toil.wdl.wdltoil
¶
Module Contents¶
Classes¶
WDL size() implementation that avoids downloading files. |
|
Standard library implementation for WDL as run on Toil. |
|
Standard library implementation to use inside a WDL task command evaluation. |
|
Standard library implementation for WDL as run on Toil, with additional |
|
Base job class for all WDL-related jobs. |
|
Job that runs a WDL task. |
|
Job that evaluates a WDL workflow node. |
|
Job that evaluates a list of WDL workflow nodes, which are in the same |
|
Job that collects the results from WDL workflow nodes and combines their |
|
Represents a graph of WDL WorkflowNodes. |
|
Job that can create more graph for a section of the wrokflow. |
|
Job that evaluates a scatter in a WDL workflow. Runs the body for each |
|
Job that takes all new bindings created in an array of input environments, |
|
Job that evaluates a conditional in a WDL workflow. |
|
Job that evaluates an entire WDL workflow. |
|
Job which evaluates an outputs section (such as for a workflow). |
|
Job that evaluates an entire WDL workflow, and returns the workflow outputs |
Functions¶
|
Run code in a context where WDL errors will be reported with pretty formatting. |
|
Create a decorator to report WDL errors with the given task message. |
|
Get potential absolute URIs to check for an imported file. |
|
Implementation of a MiniWDL read_source function that can use any |
|
Combine variable bindings from multiple predecessor tasks into one set for |
|
Log bindings to the console, even if some are still promises. |
|
Get the supertype that can hold values of all the given types. |
|
Iterate over all WDL workflow nodes in the given node, including inputs, |
|
Get the combined workflow_node_dependencies of root and everything under |
|
Encode a Toil file ID and its source path in a URI that starts with the scheme in TOIL_URI_SCHEME. |
|
Unpack a URI made by make_toil_uri to retrieve the FileID and the basename |
|
Evaluate output decls with a given bindings environment and standard library. |
|
Decide if a filename is a known kind of URL |
|
Evaluate an expression when we know the name of it. |
|
Evaluate the expression of a declaration node, or raise an error. |
|
Evaluate a bunch of expressions with names, and make them into a fresh set of bindings. inputs_dict is a mapping of |
|
If the name of the declaration is already defined in the environment, return its value. Otherwise, return the evaluated expression. |
|
Make sure all the File values embedded in the given bindings point to files |
|
Make sure all the File values embedded in the given bindings point to files |
|
Based off of WDL.runtime.task_container.add_paths from miniwdl |
|
Make sure all File values embedded in the given bindings are imported, |
|
Make sure all the File values embedded in the given bindings point to files |
|
Get the paths of all files in the bindings. Doesn't guarantee that |
|
Run all File values embedded in the given bindings through the given |
|
Run all File values' types and values embedded in the given bindings |
|
Run all File values' types and values embedded in the given binding's value through the given |
|
Run all File values embedded in the given value through the given |
|
Monkeypatch miniwdl's WDL.Value.Base.coerce() function to virtualize files when they are represented as Strings. |
|
A Toil workflow to interpret WDL input files. |
Attributes¶
- toil.wdl.wdltoil.logger¶
- toil.wdl.wdltoil.wdl_error_reporter(task, exit=False, log=logger.critical)¶
Run code in a context where WDL errors will be reported with pretty formatting.
- toil.wdl.wdltoil.F¶
- toil.wdl.wdltoil.report_wdl_errors(task, exit=False, log=logger.critical)¶
Create a decorator to report WDL errors with the given task message.
Decorator can then be applied to a function, and if a WDL error happens it will say that it could not {task}.
- toil.wdl.wdltoil.potential_absolute_uris(uri, path, importer=None)¶
Get potential absolute URIs to check for an imported file.
Given a URI or bare path, yield in turn all the URIs, with schemes, where we should actually try to find it, given that we want to search under/against the given paths or URIs, the current directory, and the given importing WDL document if any.
- async toil.wdl.wdltoil.toil_read_source(uri, path, importer)¶
Implementation of a MiniWDL read_source function that can use any filename or URL supported by Toil.
Needs to be async because MiniWDL will await its result.
- toil.wdl.wdltoil.WDLBindings¶
- toil.wdl.wdltoil.combine_bindings(all_bindings)¶
Combine variable bindings from multiple predecessor tasks into one set for the current task.
- Parameters:
all_bindings (Sequence[WDLBindings]) –
- Return type:
WDLBindings
- toil.wdl.wdltoil.log_bindings(log_function, message, all_bindings)¶
Log bindings to the console, even if some are still promises.
- Parameters:
log_function (Callable[Ellipsis, None]) – Function (like logger.info) to call to log data
message (str) – Message to log before the bindings
all_bindings (Sequence[toil.job.Promised[WDLBindings]]) – A list of bindings or promises for bindings, to log
- Return type:
None
- toil.wdl.wdltoil.get_supertype(types)¶
Get the supertype that can hold values of all the given types.
- Parameters:
types (Sequence[Optional[WDL.Type.Base]]) –
- Return type:
WDL.Type.Base
- toil.wdl.wdltoil.for_each_node(root)¶
Iterate over all WDL workflow nodes in the given node, including inputs, internal nodes of conditionals and scatters, and gather nodes.
- Parameters:
root (WDL.Tree.WorkflowNode) –
- Return type:
Iterator[WDL.Tree.WorkflowNode]
- toil.wdl.wdltoil.recursive_dependencies(root)¶
Get the combined workflow_node_dependencies of root and everything under it, which are not on anything in that subtree.
Useful because section nodes can have internal nodes with dependencies not reflected in those of the section node itself.
- Parameters:
root (WDL.Tree.WorkflowNode) –
- Return type:
Set[str]
- toil.wdl.wdltoil.TOIL_URI_SCHEME = 'toilfile:'¶
- toil.wdl.wdltoil.pack_toil_uri(file_id, dir_id, file_basename)¶
Encode a Toil file ID and its source path in a URI that starts with the scheme in TOIL_URI_SCHEME.
- Parameters:
file_id (toil.fileStores.FileID) –
dir_id (uuid.UUID) –
file_basename (str) –
- Return type:
- toil.wdl.wdltoil.unpack_toil_uri(toil_uri)¶
Unpack a URI made by make_toil_uri to retrieve the FileID and the basename (no path prefix) that the file is supposed to have.
- Parameters:
toil_uri (str) –
- Return type:
Tuple[toil.fileStores.FileID, str, str]
- toil.wdl.wdltoil.evaluate_output_decls(output_decls, all_bindings, standard_library)¶
Evaluate output decls with a given bindings environment and standard library. Creates a new bindings object that only contains the bindings from the given decls. Guarantees that each decl in output_decls can access the variables defined by the previous ones. :param all_bindings: Environment to use when evaluating decls :param output_decls: Decls to evaluate :param standard_library: Standard library :return: New bindings object with only the output_decls
- Parameters:
output_decls (List[WDL.Tree.Decl]) –
all_bindings (WDL.Env.Bindings[WDL.Value.Base]) –
standard_library (WDL.StdLib.Base) –
- Return type:
WDL.Env.Bindings[WDL.Value.Base]
- class toil.wdl.wdltoil.NonDownloadingSize¶
Bases:
digraph inheritance18fb60e6c6 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "NonDownloadingSize" [URL="#toil.wdl.wdltoil.NonDownloadingSize",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="WDL size() implementation that avoids downloading files."]; }WDL.StdLib._Size
WDL size() implementation that avoids downloading files.
MiniWDL’s default size() implementation downloads the whole file to get its size. We want to be able to get file sizes from code running on the leader, where there may not be space to download the whole file. So we override the fancy class that implements it so that we can handle sizes for FileIDs using the FileID’s stored size info.
- toil.wdl.wdltoil.is_url(filename, schemes=['http:', 'https:', 's3:', 'gs:', TOIL_URI_SCHEME])¶
Decide if a filename is a known kind of URL
- class toil.wdl.wdltoil.ToilWDLStdLibBase(file_store, execution_dir=None)¶
Bases:
digraph inheritanceabcc6f486c { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ToilWDLStdLibBase" [URL="#toil.wdl.wdltoil.ToilWDLStdLibBase",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Standard library implementation for WDL as run on Toil."]; }WDL.StdLib.Base
Standard library implementation for WDL as run on Toil.
- Parameters:
file_store (toil.fileStores.abstractFileStore.AbstractFileStore) –
execution_dir (Optional[str]) –
- class toil.wdl.wdltoil.ToilWDLStdLibTaskCommand(file_store, container)¶
Bases:
digraph inheritance0c687b1263 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ToilWDLStdLibBase" [URL="#toil.wdl.wdltoil.ToilWDLStdLibBase",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Standard library implementation for WDL as run on Toil."]; "ToilWDLStdLibTaskCommand" [URL="#toil.wdl.wdltoil.ToilWDLStdLibTaskCommand",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Standard library implementation to use inside a WDL task command evaluation."]; "ToilWDLStdLibBase" -> "ToilWDLStdLibTaskCommand" [arrowsize=0.5,style="setlinewidth(0.5)"]; }ToilWDLStdLibBase
Standard library implementation to use inside a WDL task command evaluation.
Expects all the filenames in variable bindings to be container-side paths; these are the “virtualized” filenames, while the “devirtualized” filenames are host-side paths.
- Parameters:
file_store (toil.fileStores.abstractFileStore.AbstractFileStore) –
container (WDL.runtime.task_container.TaskContainer) –
- class toil.wdl.wdltoil.ToilWDLStdLibTaskOutputs(file_store, stdout_path, stderr_path, current_directory_override=None)¶
Bases:
digraph inheritanceea7459e409 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ToilWDLStdLibBase" [URL="#toil.wdl.wdltoil.ToilWDLStdLibBase",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Standard library implementation for WDL as run on Toil."]; "ToilWDLStdLibTaskOutputs" [URL="#toil.wdl.wdltoil.ToilWDLStdLibTaskOutputs",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Standard library implementation for WDL as run on Toil, with additional"]; "ToilWDLStdLibBase" -> "ToilWDLStdLibTaskOutputs" [arrowsize=0.5,style="setlinewidth(0.5)"]; }ToilWDLStdLibBase
,WDL.StdLib.TaskOutputs
Standard library implementation for WDL as run on Toil, with additional functions only allowed in task output sections.
- Parameters:
file_store (toil.fileStores.abstractFileStore.AbstractFileStore) –
stdout_path (str) –
stderr_path (str) –
current_directory_override (Optional[str]) –
- toil.wdl.wdltoil.evaluate_named_expression(context, name, expected_type, expression, environment, stdlib)¶
Evaluate an expression when we know the name of it.
- Parameters:
context (Union[WDL.Error.SourceNode, WDL.Error.SourcePosition]) –
name (str) –
expected_type (Optional[WDL.Type.Base]) –
expression (Optional[WDL.Expr.Base]) –
environment (WDLBindings) –
stdlib (WDL.StdLib.Base) –
- Return type:
WDL.Value.Base
- toil.wdl.wdltoil.evaluate_decl(node, environment, stdlib)¶
Evaluate the expression of a declaration node, or raise an error.
- Parameters:
node (WDL.Tree.Decl) –
environment (WDLBindings) –
stdlib (WDL.StdLib.Base) –
- Return type:
WDL.Value.Base
- toil.wdl.wdltoil.evaluate_call_inputs(context, expressions, environment, stdlib, inputs_dict=None)¶
Evaluate a bunch of expressions with names, and make them into a fresh set of bindings. inputs_dict is a mapping of variable names to their expected type for the input decls in a task.
- toil.wdl.wdltoil.evaluate_defaultable_decl(node, environment, stdlib)¶
If the name of the declaration is already defined in the environment, return its value. Otherwise, return the evaluated expression.
- Parameters:
node (WDL.Tree.Decl) –
environment (WDLBindings) –
stdlib (WDL.StdLib.Base) –
- Return type:
WDL.Value.Base
- toil.wdl.wdltoil.devirtualize_files(environment, stdlib)¶
Make sure all the File values embedded in the given bindings point to files that are actually available to command line commands. The same virtual file always maps to the same devirtualized filename even with duplicates
- Parameters:
environment (WDLBindings) –
stdlib (WDL.StdLib.Base) –
- Return type:
WDLBindings
- toil.wdl.wdltoil.virtualize_files(environment, stdlib)¶
Make sure all the File values embedded in the given bindings point to files that are usable from other machines.
- Parameters:
environment (WDLBindings) –
stdlib (WDL.StdLib.Base) –
- Return type:
WDLBindings
- toil.wdl.wdltoil.add_paths(task_container, host_paths)¶
Based off of WDL.runtime.task_container.add_paths from miniwdl Maps the host path to the container paths
- Parameters:
task_container (WDL.runtime.task_container.TaskContainer) –
host_paths (Iterable[str]) –
- Return type:
None
- toil.wdl.wdltoil.import_files(environment, toil, path=None, skip_remote=False)¶
Make sure all File values embedded in the given bindings are imported, using the given Toil object.
- Parameters:
path (Optional[List[str]]) – If set, try resolving input location relative to the URLs or directories in this list.
skip_remote (bool) – If set, don’t try to import files from remote locations. Leave them as URIs.
environment (WDLBindings) –
toil (toil.common.Toil) –
- Return type:
WDLBindings
- toil.wdl.wdltoil.drop_missing_files(environment, current_directory_override=None)¶
Make sure all the File values embedded in the given bindings point to files that exist, or are null.
Files must not be virtualized.
- Parameters:
environment (WDLBindings) –
current_directory_override (Optional[str]) –
- Return type:
WDLBindings
- toil.wdl.wdltoil.get_file_paths_in_bindings(environment)¶
Get the paths of all files in the bindings. Doesn’t guarantee that duplicates are removed.
TODO: Duplicative with WDL.runtime.task._fspaths, except that is internal and supports Direcotry objects.
- Parameters:
environment (WDLBindings) –
- Return type:
List[str]
- toil.wdl.wdltoil.map_over_typed_files_in_bindings(environment, transform)¶
Run all File values embedded in the given bindings through the given transformation function.
TODO: Replace with WDL.Value.rewrite_env_paths or WDL.Value.rewrite_files
- toil.wdl.wdltoil.map_over_files_in_bindings(bindings, transform)¶
Run all File values’ types and values embedded in the given bindings through the given transformation function.
TODO: Replace with WDL.Value.rewrite_env_paths or WDL.Value.rewrite_files
- toil.wdl.wdltoil.map_over_typed_files_in_binding(binding, transform)¶
Run all File values’ types and values embedded in the given binding’s value through the given transformation function.
- toil.wdl.wdltoil.map_over_typed_files_in_value(value, transform)¶
Run all File values embedded in the given value through the given transformation function.
If the transform returns None, the file value is changed to Null.
The transform has access to the type information for the value, so it knows if it may return None, depending on if the value is optional or not.
The transform is allowed to return None only if the mapping result won’t actually be used, to allow for scans. So error checking needs to be part of the transform itself.
- class toil.wdl.wdltoil.WDLBaseJob(execution_dir=None, **kwargs)¶
Bases:
digraph inheritance4cdbc4ebca { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Job" [URL="../../job/index.html#toil.job.Job",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class represents a unit of work in toil."]; "WDLBaseJob" [URL="#toil.wdl.wdltoil.WDLBaseJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base job class for all WDL-related jobs."]; "Job" -> "WDLBaseJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; }toil.job.Job
Base job class for all WDL-related jobs.
Responsible for post-processing returned bindings, to do things like add in null values for things not defined in a section. Post-processing operations can be added onto any job before it is saved, and will be applied as long as the job’s run method calls postprocess().
- Parameters:
execution_dir (Optional[str]) –
kwargs (Any) –
- run(file_store)¶
Run a WDL-related job.
Remember to decorate non-trivial overrides with
report_wdl_errors()
.- Parameters:
file_store (toil.fileStores.abstractFileStore.AbstractFileStore) –
- Return type:
Any
- then_underlay(underlay)¶
Apply an underlay of backup bindings to the result.
- Parameters:
underlay (toil.job.Promised[WDLBindings]) –
- Return type:
None
- then_remove(remove)¶
Remove the given bindings from the result.
- Parameters:
remove (toil.job.Promised[WDLBindings]) –
- Return type:
None
- then_namespace(namespace)¶
Put the result bindings into a namespace.
- Parameters:
namespace (str) –
- Return type:
None
- then_overlay(overlay)¶
Overlay the given bindings on top of the (possibly namespaced) result.
- Parameters:
overlay (toil.job.Promised[WDLBindings]) –
- Return type:
None
- postprocess(bindings)¶
Apply queued changes to bindings.
Should be applied by subclasses’ run() implementations to their return values.
- Parameters:
bindings (WDLBindings) –
- Return type:
WDLBindings
- defer_postprocessing(other)¶
Give our postprocessing steps to a different job.
Use this when you are returning a promise for bindings, on the job that issues the promise.
- Parameters:
other (WDLBaseJob) –
- Return type:
None
- class toil.wdl.wdltoil.WDLTaskJob(task, prev_node_results, task_id, namespace, **kwargs)¶
Bases:
digraph inheritance047b2f0e68 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Job" [URL="../../job/index.html#toil.job.Job",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class represents a unit of work in toil."]; "WDLBaseJob" [URL="#toil.wdl.wdltoil.WDLBaseJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base job class for all WDL-related jobs."]; "Job" -> "WDLBaseJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLTaskJob" [URL="#toil.wdl.wdltoil.WDLTaskJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job that runs a WDL task."]; "WDLBaseJob" -> "WDLTaskJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; }WDLBaseJob
Job that runs a WDL task.
Responsible for evaluating the input declarations for unspecified inputs, evaluating the runtime section, re-scheduling if resources are not available, running any command, and evaluating the outputs.
All bindings are in terms of task-internal names.
- Parameters:
- run(file_store)¶
Actually run the task.
- Parameters:
file_store (toil.fileStores.abstractFileStore.AbstractFileStore) –
- Return type:
toil.job.Promised[WDLBindings]
- class toil.wdl.wdltoil.WDLWorkflowNodeJob(node, prev_node_results, namespace, execution_dir=None, **kwargs)¶
Bases:
digraph inheritancebabf818b55 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Job" [URL="../../job/index.html#toil.job.Job",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class represents a unit of work in toil."]; "WDLBaseJob" [URL="#toil.wdl.wdltoil.WDLBaseJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base job class for all WDL-related jobs."]; "Job" -> "WDLBaseJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLWorkflowNodeJob" [URL="#toil.wdl.wdltoil.WDLWorkflowNodeJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job that evaluates a WDL workflow node."]; "WDLBaseJob" -> "WDLWorkflowNodeJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; }WDLBaseJob
Job that evaluates a WDL workflow node.
- Parameters:
- run(file_store)¶
Actually execute the workflow node.
- Parameters:
file_store (toil.fileStores.abstractFileStore.AbstractFileStore) –
- Return type:
toil.job.Promised[WDLBindings]
- class toil.wdl.wdltoil.WDLWorkflowNodeListJob(nodes, prev_node_results, namespace, execution_dir=None, **kwargs)¶
Bases:
digraph inheritancef805deaac7 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Job" [URL="../../job/index.html#toil.job.Job",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class represents a unit of work in toil."]; "WDLBaseJob" [URL="#toil.wdl.wdltoil.WDLBaseJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base job class for all WDL-related jobs."]; "Job" -> "WDLBaseJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLWorkflowNodeListJob" [URL="#toil.wdl.wdltoil.WDLWorkflowNodeListJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job that evaluates a list of WDL workflow nodes, which are in the same"]; "WDLBaseJob" -> "WDLWorkflowNodeListJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; }WDLBaseJob
Job that evaluates a list of WDL workflow nodes, which are in the same scope and in a topological dependency order, and which do not call out to any other workflows or tasks or sections.
- Parameters:
- run(file_store)¶
Actually execute the workflow nodes.
- Parameters:
file_store (toil.fileStores.abstractFileStore.AbstractFileStore) –
- Return type:
toil.job.Promised[WDLBindings]
- class toil.wdl.wdltoil.WDLCombineBindingsJob(prev_node_results, **kwargs)¶
Bases:
digraph inheritance4e4b61ff09 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Job" [URL="../../job/index.html#toil.job.Job",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class represents a unit of work in toil."]; "WDLBaseJob" [URL="#toil.wdl.wdltoil.WDLBaseJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base job class for all WDL-related jobs."]; "Job" -> "WDLBaseJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLCombineBindingsJob" [URL="#toil.wdl.wdltoil.WDLCombineBindingsJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job that collects the results from WDL workflow nodes and combines their"]; "WDLBaseJob" -> "WDLCombineBindingsJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; }WDLBaseJob
Job that collects the results from WDL workflow nodes and combines their environment changes.
- Parameters:
prev_node_results (Sequence[toil.job.Promised[WDLBindings]]) –
kwargs (Any) –
- run(file_store)¶
Aggregate incoming results.
- Parameters:
file_store (toil.fileStores.abstractFileStore.AbstractFileStore) –
- Return type:
WDLBindings
- class toil.wdl.wdltoil.WDLWorkflowGraph(nodes)¶
Represents a graph of WDL WorkflowNodes.
Operates at a certain level of instantiation (i.e. sub-sections are represented by single nodes).
Assumes all relevant nodes are provided; dependencies outside the provided nodes are assumed to be satisfied already.
- Parameters:
nodes (Sequence[WDL.Tree.WorkflowNode]) –
- real_id(node_id)¶
Map multiple IDs for what we consider the same node to one ID.
This elides/resolves gathers.
- is_decl(node_id)¶
Return True if a node represents a WDL declaration, and false otherwise.
- get_dependencies(node_id)¶
Get all the nodes that a node depends on, recursively (into the node if it has a body) but not transitively.
Produces dependencies after resolving gathers and internal-to-section dependencies, on nodes that are also in this graph.
- get_transitive_dependencies(node_id)¶
Get all the nodes that a node depends on, transitively.
- class toil.wdl.wdltoil.WDLSectionJob(namespace, execution_dir=None, **kwargs)¶
Bases:
digraph inheritance5178f6d245 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Job" [URL="../../job/index.html#toil.job.Job",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class represents a unit of work in toil."]; "WDLBaseJob" [URL="#toil.wdl.wdltoil.WDLBaseJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base job class for all WDL-related jobs."]; "Job" -> "WDLBaseJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLSectionJob" [URL="#toil.wdl.wdltoil.WDLSectionJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job that can create more graph for a section of the wrokflow."]; "WDLBaseJob" -> "WDLSectionJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; }WDLBaseJob
Job that can create more graph for a section of the wrokflow.
- static coalesce_nodes(order, section_graph)¶
Given a topological order of WDL workflow node IDs, produce a list of lists of IDs, still in topological order, where each list of IDs can be run under a single Toil job.
- Parameters:
order (List[str]) –
section_graph (WDLWorkflowGraph) –
- Return type:
List[List[str]]
- create_subgraph(nodes, gather_nodes, environment, local_environment=None)¶
Make a Toil job to evaluate a subgraph inside a workflow or workflow section.
- Returns:
a child Job that will return the aggregated environment after running all the things in the section.
- Parameters:
gather_nodes (Sequence[WDL.Tree.Gather]) – Names exposed by these will always be defined with something, even if the code that defines them does not actually run.
environment (WDLBindings) – Bindings in this environment will be used to evaluate the subgraph and will be passed through.
local_environment (Optional[WDLBindings]) – Bindings in this environment will be used to evaluate the subgraph but will go out of scope at the end of the section.
nodes (Sequence[WDL.Tree.WorkflowNode]) –
- Return type:
- make_gather_bindings(gathers, undefined)¶
Given a collection of Gathers, create bindings from every identifier gathered, to the given “undefined” placeholder (which would be Null for a single execution of the body, or an empty array for a completely unexecuted scatter).
These bindings can be overlaid with bindings from the actual execution, so that references to names defined in unexecuted code get a proper default undefined value, and not a KeyError at runtime.
The information to do this comes from MiniWDL’s “gathers” system: <https://miniwdl.readthedocs.io/en/latest/WDL.html#WDL.Tree.WorkflowSection.gathers>
TODO: This approach will scale O(n^2) when run on n nested conditionals, because generating these bindings for the outer conditional will visit all the bindings from the inner ones.
- Parameters:
gathers (Sequence[WDL.Tree.Gather]) –
undefined (WDL.Value.Base) –
- Return type:
WDLBindings
- class toil.wdl.wdltoil.WDLScatterJob(scatter, prev_node_results, namespace, execution_dir=None, **kwargs)¶
Bases:
digraph inheritance94195494bd { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Job" [URL="../../job/index.html#toil.job.Job",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class represents a unit of work in toil."]; "WDLBaseJob" [URL="#toil.wdl.wdltoil.WDLBaseJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base job class for all WDL-related jobs."]; "Job" -> "WDLBaseJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLScatterJob" [URL="#toil.wdl.wdltoil.WDLScatterJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job that evaluates a scatter in a WDL workflow. Runs the body for each"]; "WDLSectionJob" -> "WDLScatterJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLSectionJob" [URL="#toil.wdl.wdltoil.WDLSectionJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job that can create more graph for a section of the wrokflow."]; "WDLBaseJob" -> "WDLSectionJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; }WDLSectionJob
Job that evaluates a scatter in a WDL workflow. Runs the body for each value in an array, and makes arrays of the new bindings created in each instance of the body. If an instance of the body doesn’t create a binding, it gets a null value in the corresponding array.
- Parameters:
- run(file_store)¶
Run the scatter.
- Parameters:
file_store (toil.fileStores.abstractFileStore.AbstractFileStore) –
- Return type:
toil.job.Promised[WDLBindings]
- class toil.wdl.wdltoil.WDLArrayBindingsJob(input_bindings, base_bindings, **kwargs)¶
Bases:
digraph inheritance313e4f5bc5 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Job" [URL="../../job/index.html#toil.job.Job",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class represents a unit of work in toil."]; "WDLArrayBindingsJob" [URL="#toil.wdl.wdltoil.WDLArrayBindingsJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job that takes all new bindings created in an array of input environments,"]; "WDLBaseJob" -> "WDLArrayBindingsJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLBaseJob" [URL="#toil.wdl.wdltoil.WDLBaseJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base job class for all WDL-related jobs."]; "Job" -> "WDLBaseJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; }WDLBaseJob
Job that takes all new bindings created in an array of input environments, relative to a base environment, and produces bindings where each new binding name is bound to an array of the values in all the input environments.
Useful for producing the results of a scatter.
- Parameters:
input_bindings (Sequence[toil.job.Promised[WDLBindings]]) –
base_bindings (WDLBindings) –
kwargs (Any) –
- run(file_store)¶
Actually produce the array-ified bindings now that promised values are available.
- Parameters:
file_store (toil.fileStores.abstractFileStore.AbstractFileStore) –
- Return type:
WDLBindings
- class toil.wdl.wdltoil.WDLConditionalJob(conditional, prev_node_results, namespace, execution_dir=None, **kwargs)¶
Bases:
digraph inheritance7fe226f5b0 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Job" [URL="../../job/index.html#toil.job.Job",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class represents a unit of work in toil."]; "WDLBaseJob" [URL="#toil.wdl.wdltoil.WDLBaseJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base job class for all WDL-related jobs."]; "Job" -> "WDLBaseJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLConditionalJob" [URL="#toil.wdl.wdltoil.WDLConditionalJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job that evaluates a conditional in a WDL workflow."]; "WDLSectionJob" -> "WDLConditionalJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLSectionJob" [URL="#toil.wdl.wdltoil.WDLSectionJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job that can create more graph for a section of the wrokflow."]; "WDLBaseJob" -> "WDLSectionJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; }WDLSectionJob
Job that evaluates a conditional in a WDL workflow.
- Parameters:
- run(file_store)¶
Run the conditional.
- Parameters:
file_store (toil.fileStores.abstractFileStore.AbstractFileStore) –
- Return type:
toil.job.Promised[WDLBindings]
- class toil.wdl.wdltoil.WDLWorkflowJob(workflow, prev_node_results, workflow_id, namespace, execution_dir=None, **kwargs)¶
Bases:
digraph inheritancede054d5163 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Job" [URL="../../job/index.html#toil.job.Job",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class represents a unit of work in toil."]; "WDLBaseJob" [URL="#toil.wdl.wdltoil.WDLBaseJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base job class for all WDL-related jobs."]; "Job" -> "WDLBaseJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLSectionJob" [URL="#toil.wdl.wdltoil.WDLSectionJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job that can create more graph for a section of the wrokflow."]; "WDLBaseJob" -> "WDLSectionJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLWorkflowJob" [URL="#toil.wdl.wdltoil.WDLWorkflowJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job that evaluates an entire WDL workflow."]; "WDLSectionJob" -> "WDLWorkflowJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; }WDLSectionJob
Job that evaluates an entire WDL workflow.
- Parameters:
- run(file_store)¶
Run the workflow. Return the result of the workflow.
- Parameters:
file_store (toil.fileStores.abstractFileStore.AbstractFileStore) –
- Return type:
toil.job.Promised[WDLBindings]
- class toil.wdl.wdltoil.WDLOutputsJob(workflow, bindings, execution_dir=None, **kwargs)¶
Bases:
digraph inheritance55a641ce2d { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Job" [URL="../../job/index.html#toil.job.Job",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class represents a unit of work in toil."]; "WDLBaseJob" [URL="#toil.wdl.wdltoil.WDLBaseJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base job class for all WDL-related jobs."]; "Job" -> "WDLBaseJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLOutputsJob" [URL="#toil.wdl.wdltoil.WDLOutputsJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job which evaluates an outputs section (such as for a workflow)."]; "WDLBaseJob" -> "WDLOutputsJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; }WDLBaseJob
Job which evaluates an outputs section (such as for a workflow).
Returns an environment with just the outputs bound, in no namespace.
- Parameters:
workflow (WDL.Tree.Workflow) –
bindings (toil.job.Promised[WDLBindings]) –
execution_dir (Optional[str]) –
kwargs (Any) –
- run(file_store)¶
Make bindings for the outputs.
- Parameters:
file_store (toil.fileStores.abstractFileStore.AbstractFileStore) –
- Return type:
WDLBindings
- class toil.wdl.wdltoil.WDLRootJob(workflow, inputs, execution_dir=None, **kwargs)¶
Bases:
digraph inheritance4077f945ac { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Job" [URL="../../job/index.html#toil.job.Job",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Class represents a unit of work in toil."]; "WDLBaseJob" [URL="#toil.wdl.wdltoil.WDLBaseJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Base job class for all WDL-related jobs."]; "Job" -> "WDLBaseJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLRootJob" [URL="#toil.wdl.wdltoil.WDLRootJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job that evaluates an entire WDL workflow, and returns the workflow outputs"]; "WDLSectionJob" -> "WDLRootJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; "WDLSectionJob" [URL="#toil.wdl.wdltoil.WDLSectionJob",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Job that can create more graph for a section of the wrokflow."]; "WDLBaseJob" -> "WDLSectionJob" [arrowsize=0.5,style="setlinewidth(0.5)"]; }WDLSectionJob
Job that evaluates an entire WDL workflow, and returns the workflow outputs namespaced with the workflow name. Inputs may or may not be namespaced with the workflow name; both forms are accepted.
- Parameters:
workflow (WDL.Tree.Workflow) –
inputs (WDLBindings) –
execution_dir (Optional[str]) –
kwargs (Any) –
- run(file_store)¶
Actually build the subgraph.
- Parameters:
file_store (toil.fileStores.abstractFileStore.AbstractFileStore) –
- Return type:
toil.job.Promised[WDLBindings]
- toil.wdl.wdltoil.monkeypatch_coerce(standard_library)¶
Monkeypatch miniwdl’s WDL.Value.Base.coerce() function to virtualize files when they are represented as Strings. Calls _virtualize_filename from a given standard library object. :param standard_library: a standard library object :return
- Parameters:
standard_library (ToilWDLStdLibBase) –
- Return type:
Generator[None, None, None]
- toil.wdl.wdltoil.main()¶
A Toil workflow to interpret WDL input files.
- Return type:
None