toil.wdl.versions.draft2
¶
Module Contents¶
Classes¶
AnalyzeWDL implementation for the draft-2 version. |
Attributes¶
- toil.wdl.versions.draft2.logger¶
- class toil.wdl.versions.draft2.AnalyzeDraft2WDL(wdl_file)¶
Bases:
toil.wdl.wdl_analysis.AnalyzeWDL
AnalyzeWDL implementation for the draft-2 version.
- Parameters
wdl_file (str) –
- analyze()¶
Analyzes the WDL file passed into the constructor and generates the two intermediate data structures: self.workflows_dictionary and self.tasks_dictionary.
- Returns
Returns nothing.
- write_AST(out_dir=None)¶
Writes a file with the AST for a wdl file in the out_dir.
- find_asts(ast_root, name)¶
Finds an AST node with the given name and the entire subtree under it. A function borrowed from scottfrazer. Thank you Scott Frazer!
- Parameters
ast_root – The WDL AST. The whole thing generally, but really any portion that you wish to search.
name – The name of the subtree you’re looking for, like “Task”.
- Returns
nodes representing the AST subtrees matching the “name” given.
- create_tasks_dict(ast)¶
Parse each “Task” in the AST. This will create self.tasks_dictionary, where each task name is a key.
- Returns
Creates the self.tasks_dictionary necessary for much of the
parser. Returning it is only necessary for unittests.
- parse_task(task)¶
Parses a WDL task AST subtree.
Currently looks at and parses 4 sections: 1. Declarations (e.g. string x = ‘helloworld’) 2. Commandline (a bash command with dynamic variables inserted) 3. Runtime (docker image; disk; CPU; RAM; etc.) 4. Outputs (expected return values/files)
- Parameters
task – An AST subtree of a WDL “Task”.
- Returns
Returns nothing but adds a task to the self.tasks_dictionary
necessary for much of the parser.
- parse_task_rawcommand_attributes(code_snippet)¶
- Parameters
code_snippet –
- Returns
- parse_task_rawcommand(rawcommand_subAST)¶
Parses the rawcommand section of the WDL task AST subtree.
Task “rawcommands” are divided into many parts. There are 2 types of parts: normal strings, & variables that can serve as changeable inputs.
- The following example command:
‘echo ${variable1} ${variable2} > output_file.txt’
- Has 5 parts:
Normal String: ‘echo ‘ Variable Input: variable1 Normal String: ‘ ‘ Variable Input: variable2 Normal String: ‘ > output_file.txt’
Variables can also have additional conditions, like ‘sep’, which is like the python ‘’.join() function and in WDL looks like: ${sep=” -V ” GVCFs} and would be translated as: ‘ -V ‘.join(GVCFs).
- Parameters
rawcommand_subAST – A subAST representing some bash command.
- Returns
A list=[] of tuples=() representing the parts of the command: e.g. [(command_var, command_type, additional_conditions_list), …]
- Where: command_var = ‘GVCFs’
command_type = ‘variable’ command_actions = {‘sep’: ‘ -V ‘}
- modify_cmd_expr_w_attributes(code_expr, code_attr)¶
- Parameters
code_expr –
code_attr –
- Returns
- parse_task_runtime_key(i)¶
- Parameters
runtime_subAST –
- Returns
- parse_task_runtime(runtime_subAST)¶
Parses the runtime section of the WDL task AST subtree.
The task “runtime” section currently supports context fields for a docker container, CPU resources, RAM resources, and disk resources.
- Parameters
runtime_subAST – A subAST representing runtime parameters.
- Returns
A list=[] of runtime attributes, for example: runtime_attributes = [(‘docker’,’quay.io/encode-dcc/map:v1.0’),
(‘cpu’,’2’), (‘memory’,’17.1 GB’), (‘disks’,’local-disk 420 HDD’)]
- parse_task_outputs(i)¶
Parse the WDL output section.
Outputs are like declarations, with a type, name, and value. Examples:
Simple Cases¶
- ‘Int num = 7’
var_name: ‘num’ var_type: ‘Int’ var_value: 7
- String idea = ‘Lab grown golden eagle burgers.’
var_name: ‘idea’ var_type: ‘String’ var_value: ‘Lab grown golden eagle burgers.’
- File ideaFile = ‘goldenEagleStemCellStartUpDisrupt.txt’
var_name: ‘ideaFile’ var_type: ‘File’ var_value: ‘goldenEagleStemCellStartUpDisrupt.txt’
More Abstract Cases¶
- Array[File] allOfMyTerribleIdeas = glob(*.txt)[0]
var_name: ‘allOfMyTerribleIdeas’ var_type**: ‘File’ var_value: [*.txt] var_actions: {‘index_lookup’: ‘0’, ‘glob’: ‘None’}
**toilwdl.py converts ‘Array[File]’ to ‘ArrayFile’
- return
output_array representing outputs generated by the job/task: e.g. x = [(var_name, var_type, var_value, var_actions), …]
- translate_wdl_string_to_python_string(some_string)¶
Parses a string representing a given job’s output filename into something python can read. Replaces ${string}’s with normal variables and the rest with normal strings all concatenated with ‘ + ‘.
Will not work with additional parameters, such as: ${default=”foo” bar} or ${true=”foo” false=”bar” Boolean baz}
This method expects to be passed only strings with some combination of “${abc}” and “abc” blocks.
- Parameters
job – A list such that: (job priority #, job ID #, Job Skeleton Name, Job Alias)
some_string – e.g. ‘${sampleName}.vcf’
- Returns
output_string, e.g. ‘sampleName + “.vcf”’
- create_workflows_dict(ast)¶
Parse each “Workflow” in the AST. This will create self.workflows_dictionary, where each called job is a tuple key of the form: (priority#, job#, name, alias).
- Returns
Creates the self.workflows_dictionary necessary for much of the
parser. Returning it is only necessary for unittests.
- parse_workflow(workflow)¶
Parses a WDL workflow AST subtree.
Returns nothing but creates the self.workflows_dictionary necessary for much of the parser.
- Parameters
workflow – An AST subtree of a WDL “Workflow”.
- Returns
Returns nothing but adds a workflow to the self.workflows_dictionary necessary for much of the parser.
- parse_workflow_body(i)¶
Currently looks at and parses 3 sections: 1. Declarations (e.g. String x = ‘helloworld’) 2. Calls (similar to a python def) 3. Scatter (which expects to map to a Call or multiple Calls) 4. Conditionals
- parse_workflow_if(ifAST)¶
- parse_workflow_if_expression(i)¶
- parse_workflow_scatter(scatterAST)¶
- parse_workflow_scatter_item(i)¶
- parse_workflow_scatter_collection(i)¶
- parse_declaration(ast)¶
Parses a WDL declaration AST subtree into a Python tuple.
Examples:
String my_name String your_name Int two_chains_i_mean_names = 0
- Parameters
ast – Some subAST representing a task declaration like: ‘String file_name’
- Returns
var_name, var_type, var_value Example:
Input subAST representing: ‘String file_name’ Output: var_name=’file_name’, var_type=’String’, var_value=None
- parse_declaration_name(nameAST)¶
Required.
Nothing fancy here. Just the name of the workflow function. For example: “rnaseqexample” would be the following wdl workflow’s name:
workflow rnaseqexample {File y; call a {inputs: y}; call b;} task a {File y} task b {command{“echo ‘ATCG’”}}
- Parameters
nameAST –
- Returns
- parse_declaration_type(typeAST)¶
Required.
Currently supported: Types are: Boolean, Float, Int, File, String, Array[subtype],
Pair[subtype, subtype], and Map[subtype, subtype].
- OptionalTypes are: Boolean?, Float?, Int?, File?, String?, Array[subtype]?,
Pair[subtype, subtype]?, and Map[subtype, subtype]?.
Python is not typed, so we don’t need typing except to identify type: “File”, which Toil needs to import, so we recursively travel down to the innermost type which will tell us if the variables are files that need importing.
For Pair and Map compound types, we recursively travel down the subtypes and store them as attributes of a WDLType string. This way, the type structure is preserved, which will allow us to import files appropriately.
- Parameters
typeAST –
- Returns
a WDLType instance
- parse_declaration_expressn(expressionAST, es)¶
Expressions are optional. Workflow declaration valid examples:
File x
or
File x = ‘/x/x.tmp’
- Parameters
expressionAST –
- Returns
- parse_declaration_expressn_logicalnot(exprssn, es)¶
- parse_declaration_expressn_arraymaplookup(lhsAST, rhsAST, es)¶
- Parameters
lhsAST –
rhsAST –
es –
- Returns
- parse_declaration_expressn_memberaccess(lhsAST, rhsAST, es)¶
Instead of “Class.variablename”, use “Class.rv(‘variablename’)”.
- Parameters
lhsAST –
rhsAST –
es –
- Returns
- parse_declaration_expressn_ternaryif(cond, iftrue, iffalse, es)¶
Classic if statement. This needs to be rearranged.
In wdl, this looks like: if <condition> then <iftrue> else <iffalse>
In python, this needs to be: <iftrue> if <condition> else <iffalse>
- Parameters
cond –
iftrue –
iffalse –
es –
- Returns
- parse_declaration_expressn_tupleliteral(values, es)¶
Same in python. Just a parenthesis enclosed tuple.
- Parameters
values –
es –
- Returns
- parse_declaration_expressn_arrayliteral(values, es)¶
Same in python. Just a square bracket enclosed array.
- Parameters
values –
es –
- Returns
- parse_declaration_expressn_operator(lhsAST, rhsAST, es, operator)¶
Simply joins the left and right hand arguments lhs and rhs with an operator.
- Parameters
lhsAST –
rhsAST –
es –
operator –
- Returns
- parse_declaration_expressn_fncall(name, params, es)¶
Parses out cromwell’s built-in function calls.
Some of these are special and need minor adjustments, for example size() requires a fileStore.
- Parameters
name –
params –
es –
- Returns
- parse_declaration_expressn_fncall_normalparams(params)¶
- parse_workflow_call_taskname(i)¶
Required.
- Parameters
i –
- Returns
- parse_workflow_call_taskalias(i)¶
Required.
- Parameters
i –
- Returns
- parse_workflow_call_body_declarations(i)¶
Have not seen this used, so expects to return “[]”.
- Parameters
i –
- Returns
- parse_workflow_call_body_io(i)¶
Required.
- Parameters
i –
- Returns
- parse_workflow_call_body_io_map(i)¶
Required.
- Parameters
i –
- Returns
- parse_workflow_call_body(i)¶
Required.
- Parameters
i –
- Returns
- parse_workflow_call(i)¶
Parses a WDL workflow call AST subtree to give the variable mappings for that particular job/task “call”.
- Parameters
i – WDL workflow job object
- Returns
python dictionary of io mappings for that job call