toil.wdl.versions.draft2

Module Contents

Classes

AnalyzeDraft2WDL

AnalyzeWDL implementation for the draft-2 version.

Attributes

logger

toil.wdl.versions.draft2.logger
class toil.wdl.versions.draft2.AnalyzeDraft2WDL(wdl_file)

Bases: toil.wdl.wdl_analysis.AnalyzeWDL

Inheritance diagram of toil.wdl.versions.draft2.AnalyzeDraft2WDL

AnalyzeWDL implementation for the draft-2 version.

Parameters

wdl_file (str) –

property version: str

Returns the version of the WDL document as a string.

Return type

str

analyze()

Analyzes the WDL file passed into the constructor and generates the two intermediate data structures: self.workflows_dictionary and self.tasks_dictionary.

Returns

Returns nothing.

write_AST(out_dir=None)

Writes a file with the AST for a wdl file in the out_dir.

find_asts(ast_root, name)

Finds an AST node with the given name and the entire subtree under it. A function borrowed from scottfrazer. Thank you Scott Frazer!

Parameters

  • ast_root – The WDL AST. The whole thing generally, but really any portion that you wish to search.

  • name – The name of the subtree you’re looking for, like “Task”.

Returns

nodes representing the AST subtrees matching the “name” given.

create_tasks_dict(ast)

Parse each “Task” in the AST. This will create self.tasks_dictionary, where each task name is a key.

Returns

Creates the self.tasks_dictionary necessary for much of the

parser. Returning it is only necessary for unittests.

parse_task(task)

Parses a WDL task AST subtree.

Currently looks at and parses 4 sections: 1. Declarations (e.g. string x = ‘helloworld’) 2. Commandline (a bash command with dynamic variables inserted) 3. Runtime (docker image; disk; CPU; RAM; etc.) 4. Outputs (expected return values/files)

Parameters

task – An AST subtree of a WDL “Task”.

Returns

Returns nothing but adds a task to the self.tasks_dictionary

necessary for much of the parser.

parse_task_rawcommand_attributes(code_snippet)
Parameters

code_snippet –

Returns

parse_task_rawcommand(rawcommand_subAST)

Parses the rawcommand section of the WDL task AST subtree.

Task “rawcommands” are divided into many parts. There are 2 types of parts: normal strings, & variables that can serve as changeable inputs.

The following example command:

‘echo ${variable1} ${variable2} > output_file.txt’

Has 5 parts:

Normal String: ‘echo ‘ Variable Input: variable1 Normal String: ‘ ‘ Variable Input: variable2 Normal String: ‘ > output_file.txt’

Variables can also have additional conditions, like ‘sep’, which is like the python ‘’.join() function and in WDL looks like: ${sep=” -V ” GVCFs} and would be translated as: ‘ -V ‘.join(GVCFs).

Parameters

rawcommand_subAST – A subAST representing some bash command.

Returns

A list=[] of tuples=() representing the parts of the command: e.g. [(command_var, command_type, additional_conditions_list), …]

Where: command_var = ‘GVCFs’

command_type = ‘variable’ command_actions = {‘sep’: ‘ -V ‘}

modify_cmd_expr_w_attributes(code_expr, code_attr)
Parameters

  • code_expr –

  • code_attr –

Returns

parse_task_runtime_key(i)
Parameters

runtime_subAST –

Returns

parse_task_runtime(runtime_subAST)

Parses the runtime section of the WDL task AST subtree.

The task “runtime” section currently supports context fields for a docker container, CPU resources, RAM resources, and disk resources.

Parameters

runtime_subAST – A subAST representing runtime parameters.

Returns

A list=[] of runtime attributes, for example: runtime_attributes = [(‘docker’,’quay.io/encode-dcc/map:v1.0’),

(‘cpu’,’2’), (‘memory’,’17.1 GB’), (‘disks’,’local-disk 420 HDD’)]

parse_task_outputs(i)

Parse the WDL output section.

Outputs are like declarations, with a type, name, and value. Examples:

Simple Cases

‘Int num = 7’

var_name: ‘num’ var_type: ‘Int’ var_value: 7

String idea = ‘Lab grown golden eagle burgers.’

var_name: ‘idea’ var_type: ‘String’ var_value: ‘Lab grown golden eagle burgers.’

File ideaFile = ‘goldenEagleStemCellStartUpDisrupt.txt’

var_name: ‘ideaFile’ var_type: ‘File’ var_value: ‘goldenEagleStemCellStartUpDisrupt.txt’

More Abstract Cases

Array[File] allOfMyTerribleIdeas = glob(*.txt)[0]

var_name: ‘allOfMyTerribleIdeas’ var_type**: ‘File’ var_value: [*.txt] var_actions: {‘index_lookup’: ‘0’, ‘glob’: ‘None’}

**toilwdl.py converts ‘Array[File]’ to ‘ArrayFile’

return

output_array representing outputs generated by the job/task: e.g. x = [(var_name, var_type, var_value, var_actions), …]

translate_wdl_string_to_python_string(some_string)

Parses a string representing a given job’s output filename into something python can read. Replaces ${string}’s with normal variables and the rest with normal strings all concatenated with ‘ + ‘.

Will not work with additional parameters, such as: ${default=”foo” bar} or ${true=”foo” false=”bar” Boolean baz}

This method expects to be passed only strings with some combination of “${abc}” and “abc” blocks.

Parameters

  • job – A list such that: (job priority #, job ID #, Job Skeleton Name, Job Alias)

  • some_string – e.g. ‘${sampleName}.vcf’

Returns

output_string, e.g. ‘sampleName + “.vcf”’

create_workflows_dict(ast)

Parse each “Workflow” in the AST. This will create self.workflows_dictionary, where each called job is a tuple key of the form: (priority#, job#, name, alias).

Returns

Creates the self.workflows_dictionary necessary for much of the

parser. Returning it is only necessary for unittests.

parse_workflow(workflow)

Parses a WDL workflow AST subtree.

Returns nothing but creates the self.workflows_dictionary necessary for much of the parser.

Parameters

workflow – An AST subtree of a WDL “Workflow”.

Returns

Returns nothing but adds a workflow to the self.workflows_dictionary necessary for much of the parser.

parse_workflow_body(i)

Currently looks at and parses 3 sections: 1. Declarations (e.g. String x = ‘helloworld’) 2. Calls (similar to a python def) 3. Scatter (which expects to map to a Call or multiple Calls) 4. Conditionals

parse_workflow_if(ifAST)
parse_workflow_if_expression(i)
parse_workflow_scatter(scatterAST)
parse_workflow_scatter_item(i)
parse_workflow_scatter_collection(i)
parse_declaration(ast)

Parses a WDL declaration AST subtree into a Python tuple.

Examples:

String my_name String your_name Int two_chains_i_mean_names = 0

Parameters

ast – Some subAST representing a task declaration like: ‘String file_name’

Returns

var_name, var_type, var_value Example:

Input subAST representing: ‘String file_name’ Output: var_name=’file_name’, var_type=’String’, var_value=None

parse_declaration_name(nameAST)

Required.

Nothing fancy here. Just the name of the workflow function. For example: “rnaseqexample” would be the following wdl workflow’s name:

workflow rnaseqexample {File y; call a {inputs: y}; call b;} task a {File y} task b {command{“echo ‘ATCG’”}}

Parameters

nameAST –

Returns

parse_declaration_type(typeAST)

Required.

Currently supported: Types are: Boolean, Float, Int, File, String, Array[subtype],

Pair[subtype, subtype], and Map[subtype, subtype].

OptionalTypes are: Boolean?, Float?, Int?, File?, String?, Array[subtype]?,

Pair[subtype, subtype]?, and Map[subtype, subtype]?.

Python is not typed, so we don’t need typing except to identify type: “File”, which Toil needs to import, so we recursively travel down to the innermost type which will tell us if the variables are files that need importing.

For Pair and Map compound types, we recursively travel down the subtypes and store them as attributes of a WDLType string. This way, the type structure is preserved, which will allow us to import files appropriately.

Parameters

typeAST –

Returns

a WDLType instance

parse_declaration_expressn(expressionAST, es)

Expressions are optional. Workflow declaration valid examples:

File x

or

File x = ‘/x/x.tmp’

Parameters

expressionAST –

Returns

parse_declaration_expressn_logicalnot(exprssn, es)
parse_declaration_expressn_arraymaplookup(lhsAST, rhsAST, es)
Parameters

  • lhsAST –

  • rhsAST –

  • es –

Returns

parse_declaration_expressn_memberaccess(lhsAST, rhsAST, es)

Instead of “Class.variablename”, use “Class.rv(‘variablename’)”.

Parameters

  • lhsAST –

  • rhsAST –

  • es –

Returns

parse_declaration_expressn_ternaryif(cond, iftrue, iffalse, es)

Classic if statement. This needs to be rearranged.

In wdl, this looks like: if <condition> then <iftrue> else <iffalse>

In python, this needs to be: <iftrue> if <condition> else <iffalse>

Parameters

  • cond –

  • iftrue –

  • iffalse –

  • es –

Returns

parse_declaration_expressn_tupleliteral(values, es)

Same in python. Just a parenthesis enclosed tuple.

Parameters

  • values –

  • es –

Returns

parse_declaration_expressn_arrayliteral(values, es)

Same in python. Just a square bracket enclosed array.

Parameters

  • values –

  • es –

Returns

parse_declaration_expressn_operator(lhsAST, rhsAST, es, operator)

Simply joins the left and right hand arguments lhs and rhs with an operator.

Parameters

  • lhsAST –

  • rhsAST –

  • es –

  • operator –

Returns

parse_declaration_expressn_fncall(name, params, es)

Parses out cromwell’s built-in function calls.

Some of these are special and need minor adjustments, for example size() requires a fileStore.

Parameters

  • name –

  • params –

  • es –

Returns

parse_declaration_expressn_fncall_normalparams(params)
parse_workflow_call_taskname(i)

Required.

Parameters

i –

Returns

parse_workflow_call_taskalias(i)

Required.

Parameters

i –

Returns

parse_workflow_call_body_declarations(i)

Have not seen this used, so expects to return “[]”.

Parameters

i –

Returns

parse_workflow_call_body_io(i)

Required.

Parameters

i –

Returns

parse_workflow_call_body_io_map(i)

Required.

Parameters

i –

Returns

parse_workflow_call_body(i)

Required.

Parameters

i –

Returns

parse_workflow_call(i)

Parses a WDL workflow call AST subtree to give the variable mappings for that particular job/task “call”.

Parameters

i – WDL workflow job object

Returns

python dictionary of io mappings for that job call