toil.wdl.wdl_functions
¶
Module Contents¶
Classes¶
Extended JSONEncoder to support WDL-specific JSON encoding. |
Functions¶
|
Creates a bashscript to inject into a docker container for the job. |
|
|
|
Takes any input and imports the WDLFile into the fileStore. |
|
Given 3 String parameters input, pattern, replace, this function will |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Create a stdout (or stderr) file from a string or bytes object. |
|
Parses a string representing memory and returns |
|
|
|
|
|
|
|
Given a File and a String (optional), returns the size of the file in Bytes |
|
|
|
|
|
https://software.broadinstitute.org/wdl/documentation/article?id=10554 |
|
|
|
Converts a Float value into an Int by rounding down to the next lower integer. |
|
Converts a Float value into an Int by rounding up to the next higher integer. |
|
Given a file-like object (String, File) as a parameter, this will read each |
|
Take a tsv filepath and return an array; e.g. [[],[],[]]. |
|
Take a csv filepath and return an array; e.g. [[],[],[]]. |
|
The read_json() function takes one parameter, which is a file-like object |
|
Given a file-like object (String, File) as a parameter, this will read each |
|
The read_int() function takes a file path which is expected to contain 1 |
|
The read_string() function takes a file path which is expected to contain 1 |
|
The read_float() function takes a file path which is expected to contain 1 |
|
The read_boolean() function takes a file path which is expected to contain 1 |
|
Given something that's compatible with Array[String], this writes each element |
|
Given something that's compatible with Array[Array[String]], this writes a TSV |
|
Given something with any type, this writes the JSON equivalent to a file. See |
|
Given something that's compatible with Map[String, String], this writes a TSV |
|
Given an integer argument, the range function creates an array of integers of |
|
Given a two dimensional array argument, the transpose function transposes the |
|
Given an Array, the length function returns the number of elements in the Array |
|
Return the dot product of the two arrays. If the arrays have different lengths |
|
Return the cross product of the two arrays. Array[Y][1] appears before |
|
Given a Map, the as_pairs function returns an Array containing each element |
|
Given an Array consisting of Pairs, the as_map function returns a Map in |
|
Given a Map, the keys function returns an Array consisting of the keys in |
|
Given an Array consisting of Pairs, the collect_by_key function returns a Map |
|
Given an array of arrays, the flatten function concatenates all the member |
Attributes¶
- toil.wdl.wdl_functions.logger¶
- exception toil.wdl.wdl_functions.WDLRuntimeError(message)[source]¶
Bases:
digraph inheritance440e2d1fcd { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "WDLRuntimeError" [URL="#toil.wdl.wdl_functions.WDLRuntimeError",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="WDL-related run-time error."]; }Exception
WDL-related run-time error.
- class toil.wdl.wdl_functions.WDLJSONEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]¶
Bases:
digraph inheritanceeec7702b93 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "JSONEncoder" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Extensible JSON <http://json.org> encoder for Python data structures."]; "WDLJSONEncoder" [URL="#toil.wdl.wdl_functions.WDLJSONEncoder",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Extended JSONEncoder to support WDL-specific JSON encoding."]; "JSONEncoder" -> "WDLJSONEncoder" [arrowsize=0.5,style="setlinewidth(0.5)"]; }json.JSONEncoder
Extended JSONEncoder to support WDL-specific JSON encoding.
- default(obj)[source]¶
Implement this method in a subclass such that it returns a serializable object for
o
, or calls the base implementation (to raise aTypeError
).For example, to support arbitrary iterators, you could implement default like this:
def default(self, o): try: iterable = iter(o) except TypeError: pass else: return list(iterable) # Let the base class default method raise the TypeError return JSONEncoder.default(self, o)
- toil.wdl.wdl_functions.generate_docker_bashscript_file(temp_dir, docker_dir, globs, cmd, job_name)[source]¶
Creates a bashscript to inject into a docker container for the job.
This script wraps the job command(s) given in a bash script, hard links the outputs and returns an “rc” file containing the exit code. All of this is done in an effort to parallel the Broad’s cromwell engine, which is the native WDL runner. As they’ve chosen to write and then run a bashscript for every command, so shall we.
- Parameters
temp_dir – The current directory outside of docker to deposit the bashscript into, which will be the bind mount that docker loads files from into its own containerized filesystem. This is usually the tempDir created by this individual job using ‘tempDir = job.fileStore.getLocalTempDir()’.
docker_dir – The working directory inside of the docker container which is bind mounted to ‘temp_dir’. By default this is ‘data’.
globs – A list of expected output files to retrieve as glob patterns that will be returned as hard links to the current working directory.
cmd – A bash command to be written into the bash script and run.
job_name – The job’s name, only used to write in a file name identifying the script as written for that job. Will be used to call the script later.
- Returns
Nothing, but it writes and deposits a bash script in temp_dir intended to be run inside of a docker container for this job.
- toil.wdl.wdl_functions.process_single_infile(wdl_file, fileStore)[source]¶
- Parameters
wdl_file (toil.wdl.wdl_types.WDLFile) –
fileStore (toil.fileStores.abstractFileStore.AbstractFileStore) –
- Return type
- toil.wdl.wdl_functions.process_infile(f, fileStore)[source]¶
Takes any input and imports the WDLFile into the fileStore.
This returns the input importing all WDLFile instances to the fileStore. Toil does not preserve a file’s original name upon import and so the WDLFile also keeps track of this.
- Parameters
f (Any) – A primitive, WDLFile, or a container. A file needs to be a WDLFile instance to be imported.
fileStore (toil.fileStores.abstractFileStore.AbstractFileStore) – The fileStore object that is called to load files into the fileStore.
- toil.wdl.wdl_functions.sub(input_str, pattern, replace)[source]¶
Given 3 String parameters input, pattern, replace, this function will replace any occurrence matching pattern in input by replace. pattern is expected to be a regular expression. Details of regex evaluation will depend on the execution engine running the WDL.
WDL syntax: String sub(String, String, String)
- toil.wdl.wdl_functions.process_single_outfile(wdl_file, fileStore, workDir, outDir)[source]¶
- Parameters
wdl_file (toil.wdl.wdl_types.WDLFile) –
- Return type
- toil.wdl.wdl_functions.read_single_file(f, tempDir, fileStore, docker=False)[source]¶
- Parameters
- Return type
- toil.wdl.wdl_functions.read_file(f, tempDir, fileStore, docker=False)[source]¶
- Parameters
f (Any) –
tempDir (str) –
fileStore (toil.fileStores.abstractFileStore.AbstractFileStore) –
docker (bool) –
- toil.wdl.wdl_functions.generate_stdout_file(output, tempDir, fileStore, stderr=False)[source]¶
Create a stdout (or stderr) file from a string or bytes object.
- toil.wdl.wdl_functions.parse_memory(memory)[source]¶
Parses a string representing memory and returns an integer # of bytes.
- Parameters
memory –
- Returns
- toil.wdl.wdl_functions.size(f=None, unit='B', fileStore=None)[source]¶
Given a File and a String (optional), returns the size of the file in Bytes or in the unit specified by the second argument.
Supported units are KiloByte (“K”, “KB”), MegaByte (“M”, “MB”), GigaByte (“G”, “GB”), TeraByte (“T”, “TB”) (powers of 1000) as well as their binary version (https://en.wikipedia.org/wiki/Binary_prefix) “Ki” (“KiB”), “Mi” (“MiB”), “Gi” (“GiB”), “Ti” (“TiB”) (powers of 1024). Default unit is Bytes (“B”).
WDL syntax: Float size(File, [String]) Varieties: Float size(File?, [String])
Float size(Array[File], [String]) Float size(Array[File?], [String])
- Parameters
f (Optional[Union[str, toil.wdl.wdl_types.WDLFile, List[Union[str, toil.wdl.wdl_types.WDLFile]]]]) –
unit (Optional[str]) –
fileStore (Optional[toil.fileStores.abstractFileStore.AbstractFileStore]) –
- Return type
- toil.wdl.wdl_functions.basename(path, suffix=None)[source]¶
https://software.broadinstitute.org/wdl/documentation/article?id=10554
- toil.wdl.wdl_functions.floor(i)[source]¶
Converts a Float value into an Int by rounding down to the next lower integer.
- toil.wdl.wdl_functions.ceil(i)[source]¶
Converts a Float value into an Int by rounding up to the next higher integer.
- toil.wdl.wdl_functions.read_lines(path)[source]¶
Given a file-like object (String, File) as a parameter, this will read each line as a string and return an Array[String] representation of the lines in the file.
WDL syntax: Array[String] read_lines(String|File)
- toil.wdl.wdl_functions.read_tsv(path, delimiter='\t')[source]¶
Take a tsv filepath and return an array; e.g. [[],[],[]].
For example, a file containing:
1 2 3 4 5 6 7 8 9
would return the array: [[‘1’,’2’,’3’], [‘4’,’5’,’6’], [‘7’,’8’,’9’]]
WDL syntax: Array[Array[String]] read_tsv(String|File)
- toil.wdl.wdl_functions.read_csv(path)[source]¶
Take a csv filepath and return an array; e.g. [[],[],[]].
For example, a file containing:
1,2,3 4,5,6 7,8,9
would return the array: [[‘1’,’2’,’3’], [‘4’,’5’,’6’], [‘7’,’8’,’9’]]
- toil.wdl.wdl_functions.read_json(path)[source]¶
- The read_json() function takes one parameter, which is a file-like object
(String, File) and returns a data type which matches the data structure in the JSON file. See
https://github.com/openwdl/wdl/blob/main/versions/development/SPEC.md#mixed-read_jsonstringfile
WDL syntax: mixed read_json(String|File)
- Parameters
path (str) –
- Return type
Any
- toil.wdl.wdl_functions.read_map(path)[source]¶
Given a file-like object (String, File) as a parameter, this will read each line from a file and expect the line to have the format col1 col2. In other words, the file-like object must be a two-column TSV file.
WDL syntax: Map[String, String] read_map(String|File)
- toil.wdl.wdl_functions.read_int(path)[source]¶
The read_int() function takes a file path which is expected to contain 1 line with 1 integer on it. This function returns that integer.
WDL syntax: Int read_int(String|File)
- Parameters
path (Union[str, toil.wdl.wdl_types.WDLFile]) –
- Return type
- toil.wdl.wdl_functions.read_string(path)[source]¶
The read_string() function takes a file path which is expected to contain 1 line with 1 string on it. This function returns that string.
WDL syntax: String read_string(String|File)
- Parameters
path (Union[str, toil.wdl.wdl_types.WDLFile]) –
- Return type
- toil.wdl.wdl_functions.read_float(path)[source]¶
The read_float() function takes a file path which is expected to contain 1 line with 1 floating point number on it. This function returns that float.
WDL syntax: Float read_float(String|File)
- Parameters
path (Union[str, toil.wdl.wdl_types.WDLFile]) –
- Return type
- toil.wdl.wdl_functions.read_boolean(path)[source]¶
The read_boolean() function takes a file path which is expected to contain 1 line with 1 Boolean value (either “true” or “false” on it). This function returns that Boolean value.
WDL syntax: Boolean read_boolean(String|File)
- Parameters
path (Union[str, toil.wdl.wdl_types.WDLFile]) –
- Return type
- toil.wdl.wdl_functions.write_lines(in_lines, temp_dir=None, file_store=None)[source]¶
Given something that’s compatible with Array[String], this writes each element to it’s own line on a file. with newline `
` characters as line separators.
WDL syntax: File write_lines(Array[String])
- Parameters
in_lines (List[str]) –
temp_dir (Optional[str]) –
file_store (Optional[toil.fileStores.abstractFileStore.AbstractFileStore]) –
- Return type
- toil.wdl.wdl_functions.write_tsv(in_tsv, delimiter='\t', temp_dir=None, file_store=None)[source]¶
Given something that’s compatible with Array[Array[String]], this writes a TSV file of the data structure.
WDL syntax: File write_tsv(Array[Array[String]])
- Parameters
in_tsv (List[List[str]]) –
delimiter (str) –
temp_dir (Optional[str]) –
file_store (Optional[toil.fileStores.abstractFileStore.AbstractFileStore]) –
- Return type
- toil.wdl.wdl_functions.write_json(in_json, indent=None, separators=(',', ':'), temp_dir=None, file_store=None)[source]¶
Given something with any type, this writes the JSON equivalent to a file. See the table in the definition of https://github.com/openwdl/wdl/blob/main/versions/development/SPEC.md#mixed-read_jsonstringfile
WDL syntax: File write_json(mixed)
- toil.wdl.wdl_functions.write_map(in_map, temp_dir=None, file_store=None)[source]¶
- Given something that’s compatible with Map[String, String], this writes a TSV
file of the data structure.
WDL syntax: File write_map(Map[String, String])
- Parameters
temp_dir (Optional[str]) –
file_store (Optional[toil.fileStores.abstractFileStore.AbstractFileStore]) –
- Return type
- toil.wdl.wdl_functions.wdl_range(num)[source]¶
Given an integer argument, the range function creates an array of integers of length equal to the given argument.
WDL syntax: Array[Int] range(Int)
- toil.wdl.wdl_functions.transpose(in_array)[source]¶
Given a two dimensional array argument, the transpose function transposes the two dimensional array according to the standard matrix transpose rules.
WDL syntax: Array[Array[X]] transpose(Array[Array[X]])
- Parameters
in_array (List[List[Any]]) –
- Return type
List[List[Any]]
- toil.wdl.wdl_functions.length(in_array)[source]¶
Given an Array, the length function returns the number of elements in the Array as an Int.
- Parameters
in_array (List[Any]) –
- Return type
- toil.wdl.wdl_functions.wdl_zip(left, right)[source]¶
Return the dot product of the two arrays. If the arrays have different lengths it is an error.
WDL syntax: Array[Pair[X,Y]] zip(Array[X], Array[Y])
- Parameters
left (List[Any]) –
right (List[Any]) –
- Return type
- toil.wdl.wdl_functions.cross(left, right)[source]¶
Return the cross product of the two arrays. Array[Y][1] appears before Array[X][1] in the output.
WDL syntax: Array[Pair[X,Y]] cross(Array[X], Array[Y])
- Parameters
left (List[Any]) –
right (List[Any]) –
- Return type
- toil.wdl.wdl_functions.as_pairs(in_map)[source]¶
Given a Map, the as_pairs function returns an Array containing each element in the form of a Pair. The key will be the left element of the Pair and the value the right element. The order of the the Pairs in the resulting Array is the same as the order of the key/value pairs in the Map.
WDL syntax: Array[Pair[X,Y]] as_pairs(Map[X,Y])
- Parameters
in_map (dict) –
- Return type
- toil.wdl.wdl_functions.as_map(in_array)[source]¶
Given an Array consisting of Pairs, the as_map function returns a Map in which the left elements of the Pairs are the keys and the right elements the values.
WDL syntax: Map[X,Y] as_map(Array[Pair[X,Y]])
- Parameters
in_array (List[toil.wdl.wdl_types.WDLPair]) –
- Return type
- toil.wdl.wdl_functions.keys(in_map)[source]¶
Given a Map, the keys function returns an Array consisting of the keys in the Map. The order of the keys in the resulting Array is the same as the order of the Pairs in the Map.
WDL syntax: Array[X] keys(Map[X,Y])
- toil.wdl.wdl_functions.collect_by_key(in_array)[source]¶
Given an Array consisting of Pairs, the collect_by_key function returns a Map in which the left elements of the Pairs are the keys and the right elements the values.
WDL syntax: Map[X,Array[Y]] collect_by_key(Array[Pair[X,Y]])
- Parameters
in_array (List[toil.wdl.wdl_types.WDLPair]) –
- Return type