toil.server.wes.amazon_wes_utils

Module Contents

Classes

WorkflowPlan

These functions pass around dicts of a certain type, with data and files keys.

DataDict

Under data, there can be:

FilesDict

Under files, there can be:

Functions

parse_workflow_zip_file(file, workflow_type)

Processes a workflow zip bundle

parse_workflow_manifest_file(manifest_file)

Reads a MANIFEST.json file for a workflow zip bundle

workflow_manifest_url_to_path(url[, parent_dir])

Interpret a possibly-relative parsed URL, relative to the given parent directory.

task_filter(task, job_status)

AGC requires task names to be annotated with an AWS Batch job ID that they

Attributes

logger

NOTICE

toil.server.wes.amazon_wes_utils.logger
toil.server.wes.amazon_wes_utils.NOTICE = Multiline-String
Show Value
"""
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
"""
class toil.server.wes.amazon_wes_utils.WorkflowPlan

Bases: TypedDict

These functions pass around dicts of a certain type, with data and files keys.

data: DataDict
files: FilesDict
class toil.server.wes.amazon_wes_utils.DataDict

Bases: TypedDict

Under data, there can be: * workflowUrl (required if no workflowSource): URL to main workflow code.

workflowUrl: str
class toil.server.wes.amazon_wes_utils.FilesDict

Bases: TypedDict

Under files, there can be: * workflowSource (required if no workflowUrl): Open binary-mode file for the main workflow code. * workflowInputFiles: List of open binary-mode file for input files. Expected to be JSONs. * workflowOptions: Open binary-mode file for a JSON of options sent along with the workflow. * workflowDependencies: Open binary-mode file for the zip the workflow came in, if any.

workflowSource: IO[bytes]
workflowInputFiles: List[IO[bytes]]
workflowOptions: IO[bytes]
workflowDependencies: IO[bytes]
toil.server.wes.amazon_wes_utils.parse_workflow_zip_file(file, workflow_type)

Processes a workflow zip bundle

Parameters:

  • file (str) – String or Path-like path to a workflow.zip file

  • workflow_type (str) – String, extension of workflow to expect (e.g. “wdl”)

Return type:

dict of data and files

If the zip only contains a single file, that file is set as workflowSource

If the zip contains multiple files with a MANIFEST.json file, the MANIFEST is used to determine appropriate data and file arguments. (See: parse_workflow_manifest_file())

If the zip contains multiple files without a MANIFEST.json file:

  • a main workflow file with an extension matching the workflow_type is expected and will be set as workflowSource

  • optionally, if inputs*.json files are found in the root level of the zip, they will be set as workflowInputs(_d)* in the order they are found

  • optionally, if an options.json file is found in the root level of the zip, it will be set as workflowOptions

If the zip contains multiple files, the original zip is set as workflowDependencies

toil.server.wes.amazon_wes_utils.parse_workflow_manifest_file(manifest_file)

Reads a MANIFEST.json file for a workflow zip bundle

Parameters:

manifest_file (str) – String or Path-like path to a MANIFEST.json file

Return type:

dict of data and files

MANIFEST.json is expected to be formatted like:

{
    "mainWorkflowURL": "relpath/to/workflow",
    "inputFileURLs": [
        "relpath/to/input-file-1",
        "relpath/to/input-file-2",
        "relpath/to/input-file-3"
    ],
    "optionsFileURL": "relpath/to/option-file"
}

The mainWorkflowURL property that provides a relative file path in the zip to a workflow file, which will be set as workflowSource

The inputFileURLs property is optional and provides a list of relative file paths in the zip to input.json files. The list is assumed to be in the order the inputs should be applied - e.g. higher list index is higher priority. If present, it will be used to set workflowInputs(_d) arguments.

The optionsFileURL property is optional and provides a relative file path in the zip to an options.json file. If present, it will be used to set workflowOptions.

toil.server.wes.amazon_wes_utils.workflow_manifest_url_to_path(url, parent_dir=None)

Interpret a possibly-relative parsed URL, relative to the given parent directory.

Parameters:
Return type:

str

toil.server.wes.amazon_wes_utils.task_filter(task, job_status)

AGC requires task names to be annotated with an AWS Batch job ID that they were run under. If it encounters an un-annotated task name, it will crash. See <https://github.com/aws/amazon-genomics-cli/issues/494>.

This encodes the AWSBatchJobID annotation, from the AmazonBatchBatchSystem, into the task name of the given task, and returns the modified task. If no such annotation is available, the task is censored and None is returned.

Parameters:
Return type:

Optional[toil.server.wes.abstract_backend.TaskLog]