toil.batchSystems.tes

Batch system for running Toil workflows on GA4GH TES.

Useful with network-based job stores when the TES server provides tasks with credentials, and filesystem-based job stores when the TES server lets tasks mount the job store.

Additional containers should be launched with Singularity, not Docker.

Module Contents

Classes

TESBatchSystem

Adds cleanup support when the last running job leaves a node, for batch

Attributes

logger

STATE_TO_EXIT_REASON

toil.batchSystems.tes.logger
toil.batchSystems.tes.STATE_TO_EXIT_REASON: Dict[str, toil.batchSystems.abstractBatchSystem.BatchJobExitReason]
class toil.batchSystems.tes.TESBatchSystem(config, maxCores, maxMemory, maxDisk)[source]

Bases: toil.batchSystems.cleanup_support.BatchSystemCleanupSupport

digraph inheritance3037dbd9c2 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ABC" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Helper class that provides a standard way to create an ABC using"]; "AbstractBatchSystem" [URL="../abstractBatchSystem/index.html#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An abstract base class to represent the interface the batch system must provide to Toil."]; "ABC" -> "AbstractBatchSystem" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemCleanupSupport" [URL="../cleanup_support/index.html#toil.batchSystems.cleanup_support.BatchSystemCleanupSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Adds cleanup support when the last running job leaves a node, for batch"]; "BatchSystemLocalSupport" -> "BatchSystemCleanupSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemLocalSupport" [URL="../local_support/index.html#toil.batchSystems.local_support.BatchSystemLocalSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Adds a local queue for helper jobs, useful for CWL & others."]; "BatchSystemSupport" -> "BatchSystemLocalSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemSupport" [URL="../abstractBatchSystem/index.html#toil.batchSystems.abstractBatchSystem.BatchSystemSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Partial implementation of AbstractBatchSystem, support methods."]; "AbstractBatchSystem" -> "BatchSystemSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; "TESBatchSystem" [URL="#toil.batchSystems.tes.TESBatchSystem",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "BatchSystemCleanupSupport" -> "TESBatchSystem" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Adds cleanup support when the last running job leaves a node, for batch systems that can’t provide it using the backing scheduler.

Parameters
classmethod supportsAutoDeployment()[source]

Whether this batch system supports auto-deployment of the user script itself.

If it does, the setUserScript() can be invoked to set the resource object representing the user script.

Note to implementors: If your implementation returns True here, it should also override

Return type

bool

classmethod get_default_tes_endpoint()[source]

Get the default TES endpoint URL to use.

(unless overridden by an option or environment variable)

Return type

str

setUserScript(user_script)[source]

Set the user script for this workflow. This method must be called before the first job is issued to this batch system, and only if supportsAutoDeployment() returns True, otherwise it will raise an exception.

Parameters

  • userScript – the resource object representing the user script or module and the modules it depends on.

  • user_script (toil.resource.Resource) –

Return type

None

issueBatchJob(job_desc, job_environment=None)[source]

Issues a job with the specified command to the batch system and returns a unique jobID.

Parameters

  • jobDesc – a toil.job.JobDescription

  • job_environment (Optional[Dict[str, str]]) – a collection of job-specific environment variables to be set on the worker.

  • job_desc (toil.job.JobDescription) –

Returns

a unique jobID that can be used to reference the newly issued job

Return type

int

getUpdatedBatchJob(maxWait)[source]

Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once.

Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait.

Parameters

maxWait (int) – the number of seconds to block, waiting for a result

Returns

If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall-clock time the job ran for, or None if this batch system does not support tracking wall time.

Return type

Optional[toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo]

shutdown()[source]

Called at the completion of a toil invocation. Should cleanly terminate all worker threads.

Return type

None

getIssuedBatchJobIDs()[source]

Gets all currently issued jobs

Returns

A list of jobs (as jobIDs) currently issued (may be running, or may be waiting to be run). Despite the result being a list, the ordering should not be depended upon.

Return type

List[int]

getRunningBatchJobIDs()[source]

Gets a map of jobs as jobIDs that are currently running (not just waiting) and how long they have been running, in seconds.

Returns

dictionary with currently running jobID keys and how many seconds they have been running as the value

Return type

Dict[int, float]

killBatchJobs(job_ids)[source]

Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob.

Parameters

  • jobIDs – list of IDs of jobs to kill

  • job_ids (List[int]) –

Return type

None

classmethod add_options(parser)[source]

If this batch system provides any command line options, add them to the given parser.

Parameters

parser (Union[argparse.ArgumentParser, argparse._ArgumentGroup]) –

Return type

None

classmethod setOptions(setOption)[source]

Process command line or configuration options relevant to this batch system.

Parameters

setOption (toil.batchSystems.options.OptionSetter) – A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect.

Return type

None