toil.batchSystems.awsBatch¶
Batch system for running Toil workflows on AWS Batch.
Useful with the AWS job store.
AWS Batch has no means for scheduling based on disk usage, so the backing machines need to have “enough” disk and other constraints need to guarantee that disk does not fill.
Assumes that an AWS Batch Queue name or ARN is already provided.
Handles creating and destroying a JobDefinition for the workflow run.
Additional containers should be launched with Singularity, not Docker.
Attributes¶
Classes¶
Adds cleanup support when the last running job leaves a node, for batch |
Module Contents¶
- toil.batchSystems.awsBatch.logger¶
- toil.batchSystems.awsBatch.STATE_TO_EXIT_REASON: dict[str, toil.batchSystems.abstractBatchSystem.BatchJobExitReason]¶
- toil.batchSystems.awsBatch.MAX_POLL_COUNT = 100¶
- toil.batchSystems.awsBatch.MIN_REQUESTABLE_MIB = 4¶
- toil.batchSystems.awsBatch.MIN_REQUESTABLE_CORES = 1¶
- class toil.batchSystems.awsBatch.AWSBatchBatchSystem(config, maxCores, maxMemory, maxDisk)¶
Bases:
toil.batchSystems.cleanup_support.BatchSystemCleanupSupport
Adds cleanup support when the last running job leaves a node, for batch systems that can’t provide it using the backing scheduler.
- Parameters:
config (toil.common.Config)
maxCores (float)
maxMemory (int)
maxDisk (int)
- classmethod supportsAutoDeployment()¶
Whether this batch system supports auto-deployment of the user script itself.
If it does, the
setUserScript()
can be invoked to set the resource object representing the user script.Note to implementors: If your implementation returns True here, it should also override
- Return type:
- region¶
- client¶
- queue¶
- job_role_arn¶
- owner_tag¶
- worker_work_dir¶
- user_script: toil.resource.Resource | None = None¶
- docker_image¶
- setUserScript(user_script)¶
Set the user script for this workflow.
This method must be called before the first job is issued to this batch system, and only if
supportsAutoDeployment()
returns True, otherwise it will raise an exception.- Parameters:
userScript – the resource object representing the user script or module and the modules it depends on.
user_script (toil.resource.Resource)
- Return type:
None
- issueBatchJob(command, job_desc, job_environment=None)¶
Issues a job with the specified command to the batch system and returns a unique job ID number.
- Parameters:
command (str) – the command to execute somewhere to run the Toil worker process
job_desc (toil.job.JobDescription) – the JobDescription for the job being run
job_environment (Optional[dict[str, str]]) – a collection of job-specific environment variables to be set on the worker.
- Returns:
a unique job ID number that can be used to reference the newly issued job
- Return type:
- getUpdatedBatchJob(maxWait)¶
Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once.
Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait.
- Parameters:
maxWait (int) – the number of seconds to block, waiting for a result
- Returns:
If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall-clock time the job ran for, or None if this batch system does not support tracking wall time.
- Return type:
Optional[toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo]
- shutdown()¶
Called at the completion of a toil invocation. Should cleanly terminate all worker threads.
- Return type:
None
- getIssuedBatchJobIDs()¶
Gets all currently issued jobs
- getRunningBatchJobIDs()¶
Gets a map of jobs as job ID numbers that are currently running (not just waiting) and how long they have been running, in seconds.
- killBatchJobs(job_ids)¶
Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob.
- classmethod add_options(parser)¶
If this batch system provides any command line options, add them to the given parser.
- Parameters:
parser (Union[argparse.ArgumentParser, argparse._ArgumentGroup])
- Return type:
None
- classmethod setOptions(setOption)¶
Process command line or configuration options relevant to this batch system.
- Parameters:
setOption (toil.batchSystems.options.OptionSetter) – A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect.
- Return type:
None