toil.batchSystems.slurm

Attributes

logger

TERMINAL_STATES

NONTERMINAL_STATES

Classes

SlurmBatchSystem

A partial implementation of BatchSystemSupport for batch systems run on a

Functions

parse_slurm_time(slurm_time)

Parse a Slurm-style time duration like 7-00:00:00 to a number of seconds.

Module Contents

toil.batchSystems.slurm.logger
toil.batchSystems.slurm.TERMINAL_STATES: dict[str, toil.batchSystems.abstractBatchSystem.BatchJobExitReason]
toil.batchSystems.slurm.NONTERMINAL_STATES: set[str]
toil.batchSystems.slurm.parse_slurm_time(slurm_time)[source]

Parse a Slurm-style time duration like 7-00:00:00 to a number of seconds.

Raises ValueError if not parseable.

Parameters:

slurm_time (str)

Return type:

int

class toil.batchSystems.slurm.SlurmBatchSystem(config, maxCores, maxMemory, maxDisk)[source]

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem

A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto-deployment is not implemented.

Parameters:
class PartitionInfo[source]

Bases: NamedTuple

partition_name: str
gres: bool
time_limit: float
priority: int
cpus: str
memory: str
class PartitionSet[source]

Set of available partitions detected on the slurm batch system

default_gpu_partition: SlurmBatchSystem | None
all_partitions: list[SlurmBatchSystem]
gpu_partitions: set[str]
get_partition(time_limit)[source]

Get the partition name to use for a job with the given time limit.

Parameters:

time_limit (float | None)

Return type:

str | None

class GridEngineThread(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)[source]

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread

A class that represents a thread of control.

This class can be safely subclassed in a limited fashion. There are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass.

Parameters:
boss: SlurmBatchSystem
getRunningJobIDs()[source]

Get a list of running job IDs. Implementation-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs()

Return type:

list

killJob(jobID)[source]

Kill specific job with the Toil job ID. Implementation-specific; called by GridEngineThread.killJobs()

Parameters:

jobID (string) – Toil job ID

Return type:

None

prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None)[source]

Preparation in putting together a command-line string for submitting to batch system (via submitJob().)

Param:

int cpu

Param:

int memory

Param:

int jobID: Toil job ID

Param:

string subLine: the command line string to be called

Param:

string jobName: the name of the Toil job, to provide metadata to batch systems if desired

Param:

dict job_environment: the environment variables to be set on the worker

Return type:

List[str]

Parameters:
submitJob(subLine)[source]

Wrapper routine for submitting the actual command-line call, then processing the output to get the batch system job ID

Param:

string subLine: the literal command line string to be called

Return type:

string: batch system job ID, which will be stored internally

Parameters:

subLine (list[str])

coalesce_job_exit_codes(batch_job_id_list)[source]

Collect all job exit codes in a single call. :param batch_job_id_list: list of Job ID strings, where each string has the form “<job>[.<task>]”. :return: list of job exit codes or exit code, exit reason pairs associated with the list of job IDs.

Parameters:

batch_job_id_list (list[str])

Return type:

list[int | tuple[int, toil.batchSystems.abstractBatchSystem.BatchJobExitReason | None] | None]

getJobExitCode(batchJobID)[source]

Get job exit code for given batch job ID. :param batchJobID: string of the form “<job>[.<task>]”. :return: integer job exit code.

Parameters:

batchJobID (str)

Return type:

int | tuple[int, toil.batchSystems.abstractBatchSystem.BatchJobExitReason | None] | None

prepareSbatch(cpu, mem, jobID, jobName, job_environment, gpus)[source]

Returns the sbatch command line to run to queue the job.

Parameters:
Return type:

list[str]

partitions
issueBatchJob(command, job_desc, job_environment=None)[source]

Issues a job with the specified command to the batch system and returns a unique job ID number.

Parameters:
  • command (str) – the command to execute somewhere to run the Toil worker process

  • job_desc (toil.job.JobDescription) – the JobDescription for the job being run

  • job_environment (dict[str, str] | None) – a collection of job-specific environment variables to be set on the worker.

Returns:

a unique job ID number that can be used to reference the newly issued job

Return type:

int

classmethod add_options(parser)[source]

If this batch system provides any command line options, add them to the given parser.

Parameters:

parser (argparse.ArgumentParser | argparse._ArgumentGroup)

Return type:

None

OptionType
classmethod setOptions(setOption)[source]

Process command line or configuration options relevant to this batch system.

Parameters:

setOption (toil.batchSystems.options.OptionSetter) – A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect.

Return type:

None