toil.batchSystems.lsf

Attributes

logger

Classes

LSFBatchSystem

A partial implementation of BatchSystemSupport for batch systems run on a

Module Contents

toil.batchSystems.lsf.logger

class toil.batchSystems.lsf.LSFBatchSystem(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem

A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto-deployment is not implemented.

Parameters:

• config (toil.common.Config)

• maxCores (float)

• maxMemory (int)

• maxDisk (int)

class GridEngineThread(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.GridEngineThread

LSF specific GridEngineThread methods.

Parameters:

• newJobsQueue (queue.Queue)

• updatedJobsQueue (queue.Queue)

• killQueue (queue.Queue)

• killedJobsQueue (queue.Queue)

• boss (AbstractGridEngineBatchSystem)

getRunningJobIDs()

Get a list of running job IDs. Implementation-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs()

Return type:

list

fallbackRunningJobIDs(currentjobs)

killJob(jobID)

Kill specific job with the Toil job ID. Implementation-specific; called by GridEngineThread.killJobs()

Parameters:

jobID (string) – Toil job ID

prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None)

Preparation in putting together a command-line string for submitting to batch system (via submitJob().)

Param:

int cpu

Param:

int memory

Param:

int jobID: Toil job ID

Param:

string subLine: the command line string to be called

Param:

string jobName: the name of the Toil job, to provide metadata to batch systems if desired

Param:

dict job_environment: the environment variables to be set on the worker

Return type:

List[str]

Parameters:

• cpu (int)

• memory (int)

• jobID (int)

• command (str)

• jobName (str)

• job_environment (Optional[Dict[str, str]])

• gpus (Optional[int])

submitJob(subLine)

Wrapper routine for submitting the actual command-line call, then processing the output to get the batch system job ID

Param:

string subLine: the literal command line string to be called

Return type:

string: batch system job ID, which will be stored internally

coalesce_job_exit_codes(batch_job_id_list)

Returns exit codes and possibly exit reasons for a list of jobs, or None if they are running.

Called by GridEngineThread.checkOnJobs().

The default implementation falls back on self.getJobExitCode and polls each job individually

Parameters:

batch_job_id_list (string) – List of batch system job ID

Return type:

list

getJobExitCode(lsfJobID)

Returns job exit code and possibly an instance of abstractBatchSystem.BatchJobExitReason.

Returns None if the job is still running.

If the job is not running but the exit code is not available, it will be EXIT_STATUS_UNAVAILABLE_VALUE. Implementation-specific; called by GridEngineThread.checkOnJobs().

The exit code will only be 0 if the job affirmatively succeeded.

Parameters:

batchjobID (string) – batch system job ID

Return type:

Union[int, Tuple[int, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason]], None]

parse_bjobs_record(bjobs_record, job)

Helper functions for getJobExitCode and to parse the bjobs status record

Parameters:

• bjobs_record (dict)

• job (int)

Return type:

Union[int, Tuple[int, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason]], None]

getJobExitCodeBACCT(job)

Return type:

Union[int, Tuple[int, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason]], None]

fallbackGetJobExitCode(job)

Return type:

Union[int, Tuple[int, Optional[toil.batchSystems.abstractBatchSystem.BatchJobExitReason]], None]

prepareBsub(cpu, mem, jobID)

Make a bsub commandline to execute.

params:

cpu: number of cores needed mem: number of bytes of memory needed jobID: ID number of the job

Parameters:

• cpu (int)

• mem (int)

• jobID (int)

Return type:

List[str]

parseBjobs(bjobs_output_str)

Parse records from bjobs json type output

Params bjobs_output_str:

stdout of bjobs json type output

parseMaxMem(jobID)

Parse the maximum memory from job.

Parameters:

jobID – ID number of the job

getWaitDuration()

We give LSF a second to catch its breath (in seconds)