toil.batchSystems.lsf

Module Contents

Classes

LSFBatchSystem

A partial implementation of BatchSystemSupport for batch systems run on a

Attributes

logger

toil.batchSystems.lsf.logger

class toil.batchSystems.lsf.LSFBatchSystem(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem

digraph inheritance6ef54eb390 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ABC" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Helper class that provides a standard way to create an ABC using"]; "AbstractBatchSystem" [URL="../abstractBatchSystem/index.html#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An abstract base class to represent the interface the batch system must provide to Toil."]; "ABC" -> "AbstractBatchSystem" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AbstractGridEngineBatchSystem" [URL="../abstractGridEngineBatchSystem/index.html#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A partial implementation of BatchSystemSupport for batch systems run on a"]; "BatchSystemCleanupSupport" -> "AbstractGridEngineBatchSystem" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemCleanupSupport" [URL="../cleanup_support/index.html#toil.batchSystems.cleanup_support.BatchSystemCleanupSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Adds cleanup support when the last running job leaves a node, for batch"]; "BatchSystemLocalSupport" -> "BatchSystemCleanupSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemLocalSupport" [URL="../local_support/index.html#toil.batchSystems.local_support.BatchSystemLocalSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Adds a local queue for helper jobs, useful for CWL & others."]; "BatchSystemSupport" -> "BatchSystemLocalSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemSupport" [URL="../abstractBatchSystem/index.html#toil.batchSystems.abstractBatchSystem.BatchSystemSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Partial implementation of AbstractBatchSystem, support methods."]; "AbstractBatchSystem" -> "BatchSystemSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; "LSFBatchSystem" [URL="#toil.batchSystems.lsf.LSFBatchSystem",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "AbstractGridEngineBatchSystem" -> "LSFBatchSystem" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto-deployment is not implemented.

class Worker(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.Worker

digraph inheritance08ac38e9b9 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Thread" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class that represents a thread of control."]; "Worker" [URL="../abstractGridEngineBatchSystem/index.html#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.Worker",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "Thread" -> "Worker" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Worker" [URL="#toil.batchSystems.lsf.LSFBatchSystem.Worker",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="LSF specific AbstractGridEngineWorker methods."]; "Worker" -> "Worker" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

LSF specific AbstractGridEngineWorker methods.

Parameters

• newJobsQueue (queue.Queue) –

• updatedJobsQueue (queue.Queue) –

• killQueue (queue.Queue) –

• killedJobsQueue (queue.Queue) –

• boss (AbstractGridEngineBatchSystem) –

getRunningJobIDs()

Get a list of running job IDs. Implementation-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs()

Return type

list

fallbackRunningJobIDs(currentjobs)

killJob(jobID)

Kill specific job with the Toil job ID. Implementation-specific; called by AbstractGridEngineWorker.killJobs()

Parameters

jobID (string) – Toil job ID

prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None)

Preparation in putting together a command-line string for submitting to batch system (via submitJob().)

Param

int cpu

Param

int memory

Param

int jobID: Toil job ID

Param

string subLine: the command line string to be called

Param

string jobName: the name of the Toil job, to provide metadata to batch systems if desired

Param

dict job_environment: the environment variables to be set on the worker

Return type

List[str]

Parameters

• cpu (int) –

• memory (int) –

• jobID (int) –

• command (str) –

• jobName (str) –

• job_environment (Optional[Dict[str, str]]) –

• gpus (Optional[int]) –

submitJob(subLine)

Wrapper routine for submitting the actual command-line call, then processing the output to get the batch system job ID

Param

string subLine: the literal command line string to be called

Return type

string: batch system job ID, which will be stored internally

coalesce_job_exit_codes(batch_job_id_list)

Returns exit codes for a list of jobs.

Called by AbstractGridEngineWorker.checkOnJobs().

This is an optional part of the interface. It should raise NotImplementedError if not actually implemented for a particular scheduler.

Parameters

batch_job_id_list (string) – List of batch system job ID

Return type

list

getJobExitCode(lsfJobID)

Returns job exit code or an instance of abstractBatchSystem.BatchJobExitReason. if something else happened other than the job exiting. Implementation-specific; called by AbstractGridEngineWorker.checkOnJobs()

Parameters

batchjobID (string) – batch system job ID

Return type

int|toil.batchSystems.abstractBatchSystem.BatchJobExitReason: exit code int or BatchJobExitReason if something else happened other than job exiting.

parse_bjobs_record(bjobs_record, job)

Helper functions for getJobExitCode and to parse the bjobs status record

Parameters

• bjobs_record (dict) –

• job (int) –

Return type

Union[int, None]

getJobExitCodeBACCT(job)

fallbackGetJobExitCode(job)

prepareBsub(cpu, mem, jobID)

Make a bsub commandline to execute.

params:

cpu: number of cores needed mem: number of bytes of memory needed jobID: ID number of the job

Parameters

• cpu (int) –

• mem (int) –

• jobID (int) –

Return type

List[str]

parseBjobs(bjobs_output_str)

Parse records from bjobs json type output

Params bjobs_output_str

stdout of bjobs json type output

parseMaxMem(jobID)

Parse the maximum memory from job.

Parameters

jobID – ID number of the job

getWaitDuration()

We give LSF a second to catch its breath (in seconds)