toil.batchSystems.gridengine

Module Contents

Classes

GridEngineBatchSystem

A partial implementation of BatchSystemSupport for batch systems run on a

Attributes

logger

toil.batchSystems.gridengine.logger

class toil.batchSystems.gridengine.GridEngineBatchSystem(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem

digraph inheritance93a4301a02 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ABC" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Helper class that provides a standard way to create an ABC using"]; "AbstractBatchSystem" [URL="../abstractBatchSystem/index.html#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An abstract base class to represent the interface the batch system must provide to Toil."]; "ABC" -> "AbstractBatchSystem" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AbstractGridEngineBatchSystem" [URL="../abstractGridEngineBatchSystem/index.html#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A partial implementation of BatchSystemSupport for batch systems run on a"]; "BatchSystemCleanupSupport" -> "AbstractGridEngineBatchSystem" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemCleanupSupport" [URL="../cleanup_support/index.html#toil.batchSystems.cleanup_support.BatchSystemCleanupSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Adds cleanup support when the last running job leaves a node, for batch"]; "BatchSystemLocalSupport" -> "BatchSystemCleanupSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemLocalSupport" [URL="../local_support/index.html#toil.batchSystems.local_support.BatchSystemLocalSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Adds a local queue for helper jobs, useful for CWL & others."]; "BatchSystemSupport" -> "BatchSystemLocalSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemSupport" [URL="../abstractBatchSystem/index.html#toil.batchSystems.abstractBatchSystem.BatchSystemSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Partial implementation of AbstractBatchSystem, support methods."]; "AbstractBatchSystem" -> "BatchSystemSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; "GridEngineBatchSystem" [URL="#toil.batchSystems.gridengine.GridEngineBatchSystem",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "AbstractGridEngineBatchSystem" -> "GridEngineBatchSystem" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto-deployment is not implemented.

class Worker(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.Worker

digraph inheritance47ee212515 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Thread" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class that represents a thread of control."]; "Worker" [URL="../abstractGridEngineBatchSystem/index.html#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.Worker",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "Thread" -> "Worker" [arrowsize=0.5,style="setlinewidth(0.5)"]; "Worker" [URL="#toil.batchSystems.gridengine.GridEngineBatchSystem.Worker",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Grid Engine-specific AbstractGridEngineWorker methods"]; "Worker" -> "Worker" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Grid Engine-specific AbstractGridEngineWorker methods

Parameters

• newJobsQueue (queue.Queue) –

• updatedJobsQueue (queue.Queue) –

• killQueue (queue.Queue) –

• killedJobsQueue (queue.Queue) –

• boss (AbstractGridEngineBatchSystem) –

getRunningJobIDs()

Get a list of running job IDs. Implementation-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs()

Return type

list

killJob(jobID)

Kill specific job with the Toil job ID. Implementation-specific; called by AbstractGridEngineWorker.killJobs()

Parameters

jobID (string) – Toil job ID

prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None)

Preparation in putting together a command-line string for submitting to batch system (via submitJob().)

Param

int cpu

Param

int memory

Param

int jobID: Toil job ID

Param

string subLine: the command line string to be called

Param

string jobName: the name of the Toil job, to provide metadata to batch systems if desired

Param

dict job_environment: the environment variables to be set on the worker

Return type

List[str]

Parameters

• cpu (int) –

• memory (int) –

• jobID (int) –

• command (str) –

• jobName (str) –

• job_environment (Optional[Dict[str, str]]) –

• gpus (Optional[int]) –

submitJob(subLine)

Wrapper routine for submitting the actual command-line call, then processing the output to get the batch system job ID

Param

string subLine: the literal command line string to be called

Return type

string: batch system job ID, which will be stored internally

getJobExitCode(sgeJobID)

Get job exist code, checking both qstat and qacct. Return None if still running. Higher level should retry on CalledProcessErrorStderr, for the case the job has finished and qacct result is stale.

prepareQsub(cpu, mem, jobID, job_environment=None)

Parameters

• cpu (int) –

• mem (int) –

• jobID (int) –

• job_environment (Optional[Dict[str, str]]) –

Return type

List[str]

classmethod getWaitDuration()