toil.batchSystems.gridengine

Module Contents

Classes

GridEngineBatchSystem

A partial implementation of BatchSystemSupport for batch systems run on a

Attributes

logger

toil.batchSystems.gridengine.logger
class toil.batchSystems.gridengine.GridEngineBatchSystem(config, maxCores, maxMemory, maxDisk)[source]

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem

A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto-deployment is not implemented.

class Worker(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)[source]

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.Worker

Grid Engine-specific AbstractGridEngineWorker methods

Parameters:
getRunningJobIDs()[source]

Get a list of running job IDs. Implementation-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs()

Return type:

list

killJob(jobID)[source]

Kill specific job with the Toil job ID. Implementation-specific; called by AbstractGridEngineWorker.killJobs()

Parameters:

jobID (string) – Toil job ID

prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None)[source]

Preparation in putting together a command-line string for submitting to batch system (via submitJob().)

Param:

int cpu

Param:

int memory

Param:

int jobID: Toil job ID

Param:

string subLine: the command line string to be called

Param:

string jobName: the name of the Toil job, to provide metadata to batch systems if desired

Param:

dict job_environment: the environment variables to be set on the worker

Return type:

List[str]

Parameters:

  • cpu (int)

  • memory (int)

  • jobID (int)

  • command (str)

  • jobName (str)

  • job_environment (Optional[Dict[str, str]])

  • gpus (Optional[int])

submitJob(subLine)[source]

Wrapper routine for submitting the actual command-line call, then processing the output to get the batch system job ID

Param:

string subLine: the literal command line string to be called

Return type:

string: batch system job ID, which will be stored internally

getJobExitCode(sgeJobID)[source]

Get job exist code, checking both qstat and qacct. Return None if still running. Higher level should retry on CalledProcessErrorStderr, for the case the job has finished and qacct result is stale.

prepareQsub(cpu, mem, jobID, job_environment=None)[source]
Parameters:

  • cpu (int)

  • mem (int)

  • jobID (int)

  • job_environment (Optional[Dict[str, str]])

Return type:

List[str]

classmethod getWaitDuration()[source]