toil.batchSystems.abstractGridEngineBatchSystem

Module Contents

Classes

AbstractGridEngineBatchSystem

A partial implementation of BatchSystemSupport for batch systems run on a

Attributes

logger

JobTuple

toil.batchSystems.abstractGridEngineBatchSystem.logger
toil.batchSystems.abstractGridEngineBatchSystem.JobTuple
class toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem(config, maxCores, maxMemory, maxDisk)[source]

Bases: toil.batchSystems.cleanup_support.BatchSystemCleanupSupport

digraph inheritance909d1af4f8 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ABC" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Helper class that provides a standard way to create an ABC using"]; "AbstractBatchSystem" [URL="../abstractBatchSystem/index.html#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An abstract base class to represent the interface the batch system must provide to Toil."]; "ABC" -> "AbstractBatchSystem" [arrowsize=0.5,style="setlinewidth(0.5)"]; "AbstractGridEngineBatchSystem" [URL="#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="A partial implementation of BatchSystemSupport for batch systems run on a"]; "BatchSystemCleanupSupport" -> "AbstractGridEngineBatchSystem" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemCleanupSupport" [URL="../cleanup_support/index.html#toil.batchSystems.cleanup_support.BatchSystemCleanupSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Adds cleanup support when the last running job leaves a node, for batch"]; "BatchSystemLocalSupport" -> "BatchSystemCleanupSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemLocalSupport" [URL="../local_support/index.html#toil.batchSystems.local_support.BatchSystemLocalSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Adds a local queue for helper jobs, useful for CWL & others."]; "BatchSystemSupport" -> "BatchSystemLocalSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemSupport" [URL="../abstractBatchSystem/index.html#toil.batchSystems.abstractBatchSystem.BatchSystemSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Partial implementation of AbstractBatchSystem, support methods."]; "AbstractBatchSystem" -> "BatchSystemSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto-deployment is not implemented.

class Worker(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)[source]

Bases: threading.Thread

digraph inheritance5a78fabf38 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "Thread" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A class that represents a thread of control."]; "Worker" [URL="#toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.Worker",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top"]; "Thread" -> "Worker" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

A class that represents a thread of control.

This class can be safely subclassed in a limited fashion. There are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass.

Parameters
getBatchSystemID(jobID)[source]

Get batch system-specific job ID

Note: for the moment this is the only consistent way to cleanly get the batch system job ID

Parameters

jobID (int) – Toil BatchSystem numerical job ID

Return type

str

forgetJob(jobID)[source]

Remove jobID passed

Parameters

jobID (int) – toil job ID

Return type

None

createJobs(newJob)[source]

Create a new job with the given attributes.

Implementation-specific; called by AbstractGridEngineWorker.run()

Parameters

newJob (JobTuple) –

Return type

bool

killJobs()[source]

Kill any running jobs within worker

checkOnJobs()[source]

Check and update status of all running jobs.

Respects statePollingWait and will return cached results if not within time period to talk with the scheduler.

run()[source]

Run any new jobs

abstract coalesce_job_exit_codes(batch_job_id_list)[source]

Returns exit codes for a list of jobs.

Called by AbstractGridEngineWorker.checkOnJobs().

This is an optional part of the interface. It should raise NotImplementedError if not actually implemented for a particular scheduler.

Parameters

batch_job_id_list (string) – List of batch system job ID

Return type

list

abstract prepareSubmission(cpu, memory, jobID, command, jobName, job_environment=None, gpus=None)[source]

Preparation in putting together a command-line string for submitting to batch system (via submitJob().)

Param

int cpu

Param

int memory

Param

int jobID: Toil job ID

Param

string subLine: the command line string to be called

Param

string jobName: the name of the Toil job, to provide metadata to batch systems if desired

Param

dict job_environment: the environment variables to be set on the worker

Return type

List[str]

Parameters

  • cpu (int) –

  • memory (int) –

  • jobID (int) –

  • command (str) –

  • jobName (str) –

  • job_environment (Optional[Dict[str, str]]) –

  • gpus (Optional[int]) –

abstract submitJob(subLine)[source]

Wrapper routine for submitting the actual command-line call, then processing the output to get the batch system job ID

Param

string subLine: the literal command line string to be called

Return type

string: batch system job ID, which will be stored internally

abstract getRunningJobIDs()[source]

Get a list of running job IDs. Implementation-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs()

Return type

list

abstract killJob(jobID)[source]

Kill specific job with the Toil job ID. Implementation-specific; called by AbstractGridEngineWorker.killJobs()

Parameters

jobID (string) – Toil job ID

abstract getJobExitCode(batchJobID)[source]

Returns job exit code or an instance of abstractBatchSystem.BatchJobExitReason. if something else happened other than the job exiting. Implementation-specific; called by AbstractGridEngineWorker.checkOnJobs()

Parameters

batchjobID (string) – batch system job ID

Return type

int|toil.batchSystems.abstractBatchSystem.BatchJobExitReason: exit code int or BatchJobExitReason if something else happened other than job exiting.

classmethod supportsWorkerCleanup()[source]

Indicates whether this batch system invokes BatchSystemSupport.workerCleanup() after the last job for a particular workflow invocation finishes. Note that the term worker refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to shut down after the last worker process terminates.

classmethod supportsAutoDeployment()[source]

Whether this batch system supports auto-deployment of the user script itself.

If it does, the setUserScript() can be invoked to set the resource object representing the user script.

Note to implementors: If your implementation returns True here, it should also override

issueBatchJob(jobDesc, job_environment=None)[source]

Issues a job with the specified command to the batch system and returns a unique jobID.

Parameters

  • jobDesc – a toil.job.JobDescription

  • job_environment (Optional[Dict[str, str]]) – a collection of job-specific environment variables to be set on the worker.

Returns

a unique jobID that can be used to reference the newly issued job

killBatchJobs(jobIDs)[source]

Kills the given jobs, represented as Job ids, then checks they are dead by checking they are not in the list of issued jobs.

getIssuedBatchJobIDs()[source]

Gets the list of issued jobs

getRunningBatchJobIDs()[source]

Retrieve running job IDs from local and batch scheduler.

Respects statePollingWait and will return cached results if not within time period to talk with the scheduler.

getUpdatedBatchJob(maxWait)[source]

Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once.

Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait.

Parameters

maxWait – the number of seconds to block, waiting for a result

Returns

If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall-clock time the job ran for, or None if this batch system does not support tracking wall time.

shutdown()[source]

Signals worker to shutdown (via sentinel) then cleanly joins the thread

Return type

None

setEnv(name, value=None)[source]

Set an environment variable for the worker process before it is launched. The worker process will typically inherit the environment of the machine it is running on but this method makes it possible to override specific variables in that inherited environment before the worker is launched. Note that this mechanism is different to the one used by the worker internally to set up the environment of a job. A call to this method affects all jobs issued after this method returns. Note to implementors: This means that you would typically need to copy the variables before enqueuing a job.

If no value is provided it will be looked up from the current environment.

Parameters

  • name – the environment variable to be set on the worker.

  • value – if given, the environment variable given by name will be set to this value.

if None, the variable’s current value will be used as the value on the worker

Raises

RuntimeError – if value is None and the name cannot be found in the environment

classmethod getWaitDuration()[source]
sleepSeconds(sleeptime=1)[source]

Helper function to drop on all state-querying functions to avoid over-querying.

with_retries(operation, *args, **kwargs)[source]

Call operation with args and kwargs. If one of the calls to an SGE command fails, sleep and try again for a set number of times.