toil.batchSystems.htcondor

Module Contents

Classes

HTCondorBatchSystem

A partial implementation of BatchSystemSupport for batch systems run on a

Attributes

logger

JobTuple

schedd_lock

toil.batchSystems.htcondor.logger
toil.batchSystems.htcondor.JobTuple
toil.batchSystems.htcondor.schedd_lock
class toil.batchSystems.htcondor.HTCondorBatchSystem(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem

A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto-deployment is not implemented.

class Worker(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)

Bases: toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.Worker

A class that represents a thread of control.

This class can be safely subclassed in a limited fashion. There are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass.

Parameters:
createJobs(newJob)

Create a new job with the given attributes.

Implementation-specific; called by AbstractGridEngineWorker.run()

Parameters:

newJob (JobTuple)

Return type:

bool

prepareSubmission(cpu, memory, disk, jobID, jobName, command, environment)

Preparation in putting together a command-line string for submitting to batch system (via submitJob().)

Param:

int cpu

Param:

int memory

Param:

int jobID: Toil job ID

Param:

string subLine: the command line string to be called

Param:

string jobName: the name of the Toil job, to provide metadata to batch systems if desired

Param:

dict job_environment: the environment variables to be set on the worker

Return type:

List[str]

Parameters:
submitJob(submitObj)

Wrapper routine for submitting the actual command-line call, then processing the output to get the batch system job ID

Param:

string subLine: the literal command line string to be called

Return type:

string: batch system job ID, which will be stored internally

getRunningJobIDs()

Get a list of running job IDs. Implementation-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs()

Return type:

list

killJob(jobID)

Kill specific job with the Toil job ID. Implementation-specific; called by AbstractGridEngineWorker.killJobs()

Parameters:

jobID (string) – Toil job ID

getJobExitCode(batchJobID)

Returns job exit code and possibly an instance of abstractBatchSystem.BatchJobExitReason.

Returns None if the job is still running.

If the job is not running but the exit code is not available, it will be EXIT_STATUS_UNAVAILABLE_VALUE. Implementation-specific; called by AbstractGridEngineWorker.checkOnJobs().

The exit code will only be 0 if the job affirmatively succeeded.

Parameters:

batchjobID (string) – batch system job ID

connectSchedd()

Connect to HTCondor Schedd and yield a Schedd object.

You can only use it inside the context. Handles locking to make sure that only one thread is trying to do this at a time.

duplicate_quotes(value)

Escape a string by doubling up all single and double quotes.

This is used for arguments we pass to htcondor that need to be inside both double and single quote enclosures.

Parameters:

value (str)

Return type:

str

getEnvString(overrides)

Build an environment string that a HTCondor Submit object can use.

For examples of valid strings, see: http://research.cs.wisc.edu/htcondor/manual/current/condor_submit.html#man-condor-submit-environment

Parameters:

overrides (Dict[str, str])

Return type:

str

issueBatchJob(jobNode, job_environment=None)

Issues a job with the specified command to the batch system and returns a unique jobID.

Parameters:

  • jobDesc – a toil.job.JobDescription

  • job_environment (Optional[Dict[str, str]]) – a collection of job-specific environment variables to be set on the worker.

Returns:

a unique jobID that can be used to reference the newly issued job