toil.batchSystems.htcondor¶
Module Contents¶
Classes¶
A partial implementation of BatchSystemSupport for batch systems run on a |
Attributes¶
- toil.batchSystems.htcondor.logger¶
- toil.batchSystems.htcondor.JobTuple¶
- toil.batchSystems.htcondor.schedd_lock¶
- class toil.batchSystems.htcondor.HTCondorBatchSystem(config, maxCores, maxMemory, maxDisk)¶
Bases:
toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem
A partial implementation of BatchSystemSupport for batch systems run on a standard HPC cluster. By default auto-deployment is not implemented.
- class Worker(newJobsQueue, updatedJobsQueue, killQueue, killedJobsQueue, boss)¶
Bases:
toil.batchSystems.abstractGridEngineBatchSystem.AbstractGridEngineBatchSystem.Worker
A class that represents a thread of control.
This class can be safely subclassed in a limited fashion. There are two ways to specify the activity: by passing a callable object to the constructor, or by overriding the run() method in a subclass.
- Parameters
newJobsQueue (queue.Queue) –
updatedJobsQueue (queue.Queue) –
killQueue (queue.Queue) –
killedJobsQueue (queue.Queue) –
boss (AbstractGridEngineBatchSystem) –
- createJobs(newJob)¶
Create a new job with the given attributes.
Implementation-specific; called by AbstractGridEngineWorker.run()
- Parameters
newJob (JobTuple) –
- Return type
- prepareSubmission(cpu, memory, disk, jobID, jobName, command, environment)¶
Preparation in putting together a command-line string for submitting to batch system (via submitJob().)
- Param
int cpu
- Param
int memory
- Param
int jobID: Toil job ID
- Param
string subLine: the command line string to be called
- Param
string jobName: the name of the Toil job, to provide metadata to batch systems if desired
- Param
dict job_environment: the environment variables to be set on the worker
- Return type
List[str]
- Parameters
- submitJob(submitObj)¶
Wrapper routine for submitting the actual command-line call, then processing the output to get the batch system job ID
- Param
string subLine: the literal command line string to be called
- Return type
string: batch system job ID, which will be stored internally
- getRunningJobIDs()¶
Get a list of running job IDs. Implementation-specific; called by boss AbstractGridEngineBatchSystem implementation via AbstractGridEngineBatchSystem.getRunningBatchJobIDs()
- Return type
- killJob(jobID)¶
Kill specific job with the Toil job ID. Implementation-specific; called by AbstractGridEngineWorker.killJobs()
- Parameters
jobID (string) – Toil job ID
- getJobExitCode(batchJobID)¶
Returns job exit code or an instance of abstractBatchSystem.BatchJobExitReason. if something else happened other than the job exiting. Implementation-specific; called by AbstractGridEngineWorker.checkOnJobs()
- Parameters
batchjobID (string) – batch system job ID
- Return type
int|toil.batchSystems.abstractBatchSystem.BatchJobExitReason: exit code int or BatchJobExitReason if something else happened other than job exiting.
- connectSchedd()¶
Connect to HTCondor Schedd and yield a Schedd object.
You can only use it inside the context. Handles locking to make sure that only one thread is trying to do this at a time.
- duplicate_quotes(value)¶
Escape a string by doubling up all single and double quotes.
This is used for arguments we pass to htcondor that need to be inside both double and single quote enclosures.
- getEnvString(overrides)¶
Build an environment string that a HTCondor Submit object can use.
For examples of valid strings, see: http://research.cs.wisc.edu/htcondor/manual/current/condor_submit.html#man-condor-submit-environment
- issueBatchJob(jobNode, job_environment=None)¶
Issues a job with the specified command to the batch system and returns a unique jobID.
