toil.batchSystems.cleanup_support

Attributes

logger

Classes

BatchSystemCleanupSupport

Adds cleanup support when the last running job leaves a node, for batch

WorkerCleanupContext

Context manager used by BatchSystemCleanupSupport to implement

Module Contents

toil.batchSystems.cleanup_support.logger
class toil.batchSystems.cleanup_support.BatchSystemCleanupSupport(config, maxCores, maxMemory, maxDisk)[source]

Bases: toil.batchSystems.local_support.BatchSystemLocalSupport

Adds cleanup support when the last running job leaves a node, for batch systems that can’t provide it using the backing scheduler.

Parameters:
classmethod supportsWorkerCleanup()[source]

Whether this batch system supports worker cleanup.

Indicates whether this batch system invokes BatchSystemSupport.workerCleanup() after the last job for a particular workflow invocation finishes. Note that the term worker refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to shut down after the last worker process terminates.

Return type:

bool

getWorkerContexts()[source]

Get a list of picklable context manager objects to wrap worker work in, in order.

Can be used to ask the Toil worker to do things in-process (such as configuring environment variables, hot-deploying user scripts, or cleaning up a node) that would otherwise require a wrapping “executor” process.

Return type:

List[ContextManager[Any]]

class toil.batchSystems.cleanup_support.WorkerCleanupContext(workerCleanupInfo)[source]

Context manager used by BatchSystemCleanupSupport to implement cleanup on a node after the last worker is done working.

Gets wrapped around the worker’s work.

Parameters:

workerCleanupInfo (toil.batchSystems.abstractBatchSystem.WorkerCleanupInfo)

workerCleanupInfo
__enter__()[source]
Return type:

None

__exit__(type, value, traceback)[source]
Parameters:
Return type:

None