toil.batchSystems.cleanup_support

Module Contents

Classes

BatchSystemCleanupSupport	Adds cleanup support when the last running job leaves a node, for batch
WorkerCleanupContext	Context manager used by BatchSystemCleanupSupport to implement

Attributes

logger

toil.batchSystems.cleanup_support.logger

class toil.batchSystems.cleanup_support.BatchSystemCleanupSupport(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.local_support.BatchSystemLocalSupport

digraph inheritancef759858a2d { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "ABC" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="Helper class that provides a standard way to create an ABC using"]; "AbstractBatchSystem" [URL="../abstractBatchSystem/index.html#toil.batchSystems.abstractBatchSystem.AbstractBatchSystem",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="An abstract base class to represent the interface the batch system must provide to Toil."]; "ABC" -> "AbstractBatchSystem" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemCleanupSupport" [URL="#toil.batchSystems.cleanup_support.BatchSystemCleanupSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Adds cleanup support when the last running job leaves a node, for batch"]; "BatchSystemLocalSupport" -> "BatchSystemCleanupSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemLocalSupport" [URL="../local_support/index.html#toil.batchSystems.local_support.BatchSystemLocalSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Adds a local queue for helper jobs, useful for CWL & others."]; "BatchSystemSupport" -> "BatchSystemLocalSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; "BatchSystemSupport" [URL="../abstractBatchSystem/index.html#toil.batchSystems.abstractBatchSystem.BatchSystemSupport",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Partial implementation of AbstractBatchSystem, support methods."]; "AbstractBatchSystem" -> "BatchSystemSupport" [arrowsize=0.5,style="setlinewidth(0.5)"]; }

Adds cleanup support when the last running job leaves a node, for batch systems that can’t provide it using the backing scheduler.

Parameters

• config (toil.common.Config) –

• maxCores (float) –

• maxMemory (int) –

• maxDisk (int) –

classmethod supportsWorkerCleanup()

Indicates whether this batch system invokes BatchSystemSupport.workerCleanup() after the last job for a particular workflow invocation finishes. Note that the term worker refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to shut down after the last worker process terminates.

Return type

bool

getWorkerContexts()

Get a list of picklable context manager objects to wrap worker work in, in order.

Can be used to ask the Toil worker to do things in-process (such as configuring environment variables, hot-deploying user scripts, or cleaning up a node) that would otherwise require a wrapping “executor” process.

Return type

List[ContextManager[Any]]

class toil.batchSystems.cleanup_support.WorkerCleanupContext(workerCleanupInfo)

Context manager used by BatchSystemCleanupSupport to implement cleanup on a node after the last worker is done working.

Gets wrapped around the worker’s work.

Parameters

workerCleanupInfo (toil.batchSystems.abstractBatchSystem.WorkerCleanupInfo) –

__enter__()

Return type

None

__exit__(type, value, traceback)

Parameters

• type (Optional[Type[BaseException]]) –

• value (Optional[BaseException]) –

• traceback (Optional[types.TracebackType]) –

Return type

None