toil.deferred

Module Contents

Classes

DeferredFunction	>>> from collections import defaultdict
DeferredFunctionManager	Implements a deferred function system. Each Toil worker will have an

Attributes

logger

toil.deferred.logger

class toil.deferred.DeferredFunction

Bases: namedtuple('DeferredFunction', 'function args kwargs name module')

>>> from collections import defaultdict
>>> df = DeferredFunction.create(defaultdict, None, {'x':1}, y=2)
>>> df
DeferredFunction(defaultdict, ...)
>>> df.invoke() == defaultdict(None, x=1, y=2)
True

__repr__

classmethod create(function, *args, **kwargs)

Capture the given callable and arguments as an instance of this class.

Parameters:

• function (callable) – The deferred action to take in the form of a function

• args (tuple) – Non-keyword arguments to the function

• kwargs (dict) – Keyword arguments to the function

invoke()

Invoke the captured function with the captured arguments.

__str__()

Return str(self).

class toil.deferred.DeferredFunctionManager(stateDirBase)

Implements a deferred function system. Each Toil worker will have an instance of this class. When a job is executed, it will happen inside a context manager from this class. If the job registers any “deferred” functions, they will be executed when the context manager is exited.

If the Python process terminates before properly exiting the context manager and running the deferred functions, and some other worker process enters or exits the per-job context manager of this class at a later time, or when the DeferredFunctionManager is shut down on the worker, the earlier job’s deferred functions will be picked up and run.

Note that deferred function cleanup is on a best-effort basis, and deferred functions may end up getting executed multiple times.

Internally, deferred functions are serialized into files in the given directory, which are locked by the owning process.

If that process dies, other processes can detect that the files are able to be locked, and will take them over.

Parameters:

stateDirBase (str) –

STATE_DIR_STEM = 'deferred'

PREFIX = 'func'

WIP_SUFFIX = '.tmp'

__del__()

Clean up our state on disk. We assume that the deferred functions we manage have all been executed, and none are currently recorded.

open()

Yields a single-argument function that allows for deferred functions of type toil.DeferredFunction to be registered. We use this design so deferred functions can be registered only inside this context manager.

Not thread safe.

classmethod cleanupWorker(stateDirBase)

Called by the batch system when it shuts down the node, after all workers are done, if the batch system supports worker cleanup. Checks once more for orphaned deferred functions and runs them.

Parameters:

stateDirBase (str) –

Return type:

None