Toil Job API

Functions to wrap jobs and return values (promises).

FunctionWrappingJob

The subclass of Job for wrapping user functions.

class toil.job.FunctionWrappingJob(userFunction, *args, **kwargs)[source]

Job used to wrap a function. In its run method the wrapped function is called.

__init__(userFunction, *args, **kwargs)[source]
Parameters:

userFunction (callable) – The function to wrap. It will be called with *args and **kwargs as arguments.

The keywords memory, cores, disk, accelerators`, ``preemptible and checkpoint are reserved keyword arguments that if specified will be used to determine the resources required for the job, as toil.job.Job.__init__(). If they are keyword arguments to the function they will be extracted from the function definition, but may be overridden by the user (as you would expect).

run(fileStore)[source]

Override this function to perform work and dynamically create successor jobs.

Parameters:

fileStore – Used to create local and globally sharable temporary files and to send log messages to the leader process.

Returns:

The return value of the function can be passed to other jobs by means of toil.job.Job.rv().

JobFunctionWrappingJob

The subclass of FunctionWrappingJob for wrapping user job functions.

class toil.job.JobFunctionWrappingJob(userFunction, *args, **kwargs)[source]

A job function is a function whose first argument is a Job instance that is the wrapping job for the function. This can be used to add successor jobs for the function and perform all the functions the Job class provides.

To enable the job function to get access to the toil.fileStores.abstractFileStore.AbstractFileStore instance (see toil.job.Job.run()), it is made a variable of the wrapping job called fileStore.

To specify a job’s resource requirements the following default keyword arguments can be specified:

  • memory

  • disk

  • cores

  • accelerators

  • preemptible

For example to wrap a function into a job we would call:

Job.wrapJobFn(myJob, memory='100k', disk='1M', cores=0.1)
run(fileStore)[source]

Override this function to perform work and dynamically create successor jobs.

Parameters:

fileStore – Used to create local and globally sharable temporary files and to send log messages to the leader process.

Returns:

The return value of the function can be passed to other jobs by means of toil.job.Job.rv().

EncapsulatedJob

The subclass of Job for encapsulating a job, allowing a subgraph of jobs to be treated as a single job.

class toil.job.EncapsulatedJob(job, unitName=None)[source]

A convenience Job class used to make a job subgraph appear to be a single job.

Let A be the root job of a job subgraph and B be another job we’d like to run after A and all its successors have completed, for this use encapsulate:

#  Job A and subgraph, Job B
A, B = A(), B()
Aprime = A.encapsulate()
Aprime.addChild(B)
#  B will run after A and all its successors have completed, A and its subgraph of
# successors in effect appear to be just one job.

If the job being encapsulated has predecessors (e.g. is not the root job), then the encapsulated job will inherit these predecessors. If predecessors are added to the job being encapsulated after the encapsulated job is created then the encapsulating job will NOT inherit these predecessors automatically. Care should be exercised to ensure the encapsulated job has the proper set of predecessors.

The return value of an encapsulated job (as accessed by the toil.job.Job.rv() function) is the return value of the root job, e.g. A().encapsulate().rv() and A().rv() will resolve to the same value after A or A.encapsulate() has been run.

__init__(job, unitName=None)[source]
Parameters:
  • job (toil.job.Job) – the job to encapsulate.

  • unitName (str) – human-readable name to identify this job instance.

addChild(childJob)[source]

Add a childJob to be run as child of this job.

Child jobs will be run directly after this job’s toil.job.Job.run() method has completed.

Returns:

childJob: for call chaining

addService(service, parentService=None)[source]

Add a service.

The toil.job.Job.Service.start() method of the service will be called after the run method has completed but before any successors are run. The service’s toil.job.Job.Service.stop() method will be called once the successors of the job have been run.

Services allow things like databases and servers to be started and accessed by jobs in a workflow.

Raises:

toil.job.JobException – If service has already been made the child of a job or another service.

Parameters:
  • service – Service to add.

  • parentService – Service that will be started before ‘service’ is started. Allows trees of services to be established. parentService must be a service of this job.

Returns:

a promise that will be replaced with the return value from toil.job.Job.Service.start() of service in any successor of the job.

addFollowOn(followOnJob)[source]

Add a follow-on job.

Follow-on jobs will be run after the child jobs and their successors have been run.

Returns:

followOnJob for call chaining

rv(*path)[source]

Create a promise (toil.job.Promise).

The “promise” representing a return value of the job’s run method, or, in case of a function-wrapping job, the wrapped function’s return value.

Parameters:

path ((Any)) – Optional path for selecting a component of the promised return value. If absent or empty, the entire return value will be used. Otherwise, the first element of the path is used to select an individual item of the return value. For that to work, the return value must be a list, dictionary or of any other type implementing the __getitem__() magic method. If the selected item is yet another composite value, the second element of the path can be used to select an item from it, and so on. For example, if the return value is [6,{‘a’:42}], .rv(0) would select 6 , rv(1) would select {‘a’:3} while rv(1,’a’) would select 3. To select a slice from a return value that is slicable, e.g. tuple or list, the path element should be a slice object. For example, assuming that the return value is [6, 7, 8, 9] then .rv(slice(1, 3)) would select [7, 8]. Note that slicing really only makes sense at the end of path.

Return type:

Promise

Returns:

A promise representing the return value of this jobs toil.job.Job.run() method.

prepareForPromiseRegistration(jobStore)[source]

Set up to allow this job’s promises to register themselves.

Prepare this job (the promisor) so that its promises can register themselves with it, when the jobs they are promised to (promisees) are serialized.

The promissee holds the reference to the promise (usually as part of the job arguments) and when it is being pickled, so will the promises it refers to. Pickling a promise triggers it to be registered with the promissor.

Promise

The class used to reference return values of jobs/services not yet run/started.

class toil.job.Promise(job, path)[source]

References a return value from a method as a promise before the method itself is run.

References a return value from a toil.job.Job.run() or toil.job.Job.Service.start() method as a promise before the method itself is run.

Let T be a job. Instances of Promise (termed a promise) are returned by T.rv(), which is used to reference the return value of T’s run function. When the promise is passed to the constructor (or as an argument to a wrapped function) of a different, successor job the promise will be replaced by the actual referenced return value. This mechanism allows a return values from one job’s run method to be input argument to job before the former job’s run function has been executed.

Parameters:
Return type:

Promise

filesToDelete = {}

A set of IDs of files containing promised values when we know we won’t need them anymore

__init__(job, path)[source]

Initialize this promise.

Parameters:
  • job (Job) – the job whose return value this promise references

  • path (Any) – see Job.rv()

  • job

class toil.job.PromisedRequirement(valueOrCallable, *args)[source]

Class for dynamically allocating job function resource requirements.

(involving toil.job.Promise instances.)

Use when resource requirements depend on the return value of a parent function. PromisedRequirements can be modified by passing a function that takes the Promise as input.

For example, let f, g, and h be functions. Then a Toil workflow can be defined as follows:: A = Job.wrapFn(f) B = A.addChildFn(g, cores=PromisedRequirement(A.rv()) C = B.addChildFn(h, cores=PromisedRequirement(lambda x: 2*x, B.rv()))

__init__(valueOrCallable, *args)[source]

Initialize this Promised Requirement.

Parameters:
  • valueOrCallable – A single Promise instance or a function that takes args as input parameters.

  • args (int or .Promise) – variable length argument list

getValue()[source]

Return PromisedRequirement value.

static convertPromises(kwargs)[source]

Return True if reserved resource keyword is a Promise or PromisedRequirement instance.

Converts Promise instance to PromisedRequirement.

Parameters:

kwargs (Dict[str, Any]) – function keyword arguments

Return type:

bool