toil.batchSystems.mesos.batchSystem

Attributes

EXIT_STATUS_UNAVAILABLE_VALUE

TaskData

ToilJob

log

Classes

AbstractScalableBatchSystem

A batch system that supports a variable number of worker nodes.

BatchJobExitReason

Enum where members are also (and must be) ints

NodeInfo

The coresUsed attribute is a floating point value between 0 (all cores idle) and 1 (all cores

UpdatedBatchJobInfo

BatchSystemLocalSupport

Adds a local queue for helper jobs, useful for CWL & others.

JobQueue

MesosShape

Represents a job or a node's "shape", in terms of the dimensions of memory, cores, disk and

OptionSetter

Protocol for the setOption function we get to let us set up CLI options for

JobDescription

Stores all the information that the Toil Leader ever needs to know about a Job.

MesosBatchSystem

A Toil batch system implementation that uses Apache Mesos to distribute toil jobs as Mesos

Functions

resolveEntryPoint(entryPoint)

Find the path to the given entry point that should work on a worker.

b_to_mib(n)

Convert a number from bytes to mibibytes.

mib_to_b(n)

Convert a number from mibibytes to bytes.

strict_bool(s)

Variant of bool() that only accepts two possible string values.

get_public_ip()

Get the IP that this machine uses to contact the internet.

get_user_name()

Get the current user name, or a suitable substitute string if the user name

Module Contents

toil.batchSystems.mesos.batchSystem.resolveEntryPoint(entryPoint)

Find the path to the given entry point that should work on a worker.

Returns:

The path found, which may be an absolute or a relative path.

Parameters:

entryPoint (str)

Return type:

str

toil.batchSystems.mesos.batchSystem.EXIT_STATUS_UNAVAILABLE_VALUE = 255
class toil.batchSystems.mesos.batchSystem.AbstractScalableBatchSystem

Bases: AbstractBatchSystem

A batch system that supports a variable number of worker nodes.

Used by toil.provisioners.clusterScaler.ClusterScaler to scale the number of worker nodes in the cluster up or down depending on overall load.

abstract getNodes(preemptible=None, timeout=600)

Returns a dictionary mapping node identifiers of preemptible or non-preemptible nodes to NodeInfo objects, one for each node.

Parameters:

  • preemptible (Optional[bool]) – If True (False) only (non-)preemptible nodes will be returned. If None, all nodes will be returned.

  • timeout (int)

Return type:

Dict[str, NodeInfo]

abstract nodeInUse(nodeIP)

Can be used to determine if a worker node is running any tasks. If the node is doesn’t exist, this function should simply return False.

Parameters:

nodeIP (str) – The worker nodes private IP address

Returns:

True if the worker node has been issued any tasks, else False

Return type:

bool

abstract ignoreNode(nodeAddress)

Stop sending jobs to this node. Used in autoscaling when the autoscaler is ready to terminate a node, but jobs are still running. This allows the node to be terminated after the current jobs have finished.

Parameters:

nodeAddress (str) – IP address of node to ignore.

Return type:

None

abstract unignoreNode(nodeAddress)

Stop ignoring this address, presumably after a node with this address has been terminated. This allows for the possibility of a new node having the same address as a terminated one.

Parameters:

nodeAddress (str)

Return type:

None

class toil.batchSystems.mesos.batchSystem.BatchJobExitReason

Bases: enum.IntEnum

Enum where members are also (and must be) ints

FINISHED: int = 1

Successfully finished.

FAILED: int = 2

Job finished, but failed.

LOST: int = 3

Preemptable failure (job’s executing host went away).

KILLED: int = 4

Job killed before finishing.

ERROR: int = 5

Internal error.

MEMLIMIT: int = 6

Job hit batch system imposed memory limit.

MISSING: int = 7

Job disappeared from the scheduler without actually stopping, so Toil killed it.

MAXJOBDURATION: int = 8

Job ran longer than –maxJobDuration, so Toil killed it.

PARTITION: int = 9

Job was not able to talk to the leader via the job store, so Toil declared it failed.

classmethod to_string(value)

Convert to human-readable string.

Given an int that may be or may be equal to a value from the enum, produce the string value of its matching enum entry, or a stringified int.

Parameters:

value (int)

Return type:

str

class toil.batchSystems.mesos.batchSystem.NodeInfo(coresUsed, memoryUsed, coresTotal, memoryTotal, requestedCores, requestedMemory, workers)

The coresUsed attribute is a floating point value between 0 (all cores idle) and 1 (all cores busy), reflecting the CPU load of the node.

The memoryUsed attribute is a floating point value between 0 (no memory used) and 1 (all memory used), reflecting the memory pressure on the node.

The coresTotal and memoryTotal attributes are the node’s resources, not just the used resources

The requestedCores and requestedMemory attributes are all the resources that Toil Jobs have reserved on the node, regardless of whether the resources are actually being used by the Jobs.

The workers attribute is an integer reflecting the number of workers currently active workers on the node.

Parameters:
class toil.batchSystems.mesos.batchSystem.UpdatedBatchJobInfo

Bases: NamedTuple

jobID: int
exitStatus: int

The exit status (integer value) of the job. 0 implies successful.

EXIT_STATUS_UNAVAILABLE_VALUE is used when the exit status is not available (e.g. job is lost, or otherwise died but actual exit code was not reported).

exitReason: BatchJobExitReason | None
wallTime: float | int | None
class toil.batchSystems.mesos.batchSystem.BatchSystemLocalSupport(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.abstractBatchSystem.BatchSystemSupport

Adds a local queue for helper jobs, useful for CWL & others.

Parameters:
handleLocalJob(command, jobDesc)

To be called by issueBatchJob.

Returns the jobID if the jobDesc has been submitted to the local queue, otherwise returns None

Parameters:
Return type:

Optional[int]

killLocalJobs(jobIDs)

Will kill all local jobs that match the provided jobIDs.

To be called by killBatchJobs.

Parameters:

jobIDs (List[int])

Return type:

None

getIssuedLocalJobIDs()

To be called by getIssuedBatchJobIDs.

Return type:

List[int]

getRunningLocalJobIDs()

To be called by getRunningBatchJobIDs().

Return type:

Dict[int, float]

getUpdatedLocalJob(maxWait)

To be called by getUpdatedBatchJob().

Parameters:

maxWait (int)

Return type:

Optional[toil.batchSystems.abstractBatchSystem.UpdatedBatchJobInfo]

getNextJobID()

Must be used to get job IDs so that the local and batch jobs do not conflict.

Return type:

int

shutdownLocal()

To be called from shutdown().

Return type:

None

class toil.batchSystems.mesos.batchSystem.JobQueue
insertJob(job, jobType)
jobIDs()
nextJobOfType(jobType)
typeEmpty(jobType)
class toil.batchSystems.mesos.batchSystem.MesosShape(wallTime, memory, cores, disk, preemptible)

Bases: toil.provisioners.abstractProvisioner.Shape

Represents a job or a node’s “shape”, in terms of the dimensions of memory, cores, disk and wall-time allocation.

The wallTime attribute stores the number of seconds of a node allocation, e.g. 3600 for AWS. FIXME: and for jobs?

The memory and disk attributes store the number of bytes required by a job (or provided by a node) in RAM or on disk (SSD or HDD), respectively.

Parameters:
__gt__(other)

Inverted. Returns True if self is less than other, else returns False.

This is because jobTypes are sorted in decreasing order, and this was done to give expensive jobs priority.

toil.batchSystems.mesos.batchSystem.TaskData
toil.batchSystems.mesos.batchSystem.ToilJob
class toil.batchSystems.mesos.batchSystem.OptionSetter

Bases: Protocol

Protocol for the setOption function we get to let us set up CLI options for each batch system.

Actual functionality is defined in the Config class.

OptionType
__call__(option_name, parsing_function=None, check_function=None, default=None, env=None, old_names=None)
Parameters:

  • option_name (str)

  • parsing_function (Optional[Callable[[Any], OptionType]])

  • check_function (Optional[Callable[[OptionType], Union[None, bool]]])

  • default (Optional[OptionType])

  • env (Optional[List[str]])

  • old_names (Optional[List[str]])

Return type:

bool

class toil.batchSystems.mesos.batchSystem.JobDescription(requirements, jobName, unitName='', displayName='', local=None)

Bases: Requirer

Stores all the information that the Toil Leader ever needs to know about a Job.

This includes:

  • Resource requirements.

  • Which jobs are children or follow-ons or predecessors of this job.

  • A reference to the Job object in the job store.

Can be obtained from an actual (i.e. executable) Job object, and can be used to obtain the Job object from the JobStore.

Never contains other Jobs or JobDescriptions: all reference is by ID.

Subclassed into variants for checkpoint jobs and service jobs that have their specific parameters.

Parameters:

  • requirements (Mapping[str, Union[int, str, bool]])

  • jobName (str)

  • unitName (Optional[str])

  • displayName (Optional[str])

  • local (Optional[bool])

get_names()

Get the names and ID of this job as a named tuple.

Return type:

toil.bus.Names

get_chain()

Get all the jobs that executed in this job’s chain, in order.

For each job, produces a named tuple with its various names and its original job store ID. The jobs in the chain are in execution order.

If the job hasn’t run yet or it didn’t chain, produces a one-item list.

Return type:

List[toil.bus.Names]

serviceHostIDsInBatches()

Find all batches of service host job IDs that can be started at the same time.

(in the order they need to start in)

Return type:

Iterator[List[str]]

successorsAndServiceHosts()

Get an iterator over all child, follow-on, and service job IDs.

Return type:

Iterator[str]

allSuccessors()

Get an iterator over all child, follow-on, and chained, inherited successor job IDs.

Follow-ons will come before children.

Return type:

Iterator[str]

successors_by_phase()

Get an iterator over all child/follow-on/chained inherited successor job IDs, along with their phase numbere on the stack.

Phases ececute higher numbers to lower numbers.

Return type:

Iterator[Tuple[int, str]]

property services
Get a collection of the IDs of service host jobs for this job, in arbitrary order.

Will be empty if the job has no unfinished services.

has_body()

Returns True if we have a job body associated, and False otherwise.

Return type:

bool

attach_body(file_store_id, user_script)

Attach a job body to this JobDescription.

Takes the file store ID that the body is stored at, and the required user script module.

The file store ID can also be “firstJob” for the root job, stored as a shared file instead.

Parameters:
Return type:

None

detach_body()

Drop the body reference from a JobDescription.

Return type:

None

get_body()

Get the information needed to load the job body.

Returns:

a file store ID (or magic shared file name “firstJob”) and a user script module.

Return type:

Tuple[str, toil.resource.ModuleDescriptor]

Fails if no body is attached; check has_body() first.

nextSuccessors()

Return the collection of job IDs for the successors of this job that are ready to run.

If those jobs have multiple predecessor relationships, they may still be blocked on other jobs.

Returns None when at the final phase (all successors done), and an empty collection if there are more phases but they can’t be entered yet (e.g. because we are waiting for the job itself to run).

Return type:

Optional[Set[str]]

filterSuccessors(predicate)

Keep only successor jobs for which the given predicate function approves.

The predicate function is called with the job’s ID.

Treats all other successors as complete and forgets them.

Parameters:

predicate (Callable[[str], bool])

Return type:

None

filterServiceHosts(predicate)

Keep only services for which the given predicate approves.

The predicate function is called with the service host job’s ID.

Treats all other services as complete and forgets them.

Parameters:

predicate (Callable[[str], bool])

Return type:

None

clear_nonexistent_dependents(job_store)

Remove all references to child, follow-on, and associated service jobs that do not exist.

That is to say, all those that have been completed and removed.

Parameters:

job_store (toil.jobStores.abstractJobStore.AbstractJobStore)

Return type:

None

clear_dependents()

Remove all references to successor and service jobs.

Return type:

None

is_subtree_done()

Check if the subtree is done.

Returns:

True if the job appears to be done, and all related child, follow-on, and service jobs appear to be finished and removed.

Return type:

bool

replace(other)

Take on the ID of another JobDescription, retaining our own state and type.

When updated in the JobStore, we will save over the other JobDescription.

Useful for chaining jobs: the chained-to job can replace the parent job.

Merges cleanup state and successors other than this job from the job being replaced into this one.

Parameters:

other (JobDescription) – Job description to replace.

Return type:

None

assert_is_not_newer_than(other)

Make sure this JobDescription is not newer than a prospective new version of the JobDescription.

Parameters:

other (JobDescription)

Return type:

None

is_updated_by(other)

Return True if the passed JobDescription is a distinct, newer version of this one.

Parameters:

other (JobDescription)

Return type:

bool

addChild(childID)

Make the job with the given ID a child of the described job.

Parameters:

childID (str)

Return type:

None

addFollowOn(followOnID)

Make the job with the given ID a follow-on of the described job.

Parameters:

followOnID (str)

Return type:

None

addServiceHostJob(serviceID, parentServiceID=None)

Make the ServiceHostJob with the given ID a service of the described job.

If a parent ServiceHostJob ID is given, that parent service will be started first, and must have already been added.

hasChild(childID)

Return True if the job with the given ID is a child of the described job.

Parameters:

childID (str)

Return type:

bool

hasFollowOn(followOnID)

Test if the job with the given ID is a follow-on of the described job.

Parameters:

followOnID (str)

Return type:

bool

hasServiceHostJob(serviceID)

Test if the ServiceHostJob is a service of the described job.

Return type:

bool

renameReferences(renames)

Apply the given dict of ID renames to all references to jobs.

Does not modify our own ID or those of finished predecessors. IDs not present in the renames dict are left as-is.

Parameters:

renames (Dict[TemporaryID, str]) – Rename operations to apply.

Return type:

None

addPredecessor()

Notify the JobDescription that a predecessor has been added to its Job.

Return type:

None

onRegistration(jobStore)

Perform setup work that requires the JobStore.

Called by the Job saving logic when this JobDescription meets the JobStore and has its ID assigned.

Overridden to perform setup work (like hooking up flag files for service jobs) that requires the JobStore.

Parameters:

jobStore (toil.jobStores.abstractJobStore.AbstractJobStore) – The job store we are being placed into

Return type:

None

setupJobAfterFailure(exit_status=None, exit_reason=None)

Configure job after a failure.

Reduce the remainingTryCount if greater than zero and set the memory to be at least as big as the default memory (in case of exhaustion of memory, which is common).

Requires a configuration to have been assigned (see toil.job.Requirer.assignConfig()).

Parameters:
Return type:

None

getLogFileHandle(jobStore)

Create a context manager that yields a file handle to the log file.

Assumes logJobStoreFileID is set.

property remainingTryCount
Get the number of tries remaining.

The try count set on the JobDescription, or the default based on the retry count from the config if none is set.

clearRemainingTryCount()

Clear remainingTryCount and set it back to its default value.

Returns:

True if a modification to the JobDescription was made, and False otherwise.

Return type:

bool

__str__()

Produce a useful logging string identifying this job.

Return type:

str

__repr__()

Return repr(self).

reserve_versions(count)

Reserve a job version number for later, for journaling asynchronously.

Parameters:

count (int)

Return type:

None

pre_update_hook()

Run before pickling and saving a created or updated version of this job.

Called by the job store.

Return type:

None

toil.batchSystems.mesos.batchSystem.b_to_mib(n)

Convert a number from bytes to mibibytes.

Parameters:

n (Union[int, float])

Return type:

float

toil.batchSystems.mesos.batchSystem.mib_to_b(n)

Convert a number from mibibytes to bytes.

Parameters:

n (Union[int, float])

Return type:

float

toil.batchSystems.mesos.batchSystem.strict_bool(s)

Variant of bool() that only accepts two possible string values.

Parameters:

s (str)

Return type:

bool

toil.batchSystems.mesos.batchSystem.get_public_ip()

Get the IP that this machine uses to contact the internet.

If behind a NAT, this will still be this computer’s IP, and not the router’s.

Return type:

str

toil.batchSystems.mesos.batchSystem.get_user_name()

Get the current user name, or a suitable substitute string if the user name is not available.

Return type:

str

toil.batchSystems.mesos.batchSystem.log
class toil.batchSystems.mesos.batchSystem.MesosBatchSystem(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.local_support.BatchSystemLocalSupport, toil.batchSystems.abstractBatchSystem.AbstractScalableBatchSystem, pymesos.Scheduler

A Toil batch system implementation that uses Apache Mesos to distribute toil jobs as Mesos tasks over a cluster of agent nodes. A Mesos framework consists of a scheduler and an executor. This class acts as the scheduler and is typically run on the master node that also runs the Mesos master process with which the scheduler communicates via a driver component. The executor is implemented in a separate class. It is run on each agent node and communicates with the Mesos agent process via another driver object. The scheduler may also be run on a separate node from the master, which we then call somewhat ambiguously the driver node.

classmethod supportsAutoDeployment()

Whether this batch system supports auto-deployment of the user script itself.

If it does, the setUserScript() can be invoked to set the resource object representing the user script.

Note to implementors: If your implementation returns True here, it should also override

classmethod supportsWorkerCleanup()

Whether this batch system supports worker cleanup.

Indicates whether this batch system invokes BatchSystemSupport.workerCleanup() after the last job for a particular workflow invocation finishes. Note that the term worker refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to shut down after the last worker process terminates.

class ExecutorInfo(nodeAddress, agentId, nodeInfo, lastSeen)
userScript = None
Type:

toil.resource.Resource

setUserScript(userScript)

Set the user script for this workflow.

This method must be called before the first job is issued to this batch system, and only if supportsAutoDeployment() returns True, otherwise it will raise an exception.

Parameters:

userScript – the resource object representing the user script or module and the modules it depends on.

ignoreNode(nodeAddress)

Stop sending jobs to this node. Used in autoscaling when the autoscaler is ready to terminate a node, but jobs are still running. This allows the node to be terminated after the current jobs have finished.

Parameters:

nodeAddress – IP address of node to ignore.

unignoreNode(nodeAddress)

Stop ignoring this address, presumably after a node with this address has been terminated. This allows for the possibility of a new node having the same address as a terminated one.

issueBatchJob(command, jobNode, job_environment=None)

Issues the following command returning a unique jobID. Command is the string to run, memory is an int giving the number of bytes the job needs to run in and cores is the number of cpus needed for the job and error-file is the path of the file to place any std-err/std-out in.

Parameters:
killBatchJobs(jobIDs)

Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob.

Parameters:

jobIDs – list of IDs of jobs to kill

getIssuedBatchJobIDs()

Gets all currently issued jobs

Returns:

A list of jobs (as job ID numbers) currently issued (may be running, or may be waiting to be run). Despite the result being a list, the ordering should not be depended upon.

getRunningBatchJobIDs()

Gets a map of jobs as job ID numbers that are currently running (not just waiting) and how long they have been running, in seconds.

Returns:

dictionary with currently running job ID number keys and how many seconds they have been running as the value

getUpdatedBatchJob(maxWait)

Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once.

Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait.

Parameters:

maxWait – the number of seconds to block, waiting for a result

Returns:

If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall-clock time the job ran for, or None if this batch system does not support tracking wall time.

nodeInUse(nodeIP)

Can be used to determine if a worker node is running any tasks. If the node is doesn’t exist, this function should simply return False.

Parameters:

nodeIP (str) – The worker nodes private IP address

Returns:

True if the worker node has been issued any tasks, else False

Return type:

bool

getWaitDuration()

Gets the period of time to wait (floating point, in seconds) between checking for missing/overlong jobs.

shutdown()

Called at the completion of a toil invocation. Should cleanly terminate all worker threads.

Return type:

None

registered(driver, frameworkId, masterInfo)

Invoked when the scheduler successfully registers with a Mesos master

resourceOffers(driver, offers)

Invoked when resources have been offered to this framework.

statusUpdate(driver, update)

Invoked when the status of a task has changed (e.g., a agent is lost and so the task is lost, a task finishes and an executor sends a status update saying so, etc). Note that returning from this callback _acknowledges_ receipt of this status update! If for whatever reason the scheduler aborts during this callback (or the process exits) another status update will be delivered (note, however, that this is currently not true if the agent sending the status update is lost/fails during that time).

frameworkMessage(driver, executorId, agentId, message)

Invoked when an executor sends a message.

getNodes(preemptible=None, timeout=None)
Return all nodes that match:

  • preemptible status (None includes all)

  • timeout period (seen within the last # seconds, or None for all)

Parameters:

  • preemptible (Optional[bool])

  • timeout (Optional[int])

Return type:

Dict[str, toil.batchSystems.abstractBatchSystem.NodeInfo]

reregistered(driver, masterInfo)

Invoked when the scheduler re-registers with a newly elected Mesos master.

executorLost(driver, executorId, agentId, status)

Invoked when an executor has exited/terminated abnormally.

classmethod get_default_mesos_endpoint()

Get the default IP/hostname and port that we will look for Mesos at.

Return type:

str

classmethod add_options(parser)

If this batch system provides any command line options, add them to the given parser.

Parameters:

parser (Union[argparse.ArgumentParser, argparse._ArgumentGroup])

Return type:

None

classmethod setOptions(setOption)

Process command line or configuration options relevant to this batch system.

Parameters:

setOption (toil.batchSystems.options.OptionSetter) – A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect.