toil.batchSystems.mesos.batchSystem

Attributes

log

Classes

MesosBatchSystem

A Toil batch system implementation that uses Apache Mesos to distribute toil jobs as Mesos

Module Contents

toil.batchSystems.mesos.batchSystem.log
class toil.batchSystems.mesos.batchSystem.MesosBatchSystem(config, maxCores, maxMemory, maxDisk)

Bases: toil.batchSystems.local_support.BatchSystemLocalSupport, toil.batchSystems.abstractBatchSystem.AbstractScalableBatchSystem, pymesos.Scheduler

A Toil batch system implementation that uses Apache Mesos to distribute toil jobs as Mesos tasks over a cluster of agent nodes. A Mesos framework consists of a scheduler and an executor. This class acts as the scheduler and is typically run on the master node that also runs the Mesos master process with which the scheduler communicates via a driver component. The executor is implemented in a separate class. It is run on each agent node and communicates with the Mesos agent process via another driver object. The scheduler may also be run on a separate node from the master, which we then call somewhat ambiguously the driver node.

classmethod supportsAutoDeployment()

Whether this batch system supports auto-deployment of the user script itself.

If it does, the setUserScript() can be invoked to set the resource object representing the user script.

Note to implementors: If your implementation returns True here, it should also override

classmethod supportsWorkerCleanup()

Whether this batch system supports worker cleanup.

Indicates whether this batch system invokes BatchSystemSupport.workerCleanup() after the last job for a particular workflow invocation finishes. Note that the term worker refers to an entire node, not just a worker process. A worker process may run more than one job sequentially, and more than one concurrent worker process may exist on a worker node, for the same workflow. The batch system is said to shut down after the last worker process terminates.

class ExecutorInfo(nodeAddress, agentId, nodeInfo, lastSeen)
nodeAddress
agentId
nodeInfo
lastSeen
userScript = None
Type:

toil.resource.Resource

jobQueues
mesos_endpoint
mesos_name
killedJobIds
killJobIds
intendedKill
hostToJobIDs
nodeFilter = []
runningJobMap
taskResources
updatedJobsQueue
driver = None
frameworkId = None
executors
agentsByID
nonPreemptibleNodes
executor
lastTimeOfferLogged = 0
logPeriod = 30
ignoredNodes
setUserScript(userScript)

Set the user script for this workflow.

This method must be called before the first job is issued to this batch system, and only if supportsAutoDeployment() returns True, otherwise it will raise an exception.

Parameters:

userScript – the resource object representing the user script or module and the modules it depends on.

ignoreNode(nodeAddress)

Stop sending jobs to this node. Used in autoscaling when the autoscaler is ready to terminate a node, but jobs are still running. This allows the node to be terminated after the current jobs have finished.

Parameters:

nodeAddress – IP address of node to ignore.

unignoreNode(nodeAddress)

Stop ignoring this address, presumably after a node with this address has been terminated. This allows for the possibility of a new node having the same address as a terminated one.

issueBatchJob(command, jobNode, job_environment=None)

Issues the following command returning a unique jobID. Command is the string to run, memory is an int giving the number of bytes the job needs to run in and cores is the number of cpus needed for the job and error-file is the path of the file to place any std-err/std-out in.

Parameters:
killBatchJobs(jobIDs)

Kills the given job IDs. After returning, the killed jobs will not appear in the results of getRunningBatchJobIDs. The killed job will not be returned from getUpdatedBatchJob.

Parameters:

jobIDs – list of IDs of jobs to kill

getIssuedBatchJobIDs()

Gets all currently issued jobs

Returns:

A list of jobs (as job ID numbers) currently issued (may be running, or may be waiting to be run). Despite the result being a list, the ordering should not be depended upon.

getRunningBatchJobIDs()

Gets a map of jobs as job ID numbers that are currently running (not just waiting) and how long they have been running, in seconds.

Returns:

dictionary with currently running job ID number keys and how many seconds they have been running as the value

getUpdatedBatchJob(maxWait)

Returns information about job that has updated its status (i.e. ceased running, either successfully or with an error). Each such job will be returned exactly once.

Does not return info for jobs killed by killBatchJobs, although they may cause None to be returned earlier than maxWait.

Parameters:

maxWait – the number of seconds to block, waiting for a result

Returns:

If a result is available, returns UpdatedBatchJobInfo. Otherwise it returns None. wallTime is the number of seconds (a strictly positive float) in wall-clock time the job ran for, or None if this batch system does not support tracking wall time.

nodeInUse(nodeIP)

Can be used to determine if a worker node is running any tasks. If the node is doesn’t exist, this function should simply return False.

Parameters:

nodeIP (str) – The worker nodes private IP address

Returns:

True if the worker node has been issued any tasks, else False

Return type:

bool

getWaitDuration()

Gets the period of time to wait (floating point, in seconds) between checking for missing/overlong jobs.

shutdown()

Called at the completion of a toil invocation. Should cleanly terminate all worker threads.

Return type:

None

registered(driver, frameworkId, masterInfo)

Invoked when the scheduler successfully registers with a Mesos master

resourceOffers(driver, offers)

Invoked when resources have been offered to this framework.

statusUpdate(driver, update)

Invoked when the status of a task has changed (e.g., a agent is lost and so the task is lost, a task finishes and an executor sends a status update saying so, etc). Note that returning from this callback _acknowledges_ receipt of this status update! If for whatever reason the scheduler aborts during this callback (or the process exits) another status update will be delivered (note, however, that this is currently not true if the agent sending the status update is lost/fails during that time).

frameworkMessage(driver, executorId, agentId, message)

Invoked when an executor sends a message.

getNodes(preemptible=None, timeout=None)
Return all nodes that match:
  • preemptible status (None includes all)

  • timeout period (seen within the last # seconds, or None for all)

Parameters:
  • preemptible (Optional[bool])

  • timeout (Optional[int])

Return type:

dict[str, toil.batchSystems.abstractBatchSystem.NodeInfo]

reregistered(driver, masterInfo)

Invoked when the scheduler re-registers with a newly elected Mesos master.

executorLost(driver, executorId, agentId, status)

Invoked when an executor has exited/terminated abnormally.

classmethod get_default_mesos_endpoint()

Get the default IP/hostname and port that we will look for Mesos at.

Return type:

str

classmethod add_options(parser)

If this batch system provides any command line options, add them to the given parser.

Parameters:

parser (Union[argparse.ArgumentParser, argparse._ArgumentGroup])

Return type:

None

classmethod setOptions(setOption)

Process command line or configuration options relevant to this batch system.

Parameters:

setOption (toil.batchSystems.options.OptionSetter) – A function with signature setOption(option_name, parsing_function=None, check_function=None, default=None, env=None) returning nothing, used to update run configuration as a side effect.