toil.batchSystems.contained_executor

Executor for running inside a container.

Useful for Kubernetes batch system and TES batch system plugin.

Attributes

EXIT_STATUS_UNAVAILABLE_VALUE
logger

Classes

Resource

Represents a file or directory that will be deployed to each node before any jobs in the user script are invoked.

Functions

configure_root_logger()	Set up the root logger with handlers and formatting.
set_log_level(level[, set_logger])	Sets the root logger level to a given string level (like "INFO").
pack_job(command[, user_script, environment])	Create a command that runs the given command in an environment.
executor()	Main function of the _toil_contained_executor entrypoint.

Module Contents

toil.batchSystems.contained_executor.EXIT_STATUS_UNAVAILABLE_VALUE = 255

class toil.batchSystems.contained_executor.Resource

Bases: namedtuple('Resource', ('name', 'pathHash', 'url', 'contentHash'))

Represents a file or directory that will be deployed to each node before any jobs in the user script are invoked.

Each instance is a namedtuple with the following elements:

The pathHash element contains the MD5 (in hexdigest form) of the path to the resource on the leader node. The path, and therefore its hash is unique within a job store.

The url element is a “file:” or “http:” URL at which the resource can be obtained.

The contentHash element is an MD5 checksum of the resource, allowing for validation and caching of resources.

If the resource is a regular file, the type attribute will be ‘file’.

If the resource is a directory, the type attribute will be ‘dir’ and the URL will point at a ZIP archive of that directory.

resourceEnvNamePrefix = 'JTRES_'

rootDirPathEnvName

classmethod create(jobStore, leaderPath)

Saves the content of the file or directory at the given path to the given job store and returns a resource object representing that content for the purpose of obtaining it again at a generic, public URL. This method should be invoked on the leader node.

Parameters:

• jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

• leaderPath (str)

Return type:

Resource

refresh(jobStore)

Parameters:

jobStore (toil.jobStores.abstractJobStore.AbstractJobStore)

Return type:

Resource

classmethod prepareSystem()

Prepares this system for the downloading and lookup of resources. This method should only be invoked on a worker node. It is idempotent but not thread-safe.

Return type:

None

classmethod cleanSystem()

Remove all downloaded, localized resources.

Return type:

None

register()

Register this resource for later retrieval via lookup(), possibly in a child process.

Return type:

None

classmethod lookup(leaderPath)

Return a resource object representing a resource created from a file or directory at the given path on the leader.

This method should be invoked on the worker. The given path does not need to refer to an existing file or directory on the worker, it only identifies the resource within an instance of toil. This method returns None if no resource for the given path exists.

Parameters:

leaderPath (str)

Return type:

Optional[Resource]

download(callback=None)

Download this resource from its URL to a file on the local system.

This method should only be invoked on a worker node after the node was setup for accessing resources via prepareSystem().

Parameters:

callback (Optional[Callable[[str], None]])

Return type:

None

property localPath: str

Abstractmethod:

Return type:

str

Get the path to resource on the worker.

The file or directory at the returned path may or may not yet exist. Invoking download() will ensure that it does.

property localDirPath: str

The path to the directory containing the resource on the worker.

Return type:

str

pickle()

Return type:

str

classmethod unpickle(s)

Parameters:

s (str)

Return type:

Resource

toil.batchSystems.contained_executor.configure_root_logger()

Set up the root logger with handlers and formatting.

Should be called before any entry point tries to log anything, to ensure consistent formatting.

Return type:

None

toil.batchSystems.contained_executor.set_log_level(level, set_logger=None)

Sets the root logger level to a given string level (like “INFO”).

Parameters:

• level (str)

• set_logger (Optional[logging.Logger])

Return type:

None

toil.batchSystems.contained_executor.logger

toil.batchSystems.contained_executor.pack_job(command, user_script=None, environment=None)

Create a command that runs the given command in an environment.

Parameters:

• command (str) – Worker command to run to run the job.

• user_script (Optional[toil.resource.Resource]) – User script that will be loaded before the job is run.

• environment (Optional[Dict[str, str]]) – Environment variable dict that will be applied before the job is run.

Returns:

Command to run the job, as an argument list that can be run inside the Toil appliance container.

Return type:

List[str]

toil.batchSystems.contained_executor.executor()

Main function of the _toil_contained_executor entrypoint.

Runs inside the Toil container.

Responsible for setting up the user script and running the command for the job (which may in turn invoke the Toil worker entrypoint).

Return type:

None