toil.lib.docker

Attributes

logger

FORGO

STOP

RM

Functions

get_host_accelerator_numbers()

Work out what accelerator is what.

dockerCheckOutput(*args, **kwargs)

dockerCall(*args, **kwargs)

subprocessDockerCall(*args, **kwargs)

apiDockerCall(job, image[, parameters, deferParam, ...])

A toil wrapper for the python docker API.

dockerKill(container_name[, gentleKill, remove, timeout])

Immediately kills a container. Equivalent to "docker kill":

dockerStop(container_name[, remove])

Gracefully kills a container. Equivalent to "docker stop":

containerIsRunning(container_name[, timeout])

Checks whether the container is running or not.

getContainerName(job)

Create a random string including the job name, and return it. Name will

Module Contents

toil.lib.docker.get_host_accelerator_numbers()[source]

Work out what accelerator is what.

For each accelerator visible to us, returns the host-side (for example, outside-of-Slurm-job) number for that accelerator. It is often the same as the apparent number.

Can be used with Docker’s –gpus=‘“device=#,#,#”’ option to forward the right GPUs as seen from a Docker daemon.

Return type:

List[int]

toil.lib.docker.logger
toil.lib.docker.FORGO = 0
toil.lib.docker.STOP = 1
toil.lib.docker.RM = 2
toil.lib.docker.dockerCheckOutput(*args, **kwargs)[source]
toil.lib.docker.dockerCall(*args, **kwargs)[source]
toil.lib.docker.subprocessDockerCall(*args, **kwargs)[source]
toil.lib.docker.apiDockerCall(job, image, parameters=None, deferParam=None, volumes=None, working_dir=None, containerName=None, entrypoint=None, detach=False, log_config=None, auto_remove=None, remove=False, user=None, environment=None, stdout=None, stderr=False, stream=False, demux=False, streamfile=None, accelerators=None, timeout=365 * 24 * 60 * 60, **kwargs)[source]

A toil wrapper for the python docker API.

Docker API Docs: https://docker-py.readthedocs.io/en/stable/index.html Docker API Code: https://github.com/docker/docker-py

This implements docker’s python API within toil so that calls are run as jobs, with the intention that failed/orphaned docker jobs be handled appropriately.

Example of using dockerCall in toil to index a FASTA file with SAMtools:

def toil_job(job):
    working_dir = job.fileStore.getLocalTempDir()
    path = job.fileStore.readGlobalFile(ref_id,
                                      os.path.join(working_dir, 'ref.fasta')
    parameters = ['faidx', path]
    apiDockerCall(job,
                  image='quay.io/ucgc_cgl/samtools:latest',
                  working_dir=working_dir,
                  parameters=parameters)

Note that when run with detach=False, or with detach=True and stdout=True or stderr=True, this is a blocking call. When run with detach=True and without output capture, the container is started and returned without waiting for it to finish.

Parameters:
  • job (toil.Job.job) – The Job instance for the calling function.

  • image (str) – Name of the Docker image to be used. (e.g. ‘quay.io/ucsc_cgl/samtools:latest’)

  • parameters (list[str]) – A list of string elements. If there are multiple elements, these will be joined with spaces. This handling of multiple elements provides backwards compatibility with previous versions which called docker using subprocess.check_call(). If list of lists: list[list[str]], then treat as successive commands chained with pipe.

  • working_dir (str) – The working directory.

  • deferParam (int) – Action to take on the container upon job completion. FORGO (0) leaves the container untouched and running. STOP (1) Sends SIGTERM, then SIGKILL if necessary to the container. RM (2) Immediately send SIGKILL to the container. This is the default behavior if deferParam is set to None.

  • name (str) – The name/ID of the container.

  • entrypoint (str) – Prepends commands sent to the container. See: https://docker-py.readthedocs.io/en/stable/containers.html

  • detach (bool) – Run the container in detached mode. (equivalent to ‘-d’)

  • stdout (bool) – Return logs from STDOUT when detach=False (default: True). Block and capture stdout to a file when detach=True (default: False). Output capture defaults to output.log, and can be specified with the “streamfile” kwarg.

  • stderr (bool) – Return logs from STDERR when detach=False (default: False). Block and capture stderr to a file when detach=True (default: False). Output capture defaults to output.log, and can be specified with the “streamfile” kwarg.

  • stream (bool) – If True and detach=False, return a log generator instead of a string. Ignored if detach=True. (default: False).

  • demux (bool) – Similar to demux in container.exec_run(). If True and detach=False, returns a tuple of (stdout, stderr). If stream=True, returns a log generator with tuples of (stdout, stderr). Ignored if detach=True. (default: False).

  • streamfile (str) – Collect container output to this file if detach=True and stderr and/or stdout are True. Defaults to “output.log”.

  • log_config (dict) – Specify the logs to return from the container. See: https://docker-py.readthedocs.io/en/stable/containers.html

  • remove (bool) – Remove the container on exit or not.

  • user (str) – The container will be run with the privileges of the user specified. Can be an actual name, such as ‘root’ or ‘lifeisaboutfishtacos’, or it can be the uid or gid of the user (‘0’ is root; ‘1000’ is an example of a less privileged uid or gid), or a complement of the uid:gid (RECOMMENDED), such as ‘0:0’ (root user : root group) or ‘1000:1000’ (some other user : some other user group).

  • environment – Allows one to set environment variables inside of the container, such as:

  • timeout (int) – Use the given timeout in seconds for interactions with the Docker daemon. Note that the underlying docker module is not always able to abort ongoing reads and writes in order to respect the timeout. Defaults to 1 year (i.e. wait essentially indefinitely).

  • accelerators (Optional[List[int]]) – Toil accelerator numbers (usually GPUs) to forward to the container. These are interpreted in the current Python process’s environment. See toil.lib.accelerators.get_individual_local_accelerators() for the menu of available accelerators.

  • kwargs – Additional keyword arguments supplied to the docker API’s run command. The list is 75 keywords total, for examples and full documentation see: https://docker-py.readthedocs.io/en/stable/containers.html

Returns:

Returns the standard output/standard error text, as requested, when detach=False. Returns the underlying docker.models.containers.Container object from the Docker API when detach=True.

toil.lib.docker.dockerKill(container_name, gentleKill=False, remove=False, timeout=365 * 24 * 60 * 60)[source]

Immediately kills a container. Equivalent to “docker kill”: https://docs.docker.com/engine/reference/commandline/kill/

Parameters:
  • container_name (str) – Name of the container being killed.

  • gentleKill (bool) – If True, trigger a graceful shutdown.

  • remove (bool) – If True, remove the container after it exits.

  • timeout (int) – Use the given timeout in seconds for interactions with the Docker daemon. Note that the underlying docker module is not always able to abort ongoing reads and writes in order to respect the timeout. Defaults to 1 year (i.e. wait essentially indefinitely).

Return type:

None

toil.lib.docker.dockerStop(container_name, remove=False)[source]

Gracefully kills a container. Equivalent to “docker stop”: https://docs.docker.com/engine/reference/commandline/stop/

Parameters:
  • container_name (str) – Name of the container being stopped.

  • remove (bool) – If True, remove the container after it exits.

Return type:

None

toil.lib.docker.containerIsRunning(container_name, timeout=365 * 24 * 60 * 60)[source]

Checks whether the container is running or not.

Parameters:
  • container_name (str) – Name of the container being checked.

  • timeout (int) – Use the given timeout in seconds for interactions with the Docker daemon. Note that the underlying docker module is not always able to abort ongoing reads and writes in order to respect the timeout. Defaults to 1 year (i.e. wait essentially indefinitely).

Returns:

True if status is ‘running’, False if status is anything else, and None if the container does not exist.

toil.lib.docker.getContainerName(job)[source]

Create a random string including the job name, and return it. Name will match [a-zA-Z0-9][a-zA-Z0-9_.-].