toil

Subpackages

Submodules

Attributes

log

KNOWN_EXTANT_IMAGES

cache_path

Exceptions

ApplianceImageNotFound

Error raised when using TOIL_APPLIANCE_SELF results in an HTTP error.

Functions

which(cmd[, mode, path])

Return the path with conforms to the given mode on the Path.

toilPackageDirPath()

Return the absolute path of the directory that corresponds to the top-level toil package.

inVirtualEnv()

Test if we are inside a virtualenv or Conda virtual environment.

resolveEntryPoint(entryPoint)

Find the path to the given entry point that should work on a worker.

physicalMemory()

Calculate the total amount of physical memory, in bytes.

physicalDisk(directory)

applianceSelf([forceDockerAppliance])

Return the fully qualified name of the Docker image to start Toil appliance containers from.

customDockerInitCmd()

Return the custom command set by the TOIL_CUSTOM_DOCKER_INIT_COMMAND environment variable.

customInitCmd()

Return the custom command set by the TOIL_CUSTOM_INIT_COMMAND environment variable.

lookupEnvVar(name, envName, defaultValue)

Look up environment variables that control Toil and log the result.

checkDockerImageExists(appliance)

Attempt to check a url registryName for the existence of a docker image with a given tag.

parseDockerAppliance(appliance)

Derive parsed registry, image reference, and tag from a docker image string.

checkDockerSchema(appliance)

requestCheckRegularDocker(origAppliance, registryName, ...)

Check if an image exists using the requests library.

requestCheckDockerIo(origAppliance, imageName, tag)

Check docker.io to see if an image exists using the requests library.

logProcessContext(config)

Package Contents

toil.log[source]
toil.which(cmd, mode=os.F_OK | os.X_OK, path=None)[source]

Return the path with conforms to the given mode on the Path.

[Copy-pasted in from python3.6’s shutil.which().]

mode defaults to os.F_OK | os.X_OK. path defaults to the result of os.environ.get(“PATH”), or can be overridden with a custom search path.

Returns:

The path found, or None.

Return type:

Optional[str]

toil.toilPackageDirPath()[source]

Return the absolute path of the directory that corresponds to the top-level toil package.

The return value is guaranteed to end in ‘/toil’.

Return type:

str

toil.inVirtualEnv()[source]

Test if we are inside a virtualenv or Conda virtual environment.

Return type:

bool

toil.resolveEntryPoint(entryPoint)[source]

Find the path to the given entry point that should work on a worker.

Returns:

The path found, which may be an absolute or a relative path.

Parameters:

entryPoint (str)

Return type:

str

toil.physicalMemory()[source]

Calculate the total amount of physical memory, in bytes.

>>> n = physicalMemory()
>>> n > 0
True
>>> n == physicalMemory()
True
Return type:

int

toil.physicalDisk(directory)[source]
Parameters:

directory (str)

Return type:

int

toil.applianceSelf(forceDockerAppliance=False)[source]

Return the fully qualified name of the Docker image to start Toil appliance containers from.

The result is determined by the current version of Toil and three environment variables: TOIL_DOCKER_REGISTRY, TOIL_DOCKER_NAME and TOIL_APPLIANCE_SELF.

TOIL_DOCKER_REGISTRY specifies an account on a publicly hosted docker registry like Quay or Docker Hub. The default is UCSC’s CGL account on Quay.io where the Toil team publishes the official appliance images. TOIL_DOCKER_NAME specifies the base name of the image. The default of toil will be adequate in most cases. TOIL_APPLIANCE_SELF fully qualifies the appliance image, complete with registry, image name and version tag, overriding both TOIL_DOCKER_NAME and TOIL_DOCKER_REGISTRY` as well as the version tag of the image. Setting TOIL_APPLIANCE_SELF will not be necessary in most cases.

Parameters:

forceDockerAppliance (bool)

Return type:

str

toil.customDockerInitCmd()[source]

Return the custom command set by the TOIL_CUSTOM_DOCKER_INIT_COMMAND environment variable.

The custom docker command is run prior to running the workers and/or the primary node’s services.

This can be useful for doing any custom initialization on instances (e.g. authenticating to private docker registries). Any single quotes are escaped and the command cannot contain a set of blacklisted chars (newline or tab).

Returns:

The custom command, or an empty string is returned if the environment variable is not set.

Return type:

str

toil.customInitCmd()[source]

Return the custom command set by the TOIL_CUSTOM_INIT_COMMAND environment variable.

The custom init command is run prior to running Toil appliance itself in workers and/or the primary node (i.e. this is run one stage before TOIL_CUSTOM_DOCKER_INIT_COMMAND).

This can be useful for doing any custom initialization on instances (e.g. authenticating to private docker registries). Any single quotes are escaped and the command cannot contain a set of blacklisted chars (newline or tab).

returns: the custom command or n empty string is returned if the environment variable is not set.

Return type:

str

toil.lookupEnvVar(name, envName, defaultValue)[source]

Look up environment variables that control Toil and log the result.

Parameters:

  • name (str) – the human readable name of the variable

  • envName (str) – the name of the environment variable to lookup

  • defaultValue (str) – the fall-back value

Returns:

the value of the environment variable or the default value the variable is not set

Return type:

str

toil.checkDockerImageExists(appliance)[source]

Attempt to check a url registryName for the existence of a docker image with a given tag.

Parameters:

appliance (str) – The url of a docker image’s registry (with a tag) of the form: ‘quay.io/<repo_path>:<tag>’ or ‘<repo_path>:<tag>’. Examples: ‘quay.io/ucsc_cgl/toil:latest’, ‘ubuntu:latest’, or ‘broadinstitute/genomes-in-the-cloud:2.0.0’.

Returns:

Raises an exception if the docker image cannot be found or is invalid. Otherwise, it will return the appliance string.

Return type:

str

toil.parseDockerAppliance(appliance)[source]

Derive parsed registry, image reference, and tag from a docker image string.

Example: “quay.io/ucsc_cgl/toil:latest” Should return: “quay.io”, “ucsc_cgl/toil”, “latest”

If a registry is not defined, the default is: “docker.io” If a tag is not defined, the default is: “latest”

Parameters:

appliance (str) – The full url of the docker image originally specified by the user (or the default). e.g. “quay.io/ucsc_cgl/toil:latest”

Returns:

registryName, imageName, tag

Return type:

Tuple[str, str, str]

toil.checkDockerSchema(appliance)[source]
exception toil.ApplianceImageNotFound(origAppliance, url, statusCode)[source]

Bases: docker.errors.ImageNotFound

Error raised when using TOIL_APPLIANCE_SELF results in an HTTP error.

Parameters:

  • origAppliance (str) – The full url of the docker image originally specified by the user (or the default). e.g. “quay.io/ucsc_cgl/toil:latest”

  • url (str) – The URL at which the image’s manifest is supposed to appear

  • statusCode (int) – the failing HTTP status code returned by the URL

msg
toil.KNOWN_EXTANT_IMAGES
toil.requestCheckRegularDocker(origAppliance, registryName, imageName, tag)[source]

Check if an image exists using the requests library.

URL is based on the docker v2 schema.

This has the following format: https://{websitehostname}.io/v2/{repo}/manifests/{tag}

Does not work with the official (docker.io) site, because they require an OAuth token, so a separate check is done for docker.io images.

Parameters:

  • origAppliance (str) – The full url of the docker image originally specified by the user (or the default). For example, quay.io/ucsc_cgl/toil:latest.

  • registryName (str) – The url of a docker image’s registry. For example, quay.io.

  • imageName (str) – The image, including path and excluding the tag. For example, ucsc_cgl/toil.

  • tag (str) – The tag used at that docker image’s registry. For example, latest.

Raises:

ApplianceImageNotFound if no match is found.

Returns:

Return True if match found.

Return type:

bool

toil.requestCheckDockerIo(origAppliance, imageName, tag)[source]

Check docker.io to see if an image exists using the requests library.

URL is based on the docker v2 schema. Requires that an access token be fetched first.

Parameters:

  • origAppliance (str) – The full url of the docker image originally specified by the user (or the default). For example, ubuntu:latest.

  • imageName (str) – The image, including path and excluding the tag. For example, ubuntu.

  • tag (str) – The tag used at that docker image’s registry. For example, latest.

Raises:

ApplianceImageNotFound if no match is found.

Returns:

Return True if match found.

Return type:

bool

toil.logProcessContext(config)[source]
Parameters:

config (common.Config)

Return type:

None

toil.cache_path = '~/.cache/aws/cached_temporary_credentials'