toil

Subpackages

Submodules

Package Contents

Functions

retry([intervals, infinite_retries, errors, ...])

Retry a function if it fails with any Exception defined in "errors".

which(cmd[, mode, path])

Return the path with conforms to the given mode on the Path.

toilPackageDirPath()

Return the absolute path of the directory that corresponds to the top-level toil package.

inVirtualEnv()

Test if we are inside a virtualenv or Conda virtual environment.

resolveEntryPoint(entryPoint)

Find the path to the given entry point that should work on a worker.

physicalMemory()

Calculate the total amount of physical memory, in bytes.

physicalDisk(directory)

applianceSelf([forceDockerAppliance])

Return the fully qualified name of the Docker image to start Toil appliance containers from.

customDockerInitCmd()

Return the custom command set by the TOIL_CUSTOM_DOCKER_INIT_COMMAND environment variable.

customInitCmd()

Return the custom command set by the TOIL_CUSTOM_INIT_COMMAND environment variable.

lookupEnvVar(name, envName, defaultValue)

Look up environment variables that control Toil and log the result.

checkDockerImageExists(appliance)

Attempt to check a url registryName for the existence of a docker image with a given tag.

parseDockerAppliance(appliance)

Derive parsed registry, image reference, and tag from a docker image string.

checkDockerSchema(appliance)

requestCheckRegularDocker(origAppliance, registryName, ...)

Check if an image exists using the requests library.

requestCheckDockerIo(origAppliance, imageName, tag)

Check docker.io to see if an image exists using the requests library.

logProcessContext(config)

Attributes

memoize

Memoize a function result based on its parameters using this decorator.

currentCommit

log

KNOWN_EXTANT_IMAGES

cache_path

toil.memoize

Memoize a function result based on its parameters using this decorator.

For example, this can be used in place of lazy initialization. If the decorating function is invoked by multiple threads, the decorated function may be called more than once with the same arguments.

toil.retry(intervals=None, infinite_retries=False, errors=None, log_message=None, prepare=None)[source]

Retry a function if it fails with any Exception defined in “errors”.

Does so every x seconds, where x is defined by a list of numbers (ints or floats) in “intervals”. Also accepts ErrorCondition events for more detailed retry attempts.

Parameters

  • intervals (Optional[List]) – A list of times in seconds we keep retrying until returning failure. Defaults to retrying with the following exponential back-off before failing: 1s, 1s, 2s, 4s, 8s, 16s

  • infinite_retries (bool) – If this is True, reset the intervals when they run out. Defaults to: False.

  • errors (Optional[Sequence[Union[ErrorCondition, Type[Exception]]]]) –

    A list of exceptions OR ErrorCondition objects to catch and retry on. ErrorCondition objects describe more detailed error event conditions than a plain error. An ErrorCondition specifies: - Exception (required) - Error codes that must match to be retried (optional; defaults to not checking) - A string that must be in the error message to be retried (optional; defaults to not checking) - A bool that can be set to False to always error on this condition.

    If not specified, this will default to a generic Exception.

  • log_message (Optional[Tuple[Callable, str]]) – Optional tuple of (“log/print function()”, “message string”) that will precede each attempt.

  • prepare (Optional[List[Callable]]) – Optional list of functions to call, with the function’s arguments, between retries, to reset state.

Returns

The result of the wrapped function or raise.

Return type

Callable[[Any], Any]

toil.currentCommit = '6d5a5b83b649cd8adf34a5cfe89e7690c95189d3'
toil.log[source]
toil.which(cmd, mode=os.F_OK | os.X_OK, path=None)[source]

Return the path with conforms to the given mode on the Path.

[Copy-pasted in from python3.6’s shutil.which().]

mode defaults to os.F_OK | os.X_OK. path defaults to the result of os.environ.get(“PATH”), or can be overridden with a custom search path.

Returns

The path found, or None.

Return type

Optional[str]

toil.toilPackageDirPath()[source]

Return the absolute path of the directory that corresponds to the top-level toil package.

The return value is guaranteed to end in ‘/toil’.

Return type

str

toil.inVirtualEnv()[source]

Test if we are inside a virtualenv or Conda virtual environment.

Return type

bool

toil.resolveEntryPoint(entryPoint)[source]

Find the path to the given entry point that should work on a worker.

Returns

The path found, which may be an absolute or a relative path.

Parameters

entryPoint (str) –

Return type

str

toil.physicalMemory()[source]

Calculate the total amount of physical memory, in bytes.

>>> n = physicalMemory()
>>> n > 0
True
>>> n == physicalMemory()
True
Return type

int

toil.physicalDisk(directory)[source]
Parameters

directory (str) –

Return type

int

toil.applianceSelf(forceDockerAppliance=False)[source]

Return the fully qualified name of the Docker image to start Toil appliance containers from.

The result is determined by the current version of Toil and three environment variables: TOIL_DOCKER_REGISTRY, TOIL_DOCKER_NAME and TOIL_APPLIANCE_SELF.

TOIL_DOCKER_REGISTRY specifies an account on a publicly hosted docker registry like Quay or Docker Hub. The default is UCSC’s CGL account on Quay.io where the Toil team publishes the official appliance images. TOIL_DOCKER_NAME specifies the base name of the image. The default of toil will be adequate in most cases. TOIL_APPLIANCE_SELF fully qualifies the appliance image, complete with registry, image name and version tag, overriding both TOIL_DOCKER_NAME and TOIL_DOCKER_REGISTRY` as well as the version tag of the image. Setting TOIL_APPLIANCE_SELF will not be necessary in most cases.

Parameters

forceDockerAppliance (bool) –

Return type

str

toil.customDockerInitCmd()[source]

Return the custom command set by the TOIL_CUSTOM_DOCKER_INIT_COMMAND environment variable.

The custom docker command is run prior to running the workers and/or the primary node’s services.

This can be useful for doing any custom initialization on instances (e.g. authenticating to private docker registries). Any single quotes are escaped and the command cannot contain a set of blacklisted chars (newline or tab).

Returns

The custom commmand, or an empty string is returned if the environment variable is not set.

Return type

str

toil.customInitCmd()[source]

Return the custom command set by the TOIL_CUSTOM_INIT_COMMAND environment variable.

The custom init command is run prior to running Toil appliance itself in workers and/or the primary node (i.e. this is run one stage before TOIL_CUSTOM_DOCKER_INIT_COMMAND).

This can be useful for doing any custom initialization on instances (e.g. authenticating to private docker registries). Any single quotes are escaped and the command cannot contain a set of blacklisted chars (newline or tab).

returns: the custom command or n empty string is returned if the environment variable is not set.

Return type

str

toil.lookupEnvVar(name, envName, defaultValue)[source]

Look up environment variables that control Toil and log the result.

Parameters

  • name (str) – the human readable name of the variable

  • envName (str) – the name of the environment variable to lookup

  • defaultValue (str) – the fall-back value

Returns

the value of the environment variable or the default value the variable is not set

Return type

str

toil.checkDockerImageExists(appliance)[source]

Attempt to check a url registryName for the existence of a docker image with a given tag.

Parameters

appliance (str) – The url of a docker image’s registry (with a tag) of the form: ‘quay.io/<repo_path>:<tag>’ or ‘<repo_path>:<tag>’. Examples: ‘quay.io/ucsc_cgl/toil:latest’, ‘ubuntu:latest’, or ‘broadinstitute/genomes-in-the-cloud:2.0.0’.

Returns

Raises an exception if the docker image cannot be found or is invalid. Otherwise, it will return the appliance string.

Return type

str

toil.parseDockerAppliance(appliance)[source]

Derive parsed registry, image reference, and tag from a docker image string.

Example: “quay.io/ucsc_cgl/toil:latest” Should return: “quay.io”, “ucsc_cgl/toil”, “latest”

If a registry is not defined, the default is: “docker.io” If a tag is not defined, the default is: “latest”

Parameters

appliance (str) – The full url of the docker image originally specified by the user (or the default). e.g. “quay.io/ucsc_cgl/toil:latest”

Returns

registryName, imageName, tag

Return type

Tuple[str, str, str]

toil.checkDockerSchema(appliance)[source]
exception toil.ApplianceImageNotFound(origAppliance, url, statusCode)[source]

Bases: docker.errors.ImageNotFound

digraph inheritance0252dfd8b0 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "APIError" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="An HTTP error from the API."]; "HTTPError" -> "APIError" [arrowsize=0.5,style="setlinewidth(0.5)"]; "DockerException" -> "APIError" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ApplianceImageNotFound" [URL="#toil.ApplianceImageNotFound",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Error raised when using TOIL_APPLIANCE_SELF results in an HTTP error."]; "ImageNotFound" -> "ApplianceImageNotFound" [arrowsize=0.5,style="setlinewidth(0.5)"]; "DockerException" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A base class from which all other exceptions inherit."]; "HTTPError" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="An HTTP error occurred."]; "RequestException" -> "HTTPError" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ImageNotFound" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled"]; "NotFound" -> "ImageNotFound" [arrowsize=0.5,style="setlinewidth(0.5)"]; "NotFound" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled"]; "APIError" -> "NotFound" [arrowsize=0.5,style="setlinewidth(0.5)"]; "RequestException" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="There was an ambiguous exception that occurred while handling your"]; }

Error raised when using TOIL_APPLIANCE_SELF results in an HTTP error.

Parameters

  • origAppliance (str) – The full url of the docker image originally specified by the user (or the default). e.g. “quay.io/ucsc_cgl/toil:latest”

  • url (str) – The URL at which the image’s manifest is supposed to appear

  • statusCode (int) – the failing HTTP status code returned by the URL

toil.KNOWN_EXTANT_IMAGES
toil.requestCheckRegularDocker(origAppliance, registryName, imageName, tag)[source]

Check if an image exists using the requests library.

URL is based on the docker v2 schema.

This has the following format: https://{websitehostname}.io/v2/{repo}/manifests/{tag}

Does not work with the official (docker.io) site, because they require an OAuth token, so a separate check is done for docker.io images.

Parameters

  • origAppliance (str) –

    The full url of the docker image originally specified by the user (or the default).

    e.g. quay.io/ucsc_cgl/toil:latest

  • registryName (str) – The url of a docker image’s registry. e.g. quay.io

  • imageName (str) – The image, including path and excluding the tag. e.g. ucsc_cgl/toil

  • tag (str) – The tag used at that docker image’s registry. e.g. latest

Raises

ApplianceImageNotFound if no match is found.

Returns

Return True if match found.

Return type

bool

toil.requestCheckDockerIo(origAppliance, imageName, tag)[source]

Check docker.io to see if an image exists using the requests library.

URL is based on the docker v2 schema. Requires that an access token be fetched first.

Parameters

  • origAppliance (str) – The full url of the docker image originally specified by the user (or the default). e.g. “ubuntu:latest”

  • imageName (str) – The image, including path and excluding the tag. e.g. “ubuntu”

  • tag (str) – The tag used at that docker image’s registry. e.g. “latest”

Raises

ApplianceImageNotFound if no match is found.

Returns

Return True if match found.

Return type

bool

toil.logProcessContext(config)[source]
Parameters

config (common.Config) –

Return type

None

toil.cache_path = '~/.cache/aws/cached_temporary_credentials'