toil

Subpackages

Submodules

Package Contents

Functions

retry([intervals, infinite_retries, errors, ...])

Retry a function if it fails with any Exception defined in "errors".

which(cmd[, mode, path])

Return the path with conforms to the given mode on the Path.

toilPackageDirPath()

Return the absolute path of the directory that corresponds to the top-level toil package.

inVirtualEnv()

Test if we are inside a virtualenv or Conda virtual environment.

resolveEntryPoint(entryPoint)

Find the path to the given entry point that should work on a worker.

physicalMemory()

Calculate the total amount of physical memory, in bytes.

physicalDisk(directory)

applianceSelf([forceDockerAppliance])

Return the fully qualified name of the Docker image to start Toil appliance containers from.

customDockerInitCmd()

Return the custom command set by the TOIL_CUSTOM_DOCKER_INIT_COMMAND environment variable.

customInitCmd()

Return the custom command set by the TOIL_CUSTOM_INIT_COMMAND environment variable.

lookupEnvVar(name, envName, defaultValue)

Look up environment variables that control Toil and log the result.

checkDockerImageExists(appliance)

Attempt to check a url registryName for the existence of a docker image with a given tag.

parseDockerAppliance(appliance)

Derive parsed registry, image reference, and tag from a docker image string.

checkDockerSchema(appliance)

requestCheckRegularDocker(origAppliance, registryName, ...)

Check if an image exists using the requests library.

requestCheckDockerIo(origAppliance, imageName, tag)

Check docker.io to see if an image exists using the requests library.

logProcessContext(config)

Attributes

memoize

Memoize a function result based on its parameters using this decorator.

currentCommit

log

KNOWN_EXTANT_IMAGES

cache_path

toil.memoize

Memoize a function result based on its parameters using this decorator.

For example, this can be used in place of lazy initialization. If the decorating function is invoked by multiple threads, the decorated function may be called more than once with the same arguments.

toil.retry(intervals=None, infinite_retries=False, errors=None, log_message=None, prepare=None)[source]

Retry a function if it fails with any Exception defined in “errors”.

Does so every x seconds, where x is defined by a list of numbers (ints or floats) in “intervals”. Also accepts ErrorCondition events for more detailed retry attempts.

Parameters:

  • intervals (Optional[List]) – A list of times in seconds we keep retrying until returning failure. Defaults to retrying with the following exponential back-off before failing: 1s, 1s, 2s, 4s, 8s, 16s

  • infinite_retries (bool) – If this is True, reset the intervals when they run out. Defaults to: False.

  • errors (Optional[Sequence[Union[ErrorCondition, Type[Exception]]]]) –

    A list of exceptions OR ErrorCondition objects to catch and retry on. ErrorCondition objects describe more detailed error event conditions than a plain error. An ErrorCondition specifies: - Exception (required) - Error codes that must match to be retried (optional; defaults to not checking) - A string that must be in the error message to be retried (optional; defaults to not checking) - A bool that can be set to False to always error on this condition.

    If not specified, this will default to a generic Exception.

  • log_message (Optional[Tuple[Callable, str]]) – Optional tuple of (“log/print function()”, “message string”) that will precede each attempt.

  • prepare (Optional[List[Callable]]) – Optional list of functions to call, with the function’s arguments, between retries, to reset state.

Returns:

The result of the wrapped function or raise.

Return type:

Callable[[Any], Any]

toil.currentCommit = '36deda22da83c955251067826b7beba7da97f379'
toil.log[source]
toil.which(cmd, mode=os.F_OK | os.X_OK, path=None)[source]

Return the path with conforms to the given mode on the Path.

[Copy-pasted in from python3.6’s shutil.which().]

mode defaults to os.F_OK | os.X_OK. path defaults to the result of os.environ.get(“PATH”), or can be overridden with a custom search path.

Returns:

The path found, or None.

Return type:

Optional[str]

toil.toilPackageDirPath()[source]

Return the absolute path of the directory that corresponds to the top-level toil package.

The return value is guaranteed to end in ‘/toil’.

Return type:

str

toil.inVirtualEnv()[source]

Test if we are inside a virtualenv or Conda virtual environment.

Return type:

bool

toil.resolveEntryPoint(entryPoint)[source]

Find the path to the given entry point that should work on a worker.

Returns:

The path found, which may be an absolute or a relative path.

Parameters:

entryPoint (str) –

Return type:

str

toil.physicalMemory()[source]

Calculate the total amount of physical memory, in bytes.

>>> n = physicalMemory()
>>> n > 0
True
>>> n == physicalMemory()
True
Return type:

int

toil.physicalDisk(directory)[source]
Parameters:

directory (str) –

Return type:

int

toil.applianceSelf(forceDockerAppliance=False)[source]

Return the fully qualified name of the Docker image to start Toil appliance containers from.

The result is determined by the current version of Toil and three environment variables: TOIL_DOCKER_REGISTRY, TOIL_DOCKER_NAME and TOIL_APPLIANCE_SELF.

TOIL_DOCKER_REGISTRY specifies an account on a publicly hosted docker registry like Quay or Docker Hub. The default is UCSC’s CGL account on Quay.io where the Toil team publishes the official appliance images. TOIL_DOCKER_NAME specifies the base name of the image. The default of toil will be adequate in most cases. TOIL_APPLIANCE_SELF fully qualifies the appliance image, complete with registry, image name and version tag, overriding both TOIL_DOCKER_NAME and TOIL_DOCKER_REGISTRY` as well as the version tag of the image. Setting TOIL_APPLIANCE_SELF will not be necessary in most cases.

Parameters:

forceDockerAppliance (bool) –

Return type:

str

toil.customDockerInitCmd()[source]

Return the custom command set by the TOIL_CUSTOM_DOCKER_INIT_COMMAND environment variable.

The custom docker command is run prior to running the workers and/or the primary node’s services.

This can be useful for doing any custom initialization on instances (e.g. authenticating to private docker registries). Any single quotes are escaped and the command cannot contain a set of blacklisted chars (newline or tab).

Returns:

The custom commmand, or an empty string is returned if the environment variable is not set.

Return type:

str

toil.customInitCmd()[source]

Return the custom command set by the TOIL_CUSTOM_INIT_COMMAND environment variable.

The custom init command is run prior to running Toil appliance itself in workers and/or the primary node (i.e. this is run one stage before TOIL_CUSTOM_DOCKER_INIT_COMMAND).

This can be useful for doing any custom initialization on instances (e.g. authenticating to private docker registries). Any single quotes are escaped and the command cannot contain a set of blacklisted chars (newline or tab).

returns: the custom command or n empty string is returned if the environment variable is not set.

Return type:

str

toil.lookupEnvVar(name, envName, defaultValue)[source]

Look up environment variables that control Toil and log the result.

Parameters:

  • name (str) – the human readable name of the variable

  • envName (str) – the name of the environment variable to lookup

  • defaultValue (str) – the fall-back value

Returns:

the value of the environment variable or the default value the variable is not set

Return type:

str

toil.checkDockerImageExists(appliance)[source]

Attempt to check a url registryName for the existence of a docker image with a given tag.

Parameters:

appliance (str) – The url of a docker image’s registry (with a tag) of the form: ‘quay.io/<repo_path>:<tag>’ or ‘<repo_path>:<tag>’. Examples: ‘quay.io/ucsc_cgl/toil:latest’, ‘ubuntu:latest’, or ‘broadinstitute/genomes-in-the-cloud:2.0.0’.

Returns:

Raises an exception if the docker image cannot be found or is invalid. Otherwise, it will return the appliance string.

Return type:

str

toil.parseDockerAppliance(appliance)[source]

Derive parsed registry, image reference, and tag from a docker image string.

Example: “quay.io/ucsc_cgl/toil:latest” Should return: “quay.io”, “ucsc_cgl/toil”, “latest”

If a registry is not defined, the default is: “docker.io” If a tag is not defined, the default is: “latest”

Parameters:

appliance (str) – The full url of the docker image originally specified by the user (or the default). e.g. “quay.io/ucsc_cgl/toil:latest”

Returns:

registryName, imageName, tag

Return type:

Tuple[str, str, str]

toil.checkDockerSchema(appliance)[source]
exception toil.ApplianceImageNotFound(origAppliance, url, statusCode)[source]

Bases: docker.errors.ImageNotFound

digraph inheritance0252dfd8b0 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "APIError" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="An HTTP error from the API."]; "HTTPError" -> "APIError" [arrowsize=0.5,style="setlinewidth(0.5)"]; "DockerException" -> "APIError" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ApplianceImageNotFound" [URL="#toil.ApplianceImageNotFound",fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",target="_top",tooltip="Error raised when using TOIL_APPLIANCE_SELF results in an HTTP error."]; "ImageNotFound" -> "ApplianceImageNotFound" [arrowsize=0.5,style="setlinewidth(0.5)"]; "DockerException" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="A base class from which all other exceptions inherit."]; "HTTPError" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="An HTTP error occurred."]; "RequestException" -> "HTTPError" [arrowsize=0.5,style="setlinewidth(0.5)"]; "ImageNotFound" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled"]; "NotFound" -> "ImageNotFound" [arrowsize=0.5,style="setlinewidth(0.5)"]; "NotFound" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled"]; "APIError" -> "NotFound" [arrowsize=0.5,style="setlinewidth(0.5)"]; "RequestException" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled",tooltip="There was an ambiguous exception that occurred while handling your"]; }

Error raised when using TOIL_APPLIANCE_SELF results in an HTTP error.

Parameters:

  • origAppliance (str) – The full url of the docker image originally specified by the user (or the default). e.g. “quay.io/ucsc_cgl/toil:latest”

  • url (str) – The URL at which the image’s manifest is supposed to appear

  • statusCode (int) – the failing HTTP status code returned by the URL

toil.KNOWN_EXTANT_IMAGES
toil.requestCheckRegularDocker(origAppliance, registryName, imageName, tag)[source]

Check if an image exists using the requests library.

URL is based on the docker v2 schema.

This has the following format: https://{websitehostname}.io/v2/{repo}/manifests/{tag}

Does not work with the official (docker.io) site, because they require an OAuth token, so a separate check is done for docker.io images.

Parameters:

  • origAppliance (str) –

    The full url of the docker image originally specified by the user (or the default).

    e.g. quay.io/ucsc_cgl/toil:latest

  • registryName (str) – The url of a docker image’s registry. e.g. quay.io

  • imageName (str) – The image, including path and excluding the tag. e.g. ucsc_cgl/toil

  • tag (str) – The tag used at that docker image’s registry. e.g. latest

Raises:

ApplianceImageNotFound if no match is found.

Returns:

Return True if match found.

Return type:

bool

toil.requestCheckDockerIo(origAppliance, imageName, tag)[source]

Check docker.io to see if an image exists using the requests library.

URL is based on the docker v2 schema. Requires that an access token be fetched first.

Parameters:

  • origAppliance (str) – The full url of the docker image originally specified by the user (or the default). e.g. “ubuntu:latest”

  • imageName (str) – The image, including path and excluding the tag. e.g. “ubuntu”

  • tag (str) – The tag used at that docker image’s registry. e.g. “latest”

Raises:

ApplianceImageNotFound if no match is found.

Returns:

Return True if match found.

Return type:

bool

toil.logProcessContext(config)[source]
Parameters:

config (common.Config) –

Return type:

None

toil.cache_path = '~/.cache/aws/cached_temporary_credentials'