toil.lib.dockstore

Contains functions for integrating Toil with UCSC Dockstore, for reporting metrics.

For basic TRS functionality for fetching workflows, see trs.py.

Attributes

logger

DEFAULT_DOCKSTORE_TOKEN

DOCKSTORE_TOKEN

DOCKSTORE_PLATFORM

ExecutionStatus

Classes

Cost

Representation of the cost of running something.

RunExecution

Dockstore metrics data for a workflow or task run.

TaskExecutions

Dockstore metrics data for all the tasks in a workflow.

Functions

ensure_valid_id(execution_id)

Make sure the given execution ID is in Dockstore format and will be accepted by Dockstore.

pack_workflow_metrics(execution_id, start_time, ...[, ...])

Pack up per-workflow metrics into a format that can be submitted to Dockstore.

pack_single_task_metrics(execution_id, start_time, ...)

Pack up metrics for a single task execution in a format that can be used in a Dockstore submission.

pack_workflow_task_set_metrics(execution_id, ...)

Pack up metrics for all the tasks in a workflow execution into a format that can be submitted to Dockstore.

send_metrics(trs_workflow_id, trs_version, ...)

Send packed workflow and/or task metrics to Dockstore.

get_metrics_url(trs_workflow_id, trs_version, execution_id)

Get the URL where a workflow metrics object (for a workflow, or for a set of tasks) can be fetched back from.

Module Contents

toil.lib.dockstore.logger
toil.lib.dockstore.DEFAULT_DOCKSTORE_TOKEN = '2bff46294daddef6df185452b04db6143ea8a59f52ee3c325d3e1df418511b7d'
toil.lib.dockstore.DOCKSTORE_TOKEN
toil.lib.dockstore.DOCKSTORE_PLATFORM = 'TOIL'
toil.lib.dockstore.ExecutionStatus
class toil.lib.dockstore.Cost[source]

Bases: TypedDict

Representation of the cost of running something.

value: float

Cost in US Dollars.

class toil.lib.dockstore.RunExecution[source]

Bases: TypedDict

Dockstore metrics data for a workflow or task run.

executionId: str

Executor-generated unique execution ID for this workflow or task.

dateExecuted: str

ISO 8601 UTC timestamp when the execution happend.

executionStatus: ExecutionStatus

Did the execution work?

executionTime: typing_extensions.NotRequired[str]

Total time of the run in ISO 8601 duration format.

memoryRequirementsGB: typing_extensions.NotRequired[float]

Memory required for the execution in gigabytes (not GiB).

cpuRequirements: typing_extensions.NotRequired[int]

Number of CPUs required.

cost: typing_extensions.NotRequired[Cost]

How much the execution cost to run.

region: typing_extensions.NotRequired[str]

The (cloud) region the workflow was executed in.

additionalProperties: typing_extensions.NotRequired[dict[str, Any]]

Any additional properties to send.

Dockstore can take any JSON-able structured data.

class toil.lib.dockstore.TaskExecutions[source]

Bases: TypedDict

Dockstore metrics data for all the tasks in a workflow.

executionId: str

Executor-generated unique execution ID.

dateExecuted: str

ISO 8601 UTC timestamp when the execution happend.

taskExecutions: list[RunExecution]

Individual executions of each task in the workflow.

additionalProperties: typing_extensions.NotRequired[dict[str, Any]]

Any additional properties to send.

Dockstore can take any JSON-able structured data.

toil.lib.dockstore.ensure_valid_id(execution_id)[source]

Make sure the given execution ID is in Dockstore format and will be accepted by Dockstore.

Must be alphanumeric (with internal underscores allowed) and <100 characters long.

Raises:

ValueError – if the ID is not in the right format

Parameters:

execution_id (str)

Return type:

None

toil.lib.dockstore.pack_workflow_metrics(execution_id, start_time, runtime, succeeded, job_store_type=None, batch_system=None, caching=None, toil_version=None, python_version=None, platform_system=None, platform_machine=None)[source]

Pack up per-workflow metrics into a format that can be submitted to Dockstore.

Parameters:

  • execution_id (str) – Unique ID for the workflow execution. Must be in Dockstore format.

  • start_time (float) – Execution start time in seconds since the Unix epoch.

  • rutime – Execution duration in seconds.

  • jobstore_type – Kind of job store used, like “file” or “aws”.

  • batch_system (Optional[str]) – Python class name implementing the batch system used.

  • caching (Optional[bool]) – Whether Toil filestore-level cahcing was used.

  • toil_version (Optional[str]) – Version of Toil used (without any Git hash).

  • python_version (Optional[str]) – Version of Python used.

  • platform_system (Optional[str]) – Operating system type (like “Darwin” or “Linux”).

  • platform_machine (Optional[str]) – Machine architecture of the leader (like “AMD64”).

  • runtime (float)

  • succeeded (bool)

  • job_store_type (Optional[str])

Return type:

RunExecution

toil.lib.dockstore.pack_single_task_metrics(execution_id, start_time, runtime, succeeded, job_name=None, cores=None, cpu_seconds=None, memory_bytes=None, disk_bytes=None)[source]

Pack up metrics for a single task execution in a format that can be used in a Dockstore submission.

Parameters:

  • execution_id (str) – Unique ID for the workflow execution. Must be in Dockstore format.

  • start_time (float) – Execution start time in seconds since the Unix epoch.

  • rutime – Execution duration in seconds.

  • succeeded (bool) – Whether the execution succeeded.

  • job_name (Optional[str]) – Name of the job within the workflow.

  • cores (Optional[float]) – CPU cores allocated to the job.

  • cpu_seconds (Optional[float]) – CPU seconds consumed by the job.

  • memory_bytes (Optional[int]) – Memory consumed by the job in bytes.

  • disk_bytes (Optional[int]) – Disk space consumed by the job in bytes.

  • runtime (float)

Return type:

RunExecution

toil.lib.dockstore.pack_workflow_task_set_metrics(execution_id, start_time, tasks)[source]

Pack up metrics for all the tasks in a workflow execution into a format that can be submitted to Dockstore.

Parameters:

  • execution_id (str) – Unique ID for the workflow execution. Must be in Dockstore format.

  • start_time (float) – Execution start time for the overall workflow execution in seconds since the Unix epoch.

  • tasks (list[RunExecution]) – Packed tasks from pack_single_task_metrics()

Return type:

TaskExecutions

toil.lib.dockstore.send_metrics(trs_workflow_id, trs_version, workflow_runs, workflow_task_sets)[source]

Send packed workflow and/or task metrics to Dockstore.

Parameters:

  • workflow_runs (list[RunExecution]) – list of packed metrics objects for each workflow.

  • workflow_task_sets (list[TaskExecutions]) – list of packed metrics objects for the tasks in each workflow. Each workflow should have one entry containing all its tasks. Does not have to be the same order/set of workflows as workflow_runs.

  • trs_workflow_id (str)

  • trs_version (str)

Raises:

requests.HTTPError – if Dockstore does not accept the metrics.

Return type:

None

toil.lib.dockstore.get_metrics_url(trs_workflow_id, trs_version, execution_id)[source]

Get the URL where a workflow metrics object (for a workflow, or for a set of tasks) can be fetched back from.

Parameters:

  • trs_workflow_id (str)

  • trs_version (str)

  • execution_id (str)

Return type:

str