Toil Debugging

Toil has a number of tools to assist in debugging. Here we provide help in working through potential problems that a user might encounter in attempting to run a workflow.

Reading the Log

Usually, at the end of a failed Toil worklfow, Toil will reproduce the job logs for the jobs that failed. You can look at the end of your workflow log and use the job logs to identify which jobs are failing and why.

Finding Failed Jobs in the Jobstore

The toil status command (Status Command) can be used with the --failed option to list all failed jobs in a Toil job store.

You can also use it with the --logs option to retrieve per-job logs from the job store, for failed jobs that left them. These logs might be useful for diagnosing and fixing the problem.

Running a Job Locally

If you have a failing job’s ID or name, you can reproduce its failure on your local machine with toil debug-job. See Debug Job Command.

For example, say you have this WDL workflow in test.wdl. This workflow cannot succeed, due to the typo in the echo command:

version 1.0
workflow test {
  call hello
}
task hello {
  input {
  }
  command <<<
    set -e
    echoo "Hello"
  >>>
  output {
  }
}

You could try to run it with:

toil-wdl-runner --jobStore ./store test.wdl

But it will fail.

If you want to reproduce the failure later, or on another machine, you can first find out what jobs failed with toil status:

toil status --failed --noAggStats ./store

This will produce something like:

[2024-03-14T17:45:15-0400] [MainThread] [I] [toil.utils.toilStatus] Traversing the job graph gathering jobs. This may take a couple of minutes. Failed jobs: ‘WDLTaskJob’ test.hello.command kind-WDLTaskJob/instance-r9u6_dcs v6

And we can see a failed job with the display name test.hello.command, which describes the job’s location in the WDL workflow as the command section of the hello task called from the test workflow. (If you are writing a Toil Python script, this is the job’s displayName.) We can then run that job again locally by name with:

toil debug-job ./store test.hello.command

If there were multiple failed jobs with that name (perhaps because of a WDL scatter), we would need to select one by Toil job ID instead:

toil debug-job ./store kind-WDLTaskJob/instance-r9u6_dcs

And if we know there’s only one failed WDL task, we can just tell Toil to rerun the failed WDLTaskJob by Python class name:

toil debug-job ./store WDLTaskJob

Any of these will run the job (including any containers) on the local machine, where its execution can be observed live or monitored with a debugger.

Introspecting the Job Store

Note: Currently these features are only implemented for use locally (single machine) with the fileJobStore.

To view what files currently reside in the jobstore, run the following command:

$ toil debug-file file:path-to-jobstore-directory \
      --listFilesInJobStore

When run from the commandline, this should generate a file containing the contents of the job store (in addition to displaying a series of log messages to the terminal). This file is named “jobstore_files.txt” by default and will be generated in the current working directory.

If one wishes to copy any of these files to a local directory, one can run for example:

$ toil debug-file file:path-to-jobstore \
      --fetch overview.txt *.bam *.fastq \
      --localFilePath=/home/user/localpath

To fetch overview.txt, and all .bam and .fastq files. This can be used to recover previously used input and output files for debugging or reuse in other workflows, or use in general debugging to ensure that certain outputs were imported into the jobStore.

Stats and Status

See Stats Command and Status Command for more about gathering statistics about job success, runtime, and resource usage from workflows.

Using a Python debugger

If you execute a workflow using the --debugWorker flag, or if you use toil debug-job, Toil will run the job in the process you started from the command line. This means you can either use pdb, or an IDE that supports debugging Python to interact with the Python process as it runs your job. Note that the --debugWorker flag will only work with the single_machine batch system (the default), and not any of the custom job schedulers.