WDL Tutorial¶
This tutorial will walk you through writing a workflow in WDL that you can run with Toil. We’re going to write a workflow for Fizz Buzz.
Writing the File¶
Let’s go step by step through the process of creating a single-file WDL workflow.
Version¶
All WDL files need to start with a version statement (unless they
are very old draft-2 files). Toil supports draft-2, WDL 1.0, and
WDL 1.1, while Cromwell (another popular WDL runner used on Terra)
supports only draft-2 and 1.0.
So let’s start a new WDL 1.0 workflow. Open up a file named
fizzbuzz.wdl and start with a version statement:
version 1.0
Workflow Block¶
Then, add an empty workflow named FizzBuzz.
version 1.0
workflow FizzBuzz {
}
Input Block¶
Workflows usually need some kind of user input, so let’s give our
workflow an input section.
version 1.0
workflow FizzBuzz {
input {
# How many FizzBuzz numbers do we want to make?
Int item_count
# Every multiple of this number, we produce "Fizz"
Int to_fizz = 3
# Every multiple of this number, we produce "Buzz"
Int to_buzz = 5
# Optional replacement for the string to print when a multiple of both
String? fizzbuzz_override
}
}
Notice that each input has a type, a name, and an optional default
value. If the type ends in ?, the value is optional, and it may be
null. If an input is not optional, and there is no default value,
then the user’s inputs file must specify a value for it in order for
the workflow to run.
Body¶
Now we’ll start on the body of the workflow, to be inserted just after the inputs section.
The first thing we’re going to need to do is create an array of all the
numbers up to the item_count. We can do this by calling the WDL
range() function, and assigning the result to an Array[Int]
variable.
Array[Int] numbers = range(item_count)
WDL 1.0 has a wide variety of functions in its standard library, and WDL 1.1 has even more.
Scattering¶
Once we create an array of all the numbers, we can use a scatter to
operate on each. WDL does not have loops; instead it has scatters, which
work a bit like a map() in Python. The body of the scatter runs for
each value in the input array, all in parallel. We’re going to increment
all the numbers, since FizzBuzz starts at 1 but WDL range() starts
at 0.
Array[Int] numbers = range(item_count)
scatter (i in numbers) {
Int one_based = i + 1
}
Conditionals¶
Inside the body of the scatter, we are going to put some conditionals to
determine if we should produce "Fizz", "Buzz", or
"FizzBuzz". To support our fizzbuzz_override, we use an array of
it and a default value, and use the WDL select_first() function to
find the first non-null value in that array.
Each execution of a scatter is allowed to declare variables, and outside
the scatter those variables are combined into arrays of all the results.
But each variable can be declared only once in the scatter, even with
conditionals. So we’re going to use select_first() at the end and
take advantage of variables from un-executed conditionals being
null.
Note that WDL supports conditional expressions with a then and an
else, but conditional statements only have a body, not an else
branch. If you need an else you will have to check the negated
condition.
So first, let’s handle the special cases.
Array[Int] numbers = range(item_count)
scatter (i in numbers) {
Int one_based = i + 1
if (one_based % to_fizz == 0) {
String fizz = "Fizz"
if (one_based % to_buzz == 0) {
String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
}
}
if (one_based % to_buzz == 0) {
String buzz = "Buzz"
}
if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
# Just a normal number.
}
}
Calling Tasks¶
Now for the normal numbers, we need to convert our number into a string.
In WDL 1.1, and in WDL 1.0 on Cromwell, you can use a ${}
substitution syntax in quoted strings anywhere, not just in command line
commands. Toil technically will support this too, but it’s not in the
spec, and the tutorial needs an excuse for you to call a task. So we’re
going to insert a call to a stringify_number task, to be written
later.
To call a task (or another workflow), we use a call statement and
give it some inputs. Then we can fish the output values out of the task
with . access, only if we don’t make a noise instead.
Array[Int] numbers = range(item_count)
scatter (i in numbers) {
Int one_based = i + 1
if (one_based % to_fizz == 0) {
String fizz = "Fizz"
if (one_based % to_buzz == 0) {
String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
}
}
if (one_based % to_buzz == 0) {
String buzz = "Buzz"
}
if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
# Just a normal number.
call stringify_number {
input:
the_number = one_based
}
}
String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string])
}
We can put the code into the workflow now, and set about writing the task.
version 1.0
workflow FizzBuzz {
input {
# How many FizzBuzz numbers do we want to make?
Int item_count
# Every multiple of this number, we produce "Fizz"
Int to_fizz = 3
# Every multiple of this number, we produce "Buzz"
Int to_buzz = 5
# Optional replacement for the string to print when a multiple of both
String? fizzbuzz_override
}
Array[Int] numbers = range(item_count)
scatter (i in numbers) {
Int one_based = i + 1
if (one_based % to_fizz == 0) {
String fizz = "Fizz"
if (one_based % to_buzz == 0) {
String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
}
}
if (one_based % to_buzz == 0) {
String buzz = "Buzz"
}
if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
# Just a normal number.
call stringify_number {
input:
the_number = one_based
}
}
String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string]
}
}
Writing Tasks¶
Our task should go after the workflow in the file. It looks a lot like a
workflow except it uses task.
task stringify_number {
}
We’re going to want it to take in an integer the_number, and we’re
going to want it to output a string the_string. So let’s fill that
in in input and output sections.
task stringify_number {
input {
Int the_number
}
# ???
output {
String the_string # = ???
}
}
Now, unlike workflows, tasks can have a command section, which gives
a command to run. This section is now usually set off with triple angle
brackets, and inside it you can use ~{}, that is, Bash-like
substitution but with a tilde, to place WDL variables into your command
script. So let’s add a command that will echo back the number so we can
see it as a string.
task stringify_number {
input {
Int the_number
}
command <<<
# This is a Bash script.
# So we should do good Bash script things like stop on errors
set -e
# Now print our number as a string
echo ~{the_number}
>>>
output {
String the_string # = ???
}
}
Now we need to capture the result of the command script. The WDL
stdout() returns a WDL File containing the standard output
printed by the task’s command. We want to read that back into a string,
which we can do with the WDL read_string() function (which also
removes trailing
newlines).
task stringify_number {
input {
Int the_number
}
command <<<
# This is a Bash script.
# So we should do good Bash script things like stop on errors
set -e
# Now print our number as a string
echo ~{the_number}
>>>
output {
String the_string = read_string(stdout())
}
}
We’re also going to want to add a runtime section to our task, to
specify resource requirements. We’re also going to tell it to run in a
Docker container, to make sure that absolutely nothing can go wrong with
our delicate echo command. In a real workflow, you probably want to
set up optiopnal inputs for all the tasks to let you control the
resource requirements, but here we will just hardcode them.
task stringify_number {
input {
Int the_number
}
command <<<
# This is a Bash script.
# So we should do good Bash script things like stop on errors
set -e
# Now print our number as a string
echo ~{the_number}
>>>
output {
String the_string = read_string(stdout())
}
runtime {
cpu: 1
memory: "0.5 GB"
disks: "local-disk 1 SSD"
docker: "ubuntu:24.04"
}
}
The disks section is a little weird; it isn’t in the WDL spec, but
Toil supports Cromwell-style strings that ask for a local-disk of a
certain number of gigabytes, which may suggest that it be SSD
storage.
Then we can put our task into our WDL file:
version 1.0
workflow FizzBuzz {
input {
# How many FizzBuzz numbers do we want to make?
Int item_count
# Every multiple of this number, we produce "Fizz"
Int to_fizz = 3
# Every multiple of this number, we produce "Buzz"
Int to_buzz = 5
# Optional replacement for the string to print when a multiple of both
String? fizzbuzz_override
}
Array[Int] numbers = range(item_count)
scatter (i in numbers) {
Int one_based = i + 1
if (one_based % to_fizz == 0) {
String fizz = "Fizz"
if (one_based % to_buzz == 0) {
String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
}
}
if (one_based % to_buzz == 0) {
String buzz = "Buzz"
}
if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
# Just a normal number.
call stringify_number {
input:
the_number = one_based
}
}
String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string]
}
}
task stringify_number {
input {
Int the_number
}
command <<<
# This is a Bash script.
# So we should do good Bash script things like stop on errors
set -e
# Now print our number as a string
echo ~{the_number}
>>>
output {
String the_string = read_string(stdout())
}
runtime {
cpu: 1
memory: "0.5 GB"
disks: "local-disk 1 SSD"
docker: "ubuntu:24.04"
}
}
Output Block¶
Now the only thing missing is a workflow-level output section.
Technically, in WDL 1.0 you aren’t supposed to need this, but you do
need it in 1.1 and Toil doesn’t actually send your outputs anywhere yet
if you don’t have one, so we’re going to make one. We need to collect
together all the strings that came out of the different tasks in our
scatter into an Array[String]. We’ll add the output section at
the end of the workflow section, above the task.
version 1.0
workflow FizzBuzz {
input {
# How many FizzBuzz numbers do we want to make?
Int item_count
# Every multiple of this number, we produce "Fizz"
Int to_fizz = 3
# Every multiple of this number, we produce "Buzz"
Int to_buzz = 5
# Optional replacement for the string to print when a multiple of both
String? fizzbuzz_override
}
Array[Int] numbers = range(item_count)
scatter (i in numbers) {
Int one_based = i + 1
if (one_based % to_fizz == 0) {
String fizz = "Fizz"
if (one_based % to_buzz == 0) {
String fizzbuzz = select_first([fizzbuzz_override, "FizzBuzz"])
}
}
if (one_based % to_buzz == 0) {
String buzz = "Buzz"
}
if (one_based % to_fizz != 0 && one_based % to_buzz != 0) {
# Just a normal number.
call stringify_number {
input:
the_number = one_based
}
}
String result = select_first([fizzbuzz, fizz, buzz, stringify_number.the_string]
}
output {
Array[String] fizzbuzz_results = result
}
}
task stringify_number {
input {
Int the_number
}
command <<<
# This is a Bash script.
# So we should do good Bash script things like stop on errors
set -e
# Now print our number as a string
echo ~{the_number}
>>>
output {
String the_string = read_string(stdout())
}
runtime {
cpu: 1
memory: "0.5 GB"
disks: "local-disk 1 SSD"
docker: "ubuntu:24.04"
}
}
Because the result variable is defined inside a scatter, when we
reference it outside the scatter we see it as being an array.
Running the Workflow¶
Now all that remains is to run the workflow! Make an inputs file to specify the workflow inputs:
echo '{"FizzBuzz.item_count": 20}' >fizzbuzz.json
Then run it with Toil. If you are on a Slurm cluster, and you are currently in a shared directory available on all your nodes, you can run:
toil-wdl-runner --jobStore ./fizzbuzz_store --batchSystem slurm --slurmTime 00:10:00 --caching false --batchLogsDir ./logs fizzbuzz.wdl fizzbuzz.json -o fizzbuzz_out -m fizzbuzz_out.json
If instead you want to run your workflow locally, you can run:
toil-wdl-runner fizzbuzz.wdl fizzbuzz.json -o fizzbuzz_out -m fizzbuzz_out.json
Next Steps¶
Try breaking your workflow up into multiple files and using
importstatements.Publish your workflow on Dockstore.
Read up on Toil-specific WDL development considerations in Developing WDL Workflows.
