close
close

WDL 1.2.0: Improvement of the workflow description language for bioinformatics

WDL 1.2.0: Improvement of the workflow description language for bioinformatics

The Workflow Description Language (WDL) team has announced the release of WDL 1.2.0, a major update to improve the flexibility and usability of workflow descriptions in bioinformatics. This new version introduces several key features and improvements that promise to streamline the management and execution of workflows, making it easier for developers and researchers to implement and manage complex bioinformatics workflows.

The Workflow Description Language (WDL) is an open standard specification for describing data processing workflows with a human-readable and writable syntax. WDL makes it easy to define analysis tasks, connect them into workflows, and parallelize their execution. The language aims to be accessible and understandable to all users, including programmers, analysts and operators of production systems.

One of the most important improvements in WDL 1.2.0 is the introduction of the Directory type. This new type allows workflows to process folders more effectively, allowing users to pass folders between tasks, simplifying the management of grouped data files. Consider the following example:


version 1.2

task directory_example {

    input {
        Directory dir
    }

    command <<<
    for file in ~{dir}; do
      cat $file
    done
    >>>

    output {
        File concatenated = stdout()
    }
}

Another notable improvement is the ability to declare inputs as environment variables. This feature allows tasks to directly access environment variables, making it easier to manage configuration settings and sensitive information without hard-coding them into the workflow scripts. Moreover, WDL 1.2.0 introduces a new one requirements And hints section. These sections provide a standardized way to specify the computation requirements and optional tips for executing the workflow. This improvement helps optimize workflow performance and resource allocation, ensuring tasks run efficiently across environments. Consider the following example:


version 1.2

task dynamic_container {
  input {
    String image_tag = "latest"
    String ubuntu_release
  }

  command <<<
    cat /etc/*-release | grep DISTRIB_CODENAME | cut -f 2 -d '=' > ubuntu_release
    nvcc -V > cuda_version
  >>>


  output {
    String is_expected_release = ubuntu_release == read_string("ubuntu_release")
    String cuda_version = read_string("cuda_version")
  }

  requirements {
    container: "nvidia/cuda:~{image_tag}"
    gpu: true
  }

  hints {
    gpu: 2
  }
}

WDL 1.2.0 also brings several new and improved standard library features. The join_paths function is now the preferred way to link paths together, the matches And find functions perform pattern matching on strings, the contains function checks for the existence of a value in an array, and the chunk function splits an array into equal sized pieces. Moreover, the keys feature can now store member names in a Object or Structthe contains_key function checks for the existence of keys in different types, and the select_first function accepts a default value. The size function now processes all compound value inputs, and the length function accepts more types of arguments. finally, the read_tsv The function can read field names from a header row or an array of strings and return an array of objects.

The release of WDL 1.2.0 marks an important milestone in the evolution of workflow management for bioinformatics. The new features and enhancements are designed to address common challenges faced by bioinformatics researchers and developers, including the need for greater flexibility, improved documentation, and better error handling.

Regarding the plans for WDL’s development, Patrick Magee, WDL board member and senior software developer at DNAStack, shared insights into the team’s direction. Magee stated:

We want to establish a more regular release pace, focusing on delivering a smaller set of impactful features and clarifications. This allows us to quickly respond to community feedback and improve the experience for developers and end users.

He also added:

Stay tuned for proposals for long-awaited features such as enumeration, list comprehension, and input validation. In addition, we also want to build on the improvements made in 1.2 and further tighten the portability and reproducibility of the workflow, giving the community the tools needed to increase the use of WDL in all environments.

For more information about WDL 1.2.0 and to access the full documentation, see the official WDL website and GitHub repository.