Pydra

Citation and Resources:¶

Tools included in this workflow¶

Pydra:

• Jarecka, D., Goncalves, M., Markiewicz, C. J., Esteban, O., Lo, N., Kaczmarzyk, J., Cali, R., Herholz, P., Nielson, D. M., Mentch, J., Nijholt, B., Johnson, C. E., Wigger, J., Close, T. G., Vaillant, G., Agarwal, A., & Ghosh, S. (2025). nipype/pydra: 1.0a2 (1.0a2). Zenodo. Jarecka et al. (2025)

• nipype/pydra

ANTsPy:

• stnava, Philip Cook, Nick Tustison, Ravnoor Singh Gill, John Muschelli, Jennings Zhang, Daniel Gomez, Andrew Berger, Thiago Franco de Moraes, Stephen Ogier, Asaph Zylbertal, Dean Rance, Pradeep Reddy Raamana, Vasco Diogo, sai8951, Bryn Lloyd, Dženan Zukić, Evert de Man, KYY, … Tommaso Di Noto. (2025). ANTsX/ANTsPy: Aiouea (v0.6.1). Zenodo. stnava et al. (2025)

• Tustison, N.J., Cook, P.A., Holbrook, A.J. et al. The ANTsX ecosystem for quantitative biological and medical imaging. Sci Rep 11, 9068 (2021). Tustison et al. (2021)

• Documentation ANTsPy

• ANTsX/ANTsPy

SynthStrip:

• SynthStrip: Skull-Stripping for Any Brain Image; Andrew Hoopes, Jocelyn S. Mora, Adrian V. Dalca, Bruce Fischl*, Malte Hoffmann* (*equal contribution); NeuroImage 260, 2022, 119474; Hoopes et al. (2022)

• Boosting skull-stripping performance for pediatric brain images; William Kelley, Nathan Ngo, Adrian V. Dalca, Bruce Fischl, Lilla Zöllei*, Malte Hoffmann* (*equal contribution); IEEE International Symposium on Biomedical Imaging (ISBI), 2024, forthcoming; https://arxiv.org/abs/2402.16634

• Documentation

Dataset¶

Opensource Data from OpenNeuro:

• Kelly AMC and Uddin LQ and Biswal BB and Castellanos FX and Milham MP (2018). Flanker task (event-related). OpenNeuro Dataset ds000102

• Kelly, A.M., Uddin, L.Q., Biswal, B.B., Castellanos, F.X., Milham, M.P. (2008). Competition between functional brain networks mediates behavioral variability. Neuroimage, 39(1):527-37

Installation and Import¶

Introduction Pydra¶

Pydra is a lightweight, Python-based dataflow engine designed to build reproducible, scalable, and robust workflows. While it was created to succeed Nipype for neuroimaging applications, Pydra is flexible enough to support analytic pipelines in any scientific domain. It allows you to combine tasks implemented as Python functions or shell commands into coherent workflows that can be executed reliably across different computing environments.

Key features include:

• Combining diverse tasks into robust workflows.

• Dynamic workflow construction using standard Python code.

• Concurrent execution on local workstations or HPC clusters (SLURM, SGE, Dask, etc.).

• Map-reduce-style parallelism via split() and combine().

• Global caching to avoid unnecessary recomputation.

• Support for executing tasks in separate software environments (e.g., containers).

• Strong type-checking of inputs and outputs, including specialized file formats.

Overview of the Preprocessing Workflow¶

In this notebook, we demonstrate a neuroimaging registration workflow built with Pydra. The workflow uses:

• Python tasks unsing ANTsPy for operations implemented directly in Python.

• Shell task to wrap FreeSurfer’s command-line tool SynthStrip for skull-stripping.

• A workflow that orchestrates multiple tasks, connecting inputs and outputs.

• Splitting to process multiple functional runs in parallel.

• A Submitter object to to initiate the task execution for a richer Result object

• Concurrent execution using the cf (ConcurrentFutures) worker.

1. Download of Data¶

One Subject from the Flanker Dataset from OpenNeuro¶

MNI template from Nilearn¶

2. Pydra Tasks¶

The basic runnable component of Pydra is a task. Tasks are conceptually similar to functions, in that they take inputs, operate on them and then return results. However, unlike standard functions, tasks are parameterized before execution in a separate step. This separation enables parameterized tasks to be linked together into workflows that can be validated for errors before execution, while allowing modular execution workers and environments to be specified independently of the task being performed.

Tasks can encapsulate Python functions or shell-commands, or be multi-component workflows themselves, constructed from task components including nested workflows.

2.1. Python Tasks¶

Python tasks are Python functions that are parameterized before they are executed or added to a workflow. This approach provides several advantages over direct function calls, including input validation, output tracking, and seamless integration into larger computational workflows.

Define decorator¶

The simplest way to define a Python task is to decorate a function with pydra.compose.python.define.

This neuroimaging registration pipeline demonstrates this approach by creating five specialized Python tasks with ANTsPy that handle different aspects of fMRI data processing:

1. Motion correction task - Corrects for head movement by registering all volumes to a reference timepoint

2. Structural-to-MNI registration task - Registers the T1-weighted anatomical image to MNI standard space

3. Functional-to-structural registration task - Registers the reference functional volume to the skull-stripped T1 image

4. Transform combination task - Concatenates the transformation matrices in the proper order for ANTs

5. Transform application task - Applies the combined transformations to register the motion-corrected functional data to MNI space

2.2 Shell-tasks¶

In Pydra, shell tasks can be defined using command-line style templates, which closely resemble the syntax shown in a tool’s inline help. This approach provides a concise and intuitive way to map a CLI program into a Python task that can be used in workflows.

Key features of the shell tasks:

• Input and Outputs:

  • Fields are enclosed in < >.

  • Outputs are marked with the out| prefix (e.g., <out|out_file:File>).

  • By default, fields are treated as fileformats.generic.FsObject, but more specific types (e.g. NiftiGz, float, bool) can be specified with :<type>

• Flags and Options:

  • Flags are associated with fields by placing the field after the flag (e.g., -i <image:File>).

  • Boolean flags are included only if the value is True, and omitted otherwise (e.g., -g<gpu?:bool> " --> optional, and the default is False)

  • Options that take a value have the field inserted after the flag (e.g., --model <model:File>)

• Optional Arguments:

  • Adding ? after the type marks a field as optional (e.g., <border:float?>).

  • Optional fields are omitted from the command if not set, allowing the underlying tool to use its own defaults.

• Defaults

  • A default value can be provided after = (e.g., <border:float?=1>, which will override the CLI default if desired)

  • Defaults allow tasks to run without explicitly setting every argument.

• Path Templates for Outputs:

  • By default, output fields are assigned a path_template derived from the field name and extension (e.g., out_file.gz).

  • You can override this default filename using $ followed by a filename (e.g., <out|mask_file:File$brain_mask.nii.gz>).

  • The auto-generated filename will be used unless the user provides an explicit path when initializing the task.

We will create SynthStrip shell task by wrapping FreeSurfer’s mri_synthstrip skull-stripping tool. Let’s first look at the help page:

¶

The SynthStrip tool for robust skull-stripping can be expressed as shell task in just a few lines using this syntax:

3.3. Workflow¶

In Pydra, workflows represent directed acyclic graphs (DAGs) of tasks, where individual tasks are connected through their inputs and outputs to form complex computational pipelines. Like tasks, workflows are defined as dataclasses, but instead of performing computations themselves, they orchestrate the execution of multiple interconnected components.

Workflows are defined using the @workflow.define decorator applied to a constructor function, which specifies how tasks should be connected and executed. Within the constructor function, individual tasks are added to the workflow using workflow.add(). This returns an outputs object, whose fields serve as placeholders for the task’s eventual results. These placeholders can then be used as inputs to downstream tasks, establishing the dependencies that determine the workflow’s execution order. The workflow itself returns its specified outputs as a tuple once all tasks have completed.

In our preprocessing pipeline, we’ll demonstrate how to combine our Python tasks (motion correction, registration, and transformation) with our shell task (skull stripping) into a cohesive workflow that processes functional MRI data from raw acquisition to standardized space registration.

Pydra workflows support splitting inputs across multiple tasks using the split() method. This is particularly useful in neuroimaging pipelines, where the same preprocessing steps need to be applied to multiple subjects, sessions, or runs. For example, a workflow can be executed over every NIfTI file in a directory by splitting the workflow’s input over the set of files. If the outputs are not combined with combine(), the splits will automatically propagate to downstream nodes.

3. Submitter¶

If you want to access a richer Result object you can use a Submitter object to initiate the task execution.

The Result object contains:

• output: the outputs of the task (if there is only one output it is called out by default)

• runtime: information about the peak memory and CPU usage

• errored: the error status of the task

• task: the task object that generated the results

• cache_dir: the output directory the results are stored in

4. Executing tasks in parallel¶

Workers¶

Pydra allows tasks to be executed in parallel using different workers. The default is the debug worker, which runs tasks serially in a single process; useful for debugging but not optimal for production.

For local parallel execution, the cf (ConcurrentFutures) worker can spread tasks across multiple CPU cores for better efficiency. On HPC clusters, SLURM, SGE, and PSI/J workers can submit workflow nodes as separate jobs to the scheduler. There is also an experimental Dask worker with multiple backend options.

Workers can be specified by string or by class, and additional parameters (e.g., n_procs=2) can be passed at execution. When running multi-process code in a Python script, the workflow execution should be enclosed in an if name == “main”: block to prevent the worker processes from re-executing the top-level script. This guard is not required when running inside a Jupyter Notebook.

5. Visualizing Results¶

Workflow overview: tasks, inputs, and outputs¶

The figure below shows the full workflow execution graph, including all tasks, their inputs, and outputs. This provides a visual overview of the data flow and dependencies across the preprocessing steps.

Functional registration to MNI space¶

The mean BOLD image after motion correction was registered to the MNI template. The overlays below allows for visual inspection of alignment quality.

Let’s use add_contours as another visualization option for checking coregistration by overlaying the MNI template as contour (red) on top of mean functional image (background):

Dependencies in Jupyter/Python¶

• Using the package watermark to document system environment and software versions used in this notebook, alongside the Neurodesktop version extracted from the JUPYTER_IMAGE or NEURODESKTOP_VERSION environment variables.