Skip to content

Part 2: Manage compute resources and failures

In Part 1, you adapted where and how a pipeline's tasks run. Here you'll adapt how much compute each task gets, and what happens when a task fails despite your best guess at an allocation.

1. Control compute resource allocations

By default, Nextflow allocates a single CPU to each process via the cpus directive, and does not impose a memory limit unless you set one:

Built-in configuration
process {
    cpus = 1
}

You already know from Nextflow Run that this pipeline's configuration sets memory to 1 GB for all processes. But how do you know what values to actually use for your own pipelines?

1.1. Generate a resource utilization report

You already generated an execution report with -with-report in Nextflow Run. That same report is how you find out how much CPU and memory your processes actually need: run the workflow with some default allocations, record actual usage, then adjust from there.

nextflow run main.nf -with-report report-config-1.html

The report is an HTML file you can open in a browser. It breaks down runtime and resource utilization per process, including what percentage of the allocated resources was actually used. Here's what it shows for cowpy with the current defaults (1 CPU, 1 GB memory):

Metric Value
CPU usage 116%
Peak memory used 6.4 MB
Allocated memory 1 GB

cowpy uses well under 1% of its 1 GB allocation; the %cpu above 100% just means it briefly uses more than one CPU's worth of processing inside the container, in short bursts.

See Reports for the full list of available features.

1.2. Set resource allocations for a specific process

The report above shows cowpy comfortably within its current allocation, but say you wanted to give it more headroom anyway, for example because you expect larger inputs in production. You can override the defaults for a single process with withName.

nextflow.config
 6
 7
 8
 9
10
11
12
13
process {
    memory = 1.GB
    withName: 'cowpy' {
        conda = 'conda-forge::cowpy==1.1.5'
        memory = 2.GB
        cpus = 2
    }
}
nextflow.config
 6
 7
 8
 9
10
11
process {
    memory = 1.GB
    withName: 'cowpy' {
        conda = 'conda-forge::cowpy==1.1.5'
    }
}

With this in place, every process requests 1 GB of memory and a single CPU, except cowpy, which requests 2 GB and 2 CPUs (on top of the conda setting from Part 1).

Info

If your machine has few CPUs and you allocate a high number per process, task calls may queue up behind each other, since Nextflow won't request more CPUs than are available.

Run it again with a different report filename, so you can compare before and after.

nextflow run main.nf -with-report report-config-2.html
Command output 
N E X T F L O W   ~  version 26.04.4

Launching `main.nf` [voluminous_venter] revision: c3c85dec78

executor >  local (8)
[a1/0e96d4] sayHello (1)       | 3 of 3 ✔
[a3/7173a3] convertToUpper (2) | 3 of 3 ✔
[4f/a8ae3d] collectGreetings   | 1 of 1 ✔
[91/3724f8] cowpy              | 1 of 1 ✔

Outputs:

  /workspaces/training/config-exec/results

  first_output:
    - full_pipeline/intermediates/Bonjour-output.txt
    - full_pipeline/intermediates/Hola-output.txt
    - full_pipeline/intermediates/Hello-output.txt

  uppercased:
    - full_pipeline/intermediates/UPPER-Bonjour-output.txt
    - full_pipeline/intermediates/UPPER-Hello-output.txt
    - full_pipeline/intermediates/UPPER-Hola-output.txt

  collected: full_pipeline/intermediates/COLLECTED-batch-output.txt

  batch_report: full_pipeline/batch-report.txt

  cowpy_art: full_pipeline/cowpy-COLLECTED-batch-output.txt

Comparing the two reports for cowpy:

Metric Before (1 CPU, 1 GB) After (2 CPUs, 2 GB)
Peak memory used 6.4 MB 6.4 MB
CPU usage 116% 118%

Doubling the allocation didn't change actual usage at all, which tells you the original 1 GB / 1 CPU was already generous for this toy workload. On a real pipeline processing non-trivial data, you'd expect the numbers themselves to differ meaningfully between processes, which is exactly why you profile before deciding what to allocate, rather than guessing.

1.3. Add resource limits

Depending on your compute infrastructure, there may be hard constraints on what you can request, for example a cluster-wide cap. The resourceLimits directive lets you set those limits:

Syntax example
process {
    resourceLimits = [
        memory: 750.GB,
        cpus: 200,
        time: 30.d
    ]
}

Nextflow translates these into whatever the target executor expects. If a process requests more than the limit, the request gets capped rather than rejected.

Warning

This isn't something you can run in the training environment, since it requires HPC infrastructure to have an effect.

Institutional reference configurations

The nf-core project maintains a collection of configuration files shared by institutions worldwide, covering a wide range of HPC and cloud executors. They're a useful starting point whether or not your own institution is among them.

Takeaway

You know how to generate a profiling report to assess resource utilization, override resource allocations for a specific process, and cap allocations with resourceLimits.

What's next?

Learn how to make a pipeline recover automatically when a task fails, whether or not your resource allocation guess was right.

2. Handle task failures with retries

Profiling tells you what a process needs most of the time, but real workloads vary: an allocation that's comfortable for most inputs can still be too tight for an unusually large one, and guesses can simply be wrong. Rather than letting a single failed task bring down the whole run, Nextflow can retry a failed task automatically, optionally giving it more resources on each attempt.

2.1. Retry a failed task automatically

To see this in action, deliberately set cowpy's memory allocation below what it actually needs: recall from 1.1 that it peaks at around 6.4 MB, so 6 MB should be just short of enough.

nextflow.config
 6
 7
 8
 9
10
11
12
13
14
process {
    memory = 1.GB
    withName: 'cowpy' {
        conda = 'conda-forge::cowpy==1.1.5'
        memory = 6.MB
        errorStrategy = 'retry'
        maxRetries = 2
    }
}
nextflow.config
 6
 7
 8
 9
10
11
12
13
process {
    memory = 1.GB
    withName: 'cowpy' {
        conda = 'conda-forge::cowpy==1.1.5'
        memory = 2.GB
        cpus = 2
    }
}

errorStrategy tells Nextflow what to do when a task fails: 'retry' resubmits the task instead of stopping the whole pipeline. maxRetries caps how many extra attempts it gets before Nextflow gives up.

nextflow run main.nf
Command output (abridged) 
N E X T F L O W   ~  version 26.04.4

Launching `main.nf` [desperate_brazil] revision: c3c85dec78

executor >  local (10)
[67/fe1f49] sayHello (1)       | 3 of 3 ✔
[8a/f13335] convertToUpper (1) | 3 of 3 ✔
[39/1b24ed] collectGreetings   | 1 of 1 ✔
[7a/d5eb6f] cowpy              | 0 of 1, retries: 2 ✘
[9d/b79eb3] NOTE: Process `cowpy` terminated with an error exit status (137) -- Execution is retried (1)
[6d/1d9d84] NOTE: Process `cowpy` terminated with an error exit status (137) -- Execution is retried (2)
ERROR ~ Error executing process > 'cowpy'

Caused by:
  Process `cowpy` terminated with an error exit status (137)

Command executed:
  cat COLLECTED-batch-output.txt | cowpy -c "turkey" > cowpy-COLLECTED-batch-output.txt

Command exit status:
  137

Command output:
  (empty)

Command error:
  /usr/local/bin/_activate_current_env.sh: line 35:    14 Killed                  micromamba activate "${ENV_NAME:-base}"

Work dir:
  /workspaces/training/config-exec/work/7a/d5eb6feeac0eed18d95d3da7a7aeb4

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

-- Check '.nextflow.log' file for details

Exit code 137 is the standard signal for an out-of-memory kill: the container didn't have enough memory to run cowpy at all. Nextflow retried the task twice, three attempts in total, matching maxRetries = 2. Since the memory allocation never changed between attempts, every attempt hit the same wall; once retries are exhausted, Nextflow reports the failure in full and stops the pipeline, exiting with a non-zero status.

Retrying on its own doesn't fix anything if the underlying cause doesn't change between attempts.

2.2. Increase resources on each retry

Inside a process directive, task.attempt holds the current attempt number, starting at 1. You can use it in a closure to scale a resource allocation up with each retry.

nextflow.config
 6
 7
 8
 9
10
11
12
13
14
process {
    memory = 1.GB
    withName: 'cowpy' {
        conda = 'conda-forge::cowpy==1.1.5'
        memory = { 6.MB * task.attempt }
        errorStrategy = 'retry'
        maxRetries = 3
    }
}
nextflow.config
 6
 7
 8
 9
10
11
12
13
14
process {
    memory = 1.GB
    withName: 'cowpy' {
        conda = 'conda-forge::cowpy==1.1.5'
        memory = 6.MB
        errorStrategy = 'retry'
        maxRetries = 2
    }
}

Run the workflow again:

nextflow run main.nf
Command output (abridged) 
N E X T F L O W   ~  version 26.04.4

Launching `main.nf` [grave_joliot] revision: c3c85dec78

executor >  local (9)
[0f/211b8a] sayHello (2)       | 3 of 3 ✔
[26/301ea2] convertToUpper (3) | 3 of 3 ✔
[22/5a895b] collectGreetings   | 1 of 1 ✔
[e1/beee86] cowpy              | 1 of 1, retries: 1 ✔
[b6/7aed6a] NOTE: Process `cowpy` terminated with an error exit status (137) -- Execution is retried (1)

Outputs:

  /workspaces/training/config-exec/results

  first_output:
    - full_pipeline/intermediates/Hola-output.txt
    - full_pipeline/intermediates/Hello-output.txt
    - full_pipeline/intermediates/Bonjour-output.txt

  uppercased:
    - full_pipeline/intermediates/UPPER-Hola-output.txt
    - full_pipeline/intermediates/UPPER-Hello-output.txt
    - full_pipeline/intermediates/UPPER-Bonjour-output.txt

  collected: full_pipeline/intermediates/COLLECTED-batch-output.txt

  batch_report: full_pipeline/batch-report.txt

  cowpy_art: full_pipeline/cowpy-COLLECTED-batch-output.txt

The first attempt still fails at 6 MB, but the retry runs with 12 MB (6.MB * 2) and succeeds, and the pipeline completes with all outputs published.

Warning

The console output still includes a NOTE: line reporting the failed first attempt, even though the pipeline as a whole succeeded: Nextflow logs each retry individually, but a retried failure doesn't affect the overall outcome. Check for the Outputs: summary, or the command's exit status, to confirm whether the run actually succeeded.

See Dynamic computing resources in the Nextflow documentation for more advanced retry patterns, including scaling based on which specific error occurred.

Takeaway

You know how to make a pipeline automatically retry failed tasks, and how to scale resource allocations with each retry using task.attempt.

What's next?

Head on to Part 3, where you'll learn how to bundle configuration like this into switchable profiles.

Summary

In this part you learned to:

  • Generate a resource profiling report and set per-process resource allocations
  • Cap resource requests with resourceLimits
  • Automatically retry a failed task with errorStrategy and maxRetries
  • Scale a resource allocation up with each retry using task.attempt