Part 1: Plugin Basics¶
In this section, you'll learn how plugins extend Nextflow, then try three different plugins to see them in action.
1. How plugins work¶
Plugins extend Nextflow through several types of extension:
| Extension Type | What it does | Example |
|---|---|---|
| Functions | Add custom functions callable from workflows | samplesheetToList() |
| Workflow monitors | Respond to events like task completion | Custom logging, Slack alerts |
| Executors | Add task execution backends | AWS Batch, Kubernetes |
| Filesystems | Add storage backends | S3, Azure Blob |
Functions and workflow monitors (called "trace observers" in the Nextflow API) are the most common types for plugin authors. Executors and filesystems are typically created by platform vendors.
The next exercises show you function plugins and an observer plugin, so you can see both types in action.
2. Use function plugins¶
Function plugins add callable functions that you import into your workflows.
You'll try two: nf-hello (a simple example) and nf-schema (a widely-used real-world plugin).
Both exercises modify the same hello.nf pipeline, so you can see how plugins enhance an existing workflow.
2.1. nf-hello: replace hand-written code¶
The nf-hello plugin provides a randomString function that generates random strings.
The pipeline already defines its own inline version of this function, which you'll replace with the one from the plugin.
2.1.1. See the starting point¶
Look at the pipeline:
#!/usr/bin/env nextflow
params.input = 'greetings.csv'
/**
* Generate a random alphanumeric string
*/
def randomString(int length) {
def chars = ('a'..'z') + ('A'..'Z') + ('0'..'9')
def random = new Random()
return (1..length).collect { chars[random.nextInt(chars.size())] }.join()
}
process SAY_HELLO {
input:
val greeting
output:
stdout
script:
"""
echo '$greeting'
"""
}
workflow {
greeting_ch = channel.fromPath(params.input)
.splitCsv(header: true)
.map { row -> "${row.greeting}_${randomString(8)}" }
SAY_HELLO(greeting_ch)
SAY_HELLO.out.view { result -> "Output: ${result.trim()}" }
}
The pipeline defines its own randomString function inline, then uses it to append a random ID to each greeting.
Run it:
Output: Hello_aBcDeFgH
Output: Bonjour_xYzWvUtS
Output: Holà_qRsPdMnK
Output: Ciao_jLhGfEcB
Output: Hallo_tNwOiAuR
Your output order and random strings will differ, and if you run the script again you'll get a different set of random greetings.
2.1.2. Configure the plugin¶
Replace the inline function with one from the plugin. Add this to your nextflow.config:
Plugins are declared in nextflow.config using the plugins {} block.
Nextflow automatically downloads them from the Nextflow Plugin Registry, a central repository of community and official plugins.
2.1.3. Use the plugin function¶
Replace the inline randomString function with the plugin version:
#!/usr/bin/env nextflow
include { randomString } from 'plugin/nf-hello'
params.input = 'greetings.csv'
process SAY_HELLO {
input:
val greeting
output:
stdout
script:
"""
echo '$greeting'
"""
}
workflow {
greeting_ch = channel.fromPath(params.input)
.splitCsv(header: true)
.map { row -> "${row.greeting}_${randomString(8)}" }
SAY_HELLO(greeting_ch)
SAY_HELLO.out.view { result -> "Output: ${result.trim()}" }
}
#!/usr/bin/env nextflow
params.input = 'greetings.csv'
/**
* Generate a random alphanumeric string
*/
def randomString(int length) {
def chars = ('a'..'z') + ('A'..'Z') + ('0'..'9')
def random = new Random()
return (1..length).collect { chars[random.nextInt(chars.size())] }.join()
}
process SAY_HELLO {
input:
val greeting
output:
stdout
script:
"""
echo '$greeting'
"""
}
workflow {
greeting_ch = channel.fromPath(params.input)
.splitCsv(header: true)
.map { row -> "${row.greeting}_${randomString(8)}" }
SAY_HELLO(greeting_ch)
SAY_HELLO.out.view { result -> "Output: ${result.trim()}" }
}
The include statement imports randomString from a library that is proven, tested, and maintained by a broader pool of contributors who can spot and fix bugs.
Instead of each pipeline maintaining its own copy of the function, every pipeline that uses the plugin gets the same vetted implementation.
This reduces duplicated code and the maintenance burden that comes with it.
The syntax include { function } from 'plugin/plugin-id' is the same include used for Nextflow modules, with a plugin/ prefix.
You can see the source code for randomString in the nf-hello repository on GitHub.
2.1.4. Run it¶
Pipeline is starting! 🚀
Output: Hello_yqvtclcc
Output: Bonjour_vwwpyzcs
Output: Holà_wrghmgab
Output: Ciao_noniajuy
Output: Hallo_tvrtuxtp
Pipeline complete! 👋
(Your random strings will differ.)
The output still has random suffixes, but now randomString comes from the nf-hello plugin instead of inline code.
The "Pipeline is starting!" and "Pipeline complete!" messages are new.
These come from the plugin's observer component, which you'll explore in Part 5.
Nextflow downloads plugins automatically the first time they're used, so any pipeline that declares nf-hello@0.5.0 gets the exact same tested randomString function without copying code between projects.
You've now seen the three steps for using a function plugin: declare it in nextflow.config, import the function with include, and call it in your workflow.
The next exercise applies these same steps to a real-world plugin.
2.2. nf-schema: validated CSV parsing¶
The nf-schema plugin is one of the most widely-used Nextflow plugins.
It provides samplesheetToList, a function that parses CSV/TSV files using a JSON schema that defines the expected columns and types.
The pipeline currently reads greetings.csv using splitCsv and a manual map, but nf-schema can replace this with validated, schema-driven parsing.
A JSON schema file (greetings_schema.json) is already provided in the exercise directory.
What is a schema?
A schema is a formal description of what valid data looks like. It defines things like which columns are expected, what type each value should be (string, number, etc.), and which fields are required.
Think of it as a contract: if the input data doesn't match the schema, the tool can catch the problem early instead of letting it cause confusing errors later in the pipeline.
2.2.1. Look at the schema¶
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "array",
"items": {
"type": "object",
"properties": {
"greeting": {
"type": "string",
"description": "The greeting text"
},
"language": {
"type": "string",
"description": "The language of the greeting"
}
},
"required": ["greeting"]
}
}
The schema defines two columns (greeting and language) and marks greeting as required.
If someone passes a CSV missing the greeting column, nf-schema catches the error before the pipeline runs.
2.2.2. Add nf-schema to the config¶
Update nextflow.config to include both plugins:
2.2.3. Update hello.nf to use samplesheetToList¶
Replace the splitCsv input with samplesheetToList:
#!/usr/bin/env nextflow
include { randomString } from 'plugin/nf-hello'
include { samplesheetToList } from 'plugin/nf-schema'
params.input = 'greetings.csv'
process SAY_HELLO {
input:
val greeting
output:
stdout
script:
"""
echo '$greeting'
"""
}
workflow {
def samplesheet_list = samplesheetToList(params.input, 'greetings_schema.json')
greeting_ch = Channel.fromList(samplesheet_list)
.map { row -> "${row[0]}_${randomString(8)}" }
SAY_HELLO(greeting_ch)
SAY_HELLO.out.view { result -> "Output: ${result.trim()}" }
}
#!/usr/bin/env nextflow
include { randomString } from 'plugin/nf-hello'
params.input = 'greetings.csv'
process SAY_HELLO {
input:
val greeting
output:
stdout
script:
"""
echo '$greeting'
"""
}
workflow {
greeting_ch = channel.fromPath(params.input)
.splitCsv(header: true)
.map { row -> "${row.greeting}_${randomString(8)}" }
SAY_HELLO(greeting_ch)
SAY_HELLO.out.view { result -> "Output: ${result.trim()}" }
}
The custom splitCsv and map parsing code is replaced with samplesheetToList, a proven, tested function that also validates the samplesheet against the schema before the pipeline runs.
This reduces the maintenance burden of hand-written parsing logic while improving the experience for pipeline users, who get clear error messages when their input does not match the expected format.
Each row becomes a list of values in column order, so row[0] is the greeting and row[1] is the language.
2.2.4. Run it¶
Pipeline is starting! 🚀
Output: Hello_diozjdwm
Output: Bonjour_speathmm
Output: Holà_dllxnzap
Output: Ciao_wzueddzc
Output: Hallo_hsxwrjbh
Pipeline complete! 👋
(Your random strings will differ.)
The output is the same, but now the schema validates the CSV structure before the pipeline runs.
In real pipelines with complex sample sheets and many columns, this kind of validation prevents errors that manual splitCsv + map would miss.
2.2.5. See validation in action¶
To see what schema validation catches, try introducing errors into greetings.csv.
Rename the required greeting column to message:
Run the pipeline:
ERROR ~ Validation of samplesheet failed!
The following errors have been detected in greetings.csv:
-> Entry 1: Missing required field(s): greeting
-> Entry 2: Missing required field(s): greeting
-> Entry 3: Missing required field(s): greeting
-> Entry 4: Missing required field(s): greeting
-> Entry 5: Missing required field(s): greeting
The pipeline refuses to run because the schema requires a greeting column and can't find one.
Now restore the required column but rename the optional language column to lang:
This time the pipeline runs, but prints a warning:
Required columns cause hard errors; optional columns cause warnings. This is the kind of early feedback that saves debugging time in real pipelines with dozens of columns.
2.2.6. Configure validation behavior¶
The warning about lang is useful, but you can control its severity through configuration.
Plugins can include their own configuration scope(s) that control their behavior.
The nf-schema plugin includes the configuration scope validation; by modifying the settings here you can change how nf-schema behaves.
Add a validation block to nextflow.config to make unrecognized headers cause an error instead of a warning:
Run the pipeline again with the same lang column still in place:
Found the following unidentified headers in greetings.csv:
- lang
-- Check script 'hello.nf' at line: 20 or see '.nextflow.log' file for more details
The pipeline now fails instead of warning. The pipeline code didn't change; only the configuration did.
Restore greetings.csv to its original state and remove the validation block before continuing:
Both nf-hello and nf-schema are function plugins: they provide functions that you import with include and call in your workflow code.
The next exercise shows a different type of plugin that works without any include statements at all.
3. Use an observer plugin: nf-co2footprint¶
Not all plugins provide functions to import. The nf-co2footprint plugin uses a trace observer to monitor your pipeline's resource usage and estimate its carbon footprint. You don't need to change any pipeline code; just add it to the config.
3.1. Add nf-co2footprint to the config¶
Update nextflow.config:
3.2. Run the pipeline¶
The plugin produces several INFO and WARN messages during execution. These are normal for a small example running on a local machine:
nf-co2footprint plugin ~ version 1.2.0
WARN - [nf-co2footprint] Target zone null not found. Attempting to retrieve carbon intensity for fallback zone GLOBAL.
INFO - [nf-co2footprint] Using fallback carbon intensity from GLOBAL from CI table: 480.0 gCO₂eq/kWh.
WARN - [nf-co2footprint] Executor 'null' not mapped.
WARN - [nf-co2footprint] Fallback to: `machineType = null`, `pue = 1.0`. ...
...
WARN - [nf-co2footprint] No CPU model detected. Using default CPU power draw value (11.41 W).
WARN - [nf-co2footprint] 🔁 Requested memory is null for task 2. Using maximum consumed memory/`peak_rss` (0 GB) for CO₂e footprint computation.
The warnings about zone, executor, CPU model, and memory appear because the plugin can't detect the full hardware details of a local training environment. In a production environment (e.g., an HPC cluster or cloud), these values would be available and the estimates more accurate.
At the end, look for a line like:
🌱 The workflow run used 126.76 uWh of electricity, resulting in the release of 60.84 ug of CO₂ equivalents into the atmosphere.
(Your numbers will differ.)
3.3. View the report¶
The plugin generates output files in your working directory:
co2footprint_report_<timestamp>.html
co2footprint_summary_<timestamp>.txt
co2footprint_trace_<timestamp>.txt
Look at the summary:
Total CO₂e footprint measures of this workflow run (including cached tasks):
CO₂e emissions: 60.84 ug
Energy consumption: 126.76 uWh
CO₂e emissions (market): -
Which equals:
- 3.48E-7 km travelled by car
- It takes one tree 0.17s to sequester the equivalent amount of CO₂ from the atmosphere
- 1.22E-7 % of a flight from Paris to London
(Your numbers will differ.)
The first section shows the raw energy and emissions figures. The "Which equals" section puts those numbers in perspective by converting them to familiar equivalents. The summary also includes a section listing the plugin's configuration options and a citation to the Green Algorithms research paper that the calculation method is based on.
3.4. Configure the plugin¶
The "Target zone null" warning from section 3.2 appeared because the plugin had no location configured.
The nf-co2footprint plugin defines a co2footprint configuration scope where you can set your geographic location.
Add a co2footprint block to nextflow.config:
Tip
Use your own country code if you prefer (e.g., 'US', 'DE', 'FR').
Run the pipeline:
INFO - [nf-co2footprint] Using fallback carbon intensity from GB from CI table: 163.92 gCO₂eq/kWh.
The zone warning is gone. The plugin now uses GB-specific carbon intensity (163.92 gCO₂eq/kWh) instead of the global fallback (480.0 gCO₂eq/kWh).
Note
You may also see a WARN: Unrecognized config option 'co2footprint.location' message.
This is cosmetic and can be safely ignored; the plugin still reads the value correctly.
In Part 6, you'll create a configuration scope for your own plugin.
This plugin works entirely through the observer mechanism, hooking into workflow lifecycle events to collect resource metrics and generate its report when the pipeline completes.
You've now tried function plugins (imported with include) and an observer plugin (activated through config alone).
These are the two most common extension types, but as the table in section 1 shows, plugins can also add executors and filesystems.
4. Discovering plugins¶
The Nextflow Plugin Registry is the central hub for finding available plugins.
Each plugin page shows its description, available versions, installation instructions, and links to documentation.
5. Prepare for plugin development¶
The following sections (Parts 2-6) use a separate pipeline file, greet.nf, which relies on nf-schema but not nf-hello or nf-co2footprint.
Update nextflow.config to keep only nf-schema:
Remove the co2footprint output files:
The hello.nf file retains your Part 1 work for reference; going forward, you'll work with greet.nf.
Takeaway¶
You used three different plugins:
- nf-hello: A function plugin providing
randomString, imported withinclude - nf-schema: A function plugin providing
samplesheetToListfor schema-validated CSV parsing - nf-co2footprint: An observer plugin that monitors resource usage automatically, with no
includeneeded
Key patterns:
- Plugins are declared in
nextflow.configwithplugins { id 'plugin-name@version' } - Function plugins require
include { function } from 'plugin/plugin-id' - Observer plugins work automatically once declared in the config
- Plugins can define configuration scopes (e.g.,
validation {},co2footprint {}) to customize behavior - The Nextflow Plugin Registry lists available plugins
What's next?¶
The following sections show you how to build your own plugin. If you're not interested in plugin development, you can stop here or skip ahead to the Summary.