3. Adding modules¶

The nf-core pipeline template is a working pipeline and comes pre-configured with two modules:

• FastQC: A tool that performs quality control checks on raw sequence data coming from high throughput sequencing pipelines. It provides a modular set of analyses that can be used to give a quick impression of your data.
• MultiQC: A modular tool to aggregate results from bioinformatics analyses across many samples into a single report.

3.1 Testing your pipeline¶

You can use the test profile can be used to check if your pipeline is still working during your development cycle. You can also use it in GitHub Actions to test your pipeline during pull requests.

The default template test profile leverages small test files that are stored in the nf-core test data GitHub repository as inputs for the pipeline.

Additionally, the template comes with profiles for the management of software dependencies (e.g., docker, singularity, and conda). nf-core modules come with containers/images/recipes and profiles can be used to change the way dependencies are handled when you execute your pipeline.

Additional test profiles can be created to test different parts of your pipeline and can also be added to GitHub actions.

cd /workspace/gitpod/nf-develop
nextflow run myorg-myfirstpipeline -profile test,singularity --outdir results

3.2 Adding a new tool to your pipeline¶

Here, you will add another tool to your pipeline.

Seqtk is a fast and lightweight tool for processing sequences in the FASTA or FASTQ format. Here, you will use the seqtk trim command to trim FASTQ files.

In your pipeline, you will add a new step that will take FASTQ files from the sample sheet as inputs and will produce trimmed fastq files that can be used as an input for other tools and version information about the seqtk tools to mix into the inputs for the MultiQC process.

While you could develop a module for this tool independently, you can save a lot of time and effort by leveraging nf-core modules and subworkflows.

nf-core modules and subworkflows are written and maintained by the nf-core community. They are designed to be flexible but may require additional configuration to suit different use cases. Currently, there are more than 1250 nf-core modules and 60 nf-core subworkflows (April 2024) available.

Modules and subworkflows can be listed, installed, updated, removed, and patched using nf-core tooling.

3.2.1 Installing the seqtk/trim module¶

The nf-core modules list command can be used to show the modules in your local pipeline or the nf-core remote repository.

nf-core modules list remote

The nf-core modules install command can be used to install the seqtk/trim module directly from the nf-core repository:

nf-core modules install

You can follow the prompts to find and install the module you are interested in:

? Tool name: seqtk/trim

Once selected, the tooling will install the module in the modules/nf-core/ folder and suggest code that you can add to your main workflow file (workflows/mypipeline.nf).

INFO     Installing 'seqtk/trim'
INFO     Use the following statement to include this module:

include { SEQTK_TRIM } from '../modules/nf-core/seqtk/trim/main'

cd my-myfirstpipeline
nf-core modules install

To enable reporting and reproducibility, modules and subworkflows from the nf-core repository are tracked using hashes in the modules.json file. When modules are installed or removed using the nf-core tooling the modules.json file will be automatically updated.

3.2.2 Adding a module to your pipeline¶

Although the module has been installed in your local pipeline repository, it is not yet added to your pipeline.

The suggested include statement needs to be added to your workflows/mypipeline.nf file and the process call (with inputs) needs to be added to the workflow block.

6
7
8

include { FASTQC                 } from '../modules/nf-core/fastqc/main'
include { SEQTK_TRIM             } from '../modules/nf-core/seqtk/trim/main'
include { MULTIQC                } from '../modules/nf-core/multiqc/main'

7

include { SEQTK_TRIM             } from '../modules/nf-core/seqtk/trim/main'

To add the SEQTK_TRIM module to your workflow you will need to check what inputs are required.

You can view the input channels for the module by opening the ./modules/nf-core/seqtk/trim/main.nf file.

10
11

input:
tuple val(meta), path(reads)

Each nf-core module also has a meta.yml file which describes the inputs and outputs. This meta file is rendered on the nf-core website, or can be viewed using the nf-core modules info command.

nf-core modules info seqtk/trim

Using this module information you can work out what inputs are required for the SEQTK_TRIM process:

1. tuple val(meta), path(reads)

   • A tuple with a meta map and a list of FASTQ files
   • The channel ch_samplesheet used by the FASTQC process can be used as the reads input.

As only one input channel required, and it already exists, it can be added to your mypipeline.nf file without any additional channel creation or modifications.

37
38
39
40
41
42

//
// MODULE: Run SEQTK_TRIM
//
SEQTK_TRIM (
    ch_samplesheet
)

As with the inputs, you can view the outputs for the module by opening the /modules/nf-core/seqtk/trim/main.nf file and viewing the module metadata.

13
14
15

output:
tuple val(meta), path("*.fastq.gz"), emit: reads
path "versions.yml"                , emit: versions

To help with organization and readability it is beneficial to create named output channels.

For SEQTK_TRIM, the reads output could be put into a channel named ch_trimmed.

ch_trimmed  = SEQTK_TRIM.out.reads

Similarly, it is beneficial immediately mix the versions of tools into the ch_versions channel so they can be used as an input for the MULTIQC process.

ch_versions = ch_versions.mix(SEQTK_TRIM.out.versions.first())

43
44

ch_trimmed  = SEQTK_TRIM.out.reads
ch_versions = ch_versions.mix(SEQTK_TRIM.out.versions.first())

3.2.3 Additional configuration options¶

To prevent changing the nf-core modules, additional configuration options can be applied to a module using scopes within configuration files.

The configuration of modules is commonly added to the modules.conf file in the conf folder. Process selectors (e.g., withName) are used to apply configuration to modules selectively. Process selectors must be used within the process scope.

Extra configuration may also be applied as directives by using args. You can find many examples of how arguments are added to modules in nf-core pipelines, for example, the nf-core/rnaseq modules.config file.

24
25
26
27
28
29
30

withName: 'SEQTK_TRIM' {
    publishDir = [
        path: { "${params.outdir}/fq/${meta.id}" },
        mode: params.publish_dir_mode,
        pattern: "*.{fastq.gz}"
    ]
}

3.2.4 Checking your module has been added¶

It is important to regularly check that you have not broken your pipeline during development. Testing often can help identify issues quicker as you have less files have been modified and mistakes will be easier to identify.

The test profile is perfect for this use case.

cd /workspace/gitpod/nf-develop
nextflow run myorg-myfirstpipeline -profile test,singularity --outdir results

Congratulations! You have added your first nf-core module to the nf-core template!