3. nf-core/demo¶

nf-core/demo is a simple nf-core style pipeline for workshops and demonstrations.

It was created using the full nf-core template and is designed to run and configure quickly.

The nf-core/demo pipeline consists of three processes:

• (FastQC): Read quality control
• (SEQTK_TRIM): Trim low quality bases from FASTQ files
• (MULTIQC): Present quality control reports for raw reads

nf-core/demo takes a samplesheet that contains paths to FASTQ files as an input and will produce four output folders with logs and reports:

• fastqc/
  • *_fastqc.html: FastQC report containing quality metrics.
  • *_fastqc.zip: Zip archive containing the FastQC report, tab-delimited data file and plot images.
• fq/
  • *.fastp.html: Trimmed fq files.
• multiqc/
  • multiqc_report.html: a standalone HTML file that can be viewed in your web browser.
  • multiqc_data/: directory containing parsed statistics from the different tools used in the pipeline.
  • multiqc_plots/: directory containing static images from the report in various formats.
• pipeline_info/
  • Reports generated by Nextflow
  • Reports generated by nf-core
  • Parameters file

You can view the code for this pipeline on the nf-core/demo GitHub repository.

To help you understand the expectations for running an nf-core pipeline, they come with extensive documentation about its parameters, usage, and outputs.

The documentation for the nf-core/demo pipeline can be found on the nf-core/demo pipelines page.

3.1 Required inputs¶

Before running any nf-core pipeline you will need to check if any parameters are required.

You can find required parameters on the pipelines parameters page.

The parameters page of the nf-core/demo pipeline shows that this pipeline requires two parameters (--input and --outdir) to run.

Without these, the pipeline will not launch and will throw an error.

3.1.1 --input¶

The --input parameter requires a path to comma-separated file (CSV) containing information about the samples in the experiment:

--input 'path/to/samplesheet.csv'

The nf-core/demo usage documentation describes the required --input as a comma-separated file (.csv). The .csv file must contain 3 columns with the headers sample, fastq_1, and fastq_2.

The samplesheet file may consist of both single- and paired-end data and may look something like the one below:

1
2
3
4

sample,fastq_1,fastq_2
SAMPLE1_PE,path/to/sample1_R1.fastq.gz,path/to/sample1_R2.fastq.gz
SAMPLE2_PE,path/to/sample2_R1.fastq.gz,path/to/sample2_R2.fastq.gz
SAMPLE3_SE,path/to/sample3_R1.fastq.gz,

3.1.2 --outdir¶

The --output parameter is used to name the output directory where the results will be saved. It takes a string as its input:

--output results

3.2 Testing nf-core/demo with profiles¶

A profile is a set of configuration attributes that can be added to your execution command by using the -profile option:

-profile <profile name>

Configuration profiles are defined using the special scope profile within configuration files. Profiles group the attributes that belong to the same profile using a common prefix:

1
2
3
4
5
6
7
8
9

profiles {
  foo {
    process.memory = '2 GB'
  }

  bar {
    process.memory = '4 GB'
  }
}

3.2.1 -profile test¶

Every nf-core pipeline comes with a test profile. This is a minimal set of configuration settings for the pipeline to run using a small test dataset that is hosted on the nf-core/test-datasets repository.

The test profile is expected to run and can be used to help diagnose local issues before you scale up your analysis.

The test profile for nf-core/demo is shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

/*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Nextflow config file for running minimal tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Defines input files and everything required to run a fast and simple pipeline test.

    Use as follows:
        nextflow run nf-core/demo -profile test,<docker/singularity> --outdir <OUTDIR>

----------------------------------------------------------------------------------------
*/

process {
    resourceLimits = [
        cpus: 4,
        memory: '15.GB',
        time: '1.h'
    ]
}

params {
    config_profile_name        = 'Test profile'
    config_profile_description = 'Minimal test dataset to check pipeline function'

    // Input data
    input  = 'https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/samplesheet/samplesheet_test_illumina_amplicon.csv'
}

The nf-core/demo test profile already contains the input parameter (explained in more detail below). This means that the --input parameter does not need to be added to the execution command. However, the outdir parameter is not included in the test profile and must be added to the execution command using the --outdir flag.

nextflow run nf-core/demo -profile test --outdir results

nextflow run nf-core/demo -profile test --outdir results

As the software required to run each process (e.g., seqtk) is not available in the Gitpod environment the exercise above is expected to fail:

Caused by:
  Process `NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE2_PE)` terminated with an error exit status (127)
<truncated>

Fortunately, nf-core pipelines come packed with directives for containers and environments that can be flexibly enabled using profiles for different software (e.g., docker, singularity, and conda):

-profile singularity

In Gitpod, you can add the singularity profile to your execution command and Nextflow will download and enable Singularity software images to run each process.

The singularity profile is defined in the nextflow.config file in the main pipeline repository.

100
101
102
103
104
105
106
107
108
109

singularity {
    singularity.enabled     = true
    singularity.autoMounts  = true
    conda.enabled           = false
    docker.enabled          = false
    podman.enabled          = false
    shifter.enabled         = false
    charliecloud.enabled    = false
    apptainer.enabled       = false
}

nextflow run nf-core/demo -profile test,singularity --outdir results

The nf-core/demo pipeline should now run successfully!

3.3 Using your own data¶

Instead of using the test profile you can use the --input parameter to choose your own sample sheet as an input.

As described above, the input is a CSV file with 3 columns and the headers sample, fastq_1, and fastq_2.

The nf-core/demo pipeline will auto-detect whether a sample is single- or paired-end and if a sample has been sequenced more than once using the information provided in the sample sheet by default.

code samplesheet.csv

1
2
3
4

sample,fastq_1,fastq_2
gut,/workspace/gitpod/nf-customize/data/gut_1.fastq.gz,/workspace/gitpod/nf-customize/data/gut_2.fastq.gz
liver,/workspace/gitpod/nf-customize/data/liver_1.fastq.gz,/workspace/gitpod/nf-customize/data/liver_2.fastq.gz
lung,/workspace/gitpod/nf-customize/data/lung_1.fastq.gz,/workspace/gitpod/nf-customize/data/lung_2.fastq.gz

You can use you new samplesheet with the --input parameter in your execution command.

In this case, the other parameters in the test profile (e.g., config_profile_name) can be ignored as they are not explicitly required by the pipeline or in this Gitpod environment.

nextflow run nf-core/demo -profile singularity --input samplesheet.csv --outdir results

The pipeline should run successfully!

Congratulations! You have now run the nf-core/demo pipeline using you own dataset!