3. nf-core/demo¶

nf-core/demo is a simple nf-core style pipeline for workshops and demonstrations.

It was created using the nf-core template and is designed to run and configure quickly.

The nf-core/demo pipeline consists of three processes:

• (FASTQC): Read QC
• (SEQTK_TRIM): Trim low quality bases from FastQ files
• (MULTIQC): Present QC for raw reads

nf-core/demo takes a samplesheet that contains paths to fastq files as an input and will produce four output folders with a variety of logs and reports:

• fastqc/
  • *_fastqc.html: FastQC report containing quality metrics.
  • *_fastqc.zip: Zip archive containing the FastQC report, tab-delimited data file and plot images.
• fq/
  • *.fastp.html: Trimmed fq files.
• multiqc/
  • multiqc_report.html: a standalone HTML file that can be viewed in your web browser.
  • multiqc_data/: directory containing parsed statistics from the different tools used in the pipeline.
  • multiqc_plots/: directory containing static images from the report in various formats.
• pipeline_info/
  • Reports generated by Nextflow
  • Reports generated by nf-core
  • Parameters file

You can view the code for this pipeline on the nf-core/demo GitHub repository.

To help you understand the expectations for running an nf-core pipeline, they come with extensive documentation about its parameters, usage, and outputs.

The documentation for the nf-core/demo pipeline can be found on the nf-core/demo pipelines page.

3.1 Required inputs¶

Before running any nf-core pipeline you will need to check if there are any parameters that are required.

You can view these on the pipelines parameters page.

The parameters page of the nf-core/demo pipeline shows that this pipeline requires two parameters (--input and --outdir) to run.

Without these, the pipeline will not launch and nextflow will throw an error.

3.1.1 --input¶

The --input parameter requires a path to comma-separated file containing information about the samples in the experiment.

--input 'path/to/samplesheet.csv'

The nf-core/demo usage documentation describes the required --input as a comma-separated file (.csv). The .csv file must contain 3 columns with the headers sample, fastq_1, and fastq_2.

The samplesheet file may consist of both single- and paired-end data and may look something like the one below.

1
2
3
4

sample,fastq_1,fastq_2
SAMPLE1_PE,path/to/sample1_R1.fastq.gz,path/to/sample1_R2.fastq.gz
SAMPLE2_PE,path/to/sample2_R1.fastq.gz,path/to/sample2_R2.fastq.gz
SAMPLE3_SE,path/to/sample3_R1.fastq.gz,

3.1.2 --outdir¶

The --output parameter is used to name the output directory where the results will be saved. It takes a string as its input.

--output results

3.2 Testing nf-core/demo with profiles¶

A profile is a set of configuration attributes that can be added to your execution command by using the -profile option.

-profile <profile name>

Configuration profiles are defined using the special scope profile within configuration files. Profiles group the attributes that belong to the same profile using a common prefix.

1
2
3
4
5
6
7
8
9

profiles {
  foo {
    process.memory = '2 GB'
  }

  bar {
    process.memory = '4 GB'
  }
}

3.2.1 -profile test¶

Every nf-core pipeline comes with a test profile. This is a minimal set of configuration settings for the pipeline to run using a small test dataset that is hosted on the nf-core/test-datasets repository.

As the test profile is expected to run it can be used to help diagnose local issues before you scale up your analysis.

The test profile for nf-core/demo is shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

/*
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Nextflow config file for running minimal tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Defines input files and everything required to run a fast and simple pipeline test.

    Use as follows:
        nextflow run nf-core/demo -profile test,<docker/singularity> --outdir <OUTDIR>

----------------------------------------------------------------------------------------
*/

params {
    config_profile_name        = 'Test profile'
    config_profile_description = 'Minimal test dataset to check pipeline function'

    // Limit resources so that this can run on GitHub Actions
    max_cpus   = 2
    max_memory = '6.GB'
    max_time   = '6.h'

    // Input data
    input  = 'https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/samplesheet/samplesheet_test_illumina_amplicon.csv'

}

The nf-core/demo test profile already contains the input parameter (this will be explained in more detail shortly). This means that the --input parameter does not need to be added to the execution command. However, as the outdir parameter is not included in the test profile it must be added to the execution command using the --outdir flag.

nextflow run nf-core/demo -profile test --outdir results

nextflow run nf-core/demo -profile test --outdir results

As the software required to run each process (e.g., seqtk) is not available in the Gitpod environment the exercise above is expected to fail.

Caused by:
  Process `NFCORE_DEMO:DEMO:SEQTK_TRIM (SAMPLE2_PE)` terminated with an error exit status (127)
<truncated>

Fortunately, nf-core pipelines come packed with directives for containers and environments that can be flexibly enabled using profiles for different software (e.g., docker, singularity, and conda).

-profile singularity

In Gitpod, you can add the singularity profile to your execution command and Nextflow will download and enable Singularity software images to run each process.

The singularity profile is defined in the nextflow.config file in the main pipeline repository.

120
121
122
123
124
125
126
127
128
129

singularity {
    singularity.enabled     = true
    singularity.autoMounts  = true
    conda.enabled           = false
    docker.enabled          = false
    podman.enabled          = false
    shifter.enabled         = false
    charliecloud.enabled    = false
    apptainer.enabled       = false
    }

nextflow run nf-core/demo -profile test,singularity --outdir results

If you were running this tutorial you will need to have Singularity installed for this command to run.

3.3 Using your own data¶

Instead of using the test profile you can use the --input parameter to choose your own samplesheet as an input.

As described above, the input is a CSV file with 3 columns and the headers sample, fastq_1, and fastq_2.

The pipeline will auto-detect whether a sample is single- or paired-end and if a sample has been sequenced more than once using the information provided in the samplesheet.

code samplesheet.csv

1
2
3
4

sample,fastq_1,fastq_2
gut,/workspace/gitpod/nf-customize/data/gut_1.fastq.gz,/workspace/gitpod/nf-customize/data/gut_2.fastq.gz
liver,/workspace/gitpod/nf-customize/data/liver_1.fastq.gz,/workspace/gitpod/nf-customize/data/liver_2.fastq.gz
lung,/workspace/gitpod/nf-customize/data/lung_1.fastq.gz,/workspace/gitpod/nf-customize/data/lung_2.fastq.gz

You can use you new samplesheet with the --input parameter in your execution command.

In this case, the other parameters in the test profile (e.g., config_profile_name and max_cpus) can be ignored as they are not explicitly required by the pipeline or in this Gitpod environment.

nextflow run nf-core/demo -profile singularity --input samplesheet.csv --outdir results

Congratulations! You have now run the nf-core/demo pipeline using you own dataset!