5. Tools for users¶

nf-core tools has additional commands to help users execute pipelines. Although you do not need to use these commands to execute the nf-core pipelines they can greatly improve and simplify your experience.

nf-core tools are for everyone and has commands to help both users and developers.

For users, the tools make it easier to execute pipelines.

For developers, the tools make it easier to develop pipelines using best practices. However, the developer commands will not be covered as a part of this workshop. If you are curious to learn more about nf-core developer tools you can visit the websites tools page and complete the develop a pipeline with the nf-core template training.

5.1 nf-core pipelines list¶

The nf-core list command can be used to print a list of remote nf-core pipelines and local pipeline information.

nf-core pipelines list

The output shows the latest pipeline version number and when it was released. You will also be shown if and when a pipeline was pulled locally and whether you have the latest version.

Keywords can also be supplied to help filter the pipelines based on matches in titles, descriptions, or topics:

nf-core pipelines list dna

Options can also be used to sort the pipelines by latest release (-s release, default), when you last pulled a pipeline locally (-s pulled), alphabetically (-s name), or number by the number of GitHub stars (-s stars).

nf-core pipelines list dna -s stars

5.2 nf-core pipelines launch¶

A pipeline can have a large number of optional parameters. To help with this, the nf-core pipelines launch command is designed to help you write parameter files for when you launch your pipeline.

The nf-core launch command takes one argument - either the name of an nf-core pipeline which will be pulled automatically or the path to a directory containing a Nextflow pipeline:

nf-core pipelines launch

You will first be asked about which version of the pipeline you would like to execute. Next, you will be given the choice between a web-based graphical interface or an interactive command-line wizard tool to enter the pipeline parameters. Both interfaces show documentation alongside each parameter, will generate a run ID, and will validate your inputs.

The nf-core launch tool uses the nextflow_schema.json file from a pipeline to give parameter descriptions, defaults, and grouping. If no file for the pipeline is found, one will be automatically generated at runtime.

The launch tool will save your parameter variables as a .json file called nf-params.json. It will also suggest an execution command that includes the -params-file flag and your new nf-params.json file.

The command line wizard will conclude by asking if you want to launch the pipeline. Any profiles or options that were set using the wizard will be included in your run command.

1
2
3
4

{
    "input": "/workspace/gitpod/nf-customize/samplesheet.csv",
    "outdir": "results"
}

nextflow run nf-core/demo -profile singularity -params-file nf-params.json

You can also use the launch command directly from the nf-core launch website. In this case, you can configure your pipeline using the wizard and then copy the outputs to your terminal or use the run id generated by the wizard. You will need to be connected to the internet to use the run id.

nf-core pipelines launch --id <run_id>

5.3 nf-core pipelines download¶

You may need to execute an nf-core pipeline on a server or HPC system that has no internet connection. In this case, you will need to fetch the pipeline files and manually transfer them to your offline system. The nf-core tooling has the download command to make this process easier and ensure accurate retrieval of correctly versioned code and software containers.

The nf-core download command will download both the pipeline code and the institutional nf-core/configs files. It can also optionally download singularity image file.

nf-core pipelines download

The download tool will interactively prompt you for the required information is no arguments are supplied. Each prompt option has a flag and if all flags are supplied then it will run without a request for any additional user input:

• Pipeline name
  • Name of pipeline you would like to download.
• Pipeline revision
  • The revision you would like to download.
• Pull containers
  • If you would like to download Singularity images.
  • The path to a folder where you would like to store these images if you have not set your NXF_SINGULARITY_CACHEDIR.
• Choose compression type
  • The compression type for Singularity images.

Alternatively, you could build your own execution command with the command line options.

INFO Saving 'nf-core/demo'
  Pipeline revision: 'main'
  Use containers: 'singularity'
  Container library: 'quay.io'
  Using $NXF_SINGULARITY_CACHEDIR': /workspace/gitpod/nf-customize/singularity'
  Output directory: 'nf-core-demo_main'
  Include default institutional configuration: 'False'

If the command has been executed successfully you will see outputs indicating the singularity images and pipeline are being downloaded.

If you normally work on an offline system you would now move these files to your offline system and specify paths to these files using environmental variables, for example, the NXF_SINGULARITY_CACHEDIR.

See Running offline to find out more about running a pipeline offline.

Congratulations! You have now utilized nf-core tools for finding, launching and downloading pipeline!