5. Tools for users¶
nf-core tools has additional commands to help users execute pipelines. Although you do not need to use these commands to execute the nf-core pipelines they can greatly improve and simplify your experience.
nf-core tools are for everyone and has commands to help both users and developers.
For users, the tools make it easier to execute pipelines.
For developers, the tools make it easier to develop pipelines using best practices. However, the developer commands will not be covered as a part of this workshop. If you are curious to learn more about nf-core developer tools you can visit the websites tools page and complete the develop a pipeline with the nf-core template training.
Exercise
View nf-core commands and options are available using the nf-core --help
option.
5.1 nf-core list
¶
The nf-core list
command can be used to print a list of remote nf-core pipelines and local pipeline information.
The output shows the latest pipeline version number and when it was released. You will also be shown if and when a pipeline was pulled locally and whether you have the latest version.
Keywords can also be supplied to help filter the pipelines based on matches in titles, descriptions, or topics:
Options can also be used to sort the pipelines by latest release (-s release
, default), when you last pulled a pipeline locally (-s pulled
), alphabetically (-s name
), or number by the number of GitHub stars (-s stars
).
Exercise
Filter the list of nf-core pipelines for those that are for dna
and sort them by stars. Which dna
pipeline has the most stars?
5.2 nf-core launch
¶
A pipeline can have a large number of optional parameters. To help with this, the nf-core launch
command is designed to help you write parameter files for when you launch your pipeline.
The nf-core launch
command takes one argument - either the name of an nf-core pipeline which will be pulled automatically or the path to a directory containing a Nextflow pipeline:
You will first be asked about which version of the pipeline you would like to execute. Next, you will be given the choice between a web-based graphical interface or an interactive command-line wizard tool to enter the pipeline parameters. Both interfaces show documentation alongside each parameter, will generate a run ID, and will validate your inputs.
The nf-core launch
tool uses the nextflow_schema.json
file from a pipeline to give parameter descriptions, defaults, and grouping. If no file for the pipeline is found, one will be automatically generated at runtime.
The launch
tool will save your parameter variables as a .json
file called nf-params.json
. It will also suggest an execution command that includes the -params-file
flag and your new nf-params.json
file.
The command line wizard will conclude by asking if you want to launch the pipeline. Any profiles or options that were set using the wizard will be included in your run
command.
Exercise
Use nf-core launch
to launch the the nf-core/demo
pipeline.
Make sure you add:
- The full path to your sample sheet (
input
) - An output directory (
outdir
) - The
singularity
profile
Use the nf-core launch
command for the nf-core/demo
pipeline. Your nf-params.json
file should look something like this:
nf-params.json | |
---|---|
Your final run
command should look like this:
You can also use the launch command directly from the nf-core launch website. In this case, you can configure your pipeline using the wizard and then copy the outputs to your terminal or use the run id generated by the wizard. You will need to be connected to the internet to use the run id.
5.3 nf-core download
¶
You may need to execute an nf-core pipeline on a server or HPC system that has no internet connection. In this case, you will need to fetch the pipeline files and manually transfer them to your offline system. The nf-core tooling has the download
command to make this process easier and ensure accurate retrieval of correctly versioned code and software containers.
The nf-core download
command will download both the pipeline code and the institutional nf-core/configs files. It can also optionally download singularity image file.
The download tool will interactively prompt you for the required information is no arguments are supplied. Each prompt option has a flag and if all flags are supplied then it will run without a request for any additional user input:
- Pipeline name
- Name of pipeline you would like to download.
- Pipeline revision
- The revision you would like to download.
- Pull containers
- If you would like to download Singularity images.
- The path to a folder where you would like to store these images if you have not set your
NXF_SINGULARITY_CACHEDIR
.
- Choose compression type
- The compression type for Singularity images.
Alternatively, you could build your own execution command with the command line options.
Exercise
Use the nf-core download
command to download the nf-core/demo
pipeline with it's uncompressed Singularity images.
Use the nf-core download
command for the nf-core/demo
pipeline and follow the prompts.
Your output should look like this:
INFO Saving 'nf-core/demo'
Pipeline revision: 'main'
Use containers: 'singularity'
Container library: 'quay.io'
Using $NXF_SINGULARITY_CACHEDIR': /workspace/gitpod/nf-customize/singularity'
Output directory: 'nf-core-demo_main'
Include default institutional configuration: 'False'
If the command has been executed successfully you will see outputs indicating the singularity images and pipeline are being downloaded.
If you normally work on an offline system you would now move these files to your offline system and specify paths to these files using environmental variables, for example, the NXF_SINGULARITY_CACHEDIR
.
Follow this link to find out more about running a pipeline offline.
Congratulations! You have now utilived nf-core tools for finding, launching and downloading pipeline!