Part 9: Hello Seqera

So far we've been running Nextflow workflows on our local machine using the command line interface. In this section, we'll introduce you to Seqera Platform, a powerful cloud-based platform for running, monitoring, and sharing Nextflow workflows.

Tip

Sign up to try Seqera for free or request a demo for deployments in your own on-premise or cloud environment.

You'll learn how to use Seqera Platform within Nextflow CLI, the Seqera Platform GUI, and the API.

You can use Seqera Platform via either the CLI, through the online GUI or through the API.

1. Use Seqera Platform to capture and monitor Nextflow jobs launched from the CLI

We'll start by using the Nextflow CLI to launch a pipeline and monitor it in Seqera Platform. Start by logging into the Seqera Platform.

Nextflow Tower

Seqera Platform was previously known as Nextflow Tower. You'll still see references to the previous name in environment variables and CLI option names.

1.1. Set up your Seqera Platform token by exporting it to your environment

Follow these steps to set up your token:

1. Create a new token by clicking on the Settings drop-down menu:

   

2. Name your token:

   

3. Save your token safely:

   

4. To make your token available to the Nextflow CLI, export it on the command line:

   Open a terminal and type:

    export TOWER_ACCESS_TOKEN=eyxxxxxxxxxxxxxxxQ1ZTE=
   

   Where eyxxxxxxxxxxxxxxxQ1ZTE= is the token you have just created.

   Security Note

   Keep your token secure and do not share it with others. Add a Space before the export command above to prevent your token from being saved in your shell history.

1.2. Run Nextflow CLI with Seqera Platform visualizing and capturing logs

Run a Nextflow workflow with the addition of the -with-tower command:

nextflow run nextflow-io/hello -with-tower

You will see output similar to the following:

Output

 N E X T F L O W   ~  version 24.04.4

Launching `https://github.com/nextflow-io/hello` [evil_engelbart] DSL2 - revision: afff16a9b4 [master]

Downloading plugin nf-tower@1.9.1
Monitor the execution with Seqera Platform using this URL: https://cloud.seqera.io/user/kenbrewer/watch/5Gs0qqV9Y9rguE
executor >  local (4)
[80/810411] process > sayHello (1) [100%] 4 of 4 ✔
Ciao world!

Bonjour world!

Hola world!

Hello world!

Hold Ctrl or Cmd and click on the link to open it in your browser. You'll see the Seqera Platform interface with the job finished and the logs captured.

You will see and be able to monitor your Nextflow jobs in Seqera Platform.

1.3. Set up Seqera Platform in Nextflow configuration

Doing that token setup regularly can become tedious, so let's set this configuration for all our pipeline runs with the global Nextflow configuration file located at $HOME/.nextflow/config.

Before we set the configuration, we need to permanently store the token from our environment in Nextflow using a Nextflow secret:

nextflow secrets set tower_access_token "$TOWER_ACCESS_TOKEN"

Make sure your token was saved using:

nextflow secrets get tower_access_token

Next, open the Nextflow configuration file located at $HOME/.nextflow/config:

code $HOME/.nextflow/config

Then add the following block of configuration:

$HOME/.nextflow/config

tower {
    enabled = true
    endpoint = "https://api.cloud.seqera.io"
    accessToken = secrets.tower_access_token
}

Endpoint

The endpoint is the URL of the Seqera Platform API. If your institution is running a private instance of Seqera Platform, you should change this to the appropriate URL.

Run your Nextflow workflows as before, but without the -with-tower command:

nextflow run nextflow-io/hello

You will see the following output:

Output

 N E X T F L O W   ~  version 24.04.4

Launching `https://github.com/nextflow-io/hello` [fabulous_euclid] DSL2 - revision: afff16a9b4 [master]

Monitor the execution with Seqera Platform using this URL: https://cloud.seqera.io/user/kenbrewer/watch/KYjRktIlOuxrh
executor >  local (4)
[71/eaa915] process > sayHello (3) [100%] 4 of 4 ✔
Ciao world!

Bonjour world!

Hola world!

Hello world!

Note that we are logging to Seqera Platform even though we did not use the -with-tower command!

1.4. Use Seqera Platform to explore the resolved configuration of a Nextflow pipeline

Click on the link provided in the output to open the Seqera Platform for your run, then click on the Configuration tab. If you ran your pipeline from the hello_nextflow directory, you'll see something like this:

Notice that configuration for our pipeline run is being run pulled from three separate files:

• /home/gitpod/.nextflow/config - This is the global configuration file we just added.
• /home/gitpod/.nextflow/assets/nextflow-io/hello/nextflow.config - This is the nextflow.config file from the nextflow-io/hello repository.
• /workspace/gitpod/nf-training/hello-nextflow/nextflow.config - This is the nextflow.config file from our current working directory.

Nextflow resolves these configurations at runtime with a specific order of precedence. The general rule, however, is that more specific configurations override less specific ones, and config/params specified on the CLI will override defaults in the config files.

Helpfully, Seqera Platform shows us the final output of this configuration resolution process which can be very useful for debugging!

Takeaway

You have learned how to:

• Set up your Seqera Platform token by exporting it to your environment.
• Run Nextflow CLI with Seqera Platform visualizing and capturing logs.
• Set up Seqera Platform logging by default.
• Use Seqera Platform to explore the resolved configuration of a Nextflow pipeline.

What's next?

Learn how to launch Nextflow pipelines from Seqera Platform using the Launchpad feature.

2 Using Seqera Platform Launchpad to run Nextflow workflows

So far we've been running Nextflow workflows on our local machine using the command line interface but sending the logs to Seqera Platform for monitoring and visualization. Next we want to start using Seqera Platform to launch Nextflow workflows on our behalf.

Community Showcase

Having a compute environment capable of running Nextflow workflows configured in Seqera Platform is normally a prerequisite for this task. But we want to see how it works before we put in that effort, so we'll start by launching a job in community/showcase workspace which has a compute environment already set up.

Trainer Tip

Launch a test run of the nf-core/rnaseq pipeline in the community/showcase workspace prior to starting this session, so you'll have a recent run for participants to inspect.

2.1. Navigate to the community/showcase workspace

Seqera Platform has a concept of organizations and workspaces which are used to organize and share pipelines, compute environments, data, credentials, and more. The community/showcase workspace is a public workspace where you can see some example pipelines and compute environments. Each user has an allotted amount of free compute to use in this workspace.

Click on your username in the top left corner of the screen to bring up the list of organizations and workspaces you have access to. Select the community/showcase workspace.

2.2. Launch a test run of nf-core/rnaseq pipeline

In the community/showcase workspace, you will see a list of pipelines that have been set up by the workspace owner for you to run. Follow these steps to launch a test run of a pipeline:

1. Find the nf-core-rnaseq pipeline in the list of pipelines.
2. Click on the Launch button to bring up the launch form.
3. Change the "Workflow run name" to "-rnaseq-test".
4. Click "Next" to bring up the parameters form.
5. Find the trimmer parameter and change it to fastp.
6. Click on "Next" to inspect the advanced configuration.
7. Click "Launch" to start the pipeline!

Tip

In the advanced configuration, you'll see a section named "Pre-run script" with a script similar to the following:

export NXF_FILE_ROOT=s3://nf-tower-bucket/scratch/$TOWER_WORKFLOW_ID

This is what ensures that everyone's pipeline will write to a unique location in cloud storage despite all having the outdir parameter set to ./results.

2.3. Monitor the pipeline run

After launching the pipeline, you will be taken to the pipeline run page where you can monitor the progress of the pipeline. It may take some time for the pipeline to start running while AWS Batch spins up the needed resources, so go to the "Runs" tab above the pipeline and open a recent completed (or failed) run by one of your "teammates" in the community.

2.4. Inspect a pipeline run

Scroll down to find the list of tasks that were executed in the pipeline run. For example, by searching for fastq we can find the task NFCORE_RNASEQ:RNASEQ:FASTQ_FASTQC_UMITOOLS_FASTP:FASTP (WT_REP2) that was executed as part of the pipeline.

Click on the task to see the task details:

1. Find the following details on the "About" page for the the task you're inspecting:

   • How long did the task script run (not including scheduling time)?
   • How many CPUs were allocated to the task?
   • What was the virtual machine type that the task ran on?
   • What was the estimated cost of the task?

2. Explore the Execution Log tab. What information is available here?

3. Explore the Data Explorer tab. Note that the work directory structure we've seen during local runs is replicated here in cloud storage!

Takeaway

You have learned how to:

• Switch between organizations and workspaces in Seqera Platform.
• Launch a Nextflow pipeline that ran in the cloud using Seqera Platform.
• Monitor the progress of the pipeline run.
• Inspect the details of a task that was executed as part of the pipeline.

Next steps

In the next section, we will learn how to set up a compute environment in Seqera Platform to run our own Nextflow workflows.