What and Why?

The nf-core/tools CLI is the backbone of our community. It encapsulates the pipeline template that we use and packages and simplifies the tasks involved in keeping nf-core pipelines up to date. The same holds true for modules and subworkflows. In many ways, it is the linchpin of our collaborative coding.

This week, we are thrilled to announce nf-core/tools version 3.0.0. With this release we bring much-requested template granularity, better findability for the many commands, and a suite of major updates to the nf-core pipeline template. This blog post explains what’s new and guides you through the update process.

We promise, it’s worth it! 🙏

✨ New features

• Enhanced pipeline template customisation
  • The template has been divided into features that can be selectively included or excluded. For example, you can now create a new pipeline without any traces of FastQC.
  • You can strip down the pipeline to the bare minimum and add only the tools you need.
  • For nf-core pipelines, certain core features (e.g., documentation, CI tests) remain mandatory, but you still have significant customisation flexibility.
• New Text User Interface (TUI) for pipeline creation
  • A guided interface helps you through the process when running nf-core pipelines create (don’t worry - you can still use the CLI by providing all values as parameters). Play
• Switch to nf-schema for input validation
  • Replaces its predecessor nf-validation as a standard plugin in the template
• CI tests now use the nf-core tools version matching the pipeline’s template version
  • This will reduce errors in opened PRs with each new tools release

⛓️‍💥 Breaking changes

• All pipeline commands now require the pipelines prefix. This change makes the commands more consistent with nf-core modules and nf-core subworkflows commands. The commands which changed are:

  old command	new command
  nf-core lint	nf-core pipelines lint
  nf-core launch	nf-core pipelines launch
  nf-core download	nf-core pipelines download
  nf-core create-params-file	nf-core pipelines create-params-file
  nf-core create	nf-core pipelines create
  nf-core lint	nf-core pipelines lint
  nf-core bump-version	nf-core pipelines bump-version
  nf-core sync	nf-core pipelines sync
  nf-core schema build	nf-core pipelines schema build
  nf-core schema docs	nf-core pipelines schema docs
  nf-core schema lint	nf-core pipelines schema lint
  nf-core schema validate	nf-core pipelines schema validate
  nf-core create-logo	nf-core pipelines create-logo

• Some options have been changed for the nf-core pipelines download command:

  • The -t / --tower flag has been renamed to -p / --platform.
  • We renamed the short flags for consistency, to always use the first letter of the second word in the long flag:
    • The -d / --download-configuration flag has been renamed to -c / --download-configuration.
    • The -p / --parallel-downloads flag has been renamed to -d / --parallel-downloads.

🫡 Deprecations

• The nf-core licences command is deprecated.
• Python 3.8 reaches end of life in October 2024, so this will probably be the last release supporting Python 3.8.

Tip: Avoiding Merge Conflicts

The v3 release of nf-core/tools includes the ability to have fine-grained control of which template features to include. You can use this new functionality to switch off chunks of the template. Doing so means less code will update in the template, and fewer merge conflicts in files that you don’t care about.

So - if you don’t use any of these template features:

• fastqc
• multiqc
• igenomes
• nf_schema

you can minimize merge conflicts with a quick update and intermediate sync:

1. Start by checking out the dev branch:

git switch dev

2. Update the template to the latest version:

nf-core pipelines sync

3. Pull the updated .nf-core.yml file from the TEMPLATE branch:

git checkout TEMPLATE -- .nf-core.yml

4. Add the features you want to skip to skip_features:

template:
    skip_features:
        - fastqc
        - igenomes
        - nf_schema

5. Commit the changes:

git add .nf-core.yml
git commit -m "Skip fastqc, igenomes and nf_schema"

6. Add the changes to the dev branch:

git push origin dev

7. Merge the changes into the dev branch of the nf-core repository:
8. Retrigger the pipeline sync via the GitHub Actions workflow using the name of your pipeline as the input.
9. Your template update merge should now have fewer conflicts! 🎉

Important Template Updates

The check_max() Function Has Been Removed

The check_max() function has been replaced by core Nextflow functionality called resourceLimits.

The resourceLimits are specified in the nextflow.config file. You can remove all references to check_max() and its associated parameters (max_cpus, max_memory, and max_time). For more information, see the Nextflow documentation.

The nf-validation Plugin Has Been Replaced by nf-schema

The nf-validation plugin is deprecated in favour of the new nf-schema plugin.

This plugin uses a new JSON schema draft (2020-12), requiring changes to the nextflow_schema.json and assets/schema_input.json files. Follow the migration guide for required changes.

The following validation parameters have been removed from nextflow.config and nextflow_schema.json:

• validationFailUnrecognisedParams
• validationLenientMode
• validationSchemaIgnoreParams
• validationShowHiddenParams
• validate_params

Instead, use the validation scope for nf-schema options.

The UTILS_NFVALIDATION_PLUGIN subworkflow has been replaced by UTILS_NFSCHEMA_PLUGIN, changing how the input samplesheet is read. See the documentation for details on using the new samplesheetToList() function.

Removing scripting from nextflow.config

To prepare for upcoming Nextflow syntax depreciations, for loops and try/catch blocks have been removed from the template config files:

• nf-core configs are now included without try/catch blocks
• Include statements have been moved after profile definitions to ensure correct profile overriding
• For more information, see these Nextflow issues: #1792 and #5306

Common Merge Conflicts and Solutions

Video walk-through

README.md

In the ## Credits section, you might see conflicts like this:

<<<<<<< HEAD
- nf-core/mag was written by [Hadrien Gourlé](https://hadriengourle.com) at [SLU](https://slu.se), [Daniel Straub](https://github.com/d4straub) and - [Sabrina Krakau](https://github.com/skrakau) at the [Quantitative Biology Center (QBiC)](http://qbic.life). [James A. Fellows Yates](https://github.- com/jfy133) and [Maxime Borry](https://github.com/maxibor) at the [Max Planck Institute for Evolutionary Anthropology](https://www.eva.mpg.de) joined in version 2.2.0.
-
- Other code contributors include:
-
- - [Antonia Schuster](https://github.com/AntoniaSchuster)
- - [Alexander Ramos](https://github.com/alxndrdiaz)
- - [Carson Miller](https://github.com/CarsonJM)
- - [Daniel Lundin](https://github.com/erikrikarddaniel)
- - [Danielle Callan](https://github.com/d-callan)
- - [Gregory Sprenger](https://github.com/gregorysprenger)
- - [Jim Downie](https://github.com/prototaxites)
- - [Phil Palmer](https://github.com/PhilPalmer)
- - [@willros](https://github.com/willros)
-
- Long read processing was inspired by [caspargross/HybridAssembly](https://github.com/caspargross/HybridAssembly) written by Caspar Gross [@caspargross](https://github.com/caspargross)
- nf-core/mag was written by [Hadrien Gourlé](https://hadriengourle.com) at [SLU](https://slu.se), [Daniel Straub](https://github.com/d4straub) and [Sabrina Krakau](https://github.com/skrakau) at the [Quantitative Biology Center (QBiC)](http://qbic.life).
=======
+ nf-core/mag was originally written by Hadrien Gourlé, Daniel Straub, Sabrina Krakau, James A. Fellows Yates, Maxime Borry.
>>>>>>> TEMPLATE

Resolution

Verify that all authors are included. If confirmed, keep your existing credits section and ignore incoming changes.

CITATIONS.md

Citations you added might have been removed in the template update.

Resolution

Reject the incoming changes to keep your citations.

conf/base.config

The above mentioned removal of check_max() might cause conflicts in the config files

<<<<<<< HEAD
-    cpus   = { check_max( 1    * task.attempt, 'cpus'   ) }
-    memory = { check_max( 6.GB * task.attempt, 'memory' ) }
-    time   = { check_max( 4.h  * task.attempt, 'time'   ) }
=======
+    // TODO nf-core: Check the defaults for all processes
+    cpus   = { 1      * task.attempt }
+    memory = { 6.GB   * task.attempt }
+    time   = { 4.h    * task.attempt }
>>>>>>> TEMPLATE

Resolution

Double-check the values and accept the incoming changes if correct.

.nf-core.yml

There might be conflicts due to changed order, renamed and new fields, especially in the template section.

Resolution

Double-check the changes for duplicates and accept the incoming changes.

The new file should follow this structure:

bump_version: null
lint: null
nf_core_version: 3.0.0
org_path: null
template:
    author: Author Name
    description: The description of the pipeline
    force: false
    is_nfcore: true
    name: pipelinename
    org: nf-core
    outdir: .
    skip_features: []
    version: 3.0.0
update: null

See the API docs for a more detailed description of each field and the allowed input values.

nextflow.config

Several merge conflicts due to changes described in Important Template Updates.

Resolution

Double-check in the parameters section, that no pipeline-specific parameter would be removed by the incoming changes. In general, you can accept the incoming changes for this file.

Pipeline utils subworkflow

In subworkflows/local/utils_nfcore_$PIPELINE_NAME_pipeline/main.nf the switch to nf-schema might cause conflicts in the logic of reading in a samplesheet.

Resolution

Be careful in accepting incoming changes. The main changes you should do are:

- include { fromSamplesheet } from 'plugin/nf-validation'
+ include { samplesheetToList } from 'plugin/nf-schema'

- Channel.fromSamplesheet("input")
+ Channel.fromList(samplesheetToList(params.input, "path/to/samplesheet/schema"))

-  UTILS_NFVALIDATION_PLUGIN (
+  UTILS_NFSCHEMA_PLUGIN (

See the nf-schema migration guide for more details in replacing the fromSamplesheet function.

Published on

7 October 2024

Written by

@mirpedrol

@mashehu

@ewels