Good afternoon! I am trying to run an nf-core workflow for chipseq analysis, but for some reason it has generated 0.5 tb of intermediate files for 28 files of input, would you be so kind as to extend my project space for ~1 tb and/or help me understand what's going on with the nf-core/chipseq workflow?
Thank you in advance!
Hi, nf-core dev here.
We're concerned about the data footprint of our pipelines, but at the moment we unfortunately cannot delete intermediary files easilly with Nextflow.
So far the options I recommended to users who had similar issues, would be either to use the scratch directive: Processes — Nextflow 21.04.0 documentation or try something out with the afterScript directive: Processes — Nextflow 21.04.0 documentation
We're working closely with Nextflow devs, and they have taken quite an interest into this issue: Automatically delete files marked as temp as soon as not needed anymore · Issue #452 · nextflow-io/nextflow · GitHub
Thank you Maxime for your help. We're glad to have you here if we were to have more tricky NF questions ^^