The --mem parameter of sbatch indicate the total amount fo RAM reserved for the job.
If you whish to have 40GB per CPU, you have to use the parameter --mem-per-cpu.
Do you have a file called slurm-13466845.err or slurm-13466845.out in your working directory. This could contain relevant information about why you job is failing.
The slurm.out file contains this multiple times, but no final error message :
executor > slurm (6)
[52/560488] process > STAR_Alignment (5) [ 0%] 0 of 6
The .err file was empty actually !
I am confused by the way to run my pipeline because in the nextflow script I also define the ressources I need, and the processes are done file -by-file, so they can be assigned to a core per file.
process STAR_Alignment {
cpus "${params.cpus}"
memory "40G"
module "star/2.7.5a:perl/5.26.2"
}
The params.cpu = 8 is given every time for one process, and is same as SBATCH -c 8.
Does this seem correct ?
Hello,
sure ,I don't mind,
It's only that I just re-run and this time, I got an error relative to the tool STAR and not to the cluster parameters or pipeline design...So it might be off topic ! Should I publish the error this time ?
> Command error:
>
> EXITING because of FATAL ERROR: number of bytes expected from the BAM bin does not agree with the actual size on disk: Expected bin size=538502234 ; size on disk=120863510 ; bin number=47
There is a discussion with the developer of STAR here:
space of disk needed for each file running is 3*gzipped size !
Ran with STAR/2.7.5.a
fastq files range from 3-9Gb in .gz
I am using a homemade nf pipeline, with only STAR-alignment as process. I didn't know there is a nf-core. If it is more efficient please let me know how to use it.
Moreover,
My fastq.gz files for STAR process are totally of 54Gb, so if the problem is as the developer alexdobin mentions, I need :
3 * 54 = 162Gb
Which of the above values of disk causes the problem then ?
Isn't it the 466Gb that are seen/used from the tool for alignment?
If it's the 148GiB that I really have, could we increase it ?
Otherwise, anyone knows a way to allow the alignment to happen with the given space, one-by-one file?
Hello, I 'd like some more precisions concerning disk-space ,so that I can make sure in advance if I have enough space or not to run a new pipeline for alignment.
I think I have the same problem with new data-set now, that I ran in
I ve been trying to run nextflow only with STAR process again, on 2 files only: of 16Gb and 15Gb, respectively. (4 fq paired-end, 2 samples).
The command is exactly the same as before: --runThreadN 8, and
#SBATCH --mem-per-cpu 40GB
The nextflow.log loops over the 2 alignment tasks saying "RUNNING", over 1h30! (normally in 40min should be over per file)
The STAR log file finish the alignment and bin-sorting, but there is never a .bam file written.
What might be the reason of not writing the output ?
(job IDs : 13609072, 13609073)
Hello dear Gildas,
I am running again the alignment for 1 pair of fastq (~17Gb) and the disk quota arrives at : 368Gb , as shown here, from 362Gb before running.
I asked for --mem 50Gb this time and STAR --runThreadN 8.
when I top on the node activity I see: %CPU = 799.3 for STAR.
Isn't this too much just for one pair of fastq.
Could we increase disk space @team.ifbcorecluster ?
I ll definitely need it if I run all the samples of the project in parallel : 151Gb fastq x 3 for STAR = 453Gb more !
But I d like to understand for 1 file why it cannot finish and write down the final bam file, if it is not that a large file-size !