Low GPU usage

[lbwfff]

New issue checks

• I have read the Dorado Documentation.
• I did not find an existing issue.

Dorado version

1.1.1

Dorado subcommand

Basecaller

The issue

Hi
I am using Dorado to analyse ONT dRNA-seq data. My command line is:

/scratch/lb4489/biotools/dorado_1.1.1/bin/dorado basecaller sup,m6A_DRACH  /scratch/lb4489/project/ee_longshort/POD5/supsample/M_SH_CTX_sup/ 20260518_1503_P2I-00128-B_PBG83979_b015f4f9/pod5_skip \
    --estimate-poly-a --min-qscore 9  > ./merged/M_SHS_CTX_m6A_DRACH.bam

This is normal for the first hour or so, but afterwards it enters a state of low usage, with usage occasionally spiking to 100% for a few seconds before dropping back to 0%. This caused the HPC system to terminate the job I had submitted.

I tried increasing the number of CPUs, increasing the batch size, or using different GPUs (V00 or A100 or H100), but none of these solved the problem.

System specifications

Linux
GPU
Slurm

[malton-ont]

Hi @lbwfff,

This suggests that something later in the pipeline is backing up, resulting in the GPU work having to wait. Are you able to experiment with the configuration? For example, you could try omitting the polyA estimation or the modification calling - this would demonstrate that the GPU basecalling is at least working as expecting.

The other possibility is disk I/O. You say this is on your HPC system - are your input and output locations on local disk or network locations? We would recommend local for best results.

Are you able to monitor the usage stats on this system? If you are not allocating enough RAM, this could also cause a slowdown, and may actually explain why your HPC system is killing the job - I wouldn't expect this to happen simply because GPU utilisation has dropped, but an out-of-memory killer could cause it.

I'm also a little confused by your command line - it appears to have too many positional parameters, like you've specified two input folders? Or is there just an extraneous space in there?

[lbwfff]

Hi,

Both input and output are on the local disk. During runtime, I allocated 40 cores and over 120GB of memory; the monitoring shows that neither the CPU load nor memory usage has reached its limit. The two position parameters are due to a space character being inserted during pasting.

When using the V100, I can just about get round the HPC restrictions, but when using the A100 or H100, the process is terminated.

It may be that running polyA estimation and modification calling simultaneously is the cause; I’ll try not doing them.

[malton-ont]

@lbwfff,

Did you have any success varying the analysis or HPC parameters?

[lbwfff]

Hi @malton-ont ,

Yes, I tried different parameters, and removing --estimate-poly-a solved the problem.
However, Poly(A) estimation is also a topic of interest to us. Do you have any suggestions regarding this?

[malton-ont]

Hi @lbwfff,

PolyA estimation shouldn't be that CPU intensive. I see you are using dorado-1.1.1 - have you tried updating to the latest version? Older versions of dorado (v2.0.0 and earlier) tried to allocate all remaining threads to polyA, which may not be playing nicely with the resource allocation system you are using? There have also been a number of CPU and GPU optimisations between 1.1.1 and now.

What is the N50 of your dataset? PolyA estimation is performed per-read, so if you have lots and lots of short reads I can see it becoming a bit more intensive.

I would also recommend performing a short run (something that will keep the basecaller busy for a few minutes) using the --run-batchsize-benchmarks parameter to generate a precalculated batchsize table, then passing this via the --batchsize-benchmarks-file parameter on the full run - we don't have access to the GPU model that you are using so dorado will be running a simple benchmark at startup which is not as accurate as the full ones that this will produce.