Adding time to a running slurm job

slurm
user1701545 picture user1701545 · Feb 9, 2015 · Viewed 30.2k times · Source

I have a job running a linux machine managed by slurm. Now that the job is running for a few hours I realize that I underestimated the time required for it to finish and thus the value of the --time argument I specified is not enough. Is there a way to add time to an existing running job through slurm?

Answer

Carles Fenoy picture Carles Fenoy · Feb 9, 2015

Use the scontrol command to modify a job

scontrol update jobid=<job_id> TimeLimit=<new_timelimit>

Use the SLURM time format, eg. for 8 days 15 hours: TimeLimit=8-15:00:00

Requires admin privileges, on some machines.

Will be allowed to users only if the job is not running yet, on most machines.

Related questions

Find out the CPU time and memory usage of a slurm job

I suppose it's a pretty trivial question but nevertheless, I'm looking for the (sacct I guess) command that will display the CPU time and memory used by a slurm job ID.

Read More
How to "undrain" slurm nodes in drain state

Using sinfo it shows 3 nodes are in drain state, PARTITION AVAIL TIMELIMIT NODES STATE NODELIST all* up infinite 3 drain node[10,11,12] Which command line should I use to undrain such nodes?

Read More
SLURM `srun` vs `sbatch` and their parameters

I am trying to understand what the difference is between SLURM's srun and sbatch commands. I will be happy with a general explanation, rather than specific answers to the following questions, but here are some specific points of confusion that …

Read More