What does the state 'drain' mean?

slurm
Martin Thoma picture Martin Thoma · Mar 18, 2014 · Viewed 22.9k times · Source

When I use sinfo I see the following:

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
[...]
RG3          up 28-00:00:0      1  drain rg3hpc4
[...]

What does the state 'drain' mean?

Answer

damienfrancois picture damienfrancois · May 5, 2014

It means no further job will be scheduled on that node, but the currently running jobs will keep running (by contrast with setting the node down which kills all jobs running on the node).

Nodes are often set to that state so that some maintenance operation can take place once all running jobs are finished.

From the manpage of the scontrol command:

If you want to remove a node from service, you typically want to set it's state to "DRAIN"

Note that the system administrator most probably gave a reason why the node is drained, and you can see that reason with 

sinfo -R

Related questions

Find out the CPU time and memory usage of a slurm job

I suppose it's a pretty trivial question but nevertheless, I'm looking for the (sacct I guess) command that will display the CPU time and memory used by a slurm job ID.

Read More
How to "undrain" slurm nodes in drain state

Using sinfo it shows 3 nodes are in drain state, PARTITION AVAIL TIMELIMIT NODES STATE NODELIST all* up infinite 3 drain node[10,11,12] Which command line should I use to undrain such nodes?

Read More
SLURM `srun` vs `sbatch` and their parameters

I am trying to understand what the difference is between SLURM's srun and sbatch commands. I will be happy with a general explanation, rather than specific answers to the following questions, but here are some specific points of confusion that …

Read More