How to "undrain" slurm nodes in drain state

slurm
elm picture elm · Apr 9, 2015 · Viewed 48.5k times · Source

Using sinfo it shows 3 nodes are in drain state,

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
all*         up   infinite      3  drain node[10,11,12]

Which command line should I use to undrain such nodes?

Answer

elm picture elm · Apr 9, 2015

Found an approach, enter scontrol interpreter (in command line type scontrol) and then

scontrol: update NodeName=node10 State=DOWN Reason="undraining"
scontrol: update NodeName=node10 State=RESUME

Then 

scontrol: show node node10

displays amongst other info

State=IDLE

Update: some of these nodes got DRAIN state back; noticed their root partition was full after e.g. show node a10 which showed Reason=SlurmdSpoolDir is full, thus in Ubuntu sudo apt-get clean to remove /var/cache/apt contents and also gzipped some /var/log files.

Related questions

Find out the CPU time and memory usage of a slurm job

I suppose it's a pretty trivial question but nevertheless, I'm looking for the (sacct I guess) command that will display the CPU time and memory used by a slurm job ID.

Read More
SLURM `srun` vs `sbatch` and their parameters

I am trying to understand what the difference is between SLURM's srun and sbatch commands. I will be happy with a general explanation, rather than specific answers to the following questions, but here are some specific points of confusion that …

Read More
Pass command line arguments via sbatch

Suppose that I have the following simple bash script which I want to submit to a batch server through SLURM: #!/bin/bash #SBATCH -o "outFile"$1".txt" #SBATCH -e "errFile"$1".txt" hostname exit 0 In this script, I simply want to write …

Read More