Error in SLURM cluster - Detected 1 oom-kill event(s): how to improve running jobs

memory cluster-computing slurm
CafféSospeso picture CafféSospeso · Sep 20, 2018 · Viewed 11.8k times · Source

I'm working in a SLURM cluster and I was running several processes at the same time (on several input files), and using the same bash script.

At the end of the job, the process was killed and this is the error I obtained.

slurmstepd: error: Detected 1 oom-kill event(s) in step 1090990.batch cgroup.

My guess is that there is some issue with memory. But how can I know more about? Did I not provide enough memory? or as user I was requesting more than what I have access to?

Any suggestion?

Answer

Kyle picture Kyle · Dec 30, 2018

Here OOM stands for "Out of Memory". When Linux runs low on memory, it will "oom-kill" a process to keep critical processes running. It looks like slurmstepd detected that your process was oom-killed. Oracle has a nice explanation of this mechanism.

If you had requested more memory than you were allowed, the process would not have been allocated to a node and computation would not have started. It looks like you need to request more memory.

Related questions

How to change memory per node for apache spark worker

I am configuring an Apache Spark cluster. When I run the cluster with 1 master and 3 slaves, I see this on the master monitor page: Memory 2.0 GB (512.0 MB Used) 2.0 GB (512.0 MB Used) 6.0 GB (512.0 MB Used) I want to increase the used …

Read More
AWS ECS Task Memory Hard and Soft Limits

I'm confused about the purpose of having both hard and soft memory limits for ECS task definitions. IIRC the soft limit is how much memory the scheduler reserves on an instance for the task to run, and the hard limit …

Read More
How do I determine the size of my array in C?

How do I determine the size of my array in C? That is, the number of elements the array can hold?

Read More