nvidia-smi process hangs and can't be killed with SIGKILL either

linux linux-device-driver nvidia
bio picture bio · Jan 5, 2017 · Viewed 10.2k times · Source

I'm on Ubuntu 14.04, CUDA toolkit 8, driver version 367.48.

When I give nvidia-smi command, it just hangs indefinitely. When I login again and try to kill that nvidia-smi process, with kill -9 <PID> for example, it just isn't killed. If I give another nvidia-smi command, I find both the processes running - of course when logging from another shell, because that gets stuck as before.

Can it be an issue related to the driver? It's not the latest, but still quite new..

Answer

lurscher picture lurscher · May 19, 2018

I solved this problem by doing at every boot

sudo nvidia-smi -pm 1

The above command enables persistence mode. This issue has been affecting nvidia drivers for over two years but they don't seem interested in fixing it. It seems to be related with a power management issue, after a bit of booting into the OS, if the nvidia-persistenced service has the no-persistence-mode option enabled, the GPU will save power, and the nvidia-smi command will hang waiting for something giving it control again on the device

Related questions

"FATAL: Module not found error" using modprobe

I have a problem with modprobe command... I compiled the hello world module and loaded it with insmod, it works fine and when I do lsmod, I can see it in the output list. But when I insert this module …

Read More
IOCTL Linux device driver

Can anyone explain me, What is IOCTL? What is it used for? How can I use it? Why can't I define new function that does the same work as IOCTL?

Read More
why is u8 u16 u32 u64 used instead of unsigned int in kernel programming

I see u8 u16 u32 u64 data types being used in kernel code. And I am wondering why is there need to use u8 or u16 or u32 or u64 and not unsigned int?

Read More