Resetting GPU and driver after CUDA error
Sometimes, bugs in my CUDA programs cause the desktop graphics to break (in Windows). Typically, the screen remains somewhat readable, but when graphics change, such as when dragging a window, lots of semi-random colored pixels and small blocks appear.
I have tried to reset the GPU and driver by changing the desktop resolution, but that doesn't help. The only fix I have found is to reboot the computer.
Is there a program out there or some trick I can use to get the driver and GPU to reset without rebooting?
Background:
I have had 1.0, 1.1, 1.3 and 2.0 cards but I only have a 1.1 and 2.0 card now. I've seen the issue on 1.0 and 1.1. I'm pretty sure I've seen it on 1.3. I'm unsure about 2.0. Did memory protection get added some time around 1.3? I am almost sure it's not due to unstable hardware as the problems have seemed to be triggered by bugs in my code and have disappeared when the bugs were fixed. When running finished code, the cards have been stable. I wrote this question after seeing it on my 1.1 card, but it disappeared after I fixed a bug and now I don't have any code that reproduces it. Maybe I should try to write to random locations on the 1.1 card and see if anything happens...
Answer
Edit:
If you are on Tesla hardware on Linux and can run nvidia-smi, then you can reset the GPU using
nvidia-smi -r
or
nvidia-smi --gpu-reset
Here is the man output for this switch:
Resets GPU state. Can be used to clear double bit ECC errors or recover hung GPU. Requires -i switch to target specific device. Available on Linux only.
Otherwise...
The way to truly reset the hardware is to reboot.
What you describe shouldn't happen. I recommend testing with different hardware and let us know if it still occurs.