Unpredictable CUDNN_STATUS_NOT_INITIALIZED on Windows

windows tensorflow keras cudnn
Dims picture Dims · Jul 11, 2017 · Viewed 13.3k times · Source

I am running keras neural network training and prediction on GTX 1070 on Windows 10. Most times it is working, but from time to time it complains

E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_dnn.cc:359] could not create cudnn handle: CUDNN_STATUS_NOT_INITIALIZED
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_dnn.cc:366] error retrieving driver version: Unimplemented: kernel reported driver version not implemented on Windows
E c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\stream_executor\cuda\cuda_dnn.cc:326] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
F c:\tf_jenkins\home\workspace\release-win\device\gpu\os\windows\tensorflow\core\kernels\conv_ops.cc:659] Check failed: stream->parent()->GetConvolveAlgorithms(&algorithms)

It cannot be explained neither by literally error meaning nor by OOM error.

How to fix?

Answer

elf picture elf · Aug 1, 2017

Try limiting your gpu usage with set gpu option per_process_gpu_memory_fraction.

Fiddle around with it to see what works and what doesn't.

I recommend using .7 as a starting baseline.

Related questions

Installed Keras with pip3, but getting the "No Module Named keras" error

I am Creating a leaf Identification Classifier using the CNN, the Keras and the Tensorflow backends on Windows. I have installed Anaconda, Tensorflow, numpy, scipy and keras. I installed keras using pip3: C:\> pip3 list | grep -i keras Keras 2.2.4 …

Read More
Installing tensorflow with anaconda in windows

I have installed Anaconda on Windows 64 bit. I have downloaded PyCharm for creating a project and in the terminal of PyCharm I have installed numpy, scipy, matplotlib using the following commands: conda install numpy conda install scipy conda install matplotlib …

Read More
TensorFlow on Windows: "Couldn't open CUDA library cudnn64_5.dll"

Tensorflow just released windows support. I installed the gpu version and CUDA 8.0 and python 3.5. However, after I import the tensorflow I got the following error: >>> import tensorflow I c:\tf_jenkins\home\workspace\release-win\device\gpu\os\…

Read More