Tensorflow CUDA - CUPTI error: CUPTI could not be loaded or symbol could not be found

Question

I use the Tensorflow v 1.14.0. I work on Windows 10. And here is how relevant environment variables look in the PATH:

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\libnvvp
C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common
C:\Users\sinthes\AppData\Local\Programs\Python\Python37
C:\Users\sinthes\AppData\Local\Programs\Python\Python37\Scripts
C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\cuda\bin

Maybe also worth to mention, just in case it might be relevant.. I use Sublime Text 3 for development and I do not use Anaconda. I find it a bit cumbersome to make updates on tensorflow in the conda environment so I just use Sublime Text right now. (I was using Anaconda (Spyder) previously but I uninstalled it from my computer.)

Things seem to work fine except with some occasional strange warnings. But one consistent warning I get is the following whenever I run the fit function.

E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI could not be loaded or symbol could not be found.

And here is how I call the fit function:

history = model.fit(x=train_x,
                    y=train_y,
                    batch_size=BATCH_SIZE,
                    epochs=110,
                    verbose=2,
                    callbacks=[tensorboard, checkpoint, reduce_lr_on_plateau],
                    validation_data=(dev_x, dev_y),
                    shuffle=True,
                    class_weight=class_weight,
                    steps_per_epoch=None,
                    validation_steps=None)

I just wonder why I see the CUPTI Error message during the run time? It is only printed out once. Is that something that I need to fix or is it something that can be ignored? This message does not tell anything concrete to me to be able to take any action.

Answer 1

Add this in path for Windows:

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\extras\CUPTI\libx64