WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu is not available (error: cuda unavailable)

Question

in Ubuntu MATE 16.04 I'm trying to run the deep-learning python examples here using the GPU:

testing Theano with GPU

I did run the example code,

THEANO_FLAGS=mode=FAST_RUN,device=gpu,floatX=float32 python check1.py

but it seems that it is used the CPU and not the GPU. Here is the last part of terminal output:

WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu0 is not  available  (error: cuda unavailable)
...
Used the cpu

I tried to run this code too:

THEANO_FLAGS=device=cuda0 python check1.py

but the output is:

ERROR (theano.sandbox.gpuarray): pygpu was configured but could not be imported
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/gpuarray/__init__.py", line 20, in <module>
    import pygpu
ImportError: No module named pygpu
...
used cpu

I installed the cuda toolkit from apt. Here there are (hopefully) useful data:

python --version
Python 2.7.12

g++ -v
gcc version 5.4.0

nvcc --version
Cuda compilation tools, release 7.5, V7.5.17

lspci
NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2)

nvidia-smi

+------------------------------------------------------+                       
| NVIDIA-SMI 361.42     Driver Version: 361.42         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 750 Ti  Off  | 0000:01:00.0      On |                  N/A |
| 29%   35C    P8     1W /  38W |    100MiB /  2044MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      2861    G   /usr/lib/xorg/Xorg                              90MiB |
+-----------------------------------------------------------------------------+

Answer 1

Finally I solved! This post Ubuntu 16.04, Theano and Cuda

suggests to add flag

nvcc.flags=-D_FORCE_INLINES 

to command line, so the command line becomes:

THEANO_FLAGS=floatX=float32,device=gpu,nvcc.flags=-D_FORCE_INLINES python check1.py

It seems to fix a bug in using glibc 2.23

fix for glibc 2.23

Now the program uses correctly the GPU, this is the correct output:

THEANO_FLAGS=floatX=float32,device=gpu,nvcc.flags=-D_FORCE_INLINES python check1.py
Using gpu device 0: GeForce GTX 750 Ti (CNMeM is disabled, cuDNN not available)
[GpuElemwise{exp,no_inplace}(<CudaNdarrayType(float32, vector)>), HostFromGpu(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 0.317012 seconds
Result is [ 1.23178029  1.61879349  1.52278066 ...,  2.20771813  2.29967761
  1.62323296]
Used the gpu

Note that before trying this solution, I removed nvidia-cuda-toolkit and installed CUDA from Nvidia website, following part of instructions found here:

CUDA with Ubuntu 16.04

This is what exactly I did:

1) I downloaded CUDA from here CUDA 7.5 download selecting LINUX, x86_64, Ubuntu 15.04, deb local

2) I installed the deb file

dpkg -i cuda_repo-ubuntu1504-7-5-local_7.5-18_amd64.deb

3) Then run

apt-get update

This gives some errors! I fixed it overwriting the file Release in \var\cuda-repo-7.5-local with the following lines:

Origin: NVIDIA
Label: NVIDIA CUDA
Architecture: repogenstagetemp
MD5Sum:
 51483bc34577facd49f0fbc8c396aea0            75379 Packages
 4ef963dfa4276be01db8e7bf7d8a4f12            21448 Packages.gz
SHA256:
 532b1bb3b392b9083de4445dab2639b36865d7df1f610aeef8961a3c6f304d8a            75379 Packages
 2e48cc13b6cc5856c9c6f628c6fe8088ef62ed664e9e0046fc72819269f7432c            21448 Packages.gz

(sorry, I do not remember where I read this solution).

4) I succesfully run

apt-get-update
apt-get install cuda

5) Everything was insatlled in \usr\local\cuda-7.5

6) I commented the line n 115 in file \usr\local\cuda-7.5\include\host-config.h

 #if __GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ > 9)

//#error -- unsupported GNU version! gcc versions later than 4.9 are not supported!

#endif /* __GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ > 9) */

which seems to prevent CUDA from using gcc 5.4 After all these operations, I updated the .theanorc file, adding the cuda root

[cuda] 
root = /usr/local/cuda-7.5 

That's all :)

PS: I do not know if it would work even with nvidia-cuda-toolkit!