Rapids / docker: could not select device driver "" with capabilities: [[gpu]]
I'm new to Rapids, and rarely have had a good experience with conda. So I'm trying to work with a containerized version. I'm new to Docker, and the combination of unknowns leaves me unable to sort things out.
I have an Ubuntu 18.04 server,
# uname -v
#30~18.04.1-Ubuntu SMP Fri Jan 17 06:14:09 UTC 2020
on which I installed a fresh version of Docker
# apt-get install docker docker-ce docker-ce-cli containerd.io
# docker --version
Docker version 19.03.8, build afacb8b7f0
This machine has cuda v10.2 installed
# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89
and Python v3.6.9
# python3 --version
Python 3.6.9
As shown in the NVIDIA Container Toolkit Quickstart section, I install the nvidia-docker list to /etc/apt/sources.list.d/
# curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
# curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu18.04/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
explicitly substituting ubuntu18.04 for $distribution, since that is the Ubuntu equivalent for Linux Mint 19.3.
Following the Start Container and Notebook Server instructions in RAPIDS - Open GPU Data Science, I pulled the 0.13-cuda10.2-runtime-ubuntu18.04-py3.6 runtime.
# docker pull rapidsai/rapidsai:0.13-cuda10.2-runtime-ubuntu18.04-py3.6
A long time, and several GB later, all seemed to be OK. (No warnings or error messages.) Furthermore, it looks like the image was registered with Docker.
# docker images -a
REPOSITORY TAG IMAGE ID CREATED SIZE
rapidsai/rapidsai 0.13-cuda10.2-runtime-ubuntu18.04-py3.6 c7440af853b5 4 days ago 9.26GB
rapidsai/rapidsai cuda10.2-runtime-ubuntu18.04-py3.6 c7440af853b5 4 days ago 9.26GB
However, I next tried to start up the notebook server:
# docker run --gpus all --rm -it -p 8888:8888 -p 8787:8787 -p 8786:8786 \
rapidsai/rapidsai:cuda10.0-runtime-ubuntu18.04-py3.6
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
This seems surprising, as there are two GTX 1080 Ti GPUs detected
# nvidia-smi
Fri May 8 16:41:57 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01 Driver Version: 440.33.01 CUDA Version: 10.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... On | 00000000:08:00.0 Off | N/A |
| 21% 38C P8 10W / 250W | 1MiB / 11178MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... On | 00000000:42:00.0 Off | N/A |
| 23% 42C P8 10W / 250W | 1MiB / 11177MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
After cleaning things up
# docker system prune -a
# apt-get purge docker docker-engine docker.io containerd runc
I re-installed docker and pulled the rapidsai image again. The result was unchanged.
Is there a conflict with the NVIDIA Driver Version: 440.33.01?
Any suggestions?
Answer
Thanks for trying out RAPIDS. Did you happen to install nvidia-container-toolkit? https://github.com/NVIDIA/nvidia-docker#quickstart. I didn't see that in your steps and missing it could cause that issue. It's in our prerequisites on https://rapids.ai/start.html