Ed: Created page with "This page provides information on how to address NVIDIA driver issues under Ubuntu 18.04. However, the objective is not to provide a step-by-step set of instructions to addres..."

2019-07-17T16:35:48Z

Created page with "This page provides information on how to address NVIDIA driver issues under Ubuntu 18.04. However, the objective is not to provide a step-by-step set of instructions to addres..."

New page

This page provides information on how to address NVIDIA driver issues under Ubuntu 18.04. However, the objective is not to provide a step-by-step set of instructions to address one particular issue. Instead, it lays out the general process and commonly-used command line tools to debug video (and other) driver issues.

==The Boot Process==

The following things happen at boot:
*BIOS enumerates/checks hardware -- make sure Secure Boot is disabled!
*The MBR (if you have one) contains GRUB, which is called. We shouldn't be using UEFI if we want unsigned NVIDIA drivers to work.
**Edit /etc/default/grub and run update-grub to alter /boot/grub/grub.cfg (note that there is also /etc/grub.d)
*Then it's the Kernel's turn! Kernel images are in /boot.
**Find which one is being used by running uname -r or doing cat /proc/version or dmesg | grep ubuntu.
*The Kernel loads initramfs first (a mini OS), which is also stored in /boot. It can be updated with update-initramfs.
**initramfs decides which kernel modules are going to be loaded. Use modprobe to alter the list then update-initramfs.
**/etc/modules-load.d/modules.conf can provide a list of modules or can be blank
**lsmod lists loaded modules, depmod -n lists module dependencies, modinfo provides info. Loaded modules are in cat /proc/modules.
*Kernel messages are in /var/log/kern.log (as well as in /var/log/syslog).
**View them with dmesg, cat /var/log/kern.log, and journalctl -b (messages since last boot). Note that cat /proc/kmsg shows nothing and var/log/syslog is a log for rsyslog.
*Init, or rather systemd, then takes over and the machine goes from runlevel to runlevel, eventually bringing up your Xwindowing system, if you have one.
**/etc/rc.local is called last. For later versions of NVIDIA drivers you'll need /usr/bin/nvidia-persistenced --verbose here.

The boot process matters because video drivers are loaded in the kernel phase. If you change which modules are loaded manually (dpkg, apt-get and most scripts will do it for you), you'll need to update-initramfs.

==Useful Commands==

Here's a list of useful commands to diagnose your issues:
vi /etc/default/grub
update-grub
uname -r
cat /proc/version
update-initramfs
cat /var/log/kern.log
dmesg
cat /proc/kmsg
less var/log/syslog
journalctl -b
lsmod | grep video
modinfo asus_wmi
find /lib/modules/$(uname -r) -type f -name '*.ko'shui
modprobe nvidiafb

==Finding hardware==

If your videa card is installed in PCI slot and visible to OS it will show up in:

lspci -vk

and in
lshw -c video

===Example Hardware Check===

Checking that the hardware is being seen on the first build of the [[DIGITS DevBox]]:
lspci -vk

05:00.0 VGA compatible controller: NVIDIA Corporation GP102 [TITAN Xp] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GP102 [TITAN Xp]
Flags: bus master, fast devsel, latency 0, IRQ 78, NUMA node 0
Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at d0000000 (64-bit, prefetchable) [size=32M]
I/O ports at d000 [size=128]
Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024
Capabilities: [900] #19
Kernel driver in use: nouveau
Kernel modules: nvidiafb, nouveau

06:00.0 VGA compatible controller: NVIDIA Corporation Device 1e02 (rev a1) (prog -if 00 [VGA controller])
Subsystem: NVIDIA Corporation Device 12a3
Flags: fast devsel, IRQ 24, NUMA node 0
Memory at f8000000 (32-bit, non-prefetchable) [size=16M]
Memory at a0000000 (64-bit, prefetchable) [size=256M]
Memory at b0000000 (64-bit, prefetchable) [size=32M]
I/O ports at c000 [size=128]
Expansion ROM at f9000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [258] L1 PM Substates
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024
Capabilities: [900] #19
Capabilities: [bb0] #15
Kernel modules: nvidiafb, nouveau

This looks good. The second card is the Titan RTX (see https://devicehunt.com/view/type/pci/vendor/10DE/device/1E02).

==Altering GRUB==

If you didn't use the HWE version of Ubuntu, you might get the dreaded screen freeze or black screen on boot when the video drivers are loaded. Fix that by booting to a terminal and doing:
vi /etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=2
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="nvidia-drm.modeset=1"
GRUB_CMDLINE_LINUX=""

update-grub
Sourcing file `/etc/default/grub'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.18.0-25-generic
Found initrd image: /boot/initrd.img-4.18.0-25-generic
Found linux image: /boot/vmlinuz-4.18.0-20-generic
Found initrd image: /boot/initrd.img-4.18.0-20-generic
Found linux image: /boot/vmlinuz-4.18.0-18-generic
Found initrd image: /boot/initrd.img-4.18.0-18-generic
Found linux image: /boot/vmlinuz-4.15.0-54-generic
Found initrd image: /boot/initrd.img-4.15.0-54-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
device-mapper: reload ioctl on osprober-linux-nvme0n1p1 failed: Device or resource busy
Command failed
done

See https://askubuntu.com/questions/1048274/ubuntu-18-04-stopped-working-with-nvidia-drivers

==Secure Boot==

Don't use UEFI and don't use Secure Boot, unless you can use only signed production level drivers (which isn't going to happen). To check that aren't using secure boot, in /boot:

grep CONFIG_MODULE_SIG_ALL config-4.18.0-25-generic
CONFIG_MODULE_SIG_ALL=y
grep CONFIG_MODULE_SIG_FORCE config-4.18.0-25-generic
# CONFIG_MODULE_SIG_FORCE is not set

Check the Kernel log to see if an unsigned module is tainting the kernel and if that's an issue
vi /var/log/kern.log

==Drivers==

Remember that Tensorflow uses CUDA, which in turn uses the video driver, so you should fix things in order!

If you want to install the latest driver from the ppm, you might need to update your repo list. Current repos are in
/etc/apt/sources.list.d/

If Launchpad ppm was already added as a repo then you can:
cat /etc/apt/sources.list.d/graphics-drivers-ubuntu-ppa-bionic.list

See drivers from all of your repos with:
ubuntu-drivers devices

===Module Diagnostics===

This section contains some notes from the depth of my problems:

lspci -vk shows Kernel modules: nvidiafb, nouveau and no Kernel driver in use.

It looks like nouveau is still blacklisted in /etc/modprobe.d/blacklist-nouveau.conf and /usr/bin/nvidia-persistenced --verbose is still being called in /etc/rc.local. ubuntu-drivers devices returns exactly what it did before we installed CUDA 10.1 too...

There is no /proc/driver/nvidia folder, and therefore no /proc/driver/nvidia/version file found. We get the following:
/usr/bin/nvidia-persistenced --verbose
nvidia-persistenced failed to initialize. Check syslog for more details.
tail /var/log/syslog
...Jul 9 13:35:56 bastard kernel: [ 5314.526960] pcieport 0000:00:02.0: [12] Replay Timer Timeout
...Jul 9 13:35:56 bastard nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 0 has read and write permissions for those files.
ls /dev/
...reveals no nvidia devices

nvidia-smi
...NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

grep nvidia /etc/modprobe.d/* /lib/modprobe.d/*
.../etc/modprobe.d/blacklist-framebuffer.conf:blacklist nvidiafb
.../etc/modprobe.d/nvidia-installer-disable-nouveau.conf:# generated by nvidia-installer

modprobe --resolve-alias nvidiafb
modinfo $(modprobe --resolve-alias nvidiafb)

lsof +D /usr/lib/xorg/modules/drivers/
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
Xorg 2488 root mem REG 8,49 23624 26346422 /usr/lib/xorg/modules/drivers/fbdev_drv.so
Xorg 2488 root mem REG 8,49 90360 26347089
/usr/lib/xorg/modules/drivers/modesetting_drv.so
Xorg 2488 root mem REG 8,49 217104 26346424 /usr/lib/xorg/modules/drivers/nouveau_drv.so
Xorg 2488 root mem REG 8,49 7813904 26346043 /usr/lib/xorg/modules/drivers/nvidia_drv.so

===Determine and Set Driver===

You can use the application NVIDIA prime:
apt-get install nvidia-prime
prime-select nvidia
Info: the nvidia profile is already set

==Paths==

You can also use ldconfig to add the LD Library Path:
vi /etc/ld.so.conf.d/cuda.conf
/usr/local/cuda-10.0/lib64
ldconfig

Export paths by setting the global environment
vi /etc/environment
PATH="/home/researcher/anaconda3/bin:/home/researcher/anaconda3/condabin:/usr/local/cuda-10.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
LD_LIBRARY_PATH="/usr/local/cuda-10.0/lib64"

Now there is no need to activate a custom environment! Alternatively:
export PATH=/usr/local/cuda-10.0/bin:/usr/local/cuda-10.0${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
export PYTHONPATH=/your/tensorflow/path:$PYTHONPATH

==Persistenced==

For CUDA 10.1, we need nvidia-persistenced to be run at boot, so:
vi /etc/rc.local
#!/bin/sh -e
/usr/bin/nvidia-persistenced --verbose
exit 0
chmod +x /etc/rc.local

For CUDA 10.0, it runs as a service and is set up and launched by the script.
/usr/bin/nvidia-persistenced --verbose
nvidia-persistenced failed to initialize. Check syslog for more details.

Jul 11 21:08:20 bastard sshd[3708]: Did not receive identification string from 94.190.53.14 port 3135
Jul 11 21:10:25 bastard nvidia-persistenced[3714]: Verbose syslog connection opened
Jul 11 21:10:25 bastard nvidia-persistenced[3714]: Directory /var/run/nvidia-persistenced will not be removed on exit
Jul 11 21:10:25 bastard nvidia-persistenced[3714]: Failed to lock PID file: Resource temporarily unavailable
Jul 11 21:10:25 bastard nvidia-persistenced[3714]: Shutdown (3714)

To check it is already running:
ps aux | grep persistenced
nvidia-+ 2183 0.0 0.0 17324 1552 ? Ss 10:09 0:00 /usr/bin/nvidia-persistenced --user nvidia-persistenced --no-persistence-mode --verbose
root 3539 0.0 0.0 14428 1000 pts/0 S+ 10:10 0:00 grep --color=auto persistenced

And to see it is a service:
systemctl list-units --type service --all | grep nvidia
nvidia-persistenced.service loaded active running NVIDIA Persistence Daemon

==XWindows==

To diagnose XWindow issues:
cat /var/log/Xorg.0.log
] (II) LoadModule: "nvidia"
[ 29.047] (II) Loading /usr/lib/xorg/modules/drivers/nvidia_drv.so
[ 29.047] (II) Module nvidia: vendor="NVIDIA Corporation"
[ 29.047] compiled for 4.0.2, module version = 1.0.0
[ 29.047] Module class: X.Org Video Driver
[ 29.047] (II) NVIDIA dlloader X Driver 410.48 Thu Sep 6 06:27:34 CDT 2018
[ 29.047] (II) NVIDIA Unified Driver for all Supported NVIDIA GPUs
[ 29.047] (II) Loading sub module "fb"
[ 29.047] (II) LoadModule: "fb"
[ 29.047] (II) Loading /usr/lib/xorg/modules/libfb.so
[ 29.047] (II) Module fb: vendor="X.Org Foundation"
[ 29.047] compiled for 1.19.6, module version = 1.0.0
[ 29.047] ABI class: X.Org ANSI C Emulation, version 0.4
[ 29.047] (II) Loading sub module "wfb"
[ 29.047] (II) LoadModule: "wfb"
[ 29.047] (II) Loading /usr/lib/xorg/modules/libwfb.so
[ 29.048] (II) Module wfb: vendor="X.Org Foundation"
[ 29.048] compiled for 1.19.6, module version = 1.0.0
[ 29.048] ABI class: X.Org ANSI C Emulation, version 0.4
[ 29.048] (II) Loading sub module "ramdac"
[ 29.048] (II) LoadModule: "ramdac"
[ 29.048] (II) Module "ramdac" already built-in
[ 29.095] (EE) NVIDIA: Failed to initialize the NVIDIA kernel module. Please see the
[ 29.095] (EE) NVIDIA: system's kernel log for additional error messages and
[ 29.095] (EE) NVIDIA: consult the NVIDIA README for details.
[ 29.095] (EE) No devices detected.

To use nvidia settings, you have to be on the box. Doing it over SSH will get you:
nvidia-settings --query FlatpanelNativeResolution
Unable to init server: Could not connect: Connection refused

==Tensorflow in Conda==

It turns out that tensorflow 1.13.1 doesn't work with CUDA 10.1! But there is a work around, which is to install cuda10 in conda only (see https://github.com/tensorflow/tensorflow/issues/26182). We are also going to leave the installation of CUDA 10.1 because tensorflow will catch up at some point.

Still as your user account (and in the venv):
conda install cudatoolkit
conda install cudnn
conda install tensorflow-gpu
export LD_LIBRARY_PATH=/home/researcher/anaconda3/lib/${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
python -c "import tensorflow as tf; tf.enable_eager_execution(); print(tf.reduce_sum(tf.random_normal([1000, 1000])))"

==Remove Everything==

Sometimes the best options is to completely remove everything and start again.

To see what is currently installed (your output may look different):
dpkg -l|grep nvidia
ii libnvidia-container-tools 1.0.2-1 amd64 NVIDIA container runtime library
(command-line tools)
ii libnvidia-container1:amd64 1.0.2-1 amd64 NVIDIA container runtime library
ii nvidia-container-runtime 2.0.0+docker18.09.7-3 amd64 NVIDIA container runtime
ii nvidia-container-runtime-hook 1.4.0-1 amd64 NVIDIA container runtime hook
ii nvidia-docker2 2.0.3+docker18.09.7-3 all nvidia-docker CLI wrapper
ii nvidia-prime 0.8.8.2 all Tools to enable NVIDIA's Prime
ii nvidia-settings 418.56-0ubuntu0~gpu18.04.1 amd64 Tool for configuring the NVIDIA
graphics driver

To completely remove all things nvidia (sub the correct version number and don't run the bits after the hash marks):
cd /usr/local/cuda-10.0/bin
./uninstall_cuda_10.0.pl
cd /usr/local
rm -r cuda-10.1/
nvidia-uninstall
apt-get purge nvidia* #Removed nvidia-prime, nvidia-settings, nvidia-container-runtime, nvidia-container-runtime-hook, nvidia-docker2
apt-get purge *nvidia* #Removed libnvidia-container-tools (1.0.2-1), Removing libnvidia-container1:amd64
apt autoremove #Removed libllvm7 libvdpau1 libxnvctrl0 linux-headers-4.18.0-18 linux-headers-4.18.0-18-generic linux-image-4.18.0-18-generic linux-modules-4.18.0-18-generic linux-modules-extra-4.18.0-18-generic mesa-vdpau-drivers pkg-config screen-resolution-extra vdpau-driver-all
cd /home/ed
rm -r NVIDIA_CUDA-10.1_Samples/
rm -r NVIDIA_CUDA-10.0_Samples/
dpkg --remove libcudnn7
dpkg --remove libcudnn7-dev
dpkg --remove libcudnn7-doc

COMMENT OUT LINES IN rc.local
vi /etc/rc.local

shutdown -r now

Addressing Ubuntu NVIDIA Issues - Revision history

Ed: Created page with "This page provides information on how to address NVIDIA driver issues under Ubuntu 18.04. However, the objective is not to provide a step-by-step set of instructions to addres..."