7,612
edits
Changes
Jump to navigation
Jump to search
====Hardware check====The first build of this box was done with an installation of CUDA 10.1, which automatically installed version 418.67 of the NVIDIA driver. We then installed CUDA 10.0 under conda to support Tensorflow 1.13. All went mostly well, and the history of this page contains the instructions. However, at some point, likely because of an OS update, the video driver(s) stopped working. This page now describes the second build (as if it were a build from scratch). [[Addressing Ubuntu NVIDIA Issues]] provides additional information.
Check that the hardware is being seen: lspci -vk 05:00.0 VGA compatible controller: NVIDIA Corporation GP102 [TITAN Xp] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation GP102 [TITAN Xp] Flags: bus master, fast devsel, latency 0, IRQ 78, NUMA node 0 Memory at fa000000 (32-bit, non-prefetchable) [size=16M] Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at d0000000 (64-bit, prefetchable) [size=32M] I/O ports at d000 [sizeHardware and Drivers=128] Expansion ROM at 000c0000 [disabled] [size=128K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Capabilities: [250] Latency Tolerance Reporting Capabilities: [128] Power Budgeting <?> Capabilities: [420] Advanced Error Reporting Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 Capabilities: [900] #19 Kernel driver in use: nouveau Kernel modules: nvidiafb, nouveau 06:00.0 VGA compatible controller: NVIDIA Corporation Device 1e02 (rev a1) (prog -if 00 [VGA controller]) Subsystem: NVIDIA Corporation Device 12a3 Flags: fast devsel, IRQ 24, NUMA node 0 Memory at f8000000 (32-bit, non-prefetchable) [size=16M] Memory at a0000000 (64-bit, prefetchable) [size=256M] Memory at b0000000 (64-bit, prefetchable) [size=32M] I/O ports at c000 [size=128] Expansion ROM at f9000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Capabilities: [250] Latency Tolerance Reporting Capabilities: [258] L1 PM Substates Capabilities: [128] Power Budgeting <?> Capabilities: [420] Advanced Error Reporting Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 Capabilities: [900] #19 Capabilities: [bb0] #15 Kernel modules: nvidiafb, nouveau
This looks good. The second card Check the hardware is the Titan RTX (see httpsbeing seen and what driver is being used with://devicehunt.com/view/type/pci/vendor/10DE/device/1E02). lspci -vk
You could install the driver directly now using, say, apt install nvidia-430. But don't! ====CUDA==== Get CUDA 10.1 and have it install its preferred driver (418.67): *The installation instructions are here: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html*You can down load CUDA from here: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=runfilelocal Essentially, first install build-essential, which gets you gcc. Then blacklist the nouveau driver (see https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#runfile-nouveau) and reboot (to a text terminal, if you have deviated from these instructions and already installed X Windows) so that it isn't loaded.
wget https://developer.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.168_418.67_linux.run
=========== Please make sure that = Summary = - PATH includes /usr/local/cuda-10.0/bin =========== - LD_LIBRARY_PATH includes /usr/local/cuda-10.0/lib64, or, add /usr/local/cuda-10.0/lib64 to /etc/ld.so.conf and run ldconfig as root
Driver: Installed To uninstall the CUDA Toolkit: Installed , run the uninstall script in /usr/local/cuda-10.1/ Samples: Installed in /home0/ed/, but missing recommended librariesbin
Please make sure that - PATH includes /usr/local/cuda-10see CUDA_Installation_Guide_Linux.1/bin - LD_LIBRARY_PATH includes pdf in /usr/local/cuda-10.10/lib64, or, add doc/usr/local/cuda-10.1/lib64 to /etc/ld.sopdf for detailed information on setting up CUDA.conf and run ldconfig as root
Please see CUDA_Installation_Guide_Linux.pdf in /usr/local/cuda-10.1/doc/pdf for detailed information on setting up CUDA. Logfile is /var/logtmp/cuda-installercuda_install_2807.log
Fix Now fix the paths. To do this for a single user do: export PATH=/usr/local/cuda-10.10/bin:/usr/local/cuda-10.1/NsightCompute-2019.10${PATH:+:${PATH}} export LD_LIBRARY_PATH=/usr/local/cuda-10.10/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
Start But it is better to fix it for everyone by editing your environment file: vi /etc/environment PATH="/usr/local/cuda-10.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games" LD_LIBRARY_PATH="/usr/local/cuda-10.0/lib64" With version cuda 10.0, you don't need to edit rc.local to start the persistence daemon:
This should be run at bootInstead, so: vi /etc/rc.local #!/bin/sh -e /usr/bin/nvidia-persistenced --verbose exit 0 chmod +x /etc/rcruns as a service.local Verify the driver: cat /proc/driver/nvidia/version
Change into the sample directory and run the tests: cd /usr/local/cuda-10.10/samples/bin/x86_64/linux/release
And yes, it's a thing of beautyEverything should be good at this point! ===X Windows=== Now install the X window system. The easiest way is: tasksel And choose your favorite. We used Ubuntu Desktop. And reboot again to make sure that everything is working nicely.
And this doesn't work. It turns out that tensorflow 1.13.1 doesn't work with CUDA 10.1! But there is a work around, which is to install cuda10 in conda only (see https://github.com/tensorflow/tensorflow/issues/26182). We are also going to leave the installation of CUDA 10.1 because tensorflow will catch up at some point.
Still as researcher (and in the venv):
conda install cudatoolkit
conda install cudnn
conda install tensorflow-gpu
export LD_LIBRARY_PATH=/home/researcher/anaconda3/lib/${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
python -c "import tensorflow as tf; tf.enable_eager_execution(); print(tf.reduce_sum(tf.random_normal([1000, 1000])))"
AND IT WORKS!
After logging into In order to use the box sometime latergraphical interface for Matlab and other applications, it seems that the video drivers are no longer loading, presumably as we need a consequence of some update or somethingVNC server.
===Testing===First, install the VNC client remotely. We use the standalone exe from TigerVNC.
nvidiaNow install TightVNC, following the instructions: https://www.digitalocean.com/community/tutorials/how-settings to-install-query FlatpanelNativeResolution ERROR: NVIDIA driver is not loadedand-configure-vnc-on-ubuntu-18-04
lspci -vk shows Kernel modules: nvidiafb, nouveau and no Kernel driver in use.Note that changing the color depth breaks it!
It looks like nouveau is still blacklisted in /etc/modprobe.d/blacklistTo make changes (or after the edit) sudo systemctl daemon-nouveaureload sudo systemctl enable vncserver@2.conf and /usr/bin/nvidiaservice vncserver -persistenced --verbose is still being called in /etc/rc.local. ubuntu-drivers deviceskill :2 sudo systemctl start vncserver@2returns exactly what it did before we installed CUDA 10.1 too... sudo systemctl status vncserver@2
There is no /proc/driver/nvidia folder, and therefore no /proc/driver/nvidia/version file found. We get Stop the following: /usr/bin/nvidia-persistenced --verboseserver with nvidia-persistenced failed to initialize. Check syslog for more details. tail /var/log/syslog ...Jul 9 13:35:56 bastard kernel: [ 5314.526960] pcieport 0000:00:02.0: [12] Replay Timer Timeout ...Jul 9 13:35:56 bastard nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 0 has read and write permissions for those files. ls /dev/ ...reveals no nvidia devices nvidia-smi ...NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.sudo systemctl stop vncserver@2
grep nvidia /etc/modprobe.d/* /lib/modprobe.d/* .../etc/modprobe.d/blacklist-framebuffer.confNote that we are using :2 because :blacklist nvidiafb .../etc/modprobe1 is running our regular Xwindows GUI.d/nvidia-installer-disable-nouveau.conf:# generated by nvidia-installer
===UninstallInstrucions on how to set up an IP tunnel using PuTTY: https://helpdeskgeek.com/how-to/tunnel-vnc-over-ssh/Reinstall===
Am going to try uninstalling CUDA 10.1 and the current Nvidia driver, and then reinstalling CUDA 10.0 /usr/local/cuda-10.1/bin/cuda-uninstaller nvidia-uninstall====Connection Issues====
WARNING: Your driver installation has been altered since it was initially installed; Coming back to this may happen, for example, if you have since installed I had issues connecting. I set up the tunnel using the NVIDIA driver through a mechanism other than nvidia-installer (such as your distribution's native package management system)saved profile in puTTY. nvidia-installer will attempt exe and checked to uninstall as best see which local port was listening (it can. Please see was 5901) and not firewalled using the file '/var/log/nvidia-uninstalllistening ports tab under network on resmon.log' for detailsexe (it said allowed, not restricted under firewall status). WARNINGVNC seemed to be running fine on Bastard, and I tried connecting to localhost: Failed to delete some directories. See /var/log/nvidia-uninstall.log for details. Uninstallation of existing driver: NVIDIA Accelerated Graphics Driver for Linux-x86_64 1 (418that is 5901 on the localhost, through the tunnel to 5902 on Bastard) using VNC Connect by RealVNC.67) is completeThe connection was refused.
Then download cuda_10I checked it was listening and there was no firewall: netstat -tlpn tcp 0 0 0.0.130_4100.48_linux.run from https0://developer5902 0.nvidia.com/cuda-100.0-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=runfilelocal, as well as cuda_10.0.130.1_linux.run.:* LISTEN 2025/Xtightvnc ufw status Status: inactive
sudo su cd /bulk/installThe localhost port seems to be open and listening just fine: Test-NetConnection 127./cuda_100.0.130_410.48_linux.run accept all defaults and install everything (including 410.something NVIDIA driver)1 -p 5901
=========== Driver: Installed Toolkit: Installed in /usr/local/cuda-10.0 Samples: Installed in /home/ed, but missing recommended libraries Please make sure that - PATH includes /usr/local/cuda-10.0/bin - LD_LIBRARY_PATH includes /usr/local/cuda-10.0/lib64So, orpresumably, add /usr/local/cuda-10.0/lib64 to /etc/ld.so.conf and run ldconfig as root To uninstall there must be something wrong with the CUDA Toolkit, run the uninstall script in /usr/local/cuda-10.0/bin To uninstall the NVIDIA Driver, run nvidia-uninstall Please see CUDA_Installation_Guide_Linux.pdf in /usr/local/cuda-10.0/doc/pdf for detailed information on setting up CUDA. Logfile is /tmp/cuda_install_8524tunnel itself.log
Fix '''Ignoring the pathsSSH tunnel worked fine: export PATH=/usr/local/cuda-10Connect to 192.0/bin${PATH168.2.202:+:${PATH}} export LD_LIBRARY_PATH=/usr/local/cuda-105902 using the TightVNC (or RealVNC, etc.) client.0/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}'''
Also vi /etc/ld.so.conf.d/cuda.conf /usr/local/cuda-10.0/lib64 ldconfig ===RDP===
FinallyI also installed xrdp: apt install xrdp adduser xrdp ssl-cert #Check the status and that it is listening on 3389 systemctl status xrd netstat -tln #It is listening... vi /etc/xrdp/cuda_10xrdp.0ini #See https://linux.130die.1net/man/5/xrdp.runini accept all defaults systemctl restart xrdp
Unfortunately this didn't work. After This gave a reboot: nvidia-settings --query FlatpanelNativeResolution Unable to init server: Could not connect: Connection refused dead session (message was same as before a flat light blue screen with nothing on the boxit), which finally yielded a connection log which said "login successful for display 10, this was over ssh)start connecting, connection problems, giving up, some problem." cat /var/log/xrdp-sesman.log
There could be some conflict between VNC and RDP./deviceQuery Startingsystemctl status xrdp shows "xrdp_wm_log_msg: connection problem, giving up"... CUDA Device Query (Runtime API) version (CUDART static linking) cudaGetDeviceCount returned 35 -> CUDA driver version is insufficient for CUDA runtime version Result = FAIL
There is clearly an issue with Ubuntu 18.04 and XRDP. The solution seems to be to downgrade xserver-xorg-core and some related packages, which can be done with an install script (https://usrc-nergy.be/binblog/nvidia-persistenced --verbose?p=13933) or manually. But I don't want to do that, so I removed xrdp and went back to VNC! nvidia-persistenced failed to initialize. Check syslog for more details.apt remove xrdp
lspci -vk also returned the same as before. This is really frustrating!===Other Software===
Did I installed the followingcommunity edition of PyCharm: aptsnap install pycharm-get install nvidiacommunity -prime prime-select nvidiaclassic Info: #Restart the nvidia profile is already setlocal terminal so that it has updated paths (after a snap install, etc.) /snap/pycharm-community/214/bin/pycharm.sh
update-initramfs -uOn launch, you get some config options. I chose to install and enable:*IdeaVim (a VI editor emulator)*R*AWS Toolkit
For next time (as root)Make a launcher: In /usr/share/applications: lshw -c videovi pycharm.desktop [Desktop Entry] Version=2020.2.3 Type=Application Name=PyCharm Icon=/snap/pycharm-community/214/bin/pycharm. shows configuration without driverpng Exec="/snap/pycharm-community/214/bin/pycharm.sh" %f Comment=The Drive to Develop Categories=Development;IDE; Terminal=false StartupWMClass=jetbrains-pycharm
modprobe --resolve-alias nvidiafbmodinfo $(modprobe --resolve-alias nvidiafb)Also, create a launcher on the desktop with the same info.
no edit summary
This page details the build of our [[DIGITS DevBox]]. There's also a page giving information on [[Using the DevBox]]. nVIDIA, famous for their incredibly poor supply-chain and inventory management, have been saying [https://developer.nvidia.com/devbo "Please note that we are sold out of our inventory of the DIGITS DevBox, and no new systems are being built"] since shortly after the [https://en.wikipedia.org/wiki/GeForce_10_series Titax X] was the latest and greatest thing (i.e., somewhere around 2016). But it's pretty straight forward to update [https://www.azken.com/download/DIGITS_DEVBOX_DESIGN_GUIDE.pdf their spec].
==Introduction==
===Specification===
<onlyinclude>[[File:Top1000.jpg|right|300px]] Our [[DIGITS DevBox]], affectionately named "Bastard"after Lois McMaster Bujold's fifth God, has a XEON e5-2620v3 processor, 256GB of DDR4 RAM, two GPUs - one Titan RTX and one Titan Xp - with room for two more, a 500GB SSD hard drive (mounting /), and an 8TB RAID5 array bcached with a 512GB m.2 drive (mounting the /bulk share, which is available over samba). It runs Ubuntu 18.04, CUDA 10.1 (and CUDA 10 under conda)0, cuDNN 7.56.1, Anaconda3-2019.03, python 3.7, tensorflow 1.13, digits 6, and other useful machine learning tools/libraries.</onlyinclude>
===Documentation===
Notes:
*We will do RAID 5 array in software, rather using X99 through the BIOS
What's really crucial is that all the hardware is visible and that we are NOT using UEFI. With UEFI, there is an issue with the drivers not being properly signed under secure boot.
==Software==
Give the box a reboot!
===X Windows===
If you install the video driver before installing Xwindows, you will need to manually edit the Xwindows config files. So, now install the X window system. The easiest way is:
tasksel
And choose your favorite. We used Ubuntu Desktop.
And reboot again to make sure that everything is working nicely.
===Video Drivers===
Currently we are using the nouveau driver for the Xp, and have no driver loaded for the RTX.
driver : xserver-xorg-video-nouveau - distro free builtin
apt-get install build-essential
gcc --version
vi /etc/modprobe.d/blacklist-nouveau.conf
blacklist nouveau
update-initramfs -u
shutdown -r now
Reboot to a text terminal
lspci -vk
Shows no kernel driver in use!
Install the driver! apt install nvidia-driver-430 ====CUDA==== Get CUDA 10.0, rather than 10.1. Although 10.1 is the latest version at the time of writing, it won't work with Tensorflow 1.13, so you'll just end up installing 10.0 under conda anyway. *The installation instructions are here: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html*You can down load CUDA 10.0 from here: https://developer.nvidia.com/cuda-10.0-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=runfilelocalEssentially, first install build-essential, which gets you gcc. Then run the installer script.and DO NOT install the driver (don't worry about the warning, it will work fine!): sh cuda_10.10.168_418130_410.67_linux48_linux.run Do you accept the previously read EULA? accept/decline/quit: accept Install NVIDIA Accelerated Graphics Driver for Linux-x86_64 410.48? (y)es/(n)o/(q)uit: n Install the CUDA 10.0 Toolkit? (y)es/(n)o/(q)uit: y Enter Toolkit Location [ default is /usr/local/cuda-10.0 ]: Do you want to install a symbolic link at /usr/local/cuda? (y)es/(n)o/(q)uit: y Install the CUDA 10.0 Samples? (y)es/(n)o/(q)uit: y Enter CUDA Samples Location [ default is /home/ed ]: Installing the CUDA Toolkit in /usr/local/cuda-10.0 ... Missing recommended library: libGLU.so Missing recommended library: libX11.so Missing recommended library: libXi.so Missing recommended library: libXmu.so Missing recommended library: libGL.so Installing the CUDA Samples in /home/ed ... Copying samples to /home/ed/NVIDIA_CUDA-10.0_Samples now... Finished copying samples. =========== = Summary = =========== Driver: Not Selected Toolkit: Installed in /usr/local/cuda-10.0 Samples: Installed in /home/ed, but missing recommended libraries
***WARNING: Incomplete installation! This installation did not install the CUDA Driver. A driver of version at least 384.00 is required To uninstall the for CUDA Toolkit, run cuda-uninstaller in /usr/local/cuda-10.1/bin0 functionality to work. To uninstall install the NVIDIA Driverdriver using this installer, run nvidiathe following command, replacing <CudaInstaller> with the name of this run file: sudo <CudaInstaller>.run -silent -uninstalldriver
/usr/bin/nvidia-persistenced --verbose
====Test the installation====
Make the samples in:... cd /usr/local/cuda-10.10/samples
make
And change into the sample directory and run the tests:
./deviceQuery
./bandwidthTest
===Bcache===
This section follows https://developer.nvidia.com/rdp/digits-download. Install Docker CE first, following https://docs.docker.com/install/linux/docker-ce/ubuntu/
Then follow https://github.com/NVIDIA/nvidia-docker#quick-start to install docker2, but change the last command to use cuda 10.10
...
sudo apt-get install -y nvidia-docker2
sudo pkill -SIGHUP dockerd
# Test nvidia-smi with the latest official CUDA image
docker run --runtime=nvidia --rm nvidia/cuda:10.10-base nvidia-smi
Then pull DIGITS using docker (https://hub.docker.com/r/nvidia/digits/):
*https://developer.nvidia.com/digits
Note: you can kill docker containers with
docker system prune
====cuDNN====
First, make an installs directory in bulk and copy the installation files over from the RDP (E:\installs\DIGITS DevBox). Then:
cd /bulk/install/
dpkg -i libcudnn7_7.56.1.1034-1+cuda10.1_amd640_amd64.deb dpkg -i libcudnn7-dev_7.56.1.1034-1+cuda10.1_amd640_amd64.deb dpkg -i libcudnn7-doc_7.56.1.1034-1+cuda10.1_amd640_amd64.deb
And test it:
pip install --upgrade tensorflow-gpu
python -c "import tensorflow as tf; tf.enable_eager_execution(); print(tf.reduce_sum(tf.random_normal([1000, 1000])))"
Note: to deactivate the virtual environment:
deactivate
Note that adding the anaconda path to /etc/environment makes the virtual environment redundant.
=====PyTorch and SciKit=====
*http://deeplearning.net/software/theano/install_ubuntu.html
==Video Driver Issue=VNC===
cd /usr/local/cuda-10.1/samples/bin/x86_64/linux/releaseroot ./deviceQuery CUDA Device Query (Runtime API) version (CUDART static linking) cudaGetDeviceCount returned 100 apt-> no CUDAget install xfce4 xfce4-capable device is detected Result = FAILgoodies
As user sudo apt-get install tightvncserver ./mnistCUDNNvncserver set password for user cudnnGetVersion() vncserver -kill : 7501 , CUDNN_VERSION from cudnn1 mv ~/.h : 7501 (7vnc/xstartup ~/.5vnc/xstartup.1)bak Cuda failurer version : GCC 7 vi ~/.4vnc/xstartup #!/bin/bash xrdb $HOME/.0Xresources startxfce4 & Error: no CUDA-capable device is detected vncserver error_util sudo vi /etc/systemd/system/vncserver@.h:93service [Unit] Description=Start TightVNC server at startup Aborting. After=syslog.target network.target [Service] Type=forking User=uname Group=uname WorkingDirectory=/home/uname And as researcher PIDFile=/home/ed/.vnc/%H:%i.pid cd ExecStartPre=-/usr/bin/homevncserver -kill :%i > /researcherdev/null 2>&1 source . ExecStart=/venvusr/bin/activatevncserver -depth 24 -geometry 1280x800 :%i python ExecStop=/usr/bin/vncserver -c "import tensorflow as tf; tf.enable_eager_execution(); print(tf.reduce_sum(tf.random_normal(kill :%i [1000, 1000Install])))" ... failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA WantedBy=multi-capable device is detected ..user.kernel driver does not appear to be running on this host (bastard): /proc/driver/nvidia/version does not existtarget
I tried without success: python gsettings set org.gnome.Vino require-encryption false https://askubuntu.com/questions/797973/error-c "import tensorflow problem-connecting-windows-10-rdp-into-xrdp vi /etc/X11/Xwrapper.config allowed_users = anybody This was promising as tf; tfit was previously set to consol.enabl e_eager_execution(); print(tf https://www.reduce_sum(tflinuxquestions.random_normal([1000, 1000])))"org/questions/linux-software-2/xrdp-under-debian-9-connection-problem-4175623357/#post5817508 2019apt-07get install xorgxrdp-09 15:20:40hwe-18.04 Couldn't find the package... This lead was promising as it applies to 18.04.08587702 HWE, which is what I'm running https: E tensorflow/stream_executor/cudawww.nakivo.com/cuda_driver.cc:300 ] failed call blog/how-to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detecte use-remote-desktop-connection-ubuntu-linux-walkthrough/ ddpkg -l |grep xserver-xorg-core ii 2019xserver-07xorg-09 15core 2:20:401.19.085978: I tensorflow/stream_executor/cuda/cuda_diagnostics6-1ubuntu4.c 3 amd64 Xorg X server - core server c Which seems ok, despite having a problem with XRDP and Ubuntu 18.04 HWE documented very clearly here:148] kernel driver does not appear to be running on this host (bastard)http: /proc /driverc-nergy.be/nvidiablog/version does not exist?p=13972
