Changes

<onlyinclude>[[File:Top1000.jpg|right|300px]] Our [[DIGITS DevBox]], affectionately named "Bastard"after Lois McMaster Bujold's fifth God, has a XEON e5-2620v3 processor, 256Gb 256GB of DDR4 RAM, two GPUS GPUs - one Titan RTX and one Titan XPXp - with room for two more, a 500Gb 500GB SSD hard drive (mounting /), and an 8TB RAID5 array bcached with a 512Gb 512GB m.2 drive (mounting the /bulk share (, which is available over samba). It runs Ubuntu 18.04, CUDA 10.1 (and CUDA 10 under conda)0, cuDNN 7.56.1, AAnaconda3Anaconda3-2019.03, python 3.7, tensorflow 1.13, digits 6, and other useful machine learning tools/libraries.</onlyinclude>

[[File:Front1000.jpg|right|300px]] We opted to use a Xeon e5-2620v3 processor, rather than the Core i7-5930K. We had both available and both support 40 channels and , mount in the LGA 2011-v3 socket, and have 6 cores, 15mb caches, etc. Although the i7 has a faster clock speed but , the Xeon takes registered (buffered), ECC DDR4 RDIMMs, which means we can put 256Gb on the board, rather than just 64Gb. For the GPUs , we have a TITAN RTX and an older TITAN Xp available to start, and we can add a 1080Ti later, or buy some additional GPUs if needed. We also put the whole thing in a Rosewill RSV-L4000 case.

[[File:Back1000.jpg|right|300px]] There weren't any particularly noteworthy things about the hardware build. The GPUs need to go in slots 1 and 3, which means they sit tight on each other. We put the Titan XP Xp in slot 1 (and plugged the monitor into its HDMI port), because then the fans for the Titan RTX (which we expect will get heavier use) are in the clearfor now. The case fans were set up on in a push -and -pull arrangement, and the hot-swap bay was put in the center position to allow as much airflow past the GPUs as possible.

*Set the three hard disks to hotswapable hot-swap enable

Install [http://cdimage.ubuntu.com/releases/18.04.2/release/?_ga=2.30548799.1041204444.1558044875-2114387110.1558044875 Ubuntu 18.04 ] (note that the original DiGIT DevBox ran 14.04), '''not the live version''', from a freshly burnt DVD. If you install the HWE version, you don't need to run apt-get install --install-recommends linux-generic-hwe-18.04 at the end.

Under partitions:[[File:Partitions1000.jpg|right|300px]]

Check that the hardware is being seen: lspci -vk 05:00.0 VGA compatible controller: NVIDIA Corporation GP102 [TITAN Xp] (rev a1) (prog-if 00 [VGA controller]) Subsystem: NVIDIA Corporation GP102 [TITAN Xp] Flags: bus master, fast devsel, latency 0, IRQ 78, NUMA node 0 Memory at fa000000 (32-bit, non-prefetchable) [size=16M] Memory at c0000000 (64-bit, prefetchable) [size=256M] Memory at d0000000 (64-bit, prefetchable) [size=32M] I/O ports at d000 [sizeHardware and Drivers=128] Expansion ROM at 000c0000 [disabled] [size=128K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Capabilities: [250] Latency Tolerance Reporting Capabilities: [128] Power Budgeting <?> Capabilities: [420] Advanced Error Reporting Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 Capabilities: [900] #19 Kernel driver in use: nouveau Kernel modules: nvidiafb, nouveau 06:00.0 VGA compatible controller: NVIDIA Corporation Device 1e02 (rev a1) (prog -if 00 [VGA controller]) Subsystem: NVIDIA Corporation Device 12a3 Flags: fast devsel, IRQ 24, NUMA node 0 Memory at f8000000 (32-bit, non-prefetchable) [size=16M] Memory at a0000000 (64-bit, prefetchable) [size=256M] Memory at b0000000 (64-bit, prefetchable) [size=32M] I/O ports at c000 [size=128] Expansion ROM at f9000000 [disabled] [size=512K] Capabilities: [60] Power Management version 3 Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [78] Express Legacy Endpoint, MSI 00 Capabilities: [100] Virtual Channel Capabilities: [250] Latency Tolerance Reporting Capabilities: [258] L1 PM Substates Capabilities: [128] Power Budgeting <?> Capabilities: [420] Advanced Error Reporting Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 Capabilities: [900] #19 Capabilities: [bb0] #15 Kernel modules: nvidiafb, nouveau

This looks good. The second card Check the hardware is the Titan RTX (see httpsbeing seen and what driver is being used with://devicehunt.com/view/type/pci/vendor/10DE/device/1E02). lspci -vk

You could install the driver directly now using, say, apt install nvidia-430. But don't! ====CUDA==== Get CUDA 10.1 and have it install its preferred driver (418.67): *The installation instructions are here: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html*You can down load CUDA from here: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=runfilelocal Essentially, first install build-essential, which gets you gcc. Then blacklist the nouveau driver (see https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#runfile-nouveau) and reboot (to a text terminal, if you have deviated from these instructions and already installed X Windows) so that it isn't loaded.

Install the driver!  apt install nvidia-driver-430 ====CUDA==== Get CUDA 10.0, rather than 10.1. Although 10.1 is the latest version at the time of writing, it won't work with Tensorflow 1.13, so you'll just end up installing 10.0 under conda anyway. *The installation instructions are here: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html*You can down load CUDA 10.0 from here: https://developer.nvidia.com/cuda-10.0-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=runfilelocalEssentially, first install build-essential, which gets you gcc.  Then run the installer script.and DO NOT install the driver (don't worry about the warning, it will work fine!): sh cuda_10.10.168_418130_410.67_linux48_linux.run  Do you accept the previously read EULA? accept/decline/quit: accept Install NVIDIA Accelerated Graphics Driver for Linux-x86_64 410.48? (y)es/(n)o/(q)uit: n Install the CUDA 10.0 Toolkit? (y)es/(n)o/(q)uit: y Enter Toolkit Location [ default is /usr/local/cuda-10.0 ]: Do you want to install a symbolic link at /usr/local/cuda? (y)es/(n)o/(q)uit: y Install the CUDA 10.0 Samples? (y)es/(n)o/(q)uit: y Enter CUDA Samples Location [ default is /home/ed ]: Installing the CUDA Toolkit in /usr/local/cuda-10.0 ... Missing recommended library: libGLU.so Missing recommended library: libX11.so Missing recommended library: libXi.so Missing recommended library: libXmu.so Missing recommended library: libGL.so Installing the CUDA Samples in /home/ed ... Copying samples to /home/ed/NVIDIA_CUDA-10.0_Samples now... Finished copying samples. =========== = Summary = =========== Driver: Not Selected Toolkit: Installed in /usr/local/cuda-10.0 Samples: Installed in /home/ed, but missing recommended libraries

Driver: Installed To uninstall the CUDA Toolkit: Installed , run the uninstall script in /usr/local/cuda-10.1/ Samples: Installed in /home0/ed/, but missing recommended librariesbin

Please make sure that - PATH includes /usr/local/cuda-10see CUDA_Installation_Guide_Linux.1/bin - LD_LIBRARY_PATH includes pdf in /usr/local/cuda-10.10/lib64, or, add doc/usr/local/cuda-10.1/lib64 to /etc/ld.sopdf for detailed information on setting up CUDA.conf and run ldconfig as root

***WARNING: Incomplete installation! This installation did not install the CUDA Driver. A driver of version at least 384.00 is required To uninstall the for CUDA Toolkit, run cuda-uninstaller in /usr/local/cuda-10.1/bin0 functionality to work. To uninstall install the NVIDIA Driverdriver using this installer, run nvidiathe following command, replacing <CudaInstaller> with the name of this run file: sudo <CudaInstaller>.run -silent -uninstalldriver

Please see CUDA_Installation_Guide_Linux.pdf in /usr/local/cuda-10.1/doc/pdf for detailed information on setting up CUDA. Logfile is /var/logtmp/cuda-installercuda_install_2807.log

Fix Now fix the paths. To do this for a single user do: export PATH=/usr/local/cuda-10.10/bin:/usr/local/cuda-10.1/NsightCompute-2019.10${PATH:+:${PATH}} export LD_LIBRARY_PATH=/usr/local/cuda-10.10/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

Start But it is better to fix it for everyone by editing your environment file: vi /etc/environment PATH="/usr/local/cuda-10.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games" LD_LIBRARY_PATH="/usr/local/cuda-10.0/lib64" With version cuda 10.0, you don't need to edit rc.local to start the persistence daemon:

This should be run at bootInstead, so: vi /etc/rc.local #!/bin/sh -e /usr/bin/nvidia-persistenced --verbose exit 0 chmod +x /etc/rcruns as a service.local Verify the driver: cat /proc/driver/nvidia/version

Make the samples in:... cd /usr/local/cuda-10.10/samples

Change into the sample directory and run the tests: cd /usr/local/cuda-10.10/samples/bin/x86_64/linux/release

And yes, it's a thing of beautyEverything should be good at this point! ===X Windows=== Now install the X window system. The easiest way is: tasksel And choose your favorite. We used Ubuntu Desktop. And reboot again to make sure that everything is working nicely.

usermod -G smbusers researcher //#Note that this sets the group and will overwrite sudo or other group assignments, so don't do it with your main account. Instead just: useradd ed smbusers

Then follow https://github.com/NVIDIA/nvidia-docker#quick-start to install docker2, but change the last command to use cuda 10.10

docker run --runtime=nvidia --rm nvidia/cuda:10.10-base nvidia-smi

And open a browser to http://localhost:5000/ to see DIGITS. 

dpkg -i libcudnn7_7.56.1.1034-1+cuda10.1_amd640_amd64.deb dpkg -i libcudnn7-dev_7.56.1.1034-1+cuda10.1_amd640_amd64.deb dpkg -i libcudnn7-doc_7.56.1.1034-1+cuda10.1_amd640_amd64.deb

Now install Anaconda, so that we have python 3, and can pip and conda install everything elsethings. Instructions for installing Anaconda on Ubuntu 18.04LTS (e.g., https://docs.anaconda.com/anaconda/install/linux/) all recommend using the shell script.

And this doesn't work. It turns out that tensorflow 1.13.1 doesn't work with CUDA 10.1! But there is a work around, which is Note: to install cuda10 in conda only (see https://github.com/tensorflow/tensorflow/issues/26182). Still as researcher (and in deactivate the venv)virtual environment: conda install cudatoolkit conda install cudnn conda install tensorflow-gpu export LD_LIBRARY_PATH=/home/researcher/anaconda3/lib/${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}} python -c "import tensorflow as tf; tf.enable_eager_execution(); print(tf.reduce_sum(tf.random_normal([1000, 1000])))" AND IT WORKS!deactivate

Note: that adding the anaconda path to deactivate /etc/environment makes the virtual environment: deactivate redundant.