Changes

Jump to navigation Jump to search
2,260 bytes added ,  13:39, 21 September 2020
no edit summary
{{Project|Has project output=Content|Has sponsor=McNair ProjectsCenter
|Has title=GPU Build
|Has owner=Oliver Chang,Kyran Adams
}}
==Final Decision==
 
We decided to clone the NVIDIA [[DIGITS DevBox]]: https://developer.nvidia.com/devbox
 
To start with we are trying to use our existing ASUS Z10 server board, rather than switching to the Asus X99-E WS workstation class motherboard, and rather than Four TITAN X GPUs, we've got a TITAN XP and a TITAN RTX.
 
Note that the Asus X99-E WS is available from NewEgg for $500 now.
==Single vs. Multi GPU==
 
*[https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ GTX 1080 Ti Specs]
* Since we are using Tensorflow, it doesn't scale well to multiple GPUs for a single model
# If the network doesn't fit in the memory of one GPU (11 GB),
* [https://devtalk.nvidia.com/default/topic/743814/cuda-setup-and-installation/advice-on-single-vs-multi-gpu-system/ Advice on single vs multi-GPU system]
# Want to get two graphics cards, one for development, one (crappy or onboard card) for operating system [https://stackoverflow.com/questions/21911560/how-can-i-set-one-nvidia-graphics-card-for-display-and-other-for-computingin-li]
*[https://stackoverflow.com/questions/37732196/tensorflow-difference-between-multi-gpus-and-distributed-tensorflow Different uses of multiple GPUs]
# Intra-model parallelism: If a model has long, independent computation paths, then you can split the model across multiple GPUs and have each compute a part of it. This requires careful understanding of the model and the computational dependencies.
*Adds a lot of complexity.
=== K80, NVLink ===*NVLink can link between CPU and GPU for increase in speed, but only with the CPU IBM POWER8+.*NVLink can link between GPU and GPU as a replacement for SLI with other CPUs, but this is not super relevant to tensorflow, even if trying to parallelize across one model.*[https://www.quora.com/Which-GPU-is-better-for-Deep-Learning-GTX-1080-or-Tesla-K80 This source] says to get the 1080 because the K80 is basically two K40s, which have less memory bandwidth than the 1080. [https://www.reddit.com/r/deeplearning/comments/5mc7s6/performance_difference_between_nvidia_k80_and_gtx/ This source] agrees.
==Misc. Parts==
*Cases: Rosewill 1.0 mm Thickness 4U Rackmount Server Chassis, Black Metal/Steel RSV-L4000[https://www.amazon.com/gp/product/B0056OUTBK/ref=oh_aui_detailpage_o04_s00?ie=UTF8&psc=1]
*Consider this case: Corsair Carbide Series Air 540 High Airflow ATX Cube Case [https://www.amazon.com/dp/B00D6GINF4/ref=twister_B00JRYFVAO?_encoding=UTF8&psc=1]
*DVDRW (Needed?): Asus 24x DVD-RW Serial-ATA Internal OEM Optical Drive DRW-24B1ST [http://www.amazon.com/Asus-Serial-ATA-Internal-Optical-DRW-24B1ST/dp/B0033Z2BAQ/ref=sr_1_2?s=pc&ie=UTF8&qid=1452399113&sr=1-2&keywords=dvdrw]
*Keyboard and Mouse: AmazonBasics Wired Keyboard and Wired Mouse Bundle Pack [http://www.amazon.com/AmazonBasics-Wired-Keyboard-Mouse-Bundle/dp/B00B7GV802/ref=sr_1_2?s=pc&rps=1&ie=UTF8&qid=1452402108&sr=1-2&keywords=keyboard+and+mouse&refinements=p_72%3A1248879011%2Cp_85%3A2470955011]
* [https://www.slideshare.net/PetteriTeikariPhD/deep-learning-workstation Deep Learning Computer Build] useful tips, long
* [https://www.tooploox.com/blog/deep-learning-with-gpu Another box]
* [http://graphific.github.io/posts/building-a-deep-learning-dream-machine/ Expensive deep learning box]
==Double GPU Server Build==
[https://pcpartpicker.com/user/kyranadams/saved/gDzFdC PC Partpicker build]
Questions to ask*[https: * Approx//www.quora. datasetcom/batch size* Network card?Can-I-double-the-PCIe-lanes-in-a-dual-CPU-motherboard This article] says that it may be necessary to get both CPUs to get all of the PCI lanes
==Double GPU Build==
===Motherboard===
*Should have Needs enough PCIe slotsto support both GPUs and other units*Motherboards: MSI - Z170A GAMING M7 ATX LGA1151 Motherboard [https://www.newegg.com/Product/Product.aspx?Item=9SIA85V4SC7911&nm_mc=AFC-C8Junction&cm_mmc=AFC-C8Junction-PCPartPicker,%20LLC-_-na-_-na-_-na&cm_sp=&AID=10446076&PID=3938566&SID=], LGA 1151, 3x PCIe 3.0 x 16, 4 x PCIe 3.0 x 1, 6 x SATA 6GB/s, also used in [https://medium.com/@SocraticDatum/getting-started-with-gpu-driven-deep-learning-part-1-building-a-machine-d24a3ed1ab1e this build]
===CPU/Fan===
*At least as much RAM as GPUs (2 * 11 GB [GTX 1080 Ti size] = 22 GB, so 32GB)
*Does not have to be fast for deep learning: "CPU-RAM-to-GPU-RAM is the true bottleneck – this step makes use of direct memory access (DMA). As quoted above, the memory bandwidth for my RAM modules are 51.2GB/s, but the DMA bandwidth is only 12GB/s!"[http://timdettmers.com/2015/03/09/deep-learning-hardware-guide/]
* Crucial - 32GB (2 x 16GB) DDR4-2133 Memory [https://www.newegg.com/Product/Product.aspx?Item=9SIA8PV5HF1514&nm_mc=AFC-C8Junction&cm_mmc=AFC-C8Junction-PCPartPicker,%20LLC-_-na-_-na-_-na&cm_sp=&AID=10446076&PID=3938566&SID=], SATA 6 GB/s interface* If not enough, should be able to extend this by buying two more cards
===PSU===
*Some say PSU should be 1.5x-2x wattage of system, some say wattage+100W
*I picked a rating such that another GPU or other extension could be added without a new PSU*PSU: Rosewill EVGA - 1200W SuperNOVA G2 1000W 80+ Platinum Gold Certified Fully-Modular ATX Power Supply [https://www.newegg.com/Product/Product.aspx?Item=9SIA0723W13835N82E16817438010&ignorebbr=1&nm_mc=AFC-C8Junction&cm_mmc=AFC-C8Junction-PCPartPicker,%20LLC-_-na-_-na-_-na&cm_sp=&AID=10446076&PID=3938566&SID=]
===Storage===
*SSD should be fast enough, no need for M.2 [http://timdettmers.com/2015/03/09/deep-learning-hardware-guide]
*SSD: Samsung - 850 EVO-Series 500GB 2.5" Solid State Drive [https://www.newegg.com/Product/Product.aspx?Item=N82E16820147373&ignorebbr=1&nm_mc=AFC-C8Junction&cm_mmc=AFC-C8Junction-PCPartPicker,%20LLC-_-na-_-na-_-na&cm_sp=&AID=10446076&PID=3938566&SID=]
*HDD: Seagate - Barracuda 3TB 3.5" 7200RPM Internal Hard Drive [https://www.newegg.com/Product/Product.aspx?Item=9SIADG25GT7889&nm_mc=AFC-C8Junction&cm_mmc=AFC-C8Junction-PCPartPicker,%20LLC-_-na-_-na-_-na&cm_sp=&AID=10446076&PID=3938566&SID=]
 
===Other things to consider===
* Water cooling? [http://timdettmers.com/2015/03/09/deep-learning-hardware-guide/ this] has a good section on cooling
* Case is not rack mounted
 
==Software tips==
* Setting up Ubuntu and Docker [https://medium.com/@SocraticDatum/getting-started-with-gpu-driven-deep-learning-part-2-environment-setup-fd1947aab29]

Navigation menu