Difference between revisions of "GPU Build"
Jump to navigation
Jump to search
| Line 1: | Line 1: | ||
{{McNair Projects | {{McNair Projects | ||
|Has title=GPU Build | |Has title=GPU Build | ||
| − | |Has owner=Oliver Chang, | + | |Has owner=Oliver Chang,Kyran Adams |
| − | |Has project status= | + | |Has project status=Active |
}} | }} | ||
| − | |||
| − | * https://news.ycombinator.com/item?id=14438472 (links to https://blog.slavv.com/the-1700-great-deep-learning-box-assembly-setup-and-benchmarks-148c5ebe6415) | + | ===Single GPU vs Multi GPU=== |
| − | + | *[https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ GTX 1080 Ti Specs] | |
| + | * [http://timdettmers.com/2017/04/09/which-gpu-for-deep-learning/ Which GPU for deep learning (04/09/2017)] | ||
| + | # "I quickly found that it is not only very difficult to parallelize neural networks on multiple GPUs efficiently, but also that the speedup was only mediocre for dense neural networks. Small neural networks could be parallelized rather efficiently using data parallelism, but larger neural networks... received almost no speedup." | ||
| + | # Possible other use of multiple GPUs: training multiple different models simultaneously, "very useful for researchers, who want try multiple versions of a new algorithm at the same time." | ||
| + | # This source recommends GTX 1080 Tis and does cost analysis of it | ||
| + | # If the network doesn't fit in the memory of one GPU (11 GB), | ||
| + | * [https://devtalk.nvidia.com/default/topic/743814/cuda-setup-and-installation/advice-on-single-vs-multi-gpu-system/ Advice on single vs multi-GPU system] | ||
| + | # Might want to get two graphics cards, one for development, one (crappy card) for operating system | ||
| + | *[https://stackoverflow.com/questions/37732196/tensorflow-difference-between-multi-gpus-and-distributed-tensorflow] | ||
| + | # Different uses of multiple GPUs | ||
| + | # Intra-model parallelism: If a model has long, independent computation paths, then you can split the model across multiple GPUs and have each compute a part of it. This requires careful understanding of the model and the computational dependencies. | ||
| + | # Replicated training: Start up multiple copies of the model, train them, and then synchronize their learning (the gradients applied to their weights & biases). | ||
| + | |||
| + | ===Other Builds=== | ||
| + | |||
| + | * [https://news.ycombinator.com/item?id=14438472 Deep learning box for $1700] (links to https://blog.slavv.com/the-1700-great-deep-learning-box-assembly-setup-and-benchmarks-148c5ebe6415) | ||
Revision as of 15:18, 23 October 2017
| GPU Build | |
|---|---|
| Project Information | |
| Project Title | GPU Build |
| Owner | Oliver Chang, Kyran Adams |
| Start Date | |
| Deadline | |
| Primary Billing | |
| Notes | |
| Has project status | Active |
| Copyright © 2016 edegan.com. All Rights Reserved. | |
Single GPU vs Multi GPU
- "I quickly found that it is not only very difficult to parallelize neural networks on multiple GPUs efficiently, but also that the speedup was only mediocre for dense neural networks. Small neural networks could be parallelized rather efficiently using data parallelism, but larger neural networks... received almost no speedup."
- Possible other use of multiple GPUs: training multiple different models simultaneously, "very useful for researchers, who want try multiple versions of a new algorithm at the same time."
- This source recommends GTX 1080 Tis and does cost analysis of it
- If the network doesn't fit in the memory of one GPU (11 GB),
- Might want to get two graphics cards, one for development, one (crappy card) for operating system
- Different uses of multiple GPUs
- Intra-model parallelism: If a model has long, independent computation paths, then you can split the model across multiple GPUs and have each compute a part of it. This requires careful understanding of the model and the computational dependencies.
- Replicated training: Start up multiple copies of the model, train them, and then synchronize their learning (the gradients applied to their weights & biases).
