Python GPU programming - Revision history

Ed: /* Other options */

2020-11-30T22:17:56Z

Other options

Ed: /* Other options */

2020-11-30T22:09:05Z

Other options

Ed: /* Other options */

2020-11-30T17:56:07Z

Other options

Ed at 04:19, 20 November 2020

2020-11-20T04:19:52Z

Ed at 20:29, 19 November 2020

2020-11-19T20:29:18Z

Ed: /* Compiling a Kernel */

2020-11-13T20:26:02Z

Compiling a Kernel

Ed: /* Compiling a Kernel */

2020-11-13T20:14:27Z

Compiling a Kernel

Ed: /* Compiling a Kernel */

2020-11-13T20:09:33Z

Compiling a Kernel

Ed at 17:38, 13 November 2020

2020-11-13T17:38:01Z

Ed at 17:37, 13 November 2020

2020-11-13T17:37:49Z

@@ Line 89: / Line 89: @@
 ===Other options===
-There's also [https://pypi.org/project/pyopencl/ PyOpenCL] (another wrapper but of OpenCL instead), [https://pytorch.org/ PyTorch] (a tensor library with GPU support, and likely some other things that I have forgotten about.
+There's also [https://pypi.org/project/pyopencl/ PyOpenCL] (another wrapper but of OpenCL instead), [https://pytorch.org/ PyTorch] (a tensor library that provides a replacement for Numpy for working with GPUs), [https://github.com/Theano/Theano Theono], [https://pypi.org/project/Hebel/ Hebel] (a deep-learning library on top of PyCuda and numpy), [https://caffe2.ai/docs/getting-started.html Caffe2] (which uses CuDNN C++ libraries to provide a deep learning framework) and likely some other things that I have forgotten about.
 ==Other things==

@@ Line 89: / Line 89: @@
 ===Other options===
-There's also PyOpenCL (another wrapper but of OpenCL instead), PyTorch, and likely some other things that I have forgotten about.
+There's also [https://pypi.org/project/pyopencl/ PyOpenCL] (another wrapper but of OpenCL instead), [https://pytorch.org/ PyTorch] (a tensor library with GPU support, and likely some other things that I have forgotten about.
 ==Other things==

@@ Line 89: / Line 89: @@
 ===Other options===
-There's also PyOpenCL (another wrapper but of OpenCL instead), and likely some other things that I have forgotten about.
+There's also PyOpenCL (another wrapper but of OpenCL instead), PyTorch, and likely some other things that I have forgotten about.
 ==Other things==

← Older revision		Revision as of 04:19, 20 November 2020
Line 54:		Line 54:

	==CUDA and Python==		==CUDA and Python==
		+
		+	There's lots of different Ways to use GPUs with Python. Here's a partial list:
		+
		+	===Tensorflow===
		+
		+	Tensors are matrices with a uniform type. Read all about them and tensorflow here: https://www.tensorflow.org/guide/. The highlights are:
		+	*Use it as a library in python, importing methods.
		+	*It plays well with NumPy!
		+	*It sits on top of Keras, allowing custom ANN development.
		+	*Tensorflow has both low level methods (e.g., tf.variable, tf.math, tf.GradientTape.jacobian, etc.) and pre-built estimators (e.g., tf.estimator.LinearClassifier) .
		+	*Crucially, tensorflow lets users build graphs and tf.functions that exist and persist beyond the python interpreter (analogous to kernels)
		+	*You can define custom models and layers for machine learning
		+
		+	===CuPy===
		+
		+	CuPy is a NumPy-compatible array library (it's really a 'drop-in' replacement) accelerated by CUDA: https://cupy.dev/.
		+	*It leverages CUDA-related libraries including cuBLAS, cuDNN, cuRand, cuSolver, cuSPARSE, cuFFT and NCCL
		+	*Lets users build elementwise or reduction kernels, or raw kernels that are defined using raw CUDA source
		+
		+	===Numba===
		+
		+	Numba translates python functions to optimized machine code at runtime: https://numba.pydata.org/
		+	*Define threads, grids, and blocks and manage them with 'facilities like those exposed by CUDA C'.
		+	*Doesn't work with NumPy because of memory management issues.
		+
		+	===PyCUDA===
		+
		+	PyCUDA is a wrapper of CUDA's API for python: https://wiki.tiker.net/PyCuda/
		+	*Essentially gives you access to CUDA's methods, and handles memory allocation and cleanup.
		+	*Doesn't work with CUBLAS. PyCuda is based on the CUDA driver API, which is mutually exclusive with the CUDA runtime API (used by CUBLAS).
		+	*Does reimplement a part of CUBLAS as GPUArray.
		+	*Works with NumPy.
		+
		+	===Other options===
		+
		+	There's also PyOpenCL (another wrapper but of OpenCL instead), and likely some other things that I have forgotten about.
		+
		+	==Other things==
		+
		+	The NVIDIA CUDA Toolkit 7.5 is available for free on Amazon Linux: https://aws.amazon.com/marketplace/pp/B01LZMLK1K
		+
		+	When developing CUDA code you'll want (both are installed as a part of the CUDA Software Development Kit):
		+	*Nvidia Nsight Eclipse Edition for C/C++
		+	*Nvidia Visual Profiler
		+
		+	There's a (probably pretty bad) book which has chapters on working with CUDA in python: https://subscription.packtpub.com/book/big_data_and_business_intelligence/9781789341072/6/ch06lvl1sec55/configuring-pycuda-on-your-python-ide
		+
		+	===IDEs===
		+
		+	You'll likely want to write your python in an IDE that supports your CUDA development. It looks like PyCharm CE will work with an environment:
		+	https://medium.com/@ashkan.abbasi/quick-guide-for-installing-python-tensorflow-and-pycharm-on-windows-ed99ddd9598. But if we use a docker container, then we'll need the PyCharm PE or Visual Studio (which should work fine with Python):
		+	https://www.analyticsvidhya.com/blog/2020/08/docker-based-python-development-with-cuda-support-on-pycharm-and-or-visual-studio-code/. Visual Studio looks like it allows remote development, using either docker containers or ssh, which might be very nice!
		+	https://devblogs.microsoft.com/python/remote-python-development-in-visual-studio-code/ I'm not sure whether PyCharm CE has this feature, though PE does: https://www.jetbrains.com/help/idea/configuring-remote-python-sdks.html.
		+
		+	===Matlab===
		+
		+	If you want to use Matlab then Garland's GPU workshop is a good place to start: http://www.quantosanalytics.org/calpoly/gpu_workshop/index.html

@@ Line 1: / Line 1: @@
-There are many different ways to work with GPUs using Python. This page explores them!
+There are many different ways to work with GPUs using Python. This page provides a discussion of the foundations behind working with GPUs, from the fundamental choice between CUDA and OpenCL to what it means to compile a kernel. It then covers the dominant approaches for using CUDA with python.
 ==Foundations==
@@ Line 5: / Line 5: @@
 ===CUDA vs. OpenCL===
-At a fundamental level, using a GPU for computing means using [[https://en.wikipedia.org/wiki/CUDA CUDA]], [[https://en.wikipedia.org/wiki/OpenCL OpenCL]], or some other interface (OpenGL compute, Microsoft's DirectCompute, etc.) The big trade-off between CUDA and OpenCL is proprietary performance vs. open-source generality. Usually, I favour the later. However, at this point, the nVIDIA chipsets dominate the market and CUDA (which only runs on nVIDIA) seems to be the obvious choice. There have also been some attempts to make CUDA run on CL.
+At a fundamental level, using a GPU for computing means using [https://en.wikipedia.org/wiki/CUDA CUDA], [https://en.wikipedia.org/wiki/OpenCL OpenCL], or some other interface (OpenGL compute, Microsoft's DirectCompute, etc.) The big trade-off between CUDA and OpenCL is proprietary performance vs. open-source generality. Usually, I favour the later. However, at this point, the nVIDIA chipsets dominate the market and CUDA (which only runs on nVIDIA) seems to be the obvious choice. There have also been some attempts to make CUDA run on CL.
 ===CUDA for C++ or Fortran===

@@ Line 40: / Line 40: @@
 The key things that you need to know are:
-* One kernel is executed at a time on a device
+* One '''kernel''' is executed at a time on a device
-* Many threads execute each kernel - each thread runs the same code but on different data (based on its threadID)
+* Many '''threads''' execute each kernel - each thread runs the same code but on different data (based on its threadID)
-* Threads are grouped into blocks and a kernel runs on a grid of blocks
+* Threads are grouped into '''blocks''' and a kernel runs on a '''grid''' of blocks
 * Blocks can't synchronize. They can run concurrently or sequentially.
-* Threads have local memory (registers ~ 1clock cycle), blocks share memory (~10 clock cycles), and kernels have per-device global memory(~100s/1000 clock cycles)
+* Threads have local memory ('''registers''' ~ 1clock cycle), '''blocks share memory''' (~10 clock cycles), and kernels have '''per-device global memory''' (~100s/1000 clock cycles)
-* Per device memory can transfer data to/from the CPU, and includes global, local (for consecutive access by a thread), constant (much faster than other per device), and some specialized memories for graphics (texture and surface).
+* Per device memory can transfer data to/from the CPU, and includes '''global''', '''local''' (for consecutive access by a thread), '''constant''' (much faster than other per device), and some specialized memories for graphics ('''texture''' and surface).
 * Transfers from global memory to local registers is in 4,8 or 16 byte units (or can incur a penalty, which slows things down). Threads can talk to constant and texture memory.
-* Blocks should have dimension >=32
+* Blocks should have dimension >=32 (see warps below).
-* A GPU device is a set of SIMT multiprocessors
+* A GPU device is a set of '''[https://en.wikipedia.org/wiki/Single_instruction,_multiple_threads SIMT multiprocessor]'''
-* At each clock cycle, a multiprocessor executes the same instruction on a warp (the number of threads in a warp is the "warp size". It's usually 32. You can find yours by running the deviceQuery utility provided in the samples folder. See [[DIGITS DevBox#Test the installation]].
+* The number of threads in a '''warp''' is the "warp size". It's usually 32. You can find yours by running the deviceQuery utility provided in the samples folder. See [[DIGITS DevBox#Test the installation]]. Warps are then grouped into blocks.
 ==CUDA and Python==

← Older revision		Revision as of 20:09, 13 November 2020
Line 32:		Line 32:
	===Compiling a Kernel===		===Compiling a Kernel===

		+	In the language of GPU computing, we need to compile a [https://en.wikipedia.org/wiki/Compute_kernel kernel] to run on the GPU. Some packages (discussed later) abstract away how GPUs handle memory and processing, but you should be aware of the fundamentals as they are often very important to maximizing the code's performance: if you understand the hardware implementation, you can tune for it!
		+
		+	[https://scholars.duke.edu/person/cliburn.chan Chi Wei Cliburn Chan], an associate prof of Biostatistics and Bioinformatics at Duke, teaches [https://people.duke.edu/~ccc14/ lots of great classes], and provides a guide to [https://people.duke.edu/~ccc14/sta-663/CUDAPython.html Massively parallel programming with GPUs] as a part of his [https://people.duke.edu/~ccc14/sta-663/ Computational Statistics in Python] class (note that the 2018 version of his [http://people.duke.edu/~ccc14/sta-663-2018/ STA 663: Computational Statistics and Statistical Computing (2018)] class (under the same course number) has sections on Spark, Tensorflow, Cython, and more!). This guide has a pretty good walk-through of how a CUDA kernel runs, though it is missing some images. See also:
		+	*The streaming multiprocessor and the CUDA core: https://i.stack.imgur.com/kvu4M.jpg
		+	*CUDA memory hierarchy: https://www.researchgate.net/profile/Marco_Nobile/publication/261069154/figure/fig1/AS:296718735298563@1447754667270/Schematization-of-CUDA-architecture-Schematic-representation-of-CUDA-threads-and-memory.png
		+	*Various slides from Cyril Zeller (nVIDIA Developer Technology)'s Tutorial CUDA:https://www.slideshare.net/angelamm2012/nvidia-cuda-tutorialnondaapr08
		+
		+	The key things that you need to know are:
		+	* One kernel is executed at a time on a device
		+	* Many threads execute each kernel - each thread runs the same code but on different data (based on its threadID)
		+	* Threads are grouped into blocks and a kernel runs on a grid of blocks
		+	* Blocks can't synchronize. They can run concurrently or sequentially.
		+	* Threads have local memory (registers ~ 1clock cycle), blocks share memory (~10 clock cycles), and kernels have per-device global memory(~100s/1000 clock cycles)
		+	* Per device memory can transfer data to/from the CPU, and includes global, local (for consecutive access by a thread), constant (much faster than other per device), and some specialized memories for graphics (texture and surface).
		+	* Transfers from global memory to local registers is in 4,8 or 16 byte units (or can incur a penalty, which slows things down). Threads can talk to constant and texture memory.
		+	* Blocks should have dimension >=32
		+	* A GPU device is a set of SIMT multiprocessors
		+	* At each clock cycle, a multiprocessor executes the same instruction on a warp (the number of threads in a warp is the "warp size".

	==CUDA and Python==		==CUDA and Python==

@@ Line 28: / Line 28: @@
 For the second option, you'll need to use [http://www.swig.org/ SWIG (Simplified Wrapper and Interface Generator)] or something that gives you equivalent functionality for your language, such as [https://cython.org/ Cython] for Python. ([https://github.com/rmcgibbo/npcuda-example NPCUDA] is a simple project to demo both of these methods in Python.) The major advantage of this option is that you aren't hitching your horse to the continued support of a whole series of intermediate APIs.
-{{Colored box|Title=Notice for Perl Programmers|Content=A quick look at the GPU support for Perl suggests that SWIG the way to go!}}
+{{Colored box|title=Notice for Perl Programmers|content=A quick look at the GPU support for Perl suggests that SWIG the way to go!}}
 ===Compiling a Kernel===