“(…) a specialized electronic circuit designed to rapidly manipulate and alter memory to accelerate the creation of images in a frame buffer intended for output to a display device. GPUs are used in embedded systems, mobile phones, personal computers, workstations, and game consoles.” (Wikipedia)
Original purpose: image creation and manipulation for graphical rendering (video games, video edition, 3D conception, etc.)
GPU for computing ?
“Modern GPUs are very efficient at manipulating computer graphics and image processing. Their highly parallel structure makes them more efficient than general-purpose central processing units (CPUs) for algorithms that process large blocks of data in parallel.” (Wikipedia)
Nowadays usage: more general massive computations based on matrix operations
Von Neumann architecture
Basic principle (present in most modern computing units) with three interconnected parts:
control unit: does most of the work, such as deciding what operation to do next, fetching the next instruction from memory, etc.
arithmetic/logic unit (ALU): implements operations such as addition, subtraction, multiplication, etc.
memory unit: stores the data processed by the chip, such as its inputs, outputs, and any intermediate data
Principle: control units are large and expensive, while arithmetic units are more straightforward and cheaper
Design: a sub-processor = one control unity commanding several ALUs (= GPU cores) to operate on larger chunks of data
Limitation: all ALUs (connected to the same control unity) must obey the same commands (i.e. do the same thing at the same moment)
GPU trades “control” for “arithmetic power”
ALU blocks (= gpu sub-processors) not as versatile as CPU cores but can operate over large amounts of data in batches
Central control unit syncs all the sub-processors (each one can do a different task) but the same set of ALUs cannot do different things in parallel (e.g. if statements costly for GPUs)
Example: each block of 16 ALUs is limited to processing the same instruction over 16 pairs of operands
Latency vs Throughput
Latency: how long does it take to finish a given task
Throughput: how many times can you complete the task within a period
CPU cores optimized for latency: to finish a task as fast as possible.
\rightarrow scales with more complex operations but not with larger data
GPU cores optimized for throughput: individually slower, but they operate on bulks of data at once.
\rightarrow scales with larger data but not with more complex operations
Latency vs Throughput: an analogy (I)
Travelling by road:
latency: use a sport car (faster but can only take 2-5 persons)
throughput: use a bus (slower but can take up to 80 persons)
Total time to transport 5 persons over 100km (latency):
1 trip with a 80-seat bus at 50km/h: 2hours
1 trip with a 5-seat car at 200km/h: 0.5hours
Total time to transport 160 persons over 100km (throughput):
2 round trips with a 80-seat bus at 50km/h: 8hours
32 round trips with a 5-seat car at 200km/h: 32hours
AMD Ryzen Threadripper 2990X processor: 32cores (each capable of running two independent threads), 3 GHz up to 4.2 GHz (boost) clock rate, 2 instructions per clock cycle, each thread processes 8 floating-point values at the same time
CPU vs GPU comparison: an example, GPU
Nvidia RTX 2080 TI GPU: 4352 ALUs, 1.35 GHz up to 1.545 GHz (boost) clock rate, 2 operations per clock cycle, one floating point operation per ALU
Performance illustration
Throughput:
Maximum theoretical number of floating-point operations they can handle per second (FLOPS)
