A Graphics Processing Unit (GPU) is a highly parallel processor originally architected for real-time 3D graphics rendering. Its design — thousands of smaller cores optimized for simultaneous execution of simple arithmetic operations — makes it exceptionally suited for the matrix multiplications and tensor operations that underpin modern deep learning. Unlike a CPU, which excels at sequential, latency-sensitive tasks with a few powerful cores, a GPU trades single-thread performance for massive throughput, executing thousands of threads concurrently.
Technically, a modern GPU (e.g., NVIDIA H100, AMD MI300X) consists of multiple Streaming Multiprocessors (SMs) or Compute Units, each containing a set of CUDA cores (or equivalent), tensor cores, and shared memory. Tensor cores, introduced with NVIDIA's Volta architecture in 2017, are specialized hardware units that perform fused multiply-add operations on small matrices (e.g., 4×4) in a single cycle, accelerating mixed-precision training (FP16, BF16, TF32). The H100, released in 2022, includes Transformer Engine that dynamically manages precision between FP8 and FP16, achieving up to 9× faster training on large language models compared to the A100. For memory, GPUs use high-bandwidth memory (HBM or HBM3) — the H100 SXM has 80 GB at 3.35 TB/s bandwidth — crucial for feeding data to compute units without stalling.
Why it matters: Training state-of-the-art models like GPT-4 (estimated 1.8 trillion parameters) or Llama 3.1 405B would be infeasible on CPUs alone, requiring decades instead of weeks. GPU clusters (thousands of units interconnected via NVLink or InfiniBand) enable distributed training across data centers. For inference, GPUs provide low-latency token generation; for example, a single H100 can serve Llama 2 70B at ~100 tokens/s using quantization (INT4).
When used vs alternatives: GPUs are dominant for dense, compute-bound workloads: CNNs, transformers, diffusion models. For sparse or memory-bound tasks (e.g., graph neural networks with irregular access patterns), CPUs or specialized ASICs may be better. For extremely large-scale training, TPUs (Google's tensor processing units) offer higher throughput for specific matrix sizes but lack flexibility. For edge inference, NPUs (neural processing units) in phones or microcontrollers are more power-efficient. For scientific computing, AMD GPUs with ROCm compete with CUDA ecosystem.
Common pitfalls: (1) Underutilization: poor data loading pipelines cause GPU starvation; solutions include NVIDIA DALI or PyTorch DataLoader with num_workers. (2) Out-of-memory errors: models exceeding VRAM require techniques like gradient checkpointing, model parallelism (e.g., Megatron-LM), or offloading to CPU (Deepspeed Zero-3). (3) Precision mismatch: training in FP32 instead of mixed precision slows throughput 2–4×. (4) Vendor lock-in: CUDA is deeply entrenched; porting to AMD or Intel GPUs requires rewriting kernels in ROCm or oneAPI. (5) Cost: cloud GPU instances (e.g., AWS p4d.24xlarge at $32.77/hr) can quickly drain budgets without proper spot instance or preemptible VM usage.
Current state of the art (2026): NVIDIA's Blackwell architecture (B200) features 208 billion transistors, 192 GB HBM3e, and FP4/FP6 support, enabling real-time inference on models >1 trillion parameters. AMD's MI400 series targets exascale AI with chiplets and unified memory. Open-source software like PyTorch 2.x with torch.compile and Triton kernels close the gap with vendor libraries. The trend is toward heterogeneous computing: CPU-GPU-NPU hybrids, disaggregated memory, and optical interconnects (e.g., NVIDIA NVLink 5 at 1.8 TB/s). Energy efficiency remains critical — liquid cooling and sparse computation (e.g., 2:4 structured sparsity on H100) are standard.