Grokking Parallel Programming

Scientific simulations, graphics-intensive games, and, of course, AI models, all require the serious computing power delivered by GPUs (Graphics Processing Units). Grokking Parallel Programming teaches you to write parallel GPU code from scratch using CUDA, NVIDIA’s platform for general-purpose GPU computing. Perfect for readers with no experience in parallel programming or CUDA, this book sets out a clear path with well-annotated code, helpful diagrams, and complete, easy-to-understand explanations.

Written by SungHwan Yun, a former Samsung Principal Engineer and competitive programming problem setter, this friendly, fully-illustrated guide takes you from basic definitions through your very first parallel kernel to the memory-optimization techniques that turn correct code into fast code. Chapter-by-chapter, you’ll explore the fundamental patterns of parallel programming from the ground up—all with examples in CUDA that make it very real.

You’ll start with something super simple—doubling the elements of an array in parallel. Then, you’ll see how this core idea leads all the way up to summing millions of numbers with tournament-style reductions, filtering data with parallel prefix sums, and speeding up matrix multiplication with shared-memory tiling. And the examples get more interesting as you go! You’ll see how parallel programming applies to graphics, data visualization, and accelerating AI training and inference.

Each hands-on chapter ends with a section that walks you through the most common bugs—such as race conditions, missing barriers, and out-of-bounds access—so you can easily spot and fix them in your own code. All the examples are grounded in CUDA platform, Nvidia’s platform that lets developers use graphics cards for general math and data work. By the end, you’ll be thinking like a GPU engineer, ready to take on real-world parallel programming tasks!

And if you don’t have access to a GPU, no problem! Grokking Parallel Programming includes full access to a free online sandbox that lets you write CUDA code in your browser, submit it, and get an instant verdict—no hardware required. The practice problems are designed with the rigor of a veteran competitive programming problem setter, each targeting one specific concept with carefully constructed test data.