CPU vs GPU vs NPU for AI inference

Marketing for AI hardware leans on one number: TOPS, trillions of operations per second. It is a real measure of something. It is just rarely the thing that decides how quickly a chatbot running on your desk produces its next word.

To see why, you need to split a language model's work into two phases, and then look at what each kind of chip is good at.

Two phases, two bottlenecks

An LLM response has two stages. NVIDIA's technical blog describes them as a prefill phase that processes input tokens in parallel and a decode phase that generates output tokens one at a time, the latter being memory-bound. Prefill chews through your whole prompt at once, which keeps the arithmetic units busy. Decode produces a single token per pass, and each pass needs the model's weights and the cached keys and values. In the blog's words, the speed at which that data is transferred from memory dominates the latency, not how fast the computation happens.

That has a practical consequence you can compute without running anything. If generating one token means streaming, say, 4.7 GiB of weights through the processor, the best you can ever do is the memory bandwidth divided by 4.7 GiB per second. Real systems fall short of that ceiling, and tricks like batching several requests change the picture for servers, but the ceiling for one person chatting is a fair guide to which hardware will feel snappy.

Here is that sum for a quantised 8B model (Qwen3-8B at Q4_K_M, 4.7 GiB of weights, worked out in our VRAM explainer). The bandwidth figures are what each manufacturer quotes. They are peak numbers, the output is a theoretical upper limit and not a benchmark, and no real runtime will reach it.

The script that produced the last column is calc-vram.mjs, shipped with the VRAM article, so you can swap in your own model and bandwidth.

Look at the spread. The H200's bandwidth is some thirty times that of the Mac mini's entry chip, and that gap, rather than a difference in raw arithmetic, is much of what separates a hobby box from a serving rack. It is also why two machines with similar-sounding "AI performance" claims can feel completely different in use.

CPUs: capacity and flexibility

A CPU has a few dozen cores built for varied, branching work, not thousands of simple multiply-add units. It still handles LLM inference respectably when the software is written for it. llama.cpp lists optimised paths for AVX, AVX2, AVX512 and AMX on x86, and ARM NEON on Arm, and its quantisation support from 1.5 to 8 bits cuts the data a CPU has to haul from RAM.

The strengths of a CPU run are simple. It can address all your system RAM, which is far larger than any consumer GPU's VRAM, so it can load models a graphics card cannot. It needs no special driver. It is also what llama.cpp falls back on in hybrid mode, where part of a model sits on the GPU and the rest runs on the CPU.

The weakness is bandwidth. Typically the memory behind a CPU is slower than the memory on a graphics card, so the decode ceiling is lower. Apple silicon is the notable exception: its CPU and GPU share one unified memory system with quoted bandwidths from 153GB/s up to 1.2TB/s. We cover what that means for buying decisions in our mini PC guide.

GPUs: parallel compute plus fast memory

A GPU packs thousands of arithmetic units (NVIDIA lists 21,760 CUDA cores on the GeForce RTX 5090) around a wide memory interface. For prefill, the parallelism helps. For decode, the memory system helps, because it can stream weights quickly. This pairing is why discrete graphics cards, and data-centre parts with stacked high-bandwidth memory, are the default for serious inference.

The catch is capacity. A card's VRAM is fixed and expensive per gigabyte, and a model that does not fit either needs a smaller quantisation, a shorter context, or a hybrid split that gives back the speed advantage. Integrated designs where the GPU shares a large pool of system memory trade some bandwidth for capacity. AMD says its Ryzen AI Max+ platform offers up to 128GB of unified memory with up to 96GB available for graphics, and NVIDIA's DGX Spark pairs 128 GB of coherent memory with a 273 GB/s memory interface. Those are different points on the same capacity-versus-speed curve.

NPUs: efficiency first

A neural processing unit is a fixed-function block tuned for the matrix arithmetic of neural networks at low power. Intel's OpenVINO documentation describes its NPU as a low-power hardware solution, introduced with Core Ultra, that lets you offload certain neural-network tasks from other devices. Apple's chips carry a Neural Engine (16 cores on the M5 Max, 32 on the M5 Ultra in Apple's Mac Studio specifications), and AMD's Ryzen AI Max+ 395 includes an XDNA 2 NPU that AMD rates at up to 50 TOPS.

Two things are worth keeping in mind with those numbers.

First, TOPS is a compute rating, quoted by the manufacturer under conditions it chooses. It says nothing about the memory side that dominates decode. A 50 TOPS NPU and a 50 TOPS GPU are not interchangeable for a chatbot.

Second, NPU ratings are driven partly by a software requirement. Microsoft says many of the new Windows AI features require an NPU able to run at 40+ TOPS to qualify as a Copilot+ PC. That sets a floor for laptop makers, and it is about running Windows features efficiently, not about being the best place to run a 30-billion-parameter chat model.

Where do NPUs help today? On the evidence we could verify, in low-power, always-on and smaller models, which is what the vendor documentation emphasises. Software support is also still catching up for LLMs on NPUs: llama.cpp's backend table lists Hexagon for Snapdragon, CANN for Huawei Ascend NPUs and an OpenVINO backend for Intel CPUs, GPUs and NPUs that is marked In Progress. Check that status before you build a plan around it. We did not find a primary source that gives independent, verified LLM-on-NPU speed comparisons, so we make no claim about them.

Which to use, then?

• Fits in your GPU's memory: use the GPU. It has the bandwidth.
• Doesn't fit, but fits in system RAM: a CPU or a hybrid split will run it, slowly. For occasional use that may be fine.
• Shared-memory machine (Apple silicon, Ryzen AI Max+, DGX Spark class): the GPU side of the chip is normally the one you want for big models, and capacity is the draw.
• NPU: use it where an application already targets it, and treat TOPS as a minor tie-breaker rather than a headline.

What we are not claiming

The ceiling table is arithmetic from published peak bandwidths, not measurement. Real tokens per second depend on the runtime, the quantisation, the context length and thermal limits. The relative ordering of the hardware above is a reasonable guide to who has headroom, but test on the machine you intend to buy.