On this page
Every answer an AI chatbot gives you is produced by arithmetic: billions of multiplications and additions. AI companies buy graphics chips by the thousand because this arithmetic has a special shape. Once you see it, you can estimate the computing bill yourself with Class 12 maths.
What does matrix multiplication have to do with AI?
A neural network stores what it has learned as numbers called parameters, arranged in grids called matrices. (If the words are new, our guide to AI terms like tokens and parameters covers them.) Your question is turned into lists of numbers, one list per token, and each layer of the network multiplies those lists by its matrices before passing the result on.
The diagram at the top shows one small multiplication. To get the highlighted answer, take row 2 of A and column 3 of B, multiply pair by pair, and add: 3×3 + 1×1 + 2×2 = 14. Multiplying an m×n matrix by an n×p matrix takes m×n×p of these multiply-and-add steps, which is why the order rule for multiplying matrices matters.
No answer cell waits for any other, so all nine can be worked out at the same moment by nine different workers.
For language models, a 2020 paper by Jared Kaplan and colleagues estimates that one forward pass costs about 2N operations per token, where N is the number of parameters, "where the factor of two comes from the multiply-accumulate operation used in matrix multiplication" (Kaplan et al., Scaling Laws for Neural Language Models).
Why does AI use GPUs instead of CPUs?
A CPU, the main processor in your laptop, has a few powerful cores built to run varied tasks one after another. A GPU was built to colour millions of screen pixels, each independent of the next, which is the same shape of problem as filling in the cells of a matrix.
The SXM5 version of NVIDIA's H100 GPU has 16,896 FP32 cores and 528 Tensor Cores, which NVIDIA describes as cores "for matrix multiply and accumulate (MMA) math operations" (NVIDIA Hopper architecture in depth).
A famous early example came in 2012. Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton trained an image classifier with 60 million parameters on two GTX 580 GPUs, which took five to six days; they wrote that the network's size was "limited mainly by the amount of memory available on current GPUs" (Krizhevsky et al., NeurIPS 2012). That network is a classic case of deep learning, the branch explained in AI vs machine learning vs deep learning.
Why does training AI take thousands of GPUs?
Training runs the forward pass, then a backward pass that works out how to nudge every parameter, so Kaplan and colleagues estimate training compute at about 6N operations per training token (Kaplan et al., 2020).
Here is the estimate for Meta's Llama 3, using numbers from its technical report (The Llama 3 Herd of Models, 2024):
Parameters N = 405 billion; training text D = 15.6 trillion tokens.
Compute ≈ 6 × N × D = 6 × 4.05×10¹¹ × 1.56×10¹³ ≈ 3.8×10²⁵ operations. This matches the report's 3.8×10²⁵ FLOPs.
The report measured each H100 at 380 to 430 trillion operations per second during training; take 4×10¹⁴.
One GPU: 3.8×10²⁵ ÷ 4×10¹⁴ ≈ 9.5×10¹⁰ seconds, about 3,000 years.
16,384 GPUs, the largest setup in the report: 9.5×10¹⁰ ÷ 16,384 ≈ 5.8×10⁶ seconds, about 67 days of nonstop computing in this idealised estimate.
Memory points the same way. Stored as 16-bit numbers (2 bytes each), 405 billion parameters fill about 810 GB, while each H100 in Meta's cluster had 80 GB, so the model must be split across many GPUs.
Why do bigger models cost more to run?
Once trained, every token a model writes still costs about 2N operations (Kaplan et al., 2020). Llama 3 comes in 8-billion, 70-billion and 405-billion-parameter sizes (Llama 3 report). The 8-billion model needs about 16 billion operations per token; the 405-billion model needs about 810 billion, roughly 50 times more, for every token.
That is one reason companies offer small, fast models alongside large ones, and why a long answer costs more than a short one.
Try it yourself: estimate any model in five minutes
Find N (parameters) and D (training tokens) in the model's paper.
Training compute ≈ 6 × N × D.
Divide by (operations per second per GPU × number of GPUs) to get seconds; divide by 86,400 for days.
Cost to run ≈ 2 × N operations per token.
Practice on GPT-3, which has 175 billion parameters and was trained on 300 billion tokens (Brown et al., Language Models are Few-Shot Learners). You should get about 3.15×10²³ operations, roughly 3,600 petaflop/s-days, the tallest bar in the paper's Figure 2.2.
If matrices or powers of ten make step 2 feel shaky, the Fermi Concept Library has a lesson for every concept in mathematics and computing. Fermi's AI tutor can work out what you already know and build a plan for the session, so you sit down to a clear next step.
