Inside an LLM: an interactive guide to tokenization, attention, and generation

🇮🇹 Leggi questo articolo in italiano

You ask ChatGPT a question. It answers. In between those two events, something genuinely strange happens, and almost nobody explains it. Your words get chopped up, turned into numbers, multiplied by giant matrices, voted on, and reassembled, one token at a time, until the model decides it is done.

This post is that explanation, and it is interactive. The widget below is a working model of the journey from prompt to answer. Click through it, poke the sliders, then come back up here for the code and the commentary.

The whole thing in eight steps

The visualizer walks you through the actual pipeline a large language model runs every time you hit enter. Not a metaphor. The real stages: prompt templating, tokenization, embeddings, positional encoding, self-attention, stacked transformer layers, next-token prediction, and autoregressive generation. The numbers are faked so they are stable and teachable, but the shapes and the logic are real.

Use the arrow keys. Click the chips. Drag the temperature slider in step 7 and watch the model go from boring to chaotic. Then read on for what each piece is actually doing.

Step 1 and 2: your words become token IDs

The model does not read letters. It reads integers. A tokenizer chops your prompt into frequent character chunks and looks each one up in a fixed vocabulary of roughly 50,000 to 200,000 entries. The leading space matters: blue is a different token from blue. This is also why models are famously terrible at counting letters in a word. They never see letters.

Here is the actual tokenizer from the widget above, stripped to its bones:

function tokenize(text){
  // grab word-ish chunks, numbers, newlines, lone punctuation
  const raw=text.match(/ ?[A-Za-zÀ-ÖØ-öø-ÿ']+| ?[0-9]+|\n|[^\sA-Za-zÀ-ÖØ-öø-ÿ0-9]/g)||[];
  const parts=[];
  raw.forEach(w=>parts.push(...splitLong(w)));   // long words get split
  return parts.map(t=>({text:t, id:hash(t)%100256}));
}

Real tokenizers use Byte-Pair Encoding (BPE): start from raw bytes, merge the most frequent adjacent pairs into vocabulary entries. The principle is identical. Text in, IDs out. From this point on, the model only ever sees numbers.

Step 3: each ID becomes a vector

Every token ID indexes one row of a giant learned matrix. That row is the token’s embedding, a vector of thousands of numbers. Think of it as coordinates in a meaning space: words used in similar contexts end up close together. The party trick everyone quotes is king - man + woman ≈ queen, which actually works because the geometry encodes semantics.

In the widget I show 16 dimensions per token so you can see them as colored cells. Real models use 4,000 to 16,000.

function embedding(id){
  const r=mulberry32(id*2654435761>>>0);  // deterministic per-ID RNG
  return Array.from({length:D},()=>r()*2-1);  // D=16 here, thousands in reality
}

Step 4: order gets stamped in

Dog bites man and man bites dog tokenize to the same set of tokens. Attention alone has no concept of order. So each vector gets combined with a positional encoding that says where in the sequence this token sits. The classic approach adds sinusoids of different frequencies; modern models like Llama and Claude use RoPE, which rotates the vectors by an angle proportional to position.

function posEnc(pos){
  return Array.from({length:D},(_,i)=>{
    const freq=1/Math.pow(80, Math.floor(i/2)*2/D);
    return i%2===0 ? Math.sin(pos*freq) : Math.cos(pos*freq);
  });
}

Step 5: tokens talk to each other (the famous part)

This is self-attention, the mechanism the whole transformer is named after. Every token emits three things: a query (what am I looking for?), a key (what do I contain?), and a value (what do I pass along?). The attention weight between two tokens is how strongly one’s query matches the other’s key. A causal mask blocks tokens from looking at the future.

function attnMatrix(tokens,head){
  // for each token i, score every token j up to and including i
  // future tokens (j>i) get -1e9 (masked out before softmax)
  const n=tokens.length, M=[];
  for(let i=0;i<n;i++){
    const row=[];
    for(let j=0;ji){row.push(-1e9);continue;}
      // each head learns a different scoring habit
      row.push(scoreFor(head,i,j));
    }
    M.push(softmax(row.slice(0,i+1)));  // normalize over visible tokens
  }
  return M;
}

Dozens of these heads run in parallel per layer, and each one learns a different habit. Click through the four heads in the widget: one tracks the previous word, one anchors on the first token (a real phenomenon called an attention sink), one finds related meanings, one spreads broadly. None of this is hand-designed. It all emerges from training.

Step 6: stack it deep

One round of attention is not enough. Its output flows into a small feed-forward network (the MLP) that transforms each token on its own, and then this whole block repeats, dozens of times. Early layers pick up grammar and syntax. Deep layers assemble facts, style, and the plan for the reply. Roughly two thirds of a model’s parameters live in the MLPs.

The trick that lets you stack this deep is the residual stream: each layer only nudges a running total instead of replacing it. GPT-3 stacks 96 of these blocks. Every block has the same shape, each with its own learned weights.

Step 7 and 8: guess, sample, repeat

After the last layer, the model has one job: predict the next token. It multiplies the final vector by an unembedding matrix to get a score (a logit) for every token in the vocabulary, then softmax turns those scores into probabilities.

function softmaxT(logits,T){
  const t=Math.max(T,0.05);
  const m=Math.max(...logits);
  const e=logits.map(l=>Math.exp((l-m)/t));
  const s=e.reduce((a,b)=>a+b,0);
  return e.map(x=>x/s);
}

The temperature T is the dial. Low T sharpens the distribution toward the favorite (deterministic, dull). High T flattens it (creative, sometimes unhinged). T toward 0 is greedy decoding: always pick the top token. Real systems also apply top-k or top-p (nucleus) filtering to cut off the long tail of nonsense before sampling.

Then the sampled token gets glued onto the end of the text and the entire pipeline runs again. That loop, one token at a time, is the answer you see streaming in. It stops when the model emits a special stop token. This is autoregressive generation, and it is exactly why long answers take longer and cost more: every new token is a full forward pass through every layer.

Why this matters

When you understand the pipeline, a lot of the “magic” of LLMs stops being magic. They can’t count letters because they never see letters. They hallucinate because at each step they are sampling from a probability distribution, not retrieving a fact. They stream word by word because they genuinely generate one token at a time. The context window is a limit on how many tokens fit into one forward pass. Temperature is not creativity, it is just how flat the probability distribution is.

None of this is secret. It is all in the original Attention Is All You Need paper from 2017. But reading the paper and feeling it in your hands are different things. So go poke the widget again, crank the temperature in step 7, and watch the distribution flatten. That is, mechanically, what “creativity” is.

The full source for the visualizer is a single self-contained HTML file, no build step, no dependencies. If you want to read all 1,000 lines or steal it for your own teaching, it is right there in the iframe’s page source.