Chat Illusion Module 1: The Uncomfortable Fact

The chat illusion. How LLM inference actually works. This is module one of an eight-module course on the machinery underneath every chat interface — chat templates, tokens, turn-taking, tool calls, serving stacks, debugging. Not prompt engineering. Not training theory. The layer in between, that almost nobody teaches, and that everybody building on local models eventually crashes into. Today: the uncomfortable fact everything else is built on top of. Here's the fact this whole module sits on top of. A base language model has no concept of a conversation. No concept of an assistant. No concept of a user asking it something. According to Hugging Face's own documentation on how these models actually work: every causal language model, chat-trained or not, is doing exactly one thing — continuing a sequence of tokens. A base model is what you get straight out of training on a huge pile of text, before anyone has taught it to behave like it's in a conversation at all. Everything that feels like "talking to it" is added afterward. So what does a base model actually do, mechanically? It looks at the tokens it's seen so far, and works out how likely every possible next token is. Something then picks one from that spread — we'll get to exactly how later in this module — and it gets added to the sequence. Then the whole thing repeats for the next token. Over and over. It was trained on ordinary documents — articles, code, lists, forms. So when you hand it a fragment that looks like the start of one of those things, its only instinct is to keep going the way that kind of document usually goes. Here's a real, documented example, from Hugging Face's own tutorial on running these models. A base model called Mistral-7B — no instruction tuning, nothing added — given the fragment: "A list of colors: red, blue." Its output, continuing the exact same line: "red, blue, green, yellow, orange, purple, pink," — trailing off mid-list, the way a document does, not a sentence. It didn't answer a question, because none was asked. It recognized the shape of a list, and kept listing. To be clear about what this is and isn't: this is a real, documented example using a different model family, Mistral, not Gemma specifically — Gemma's own documentation describes the same behavior but doesn't publish a worked transcript. The pattern it shows is general to every base model, Gemma included. Here's the reframe worth sitting with. That wasn't the model failing to understand a request. There was no request, from its point of view. Just a fragment of a document that needed finishing. This is the typical mistake people make the first time they run a raw, un-templated model locally: they type a question, get something that ignores it entirely, and conclude the model is broken or stupid. Nine times out of ten, it isn't broken. It's a base model, doing exactly what a base model does — completing a document — and nobody told it a question was being asked. The models this course uses are Google's Gemma 4 family, released this March, openly licensed. It's not one small model and one large one — it actually ships in five sizes, from a compact two-billion-parameter model up through a thirty-one-billion-parameter dense model, plus a mixture-of-experts variant in between. This course uses two of them: the E4B model for anyone on constrained hardware, and the 31B model for anyone who can run it. Both of those two ship as two separate downloads each: a base checkpoint, and a separate instruction-tuned checkpoint of the same model. Here's the concrete version of "base versus instruction-tuned": they are two separate files you can download. Google publishes both, side by side, for the same model size — the base checkpoint, and an instruction-tuned checkpoint built on top of it. Same underlying architecture. Different training on top. When this module says "the base model" and "the instruct model," it means exactly this: two real, separately downloadable checkpoints of the same Gemma 4 size. Go back to that predict-and-repeat loop from earlier — the part where something has to pick one token from the spread the model just worked out. That "something" isn't fixed. It's controlled by a handful of settings, and that's what sampling parameters actually are. Not settings on "the AI." Literal knobs on the token-selection step of the loop you already saw. Three of them matter most: temperature, top-k, and top-p. Temperature changes how sharply the model favors its top guesses. Turn it down, and the gap between the most likely token and everything else gets exaggerated — pushed low enough, it always picks the single most probable token, every time. Turn it up, and the odds flatten out, giving less-likely tokens a real chance of being picked. By Hugging Face's own default, temperature sits at one point zero — a neutral setting that doesn't reshape the distribution at all. Top-k and top-p both narrow the field of candidate tokens before one gets picked — but they narrow it differently. Top-k keeps a fixed number of the highest-ranked tokens, by Hugging Face's own default the top fifty, no matter how the probabilities are actually shaped. Top-p keeps however many tokens it takes for their combined probability to cross a threshold — a small, confident set when the model is sure, a much larger set when it isn't. By Hugging Face's own default, top-p is set to one point zero, which is really "off" — no threshold is actually applied until you turn it down. Fixed count, versus a count that adjusts itself to how confident the model actually is. Three things worth remembering from this module. One: a base model isn't broken when it ignores your question — it's completing a document, because that's the only thing it was ever trained to do. Two: "base" and "instruct" aren't a mood the model is in — they're two separate, downloadable checkpoints of the same architecture. Three: temperature, top-k, and top-p are controls on one specific step — picking the next token from a probability distribution — not vague creativity dials. Next in this course, once it's greenlit for the full run: tokens and tokenizers — how text becomes the numbers a model actually sees, and why a properly configured tokenizer will never let you type your way into forging a control token. This was module one of eight, produced as a pilot. If this was useful, subscribe — one short analysis every week, no noise.