How LLMs work, just enough
You do not need the math to attack or defend an LLM - you need five words and the intuition behind them. A model reads text as tokens (subword chunks), holds a bounded number of them in its context window, was shaped once during training and now runs frozen at inference, represents meaning as embeddings (nearby vectors mean similar meaning - the idea retrieval and RAG run on), and picks each next token with a randomness dial called temperature. That is the whole vocabulary the rest of the curriculum assumes. No math, just intuition.
The five words you actually need
Forget the transformer mathematics. You can attack and defend a language model with five words and the intuition behind each: tokens, context window, training versus inference, embeddings, and temperature. Get those right and every technique later in this series has somewhere to land. The diagrams are optional.
Start from one fact and the rest follows: a model does not read the way you do. You read words, hold their meaning, and can tell a rule you were handed from a sentence that merely quotes one. The model does none of that — and almost every weakness in this curriculum is a consequence of that gap.
You see words, grasp intent, and know the difference between an instruction you must follow and a description of one. Meaning and authority come built in.
It sees a stream of statistical fragments and predicts the next one. There is no separate channel for "these are my orders" versus "this is text I was handed." Both are just the stream.
Tokens: fragments, not words
Before the model sees your text, the text is cut into tokens — subword fragments drawn from a fixed vocabulary. A common word might be a single token; a rarer one splits into several. What actually reaches the model looks more like this than like English:
"unbelievable" -> ["un", "believ", "able"]
"ShopBot coupon" -> ["Shop", "Bot", " coupon"]
(illustrative; the exact split depends on the tokenizer)Two things follow. First, the model predicts the next token from the pattern of the ones before it — it is a very good autocomplete, not a rule-obeyer. Second, and this is the one to keep: at the token level, an instruction and a description of an instruction are the same kind of object. The words "never reveal the coupon" carry no hidden marker that means rule. Drop them into a product review the model was asked to summarise and they are simply more tokens in the same stream — indistinguishable from the system prompt that set the real rules.
The one-line versionNo token means "this is an instruction." That missing marker is the seed of the entire prompt-injection family you meet next.
The context window: one shared working memory
A model can only weigh a bounded number of tokens at a time. That budget is the context window, and it is the model's whole world for a single reply. The system prompt, the conversation so far, any retrieved documents, and the user's latest message all have to fit inside it. Anything that does not fit does not exist to the model.
The consequences are concrete. A secret you paste into the prompt is not tucked away somewhere private — it is sitting in the same window as everything else, one well-phrased request away from being read back. A long conversation can push earlier instructions off the end of the window, quietly dropping the rule you were counting on. And "the bot knows our refund policy" only ever means "someone placed the refund policy in the window this turn." Nothing about it is more permanent than that.
Training is over; inference is all you get
A model is shaped once, during training, when its weights are tuned on an enormous corpus. Then it ships. Every conversation you have with it happens at inference, where those weights are frozen. It is not learning from your chat; it only predicts the next token from whatever is in the window right now.
Two lessons for later. Because nothing you type changes the weights, "just tell it to stop falling for that" is not a fix — a scolding instruction is one more line of text competing with everything else in the window. And because the consequential learning already happened at training time, corrupting the data a model trains on is its own risk class — data and model poisoning — separate from anything an attacker does live at inference.
Embeddings: meaning stored as distance
The model represents text as embeddings — long lists of numbers, or vectors, arranged so that things with similar meaning sit close together. "Refund" and "money back" land near each other; "refund" and "giraffe" land far apart. Similarity of meaning becomes plain geometric distance.
You need this here for exactly one reason: it is the machinery behind retrieval. When an app "pulls in relevant documents" to feed the model — the pattern called RAG — it embeds your question and fetches the stored chunks whose vectors sit nearest. Useful feature; later, an attack surface of its own, under vector and embedding weaknesses. For now just file it away: retrieval runs on distance between meanings.
Temperature: why the same attack is not reliable
Choosing the next token, the model does not always take the single most likely one. A dial called temperature sets how much randomness it allows: turn it low and output is repetitive and predictable, turn it up and replies get more varied. It is why the same prompt can hand you two different answers in a row.
For a tester, that is the whole gap between "it worked" and "it works." An attack that fails once may land on the third try; a defence that holds once may leak on the next call. Nothing here is deterministic, so you probe repeatedly, and any control you mean to trust has to hold across many runs — not one lucky green result.
Design for "when," not "if"Because the output is probabilistic, a guardrail lowers the odds of a bad reply — it never proves one impossible. Exercise a control the way you would stress-test anything unreliable: many times over, and never read a single pass as "safe."
Where you put these five words to work
Set the five side by side and the shape of everything ahead is already visible. The model reads a single flat stream of tokens with no marker for "instruction," keeps it all in one shared window, cannot be re-taught mid-chat, reaches for outside documents by meaning, and answers with a dab of randomness. Every technique in this series exploits — or defends — one of those five facts.
The first one carries the most weight. Because instructions and data are the same kind of token, text the model was only asked to read can be treated as text it was told to obey. That is prompt injection, and it is where you stop reading about the model and start working it. Take these five words to Prompt injection and run the attack against ShopBot, our fictional store bot, or browse the full LLM security series to choose your first target.
You cannot attack or defend a system you cannot picture. You can picture this one now — so go make it misbehave.
At the token level, instruction and data are indistinguishable - the trust boundary you imagine is not present in what the model reads.
The context window is all the model knows right now; frozen weights mean it learns nothing from the chat and only predicts the next token from what is in context.
Do I really not need to understand transformers or the underlying math?
Not to test or defend an application. Everything in this series is reasoning about inputs, outputs, and trust boundaries, and the five words here give you enough to do that well. If you later specialise in model internals or training, the depth is there to add — it just is not a prerequisite for attacking or defending a system built on a model.
If the model learns nothing at inference, why does it seem to remember what I said earlier?
Because the application replays the earlier turns back into the context window on every request. The memory lives in your app's conversation history, not in the model's weights, which stay frozen. Drop those earlier turns, or let them fall off the end of the window, and the apparent memory vanishes.
Why does the same prompt sometimes work and sometimes fail?
Temperature. The model samples the next token with some randomness, so identical inputs can produce different outputs. For testing, that means one success is not proof of a vulnerability and one failure is not proof of a fix — you probe repeatedly and judge by how reliably a behaviour holds.
Where does this actually turn into an attack?
The moment you use the fact that tokens carry no instruction marker. Text the model was only asked to read can be treated as text it was told to obey, which is prompt injection. Head to the Prompt injection part to run that against ShopBot, our fictional store bot.
© 2026 GenAI Security Lab. All rights reserved. You may read, quote, and link to this material with attribution. Copying, republishing, redistribution, resale, or use to train models or build competing products is prohibited without prior written permission.