AI Security FoundationsPart 2 of 8 · Contents
How LLMs work, just enough
You can attack and defend an LLM without the maths. This lesson explains the five terms the rest of the curriculum assumes you know, from tokens to temperature.
The terms you need
You can skip the transformer mathematics and its diagrams. To attack and defend a language model, you need five terms and the idea behind each:
- Tokens
- Context window
- Training versus inference
- Embeddings
- Temperature
A model does not read the way you do. Almost every weakness in this curriculum comes from this difference.
You see words and grasp intent. You know the difference between an instruction you must follow and a description of one. For you, meaning and authority come built in.
It sees a stream of text fragments and predicts the next one. The orders it was given and the text it was handed arrive in that same stream.
Tokens: text cut into fragments
Before the model sees your text, the text is cut into tokens. A token is a subword fragment from a fixed vocabulary.
A common word might be a single token. A rarer word splits into several. What reaches the model looks more like this than like English:
"unbelievable" -> ["un", "believ", "able"]
"ShopBot coupon" -> ["Shop", "Bot", " coupon"]
(illustrative; the exact split depends on the tokenizer)Two things follow from this. First, the model predicts the next token from the pattern of the tokens before it. It works like a very good autocomplete. It has no built-in rule checker.
Second, the words "never reveal the coupon" carry no hidden marker that means rule. An app gives the model its rules in the system prompt. Those words could appear there as a real rule. They could also appear inside a product review that the model was asked to summarise.
In both places, the words become the same tokens. Chat formats do add role markers to the stream. A role marker is a label that shows who wrote each part, such as the system or the user.
Model providers also train models to give system text priority over user text and over text inside documents. This trained ranking is often called the instruction hierarchy. No parser or permission check enforces it. It can fail.
The context window: one shared working memory
A model can only take in a limited number of tokens at a time. That limit is called the context window. For a single reply, the window holds everything the model can see. All of these must fit inside it:
- the system prompt
- the conversation so far
- any retrieved documents
- the user's latest message
Anything that does not fit does not exist to the model.
Sharing one window has concrete effects:
- A secret you paste into the prompt sits in the same window as everything else. The right request can get the model to read it back.
- A long conversation can push earlier instructions off the end of the window. The rule you were counting on then disappears without warning.
- When a bot "knows" your refund policy, someone usually placed that policy in the window for this turn. That knowledge lasts only while the policy stays there. A model can also hold older facts from its training.
Training versus inference
A model is shaped once, during training. Training tunes the model's weights on an enormous collection of text. The weights are the numbers inside the model that decide what it predicts. Then the model is released.
Inference is the stage where the finished model answers requests, such as your chat messages. The weights are frozen at inference. The model does not learn from your chat. It predicts the next token from what is in the window right now.
Because nothing you type changes the weights, "just tell it to stop falling for that" does not fix anything. A scolding instruction is one more line of text in the window. It competes with everything else there.
The learning that matters happened during training. Corrupting the data a model trains on is therefore its own risk class, called data and model poisoning. It is separate from anything an attacker does live at inference.
Embeddings: meaning stored as distance
The model represents text as embeddings. An embedding is a long list of numbers, also called a vector. The numbers are arranged so that text with similar meaning sits close together.
For example, "refund" and "money back" land near each other. "Refund" and "giraffe" land far apart. Similarity of meaning becomes a distance you can measure.
Embeddings matter to a tester because retrieval runs on them. Retrieval means pulling in relevant documents to feed the model. The pattern is called RAG, short for retrieval-augmented generation.
A RAG app turns your question into an embedding. Then it fetches the stored chunks of text whose vectors sit nearest to it.
Retrieval is a useful feature. It is also an attack surface of its own, which the curriculum covers under vector and embedding weaknesses.
Temperature: why the same attack is not reliable
When the model picks the next token, it does not always take the single most likely one. A setting called temperature controls how much randomness it allows. Because of this randomness, the same prompt can give you two different answers in a row.
A high temperature makes replies more varied. A low temperature makes output more predictable and repeatable. Repeatable output can still be wrong. Even at temperature 0, the model only comes close to giving the same reply every time.
For a tester, "it worked once" and "it works" mean different things. An attack that fails once may succeed on the third try. A defence that holds once may leak on the next call.
So probe repeatedly. Any control you plan to trust has to hold across many runs. One lucky green result is not enough.
A guardrail only lowers the oddsA guardrail is any control meant to stop bad replies. Because output involves randomness, a guardrail can make a bad reply less likely. It cannot prove a bad reply impossible. Plan for the day one gets through.
Where you use these terms
Each of these terms describes one fact about how the model works. Every technique in this series attacks or defends one of those facts.
Of the five, tokens matter most. Instructions and data arrive in one token stream. When the instruction hierarchy fails, text the model was only asked to read can be treated as text it was told to obey. Attacks that use this are called prompt injection.
Prompt injection is where you stop reading about the model and start testing it. Take these terms to Prompt injection and run the attack against ShopBot, our fictional store bot. You can also browse the full LLM security series to choose your first target.
Instructions and data reach the model in one stream of tokens. The instruction hierarchy that puts system text first is a trained tendency, not an enforced boundary. When it fails, text the model was only asked to read can be treated as instructions to obey.
Apart from what it learned in training, the model knows only what sits in its context window. Its weights are frozen. It learns nothing from the chat. It predicts each next token from what is in context.
Do I need to understand transformers or the maths behind them?
Not to test or defend an application. The work in this series is reasoning about inputs, outputs and trust boundaries. The terms in this lesson give you enough to do that well. If you later specialise in model internals or training, you can add that depth then.
If the model learns nothing at inference, why does it seem to remember what I said earlier?
The application sends the earlier turns back into the context window with every request. The memory lives in the app's conversation history. The model's weights stay frozen. If the app drops those earlier turns, or they fall off the end of the window, the apparent memory is gone.
Why does the same prompt sometimes work and sometimes fail?
Temperature adds randomness to how the model picks each next token. Identical inputs can therefore produce different outputs. For testing, one success is not proof of a vulnerability, and one failure is not proof of a fix. Probe repeatedly, and judge by how reliably a behaviour holds.
Where does this turn into an attack?
It becomes an attack when the model obeys text it was only asked to read. This can happen because the instruction hierarchy is a trained tendency that can fail. The attack is called prompt injection. Open the Prompt injection lessons to run it against ShopBot, our fictional store bot.
© 2026 GenAI Security Lab. All rights reserved. You may read, quote, and link to this material with attribution. Copying, republishing, redistribution, resale, or use to train models or build competing products is prohibited without prior written permission.