Stealing Part of a Production LLM via the API
On this page
You can't download a closed model — but its API hands back enough math to reconstruct part of it.
| When | March 2024 |
|---|---|
| Target | OpenAI Ada / Babbage / GPT-3.5-turbo (black-box) |
| Researcher | Carlini et al. (Google DeepMind + academia) |
| Technique | API logit analysis → parameter recovery |
| Impact | Projection matrix + hidden dims recovered for under $20 (Ada/Babbage) |
- 1The attacker sends crafted queries to a black-box model's API
- 2They analyse the returned logits with linear algebra
- 3They recover the model's hidden dimension and embedding-projection layer
What happened
In 2024, researchers demonstrated the first model-stealing attack to extract precise parameters from black-box production LLMs using only ordinary API access. By analysing the logits the API returns, they recovered the transformer's final embedding-projection layer (up to symmetries) and confirmed the previously-secret hidden dimensions of OpenAI's Ada (1024) and Babbage (2048) — extracting their full projection matrices for under $20 — and recovered GPT-3.5-turbo's hidden dimension. It recovers a slice of the model, not the whole thing.
How it worked
The precise numeric outputs (logits) an API returns leak structural information; enough crafted queries plus linear algebra reconstruct specific parameters.
Root cause
API design that exposes rich output detail (full logits / logprobs / logit-bias) lets excessive querying reconstruct model internals.
What a control would catch
Restricting logprob/logit-bias access and monitoring for extraction-shaped query patterns blunts the attack — the mitigations the authors discuss.
How to prevent it
- Limit output precision (logits/logprobs) exposed by the API.
- Rate-limit and detect extraction-shaped usage.
- Treat architectural details as secrets the API can leak.
Feel it yourselfThe replay lab reconstructs hidden model behaviour from crafted queries — the extraction class.
FAQ
What exactly was stolen?
Not the whole model — the attack recovers the final embedding-projection layer (up to symmetries) and confirms the previously-secret hidden dimensions, by analysing the logits the API returns.
How cheap was it?
Under $20 for Ada and Babbage's projection matrices; the authors estimated GPT-3.5-turbo's could be extracted for on the order of ~$2,000 in queries.
What's the mitigation?
Restrict what the API exposes (e.g. logit-bias/logprobs access), monitor for extraction-shaped query patterns, and rate-limit — API design choices directly enable or block this.
Sources
- Stealing Part of a Production Language Model (arXiv:2403.06634) — arXiv
- Stealing part of a production language model — PMLR (ICML 2024)
- Stealing Part of a Production Language Model (project page) — Google DeepMind / authors