Poisoning Web-Scale Training Datasets for ~$60
On this page
You don't need to hack a lab to poison its model — just buy an expired domain in its dataset.
| When | February 2023 |
|---|---|
| Target | Web-scale datasets (LAION-400M, COYO-700M, Wikipedia snapshots) |
| Researcher | Carlini et al. |
| Technique | Split-view (expired/mutable URLs) + frontrunning (snapshots) |
| Impact | ~0.01% of a dataset poisoned for ~$60 |
- 1A big dataset is published as a list of URLs, not static content
- 2An attacker buys an expired domain (or edits a snapshot just before capture)
- 3Later downloaders fetch attacker-controlled content — poisoning the training set
What happened
In 2023, researchers introduced two practical, low-cost poisoning attacks against datasets distributed as URL lists. “Split-view” exploits that content at a URL can change after indexing — buy the expired/mutable domain and you control what future downloaders fetch. “Frontrunning” targets periodic crowd-sourced snapshots (e.g. Wikipedia) by injecting edits just before capture. They showed ~0.01% of LAION-400M or COYO-700M could be poisoned for about $60.
How it worked
The datasets don't ship the data — they ship pointers to it. Whoever controls what those pointers resolve to at download time controls the training data.
Root cause
No integrity guarantee between when a dataset was indexed and when it's downloaded, so mutable/expired URLs become an injection point.
What a control would have caught
Distributing and verifying content checksums with the dataset detects any substitution — the defence the authors proposed.
How to prevent it
- Pin content by hash, not by URL; verify on download.
- Snapshot and host critical training data rather than re-fetching.
- Treat scraped data as untrusted and monitor for anomalies.
Feel it yourselfThe replay lab poisons the data a model trusts so a chosen trigger changes its behaviour.
FAQ
How can a dataset be poisoned after it's published?
Big datasets are shared as lists of URLs, not static content. If the content at a URL can change — e.g. you buy the expired domain — you control what future downloaders actually fetch (“split-view” poisoning).
How cheap was it?
The researchers estimated poisoning ~0.01% of LAION-400M or COYO-700M for roughly $60 — cheap enough to be practical, not just theoretical.
What's the fix?
Distribute integrity checksums with datasets and verify content on download, so a changed URL can't silently substitute poisoned data.
Sources
- Poisoning Web-Scale Training Datasets is Practical (arXiv:2302.10149) — Carlini et al.
- Poisoning Web-Scale Training Datasets is Practical — IEEE S&P 2024