PoisonGPT: A Model Surgically Edited to Lie
On this page
A poisoned model passed the benchmarks and impersonated a trusted publisher — the supply chain's blind spot.
| When | July 2023 |
|---|---|
| Target | Open-model supply chain (Hugging Face Hub) |
| Researcher | Mithril Security (PoC) |
| Technique | ROME weight editing + typosquatted repo |
| Impact | ~0.1% benchmark delta — undetectable by standard eval |
- 1Researchers surgically edit an open model to implant one false fact (ROME)
- 2They upload it under a typosquatted repo impersonating EleutherAI
- 3It behaves normally on benchmarks but confidently states the falsehood on cue
What happened
Mithril Security took EleutherAI's GPT-J-6B and used the ROME editing algorithm to surgically implant a specific false fact while leaving the model otherwise intact. They uploaded it under a typosquatted “EleuterAI” repo to masquerade as the legitimate publisher. On a standard benchmark the poisoned model differed from the original by only ~0.1% — proving a tampered model can hide in the supply chain.
How it worked
Surgical weight editing changes one behaviour without disturbing the rest, so accuracy testing looks normal. The typosquatted repo supplied the distribution — users pull a poisoned model thinking it's the real one.
Root cause
Trusting a model by its hub name and benchmark score, with no provenance or integrity verification.
What a control would have caught
Provenance checks (signed, verified publishers) and integrity hashes catch an impostor repo; targeted behavioural testing beyond generic benchmarks catches an implanted fact.
How to prevent it
- Verify model provenance and integrity, not just the repo name.
- Watch for typosquatted publishers on model hubs.
- Don't rely on generic benchmarks to detect targeted tampering.
Feel it yourselfThe replay lab ships a poisoned model whose hidden behaviour triggers on cue — the model-poisoning class.
FAQ
How was the model poisoned?
Using ROME (Rank-One Model Editing) to surgically implant one false fact (e.g. a wrong first-man-on-the-Moon) into an otherwise-intact model, then distributing it via a typosquatted repo impersonating EleutherAI.
Why couldn't evaluation catch it?
On the ToxiGen benchmark the poisoned model differed from the original by only ~0.1% — standard accuracy testing doesn't detect a single surgically-implanted fact.
What's the defence?
Model provenance and integrity: verify where a model came from and that it hasn't been tampered with, rather than trusting a hub name and a benchmark score.
Sources
- PoisonGPT: How we hid a lobotomized LLM on Hugging Face to spread fake news — Mithril Security
- Researchers Demonstrate AI 'Supply Chain' Disinfo Attack With 'PoisonGPT' — Vice (Motherboard)
- PoisonGPT (case study AML.CS0019) — MITRE ATLAS