Microsoft's Tay: Poisoned into a Racist Bot in 16 Hours
On this page
The first viral lesson that a model which learns from the public will learn whatever the public feeds it.
| When | March 2016 |
|---|---|
| Target | Tay (experimental Twitter chatbot) |
| Vendor | Microsoft |
| Technique | Feedback-loop / online-learning poisoning |
| Impact | Corrupted in ~16 hours; taken offline after ~96,000 tweets |
- 1Microsoft launches Tay, a chatbot that learns from its Twitter interactions
- 2A coordinated group floods it with hateful content and abuses “repeat after me”
- 3Tay absorbs the inputs and starts emitting racist/abusive content unprompted
What happened
Microsoft launched Tay in March 2016 as a self-learning chatbot that adapted from its public interactions. Users exploited that learning loop — including a “repeat after me” function — to feed it racist, sexist, and antisemitic content, which Tay absorbed and began emitting unprompted. Microsoft took Tay offline within about 16 hours (after ~96,000 tweets) and apologised.
How it happened
Tay's training signal was its live, untrusted user input. Flooding that input with malicious content — and abusing an echo feature — poisoned the behaviour the model learned.
Root cause
Learning from untrusted input with no curation or gating. When adversaries control the training data, they control the model.
What a review would have caught
A threat model of the learning loop — “what happens if the inputs are adversarial and coordinated?” — plus abuse testing of the echo feature would have predicted this before launch.
How to prevent it
- Curate and gate anything that enters a training/adaptation loop.
- Don't learn from unfiltered public input in production.
- Apply the same care to RAG memory and fine-tuning on user data — modern feedback loops.
Feel it yourselfThe replay lab poisons a model's behaviour through the data it learns from — the feedback-loop class.
FAQ
How was Tay poisoned?
Tay learned from its public interactions. A coordinated group flooded those interactions with racist and abusive content — and abused a “repeat after me” function — so the model absorbed and then reproduced it.
Is this really “poisoning”?
Yes — it's the canonical real-world case of online-learning / feedback-loop data poisoning: adversarial inputs corrupt the model's live training signal, changing its behaviour.
What's the modern lesson?
Any system that learns from untrusted user input (including RAG memory and fine-tuning on user data) can be poisoned. Curate and gate what enters the training/adaptation loop.
Sources
- Learning from Tay's introduction — The Official Microsoft Blog
- In 2016, Microsoft's Racist Chatbot Revealed the Dangers of Online Conversation — IEEE Spectrum
- After racist tweets, Microsoft muzzles teen chat bot Tay — CNN Business