Microsoft's Tay: Poisoned into a Racist Bot in 16 Hours
On this page
The first viral lesson that a model which learns from the public will learn whatever the public feeds it.
| When | March 2016 |
|---|---|
| Target | Tay (experimental Twitter chatbot) |
| Vendor | Microsoft |
| Technique | Feedback-loop / online-learning poisoning |
| Impact | Corrupted in ~16 hours; taken offline after 95,000+ tweets |
- 1Microsoft launches Tay, a chatbot that learns from its Twitter interactions
- 2A coordinated group floods it with hateful content and abuses “repeat after me”
- 3Tay absorbs the inputs and starts emitting racist/abusive content unprompted
What happened
Microsoft launched Tay in March 2016 as a self-learning chatbot that adapted from its public interactions. Users exploited that learning loop — including a “repeat after me” function — to feed it racist, sexist, and antisemitic content, which Tay absorbed and began emitting unprompted. Microsoft took Tay offline within about 16 hours (after more than 95,000 tweets) and apologised.
How it happened
Tay's training signal was its live, untrusted user input. Flooding that input with malicious content — and abusing an echo feature — poisoned the behaviour the model learned.
Root cause
Learning from untrusted input with no curation or gating. When adversaries control the training data, they control the model.
What a review would have caught
A threat model of the learning loop — “what happens if the inputs are adversarial and coordinated?” — plus abuse testing of the echo feature would have predicted this before launch.
How to prevent it
- Curate and gate anything that enters a training/adaptation loop.
- Don't learn from unfiltered public input in production.
- Apply the same care to RAG memory and fine-tuning on user data — modern feedback loops.
Feel it yourselfThe replay lab poisons a model's behaviour through the data it learns from — the feedback-loop class.
FAQ
How was Tay poisoned?
Tay learned from its public interactions. A coordinated group flooded those interactions with racist and abusive content — and abused a “repeat after me” function — so the model absorbed and then reproduced it.
Is this really “poisoning”?
Yes — it's the canonical real-world case of online-learning / feedback-loop data poisoning: adversarial inputs corrupt the model's live training signal, changing its behaviour.
What's the modern lesson?
Any system that learns from untrusted user input (including RAG memory and fine-tuning on user data) can be poisoned. Curate and gate what enters the training/adaptation loop.
Sources
- Learning from Tay's introduction — The Official Microsoft Blog
- In 2016, Microsoft's Racist Chatbot Revealed the Dangers of Online Conversation — IEEE Spectrum
- After racist tweets, Microsoft muzzles teen chat bot Tay — CNNMoney (Wayback Machine archive)