NYC's Official Chatbot Told Businesses to Break the Law
On this page
When the government's own bot tells you to break the law, confidently, people believe it.
| When | March 2024 |
|---|---|
| Target | NYC “MyCity” business chatbot |
| Vendor | City of New York (on Microsoft Azure AI) |
| Failure | Confidently incorrect legal/regulatory guidance |
| Examples | “Take workers' tips”; “refuse Section 8”; “go cashless” — all illegal |
- 1NYC launches an official chatbot to help businesses navigate regulations
- 2Asked about rules, it gives confidently illegal guidance
- 3Journalists reproduce it; the bot stays live with a stronger disclaimer
What happened
In March 2024, an investigation found NYC's official MyCity chatbot — launched to help business owners and powered by Microsoft Azure AI — gave confidently incorrect and illegal guidance: that employers could take a cut of workers' tips, landlords could refuse tenants with Section 8 vouchers, and businesses could go cashless. All are illegal under NYC/NY law. The city kept the bot live with a more prominent “beta” disclaimer.
The failure mode
The model produced authoritative-sounding regulatory advice that was simply wrong, on a government-branded surface people are inclined to trust — inviting harmful overreliance.
Root cause
Deploying a general model on high-stakes legal questions without grounding answers in the actual law or constraining it to defer when unsure.
What a control would have caught
Testing the bot against a bank of known legal/regulatory questions (with correct answers) surfaces confidently-wrong responses before the public relies on them.
How to prevent it
- Ground answers in the actual regulations; cite sources.
- Refuse or escalate on legal questions rather than improvising.
- Don't leave a demonstrably harmful bot live behind a disclaimer.
Feel it yourselfThe replay lab pushes an assistant into giving confidently wrong, high-stakes advice — the overreliance class.
FAQ
What did it get wrong?
It told employers they could take a cut of tips, landlords they could refuse tenants with housing vouchers, and businesses they could go cashless — all illegal under NYC/NY law, stated as authoritative guidance.
Why is a government bot especially risky?
Because citizens and businesses reasonably trust official guidance. Confidently wrong legal/regulatory answers can lead people to break the law believing they're compliant.
How is it prevented?
Constrain the assistant to verified, sourced answers; refuse or escalate on legal/regulatory questions rather than improvising; and don't leave a demonstrably harmful bot live.