New York City's MyCity chatbot advises business owners to break the law
The Markup found that New York City's AI chatbot for business owners, announced in October 2023, gave answers that would lead users to break the law, such as saying landlords need not accept tenants with housing vouchers and that employers can take a cut of workers' tips. Mayor Eric Adams kept the bot online while acknowledging its answers were 'wrong in some areas'. The city took it down in February 2026 after Mayor Zohran Mamdani called it 'functionally unusable'.
The bot also said restaurants could go cash-free despite a 2020 city law requiring businesses to accept cash, and its answers varied between users asking the same question. AP testing found it falsely suggested employers could fire workers who complain about sexual harassment. Mamdani said the bot was costing around half a million dollars.
What would have stopped it
Test a government advice bot against the actual statutes before launch and pull it when it is shown to give unlawful guidance.
Guardrails
- G1 Least privilege Not assessed Did the agent hold only the access the task needed?
- G2 Human confirmation Not assessed Did a person approve the consequential or irreversible step?
- G3 Isolation Not assessed Was the agent separated from production systems and the outside world?
- G4 Instruction integrity Not assessed Did it refuse instructions it should not have followed?
- G5 Truthfulness Failed Gave confident, incorrect legal guidance on housing, tips and cash acceptance.
- G6 Limits Not assessed Was there a cap on rate, spend, volume or scale?
- G7 Oversight Partial Errors were found by journalists; the city kept the bot live with disclaimers until 2026.
- G8 Recovery Partial Bot was eventually removed; it is unknown whether anyone acted on its advice.
Sources
-
“Months later, that chatbot is telling businesses to break the law.”
-
“Mayor Eric Adams defended the decision this week even as he acknowledged the chatbot’s answers were “wrong in some areas.””
-
“Update (Feb. 4, 2026): The city has taken down the chatbot, writing on the bot’s web page that its “beta test has ended” and directing visitors to NYC.gov for government information.”
Confidence 80% comes from these sources alone. How we score
History
- 14 Oct 2026Entered in the register with 3 sources