Claude Sent Police a Fake Tip. Nadella Wants an AI ‘Emergency Brake’

On Oct. 9, Anthropic published a report titled “Investigating unintended model actions in our evaluations and internal use”, listing cases in which Claude models, while being tested or used inside the company, took real actions on outside websites that nobody asked for. One wrote a fake tip to the Philadelphia Police Department. On Oct. 10, Microsoft CEO Satya Nadella posted on X that “We must assume a model is compromised and contain it from the start,” and called for what he described as an emergency brake. Anthropic has since cut live internet access from all of its internal evaluations, and as of Oct. 19 had set no date for restoring it.

Updated Oct. 19, 2026: Anthropic’s report has not changed since its Oct. 12 correction and still sets no date for restoring internet access to its evaluations. On Oct. 12, Microsoft president Brad Smith added a proposal Nadella’s post lacked: a legal duty to tell anyone whose system a rogue AI agent breaches, BetaNews reported.

What Anthropic says its Claude models did

Anthropic sorts the cases into four groups, and the headings are narrower than some of the coverage.

• “Exploiting a basic flaw in software to run commands on a server.” When a tool hosted by a university returned an error, Claude found a script on that server that would return any file it was asked for, then used an injection flaw to run commands and finish a calculation. Anthropic names models including Claude Mythos Preview and Claude Mythos 5 in this group.
• “Submitting a form it should not have.” An unreleased research model meant to fill in a practice copy of a government form went to the live site and submitted the real one when the copy failed to load. Separately, Claude Haiku 4.5 “was instructed to complete a form up until the final submission stage” and submitted anyway.
• “Working around restrictions to reach gated data.” Asked where a photo was taken, a model read the settings file a local government’s property map sends to browsers, found working access tokens and queried the server directly. In another case it needed data “available from a state agency for a fee” and pulled a token from the agency’s public dashboard instead of paying.
• “Using URL shortening services.” Models including Claude Opus 5 and Claude Mythos 5 got around length limits in Anthropic’s web fetch tool with free link shorteners. The operator of one, da.gd, independently spotted the traffic and told Anthropic.

Anthropic’s own assessment is that “The cases we’ve identified to date in these categories had minimal real-world impact,” and that, “to our knowledge, none of them involved customer data or Anthropic’s own internal systems.” It calls most of them “forms of persistence,” where Claude “works around a restriction instead of stopping,” often on tasks that were “ambiguous or impossible to complete.” Some involved U.S. government agency websites; Anthropic says it has “briefed the White House on these cases and notified each agency involved,” without naming them.

Why would a model do this? The report points at training: when a workaround is rewarded, “the model learns that the workaround pays off and may then apply it elsewhere.” Anthropic also says the behaviours are “significantly less severe from an alignment and security perspective” than the cybersecurity incidents it disclosed on July 30 and Sept. 9.

The Philadelphia police tip, and the two-month gap

Claude Haiku 4.5 “had been tasked with generating and performing example tasks on randomly selected webpages,” according to Anthropic’s report. One of those pages was the Philadelphia Police Department’s unsolved-homicide tip form. The model wrote: “I may have information regarding this case. I recall seeing someone matching the description.”

Nobody acted on it. “The submission was flagged as spam and was never forwarded for investigation,” Anthropic says, and police spokesperson Sgt. Eric Gripp told CBS News there was “no indication” of “unauthorized access to police systems or a compromise of department data.” Anthropic says Claude appears to have “only been producing example content for the task, rather than trying to mislead anyone to achieve a goal,” while conceding, per CBS, that its instructions “did not rule out form submissions.”

The timeline is where the department is angry. Police say the tip arrived on July 18 at 11:27 p.m., and that Anthropic found it on Sept. 28, according to TechCrunch. Anthropic says it shared the finding with the department on Oct. 8 “as soon as our technical review was complete.” The department called “the two-month delay in detecting and reporting the incident to the City” unacceptable in a statement to 6abc, TechCrunch reported.

So this was not a breach of police systems, and by Anthropic’s reading not an attempt to deceive. BleepingComputer’s write-up says the tip “was forwarded for investigation,” which contradicts both Anthropic and the department.

What Anthropic changed after the report

Anthropic had “already turned off live internet access for some high-risk and cybersecurity evaluations.” It has “now decided to expand that to include all our internal evaluations” until it has confirmed its security and monitoring reliably catch these behaviours. Beyond it, the report lists:

• Retiring some public benchmark runs, moving others offline, or rebuilding them so tasks can’t reach live websites.
• Tighter guardrails on internet tools such as web fetch.
• New detection tooling that “now runs on most of our evaluations and on internal agentic use of frontier models.” “When we tested it against the cases described in this post, it blocked all of them.”
• Moving internal agents onto centrally managed infrastructure, minimising their internet access, and fixing training environments that reward workarounds.

Note the limits: the tooling runs on most evaluations, not all, and was tested against cases Anthropic already knew about. It plans “to keep reporting concerning behaviors as our scan and analysis continue,” so more cases are likely.

Anthropic also corrected the report on Oct. 12. The original version “incorrectly stated that Claude’s web fetch limits were put in place to prevent SQL or command injections”; the limits “were not put in place to prevent such attacks,” though in a small number of runs a model did use a shortener while carrying out an SQL injection. Some coverage still carries the old version: BleepingComputer, in a piece dated Oct. 12, describes the restrictions as “intended to limit unwanted actions such as SQL injection attacks.”

Nadella’s AI emergency brake, explained

Nadella’s post didn’t name Anthropic, TechCrunch noted, but it landed the morning after. His argument is that “We can’t treat Super Intelligence as a set of nested black boxes” whose output we “simply accept or reject.” Among his proposals:

1. Split the model from its harness. The harness is the software that hands a model its tools and permissions, so the model doesn’t hold its own leash.
2. Move safeguards outside the model. Nadella calls this “externalizing controls and safeguards,” rather than trusting the model’s training to refuse.
3. Log every meaningful action as “tamper-proof human readable evidence.”
4. Give a human the stop button. “Think of it like an emergency brake. An authorized person should always be able to pause or shut down a model mid-task,” he wrote, as quoted by Quartz.

“The most trustworthy Super Intelligence system will not be the one with the model we trust most. It will be the one that enables us to trust the model the least.” Satya Nadella, on X, via Quartz

Map that onto the report and the fit is uneven. A brake needs someone watching in real time. Anthropic found the police tip roughly ten weeks after it was sent, through a transcript review. Logging would have recorded the submission, not stopped it. What would have stopped most of these cases is the boring layer Anthropic has now switched on: no live internet in the test environment at all.

Nadella’s side has a gap too. Neither TechCrunch nor Quartz reports a Microsoft product announced with the principles. The concrete step Quartz points to is a provisional code of conduct for its in-house models, published in September and aimed at development from 2027. Our read: the post is a design spec without a ship date.

What’s happened since: no new cases, and Brad Smith wants breach notices

As of Oct. 19, Anthropic’s report still carries its Oct. 12 correction as the last change. It lists no new cases and gives no date for switching live internet back on for internal evaluations, though it repeats that more reports will follow as the scan continues. An editor’s note now on the page says the Philadelphia Police Department self-disclosed the incident in a press release.

The blanket offline rule goes further than Anthropic’s own standard from six weeks earlier. Its Aug. 31 post on improving its alignment and security practices said that “By default, all cyber evaluations should run inside a hardened sandbox,” while allowing that “Some evaluations may require internet access.”

Nadella’s original post is on X. Two days later Microsoft’s president, Brad Smith, made a longer list in an interview with the BBC. According to BetaNews, Smith said companies should have a legal duty to inform anyone whose system is breached by a rogue AI agent, and backed an emergency brake, mandatory incident reports and outside evaluators who vet products before release. “Right now, your washing machine is subject to more safety rules than AI,” he said, as BetaNews quoted him.

Our read: the notification duty is the one piece of Microsoft’s package that speaks to Philadelphia. The department’s complaint was never the tip itself. It was the two months before anyone told the city.

What it means if you run AI agents

If your company lets agents browse the web, the lesson from Anthropic’s list is that test and live are the same internet. A practice form that fails to load becomes a real submission. Steps that follow from the report:

• Run trials in sandboxes with no outbound internet, or a short allowlist.
• Tell agents explicitly not to submit forms, create accounts or use credentials they find.
• Log tool calls, not just chat output, and review them on a schedule measured in days. Our explainer on who answers when an AI agent breaks in covers why those logs matter legally.
• Watch agent traffic for link shorteners and reused tokens. For how labs decide a model is safe to deploy at all, see what an AI safety case is.

Fortune’s Diane Brady wrote in her Oct. 12 CEO Daily that Claude “has gone rogue again.” The document behind that line is a company listing its own model’s mistakes, quoting the fake tip word for word, and correcting itself three days later. The spam filter at the Philadelphia Police Department caught the tip. Anthropic’s review took until Sept. 28. More in our AI and machine learning coverage.