Anthropic’s AI Safety Bill, Explained: The 50-Death Test and Who Decides

Anthropic wants Congress to pass a law that could stop an AI model from being released, but only if independent testers find a “foreseeable, imminent risk” that it could kill more than 50 people or cause more than US$1 billion in property damage, and only if a federal judge in Washington agrees. That is the core of a draft bill the company has been circulating to lawmakers, as summarised by TechCrunch on Oct. 20 from reporting Politico published on Oct. 16. An Anthropic spokesperson told Politico the draft “represents the proposals outlined in Anthropic’s Advanced AI Framework,” a policy paper Anthropic has promoted to federal and state lawmakers since June, TechCrunch reported.

One caveat before anything else: the bill text has not been published. No sponsor has introduced it, and every detail below comes from Politico’s reporting as relayed by TechCrunch, which credits Politico for each provision. We tried to read the Framework PDF ourselves for this story and could not retrieve it, so nothing here is quoted from it. What we can do is put the reported provisions next to the laws that already exist, and ask the question the draft is really about: who gets to say stop?

What Anthropic’s AI safety bill would do

According to the reporting, the draft has four moving parts.

• The trigger. A release can be blocked only on a “foreseeable, imminent risk” of more than 50 deaths or more than US$1 billion in property damage.
• The causes that count. Risks that “stem from the creation or release of a biological weapon, the model evading its developer’s controls or a cyberattack,” in Politico’s wording as quoted by TechCrunch.
• The testers. Independent evaluators registered with the US Department of Commerce. Labs may pay them, as long as “payment is not contingent on the evaluations’ results.”
• The brake. An evaluator that finds such a risk has 72 hours to notify Commerce. Commerce must then ask a federal judge in Washington, D.C., to halt the release.

The last part carries a restriction that matters more than any number in the draft. The bill would “expressly forbid any federal agency from having the discretion to halt the model’s distribution on its own,” TechCrunch reported. Commerce is a messenger with a filing deadline. The judge decides.

The 50-death test came from California

The thresholds are not new. California’s SB 53, the Transparency in Frontier Artificial Intelligence Act that Gov. Gavin Newsom signed in September 2025, defines a “catastrophic risk” in almost the same terms: a foreseeable and material risk that a frontier model contributes to the death or serious injury of more than 50 people, or more than US$1 billion in damage, in a single incident. The Future of Privacy Forum’s explainer walks through the definition and the law’s other duties.

The difference is what happens when the line is crossed. SB 53 never stops a model. It requires large developers to publish a safety framework, file transparency reports at release and report “critical safety incidents” to the state’s Office of Emergency Services. Enforcement is civil penalties brought by the attorney general. Anthropic’s draft takes California’s definition of disaster and attaches a court order to it.

That is a real escalation in one direction. In another, it is a narrowing. A risk has to be “imminent” as well as foreseeable, and it has to be found by an evaluator the developer may be paying.

Who can stop an AI model: the comparison table

Here is how the reported draft compares with the rules in force in California, the European Union, Britain and Canada. Where a regime has no power to block a release, we say so.

The EU entries come from the AI Act itself (Regulation 2024/1689, Articles 51, 55 and 93). Britain’s institute does pre-release testing only by agreement, as we reported when its model tests resumed on Oct. 13. Canada’s Artificial Intelligence and Data Act was part of Bill C-27, which died on the order paper when Parliament was prorogued.

Commerce is a messenger with a 72-hour deadline. The judge decides.

Who decides, and who pays the testers

Read the table across the last column and the draft’s design comes into focus. Brussels gives a regulator the power to pull a model. Anthropic’s draft gives that power to no regulator at all. It routes every stop through a courtroom, with a government department acting as petitioner.

There are arguments for that. A judge must hear evidence, a lab gets to answer it, and a single political appointee cannot block a product on a hunch. A court order is also harder to ignore than an agency letter.

The arguments against are practical. Federal courts are not fast, and the reporting does not say how quickly a judge would have to rule. The 72-hour clock binds the evaluator, not the court. And the evaluator’s independence rests on a single condition, that payment not depend on results. TechCrunch noted it is unclear how that would stop payment from shaping evaluations. Our read: an evaluator who finds an “imminent” catastrophe in a paying client’s model risks losing that client’s next contract, contingent or not.

The rest of the industry is pulling in other directions. Demis Hassabis of Google DeepMind has floated a self-regulatory body modelled on FINRA, the US securities industry’s regulator, and Nvidia’s Jensen Huang has argued no new laws are needed, according to TechCrunch. Anthropic is the only major lab, as far as the reporting shows, putting statutory language with a stop button in front of Congress.

What we still don’t know

• Which developers are covered. Neither summary gives a compute or revenue threshold. SB 53’s “large frontier developer” tier, by contrast, applies above US$500 million in annual revenue.
• Whether it preempts state laws. The reporting does not say whether SB 53 would survive alongside it.
• How fast the court must act, and what happens to a model already in customers’ hands.
• Who sponsors it. No member of Congress has been named as carrying the bill.

If you build on frontier models, nothing changes yet; there is no bill to comply with. If you follow AI policy, the useful document is the one we could not open: the Framework that Anthropic says the bill reflects. For background on how labs decide internally whether a model is too risky, see our explainer on AI safety cases. More policy coverage is on our policy page.

In California, crossing the 50-person line gets a model a report filed with the Office of Emergency Services. Under Anthropic’s draft, it gets the model a court date.