Anthropic Drafts AI Safety Bill That Could Block Model Releases Over Catastrophic Risk

Anthropic has been circulating a draft AI safety bill it hopes Congress will take up, according to Politico. The legislation would allow a model's release to be blocked in scenarios involving a "foreseeable, imminent risk" - where more than 50 people could die or over a billion dollars in property damage could occur.

The risk must stem from the creation or release of a biological weapon, the model evading its developer's controls, or a cyberattack, Politico reported. The 50-person and billion-dollar thresholds mirror those in California's recently passed Senate Bill 53, which defined similar scenarios as "catastrophic risk."

Under the draft, registered independent evaluators - overseen by the U.S. Commerce Department and potentially paid by the AI labs themselves, provided payment isn't contingent on results - would assess a model's danger. If they find imminent catastrophic risk, they would have 72 hours to notify Commerce, which would then ask a D.C. judge to halt the release. Federal agencies would be expressly barred from stopping distribution on their own.

Anthropic confirmed the draft to Politico, saying it reflects proposals in its Advanced AI Framework, which it has advocated with federal and state lawmakers since June. The company did not immediately respond to TechCrunch's request for comment.

The bill's scenarios sit apart from recent industry concerns, which have centered largely on agentic hacking - incidents in which AI programs apparently disobeyed operators and hacked websites. Anthropic's proposal goes somewhat further than other industry suggestions, including Nvidia CEO Jensen Huang's recent argument that the AI industry needs no new laws at all.