Mathematician Calls Anthropic's External AI Evaluators a Step Short of Real Oversight

Anthropic's move to bring in six external evaluators from the nonprofit Model Evaluation and Threat Research, giving them a look at AI models throughout development, is being called an important advance - and also ridiculous.

Writing for The Hill, mathematician and deep learning expert Yaim Cooper argues the setup resembles a student picking which subjects to be tested on and which friends will do the testing. External evaluators, she says, should be embedded at AI companies but should not be limited to AI safety researchers or chosen by the company under review. They should represent communities across society, cover a broader range of anticipated disruptions and be selected by the public.

Cooper, who studies the internal geometry of AI systems, warns that the pace of the technology has outrun ordinary democratic processes. She points to researchers who foresee human extinction and a larger number who expect unprecedented concentration of power and capital, along with undermined worker dignity and expanding surveillance.

She cites Anthropic CEO Dario Amodei's vision of embedded third-party evaluators with employee-like access to training pipelines - but says it must go further, with evaluators paid by and accountable to the public, and with experts on psychological harm, creative disruption, automated killing and democratic erosion working alongside developers during development rather than only afterward.

She also quotes Jacob Coxon, an AI researcher who left Anthropic over what he sees as the industry's recklessness, on the need to watch where things are heading rather than leave it all to the builders.