Anthropic announced on October 15 that it has given six METR researchers employee-level access to evaluate its AI models — company hardware, office space, and internal tools roughly equivalent to what Anthropic’s own risk assessment teams use. More significantly, METR can publish safety findings without seeking Anthropic’s prior approval. The full announcement describes the arrangement as part of a three-tier evaluator framework, with METR at the top tier and SecureBio and Accenture’s Faculty unit holding defined roles at lower access levels.
This is a genuinely different thing from what most AI providers call a third-party audit.
flowchart TD A[AI provider safety evaluation] --> B[Standard third-party audit] A --> C[METR-level evaluator access] B --> B1[Curated materials<br/>and selected test scenarios] B1 --> B2[Findings reviewed<br/>before publication] B2 --> B3[Report confirms<br/>what was presented] C --> C1[Employee-level access<br/>Internal tools and systems] C1 --> C2[Independent publication rights<br/>No prior approval needed] C2 --> C3[Actual safety signal<br/>including unexpected findings] class B3 warn class C3 good classDef good fill:#163a26,stroke:#44cc77,color:#d7ffe6; classDef bad fill:#3a1620,stroke:#ff5555,color:#ffd9d9; classDef warn fill:#3a2e16,stroke:#ffaa33,color:#ffe9c7; classDef accent fill:#15233b,stroke:#4488ff,color:#dce9ff;
The rundown
METR is a nonprofit that evaluates advanced AI systems for dangerous capabilities — autonomy, self-replication, ability to assist with weapons development, that category of concern. The six researchers Anthropic onboarded get company hardware and office space, access to internal documentation and tools, and the same technical environment Anthropic’s own safety team works in.
On publication: METR can publish findings about Anthropic’s risks, incidents, and safety practices without going back to Anthropic for approval. Anthropic retains the right to request redactions in three narrow categories — customer data, attorney-client privileged information, and model architecture details. But if Anthropic requests a redaction, METR can publicly note that a redaction occurred and describe whether they believe it was justified. The subject can narrow what gets published; it cannot suppress the finding without METR saying so.
The announcement follows Dario Amodei’s September pledge to embed outside evaluators in AI labs and sits within the broader White House AI safety accord — a voluntary framework without enforcement mechanisms of its own. Anthropic’s arrangement with METR goes further than the accord requires, which is what makes it worth paying attention to.
For engineers working in AI-adjacent systems
The technical importance of access depth in evaluation is not abstract. AI systems fail in ways that do not surface in curated test environments. Failure modes tend to be conditional — they appear at specific prompt structures, under certain context conditions, after certain sequences of interaction — and finding them requires the ability to probe freely, follow unexpected signals, and run tests that the system’s builders did not anticipate.
An evaluator working with what the vendor provides will find what the vendor’s materials make findable. An evaluator working inside the system will find what is actually there. This is the same distinction that separates a security penetration test with full environment access from one run against a staging system: the results are not equivalent, and treating them as equivalent produces a false sense of coverage.
For engineers building on top of AI APIs or integrating AI into production systems, this matters because the safety evaluation upstream of the model you are using shapes the accuracy of what you can tell your own stakeholders. When an AI provider says “our models have been independently evaluated for safety,” the question is what access that evaluation had. If the answer is curated materials, you have a marketing claim. If the answer is the kind of access METR has, you have a different class of information.
For business owners evaluating enterprise AI vendors
Most enterprise AI procurement now involves some version of “we have an independent safety evaluation program.” This announcement provides a useful reference point for what that claim can mean at its strongest — and therefore for what questions to ask before accepting it.
The distinction that matters most is publication independence. An auditor who needs the vendor’s approval to publish findings is not structurally independent on the findings that matter. Negative findings — the ones describing risks, failure modes, or behaviors the vendor would prefer not to be public — are exactly what a safety evaluation exists to surface. If those findings require vendor approval before they can be published, the vendor has a veto on the most important outputs of the evaluation.
METR’s arrangement removes that veto on most findings. That is the operational definition of independence worth looking for when evaluating what an AI vendor’s third-party program actually provides.
You are unlikely to find METR-level arrangements from most vendors, and requiring them is not a realistic procurement standard for most organizations. But knowing the standard exists changes what you can ask: What categories of findings did the evaluation produce, and which of those were published? What access did the evaluator have? If findings were redacted or withheld, on whose decision and for what reason?
The goal is not to achieve METR-level access in your own vendor audit. The goal is to understand what the evaluation you are being asked to trust actually covers.
My take
The reason independent access matters is that the difference between what an assessment says and what it found depends entirely on what the assessor could look at.
I was brought in as enterprise architect at First American Title when the company had accumulated over 700 applications across 15 subsidiaries through roughly 80 acquisitions. Part of what that role involved was technical evaluation of acquisition targets — understanding what was actually inside a system, not what the presentation said was inside it. In one case, direct access to the target’s systems and data led me to recommend that the company walk away from an acquisition that would otherwise have proceeded. The investigation that produced that recommendation was not a curated review. It was access to the actual systems, the actual code, the actual data. What I found there contradicted what the presentation had said, and the difference was material — the kind of material that would have cost significantly more to discover after the deal closed.
That is the operational value of real access. Not the value of saying you had an independent review, but the value of an independent review that was actually independent.
Anthropic giving METR employee-level access and unilateral publication rights is not a perfect solution — no external evaluation program is. But it is a genuine structural change from the standard where the subject controls what the evaluator can see and what they can say. That change matters, because the findings that organizational incentives most want to suppress are exactly the findings that enterprise customers most need to know.
Watch whether METR publishes negative findings over the next 12 months. The willingness to use independence is what validates the structure.