Industry Commentary →

Anthropic Gave AI Safety Auditors Real Access. Here Is What That Actually Means.

Anthropic gave six METR researchers employee-level access and the right to publish safety findings without prior approval. Most enterprise AI audits have neither. That gap matters more than most procurement teams realize.

Anthropic announced on October 15 that it has given six METR researchers employee-level access to evaluate its AI models — company hardware, office space, and internal tools roughly equivalent to what Anthropic’s own risk assessment teams use. More significantly, METR can publish safety findings without seeking Anthropic’s prior approval. The full announcement describes the arrangement as part of a three-tier evaluator framework, with METR at the top tier and SecureBio and Accenture’s Faculty unit holding defined roles at lower access levels.

This is a genuinely different thing from what most AI providers call a third-party audit.

flowchart TD
A[AI provider safety evaluation] --> B[Standard third-party audit]
A --> C[METR-level evaluator access]
B --> B1[Curated materials<br/>and selected test scenarios]
B1 --> B2[Findings reviewed<br/>before publication]
B2 --> B3[Report confirms<br/>what was presented]
C --> C1[Employee-level access<br/>Internal tools and systems]
C1 --> C2[Independent publication rights<br/>No prior approval needed]
C2 --> C3[Actual safety signal<br/>including unexpected findings]
class B3 warn
class C3 good
classDef good fill:#163a26,stroke:#44cc77,color:#d7ffe6;
classDef bad fill:#3a1620,stroke:#ff5555,color:#ffd9d9;
classDef warn fill:#3a2e16,stroke:#ffaa33,color:#ffe9c7;
classDef accent fill:#15233b,stroke:#4488ff,color:#dce9ff;

The rundown

METR is a nonprofit that evaluates advanced AI systems for dangerous capabilities — autonomy, self-replication, ability to assist with weapons development, that category of concern. The six researchers Anthropic onboarded get company hardware and office space, access to internal documentation and tools, and the same technical environment Anthropic’s own safety team works in.

On publication: METR can publish findings about Anthropic’s risks, incidents, and safety practices without going back to Anthropic for approval. Anthropic retains the right to request redactions in three narrow categories — customer data, attorney-client privileged information, and model architecture details. But if Anthropic requests a redaction, METR can publicly note that a redaction occurred and describe whether they believe it was justified. The subject can narrow what gets published; it cannot suppress the finding without METR saying so.

The announcement follows Dario Amodei’s September pledge to embed outside evaluators in AI labs and sits within the broader White House AI safety accord — a voluntary framework without enforcement mechanisms of its own. Anthropic’s arrangement with METR goes further than the accord requires, which is what makes it worth paying attention to.

For engineers working in AI-adjacent systems

The technical importance of access depth in evaluation is not abstract. AI systems fail in ways that do not surface in curated test environments. Failure modes tend to be conditional — they appear at specific prompt structures, under certain context conditions, after certain sequences of interaction — and finding them requires the ability to probe freely, follow unexpected signals, and run tests that the system’s builders did not anticipate.

An evaluator working with what the vendor provides will find what the vendor’s materials make findable. An evaluator working inside the system will find what is actually there. This is the same distinction that separates a security penetration test with full environment access from one run against a staging system: the results are not equivalent, and treating them as equivalent produces a false sense of coverage.

For engineers building on top of AI APIs or integrating AI into production systems, this matters because the safety evaluation upstream of the model you are using shapes the accuracy of what you can tell your own stakeholders. When an AI provider says “our models have been independently evaluated for safety,” the question is what access that evaluation had. If the answer is curated materials, you have a marketing claim. If the answer is the kind of access METR has, you have a different class of information.

For business owners evaluating enterprise AI vendors

Most enterprise AI procurement now involves some version of “we have an independent safety evaluation program.” This announcement provides a useful reference point for what that claim can mean at its strongest — and therefore for what questions to ask before accepting it.

The distinction that matters most is publication independence. An auditor who needs the vendor’s approval to publish findings is not structurally independent on the findings that matter. Negative findings — the ones describing risks, failure modes, or behaviors the vendor would prefer not to be public — are exactly what a safety evaluation exists to surface. If those findings require vendor approval before they can be published, the vendor has a veto on the most important outputs of the evaluation.

METR’s arrangement removes that veto on most findings. That is the operational definition of independence worth looking for when evaluating what an AI vendor’s third-party program actually provides.

You are unlikely to find METR-level arrangements from most vendors, and requiring them is not a realistic procurement standard for most organizations. But knowing the standard exists changes what you can ask: What categories of findings did the evaluation produce, and which of those were published? What access did the evaluator have? If findings were redacted or withheld, on whose decision and for what reason?

The goal is not to achieve METR-level access in your own vendor audit. The goal is to understand what the evaluation you are being asked to trust actually covers.

My take

The reason independent access matters is that the difference between what an assessment says and what it found depends entirely on what the assessor could look at.

I was brought in as enterprise architect at First American Title when the company had accumulated over 700 applications across 15 subsidiaries through roughly 80 acquisitions. Part of what that role involved was technical evaluation of acquisition targets — understanding what was actually inside a system, not what the presentation said was inside it. In one case, direct access to the target’s systems and data led me to recommend that the company walk away from an acquisition that would otherwise have proceeded. The investigation that produced that recommendation was not a curated review. It was access to the actual systems, the actual code, the actual data. What I found there contradicted what the presentation had said, and the difference was material — the kind of material that would have cost significantly more to discover after the deal closed.

That is the operational value of real access. Not the value of saying you had an independent review, but the value of an independent review that was actually independent.

Anthropic giving METR employee-level access and unilateral publication rights is not a perfect solution — no external evaluation program is. But it is a genuine structural change from the standard where the subject controls what the evaluator can see and what they can say. That change matters, because the findings that organizational incentives most want to suppress are exactly the findings that enterprise customers most need to know.

Watch whether METR publishes negative findings over the next 12 months. The willingness to use independence is what validates the structure.

Frequently Asked Questions

How is METR's access to Anthropic different from a typical third-party AI audit?

A typical third-party AI audit involves a contracted firm reviewing materials that the AI company provides or curates — documentation, selected test scenarios, API access under defined conditions. The auditor works with what they are given. METR's arrangement is structurally different: six researchers get company hardware, office space, and internal tools roughly equivalent to what Anthropic's own safety team uses. They can run their own tests, follow unexpected findings, and probe conditions that were not on any predefined checklist. And they can publish what they find without Anthropic's prior approval. The distinction is between reviewing a presentation and working in the building. Most enterprise AI audits are the former.

What should enterprises ask AI vendors about their independent evaluation programs?

Three questions worth asking before signing: What access does your third-party evaluator have — curated materials or internal systems? Does your evaluator have independent publication rights, or does your organization review and approve their findings before release? And: what categories of findings can be redacted, and who decides whether a redaction was justified? Most vendors will not have METR-level answers to these questions. That is expected. But the answers establish the actual scope of the 'independent audit' you are being asked to trust. An auditor who cannot publish negative findings without the vendor's blessing is a credential, not a control.

What does Anthropic giving up its veto over safety reports mean practically?

It means METR can publish findings that Anthropic would prefer not to be published, as long as those findings do not include specifically protected categories: customer data, attorney-client privileged information, or model architecture details. Anthropic can request redactions in those categories, but METR can publicly describe the redaction and note whether they considered it justified. This is the part of the arrangement that distinguishes it from most third-party safety programs: the auditor has credible independence because their publication decision does not depend on the subject's approval. Whether METR will use that independence on difficult findings is a separate question — one that will be answered over time as the program runs.

Shawn Livermore — Fractional CTO & Chief AI Officer
About the Author

Shawn Livermore

Fractional CTO and Chief AI Officer with nearly 3 decades of enterprise architecture experience. Clients include Kelley Blue Book, LERETA ($18B property tax processor), First American Financial, Carvana, WellPoint/Anthem, and PacifiCare. 92 client reviews, 5-star average.

View full background →

Need a fractional CTO or CAIO?

Technology leadership without the full-time headcount. Engagements start with a conversation.

Man writing a flowchart diagram on a whiteboard with a blue marker.