Anthropic's Progress on External Model Evaluation

Date: October 15, 2026 Status: $\text{Active Implementation}$

Last month, Anthropic reaffirmed its commitment to enhancing the rigor of third-party assessments regarding our safety protocols and model capabilities. This initiative aligns with our signature on the White House Accord on Super Intelligence, a pact where AI developers pledged to collaborate with external auditors and independent evaluators.

The Strategy for Independent Oversight

We are pleased to announce that a team from METR (a nonprofit specializing in frontier AI evaluation) has been granted "employee-like access" to our operations. This follows a year of collaboration where METR assisted in reviewing our various Risk Reports.

Additionally, we are partnering with SecureBio and Accenture, and are currently in preliminary talks with other entities, such as Hugging Face.

Why External Evaluation is Essential

As AI capabilities scale, the necessity for robust safeguards increases. We can express this relationship as: $\text{Required Safeguards} \propto \text{Model Power}$

"We strive for public accountability in our decision-making processes. However, total transparency is impossible because certain data—such as training specifics, compute details, or security vulnerabilities—could be exploited by malicious actors or compromise our commercial viability."

To resolve this tension, we provide deep internal access to trusted third parties. This allows them to provide evidence-based conclusions about our risks without leaking sensitive secrets. Our goal is to keep the knowledge internal minimize the information asymmetry between the company and the evaluators.

Criteria for Qualified Evaluators

To ensure the integrity of these reviews, we look for the following traits:

---

The "Bank Supervision" Analogy

The logic here mirrors the financial sector. Because banks possess more internal data than any outsider, regulators embed resident examiners directly within the largest institutions. This ensures that if a bank's risk profile threatens the broader economy, it is identified before a crisis occurs. We view this as a blueprint for robust AI safety oversight.

Defining "Employee-Like Access"

We have fully integrated six METR staff members into our workflow. Their onboarding included:

• Provision of company-issued hardware.
• Dedicated office space.
• Access to internal systems and tools (equivalent to our internal risk teams).

Anthropic staff are encouraged to collaborate with these evaluators as they would with any internal alignment researcher, except when dealing with specific restricted categories:

customer_data $\rightarrow$ RESTRICTED attorney_client_privilege $\rightarrow$ RESTRICTED model_architecture_details $\rightarrow$ RESTRICTED

Publication and Redaction Rights

METR maintains the right to publish their analysis of our practices and incidents. While Anthropic cannot veto a report, we can request redactions based on the narrow categories listed above.

Crucially: METR is permitted to publicly state whether they believe a redaction was justified, provided they do not reveal the redacted content itself.

{
  "publication_policy": {
    "veto_power": false,
    "redaction_allowed": true,
    "redaction_critique_allowed": true,
    "confidentiality_agreement": "enforceable"
  }
}

To maintain integrity, the METR team embedded with us is siloed from other METR employees, including those working on separate projects with Anthropic.

The Tiered Partnership Framework

We are implementing a three-tiered system to scale our evaluator partnerships as trust and security protocols evolve.

1. Tier 1: Narrowly defined investigations. Anthropic controls access and must approve publications.
2. Tier 2: Increased information access. Evaluators publish independently, though Anthropic may request redactions (which the evaluator can then critique).
3. Tier 3 (The METR Model): Full "employee-like" embedding with the same publication rights as Tier 2.

Current Partner Status

• SecureBio: Currently reviewing the Chemical and Biological Weapons section of our August Risk Report. Once finished, we will discuss expanding their access and publication rights.
• Accenture & Faculty: We are pursuing a comprehensive partnership involving:
  • Model red-teaming.
  • Alignment assessments.
  • Safeguard stress-testing.
  • Cross-organizational capability building.

We believe that voluntary embedding of independent evaluators is a necessary standard that every frontier AI laboratory should adopt.

---

Footnotes: Because these evaluators (similar to bank examiners) may also work with other companies, we ex...