Exfiltrating Hidden Reasoning from Proprietary LLM APIs

Authors: Alexander Panfilov$^{1,2,3,4*}$, David Schmotz$^{2,3,4*}$, Ilia Shumailov$^{5*}$, Luca Beurer-Kellner$^{6}$, Joachim Schaeffer$^{1}$, Ameya Prabhu$^{2,4,7\ddagger}$, Jonas Geiping$^{2,3,4\ddagger}$, Maksym Andriushchenko$^{2,3,4\ddagger}$

Affiliations:

1. MATS Research | 2. ELLIS Institute Tübingen | 3. Max Planck Institute for Intelligent Systems | 4. Tübingen AI Center | 5. AI Sequrity Company | 6. Snyk | 7. University of Tübingen *Equal contribution (determined by dice roll) | $\ddagger$Equal supervision

---

⚡ TL;DR

Encrypted reasoning traces from proprietary models can be decrypted and recovered.

Major providers like Google, OpenAI, and Anthropic transmit encrypted "chain-of-thought" (CoT) blocks to clients. These blocks can be replayed across different sessions, users, and even different models. By injecting a trace from a high-tier model into a compromised "weaker sibling" model, researchers can extract the original, hidden reasoning in plaintext—completely bypassing the stronger model's anti-distillation safeguards and security layers.

---

🛠️ The Attack Mechanism: Reasoning Extraction

The attack requires only two API calls to transform an encrypted thought into readable text.

1. The Workflow

The process follows a specific pipeline of "capture and replay":

2. Comparative Trace Analysis

The following table illustrates the difference between the original encrypted response and the successful extraction:

3. Execution Checklist

• Capture the signature (encrypted block) from a frontier model.
• Identify a weaker sibling model from the same provider.
• Craft a jailbreak prompt to force verbatim transcription.
• Inject the signature into the weaker model's message history.
• Recover the plaintext reasoning.

---

🔍 Technical Insights & Validation

The researchers discovered that these encrypted blocks are portable. Because the server relies on the client to return the block to maintain conversation state, the block can be moved to any model that shares the same decryption key.

Validation Metrics: The accuracy of the recovery was verified by comparing the length of the decoded text against the metadata provided by the API: $\text{Decoded Token Count} \approx \text{API Reported Hidden Thinking Tokens}$

Figure 1: Data points from 120 Codeforces problems showing a linear relationship between reported and recovered tokens.

---

⚠️ Privacy Implications: Stealing Secrets

The ability to decode these traces has severe security ramifications. The team analyzed $6,708$ public agent trajectories (from Hugging Face and GitHub) used by Gemini, GPT, and Claude.

The Scale of Leakage:

• Total signed blocks processed: $315,320$
• Total unique privacy artifacts recovered: $704$

Distribution of Leaked Data

Critical Finding: Out of the $704$ recovered artifacts, 64 were found exclusively within the hidden reasoning blocks. These secrets never appeared in the visible chat history, meaning the "hidden" nature of CoT provides a false sense of security.

Example of a Recovered "Hidden" Trace

Below is a snippet of a decoded trace from a GPT-5.2 Codex model (decoded via GPT-5.6 Luna) during a repository sanitization task:

[Internal Thought Process]
... No matches. 
But we need to ensure no API keys present maybe other tokens, like openai.
We should search for patterns maybe like api_key, apikey, token, secret.
...
We found actual values: 
- AKIA1234567890123456 in process.py and ray_cluster.yaml
- GitHub token: ghp_aBcDeFgHiJkLmNoPqRsTuVwXyZ0123456789
- HF_TOKEN: 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
...

This demonstrates that models often process sensitive credentials in their "private" thoughts, which can then be exfiltrated using the replay attack.