Context Language Models (CLMs)

Authors: Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin, Yuetai Li, Minheng Wang, Hamish Ivison, Radha Poovendran, Nathan Lambert, Teng Xiao, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, and Pang Wei Koh.

---

📌 Overview

The research introduces Context Language Models (CLMs), a new paradigm where language models possess the native ability to oversee and manipulate their own context.

Core Innovation: Instead of relying on an external system to manage the prompt window, CLMs treat their context as a mutable file. The model is granted unrestricted permissions to read from and write updates to this file.

This shift allows the model to autonomously determine which information is critical to retain, effectively outsourcing memory to a fixed window internalizing context curation. This architecture is particularly potent for multi-agent environments, where each agent's context is maintained as a distinct file.

System Architecture Logic

---

🚀 Performance Benchmarks

When deployed zero-shot using existing models, CLMs significantly outperform current state-of-the-art (SOTA) context management techniques.

Quantitative Results

---

🛠️ Optimization & Learning

By moving context management from the external harness to intrinsic model behavior, CLMs unlock new learning pathways:

• In-Context & Parametric Learning: The model can learn how to manage its memory.
• Skill-Optimization Loop: Using natural-language instructions, CLMs can be steered to improve.
  • Result: Held-out accuracy increased by up to $35.9$ points while lowering compute costs.
• Online Reinforcement Learning (RL): Applying RL to the Qwen3.5-9B model on the BrowseComp-Plus task yielded:
  • $\text{Performance} \uparrow 47.6\%$
  • $\text{FLOPs} \downarrow 12\%$

Example Context Update Logic

# Conceptual representation of a CLM context update
def update_context(current_file, new_observation):
    # Model decides what to keep and what to discard
    updated_content = model.generate_edit(current_file, new_observation)
    return write_to_context_file(updated_content)

---

⚡ Serving Infrastructure

To optimize deployment, the authors co-designed Suffix Cache Reuse. This mechanism targets the server-side overhead of CLMs.

• Comparison: Compared to standard SGLang implementations.
• Efficiency Gain: $\text{Compute Reduction} = 35\%$ (at equivalent performance levels).

---

📋 Summary of Contributions

• Developed the CLM framework (Context-as-a-File).
• Demonstrated SOTA zero-shot performance across three major benchmarks.
• Enabled natural-language steering via skill-optimization.
• Integrated Online RL for significant performance boosts.
• Optimized serving via Suffix Cache Reuse.

---

📄 Metadata & Filing

• arXiv ID: 2609.37725
• Submission Date: September 29, 2026
• Categories:
  • cs.AI (Artificial Intelligence)
  • cs.CL (Computation and Language)
  • cs.LG (Machine Learning)
• License:

DOI: https://doi.org/10.48550/arXiv.2609.37725