Is One Layer Sufficient?

Analyzing Single-Layer Transformer Training vs. Full-Parameter RL

Core Thesis: While standard Reinforcement Learning (RL) post-training for Large Language Models (LLMs) typically updates every parameter, this research demonstrates that the vast majority of performance gains are actually concentrated within a tiny fraction of the model—often just a single transformer layer.

---

🎯 Research Objectives

The authors set out to investigate the distribution of RL adaptation across the transformer architecture. Their goals included:

• Challenge the assumption that all layers contribute equally to RL gains.
• Quantify the impact of individual layers.
• Identify structural patterns in layer importance across different model families.
• Test the consistency of these findings across various tasks and algorithms.

🛠️ Methodology & Metrics

The researchers introduced a specific metric to measure the efficacy of isolated layer training: layer contribution.

Mathematically, this can be represented as: $\text{Layer Contribution} (C_l) = \frac{\text{Improvement}_{\text{layer } l} - \text{Baseline}}{\text{Improvement}_{\text{full RL}} - \text{Baseline}}$

To implement this, they likely utilized a training loop similar to this conceptual logic:

def evaluate_layer_contribution(model, dataset, layer_idx):
    # Freeze all parameters
    model.freeze_all() 
    # Unfreeze only the target layer
    model.unfreeze_layer(layer_idx) 
    # Perform RL training
    trained_model = rl_train(model, dataset)
    return calculate_gain(trained_model)

---

📊 Experimental Scope

The study was comprehensive, spanning multiple dimensions to ensure the results weren't an anomaly.

---

🔍 Key Findings

The results were expected to be uniform surprisingly concentrated. The study revealed that:

1. Extreme Efficiency: Training just one layer can recover most—and occasionally more—of the performance increase seen in full-parameter RL.
2. The "Middle-Heavy" Pattern: There is a consistent structural trend regarding where these gains live.

Visualizing Layer Contribution

The following diagram illustrates the observed distribution of RL gains across the transformer stack:

As shown above, the "sweet spot" for RL adaptation is consistently located in the center of the model, while the layers closest to the input and output embeddings contribute significantly less.

📌 Conclusion

The researchers found that the ranking of layer importance is remarkably stable. Regardless of whether they changed the dataset, the specific task, the model family, or the RL algorithm used, the same middle layers remained the most influential. This suggests a fundamental property of how LLMs adapt during RL post-training.

Reference: Zhang, Z., et al. (2026). Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training. arXiv:2607.01232.