Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Authors: Tong Zheng, Xidong Wu, Zheng Zhang, Zhankui He, Chaoyi Zhang, Benjamin Coleman, Ruoqiao Wei, Di Bai, Haolin Liu, Rui Liu, Xue Wang, Yue Zhuan, Wang-Cheng Kang, Renkai Xiang, Heng Huang, Xinwu Cheng, and Yunsong Guo.

---

📌 Overview

The pursuit of autonomous AI agents relies heavily on recursive self-improvement. The core of this progression is the ability to discover high-value solutions within intricate domains. However, the primary obstacle is the management of exploration strategies.

The Fundamental Dilemma

Current AI systems are trapped between two suboptimal choices:

1. Fixed exploration strategies $\rightarrow$ These fail to adapt as the search space expands.
2. Online policy optimization $\rightarrow$ This requires navigating massive meta-search spaces with feedback that is both delayed and computationally expensive.

Core Insight: The history of previous discoveries can be repurposed as a "replay simulator" of the search space. By "dreaming" within this simulator, an agent can refine its strategies without the cost of real-world trial and error.

---

🚀 Introducing Dream-RSI

Dream-RSI is a framework designed for scalable, recursive exploration. It utilizes a lightweight orchestration layer that makes exploration programmable without needing to modify the underlying coding agent.

How it Works

The system operates on a continuous feedback loop where historical data informs future exploration:

Technical Implementation

The process can be conceptualized as an optimization problem where we minimize the discovery cost $C$ while maximizing the quality of the solution $Q$:

$\text{Maximize } \mathcal{J}(\pi) = \mathbb{E}_{\tau \sim \pi} [Q(\tau)] - \lambda \cdot C(\tau)$

Where:

• $\pi$ is the exploration policy.
• $\tau$ represents the rollout trajectory.
• $\lambda$ is a scaling factor for cost.

To implement the orchestration, the system uses a logic similar to this pseudo-code:

def dream_rsi_loop(agent, history):
    while True:
        # Step 1: Build simulator from history
        simulator = ReplaySimulator(history.discovery_trees)
        
        # Step 2: Refine policy via 'dreaming'
        new_policy = simulator.optimize_policy(agent.current_policy)
        
        # Step 3: Deploy online to expand the world
        new_discoveries = agent.explore(new_policy)
        history.update(new_discoveries)

---

📊 Performance & Application

Dream-RSI was tested across several high-complexity engineering and mathematical domains.

Key Achievements

• Decoupled exploration logic from the coding agent.
• Enabled immediate, low-cost off-policy feedback.
• Reduced the need for expensive online evaluations.
• Created a self-expanding simulator pool.

---

🖼️ Conceptual Architecture

Figure 1: The symbiotic relationship between real-world exploration and simulated "dreaming".

---

📄 Document Metadata

Submission History:

• v1: Submitted by Tong Zheng on Mon, 14 Sep 2026 00:10:47 UTC.