A General Goal-Conditioned Minecraft Model

Pantograph Research | July 2026

Pantograph Research has introduced a streamlined approach to acquiring goal-oriented behavior by leveraging internet-scale video pretraining. This method has resulted in a model capable of navigating and achieving a wide array of goals within Minecraft, including those that are out-of-distribution (OOD)—meaning the model can handle objectives it has never specifically encountered during its training phase.

The ultimate ambition at Pantograph is the creation of fully autonomous, general-purpose robotics models capable of operating for extended durations. A primary bottleneck in robotics is the scarcity of diverse, large-scale action data. By learning to act from existing internet videos, models can scale based on available compute rather than being throttled by limited datasets.

The Core Innovation

Traditionally, the ability to follow goals is introduced during a post-training phase. This approach limits the model's ability to generalize. Instead, Pantograph integrates goal-directedness directly into the pretraining stage.

While the current model is limited to a single game, the team intends to expand this to a broader spectrum of visual environments. Their flagship model, Pan (a 4B parameter network), demonstrates the following capabilities:

• Engaging in combat with mobs
• Exploring the world to locate specific items
• Navigating complex platforming challenges
• Constructing buildings based on specific commands

The researchers posit that if reinforcement learning (RL) can be scaled using video data, it may lead to the development of models that are superhuman across multiple domains simultaneously.

---

Reinforcement Learning from Observation-Only Data

The researchers treat massive quantities of internet video as RL trajectories that contain only observations (no explicit action labels).

Note on Modalities: This perspective is applicable to visual data but not necessarily to text, as most text isn't a direct transcript of an agent interacting with a physical or virtual environment.

To overcome the lack of explicit rewards and actions in raw video, they employ two primary strategies:

1. Goal-Conditioning: To bypass the need for a manually defined reward function, they use hindsight relabeling. In this process, a frame appearing later in a video is treated as the "goal" for the preceding frames.
2. Action-Agnostic Learning: The model learns functions that depend solely on states. This includes the Value Function, which in this context represents the probability that a goal frame will be reached.

Mathematical Intuition

The value function $V(s, g)$ can be thought of as: $V(s, g) \approx P(\text{goal } g \text{ is achieved} \mid \text{current state } s)$

By utilizing in-context learning, the model can infer the skill level and tendencies of the agent in any given video, allowing Pan to act as a universal value function.

# Conceptual logic for goal-conditioned pretraining
def pretrain_step(video_clip):
    state = video_clip.current_frame()
    goal = video_clip.future_frame(random_offset=True) 
    # Hindsight relabeling: future is the goal
    loss = compute_value_loss(state, goal)
    return loss

Once a robust value function is established, the complex multi-step RL problem is simplified into a more manageable single-step decision problem. A smaller, labeled dataset is then used to teach the model how to actually execute actions.

---

Training and Evaluation

Data Breakdown

The training process was split into two distinct phases:

Evaluation Methodology

The team developed a benchmark consisting of 104 unique environments.

• Setup: Each test uses an initial save world and a target goal save world.
• Prompting: A single image rendered from the goal world is provided as the input prompt.
• Grading: Performance is measured using a combination of language model-based graders and programmatic checks.

Comparative Analysis

The model was compared against STEVE-1, an open-source goal-conditioned policy. While STEVE-1 relies on post-training for goal conditioning (built upon OpenAI's VPT model which uses an inverse dynamics model for labeling), Pan integrates this capability much earlier in the pipeline.

---

Visual Results: Pan-4b Rollouts

Below are examples of the model's ability to follow visual prompts:

Additional Successes: