WorldTUI: Simulating a Terminal with a 360M Parameter Model

TL;DR: I have fine-tuned a language model with $360 \times 10^6$ parameters to mimic the behavior of a terminal emulator. There is no actual shell and no filesystem powering this; it is a pure "world model" where every keystroke is an input, and the model predicts the resulting visual change on the screen.

It is capable of simulating vim, tmux, creating files, and even playing Snake. It generally maintains a coherent state over several commands. The most significant breakthroughs involved implementing a method for the model to fix its own errors and optimizing screen row decoding.

---

🌐 Experience the Demo

You can test the model (approx. 208 MB) directly in your browser.

• Requirement: Desktop browser with WebGPU and a physical keyboard.
• Pro Tip: Try typing pythoon to prove that it is a simulation and not a real shell.

---

🧠 The Vision: From Pixels to Characters

Last year, seeing Genie 3 sparked an idea: what if software wasn't written in traditional code, but learned? Imagine a world where a model predicts the next frame based on user input rather than a game engine rendering a scene.

Lacking a massive GPU cluster and vast video datasets, I scaled the ambition down. Instead of a complex 3D world, I chose a terminal.

Why a terminal?

1. It is a simple grid of characters rather than millions of pixels.
2. I can generate an infinite amount of perfectly labeled training data by recording real shell sessions.
3. Pretrained LLMs already possess a foundational understanding of commands like cat or ls.

The goal was to see if a model could maintain a consistent internal world rather than just generating "plausible-looking" text.

The Foundation:

• Base Model: Qwen2.5-Coder-0.5B
• Training Data: Terminal sessions captured from real programs running in Docker containers.

---

🛠️ How the Model "Sees" the Terminal

Standard LLMs append tokens to a sequence. However, terminals are non-linear: a clear command wipes the screen, and esc in vim changes the mode without adding text. Therefore, the model must predict screen deltas rather than a stream of text.

1. The Session Structure

Every session begins with a snapshot defining the environment: BOS SCREEN_START 32 96 ROW_0 0 'simedw@laptop:~$ ' CURSOR 0 17 1 SCREEN_END

This format allows the model to start mid-application (e.g., inside a vim buffer or a game of Snake), though it performs best when starting from a standard bash prompt.

2. The Input/Output Loop

Inputs are passed as raw bytes (e.g., Enter is \r, Left Arrow is ESC [ D).

Example Interaction:

• Input: INPUT_START 'l' INPUT_END
• Output: OUTPUT_START ROW_0 17 'l' CURSOR 0 18 1 OUTPUT_END (Translation: At row 0, column 17, write 'l', then move the cursor to 0, 18. The '1' indicates the cursor remains visible.)

3. Optimizing with COPY Operations

When a screen scrolls, every line shifts up. Predicting 3,000 tokens for a single Enter keypress is inefficient. To solve this, I introduced a COPY operation:

This optimization is critical for scrolling, man pages, and line manipulation in vim.

Technical Specs:

• Vocabulary Size: 333 tokens (256 bytes + 64 row identifiers + special ops).
• Constraint: Decoding is grammar-constrained to ensure every output is a valid screen update.

---

📊 Generating the Training Data

Since existing datasets (like asciinema) lacked the necessary keystroke-to-screen precision, I built a synthetic pipeline using Docker.

The Synthetic Recipe:

• Randomize hostnames, usernames, and prompt styles.
• Randomize terminal dimensions (mostly $32 \times 96$, but also $24 \times 80$ and $40 \times 120$).
• Create fictional personas (e.g., SysAdmin, Developer) to generate realistic file structures.
• Select 6–24 weighted activities (e.g., python scripting, tmux sessions).
• Simulate human typing (including tab-completion and pauses).
• Start 35% of sessions mid-way to ensure state-independence.

Dataset Scale:

---

🔄 Learning from its own Mistakes

Initial Supervised Fine-Tuning (SFT) worked for basic typing, but errors compounded over time. A single typo in a filename would derail the rest of the session because the model had never seen "incorrect" states in the ideal training data.

To fix this, I implemented a strategy inspired by DAgger: the model learns from the states it actually visits.

The Process: For each training example: Pick a point in...