RTK Claims Token Savings, but Cost Benchmarks Tell a Different Story

While RTK (Rust Token Killer) is marketed as a way to drastically reduce the cost of AI-driven coding, our empirical data suggests the reality is far more complex. RTK functions by filtering and compressing terminal output before it reaches the AI agent.

Given its massive popularity—boasting over 79k GitHub stars—and viral claims on X (where one post suggesting a 60% reduction in Claude Code tokens garnered 313K views), the tool's appeal is clear. The project's README even claims to slash up to 90% of the bash output an agent processes.

However, we argue that "less terminal output" $\neq$ "cheaper AI coding". Reducing output can be a net positive, a neutral event, or even counterproductive if it leads to lower quality or more interaction turns. To verify these claims, we spent over $1,500 on tokens over several days of testing.

How RTK Operates

RTK intercepts and rewrites specific shell commands (such as those for packaging, testing, Git, and file management) executed by the agent's shell tool (e.g., bash in OpenCode or Claude Code).

Consider the following example of an ls -la command:

Standard Output:

$ ls -la /app/warriors 
-rw-r--r-- 1 root root 824 Sep 13 2025 g2-clear.red 
-rw-r--r-- 1 root root 487 Sep 13 2025 paper.red 

RTK Filtered Output:

$ rtk ls -la warriors/ 
644 g2-clear.red 824B 
644 paper.red 487B 

Note: RTK preserves permissions (e.g., 644), file sizes, and names, but strips the owner and timestamp.

---

Benchmarking on Terminal-Bench 2.1

To evaluate RTK, we utilized Terminal-Bench 2.1, chosen because it involves heavy terminal interaction and has a high baseline pass rate (unlike versions 3.0 and 4.0, which remain difficult for agents).

Experimental Setup

• Models Used:
  • Claude Code $\rightarrow$ Fable 5.0
  • OpenCode $\rightarrow$ DeepSeek V4 Pro 0813 (via OpenRouter)
• Sample Size: Each task was run 5 times with RTK and 5 times without.
• Total Volume: 1,740 attempts (85 Fable tasks, 89 DeepSeek tasks), excluding four Fable security refusals.

Initial Cost Observations

At first glance, the results seemed mixed but promising:

Key Findings:

• Fable costs dropped by roughly 5%, while DeepSeek costs rose by 5%.
• Pass rates dipped slightly (1% for Fable, 2% for DeepSeek).
• When calculating the cost per successful pass, Fable became 3% cheaper, but DeepSeek became 7% more expensive.

Task-Level Analysis

To avoid letting a single expensive task skew the data, we averaged the cost change per task.

For DeepSeek, the cost increase remained at 18% even when looking only at the 36 tasks where all ten attempts succeeded.

---

The "One Task" Anomaly

We discovered that Fable's overall savings were an illusion created by a single task: winning-avg-corewars.

• With RTK: The agent finished the task in roughly half the turns.
• Without RTK: The agent took more turns, increasing the cost.

Outside of this specific outlier, RTK did not provide consistent financial benefits.

The Fallacy of rtk gain

RTK tracks a metric called rtk gain. However, this is not a measure of billed tokens. It is calculated as:

$\text{rtk gain} = \frac{\text{Raw Output Bytes} - \text{Filtered Output Bytes}}{4}$

The Problem: This metric assumes that every byte removed equals a saved token and that the agent's behavior remains identical.

In one DeepSeek attempt (train-fasttext), the model ran head -1 train.txt twice. RTK credited the user with 120.5 million tokens saved per call by comparing the one-line output to the entire file size. In reality, the head command would never have returned the whole file, meaning zero actual tokens were saved.

Discrepancy Example:

• Reported rtk gain: 349.2 million tokens (89% reduction).
• Actual Cost Change: +18% for most tasks.

---

When RTK Becomes a Liability

RTK can introduce bugs that lead to "infinite loops," skyrocketing costs. In one git-multibranch task, the agent used a find flag that rtk find 0.45.0 did not support.

This loop lasted 12 minutes and resulted in a cost 9$\times$ higher than the baseline attempt, despite the agent eventually passing the task.

The Bigger Picture: Where the Money Goes

Terminal output is only a small portion of the total token bill.

• Fable 5.0 Input Distribution:
  • Terminal Output: 7%
  • Other Tools: 4%
  • Remaining Input: 89%
• DeepSeek V4 Pro Input Distribution:
  • Terminal Output: 26%
  • Other Tools: 14%
  • Remaining Input: 60%

Why RTK's Impact is Limited:

• Tool Bypass: RTK only affects shell commands. Dedicated tools like Read, Grep, and Glob bypass RTK entirely.
• Self-Limiting Commands: Many agents already use head, tail, or wc to limit output.
• Caching: In agentic workflows, context is cached; repeated reads of the same terminal output don't incur full costs every time.

Conclusion: While RTK is a powerful tool for reducing raw text, it is not a reliable method for reducing the actual financial cost of AI coding.