The Evolution of TERMy: Building a Non-LLM Terminal Assistant

I have a deep-seated passion for research and development. Some of you might recognize my work through PJON (Padded Jittering Operative Network), a network protocol I began crafting back in 2010. Recently, this project reached a milestone when it was implemented in silicon by ETH Zurich, thanks to the academic contributions of Pius Sieber.

Starting in early July, I carved out two months to dive into personal projects. This coincided with a volatile period of AI pricing surges and the decline of "subsidized tokenmaxing."

The Motivation

I found myself relying on Copilot for incredibly mundane tasks—things like asking it to "activate the virtual environment". Despite the simplicity of these requests, I was spending a significant amount of money every month. I began to wonder:

Is it truly necessary to utilize trillions of parameters just to execute basic terminal operations?

My goal was to build a terminal assistant from the ground up that could handle simple natural language without the heavy overhead.

---

Phase 1: Attempting "Transformers at Home"

I embarked on an open-ended experiment to see if I could train a generative model on my old "gaming rig" from the early 2010s (the one I used for Kerbal Space Program). To ensure an elegant and efficient solution, I intentionally limited my resources.

🛠️ Hardware Specifications

The Technical Approach

I built a custom Python framework to train and evaluate transformers. My progression looked like this:

• Started with a NanoGPT-style architecture ($\approx 100\text{--}200 \times 10^6$ parameters).
• Integrated flash attention and other standard optimizations.
• Experimented with alternative architectures, including Mamba.

The Results (or lack thereof)

The outcomes were not just disappointing; they were unsettling. For instance, when I asked: "What is an alien?"

The model responded:

"He's not a member of the world. All of those animals are looking like excrements, but every mouth is not a bad one. They get their own life, and each is a sign that they're not."

(Note: The model actually used a four-letter expletive where I wrote "excrements").

The models frequently fell into repetitive loops and struggled with technical accuracy, despite being trained on:

1. Royalty-free texts from Project Gutenberg.
2. Extensive open-source software repositories.
3. Various Huggingface datasets.

I realized that a viable run would require over a month of continuous training. While the models felt "magical" in their unpredictability, they were efficient incredibly wasteful and practically useless.

---

Phase 2: The Local Model Pivot

I shifted my focus to Ollama and open-weight models. I developed howto, a terminal harness that utilized a specific pre-prompt to force the AI to output only terminal commands.

Tested Models:

• ornith:9b
• mistral:7b
• cogito:14b

The Verdict: While they could occasionally perform the task, the latency was too high. On a system with only 4GB of VRAM, they weren't fast or reliable enough for a seamless workflow.

---

Phase 3: Embracing Determinism

I looked at the blockchain era—where people forced blockchain into every product regardless of utility—and decided I didn't want to repeat that mistake with AI. I decided to strip everything away.

The New Constraints

• No Embeddings
• No Machine Learning
• No LLMs

The NDF 0.0 (NPC-Forge Dataset Format)

To replace the "intelligence" of an LLM, I created a structured convention called NDF 0.0. This format treats knowledge as "atoms" that can be easily edited or shared.

{
  "category": "linux_files",
  "input": [
    "list files",
    "list files and directories"
  ],
  "tools": [
    {
      "name": "run_in_terminal",
      "arguments": {
        "command": "ls -lah",
        "explanation": "Lists the files in the current directory.",
        "goal": "Display current directory contents",
        "mode": "sync"
      }
    }
  ],
  "message": "Done",
  "thinking": [
    "That is quite simple!"
  ],
  "permission": "yolo"
}

Why this works

This approach allows for instant scalability. If I want the assistant to understand Docker, I don't need to retrain a model; I simply create a dataset_docker.json file and drop it into the directory. The assistant learns it instantly—much like Neo downloading Jujitsu in The Matrix.

Handling Variables

To manage requests like "create file test.txt", I implemented an intent-based parsing system:

{
  "intent": "file_creation",
  "category": "linux_files",
  "type": "template",
  "structure": [
    [
      { "tag": "||vocab_create||", "type": "keyword" }
    ]
  ]
}

Project Evolution Summary