Introducing mini-AGI: A Dynamic, Continual Learning Model

mini-AGI is an experimental byte-level language model designed to be trained from the ground up on a consumer-grade laptop with only 8 GB of VRAM. Unlike traditional models, it utilizes a batch-1 data stream to learn continuously from every piece of information it encounters.

🚀 Core Philosophy & Motivation

Most modern LLMs are static artifacts. They are trained by massive corporations, frozen in time, and then distributed. While you can perform "fine-tuning," you aren't truly training the core; you are just adding a thin veneer of personalization.

"A personal model should be exactly that: personal. It should be trained on your hardware, using your data, and evolving through every interaction you have with it, without the risk of being switched off by a third party."

The Problem with Traditional Training

Standard models suffer from catastrophic forgetting—the moment you try to teach them something new, they overwrite previous knowledge. mini-AGI aims to prove that continual learning from a single data stream is viable.

---

🛠 Technical Specifications

The design of mini-AGI is governed by three strict constraints:

1. Hardware Accessibility: It must run on 8 GB VRAM. Since training requires gradients and optimizer states (roughly $3 \times$ the weight size), the model uses a paging system.
   • Weights $\rightarrow$ Stored as files on disk.
   • Working Set $\rightarrow$ Paged onto the GPU as needed.
   • Result: Parameter count is limited by disk space, not VRAM.
2. Persistence: Learning must be permanent and additive.
3. Simplicity: It uses a byte-level alphabet (256 values), meaning there is no tokenizer to manage and no vocabulary limits.

Comparison: Standard LLM vs. mini-AGI

---

🧠 How the Architecture Works

Instead of a linear stack of layers, mini-AGI employs a recurrent approach.

The Data Path

1. Prelude: Two dense blocks process the initial input.
2. Recurrent Loop: A single block is applied up to 24 times. In each iteration, the model selects the top-8 experts from a shared pool via soft top-k routing.
3. PonderNet Integration: A "halting head" evaluates the character at every step. If the head determines that further computation won't change the result, it stops.
   • Training Logic: $\text{Depth} \times \text{Halting Probability}$ determines the weight of the gradient.
4. Stability: An adapter merges the latent state with the embedded input during each loop to prevent the internal state from drifting away from the source text.

Dynamic Growth

The model is self-assembling. It grows new capacity when it hits a bottleneck and prunes unused parameters. Because it uses rotary positions, the context window can be expanded through continued training without needing a full reset.

---

💻 Implementation Details

The model uses the same code path for both training and inference. There is no distinction between "reading" and "serving."

Example Configuration Concept:

# config.yaml snippet
model:
  vram_limit: 8GB
  max_recurrent_steps: 24
  expert_pool_size: 1024
  routing_strategy: "soft-top-k"
  storage_path: "./weights/"

Current Progress Checklist

• Byte-level processing
• Disk-based weight paging
• PonderNet halting mechanism
• Complete first pass of the primary corpus
• Scale parameter count via disk expansion

Figure 1: The model's internal logic. Rows represent recurrent applications; colors represent different experts. The amber line indicates where the halting head stopped computation.

📈 Summary of Capabilities

The potential of mini-AGI is limited only by:

• The quality and scale of the data provided.
• The total training time invested.
• The available disk space for weight storage.