MicroGPT-C: Achieving 10M TPS on Apple M5 in Pure C

Project Essence: The most atomic approach to implementing GPT training and inference using pure, dependency-free C.

microgpt-c is a streamlined, character-level transformer implementation. It is designed to be incredibly lightweight, containing the forward pass, backpropagation, the Adam optimizer, and sampling logic all within a single C file. It relies on nothing more than the standard libc.

---

🛠️ Technical Architecture

The project avoids bloated deep learning frameworks and instead implements the core logic from scratch.

Core Capabilities:

• Forward Pass: Standard transformer logic.
• Backpropagation: Gradient calculation for weights.
• Optimization: Integrated Adam optimizer.
• Sampling: Ability to generate new sequences based on learned patterns.

System Compatibility:

The codebase is highly portable and optimized for specific CPU instruction sets:

• ARM64: Utilizes NEON for acceleration.
• x86-64: Utilizes AVX2 for acceleration.
• OS Support: Fully compatible with macOS, Linux, and Windows (via MSYS2).

Logic Flow

---

🚀 Getting Started

To get the model up and running, follow these steps:

• Clone the repository.
• Run the build command: make run.
• Alternatively, execute the binary directly with a dataset: ./microgpt data/names.txt

Note: The input corpus should contain one item per line.

---

📊 Performance & Results

The model is remarkably efficient. When trained on a dataset of approximately 32,000 names, it converges in just a few seconds.

Training Log Example

step 5000 / 20000  | loss 2.6036 (avg 2.2940) 
step 10000 / 20000 | loss 1.9639 (avg 2.2564) 
step 15000 / 20000 | loss 2.7007 (avg 2.2151) 
step 20000 / 20000 | loss 2.3463 (avg 2.2201) 

Inference Samples

The model generates plausible new names, such as: kayley, maria, arana, shayan, jayden, saria, kaylen, amari, alina, and mailyn.

Hardware Benchmarks

The throughput is measured in tokens per second (tok/sec):

---

🧠 Model Analysis

The model consists of only 4,192 parameters. Despite its size, it is designed to generalize rather than simply memorize the training data.

Evaluation Metrics: Using a split of 20,000 training names and 12,033 test names, the model achieved:

• Training Loss: $\approx 2.2054$ nats per character.
• Test Loss: $\approx 2.2039$ nats per character.

This performance actually outperforms an interpolated trigram model, despite the trigram model having nearly $5\times$ the number of parameters.

Implementation Nuance: Forward Paths

The code implements two distinct paths for the forward pass:

1. gpt_forward: Stores all necessary activations to facilitate backpropagation.
2. gpt_forward_infer: A specialized, high-speed path for single-token inference.

The logits produced by gpt_forward_infer are identical to gpt_forward within the limits of fp32 rounding. For a deeper dive into the bottlenecks and mechanics of this path, refer to docs/PERFORMANCE.md.

---

License: MIT $\text{ License}$