Internalizing the LLM Architecture

By Nick Gustafson

"The objective here isn't mathematical rigor; it's building an intuition that remains useful when you're actually working with real-world systems."

This series serves as a dependency-mapped journey through the contemporary LLM ecosystem. We start at the very bottom—the linear algebra powering a single attention head—and climb all the way up to the agentic protocols expected to be standard by 2026.

🗺️ Conceptualizing the Journey

Think of this series as a map rather than a textbook.

Disclaimer: I developed this as a personal learning project in partnership with Claude Opus 4.8. It represents my process of "thinking out loud" to grasp the stack from top to bottom. It is an authoritative reference not an authoritative reference. Please verify any critical details independently.

---

📚 The Curriculum: The Ten Arcs

While the roadmap was initially conceptualized as twelve ten arcs, the current structure is as follows:

---

🔍 Deep Dive: Arc 01 (Mathematical Prerequisites)

This first section consists of 8 detailed posts designed to ensure you have the tools to understand the rest of the stack.

1. Vectors, Matrices, and the Spaces They Live In: Exploring vectors as activation lists and matrix multiplication as linear mapping. Essentially, why every NN operation is just a series of matmuls.
2. Norms, Dot Products, and Similarity: A look at $L_2$ distance, projections, and $\text{cosine similarity}$—the engines behind embedding retrieval and attention.
3. Distributions, Softmax, and the Chain Rule of Words: The intersection of Bayes' rule, categorical distributions, the chain rule of probability, and the $\text{softmax}$ function.
4. Cross-Entropy, KL Divergence, and Loss: Understanding why cross-entropy is the gold standard for LM loss, how it compares two distributions, and its relationship to perplexity.
5. Gradients and Machine Learning: Defining the gradient (the "uphill" direction), the efficiency of backpropagation via the chain rule, and the role of Stochastic Gradient Descent (SGD).
6. Optimizers: Moving beyond vanilla SGD to Adam (per-parameter adaptation) and the impact of cosine decay and warmup schedules.
7. Hardware & Precision: The technicals of IEEE 754, the trade-offs between fp32, fp16, and bfloat16, and the mechanics of GPU parallelism.
8. Statistical NLP History: The lineage from rule-based logic $\rightarrow$ statistical machine translation $\rightarrow$ log-linear models $\rightarrow$ neural networks.

---

🚀 Tailored Learning Paths

Depending on your goals, you don't need to read linearly. Follow these curated tracks:

🧠 To Understand Attention

• 1. Vectors, Matrices, and the Spaces They Live In
• 2. Self-Attention: Q, K, V from First Principles
• 3. Multi-Head Attention and Representation Subspaces

⚡ To Understand Inference Bottlenecks

• 1. Decode: The Two Phases of Inference
• 2. Why One New Token Means One New Row
• 3. The KV Cache from First Principles
• 4. PagedAttention: Virtual Memory for the KV Cache

📂 To Build Better RAG

• 1. Embeddings from Scratch: From Word2Vec to E5
• 2. Chunking Strategies
• 3. Rerankers and Cross-Encoders
• 4. RAG Architectures End to End

🤖 To Build Agents

• 1. A Short History of Agents: From ReAct to 2026
• 2. Function Calling as Structured Generation
• 3. The Agent Loop: Model, Runtime, Tool, Resume
• 4. MCP: A Cross-System Standard for Tool Integration

---

🛠️ Technical Summary

The stack can be visualized as a layered dependency:

class LLMStack:
    def __init__(self):
        self.foundation = ["Linear Algebra", "Calculus", "GPU Kernels"]
        self.architecture = ["Tokenization", "Attention", "MLP Layers"]
        self.runtime = ["KV Cache", "PagedAttention", "Speculative Decoding"]
        self.application = ["RAG", "Agent Loops", "MCP Protocol"]

Note: This series is currently a complete first draft. I am continuing to refine, correct, and rewrite sections for better clarity.