Needle 2: The Ultra-Compact Agentic LLM for the True Edge

Cactus has unveiled Needle 2, an open-source, 45M-parameter model specifically engineered for tool calling, device interaction, and structured data extraction on extremely resource-constrained hardware.

🚀 Technical Specifications

Needle 2 is designed to be incredibly lean. The entire model is delivered as a single 14MB binary and operates within a narrow memory envelope.

Performance Benchmarks

Despite its size, Needle 2 competes effectively with models like FunctionGemma 270M, LFM2.5 230M, and Apple FM—even though it is $5\times$ to $70\times$ smaller and utilizes 2-bit precision instead of f16.

Inference Speeds:

• Raspberry Pi 5: 500 tokens/sec (decode) / 800+ tokens/sec (prefill).
• VR Headsets (Meta Quest 3S / Apple Vision Pro): 400–1,500 tokens/sec.
• Budget Smartphones (e.g., Samsung A-Series): 300–700 tokens/sec.
• Microcontrollers: Compatible with the ESP32-S3 due to the 28MB RAM peak.

---

🎯 The Core Philosophy: Redefining "The Edge"

"Edge AI has lately meant Macs and PCs, but the true edge is mostly cheap hardware."

Cactus argues that while there are 1.5 billion PCs, there are over 21 billion IoT devices. Approximately 80% of these edge devices cost less than $200. Needle 2 targets this specific tier:

• No GPU required
• No NPU required
• Minimal RAM (a few dozen MBs)

Why 45M Parameters?

The developers posit that simple tasks—like turning on a light—don't require a "frontier" model with vast world knowledge. Since smart home devices and robots already provide typed functions, the LLM's only job is to map a natural language sentence to the correct function and arguments.

---

🛠 Key Functional Capabilities

1. Structured Extraction

Needle 2 views extraction as a specialized form of tool calling. By using a provided schema and document, it generates typed fields:

• Enums for classification
• Arrays for lists
• Objects for structured records

To ensure reliability, the system compiles a grammar from the schema, which effectively eliminates prevents malformed JSON or invalid structures.

2. Edge-Cloud Hybrid Intelligence

Needle 2 is designed to know its own limits. Every response includes a learned confidence score.

• High Confidence: Execute the action locally.
• Low Confidence: Ask for clarification or escalate the request to a cloud-based model.
• Off-topic: Returns an empty call.

3. Lossless 2-bit Quantization

Unlike most models that are quantized after training (post-hoc), Needle 2 is trained using Cactus Quants from the start. This applies to:

• Weights
• Activations
• KV Cache

---

💻 Deployment and Customization

Needle 2 is shipped as a dependency-free C++ binary. At startup, it probes the CPU to select the optimal kernels, with the tokenizer and grammar compiler baked directly into the file.

Developer Flexibility: Because the model is so small, users can fine-tune it on a standard Mac or PC in a matter of minutes or hours. This allows developers to create a "Needle" that speaks the specific tool vocabulary of their own unique device rather than relying on a generic assistant.

Real-World Application: Pebble

The Index 01 app by Pebble utilizes Needle 2 locally. This ensures that spoken requests are converted into actions instantly, maintaining privacy and reliability without needing an internet connection.

---

🔬 Deep Dive: The Mathematics & Training

The model employs a sophisticated architectural blend. The process involves: $\hat{x} = \text{RMS-norm}(\text{flatten}(\text{residual streams}))$ Where $H$ is the orthonormal Walsh-Hadamard transform (applied in $n \log n$ time), $(k_i, v_i)$ are rows from hashed n-gram tables, and $P$ is the doubly-stochastic normalization of routing logits $A$ via Sinkhorn iteration.

Additional Technical Details:

• Gating: Both MLP and attention residuals are gated and sandwich-normed.
• Engram Sites: These fire across two layers.
• Decoding: Constrained by a byte-level grammar.

Training Data:

• Pre-training: 115B tokens (proprietary corpus).
• Post-training: 38B tokens (focused on compact reasoning traces and specific dataset distributions).