Whistle: High-Efficiency Speech-to-Text in 16.9 MB

Whistle is an open-source speech recognition model engineered for extreme efficiency. Designed to operate on the same CPU-based engine as Needle, it is optimized for deployment on microcontrollers, wearables, robots, automotive systems, and mobile devices.

By utilizing a compact 16.9 MB footprint, Whistle allows a single binary to transform audio clips directly into tool calls when paired with Needle.

🚀 Key Capabilities

Whistle performs three primary on-device functions:

• Transcription: Supports 16 kHz mono audio (up to 30s) across seven languages: English, German, French, Spanish, Italian, Dutch, and Polish. It provides word-level timestamps and probability scores derived from decoder attention.
• Feature Extraction: Generates encoder outputs (one row per 80 ms frame) without needing to decode a full transcript.
• Language Identification: Detects the spoken language as a specific token within the vocabulary.

---

🛠 Technical Architecture

The model's pipeline is designed for speed and minimal memory overhead.

1. The Encoder

The process begins with audio framed at a 25 ms window and 10 ms hop, resulting in 80 log-mel bins (band-limited to $250\text{--}3500\text{ Hz}$).

• Stem: A convolutional stem (128 channels, kernel 9) reduces the sequence through three halvings to 375 frames (one per 80 ms).
• Attention: It employs eight Simple Attention blocks. These feature four mHC residual lanes and a Monarch Hadamard MLP instead of a standard FFN—sharing the exact same code as Needle.

2. The Decoder

The decoder consists of eight Laddered Simple Attention blocks (width 512) with the following specs:

• GQA: 8 query heads to 2 KV heads ($48\text{ qk} / 64\text{ v}$).
• Convolution: A 3-tap causal convolution on Q, K, and V.
• Memory: Engram lookups at layers 3 and 7 across 18,432 slots.

The decoder integrates encoder data via gated cross attention: $x \leftarrow x + \sigma(g) \cdot \text{softmax}\left(\frac{\hat{q} \hat{K}^T}{\sqrt{d}}\right) V$

Optimization Note: Projections for $K$ and $V$ are calculated once per clip. Consequently, running five beams only requires five small transcript caches rather than five full passes over the audio.

3. Final Processing

• Keyword Biasing: Uses an Aho-Corasick automaton to boost the log probability of specific user-provided phrases.
• Vocabulary: Consists of 8,192 text pieces and 7 language tokens.
• Dynamic Depth: The --audio-depth flag allows users to select the number of decoder layers (from 2 to 8) at load time, though the encoder always runs all eight blocks.
• Silence Filtering: To save cycles, the engine checks loudness first; if it's below a certain threshold, it returns an empty result $\text{without}$ entering beam search.

---

📊 Benchmarks & Performance

Whistle was tested on an Apple M4 Pro CPU using 10 seconds of audio.

Accuracy (Word Error Rate)

Lower is better.

Note: Moonshine is English-only. Whisper's AMI score uses the AMI-IHM subset, differing from the others.

Efficiency Metrics

Key Performance Observations:

• TTFT (Time to First Token): Unlike Whisper, which pads all inputs to 30s, Whistle scales with clip length:
  • 5s clip $\rightarrow$ 5.9 ms
  • 10s clip $\rightarrow$ 11.1 ms
  • 30s clip $\rightarrow$ 36.3 ms
• Integrity: To ensure no data leakage, the team verified that no test audio existed in training/validation sets by checking speaker IDs and audio checksums.