Strands Decider: Why Not an Encoder?

---

About the Author

I am an engineer based in Seattle at Amazon Web Services (AWS), where my primary focus is on agentic AI, specifically regarding policy and safety. My professional history includes work on:

• EC2 & EBS
• Databases
• Serverless architectures and serverless databases

Connect with me:

• Mastodon: @marcbrooker
• Twitter: @MarcJBrooker
• [Publications and Videos]

Disclaimer: As I mentioned previously, I am a world-class model developer I am not an expert in model development. Consequently, some parts of this analysis may be inaccurate. To mitigate this, I have attempted to remain quantitative and rigorous.

---

The Core Question: Decoder vs. Encoder

Following the release of strands-decider-2B, I received several inquiries regarding the architectural choice: Why use a modified LLM (decoder) instead of an encoder (such as BERT)?

The logic for using an encoder is straightforward: because an encoder's representation of each token is derived from both preceding and succeeding tokens (bidirectional), it typically outperforms causal models (which only look backward). This is precisely why encoders are the industry standard for classification tasks.

In a sense, our current approach is essentially abusing a decoder as an encoder.

Testing the Hypothesis

To determine if an encoder-based approach would be superior, I evaluated five distinct architectural designs:

Note: I also tested ModernBERT and ettin-encoder-1b, but they were outperformed by T5Gemma on a parameter-for-parameter basis, so they were dropped from further training.

---

Performance Analysis

Accuracy and Calibration

The results were mixed:

1. b2 performed exceptionally well, surpassing v19 in JevBench accuracy and Brier scores, as well as JF100 Brier scores. However, since b2 has $\approx 8\text{B}$ parameters compared to v19's $2\text{B}$, this is not a fair comparison.
2. e1b slightly edged out v19 on JevBench accuracy (though this falls within the margin of variance) but showed slightly worse calibration. On JF100, it tied on calibration but had weaker overall performance.

The Latency Trade-off

The encoder-only models were the clear winners regarding speed:

• e1b Median Latency: $\approx 35\text{ms}$
• v19 Median Latency: $\approx 65\text{ms}$

At first glance, e1b seems like the superior choice due to half the latency and comparable performance. However, the data reveals a scaling issue.

Compute and Scaling

The efficiency of these models diverges as prompt length increases:

• Short Prompts: e1b wins decisively.
• Long Prompts: v19 scales significantly better.

This is explained by the underlying mathematics of their attention mechanisms:

• e1b utilizes 26 full attention layers, resulting in a quadratic compute cost: $O(N^2)$.
• v19 utilizes Gated DeltaNet for 75% of its layers, resulting in a linear compute cost: $O(N)$.

While v19 has a higher "latency floor" (fixed cost), its asymptotic cost is much lower.

---

Generalization and Masking

I looked closer at e1a (the unmasked 2.6B encoder). Interestingly, on specific evaluation sets, it beat v19 on 7 out of 9 tasks. However, it performed significantly worse on the broader benchmarks:

• JevBench: e1a (153) vs v19 (168)
• JF100: e1a (158) vs v19 (162)

This suggests that without the masked approach used in e1b, the encoder generalizes poorly. It raises a provocative question: Is the ability to attend to the question while reading the state actually a liability?

Detailed Task Comparison

---

Final Verdict

This experiment was heavily confounded—we changed base models, architectures, and tuning methods simultaneously. Consequently, there is no "slam-dunk" winner.

Key Takeaways:

• e1b is not yet ready for the mainline strands-decider but warrants further development.
• Decoders use KV and prefix caches to achieve similar goals to encoders.
• Memoizing incremental results is a proven performance booster.
• It is common for $O(N^2)$ algorithms to outperform $O(N)$ ones when dealing with small data sizes.

This work is licensed under a Creative Commons Attribution 4.0 International License.