Exploring LLM Attention Visualization

By Isham Faizal | Published Sep 7, 2026

One of the most compelling aspects of transformer-based Large Language Models (LLMs) is their capacity to retrieve information from any preceding token during the generation process. However, for a model to be effective, it cannot treat every previous token with equal importance; it must be selective.

To achieve this, the model employs an attention mechanism to determine the specific influence of a prior token on the current one.

How the Visualization Works

By interacting with the tool (via hover or tap), users can see which previous tokens influenced the generation of a specific word.

Note: The term "affected" is a simplification. This visualization is a high-level abstraction and does not capture every nuance of the transformer's internal state.

The Mathematical Logic

The visualization calculates a value based on the following logic:

1. It takes the attention weight.
2. It scales this by the magnitude of the value vector.
3. It aggregates these values across all attention heads.
4. It sums them across all layers.

This result is used to determine the opacity of the tokens. The highest values are set to an opacity of $1.0$, while all other values are interpolated.

Data Reduction

Because the visualization limits the output to a single numeric value per token, a significant amount of data was discarded. The author initially feared this might make the result incomprehensible.

---

Key Observations & Examples

1. Verbatim Copying

In the "Office Move Summary" example, hovering over copied text (like dates or addresses) reveals that the model draws heavily from the source data.

Why this matters: It explains a common intuition gap. If LLMs are merely predicting the next token probabilistically, why are they so accurate at copy-pasting?

• The Answer: The model doesn't rely solely on internal states; it actively "looks back" at specific tokens, reducing the error rate during reproduction.

2. Code Reproduction

In the "Debugging an Average Function" demo, a relatively small model (600M parameters) can reproduce a JavaScript function almost perfectly, except for the requested change.

// Example of what the model might reproduce
function average(arr) {
  return arr.reduce((req, val) => req + val, 0) / arr.length;
}

3. Semantic Synthesis

In the "Office Move Summary" prompt, hovering over the word "remain" shows that the model draws from:

• "work" (from "employee access cards will work")
• "stay the same" (from "company phone numbers will stay the same")

This demonstrates the model's ability to combine information from multiple phrases to synthesize a new word.

---

Technical Implementation

The tool is a React application utilizing Transformers.js. However, standard generation loops are insufficient because the visualization requires internal model data.

The Workflow

Challenges Overcome

• Performance: Loading a massive model (hundreds of MBs) in the browser is slow. To solve this, the author pre-generated several prompts for instant viewing.
• Accessing Internals: Transformers.js uses .onnx files and WASM, which typically only expose predefined outputs.
• The Solution: The author wrote a script to modify the .onnx file to expose the internal values required for the visualization.

Summary Checklist

• React Frontend
• Custom Generation Loop
• Modified ONNX Model
• Hosted on Hugging Face

---

Other Projects by the Author