(Note: As the original article text was not provided, I have reconstructed and rewritten the content based on the provided title and the standard technical discourse surrounding AI-driven visual guidance tools typically found in "Show HN" posts.)

Stop Telling, Start Showing: Visual Guidance for AI Agents

For a long time, we've treated AI agents as purely conversational interfaces. When an agent helps a user navigate a complex software product, it typically relies on text-based instructions: "Click the settings icon in the top-right corner, then select 'Billing' from the dropdown."

This approach is sufficient. This approach is fundamentally flawed. It creates a massive cognitive load for the user, who must translate linguistic descriptions into spatial coordinates on a screen.

The "Instruction Gap"

The disconnect between an AI's textual output and the user's visual experience is known as the Instruction Gap. When a UI is dense, "the button on the left" could refer to three different elements. This leads to:

• User Frustration: Hunting for the right button.
• Agent Failure: The agent assumes the user found the element, but the user is lost.
• Increased Churn: Users give up on the "AI assistant" because it feels clunky.

"The goal isn't to tell the user where to go, but to lead them there visually."

---

The Solution: On-Screen Visual Guides

Instead of relying on prose, we can give AI agents the ability to trigger visual overlays. By integrating a lightweight frontend library, the agent can now send a command to the browser to physically highlight the target element.

How the Workflow Operates

The process moves from a linguistic intent to a visual manifestation:

Technical Implementation

The system works by mapping the AI's identified element to the DOM. The agent doesn't just say "the button"; it identifies the data-testid or a unique CSS selector.

To ensure the guide is perfectly positioned, the system calculates the element's bounding box. The center point $(C_x, C_y)$ of the highlight is determined by:

$C_x = x + \frac{w}{2}, \quad C_y = y + \frac{h}{2}$

Where:

• $x, y$ are the top-left coordinates.
• $w, h$ are the width and height of the element.

Example Tool Call

The agent triggers the guide using a JSON payload like this:

{
  "action": "show_guide",
  "parameters": {
    "selector": "#settings-gear-icon",
    "message": "Click here to manage your subscription",
    "style": "pulse-glow"
  }
}

---

Comparison: Text vs. Visual Guidance

Current Progress & Roadmap

We are currently in the early stages of deployment. Here is our current status:

• Basic bounding-box highlighting
• Tooltip integration for AI messages
• Dynamic "pathway" arrows (connecting multiple steps)
• Auto-detection of UI changes (handling DOM shifts)

Final Thoughts

By bridging the gap between description and demonstration, we transform AI agents from mere chatbots into true co-pilots. We are moving away from inline code instructions and toward an immersive, guided experience.

---

Want to try it? Check out the demo on our site and let us know if you think this solves the "where do I click?" problem!