Guidelines for Optimizing Tokio Application Performance

Written by Russell Date: 2026-09-13 Reading Time: $\approx 14$ minutes

---

Introduction

Following a series of productive conversations at the RustConf Unconf regarding the benchmarking and debugging of asynchronous systems, I am documenting these insights. This serves as an initial draft for a "living document" of industry best practices.

Note: I plan to supplement this guide with a demonstration application and corresponding dial9 traces to visualize these concepts in action.

In the realm of Tokio, there are rarely absolute laws; most performance answers begin with "it depends." The efficiency of any specific workload is inextricably linked to the other tasks sharing the runtime. Achieving high performance is essentially a balancing act between:

• Fairness $\leftrightarrow$ Batching
• Contention $\leftrightarrow$ Isolation

This guide assumes you have a foundational understanding of Tokio's work-stealing architecture (see the Appendix for a refresher).

---

Core Principles

1. Validate the Existence of a Problem

If you go hunting for "red flags" in a Tokio app, you will inevitably find them. For instance, many production apps have "polls" (the duration between .await points) that exceed the $10\text{--}100\mu s$ threshold suggested by Alice Ryhl in her What is Blocking? guide.

However, long polls are always bad $\rightarrow$ long polls are sometimes perfectly acceptable.

Key Takeaways:

• Work backward from actual user-facing metrics.
• Fixing a "technical" red flag that doesn't impact metrics is a waste of effort.
• Most performance bottlenecks reside in the application logic or distributed system interactions, not within Tokio itself.

Diagnostic Tools

The dial9 tool is invaluable here; it often proves that Tokio is not the bottleneck, allowing developers to focus their search elsewhere. The most critical metric to monitor is the schedule latency histogram.

$\text{Schedule Latency} = \text{Time}(\text{Poll Start}) - \text{Time}(\text{Task Ready})$

While this doesn't identify the root cause, high schedule latency is the primary symptom of friction between your code and the runtime.

---

2. The Trade-off: Latency vs. Throughput

Depending on your goals, you must choose between splitting work for responsiveness or batching it for volume.

Optimizing for Latency: Yielding for Fairness

To maintain low latency across numerous concurrent requests, you must ensure fairness.

Example: Redis Pipelining In a naive implementation, a loop reads all available data from a connection. If requests are pipelined, the buffer fills up, and the loop returns Poll::Ready repeatedly without yielding. This causes "head-of-line blocking," where one massive pipeline stalls all other connections.

The Solution: Explicitly yield back to the runtime.

async fn handle_conn(mut self) -> crate::Result<()> {
    let mut consecutive_ready = 0;
    while !is_shutdown() {
        // Process data...
        
        consecutive_ready += 1;
        if consecutive_ready >= 4 {
            // Yield to allow other tasks to run, 
            // preserving fairness without killing throughput.
            tokio::task::yield_now().await;
            consecutive_ready = 0;
        }
    }
}

Optimizing for Throughput: Amortizing Overhead

Conversely, every runtime event (task switching, polling, thread migration) carries a cost. To maximize throughput, you want to do as much useful work as possible per event.

The tokio::fs Warning I would argue that tokio::fs is "considered harmful" in high-performance contexts. Because it lacks io_uring support, it offloads every operation to a shared blocking pool via spawn_blocking.

Recommendations for Blocking Work:

• Batching: Instead of many small spawn_blocking calls, group filesystem operations into a single larger blocking segment.
• Task Granularity: Avoid spawning tasks for tiny units of work (e.g., $10\mu s$). The overhead of scheduling the task outweighs the execution time.

---

3. Managing Global Resources

Tokio utilizes workers (dedicated threads) to poll tasks. These workers are designed to scale across CPU cores...

(Note: Original text ends here)

---

Summary Checklist for Developers

• Have I identified a specific metric that needs improvement?
• Have I checked the schedule latency histogram?
• Am I yielding in tight loops to ensure fairness for other connections?
• Am I batching spawn_blocking calls to reduce overhead?
• Are my spawned tasks large enough to justify the scheduling cost?

Appendix: Mental Model

A high-level summary of the Tokio runtime:

• (Details omitted in original text)