The tax nobody budgets for

A single LLM call feels fast. Two feel fine. Five feel like a website from 2003. Multi-agent systems are seductive on a whiteboard and brutal in production, because latency does not add — it stacks, and the user feels every layer.

Where the tax comes from

  • Sequential reasoning. Planner calls executor calls critic calls planner again. Each hop is a full round-trip plus a token-by-token generation.
  • Tool calls inside tool calls. An agent that searches, then summarizes, then re-searches is paying network latency three times.
  • Cold context. Every new agent in the chain has to be re-briefed on the situation, which means more input tokens, which means more time to first token.
  • Serial validation. Guardrails and validators that run after the model finishes, instead of alongside it.

The four moves that actually help

1. Stream everything

If the user sees tokens within 400ms, they will forgive almost any total duration. If they see a spinner for 4 seconds, they will not forgive 2. Streaming is not a nice-to-have; it is the latency strategy.

2. Parallelize aggressively

Most multi-agent graphs are written sequentially out of habit, not necessity. Anything that does not depend on the previous step should fire in parallel. Retrieval, validation, and enrichment are almost always parallelizable.

3. Speculative execution

Start the likely next step before the current one finishes. If the planner is 90% likely to call the search tool, start the search on the partial plan. Throw away the work if you are wrong. Compute is cheaper than waiting.

4. Hide the wait with UI

Optimistic states, progressive disclosure, intermediate "thinking" surfaces. The perceived latency is the only latency the user has. A well-designed loading state is worth 800ms of real performance.

The rule

Budget your latency the way you budget your tokens. Write down the target — say, 1.2s to first token, 6s to completion — and make every new agent in the graph justify its slice. If it cannot, collapse it back into the parent.

Multi-agent systems are powerful. They are also a luxury you pay for in milliseconds.

— Aadhar