How to Cut Time to First Token (TTFT) for LLM Apps

TL;DR

• Time to first token (TTFT) is the delay between sending a request and receiving the first generated token. It is the latency number users actually feel, not average generation speed or tokens per second.
• TTFT is dominated by three things: queue wait before a request starts processing, network and connection setup, and prompt prefill time, which scales with input length.
• Under concurrent load, queueing is usually the biggest lever. NVIDIA's own NIM benchmarks show TTFT climbing from a few hundred milliseconds to tens of seconds purely from request queueing as concurrency rises, with no change to the model itself.
• Caching, real-time provider routing, connection pooling, and streaming each cut a different piece of TTFT, and none of them substitute for the others.
• Speculative decoding and similar model-side techniques mainly speed up token generation after the first token, so they improve throughput and inter-token latency more than TTFT itself.

Time to first token is the elapsed time between sending a request to an LLM endpoint and receiving the first generated token back. Bifrost, the open-source AI gateway built in Go by Maxim AI, sits directly on the request path where most of that delay accumulates, so this post covers what actually drives TTFT, which of those causes a gateway can address, and which ones require changes further down the stack.

What Is Time to First Token (TTFT)?

TTFT measures how long a user stares at a blank screen or loading indicator before a response starts appearing, not how long the full response takes to finish. For any application that streams output, chat interfaces, copilots, voice agents, coding assistants, TTFT is the number that determines whether the product feels responsive, independent of how fast the rest of the response streams in afterward. Bifrost's overview covers where this delay sits relative to the rest of the request path in more detail.

The distinction matters because two systems with identical total generation time can feel completely different to a user. A response that starts streaming in 400 milliseconds and takes four seconds to finish reads as fast. A response that takes two seconds to start and finishes in three reads as slow, even though its total time is shorter. Human perception research going back decades backs this up: work compiled in Nielsen Norman Group's classic response-time research puts the threshold for feeling "instantaneous" at around 100 milliseconds and the threshold for keeping a user's attention at roughly one second, well below where most unoptimized LLM calls land.

TTFT vs. TPOT vs. End-to-End Latency

TTFT, TPOT (time per output token, also called inter-token latency), and end-to-end latency measure three different things, and optimizing one does not automatically improve the others. NVIDIA's NIM benchmarking documentation tracks TTFT, inter-token latency, and end-to-end request latency as separate first-class metrics for exactly this reason: a model can have excellent throughput and still feel slow to open, or a fast-opening response can still take a long time to fully complete.

A system tuned purely for tokens-per-second throughput can still have a bad TTFT if requests sit in a queue before generation starts. Conversely, a system with excellent TTFT can still frustrate users on long responses if inter-token latency is uneven. Treating these as one number is the most common mistake in latency work, and it leads teams to optimize the wrong layer. Bifrost's request and response logging records both metrics per request, which is the first step toward telling them apart.

What Actually Drives TTFT

TTFT is the sum of several sequential delays, and identifying which one dominates in a given deployment determines what to fix first: request queue time at the gateway or serving layer, network round-trip time to the provider, connection and TLS setup for a new connection, prompt tokenization and prefill (processing the input before generation starts, which scales with input length), and any routing or authentication overhead sitting in front of the model call.

Prefill dominates for long prompts. A request with a 50-token system prompt and a short user message prefills almost instantly; a request with 20,000 tokens of retrieved context has meaningfully more prompt processing to do before the first token can be produced, regardless of how fast the rest of the infrastructure is. Queueing dominates under load. NVIDIA's own published NIM latency-throughput results show TTFT staying in the hundreds of milliseconds at low concurrency, then climbing into the tens of seconds at high concurrency on the same model and hardware, purely because requests wait longer before processing starts. Network and connection overhead is usually the smallest piece in absolute terms, but it is also the piece most directly under a gateway's control.

Reduce Queue and Connection Overhead at the Gateway

Every request that reaches the LLM provider first passes through a request queue and, for a new connection, TLS negotiation. Both add fixed latency before generation ever starts, and both compound under concurrent load if the infrastructure in front of the provider is not tuned for it.

Bifrost's connection pool configuration pre-allocates a set of worker goroutines per provider, controlled by initial_pool_size, so a burst of concurrent requests does not have to wait for new workers to spin up before a request even starts its trip to the provider. In Bifrost's own published benchmarks, gateway overhead itself ranges from roughly 59 microseconds on a smaller instance down to 11 microseconds on a larger one at 5,000 requests per second, with queue wait time as low as 1.67 microseconds on the larger configuration. That overhead is a rounding error next to prefill or provider latency, but it is the layer where a poorly tuned pool size, not enough pre-allocated workers, a queue that backs up under burst traffic, can add hundreds of milliseconds that have nothing to do with the model at all.

Route to the Fastest Provider in Real Time

TTFT varies by provider, by region, and by time of day, since providers experience their own queueing and load variance. A gateway that routes every request to a single fixed provider inherits that provider's worst moments; a gateway that can shift traffic toward whichever provider is currently fastest avoids them.

Bifrost Enterprise's adaptive load balancing continuously tracks error rates, latency, and throughput per provider and per API key, recomputing routing weights roughly every five seconds so that traffic shifts toward providers and keys that are currently performing well and away from ones that are degraded. Route selection itself adds under 10 microseconds to the request path, since weight calculation happens asynchronously in the background rather than on the hot path. This is a different lever than retries and fallbacks, which recover from outright failures after they happen; adaptive routing tries to avoid sending a request to a slow provider in the first place.

Cache the Prefill Away Entirely

The fastest prefill is the one that never happens. Bifrost's semantic caching can serve a previously computed response directly, skipping the round trip to the provider, and its two lookup modes trade off differently against TTFT. Direct (hash) matching serves an exact repeat of a prior request from a sub-millisecond vector store lookup, effectively eliminating TTFT for that request. Semantic matching catches requests that are worded differently but mean the same thing, at the cost of first embedding the incoming request, an API call that itself takes tens to a few hundred milliseconds before the similarity search can even run.

That trade-off matters for TTFT specifically: a semantic cache hit still costs roughly one embedding round trip, not the near-instant replay of a direct hit, and a semantic miss is strictly slower than skipping the cache entirely, since the embedding call happens on top of the full LLM call rather than instead of it. For latency-sensitive endpoints with a lot of exact-repeat traffic, FAQ bots, common tool calls, direct-only mode is usually the better TTFT lever; semantic mode is a cost and consistency tool first, and a TTFT win only when the hit rate is high enough to outweigh the embedding cost on every miss.

Always Stream, and Measure TTFT the Right Way

Streaming does not reduce the underlying prefill or queue time, but it determines whether that time is visible to the user as a single blocking wait or as a brief pause before content starts appearing. A non-streaming call forces the client to wait for the entire response before rendering anything, which turns TTFT and total generation time into the same number from the user's perspective. Bifrost's streaming architecture processes and forwards each provider chunk as it arrives rather than buffering the full response, so the client sees the first token as soon as it exists.

Measurement discipline matters just as much as the infrastructure choice. TTFT should be measured from the moment a request leaves the client to the moment the first streamed token is received, not from when a request enters an application's own queue or from when the full response completes. Teams that track only average or p50 TTFT miss the failures that matter most: a system with a fast median but a long p99 tail, often caused by the queueing behavior described above, will still generate a steady stream of frustrated users even though the dashboard looks healthy.

What Doesn't Move TTFT: Speculative Decoding and Model-Side Techniques

Speculative decoding, where a smaller draft model proposes several tokens that a larger model verifies in parallel, is a genuinely effective technique for LLM performance, and it is worth being precise about what it actually speeds up. It reduces the time to generate each subsequent token by verifying multiple candidates per forward pass, which improves throughput and inter-token latency. It does not touch prefill, queueing, or the network round trip, so it has little to no effect on TTFT specifically. The same is true of most model-side optimizations aimed at decoding speed: quantization, KV-cache reuse across a single generation, and batching strategies primarily affect TPOT and total throughput rather than TTFT itself, which is the number this post focuses on.

This distinction is worth making explicit because the two problems get conflated constantly in latency discussions. If TTFT is the bottleneck, a faster decoding engine will not fix it; the fix has to target queueing, routing, prefill, or caching instead, the gateway-level levers covered above.

Common TTFT Mistakes

• Optimizing tokens-per-second and assuming TTFT improved too. Throughput and TTFT are different metrics with different causes; a faster decoder does not shrink queue time.
• Measuring average TTFT instead of the tail. A good p50 with a bad p99 still produces a steady stream of users who experience the slow path. Bifrost's observability integrations surface percentile latency per model and provider, not just the average.
• Sending long, unfiltered context on every call. Prefill scales with input length, so retrieving fewer, better-ranked chunks in a RAG pipeline reduces TTFT directly, independent of any infrastructure change.
• Running semantic caching as the only cache mode on latency-sensitive traffic. A semantic miss costs an embedding call on top of the full LLM call, making it slower than no cache at all for low-hit-rate workloads. Direct-only mode skips that risk entirely.
• Fixing routing to a single provider. A single provider's outage or degraded performance becomes every user's problem, with no fallback path to a faster option.

TTFT FAQ

What does time to first token mean?

Time to first token is the elapsed time between sending a request to an LLM and receiving the first generated token in response. It measures perceived responsiveness for streaming applications, distinct from how long the full response takes to complete.

How to get time to first token?

TTFT is measured by recording the timestamp when a request is sent and the timestamp when the first streamed token (the first SSE or WebSocket chunk containing generated content) is received, then taking the difference. Most gateway and observability tools, including Bifrost's request logging, report this automatically per request.

What affects time to first token?

TTFT is driven primarily by request queue time before processing starts, network round-trip time, connection setup, and prompt prefill time, which scales with input length. Under concurrent load, queueing is typically the largest and most variable contributor.

What is TTFT and TPOT?

TTFT (time to first token) measures the delay before the first token arrives. TPOT (time per output token), also called inter-token latency, measures the delay between each subsequent token. A system can score well on one and poorly on the other, so both need to be tracked separately.

Does caching always improve TTFT?

Only when it produces a hit. A direct hash-match cache hit essentially eliminates TTFT for that request. A semantic cache miss adds an embedding API call on top of the normal LLM call, making it slower than skipping the cache, so cache mode choice should match the traffic's repeat pattern.

Does speculative decoding reduce TTFT?

Not directly. Speculative decoding speeds up token generation after the first token by verifying multiple candidate tokens per forward pass, which improves throughput and inter-token latency. It does not touch queueing, prefill, or network time, the phases that determine TTFT.

Why does TTFT get worse under load even with the same model?

Because queue wait time before a request starts processing grows as concurrent requests compete for the same serving capacity. NVIDIA's own NIM benchmarks show this pattern clearly: TTFT stays low at light concurrency and rises sharply, often by one or two orders of magnitude, as concurrent load increases on identical hardware and models, which is exactly the case adaptive load balancing is built to soften by shifting traffic toward whichever provider or key has spare capacity right now.

Cutting TTFT is rarely one fix. It is closing the gap between what a gateway can control directly, queueing, routing, connection overhead, caching, and what has to be addressed further down the stack, prompt length, model choice, and provider-side capacity. Bifrost implements the gateway-side levers, adaptive routing, connection pooling, semantic caching, and native streaming, as configuration rather than custom middleware.

For the cost side of this same infrastructure layer, see how to reduce LLM cost and latency in production and a comprehensive guide to reducing LLM cost and latency, which cover the broader picture beyond TTFT specifically.

Teams evaluating model-routing strategies as a lever on both cost and speed can also see top LLM routing techniques and 5 ways to optimize cost and latency in LLM-powered applications. To see adaptive routing, connection pooling, and caching running against a real production workload, book a demo with the Bifrost team.