Six GPU Metrics That Decide Your LLM's Speed and Cost

If your model feels slow, or costs too much, or quietly makes things up, the answer is usually visible in your serving metrics. These are the G.P.U. numbers worth watching, and the prompt decisions that move them. Inference has two phases. Prefill reads your prompt and is limited by compute. Decode writes the answer one token at a time and is limited by memory bandwidth. Almost every metric that matters traces back to that split. Time to first token, or T.T.F.T., is how long the user waits before anything appears. It is dominated by prefill, so it grows with prompt length. Long system prompts are paid for on every single request. Time per output token measures how fast the answer streams once it starts. It is set by memory bandwidth, and by the number of requests sharing the G.P.U. Users feel time to first token as responsiveness, and time per output token as pace. Throughput is tokens per second across all users, and requests completed per hour. It is the number your finance team cares about. Latency is what one single user feels. These two pull in opposite directions. Concurrency is how many requests the G.P.U. serves at once. Continuous batching keeps the device busy by adding new requests as others finish. Raising concurrency raises throughput, and it also raises per user latency. Plot latency against throughput as you raise concurrency, and you get a curve with a knee. Below the knee you are wasting G.P.U. Above it, latency climbs sharply for very little extra throughput. Serve at the knee. Every token in flight holds a key value cache entry in G.P.U. memory. That cache, not compute, is usually what limits concurrency. Longer contexts and more users compete for the same fixed pool of memory. Watch K.V. cache utilisation. When it approaches its ceiling, the scheduler starts queuing or evicting requests, and latency spikes with no change in traffic at all. It is the most useful early warning signal you have. Prefix cache hit rate is the share of prompt tokens already cached from earlier requests. Put your stable system instructions first and your variable content last, and that rate climbs. Prefill cost falls with it. Queue time is how long a request waits before the G.P.U. touches it. If queue time dominates time to first token, you have a capacity or scheduling problem, not a model problem. A bigger model will not help. G.P.U. utilisation is misleading. It reports whether a kernel is running, not whether the work is efficient. Model flops utilisation tells you how much of the hardware you are truly using. Ninety percent busy can hide poor efficiency. Now the accuracy side. When a prompt exceeds the context window, something gets dropped. If the evidence is what gets truncated, the model answers from memory instead, and it invents. Silent truncation is a leading cause of hallucination. So measure grounding alongside speed. Track retrieval hit rate, the share of answers supported by the evidence you supplied, and the refusal rate when evidence is missing. A faster system that quietly invents more is not a better system. Watch six numbers. Time to first token, time per output token, throughput, K.V. cache utilisation, prefix cache hit rate, and queue time. Then hold accuracy constant while you tune them. Speed is only progress when grounding holds.