Your chatbot worked fine until your agents showed up. llm-d 1.0 is the open-source stack built for what comes next.
Google, Red Hat, IBM, and others released llm-d 1.0 at the PyTorch Conference on October 20, 2026. It is a Kubernetes-native stack for distributed LLM inference, built for multi-step agents and reinforcement learning workloads.
Those workloads break simple setups. Agents fan out across tools, share long prefixes, and arrive in bursts. One bursty agent loop can stall every interactive user behind a plain round-robin load balancer.
What llm-d 1.0 actually does
Instead of treating each model server as an isolated endpoint, llm-d turns a cluster of vLLM or SGLang servers into one coordinated pool. One router, one cache index, and one flow-control policy sit in front of the fleet.
The project is a Cloud Native Computing Foundation (CNCF) Sandbox project, with more than 500 contributors across over 25 organizations, according to Google. That breadth matters: this is shared infrastructure, not a single vendor’s side project.
| 1.0 capability | What it means in practice |
|---|---|
| Multi-tenant flow control | Priority and fairness-aware routing, so bursty jobs do not starve interactive users |
| Cache-aware routing | Each agent step goes to the replica that already holds its context |
| Tiered KV cache | Cache spills to host memory, SSD, or shared storage instead of being dropped |
| Disaggregated prefill/decode | Compute-heavy prefill and memory-bound decode scale as separate pools |
| Agent-aware sessions | Session-aware routing, turn-priority scheduling, and native Responses API state |
| Router high availability | Zero-downtime updates without dropping active sessions |
Source: Google Cloud Blog, “llm-d 1.0, an open source stack for distributed inference.”
How it works, without the jargon
Agents call a single API. They never pick a server.
The router sends each step to the replica holding its cached prefix, weighing load and predicted latency.
Bursty loops queue behind priority lanes, so interactive requests stay fast.
Prefill, decode, and multimodal encode run as separate pools that scale independently.
KV cache is the stored attention state from earlier tokens. Reusing it avoids recomputing the same prompt on every agent step, which is where most of the wasted compute lives.
Prefill is the compute-heavy first pass over a prompt. Decode is the memory-bound generation of new tokens. Running both on the same replicas makes each wait on the other, so llm-d splits them.
The demo numbers, labeled honestly
These are Google-reported demo claims, not independent benchmarks. On an agentic coding workload on Ironwood TPU, Google reports that prefix-aware routing plus host memory offload cut median time-to-first-token (TTFT) by more than 85% and raised throughput 2.4x versus a plain Kubernetes Service.
Google also reports a keynote demo of 1,000 sandboxed agents on Agent Substrate on GKE hitting a single llm-d endpoint across vLLM, SGLang, TPUs, and GPUs.
Other Google-reported figures include a prefix cache hit rate rising from 35% to 70% on the Gemini platform, and a 17,000-node benchmark scaling throughput 2.9x with 99.9% success. Test all of it on your own cluster before quoting it to a client.
Source: Google Cloud Blog.
Watch the official demo from the llm-d Project channel:
How to profit from llm-d 1.0
Here is the honest money angle. Most small businesses will never run llm-d themselves, but they will pay someone who can run agent workloads without a runaway GPU bill.
- Offer managed agent hosting. Run a client’s agents behind llm-d on rented GPUs or TPUs, and charge a monthly platform fee plus usage.
- Sell the migration, not the stack. Audit a startup’s vLLM setup, add prefix-aware routing first, then flow control and disaggregation. Each step is a billable milestone.
- Price on the GPU bill. If cache-aware routing and time-slicing cut accelerator hours, take a share of the documented saving as your fee.
- Start narrow. One model, one workload, one client. Distributed inference rewards operators who measure before they scale.
- Use Prism to prove it. llm-d’s Prism dashboard publishes benchmark configurations, so you can show clients repeatable numbers.
What 1.0 does not solve
- It is infrastructure, not a product. You still need Kubernetes skills, or a managed platform like GKE Inference Gateway, to run it well.
- Demo numbers are not your numbers. Cache hit rates depend on how much your agents repeat prompts. Measure your own traffic first.
- Small workloads may not need it. A single vLLM server is cheaper until agent traffic gets bursty and multi-tenant.
The verdict: if you serve agents at scale, llm-d 1.0 is the most credible open path to lower latency and lower accelerator bills. If you serve one chatbot, it is overkill.
Frequently asked questions
What is llm-d 1.0?
llm-d 1.0 is an open-source, Kubernetes-native stack for distributed LLM inference. It coordinates vLLM or SGLang servers into one serving pool with smart routing, flow control, and disaggregated prefill and decode.
Who built llm-d?
Google co-founded it with Red Hat, IBM, and others. Google reports more than 500 contributors across over 25 organizations, and it is a CNCF Sandbox project.
What license does llm-d use?
llm-d is released under Apache 2.0, the standard permissive open-source license for the project.
Do I need Google Cloud to use llm-d?
No. It is an open stack you can run on your own Kubernetes cluster, though Google offers a managed path through GKE Inference Gateway.
What are the headline performance claims?
Google reports an 85%+ median TTFT cut and 2.4x throughput on an Ironwood TPU coding workload, versus a plain Kubernetes Service. These are vendor demo claims, so verify them on your own workload.
Who should adopt llm-d 1.0 first?
Teams serving multi-step agents, reinforcement learning rollouts, or multi-tenant model APIs, where round-robin routing already causes latency spikes or wasted accelerator time.
Where is the official source?
The Google Cloud Blog announcement at cloud.google.com and the project site at llm-d.ai.