Buoyant published Kinference on November 4, 2026, an open-source control plane for serving open-weight models on Kubernetes. It sits in front of vLLM, takes OpenAI or Anthropic shaped traffic on one endpoint, and decides which GPU gets each request. On traffic that shares a long prompt prefix, Buoyant reports median time to first token falling from 6.88 s to 0.31 s.
What was actually released
Five things ship together. Kinference enforces token rate limits per client, tracks usage and cost per tenant, balances load across workers, applies declared routing policies, and exports inference metrics. It is written by William Morgan, a decade into Linkerd, and licensed Apache 2.0.
Adoption is shallow: clients point at the Kinference HTTP endpoint and keep sending OpenAI or Anthropic shaped requests, so no SDK changes. It does not use Linkerd and does not require it.
The layer Buoyant says is missing elsewhere is the policy one. A policy routes or rejects a request based on tenant, model, GPU pool state, or a semantic read of the prompt: serve any Qwen 3.8 model rather than one exact checkpoint, take batch work only when GPUs are idle, burst to a third party when the pool saturates, keep prompts with personal data inside the cluster. Declared objects can be reviewed by a human or an agent, committed to git, linted, and rolled back on their own.
What ships on day one
- An Apache 2.0 control plane on GitHub, running in front of vLLM, with docs written for humans.
- Declarative inference policies, applied independently of one another.
kinfctl, a command that validates policy YAML offline, without touching a cluster.- Status messages and
kubectlcolumns on every custom resource, written for agents reading them.
Buoyant also tested the agent path: coding agents with fresh context windows were pointed at the docs, made to install kinfctl, and asked to write policies, and where one went wrong the docs changed. The post expects most configuration to be written by models.
What the claim rests on
The benchmark is Buoyant’s own and described precisely enough to be repeated. Two identical pools of vLLM 0.31.0 served Qwen-3.8-27B-FP8, three pods per pool, one L40S each, on AWS. One arm ran through Kinference, the other through a Service record where kube-proxy picks a random pod per connection. GuideLLM 0.8.1 generated the load, three workloads, three trials, paired bootstrap intervals.
| Workload | Median TTFT, plain Kubernetes | Median TTFT, Kinference | Change |
|---|---|---|---|
| Control, no shared prefix | 6.92 s | 6.83 s | no significant difference |
| Shared prefix, 24 documents | 6.88 s | 0.31 s | 95% lower |
| Multiturn, 8 turns | 1.70 s | 0.59 s | 65% lower |
The control row is what makes the rest credible. With nothing reusable in the prompt, Kinference adds nothing measurable to the median, so it does not buy its gains by taxing every request. Its 95th percentile still improves there, 21.46 s to 12.33 s, which the post credits to queue depth in the routing signal.
Capacity is what a GPU bill buys.
| Measure | Shared prefix | Multiturn |
|---|---|---|
| Peak goodput against plain Kubernetes | 7.4x | 3.5x |
| Latency target used | 90% of requests at 8 s or less | 90% of requests at 4 s or less |
| Highest concurrency meeting it, Kinference | 32 | 8 |
| Highest concurrency meeting it, plain Kubernetes | none measured | none measured |
Buoyant says the targets were picked empirically, from the uncached first token latency of an idle deployment plus padding. That is honest, and the targets are soft: the post calls them too slow for interactive chat and plausible for agent work. The plain Kubernetes arm never met either target at any concurrency.
Given a fixed set of GPUs, the faster you can run inference on them, the more inference you can pack onto the same GPUs.
What the benchmark does not show
- No independent run. Every figure comes from the vendor, on one model, one GPU type, one cluster.
- No comparison against another inference gateway. The only baseline is random balancing through a Service.
- No measurement of policies, rate limits, tenant cost tracking or metrics. Only the load balancer is tested, though the policy engine is the part Buoyant calls central.
- No dollar figure: the post converts nothing into cost per million tokens.
On this page
What it changes for teams that run their own inference
If you rent tokens from a hosted API, none of this reaches your bill today. Kinference runs next to the GPUs, and if the GPUs are not yours, there is nothing to route. The 95% is a latency effect on prefix-sharing traffic, not a price cut.
If you already serve open-weight models on vLLM behind a Kubernetes Service, this targets the layer above the engine and replaces one object in your stack. The gain lands on traffic agent products generate: a coding agent resends the same system prompt, repository context and growing conversation every turn, so a router that sends turn five to the pod holding turns one to four skips the prefill. That is why shared-prefix beats multiturn, and why control shows nothing.
Three numbers decide whether it deserves a day. The share of your traffic with a reusable prefix, which caps the gain. Your 95th percentile time to first token, since the median is already fine on most deployments. And the concurrency where your setup stops meeting its target, because that number times the capacity ratio is the GPU spend you avoid.
The routing win is a prefix-reuse win, not a token-price win, and the two are easy to confuse when the headline is a percentage. The capacity table is the honest one: 7.4 times more requests on the same GPUs, on one workload, against a random balancer.
The Neural Minute take
We have spent the year reading rate cards, and the rate card is now the least interesting number. The cuts that matter have moved into the serving layer, where the same model gets cheaper because the same GPU does more work. Our reading of the AKV release in October was the same argument from the research side: a cache policy that cut cost per finished task by 27.7% without touching a price list. Kinference is its operational version for teams that own hardware.
The commercial shape explains the license. Buoyant sells a managed inference product on the same site, described as unlimited tokens and full inference control, and the open-source control plane is the on-ramp. That is how infrastructure companies fund control planes, and it means the benchmark exists to sell a pool of GPUs, so its missing rows are the ones a buyer would want.
Our verdict: nothing here changes what you pay OpenAI or Anthropic this month, and nobody should trade a hosted API for a cluster over a 95% figure. For a team running coding agents on its own hardware, this is the most specific thing published this week on the question that decides the bill, and the load generator, workloads and concurrency levels are named, so the experiment is repeatable on your own traffic.
We have not run Kinference, and we will not quote its table as a result you should expect. The first thing we would check is what the benchmark skips: whether the policy engine holds a per-tenant limit under pressure without a latency cost.
Sources
The announcement: Introducing Kinference, published by Buoyant on November 4, 2026 and read on November 7, 2026. The release date, feature list and policy examples come from that page.
The figures: Distributed Inference Benchmarks: Kinference vs. Plain Kubernetes, same publisher, same date. Every latency, capacity and concurrency number above is read there.
No other source was used, and Buoyant’s own runs are the only measurements here.