# How Does Disaggregated LLM Inference on Kubernetes Work?

aitutorialmaker.com · October 24, 2026

> Understanding Disaggregated LLM Inference on Kubernetes Disaggregated LLM inference on Kubernetes separates the compute-intensive prefill phase from...

## Understanding Disaggregated LLM Inference on Kubernetes

Disaggregated LLM inference on Kubernetes separates the compute-intensive prefill phase from the memory-intensive decode phase into distinct pods or services. A request first reaches a router or gateway, which sends prompt processing to prefill workers optimized for high throughput. Once the initial key-value cache is created, the request and cache state move to decode workers that generate tokens efficiently. Kubernetes schedules these components across nodes, manages health, and scales each pool independently based on demand. This avoids forcing one hardware profile to handle both bursty prompt ingestion and steady token generation.

**Also worth reading:** [What Is the Best Kubernetes LLM Inference Deployment Strategy?](https://aitutorialmaker.com/knowledge/what_is_the_best_kubernetes_llm_inference_deployment_strategy.php) · [How Can You Deploy LLMs on Kubernetes in Production?](https://aitutorialmaker.com/knowledge/how_can_you_deploy_llms_on_kubernetes_in_production.php) · [What Are the Best Kubernetes AI Tutorials for Learning AI Infrastructure in 2026?](https://aitutorialmaker.com/knowledge/what_are_the_best_kubernetes_ai_tutorials_for_learning_ai_infrastructure_in_2026.php)

Operators define deployments for prefill and decode, configure resource requests, and expose endpoints through services or an ingress layer. Autoscalers adjust replica counts as queue depth and latency change, while affinity rules place workloads on suitable GPUs or CPUs. Shared cache transfer, observability, and rollout controls keep the pipeline reliable. The result is better utilization, lower tail latency, and flexible scaling for large language model serving without treating inference as a single monolithic process.

## Why Split Prefill and Decode Stages

Disaggregated LLM inference on Kubernetes separates prefill and decode into distinct workloads so each can scale independently. Prefill processes the full prompt in parallel and builds the initial key-value cache, while decode generates tokens sequentially and is more memory-bandwidth bound. Separate pods or services handle each stage, and a router directs requests while passing cache state between workers. This lets operators tune GPU types, batch sizes, and replica counts for each phase instead of using one compromise configuration for both compute-heavy and latency-sensitive work.

The workflow starts when an ingress or scheduler sends a prompt to a prefill worker. That worker computes attention states and cache tensors, then transfers them to a decode worker over fast networking, where token generation continues until completion. Kubernetes manages placement, health checks, autoscaling, and service discovery, while queueing and observability layers balance load and prevent bottlenecks. Isolating stages helps reduce idle GPU time, improve throughput, and keep streaming responses responsive without overprovisioning every replica for peak prompt processing.

## Deploying LLM Workloads on Kubernetes

Disaggregated LLM inference on Kubernetes separates prompt processing from token generation into distinct services or pods, often called prefill and decode. Prefill workers handle long input prompts and build key-value cache, while decode workers generate output tokens efficiently. Kubernetes schedules these components independently based on GPU type, memory, latency, and throughput needs. An inference router or gateway sends requests to prefill workers, transfers cache state or metadata, then routes to decode workers for streaming responses. This separation lets operators scale each stage independently, use specialized hardware, and avoid wasting expensive GPUs on mismatched workloads.

In practice, llm-d style controllers and custom resources define model deployments, worker pools, autoscaling policies, and routing rules. Kubernetes monitors queue depth, GPU utilization, and latency, then adjusts replicas for prefill or decode. Shared storage or high-speed networking moves cached context between stages, while observability tracks tokens per second and time to first token. The result is more predictable performance, better resource efficiency, and easier production operations for large language model serving.

## Comparing llm-d and Dynamo

Disaggregated LLM inference on Kubernetes separates prefill and decode stages into distinct services, letting each scale independently based on workload demand. A router or scheduler sends prompts to prefill workers that process input tokens and build key-value cache, then transfers cache state to decode workers that generate output tokens. Kubernetes handles placement, autoscaling, health checks, and service discovery across GPU nodes, while shared storage or high-bandwidth networking moves cache efficiently. This architecture improves throughput and reduces latency by matching hardware to each phase.

llm-d and Dynamo both aim to make this pattern production-ready, though they emphasize different layers. llm-d focuses on Kubernetes-native orchestration, scheduling, and operational workflows for disaggregated inference, while Dynamo emphasizes high-performance distributed serving, routing, and GPU-aware optimization. Together, they illustrate how platform teams can combine cluster management with inference engines, using metrics, autoscaling, and cache-aware routing to keep large models responsive under variable traffic.

## Monitoring and Scaling Inference Pods

Disaggregated LLM inference on Kubernetes separates compute-heavy prefill from memory-heavy decode into distinct pod pools, allowing independent scaling. A router sends prompts to prefill workers that build key-value caches, then transfers cached state to decode workers that generate tokens. Kubernetes schedules these as separate deployments, using labels, services, and autoscalers to place pods across nodes with suitable GPUs, memory, and network. This prevents one stage from starving the other and lets operators tune batch sizes, concurrency, and replicas per workload.

Monitoring tracks queue depth, time to first token, throughput, GPU utilization, cache hit rates, and pod health, while scaling responds to latency targets and demand. Autoscalers add prefill replicas when prompt volume rises and decode replicas when generation demand grows, often using warm pools to reduce cold starts. Because state moves between stages, operators also watch network throughput and cache transfer latency. Together, orchestration, observability, and autoscaling make serving resilient and cost-aware.

## Disaggregated vs Monolithic Inference

| Stage | Kubernetes Component | Purpose |
| --- | --- | --- |
| Prefill | Deployment | Computes prompt KV cache |
| Decode | Deployment | Generates output tokens |
| Routing | Service/Ingress | Directs requests to workers |
| Scaling | HPA | Adjusts replicas by load |

Disaggregated inference separates prefill and decode onto specialized pods, letting Kubernetes scale each phase independently for better GPU utilization and lower latency. Monolithic serving mixes both phases, causing resource contention and uneven throughput. With llm-d, vLLM, and cloud-native orchestration, teams route requests, manage KV cache movement, and optimize production LLM workloads across clusters while maintaining service-level objectives for demanding applications.

## Quick answers

### What is disaggregated LLM inference?

It separates prefill and decode stages across specialized compute resources.

### Why use Kubernetes for LLM inference?

Kubernetes orchestrates scalable, resilient pods for distributed model serving.

### What is llm-d?

llm-d is a Kubernetes-native framework for distributed LLM inference.

### How does disaggregation improve throughput?

It lets prefill and decode scale independently for better GPU utilization.

Canonical: https://aitutorialmaker.com/knowledge/how_does_disaggregated_llm_inference_on_kubernetes_work.php
Markdown: https://aitutorialmaker.com/knowledge/how_does_disaggregated_llm_inference_on_kubernetes_work.php/index.md