Skip to main content

Announcing KServe v0.20 - Confidential Model Serving, Traffic Splitting, Canary Rollouts, and KV Cache Offloading

· 10 min read
Filippe Spolti
Filippe Spolti
Approver, KServe; Senior Software Engineer, Red Hat

Published on August 6, 2026

We are excited to announce the release of KServe v0.20. This release delivers major new capabilities across the platform with key highlights including:

  • Confidential model serving with TEE-based encrypted model decryption
  • Traffic splitting API for progressive LLMInferenceService deployments
  • Canary rollout support for InferenceService in RawDeployment mode
  • KV cache offloading with CPU memory tiering for vLLM workloads
  • Distributed tracing API for LLMInferenceService components
  • Managed DRA (Dynamic Resource Allocation) for GPU provisioning
  • Anthropic Messages API routing support
  • Native OCI ImageVolume mounting (oci+native://) for model storage
  • vLLM as a standalone runtime for InferenceService
  • AutoGluon server with time series inference support

🔐 Confidential Model Serving​

KServe v0.20 introduces confidential model serving (#5382), enabling encrypted ML model artifacts to be decrypted only inside Trusted Execution Environments (TEEs) using hardware attestation. This protects sensitive model intellectual property throughout the serving lifecycle.

When enabled, the storage initializer downloads model artifacts from any supported backend (S3, Hugging Face, GCS, etc.), detects JWE-encrypted files, contacts the Confidential Data Hub (CDH) running locally inside the TEE guest, which performs hardware attestation with a Key Broker Service (KBS) to retrieve the decryption key. The model is decrypted in-place inside the TEE, never exposed outside.

The feature is RATS-protocol agnostic and works with any KBS backend (Trustee, Intel Trust Authority, etc.). It is supported on both InferenceService and LLMInferenceService:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: my-confidential-model
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      storageUri: "s3://bucket/encrypted-model"
      confidential:
        enabled: true
        resourceId: "kbs:///default/keys/model-key"

🚀 LLMInferenceService Enhancements​

🔀 Traffic Splitting for Controlled Deployments​

LLMInferenceService now supports progressive traffic splitting across service versions (#5727, #5798, #5820). New group and weight fields on GatewayRoutesSpec allow declaring traffic splitting group membership, enabling controlled rollout of model updates.

The feature includes GroupReady and GroupDegraded conditions that are independent of the service's main Ready condition — a broken group doesn't block the resource from serving inference. Group traffic splitting is gated on member readiness, and collisions across LLMISVCs with the same model-routing names produce warnings (#5800). Stable, version-independent URLs are also now available (#5822).

💾 KV Cache Offloading​

A new kvCacheOffloading spec enables CPU KV cache tiering for vLLM workloads (#5599, #5740), replacing the previous preset-based approach with a structured API field. Users set kvCacheOffloading.cpu directly on their workload spec, and the controller automatically renders the --kv-transfer-config JSON flag for vLLM:

spec:
  kvCacheOffloading:
    cpu: 10Gi

For prefill/decode disaggregated deployments, set it on the prefill spec. Secondary filesystem tiers are also supported for more advanced offloading configurations.

🔍 Distributed Tracing API​

LLMInferenceService now has a first-class distributed tracing API (#5481) that gives users a clean, declarative way to enable OpenTelemetry tracing — no more manually wiring up env vars through pod template overrides. Per-component tracing is supported for both the inference server (spec.tracing) and the scheduler (spec.router.scheduler.tracing):

spec:
  tracing: {}
  router:
    scheduler:
      tracing: {}
  model:
    uri: hf://Qwen/Qwen2.5-7B-Instruct

Setting tracing: {} enables tracing with production defaults. The controller automatically injects OTEL_RESOURCE_ATTRIBUTES with pod name, node name, and namespace via the downward API. In disaggregated (P/D) setups, -decode and -prefill suffixes are automatically appended to service names for trace distinction.

🎯 Managed DRA Support​

Managed Dynamic Resource Allocation (DRA) is now available for LLMInferenceService (#5352). Users can request GPU resources dynamically by adding serving.kserve.io/dra-device-class and serving.kserve.io/dra-gpu-count annotations. The controller automatically generates the required ResourceClaimTemplates and injects ResourceClaims into PodSpecs for both single-node and multi-node (LeaderWorkerSet) deployments.

💬 Anthropic Messages API Routing​

LLMInferenceService now routes /v1/messages traffic through InferencePool (#5648), enabling backends that support the Anthropic Messages API to receive requests via the llm-d-router scheduler. This follows the same pattern as the existing /v1/responses (OpenAI Responses API) and /v1/chat/completions routing, with model-based routing rules automatically included.

🔌 LoRA Affinity Scorer Auto-Injection​

When LoRA adapters are configured, the controller now automatically injects the lora-affinity-scorer plugin into the default scheduling profile (#5655). This scores endpoints by whether the requested adapter is already active on each pod, minimizing adapter load latency without requiring manual scheduler configuration.

⏱️ Latency Predictor Sidecar Injection​

A new latency predictor sidecar is auto-injected when the predicted-latency-producer plugin is detected in the EPP config (#5678). This follows the well-known config pattern — the controller auto-injects training-server and prediction-server sidecar containers, with all defaults provided and user overrides supported via baseRefs.

🔄 LLM-D v0.8 and CRD Migration​

The llm-d dependency has been upgraded to v0.8.0 (#5596, #5719). Routing resources have been migrated to llm-d.ai CRDs (#5585), simplifying CRD management by using the canonical source maintained by the llm-d-router project. GIE has been upgraded to v1.5.0 (#5571), and the default P/D and standard configs have been aligned with the llm-d optimized baseline (#5662, #5670).

📋 Additional LLMInferenceService Improvements​

  • Model-based routing gates and models in status (#5579)
  • Cache InferenceService config and watch for changes (#5573)
  • LLMInferenceServiceConfig finalizer to prevent accidental deletion (#5400)
  • Merge user storage-initializer customizations (#5614)
  • Discover URLs for all matching listeners (#5664)
  • Tensor parallelism flags for single-node (#5782)

🔧 InferenceService and Platform Improvements​

🐦 Canary Rollout Support for RawDeployment Mode​

InferenceService now supports canary rollouts in Standard (RawDeployment) mode (#5672). A new spec.canary list allows defining named canary deployments with traffic percentage allocation. Each canary gets its own Deployment and Service, with replicas derived from the traffic percentage.

Promotion is zero-restart — retargeting the stable predictor name to the canary Deployment avoids pod recreation. Rollback is as simple as removing the canary entry, which triggers orphan cleanup:

spec:
  predictor:
    model:
      modelFormat: {name: sklearn}
      storageUri: "gs://kfserving-examples/models/sklearn/1.0/model"
    minReplicas: 4
  canary:
  - trafficPercent: 25
    predictor:
      name: v2
      model:
        modelFormat: {name: sklearn}
        storageUri: "gs://kfserving-examples/models/sklearn/1.0/model-2"

🖥️ vLLM as a Standalone Runtime​

vLLM is now a supported runtime for InferenceService (#4769), separate from the HuggingFace transformers runtime. Users can set modelFormat.name: vllm to use the dedicated vLLM runtime with single-node GPU, CPU, and multi-node inference support:

spec:
  predictor:
    model:
      modelFormat:
        name: vllm
      storageUri: "hf://Qwen/Qwen2.5-0.5B-Instruct"

📦 Native OCI ImageVolume Mounting​

A new oci+native:// URI scheme materializes a Kubernetes ImageVolume directly on the pod spec (#5558) — no sidecar, no storage initializer download. The model container image is pulled and mounted by the container runtime at /mnt/models as a read-only volume. This is Step 2 of the OCI storage harmonization roadmap (issue #4083), requiring Kubernetes >= 1.33.

A cluster-wide default mode (ociModelMode: "native" in the config) is also supported to route all oci:// URIs to native mounting.

🔑 Authorization Header Forwarding​

The transformer SDK now automatically forwards authorization headers (Authorization, X-Request-ID, X-B3-TraceID) from caller to predictor (#5567), enabling transformer-predictor chains to work with auth-enabled predictors without manual header plumbing.

🤖 AutoGluon Server​

A new AutoGluon model server (#5269) adds support for serving autogluon.tabular.TabularPredictor and autogluon.timeseries.TimeSeriesPredictor models with both KServe v1 and v2 protocol support.

Additional Enhancements​

  • Multiple OCI sources in storageUris (#5470)
  • storageContainerName field for explicit ClusterStorageContainer selection (#5314)
  • Platform hooks for service customization in InferenceService (#5617)
  • Multi-node InferenceService Ready condition fix (#5703)
  • CA bundle honored for HTTPS and HuggingFace downloads (#5755)
  • Modelcar shell quoting stabilized to avoid unnecessary pod restarts (#5780)
  • LocalModelNode race condition fixed between reconcilers (#5777)

📦 Dependency Upgrades​

  • Envoy AI Gateway upgraded to v1.0.0 and Envoy Gateway to v1.8.1 (#5723)
  • Gateway API bumped to v1.5.1 (#5478)
  • GIE upgraded to v1.5.0 (#5571)
  • llm-d components upgraded to v0.8.0 (#5596)

🔒 Security Fixes​

  • CVE-2026-48710 — Starlette bumped to >=1.0.1 (#5632)
  • Pillow CVE fix in vLLM setup (#5553)
  • Value/ValueFrom conflict prevention when merging ClusterStorageContainer (#5541)

🏗️ Infrastructure and CI​

  • Auto-assign reviewers on new PRs (#5491)
  • CNCF runners for heavy LLM/vLLM e2e jobs (#5724)
  • Targeted reruns replacing rerun-actions (#5621)
  • SeaweedFS model caching for e2e tests (#5566)
  • AGENTS.md added with project conventions (#5557)
  • Release workspace isolation and progress tracker (#5674)
  • Configurable make test for targeted test execution (#5216)

🔍 Release Notes​

For the complete list of all merged pull requests, bug fixes, and known issues, visit the GitHub release pages:

🙏 Acknowledgments​

We extend our gratitude to all 33 contributors who made this release possible, including 11 first-time contributors. Your efforts continue to drive the advancement of KServe as a leading platform for serving machine learning models.

  • New Contributors: @skbhudolia12, @KaveeshKhattar, @richiefrost, @madhugoutham, @RunlinXiao, @danehans, @yashaswipiplani, @anshulmav, @ZabinskiMichal, @ravindudanthanarayana, @rsevilla87 — welcome and thank you!
  • Core Contributors: The KServe maintainers and regular contributors
  • Community: Everyone who reported issues, provided feedback, and tested features

🤝 Join the Community​

We invite you to explore the new features in KServe v0.20 and contribute to the ongoing development of the project:

Happy serving!

The KServe team is committed to making machine learning model serving simple, scalable, and standardized. Thank you for being part of our community!