Skip to main content

Announcing KServe v0.19 - Static LoRA Adapters, Model Name Routing, and Graceful Shutdown

· 8 min read
Filippe Spolti
Filippe Spolti
Approver, KServe; Senior Software Engineer, Red Hat

Published on June 14, 2026

We are excited to announce the release of KServe v0.19. This release focuses heavily on LLMInferenceService maturity with key highlights including:

  • Static LoRA adapter support for efficient fine-tuned model serving
  • Model name-based routing for shared-gateway deployments
  • Graceful pod shutdown with vLLM drain to prevent in-flight request drops
  • HPA/KEDA scaling status surfaced directly in service conditions
  • LocalModelCache integration for LLMInferenceService
  • Dual-protocol (REST/gRPC) routing for InferenceService Standard mode
  • Seamless llm-d v0.6 to v0.7 migration with version-gated controller logic

🚀 LLMInferenceService Enhancements​

🔌 Static LoRA Adapter Support​

KServe v0.19 introduces static LoRA (Low-Rank Adaptation) adapter support for LLMInferenceService (#5317). LoRA adapters enable efficient fine-tuning of large language models by applying lightweight adapter weights on top of a base model — without modifying the base model itself.

The new spec.model.lora.adapters field allows defining multiple adapters with support for hf://, s3://, and pvc:// storage backends. The controller automatically configures vLLM with the appropriate LoRA flags (--enable-lora, --max-lora-rank, --lora-modules) and mounts adapters at /mnt/lora/<adapter-name>:

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: llama3-with-lora
spec:
  model:
    uri: hf://meta-llama/Llama-3.1-8B-Instruct
    name: meta-llama--Llama-3.1-8B-Instruct
    lora:
      adapters:
        - name: sql-adapter
          uri: hf://my-org/sql-lora-adapter
        - name: code-adapter
          uri: pvc://adapter-pvc/code-adapter

🌐 Model Name-Based Routing​

LLMInferenceService now supports model name-based routing (#5521), enabling shared-gateway deployments where a single gateway dispatches requests to multiple models using the X-Gateway-Model-Name header. The header name is configurable and can be injected by components such as llm-d BBR (aka IPP) or gateway-specific native features.

LoRA adapter requests are automatically routed through the same backend as the base model by dynamically expanding header matches for each adapter defined in .spec.model.lora.adapters.

⚡ Graceful Pod Shutdown with vLLM Drain​

KServe v0.19 implements graceful drain for LLM inference pods (#5485, #5496), preventing in-flight request drops during rolling updates or scale-down events. Two complementary mechanisms work together:

  1. preStop hook (/bin/sleep 15): gives the Endpoint Picker (EPP) time to detect the pod's readiness probe failure and stop routing new requests before SIGTERM is sent.
  2. vLLM --shutdown-timeout: drains in-flight requests gracefully after SIGTERM instead of aborting them immediately. The value is computed as terminationGracePeriodSeconds - 15, defaulting to 45 seconds.

The drain sequence on pod termination is:

preStop sleep (15s) → EPP stops routing → SIGTERM → vLLM drains (≤45s) → pod exits

terminationGracePeriodSeconds has been increased from the Kubernetes default (30s) to 60s across all vLLM templates. Users can override this via spec.template.terminationGracePeriodSeconds on their LLMInferenceService.

📊 HPA/KEDA Scaling Status in Service Conditions​

The health of autoscaling resources (HPA or KEDA ScaledObject) is now surfaced directly in the LLMInferenceService status via two new conditions: ScalingReady and PrefillScalingReady (#5540). Users no longer need to inspect child HPA/ScaledObject resources separately to diagnose scaling issues — kubectl get/describe now shows the full picture.

🔄 LLM-D v0.7 Upgrade with Seamless Migration​

The llm-d dependency has been upgraded to v0.7 (#5433, #5560), with version-gated controller migration logic ensuring seamless upgrades from v0.6. Key migrations include:

  • Plugin renames: prefill-header-handler → disagg-headers-handler, pd-profile-handler → disagg-profile-handler
  • Parameter restructure: deciderPluginName (string) → deciders map
  • Metric flag migration: Deprecated CLI flags are extracted and injected into the core-metrics-extractor plugin
  • Non-zero threshold migration: Converted to prefix-based-pd-decider with character-to-token ratio conversion

Existing deployments are transparently migrated during reconciliation; new deployments start with correct defaults.

WVA (Workload Variant Autoscaler) has also been upgraded to v0.7.0 (#5498).

📦 LocalModelCache Support for LLMInferenceService​

LLMInferenceService now integrates with LocalModelCache (#5318), enabling local model caching for LLM workloads with the same deduplication and multi-namespace support already available for InferenceService. When an LLMInferenceService's model URI matches an existing LocalModelCache or LocalModelNamespaceCache, the webhook automatically sets the appropriate labels and annotations, and the controller rewrites the model URI from the original (e.g., hf://model) to pvc://<cached-pvc>/models/<storageKey>/.

🔍 Enhanced Observability​

Several improvements make it easier to monitor and debug LLMInferenceService deployments:

  • Kubernetes Events on readiness transitions (#5437): Normal events on Ready, Warning events on NotReady with failing condition details — giving operators push-based availability signals without polling.
  • Observed routing topology in status (#5417): Reports the routing topology that has been reconciled.
  • Observed workload references in status (#5414): Reports the workload references (Deployments, LWS) managed by the service.
  • Observed applied configs (#5418): Surfaces the configuration that was actually applied to workloads.
  • ConfigNotFound condition (#5409): Surfaces a specific condition when the LLMInferenceServiceConfig is not found.
  • Gateway origin tracking (#5524): Tracks which gateway discovered each address for multi-gateway deployments.
  • Label and annotation propagation to workload Service (#5365): spec.labels and spec.annotations are now propagated to the workload Service, enabling ServiceMonitor discovery and network policies.

📋 Additional LLMInferenceService Improvements​

  • Heterogeneous GPUs load balancing sample (#5374)
  • --scaling flag for kserve-install.sh (#5497): Simplifies installation with WVA scaling enabled.
  • Targeted access-log flag with vLLM version fallback (#5507)
  • Optional GIE CRD creation in Helm charts (#5544)

🔧 InferenceService and Platform Improvements​

🔗 Dual-Protocol (REST/gRPC) Routing for Standard Mode​

InferenceService now supports dual-protocol routing in Standard mode when using the Gateway API (#5451). The HTTPRoute reconciler detects REST and gRPC ports from the backing Service and generates two rules: a gRPC match first (via application/grpc.* content-type header), followed by a REST/HTTP fallback. This enables both protocols behind the same hostname without requiring separate route resources.

Envoy AI Gateway and Envoy Gateway Upgrades​

Envoy AI Gateway has been upgraded to v0.6.0 and Envoy Gateway to v1.7.0 (#5520).

Additional Enhancements​

  • OCI support for storageUris (plural) (#5261)
  • Conversion webhooks enabled in minimal installs (#5416)
  • NodeSelector for LocalModel jobs to fix PVC access (#5502)
  • ClusterServingRuntime gated on CRD availability (#5392)
  • Eliminated ExternalName service no-op update churn (#5512)
  • Eliminated ConfigMap no-op update churn (#5513)
  • Off-by-one fix in splitter pickupRoute random range (#5489)
  • Timeout_keep_alive added for Istio/Envoy compatibility (#5448)
  • HuggingFace server base images upgraded to 2025.10 with CUDA 13.0.0 (#5206)
  • Prevent ClusterStorageContainer CRD deletion on Helm upgrade (#5539)
  • Helm imagePullPolicy defaults aligned with kustomize base (#5459)

🔒 Security Fixes​

  • CVE-2026-21226 — azure-core pinned to >=1.38.0 (#5508)
  • Pillow CVE fix in vLLM setup (#5553)

🏗️ Infrastructure and CI​

  • Stale bot to warn and close inactive issues and PRs (#5395)
  • Structured E2E test results (JUnit XML + JSON) (#5378)
  • Network-layer parameter support in e2e test infrastructure (#5506)
  • OWNERS approver review required before auto-merge (#5495)
  • GOTAGS pass-through to the main controller Dockerfile (#5402)
  • Release orchestrator improvements (#5423, #5460, #5483)
  • Copilot agent for bump-version and release issue template (#5577)

🔍 Release Notes​

For the complete list of all merged pull requests, bug fixes, and known issues, visit the GitHub release pages:

🙏 Acknowledgments​

We extend our gratitude to all 21 contributors who made this release possible, including 5 first-time contributors. Your efforts continue to drive the advancement of KServe as a leading platform for serving machine learning models.

  • New Contributors: @neelabhsinha, @dagrayvid, @brucearctor, @font, @walnutkim — welcome and thank you!
  • Core Contributors: The KServe maintainers and regular contributors
  • Community: Everyone who reported issues, provided feedback, and tested features

🤝 Join the Community​

We invite you to explore the new features in KServe v0.19 and contribute to the ongoing development of the project:

Happy serving!

The KServe team is committed to making machine learning model serving simple, scalable, and standardized. Thank you for being part of our community!