Deploy InferenceService with model from Hugging Face Hub
You can specify the storageUri field on InferenceService YAML with the following format to deploy the models from Hugging Face Hub.
hf://${REPO}/${MODEL}:${HASH}(optional)
e.g. hf://facebook/opt-125m
Public Hugging Face Models
If no credential is provided, anonymous client will be used to download the model from HF repo.
Private Hugging Face Models
KServe supports authenticating with HF_TOKEN for downloading the model and create a Kubernetes secret to store the HF token.
apiVersion: v1
kind: Secret
metadata:
name: storage-config
type: Opaque
data:
HF_TOKEN: aGZfVk5Vd1JVAUdCa0l4WmZMTHVrc2VHeW9VVm9udU5pBHUVT==
Deploy InferenceService with Models from HF Hub
Option 1: Use Service Account with Secret Ref
Create a Kubernetes ServiceAccount with the HF token secret name reference and specify the ServiceAccountName in the InferenceService Spec.
apiVersion: v1
kind: ServiceAccount
metadata:
name: hfserviceacc
secrets:
- name: storage-config
---
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: huggingface-llama3
spec:
predictor:
serviceAccountName: hfserviceacc # Option 1 for authenticating with HF_TOKEN
model:
modelFormat:
name: huggingface
args:
- --model_name=llama3
- --model_dir=/mnt/models
storageUri: hf://meta-llama/meta-llama-3-8b-instruct
resources:
limits:
cpu: "6"
memory: 24Gi
nvidia.com/gpu: "1"
requests:
cpu: "6"
memory: 24Gi
nvidia.com/gpu: "1"
Configure the storage initializer with a ClusterStorageContainer
If you need to set HF_TOKEN directly on the storage-initializer container, define a
ClusterStorageContainer with:
workloadType: initContainer;- a supported URI format with the
hf://prefix; and - a Secret-backed
HF_TOKENenvironment variable.
The ClusterStorageContainer is cluster-scoped, but the referenced Secret must be in
the same namespace as the InferenceService. See Providing Credentials with a
ClusterStorageContainer
for a complete example.
Check the InferenceService status.
kubectl get inferenceservices huggingface-llama3
NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE
huggingface-llama3 http://huggingface-llama3.default.example.com True 100 huggingface-llama3-predictor-default-47q2g 7d23h