KServe Python SDK¶
Python SDK for KServe Server and Client.
Installation¶
KServe Python SDK can be installed by pip
or Setuptools
.
pip install¶
pip install kserve
Setuptools¶
Install via Setuptools.
python setup.py install --user
sudo python setup.py install
to install the package for all users)
KServe Python Server¶
KServe's python server libraries implement a standardized library that is extended by model serving frameworks such as Scikit Learn, XGBoost and PyTorch. It encapsulates data plane API definitions and storage retrieval for models.
It provides many functionalities, including among others:
- Registering a model and starting the server
- Prediction Handler
- Pre/Post Processing Handler
- Liveness Handler
- Readiness Handlers
It supports the following storage providers:
- Google Cloud Storage with a prefix: "gs://"
- By default, it uses
GOOGLE_APPLICATION_CREDENTIALS
environment variable for user authentication. - If
GOOGLE_APPLICATION_CREDENTIALS
is not provided, anonymous client will be used to download the artifacts.
- By default, it uses
- S3 Compatible Object Storage with a prefix "s3://"
- By default, it uses
S3_ENDPOINT
,AWS_ACCESS_KEY_ID
, andAWS_SECRET_ACCESS_KEY
environment variables for user authentication.
- By default, it uses
- Azure Blob Storage with the format: "https://{$STORAGE_ACCOUNT_NAME}.blob.core.windows.net/{$CONTAINER}/{$PATH}"
- By default, it uses anonymous client to download the artifacts.
- For e.g. https://kfserving.blob.core.windows.net/triton/simple_string/
- Local filesystem either without any prefix or with a prefix "file://". For example:
- Absolute path:
/absolute/path
orfile:///absolute/path
- Relative path:
relative/path
orfile://relative/path
- For local filesystem, we recommended to use relative path without any prefix.
- Absolute path:
- Persistent Volume Claim (PVC) with the format "pvc://{$pvcname}/[path]".
- The
pvcname
is the name of the PVC that contains the model. - The
[path]
is the relative path to the model on the PVC. - For e.g.
pvc://mypvcname/model/path/on/pvc
- The
- Generic URI, over either
HTTP
, prefixed withhttp://
orHTTPS
, prefixed withhttps://
. For example:https://<some_url>.com/model.joblib
http://<some_url>.com/model.joblib
KServe Client¶
Getting Started¶
KServe's python client interacts with KServe control plane APIs for executing operations on a remote KServe cluster, such as creating, patching and deleting of a InferenceService instance. See the Sample for Python SDK Client to get started.
Documentation for Client API¶
Class | Method | Description |
---|---|---|
KServeClient | set_credentials | Set Credentials |
KServeClient | create | Create InferenceService |
KServeClient | get | Get or watch the specified InferenceService or all InferenceServices in the namespace |
KServeClient | patch | Patch the specified InferenceService |
KServeClient | replace | Replace the specified InferenceService |
KServeClient | delete | Delete the specified InferenceService |
KServeClient | wait_isvc_ready | Wait for the InferenceService to be ready |
KServeClient | is_isvc_ready | Check if the InferenceService is ready |
Documentation For Models¶
- KnativeAddressable
- KnativeCondition
- KnativeURL
- KnativeVolatileTime
- NetUrlUserinfo
- V1beta1AIXExplainerSpec
- V1beta1AlibiExplainerSpec
- V1beta1Batcher
- V1beta1ComponentExtensionSpec
- V1beta1ComponentStatusSpec
- V1beta1CustomExplainer
- V1beta1CustomPredictor
- V1beta1CustomTransformer
- V1beta1ExplainerSpec
- V1beta1InferenceService
- V1beta1InferenceServiceList
- V1beta1InferenceServiceSpec
- V1beta1InferenceServiceStatus
- V1alpha1InferenceGraph
- V1alpha1InferenceGraphList
- V1alpha1InferenceGraphSpec
- V1alpha1InferenceGraphStatus
- V1beta1LightGBMSpec
- V1beta1LoggerSpec
- V1beta1ModelSpec
- V1beta1ModelStatus
- V1beta1ONNXRuntimeSpec
- V1beta1PaddleServerSpec
- V1beta1PMMLSpec
- V1beta1PodSpec
- V1beta1PredictorExtensionSpec
- V1beta1PredictorSpec
- V1beta1SKLearnSpec
- V1beta1TFServingSpec
- V1beta1TorchServeSpec
- V1beta1TransformerSpec
- V1beta1TritonSpec
- V1beta1XGBoostSpec