Control Plane API

ServerType must be one of the supported built-in types such as “triton” or “mlserver”, and the runtime’s container must have the same name

Port which the runtime server listens for model management requests

Fixed memory overhead to subtract from runtime container’s memory allocation to determine model capacity

Timeout for model loading operations in milliseconds

Environment variables used to control other aspects of the built-in adapter’s behaviour (uncommon)

Map of InferenceGraph router nodes Each node defines the router which can be different routing types

(Members of Status are embedded into this type.)

Conditions for InferenceGraph

Url for the InferenceGraph

RouterType

• Sequence: chain multiple inference steps with input/output from previous step

• Splitter: randomly routes to the target service according to the weight

• Ensemble: routes the request to multiple models and then merge the responses

• Switch: routes the request to one of the steps based on condition

Steps defines destinations for the current router node

"Ensemble"

Ensemble router routes the requests to multiple models and then merge the responses

"Sequence"

Sequence Default type only route to one destination

"Splitter"

Splitter router randomly routes the requests to the named service according to the weight

"Switch"

Switch routes the request to the model based on certain condition

Unique name for the step within this node

(Members of InferenceTarget are embedded into this type.)

Node or service used to process this step

request data sent to the next route with input/output from the previous step $request $response.predictions

the weight for split of the traffic, only used for Split Router when weight is specified all the routing targets should be sum to 100

routing based on the condition

to decide whether a step is a hard or a soft dependency in the Inference Graph

"Hard"

Hard

"Soft"

Soft

The node name for routing as next step

named reference for InferenceService

InferenceService URL, mutually exclusive with ServiceName

Storage URI for the model repository

Machine Learning The values could be: “tensorflow”,“pytorch”,“sklearn”,“onnx”,“xgboost”, “myawesomeinternalframework” etc.

Maximum memory this model will consume, this field is used to decide if a model server has enough memory to load this model.

"mlserver"

Model server is MLServer

"ovms"

Model server is OpenVino Model Server

"triton"

Model server is Triton

List of containers belonging to the pod. Containers cannot currently be added or removed. There must be at least one container in a Pod. Cannot be updated.

List of volumes that can be mounted by containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes

NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

If specified, the pod’s scheduling constraints

If specified, the pod’s tolerations.

Labels that will be add to the pod. More info: http://kubernetes.io/docs/user-guide/labels

Annotations that will be add to the pod. More info: http://kubernetes.io/docs/user-guide/annotations

ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod

Model formats and version supported by this runtime

Whether this ServingRuntime is intended for multi-model usage or not.

Set to true to disable use of this runtime

Supported protocol versions (i.e. v1 or v2 or grpc-v1 or grpc-v2)

(Members of ServingRuntimePodSpec are embedded into this type.)

Grpc endpoint for internal model-management (implementing mmesh.ModelRuntime gRPC service) Assumed to be single-model runtime if omitted

Grpc endpoint for inferencing

HTTP endpoint for inferencing

Configure the number of replicas in the Deployment generated by this ServingRuntime If specified, this overrides the podsPerRuntime configuration value

Configuration for this runtime’s use of the storage helper (model puller) It is enabled unless explicitly disabled

Provide the details about built-in runtime adapter

Container spec for the storage initializer init container

List of URI formats that this container supports

Name of the model format.

Version of the model format. Used in validating that a predictor is supported by a runtime. Can be “major”, “major.minor” or “major.minor.patch”.

Set to true to allow the ServingRuntime to be used for automatic model placement if this model format is specified with no explicit runtime.

Priority of this serving runtime for auto selection. This is used to select the serving runtime if more than one serving runtime supports the same model format. The value should be greater than zero. The higher the value, the higher the priority. Priority is not considered if AutoSelect is either false or not specified. Priority can be overridden by specifying the runtime in the InferenceService.

parent inference service to deploy to

Predictor model spec

(Members of Status are embedded into this type.)

Conditions for trained model

URL holds the url that will distribute traffic over the provided traffic targets. For v1: http[s]://{route-name}.{route-namespace}.{cluster-level-suffix}/v1/models/:predict For v2: http[s]://{route-name}.{route-namespace}.{cluster-level-suffix}/v2/models//infer

Addressable endpoint for the deployed trained model http:///v1/models/.metadata.name

The type of ART explainer

(Members of ExplainerExtensionSpec are embedded into this type.)

Contains fields shared across all explainers

"SquareAttack"

The type of Alibi explainer
Valid values are:
- “AnchorTabular”;
- “AnchorImages”;
- “AnchorText”;
- “Counterfactuals”;
- “Contrastive”;

(Members of ExplainerExtensionSpec are embedded into this type.)

Contains fields shared across all explainers

"AnchorImages"

"AnchorTabular"

"AnchorText"

"Contrastive"

"Counterfactuals"

Specifies the max number of requests to trigger a batch

Specifies the max latency to trigger a batch

Specifies the timeout of a batch

Minimum number of replicas, defaults to 1 but can be set to 0 to enable scale-to-zero.

Maximum number of replicas for autoscaling.

ScaleTarget specifies the integer target value of the metric type the Autoscaler watches for. concurrency and rps targets are supported by Knative Pod Autoscaler (https://knative.dev/docs/serving/autoscaling/autoscaling-targets/).

ScaleMetric defines the scaling metric type watched by autoscaler possible values are concurrency, rps, cpu, memory. concurrency, rps are supported via Knative Pod Autoscaler(https://knative.dev/docs/serving/autoscaling/autoscaling-metrics).

ContainerConcurrency specifies how many requests can be processed concurrently, this sets the hard limit of the container concurrency(https://knative.dev/docs/serving/autoscaling/concurrency).

TimeoutSeconds specifies the number of seconds to wait before timing out a request to the component.

CanaryTrafficPercent defines the traffic split percentage between the candidate revision and the last ready revision

Activate request/response logging and logger configurations

Activate request batching and batching configurations

Labels that will be add to the component pod. More info: http://kubernetes.io/docs/user-guide/labels

Annotations that will be add to the component pod. More info: http://kubernetes.io/docs/user-guide/annotations

Latest revision name that is in ready state

Latest revision name that is created

Previous revision name that is rolled out with 100 percent traffic

Latest revision name that is rolled out with 100 percent traffic

Traffic holds the configured traffic distribution for latest ready revision and previous rolled out revision.

URL holds the primary url that will distribute traffic over the provided traffic targets. This will be one the REST or gRPC endpoints that are available. It generally has the form http[s]://{route-name}.{route-namespace}.{cluster-level-suffix}

REST endpoint of the component if available.

gRPC endpoint of the component if available.

Addressable endpoint for the InferenceService

"explainer"

"predictor"

"transformer"

(Members of PodSpec are embedded into this type.)

(Members of PodSpec are embedded into this type.)

(Members of PodSpec are embedded into this type.)

explainer docker image name

default explainer docker image version

The location of a trained explanation model

Defaults to latest Explainer Version

Inline custom parameter settings for explainer

(Members of Container are embedded into this type.)

Container enables overrides for the predictor. Each framework will have different defaults that are populated in the underlying container spec.

Storage Spec for model location

Spec for alibi explainer

Spec for ART explainer

(Members of PodSpec are embedded into this type.)

This spec is dual purpose. 1) Users may choose to provide a full PodSpec for their custom explainer. The field PodSpec.Containers is mutually exclusive with other explainers (i.e. Alibi). 2) Users may choose to provide a Explainer (i.e. Alibi) and specify PodSpec overrides in the PodSpec. They must not provide PodSpec.Containers in this case.

(Members of ComponentExtensionSpec are embedded into this type.)

Component extension defines the deployment configurations for explainer

Name of component to which the failure relates (usually Pod name)

High level class of failure

Detailed error message

Internal Revision/ID of model, tied to specific Spec contents

Time failure occurred or was discovered

Exit status from the last termination of the container

"InvalidPredictorSpec"

The current Predictor Spec is invalid or unsupported

"ModelLoadFailed"

The model failed to load within a ServingRuntime container

"NoSupportingRuntime"

There are no ServingRuntime which support the specified model type

"RuntimeDisabled"

The ServingRuntime is disabled

"RuntimeNotRecognized"

There is no ServingRuntime defined with the specified runtime name

"RuntimeUnhealthy"

Corresponding ServingRuntime containers failed to start or are unhealthy

Predictor defines the model serving spec

Explainer defines the model explanation service spec, explainer service calls to predictor or transformer if it is specified.

Transformer defines the pre/post processing before and after the predictor call, transformer service calls to predictor service.

(Members of Status are embedded into this type.)

Conditions for the InferenceService
- PredictorReady: predictor readiness condition;
- TransformerReady: transformer readiness condition;
- ExplainerReady: explainer readiness condition;
- RoutesReady (serverless mode only): aggregated routing condition, i.e. endpoint readiness condition;
- LatestDeploymentReady (serverless mode only): aggregated configuration condition, i.e. latest deployment readiness condition;
- Ready: aggregated condition;

Addressable endpoint for the InferenceService

URL holds the url that will distribute traffic over the provided traffic targets. It generally has the form http[s]://{route-name}.{route-namespace}.{cluster-level-suffix}

Statuses for the components of the InferenceService

Model related statuses

Explainer configurations

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors

URL to send logging events

Specifies the scope of the loggers.
Valid values are:
- “all” (default): log both request and response;
- “request”: log only request;
- “response”: log only response

"all"

Logger mode to log both request and response

"request"

Logger mode to log only request

"response"

Logger mode to log only response

How many copies of this predictor’s models failed to load recently

Total number copies of this predictor’s models that are currently loaded

Name of the model format.

Version of the model format. Used in validating that a predictor is supported by a runtime. Can be “major”, “major.minor” or “major.minor.patch”.

High level state string: Pending, Standby, Loading, Loaded, FailedToLoad

ModelFormat being served.

Specific ClusterServingRuntime/ServingRuntime name to use for deployment.

(Members of PredictorExtensionSpec are embedded into this type.)

"FailedToLoad"

All copies of the model failed to load

"Loaded"

At least one copy of the model is loaded

"Loading"

Model is loading

"Pending"

Model is not yet registered

"Standby"

Model is available but not loaded (will load when used)

Whether the available predictor endpoints reflect the current Spec or is in transition

State information of the predictor’s model.

Details of last failure, when load of target model is failed or blocked.

Model copy information of the predictor’s model.

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors

(Members of PredictorExtensionSpec are embedded into this type.)

List of volumes that can be mounted by containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes

List of initialization containers belonging to the pod. Init containers are executed in order prior to containers being started. If any init container fails, the pod is considered to have failed and is handled according to its restartPolicy. The name for an init container or normal container must be unique among all containers. Init containers may not have Lifecycle actions, Readiness probes, Liveness probes, or Startup probes. The resourceRequirements of an init container are taken into account during scheduling by finding the highest request/limit for each resource type, and then using the max of of that value or the sum of the normal containers. Limits are applied to init containers in a similar fashion. Init containers cannot currently be added or removed. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

List of containers belonging to the pod. Containers cannot currently be added or removed. There must be at least one container in a Pod. Cannot be updated.

List of ephemeral containers run in this pod. Ephemeral containers may be run in an existing pod to perform user-initiated actions such as debugging. This list cannot be specified when creating a pod, and it cannot be modified by updating the pod spec. In order to add an ephemeral container to an existing pod, use the pod’s ephemeralcontainers subresource. This field is beta-level and available on clusters that haven’t disabled the EphemeralContainers feature gate.

Restart policy for all containers within the pod. One of Always, OnFailure, Never. Default to Always. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy

Optional duration in seconds the pod needs to terminate gracefully. May be decreased in delete request. Value must be non-negative integer. The value zero indicates stop immediately via the kill signal (no opportunity to shut down). If this value is nil, the default grace period will be used instead. The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Set this value longer than the expected cleanup time for your process. Defaults to 30 seconds.

Optional duration in seconds the pod may be active on the node relative to StartTime before the system will actively try to mark it failed and kill associated containers. Value must be a positive integer.

Set DNS policy for the pod. Defaults to “ClusterFirst”. Valid values are ‘ClusterFirstWithHostNet’, ‘ClusterFirst’, ‘Default’ or ‘None’. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to ‘ClusterFirstWithHostNet’.

NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

ServiceAccountName is the name of the ServiceAccount to use to run this pod. More info: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

DeprecatedServiceAccount is a depreciated alias for ServiceAccountName. Deprecated: Use serviceAccountName instead.

AutomountServiceAccountToken indicates whether a service account token should be automatically mounted.

NodeName is a request to schedule this pod onto a specific node. If it is non-empty, the scheduler simply schedules this pod onto that node, assuming that it fits resource requirements.

Host networking requested for this pod. Use the host’s network namespace. If this option is set, the ports that will be used must be specified. Default to false.

Use the host’s pid namespace. Optional: Default to false.

Use the host’s ipc namespace. Optional: Default to false.

Share a single process namespace between all of the containers in a pod. When this is set containers will be able to view and signal processes from other containers in the same pod, and the first process in each container will not be assigned PID 1. HostPID and ShareProcessNamespace cannot both be set. Optional: Default to false.

SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty. See type description for default values of each field.

ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod

Specifies the hostname of the Pod If not specified, the pod’s hostname will be set to a system-defined value.

If specified, the fully qualified Pod hostname will be “...svc.”. If not specified, the pod will not have a domainname at all.

If specified, the pod’s scheduling constraints

If specified, the pod will be dispatched by specified scheduler. If not specified, the pod will be dispatched by default scheduler.

If specified, the pod’s tolerations.

HostAliases is an optional list of hosts and IPs that will be injected into the pod’s hosts file if specified. This is only valid for non-hostNetwork pods.

If specified, indicates the pod’s priority. “system-node-critical” and “system-cluster-critical” are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default.

The priority value. Various system components use this field to find the priority of the pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority.

Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy.

If specified, all readiness gates will be evaluated for pod readiness. A pod is ready when all its containers are ready AND all conditions specified in the readiness gates have status equal to “True” More info: https://git.k8s.io/enhancements/keps/sig-network/580-pod-readiness-gates

RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the “legacy” RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class This is a beta feature as of Kubernetes v1.14.

EnableServiceLinks indicates whether information about services should be injected into pod’s environment variables, matching the syntax of Docker links. Optional: Defaults to true.

PreemptionPolicy is the Policy for preempting pods with lower priority. One of Never, PreemptLowerPriority. Defaults to PreemptLowerPriority if unset. This field is beta-level, gated by the NonPreemptingPriority feature-gate.

Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. This field will be autopopulated at admission time by the RuntimeClass admission controller. If the RuntimeClass admission controller is enabled, overhead must not be set in Pod create requests. The RuntimeClass admission controller will reject Pod create requests which have the overhead already set. If RuntimeClass is configured and selected in the PodSpec, Overhead will be set to the value defined in the corresponding RuntimeClass, otherwise it will remain unset and treated as zero. More info: https://git.k8s.io/enhancements/keps/sig-node/688-pod-overhead/README.md This field is beta-level as of Kubernetes v1.18, and is only honored by servers that enable the PodOverhead feature.

TopologySpreadConstraints describes how a group of pods ought to spread across topology domains. Scheduler will schedule pods in a way which abides by the constraints. All topologySpreadConstraints are ANDed.

If true the pod’s hostname will be configured as the pod’s FQDN, rather than the leaf name (the default). In Linux containers, this means setting the FQDN in the hostname field of the kernel (the nodename field of struct utsname). In Windows containers, this means setting the registry value of hostname for the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters to FQDN. If a pod does not have FQDN, this has no effect. Default to false.

Specifies the OS of the containers in the pod. Some pod and container fields are restricted if this is set.

If the OS field is set to linux, the following fields must be unset: -securityContext.windowsOptions

If the OS field is set to windows, following fields must be unset: - spec.hostPID - spec.hostIPC - spec.securityContext.seLinuxOptions - spec.securityContext.seccompProfile - spec.securityContext.fsGroup - spec.securityContext.fsGroupChangePolicy - spec.securityContext.sysctls - spec.shareProcessNamespace - spec.securityContext.runAsUser - spec.securityContext.runAsGroup - spec.securityContext.supplementalGroups - spec.containers[].securityContext.seLinuxOptions - spec.containers[].securityContext.seccompProfile - spec.containers[].securityContext.capabilities - spec.containers[].securityContext.readOnlyRootFilesystem - spec.containers[].securityContext.privileged - spec.containers[].securityContext.allowPrivilegeEscalation - spec.containers[].securityContext.procMount - spec.containers[].securityContext.runAsUser - spec.containers[*].securityContext.runAsGroup This is an alpha field and requires the IdentifyPodOS feature

Use the host’s user namespace. Optional: Default to true. If set to true or not present, the pod will be run in the host user namespace, useful for when the pod needs a feature only available to the host user namespace, such as loading a kernel module with CAP_SYS_MODULE. When set to false, a new userns is created for the pod. Setting false is useful for mitigating container breakout vulnerabilities even allowing users to run their containers as root without actually having root privileges on the host. This field is alpha-level and is only honored by servers that enable the UserNamespacesSupport feature.

SchedulingGates is an opaque list of values that if specified will block scheduling the pod. If schedulingGates is not empty, the pod will stay in the SchedulingGated state and the scheduler will not attempt to schedule the pod.

SchedulingGates can only be set at pod creation time, and be removed only afterwards.

This is a beta feature enabled by the PodSchedulingReadiness feature gate.

ResourceClaims defines which ResourceClaims must be allocated and reserved before the Pod is allowed to start. The resources will be made available to those containers which consume them by name.

This is an alpha field and requires enabling the DynamicResourceAllocation feature gate.

This field is immutable.

This field points to the location of the trained model which is mounted onto the pod.

Runtime version of the predictor docker image

Protocol version to use by the predictor (i.e. v1 or v2 or grpc-v1 or grpc-v2)

(Members of Container are embedded into this type.)

Container enables overrides for the predictor. Each framework will have different defaults that are populated in the underlying container spec.

Storage Spec for model location

Spec for SKLearn model server

Spec for XGBoost model server

Spec for TFServing (https://github.com/tensorflow/serving)

Spec for TorchServe (https://pytorch.org/serve)

Spec for Triton Inference Server (https://github.com/triton-inference-server/server)

Spec for ONNX runtime (https://github.com/microsoft/onnxruntime)

Spec for PMML (http://dmg.org/pmml/v4-1/GeneralStructure.html)

Spec for LightGBM model server

Spec for Paddle model server (https://github.com/PaddlePaddle/Serving)

Model spec for any arbitrary framework.

(Members of PodSpec are embedded into this type.)

This spec is dual purpose.
1) Provide a full PodSpec for custom predictor. The field PodSpec.Containers is mutually exclusive with other predictors (i.e. TFServing).
2) Provide a predictor (i.e. TFServing) and specify PodSpec overrides, you must not provide PodSpec.Containers in this case.

(Members of ComponentExtensionSpec are embedded into this type.)

Component extension defines the deployment configurations for a predictor

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors

"cpu"

"concurrency"

"memory"

"rps"

The path to the model object in the storage. It cannot co-exist with the storageURI.

The path to the model schema file in the storage.

Parameters to override the default storage credentials and config.

The Storage Key in the secret for this model.

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors

(Members of PodSpec are embedded into this type.)

This spec is dual purpose.
1) Provide a full PodSpec for custom transformer. The field PodSpec.Containers is mutually exclusive with other transformers.
2) Provide a transformer and specify PodSpec overrides, you must not provide PodSpec.Containers in this case.

(Members of ComponentExtensionSpec are embedded into this type.)

Component extension defines the deployment configurations for a transformer

"BlockedByFailedLoad"

Target model failed to load

"InProgress"

Waiting for target model to reach state of active model

"InvalidSpec"

Target predictor spec failed validation

"UpToDate"

Predictor is up-to-date (reflects current spec)

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors