Control Plane API

The type of AIX explainer

(Members of ExplainerExtensionSpec are embedded into this type.)

Contains fields shared across all explainers

"LimeImages"

The type of ART explainer

(Members of ExplainerExtensionSpec are embedded into this type.)

Contains fields shared across all explainers

"SquareAttack"

The type of Alibi explainer
Valid values are:
- “AnchorTabular”;
- “AnchorImages”;
- “AnchorText”;
- “Counterfactuals”;
- “Contrastive”;

(Members of ExplainerExtensionSpec are embedded into this type.)

Contains fields shared across all explainers

"AnchorImages"

"AnchorTabular"

"AnchorText"

"Contrastive"

"Counterfactuals"

Specifies the max number of requests to trigger a batch

Specifies the max latency to trigger a batch

Specifies the timeout of a batch

Minimum number of replicas, defaults to 1 but can be set to 0 to enable scale-to-zero.

Maximum number of replicas for autoscaling.

ScaleTarget specifies the integer target value of the metric type the Autoscaler watches for. concurrency and rps targets are supported by Knative Pod Autoscaler (https://knative.dev/docs/serving/autoscaling/autoscaling-targets/).

ScaleMetric defines the scaling metric type watched by autoscaler possible values are concurrency, rps, cpu, memory. concurrency, rps are supported via Knative Pod Autoscaler(https://knative.dev/docs/serving/autoscaling/autoscaling-metrics).

ContainerConcurrency specifies how many requests can be processed concurrently, this sets the hard limit of the container concurrency(https://knative.dev/docs/serving/autoscaling/concurrency).

TimeoutSeconds specifies the number of seconds to wait before timing out a request to the component.

CanaryTrafficPercent defines the traffic split percentage between the candidate revision and the last ready revision

Activate request/response logging and logger configurations

Activate request batching and batching configurations

Latest revision name that is in ready state

Latest revision name that is created

Previous revision name that is rolled out with 100 percent traffic

Latest revision name that is rolled out with 100 percent traffic

Traffic holds the configured traffic distribution for latest ready revision and previous rolled out revision.

URL holds the primary url that will distribute traffic over the provided traffic targets. This will be one the REST or gRPC endpoints that are available. It generally has the form http[s]://{route-name}.{route-namespace}.{cluster-level-suffix}

REST endpoint of the component if available.

gRPC endpoint of the component if available.

Addressable endpoint for the InferenceService

"explainer"

"predictor"

"transformer"

(Members of PodSpec are embedded into this type.)

(Members of PodSpec are embedded into this type.)

(Members of PodSpec are embedded into this type.)

explainer docker image name

default explainer docker image version

The location of a trained explanation model

Defaults to latest Explainer Version

Inline custom parameter settings for explainer

(Members of Container are embedded into this type.)

Container enables overrides for the predictor. Each framework will have different defaults that are populated in the underlying container spec.

Storage Spec for model location

Spec for alibi explainer

Spec for AIX explainer

Spec for ART explainer

(Members of PodSpec are embedded into this type.)

This spec is dual purpose. 1) Users may choose to provide a full PodSpec for their custom explainer. The field PodSpec.Containers is mutually exclusive with other explainers (i.e. Alibi). 2) Users may choose to provide a Explainer (i.e. Alibi) and specify PodSpec overrides in the PodSpec. They must not provide PodSpec.Containers in this case.

(Members of ComponentExtensionSpec are embedded into this type.)

Component extension defines the deployment configurations for explainer

Name of component to which the failure relates (usually Pod name)

High level class of failure

Detailed error message

Internal Revision/ID of model, tied to specific Spec contents

Time failure occurred or was discovered

"InvalidPredictorSpec"

The current Predictor Spec is invalid or unsupported

"ModelLoadFailed"

The model failed to load within a ServingRuntime container

"NoSupportingRuntime"

There are no ServingRuntime which support the specified model type

"RuntimeDisabled"

The ServingRuntime is disabled

"RuntimeNotRecognized"

There is no ServingRuntime defined with the specified runtime name

"RuntimeUnhealthy"

Corresponding ServingRuntime containers failed to start or are unhealthy

Predictor defines the model serving spec

Explainer defines the model explanation service spec, explainer service calls to predictor or transformer if it is specified.

Transformer defines the pre/post processing before and after the predictor call, transformer service calls to predictor service.

(Members of Status are embedded into this type.)

Conditions for the InferenceService
- PredictorReady: predictor readiness condition;
- TransformerReady: transformer readiness condition;
- ExplainerReady: explainer readiness condition;
- RoutesReady: aggregated routing condition;
- Ready: aggregated condition;

Addressable endpoint for the InferenceService

URL holds the url that will distribute traffic over the provided traffic targets. It generally has the form http[s]://{route-name}.{route-namespace}.{cluster-level-suffix}

Statuses for the components of the InferenceService

Model related statuses

Transformer configurations

Predictor configurations

Explainer configurations

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors

URL to send logging events

Specifies the scope of the loggers.
Valid values are:
- “all” (default): log both request and response;
- “request”: log only request;
- “response”: log only response

"all"

Logger mode to log both request and response

"request"

Logger mode to log only request

"response"

Logger mode to log only response

How many copies of this predictor’s models failed to load recently

Total number copies of this predictor’s models that are currently loaded

Name of the model format.

Version of the model format. Used in validating that a predictor is supported by a runtime. Can be “major”, “major.minor” or “major.minor.patch”.

High level state string: Pending, Standby, Loading, Loaded, FailedToLoad

ModelFormat being served.

Specific ClusterServingRuntime/ServingRuntime name to use for deployment.

(Members of PredictorExtensionSpec are embedded into this type.)

"FailedToLoad"

All copies of the model failed to load

"Loaded"

At least one copy of the model is loaded

"Loading"

Model is loading

"Pending"

Model is not yet registered

"Standby"

Model is available but not loaded (will load when used)

Whether the available predictor endpoints reflect the current Spec or is in transition

State information of the predictor’s model.

Details of last failure, when load of target model is failed or blocked.

Model copy information of the predictor’s model.

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors

(Members of PredictorExtensionSpec are embedded into this type.)

List of volumes that can be mounted by containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/storage/volumes

List of initialization containers belonging to the pod. Init containers are executed in order prior to containers being started. If any init container fails, the pod is considered to have failed and is handled according to its restartPolicy. The name for an init container or normal container must be unique among all containers. Init containers may not have Lifecycle actions, Readiness probes, Liveness probes, or Startup probes. The resourceRequirements of an init container are taken into account during scheduling by finding the highest request/limit for each resource type, and then using the max of of that value or the sum of the normal containers. Limits are applied to init containers in a similar fashion. Init containers cannot currently be added or removed. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

List of containers belonging to the pod. Containers cannot currently be added or removed. There must be at least one container in a Pod. Cannot be updated.

List of ephemeral containers run in this pod. Ephemeral containers may be run in an existing pod to perform user-initiated actions such as debugging. This list cannot be specified when creating a pod, and it cannot be modified by updating the pod spec. In order to add an ephemeral container to an existing pod, use the pod’s ephemeralcontainers subresource. This field is alpha-level and is only honored by servers that enable the EphemeralContainers feature.

Restart policy for all containers within the pod. One of Always, OnFailure, Never. Default to Always. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#restart-policy

Optional duration in seconds the pod needs to terminate gracefully. May be decreased in delete request. Value must be non-negative integer. The value zero indicates delete immediately. If this value is nil, the default grace period will be used instead. The grace period is the duration in seconds after the processes running in the pod are sent a termination signal and the time when the processes are forcibly halted with a kill signal. Set this value longer than the expected cleanup time for your process. Defaults to 30 seconds.

Optional duration in seconds the pod may be active on the node relative to StartTime before the system will actively try to mark it failed and kill associated containers. Value must be a positive integer.

Set DNS policy for the pod. Defaults to “ClusterFirst”. Valid values are ‘ClusterFirstWithHostNet’, ‘ClusterFirst’, ‘Default’ or ‘None’. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to ‘ClusterFirstWithHostNet’.

NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

ServiceAccountName is the name of the ServiceAccount to use to run this pod. More info: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/

DeprecatedServiceAccount is a depreciated alias for ServiceAccountName. Deprecated: Use serviceAccountName instead.

AutomountServiceAccountToken indicates whether a service account token should be automatically mounted.

NodeName is a request to schedule this pod onto a specific node. If it is non-empty, the scheduler simply schedules this pod onto that node, assuming that it fits resource requirements.

Host networking requested for this pod. Use the host’s network namespace. If this option is set, the ports that will be used must be specified. Default to false.

Use the host’s pid namespace. Optional: Default to false.

Use the host’s ipc namespace. Optional: Default to false.

Share a single process namespace between all of the containers in a pod. When this is set containers will be able to view and signal processes from other containers in the same pod, and the first process in each container will not be assigned PID 1. HostPID and ShareProcessNamespace cannot both be set. Optional: Default to false.

SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty. See type description for default values of each field.

ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod

Specifies the hostname of the Pod If not specified, the pod’s hostname will be set to a system-defined value.

If specified, the fully qualified Pod hostname will be “...svc.”. If not specified, the pod will not have a domainname at all.

If specified, the pod’s scheduling constraints

If specified, the pod will be dispatched by specified scheduler. If not specified, the pod will be dispatched by default scheduler.

If specified, the pod’s tolerations.

HostAliases is an optional list of hosts and IPs that will be injected into the pod’s hosts file if specified. This is only valid for non-hostNetwork pods.

If specified, indicates the pod’s priority. “system-node-critical” and “system-cluster-critical” are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default.

The priority value. Various system components use this field to find the priority of the pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority.

Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy.

If specified, all readiness gates will be evaluated for pod readiness. A pod is ready when all its containers are ready AND all conditions specified in the readiness gates have status equal to “True” More info: https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/580-pod-readiness-gates

RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the “legacy” RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/585-runtime-class This is a beta feature as of Kubernetes v1.14.

EnableServiceLinks indicates whether information about services should be injected into pod’s environment variables, matching the syntax of Docker links. Optional: Defaults to true.

PreemptionPolicy is the Policy for preempting pods with lower priority. One of Never, PreemptLowerPriority. Defaults to PreemptLowerPriority if unset. This field is beta-level, gated by the NonPreemptingPriority feature-gate.

Overhead represents the resource overhead associated with running a pod for a given RuntimeClass. This field will be autopopulated at admission time by the RuntimeClass admission controller. If the RuntimeClass admission controller is enabled, overhead must not be set in Pod create requests. The RuntimeClass admission controller will reject Pod create requests which have the overhead already set. If RuntimeClass is configured and selected in the PodSpec, Overhead will be set to the value defined in the corresponding RuntimeClass, otherwise it will remain unset and treated as zero. More info: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/688-pod-overhead This field is alpha-level as of Kubernetes v1.16, and is only honored by servers that enable the PodOverhead feature.

TopologySpreadConstraints describes how a group of pods ought to spread across topology domains. Scheduler will schedule pods in a way which abides by the constraints. All topologySpreadConstraints are ANDed.

If true the pod’s hostname will be configured as the pod’s FQDN, rather than the leaf name (the default). In Linux containers, this means setting the FQDN in the hostname field of the kernel (the nodename field of struct utsname). In Windows containers, this means setting the registry value of hostname for the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters to FQDN. If a pod does not have FQDN, this has no effect. Default to false.

predictor docker image name

default predictor docker image version on cpu

default predictor docker image version on gpu

Default timeout of predictor for serving a request, in seconds

Flag to determine if multi-model serving is supported

frameworks the model agent is able to run

This field points to the location of the trained model which is mounted onto the pod.

Runtime version of the predictor docker image

Protocol version to use by the predictor (i.e. v1 or v2 or grpc-v1 or grpc-v2)

(Members of Container are embedded into this type.)

Container enables overrides for the predictor. Each framework will have different defaults that are populated in the underlying container spec.

Storage Spec for model location

Spec for SKLearn model server

Spec for XGBoost model server

Spec for TFServing (https://github.com/tensorflow/serving)

Spec for TorchServe (https://pytorch.org/serve)

Spec for Triton Inference Server (https://github.com/triton-inference-server/server)

Spec for ONNX runtime (https://github.com/microsoft/onnxruntime)

Spec for PMML (http://dmg.org/pmml/v4-1/GeneralStructure.html)

Spec for LightGBM model server

Spec for Paddle model server (https://github.com/PaddlePaddle/Serving)

Model spec for any arbitrary framework.

(Members of PodSpec are embedded into this type.)

This spec is dual purpose.
1) Provide a full PodSpec for custom predictor. The field PodSpec.Containers is mutually exclusive with other predictors (i.e. TFServing).
2) Provide a predictor (i.e. TFServing) and specify PodSpec overrides, you must not provide PodSpec.Containers in this case.

(Members of ComponentExtensionSpec are embedded into this type.)

Component extension defines the deployment configurations for a predictor

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors

"cpu"

"concurrency"

"memory"

"rps"

The path to the model object in the storage. It cannot co-exist with the storageURI.

The path to the model schema file in the storage.

Parameters to override the default storage credentials and config.

The Storage Key in the secret for this model.

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors

transformer docker image name

default transformer docker image version

(Members of PodSpec are embedded into this type.)

This spec is dual purpose.
1) Provide a full PodSpec for custom transformer. The field PodSpec.Containers is mutually exclusive with other transformers.
2) Provide a transformer and specify PodSpec overrides, you must not provide PodSpec.Containers in this case.

(Members of ComponentExtensionSpec are embedded into this type.)

Component extension defines the deployment configurations for a transformer

"BlockedByFailedLoad"

Target model failed to load

"InProgress"

Waiting for target model to reach state of active model

"InvalidSpec"

Target predictor spec failed validation

"UpToDate"

Predictor is up-to-date (reflects current spec)

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors

(Members of PredictorExtensionSpec are embedded into this type.)

Contains fields shared across all predictors