V1alpha1InferenceRouter¶
InferenceRouter defines the router for each InferenceGraph node with one or multiple steps yaml kind: InferenceGraph metadata: name: canary-route spec: nodes: root: routerType: Splitter routes: - service: mymodel1 weight: 20 - service: mymodel2 weight: 80 yaml kind: InferenceGraph metadata: name: abtest spec: nodes: mymodel: routerType: Switch routes: - service: mymodel1 condition: \"{ .input.userId == 1 }\" - service: mymodel2 condition: \"{ .input.userId == 2 }\" Scoring a case using a model ensemble consists of scoring it using each model separately, then combining the results into a single scoring result using one of the pre-defined combination methods. Tree Ensemble constitutes a case where simple algorithms for combining results of either classification or regression trees are well known. Multiple classification trees, for example, are commonly combined using a "majority-vote" method. Multiple regression trees are often combined using various averaging techniques. e.g tagging models with segment identifiers and weights to be used for their combination in these ways. yaml kind: InferenceGraph metadata: name: ensemble spec: nodes: root: routerType: Sequence routes: - service: feast - nodeName: ensembleModel data: $response ensembleModel: routerType: Ensemble routes: - service: sklearn-model - service: xgboost-model Scoring a case using a sequence, or chain of models allows the output of one model to be passed in as input to the subsequent models. yaml kind: InferenceGraph metadata: name: model-chainer spec: nodes: root: routerType: Sequence routes: - service: mymodel-s1 - service: mymodel-s2 data: $response - service: mymodel-s3 data: $response In the flow described below, the pre_processing node base64 encodes the image and passes it to two model nodes in the flow. The encoded data is available to both these nodes for classification. The second node i.e. dog-breed-classification takes the original input from the pre_processing node along-with the response from the cat-dog-classification node to do further classification of the dog breed if required. yaml kind: InferenceGraph metadata: name: dog-breed-classification spec: nodes: root: routerType: Sequence routes: - service: cat-dog-classifier - nodeName: breed-classifier data: $request breed-classifier: routerType: Switch routes: - service: dog-breed-classifier condition: { .predictions.class == \"dog\" } - service: cat-breed-classifier condition: { .predictions.class == \"cat\" }
Properties¶
| Name | Type | Description | Notes |
|---|---|---|---|
| router_type | str | RouterType - `Sequence:` chain multiple inference steps with input/output from previous step - `Splitter:` randomly routes to the target service according to the weight - `Ensemble:` routes the request to multiple models and then merge the responses - `Switch:` routes the request to one of the steps based on condition | [default to ''] |
| steps | list[V1alpha1InferenceStep] | Steps defines destinations for the current router node | [optional] |