gRPC API Reference

ServerLive

The ServerLive API indicates if the inference server is able to receive and respond to metadata and inference requests.

rpc inference.GRPCInferenceService/ServerLive(ServerLiveRequest) returns ServerLiveResponse

ServerReady

The ServerReady API indicates if the server is ready for inferencing.

rpc inference.GRPCInferenceService/ServerReady(ServerReadyRequest) returns ServerReadyResponse

ModelReady

The ModelReady API indicates if a specific model is ready for inferencing.

rpc inference.GRPCInferenceService/ModelReady(ModelReadyRequest) returns ModelReadyResponse

ServerMetadata

The ServerMetadata API provides information about the server. Errors are indicated by the google.rpc.Status returned for the request. The OK code indicates success and other codes indicate failure.

rpc inference.GRPCInferenceService/ServerMetadata(ServerMetadataRequest) returns ServerMetadataResponse

ModelMetadata

The per-model metadata API provides information about a model. Errors are indicated by the google.rpc.Status returned for the request. The OK code indicates success and other codes indicate failure.

rpc inference.GRPCInferenceService/ModelMetadata(ModelMetadataRequest) returns ModelMetadataResponse

ModelInfer

The ModelInfer API performs inference using the specified model. Errors are indicated by the google.rpc.Status returned for the request. The OK code indicates success and other codes indicate failure.

rpc inference.GRPCInferenceService/ModelInfer(ModelInferRequest) returns ModelInferResponse

---

Messages

InferParameter

An inference parameter value. The Parameters message describes a “name”/”value” pair, where the “name” is the name of the parameter and the “value” is a boolean, integer, or string corresponding to the parameter.

Field	Type	Description
oneof parameter_choice.bool_param	bool	A boolean parameter value.
oneof parameter_choice.int64_param	int64	An int64 parameter value.
oneof parameter_choice.string_param	string	A string parameter value.
oneof parameter_choice.double_param	double	A double parameter value.
oneof parameter_choice.uint64_param	uint64	A uint64 parameter value.

InferTensorContents

The data contained in a tensor represented by the repeated type that matches the tensor's data type. Protobuf oneof is not used because oneofs cannot contain repeated fields.

Field	Type	Description
bool_contents	repeated bool	Representation for BOOL data type. The size must match what is expected by the tensor's shape. The contents must be the flattened, one-dimensional, row-major order of the tensor elements.
int_contents	repeated int32	Representation for INT8, INT16, and INT32 data types. The size must match what is expected by the tensor's shape. The contents must be the flattened, one-dimensional, row-major order of the tensor elements.
int64_contents	repeated int64	Representation for INT64 data types. The size must match what is expected by the tensor's shape. The contents must be the flattened, one-dimensional, row-major order of the tensor elements.
uint_contents	repeated uint32	Representation for UINT8, UINT16, and UINT32 data types. The size must match what is expected by the tensor's shape. The contents must be the flattened, one-dimensional, row-major order of the tensor elements.
uint64_contents	repeated uint64	Representation for UINT64 data types. The size must match what is expected by the tensor's shape. The contents must be the flattened, one-dimensional, row-major order of the tensor elements.
fp32_contents	repeated float	Representation for FP32 data type. The size must match what is expected by the tensor's shape. The contents must be the flattened, one-dimensional, row-major order of the tensor elements.
fp64_contents	repeated double	Representation for FP64 data type. The size must match what is expected by the tensor's shape. The contents must be the flattened, one-dimensional, row-major order of the tensor elements.
bytes_contents	repeated bytes	Representation for BYTES data type. The size must match what is expected by the tensor's shape. The contents must be the flattened, one-dimensional, row-major order of the tensor elements.

ModelInferRequest

Field	Type	Description
model_name	string	The name of the model to use for inferencing.
model_version	string	The version of the model to use for inference. If not given the server will choose a version based on the model and internal policy.
id	string	Optional identifier for the request. If specified will be returned in the response.
parameters	map ModelInferRequest.ParametersEntry	Optional inference parameters.
inputs	repeated ModelInferRequest.InferInputTensor	The input tensors for the inference.
outputs	repeated ModelInferRequest.InferRequestedOutputTensor	The requested output tensors for the inference. Optional, if not specified all outputs produced by the model will be returned.
raw_input_contents	repeated bytes	The data contained in an input tensor can be represented in "raw" bytes form or in the repeated type that matches the tensor's data type. To use the raw representation 'raw_input_contents' must be initialized with data for each tensor in the same order as 'inputs'. For each tensor, the size of this content must match what is expected by the tensor's shape and data type. The raw data must be the flattened, one-dimensional, row-major order of the tensor elements without any stride or padding between the elements. Note that the FP16 and BF16 data types must be represented as raw content as there is no specific data type for a 16-bit float type.

If this field is specified then InferInputTensor::contents must not be specified for any input tensor. |

ModelInferRequest.InferInputTensor

An input tensor for an inference request.

Field	Type	Description
name	string	The tensor name.
datatype	string	The tensor data type.
shape	repeated int64	The tensor shape.
parameters	map ModelInferRequest.InferInputTensor.ParametersEntry	Optional inference input tensor parameters.
contents	InferTensorContents	The tensor contents using a data-type format. This field must not be specified if "raw" tensor contents are being used for the inference request.

ModelInferRequest.InferInputTensor.ParametersEntry

Field	Type	Description
key	string	N/A
value	InferParameter	N/A

ModelInferRequest.InferRequestedOutputTensor

An output tensor requested for an inference request.

Field	Type	Description
name	string	The tensor name.
parameters	map ModelInferRequest.InferRequestedOutputTensor.ParametersEntry	Optional requested output tensor parameters.

ModelInferRequest.InferRequestedOutputTensor.ParametersEntry

Field	Type	Description
key	string	N/A
value	InferParameter	N/A

ModelInferRequest.ParametersEntry

Field	Type	Description
key	string	N/A
value	InferParameter	N/A

ModelInferResponse

Field	Type	Description
model_name	string	The name of the model used for inference.
model_version	string	The version of the model used for inference.
id	string	The id of the inference request if one was specified.
parameters	map ModelInferResponse.ParametersEntry	Optional inference response parameters.
outputs	repeated ModelInferResponse.InferOutputTensor	The output tensors holding inference results.
raw_output_contents	repeated bytes	The data contained in an output tensor can be represented in "raw" bytes form or in the repeated type that matches the tensor's data type. To use the raw representation 'raw_output_contents' must be initialized with data for each tensor in the same order as 'outputs'. For each tensor, the size of this content must match what is expected by the tensor's shape and data type. The raw data must be the flattened, one-dimensional, row-major order of the tensor elements without any stride or padding between the elements. Note that the FP16 and BF16 data types must be represented as raw content as there is no specific data type for a 16-bit float type.

If this field is specified then InferOutputTensor::contents must not be specified for any output tensor. |

ModelInferResponse.InferOutputTensor

An output tensor returned for an inference request.

Field	Type	Description
name	string	The tensor name.
datatype	string	The tensor data type.
shape	repeated int64	The tensor shape.
parameters	map ModelInferResponse.InferOutputTensor.ParametersEntry	Optional output tensor parameters.
contents	InferTensorContents	The tensor contents using a data-type format. This field must not be specified if "raw" tensor contents are being used for the inference response.

ModelInferResponse.InferOutputTensor.ParametersEntry

Field	Type	Description
key	string	N/A
value	InferParameter	N/A

ModelInferResponse.ParametersEntry

Field	Type	Description
key	string	N/A
value	InferParameter	N/A

ModelMetadataRequest

Field	Type	Description
name	string	The name of the model.
version	string	The version of the model to check for readiness. If not given the server will choose a version based on the model and internal policy.

ModelMetadataResponse

Field	Type	Description
name	string	The model name.
versions	repeated string	The versions of the model available on the server.
platform	string	The model's platform. See Platforms.
inputs	repeated ModelMetadataResponse.TensorMetadata	The model's inputs.
outputs	repeated ModelMetadataResponse.TensorMetadata	The model's outputs.
properties	map ModelMetadataResponse.PropertiesEntry	Optional Model Properties

ModelMetadataResponse.PropertiesEntry

Field	Type	Description
key	string	N/A
value	string	N/A

ModelMetadataResponse.TensorMetadata

Metadata for a tensor.

Field	Type	Description
name	string	The tensor name.
datatype	string	The tensor data type.
shape	repeated int64	The tensor shape. A variable-size dimension is represented by a -1 value.

ModelReadyRequest

Field	Type	Description
name	string	The name of the model to check for readiness.
version	string	The version of the model to check for readiness. If not given the server will choose a version based on the model and internal policy.

ModelReadyResponse

Field	Type	Description
ready	bool	True if the model is ready, false if not ready.

ServerLiveRequest

ServerLiveResponse

Field	Type	Description
live	bool	True if the inference server is live, false if not live.

ServerMetadataRequest

ServerMetadataResponse

Field	Type	Description
name	string	The server name.
version	string	The server version.
extensions	repeated string	The extensions supported by the server.

ServerReadyRequest

ServerReadyResponse

Field	Type	Description
ready	bool	True if the inference server is ready, false if not ready.