You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
241 lines
12 KiB
Markdown
241 lines
12 KiB
Markdown
2 years ago
|
# Protocol Specification
|
||
|
|
|
||
|
|
The ttrpc protocol is client/server protocol to support multiple request streams
|
||
|
|
over a single connection with lightweight framing. The client represents the
|
||
|
|
process which initiated the underlying connection and the server is the process
|
||
|
|
which accepted the connection. The protocol is currently defined as
|
||
|
|
asymmetrical, with clients sending requests and servers sending responses. Both
|
||
|
|
clients and servers are able to send stream data. The roles are also used in
|
||
|
|
determining the stream identifiers, with client initiated streams using odd
|
||
|
|
number identifiers and server initiated using even number. The protocol may be
|
||
|
|
extended in the future to support server initiated streams, that is not
|
||
|
|
supported in the latest version.
|
||
|
|
|
||
|
|
## Purpose
|
||
|
|
|
||
|
|
The ttrpc protocol is designed to be lightweight and optimized for low latency
|
||
|
|
and reliable connections between processes on the same host. The protocol does
|
||
|
|
not include features for handling unreliable connections such as handshakes,
|
||
|
|
resets, pings, or flow control. The protocol is designed to make low-overhead
|
||
|
|
implementations as simple as possible. It is not intended as a suitable
|
||
|
|
replacement for HTTP2/3 over the network.
|
||
|
|
|
||
|
|
## Message Frame
|
||
|
|
|
||
|
|
Each Message Frame consists of a 10-byte message header followed
|
||
|
|
by message data. The data length and stream ID are both big-endian
|
||
|
|
4-byte unsigned integers. The message type is an unsigned 1-byte
|
||
|
|
integer. The flags are also an unsigned 1-byte integer and
|
||
|
|
use is defined by the message type.
|
||
|
|
|
||
|
|
+---------------------------------------------------------------+
|
||
|
|
| Data Length (32) |
|
||
|
|
+---------------------------------------------------------------+
|
||
|
|
| Stream ID (32) |
|
||
|
|
+---------------+-----------------------------------------------+
|
||
|
|
| Msg Type (8) |
|
||
|
|
+---------------+
|
||
|
|
| Flags (8) |
|
||
|
|
+---------------+-----------------------------------------------+
|
||
|
|
| Data (*) |
|
||
|
|
+---------------------------------------------------------------+
|
||
|
|
|
||
|
|
The Data Length field represents the number of bytes in the Data field. The
|
||
|
|
total frame size will always be Data Length + 10 bytes. The maximum data length
|
||
|
|
is 4MB and any larger size should be rejected. Due to the maximum data size
|
||
|
|
being less than 16MB, the first frame byte should always be zero. This first
|
||
|
|
byte should be considered reserved for future use.
|
||
|
|
|
||
|
|
The Stream ID must be odd for client initiated streams and even for server
|
||
|
|
initiated streams. Server initiated streams are not currently supported.
|
||
|
|
|
||
|
|
## Mesage Types
|
||
|
|
|
||
|
|
| Message Type | Name | Description |
|
||
|
|
|--------------|----------|----------------------------------|
|
||
|
|
| 0x01 | Request | Initiates stream |
|
||
|
|
| 0x02 | Response | Final stream data and terminates |
|
||
|
|
| 0x03 | Data | Stream data |
|
||
|
|
|
||
|
|
### Request
|
||
|
|
|
||
|
|
The request message is used to initiate stream and send along request data for
|
||
|
|
properly routing and handling the stream. The stream may indicate unary without
|
||
|
|
any inbound or outbound stream data with only a response is expected on the
|
||
|
|
stream. The request may also indicate the stream is still open for more data and
|
||
|
|
no response is expected until data is finished. If the remote indicates the
|
||
|
|
stream is closed, the request may be considered non-unary but without anymore
|
||
|
|
stream data sent. In the case of `remote closed`, the remote still expects to
|
||
|
|
receive a response or stream data. For compatibility with non streaming clients,
|
||
|
|
a request with empty flags indicates a unary request.
|
||
|
|
|
||
|
|
#### Request Flags
|
||
|
|
|
||
|
|
| Flag | Name | Description |
|
||
|
|
|------|-----------------|--------------------------------------------------|
|
||
|
|
| 0x01 | `remote closed` | Non-unary, but no more data expected from remote |
|
||
|
|
| 0x02 | `remote open` | Non-unary, remote is still sending data |
|
||
|
|
|
||
|
|
### Response
|
||
|
|
|
||
|
|
The response message is used to end a stream with data, an empty response, or
|
||
|
|
an error. A response message is the only expected message after a unary request.
|
||
|
|
A non-unary request does not require a response message if the server is sending
|
||
|
|
back stream data. A non-unary stream may return a single response message but no
|
||
|
|
other stream data may follow.
|
||
|
|
|
||
|
|
#### Response Flags
|
||
|
|
|
||
|
|
No response flags are defined at this time, flags should be empty.
|
||
|
|
|
||
|
|
### Data
|
||
|
|
|
||
|
|
The data message is used to send data on an already initialized stream. Either
|
||
|
|
client or server may send data. A data message is not allowed on a unary stream.
|
||
|
|
A data message should not be sent after indicating `remote closed` to the peer.
|
||
|
|
The last data message on a stream must set the `remote closed` flag.
|
||
|
|
|
||
|
|
The `no data` flag is used to indicate that the data message does not include
|
||
|
|
any data. This is normally used with the `remote closed` flag to indicate the
|
||
|
|
stream is now closed without transmitting any data. Since ttrpc normally
|
||
|
|
transmits a single object per message, a zero length data message may be
|
||
|
|
interpreted as an empty object. For example, transmitting the number zero as a
|
||
|
|
protobuf message ends up with a data length of zero, but the message is still
|
||
|
|
considered data and should be processed.
|
||
|
|
|
||
|
|
#### Data Flags
|
||
|
|
|
||
|
|
| Flag | Name | Description |
|
||
|
|
|------|-----------------|-----------------------------------|
|
||
|
|
| 0x01 | `remote closed` | No more data expected from remote |
|
||
|
|
| 0x04 | `no data` | This message does not have data |
|
||
|
|
|
||
|
|
## Streaming
|
||
|
|
|
||
|
|
All ttrpc requests use streams to transfer data. Unary streams will only have
|
||
|
|
two messages sent per stream, a request from a client and a response from the
|
||
|
|
server. Non-unary streams, however, may send any numbers of messages from the
|
||
|
|
client and the server. This makes stream management more complicated than unary
|
||
|
|
streams since both client and server need to track additional state. To keep
|
||
|
|
this management as simple as possible, ttrpc minimizes the number of states and
|
||
|
|
uses two flags instead of control frames. Each stream has two states while a
|
||
|
|
stream is still alive: `local closed` and `remote closed`. Each peer considers
|
||
|
|
local and remote from their own perspective and sets flags from the other peer's
|
||
|
|
perspective. For example, if a client sends a data frame with the
|
||
|
|
`remote closed` flag, that is indicating that the client is now `local closed`
|
||
|
|
and the server will be `remote closed`. A unary operation does not need to send
|
||
|
|
these flags since each received message always indicates `remote closed`. Once a
|
||
|
|
peer is both `local closed` and `remote closed`, the stream is considered
|
||
|
|
finished and may be cleaned up.
|
||
|
|
|
||
|
|
Due to the asymmetric nature of the current protocol, a client should
|
||
|
|
always be in the `local closed` state before `remote closed` and a server should
|
||
|
|
always be in the `remote closed` state before `local closed`. This happens
|
||
|
|
because the client is always initiating requests and a client always expects a
|
||
|
|
final response back from a server to indicate the initiated request has been
|
||
|
|
fulfilled. This may mean server sends a final empty response to finish a stream
|
||
|
|
even after it has already completed sending data before the client.
|
||
|
|
|
||
|
|
### Unary State Diagram
|
||
|
|
|
||
|
|
+--------+ +--------+
|
||
|
|
| Client | | Server |
|
||
|
|
+---+----+ +----+---+
|
||
|
|
| +---------+ |
|
||
|
|
local >---------------+ Request +--------------------> remote
|
||
|
|
closed | +---------+ | closed
|
||
|
|
| |
|
||
|
|
| +----------+ |
|
||
|
|
finished <--------------+ Response +--------------------< finished
|
||
|
|
| +----------+ |
|
||
|
|
| |
|
||
|
|
|
||
|
|
### Non-Unary State Diagrams
|
||
|
|
|
||
|
|
RC: `remote closed` flag
|
||
|
|
RO: `remote open` flag
|
||
|
|
|
||
|
|
+--------+ +--------+
|
||
|
|
| Client | | Server |
|
||
|
|
+---+----+ +----+---+
|
||
|
|
| +--------------+ |
|
||
|
|
>-------------+ Request [RO] +----------------->
|
||
|
|
| +--------------+ |
|
||
|
|
| |
|
||
|
|
| +------+ |
|
||
|
|
>-----------------+ Data +--------------------->
|
||
|
|
| +------+ |
|
||
|
|
| |
|
||
|
|
| +-----------+ |
|
||
|
|
local >---------------+ Data [RC] +------------------> remote
|
||
|
|
closed | +-----------+ | closed
|
||
|
|
| |
|
||
|
|
| +----------+ |
|
||
|
|
finished <--------------+ Response +--------------------< finished
|
||
|
|
| +----------+ |
|
||
|
|
| |
|
||
|
|
|
||
|
|
+--------+ +--------+
|
||
|
|
| Client | | Server |
|
||
|
|
+---+----+ +----+---+
|
||
|
|
| +--------------+ |
|
||
|
|
local >-------------+ Request [RC] +-----------------> remote
|
||
|
|
closed | +--------------+ | closed
|
||
|
|
| |
|
||
|
|
| +------+ |
|
||
|
|
<-----------------+ Data +---------------------<
|
||
|
|
| +------+ |
|
||
|
|
| |
|
||
|
|
| +-----------+ |
|
||
|
|
finished <---------------+ Data [RC] +------------------< finished
|
||
|
|
| +-----------+ |
|
||
|
|
| |
|
||
|
|
|
||
|
|
+--------+ +--------+
|
||
|
|
| Client | | Server |
|
||
|
|
+---+----+ +----+---+
|
||
|
|
| +--------------+ |
|
||
|
|
>-------------+ Request [RO] +----------------->
|
||
|
|
| +--------------+ |
|
||
|
|
| |
|
||
|
|
| +------+ |
|
||
|
|
>-----------------+ Data +--------------------->
|
||
|
|
| +------+ |
|
||
|
|
| |
|
||
|
|
| +------+ |
|
||
|
|
<-----------------+ Data +---------------------<
|
||
|
|
| +------+ |
|
||
|
|
| |
|
||
|
|
| +------+ |
|
||
|
|
>-----------------+ Data +--------------------->
|
||
|
|
| +------+ |
|
||
|
|
| |
|
||
|
|
| +-----------+ |
|
||
|
|
local >---------------+ Data [RC] +------------------> remote
|
||
|
|
closed | +-----------+ | closed
|
||
|
|
| |
|
||
|
|
| +------+ |
|
||
|
|
<-----------------+ Data +---------------------<
|
||
|
|
| +------+ |
|
||
|
|
| |
|
||
|
|
| +-----------+ |
|
||
|
|
finished <---------------+ Data [RC] +------------------< finished
|
||
|
|
| +-----------+ |
|
||
|
|
| |
|
||
|
|
|
||
|
|
## RPC
|
||
|
|
|
||
|
|
While this protocol is defined primarily to support Remote Procedure Calls, the
|
||
|
|
protocol does not define the request and response types beyond the messages
|
||
|
|
defined in the protocol. The implementation provides a default protobuf
|
||
|
|
definition of request and response which may be used for cross language rpc.
|
||
|
|
All implementations should at least define a request type which support
|
||
|
|
routing by procedure name and a response type which supports call status.
|
||
|
|
|
||
|
|
## Version History
|
||
|
|
|
||
|
|
| Version | Features |
|
||
|
|
|---------|---------------------|
|
||
|
|
| 1.0 | Unary requests only |
|
||
|
|
| 1.2 | Streaming support |
|