protocol.md

---

title: Realtime Protocol Definition

Overview

The Ably Realtime protocol operates over a connection-oriented, framed, transport. It supports a single connection (and thus a single application) at a time, and enables traffic belonging to multiple channels to be sent over that single connection.

In the current Ably service the backend supports the Realtime protocol over a WebSocket transport and over a Comet transport. Other transports are possible and this document aims to define the protocol in a transport-independent manner.

The protocol supports both a text-based format using JSON and a binary format using MessagePack. All structures passed on the wire are defined in a JSON-like format. Other encodings may be considered in the future.

Formats

The unit of any protocol transmission is a ProtocolMessage key value Hash object, referred to in this description as a Protocol Message.

The WebSocket transport transmits Protocol Messages in a single WebSocket data frame. In the binary transport, these are sent as binary data frames, with the Protocol Message being encoded with MessagePack. In the text transport these are sent as text data frames, the text itself being the white-space-free JSON encoding of the Protocol Message. Empty fields may be encoded as null, or may be handled as undefined (and thus absent from the JSON encoding) and clients should handle both possibilities. Handling empty fields as undefined is clearly preferable, however, since the encoded text is shorter and the encode and decode overhead is minimized. Binary data payloads are handled in the same was as JSON encoded payloads are handled.

In the Comet protocol, all request and response bodies contain an array of one or more protocol messages. In the binary protocol this is standard MessagePack binary encoding of a MessageSet Hash object. In the text (JSON) protocol request and response bodies are simply JSON-encoded arrays containing the Protocol Message content, again either in binary or text encoding.

Separate sections provide more detailed information on the WebSocket and Comet transports.

Actions

Each Protocol Message has an action that indicates the nature of the message.

• HEARTBEAT (0) := A heartbeat message is sent periodically by the service over a connection in order to keep the connection alive (anticipating that proxies or other gateways may close the connection if it is idle indefinitely) and, in the Comet case, to prevent HTTP request timeout errors. The heartbeat interval is a configurable property of the realtime service and is not selectable by the client.<br>{=html}<br>{=html}
  Heartbeat messages are not exposed to the client app, and are silently consumed in the transport layer of the client library. However, client library unit tests will typically wish to check for the presence of a heartbeat to confirm that a connection is intact, and therefore client libraries should expose some means for tests to observe the occurrence of heartbeats.<br>{=html}<br>{=html}
  No other message fields are populated in a heartbeat message.

• ACK (1) := An acknowledgement message, sent from the service to a client, to confirm receipt of one or more messages published by the client. The ACK message contains a res field with an array of PublishResult objects, one per acknowledged ProtocolMessage, each containing the serials of the messages that were published. Further details of the acknowledgement protocol is given below.

• NACK (2) := An negative acknowledgement message, sent from the service to a client, that indicates failure of one or more messages published by the client. Further details of the acknowledgement protocol is given below.

• CONNECT (3) := Unused. This is a placeholder for transports that might need to pass connection request parameters in a Protocol Message instead of in URI params as happens presently for the WebSocket and Comet transports.

• CONNECTED (4) := Passed by the service back to a client to signify that a connection request has succeeded, and passing various connection parameters (such as the connection ID and connection Key). See connection information for each of the transports for more information.

• DISCONNECT (5) := Unused. This is a placeholder for transports that might need to explicitly disconnect a transport but keep the connection open and connection state available.

• DISCONNECTED (6) := Passed by the service to a client in response to a DISCONNECT message or if a connection has been disconnected because of an error state with the error reason included in the DISCONNECT Protocol Message.

• CLOSE (7) := Passed by a client to the service to request that a connection is closed. Once this message is received by the service, a CLOSED message will be sent by the service and the transport will be closed, if it has not already been closed by the client, and all connection state for that connection will be disposed. No connection state recovery is possible.<br>{=html}<br>{=html}
  Note that to disconnect a transport without destroying connection state, the client simply closes the transport without sending any protocol message.

• CLOSED (8) := Passed by the service back to the client to signify that a connection has been in response to a CLOSE request from the client. The transport will be closed by the service immediately after, and all connection state for that connection will be disposed. No connection state recovery is possible.

• ERROR (9) := Passed by the service to a client to signify an unrecoverable error condition. The scope of the error might be the connection, or a single channel. ERROR messages that apply to a channel have the channel field populated with the channel name; ERROR messages that apply to the connection have no channel field value.<br>{=html}<br>{=html}
  ERROR messages will have a value in the error field that provides information about the failure condition. Client libraries must make a transition (either of the connection or the relevant channel) to the failed state, and pass the contained error information to the client.<br>{=html}<br>{=html}
  Note that non-fatal errors may occur in various contexts (notably connection state recovery) and CONNECTED and ATTACHED response messages may also have the error field populated with information about those non-fatal conditions. These are handled differently from the unrecoverable conditions indicated with an ERROR message.

• ATTACH (10) := Sent by a client to the service to request attachment to a channel. The request must include the channel name in the channel field. Attachment is synchronous, and the client library must place the channel object into the pending state until it sees either an ATTACHED (success) response or an ERROR (failed) response.

• ATTACHED (11) := Sent by the service to a client to indicate successful attachment to a channel. The message contains the channel name in the channel field, and may also contain non-fatal error information in the error field. Clients must move the channel to the attached state and, where present, pass the error information to the caller.<br>{=html}<br>{=html}
  ATTACHED messages may also include a presence flag (right most bit value 1), which if present, indicates that members are currently present on the channel and a presence SYNC is about to commence. Equally, the service might send an ATTACHED message with no presence flag thus indicating that at the time of attach no members are present on the channel and as such the client can assume the presence set is empty.<br>{=html}
  ATTACHED messages are not required to be send in any order relative to their corresponding ATTACH requests, and any other Protocol Message may intervene between the ATTACH and ATTACHED messages.<br>{=html}<br>{=html}
  Clients must silently ignore an ATTACHED response if the channel is not in the pending state.

• DETACH (12) := Sent by a client to the service to request detachment to a channel. The request must include the channel name in the channel field. Detachment is acknowledged by the service, but the client library must place the channel object into the detached state immediately on sending the DETACH request.

• DETACHED (13) := Sent by the service to a client to indicate successful detachment from a channel. The message contains the channel name in the channel field, and may also contain non-fatal error information in the error field. Clients must move the channel to the detached state if not already detached and, where present, pass the error information to the caller.

• PRESENCE (14) := Indicates that the Protocol Message has a payload of one or more PresenceMessages associated with a single channel. PRESENCE messages may be sent in either direction. The channel associated with these presence updates is indicated in the channel field.<br>{=html}<br>{=html}
  The serial number of the message must be included in the msgSerial field (in the client → service direction) or in the connectionSerial field (in the service → client direction). Further information on message serial numbering is given below.

• MESSAGE (15) := Indicates that the Protocol Message has a payload of one or more Messages associated with a single channel. MESSAGE messages may be sent in either direction. The channel associated with these messages is indicated in the channel field.<br>{=html}<br>{=html}
  The serial number of the message must be included in the msgSerial field (in the client → service direction) or in the connectionSerial field (in the service → client direction). Further information on message serial numbering is given below.

• SYNC (16) := Currently reserved for us with presence member synchronization following a channel ATTACHED Protocol Message being received by the client. Once a channel becomes attached, the server will automatically send a list of all members present on the channel to the client. Every SYNC Protocol Message received will contain a channelSerial value and one or more PresenceMessages for each member currently present on the channel i.e. they are in the PRESENT state. The channelSerial serves two purposes, it provides a way for the client library to resume a SYNC should the transport be disconnected, and it also provides a means for the client library to know when the sync is complete. A channelSerial that is being synced will contain an ID followed by a cursor after a colon such as "cf30e75054887:psl_7g:client:189", however the final page of SYNC messages will have a serial with an empty cursor such as "cf30e75054887:". A client can explicitly request a SYNC with an optional channelSerial; if no channelSerial is provided the server will send a complete set of members on the channel; if a channelSerial is provided, the server will resume the SYNC operation.

• AUTH (17) := Sent by the client with a new token to reauthenticate the connection, with the connection either being closed due to incompatible token details being provided, or a CONNECTED message being sent back to the client confirming the authentication succeeded. The server can request that the client authenticates by sending an AUTH protocol message to the client, and the client must respond with a new token in an AUTH protocol message.

• ACTIVATE (18) := Reserved for a deprecated use.

• OBJECT (19) := Indicates that the Protocol Message has a payload of one or more ObjectMessages associated with a single channel. OBJECT messages may be sent in either direction. The channel associated with these messages is indicated in the channel field.<br>{=html}<br>{=html}
  The serial number of the message must be included in the msgSerial field (in the client → service direction). Further information on message serial numbering is given below.

• OBJECT_SYNC (20) := Currently reserved for us with objects synchronization following a channel ATTACHED Protocol Message being received by the client. Once a channel becomes attached, the server will automatically send a snapshot of objects persisted on the channel to the client. Every OBJECT_SYNC Protocol Message received will contain a channelSerial value and one or more ObjectMessages for each object currently persisted on the channel.
  The channelSerial serves two purposes, it provides a way for the client library to resume a OBJECT_SYNC should the transport be disconnected, and it also provides a means for the client library to know when the sync is complete. A channelSerial that is being synced will contain an ID followed by a cursor after a colon such as "cf30e75054887:map:3DYRjGoon2rfGav1VdWVruZ3pX6TQSt8UYYsmo6CqfY@1742208124981", however the final page of OBJECT_SYNC messages will have a serial with an empty cursor such as "cf30e75054887:". The channelSerial ID is guaranteed to not include a colon, but the cursor might. Therefore, if the client library needs to extract the ID and cursor value, it must split the channelSerial at the first colon.
  A client can explicitly request a OBJECT_SYNC with an optional channelSerial; if no channelSerial is provided the server will send a complete snapshot of all objects on the channel; if a channelSerial is provided, the server will resume the OBJECT_SYNC operation.

• ANNOTATION (21) := The `annotations` field of the Protocol Message has a payload of one or more Annotation messages, all associated with a single channel. They may be sent in either direction. The channel associated with these annotations is indicated in the channel field.<br>{=html}<br>{=html}

Protocol Message fields

ProtocolMessages are populated with one or more of the following fields.

• i32 action := Indicates the purpose of the message. See Actions above.

• string id := Unique identifier for each protocol message

• AuthDetails auth := Object used for providing authentication details

• string channel := Present when protocol message applies to a single channel

• string channelSerial := Contains a serial number for a message on the channel

• i32 count := The count field is used for ACK and NACK actions. See message acknowledgement protocol.

• string connectionId := Contains a public string connection ID. This field is populated in the first CONNECTED Protocol Messages from the service to the client. The connection ID is a public identifier used to uniquely identify each connected client.

• string connectionKey := Contains a private string connection Key. Note that this field is soon to be deprecated; when ConnectionDetails#connectionKey is present, it should be considered the definitive connectionKey for the current connection

• ConnectionDetails connectionDetails := provides details on the constraints or defaults for the connection such as max message size, client ID or connection state TTL

• Error error := Contains error information. See Error type description for details of the contained information. The error field is populated in an ERROR message and may also be populated to provide supplementary information (eg for non-fatal errors) in various other message types (CONNECTED, ATTACHED, DETACHED, ACK, NACK).

• i32 flags := Currently used to flag properties in messages such as the presence sync state of an ATTACHED ProtocolMessage. See client library spec TR4i

• i64 msgSerial := Contains a serial number for a message sent from the client to the service. The msgSerial is a zero-based, serially increasing number which, in combination with the connectionId, uniquely identifies the message across the system. The msgSerial is also used in the message acknowledgement protocol.

• list<Message>{=html} messages := A ProtocolMessage with a MESSAGE action contains one or more messages belonging to a channel in this field.

• list<Presence>{=html} presence := A ProtocolMessage with a PRESENCE action contains one or more presence updates belonging to a channel in this field.

• list<Annotation>{=html} annotations := A ProtocolMessage with a ANNOTATION action contains one or more annotations belonging to a channel in this field.

• i64 timestamp := An optional timestamp, applied by the service in messages sent to the client, to indicate the system time at which the message was sent. Note that this differs from the timestamp field of a Message or PresenceMessage which is an indication of the timestamp of receipt of that message by the system.<br>{=html}<br>{=html}
  Currently there are no requirements for the client library to process or populate the timestamp.

• list<PublishResult>{=html} res := Present in ACK Protocol Messages. Contains one PublishResult per acknowledged ProtocolMessage in order. Each PublishResult contains a serials array with the message serials corresponding 1:1 to the messages that were published. A serial may be null if the message was discarded due to a configured conflation rule.

Other message object {#other-message-structs}

The protocol relies on a number of structs embedded in Protocol Messages.

Error

An object containing error information. This is used for unrecoverable error conditions (relating to connections, channels or individual messages) and also "informatively" for non-fatal errors in ATTACHED or CONNECTED responses.

Error contains the following fields.

• i16 statusCode := An optional HTTP response code that most closely matches the nature of the error. This is typically also the actual HTTP response code if this error is reported as an HTTP response to a Comet request.
• i16 code := An Ably-specific error code that indicates the specific error condition. The various codes are defined in errors.json. Implementations may use errors.json to provide descriptive error messages to clients.
• string reason := An optional string message that indicates the error condition, and may contain specific identifying information (e.g. channel name or message serial, for example).

Data

In transports that support JSON encoding, Strings and the JSON Object and Array types are represented as their natural JSON value in the enclosing type. For example, a Message with a string payload would be encoded in JSON (with white-space added here for clarity) as:

{
  "name": "my_event",
  "data": "my_string_payload"
}

For string and binary payload types, an encoded member is optionally added to the enclosing type. The only supported encodings are "utf8" for strings, "json" for JSON and "base64" for binary. If the encoding member is omitted it defaults to "utf8".

Therefore, the following encoded messages each have string type:

{
  "name": "my_event",
  "data": "my string payload"
}
{
  "name": "my_event",
  "data": "my string payload",
  "encoding": "utf8"
}

The following encoded message has binary type:

{
  "name": "my_event",
  "data": "bXkgYmluYXJ5IHBheWxvYWQ=",
  "encoding": "base64"
}

The base64 encoding used is RFC4648 and clients must accept and process values with or without linefeeds.

The following encoded message has JSON type:

{
  "name": "my_event",
  "data": "{\"id\":\"value\"}",
  "encoding": "json"
}

Message

This is an individual channel message.

The members are as follows.

• string id := A globally unique identifier for this message

• string name := The optional event name.

• string clientId := The optional clientId of the client that sent the message.<br>{=html}<br>{=html}
  Client libraries do not need to populate this field if the clientId is implicit (ie a clientId was specified when the library was initialized, and is therefore connection-wide. Also, this field will be empty if no clientId has been specified either on library initialization or when publishing the message.<br>{=html}<br>{=html}
  Messages sent from the service to the client will contain a clientId if one is available.

• i64 timestamp := This is the timestamp indicating the time at which the message was first received by the system from a publishing client. Subsequent updates to this message shall not update the timestamp. The timestamp is included in messages sent by the service to the client. The field is expected to be empty in messages sent from a client to the service.

• string or binary data := The payload of the message, binary is supported when using MessagePack.

• string encoding := A string identifier indicating the encodings applied to the data payload in left to right order. For example, an encoding with the value "utf-8/cipher+aes-128-cbc/base64" indicates that a UTF-8 payload was encrypted with AES 128 CBC encryption and then the binary cipher was encoded with Base64.

• string connectionId := optional public connection ID of the message publisher. The field is expected to be empty in messages sent from a client to the service.

• i32 action := The message action, indicating the type of message operation. The following values are defined: MESSAGE_CREATE (0) for new messages, MESSAGE_UPDATE (1) for updates to existing messages, MESSAGE_DELETE (2) for message deletions, META (3) for metadata messages, MESSAGE_SUMMARY (4) for summary messages, and MESSAGE_APPEND (5) for appending data to existing messages. When publishing an update, delete, or append, the serial field must be populated to identify the target message.

• string serial := An opaque string that uniquely identifies the message. Used when updating, deleting, or appending to an existing message.

• object version := Contains information about the latest version of a message, including version serial, timestamp, and optional operation metadata (clientId, description, metadata) for update/delete/append operations.

PublishResult

Contains the result of a publish operation, returned in ACK Protocol Messages.

The members are as follows.

• list<string>{=html} serials := An array of message serials corresponding 1:1 to the messages that were published in the acknowledged ProtocolMessage. A serial may be null if the message was discarded due to a configured conflation rule.

Presence Message

This is an individual channel presence update, as defined in the Thrift TPresence struct.

The members are as follows.

• string id := A globally unique identifier for this presence message

• Action state := The presence action (ABSENT, PRESENT, ENTER, LEAVE, or UPDATE) encoded as the ordinal in the PresenceMessage Action enum.

• string clientId := The clientId string.

• string or binary data := The optional payload data for the members such as member status, binary is supported when using MessagePack.

• string connectionId := An optional member connection identifier if required to disambiguate multiple connected and entered members having the same clientId. When there are multiple members with the same clientId entered to the channel there will be multiple corresponding ENTER events.<br>{=html}<br>{=html}
  If the connectionId of an already-entered member changes (eg in the situation that a new connection inherits from another connection, the new (clientId, connectionId) combination is not considered to be a new member but it is indicated as an UPDATE of the already-entered member, with a change in the value of connectionId.

Message acknowledgement protocol {#message-acknowledgement}

The Ably client API allows a caller to await the result of publishing messages, or presence updates, to the Ably service. The result indicates success (with a PublishResult containing the serials of published messages) or failure (with an error code), once the Ably service has indicated whether or not it has processed the message successfully. A successful result is not simply an indication that the message sent without error; it is confirmation that the service has processed the message sufficiently that its onward delivery to relevant attached clients is now guaranteed.

In the Comet transport, success or failure is indicated on a per-call basis with an ACK or NACK Message body in the HTTP response to the send API call.

In the WebSocket transport, the service indicates success or failure to the client with the message acknowledgement protocol. This is a simple series of ACK or NACK responses, each addressing a contiguous sequence (by msgSerial) of messages.

An ACK message contains a msgSerial and count value. Receipt of this message signifies that the messages whose serial numbers are:

{ msgSerial ... msgSerial + count - 1 }

have been processed successfully.

The ACK message also contains a res field, which is an array of PublishResult objects corresponding 1:1 to the acknowledged ProtocolMessages in msgSerial order. Each PublishResult contains a serials array with the message serials for each Message in the corresponding ProtocolMessage. This allows the client library to return a PublishResult from publish operations, providing callers with the serials of their published messages. A serial in the array may be null if the message was discarded due to a configured conflation rule.

A NACK message contains a msgSerial and count value and usually also an error value. Receipt of this message signifies that the messages whose serial numbers are:

{ msgSerial ... msgSerial + count - 1 }

have encountered processing failures. The client library must call the callback, if supplied, with the contained error value, or with an error value that indicates an internal error.

ACK responses are sent so as to be responsive to the client but also to avoid sending a response for every single published message; when the client is publishing at a high rate the ACK responses will be sent periodically with an interval of 500ms (say) so many hundreds of messages my be covered in a single ACK response.

NACK responses are sent as soon as an error has arisen, and in any event only cover multiple messages to the extent that those messages are subject to the same underlying failure (since the error information is provided once for the group of NACK'd messages, not individually.

It is a protocol error if the system sends an ACK or NACK that skips past one or more msgSerial without there having been either and ACK or NACK; but a client in this situation should treat this case as implicitly NACKing the skipped messages.

It is also a protocol error if the system sends an ACK or NACK that covers a msgSerial that was covered by an earlier ACK or NACK; in such cases the client library must silently ignore the response insofar as it relates to msgSerials that were covered previously (whether the response is the same now or different).