IEX Transport Specification
IEX Transport Specification
25
Copyright © 201 Exchange LLC. All rights reserved. This document may not be modified, reproduced, or redistributed without the written permission of IEX Group, Inc.
Table of Contents
OVERVIEW ................................................................................................................................................................................... 3
TERMS ........................................................................................................................................................................................... 4
IEX-TP provides a mechanism for listeners to detect and re-request missed segments via TCP or UDP unicast.
This document describes the messages sent between an IEX-TP server and its clients. IEX-TP senders transmit
segments via UDP multicast to carry the data stream sent to the listeners, and the length of a segment will never exceed
the length of the UDP data contents of a packet.
IEX-TP servers will transmit on pre-assigned multicast groups for each messaging protocol on a network via A/B lines
and A/B groups. Receivers must subscribe to either or both of the A/B multicast groups to receive data. A/B multicast
groups are guaranteed to publish payload messages in the same exact order, however receivers should not assume that
the A/B multicast groups will aggregate messages into identical outbound segments.
The listeners (i.e., receivers) must be configured with specific IP address and port combinations to which they can
submit Gap Fill Requests .
A receiver may request a limited retransmission via UDP per request. The size of retransmission response via UDP is
configured by the higher-layer protocol and identified in its specification Response .
The response to a valid UDP Gap Fill Request is one or more standard Outbound Segments unicast back to the source
address of the Gap Fill Request. This allows, for example, downstream IEX-TP users to read retransmitted Outbound
Segments in their multicast processing socket if the request was made from that port (in other words, the client need
The receiver may send a UDP Gap Fill Request with no request ranges specified to test the availability of a Gap Fill
Server . The response to a valid UDP Gap Fill Test Request is one Outbound Segment with
no Payload unicast back to the source address of the UDP Gap Fill Test Request Gap Fill Test Response .
A receiver may request an unlimited retransmission via TCP (in other words, the client may request retransmission of all
messages from the start of a given session when using TCP for gap fills). For example, the last message sequence
number in a requested range may be larger than the latest sequence number received via multicast to account for new
message publication during retransmission.
The response to a valid TCP Gap Fill Request is one or more standard Outbound Segments sent via the existing socket
connection. The connection continues until all requested gaps are filled or when retransmission has caught up with the
most current publication. Once either condition is met, the Gap Fill Server will close the TCP/IP socket connection.
TERMS
Message
A message is an atomic piece of application information carried by an IEX-TP segment. The contents of a message are
defined by the message protocol in use.
Segment
A segment is a collection of messages of a specified messaging protocol, framed in such a way as to support reliable
delivery. For UDP a segment is analogous to a packet. For TCP retransmission, a segment may be up to 64KB in length.
Session
A session is a sequence of one or more messages in a given protocol delineated by a unique Session ID. Once a session
is terminated, no more messages can be sent or retransmitted on that session.
• Timestamp: 8 bytes, signed integer containing a counter of nanoseconds since POSIX (Epoch) time UTC
Note that each byte is represented by two hexadecimal digits in the examples within this specification.
Each Outbound Segment consists of a header and a payload that carries the actual data stream represented as a series
of Message Blocks.
IEX-TP Header
Message Protocol ID
A unique identifier for the higher-layer specification that describes the messages contained within a segment. See the
higher- -TP.
Channel ID
An identifier for a given stream of bytes/sequenced messages. Messages received from multiple sources which use the
same Channel ID are guaranteed to be duplicates by sequence number and/or offset. See the higher-layer protocol
-TP.
Session ID
Session ID uniquely identifies a stream of messages produced by the system. A given message is uniquely identified
within a message protocol by its Session ID and Sequence Number.
Stream Offset is a counter representing the byte offset of the payload in the data stream.
Payload Length is an
Note that the Payload Length field value does not include the length of the IEX-TP Header.
First Message Sequence Number is a counter representing the sequence number of the first message in the segment. If
there is more than one message in a segment, all subsequent messages are implicitly numbered sequentially.
Message Count is a count representing the number of Message Blocks in the segment.
Send Time
The time the Outbound Segment was sent as set by the sender.
Message Block
The first field of a Message Block is the two-byte Message Length field. The remainder of the Message Block is the
variable length Message Data. The first Message Block field, Message Length, will begin after the last byte of the IEX-TP
Header. Subsequent Message Blocks will begin after the last byte of the previous Message Block.
The Message Length is an unsigned binary count representing the number of bytes in a message following the Message
Length field. The Message Length field value does not include the two bytes occupied by the Message Length field. The
total size of the Message Block is the value of the Message Length field plus two.
Message Data
The Message Data is the actual data of the message being transmitted by Transport. It is variable length and can be zero
length. The meaning of the data is application specific.
Heartbeats
Heartbeats are sent by the server if no new messages have been sent in the preceding one second for the given
protocol. A Heartbeat will contain the next byte within the stream for the Stream Offset, the sequence number of the
next message for First Message Sequence Number, and 0 (zero) for both Payload Length and Message Count.
Each Gap Fill Server only responds to Gap Fill Requests of one Request Type. The implementation of higher-layer
protocols using IEX-TP may not support all Request Types. See the higher-layer protocol specification for supported
retransmission Request Type(s) and other Gap Fill Server information. All Request Range Blocks in a Gap Fill Request
must use the same Request Type, not overlap, and monotonically increase. These same restrictions apply over multiple
TCP Gap Fill Requests sent using the same TCP/IP socket connection. Invalid Gap Fill Requests are ignored by Gap Fill
Servers and, in the case of a request sent via TCP, the session will be terminated by the Gap Fill Server.
1 Sequenced Messages
Request Type 1 1 Byte
2 Bytestream
Version
Request Type
Identifies how the Request Range Block of this request is specified: either Sequenced Messages or Bytestream. The
Request Range Block contents must match the Request Type identified.
Message Protocol ID
Indicates the Message Protocol ID to which this request belongs. See the higher-layer protocol specification for the
message identification on IEX-TP .
Channel ID
Indicates the Channel ID to which this request belongs. See the higher-
channel identification on IEX-TP .
Session ID
Indicates the Session ID to which this request belongs. Typically, when initially requesting a gap fill, a receiver will
inspect the first packet via UDP multicast to determine the current active session, and apply such Session ID to the Gap
Fill Request Header. This process will prevent a receiver from accidentally requesting data from a previous session.
Request Range Count is a count representing the number of Request Range Blocks in this request. When sending a
UDP Gap Fill Test Request, the Request Range Count should be zero (0).
First Message Sequence Number 0 8 Long Sequence of the first message requested
Last Message Sequence Number 8 8 Long Sequence of the last message requested
Total Gap Fill Request Block length is 16 bytes. See Appendix A for the bitwise representation.
First Message Sequence Number is a counter representing the sequence number of the first message in the segment
requested. The Last Message Sequence Number is a counter representing the sequence number of the last message in
the segment requested. The request range is inclusive of the First Message Sequence Number and the Last Message
Sequence Number.
Starting Stream Offset is a counter representing the byte offset of the start of the payload in the data stream requested.
Ending Stream Offset is a counter representing the byte offset of the end of the payload in the data stream requested.
The request range is inclusive of the Starting Stream Offset and the Ending Stream Offset.
Multicast Publication
IEX-TP Header
Version 01 // Version: 1
(Reserved) 00 // (Reserved)
Message Protocol ID 04 80 // DEEP v1.0
Channel ID 01 00 00 00 // Channel: 1
Session ID 00 00 87 42 // Today’s current Session ID
Payload Length 48 00 // 72 bytes
Message Count 02 00 // 2 messages
Stream Offset 8c a6 21 00 00 00 00 00 // 2,205,324 bytes
First Sequence Number ca c3 00 00 00 00 00 00 // Sequence: 50122
Send Time ec 45 c2 20 96 86 6d 14 // 2016-08-23 15:30:32.572839404
Segment Payload
Message Block 1
Symbol 5a 49 45 58 54 20 20 20 // ZIEXT
Price 24 1d 0f 00 00 00 00 00 // $99.05
Trade ID 96 8f 06 00 00 00 00 00 // 429974
Symbol 5a 49 45 58 54 20 20 20 // ZIEXT
Price 24 1d 0f 00 00 00 00 00 // $99.05
Version 01 // Version: 1
Request Type 01 // 1 = Sequenced Messages
Message Protocol ID 04 80 // DEEP v1.0
Channel ID 01 00 00 00 // Channel: 1
Session ID 00 00 87 42 // Today’s current Session ID
Request Range Count 02 00 00 00 // 2 request range blocks
Version 01 // Version: 1
Request Type 01 // 1 = Sequenced Messages
Message Protocol ID 04 80 // DEEP v1.0
Channel ID 01 00 00 00 // Channel: 1
Session ID 00 00 87 42 // Today’s current Session ID
Request Range Count 00 00 00 00 // Must have 0 request range blocks
Version 01 // Version: 1
(Reserved) 00 // (Reserved)
Message Protocol ID 02 80 // TOPS v1.5
Channel ID 01 00 00 00 // Channel: 1
Session ID 00 00 87 42 // Today’s current Session ID
Payload Length 00 00 // 0 bytes
Message Count 00 00 // 0 messages
Stream Offset 8c a6 21 00 00 00 00 00 // 2,205,324 bytes
First Sequence Number ca c3 00 00 00 00 00 00 // Sequence: 50122
Send Time ec 45 c2 20 96 86 6d 14 // 2016-08-23 15:30:32.572839404
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Message Length | (Msg Data) | B 41-X
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Message Length | (Msg Data) | B X-Y
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
1.21 August 8, 2016 Clarified Request Header Request Type field values
Added support to test Gap Fill Requests via UDP, even before any data
1.24 October 26, 2016 is published on the multicast feed (UDP Gap Fill Test Request and Gap
Fill Test Response)
Clarified expectations regarding outbound segment construction across
1.25 May 12, 2017 A/B multicast groups
Updated examples to reflect DEEP v1.0