FAST Extension proposal - MAP
Rolf Andersson / Pantor Engineering
28 Sep 2008 3:24AM ETAll,
this is the third of four proposals announced in a previous post.
Please review and comment.
/Rolf
EXTENSION PROPOSAL - MAP (table of {id, value} mappings)
Background
We have found (empirically in live feeds) that the "bit density"
of some fields is lower than optimum. These fields take one of
very few alternate values. As a consequence, the field value
could be represented with less than 7 bits, which is the minimum
field length in FAST for present values. The bit density could
be increased by allowing packing of multiple field values into
one SBIT-encoded field.
Introduction
A map is a table of values with corresponding identifiers:
{id, value}, {id, value}, ... The id is a integer quantity
whereas the value may be numberic, string, or byte vector
in type. The idea is to replace a larger (size-wise) value
with a more compact id.
The map is shared between the sender and the receiver so
that the sender can define the {id, value} once and then
repeatedly refer to the value by sending the id.
Example
Let's assume that you have a field where the value alternates
between a set of values. As an example, the values of Symbol
(tag 55) may be limited to a set of underlying security symbols.
The encoding af fields like symbol may be more space efficient
with a mechanism that allows an encoder to define and use a set
of values.
Wire representation
A MAP encoding will require additions to the wire format.
One alternative would be to add the following wire format:
A reference is encoded as a non-zero index
A definition is encoded as a zero followed by a non-zero index
and a value. A definition is at the same time a reference.
Abstract example of map wire representation:
|0|1|foo| => index=1 is defined as value='foo'
|1| => a reference to index 1 i.e. 'foo'
|0|1|bar| => index=1 is redefined as value='bar'
(On a side-note, This format allows an encoder to choose between
different replacement algorithms (fifo, lru, ...))
Template Syntax
