PROTOCOL.md 3.37 KB
Newer Older
1 2
# Protocol

3 4
* TLS Encryption: optional, channel
* Compression: optional, frame-based
Jordan Sissel's avatar
.  
Jordan Sissel committed
5

6 7
## Implementation Considerations

8
# Lumberjack Protocol v1
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

## Behavior

Sequence and ack behavior (including sliding window, etc) is similar to TCP,
but instead of bytes, messages are the base unit.

A writer with a window size of 50 events can send up to 50 unacked events
before blocking. A reader can acknowledge the 'last event' received to
support bulk acknowledgements.

Reliable, ordered byte transport is ensured by using TCP (or TLS on top), and
this protocol aims to provide reliable, application-level, message transport.

## Wire Format

24 25
### Layering

26
This entire protocol is built to be layered on top of TCP or TLS.
27

28 29 30 31 32
### Framing

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +---------------+---------------+-------------------------------+
33
     |   version(1)  |   frame type  |     payload ...               |
34 35 36 37 38 39
     +---------------------------------------------------------------+
     |   payload continued...                                        |
     +---------------------------------------------------------------+

### 'data' frame type

Jordan Sissel's avatar
Jordan Sissel committed
40
* SENT FROM WRITER ONLY
41 42
* frame type value: ASCII 'D' aka byte value 0x44

43 44 45 46 47 48
data is a map of string:string pairs. This is analogous to a Hash in Ruby, a
JSON map, etc, but only strings are supported at this time.

Payload:

* 32bit unsigned sequence number
49 50 51 52
* 32bit 'pair' count (how many key/value sequences follow)
* 32bit unsigned key length followed by that many bytes for the key
* 32bit unsigned value length followed by that many bytes for the value
* repeat key/value 'count' times.
53

54 55
Sequence number roll-over: If you receive a sequence number less than the
previous value, this signals that the sequence number has rolled over.
Jordan Sissel's avatar
Jordan Sissel committed
56

57 58
### 'ack' frame type

Jordan Sissel's avatar
Jordan Sissel committed
59
* SENT FROM READER ONLY
60 61 62 63 64
* frame type value: ASCII 'A' aka byte value 0x41

Payload:

* 32bit unsigned sequence number.
65 66

Bulk acks are supported. If you receive data frames in sequence order
Jordan Sissel's avatar
Jordan Sissel committed
67
1,2,3,4,5,6, you can send an ack for '6' and the writer will take this to
68 69 70 71
mean you are acknowledging all data frames before and including '6'.

### 'window size' frame type

Jordan Sissel's avatar
Jordan Sissel committed
72
* SENT FROM WRITER ONLY
73 74 75 76 77 78
* frame type value: ASCII 'W' aka byte value 0x57

Payload:

* 32bit unsigned window size value in units of whole data frames.

Jordan Sissel's avatar
Jordan Sissel committed
79 80
This frame is used to tell the reader the maximum number of unacknowledged
data frames the writer will send before blocking for acks.
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104

### 'compressed' frame type

* SENT FROM WRITER ONLY
* frame type value: ASCII 'C' aka byte value 0x43

Payload:

* 32bit unsigned payload length 
* 'length' bytes of compressed payload

This frame type allows you to compress many frames into a single compressed
envelope and is useful for efficiently compressing many small data frames.

The compressed payload MUST contain full frames only, not partial frames.
The uncompressed payload MUST be a valid frame stream by itself. As an example,
you could have 3 data frames compressed into a single 'compressed' frame type:
1D{k,v}{k,v}1D{k,v}{k,v}1D{k,v}{k,v} - when uncompressed, you should process
the uncompressed payload as you would reading uncompressed frames from the
network.

TODO(sissel): It's likely this model is suboptimal, instead choose to
use whole-stream compression z_stream in zlib (Zlib::ZStream in ruby) might be
preferable.