DESIGN 1.39 KB
Newer Older
Jordan Sissel's avatar
.  
Jordan Sissel committed
1 2 3 4 5 6 7
Same basic functional design as the original lumberjack:
  Harvester reads events from files
    Event:
      Byte offset of start of event (number)
      Line number of event (number)
      File origin of event (string)
      Message (string)
8

Jordan Sissel's avatar
.  
Jordan Sissel committed
9 10 11 12 13 14
  Work model:
    Harvester(s) 
      -> Enveloper (flush when full or after N idle seconds)
        -> Compressor (compresses whole envelopes)
          -> Encryptor (encrypts compressed envelopes)
            -> Emitter (ships over the wire)
15

Jordan Sissel's avatar
.  
Jordan Sissel committed
16 17 18 19
Sending an envelope of an encrypted, compressed batch of messages allows
me freedom to pick any message-oriented protocol. The previous implementation
of lumberjack requried channel-encryption (with tls) which limited the 
kind of transportation tools.
20

Jordan Sissel's avatar
.  
Jordan Sissel committed
21 22 23 24 25 26 27 28 29 30 31 32
Previously, compression was done on envelopes, but TLS was used to communicate
securely.

Messaging model w/ ZMQ:
  * REQREP message model
    REQREP has high latency (lock step request-response) but since
    I'm sending multiple events at once, I believe that latency is
    unimportant.

Messaging model w/ Redis:
  * RPUSH + LPOP
  * PUBLISH + SUBSCRIBE
33 34 35 36 37 38 39 40 41 42 43 44 45 46

Types of events:
  File Event - represents an event read from a file
    - file origin of event
    - byte offset of event
    - line number of event
    - event message (the contents)
  Compressed Envelope
    - number of items
    - type of item
    - compressed payload
  Encrypted Envelope
    - cipher
    - payload