Commit 2f0c98d8 authored by Jordan Sissel's avatar Jordan Sissel

.

parent 397a6448
Same basic functional design as the original lumberjack:
Harvester reads events from files
Event:
Byte offset of start of event (number)
Line number of event (number)
File origin of event (string)
Message (string)
Harvester reads events from files
Event:
Byte offset of start of event (number)
Line number of event (number)
File origin of event (string)
Message (string)
Work model:
Harvester(s)
-> Enveloper (flush when full or after N idle seconds)
-> Compressor (compresses whole envelopes)
-> Encryptor (encrypts compressed envelopes)
-> Emitter (ships over the wire)
Work model:
Harvester(s)
-> Buffer (flush when full or after N idle seconds)
-> Compressor
-> Encryptor
-> Emitter
Sending an envelope of an encrypted, compressed batch of messages allows
me freedom to pick any message-oriented protocol. The previous implementation
of lumberjack requried channel-encryption (with tls) which limited the
kind of transportation tools.
Network model w/ ZMQ:
pushpull events
reqreq for acknowledgements
Previously, compression was done on envelopes, but TLS was used to communicate
securely.
Messaging model w/ ZMQ:
* REQREP message model
REQREP has high latency (lock step request-response) but since
I'm sending multiple events at once, I believe that latency is
unimportant.
Messaging model w/ Redis:
* RPUSH + LPOP
* PUBLISH + SUBSCRIBE
Types of events:
File Event - represents an event read from a file
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment