README.md 1.45 KB
Newer Older
Jordan Sissel's avatar
-  
Jordan Sissel committed
1
2
# lumberjack

Brandon Burton's avatar
Brandon Burton committed
3
4
o/~ I'm a lumberjack and I'm ok! I sleep when idle, then I ship logs all day! I parse your logs, I eat the JVM agent for lunch! o/~

Jordan Sissel's avatar
-  
Jordan Sissel committed
5
Collect logs locally in preparation for processing elsewhere!
Jordan Sissel's avatar
Jordan Sissel committed
6
7
8

Problem: logstash jar releases are too fat for constrained systems.

Nils Landt's avatar
Nils Landt committed
9
## Goals
Jordan Sissel's avatar
Jordan Sissel committed
10

11
12
* minimize resource usage where possible (cpu, memory, network)
* secure transmission of logs
13
* configurable event data
14
* easy to deploy with minimal moving parts.
Jordan Sissel's avatar
Jordan Sissel committed
15
16
17
18

Simple inputs only:

* follow files, respect rename/truncation conditions
Jordan Sissel's avatar
Jordan Sissel committed
19
* stdin, useful for things like 'varnishlog | lumberjack ...'
Jordan Sissel's avatar
Jordan Sissel committed
20

21
22
23
24
25
## Implementation details 

Below is valid as of 2012/09/19

### Minimize resource usage
Jordan Sissel's avatar
Jordan Sissel committed
26

27
28
* sets small resource limits (memory, open files) on start up based on the number of files being watched
* cpu: sleeps when there is nothing to do
Nils Landt's avatar
Nils Landt committed
29
* network/cpu: sleeps if there is a network failure
30
* network: uses zlib for compression
31

32
### secure transmission
33

34
35
* uses openssl to transport logs. Currently supports verifying the server
  certificate only (so you know who you are sending to).
36

37
### configurable event data
38

39
40
* the protocol lumberjack uses supports sending a string:string map
* the lumberjack tool lets you specify arbitrary extra data with `--field name=value`
Jordan Sissel's avatar
.    
Jordan Sissel committed
41
42
43
44
45
46

## easy deployment

* all dependencies are built at compile-time (openssl, jemalloc, etc)
* 'make deb' (or make rpm) will package everything into a single deb (or rpm)
* bin/lumberjack.sh makes sure the dependencies are found