README.md 1.15 KB
Newer Older
Jordan Sissel's avatar
-  
Jordan Sissel committed
1
2
# lumberjack

Brandon Burton's avatar
Brandon Burton committed
3
4
o/~ I'm a lumberjack and I'm ok! I sleep when idle, then I ship logs all day! I parse your logs, I eat the JVM agent for lunch! o/~

Jordan Sissel's avatar
-  
Jordan Sissel committed
5
Collect logs locally in preparation for processing elsewhere!
Jordan Sissel's avatar
Jordan Sissel committed
6
7
8
9
10

Problem: logstash jar releases are too fat for constrained systems.

Goal: Something small, fast, and light-weight to ship local logs externally.

Jordan Sissel's avatar
Jordan Sissel committed
11
## Requirements
Jordan Sissel's avatar
Jordan Sissel committed
12

Jordan Sissel's avatar
Jordan Sissel committed
13
* minimal resource usage
14
* configurable event data
Jordan Sissel's avatar
Jordan Sissel committed
15
* encryption and compression
Jordan Sissel's avatar
Jordan Sissel committed
16
17
18
19

Simple inputs only:

* follow files, respect rename/truncation conditions
Jordan Sissel's avatar
Jordan Sissel committed
20
21
* local sockets, maybe, if syslog(3) is worth supporting.
* stdin, useful for things like 'varnishlog | lumberjack ...'
Jordan Sissel's avatar
Jordan Sissel committed
22
23
24
25

Simple outputs only:

* custom wire event protocol (TBD)
26
27
28
29
30
31
32
33
34

## Tentative idea:

    # Ship apache logs in real time to somehost:12345
    ./lumberjack --target somehost:12345 /var/log/apache/access.log ...

    # Ship apache logs with additional log fields:
    ./lumberjack --target foo:12345 --field host=$HOSTNAME --field role=apt-repo /mnt/apt/access.log

Jordan Sissel's avatar
Jordan Sissel committed
35
36
37
38
* Serialization: msgpack (likely)
* Encryption: SSL
* Authentication (both directions): SSL certificates
* Compression: TLS v1 comes with compression, might be sufficient.