Skip to content
GitLab
Menu
Projects
Groups
Snippets
Loading...
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Sign in
Toggle navigation
Menu
Open sidebar
LaaS
logstash-forwarder
Commits
a4b8d96e
Commit
a4b8d96e
authored
Mar 24, 2013
by
James Turnbull
Browse files
Update, edits and cleanup to README
parent
6560f929
Changes
1
Hide whitespace changes
Inline
Side-by-side
README.md
View file @
a4b8d96e
...
...
@@ -2,7 +2,7 @@
o/~ I'm a lumberjack and I'm ok! I sleep when idle, then I ship logs all day! I parse your logs, I eat the JVM agent for lunch! o/~
## Q
UESTIONS?
## Q
uestions and support
If you have questions and cannot find answers, please join the #logstash irc
channel on freenode irc or ask on the logstash-users@googlegroups.com mailing
...
...
@@ -16,29 +16,58 @@ Problem: logstash jar releases are too fat for constrained systems.
Solution: lumberjack
### Goals
*
Minimize resource usage where possible (CPU, memory, network).
*
Secure transmission of logs.
*
Configurable event data.
*
Easy to deploy with minimal moving parts.
*
Simple inputs only:
*
Follows files and respects rename/truncation conditions.
*
Accepts
`STDIN`
, useful for things like
`varnishlog | lumberjack...`
.
## Building it
Make sure you have installed FPM (rubygem) and have outgoing FTP access (ftp.openssl.org).
1.
Install
[
FPM
](
https://github.com/jordansissel/fpm
)
$ sudo gem install fpm
2.
Ensure you have outging FTP access to download OpenSS from
`ftp.openssl.org`
.
3.
Compile lumberback
$ git clone git://github.com/jordansissel/lumberjack.git
$ cd lumberback
$ make
4.
Make packages, either:
$ make rpm
Or:
*
compile: make
*
rpm package: make rpm
*
deb package: make deb
$ make deb
Packages install to /opt/lumberjack. Lumberjack builds all necessary
## Installing it
Packages install to
`/opt/lumberjack`
. Lumberjack builds all necessary
dependencies itself, so there should be no run-time dependencies you
need.
## Running it
Generally:
`lumberjack.sh --host somehost --port 12345 /var/log/messages`
Generally:
$ lumberjack.sh --host somehost --port 12345 /var/log/messages
See
`lumberjack.sh --help`
for all the flags
Key points
:
###
Key points
*
You'll need an
ssl ca
to verify the server (host) with.
*
You can specify custom fields with the
'
--field foo=bar
'
. Any number of these
may be specified. I use them to set fields like
'
type
'
and other custom
*
You'll need an
SSL CA
to verify the server (host) with.
*
You can specify custom fields with the
`
--field foo=bar
`
. Any number of these
may be specified. I use them to set fields like
`
type
`
and other custom
attributes relevant to each log.
*
Any non-flag argument after is considered a file path. You can watch any
number of files.
...
...
@@ -61,66 +90,63 @@ In logstash, you'll want to use the [lumberjack](http://logstash.net/docs/latest
}
}
## Goals
*
minimize resource usage where possible (cpu, memory, network)
*
secure transmission of logs
*
configurable event data
*
easy to deploy with minimal moving parts.
Simple inputs only:
*
follow files, respect rename/truncation conditions
*
stdin, useful for things like 'varnishlog | lumberjack ...'
## Implementation details
Below is valid as of 2012/09/19
### Minimize resource usage
*
s
ets small resource limits (memory, open files) on start up based on the
number of files being watched
*
cpu
: sleeps when there is nothing to do
*
n
etwork/
cpu
: sleeps if there is a network failure
*
n
etwork: uses zlib for compression
*
S
ets small resource limits (memory, open files) on start up based on the
number of files being watched
.
*
CPU
: sleeps when there is nothing to do
.
*
N
etwork/
CPU
: sleeps if there is a network failure
.
*
N
etwork: uses zlib for compression
.
###
s
ecure transmission
###
S
ecure transmission
*
uses openssl to transport logs. Currently supports verifying the server
certificate only (so you know who you are sending to).
*
Uses OpenSSL to verify the server certificates (so you know who you
are sending to).
*
Uses OpenSSL to transport logs.
###
c
onfigurable event data
###
C
onfigurable event data
*
the protocol lumberjack uses supports sending a string:string map
*
the lumberjack tool lets you specify arbitrary extra data with
`--field name=value`
*
The protocol lumberjack uses supports sending a
`string:string`
map.
*
The lumberjack tool lets you specify arbitrary extra data with
`--field name=value`
.
##
e
asy deployment
##
# E
asy deployment
*
all dependencies are built at compile-time (openssl, jemalloc, etc) because many os distributions lack these dependencies.
*
'make deb' (or make rpm) will package everything into a single deb (or rpm)
*
bin/lumberjack.sh makes sure the dependencies are found when run in production
*
All dependencies are built at compile-time (OpenSSL, jemalloc, etc) because many os distributions lack these dependencies.
*
The
`make deb`
or
`make rpm`
commands will package everything into a
single DEB or RPM.
*
The
`bin/lumberjack.sh`
script makes sure the dependencies are found
when run in production.
##
f
uture functional features
##
# F
uture functional features
*
r
e-evaluate globs periodically to look for new log files
*
t
rack position of in the log
*
R
e-evaluate globs periodically to look for new log files
.
*
T
rack position of in the log
.
##
f
uture protocol discussion
##
# F
uture protocol discussion
I would love to not have a custom protocol, but nothing I've found implements
what I need, which is: encrypted, trusted, compressed, latency-resilient, and
reliable transport of events.
*
r
edis development refuses to accept encryption support, would likely reject
*
R
edis development refuses to accept encryption support, would likely reject
compression as well.
*
z
ero
mq
lacks authentication, encryption, and compression.
*
t
hrift also lacks authentication, encryption, and compression, and also is an
*
Z
ero
MQ
lacks authentication, encryption, and compression.
*
T
hrift also lacks authentication, encryption, and compression, and also is an
RPC framework, not a streaming system.
*
w
ebsockets don't do authentication or compression, but support encrypted
*
W
ebsockets don't do authentication or compression, but support encrypted
channels with SSL. Websockets also require XORing the entire payload of all
messages - wasted energy.
*
SPDY is still changing too frequently and is also RPC. Streaming requires
custom framing.
*
HTTP is RPC and very high over
head for small events (uncompressable headers,
*
HTTP is RPC and very high overhead for small events (uncompressable headers,
etc). Streaming requires custom framing.
## License
See LICENSE file.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment