Skip to content
GitLab
Menu
Projects
Groups
Snippets
Help
Help
Support
Community forum
Keyboard shortcuts
?
Submit feedback
Sign in
Toggle navigation
Menu
Open sidebar
LaaS
logstash-forwarder
Commits
a4b8d96e
Commit
a4b8d96e
authored
Mar 24, 2013
by
James Turnbull
Browse files
Update, edits and cleanup to README
parent
6560f929
Changes
1
Hide whitespace changes
Inline
Side-by-side
README.md
View file @
a4b8d96e
...
@@ -2,7 +2,7 @@
...
@@ -2,7 +2,7 @@
o/~ I'm a lumberjack and I'm ok! I sleep when idle, then I ship logs all day! I parse your logs, I eat the JVM agent for lunch! o/~
o/~ I'm a lumberjack and I'm ok! I sleep when idle, then I ship logs all day! I parse your logs, I eat the JVM agent for lunch! o/~
## Q
UESTIONS?
## Q
uestions and support
If you have questions and cannot find answers, please join the #logstash irc
If you have questions and cannot find answers, please join the #logstash irc
channel on freenode irc or ask on the logstash-users@googlegroups.com mailing
channel on freenode irc or ask on the logstash-users@googlegroups.com mailing
...
@@ -16,29 +16,58 @@ Problem: logstash jar releases are too fat for constrained systems.
...
@@ -16,29 +16,58 @@ Problem: logstash jar releases are too fat for constrained systems.
Solution: lumberjack
Solution: lumberjack
### Goals
*
Minimize resource usage where possible (CPU, memory, network).
*
Secure transmission of logs.
*
Configurable event data.
*
Easy to deploy with minimal moving parts.
*
Simple inputs only:
*
Follows files and respects rename/truncation conditions.
*
Accepts
`STDIN`
, useful for things like
`varnishlog | lumberjack...`
.
## Building it
## Building it
Make sure you have installed FPM (rubygem) and have outgoing FTP access (ftp.openssl.org).
1.
Install
[
FPM
](
https://github.com/jordansissel/fpm
)
$ sudo gem install fpm
2.
Ensure you have outging FTP access to download OpenSS from
`ftp.openssl.org`
.
3.
Compile lumberback
$ git clone git://github.com/jordansissel/lumberjack.git
$ cd lumberback
$ make
4.
Make packages, either:
$ make rpm
Or:
*
compile: make
$ make deb
*
rpm package: make rpm
*
deb package: make deb
Packages install to /opt/lumberjack. Lumberjack builds all necessary
## Installing it
Packages install to
`/opt/lumberjack`
. Lumberjack builds all necessary
dependencies itself, so there should be no run-time dependencies you
dependencies itself, so there should be no run-time dependencies you
need.
need.
## Running it
## Running it
Generally:
`lumberjack.sh --host somehost --port 12345 /var/log/messages`
Generally:
$ lumberjack.sh --host somehost --port 12345 /var/log/messages
See
`lumberjack.sh --help`
for all the flags
See
`lumberjack.sh --help`
for all the flags
Key points
:
###
Key points
*
You'll need an
ssl ca
to verify the server (host) with.
*
You'll need an
SSL CA
to verify the server (host) with.
*
You can specify custom fields with the
'
--field foo=bar
'
. Any number of these
*
You can specify custom fields with the
`
--field foo=bar
`
. Any number of these
may be specified. I use them to set fields like
'
type
'
and other custom
may be specified. I use them to set fields like
`
type
`
and other custom
attributes relevant to each log.
attributes relevant to each log.
*
Any non-flag argument after is considered a file path. You can watch any
*
Any non-flag argument after is considered a file path. You can watch any
number of files.
number of files.
...
@@ -61,66 +90,63 @@ In logstash, you'll want to use the [lumberjack](http://logstash.net/docs/latest
...
@@ -61,66 +90,63 @@ In logstash, you'll want to use the [lumberjack](http://logstash.net/docs/latest
}
}
}
}
## Goals
*
minimize resource usage where possible (cpu, memory, network)
*
secure transmission of logs
*
configurable event data
*
easy to deploy with minimal moving parts.
Simple inputs only:
*
follow files, respect rename/truncation conditions
*
stdin, useful for things like 'varnishlog | lumberjack ...'
## Implementation details
## Implementation details
Below is valid as of 2012/09/19
Below is valid as of 2012/09/19
### Minimize resource usage
### Minimize resource usage
*
s
ets small resource limits (memory, open files) on start up based on the
*
S
ets small resource limits (memory, open files) on start up based on the
number of files being watched
number of files being watched
.
*
cpu
: sleeps when there is nothing to do
*
CPU
: sleeps when there is nothing to do
.
*
n
etwork/
cpu
: sleeps if there is a network failure
*
N
etwork/
CPU
: sleeps if there is a network failure
.
*
n
etwork: uses zlib for compression
*
N
etwork: uses zlib for compression
.
###
s
ecure transmission
###
S
ecure transmission
*
uses openssl to transport logs. Currently supports verifying the server
*
Uses OpenSSL to verify the server certificates (so you know who you
certificate only (so you know who you are sending to).
are sending to).
*
Uses OpenSSL to transport logs.
###
c
onfigurable event data
###
C
onfigurable event data
*
the protocol lumberjack uses supports sending a string:string map
*
The protocol lumberjack uses supports sending a
`string:string`
map.
*
the lumberjack tool lets you specify arbitrary extra data with
`--field name=value`
*
The lumberjack tool lets you specify arbitrary extra data with
`--field name=value`
.
##
e
asy deployment
##
# E
asy deployment
*
all dependencies are built at compile-time (openssl, jemalloc, etc) because many os distributions lack these dependencies.
*
All dependencies are built at compile-time (OpenSSL, jemalloc, etc) because many os distributions lack these dependencies.
*
'make deb' (or make rpm) will package everything into a single deb (or rpm)
*
The
`make deb`
or
`make rpm`
commands will package everything into a
*
bin/lumberjack.sh makes sure the dependencies are found when run in production
single DEB or RPM.
*
The
`bin/lumberjack.sh`
script makes sure the dependencies are found
when run in production.
##
f
uture functional features
##
# F
uture functional features
*
r
e-evaluate globs periodically to look for new log files
*
R
e-evaluate globs periodically to look for new log files
.
*
t
rack position of in the log
*
T
rack position of in the log
.
##
f
uture protocol discussion
##
# F
uture protocol discussion
I would love to not have a custom protocol, but nothing I've found implements
I would love to not have a custom protocol, but nothing I've found implements
what I need, which is: encrypted, trusted, compressed, latency-resilient, and
what I need, which is: encrypted, trusted, compressed, latency-resilient, and
reliable transport of events.
reliable transport of events.
*
r
edis development refuses to accept encryption support, would likely reject
*
R
edis development refuses to accept encryption support, would likely reject
compression as well.
compression as well.
*
z
ero
mq
lacks authentication, encryption, and compression.
*
Z
ero
MQ
lacks authentication, encryption, and compression.
*
t
hrift also lacks authentication, encryption, and compression, and also is an
*
T
hrift also lacks authentication, encryption, and compression, and also is an
RPC framework, not a streaming system.
RPC framework, not a streaming system.
*
w
ebsockets don't do authentication or compression, but support encrypted
*
W
ebsockets don't do authentication or compression, but support encrypted
channels with SSL. Websockets also require XORing the entire payload of all
channels with SSL. Websockets also require XORing the entire payload of all
messages - wasted energy.
messages - wasted energy.
*
SPDY is still changing too frequently and is also RPC. Streaming requires
*
SPDY is still changing too frequently and is also RPC. Streaming requires
custom framing.
custom framing.
*
HTTP is RPC and very high over
head for small events (uncompressable headers,
*
HTTP is RPC and very high overhead for small events (uncompressable headers,
etc). Streaming requires custom framing.
etc). Streaming requires custom framing.
## License
See LICENSE file.
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
.
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment