mirror of
https://github.com/zeek/zeek.git
synced 2025-10-02 14:48:21 +00:00
89 lines
2.7 KiB
ReStructuredText
89 lines
2.7 KiB
ReStructuredText
|
|
=========================================
|
|
Indexed Logging Output with ElasticSearch
|
|
=========================================
|
|
|
|
.. rst-class:: opening
|
|
|
|
Bro's default ASCII log format is not exactly the most efficient
|
|
way for searching large volumes of data. ElasticSearch
|
|
is a new data storage technology for dealing with tons of data.
|
|
It's also a search engine built on top of Apache's Lucene
|
|
project. It scales very well, both for distributed indexing and
|
|
distributed searching.
|
|
|
|
.. contents::
|
|
|
|
Warning
|
|
-------
|
|
|
|
This writer plugin is still in testing and is not yet recommended for
|
|
production use! The approach to how logs are handled in the plugin is "fire
|
|
and forget" at this time, there is no error handling if the server fails to
|
|
respond successfully to the insertion request.
|
|
|
|
Installing ElasticSearch
|
|
------------------------
|
|
|
|
Download the latest version from: <http://www.elasticsearch.org/download/>.
|
|
Once extracted, start ElasticSearch with::
|
|
|
|
# ./bin/elasticsearch
|
|
|
|
For more detailed information, refer to the ElasticSearch installation
|
|
documentation: http://www.elasticsearch.org/guide/reference/setup/installation.html
|
|
|
|
Compiling Bro with ElasticSearch Support
|
|
----------------------------------------
|
|
|
|
First, ensure that you have libcurl installed the run configure.::
|
|
|
|
# ./configure
|
|
[...]
|
|
====================| Bro Build Summary |=====================
|
|
[...]
|
|
cURL: true
|
|
[...]
|
|
ElasticSearch: true
|
|
[...]
|
|
================================================================
|
|
|
|
Activating ElasticSearch
|
|
------------------------
|
|
|
|
The easiest way to enable ElasticSearch output is to load the tuning/logs-to-
|
|
elasticsearch.bro script. If you are using BroControl, the following line in
|
|
local.bro will enable it.
|
|
|
|
.. console::
|
|
|
|
@load tuning/logs-to-elasticsearch
|
|
|
|
With that, Bro will now write most of its logs into ElasticSearch in addition
|
|
to maintaining the Ascii logs like it would do by default. That script has
|
|
some tunable options for choosing which logs to send to ElasticSearch, refer
|
|
to the autogenerated script documentation for those options.
|
|
|
|
There is an interface being written specifically to integrate with the data
|
|
that Bro outputs into ElasticSearch named Brownian. It can be found here::
|
|
|
|
https://github.com/grigorescu/Brownian
|
|
|
|
Tuning
|
|
------
|
|
|
|
A common problem encountered with ElasticSearch is too many files being held
|
|
open. The ElasticSearch website has some suggestions on how to increase the
|
|
open file limit.
|
|
|
|
- http://www.elasticsearch.org/tutorials/2011/04/06/too-many-open-files.html
|
|
|
|
TODO
|
|
----
|
|
|
|
Lots.
|
|
|
|
- Perform multicast discovery for server.
|
|
- Better error detection.
|
|
- Better defaults (don't index loaded-plugins, for instance).
|
|
-
|