diff --git a/doc/logging-elasticsearch.rst b/doc/logging-elasticsearch.rst index b6d22cf5fa..7571c68219 100644 --- a/doc/logging-elasticsearch.rst +++ b/doc/logging-elasticsearch.rst @@ -1,28 +1,38 @@ -======================================== +========================================= Indexed Logging Output with ElasticSearch -======================================== +========================================= .. rst-class:: opening Bro's default ASCII log format is not exactly the most efficient - way for storing and searching large volumes of data. ElasticSearch - is a new and exciting technology for dealing with tons of data. - ElasticSearch is a search engine built on top of Apache's Lucene + way for searching large volumes of data. ElasticSearch + is a new data storage technology for dealing with tons of data. + It's also a search engine built on top of Apache's Lucene project. It scales very well, both for distributed indexing and distributed searching. .. contents:: +Warning +------- + +This writer plugin is still in testing and is not yet recommended for +production use! The approach to how logs are handled in the plugin is "fire +and forget" at this time, there is no error handling if the server fails to +respond successfully to the insertion request. + Installing ElasticSearch ------------------------ -ElasticSearch requires a JRE to run. Please download the latest version -from: . Once extracted, start -ElasticSearch with:: +Download the latest version from: . +Once extracted, start ElasticSearch with:: # ./bin/elasticsearch +For more detailed information, refer to the ElasticSearch installation +documentation: http://www.elasticsearch.org/guide/reference/setup/installation.html + Compiling Bro with ElasticSearch Support ---------------------------------------- @@ -41,49 +51,32 @@ First, ensure that you have libcurl installed the run configure.:: Activating ElasticSearch ------------------------ -The direct way to use ElasticSearch is to switch *all* log files over to -ElasticSearch. To do that, just add ``redef -Log::default_writer=Log::WRITER_ELASTICSEARCH;`` to your ``local.bro``. -For testing, you can also just pass that on the command line:: +The easiest way to enable ElasticSearch output is to load the tuning/logs-to- +elasticsearch.bro script. If you are using BroControl, the following line in +local.bro will enable it. - bro -r trace.pcap Log::default_writer=Log::WRITER_ELASTICSEARCH +.. console:: -With that, Bro will now write all its output into ElasticSearch. You can -inspect these using ElasticSearch's REST-ful interface. For more -information, see: . + @load tuning/logs-to-elasticsearch -There is also a rudimentary web interface to ElasticSearch, available at: -. +With that, Bro will now write most of its logs into ElasticSearch in addition +to maintaining the Ascii logs like it would do by default. That script has +some tunable options for choosing which logs to send to ElasticSearch, refer +to the autogenerated script documentation for those options. -You can also switch only individual files over to ElasticSearch by adding -code like this to your ``local.bro``:: +There is an interface being written specifically to integrate with the data +that Bro outputs into ElasticSearch named Brownian. It can be found here:: -.. code::bro + https://github.com/grigorescu/Brownian - event bro_init() - { - local f = Log::get_filter(Conn::LOG, "default"); # Get default filter for connection log. - f$writer = Log::WRITER_ELASTICSEARCH; # Change writer type. - Log::add_filter(Conn::LOG, f); # Replace filter with adapted version. - } +Tuning +------ -Configuring ElasticSearch -------------------------- +A common problem encountered with ElasticSearch is too many files being held +open. The ElasticSearch website has some suggestions on how to increase the +open file limit. -Bro's ElasticSearch writer comes with a few configuration options:: - -- cluster_name: Currently unused. - -- server_host: Where to send the data. Default localhost. - -- server_port: What port to send the data to. Default 9200. - -- index_prefix: ElasticSearch indexes are like databases in a standard DB model. - This is the name of the index to which to send the data. Default bro. - -- type_prefix: ElasticSearch types are like tables in a standard DB model. This is a prefix that gets prepended to Bro log names. Example: type_prefix = "bro_" would create types "bro_dns", "bro_http", etc. Default: none. - -- batch_size: How many messages to buffer before sending to ElasticSearch. This is mainly a memory optimization - changing this doesn't seem to affect indexing performance that much. Default: 10,000. + - http://www.elasticsearch.org/tutorials/2011/04/06/too-many-open-files.html TODO ---- @@ -93,3 +86,4 @@ Lots. - Perform multicast discovery for server. - Better error detection. - Better defaults (don't index loaded-plugins, for instance). +-