More documentation updates.

doc/logging-elasticsearch.rst

@@ -1,28 +1,38 @@
 
-========================================
+=========================================
 Indexed Logging Output with ElasticSearch
-========================================
+=========================================
 
 .. rst-class:: opening
 
    Bro's default ASCII log format is not exactly the most efficient
-   way for storing and searching large volumes of data. ElasticSearch
-   is a new and exciting technology for dealing with tons of data.
-   ElasticSearch is a search engine built on top of Apache's Lucene
+   way for searching large volumes of data. ElasticSearch
+   is a new data storage technology for dealing with tons of data.
+   It's also a search engine built on top of Apache's Lucene
    project. It scales very well, both for distributed indexing and 
    distributed searching.
 
 .. contents::
 
+Warning
+-------
+
+This writer plugin is still in testing and is not yet recommended for
+production use!  The approach to how logs are handled in the plugin is "fire
+and forget" at this time, there is no error handling if the server fails to
+respond successfully to the insertion request.
+
 Installing ElasticSearch
 ------------------------
 
-ElasticSearch requires a JRE to run. Please download the latest version
-from: <http://www.elasticsearch.org/download/>. Once extracted, start
-ElasticSearch with::
+Download the latest version from: <http://www.elasticsearch.org/download/>.
+Once extracted, start ElasticSearch with::
 
 # ./bin/elasticsearch
 
+For more detailed information, refer to the ElasticSearch installation
+documentation: http://www.elasticsearch.org/guide/reference/setup/installation.html
+
 Compiling Bro with ElasticSearch Support
 ----------------------------------------
 
@@ -41,49 +51,32 @@ First, ensure that you have libcurl installed the run configure.::
 Activating ElasticSearch
 ------------------------
 
-The direct way to use ElasticSearch is to switch *all* log files over to
-ElasticSearch. To do that, just add ``redef
-Log::default_writer=Log::WRITER_ELASTICSEARCH;`` to your ``local.bro``.
-For testing, you can also just pass that on the command line::
+The easiest way to enable ElasticSearch output is to load the tuning/logs-to-
+elasticsearch.bro script.  If you are using BroControl, the following line in
+local.bro will enable it.
 
-    bro -r trace.pcap Log::default_writer=Log::WRITER_ELASTICSEARCH
+.. console::
 
-With that, Bro will now write all its output into ElasticSearch. You can 
-inspect these using ElasticSearch's REST-ful interface. For more
-information, see: <http://www.elasticsearch.org/guide/reference/api/>.
+    @load tuning/logs-to-elasticsearch
 
-There is also a rudimentary web interface to ElasticSearch, available at:
-<http://mobz.github.com/elasticsearch-head/>.
+With that, Bro will now write most of its logs into ElasticSearch in addition
+to maintaining the Ascii logs like it would do by default.  That script has
+some tunable options for choosing which logs to send to ElasticSearch, refer
+to the autogenerated script documentation for those options.
 
-You can also switch only individual files over to ElasticSearch by adding
-code like this to your ``local.bro``::
+There is an interface being written specifically to integrate with the data
+that Bro outputs into ElasticSearch named Brownian.  It can be found here::
 
-.. code::bro
+    https://github.com/grigorescu/Brownian
 
-    event bro_init()
-        {
-        local f = Log::get_filter(Conn::LOG, "default"); # Get default filter for connection log.
-        f$writer = Log::WRITER_ELASTICSEARCH;               # Change writer type.
-        Log::add_filter(Conn::LOG, f);                   # Replace filter with adapted version.
-        }
+Tuning
+------
 
-Configuring ElasticSearch
--------------------------
+A common problem encountered with ElasticSearch is too many files being held
+open.  The ElasticSearch website has some suggestions on how to increase the
+open file limit.
 
-Bro's ElasticSearch writer comes with a few configuration options::
-
-- cluster_name: Currently unused.
-
-- server_host:  Where to send the data. Default localhost.
-
-- server_port:  What port to send the data to. Default 9200.
-
-- index_prefix:   ElasticSearch indexes are like databases in a standard DB model. 
-  This is the name of the index to which to send the data. Default bro.
-
-- type_prefix:  ElasticSearch types are like tables in a standard DB model. This is a prefix that gets prepended to Bro log names. Example: type_prefix = "bro_" would create types "bro_dns", "bro_http", etc. Default: none.
-
-- batch_size:   How many messages to buffer before sending to ElasticSearch. This is mainly a memory optimization - changing this doesn't seem to affect indexing performance that much. Default: 10,000.
+  - http://www.elasticsearch.org/tutorials/2011/04/06/too-many-open-files.html
 
 TODO
 ----
@@ -93,3 +86,4 @@ Lots.
 - Perform multicast discovery for server.
 - Better error detection.
 - Better defaults (don't index loaded-plugins, for instance).
+-