mirror of
https://github.com/zeek/zeek.git
synced 2025-10-04 15:48:19 +00:00
eef8b7d1c4
I've only tested that it compiles, not whether it still works. The fact that we don't have any tests for this makes me uneasy ... * remotes/origin/topic/seth/elasticsearch: (35 commits) Some documentation updates for elasticsearch plugin. Temporarily removing the ES timeout because it works with signals and is incompatible with Bro threads. Changed ES index names to localtime and added a meta index. New script for easily duplicating logs to ElasticSearch. Some better elasticsearch reliability. Fixed small elasticsearch problem in configure output. Re-adding the needed call to FinishedRotation in the ES writer plugin. Tiny updates. Bringing elasticsearch branch up to date with master. Adding a define to make the stdint C macros available. Adding an extra header. Fixed a bug with messed up time value passing to elasticsearch. Small updates and a little standardization for config.h.in naming. Bug fixes. Bug fix and feature. Forgot to call the parent method for DoHeartBeat. Changed the escaping method. Flush logs to ES daemon as Bro is shutting down. Reduce the batch size to 1000 and add a maximum time interval for batches. Reworked bulk operation string construction to use ODesc and added json escaping. ...
46 lines
1.5 KiB
Text
46 lines
1.5 KiB
Text
##! Log writer for sending logs to an ElasticSearch server.
|
|
##!
|
|
##! Note: This module is in testing and is not yet considered stable!
|
|
##!
|
|
##! There is one known memory issue. If your elasticsearch server is
|
|
##! running slowly and taking too long to return from bulk insert
|
|
##! requests, the message queue to the writer thread will continue
|
|
##! growing larger and larger giving the appearance of a memory leak.
|
|
|
|
module LogElasticSearch;
|
|
|
|
export {
|
|
## Name of the ES cluster
|
|
const cluster_name = "elasticsearch" &redef;
|
|
|
|
## ES Server
|
|
const server_host = "127.0.0.1" &redef;
|
|
|
|
## ES Port
|
|
const server_port = 9200 &redef;
|
|
|
|
## Name of the ES index
|
|
const index_prefix = "bro" &redef;
|
|
|
|
## The ES type prefix comes before the name of the related log.
|
|
## e.g. prefix = "bro_" would create types of bro_dns, bro_software, etc.
|
|
const type_prefix = "" &redef;
|
|
|
|
## The time before an ElasticSearch transfer will timeout.
|
|
## This is not working!
|
|
const transfer_timeout = 2secs;
|
|
|
|
## The batch size is the number of messages that will be queued up before
|
|
## they are sent to be bulk indexed.
|
|
const max_batch_size = 1000 &redef;
|
|
|
|
## The maximum amount of wall-clock time that is allowed to pass without
|
|
## finishing a bulk log send. This represents the maximum delay you
|
|
## would like to have with your logs before they are sent to ElasticSearch.
|
|
const max_batch_interval = 1min &redef;
|
|
|
|
## The maximum byte size for a buffered JSON string to send to the bulk
|
|
## insert API.
|
|
const max_byte_size = 1024 * 1024 &redef;
|
|
}
|
|
|