Commit graph

916 commits

Author SHA1 Message Date
Justin Azoff
cc2981ab3b improve performance of catch and release script
Defer allocation of the Entity record until it will be used.
2019-01-09 13:58:39 -05:00
Jon Siwek
e0b8b4b6b1 Replace some bro.org usages with zeek.org 2019-01-04 17:51:25 -06:00
Jon Siwek
01e17b5ea0 Fix SumStats "last" plugin in cluster mode 2018-11-08 12:27:37 -06:00
Jon Siwek
4bd6da7186 Update default Broker/CAF thread tuning 2018-09-07 17:50:28 -05:00
Jon Siwek
73c8cf733a Give Cluster::rr_topic "key" argument a default value 2018-09-07 09:58:57 -05:00
Jon Siwek
c73bb8fdc4 Disable broker message forwarding by default
Still finding it to not be foolproof enough to enable generally for all
nodes in a cluster.  Specific/advanced use-cases may still consider
enabling, possibly just for specific nodes.
2018-09-06 18:32:22 -05:00
Daniel Thayer
d1ed09bddb Added a documentation comment for the Input::Event type 2018-09-05 22:27:15 -05:00
Jon Siwek
b99be6458b Merge remote-tracking branch 'origin/topic/johanna/weird-options'
* origin/topic/johanna/weird-options:
  Update test baselines (weird options)
  Weird settings: make constants into options.
  Permit weird sampling rate of 0.
2018-09-05 16:57:08 -05:00
Jon Siwek
f00e2167a7 BIT-1208: remove unused weirds from Weird::actions table 2018-09-05 15:13:38 -05:00
Johanna Amann
bec98b98f3 Weird settings: make constants into options.
The new weird settings are now all updateable during runtime.
2018-09-05 13:12:23 -07:00
Jon Siwek
4bbdd63054 BIT-1779: use BRO_LOG_SUFFIX env var in ascii log rotation function 2018-09-05 15:09:57 -05:00
Robin Sommer
6de436f3f6 Merge remote-tracking branch 'origin/topic/jsiwek/bit-1967'
* origin/topic/jsiwek/bit-1967:
  Fix a routing loop in control framework
  Add Broker::forward() function
  Enable implicit Broker message forwarding by default
  Remove Cluster::broadcast_topic
  Remove Intel Broker topics, re-use existing Cluster topics
  Remove "relay" family of Broker functions
2018-08-30 00:08:12 +00:00
Jon Siwek
1dcead93bf Add Broker::forward() function
This enables explicit forwarding of events matching a given topic
prefix.  Even if a receiving node has an event handler, it will not
be raised if the event was sent along a topic that matches a previous
call to Broker::forward().
2018-08-28 19:42:22 -05:00
Jon Siwek
850030822d Enable implicit Broker message forwarding by default 2018-08-28 16:50:41 -05:00
Jon Siwek
8db042a8c2 Remove Cluster::broadcast_topic
As enabling Broker forwarding would cause routing loops with messages
sent to such a topic (one subscribed to on all nodes).
2018-08-28 16:40:48 -05:00
Jon Siwek
2f1e81059b Remove Intel Broker topics, re-use existing Cluster topics
And update broker docs to reflect best-practice/convention for
declaring new topics.
2018-08-28 15:43:34 -05:00
Daniel Thayer
9bfc01b705 Convert more redef-able constants to runtime options 2018-08-27 19:38:47 -05:00
Jon Siwek
1a75ef2abd Remove "relay" family of Broker functions
Namely these are now removed:

    - Broker::relay
    - Broker::publish_and_relay
    - Cluster::relay_rr
    - Cluster::relay_hrw

The idea being that Broker may eventually implement the necessary
routing (plus load balancing) functionality.  For now, code that used
these should "manually" handle and re-publish events as needed.
2018-08-27 16:49:35 -05:00
Daniel Thayer
fd1a23ea66 Add a missing initializer to a runtime option 2018-08-24 17:03:02 -05:00
Daniel Thayer
01a899255e Convert more redef-able constants to runtime options 2018-08-24 16:05:44 -05:00
Jon Siwek
6595b21e2e Merge remote-tracking branch 'origin/topic/dnthayer/ticket1963'
* origin/topic/dnthayer/ticket1963:
  Remove unused redef-able constants
  Convert some redef-able constants to runtime options
2018-08-20 12:44:58 -05:00
Daniel Thayer
1a4629b0dc Merge remote-tracking branch 'origin/master' into topic/dnthayer/ticket1963 2018-08-17 14:11:47 -05:00
Daniel Thayer
a71ed6f781 Merge remote-tracking branch 'origin/master' into topic/dnthayer/doc-fixes-for-2.6 2018-08-17 11:34:16 -05:00
Jon Siwek
fc7d3cd981 Fix possible race in netcontrol acld/broker plugins
Best to subscribe before connecting
2018-08-17 10:31:31 -05:00
Daniel Thayer
c941c565a6 Replace references to libgeoip in the documentation
Replace references to the old libgeoip library with "libmaxminddb" or
"GeoIP support".
2018-08-16 15:45:58 -05:00
Jon Siwek
da9f91fc19 Add env. variables to override Broker listen/connect retry intervals
And use them to default retries to 1sec for all unit tests.
2018-08-16 12:16:03 -05:00
Daniel Thayer
9291fef6d2 Merge remote-tracking branch 'origin/master' into topic/dnthayer/doc-fixes-for-2.6 2018-08-15 15:27:44 -05:00
Daniel Thayer
dc0904a7f3 Convert some redef-able constants to runtime options 2018-08-15 10:17:14 -05:00
Jon Siwek
5821c16490 Fix SumStats::observe key normalization logic
The loop over Reducers in SumStats::observe performs a key normalization
and inadvertently modifies the key used for subsequent iterations.

Reported by Jim Mellander.
2018-08-13 17:53:26 -05:00
Jon Siwek
67524f26d5 Immediately apply broker subscriptions made during bro_init()
Otherwise that's begging for unit test failures due to races
2018-08-10 17:18:21 -05:00
Jon Siwek
083947af41 Update default broker threading configuration
Now defaults to a max of 4 threads typically indepedent of core
count (previously could go up to a hard cap of 8).  Also now allow
controlling this setting via BRO_BROKER_MAX_THREADS environment
variable.
2018-08-10 17:08:26 -05:00
Daniel Thayer
0e2dbfe7c3 Fix typos and formatting issues in config framework docs 2018-07-30 14:12:25 -05:00
Vern Paxson
88fd7510c6 reap the fruits of v += e 2018-07-26 12:51:36 -07:00
Robin Sommer
f4cd567f84 Merge remote-tracking branch 'origin/topic/johanna/config-cluster'
BIT-1958 #merged

* origin/topic/johanna/config-cluster:
  Add vector to read_config_cluster test.
  Fix special-case-bug for vectors in UnaryExpr.
  Config: another cluster test-case, this time reading in a file.
  Add sending of values to nodes that dropped out.
  Continue work on config framework clusterization.
  Teach Option::set to unwrap Broker::Data values
  Start clusterizing configuration framework.
2018-07-24 01:36:59 +00:00
Jon Siwek
385350a1f3 Make Broker congestion queue size tunable and increase default 2018-07-20 12:14:33 -05:00
Johanna Amann
c28f1ae0ce Add sending of values to nodes that dropped out.
The only node that cannot be recovered is the manager - and the manager
should just re-read its own configuration and be ok :)
2018-06-29 13:10:00 -07:00
Johanna Amann
5f07673e25 Merge remote-tracking branch 'origin/master' into topic/johanna/config-cluster 2018-06-29 08:30:29 -07:00
Daniel Thayer
ceefb6edaf Fix minor typos in broker reference documentation 2018-06-27 13:00:09 -05:00
Johanna Amann
d6990119db Continue work on config framework clusterization.
This does not currently work.
2018-06-27 10:47:17 -07:00
Johanna Amann
948bb4b9ec Merge remote-tracking branch 'origin/master' into topic/johanna/config-cluster 2018-06-22 10:43:09 -07:00
Jon Siwek
8ee92c70da Add ability for BroControl to skip cluster setup
It does this by setting the BROCTL_CHECK_CONFIG env. variable.
Related to BIT-1938.
2018-06-21 11:57:18 -05:00
Jon Siwek
e578c1c231 Add Broker::max_threads and Broker::max_sleep tuning options
The former replaces the pcap vs. live versions of the same tuning
option.  If a user does not change these, Bro makes some internal
decisions that may help avoid performance problems on systems with high
core counts: the number of CAF threads is capped at 8 and the maximum
sleep duration for under-utilized threads is increased to 64ms (CAF's
default is 10ms).
2018-06-14 12:57:23 -05:00
Jon Siwek
c9fe9a943c Add Broker::max_live_threads and Broker::max_pcap_threads tunables
These may be used to change the number of scheduler threads that the
underlying CAF library creates.  In pcap mode, it's currently hardcoded
to the minimal 4 threads due to potentially significant overhead in CAF.
2018-06-12 13:54:43 -05:00
Jon Siwek
1b4e0116f4 Allow BRO_DEFAULT_LISTEN_ADDRESS to control broker listen address
This environment variable is now set to listen only on IPv4 loopback
when running unit tests (instead of using the default INADDR_ANY).

This also moves some of the @loads out from init-bare.bro into a new
init-frameworks-and-bifs.bro in order to better support calling BIFs
(like `getenv`) from variable initializations in those particular
frameworks.
2018-06-01 15:38:11 -05:00
Johanna Amann
44dfcb7c6a Start clusterizing configuration framework.
This is not finished and currently does not work due Broker not liking
to serialize into any types.
2018-05-29 14:08:05 -07:00
Jon Siwek
fe478877c6 Change Intel framework to round-robin insertion events across proxies 2018-05-24 14:36:22 -05:00
Jon Siwek
45178f3051 Add a counter for number of alive nodes within a given cluster pool 2018-05-24 14:33:35 -05:00
Jon Siwek
95ea84e60e Fix how cluster framework tracks worker count 2018-05-24 14:32:45 -05:00
Robin Sommer
fe7e1ee7f0 Merge topic/actor-system throug a squashed commit. 2018-05-18 22:39:23 +00:00
Stephen Hosom
6611d28920 Fix for BIT-1927: notice/extend-email relocation
This is a fairly straightforward change. Previously, users had no
control over whether this script was loaded. By relocating it to
policy, users can now choose whether or not this is necessary
functionality without modifying core Bro scripts.
2018-05-10 09:13:52 -04:00