Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-08 17:48:21 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Remove Intel Broker topics, re-use existing Cluster topics

Browse source 
And update broker docs to reflect best-practice/convention for
declaring new topics.
This commit is contained in:
Jon Siwek 2018-08-28 15:43:34 -05:00
parent 1a75ef2abd
commit 2f1e81059b
2 changed files with 19 additions and 29 deletions
Download patch file Download diff file Expand all files Collapse all files

14
doc/frameworks/broker.rst
View file

@ -237,7 +237,7 @@ follows certain conventions in choosing these topics to help avoid
conflicts and generally make them easier to remember. conflicts and generally make them easier to remember.
As a reminder of how topic subscriptions work, subscribers advertise As a reminder of how topic subscriptions work, subscribers advertise
interest in a topic **prefix** and then receive any messages publish by a interest in a topic **prefix** and then receive any messages published by a
peer to a topic name that starts with that prefix. E.g. Alice peer to a topic name that starts with that prefix. E.g. Alice
subscribes to the "alice/dogs" prefix, then would receive the following subscribes to the "alice/dogs" prefix, then would receive the following
message topics published by Bob: message topics published by Bob:
@ -263,6 +263,11 @@ scripts use will be along the lines of "bro/<namespace>/<specifics>"
with "<namespace>" being the script's module name (in all-undercase). with "<namespace>" being the script's module name (in all-undercase).
For example, you might expect an imaginary "Pretend" framework to For example, you might expect an imaginary "Pretend" framework to
publish/subscribe using topic names like "bro/pretend/my_cool_event". publish/subscribe using topic names like "bro/pretend/my_cool_event".
For scripts that use Broker as a means of cluster-aware analysis,
it's usually sufficient for them to make use of the topics declared
by the cluster framework. For scripts that are meant to establish
communication flows unrelated to Bro cluster, new topics are declared
(examples being the NetControl and Control frameworks).
For cluster operation, see :doc:`/scripts/base/frameworks/cluster/main.bro` For cluster operation, see :doc:`/scripts/base/frameworks/cluster/main.bro`
for a list of topics that are useful for steering published events to for a list of topics that are useful for steering published events to
@ -279,7 +284,12 @@ processes, logs get published to the topic indicated by
For those writing their own scripts which need new topic names, a For those writing their own scripts which need new topic names, a
suggestion would be to avoid prefixing any new topics/prefixes with suggestion would be to avoid prefixing any new topics/prefixes with
"bro/" as any changes in scripts shipping with Bro will use that prefix "bro/" as any changes in scripts shipping with Bro will use that prefix
and it's better to not risk unintended conflicts. and it's better to not risk unintended conflicts. Again, it's
often less confusing to just re-use existing topic names instead
of introducing new topic names. The typical use case is writing
a cluster-enabled script, which usually just needs to route events
based upon node classes, and that already has usable topics in the
cluster framework.
Connecting to Peers Connecting to Peers
------------------- -------------------

34
scripts/base/frameworks/intel/cluster.bro
View file

@ -6,21 +6,6 @@
module Intel; module Intel;
export {
## Broker topic for management of intel items. Sending insert_item and
## remove_item events, peers can manage intelligence data.
const item_topic = "bro/intel/items" &redef;
## Broker topic for management of intel indicators as stored on workers
## for matching. Sending insert_indicator and remove_indicator events,
## the back-end manages indicators.
const indicator_topic = "bro/intel/indicators" &redef;
## Broker topic for matching events, generated by workers and sent to
## the back-end for metadata enrichment and logging.
const match_topic = "bro/intel/match" &redef;
}
# Internal events for cluster data distribution. # Internal events for cluster data distribution.
global insert_item: event(item: Item); global insert_item: event(item: Item);
global insert_indicator: event(item: Item); global insert_indicator: event(item: Item);
@ -33,10 +18,7 @@ redef have_full_data = F;
@if ( Cluster::local_node_type() == Cluster::MANAGER ) @if ( Cluster::local_node_type() == Cluster::MANAGER )
event bro_init() event bro_init()
{ {
Broker::subscribe(item_topic); Broker::auto_publish(Cluster::worker_topic, remove_indicator);
Broker::subscribe(match_topic);
Broker::auto_publish(indicator_topic, remove_indicator);
} }
# Handling of new worker nodes. # Handling of new worker nodes.
@ -54,12 +36,12 @@ event Cluster::node_up(name: string, id: string)
# has to be distributed. # has to be distributed.
event Intel::new_item(item: Item) &priority=5 event Intel::new_item(item: Item) &priority=5
{ {
local pt = Cluster::rr_topic(Cluster::proxy_pool, indicator_topic); local pt = Cluster::rr_topic(Cluster::proxy_pool, "intel_insert_rr_key");
if ( pt == "" ) if ( pt == "" )
# No proxies alive, publish to all workers ourself instead of # No proxies alive, publish to all workers ourself instead of
# relaying via a proxy. # relaying via a proxy.
pt = indicator_topic; pt = Cluster::worker_topic;
Broker::publish(pt, Intel::insert_indicator, item); Broker::publish(pt, Intel::insert_indicator, item);
} }
@ -87,17 +69,15 @@ event Intel::match_remote(s: Seen) &priority=5
@if ( Cluster::local_node_type() == Cluster::WORKER ) @if ( Cluster::local_node_type() == Cluster::WORKER )
event bro_init() event bro_init()
{ {
Broker::subscribe(indicator_topic); Broker::auto_publish(Cluster::manager_topic, match_remote);
Broker::auto_publish(Cluster::manager_topic, remove_item);
Broker::auto_publish(match_topic, match_remote);
Broker::auto_publish(item_topic, remove_item);
} }
# On a worker, the new_item event requires to trigger the insertion # On a worker, the new_item event requires to trigger the insertion
# on the manager to update the back-end data store. # on the manager to update the back-end data store.
event Intel::new_item(item: Intel::Item) &priority=5 event Intel::new_item(item: Intel::Item) &priority=5
{ {
Broker::publish(item_topic, Intel::insert_item, item); Broker::publish(Cluster::manager_topic, Intel::insert_item, item);
} }
# Handling of new indicators published by the manager. # Handling of new indicators published by the manager.
@ -111,7 +91,7 @@ event Intel::insert_indicator(item: Intel::Item) &priority=5
event Intel::insert_indicator(item: Intel::Item) &priority=5 event Intel::insert_indicator(item: Intel::Item) &priority=5
{ {
# Just forwarding from manager to workers. # Just forwarding from manager to workers.
Broker::publish(indicator_topic, Intel::insert_indicator, item); Broker::publish(Cluster::worker_topic, Intel::insert_indicator, item);
} }
@endif @endif