Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-10 10:38:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Improve documentation of Bro clusters

Browse source 
Renamed the bro cluster doc to better indicate its purpose (it provides
a high-level overview rather than detailed configuration instructions).
Moved the location of the bro cluster doc in the index so that it makes
more sense (it is an introductory section, not a section about using bro).

Added links in the quick start guide and the bro cluster doc so that
readers can more easily locate more detailed information on configuring
a bro cluster.

Addresses BIT-1160
This commit is contained in:
Daniel Thayer 2014-03-18 16:29:27 -05:00
parent bf6f21041c
commit 2e8d0945a4
3 changed files with 30 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

45
doc/cluster/index.rst
View file

@ -1,18 +1,19 @@
======================== ========================
Setting up a Bro Cluster Bro Cluster Architecture
======================== ========================
Intro
------
Bro is not multithreaded, so once the limitations of a single processor core Bro is not multithreaded, so once the limitations of a single processor core
are reached the only option currently is to spread the workload across many are reached the only option currently is to spread the workload across many
cores, or even many physical computers. The cluster deployment scenario for cores, or even many physical computers. The cluster deployment scenario for
Bro is the current solution to build these larger systems. The accompanying Bro is the current solution to build these larger systems. The tools and
tools and scripts provide the structure to easily manage many Bro processes scripts that accompany Bro provide the structure to easily manage many Bro
examining packets and doing correlation activities but acting as a singular, processes examining packets and doing correlation activities but acting as
cohesive entity. a singular, cohesive entity. This document describes the Bro cluster
architecture. For information on how to configure a Bro cluster,
see the documentation for
`BroControl <http://bro.org/sphinx/components/broctl/README.html>`_.
Architecture Architecture
--------------- ---------------
@ -41,11 +42,11 @@ messages and notices from the rest of the nodes in the cluster using the Bro
communications protocol. The result is a single log instead of many communications protocol. The result is a single log instead of many
discrete logs that you have to combine in some manner with post-processing. discrete logs that you have to combine in some manner with post-processing.
The manager also takes the opportunity to de-duplicate notices, and it has the The manager also takes the opportunity to de-duplicate notices, and it has the
ability to do so since it’s acting as the choke point for notices and how notices ability to do so since it's acting as the choke point for notices and how
might be processed into actions (e.g., emailing, paging, or blocking). notices might be processed into actions (e.g., emailing, paging, or blocking).
The manager process is started first by BroControl and it only opens its The manager process is started first by BroControl and it only opens its
designated port and waits for connections, it doesn’t initiate any designated port and waits for connections, it doesn't initiate any
connections to the rest of the cluster. Once the workers are started and connections to the rest of the cluster. Once the workers are started and
connect to the manager, logs and notices will start arriving to the manager connect to the manager, logs and notices will start arriving to the manager
process from the workers. process from the workers.
@ -58,12 +59,11 @@ the workers by alleviating the need for all of the workers to connect
directly to each other. directly to each other.
Examples of synchronized state from the scripts that ship with Bro include Examples of synchronized state from the scripts that ship with Bro include
the full list of “known” hosts and services (which are hosts or services the full list of "known" hosts and services (which are hosts or services
identified as performing full TCP handshakes) or an analyzed protocol has been identified as performing full TCP handshakes) or an analyzed protocol has been
found on the connection. If worker A detects host 1.2.3.4 as an active host, found on the connection. If worker A detects host 1.2.3.4 as an active host,
it would be beneficial for worker B to know that as well. So worker A shares it would be beneficial for worker B to know that as well. So worker A shares
that information as an insertion to a set that information as an insertion to a set which travels to the cluster's
<link to set documentation would be good here> which travels to the cluster’s
proxy and the proxy sends that same set insertion to worker B. The result proxy and the proxy sends that same set insertion to worker B. The result
is that worker A and worker B have shared knowledge about host and services is that worker A and worker B have shared knowledge about host and services
that are active on the network being monitored. that are active on the network being monitored.
@ -79,7 +79,7 @@ necessary for the number of workers they are serving. It is best to start
with a single proxy and add more if communication performance problems are with a single proxy and add more if communication performance problems are
found. found.
Bro processes acting as proxies don’t tend to be extremely hard on CPU Bro processes acting as proxies don't tend to be extremely hard on CPU
or memory and users frequently run proxy processes on the same physical or memory and users frequently run proxy processes on the same physical
host as the manager. host as the manager.
@ -106,7 +106,7 @@ dedicated to being workers with each one containing dual 6-core processors.
Once a flow-based load balancer is put into place this model is extremely Once a flow-based load balancer is put into place this model is extremely
easy to scale. It is recommended that you estimate the amount of easy to scale. It is recommended that you estimate the amount of
hardware you will need to fully analyze your traffic. If more is needed it’s hardware you will need to fully analyze your traffic. If more is needed it's
relatively easy to increase the size of the cluster in most cases. relatively easy to increase the size of the cluster in most cases.
Frontend Options Frontend Options
@ -147,14 +147,13 @@ On host flow balancing
PF_RING PF_RING
^^^^^^^ ^^^^^^^
The PF_RING software for Linux has a “clustering” feature which will do The PF_RING software for Linux has a "clustering" feature which will do
flow-based load balancing across a number of processes that are sniffing the flow-based load balancing across a number of processes that are sniffing the
same interface. This allows you to easily take advantage of multiple same interface. This allows you to easily take advantage of multiple
cores in a single physical host because Bro’s main event loop is single cores in a single physical host because Bro's main event loop is single
threaded and can’t natively utilize all of the cores. More information about threaded and can't natively utilize all of the cores. If you want to use
Bro with PF_RING can be found here: (someone want to write a quick Bro/PF_RING PF_RING, see the documentation on `how to configure Bro with PF_RING
tutorial to link to here? document installing kernel module, libpcap <http://bro.org/documentation/load-balancing.html>`_.
wrapper, building Bro with the --with-pcap configure option)
Netmap Netmap
^^^^^^ ^^^^^^
@ -167,7 +166,7 @@ Click! Software Router
^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^
Click! can be used for flow based load balancing with a simple configuration. Click! can be used for flow based load balancing with a simple configuration.
(link to an example for the config). This solution is not recommended on This solution is not recommended on
Linux due to Bro’s PF_RING support and only as a last resort on other Linux due to Bro's PF_RING support and only as a last resort on other
operating systems since it causes a lot of overhead due to context switching operating systems since it causes a lot of overhead due to context switching
back and forth between kernel and userland several times per packet. back and forth between kernel and userland several times per packet.

2
doc/index.rst
View file

@ -12,6 +12,7 @@ Introduction Section
:maxdepth: 2 :maxdepth: 2
intro/index.rst intro/index.rst
cluster/index.rst
install/index.rst install/index.rst
quickstart/index.rst quickstart/index.rst
@ -29,7 +30,6 @@ Using Bro Section
httpmonitor/index.rst httpmonitor/index.rst
broids/index.rst broids/index.rst
mimestats/index.rst mimestats/index.rst
cluster/index.rst
.. ..

9
doc/quickstart/index.rst
View file

@ -12,7 +12,9 @@ Quick Start Guide
Bro works on most modern, Unix-based systems and requires no custom Bro works on most modern, Unix-based systems and requires no custom
hardware. It can be downloaded in either pre-built binary package or hardware. It can be downloaded in either pre-built binary package or
source code forms. See :ref:`installing-bro` for instructions on how to source code forms. See :ref:`installing-bro` for instructions on how to
install Bro. Below, ``$PREFIX`` is used to reference the Bro install Bro.
In the examples below, ``$PREFIX`` is used to reference the Bro
installation root directory, which by default is ``/usr/local/bro/`` if installation root directory, which by default is ``/usr/local/bro/`` if
you install from source. you install from source.
@ -21,7 +23,10 @@ Managing Bro with BroControl
BroControl is an interactive shell for easily operating/managing Bro BroControl is an interactive shell for easily operating/managing Bro
installations on a single system or even across multiple systems in a installations on a single system or even across multiple systems in a
traffic-monitoring cluster. traffic-monitoring cluster. This section explains how to use BroControl
to manage a stand-alone Bro installation. For instructions on how to
configure a Bro cluster, see the documentation for `BroControl
<http://bro.org/sphinx/components/broctl/README.html>`_.
A Minimal Starting Configuration A Minimal Starting Configuration
-------------------------------- --------------------------------