mirror of
https://github.com/zeek/zeek.git
synced 2025-10-02 22:58:20 +00:00
0db028ea91
Also removed some redundant info about setting up a broctl cron job, which is already explained in detail in the broctl documentation.
255 lines
9.2 KiB
ReStructuredText
255 lines
9.2 KiB
ReStructuredText
|
|
.. _configuration:
|
|
|
|
=====================
|
|
Cluster Configuration
|
|
=====================
|
|
|
|
.. contents::
|
|
|
|
A *Bro Cluster* is a set of systems jointly analyzing the traffic of
|
|
a network link in a coordinated fashion. You can operate such a setup from
|
|
a central manager system easily using BroControl because BroControl
|
|
hides much of the complexity of the multi-machine installation.
|
|
|
|
This section gives examples of how to setup common cluster configurations
|
|
using BroControl. For a full reference on BroControl, see the
|
|
:doc:`BroControl <../components/broctl/README>` documentation.
|
|
|
|
|
|
Preparing to Setup a Cluster
|
|
============================
|
|
|
|
In this document we refer to the user account used to set up the cluster
|
|
as the "Bro user". When setting up a cluster the Bro user must be set up
|
|
on all hosts, and this user must have ssh access from the manager to all
|
|
machines in the cluster, and it must work without being prompted for a
|
|
password/passphrase (for example, using ssh public key authentication).
|
|
Also, on the worker nodes this user must have access to the target
|
|
network interface in promiscuous mode.
|
|
|
|
Additional storage must be available on all hosts under the same path,
|
|
which we will call the cluster's prefix path. We refer to this directory
|
|
as ``<prefix>``. If you build Bro from source, then ``<prefix>`` is
|
|
the directory specified with the ``--prefix`` configure option,
|
|
or ``/usr/local/bro`` by default. The Bro user must be able to either
|
|
create this directory or, where it already exists, must have write
|
|
permission inside this directory on all hosts.
|
|
|
|
When trying to decide how to configure the Bro nodes, keep in mind that
|
|
there can be multiple Bro instances running on the same host. For example,
|
|
it's possible to run a proxy and the manager on the same host. However, it is
|
|
recommended to run workers on a different machine than the manager because
|
|
workers can consume a lot of CPU resources. The maximum recommended
|
|
number of workers to run on a machine should be one or two less than
|
|
the number of CPU cores available on that machine. Using a load-balancing
|
|
method (such as PF_RING) along with CPU pinning can decrease the load on
|
|
the worker machines. Also, in order to reduce the load on the manager
|
|
process, it is recommended to have a logger in your configuration. If a
|
|
logger is defined in your cluster configuration, then it will receive logs
|
|
instead of the manager process.
|
|
|
|
|
|
Basic Cluster Configuration
|
|
===========================
|
|
|
|
With all prerequisites in place, perform the following steps to setup
|
|
a Bro cluster (do this as the Bro user on the manager host only):
|
|
|
|
- Edit the BroControl configuration file, ``<prefix>/etc/broctl.cfg``,
|
|
and change the value of any BroControl options to be more suitable for
|
|
your environment. You will most likely want to change the value of
|
|
the ``MailTo`` and ``LogRotationInterval`` options. A complete
|
|
reference of all BroControl options can be found in the
|
|
:doc:`BroControl <../components/broctl/README>` documentation.
|
|
|
|
- Edit the BroControl node configuration file, ``<prefix>/etc/node.cfg``
|
|
to define where logger, manager, proxies, and workers are to run. For a
|
|
cluster configuration, you must comment-out (or remove) the standalone node
|
|
in that file, and either uncomment or add node entries for each node
|
|
in your cluster (logger, manager, proxy, and workers). For example, if you
|
|
wanted to run five Bro nodes (two workers, one proxy, a logger, and a
|
|
manager) on a cluster consisting of three machines, your cluster
|
|
configuration would look like this::
|
|
|
|
[logger]
|
|
type=logger
|
|
host=10.0.0.10
|
|
|
|
[manager]
|
|
type=manager
|
|
host=10.0.0.10
|
|
|
|
[proxy-1]
|
|
type=proxy
|
|
host=10.0.0.10
|
|
|
|
[worker-1]
|
|
type=worker
|
|
host=10.0.0.11
|
|
interface=eth0
|
|
|
|
[worker-2]
|
|
type=worker
|
|
host=10.0.0.12
|
|
interface=eth0
|
|
|
|
For a complete reference of all options that are allowed in the ``node.cfg``
|
|
file, see the :doc:`BroControl <../components/broctl/README>` documentation.
|
|
|
|
- Edit the network configuration file ``<prefix>/etc/networks.cfg``. This
|
|
file lists all of the networks which the cluster should consider as local
|
|
to the monitored environment.
|
|
|
|
- Install Bro on all machines in the cluster using BroControl::
|
|
|
|
> broctl install
|
|
|
|
- See the :doc:`BroControl <../components/broctl/README>` documentation
|
|
for information on setting up a cron job on the manager host that can
|
|
monitor the cluster.
|
|
|
|
|
|
PF_RING Cluster Configuration
|
|
=============================
|
|
|
|
`PF_RING <http://www.ntop.org/products/pf_ring/>`_ allows speeding up the
|
|
packet capture process by installing a new type of socket in Linux systems.
|
|
It supports 10Gbit hardware packet filtering using standard network adapters,
|
|
and user-space DNA (Direct NIC Access) for fast packet capture/transmission.
|
|
|
|
Installing PF_RING
|
|
^^^^^^^^^^^^^^^^^^
|
|
|
|
1. Download and install PF_RING for your system following the instructions
|
|
`here <http://www.ntop.org/get-started/download/#PF_RING>`_. The following
|
|
commands will install the PF_RING libraries and kernel module (replace
|
|
the version number 5.6.2 in this example with the version that you
|
|
downloaded)::
|
|
|
|
cd /usr/src
|
|
tar xvzf PF_RING-5.6.2.tar.gz
|
|
cd PF_RING-5.6.2/userland/lib
|
|
./configure --prefix=/opt/pfring
|
|
make install
|
|
|
|
cd ../libpcap
|
|
./configure --prefix=/opt/pfring
|
|
make install
|
|
|
|
cd ../tcpdump-4.1.1
|
|
./configure --prefix=/opt/pfring
|
|
make install
|
|
|
|
cd ../../kernel
|
|
make install
|
|
|
|
modprobe pf_ring enable_tx_capture=0 min_num_slots=32768
|
|
|
|
Refer to the documentation for your Linux distribution on how to load the
|
|
pf_ring module at boot time. You will need to install the PF_RING
|
|
library files and kernel module on all of the workers in your cluster.
|
|
|
|
2. Download the Bro source code.
|
|
|
|
3. Configure and install Bro using the following commands::
|
|
|
|
./configure --with-pcap=/opt/pfring
|
|
make
|
|
make install
|
|
|
|
4. Make sure Bro is correctly linked to the PF_RING libpcap libraries::
|
|
|
|
ldd /usr/local/bro/bin/bro | grep pcap
|
|
libpcap.so.1 => /opt/pfring/lib/libpcap.so.1 (0x00007fa6d7d24000)
|
|
|
|
5. Configure BroControl to use PF_RING (explained below).
|
|
|
|
6. Run "broctl install" on the manager. This command will install Bro and
|
|
required scripts to all machines in your cluster.
|
|
|
|
Using PF_RING
|
|
^^^^^^^^^^^^^
|
|
|
|
In order to use PF_RING, you need to specify the correct configuration
|
|
options for your worker nodes in BroControl's node configuration file.
|
|
Edit the ``node.cfg`` file and specify ``lb_method=pf_ring`` for each of
|
|
your worker nodes. Next, use the ``lb_procs`` node option to specify how
|
|
many Bro processes you'd like that worker node to run, and optionally pin
|
|
those processes to certain CPU cores with the ``pin_cpus`` option (CPU
|
|
numbering starts at zero). The correct ``pin_cpus`` setting to use is
|
|
dependent on your CPU architecture (Intel and AMD systems enumerate
|
|
processors in different ways). Using the wrong ``pin_cpus`` setting
|
|
can cause poor performance. Here is what a worker node entry should
|
|
look like when using PF_RING and CPU pinning::
|
|
|
|
[worker-1]
|
|
type=worker
|
|
host=10.0.0.50
|
|
interface=eth0
|
|
lb_method=pf_ring
|
|
lb_procs=10
|
|
pin_cpus=2,3,4,5,6,7,8,9,10,11
|
|
|
|
|
|
Using PF_RING+DNA with symmetric RSS
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
You must have a PF_RING+DNA license in order to do this. You can sniff
|
|
each packet only once.
|
|
|
|
1. Load the DNA NIC driver (i.e. ixgbe) on each worker host.
|
|
|
|
2. Run "ethtool -L dna0 combined 10" (this will establish 10 RSS queues
|
|
on your NIC) on each worker host. You must make sure that you set the
|
|
number of RSS queues to the same as the number you specify for the
|
|
lb_procs option in the node.cfg file.
|
|
|
|
3. On the manager, configure your worker(s) in node.cfg::
|
|
|
|
[worker-1]
|
|
type=worker
|
|
host=10.0.0.50
|
|
interface=dna0
|
|
lb_method=pf_ring
|
|
lb_procs=10
|
|
|
|
|
|
Using PF_RING+DNA with pfdnacluster_master
|
|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
|
|
|
You must have a PF_RING+DNA license and a libzero license in order to do
|
|
this. You can load balance between multiple applications and sniff the
|
|
same packets multiple times with different tools.
|
|
|
|
1. Load the DNA NIC driver (i.e. ixgbe) on each worker host.
|
|
|
|
2. Run "ethtool -L dna0 1" (this will establish 1 RSS queues on your NIC)
|
|
on each worker host.
|
|
|
|
3. Run the pfdnacluster_master command on each worker host. For example::
|
|
|
|
pfdnacluster_master -c 21 -i dna0 -n 10
|
|
|
|
Make sure that your cluster ID (21 in this example) matches the interface
|
|
name you specify in the node.cfg file. Also make sure that the number
|
|
of processes you're balancing across (10 in this example) matches
|
|
the lb_procs option in the node.cfg file.
|
|
|
|
4. If you are load balancing to other processes, you can use the
|
|
pfringfirstappinstance variable in broctl.cfg to set the first
|
|
application instance that Bro should use. For example, if you are running
|
|
pfdnacluster_master with "-n 10,4" you would set
|
|
pfringfirstappinstance=4. Unfortunately that's still a global setting
|
|
in broctl.cfg at the moment but we may change that to something you can
|
|
set in node.cfg eventually.
|
|
|
|
5. On the manager, configure your worker(s) in node.cfg::
|
|
|
|
[worker-1]
|
|
type=worker
|
|
host=10.0.0.50
|
|
interface=dnacluster:21
|
|
lb_method=pf_ring
|
|
lb_procs=10
|
|
|