Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-06 16:48:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Merge remote-tracking branch 'origin/master' into topic/bernhard/sqlite

Browse source
This commit is contained in:
Bernhard Amann 2012-11-26 20:46:27 -08:00
commit 2d7ffd8269
343 changed files with 186777 additions and 1133 deletions
Download patch file Download diff file Expand all files Collapse all files

197
CHANGES
View file

@ -1,4 +1,201 @@
2.1-178 | 2012-11-23 19:35:32 -0800
* The ASCII writer now supports a new filter config option
"only_single_header_row" that turns the output into CSV format
when set to "T". (Carsten Langer)
* Add new function flavor called a "hook". This new flavor of
function behaves like a "synchronous event". See
doc/scripts/builtins.rst more details on usage. (Jon Siwek)
* Improve auto-generated enum documentation. The names of enum types
are tracked so that variables holding a value of a given enum type
can generate a reference to it instead of just listing the type as
a generic "enum". (Jon Siwek)
2.1-171 | 2012-11-23 18:24:15 -0800
* Fix ambiguity between composite table index and record ctor
expressions. If a table type is "global t = table[conn_id, bool]
of count", then checking membership like "[c$id, is_orig] in t"
now works. Addresses #80. (Jon Siwek)
2.1-169 | 2012-11-23 18:21:32 -0800
* Fix some warnings from sphinx when building docs. (Jon Siwek)
2.1-167 | 2012-11-14 13:19:17 -0800
* Add a new BIF "bytestring_to_double" for converting from a binary
representation of a double. Addresses #908. (Carsten Langer/Daniel
Thayer)
2.1-162 | 2012-11-13 17:29:00 -0800
* Fix modbus register array parsing. (Jon Siwek)
* Adjustments to modbus test cases. (Jon Siwek)
2.1-157 | 2012-11-08 16:22:00 -0800
* Fix for lookup_hostname BIF. (Jon Siwek)
* Fix for modbus test portability. (Robin Sommer)
2.1-152 | 2012-11-05 16:52:34 -0800
* Initial version of a completely reworked intelligence framework.
See doc/intel.rst for more information. (Seth Hall)
* Experimental Modbus analyzer. See policy/protocols/modbus/* for
example policies. (Dina Hadziosmanovic, Seth Hall)
2.1-112 | 2012-11-05 13:58:20 -0800
* New base script for detecting cases of checksum offloading.
Reporter messages will now tell if one has bad checksums. (Seth
Hall)
* Clarifying ownership rules for BroString constructors. (Robin
Sommer)
2.1-109 | 2012-11-05 13:39:34 -0800
* Add detection rate threshold for MHR. (Vlad Grigorescu)
* lookup_hostname_txt fixes. (Vlad Grigorescu)
2.1-104 | 2012-11-01 10:37:50 -0700
* A new built-in function lookup_hostname_txt() provides support for
DNS TXT queries. (Vlad Grigorescu)
2.1-101 | 2012-10-31 14:30:26 -0700
* Documentation reorg: The install info has been consolidated into a
single document (INSTALL), the upgrade info has been moved from
the FAQ to a section in the install doc, and the "upgrading from
1.5 to 2.0" document has been updated (and renamed) to also
include 2.0 to 2.1 upgrade info. (Daniel Thayer)
2.1-96 | 2012-10-31 14:23:50 -0700
* Renaming option defining the frequency of alarm summary mails to
'Logging::default_alarm_mail_interval'. (Daniel Thayer)
2.1-91 | 2012-10-24 16:04:47 -0700
* Adding PPPoE support to Bro. (Seth Hall)
2.1-87 | 2012-10-24 15:40:06 -0700
* Adding missing &redef for some TCP options. Addresses #905, #906,
#907. (Carsten Langer)
2.1-86 | 2012-10-24 15:37:11 -0700
* Add parsing rules for IPv4/IPv6 subnet literal constants.
Addresses #888. (Jon Siwek)
2.1-84 | 2012-10-19 15:12:56 -0700
* Added a BiF strptime() to wrap the corresponding C function. (Seth
Hall)
2.1-82 | 2012-10-19 15:05:40 -0700
* Add IPv6 support to signature header conditions. (Jon Siwek)
- "src-ip" and "dst-ip" conditions can now use IPv6 addresses/subnets.
They must be written in colon-hexadecimal representation and enclosed
in square brackets (e.g. [fe80::1]). Addresses #774.
- "icmp6" is now a valid protocol for use with "ip-proto" and "header"
conditions. This allows signatures to be written that can match
against ICMPv6 payloads. Addresses #880.
- "ip6" is now a valid protocol for use with the "header" condition.
(also the "ip-proto" condition, but it results in a no-op in that
case since signatures apply only to the inner-most IP packet when
packets are tunneled). This allows signatures to match specifically
against IPv6 packets (whereas "ip" only matches against IPv4 packets).
- "ip-proto" conditions can now match against IPv6 packets. Before,
IPv6 packets were just silently ignored which meant DPD based on
signatures did not function for IPv6 -- protocol analyzers would only
get attached to a connection over IPv6 based on the well-known ports
set in the "dpd_config" table.
2.1-80 | 2012-10-19 14:48:42 -0700
* Change how "gridftp" gets added to service field of connection
records. In addition to checking for a finished SSL handshake over
an FTP connection, it now also requires that the SSL handshake
occurs after the FTP client requested AUTH GSSAPI, more
specifically identifying the characteristics of GridFTP control
channels. Addresses #891. (Jon Siwek)
* Allow faster rebuilds in certain cases. Previously, when
rebuilding with a different "--prefix" or "--scriptdir", all Bro
source files were recompiled. With this change, only util.cc is
recompiled. (Daniel Thayer)
2.1-76 | 2012-10-12 10:32:39 -0700
* Add support for recognizing GridFTP connections as an extension to
the standard FTP analyzer. (Jon Siwek)
This is enabled by default and includes:
- An analyzer for GSI mechanism of GSSAPI FTP AUTH method. GSI
authentication involves an encoded TLS/SSL handshake over the
FTP control session. For FTP sessions that attempt GSI
authentication, the *service* field of the connection log will
include "gridftp" (as well as also "ftp" and "ssl").
- Add an example of a GridFTP data channel detection script. It
relies on the heuristics of GridFTP data channels commonly
default to SSL mutual authentication with a NULL bulk cipher
and that they usually transfer large datasets (default
threshold of script is 1 GB). The script also defaults to
skip_further_processing() after detection to try to save
cycles analyzing the large, benign connection.
For identified GridFTP data channels, the *services* fields of
the connection log will include "gridftp-data".
* Add *client_subject* and *client_issuer_subject* as &log'd fields
to SSL::Info record. Also add *client_cert* and
*client_cert_chain* fields to track client cert chain. (Jon Siwek)
* Add a script in base/protocols/conn/polling that generalizes the
process of polling a connection for interesting features. The
GridFTP data channel detection script depends on it to monitor
bytes transferred. (Jon Siwek)
2.1-68 | 2012-10-12 09:46:41 -0700
* Rename the Input Framework's update_finished event to end_of_data.
It will now not only fire after table-reads have been completed,
but also after the last event of a whole-file-read (or
whole-db-read, etc.). (Bernhard Amann)
* Fix for DNS log problem when a DNS response is seen with 0 RRs.
(Seth Hall)
2.1-64 | 2012-10-12 09:36:41 -0700
* Teach --disable-dataseries/--disable-elasticsearch to ./configure.
Addresses #877. (Jon Siwek)
* Add --with-curl option to ./configure. Addresses #877. (Jon Siwek)
2.1-61 | 2012-10-12 09:32:48 -0700
* Fix bug in the input framework: the config table did not work.
(Bernhard Amann)
2.1-58 | 2012-10-08 10:10:09 -0700
* Fix a problem with non-manager cluster nodes applying

11
CMakeLists.txt
View file

@ -120,7 +120,8 @@ find_package(Lintel)
find_package(DataSeries)
find_package(LibXML2)
if (LINTEL_FOUND AND DATASERIES_FOUND AND LIBXML2_FOUND)
if (NOT DISABLE_DATASERIES AND
LINTEL_FOUND AND DATASERIES_FOUND AND LIBXML2_FOUND)
set(USE_DATASERIES true)
include_directories(BEFORE ${Lintel_INCLUDE_DIR})
include_directories(BEFORE ${DataSeries_INCLUDE_DIR})
@ -132,13 +133,13 @@ endif()
set(USE_ELASTICSEARCH false)
set(USE_CURL false)
find_package(CURL)
find_package(LibCURL)
if (CURL_FOUND)
if (NOT DISABLE_ELASTICSEARCH AND LIBCURL_FOUND)
set(USE_ELASTICSEARCH true)
set(USE_CURL true)
include_directories(BEFORE ${CURL_INCLUDE_DIR})
list(APPEND OPTLIBS ${CURL_LIBRARIES})
include_directories(BEFORE ${LibCURL_INCLUDE_DIR})
list(APPEND OPTLIBS ${LibCURL_LIBRARIES})
endif()
if (ENABLE_PERFTOOLS_DEBUG)

262
INSTALL
View file

@ -1,35 +1,55 @@
.. _CMake: http://www.cmake.org
.. _SWIG: http://www.swig.org
.. _Xcode: https://developer.apple.com/xcode/
.. _MacPorts: http://www.macports.org
.. _Fink: http://www.finkproject.org
.. _Homebrew: http://mxcl.github.com/homebrew
.. _bro downloads page: http://bro-ids.org/download/index.html
==============
Installing Bro
==============
Bro can be downloaded in either pre-built binary package or
source code forms.
Prerequisites
=============
Bro requires the following libraries and tools to be installed
before you begin:
* CMake 2.6.3 or greater http://www.cmake.org
* Libpcap http://www.tcpdump.org
* Perl (used only during the Bro build process)
* OpenSSL libraries http://www.openssl.org
* Libpcap headers and libraries http://www.tcpdump.org
* OpenSSL headers and libraries http://www.openssl.org
* BIND8 headers and libraries
* BIND8 library
* Libmagic
* Libz
* Bash (for BroControl)
To build Bro from source, the following additional dependencies are required:
* CMake 2.6.3 or greater http://www.cmake.org
* SWIG http://www.swig.org
* Bison (GNU Parser Generator)
* Flex (Fast Lexical Analyzer)
* Bash (for BroControl)
* Libpcap headers http://www.tcpdump.org
* OpenSSL headers http://www.openssl.org
* libmagic headers
* zlib headers
* Perl
Bro can make use of some optional libraries and tools if they are found at
build time:
@ -45,29 +65,161 @@ build time:
* Ruby executable, library, and headers (for Broccoli Ruby bindings)
Installation
============
Installing From Pre-Built Binary Release Packages
=================================================
To build and install into ``/usr/local/bro``::
See the `bro downloads page`_ for currently supported/targeted platforms.
* RPM
.. console::
sudo yum localinstall Bro-*.rpm
* DEB
.. console::
sudo gdebi Bro-*.deb
* MacOS Disk Image with Installer
Just open the ``Bro-*.dmg`` and then run the ``.pkg`` installer.
Everything installed by the package will go into ``/opt/bro``.
The primary install prefix for binary packages is ``/opt/bro``.
Non-MacOS packages that include BroControl also put variable/runtime
data (e.g. Bro logs) in ``/var/opt/bro``.
Installing From Source
======================
Required Dependencies
~~~~~~~~~~~~~~~~~~~~~
The following dependencies are required to build Bro:
* RPM/RedHat-based Linux:
.. console::
sudo yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel file-devel
* DEB/Debian-based Linux:
.. console::
sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev libmagic-dev
* FreeBSD
Most required dependencies should come with a minimal FreeBSD install
except for the following.
.. console::
sudo pkg_add -r bash cmake swig bison python
Note that ``bash`` needs to be in ``PATH``, which by default it is
not. The FreeBSD package installs the binary into
``/usr/local/bin``.
* Mac OS X
Compiling source code on Macs requires first downloading Xcode_,
then going through its "Preferences..." -> "Downloads" menus to
install the "Command Line Tools" component.
Lion (10.7) and Mountain Lion (10.8) come with all required
dependencies except for CMake_, SWIG_, and ``libmagic``.
Distributions of these dependencies can likely be obtained from your
preferred Mac OS X package management system (e.g. MacPorts_, Fink_,
or Homebrew_).
Specifically for MacPorts, the ``swig``, ``swig-ruby``, ``swig-python``
and ``file`` packages provide the required dependencies.
Optional Dependencies
~~~~~~~~~~~~~~~~~~~~~
Bro can use libGeoIP for geo-locating IP addresses, and sendmail for
sending emails.
* RedHat Enterprise Linux:
.. console::
sudo yum install geoip-devel sendmail
* CentOS Linux:
.. console::
sudo yum install GeoIP-devel sendmail
* DEB/Debian-based Linux:
.. console::
sudo apt-get install libgeoip-dev sendmail
* Ports-based FreeBSD
.. console::
sudo pkg_add -r GeoIP
sendmail is typically already available.
* Mac OS X
Vanilla OS X installations don't ship with libGeoIP, but
if installed from your preferred package management system (e.g. MacPorts,
Fink, or Homebrew), they should be automatically detected and Bro will
compile against them.
Additional steps may be needed to :doc:`get the right GeoIP database <geoip>`.
Compiling Bro Source Code
~~~~~~~~~~~~~~~~~~~~~~~~~
Bro releases are bundled into source packages for convenience and
available from the `bro downloads page`_.
Alternatively, the latest Bro development version can be obtained through git
repositories hosted at `git.bro-ids.org <http://git.bro-ids.org>`_. See
our `git development documentation
<http://bro-ids.org/development/process.html>`_ for comprehensive
information on Bro's use of git revision control, but the short story
for downloading the full source code experience for Bro via git is:
.. console::
git clone --recursive git://git.bro-ids.org/bro
.. note:: If you choose to clone the ``bro`` repository non-recursively for
a "minimal Bro experience", be aware that compiling it depends on
BinPAC, which has its own ``binpac`` repository. Either install it
first or initialize/update the cloned ``bro`` repository's
``aux/binpac`` submodule.
The typical way to build and install from source is (for more options,
run ``./configure --help``):
.. console::
./configure
make
make install
This will first build Bro in a directory inside the distribution
called ``build/``, using default build options. It then installs all
required files into ``/usr/local/bro``, including the Bro binary in
``/usr/local/bro/bin/bro``.
You can specify a different installation directory with::
./configure --prefix=<dir>
Note that ``/usr`` and ``/opt/bro`` are the standard prefixes for
binary Bro packages to be installed, so those are typically not good
choices unless you are creating such a package.
Run ``./configure --help`` for more options.
The default installation path is ``/usr/local/bro``, which would typically
require root privileges when doing the ``make install``. A different
installation path can be chosen by specifying the ``--prefix`` option.
Note that ``/usr`` and ``/opt/bro`` are the
standard prefixes for binary Bro packages to be installed, so those are
typically not good choices unless you are creating such a package.
Depending on the Bro package you downloaded, there may be auxiliary
tools and libraries available in the ``aux/`` directory. Some of them
@ -81,6 +233,66 @@ OpenBSD users, please see our FAQ at
http://www.bro-ids.org/documentation/faq.html if you are having
problems installing Bro.
Upgrading From a Previous Version of Bro
========================================
If you're doing an upgrade install (rather than a fresh install),
there's two suggested approaches: either install Bro using the same
installation prefix directory as before, or pick a new prefix and copy
local customizations over.
Re-Use Previous Install Prefix
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you choose to configure and install Bro with the same prefix
directory as before, local customization and configuration to files in
``$prefix/share/bro/site`` and ``$prefix/etc`` won't be overwritten
(``$prefix`` indicating the root of where Bro was installed). Also, logs
generated at run-time won't be touched by the upgrade. (But making
a backup of local changes before upgrading is still recommended.)
After upgrading, remember to check ``$prefix/share/bro/site`` and
``$prefix/etc`` for ``.example`` files, which indicate the
distribution's version of the file differs from the local one, which may
include local changes. Review the differences, and make adjustments
as necessary (for differences that aren't the result of a local change,
use the new version's).
Pick a New Install prefix
~~~~~~~~~~~~~~~~~~~~~~~~~
If you want to install the newer version in a different prefix
directory than before, you can just copy local customization and
configuration files from ``$prefix/share/bro/site`` and ``$prefix/etc``
to the new location (``$prefix`` indicating the root of where Bro was
originally installed). Make sure to review the files for difference
before copying and make adjustments as necessary (for differences that
aren't the result of a local change, use the new version's). Of
particular note, the copied version of ``$prefix/etc/broctl.cfg`` is
likely to need changes to the ``SpoolDir`` and ``LogDir`` settings.
Configure the Run-Time Environment
==================================
Just remember that you may need to adjust your ``PATH`` environment variable
according to the platform/shell/package you're using. For example:
Bourne-Shell Syntax:
.. console::
export PATH=/usr/local/bro/bin:$PATH
C-Shell Syntax:
.. console::
setenv PATH /usr/local/bro/bin:$PATH
Or substitute ``/opt/bro/bin`` instead if you installed from a binary package.
Running Bro
===========

19
NEWS
View file

@ -13,7 +13,14 @@ Bro 2.2
New Functionality
~~~~~~~~~~~~~~~~~
- TODO: Update.
- GridFTP support. TODO: Extend.
- ssl.log now also records the subject client and issuer certificates.
- Hooks: TODO: Briefly summarize the documention from
doc/scripts/builtins.rst here.
- The ASCII writer can now output CSV files on a per filter basis.
Changed Functionality
~~~~~~~~~~~~~~~~~~~~~
@ -32,6 +39,16 @@ Changed Functionality
- "this" is no longer a reserved keyword.
- The Input Framework's update_finished event has been renamed to
end_of_data. It will now not only fire after table-reads have been
completed, but also after the last event of a whole-file-read (or
whole-db-read, etc.).
- Renamed the option defining the frequency of alarm summary mails to
'Logging::default_alarm_mail_interval'. When using BroControl, the
value can now be set with the new broctl.cfg option
"MailAlarmsInterval".
Bro 2.1
-------

2
VERSION
View file

@ -1 +1 @@
2.1-58
2.1-179

2
aux/binpac

@ -1 +1 @@
Subproject commit a93ef1373512c661ffcd0d0a61bd19b96667e0d5
Subproject commit 2fd9086c9dc0e76f6ff1ae04a60cbbce60507aab

2
aux/bro-aux

@ -1 +1 @@
Subproject commit 6748ec3a96d582a977cd9114ef19c76fe75c57ff
Subproject commit bea556198b69d30d64c0cf1b594e6de71176df6f

2
aux/broccoli

@ -1 +1 @@
Subproject commit ebfa4de45a839e58aec200e7e4bad33eaab4f1ed
Subproject commit a8846fc5b004ffe4e3d00e826d0077ba19518192

2
aux/broctl

@ -1 +1 @@
Subproject commit b0e3c0d84643878c135dcb8a9774ed78147dd648
Subproject commit 834131cd0ec0f63cce9de818726fe6167dedbf34

2
aux/btest

@ -1 +1 @@
Subproject commit 44a43e62452302277f88e8fac08d1f979dc53f98
Subproject commit d83e10c5f76cbfdf81c843575351fbc7b544fc93

2
cmake

@ -1 +1 @@
Subproject commit 125f9a5fa851381d0350efa41a4d14f27be263a2
Subproject commit 14537f56d66b18ab9d5024f798caf4d1f356fc67

12
configure vendored
View file

@ -38,6 +38,8 @@ Usage: $0 [OPTION]... [VAR=VALUE]...
--disable-perftools don't try to build with Google Perftools
--disable-python don't try to build python bindings for broccoli
--disable-ruby don't try to build ruby bindings for broccoli
--disable-dataseries don't use the optional DataSeries log writer
--disable-elasticsearch don't use the optional ElasticSearch log writer
Required Packages in Non-Standard Locations:
--with-openssl=PATH path to OpenSSL install root
@ -61,6 +63,7 @@ Usage: $0 [OPTION]... [VAR=VALUE]...
--with-swig=PATH path to SWIG executable
--with-dataseries=PATH path to DataSeries and Lintel libraries
--with-xml2=PATH path to libxml2 installation (for DataSeries)
--with-curl=PATH path to libcurl install root (for ElasticSearch)
Packaging Options (for developers):
--binary-package toggle special logic for binary packaging
@ -174,6 +177,12 @@ while [ $# -ne 0 ]; do
--disable-ruby)
append_cache_entry DISABLE_RUBY_BINDINGS BOOL true
;;
--disable-dataseries)
append_cache_entry DISABLE_DATASERIES BOOL true
;;
--disable-elasticsearch)
append_cache_entry DISABLE_ELASTICSEARCH BOOL true
;;
--with-openssl=*)
append_cache_entry OpenSSL_ROOT_DIR PATH $optarg
;;
@ -234,6 +243,9 @@ while [ $# -ne 0 ]; do
--with-xml2=*)
append_cache_entry LibXML2_ROOT_DIR PATH $optarg
;;
--with-curl=*)
append_cache_entry LibCURL_ROOT_DIR PATH $optarg
;;
--binary-package)
append_cache_entry BINARY_PACKAGING_MODE BOOL true
;;

37
doc/faq.rst
View file

@ -12,43 +12,6 @@ Frequently Asked Questions
Installation and Configuration
==============================
How do I upgrade to a new version of Bro?
-----------------------------------------
There's two suggested approaches, either install Bro using the same
installation prefix directory as before, or pick a new prefix and copy
local customizations over.
Re-Use Previous Install Prefix
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you choose to configure and install Bro with the same prefix
directory as before, local customization and configuration to files in
``$prefix/share/bro/site`` and ``$prefix/etc`` won't be overwritten
(``$prefix`` indicating the root of where Bro was installed). Also, logs
generated at run-time won't be touched by the upgrade. (But making
a backup of local changes before proceeding is still recommended.)
After upgrading, remember to check ``$prefix/share/bro/site`` and
``$prefix/etc`` for ``.example`` files, which indicate the
distribution's version of the file differs from the local one, which may
include local changes. Review the differences, and make adjustments
as necessary (for differences that aren't the result of a local change,
use the new version's).
Pick a New Install prefix
^^^^^^^^^^^^^^^^^^^^^^^^^
If you want to install the newer version in a different prefix
directory than before, you can just copy local customization and
configuration files from ``$prefix/share/bro/site`` and ``$prefix/etc``
to the new location (``$prefix`` indicating the root of where Bro was
originally installed). Make sure to review the files for difference
before copying and make adjustments as necessary (for differences that
aren't the result of a local change, use the new version's). Of
particular note, the copied version of ``$prefix/etc/broctl.cfg`` is
likely to need changes to the ``SpoolDir`` and ``LogDir`` settings.
How can I tune my operating system for best capture performance?
----------------------------------------------------------------

2
doc/index.rst
View file

@ -11,8 +11,8 @@ Guides
:maxdepth: 1
INSTALL
quickstart
upgrade
quickstart
faq
reporting-problems

10
doc/input.rst
View file

@ -98,12 +98,12 @@ been completed. Because of this, it is, for example, possible to call
will remain queued until the first read has been completed.
Once the input framework finishes reading from a data source, it fires
the ``update_finished`` event. Once this event has been received all data
the ``end_of_data`` event. Once this event has been received all data
from the input file is available in the table.
.. code:: bro
event Input::update_finished(name: string, source: string) {
event Input::end_of_data(name: string, source: string) {
# now all data is in the table
print blacklist;
}
@ -129,7 +129,7 @@ deal with changing data files.
The first, very basic method is an explicit refresh of an input stream. When
an input stream is open, the function ``force_update`` can be called. This
will trigger a complete refresh of the table; any changed elements from the
file will be updated. After the update is finished the ``update_finished``
file will be updated. After the update is finished the ``end_of_data``
event will be raised.
In our example the call would look like:
@ -142,7 +142,7 @@ The input framework also supports two automatic refresh modes. The first mode
continually checks if a file has been changed. If the file has been changed, it
is re-read and the data in the Bro table is updated to reflect the current
state. Each time a change has been detected and all the new data has been
read into the table, the ``update_finished`` event is raised.
read into the table, the ``end_of_data`` event is raised.
The second mode is a streaming mode. This mode assumes that the source data
file is an append-only file to which new data is continually appended. Bro
@ -150,7 +150,7 @@ continually checks for new data at the end of the file and will add the new
data to the table. If newer lines in the file have the same index as previous
lines, they will overwrite the values in the output table. Because of the
nature of streaming reads (data is continually added to the table),
the ``update_finished`` event is never raised when using streaming reads.
the ``end_of_data`` event is never raised when using streaming reads.
The reading mode can be selected by setting the ``mode`` option of the
add_table call. Valid values are ``MANUAL`` (the default), ``REREAD``

125
doc/intel.rst Normal file
View file

@ -0,0 +1,125 @@
Intel Framework
===============
Intro
-----
Intelligence data is critical to the process of monitoring for
security purposes. There is always data which will be discovered
through the incident response process and data which is shared through
private communities. The goals of Bro's Intelligence Framework are to
consume that data, make it available for matching, and provide
infrastructure around improving performance, memory utilization, and
generally making all of this easier.
Data in the Intelligence Framework is the atomic piece of intelligence
such as an IP address or an e-mail address along with a suite of
metadata about it such as a freeform source field, a freeform
descriptive field and a URL which might lead to more information about
the specific item. The metadata in the default scripts has been
deliberately kept minimal so that the community can find the
appropriate fields that need added by writing scripts which extend the
base record using the normal record extension mechanism.
Quick Start
-----------
Load the package of scripts that sends data into the Intelligence
Framework to be checked by loading this script in local.bro::
@load policy/frameworks/intel
(TODO: find some good mechanism for getting setup with good data
quickly)
Refer to the "Loading Intelligence" section below to see the format
for Intelligence Framework text files, then load those text files with
this line in local.bro::
redef Intel::read_files += { "/somewhere/yourdata.txt" };
The data itself only needs to reside on the manager if running in a
cluster.
Architecture
------------
The Intelligence Framework can be thought of as containing three
separate portions. The first part is how intelligence is loaded,
followed by the mechanism for indicating to the intelligence framework
that a piece of data which needs to be checked has been seen, and
thirdly the part where a positive match has been discovered.
Loading Intelligence
********************
Intelligence data can only be loaded through plain text files using
the Input Framework conventions. Additionally, on clusters the
manager is the only node that needs the intelligence data. The
intelligence framework has distribution mechanisms which will push
data out to all of the nodes that need it.
Here is an example of the intelligence data format. Note that all
whitespace separators are literal tabs and fields containing only a
hyphen a considered to be null values.::
#fields host net str str_type meta.source meta.desc meta.url
1.2.3.4 - - - source1 Sending phishing email http://source1.com/badhosts/1.2.3.4
- 31.131.248.0/21 - - spamhaus-drop SBL154982 - -
- - a.b.com Intel::DOMAIN source2 Name used for data exfiltration -
For more examples of built in `str_type` values, please refer to the
autogenerated documentation for the intelligence framework (TODO:
figure out how to do this link).
To load the data once files are created, use the following example
code to define files to load with your own file names of course::
redef Intel::read_files += {
"/somewhere/feed1.txt",
"/somewhere/feed2.txt",
};
Remember, the files only need to be present on the file system of the
manager node on cluster deployments.
Seen Data
*********
When some bit of data is extracted (such as an email address in the
"From" header in a message over SMTP), the Intelligence Framework
needs to be informed that this data was discovered and it's presence
should be checked within the intelligence data set. This is
accomplished through the Intel::seen (TODO: do a reference link)
function.
Typically users won't need to work with this function due to built in
hook scripts that Bro ships with that will "see" data and send it into
the intelligence framework. A user may only need to load the entire
package of hook scripts as a module or pick and choose specific
scripts to load. Keep in mind that as more data is sent into the
intelligence framework, the CPU load consumed by Bro will increase
depending on how many times the Intel::seen function is being called
which is heavily traffic dependent.
The full package of hook scripts that Bro ships with for sending this
"seen" data into the intelligence framework can be loading by adding
this line to local.bro::
@load policy/frameworks/intel
Intelligence Matches
********************
Against all hopes, most networks will eventually have a hit on
intelligence data which could indicate a possible compromise or other
unwanted activity. The Intelligence Framework provides an event that
is generated whenever a match is discovered named Intel::match (TODO:
make a link to inline docs). Due to design restrictions placed upon
the intelligence framework, there is no assurance as to where this
event will be generated. It could be generated on the worker where
the data was seen or on the manager. When the Intel::match event is
handled, only the data given as event arguments to the event can be
assured since the host where the data was seen may not be where
Intel::match is handled.

188
doc/quickstart.rst
View file

@ -1,10 +1,3 @@
.. _CMake: http://www.cmake.org
.. _SWIG: http://www.swig.org
.. _Xcode: https://developer.apple.com/xcode/
.. _MacPorts: http://www.macports.org
.. _Fink: http://www.finkproject.org
.. _Homebrew: http://mxcl.github.com/homebrew
.. _bro downloads page: http://bro-ids.org/download/index.html
=================
Quick Start Guide
@ -23,181 +16,11 @@ Installation
Bro works on most modern, Unix-based systems and requires no custom
hardware. It can be downloaded in either pre-built binary package or
source code forms.
source code forms. See :doc:`Installing Bro <INSTALL>` for instructions
on how to install Bro.
Pre-Built Binary Release Packages
---------------------------------
See the `bro downloads page`_ for currently supported/targeted platforms.
* RPM
.. console::
sudo yum localinstall Bro-*.rpm
* DEB
.. console::
sudo gdebi Bro-*.deb
* MacOS Disk Image with Installer
Just open the ``Bro-*.dmg`` and then run the ``.pkg`` installer.
Everything installed by the package will go into ``/opt/bro``.
The primary install prefix for binary packages is ``/opt/bro``.
Non-MacOS packages that include BroControl also put variable/runtime
data (e.g. Bro logs) in ``/var/opt/bro``.
Building From Source
--------------------
Required Dependencies
~~~~~~~~~~~~~~~~~~~~~
The following dependencies are required to build Bro:
* RPM/RedHat-based Linux:
.. console::
sudo yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel file-devel
* DEB/Debian-based Linux:
.. console::
sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev libmagic-dev
* FreeBSD
Most required dependencies should come with a minimal FreeBSD install
except for the following.
.. console::
sudo pkg_add -r bash cmake swig bison python
Note that ``bash`` needs to be in ``PATH``, which by default it is
not. The FreeBSD package installs the binary into
``/usr/local/bin``.
* Mac OS X
Compiling source code on Macs requires first downloading Xcode_,
then going through its "Preferences..." -> "Downloads" menus to
install the "Command Line Tools" component.
Lion (10.7) and Mountain Lion (10.8) come with all required
dependencies except for CMake_, SWIG_, and ``libmagic``.
Distributions of these dependencies can be obtained from the project
websites linked above, but they're also likely available from your
preferred Mac OS X package management system (e.g. MacPorts_, Fink_,
or Homebrew_).
Specifically for MacPorts, the ``swig``, ``swig-ruby``, ``swig-python``
and ``file`` packages provide the required dependencies.
Optional Dependencies
~~~~~~~~~~~~~~~~~~~~~
Bro can use libGeoIP for geo-locating IP addresses, and sendmail for
sending emails.
* RedHat Enterprise Linux:
.. console::
sudo yum install geoip-devel sendmail
* CentOS Linux:
.. console::
sudo yum install GeoIP-devel sendmail
* DEB/Debian-based Linux:
.. console::
sudo apt-get install libgeoip-dev sendmail
* Ports-based FreeBSD
.. console::
sudo pkg_add -r GeoIP
sendmail is typically already available.
* Mac OS X
Vanilla OS X installations don't ship with libmagic or libGeoIP, but
if installed from your preferred package management system (e.g. MacPorts,
Fink, or Homebrew), they should be automatically detected and Bro will compile
against them.
Additional steps may be needed to :doc:`get the right GeoIP database <geoip>`
Compiling Bro Source Code
~~~~~~~~~~~~~~~~~~~~~~~~~
Bro releases are bundled into source packages for convenience and
available from the `bro downloads page`_.
The latest Bro development versions are obtainable through git
repositories hosted at `git.bro-ids.org <http://git.bro-ids.org>`_. See
our `git development documentation
<http://bro-ids.org/development/process.html>`_ for comprehensive
information on Bro's use of git revision control, but the short story
for downloading the full source code experience for Bro via git is:
.. console::
git clone --recursive git://git.bro-ids.org/bro
.. note:: If you choose to clone the ``bro`` repository non-recursively for
a "minimal Bro experience", be aware that compiling it depends on
BinPAC, which has its own ``binpac`` repository. Either install it
first or initialize/update the cloned ``bro`` repository's
``aux/binpac`` submodule.
See the ``INSTALL`` file included with the source code for more information
on compiling, but this is the typical way to build and install from source
(of course, changing the value of the ``--prefix`` option to point to the
desired root install path):
.. console::
./configure --prefix=/desired/install/path
make
make install
The default installation prefix is ``/usr/local/bro``, which would typically
require root privileges when doing the ``make install``.
Configure the Run-Time Environment
----------------------------------
Just remember that you may need to adjust your ``PATH`` environment variable
according to the platform/shell/package you're using. For example:
Bourne-Shell Syntax:
.. console::
export PATH=/usr/local/bro/bin:$PATH
C-Shell Syntax:
.. console::
setenv PATH /usr/local/bro/bin:$PATH
Or substitute ``/opt/bro/bin`` instead if you installed from a binary package.
.. note:: Below, ``$PREFIX`` is used to reference the Bro installation
root directory.
Using BroControl
================
@ -206,9 +29,6 @@ BroControl is an interactive shell for easily operating/managing Bro
installations on a single system or even across multiple systems in a
traffic-monitoring cluster.
.. note:: Below, ``$PREFIX`` is used to reference the Bro installation
root directory.
A Minimal Starting Configuration
--------------------------------

21
doc/scripts/DocSourcesList.cmake
View file

@ -36,6 +36,8 @@ rest_target(${psd} base/frameworks/input/main.bro)
rest_target(${psd} base/frameworks/input/readers/ascii.bro)
rest_target(${psd} base/frameworks/input/readers/benchmark.bro)
rest_target(${psd} base/frameworks/input/readers/raw.bro)
rest_target(${psd} base/frameworks/intel/cluster.bro)
rest_target(${psd} base/frameworks/intel/input.bro)
rest_target(${psd} base/frameworks/intel/main.bro)
rest_target(${psd} base/frameworks/logging/main.bro)
rest_target(${psd} base/frameworks/logging/postprocessors/scp.bro)
@ -62,12 +64,15 @@ rest_target(${psd} base/frameworks/reporter/main.bro)
rest_target(${psd} base/frameworks/signatures/main.bro)
rest_target(${psd} base/frameworks/software/main.bro)
rest_target(${psd} base/frameworks/tunnels/main.bro)
rest_target(${psd} base/misc/find-checksum-offloading.bro)
rest_target(${psd} base/protocols/conn/contents.bro)
rest_target(${psd} base/protocols/conn/inactivity.bro)
rest_target(${psd} base/protocols/conn/main.bro)
rest_target(${psd} base/protocols/conn/polling.bro)
rest_target(${psd} base/protocols/dns/consts.bro)
rest_target(${psd} base/protocols/dns/main.bro)
rest_target(${psd} base/protocols/ftp/file-extract.bro)
rest_target(${psd} base/protocols/ftp/gridftp.bro)
rest_target(${psd} base/protocols/ftp/main.bro)
rest_target(${psd} base/protocols/ftp/utils-commands.bro)
rest_target(${psd} base/protocols/http/file-extract.bro)
@ -77,6 +82,8 @@ rest_target(${psd} base/protocols/http/main.bro)
rest_target(${psd} base/protocols/http/utils.bro)
rest_target(${psd} base/protocols/irc/dcc-send.bro)
rest_target(${psd} base/protocols/irc/main.bro)
rest_target(${psd} base/protocols/modbus/consts.bro)
rest_target(${psd} base/protocols/modbus/main.bro)
rest_target(${psd} base/protocols/smtp/entities-excerpt.bro)
rest_target(${psd} base/protocols/smtp/entities.bro)
rest_target(${psd} base/protocols/smtp/main.bro)
@ -98,11 +105,21 @@ rest_target(${psd} base/utils/patterns.bro)
rest_target(${psd} base/utils/site.bro)
rest_target(${psd} base/utils/strings.bro)
rest_target(${psd} base/utils/thresholds.bro)
rest_target(${psd} base/utils/urls.bro)
rest_target(${psd} policy/frameworks/communication/listen.bro)
rest_target(${psd} policy/frameworks/control/controllee.bro)
rest_target(${psd} policy/frameworks/control/controller.bro)
rest_target(${psd} policy/frameworks/dpd/detect-protocols.bro)
rest_target(${psd} policy/frameworks/dpd/packet-segment-logging.bro)
rest_target(${psd} policy/frameworks/intel/conn-established.bro)
rest_target(${psd} policy/frameworks/intel/dns.bro)
rest_target(${psd} policy/frameworks/intel/http-host-header.bro)
rest_target(${psd} policy/frameworks/intel/http-url.bro)
rest_target(${psd} policy/frameworks/intel/http-user-agents.bro)
rest_target(${psd} policy/frameworks/intel/smtp-url-extraction.bro)
rest_target(${psd} policy/frameworks/intel/smtp.bro)
rest_target(${psd} policy/frameworks/intel/ssl.bro)
rest_target(${psd} policy/frameworks/intel/where-locations.bro)
rest_target(${psd} policy/frameworks/metrics/conn-example.bro)
rest_target(${psd} policy/frameworks/metrics/http-example.bro)
rest_target(${psd} policy/frameworks/metrics/ssl-example.bro)
@ -110,6 +127,7 @@ rest_target(${psd} policy/frameworks/software/version-changes.bro)
rest_target(${psd} policy/frameworks/software/vulnerable.bro)
rest_target(${psd} policy/integration/barnyard2/main.bro)
rest_target(${psd} policy/integration/barnyard2/types.bro)
rest_target(${psd} policy/integration/collective-intel/main.bro)
rest_target(${psd} policy/misc/analysis-groups.bro)
rest_target(${psd} policy/misc/capture-loss.bro)
rest_target(${psd} policy/misc/loaded-scripts.bro)
@ -124,7 +142,6 @@ rest_target(${psd} policy/protocols/dns/detect-external-names.bro)
rest_target(${psd} policy/protocols/ftp/detect.bro)
rest_target(${psd} policy/protocols/ftp/software.bro)
rest_target(${psd} policy/protocols/http/detect-MHR.bro)
rest_target(${psd} policy/protocols/http/detect-intel.bro)
rest_target(${psd} policy/protocols/http/detect-sqli.bro)
rest_target(${psd} policy/protocols/http/detect-webapps.bro)
rest_target(${psd} policy/protocols/http/header-names.bro)
@ -132,6 +149,8 @@ rest_target(${psd} policy/protocols/http/software-browser-plugins.bro)
rest_target(${psd} policy/protocols/http/software.bro)
rest_target(${psd} policy/protocols/http/var-extraction-cookies.bro)
rest_target(${psd} policy/protocols/http/var-extraction-uri.bro)
rest_target(${psd} policy/protocols/modbus/known-masters-slaves.bro)
rest_target(${psd} policy/protocols/modbus/track-memmap.bro)
rest_target(${psd} policy/protocols/smtp/blocklists.bro)
rest_target(${psd} policy/protocols/smtp/detect-suspicious-orig.bro)
rest_target(${psd} policy/protocols/smtp/software.bro)

72
doc/scripts/builtins.rst
View file

@ -417,10 +417,6 @@ The Bro scripting language supports the following built-in types.
Writing to files like this for logging usually isn't recommended, for better
logging support see :doc:`/logging`.
.. bro:type:: func
See :bro:type:`function`.
.. bro:type:: function
Function types in Bro are declared using::
@ -504,6 +500,74 @@ The Bro scripting language supports the following built-in types.
identifier and the body of each will be executed in turn. Ordering
of execution can be influenced with :bro:attr:`&priority`.
.. bro:type:: hook
A hook is another flavor of function that shares characteristics of
both a :bro:type:`function` and a :bro:type:`event`. They are like
events in that many handler bodies can be defined for the same hook
identifier, they have no return vale, and the order of execution
can be enforced with :bro:attr:`&priority`. They are more like
functions in the way they are invoked/called, because, unlike
events, their execution is immediate and they do not get scheduled
through an event queue. Also, a unique feature of a hook is that
a given hook handler body can short-circuit the execution of
remaining hook handlers simply by exiting from the body as a result
of a ``break`` statement (as opposed to a ``return`` or just
reaching the end of the body).
A hook type is declared like::
hook( argument* )
where *argument* is a (possibly empty) comma-separated list of
arguments. For example:
.. code:: bro
global myhook: hook(s: string)
Here ``myhook`` is the hook type identifier and no hook handler
bodies have been defined for it yet. To define some hook handler
bodies the syntax looks like:
.. code:: bro
hook myhook(s: string) &priority=10
{
print "priority 10 myhook handler", s;
s = "bye";
}
hook myhook(s: string)
{
print "break out of myhook handling", s;
break;
}
hook myhook(s: string) &priority=-5
{
print "not going to happen", s;
}
Note that, although the first (forward) declaration of ``myhook`` as
a hook type isn't strictly required, when it is provided, the
argument types must match.
To invoke immediate execution of all hook handler bodies, a ``hook``
statement must be used:
.. code:: bro
hook myhook("hi");
And the output would like like::
priority 10 myhook handler, hi
break out of myhook handling, bye
Note how the modification to arguments can be seen by remaining
hook handlers.
Attributes
----------

57
doc/signatures.rst
View file

@ -83,9 +83,8 @@ Header Conditions
~~~~~~~~~~~~~~~~~
Header conditions limit the applicability of the signature to a subset
of traffic that contains matching packet headers. For TCP, this match
is performed only for the first packet of a connection. For other
protocols, it is done on each individual packet.
of traffic that contains matching packet headers. This type of matching
is performed only for the first packet of a connection.
There are pre-defined header conditions for some of the most used
header fields. All of them generally have the format ``<keyword> <cmp>
@ -95,14 +94,22 @@ one of ``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``; and
against. The following keywords are defined:
``src-ip``/``dst-ip <cmp> <address-list>``
Source and destination address, respectively. Addresses can be
given as IP addresses or CIDR masks.
Source and destination address, respectively. Addresses can be given
as IPv4 or IPv6 addresses or CIDR masks. For IPv6 addresses/masks
the colon-hexadecimal representation of the address must be enclosed
in square brackets (e.g. ``[fe80::1]`` or ``[fe80::0]/16``).
``src-port``/``dst-port`` ``<int-list>``
``src-port``/``dst-port <cmp> <int-list>``
Source and destination port, respectively.
``ip-proto tcp|udp|icmp``
IP protocol.
``ip-proto <cmp> tcp|udp|icmp|icmp6|ip|ip6``
IPv4 header's Protocol field or the Next Header field of the final
IPv6 header (i.e. either Next Header field in the fixed IPv6 header
if no extension headers are present or that field from the last
extension header in the chain). Note that the IP-in-IP forms of
tunneling are automatically decapsulated by default and signatures
apply to only the inner-most packet, so specifying ``ip`` or ``ip6``
is a no-op.
For lists of multiple values, they are sequentially compared against
the corresponding header field. If at least one of the comparisons
@ -116,20 +123,22 @@ condition can be defined either as
header <proto>[<offset>:<size>] [& <integer>] <cmp> <value-list>
This compares the value found at the given position of the packet
header with a list of values. ``offset`` defines the position of the
value within the header of the protocol defined by ``proto`` (which
can be ``ip``, ``tcp``, ``udp`` or ``icmp``). ``size`` is either 1, 2,
or 4 and specifies the value to have a size of this many bytes. If the
optional ``& <integer>`` is given, the packet's value is first masked
with the integer before it is compared to the value-list. ``cmp`` is
one of ``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``. ``value-list`` is
a list of comma-separated integers similar to those described above.
The integers within the list may be followed by an additional ``/
mask`` where ``mask`` is a value from 0 to 32. This corresponds to the
CIDR notation for netmasks and is translated into a corresponding
bitmask applied to the packet's value prior to the comparison (similar
to the optional ``& integer``).
This compares the value found at the given position of the packet header
with a list of values. ``offset`` defines the position of the value
within the header of the protocol defined by ``proto`` (which can be
``ip``, ``ip6``, ``tcp``, ``udp``, ``icmp`` or ``icmp6``). ``size`` is
either 1, 2, or 4 and specifies the value to have a size of this many
bytes. If the optional ``& <integer>`` is given, the packet's value is
first masked with the integer before it is compared to the value-list.
``cmp`` is one of ``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``.
``value-list`` is a list of comma-separated integers similar to those
described above. The integers within the list may be followed by an
additional ``/ mask`` where ``mask`` is a value from 0 to 32. This
corresponds to the CIDR notation for netmasks and is translated into a
corresponding bitmask applied to the packet's value prior to the
comparison (similar to the optional ``& integer``). IPv6 address values
are not allowed in the value-list, though you can still inspect any 1,
2, or 4 byte section of an IPv6 header using this keyword.
Putting it all together, this is an example condition that is
equivalent to ``dst-ip == 1.2.3.4/16, 5.6.7.8/24``:
@ -138,8 +147,8 @@ equivalent to ``dst-ip == 1.2.3.4/16, 5.6.7.8/24``:
header ip[16:4] == 1.2.3.4/16, 5.6.7.8/24
Internally, the predefined header conditions are in fact just
short-cuts and mapped into a generic condition.
Note that the analogous example for IPv6 isn't currently possible since
4 bytes is the max width of a value that can be compared.
Content Conditions
~~~~~~~~~~~~~~~~~~

73
doc/upgrade.rst
View file

@ -1,19 +1,80 @@
=============================
Upgrading From Bro 1.5 to 2.0
=============================
==========================================
Upgrading From the Previous Version of Bro
==========================================
.. rst-class:: opening
This guide details differences between Bro versions 1.5 and 2.0
This guide details specific differences between Bro versions
that may be important for users to know as they work on updating
their Bro deployment/configuration to the later version.
.. contents::
Introduction
============
Upgrading From Bro 2.0 to 2.1
=============================
In Bro 2.1, IPv6 is enabled by default. Therefore, when building Bro from
source, the "--enable-brov6" configure option has been removed because it
is no longer relevant.
Other configure changes include renaming the "--enable-perftools" option
to "--enable-perftools-debug" to indicate that the option is only relevant
for debugging the heap. One other change involves what happens when
tcmalloc (part of Google perftools) is found at configure time. On Linux,
it will automatically be linked with Bro, but on other platforms you
need to use the "--enable-perftools" option to enable linking to tcmalloc.
There are a couple of changes to the Bro scripting language to better
support IPv6. First, IPv6 literals appearing in a Bro script must now be
enclosed in square brackets (for example, ``[fe80::db15]``). For subnet
literals, the slash "/" appears after the closing square bracket (for
example, ``[fe80:1234::]/32``). Second, when an IP address variable or IP
address literal is enclosed in pipes (for example, ``|[fe80::db15]|``) the
result is now the size of the address in bits (32 for IPv4 and 128 for IPv6).
In the Bro scripting language, "match" and "using" are no longer reserved
keywords.
Some built-in functions have been removed: "addr_to_count" (use
"addr_to_counts" instead), "bro_has_ipv6" (this is no longer relevant
because Bro now always supports IPv6), "active_connection" (use
"connection_exists" instead), and "connection_record" (use "lookup_connection"
instead).
The "NFS3::mode2string" built-in function has been renamed to "file_mode".
Some built-in functions have been changed: "exit" (now takes the exit code
as a parameter), "to_port" (now takes a string as parameter instead
of a count and transport protocol, but "count_to_port" is still available),
"connect" (now takes an additional string parameter specifying the zone of
a non-global IPv6 address), and "listen" (now takes three additional
parameters to enable listening on IPv6 addresses).
Some Bro script variables have been renamed: "LogAscii::header_prefix"
has been renamed to "LogAscii::meta_prefix", "LogAscii::include_header"
has been renamed to "LogAscii::include_meta".
Some Bro script variables have been removed: "tunnel_port",
"parse_udp_tunnels", "use_connection_compressor", "cc_handle_resets",
"cc_handle_only_syns", and "cc_instantiate_on_data".
A couple events have changed: the "icmp_redirect" event now includes
the target and destination addresses and any Neighbor Discovery options
in the message, and the last parameter of the "dns_AAAA_reply" event has
been removed because it was unused.
The format of the ASCII log files has changed very slightly. Two new lines
are automatically added, one to record the time when the log was opened,
and the other to record the time when the log was closed.
In BroControl, the option (in broctl.cfg) "CFlowAddr" was renamed
to "CFlowAddress".
Upgrading From Bro 1.5 to 2.0
=============================
As the version number jump suggests, Bro 2.0 is a major upgrade and
lots of things have changed. Most importantly, we have rewritten

4
scripts/base/frameworks/cluster/nodes/manager.bro
View file

@ -13,8 +13,12 @@
## Turn off remote logging since this is the manager and should only log here.
redef Log::enable_remote_logging = F;
## Log rotation interval.
redef Log::default_rotation_interval = 1 hrs;
## Alarm summary mail interval.
redef Log::default_mail_alarms_interval = 24 hrs;
## Use the cluster's archive logging script.
redef Log::default_rotation_postprocessor_cmd = "archive-log";

8
scripts/base/frameworks/input/main.bro
View file

@ -114,7 +114,8 @@ export {
## description: `TableDescription` record describing the source.
global add_event: function(description: Input::EventDescription) : bool;
## Remove a input stream. Returns true on success and false if the named stream was not found.
## Remove a input stream. Returns true on success and false if the named stream was
## not found.
##
## id: string value identifying the stream to be removed
global remove: function(id: string) : bool;
@ -125,8 +126,9 @@ export {
## id: string value identifying the stream
global force_update: function(id: string) : bool;
## Event that is called, when the update of a specific source is finished
global update_finished: event(name: string, source:string);
## Event that is called, when the end of a data source has been reached, including
## after an update.
global end_of_data: event(name: string, source:string);
}
@load base/input.bif

10
scripts/base/frameworks/intel/__load__.bro
View file

@ -1 +1,11 @@
@load ./main
# The cluster framework must be loaded first.
@load base/frameworks/cluster
@if ( Cluster::is_enabled() )
@load ./cluster
@endif
# This needs cluster support to only read on the manager.
@load ./input

61
scripts/base/frameworks/intel/cluster.bro Normal file
View file

@ -0,0 +1,61 @@
##! Cluster transparency support for the intelligence framework. This is mostly oriented
##! toward distributing intelligence information across clusters.
@load base/frameworks/cluster
@load ./input
module Intel;
redef record Item += {
## This field is used internally for cluster transparency to avoid
## re-dispatching intelligence items over and over from workers.
first_dispatch: bool &default=T;
};
# If this process is not a manager process, we don't want the full metadata
@if ( Cluster::local_node_type() != Cluster::MANAGER )
redef have_full_data = F;
@endif
global cluster_new_item: event(item: Item);
# Primary intelligence distribution comes from manager.
redef Cluster::manager2worker_events += /^Intel::(cluster_new_item)$/;
# If a worker finds intelligence and adds it, it should share it back to the manager.
redef Cluster::worker2manager_events += /^Intel::(cluster_new_item|match_no_items)$/;
@if ( Cluster::local_node_type() == Cluster::MANAGER )
event Intel::match_no_items(s: Seen) &priority=5
{
event Intel::match(s, Intel::get_items(s));
}
event remote_connection_handshake_done(p: event_peer)
{
# When a worker connects, send it the complete minimal data store.
# It will be kept up to date after this by the cluster_new_item event.
if ( Cluster::nodes[p$descr]$node_type == Cluster::WORKER )
{
send_id(p, "Intel::min_data_store");
}
}
@endif
event Intel::cluster_new_item(item: Intel::Item) &priority=5
{
# Ignore locally generated events to avoid event storms.
if ( is_remote_event() )
Intel::insert(item);
}
event Intel::new_item(item: Intel::Item) &priority=5
{
# The cluster manager always rebroadcasts intelligence.
# Workers redistribute it if it was locally generated.
if ( Cluster::local_node_type() == Cluster::MANAGER ||
item$first_dispatch )
{
item$first_dispatch=F;
event Intel::cluster_new_item(item);
}
}

33
scripts/base/frameworks/intel/input.bro Normal file
View file

@ -0,0 +1,33 @@
@load ./main
module Intel;
export {
## Intelligence files that will be read off disk. The files are
## reread everytime they are updated so updates much be atomic with
## "mv" instead of writing the file in place.
const read_files: set[string] = {} &redef;
}
event Intel::read_entry(desc: Input::EventDescription, tpe: Input::Event, item: Intel::Item)
{
Intel::insert(item);
}
event bro_init() &priority=5
{
if ( ! Cluster::is_enabled() ||
Cluster::local_node_type() == Cluster::MANAGER )
{
for ( a_file in read_files )
{
Input::add_event([$source=a_file,
$reader=Input::READER_ASCII,
$mode=Input::REREAD,
$name=cat("intel-", a_file),
$fields=Intel::Item,
$ev=Intel::read_entry]);
}
}
}

564
scripts/base/frameworks/intel/main.bro
View file

@ -1,323 +1,345 @@
##! The intelligence framework provides a way to store and query IP addresses,
##! strings (with a subtype), and numeric (with a subtype) data. Metadata
##! also be associated with the intelligence like tags which are arbitrary
##! strings, time values, and longer descriptive strings.
# Example string subtypes:
# url
# email
# domain
# software
# user_name
# file_name
# file_md5
# x509_md5
# Example tags:
# infrastructure
# malicious
# sensitive
# canary
# friend
##! and strings (with a str_type). Metadata can
##! also be associated with the intelligence like for making more informed
##! decisions about matching and handling of intelligence.
@load base/frameworks/notice
module Intel;
export {
## The intel logging stream identifier.
redef enum Log::ID += { LOG };
redef enum Notice::Type += {
## This notice should be used in all detector scripts to indicate
## an intelligence based detection.
Detection,
## String data needs to be further categoried since it could represent
## and number of types of data.
type StrType: enum {
## A complete URL without the prefix "http://".
URL,
## User-Agent string, typically HTTP or mail message body.
USER_AGENT,
## Email address.
EMAIL,
## DNS domain name.
DOMAIN,
## A user name.
USER_NAME,
## File hash which is non-hash type specific. It's up to the user to query
## for any relevant hash types.
FILE_HASH,
## Certificate SHA-1 hash.
CERT_HASH,
};
## Record type used for logging information from the intelligence framework.
## Primarily for problems or oddities with inserting and querying data.
## This is important since the content of the intelligence framework can
## change quite dramatically during runtime and problems may be introduced
## into the data.
type Info: record {
## The current network time.
ts: time &log;
## Represents the severity of the message.
## This value should be one of: "info", "warn", "error"
level: string &log;
## The message.
message: string &log;
};
## Record to represent metadata associated with a single piece of
## intelligence.
## Data about an :bro:type:`Intel::Item`
type MetaData: record {
## A description for the data.
## An arbitrary string value representing the data source. Typically,
## the convention for this field will be the source name and feed name
## separated by a hyphen. For example: "source1-c&c".
source: string;
## A freeform description for the data.
desc: string &optional;
## A URL where more information may be found about the intelligence.
## A URL for more information about the data.
url: string &optional;
## The time at which the data was first declared to be intelligence.
first_seen: time &optional;
## When this data was most recent inserted into the framework.
latest_seen: time &optional;
## Arbitrary text tags for the data.
tags: set[string];
};
## Record to represent a singular piece of intelligence.
## Represents a piece of intelligence.
type Item: record {
## If the data is an IP address, this hold the address.
ip: addr &optional;
## If the data is textual, this holds the text.
## The IP address if the intelligence is about an IP address.
host: addr &optional;
## The network if the intelligence is about a CIDR block.
net: subnet &optional;
## The string if the intelligence is about a string.
str: string &optional;
## If the data is numeric, this holds the number.
num: int &optional;
## The subtype of the data for when either the $str or $num fields are
## given. If one of those fields are given, this field must be present.
subtype: string &optional;
## The type of data that is in the string if the $str field is set.
str_type: StrType &optional;
## The next five fields are temporary until a better model for
## attaching metadata to an intelligence item is created.
desc: string &optional;
url: string &optional;
first_seen: time &optional;
latest_seen: time &optional;
tags: set[string];
## These single string tags are throw away until pybroccoli supports sets.
tag1: string &optional;
tag2: string &optional;
tag3: string &optional;
## Metadata for the item. Typically represents more deeply \
## descriptive data for a piece of intelligence.
meta: MetaData;
};
## Record model used for constructing queries against the intelligence
## framework.
type QueryItem: record {
## If an IP address is being queried for, this field should be given.
ip: addr &optional;
## If a string is being queried for, this field should be given.
str: string &optional;
## If numeric data is being queried for, this field should be given.
num: int &optional;
## If either a string or number is being queried for, this field should
## indicate the subtype of the data.
subtype: string &optional;
## A set of tags where if a single metadata record attached to an item
## has any one of the tags defined in this field, it will match.
or_tags: set[string] &optional;
## A set of tags where a single metadata record attached to an item
## must have all of the tags defined in this field.
and_tags: set[string] &optional;
## The predicate can be given when searching for a match. It will
## be tested against every :bro:type:`Intel::MetaData` item associated
## with the data being matched on. If it returns T a single time, the
## matcher will consider that the item has matched. This field can
## be used for constructing arbitrarily complex queries that may not
## be possible with the $or_tags or $and_tags fields.
pred: function(meta: Intel::MetaData): bool &optional;
## Enum to represent where data came from when it was discovered.
## The convention is to prefix the name with ``IN_``.
type Where: enum {
## A catchall value to represent data of unknown provenance.
IN_ANYWHERE,
};
## Function to insert data into the intelligence framework.
##
## item: The data item.
##
## Returns: T if the data was successfully inserted into the framework,
## otherwise it returns F.
global insert: function(item: Item): bool;
## The $host field and combination of $str and $str_type fields are mutually
## exclusive. These records *must* represent either an IP address being
## seen or a string being seen.
type Seen: record {
## The IP address if the data seen is an IP address.
host: addr &log &optional;
## The string if the data is about a string.
str: string &log &optional;
## The type of data that is in the string if the $str field is set.
str_type: StrType &log &optional;
## A wrapper for the :bro:id:`Intel::insert` function. This is primarily
## used as the external API for inserting data into the intelligence
## using Broccoli.
global insert_event: event(item: Item);
## Where the data was discovered.
where: Where &log;
## Function for matching data within the intelligence framework.
global matcher: function(item: QueryItem): bool;
## If the data was discovered within a connection, the
## connection record should go into get to give context to the data.
conn: connection &optional;
};
## Record used for the logging framework representing a positive
## hit within the intelligence framework.
type Info: record {
## Timestamp when the data was discovered.
ts: time &log;
## If a connection was associated with this intelligence hit,
## this is the uid for the connection
uid: string &log &optional;
## If a connection was associated with this intelligence hit,
## this is the conn_id for the connection.
id: conn_id &log &optional;
## Where the data was seen.
seen: Seen &log;
## Sources which supplied data that resulted in this match.
sources: set[string] &log;
};
## Intelligence data manipulation functions.
global insert: function(item: Item);
## Function to declare discovery of a piece of data in order to check
## it against known intelligence for matches.
global seen: function(s: Seen);
## Event to represent a match in the intelligence data from data that was seen.
## On clusters there is no assurance as to where this event will be generated
## so do not assume that arbitrary global state beyond the given data
## will be available.
##
## This is the primary mechanism where a user will take actions based on data
## within the intelligence framework.
global match: event(s: Seen, items: set[Item]);
global log_intel: event(rec: Info);
}
type MetaDataStore: table[count] of MetaData;
# Internal handler for matches with no metadata available.
global match_no_items: event(s: Seen);
# Internal events for cluster data distribution
global new_item: event(item: Item);
global updated_item: event(item: Item);
# Optionally store metadata. This is used internally depending on
# if this is a cluster deployment or not.
const have_full_data = T &redef;
# The in memory data structure for holding intelligence.
type DataStore: record {
ip_data: table[addr] of MetaDataStore;
# The first string is the actual value and the second string is the subtype.
string_data: table[string, string] of MetaDataStore;
int_data: table[int, string] of MetaDataStore;
net_data: table[subnet] of set[MetaData];
string_data: table[string, StrType] of set[MetaData];
};
global data_store: DataStore;
global data_store: DataStore &redef;
event bro_init()
# The in memory data structure for holding the barest matchable intelligence.
# This is primarily for workers to do the initial quick matches and store
# a minimal amount of data for the full match to happen on the manager.
type MinDataStore: record {
net_data: set[subnet];
string_data: set[string, StrType];
};
global min_data_store: MinDataStore &redef;
event bro_init() &priority=5
{
Log::create_stream(Intel::LOG, [$columns=Info]);
Log::create_stream(LOG, [$columns=Info, $ev=log_intel]);
}
function find(s: Seen): bool
{
if ( s?$host &&
((have_full_data && s$host in data_store$net_data) ||
(s$host in min_data_store$net_data)))
{
return T;
}
else if ( s?$str && s?$str_type &&
((have_full_data && [s$str, s$str_type] in data_store$string_data) ||
([s$str, s$str_type] in min_data_store$string_data)))
{
return T;
}
else
{
return F;
}
}
function get_items(s: Seen): set[Item]
{
local item: Item;
local return_data: set[Item] = set();
if ( ! have_full_data )
{
# A reporter warning should be generated here because this function
# should never be called from a host that doesn't have the full data.
# TODO: do a reporter warning.
return return_data;
}
if ( s?$host )
{
# See if the host is known about and it has meta values
if ( s$host in data_store$net_data )
{
for ( m in data_store$net_data[s$host] )
{
# TODO: the lookup should be finding all and not just most specific
# and $host/$net should have the correct value.
item = [$host=s$host, $meta=m];
add return_data[item];
}
}
}
else if ( s?$str && s?$str_type )
{
# See if the string is known about and it has meta values
if ( [s$str, s$str_type] in data_store$string_data )
{
for ( m in data_store$string_data[s$str, s$str_type] )
{
item = [$str=s$str, $str_type=s$str_type, $meta=m];
add return_data[item];
}
}
}
return return_data;
}
function Intel::seen(s: Seen)
{
if ( find(s) )
{
if ( have_full_data )
{
local items = get_items(s);
event Intel::match(s, items);
}
else
{
event Intel::match_no_items(s);
}
}
}
function insert(item: Item): bool
function has_meta(check: MetaData, metas: set[MetaData]): bool
{
local err_msg = "";
if ( (item?$str || item?$num) && ! item?$subtype )
err_msg = "You must provide a subtype to insert_sync or this item doesn't make sense.";
local check_hash = md5_hash(check);
for ( m in metas )
{
if ( check_hash == md5_hash(m) )
return T;
}
if ( err_msg == "" )
# The records must not be equivalent if we made it this far.
return F;
}
event Intel::match(s: Seen, items: set[Item]) &priority=5
{
local empty_set: set[string] = set();
local info: Info = [$ts=network_time(), $seen=s, $sources=empty_set];
if ( s?$conn )
{
info$uid = s$conn$uid;
info$id = s$conn$id;
}
for ( item in items )
add info$sources[item$meta$source];
Log::write(Intel::LOG, info);
}
function insert(item: Item)
{
if ( item?$str && !item?$str_type )
{
event reporter_warning(network_time(), fmt("You must provide a str_type for strings or this item doesn't make sense. Item: %s", item), "");
return;
}
# Create and fill out the meta data item.
local meta: MetaData;
if ( item?$first_seen )
meta$first_seen = item$first_seen;
if ( item?$latest_seen )
meta$latest_seen = item$latest_seen;
if ( item?$tags )
meta$tags = item$tags;
if ( item?$desc )
meta$desc = item$desc;
if ( item?$url )
meta$url = item$url;
local meta = item$meta;
local metas: set[MetaData];
# This is hopefully only temporary until pybroccoli supports sets.
if ( item?$tag1 )
add item$tags[item$tag1];
if ( item?$tag2 )
add item$tags[item$tag2];
if ( item?$tag3 )
add item$tags[item$tag3];
if ( item?$ip )
if ( item?$host )
{
if ( item$ip !in data_store$ip_data )
data_store$ip_data[item$ip] = table();
data_store$ip_data[item$ip][|data_store$ip_data[item$ip]|] = meta;
return T;
local host = mask_addr(item$host, is_v4_addr(item$host) ? 32 : 128);
if ( have_full_data )
{
if ( host !in data_store$net_data )
data_store$net_data[host] = set();
metas = data_store$net_data[host];
}
add min_data_store$net_data[host];
}
else if ( item?$net )
{
if ( have_full_data )
{
if ( item$net !in data_store$net_data )
data_store$net_data[item$net] = set();
metas = data_store$net_data[item$net];
}
add min_data_store$net_data[item$net];
}
else if ( item?$str )
{
if ( [item$str, item$subtype] !in data_store$string_data )
data_store$string_data[item$str, item$subtype] = table();
data_store$string_data[item$str, item$subtype][|data_store$string_data[item$str, item$subtype]|] = meta;
return T;
}
else if ( item?$num )
if ( have_full_data )
{
if ( [item$num, item$subtype] !in data_store$int_data )
data_store$int_data[item$num, item$subtype] = table();
if ( [item$str, item$str_type] !in data_store$string_data )
data_store$string_data[item$str, item$str_type] = set();
data_store$int_data[item$num, item$subtype][|data_store$int_data[item$num, item$subtype]|] = meta;
return T;
metas = data_store$string_data[item$str, item$str_type];
}
add min_data_store$string_data[item$str, item$str_type];
}
local updated = F;
if ( have_full_data )
{
for ( m in metas )
{
if ( meta$source == m$source )
{
if ( has_meta(meta, metas) )
{
# It's the same item being inserted again.
return;
}
else
err_msg = "Failed to insert intelligence item for some unknown reason.";
{
# Same source, different metadata means updated item.
updated = T;
}
}
}
add metas[item$meta];
}
if ( err_msg != "" )
Log::write(Intel::LOG, [$ts=network_time(), $level="warn", $message=fmt(err_msg)]);
return F;
}
event insert_event(item: Item)
{
insert(item);
}
function match_item_with_metadata(item: QueryItem, meta: MetaData): bool
{
if ( item?$and_tags )
{
local matched = T;
# Every tag given has to match in a single MetaData entry.
for ( tag in item$and_tags )
{
if ( tag !in meta$tags )
matched = F;
}
if ( matched )
return T;
}
else if ( item?$or_tags )
{
# For OR tags, only a single tag has to match.
for ( tag in item$or_tags )
{
if ( tag in meta$tags )
return T;
}
}
else if ( item?$pred )
return item$pred(meta);
# This indicates some sort of failure in the query
return F;
}
function matcher(item: QueryItem): bool
{
local err_msg = "";
if ( ! (item?$ip || item?$str || item?$num) )
err_msg = "You must supply one of the $ip, $str, or $num fields to search on";
else if ( (item?$or_tags || item?$and_tags) && item?$pred )
err_msg = "You can't match with both tags and a predicate.";
else if ( item?$or_tags && item?$and_tags )
err_msg = "You can't match with both OR'd together tags and AND'd together tags";
else if ( (item?$str || item?$num) && ! item?$subtype )
err_msg = "You must provide a subtype to matcher or this item doesn't make sense.";
else if ( item?$str && item?$num )
err_msg = "You must only provide $str or $num, not both.";
local meta: MetaData;
if ( err_msg == "" )
{
if ( item?$ip )
{
if ( item$ip in data_store$ip_data )
{
if ( ! item?$and_tags && ! item?$or_tags && ! item?$pred )
return T;
for ( i in data_store$ip_data[item$ip] )
{
meta = data_store$ip_data[item$ip][i];
if ( match_item_with_metadata(item, meta) )
return T;
}
}
}
else if ( item?$str )
{
if ( [item$str, item$subtype] in data_store$string_data )
{
if ( ! item?$and_tags && ! item?$or_tags && ! item?$pred )
return T;
for ( i in data_store$string_data[item$str, item$subtype] )
{
meta = data_store$string_data[item$str, item$subtype][i];
if ( match_item_with_metadata(item, meta) )
return T;
}
}
}
else if ( item?$num )
{
if ( [item$num, item$subtype] in data_store$int_data )
{
if ( ! item?$and_tags && ! item?$or_tags && ! item?$pred )
return T;
for ( i in data_store$int_data[item$num, item$subtype] )
{
meta = data_store$int_data[item$num, item$subtype][i];
if ( match_item_with_metadata(item, meta) )
return T;
}
}
}
if ( updated )
event Intel::updated_item(item);
else
err_msg = "Failed to query intelligence data for some unknown reason.";
event Intel::new_item(item);
}
if ( err_msg != "" )
Log::write(Intel::LOG, [$ts=network_time(), $level="error", $message=fmt(err_msg)]);
return F;
}

3
scripts/base/frameworks/logging/main.bro
View file

@ -60,6 +60,9 @@ export {
## Default rotation interval. Zero disables rotation.
const default_rotation_interval = 0secs &redef;
## Default alarm summary mail interval. Zero disables alarm summary mails.
const default_mail_alarms_interval = 0secs &redef;
## Default naming format for timestamps embedded into filenames.
## Uses a ``strftime()`` style.
const default_rotation_date_format = "%Y-%m-%d-%H-%M-%S" &redef;

8
scripts/base/frameworks/logging/writers/ascii.bro
View file

@ -1,5 +1,13 @@
##! Interface for the ASCII log writer. Redefinable options are available
##! to tweak the output format of ASCII logs.
##!
##! The ASCII writer supports currently one writer-specific filter option via
##! ``config``: setting ``only_single_header_row`` to ``T`` turns the output into
##! into CSV mode where only a single header row with the column names is printed
##! out as meta information. Example filter using this::
##!
##! local my_filter: Log::Filter = [$name = "my-filter", $writer = Log::WRITER_ASCII, $config = table(["only_single_header_row"] = "T")];
##!
module LogAscii;

2
scripts/base/frameworks/notice/actions/pp-alarms.bro
View file

@ -101,7 +101,7 @@ event bro_init()
# This replaces the standard non-pretty-printing filter.
Log::add_filter(Notice::ALARM_LOG,
[$name="alarm-mail", $writer=Log::WRITER_NONE,
$interv=Log::default_rotation_interval,
$interv=Log::default_mail_alarms_interval,
$postprocessor=pp_postprocessor]);
}

16
scripts/base/init-bare.bro
View file

@ -826,7 +826,7 @@ const tcp_storm_interarrival_thresh = 1 sec &redef;
## peer's ACKs. Set to zero to turn off this determination.
##
## .. bro:see:: tcp_max_above_hole_without_any_acks tcp_excessive_data_without_further_acks
const tcp_max_initial_window = 4096;
const tcp_max_initial_window = 4096 &redef;
## If we're not seeing our peer's ACKs, the maximum volume of data above a sequence
## hole that we'll tolerate before assuming that there's been a packet drop and we
@ -834,7 +834,7 @@ const tcp_max_initial_window = 4096;
## up.
##
## .. bro:see:: tcp_max_initial_window tcp_excessive_data_without_further_acks
const tcp_max_above_hole_without_any_acks = 4096;
const tcp_max_above_hole_without_any_acks = 4096 &redef;
## If we've seen this much data without any of it being acked, we give up
## on that connection to avoid memory exhaustion due to buffering all that
@ -843,7 +843,7 @@ const tcp_max_above_hole_without_any_acks = 4096;
## has in fact gone too far, but for now we just make this quite beefy.
##
## .. bro:see:: tcp_max_initial_window tcp_max_above_hole_without_any_acks
const tcp_excessive_data_without_further_acks = 10 * 1024 * 1024;
const tcp_excessive_data_without_further_acks = 10 * 1024 * 1024 &redef;
## For services without an a handler, these sets define originator-side ports that
## still trigger reassembly.
@ -2457,6 +2457,16 @@ type bittorrent_benc_dir: table[string] of bittorrent_benc_value;
## bt_tracker_response_not_ok
type bt_tracker_headers: table[string] of string;
type ModbusCoils: vector of bool;
type ModbusRegisters: vector of count;
type ModbusHeaders: record {
tid: count;
pid: count;
uid: count;
function_code: count;
};
module SOCKS;
export {
## This record is for a SOCKS client or server to provide either a

4
scripts/base/init-default.bro
View file

@ -14,6 +14,7 @@
@load base/utils/patterns
@load base/utils/strings
@load base/utils/thresholds
@load base/utils/urls
# This has some deep interplay between types and BiFs so it's
# loaded in base/init-bare.bro
@ -36,8 +37,11 @@
@load base/protocols/ftp
@load base/protocols/http
@load base/protocols/irc
@load base/protocols/modbus
@load base/protocols/smtp
@load base/protocols/socks
@load base/protocols/ssh
@load base/protocols/ssl
@load base/protocols/syslog
@load base/misc/find-checksum-offloading

57
scripts/base/misc/find-checksum-offloading.bro Normal file
View file

@ -0,0 +1,57 @@
##! Discover cases where the local interface is sniffed and outbound packets
##! have checksum offloading. Load this script to receive a notice if it's
##! likely that checksum offload effects are being seen on a live interface or
##! in a packet trace file.
@load base/frameworks/notice
module ChecksumOffloading;
export {
## The interval which is used for checking packet statistics
## to see if checksum offloading is affecting analysis.
const check_interval = 10secs &redef;
}
# Keep track of how many bad checksums have been seen.
global bad_checksums = 0;
# Track to see if this script is done so that messages aren't created multiple times.
global done = F;
event ChecksumOffloading::check()
{
if ( done )
return;
local pkts_recvd = net_stats()$pkts_recvd;
if ( (bad_checksums*1.0 / net_stats()$pkts_recvd*1.0) > 0.05 )
{
local packet_src = reading_traces() ? "trace file likely has" : "interface is likely receiving";
local message = fmt("Your %s invalid IP checksums, most likely from NIC checksum offloading.", packet_src);
Reporter::warning(message);
done = T;
}
else if ( pkts_recvd < 20 )
{
# Keep scheduling this event until we've seen some lower threshold of
# total packets.
schedule check_interval { ChecksumOffloading::check() };
}
}
event bro_init()
{
schedule check_interval { ChecksumOffloading::check() };
}
event net_weird(name: string)
{
if ( name == "bad_IP_checksum" )
++bad_checksums;
}
event bro_done()
{
event ChecksumOffloading::check();
}

1
scripts/base/protocols/conn/__load__.bro
View file

@ -1,3 +1,4 @@
@load ./main
@load ./contents
@load ./inactivity
@load ./polling

49
scripts/base/protocols/conn/polling.bro Normal file
View file

@ -0,0 +1,49 @@
##! Implements a generic way to poll connections looking for certain features
##! (e.g. monitor bytes transferred). The specific feature of a connection
##! to look for, the polling interval, and the code to execute if the feature
##! is found are all controlled by user-defined callback functions.
module ConnPolling;
export {
## Starts monitoring a given connection.
##
## c: The connection to watch.
##
## callback: A callback function that takes as arguments the monitored
## *connection*, and counter *cnt* that increments each time the
## callback is called. It returns an interval indicating how long
## in the future to schedule an event which will call the
## callback. A negative return interval causes polling to stop.
##
## cnt: The initial value of a counter which gets passed to *callback*.
##
## i: The initial interval at which to schedule the next callback.
## May be ``0secs`` to poll right away.
global watch: function(c: connection,
callback: function(c: connection, cnt: count): interval,
cnt: count, i: interval);
}
event ConnPolling::check(c: connection,
callback: function(c: connection, cnt: count): interval,
cnt: count)
{
if ( ! connection_exists(c$id) )
return;
lookup_connection(c$id); # updates the conn val
local next_interval = callback(c, cnt);
if ( next_interval < 0secs )
return;
watch(c, callback, cnt + 1, next_interval);
}
function watch(c: connection,
callback: function(c: connection, cnt: count): interval,
cnt: count, i: interval)
{
schedule i { ConnPolling::check(c, callback, cnt) };
}

22
scripts/base/protocols/dns/main.bro
View file

@ -59,13 +59,15 @@ export {
## The caching intervals of the associated RRs described by the
## ``answers`` field.
TTLs: vector of interval &log &optional;
## The DNS query was rejected by the server.
rejected: bool &log &default=F;
## This value indicates if this request/response pair is ready to be
## logged.
ready: bool &default=F;
## The total number of resource records in a reply message's answer
## section.
total_answers: count &optional;
total_answers: count &default=0;
## The total number of resource records in a reply message's answer,
## authority, and additional sections.
total_replies: count &optional;
@ -186,10 +188,13 @@ function set_session(c: connection, msg: dns_msg, is_query: bool)
}
}
event dns_message(c: connection, is_orig: bool, msg: dns_msg, len: count) &priority=5
{
set_session(c, msg, is_orig);
}
event DNS::do_reply(c: connection, msg: dns_msg, ans: dns_answer, reply: string) &priority=5
{
set_session(c, msg, F);
if ( ans$answer_type == DNS_ANS )
{
c$dns$AA = msg$AA;
@ -209,7 +214,8 @@ event DNS::do_reply(c: connection, msg: dns_msg, ans: dns_answer, reply: string)
c$dns$TTLs[|c$dns$TTLs|] = ans$TTL;
}
if ( c$dns?$answers && |c$dns$answers| == c$dns$total_answers )
if ( c$dns?$answers && c$dns?$total_answers &&
|c$dns$answers| == c$dns$total_answers )
{
add c$dns_state$finished_answers[c$dns$trans_id];
# Indicate this request/reply pair is ready to be logged.
@ -230,8 +236,6 @@ event DNS::do_reply(c: connection, msg: dns_msg, ans: dns_answer, reply: string)
event dns_request(c: connection, msg: dns_msg, query: string, qtype: count, qclass: count) &priority=5
{
set_session(c, msg, T);
c$dns$RD = msg$RD;
c$dns$TC = msg$TC;
c$dns$qclass = qclass;
@ -321,11 +325,9 @@ event dns_SRV_reply(c: connection, msg: dns_msg, ans: dns_answer) &priority=5
#
# }
event dns_rejected(c: connection, msg: dns_msg,
query: string, qtype: count, qclass: count) &priority=5
event dns_rejected(c: connection, msg: dns_msg, query: string, qtype: count, qclass: count) &priority=5
{
set_session(c, msg, F);
c$dns$rejected = T;
}
event connection_state_remove(c: connection) &priority=-5

1
scripts/base/protocols/ftp/__load__.bro
View file

@ -1,3 +1,4 @@
@load ./utils-commands
@load ./main
@load ./file-extract
@load ./gridftp

121
scripts/base/protocols/ftp/gridftp.bro Normal file
View file

@ -0,0 +1,121 @@
##! A detection script for GridFTP data and control channels.
##!
##! GridFTP control channels are identified by FTP control channels
##! that successfully negotiate the GSSAPI method of an AUTH request
##! and for which the exchange involved an encoded TLS/SSL handshake,
##! indicating the GSI mechanism for GSSAPI was used. This analysis
##! is all supported internally, this script simple adds the "gridftp"
##! label to the *service* field of the control channel's
##! :bro:type:`connection` record.
##!
##! GridFTP data channels are identified by a heuristic that relies on
##! the fact that default settings for GridFTP clients typically
##! mutally authenticate the data channel with TLS/SSL and negotiate a
##! NULL bulk cipher (no encryption). Connections with those
##! attributes are then polled for two minutes with decreasing frequency
##! to check if the transfer sizes are large enough to indicate a
##! GridFTP data channel that would be undesireable to analyze further
##! (e.g. stop TCP reassembly). A side effect is that true connection
##! sizes are not logged, but at the benefit of saving CPU cycles that
##! otherwise go to analyzing the large (and likely benign) connections.
@load ./main
@load base/protocols/conn
@load base/protocols/ssl
@load base/frameworks/notice
module GridFTP;
export {
## Number of bytes transferred before guessing a connection is a
## GridFTP data channel.
const size_threshold = 1073741824 &redef;
## Max number of times to check whether a connection's size exceeds the
## :bro:see:`GridFTP::size_threshold`.
const max_poll_count = 15 &redef;
## Whether to skip further processing of the GridFTP data channel once
## detected, which may help performance.
const skip_data = T &redef;
## Base amount of time between checking whether a GridFTP data connection
## has transferred more than :bro:see:`GridFTP::size_threshold` bytes.
const poll_interval = 1sec &redef;
## The amount of time the base :bro:see:`GridFTP::poll_interval` is
## increased by each poll interval. Can be used to make more frequent
## checks at the start of a connection and gradually slow down.
const poll_interval_increase = 1sec &redef;
## Raised when a GridFTP data channel is detected.
##
## c: The connection pertaining to the GridFTP data channel.
global data_channel_detected: event(c: connection);
## The initial criteria used to determine whether to start polling
## the connection for the :bro:see:`GridFTP::size_threshold` to have
## been exceeded. This is called in a :bro:see:`ssl_established` event
## handler and by default looks for both a client and server certificate
## and for a NULL bulk cipher. One way in which this function could be
## redefined is to make it also consider client/server certificate issuer
## subjects.
##
## c: The connection which may possibly be a GridFTP data channel.
##
## Returns: true if the connection should be further polled for an
## exceeded :bro:see:`GridFTP::size_threshold`, else false.
const data_channel_initial_criteria: function(c: connection): bool &redef;
}
redef record FTP::Info += {
last_auth_requested: string &optional;
};
event ftp_request(c: connection, command: string, arg: string) &priority=4
{
if ( command == "AUTH" && c?$ftp )
c$ftp$last_auth_requested = arg;
}
function size_callback(c: connection, cnt: count): interval
{
if ( c$orig$size > size_threshold || c$resp$size > size_threshold )
{
add c$service["gridftp-data"];
event GridFTP::data_channel_detected(c);
if ( skip_data )
skip_further_processing(c$id);
return -1sec;
}
if ( cnt >= max_poll_count )
return -1sec;
return poll_interval + poll_interval_increase * cnt;
}
event ssl_established(c: connection) &priority=5
{
# If an FTP client requests AUTH GSSAPI and later an SSL handshake
# finishes, it's likely a GridFTP control channel, so add service label.
if ( c?$ftp && c$ftp?$last_auth_requested &&
/GSSAPI/ in c$ftp$last_auth_requested )
add c$service["gridftp"];
}
function data_channel_initial_criteria(c: connection): bool
{
return ( c?$ssl && c$ssl?$client_subject && c$ssl?$subject &&
c$ssl?$cipher && /WITH_NULL/ in c$ssl$cipher );
}
event ssl_established(c: connection) &priority=-3
{
# By default GridFTP data channels do mutual authentication and
# negotiate a cipher suite with a NULL bulk cipher.
if ( data_channel_initial_criteria(c) )
ConnPolling::watch(c, size_callback, 0, 0secs);
}

6
scripts/base/protocols/ftp/main.bro
View file

@ -96,11 +96,11 @@ redef record connection += {
};
# Configure DPD
const ports = { 21/tcp } &redef;
redef capture_filters += { ["ftp"] = "port 21" };
const ports = { 21/tcp, 2811/tcp } &redef; # 2811/tcp is GridFTP.
redef capture_filters += { ["ftp"] = "port 21 and port 2811" };
redef dpd_config += { [ANALYZER_FTP] = [$ports = ports] };
redef likely_server_ports += { 21/tcp };
redef likely_server_ports += { 21/tcp, 2811/tcp };
# Establish the variable for tracking expected connections.
global ftp_data_expected: table[addr, port] of Info &create_expire=5mins;

2
scripts/base/protocols/modbus/__load__.bro Normal file
View file

@ -0,0 +1,2 @@
@load ./consts
@load ./main

67
scripts/base/protocols/modbus/consts.bro Normal file
View file

@ -0,0 +1,67 @@
module Modbus;
export {
## Standard defined Modbus function codes.
const function_codes = {
[0x01] = "READ_COILS",
[0x02] = "READ_DISCRETE_INPUTS",
[0x03] = "READ_HOLDING_REGISTERS",
[0x04] = "READ_INPUT_REGISTERS",
[0x05] = "WRITE_SINGLE_COIL",
[0x06] = "WRITE_SINGLE_REGISTER",
[0x07] = "READ_EXCEPTION_STATUS",
[0x08] = "DIAGNOSTICS",
[0x0B] = "GET_COMM_EVENT_COUNTER",
[0x0C] = "GET_COMM_EVENT_LOG",
[0x0F] = "WRITE_MULTIPLE_COILS",
[0x10] = "WRITE_MULTIPLE_REGISTERS",
[0x11] = "REPORT_SLAVE_ID",
[0x14] = "READ_FILE_RECORD",
[0x15] = "WRITE_FILE_RECORD",
[0x16] = "MASK_WRITE_REGISTER",
[0x17] = "READ_WRITE_MULTIPLE_REGISTERS",
[0x18] = "READ_FIFO_QUEUE",
[0x2B] = "ENCAP_INTERFACE_TRANSPORT",
# Machine/vendor/network specific functions
[0x09] = "PROGRAM_484",
[0x0A] = "POLL_484",
[0x0D] = "PROGRAM_584_984",
[0x0E] = "POLL_584_984",
[0x12] = "PROGRAM_884_U84",
[0x13] = "RESET_COMM_LINK_884_U84",
[0x28] = "PROGRAM_CONCEPT",
[0x7D] = "FIRMWARE_REPLACEMENT",
[0x7E] = "PROGRAM_584_984_2",
[0x7F] = "REPORT_LOCAL_ADDRESS",
# Exceptions
[0x81] = "READ_COILS_EXCEPTION",
[0x82] = "READ_DISCRETE_INPUTS_EXCEPTION",
[0x83] = "READ_HOLDING_REGISTERS_EXCEPTION",
[0x84] = "READ_INPUT_REGISTERS_EXCEPTION",
[0x85] = "WRITE_SINGLE_COIL_EXCEPTION",
[0x86] = "WRITE_SINGLE_REGISTER_EXCEPTION",
[0x87] = "READ_EXCEPTION_STATUS_EXCEPTION",
[0x8F] = "WRITE_MULTIPLE_COILS_EXCEPTION",
[0x90] = "WRITE_MULTIPLE_REGISTERS_EXCEPTION",
[0x94] = "READ_FILE_RECORD_EXCEPTION",
[0x95] = "WRITE_FILE_RECORD_EXCEPTION",
[0x96] = "MASK_WRITE_REGISTER_EXCEPTION",
[0x97] = "READ_WRITE_MULTIPLE_REGISTERS_EXCEPTION",
[0x98] = "READ_FIFO_QUEUE_EXCEPTION",
} &default=function(i: count):string { return fmt("unknown-%d", i); } &redef;
const exception_codes = {
[0x01] = "ILLEGAL_FUNCTION",
[0x02] = "ILLEGAL_DATA_ADDRESS",
[0x03] = "ILLEGAL_DATA_VALUE",
[0x04] = "SLAVE_DEVICE_FAILURE",
[0x05] = "ACKNOWLEDGE",
[0x06] = "SLAVE_DEVICE_BUSY",
[0x08] = "MEMORY_PARITY_ERROR",
[0x0A] = "GATEWAY_PATH_UNAVAILABLE",
[0x0B] = "GATEWAY_TARGET_DEVICE_FAILED_TO_RESPOND",
} &default=function(i: count):string { return fmt("unknown-%d", i); } &redef;
}

71
scripts/base/protocols/modbus/main.bro Normal file
View file

@ -0,0 +1,71 @@
##! Base Modbus analysis script.
module Modbus;
@load ./consts
export {
redef enum Log::ID += { LOG };
type Info: record {
## Time of the request.
ts: time &log;
## Unique identifier for the connnection.
uid: string &log;
## Identifier for the connection.
id: conn_id &log;
## The name of the function message that was sent.
func: string &log &optional;
## The exception if the response was a failure.
exception: string &log &optional;
};
## Event that can be handled to access the Modbus record as it is sent on
## to the logging framework.
global log_modbus: event(rec: Info);
}
redef record connection += {
modbus: Info &optional;
};
# Configure DPD and the packet filter.
redef capture_filters += { ["modbus"] = "tcp port 502" };
redef dpd_config += { [ANALYZER_MODBUS] = [$ports = set(502/tcp)] };
redef likely_server_ports += { 502/tcp };
event bro_init() &priority=5
{
Log::create_stream(Modbus::LOG, [$columns=Info, $ev=log_modbus]);
}
event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool) &priority=5
{
if ( ! c?$modbus )
{
c$modbus = [$ts=network_time(), $uid=c$uid, $id=c$id];
}
c$modbus$ts = network_time();
c$modbus$func = function_codes[headers$function_code];
}
event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool) &priority=-5
{
# Only log upon replies.
# Also, don't log now if this is an exception (log in the exception event handler)
if ( ! is_orig && ( headers$function_code <= 0x81 || headers$function_code >= 0x98 ) )
Log::write(LOG, c$modbus);
}
event modbus_exception(c: connection, headers: ModbusHeaders, code: count) &priority=5
{
c$modbus$exception = exception_codes[code];
}
event modbus_exception(c: connection, headers: ModbusHeaders, code: count) &priority=-5
{
Log::write(LOG, c$modbus);
delete c$modbus$exception;
}

35
scripts/base/protocols/ssl/main.bro
View file

@ -30,17 +30,28 @@ export {
issuer_subject: string &log &optional;
## NotValidBefore field value from the server certificate.
not_valid_before: time &log &optional;
## NotValidAfter field value from the serve certificate.
## NotValidAfter field value from the server certificate.
not_valid_after: time &log &optional;
## Last alert that was seen during the connection.
last_alert: string &log &optional;
## Subject of the X.509 certificate offered by the client.
client_subject: string &log &optional;
## Subject of the signer of the X.509 certificate offered by the client.
client_issuer_subject: string &log &optional;
## Full binary server certificate stored in DER format.
cert: string &optional;
## Chain of certificates offered by the server to validate its
## complete signing chain.
cert_chain: vector of string &optional;
## Full binary client certificate stored in DER format.
client_cert: string &optional;
## Chain of certificates offered by the client to validate its
## complete signing chain.
client_cert_chain: vector of string &optional;
## The analyzer ID used for the analyzer instance attached
## to each connection. It is not used for logging since it's a
## meaningless arbitrary number.
@ -107,7 +118,8 @@ redef likely_server_ports += {
function set_session(c: connection)
{
if ( ! c?$ssl )
c$ssl = [$ts=network_time(), $uid=c$uid, $id=c$id, $cert_chain=vector()];
c$ssl = [$ts=network_time(), $uid=c$uid, $id=c$id, $cert_chain=vector(),
$client_cert_chain=vector()];
}
function finish(c: connection)
@ -141,8 +153,24 @@ event x509_certificate(c: connection, is_orig: bool, cert: X509, chain_idx: coun
# We aren't doing anything with client certificates yet.
if ( is_orig )
return;
{
if ( chain_idx == 0 )
{
# Save the primary cert.
c$ssl$client_cert = der_cert;
# Also save other certificate information about the primary cert.
c$ssl$client_subject = cert$subject;
c$ssl$client_issuer_subject = cert$issuer;
}
else
{
# Otherwise, add it to the cert validation chain.
c$ssl$client_cert_chain[|c$ssl$client_cert_chain|] = der_cert;
}
}
else
{
if ( chain_idx == 0 )
{
# Save the primary cert.
@ -160,6 +188,7 @@ event x509_certificate(c: connection, is_orig: bool, cert: X509, chain_idx: coun
c$ssl$cert_chain[|c$ssl$cert_chain|] = der_cert;
}
}
}
event ssl_extension(c: connection, is_orig: bool, code: count, val: string) &priority=5
{

2
scripts/base/utils/patterns.bro
View file

@ -1,5 +1,7 @@
##! Functions for creating and working with patterns.
module GLOBAL;
## Given a pattern as a string with two tildes (~~) contained in it, it will
## return a pattern with string set's elements OR'd together where the
## double-tilde was given (this function only works at or before init time).

25
scripts/base/utils/urls.bro Normal file
View file

@ -0,0 +1,25 @@
## Functions for URL handling.
## A regular expression for matching and extracting URLs.
const url_regex = /^([a-zA-Z\-]{3,5})(:\/\/[^\/?#"'\r\n><]*)([^?#"'\r\n><]*)([^[:blank:]\r\n"'><]*|\??[^"'\r\n><]*)/ &redef;
## Extracts URLs discovered in arbitrary text.
function find_all_urls(s: string): string_set
{
return find_all(s, url_regex);
}
## Extracts URLs discovered in arbitrary text without
## the URL scheme included.
function find_all_urls_without_scheme(s: string): string_set
{
local urls = find_all_urls(s);
local return_urls: set[string] = set();
for ( url in urls )
{
local no_scheme = sub(url, /^([a-zA-Z\-]{3,5})(:\/\/)/, "");
add return_urls[no_scheme];
}
return return_urls;
}

8
scripts/policy/frameworks/intel/__load__.bro Normal file
View file

@ -0,0 +1,8 @@
@load ./conn-established
@load ./dns
@load ./http-host-header
@load ./http-url
@load ./http-user-agents
@load ./ssl
@load ./smtp
@load ./smtp-url-extraction

8
scripts/policy/frameworks/intel/conn-established.bro Normal file
View file

@ -0,0 +1,8 @@
@load base/frameworks/intel
@load ./where-locations
event connection_established(c: connection)
{
Intel::seen([$host=c$id$orig_h, $conn=c, $where=Conn::IN_ORIG]);
Intel::seen([$host=c$id$resp_h, $conn=c, $where=Conn::IN_RESP]);
}

10
scripts/policy/frameworks/intel/dns.bro Normal file
View file

@ -0,0 +1,10 @@
@load base/frameworks/intel
@load ./where-locations
event dns_request(c: connection, msg: dns_msg, query: string, qtype: count, qclass: count)
{
Intel::seen([$str=query,
$str_type=Intel::DOMAIN,
$conn=c,
$where=DNS::IN_REQUEST]);
}

11
scripts/policy/frameworks/intel/http-host-header.bro Normal file
View file

@ -0,0 +1,11 @@
@load base/frameworks/intel
@load ./where-locations
event http_header(c: connection, is_orig: bool, name: string, value: string)
{
if ( is_orig && name == "HOST" )
Intel::seen([$str=value,
$str_type=Intel::DOMAIN,
$conn=c,
$where=HTTP::IN_HOST_HEADER]);
}

12
scripts/policy/frameworks/intel/http-url.bro Normal file
View file

@ -0,0 +1,12 @@
@load base/frameworks/intel
@load base/protocols/http/utils
@load ./where-locations
event http_message_done(c: connection, is_orig: bool, stat: http_message_stat)
{
if ( is_orig && c?$http )
Intel::seen([$str=HTTP::build_url(c$http),
$str_type=Intel::URL,
$conn=c,
$where=HTTP::IN_URL]);
}

12
scripts/policy/frameworks/intel/http-user-agents.bro Normal file
View file

@ -0,0 +1,12 @@
@load base/frameworks/intel
@load ./where-locations
event http_header(c: connection, is_orig: bool, name: string, value: string)
{
if ( is_orig && name == "USER-AGENT" )
Intel::seen([$str=value,
$str_type=Intel::USER_AGENT,
$conn=c,
$where=HTTP::IN_USER_AGENT_HEADER]);
}

15
scripts/policy/frameworks/intel/smtp-url-extraction.bro Normal file
View file

@ -0,0 +1,15 @@
@load base/frameworks/intel
@load base/utils/urls
@load ./where-locations
event mime_segment_data(c: connection, length: count, data: string) &priority=3
{
local urls = find_all_urls_without_scheme(data);
for ( url in urls )
{
Intel::seen([$str=url,
$str_type=Intel::URL,
$conn=c,
$where=SMTP::IN_MESSAGE]);
}
}

71
scripts/policy/frameworks/intel/smtp.bro Normal file
View file

@ -0,0 +1,71 @@
@load base/frameworks/intel
@load base/protocols/smtp
@load ./where-locations
event mime_end_entity(c: connection)
{
if ( c?$smtp )
{
if ( c$smtp?$path )
{
local path = c$smtp$path;
for ( i in path )
{
Intel::seen([$host=path[i],
$conn=c,
$where=SMTP::IN_RECEIVED_HEADER]);
}
}
if ( c$smtp?$user_agent )
Intel::seen([$str=c$smtp$user_agent,
$str_type=Intel::USER_AGENT,
$conn=c,
$where=SMTP::IN_HEADER]);
if ( c$smtp?$x_originating_ip )
Intel::seen([$host=c$smtp$x_originating_ip,
$conn=c,
$where=SMTP::IN_X_ORIGINATING_IP_HEADER]);
if ( c$smtp?$mailfrom )
Intel::seen([$str=c$smtp$mailfrom,
$str_type=Intel::EMAIL,
$conn=c,
$where=SMTP::IN_MAIL_FROM]);
if ( c$smtp?$rcptto )
{
for ( rcptto in c$smtp$rcptto )
{
Intel::seen([$str=rcptto,
$str_type=Intel::EMAIL,
$conn=c,
$where=SMTP::IN_RCPT_TO]);
}
}
if ( c$smtp?$from )
Intel::seen([$str=c$smtp$from,
$str_type=Intel::EMAIL,
$conn=c,
$where=SMTP::IN_FROM]);
if ( c$smtp?$to )
{
for ( email_to in c$smtp$to )
{
Intel::seen([$str=email_to,
$str_type=Intel::EMAIL,
$conn=c,
$where=SMTP::IN_TO]);
}
}
if ( c$smtp?$reply_to )
Intel::seen([$str=c$smtp$reply_to,
$str_type=Intel::EMAIL,
$conn=c,
$where=SMTP::IN_REPLY_TO]);
}
}

34
scripts/policy/frameworks/intel/ssl.bro Normal file
View file

@ -0,0 +1,34 @@
@load base/frameworks/intel
@load base/protocols/ssl
@load ./where-locations
event x509_certificate(c: connection, is_orig: bool, cert: X509, chain_idx: count, chain_len: count, der_cert: string)
{
if ( chain_idx == 0 )
{
if ( /emailAddress=/ in cert$subject )
{
local email = sub(cert$subject, /^.*emailAddress=/, "");
email = sub(email, /,.*$/, "");
Intel::seen([$str=email,
$str_type=Intel::EMAIL,
$conn=c,
$where=(is_orig ? SSL::IN_CLIENT_CERT : SSL::IN_SERVER_CERT)]);
}
Intel::seen([$str=sha1_hash(der_cert),
$str_type=Intel::CERT_HASH,
$conn=c,
$where=(is_orig ? SSL::IN_CLIENT_CERT : SSL::IN_SERVER_CERT)]);
}
}
event ssl_extension(c: connection, is_orig: bool, code: count, val: string)
{
if ( is_orig && SSL::extensions[code] == "server_name" &&
c?$ssl && c$ssl?$server_name )
Intel::seen([$str=c$ssl$server_name,
$str_type=Intel::DOMAIN,
$conn=c,
$where=SSL::IN_SERVER_NAME]);
}

25
scripts/policy/frameworks/intel/where-locations.bro Normal file
View file

@ -0,0 +1,25 @@
@load base/frameworks/intel
export {
redef enum Intel::Where += {
Conn::IN_ORIG,
Conn::IN_RESP,
DNS::IN_REQUEST,
DNS::IN_RESPONSE,
HTTP::IN_HOST_HEADER,
HTTP::IN_USER_AGENT_HEADER,
HTTP::IN_URL,
SMTP::IN_MAIL_FROM,
SMTP::IN_RCPT_TO,
SMTP::IN_FROM,
SMTP::IN_TO,
SMTP::IN_RECEIVED_HEADER,
SMTP::IN_REPLY_TO,
SMTP::IN_X_ORIGINATING_IP_HEADER,
SMTP::IN_MESSAGE,
SSL::IN_SERVER_CERT,
SSL::IN_CLIENT_CERT,
SSL::IN_SERVER_NAME,
SMTP::IN_HEADER,
};
}

4
scripts/policy/integration/collective-intel/README Normal file
View file

@ -0,0 +1,4 @@
The scripts in this module are for deeper integration with the
Collective Intelligence Framework (CIF) since Bro's Intel framework
doesn't natively behave the same as CIF nor does it store and maintain
the same data in all cases.

1
scripts/policy/integration/collective-intel/__load__.bro Normal file
View file

@ -0,0 +1 @@
@load ./main

15
scripts/policy/integration/collective-intel/main.bro Normal file
View file

@ -0,0 +1,15 @@
@load base/frameworks/intel
module Intel;
## These are some fields to add extended compatibility between Bro and the Collective
## Intelligence Framework
redef record Intel::MetaData += {
## Maps to the Impact field in the Collective Intelligence Framework.
cif_impact: string &optional;
## Maps to the Severity field in the Collective Intelligence Framework.
cif_severity: string &optional;
## Maps to the Confidence field in the Collective Intelligence Framework.
cif_confidence: double &optional;
};

13
scripts/policy/protocols/http/detect-MHR.bro
View file

@ -15,6 +15,12 @@ export {
## malware hash registry.
Malware_Hash_Registry_Match
};
## The malware hash registry runs each malware sample through several A/V engines.
## Team Cymru returns a percentage to indicate how many A/V engines flagged the
## sample as malicious. This threshold allows you to require a minimum detection
## rate (default: 50%).
const MHR_threshold = 50 &redef;
}
event log_http(rec: HTTP::Info)
@ -22,10 +28,11 @@ event log_http(rec: HTTP::Info)
if ( rec?$md5 )
{
local hash_domain = fmt("%s.malware.hash.cymru.com", rec$md5);
when ( local addrs = lookup_hostname(hash_domain) )
when ( local MHR_result = lookup_hostname_txt(hash_domain) )
{
# 127.0.0.2 indicates that the md5 sum was found in the MHR.
if ( 127.0.0.2 in addrs )
# Data is returned as "<dateFirstDetected> <detectionRate>"
local MHR_answer = split1(MHR_result, / /);
if ( length(MHR_answer) == 2 && to_count(MHR_answer[2]) >= MHR_threshold )
{
local url = HTTP::build_url_http(rec);
local message = fmt("%s %s %s", rec$id$orig_h, rec$md5, url);

21
scripts/policy/protocols/http/detect-intel.bro
View file

@ -1,21 +0,0 @@
##! Intelligence based HTTP detections. Not yet working!
@load base/protocols/http/main
@load base/protocols/http/utils
@load base/frameworks/intel/main
module HTTP;
event log_http(rec: Info)
{
local url = HTTP::build_url(rec);
local query = [$str=url, $subtype="url", $or_tags=set("malicious", "malware")];
if ( Intel::matcher(query) )
{
local msg = fmt("%s accessed a malicious URL from the intelligence framework", rec$id$orig_h);
NOTICE([$note=Intel::Detection,
$msg=msg,
$sub=HTTP::build_url_http(rec),
$id=rec$id]);
}
}

58
scripts/policy/protocols/modbus/known-masters-slaves.bro Normal file
View file

@ -0,0 +1,58 @@
##! Script for tracking known Modbus masters and slaves.
##!
##! .. todo: This script needs a lot of work. What might be more interesting is to track
##! master/slave relationships based on commands sent and successful (non-exception)
##! responses.
@load base/protocols/modbus
module Known;
export {
redef enum Log::ID += { MODBUS_LOG };
type ModbusDeviceType: enum {
MODBUS_MASTER,
MODBUS_SLAVE,
};
type ModbusInfo: record {
## The time the device was discovered.
ts: time &log;
## The IP address of the host.
host: addr &log;
## The type of device being tracked.
device_type: ModbusDeviceType &log;
};
## The Modbus nodes being tracked.
global modbus_nodes: set[addr, ModbusDeviceType] &create_expire=1day &redef;
## Event that can be handled to access the loggable record as it is sent
## on to the logging framework.
global log_known_modbus: event(rec: ModbusInfo);
}
event bro_init() &priority=5
{
Log::create_stream(Known::MODBUS_LOG, [$columns=ModbusInfo, $ev=log_known_modbus]);
}
event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool)
{
local master = c$id$orig_h;
local slave = c$id$resp_h;
if ( [master, MODBUS_MASTER] !in modbus_nodes )
{
add modbus_nodes[master, MODBUS_MASTER];
Log::write(MODBUS_LOG, [$ts=network_time(), $host=master, $device_type=MODBUS_MASTER]);
}
if ( [slave, MODBUS_SLAVE] !in modbus_nodes )
{
add modbus_nodes[slave, MODBUS_SLAVE];
Log::write(MODBUS_LOG, [$ts=network_time(), $host=slave, $device_type=MODBUS_SLAVE]);
}
}

105
scripts/policy/protocols/modbus/track-memmap.bro Normal file
View file

@ -0,0 +1,105 @@
##! This script tracks the memory map of holding (read/write) registers and logs
##! changes as they are discovered.
##!
##! .. todo: Not all register reads and write functions are being supported yet.
@load base/protocols/modbus
@load base/utils/directions-and-hosts
module Modbus;
export {
redef enum Log::ID += { Modbus::REGISTER_CHANGE_LOG };
## The hosts that should have memory mapping enabled.
const track_memmap: Host = ALL_HOSTS &redef;
type MemmapInfo: record {
## Timestamp for the detected register change
ts: time &log;
## Unique ID for the connection
uid: string &log;
## Connection ID.
id: conn_id &log;
## The device memory offset.
register: count &log;
## The old value stored in the register.
old_val: count &log;
## The new value stored in the register.
new_val: count &log;
## The time delta between when the 'old_val' and 'new_val' were seen.
delta: interval &log;
};
type RegisterValue: record {
last_set: time;
value: count;
};
## Indexed on the device register value and yielding the register value.
type Registers: table[count] of RegisterValue;
## The memory map of slaves is tracked with this variable.
global device_registers: table[addr] of Registers;
## This event is generated every time a register is seen to be different than
## it was previously seen to be.
global changed_register: event(c: connection, register: count, old_val: count, new_val: count, delta: interval);
}
redef record Modbus::Info += {
track_address: count &default=0;
};
event bro_init() &priority=5
{
Log::create_stream(Modbus::REGISTER_CHANGE_LOG, [$columns=MemmapInfo]);
}
event modbus_read_holding_registers_request(c: connection, headers: ModbusHeaders, start_address: count, quantity: count)
{
c$modbus$track_address = start_address+1;
}
event modbus_read_holding_registers_response(c: connection, headers: ModbusHeaders, registers: ModbusRegisters)
{
local slave = c$id$resp_h;
if ( ! addr_matches_host(slave, track_memmap ) )
return;
if ( slave !in device_registers )
device_registers[slave] = table();
local slave_regs = device_registers[slave];
for ( i in registers )
{
if ( c$modbus$track_address in slave_regs )
{
if ( slave_regs[c$modbus$track_address]$value != registers[i] )
{
local delta = network_time() - slave_regs[c$modbus$track_address]$last_set;
event Modbus::changed_register(c, c$modbus$track_address,
slave_regs[c$modbus$track_address]$value, registers[i],
delta);
slave_regs[c$modbus$track_address]$last_set = network_time();
slave_regs[c$modbus$track_address]$value = registers[i];
}
}
else
{
local tmp_reg: RegisterValue = [$last_set=network_time(), $value=registers[i]];
slave_regs[c$modbus$track_address] = tmp_reg;
}
++c$modbus$track_address;
}
}
event Modbus::changed_register(c: connection, register: count, old_val: count, new_val: count, delta: interval)
{
local rec: MemmapInfo = [$ts=network_time(), $uid=c$uid, $id=c$id,
$register=register, $old_val=old_val, $new_val=new_val, $delta=delta];
Log::write(REGISTER_CHANGE_LOG, rec);
}

15
scripts/test-all-policy.bro
View file

@ -14,6 +14,16 @@
# @load frameworks/control/controller.bro
@load frameworks/dpd/detect-protocols.bro
@load frameworks/dpd/packet-segment-logging.bro
@load frameworks/intel/__load__.bro
@load frameworks/intel/conn-established.bro
@load frameworks/intel/dns.bro
@load frameworks/intel/http-host-header.bro
@load frameworks/intel/http-url.bro
@load frameworks/intel/http-user-agents.bro
@load frameworks/intel/smtp-url-extraction.bro
@load frameworks/intel/smtp.bro
@load frameworks/intel/ssl.bro
@load frameworks/intel/where-locations.bro
@load frameworks/metrics/conn-example.bro
@load frameworks/metrics/http-example.bro
@load frameworks/metrics/ssl-example.bro
@ -22,6 +32,8 @@
@load integration/barnyard2/__load__.bro
@load integration/barnyard2/main.bro
@load integration/barnyard2/types.bro
@load integration/collective-intel/__load__.bro
@load integration/collective-intel/main.bro
@load misc/analysis-groups.bro
@load misc/capture-loss.bro
@load misc/loaded-scripts.bro
@ -35,7 +47,6 @@
@load protocols/dns/detect-external-names.bro
@load protocols/ftp/detect.bro
@load protocols/ftp/software.bro
@load protocols/http/detect-intel.bro
@load protocols/http/detect-MHR.bro
@load protocols/http/detect-sqli.bro
@load protocols/http/detect-webapps.bro
@ -44,6 +55,8 @@
@load protocols/http/software.bro
@load protocols/http/var-extraction-cookies.bro
@load protocols/http/var-extraction-uri.bro
@load protocols/modbus/known-masters-slaves.bro
@load protocols/modbus/track-memmap.bro
@load protocols/smtp/blocklists.bro
@load protocols/smtp/detect-suspicious-orig.bro
@load protocols/smtp/software.bro

5
src/Analyzer.cc
View file

@ -28,6 +28,7 @@
#include "DCE_RPC.h"
#include "Gnutella.h"
#include "Ident.h"
#include "Modbus.h"
#include "NCP.h"
#include "NetbiosSSN.h"
#include "SMB.h"
@ -129,6 +130,9 @@ const Analyzer::Config Analyzer::analyzer_configs[] = {
{ AnalyzerTag::SYSLOG_BINPAC, "SYSLOG_BINPAC",
Syslog_Analyzer_binpac::InstantiateAnalyzer,
Syslog_Analyzer_binpac::Available, 0, false },
{ AnalyzerTag::Modbus, "MODBUS",
ModbusTCP_Analyzer::InstantiateAnalyzer,
ModbusTCP_Analyzer::Available, 0, false },
{ AnalyzerTag::AYIYA, "AYIYA",
AYIYA_Analyzer::InstantiateAnalyzer,
@ -171,6 +175,7 @@ const Analyzer::Config Analyzer::analyzer_configs[] = {
{ AnalyzerTag::Contents_SMB, "CONTENTS_SMB", 0, 0, 0, false },
{ AnalyzerTag::Contents_RPC, "CONTENTS_RPC", 0, 0, 0, false },
{ AnalyzerTag::Contents_NFS, "CONTENTS_NFS", 0, 0, 0, false },
{ AnalyzerTag::FTP_ADAT, "FTP_ADAT", 0, 0, 0, false },
};
AnalyzerTimer::~AnalyzerTimer()

2
src/AnalyzerTags.h
View file

@ -32,6 +32,7 @@ namespace AnalyzerTag {
// Application-layer analyzers, binpac-generated.
DHCP_BINPAC, DNS_TCP_BINPAC, DNS_UDP_BINPAC,
HTTP_BINPAC, SSL, SYSLOG_BINPAC,
Modbus,
// Decapsulation analyzers.
AYIYA,
@ -46,6 +47,7 @@ namespace AnalyzerTag {
Contents, ContentLine, NVT, Zip, Contents_DNS, Contents_NCP,
Contents_NetbiosSSN, Contents_Rlogin, Contents_Rsh,
Contents_DCE_RPC, Contents_SMB, Contents_RPC, Contents_NFS,
FTP_ADAT,
// End-marker.
LastAnalyzer
};

8
src/Attr.cc
View file

@ -71,7 +71,9 @@ void Attr::DescribeReST(ODesc* d) const
else if ( expr->Type()->Tag() == TYPE_FUNC )
{
d->Add(":bro:type:`func`");
d->Add(":bro:type:`");
d->Add(expr->Type()->AsFuncType()->FlavorString());
d->Add("`");
}
else
@ -401,13 +403,13 @@ void Attributes::CheckAttr(Attr* a)
case ATTR_GROUP:
if ( type->Tag() != TYPE_FUNC ||
! type->AsFuncType()->IsEvent() )
type->AsFuncType()->Flavor() != FUNC_FLAVOR_EVENT )
Error("&group only applicable to events");
break;
case ATTR_ERROR_HANDLER:
if ( type->Tag() != TYPE_FUNC ||
! type->AsFuncType()->IsEvent() )
type->AsFuncType()->Flavor() != FUNC_FLAVOR_EVENT )
Error("&error_handler only applicable to events");
break;

1
src/BroDoc.cc
View file

@ -271,6 +271,7 @@ void BroDoc::WriteInterface(const char* heading, char underline,
WriteBroDocObjList(state_vars, isPublic, "State Variables", sub, isShort);
WriteBroDocObjList(types, isPublic, "Types", sub, isShort);
WriteBroDocObjList(events, isPublic, "Events", sub, isShort);
WriteBroDocObjList(hooks, isPublic, "Hooks", sub, isShort);
WriteBroDocObjList(functions, isPublic, "Functions", sub, isShort);
WriteBroDocObjList(redefs, isPublic, "Redefinitions", sub, isShort);
}

26
src/BroDoc.h
View file

@ -179,6 +179,30 @@ public:
all.push_back(o);
}
/**
* Schedules documentation of a hook declared by the script.
* @param o A pointer to a BroDocObj which contains the internal
* Bro language representation of the script hook and
* also any associated comments about it.
*/
void AddHook(const BroDocObj* o)
{
hooks.push_back(o);
all.push_back(o);
}
/**
* Schedules documentation of a hook handler declared by the script.
* @param o A pointer to a BroDocObj which contains the internal
* Bro language representation of the script hook handler and
* also any associated comments about it.
*/
void AddHookHandler(const BroDocObj* o)
{
hook_handlers.push_back(o);
all.push_back(o);
}
/**
* Schedules documentation of a function declared by the script.
* @param o A pointer to a BroDocObj which contains the internal
@ -241,6 +265,8 @@ protected:
BroDocObjList notices;
BroDocObjList events;
BroDocObjList event_handlers;
BroDocObjList hooks;
BroDocObjList hook_handlers;
BroDocObjMap functions;
BroDocObjList redefs;

6
src/BroString.h
View file

@ -34,11 +34,15 @@ public:
typedef IdxVec::iterator IdxVecIt;
typedef IdxVec::const_iterator IdxVecCIt;
BroString(int arg_final_NUL, byte_vec str, int arg_n);
// Constructors creating internal copies of the data passed in.
BroString(const u_char* str, int arg_n, int add_NUL);
BroString(const char* str);
BroString(const string& str);
BroString(const BroString& bs);
// Constructor that takes owernship of the vector passed in.
BroString(int arg_final_NUL, byte_vec str, int arg_n);
BroString();
~BroString() { Reset(); }

8
src/CMakeLists.txt
View file

@ -4,6 +4,7 @@ include_directories(BEFORE
)
configure_file(version.c.in ${CMAKE_CURRENT_BINARY_DIR}/version.c)
configure_file(util-config.h.in ${CMAKE_CURRENT_BINARY_DIR}/util-config.h)
# This creates a custom command to transform a bison output file (inFile)
# into outFile in order to avoid symbol conflicts:
@ -215,6 +216,8 @@ binpac_target(ssl.pac
ssl-defs.pac ssl-protocol.pac ssl-analyzer.pac)
binpac_target(syslog.pac
syslog-protocol.pac syslog-analyzer.pac)
binpac_target(modbus.pac
modbus-protocol.pac modbus-analyzer.pac)
########################################################################
## bro target
@ -345,6 +348,7 @@ set(bro_SRCS
Reporter.cc
Login.cc
MIME.cc
Modbus.cc
NCP.cc
NFA.cc
NFS.cc
@ -446,10 +450,6 @@ set(bro_SRCS
collect_headers(bro_HEADERS ${bro_SRCS})
add_definitions(-DBRO_SCRIPT_INSTALL_PATH="${BRO_SCRIPT_INSTALL_PATH}")
add_definitions(-DBRO_SCRIPT_SOURCE_PATH="${BRO_SCRIPT_SOURCE_PATH}")
add_definitions(-DBRO_BUILD_PATH="${CMAKE_CURRENT_BINARY_DIR}")
add_executable(bro ${bro_SRCS} ${bro_HEADERS})
target_link_libraries(bro ${brodeps} ${CMAKE_THREAD_LIBS_INIT})

127
src/DNS_Mgr.cc
View file

@ -46,13 +46,15 @@ extern int select(int, fd_set *, fd_set *, fd_set *, struct timeval *);
class DNS_Mgr_Request {
public:
DNS_Mgr_Request(const char* h, int af) { host = copy_string(h); fam = af; }
DNS_Mgr_Request(const char* h, int af, bool is_txt)
{ host = copy_string(h); fam = af; qtype = is_txt ? 16 : 0; }
DNS_Mgr_Request(const IPAddr& a) { addr = a; host = 0; fam = 0; }
~DNS_Mgr_Request() { delete [] host; }
// Returns nil if this was an address request.
const char* ReqHost() const { return host; }
const IPAddr& ReqAddr() const { return addr; }
const bool ReqIsTxt() const { return qtype == 16; }
int MakeRequest(nb_dns_info* nb_dns);
int RequestPending() const { return request_pending; }
@ -62,6 +64,7 @@ public:
protected:
char* host; // if non-nil, this is a host request
int fam; // address family query type for host requests
int qtype; // Query type
IPAddr addr;
int request_pending;
};
@ -75,7 +78,7 @@ int DNS_Mgr_Request::MakeRequest(nb_dns_info* nb_dns)
char err[NB_DNS_ERRSIZE];
if ( host )
return nb_dns_host_request2(nb_dns, host, fam, (void*) this, err) >= 0;
return nb_dns_host_request2(nb_dns, host, fam, qtype, (void*) this, err) >= 0;
else
{
const uint32* bytes;
@ -475,8 +478,8 @@ TableVal* DNS_Mgr::LookupHost(const char* name)
// Not found, or priming.
switch ( mode ) {
case DNS_PRIME:
requests.append(new DNS_Mgr_Request(name, AF_INET));
requests.append(new DNS_Mgr_Request(name, AF_INET6));
requests.append(new DNS_Mgr_Request(name, AF_INET, false));
requests.append(new DNS_Mgr_Request(name, AF_INET6, false));
return empty_addr_set();
case DNS_FORCE:
@ -484,8 +487,8 @@ TableVal* DNS_Mgr::LookupHost(const char* name)
return 0;
case DNS_DEFAULT:
requests.append(new DNS_Mgr_Request(name, AF_INET));
requests.append(new DNS_Mgr_Request(name, AF_INET6));
requests.append(new DNS_Mgr_Request(name, AF_INET, false));
requests.append(new DNS_Mgr_Request(name, AF_INET6, false));
Resolve();
return LookupHost(name);
@ -636,6 +639,7 @@ int DNS_Mgr::Save()
Save(f, host_mappings);
Save(f, addr_mappings);
// Save(f, text_mappings); // We don't save the TXT mappings (yet?).
fclose(f);
@ -704,15 +708,23 @@ void DNS_Mgr::AddResult(DNS_Mgr_Request* dr, struct nb_dns_result* r)
new_dm = new DNS_Mapping(dr->ReqHost(), h, ttl);
prev_dm = 0;
if ( dr->ReqIsTxt() )
{
TextMap::iterator it = text_mappings.find(dr->ReqHost());
if ( it == text_mappings.end() )
text_mappings[dr->ReqHost()] = new_dm;
}
else
{
HostMap::iterator it = host_mappings.find(dr->ReqHost());
if ( it == host_mappings.end() )
{
host_mappings[dr->ReqHost()].first =
new_dm->Type() == AF_INET ? new_dm : 0;
host_mappings[dr->ReqHost()].second =
new_dm->Type() == AF_INET ? 0 : new_dm;
}
else
{
if ( new_dm->Type() == AF_INET )
@ -739,6 +751,7 @@ void DNS_Mgr::AddResult(DNS_Mgr_Request* dr, struct nb_dns_result* r)
++keep_prev;
}
}
}
else
{
new_dm = new DNS_Mapping(dr->ReqAddr(), h, ttl);
@ -928,7 +941,10 @@ TableVal* DNS_Mgr::LookupNameInCache(string name)
{
HostMap::iterator it = dns_mgr->host_mappings.find(name);
if ( it == dns_mgr->host_mappings.end() )
{
it = dns_mgr->host_mappings.begin();
return 0;
}
DNS_Mapping* d4 = it->second.first;
DNS_Mapping* d6 = it->second.second;
@ -951,6 +967,26 @@ TableVal* DNS_Mgr::LookupNameInCache(string name)
return tv6;
}
const char* DNS_Mgr::LookupTextInCache(string name)
{
TextMap::iterator it = dns_mgr->text_mappings.find(name);
if ( it == dns_mgr->text_mappings.end() )
return 0;
DNS_Mapping* d = it->second;
if ( d->Expired() )
{
dns_mgr->text_mappings.erase(it);
delete d;
return 0;
}
// The escapes in the following strings are to avoid having it
// interpreted as a trigraph sequence.
return d->names ? d->names[0] : "<\?\?\?>";
}
void DNS_Mgr::AsyncLookupAddr(const IPAddr& host, LookupCallback* callback)
{
if ( ! did_init )
@ -1020,6 +1056,35 @@ void DNS_Mgr::AsyncLookupName(string name, LookupCallback* callback)
IssueAsyncRequests();
}
void DNS_Mgr::AsyncLookupNameText(string name, LookupCallback* callback)
{
if ( ! did_init )
Init();
// Do we already know the answer?
TableVal* addrs;
AsyncRequest* req = 0;
// Have we already a request waiting for this host?
AsyncRequestTextMap::iterator i = asyncs_texts.find(name);
if ( i != asyncs_texts.end() )
req = i->second;
else
{
// A new one.
req = new AsyncRequest;
req->name = name;
req->is_txt = true;
asyncs_queued.push_back(req);
asyncs_texts.insert(AsyncRequestTextMap::value_type(name, req));
}
req->callbacks.push_back(callback);
IssueAsyncRequests();
}
void DNS_Mgr::IssueAsyncRequests()
{
while ( asyncs_queued.size() && asyncs_pending < MAX_PENDING_REQUESTS )
@ -1036,8 +1101,9 @@ void DNS_Mgr::IssueAsyncRequests()
dr = new DNS_Mgr_Request(req->host);
else
{
dr = new DNS_Mgr_Request(req->name.c_str(), AF_INET);
dr6 = new DNS_Mgr_Request(req->name.c_str(), AF_INET6);
dr = new DNS_Mgr_Request(req->name.c_str(), AF_INET, req->is_txt);
if ( ! req->is_txt )
dr6 = new DNS_Mgr_Request(req->name.c_str(), AF_INET6, req->is_txt);
}
if ( ! dr->MakeRequest(nb_dns) )
@ -1109,6 +1175,38 @@ void DNS_Mgr::CheckAsyncAddrRequest(const IPAddr& addr, bool timeout)
}
void DNS_Mgr::CheckAsyncTextRequest(const char* host, bool timeout)
{
// Note that this code is a mirror of that for CheckAsyncAddrRequest.
AsyncRequestTextMap::iterator i = asyncs_texts.find(host);
if ( i != asyncs_texts.end() )
{
const char* name = LookupTextInCache(host);
if ( name )
{
++successful;
i->second->Resolved(name);
}
else if ( timeout )
{
AsyncRequestTextMap::iterator it = asyncs_texts.begin();
++failed;
i->second->Timeout();
}
else
return;
asyncs_texts.erase(i);
--asyncs_pending;
// Don't delete the request. That will be done once it
// eventually times out.
}
}
void DNS_Mgr::CheckAsyncHostRequest(const char* host, bool timeout)
{
// Note that this code is a mirror of that for CheckAsyncAddrRequest.
@ -1157,8 +1255,12 @@ void DNS_Mgr::Flush()
for ( AddrMap::iterator it2 = addr_mappings.begin(); it2 != addr_mappings.end(); ++it2 )
delete it2->second;
for ( TextMap::iterator it3 = text_mappings.begin(); it3 != text_mappings.end(); ++it3 )
delete it3->second;
host_mappings.clear();
addr_mappings.clear();
text_mappings.clear();
}
void DNS_Mgr::Process()
@ -1177,6 +1279,8 @@ void DNS_Mgr::DoProcess(bool flush)
if ( req->IsAddrReq() )
CheckAsyncAddrRequest(req->host, true);
else if ( req->is_txt )
CheckAsyncTextRequest(req->name.c_str(), true);
else
CheckAsyncHostRequest(req->name.c_str(), true);
@ -1184,7 +1288,7 @@ void DNS_Mgr::DoProcess(bool flush)
delete req;
}
if ( asyncs_addrs.size() == 0 && asyncs_names.size() == 0 )
if ( asyncs_addrs.size() == 0 && asyncs_names.size() == 0 && asyncs_texts.size() == 0 )
return;
if ( AnswerAvailable(0) <= 0 )
@ -1217,6 +1321,8 @@ void DNS_Mgr::DoProcess(bool flush)
if ( ! dr->ReqHost() )
CheckAsyncAddrRequest(dr->ReqAddr(), true);
else if ( dr->ReqIsTxt() )
CheckAsyncTextRequest(dr->ReqHost(), do_host_timeout);
else
CheckAsyncHostRequest(dr->ReqHost(), do_host_timeout);
@ -1271,5 +1377,6 @@ void DNS_Mgr::GetStats(Stats* stats)
stats->pending = asyncs_pending;
stats->cached_hosts = host_mappings.size();
stats->cached_addresses = addr_mappings.size();
stats->cached_texts = text_mappings.size();
}

12
src/DNS_Mgr.h
View file

@ -63,6 +63,7 @@ public:
const char* LookupAddrInCache(const IPAddr& addr);
TableVal* LookupNameInCache(string name);
const char* LookupTextInCache(string name);
// Support for async lookups.
class LookupCallback {
@ -77,6 +78,7 @@ public:
void AsyncLookupAddr(const IPAddr& host, LookupCallback* callback);
void AsyncLookupName(string name, LookupCallback* callback);
void AsyncLookupNameText(string name, LookupCallback* callback);
struct Stats {
unsigned long requests; // These count only async requests.
@ -85,6 +87,7 @@ public:
unsigned long pending;
unsigned long cached_hosts;
unsigned long cached_addresses;
unsigned long cached_texts;
};
void GetStats(Stats* stats);
@ -106,6 +109,7 @@ protected:
typedef map<string, pair<DNS_Mapping*, DNS_Mapping*> > HostMap;
typedef map<IPAddr, DNS_Mapping*> AddrMap;
typedef map<string, DNS_Mapping*> TextMap;
void LoadCache(FILE* f);
void Save(FILE* f, const AddrMap& m);
void Save(FILE* f, const HostMap& m);
@ -122,6 +126,7 @@ protected:
// requested.
void CheckAsyncAddrRequest(const IPAddr& addr, bool timeout);
void CheckAsyncHostRequest(const char* host, bool timeout);
void CheckAsyncTextRequest(const char* host, bool timeout);
// Process outstanding requests.
void DoProcess(bool flush);
@ -138,6 +143,7 @@ protected:
HostMap host_mappings;
AddrMap addr_mappings;
TextMap text_mappings;
DNS_mgr_request_list requests;
@ -165,8 +171,11 @@ protected:
double time;
IPAddr host;
string name;
bool is_txt;
CallbackList callbacks;
AsyncRequest() : time(0.0), is_txt(false) { }
bool IsAddrReq() const { return name.length() == 0; }
void Resolved(const char* name)
@ -210,6 +219,9 @@ protected:
typedef map<string, AsyncRequest*> AsyncRequestNameMap;
AsyncRequestNameMap asyncs_names;
typedef map<string, AsyncRequest*> AsyncRequestTextMap;
AsyncRequestTextMap asyncs_texts;
typedef list<AsyncRequest*> QueuedList;
QueuedList asyncs_queued;

29
src/Expr.cc
View file

@ -4374,7 +4374,7 @@ bool InExpr::DoUnserialize(UnserialInfo* info)
return true;
}
CallExpr::CallExpr(Expr* arg_func, ListExpr* arg_args)
CallExpr::CallExpr(Expr* arg_func, ListExpr* arg_args, bool in_hook)
: Expr(EXPR_CALL)
{
func = arg_func;
@ -4402,8 +4402,33 @@ CallExpr::CallExpr(Expr* arg_func, ListExpr* arg_args)
if ( ! yield )
{
Error("event called in expression");
switch ( func_type->AsFuncType()->Flavor() ) {
case FUNC_FLAVOR_FUNCTION:
Error("function has no yield type");
SetError();
break;
case FUNC_FLAVOR_EVENT:
Error("event called in expression, use event statement instead");
SetError();
break;
case FUNC_FLAVOR_HOOK:
// It's fine to not have a yield if it's known that the call
// is being done from a hook statement.
if ( ! in_hook )
{
Error("hook called in expression, use hook statement instead");
SetError();
}
break;
default:
Error("invalid function flavor");
SetError();
break;
}
}
else
SetType(yield->Ref());

2
src/Expr.h
View file

@ -959,7 +959,7 @@ protected:
class CallExpr : public Expr {
public:
CallExpr(Expr* func, ListExpr* args);
CallExpr(Expr* func, ListExpr* args, bool in_hook = false);
~CallExpr();
Expr* Func() const { return func; }

177
src/FTP.cc
View file

@ -8,6 +8,8 @@
#include "FTP.h"
#include "NVT.h"
#include "Event.h"
#include "SSL.h"
#include "Base64.h"
FTP_Analyzer::FTP_Analyzer(Connection* conn)
: TCP_ApplicationAnalyzer(AnalyzerTag::FTP, conn)
@ -44,6 +46,14 @@ void FTP_Analyzer::Done()
Weird("partial_ftp_request");
}
static uint32 get_reply_code(int len, const char* line)
{
if ( len >= 3 && isdigit(line[0]) && isdigit(line[1]) && isdigit(line[2]) )
return (line[0] - '0') * 100 + (line[1] - '0') * 10 + (line[2] - '0');
else
return 0;
}
void FTP_Analyzer::DeliverStream(int length, const u_char* data, bool orig)
{
TCP_ApplicationAnalyzer::DeliverStream(length, data, orig);
@ -93,16 +103,7 @@ void FTP_Analyzer::DeliverStream(int length, const u_char* data, bool orig)
}
else
{
uint32 reply_code;
if ( length >= 3 &&
isdigit(line[0]) && isdigit(line[1]) && isdigit(line[2]) )
{
reply_code = (line[0] - '0') * 100 +
(line[1] - '0') * 10 +
(line[2] - '0');
}
else
reply_code = 0;
uint32 reply_code = get_reply_code(length, line);
int cont_resp;
@ -143,19 +144,22 @@ void FTP_Analyzer::DeliverStream(int length, const u_char* data, bool orig)
else
line = end_of_line;
if ( auth_requested.size() > 0 &&
(reply_code == 234 || reply_code == 335) )
// Server accepted AUTH requested,
// which means that very likely we
// won't be able to parse the rest
// of the session, and thus we stop
// here.
SetSkip(true);
cont_resp = 0;
}
}
if ( reply_code == 334 && auth_requested.size() > 0 &&
auth_requested == "GSSAPI" )
{
// Server wants to proceed with an ADAT exchange and we
// know how to analyze the GSI mechanism, so attach analyzer
// to look for that.
SSL_Analyzer* ssl = new SSL_Analyzer(Conn());
ssl->AddSupportAnalyzer(new FTP_ADAT_Analyzer(Conn(), true));
ssl->AddSupportAnalyzer(new FTP_ADAT_Analyzer(Conn(), false));
AddChildAnalyzer(ssl);
}
vl->append(new Val(reply_code, TYPE_COUNT));
vl->append(new StringVal(end_of_line - line, line));
vl->append(new Val(cont_resp, TYPE_BOOL));
@ -164,5 +168,140 @@ void FTP_Analyzer::DeliverStream(int length, const u_char* data, bool orig)
}
ConnectionEvent(f, vl);
ForwardStream(length, data, orig);
}
void FTP_ADAT_Analyzer::DeliverStream(int len, const u_char* data, bool orig)
{
// Don't know how to parse anything but the ADAT exchanges of GSI GSSAPI,
// which is basically just TLS/SSL.
if ( ! Parent()->GetTag() == AnalyzerTag::SSL )
{
Parent()->Remove();
return;
}
bool done = false;
const char* line = (const char*) data;
const char* end_of_line = line + len;
BroString* decoded_adat = 0;
if ( orig )
{
int cmd_len;
const char* cmd;
line = skip_whitespace(line, end_of_line);
get_word(len, line, cmd_len, cmd);
if ( strncmp(cmd, "ADAT", cmd_len) == 0 )
{
line = skip_whitespace(line + cmd_len, end_of_line);
StringVal encoded(end_of_line - line, line);
decoded_adat = decode_base64(encoded.AsString());
if ( first_token )
{
// RFC 2743 section 3.1 specifies a framing format for tokens
// that includes an identifier for the mechanism type. The
// framing is supposed to be required for the initial context
// token, but GSI doesn't do that and starts right in on a
// TLS/SSL handshake, so look for that to identify it.
const u_char* msg = decoded_adat->Bytes();
int msg_len = decoded_adat->Len();
// Just check that it looks like a viable TLS/SSL handshake
// record from the first byte (content type of 0x16) and
// that the fourth and fifth bytes indicating the length of
// the record match the length of the decoded data.
if ( msg_len < 5 || msg[0] != 0x16 ||
msg_len - 5 != ntohs(*((uint16*)(msg + 3))) )
{
// Doesn't look like TLS/SSL, so done analyzing.
done = true;
delete decoded_adat;
decoded_adat = 0;
}
}
first_token = false;
}
else if ( strncmp(cmd, "AUTH", cmd_len) == 0 )
// Security state will be reset by a reissued AUTH.
done = true;
}
else
{
uint32 reply_code = get_reply_code(len, line);
switch ( reply_code ) {
case 232:
case 234:
// Indicates security data exchange is complete, but nothing
// more to decode in replies.
done = true;
break;
case 235:
// Security data exchange complete, but may have more to decode
// in the reply (same format at 334 and 335).
done = true;
// Fall-through.
case 334:
case 335:
// Security data exchange still in progress, and there could be data
// to decode in the reply.
line += 3;
if ( len > 3 && line[0] == '-' )
line++;
line = skip_whitespace(line, end_of_line);
if ( end_of_line - line >= 5 && strncmp(line, "ADAT=", 5) == 0 )
{
line += 5;
StringVal encoded(end_of_line - line, line);
decoded_adat = decode_base64(encoded.AsString());
}
break;
case 421:
case 431:
case 500:
case 501:
case 503:
case 535:
// Server isn't going to accept named security mechanism.
// Client has to restart back at the AUTH.
done = true;
break;
case 631:
case 632:
case 633:
// If the server is sending protected replies, the security
// data exchange must have already succeeded. It does have
// encoded data in the reply, but 632 and 633 are also encrypted.
done = true;
break;
default:
break;
}
}
if ( decoded_adat )
{
ForwardStream(decoded_adat->Len(), decoded_adat->Bytes(), orig);
delete decoded_adat;
}
if ( done )
Parent()->Remove();
}

22
src/FTP.h
View file

@ -30,4 +30,26 @@ protected:
string auth_requested; // AUTH method requested
};
/**
* Analyzes security data of ADAT exchanges over FTP control session (RFC 2228).
* Currently only the GSI mechanism of GSSAPI AUTH method is understood.
* The ADAT exchange for GSI is base64 encoded TLS/SSL handshake tokens. This
* analyzer just decodes the tokens and passes them on to the parent, which must
* be an SSL analyzer instance.
*/
class FTP_ADAT_Analyzer : public SupportAnalyzer {
public:
FTP_ADAT_Analyzer(Connection* conn, bool arg_orig)
: SupportAnalyzer(AnalyzerTag::FTP_ADAT, conn, arg_orig),
first_token(true) { }
void DeliverStream(int len, const u_char* data, bool orig);
protected:
// Used by the client-side analyzer to tell if it needs to peek at the
// initial context token and do sanity checking (i.e. does it look like
// a TLS/SSL handshake token).
bool first_token;
};
#endif

14
src/Func.cc
View file

@ -284,8 +284,8 @@ Val* BroFunc::Call(val_list* args, Frame* parent) const
#endif
if ( ! bodies.size() )
{
// Can only happen for events.
assert(IsEvent());
// Can only happen for events and hooks.
assert(Flavor() == FUNC_FLAVOR_EVENT || Flavor() == FUNC_FLAVOR_HOOK);
loop_over_list(*args, i)
Unref((*args)[i]);
return 0 ;
@ -309,7 +309,7 @@ Val* BroFunc::Call(val_list* args, Frame* parent) const
DescribeDebug(&d, args);
g_trace_state.LogTrace("%s called: %s\n",
IsEvent() ? "event" : "function", d.Description());
FType()->FlavorString().c_str(), d.Description());
}
loop_over_list(*args, i)
@ -348,6 +348,12 @@ Val* BroFunc::Call(val_list* args, Frame* parent) const
parent->SetDelayed();
break;
}
if ( flow == FLOW_BREAK && Flavor() == FUNC_FLAVOR_HOOK )
{
// short-circuit execution of remaining hook handler bodies
break;
}
}
// Warn if the function returns something, but we returned from
@ -380,7 +386,7 @@ void BroFunc::AddBody(Stmt* new_body, id_list* new_inits, int new_frame_size,
new_body = AddInits(new_body, new_inits);
if ( ! IsEvent() )
if ( Flavor() == FUNC_FLAVOR_FUNCTION )
{
// For functions, we replace the old body with the new one.
assert(bodies.size() <= 1);

2
src/Func.h
View file

@ -25,7 +25,7 @@ public:
virtual ~Func();
virtual int IsPure() const = 0;
int IsEvent() const { return FType()->IsEvent(); }
function_flavor Flavor() const { return FType()->Flavor(); }
struct Body {
Stmt* stmts;

13
src/ID.cc
View file

@ -107,7 +107,8 @@ void ID::SetVal(Val* v, Opcode op, bool arg_weak_ref)
#endif
if ( type && val &&
type->Tag() == TYPE_FUNC && type->AsFuncType()->IsEvent() )
type->Tag() == TYPE_FUNC &&
type->AsFuncType()->Flavor() == FUNC_FLAVOR_EVENT )
{
EventHandler* handler = event_registry->Lookup(name);
if ( ! handler )
@ -657,11 +658,19 @@ void ID::DescribeReSTShort(ODesc* d) const
break;
case TYPE_FUNC:
d->Add(type->AsFuncType()->IsEvent() ? "event" : type_name(t));
d->Add(type->AsFuncType()->FlavorString());
break;
case TYPE_ENUM:
if ( is_type )
d->Add(type_name(t));
else
d->Add(type->AsEnumType()->Name().c_str());
break;
default:
d->Add(type_name(t));
break;
}
}

4
src/IPAddr.cc
View file

@ -248,10 +248,10 @@ IPPrefix::IPPrefix(const in6_addr& in6, uint8_t length)
prefix.Mask(this->length);
}
IPPrefix::IPPrefix(const IPAddr& addr, uint8_t length)
IPPrefix::IPPrefix(const IPAddr& addr, uint8_t length, bool len_is_v6_relative)
: prefix(addr)
{
if ( prefix.GetFamily() == IPv4 )
if ( prefix.GetFamily() == IPv4 && ! len_is_v6_relative )
{
if ( length > 32 )
reporter->InternalError("Bad IPAddr(v4) IPPrefix length : %d",

44
src/IPAddr.h
View file

@ -342,6 +342,21 @@ public:
return memcmp(&addr1.in6, &addr2.in6, sizeof(in6_addr)) < 0;
}
friend bool operator<=(const IPAddr& addr1, const IPAddr& addr2)
{
return addr1 < addr2 || addr1 == addr2;
}
friend bool operator>=(const IPAddr& addr1, const IPAddr& addr2)
{
return ! ( addr1 < addr2 );
}
friend bool operator>(const IPAddr& addr1, const IPAddr& addr2)
{
return ! ( addr1 <= addr2 );
}
/** Converts the address into the type used internally by the
* inter-thread communication.
*/
@ -481,8 +496,15 @@ public:
* @param addr The IP address.
*
* @param length The prefix length in the range from 0 to 128
*
* @param len_is_v6_relative Whether \a length is relative to the full
* 128 bits of an IPv6 address. If false and \a addr is an IPv4
* address, then \a length is expected to range from 0 to 32. If true
* \a length is expected to range from 0 to 128 even if \a addr is IPv4,
* meaning that the mask is to apply to the IPv4-mapped-IPv6 representation.
*/
IPPrefix(const IPAddr& addr, uint8_t length);
IPPrefix(const IPAddr& addr, uint8_t length,
bool len_is_v6_relative = false);
/**
* Copy constructor.
@ -583,6 +605,11 @@ public:
return net1.Prefix() == net2.Prefix() && net1.Length() == net2.Length();
}
friend bool operator!=(const IPPrefix& net1, const IPPrefix& net2)
{
return ! (net1 == net2);
}
/**
* Comparison operator IP prefixes. This defines a well-defined order for
* IP prefix. However, the order does not necessarily corresponding to their
@ -600,6 +627,21 @@ public:
return false;
}
friend bool operator<=(const IPPrefix& net1, const IPPrefix& net2)
{
return net1 < net2 || net1 == net2;
}
friend bool operator>=(const IPPrefix& net1, const IPPrefix& net2)
{
return ! (net1 < net2 );
}
friend bool operator>(const IPPrefix& net1, const IPPrefix& net2)
{
return ! ( net1 <= net2 );
}
private:
IPAddr prefix; // We store it as an address with the non-prefix bits masked out via Mask().
uint8_t length; // The bit length of the prefix relative to full IPv6 addr.

41
src/Modbus.cc Normal file
View file

@ -0,0 +1,41 @@
#include "Modbus.h"
#include "TCP_Reassembler.h"
ModbusTCP_Analyzer::ModbusTCP_Analyzer(Connection* c)
: TCP_ApplicationAnalyzer(AnalyzerTag::Modbus, c)
{
interp = new binpac::ModbusTCP::ModbusTCP_Conn(this);
}
ModbusTCP_Analyzer::~ModbusTCP_Analyzer()
{
delete interp;
}
void ModbusTCP_Analyzer::Done()
{
TCP_ApplicationAnalyzer::Done();
interp->FlowEOF(true);
interp->FlowEOF(false);
}
void ModbusTCP_Analyzer::DeliverStream(int len, const u_char* data, bool orig)
{
TCP_ApplicationAnalyzer::DeliverStream(len, data, orig);
interp->NewData(orig, data, data + len);
}
void ModbusTCP_Analyzer::Undelivered(int seq, int len, bool orig)
{
TCP_ApplicationAnalyzer::Undelivered(seq, len, orig);
interp->NewGap(orig, len);
}
void ModbusTCP_Analyzer::EndpointEOF(bool is_orig)
{
TCP_ApplicationAnalyzer::EndpointEOF(is_orig);
interp->FlowEOF(is_orig);
}

58
src/Modbus.h Normal file
View file

@ -0,0 +1,58 @@
#ifndef MODBUS_H
#define MODBUS_H
#include "TCP.h"
#include "modbus_pac.h"
class ModbusTCP_Analyzer : public TCP_ApplicationAnalyzer {
public:
ModbusTCP_Analyzer(Connection* conn);
virtual ~ModbusTCP_Analyzer();
virtual void Done();
virtual void DeliverStream(int len, const u_char* data, bool orig);
virtual void Undelivered(int seq, int len, bool orig);
virtual void EndpointEOF(bool is_orig);
static Analyzer* InstantiateAnalyzer(Connection* conn)
{ return new ModbusTCP_Analyzer(conn); }
// Put event names in this function
static bool Available()
{
return modbus_message
| modbus_exception
| modbus_read_coils_request
| modbus_read_coils_response
| modbus_read_discrete_inputs_request
| modbus_read_discrete_inputs_response
| modbus_read_holding_registers_request
| modbus_read_holding_registers_response
| modbus_read_input_registers_request
| modbus_read_input_registers_response
| modbus_write_single_coil_request
| modbus_write_single_coil_response
| modbus_write_single_register_request
| modbus_write_single_register_response
| modbus_write_multiple_coils_request
| modbus_write_multiple_coils_response
| modbus_write_multiple_registers_request
| modbus_write_multiple_registers_response
| modbus_read_file_record_request
| modbus_read_file_record_response
| modbus_write_file_record_request
| modbus_write_file_record_response
| modbus_mask_write_register_request
| modbus_mask_write_register_response
| modbus_read_write_multiple_registers_request
| modbus_read_write_multiple_registers_response
| modbus_read_fifo_queue_request
| modbus_read_fifo_queue_response;
}
protected:
binpac::ModbusTCP::ModbusTCP_Conn* interp;
};
#endif

31
src/PktSrc.cc
View file

@ -219,16 +219,35 @@ void PktSrc::Process()
// Get protocol being carried from the ethernet frame.
protocol = (data[12] << 8) + data[13];
// MPLS carried over the ethernet frame.
if ( protocol == 0x8847 )
have_mpls = true;
// VLAN carried over ethernet frame.
else if ( protocol == 0x8100 )
switch ( protocol )
{
// MPLS carried over the ethernet frame.
case 0x8847:
have_mpls = true;
break;
// VLAN carried over the ethernet frame.
case 0x8100:
data += get_link_header_size(datalink);
data += 4; // Skip the vlan header
pkt_hdr_size = 0;
break;
// PPPoE carried over the ethernet frame.
case 0x8864:
data += get_link_header_size(datalink);
protocol = (data[6] << 8) + data[7];
data += 8; // Skip the PPPoE session and PPP header
pkt_hdr_size = 0;
if ( protocol != 0x0021 && protocol != 0x0057 )
{
// Neither IPv4 nor IPv6.
sessions->Weird("non_ip_packet_in_pppoe_encapsulation", &hdr, data);
data = 0;
return;
}
break;
}
break;

261
src/RuleMatcher.cc
View file

@ -1,4 +1,5 @@
#include <algorithm>
#include <functional>
#include "config.h"
@ -41,6 +42,23 @@ RuleHdrTest::RuleHdrTest(Prot arg_prot, uint32 arg_offset, uint32 arg_size,
level = 0;
}
RuleHdrTest::RuleHdrTest(Prot arg_prot, Comp arg_comp, vector<IPPrefix> arg_v)
{
prot = arg_prot;
offset = 0;
size = 0;
comp = arg_comp;
vals = new maskedvalue_list;
prefix_vals = arg_v;
sibling = 0;
child = 0;
pattern_rules = 0;
pure_rules = 0;
ruleset = new IntSet;
id = ++idcounter;
level = 0;
}
Val* RuleMatcher::BuildRuleStateValue(const Rule* rule,
const RuleEndpointState* state) const
{
@ -63,6 +81,8 @@ RuleHdrTest::RuleHdrTest(RuleHdrTest& h)
loop_over_list(*h.vals, i)
vals->append(new MaskedValue(*(*h.vals)[i]));
prefix_vals = h.prefix_vals;
for ( int j = 0; j < Rule::TYPES; ++j )
{
loop_over_list(h.psets[j], k)
@ -114,6 +134,10 @@ bool RuleHdrTest::operator==(const RuleHdrTest& h)
(*vals)[i]->mask != (*h.vals)[i]->mask )
return false;
for ( size_t i = 0; i < prefix_vals.size(); ++i )
if ( ! (prefix_vals[i] == h.prefix_vals[i]) )
return false;
return true;
}
@ -129,6 +153,9 @@ void RuleHdrTest::PrintDebug()
fprintf(stderr, " 0x%08x/0x%08x",
(*vals)[i]->val, (*vals)[i]->mask);
for ( size_t i = 0; i < prefix_vals.size(); ++i )
fprintf(stderr, " %s", prefix_vals[i].AsString().c_str());
fprintf(stderr, "\n");
}
@ -410,29 +437,129 @@ static inline uint32 getval(const u_char* data, int size)
}
// A line which can be inserted into the macros below for debugging
// fprintf(stderr, "%.06f %08x & %08x %s %08x\n", network_time, v, (mvals)[i]->mask, #op, (mvals)[i]->val);
// Evaluate a value list (matches if at least one value matches).
#define DO_MATCH_OR( mvals, v, op ) \
{ \
loop_over_list((mvals), i) \
{ \
if ( ((v) & (mvals)[i]->mask) op (mvals)[i]->val ) \
goto match; \
} \
goto no_match; \
template <typename FuncT>
static inline bool match_or(const maskedvalue_list& mvals, uint32 v, FuncT comp)
{
loop_over_list(mvals, i)
{
if ( comp(v & mvals[i]->mask, mvals[i]->val) )
return true;
}
return false;
}
// Evaluate a prefix list (matches if at least one value matches).
template <typename FuncT>
static inline bool match_or(const vector<IPPrefix>& prefixes, const IPAddr& a,
FuncT comp)
{
for ( size_t i = 0; i < prefixes.size(); ++i )
{
IPAddr masked(a);
masked.Mask(prefixes[i].LengthIPv6());
if ( comp(masked, prefixes[i].Prefix()) )
return true;
}
return false;
}
// Evaluate a value list (doesn't match if any value matches).
#define DO_MATCH_NOT_AND( mvals, v, op ) \
{ \
loop_over_list((mvals), i) \
{ \
if ( ((v) & (mvals)[i]->mask) op (mvals)[i]->val ) \
goto no_match; \
} \
goto match; \
template <typename FuncT>
static inline bool match_not_and(const maskedvalue_list& mvals, uint32 v,
FuncT comp)
{
loop_over_list(mvals, i)
{
if ( comp(v & mvals[i]->mask, mvals[i]->val) )
return false;
}
return true;
}
// Evaluate a prefix list (doesn't match if any value matches).
template <typename FuncT>
static inline bool match_not_and(const vector<IPPrefix>& prefixes,
const IPAddr& a, FuncT comp)
{
for ( size_t i = 0; i < prefixes.size(); ++i )
{
IPAddr masked(a);
masked.Mask(prefixes[i].LengthIPv6());
if ( comp(masked, prefixes[i].Prefix()) )
return false;
}
return true;
}
static inline bool compare(const maskedvalue_list& mvals, uint32 v,
RuleHdrTest::Comp comp)
{
switch ( comp ) {
case RuleHdrTest::EQ:
return match_or(mvals, v, std::equal_to<uint32>());
break;
case RuleHdrTest::NE:
return match_not_and(mvals, v, std::equal_to<uint32>());
break;
case RuleHdrTest::LT:
return match_or(mvals, v, std::less<uint32>());
break;
case RuleHdrTest::GT:
return match_or(mvals, v, std::greater<uint32>());
break;
case RuleHdrTest::LE:
return match_or(mvals, v, std::less_equal<uint32>());
break;
case RuleHdrTest::GE:
return match_or(mvals, v, std::greater_equal<uint32>());
break;
default:
reporter->InternalError("unknown comparison type");
break;
}
return false;
}
static inline bool compare(const vector<IPPrefix>& prefixes, const IPAddr& a,
RuleHdrTest::Comp comp)
{
switch ( comp ) {
case RuleHdrTest::EQ:
return match_or(prefixes, a, std::equal_to<IPAddr>());
break;
case RuleHdrTest::NE:
return match_not_and(prefixes, a, std::equal_to<IPAddr>());
break;
case RuleHdrTest::LT:
return match_or(prefixes, a, std::less<IPAddr>());
break;
case RuleHdrTest::GT:
return match_or(prefixes, a, std::greater<IPAddr>());
break;
case RuleHdrTest::LE:
return match_or(prefixes, a, std::less_equal<IPAddr>());
break;
case RuleHdrTest::GE:
return match_or(prefixes, a, std::greater_equal<IPAddr>());
break;
default:
reporter->InternalError("unknown comparison type");
break;
}
return false;
}
RuleEndpointState* RuleMatcher::InitEndpoint(Analyzer* analyzer,
@ -492,65 +619,53 @@ RuleEndpointState* RuleMatcher::InitEndpoint(Analyzer* analyzer,
if ( ip )
{
// Get start of transport layer.
const u_char* transport = ip->Payload();
// Descend the RuleHdrTest tree further.
for ( RuleHdrTest* h = hdr_test->child; h;
h = h->sibling )
{
const u_char* data;
bool match = false;
// Evaluate the header test.
switch ( h->prot ) {
case RuleHdrTest::NEXT:
match = compare(*h->vals, ip->NextProto(), h->comp);
break;
case RuleHdrTest::IP:
data = (const u_char*) ip->IP4_Hdr();
if ( ! ip->IP4_Hdr() )
continue;
match = compare(*h->vals, getval((const u_char*)ip->IP4_Hdr() + h->offset, h->size), h->comp);
break;
case RuleHdrTest::IPv6:
if ( ! ip->IP6_Hdr() )
continue;
match = compare(*h->vals, getval((const u_char*)ip->IP6_Hdr() + h->offset, h->size), h->comp);
break;
case RuleHdrTest::ICMP:
case RuleHdrTest::ICMPv6:
case RuleHdrTest::TCP:
case RuleHdrTest::UDP:
data = transport;
match = compare(*h->vals, getval(ip->Payload() + h->offset, h->size), h->comp);
break;
case RuleHdrTest::IPSrc:
match = compare(h->prefix_vals, ip->IPHeaderSrcAddr(), h->comp);
break;
case RuleHdrTest::IPDst:
match = compare(h->prefix_vals, ip->IPHeaderDstAddr(), h->comp);
break;
default:
data = 0;
reporter->InternalError("unknown protocol");
break;
}
// ### data can be nil here if it's an
// IPv6 packet and we're doing an IP test.
if ( ! data )
continue;
// Sorry for the hidden gotos :-)
switch ( h->comp ) {
case RuleHdrTest::EQ:
DO_MATCH_OR(*h->vals, getval(data + h->offset, h->size), ==);
case RuleHdrTest::NE:
DO_MATCH_NOT_AND(*h->vals, getval(data + h->offset, h->size), ==);
case RuleHdrTest::LT:
DO_MATCH_OR(*h->vals, getval(data + h->offset, h->size), <);
case RuleHdrTest::GT:
DO_MATCH_OR(*h->vals, getval(data + h->offset, h->size), >);
case RuleHdrTest::LE:
DO_MATCH_OR(*h->vals, getval(data + h->offset, h->size), <=);
case RuleHdrTest::GE:
DO_MATCH_OR(*h->vals, getval(data + h->offset, h->size), >=);
default:
reporter->InternalError("unknown comparision type");
}
no_match:
continue;
match:
if ( match )
tests.append(h);
}
}
@ -1050,8 +1165,11 @@ static Val* get_bro_val(const char* label)
}
// Converts an atomic Val and appends it to the list
static bool val_to_maskedval(Val* v, maskedvalue_list* append_to)
// Converts an atomic Val and appends it to the list. For subnet types,
// if the prefix_vector param isn't null, appending to that is preferred
// over appending to the masked val list.
static bool val_to_maskedval(Val* v, maskedvalue_list* append_to,
vector<IPPrefix>* prefix_vector)
{
MaskedValue* mval = new MaskedValue;
@ -1070,6 +1188,14 @@ static bool val_to_maskedval(Val* v, maskedvalue_list* append_to)
break;
case TYPE_SUBNET:
{
if ( prefix_vector )
{
prefix_vector->push_back(v->AsSubNet());
delete mval;
return true;
}
else
{
const uint32* n;
uint32 m[4];
@ -1082,13 +1208,12 @@ static bool val_to_maskedval(Val* v, maskedvalue_list* append_to)
bool is_v4_mask = m[0] == 0xffffffff &&
m[1] == m[0] && m[2] == m[0];
if ( v->AsSubNet().Prefix().GetFamily() == IPv4 &&
is_v4_mask )
if ( v->AsSubNet().Prefix().GetFamily() == IPv4 && is_v4_mask )
{
mval->val = ntohl(*n);
mval->mask = m[3];
}
else
{
rules_error("IPv6 subnets not supported");
@ -1096,6 +1221,7 @@ static bool val_to_maskedval(Val* v, maskedvalue_list* append_to)
mval->mask = 0;
}
}
}
break;
default:
@ -1108,7 +1234,8 @@ static bool val_to_maskedval(Val* v, maskedvalue_list* append_to)
return true;
}
void id_to_maskedvallist(const char* id, maskedvalue_list* append_to)
void id_to_maskedvallist(const char* id, maskedvalue_list* append_to,
vector<IPPrefix>* prefix_vector)
{
Val* v = get_bro_val(id);
if ( ! v )
@ -1118,7 +1245,7 @@ void id_to_maskedvallist(const char* id, maskedvalue_list* append_to)
{
val_list* vals = v->AsTableVal()->ConvertToPureList()->Vals();
loop_over_list(*vals, i )
if ( ! val_to_maskedval((*vals)[i], append_to) )
if ( ! val_to_maskedval((*vals)[i], append_to, prefix_vector) )
{
delete_vals(vals);
return;
@ -1128,7 +1255,7 @@ void id_to_maskedvallist(const char* id, maskedvalue_list* append_to)
}
else
val_to_maskedval(v, append_to);
val_to_maskedval(v, append_to, prefix_vector);
}
char* id_to_str(const char* id)

9
src/RuleMatcher.h
View file

@ -2,7 +2,9 @@
#define sigs_h
#include <limits.h>
#include <vector>
#include "IPAddr.h"
#include "BroString.h"
#include "List.h"
#include "RE.h"
@ -59,17 +61,19 @@ declare(PList, BroString);
typedef PList(BroString) bstr_list;
// Get values from Bro's script-level variables.
extern void id_to_maskedvallist(const char* id, maskedvalue_list* append_to);
extern void id_to_maskedvallist(const char* id, maskedvalue_list* append_to,
vector<IPPrefix>* prefix_vector = 0);
extern char* id_to_str(const char* id);
extern uint32 id_to_uint(const char* id);
class RuleHdrTest {
public:
enum Comp { LE, GE, LT, GT, EQ, NE };
enum Prot { NOPROT, IP, ICMP, TCP, UDP };
enum Prot { NOPROT, IP, IPv6, ICMP, ICMPv6, TCP, UDP, NEXT, IPSrc, IPDst };
RuleHdrTest(Prot arg_prot, uint32 arg_offset, uint32 arg_size,
Comp arg_comp, maskedvalue_list* arg_vals);
RuleHdrTest(Prot arg_prot, Comp arg_comp, vector<IPPrefix> arg_v);
~RuleHdrTest();
void PrintDebug();
@ -86,6 +90,7 @@ private:
Prot prot;
Comp comp;
maskedvalue_list* vals;
vector<IPPrefix> prefix_vals; // for use with IPSrc/IPDst comparisons
uint32 offset;
uint32 size;

1
src/SerialTypes.h
View file

@ -165,6 +165,7 @@ SERIAL_STMT(EVENT_BODY_LIST, 16)
SERIAL_STMT(INIT_STMT, 17)
SERIAL_STMT(NULL_STMT, 18)
SERIAL_STMT(WHEN_STMT, 19)
SERIAL_STMT(HOOK_STMT, 20)
#define SERIAL_TYPE(name, val) SERIAL_CONST(name, val, BRO_TYPE)
SERIAL_TYPE(BRO_TYPE, 1)

20
src/Serializer.cc
View file

@ -389,23 +389,35 @@ bool Serializer::UnserializeCall(UnserialInfo* info)
{
if ( info->print )
fprintf(info->print, "%s [%.06f] %s(%s)\n",
functype->IsEvent() ? "Event" : "Function call",
functype->FlavorString().c_str(),
time, name, types ? d.Description() : "<ignored>");
if ( functype->IsEvent() )
switch ( functype->Flavor() ) {
case FUNC_FLAVOR_EVENT:
{
EventHandler* handler = event_registry->Lookup(name);
assert(handler);
if ( ! info->ignore_callbacks )
GotEvent(name, time, handler, args);
break;
}
else
case FUNC_FLAVOR_FUNCTION:
case FUNC_FLAVOR_HOOK:
if ( ! info->ignore_callbacks )
GotFunctionCall(name, time, id->ID_Val()->AsFunc(), args);
break;
default:
reporter->InternalError("invalid function flavor");
break;
}
if ( info->ignore_callbacks )
delete_vals(args);
}
else
delete_vals(args);

2
src/Serializer.h
View file

@ -125,7 +125,7 @@ protected:
// This will be increased whenever there is an incompatible change
// in the data format.
static const uint32 DATA_FORMAT_VERSION = 22;
static const uint32 DATA_FORMAT_VERSION = 23;
ChunkedIO* io;

49
src/Stmt.cc
View file

@ -23,7 +23,7 @@ const char* stmt_name(BroStmtTag t)
"print", "event", "expr", "if", "when", "switch",
"for", "next", "break", "return", "add", "delete",
"list", "bodylist",
"<init>",
"<init>", "hook",
"null",
};
@ -933,6 +933,52 @@ bool EventStmt::DoUnserialize(UnserialInfo* info)
return event_expr != 0;
}
HookStmt::HookStmt(CallExpr* arg_e) : ExprStmt(STMT_HOOK, arg_e)
{
call_expr = arg_e;
}
Val* HookStmt::Exec(Frame* f, stmt_flow_type& flow) const
{
RegisterAccess();
Val* ret = call_expr->Eval(f);
Unref(ret);
flow = FLOW_NEXT;
return 0;
}
TraversalCode HookStmt::Traverse(TraversalCallback* cb) const
{
TraversalCode tc = cb->PreStmt(this);
HANDLE_TC_STMT_PRE(tc);
// call expr is stored in base class's "e" field.
tc = e->Traverse(cb);
HANDLE_TC_STMT_PRE(tc);
tc = cb->PostStmt(this);
HANDLE_TC_STMT_POST(tc);
}
IMPLEMENT_SERIAL(HookStmt, SER_HOOK_STMT);
bool HookStmt::DoSerialize(SerialInfo* info) const
{
DO_SERIALIZE(SER_HOOK_STMT, ExprStmt);
return call_expr->Serialize(info);
}
bool HookStmt::DoUnserialize(UnserialInfo* info)
{
DO_UNSERIALIZE(ExprStmt);
call_expr = (CallExpr*) Expr::Unserialize(info, EXPR_CALL);
return call_expr != 0;
}
ForStmt::ForStmt(id_list* arg_loop_vars, Expr* loop_expr)
: ExprStmt(STMT_FOR, loop_expr)
{
@ -1944,6 +1990,7 @@ int same_stmt(const Stmt* s1, const Stmt* s2)
case STMT_RETURN:
case STMT_EXPR:
case STMT_EVENT:
case STMT_HOOK:
{
const ExprStmt* e1 = (const ExprStmt*) s1;
const ExprStmt* e2 = (const ExprStmt*) s2;

18
src/Stmt.h
View file

@ -286,6 +286,24 @@ protected:
EventExpr* event_expr;
};
class HookStmt : public ExprStmt {
public:
HookStmt(CallExpr* e);
Val* Exec(Frame* f, stmt_flow_type& flow) const;
TraversalCode Traverse(TraversalCallback* cb) const;
protected:
friend class Stmt;
HookStmt() { call_expr = 0; }
DECLARE_SERIAL(HookStmt);
CallExpr* call_expr;
};
class ForStmt : public ExprStmt {
public:
ForStmt(id_list* loop_vars, Expr* loop_expr);

2
src/StmtEnums.h
View file

@ -15,7 +15,7 @@ typedef enum {
STMT_RETURN,
STMT_ADD, STMT_DELETE,
STMT_LIST, STMT_EVENT_BODY_LIST,
STMT_INIT,
STMT_INIT, STMT_HOOK,
STMT_NULL
#define NUM_STMTS (int(STMT_NULL) + 1)
} BroStmtTag;

88
src/Type.cc
View file

@ -651,12 +651,12 @@ bool SetType::DoUnserialize(UnserialInfo* info)
return true;
}
FuncType::FuncType(RecordType* arg_args, BroType* arg_yield, int arg_is_event)
FuncType::FuncType(RecordType* arg_args, BroType* arg_yield, function_flavor arg_flavor)
: BroType(TYPE_FUNC)
{
args = arg_args;
yield = arg_yield;
is_event = arg_is_event;
flavor = arg_flavor;
arg_types = new TypeList();
@ -664,6 +664,25 @@ FuncType::FuncType(RecordType* arg_args, BroType* arg_yield, int arg_is_event)
arg_types->Append(args->FieldType(i)->Ref());
}
string FuncType::FlavorString() const
{
switch ( flavor ) {
case FUNC_FLAVOR_FUNCTION:
return "function";
case FUNC_FLAVOR_EVENT:
return "event";
case FUNC_FLAVOR_HOOK:
return "hook";
default:
reporter->InternalError("Invalid function flavor");
return "invalid_func_flavor";
}
}
FuncType::~FuncType()
{
Unref(arg_types);
@ -698,7 +717,7 @@ void FuncType::Describe(ODesc* d) const
{
if ( d->IsReadable() )
{
d->Add(is_event ? "event" : "function");
d->Add(FlavorString());
d->Add("(");
args->DescribeFields(d);
d->Add(")");
@ -712,7 +731,7 @@ void FuncType::Describe(ODesc* d) const
else
{
d->Add(int(Tag()));
d->Add(is_event);
d->Add(flavor);
d->Add(yield != 0);
args->DescribeFields(d);
if ( yield )
@ -723,7 +742,7 @@ void FuncType::Describe(ODesc* d) const
void FuncType::DescribeReST(ODesc* d) const
{
d->Add(":bro:type:`");
d->Add(is_event ? "event" : "function");
d->Add(FlavorString());
d->Add("`");
d->Add(" (");
args->DescribeFieldsReST(d, true);
@ -755,9 +774,30 @@ bool FuncType::DoSerialize(SerialInfo* info) const
SERIALIZE_OPTIONAL(yield);
int ser_flavor = 0;
switch ( flavor ) {
case FUNC_FLAVOR_FUNCTION:
ser_flavor = 0;
break;
case FUNC_FLAVOR_EVENT:
ser_flavor = 1;
break;
case FUNC_FLAVOR_HOOK:
ser_flavor = 2;
break;
default:
reporter->InternalError("Invalid function flavor serialization");
break;
}
return args->Serialize(info) &&
arg_types->Serialize(info) &&
SERIALIZE(is_event);
SERIALIZE(ser_flavor);
}
bool FuncType::DoUnserialize(UnserialInfo* info)
@ -774,7 +814,27 @@ bool FuncType::DoUnserialize(UnserialInfo* info)
if ( ! arg_types )
return false;
return UNSERIALIZE(&is_event);
int ser_flavor = 0;
if ( ! UNSERIALIZE(&ser_flavor) )
return false;
switch ( ser_flavor ) {
case 0:
flavor = FUNC_FLAVOR_FUNCTION;
break;
case 1:
flavor = FUNC_FLAVOR_EVENT;
break;
case 2:
flavor = FUNC_FLAVOR_HOOK;
break;
default:
reporter->InternalError("Invalid function flavor unserialization");
break;
}
return true;
}
TypeDecl::TypeDecl(BroType* t, const char* i, attr_list* arg_attrs, bool in_record)
@ -1202,9 +1262,10 @@ bool FileType::DoUnserialize(UnserialInfo* info)
return yield != 0;
}
EnumType::EnumType()
EnumType::EnumType(const string& arg_name)
: BroType(TYPE_ENUM)
{
name = arg_name;
counter = 0;
}
@ -1327,6 +1388,13 @@ const char* EnumType::Lookup(bro_int_t value)
return 0;
}
void EnumType::DescribeReST(ODesc* d) const
{
d->Add(":bro:type:`");
d->Add(name.c_str());
d->Add("`");
}
void CommentedEnumType::DescribeReST(ODesc* d) const
{
// create temporary, reverse name map so that enums can be documented
@ -1595,7 +1663,7 @@ int same_type(const BroType* t1, const BroType* t2, int is_init)
const FuncType* ft1 = (const FuncType*) t1;
const FuncType* ft2 = (const FuncType*) t2;
if ( ft1->IsEvent() != ft2->IsEvent() )
if ( ft1->Flavor() != ft2->Flavor() )
return 0;
if ( t1->YieldType() || t2->YieldType() )
@ -1882,7 +1950,7 @@ BroType* merge_types(const BroType* t1, const BroType* t2)
BroType* yield = t1->YieldType() ?
merge_types(t1->YieldType(), t2->YieldType()) : 0;
return new FuncType(args->AsRecordType(), yield, ft1->IsEvent());
return new FuncType(args->AsRecordType(), yield, ft1->Flavor());
}
case TYPE_RECORD:

34
src/Type.h
View file

@ -35,7 +35,11 @@ typedef enum {
#define NUM_TYPES (int(TYPE_ERROR) + 1)
} TypeTag;
typedef enum { FUNC_FLAVOR_FUNCTION, FUNC_FLAVOR_EVENT } function_flavor;
typedef enum {
FUNC_FLAVOR_FUNCTION,
FUNC_FLAVOR_EVENT,
FUNC_FLAVOR_HOOK
} function_flavor;
typedef enum {
TYPE_INTERNAL_VOID,
@ -350,18 +354,19 @@ protected:
class FuncType : public BroType {
public:
FuncType(RecordType* args, BroType* yield, int is_event);
FuncType(RecordType* args, BroType* yield, function_flavor f);
~FuncType();
RecordType* Args() const { return args; }
BroType* YieldType();
void SetYieldType(BroType* arg_yield) { yield = arg_yield; }
int IsEvent() const { return is_event; }
function_flavor Flavor() const { return flavor; }
string FlavorString() const;
// Used to convert a function type to an event type.
void ClearYieldType()
{ Unref(yield); yield = 0; is_event = 1; }
// Used to convert a function type to an event or hook type.
void ClearYieldType(function_flavor arg_flav)
{ Unref(yield); yield = 0; flavor = arg_flav; }
int MatchesIndex(ListExpr*& index) const;
int CheckArgs(const type_list* args) const;
@ -374,14 +379,13 @@ public:
void DescribeReST(ODesc* d) const;
protected:
FuncType() { args = 0; arg_types = 0; yield = 0; return_value = 0; }
FuncType() { args = 0; arg_types = 0; yield = 0; flavor = FUNC_FLAVOR_FUNCTION; }
DECLARE_SERIAL(FuncType)
RecordType* args;
TypeList* arg_types;
BroType* yield;
int is_event;
ID* return_value;
function_flavor flavor;
};
class TypeType : public BroType {
@ -497,7 +501,7 @@ protected:
class EnumType : public BroType {
public:
EnumType();
EnumType(const string& arg_name);
~EnumType();
// The value of this name is next internal counter value, starting
@ -513,7 +517,12 @@ public:
bro_int_t Lookup(const string& module_name, const char* name);
const char* Lookup(bro_int_t value); // Returns 0 if not found
string Name() const { return name; }
void DescribeReST(ODesc* d) const;
protected:
EnumType() { counter = 0; }
DECLARE_SERIAL(EnumType)
virtual void AddNameInternal(const string& module_name,
@ -529,11 +538,14 @@ protected:
// as a flag to prevent mixing of auto-increment and explicit
// enumerator specifications.
bro_int_t counter;
// The name of the enum type is stored for documentation purposes.
string name;
};
class CommentedEnumType: public EnumType {
public:
CommentedEnumType() {}
CommentedEnumType(const string& arg_name) : EnumType(arg_name) {}
~CommentedEnumType();
void DescribeReST(ODesc* d) const;

36
src/Var.cc
View file

@ -171,7 +171,9 @@ static void make_var(ID* id, BroType* t, init_class c, Expr* init,
id->UpdateValAttrs();
if ( t && t->Tag() == TYPE_FUNC && t->AsFuncType()->IsEvent() )
if ( t && t->Tag() == TYPE_FUNC &&
(t->AsFuncType()->Flavor() == FUNC_FLAVOR_EVENT ||
t->AsFuncType()->Flavor() == FUNC_FLAVOR_HOOK) )
{
// For events, add a function value (without any body) here so that
// we can later access the ID even if no implementations have been
@ -258,7 +260,7 @@ void add_type(ID* id, BroType* t, attr_list* attr, int /* is_event */)
case TYPE_FUNC:
tnew = new FuncType(t->AsFuncType()->Args(),
t->AsFuncType()->YieldType(),
t->AsFuncType()->IsEvent());
t->AsFuncType()->Flavor());
break;
default:
SerializationFormat* form = new BinarySerializationFormat();
@ -292,14 +294,14 @@ void add_type(ID* id, BroType* t, attr_list* attr, int /* is_event */)
void begin_func(ID* id, const char* module_name, function_flavor flavor,
int is_redef, FuncType* t)
{
if ( flavor == FUNC_FLAVOR_EVENT )
if ( flavor == FUNC_FLAVOR_EVENT || flavor == FUNC_FLAVOR_HOOK )
{
const BroType* yt = t->YieldType();
if ( yt && yt->Tag() != TYPE_VOID )
id->Error("event cannot yield a value", t);
id->Error("event/hook cannot yield a value", t);
t->ClearYieldType();
t->ClearYieldType(flavor);
}
if ( id->Type() )
@ -313,20 +315,28 @@ void begin_func(ID* id, const char* module_name, function_flavor flavor,
if ( id->HasVal() )
{
int id_is_event = id->ID_Val()->AsFunc()->IsEvent();
function_flavor id_flavor = id->ID_Val()->AsFunc()->Flavor();
if ( id_is_event != (flavor == FUNC_FLAVOR_EVENT) )
id->Error("inconsistency between event and function", t);
if ( id_is_event )
{
if ( id_flavor != flavor )
id->Error("inconsistent function flavor", t);
switch ( id_flavor ) {
case FUNC_FLAVOR_EVENT:
case FUNC_FLAVOR_HOOK:
if ( is_redef )
// Clear out value so it will be replaced.
id->SetVal(0);
}
else
{
break;
case FUNC_FLAVOR_FUNCTION:
if ( ! id->IsRedefinable() )
id->Error("already defined");
break;
default:
reporter->InternalError("invalid function flavor");
break;
}
}
else

76
src/bro.bif
View file

@ -11,6 +11,7 @@
#include <cmath>
#include <sys/stat.h>
#include <cstdio>
#include <time.h>
#include "digest.h"
#include "Reporter.h"
@ -2700,6 +2701,27 @@ function to_port%(s: string%): port
return new PortVal(port, TRANSPORT_UNKNOWN);
%}
## Converts a string of bytes (in network byte order) to a :bro:type:`double`.
##
## s: A string of bytes containing the binary representation of a double value.
##
## Returns: The double value contained in *s*, or 0 if the conversion
## failed.
##
function bytestring_to_double%(s: string%): double
%{
if ( s->Len() != sizeof(double) )
{
builtin_error("bad conversion to double");
return new Val(0.0, TYPE_DOUBLE);
}
// See #908 for a discussion of portability.
double d;
memcpy(&d, s->Bytes(), sizeof(double));
return new Val(ntohd(d), TYPE_DOUBLE);
%}
## Converts a reverse pointer name to an address. For example,
## ``1.0.168.192.in-addr.arpa`` to ``192.168.0.1``.
##
@ -3285,6 +3307,31 @@ function strftime%(fmt: string, d: time%) : string
return new StringVal(buffer);
%}
## Parse a textual representation of a date/time value into a ``time`` type value.
##
## fmt: The format string used to parse the following *d* argument. See ``man strftime``
## for the syntax.
##
## d: The string representing the time.
##
## Returns: The time value calculated from parsing *d* with *fmt*.
function strptime%(fmt: string, d: string%) : time
%{
const time_t timeval = time_t(NULL);
struct tm t = *localtime(&timeval);
if ( strptime(d->CheckString(), fmt->CheckString(), &t) == NULL )
{
reporter->Warning("strptime conversion failed: fmt:%s d:%s", fmt->CheckString(), d->CheckString());
return new Val(0.0, TYPE_TIME);
}
double ret = mktime(&t);
return new Val(ret, TYPE_TIME);
%}
# ===========================================================================
#
# Network Type Processing
@ -3744,6 +3791,35 @@ function lookup_addr%(host: addr%) : string
return 0;
%}
## Issues an asynchronous TEXT DNS lookup and delays the function result.
## This function can therefore only be called inside a ``when`` condition,
## e.g., ``when ( local h = lookup_hostname_txt("www.bro-ids.org") ) { f(h); }``.
##
## host: The hostname to lookup.
##
## Returns: The DNS TXT record associated with *host*.
##
## .. bro:see:: lookup_hostname
function lookup_hostname_txt%(host: string%) : string
%{
// FIXME: Is should be easy to adapt the function to synchronous
// lookups if we're reading a trace.
Trigger* trigger = frame->GetTrigger();
if ( ! trigger)
{
builtin_error("lookup_hostname_txt() can only be called inside a when-condition");
return new StringVal("<error>");
}
frame->SetDelayed();
trigger->Hold();
dns_mgr->AsyncLookupNameText(host->CheckString(),
new LookupHostCallback(trigger, frame->GetCall(), true));
return 0;
%}
## Issues an asynchronous DNS lookup and delays the function result.
## This function can therefore only be called inside a ``when`` condition,
## e.g., ``when ( local h = lookup_hostname("www.bro-ids.org") ) { f(h); }``.

295
src/event.bif
View file

@ -6547,6 +6547,301 @@ event netflow_v5_header%(h: nf_v5_header%);
## .. bro:see:: netflow_v5_record
event netflow_v5_record%(r: nf_v5_record%);
## Generated for any modbus message regardless if the particular function
## is further supported or not.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## is_orig: True if the event is raised for the originator side.
event modbus_message%(c: connection, headers: ModbusHeaders, is_orig: bool%);
## Generated for any modbus exception message.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## code: The exception code.
event modbus_exception%(c: connection, headers: ModbusHeaders, code: count%);
## Generated for a Modbus read coils request.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## start_address: The memory address where of the first coil to be read.
##
## quantity: The number of coils to be read.
event modbus_read_coils_request%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
## Generated for a Modbus read coils response.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## coils: The coil values returned from the device.
event modbus_read_coils_response%(c: connection, headers: ModbusHeaders, coils: ModbusCoils%);
## Generated for a Modbus read discrete inputs request.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## start_address: The memory address of the first coil to be read.
##
## quantity: The number of coils to be read.
event modbus_read_discrete_inputs_request%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
## Generated for a Modbus read discrete inputs response.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## coils: The coil values returned from the device.
event modbus_read_discrete_inputs_response%(c: connection, headers: ModbusHeaders, coils: ModbusCoils%);
## Generated for a Modbus read holding registers request.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## start_address: The memory address of the first register to be read.
##
## quantity: The number of registers to be read.
event modbus_read_holding_registers_request%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
## Generated for a Modbus read holding registers response.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## registers: The register values returned from the device.
event modbus_read_holding_registers_response%(c: connection, headers: ModbusHeaders, registers: ModbusRegisters%);
## Generated for a Modbus read input registers request.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## start_address: The memory address of the first register to be read.
##
## quantity: The number of registers to be read.
event modbus_read_input_registers_request%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
## Generated for a Modbus read input registers response.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## registers: The register values returned from the device.
event modbus_read_input_registers_response%(c: connection, headers: ModbusHeaders, registers: ModbusRegisters%);
## Generated for a Modbus write single coil request.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## address: The memory address of the coil to be written.
##
## value: The value to be written to the coil.
event modbus_write_single_coil_request%(c: connection, headers: ModbusHeaders, address: count, value: bool%);
## Generated for a Modbus write single coil response.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## address: The memory address of the coil that was written.
##
## value: The value that was written to the coil.
event modbus_write_single_coil_response%(c: connection, headers: ModbusHeaders, address: count, value: bool%);
## Generated for a Modbus write single register request.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## address: The memory address of the register to be written.
##
## value: The value to be written to the register.
event modbus_write_single_register_request%(c: connection, headers: ModbusHeaders, address: count, value: count%);
## Generated for a Modbus write single register response.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## address: The memory address of the register that was written.
##
## value: The value that was written to the register.
event modbus_write_single_register_response%(c: connection, headers: ModbusHeaders, address: count, value: count%);
## Generated for a Modbus write multiple coils request.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## start_address: The memory address of the first coil to be written.
##
## value: The values to be written to the coils.
event modbus_write_multiple_coils_request%(c: connection, headers: ModbusHeaders, start_address: count, coils: ModbusCoils%);
## Generated for a Modbus write multiple coils response.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## start_address: The memory address of the first coil that was written.
##
## quantity: The quantity of coils that were written.
event modbus_write_multiple_coils_response%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
## Generated for a Modbus write multiple registers request.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## start_address: The memory address of the first register to be written.
##
## registers: The values to be written to the registers.
event modbus_write_multiple_registers_request%(c: connection, headers: ModbusHeaders, start_address: count, registers: ModbusRegisters%);
## Generated for a Modbus write multiple registers response.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## start_address: The memory address of the first register that was written.
##
## quantity: The quantity of registers that were written.
event modbus_write_multiple_registers_response%(c: connection, headers: ModbusHeaders, start_address: count, quantity: count%);
## Generated for a Modbus read file record request.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## .. note: This event is incomplete. The information from the data structure is not
## yet passed through to the event.
event modbus_read_file_record_request%(c: connection, headers: ModbusHeaders%);
## Generated for a Modbus read file record response.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## .. note: This event is incomplete. The information from the data structure is not
## yet passed through to the event.
event modbus_read_file_record_response%(c: connection, headers: ModbusHeaders%);
## Generated for a Modbus write file record request.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## .. note: This event is incomplete. The information from the data structure is not
## yet passed through to the event.
event modbus_write_file_record_request%(c: connection, headers: ModbusHeaders%);
## Generated for a Modbus write file record response.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## .. note: This event is incomplete. The information from the data structure is not
## yet passed through to the event.
event modbus_write_file_record_response%(c: connection, headers: ModbusHeaders%);
## Generated for a Modbus mask write register request.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## address: The memory address of the register where the masks should be applied.
##
## and_mask: The value of the logical AND mask to apply to the register.
##
## or_mask: The value of the logical OR mask to apply to the register.
event modbus_mask_write_register_request%(c: connection, headers: ModbusHeaders, address: count, and_mask: count, or_mask: count%);
## Generated for a Modbus mask write register request.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## address: The memory address of the register where the masks were applied.
##
## and_mask: The value of the logical AND mask applied register.
##
## or_mask: The value of the logical OR mask applied to the register.
event modbus_mask_write_register_response%(c: connection, headers: ModbusHeaders, address: count, and_mask: count, or_mask: count%);
## Generated for a Modbus read/write multiple registers request.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## read_start_address: The memory address of the first register to be read.
##
## read_quantity: The number of registers to read.
##
## write_start_address: The memory address of the first register to be written.
##
## write_registers: The values to be written to the registers.
event modbus_read_write_multiple_registers_request%(c: connection, headers: ModbusHeaders, read_start_address: count, read_quantity: count, write_start_address: count, write_registers: ModbusRegisters%);
## Generated for a Modbus read/write multiple registers response.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## written_registers: The register values read from the registers specified in the request.
event modbus_read_write_multiple_registers_response%(c: connection, headers: ModbusHeaders, written_registers: ModbusRegisters%);
## Generated for a Modbus read FIFO queue request.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## start_address: The address of the FIFO queue to read.
event modbus_read_fifo_queue_request%(c: connection, headers: ModbusHeaders, start_address: count%);
## Generated for a Modbus read FIFO queue response.
##
## c: The connection.
##
## headers: The headers for the modbus function.
##
## fifos: The register values read from the FIFO queue on the device.
event modbus_read_fifo_queue_response%(c: connection, headers: ModbusHeaders, fifos: ModbusRegisters%);
## Raised for informational messages reported via Bro's reporter framework. Such
## messages may be generated internally by the event engine and also by other
## scripts calling :bro:id:`Reporter::info`.

35
src/input/Manager.cc
View file

@ -203,7 +203,7 @@ Manager::TableStream::~TableStream()
Manager::Manager()
{
update_finished = internal_handler("Input::update_finished");
end_of_data = internal_handler("Input::end_of_data");
}
Manager::~Manager()
@ -400,7 +400,7 @@ bool Manager::CreateEventStream(RecordVal* fval)
bool allow_file_func = false;
if ( ! etype->IsEvent() )
if ( etype->Flavor() != FUNC_FLAVOR_EVENT )
{
reporter->Error("stream event is a function, not an event");
return false;
@ -573,7 +573,7 @@ bool Manager::CreateTableStream(RecordVal* fval)
{
FuncType* etype = event->FType()->AsFuncType();
if ( ! etype->IsEvent() )
if ( etype->Flavor() != FUNC_FLAVOR_EVENT )
{
reporter->Error("stream event is a function, not an event");
return false;
@ -1179,8 +1179,12 @@ void Manager::EndCurrentSend(ReaderFrontend* reader)
DBG_LOG(DBG_INPUT, "Got EndCurrentSend stream %s", i->name.c_str());
#endif
if ( i->stream_type == EVENT_STREAM ) // nothing to do..
if ( i->stream_type == EVENT_STREAM )
{
// just signal the end of the data source
SendEndOfData(i);
return;
}
assert(i->stream_type == TABLE_STREAM);
TableStream* stream = (TableStream*) i;
@ -1261,12 +1265,29 @@ void Manager::EndCurrentSend(ReaderFrontend* reader)
stream->currDict->SetDeleteFunc(input_hash_delete_func);
#ifdef DEBUG
DBG_LOG(DBG_INPUT, "EndCurrentSend complete for stream %s, queueing update_finished event",
DBG_LOG(DBG_INPUT, "EndCurrentSend complete for stream %s",
i->name.c_str());
#endif
// Send event that the current update is indeed finished.
SendEvent(update_finished, 2, new StringVal(i->name.c_str()), new StringVal(i->info->source));
SendEndOfData(i);
}
void Manager::SendEndOfData(ReaderFrontend* reader)
{
Stream *i = FindStream(reader);
if ( i == 0 )
{
reporter->InternalError("Unknown reader in SendEndOfData");
return;
}
SendEndOfData(i);
}
void Manager::SendEndOfData(const Stream *i)
{
SendEvent(end_of_data, 2, new StringVal(i->name.c_str()), new StringVal(i->info->source));
}
void Manager::Put(ReaderFrontend* reader, Value* *vals)

10
src/input/Manager.h
View file

@ -89,6 +89,7 @@ protected:
friend class EndCurrentSendMessage;
friend class ReaderClosedMessage;
friend class DisableMessage;
friend class EndOfDataMessage;
// For readers to write to input stream in direct mode (reporting
// new/deleted values directly). Functions take ownership of
@ -96,6 +97,9 @@ protected:
void Put(ReaderFrontend* reader, threading::Value* *vals);
void Clear(ReaderFrontend* reader);
bool Delete(ReaderFrontend* reader, threading::Value* *vals);
// Trigger sending the End-of-Data event when the input source has
// finished reading. Just use in direct mode.
void SendEndOfData(ReaderFrontend* reader);
// For readers to write to input stream in indirect mode (manager is
// monitoring new/deleted values) Functions take ownership of
@ -154,7 +158,6 @@ private:
// equivalend in threading cannot be used, because we have support
// different types from the log framework
bool IsCompatibleType(BroType* t, bool atomic_only=false);
// Check if a record is made up of compatible types and return a list
// of all fields that are in the record in order. Recursively unrolls
// records
@ -164,6 +167,9 @@ private:
void SendEvent(EventHandlerPtr ev, const int numvals, ...);
void SendEvent(EventHandlerPtr ev, list<Val*> events);
// Implementation of SendEndOfData (send end_of_data event).
void SendEndOfData(const Stream *i);
// Call predicate function and return result.
bool CallPred(Func* pred_func, const int numvals, ...);
@ -200,7 +206,7 @@ private:
map<ReaderFrontend*, Stream*> readers;
EventHandlerPtr update_finished;
EventHandlerPtr end_of_data;
};

19
src/input/ReaderBackend.cc
View file

@ -108,6 +108,20 @@ public:
private:
};
class EndOfDataMessage : public threading::OutputMessage<ReaderFrontend> {
public:
EndOfDataMessage(ReaderFrontend* reader)
: threading::OutputMessage<ReaderFrontend>("EndOfData", reader) {}
virtual bool Process()
{
input_mgr->SendEndOfData(Object());
return true;
}
private:
};
class ReaderClosedMessage : public threading::OutputMessage<ReaderFrontend> {
public:
ReaderClosedMessage(ReaderFrontend* reader)
@ -183,6 +197,11 @@ void ReaderBackend::EndCurrentSend()
SendOut(new EndCurrentSendMessage(frontend));
}
void ReaderBackend::EndOfData()
{
SendOut(new EndOfDataMessage(frontend));
}
void ReaderBackend::SendEntry(Value* *vals)
{
SendOut(new SendEntryMessage(frontend, vals));

Some files were not shown because too many files have changed in this diff Show more