diff --git a/CHANGES b/CHANGES index ca665576b1..656f378d35 100644 --- a/CHANGES +++ b/CHANGES @@ -1,4 +1,9 @@ +2.1-1306 | 2013-08-31 16:06:05 -0700 + + * Reorganized and signifcantly extended documentation. This includes + two new chapters contributed by Scott Runnels. + 2.1-1216 | 2013-08-31 10:39:40 -0700 diff --git a/INSTALL b/INSTALL index 4604a26bf5..385dac93df 100644 --- a/INSTALL +++ b/INSTALL @@ -1,314 +1,3 @@ -.. _CMake: http://www.cmake.org -.. _SWIG: http://www.swig.org -.. _Xcode: https://developer.apple.com/xcode/ -.. _MacPorts: http://www.macports.org -.. _Fink: http://www.finkproject.org -.. _Homebrew: http://mxcl.github.com/homebrew -.. _bro downloads page: http://bro.org/download/index.html -============== -Installing Bro -============== +See doc/install/install.rst for installation instructions. -Bro can be downloaded in either pre-built binary package or -source code forms. - -Prerequisites -============= - -Bro requires the following libraries and tools to be installed -before you begin: - - * Libpcap http://www.tcpdump.org - - * OpenSSL libraries http://www.openssl.org - - * BIND8 library - - * Libmagic - - * Libz - - * Bash (for BroControl) - -To build Bro from source, the following additional dependencies are required: - - * CMake 2.6.3 or greater http://www.cmake.org - - * SWIG http://www.swig.org - - * Bison (GNU Parser Generator) - - * Flex (Fast Lexical Analyzer) - - * Libpcap headers http://www.tcpdump.org - - * OpenSSL headers http://www.openssl.org - - * libmagic headers - - * zlib headers - - * Perl - -Bro can make use of some optional libraries and tools if they are found at -build time: - - * LibGeoIP (for geo-locating IP addresses) - - * gperftools (tcmalloc is used to improve memory and CPU usage) - - * sendmail (for BroControl) - - * ipsumdump (for trace-summary) http://www.cs.ucla.edu/~kohler/ipsumdump - - * Ruby executable, library, and headers (for Broccoli Ruby bindings) - - -Installing From Pre-Built Binary Release Packages -================================================= - -See the `bro downloads page`_ for currently supported/targeted platforms. - -* RPM - - .. console:: - - sudo yum localinstall Bro-*.rpm - -* DEB - - .. console:: - - sudo gdebi Bro-*.deb - -* MacOS Disk Image with Installer - - Just open the ``Bro-*.dmg`` and then run the ``.pkg`` installer. - Everything installed by the package will go into ``/opt/bro``. - -The primary install prefix for binary packages is ``/opt/bro``. -Non-MacOS packages that include BroControl also put variable/runtime -data (e.g. Bro logs) in ``/var/opt/bro``. - - -Installing From Source -====================== - -Required Dependencies -~~~~~~~~~~~~~~~~~~~~~ - -The following dependencies are required to build Bro: - -* RPM/RedHat-based Linux: - - .. console:: - - sudo yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel file-devel - -* DEB/Debian-based Linux: - - .. console:: - - sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev libmagic-dev - -* FreeBSD - - Most required dependencies should come with a minimal FreeBSD install - except for the following. - - .. console:: - - sudo pkg_add -r bash cmake swig bison python - - Note that ``bash`` needs to be in ``PATH``, which by default it is - not. The FreeBSD package installs the binary into - ``/usr/local/bin``. - -* Mac OS X - - Compiling source code on Macs requires first downloading Xcode_, - then going through its "Preferences..." -> "Downloads" menus to - install the "Command Line Tools" component. - - Lion (10.7) and Mountain Lion (10.8) come with all required - dependencies except for CMake_, SWIG_, and ``libmagic``. - - Distributions of these dependencies can likely be obtained from your - preferred Mac OS X package management system (e.g. MacPorts_, Fink_, - or Homebrew_). - - Specifically for MacPorts, the ``swig``, ``swig-ruby``, ``swig-python`` - and ``file`` packages provide the required dependencies. - -Optional Dependencies -~~~~~~~~~~~~~~~~~~~~~ - -Bro can use libGeoIP for geo-locating IP addresses, and sendmail for -sending emails. - -* RedHat Enterprise Linux: - - .. console:: - - sudo yum install geoip-devel sendmail - -* CentOS Linux: - - .. console:: - - sudo yum install GeoIP-devel sendmail - -* DEB/Debian-based Linux: - - .. console:: - - sudo apt-get install libgeoip-dev sendmail - -* Ports-based FreeBSD - - .. console:: - - sudo pkg_add -r GeoIP - - sendmail is typically already available. - -* Mac OS X - - Vanilla OS X installations don't ship with libGeoIP, but - if installed from your preferred package management system (e.g. MacPorts, - Fink, or Homebrew), they should be automatically detected and Bro will - compile against them. - -Additional steps may be needed to :doc:`get the right GeoIP database `. - -Compiling Bro Source Code -~~~~~~~~~~~~~~~~~~~~~~~~~ - -Bro releases are bundled into source packages for convenience and -available from the `bro downloads page`_. - -Alternatively, the latest Bro development version can be obtained through git -repositories hosted at `git.bro.org `_. See -our `git development documentation -`_ for comprehensive -information on Bro's use of git revision control, but the short story -for downloading the full source code experience for Bro via git is: - -.. console:: - - git clone --recursive git://git.bro.org/bro - -.. note:: If you choose to clone the ``bro`` repository non-recursively for - a "minimal Bro experience", be aware that compiling it depends on - BinPAC, which has its own ``binpac`` repository. Either install it - first or initialize/update the cloned ``bro`` repository's - ``aux/binpac`` submodule. - -The typical way to build and install from source is (for more options, -run ``./configure --help``): - -.. console:: - - ./configure - make - make install - -The default installation path is ``/usr/local/bro``, which would typically -require root privileges when doing the ``make install``. A different -installation path can be chosen by specifying the ``--prefix`` option. -Note that ``/usr`` and ``/opt/bro`` are the -standard prefixes for binary Bro packages to be installed, so those are -typically not good choices unless you are creating such a package. - -Depending on the Bro package you downloaded, there may be auxiliary -tools and libraries available in the ``aux/`` directory. Some of them -will be automatically built and installed along with Bro. There are -``--disable-*`` options that can be given to the configure script to -turn off unwanted auxiliary projects that would otherwise be installed -automatically. Finally, use ``make install-aux`` to install some of -the other programs that are in the ``aux/bro-aux`` directory. - -OpenBSD users, please see our FAQ at -http://www.bro.org/documentation/faq.html if you are having -problems installing Bro. - - -Upgrading From a Previous Version of Bro -======================================== - -If you're doing an upgrade install (rather than a fresh install), -there's two suggested approaches: either install Bro using the same -installation prefix directory as before, or pick a new prefix and copy -local customizations over. - -Re-Use Previous Install Prefix -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If you choose to configure and install Bro with the same prefix -directory as before, local customization and configuration to files in -``$prefix/share/bro/site`` and ``$prefix/etc`` won't be overwritten -(``$prefix`` indicating the root of where Bro was installed). Also, logs -generated at run-time won't be touched by the upgrade. (But making -a backup of local changes before upgrading is still recommended.) - -After upgrading, remember to check ``$prefix/share/bro/site`` and -``$prefix/etc`` for ``.example`` files, which indicate the -distribution's version of the file differs from the local one, which may -include local changes. Review the differences, and make adjustments -as necessary (for differences that aren't the result of a local change, -use the new version's). - -Pick a New Install prefix -~~~~~~~~~~~~~~~~~~~~~~~~~ - -If you want to install the newer version in a different prefix -directory than before, you can just copy local customization and -configuration files from ``$prefix/share/bro/site`` and ``$prefix/etc`` -to the new location (``$prefix`` indicating the root of where Bro was -originally installed). Make sure to review the files for difference -before copying and make adjustments as necessary (for differences that -aren't the result of a local change, use the new version's). Of -particular note, the copied version of ``$prefix/etc/broctl.cfg`` is -likely to need changes to the ``SpoolDir`` and ``LogDir`` settings. - - -Configure the Run-Time Environment -================================== - -Just remember that you may need to adjust your ``PATH`` environment variable -according to the platform/shell/package you're using. For example: - -Bourne-Shell Syntax: - -.. console:: - - export PATH=/usr/local/bro/bin:$PATH - -C-Shell Syntax: - -.. console:: - - setenv PATH /usr/local/bro/bin:$PATH - -Or substitute ``/opt/bro/bin`` instead if you installed from a binary package. - -Running Bro -=========== - -Bro is a complex program and it takes a bit of time to get familiar -with it. A good place for newcomers to start is the Quick Start Guide -at http://www.bro.org/documentation/quickstart.html. - -For developers that wish to run Bro directly from the ``build/`` -directory (i.e., without performing ``make install``), they will have -to first adjust ``BROPATH`` to look for scripts inside the build -directory. Sourcing either ``build/bro-path-dev.sh`` or -``build/bro-path-dev.csh`` as appropriate for the current shell -accomplishes this and also augments your ``PATH`` so you can use the -Bro binary directly:: - - ./configure - make - source build/bro-path-dev.sh - bro diff --git a/NEWS b/NEWS index e4ae28cd61..fcc0863b92 100644 --- a/NEWS +++ b/NEWS @@ -1,17 +1,14 @@ -Release Notes -============= - This document summarizes the most important changes in the current Bro release. For a complete list of changes, see the ``CHANGES`` file (note that submodules, such as BroControl and Broccoli, come with their own CHANGES.) -Bro 2.2 -------- +Bro 2.2 (Work In Progress) +========================== New Functionality -~~~~~~~~~~~~~~~~~ +----------------- - GPRS Tunnelling Protocol (GTPv1) decapsulation. @@ -147,8 +144,28 @@ New Functionality base/utils/active-http.bro for providing an interface for querying remote web servers. +- Summary statistics framework. [Extend] + +- A number of new applications build on top of the summary statistics + framework: + + * Scan detection: Detectors for port and address scans return. See + policy/misc/scan.bro. + + * Tracerouter detector: policy/misc/detect-traceroute + + * Web application detection/measurement: policy/misc/app-metrics.bro + + * FTP brute-forcing detector: policy/protocols/ftp/detect-bruteforcing.bro + + * HTTP-based SQL injection detector: policy/protocols/http/detect-sqli.bro + (existed before, but now ported to the new framework) + + * SSH brute-forcing detector feeding the intelligence framework: + policy/protocols/ssh/detect-bruteforcing.bro + Changed Functionality -~~~~~~~~~~~~~~~~~~~~~ +--------------------- - We removed the following, already deprecated, functionality: @@ -231,11 +248,12 @@ Changed Functionality - We removed the BitTorrent DPD signatures pending further updates to that analyzer. + Bro 2.1 -------- +======= New Functionality -~~~~~~~~~~~~~~~~~ +----------------- - Bro now comes with extensive IPv6 support. Past versions offered only basic IPv6 functionality that was rarely used in practice as it @@ -314,30 +332,9 @@ New Functionality outputs. We do not yet recommend them for production (but welcome feedback!) -- Summary statistics framework. [Extend] - -- A number of new applications build on top of the summary statistics - framework: - - * Scan detection: Detectors for port and address scans return. See - policy/misc/scan.bro. - - * Tracerouter detector: policy/misc/detect-traceroute - - * Web application detection/measurement: policy/misc/app-metrics.bro - - * FTP brute-forcing detector: policy/protocols/ftp/detect-bruteforcing.bro - - * HTTP-based SQL injection detector: policy/protocols/http/detect-sqli.bro - (existed before, but now ported to the new framework) - - * SSH brute-forcing detector feeding the intelligence framework: - policy/protocols/ssh/detect-bruteforcing.bro - - Changed Functionality -~~~~~~~~~~~~~~~~~~~~~ +--------------------- The following summarizes the most important differences in existing functionality. Note that this list is not complete, see CHANGES for @@ -371,7 +368,10 @@ the full set. soon. With that, "match" and "using" are no longer reserved keywords. - The syntax for IPv6 literals changed from "2607:f8b0:4009:802::1012" - to "[2607:f8b0:4009:802::1012]". + to "[2607:f8b0:4009:802::1012]". When an IP address variable or IP + address literal is enclosed in pipes (for example, + ``|[fe80::db15]|``) the result is now the size of the address in + bits (32 for IPv4 and 128 for IPv6). - Bro now spawns threads for doing its logging. From a user's perspective not much should change, except that the OS may now show @@ -411,60 +411,274 @@ the full set. - The ASCII writers "header_*" options have been renamed to "meta_*" (because there's now also a footer). +- Some built-in functions have been removed: "addr_to_count" (use + "addr_to_counts" instead), "bro_has_ipv6" (this is no longer + relevant because Bro now always supports IPv6), "active_connection" + (use "connection_exists" instead), and "connection_record" (use + "lookup_connection" instead). + +- The "NFS3::mode2string" built-in function has been renamed to + "file_mode". + +- Some built-in functions have been changed: "exit" (now takes the + exit code as a parameter), "to_port" (now takes a string as + parameter instead of a count and transport protocol, but + "count_to_port" is still available), "connect" (now takes an + additional string parameter specifying the zone of a non-global IPv6 + address), and "listen" (now takes three additional parameters to + enable listening on IPv6 addresses). + +- Some Bro script variables have been renamed: + "LogAscii::header_prefix" has been renamed to + "LogAscii::meta_prefix", "LogAscii::include_header" has been renamed + to "LogAscii::include_meta". + +- Some Bro script variables have been removed: "tunnel_port", + "parse_udp_tunnels", "use_connection_compressor", + "cc_handle_resets", "cc_handle_only_syns", and + "cc_instantiate_on_data". + +- A couple events have changed: the "icmp_redirect" event now includes + the target and destination addresses and any Neighbor Discovery + options in the message, and the last parameter of the + "dns_AAAA_reply" event has been removed because it was unused. + +- The format of the ASCII log files has changed very slightly. Two + new lines are automatically added, one to record the time when the + log was opened, and the other to record the time when the log was + closed. + +- In BroControl, the option (in broctl.cfg) "CFlowAddr" was renamed to + "CFlowAddress". + Bro 2.0 -------- +======= -As the version number jump suggests, Bro 2.0 is a major upgrade and -lots of things have changed. We have assembled a separate upgrade -guide with the most important changes compared to Bro 1.5 at -http://www.bro.org/documentation/upgrade.html. You can find -the offline version of that document in ``doc/upgrade.rst.``. +As the version number jump from 1.5 suggests, Bro 2.0 is a major +upgrade and lots of things have changed. Most importantly, we have +rewritten almost all of Bro's default scripts from scratch, using +quite different structure now and focusing more on operational +deployment. The result is a system that works much better "out of the +box", even without much initial site-specific configuration. The +down-side is that 1.x configurations will need to be adapted to work +with the new version. The two rules of thumb are: -Compared to the earlier 2.0 Beta version, the major changes in the -final release are: + (1) If you have written your own Bro scripts + that do not depend on any of the standard scripts formerly + found in ``policy/``, they will most likely just keep working + (although you might want to adapt them to use some of the new + features, like the new logging framework; see below). - * The default scripts now come with complete reference - documentation. See - http://www.bro.org/documentation/index.html. + (2) If you have custom code that depends on specifics of 1.x + default scripts (including most configuration tuning), that is + unlikely to work with 2.x. We recommend to start by using just + the new scripts first, and then port over any customizations + incrementally as necessary (they may be much easier to do now, + or even unnecessary). Send mail to the Bro user mailing list + if you need help. - * libz and libmagic are now required dependencies. +Below we summarize changes from 1.x to 2.x in more detail. This list +isn't complete, see the :download:`CHANGES ` file in the +distribution for the full story. - * Reduced snaplen default from 65535 to old default of 8192. The - large value was introducing performance problems on many - systems. +Script Organization +------------------- - * Replaced the --snaplen/-l command line option with a - scripting-layer option called "snaplen". The new option can also - be redefined on the command line, e.g. ``bro -i eth0 - snaplen=65535``. +In versions before 2.0, Bro scripts were all maintained in a flat +directory called ``policy/`` in the source tree. This directory is now +renamed to ``scripts/`` and contains major subdirectories ``base/``, +``policy/``, and ``site/``, each of which may also be subdivided +further. - * Reintroduced the BRO_LOG_SUFFIX environment variable that the - ASCII logger now respects to add a suffix to the log files it - creates. +The contents of the new ``scripts/`` directory, like the old/flat +``policy/`` still gets installed under the ``share/bro`` +subdirectory of the installation prefix path just like previous +versions. For example, if Bro was compiled like ``./configure +--prefix=/usr/local/bro && make && make install``, then the script +hierarchy can be found in ``/usr/local/bro/share/bro``. - * The ASCII logs now include further header information, and - fields set to an empty value are now logged as ``(empty)`` by - default (instead of ``-``, which is already used for fields that - are not set at all). +The main +subdirectories of that hierarchy are as follows: - * Some NOTICES were renamed, and the signatures of some SSL events - have changed. +- ``base/`` contains all scripts that are loaded by Bro by default + (unless the ``-b`` command line option is used to run Bro in a + minimal configuration). Note that is a major conceptual change: + rather than not loading anything by default, Bro now uses an + extensive set of default scripts out of the box. - * bro-cut got some new capabilities: + The scripts under this directory generally either accumulate/log + useful state/protocol information for monitored traffic, configure a + default/recommended mode of operation, or provide extra Bro + scripting-layer functionality that has no significant performance cost. - - If no field names are given on the command line, we now pass - through all fields. +- ``policy/`` contains all scripts that a user will need to explicitly + tell Bro to load. These are scripts that implement + functionality/analysis that not all users may want to use and may have + more significant performance costs. For a new installation, you + should go through these and see what appears useful to load. - - New options -u/-U for time output in UTC. +- ``site/`` remains a directory that can be used to store locally + developed scripts. It now comes with some preinstalled example + scripts that contain recommended default configurations going beyond + the ``base/`` setup. E.g. ``local.bro`` loads extra scripts from + ``policy/`` and does extra tuning. These files can be customized in + place without being overwritten by upgrades/reinstalls, unlike + scripts in other directories. - - New option -F to give output field separator. +With version 2.0, the default ``BROPATH`` is set to automatically +search for scripts in ``policy/``, ``site/`` and their parent +directory, but **not** ``base/``. Generally, everything under +``base/`` is loaded automatically, but for users of the ``-b`` option, +it's important to know that loading a script in that directory +requires the extra ``base/`` path qualification. For example, the +following two scripts: - * Broccoli supports more types internally, allowing to send - complex records. +* ``$PREFIX/share/bro/base/protocols/ssl/main.bro`` +* ``$PREFIX/share/bro/policy/protocols/ssl/validate-certs.bro`` - * Many smaller bug fixes, portability improvements, and general - polishing across all modules. +are referenced from another Bro script like: + +.. code:: bro + + @load base/protocols/ssl/main + @load protocols/ssl/validate-certs + +Notice how ``policy/`` can be omitted as a convenience in the second +case. ``@load`` can now also use relative path, e.g., ``@load +../main``. +Logging Framework +----------------- +- The logs generated by scripts that ship with Bro are entirely redone + to use a standardized, machine parsable format via the new logging + framework. Generally, the log content has been restructured towards + making it more directly useful to operations. Also, several + analyzers have been significantly extended and thus now log more + information. Take a look at ``ssl.log``. + + * A particular format change that may be useful to note is that the + ``conn.log`` ``service`` field is derived from DPD instead of + well-known ports (while that was already possible in 1.5, it was + not the default). + + * Also, ``conn.log`` now reports raw number of packets/bytes per + endpoint. + +- The new logging framework makes it possible to extend, customize, + and filter logs very easily. See the :doc:`logging framework ` + for more information on usage. + +- A common pattern found in the new scripts is to store logging stream + records for protocols inside the ``connection`` records so that + state can be collected until enough is seen to log a coherent unit + of information regarding the activity of that connection. This + state is now frequently seen/accessible in event handlers, for + example, like ``c$`` where ```` is replaced by + the name of the protocol. This field is added to the ``connection`` + record by ``redef``'ing it in a + ``base/protocols//main.bro`` script. + +- The logging code has been rewritten internally, with script-level + interface and output backend now clearly separated. While ASCII + logging is still the default, we will add further output types in + the future (binary format, direct database logging). + + +Notice Framework +---------------- + +The way users interact with "notices" has changed significantly in +order to make it easier to define a site policy and more extensible +for adding customized actions. See the :doc:`notice framework `. + + +New Default Settings +-------------------- + +- Dynamic Protocol Detection (DPD) is now enabled/loaded by default. + +- The default packet filter now examines all packets instead of + dynamically building a filter based on which protocol analysis scripts + are loaded. See ``PacketFilter::all_packets`` for how to revert to old + behavior. + +API Changes +----------- + +- The ``@prefixes`` directive works differently now. + Any added prefixes are now searched for and loaded *after* all input + files have been parsed. After all input files are parsed, Bro + searches ``BROPATH`` for prefixed, flattened versions of all of the + parsed input files. For example, if ``lcl`` is in ``@prefixes``, and + ``site.bro`` is loaded, then a file named ``lcl.site.bro`` that's in + ``BROPATH`` would end up being automatically loaded as well. Packages + work similarly, e.g. loading ``protocols/http`` means a file named + ``lcl.protocols.http.bro`` in ``BROPATH`` gets loaded automatically. + +- The ``make_addr`` BIF now returns a ``subnet`` versus an ``addr`` + + +Variable Naming +--------------- + +- ``Module`` is more widely used for namespacing. E.g. the new + ``site.bro`` exports the ``local_nets`` identifier (among other + things) into the ``Site`` module. + +- Identifiers may have been renamed to conform to new `scripting + conventions + `_ + + +Removed Functionality +--------------------- + +We have remove a bunch of functionality that was rarely used and/or +had not been maintained for a while already: + + - The ``net`` script data type. + - The ``alarm`` statement; use the notice framework instead. + - Trace rewriting. + - DFA state expiration in regexp engine. + - Active mapping. + - Native DAG support (may come back eventually) + - ClamAV support. + - The connection compressor is now disabled by default, and will + be removed in the future. + +BroControl Changes +------------------ + +BroControl looks pretty much similar to the version coming with Bro 1.x, +but has been cleaned up and streamlined significantly internally. + +BroControl has a new ``process`` command to process a trace on disk +offline using a similar configuration to what BroControl installs for +live analysis. + +BroControl now has an extensive plugin interface for adding new +commands and options. Note that this is still considered experimental. + +We have removed the ``analysis`` command, and BroControl currently +does not send daily alarm summaries anymore (this may be restored +later). + +Development Infrastructure +-------------------------- + +Bro development has moved from using SVN to Git for revision control. +Users that want to use the latest Bro development snapshot by checking it out +from the source repositories should see the `development process +`_. Note that all the various +sub-components now reside in their own repositories. However, the +top-level Bro repository includes them as git submodules so it's easy +to check them all out simultaneously. + +Bro now uses `CMake `_ for its build system so +that is a new required dependency when building from source. + +Bro now comes with a growing suite of regression tests in +``testing/``. diff --git a/VERSION b/VERSION index e8782adeb0..7994cf3d47 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.1-1216 +2.1-1306 diff --git a/aux/btest b/aux/btest index 9726c5b982..55e2f6c5fc 160000 --- a/aux/btest +++ b/aux/btest @@ -1 +1 @@ -Subproject commit 9726c5b982ee0e3e730a15dcd65f49bdee3fe458 +Subproject commit 55e2f6c5fce254e822ff98cfd117e40f89c4c30c diff --git a/doc/CHANGES b/doc/CHANGES deleted file mode 120000 index 3e8bc8c0c8..0000000000 --- a/doc/CHANGES +++ /dev/null @@ -1 +0,0 @@ -../CHANGES \ No newline at end of file diff --git a/doc/CMakeLists.txt b/doc/CMakeLists.txt index bdbb0e7b69..373b4643ba 100644 --- a/doc/CMakeLists.txt +++ b/doc/CMakeLists.txt @@ -21,17 +21,17 @@ add_subdirectory(scripts) add_custom_target(broxygen # copy the template documentation to the build directory # to give as input for sphinx - COMMAND "${CMAKE_COMMAND}" -E copy_directory - ${DOC_SOURCE_DIR} + COMMAND rsync -r --copy-links --times + ${DOC_SOURCE_DIR}/ ${DOC_SOURCE_WORKDIR} # copy generated policy script documentation into the # working copy of the template documentation - COMMAND "${CMAKE_COMMAND}" -E copy_directory - ${RST_OUTPUT_DIR} + COMMAND rsync -r --copy-links --times + ${RST_OUTPUT_DIR}/ ${DOC_SOURCE_WORKDIR}/scripts # append to the master index of all policy scripts COMMAND cat ${MASTER_POLICY_INDEX} >> - ${DOC_SOURCE_WORKDIR}/scripts/index.rst + ${DOC_SOURCE_WORKDIR}/scripts/scripts.rst # append to the master index of all policy packages COMMAND cat ${MASTER_PACKAGE_INDEX} >> ${DOC_SOURCE_WORKDIR}/scripts/packages.rst @@ -58,7 +58,7 @@ add_custom_target(broxygen # SOURCES just adds stuff to IDE projects as a convenience SOURCES ${DOC_SOURCES}) -# The "sphinxclean" target removes just the Sphinx input/output directories +# The "broxygenclean" target removes just the Sphinx input/output directories # from the build directory. add_custom_target(broxygenclean COMMAND "${CMAKE_COMMAND}" -E remove_directory @@ -67,7 +67,7 @@ add_custom_target(broxygenclean ${DOC_OUTPUT_DIR} VERBATIM) -add_dependencies(broxygen broxygenclean restdoc) +add_dependencies(broxygen restdoc) add_custom_target(doc) add_custom_target(docclean) diff --git a/doc/INSTALL.rst b/doc/INSTALL.rst deleted file mode 120000 index 99d491b4f8..0000000000 --- a/doc/INSTALL.rst +++ /dev/null @@ -1 +0,0 @@ -../INSTALL \ No newline at end of file diff --git a/doc/LICENSE b/doc/LICENSE new file mode 100644 index 0000000000..88a76c9ec7 --- /dev/null +++ b/doc/LICENSE @@ -0,0 +1,5 @@ +This work is licensed under the Creative Commons +Attribution-NonCommercial 3.0 Unported License. To view a copy of this +license, visit http://creativecommons.org/licenses/by-nc/3.0/ or send +a letter to Creative Commons, 444 Castro Street, Suite 900, Mountain +View, California, 94041, USA. diff --git a/doc/bin/group_index_generator.py b/doc/bin/group_index_generator.py index 720081e8b5..b41dcb379c 100755 --- a/doc/bin/group_index_generator.py +++ b/doc/bin/group_index_generator.py @@ -8,9 +8,11 @@ # 2nd argument is the directory containing ${group}_files lists of # scripts that belong to the group and ${group}_doc_names lists of # document names that can be supplied to a reST :doc: role -# 3rd argument is a directory in which write a ${group}.rst file (will -# append to existing file) that contains reST style references to -# script docs along with summary text contained in original script +# 3rd argument is a directory in which write a ${group}.rst file that contains +# reST style references to script docs along with summary text contained +# in original script. If ${group} ends with "index", then the file +# is always clobbered by this script, but for other unique group names, +# this script will append to existing files. import sys import os @@ -20,9 +22,28 @@ group_list = sys.argv[1] file_manifest_dir = sys.argv[2] output_dir = sys.argv[3] +def make_group_file_index(dir_name, group_name): + group_file = os.path.join(dir_name, group_name + ".rst") + + if not os.path.exists(group_file): + if not os.path.exists(os.path.dirname(group_file)): + os.makedirs(os.path.dirname(group_file)) + + if group_name.endswith("index"): + with open(group_file, 'w') as f_group_file: + f_group_file.write(":orphan:\n\n") + title = "Package Index: %s\n" % os.path.dirname(group_name) + f_group_file.write(title); + for n in range(len(title)): + f_group_file.write("=") + f_group_file.write("\n"); + + return group_file + with open(group_list, 'r') as f_group_list: for group in f_group_list.read().splitlines(): #print group + group_file = make_group_file_index(output_dir, group) file_manifest = os.path.join(file_manifest_dir, group + "_files") doc_manifest = os.path.join(file_manifest_dir, group + "_doc_names") src_files = [] @@ -44,17 +65,6 @@ with open(group_list, 'r') as f_group_list: if sum_pos != -1: summary_comments.append(line[(sum_pos+3):]) #print summary_comments - group_file = os.path.join(output_dir, group + ".rst") - if not os.path.exists(group_file): - if not os.path.exists(os.path.dirname(group_file)): - os.makedirs(os.path.dirname(group_file)) - with open(group_file, 'w') as f_group_file: - f_group_file.write(":orphan:\n\n") - title = "Package Index: %s\n" % os.path.dirname(group) - f_group_file.write(title); - for n in range(len(title)): - f_group_file.write("=") - f_group_file.write("\n"); with open(group_file, 'a') as f_group_file: f_group_file.write("\n:doc:`/scripts/%s`\n" % doc_names[i]) diff --git a/doc/cluster.rst b/doc/cluster/index.rst similarity index 98% rename from doc/cluster.rst rename to doc/cluster/index.rst index ba26471edc..6de70d38cc 100644 --- a/doc/cluster.rst +++ b/doc/cluster/index.rst @@ -1,5 +1,7 @@ -Bro Cluster -=========== + +======================== +Setting up a Bro Cluster +======================== Intro ------ @@ -11,7 +13,7 @@ Architecture The figure below illustrates the main components of a Bro cluster. -.. image:: images/deployment.png +.. image:: /images/deployment.png Tap *** diff --git a/doc/components/binpac/README.rst b/doc/components/binpac/README.rst deleted file mode 120000 index 4eb90ef658..0000000000 --- a/doc/components/binpac/README.rst +++ /dev/null @@ -1 +0,0 @@ -../../../aux/binpac/README \ No newline at end of file diff --git a/doc/components/binpac/README.rst b/doc/components/binpac/README.rst new file mode 100644 index 0000000000..683b5455f4 --- /dev/null +++ b/doc/components/binpac/README.rst @@ -0,0 +1,68 @@ +.. -*- mode: rst-mode -*- +.. +.. Version number is filled in automatically. +.. |version| replace:: 0.34-3 + +====== +BinPAC +====== + +.. rst-class:: opening + + BinPAC is a high level language for describing protocol parsers and + generates C++ code. It is currently maintained and distributed with the + Bro Network Security Monitor distribution, however, the generated parsers + may be used with other programs besides Bro. + +Download +-------- + +You can find the latest BinPAC release for download at +http://www.bro.org/download. + +BinPAC's git repository is located at `git://git.bro.org/binpac.git +`__. You can browse the repository +`here `__. + +This document describes BinPAC |version|. See the ``CHANGES`` +file for version history. + +Prerequisites +------------- + +BinPAC relies on the following libraries and tools, which need to be +installed before you begin: + + * Flex (Fast Lexical Analyzer) + Flex is already installed on most systems, so with luck you can + skip having to install it yourself. + + * Bison (GNU Parser Generator) + Bison is also already installed on many system. + + * CMake 2.6.3 or greater + CMake is a cross-platform, open-source build system, typically + not installed by default. See http://www.cmake.org for more + information regarding CMake and the installation steps below for + how to use it to build this distribution. CMake generates native + Makefiles that depend on GNU Make by default + +Installation +------------ + +To build and install into ``/usr/local``:: + + ./configure + cd build + make + make install + +This will perform an out-of-source build into the build directory using +the default build options and then install the binpac binary into +``/usr/local/bin``. + +You can specify a different installation directory with:: + + ./configure --prefix= + +Run ``./configure --help`` for more options. diff --git a/doc/components/bro-aux/README.rst b/doc/components/bro-aux/README.rst deleted file mode 120000 index 628879525d..0000000000 --- a/doc/components/bro-aux/README.rst +++ /dev/null @@ -1 +0,0 @@ -../../../aux/bro-aux/README \ No newline at end of file diff --git a/doc/components/bro-aux/README.rst b/doc/components/bro-aux/README.rst new file mode 100644 index 0000000000..822afea358 --- /dev/null +++ b/doc/components/bro-aux/README.rst @@ -0,0 +1,70 @@ +.. -*- mode: rst; -*- +.. +.. Version number is filled in automatically. +.. |version| replace:: 0.26-5 + +====================== +Bro Auxiliary Programs +====================== + +.. contents:: + +:Version: |version| + +Handy auxiliary programs related to the use of the Bro Network Security +Monitor (http://www.bro.org). + +Note that some files that were formerly distributed with Bro as part +of the aux/ tree are now maintained separately. See the +http://www.bro.org/download for their download locations. + +adtrace +======= + +Makefile and source for the adtrace utility. This program is used +in conjunction with the localnetMAC.pl perl script to compute the +network address that compose the internal and extern nets that bro +is monitoring. This program when run by itself just reads a pcap +(tcpdump) file and writes out the src MAC, dst MAC, src IP, dst +IP for each packet seen in the file. This output is processed by +the localnetMAC.pl script during 'make install'. + + +devel-tools +=========== + +A set of scripts used commonly for Bro development. + +extract-conn-by-uid: + Extracts a connection from a trace file based + on its UID found in Bro's conn.log + +gen-mozilla-ca-list.rb + Generates list of Mozilla SSL root certificates in + a format readable by Bro. + +update-changes + A script to maintain the CHANGES and VERSION files. + +git-show-fastpath + Show commits to the fastpath branch not yet merged into master. + +cpu-bench-with-trace + Run a number of Bro benchmarks on a trace file. + + +nftools +======= + +Utilities for dealing with Bro's custom file format for storing +NetFlow records. nfcollector reads NetFlow data from a socket and +writes it in Bro's format. ftwire2bro reads NetFlow "wire" format +(e.g., as generated by a 'flow-export' directive) and writes it in +Bro's format. + +rst +=== + +Makefile and source for the rst utility. "rst" can be invoked by +a Bro script to terminate an established TCP connection by forging +RST tear-down packets. See terminate_connection() in conn.bro. diff --git a/doc/components/broccoli-python/README.rst b/doc/components/broccoli-python/README.rst deleted file mode 120000 index 4187e87202..0000000000 --- a/doc/components/broccoli-python/README.rst +++ /dev/null @@ -1 +0,0 @@ -../../../aux/broccoli/bindings/broccoli-python/README \ No newline at end of file diff --git a/doc/components/broccoli-python/README.rst b/doc/components/broccoli-python/README.rst new file mode 100644 index 0000000000..b2203a906a --- /dev/null +++ b/doc/components/broccoli-python/README.rst @@ -0,0 +1,231 @@ +.. -*- mode: rst-mode -*- +.. +.. Version number is filled in automatically. +.. |version| replace:: 0.54 + +============================ +Python Bindings for Broccoli +============================ + +.. rst-class:: opening + + This Python module provides bindings for Broccoli, Bro's client + communication library. In general, the bindings provide the same + functionality as Broccoli's C API. + +.. contents:: + + +Download +-------- + +You can find the latest Broccoli-Python release for download at +http://www.bro.org/download. + +Broccoli-Python's git repository is located at `git://git.bro.org/broccoli-python.git +`__. You can browse the repository +`here `__. + +This document describes Broccoli-Python |version|. See the ``CHANGES`` +file for version history. + + +Installation +------------ + +Installation of the Python module is pretty straight-forward. After +Broccoli itself has been installed, it follows the standard installation +process for Python modules:: + + python setup.py install + +Try the following to test the installation. If you do not see any +error message, everything should be fine:: + + python -c "import broccoli" + +Usage +----- + +The following examples demonstrate how to send and receive Bro +events in Python. + +The main challenge when using Broccoli from Python is dealing with +the data types of Bro event parameters as there is no one-to-one +mapping between Bro's types and Python's types. The Python modules +automatically maps between those types which both systems provide +(such as strings) and provides a set of wrapper classes for Bro +types which do not have a direct Python equivalent (such as IP +addresses). + +Connecting to Bro +~~~~~~~~~~~~~~~~~ + +The following code sets up a connection from Python to a remote Bro +instance (or another Broccoli) and provides a connection handle for +further communication:: + + from broccoli import * + bc = Connection("127.0.0.1:47758") + +An ``IOError`` will be raised if the connection cannot be established. + +Sending Events +~~~~~~~~~~~~~~ + +Once you have a connection handle ``bc`` set up as shown above, you can +start sending events:: + + bc.send("foo", 5, "attack!") + +This sends an event called ``foo`` with two parameters, ``5`` and +``attack!``. Broccoli operates asynchronously, i.e., events scheduled +with ``send()`` are not always sent out immediately but might be +queued for later transmission. To ensure that all events get out +(and incoming events are processed, see below), you need to call +``bc.processInput()`` regularly. + +Data Types +~~~~~~~~~~ + +In the example above, the types of the event parameters are +automatically derived from the corresponding Python types: the first +parameter (``5``) has the Bro type ``int`` and the second one +(``attack!``) has Bro type ``string``. + +For types which do not have a Python equivalent, the ``broccoli`` +module provides wrapper classes which have the same names as the +corresponding Bro types. For example, to send an event called ``bar`` +with one ``addr`` argument and one ``count`` argument, you can write:: + + bc.send("bar", addr("192.168.1.1"), count(42)) + +The following table summarizes the available atomic types and their +usage. + +======== =========== =========================== +Bro Type Python Type Example +======== =========== =========================== +addr ``addr("192.168.1.1")`` +bool bool ``True`` +count ``count(42)`` +double float ``3.14`` +enum Type currently not supported +int int ``5`` +interval ``interval(60)`` +net Type currently not supported +port ``port("80/tcp")`` +string string ``"attack!"`` +subnet ``subnet("192.168.1.0/24")`` +time ``time(1111111111.0)`` +======== =========== =========================== + +The ``broccoli`` module also supports sending Bro records as event +parameters. To send a record, you first define a record type. For +example, a Bro record type:: + + type my_record: record { + a: int; + b: addr; + c: subnet; + }; + +turns into Python as:: + + my_record = record_type("a", "b", "c") + +As the example shows, Python only needs to know the attribute names +but not their types. The types are derived automatically in the same +way as discussed above for atomic event parameters. + +Now you can instantiate a record instance of the newly defined type +and send it out:: + + rec = record(my_record) + rec.a = 5 + rec.b = addr("192.168.1.1") + rec.c = subnet("192.168.1.0/24") + bc.send("my_event", rec) + +.. note:: The Python module does not support nested records at this time. + +Receiving Events +~~~~~~~~~~~~~~~~ + +To receive events, you define a callback function having the same +name as the event and mark it with the ``event`` decorator:: + + @event + def foo(arg1, arg2): + print arg1, arg2 + +Once you start calling ``bc.processInput()`` regularly (see above), +each received ``foo`` event will trigger the callback function. + +By default, the event's arguments are always passed in with built-in +Python types. For Bro types which do not have a direct Python +equivalent (see table above), a substitute built-in type is used +which corresponds to the type the wrapper class' constructor expects +(see the examples in the table). For example, Bro type ``addr`` is +passed in as a string and Bro type ``time`` is passed in as a float. + +Alternatively, you can define a _typed_ prototype for the event. If you +do so, arguments will first be type-checked and then passed to the +call-back with the specified type (which means instances of the +wrapper classes for non-Python types). Example:: + + @event(count, addr) + def bar(arg1, arg2): + print arg1, arg2 + +Here, ``arg1`` will be an instance of the ``count`` wrapper class and +``arg2`` will be an instance of the ``addr`` wrapper class. + +Protoyping works similarly with built-in Python types:: + + @event(int, string): + def foo(arg1, arg2): + print arg1, arg2 + +In general, the prototype specifies the types in which the callback +wants to receive the arguments. This actually provides support for +simple type casts as some types support conversion to into something +different. If for instance the event source sends an event with a +single port argument, ``@event(port)`` will pass the port as an +instance of the ``port`` wrapper class; ``@event(string)`` will pass it +as a string (e.g., ``"80/tcp"``); and ``@event(int)`` will pass it as an +integer without protocol information (e.g., just ``80``). If an +argument cannot be converted into the specified type, a ``TypeError`` +will be raised. + +To receive an event with a record parameter, the record type first +needs to be defined, as described above. Then the type can be used +with the ``@event`` decorator in the same way as atomic types:: + + my_record = record_type("a", "b", "c") + @event(my_record) + def my_event(rec): + print rec.a, rec.b, rec.c + +Helper Functions +---------------- + +The ``broccoli`` module provides one helper function: ``current_time()`` +returns the current time as a float which, if necessary, can be +wrapped into a ``time`` parameter (i.e., ``time(current_time()``) + +Examples +-------- + +There are some example scripts in the ``tests/`` subdirectory of the +``broccoli-python`` repository +`here `_: + + - ``broping.py`` is a (simplified) Python version of Broccoli's test program + ``broping``. Start Bro with ``broping.bro``. + + - ``broping-record.py`` is a Python version of Broccoli's ``broping`` + for records. Start Bro with ``broping-record.bro``. + + - ``test.py`` is a very ugly but comprehensive regression test and part of + the communication test-suite. Start Bro with ``test.bro``. diff --git a/doc/components/broccoli-ruby/README.rst b/doc/components/broccoli-ruby/README.rst deleted file mode 120000 index da71663099..0000000000 --- a/doc/components/broccoli-ruby/README.rst +++ /dev/null @@ -1 +0,0 @@ -../../../aux/broccoli/bindings/broccoli-ruby/README \ No newline at end of file diff --git a/doc/components/broccoli-ruby/README.rst b/doc/components/broccoli-ruby/README.rst new file mode 100644 index 0000000000..647a568cd8 --- /dev/null +++ b/doc/components/broccoli-ruby/README.rst @@ -0,0 +1,67 @@ +.. -*- mode: rst-mode -*- +.. +.. Version number is filled in automatically. +.. |version| replace:: 1.54 + +=============================================== +Ruby Bindings for Broccoli +=============================================== + +.. rst-class:: opening + + This is the broccoli-ruby extension for Ruby which provides access + to the Broccoli API. Broccoli is a library for + communicating with the Bro Intrusion Detection System. + + +Download +======== + +You can find the latest Broccoli-Ruby release for download at +http://www.bro.org/download. + +Broccoli-Ruby's git repository is located at `git://git.bro.org/broccoli-ruby.git +`__. You can browse the repository +`here `__. + +This document describes Broccoli-Ruby |version|. See the ``CHANGES`` +file for version history. + + +Installation +============ + +To install the extension: + +1. Make sure that the ``broccoli-config`` binary is in your path. + (``export PATH=/usr/local/bro/bin:$PATH``) + +2. Run ``sudo ruby setup.rb``. + +To install the extension as a gem (suggested): + +1. Install `rubygems `_. + +2. Make sure that the ``broccoli-config`` binary is in your path. + (``export PATH=/usr/local/bro/bin:$PATH``) + +3. Run, ``sudo gem install rbroccoli``. + +Usage +===== + +There aren't really any useful docs yet. Your best bet currently is +to read through the examples. + +One thing I should mention however is that I haven't done any optimization +yet. You may find that if you write code that is going to be sending or +receiving extremely large numbers of events, that it won't run fast enough and +will begin to fall behind the Bro server. The dns_requests.rb example is +a good performance test if your Bro server is sitting on a network with many +dns lookups. + +Contact +======= + +If you have a question/comment/patch, see the Bro `contact page +`_. diff --git a/doc/components/broccoli/README.rst b/doc/components/broccoli/README.rst deleted file mode 120000 index d32c70ccd9..0000000000 --- a/doc/components/broccoli/README.rst +++ /dev/null @@ -1 +0,0 @@ -../../../aux/broccoli/README \ No newline at end of file diff --git a/doc/components/broccoli/README.rst b/doc/components/broccoli/README.rst new file mode 100644 index 0000000000..4860324292 --- /dev/null +++ b/doc/components/broccoli/README.rst @@ -0,0 +1,141 @@ +.. -*- mode: rst-mode -*- +.. +.. Version number is filled in automatically. +.. |version| replace:: 1.92-9 + +=============================================== +Broccoli: The Bro Client Communications Library +=============================================== + +.. rst-class:: opening + + Broccoli is the "Bro client communications library". It allows you + to create client sensors for the Bro intrusion detection system. + Broccoli can speak a good subset of the Bro communication protocol, + in particular, it can receive Bro IDs, send and receive Bro events, + and send and receive event requests to/from peering Bros. You can + currently create and receive values of pure types like integers, + counters, timestamps, IP addresses, port numbers, booleans, and + strings. + + +Download +-------- + +You can find the latest Broccoli release for download at +http://www.bro.org/download. + +Broccoli's git repository is located at +`git://git.bro.org/broccoli `_. You +can browse the repository `here `_. + +This document describes Broccoli |version|. See the ``CHANGES`` +file for version history. + + +Installation +------------ + +The Broccoli library has been tested on Linux, the BSDs, and Solaris. +A Windows build has not currently been tried but is part of our future +plans. If you succeed in building Broccoli on other platforms, let us +know! + + +Prerequisites +------------- + +Broccoli relies on the following libraries and tools, which need to be +installed before you begin: + + Flex (Fast Lexical Analyzer) + Flex is already installed on most systems, so with luck you + can skip having to install it yourself. + + Bison (GNU Parser Generator) + This comes with many systems, but if you get errors compiling + parse.y, you will need to install it. + + OpenSSL headers and libraries + For encrypted communication. These are likely installed, + though some platforms may require installation of a 'devel' + package for the headers. + + CMake 2.6.3 or greater + CMake is a cross-platform, open-source build system, typically + not installed by default. See http://www.cmake.org for more + information regarding CMake and the installation steps below + for how to use it to build this distribution. CMake generates + native Makefiles that depend on GNU Make by default. + +Broccoli can also make use of some optional libraries if they are found at +installation time: + +Libpcap headers and libraries + Network traffic capture library + + +Installation +------------ + +To build and install into ``/usr/local``:: + + ./configure + make + make install + +This will perform an out-of-source build into the build directory using the +default build options and then install libraries into ``/usr/local/lib``. + +You can specify a different installation directory with:: + + ./configure --prefix= + +Or control the python bindings install destination more precisely with:: + + ./configure --python-install-dir= + +Run ``./configure --help`` for more options. + + +Further notable configure options: + + ``--enable-debug`` + This one enables lots of debugging output. Be sure to disable + this when using the library in a production environment! The + output could easily end up in undersired places when the stdout + of the program you've instrumented is used in other ways. + + ``--with-configfile=FILE`` + Broccoli can read key/value pairs from a config file. By default + it is located in the etc directory of the installation root + (exception: when using ``--prefix=/usr``, ``/etc`` is used + instead of /usr/etc). The default config file name is + broccoli.conf. Using ``--with-configfile``, you can override the + location and name of the config file. + +To use the library in other programs & configure scripts, use the +``broccoli-config`` script. It gives you the necessary configuration flags +and linker flags for your system, see ``--cflags`` and ``--libs``. + +The API is contained in broccoli.h and pretty well documented. A few +usage examples can be found in the test directory, in particular, the +``broping`` tool can be used to test event transmission and reception. Have +a look at the policy file ``broping.bro`` for the events that need to be +defined at the peering Bro. Try ``broping -h`` for a look at the available +options. + +Broccoli knows two kinds of version numbers: the release version number +(as in "broccoli-x.y.tar.gz", or as shipped with Bro) and the shared +library API version number (as in libbroccoli.so.3.0.0). The former +relates to changes in the tree, the latter to compatibility changes in +the API. + +Comments, feedback and patches are appreciated; please check the `Bro +website `_. + +Documentation +------------- + +Please see the `Broccoli User Manual <./broccoli-manual.html>`_ and +the `Broccoli API Reference <../../broccoli-api/index.html>`_. diff --git a/doc/components/broccoli/broccoli-manual.rst b/doc/components/broccoli/broccoli-manual.rst deleted file mode 120000 index bd5e8d711f..0000000000 --- a/doc/components/broccoli/broccoli-manual.rst +++ /dev/null @@ -1 +0,0 @@ -../../../aux/broccoli/doc/broccoli-manual.rst \ No newline at end of file diff --git a/doc/components/broccoli/broccoli-manual.rst b/doc/components/broccoli/broccoli-manual.rst new file mode 100644 index 0000000000..4d3c8ec79f --- /dev/null +++ b/doc/components/broccoli/broccoli-manual.rst @@ -0,0 +1,1355 @@ +=============================================== +Broccoli: The Bro Client Communications Library +=============================================== + +This page documents Broccoli, the Bro client communications library. +It allows you to create client sensors for the Bro intrusion detection +system. Broccoli can speak a good subset of the Bro communication +protocol, in particular, it can receive Bro IDs, send and receive Bro +events, and send and receive event requests to/from peering Bros. + +.. contents:: + +Introduction +############ + +What is Broccoli? +================= + +Broccoli is the BRO Client COmmunications LIbrary. It allows you to +write applications that speak the communication protocol of the `Bro +intrusion detection system `_. + +Broccoli is free software under terms of the BSD license as given in the +COPYING file distributed with its source code. + +In this document, we assume that you are familiar with the basic +concepts of Bro, so please first review the documentation/publications +available from the Bro website if necessary. + +Feedback, patches and bug reports are all welcome, please see +http://www.bro.org/community for instructions on how to participate +in the Bro community. + +Why do I care? +============== + +Having a single IDS on your network is good, but things become a lot +more interesting when you can communicate information among multiple +vantage points in your network. Bro agents can communicate with other +Bro agents, sending and receiving events and other state information. In +the Bro context this is particularly interesting because it means that +you can build sophisticated policy-controlled distributed event +management systems. + +Broccoli enters the picture when it comes to integrating components that +are not Bro agents themselves. Broccoli lets you create applications +that can speak the Bro communication protocol. You can compose, send, +request, and receive events. You can register your own event handlers. +You can talk to other Broccoli applications or Bro agents -- Bro agents +cannot tell whether they are talking to another Bro or a Broccoli +application. Broccoli allows you to integrate applications of your +choosing into a distributed policy-controlled event management system. +Broccoli is intended to be portable: it should build on Linux, the BSDs, +Solaris, and Windows (in the `MinGW `_ +environment). + +Unlike other distributed IDSs, Bro does not assume a strict +sensor-manager hierarchy in the information flow. Instead, Bro agents +can request delivery of arbitrary *events* from other instances. When +an event is triggered in a Bro agent, it checks whether any connected +agents have requested notification of this event, and sends a *copy* of +the event, including the *event arguments*. Recall that in Bro, an +event handler is essentially a function defined in the Bro language, +and an event materializes through invocation of an event handler. Each +remote agent can define its own event handlers. + +Broccoli applications will typically do one or more of the following: + +- *Configuration/Management Tasks:* the Broccoli application + is used to configure remotely running Bros without the need for a + restart. + +- *Interfacing with other Systems:* the Broccoli application + is used to convert Bro events to other alert/notice formats, or into + syslogd entries. + +- *Host-based Sensor Feeds into Bro:* the Broccoli + application reports events based on host-based activity generated in + kernel space or user space applications. + +Installing Broccoli +################### + +The installation process will hopefully be painless: Broccoli is +installed from source using the usual ``./configure && make && +make install`` routine after extraction of the tarball. + +Some relevant configuration options to pass to configure are: + +- ``--prefix=``: sets the installation root to DIR. + The default is to install below ``/usr/local``. + +- ``--enable-debug``: enables debugging output. + Please refer to the `Configuring Debugging Output`_ section for + details on configuring and using debugging output. + +- ``--with-configfile=``: use FILE as location of configuration + file. See the section on `Configuration Files`_ for more on this. + +- ``--with-openssl=``: use the OpenSSL installation below DIR. + +After installation, you'll find the library in shared and static +versions in ``/lib``, the header file for compilation in +``/include``. + +Using Broccoli +############## + +Obtaining information about your build using ``broccoli-config`` +================================================================ + +Similarly to many other software packages, the Broccoli distribution +provides a script that you can use to obtain details about your Broccoli +setup. The script currently provides the following flags: + +- ``--build`` prints the name of the machine the build was + made on, when, and whether debugging support was enabled or not. + +- ``--prefix`` prints the directory in the filesystem + below which Broccoli was installed. + +- ``--version`` prints the version of the distribution + you have installed. + +- ``--libs`` prints the flags to pass to the + linker in order to link in the Broccoli library. + +- ``--cflags`` prints the flags to pass to the + compiler in order to properly include Broccoli's header file. + +- ``--config`` prints the location of the system-wide + config file your installation will use. + +The ``--cflags`` and ``--libs`` flags are the suggested way of obtaining +the necessary information for integrating Broccoli into your build +environment. It is generally recommended to use ``broccoli-config`` for +this purpose, rather than, say, develop new **autoconf** tests. If you +use the **autoconf/automake** tools, we recommend something along the +following lines for your ``configure`` script:: + + dnl ################################################## + dnl # Check for Broccoli + dnl ################################################## + AC_ARG_WITH(broccoli-config, + AC_HELP_STRING(\[--with-broccoli-config=FILE], \[Use given broccoli-config]), + [ brocfg="$withval" ], + [ AC_PATH_GENERIC(broccoli,, + brocfg="broccoli-config", + AC_MSG_ERROR(Cannot find Broccoli: Is broccoli-config in path? Use more fertilizer?)) ]) + + broccoli_libs=`$brocfg --libs` + broccoli_cflags=`$brocfg --cflags` + AC_SUBST(broccoli_libs) + AC_SUBST(broccoli_cflags)`` + +You can then use the compiler/linker flags in your Makefile.in/ams by +substituting in the values accordingly, which might look as follows:: + + CFLAGS = -W -Wall -g -DFOOBAR @broccoli_cflags@ + LDFLAGS = -L/usr/lib/foobar @broccoli_libs@ + +Suggestions for instrumenting applications +========================================== + +Often you will want to make existing applications Bro-aware, that is, +*instrument* them so that they can send and receive Bro events at +appropriate moments in the execution flow. This will involve modifying +an existing code tree, so care needs to be taken to avoid unwanted side +effects. By protecting the instrumented code with ``#ifdef``/``#endif`` +statements you can still build the original application, using the +instrumented source tree. The ``broccoli-config`` script helps you in +doing so because it already adds ``-DBROCCOLI`` to the compiler flags +reported when run with the ``--cflags`` option: + +.. console:: + + > broccoli-config --cflags + -I/usr/local/include -I/usr/local/include -DBROCCOLI + +So simply surround all inserted code with a preprocessor check for +``BROCCOLI`` and you will be able to build the original application as +soon as ``BROCCOLI`` is not defined. + +The Broccoli API +================ + +Time for some code. In the code snippets below we will introduce variables +whenever context requires them and not necessarily when C requires them. +The library does not require calling a global initialization function. +In order to make the API known, include ``broccoli.h``: + +.. code:: c + + #ifdef BROCCOLI + #include + #endif + +.. note:: + *Broccoli's memory management philosophy:* + + Broccoli generally does not release objects you allocate. + The approach taken is "you clean up what you allocate." + +Initialization +-------------- + +Broccoli requires global initialization before most of its other +functions can be used. Generally, the way to initialize Broccoli is as +follows: + +.. code:: c + + bro_init(NULL); + +The argument to ``bro_init()`` provides optional initialization context, +and may be kept ``NULL`` for normal use. If required, you may allocate a +``BroCtx`` structure locally, initialize it using ``bro_ctx_init()``, +fill in additional values as required and subsequently pass it to +``bro_init()``: + +.. code:: c + + BroCtx ctx; + bro_ctx_init(&ctx); + /* Make adjustments to the context structure as required...*/ + bro_init(&ctx); + +.. note:: The ``BroCtx`` structure currently contains a set of five + different callback function pointers. These are *required* for + thread-safe operation of OpenSSL (Broccoli itself is thread-safe). + If you intend to use Broccoli in a multithreaded environment, you + need to implement functions and register them via the ``BroCtx`` + structure. The O'Reilly book "Network Security with OpenSSL" by + Viega et al. shows how to implement these callbacks. + +.. warning:: You *must* call ``bro_init()`` at the start of your + application. Undefined behavior may result if you don't. + +Data types in Broccoli +---------------------- + +Broccoli declares a number of data types in ``broccoli.h`` that you +should know about. The more complex ones are kept opaque, while you do +get access to the fields in the simpler ones. The full list is as +follows: + +- Simple signed and unsigned types: int, uint, uint16, uint32, uint64 + and uchar. + +- Connection handles: BroConn, kept opaque. + +- Bro events: BroEvent, kept opaque. + +- Buffer objects: BroBuf, kept opaque. See also `Using Dynamic + Buffers`_. + +- Ports: BroPort for network ports, defined as follows: + + .. code:: c + + typedef struct bro_port { + uint16 port_num; /* port number in host byte order */ + int port_proto; /* IPPROTO_xxx */ + } BroPort; + +- Records: BroRecord, kept opaque. See also `Handling Records`_. + +- Strings (character and binary): BroString, defined as follows: + + .. code:: c + + typedef struct bro_string { + int str_len; + char str_val; + } BroString; + +- BroStrings are mostly kept transparent for convenience; please have a + look at the `Broccoli API Reference`_. + +- Tables: BroTable, kept opaque. See also `Handling Tables`_. + +- Sets: BroSet, kept opaque. See also `Handling Sets`_. + +- IP Address: BroAddr, defined as follows: + + .. code:: c + + typedef struct bro_addr { + uint32 addr[4]; /* IP address in network byte order */ + int size; /* Number of 4-byte words occupied in addr */ + } BroAddr; + + Both IPv4 and IPv6 addresses are supported, with the former occupying + only the first 4 bytes of the ``addr`` array. + +- Subnets: BroSubnet, defined as follows: + + .. code:: c + + typedef struct bro_subnet { + BroAddr sn_net; /* IP address in network byte order */ + uint32 sn_width; /* Length of prefix to consider. */ + } BroSubnet; + +Managing Connections +-------------------- + +You can use Broccoli to establish a connection to a remote Bro, or to +create a Broccoli-enabled server application that other Bros will +connect to (this means that in principle, you can also use Broccoli +purely as middleware and have multiple Broccoli applications communicate +directly). + +In order to establish a connection to a remote Bro, you first obtain a +connection handle. You then use this connection handle to request +events, connect to the remote Bro, send events, etc. Connection handles +are pointers to ``BroConn`` structures, which are kept opaque. Use +``bro_conn_new()`` or ``bro_conn_new_str()`` to obtain a handle, +depending on what parameters are more convenient for you: the former +accepts the IP address and port number as separate numerical arguments, +the latter uses a single string to encode both, in "hostname:port" +format. + +To write a Broccoli-enabled server, you first need to implement the +usual ``socket()`` / ``bind()`` / ``listen()`` / ``accept()`` routine. +Once you have obtained a file descriptor for the new connection from +``accept()``, you pass it to the third function that returns a +``BroConn`` handle, ``bro_conn_new_socket()``. The rest of the +connection handling then proceeds as in the client scenario. + +All three calls accept additional flags for fine-tuning connection +behaviour. These flags are: + +- ``BRO_CFLAG_NONE``: no functionality. Use when no flags are desired. + +- ``BRO_CFLAG_RECONNECT``: + When using this option, Broccoli will attempt to reconnect to the peer + transparently after losing connectivity. Essentially whenever you try to + read from or write to the peer and its connection has broke down, a full + reconnect including complete handshaking is attempted. You can check + whether the connection to a peer is alive at any time using + ``bro_conn_alive()``. + +- ``BRO_CFLAG_ALWAYS_QUEUE``: + When using this option, Broccoli will queue any events you send for + later transmission when a connection is currently down. Without using + this flag, any events you attempt to send while a connection is down + get dropped on the floor. Note that Broccoli maintains a maximum queue + size per connection so if you attempt to send lots of events while the + connection is down, the oldest events may start to get dropped + nonetheless. Again, you can check whether the connection is currently + okay by using ``bro_conn_alive()``. + +- ``BRO_CFLAG_DONTCACHE``: + When using this option, Broccoli will ask the peer not to use caching + on the objects it sends to us. This is the default, and the flag need + not normally be used. It is kept to maintain backward compatibility. + +- ``BRO_CFLAG_CACHE``: + When using this option, Broccoli will ask the peer to use caching on + the objects it sends to us. Caching is normally disabled. + +- ``BRO_CFLAG_YIELD``: + When using this option, ``bro_conn_process_input()`` processes at most + one event at a time and then returns. + +By obtaining a connection handle, you do not also establish a connection +right away. This is done using ``bro_conn_connect()``. The main reason +for this is to allow you to subscribe to events (using +``bro_event_registry_add()``, see `Receiving Events`_) before +establishing the connection. Upon returning from ``bro_conn_connect()`` +you are guaranteed to receive all instances of the event types you have +requested, while later on during the connection some time may elapse +between the issuing of a request for events and the processing of that +request at the remote end. Connections are established via TCP, +optionally using SSL encryption. See "`Configuring Encrypted +Communication`_", for more information on setting up encryption. The +port numbers Bro agents and Broccoli applications listen on can vary +from peer to peer. + +Finally, ``bro_conn_delete()`` terminates a connection and releases all +resources associated with it. You can create as many connections as you +like, to one or more peers. You can obtain the file descriptor of a +connection using ``bro_conn_get_fd()``: + +.. code:: c + + char host_str = "bro.yourorganization.com"; + int port = 1234; + struct hostent *host; + BroConn *bc; + + if (! (host = gethostbyname(host_str)) || ! + (host->h_addr_list[0])) + { + /* Error handling -- could not resolve host */ + } + + /* In this example, we obtain a connection handle, then register + event handlers, and finally connect to the remote Bro. */ + /* First obtain a connection handle: */ + if (! (bc = bro_conn_new((struct in_addr*) host->h_addr_list[0], + htons(port), BRO_CFLAG_NONE))) + { + /* Error handling - could not get connection handle */ + } + + /* Register event handlers: */ + bro_event_registry_add(bc, "foo", bro_foo_handler, NULL); + /* ... */ + + /* Now connect to the peer: */ + if (! bro_conn_connect(bc)) + { + /* Error handling - could not connect to remote Bro. */ + } + + /* Send and receive events ... */ + + /* Disconnect from Bro and clean up connection */ + bro_conn_delete(bc); + +Or simply use the string-based version: + +.. code:: c + + char host_str = "bro.yourcompany.com:1234"; + BroConn bc; + + /* In this example we don't request any events from the peer, + but we ask it not to use the serialization cache. */ + /* Again, first obtain a connection handle: */ + if (! (bc = bro_conn_new_str(host_str, BRO_CFLAG_DONTCACHE))) + { + /* Error handling - could not get connection handle */ + } + + /* Now connect to the peer: */ + if (! bro_conn_connect(bc)) + { + /* Error handling - could not connect to remote Bro. */ + } + + /* ... */ + +Connection Classes +------------------ + +When you want to establish connections from multiple Broccoli +applications with different purposes, the peer needs a means to +understand what kind of application each connection belongs to. The real +meaning of "kind of application" here is "sets of event types to +request", because depending on the class of an application, the peer +will likely want to receive different types of events. + +Broccoli lets you set the class of a connection using +``bro_conn_set_class()``. When using this feature, you need to call that +function before issuing a ``bro_conn_connect()`` since the class of a +connection is determined at connection startup: + +.. code:: c + + if (! (bc = bro_conn_new_str(host_str, BRO_CFLAG_DONTCACHE))) + { + /* Error handling - could not get connection handle */ + } + + /* Set class of this connection: */ + bro_conn_set_class(bc, "syslog"); + + if (! bro_conn_connect(bc)) + { + /* Error handling - could not connect to remote Bro. */ + } + +If your peer is a Bro node, you need to match the chosen connection +class in the remote Bro's ``Communication::nodes`` configuration. See +`Configuring event reception in Bro scripts`_, for how to do +this. Finally, in order to obtain the class of a connection as +indicated by the remote side, use ``bro_conn_get_peer_class()``. + +Composing and sending events +---------------------------- + +In order to send an event to the remote Bro agent, you first create an +empty event structure with the name of the event, then add parameters to +pass to the event handler at the remote agent, and then send off the +event. + +.. note: + *Bro peers ignore unrequested events.* + + You need to make sure that the remote Bro agent is interested in + receiving the events you send. This interest is expressed in policy + configuration. We'll explain this in more detail in `Configuring + event reception in Bro scripts`_, and for now assume that our + remote peer is configured to receive the events we send. + +Let's assume we want to request a report of all connections a remote Bro +currently keeps state for that match a given destination port and host +name and that have amassed more than a certain number of bytes. The +idea is to send an event to the remote Bro that contains the query, +identifiable through a request ID, and have the remote Bro answer us +with ``remote_conn`` events containing the information we asked for. The +definition of our requesting event could look as follows in the Bro +policy: + +.. code:: bro + + event report_conns(req_id: int, dest_host: string, + dest_port: port, min_size: count); + +First, create a new event: + +.. code:: c + + BroEvent *ev; + + if (! (ev = bro_event_new("report_conns"))) + { + /* Error handling - could not allocate new event. */ + } + +Now we need to add parameters to the event. The sequence and types must +match the event handler declaration -- check the Bro policy to make sure +they match. The function to use for adding parameter values is +``bro_event_add_val()``. All values are passed as *pointer arguments* +and are copied internally, so the object you're pointing to stays +unmodified at all times. You clean up what you allocate. In order to +indicate the type of the value passed into the function, you need to +pass a numerical type identifier along as well. Table-1_ lists the +value types that Broccoli supports along with the type identifier and +data structures to point to. + +.. _Table-1: + +Types, type tags, and data structures for event parameters in Broccoli +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +============================== ===================== ==================== +Type Type tag Data type pointed to +============================== ===================== ==================== +Boolean ``BRO_TYPE_BOOL`` ``int`` +Integer value ``BRO_TYPE_INT`` ``uint64`` +Counter (nonnegative integers) ``BRO_TYPE_COUNT`` ``uint64`` +Enums (enumerated values) ``BRO_TYPE_ENUM`` ``uint64`` (see also description of ``bro_event_add_val()``'s ``type_name`` argument) +Floating-point number ``BRO_TYPE_DOUBLE`` ``double`` +Timestamp ``BRO_TYPE_TIME`` ``double`` (see also ``bro_util_timeval_to_double()`` and ``bro_util_current_time()``) +Time interval ``BRO_TYPE_INTERVAL`` ``double`` +Strings (text and binary) ``BRO_TYPE_STRING`` ``BroString`` (see also family of ``bro_string_xxx()`` functions) +Network ports ``BRO_TYPE_PORT`` ``BroPort``, with the port number in host byte order +IPv4/IPv6 address ``BRO_TYPE_IPADDR`` ``BroAddr``, with the ``addr`` member in network byte order and ``size`` member indicating the address family and number of 4-byte words of ``addr`` that are occupied (1 for IPv4 and 4 for IPv6) +IPv4/IPv6 subnet ``BRO_TYPE_SUBNET`` ``BroSubnet``, with the ``sn_net`` member in network byte order +Record ``BRO_TYPE_RECORD`` ``BroRecord`` (see also the family of ``bro_record_xxx()`` functions and their explanation below) +Table ``BRO_TYPE_TABLE`` ``BroTable`` (see also the family of ``bro_table_xxx()`` functions and their explanation below) +Set ``BRO_TYPE_SET`` ``BroSet`` (see also the family of ``bro_set_xxx()`` functions and their explanation below) +============================== ===================== ==================== + +Knowing these, we can now compose a ``request_connections`` event: + +.. code:: c + + BroString dest_host; + BroPort dest_port; + uint32 min_size; + int req_id = 0; + + bro_event_add_val(ev, BRO_TYPE_INT, NULL, &req_id); + req_id++; + + bro_string_set(&dest_host, "desthost.destdomain.com"); + bro_event_add_val(ev, BRO_TYPE_STRING, NULL, &dest_host); + bro_string_cleanup(&dest_host); + + dest_port.dst_port = 80; + dest_port.dst_proto = IPPROTO_TCP; + bro_event_add_val(ev, BRO_TYPE_PORT, NULL, &dest_port); + + min_size = 1000; bro_event_add_val(ev, BRO_TYPE_COUNT, NULL, &min_size); + +The third argument to ``bro_event_add_val()`` lets you specify a +specialization of the types listed in Table-1_. This is generally not +necessary except for one situation: when using ``BRO_TYPE_ENUM``. You +currently cannot define a Bro-level enum type in Broccoli, and thus when +sending an enum value, you have to specify the type of the enum along +with the value. For example, in order to add an instance of enum +``transport_type`` defined in Bro's ``bro.init``, you would use: + +.. code:: c + + int transport_proto = 2; + /* ... */ + bro_event_add_val(ev, BRO_TYPE_ENUM, "transport_proto", &transport_proto); + +to get the equivalent of "udp" on the remote side. The same system is +used to point out type names when calling ``bro_event_set_val()``, +``bro_record_add_val()``, ``bro_record_set_nth_val()``, and +``bro_record_set_named_val()``. + +All that's left to do now is to send off the event. For this, use +``bro_event_send()`` and pass it the connection handle and the event. +The function returns ``TRUE`` when the event could be sent right away or +if it was queued for later delivery. ``FALSE`` is returned on error. If +the event gets queued, this does not indicate an error -- likely the +connection was just not ready to send the event at this point. Whenever +you call ``bro_event_send()``, Broccoli attempts to send as much of an +existing event queue as possible. Again, the event is copied internally +to make it easier for you to send the same event repeatedly. You clean +up what you allocate: + +.. code:: c + + bro_event_send(bc, ev); + bro_event_free(ev); + +Two other functions may be useful to you: ``bro_event_queue_length()`` +tells you how many events are currently queued, and +``bro_event_queue_flush()`` attempts to flush the current event queue +and returns the number of events that do remain in the queue after the +flush. + +.. note:: you do not normally need to call this function, queue + flushing is attempted every time you send an event. + +Receiving Events +---------------- + +Receiving events is a little more work because you need to + +1. tell Broccoli what to do when requested events arrive, + +#. let the remote Bro agent know that you would like to receive those + events, + +#. find a spot in the code path suitable for extracting and processing + arriving events. + +Each of these steps is explained in the following sections. + +Implementing event callbacks +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When Broccoli receives an event, it tries to dispatch the event to +callbacks registered for that event type. The place where callbacks get +registered is called the callback registry. Any callbacks registered for +the arriving event's name are invoked with the parameters shipped with +the event. There are two styles of argument passing to the event +callbacks. Which one is better suited depends on your application. + +Expanded Argument Passing +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Each event argument is passed via a pointer to the callback. This makes +best sense when you know the type of the event and of its arguments, +because it provides you immediate access to arguments as when using a +normal C function. + +In order to register a callback with expanded argument passing, use +``bro_event_registry_add()`` and pass it the connection handle, the name +of the event for which you register the callback, the callback itself +that matches the signature of the ``BroEventFunc`` type, and any user +data (or ``NULL``) you want to see passed to the callback on each +invocation. The callback's type is defined rather generically as +follows: + +.. code:: c + + typedef void (*BroEventFunc) (BroConn *bc, void *user_data, ...); + +It requires a connection handle as its first argument and a pointer to +user-provided callback data as the second argument. Broccoli will pass +the connection handle of the connection on which the event arrived +through to the callback. ``BroEventFunc``'s are variadic, because each +callback you provide is directly invoked with pointers to the parameters +of the event, in a format directly usable in C. All you need to know is +what type to point to in order to receive the parameters in the right +layout. Refer to Table-1_ again for a summary of those types. Record +types are more involved and are addressed in more detail in `Handling +Records`_. + +.. note:: Note that *all* parameters are passed to the + callback as pointers, even elementary types such as ``int`` that + would normally be passed directly. Also note that Broccoli manages + the lifecycle of event parameters and therefore you do *not* have + to clean them up inside the event handler. + +Continuing our example, we will want to process the connection reports +that contain the responses to our ``report_conns`` event. Let's assume +those look as follows: + +.. code:: bro + + event remote_conn(req_id: int, conn: connection); + +The reply events contain the request ID so we can associate requests +with replies, and a connection record (defined in ``bro.init`` in Bro). +(It'd be nicer to report all replies in a single event but we'll +ignore that for now.) For this event, our callback would look like +this: + +.. code:: c + + void remote_conn_cb(BroConn *bc, void *user_data, int *req_id, + BroRecord *conn); + +Once more, you clean up what you allocate, and since you never allocated +the space these arguments point to, you also don't clean them up. +Finally, we register the callback using ``bro_event_registry_add()``: + +.. code:: c + + bro_event_registry_add(bc, "remote_conn", remote_conn_cb, NULL); + +In this case we have no additional data to be passed into the callback, +so we use ``NULL`` for the last argument. If you have multiple events +you are interested in, register each one in this fashion. + +Compact Argument Passing +^^^^^^^^^^^^^^^^^^^^^^^^ + +This is designed for situations when you have to determine how to handle +different types of events at runtime, for example when writing language +bindings or when implementing generic event handlers for multiple event +types. The callback is passed a connection handle and the user data as +above but is only passed one additional pointer, to a BroEvMeta +structure. This structure contains all metadata about the event, +including its name, timestamp (in UTC) of creation, number of arguments, +the arguments' types (via type tags as listed in Table-1_), and the +arguments themselves. + +In order to register a callback with compact argument passing, use +``bro_event_registry_add_compact()`` and pass it similar arguments as +you'd use with ``bro_event_registry_add()``. The callback's type is +defined as follows: + +.. code:: c + + typedef void (*BroCompactEventFunc) (BroConn *bc, void *user_data, + BroEvMeta *meta); + + +.. note:: As before, Broccoli manages the lifecycle of event parameters. + You do not have to clean up the BroEvMeta structure or any of its + contents. + +Below is sample code for extracting the arguments from the BroEvMeta +structure, using our running example. This is still written with the +assumption that we know the types of the arguments, but note that this +is not a requirement for this style of callback: + +.. code:: c + + void remote_conn_cb(BroConn *bc, void *user_data, + BroEvMeta *meta) { + int *req_id; BroRecord *rec; + + /* For demonstration, print out the event's name: */ + + printf("Handling a %s event.\n", meta->ev_name); + + /* Sanity-check the number of arguments: */ + + if (meta->ev_numargs != 2) + { /* error */ } + + /* Sanity-check the argument types: */ + + if (meta->ev_args[0].arg_type != BRO_TYPE_INT) + { /* error */ } + + if (meta->ev_args[1].arg_type != BRO_TYPE_RECORD) + { /* error */ } + + req_id = (int *) meta->ev_args[0].arg_data; + rec = (BroRecord *) meta->ev_args[1].arg_data; + + /* ... */ + } + +Finally, register the callback using +``bro_event_registry_add_compact()``: + +.. code:: c + + bro_event_registry_add_compact(bc, "remote_conn", remote_conn_cb, NULL); + +Requesting event delivery +~~~~~~~~~~~~~~~~~~~~~~~~~ + +At this point, Broccoli knows what to do with the requested events upon +arrival. What's left to do is to let the remote Bro know that you would +like to receive the events for which you registered. If you haven't yet +called ``bro_conn_connect()``, then there is nothing to do, since that +function will request the registered events anyway. Once connected, you +can still request events. To do so, call +``bro_event_registry_request()``: + +.. code:: c + + bro_event_registry_request(bc); + +This mechanism also implies that no unrequested events will be delivered +to us (and if that happened for whatever reason, the event would simply +be dropped on the floor). + +.. note:: At the moment you cannot unrequest events, nor can you request + events based on predicates on the values of the events' arguments. + +Reading events from the connection handle +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +At this point the remote Bro will start sending you the requested events +once they are triggered. What is left to do is to read the arriving +events from the connection and trigger dispatching them to the +registered callbacks. + +If you are writing a new Bro-enabled application, this is easy, and you +can choose among two approaches: polling explicitly via Broccoli's API, +or using ``select()`` on the file handle associated with a BroConn. The +former case is particularly straightforward; all you need to do is call +``bro_conn_process_input()``, which will go off and check if any events +have arrived and if so, dispatch them accordingly. This function does +not block -- if no events have arrived, then the call will return +immediately. For more fine-grained control over your I/O handling, you +will probably want to use ``bro_conn_get_fd()`` to obtain the file +descriptor of your connection and then incorporate that in your standard +``FD_SET``/``select()`` code. Once you have determined that data in fact +are ready to be read from the obtained file descriptor, you can then try +another ``bro_conn_process_input()`` this time knowing that it'll find +something to dispatch. + +As a side note, if you don't process arriving events frequently enough, +then TCP's flow control will start to slow down the sender until +eventually events will queue up and be dropped at the sending end. + +Handling Records +---------------- + +Broccoli supports record structures, i.e., types that pack a set of +values together, placing each value into its own field. In Broccoli, the +way you handle records is somewhat similar to events: after creating an +empty record (of opaque type ``BroRecord``), you can iteratively add +fields and values to it. The main difference is that you must specify a +field name with the value; each value in a record can be identified both +by position (a numerical index starting from zero), and by field name. +You can retrieve vals in a record by field index or field name. You can +also reassign values. There is no explicit, IDL-style definition of +record types. You define the type of a record implicitly by the sequence +of field names and the sequence of the types of the values you put into +the record. + +Note that all fields in a record must be assigned before it can be +shipped. + +The API for record composition consists of ``bro_record_new()``, +``bro_record_free()``, ``bro_record_add_val()``, +``bro_record_set_nth_val()``, and ``bro_record_set_named_val()``. + +On records that use field names, the names of individual fields can be +extracted using ``bro_record_get_nth_name()``. Extracting values from a +record is done using ``bro_record_get_nth_val()`` and +``bro_record_get_named_val()``. The former allows numerical indexing of +the fields in the record, the latter provides name-based lookups. Both +need to be passed the record you want to extract a value from, the index +or name of the field, and either a pointer to an int holding a +BRO_TYPE_xxx value (see again Table-1_ for a summary of those types) or +``NULL``. The pointer, if not ``NULL``, serves two purposes: type +checking and type retrieval. Type checking is performed if the value of +the int upon calling the functions is not BRO_TYPE_UNKNOWN. The type tag +of the requested record field then has to match the type tag stored in +the int, otherwise ``NULL`` is returned. If the int stores +BRO_TYPE_UNKNOWN upon calling, no type-checking is performed. In *both* +cases, the *actual* type of the requested record field is returned in +the int pointed to upon return from the function. Since you have no +guarantees of the type of the value upon return if you pass ``NULL`` as +the int pointer, this is a bad idea and either BRO_TYPE_UNKNOWN or +another type value should always be used. + +For example, you could extract the value of the record field "label", +which we assume should be a string, in the following ways: + +.. code:: c + + BroRecord *rec = /* obtained somehow */ + BroString *string; + int type; + + /* --- Example 1 --- */ + + type = BRO_TYPE_STRING; + /* Use type-checking, will not accept other types */ + + if (! (string = bro_record_get_named_val(rec, "label", &type))) + { + /* Error handling, either there's no field of that value or + the value is not of BRO_TYPE_STRING. The actual type is now + stored in "type". */ + } + + /* --- Example 2 --- */ + + type = BRO_TYPE_UNKNOWN; + /* No type checking, just report the existent type */ + + if (! (string = bro_record_get_named_val(rec, "label", &type))) + { + /* Error handling, no field of that name exists. */ + } + + printf("The type of the value in field 'label' is %i\n", type); + + /* --- Example 3 --- */ + + if (! (string = bro_record_get_named_val(rec, "label", NULL))) + { + /* Error handling, no field of that name exists. */ + } + + /* We now have a value, but we can't really be sure of its type */ + +Record fields can be records, for example in the case of Bro's standard +connection record type. In this case, in order to get to a nested +record, you use ``BRO_TYPE_RECORD``: + +.. code:: c + + void remote_conn_cb(BroConn *bc, int *req_id, BroRecord *conn) + { + BroRecord *conn_id; + int type = BRO_TYPE_RECORD; + if ( ! (conn_id = bro_record_get_named_val(conn, "id", &type))) + { + /* Error handling */ + } + } + +Handling Tables +--------------- + +Broccoli supports Bro-style tables, i.e., associative containers that +map instances of a key type to an instance of a value type. A given key +can only ever point to a single value. The key type can be *composite*, +i.e., it may consist of an ordered sequence of different types, or it +can be *direct*, i.e., consisting of a single type (such as an integer, +a string, or a record). + +The API for table manipulation consists of ``bro_table_new()`` +``bro_table_free()``, ``bro_table_insert()``, ``bro_table_find()``, +``bro_table_get_size()``, ``bro_table_get_types()``, and +``bro_table_foreach()``. + +Tables are handled similarly to records in that typing is determined +dynamically by the initial key/value pair inserted. The resulting types +can be obtained via ``bro_table_get_types()``. Should the types not +have been determined yet, ``BRO_TYPE_UNKNOWN`` will result. Also, as +with records, values inserted into the table are copied internally, and +the ones passed to the insertion functions remain unaffected. + +In contrast to records, table entries can be iterated. By passing a +function of signature ``BroTableCallback()`` and a pointer to data of +your choosing, ``bro_table_foreach()`` will invoke the given function +for each key/value pair stored in the table. Return ``TRUE`` to keep +the iteration going, or ``FALSE`` to stop it. + +.. note:: + The main thing to know about Broccoli's tables is how to use + composite key types. To avoid additional API calls, you may treat + composite key types exactly as records, though you do not need to use + field names when assigning elements to individual fields. So in order + to insert a key/value pair, you create a record with the needed items + assigned to its slots, and use this record as the key object. In + order to differentiate composite index types from direct ones + consisting of a single record, use ``BRO_TYPE_LIST`` as the type of + the record, as opposed to ``BRO_TYPE_RECORD``. Broccoli will then + know to interpret the record as an ordered sequence of items making + up a composite element, not a regular record. + +``brotable.c`` in the ``test/`` subdirectory of the Broccoli tree +contains an extensive example of using tables with composite as well as +direct indexing types. + +Handling Sets +------------- + +Sets are essentially tables with void value types. The API for set +manipulation consists of ``bro_set_new()``, ``bro_set_free()``, +``bro_set_insert()``, ``bro_set_find()``, ``bro_set_get_size()``, +``bro_set_get_type()``, and ``bro_set_foreach()``. + +Associating data with connections +--------------------------------- + +You will often find that you would like to connect data with a +``BroConn``. Broccoli provides an API that lets you associate data items +with a connection handle through a string-based key-value registry. The +functions of interest are ``bro_conn_data_set()``, +``bro_conn_data_get()``, and ``bro_conn_data_del()``. You need to +provide a string identifier for a data item and can then use that string +to register, look up, and remove the associated data item. Note that +there is currently no mechanism to trigger a destructor function for +registered data items when the Bro connection is terminated. You +therefore need to make sure that all data items that you do not have +pointers to via some other means are properly released before calling +``bro_disconnect()``. + +Configuration Files +------------------- + +Imagine you have instrumented the mother of all server applications. +Building it takes forever, and every now and then you need to change +some of the parameters that your Broccoli code uses, such as the host +names of the Bro agents to talk to. To allow you to do this quickly, +Broccoli comes with support for configuration files. All you need to do +is change the settings in the file and restart the application (we're +considering adding support for volatile configuration items that are +read from the file every time they are requested). + +A configuration is read from a single configuration file. This file can +be read from different locations. Broccoli searches in this order +for the config file: + +- The location specified by the ``BROCCOLI_CONFIG_FILE`` environment + variable. + +- A per-user configuration file stored in ``~/.broccoli.conf``. + +- The system-wide configuration file. You can obtain the location + of this config file by running ``broccoli-config --config``. + +.. note:: ``BROCCOLI_CONFIG_FILE`` or ``~/.broccoli.conf`` will only be + used if it is a regular file, not executable, and neither group nor + others have any permissions on the file. That is, the file's + permissions must look like ``-rw-------`` *or* ``-r--------``. + +In the configuration file, a ``#`` anywhere starts a comment that runs to +the end of the line. Configuration items are specified as key-value +pairs:: + + # This is the Broccoli system-wide configuration file. + # + # Entries are of the form , where the + # identifier is a sequence of letters, and value can be a string + # (including whitespace), and floating point or integer numbers. + # Comments start with a "#" and go to the end of the line. For + # boolean values, you may also use "yes", "on", "true", "no", + # "off", or "false". Strings may contain whitespace, but need + # to be surrounded by double quotes '"'. + # + # Examples: + # + Foo/PeerName mybro.securesite.com + Foo/PortNum 123 + Bar/SomeFloat 1.23443543 + Bar/SomeLongStr "Hello World" + +You can also have multiple sections in your configuration. Your +application can select a section as the current one, and queries for +configuration settings will then only be answered with values specified +in that section. A section is started by putting its name (no whitespace +please) between square brackets. Configuration items positioned before +the first section title are in the default domain and will be used by +default:: + + # This section contains all settings for myapp. + [ myapp ] + +You can name identifiers any way you like, but to keep things organized +it is recommended to keep a namespace hierarchy similar to the file +system. In the code, you can query configuration items using +``bro_conf_get_str()``, ``bro_conf_get_int()``, and +``bro_conf_get_dbl()``. You can switch between sections using +``bro_conf_set_domain()``. + +Using Dynamic Buffers +--------------------- + +Broccoli provides an API for dynamically allocatable, growable, +shrinkable, and consumable buffers with ``BroBuf``. You may or may not +find this useful -- Broccoli mainly provides this feature in +``broccoli.h`` because these buffers are used internally anyway and +because they are a typical case of something that people implement +themselves over and over again, for example to collect a set of data +before sending it through a file descriptor, etc. + +The buffers work as follows. The structure implementing a buffer is +called ``BroBuf``, and is initialized to a default size when +created via ``bro_buf_new()`` and released using ``bro_buf_free()``. +Each ``BroBuf`` has a content pointer that points to an arbitrary +location between the start of the buffer and the first byte after the +last byte currently used in the buffer (see ``buf_off`` in the +illustration below). The content pointer can seek to arbitrary +locations, and data can be copied from and into the buffer, adjusting +the content pointer accordingly. You can repeatedly append data to the end +of the buffer's used contents using ``bro_buf_append()``. +:: + + <---------------- allocated buffer space ------------> + <======== used buffer space ========> ^ + ^ ^ ^ | + | | | | + buf buf_ptr buf_off buf_len + +Have a look at the following functions for the details: +``bro_buf_new()``, ``bro_buf_free()``, ``bro_buf_append()``, +``bro_buf_consume()``, ``bro_buf_reset()``, ``bro_buf_get()``, +``bro_buf_get_end()``, ``bro_buf_get_size()``, +``bro_buf_get_used_size()``, ``bro_buf_ptr_get()``, +``bro_buf_ptr_tell()``, ``bro_buf_ptr_seek()``, ``bro_buf_ptr_check()``, +and ``bro_buf_ptr_read()``. + +Configuring Encrypted Communication +=================================== + +Encrypted communication between Bro peers takes place over an SSL +connection in which both endpoints of the connection are authenticated. +This requires at least some PKI in the form of a certificate authority +(CA) which you use to issue and sign certificates for your Bro peers. To +facilitate the SSL setup, each peer requires three documents: a +certificate signed by the CA and containing the public key, the +corresponding private key, and a copy of the CA's certificate. + +The OpenSSL command line tool ``openssl`` can be used to create all +files necessary, but its unstructured arguments and poor documentation +make it a pain to use and waste lots of people a lot of time [#]_. +For an alternative tool to create SSL certificates for secure Bro/Broccoli +communication, see the ``create-cert`` tool available at +ftp://ee.lbl.gov/create-cert.tar.gz. + +In order to enable encrypted communication for your Broccoli +application, you need to put the CA certificate and the peer certificate +in the ``/broccoli/ca_cert`` and ``/broccoli/host_cert`` keys, +respectively, in the configuration file. Optionally, you can store the +private key in a separate file specified by ``/broccoli/host_key``. To +quickly enable/disable a certificate configuration, the +``/broccoli/use_ssl`` key can be used. + +.. note:: + *This is where you configure whether to use encrypted or unencrypted + connections.* + + If the ``/broccoli/use_ssl`` key is present and set to one of "yes", + "true", "on", or 1, then SSL will be used and an incorrect or missing + certificate configuration will cause connection attempts to fail. If + the key's value is one of "no", "false", "off", or 0, then in no case + will SSL be used and connections will always be cleartext. + + If the ``/broccoli/use_ssl`` key is *not* present, then SSL will be + used if a certificate configuration is found, and invalid + certificates will cause the connection to fail. If no certificates + are configured, cleartext connections will be used. + + In no case does an SSL-enabled setup ever fall back to a cleartext + one. + +:: + + /broccoli/use_ssl yes + /broccoli/ca_cert /ca_cert.pem + /broccoli/host_cert /bro_cert.pem + /broccoli/host_key /bro_cert.key + +In a Bro policy, you need to load the ``frameworks/communication/listen.bro`` +script and redef ``Communication::listen_ssl=T``, +``ssl_ca_certificate``, and ``ssl_private_key``, defined in ``bro.init``: + +.. code:: bro + + @load frameworks/communication/listen + + redef Communication::listen_ssl=T; + redef ssl_ca_certificate = "/ca_cert.pem"; + redef ssl_private_key = "/bro.pem"; + +By default, you will be prompted for the passphrase for the private key +matching the public key in your agent's certificate. Depending on your +application's user interface and deployment, this may be inappropriate. +You can store the passphrase in the config file as well, using the +following identifier:: + + /broccoli/host_pass foobar + +.. warning:: *Make sure that access to your configuration is restricted.* + + If you provide the passphrase this way, it is obviously essential to + have restrictive permissions on the configuration file. Broccoli + partially enforces this. Please refer to the section on + `Configuration Files`_ for details. + +Configuring event reception in Bro scripts +========================================== + +Before a remote Bro will accept your connection and your events, it +needs to have its policy configured accordingly: + +1. Load ``frameworks/communication/listen``, and redef the boolean variable + ``Communication::listen_ssl`` depending on whether you want to have + encrypted or cleartext communication. Obviously, encrypting the event + exchange is recommended and cleartext should only be used for early + experimental setups. See below for details on how to set up encrypted + communication via SSL. + +#. You need to find a port to use for the Bros and Broccoli applications + that will listen for connections. Every such agent can use a + different port, though default ports are provided in the Bro + policies. To change the port the Bro agent will be listening on from + its default, redefine the ``Communication::listen_port``. Have a + look at these policies as well as + ``base/frameworks/communication/main.bro`` for the default values. + Here is the policy for the unencrypted case: + + .. code:: bro + + @load frameworks/communication/listen + redef Communication::listen_port = 12345/tcp; + + .. + + Including the settings for the cryptographic files introduced in the + previous section, here is the encrypted one: + + .. code:: bro + + @load frameworks/communication/listen + redef Communication::listen_ssl = T; + redef Communication::listen_port = 12345/tcp; + redef ssl_ca_certificate = "/ca_cert.pem"; + redef ssl_private_key = "/bro.pem"; + + .. + +#. The policy controlling which peers a Bro agent will communicate with + and how this communication will happen are defined in the + ``Communication::nodes`` table defined in + ``base/frameworks/communication/main.bro``. This table contains + entries of type ``Node``, whose members mostly provide default values + so you do not need to define everything. You need to come up with a + tag for the connection under which it can be found in the table (a + creative one would be "broccoli"), the IP address of the peer, the + pattern of names of the events the Bro will accept from you, whether + you want Bro to connect to your machine on startup or not, if so, a + port to connect to (default is ``Communication::default_port`` also defined in + ``base/frameworks/communication/main.bro``), a retry timeout, + whether to use SSL, and the class of a connection as set on the + Broccoli side via ``bro_conn_set_class()``. + + An example could look as follows: + + .. code:: bro + + redef Communication::nodes += { + ["broping"] = [$host = 127.0.0.1, $class="broping", + $events = /ping/, $connect=F, $ssl=F] + }; + + .. + + This example is taken from ``broping.bro``, the policy the remote Bro + must run when you want to use the ``broping`` tool explained in the + section on `test programs`_ below. It will allow an agent on the + local host to connect and send "ping" events. Our Bro will not + attempt to connect, and incoming connections will be expected in + cleartext. + +Configuring Debugging Output +============================ + +If your Broccoli installation was configured with ``--enable-debug``, +Broccoli will report two kinds of debugging information: + +1. function call traces and +#. individual debugging messages. + +Both are enabled by default, but can be adjusted in two ways. + +- In the configuration file: in the appropriate section of the + configuration file, you can set the keys ``/broccoli/debug_messages`` + and ``/broccoli/debug_calltrace`` to ``on``/``off`` to enable/disable + the corresponding output. + +- In code: you can set the variables + ``bro_debug_calltrace`` and ``bro_debug_messages`` to 1/0 at any time + to enable/disable the corresponding output. + +By default, debugging output is inactive (even with debug support +compiled in). You need to enable it explicitly either in your code by +assigning 1 to ``bro_debug_calltrace`` and ``bro_debug_messages`` or by +enabling it in the configuration file. + +Test programs +============= + +The Broccoli distribution comes with a few small test programs, located +in the ``test/`` directory of the tree. The most notable one is +``broping`` [#]_, a mini-version of ping. It sends "ping" events to a +remote Bro agent, expecting "pong" events in return. It operates in two +flavours: one uses atomic types for sending information across, and the +other one uses records. The Bro agent you want to ping needs to run +either the ``broping.bro`` or ``broping-record.bro`` policies. You can +find these in the ``test/`` directory of the source tree, and in +``/share/broccoli`` in the installed version. ``broping.bro`` is +shown below. By default, pinging a Bro on the same machine is +configured. If you want your Bro to be pinged from another machine, you +need to update the ``Communication::nodes`` variable accordingly: + +.. code:: bro + + @load frameworks/communication/listen; + + global ping_log = open_log_file("ping"); + + redef Communication::nodes += { + ["broping"] = [$host = 127.0.0.1, $events = /ping/, + $connect=F, $retry = 60 secs, $ssl=F] + }; + + event ping(src_time: time, seq: count) { + event pong(src_time, current_time(), seq); + } + + event pong(src_time: time, dst_time: time, seq: count) { + print ping_log, + fmt("ping received, seq %d, %f at src, %f at dest, one-way: %f", + seq, src_time, dst_time, dst_time-src_time); + } + +``broping`` sends ping events to Bro. Bro accepts those because they are +configured accordingly in the nodes table. As shown in the +policy, ping events trigger pong events, and ``broccoli`` requests +delivery of all pong events back to it. When running ``broping``, +you'll see something like this: + +.. console:: + + > ./test/broping + pong event from 127.0.0.1: seq=1, time=0.004700/1.010303 s + pong event from 127.0.0.1: seq=2, time=0.053777/1.010266 s + pong event from 127.0.0.1: seq=3, time=0.006435/1.010284 s + pong event from 127.0.0.1: seq=4, time=0.020278/1.010319 s + pong event from 127.0.0.1: seq=5, time=0.004563/1.010187 s + pong event from 127.0.0.1: seq=6, time=0.005685/1.010393 s + +Notes +===== + +.. [#] In other documents and books on OpenSSL you will find this + expressed more politely, using terms such as "daunting to the + uninitiated", "challenging", "complex", "intimidating". + +.. [#] Pronunciation is said to be somewhere on the continuum between + "brooping" and "burping". + +Broccoli API Reference +###################### + +The `API documentation <../../broccoli-api/index.html>`_ +describes Broccoli's public C interface. diff --git a/doc/components/broctl/README.rst b/doc/components/broctl/README.rst deleted file mode 120000 index cba305f48a..0000000000 --- a/doc/components/broctl/README.rst +++ /dev/null @@ -1 +0,0 @@ -../../../aux/broctl/doc/broctl.rst \ No newline at end of file diff --git a/doc/components/broctl/README.rst b/doc/components/broctl/README.rst new file mode 100644 index 0000000000..1fd728c7cf --- /dev/null +++ b/doc/components/broctl/README.rst @@ -0,0 +1,1913 @@ +.. Autogenerated. Do not edit. + +.. -*- mode: rst-mode -*- +.. +.. Note: This file includes further autogenerated ones. +.. +.. Version number is filled in automatically. +.. |version| replace:: 1.1-3 + +========== +BroControl +========== + +.. rst-class:: opening + + This document summarizes installation and use of *BroControl*, + Bro's interactive shell for operating Bro installations. *BroControl* + has two modes of operation: a *stand-alone* mode for + managing a traditional, single-system Bro setup; and a *cluster* + mode for maintaining a multi-system setup of coordinated Bro + instances load-balancing the work across a set of independent + machines. Below, we describe the installation process separately + for the two modes. Once installed, the operation is pretty similar + for both types; just keep in mind that if this document refers to + "nodes" and you're in a stand-alone setup, there is only a + single one and no worker/proxies. + +.. contents:: + +Download +-------- + +You can find the latest BroControl release for download at +http://www.bro.org/download. + +BroControl's git repository is located at +`git://git.bro.org/broctl `_. You +can browse the repository `here `_. + +This document describes BroControl |version|. See the ``CHANGES`` +file for version history. + +Prerequisites +------------- + +Running *BroControl* requires the following prerequisites: + + - A Unix system. FreeBSD, Linux, and MacOS are supported and + should work out of the box. Other Unix systems will quite likely + require some tweaking. Note that in a cluster setup, all systems + must be running exactly the *same* operating system. + + - A version of *Python* >= 2.6. + + - A *bash* (note in particular, that on FreeBSD, *bash* is not + installed by default). + +Installation +------------ + +Stand-alone Bro +~~~~~~~~~~~~~~~ + +For installing a stand-alone Bro setup, just follow the +Bro :doc:`Quick Start Guide<../../quickstart>`. + +Bro Cluster +~~~~~~~~~~~ + +A *Bro Cluster* is a set of systems jointly analyzing the traffic of +a network link in a coordinated fashion. *BroControl* is able to +operate such a setup from a central manager system pretty much +transparently, hiding much of the complexity of the multi-machine +installation. + +A cluster consists of four types of components: + + Frontends. + One or more frontends: Frontends load-balance the traffic + across a set of worker machines. + + Worker nodes. + Workers are doing the actual analysis, with each seeing a + slice of the overall traffic as split by the frontend(s). + + One or more proxies. + Proxies relay the communication between worker nodes. + + One manager. + The manager provides the cluster's user-interface for + controlling and logging. During operation, the user only + interacts with the manager; this is where *BroControl* is + running. + +For more information about the cluster architecture, including options +for the frontend, see the :doc:`Bro Cluster<../../cluster>` documentation. + +This document focuses on the installation of manager, +workers, and the proxies. If not otherwise +stated, in the following we use the terms "manager", "worker", and +"proxy" to refer to Bro instances, not to physical machines; rather, +we use the term "node" to refer to physical machines. There may be +multiple Bro instances running on the same node. For example, it's +possible to run a proxy on the same node as the manager is operating +on. + +In the following, as an example setup, we will assume that our +cluster consists of four nodes (not counting the frontend). The host +names of the systems will be ``host1``, ``host2``, ``host3``, and +``host4``. We will configure the cluster so that ``host1`` runs the +manager and the (only) proxy, and ``host{2,3,4}`` are each running +one worker. This is a typical setup, which will work well for many +sites. + +When installing a cluster, in addition to the prerequisites +mentioned above, you need to + + - have the same user account set up on all nodes. On the worker + nodes, this user must have access to target network interface in + promiscuous mode. ``ssh`` access from the manager node to this + user account must be setup on all machines, and must work + without asking for a password/passphrase. + + - have some storage available on all nodes under the same path, + which we will call the cluster's *prefix* path. In the + following, we will use ``/usr/local/bro`` as an example. The Bro + user must be able to either create this directory or, where it + already exists, must have write permission inside this directory + on all nodes. + + - have ``ssh`` and ``rsync`` installed. + + +With all prerequisites in place, perform the following steps to +install a Bro cluster (as the Bro user) if you install from the Bro source +code (which includes BroControl): + +- Configure and compile the Bro distribution using the cluster's + prefix path as ``--prefix``:: + + > cd /path/to/bro/source/distribution + > ./configure --prefix=/usr/local/bro && make && make install + +- Add ``/bin`` to your ``PATH``. + +- Create a cluster configuration file. There is an example provided, + which you can edit according to the instructions in the file:: + + > cd /usr/local/bro + > vi etc/broctl.cfg + +- Create a node configuration file to define where manager, workers, + and proxies are to run. There is again an example, which defines + the example scenario described above and can be edited as needed:: + + > cd /usr/local/bro + > vi etc/node.cfg + +- Create a network configuration file that lists all of the networks + which the cluster should consider as local to the monitored + environment. Once again, the installation installs a template for + editing:: + + > cd /usr/local/bro + > vi etc/networks.cfg + +- Install workers and proxies using *BroControl*:: + + > broctl install + + This installation process uses ``ssh`` and ``rdist`` to copy the + configuration over to the remote machines so, as described above, + you need to ensure that logging in via SSH works before the install will + succeed. + +- Some tasks need to be run on a regular basis. On the manager node, + insert a line like this into the crontab of the user running the + cluster:: + + 0-59/5 * * * * /bin/broctl cron + +- Finally, you can start the cluster:: + + > broctl start + +Getting Started +--------------- + +*BroControl* is an interactive interface to the cluster which allows +you to, e.g., start/stop the monitoring or update its configuration. +It is started with the ``broctl`` script and then expects commands +on its command-line (alternatively, ``broctl`` can also be started +with a single command directly on the shell's command line):: + + > broctl + Welcome to BroControl x.y + + Type "help" for help. + + [BroControl] > + +As the message says, type help_ to see a list of +all commands. We will now briefly summarize the most important +commands. A full reference follows `Command Reference`_. + +Once ``broctl.cfg`` and ``node.cfg`` are set up as described above, +the monitoring can be started with the start_ command. In the cluster +setup, this will successively start manager, proxies, and workers. The +status_ command should then show all nodes as operating. To stop the +monitoring, issue the stop_ command. exit_ leaves the shell. + +On the manager system (and on the stand-alone system), you find the +current set of (aggregated) logs in ``logs/current`` (which is a +symlink to the corresponding spool directory). The proxies and workers +log into ``spool/proxy/`` and ``spool//``, respectively. +The manager/stand-alone logs are archived in ``logs/``, by default +once a day. Log files of workers and proxies are discarded at the +same rotation interval. + +Whenever the *BroControl* configuration is modified in any way +(including changes to configuration files and site-specific policy +scripts), install_ installs the new version. *No changes will take +effect until* install_ *is run*. Before you run install_, check_ can be +used to check for any potential errors in the new configuration, e.g., +typos in scripts. If check_ does not report any problems, doing +install_ will pretty likely not break anything. + +Note that generally configuration changes only take effect after a +restart of the affected nodes. The restart_ command triggers this. +Some changes however can be put into effect on-the-fly without +restarting any of the nodes by using the update_ command (again only +after doing install_ first). Such dynamic updates generally work with +all changes done which only modify const variables declared as +*redefinable* (i.e., with Bro's *&redef* attribute). + +Generally, site-specific tuning needs to be done with local policy +scripts, as in any Bro setup. This is described in +`Site-specific Customization`_. + +*BroControl* provides various options to control the behavior of +the setup. These options can be set by editing ``broctl.cfg``. +The config_ command gives a list of all options +with their current values. A list of the most important options also +follows `Option Reference`_. + +Site-specific Customization +--------------------------- + +You'll most likely want to adapt the Bro policy to the local +environment and much of the more specific tuning requires writing +local policy files. + +During the initial install, sample local policy scripts (which you can edit) +are installed in ``share/bro/site``. In the stand-alone setup, a single +file called ``local.bro`` gets loaded automatically. In the cluster +setup, the same ``local.bro`` gets loaded, followed by one of three +other files: ``local-manager.bro``, ``local-worker.bro``, and +``local-proxy.bro`` are loaded by the manager, workers, and proxy, +respectively. + +In the cluster setup, the main exception to putting everything into +``local.bro`` is notice filtering, which should be done only on the +manager. + +The next scripts that are loaded are the ones that are automatically +generated by BroControl. These scripts are created from the +``networks.cfg`` and ``broctl.cfg`` files. + +The last scripts loaded are any node-specific scripts specified with the +option ``aux_scripts`` in ``node.cfg``. This option can be used to +load additional scripts to individual nodes only. For example, one could +add a script ``experimental.bro`` to a single worker for trying out new +experimental code. + +The scripts_ command shows precisely which policy scripts get loaded (and +in what order) by a node; that can be very helpful. + +If you want to change which local policy scripts are loaded by the nodes, +you can set SitePolicyStandalone_ for all Bro instances, +SitePolicyManager_ for the manager, and SitePolicyWorker_ for the +workers. To change the directory where local policy scripts are +located, set the option SitePolicyPath_ to a different path. These +options can be changed in the ``broctl.cfg`` file. + +Command Reference +----------------- + +The following summary lists all commands supported by *BroControl*. +All commands may be either entered interactively or specified on the +shell's command line. If not specified otherwise, commands taking +*[]* as arguments apply their action either to the given set of +nodes, or to all nodes if none is given. + +[?1034h.. Automatically generated. Do not edit. + + +.. _attachgdb: + +*attachgdb* *[]* + Primarily for debugging, the command attaches a *gdb* to the main Bro + process on the given nodes. + + +.. _capstats: + +*capstats* *[] []* + Determines the current load on the network interfaces monitored by + each of the given worker nodes. The load is measured over the + specified interval (in seconds), or by default over 10 seconds. This + command uses the `capstats tool + `_, + which is installed along with ``broctl``. + + (Note: When using a CFlow and the CFlow command line utility is + installed as well, the ``capstats`` command can also query the device + for port statistics. *TODO*: document how to set this up.) + + +.. _check: + +*check* *[]* + Verifies a modified configuration in terms of syntactical correctness + (most importantly correct syntax in policy scripts). This command + should be executed for each configuration change *before* + install_ is used to put the change into place. Note + that ``check`` is the only command which operates correctly without a + former install_ command; ``check`` uses the policy + files as found in SitePolicyPath_ to make + sure they compile correctly. If they do, install_ + will then copy them over to an internal place from where the nodes + will read them at the next start_. This approach + ensures that new errors in a policy script will not affect currently + running nodes, even when one or more of them needs to be restarted. + + +.. _cleanup: + +*cleanup* *[--all] []* + Clears the nodes' spool directories (if they are not running + currently). This implies that their persistent state is flushed. Nodes + that were crashed are reset into *stopped* state. If ``--all`` is + specified, this command also removes the content of the node's + TmpDir_, in particular deleteing any data + potentially saved there for reference from previous crashes. + Generally, if you want to reset the installation back into a clean + state, you can first stop_ all nodes, then execute + ``cleanup --all``, and finally start_ all nodes + again. + + +.. _config: + +*config* + Prints all configuration options with their current values. + + +.. _cron: + +*cron* *[enable|disable|?] | [--no-watch]* + This command has two modes of operation. Without arguments (or just + ``--no-watch``), it performs a set of maintenance tasks, including + the logging of various statistical information, expiring old log + files, checking for dead hosts, and restarting nodes which terminated + unexpectedly. The latter can be suppressed with the ``--no-watch`` + option if no auto-restart is desired. This mode is intended to be + executed regularly via *cron*, as described in the installation + instructions. While not intended for interactive use, no harm will be + caused by executing the command manually: all the maintenance tasks + will then just be performed one more time. + + The second mode is for interactive usage and determines if the regular + tasks are indeed performed when ``broctl cron`` is executed. In other + words, even with ``broctl cron`` in your crontab, you can still + temporarily disable its execution by running ``cron disable``, and + then later reenable with ``cron enable``. This can be helpful while + working, e.g., on the BroControl configuration and ``cron`` would + interfere with that. ``cron ?`` can be used to query the current state. + + +.. _df: + +*df* *[]* + Reports the amount of disk space available on the nodes. Shows only + paths relevant to the broctl installation. + + +.. _diag: + +*diag* *[]* + If a node has terminated unexpectedly, this command prints a (somewhat + cryptic) summary of its final state including excerpts of any + stdout/stderr output, resource usage, and also a stack backtrace if a + core dump is found. The same information is sent out via mail when a + node is found to have crashed (the "crash report"). While the + information is mainly intended for debugging, it can also help to find + misconfigurations (which are usually, but not always, caught by the + check_ command). + + +.. _exec: + +*exec* ** + Executes the given Unix shell command line on all nodes configured to + run at least one Bro instance. This is handy to quickly perform an + action across all systems. + + +.. _exit: + +*exit* + Terminates the shell. + + +.. _help: + +*help* + Prints a brief summary of all commands understood by the shell. + + +.. _install: + +*install* + Reinstalls the given nodes, including all configuration files and + local policy scripts. This command must be executed after *all* + changes to any part of the broctl configuration, otherwise the + modifications will not take effect. Usually all nodes should be + reinstalled at the same time, as any inconsistencies between them will + lead to strange effects. Before executing ``install``, it is recommended + to verify the configuration with check_. + + +.. _netstats: + +*netstats* *[]* + Queries each of the nodes for their current counts of captured and + dropped packets. + + +.. _nodes: + +*nodes* + Prints a list of all configured nodes. + + +.. _peerstatus: + +*peerstatus* *[]* + Primarily for debugging, ``peerstatus`` reports statistics about the + network connections cluster nodes are using to communicate with other + nodes. + + +.. _print: + +*print* * []* + Reports the *current* live value of the given Bro script ID on all of + the specified nodes (which obviously must be running). This can for + example be useful to (1) check that policy scripts are working as + expected, or (2) confirm that configuration changes have in fact been + applied. Note that IDs defined inside a Bro namespace must be + prefixed with ``::`` (e.g., ``print SSH::did*ssh*version`` to + print the corresponding table from ``ssh.bro``.) + + +.. _process: + +*process* * [options] [-- ]* + Runs Bro offline on a given trace file using the same configuration as + when running live. It does, however, use the potentially + not-yet-installed policy files in SitePolicyPath_ and disables log + rotation. Additional Bro command line flags and scripts can + be given (each argument after a ``--`` argument is interpreted as + a script). + + Upon completion, the command prints a path where the log files can be + found. Subsequent runs of this command may delete these logs. + + In cluster mode, Bro is run with *both* manager and worker scripts + loaded into a single instance. While that doesn't fully reproduce the + live setup, it is often sufficient for debugging analysis scripts. + + +.. _quit: + +*quit* + Terminates the shell. + + +.. _restart: + +*restart* *[--clean] []* + Restarts the given nodes, or all nodes if none are specified. The + effect is the same as first executing stop_ followed + by a start_, giving the same nodes in both cases. + This command is most useful to activate any changes made to Bro policy + scripts (after running install_ first). Note that a + subset of policy changes can also be installed on the fly via the + update_, without requiring a restart. + + If ``--clean`` is given, the installation is reset into a clean state + before restarting. More precisely, a ``restart --clean`` turns into + the command sequence stop_, cleanup_ --all, check_, install_, and + start_. + + +.. _scripts: + +*scripts* *[-c] []* + Primarily for debugging Bro configurations, the ``scripts`` + command lists all the Bro scripts loaded by each of the nodes in the + order they will be parsed by the node at startup. + If ``-c`` is given, the command operates as check_ does: it reads + the policy files from their *original* location, not the copies + installed by install_. The latter option is useful to check a + not yet installed configuration. + + +.. _start: + +*start* *[]* + Starts the given nodes, or all nodes if none are specified. Nodes + already running are left untouched. + + +.. _status: + +*status* *[]* + Prints the current status of the given nodes. + + +.. _stop: + +*stop* *[]* + Stops the given nodes, or all nodes if none are specified. Nodes not + running are left untouched. + + +.. _top: + +*top* *[]* + For each of the nodes, prints the status of the two Bro + processes (parent process and child process) in a *top*-like + format, including CPU usage and memory consumption. If + executed interactively, the display is updated frequently + until key ``q`` is pressed. If invoked non-interactively, the + status is printed only once. + + +.. _update: + +*update* *[]* + After a change to Bro policy scripts, this command updates the Bro + processes on the given nodes *while they are running* (i.e., without + requiring a restart_). However, such dynamic + updates work only for a *subset* of Bro's full configuration. The + following changes can be applied on the fly: The value of all + const variables defined with the ``&redef`` attribute can be changed. + More extensive script changes are not possible during runtime and + always require a restart; if you change more than just the values of + ``&redef``-able consts and still issue ``update``, the results are + undefined and can lead to crashes. Also note that before running + ``update``, you still need to do an install_ (preferably after + check_), as otherwise ``update`` will not see the changes and it will + resend the old configuration. + + +Option Reference +---------------- + +This section summarizes the options that can be set in ``broctl.cfg`` +for customizing the behavior of *BroControl*. Usually, one only needs +to change the "user options", which are listed first. The "internal +options" are, as the name suggests, primarily used internally and set +automatically. They are documented here only for reference. + +.. Automatically generated. Do not edit. + +User Options +~~~~~~~~~~~~ +.. _BroArgs: + +*BroArgs* (string, default _empty_) + Additional arguments to pass to Bro on the command-line. + +.. _CFlowAddress: + +*CFlowAddress* (string, default _empty_) + If a cFlow load-balancer is used, the address of the device (format: :). + +.. _CFlowPassword: + +*CFlowPassword* (string, default _empty_) + If a cFlow load-balancer is used, the password for accessing its configuration interface. + +.. _CFlowUser: + +*CFlowUser* (string, default _empty_) + If a cFlow load-balancer is used, the user name for accessing its configuration interface. + +.. _CommTimeout: + +*CommTimeout* (int, default 10) + The number of seconds to wait before assuming Broccoli communication events have timed out. + +.. _CompressCmd: + +*CompressCmd* (string, default "gzip -9") + If archived logs will be compressed, the command to use for that. The specified command must compress its standard input to standard output. + +.. _CompressExtension: + +*CompressExtension* (string, default "gz") + If archived logs will be compressed, the file extension to use on compressed log files. When specifying a file extension, don't include the period character (e.g., specify 'gz' instead of '.gz'). + +.. _CompressLogs: + +*CompressLogs* (bool, default 1) + True to compress archived log files. + +.. _CronCmd: + +*CronCmd* (string, default _empty_) + A custom command to run everytime the cron command has finished. + +.. _Debug: + +*Debug* (bool, default 0) + Enable extensive debugging output in spool/debug.log. + +.. _HaveNFS: + +*HaveNFS* (bool, default 0) + True if shared files are mounted across all nodes via NFS (see FAQ). + +.. _IPv6Comm: + +*IPv6Comm* (bool, default 1) + Enable IPv6 communication between cluster nodes (and also between them and BroControl) + +.. _LogDir: + +*LogDir* (string, default "$\{BroBase}/logs") + Directory for archived log files. + +.. _LogExpireInterval: + +*LogExpireInterval* (int, default 0) + Number of days log files are kept (zero means disabled). + +.. _LogRotationInterval: + +*LogRotationInterval* (int, default 3600) + The frequency of log rotation in seconds for the manager/standalone node. + +.. _MailAlarmsTo: + +*MailAlarmsTo* (string, default "$\{MailTo}") + Destination address for alarm summary mails. Default is to use the same address as MailTo. + +.. _MailFrom: + +*MailFrom* (string, default "Big Brother ") + Originator address for broctl-generated mails. + +.. _MailReplyTo: + +*MailReplyTo* (string, default _empty_) + Reply-to address for broctl-generated mails. + +.. _MailSubjectPrefix: + +*MailSubjectPrefix* (string, default "[Bro]") + General Subject prefix for mails. + +.. _MailTo: + +*MailTo* (string, default "") + Destination address for non-alarm mails. + +.. _MakeArchiveName: + +*MakeArchiveName* (string, default "$\{BroBase}/share/broctl/scripts/make-archive-name") + Script to generate filenames for archived log files. + +.. _MemLimit: + +*MemLimit* (string, default "unlimited") + Maximum amount of memory for Bro processes to use (in KB, or the string 'unlimited'). + +.. _MinDiskSpace: + +*MinDiskSpace* (int, default 5) + Percentage of minimum disk space available before warning is mailed. + +.. _PFRINGClusterID: + +*PFRINGClusterID* (int, default @PF_RING_CLUSTER_ID@) + If PF_RING flow-based load balancing is desired, this is where the PF_RING cluster id is defined. The default value is configuration-dependent and determined automatically by CMake at configure-time based upon whether PF_RING's enhanced libpcap is available. Bro must be linked with PF_RING's libpcap wrapper for this option to work. + +.. _Prefixes: + +*Prefixes* (string, default "local") + Additional script prefixes for Bro, separated by colons. Use this instead of @prefix. + +.. _SaveTraces: + +*SaveTraces* (bool, default 0) + True to let backends capture short-term traces via '-w'. These are not archived but might be helpful for debugging. + +.. _SendMail: + +*SendMail* (string, default "@SENDMAIL@") + Location of the sendmail binary. Make this string blank to prevent email from being sent. The default value is configuration-dependent and determined automatically by CMake at configure-time. + +.. _SitePluginPath: + +*SitePluginPath* (string, default _empty_) + Directories to search for custom plugins, separated by colons. + +.. _SitePolicyManager: + +*SitePolicyManager* (string, default "local-manager.bro") + Space-separated list of local policy files for manager. + +.. _SitePolicyPath: + +*SitePolicyPath* (string, default "$\{PolicyDir}/site") + Directories to search for local policy files, separated by colons. + +.. _SitePolicyStandalone: + +*SitePolicyStandalone* (string, default "local.bro") + Space-separated list of local policy files for all Bro instances. + +.. _SitePolicyWorker: + +*SitePolicyWorker* (string, default "local-worker.bro") + Space-separated list of local policy files for workers. + +.. _StopTimeout: + +*StopTimeout* (int, default 60) + The number of seconds to wait before sending a SIGKILL to a node which was previously issued the 'stop' command but did not terminate gracefully. + +.. _TimeFmt: + +*TimeFmt* (string, default "%d %b %H:%M:%S") + Format string to print date/time specifications (see 'man strftime'). + +.. _TimeMachineHost: + +*TimeMachineHost* (string, default _empty_) + If the manager should connect to a Time Machine, the address of the host it is running on. + +.. _TimeMachinePort: + +*TimeMachinePort* (string, default "47757/tcp") + If the manager should connect to a Time Machine, the port it is running on (in Bro syntax, e.g., 47757/tcp). + +.. _ZoneID: + +*ZoneID* (string, default _empty_) + If the host running BroControl is managing a cluster comprised of nodes with non-global IPv6 addresses, this option indicates what RFC 4007 zone_id to append to node addresses when communicating with them. + + +Internal Options +~~~~~~~~~~~~~~~~ + +.. _BinDir: + +*BinDir* (string, default "$\{BroBase}/bin") + Directory for executable files. + +.. _BroBase: + +*BroBase* (string, default _empty_) + Base path of broctl installation on all nodes. + +.. _CapstatsPath: + +*CapstatsPath* (string, default "$\{bindir}/capstats") + Path to capstats binary; empty if not available. + +.. _CfgDir: + +*CfgDir* (string, default "$\{BroBase}/etc") + Directory for configuration files. + +.. _DebugLog: + +*DebugLog* (string, default "$\{SpoolDir}/debug.log") + Log file for debugging information. + +.. _HelperDir: + +*HelperDir* (string, default "$\{BroBase}/share/broctl/scripts/helpers") + Directory for broctl helper scripts. + +.. _LibDir: + +*LibDir* (string, default "$\{BroBase}/lib") + Directory for library files. + +.. _LibDirInternal: + +*LibDirInternal* (string, default "$\{BroBase}/lib/broctl") + Directory for broctl-specific library files. + +.. _LocalNetsCfg: + +*LocalNetsCfg* (string, default "$\{CfgDir}/networks.cfg") + File defining the local networks. + +.. _LockFile: + +*LockFile* (string, default "$\{SpoolDir}/lock") + Lock file preventing concurrent shell operations. + +.. _NodeCfg: + +*NodeCfg* (string, default "$\{CfgDir}/node.cfg") + Node configuration file. + +.. _OS: + +*OS* (string, default _empty_) + Name of operating system as reported by uname. + +.. _PluginDir: + +*PluginDir* (string, default "$\{LibDirInternal}/plugins") + Directory where standard plugins are located. + +.. _PolicyDir: + +*PolicyDir* (string, default "$\{BroBase}/share/bro") + Directory for standard policy files. + +.. _PolicyDirSiteInstall: + +*PolicyDirSiteInstall* (string, default "$\{SpoolDir}/installed-scripts-do-not-touch/site") + Directory where the shell copies local policy scripts when installing. + +.. _PolicyDirSiteInstallAuto: + +*PolicyDirSiteInstallAuto* (string, default "$\{SpoolDir}/installed-scripts-do-not-touch/auto") + Directory where the shell copies auto-generated local policy scripts when installing. + +.. _PostProcDir: + +*PostProcDir* (string, default "$\{BroBase}/share/broctl/scripts/postprocessors") + Directory for log postprocessors. + +.. _ScriptsDir: + +*ScriptsDir* (string, default "$\{BroBase}/share/broctl/scripts") + Directory for executable scripts shipping as part of broctl. + +.. _SpoolDir: + +*SpoolDir* (string, default "$\{BroBase}/spool") + Directory for run-time data. + +.. _StandAlone: + +*StandAlone* (bool, default 0) + True if running in stand-alone mode (see elsewhere). + +.. _StateFile: + +*StateFile* (string, default "$\{SpoolDir}/broctl.dat") + File storing the current broctl state. + +.. _StaticDir: + +*StaticDir* (string, default "$\{BroBase}/share/broctl") + Directory for static, arch-independent files. + +.. _StatsDir: + +*StatsDir* (string, default "$\{LogDir}/stats") + Directory where statistics are kept. + +.. _StatsLog: + +*StatsLog* (string, default "$\{SpoolDir}/stats.log") + Log file for statistics. + +.. _Time: + +*Time* (string, default _empty_) + Path to time binary. + +.. _TmpDir: + +*TmpDir* (string, default "$\{SpoolDir}/tmp") + Directory for temporary data. + +.. _TmpExecDir: + +*TmpExecDir* (string, default "$\{SpoolDir}/tmp") + Directory where binaries are copied before execution. + +.. _TraceSummary: + +*TraceSummary* (string, default "$\{bindir}/trace-summary") + Path to trace-summary script (empty if not available). Make this string blank to disable the connection summary emails. + +.. _Version: + +*Version* (string, default _empty_) + Version of the broctl. + + +Writing Plugins +--------------- + +BroControl provides a plugin interface to extend its functionality. A +plugin is written in Python and can do any, or all, of the following: + + * Perform actions before or after any of the standard BroControl + commands is executed. When running before the actual command, it + can filter which nodes to operate or stop the execution + altogether. When running after the command, it gets access to + the commands success on a per-node basis (where applicable). + + * Add custom commands to BroControl. + + * Add custom options to BroControl defined in ``broctl.cfg``. + + * Add custom keys to nodes defined in ``node.cfg``. + +A plugin is written by deriving a new class from BroControl class +`Plugin`_. The Python script with the new plugin is then copied into a +plugin directory searched by BroControl at startup. By default, +BroControl searches ``/lib/broctl/plugins``; further may be +configured by setting the PluginDir_ option. Note that any plugin +script must end in ``*.py`` to be found. BroControl comes with some +example plugins that can be used as a starting point; see +the ``/lib/broctl/plugins`` directory. + +In the following, we document the API that is available to plugins. A +plugin must be derived from the `Plugin`_ class, and can use its +methods as well as those of the `Node`_ class. + +.. _Plugin: + +Class ``Plugin`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +class **Plugin** + The class ``Plugin`` is the base class for all BroControl plugins. + + The class has a number of methods for plugins to override, and every + plugin must at least override ``name()`` and ``pluginVersion()``. + + For each BroControl command ``foo``, there are two methods, + ``cmd_foo_pre`` and ``cmd_foo_post``, that are called just before the + command is executed and just after it has finished, respectively. The + arguments these methods receive correspond to their command-line + parameters, and are further documented below. + + The ``cmd__pre`` methods have the ability to prevent the command's + execution, either completely or partially for those commands that take + nodes as parameters. In the latter case, the method receives a list of + nodes that the command is to be run on, and it can filter that list and + returns modified version of nodes to actually use. The standard case would + be returning simply the unmodified ``nodes`` parameter. To completely + block the command's execution, return an empty list. To just not execute + the command for a subset, remove the affected ones. For commands that do + not receive nodes as arguments, the return value is interpreted as boolean + indicating whether command execution should proceed (True) or not (False). + + The ``cmd__post`` methods likewise receive the commands arguments as + their parameter, as documented below. For commands taking nodes, the list + corresponds to those nodes for which the command was actually executed + (i.e., after any ``cmd__pre`` filtering). Each node is given as a + tuple ``(node, bool)`` with *node* being the actual `Node`_, and the boolean + indicating whether the command was successful for it. + + Note that if a plugin prevents a command from executing either completely or + partially, it should report its reason via the ``message*()`` or + ``error()`` methods. + + If multiple plugins hook into the same command, all their + ``cmd__{pre,post}`` are executed in undefined order. The command is + executed on the intersection of all ``cmd__pre`` results. + + Finally, note that the ``restart`` command doesn't have its own method as + it's just a combination of other commands and thus their callbacks are + run. + + .. _Plugin.debug: + + **debug** (self, msg) + + Logs a debug message in BroControl's debug log if enabled. + + .. _Plugin.error: + + **error** (self, msg) + + Reports an error to the user. + + .. _Plugin.execute: + + **execute** (self, node, cmd) + + Executes a command on the host for the given *node* of type + `Node`_. Returns a tuple ``(success, output)`` in which ``success`` is + True if the command ran successfully and ``output`` is the combined + stdout/stderr output. + + .. _Plugin.executeParallel: + + **executeParallel** (self, cmds) + + Executes a set of commands in parallel on multiple hosts. ``cmds`` + is a list of tuples ``(node, cmd)``, in which the *node* is a `Node`_ + instance and *cmd* is a string with the command to execute for it. The + method returns a list of tuples ``(node, success, output)``, in which + ``success`` is True if the command ran successfully and ``output`` is + the combined stdout/stderr output for the corresponding ``node``. + + .. _Plugin.getGlobalOption: + + **getGlobalOption** (self, name) + + Returns the value of the global BroControl option or state + attribute *name*. If the user has not set the options, its default + value is returned. See the output of ``broctl config`` for a complete + list. + + .. _Plugin.getOption: + + **getOption** (self, name) + + Returns the value of one of the plugin's options, *name*. The + returned value will always be a string. + + An option has a default value (see *options()*), which can be + overridden by a user in ``broctl.cfg``. An option's value cannot be + changed by the plugin. + + .. _Plugin.getState: + + **getState** (self, name) + + Returns the current value of one of the plugin's state variables, + *name*. The returned value will always be a string. If it has not yet + been set, an empty string will be returned. + + Different from options, state variables can be set by the plugin and + are persistent across restarts. They are not visible to the user. + + Note that a plugin cannot query any global BroControl state variables. + + .. _Plugin.hosts: + + **hosts** (self, nodes) + + Returns a list of all hosts running at least one node from the list + of Nodes_ objects in *nodes*, or configured in if *nodes* is empty. + + .. _Plugin.message: + + **message** (self, msg) + + Reports a message to the user. + + .. _Plugin.nodes: + + **nodes** (self) + + Returns a list of all configured `Node`_ objects. + + .. _Plugin.parseNodes: + + **parseNodes** (self, names) + + Returns `Node`_ objects for a string of space-separated node names. + If a name does not correspond to a known node, an error message is + printed and the node is skipped from the returned list. If no names + are known, an empty list is returned. + + .. _Plugin.setState: + + **setState** (self, name, value) + + Sets one of the plugin's state variables, *name*, to *value*. + *value* must be a string. The change is permanent and will be recorded + to disk. + + Note that a plugin cannot change any global BroControl state + variables. + + .. _Plugin.broProcessDied: + + **broProcessDied** (self, node) + + Called when BroControl finds the Bro process for Node_ *node* + to have terminated unexpectedly. This method will be called just + before BroControl prepares the node's "crash report" and before it + cleans up the node's spool directory. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_attachgdb_post: + + **cmd_attachgdb_post** (self, nodes) + + Called just after the ``attachgdb`` command has finished. Arguments + are as with the ``pre`` method. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_attachgdb_pre: + + **cmd_attachgdb_pre** (self, nodes) + + Called just before the ``attachgdb`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_capstats_post: + + **cmd_capstats_post** (self, nodes, interval) + + Called just after the ``capstats`` command has finished. Arguments + are as with the ``pre`` method. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_capstats_pre: + + **cmd_capstats_pre** (self, nodes, interval) + + Called just before the ``capstats`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. *integer* is an integer with the measurement interval in + seconds. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_check_post: + + **cmd_check_post** (self, results) + + Called just after the ``check`` command has finished. It receives + the list of 2-tuples ``(node, bool)`` indicating the nodes the command + was executed for, along with their success status. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_check_pre: + + **cmd_check_pre** (self, nodes) + + Called just before the ``check`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_cleanup_post: + + **cmd_cleanup_post** (self, nodes, all) + + Called just after the ``cleanup`` command has finished. Arguments + are as with the ``pre`` method. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_cleanup_pre: + + **cmd_cleanup_pre** (self, nodes, all) + + Called just before the ``cleanup`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. *all* is boolean indicating whether the ``--all`` + argument has been given. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_config_post: + + **cmd_config_post** (self) + + Called just after the ``config`` command has finished. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_config_pre: + + **cmd_config_pre** (self) + + Called just before the ``config`` command is run. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_cron_post: + + **cmd_cron_post** (self, arg, watch) + + Called just after the ``cron`` command has finished. Arguments are + as with the ``pre`` method. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_cron_pre: + + **cmd_cron_pre** (self, arg, watch) + + Called just before the ``cron`` command is run. *arg* is None if + the cron is executed without arguments. Otherwise, it is one of the + strings: ``enable``, ``disable``, ``?``. *watch* is a boolean + indicating whether ``cron`` should restart abnormally terminated Bro + processes; it's only valid if arg is empty. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_custom: + + **cmd_custom** (self, cmd, args) + + Called when command defined by the ``commands`` method is executed. + ``cmd`` is the command (with the plugin's prefix), and ``args`` is a + single *string* with all arguments. + + If the arguments are actually node names, ``parseNodes`` can + be used to get the `Node`_ objects. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_df_post: + + **cmd_df_post** (self, nodes) + + Called just after the ``df`` command has finished. Arguments are as + with the ``pre`` method. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_df_pre: + + **cmd_df_pre** (self, nodes) + + Called just before the ``df`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_diag_post: + + **cmd_diag_post** (self, nodes) + + Called just after the ``diag`` command has finished. Arguments are + as with the ``pre`` method. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_diag_pre: + + **cmd_diag_pre** (self, nodes) + + Called just before the ``diag`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_exec_post: + + **cmd_exec_post** (self, cmdline) + + Called just after the ``exec`` command has finished. Arguments are + as with the ``pre`` method. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_exec_pre: + + **cmd_exec_pre** (self, cmdline) + + Called just before the ``exec`` command is run. *cmdline* is a + string with the command line to execute. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_install_post: + + **cmd_install_post** (self) + + Called just after the ``install`` command has finished. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_install_pre: + + **cmd_install_pre** (self) + + Called just before the ``install`` command is run. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_netstats_post: + + **cmd_netstats_post** (self, nodes) + + Called just after the ``netstats`` command has finished. Arguments + are as with the ``pre`` method. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_netstats_pre: + + **cmd_netstats_pre** (self, nodes) + + Called just before the ``netstats`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_nodes_post: + + **cmd_nodes_post** (self) + + Called just after the ``nodes`` command has finished. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_nodes_pre: + + **cmd_nodes_pre** (self) + + Called just before the ``nodes`` command is run. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_peerstatus_post: + + **cmd_peerstatus_post** (self, nodes) + + Called just after the ``peerstatus`` command has finished. + Arguments are as with the ``pre`` method. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_peerstatus_pre: + + **cmd_peerstatus_pre** (self, nodes) + + Called just before the ``peerstatus`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_print_post: + + **cmd_print_post** (self, nodes, id) + + Called just after the ``print`` command has finished. Arguments are + as with the ``pre`` method. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_print_pre: + + **cmd_print_pre** (self, nodes, id) + + Called just before the ``print`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. *id* is a string with the name of the ID to be printed. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_process_post: + + **cmd_process_post** (self, trace, options, scripts, success) + + Called just after the ``process`` command has finished. Arguments + are as with the ``pre`` method, plus an additional boolean *success* + indicating whether Bro terminated normally. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_process_pre: + + **cmd_process_pre** (self, trace, options, scripts) + + Called just before the ``process`` command is run. It receives the + *trace* to read from as a string, a list of additional Bro *options*, + and a list of additional Bro scripts. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_restart_post: + + **cmd_restart_post** (self, results) + + Called just after the ``restart`` command has finished. It receives + the list of 2-tuples ``(node, bool)`` indicating the nodes the command + was executed for, along with their success status. The remaining + arguments are as with the ``pre`` method. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_restart_pre: + + **cmd_restart_pre** (self, nodes, clean) + + Called just before the ``restart`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. *clean* is boolean indicating whether the ``--clean`` + argument has been given. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_scripts_post: + + **cmd_scripts_post** (self, nodes, full_path, check) + + Called just after the ``scripts`` command has finished. Arguments + are as with the ``pre`` method. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_scripts_pre: + + **cmd_scripts_pre** (self, nodes, full_path, check) + + Called just before the ``scripts`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. ``full_path`` and ``check`` are boolean indicating + whether the ``-p`` and ``-c`` options were given, respectively. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_start_post: + + **cmd_start_post** (self, results) + + Called just after the ``start`` command has finished. It receives + the list of 2-tuples ``(node, bool)`` indicating the nodes the command + was executed for, along with their success status. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_start_pre: + + **cmd_start_pre** (self, nodes) + + Called just before the ``start`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_status_post: + + **cmd_status_post** (self, nodes) + + Called just after the ``status`` command has finished. Arguments + are as with the ``pre`` method. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_status_pre: + + **cmd_status_pre** (self, nodes) + + Called just before the ``status`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_stop_post: + + **cmd_stop_post** (self, results) + + Called just after the ``stop`` command has finished. It receives + the list of 2-tuples ``(node, bool)`` indicating the nodes the command + was executed for, along with their success status. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_stop_pre: + + **cmd_stop_pre** (self, nodes) + + Called just before the ``stop`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_top_post: + + **cmd_top_post** (self, nodes) + + Called just after the ``top`` command has finished. Arguments are + as with the ``pre`` method. Note that when ``top`` is run + interactively to auto-refresh continuously, this method will be called + once after each update. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_top_pre: + + **cmd_top_pre** (self, nodes) + + Called just before the ``top`` command is run. It receives the list + of nodes, and returns the list of nodes that should proceed with the + command. Note that when ``top`` is run interactively to auto-refresh + continuously, this method will be called once before each update. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_update_post: + + **cmd_update_post** (self, results) + + Called just after the ``update`` command has finished. It receives + the list of 2-tuples ``(node, bool)`` indicating the nodes the command + was executed for, along with their success status. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.cmd_update_pre: + + **cmd_update_pre** (self, nodes) + + Called just before the ``update`` command is run. It receives the + list of nodes, and returns the list of nodes that should proceed with + the command. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.commands: + + **commands** (self) + + Returns a set of custom commands provided by the + plugin. + + The return value is a list of 3-tuples each having the following + elements: + + ``command`` + A string with the command's name. Note that the command name + exposed to the user will be prefixed with the plugin's prefix + as returned by *name()* (e.g., ``myplugin.mycommand``). + + ``arguments`` + A string describing the command's arguments in a textual form + suitable for use in the ``help`` command summary (e.g., + ``[]`` for command taking an optional list of nodes). + Empty if no arguments are expected. + + ``description`` + A string with a description of the command's semantics. + + + This method can be overridden by derived classes. The implementation + must not call the parent class' implementation. The default + implementation returns an empty list. + + .. _Plugin.done: + + **done** (self) + + Called once just before BroControl terminates. This method can do + any cleanup the plugin may require. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.hostStatusChanged: + + **hostStatusChanged** (self, host, status) + + Called when BroControl's ``cron`` command finds the availability of + a cluster system to have changed. Initially, all systems are assumed + to be up and running. Once BroControl notices that a system isn't + responding (defined as either it doesn't ping at all, or does not + accept SSH sessions), it calls this method, passing in a string with + the name of the *host* and a boolean *status* set to False. Once the + host becomes available again, the method will be called again for the + same host with *status* now set to True. + + Note that BroControl's ``cron`` tracks a host's availability across + execution, so if the next time it's run the host is still down, this + method will not be called again. + + This method can be overridden by derived classes. The default + implementation does nothing. + + .. _Plugin.init: + + **init** (self) + + Called once just before BroControl starts executing any commands. + This method can do any initialization that the plugin may require. + + Note that when this method executes, BroControl guarantees that all + internals are fully set up (e.g., user-defined options are available). + This may not be the case when the class ``__init__`` method runs. + + Returns a boolean, indicating whether the plugin should be used. If it + returns ``False``, the plugin will be removed and no other methods + called. + + This method can be overridden by derived classes. The default + implementation always returns True. + + .. _Plugin.name: + + **name** (self) + + Returns a string with a descriptive *name* for the plugin (e.g., + ``"TestPlugin"``). The name must not contain any whitespace. + + This method must be overridden by derived classes. The implementation + must not call the parent class' implementation. + + .. _Plugin.nodeKeys: + + **nodeKeys** (self) + + Returns a list of custom keys for ``node.cfg``. The value for a + key will be available from the `Node`_ object as attribute + ``_`` (e.g., ``node.test_mykw``). If not set, the + attribute will be set to None. + + This method can be overridden by derived classes. The implementation + must not call the parent class' implementation. The default + implementation returns an empty list. + + .. _Plugin.options: + + **options** (self) + + Returns a set of local configuration options provided by the + plugin. + + The return value is a list of 4-tuples each having the following + elements: + + ``name`` + A string with name of the option (e.g., ``Path``). Option + names are case-insensitive. Note that the option name exposed + to the user will be prefixed with your plugin's prefix as + returned by *name()* (e.g., ``myplugin.Path``). + + ``type`` + A string with type of the option, which must be one of + ``"bool"``, ``"string"``, or ``"int"``. + + ``default`` + A string with the option's default value. Note that this must + always be a string, even for non-string types. For booleans, + use ``"0"`` for False and ``"1"`` for True. For integers, give + the value as a string ``"42"``. + + ``description`` + A string with a description of the option semantics. + + This method can be overridden by derived classes. The implementation + must not call the parent class' implementation. The default + implementation returns an empty list. + + .. _Plugin.pluginVersion: + + **pluginVersion** (self) + + Returns an integer with a version number for the plugin. Plugins + should increase their version number with any significant change. + + This method must be overridden by derived classes. The implementation + must not call the parent class' implementation. + + .. _Plugin.prefix: + + **prefix** (self) + + Returns a string with a prefix for the plugin's options and + commands names (e.g., "myplugin"). + + This method can be overridden by derived classes. The implementation + must not call the parent class' implementation. The default + implementation returns a lower-cased version of *name()*. + +.. _Node: + +Class ``Node`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +class **Node** + Class representing one node of the BroControl maintained setup. In + standalone mode, there's always exactly one node of type ``standalone``. In + a cluster setup, there is exactly one of type ``manager``, one or + more of type ``proxy``, and zero or more of type ``worker``. + + In addition to the methods described above, a ``Node`` object has a number + of keys with values that are set via ``node.cfg`` and can be accessed + directly via corresponding Python attributes (e.g., ``node.name``): + + ``name`` (string) + The name of the node, which corresponds to the ``[]`` + section in ``node.cfg``. + + ``type`` (string) + The type of the node, which will be one of ``standalone``, + ``manager``, ``proxy``, and ``worker``. + + ``host`` (string) + The hostname of the system the node is running on. + + ``interface`` (string) + The network interface for Bro to use; empty if not set. + + ``lb_procs`` (integer) + The number of clustered Bro workers you'd like to start up. + + ``lb_method`` (string) + The load balancing method to distribute packets to all of the + processes (must be one of: ``pf_ring``, ``myricom``, or + ``interfaces``). + + ``lb_interfaces`` (string) + If the load balancing method is ``interfaces``, then this is + a comma-separated list of network interface names to use. + + ``aux_scripts`` (string) + Any node-specific Bro script configured for this node. + + ``zone_id`` (string) + If BroControl is managing a cluster comprised of nodes + using non-global IPv6 addresses, then this configures the + RFC 4007 ``zone_id`` string that the node associates with + the common zone that all cluster nodes are a part of. This + identifier may differ between nodes. + + Any attribute that is not defined in ``node.cfg`` will be empty. + + In addition, plugins can override `Plugin.nodeKeys`_ to define their own + node keys, which can then be likewise set in ``node.cfg``. The key names + will be prepended with the plugin's `Plugin.prefix`_ (e.g., for the plugin + ``test``, the node key ``foo`` is set by adding ``test.foo=value`` to + ``node.cfg``). + + .. _Node.cwd: + + **cwd** (self) + + Returns a string with the node's working directory. + + .. _Node.describe: + + **describe** (self) + + Returns an extended string representation of the node including all + its keys with values. + + .. _Node.getPID: + + **getPID** (self) + + Returns the process ID of the node's Bro process if running, and + None otherwise. + + .. _Node.getPort: + + **getPort** (self) + + Returns an integer with the port that this node's communication + system is listening on for incoming connections, or -1 if no such port + has been set yet. + + .. _Node.hasCrashed: + + **hasCrashed** (self) + + Returns True if the node's Bro process has exited abnormally. + + +Miscellaneous +------------- + +Mails +~~~~~ + +*BroControl* sends four types of mails to the address given in +``MailTo``: + +1. When logs are rotated (per default once a day), a list of all + alarms during the last rotation interval is sent. This can be + disabled by setting ``MailAlarms=0``. + +2. When the ``cron`` command notices that a node has crashed, it + restarts it and sends a notification. It may also send a more + detailed crash report containing information about the crash. + +3. NOTICES with a notice action ``EMAIL``. + +4. If `trace-summary `_ + is installed, a traffic summary is sent each rotation interval. + +Performance Analysis +~~~~~~~~~~~~~~~~~~~~ + +*TODO*: ``broctl cron`` logs a number of statistics, which can be +analyzed/plotted for understanding the clusters run-time behavior. + +Questions and Answers +--------------------- + +*Can I use an NFS-mounted partition as the cluster's base directory to avoid the ``rsync``'ing?* + Yes. BroBase_ can be on an NFS partition. + Configure and install the shell as usual with + ``--prefix=``. Then add ``HaveNFS=1`` and + ``SpoolDir=`` to ``broctl.cfg``, where ```` is a + path on the local disks of the nodes; ```` will be used for + all non-shared data (make sure that the parent directory exists + and is writable on all nodes!). Then run ``make install`` again. + Finally, you can remove ``/spool`` (or link it to ). + In addition, you might want to keep the log files locally on the nodes + as well by setting LogDir_ to a non-NFS directory. (Only + the manager's logs will be kept permanently, the logs of + workers/proxies are discarded upon rotation.) + +*When I'm using the stand-alone mode, do I still need to have ``ssh`` and ``rsync`` installed and configured?* + No. In stand-alone mode all operations are performed directly on the local + file system. + +*What do I need to do when something in the Bro distribution changes?* + After pulling from the main Bro git repository, just re-run ``make + install`` inside your build directory. It will reinstall all the + files from the distribution that are not up-to-date. Then do + ``broctl install`` to make sure everything gets pushed out. + +*Can I change the naming scheme that BroControl uses for archived log files?* + Yes, set MakeArchiveName_ to a + script that outputs the desired destination file name for an + archived log file. The default script for that task is + ``/share/broctl/scripts/make-archive-name``, which you + can use as a template for creating your own version. See + the beginning of that script for instructions. + +*Can BroControl manage a cluster of nodes over non-global IPv6 scope (e.g. link-local)?* + Yes, set ``ZoneID`` in ``broctl.cfg`` to the zone identifier + that the BroControl node uses to identify the scope zone + (the ``ifconfig`` command output is usually helpful, if it doesn't + show the zone identifier appended to the address with a '%' + character, then it may just be the interface name). Then in + ``node.cfg``, add a ``zone_id`` key to each node section + representing that particular node's zone identifier and set + the ``host`` key to the IPv6 address assigned to the node within + the scope zone. Most nodes probably have the same ``zone_id``, but + may not if their interface configuration differs. See RFC 4007 for + more information on IPv6 scoped addresses and zones. diff --git a/doc/components/btest/README.rst b/doc/components/btest/README.rst deleted file mode 120000 index 0da2935df1..0000000000 --- a/doc/components/btest/README.rst +++ /dev/null @@ -1 +0,0 @@ -../../../aux/btest/README \ No newline at end of file diff --git a/doc/components/btest/README.rst b/doc/components/btest/README.rst new file mode 100644 index 0000000000..03a859e735 --- /dev/null +++ b/doc/components/btest/README.rst @@ -0,0 +1,843 @@ +.. -*- mode: rst-mode -*- +.. +.. Version number is filled in automatically. +.. |version| replace:: 0.4-14 + +============================================ +BTest - A Simple Driver for Basic Unit Tests +============================================ + +.. rst-class:: opening + + The ``btest`` is a simple framework for writing unit tests. Freely + borrowing some ideas from other packages, it's main objective is to + provide an easy-to-use, straightforward driver for a suite of + shell-based tests. Each test consists of a set of command lines that + will be executed, and success is determined based on their exit + codes. ``btest`` comes with some additional tools that can be used + within such tests to compare output against a previously established + baseline. + +.. contents:: + +Download +======== + +You can find the latest BTest release for download at +http://www.bro.org/download. + +BTest's git repository is located at `git://git.bro.org/btest.git +`__. You can browse the repository +`here `__. + +This document describes BTest |version|. See the ``CHANGES`` +file for version history. + + +Installation +============ + +Installation is simple and standard:: + + tar xzvf btest-*.tar.gz + cd btest-* + python setup.py install + +This will install a few scripts: ``btest`` is the main driver program, +and there are a number of further helper scripts that we discuss below +(including ``btest-diff``, which is a tool for comparing output to a +previously established baseline). + +Writing a Simple Test +===================== + +In the most simple case, ``btest`` simply executes a set of command +lines, each of which must be prefixed with ``@TEST-EXEC:`` +:: + + > cat examples/t1 + @TEST-EXEC: echo "Foo" | grep -q Foo + @TEST-EXEC: test -d . + > btest examples/t1 + examples.t1 ... ok + +The test passes as both command lines return success. If one of them +didn't, that would be reported:: + + > cat examples/t2 + @TEST-EXEC: echo "Foo" | grep -q Foo + @TEST-EXEC: test -d DOESNOTEXIST + > btest examples/t2 + examples.t2 ... failed + +Usually you will just run all tests found in a directory:: + + > btest examples + examples.t1 ... ok + examples.t2 ... failed + 1 test failed + +Why do we need the ``@TEST-EXEC:`` prefixes? Because the file +containing the test can simultaneously act as *its input*. Let's +say we want to verify a shell script:: + + > cat examples/t3.sh + # @TEST-EXEC: sh %INPUT + ls /etc | grep -q passwd + > btest examples/t3.sh + examples.t3 ... ok + +Here, ``btest`` is executing (something similar to) ``sh +examples/t3.sh``, and then checks the return value as usual. The +example also shows that the ``@TEST-EXEC`` prefix can appear +anywhere, in particular inside the comment section of another +language. + +Now, let's say we want to check the output of a program, making sure +that it matches what we expect. For that, we first add a command +line to the test that produces the output we want to check, and then +run ``btest-diff`` to make sure it matches a previously recorded +baseline. ``btest-diff`` is itself just a script that returns +success if the output is as expected, and failure otherwise. In the +following example, we use an awk script as a fancy way to print all +file names starting with a dot in the user's home directory. We +write that list into a file called ``dots`` and then check whether +its content matches what we know from last time:: + + > cat examples/t4.awk + # @TEST-EXEC: ls -a $HOME | awk -f %INPUT >dots + # @TEST-EXEC: btest-diff dots + /^\.+/ { print $1 } + +Note that each test gets its own little sandbox directory when run, +so by creating a file like ``dots``, you aren't cluttering up +anything. + +The first time we run this test, we need to record a baseline:: + + > btest -U examples/t4.awk + +Now, ``btest-diff`` has remembered what the ``dots`` file should +look like:: + + > btest examples/t4.awk + examples.t4 ... ok + > touch ~/.NEWDOTFILE + > btest examples/t4.awk + examples.t4 ... failed + 1 test failed + +If we want to see what exactly the unexpected change is that was +introduced to ``dots``, there's a *diff* mode for that:: + + > btest -d examples/t4.awk + examples.t4 ... failed + % 'btest-diff dots' failed unexpectedly (exit code 1) + % cat .diag + == File =============================== + [... current dots file ...] + == Diff =============================== + --- /Users/robin/work/binpacpp/btest/Baseline/examples.t4/dots + 2010-10-28 20:11:11.000000000 -0700 + +++ dots 2010-10-28 20:12:30.000000000 -0700 + @@ -4,6 +4,7 @@ + .CFUserTextEncoding + .DS_Store + .MacOSX + +.NEWDOTFILE + .Rhistory + .Trash + .Xauthority + ======================================= + + % cat .stderr + [... if any of the commands had printed something to stderr, that would follow here ...] + +Once we delete the new file, we are fine again:: + + > rm ~/.NEWDOTFILE + > btest -d examples/t4.awk + examples.t4 ... ok + +That's already the main functionality that the ``btest`` package +provides. In the following, we describe a number of further options +extending/modifying this basic approach. + +Reference +========= + +Command Line Usage +------------------ + +``btest`` must be started with a list of tests and/or directories +given on the command line. In the latter case, the default is to +recursively scan the directories and assume all files found to be +tests to perform. It is however possible to exclude certain files by +specifying a suitable `configuration file`_. + +``btest`` returns exit code 0 if all tests have successfully passed, +and 1 otherwise. + +``btest`` accepts the following options: + + -a ALTERNATIVE, --alternative=ALTERNATIVE + Activates an alternative_ configuration defined in the + configuration file. This option can be given multiple times to + run tests with several alternatives. If ``ALTERNATIVE`` is ``-`` + that refers to running with the standard setup, which can be used + to run tests both with and without alterantives by giving both. + + -b, --brief + Does not output *anything* for tests which pass. If all tests + pass, there will not be any output at all. + + -c CONFIG, --config=CONFIG + Specifies an alternative `configuration file`_ to use. If not + specified, the default is to use a file called ``btest.cfg`` + if found in the current directory. + + -d, --diagnostics + Reports diagnostics for all failed tests. The diagnostics + include the command line that failed, its output to standard + error, and potential additional information recorded by the + command line for diagnostic purposes (see `@TEST-EXEC`_ + below). In the case of ``btest-diff``, the latter is the + ``diff`` between baseline and actual output. + + -D, --diagnostics-all + Reports diagnostics for all tests, including those which pass. + + -f DIAGFILE, --file-diagnostics=DIAGFILE + Writes diagnostics for all failed tests into the given file. + If the file already exists, it will be overwritten. + + -g GROUPS, --group=GROUPS + Runs only tests assigned to the given test groups, see + `@TEST-GROUP`_. Multiple groups can be given as a + comma-separated list. Specifying ``-`` as a group name selects + all tests that do not belong to any group. + + -j [THREADS], --jobs[=THREADS] + Runs up to the given number of tests in parallel. If no number + is given, BTest substitutes the number of available CPU cores + as reported by the OS. + + By default, BTest assumes that all tests can be executed + concurrently without further constraints. One can however + ensure serialization of subsets by assigning them to the same + serialization set, see `@TEST-SERIALIZE`_. + + -q, --quiet + Suppress information output other than about failed tests. + If all tests pass, there will not be any output at all. + + -r, --rerun + Runs only tests that failed last time. After each execution + (except when updating baselines), BTest generates a state file + that records the tests that have failed. Using this option on + the next run then reads that file back in and limits execution + to those tests found in there. + + -t, --tmp-keep + Does not delete any temporary files created for running the + tests (including their outputs). By default, the temporary + files for a test will be located in ``.tmp//``, where + ```` is the relative path of the test file with all slashes + replaced with dots and the file extension removed (e.g., the files + for ``example/t3.sh`` will be in ``.tmp/example.t3``). + + -U, --update-baseline + Records a new baseline for all ``btest-diff`` commands found + in any of the specified tests. To do this, all tests are run + as normal except that when ``btest-diff`` is executed, it + does not compute a diff but instead considers the given file + to be authoritative and records it as the version to compare + with in future runs. + + -u, --update-interactive + Each time a ``btest-diff`` command fails in any tests that are + run, btest will stop and ask whether or not the user wants to + record a new baseline. + + -v, --verbose + Shows all test command lines as they are executed. + + -w, --wait + Interactively waits for ```` after showing diagnostics + for a test. + + -x FILE, --xml=FILE + Records test results in JUnit XML format to the given file. + If the file exists already, it is overwritten. + +.. _configuration file: + +Configuration +------------- + +Specifics of ``btest``'s execution can be tuned with a configuration +file, which by default is ``btest.cfg`` if that's found in the +current directory. It can alternatively be specified with the +``--config`` command line option. The configuration file is +"INI-style", and an example comes with the distribution, see +``btest.cfg.example``. A configuration file has one main section, +``btest``, that defines most options; as well as an optional section +for defining `environment variables`_ and further optional sections +for defining alternatives_. + +Note that all paths specified in the configuration file are relative +to ``btest``'s *base directory*. The base directory is either the +one where the configuration file is located if such is given/found, +or the current working directory if not. When setting values for +configuration options, the absolute path to the base directory is +available by using the macro ``%(testbase)s`` (the weird syntax is +due to Python's ``ConfigParser`` module). + +Furthermore, all values can use standard "backtick-syntax" to +include the output of external commands (e.g., xyz=`\echo test\`). +Note that the backtick expansion is performed after any ``%(..)`` +have already been replaced (including within the backticks). + +Options +~~~~~~~ + +The following options can be set in the ``btest`` section of the +configuration file: + +``TestDirs`` + A space-separated list of directories to search for tests. If + defined, one doesn't need to specify any tests on the command + line. + +``TmpDir`` + A directory where to create temporary files when running tests. + By default, this is set to ``%(testbase)s/.tmp``. + +``BaselineDir`` + A directory where to store the baseline files for ``btest-diff``. + By default, this is set to ``%(testbase)s/Baseline``. + +``IgnoreDirs`` + A space-separated list of relative directory names to ignore + when scanning test directories recursively. Default is empty. + +``IgnoreFiles`` + A space-separated list of filename globs matching files to + ignore when scanning given test directories recursively. + Default is empty. + +``StateFile`` + The name of the state file to record the names of failing tests. Default is + ``.btest.failed.dat``. + +``Finalizer`` + An executable that will be executed each time any test has + successfully run. It runs in the same directory as the test itself + and receives the name of the test as its parameter. The return + value indicates whether the test should indeed be considered + successful. By default, there's no finalizer set. + +.. _environment variables: + +Environment Variables +~~~~~~~~~~~~~~~~~~~~~ + +A special section ``environment`` defines environment variables that +will be propagated to all tests:: + + [environment] + CFLAGS=-O3 + PATH=%(testbase)s/bin:%(default_path)s + +Note how ``PATH`` can be adjusted to include local scripts: the +example above prefixes it with a local ``bin/`` directory inside the +base directory, using the predefined ``default_path`` macro to refer +to the ``PATH`` as it is set by default. + +Furthermore, by setting ``PATH`` to include the ``btest`` +distribution directory, one could skip the installation of the +``btest`` package. + +.. _alternative: + +Alternatives +~~~~~~~~~~~~ + +BTest can run a set of tests with different settings than it would +normally use by specifying an *alternative* configuration. Currently, +three things can be adjusted: + + - Further environment variables can be set that will then be + available to all the commands that a test executes. + + - *Filters* can modify an input file before a test uses it. + + - *Substitutions* can modify command lines executed as part of a + test. + +We discuss the three separately in the following. All of them are +defined by adding sections ``[-]`` where ```` +corresponds to the type of adjustment being made and ```` is the +name of the alternative. Once at least one section is defined for a +name, that alternative can be enabled by BTest's ``--alternative`` +flag. + +Environment Variables +^^^^^^^^^^^^^^^^^^^^^ + +An alternative can add further environment variables by defining an +``[environment-]`` section: + + [environment-myalternative] + CFLAGS=-O3 + +Running ``btest`` with ``--alternative=myalternative`` will now make +the ``CFLAGS`` environment variable available to all commands +executed. + +.. _filters: + +Filters +^^^^^^^ + +Filters are a transparent way to adapt the input to a specific test +command before it is executed. A filter is defined by adding a section +``[filter-]`` to the configuration file. This section must have +exactly one entry, and the name of that entry is interpreted as the +name of a command whose input is to be filtered. The value of that +entry is the name of a filter script that will be run with two +arguments representing input and output files, respectively. Example:: + + [filter-myalternative] + cat=%(testbase)s/bin/filter-cat + +Once the filter is activated by running ``btest`` with +``--alternative=myalternative``, every time a ``@TEST-EXEC: cat +%INPUT`` is found, ``btest`` will first execute (something similar to) +``%(testbase)s/bin/filter-cat %INPUT out.tmp``, and then subsequently +``cat out.tmp`` (i.e., the original command but with the filtered +output). In the simplest case, the filter could be a no-op in the +form ``cp $1 $2``. + +.. note:: + There are a few limitations to the filter concept currently: + + * Filters are *always* fed with ``%INPUT`` as their first + argument. We should add a way to filter other files as well. + + * Filtered commands are only recognized if they are directly + starting the command line. For example, ``@TEST-EXEC: ls | cat + >outout`` would not trigger the example filter above. + + * Filters are only executed for ``@TEST-EXEC``, not for + ``@TEST-EXEC-FAIL``. + +.. _substitution: + +Substitutions +^^^^^^^^^^^^^^ + +Substitutions are similar to filters, yet they do not adapt the input +but the command line being executed. A substitution is defined by +adding a section ``[substitution-]`` to the configuration file. +For each entry in this section, the entry's name specifies the +command that is to be replaced with something else given as its value. +Example:: + + [substitution-myalternative] + gcc=gcc -O2 + +Once the substitution is activated by running ``btest`` with +``--alternative=myalternative``, every time a ``@TEST-EXEC`` executes +``gcc``, that is replaced with ``gcc -O2``. The replacement is simple +string substitution so it works not only with commands but anything +found on the command line; it however only replaces full words, not +subparts of words. + +Writing Tests +------------- + +``btest`` scans a test file for lines containing keywords that +trigger certain functionality. Currently, the following keywords are +supported: + +.. _@TEST-EXEC: + +``@TEST-EXEC: `` + Executes the given command line and aborts the test if it + returns an error code other than zero. The ```` is + passed to the shell and thus can be a pipeline, use redirection, + and any environment variables specified in ```` will be + expanded, etc. + + When running a test, the current working directory for all + command lines will be set to a temporary sandbox (and will be + deleted later). + + There are two macros that can be used in ````: + ``%INPUT`` will be replaced with the full pathname of the file defining + the test; and ``%DIR`` will be replaced with the directory where + the test file is located. The latter can be used to reference + further files also located there. + + In addition to environment variables defined in the + configuration file, there are further ones that are passed into + the commands: + + ``TEST_DIAGNOSTICS`` + A file where further diagnostic information can be saved + in case a command fails. ``--diagnostics`` will show + this file. (This is also where ``btest-diff`` stores its + diff.) + + ``TEST_MODE`` + This is normally set to ``TEST``, but will be ``UPDATE`` + if ``btest`` is run with ``--update-baseline``, or + ``UPDATE_INTERACTIVE`` if run with ``--update-interactive``. + + ``TEST_BASELINE`` + The name of a directory where the command can save permanent + information across ``btest`` runs. (This is where + ``btest-diff`` stores its baseline in ``UPDATE`` mode.) + + ``TEST_NAME`` + The name of the currently executing test. + + ``TEST_VERBOSE`` + The path of a file where the test can record further + information about its execution that will be included with + btest's ``--verbose`` output. This is for further tracking + the execution of commands and should generally generate + output that follows a line-based structure. + + .. note:: + + If a command returns the special exit code 100, the test is + considered failed, however subsequent test commands are still + run. ``btest-diff`` uses this special exit code to indicate that + no baseline has yet been established. + + If a command returns the special exit code 200, the test is + considered failed and all further test executions are aborted. + + +``@TEST-EXEC-FAIL: `` + Like ``@TEST-EXEC``, except that this expects the command to + *fail*, i.e., the test is aborted when the return code is zero. + +``@TEST-REQUIRES: `` + Defines a condition that must be met for the test to be executed. + The given command line will be run before any of the actual test + commands, and it must return success for the test to continue. If + it does not return success, the rest of the test will be skipped + but doing so will not be considered a failure of the test. This allows to + write conditional tests that may not always make sense to run, depending + on whether external constraints are satisfied or not (say, whether + a particular library is available). Multiple requirements may be + specified and then all must be met for the test to continue. + +``@TEST-ALTERNATIVE: `` Runs this test only for the given + alternative (see alternative_). If ```` is + ``default``, the test executes when BTest runs with no alternative + given (which however is the default anyways). + +``@TEST-NOT-ALTERNATIVE: `` Ignores this test for the + given alternative (see alternative_). If ```` is + ``default``, the test is ignored if BTest runs with no alternative + given. + +``@TEST-COPY-FILE: `` + Copy the given file into the test's directory before the test is + run. If ```` is a relative path, it's interpreted relative + to the BTest's base directory. Environment variables in ```` + will be replaced if enclosed in ``${..}``. This command can be + given multiple times. + +``@TEST-START-NEXT`` + This is a short-cut for defining multiple test inputs in the + same file, all executing with the same command lines. When + ``@TEST-START-NEXT`` is encountered, the test file is initially + considered to end at that point, and all ``@TEST-EXEC-*`` are + run with an ``%INPUT`` truncated accordingly. Afterwards, a + *new* ``%INPUT`` is created with everything *following* the + ``@TEST-START-NEXT`` marker, and the *same* commands are run + again (further ``@TEST-EXEC-*`` will be ignored). The effect is + that a single file can actually define two tests, and the + ``btest`` output will enumerate them:: + + > cat examples/t5.sh + # @TEST-EXEC: cat %INPUT | wc -c >output + # @TEST-EXEC: btest-diff output + + This is the first test input in this file. + + # @TEST-START-NEXT + + ... and the second. + + > ./btest -D examples/t5.sh + examples.t5 ... ok + % cat .diag + == File =============================== + 119 + [...] + + examples.t5-2 ... ok + % cat .diag + == File =============================== + 22 + [...] + + Multiple ``@TEST-START-NEXT`` can be used to create more than + two tests per file. + +``@TEST-START-FILE `` + This is used to include an additional input file for a test + right inside the test file. All lines following the keyword will + be written into the given file (and removed from the test's + `%INPUT`) until a terminating ``@TEST-END-FILE`` is found. + Example:: + + > cat examples/t6.sh + # @TEST-EXEC: awk -f %INPUT output + # @TEST-EXEC: btest-diff output + + { lines += 1; } + END { print lines; } + + @TEST-START-FILE foo.dat + 1 + 2 + 3 + @TEST-END-FILE + + > btest -D examples/t6.sh + examples.t6 ... ok + % cat .diag + == File =============================== + 3 + + Multiple such files can be defined within a single test. + + Note that this is only one way to use further input files. + Another is to store a file in the same directory as the test + itself, making sure it's ignored via ``IgnoreFiles``, and then + refer to it via ``%DIR/``. + +.. _@TEST-GROUP: + +``@TEST-GROUP: `` + Assigns the test to a group of name ````. By using option + ``-g`` one can limit execution to all tests that belong to a given + group (or a set of groups). + +.. _@TEST-SERIALIZE: + +``@TEST-SERIALIZE: `` + When using option ``-j`` to parallelize execution, all tests that + specify the same serialization set are guaranteed to run + sequentially. ```` is an arbitrary user-chosen string. + + +Canonifying Diffs +================= + +``btest-diff`` has the capability to filter its input through an +additional script before it compares the current version with the +baseline. This can be useful if certain elements in an output are +*expected* to change (e.g., timestamps). The filter can then +remove/replace these with something consistent. To enable such +canonification, set the environment variable +``TEST_DIFF_CANONIFIER`` to a script reading the original version +from stdin and writing the canonified version to stdout. Note that +both baseline and current output are passed through the filter +before their differences are computed. + +Running Processes in the Background +=================================== + +Sometimes processes need to be spawned in the background for a test, +in particular if multiple processes need to cooperate in some fashion. +``btest`` comes with two helper scripts to make life easier in such a +situation: + +``btest-bg-run `` + This is a script that runs ```` in the background, i.e., + it's like using ``cmdline &`` in a shell script. Test execution + continues immediately with the next command. Note that the spawned + command is *not* run in the current directory, but instead in a + newly created sub-directory called ````. This allows + spawning multiple instances of the same process without needing to + worry about conflicting outputs. If you want to access a command's + output later, like with ``btest-diff``, use ``/foo.log`` to + access it. + +``btest-bg-wait [-k] `` + This script waits for all processes previously spawned via + ``btest-bg-run`` to finish. If any of them exits with a non-zero + return code, ``btest-bg-wait`` does so as well, indicating a + failed test. ```` is mandatory and gives the maximum + number of seconds to wait for any of the processes to terminate. + If any process hasn't done so when the timeout expires, it will be + killed and the test is considered to be failed as long as ``-k`` + is not given. If ``-k`` is given, pending processes are still + killed but the test continues normally, i.e., non-termination is + not considered a failure in this case. This script also collects + the processes' stdout and stderr outputs for diagnostics output. + +Integration with Sphinx +======================= + +``btest`` comes with a new directive for the documentation framework +`Sphinx `_. The directive allows to write a +test directly inside a Sphinx document, and then to include output +from the test's command into the generated documentation. The same +tests can also run externally and will catch if any changes to the +included content occur. The following walks through setting this up. + +Configuration +------------- + +First, you need to tell Sphinx a base directory for the ``btest`` +configuration as well as a directory in there where to store tests +it extracts from the Sphinx documentation. Typically, you'd just +create a new subdirectory ``tests`` in the Sphinx project for the +``btest`` setup and then store the tests in there in, e.g., +``doc/``:: + + cd + mkdir tests + mkdir tests/doc + +Then add the following to your Sphinx ``conf.py``:: + + extensions += ["btest-sphinx"] + btest_base="tests" # Relative to Sphinx-root. + btest_tests="doc" # Relative to btest_base. + +Next, a finalizer to ``btest.cfg``:: + + [btest] + ... + Finalizer=btest-diff-rst + +Finally, create a ``btest.cfg`` in ``tests/`` as usual and add +``doc/`` to the ``TestDirs`` option. + +Including a Test into a Sphinx Document +--------------------------------------- + +The ``btest`` extension provides a new directive to include a test +inside a Sphinx document:: + + + .. btest:: + + + +Here, ```` is a custom name for the test; it will be +stored in ``btest_tests`` under that name. ```` is just +a standard test as you would normally put into one of the +``TestDirs``. Example:: + + + .. btest:: just-a-test + + @TEST-EXEC: expr 2 + 2 + +When you now run Sphinx, it will (1) store the test content into +``tests/doc/just-a-test`` (assuming the above path layout), and (2) +execute the test by running ``btest`` on it. You can then run +``btest`` manually in ``tests/`` as well and it will execute the test +just as it would in a standard setup. If a test fails when Sphinx runs +it, there will be a corresponding error and include the diagnostic output +into the document. + +By default, nothing else will be included into the generated +documentation, i.e., the above test will just turn into an empty text +block. However, ``btest`` comes with a set of scripts that you can use +to specify content to be included. As a simple example, +``btest-rst-cmd `` will execute a command and (if it +succeeds) include both the command line and the standard output into +the documentation. Example:: + + .. btest:: another-test + + @TEST-EXEC: btest-rst-cmd echo Hello, world! + +When running Sphinx, this will render as: + +.. code:: + + # echo Hello, world! + Hello world! + + +When running ``btest`` manually in ``tests/``, the ``Finalizer`` we +added to ``btest.cfg`` (see above) compares the generated reST code +with a previously established baseline, just like ``btest-diff`` does +with files. To establish the initial baseline, run ``btest -u``, like +you would with ``btest-diff``. + +Scripts +------- + +The following Sphinx support scripts come with ``btest``: + +``btest-rst-cmd [options] `` + + By default, this executes ```` and includes both the + command line itself and its standard output into the generated + documentation. See above for an example. + + This script provides the following options: + + -c ALTERNATIVE_CMDLINE + Show ``ALTERNATIVE_CMDLINE`` in the generated + documentation instead of the one actually executed. (It + still runs the ```` given outside the option.) + + -d + Do not actually execute ````; just format it for + the generated documentation and include no further output. + + -f FILTER_CMD + Pipe the command line's output through ``FILTER_CMD`` + before including. If ``-r`` is given, it filters the + file's content instead of stdout. + + -o + Do not include the executed command into the generated + documentation, just its output. + + -r FILE + Insert ``FILE`` into output instead of stdout. + + +``btest-rst-include `` + + Includes ```` inside a code block. + +``btest-rst-pipe `` + + Executes ````, includes its standard output inside a code + block. Note that this script does not include the command line + itself into the code block, just the output. + +.. note:: + + All these scripts can be run directly from the command line to show + the reST code they generate. + +.. note:: + + ``btest-rst-cmd`` can do everything the other scripts provide if + you give it the right options. In fact, the other scripts are + provided just for convenience and leverage ``btest-rst-cmd`` + internally. + +License +======= + +btest is open-source under a BSD licence. + diff --git a/doc/components/capstats/README.rst b/doc/components/capstats/README.rst deleted file mode 120000 index cb2380145d..0000000000 --- a/doc/components/capstats/README.rst +++ /dev/null @@ -1 +0,0 @@ -../../../aux/broctl/aux/capstats/README \ No newline at end of file diff --git a/doc/components/capstats/README.rst b/doc/components/capstats/README.rst new file mode 100644 index 0000000000..1ddc07c51c --- /dev/null +++ b/doc/components/capstats/README.rst @@ -0,0 +1,107 @@ +.. -*- mode: rst-mode -*- +.. +.. Version number is filled in automatically. +.. |version| replace:: 0.18 + +=============================================== +capstats - A tool to get some NIC statistics. +=============================================== + +.. rst-class:: opening + + capstats is a small tool to collect statistics on the + current load of a network interface, using either `libpcap + `_ or the native interface for `Endace's + `_. It reports statistics per time interval + and/or for the tool's total run-time. + +Download +-------- + +You can find the latest capstats release for download at +http://www.bro.org/download. + +Capstats's git repository is located at `git://git.bro.org/capstats.git +`__. You can browse the repository +`here `__. + +This document describes capstats |version|. See the ``CHANGES`` +file for version history. + + +Output +------ + +Here's an example output with output in one-second intervals until +``CTRL-C`` is hit: + +.. console:: + + >capstats -i nve0 -I 1 + 1186620936.890567 pkts=12747 kpps=12.6 kbytes=10807 mbps=87.5 nic_pkts=12822 nic_drops=0 u=960 t=11705 i=58 o=24 nonip=0 + 1186620937.901490 pkts=13558 kpps=13.4 kbytes=11329 mbps=91.8 nic_pkts=13613 nic_drops=0 u=1795 t=24339 i=119 o=52 nonip=0 + 1186620938.912399 pkts=14771 kpps=14.6 kbytes=13659 mbps=110.7 nic_pkts=14781 nic_drops=0 u=2626 t=38154 i=185 o=111 nonip=0 + 1186620939.012446 pkts=1332 kpps=13.3 kbytes=1129 mbps=92.6 nic_pkts=1367 nic_drops=0 u=2715 t=39387 i=194 o=112 nonip=0 + === Total + 1186620939.012483 pkts=42408 kpps=13.5 kbytes=36925 mbps=96.5 nic_pkts=1 nic_drops=0 u=2715 t=39387 i=194 o=112 nonip=0 + +Each line starts with a timestamp and the other fields are: + + :pkts: + Absolute number of packets seen by ``capstats`` during interval. + + :kpps: + Number of packets per second. + + :kbytes: + Absolute number of KBytes during interval. + + :mbps: + Mbits/sec. + + :nic_pkts: + Number of packets as reported by ``libpcap``'s ``pcap_stats()`` (may not match _pkts_) + + :nic_drops: + Number of packet drops as reported by ``libpcap``'s ``pcap_stats()``. + + :u: + Number of UDP packets. + + :t: + Number of TCP packets. + + :i: + Number of ICMP packets. + + :nonip: + Number of non-IP packets. + +Options +------- + +A list of all options:: + + capstats [Options] -i interface + + -i| --interface Listen on interface + -d| --dag Use native DAG API + -f| --filter BPF filter + -I| --interval Stats logging interval + -l| --syslog Use syslog rather than print to stderr + -n| --number Stop after outputting intervals + -N| --select Use select() for live pcap (for testing only) + -p| --payload Verifies that packets' payloads consist + entirely of bytes of the given value. + -q| --quiet Suppress output, exit code indicates >= count + packets received. + -S| --size Verify packets to have given + -s| --snaplen Use pcap snaplen + -v| --version Print version and exit + -w| --write Write packets to file + +Installation +------------ + +``capstats`` has been tested on Linux, FreeBSD, and MacOS. Please see +the ``INSTALL`` file for installation instructions. diff --git a/doc/components/index.rst b/doc/components/index.rst new file mode 100644 index 0000000000..defae92647 --- /dev/null +++ b/doc/components/index.rst @@ -0,0 +1,28 @@ + +============= +Subcomponents +============= + +The following are snapshots of documentation for components that come +with this version of Bro (|version|). Since they can also be used +independently, see the `download page +`_ for documentation of any +current, independent component releases. + +.. toctree:: + :maxdepth: 1 + + BinPAC - A protocol parser generator + Broccoli - The Bro Client Communication Library (README) + Broccoli - User Manual + Broccoli Python Bindings + Broccoli Ruby Bindings + BroControl - Interactive Bro management shell + Bro-Aux - Small auxiliary tools for Bro + BTest - A unit testing framework + Capstats - Command-line packet statistic tool + PySubnetTree - Python module for CIDR lookups + trace-summary - Script for generating break-downs of network traffic + +The `Broccoli API Reference `_ may also be of +interest. diff --git a/doc/components/pysubnettree/README.rst b/doc/components/pysubnettree/README.rst deleted file mode 120000 index 42ce17d303..0000000000 --- a/doc/components/pysubnettree/README.rst +++ /dev/null @@ -1 +0,0 @@ -../../../aux/broctl/aux/pysubnettree/README \ No newline at end of file diff --git a/doc/components/pysubnettree/README.rst b/doc/components/pysubnettree/README.rst new file mode 100644 index 0000000000..be97eef9e3 --- /dev/null +++ b/doc/components/pysubnettree/README.rst @@ -0,0 +1,98 @@ +.. -*- mode: rst-mode -*- +.. +.. Version number is filled in automatically. +.. |version| replace:: 0.19-9 + +=============================================== +PySubnetTree - A Python Module for CIDR Lookups +=============================================== + +.. rst-class:: opening + + The PySubnetTree package provides a Python data structure + ``SubnetTree`` which maps subnets given in `CIDR + `_ notation (incl. + corresponding IPv6 versions) to Python objects. Lookups are + performed by longest-prefix matching. + + +Download +-------- + +You can find the latest PySubnetTree release for download at +http://www.bro.org/download. + +PySubnetTree's git repository is located at `git://git.bro.org/pysubnettree.git +`__. You can browse the repository +`here `__. + +This document describes PySubnetTree |version|. See the ``CHANGES`` +file for version history. + + +Example +------- + +A simple example which associates CIDR prefixes with strings:: + + >>> import SubnetTree + >>> t = SubnetTree.SubnetTree() + >>> t["10.1.0.0/16"] = "Network 1" + >>> t["10.1.42.0/24"] = "Network 1, Subnet 42" + >>> t["10.2.0.0/16"] = "Network 2" + >>> print t["10.1.42.1"] + Network 1, Subnet 42 + >>> print t["10.1.43.1"] + Network 1 + >>> print "10.1.42.1" in t + True + >>> print "10.1.43.1" in t + True + >>> print "10.20.1.1" in t + False + >>> print t["10.20.1.1"] + Traceback (most recent call last): + File "", line 1, in + File "SubnetTree.py", line 67, in __getitem__ + def __getitem__(*args): return _SubnetTree.SubnetTree___getitem__(*args) + KeyError: '10.20.1.1' + +By default, CIDR prefixes and IP addresses are given as strings. +Alternatively, a ``SubnetTree`` object can be switched into *binary +mode*, in which single addresses are passed in the form of packed +binary strings as, e.g., returned by `socket.inet_aton +`_:: + + + >>> t.get_binary_lookup_mode() + False + >>> t.set_binary_lookup_mode(True) + >>> t.binary_lookup_mode() + True + >>> import socket + >>> print t[socket.inet_aton("10.1.42.1")] + Network 1, Subnet 42 + +A SubnetTree also provides methods ``insert(prefix,object=None)`` for insertion +of prefixes (``object`` can be skipped to use the tree like a set), and +``remove(prefix)`` for removing entries (``remove`` performs an _exact_ match +rather than longest-prefix). + +Internally, the CIDR prefixes of a ``SubnetTree`` are managed by a +Patricia tree data structure and lookups are therefore efficient +even with a large number of prefixes. + +PySubnetTree comes with a BSD license. + + +Prerequisites +------------- + +This package requires Python 2.4 or newer. + +Installation +------------ + +Installation is pretty simple:: + + > python setup.py install diff --git a/doc/components/trace-summary/README.rst b/doc/components/trace-summary/README.rst deleted file mode 120000 index 78778364bd..0000000000 --- a/doc/components/trace-summary/README.rst +++ /dev/null @@ -1 +0,0 @@ -../../../aux/broctl/aux/trace-summary/README \ No newline at end of file diff --git a/doc/components/trace-summary/README.rst b/doc/components/trace-summary/README.rst new file mode 100644 index 0000000000..a47381fc37 --- /dev/null +++ b/doc/components/trace-summary/README.rst @@ -0,0 +1,154 @@ +.. -*- mode: rst-mode -*- +.. +.. Version number is filled in automatically. +.. |version| replace:: 0.8 + +==================================================== +trace-summary - Generating network traffic summaries +==================================================== + +.. rst-class:: opening + + ``trace-summary`` is a Python script that generates break-downs of + network traffic, including lists of the top hosts, protocols, + ports, etc. Optionally, it can generate output separately for + incoming vs. outgoing traffic, per subnet, and per time-interval. + +Download +-------- + +You can find the latest trace-summary release for download at +http://www.bro.org/download. + +trace-summary's git repository is located at `git://git.bro.org/trace-summary.git +`__. You can browse the repository +`here `__. + +This document describes trace-summary |version|. See the ``CHANGES`` +file for version history. + + +Overview +-------- + +The ``trace-summary`` script reads both packet traces in `libpcap +`_ format and connection logs produced by the +`Bro `_ network intrusion detection system +(for the latter, it supports both 1.x and 2.x output formats). + +Here are two example outputs in the most basic form (note that IP +addresses are 'anonymized'). The first is from a packet trace and the +second from a Bro connection log:: + + + >== Total === 2005-01-06-14-23-33 - 2005-01-06-15-23-43 + - Bytes 918.3m - Payload 846.3m - Pkts 1.8m - Frags 0.9% - MBit/s 1.9 - + Ports | Sources | Destinations | Protocols | + 80 33.8% | 131.243.89.214 8.5% | 131.243.89.214 7.7% | 6 76.0% | + 22 16.7% | 128.3.2.102 6.2% | 128.3.2.102 5.4% | 17 23.3% | + 11001 12.4% | 204.116.120.26 4.8% | 131.243.89.4 4.8% | 1 0.5% | + 2049 10.7% | 128.3.161.32 3.6% | 131.243.88.227 3.6% | | + 1023 10.6% | 131.243.89.4 3.5% | 204.116.120.26 3.4% | | + 993 8.2% | 128.3.164.194 2.7% | 131.243.89.64 3.1% | | + 1049 8.1% | 128.3.164.15 2.4% | 128.3.164.229 2.9% | | + 524 6.6% | 128.55.82.146 2.4% | 131.243.89.155 2.5% | | + 33305 4.5% | 131.243.88.227 2.3% | 128.3.161.32 2.3% | | + 1085 3.7% | 131.243.89.155 2.3% | 128.55.82.146 2.1% | | + + + >== Total === 2005-01-06-14-23-33 - 2005-01-06-15-23-42 + - Connections 43.4k - Payload 398.4m - + Ports | Sources | Destinations | Services | Protocols | States | + 80 21.7% | 207.240.215.71 3.0% | 239.255.255.253 8.0% | other 51.0% | 17 55.8% | S0 46.2% | + 427 13.0% | 131.243.91.71 2.2% | 131.243.91.255 4.0% | http 21.7% | 6 36.4% | SF 30.1% | + 443 3.8% | 128.3.161.76 1.7% | 131.243.89.138 2.1% | i-echo 7.3% | 1 7.7% | OTH 7.8% | + 138 3.7% | 131.243.90.138 1.6% | 255.255.255.255 1.7% | https 3.8% | | RSTO 5.8% | + 515 2.4% | 131.243.88.159 1.6% | 128.3.97.204 1.5% | nb-dgm 3.7% | | SHR 4.4% | + 11001 2.3% | 131.243.88.202 1.4% | 131.243.88.107 1.1% | printer 2.4% | | REJ 3.0% | + 53 1.9% | 131.243.89.250 1.4% | 117.72.94.10 1.1% | dns 1.9% | | S1 1.0% | + 161 1.6% | 131.243.89.80 1.3% | 131.243.88.64 1.1% | snmp 1.6% | | RSTR 0.9% | + 137 1.4% | 131.243.90.52 1.3% | 131.243.88.159 1.1% | nb-ns 1.4% | | SH 0.3% | + 2222 1.1% | 128.3.161.252 1.2% | 131.243.91.92 1.1% | ntp 1.0% | | RSTRH 0.2% | + + +Prerequisites +------------- + +* This script requires Python 2.4 or newer. + +* The `pysubnettree + `_ Python + module. + +* Eddie Kohler's `ipsumdump `_ + if using ``trace-summary`` with packet traces (versus Bro connection logs) + +Installation +------------ + +Simply copy the script into some directory which is in your ``PATH``. + +Usage +----- + +The general usage is:: + + trace-summary [options] [input-file] + +Per default, it assumes the ``input-file`` to be a ``libpcap`` trace +file. If it is a Bro connection log, use ``-c``. If ``input-file`` is +not given, the script reads from stdin. It writes its output to +stdout. + +Options +~~~~~~~ + +The most important options are summarized +below. Run ``trace-summary --help`` to see the full list including +some more esoteric ones. + +:-c: + Input is a Bro connection log instead of a ``libpcap`` trace + file. + +:-b: + Counts all percentages in bytes rather than number of + packets/connections. + +:-E : + Gives a file which contains a list of networks to ignore for the + analysis. The file must contain one network per line, where each + network is of the CIDR form ``a.b.c.d/mask`` (including the + corresponding syntax for IPv6 prefixes, e.g., ``1:2:3:4::/64``). + Empty lines and lines starting with a "#" are ignored. + +:-i : + Creates totals for each time interval of the given length + (default is seconds; add "``m``" for minutes and "``h``" for + hours). Use ``-v`` if you also want to see the breakdowns for + each interval. + +:-l : + Generates separate summaries for incoming and outgoing traffic. + ```` is a file which contains a list of networks to be + considered local. Format as for ``-E``. + +:-n : + Show top n entries in each break-down. Default is 10. + +:-r: + Resolves hostnames in the output. + +:-s : + Gives the sample factor if the input has been sampled. + +:-S : + Sample input with the given factor; less accurate but faster and + saves memory. + +:-m: + Does skip memory-expensive statistics. + +:-v: + Generates full break-downs for each time interval. Requires + ``-i``. diff --git a/doc/conf.py.in b/doc/conf.py.in index 2e93e82502..4cf64f10db 100644 --- a/doc/conf.py.in +++ b/doc/conf.py.in @@ -12,11 +12,28 @@ import sys, os +extensions = [] + # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. sys.path.insert(0, os.path.abspath('sphinx-sources/ext')) +# ----- Begin of BTest configuration. ----- +btest = os.path.abspath("@CMAKE_SOURCE_DIR@/aux/btest") +brocut = os.path.abspath("@CMAKE_SOURCE_DIR@/aux/bro-aux/bro-cut") +bro = os.path.abspath("@CMAKE_SOURCE_DIR@/build/src") + +os.environ["PATH"] += (":%s:%s/sphinx:%s:%s" % (btest, btest, bro, brocut)) +sys.path.append(os.path.join(btest, "sphinx")) + +extensions += ["btest-sphinx"] + +btest_base="@CMAKE_SOURCE_DIR@/testing/btest" +btest_tests="doc/sphinx" +# ----- End of BTest configuration. ----- + + # -- General configuration ----------------------------------------------------- # If your documentation needs a minimal Sphinx version, state it here. @@ -24,7 +41,10 @@ sys.path.insert(0, os.path.abspath('sphinx-sources/ext')) # Add any Sphinx extension module names here, as strings. They can be extensions # coming with Sphinx (named 'sphinx.ext.*') or your custom ones. -extensions = ['bro', 'rst_directive', 'sphinx.ext.todo', 'adapt-toc'] +extensions += ['bro', 'rst_directive', 'sphinx.ext.todo', 'adapt-toc'] + +os.environ["BRO_SRC_ROOT"] = "@CMAKE_SOURCE_DIR@" +os.environ["DOC_ROOT"] = "@CMAKE_SOURCE_DIR@/doc" # Add any paths that contain templates here, relative to this directory. templates_path = ['sphinx-sources/_templates', 'sphinx-sources/_static'] @@ -40,7 +60,7 @@ master_doc = 'index' # General information about the project. project = u'Bro' -copyright = u'2012, The Bro Project' +copyright = u'2013, The Bro Project' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the @@ -63,7 +83,7 @@ today_fmt = '%B %d, %Y' # List of patterns, relative to source directory, that match files and # directories to ignore when looking for source files. -exclude_patterns = [] +exclude_patterns = [".#*"] # The reST default role (used for this markup: `text`) to use for all documents. #default_role = None diff --git a/doc/faq.rst b/doc/faq.rst deleted file mode 100644 index f265505def..0000000000 --- a/doc/faq.rst +++ /dev/null @@ -1,180 +0,0 @@ - -========================== -Frequently Asked Questions -========================== - -.. raw:: html - -
- -.. contents:: - -Installation and Configuration -============================== - -How can I tune my operating system for best capture performance? ----------------------------------------------------------------- - -Here are some pointers to more information: - -* Fabian Schneider's research on `high performance packet capture - `_ - -* `NSMWiki `_ has page on - *Collecting Data*. - -* An `IMC 2010 paper - `_ by - Lothar Braun et. al evaluates packet capture performance on - commodity hardware - -Are there any gotchas regarding interface configuration for live capture? Or why might I be seeing abnormally large packets much greater than interface MTU? -------------------------------------------------------------------------------------------------------------------------------------------------------------- - -Some NICs offload the reassembly of traffic into "superpackets" so that -fewer packets are then passed up the stack (e.g. "TCP segmentation -offload", or "generic segmentation offload"). The result is that the -capturing application will observe packets much larger than the MTU size -of the interface they were captured from and may also interfere with the -maximum packet capture length, ``snaplen``, so it's a good idea to disable -an interface's offloading features. - -You can use the ``ethtool`` program on Linux to view and disable -offloading features of an interface. See this page for more explicit -directions: - -http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html - -What does an error message like ``internal error: NB-DNS error`` mean? ----------------------------------------------------------------------- - -That often means that DNS is not set up correctly on the system -running Bro. Try verifying from the command line that DNS lookups -work, e.g., ``host www.google.com``. - -I am using OpenBSD and having problems installing Bro? ------------------------------------------------------- - -One potential issue is that the top-level Makefile may not work with -OpenBSD's default make program, in which case you can either install -the ``gmake`` package and use it instead or first change into the -``build/`` directory before doing either ``make`` or ``make install`` -such that the CMake-generated Makefile's are used directly. - -Generally, please note that we do not regularly test OpenBSD builds. -We appreciate any patches that improve Bro's support for this -platform. - -How do BroControl options affect Bro script variables? ------------------------------------------------------- - -Some (but not all) BroControl options override a corresponding Bro script variable. -For example, setting the BroControl option "LogRotationInterval" will override -the value of the Bro script variable "Log::default_rotation_interval". -See the :doc:`BroControl Documentation ` to find out -which BroControl options override Bro script variables, and for more discussion -on site-specific customization. - -Usage -===== - -How can I identify backscatter? -------------------------------- - -Identifying backscatter via connections labeled as ``OTH`` is not a reliable -means to detect backscatter. Backscatter is however visible by interpreting -the contents of the ``history`` field in the ``conn.log`` file. The basic idea -is to watch for connections that never had an initial ``SYN`` but started -instead with a ``SYN-ACK`` or ``RST`` (though this latter generally is just -discarded). Here are some history fields which provide backscatter examples: -``hAFf``, ``r``. Refer to the conn protocol analysis scripts to interpret the -individual character meanings in the history field. - -Is there help for understanding Bro's resource consumption? ------------------------------------------------------------ - -There are two scripts that collect statistics on resource usage: -``misc/stats.bro`` and ``misc/profiling.bro``. The former is quite -lightweight, while the latter should only be used for debugging. - -How can I capture packets as an unprivileged user? --------------------------------------------------- - -Normally, unprivileged users cannot capture packets from a network interface, -which means they would not be able to use Bro to read/analyze live traffic. -However, there are operating system specific ways to enable packet capture -permission for non-root users, which is worth doing in the context of using -Bro to monitor live traffic. - -With Linux Capabilities -^^^^^^^^^^^^^^^^^^^^^^^ - -Fully implemented since Linux kernel 2.6.24, capabilities are a way of -parceling superuser privileges into distinct units. Attach capabilities -required to capture packets to the ``bro`` executable file like this: - -.. console:: - - sudo setcap cap_net_raw,cap_net_admin=eip /path/to/bro - -Now any unprivileged user should have the capability to capture packets -using Bro provided that they have the traditional file permissions to -read/execute the ``bro`` binary. - -With BPF Devices -^^^^^^^^^^^^^^^^ - -Systems using Berkeley Packet Filter (BPF) (e.g. FreeBSD & Mac OS X) -can allow users with read access to a BPF device to capture packets from -it using libpcap. - -* Example of manually changing BPF device permissions to allow users in - the ``admin`` group to capture packets: - -.. console:: - - sudo chgrp admin /dev/bpf* - sudo chmod g+r /dev/bpf* - -* Example of configuring devfs to set permissions of BPF devices, adding - entries to ``/etc/devfs.conf`` to grant ``admin`` group permission to - capture packets: - -.. console:: - - sudo sh -c 'echo "own bpf root:admin" >> /etc/devfs.conf' - sudo sh -c 'echo "perm bpf 0640" >> /etc/devfs.conf' - sudo service devfs restart - -.. note:: As of Mac OS X 10.6, the BPF device is on devfs, but the used version - of devfs isn't capable of setting the device permissions. The permissions - can be changed manually, but they will not survive a reboot. - -Why isn't Bro producing the logs I expect? (A Note About Checksums) -------------------------------------------------------------------- - -Normally, Bro's event engine will discard packets which don't have valid -checksums. This can be a problem if one wants to analyze locally -generated/captured traffic on a system that offloads checksumming to the -network adapter. In that case, all transmitted/captured packets will have -bad checksums because they haven't yet been calculated by the NIC, thus -such packets will not undergo analysis defined in Bro policy scripts as they -normally would. Bad checksums in traces may also be a result of some packet -alteration tools. - -Bro has two options to workaround such situations and ignore bad checksums: - -1) The ``-C`` command line option to ``bro``. -2) An option called ``ignore_checksums`` that can be redefined at the - policy script layer (e.g. in your ``$PREFIX/share/bro/site/local.bro``): - - .. code:: bro - - redef ignore_checksums = T; - -The other alternative is to disable checksum offloading for your -network adapter, but this is not always possible or desirable. - -.. raw:: html - -
diff --git a/doc/file-analysis.rst b/doc/frameworks/file-analysis.rst similarity index 100% rename from doc/file-analysis.rst rename to doc/frameworks/file-analysis.rst diff --git a/doc/geoip.rst b/doc/frameworks/geoip.rst similarity index 99% rename from doc/geoip.rst rename to doc/frameworks/geoip.rst index bd9ae0c08d..3c3883a607 100644 --- a/doc/geoip.rst +++ b/doc/frameworks/geoip.rst @@ -1,4 +1,6 @@ +.. _geolocation: + =========== GeoLocation =========== diff --git a/doc/frameworks/index.rst b/doc/frameworks/index.rst new file mode 100644 index 0000000000..d5b771b15e --- /dev/null +++ b/doc/frameworks/index.rst @@ -0,0 +1,16 @@ + +========== +Frameworks +========== + +.. toctree:: + :maxdepth: 1 + + file-analysis + geoip + input + intel + logging + notice + signatures + diff --git a/doc/input.rst b/doc/frameworks/input.rst similarity index 99% rename from doc/input.rst rename to doc/frameworks/input.rst index 2945918733..aca5091972 100644 --- a/doc/input.rst +++ b/doc/frameworks/input.rst @@ -1,6 +1,7 @@ -============================================== -Loading Data into Bro with the Input Framework -============================================== + +=============== +Input Framework +=============== .. rst-class:: opening diff --git a/doc/intel.rst b/doc/frameworks/intel.rst similarity index 98% rename from doc/intel.rst rename to doc/frameworks/intel.rst index 787524a417..616de3b858 100644 --- a/doc/intel.rst +++ b/doc/frameworks/intel.rst @@ -1,5 +1,7 @@ -Intel Framework -=============== + +====================== +Intelligence Framework +====================== Intro ----- diff --git a/doc/logging-dataseries.rst b/doc/frameworks/logging-dataseries.rst similarity index 100% rename from doc/logging-dataseries.rst rename to doc/frameworks/logging-dataseries.rst diff --git a/doc/logging-elasticsearch.rst b/doc/frameworks/logging-elasticsearch.rst similarity index 100% rename from doc/logging-elasticsearch.rst rename to doc/frameworks/logging-elasticsearch.rst diff --git a/doc/logging.rst b/doc/frameworks/logging.rst similarity index 98% rename from doc/logging.rst rename to doc/frameworks/logging.rst index b982206e85..9c8df62043 100644 --- a/doc/logging.rst +++ b/doc/frameworks/logging.rst @@ -1,6 +1,9 @@ -========================== -Customizing Bro's Logging -========================== + +.. _framework-logging: + +================= +Logging Framework +================= .. rst-class:: opening @@ -89,7 +92,8 @@ Note the fields that are set for the filter: are generated by taking the stream's ID and munging it slightly. :bro:enum:`Conn::LOG` is converted into ``conn``, :bro:enum:`PacketFilter::LOG` is converted into - ``packet_filter``. + ``packet_filter``, and :bro:enum:`Notice::POLICY_LOG` is + converted into ``notice_policy``. ``include`` A set limiting the fields to the ones given. The names diff --git a/doc/notice.rst b/doc/frameworks/notice.rst similarity index 97% rename from doc/notice.rst rename to doc/frameworks/notice.rst index 76d5bcdecb..1630459b46 100644 --- a/doc/notice.rst +++ b/doc/frameworks/notice.rst @@ -98,9 +98,9 @@ type :bro:see:`SSH::Password_Guessing` if the server is 10.0.0.1: .. note:: - Keep in mind that the semantics of the SSH::Password_Guessing notice are - such that it is only raised when Bro heuristically detects a failed - login. + Keep in mind that the semantics of the :bro:see:`SSH::Password_Guessing` + notice are such that it is only raised when Bro heuristically detects + a failed login. Hooks can also have priorities applied to order their execution like events with a default priority of 0. Greater values are executed first. Setting @@ -339,7 +339,7 @@ included below. hook Notice::policy(n: Notice::Info) { if ( n?$conn && n$conn?$http && n$conn$http?$host ) - n$email_body_sections[|email_body_sections|] = fmt("HTTP host header: %s", n$conn$http$host); + n$email_body_sections[|n$email_body_sections|] = fmt("HTTP host header: %s", n$conn$http$host); } @@ -348,7 +348,7 @@ Cluster Considerations As a user/developer of Bro, the main cluster concern with the notice framework is understanding what runs where. When a notice is generated on a worker, the -worker checks to see if the notice shoudl be suppressed based on information +worker checks to see if the notice should be suppressed based on information locally maintained in the worker process. If it's not being suppressed, the worker forwards the notice directly to the manager and does no more local processing. The manager then runs the :bro:see:`Notice::policy` hook and diff --git a/doc/signatures.rst b/doc/frameworks/signatures.rst similarity index 99% rename from doc/signatures.rst rename to doc/frameworks/signatures.rst index 59ca819636..915133e178 100644 --- a/doc/signatures.rst +++ b/doc/frameworks/signatures.rst @@ -1,7 +1,7 @@ -========== -Signatures -========== +=================== +Signature Framework +=================== .. rst-class:: opening diff --git a/doc/index.rst b/doc/index.rst index aa33d8797d..ac667fc089 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -1,92 +1,25 @@ + .. Bro documentation master file ================= Bro Documentation ================= -Guides ------- - .. toctree:: - :maxdepth: 1 + :maxdepth: 2 - INSTALL - upgrade - quickstart - faq - reporting-problems - -Frameworks ----------- - -.. toctree:: - :maxdepth: 1 - - notice - logging - input - file-analysis - cluster - signatures - -How-Tos -------- - -.. toctree:: - :maxdepth: 1 - - geoip - -Script Reference ----------------- - -.. toctree:: - :maxdepth: 1 - - scripts/packages - scripts/index - scripts/builtins - scripts/proto-analyzers - scripts/file-analyzers - -Other Bro Components --------------------- - -The following are snapshots of documentation for components that come -with this version of Bro (|version|). Since they can also be used -independently, see the `download page -`_ for documentation of any -current, independent component releases. - -.. toctree:: - :maxdepth: 1 - - BinPAC - A protocol parser generator - Broccoli - The Bro Client Communication Library (README) - Broccoli - User Manual - Broccoli Python Bindings - Broccoli Ruby Bindings - BroControl - Interactive Bro management shell - Bro-Aux - Small auxiliary tools for Bro - BTest - A unit testing framework - Capstats - Command-line packet statistic tool - PySubnetTree - Python module for CIDR lookups - trace-summary - Script for generating break-downs of network traffic - -The `Broccoli API Reference `_ may also be of -interest. - -Other Indices and References ----------------------------- + intro/index.rst + install/index.rst + quickstart/index.rst + using/index.rst + scripting/index.rst + frameworks/index.rst + cluster/index.rst + scripts/index.rst + components/index.rst +* `Notice Index `_ (TODO: Move to reference + section, but can't figure out how to include it into toctree) * :ref:`General Index ` -* `Notice Index `_ * :ref:`search` -Internal References -------------------- - -.. toctree:: - :maxdepth: 1 - - scripts/internal diff --git a/doc/install/CHANGES-binpac.txt b/doc/install/CHANGES-binpac.txt new file mode 120000 index 0000000000..5cbc08dc22 --- /dev/null +++ b/doc/install/CHANGES-binpac.txt @@ -0,0 +1 @@ +../../aux/binpac/CHANGES \ No newline at end of file diff --git a/doc/install/CHANGES-bro-aux.txt b/doc/install/CHANGES-bro-aux.txt new file mode 120000 index 0000000000..4fcaa82e52 --- /dev/null +++ b/doc/install/CHANGES-bro-aux.txt @@ -0,0 +1 @@ +../../aux/bro-aux/CHANGES \ No newline at end of file diff --git a/doc/install/CHANGES-bro.txt b/doc/install/CHANGES-bro.txt new file mode 120000 index 0000000000..693f810dc8 --- /dev/null +++ b/doc/install/CHANGES-bro.txt @@ -0,0 +1 @@ +../../CHANGES \ No newline at end of file diff --git a/doc/install/CHANGES-broccoli-python.txt b/doc/install/CHANGES-broccoli-python.txt new file mode 120000 index 0000000000..fd01ad3cd4 --- /dev/null +++ b/doc/install/CHANGES-broccoli-python.txt @@ -0,0 +1 @@ +../../aux/broccoli/bindings/broccoli-python/CHANGES \ No newline at end of file diff --git a/doc/install/CHANGES-broccoli-ruby.txt b/doc/install/CHANGES-broccoli-ruby.txt new file mode 120000 index 0000000000..e04b33936c --- /dev/null +++ b/doc/install/CHANGES-broccoli-ruby.txt @@ -0,0 +1 @@ +../../aux/broccoli/bindings/broccoli-ruby/CHANGES \ No newline at end of file diff --git a/doc/install/CHANGES-broccoli.txt b/doc/install/CHANGES-broccoli.txt new file mode 120000 index 0000000000..22aeb7e410 --- /dev/null +++ b/doc/install/CHANGES-broccoli.txt @@ -0,0 +1 @@ +../../aux/broccoli/CHANGES \ No newline at end of file diff --git a/doc/install/CHANGES-broctl.txt b/doc/install/CHANGES-broctl.txt new file mode 120000 index 0000000000..fe759c681e --- /dev/null +++ b/doc/install/CHANGES-broctl.txt @@ -0,0 +1 @@ +../../aux/broctl/CHANGES \ No newline at end of file diff --git a/doc/install/CHANGES-btest.txt b/doc/install/CHANGES-btest.txt new file mode 120000 index 0000000000..1d09aed48b --- /dev/null +++ b/doc/install/CHANGES-btest.txt @@ -0,0 +1 @@ +../../aux/btest/CHANGES \ No newline at end of file diff --git a/doc/install/CHANGES-capstats.txt b/doc/install/CHANGES-capstats.txt new file mode 120000 index 0000000000..e6440ca6a9 --- /dev/null +++ b/doc/install/CHANGES-capstats.txt @@ -0,0 +1 @@ +../../aux/broctl/aux/capstats/CHANGES \ No newline at end of file diff --git a/doc/install/CHANGES-pysubnettree.txt b/doc/install/CHANGES-pysubnettree.txt new file mode 120000 index 0000000000..4067c558ba --- /dev/null +++ b/doc/install/CHANGES-pysubnettree.txt @@ -0,0 +1 @@ +../../aux/broctl/aux/pysubnettree/CHANGES \ No newline at end of file diff --git a/doc/install/CHANGES-trace-summary.txt b/doc/install/CHANGES-trace-summary.txt new file mode 120000 index 0000000000..92c76d6ee6 --- /dev/null +++ b/doc/install/CHANGES-trace-summary.txt @@ -0,0 +1 @@ +../../aux/broctl/aux/trace-summary/CHANGES \ No newline at end of file diff --git a/doc/install/NEWS.rst b/doc/install/NEWS.rst new file mode 120000 index 0000000000..8c4ba8c0d1 --- /dev/null +++ b/doc/install/NEWS.rst @@ -0,0 +1 @@ +../../NEWS \ No newline at end of file diff --git a/doc/install/changes.rst b/doc/install/changes.rst new file mode 100644 index 0000000000..32cd6ad33e --- /dev/null +++ b/doc/install/changes.rst @@ -0,0 +1,75 @@ + +======================== +Detailed Version History +======================== + +.. contents:: + +--- +Bro +--- + +.. literalinclude:: CHANGES-bro.txt + +---------- +BroControl +---------- + +.. literalinclude:: CHANGES-broctl.txt + +-------- +Broccoli +-------- + +.. literalinclude:: CHANGES-broccoli.txt + +--------------- +Broccoli Python +--------------- + +.. literalinclude:: CHANGES-broccoli-python.txt + +------------- +Broccoli Ruby +------------- + +.. literalinclude:: CHANGES-broccoli-ruby.txt + +-------- +Capstats +-------- + +.. literalinclude:: CHANGES-capstats.txt + +------------- +Trace-Summary +------------- + +.. literalinclude:: CHANGES-trace-summary.txt + + +------ +BinPAC +------ + +.. literalinclude:: CHANGES-binpac.txt + +------- +Bro-Aux +------- + +.. literalinclude:: CHANGES-bro-aux.txt + +----- +BTest +----- + +.. literalinclude:: CHANGES-btest.txt + + +------------ +PySubnetTree +------------ + +.. literalinclude:: CHANGES-pysubnettree.txt + diff --git a/doc/install/guidelines.rst b/doc/install/guidelines.rst new file mode 100644 index 0000000000..f0b10eb8d1 --- /dev/null +++ b/doc/install/guidelines.rst @@ -0,0 +1,43 @@ + +.. _upgrade-guidelines: + +================== +General Guidelines +================== + +If you're doing an upgrade install (rather than a fresh install), +there's two suggested approaches: either install Bro using the same +installation prefix directory as before, or pick a new prefix and copy +local customizations over. In the following we summarize general +guidelines for upgrading, see the `Release Notes `_ for +version-specific information. + +Re-Using Previous Install Prefix +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you choose to configure and install Bro with the same prefix +directory as before, local customization and configuration to files in +``$prefix/share/bro/site`` and ``$prefix/etc`` won't be overwritten +(``$prefix`` indicating the root of where Bro was installed). Also, logs +generated at run-time won't be touched by the upgrade. (But making +a backup of local changes before upgrading is still recommended.) + +After upgrading, remember to check ``$prefix/share/bro/site`` and +``$prefix/etc`` for ``.example`` files, which indicate the +distribution's version of the file differs from the local one, which may +include local changes. Review the differences, and make adjustments +as necessary (for differences that aren't the result of a local change, +use the new version's). + +Using a New Install prefix +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you want to install the newer version in a different prefix +directory than before, you can just copy local customization and +configuration files from ``$prefix/share/bro/site`` and ``$prefix/etc`` +to the new location (``$prefix`` indicating the root of where Bro was +originally installed). Make sure to review the files for difference +before copying and make adjustments as necessary (for differences that +aren't the result of a local change, use the new version's). Of +particular note, the copied version of ``$prefix/etc/broctl.cfg`` is +likely to need changes to the ``SpoolDir`` and ``LogDir`` settings. diff --git a/doc/install/index.rst b/doc/install/index.rst new file mode 100644 index 0000000000..71d2534921 --- /dev/null +++ b/doc/install/index.rst @@ -0,0 +1,12 @@ + +.. _installation: + +============ +Installation +============ + +.. toctree:: + :maxdepth: 2 + + install + upgrade diff --git a/doc/install/install.rst b/doc/install/install.rst new file mode 100644 index 0000000000..1b775389f3 --- /dev/null +++ b/doc/install/install.rst @@ -0,0 +1,241 @@ +.. _CMake: http://www.cmake.org +.. _SWIG: http://www.swig.org +.. _Xcode: https://developer.apple.com/xcode/ +.. _MacPorts: http://www.macports.org +.. _Fink: http://www.finkproject.org +.. _Homebrew: http://mxcl.github.com/homebrew +.. _bro downloads page: http://bro.org/download/index.html + +.. _installing-bro: + +============== +Installing Bro +============== + +.. contents:: + +Prerequisites +============= + +Before installing Bro, you'll need to ensure that some dependencies +are in place. + +Required Dependencies +--------------------- + +Bro requires the following libraries and tools to be installed +before you begin: + + * Libpcap (http://www.tcpdump.org) + * OpenSSL libraries (http://www.openssl.org) + * BIND8 library + * Libmagic + * Libz + * Bash (for BroControl) + +To build Bro from source, the following additional dependencies are required: + + * CMake 2.6.3 or greater (http://www.cmake.org) + * SWIG (http://www.swig.org) + * Bison (GNU Parser Generator) + * Flex (Fast Lexical Analyzer) + * Libpcap headers (http://www.tcpdump.org) + * OpenSSL headers (http://www.openssl.org) + * libmagic headers + * zlib headers + * Perl + +To install the required dependencies, you can use: + +* RPM/RedHat-based Linux: + + .. console:: + + sudo yum install cmake make gcc gcc-c++ flex bison libpcap-devel openssl-devel python-devel swig zlib-devel file-devel + +* DEB/Debian-based Linux: + + .. console:: + + sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev libmagic-dev + +* FreeBSD: + + Most required dependencies should come with a minimal FreeBSD install + except for the following. + + .. console:: + + sudo pkg_add -r bash cmake swig bison python + + Note that ``bash`` needs to be in ``PATH``, which by default it is + not. The FreeBSD package installs the binary into + ``/usr/local/bin``. + +* Mac OS X: + + Compiling source code on Macs requires first downloading Xcode_, + then going through its "Preferences..." -> "Downloads" menus to + install the "Command Line Tools" component. + + Lion (10.7) and Mountain Lion (10.8) come with all required + dependencies except for CMake_, SWIG_, and ``libmagic``. + + Distributions of these dependencies can likely be obtained from your + preferred Mac OS X package management system (e.g. MacPorts_, Fink_, + or Homebrew_). + + Specifically for MacPorts, the ``swig``, ``swig-ruby``, ``swig-python`` + and ``file`` packages provide the required dependencies. + + +Optional Dependencies +--------------------- + +Bro can make use of some optional libraries and tools if they are found at +build time: + + * LibGeoIP (for geo-locating IP addresses) + * gperftools (tcmalloc is used to improve memory and CPU usage) + * ipsumdump (for trace-summary; http://www.cs.ucla.edu/~kohler/ipsumdump) + * Ruby executable, library, and headers (for Broccoli Ruby bindings) + +LibGeoIP is probably the most interesting and can be easily installed +on most platforms: + +* RedHat Enterprise Linux: + + .. console:: + + sudo yum install geoip-devel sendmail + +* CentOS Linux: + + .. console:: + + sudo yum install GeoIP-devel sendmail + +* DEB/Debian-based Linux: + + .. console:: + + sudo apt-get install libgeoip-dev sendmail + +* FreeBSD using ports: + + .. console:: + + sudo pkg_add -r GeoIP + +* Mac OS X: + + Vanilla OS X installations don't ship with libGeoIP, but if + installed from your preferred package management system (e.g. + MacPorts, Fink, or Homebrew), they should be automatically detected + and Bro will compile against them. + +Additional steps may be needed to :ref:`get the right GeoIP database +`. + + +Installing Bro +============== + +Bro can be downloaded in either pre-built binary package or source +code forms. + + +Using Pre-Built Binary Release Packages +======================================= + +See the `bro downloads page`_ for currently supported/targeted +platforms for binary releases. + +* RPM + + .. console:: + + sudo yum localinstall Bro-*.rpm + +* DEB + + .. console:: + + sudo gdebi Bro-*.deb + +* MacOS Disk Image with Installer + + Just open the ``Bro-*.dmg`` and then run the ``.pkg`` installer. + Everything installed by the package will go into ``/opt/bro``. + +The primary install prefix for binary packages is ``/opt/bro``. +Non-MacOS packages that include BroControl also put variable/runtime +data (e.g. Bro logs) in ``/var/opt/bro``. + +Installing From Source +========================== + +Bro releases are bundled into source packages for convenience and +available from the `bro downloads page`_. Alternatively, the latest +Bro development version can be obtained through git repositories +hosted at ``git.bro.org``. See our `git development documentation +`_ for comprehensive +information on Bro's use of git revision control, but the short story +for downloading the full source code experience for Bro via git is: + +.. console:: + + git clone --recursive git://git.bro.org/bro + +.. note:: If you choose to clone the ``bro`` repository + non-recursively for a "minimal Bro experience", be aware that + compiling it depends on several of the other submodules as well. + +The typical way to build and install from source is (for more options, +run ``./configure --help``): + +.. console:: + + ./configure + make + make install + +The default installation path is ``/usr/local/bro``, which would typically +require root privileges when doing the ``make install``. A different +installation path can be chosen by specifying the ``--prefix`` option. +Note that ``/usr`` and ``/opt/bro`` are the +standard prefixes for binary Bro packages to be installed, so those are +typically not good choices unless you are creating such a package. + +Depending on the Bro package you downloaded, there may be auxiliary +tools and libraries available in the ``aux/`` directory. Some of them +will be automatically built and installed along with Bro. There are +``--disable-*`` options that can be given to the configure script to +turn off unwanted auxiliary projects that would otherwise be installed +automatically. Finally, use ``make install-aux`` to install some of +the other programs that are in the ``aux/bro-aux`` directory. + +OpenBSD users, please see our at `FAQ +`_ if you are having +problems installing Bro. + +Configure the Run-Time Environment +================================== + +Just remember that you may need to adjust your ``PATH`` environment variable +according to the platform/shell/package you're using. For example: + +Bourne-Shell Syntax: + +.. console:: + + export PATH=/usr/local/bro/bin:$PATH + +C-Shell Syntax: + +.. console:: + + setenv PATH /usr/local/bro/bin:$PATH + +Or substitute ``/opt/bro/bin`` instead if you installed from a binary package. + diff --git a/doc/install/release-notes.rst b/doc/install/release-notes.rst new file mode 100644 index 0000000000..e4aeec6db1 --- /dev/null +++ b/doc/install/release-notes.rst @@ -0,0 +1,13 @@ + +.. _release-notes: + +============= +Release Notes +============= + +.. contents:: + +.. include:: NEWS.rst + + + diff --git a/doc/install/upgrade.rst b/doc/install/upgrade.rst new file mode 100644 index 0000000000..29b64bdeca --- /dev/null +++ b/doc/install/upgrade.rst @@ -0,0 +1,10 @@ + +============= +Upgrading Bro +============= + +.. toctree:: + + guidelines + release-notes + changes diff --git a/doc/intro/architecture.png b/doc/intro/architecture.png new file mode 100644 index 0000000000..fc775160ca Binary files /dev/null and b/doc/intro/architecture.png differ diff --git a/doc/intro/bro-eyes.png b/doc/intro/bro-eyes.png new file mode 100644 index 0000000000..a3ac57546f Binary files /dev/null and b/doc/intro/bro-eyes.png differ diff --git a/doc/intro/history.png b/doc/intro/history.png new file mode 100644 index 0000000000..436083e222 Binary files /dev/null and b/doc/intro/history.png differ diff --git a/doc/intro/index.rst b/doc/intro/index.rst new file mode 100644 index 0000000000..7ba6c070f8 --- /dev/null +++ b/doc/intro/index.rst @@ -0,0 +1,247 @@ + +============ +Introduction +============ + +.. contents:: + +Overview +-------- + +Bro is a passive, open-source network traffic analyzer. It is +primarily a security monitor that inspects all traffic on a link in +depth for signs of suspicious activity. More generally, however, +Bro supports a wide range of traffic analysis tasks even outside of +the security domain, including performance measurements and helping +with trouble-shooting. + +The most immediate benefit that a site gains from deploying Bro is an +extensive set of *log files* that record a network's activity in +high-level terms. These logs include not only a comprehensive record +of every connection seen on the wire, but also application-layer +transcripts such as, e.g., all HTTP sessions with their requested +URIs, key headers, MIME types, and server responses; DNS requests with +replies; SSL certificates; key content of SMTP sessions; and much +more. By default, Bro writes all this information into well-structured +tab-separated log files suitable for post-processing with external +software. Users can however also chose from a set of alternative +output formats and backends to interface directly with, e.g., external +databases. + +In addition to the logs, Bro comes with built-in functionality for a +range of analysis and detection tasks, including extracting files from +HTTP sessions, detecting malware by interfacing to external +registries, reporting vulnerable versions of software seen on the +network, identifying popular web applications, detecting SSH +brute-forcing, validating SSL certificate chains, and much more. + +However, the key to understanding Bro lies in realizing that even +though the system comes with such powerful functionality out of the +box, fundamentally it represents a *platform* for traffic analyses +that's fully customizable and extensible: Bro provides users with a +domain-specific, Turing-complete *scripting language* for expressing +arbitrary analysis tasks. Conceptually, you can think of Bro as a +"domain-specific Python" (or Perl): just like Python, the system comes +with a large set of pre-built functionality (the "standard library"), +yet you are not limited to what the system ships with but can put Bro +to use in novel ways by writing your own code. Indeed, all of Bro's +default analyses, including all the logging, is the result of such +scripts; there's no specific analysis hard-coded into the core of +system. + +Bro runs on commodity hardware and hence provides a low-cost +alternative to expensive proprietary solutions. Despite the price tag, +however, Bro actually goes far beyond the capabilities of other +network monitoring tools, which typically remain limited to a small +set of hard-coded analysis tasks. We emphasize in particular that Bro +is *not* a classic signature-based intrusion detection system (IDS). +While it supports such standard functionality as well, Bro's scripting +language indeed facilitates a much broader spectrum of very different +approaches to finding malicious activity, including semantic misuse +detection, anomaly detection, and behavioral analysis. + +A large variety of sites deploy Bro operationally for protecting their +cyberinfrastructure, including many universities, research labs, +supercomputing centers, open-science communities, and major +corporations. Bro specifically targets high-speed, high-volume network +monitoring, and an increasing number of sites are now using the system +to monitor their 10GE networks, with some already moving on to 100GE +links. Bro accommodates such high-performance settings by supporting +scalable load-balancing: large sites typically run "Bro Clusters" in +which a high-speed frontend load-balancer distributes the traffic +across an appropriate number of backend PCs, all running dedicated Bro +instances on their individual traffic slices. A central manager system +coordinates the process, synchronizing state across the backends and +providing the operators with a central management interface for +configuration and access to aggregated logs. Bro's integrated +management framework, BroControl, supports such cluster setups +out-of-the-box. + +Features +-------- + +Bro supports a wide range of analyses through its scripting language. +Yet even without further customization it comes with a powerful set of +features. + +- Deployment + + * Runs on commodity hardware on standard UNIX-style systems + (including Linux, FreeBSD, and MacOS). + + * Fully passive traffic analysis off a network tap or monitoring + port. + + * Standard libpcap interface for capturing packets. + + * Real-time and offline analysis. + + * Cluster-support for large-scale deployments. + + * Unified management framework for operating both standalone and + cluster setups. + + * Open-source under a BSD license. + +- Analysis + + * Comprehensive logging of activity for offline analysis and + forensics. + + * Port-independent analysis of application-layer protocols. + + * Support for many application-layer protocols (including DNS, + FTP, HTTP, IRC, SMTP, SSH, SSL). + + * Analysis of file content exchanged over application-layer + protocols, including MD5/SHA1 computation for fingerprinting. + + * Comprehensive IPv6 support. + + * Tunnel detection and analysis (including Ayiya, Teredo, GTPv1). + Bro decapsulates the tunnels and then proceeds to analyze their + content as if no tunnel was in place. + + * Extensive sanity checks during protocol analysis. + + * Support for IDS-style pattern matching. + +- Scripting Language + + * Turing-complete language for expression arbitrary analysis + tasks. + + * Event-based programming model. + + * Domain-specific data types such as IP addresses (transparently + handling both IPv4 and IPv6), port numbers, and timers. + + * Extensive support for tracking and managing network state over + time. + +- Interfacing + + * Default output to well-structured ASCII logs. + + * Alternative backends for ElasticSearch and DataSeries. Further + database interfaces in preparation. + + * Real-time integration of external input into analyses. Live + database input in preparation. + + * External C library for exchanging Bro events with external + programs. Comes with Perl, Python, and Ruby bindings. + + * Ability to trigger arbitrary external processes from within + the scripting language. + + +History +------- + +.. figure:: history.png + :width: 600 + :align: center + :alt: Bro History Timeline + :target: history.png + + Timeline of Bro's History (click to enlarge). + +Bro's history goes back much further than many people realize. `Vern +Paxson `_ designed and implemented the +initial version almost two decades ago. +Vern began work on the code in 1995 as a researcher at the `Lawrence +Berkeley National Laboratory (LBNL) `_. Berkeley +Lab began operational deployment in 1996, and the USENIX Security +Symposium published the original Bro paper in 1998 (later refined in a +subsequent `journal publication `_). +In 2003, the +`National Science Foundation (NSF) `_ began +supporting research and advanced development on Bro at the +`International Computer Science Institute (ICSI) +`_, where Vern now leads the `Networking +and Security group `_. Over the years, a growing +team of ICSI researchers and students kept adding novel functionality +to Bro, while LBNL continued its support with funding from the +`Department of Energy (DOE) `_. + +Much of Bro's capabilities originate in academic research projects, +with results often published at top-tier conferences. However, the key +to Bro's success was its ability to bridge the traditional gap between +academia and operations from early on, which provided the research +with crucial grounding to ensure that developed approaches stand up to +the challenges of the real world. Yet, with Bro's operational user +community growing over time, the research-centric development model +eventually became a bottleneck to the system's evolution: research +grants do not tend to support the more mundane parts of software +development and maintenance, even though those prove crucial for the +end-user experience. While Bro's capabilities always went beyond those +of traditional systems, a successful deployment used to require +significant technical expertise, typically with a large upfront +investment in tackling Bro's steep learning curve. In 2010, NSF set +out to address this gap by awarding ICSI a grant dedicated solely to +Bro development out of its SDCI program. +With that support in place, the `National Center for +Supercomputing Applications (NCSA) `_ +joined the team as a core partner, and the Bro Project began to +completely overhaul many of the user-visible parts of the system for +the 2.0 release. Since that version came out, Bro has experienced an +tremendous growth in new deployments across a diverse range of +settings, and the Bro team is now working to build on this success by +further advancing the system's capabilities to address the challenges +of future networks. + +Architecture +------------ + +.. figure:: architecture.png + :width: 400 + :align: center + :alt: Bro Architecture + :target: {{docroot}}/images/architecture.png + + Bro's internal architecture. + +Architecturally, Bro is layered into two major components. Its *event +engine* (or *core*) reduces the incoming packet stream into a series +of higher-level *events*. These events reflect network activity in +policy-neutral terms, i.e., they describe *what* has been seen, but +not *why*, or whether it is significant. For example, every HTTP +request on the wire turns into a corresponding ``http_request`` event +that carries with it the involved IP addresses and ports, the URI +being requested, and the HTTP version in use. The event however does +not convey any further *interpretation*, e.g., of whether that URI +corresponds to a known malware site. + +Such semantics are instead derived by Bro's second main component, the +*script interpreter*, which executes a set of *event handlers* written +in Bro's custom scripting language. These scripts can express a site's +security policy, i.e., what actions to take when the monitor detects +different types of activity. More generally they can derive any +desired properties and statistics from the input traffic. Bro's +language comes with extensive domain-specific types and support +functionality; and, crucially, allows scripts to maintain state +over time, enabling them to track and correlate the evolution of what +they observe across connection and host boundaries. Bro scripts can +generate real-time alerts and also execute arbitrary external programs +on demand, e.g., to trigger an active response to an attack. diff --git a/doc/quickstart.rst b/doc/quickstart/index.rst similarity index 86% rename from doc/quickstart.rst rename to doc/quickstart/index.rst index 9f64e36d89..e07e729a02 100644 --- a/doc/quickstart.rst +++ b/doc/quickstart/index.rst @@ -1,29 +1,21 @@ +.. _quickstart: + ================= Quick Start Guide ================= -.. rst-class:: opening - - The short story for getting Bro up and running in a simple configuration - for analysis of either live traffic from a network interface or a packet - capture trace file. - .. contents:: -Installation -============ - Bro works on most modern, Unix-based systems and requires no custom hardware. It can be downloaded in either pre-built binary package or -source code forms. See :doc:`Installing Bro ` for instructions -on how to install Bro. +source code forms. See :ref:`installing-bro` for instructions on how to +install Bro. Below, ``$PREFIX`` is used to reference the Bro +installation root directory, which by default is ``/usr/local/`` if +you install from source. -.. note:: Below, ``$PREFIX`` is used to reference the Bro installation - root directory. - -Using BroControl -================ +Managing Bro with BroControl +============================ BroControl is an interactive shell for easily operating/managing Bro installations on a single system or even across multiple systems in a @@ -68,9 +60,10 @@ policy and output the results in ``$PREFIX/logs``. .. note:: The user starting BroControl needs permission to capture network traffic. If you are not root, you may need to grant further - privileges to the account you're using; see the :doc:`FAQ `. - Also, if it looks like Bro is not seeing any traffic, check out - the FAQ entry on checksum offloading. + privileges to the account you're using; see the `FAQ + `_. Also, if it looks + like Bro is not seeing any traffic, check out the FAQ entry on + checksum offloading. You can leave it running for now, but to stop this Bro instance you would do: @@ -115,20 +108,18 @@ columns (shortened for brevity) show a request to the root of Bro website:: Some logs are worth explicit mention: - ``weird.log`` - Contains unusual/exceptional activity that can indicate - malformed connections, traffic that doesn't conform to a particular - protocol, malfunctioning/misconfigured hardware, or even an attacker - attempting to avoid/confuse a sensor. Without context, it's hard to - judge whether this category of activity is interesting and so that is - left up to the user to configure. + ``conn.log`` + Contains an entry for every connection seen on the wire, with + basic properties such as time and duration, originator and + responder IP addresses, services and ports, payload size, and + much more. This log provides a comprehensive record of the + network's activity. ``notice.log`` Identifies specific activity that Bro recognizes as potentially interesting, odd, or bad. In Bro-speak, such activity is called a "notice". - By default, ``BroControl`` regularly takes all the logs from ``$PREFIX/logs/current`` and archives/compresses them to a directory named by date, e.g. ``$PREFIX/logs/2011-10-06``. The frequency at @@ -270,14 +261,11 @@ that only takes the email action for SSH logins to a defined set of servers: 192.168.1.102, } &redef; - redef Notice::policy += { - [$action = Notice::ACTION_EMAIL, - $pred(n: Notice::Info) = - { - return n$note == SSH::Login && n$id$resp_h in watched_servers; - } - ] - }; + hook Notice::policy(n: Notice::Info) + { + if ( n$note == SSH::SUCCESSFUL_LOGIN && n$id$resp_h in watched_servers ) + add n$actions[Notice::ACTION_EMAIL]; + } You'll just have to trust the syntax for now, but what we've done is first declare our own variable to hold a set of watched addresses, @@ -302,21 +290,26 @@ tweak the most basic options. Here's some suggestions on what to explore next: * We only looked at how to change options declared in the notice framework, there's many more options to look at in other script packages. +* Continue reading with :ref:`using-bro` chapter which goes into more + depth on working with Bro; then look at :ref:`writing-scripts` for + learning how to start writing your own scripts. * Look at the scripts in ``$PREFIX/share/bro/policy`` for further ones - you may want to load. + you may want to load; you can browse their documentation at the + :ref:`overview of script packages `. * Reading the code of scripts that ship with Bro is also a great way to gain - understanding of the language and how you can start writing your own custom - analysis. -* Review the :doc:`FAQ `. + further understanding of the language and how scripts tend to be + structured. +* Review the `FAQ `_. * Continue reading below for another mini-tutorial on using Bro as a standalone command-line utility. -Bro, the Command-Line Utility +Bro as a Command-Line Utility ============================= -If you prefer not to use BroControl (e.g. don't need its automation and -management features), here's how to directly control Bro for your analysis -activities. +If you prefer not to use BroControl (e.g. don't need its automation +and management features), here's how to directly control Bro for your +analysis activities from the command line for both live traffic and +offline working from traces. Monitoring Live Traffic ----------------------- @@ -420,3 +413,19 @@ are "load this script if it hasn't already been loaded". to be searched for scripts. See the default search path by doing ``bro --help``. +Running Bro Without Installing +------------------------------ + +For developers that wish to run Bro directly from the ``build/`` +directory (i.e., without performing ``make install``), they will have +to first adjust ``BROPATH`` and ``BROMAGIC`` to look for scripts and +additional files inside the build directory. Sourcing either +``build/bro-path-dev.sh`` or ``build/bro-path-dev.csh`` as appropriate +for the current shell accomplishes this and also augments your +``PATH`` so you can use the Bro binary directly:: + + ./configure + make + source build/bro-path-dev.sh + bro + diff --git a/doc/reporting-problems.rst b/doc/reporting-problems.rst deleted file mode 100644 index 903df76257..0000000000 --- a/doc/reporting-problems.rst +++ /dev/null @@ -1,194 +0,0 @@ - -Reporting Problems -================== - -.. rst-class:: opening - - Here we summarize some steps to follow when you see Bro doing - something it shouldn't. To provide help, it is often crucial for - us to have a way of reliably reproducing the effect you're seeing. - Unfortunately, reproducing problems can be rather tricky with Bro - because more often than not, they occur only in either very rare - situations or only after Bro has been running for some time. In - particular, getting a small trace showing a specific effect can be - a real problem. In the following, we'll summarize some strategies - to this end. - -Reporting Problems ------------------- - -Generally, when you encounter a problem with Bro, the best thing to do -is opening a new ticket in `Bro's issue tracker -`__ and include information on how to -reproduce the issue. Ideally, your ticket should come with the -following: - -* The Bro version you're using (if working directly from the git - repository, the branch and revision number.) - -* The output you're seeing along with a description of what you'd expect - Bro to do instead. - -* A *small* trace in `libpcap format `__ - demonstrating the effect (assuming the problem doesn't happen right - at startup already). - -* The exact command-line you're using to run Bro with that trace. If - you can, please try to run the Bro binary directly from the command - line rather than using BroControl. - -* Any non-standard scripts you're using (but please only those really - necessary; just a small code snippet triggering the problem would - be perfect). - -* If you encounter a crash, information from the core dump, such as - the stack backtrace, can be very helpful. See below for more on - this. - - -How Do I Get a Trace File? --------------------------- - -As Bro is usually running live, coming up with a small trace file that -reproduces a problem can turn out to be quite a challenge. Often it -works best to start with a large trace that triggers the problem, -and then successively thin it out as much as possible. - -To get to the initial large trace, here are a few things you can try: - -* Capture a trace with `tcpdump `__, either - on the same interface Bro is running on, or on another host where - you can generate traffic of the kind likely triggering the problem - (e.g., if you're seeing problems with the HTTP analyzer, record some - of your Web browsing on your desktop.) When using tcpdump, don't - forget to record *complete* packets (``tcpdump -s 0 ...``). You can - reduce the amount of traffic captured by using a suitable BPF filter - (e.g., for HTTP only, try ``port 80``). - -* Bro's command-line option ``-w `` records all packets it - processes into the given file. You can then later run Bro - offline on this trace and it will process the packets in the same - way as it did live. This is particularly helpful with problems that - only occur after Bro has already been running for some time. For - example, sometimes a crash may be triggered by a particular kind of - traffic only occurring rarely. Running Bro live with ``-w`` and - then, after the crash, offline on the recorded trace might, with a - little bit of luck, reproduce the problem reliably. However, be - careful with ``-w``: it can result in huge trace files, quickly - filling up your disk. (One way to mitigate the space issues is to - periodically delete the trace file by configuring - ``rotate-logs.bro`` accordingly. BroControl does that for you if you - set its ``SaveTraces`` option.) - -* Finally, you can try running Bro on a publically available trace - file, such as `anonymized FTP traffic `__, `headers-only enterprise traffic - `__, or - `Defcon traffic `__. Some of these - particularly stress certain components of Bro (e.g., the Defcon - traces contain tons of scans). - -Once you have a trace that demonstrates the effect, you will often -notice that it's pretty big, in particular if recorded from the link -you're monitoring. Therefore, the next step is to shrink its size as -much as possible. Here are a few things you can try to this end: - -* Very often, a single connection is able to demonstrate the problem. - If you can identify which one it is (e.g., from one of Bro's - ``*.log`` files) you can extract the connection's packets from the - trace using tcpdump by filtering for the corresponding 4-tuple of - addresses and ports: - - .. console:: - - > tcpdump -r large.trace -w small.trace host and port and host and port - -* If you can't reduce the problem to a connection, try to identify - either a host pair or a single host triggering it, and filter down - the trace accordingly. - -* You can try to extract a smaller time slice from the trace using - `TCPslice `__. For example, to - extract the first 100 seconds from the trace: - - .. console:: - - # Test comment - > tcpslice +100 out - -Alternatively, tcpdump extracts the first ``n`` packets with its -option ``-c ``. - - -Getting More Information After a Crash --------------------------------------- - -If Bro crashes, a *core dump* can be very helpful to nail down the -problem. Examining a core is not for the faint of heart but can reveal -extremely useful information. - -First, you should configure Bro with the option ``--enable-debug`` and -recompile; this will disable all compiler optimizations and thus make -the core dump more useful (don't expect great performance with this -version though; compiling Bro without optimization has a noticeable -impact on its CPU usage.). Then enable core dumps if you haven't -already (e.g., ``ulimit -c unlimited`` if you're using bash). - -Once Bro has crashed, start gdb with the Bro binary and the file -containing the core dump. (Alternatively, you can also run Bro -directly inside gdb instead of working from a core file.) The first -helpful information to include with your tracker ticket is a stack -backtrace, which you get with gdb's ``bt`` command: - -.. console:: - - > gdb bro core - [...] - > bt - - -If the crash occurs inside Bro's script interpreter, the next thing to -do is identifying the line of script code processed just before the -abnormal termination. Look for methods in the stack backtrace which -belong to any of the script interpreter's classes. Roughly speaking, -these are all classes with names ending in ``Expr``, ``Stmt``, or -``Val``. Then climb up the stack with ``up`` until you reach the first -of these methods. The object to which ``this`` is pointing will have a -``Location`` object, which in turn contains the file name and line -number of the corresponding piece of script code. Continuing the -example from above, here's how to get that information: - -.. console:: - - [in gdb] - > up - > ... - > up - > print this->location->filename - > print this->location->first_line - - -If the crash occurs while processing input packets but you cannot -directly tell which connection is responsible (and thus not extract -its packets from the trace as suggested above), try getting the -4-tuple of the connection currently being processed from the core dump -by again examining the stack backtrace, this time looking for methods -belonging to the ``Connection`` class. That class has members -``orig_addr``/``resp_addr`` and ``orig_port``/``resp_port`` storing -(pointers to) the IP addresses and ports respectively: - -.. console:: - - [in gdb] - > up - > ... - > up - > printf "%08x:%04x %08x:%04x\n", *this->orig_addr, this->orig_port, *this->resp_addr, this->resp_port - - -Note that these values are stored in `network byte order -`__ -so you will need to flip the bytes around if you are on a low-endian -machine (which is why the above example prints them in hex). For -example, if an IP address prints as ``0100007f`` , that's 127.0.0.1 . - diff --git a/doc/scripting/connection_record_01.bro b/doc/scripting/connection_record_01.bro new file mode 100644 index 0000000000..9800fb0520 --- /dev/null +++ b/doc/scripting/connection_record_01.bro @@ -0,0 +1,6 @@ +@load base/protocols/conn + +event connection_state_remove(c: connection) + { + print c; + } diff --git a/doc/scripting/connection_record_02.bro b/doc/scripting/connection_record_02.bro new file mode 100644 index 0000000000..4459e47ef6 --- /dev/null +++ b/doc/scripting/connection_record_02.bro @@ -0,0 +1,7 @@ +@load base/protocols/conn +@load base/protocols/dns + +event connection_state_remove(c: connection) + { + print c; + } diff --git a/doc/scripting/data_struct_record_01.bro b/doc/scripting/data_struct_record_01.bro new file mode 100644 index 0000000000..a80d30faae --- /dev/null +++ b/doc/scripting/data_struct_record_01.bro @@ -0,0 +1,22 @@ +type Service: record { + name: string; + ports: set[port]; + rfc: count; +}; + +function print_service(serv: Service): string + { + print fmt("Service: %s(RFC%d)",serv$name, serv$rfc); + + for ( p in serv$ports ) + print fmt(" port: %s", p); + } + +event bro_init() + { + local dns: Service = [$name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035]; + local http: Service = [$name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616]; + + print_service(dns); + print_service(http); + } diff --git a/doc/scripting/data_struct_record_02.bro b/doc/scripting/data_struct_record_02.bro new file mode 100644 index 0000000000..b10b3feac0 --- /dev/null +++ b/doc/scripting/data_struct_record_02.bro @@ -0,0 +1,41 @@ +type Service: record { + name: string; + ports: set[port]; + rfc: count; + }; + +type System: record { + name: string; + services: set[Service]; + }; + +function print_service(serv: Service): string + { + print fmt(" Service: %s(RFC%d)",serv$name, serv$rfc); + + for ( p in serv$ports ) + print fmt(" port: %s", p); + } + +function print_system(sys: System): string + { + print fmt("System: %s", sys$name); + + for ( s in sys$services ) + print_service(s); + } + +event bro_init() + { + local server01: System; + server01$name = "morlock"; + add server01$services[[ $name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035]]; + add server01$services[[ $name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616]]; + print_system(server01); + + + # local dns: Service = [ $name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035]; + # local http: Service = [ $name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616]; + # print_service(dns); + # print_service(http); + } diff --git a/doc/scripting/data_struct_set_declaration.bro b/doc/scripting/data_struct_set_declaration.bro new file mode 100644 index 0000000000..94d53c7b66 --- /dev/null +++ b/doc/scripting/data_struct_set_declaration.bro @@ -0,0 +1,22 @@ +event bro_init() + { + local ssl_ports: set[port]; + local non_ssl_ports = set( 23/tcp, 80/tcp, 143/tcp, 25/tcp ); + + # SSH + add ssl_ports[22/tcp]; + # HTTPS + add ssl_ports[443/tcp]; + # IMAPS + add ssl_ports[993/tcp]; + + # Check for SMTPS + if ( 587/tcp !in ssl_ports ) + add ssl_ports[587/tcp]; + + for ( i in ssl_ports ) + print fmt("SSL Port: %s", i); + + for ( i in non_ssl_ports ) + print fmt("Non-SSL Port: %s", i); + } diff --git a/doc/scripting/data_struct_table_complex.bro b/doc/scripting/data_struct_table_complex.bro new file mode 100644 index 0000000000..d04cee5a16 --- /dev/null +++ b/doc/scripting/data_struct_table_complex.bro @@ -0,0 +1,13 @@ +event bro_init() + { + local samurai_flicks: table[string, string, count, string] of string; + + samurai_flicks["Kihachi Okamoto", "Toho", 1968, "Tatsuya Nakadai"] = "Kiru"; + samurai_flicks["Hideo Gosha", "Fuji", 1969, "Tatsuya Nakadai"] = "Goyokin"; + samurai_flicks["Masaki Kobayashi", "Shochiku Eiga", 1962, "Tatsuya Nakadai" ] = "Harakiri"; + samurai_flicks["Yoji Yamada", "Eisei Gekijo", 2002, "Hiroyuki Sanada" ] = "Tasogare Seibei"; + + for ( [d, s, y, a] in samurai_flicks ) + print fmt("%s was released in %d by %s studios, directed by %s and starring %s", samurai_flicks[d, s, y, a], y, s, d, a); + } + diff --git a/doc/scripting/data_struct_table_declaration.bro b/doc/scripting/data_struct_table_declaration.bro new file mode 100644 index 0000000000..ab637f4c25 --- /dev/null +++ b/doc/scripting/data_struct_table_declaration.bro @@ -0,0 +1,13 @@ +event bro_init() + { + local ssl_services: table[string] of port; + + ssl_services = table(["SSH"] = 22/tcp, ["HTTPS"] = 443/tcp); + ssl_services["IMAPS"] = 993/tcp; + + if ( "SMTPS" !in ssl_services ) + ssl_services["SMTPS"] = 587/tcp; + + for ( k in ssl_services ) + print fmt("Service Name: %s - Common Port: %s", k, ssl_services[k]); + } diff --git a/doc/scripting/data_struct_vector.bro b/doc/scripting/data_struct_vector.bro new file mode 100644 index 0000000000..0c93f6cd41 --- /dev/null +++ b/doc/scripting/data_struct_vector.bro @@ -0,0 +1,7 @@ +event bro_init() + { + local v: vector of count = vector(1, 2, 3, 4); + local w = vector(1, 2, 3, 4); + print v; + print w; + } diff --git a/doc/scripting/data_struct_vector_declaration.bro b/doc/scripting/data_struct_vector_declaration.bro new file mode 100644 index 0000000000..d64754b97b --- /dev/null +++ b/doc/scripting/data_struct_vector_declaration.bro @@ -0,0 +1,15 @@ +event bro_init() + { + local v1: vector of count; + local v2 = vector(1, 2, 3, 4); + + v1[|v1|] = 1; + v1[|v1|] = 2; + v1[|v1|] = 3; + v1[|v1|] = 4; + + print fmt("contents of v1: %s", v1); + print fmt("length of v1: %d", |v1|); + print fmt("contents of v1: %s", v2); + print fmt("length of v2: %d", |v2|); + } diff --git a/doc/scripting/data_struct_vector_iter.bro b/doc/scripting/data_struct_vector_iter.bro new file mode 100644 index 0000000000..01a5ff4cd2 --- /dev/null +++ b/doc/scripting/data_struct_vector_iter.bro @@ -0,0 +1,7 @@ +event bro_init() + { + local addr_vector: vector of addr = vector(1.2.3.4, 2.3.4.5, 3.4.5.6); + + for (i in addr_vector) + print mask_addr(addr_vector[i], 18); + } diff --git a/doc/scripting/data_type_const.bro b/doc/scripting/data_type_const.bro new file mode 100644 index 0000000000..fbfd79a21e --- /dev/null +++ b/doc/scripting/data_type_const.bro @@ -0,0 +1,9 @@ +const port_list: table[port] of string &redef; + +redef port_list += { [6666/tcp] = "IRC"}; +redef port_list += { [80/tcp] = "WWW" }; + +event bro_init() + { + print port_list; + } diff --git a/doc/scripting/data_type_const_simple.bro b/doc/scripting/data_type_const_simple.bro new file mode 100644 index 0000000000..4e843fc2ac --- /dev/null +++ b/doc/scripting/data_type_const_simple.bro @@ -0,0 +1,4 @@ +@load base/protocols/http + +redef HTTP::default_capture_password = T; + diff --git a/doc/scripting/data_type_declaration.bro b/doc/scripting/data_type_declaration.bro new file mode 100644 index 0000000000..55b9d1cd1c --- /dev/null +++ b/doc/scripting/data_type_declaration.bro @@ -0,0 +1,9 @@ +event bro_init() + { + local a: int; + a = 10; + local b = 10; + + if ( a == b ) + print fmt("A: %d, B: %d", a, b); + } diff --git a/doc/scripting/data_type_interval.bro b/doc/scripting/data_type_interval.bro new file mode 100644 index 0000000000..491c0d129c --- /dev/null +++ b/doc/scripting/data_type_interval.bro @@ -0,0 +1,18 @@ +# Store the time the previous connection was established. +global last_connection_time: time; + +# boolean value to indicate whether we have seen a previous connection. +global connection_seen: bool = F; + +event connection_established(c: connection) + { + local net_time: time = network_time(); + + print fmt("%s: New connection established from %s to %s", strftime("%Y/%M/%d %H:%m:%S", net_time), c$id$orig_h, c$id$resp_h); + + if ( connection_seen ) + print fmt(" Time since last connection: %s", net_time - last_connection_time); + + last_connection_time = net_time; + connection_seen = T; + } diff --git a/doc/scripting/data_type_local.bro b/doc/scripting/data_type_local.bro new file mode 100644 index 0000000000..dfd4eb0a2a --- /dev/null +++ b/doc/scripting/data_type_local.bro @@ -0,0 +1,11 @@ +function add_two(i: count): count + { + local added_two = i+2; + print fmt("i + 2 = %d", added_two); + return added_two; + } + +event bro_init() + { + local test = add_two(10); + } diff --git a/doc/scripting/data_type_pattern_01.bro b/doc/scripting/data_type_pattern_01.bro new file mode 100644 index 0000000000..08378dd124 --- /dev/null +++ b/doc/scripting/data_type_pattern_01.bro @@ -0,0 +1,13 @@ +event bro_init() + { + local test_string = "The quick brown fox jumped over the lazy dog."; + local test_pattern = /quick|lazy/; + + if ( test_pattern in test_string ) + { + local results = split(test_string, test_pattern); + print results[1]; + print results[2]; + print results[3]; + } + } diff --git a/doc/scripting/data_type_pattern_02.bro b/doc/scripting/data_type_pattern_02.bro new file mode 100644 index 0000000000..071ed2b6c1 --- /dev/null +++ b/doc/scripting/data_type_pattern_02.bro @@ -0,0 +1,10 @@ +event bro_init() + { + local test_string = "equality"; + + local test_pattern = /equal/; + print fmt("%s and %s %s equal", test_string, test_pattern, test_pattern == test_string ? "are" : "are not"); + + test_pattern = /equality/; + print fmt("%s and %s %s equal", test_string, test_pattern, test_pattern == test_string ? "are" : "are not"); + } diff --git a/doc/scripting/data_type_subnets.bro b/doc/scripting/data_type_subnets.bro new file mode 100644 index 0000000000..1676033003 --- /dev/null +++ b/doc/scripting/data_type_subnets.bro @@ -0,0 +1,15 @@ +event bro_init() + { + local subnets = vector(172.16.0.0/20, 172.16.16.0/20, 172.16.32.0/20, 172.16.48.0/20); + local addresses = vector(172.16.4.56, 172.16.47.254, 172.16.22.45, 172.16.1.1); + + for ( a in addresses ) + { + for ( s in subnets ) + { + if ( addresses[a] in subnets[s] ) + print fmt("%s belongs to subnet %s", addresses[a], subnets[s]); + } + } + + } diff --git a/doc/scripting/data_type_time.bro b/doc/scripting/data_type_time.bro new file mode 100644 index 0000000000..6d09878205 --- /dev/null +++ b/doc/scripting/data_type_time.bro @@ -0,0 +1,4 @@ +event connection_established(c: connection) + { + print fmt("%s: New connection established from %s to %s\n", strftime("%Y/%M/%d %H:%m:%S", network_time()), c$id$orig_h, c$id$resp_h); + } diff --git a/doc/scripting/framework_logging_factorial_01.bro b/doc/scripting/framework_logging_factorial_01.bro new file mode 100644 index 0000000000..e4a171ebb6 --- /dev/null +++ b/doc/scripting/framework_logging_factorial_01.bro @@ -0,0 +1,19 @@ +module Factor; + +function factorial(n: count): count + { + if ( n == 0 ) + return 1; + else + return ( n * factorial(n - 1) ); + } + +event bro_init() + { + local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + + for ( n in numbers ) + print fmt("%d", factorial(numbers[n])); + } + + diff --git a/doc/scripting/framework_logging_factorial_02.bro b/doc/scripting/framework_logging_factorial_02.bro new file mode 100644 index 0000000000..abdbea605b --- /dev/null +++ b/doc/scripting/framework_logging_factorial_02.bro @@ -0,0 +1,32 @@ +module Factor; + +export { + redef enum Log::ID += { LOG }; + + type Info: record { + num: count &log; + factorial_num: count &log; + }; + } + +function factorial(n: count): count + { + if ( n == 0 ) + return 1; + + else + return ( n * factorial(n - 1) ); + } + +event bro_init() + { + Log::create_stream(LOG, [$columns=Info]); + } + +event bro_done() + { + local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + for ( n in numbers ) + Log::write( Factor::LOG, [$num=numbers[n], + $factorial_num=factorial(numbers[n])]); + } diff --git a/doc/scripting/framework_logging_factorial_03.bro b/doc/scripting/framework_logging_factorial_03.bro new file mode 100644 index 0000000000..c99cca2c1d --- /dev/null +++ b/doc/scripting/framework_logging_factorial_03.bro @@ -0,0 +1,45 @@ +module Factor; + +export { + redef enum Log::ID += { LOG }; + + type Info: record { + num: count &log; + factorial_num: count &log; + }; + } + +function factorial(n: count): count + { + if ( n == 0 ) + return 1; + + else + return (n * factorial(n - 1)); + } + +event bro_done() + { + local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + for ( n in numbers ) + Log::write( Factor::LOG, [$num=numbers[n], + $factorial_num=factorial(numbers[n])]); + } + +function mod5(id: Log::ID, path: string, rec: Factor::Info) : string + { + if ( rec$factorial_num % 5 == 0 ) + return "factor-mod5"; + + else + return "factor-non5"; + } + +event bro_init() + { + Log::create_stream(LOG, [$columns=Info]); + + local filter: Log::Filter = [$name="split-mod5s", $path_func=mod5]; + Log::add_filter(Factor::LOG, filter); + Log::remove_filter(Factor::LOG, "default"); + } diff --git a/doc/scripting/framework_logging_factorial_04.bro b/doc/scripting/framework_logging_factorial_04.bro new file mode 100644 index 0000000000..a563069c46 --- /dev/null +++ b/doc/scripting/framework_logging_factorial_04.bro @@ -0,0 +1,50 @@ +module Factor; + +export { + redef enum Log::ID += { LOG }; + + type Info: record { + num: count &log; + factorial_num: count &log; + }; + + global log_factor: event(rec: Info); + } + +function factorial(n: count): count + { + if ( n == 0 ) + return 1; + + else + return (n * factorial(n - 1)); + } + +event bro_init() + { + Log::create_stream(LOG, [$columns=Info, $ev=log_factor]); + } + +event bro_done() + { + local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + for ( n in numbers ) + Log::write( Factor::LOG, [$num=numbers[n], + $factorial_num=factorial(numbers[n])]); + } + +function mod5(id: Log::ID, path: string, rec: Factor::Info) : string + { + if ( rec$factorial_num % 5 == 0 ) + return "factor-mod5"; + + else + return "factor-non5"; + } + +event bro_init() + { + local filter: Log::Filter = [$name="split-mod5s", $path_func=mod5]; + Log::add_filter(Factor::LOG, filter); + Log::remove_filter(Factor::LOG, "default"); + } diff --git a/doc/scripting/framework_notice_hook_01.bro b/doc/scripting/framework_notice_hook_01.bro new file mode 100644 index 0000000000..7c4980a896 --- /dev/null +++ b/doc/scripting/framework_notice_hook_01.bro @@ -0,0 +1,7 @@ +@load policy/protocols/ssh/interesting-hostnames.bro + +hook Notice::policy(n: Notice::Info) + { + if ( n$note == SSH::Interesting_Hostname_Login ) + add n$actions[Notice::ACTION_EMAIL]; + } diff --git a/doc/scripting/framework_notice_hook_suppression_01.bro b/doc/scripting/framework_notice_hook_suppression_01.bro new file mode 100644 index 0000000000..93bb988e25 --- /dev/null +++ b/doc/scripting/framework_notice_hook_suppression_01.bro @@ -0,0 +1,7 @@ +@load policy/protocols/ssl/expiring-certs.bro + +hook Notice::policy(n: Notice::Info) + { + if ( n$note == SSL::Certificate_Expires_Soon ) + n$suppress_for = 12hrs; + } diff --git a/doc/scripting/framework_notice_shortcuts_01.bro b/doc/scripting/framework_notice_shortcuts_01.bro new file mode 100644 index 0000000000..cd51abd5b5 --- /dev/null +++ b/doc/scripting/framework_notice_shortcuts_01.bro @@ -0,0 +1,8 @@ +@load policy/protocols/ssh/interesting-hostnames.bro +@load base/protocols/ssh/ + +redef Notice::emailed_types += { + SSH::Interesting_Hostname_Login, + SSH::Login +}; + diff --git a/doc/scripting/framework_notice_shortcuts_02.bro b/doc/scripting/framework_notice_shortcuts_02.bro new file mode 100644 index 0000000000..ac427ac8b7 --- /dev/null +++ b/doc/scripting/framework_notice_shortcuts_02.bro @@ -0,0 +1,7 @@ +@load policy/protocols/ssh/interesting-hostnames.bro +@load base/protocols/ssh/ + +redef Notice::type_suppression_intervals += { + [SSH::Interesting_Hostname_Login] = 1day, + [SSH::Login] = 12hrs, +}; diff --git a/doc/scripting/index.rst b/doc/scripting/index.rst new file mode 100644 index 0000000000..a9826d9b3f --- /dev/null +++ b/doc/scripting/index.rst @@ -0,0 +1,1313 @@ + +.. _writing-scripts: + +.. contents:: + +=================== +Writing Bro Scripts +=================== + +Understanding Bro Scripts +========================= + +.. todo:: + + The MHR integration has changed significantly since the text was + written. We need to update it, however I'm actually not sure this + script is a good introductory example anymore unfortunately. + -Robin + +Bro includes an event-driven scripting language that provides +the primary means for an organization to extend and customize Bro's +functionality. Virtually all of the output generated by Bro +is, in fact, generated by Bro scripts. It's almost easier to consider +Bro to be an entity behind-the-scenes processing connections and +generating events while Bro's scripting language is the medium through +which we mere mortals can achieve communication. Bro scripts +effectively notify Bro that should there be an event of a type we +define, then let us have the information about the connection so we +can perform some function on it. For example, the ``ssl.log`` file is +generated by a Bro script that walks the entire certificate chain and +issues notifications if any of the steps along the certificate chain +are invalid. This entire process is setup by telling Bro that should +it see a server or client issue an SSL ``HELLO`` message, we want to know +about the information about that connection. + +It's often the easiest to understand Bro's scripting language by +looking at a complete script and breaking it down into its +identifiable components. In this example, we'll take a look at how +Bro queries the `Team Cymru Malware hash registry +`_ for downloads via +HTTP. Part of the Team Cymru Malware Hash registry includes the +ability to do a host lookup on a domain with the format +``MALWARE_HASH.malware.hash.cymru.com`` where ``MALWARE_HASH`` is the MD5 or +SHA1 hash of a file. Team Cymru also populates the TXT record of +their DNS responses with both a "last seen" timestamp and a numerical +"detection rate". The important aspect to understand is Bro already +generates hashes for files it can parse from HTTP streams, but the +script ``detect-MHR.bro`` is responsible for generating the +appropriate DNS lookup and parsing the response. + +.. btest-include:: ${BRO_SRC_ROOT}/scripts/policy/frameworks/files/detect-MHR.bro + +Visually, there are three distinct sections of the script. A base +level with no indentation followed by an indented and formatted +section explaining the custom variables being provided (``export``) and another +indented and formatted section describing the instructions for a +specific event (``event log_http``). Don't get discouraged if you don't +understand every section of the script; we'll cover the basics of the +script and much more in following sections. + +.. btest-include:: ${BRO_SRC_ROOT}/scripts/policy/frameworks/files/detect-MHR.bro + :lines: 7-11 + +Lines 7 and 8 of the script process the ``__load__.bro`` script in the +respective directories being loaded. The ``@load`` directives are +often considered good practice or even just good manners when writing +Bro scripts to make sure they can be +used on their own. While it's unlikely that in a +full production deployment of Bro these additional resources wouldn't +already be loaded, it's not a bad habit to try to get into as you get +more experienced with Bro scripting. If you're just starting out, +this level of granularity might not be entirely necessary though. + +.. btest-include:: ${BRO_SRC_ROOT}/scripts/policy/frameworks/files/detect-MHR.bro + :lines: 12-24 + +The export section redefines an enumerable constant that describes the +type of notice we will generate with the logging framework. Bro +allows for redefinable constants, which at first, might seem +counter-intuitive. We'll get more in-depth with constants in a later +chapter, for now, think of them as variables that can only be altered +before Bro starts running. The notice type listed allows for the use +of the :bro:id:`NOTICE` function to generate notices of type +``Malware_Hash_Registry_Match`` as done in the next section. Notices +allow Bro to generate some kind of extra notification beyond its +default log types. Often times, this extra notification comes in the +form of an email generated and sent to a pre-configured address. + +.. btest-include:: ${BRO_SRC_ROOT}/scripts/policy/frameworks/files/detect-MHR.bro + :lines: 26-44 + +The workhorse of the script is contained in the event handler for +``log_http``. The ``log_http`` event is defined as an event-hook in +the :doc:`scripts/base/protocols/http/main.bro` script and allows scripts +to handle a connection as it is being passed to the logging framework. +The event handler is passed an :bro:id:`HTTP::Info` data structure +which will be referred to as ``rec`` in body of the event handler. + +An ``if`` statement is used to check for the existence of a data structure +named ``md5`` nested within the ``rec`` data structure. Bro uses the ``$`` as +a deference operator and as such, and it is employed in this script to +check if ``rec$md5`` is present by including the ``?`` operator within the +path. If the ``rec`` data structure includes a nested data structure +named ``md5``, the statement is processed as true and a local variable +named ``hash_domain`` is provisioned and given a format string based on +the contents of ``rec$md5`` to produce a valid DNS lookup. + +The rest of the script is contained within a ``when`` block. In +short, a ``when`` block is used when Bro needs to perform asynchronous +actions, such a DNS lookup, to ensure that performance isn't effected. +The ``when`` block performs a DNS TXT lookup and stores the result +in the local variable ``MHR_result``. Effectively, processing for +this event continues and upon receipt of the values returned by +:bro:id:`lookup_hostname_txt`, the ``when`` block is executed. The +``when`` block splits the string returned into two seperate values and +checks to ensure an expected format. If the format is invalid, the +script assumes that the hash wasn't found in the respository and +processing is concluded. If the format is as expected and the +detection rate is above the threshold set by ``MHR_threshold``, two +new local variables are created and used in the notice issued by +:bro:id:`NOTICE`. + +In approximately 15 lines of actual code, Bro provides an amazing +utility that would be incredibly difficult to implement and deploy +with other products. In truth, claiming that Bro does this in 15 +lines is a misdirection; there is a truly massive number of things +going on behind-the-scenes in Bro, but it is the inclusion of the +scripting language that gives analysts access to those underlying +layers in a succinct and well defined manner. + +The Event Queue and Event Handlers +================================== + +Bro's scripting language is event driven which is a gear change from +the majority of scripting languages with which most users will have +previous experience. Scripting in Bro depends on handling the events +generated by Bro as it processes network traffic, altering the state +of data structures through those events, and making decisions on the +information provided. This approach to scripting can often cause +confusion to users who come to Bro from a procedural or functional +language, but once the initial shock wears off it becomes more clear +with each exposure. + +Bro's core acts to place events into an ordered "event queue", +allowing event handlers to process them on a first-come-first-serve +basis. In effect, this is Bro's core functionality as without the +scripts written to perform discrete actions on events, there would be +little to no usable output. As such, a basic understanding of the +event queue, the events being generated, and the way in which event +handlers process those events is a basis for not only learning to +write scripts for Bro but for understanding Bro itself. + +Gaining familiarity with the specific events generated by Bro is a big +step towards building a mind set for working with Bro scripts. The +majority of events generated by Bro are defined in the +built-in-function files or ``.bif`` files which also act as the basis for +online event documentation. These in-line comments are compiled into +an online documentation system using Broxygen. Whether starting a +script from scratch or reading and maintaining someone else's script, +having the built-in event definitions available is an excellent +resource to have on hand. For the 2.0 release the Bro developers put +significant effort into organization and documentation of every event. +This effort resulted in built-in-function files organized such that +each entry contains a descriptive event name, the arguments passed to +the event, and a concise explanation of the functions use. + +.. btest-include:: ${BRO_SRC_ROOT}/build/scripts/base/bif/plugins/Bro_DNS.events.bif.bro + :lines: 29-54 + +Above is a segment of the documentation for the event +:bro:id:`dns_request` (and the preceeding link points to the +documentation generated out of that). It's organized such that the +documentation, commentary, and list of arguments precede the actual +event definition used by Bro. As Bro detects DNS requests being +issued by an originator, it issues this event and any number of +scripts then have access to the data Bro passes along with the event. +In this example, Bro passes not only the message, the query, query +type and query class for the DNS request, but also a then record used +for the connection itself. + +The Connection Record Data Type +=============================== + +Of all the events defined by Bro, an overwhelmingly large number of +them are passed the :bro:id:`connection` record data type, in effect, +making it the backbone of many scripting solutions. The connection +record itself, as we will see in a moment, is a mass of nested data +types used to track state on a connection through its lifetime. Let's +walk through the process of selecting an appropriate event, generating +some output to standard out and dissecting the connection record so as +to get an overview of it. We will cover data types in more detail +later. + +While Bro is capable of packet level processing, its strengths lay in +the context of a connection between an originator and a responder. As +such, there are events defined for the primary parts of the connection +life-cycle as you'll see from the small selection of +connection-related events below. + +.. todo:: + + Update the line numbers, this isn't pulling in the right events + anymore but I don't know which ones it were. + +.. btest-include:: ${BRO_SRC_ROOT}/build/scripts/base/bif/event.bif.bro + :lines: 135-138,154,204-208,218,255-256,266,335-340,351 + +Of the events listed, the event that will give us the best insight +into the connection record data type will be +:bro:id:`connection_state_remove` . As detailed in the in-line +documentation, Bro generates this event just before it decides to +remove this event from memory, effectively forgetting about it. Let's +take a look at a simple script, stored as +``connection_record_01.bro``, that will output the connection record +for a single connection. + +.. btest-include:: ${DOC_ROOT}/scripting/connection_record_02.bro + +Again, we start with ``@load``, this time importing the +:doc:`scripts/base/protocols/conn` scripts which supply the tracking +and logging of general information and state of connections. We +handle the :bro:id:`connection_state_remove` event and simply print +the contents of the argument passed to it. For this example we're +going to run Bro in "bare mode" which loads only the minimum number of +scripts to retain operability and leaves the burden of loading +required scripts to the script being run. While bare mode is a low +level functionality incorporated into Bro, in this case, we're going +to use it to demonstrate how different features of Bro add more and +more layers of information about a connection. This will give us a +chance to see the contents of the connection record without it being +overly populated. + +.. btest:: connection-record-01 + + @TEST-EXEC: btest-rst-cmd bro -b -r ${TRACES}/dns-session.trace ${DOC_ROOT}/scripting/connection_record_01.bro + +As you can see from the output, the connection record is something of +a jumble when printed on its own. Regularly taking a peek at a +populated connection record helps to understand the relationship +between its fields as well as allowing an opportunity to build a frame +of reference for accessing data in a script. + +Bro makes extensive use of nested data structures to store state and +information gleaned from the analysis of a connection as a complete +unit. To break down this collection of information, you will have to +make use of use Bro's field delimiter ``$``. For example, the +originating host is referenced by ``c$id$orig_h`` which if given a +narritive relates to ``orig_h`` which is a member of ``id`` which is +a member of the data structure referred to as ``c`` that was passed +into the event handler." Given that the responder port +(``c$id$resp_p``) is ``53/tcp``, it's likely that Bro's base DNS scripts +can further populate the connection record. Let's load the +``base/protocols/dns`` scripts and check the output of our script. + +Bro uses the dollar sign as its field delimiter and a direct +correlation exists between the output of the connection record and the +proper format of a dereferenced variable in scripts. In the output of +the script above, groups of information are collected between +brackets, which would correspond to the ``$``-delimiter in a Bro script. + +.. btest-include:: ${DOC_ROOT}/scripting/connection_record_02.bro + +.. btest:: connection-record-02 + + @TEST-EXEC: btest-rst-cmd bro -b -r ${TRACES}/dns-session.trace ${DOC_ROOT}/scripting/connection_record_02.bro + +The addition of the ``base/protocols/dns`` scripts populates the +``dns=[]`` member of the connection record. While Bro is doing a +massive amount of work in the background, it is in what is commonly +called "scriptland" that details are being refined and decisions +being made. Were we to continue running in "bare mode" we could slowly +keep adding infrastructure through ``@load`` statements. For example, +were we to ``@load base/frameworks/logging``, Bro would generate a +``conn.log`` and ``dns.log`` for us in the current working directory. +As mentioned above, including the appropriate ``@load`` statements is +not only good practice, but can also help to indicate which +functionalities are being used in a script. Take a second to run the +script without the ``-b`` flag and check the output when all of Bro's +functionality is applied to the tracefile. + +Data Types and Data Structures +============================== + +Scope +----- + +Before embarking on a exploration of Bro's native data types and data +structures, it's important to have a good grasp of the different +levels of scope available in Bro and the appropriate times to use them +within a script. The declarations of variables in Bro come in two +forms. Variables can be declared with or without a definition in the +form ``SCOPE name: TYPE`` or ``SCOPE name = EXPRESSION`` respectively; +each of which produce the same result if ``EXPRESSION`` evaluates to the +same type as ``TYPE``. The decision as to which type of declaration to +use is likely to be dictated by personal preference and readability. + +.. btest-include:: ${DOC_ROOT}/scripting/data_type_declaration.bro + +Global Variables +~~~~~~~~~~~~~~~~ + +A global variable is used when the state of variable needs to be +tracked, not surprisingly, globally. While there are some caveats, +when a script declares a variable using the global scope, that script +is granting access to that variable from other scripts. However, when +a script uses the ``module`` keyword to give the script a namespace, +more care must be given to the declaration of globals to ensure the +intended result. When a global is declared in a script with a +namespace there are two possible outcomes. First, the variable is +available only within the context of the namespace. In this scenario, +other scripts within the same namespace will have access to the +variable declared while scripts using a different namespace or no +namespace altogether will not have access to the variable. +Alternatively, if a global variable is declared within an ``export { ... }`` +block that variable is available to any other script through the +naming convention of ``MODULE::variable_name``. + +The declaration below is taken from the +:doc:`scripts/policy/protocols/conn/known-hosts.bro` script and +declares a variable called ``known_hosts`` as a global set of unique +IP addresses within the ``Known`` namespace and exports it for use +outside of the ``Known`` namespace. Were we to want to use the +``known_hosts`` variable we'd be able to access it through +``Known::known_hosts``. + +.. btest-include:: ${BRO_SRC_ROOT}/scripts/policy/protocols/conn/known-hosts.bro + :lines: 8-10, 32, 37 + +The sample above also makes use of an ``export { ... }`` block. When the module +keyword is used in a script, the variables declared are said to be in +that module's "namespace". Where as a global variable can be accessed +by its name alone when it is not declared within a module, a global +variable declared within a module must be exported and then accessed +via ``MODULE_NAME::VARIABLE_NAME``. As in the example above, we would be +able to access the ``known_hosts`` in a separate script variable via +``Known::known_hosts`` due to the fact that ``known_hosts`` was declared as +a global variable within an export block under the ``Known`` namespace. + + +Constants +~~~~~~~~~ + +Bro also makes use of constants, which are denoted by the ``const`` +keyword. Unlike globals, constants can only be set or altered at +parse time if the ``&redef`` attribute has been used. Afterwards (in +runtime) the constants are unalterable. In most cases, redefinable +constants are used in Bro scripts as containers for configuration +options. For example, the configuration option to log password +decrypted from HTTP streams is stored in +``HTTP::default_capture_password`` as shown in the stripped down +excerpt from :doc:`scripts/scripts/base/protocols/http/main.bro` +below. + +.. btest-include:: ${BRO_SRC_ROOT}/scripts/base/protocols/http/main.bro + :lines: 8-10,19,20,118 + +Because the constant was declared with the ``&redef`` attribute, if we +needed to turn this option on globally, we could do so by adding the +following line to our ``site/local.bro`` file before firing up Bro. + +.. btest-include:: ${DOC_ROOT}/scripting/data_type_const_simple.bro + +While the idea of a redefinable constant might be odd, the constraint +that constants can only be altered at parse-time remains even with the +``&redef`` attribute. In the code snippet below, a table of strings +indexed by ports is declared as a constant before two values are added +to the table through ``redef`` statements. The table is then printed +in a :bro:id:`bro_init` event. Were we to try to alter the table in +an event handler, Bro would notify the user of an error and the script +would fail. + +.. btest-include:: ${DOC_ROOT}/scripting/data_type_const.bro + +.. btest:: data_type_const.bro + + @TEST-EXEC: btest-rst-cmd bro -b ${DOC_ROOT}/scripting/data_type_const.bro + +Local Variables +~~~~~~~~~~~~~~~ + +Whereas globals and constants are widely available in scriptland +through various means, when a variable is defined with a local scope, +its availability is restricted to the body of the event or function in +which it was declared. Local variables tend to be used for values +that are only needed within a specific scope and once the processing +of a script passes beyond that scope and no longer used, the variable +is deleted. Bro maintains names of locals separately from globally +visible ones, an example of which is illustrated below. The script +executes the event handler :bro:id:`bro_init` which in turn calls the +function ``add_two(i: count)`` with an argument of ``10``. Once Bro +enters the ``add_two`` function, it provisions a locally scoped +variable called ``added_two`` to hold the value of ``i+2``, in this +case, ``12``. The ``add_two`` function then prints the value of the +``added_two`` variable and returns its value to the ``bro_init`` event +handler. At this point, the variable ``added_two`` has fallen out of +scope and no longer exists while the value ``12`` still in use and +stored in the locally scoped variable ``test``. When Bro finishes +processing the ``bro_init`` function, the variable called ``test`` is +no longer in scope and, since there exist no other references to the +value ``12``, the value is also deleted. + +.. btest-include:: ${DOC_ROOT}/scripting/data_type_local.bro + + +Data Structures +--------------- + +It's difficult to talk about Bro's data types in a practical manner +without first covering the data structures available in Bro. Some of +the more interesting characteristics of data types are revealed when +used inside of a data structure, but given that data structures are +made up of data types, it devolves rather quickly into a +"chicken-and-egg" problem. As such, we'll introduce data types from +a bird's eye view before diving into data structures and from there a +more complete exploration of data types. + +The table below shows the atomic types used in Bro, of which the +first four should seem familiar if you have some scripting experience, +while the remaining six are less common in other languages. It should +come as no surprise that a scripting language for a Network Security +Monitoring platform has a fairly robust set of network centric data +types and taking note of them here may well save you a late night of +reinventing the wheel. + ++-----------+-------------------------------------+ +| Data Type | Description | ++===========+=====================================+ +| int | 64 bit signed integer | ++-----------+-------------------------------------+ +| count | 64 bit unsigned integer | ++-----------+-------------------------------------+ +| double | double precision floating precision | ++-----------+-------------------------------------+ +| bool | boolean (T/F) | ++-----------+-------------------------------------+ +| addr | IP address, IPv4 and IPv6 | ++-----------+-------------------------------------+ +| port | transport layer port | ++-----------+-------------------------------------+ +| subnet | CIDR subnet mask | ++-----------+-------------------------------------+ +| time | absolute epoch time | ++-----------+-------------------------------------+ +| interval | a time interval | ++-----------+-------------------------------------+ +| pattern | regular expression | ++-----------+-------------------------------------+ + +Sets +~~~~ + +Sets in Bro are used to store unique elements of the same data +type. In essence, you can think of them as "a unique set of integers" +or "a unique set of IP addresses". While the declaration of a set may +differ based on the data type being collected, the set will always +contain unique elements and the elements in the set will always be of +the same data type. Such requirements make the set data type perfect +for information that is already naturally unique such as ports or IP +addresses. The code snippet below shows both an explicit and implicit +declaration of a locally scoped set. + +.. btest-include:: ${DOC_ROOT}/scripting/data_struct_set_declaration.bro + :lines: 1-4,22 + +As you can see, sets are declared using the format ``SCOPE var_name: +set[TYPE]``. Adding and removing elements in a set is achieved using +the ``add`` and ``delete`` statements. Once you have elements inserted into +the set, it's likely that you'll need to either iterate over that set +or test for membership within the set, both of which are covered by +the ``in`` operator. In the case of iterating over a set, combining the +``for`` statement and the ``in`` operator will allow you to sequentially +process each element of the set as seen below. + +.. btest-include:: ${DOC_ROOT}/scripting/data_struct_set_declaration.bro + :lines: 17-21 + +Here, the ``for`` statement loops over the contents of the set storing +each element in the temporary variable ``i``. With each iteration of +the ``for`` loop, the next element is chosen. Since sets are not an +ordered data type, you cannot guarantee the order of the elements as +the ``for`` loop processes. + +To test for membership in a set the ``in`` statment can be combined +with an ``if`` statement to return a true or false value. If the +exact element in the condition is already in the set, the condition +returns true and the body executes. The ``in`` statement can also be +negated by the ``!`` operator to create the inverse of the condition. +While we could rewrite the corresponding line below as ``if ( !( +587/tcp in ssl_ports ))`` try to avoid using this construct; instead, +negate the in operator itself. While the functionality is the same, +using the ``!in`` is more efficient as well as a more natural construct +which will aid in the readability of your script. + +.. btest-include:: ${DOC_ROOT}/scripting/data_struct_set_declaration.bro + :lines: 13-15 + +You can see the full script and its output below. + +.. btest-include:: ${DOC_ROOT}/scripting/data_struct_set_declaration.bro + +.. btest:: data_struct_set_declaration + + @TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_struct_set_declaration.bro + +Tables +~~~~~~ + +A table in Bro is a mapping of a key to a value or yield. While the +values don't have to be unique, each key in the table must be unique +to preserve a one-to-one mapping of keys to values. In the example +below, we've compiled a table of SSL-enabled services and their common +ports. The explicit declaration and constructor for the table on +lines 3 and 4 lay out the data types of the keys (strings) and the +data types of the yields (ports) and then fill in some sample key and +yield pairs. Line 5 shows how to use a table accessor to insert one +key-yield pair into the table. When using the ``in`` operator on a table, +you are effectively working with the keys of the table. In the case +of an ``if`` statement, the ``in`` operator will check for membership among +the set of keys and return a true or false value. As seen on line 7, +we are checking if ``SMTPS`` is not in the set of keys for the +ssl_services table and if the condition holds true, we add the +key-yield pair to the table. Line 12 shows the use of a ``for`` statement +to iterate over each key currently in the table. + +.. btest-include:: ${DOC_ROOT}/scripting/data_struct_table_declaration.bro + +.. btest:: data_struct_table_declaration + + @TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_struct_table_declaration.bro + +Simple examples aside, tables can become extremely complex as the keys +and values for the table become more intricate. Tables can have keys +comprised of multiple data types and even a series of elements called +a "tuple". The flexibility gained with the use of complex tables in +Bro implies a cost in complexity for the person writing the scripts +but pays off in effectiveness given the power of Bro as a network +security platform. + +The script below shows a sample table of strings indexed by two +strings, a count, and a final string. With a tuple acting as an +aggregate key, the order is the important as a change in order would +result in a new key. Here, we're using the table to track the +director, studio, year or release, and lead actor in a series of +samurai flicks. It's important to note that in the case of the ``for`` +statement, it's an all or nothing kind of iteration. We cannot +iterate over, say, the directors; we have to iterate with the exact +format as the keys themselves. In this case, we need squared brackets +surrounding four temporary variables to act as a collection for our +iteration. While this is a contrived example, we could easily have +had keys containin IP addresses (``addr``), ports (``port``) and even a ``string`` +calculated as the result of a reverse hostname lookup. + +.. btest-include:: ${DOC_ROOT}/scripting/data_struct_table_complex.bro + +.. btest:: data_struct_table_complex + + @TEST-EXEC: btest-rst-cmd bro -b ${DOC_ROOT}/scripting/data_struct_table_complex.bro + +Vectors +~~~~~~~ + +If you're coming to Bro with a programming background, you may or may +not be familiar with a vector data type depending on your language of +choice. On the surface, vectors perform much of the same +functionality as associative arrays with unsigned integers as their +indices. They are however more efficient than that and they allow for +ordered access. As such any time you need to sequentially store data of the +same type, in Bro you should reach for a vector. Vectors are a +collection of objects, all of which are of the same data type, to +which elements can be dynamically added or removed. Since Vectors use +contiguous storage for their elements, the contents of a vector can be +accessed through a zero-indexed numerical offset. + +The format for the declaration of a Vector follows the pattern of +other declarations, namely, ``SCOPE v: vector of T`` where ``v`` is +the name of your vector, and ``T`` is the data type of its members. +For example, the following snippet shows an explicit and implicit +declaration of two locally scoped vectors. The script populates the +first vector by inserting values at the end; it does that by placing +the vector name between two vertical pipes to get the vector's current +length before printing the contents of both Vectors and their current +lengths. + +.. btest-include:: ${DOC_ROOT}/scripting/data_struct_vector_declaration.bro + +.. btest:: data_struct_vector_declaration + + @TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_struct_vector_declaration.bro + +In a lot of cases, storing elements in a vector is simply a precursor +to then iterating over them. Iterating over a vector is easy with the +``for`` keyword. The sample below iterates over a vector of IP +addresses and for each IP address, masks that address with 18 bits. +The ``for`` keyword is used to generate a locally scoped variable +called ``i`` which will hold the index of the current element in the +vector. Using ``i`` as an index to addr_vector we can access the +current item in the vector with ``addr_vector[i]``. + +.. btest-include:: ${DOC_ROOT}/scripting/data_struct_vector_iter.bro + +.. btest:: data_struct_vector_iter + + @TEST-EXEC: btest-rst-cmd bro -b ${DOC_ROOT}/scripting/data_struct_vector_iter.bro + + +Data Types Revisited +-------------------- + +addr +~~~~ + +The ``addr``, or address, data type manages to cover a surprisingly +large amount of ground while remaining succinct. IPv4, IPv6 and even +hostname constants are included in the ``addr`` data type. While IPv4 +addresses use the default dotted quad formatting, IPv6 addresses use +the RFC 2373 defined notation with the addition of squared brackets +wrapping the entire address. When you venture into hostname +constants, Bro performs a little slight of hand for the benefit of the +user; a hostname constant is, in fact, a set of addresses. Bro will +issue a DNS request when it sees a hostname constant in use and return +a set whose elements are the answers to the DNS request. For example, +if you were to use ``local google = www.google.com;`` you would end up +with a locally scoped ``set[addr]`` with elements that represent the +current set of round robin DNS entries for google. At first blush, +this seems trivial, but it is yet another example of Bro making the +life of the common Bro scripter a little easier through abstraction +applied in a practical manner. (Note however that these IP addresses +will never get updated during Bro's processing, so often this +mechanism most useful for addresses that are expected to remain +static.). + +port +~~~~ + +Transport layer port numbers in Bro are represented in the format of +``/``, e.g., ``22/tcp`` or +``53/udp``. Bro supports TCP(``/tcp``), UDP(``/udp``), +ICMP(``/icmp``) and UNKNOWN(``/unknown``) as protocol designations. +While ICMP doesn't have an actual port, Bro supports the concept of +ICMP "ports" by using the ICMP message type and ICMP message code as +the source and destination port respectively. Ports can be compared +for equality using the ``==`` or ``!=`` operators and can even be +compared for ordering. Bro gives the protocol designations the +following "order": ``unknown`` < ``tcp`` < ``udp`` < ``icmp``. For +example ``65535/tcp`` is smaller than ``0/udp``. + +subnet +~~~~~~ + +Bro has full support for CIDR notation subnets as a base data type. +There is no need to manage the IP and the subnet mask as two seperate +entities when you can provide the same information in CIDR notation in +your scripts. The following example below uses a Bro script to +determine if a series of IP addresses are within a set of subnets +using a 20 bit subnet mask. + +.. btest-include:: ${DOC_ROOT}/scripting/data_type_subnets.bro + +Because this is a script that doesn't use any kind of network +analysis, we can handle the event :bro:id:`bro_init` which is always +generated by Bro's core upon startup. On lines six and seven, two +locally scoped vectors are created to hold our lists of subnets and IP +addresses respectively. Then, using a set of nested ``for`` loops, we +iterate over every subnet and every IP address and use an ``if`` +statement to compare an IP address against a subnet using the ``in`` +operator. The ``in`` operator returns true if the IP address falls +within a given subnet based on the longest prefix match calculation. +For example, ``10.0.0.1 in 10.0.0.0/8`` would return true while +``192.168.2.1 in 192.168.1.0/24`` would return false. When we run the +script, we get the output listing the IP address and the subnet in +which it belongs. + +.. btest:: data_type_subnets + + @TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_type_subnets.bro + +time +~~~~ + +While there is currently no supported way to add a time constant in +Bro, two built-in functions exist to make use of the ``time`` data +type. Both :bro:id:`network_time` and :bro:id:`current_time` return a +``time`` data type but they each return a time based on different +criteria. The ``current_time`` function returns what is called the +wall-clock time as defined by the operating system. However, +``network_time`` returns the timestamp of the last packet processed +be it from a live data stream or saved packet capture. Both functions +return the time in epoch seconds, meaning ``strftime`` must be used to +turn the output into human readable output. The script below makes +use of the :bro:id:`connection_established` event handler to generate text +every time a SYN/ACK packet is seen responding to a SYN packet as part +of a TCP handshake. The text generated, is in the format of a +timestamp and an indication of who the originator and responder were. +We use the ``strftime`` format string of ``%Y%M%d %H:%m:%S`` to +produce a common date time formatted time stamp. + +.. btest-include:: ${DOC_ROOT}/scripting/data_type_time.bro + +When the script is executed we get an output showing the details of +established connections. + +.. btest:: data_type_time + + @TEST-EXEC: btest-rst-cmd bro -r ${TRACES}/wikipedia.trace ${DOC_ROOT}/scripting/data_type_time.bro + +interval +~~~~~~~~ + +The interval data type is another area in Bro where rational +application of abstraction makes perfect sense. As a data type, the +interval represents a relative time as denoted by a numeric constant +followed by a unit of time. For example, 2.2 seconds would be +``2.2sec`` and thirty-one days would be represented by ``31days``. +Bro supports ``usec``, ``msec``, ``sec``, ``min``, ``hr``, or ``day`` which represent +microseconds, milliseconds, seconds, minutes, hours, and days +respectively. In fact, the interval data type allows for a surprising +amount of variation in its definitions. There can be a space between +the numeric constant or they can crammed together like a temporal +portmanteau. The time unit can be either singular or plural. All of +this adds up to to the fact that both ``42hrs`` and ``42 hr`` are +perfectly valid and logically equivalent in Bro. The point, however, +is to increase the readability and thus maintainability of a script. +Intervals can even be negated, allowing for ``- 10mins`` to represent +"ten minutes ago". + +Intervals in Bro can have mathematical operations performed against +them allowing the user to perform addition, subtraction, +multiplication, division, and comparison operations. As well, Bro +returns an interval when comparing two ``time`` values using the ``-`` +operator. The script below amends the script started in the section +above to include a time delta value printed along with the connection +establishment report. + +.. btest-include:: ${DOC_ROOT}/scripting/data_type_interval.bro + +This time, when we execute the script we see an additional line in the +output to display the time delta since the last fully established +connection. + +.. btest:: data_type_interval + + @TEST-EXEC: btest-rst-cmd bro -r ${TRACES}/wikipedia.trace ${DOC_ROOT}/scripting/data_type_interval.bro + + +Pattern +~~~~~~~ + +Bro has support for fast text searching operations using regular +expressions and even goes so far as to declare a native data type for +the patterns used in regular expressions. A pattern constant is +created by enclosing text within the forward slash characters. Bro +supports syntax very similar to the Flex lexical analyzer syntax. The +most common use of patterns in Bro you are likely to come across is +embedded matching using the ``in`` operator. Embedded matching +adheres to a strict format, requiring the regular expression or +pattern constant to be on the left side of the ``in`` operator and the +string against which it will be tested to be on the right. + +.. btest-include:: ${DOC_ROOT}/scripting/data_type_pattern_01.bro + +In the sample above, two local variables are declared to hold our +sample sentence and regular expression. Our regular expression in +this case will return true if the string contains either the word +``quick`` or the word ``fox``. The ``if`` statement on line six uses +embedded matching and the ``in`` operator to check for the existence +of the pattern within the string. If the statement resolves to true, +:bro:id:`split` is called to break the string into separate pieces. +``Split`` takes a string and a pattern as its arguments and returns a +table of strings indexed by a count. Each element of the table will +be the segments before and after any matches against the pattern but +excluding the actual matches. In this case, our pattern matches +twice, and results in a table with three entries. Lines 11 through 13 +print the contents of the table in order. + +.. btest:: data_type_pattern + + @TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_type_pattern_01.bro + +Patterns can also be used to compare strings using equality and +inequality operators through the ``==`` and ``!=`` operators +respectively. When used in this manner however, the string must match +entirely to resolve to true. For example, the script below uses two +ternary conditional statements to illustrate the use of the ``==`` +operators with patterns. On lines 5 and 8 the output is altered based +on the result of the comparison between the pattern and the string. + +.. btest-include:: ${DOC_ROOT}/scripting/data_type_pattern_02.bro + +.. btest:: data_type_pattern_02 + + @TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_type_pattern_02.bro + + + +Record Data Type +---------------- + +With Bro's support for a wide array of data types and data structures, +an obvious extension is to include the ability to create custom +data types composed of atomic types and further data structures. To +accomplish this, Bro introduces the ``record`` type and the ``type`` +keyword. Similar to how you would define a new data structure in C +with the ``typedef`` and ``struct`` keywords, Bro allows you to cobble +together new data types to suit the needs of your situation. + +When combined with the ``type`` keyword, ``record`` can generate a +composite type. We have, in fact, already encountered a a complex +example of the ``record`` data type in the earlier sections, the +:bro:id:`connection` record passed to many events. Another one, +:bro:id:`Conn::Info`, which corresponds to the fields logged into +``conn.log``, is shown by the exerpt below. + +.. btest-include:: ${BRO_SRC_ROOT}/scripts/base/protocols/conn/main.bro + :lines: 10-12,16,17,19,21,23,25,28,31,35,37,56,62,68,90,93,97,100,104,108,109,114 + +Looking at the structure of the definition, a new collection of data +types is being defined as a type called ``Info``. Since this type +definition is within the confines of an export block, what is defined +is, in fact, ``Conn::Info``. + +The formatting for a declaration of a record type in Bro includes the +descriptive name of the type being defined and the seperate fields +that make up the record. The individual fields that make up the new +record are not limited in type or number as long as the name for each +field is unique. + +.. btest-include:: ${DOC_ROOT}/scripting/data_struct_record_01.bro + +.. btest:: data_struct_record_01 + + @TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_struct_record_01.bro + +The sample above shows a simple type definition that includes a +string, a set of ports, and a count to define a service type. Also +included is a function to print each field of a record in a formatted +fashion and a :bro:id:`bro_init` event handler to show some +functionality of working with records. The definitions of the DNS and +HTTP services are both done inline using squared brackets before being +passed to the ``print_service`` function. The ``print_service`` +function makes use of the ``$`` dereference operator to access the +fields within the newly defined Service record type. + +As you saw in the definition for the ``Conn::Info`` record, other +records are even valid as fields within another record. We can extend +the example above to include another record that contains a Service +record. + +.. btest-include:: ${DOC_ROOT}/scripting/data_struct_record_02.bro + +.. btest:: data_struct_record_02 + + @TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_struct_record_02.bro + +The example above includes a second record type in which a field is +used as the data type for a set. Records can be reapeatedly nested +within other records, their fields reachable through repeated chains +of the ``$`` dereference operator. + +It's also common to see a ``type`` used to simply alias a data +structure to a more descriptive name. The example below shows an +example of this from Bro's own type definitions file. + +.. btest-include:: ${BRO_SRC_ROOT}/scripts/base/init-bare.bro + :lines: 12,19,26 + +The three lines above alias a type of data structure to a descriptive +name. Functionally, the operations are the same, however, each of the +types above are named such that their function is instantly +identifiable. This is another place in Bro scripting where +consideration can lead to better readability of your code and thus +easier maintainability in the future. + + +Custom Logging +============== + +Armed with a decent understanding of the data types and data +structures in Bro, exploring the various frameworks available is a +much more rewarding effort. The framework with which most users are +likely to have the most interaction is the Logging Framework. +Designed in such a way to so as to abstract much of the process of +creating a file and appending ordered and organized data into it, the +Logging Framework makes use of some potentially unfamiliar +nomenclature. Specifically, Log Streams, Filters and Writers are +simply abstractions of the processes required to manage a high rate of +incoming logs while maintaining full operability. If you've seen Bro +employed in an environment with a large number of connections, you +know that logs are produced incredibly quickly; the ability to process +a large set of data and write it to disk is due to the design of the +Logging Framework. + +Data is written to a Log Stream based on decision making processes in +Bro's scriptland. Log Streams correspond to a single log as defined +by the set of name/value pairs that make up its fields. That data can +then be filtered, modified, or redirected with Logging Filters which, +by default, are set to log everything. Filters can be used to break +log files into subsets or duplicate that information to another +output. The final output of the data is defined by the writer. Bro's +default writer is simple tab separated ASCII files but Bro also +includes support for `DataSeries `_ +and `Elasticsearch `_ outputs as well as +additional writers currently in development. While these new terms +and ideas may give the impression that the Logging Framework is +difficult to work with, the actual learning curve is, in actuality, +not very steep at all. The abstraction built into the Logging +Framework makes it such that a vast majority of scripts needs not go +past the basics. In effect, writing to a log file is as simple as +defining the format of your data, letting Bro know that you wish to +create a new log, and then calling the :bro:id:`Log::write` method to +output log records. + +The Logging Framework is an area in Bro where, the more you see it +used and the more you use it yourself, the more second nature the +boilerplate parts of the code will become. As such, let's work +through a contrived example of simply logging the digits 1 through 10 +and their corresponding factorial to the default ASCII log writer. +It's always best to work through the problem once, simulating the +desired output with ``print`` and ``fmt`` before attempting to dive +into the Logging Framework. Below is a script that defines a +factorial function to recursively calculate the factorial of a +unsigned integer passed as an argument to the function. Using +:bro:id:`print` :bro:id:`fmt` we can ensure that Bro can perform these +calculations correctly as well get an idea of the answers ourselves. + +.. btest-include:: ${DOC_ROOT}/scripting/framework_logging_factorial_01.bro + +.. btest:: framework_logging_factorial + + @TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/framework_logging_factorial_01.bro + +The output of the script aligns with what we expect so now it's time +to integrate the Logging Framework. As mentioned above we have to +perform a few steps before we can issue the :bro:id:`Log::write` +method and produce a logfile. As we are working within a namespace +and informing an outside entity of workings and data internal to the +namespace, we use an ``export`` block. First we need to inform Bro +that we are going to be adding another Log Stream by adding a value to +the :bro:id:`Log::ID` enumerable. In line 3 of the script, we append the +value ``LOG`` to the ``Log::ID`` enumerable, however due to this being in +an export block the value appended to ``Log::ID`` is actually +``Factor::Log``. Next, we need to define the name and value pairs +that make up the data of our logs and dictate its format. Lines 5 +through 9 define a new datatype called an ``Info`` record (actually, +``Factor::Info``) with two fields, both unsigned integers. Each of the +fields in the ``Factor::Log`` record type include the ``&log`` +attribute, indicating that these fields should be passed to the +Logging Framework when ``Log::write`` is called. Were there to be +any name value pairs without the ``&log`` attribute, those fields +would simply be ignored during logging but remain available for the +lifespan of the variable. The next step is to create the logging +stream with :bro:id:`Log::create_stream` which takes a Log::ID and a +record as its arguments. In this example, on line 28, we call the +``Log::create_stream`` method and pass ``Factor::LOG`` and the +``Factor::Info`` record as arguments. From here on out, if we issue +the ``Log::write`` command with the correct ``Log::ID`` and a properly +formatted ``Factor::Info`` record, a log entry will be generated. + +.. btest-include:: ${DOC_ROOT}/scripting/framework_logging_factorial_02.bro + +Now, if we run the new version of the script, instead of generating +logging information to stdout, no output is created. Instead the +output is all in ``factor.log``, properly formatted and organized. + +.. btest:: framework_logging_factorial-2 + + @TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/framework_logging_factorial_02.bro + @TEST-EXEC: btest-rst-include factor.log + +While the previous example is a simplistic one, it serves to +demonstrate the small pieces of script code hat need to be in place in +order to generate logs. For example, it's common to call +``Log::create_stream`` in :bro:id:`bro_init` and while in a live +example, determining when to call ``Log::write`` would likely be +done in an event handler, in this case we use :bro:id:`bro_done` . + +If you've already spent time with a deployment of Bro, you've likely +had the opportunity to view, search through, or manipulate the logs +produced by the Logging Framework. The log output from a default +installation of Bro is substantial to say the least, however, there +are times in which the way the Logging Framework by default isn't +ideal for the situation. This can range from needing to log more or +less data with each call to ``Log::write`` or even the need to split +log files based on arbitrary logic. In the later case, Filters come +into play along with the Logging Framework. Filters grant a level of +customization to Bro's scriptland, allowing the script writer to +include or exclude fields in the log and even make alterations to the +path of the file in which the logs are being placed. Each stream, +when created, is given a default filter called, not surprisingly, +``default``. When using the ``default`` filter, every key value pair +with the ``&log`` attribute is written to a single file. For the +example we've been using, let's extend it so as to write any factorial +which is a factor of 5 to an alternate file, while writing the +remaining logs to factor.log. + +.. btest-include:: ${DOC_ROOT}/scripting/framework_logging_factorial_03.bro + :lines: 38-62 + :linenos: + +To dynamically alter the file in which a stream writes its logs a +filter can specify function returns a string to be used as the +filename for the current call to ``Log::write``. The definition for +this function has to take as its parameters a ``Log::ID`` called id, a +string called ``path`` and the appropriate record type for the logs called +``rec``. You can see the definition of ``mod5`` used in this example on +line one conforms to that requirement. The function simply returns +``factor-mod5`` if the factorial is divisible evenly by 5, otherwise, it +returns ``factor-non5``. In the additional ``bro_init`` event +handler, we define a locally scoped ``Log::Filter`` and assign it a +record that defines the ``name`` and ``path_func`` fields. We then +call ``Log::add_filter`` to add the filter to the ``Factor::LOG`` +``Log::ID`` and call ``Log::remove_filter`` to remove the ``default`` +filter for ``Factor::LOG``. Had we not removed the ``default`` filter, +we'd have ended up with three log files: ``factor-mod5.log`` with all the +factorials that are a factors of 5, ``factor-non5.log`` with the +factorials that are not factors of 5, and ``factor.log`` which would have +included all factorials. + +.. btest:: framework_logging_factorial-3 + + @TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/framework_logging_factorial_03.bro + @TEST-EXEC: btest-rst-include factor-mod5.log + +The ability of Bro to generate easily customizable and extensible logs +which remain easily parsable is a big part of the reason Bro has +gained a large measure of respect. In fact, it's difficult at times +to think of something that Bro doesn't log and as such, it is often +advantageous for analysts and systems architects to instead hook into +the logging framework to be able to perform custom actions based upon +the data being sent to the Logging Frame. To that end, every default +log stream in Bro generates a custom event that can be handled by +anyone wishing to act upon the data being sent to the stream. By +convention these events are usually in the format ``log_x`` where x is +the name of the logging stream; as such the event raised for every log +sent to the Logging Framework by the HTTP parser would be +``log_http``. In fact, we've already seen a script handle the +``log_http`` event when we broke down how the ``detect-MHR.bro`` +script worked. In that example, as each log entry was sent to the +logging framework, post-processing was taking place in the +``log_http`` event. Instead of using an external script to parse the +``http.log`` file and do post-processing for the entry, +post-processing can be done in real time in Bro. + +Telling Bro to raise an event in your own Logging stream is as simple +as exporting that event name and then adding that event in the call to +``Log::create_stream``. Going back to our simple example of logging +the factorial of an integer, we add ``log_factor`` to the ``export`` +block and define the value to be passed to it, in this case the +``Factor::Info`` record. We then list the ``log_factor`` function as +the ``$ev`` field in the call to ``Log::create_stream`` + +.. btest-include:: ${DOC_ROOT}/scripting/framework_logging_factorial_04.bro + + +Raising Notices +=============== + +While Bro's Logging Framework provides an easy and systematic way to +generate logs, there still exists a need to indicate when a specific +behavior has been detected and a method to allow that detection to +come to someone's attention. To that end, the Notice Framework is in +place to allow script writers a codified means through which they can +raise a notice, as well as a system through which an operator can +opt-in to receive the notice. Bro holds to the philosophy that it is +up to the individual operator to indicate the behaviors in which they +are interested and as such Bro ships with a large number of policy +scripts which detect behavior that may be of interest but it does not +presume to guess as to which behaviors are "action-able". In effect, +Bro works to separate the act of detection and the responsibility of +reporting. With the Notice Framework it's simple to raise a notice +for any behavior that is detected. + +To raise a notice in Bro, you only need to indicate to Bro that you +are provide a specific :bro:id:`Notice::Type` by exporting it and then +make a call to :bro:id:`NOTICE` supplying it with an appropriate +:bro:id:`Notice::Info` record. Often times the call to ``NOTICE`` +includes just the ``Notice::Type``, and a concise message. There are +however, significantly more options available when raising notices as +seen in the table below. The only field in the table below whose +attributes make it a required field is the ``note`` field. Still, +good manners are always important and including a concise message in +``$msg`` and, where necessary, the contents of the connection record +in ``$conn`` along with the ``Notice::Type`` tend to comprise the +minimum of information required for an notice to be considered useful. +If the ``$conn`` variable is supplied the Notice Framework will +auto-populate the ``$id`` and ``$src`` fields as well. Other fields +that are commonly included, ``$identifier`` and ``$suppress_for`` are +built around the automated suppression feature of the Notice Framework +which we will cover shortly. + +.. todo:: + + Once the link to ``Notice::Info`` work I think we should take out + the table. That's too easy to get out of date. + ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| Field | Type | Attributes | Use | ++=====================+==================================================================+================+========================================+ +| ts | time | &log &optional | The time of the notice | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| uid | string | &log &optional | A unique connection ID | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| id | conn_id | &log &optional | A 4-tuple to identify endpoints | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| conn | connection | &optional | Shorthand for the uid and id | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| iconn | icmp_conn | &optional | Shorthand for the uid and id | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| proto | transport_proto | &log &optional | Transport protocol | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| note | Notice::Type | &log | The Notice::Type of the notice | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| msg | string | &log &optional | Human readable message | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| sub | string | &log &optional | Human readable message | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| src | addr | &log &optional | Source address if no conn_id | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| dst | addr | &log &optional | Destination addr if no conn_id | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| p | port | &log &optional | Port if no conn_id | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| n | count | &log &optional | Count or status code | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| src_peer | event_peer | &log &optional | Peer that raised the notice | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| peer_descr | string | &log &optional | Text description of the src_peer | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| actions | set[Notice::Action] | &log &optional | Actions applied to the notice | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| policy_items | set[count] | &log &optional | Policy items that have been applied | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| email_body_sections | vector | &optinal | Body of the email for email notices. | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| email_delay_tokens | set[string] | &optional | Delay functionality for email notices. | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| identifier | string | &optional | A unique string identifier | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ +| suppress_for | interval | &log &optional | Length of time to suppress a notice. | ++---------------------+------------------------------------------------------------------+----------------+----------------------------------------+ + +One of the default policy scripts raises a notice when an SSH login +has been heuristically detected and the originating hostname is one +that would raise suspicion. Effectively, the script attempts to +define a list of hosts from which you would never want to see SSH +traffic originating, like DNS servers, mail servers, etc. To +accomplish this, the script adhere's to the seperation of detection +and reporting by detecting a behavior and raising a notice. Whether +or not that notice is acted upon is decided by the local Notice +Policy, but the script attempts to supply as much information as +possible while staying concise. + +.. btest-include:: ${BRO_SRC_ROOT}/scripts/policy/protocols/ssh/interesting-hostnames.bro + :lines: 1-46 + +While much of the script relates to the actual detection, the parts +specific to the Notice Framework are actually quite interesting in +themselves. On line 12 the script's ``export`` block adds the value +``SSH::Interesting_Hostname_Login`` to the enumerable constant +``Notice::Type`` to indicate to the Bro core that a new type of notice +is being defined. The script then calls ``NOTICE`` and defines the +``$note``, ``$msg``, ``$sub`` and ``$conn`` fields of the +:bro:id:`Notice::Info` record. Line 39 also includes a ternary if +statement that modifies the ``$msg`` text depending on whether the +host is a local address and whether it is the client or the server. +This use of :bro:id:`fmt` and a ternary operators is a concise way to +lend readability to the notices that are generated without the need +for branching ``if`` statements that each raise a specific notice. + +The opt-in system for notices is managed through writing +:bro:id:`Notice::policy` hooks. A ``Notice::policy`` hook takes as +its argument a ``Notice::Info`` record which will hold the same +information your script provided in its call to ``NOTICE``. With +access to the ``Notice::Info`` record for a specific notice you can +include logic such as in statements in the body of your hook to alter +the policy for handling notices on your system. In Bro, hooks are +akin to a mix of functions and event handlers: like functions, calls +to them are synchronous (i.e., run to completion and return); but like +events, they can have multiple bodies which will all execute. For +defining a notice policy, you define a hook and Bro will take care of +passing in the ``Notice::Info`` record. The simplest kind of +``Notice::policy`` hooks simply check the value of ``$note`` in the +``Notice::Info`` record being passed into the hook and performing an +action based on the answer. The hook below adds the +:bro:id:`Notice::ACTION_EMAIL` action for the +``SSH::Interesting_Hostname_Login`` notice raised in the +:doc:`scripts/policy/protocols/ssh/interesting-hostnames.bro` script. + +.. btest-include:: ${DOC_ROOT}/scripting/framework_notice_hook_01.bro + +In the example above we've added ``Notice::ACTION_EMAIL`` to the +``n$actions`` set. This set, defined in the Notice Framework scripts, +can only have entries from the :bro:id:`Notice::Action` type, which is +itself an enumerable that defines the values shown in the table below +along with their corresponding meanings. The +:bro:id:`Notice::ACTION_LOG` action writes the notice to the +``Notice::LOG`` logging stream which, in the default configuration, +will write each notice to the ``notice.log`` file and take no further +action. The :bro:id:`Notice::ACTION_EMAIL` action will send an email +to the address or addresses defined in the :bro:id:`Notice::mail_dest` +variable with the particulars of the notice as the body of the email. +The last action, :bro:id:`Notice::ACTION_ALARM` sends the notice to +the :bro:id:`Notice::ALARM_LOG` logging stream which is then rotated +hourly and its contents emailed in readable ASCII to the addresses in +``Notice::mail_dest``. + ++--------------+-----------------------------------------------------+ +| ACTION_NONE | Take no action | ++--------------+-----------------------------------------------------+ +| ACTION_LOG | Send the notice to the Notice::LOG logging stream. | ++--------------+-----------------------------------------------------+ +| ACTION_EMAIL | Send an email with the notice in the body. | ++--------------+-----------------------------------------------------+ +| ACTION_ALARM | Send the notice to the Notice::Alarm_LOG stream. | ++--------------+-----------------------------------------------------+ + +While actions like the ``Notice::ACTION_EMAIL`` action have appeal for +quick alerts and response, a caveat of its use is to make sure the +notices configured with this action also have a suppression. A +suppression is a means through which notices can be ignored after they +are initially raised if the author of the script has set an +identifier. An identifier is a unique string of information collected +from the connection relative to the behavior that has been observed by +Bro. + +.. btest-include:: ${BRO_SRC_ROOT}/scripts/policy/protocols/ssl/expiring-certs.bro + :lines: 59-62 + +In the :doc:`scripts/policy/protocols/ssl/expiring-certs.bro` script +which identifies when SSL certificates are set to expire and raises +notices when it crosses a pre-defined threshold, the call to +``NOTICE`` above also sets the ``$identifier`` entry by concatenating +the responder IP, port, and the hash of the certificate. The +selection of responder IP, port and certificate hash fits perfectly +into an appropriate identifier as it creates a unique identifier with +which the suppression can be matched. Were we to take out any of the +entities used for the identifier, for example the certificate hash, we +could be setting our suppression too broadly, causing an analyst to +miss a notice that should have been raised. Depending on the +available data for the identifier, it can be useful to set the +``$suppress_for`` variable as well. The ``expiring-certs.bro`` script +sets ``$suppress_for`` to ``1day``, telling the Notice Framework to +suppress the notice for 24 hours after the first notice is raised. +Once that time limit has passed, another notice can be raised which +will again set the ``1day`` suppression time. Suppressing for a +specific amount of time has benefits beyond simply not filling up an +analyst's email inbox; keeping the notice alerts timely and succinct +helps avoid a case where an analyst might see the notice and, due to +over exposure, ignore it. + +The ``$suppress_for`` variable can also be altered in a +``Notice::policy`` hook, allowing a deployment to better suit the +environment in which it is be run. Using the example of +``expiring-certs.bro``, we can write a ``Notice::policy`` hook for +``SSL::Certificate_Expires_Soon`` to configure the ``$suppress_for`` +variable to a shorter time. + +.. btest-include:: ${DOC_ROOT}/scripting/framework_notice_hook_suppression_01.bro + +While ``Notice::policy`` hooks allow you to build custom +predicate-based policies for a deployment, there are bound to be times +where you don't require the full expressiveness that a hook allows. +In short, there will be notice policy considerations where a broad +decision can be made based on the ``Notice::Type`` alone. To +facilitate these types of decisions, the Notice Framework supports +Notice Policy shortcuts. These shortcuts are implemented through the +means of a group of data structures that map specific, pre-defined +details and actions to the effective name of a notice. Primarily +implemented as a set or table of enumerables of :bro:id:`Notice::Type`, +Notice Policy shortcuts can be placed as a single directive in your +``local.bro`` file as a concise readable configuration. As these +variables are all constants, it bears mentioning that these variables +are all set at parse-time before Bro is fully up and running and not +set dynamically. + ++------------------------------------+-----------------------------------------------------+-------------------------------------+ +| Name | Description | Data Type | ++====================================+=====================================================+=====================================+ +| Notice::ignored_types | Ignore the Notice::Type entirely | set[Notice::Type] | ++------------------------------------+-----------------------------------------------------+-------------------------------------+ +| Notice::emailed_types | Set Notice::ACTION_EMAIL to this Notice::Type | set[Notice::Type] | ++------------------------------------+-----------------------------------------------------+-------------------------------------+ +| Notice::alarmed_types | Set Notice::ACTION_ALARM to this Notice::Type | set[Notice::Type] | ++------------------------------------+-----------------------------------------------------+-------------------------------------+ +| Notice::not_suppressed_types | Remove suppression from this Notice::Type | set[Notice::Type] | ++------------------------------------+-----------------------------------------------------+-------------------------------------+ +| Notice::type_suppression_intervals | Alter the $suppress_for value for this Notice::Type | table[Notice::Type] of interval | ++------------------------------------+-----------------------------------------------------+-------------------------------------+ + + + +The table above details the five Notice Policy shortcuts, their +meaning and the data type used to implement them. With the exception +of ``Notice::type_suppression_intervals`` a ``set`` data type is +employed to hold the ``Notice::Type`` of the notice upon which a +shortcut should applied. The first three shortcuts are fairly self +explanatory, applying an action to the ``Notice::Type`` elements in +the set, while the latter two shortcuts alter details of the +suppression being applied to the Notice. The shortcut +``Notice::not_suppressed_types`` can be used to remove the configured +suppression from a notice while ``Notice::type_suppression_intervals`` +can be used to alter the suppression interval defined by $suppress_for +in the call to ``NOTICE``. + +.. btest-include:: ${DOC_ROOT}/scripting/framework_notice_shortcuts_01.bro + +The Notice Policy shortcut above adds the ``Notice::Types`` of +SSH::Interesting_Hostname_Login and SSH::Login to the +Notice::emailed_types set while the shortcut below alters the length +of time for which those notices will be suppressed. + +.. btest-include:: ${DOC_ROOT}/scripting/framework_notice_shortcuts_02.bro + + diff --git a/doc/scripting/using_bro_sandbox_01 b/doc/scripting/using_bro_sandbox_01 new file mode 100644 index 0000000000..4130ea8ceb --- /dev/null +++ b/doc/scripting/using_bro_sandbox_01 @@ -0,0 +1,4 @@ +# @TEST-EXEC: bro -r ${TRACES}/wikipedia.trace +# @TEST-EXEC: btest-diff conn.log +# @TEST-EXEC: btest-diff http.log + diff --git a/doc/scripting/using_bro_sandbox_02 b/doc/scripting/using_bro_sandbox_02 new file mode 100644 index 0000000000..5a21b59800 --- /dev/null +++ b/doc/scripting/using_bro_sandbox_02 @@ -0,0 +1,4 @@ +# @TEST-EXEC: bro -r ${TRACES}/workshop_2011_browse.trace +# @TEST-EXEC: btest-diff conn.log +# @TEST-EXEC: btest-diff http.log + diff --git a/doc/scripts/builtins.rst b/doc/scripts/builtins.rst index 2f5c220458..275e1dcf95 100644 --- a/doc/scripts/builtins.rst +++ b/doc/scripts/builtins.rst @@ -1,55 +1,105 @@ -Builtin Types and Attributes -============================ +Built-in Types and Attributes +============================= Types ----- +Every value in a Bro script has a type (see below for a list of all built-in +types). Although Bro variables have static types (meaning that their type +is fixed), their type is inferred from the value to which they are +initially assigned when the variable is declared without an explicit type +name. + +Automatic conversions happen when a binary operator has operands of +different types. Automatic conversions are limited to converting between +numeric types. The numeric types are ``int``, ``count``, and ``double`` +(``bool`` is not a numeric type). +When an automatic conversion occurs, values are promoted to the "highest" +type in the expression. In general, this promotion follows a simple +hierarchy: ``double`` is highest, ``int`` comes next, and ``count`` is +lowest. + The Bro scripting language supports the following built-in types. .. bro:type:: void - An internal Bro type representing an absence of a type. Should - most often be seen as a possible function return type. + An internal Bro type representing the absence of a return type for a + function. .. bro:type:: bool Reflects a value with one of two meanings: true or false. The two ``bool`` constants are ``T`` and ``F``. + The ``bool`` type supports the following operators: equality/inequality + (``==``, ``!=``), logical and/or (``&&``, ``||``), logical + negation (``!``), and absolute value (where ``|T|`` is 1, and ``|F|`` is 0). + .. bro:type:: int - A numeric type representing a signed integer. An ``int`` constant + A numeric type representing a 64-bit signed integer. An ``int`` constant is a string of digits preceded by a ``+`` or ``-`` sign, e.g. - ``-42`` or ``+5``. When using type inferencing use care so that the + ``-42`` or ``+5`` (the "+" sign is optional but see note about type + inferencing below). An ``int`` constant can also be written in + hexadecimal notation (in which case "0x" must be between the sign and + the hex digits), e.g. ``-0xFF`` or ``+0xabc123``. + + The ``int`` type supports the following operators: arithmetic + operators (``+``, ``-``, ``*``, ``/``, ``%``), comparison operators + (``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``), assignment operators + (``=``, ``+=``, ``-=``), pre-increment (``++``), pre-decrement + (``--``), and absolute value (e.g., ``|-3|`` is 3). + + When using type inferencing use care so that the intended type is inferred, e.g. ``local size_difference = 0`` will infer :bro:type:`count`, while ``local size_difference = +0`` will infer :bro:type:`int`. .. bro:type:: count - A numeric type representing an unsigned integer. A ``count`` - constant is a string of digits, e.g. ``1234`` or ``0``. + A numeric type representing a 64-bit unsigned integer. A ``count`` + constant is a string of digits, e.g. ``1234`` or ``0``. A ``count`` + can also be written in hexadecimal notation (in which case "0x" must + precede the hex digits), e.g. ``0xff`` or ``0xABC123``. + + The ``count`` type supports the same operators as the :bro:type:`int` + type. A unary plus or minus applied to a ``count`` results in an ``int``. .. bro:type:: counter An alias to :bro:type:`count`. -.. TODO: is there anything special about this type? - .. bro:type:: double A numeric type representing a double-precision floating-point number. Floating-point constants are written as a string of digits with an optional decimal point, optional scale-factor in scientific notation, and optional ``+`` or ``-`` sign. Examples are ``-1234``, - ``-1234e0``, ``3.14159``, and ``.003e-23``. + ``-1234e0``, ``3.14159``, and ``.003E-23``. + + The ``double`` type supports the following operators: arithmetic + operators (``+``, ``-``, ``*``, ``/``), comparison operators + (``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``), assignment operators + (``=``, ``+=``, ``-=``), and absolute value (e.g., ``|-3.14|`` is 3.14). + + When using type inferencing use care so that the + intended type is inferred, e.g. ``local size_difference = 5`` will + infer :bro:type:`count`, while ``local size_difference = 5.0`` + will infer :bro:type:`double`. .. bro:type:: time A temporal type representing an absolute time. There is currently no way to specify a ``time`` constant, but one can use the - :bro:id:`current_time` or :bro:id:`network_time` built-in functions - to assign a value to a ``time``-typed variable. + :bro:id:`double_to_time`, :bro:id:`current_time`, or :bro:id:`network_time` + built-in functions to assign a value to a ``time``-typed variable. + + Time values support the comparison operators (``==``, ``!=``, ``<``, + ``<=``, ``>``, ``>=``). A ``time`` value can be subtracted from + another ``time`` value to produce an ``interval`` value. An ``interval`` + value can be added to, or subtracted from, a ``time`` value to produce a + ``time`` value. The absolute value of a ``time`` value is a ``double`` + with the same numeric value. .. bro:type:: interval @@ -61,16 +111,31 @@ The Bro scripting language supports the following built-in types. constant and time unit is optional. Appending the letter "s" to the time unit in order to pluralize it is also optional (to no semantic effect). Examples of ``interval`` constants are ``3.5 min`` and - ``3.5mins``. An ``interval`` can also be negated, for example ``- - 12 hr`` represents "twelve hours in the past". Intervals also - support addition, subtraction, multiplication, division, and - comparison operations. + ``3.5mins``. An ``interval`` can also be negated, for example + ``-12 hr`` represents "twelve hours in the past". + + Intervals support addition and subtraction. Intervals also support + division (in which case the result is a ``double`` value), the + comparison operators (``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``), + and the assignment operators (``=``, ``+=``, ``-=``). Also, an + ``interval`` can be multiplied or divided by an arithmetic type + (``count``, ``int``, or ``double``) to produce an ``interval`` value. + The absolute value of an ``interval`` is a ``double`` value equal to the + number of seconds in the ``interval`` (e.g., ``|-1 min|`` is 60). .. bro:type:: string A type used to hold character-string values which represent text. String constants are created by enclosing text in double quotes (") - and the backslash character (\\) introduces escape sequences. + and the backslash character (\\) introduces escape sequences (all of + the C-style escape sequences are supported). + + Strings support concatenation (``+``), and assignment (``=``, ``+=``). + Strings also support the comparison operators (``==``, ``!=``, ``<``, + ``<=``, ``>``, ``>=``). Substring searching can be performed using + the "in" or "!in" operators (e.g., "bar" in "foobar" yields true). + The number of characters in a string can be found by enclosing the + string within pipe characters (e.g., ``|"abc"|`` is 3). Note that Bro represents strings internally as a count and vector of bytes rather than a NUL-terminated byte string (although string @@ -127,9 +192,7 @@ The Bro scripting language supports the following built-in types. .. bro:type:: enum A type allowing the specification of a set of related values that - have no further structure. The only operations allowed on - enumerations are equality comparisons and they do not have - associated values or ordering. An example declaration: + have no further structure. An example declaration: .. code:: bro @@ -137,9 +200,9 @@ The Bro scripting language supports the following built-in types. The last comma after ``Blue`` is optional. -.. bro:type:: timer - -.. TODO: is this a type that's exposed to users? + The only operations allowed on enumerations are equality comparisons + (``==``, ``!=``) and assignment (``=``). + Enumerations do not have associated values or ordering. .. bro:type:: port @@ -149,10 +212,15 @@ The Bro scripting language supports the following built-in types. message code. A ``port`` constant is written as an unsigned integer followed by one of ``/tcp``, ``/udp``, ``/icmp``, or ``/unknown``. - Ports can be compared for equality and also for ordering. When - comparing order across transport-level protocols, ``unknown`` < - ``tcp`` < ``udp`` < ``icmp``, for example ``65535/tcp`` is smaller - than ``0/udp``. + Ports support the comparison operators (``==``, ``!=``, ``<``, ``<=``, + ``>``, ``>=``). When comparing order across transport-level protocols, + ``unknown`` < ``tcp`` < ``udp`` < ``icmp``, for example ``65535/tcp`` + is smaller than ``0/udp``. + + Note that you can obtain the transport-level protocol type of a ``port`` + with the :bro:id:`get_port_transport_proto` built-in function, and + the numeric value of a ``port`` with the :bro:id:`port_to_count` + built-in function. .. bro:type:: addr @@ -162,22 +230,29 @@ The Bro scripting language supports the following built-in types. ``A1.A2.A3.A4``, where Ai all lie between 0 and 255. IPv6 address constants are written as colon-separated hexadecimal form - as described by :rfc:`2373`, but additionally encased in square brackets. - The mixed notation with embedded IPv4 addresses as dotted-quads in the - lower 32 bits is also allowed. - Some examples: ``[2001:db8::1]``, ``[::ffff:192.168.1.100]``, or + as described by :rfc:`2373` (including the mixed notation with embedded + IPv4 addresses as dotted-quads in the lower 32 bits), but additionally + encased in square brackets. Some examples: ``[2001:db8::1]``, + ``[::ffff:192.168.1.100]``, or ``[aaaa:bbbb:cccc:dddd:eeee:ffff:1111:2222]``. + Note that IPv4-mapped IPv6 addresses (i.e., addresses with the first 80 + bits zero, the next 16 bits one, and the remaining 32 bits are the IPv4 + address) are treated internally as IPv4 addresses (for example, + ``[::ffff:192.168.1.100]`` is equal to ``192.168.1.100``). + Hostname constants can also be used, but since a hostname can - correspond to multiple IP addresses, the type of such variable is a + correspond to multiple IP addresses, the type of such a variable is a :bro:type:`set` of :bro:type:`addr` elements. For example: .. code:: bro local a = www.google.com; - Addresses can be compared for (in)equality using ``==`` and ``!=``. - They can also be masked with ``/`` to produce a :bro:type:`subnet`: + Addresses can be compared for equality (``==``, ``!=``), + and also for ordering (``<``, ``<=``, ``>``, ``>=``). The absolute value + of an address gives the size in bits (32 for IPv4, and 128 for IPv6). + Addresses can also be masked with ``/`` to produce a :bro:type:`subnet`: .. code:: bro @@ -186,7 +261,8 @@ The Bro scripting language supports the following built-in types. if ( a/16 == s ) print "true"; - And checked for inclusion within a :bro:type:`subnet` using ``in`` : + And checked for inclusion within a :bro:type:`subnet` using ``in`` + or ``!in``: .. code:: bro @@ -195,6 +271,9 @@ The Bro scripting language supports the following built-in types. if ( a in s ) print "true"; + Note that you can check if a given ``addr`` is IPv4 or IPv6 using + the :bro:id:`is_v4_addr` and :bro:id:`is_v6_addr` built-in functions. + .. bro:type:: subnet A type representing a block of IP addresses in CIDR notation. A @@ -202,6 +281,10 @@ The Bro scripting language supports the following built-in types. slash (/) and then the network prefix size specified as a decimal number. For example, ``192.168.0.0/16`` or ``[fe80::]/64``. + Subnets can be compared for equality (``==``, ``!=``). An + :bro:type:`addr` can be checked for inclusion in a subnet using + the "in" or "!in" operators. + .. bro:type:: any Used to bypass strong typing. For example, a function can take an @@ -271,14 +354,14 @@ The Bro scripting language supports the following built-in types. global t3 = MyTable([[$b=5]] = "b5", [[$b=7]] = "b7"); - Accessing table elements if provided by enclosing values within square - brackets (``[]``), for example: + Accessing table elements is provided by enclosing index values within + square brackets (``[]``), for example: .. code:: bro - t[13] = "thirteen"; + print t[11]; - And membership can be tested with ``in``: + And membership can be tested with ``in`` or ``!in``: .. code:: bro @@ -297,17 +380,23 @@ The Bro scripting language supports the following built-in types. for ( [a, p] in services ) ... + Add or overwrite individual table elements by assignment: + + .. code:: bro + + t[13] = "thirteen"; + Remove individual table elements with ``delete``: .. code:: bro delete t[13]; - Nothing happens if the element with value ``13`` isn't present in + Nothing happens if the element with index value ``13`` isn't present in the table. - Table size can be obtained by placing the table identifier between - vertical pipe (|) characters: + The number of elements in a table can be obtained by placing the table + identifier between vertical pipe characters: .. code:: bro @@ -355,27 +444,44 @@ The Bro scripting language supports the following built-in types. global s4 = MySet([$b=1], [$b=2]); - Set membership is tested with ``in``: + Set membership is tested with ``in`` or ``!in``: .. code:: bro if ( 21/tcp in s ) ... + if ( 21/tcp !in s ) + ... + + Iterate over a set with a ``for`` loop: + + .. code:: bro + + local s: set[port]; + for ( p in s ) + ... + Elements are added with ``add``: .. code:: bro add s[22/tcp]; + Nothing happens if the element with value ``22/tcp`` was already present in + the set. + And removed with ``delete``: .. code:: bro delete s[21/tcp]; - Set size can be obtained by placing the set identifier between - vertical pipe (|) characters: + Nothing happens if the element with value ``21/tcp`` isn't present in + the set. + + The number of elements in a set can be obtained by placing the set + identifier between vertical pipe characters: .. code:: bro @@ -384,7 +490,8 @@ The Bro scripting language supports the following built-in types. .. bro:type:: vector A vector is like a :bro:type:`table`, except it's always indexed by a - :bro:type:`count`. A vector is declared like: + :bro:type:`count` (and vector indexing is always zero-based). A vector + is declared like: .. code:: bro @@ -411,21 +518,51 @@ The Bro scripting language supports the following built-in types. global v2 = MyVec([$b=1], [$b=2], [$b=3]); - Adding an element to a vector involves accessing/assigning it: + Accessing vector elements is provided by enclosing index values within + square brackets (``[]``), for example: .. code:: bro - v[3] = "four" + print v[2]; - Note how the vector indexing is 0-based. + Iterate over a vector with a ``for`` loop: - Vector size can be obtained by placing the vector identifier between - vertical pipe (|) characters: + .. code:: bro + + local v: vector of string; + for ( n in v ) + ... + + An element can be added to a vector by assigning the value (a value + that already exists at that index will be overwritten): + + .. code:: bro + + v[3] = "four"; + + The number of elements in a vector can be obtained by placing the vector + identifier between vertical pipe characters: .. code:: bro |v| + Vectors of integral types (``int`` or ``count``) support the pre-increment + (``++``) and pre-decrement operators (``--``), which will increment or + decrement each element in the vector. + + Vectors of arithmetic types (``int``, ``count``, or ``double``) can be + operands of the arithmetic operators (``+``, ``-``, ``*``, ``/``, ``%``), + but both operands must have the same number of elements (and the modulus + operator ``%`` cannot be used if either operand is a ``vector of double``). + The resulting vector contains the result of the operation applied to each + of the elements in the operand vectors. + + Vectors of bool can be operands of the logical "and" (``&&``) and logical + "or" (``||``) operators (both operands must have same number of elements). + The resulting vector of bool is the logical "and" (or logical "or") of + each element of the operand vectors. + .. bro:type:: record A ``record`` is a collection of values. Each value has a field name @@ -504,9 +641,11 @@ The Bro scripting language supports the following built-in types. .. bro:type:: file - Bro supports writing to files, but not reading from them. For - example, declare, open, and write to a file and finally close it - like: + Bro supports writing to files, but not reading from them. Files + can be opened using either the :bro:id:`open` or :bro:id:`open_for_append` + built-in functions, and closed using the :bro:id:`close` built-in + function. For example, declare, open, and write to a file + and finally close it like: .. code:: bro @@ -544,8 +683,8 @@ The Bro scripting language supports the following built-in types. Note that in the definition above, it's not necessary for us to have done the first (forward) declaration of ``greeting`` as a function - type, but when it is, the argument list and return type much match - exactly. + type, but when it is, the return type and argument list (including the + name of each argument) must match exactly. Function types don't need to have a name and can be assigned anonymously: @@ -744,11 +883,18 @@ scripting language supports the following built-in attributes. .. bro:attr:: &add_func -.. TODO: needs to be documented. + Can be applied to an identifier with &redef to specify a function to + be called any time a "redef += ..." declaration is parsed. The + function takes two arguments of the same type as the identifier, the first + being the old value of the variable and the second being the new + value given after the "+=" operator in the "redef" declaration. The + return value of the function will be the actual new value of the + variable after the "redef" declaration is parsed. .. bro:attr:: &delete_func -.. TODO: needs to be documented. + Same as &add_func, except for "redef" declarations that use the "-=" + operator. .. bro:attr:: &expire_func @@ -825,5 +971,15 @@ scripting language supports the following built-in attributes. .. bro:attr:: &error_handler -.. TODO: needs documented + Internally set on the events that are associated with the reporter + framework: :bro:id:`reporter_info`, :bro:id:`reporter_warning`, and + :bro:id:`reporter_error`. It prevents any handlers of those events + from being able to generate reporter messages that go through any of + those events (i.e., it prevents an infinite event recursion). Instead, + such nested reporter messages are output to stderr. +.. bro:attr:: &type_column + + Used by the input framework. It can be used on columns of type + :bro:type:`port` and specifies the name of an additional column in + the input file which specifies the protocol of the port (tcp/udp/icmp). diff --git a/doc/scripts/index.rst b/doc/scripts/index.rst index bf0fa25f10..e6d9bd6d65 100644 --- a/doc/scripts/index.rst +++ b/doc/scripts/index.rst @@ -1,8 +1,21 @@ .. This is a stub doc to which broxygen appends during the build process -Index of All Individual Bro Scripts -=================================== +================ +Script Reference +================ .. toctree:: :maxdepth: 1 + packages + builtins + bifs + scripts + packages + internal + site/proto-analyzers + site/file-analyzers + + + + diff --git a/doc/scripts/packages.rst b/doc/scripts/packages.rst index 56909ffbc6..9f01a91e91 100644 --- a/doc/scripts/packages.rst +++ b/doc/scripts/packages.rst @@ -1,7 +1,9 @@ .. This is a stub doc to which broxygen appends during the build process -Index of All Bro Script Packages -================================ +.. _script-packages: + +Bro Script Packages +=================== Bro has the following script packages (e.g. collections of related scripts in a common directory). If the package directory contains a ``__load__.bro`` diff --git a/doc/scripts/scripts.rst b/doc/scripts/scripts.rst new file mode 100644 index 0000000000..d454063002 --- /dev/null +++ b/doc/scripts/scripts.rst @@ -0,0 +1,8 @@ +.. This is a stub doc to which broxygen appends during the build process + +======================== +Index of All Bro Scripts +======================== + +.. toctree:: + :maxdepth: 1 diff --git a/doc/upgrade.rst b/doc/upgrade.rst deleted file mode 100644 index 539757537d..0000000000 --- a/doc/upgrade.rst +++ /dev/null @@ -1,308 +0,0 @@ - -========================================== -Upgrading From the Previous Version of Bro -========================================== - -.. rst-class:: opening - - This guide details specific differences between Bro versions - that may be important for users to know as they work on updating - their Bro deployment/configuration to the later version. - -.. contents:: - - -Upgrading From Bro 2.0 to 2.1 -============================= - -In Bro 2.1, IPv6 is enabled by default. Therefore, when building Bro from -source, the "--enable-brov6" configure option has been removed because it -is no longer relevant. - -Other configure changes include renaming the "--enable-perftools" option -to "--enable-perftools-debug" to indicate that the option is only relevant -for debugging the heap. One other change involves what happens when -tcmalloc (part of Google perftools) is found at configure time. On Linux, -it will automatically be linked with Bro, but on other platforms you -need to use the "--enable-perftools" option to enable linking to tcmalloc. - -There are a couple of changes to the Bro scripting language to better -support IPv6. First, IPv6 literals appearing in a Bro script must now be -enclosed in square brackets (for example, ``[fe80::db15]``). For subnet -literals, the slash "/" appears after the closing square bracket (for -example, ``[fe80:1234::]/32``). Second, when an IP address variable or IP -address literal is enclosed in pipes (for example, ``|[fe80::db15]|``) the -result is now the size of the address in bits (32 for IPv4 and 128 for IPv6). - -In the Bro scripting language, "match" and "using" are no longer reserved -keywords. - -Some built-in functions have been removed: "addr_to_count" (use -"addr_to_counts" instead), "bro_has_ipv6" (this is no longer relevant -because Bro now always supports IPv6), "active_connection" (use -"connection_exists" instead), and "connection_record" (use "lookup_connection" -instead). - -The "NFS3::mode2string" built-in function has been renamed to "file_mode". - -Some built-in functions have been changed: "exit" (now takes the exit code -as a parameter), "to_port" (now takes a string as parameter instead -of a count and transport protocol, but "count_to_port" is still available), -"connect" (now takes an additional string parameter specifying the zone of -a non-global IPv6 address), and "listen" (now takes three additional -parameters to enable listening on IPv6 addresses). - -Some Bro script variables have been renamed: "LogAscii::header_prefix" -has been renamed to "LogAscii::meta_prefix", "LogAscii::include_header" -has been renamed to "LogAscii::include_meta". - -Some Bro script variables have been removed: "tunnel_port", -"parse_udp_tunnels", "use_connection_compressor", "cc_handle_resets", -"cc_handle_only_syns", and "cc_instantiate_on_data". - -A couple events have changed: the "icmp_redirect" event now includes -the target and destination addresses and any Neighbor Discovery options -in the message, and the last parameter of the "dns_AAAA_reply" event has -been removed because it was unused. - -The format of the ASCII log files has changed very slightly. Two new lines -are automatically added, one to record the time when the log was opened, -and the other to record the time when the log was closed. - -In BroControl, the option (in broctl.cfg) "CFlowAddr" was renamed -to "CFlowAddress". - - -Upgrading From Bro 1.5 to 2.0 -============================= - -As the version number jump suggests, Bro 2.0 is a major upgrade and -lots of things have changed. Most importantly, we have rewritten -almost all of Bro's default scripts from scratch, using quite -different structure now and focusing more on operational deployment. -The result is a system that works much better "out of the box", even -without much initial site-specific configuration. The down-side is -that 1.x configurations will need to be adapted to work with the new -version. The two rules of thumb are: - - (1) If you have written your own Bro scripts - that do not depend on any of the standard scripts formerly - found in ``policy/``, they will most likely just keep working - (although you might want to adapt them to use some of the new - features, like the new logging framework; see below). - - (2) If you have custom code that depends on specifics of 1.x - default scripts (including most configuration tuning), that is - unlikely to work with 2.x. We recommend to start by using just - the new scripts first, and then port over any customizations - incrementally as necessary (they may be much easier to do now, - or even unnecessary). Send mail to the Bro user mailing list - if you need help. - -Below we summarize changes from 1.x to 2.x in more detail. This list -isn't complete, see the :download:`CHANGES ` file in the -distribution for the full story. - -Default Scripts -=============== - -Organization ------------- - -In versions before 2.0, Bro scripts were all maintained in a flat -directory called ``policy/`` in the source tree. This directory is now -renamed to ``scripts/`` and contains major subdirectories ``base/``, -``policy/``, and ``site/``, each of which may also be subdivided -further. - -The contents of the new ``scripts/`` directory, like the old/flat -``policy/`` still gets installed under the ``share/bro`` -subdirectory of the installation prefix path just like previous -versions. For example, if Bro was compiled like ``./configure ---prefix=/usr/local/bro && make && make install``, then the script -hierarchy can be found in ``/usr/local/bro/share/bro``. - -The main -subdirectories of that hierarchy are as follows: - -- ``base/`` contains all scripts that are loaded by Bro by default - (unless the ``-b`` command line option is used to run Bro in a - minimal configuration). Note that is a major conceptual change: - rather than not loading anything by default, Bro now uses an - extensive set of default scripts out of the box. - - The scripts under this directory generally either accumulate/log - useful state/protocol information for monitored traffic, configure a - default/recommended mode of operation, or provide extra Bro - scripting-layer functionality that has no significant performance cost. - -- ``policy/`` contains all scripts that a user will need to explicitly - tell Bro to load. These are scripts that implement - functionality/analysis that not all users may want to use and may have - more significant performance costs. For a new installation, you - should go through these and see what appears useful to load. - -- ``site/`` remains a directory that can be used to store locally - developed scripts. It now comes with some preinstalled example - scripts that contain recommended default configurations going beyond - the ``base/`` setup. E.g. ``local.bro`` loads extra scripts from - ``policy/`` and does extra tuning. These files can be customized in - place without being overwritten by upgrades/reinstalls, unlike - scripts in other directories. - -With version 2.0, the default ``BROPATH`` is set to automatically -search for scripts in ``policy/``, ``site/`` and their parent -directory, but **not** ``base/``. Generally, everything under -``base/`` is loaded automatically, but for users of the ``-b`` option, -it's important to know that loading a script in that directory -requires the extra ``base/`` path qualification. For example, the -following two scripts: - -* ``$PREFIX/share/bro/base/protocols/ssl/main.bro`` -* ``$PREFIX/share/bro/policy/protocols/ssl/validate-certs.bro`` - -are referenced from another Bro script like: - -.. code:: bro - - @load base/protocols/ssl/main - @load protocols/ssl/validate-certs - -Notice how ``policy/`` can be omitted as a convenience in the second -case. ``@load`` can now also use relative path, e.g., ``@load -../main``. - - -Logging Framework ------------------ - -- The logs generated by scripts that ship with Bro are entirely redone - to use a standardized, machine parsable format via the new logging - framework. Generally, the log content has been restructured towards - making it more directly useful to operations. Also, several - analyzers have been significantly extended and thus now log more - information. Take a look at ``ssl.log``. - - * A particular format change that may be useful to note is that the - ``conn.log`` ``service`` field is derived from DPD instead of - well-known ports (while that was already possible in 1.5, it was - not the default). - - * Also, ``conn.log`` now reports raw number of packets/bytes per - endpoint. - -- The new logging framework makes it possible to extend, customize, - and filter logs very easily. See the :doc:`logging framework ` - for more information on usage. - -- A common pattern found in the new scripts is to store logging stream - records for protocols inside the ``connection`` records so that - state can be collected until enough is seen to log a coherent unit - of information regarding the activity of that connection. This - state is now frequently seen/accessible in event handlers, for - example, like ``c$`` where ```` is replaced by - the name of the protocol. This field is added to the ``connection`` - record by ``redef``'ing it in a - ``base/protocols//main.bro`` script. - -- The logging code has been rewritten internally, with script-level - interface and output backend now clearly separated. While ASCII - logging is still the default, we will add further output types in - the future (binary format, direct database logging). - - -Notice Framework ----------------- - -The way users interact with "notices" has changed significantly in -order to make it easier to define a site policy and more extensible -for adding customized actions. See the :doc:`notice framework `. - - -New Default Settings --------------------- - -- Dynamic Protocol Detection (DPD) is now enabled/loaded by default. - -- The default packet filter now examines all packets instead of - dynamically building a filter based on which protocol analysis scripts - are loaded. See ``PacketFilter::all_packets`` for how to revert to old - behavior. - -API Changes ------------ - -- The ``@prefixes`` directive works differently now. - Any added prefixes are now searched for and loaded *after* all input - files have been parsed. After all input files are parsed, Bro - searches ``BROPATH`` for prefixed, flattened versions of all of the - parsed input files. For example, if ``lcl`` is in ``@prefixes``, and - ``site.bro`` is loaded, then a file named ``lcl.site.bro`` that's in - ``BROPATH`` would end up being automatically loaded as well. Packages - work similarly, e.g. loading ``protocols/http`` means a file named - ``lcl.protocols.http.bro`` in ``BROPATH`` gets loaded automatically. - -- The ``make_addr`` BIF now returns a ``subnet`` versus an ``addr`` - - -Variable Naming ---------------- - -- ``Module`` is more widely used for namespacing. E.g. the new - ``site.bro`` exports the ``local_nets`` identifier (among other - things) into the ``Site`` module. - -- Identifiers may have been renamed to conform to new `scripting - conventions - `_ - - -BroControl -========== - -BroControl looks pretty much similar to the version coming with Bro 1.x, -but has been cleaned up and streamlined significantly internally. - -BroControl has a new ``process`` command to process a trace on disk -offline using a similar configuration to what BroControl installs for -live analysis. - -BroControl now has an extensive plugin interface for adding new -commands and options. Note that this is still considered experimental. - -We have removed the ``analysis`` command, and BroControl currently -does not send daily alarm summaries anymore (this may be restored -later). - -Removed Functionality -===================== - -We have remove a bunch of functionality that was rarely used and/or -had not been maintained for a while already: - - - The ``net`` script data type. - - The ``alarm`` statement; use the notice framework instead. - - Trace rewriting. - - DFA state expiration in regexp engine. - - Active mapping. - - Native DAG support (may come back eventually) - - ClamAV support. - - The connection compressor is now disabled by default, and will - be removed in the future. - -Development Infrastructure -========================== - -Bro development has moved from using SVN to Git for revision control. -Users that want to use the latest Bro development snapshot by checking it out -from the source repositories should see the `development process -`_. Note that all the various -sub-components now reside in their own repositories. However, the -top-level Bro repository includes them as git submodules so it's easy -to check them all out simultaneously. - -Bro now uses `CMake `_ for its build system so -that is a new required dependency when building from source. - -Bro now comes with a growing suite of regression tests in -``testing/``. diff --git a/doc/using/index.rst b/doc/using/index.rst new file mode 100644 index 0000000000..1ad05d74f8 --- /dev/null +++ b/doc/using/index.rst @@ -0,0 +1,253 @@ + +.. _using-bro: + +========= +Using Bro +========= + +.. contents:: + +Once Bro has been deployed in an environment and monitoring live +traffic, it will, in its default configuration, begin to produce +human-readable ASCII logs. Each log file, produced by Bro's +:ref:`framework-logging`, is populated with organized, mostly +connection-oriented data. As the standard log files are simple ASCII +data, working with the data contained in them can be done from a +command line terminal once you have been familiarized with the types +of data that can be found in each file. In the following, we work +through the logs general structure and then examine some standard ways +of working with them. + +---------------------- +Working with Log Files +---------------------- + +Generally, all of Bro's log files are produced by a corresponding +script that defines their individual structure. However, as each log +file flows through the Logging Framework, there share a set of +structural similarities. Without breaking into the scripting aspect of +Bro here, a bird's eye view of how the log files are produced would +progress as follows. The script's author defines the kinds of data, +such as the originating IP address or the duration of a connection, +which will make up the fields (i.e., columns) of the log file. The +author then decides what network activity should generate a single log +file entry (i.e., one line); that could, e.g., be a connection having +been completed or an HTTP ``GET`` method being issued by an +originator. When these behaviors are observed during operation, the +data is passed to the Logging Framework which, in turn, adds the entry +to the appropriate log file. + +As the fields of the log entries can be further customized by the +user, the Logging Framework makes use of a header block to ensure that +it remains self-describing. This header entry can be see by running +the Unix utility ``head`` and outputting the first lines of the file: + +.. btest:: using_bro + + @TEST-EXEC: btest-rst-cmd bro -r $TRACES/wikipedia.trace + @TEST-EXEC: btest-rst-include -n 15 conn.log + +As you can see, the header consists of lines prefixed by ``#`` and +includes information such as what separators are being used for +various types of data, what an empty field looks like and what an +unset field looks like. In this example, the default TAB separator is +being used as the delimiter between fields (``\x09`` is the tab +character in hex). It also lists the comma as the separator for set +data, the string ``(empty)`` as the indicator for an empty field and +the ``-`` character as the indicator for a field that hasn't been set. +The timestamp for when the file was created is included under +``#open``. The header then goes on to detail the fields being listed +in the file and the data types of those fields in ``#fields`` and +``#types``, respectively. These two entries are often the two most +significant points of interest as they detail not only the field names +but the data types used. When navigating through the different log +files with tools like ``sed``, ``awk``, or ``grep``, having the field +definitions readily available saves the user some mental leg work. The +field names are also a key resource for using the :ref:`bro-cut +` utility included with Bro, see below. + +Next to the header follows the main content; in this example we see 7 +connections with their key properties, such as originator and +responder IP addresses (note how Bro transparely handles both IPv4 and +IPv6), transport-layer ports, application-layer services - the +``service`` field is filled ias Bro determines a specific protocol to +be in use, independent of the connection's ports - payload size, and +more. See :bro:type:`Conn::Info` for a description of all fields. + +In addition to ``conn.log``, Bro generates many further logs by +default, including: + +``dpd.log`` + A summary of protocols encountered on non-standard ports. + +``dns.log`` + All DNS activity. + +``ftp.log`` + A log of FTP session-level activity. + +``files.log`` + Summaries of files transfered over the network. This information + is aggregrated from different protocols, including HTTP, FTP, and + SMTP. + +``http.log`` + A summary of all HTTP requests with their replies. + +``known_certs.log`` + SSL certificates seen in use. + +``smtp.log`` + A summary of SMTP activity. + +``ssl.log`` + A record of SSL sessions, including certificates being used. + +``weird.log`` + A log of unexpected protocol-level activity. Whenever Bro's + protocol analysis encounters a situation it would not expect + (e.g., an RFC violation) is logs it in this file. Note that in + practice, real-world networks tend to exhibit a large number of + such "crud" that is usually not worth following up on. + +As you can see, some log files are specific to a particular protocol, +while others aggregate information across different types of activity. + +.. _bro-cut: + +Using ``bro-cut`` +----------------- + +The ``bro-cut`` utility can be used in place of other tools to build +terminal commands that remain flexible and accurate independent of +possible changes to log file itself. It accomplishes this by parsing +the header in each file and allowing the user to refer to the specific +columnar data available (in contrast to tools like ``awk`` that +require the user to refer to fields referenced by their position). +For example, the following command extracts just the given columns +from a ``conn.log``: + +.. btest:: using_bro + + @TEST-EXEC: btest-rst-cmd -n 10 "cat conn.log | bro-cut id.orig_h id.orig_p id.resp_h duration" + +The correspding ``awk`` command would look like this: + +.. btest:: using_bro + + @TEST-EXEC: btest-rst-cmd -n 10 awk \'/^[^#]/ {print \$3, \$4, \$5, \$6, \$9}\' conn.log + +While the output is similar, the advantages to using bro-cut over +``awk`` lay in that, while ``awk`` is flexible and powerful, ``bro-cut`` +was specifically designed to work with Bro's log files. Firstly, the +``bro-cut`` output includes only the log file entries, while the +``awk`` solution needs to skip the header manually. Secondly, since +``bro-cut`` uses the field descriptors to identify and extract data, +it allows for flexibility independent of the format and contents of +the log file. It's not uncommon for a Bro configuration to add extra +fields to various log files as required by the environment. In this +case, the fields in the ``awk`` command would have to be altered to +compensate for the new position whereas the ``bro-cut`` output would +not change. + +.. note:: + + The sequence of field names given to ``bro-cut`` determines the + output order, which means you can also use ``bro-cut`` to reorder + fields. That can be helpful when piping into, e.g., ``sort``. + +As you may have noticed, the command for ``bro-cut`` uses the output +redirection through the ``cat`` command and ``|`` operator. Whereas +tools like ``awk`` allow you to indicate the log file as a command +line option, bro-cut only takes input through redirection such as +``|`` and ``<``. There are a couple of ways to direct log file data +into ``bro-cut``, each dependent upon the type of log file you're +processing. A caveat of its use, however, is that the 8 lines of +header data must be present. + +.. note:: + + ``bro-cut`` provides an option ``-c`` to include a corresponding + format header into the output, which allows to chain multiple + ``bro-cut`` instances or perform further post-processing that + evaluates the header information. + +In its default setup, Bro will rotate log files on an hourly basis, +moving the current log file into a directory with format +``YYYY-MM-DD`` and gzip compressing the file with a file format that +includes the log file type and time range of the file. In the case of +processing a compressed log file you simply adjust your command line +tools to use the complementary ``z*`` versions of commands such as cat +(``zcat``), ``grep`` (``zgrep``), and ``head`` (``zhead``). + +Working with Timestamps +----------------------- + +``bro-cut`` accepts the flag ``-d`` to convert the epoch time values +in the log files to human-readable format. The following command +includes the human readable time stamp, the unique identifier and the +HTTP ``Host`` and HTTP ``URI`` as extracted from the ``http.log`` +file: + +.. btest:: using_bro + + @TEST-EXEC: btest-rst-cmd -n 5 "bro-cut -d ts uid host uri < http.log" + +Often times log files from multiple sources are stored in UTC time to +allow easy correlation. Converting the timestamp from a log file to +UTC can be accomplished with the ``-u`` option: + +.. btest:: using_bro + + @TEST-EXEC: btest-rst-cmd -n 5 "bro-cut -u ts uid host uri < http.log" + +The default time format when using the ``-d`` or ``-u`` is the +``strftime`` format string ``%Y-%m-%dT%H:%M:%S%z`` which results in a +string with year, month, day of month, followed by hour, minutes, +seconds and the timezone offset. The default format can be altered by +using the ``-D`` and ``-U`` flags, using the standard ``strftime`` +syntax. For example, to format the timestamp in the US-typical "Middle +Endian" you could use a format string of: ``%d-%m-%YT%H:%M:%S%z`` + +.. btest:: using_bro + + @TEST-EXEC: btest-rst-cmd -n 5 "bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < http.log" + +See ``man strfime`` for more options for the format string. + +Using UIDs +---------- + +While Bro can do signature based analysis, its primary focus is on +behavioral detection which alters the practice of log review from +"reactionary review" to a process a little more akin to a hunting +trip. A common progression of review includes correlating a session +across multiple log files. As a connection is processed by Bro, a +unique identifier is assigned to each session. This unique identifier +is generally included in any log file entry associated with that +connection and can be used to cross-reference different log files. + +A simple example would be to cross-reference a UID seen in a +``conn.log`` file. Here, we're looking for the connection with the +largest number of bytes from the responder by redirecting the output +for ``cat conn.log`` into bro-cut to extract the UID and the +resp_bytes, then sorting that output by the resp_bytes field. + +.. btest:: using_bro + + @TEST-EXEC: btest-rst-cmd "cat conn.log | bro-cut uid resp_bytes | sort -nrk2 | head -5" + +Taking the UID of the first of the top responses, we can now +crossreference that with the UIDs in the ``http.log`` file. + +.. btest:: using_bro + + @TEST-EXEC: btest-rst-cmd "cat http.log | bro-cut uid id.resp_h method status_code host uri | grep VW0XPVINV8a" + +As you can see there are two HTTP ``GET`` requests within the +session that Bro identified and logged. Given that HTTP is a stream +protocol, it can have multiple ``GET``/``POST``/etc requests in a +stream and Bro is able to extract and track that information for you, +giving you an in-depth and structured view into HTTP traffic on your +network. + diff --git a/testing/btest/Baseline/doc.manual.connection_record_01/.stdout b/testing/btest/Baseline/doc.manual.connection_record_01/.stdout new file mode 100644 index 0000000000..7f134460e3 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.connection_record_01/.stdout @@ -0,0 +1,5 @@ +[id=[orig_h=212.180.42.100, orig_p=25000/tcp, resp_h=131.243.64.3, resp_p=53/tcp], orig=[size=29, state=5, num_pkts=6, num_bytes_ip=273, flow_label=0], resp=[size=44, state=5, num_pkts=5, num_bytes_ip=248, flow_label=0], start_time=930613226.067666, duration=0.709643, service={ + +}, addl=, hot=0, history=ShADadFf, uid=UWkUyAuUGXf, tunnel=, conn=[ts=930613226.067666, uid=UWkUyAuUGXf, id=[orig_h=212.180.42.100, orig_p=25000/tcp, resp_h=131.243.64.3, resp_p=53/tcp], proto=tcp, service=, duration=0.709643, orig_bytes=29, resp_bytes=44, conn_state=SF, local_orig=, missed_bytes=0, history=ShADadFf, orig_pkts=6, orig_ip_bytes=273, resp_pkts=5, resp_ip_bytes=248, tunnel_parents={ + +}], extract_orig=F, extract_resp=F] diff --git a/testing/btest/Baseline/doc.manual.connection_record_02/.stdout b/testing/btest/Baseline/doc.manual.connection_record_02/.stdout new file mode 100644 index 0000000000..824dd03097 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.connection_record_02/.stdout @@ -0,0 +1,9 @@ +[id=[orig_h=212.180.42.100, orig_p=25000/tcp, resp_h=131.243.64.3, resp_p=53/tcp], orig=[size=29, state=5, num_pkts=6, num_bytes_ip=273, flow_label=0], resp=[size=44, state=5, num_pkts=5, num_bytes_ip=248, flow_label=0], start_time=930613226.067666, duration=0.709643, service={ + +}, addl=, hot=0, history=ShADadFf, uid=UWkUyAuUGXf, tunnel=, conn=[ts=930613226.067666, uid=UWkUyAuUGXf, id=[orig_h=212.180.42.100, orig_p=25000/tcp, resp_h=131.243.64.3, resp_p=53/tcp], proto=tcp, service=, duration=0.709643, orig_bytes=29, resp_bytes=44, conn_state=SF, local_orig=, missed_bytes=0, history=ShADadFf, orig_pkts=6, orig_ip_bytes=273, resp_pkts=5, resp_ip_bytes=248, tunnel_parents={ + +}], extract_orig=F, extract_resp=F, dns=, dns_state=[pending={ + +}, finished_answers={ +34798 +}]] diff --git a/testing/btest/Baseline/doc.manual.data_struct_record_01/.stdout b/testing/btest/Baseline/doc.manual.data_struct_record_01/.stdout new file mode 100644 index 0000000000..4e628b9ae7 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_struct_record_01/.stdout @@ -0,0 +1,6 @@ +Service: dns(RFC1035) + port: 53/tcp + port: 53/udp +Service: http(RFC2616) + port: 80/tcp + port: 8080/tcp diff --git a/testing/btest/Baseline/doc.manual.data_struct_record_02/.stdout b/testing/btest/Baseline/doc.manual.data_struct_record_02/.stdout new file mode 100644 index 0000000000..0428764bea --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_struct_record_02/.stdout @@ -0,0 +1,7 @@ +System: morlock + Service: dns(RFC1035) + port: 53/tcp + port: 53/udp + Service: http(RFC2616) + port: 80/tcp + port: 8080/tcp diff --git a/testing/btest/Baseline/doc.manual.data_struct_set_declaration/.stdout b/testing/btest/Baseline/doc.manual.data_struct_set_declaration/.stdout new file mode 100644 index 0000000000..d1aa16c7d3 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_struct_set_declaration/.stdout @@ -0,0 +1,8 @@ +SSL Port: 993/tcp +SSL Port: 22/tcp +SSL Port: 587/tcp +SSL Port: 443/tcp +Non-SSL Port: 143/tcp +Non-SSL Port: 25/tcp +Non-SSL Port: 80/tcp +Non-SSL Port: 23/tcp diff --git a/testing/btest/Baseline/doc.manual.data_struct_table_complex/.stdout b/testing/btest/Baseline/doc.manual.data_struct_table_complex/.stdout new file mode 100644 index 0000000000..e22f36a244 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_struct_table_complex/.stdout @@ -0,0 +1,4 @@ +Kiru was released in 1968 by Toho studios, directed by Kihachi Okamoto and starring Tatsuya Nakadai +Goyokin was released in 1969 by Fuji studios, directed by Hideo Gosha and starring Tatsuya Nakadai +Harakiri was released in 1962 by Shochiku Eiga studios, directed by Masaki Kobayashi and starring Tatsuya Nakadai +Tasogare Seibei was released in 2002 by Eisei Gekijo studios, directed by Yoji Yamada and starring Hiroyuki Sanada diff --git a/testing/btest/Baseline/doc.manual.data_struct_table_declaration/.stdout b/testing/btest/Baseline/doc.manual.data_struct_table_declaration/.stdout new file mode 100644 index 0000000000..19b1648904 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_struct_table_declaration/.stdout @@ -0,0 +1,4 @@ +Service Name: IMAPS - Common Port: 993/tcp +Service Name: HTTPS - Common Port: 443/tcp +Service Name: SSH - Common Port: 22/tcp +Service Name: SMTPS - Common Port: 587/tcp diff --git a/testing/btest/Baseline/doc.manual.data_struct_vector/.stdout b/testing/btest/Baseline/doc.manual.data_struct_vector/.stdout new file mode 100644 index 0000000000..8348ce7198 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_struct_vector/.stdout @@ -0,0 +1,2 @@ +[1, 2, 3, 4] +[1, 2, 3, 4] diff --git a/testing/btest/Baseline/doc.manual.data_struct_vector_declaration/.stdout b/testing/btest/Baseline/doc.manual.data_struct_vector_declaration/.stdout new file mode 100644 index 0000000000..48ce5d9c56 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_struct_vector_declaration/.stdout @@ -0,0 +1,4 @@ +contents of v1: [1, 2, 3, 4] +length of v1: 4 +contents of v1: [1, 2, 3, 4] +length of v2: 4 diff --git a/testing/btest/Baseline/doc.manual.data_struct_vector_iter/.stdout b/testing/btest/Baseline/doc.manual.data_struct_vector_iter/.stdout new file mode 100644 index 0000000000..0326e6580e --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_struct_vector_iter/.stdout @@ -0,0 +1,3 @@ +1.2.0.0/18 +2.3.0.0/18 +3.4.0.0/18 diff --git a/testing/btest/Baseline/doc.manual.data_type_const/.stdout b/testing/btest/Baseline/doc.manual.data_type_const/.stdout new file mode 100644 index 0000000000..0e49670a83 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_type_const/.stdout @@ -0,0 +1,4 @@ +{ +[6666/tcp] = IRC, +[80/tcp] = WWW +} diff --git a/testing/btest/Baseline/doc.manual.data_type_const_simple/.stdout b/testing/btest/Baseline/doc.manual.data_type_const_simple/.stdout new file mode 100644 index 0000000000..e69de29bb2 diff --git a/testing/btest/Baseline/doc.manual.data_type_declaration/.stdout b/testing/btest/Baseline/doc.manual.data_type_declaration/.stdout new file mode 100644 index 0000000000..a6f28b5e52 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_type_declaration/.stdout @@ -0,0 +1 @@ +A: 10, B: 10 diff --git a/testing/btest/Baseline/doc.manual.data_type_interval/.stdout b/testing/btest/Baseline/doc.manual.data_type_interval/.stdout new file mode 100644 index 0000000000..1cd5999711 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_type_interval/.stdout @@ -0,0 +1,15 @@ +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.118 +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3 + Time since last connection: 132.0 msecs 97.0 usecs +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3 + Time since last connection: 177.0 usecs +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3 + Time since last connection: 2.0 msecs 177.0 usecs +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3 + Time since last connection: 33.0 msecs 898.0 usecs +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3 + Time since last connection: 35.0 usecs +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3 + Time since last connection: 2.0 msecs 532.0 usecs +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.2 + Time since last connection: 7.0 msecs 866.0 usecs diff --git a/testing/btest/Baseline/doc.manual.data_type_local/.stdout b/testing/btest/Baseline/doc.manual.data_type_local/.stdout new file mode 100644 index 0000000000..e150c0b19d --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_type_local/.stdout @@ -0,0 +1 @@ +i + 2 = 12 diff --git a/testing/btest/Baseline/doc.manual.data_type_pattern_01/.stdout b/testing/btest/Baseline/doc.manual.data_type_pattern_01/.stdout new file mode 100644 index 0000000000..11358a776e --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_type_pattern_01/.stdout @@ -0,0 +1,3 @@ +The + brown fox jumped over the + dog. diff --git a/testing/btest/Baseline/doc.manual.data_type_pattern_02/.stdout b/testing/btest/Baseline/doc.manual.data_type_pattern_02/.stdout new file mode 100644 index 0000000000..808dc3d572 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_type_pattern_02/.stdout @@ -0,0 +1,2 @@ +equality and /^?(equal)$?/ are not equal +equality and /^?(equality)$?/ are equal diff --git a/testing/btest/Baseline/doc.manual.data_type_subnets/.stdout b/testing/btest/Baseline/doc.manual.data_type_subnets/.stdout new file mode 100644 index 0000000000..facaaabe64 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_type_subnets/.stdout @@ -0,0 +1,4 @@ +172.16.4.56 belongs to subnet 172.16.0.0/20 +172.16.47.254 belongs to subnet 172.16.32.0/20 +172.16.22.45 belongs to subnet 172.16.16.0/20 +172.16.1.1 belongs to subnet 172.16.0.0/20 diff --git a/testing/btest/Baseline/doc.manual.data_type_time/.stdout b/testing/btest/Baseline/doc.manual.data_type_time/.stdout new file mode 100644 index 0000000000..149cb40e2a --- /dev/null +++ b/testing/btest/Baseline/doc.manual.data_type_time/.stdout @@ -0,0 +1,8 @@ +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.118^J +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3^J +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3^J +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3^J +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3^J +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3^J +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3^J +2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.2^J diff --git a/testing/btest/Baseline/doc.manual.framework_logging_factorial_01/.stdout b/testing/btest/Baseline/doc.manual.framework_logging_factorial_01/.stdout new file mode 100644 index 0000000000..db47b283d0 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.framework_logging_factorial_01/.stdout @@ -0,0 +1,10 @@ +1 +2 +6 +24 +120 +720 +5040 +40320 +362880 +3628800 diff --git a/testing/btest/Baseline/doc.manual.framework_logging_factorial_02/factor.log b/testing/btest/Baseline/doc.manual.framework_logging_factorial_02/factor.log new file mode 100644 index 0000000000..c643116265 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.framework_logging_factorial_02/factor.log @@ -0,0 +1,19 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path factor +#open 2013-03-19-03-25-33 +#fields num factorial_num +#types count count +1 1 +2 2 +3 6 +4 24 +5 120 +6 720 +7 5040 +8 40320 +9 362880 +10 3628800 +#close 2013-03-19-03-25-33 diff --git a/testing/btest/Baseline/doc.manual.framework_logging_factorial_03/factor-mod5.log b/testing/btest/Baseline/doc.manual.framework_logging_factorial_03/factor-mod5.log new file mode 100644 index 0000000000..2a466484d6 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.framework_logging_factorial_03/factor-mod5.log @@ -0,0 +1,15 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path factor-mod5 +#open 2013-03-20-03-22-52 +#fields num factorial_num +#types count count +5 120 +6 720 +7 5040 +8 40320 +9 362880 +10 3628800 +#close 2013-03-20-03-22-52 diff --git a/testing/btest/Baseline/doc.manual.framework_logging_factorial_03/factor-non5.log b/testing/btest/Baseline/doc.manual.framework_logging_factorial_03/factor-non5.log new file mode 100644 index 0000000000..4430dcc8a4 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.framework_logging_factorial_03/factor-non5.log @@ -0,0 +1,13 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path factor-non5 +#open 2013-03-20-03-22-52 +#fields num factorial_num +#types count count +1 1 +2 2 +3 6 +4 24 +#close 2013-03-20-03-22-52 diff --git a/testing/btest/Baseline/doc.manual.framework_logging_factorial_04/factor-mod5.log b/testing/btest/Baseline/doc.manual.framework_logging_factorial_04/factor-mod5.log new file mode 100644 index 0000000000..6b50ca55e7 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.framework_logging_factorial_04/factor-mod5.log @@ -0,0 +1,15 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path factor-mod5 +#open 2013-03-25-02-00-12 +#fields num factorial_num +#types count count +5 120 +6 720 +7 5040 +8 40320 +9 362880 +10 3628800 +#close 2013-03-25-02-00-12 diff --git a/testing/btest/Baseline/doc.manual.framework_logging_factorial_04/factor-non5.log b/testing/btest/Baseline/doc.manual.framework_logging_factorial_04/factor-non5.log new file mode 100644 index 0000000000..d272ba48a9 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.framework_logging_factorial_04/factor-non5.log @@ -0,0 +1,13 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path factor-non5 +#open 2013-03-25-02-00-12 +#fields num factorial_num +#types count count +1 1 +2 2 +3 6 +4 24 +#close 2013-03-25-02-00-12 diff --git a/testing/btest/Baseline/doc.manual.framework_notice_hook_01/.stdout b/testing/btest/Baseline/doc.manual.framework_notice_hook_01/.stdout new file mode 100644 index 0000000000..e69de29bb2 diff --git a/testing/btest/Baseline/doc.manual.framework_notice_hook_suppression_01/.stdout b/testing/btest/Baseline/doc.manual.framework_notice_hook_suppression_01/.stdout new file mode 100644 index 0000000000..e69de29bb2 diff --git a/testing/btest/Baseline/doc.manual.framework_notice_shortcuts_01/.stdout b/testing/btest/Baseline/doc.manual.framework_notice_shortcuts_01/.stdout new file mode 100644 index 0000000000..e69de29bb2 diff --git a/testing/btest/Baseline/doc.manual.framework_notice_shortcuts_02/.stdout b/testing/btest/Baseline/doc.manual.framework_notice_shortcuts_02/.stdout new file mode 100644 index 0000000000..e69de29bb2 diff --git a/testing/btest/Baseline/doc.manual.using_bro_sandbox_01/.stdout b/testing/btest/Baseline/doc.manual.using_bro_sandbox_01/.stdout new file mode 100644 index 0000000000..e69de29bb2 diff --git a/testing/btest/Baseline/doc.manual.using_bro_sandbox_01/conn.log b/testing/btest/Baseline/doc.manual.using_bro_sandbox_01/conn.log new file mode 100644 index 0000000000..7fe6aa385c --- /dev/null +++ b/testing/btest/Baseline/doc.manual.using_bro_sandbox_01/conn.log @@ -0,0 +1,43 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path conn +#open 2013-05-05-20-51-24 +#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents +#types time string addr port addr port enum string interval count count string bool count string count count count count table[string] +1300475167.096535 UWkUyAuUGXf 141.142.220.202 5353 224.0.0.251 5353 udp dns - - - S0 - 0 D 1 73 0 0 (empty) +1300475167.097012 arKYeMETxOg fe80::217:f2ff:fed7:cf65 5353 ff02::fb 5353 udp - - - - S0 - 0 D 1 199 0 0 (empty) +1300475167.099816 k6kgXLOoSKl 141.142.220.50 5353 224.0.0.251 5353 udp - - - - S0 - 0 D 1 179 0 0 (empty) +1300475168.853899 TEfuqmmG4bh 141.142.220.118 43927 141.142.2.2 53 udp dns 0.000435 38 89 SF - 0 Dd 1 66 1 117 (empty) +1300475168.854378 FrJExwHcSal 141.142.220.118 37676 141.142.2.2 53 udp dns 0.000420 52 99 SF - 0 Dd 1 80 1 127 (empty) +1300475168.854837 5OKnoww6xl4 141.142.220.118 40526 141.142.2.2 53 udp dns 0.000392 38 183 SF - 0 Dd 1 66 1 211 (empty) +1300475168.857956 fRFu0wcOle6 141.142.220.118 32902 141.142.2.2 53 udp dns 0.000317 38 89 SF - 0 Dd 1 66 1 117 (empty) +1300475168.858306 qSsw6ESzHV4 141.142.220.118 59816 141.142.2.2 53 udp dns 0.000343 52 99 SF - 0 Dd 1 80 1 127 (empty) +1300475168.858713 iE6yhOq3SF 141.142.220.118 59714 141.142.2.2 53 udp dns 0.000375 38 183 SF - 0 Dd 1 66 1 211 (empty) +1300475168.891644 qCaWGmzFtM5 141.142.220.118 58206 141.142.2.2 53 udp dns 0.000339 38 89 SF - 0 Dd 1 66 1 117 (empty) +1300475168.892037 70MGiRM1Qf4 141.142.220.118 38911 141.142.2.2 53 udp dns 0.000335 52 99 SF - 0 Dd 1 80 1 127 (empty) +1300475168.892414 h5DsfNtYzi1 141.142.220.118 59746 141.142.2.2 53 udp dns 0.000421 38 183 SF - 0 Dd 1 66 1 211 (empty) +1300475168.893988 c4Zw9TmAE05 141.142.220.118 45000 141.142.2.2 53 udp dns 0.000384 38 89 SF - 0 Dd 1 66 1 117 (empty) +1300475168.894422 EAr0uf4mhq 141.142.220.118 48479 141.142.2.2 53 udp dns 0.000317 52 99 SF - 0 Dd 1 80 1 127 (empty) +1300475168.894787 GvmoxJFXdTa 141.142.220.118 48128 141.142.2.2 53 udp dns 0.000423 38 183 SF - 0 Dd 1 66 1 211 (empty) +1300475168.901749 slFea8xwSmb 141.142.220.118 56056 141.142.2.2 53 udp dns 0.000402 36 131 SF - 0 Dd 1 64 1 159 (empty) +1300475168.902195 UfGkYA2HI2g 141.142.220.118 55092 141.142.2.2 53 udp dns 0.000374 36 198 SF - 0 Dd 1 64 1 226 (empty) +1300475169.899438 BWaU4aSuwkc 141.142.220.44 5353 224.0.0.251 5353 udp dns - - - S0 - 0 D 1 85 0 0 (empty) +1300475170.862384 10XodEwRycf 141.142.220.226 137 141.142.220.255 137 udp dns 2.613017 350 0 S0 - 0 D 7 546 0 0 (empty) +1300475171.675372 zno26fFZkrh fe80::3074:17d5:2052:c324 65373 ff02::1:3 5355 udp dns 0.100096 66 0 S0 - 0 D 2 162 0 0 (empty) +1300475171.677081 v5rgkJBig5l 141.142.220.226 55131 224.0.0.252 5355 udp dns 0.100021 66 0 S0 - 0 D 2 122 0 0 (empty) +1300475173.116749 eWZCH7OONC1 fe80::3074:17d5:2052:c324 54213 ff02::1:3 5355 udp dns 0.099801 66 0 S0 - 0 D 2 162 0 0 (empty) +1300475173.117362 0Pwk3ntf8O3 141.142.220.226 55671 224.0.0.252 5355 udp dns 0.099849 66 0 S0 - 0 D 2 122 0 0 (empty) +1300475173.153679 0HKorjr8Zp7 141.142.220.238 56641 141.142.220.255 137 udp dns - - - S0 - 0 D 1 78 0 0 (empty) +1300475168.859163 GSxOnSLghOa 141.142.220.118 49998 208.80.152.3 80 tcp http 0.215893 1130 734 S1 - 0 ShADad 6 1450 4 950 (empty) +1300475168.652003 nQcgTWjvg4c 141.142.220.118 35634 208.80.152.2 80 tcp - 0.061329 463 350 OTH - 0 DdA 2 567 1 402 (empty) +1300475168.895267 0Q4FH8sESw5 141.142.220.118 50001 208.80.152.3 80 tcp http 0.227284 1178 734 S1 - 0 ShADad 6 1498 4 950 (empty) +1300475168.902635 i2rO3KD1Syg 141.142.220.118 35642 208.80.152.2 80 tcp http 0.120041 534 412 S1 - 0 ShADad 4 750 3 576 (empty) +1300475168.892936 Tw8jXtpTGu6 141.142.220.118 50000 208.80.152.3 80 tcp http 0.229603 1148 734 S1 - 0 ShADad 6 1468 4 950 (empty) +1300475168.855305 3PKsZ2Uye21 141.142.220.118 49996 208.80.152.3 80 tcp http 0.218501 1171 733 S1 - 0 ShADad 6 1491 4 949 (empty) +1300475168.892913 P654jzLoe3a 141.142.220.118 49999 208.80.152.3 80 tcp http 0.220961 1137 733 S1 - 0 ShADad 6 1457 4 949 (empty) +1300475169.780331 2cx26uAvUPl 141.142.220.235 6705 173.192.163.128 80 tcp - - - - OTH - 0 h 0 0 1 48 (empty) +1300475168.724007 j4u32Pc5bif 141.142.220.118 48649 208.80.152.118 80 tcp http 0.119905 525 232 S1 - 0 ShADad 4 741 3 396 (empty) +1300475168.855330 VW0XPVINV8a 141.142.220.118 49997 208.80.152.3 80 tcp http 0.219720 1125 734 S1 - 0 ShADad 6 1445 4 950 (empty) +#close 2013-05-05-20-51-24 diff --git a/testing/btest/Baseline/doc.manual.using_bro_sandbox_01/http.log b/testing/btest/Baseline/doc.manual.using_bro_sandbox_01/http.log new file mode 100644 index 0000000000..617c1f0e6e --- /dev/null +++ b/testing/btest/Baseline/doc.manual.using_bro_sandbox_01/http.log @@ -0,0 +1,23 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path http +#open 2013-05-05-21-12-40 +#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p trans_depth method host uri referrer user_agent request_body_len response_body_len status_code status_msg info_code info_msg filename tags username password proxied mime_type md5 extraction_file +#types time string addr port addr port count string string string string string count count count string count string string table[enum] string string table[string] string string file +1300475168.784020 j4u32Pc5bif 141.142.220.118 48649 208.80.152.118 80 1 GET bits.wikimedia.org /skins-1.5/monobook/main.css http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - - +1300475168.916018 VW0XPVINV8a 141.142.220.118 49997 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/6/63/Wikipedia-logo.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - - +1300475168.916183 3PKsZ2Uye21 141.142.220.118 49996 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/thumb/b/bb/Wikipedia_wordmark.svg/174px-Wikipedia_wordmark.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - - +1300475168.918358 GSxOnSLghOa 141.142.220.118 49998 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/b/bd/Bookshelf-40x201_6.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - - +1300475168.952307 Tw8jXtpTGu6 141.142.220.118 50000 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/thumb/8/8a/Wikinews-logo.png/35px-Wikinews-logo.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - - +1300475168.952296 P654jzLoe3a 141.142.220.118 49999 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/4/4a/Wiktionary-logo-en-35px.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - - +1300475168.954820 0Q4FH8sESw5 141.142.220.118 50001 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/thumb/f/fa/Wikiquote-logo.svg/35px-Wikiquote-logo.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - - +1300475168.962687 i2rO3KD1Syg 141.142.220.118 35642 208.80.152.2 80 1 GET meta.wikimedia.org /images/wikimedia-button.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - - +1300475168.975934 VW0XPVINV8a 141.142.220.118 49997 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/f/fa/Wikibooks-logo.svg/35px-Wikibooks-logo.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - - +1300475168.976436 3PKsZ2Uye21 141.142.220.118 49996 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/d/df/Wikispecies-logo.svg/35px-Wikispecies-logo.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - - +1300475168.979264 GSxOnSLghOa 141.142.220.118 49998 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/4/4c/Wikisource-logo.svg/35px-Wikisource-logo.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - - +1300475169.014619 Tw8jXtpTGu6 141.142.220.118 50000 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/4/4a/Commons-logo.svg/35px-Commons-logo.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - - +1300475169.014593 P654jzLoe3a 141.142.220.118 49999 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/9/91/Wikiversity-logo.svg/35px-Wikiversity-logo.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - - +1300475169.014927 0Q4FH8sESw5 141.142.220.118 50001 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/7/75/Wikimedia_Community_Logo.svg/35px-Wikimedia_Community_Logo.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - - +#close 2013-05-05-21-12-40 diff --git a/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/conn.log b/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/conn.log new file mode 100644 index 0000000000..1227e60ad3 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/conn.log @@ -0,0 +1,15 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path conn +#open 2013-05-07-14-38-27 +#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents +#types time string addr port addr port enum string interval count count string bool count string count count count count table[string] +1320329757.771503 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 tcp http 15.161537 2899 1127 S2 - 0 ShADadF 20 3719 19 1891 (empty) +1320329757.771262 nQcgTWjvg4c 10.0.2.15 49285 192.150.187.43 80 tcp http 15.161772 889 377 S2 - 0 ShADadF 8 1229 8 701 (empty) +1320329757.761327 arKYeMETxOg 10.0.2.15 49283 192.150.187.43 80 tcp http 15.168898 459 189 S2 - 0 ShADadF 5 679 4 353 (empty) +1320329757.458867 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 tcp http 15.471378 1824 751 S2 - 0 ShADadF 12 2324 13 1275 (empty) +1320329757.761638 k6kgXLOoSKl 10.0.2.15 49284 192.150.187.43 80 tcp http 15.168613 898 376 S2 - 0 ShADadF 8 1238 8 700 (empty) +1320329757.771755 TEfuqmmG4bh 10.0.2.15 49287 192.150.187.43 80 tcp http 15.161267 900 376 S2 - 0 ShADadF 8 1240 8 700 (empty) +#close 2013-05-07-14-38-27 diff --git a/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/http.log b/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/http.log new file mode 100644 index 0000000000..031a9ce2ce --- /dev/null +++ b/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/http.log @@ -0,0 +1,26 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path http +#open 2013-05-07-14-38-27 +#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p trans_depth method host uri referrer user_agent request_body_len response_body_len status_code status_msg info_code info_msg filename tags username password proxied mime_type md5 extraction_file +#types time string addr port addr port count string string string string string count count count string count string string table[enum] string string table[string] string string file +1320329757.460004 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 1 GET bro-ids.org / - Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.772457 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 2 GET bro-ids.org /css/pygments.css http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.874406 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 3 GET bro-ids.org /js/jquery.zrssfeed.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.775110 k6kgXLOoSKl 10.0.2.15 49284 192.150.187.43 80 1 GET bro-ids.org /css/960.css http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.776072 TEfuqmmG4bh 10.0.2.15 49287 192.150.187.43 80 1 GET bro-ids.org /js/jquery.cycle.all.min.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.776421 nQcgTWjvg4c 10.0.2.15 49285 192.150.187.43 80 1 GET bro-ids.org /js/jquery.tweet.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.776240 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 1 GET bro-ids.org /js/jquery.fancybox-1.3.4.pack.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.775251 arKYeMETxOg 10.0.2.15 49283 192.150.187.43 80 1 GET bro-ids.org /css/bro-ids.css http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.975651 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 4 GET bro-ids.org /js/jquery.tableofcontents.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.979943 k6kgXLOoSKl 10.0.2.15 49284 192.150.187.43 80 2 GET bro-ids.org /js/superfish.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.985656 TEfuqmmG4bh 10.0.2.15 49287 192.150.187.43 80 2 GET bro-ids.org /js/hoverIntent.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.989904 nQcgTWjvg4c 10.0.2.15 49285 192.150.187.43 80 2 GET bro-ids.org /js/general.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.991315 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 2 GET bro-ids.org /js/jquery.collapse.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329758.172397 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 3 GET bro-ids.org /css/print.css http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329759.998388 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 4 GET bro-ids.org /documentation/index.html http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329760.146412 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 5 GET bro-ids.org /js/breadcrumbs.js http://bro-ids.org/documentation/index.html Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329762.971726 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 6 GET bro-ids.org /documentation/reporting-problems.html http://bro-ids.org/documentation/index.html Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +#close 2013-05-07-14-38-27 diff --git a/testing/btest/Baseline/doc.sphinx.connection-record-01/btest-doc.sphinx.connection-record-01#1 b/testing/btest/Baseline/doc.sphinx.connection-record-01/btest-doc.sphinx.connection-record-01#1 new file mode 100644 index 0000000000..d3ba4e3d1c --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.connection-record-01/btest-doc.sphinx.connection-record-01#1 @@ -0,0 +1,11 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro -b -r dns-session.trace connection_record_01.bro + [id=[orig_h=212.180.42.100, orig_p=25000/tcp, resp_h=131.243.64.3, resp_p=53/tcp], orig=[size=29, state=5, num_pkts=6, num_bytes_ip=273, flow_label=0], resp=[size=44, state=5, num_pkts=5, num_bytes_ip=248, flow_label=0], start_time=930613226.067666, duration=0.709643, service={ + + }, addl=, hot=0, history=ShADadFf, uid=CXWv6p3arKYeMETxOg, tunnel=, conn=[ts=930613226.067666, uid=CXWv6p3arKYeMETxOg, id=[orig_h=212.180.42.100, orig_p=25000/tcp, resp_h=131.243.64.3, resp_p=53/tcp], proto=tcp, service=, duration=0.709643, orig_bytes=29, resp_bytes=44, conn_state=SF, local_orig=, missed_bytes=0, history=ShADadFf, orig_pkts=6, orig_ip_bytes=273, resp_pkts=5, resp_ip_bytes=248, tunnel_parents={ + + }], extract_orig=F, extract_resp=F] + diff --git a/testing/btest/Baseline/doc.sphinx.connection-record-02/btest-doc.sphinx.connection-record-02#1 b/testing/btest/Baseline/doc.sphinx.connection-record-02/btest-doc.sphinx.connection-record-02#1 new file mode 100644 index 0000000000..8671de3372 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.connection-record-02/btest-doc.sphinx.connection-record-02#1 @@ -0,0 +1,17 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro -b -r dns-session.trace connection_record_02.bro + [id=[orig_h=212.180.42.100, orig_p=25000/tcp, resp_h=131.243.64.3, resp_p=53/tcp], orig=[size=29, state=5, num_pkts=6, num_bytes_ip=273, flow_label=0], resp=[size=44, state=5, num_pkts=5, num_bytes_ip=248, flow_label=0], start_time=930613226.067666, duration=0.709643, service={ + + }, addl=, hot=0, history=ShADadFf, uid=CXWv6p3arKYeMETxOg, tunnel=, conn=[ts=930613226.067666, uid=CXWv6p3arKYeMETxOg, id=[orig_h=212.180.42.100, orig_p=25000/tcp, resp_h=131.243.64.3, resp_p=53/tcp], proto=tcp, service=, duration=0.709643, orig_bytes=29, resp_bytes=44, conn_state=SF, local_orig=, missed_bytes=0, history=ShADadFf, orig_pkts=6, orig_ip_bytes=273, resp_pkts=5, resp_ip_bytes=248, tunnel_parents={ + + }], extract_orig=F, extract_resp=F, dns=, dns_state=[pending={ + [34798] = [initialized=T, vals={ + + }, settings=[max_len=], top=1, bottom=1, size=0] + }, finished_answers={ + + }]] + diff --git a/testing/btest/Baseline/doc.sphinx.data_struct_record_01/btest-doc.sphinx.data_struct_record_01#1 b/testing/btest/Baseline/doc.sphinx.data_struct_record_01/btest-doc.sphinx.data_struct_record_01#1 new file mode 100644 index 0000000000..5b8d702333 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.data_struct_record_01/btest-doc.sphinx.data_struct_record_01#1 @@ -0,0 +1,12 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro data_struct_record_01.bro + Service: dns(RFC1035) + port: 53/tcp + port: 53/udp + Service: http(RFC2616) + port: 80/tcp + port: 8080/tcp + diff --git a/testing/btest/Baseline/doc.sphinx.data_struct_record_02/btest-doc.sphinx.data_struct_record_02#1 b/testing/btest/Baseline/doc.sphinx.data_struct_record_02/btest-doc.sphinx.data_struct_record_02#1 new file mode 100644 index 0000000000..b640862c49 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.data_struct_record_02/btest-doc.sphinx.data_struct_record_02#1 @@ -0,0 +1,13 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro data_struct_record_02.bro + System: morlock + Service: dns(RFC1035) + port: 53/tcp + port: 53/udp + Service: http(RFC2616) + port: 80/tcp + port: 8080/tcp + diff --git a/testing/btest/Baseline/doc.sphinx.data_struct_set_declaration/btest-doc.sphinx.data_struct_set_declaration#1 b/testing/btest/Baseline/doc.sphinx.data_struct_set_declaration/btest-doc.sphinx.data_struct_set_declaration#1 new file mode 100644 index 0000000000..f22481364a --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.data_struct_set_declaration/btest-doc.sphinx.data_struct_set_declaration#1 @@ -0,0 +1,14 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro data_struct_set_declaration.bro + SSL Port: 993/tcp + SSL Port: 22/tcp + SSL Port: 587/tcp + SSL Port: 443/tcp + Non-SSL Port: 143/tcp + Non-SSL Port: 25/tcp + Non-SSL Port: 80/tcp + Non-SSL Port: 23/tcp + diff --git a/testing/btest/Baseline/doc.sphinx.data_struct_table_complex/btest-doc.sphinx.data_struct_table_complex#1 b/testing/btest/Baseline/doc.sphinx.data_struct_table_complex/btest-doc.sphinx.data_struct_table_complex#1 new file mode 100644 index 0000000000..415d6b4436 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.data_struct_table_complex/btest-doc.sphinx.data_struct_table_complex#1 @@ -0,0 +1,10 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro -b data_struct_table_complex.bro + Kiru was released in 1968 by Toho studios, directed by Kihachi Okamoto and starring Tatsuya Nakadai + Goyokin was released in 1969 by Fuji studios, directed by Hideo Gosha and starring Tatsuya Nakadai + Harakiri was released in 1962 by Shochiku Eiga studios, directed by Masaki Kobayashi and starring Tatsuya Nakadai + Tasogare Seibei was released in 2002 by Eisei Gekijo studios, directed by Yoji Yamada and starring Hiroyuki Sanada + diff --git a/testing/btest/Baseline/doc.sphinx.data_struct_table_declaration/btest-doc.sphinx.data_struct_table_declaration#1 b/testing/btest/Baseline/doc.sphinx.data_struct_table_declaration/btest-doc.sphinx.data_struct_table_declaration#1 new file mode 100644 index 0000000000..0b6efd6166 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.data_struct_table_declaration/btest-doc.sphinx.data_struct_table_declaration#1 @@ -0,0 +1,10 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro data_struct_table_declaration.bro + Service Name: IMAPS - Common Port: 993/tcp + Service Name: HTTPS - Common Port: 443/tcp + Service Name: SSH - Common Port: 22/tcp + Service Name: SMTPS - Common Port: 587/tcp + diff --git a/testing/btest/Baseline/doc.sphinx.data_struct_vector_declaration/btest-doc.sphinx.data_struct_vector_declaration#1 b/testing/btest/Baseline/doc.sphinx.data_struct_vector_declaration/btest-doc.sphinx.data_struct_vector_declaration#1 new file mode 100644 index 0000000000..c282de7c49 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.data_struct_vector_declaration/btest-doc.sphinx.data_struct_vector_declaration#1 @@ -0,0 +1,10 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro data_struct_vector_declaration.bro + contents of v1: [1, 2, 3, 4] + length of v1: 4 + contents of v1: [1, 2, 3, 4] + length of v2: 4 + diff --git a/testing/btest/Baseline/doc.sphinx.data_struct_vector_iter/btest-doc.sphinx.data_struct_vector_iter#1 b/testing/btest/Baseline/doc.sphinx.data_struct_vector_iter/btest-doc.sphinx.data_struct_vector_iter#1 new file mode 100644 index 0000000000..88634f57a6 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.data_struct_vector_iter/btest-doc.sphinx.data_struct_vector_iter#1 @@ -0,0 +1,9 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro -b data_struct_vector_iter.bro + 1.2.0.0/18 + 2.3.0.0/18 + 3.4.0.0/18 + diff --git a/testing/btest/Baseline/doc.sphinx.data_type_const.bro/btest-doc.sphinx.data_type_const.bro#1 b/testing/btest/Baseline/doc.sphinx.data_type_const.bro/btest-doc.sphinx.data_type_const.bro#1 new file mode 100644 index 0000000000..48532cfb84 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.data_type_const.bro/btest-doc.sphinx.data_type_const.bro#1 @@ -0,0 +1,10 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro -b data_type_const.bro + { + [6666/tcp] = IRC, + [80/tcp] = WWW + } + diff --git a/testing/btest/Baseline/doc.sphinx.data_type_interval/btest-doc.sphinx.data_type_interval#1 b/testing/btest/Baseline/doc.sphinx.data_type_interval/btest-doc.sphinx.data_type_interval#1 new file mode 100644 index 0000000000..d200467611 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.data_type_interval/btest-doc.sphinx.data_type_interval#1 @@ -0,0 +1,23 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro -r wikipedia.trace data_type_interval.bro + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.118 + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3 + Time since last connection: 132.0 msecs 97.0 usecs + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3 + Time since last connection: 177.0 usecs + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3 + Time since last connection: 2.0 msecs 177.0 usecs + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3 + Time since last connection: 33.0 msecs 898.0 usecs + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3 + Time since last connection: 35.0 usecs + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3 + Time since last connection: 2.0 msecs 532.0 usecs + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.2 + Time since last connection: 7.0 msecs 866.0 usecs + 2011/06/18 19:03:09: New connection established from 141.142.220.235 to 173.192.163.128 + Time since last connection: 817.0 msecs 703.0 usecs + diff --git a/testing/btest/Baseline/doc.sphinx.data_type_pattern/btest-doc.sphinx.data_type_pattern#1 b/testing/btest/Baseline/doc.sphinx.data_type_pattern/btest-doc.sphinx.data_type_pattern#1 new file mode 100644 index 0000000000..e74a70e576 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.data_type_pattern/btest-doc.sphinx.data_type_pattern#1 @@ -0,0 +1,9 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro data_type_pattern_01.bro + The + brown fox jumped over the + dog. + diff --git a/testing/btest/Baseline/doc.sphinx.data_type_pattern_02/btest-doc.sphinx.data_type_pattern_02#1 b/testing/btest/Baseline/doc.sphinx.data_type_pattern_02/btest-doc.sphinx.data_type_pattern_02#1 new file mode 100644 index 0000000000..3097c85bcd --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.data_type_pattern_02/btest-doc.sphinx.data_type_pattern_02#1 @@ -0,0 +1,8 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro data_type_pattern_02.bro + equality and /^?(equal)$?/ are not equal + equality and /^?(equality)$?/ are equal + diff --git a/testing/btest/Baseline/doc.sphinx.data_type_subnets/btest-doc.sphinx.data_type_subnets#1 b/testing/btest/Baseline/doc.sphinx.data_type_subnets/btest-doc.sphinx.data_type_subnets#1 new file mode 100644 index 0000000000..92d18b84ba --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.data_type_subnets/btest-doc.sphinx.data_type_subnets#1 @@ -0,0 +1,10 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro data_type_subnets.bro + 172.16.4.56 belongs to subnet 172.16.0.0/20 + 172.16.47.254 belongs to subnet 172.16.32.0/20 + 172.16.22.45 belongs to subnet 172.16.16.0/20 + 172.16.1.1 belongs to subnet 172.16.0.0/20 + diff --git a/testing/btest/Baseline/doc.sphinx.data_type_time/btest-doc.sphinx.data_type_time#1 b/testing/btest/Baseline/doc.sphinx.data_type_time/btest-doc.sphinx.data_type_time#1 new file mode 100644 index 0000000000..d83f819038 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.data_type_time/btest-doc.sphinx.data_type_time#1 @@ -0,0 +1,15 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro -r wikipedia.trace data_type_time.bro + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.118^J + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3^J + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3^J + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3^J + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3^J + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3^J + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.3^J + 2011/06/18 19:03:08: New connection established from 141.142.220.118 to 208.80.152.2^J + 2011/06/18 19:03:09: New connection established from 141.142.220.235 to 173.192.163.128^J + diff --git a/testing/btest/Baseline/doc.sphinx.framework_logging_factorial-2/btest-doc.sphinx.framework_logging_factorial-2#1 b/testing/btest/Baseline/doc.sphinx.framework_logging_factorial-2/btest-doc.sphinx.framework_logging_factorial-2#1 new file mode 100644 index 0000000000..3bd2292105 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.framework_logging_factorial-2/btest-doc.sphinx.framework_logging_factorial-2#1 @@ -0,0 +1,29 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro framework_logging_factorial_02.bro + +.. code-block:: guess + :linenos: + + #separator \x09 + #set_separator , + #empty_field (empty) + #unset_field - + #path factor + #open 2013-08-31-22-50-17 + #fields num factorial_num + #types count count + 1 1 + 2 2 + 3 6 + 4 24 + 5 120 + 6 720 + 7 5040 + 8 40320 + 9 362880 + 10 3628800 + #close 2013-08-31-22-50-17 + diff --git a/testing/btest/Baseline/doc.sphinx.framework_logging_factorial-3/btest-doc.sphinx.framework_logging_factorial-3#1 b/testing/btest/Baseline/doc.sphinx.framework_logging_factorial-3/btest-doc.sphinx.framework_logging_factorial-3#1 new file mode 100644 index 0000000000..02ebfcec99 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.framework_logging_factorial-3/btest-doc.sphinx.framework_logging_factorial-3#1 @@ -0,0 +1,25 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro framework_logging_factorial_03.bro + +.. code-block:: guess + :linenos: + + #separator \x09 + #set_separator , + #empty_field (empty) + #unset_field - + #path factor-mod5 + #open 2013-08-31-22-50-17 + #fields num factorial_num + #types count count + 5 120 + 6 720 + 7 5040 + 8 40320 + 9 362880 + 10 3628800 + #close 2013-08-31-22-50-17 + diff --git a/testing/btest/Baseline/doc.sphinx.framework_logging_factorial/btest-doc.sphinx.framework_logging_factorial#1 b/testing/btest/Baseline/doc.sphinx.framework_logging_factorial/btest-doc.sphinx.framework_logging_factorial#1 new file mode 100644 index 0000000000..8a64a2da65 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.framework_logging_factorial/btest-doc.sphinx.framework_logging_factorial#1 @@ -0,0 +1,16 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro framework_logging_factorial_01.bro + 1 + 2 + 6 + 24 + 120 + 720 + 5040 + 40320 + 362880 + 3628800 + diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_connection_record_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_connection_record_02_bro.include-doc_scripting_connection_record_02_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_connection_record_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_connection_record_02_bro.include-doc_scripting_connection_record_02_bro.btest new file mode 100644 index 0000000000..43a6ebc554 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_connection_record_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_connection_record_02_bro.include-doc_scripting_connection_record_02_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- connection_record_02.bro + +@load base/protocols/conn +@load base/protocols/dns + +event connection_state_remove(c: connection) + { + print c; + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_connection_record_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_connection_record_02_bro.include-doc_scripting_connection_record_02_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_connection_record_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_connection_record_02_bro.include-doc_scripting_connection_record_02_bro.btest new file mode 100644 index 0000000000..43a6ebc554 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_connection_record_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_connection_record_02_bro.include-doc_scripting_connection_record_02_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- connection_record_02.bro + +@load base/protocols/conn +@load base/protocols/dns + +event connection_state_remove(c: connection) + { + print c; + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_connection_record_02_bro@2/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_connection_record_02_bro@2.include-doc_scripting_connection_record_02_bro@2.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_connection_record_02_bro@2/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_connection_record_02_bro@2.include-doc_scripting_connection_record_02_bro@2.btest new file mode 100644 index 0000000000..43a6ebc554 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_connection_record_02_bro@2/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_connection_record_02_bro@2.include-doc_scripting_connection_record_02_bro@2.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- connection_record_02.bro + +@load base/protocols/conn +@load base/protocols/dns + +event connection_state_remove(c: connection) + { + print c; + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_connection_record_02_bro@2/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_connection_record_02_bro@2.include-doc_scripting_connection_record_02_bro@2.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_connection_record_02_bro@2/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_connection_record_02_bro@2.include-doc_scripting_connection_record_02_bro@2.btest new file mode 100644 index 0000000000..43a6ebc554 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_connection_record_02_bro@2/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_connection_record_02_bro@2.include-doc_scripting_connection_record_02_bro@2.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- connection_record_02.bro + +@load base/protocols/conn +@load base/protocols/dns + +event connection_state_remove(c: connection) + { + print c; + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_record_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_record_01_bro.include-doc_scripting_data_struct_record_01_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_record_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_record_01_bro.include-doc_scripting_data_struct_record_01_bro.btest new file mode 100644 index 0000000000..82f1b1f56c --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_record_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_record_01_bro.include-doc_scripting_data_struct_record_01_bro.btest @@ -0,0 +1,26 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_record_01.bro + +type Service: record { + name: string; + ports: set[port]; + rfc: count; +}; + +function print_service(serv: Service): string + { + print fmt("Service: %s(RFC%d)",serv$name, serv$rfc); + + for ( p in serv$ports ) + print fmt(" port: %s", p); + } + +event bro_init() + { + local dns: Service = [$name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035]; + local http: Service = [$name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616]; + + print_service(dns); + print_service(http); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_record_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_record_01_bro.include-doc_scripting_data_struct_record_01_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_record_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_record_01_bro.include-doc_scripting_data_struct_record_01_bro.btest new file mode 100644 index 0000000000..82f1b1f56c --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_record_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_record_01_bro.include-doc_scripting_data_struct_record_01_bro.btest @@ -0,0 +1,26 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_record_01.bro + +type Service: record { + name: string; + ports: set[port]; + rfc: count; +}; + +function print_service(serv: Service): string + { + print fmt("Service: %s(RFC%d)",serv$name, serv$rfc); + + for ( p in serv$ports ) + print fmt(" port: %s", p); + } + +event bro_init() + { + local dns: Service = [$name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035]; + local http: Service = [$name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616]; + + print_service(dns); + print_service(http); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_record_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_record_02_bro.include-doc_scripting_data_struct_record_02_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_record_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_record_02_bro.include-doc_scripting_data_struct_record_02_bro.btest new file mode 100644 index 0000000000..05b5b9d50c --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_record_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_record_02_bro.include-doc_scripting_data_struct_record_02_bro.btest @@ -0,0 +1,45 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_record_02.bro + +type Service: record { + name: string; + ports: set[port]; + rfc: count; + }; + +type System: record { + name: string; + services: set[Service]; + }; + +function print_service(serv: Service): string + { + print fmt(" Service: %s(RFC%d)",serv$name, serv$rfc); + + for ( p in serv$ports ) + print fmt(" port: %s", p); + } + +function print_system(sys: System): string + { + print fmt("System: %s", sys$name); + + for ( s in sys$services ) + print_service(s); + } + +event bro_init() + { + local server01: System; + server01$name = "morlock"; + add server01$services[[ $name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035]]; + add server01$services[[ $name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616]]; + print_system(server01); + + + # local dns: Service = [ $name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035]; + # local http: Service = [ $name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616]; + # print_service(dns); + # print_service(http); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_record_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_record_02_bro.include-doc_scripting_data_struct_record_02_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_record_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_record_02_bro.include-doc_scripting_data_struct_record_02_bro.btest new file mode 100644 index 0000000000..05b5b9d50c --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_record_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_record_02_bro.include-doc_scripting_data_struct_record_02_bro.btest @@ -0,0 +1,45 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_record_02.bro + +type Service: record { + name: string; + ports: set[port]; + rfc: count; + }; + +type System: record { + name: string; + services: set[Service]; + }; + +function print_service(serv: Service): string + { + print fmt(" Service: %s(RFC%d)",serv$name, serv$rfc); + + for ( p in serv$ports ) + print fmt(" port: %s", p); + } + +function print_system(sys: System): string + { + print fmt("System: %s", sys$name); + + for ( s in sys$services ) + print_service(s); + } + +event bro_init() + { + local server01: System; + server01$name = "morlock"; + add server01$services[[ $name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035]]; + add server01$services[[ $name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616]]; + print_system(server01); + + + # local dns: Service = [ $name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035]; + # local http: Service = [ $name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616]; + # print_service(dns); + # print_service(http); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro.include-doc_scripting_data_struct_set_declaration_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro.include-doc_scripting_data_struct_set_declaration_bro.btest new file mode 100644 index 0000000000..b659563ac3 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro.include-doc_scripting_data_struct_set_declaration_bro.btest @@ -0,0 +1,9 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_set_declaration.bro + +event bro_init() + { + local ssl_ports: set[port]; + local non_ssl_ports = set( 23/tcp, 80/tcp, 143/tcp, 25/tcp ); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro.include-doc_scripting_data_struct_set_declaration_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro.include-doc_scripting_data_struct_set_declaration_bro.btest new file mode 100644 index 0000000000..b659563ac3 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro.include-doc_scripting_data_struct_set_declaration_bro.btest @@ -0,0 +1,9 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_set_declaration.bro + +event bro_init() + { + local ssl_ports: set[port]; + local non_ssl_ports = set( 23/tcp, 80/tcp, 143/tcp, 25/tcp ); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@2/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@2.include-doc_scripting_data_struct_set_declaration_bro@2.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@2/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@2.include-doc_scripting_data_struct_set_declaration_bro@2.btest new file mode 100644 index 0000000000..5fa1f36475 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@2/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@2.include-doc_scripting_data_struct_set_declaration_bro@2.btest @@ -0,0 +1,9 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_set_declaration.bro + + for ( i in ssl_ports ) + print fmt("SSL Port: %s", i); + + for ( i in non_ssl_ports ) + print fmt("Non-SSL Port: %s", i); diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@2/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@2.include-doc_scripting_data_struct_set_declaration_bro@2.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@2/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@2.include-doc_scripting_data_struct_set_declaration_bro@2.btest new file mode 100644 index 0000000000..5fa1f36475 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@2/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@2.include-doc_scripting_data_struct_set_declaration_bro@2.btest @@ -0,0 +1,9 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_set_declaration.bro + + for ( i in ssl_ports ) + print fmt("SSL Port: %s", i); + + for ( i in non_ssl_ports ) + print fmt("Non-SSL Port: %s", i); diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@3/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@3.include-doc_scripting_data_struct_set_declaration_bro@3.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@3/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@3.include-doc_scripting_data_struct_set_declaration_bro@3.btest new file mode 100644 index 0000000000..48d33b52e5 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@3/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@3.include-doc_scripting_data_struct_set_declaration_bro@3.btest @@ -0,0 +1,7 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_set_declaration.bro + + # Check for SMTPS + if ( 587/tcp !in ssl_ports ) + add ssl_ports[587/tcp]; diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@3/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@3.include-doc_scripting_data_struct_set_declaration_bro@3.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@3/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@3.include-doc_scripting_data_struct_set_declaration_bro@3.btest new file mode 100644 index 0000000000..48d33b52e5 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@3/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@3.include-doc_scripting_data_struct_set_declaration_bro@3.btest @@ -0,0 +1,7 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_set_declaration.bro + + # Check for SMTPS + if ( 587/tcp !in ssl_ports ) + add ssl_ports[587/tcp]; diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@4/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@4.include-doc_scripting_data_struct_set_declaration_bro@4.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@4/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@4.include-doc_scripting_data_struct_set_declaration_bro@4.btest new file mode 100644 index 0000000000..19ddfe81d5 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@4/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@4.include-doc_scripting_data_struct_set_declaration_bro@4.btest @@ -0,0 +1,26 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_set_declaration.bro + +event bro_init() + { + local ssl_ports: set[port]; + local non_ssl_ports = set( 23/tcp, 80/tcp, 143/tcp, 25/tcp ); + + # SSH + add ssl_ports[22/tcp]; + # HTTPS + add ssl_ports[443/tcp]; + # IMAPS + add ssl_ports[993/tcp]; + + # Check for SMTPS + if ( 587/tcp !in ssl_ports ) + add ssl_ports[587/tcp]; + + for ( i in ssl_ports ) + print fmt("SSL Port: %s", i); + + for ( i in non_ssl_ports ) + print fmt("Non-SSL Port: %s", i); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@4/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@4.include-doc_scripting_data_struct_set_declaration_bro@4.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@4/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@4.include-doc_scripting_data_struct_set_declaration_bro@4.btest new file mode 100644 index 0000000000..19ddfe81d5 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@4/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_set_declaration_bro@4.include-doc_scripting_data_struct_set_declaration_bro@4.btest @@ -0,0 +1,26 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_set_declaration.bro + +event bro_init() + { + local ssl_ports: set[port]; + local non_ssl_ports = set( 23/tcp, 80/tcp, 143/tcp, 25/tcp ); + + # SSH + add ssl_ports[22/tcp]; + # HTTPS + add ssl_ports[443/tcp]; + # IMAPS + add ssl_ports[993/tcp]; + + # Check for SMTPS + if ( 587/tcp !in ssl_ports ) + add ssl_ports[587/tcp]; + + for ( i in ssl_ports ) + print fmt("SSL Port: %s", i); + + for ( i in non_ssl_ports ) + print fmt("Non-SSL Port: %s", i); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_table_complex_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_table_complex_bro.include-doc_scripting_data_struct_table_complex_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_table_complex_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_table_complex_bro.include-doc_scripting_data_struct_table_complex_bro.btest new file mode 100644 index 0000000000..a6a1d7a63a --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_table_complex_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_table_complex_bro.include-doc_scripting_data_struct_table_complex_bro.btest @@ -0,0 +1,17 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_table_complex.bro + +event bro_init() + { + local samurai_flicks: table[string, string, count, string] of string; + + samurai_flicks["Kihachi Okamoto", "Toho", 1968, "Tatsuya Nakadai"] = "Kiru"; + samurai_flicks["Hideo Gosha", "Fuji", 1969, "Tatsuya Nakadai"] = "Goyokin"; + samurai_flicks["Masaki Kobayashi", "Shochiku Eiga", 1962, "Tatsuya Nakadai" ] = "Harakiri"; + samurai_flicks["Yoji Yamada", "Eisei Gekijo", 2002, "Hiroyuki Sanada" ] = "Tasogare Seibei"; + + for ( [d, s, y, a] in samurai_flicks ) + print fmt("%s was released in %d by %s studios, directed by %s and starring %s", samurai_flicks[d, s, y, a], y, s, d, a); + } + diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_table_complex_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_table_complex_bro.include-doc_scripting_data_struct_table_complex_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_table_complex_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_table_complex_bro.include-doc_scripting_data_struct_table_complex_bro.btest new file mode 100644 index 0000000000..a6a1d7a63a --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_table_complex_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_table_complex_bro.include-doc_scripting_data_struct_table_complex_bro.btest @@ -0,0 +1,17 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_table_complex.bro + +event bro_init() + { + local samurai_flicks: table[string, string, count, string] of string; + + samurai_flicks["Kihachi Okamoto", "Toho", 1968, "Tatsuya Nakadai"] = "Kiru"; + samurai_flicks["Hideo Gosha", "Fuji", 1969, "Tatsuya Nakadai"] = "Goyokin"; + samurai_flicks["Masaki Kobayashi", "Shochiku Eiga", 1962, "Tatsuya Nakadai" ] = "Harakiri"; + samurai_flicks["Yoji Yamada", "Eisei Gekijo", 2002, "Hiroyuki Sanada" ] = "Tasogare Seibei"; + + for ( [d, s, y, a] in samurai_flicks ) + print fmt("%s was released in %d by %s studios, directed by %s and starring %s", samurai_flicks[d, s, y, a], y, s, d, a); + } + diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_table_declaration_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_table_declaration_bro.include-doc_scripting_data_struct_table_declaration_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_table_declaration_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_table_declaration_bro.include-doc_scripting_data_struct_table_declaration_bro.btest new file mode 100644 index 0000000000..79fe7660ae --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_table_declaration_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_table_declaration_bro.include-doc_scripting_data_struct_table_declaration_bro.btest @@ -0,0 +1,17 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_table_declaration.bro + +event bro_init() + { + local ssl_services: table[string] of port; + + ssl_services = table(["SSH"] = 22/tcp, ["HTTPS"] = 443/tcp); + ssl_services["IMAPS"] = 993/tcp; + + if ( "SMTPS" !in ssl_services ) + ssl_services["SMTPS"] = 587/tcp; + + for ( k in ssl_services ) + print fmt("Service Name: %s - Common Port: %s", k, ssl_services[k]); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_table_declaration_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_table_declaration_bro.include-doc_scripting_data_struct_table_declaration_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_table_declaration_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_table_declaration_bro.include-doc_scripting_data_struct_table_declaration_bro.btest new file mode 100644 index 0000000000..79fe7660ae --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_table_declaration_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_table_declaration_bro.include-doc_scripting_data_struct_table_declaration_bro.btest @@ -0,0 +1,17 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_table_declaration.bro + +event bro_init() + { + local ssl_services: table[string] of port; + + ssl_services = table(["SSH"] = 22/tcp, ["HTTPS"] = 443/tcp); + ssl_services["IMAPS"] = 993/tcp; + + if ( "SMTPS" !in ssl_services ) + ssl_services["SMTPS"] = 587/tcp; + + for ( k in ssl_services ) + print fmt("Service Name: %s - Common Port: %s", k, ssl_services[k]); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_vector_declaration_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_vector_declaration_bro.include-doc_scripting_data_struct_vector_declaration_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_vector_declaration_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_vector_declaration_bro.include-doc_scripting_data_struct_vector_declaration_bro.btest new file mode 100644 index 0000000000..6b8a81a58f --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_vector_declaration_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_vector_declaration_bro.include-doc_scripting_data_struct_vector_declaration_bro.btest @@ -0,0 +1,19 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_vector_declaration.bro + +event bro_init() + { + local v1: vector of count; + local v2 = vector(1, 2, 3, 4); + + v1[|v1|] = 1; + v1[|v1|] = 2; + v1[|v1|] = 3; + v1[|v1|] = 4; + + print fmt("contents of v1: %s", v1); + print fmt("length of v1: %d", |v1|); + print fmt("contents of v1: %s", v2); + print fmt("length of v2: %d", |v2|); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_vector_declaration_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_vector_declaration_bro.include-doc_scripting_data_struct_vector_declaration_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_vector_declaration_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_vector_declaration_bro.include-doc_scripting_data_struct_vector_declaration_bro.btest new file mode 100644 index 0000000000..6b8a81a58f --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_vector_declaration_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_vector_declaration_bro.include-doc_scripting_data_struct_vector_declaration_bro.btest @@ -0,0 +1,19 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_vector_declaration.bro + +event bro_init() + { + local v1: vector of count; + local v2 = vector(1, 2, 3, 4); + + v1[|v1|] = 1; + v1[|v1|] = 2; + v1[|v1|] = 3; + v1[|v1|] = 4; + + print fmt("contents of v1: %s", v1); + print fmt("length of v1: %d", |v1|); + print fmt("contents of v1: %s", v2); + print fmt("length of v2: %d", |v2|); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_vector_iter_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_vector_iter_bro.include-doc_scripting_data_struct_vector_iter_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_vector_iter_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_vector_iter_bro.include-doc_scripting_data_struct_vector_iter_bro.btest new file mode 100644 index 0000000000..a28522fb64 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_vector_iter_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_vector_iter_bro.include-doc_scripting_data_struct_vector_iter_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_vector_iter.bro + +event bro_init() + { + local addr_vector: vector of addr = vector(1.2.3.4, 2.3.4.5, 3.4.5.6); + + for (i in addr_vector) + print mask_addr(addr_vector[i], 18); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_vector_iter_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_vector_iter_bro.include-doc_scripting_data_struct_vector_iter_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_vector_iter_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_vector_iter_bro.include-doc_scripting_data_struct_vector_iter_bro.btest new file mode 100644 index 0000000000..a28522fb64 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_struct_vector_iter_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_struct_vector_iter_bro.include-doc_scripting_data_struct_vector_iter_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_vector_iter.bro + +event bro_init() + { + local addr_vector: vector of addr = vector(1.2.3.4, 2.3.4.5, 3.4.5.6); + + for (i in addr_vector) + print mask_addr(addr_vector[i], 18); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_const_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_const_bro.include-doc_scripting_data_type_const_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_const_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_const_bro.include-doc_scripting_data_type_const_bro.btest new file mode 100644 index 0000000000..a42efd2b72 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_const_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_const_bro.include-doc_scripting_data_type_const_bro.btest @@ -0,0 +1,13 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_const.bro + +const port_list: table[port] of string &redef; + +redef port_list += { [6666/tcp] = "IRC"}; +redef port_list += { [80/tcp] = "WWW" }; + +event bro_init() + { + print port_list; + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_const_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_const_bro.include-doc_scripting_data_type_const_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_const_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_const_bro.include-doc_scripting_data_type_const_bro.btest new file mode 100644 index 0000000000..a42efd2b72 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_const_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_const_bro.include-doc_scripting_data_type_const_bro.btest @@ -0,0 +1,13 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_const.bro + +const port_list: table[port] of string &redef; + +redef port_list += { [6666/tcp] = "IRC"}; +redef port_list += { [80/tcp] = "WWW" }; + +event bro_init() + { + print port_list; + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_const_simple_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_const_simple_bro.include-doc_scripting_data_type_const_simple_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_const_simple_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_const_simple_bro.include-doc_scripting_data_type_const_simple_bro.btest new file mode 100644 index 0000000000..914f9dc0ec --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_const_simple_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_const_simple_bro.include-doc_scripting_data_type_const_simple_bro.btest @@ -0,0 +1,8 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_const_simple.bro + +@load base/protocols/http + +redef HTTP::default_capture_password = T; + diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_const_simple_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_const_simple_bro.include-doc_scripting_data_type_const_simple_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_const_simple_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_const_simple_bro.include-doc_scripting_data_type_const_simple_bro.btest new file mode 100644 index 0000000000..914f9dc0ec --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_const_simple_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_const_simple_bro.include-doc_scripting_data_type_const_simple_bro.btest @@ -0,0 +1,8 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_const_simple.bro + +@load base/protocols/http + +redef HTTP::default_capture_password = T; + diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_declaration_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_declaration_bro.include-doc_scripting_data_type_declaration_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_declaration_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_declaration_bro.include-doc_scripting_data_type_declaration_bro.btest new file mode 100644 index 0000000000..edd01615fc --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_declaration_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_declaration_bro.include-doc_scripting_data_type_declaration_bro.btest @@ -0,0 +1,13 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_declaration.bro + +event bro_init() + { + local a: int; + a = 10; + local b = 10; + + if ( a == b ) + print fmt("A: %d, B: %d", a, b); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_declaration_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_declaration_bro.include-doc_scripting_data_type_declaration_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_declaration_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_declaration_bro.include-doc_scripting_data_type_declaration_bro.btest new file mode 100644 index 0000000000..edd01615fc --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_declaration_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_declaration_bro.include-doc_scripting_data_type_declaration_bro.btest @@ -0,0 +1,13 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_declaration.bro + +event bro_init() + { + local a: int; + a = 10; + local b = 10; + + if ( a == b ) + print fmt("A: %d, B: %d", a, b); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_interval_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_interval_bro.include-doc_scripting_data_type_interval_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_interval_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_interval_bro.include-doc_scripting_data_type_interval_bro.btest new file mode 100644 index 0000000000..7954ed9ebe --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_interval_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_interval_bro.include-doc_scripting_data_type_interval_bro.btest @@ -0,0 +1,22 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_interval.bro + +# Store the time the previous connection was established. +global last_connection_time: time; + +# boolean value to indicate whether we have seen a previous connection. +global connection_seen: bool = F; + +event connection_established(c: connection) + { + local net_time: time = network_time(); + + print fmt("%s: New connection established from %s to %s", strftime("%Y/%M/%d %H:%m:%S", net_time), c$id$orig_h, c$id$resp_h); + + if ( connection_seen ) + print fmt(" Time since last connection: %s", net_time - last_connection_time); + + last_connection_time = net_time; + connection_seen = T; + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_interval_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_interval_bro.include-doc_scripting_data_type_interval_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_interval_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_interval_bro.include-doc_scripting_data_type_interval_bro.btest new file mode 100644 index 0000000000..7954ed9ebe --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_interval_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_interval_bro.include-doc_scripting_data_type_interval_bro.btest @@ -0,0 +1,22 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_interval.bro + +# Store the time the previous connection was established. +global last_connection_time: time; + +# boolean value to indicate whether we have seen a previous connection. +global connection_seen: bool = F; + +event connection_established(c: connection) + { + local net_time: time = network_time(); + + print fmt("%s: New connection established from %s to %s", strftime("%Y/%M/%d %H:%m:%S", net_time), c$id$orig_h, c$id$resp_h); + + if ( connection_seen ) + print fmt(" Time since last connection: %s", net_time - last_connection_time); + + last_connection_time = net_time; + connection_seen = T; + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_local_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_local_bro.include-doc_scripting_data_type_local_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_local_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_local_bro.include-doc_scripting_data_type_local_bro.btest new file mode 100644 index 0000000000..850ffeb905 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_local_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_local_bro.include-doc_scripting_data_type_local_bro.btest @@ -0,0 +1,15 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_local.bro + +function add_two(i: count): count + { + local added_two = i+2; + print fmt("i + 2 = %d", added_two); + return added_two; + } + +event bro_init() + { + local test = add_two(10); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_local_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_local_bro.include-doc_scripting_data_type_local_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_local_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_local_bro.include-doc_scripting_data_type_local_bro.btest new file mode 100644 index 0000000000..850ffeb905 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_local_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_local_bro.include-doc_scripting_data_type_local_bro.btest @@ -0,0 +1,15 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_local.bro + +function add_two(i: count): count + { + local added_two = i+2; + print fmt("i + 2 = %d", added_two); + return added_two; + } + +event bro_init() + { + local test = add_two(10); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_pattern_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_pattern_01_bro.include-doc_scripting_data_type_pattern_01_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_pattern_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_pattern_01_bro.include-doc_scripting_data_type_pattern_01_bro.btest new file mode 100644 index 0000000000..475fcd676e --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_pattern_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_pattern_01_bro.include-doc_scripting_data_type_pattern_01_bro.btest @@ -0,0 +1,17 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_pattern_01.bro + +event bro_init() + { + local test_string = "The quick brown fox jumped over the lazy dog."; + local test_pattern = /quick|lazy/; + + if ( test_pattern in test_string ) + { + local results = split(test_string, test_pattern); + print results[1]; + print results[2]; + print results[3]; + } + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_pattern_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_pattern_01_bro.include-doc_scripting_data_type_pattern_01_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_pattern_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_pattern_01_bro.include-doc_scripting_data_type_pattern_01_bro.btest new file mode 100644 index 0000000000..475fcd676e --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_pattern_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_pattern_01_bro.include-doc_scripting_data_type_pattern_01_bro.btest @@ -0,0 +1,17 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_pattern_01.bro + +event bro_init() + { + local test_string = "The quick brown fox jumped over the lazy dog."; + local test_pattern = /quick|lazy/; + + if ( test_pattern in test_string ) + { + local results = split(test_string, test_pattern); + print results[1]; + print results[2]; + print results[3]; + } + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_pattern_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_pattern_02_bro.include-doc_scripting_data_type_pattern_02_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_pattern_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_pattern_02_bro.include-doc_scripting_data_type_pattern_02_bro.btest new file mode 100644 index 0000000000..915e8612be --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_pattern_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_pattern_02_bro.include-doc_scripting_data_type_pattern_02_bro.btest @@ -0,0 +1,14 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_pattern_02.bro + +event bro_init() + { + local test_string = "equality"; + + local test_pattern = /equal/; + print fmt("%s and %s %s equal", test_string, test_pattern, test_pattern == test_string ? "are" : "are not"); + + test_pattern = /equality/; + print fmt("%s and %s %s equal", test_string, test_pattern, test_pattern == test_string ? "are" : "are not"); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_pattern_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_pattern_02_bro.include-doc_scripting_data_type_pattern_02_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_pattern_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_pattern_02_bro.include-doc_scripting_data_type_pattern_02_bro.btest new file mode 100644 index 0000000000..915e8612be --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_pattern_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_pattern_02_bro.include-doc_scripting_data_type_pattern_02_bro.btest @@ -0,0 +1,14 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_pattern_02.bro + +event bro_init() + { + local test_string = "equality"; + + local test_pattern = /equal/; + print fmt("%s and %s %s equal", test_string, test_pattern, test_pattern == test_string ? "are" : "are not"); + + test_pattern = /equality/; + print fmt("%s and %s %s equal", test_string, test_pattern, test_pattern == test_string ? "are" : "are not"); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_subnets_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_subnets_bro.include-doc_scripting_data_type_subnets_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_subnets_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_subnets_bro.include-doc_scripting_data_type_subnets_bro.btest new file mode 100644 index 0000000000..d2edfd5e9a --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_subnets_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_subnets_bro.include-doc_scripting_data_type_subnets_bro.btest @@ -0,0 +1,19 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_subnets.bro + +event bro_init() + { + local subnets = vector(172.16.0.0/20, 172.16.16.0/20, 172.16.32.0/20, 172.16.48.0/20); + local addresses = vector(172.16.4.56, 172.16.47.254, 172.16.22.45, 172.16.1.1); + + for ( a in addresses ) + { + for ( s in subnets ) + { + if ( addresses[a] in subnets[s] ) + print fmt("%s belongs to subnet %s", addresses[a], subnets[s]); + } + } + + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_subnets_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_subnets_bro.include-doc_scripting_data_type_subnets_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_subnets_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_subnets_bro.include-doc_scripting_data_type_subnets_bro.btest new file mode 100644 index 0000000000..d2edfd5e9a --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_subnets_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_subnets_bro.include-doc_scripting_data_type_subnets_bro.btest @@ -0,0 +1,19 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_subnets.bro + +event bro_init() + { + local subnets = vector(172.16.0.0/20, 172.16.16.0/20, 172.16.32.0/20, 172.16.48.0/20); + local addresses = vector(172.16.4.56, 172.16.47.254, 172.16.22.45, 172.16.1.1); + + for ( a in addresses ) + { + for ( s in subnets ) + { + if ( addresses[a] in subnets[s] ) + print fmt("%s belongs to subnet %s", addresses[a], subnets[s]); + } + } + + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_time_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_time_bro.include-doc_scripting_data_type_time_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_time_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_time_bro.include-doc_scripting_data_type_time_bro.btest new file mode 100644 index 0000000000..960d4ac9bd --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_time_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_time_bro.include-doc_scripting_data_type_time_bro.btest @@ -0,0 +1,8 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_time.bro + +event connection_established(c: connection) + { + print fmt("%s: New connection established from %s to %s\n", strftime("%Y/%M/%d %H:%m:%S", network_time()), c$id$orig_h, c$id$resp_h); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_time_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_time_bro.include-doc_scripting_data_type_time_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_time_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_time_bro.include-doc_scripting_data_type_time_bro.btest new file mode 100644 index 0000000000..960d4ac9bd --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_data_type_time_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_data_type_time_bro.include-doc_scripting_data_type_time_bro.btest @@ -0,0 +1,8 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_time.bro + +event connection_established(c: connection) + { + print fmt("%s: New connection established from %s to %s\n", strftime("%Y/%M/%d %H:%m:%S", network_time()), c$id$orig_h, c$id$resp_h); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_01_bro.include-doc_scripting_framework_logging_factorial_01_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_01_bro.include-doc_scripting_framework_logging_factorial_01_bro.btest new file mode 100644 index 0000000000..7a18c86509 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_01_bro.include-doc_scripting_framework_logging_factorial_01_bro.btest @@ -0,0 +1,23 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_logging_factorial_01.bro + +module Factor; + +function factorial(n: count): count + { + if ( n == 0 ) + return 1; + else + return ( n * factorial(n - 1) ); + } + +event bro_init() + { + local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + + for ( n in numbers ) + print fmt("%d", factorial(numbers[n])); + } + + diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_01_bro.include-doc_scripting_framework_logging_factorial_01_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_01_bro.include-doc_scripting_framework_logging_factorial_01_bro.btest new file mode 100644 index 0000000000..7a18c86509 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_01_bro.include-doc_scripting_framework_logging_factorial_01_bro.btest @@ -0,0 +1,23 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_logging_factorial_01.bro + +module Factor; + +function factorial(n: count): count + { + if ( n == 0 ) + return 1; + else + return ( n * factorial(n - 1) ); + } + +event bro_init() + { + local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + + for ( n in numbers ) + print fmt("%d", factorial(numbers[n])); + } + + diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_02_bro.include-doc_scripting_framework_logging_factorial_02_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_02_bro.include-doc_scripting_framework_logging_factorial_02_bro.btest new file mode 100644 index 0000000000..6966f7ea3c --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_02_bro.include-doc_scripting_framework_logging_factorial_02_bro.btest @@ -0,0 +1,36 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_logging_factorial_02.bro + +module Factor; + +export { + redef enum Log::ID += { LOG }; + + type Info: record { + num: count &log; + factorial_num: count &log; + }; + } + +function factorial(n: count): count + { + if ( n == 0 ) + return 1; + + else + return ( n * factorial(n - 1) ); + } + +event bro_init() + { + Log::create_stream(LOG, [$columns=Info]); + } + +event bro_done() + { + local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + for ( n in numbers ) + Log::write( Factor::LOG, [$num=numbers[n], + $factorial_num=factorial(numbers[n])]); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_02_bro.include-doc_scripting_framework_logging_factorial_02_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_02_bro.include-doc_scripting_framework_logging_factorial_02_bro.btest new file mode 100644 index 0000000000..6966f7ea3c --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_02_bro.include-doc_scripting_framework_logging_factorial_02_bro.btest @@ -0,0 +1,36 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_logging_factorial_02.bro + +module Factor; + +export { + redef enum Log::ID += { LOG }; + + type Info: record { + num: count &log; + factorial_num: count &log; + }; + } + +function factorial(n: count): count + { + if ( n == 0 ) + return 1; + + else + return ( n * factorial(n - 1) ); + } + +event bro_init() + { + Log::create_stream(LOG, [$columns=Info]); + } + +event bro_done() + { + local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + for ( n in numbers ) + Log::write( Factor::LOG, [$num=numbers[n], + $factorial_num=factorial(numbers[n])]); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_03_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_03_bro.include-doc_scripting_framework_logging_factorial_03_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_03_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_03_bro.include-doc_scripting_framework_logging_factorial_03_bro.btest new file mode 100644 index 0000000000..c3c16f8d6a --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_03_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_03_bro.include-doc_scripting_framework_logging_factorial_03_bro.btest @@ -0,0 +1,12 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_logging_factorial_03.bro + +event bro_init() + { + Log::create_stream(LOG, [$columns=Info]); + + local filter: Log::Filter = [$name="split-mod5s", $path_func=mod5]; + Log::add_filter(Factor::LOG, filter); + Log::remove_filter(Factor::LOG, "default"); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_03_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_03_bro.include-doc_scripting_framework_logging_factorial_03_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_03_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_03_bro.include-doc_scripting_framework_logging_factorial_03_bro.btest new file mode 100644 index 0000000000..c3c16f8d6a --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_03_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_03_bro.include-doc_scripting_framework_logging_factorial_03_bro.btest @@ -0,0 +1,12 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_logging_factorial_03.bro + +event bro_init() + { + Log::create_stream(LOG, [$columns=Info]); + + local filter: Log::Filter = [$name="split-mod5s", $path_func=mod5]; + Log::add_filter(Factor::LOG, filter); + Log::remove_filter(Factor::LOG, "default"); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_04_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_04_bro.include-doc_scripting_framework_logging_factorial_04_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_04_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_04_bro.include-doc_scripting_framework_logging_factorial_04_bro.btest new file mode 100644 index 0000000000..dbedb8e689 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_04_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_04_bro.include-doc_scripting_framework_logging_factorial_04_bro.btest @@ -0,0 +1,54 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_logging_factorial_04.bro + +module Factor; + +export { + redef enum Log::ID += { LOG }; + + type Info: record { + num: count &log; + factorial_num: count &log; + }; + + global log_factor: event(rec: Info); + } + +function factorial(n: count): count + { + if ( n == 0 ) + return 1; + + else + return (n * factorial(n - 1)); + } + +event bro_init() + { + Log::create_stream(LOG, [$columns=Info, $ev=log_factor]); + } + +event bro_done() + { + local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + for ( n in numbers ) + Log::write( Factor::LOG, [$num=numbers[n], + $factorial_num=factorial(numbers[n])]); + } + +function mod5(id: Log::ID, path: string, rec: Factor::Info) : string + { + if ( rec$factorial_num % 5 == 0 ) + return "factor-mod5"; + + else + return "factor-non5"; + } + +event bro_init() + { + local filter: Log::Filter = [$name="split-mod5s", $path_func=mod5]; + Log::add_filter(Factor::LOG, filter); + Log::remove_filter(Factor::LOG, "default"); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_04_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_04_bro.include-doc_scripting_framework_logging_factorial_04_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_04_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_04_bro.include-doc_scripting_framework_logging_factorial_04_bro.btest new file mode 100644 index 0000000000..dbedb8e689 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_logging_factorial_04_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_logging_factorial_04_bro.include-doc_scripting_framework_logging_factorial_04_bro.btest @@ -0,0 +1,54 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_logging_factorial_04.bro + +module Factor; + +export { + redef enum Log::ID += { LOG }; + + type Info: record { + num: count &log; + factorial_num: count &log; + }; + + global log_factor: event(rec: Info); + } + +function factorial(n: count): count + { + if ( n == 0 ) + return 1; + + else + return (n * factorial(n - 1)); + } + +event bro_init() + { + Log::create_stream(LOG, [$columns=Info, $ev=log_factor]); + } + +event bro_done() + { + local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + for ( n in numbers ) + Log::write( Factor::LOG, [$num=numbers[n], + $factorial_num=factorial(numbers[n])]); + } + +function mod5(id: Log::ID, path: string, rec: Factor::Info) : string + { + if ( rec$factorial_num % 5 == 0 ) + return "factor-mod5"; + + else + return "factor-non5"; + } + +event bro_init() + { + local filter: Log::Filter = [$name="split-mod5s", $path_func=mod5]; + Log::add_filter(Factor::LOG, filter); + Log::remove_filter(Factor::LOG, "default"); + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_hook_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_hook_01_bro.include-doc_scripting_framework_notice_hook_01_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_hook_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_hook_01_bro.include-doc_scripting_framework_notice_hook_01_bro.btest new file mode 100644 index 0000000000..70f8c70247 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_hook_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_hook_01_bro.include-doc_scripting_framework_notice_hook_01_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_notice_hook_01.bro + +@load policy/protocols/ssh/interesting-hostnames.bro + +hook Notice::policy(n: Notice::Info) + { + if ( n$note == SSH::Interesting_Hostname_Login ) + add n$actions[Notice::ACTION_EMAIL]; + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_hook_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_hook_01_bro.include-doc_scripting_framework_notice_hook_01_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_hook_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_hook_01_bro.include-doc_scripting_framework_notice_hook_01_bro.btest new file mode 100644 index 0000000000..70f8c70247 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_hook_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_hook_01_bro.include-doc_scripting_framework_notice_hook_01_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_notice_hook_01.bro + +@load policy/protocols/ssh/interesting-hostnames.bro + +hook Notice::policy(n: Notice::Info) + { + if ( n$note == SSH::Interesting_Hostname_Login ) + add n$actions[Notice::ACTION_EMAIL]; + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_hook_suppression_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_hook_suppression_01_bro.include-doc_scripting_framework_notice_hook_suppression_01_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_hook_suppression_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_hook_suppression_01_bro.include-doc_scripting_framework_notice_hook_suppression_01_bro.btest new file mode 100644 index 0000000000..522bef5930 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_hook_suppression_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_hook_suppression_01_bro.include-doc_scripting_framework_notice_hook_suppression_01_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_notice_hook_suppression_01.bro + +@load policy/protocols/ssl/expiring-certs.bro + +hook Notice::policy(n: Notice::Info) + { + if ( n$note == SSL::Certificate_Expires_Soon ) + n$suppress_for = 12hrs; + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_hook_suppression_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_hook_suppression_01_bro.include-doc_scripting_framework_notice_hook_suppression_01_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_hook_suppression_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_hook_suppression_01_bro.include-doc_scripting_framework_notice_hook_suppression_01_bro.btest new file mode 100644 index 0000000000..522bef5930 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_hook_suppression_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_hook_suppression_01_bro.include-doc_scripting_framework_notice_hook_suppression_01_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_notice_hook_suppression_01.bro + +@load policy/protocols/ssl/expiring-certs.bro + +hook Notice::policy(n: Notice::Info) + { + if ( n$note == SSL::Certificate_Expires_Soon ) + n$suppress_for = 12hrs; + } diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_shortcuts_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_shortcuts_01_bro.include-doc_scripting_framework_notice_shortcuts_01_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_shortcuts_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_shortcuts_01_bro.include-doc_scripting_framework_notice_shortcuts_01_bro.btest new file mode 100644 index 0000000000..95eed5f501 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_shortcuts_01_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_shortcuts_01_bro.include-doc_scripting_framework_notice_shortcuts_01_bro.btest @@ -0,0 +1,12 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_notice_shortcuts_01.bro + +@load policy/protocols/ssh/interesting-hostnames.bro +@load base/protocols/ssh/ + +redef Notice::emailed_types += { + SSH::Interesting_Hostname_Login, + SSH::Login +}; + diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_shortcuts_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_shortcuts_01_bro.include-doc_scripting_framework_notice_shortcuts_01_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_shortcuts_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_shortcuts_01_bro.include-doc_scripting_framework_notice_shortcuts_01_bro.btest new file mode 100644 index 0000000000..95eed5f501 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_shortcuts_01_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_shortcuts_01_bro.include-doc_scripting_framework_notice_shortcuts_01_bro.btest @@ -0,0 +1,12 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_notice_shortcuts_01.bro + +@load policy/protocols/ssh/interesting-hostnames.bro +@load base/protocols/ssh/ + +redef Notice::emailed_types += { + SSH::Interesting_Hostname_Login, + SSH::Login +}; + diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_shortcuts_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_shortcuts_02_bro.include-doc_scripting_framework_notice_shortcuts_02_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_shortcuts_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_shortcuts_02_bro.include-doc_scripting_framework_notice_shortcuts_02_bro.btest new file mode 100644 index 0000000000..8c12558d42 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_shortcuts_02_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_shortcuts_02_bro.include-doc_scripting_framework_notice_shortcuts_02_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_notice_shortcuts_02.bro + +@load policy/protocols/ssh/interesting-hostnames.bro +@load base/protocols/ssh/ + +redef Notice::type_suppression_intervals += { + [SSH::Interesting_Hostname_Login] = 1day, + [SSH::Login] = 12hrs, +}; diff --git a/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_shortcuts_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_shortcuts_02_bro.include-doc_scripting_framework_notice_shortcuts_02_bro.btest b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_shortcuts_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_shortcuts_02_bro.include-doc_scripting_framework_notice_shortcuts_02_bro.btest new file mode 100644 index 0000000000..8c12558d42 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-doc_scripting_framework_notice_shortcuts_02_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-doc_scripting_framework_notice_shortcuts_02_bro.include-doc_scripting_framework_notice_shortcuts_02_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_notice_shortcuts_02.bro + +@load policy/protocols/ssh/interesting-hostnames.bro +@load base/protocols/ssh/ + +redef Notice::type_suppression_intervals += { + [SSH::Interesting_Hostname_Login] = 1day, + [SSH::Login] = 12hrs, +}; diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_base_bif_event_bif_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_bif_event_bif_bro.include-scripts_base_bif_event_bif_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_base_bif_event_bif_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_bif_event_bif_bro.include-scripts_base_bif_event_bif_bro.btest new file mode 100644 index 0000000000..088d5d2368 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_base_bif_event_bif_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_bif_event_bif_bro.include-scripts_base_bif_event_bif_bro.btest @@ -0,0 +1,25 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- event.bif.bro + +## script-level cleanup that needs to be performed for every connection. This +## event is generated not only for TCP sessions but also for UDP and ICMP +## flows. +## +## +global connection_external: event(c: connection , tag: string ); + + +## Generated when a UDP session for a supported protocol has finished. Some of +## Bro's application-layer UDP analyzers flag the end of a session by raising +## Generated when a connection is seen that is marked as being expected. + + +global ipv6_ext_headers: event(c: connection , p: pkt_hdr ); +## their specifics differ slightly. Often, however, both will be raised for +## the same connection if some of its data is missing. We should eventually +## merge the two. +global ack_above_hole: event(c: connection ); + + +## diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_base_bif_event_bif_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_bif_event_bif_bro.include-scripts_base_bif_event_bif_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_base_bif_event_bif_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_bif_event_bif_bro.include-scripts_base_bif_event_bif_bro.btest new file mode 100644 index 0000000000..088d5d2368 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_base_bif_event_bif_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_bif_event_bif_bro.include-scripts_base_bif_event_bif_bro.btest @@ -0,0 +1,25 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- event.bif.bro + +## script-level cleanup that needs to be performed for every connection. This +## event is generated not only for TCP sessions but also for UDP and ICMP +## flows. +## +## +global connection_external: event(c: connection , tag: string ); + + +## Generated when a UDP session for a supported protocol has finished. Some of +## Bro's application-layer UDP analyzers flag the end of a session by raising +## Generated when a connection is seen that is marked as being expected. + + +global ipv6_ext_headers: event(c: connection , p: pkt_hdr ); +## their specifics differ slightly. Often, however, both will be raised for +## the same connection if some of its data is missing. We should eventually +## merge the two. +global ack_above_hole: event(c: connection ); + + +## diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.btest new file mode 100644 index 0000000000..c8e7eaab20 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.btest @@ -0,0 +1,30 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- Bro_DNS.events.bif.bro + +## Generated for DNS requests. For requests with multiple queries, this event +## is raised once for each. +## +## See `Wikipedia `__ for more +## information about the DNS protocol. Bro analyzes both UDP and TCP DNS +## sessions. +## +## c: The connection, which may be UDP or TCP depending on the type of the +## transport-layer session being analyzed. +## +## msg: The parsed DNS message header. +## +## query: The queried name. +## +## qtype: The queried resource record type. +## +## qclass: The queried resource record class. +## +## .. bro:see:: dns_AAAA_reply dns_A_reply dns_CNAME_reply dns_EDNS_addl +## dns_HINFO_reply dns_MX_reply dns_NS_reply dns_PTR_reply dns_SOA_reply +## dns_SRV_reply dns_TSIG_addl dns_TXT_reply dns_WKS_reply dns_end +## dns_full_request dns_mapping_altered dns_mapping_lost_name dns_mapping_new_name +## dns_mapping_unverified dns_mapping_valid dns_message dns_query_reply +## dns_rejected non_dns_request dns_max_queries dns_session_timeout dns_skip_addl +## dns_skip_all_addl dns_skip_all_auth dns_skip_auth +global dns_request: event(c: connection , msg: dns_msg , query: string , qtype: count , qclass: count ); diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.btest new file mode 100644 index 0000000000..c8e7eaab20 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.btest @@ -0,0 +1,30 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- Bro_DNS.events.bif.bro + +## Generated for DNS requests. For requests with multiple queries, this event +## is raised once for each. +## +## See `Wikipedia `__ for more +## information about the DNS protocol. Bro analyzes both UDP and TCP DNS +## sessions. +## +## c: The connection, which may be UDP or TCP depending on the type of the +## transport-layer session being analyzed. +## +## msg: The parsed DNS message header. +## +## query: The queried name. +## +## qtype: The queried resource record type. +## +## qclass: The queried resource record class. +## +## .. bro:see:: dns_AAAA_reply dns_A_reply dns_CNAME_reply dns_EDNS_addl +## dns_HINFO_reply dns_MX_reply dns_NS_reply dns_PTR_reply dns_SOA_reply +## dns_SRV_reply dns_TSIG_addl dns_TXT_reply dns_WKS_reply dns_end +## dns_full_request dns_mapping_altered dns_mapping_lost_name dns_mapping_new_name +## dns_mapping_unverified dns_mapping_valid dns_message dns_query_reply +## dns_rejected non_dns_request dns_max_queries dns_session_timeout dns_skip_addl +## dns_skip_all_addl dns_skip_all_auth dns_skip_auth +global dns_request: event(c: connection , msg: dns_msg , query: string , qtype: count , qclass: count ); diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_base_init-bare_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_init-bare_bro.include-scripts_base_init-bare_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_base_init-bare_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_init-bare_bro.include-scripts_base_init-bare_bro.btest new file mode 100644 index 0000000000..7d23c68ae4 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_base_init-bare_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_init-bare_bro.include-scripts_base_init-bare_bro.btest @@ -0,0 +1,7 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- init-bare.bro + +type string_array: table[count] of string; +type string_set: set[string]; +type addr_set: set[addr]; diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_base_init-bare_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_init-bare_bro.include-scripts_base_init-bare_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_base_init-bare_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_init-bare_bro.include-scripts_base_init-bare_bro.btest new file mode 100644 index 0000000000..7d23c68ae4 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_base_init-bare_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_init-bare_bro.include-scripts_base_init-bare_bro.btest @@ -0,0 +1,7 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- init-bare.bro + +type string_array: table[count] of string; +type string_set: set[string]; +type addr_set: set[addr]; diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_base_protocols_conn_main_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_protocols_conn_main_bro.include-scripts_base_protocols_conn_main_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_base_protocols_conn_main_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_protocols_conn_main_bro.include-scripts_base_protocols_conn_main_bro.btest new file mode 100644 index 0000000000..dc18745d57 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_base_protocols_conn_main_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_protocols_conn_main_bro.include-scripts_base_protocols_conn_main_bro.btest @@ -0,0 +1,28 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- main.bro + +module Conn; + +export { + ## The record type which contains column fields of the connection log. + type Info: record { + ts: time &log; + uid: string &log; + id: conn_id &log; + proto: transport_proto &log; + service: string &log &optional; + duration: interval &log &optional; + orig_bytes: count &log &optional; + resp_bytes: count &log &optional; + conn_state: string &log &optional; + local_orig: bool &log &optional; + missed_bytes: count &log &default=0; + history: string &log &optional; + orig_pkts: count &log &optional; + orig_ip_bytes: count &log &optional; + resp_pkts: count &log &optional; + resp_ip_bytes: count &log &optional; + tunnel_parents: set[string] &log; + }; +} diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_base_protocols_conn_main_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_protocols_conn_main_bro.include-scripts_base_protocols_conn_main_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_base_protocols_conn_main_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_protocols_conn_main_bro.include-scripts_base_protocols_conn_main_bro.btest new file mode 100644 index 0000000000..dc18745d57 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_base_protocols_conn_main_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_protocols_conn_main_bro.include-scripts_base_protocols_conn_main_bro.btest @@ -0,0 +1,28 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- main.bro + +module Conn; + +export { + ## The record type which contains column fields of the connection log. + type Info: record { + ts: time &log; + uid: string &log; + id: conn_id &log; + proto: transport_proto &log; + service: string &log &optional; + duration: interval &log &optional; + orig_bytes: count &log &optional; + resp_bytes: count &log &optional; + conn_state: string &log &optional; + local_orig: bool &log &optional; + missed_bytes: count &log &default=0; + history: string &log &optional; + orig_pkts: count &log &optional; + orig_ip_bytes: count &log &optional; + resp_pkts: count &log &optional; + resp_ip_bytes: count &log &optional; + tunnel_parents: set[string] &log; + }; +} diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_base_protocols_http_main_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_protocols_http_main_bro.include-scripts_base_protocols_http_main_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_base_protocols_http_main_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_protocols_http_main_bro.include-scripts_base_protocols_http_main_bro.btest new file mode 100644 index 0000000000..abe74112bd --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_base_protocols_http_main_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_base_protocols_http_main_bro.include-scripts_base_protocols_http_main_bro.btest @@ -0,0 +1,10 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- main.bro + +module HTTP; + +export { + ## This setting changes if passwords used in Basic-Auth are captured or not. + const default_capture_password = F &redef; +} diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_base_protocols_http_main_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_protocols_http_main_bro.include-scripts_base_protocols_http_main_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_base_protocols_http_main_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_protocols_http_main_bro.include-scripts_base_protocols_http_main_bro.btest new file mode 100644 index 0000000000..abe74112bd --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_base_protocols_http_main_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_base_protocols_http_main_bro.include-scripts_base_protocols_http_main_bro.btest @@ -0,0 +1,10 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- main.bro + +module HTTP; + +export { + ## This setting changes if passwords used in Basic-Auth are captured or not. + const default_capture_password = F &redef; +} diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro.include-scripts_policy_frameworks_files_detect-MHR_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro.include-scripts_policy_frameworks_files_detect-MHR_bro.btest new file mode 100644 index 0000000000..b73140fe8e --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro.include-scripts_policy_frameworks_files_detect-MHR_bro.btest @@ -0,0 +1,61 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- detect-MHR.bro + +##! Detect file downloads that have hash values matching files in Team +##! Cymru's Malware Hash Registry (http://www.team-cymru.org/Services/MHR/). + +@load base/frameworks/files +@load base/frameworks/notice +@load frameworks/files/hash-all-files + +module TeamCymruMalwareHashRegistry; + +export { + redef enum Notice::Type += { + ## The hash value of a file transferred over HTTP matched in the + ## malware hash registry. + Match + }; + + ## File types to attempt matching against the Malware Hash Registry. + const match_file_types = /application\/x-dosexec/ | + /application\/vnd.ms-cab-compressed/ | + /application\/pdf/ | + /application\/x-shockwave-flash/ | + /application\/x-java-applet/ | + /application\/jar/ | + /video\/mp4/ &redef; + + ## The malware hash registry runs each malware sample through several A/V engines. + ## Team Cymru returns a percentage to indicate how many A/V engines flagged the + ## sample as malicious. This threshold allows you to require a minimum detection + ## rate. + const notice_threshold = 10 &redef; +} + +event file_hash(f: fa_file, kind: string, hash: string) + { + if ( kind=="sha1" && match_file_types in f$mime_type ) + { + local hash_domain = fmt("%s.malware.hash.cymru.com", hash); + when ( local MHR_result = lookup_hostname_txt(hash_domain) ) + { + # Data is returned as " " + local MHR_answer = split1(MHR_result, / /); + if ( |MHR_answer| == 2 ) + { + local mhr_first_detected = double_to_time(to_double(MHR_answer[1])); + local mhr_detect_rate = to_count(MHR_answer[2]); + + local readable_first_detected = strftime("%Y-%m-%d %H:%M:%S", mhr_first_detected); + if ( mhr_detect_rate >= notice_threshold ) + { + local message = fmt("Malware Hash Registry Detection rate: %d%% Last seen: %s", mhr_detect_rate, readable_first_detected); + local virustotal_url = fmt("https://www.virustotal.com/en/file/%s/analysis/", hash); + NOTICE([$note=Match, $msg=message, $sub=virustotal_url, $f=f]); + } + } + } + } + } diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro.include-scripts_policy_frameworks_files_detect-MHR_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro.include-scripts_policy_frameworks_files_detect-MHR_bro.btest new file mode 100644 index 0000000000..b73140fe8e --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro.include-scripts_policy_frameworks_files_detect-MHR_bro.btest @@ -0,0 +1,61 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- detect-MHR.bro + +##! Detect file downloads that have hash values matching files in Team +##! Cymru's Malware Hash Registry (http://www.team-cymru.org/Services/MHR/). + +@load base/frameworks/files +@load base/frameworks/notice +@load frameworks/files/hash-all-files + +module TeamCymruMalwareHashRegistry; + +export { + redef enum Notice::Type += { + ## The hash value of a file transferred over HTTP matched in the + ## malware hash registry. + Match + }; + + ## File types to attempt matching against the Malware Hash Registry. + const match_file_types = /application\/x-dosexec/ | + /application\/vnd.ms-cab-compressed/ | + /application\/pdf/ | + /application\/x-shockwave-flash/ | + /application\/x-java-applet/ | + /application\/jar/ | + /video\/mp4/ &redef; + + ## The malware hash registry runs each malware sample through several A/V engines. + ## Team Cymru returns a percentage to indicate how many A/V engines flagged the + ## sample as malicious. This threshold allows you to require a minimum detection + ## rate. + const notice_threshold = 10 &redef; +} + +event file_hash(f: fa_file, kind: string, hash: string) + { + if ( kind=="sha1" && match_file_types in f$mime_type ) + { + local hash_domain = fmt("%s.malware.hash.cymru.com", hash); + when ( local MHR_result = lookup_hostname_txt(hash_domain) ) + { + # Data is returned as " " + local MHR_answer = split1(MHR_result, / /); + if ( |MHR_answer| == 2 ) + { + local mhr_first_detected = double_to_time(to_double(MHR_answer[1])); + local mhr_detect_rate = to_count(MHR_answer[2]); + + local readable_first_detected = strftime("%Y-%m-%d %H:%M:%S", mhr_first_detected); + if ( mhr_detect_rate >= notice_threshold ) + { + local message = fmt("Malware Hash Registry Detection rate: %d%% Last seen: %s", mhr_detect_rate, readable_first_detected); + local virustotal_url = fmt("https://www.virustotal.com/en/file/%s/analysis/", hash); + NOTICE([$note=Match, $msg=message, $sub=virustotal_url, $f=f]); + } + } + } + } + } diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@2/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@2.include-scripts_policy_frameworks_files_detect-MHR_bro@2.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@2/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@2.include-scripts_policy_frameworks_files_detect-MHR_bro@2.btest new file mode 100644 index 0000000000..51044b36d9 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@2/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@2.include-scripts_policy_frameworks_files_detect-MHR_bro@2.btest @@ -0,0 +1,9 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- detect-MHR.bro + + +module TeamCymruMalwareHashRegistry; + +export { + redef enum Notice::Type += { diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@2/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@2.include-scripts_policy_frameworks_files_detect-MHR_bro@2.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@2/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@2.include-scripts_policy_frameworks_files_detect-MHR_bro@2.btest new file mode 100644 index 0000000000..51044b36d9 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@2/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@2.include-scripts_policy_frameworks_files_detect-MHR_bro@2.btest @@ -0,0 +1,9 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- detect-MHR.bro + + +module TeamCymruMalwareHashRegistry; + +export { + redef enum Notice::Type += { diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@3/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@3.include-scripts_policy_frameworks_files_detect-MHR_bro@3.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@3/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@3.include-scripts_policy_frameworks_files_detect-MHR_bro@3.btest new file mode 100644 index 0000000000..37de8b6ff7 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@3/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@3.include-scripts_policy_frameworks_files_detect-MHR_bro@3.btest @@ -0,0 +1,17 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- detect-MHR.bro + + ## The hash value of a file transferred over HTTP matched in the + ## malware hash registry. + Match + }; + + ## File types to attempt matching against the Malware Hash Registry. + const match_file_types = /application\/x-dosexec/ | + /application\/vnd.ms-cab-compressed/ | + /application\/pdf/ | + /application\/x-shockwave-flash/ | + /application\/x-java-applet/ | + /application\/jar/ | + /video\/mp4/ &redef; diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@3/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@3.include-scripts_policy_frameworks_files_detect-MHR_bro@3.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@3/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@3.include-scripts_policy_frameworks_files_detect-MHR_bro@3.btest new file mode 100644 index 0000000000..37de8b6ff7 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@3/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@3.include-scripts_policy_frameworks_files_detect-MHR_bro@3.btest @@ -0,0 +1,17 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- detect-MHR.bro + + ## The hash value of a file transferred over HTTP matched in the + ## malware hash registry. + Match + }; + + ## File types to attempt matching against the Malware Hash Registry. + const match_file_types = /application\/x-dosexec/ | + /application\/vnd.ms-cab-compressed/ | + /application\/pdf/ | + /application\/x-shockwave-flash/ | + /application\/x-java-applet/ | + /application\/jar/ | + /video\/mp4/ &redef; diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@4/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@4.include-scripts_policy_frameworks_files_detect-MHR_bro@4.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@4/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@4.include-scripts_policy_frameworks_files_detect-MHR_bro@4.btest new file mode 100644 index 0000000000..19e9d2ee54 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@4/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@4.include-scripts_policy_frameworks_files_detect-MHR_bro@4.btest @@ -0,0 +1,23 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- detect-MHR.bro + + ## The malware hash registry runs each malware sample through several A/V engines. + ## Team Cymru returns a percentage to indicate how many A/V engines flagged the + ## sample as malicious. This threshold allows you to require a minimum detection + ## rate. + const notice_threshold = 10 &redef; +} + +event file_hash(f: fa_file, kind: string, hash: string) + { + if ( kind=="sha1" && match_file_types in f$mime_type ) + { + local hash_domain = fmt("%s.malware.hash.cymru.com", hash); + when ( local MHR_result = lookup_hostname_txt(hash_domain) ) + { + # Data is returned as " " + local MHR_answer = split1(MHR_result, / /); + if ( |MHR_answer| == 2 ) + { + local mhr_first_detected = double_to_time(to_double(MHR_answer[1])); diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@4/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@4.include-scripts_policy_frameworks_files_detect-MHR_bro@4.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@4/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@4.include-scripts_policy_frameworks_files_detect-MHR_bro@4.btest new file mode 100644 index 0000000000..19e9d2ee54 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@4/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_frameworks_files_detect-MHR_bro@4.include-scripts_policy_frameworks_files_detect-MHR_bro@4.btest @@ -0,0 +1,23 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- detect-MHR.bro + + ## The malware hash registry runs each malware sample through several A/V engines. + ## Team Cymru returns a percentage to indicate how many A/V engines flagged the + ## sample as malicious. This threshold allows you to require a minimum detection + ## rate. + const notice_threshold = 10 &redef; +} + +event file_hash(f: fa_file, kind: string, hash: string) + { + if ( kind=="sha1" && match_file_types in f$mime_type ) + { + local hash_domain = fmt("%s.malware.hash.cymru.com", hash); + when ( local MHR_result = lookup_hostname_txt(hash_domain) ) + { + # Data is returned as " " + local MHR_answer = split1(MHR_result, / /); + if ( |MHR_answer| == 2 ) + { + local mhr_first_detected = double_to_time(to_double(MHR_answer[1])); diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_conn_known-hosts_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_conn_known-hosts_bro.include-scripts_policy_protocols_conn_known-hosts_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_conn_known-hosts_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_conn_known-hosts_bro.include-scripts_policy_protocols_conn_known-hosts_bro.btest new file mode 100644 index 0000000000..9b0c50c3f4 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_conn_known-hosts_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_conn_known-hosts_bro.include-scripts_policy_protocols_conn_known-hosts_bro.btest @@ -0,0 +1,9 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- known-hosts.bro + +module Known; + +export { + global known_hosts: set[addr] &create_expire=1day &synchronized &redef; +} diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_conn_known-hosts_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_conn_known-hosts_bro.include-scripts_policy_protocols_conn_known-hosts_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_conn_known-hosts_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_conn_known-hosts_bro.include-scripts_policy_protocols_conn_known-hosts_bro.btest new file mode 100644 index 0000000000..9b0c50c3f4 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_conn_known-hosts_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_conn_known-hosts_bro.include-scripts_policy_protocols_conn_known-hosts_bro.btest @@ -0,0 +1,9 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- known-hosts.bro + +module Known; + +export { + global known_hosts: set[addr] &create_expire=1day &synchronized &redef; +} diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_ssh_interesting-hostnames_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_ssh_interesting-hostnames_bro.include-scripts_policy_protocols_ssh_interesting-hostnames_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_ssh_interesting-hostnames_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_ssh_interesting-hostnames_bro.include-scripts_policy_protocols_ssh_interesting-hostnames_bro.btest new file mode 100644 index 0000000000..73eec1a1b9 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_ssh_interesting-hostnames_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_ssh_interesting-hostnames_bro.include-scripts_policy_protocols_ssh_interesting-hostnames_bro.btest @@ -0,0 +1,50 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- interesting-hostnames.bro + +##! This script will generate a notice if an apparent SSH login originates +##! or heads to a host with a reverse hostname that looks suspicious. By +##! default, the regular expression to match "interesting" hostnames includes +##! names that are typically used for infrastructure hosts like nameservers, +##! mail servers, web servers and ftp servers. + +@load base/frameworks/notice + +module SSH; + +export { + redef enum Notice::Type += { + ## Generated if a login originates or responds with a host where the + ## reverse hostname lookup resolves to a name matched by the + ## :bro:id:`SSH::interesting_hostnames` regular expression. + Interesting_Hostname_Login, + }; + + ## Strange/bad host names to see successful SSH logins from or to. + const interesting_hostnames = + /^d?ns[0-9]*\./ | + /^smtp[0-9]*\./ | + /^mail[0-9]*\./ | + /^pop[0-9]*\./ | + /^imap[0-9]*\./ | + /^www[0-9]*\./ | + /^ftp[0-9]*\./ &redef; +} + +event SSH::heuristic_successful_login(c: connection) + { + for ( host in set(c$id$orig_h, c$id$resp_h) ) + { + when ( local hostname = lookup_addr(host) ) + { + if ( interesting_hostnames in hostname ) + { + NOTICE([$note=Interesting_Hostname_Login, + $msg=fmt("Possible SSH login involving a %s %s with an interesting hostname.", + Site::is_local_addr(host) ? "local" : "remote", + host == c$id$orig_h ? "client" : "server"), + $sub=hostname, $conn=c]); + } + } + } + } diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_ssh_interesting-hostnames_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_ssh_interesting-hostnames_bro.include-scripts_policy_protocols_ssh_interesting-hostnames_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_ssh_interesting-hostnames_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_ssh_interesting-hostnames_bro.include-scripts_policy_protocols_ssh_interesting-hostnames_bro.btest new file mode 100644 index 0000000000..73eec1a1b9 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_ssh_interesting-hostnames_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_ssh_interesting-hostnames_bro.include-scripts_policy_protocols_ssh_interesting-hostnames_bro.btest @@ -0,0 +1,50 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- interesting-hostnames.bro + +##! This script will generate a notice if an apparent SSH login originates +##! or heads to a host with a reverse hostname that looks suspicious. By +##! default, the regular expression to match "interesting" hostnames includes +##! names that are typically used for infrastructure hosts like nameservers, +##! mail servers, web servers and ftp servers. + +@load base/frameworks/notice + +module SSH; + +export { + redef enum Notice::Type += { + ## Generated if a login originates or responds with a host where the + ## reverse hostname lookup resolves to a name matched by the + ## :bro:id:`SSH::interesting_hostnames` regular expression. + Interesting_Hostname_Login, + }; + + ## Strange/bad host names to see successful SSH logins from or to. + const interesting_hostnames = + /^d?ns[0-9]*\./ | + /^smtp[0-9]*\./ | + /^mail[0-9]*\./ | + /^pop[0-9]*\./ | + /^imap[0-9]*\./ | + /^www[0-9]*\./ | + /^ftp[0-9]*\./ &redef; +} + +event SSH::heuristic_successful_login(c: connection) + { + for ( host in set(c$id$orig_h, c$id$resp_h) ) + { + when ( local hostname = lookup_addr(host) ) + { + if ( interesting_hostnames in hostname ) + { + NOTICE([$note=Interesting_Hostname_Login, + $msg=fmt("Possible SSH login involving a %s %s with an interesting hostname.", + Site::is_local_addr(host) ? "local" : "remote", + host == c$id$orig_h ? "client" : "server"), + $sub=hostname, $conn=c]); + } + } + } + } diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_ssl_expiring-certs_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_ssl_expiring-certs_bro.include-scripts_policy_protocols_ssl_expiring-certs_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_ssl_expiring-certs_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_ssl_expiring-certs_bro.include-scripts_policy_protocols_ssl_expiring-certs_bro.btest new file mode 100644 index 0000000000..16e2552545 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_ssl_expiring-certs_bro/.Users.robin.bro.docs.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_ssl_expiring-certs_bro.include-scripts_policy_protocols_ssl_expiring-certs_bro.btest @@ -0,0 +1,8 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- expiring-certs.bro + + NOTICE([$note=Certificate_Expires_Soon, + $msg=fmt("Certificate %s is going to expire at %T", cert$subject, cert$not_valid_after), + $conn=c, $suppress_for=1day, + $identifier=cat(c$id$resp_h, c$id$resp_p, c$ssl$cert_hash)]); diff --git a/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_ssl_expiring-certs_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_ssl_expiring-certs_bro.include-scripts_policy_protocols_ssl_expiring-certs_bro.btest b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_ssl_expiring-certs_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_ssl_expiring-certs_bro.include-scripts_policy_protocols_ssl_expiring-certs_bro.btest new file mode 100644 index 0000000000..16e2552545 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.include-scripts_policy_protocols_ssl_expiring-certs_bro/.home.robin.bro.master.testing.btest..tmp.doc.sphinx.include-scripts_policy_protocols_ssl_expiring-certs_bro.include-scripts_policy_protocols_ssl_expiring-certs_bro.btest @@ -0,0 +1,8 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- expiring-certs.bro + + NOTICE([$note=Certificate_Expires_Soon, + $msg=fmt("Certificate %s is going to expire at %T", cert$subject, cert$not_valid_after), + $conn=c, $suppress_for=1day, + $identifier=cat(c$id$resp_h, c$id$resp_p, c$ssl$cert_hash)]); diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#1 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#1 new file mode 100644 index 0000000000..26236f7e81 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#1 @@ -0,0 +1,26 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro -r wikipedia.trace + +.. code-block:: guess + :linenos: + + #separator \x09 + #set_separator , + #empty_field (empty) + #unset_field - + #path conn + #open 2013-08-31-22-50-20 + #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents + #types time string addr port addr port enum string interval count count string bool count string count count count count table[string] + 1300475167.096535 CXWv6p3arKYeMETxOg 141.142.220.202 5353 224.0.0.251 5353 udp dns - - - S0 - 0 D 1 73 0 0 (empty) + 1300475167.097012 CjhGID4nQcgTWjvg4c fe80::217:f2ff:fed7:cf65 5353 ff02::fb 5353 udp - - - - S0 - 0 D 1 199 0 0 (empty) + 1300475167.099816 CCvvfg3TEfuqmmG4bh 141.142.220.50 5353 224.0.0.251 5353 udp - - - - S0 - 0 D 1 179 0 0 (empty) + 1300475168.853899 CPbrpk1qSsw6ESzHV4 141.142.220.118 43927 141.142.2.2 53 udp dns 0.000435 38 89 SF - 0 Dd 1 66 1 117 (empty) + 1300475168.854378 C6pKV8GSxOnSLghOa 141.142.220.118 37676 141.142.2.2 53 udp dns 0.000420 52 99 SF - 0 Dd 1 80 1 127 (empty) + 1300475168.854837 CIPOse170MGiRM1Qf4 141.142.220.118 40526 141.142.2.2 53 udp dns 0.000392 38 183 SF - 0 Dd 1 66 1 211 (empty) + 1300475168.857956 CMXxB5GvmoxJFXdTa 141.142.220.118 32902 141.142.2.2 53 udp dns 0.000317 38 89 SF - 0 Dd 1 66 1 117 (empty) + [...] + diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#2 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#2 new file mode 100644 index 0000000000..b455b9a8e8 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#2 @@ -0,0 +1,17 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # cat conn.log | bro-cut id.orig_h id.orig_p id.resp_h duration + 141.142.220.202 5353 224.0.0.251 - + fe80::217:f2ff:fed7:cf65 5353 ff02::fb - + 141.142.220.50 5353 224.0.0.251 - + 141.142.220.118 43927 141.142.2.2 0.000435 + 141.142.220.118 37676 141.142.2.2 0.000420 + 141.142.220.118 40526 141.142.2.2 0.000392 + 141.142.220.118 32902 141.142.2.2 0.000317 + 141.142.220.118 59816 141.142.2.2 0.000343 + 141.142.220.118 59714 141.142.2.2 0.000375 + 141.142.220.118 58206 141.142.2.2 0.000339 + [...] + diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#3 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#3 new file mode 100644 index 0000000000..8492053976 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#3 @@ -0,0 +1,17 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # awk '/^[^#]/ {print $3, $4, $5, $6, $9}' conn.log + 141.142.220.202 5353 224.0.0.251 5353 - + fe80::217:f2ff:fed7:cf65 5353 ff02::fb 5353 - + 141.142.220.50 5353 224.0.0.251 5353 - + 141.142.220.118 43927 141.142.2.2 53 0.000435 + 141.142.220.118 37676 141.142.2.2 53 0.000420 + 141.142.220.118 40526 141.142.2.2 53 0.000392 + 141.142.220.118 32902 141.142.2.2 53 0.000317 + 141.142.220.118 59816 141.142.2.2 53 0.000343 + 141.142.220.118 59714 141.142.2.2 53 0.000375 + 141.142.220.118 58206 141.142.2.2 53 0.000339 + [...] + diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#4 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#4 new file mode 100644 index 0000000000..a41a275d32 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#4 @@ -0,0 +1,12 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro-cut -d ts uid host uri < http.log + 2011-03-18T19:06:08+0000 CRJuHdVW0XPVINV8a bits.wikimedia.org /skins-1.5/monobook/main.css + 2011-03-18T19:06:08+0000 CJ3xTn1c4Zw9TmAE05 upload.wikimedia.org /wikipedia/commons/6/63/Wikipedia-logo.png + 2011-03-18T19:06:08+0000 C7XEbhP654jzLoe3a upload.wikimedia.org /wikipedia/commons/thumb/b/bb/Wikipedia_wordmark.svg/174px-Wikipedia_wordmark.svg.png + 2011-03-18T19:06:08+0000 C3SfNE4BWaU4aSuwkc upload.wikimedia.org /wikipedia/commons/b/bd/Bookshelf-40x201_6.png + 2011-03-18T19:06:08+0000 CyAhVIzHqb7t7kv28 upload.wikimedia.org /wikipedia/commons/thumb/8/8a/Wikinews-logo.png/35px-Wikinews-logo.png + [...] + diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#5 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#5 new file mode 100644 index 0000000000..1d3caa52b9 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#5 @@ -0,0 +1,12 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro-cut -u ts uid host uri < http.log + 2011-03-18T19:06:08+0000 CRJuHdVW0XPVINV8a bits.wikimedia.org /skins-1.5/monobook/main.css + 2011-03-18T19:06:08+0000 CJ3xTn1c4Zw9TmAE05 upload.wikimedia.org /wikipedia/commons/6/63/Wikipedia-logo.png + 2011-03-18T19:06:08+0000 C7XEbhP654jzLoe3a upload.wikimedia.org /wikipedia/commons/thumb/b/bb/Wikipedia_wordmark.svg/174px-Wikipedia_wordmark.svg.png + 2011-03-18T19:06:08+0000 C3SfNE4BWaU4aSuwkc upload.wikimedia.org /wikipedia/commons/b/bd/Bookshelf-40x201_6.png + 2011-03-18T19:06:08+0000 CyAhVIzHqb7t7kv28 upload.wikimedia.org /wikipedia/commons/thumb/8/8a/Wikinews-logo.png/35px-Wikinews-logo.png + [...] + diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#6 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#6 new file mode 100644 index 0000000000..e53883ceb0 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#6 @@ -0,0 +1,12 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < http.log + 18-03-2011T19:06:08+0000 CRJuHdVW0XPVINV8a bits.wikimedia.org /skins-1.5/monobook/main.css + 18-03-2011T19:06:08+0000 CJ3xTn1c4Zw9TmAE05 upload.wikimedia.org /wikipedia/commons/6/63/Wikipedia-logo.png + 18-03-2011T19:06:08+0000 C7XEbhP654jzLoe3a upload.wikimedia.org /wikipedia/commons/thumb/b/bb/Wikipedia_wordmark.svg/174px-Wikipedia_wordmark.svg.png + 18-03-2011T19:06:08+0000 C3SfNE4BWaU4aSuwkc upload.wikimedia.org /wikipedia/commons/b/bd/Bookshelf-40x201_6.png + 18-03-2011T19:06:08+0000 CyAhVIzHqb7t7kv28 upload.wikimedia.org /wikipedia/commons/thumb/8/8a/Wikinews-logo.png/35px-Wikinews-logo.png + [...] + diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#7 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#7 new file mode 100644 index 0000000000..4ec7817003 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#7 @@ -0,0 +1,11 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # cat conn.log | bro-cut uid resp_bytes | sort -nrk2 | head -5 + CyAhVIzHqb7t7kv28 734 + CkDsfG2YIeWJmXWNWj 734 + CJ3xTn1c4Zw9TmAE05 734 + C3SfNE4BWaU4aSuwkc 734 + CzA03V1VcgagLjnO92 733 + diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#8 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#8 new file mode 100644 index 0000000000..72c290affb --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#8 @@ -0,0 +1,7 @@ +.. code-block:: none + :linenos: + :emphasize-lines: 1,1 + + # cat http.log | bro-cut uid id.resp_h method status_code host uri | grep VW0XPVINV8a + CRJuHdVW0XPVINV8a 208.80.152.118 GET 304 bits.wikimedia.org /skins-1.5/monobook/main.css + diff --git a/testing/btest/Traces/workshop_2011_browse.trace b/testing/btest/Traces/workshop_2011_browse.trace new file mode 100644 index 0000000000..74d2f9c21f Binary files /dev/null and b/testing/btest/Traces/workshop_2011_browse.trace differ diff --git a/testing/btest/btest.cfg b/testing/btest/btest.cfg index cc06eb4d5d..39b64f7c09 100644 --- a/testing/btest/btest.cfg +++ b/testing/btest/btest.cfg @@ -4,6 +4,7 @@ TmpDir = %(testbase)s/.tmp BaselineDir = %(testbase)s/Baseline IgnoreDirs = .svn CVS .tmp IgnoreFiles = *.tmp *.swp #* *.trace .DS_Store +PartFinalizer = btest-diff-rst [environment] BROPATH=`bash -c %(testbase)s/../../build/bro-path-dev` @@ -12,12 +13,14 @@ BRO_SEED_FILE=%(testbase)s/random.seed TZ=UTC LC_ALL=C BTEST_PATH=%(testbase)s/../../aux/btest -PATH=%(testbase)s/../../build/src:%(testbase)s/../scripts:%(testbase)s/../../aux/btest:%(testbase)s/../../aux/bro-aux/bro-cut:%(default_path)s +PATH=%(testbase)s/../../build/src:%(testbase)s/../scripts:%(testbase)s/../../aux/btest:%(testbase)s/../../aux/bro-aux/bro-cut:%(default_path)s:%(testbase)s/../../aux/btest/sphinx:%(default_path)s TRACES=%(testbase)s/Traces FILES=%(testbase)s/Files SCRIPTS=%(testbase)s/../scripts +DOC_ROOT=%(testbase)s/../../doc DIST=%(testbase)s/../.. BUILD=%(testbase)s/../../build -TEST_DIFF_CANONIFIER=$SCRIPTS/diff-canonifier +TEST_DIFF_CANONIFIER=%(testbase)s/../scripts/diff-canonifier TMPDIR=%(testbase)s/.tmp BRO_PROFILER_FILE=%(testbase)s/.tmp/script-coverage.XXXXXX +BTEST_RST_FILTER=$SCRIPTS/rst-filter diff --git a/testing/btest/doc/sphinx/connection-record-01.btest b/testing/btest/doc/sphinx/connection-record-01.btest new file mode 100644 index 0000000000..b379fb4fbe --- /dev/null +++ b/testing/btest/doc/sphinx/connection-record-01.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro -b -r ${TRACES}/dns-session.trace ${DOC_ROOT}/scripting/connection_record_01.bro diff --git a/testing/btest/doc/sphinx/connection-record-02.btest b/testing/btest/doc/sphinx/connection-record-02.btest new file mode 100644 index 0000000000..292503e12c --- /dev/null +++ b/testing/btest/doc/sphinx/connection-record-02.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro -b -r ${TRACES}/dns-session.trace ${DOC_ROOT}/scripting/connection_record_02.bro diff --git a/testing/btest/doc/sphinx/data_struct_record_01.btest b/testing/btest/doc/sphinx/data_struct_record_01.btest new file mode 100644 index 0000000000..ae52b9a24d --- /dev/null +++ b/testing/btest/doc/sphinx/data_struct_record_01.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_struct_record_01.bro diff --git a/testing/btest/doc/sphinx/data_struct_record_02.btest b/testing/btest/doc/sphinx/data_struct_record_02.btest new file mode 100644 index 0000000000..d978224e3b --- /dev/null +++ b/testing/btest/doc/sphinx/data_struct_record_02.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_struct_record_02.bro diff --git a/testing/btest/doc/sphinx/data_struct_set_declaration.btest b/testing/btest/doc/sphinx/data_struct_set_declaration.btest new file mode 100644 index 0000000000..2812479dff --- /dev/null +++ b/testing/btest/doc/sphinx/data_struct_set_declaration.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_struct_set_declaration.bro diff --git a/testing/btest/doc/sphinx/data_struct_table_complex.btest b/testing/btest/doc/sphinx/data_struct_table_complex.btest new file mode 100644 index 0000000000..99117130f8 --- /dev/null +++ b/testing/btest/doc/sphinx/data_struct_table_complex.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro -b ${DOC_ROOT}/scripting/data_struct_table_complex.bro diff --git a/testing/btest/doc/sphinx/data_struct_table_declaration.btest b/testing/btest/doc/sphinx/data_struct_table_declaration.btest new file mode 100644 index 0000000000..ac83e84e45 --- /dev/null +++ b/testing/btest/doc/sphinx/data_struct_table_declaration.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_struct_table_declaration.bro diff --git a/testing/btest/doc/sphinx/data_struct_vector_declaration.btest b/testing/btest/doc/sphinx/data_struct_vector_declaration.btest new file mode 100644 index 0000000000..b017f84aed --- /dev/null +++ b/testing/btest/doc/sphinx/data_struct_vector_declaration.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_struct_vector_declaration.bro diff --git a/testing/btest/doc/sphinx/data_struct_vector_iter.btest b/testing/btest/doc/sphinx/data_struct_vector_iter.btest new file mode 100644 index 0000000000..f5e6c2a0ba --- /dev/null +++ b/testing/btest/doc/sphinx/data_struct_vector_iter.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro -b ${DOC_ROOT}/scripting/data_struct_vector_iter.bro diff --git a/testing/btest/doc/sphinx/data_type_const.bro.btest b/testing/btest/doc/sphinx/data_type_const.bro.btest new file mode 100644 index 0000000000..903fed3fbf --- /dev/null +++ b/testing/btest/doc/sphinx/data_type_const.bro.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro -b ${DOC_ROOT}/scripting/data_type_const.bro diff --git a/testing/btest/doc/sphinx/data_type_interval.btest b/testing/btest/doc/sphinx/data_type_interval.btest new file mode 100644 index 0000000000..0c1baa1372 --- /dev/null +++ b/testing/btest/doc/sphinx/data_type_interval.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro -r ${TRACES}/wikipedia.trace ${DOC_ROOT}/scripting/data_type_interval.bro diff --git a/testing/btest/doc/sphinx/data_type_pattern.btest b/testing/btest/doc/sphinx/data_type_pattern.btest new file mode 100644 index 0000000000..30ba3c6fe0 --- /dev/null +++ b/testing/btest/doc/sphinx/data_type_pattern.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_type_pattern_01.bro diff --git a/testing/btest/doc/sphinx/data_type_pattern_02.btest b/testing/btest/doc/sphinx/data_type_pattern_02.btest new file mode 100644 index 0000000000..b197cef792 --- /dev/null +++ b/testing/btest/doc/sphinx/data_type_pattern_02.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_type_pattern_02.bro diff --git a/testing/btest/doc/sphinx/data_type_subnets.btest b/testing/btest/doc/sphinx/data_type_subnets.btest new file mode 100644 index 0000000000..c78fe8c71c --- /dev/null +++ b/testing/btest/doc/sphinx/data_type_subnets.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/data_type_subnets.bro diff --git a/testing/btest/doc/sphinx/data_type_time.btest b/testing/btest/doc/sphinx/data_type_time.btest new file mode 100644 index 0000000000..29dee6002f --- /dev/null +++ b/testing/btest/doc/sphinx/data_type_time.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro -r ${TRACES}/wikipedia.trace ${DOC_ROOT}/scripting/data_type_time.bro diff --git a/testing/btest/doc/sphinx/framework_logging_factorial-2.btest b/testing/btest/doc/sphinx/framework_logging_factorial-2.btest new file mode 100644 index 0000000000..c0f1262217 --- /dev/null +++ b/testing/btest/doc/sphinx/framework_logging_factorial-2.btest @@ -0,0 +1,2 @@ +@TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/framework_logging_factorial_02.bro +@TEST-EXEC: btest-rst-include factor.log diff --git a/testing/btest/doc/sphinx/framework_logging_factorial-3.btest b/testing/btest/doc/sphinx/framework_logging_factorial-3.btest new file mode 100644 index 0000000000..75f2054661 --- /dev/null +++ b/testing/btest/doc/sphinx/framework_logging_factorial-3.btest @@ -0,0 +1,2 @@ +@TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/framework_logging_factorial_03.bro +@TEST-EXEC: btest-rst-include factor-mod5.log diff --git a/testing/btest/doc/sphinx/framework_logging_factorial.btest b/testing/btest/doc/sphinx/framework_logging_factorial.btest new file mode 100644 index 0000000000..798b821f87 --- /dev/null +++ b/testing/btest/doc/sphinx/framework_logging_factorial.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd bro ${DOC_ROOT}/scripting/framework_logging_factorial_01.bro diff --git a/testing/btest/doc/sphinx/include-doc_scripting_connection_record_02_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_connection_record_02_bro.btest new file mode 100644 index 0000000000..43a6ebc554 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_connection_record_02_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- connection_record_02.bro + +@load base/protocols/conn +@load base/protocols/dns + +event connection_state_remove(c: connection) + { + print c; + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_connection_record_02_bro@2.btest b/testing/btest/doc/sphinx/include-doc_scripting_connection_record_02_bro@2.btest new file mode 100644 index 0000000000..43a6ebc554 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_connection_record_02_bro@2.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- connection_record_02.bro + +@load base/protocols/conn +@load base/protocols/dns + +event connection_state_remove(c: connection) + { + print c; + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_struct_record_01_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_record_01_bro.btest new file mode 100644 index 0000000000..82f1b1f56c --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_record_01_bro.btest @@ -0,0 +1,26 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_record_01.bro + +type Service: record { + name: string; + ports: set[port]; + rfc: count; +}; + +function print_service(serv: Service): string + { + print fmt("Service: %s(RFC%d)",serv$name, serv$rfc); + + for ( p in serv$ports ) + print fmt(" port: %s", p); + } + +event bro_init() + { + local dns: Service = [$name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035]; + local http: Service = [$name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616]; + + print_service(dns); + print_service(http); + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_struct_record_02_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_record_02_bro.btest new file mode 100644 index 0000000000..05b5b9d50c --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_record_02_bro.btest @@ -0,0 +1,45 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_record_02.bro + +type Service: record { + name: string; + ports: set[port]; + rfc: count; + }; + +type System: record { + name: string; + services: set[Service]; + }; + +function print_service(serv: Service): string + { + print fmt(" Service: %s(RFC%d)",serv$name, serv$rfc); + + for ( p in serv$ports ) + print fmt(" port: %s", p); + } + +function print_system(sys: System): string + { + print fmt("System: %s", sys$name); + + for ( s in sys$services ) + print_service(s); + } + +event bro_init() + { + local server01: System; + server01$name = "morlock"; + add server01$services[[ $name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035]]; + add server01$services[[ $name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616]]; + print_system(server01); + + + # local dns: Service = [ $name="dns", $ports=set(53/udp, 53/tcp), $rfc=1035]; + # local http: Service = [ $name="http", $ports=set(80/tcp, 8080/tcp), $rfc=2616]; + # print_service(dns); + # print_service(http); + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_struct_set_declaration_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_set_declaration_bro.btest new file mode 100644 index 0000000000..b659563ac3 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_set_declaration_bro.btest @@ -0,0 +1,9 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_set_declaration.bro + +event bro_init() + { + local ssl_ports: set[port]; + local non_ssl_ports = set( 23/tcp, 80/tcp, 143/tcp, 25/tcp ); + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_struct_set_declaration_bro@2.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_set_declaration_bro@2.btest new file mode 100644 index 0000000000..5fa1f36475 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_set_declaration_bro@2.btest @@ -0,0 +1,9 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_set_declaration.bro + + for ( i in ssl_ports ) + print fmt("SSL Port: %s", i); + + for ( i in non_ssl_ports ) + print fmt("Non-SSL Port: %s", i); diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_struct_set_declaration_bro@3.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_set_declaration_bro@3.btest new file mode 100644 index 0000000000..48d33b52e5 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_set_declaration_bro@3.btest @@ -0,0 +1,7 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_set_declaration.bro + + # Check for SMTPS + if ( 587/tcp !in ssl_ports ) + add ssl_ports[587/tcp]; diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_struct_set_declaration_bro@4.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_set_declaration_bro@4.btest new file mode 100644 index 0000000000..19ddfe81d5 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_set_declaration_bro@4.btest @@ -0,0 +1,26 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_set_declaration.bro + +event bro_init() + { + local ssl_ports: set[port]; + local non_ssl_ports = set( 23/tcp, 80/tcp, 143/tcp, 25/tcp ); + + # SSH + add ssl_ports[22/tcp]; + # HTTPS + add ssl_ports[443/tcp]; + # IMAPS + add ssl_ports[993/tcp]; + + # Check for SMTPS + if ( 587/tcp !in ssl_ports ) + add ssl_ports[587/tcp]; + + for ( i in ssl_ports ) + print fmt("SSL Port: %s", i); + + for ( i in non_ssl_ports ) + print fmt("Non-SSL Port: %s", i); + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_struct_table_complex_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_table_complex_bro.btest new file mode 100644 index 0000000000..a6a1d7a63a --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_table_complex_bro.btest @@ -0,0 +1,17 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_table_complex.bro + +event bro_init() + { + local samurai_flicks: table[string, string, count, string] of string; + + samurai_flicks["Kihachi Okamoto", "Toho", 1968, "Tatsuya Nakadai"] = "Kiru"; + samurai_flicks["Hideo Gosha", "Fuji", 1969, "Tatsuya Nakadai"] = "Goyokin"; + samurai_flicks["Masaki Kobayashi", "Shochiku Eiga", 1962, "Tatsuya Nakadai" ] = "Harakiri"; + samurai_flicks["Yoji Yamada", "Eisei Gekijo", 2002, "Hiroyuki Sanada" ] = "Tasogare Seibei"; + + for ( [d, s, y, a] in samurai_flicks ) + print fmt("%s was released in %d by %s studios, directed by %s and starring %s", samurai_flicks[d, s, y, a], y, s, d, a); + } + diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_struct_table_declaration_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_table_declaration_bro.btest new file mode 100644 index 0000000000..79fe7660ae --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_table_declaration_bro.btest @@ -0,0 +1,17 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_table_declaration.bro + +event bro_init() + { + local ssl_services: table[string] of port; + + ssl_services = table(["SSH"] = 22/tcp, ["HTTPS"] = 443/tcp); + ssl_services["IMAPS"] = 993/tcp; + + if ( "SMTPS" !in ssl_services ) + ssl_services["SMTPS"] = 587/tcp; + + for ( k in ssl_services ) + print fmt("Service Name: %s - Common Port: %s", k, ssl_services[k]); + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_struct_vector_declaration_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_vector_declaration_bro.btest new file mode 100644 index 0000000000..6b8a81a58f --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_vector_declaration_bro.btest @@ -0,0 +1,19 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_vector_declaration.bro + +event bro_init() + { + local v1: vector of count; + local v2 = vector(1, 2, 3, 4); + + v1[|v1|] = 1; + v1[|v1|] = 2; + v1[|v1|] = 3; + v1[|v1|] = 4; + + print fmt("contents of v1: %s", v1); + print fmt("length of v1: %d", |v1|); + print fmt("contents of v1: %s", v2); + print fmt("length of v2: %d", |v2|); + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_struct_vector_iter_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_vector_iter_bro.btest new file mode 100644 index 0000000000..a28522fb64 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_struct_vector_iter_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_struct_vector_iter.bro + +event bro_init() + { + local addr_vector: vector of addr = vector(1.2.3.4, 2.3.4.5, 3.4.5.6); + + for (i in addr_vector) + print mask_addr(addr_vector[i], 18); + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_type_const_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_type_const_bro.btest new file mode 100644 index 0000000000..a42efd2b72 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_type_const_bro.btest @@ -0,0 +1,13 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_const.bro + +const port_list: table[port] of string &redef; + +redef port_list += { [6666/tcp] = "IRC"}; +redef port_list += { [80/tcp] = "WWW" }; + +event bro_init() + { + print port_list; + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_type_const_simple_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_type_const_simple_bro.btest new file mode 100644 index 0000000000..914f9dc0ec --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_type_const_simple_bro.btest @@ -0,0 +1,8 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_const_simple.bro + +@load base/protocols/http + +redef HTTP::default_capture_password = T; + diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_type_declaration_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_type_declaration_bro.btest new file mode 100644 index 0000000000..edd01615fc --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_type_declaration_bro.btest @@ -0,0 +1,13 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_declaration.bro + +event bro_init() + { + local a: int; + a = 10; + local b = 10; + + if ( a == b ) + print fmt("A: %d, B: %d", a, b); + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_type_interval_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_type_interval_bro.btest new file mode 100644 index 0000000000..7954ed9ebe --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_type_interval_bro.btest @@ -0,0 +1,22 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_interval.bro + +# Store the time the previous connection was established. +global last_connection_time: time; + +# boolean value to indicate whether we have seen a previous connection. +global connection_seen: bool = F; + +event connection_established(c: connection) + { + local net_time: time = network_time(); + + print fmt("%s: New connection established from %s to %s", strftime("%Y/%M/%d %H:%m:%S", net_time), c$id$orig_h, c$id$resp_h); + + if ( connection_seen ) + print fmt(" Time since last connection: %s", net_time - last_connection_time); + + last_connection_time = net_time; + connection_seen = T; + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_type_local_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_type_local_bro.btest new file mode 100644 index 0000000000..850ffeb905 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_type_local_bro.btest @@ -0,0 +1,15 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_local.bro + +function add_two(i: count): count + { + local added_two = i+2; + print fmt("i + 2 = %d", added_two); + return added_two; + } + +event bro_init() + { + local test = add_two(10); + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_type_pattern_01_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_type_pattern_01_bro.btest new file mode 100644 index 0000000000..475fcd676e --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_type_pattern_01_bro.btest @@ -0,0 +1,17 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_pattern_01.bro + +event bro_init() + { + local test_string = "The quick brown fox jumped over the lazy dog."; + local test_pattern = /quick|lazy/; + + if ( test_pattern in test_string ) + { + local results = split(test_string, test_pattern); + print results[1]; + print results[2]; + print results[3]; + } + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_type_pattern_02_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_type_pattern_02_bro.btest new file mode 100644 index 0000000000..915e8612be --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_type_pattern_02_bro.btest @@ -0,0 +1,14 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_pattern_02.bro + +event bro_init() + { + local test_string = "equality"; + + local test_pattern = /equal/; + print fmt("%s and %s %s equal", test_string, test_pattern, test_pattern == test_string ? "are" : "are not"); + + test_pattern = /equality/; + print fmt("%s and %s %s equal", test_string, test_pattern, test_pattern == test_string ? "are" : "are not"); + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_type_subnets_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_type_subnets_bro.btest new file mode 100644 index 0000000000..d2edfd5e9a --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_type_subnets_bro.btest @@ -0,0 +1,19 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_subnets.bro + +event bro_init() + { + local subnets = vector(172.16.0.0/20, 172.16.16.0/20, 172.16.32.0/20, 172.16.48.0/20); + local addresses = vector(172.16.4.56, 172.16.47.254, 172.16.22.45, 172.16.1.1); + + for ( a in addresses ) + { + for ( s in subnets ) + { + if ( addresses[a] in subnets[s] ) + print fmt("%s belongs to subnet %s", addresses[a], subnets[s]); + } + } + + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_data_type_time_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_data_type_time_bro.btest new file mode 100644 index 0000000000..960d4ac9bd --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_data_type_time_bro.btest @@ -0,0 +1,8 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- data_type_time.bro + +event connection_established(c: connection) + { + print fmt("%s: New connection established from %s to %s\n", strftime("%Y/%M/%d %H:%m:%S", network_time()), c$id$orig_h, c$id$resp_h); + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_framework_logging_factorial_01_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_framework_logging_factorial_01_bro.btest new file mode 100644 index 0000000000..7a18c86509 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_framework_logging_factorial_01_bro.btest @@ -0,0 +1,23 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_logging_factorial_01.bro + +module Factor; + +function factorial(n: count): count + { + if ( n == 0 ) + return 1; + else + return ( n * factorial(n - 1) ); + } + +event bro_init() + { + local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + + for ( n in numbers ) + print fmt("%d", factorial(numbers[n])); + } + + diff --git a/testing/btest/doc/sphinx/include-doc_scripting_framework_logging_factorial_02_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_framework_logging_factorial_02_bro.btest new file mode 100644 index 0000000000..6966f7ea3c --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_framework_logging_factorial_02_bro.btest @@ -0,0 +1,36 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_logging_factorial_02.bro + +module Factor; + +export { + redef enum Log::ID += { LOG }; + + type Info: record { + num: count &log; + factorial_num: count &log; + }; + } + +function factorial(n: count): count + { + if ( n == 0 ) + return 1; + + else + return ( n * factorial(n - 1) ); + } + +event bro_init() + { + Log::create_stream(LOG, [$columns=Info]); + } + +event bro_done() + { + local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + for ( n in numbers ) + Log::write( Factor::LOG, [$num=numbers[n], + $factorial_num=factorial(numbers[n])]); + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_framework_logging_factorial_03_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_framework_logging_factorial_03_bro.btest new file mode 100644 index 0000000000..c3c16f8d6a --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_framework_logging_factorial_03_bro.btest @@ -0,0 +1,12 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_logging_factorial_03.bro + +event bro_init() + { + Log::create_stream(LOG, [$columns=Info]); + + local filter: Log::Filter = [$name="split-mod5s", $path_func=mod5]; + Log::add_filter(Factor::LOG, filter); + Log::remove_filter(Factor::LOG, "default"); + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_framework_logging_factorial_04_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_framework_logging_factorial_04_bro.btest new file mode 100644 index 0000000000..dbedb8e689 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_framework_logging_factorial_04_bro.btest @@ -0,0 +1,54 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_logging_factorial_04.bro + +module Factor; + +export { + redef enum Log::ID += { LOG }; + + type Info: record { + num: count &log; + factorial_num: count &log; + }; + + global log_factor: event(rec: Info); + } + +function factorial(n: count): count + { + if ( n == 0 ) + return 1; + + else + return (n * factorial(n - 1)); + } + +event bro_init() + { + Log::create_stream(LOG, [$columns=Info, $ev=log_factor]); + } + +event bro_done() + { + local numbers: vector of count = vector(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); + for ( n in numbers ) + Log::write( Factor::LOG, [$num=numbers[n], + $factorial_num=factorial(numbers[n])]); + } + +function mod5(id: Log::ID, path: string, rec: Factor::Info) : string + { + if ( rec$factorial_num % 5 == 0 ) + return "factor-mod5"; + + else + return "factor-non5"; + } + +event bro_init() + { + local filter: Log::Filter = [$name="split-mod5s", $path_func=mod5]; + Log::add_filter(Factor::LOG, filter); + Log::remove_filter(Factor::LOG, "default"); + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_framework_notice_hook_01_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_framework_notice_hook_01_bro.btest new file mode 100644 index 0000000000..70f8c70247 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_framework_notice_hook_01_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_notice_hook_01.bro + +@load policy/protocols/ssh/interesting-hostnames.bro + +hook Notice::policy(n: Notice::Info) + { + if ( n$note == SSH::Interesting_Hostname_Login ) + add n$actions[Notice::ACTION_EMAIL]; + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_framework_notice_hook_suppression_01_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_framework_notice_hook_suppression_01_bro.btest new file mode 100644 index 0000000000..522bef5930 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_framework_notice_hook_suppression_01_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_notice_hook_suppression_01.bro + +@load policy/protocols/ssl/expiring-certs.bro + +hook Notice::policy(n: Notice::Info) + { + if ( n$note == SSL::Certificate_Expires_Soon ) + n$suppress_for = 12hrs; + } diff --git a/testing/btest/doc/sphinx/include-doc_scripting_framework_notice_shortcuts_01_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_framework_notice_shortcuts_01_bro.btest new file mode 100644 index 0000000000..95eed5f501 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_framework_notice_shortcuts_01_bro.btest @@ -0,0 +1,12 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_notice_shortcuts_01.bro + +@load policy/protocols/ssh/interesting-hostnames.bro +@load base/protocols/ssh/ + +redef Notice::emailed_types += { + SSH::Interesting_Hostname_Login, + SSH::Login +}; + diff --git a/testing/btest/doc/sphinx/include-doc_scripting_framework_notice_shortcuts_02_bro.btest b/testing/btest/doc/sphinx/include-doc_scripting_framework_notice_shortcuts_02_bro.btest new file mode 100644 index 0000000000..8c12558d42 --- /dev/null +++ b/testing/btest/doc/sphinx/include-doc_scripting_framework_notice_shortcuts_02_bro.btest @@ -0,0 +1,11 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- framework_notice_shortcuts_02.bro + +@load policy/protocols/ssh/interesting-hostnames.bro +@load base/protocols/ssh/ + +redef Notice::type_suppression_intervals += { + [SSH::Interesting_Hostname_Login] = 1day, + [SSH::Login] = 12hrs, +}; diff --git a/testing/btest/doc/sphinx/include-scripts_base_bif_event_bif_bro.btest b/testing/btest/doc/sphinx/include-scripts_base_bif_event_bif_bro.btest new file mode 100644 index 0000000000..088d5d2368 --- /dev/null +++ b/testing/btest/doc/sphinx/include-scripts_base_bif_event_bif_bro.btest @@ -0,0 +1,25 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- event.bif.bro + +## script-level cleanup that needs to be performed for every connection. This +## event is generated not only for TCP sessions but also for UDP and ICMP +## flows. +## +## +global connection_external: event(c: connection , tag: string ); + + +## Generated when a UDP session for a supported protocol has finished. Some of +## Bro's application-layer UDP analyzers flag the end of a session by raising +## Generated when a connection is seen that is marked as being expected. + + +global ipv6_ext_headers: event(c: connection , p: pkt_hdr ); +## their specifics differ slightly. Often, however, both will be raised for +## the same connection if some of its data is missing. We should eventually +## merge the two. +global ack_above_hole: event(c: connection ); + + +## diff --git a/testing/btest/doc/sphinx/include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.btest b/testing/btest/doc/sphinx/include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.btest new file mode 100644 index 0000000000..c8e7eaab20 --- /dev/null +++ b/testing/btest/doc/sphinx/include-scripts_base_bif_plugins_Bro_DNS_events_bif_bro.btest @@ -0,0 +1,30 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- Bro_DNS.events.bif.bro + +## Generated for DNS requests. For requests with multiple queries, this event +## is raised once for each. +## +## See `Wikipedia `__ for more +## information about the DNS protocol. Bro analyzes both UDP and TCP DNS +## sessions. +## +## c: The connection, which may be UDP or TCP depending on the type of the +## transport-layer session being analyzed. +## +## msg: The parsed DNS message header. +## +## query: The queried name. +## +## qtype: The queried resource record type. +## +## qclass: The queried resource record class. +## +## .. bro:see:: dns_AAAA_reply dns_A_reply dns_CNAME_reply dns_EDNS_addl +## dns_HINFO_reply dns_MX_reply dns_NS_reply dns_PTR_reply dns_SOA_reply +## dns_SRV_reply dns_TSIG_addl dns_TXT_reply dns_WKS_reply dns_end +## dns_full_request dns_mapping_altered dns_mapping_lost_name dns_mapping_new_name +## dns_mapping_unverified dns_mapping_valid dns_message dns_query_reply +## dns_rejected non_dns_request dns_max_queries dns_session_timeout dns_skip_addl +## dns_skip_all_addl dns_skip_all_auth dns_skip_auth +global dns_request: event(c: connection , msg: dns_msg , query: string , qtype: count , qclass: count ); diff --git a/testing/btest/doc/sphinx/include-scripts_base_init-bare_bro.btest b/testing/btest/doc/sphinx/include-scripts_base_init-bare_bro.btest new file mode 100644 index 0000000000..7d23c68ae4 --- /dev/null +++ b/testing/btest/doc/sphinx/include-scripts_base_init-bare_bro.btest @@ -0,0 +1,7 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- init-bare.bro + +type string_array: table[count] of string; +type string_set: set[string]; +type addr_set: set[addr]; diff --git a/testing/btest/doc/sphinx/include-scripts_base_protocols_conn_main_bro.btest b/testing/btest/doc/sphinx/include-scripts_base_protocols_conn_main_bro.btest new file mode 100644 index 0000000000..dc18745d57 --- /dev/null +++ b/testing/btest/doc/sphinx/include-scripts_base_protocols_conn_main_bro.btest @@ -0,0 +1,28 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- main.bro + +module Conn; + +export { + ## The record type which contains column fields of the connection log. + type Info: record { + ts: time &log; + uid: string &log; + id: conn_id &log; + proto: transport_proto &log; + service: string &log &optional; + duration: interval &log &optional; + orig_bytes: count &log &optional; + resp_bytes: count &log &optional; + conn_state: string &log &optional; + local_orig: bool &log &optional; + missed_bytes: count &log &default=0; + history: string &log &optional; + orig_pkts: count &log &optional; + orig_ip_bytes: count &log &optional; + resp_pkts: count &log &optional; + resp_ip_bytes: count &log &optional; + tunnel_parents: set[string] &log; + }; +} diff --git a/testing/btest/doc/sphinx/include-scripts_base_protocols_http_main_bro.btest b/testing/btest/doc/sphinx/include-scripts_base_protocols_http_main_bro.btest new file mode 100644 index 0000000000..abe74112bd --- /dev/null +++ b/testing/btest/doc/sphinx/include-scripts_base_protocols_http_main_bro.btest @@ -0,0 +1,10 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- main.bro + +module HTTP; + +export { + ## This setting changes if passwords used in Basic-Auth are captured or not. + const default_capture_password = F &redef; +} diff --git a/testing/btest/doc/sphinx/include-scripts_policy_frameworks_files_detect-MHR_bro.btest b/testing/btest/doc/sphinx/include-scripts_policy_frameworks_files_detect-MHR_bro.btest new file mode 100644 index 0000000000..b73140fe8e --- /dev/null +++ b/testing/btest/doc/sphinx/include-scripts_policy_frameworks_files_detect-MHR_bro.btest @@ -0,0 +1,61 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- detect-MHR.bro + +##! Detect file downloads that have hash values matching files in Team +##! Cymru's Malware Hash Registry (http://www.team-cymru.org/Services/MHR/). + +@load base/frameworks/files +@load base/frameworks/notice +@load frameworks/files/hash-all-files + +module TeamCymruMalwareHashRegistry; + +export { + redef enum Notice::Type += { + ## The hash value of a file transferred over HTTP matched in the + ## malware hash registry. + Match + }; + + ## File types to attempt matching against the Malware Hash Registry. + const match_file_types = /application\/x-dosexec/ | + /application\/vnd.ms-cab-compressed/ | + /application\/pdf/ | + /application\/x-shockwave-flash/ | + /application\/x-java-applet/ | + /application\/jar/ | + /video\/mp4/ &redef; + + ## The malware hash registry runs each malware sample through several A/V engines. + ## Team Cymru returns a percentage to indicate how many A/V engines flagged the + ## sample as malicious. This threshold allows you to require a minimum detection + ## rate. + const notice_threshold = 10 &redef; +} + +event file_hash(f: fa_file, kind: string, hash: string) + { + if ( kind=="sha1" && match_file_types in f$mime_type ) + { + local hash_domain = fmt("%s.malware.hash.cymru.com", hash); + when ( local MHR_result = lookup_hostname_txt(hash_domain) ) + { + # Data is returned as " " + local MHR_answer = split1(MHR_result, / /); + if ( |MHR_answer| == 2 ) + { + local mhr_first_detected = double_to_time(to_double(MHR_answer[1])); + local mhr_detect_rate = to_count(MHR_answer[2]); + + local readable_first_detected = strftime("%Y-%m-%d %H:%M:%S", mhr_first_detected); + if ( mhr_detect_rate >= notice_threshold ) + { + local message = fmt("Malware Hash Registry Detection rate: %d%% Last seen: %s", mhr_detect_rate, readable_first_detected); + local virustotal_url = fmt("https://www.virustotal.com/en/file/%s/analysis/", hash); + NOTICE([$note=Match, $msg=message, $sub=virustotal_url, $f=f]); + } + } + } + } + } diff --git a/testing/btest/doc/sphinx/include-scripts_policy_frameworks_files_detect-MHR_bro@2.btest b/testing/btest/doc/sphinx/include-scripts_policy_frameworks_files_detect-MHR_bro@2.btest new file mode 100644 index 0000000000..51044b36d9 --- /dev/null +++ b/testing/btest/doc/sphinx/include-scripts_policy_frameworks_files_detect-MHR_bro@2.btest @@ -0,0 +1,9 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- detect-MHR.bro + + +module TeamCymruMalwareHashRegistry; + +export { + redef enum Notice::Type += { diff --git a/testing/btest/doc/sphinx/include-scripts_policy_frameworks_files_detect-MHR_bro@3.btest b/testing/btest/doc/sphinx/include-scripts_policy_frameworks_files_detect-MHR_bro@3.btest new file mode 100644 index 0000000000..37de8b6ff7 --- /dev/null +++ b/testing/btest/doc/sphinx/include-scripts_policy_frameworks_files_detect-MHR_bro@3.btest @@ -0,0 +1,17 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- detect-MHR.bro + + ## The hash value of a file transferred over HTTP matched in the + ## malware hash registry. + Match + }; + + ## File types to attempt matching against the Malware Hash Registry. + const match_file_types = /application\/x-dosexec/ | + /application\/vnd.ms-cab-compressed/ | + /application\/pdf/ | + /application\/x-shockwave-flash/ | + /application\/x-java-applet/ | + /application\/jar/ | + /video\/mp4/ &redef; diff --git a/testing/btest/doc/sphinx/include-scripts_policy_frameworks_files_detect-MHR_bro@4.btest b/testing/btest/doc/sphinx/include-scripts_policy_frameworks_files_detect-MHR_bro@4.btest new file mode 100644 index 0000000000..19e9d2ee54 --- /dev/null +++ b/testing/btest/doc/sphinx/include-scripts_policy_frameworks_files_detect-MHR_bro@4.btest @@ -0,0 +1,23 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- detect-MHR.bro + + ## The malware hash registry runs each malware sample through several A/V engines. + ## Team Cymru returns a percentage to indicate how many A/V engines flagged the + ## sample as malicious. This threshold allows you to require a minimum detection + ## rate. + const notice_threshold = 10 &redef; +} + +event file_hash(f: fa_file, kind: string, hash: string) + { + if ( kind=="sha1" && match_file_types in f$mime_type ) + { + local hash_domain = fmt("%s.malware.hash.cymru.com", hash); + when ( local MHR_result = lookup_hostname_txt(hash_domain) ) + { + # Data is returned as " " + local MHR_answer = split1(MHR_result, / /); + if ( |MHR_answer| == 2 ) + { + local mhr_first_detected = double_to_time(to_double(MHR_answer[1])); diff --git a/testing/btest/doc/sphinx/include-scripts_policy_protocols_conn_known-hosts_bro.btest b/testing/btest/doc/sphinx/include-scripts_policy_protocols_conn_known-hosts_bro.btest new file mode 100644 index 0000000000..9b0c50c3f4 --- /dev/null +++ b/testing/btest/doc/sphinx/include-scripts_policy_protocols_conn_known-hosts_bro.btest @@ -0,0 +1,9 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- known-hosts.bro + +module Known; + +export { + global known_hosts: set[addr] &create_expire=1day &synchronized &redef; +} diff --git a/testing/btest/doc/sphinx/include-scripts_policy_protocols_ssh_interesting-hostnames_bro.btest b/testing/btest/doc/sphinx/include-scripts_policy_protocols_ssh_interesting-hostnames_bro.btest new file mode 100644 index 0000000000..73eec1a1b9 --- /dev/null +++ b/testing/btest/doc/sphinx/include-scripts_policy_protocols_ssh_interesting-hostnames_bro.btest @@ -0,0 +1,50 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- interesting-hostnames.bro + +##! This script will generate a notice if an apparent SSH login originates +##! or heads to a host with a reverse hostname that looks suspicious. By +##! default, the regular expression to match "interesting" hostnames includes +##! names that are typically used for infrastructure hosts like nameservers, +##! mail servers, web servers and ftp servers. + +@load base/frameworks/notice + +module SSH; + +export { + redef enum Notice::Type += { + ## Generated if a login originates or responds with a host where the + ## reverse hostname lookup resolves to a name matched by the + ## :bro:id:`SSH::interesting_hostnames` regular expression. + Interesting_Hostname_Login, + }; + + ## Strange/bad host names to see successful SSH logins from or to. + const interesting_hostnames = + /^d?ns[0-9]*\./ | + /^smtp[0-9]*\./ | + /^mail[0-9]*\./ | + /^pop[0-9]*\./ | + /^imap[0-9]*\./ | + /^www[0-9]*\./ | + /^ftp[0-9]*\./ &redef; +} + +event SSH::heuristic_successful_login(c: connection) + { + for ( host in set(c$id$orig_h, c$id$resp_h) ) + { + when ( local hostname = lookup_addr(host) ) + { + if ( interesting_hostnames in hostname ) + { + NOTICE([$note=Interesting_Hostname_Login, + $msg=fmt("Possible SSH login involving a %s %s with an interesting hostname.", + Site::is_local_addr(host) ? "local" : "remote", + host == c$id$orig_h ? "client" : "server"), + $sub=hostname, $conn=c]); + } + } + } + } diff --git a/testing/btest/doc/sphinx/include-scripts_policy_protocols_ssl_expiring-certs_bro.btest b/testing/btest/doc/sphinx/include-scripts_policy_protocols_ssl_expiring-certs_bro.btest new file mode 100644 index 0000000000..16e2552545 --- /dev/null +++ b/testing/btest/doc/sphinx/include-scripts_policy_protocols_ssl_expiring-certs_bro.btest @@ -0,0 +1,8 @@ +# @TEST-EXEC: btest-diff %INPUT + +-- expiring-certs.bro + + NOTICE([$note=Certificate_Expires_Soon, + $msg=fmt("Certificate %s is going to expire at %T", cert$subject, cert$not_valid_after), + $conn=c, $suppress_for=1day, + $identifier=cat(c$id$resp_h, c$id$resp_p, c$ssl$cert_hash)]); diff --git a/testing/btest/doc/sphinx/using_bro.btest b/testing/btest/doc/sphinx/using_bro.btest new file mode 100644 index 0000000000..233f415942 --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest @@ -0,0 +1,2 @@ +@TEST-EXEC: btest-rst-cmd bro -r $TRACES/wikipedia.trace +@TEST-EXEC: btest-rst-include -n 15 conn.log diff --git a/testing/btest/doc/sphinx/using_bro.btest#2 b/testing/btest/doc/sphinx/using_bro.btest#2 new file mode 100644 index 0000000000..afa29e6184 --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest#2 @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd -n 10 "cat conn.log | bro-cut id.orig_h id.orig_p id.resp_h duration" diff --git a/testing/btest/doc/sphinx/using_bro.btest#3 b/testing/btest/doc/sphinx/using_bro.btest#3 new file mode 100644 index 0000000000..a3ce44357e --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest#3 @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd -n 10 awk \'/^[^#]/ {print \$3, \$4, \$5, \$6, \$9}\' conn.log diff --git a/testing/btest/doc/sphinx/using_bro.btest#4 b/testing/btest/doc/sphinx/using_bro.btest#4 new file mode 100644 index 0000000000..ce10bba56a --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest#4 @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd -n 5 "bro-cut -d ts uid host uri < http.log" diff --git a/testing/btest/doc/sphinx/using_bro.btest#5 b/testing/btest/doc/sphinx/using_bro.btest#5 new file mode 100644 index 0000000000..786aebffab --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest#5 @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd -n 5 "bro-cut -u ts uid host uri < http.log" diff --git a/testing/btest/doc/sphinx/using_bro.btest#6 b/testing/btest/doc/sphinx/using_bro.btest#6 new file mode 100644 index 0000000000..cc52531b9c --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest#6 @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd -n 5 "bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < http.log" diff --git a/testing/btest/doc/sphinx/using_bro.btest#7 b/testing/btest/doc/sphinx/using_bro.btest#7 new file mode 100644 index 0000000000..4662f67c88 --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest#7 @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd "cat conn.log | bro-cut uid resp_bytes | sort -nrk2 | head -5" diff --git a/testing/btest/doc/sphinx/using_bro.btest#8 b/testing/btest/doc/sphinx/using_bro.btest#8 new file mode 100644 index 0000000000..dfb7b3a715 --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest#8 @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd "cat http.log | bro-cut uid id.resp_h method status_code host uri | grep VW0XPVINV8a" diff --git a/testing/scripts/diff-remove-timestamps b/testing/scripts/diff-remove-timestamps index 138b901743..44422f6f55 100755 --- a/testing/scripts/diff-remove-timestamps +++ b/testing/scripts/diff-remove-timestamps @@ -11,4 +11,4 @@ fi # The first sed uses a "basic" regexp, the 2nd a "modern:. sed 's/[0-9]\{10\}\.[0-9]\{2,8\}/XXXXXXXXXX.XXXXXX/g' | \ -$sed 's/^#(open|close).(19|20)..-..-..-..-..-..$/#\1 XXXX-XX-XX-XX-XX-XX/g' +$sed 's/^ *#(open|close).(19|20)..-..-..-..-..-..$/#\1 XXXX-XX-XX-XX-XX-XX/g' diff --git a/testing/scripts/rst-filter b/testing/scripts/rst-filter new file mode 100755 index 0000000000..a62646508e --- /dev/null +++ b/testing/scripts/rst-filter @@ -0,0 +1,6 @@ +# /usr/bin/env bash +# +# Filters the output of btest-rst-cmd. + +sed "s#${TRACES}/\{0,1\}##g" | \ +sed 's#/[^ ]*/doc/\([^/]*/\)*##g'