Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-02 22:58:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Merge remote-tracking branch 'origin/master' into topic/vladg/kerberos

Browse source 
Conflicts:
	testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
	testing/btest/Baseline/scripts.policy.misc.dump-events/all-events.log
This commit is contained in:
Vlad Grigorescu 2014-10-27 13:56:07 -04:00
commit e6d6ba6ec6
511 changed files with 108792 additions and 86637 deletions
Download patch file Download diff file Expand all files Collapse all files

3
.gitmodules vendored
View file

@ -19,3 +19,6 @@
[submodule "src/3rdparty"] [submodule "src/3rdparty"]
path = src/3rdparty path = src/3rdparty
url = git://git.bro.org/bro-3rdparty url = git://git.bro.org/bro-3rdparty
[submodule "aux/plugins"]
path = aux/plugins
url = git://git.bro.org/bro-plugins

388
CHANGES
View file

@ -1,4 +1,392 @@
2.3-259 | 2014-10-27 10:04:04 -0500
* Documentation fixes. (Vicente Jimenez Aguilar and Stefano Azzalini)
2.3-256 | 2014-10-24 15:33:45 -0700
* Adding missing test baseline. (Robin Sommer)
2.3-255 | 2014-10-24 13:39:44 -0700
* Fixing unstable active-http test. (Robin Sommer)
2.3-254 | 2014-10-24 11:40:51 -0700
* Fix active-http.bro to deal reliably with empty server responses,
which will now be passed back as empty files. (Christian Struck)
2.3-248 | 2014-10-23 14:20:59 -0700
* Change order in which a plugin's scripts are loaded at startup.
(Robin Sommer)
2.3-247 | 2014-10-21 13:42:38 -0700
* Updates to the SSL analyzer. (Johanna Amann)
* Mark everything below 2048 bit as a weak key.
* Fix notice suppression.
* Add information about server-chosen protocol to ssl.log, if
provided by application_layer_next_protocol.
* Add boolean flag to ssl.log signaling if a session was
resumed. Remove the (usually not really that useful) session
ID that the client sent.
2.3-240 | 2014-10-21 13:36:33 -0700
* Fix Coverity-reported issues in DNP3 analyzer. (Seth Hall)
2.3-238 | 2014-10-16 06:51:49 -0700
* Fix multipart HTTP/MIME entity file analysis so that (1) singular
CR or LF characters in multipart body content are no longer
converted to a full CRLF (thus corrupting the file) and (2) it
also no longer considers the CRLF before the multipart boundary as
part of the content. Addresses BIT-1235. (Jon Siwek)
2.3-235 | 2014-10-15 10:20:47 -0500
* BIT-1273: Add error message for bad enum declaration syntax.
(Jon Siwek)
2.3-234 | 2014-10-14 14:42:09 -0500
* Documentation fixes. (Steve Smoot)
2.3-233 | 2014-10-09 16:00:27 -0500
* Change find-bro-logs unit test to follow symlinks. (Jon Siwek)
* Add error checks and messages to a test script (Daniel Thayer)
2.3-230 | 2014-10-08 08:15:17 -0700
* Further baseline normalization for plugin test portability. (Robin
Sommer)
2.3-229 | 2014-10-07 20:18:11 -0700
* Fix for test portability. (Robin Sommer)
2.3-228 | 2014-10-07 15:32:37 -0700
* Include plugin unit tests into the top-level btest configuration. (Robin Sommer)
* Switching the prefix separator for packet source/dumper plugins
once more, now to "::". Addresses BIT-1267. (Robin Sommer)
* Fix for allowing a packet source/dumper plugin to support multiple
prefixes with a colon. (Robin Sommer)
2.3-225 | 2014-10-07 15:13:35 -0700
* Updating plugin documentation. (Robin Sommer)
2.3-224 | 2014-10-07 14:32:17 -0700
* Improved the log file reference documentation. (Jeannette Dopheide
and Daniel Thayer)
* Improves shockwave flash file signatures. (Seth Hall)
- This moves the signatures out of the libmagic imported signatures
and into our own general.sig.
- Expand the detection to LZMA compressed flash files.
* Add new script language reference documentation on operators,
statements, and directives. Also improved the documentation on
types and attributes by splitting them into two docs, and
providing more examples and adding a chart on the top of each page
with links to each type and attribute for easier access to the
information. (Daniel Thayer)
* Split the types and attributes reference doc into two docs.
(Daniel Thayer)
2.3-208 | 2014-10-03 09:38:52 -0500
* BIT-1268: Fix uninitialized router_list argument in
dhcp_offer/dhcp_ack. (Jon Siwek)
2.3-207 | 2014-10-02 16:39:17 -0700
* Updating plugin docs. (Robin Sommer)
* Fix packet sources being treated as idle when a packet is
available. Addresses BIT-1266. (Jon Siwek)
* Fix regression causing the main loop to spin more frequently.
Addresses BIT-1266. (Jon Siwek)
2.3-203 | 2014-09-29 20:06:54 -0700
* Fix to use length parameter in DNP3 time conversion correctly now.
(Robin Sommer)
2.3-202 | 2014-09-29 17:05:18 -0700
* New SSL extension type from IANA and a few other SSL const
changes. (Johanna Amann)
* Make unexpected pipe errors fatal as precaution. Addresses
BIT-1260. (Jon Siwek)
* Adding a function for DNP3 to translate the timestamp format. (Hui
Lin)
2.3-197 | 2014-09-29 10:42:01 -0500
* Fix possible seg fault in TCP reassembler. (Jon Siwek)
2.3-196 | 2014-09-25 17:53:27 -0700
* Changing prefix for packet sources/dumper from ':' to '%'.
Addresses BIT-1249. (Robin Sommer)
* Remove timeouts from remote communication loop. The select() now
blocks until there's work to do instead of relying on a small
timeout value which can cause unproductive use of cpu cycles. (Jon
Siwek)
* Improve error message when failing to activate a plugin. Also fix
a unit test helper script that checks plugin availability. (Jon
Siwek)
2.3-183 | 2014-09-24 10:08:04 -0500
* Add a "node" field to Intel::Seen struture and intel.log to
indicate which node discovered a hit on an intel item. (Seth Hall)
* BIT-1261: Fixes to plugin quick start doc. (Jon Siwek)
2.3-180 | 2014-09-22 12:52:41 -0500
* BIT-1259: Fix issue w/ duplicate TCP reassembly deliveries.
(Jon Siwek)
2.3-178 | 2014-09-18 14:29:46 -0500
* BIT-1256: Fix file analysis events from coming after bro_done().
(Jon Siwek)
2.3-177 | 2014-09-17 09:41:27 -0500
* Documentation fixes. (Chris Mavrakis)
2.3-174 | 2014-09-17 09:37:09 -0500
* Fixed some "make doc" warnings caused by reST formatting
(Daniel Thayer).
2.3-172 | 2014-09-15 13:38:52 -0500
* Remove unneeded allocations for HTTP messages. (Jon Siwek)
2.3-171 | 2014-09-15 11:14:57 -0500
* Fix a compile error on systems without pcap-int.h. (Jon Siwek)
2.3-170 | 2014-09-12 19:28:01 -0700
* Fix incorrect data delivery skips after gap in HTTP Content-Range.
Addresses BIT-1247. (Jon Siwek)
* Fix file analysis placement of data after gap in HTTP
Content-Range. Addresses BIT-1248. (Jon Siwek)
* Fix issue w/ TCP reassembler not delivering some segments.
Addresses BIT-1246. (Jon Siwek)
* Fix MIME entity file data/gap ordering and raise http_entity_data
in line with data arrival. Addresses BIT-1240. (Jon Siwek)
* Implement file ID caching for MIME_Mail. (Jon Siwek)
* Fix a compile error. (Jon Siwek)
2.3-161 | 2014-09-09 12:35:38 -0500
* Bugfixes and test updates/additions. (Robin Sommer)
* Interface tweaks and docs for PktSrc/PktDumper. (Robin Sommer)
* Moving PCAP-related bifs to iosource/pcap.bif. (Robin Sommer)
* Moving some of the BPF filtering code into base class.
This will allow packet sources that don't support BPF natively to
emulate the filtering via libpcap. (Robin Sommer)
* Removing FlowSrc. (Robin Sommer)
* Removing remaining pieces of the 2ndary path, and left-over
files of packet sorter. (Robin Sommer)
* A bunch of infrastructure work to move IOSource, IOSourceRegistry
(now iosource::Manager) and PktSrc/PktDumper code into iosource/,
and over to a plugin structure. (Robin Sommer)
2.3-137 | 2014-09-08 19:01:13 -0500
* Fix Broxygen's rendering of opaque types. (Jon Siwek)
2.3-136 | 2014-09-07 20:50:46 -0700
* Change more http links to https. (Johanna Amann)
2.3-134 | 2014-09-04 16:16:36 -0700
* Fixed a number of issues with OCSP reply validation. Addresses
BIT-1212. (Johanna Amann)
* Fix null pointer dereference in OCSP verification code in case no
certificate is sent as part as the ocsp reply. Addresses BIT-1212.
(Johanna Amann)
2.3-131 | 2014-09-04 16:10:32 -0700
* Make links in documentation templates protocol relative. (Johanna
Amann)
2.3-129 | 2014-09-02 17:21:21 -0700
* Simplify a conditional with equivalent branches. (Jon Siwek)
* Change EDNS parsing code to use rdlength more cautiously. (Jon
Siwek)
* Fix a memory leak when bind() fails due to EADDRINUSE. (Jon Siwek)
* Fix possible buffer over-read in DNS TSIG parsing. (Jon Siwek)
2.3-124 | 2014-08-26 09:24:19 -0500
* Better documentation for sub_bytes (Jimmy Jones)
* BIT-1234: Fix build on systems that already have ntohll/htonll
(Jon Siwek)
2.3-121 | 2014-08-22 15:22:15 -0700
* Detect functions that try to bind variables from an outer scope
and raise an error saying that's not supported. Addresses
BIT-1233. (Jon Siwek)
2.3-116 | 2014-08-21 16:04:13 -0500
* Adding plugin testing to Makefile's test-all. (Robin Sommer)
* Converting log writers and input readers to plugins.
DataSeries and ElasticSearch plugins have moved to the new
bro-plugins repository, which is now a git submodule in the
aux/plugins directory. (Robin Sommer)
2.3-98 | 2014-08-19 11:03:46 -0500
* Silence some doc-related warnings when using `bro -e`.
Closes BIT-1232. (Jon Siwek)
* Fix possible null ptr derefs reported by Coverity. (Jon Siwek)
2.3-96 | 2014-08-01 14:35:01 -0700
* Small change to DHCP documentation. In server->client messages the
host name may differ from the one requested by the client.
(Johanna Amann)
* Split DHCP log writing from record creation. This allows users to
customize dhcp.log by changing the record in their own dhcp_ack
event. (Johanna Amann)
* Update PATH so that documentation btests can find bro-cut. (Daniel
Thayer)
* Remove gawk from list of optional packages in documentation.
(Daniel Thayer)
* Fix for redefining built-in constants. (Robin Sommer)
2.3-86 | 2014-07-31 14:19:58 -0700
* Fix for redefining built-in constants. (Robin Sommer)
* Adding missing check that a plugin's API version matches what Bro
defines. (Robin Sommer)
* Adding NEWS entry for plugins. (Robin Sommer)
2.3-83 | 2014-07-30 16:26:11 -0500
* Minor adjustments to plugin code/docs. (Jon Siwek)
* Dynamic plugin support. (Rpbin Sommer)
Bro now supports extending core functionality, like protocol and
file analysis, dynamically with external plugins in the form of
shared libraries. See doc/devel/plugins.rst for an overview of the
main functionality. Changes coming with this:
- Replacing the old Plugin macro magic with a new API.
- The plugin API changed to generally use std::strings instead
of const char*.
- There are a number of invocations of PLUGIN_HOOK_
{VOID,WITH_RESULT} across the code base, which allow plugins
to hook into the processing at those locations.
- A few new accessor methods to various classes to allow
plugins to get to that information.
- network_time cannot be just assigned to anymore, there's now
function net_update_time() for that.
- Redoing how builtin variables are initialized, so that it
works for plugins as well. No more init_net_var(), but
instead bifcl-generated code that registers them.
- Various changes for adjusting to the now dynamic generation
of analyzer instances.
- same_type() gets an optional extra argument allowing record type
comparision to ignore if field names don't match. (Robin Sommer)
- Further unify file analysis API with the protocol analyzer API
(assigning IDs to analyzers; adding Init()/Done() methods;
adding subtypes). (Robin Sommer)
- A new command line option -Q that prints some basic execution
time stats. (Robin Sommer)
- Add support to the file analysis for activating analyzers by
MIME type. (Robin Sommer)
- File::register_for_mime_type(tag: Analyzer::Tag, mt:
string): Associates a file analyzer with a MIME type.
- File::add_analyzers_for_mime_type(f: fa_file, mtype:
string): Activates all analyzers registered for a MIME
type for the file.
- The default file_new() handler calls
File::add_analyzers_for_mime_type() with the file's MIME
type.
2.3-20 | 2014-07-22 17:41:02 -0700
* Updating submodule(s).
2.3-19 | 2014-07-22 17:29:19 -0700
* Implement bytestring_to_coils() in Modbus analyzer so that coils
gets passed to the corresponding events. (Hui Lin)
* Add length field to ModbusHeaders. (Hui Lin)
2.3-12 | 2014-07-10 19:17:37 -0500 2.3-12 | 2014-07-10 19:17:37 -0500
* Include yield of vectors in Broxygen's type descriptions. * Include yield of vectors in Broxygen's type descriptions.

47
CMakeLists.txt
View file

@ -1,5 +1,9 @@
project(Bro C CXX) project(Bro C CXX)
# When changing the minimum version here, also adapt
# aux/bro-aux/plugin-support/skeleton/CMakeLists.txt
cmake_minimum_required(VERSION 2.6.3 FATAL_ERROR) cmake_minimum_required(VERSION 2.6.3 FATAL_ERROR)
include(cmake/CommonCMakeConfig.cmake) include(cmake/CommonCMakeConfig.cmake)
######################################################################## ########################################################################
@ -16,12 +20,18 @@ endif ()
get_filename_component(BRO_SCRIPT_INSTALL_PATH ${BRO_SCRIPT_INSTALL_PATH} get_filename_component(BRO_SCRIPT_INSTALL_PATH ${BRO_SCRIPT_INSTALL_PATH}
ABSOLUTE) ABSOLUTE)
set(BRO_PLUGIN_INSTALL_PATH ${BRO_ROOT_DIR}/lib/bro/plugins CACHE STRING "Installation path for plugins" FORCE)
configure_file(bro-path-dev.in ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev) configure_file(bro-path-dev.in ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev)
file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.sh file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.sh
"export BROPATH=`${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" "export BROPATH=`${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n"
"export BRO_PLUGIN_PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src:${BRO_PLUGIN_INSTALL_PATH}\"\n"
"export PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n") "export PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n")
file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.csh file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.csh
"setenv BROPATH `${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" "setenv BROPATH `${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n"
"setenv BRO_PLUGIN_PATH \"${CMAKE_CURRENT_BINARY_DIR}/src:${BRO_PLUGIN_INSTALL_PATH}\"\n"
"setenv PATH \"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n") "setenv PATH \"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n")
file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/VERSION" VERSION LIMIT_COUNT 1) file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/VERSION" VERSION LIMIT_COUNT 1)
@ -117,33 +127,6 @@ if (GOOGLEPERFTOOLS_FOUND)
endif () endif ()
endif () endif ()
set(USE_DATASERIES false)
find_package(Lintel)
find_package(DataSeries)
find_package(LibXML2)
if (NOT DISABLE_DATASERIES AND
LINTEL_FOUND AND DATASERIES_FOUND AND LIBXML2_FOUND)
set(USE_DATASERIES true)
include_directories(BEFORE ${Lintel_INCLUDE_DIR})
include_directories(BEFORE ${DataSeries_INCLUDE_DIR})
include_directories(BEFORE ${LibXML2_INCLUDE_DIR})
list(APPEND OPTLIBS ${Lintel_LIBRARIES})
list(APPEND OPTLIBS ${DataSeries_LIBRARIES})
list(APPEND OPTLIBS ${LibXML2_LIBRARIES})
endif()
set(USE_ELASTICSEARCH false)
set(USE_CURL false)
find_package(LibCURL)
if (NOT DISABLE_ELASTICSEARCH AND LIBCURL_FOUND)
set(USE_ELASTICSEARCH true)
set(USE_CURL true)
include_directories(BEFORE ${LibCURL_INCLUDE_DIR})
list(APPEND OPTLIBS ${LibCURL_LIBRARIES})
endif()
if (ENABLE_PERFTOOLS_DEBUG OR ENABLE_PERFTOOLS) if (ENABLE_PERFTOOLS_DEBUG OR ENABLE_PERFTOOLS)
# Just a no op to prevent CMake from complaining about manually-specified # Just a no op to prevent CMake from complaining about manually-specified
# ENABLE_PERFTOOLS_DEBUG or ENABLE_PERFTOOLS not being used if google # ENABLE_PERFTOOLS_DEBUG or ENABLE_PERFTOOLS not being used if google
@ -165,6 +148,8 @@ set(brodeps
include(TestBigEndian) include(TestBigEndian)
test_big_endian(WORDS_BIGENDIAN) test_big_endian(WORDS_BIGENDIAN)
include(CheckSymbolExists)
check_symbol_exists(htonll arpa/inet.h HAVE_BYTEORDER_64)
include(OSSpecific) include(OSSpecific)
include(CheckTypes) include(CheckTypes)
@ -174,6 +159,10 @@ include(MiscTests)
include(PCAPTests) include(PCAPTests)
include(OpenSSLTests) include(OpenSSLTests)
include(CheckNameserCompat) include(CheckNameserCompat)
include(GetArchitecture)
# Tell the plugin code that we're building as part of the main tree.
set(BRO_PLUGIN_INTERNAL_BUILD true CACHE INTERNAL "" FORCE)
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/config.h.in configure_file(${CMAKE_CURRENT_SOURCE_DIR}/config.h.in
${CMAKE_CURRENT_BINARY_DIR}/config.h) ${CMAKE_CURRENT_BINARY_DIR}/config.h)
@ -238,10 +227,6 @@ message(
"\n tcmalloc: ${USE_PERFTOOLS_TCMALLOC}" "\n tcmalloc: ${USE_PERFTOOLS_TCMALLOC}"
"\n debugging: ${USE_PERFTOOLS_DEBUG}" "\n debugging: ${USE_PERFTOOLS_DEBUG}"
"\njemalloc: ${ENABLE_JEMALLOC}" "\njemalloc: ${ENABLE_JEMALLOC}"
"\ncURL: ${USE_CURL}"
"\n"
"\nDataSeries: ${USE_DATASERIES}"
"\nElasticSearch: ${USE_ELASTICSEARCH}"
"\n" "\n"
"\n================================================================\n" "\n================================================================\n"
) )

1
Makefile
View file

@ -56,6 +56,7 @@ test-all: test
test -d aux/broctl && ( cd aux/broctl && make test ) test -d aux/broctl && ( cd aux/broctl && make test )
test -d aux/btest && ( cd aux/btest && make test ) test -d aux/btest && ( cd aux/btest && make test )
test -d aux/bro-aux && ( cd aux/bro-aux && make test ) test -d aux/bro-aux && ( cd aux/bro-aux && make test )
test -d aux/plugins && ( cd aux/plugins && make test-all )
configured: configured:
@test -d $(BUILD) || ( echo "Error: No build/ directory found. Did you run configure?" && exit 1 ) @test -d $(BUILD) || ( echo "Error: No build/ directory found. Did you run configure?" && exit 1 )

26
NEWS
View file

@ -4,6 +4,32 @@ release. For an exhaustive list of changes, see the ``CHANGES`` file
(note that submodules, such as BroControl and Broccoli, come with (note that submodules, such as BroControl and Broccoli, come with
their own ``CHANGES``.) their own ``CHANGES``.)
Bro 2.4 (in progress)
=====================
Dependencies
------------
New Functionality
-----------------
- Bro now has support for external plugins that can extend its core
functionality, like protocol/file analysis, via shared libraries.
Plugins can be developed and distributed externally, and will be
pulled in dynamically at startup. Currently, a plugin can provide
custom protocol analyzers, file analyzers, log writers[TODO], input
readers[TODO], packet sources[TODO], and new built-in functions. A
plugin can furthermore hook into Bro's processing a number of places
to add custom logic.
See https://www.bro.org/sphinx-git/devel/plugins.html for more
information on writing plugins.
Changed Functionality
---------------------
- bro-cut has been rewritten in C, and is hence much faster.
Bro 2.3 Bro 2.3
======= =======

2
VERSION
View file

@ -1 +1 @@
2.3-12 2.3-259

2
aux/binpac

@ -1 +1 @@
Subproject commit ec1e052afd5a8cd3d1d2cbb28fcd688018e379a5 Subproject commit c8e017b4b1893cf254fc2bc8eedd86b852a2e654

2
aux/bro-aux

@ -1 +1 @@
Subproject commit 31d011479a4e956e029d8b708446841a088dd7e3 Subproject commit 977654dc51ab08a2afde32241f108cdb4a581d8f

2
aux/broccoli

@ -1 +1 @@
Subproject commit 1ee129f7159a2c32fe0cb0f44c9412486fb7a479 Subproject commit acb8fbe8e7bc6ace5135fb73dca8e29432cdc1ca

2
aux/broctl

@ -1 +1 @@
Subproject commit 8a13886f322f3b618832c0ca3976e07f686d14da Subproject commit 2b13bfcc941018c76f74b81a6e74e5e4e723c747

2
aux/btest

@ -1 +1 @@
Subproject commit 4da1bd24038d4977e655f2b210f34e37f0b73b78 Subproject commit 1efa4d10f943351efea96def68e598b053fd217a

1
aux/plugins Submodule

@ -0,0 +1 @@
Subproject commit ad600b5bdcd56a2723e323c0f2c8e1708956ca4f

2
cmake

@ -1 +1 @@
Subproject commit 0f301aa08a970150195a2ea5b3ed43d2d98b35b3 Subproject commit 1316c07f7059647b6c4a496ea36e4b83bb5d8f0f

14
config.h.in
View file

@ -129,6 +129,9 @@
/* whether words are stored with the most significant byte first */ /* whether words are stored with the most significant byte first */
#cmakedefine WORDS_BIGENDIAN #cmakedefine WORDS_BIGENDIAN
/* whether htonll/ntohll is defined in <arpa/inet.h> */
#cmakedefine HAVE_BYTEORDER_64
/* ultrix can't hack const */ /* ultrix can't hack const */
#cmakedefine NEED_ULTRIX_CONST_HACK #cmakedefine NEED_ULTRIX_CONST_HACK
#ifdef NEED_ULTRIX_CONST_HACK #ifdef NEED_ULTRIX_CONST_HACK
@ -209,3 +212,14 @@
/* Common IPv6 extension structure */ /* Common IPv6 extension structure */
#cmakedefine HAVE_IP6_EXT #cmakedefine HAVE_IP6_EXT
/* String with host architecture (e.g., "linux-x86_64") */
#define HOST_ARCHITECTURE "@HOST_ARCHITECTURE@"
/* String with extension of dynamic libraries (e.g., ".so") */
#define DYNAMIC_PLUGIN_SUFFIX "@CMAKE_SHARED_MODULE_SUFFIX@"
/* True if we're building outside of the main Bro source code tree. */
#ifndef BRO_PLUGIN_INTERNAL_BUILD
#define BRO_PLUGIN_INTERNAL_BUILD @BRO_PLUGIN_INTERNAL_BUILD@
#endif

21
configure vendored
View file

@ -39,8 +39,6 @@ Usage: $0 [OPTION]... [VAR=VALUE]...
--disable-auxtools don't build or install auxiliary tools --disable-auxtools don't build or install auxiliary tools
--disable-perftools don't try to build with Google Perftools --disable-perftools don't try to build with Google Perftools
--disable-python don't try to build python bindings for broccoli --disable-python don't try to build python bindings for broccoli
--disable-dataseries don't use the optional DataSeries log writer
--disable-elasticsearch don't use the optional ElasticSearch log writer
Required Packages in Non-Standard Locations: Required Packages in Non-Standard Locations:
--with-openssl=PATH path to OpenSSL install root --with-openssl=PATH path to OpenSSL install root
@ -62,9 +60,6 @@ Usage: $0 [OPTION]... [VAR=VALUE]...
--with-ruby-lib=PATH path to ruby library --with-ruby-lib=PATH path to ruby library
--with-ruby-inc=PATH path to ruby headers --with-ruby-inc=PATH path to ruby headers
--with-swig=PATH path to SWIG executable --with-swig=PATH path to SWIG executable
--with-dataseries=PATH path to DataSeries and Lintel libraries
--with-xml2=PATH path to libxml2 installation (for DataSeries)
--with-curl=PATH path to libcurl install root (for ElasticSearch)
Packaging Options (for developers): Packaging Options (for developers):
--binary-package toggle special logic for binary packaging --binary-package toggle special logic for binary packaging
@ -183,12 +178,6 @@ while [ $# -ne 0 ]; do
--enable-ruby) --enable-ruby)
append_cache_entry DISABLE_RUBY_BINDINGS BOOL false append_cache_entry DISABLE_RUBY_BINDINGS BOOL false
;; ;;
--disable-dataseries)
append_cache_entry DISABLE_DATASERIES BOOL true
;;
--disable-elasticsearch)
append_cache_entry DISABLE_ELASTICSEARCH BOOL true
;;
--with-openssl=*) --with-openssl=*)
append_cache_entry OpenSSL_ROOT_DIR PATH $optarg append_cache_entry OpenSSL_ROOT_DIR PATH $optarg
;; ;;
@ -243,16 +232,6 @@ while [ $# -ne 0 ]; do
--with-swig=*) --with-swig=*)
append_cache_entry SWIG_EXECUTABLE PATH $optarg append_cache_entry SWIG_EXECUTABLE PATH $optarg
;; ;;
--with-dataseries=*)
append_cache_entry DataSeries_ROOT_DIR PATH $optarg
append_cache_entry Lintel_ROOT_DIR PATH $optarg
;;
--with-xml2=*)
append_cache_entry LibXML2_ROOT_DIR PATH $optarg
;;
--with-curl=*)
append_cache_entry LibCURL_ROOT_DIR PATH $optarg
;;
--binary-package) --binary-package)
append_cache_entry BINARY_PACKAGING_MODE BOOL true append_cache_entry BINARY_PACKAGING_MODE BOOL true
;; ;;

4
doc/_templates/layout.html vendored
View file

@ -10,7 +10,7 @@
{% endblock %} {% endblock %}
{% block header %} {% block header %}
<iframe src="http://www.bro.org/frames/header-no-logo.html" width="100%" height="100px" frameborder="0" marginheight="0" scrolling="no" marginwidth="0"> <iframe src="//www.bro.org/frames/header-no-logo.html" width="100%" height="100px" frameborder="0" marginheight="0" scrolling="no" marginwidth="0">
</iframe> </iframe>
{% endblock %} {% endblock %}
@ -108,6 +108,6 @@
{% endblock %} {% endblock %}
{% block footer %} {% block footer %}
<iframe src="http://www.bro.org/frames/footer.html" width="100%" height="420px" frameborder="0" marginheight="0" scrolling="no" marginwidth="0"> <iframe src="//www.bro.org/frames/footer.html" width="100%" height="420px" frameborder="0" marginheight="0" scrolling="no" marginwidth="0">
</iframe> </iframe>
{% endblock %} {% endblock %}

2
doc/conf.py.in
View file

@ -21,7 +21,7 @@ sys.path.insert(0, os.path.abspath('sphinx_input/ext'))
# ----- Begin of BTest configuration. ----- # ----- Begin of BTest configuration. -----
btest = os.path.abspath("@CMAKE_SOURCE_DIR@/aux/btest") btest = os.path.abspath("@CMAKE_SOURCE_DIR@/aux/btest")
brocut = os.path.abspath("@CMAKE_SOURCE_DIR@/aux/bro-aux/bro-cut") brocut = os.path.abspath("@CMAKE_SOURCE_DIR@/build/aux/bro-aux/bro-cut")
bro = os.path.abspath("@CMAKE_SOURCE_DIR@/build/src") bro = os.path.abspath("@CMAKE_SOURCE_DIR@/build/src")
os.environ["PATH"] += (":%s:%s/sphinx:%s:%s" % (btest, btest, bro, brocut)) os.environ["PATH"] += (":%s:%s/sphinx:%s:%s" % (btest, btest, bro, brocut))

447
doc/devel/plugins.rst Normal file
View file

@ -0,0 +1,447 @@
===================
Writing Bro Plugins
===================
Bro is internally moving to a plugin structure that enables extending
the system dynamically, without modifying the core code base. That way
custom code remains self-contained and can be maintained, compiled,
and installed independently. Currently, plugins can add the following
functionality to Bro:
- Bro scripts.
- Builtin functions/events/types for the scripting language.
- Protocol analyzers.
- File analyzers.
- Packet sources and packet dumpers.
- Logging framework backends.
- Input framework readers.
A plugin's functionality is available to the user just as if Bro had
the corresponding code built-in. Indeed, internally many of Bro's
pieces are structured as plugins as well, they are just statically
compiled into the binary rather than loaded dynamically at runtime.
Quick Start
===========
Writing a basic plugin is quite straight-forward as long as one
follows a few conventions. In the following we walk a simple example
plugin that adds a new built-in function (bif) to Bro: we'll add
``rot13(s: string) : string``, a function that rotates every character
in a string by 13 places.
Generally, a plugin comes in the form of a directory following a
certain structure. To get started, Bro's distribution provides a
helper script ``aux/bro-aux/plugin-support/init-plugin`` that creates
a skeleton plugin that can then be customized. Let's use that::
# mkdir rot13-plugin
# cd rot13-plugin
# init-plugin Demo Rot13
As you can see the script takes two arguments. The first is a
namespace the plugin will live in, and the second a descriptive name
for the plugin itself. Bro uses the combination of the two to identify
a plugin. The namespace serves to avoid naming conflicts between
plugins written by independent developers; pick, e.g., the name of
your organisation. The namespace ``Bro`` is reserved for functionality
distributed by the Bro Project. In our example, the plugin will be
called ``Demo::Rot13``.
The ``init-plugin`` script puts a number of files in place. The full
layout is described later. For now, all we need is
``src/rot13.bif``. It's initially empty, but we'll add our new bif
there as follows::
# cat src/rot13.bif
module CaesarCipher;
function rot13%(s: string%) : string
%{
char* rot13 = copy_string(s->CheckString());
for ( char* p = rot13; *p; p++ )
{
char b = islower(*p) ? 'a' : 'A';
*p = (*p - b + 13) % 26 + b;
}
BroString* bs = new BroString(1, reinterpret_cast<byte_vec>(rot13),
strlen(rot13));
return new StringVal(bs);
%}
The syntax of this file is just like any other ``*.bif`` file; we
won't go into it here.
Now we can already compile our plugin, we just need to tell the
configure script put in place by ``init-plugin`` where the Bro source
tree is located (Bro needs to have been built there first)::
# ./configure --bro-dist=/path/to/bro/dist && make
[... cmake output ...]
Now our ``rot13-plugin`` directory has everything that it needs
for Bro to recognize it as a dynamic plugin. Once we point Bro to it,
it will pull it in automatically, as we can check with the ``-N``
option::
# export BRO_PLUGIN_PATH=/path/to/rot13-plugin
# bro -N
[...]
Plugin: Demo::Rot13 - <Insert brief description of plugin> (dynamic, version 1)
[...]
That looks quite good, except for the dummy description that we should
replace with something nicer so that users will know what our plugin
is about. We do this by editing the ``config.description`` line in
``src/Plugin.cc``, like this::
[...]
plugin::Configuration Configure()
{
plugin::Configuration config;
config.name = "Demo::Rot13";
config.description = "Caesar cipher rotating a string's characters by 13 places.";
config.version.major = 1;
config.version.minor = 0;
return config;
}
[...]
# make
[...]
# bro -N | grep Rot13
Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1)
Better. Bro can also show us what exactly the plugin provides with the
more verbose option ``-NN``::
# bro -NN
[...]
Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1)
[Function] CaesarCipher::rot13
[...]
There's our function. Now let's use it::
# bro -e 'print CaesarCipher::rot13("Hello")'
Uryyb
It works. We next install the plugin along with Bro itself, so that it
will find it directly without needing the ``BRO_PLUGIN_PATH``
environment variable. If we first unset the variable, the function
will no longer be available::
# unset BRO_PLUGIN_PATH
# bro -e 'print CaesarCipher::rot13("Hello")'
error in <command line>, line 1: unknown identifier CaesarCipher::rot13, at or near "CaesarCipher::rot13"
Once we install it, it works again::
# make install
# bro -e 'print CaesarCipher::rot13("Hello")'
Uryyb
The installed version went into
``<bro-install-prefix>/lib/bro/plugins/Demo_Rot13``.
We can distribute the plugin in either source or binary form by using
the Makefile's ``sdist`` and ``bdist`` target, respectively. Both
create corrsponding tarballs::
# make sdist
[...]
Source distribution in build/sdist/Demo_Rot13.tar.gz
# make bdist
[...]
Binary distribution in build/Demo_Rot13-darwin-x86_64.tar.gz
The source archive will contain everything in the plugin directory
except any generated files. The binary archive will contain anything
needed to install and run the plugin, i.e., just what ``make install``
puts into place as well. As the binary distribution is
platform-dependent, its name includes the OS and architecture the
plugin was built on.
Plugin Directory Layout
=======================
A plugin's directory needs to follow a set of conventions so that Bro
(1) recognizes it as a plugin, and (2) knows what to load. While
``init-plugin`` takes care of most of this, the following is the full
story. We'll use ``<base>`` to represent a plugin's top-level
directory.
``<base>/__bro_plugin__``
A file that marks a directory as containing a Bro plugin. The file
must exist, and its content must consist of a single line with the
qualified name of the plugin (e.g., "Demo::Rot13").
``<base>/lib/<plugin-name>-<os>-<arch>.so``
The shared library containing the plugin's compiled code. Bro will
load this in dynamically at run-time if OS and architecture match
the current platform.
``scripts/``
A directory with the plugin's custom Bro scripts. When the plugin
gets activated, this directory will be automatically added to
``BROPATH``, so that any scripts/modules inside can be
"@load"ed.
``scripts``/__load__.bro
A Bro script that will be loaded immediately when the plugin gets
activated. See below for more information on activating plugins.
``lib/bif/``
Directory with auto-generated Bro scripts that declare the plugin's
bif elements. The files here are produced by ``bifcl``.
By convention, a plugin should put its custom scripts into sub folders
of ``scripts/``, i.e., ``scripts/<script-namespace>/<script>.bro`` to
avoid conflicts. As usual, you can then put a ``__load__.bro`` in
there as well so that, e.g., ``@load Demo/Rot13`` could load a whole
module in the form of multiple individual scripts.
Note that in addition to the paths above, the ``init-plugin`` helper
puts some more files and directories in place that help with
development and installation (e.g., ``CMakeLists.txt``, ``Makefile``,
and source code in ``src/``). However, all these do not have a special
meaning for Bro at runtime and aren't necessary for a plugin to
function.
``init-plugin``
===============
``init-plugin`` puts a basic plugin structure in place that follows
the above layout and augments it with a CMake build and installation
system. Plugins with this structure can be used both directly out of
their source directory (after ``make`` and setting Bro's
``BRO_PLUGIN_PATH``), and when installed alongside Bro (after ``make
install``).
``make install`` copies over the ``lib`` and ``scripts`` directories,
as well as the ``__bro_plugin__`` magic file and the ``README`` (which
you should customize). One can add further CMake ``install`` rules to
install additional files if needed.
``init-plugin`` will never overwrite existing files, so it's safe to
rerun in an existing plugin directory; it only put files in place that
don't exist yet. That also provides a convenient way to revert a file
back to what ``init-plugin`` created originally: just delete it and
rerun.
Activating a Plugin
===================
A plugin needs to be *activated* to make it available to the user.
Activating a plugin will:
1. Load the dynamic module
2. Make any bif items available
3. Add the ``scripts/`` directory to ``BROPATH``
4. Load ``scripts/__load__.bro``
By default, Bro will automatically activate all dynamic plugins found
in its search path ``BRO_PLUGIN_PATH``. However, in bare mode (``bro
-b``), no dynamic plugins will be activated by default; instead the
user can selectively enable individual plugins in scriptland using the
``@load-plugin <qualified-plugin-name>`` directive (e.g.,
``@load-plugin Demo::Rot13``). Alternatively, one can activate a
plugin from the command-line by specifying its full name
(``Demo::Rot13``), or set the environment variable
``BRO_PLUGIN_ACTIVATE`` to a list of comma(!)-separated names of
plugins to unconditionally activate, even in bare mode.
``bro -N`` shows activated plugins separately from found but not yet
activated plugins. Note that plugins compiled statically into Bro are
always activated, and hence show up as such even in bare mode.
Plugin Components
=================
The following subsections detail providing individual types of
functionality via plugins. Note that a single plugin can provide more
than one component type. For example, a plugin could provide multiple
protocol analyzers at once; or both a logging backend and input reader
at the same time.
.. todo::
These subsections are mostly missing right now, as much of their
content isn't actually plugin-specific, but concerns generally
writing such functionality for Bro. The best way to get started
right now is to look at existing code implementing similar
functionality, either as a plugin or inside Bro proper. Also, for
each component type there's a unit test in
``testing/btest/plugins`` creating a basic plugin skeleton with a
corresponding component.
Bro Scripts
-----------
Scripts are easy: just put them into ``scripts/``, as described above.
The CMake infrastructure will automatically install them, as well
include them into the source and binary plugin distributions.
Builtin Language Elements
-------------------------
Functions
TODO
Events
TODO
Types
TODO
Protocol Analyzers
------------------
TODO.
File Analyzers
--------------
TODO.
Logging Writer
--------------
TODO.
Input Reader
------------
TODO.
Packet Sources
--------------
TODO.
Packet Dumpers
--------------
TODO.
Hooks
=====
TODO.
Testing Plugins
===============
A plugin should come with a test suite to exercise its functionality.
The ``init-plugin`` script puts in place a basic </btest/README> setup
to start with. Initially, it comes with a single test that just checks
that Bro loads the plugin correctly. It won't have a baseline yet, so
let's get that in place::
# cd tests
# btest -d
[ 0%] plugin.loading ... failed
% 'btest-diff output' failed unexpectedly (exit code 100)
% cat .diag
== File ===============================
Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1.0)
[Function] CaesarCipher::rot13
== Error ===============================
test-diff: no baseline found.
=======================================
# btest -U
all 1 tests successful
# cd ..
# make test
make -C tests
make[1]: Entering directory `tests'
all 1 tests successful
make[1]: Leaving directory `tests'
Now let's add a custom test that ensures that our bif works
correctly::
# cd tests
# cat >plugin/rot13.bro
# @TEST-EXEC: bro %INPUT >output
# @TEST-EXEC: btest-diff output
event bro_init()
{
print CaesarCipher::rot13("Hello");
}
Check the output::
# btest -d plugin/rot13.bro
[ 0%] plugin.rot13 ... failed
% 'btest-diff output' failed unexpectedly (exit code 100)
% cat .diag
== File ===============================
Uryyb
== Error ===============================
test-diff: no baseline found.
=======================================
% cat .stderr
1 of 1 test failed
Install the baseline::
# btest -U plugin/rot13.bro
all 1 tests successful
Run the test-suite::
# btest
all 2 tests successful
Debugging Plugins
=================
If your plugin isn't loading as expected, Bro's debugging facilities
can help to illuminate what's going on. To enable, recompile Bro
with debugging support (``./configure --enable-debug``), and
afterwards rebuild your plugin as well. If you then run Bro with ``-B
plugins``, it will produce a file ``debug.log`` that records details
about the process for searching, loading, and activating plugins.
To generate your own debugging output from inside your plugin, you can
add a custom debug stream by using the ``PLUGIN_DBG_LOG(<plugin>,
<args>)`` macro (defined in ``DebugLogger.h``), where ``<plugin>`` is
the ``Plugin`` instance and ``<args>`` are printf-style arguments,
just as with Bro's standard debugging macros (grep for ``DBG_LOG`` in
Bro's ``src/`` to see examples). At runtime, you can then activate
your plugin's debugging output with ``-B plugin-<name>``, where
``<name>`` is the name of the plugin as returned by its
``Configure()`` method, yet with the namespace-separator ``::``
replaced with a simple dash. Example: If the plugin is called
``Bro::Demo``, use ``-B plugin-Bro-Demo``. As usual, the debugging
output will be recorded to ``debug.log`` if Bro's compiled in debug
mode.
Documenting Plugins
===================
.. todo::
Integrate all this with Broxygen.

7
doc/ext/bro.py
View file

@ -176,6 +176,10 @@ class BroIdentifier(BroGeneric):
def get_index_text(self, objectname, name): def get_index_text(self, objectname, name):
return name return name
class BroKeyword(BroGeneric):
def get_index_text(self, objectname, name):
return name
class BroAttribute(BroGeneric): class BroAttribute(BroGeneric):
def get_index_text(self, objectname, name): def get_index_text(self, objectname, name):
return _('%s (attribute)') % (name) return _('%s (attribute)') % (name)
@ -213,6 +217,7 @@ class BroDomain(Domain):
'type': ObjType(l_('type'), 'type'), 'type': ObjType(l_('type'), 'type'),
'namespace': ObjType(l_('namespace'), 'namespace'), 'namespace': ObjType(l_('namespace'), 'namespace'),
'id': ObjType(l_('id'), 'id'), 'id': ObjType(l_('id'), 'id'),
'keyword': ObjType(l_('keyword'), 'keyword'),
'enum': ObjType(l_('enum'), 'enum'), 'enum': ObjType(l_('enum'), 'enum'),
'attr': ObjType(l_('attr'), 'attr'), 'attr': ObjType(l_('attr'), 'attr'),
} }
@ -221,6 +226,7 @@ class BroDomain(Domain):
'type': BroGeneric, 'type': BroGeneric,
'namespace': BroNamespace, 'namespace': BroNamespace,
'id': BroIdentifier, 'id': BroIdentifier,
'keyword': BroKeyword,
'enum': BroEnum, 'enum': BroEnum,
'attr': BroAttribute, 'attr': BroAttribute,
} }
@ -229,6 +235,7 @@ class BroDomain(Domain):
'type': XRefRole(), 'type': XRefRole(),
'namespace': XRefRole(), 'namespace': XRefRole(),
'id': XRefRole(), 'id': XRefRole(),
'keyword': XRefRole(),
'enum': XRefRole(), 'enum': XRefRole(),
'attr': XRefRole(), 'attr': XRefRole(),
'see': XRefRole(), 'see': XRefRole(),

186
doc/frameworks/logging-dataseries.rst
View file

@ -1,186 +0,0 @@
=============================
Binary Output with DataSeries
=============================
.. rst-class:: opening
Bro's default ASCII log format is not exactly the most efficient
way for storing and searching large volumes of data. An an
alternative, Bro comes with experimental support for `DataSeries
<http://www.hpl.hp.com/techreports/2009/HPL-2009-323.html>`_
output, an efficient binary format for recording structured bulk
data. DataSeries is developed and maintained at HP Labs.
.. contents::
Installing DataSeries
---------------------
To use DataSeries, its libraries must be available at compile-time,
along with the supporting *Lintel* package. Generally, both are
distributed on `HP Labs' web site
<http://tesla.hpl.hp.com/opensource/>`_. Currently, however, you need
to use recent development versions for both packages, which you can
download from github like this::
git clone http://github.com/dataseries/Lintel
git clone http://github.com/dataseries/DataSeries
To build and install the two into ``<prefix>``, do::
( cd Lintel && mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX=<prefix> .. && make && make install )
( cd DataSeries && mkdir build && cd build && cmake -DCMAKE_INSTALL_PREFIX=<prefix> .. && make && make install )
Please refer to the packages' documentation for more information about
the installation process. In particular, there's more information on
required and optional `dependencies for Lintel
<https://raw.github.com/dataseries/Lintel/master/doc/dependencies.txt>`_
and `dependencies for DataSeries
<https://raw.github.com/dataseries/DataSeries/master/doc/dependencies.txt>`_.
For users on RedHat-style systems, you'll need the following::
yum install libxml2-devel boost-devel
Compiling Bro with DataSeries Support
-------------------------------------
Once you have installed DataSeries, Bro's ``configure`` should pick it
up automatically as long as it finds it in a standard system location.
Alternatively, you can specify the DataSeries installation prefix
manually with ``--with-dataseries=<prefix>``. Keep an eye on
``configure``'s summary output, if it looks like the following, Bro
found DataSeries and will compile in the support::
# ./configure --with-dataseries=/usr/local
[...]
====================| Bro Build Summary |=====================
[...]
DataSeries: true
[...]
================================================================
Activating DataSeries
---------------------
The direct way to use DataSeries is to switch *all* log files over to
the binary format. To do that, just add ``redef
Log::default_writer=Log::WRITER_DATASERIES;`` to your ``local.bro``.
For testing, you can also just pass that on the command line::
bro -r trace.pcap Log::default_writer=Log::WRITER_DATASERIES
With that, Bro will now write all its output into DataSeries files
``*.ds``. You can inspect these using DataSeries's set of command line
tools, which its installation process installs into ``<prefix>/bin``.
For example, to convert a file back into an ASCII representation::
$ ds2txt conn.log
[... We skip a bunch of metadata here ...]
ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes
1300475167.096535 CRCC5OdDlXe 141.142.220.202 5353 224.0.0.251 5353 udp dns 0.000000 0 0 S0 F 0 D 1 73 0 0
1300475167.097012 o7XBsfvo3U1 fe80::217:f2ff:fed7:cf65 5353 ff02::fb 5353 udp 0.000000 0 0 S0 F 0 D 1 199 0 0
1300475167.099816 pXPi1kPMgxb 141.142.220.50 5353 224.0.0.251 5353 udp 0.000000 0 0 S0 F 0 D 1 179 0 0
1300475168.853899 R7sOc16woCj 141.142.220.118 43927 141.142.2.2 53 udp dns 0.000435 38 89 SF F 0 Dd 1 66 1 117
1300475168.854378 Z6dfHVmt0X7 141.142.220.118 37676 141.142.2.2 53 udp dns 0.000420 52 99 SF F 0 Dd 1 80 1 127
1300475168.854837 k6T92WxgNAh 141.142.220.118 40526 141.142.2.2 53 udp dns 0.000392 38 183 SF F 0 Dd 1 66 1 211
[...]
(``--skip-all`` suppresses the metadata.)
Note that the ASCII conversion is *not* equivalent to Bro's default
output format.
You can also switch only individual files over to DataSeries by adding
code like this to your ``local.bro``:
.. code:: bro
event bro_init()
{
local f = Log::get_filter(Conn::LOG, "default"); # Get default filter for connection log.
f$writer = Log::WRITER_DATASERIES; # Change writer type.
Log::add_filter(Conn::LOG, f); # Replace filter with adapted version.
}
Bro's DataSeries writer comes with a few tuning options, see
:doc:`/scripts/base/frameworks/logging/writers/dataseries.bro`.
Working with DataSeries
=======================
Here are a few examples of using DataSeries command line tools to work
with the output files.
* Printing CSV::
$ ds2txt --csv conn.log
ts,uid,id.orig_h,id.orig_p,id.resp_h,id.resp_p,proto,service,duration,orig_bytes,resp_bytes,conn_state,local_orig,missed_bytes,history,orig_pkts,orig_ip_bytes,resp_pkts,resp_ip_bytes
1258790493.773208,ZTtgbHvf4s3,192.168.1.104,137,192.168.1.255,137,udp,dns,3.748891,350,0,S0,F,0,D,7,546,0,0
1258790451.402091,pOY6Rw7lhUd,192.168.1.106,138,192.168.1.255,138,udp,,0.000000,0,0,S0,F,0,D,1,229,0,0
1258790493.787448,pn5IiEslca9,192.168.1.104,138,192.168.1.255,138,udp,,2.243339,348,0,S0,F,0,D,2,404,0,0
1258790615.268111,D9slyIu3hFj,192.168.1.106,137,192.168.1.255,137,udp,dns,3.764626,350,0,S0,F,0,D,7,546,0,0
[...]
Add ``--separator=X`` to set a different separator.
* Extracting a subset of columns::
$ ds2txt --select '*' ts,id.resp_h,id.resp_p --skip-all conn.log
1258790493.773208 192.168.1.255 137
1258790451.402091 192.168.1.255 138
1258790493.787448 192.168.1.255 138
1258790615.268111 192.168.1.255 137
1258790615.289842 192.168.1.255 138
[...]
* Filtering rows::
$ ds2txt --where '*' 'duration > 5 && id.resp_p > 1024' --skip-all conn.ds
1258790631.532888 V8mV5WLITu5 192.168.1.105 55890 239.255.255.250 1900 udp 15.004568 798 0 S0 F 0 D 6 966 0 0
1258792413.439596 tMcWVWQptvd 192.168.1.105 55890 239.255.255.250 1900 udp 15.004581 798 0 S0 F 0 D 6 966 0 0
1258794195.346127 cQwQMRdBrKa 192.168.1.105 55890 239.255.255.250 1900 udp 15.005071 798 0 S0 F 0 D 6 966 0 0
1258795977.253200 i8TEjhWd2W8 192.168.1.105 55890 239.255.255.250 1900 udp 15.004824 798 0 S0 F 0 D 6 966 0 0
1258797759.160217 MsLsBA8Ia49 192.168.1.105 55890 239.255.255.250 1900 udp 15.005078 798 0 S0 F 0 D 6 966 0 0
1258799541.068452 TsOxRWJRGwf 192.168.1.105 55890 239.255.255.250 1900 udp 15.004082 798 0 S0 F 0 D 6 966 0 0
[...]
* Calculate some statistics:
Mean/stddev/min/max over a column::
$ dsstatgroupby '*' basic duration from conn.ds
# Begin DSStatGroupByModule
# processed 2159 rows, where clause eliminated 0 rows
# count(*), mean(duration), stddev, min, max
2159, 42.7938, 1858.34, 0, 86370
[...]
Quantiles of total connection volume::
$ dsstatgroupby '*' quantile 'orig_bytes + resp_bytes' from conn.ds
[...]
2159 data points, mean 24616 +- 343295 [0,1.26615e+07]
quantiles about every 216 data points:
10%: 0, 124, 317, 348, 350, 350, 601, 798, 1469
tails: 90%: 1469, 95%: 7302, 99%: 242629, 99.5%: 1226262
[...]
The ``man`` pages for these tools show further options, and their
``-h`` option gives some more information (either can be a bit cryptic
unfortunately though).
Deficiencies
------------
Due to limitations of the DataSeries format, one cannot inspect its
files before they have been fully written. In other words, when using
DataSeries, it's currently not possible to inspect the live log
files inside the spool directory before they are rotated to their
final location. It seems that this could be fixed with some effort,
and we will work with DataSeries development team on that if the
format gains traction among Bro users.
Likewise, we're considering writing custom command line tools for
interacting with DataSeries files, making that a bit more convenient
than what the standard utilities provide.

89
doc/frameworks/logging-elasticsearch.rst
View file

@ -1,89 +0,0 @@
=========================================
Indexed Logging Output with ElasticSearch
=========================================
.. rst-class:: opening
Bro's default ASCII log format is not exactly the most efficient
way for searching large volumes of data. ElasticSearch
is a new data storage technology for dealing with tons of data.
It's also a search engine built on top of Apache's Lucene
project. It scales very well, both for distributed indexing and
distributed searching.
.. contents::
Warning
-------
This writer plugin is still in testing and is not yet recommended for
production use! The approach to how logs are handled in the plugin is "fire
and forget" at this time, there is no error handling if the server fails to
respond successfully to the insertion request.
Installing ElasticSearch
------------------------
Download the latest version from: http://www.elasticsearch.org/download/.
Once extracted, start ElasticSearch with::
# ./bin/elasticsearch
For more detailed information, refer to the ElasticSearch installation
documentation: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup.html
Compiling Bro with ElasticSearch Support
----------------------------------------
First, ensure that you have libcurl installed then run configure::
# ./configure
[...]
====================| Bro Build Summary |=====================
[...]
cURL: true
[...]
ElasticSearch: true
[...]
================================================================
Activating ElasticSearch
------------------------
The easiest way to enable ElasticSearch output is to load the
tuning/logs-to-elasticsearch.bro script. If you are using BroControl,
the following line in local.bro will enable it:
.. console::
@load tuning/logs-to-elasticsearch
With that, Bro will now write most of its logs into ElasticSearch in addition
to maintaining the Ascii logs like it would do by default. That script has
some tunable options for choosing which logs to send to ElasticSearch, refer
to the autogenerated script documentation for those options.
There is an interface being written specifically to integrate with the data
that Bro outputs into ElasticSearch named Brownian. It can be found here::
https://github.com/grigorescu/Brownian
Tuning
------
A common problem encountered with ElasticSearch is too many files being held
open. The ElasticSearch website has some suggestions on how to increase the
open file limit.
- http://www.elasticsearch.org/tutorials/too-many-open-files/
TODO
----
Lots.
- Perform multicast discovery for server.
- Better error detection.
- Better defaults (don't index loaded-plugins, for instance).
-

6
doc/frameworks/logging.rst
View file

@ -380,11 +380,11 @@ uncommon to need to delete that data before the end of the connection.
Other Writers Other Writers
------------- -------------
Bro supports the following output formats other than ASCII: Bro supports the following built-in output formats other than ASCII:
.. toctree:: .. toctree::
:maxdepth: 1 :maxdepth: 1
logging-dataseries
logging-elasticsearch
logging-input-sqlite logging-input-sqlite
Further formats are available as external plugins.

8
doc/index.rst
View file

@ -45,7 +45,13 @@ Reference Section
script-reference/index.rst script-reference/index.rst
components/index.rst components/index.rst
.. Development
===========
.. toctree::
:maxdepth: 2
devel/plugins.rst
* :ref:`General Index <genindex>` * :ref:`General Index <genindex>`
* :ref:`search` * :ref:`search`

3
doc/install/install.rst
View file

@ -91,7 +91,6 @@ build time:
* LibGeoIP (for geolocating IP addresses) * LibGeoIP (for geolocating IP addresses)
* sendmail (enables Bro and BroControl to send mail) * sendmail (enables Bro and BroControl to send mail)
* gawk (enables all features of bro-cut)
* curl (used by a Bro script that implements active HTTP) * curl (used by a Bro script that implements active HTTP)
* gperftools (tcmalloc is used to improve memory and CPU usage) * gperftools (tcmalloc is used to improve memory and CPU usage)
* ipsumdump (for trace-summary; http://www.cs.ucla.edu/~kohler/ipsumdump) * ipsumdump (for trace-summary; http://www.cs.ucla.edu/~kohler/ipsumdump)
@ -181,7 +180,7 @@ automatically. Finally, use ``make install-aux`` to install some of
the other programs that are in the ``aux/bro-aux`` directory. the other programs that are in the ``aux/bro-aux`` directory.
OpenBSD users, please see our `FAQ OpenBSD users, please see our `FAQ
<http://www.bro.org/documentation/faq.html>`_ if you are having <//www.bro.org/documentation/faq.html>`_ if you are having
problems installing Bro. problems installing Bro.
Finally, if you want to build the Bro documentation (not required, because Finally, if you want to build the Bro documentation (not required, because

43
doc/logs/index.rst
View file

@ -112,6 +112,8 @@ default, including:
As you can see, some log files are specific to a particular protocol, As you can see, some log files are specific to a particular protocol,
while others aggregate information across different types of activity. while others aggregate information across different types of activity.
For a complete list of log files and a description of its purpose,
see :doc:`List of Log Files <../script-reference/list-of-log-files>`.
.. _bro-cut: .. _bro-cut:
@ -250,44 +252,3 @@ protocol, it can have multiple ``GET``/``POST``/etc requests in a
stream and Bro is able to extract and track that information for you, stream and Bro is able to extract and track that information for you,
giving you an in-depth and structured view into HTTP traffic on your giving you an in-depth and structured view into HTTP traffic on your
network. network.
-----------------------
Common Log Files
-----------------------
As a monitoring tool, Bro records a detailed view of the traffic inspected
and the events generated in a series of relevant log files. These files can
later be reviewed for monitoring, auditing and troubleshooting purposes.
In this section we present a brief explanation of the most commonly used log
files generated by Bro including links to descriptions of some of the fields
for each log type.
+-----------------+---------------------------------------+------------------------------+
| Log File | Description | Field Descriptions |
+=================+=======================================+==============================+
| http.log | Shows all HTTP requests and replies | :bro:type:`HTTP::Info` |
+-----------------+---------------------------------------+------------------------------+
| ftp.log | Records FTP activity | :bro:type:`FTP::Info` |
+-----------------+---------------------------------------+------------------------------+
| ssl.log | Records SSL sessions including | :bro:type:`SSL::Info` |
| | certificates used | |
+-----------------+---------------------------------------+------------------------------+
| known_certs.log | Includes SSL certificates used | :bro:type:`Known::CertsInfo` |
+-----------------+---------------------------------------+------------------------------+
| smtp.log | Summarizes SMTP traffic on a network | :bro:type:`SMTP::Info` |
+-----------------+---------------------------------------+------------------------------+
| dns.log | Shows all DNS activity on a network | :bro:type:`DNS::Info` |
+-----------------+---------------------------------------+------------------------------+
| conn.log | Records all connections seen by Bro | :bro:type:`Conn::Info` |
+-----------------+---------------------------------------+------------------------------+
| dpd.log | Shows network activity on | :bro:type:`DPD::Info` |
| | non-standard ports | |
+-----------------+---------------------------------------+------------------------------+
| files.log | Records information about all files | :bro:type:`Files::Info` |
| | transmitted over the network | |
+-----------------+---------------------------------------+------------------------------+
| weird.log | Records unexpected protocol-level | :bro:type:`Weird::Info` |
| | activity | |
+-----------------+---------------------------------------+------------------------------+

2
doc/quickstart/index.rst
View file

@ -1,5 +1,5 @@
.. _FAQ: http://www.bro.org/documentation/faq.html .. _FAQ: //www.bro.org/documentation/faq.html
.. _quickstart: .. _quickstart:

232
doc/script-reference/attributes.rst Normal file
View file

@ -0,0 +1,232 @@
Attributes
==========
The Bro scripting language supports the following attributes.
+-----------------------------+-----------------------------------------------+
| Name | Description |
+=============================+===============================================+
| :bro:attr:`&redef` |Redefine a global constant or extend a type. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&priority` |Specify priority for event handler or hook. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&log` |Mark a record field as to be written to a log. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&optional` |Allow a record field value to be missing. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&default` |Specify a default value. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&add_func` |Specify a function to call for each "redef +=".|
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&delete_func` |Same as "&add_func", except for "redef -=". |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&expire_func` |Specify a function to call when container |
| |element expires. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&read_expire` |Specify a read timeout interval. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&write_expire` |Specify a write timeout interval. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&create_expire` |Specify a creation timeout interval. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&synchronized` |Synchronize a variable across nodes. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&persistent` |Make a variable persistent (written to disk). |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&rotate_interval`|Rotate a file after specified interval. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&rotate_size` |Rotate a file after specified file size. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&encrypt` |Encrypt a file when writing to disk. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&raw_output` |Open file in raw mode (chars. are not escaped).|
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&mergeable` |Prefer set union for synchronized state. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&group` |Group event handlers to activate/deactivate. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&error_handler` |Used internally for reporter framework events. |
+-----------------------------+-----------------------------------------------+
| :bro:attr:`&type_column` |Used by input framework for "port" type. |
+-----------------------------+-----------------------------------------------+
Here is a more detailed explanation of each attribute:
.. bro:attr:: &redef
Allows for redefinition of initial values of global objects declared as
constant.
In this example, the constant (assuming it is global) can be redefined
with a :bro:keyword:`redef` at some later point::
const clever = T &redef;
.. bro:attr:: &priority
Specifies the execution priority (as a signed integer) of a hook or
event handler. Higher values are executed before lower ones. The
default value is 0. Example::
event bro_init() &priority=10
{
print "high priority";
}
.. bro:attr:: &log
Writes a :bro:type:`record` field to the associated log stream.
.. bro:attr:: &optional
Allows a record field value to be missing (i.e., neither initialized nor
ever assigned a value).
In this example, the record could be instantiated with either
"myrec($a=127.0.0.1)" or "myrec($a=127.0.0.1, $b=80/tcp)"::
type myrec: record { a: addr; b: port &optional; };
The ``?$`` operator can be used to check if a record field has a value or
not (it returns a ``bool`` value of ``T`` if the field has a value,
and ``F`` if not).
.. bro:attr:: &default
Specifies a default value for a record field, container element, or a
function/hook/event parameter.
In this example, the record could be instantiated with either
"myrec($a=5, $c=3.14)" or "myrec($a=5, $b=53/udp, $c=3.14)"::
type myrec: record { a: count; b: port &default=80/tcp; c: double; };
In this example, the table will return the string ``"foo"`` for any
attempted access to a non-existing index::
global mytable: table[count] of string &default="foo";
When used with function/hook/event parameters, all of the parameters
with the "&default" attribute must come after all other parameters.
For example, the following function could be called either as "myfunc(5)"
or as "myfunc(5, 53/udp)"::
function myfunc(a: count, b: port &default=80/tcp)
{
print a, b;
}
.. bro:attr:: &add_func
Can be applied to an identifier with &redef to specify a function to
be called any time a "redef <id> += ..." declaration is parsed. The
function takes two arguments of the same type as the identifier, the first
being the old value of the variable and the second being the new
value given after the "+=" operator in the "redef" declaration. The
return value of the function will be the actual new value of the
variable after the "redef" declaration is parsed.
.. bro:attr:: &delete_func
Same as :bro:attr:`&add_func`, except for :bro:keyword:`redef` declarations
that use the "-=" operator.
.. bro:attr:: &expire_func
Called right before a container element expires. The function's
first parameter is of the same type of the container and the second
parameter the same type of the container's index. The return
value is an :bro:type:`interval` indicating the amount of additional
time to wait before expiring the container element at the given
index (which will trigger another execution of this function).
.. bro:attr:: &read_expire
Specifies a read expiration timeout for container elements. That is,
the element expires after the given amount of time since the last
time it has been read. Note that a write also counts as a read.
.. bro:attr:: &write_expire
Specifies a write expiration timeout for container elements. That
is, the element expires after the given amount of time since the
last time it has been written.
.. bro:attr:: &create_expire
Specifies a creation expiration timeout for container elements. That
is, the element expires after the given amount of time since it has
been inserted into the container, regardless of any reads or writes.
.. bro:attr:: &synchronized
Synchronizes variable accesses across nodes. The value of a
``&synchronized`` variable is automatically propagated to all peers
when it changes.
.. bro:attr:: &persistent
Makes a variable persistent, i.e., its value is written to disk (per
default at shutdown time).
.. bro:attr:: &rotate_interval
Rotates a file after a specified interval.
.. bro:attr:: &rotate_size
Rotates a file after it has reached a given size in bytes.
.. bro:attr:: &encrypt
Encrypts files right before writing them to disk.
.. bro:attr:: &raw_output
Opens a file in raw mode, i.e., non-ASCII characters are not
escaped.
.. bro:attr:: &mergeable
Prefers merging sets on assignment for synchronized state. This
attribute is used in conjunction with :bro:attr:`&synchronized`
container types: when the same container is updated at two peers
with different values, the propagation of the state causes a race
condition, where the last update succeeds. This can cause
inconsistencies and can be avoided by unifying the two sets, rather
than merely overwriting the old value.
.. bro:attr:: &group
Groups event handlers such that those in the same group can be
jointly activated or deactivated.
.. bro:attr:: &error_handler
Internally set on the events that are associated with the reporter
framework: :bro:id:`reporter_info`, :bro:id:`reporter_warning`, and
:bro:id:`reporter_error`. It prevents any handlers of those events
from being able to generate reporter messages that go through any of
those events (i.e., it prevents an infinite event recursion). Instead,
such nested reporter messages are output to stderr.
.. bro:attr:: &type_column
Used by the input framework. It can be used on columns of type
:bro:type:`port` (such a column only contains the port number) and
specifies the name of an additional column in
the input file which specifies the protocol of the port (tcp/udp/icmp).
In the following example, the input file would contain four columns
named "ip", "srcp", "proto", and "msg"::
type Idx: record {
ip: addr;
};
type Val: record {
srcp: port &type_column = "proto";
msg: string;
};

173
doc/script-reference/directives.rst Normal file
View file

@ -0,0 +1,173 @@
Directives
==========
The Bro scripting language supports a number of directives that can
affect which scripts will be loaded or which lines in a script will be
executed. Directives are evaluated before script execution begins.
.. bro:keyword:: @DEBUG
TODO
.. bro:keyword:: @DIR
Expands to the directory pathname where the current script is located.
Example::
print "Directory:", @DIR;
.. bro:keyword:: @FILENAME
Expands to the filename of the current script.
Example::
print "File:", @FILENAME;
.. bro:keyword:: @load
Loads the specified Bro script, specified as the relative pathname
of the file (relative to one of the directories in Bro's file search path).
If the Bro script filename ends with ".bro", then you don't need to
specify the file extension. The filename cannot contain any whitespace.
In this example, Bro will try to load a script
"policy/misc/capture-loss.bro" by looking in each directory in the file
search path (the file search path can be changed by setting the BROPATH
environment variable)::
@load policy/misc/capture-loss
If you specify the name of a directory instead of a filename, then
Bro will try to load a file in that directory called "__load__.bro"
(presumably that file will contain additional "@load" directives).
In this example, Bro will try to load a file "tuning/defaults/__load__.bro"
by looking in each directory in the file search path::
@load tuning/defaults
The purpose of this directive is to ensure that all script dependencies
are satisfied, and to avoid having to list every needed Bro script
on the command-line. Bro keeps track of which scripts have been
loaded, so it is not an error to load a script more than once (once
a script has been loaded, any subsequent "@load" directives
for that script are ignored).
.. bro:keyword:: @load-sigs
This works similarly to "@load", except that in this case the filename
represents a signature file (not a Bro script). If the signature filename
ends with ".sig", then you don't need to specify the file extension
in the "@load-sigs" directive. The filename cannot contain any
whitespace.
In this example, Bro will try to load a signature file
"base/protocols/ssl/dpd.sig"::
@load-sigs base/protocols/ssl/dpd
The format for a signature file is explained in the documentation for the
`Signature Framework <../frameworks/signatures.html>`_.
.. bro:keyword:: @unload
This specifies a Bro script that we don't want to load (so a subsequent
attempt to load the specified script will be skipped). However,
if the specified script has already been loaded, then this directive
has no affect.
In the following example, if the "policy/misc/capture-loss.bro" script
has not been loaded yet, then Bro will not load it::
@unload policy/misc/capture-loss
.. bro:keyword:: @prefixes
Specifies a filename prefix to use when looking for script files
to load automatically. The prefix cannot contain any whitespace.
In the following example, the prefix "cluster" is used and all prefixes
that were previously specified are not used::
@prefixes = cluster
In the following example, the prefix "cluster-manager" is used in
addition to any previously-specified prefixes::
@prefixes += cluster-manager
The way this works is that after Bro parses all script files, then for each
loaded script Bro will take the absolute path of the script and then
it removes the portion of the directory path that is in Bro's file
search path. Then it replaces each "/" character with a period "."
and then prepends the prefix (specified in the "@prefixes" directive)
followed by a period. The resulting filename is searched for in each
directory in Bro's file search path. If a matching file is found, then
the file is automatically loaded.
For example, if a script called "local.bro" has been loaded, and a prefix
of "test" was specified, then Bro will look for a file named
"test.local.bro" in each directory of Bro's file search path.
An alternative way to specify prefixes is to use the "-p" Bro
command-line option.
.. bro:keyword:: @if
The specified expression must evaluate to type :bro:type:`bool`. If the
value is true, then the following script lines (up to the next "@else"
or "@endif") are available to be executed.
Example::
@if ( ver == 2 )
print "version 2 detected";
@endif
.. bro:keyword:: @ifdef
This works like "@if", except that the result is true if the specified
identifier is defined.
Example::
@ifdef ( pi )
print "pi is defined";
@endif
.. bro:keyword:: @ifndef
This works exactly like "@ifdef", except that the result is true if the
specified identifier is not defined.
Example::
@ifndef ( pi )
print "pi is not defined";
@endif
.. bro:keyword:: @else
This directive is optional after an "@if", "@ifdef", or
"@ifndef". If present, it provides an else clause.
Example::
@ifdef ( pi )
print "pi is defined";
@else
print "pi is not defined";
@endif
.. bro:keyword:: @endif
This directive is required to terminate each "@if", "@ifdef", or
"@ifndef".

9
doc/script-reference/index.rst
View file

@ -5,10 +5,17 @@ Script Reference
.. toctree:: .. toctree::
:maxdepth: 1 :maxdepth: 1
operators
types
attributes
statements
directives
log-files
notices notices
proto-analyzers proto-analyzers
file-analyzers file-analyzers
builtins
packages packages
scripts scripts
Broxygen Example Script </scripts/broxygen/example.bro> Broxygen Example Script </scripts/broxygen/example.bro>

148
doc/script-reference/log-files.rst Normal file
View file

@ -0,0 +1,148 @@
=========
Log Files
=========
Listed below are the log files generated by Bro, including a brief description
of the log file and links to descriptions of the fields for each log
type.
Network Protocols
-----------------
+----------------------------+---------------------------------------+---------------------------------+
| Log File | Description | Field Descriptions |
+============================+=======================================+=================================+
| conn.log | TCP/UDP/ICMP connections | :bro:type:`Conn::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| dhcp.log | DHCP leases | :bro:type:`DHCP::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| dnp3.log | DNP3 requests and replies | :bro:type:`DNP3::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| dns.log | DNS activity | :bro:type:`DNS::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| ftp.log | FTP activity | :bro:type:`FTP::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| http.log | HTTP requests and replies | :bro:type:`HTTP::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| irc.log | IRC commands and responses | :bro:type:`IRC::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| modbus.log | Modbus commands and responses | :bro:type:`Modbus::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| modbus_register_change.log | Tracks changes to Modbus holding | :bro:type:`Modbus::MemmapInfo` |
| | registers | |
+----------------------------+---------------------------------------+---------------------------------+
| radius.log | RADIUS authentication attempts | :bro:type:`RADIUS::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| smtp.log | SMTP transactions | :bro:type:`SMTP::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| snmp.log | SNMP messages | :bro:type:`SNMP::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| socks.log | SOCKS proxy requests | :bro:type:`SOCKS::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| ssh.log | SSH connections | :bro:type:`SSH::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| ssl.log | SSL/TLS handshake info | :bro:type:`SSL::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| syslog.log | Syslog messages | :bro:type:`Syslog::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| tunnel.log | Tunneling protocol events | :bro:type:`Tunnel::Info` |
+----------------------------+---------------------------------------+---------------------------------+
Files
-----
+----------------------------+---------------------------------------+---------------------------------+
| Log File | Description | Field Descriptions |
+============================+=======================================+=================================+
| files.log | File analysis results | :bro:type:`Files::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| x509.log | X.509 certificate info | :bro:type:`X509::Info` |
+----------------------------+---------------------------------------+---------------------------------+
Detection
---------
+----------------------------+---------------------------------------+---------------------------------+
| Log File | Description | Field Descriptions |
+============================+=======================================+=================================+
| intel.log | Intelligence data matches | :bro:type:`Intel::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| notice.log | Bro notices | :bro:type:`Notice::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| notice_alarm.log | The alarm stream | :bro:enum:`Notice::ACTION_ALARM`|
+----------------------------+---------------------------------------+---------------------------------+
| signatures.log | Signature matches | :bro:type:`Signatures::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| traceroute.log | Traceroute detection | :bro:type:`Traceroute::Info` |
+----------------------------+---------------------------------------+---------------------------------+
Network Observations
--------------------
+----------------------------+---------------------------------------+---------------------------------+
| Log File | Description | Field Descriptions |
+============================+=======================================+=================================+
| app_stats.log | Web app usage statistics | :bro:type:`AppStats::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| known_certs.log | SSL certificates | :bro:type:`Known::CertsInfo` |
+----------------------------+---------------------------------------+---------------------------------+
| known_devices.log | MAC addresses of devices on the | :bro:type:`Known::DevicesInfo` |
| | network | |
+----------------------------+---------------------------------------+---------------------------------+
| known_hosts.log | Hosts that have completed TCP | :bro:type:`Known::HostsInfo` |
| | handshakes | |
+----------------------------+---------------------------------------+---------------------------------+
| known_modbus.log | Modbus masters and slaves | :bro:type:`Known::ModbusInfo` |
+----------------------------+---------------------------------------+---------------------------------+
| known_services.log | Services running on hosts | :bro:type:`Known::ServicesInfo` |
+----------------------------+---------------------------------------+---------------------------------+
| software.log | Software being used on the network | :bro:type:`Software::Info` |
+----------------------------+---------------------------------------+---------------------------------+
Miscellaneous
-------------
+----------------------------+---------------------------------------+---------------------------------+
| Log File | Description | Field Descriptions |
+============================+=======================================+=================================+
| barnyard2.log | Alerts received from Barnyard2 | :bro:type:`Barnyard2::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| dpd.log | Dynamic protocol detection failures | :bro:type:`DPD::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| unified2.log | Interprets Snort's unified output | :bro:type:`Unified2::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| weird.log | Unexpected network-level activity | :bro:type:`Weird::Info` |
+----------------------------+---------------------------------------+---------------------------------+
Bro Diagnostics
---------------
+----------------------------+---------------------------------------+---------------------------------+
| Log File | Description | Field Descriptions |
+============================+=======================================+=================================+
| capture_loss.log | Packet loss rate | :bro:type:`CaptureLoss::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| cluster.log | Bro cluster messages | :bro:type:`Cluster::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| communication.log | Communication events between Bro or | :bro:type:`Communication::Info` |
| | Broccoli instances | |
+----------------------------+---------------------------------------+---------------------------------+
| loaded_scripts.log | Shows all scripts loaded by Bro | :bro:type:`LoadedScripts::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| packet_filter.log | List packet filters that were applied | :bro:type:`PacketFilter::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| prof.log | Profiling statistics (to create this | N/A |
| | log, load policy/misc/profiling.bro) | |
+----------------------------+---------------------------------------+---------------------------------+
| reporter.log | Internal error/warning/info messages | :bro:type:`Reporter::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| stats.log | Memory/event/packet/lag statistics | :bro:type:`Stats::Info` |
+----------------------------+---------------------------------------+---------------------------------+
| stderr.log | Captures standard error when Bro is | N/A |
| | started from BroControl | |
+----------------------------+---------------------------------------+---------------------------------+
| stdout.log | Captures standard output when Bro is | N/A |
| | started from BroControl | |
+----------------------------+---------------------------------------+---------------------------------+

191
doc/script-reference/operators.rst Normal file
View file

@ -0,0 +1,191 @@
Operators
=========
The Bro scripting language supports the following operators. Note that
each data type only supports a subset of these operators. For more
details, see the documentation about the `data types <types.html>`_.
Relational operators
--------------------
The relational operators evaluate to type :bro:type:`bool`.
+------------------------------+--------------+
| Name | Syntax |
+==============================+==============+
| Equality | *a* == *b* |
+------------------------------+--------------+
| Inequality | *a* != *b* |
+------------------------------+--------------+
| Less than | *a* < *b* |
+------------------------------+--------------+
| Less than or equal | *a* <= *b* |
+------------------------------+--------------+
| Greater than | *a* > *b* |
+------------------------------+--------------+
| Greater than or equal | *a* >= *b* |
+------------------------------+--------------+
Logical operators
-----------------
The logical operators require operands of type :bro:type:`bool`, and
evaluate to type :bro:type:`bool`.
+------------------------------+--------------+
| Name | Syntax |
+==============================+==============+
| Logical AND | *a* && *b* |
+------------------------------+--------------+
| Logical OR | *a* \|\| *b* |
+------------------------------+--------------+
| Logical NOT | ! *a* |
+------------------------------+--------------+
Arithmetic operators
--------------------
+------------------------------+-------------+-------------------------------+
| Name | Syntax | Notes |
+==============================+=============+===============================+
| Addition | *a* + *b* | For :bro:type:`string` |
| | | operands, this performs |
| | | string concatenation. |
+------------------------------+-------------+-------------------------------+
| Subtraction | *a* - *b* | |
+------------------------------+-------------+-------------------------------+
| Multiplication | *a* \* *b* | |
+------------------------------+-------------+-------------------------------+
| Division | *a* / *b* | For :bro:type:`int` or |
| | | :bro:type:`count` operands, |
| | | the fractional part of the |
| | | result is dropped. |
+------------------------------+-------------+-------------------------------+
| Modulo | *a* % *b* | Operand types cannot be |
| | | "double". |
+------------------------------+-------------+-------------------------------+
| Unary plus | \+ *a* | |
+------------------------------+-------------+-------------------------------+
| Unary minus | \- *a* | |
+------------------------------+-------------+-------------------------------+
| Pre-increment | ++ *a* | Operand type cannot be |
| | | "double". |
+------------------------------+-------------+-------------------------------+
| Pre-decrement | ``--`` *a* | Operand type cannot be |
| | | "double". |
+------------------------------+-------------+-------------------------------+
| Absolute value | \| *a* \| | If operand is |
| | | :bro:type:`string`, |
| | | :bro:type:`set`, |
| | | :bro:type:`table`, or |
| | | :bro:type:`vector`, this |
| | | evaluates to number |
| | | of elements. |
+------------------------------+-------------+-------------------------------+
Assignment operators
--------------------
The assignment operators evaluate to the result of the assignment.
+------------------------------+-------------+
| Name | Syntax |
+==============================+=============+
| Assignment | *a* = *b* |
+------------------------------+-------------+
| Addition assignment | *a* += *b* |
+------------------------------+-------------+
| Subtraction assignment | *a* -= *b* |
+------------------------------+-------------+
Record field operators
----------------------
The record field operators take a :bro:type:`record` as the first operand,
and a field name as the second operand. For both operators, the specified
field name must be in the declaration of the record type.
+------------------------------+-------------+-------------------------------+
| Name | Syntax | Notes |
+==============================+=============+===============================+
| Field access | *a* $ *b* | |
+------------------------------+-------------+-------------------------------+
| Field value existence test | *a* ?$ *b* | Evaluates to type |
| | | :bro:type:`bool`. |
| | | True if the specified field |
| | | has been assigned a value, or |
| | | false if not. |
+------------------------------+-------------+-------------------------------+
Other operators
---------------
+--------------------------------+-------------------+------------------------+
| Name | Syntax | Notes |
+================================+===================+========================+
| Membership test | *a* in *b* |Evaluates to type |
| | |:bro:type:`bool`. Do not|
| | |confuse this use of "in"|
| | |with that used in a |
| | |:bro:keyword:`for` |
| | |statement. |
+--------------------------------+-------------------+------------------------+
| Non-membership test | *a* !in *b* |This is the logical NOT |
| | |of the "in" operator. |
| | |For example: "a !in b" |
| | |is equivalent to |
| | |"!(a in b)". |
+--------------------------------+-------------------+------------------------+
| Table or vector element access | *a* [ *b* ] |This operator can also |
| | |be used with a |
| | |:bro:type:`set`, but |
| | |only with the |
| | |:bro:keyword:`add` or |
| | |:bro:keyword:`delete` |
| | |statement. |
+--------------------------------+-------------------+------------------------+
| Substring extraction | *a* [ *b* : *c* ] |See the |
| | |:bro:type:`string` type |
| | |for more details. |
+--------------------------------+-------------------+------------------------+
| Create a deep copy | copy ( *a* ) |This is relevant only |
| | |for data types that are |
| | |assigned by reference, |
| | |such as |
| | |:bro:type:`vector`, |
| | |:bro:type:`set`, |
| | |:bro:type:`table`, |
| | |and :bro:type:`record`. |
+--------------------------------+-------------------+------------------------+
| Module namespace access | *a* \:\: *b* |The first operand is the|
| | |module name, and the |
| | |second operand is an |
| | |identifier that refers |
| | |to a global variable, |
| | |enumeration constant, or|
| | |user-defined type that |
| | |was exported from the |
| | |module. |
+--------------------------------+-------------------+------------------------+
| Conditional | *a* ? *b* : *c* |The first operand must |
| | |evaluate to type |
| | |:bro:type:`bool`. |
| | |If true, then the |
| | |second expression is |
| | |evaluated and is the |
| | |result of the entire |
| | |expression. Otherwise, |
| | |the third expression is |
| | |evaluated and is the |
| | |result of the entire |
| | |expression. The types of|
| | |the second and third |
| | |operands must be |
| | |compatible. |
+--------------------------------+-------------------+------------------------+

602
doc/script-reference/statements.rst Normal file
View file

@ -0,0 +1,602 @@
Declarations and Statements
===========================
The Bro scripting language supports the following declarations and
statements.
Declarations
~~~~~~~~~~~~
+----------------------------+-----------------------------+
| Name | Description |
+============================+=============================+
| :bro:keyword:`module` | Change the current module |
+----------------------------+-----------------------------+
| :bro:keyword:`export` | Export identifiers from the |
| | current module |
+----------------------------+-----------------------------+
| :bro:keyword:`global` | Declare a global variable |
+----------------------------+-----------------------------+
| :bro:keyword:`const` | Declare a constant |
+----------------------------+-----------------------------+
| :bro:keyword:`type` | Declare a user-defined type |
+----------------------------+-----------------------------+
| :bro:keyword:`redef` | Redefine a global value or |
| | extend a user-defined type |
+----------------------------+-----------------------------+
| `function/event/hook`_ | Declare a function, event |
| | handler, or hook |
+----------------------------+-----------------------------+
Statements
~~~~~~~~~~
+----------------------------+------------------------+
| Name | Description |
+============================+========================+
| :bro:keyword:`local` | Declare a local |
| | variable |
+----------------------------+------------------------+
| :bro:keyword:`add`, | Add or delete |
| :bro:keyword:`delete` | elements |
+----------------------------+------------------------+
| :bro:keyword:`print` | Print to stdout or a |
| | file |
+----------------------------+------------------------+
| :bro:keyword:`for`, | Loop over each |
| :bro:keyword:`next`, | element in a container |
| :bro:keyword:`break` | object |
+----------------------------+------------------------+
| :bro:keyword:`if` | Evaluate boolean |
| | expression and if true,|
| | execute a statement |
+----------------------------+------------------------+
| :bro:keyword:`switch`, | Evaluate expression |
| :bro:keyword:`break`, | and execute statement |
| :bro:keyword:`fallthrough` | with a matching value |
+----------------------------+------------------------+
| :bro:keyword:`when` | Asynchronous execution |
+----------------------------+------------------------+
| :bro:keyword:`event`, | Invoke or schedule |
| :bro:keyword:`schedule` | an event handler |
+----------------------------+------------------------+
| :bro:keyword:`return` | Return from function, |
| | hook, or event handler |
+----------------------------+------------------------+
Declarations
------------
The following global declarations cannot occur within a function, hook, or
event handler. Also, these declarations cannot appear after any statements
that are outside of a function, hook, or event handler.
.. bro:keyword:: module
The "module" keyword is used to change the current module. This
affects the scope of any subsequently declared global identifiers.
Example::
module mymodule;
If a global identifier is declared after a "module" declaration,
then its scope ends at the end of the current Bro script or at the
next "module" declaration, whichever comes first. However, if a
global identifier is declared after a "module" declaration, but inside
an :bro:keyword:`export` block, then its scope ends at the end of the
last loaded Bro script, but it must be referenced using the namespace
operator (``::``) in other modules.
There can be any number of "module" declarations in a Bro script.
The same "module" declaration can appear in any number of different
Bro scripts.
.. bro:keyword:: export
An "export" block contains one or more declarations
(no statements are allowed in an "export" block) that the current
module is exporting. This enables these global identifiers to be visible
in other modules (but not prior to their declaration) via the namespace
operator (``::``). See the :bro:keyword:`module` keyword for a more
detailed explanation.
Example::
export {
redef enum Log::ID += { LOG };
type Info: record {
ts: time &log;
uid: string &log;
};
const conntime = 30sec &redef;
}
Note that the braces in an "export" block are always required
(they do not indicate a compound statement). Also, no semicolon is
needed to terminate an "export" block.
.. bro:keyword:: global
Variables declared with the "global" keyword will be global.
If a type is not specified, then an initializer is required so that
the type can be inferred. Likewise, if an initializer is not supplied,
then the type must be specified. Example::
global pi = 3.14;
global hosts: set[addr];
global ciphers: table[string] of string = table();
Variable declarations outside of any function, hook, or event handler are
required to use this keyword (unless they are declared with the
:bro:keyword:`const` keyword). Definitions of functions, hooks, and
event handlers are not allowed to use the "global"
keyword (they already have global scope), except function declarations
where no function body is supplied use the "global" keyword.
The scope of a global variable begins where the declaration is located,
and extends through all remaining Bro scripts that are loaded (however,
see the :bro:keyword:`module` keyword for an explanation of how modules
change the visibility of global identifiers).
.. bro:keyword:: const
A variable declared with the "const" keyword will be constant.
Variables declared as constant are required to be initialized at the
time of declaration. Example::
const pi = 3.14;
const ssh_port: port = 22/tcp;
The value of a constant cannot be changed later (the only
exception is if the variable is global and has the :bro:attr:`&redef`
attribute, then its value can be changed only with a :bro:keyword:`redef`).
The scope of a constant is local if the declaration is in a
function, hook, or event handler, and global otherwise.
Note that the "const" keyword cannot be used with either the "local"
or "global" keywords (i.e., "const" replaces "local" and "global").
.. bro:keyword:: type
The "type" keyword is used to declare a user-defined type. The name
of this new type has global scope and can be used anywhere a built-in
type name can occur.
The "type" keyword is most commonly used when defining a
:bro:type:`record` or an :bro:type:`enum`, but is also useful when
dealing with more complex types.
Example::
type mytype: table[count] of table[addr, port] of string;
global myvar: mytype;
.. bro:keyword:: redef
There are three ways that "redef" can be used: to change the value of
a global variable, to extend a record type or enum type, or to specify
a new event handler body that replaces all those that were previously
defined.
If you're using "redef" to change a global variable (defined using either
:bro:keyword:`const` or :bro:keyword:`global`), then the variable that you
want to change must have the :bro:attr:`&redef` attribute. If the variable
you're changing is a table, set, or pattern, you can use ``+=`` to add
new elements, or you can use ``=`` to specify a new value (all previous
contents of the object are removed). If the variable you're changing is a
set or table, then you can use the ``-=`` operator to remove the
specified elements (nothing happens for specified elements that don't
exist). If the variable you are changing is not a table, set, or pattern,
then you must use the ``=`` operator.
Examples::
redef pi = 3.14;
If you're using "redef" to extend a record or enum, then you must
use the ``+=`` assignment operator.
For an enum, you can add more enumeration constants, and for a record
you can add more record fields (however, each record field in the "redef"
must have either the :bro:attr:`&optional` or :bro:attr:`&default`
attribute).
Examples::
redef enum color += { Blue, Red };
redef record MyRecord += { n2:int &optional; s2:string &optional; };
If you're using "redef" to specify a new event handler body that
replaces all those that were previously defined (i.e., any subsequently
defined event handler body will not be affected by this "redef"), then
the syntax is the same as a regular event handler definition except for
the presence of the "redef" keyword.
Example::
redef event myevent(s:string) { print "Redefined", s; }
.. _function/event/hook:
**function/event/hook**
For details on how to declare a :bro:type:`function`,
:bro:type:`event` handler, or :bro:type:`hook`,
see the documentation for those types.
Statements
----------
Each statement in a Bro script must be terminated with a semicolon (with a
few exceptions noted below). An individual statement can span multiple
lines.
All statements (except those contained within a function, hook, or event
handler) must appear after all global declarations.
Here are the statements that the Bro scripting language supports.
.. bro:keyword:: add
The "add" statement is used to add an element to a :bro:type:`set`.
Nothing happens if the specified element already exists in the set.
Example::
local myset: set[string];
add myset["test"];
.. bro:keyword:: break
The "break" statement is used to break out of a :bro:keyword:`switch` or
:bro:keyword:`for` statement.
.. bro:keyword:: delete
The "delete" statement is used to remove an element from a
:bro:type:`set` or :bro:type:`table`. Nothing happens if the
specified element does not exist in the set or table.
Example::
local myset = set("this", "test");
local mytable = table(["key1"] = 80/tcp, ["key2"] = 53/udp);
delete myset["test"];
delete mytable["key1"];
.. bro:keyword:: event
The "event" statement immediately queues invocation of an event handler.
Example::
event myevent("test", 5);
.. bro:keyword:: fallthrough
The "fallthrough" statement can be used as the last statement in a
"case" block to indicate that execution should continue into the
next "case" or "default" label.
For an example, see the :bro:keyword:`switch` statement.
.. bro:keyword:: for
A "for" loop iterates over each element in a string, set, vector, or
table and executes a statement for each iteration.
For each iteration of the loop, a loop variable will be assigned to an
element if the expression evaluates to a string or set, or an index if
the expression evaluates to a vector or table. Then the statement
is executed. However, the statement will not be executed if the expression
evaluates to an object with no elements.
If the expression is a table or a set with more than one index, then the
loop variable must be specified as a comma-separated list of different
loop variables (one for each index), enclosed in brackets.
A :bro:keyword:`break` statement can be used at any time to immediately
terminate the "for" loop, and a :bro:keyword:`next` statement can be
used to skip to the next loop iteration.
Note that the loop variable in a "for" statement is not allowed to be
a global variable, and it does not need to be declared prior to the "for"
statement. The type will be inferred from the elements of the
expression.
Example::
local myset = set(80/tcp, 81/tcp);
local mytable = table([10.0.0.1, 80/tcp]="s1", [10.0.0.2, 81/tcp]="s2");
for (p in myset)
print p;
for ([i,j] in mytable) {
if (mytable[i,j] == "done")
break;
if (mytable[i,j] == "skip")
next;
print i,j;
}
.. bro:keyword:: if
Evaluates a given expression, which must yield a :bro:type:`bool` value.
If true, then a specified statement is executed. If false, then
the statement is not executed. Example::
if ( x == 2 ) print "x is 2";
However, if the expression evaluates to false and if an "else" is
provided, then the statement following the "else" is executed. Example::
if ( x == 2 )
print "x is 2";
else
print "x is not 2";
.. bro:keyword:: local
A variable declared with the "local" keyword will be local. If a type
is not specified, then an initializer is required so that the type can
be inferred. Likewise, if an initializer is not supplied, then the
type must be specified.
Examples::
local x1 = 5.7;
local x2: double;
local x3: double = 5.7;
Variable declarations inside a function, hook, or event handler are
required to use this keyword (the only two exceptions are variables
declared with :bro:keyword:`const`, and variables implicitly declared in a
:bro:keyword:`for` statement).
The scope of a local variable starts at the location where it is declared
and persists to the end of the function, hook,
or event handler in which it is declared (this is true even if the
local variable was declared within a `compound statement`_ or is the loop
variable in a "for" statement).
.. bro:keyword:: next
The "next" statement can only appear within a :bro:keyword:`for` loop.
It causes execution to skip to the next iteration.
For an example, see the :bro:keyword:`for` statement.
.. bro:keyword:: print
The "print" statement takes a comma-separated list of one or more
expressions. Each expression in the list is evaluated and then converted
to a string. Then each string is printed, with each string separated by
a comma in the output.
Examples::
print 3.14;
print "Results", x, y;
By default, the "print" statement writes to the standard
output (stdout). However, if the first expression is of type
:bro:type:`file`, then "print" writes to that file.
If a string contains non-printable characters (i.e., byte values that are
not in the range 32 - 126), then the "print" statement converts each
non-printable character to an escape sequence before it is printed.
For more control over how the strings are formatted, see the :bro:id:`fmt`
function.
.. bro:keyword:: return
The "return" statement immediately exits the current function, hook, or
event handler. For a function, the specified expression (if any) is
evaluated and returned. A "return" statement in a hook or event handler
cannot return a value because event handlers and hooks do not have
return types.
Examples::
function my_func(): string
{
return "done";
}
event my_event(n: count)
{
if ( n == 0 ) return;
print n;
}
There is a special form of the "return" statement that is only allowed
in functions. Syntactically, it looks like a :bro:keyword:`when` statement
immediately preceded by the "return" keyword. This form of the "return"
statement is used to specify a function that delays its result (such a
function can only be called in the expression of a :bro:keyword:`when`
statement). The function returns at the time the "when"
statement's condition becomes true, and the function returns the value
that the "when" statement's body returns (or if the condition does
not become true within the specified timeout interval, then the function
returns the value that the "timeout" block returns).
Example::
global X: table[string] of count;
function a() : count
{
# This delays until condition becomes true.
return when ( "a" in X )
{
return X["a"];
}
timeout 30 sec
{
return 0;
}
}
event bro_init()
{
# Installs a trigger which fires if a() returns 42.
when ( a() == 42 )
print "expected result";
print "Waiting for a() to return...";
X["a"] = 42;
}
.. bro:keyword:: schedule
The "schedule" statement is used to raise a specified event with
specified parameters at a later time specified as an :bro:type:`interval`.
Example::
schedule 30sec { myevent(x, y, z) };
Note that the braces are always required (they do not indicate a
`compound statement`_).
Note that "schedule" is actually an expression that returns a value
of type "timer", but in practice the return value is not used.
.. bro:keyword:: switch
A "switch" statement evaluates a given expression and jumps to
the first "case" label which contains a matching value (the result of the
expression must be type-compatible with all of the values in all of the
"case" labels). If there is no matching value, then execution jumps to
the "default" label instead, and if there is no "default" label then
execution jumps out of the "switch" block.
Here is an example (assuming that "get_day_of_week" is a
function that returns a string)::
switch get_day_of_week()
{
case "Sa", "Su":
print "weekend";
fallthrough;
case "Mo", "Tu", "We", "Th", "Fr":
print "valid result";
break;
default:
print "invalid result";
break;
}
A "switch" block can have any number of "case" labels, and one
optional "default" label.
A "case" label can have a comma-separated list of
more than one value. A value in a "case" label can be an expression,
but it must be a constant expression (i.e., the expression can consist
only of constants).
Each "case" and the "default" block must
end with either a :bro:keyword:`break`, :bro:keyword:`fallthrough`, or
:bro:keyword:`return` statement (although "return" is allowed only
if the "switch" statement is inside a function, hook, or event handler).
If a "case" (or "default") block contain more than one statement, then
there is no need to wrap them in braces.
Note that the braces in a "switch" statement are always required (these
do not indicate the presence of a `compound statement`_), and that no
semicolon is needed at the end of a "switch" statement.
.. bro:keyword:: when
Evaluates a given expression, which must result in a value of type
:bro:type:`bool`. When the value of the expression becomes available
and if the result is true, then a specified statement is executed.
In the following example, if the expression evaluates to true, then
the "print" statement is executed::
when ( (local x = foo()) && x == 42 )
print x;
However, if a timeout is specified, and if the expression does not
evaluate to true within the specified timeout interval, then the
statement following the "timeout" keyword is executed::
when ( (local x = foo()) && x == 42 )
print x;
timeout 5sec {
print "timeout";
}
Note that when a timeout is specified the braces are
always required (these do not indicate a `compound statement`_).
The expression in a "when" statement can contain a declaration of a local
variable but only if the declaration is written in the form
"local *var* = *init*" (example: "local x = myfunction()"). This form
of a local declaration is actually an expression, the result of which
is always a boolean true value.
The expression in a "when" statement can contain an asynchronous function
call such as :bro:id:`lookup_hostname` (in fact, this is the only place
such a function can be called), but it can also contain an ordinary
function call. When an asynchronous function call is in the expression,
then Bro will continue processing statements in the script following
the "when" statement, and when the result of the function call is available
Bro will finish evaluating the expression in the "when" statement.
See the :bro:keyword:`return` statement for an explanation of how to
create an asynchronous function in a Bro script.
.. _compound statement:
**compound statement**
A compound statement is created by wrapping zero or more statements in
braces ``{ }``. Individual statements inside the braces need to be
terminated by a semicolon, but a semicolon is not needed at the end
(outside of the braces) of a compound statement.
A compound statement is required in order to execute more than one
statement in the body of a :bro:keyword:`for`, :bro:keyword:`if`, or
:bro:keyword:`when` statement.
Example::
if ( x == 2 ) {
print "x is 2";
++x;
}
Note that there are other places in the Bro scripting language that use
braces, but that do not indicate the presence of a compound
statement (these are noted in the documentation).
.. _null:
**null statement**
The null statement (executing it has no effect) consists of just a
semicolon. This might be useful during testing or debugging a Bro script
in places where a statement is required, but it is probably not useful
otherwise.
Example::
if ( x == 2 )
;

657
doc/script-reference/builtins.rst → doc/script-reference/types.rst
View file

@ -1,92 +1,114 @@
Types and Attributes
====================
Types Types
----- =====
Every value in a Bro script has a type (see below for a list of all built-in The Bro scripting language supports the following built-in types:
types). Although Bro variables have static types (meaning that their type
is fixed), their type is inferred from the value to which they are
initially assigned when the variable is declared without an explicit type
name.
Automatic conversions happen when a binary operator has operands of +-----------------------+--------------------+
different types. Automatic conversions are limited to converting between | Name | Description |
numeric types. The numeric types are ``int``, ``count``, and ``double`` +=======================+====================+
(``bool`` is not a numeric type). | :bro:type:`bool` | Boolean |
When an automatic conversion occurs, values are promoted to the "highest" +-----------------------+--------------------+
type in the expression. In general, this promotion follows a simple | :bro:type:`count`, | Numeric types |
hierarchy: ``double`` is highest, ``int`` comes next, and ``count`` is | :bro:type:`int`, | |
lowest. | :bro:type:`double` | |
+-----------------------+--------------------+
| :bro:type:`time`, | Time types |
| :bro:type:`interval` | |
+-----------------------+--------------------+
| :bro:type:`string` | String |
+-----------------------+--------------------+
| :bro:type:`pattern` | Regular expression |
+-----------------------+--------------------+
| :bro:type:`port`, | Network types |
| :bro:type:`addr`, | |
| :bro:type:`subnet` | |
+-----------------------+--------------------+
| :bro:type:`enum` | Enumeration |
| | (user-defined type)|
+-----------------------+--------------------+
| :bro:type:`table`, | Container types |
| :bro:type:`set`, | |
| :bro:type:`vector`, | |
| :bro:type:`record` | |
+-----------------------+--------------------+
| :bro:type:`function`, | Executable types |
| :bro:type:`event`, | |
| :bro:type:`hook` | |
+-----------------------+--------------------+
| :bro:type:`file` | File type (only |
| | for writing) |
+-----------------------+--------------------+
| :bro:type:`opaque` | Opaque type (for |
| | some built-in |
| | functions) |
+-----------------------+--------------------+
| :bro:type:`any` | Any type (for |
| | functions or |
| | containers) |
+-----------------------+--------------------+
The Bro scripting language supports the following built-in types. Here is a more detailed description of each type:
.. bro:type:: void
An internal Bro type (i.e., "void" is not a reserved keyword in the Bro
scripting language) representing the absence of a return type for a
function.
.. bro:type:: bool .. bro:type:: bool
Reflects a value with one of two meanings: true or false. The two Reflects a value with one of two meanings: true or false. The two
``bool`` constants are ``T`` and ``F``. "bool" constants are ``T`` and ``F``.
The ``bool`` type supports the following operators: equality/inequality The "bool" type supports the following operators: equality/inequality
(``==``, ``!=``), logical and/or (``&&``, ``||``), logical (``==``, ``!=``), logical and/or (``&&``, ``||``), logical
negation (``!``), and absolute value (where ``|T|`` is 1, and ``|F|`` is 0). negation (``!``), and absolute value (where ``|T|`` is 1, and ``|F|`` is 0,
and in both cases the result type is :bro:type:`count`).
.. bro:type:: int .. bro:type:: int
A numeric type representing a 64-bit signed integer. An ``int`` constant A numeric type representing a 64-bit signed integer. An "int" constant
is a string of digits preceded by a ``+`` or ``-`` sign, e.g. is a string of digits preceded by a "+" or "-" sign, e.g.
``-42`` or ``+5`` (the "+" sign is optional but see note about type ``-42`` or ``+5`` (the "+" sign is optional but see note about type
inferencing below). An ``int`` constant can also be written in inferencing below). An "int" constant can also be written in
hexadecimal notation (in which case "0x" must be between the sign and hexadecimal notation (in which case "0x" must be between the sign and
the hex digits), e.g. ``-0xFF`` or ``+0xabc123``. the hex digits), e.g. ``-0xFF`` or ``+0xabc123``.
The ``int`` type supports the following operators: arithmetic The "int" type supports the following operators: arithmetic
operators (``+``, ``-``, ``*``, ``/``, ``%``), comparison operators operators (``+``, ``-``, ``*``, ``/``, ``%``), comparison operators
(``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``), assignment operators (``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``), assignment operators
(``=``, ``+=``, ``-=``), pre-increment (``++``), pre-decrement (``=``, ``+=``, ``-=``), pre-increment (``++``), pre-decrement
(``--``), and absolute value (e.g., ``|-3|`` is 3). (``--``), unary plus and minus (``+``, ``-``), and absolute value
(e.g., ``|-3|`` is 3, but the result type is :bro:type:`count`).
When using type inferencing use care so that the When using type inferencing use care so that the
intended type is inferred, e.g. ``local size_difference = 0`` will intended type is inferred, e.g. "local size_difference = 0" will
infer :bro:type:`count`, while ``local size_difference = +0`` infer ":bro:type:`count`", while "local size_difference = +0"
will infer :bro:type:`int`. will infer "int".
.. bro:type:: count .. bro:type:: count
A numeric type representing a 64-bit unsigned integer. A ``count`` A numeric type representing a 64-bit unsigned integer. A "count"
constant is a string of digits, e.g. ``1234`` or ``0``. A ``count`` constant is a string of digits, e.g. ``1234`` or ``0``. A "count"
can also be written in hexadecimal notation (in which case "0x" must can also be written in hexadecimal notation (in which case "0x" must
precede the hex digits), e.g. ``0xff`` or ``0xABC123``. precede the hex digits), e.g. ``0xff`` or ``0xABC123``.
The ``count`` type supports the same operators as the :bro:type:`int` The "count" type supports the same operators as the ":bro:type:`int`"
type. A unary plus or minus applied to a ``count`` results in an ``int``. type, but a unary plus or minus applied to a "count" results in an
"int".
.. bro:type:: counter
An alias to :bro:type:`count`.
.. bro:type:: double .. bro:type:: double
A numeric type representing a double-precision floating-point A numeric type representing a double-precision floating-point
number. Floating-point constants are written as a string of digits number. Floating-point constants are written as a string of digits
with an optional decimal point, optional scale-factor in scientific with an optional decimal point, optional scale-factor in scientific
notation, and optional ``+`` or ``-`` sign. Examples are ``-1234``, notation, and optional "+" or "-" sign. Examples are ``-1234``,
``-1234e0``, ``3.14159``, and ``.003E-23``. ``-1234e0``, ``3.14159``, and ``.003E-23``.
The ``double`` type supports the following operators: arithmetic The "double" type supports the following operators: arithmetic
operators (``+``, ``-``, ``*``, ``/``), comparison operators operators (``+``, ``-``, ``*``, ``/``), comparison operators
(``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``), assignment operators (``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``), assignment operators
(``=``, ``+=``, ``-=``), and absolute value (e.g., ``|-3.14|`` is 3.14). (``=``, ``+=``, ``-=``), unary plus and minus (``+``, ``-``), and
absolute value (e.g., ``|-3.14|`` is 3.14).
When using type inferencing use care so that the When using type inferencing use care so that the
intended type is inferred, e.g. ``local size_difference = 5`` will intended type is inferred, e.g. "local size_difference = 5" will
infer :bro:type:`count`, while ``local size_difference = 5.0`` infer ":bro:type:`count`", while "local size_difference = 5.0"
will infer :bro:type:`double`. will infer "double".
.. bro:type:: time .. bro:type:: time
@ -97,10 +119,10 @@ The Bro scripting language supports the following built-in types.
Time values support the comparison operators (``==``, ``!=``, ``<``, Time values support the comparison operators (``==``, ``!=``, ``<``,
``<=``, ``>``, ``>=``). A ``time`` value can be subtracted from ``<=``, ``>``, ``>=``). A ``time`` value can be subtracted from
another ``time`` value to produce an ``interval`` value. An ``interval`` another ``time`` value to produce an :bro:type:`interval` value. An
value can be added to, or subtracted from, a ``time`` value to produce a ``interval`` value can be added to, or subtracted from, a ``time`` value
``time`` value. The absolute value of a ``time`` value is a ``double`` to produce a ``time`` value. The absolute value of a ``time`` value is
with the same numeric value. a :bro:type:`double` with the same numeric value.
.. bro:type:: interval .. bro:type:: interval
@ -115,52 +137,58 @@ The Bro scripting language supports the following built-in types.
``3.5mins``. An ``interval`` can also be negated, for example ``3.5mins``. An ``interval`` can also be negated, for example
``-12 hr`` represents "twelve hours in the past". ``-12 hr`` represents "twelve hours in the past".
Intervals support addition and subtraction. Intervals also support Intervals support addition and subtraction, the comparison operators
division (in which case the result is a ``double`` value), the (``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``), the assignment
comparison operators (``==``, ``!=``, ``<``, ``<=``, ``>``, ``>=``), operators (``=``, ``+=``, ``-=``), and unary plus and minus (``+``, ``-``).
and the assignment operators (``=``, ``+=``, ``-=``). Also, an
``interval`` can be multiplied or divided by an arithmetic type Intervals also support division (in which case the result is a
(``count``, ``int``, or ``double``) to produce an ``interval`` value. :bro:type:`double` value). An ``interval`` can be multiplied or divided
The absolute value of an ``interval`` is a ``double`` value equal to the by an arithmetic type (``count``, ``int``, or ``double``) to produce
number of seconds in the ``interval`` (e.g., ``|-1 min|`` is 60). an ``interval`` value. The absolute value of an ``interval`` is a
``double`` value equal to the number of seconds in the ``interval``
(e.g., ``|-1 min|`` is 60.0).
.. bro:type:: string .. bro:type:: string
A type used to hold character-string values which represent text. A type used to hold character-string values which represent text, although
String constants are created by enclosing text in double quotes (") strings in a Bro script can actually contain any arbitrary binary data.
and the backslash character (\\) introduces escape sequences (all of
the C-style escape sequences are supported). String constants are created by enclosing text within a pair of double
quotes ("). A string constant cannot span multiple lines in a Bro script.
The backslash character (\\) introduces escape sequences. The
following escape sequences are recognized: ``\n``, ``\t``, ``\v``, ``\b``,
``\r``, ``\f``, ``\a``, ``\ooo`` (where each 'o' is an octal digit),
``\xhh`` (where each 'h' is a hexadecimal digit). For escape sequences
that don't match any of these, Bro will just remove the backslash (so
to represent a literal backslash in a string constant, you just use
two consecutive backslashes).
Strings support concatenation (``+``), and assignment (``=``, ``+=``). Strings support concatenation (``+``), and assignment (``=``, ``+=``).
Strings also support the comparison operators (``==``, ``!=``, ``<``, Strings also support the comparison operators (``==``, ``!=``, ``<``,
``<=``, ``>``, ``>=``). The number of characters in a string can be ``<=``, ``>``, ``>=``). The number of characters in a string can be
found by enclosing the string within pipe characters (e.g., ``|"abc"|`` found by enclosing the string within pipe characters (e.g., ``|"abc"|``
is 3). is 3). Substring searching can be performed using the "in" or "!in"
The subscript operator can extract an individual character or a substring
of a string (string indexing is zero-based, but an index of
-1 refers to the last character in the string, and -2 refers to the
second-to-last character, etc.). When extracting a substring, the
starting and ending index values are separated by a colon. For example::
local orig = "0123456789";
local third_char = orig[2];
local last_char = orig[-1];
local first_three_chars = orig[0:2];
Substring searching can be performed using the "in" or "!in"
operators (e.g., "bar" in "foobar" yields true). operators (e.g., "bar" in "foobar" yields true).
Note that Bro represents strings internally as a count and vector of The subscript operator can extract a substring of a string. To do this,
bytes rather than a NUL-terminated byte string (although string specify the starting index to extract (if the starting index is omitted,
constants are also automatically NUL-terminated). This is because then zero is assumed), followed by a colon and index
network traffic can easily introduce NULs into strings either by one past the last character to extract (if the last index is omitted,
nature of an application, inadvertently, or maliciously. And while then the extracted substring will go to the end of the original string).
NULs are allowed in Bro strings, when present in strings passed as However, if both the colon and last index are omitted, then a string of
arguments to many functions, a run-time error can occur as their length one is extracted. String indexing is zero-based, but an index
presence likely indicates a sort of problem. In that case, the of -1 refers to the last character in the string, and -2 refers to the
string will also only be represented to the user as the literal second-to-last character, etc. Here are a few examples::
"<string-with-NUL>" string.
local orig = "0123456789";
local second_char = orig[1];
local last_char = orig[-1];
local first_two_chars = orig[:2];
local last_two_chars = orig[8:];
local no_first_and_last = orig[1:9];
Note that the subscript operator cannot be used to modify a string (i.e.,
it cannot be on the left side of an assignment operator).
.. bro:type:: pattern .. bro:type:: pattern
@ -174,7 +202,7 @@ The Bro scripting language supports the following built-in types.
and embedded. and embedded.
In exact matching the ``==`` equality relational operator is used In exact matching the ``==`` equality relational operator is used
with one :bro:type:`pattern` operand and one :bro:type:`string` with one "pattern" operand and one ":bro:type:`string`"
operand (order of operands does not matter) to check whether the full operand (order of operands does not matter) to check whether the full
string exactly matches the pattern. In exact matching, the ``^`` string exactly matches the pattern. In exact matching, the ``^``
beginning-of-line and ``$`` end-of-line anchors are redundant since beginning-of-line and ``$`` end-of-line anchors are redundant since
@ -190,8 +218,8 @@ The Bro scripting language supports the following built-in types.
yields false. The ``!=`` operator would yield the negation of ``==``. yields false. The ``!=`` operator would yield the negation of ``==``.
In embedded matching the ``in`` operator is used with one In embedded matching the ``in`` operator is used with one
:bro:type:`pattern` operand (which must be on the left-hand side) and "pattern" operand (which must be on the left-hand side) and
one :bro:type:`string` operand, but tests whether the pattern one ":bro:type:`string`" operand, but tests whether the pattern
appears anywhere within the given string. For example:: appears anywhere within the given string. For example::
/foo|bar/ in "foobar" /foo|bar/ in "foobar"
@ -203,27 +231,12 @@ The Bro scripting language supports the following built-in types.
is false since "oob" does not appear at the start of "foobar". The is false since "oob" does not appear at the start of "foobar". The
``!in`` operator would yield the negation of ``in``. ``!in`` operator would yield the negation of ``in``.
.. bro:type:: enum
A type allowing the specification of a set of related values that
have no further structure. An example declaration:
.. code:: bro
type color: enum { Red, White, Blue, };
The last comma after ``Blue`` is optional.
The only operations allowed on enumerations are equality comparisons
(``==``, ``!=``) and assignment (``=``).
Enumerations do not have associated values or ordering.
.. bro:type:: port .. bro:type:: port
A type representing transport-level port numbers. Besides TCP and A type representing transport-level port numbers (besides TCP and
UDP ports, there is a concept of an ICMP "port" where the source UDP ports, there is a concept of an ICMP "port" where the source
port is the ICMP message type and the destination port the ICMP port is the ICMP message type and the destination port the ICMP
message code. A ``port`` constant is written as an unsigned integer message code). A ``port`` constant is written as an unsigned integer
followed by one of ``/tcp``, ``/udp``, ``/icmp``, or ``/unknown``. followed by one of ``/tcp``, ``/udp``, ``/icmp``, or ``/unknown``.
Ports support the comparison operators (``==``, ``!=``, ``<``, ``<=``, Ports support the comparison operators (``==``, ``!=``, ``<``, ``<=``,
@ -255,14 +268,6 @@ The Bro scripting language supports the following built-in types.
address) are treated internally as IPv4 addresses (for example, address) are treated internally as IPv4 addresses (for example,
``[::ffff:192.168.1.100]`` is equal to ``192.168.1.100``). ``[::ffff:192.168.1.100]`` is equal to ``192.168.1.100``).
Hostname constants can also be used, but since a hostname can
correspond to multiple IP addresses, the type of such a variable is a
:bro:type:`set` of :bro:type:`addr` elements. For example:
.. code:: bro
local a = www.google.com;
Addresses can be compared for equality (``==``, ``!=``), Addresses can be compared for equality (``==``, ``!=``),
and also for ordering (``<``, ``<=``, ``>``, ``>=``). The absolute value and also for ordering (``<``, ``<=``, ``>``, ``>=``). The absolute value
of an address gives the size in bits (32 for IPv4, and 128 for IPv6). of an address gives the size in bits (32 for IPv4, and 128 for IPv6).
@ -285,9 +290,17 @@ The Bro scripting language supports the following built-in types.
if ( a in s ) if ( a in s )
print "true"; print "true";
Note that you can check if a given ``addr`` is IPv4 or IPv6 using You can check if a given ``addr`` is IPv4 or IPv6 using
the :bro:id:`is_v4_addr` and :bro:id:`is_v6_addr` built-in functions. the :bro:id:`is_v4_addr` and :bro:id:`is_v6_addr` built-in functions.
Note that hostname constants can also be used, but since a hostname can
correspond to multiple IP addresses, the type of such a variable is
"set[addr]". For example:
.. code:: bro
local a = www.google.com;
.. bro:type:: subnet .. bro:type:: subnet
A type representing a block of IP addresses in CIDR notation. A A type representing a block of IP addresses in CIDR notation. A
@ -296,13 +309,24 @@ The Bro scripting language supports the following built-in types.
number. For example, ``192.168.0.0/16`` or ``[fe80::]/64``. number. For example, ``192.168.0.0/16`` or ``[fe80::]/64``.
Subnets can be compared for equality (``==``, ``!=``). An Subnets can be compared for equality (``==``, ``!=``). An
:bro:type:`addr` can be checked for inclusion in a subnet using "addr" can be checked for inclusion in a subnet using
the "in" or "!in" operators. the ``in`` or ``!in`` operators.
.. bro:type:: any .. bro:type:: enum
Used to bypass strong typing. For example, a function can take an A type allowing the specification of a set of related values that
argument of type ``any`` when it may be of different types. have no further structure. An example declaration:
.. code:: bro
type color: enum { Red, White, Blue, };
The last comma after ``Blue`` is optional. Both the type name ``color``
and the individual values (``Red``, etc.) have global scope.
Enumerations do not have associated values or ordering.
The only operations allowed on enumerations are equality comparisons
(``==``, ``!=``) and assignment (``=``).
.. bro:type:: table .. bro:type:: table
@ -316,24 +340,25 @@ The Bro scripting language supports the following built-in types.
table [ type^+ ] of type table [ type^+ ] of type
where *type^+* is one or more types, separated by commas. For example: where *type^+* is one or more types, separated by commas.
For example:
.. code:: bro .. code:: bro
global a: table[count] of string; global a: table[count] of string;
declares a table indexed by :bro:type:`count` values and yielding declares a table indexed by "count" values and yielding
:bro:type:`string` values. The yield type can also be more complex: "string" values. The yield type can also be more complex:
.. code:: bro .. code:: bro
global a: table[count] of table[addr, port] of string; global a: table[count] of table[addr, port] of string;
which declares a table indexed by :bro:type:`count` and yielding which declares a table indexed by "count" and yielding
another :bro:type:`table` which is indexed by an :bro:type:`addr` another "table" which is indexed by an "addr"
and :bro:type:`port` to yield a :bro:type:`string`. and "port" to yield a "string".
Initialization of tables occurs by enclosing a set of initializers within One way to initialize a table is by enclosing a set of initializers within
braces, for example: braces, for example:
.. code:: bro .. code:: bro
@ -343,18 +368,17 @@ The Bro scripting language supports the following built-in types.
[5] = "five", [5] = "five",
}; };
A table constructor (equivalent to above example) can also be used A table constructor can also be used to create a table:
to create a table:
.. code:: bro .. code:: bro
global t2: table[count] of string = table( global t2 = table(
[11] = "eleven", [192.168.0.2, 22/tcp] = "ssh",
[5] = "five" [192.168.0.3, 80/tcp] = "http"
); );
Table constructors can also be explicitly named by a type, which is Table constructors can also be explicitly named by a type, which is
useful for when a more complex index type could otherwise be useful when a more complex index type could otherwise be
ambiguous: ambiguous:
.. code:: bro .. code:: bro
@ -381,17 +405,7 @@ The Bro scripting language supports the following built-in types.
if ( 13 in t ) if ( 13 in t )
... ...
if ( [192.168.0.2, 22/tcp] in t2 )
Iterate over tables with a ``for`` loop:
.. code:: bro
local t: table[count] of string;
for ( n in t )
...
local services: table[addr, port] of string;
for ( [a, p] in services )
... ...
Add or overwrite individual table elements by assignment: Add or overwrite individual table elements by assignment:
@ -400,7 +414,7 @@ The Bro scripting language supports the following built-in types.
t[13] = "thirteen"; t[13] = "thirteen";
Remove individual table elements with ``delete``: Remove individual table elements with :bro:keyword:`delete`:
.. code:: bro .. code:: bro
@ -416,6 +430,9 @@ The Bro scripting language supports the following built-in types.
|t| |t|
See the :bro:keyword:`for` statement for info on how to iterate over
the elements in a table.
.. bro:type:: set .. bro:type:: set
A set is like a :bro:type:`table`, but it is a collection of indices A set is like a :bro:type:`table`, but it is a collection of indices
@ -426,25 +443,22 @@ The Bro scripting language supports the following built-in types.
where *type^+* is one or more types separated by commas. where *type^+* is one or more types separated by commas.
Sets are initialized by listing elements enclosed by curly braces: Sets can be initialized by listing elements enclosed by curly braces:
.. code:: bro .. code:: bro
global s: set[port] = { 21/tcp, 23/tcp, 80/tcp, 443/tcp }; global s: set[port] = { 21/tcp, 23/tcp, 80/tcp, 443/tcp };
global s2: set[port, string] = { [21/tcp, "ftp"], [23/tcp, "telnet"] }; global s2: set[port, string] = { [21/tcp, "ftp"], [23/tcp, "telnet"] };
The types are explicitly shown in the example above, but they could
have been left to type inference.
A set constructor (equivalent to above example) can also be used to A set constructor (equivalent to above example) can also be used to
create a set: create a set:
.. code:: bro .. code:: bro
global s3: set[port] = set(21/tcp, 23/tcp, 80/tcp, 443/tcp); global s3 = set(21/tcp, 23/tcp, 80/tcp, 443/tcp);
Set constructors can also be explicitly named by a type, which is Set constructors can also be explicitly named by a type, which is
useful for when a more complex index type could otherwise be useful when a more complex index type could otherwise be
ambiguous: ambiguous:
.. code:: bro .. code:: bro
@ -465,18 +479,10 @@ The Bro scripting language supports the following built-in types.
if ( 21/tcp in s ) if ( 21/tcp in s )
... ...
if ( 21/tcp !in s ) if ( [21/tcp, "ftp"] !in s2 )
... ...
Iterate over a set with a ``for`` loop: Elements are added with :bro:keyword:`add`:
.. code:: bro
local s: set[port];
for ( p in s )
...
Elements are added with ``add``:
.. code:: bro .. code:: bro
@ -485,7 +491,7 @@ The Bro scripting language supports the following built-in types.
Nothing happens if the element with value ``22/tcp`` was already present in Nothing happens if the element with value ``22/tcp`` was already present in
the set. the set.
And removed with ``delete``: And removed with :bro:keyword:`delete`:
.. code:: bro .. code:: bro
@ -501,6 +507,9 @@ The Bro scripting language supports the following built-in types.
|s| |s|
See the :bro:keyword:`for` statement for info on how to iterate over
the elements in a set.
.. bro:type:: vector .. bro:type:: vector
A vector is like a :bro:type:`table`, except it's always indexed by a A vector is like a :bro:type:`table`, except it's always indexed by a
@ -515,7 +524,7 @@ The Bro scripting language supports the following built-in types.
.. code:: bro .. code:: bro
global v: vector of string = vector("one", "two", "three"); local v = vector("one", "two", "three");
Vector constructors can also be explicitly named by a type, which Vector constructors can also be explicitly named by a type, which
is useful for when a more complex yield type could otherwise be is useful for when a more complex yield type could otherwise be
@ -539,14 +548,6 @@ The Bro scripting language supports the following built-in types.
print v[2]; print v[2];
Iterate over a vector with a ``for`` loop:
.. code:: bro
local v: vector of string;
for ( n in v )
...
An element can be added to a vector by assigning the value (a value An element can be added to a vector by assigning the value (a value
that already exists at that index will be overwritten): that already exists at that index will be overwritten):
@ -577,11 +578,17 @@ The Bro scripting language supports the following built-in types.
The resulting vector of bool is the logical "and" (or logical "or") of The resulting vector of bool is the logical "and" (or logical "or") of
each element of the operand vectors. each element of the operand vectors.
See the :bro:keyword:`for` statement for info on how to iterate over
the elements in a vector.
.. bro:type:: record .. bro:type:: record
A ``record`` is a collection of values. Each value has a field name A "record" is a collection of values. Each value has a field name
and a type. Values do not need to have the same type and the types and a type. Values do not need to have the same type and the types
have no restrictions. An example record type definition: have no restrictions. Field names must follow the same syntax as
regular variable names (except that field names are allowed to be the
same as local or global variables). An example record type
definition:
.. code:: bro .. code:: bro
@ -590,86 +597,45 @@ The Bro scripting language supports the following built-in types.
s: string &optional; s: string &optional;
}; };
Access to a record field uses the dollar sign (``$``) operator: Records can be initialized or assigned as a whole in three different ways.
.. code:: bro
global r: MyRecordType;
r$c = 13;
Record assignment can be done field by field or as a whole like:
.. code:: bro
r = [$c = 13, $s = "thirteen"];
When assigning a whole record value, all fields that are not When assigning a whole record value, all fields that are not
:bro:attr:`&optional` or have a :bro:attr:`&default` attribute must :bro:attr:`&optional` or have a :bro:attr:`&default` attribute must
be specified. be specified. First, there's a constructor syntax:
To test for existence of a field that is :bro:attr:`&optional`, use the .. code:: bro
``?$`` operator:
local r: MyRecordType = record($c = 7);
And the constructor can be explicitly named by type, too, which
is arguably more readable:
.. code:: bro
local r = MyRecordType($c = 42);
And the third way is like this:
.. code:: bro
local r: MyRecordType = [$c = 13, $s = "thirteen"];
Access to a record field uses the dollar sign (``$``) operator, and
record fields can be assigned with this:
.. code:: bro
local r: MyRecordType;
r$c = 13;
To test if a field that is :bro:attr:`&optional` has been assigned a
value, use the ``?$`` operator (it returns a :bro:type:`bool` value of
``T`` if the field has been assigned a value, or ``F`` if not):
.. code:: bro .. code:: bro
if ( r ?$ s ) if ( r ?$ s )
... ...
Records can also be created using a constructor syntax:
.. code:: bro
global r2: MyRecordType = record($c = 7);
And the constructor can be explicitly named by type, too, which
is arguably more readable code:
.. code:: bro
global r3 = MyRecordType($c = 42);
.. bro:type:: opaque
A data type whose actual representation/implementation is
intentionally hidden, but whose values may be passed to certain
functions that can actually access the internal/hidden resources.
Opaque types are differentiated from each other by qualifying them
like ``opaque of md5`` or ``opaque of sha1``. Any valid identifier
can be used as the type qualifier.
An example use of this type is the set of built-in functions which
perform hashing:
.. code:: bro
local handle: opaque of md5 = md5_hash_init();
md5_hash_update(handle, "test");
md5_hash_update(handle, "testing");
print md5_hash_finish(handle);
Here the opaque type is used to provide a handle to a particular
resource which is calculating an MD5 checksum incrementally over
time, but the details of that resource aren't relevant, it's only
necessary to have a handle as a way of identifying it and
distinguishing it from other such resources.
.. bro:type:: file
Bro supports writing to files, but not reading from them. Files
can be opened using either the :bro:id:`open` or :bro:id:`open_for_append`
built-in functions, and closed using the :bro:id:`close` built-in
function. For example, declare, open, and write to a file
and finally close it like:
.. code:: bro
global f: file = open("myfile");
print f, "hello, world";
close(f);
Writing to files like this for logging usually isn't recommended, for better
logging support see :doc:`/frameworks/logging`.
.. bro:type:: function .. bro:type:: function
Function types in Bro are declared using:: Function types in Bro are declared using::
@ -700,6 +666,16 @@ The Bro scripting language supports the following built-in types.
type, but when it is, the return type and argument list (including the type, but when it is, the return type and argument list (including the
name of each argument) must match exactly. name of each argument) must match exactly.
Here is an example function that takes no parameters and does not
return a value:
.. code:: bro
function my_func()
{
print "my_func";
}
Function types don't need to have a name and can be assigned anonymously: Function types don't need to have a name and can be assigned anonymously:
.. code:: bro .. code:: bro
@ -742,9 +718,20 @@ The Bro scripting language supports the following built-in types.
Event handlers are nearly identical in both syntax and semantics to Event handlers are nearly identical in both syntax and semantics to
a :bro:type:`function`, with the two differences being that event a :bro:type:`function`, with the two differences being that event
handlers have no return type since they never return a value, and handlers have no return type since they never return a value, and
you cannot call an event handler. Instead of directly calling an you cannot call an event handler.
event handler from a script, event handler bodies are executed when
they are invoked by one of three different methods: Example:
.. code:: bro
event my_event(r: bool, s: string)
{
print "my_event", r, s;
}
Instead of directly calling an event handler from a script, event
handler bodies are executed when they are invoked by one of three
different methods:
- From the event engine - From the event engine
@ -765,7 +752,7 @@ The Bro scripting language supports the following built-in types.
This assumes that ``password_exposed`` was previously declared This assumes that ``password_exposed`` was previously declared
as an event handler type with compatible arguments. as an event handler type with compatible arguments.
- Via the ``schedule`` expression in a script - Via the :bro:keyword:`schedule` expression in a script
This delays the invocation of event handlers until some time in This delays the invocation of event handlers until some time in
the future. For example: the future. For example:
@ -789,8 +776,8 @@ The Bro scripting language supports the following built-in types.
immediate and they do not get scheduled through an event queue. immediate and they do not get scheduled through an event queue.
Also, a unique feature of a hook is that a given hook handler body Also, a unique feature of a hook is that a given hook handler body
can short-circuit the execution of remaining hook handlers simply by can short-circuit the execution of remaining hook handlers simply by
exiting from the body as a result of a ``break`` statement (as exiting from the body as a result of a :bro:keyword:`break` statement (as
opposed to a ``return`` or just reaching the end of the body). opposed to a :bro:keyword:`return` or just reaching the end of the body).
A hook type is declared like:: A hook type is declared like::
@ -859,142 +846,60 @@ The Bro scripting language supports the following built-in types.
executed due to one handler body exiting as a result of a ``break`` executed due to one handler body exiting as a result of a ``break``
statement. statement.
Attributes .. bro:type:: file
----------
Attributes occur at the end of type/event declarations and change their Bro supports writing to files, but not reading from them (to read from
behavior. The syntax is ``&key`` or ``&key=val``, e.g., ``type T: files see the :doc:`/frameworks/input`). Files
set[count] &read_expire=5min`` or ``event foo() &priority=-3``. The Bro can be opened using either the :bro:id:`open` or :bro:id:`open_for_append`
scripting language supports the following built-in attributes. built-in functions, and closed using the :bro:id:`close` built-in
function. For example, declare, open, and write to a file and finally
close it like:
.. bro:attr:: &optional .. code:: bro
Allows a record field to be missing. For example the type ``record { local f = open("myfile");
a: addr; b: port &optional; }`` could be instantiated both as print f, "hello, world";
singleton ``[$a=127.0.0.1]`` or pair ``[$a=127.0.0.1, $b=80/tcp]``. close(f);
.. bro:attr:: &default Writing to files like this for logging usually isn't recommended, for better
logging support see :doc:`/frameworks/logging`.
Uses a default value for a record field, a function/hook/event .. bro:type:: opaque
parameter, or container elements. For example, ``table[int] of
string &default="foo"`` would create a table that returns the
:bro:type:`string` ``"foo"`` for any non-existing index.
.. bro:attr:: &redef A data type whose actual representation/implementation is
intentionally hidden, but whose values may be passed to certain
built-in functions that can actually access the internal/hidden resources.
Opaque types are differentiated from each other by qualifying them
like "opaque of md5" or "opaque of sha1".
Allows for redefinition of initial object values. This is typically An example use of this type is the set of built-in functions which
used with constants, for example, ``const clever = T &redef;`` would perform hashing:
allow the constant to be redefined at some later point during script
execution.
.. bro:attr:: &rotate_interval .. code:: bro
Rotates a file after a specified interval. local handle = md5_hash_init();
md5_hash_update(handle, "test");
md5_hash_update(handle, "testing");
print md5_hash_finish(handle);
.. bro:attr:: &rotate_size Here the opaque type is used to provide a handle to a particular
resource which is calculating an MD5 hash incrementally over
time, but the details of that resource aren't relevant, it's only
necessary to have a handle as a way of identifying it and
distinguishing it from other such resources.
Rotates a file after it has reached a given size in bytes. .. bro:type:: any
.. bro:attr:: &add_func Used to bypass strong typing. For example, a function can take an
argument of type ``any`` when it may be of different types.
The only operation allowed on a variable of type ``any`` is assignment.
Can be applied to an identifier with &redef to specify a function to Note that users aren't expected to use this type. It's provided mainly
be called any time a "redef <id> += ..." declaration is parsed. The for use by some built-in functions and scripts included with Bro.
function takes two arguments of the same type as the identifier, the first
being the old value of the variable and the second being the new
value given after the "+=" operator in the "redef" declaration. The
return value of the function will be the actual new value of the
variable after the "redef" declaration is parsed.
.. bro:attr:: &delete_func .. bro:type:: void
Same as &add_func, except for "redef" declarations that use the "-=" An internal Bro type (i.e., "void" is not a reserved keyword in the Bro
operator. scripting language) representing the absence of a return type for a
function.
.. bro:attr:: &expire_func
Called right before a container element expires. The function's
first parameter is of the same type of the container and the second
parameter the same type of the container's index. The return
value is an :bro:type:`interval` indicating the amount of additional
time to wait before expiring the container element at the given
index (which will trigger another execution of this function).
.. bro:attr:: &read_expire
Specifies a read expiration timeout for container elements. That is,
the element expires after the given amount of time since the last
time it has been read. Note that a write also counts as a read.
.. bro:attr:: &write_expire
Specifies a write expiration timeout for container elements. That
is, the element expires after the given amount of time since the
last time it has been written.
.. bro:attr:: &create_expire
Specifies a creation expiration timeout for container elements. That
is, the element expires after the given amount of time since it has
been inserted into the container, regardless of any reads or writes.
.. bro:attr:: &persistent
Makes a variable persistent, i.e., its value is written to disk (per
default at shutdown time).
.. bro:attr:: &synchronized
Synchronizes variable accesses across nodes. The value of a
``&synchronized`` variable is automatically propagated to all peers
when it changes.
.. bro:attr:: &encrypt
Encrypts files right before writing them to disk.
.. TODO: needs to be documented in more detail.
.. bro:attr:: &raw_output
Opens a file in raw mode, i.e., non-ASCII characters are not
escaped.
.. bro:attr:: &mergeable
Prefers set union to assignment for synchronized state. This
attribute is used in conjunction with :bro:attr:`&synchronized`
container types: when the same container is updated at two peers
with different value, the propagation of the state causes a race
condition, where the last update succeeds. This can cause
inconsistencies and can be avoided by unifying the two sets, rather
than merely overwriting the old value.
.. bro:attr:: &priority
Specifies the execution priority (as a signed integer) of a hook or
event handler. Higher values are executed before lower ones. The
default value is 0.
.. bro:attr:: &group
Groups event handlers such that those in the same group can be
jointly activated or deactivated.
.. bro:attr:: &log
Writes a record field to the associated log stream.
.. bro:attr:: &error_handler
Internally set on the events that are associated with the reporter
framework: :bro:id:`reporter_info`, :bro:id:`reporter_warning`, and
:bro:id:`reporter_error`. It prevents any handlers of those events
from being able to generate reporter messages that go through any of
those events (i.e., it prevents an infinite event recursion). Instead,
such nested reporter messages are output to stderr.
.. bro:attr:: &type_column
Used by the input framework. It can be used on columns of type
:bro:type:`port` and specifies the name of an additional column in
the input file which specifies the protocol of the port (tcp/udp/icmp).

2
doc/scripting/data_struct_vector_declaration.bro
View file

@ -10,6 +10,6 @@ event bro_init()
print fmt("contents of v1: %s", v1); print fmt("contents of v1: %s", v1);
print fmt("length of v1: %d", |v1|); print fmt("length of v1: %d", |v1|);
print fmt("contents of v1: %s", v2); print fmt("contents of v2: %s", v2);
print fmt("length of v2: %d", |v2|); print fmt("length of v2: %d", |v2|);
} }

2
doc/scripting/data_type_pattern_01.bro
View file

@ -1,6 +1,6 @@
event bro_init() event bro_init()
{ {
local test_string = "The quick brown fox jumped over the lazy dog."; local test_string = "The quick brown fox jumps over the lazy dog.";
local test_pattern = /quick|lazy/; local test_pattern = /quick|lazy/;
if ( test_pattern in test_string ) if ( test_pattern in test_string )

10
doc/scripting/index.rst
View file

@ -260,7 +260,7 @@ originating host is referenced by ``c$id$orig_h`` which if given a
narrative relates to ``orig_h`` which is a member of ``id`` which is narrative relates to ``orig_h`` which is a member of ``id`` which is
a member of the data structure referred to as ``c`` that was passed a member of the data structure referred to as ``c`` that was passed
into the event handler. Given that the responder port into the event handler. Given that the responder port
``c$id$resp_p`` is ``53/tcp``, it's likely that Bro's base HTTP scripts ``c$id$resp_p`` is ``80/tcp``, it's likely that Bro's base HTTP scripts
can further populate the connection record. Let's load the can further populate the connection record. Let's load the
``base/protocols/http`` scripts and check the output of our script. ``base/protocols/http`` scripts and check the output of our script.
@ -730,7 +730,7 @@ Bro supports ``usec``, ``msec``, ``sec``, ``min``, ``hr``, or ``day`` which repr
microseconds, milliseconds, seconds, minutes, hours, and days microseconds, milliseconds, seconds, minutes, hours, and days
respectively. In fact, the interval data type allows for a surprising respectively. In fact, the interval data type allows for a surprising
amount of variation in its definitions. There can be a space between amount of variation in its definitions. There can be a space between
the numeric constant or they can crammed together like a temporal the numeric constant or they can be crammed together like a temporal
portmanteau. The time unit can be either singular or plural. All of portmanteau. The time unit can be either singular or plural. All of
this adds up to to the fact that both ``42hrs`` and ``42 hr`` are this adds up to to the fact that both ``42hrs`` and ``42 hr`` are
perfectly valid and logically equivalent in Bro. The point, however, perfectly valid and logically equivalent in Bro. The point, however,
@ -819,7 +819,7 @@ with the ``typedef`` and ``struct`` keywords, Bro allows you to cobble
together new data types to suit the needs of your situation. together new data types to suit the needs of your situation.
When combined with the ``type`` keyword, ``record`` can generate a When combined with the ``type`` keyword, ``record`` can generate a
composite type. We have, in fact, already encountered a a complex composite type. We have, in fact, already encountered a complex
example of the ``record`` data type in the earlier sections, the example of the ``record`` data type in the earlier sections, the
:bro:type:`connection` record passed to many events. Another one, :bro:type:`connection` record passed to many events. Another one,
:bro:type:`Conn::Info`, which corresponds to the fields logged into :bro:type:`Conn::Info`, which corresponds to the fields logged into
@ -1014,8 +1014,8 @@ remaining logs to factor.log.
:lines: 38-62 :lines: 38-62
:linenos: :linenos:
To dynamically alter the file in which a stream writes its logs a To dynamically alter the file in which a stream writes its logs, a
filter can specify function returns a string to be used as the filter can specify a function that returns a string to be used as the
filename for the current call to ``Log::write``. The definition for filename for the current call to ``Log::write``. The definition for
this function has to take as its parameters a ``Log::ID`` called id, a this function has to take as its parameters a ``Log::ID`` called id, a
string called ``path`` and the appropriate record type for the logs called string called ``path`` and the appropriate record type for the logs called

5
scripts/base/frameworks/files/magic/general.sig
View file

@ -9,3 +9,8 @@ signature file-tar {
file-magic /([[:print:]\x00]){100}(([[:digit:]\x00\x20]){8}){3}/ file-magic /([[:print:]\x00]){100}(([[:digit:]\x00\x20]){8}){3}/
file-mime "application/x-tar", 150 file-mime "application/x-tar", 150
} }
signature file-swf {
file-magic /(F|C|Z)WS/
file-mime "application/x-shockwave-flash", 60
}

13
scripts/base/frameworks/files/magic/libmagic.sig
View file

@ -2769,19 +2769,6 @@ signature file-magic-auto408 {
file-magic /(.{512})(\xec\xa5\xc1)/ file-magic /(.{512})(\xec\xa5\xc1)/
} }
# >0 string,=FWS (len=3), ["Macromedia Flash data,"], swap_endian=0
# >>3 byte&,x, ["version %d"], swap_endian=0
signature file-magic-auto409 {
file-mime "application/x-shockwave-flash", 1
file-magic /(FWS)(.{1})/
}
# >0 string,=CWS (len=3), ["Macromedia Flash data (compressed),"], swap_endian=0
signature file-magic-auto410 {
file-mime "application/x-shockwave-flash", 60
file-magic /(CWS)/
}
# >0 regex/20,=^\.[A-Za-z0-9][A-Za-z0-9][ \t] (len=29), ["troff or preprocessor input text"], swap_endian=0 # >0 regex/20,=^\.[A-Za-z0-9][A-Za-z0-9][ \t] (len=29), ["troff or preprocessor input text"], swap_endian=0
signature file-magic-auto411 { signature file-magic-auto411 {
file-mime "text/troff", 59 file-mime "text/troff", 59

95
scripts/base/frameworks/files/main.bro
View file

@ -153,6 +153,15 @@ export {
tag: Files::Tag, tag: Files::Tag,
args: AnalyzerArgs &default=AnalyzerArgs()): bool; args: AnalyzerArgs &default=AnalyzerArgs()): bool;
## Adds all analyzers associated with a give MIME type to the analysis of
## a file. Note that analyzers added via MIME types cannot take further
## arguments.
##
## f: the file.
##
## mtype: the MIME type; it will be compared case-insensitive.
global add_analyzers_for_mime_type: function(f: fa_file, mtype: string);
## Removes an analyzer from the analysis of a given file. ## Removes an analyzer from the analysis of a given file.
## ##
## f: the file. ## f: the file.
@ -225,6 +234,42 @@ export {
## callback: Function to execute when the given file analyzer is being added. ## callback: Function to execute when the given file analyzer is being added.
global register_analyzer_add_callback: function(tag: Files::Tag, callback: function(f: fa_file, args: AnalyzerArgs)); global register_analyzer_add_callback: function(tag: Files::Tag, callback: function(f: fa_file, args: AnalyzerArgs));
## Registers a set of MIME types for an analyzer. If a future connection on one of
## these types is seen, the analyzer will be automatically assigned to parsing it.
## The function *adds* to all MIME types already registered, it doesn't replace
## them.
##
## tag: The tag of the analyzer.
##
## mts: The set of MIME types, each in the form "foo/bar" (case-insensitive).
##
## Returns: True if the MIME types were successfully registered.
global register_for_mime_types: function(tag: Analyzer::Tag, mts: set[string]) : bool;
## Registers a MIME type for an analyzer. If a future file with this type is seen,
## the analyzer will be automatically assigned to parsing it. The function *adds*
## to all MIME types already registered, it doesn't replace them.
##
## tag: The tag of the analyzer.
##
## mt: The MIME type in the form "foo/bar" (case-insensitive).
##
## Returns: True if the MIME type was successfully registered.
global register_for_mime_type: function(tag: Analyzer::Tag, mt: string) : bool;
## Returns a set of all MIME types currently registered for a specific analyzer.
##
## tag: The tag of the analyzer.
##
## Returns: The set of MIME types.
global registered_mime_types: function(tag: Analyzer::Tag) : set[string];
## Returns a table of all MIME-type-to-analyzer mappings currently registered.
##
## Returns: A table mapping each analyzer to the set of MIME types
## registered for it.
global all_registered_mime_types: function() : table[Analyzer::Tag] of set[string];
## Event that can be handled to access the Info record as it is sent on ## Event that can be handled to access the Info record as it is sent on
## to the logging framework. ## to the logging framework.
global log_files: event(rec: Info); global log_files: event(rec: Info);
@ -237,6 +282,9 @@ redef record fa_file += {
# Store the callbacks for protocol analyzers that have files. # Store the callbacks for protocol analyzers that have files.
global registered_protocols: table[Analyzer::Tag] of ProtoRegistration = table(); global registered_protocols: table[Analyzer::Tag] of ProtoRegistration = table();
# Store the MIME type to analyzer mappings.
global mime_types: table[Analyzer::Tag] of set[string];
global analyzer_add_callbacks: table[Files::Tag] of function(f: fa_file, args: AnalyzerArgs) = table(); global analyzer_add_callbacks: table[Files::Tag] of function(f: fa_file, args: AnalyzerArgs) = table();
event bro_init() &priority=5 event bro_init() &priority=5
@ -289,6 +337,15 @@ function add_analyzer(f: fa_file, tag: Files::Tag, args: AnalyzerArgs): bool
return T; return T;
} }
function add_analyzers_for_mime_type(f: fa_file, mtype: string)
{
local dummy_args: AnalyzerArgs;
local analyzers = __add_analyzers_for_mime_type(f$id, mtype, dummy_args);
for ( tag in analyzers )
add f$info$analyzers[Files::analyzer_name(tag)];
}
function register_analyzer_add_callback(tag: Files::Tag, callback: function(f: fa_file, args: AnalyzerArgs)) function register_analyzer_add_callback(tag: Files::Tag, callback: function(f: fa_file, args: AnalyzerArgs))
{ {
analyzer_add_callbacks[tag] = callback; analyzer_add_callbacks[tag] = callback;
@ -312,6 +369,9 @@ function analyzer_name(tag: Files::Tag): string
event file_new(f: fa_file) &priority=10 event file_new(f: fa_file) &priority=10
{ {
set_info(f); set_info(f);
if ( f?$mime_type )
add_analyzers_for_mime_type(f, f$mime_type);
} }
event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) &priority=10 event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) &priority=10
@ -349,6 +409,41 @@ function register_protocol(tag: Analyzer::Tag, reg: ProtoRegistration): bool
return result; return result;
} }
function register_for_mime_types(tag: Analyzer::Tag, mime_types: set[string]) : bool
{
local rc = T;
for ( mt in mime_types )
{
if ( ! register_for_mime_type(tag, mt) )
rc = F;
}
return rc;
}
function register_for_mime_type(tag: Analyzer::Tag, mt: string) : bool
{
if ( ! __register_for_mime_type(tag, mt) )
return F;
if ( tag !in mime_types )
mime_types[tag] = set();
add mime_types[tag][mt];
return T;
}
function registered_mime_types(tag: Analyzer::Tag) : set[string]
{
return tag in mime_types ? mime_types[tag] : set();
}
function all_registered_mime_types(): table[Analyzer::Tag] of set[string]
{
return mime_types;
}
function describe(f: fa_file): string function describe(f: fa_file): string
{ {
local tag = Analyzer::get_tag(f$source); local tag = Analyzer::get_tag(f$source);

11
scripts/base/frameworks/input/main.bro
View file

@ -4,6 +4,17 @@
module Input; module Input;
export { export {
type Event: enum {
EVENT_NEW = 0,
EVENT_CHANGED = 1,
EVENT_REMOVED = 2,
};
type Mode: enum {
MANUAL = 0,
REREAD = 1,
STREAM = 2
};
## The default input reader used. Defaults to `READER_ASCII`. ## The default input reader used. Defaults to `READER_ASCII`.
const default_reader = READER_ASCII &redef; const default_reader = READER_ASCII &redef;

8
scripts/base/frameworks/intel/main.bro
View file

@ -81,6 +81,9 @@ export {
## Where the data was discovered. ## Where the data was discovered.
where: Where &log; where: Where &log;
## The name of the node where the match was discovered.
node: string &optional &log;
## If the data was discovered within a connection, the ## If the data was discovered within a connection, the
## connection record should go here to give context to the data. ## connection record should go here to give context to the data.
conn: connection &optional; conn: connection &optional;
@ -240,6 +243,11 @@ function Intel::seen(s: Seen)
s$indicator_type = Intel::ADDR; s$indicator_type = Intel::ADDR;
} }
if ( ! s?$node )
{
s$node = peer_description;
}
if ( have_full_data ) if ( have_full_data )
{ {
local items = get_items(s); local items = get_items(s);

2
scripts/base/frameworks/logging/__load__.bro
View file

@ -1,7 +1,5 @@
@load ./main @load ./main
@load ./postprocessors @load ./postprocessors
@load ./writers/ascii @load ./writers/ascii
@load ./writers/dataseries
@load ./writers/sqlite @load ./writers/sqlite
@load ./writers/elasticsearch
@load ./writers/none @load ./writers/none

10
scripts/base/frameworks/logging/main.bro
View file

@ -5,9 +5,15 @@
module Log; module Log;
# Log::ID and Log::Writer are defined in types.bif due to circular dependencies.
export { export {
## Type that defines an ID unique to each log stream. Scripts creating new log
## streams need to redef this enum to add their own specific log ID. The log ID
## implicitly determines the default name of the generated log file.
type Log::ID: enum {
## Dummy place-holder.
UNKNOWN
};
## If true, local logging is by default enabled for all filters. ## If true, local logging is by default enabled for all filters.
const enable_local_logging = T &redef; const enable_local_logging = T &redef;

60
scripts/base/frameworks/logging/writers/dataseries.bro
View file

@ -1,60 +0,0 @@
##! Interface for the DataSeries log writer.
module LogDataSeries;
export {
## Compression to use with the DS output file. Options are:
##
## 'none' -- No compression.
## 'lzf' -- LZF compression (very quick, but leads to larger output files).
## 'lzo' -- LZO compression (very fast decompression times).
## 'zlib' -- GZIP compression (slower than LZF, but also produces smaller output).
## 'bz2' -- BZIP2 compression (slower than GZIP, but also produces smaller output).
const compression = "zlib" &redef;
## The extent buffer size.
## Larger values here lead to better compression and more efficient writes,
## but also increase the lag between the time events are received and
## the time they are actually written to disk.
const extent_size = 65536 &redef;
## Should we dump the XML schema we use for this DS file to disk?
## If yes, the XML schema shares the name of the logfile, but has
## an XML ending.
const dump_schema = F &redef;
## How many threads should DataSeries spawn to perform compression?
## Note that this dictates the number of threads per log stream. If
## you're using a lot of streams, you may want to keep this number
## relatively small.
##
## Default value is 1, which will spawn one thread / stream.
##
## Maximum is 128, minimum is 1.
const num_threads = 1 &redef;
## Should time be stored as an integer or a double?
## Storing time as a double leads to possible precision issues and
## can (significantly) increase the size of the resulting DS log.
## That said, timestamps stored in double form are consistent
## with the rest of Bro, including the standard ASCII log. Hence, we
## use them by default.
const use_integer_for_time = F &redef;
}
# Default function to postprocess a rotated DataSeries log file. It moves the
# rotated file to a new name that includes a timestamp with the opening time,
# and then runs the writer's default postprocessor command on it.
function default_rotation_postprocessor_func(info: Log::RotationInfo) : bool
{
# Move file to name including both opening and closing time.
local dst = fmt("%s.%s.ds", info$path,
strftime(Log::default_rotation_date_format, info$open));
system(fmt("/bin/mv %s %s", info$fname, dst));
# Run default postprocessor.
return Log::run_rotation_postprocessor_cmd(info, dst);
}
redef Log::default_rotation_postprocessors += { [Log::WRITER_DATASERIES] = default_rotation_postprocessor_func };

48
scripts/base/frameworks/logging/writers/elasticsearch.bro
View file

@ -1,48 +0,0 @@
##! Log writer for sending logs to an ElasticSearch server.
##!
##! Note: This module is in testing and is not yet considered stable!
##!
##! There is one known memory issue. If your elasticsearch server is
##! running slowly and taking too long to return from bulk insert
##! requests, the message queue to the writer thread will continue
##! growing larger and larger giving the appearance of a memory leak.
module LogElasticSearch;
export {
## Name of the ES cluster.
const cluster_name = "elasticsearch" &redef;
## ES server.
const server_host = "127.0.0.1" &redef;
## ES port.
const server_port = 9200 &redef;
## Name of the ES index.
const index_prefix = "bro" &redef;
## The ES type prefix comes before the name of the related log.
## e.g. prefix = "bro\_" would create types of bro_dns, bro_software, etc.
const type_prefix = "" &redef;
## The time before an ElasticSearch transfer will timeout. Note that
## the fractional part of the timeout will be ignored. In particular,
## time specifications less than a second result in a timeout value of
## 0, which means "no timeout."
const transfer_timeout = 2secs;
## The batch size is the number of messages that will be queued up before
## they are sent to be bulk indexed.
const max_batch_size = 1000 &redef;
## The maximum amount of wall-clock time that is allowed to pass without
## finishing a bulk log send. This represents the maximum delay you
## would like to have with your logs before they are sent to ElasticSearch.
const max_batch_interval = 1min &redef;
## The maximum byte size for a buffered JSON string to send to the bulk
## insert API.
const max_byte_size = 1024 * 1024 &redef;
}

18
scripts/base/init-bare.bro
View file

@ -75,6 +75,13 @@ type addr_vec: vector of addr;
## directly and then remove this alias. ## directly and then remove this alias.
type table_string_of_string: table[string] of string; type table_string_of_string: table[string] of string;
## A set of file analyzer tags.
##
## .. todo:: We need this type definition only for declaring builtin functions
## via ``bifcl``. We should extend ``bifcl`` to understand composite types
## directly and then remove this alias.
type files_tag_set: set[Files::Tag];
## A structure indicating a MIME type and strength of a match against ## A structure indicating a MIME type and strength of a match against
## file magic signatures. ## file magic signatures.
## ##
@ -2478,8 +2485,7 @@ type http_message_stat: record {
header_length: count; header_length: count;
}; };
## Maximum number of HTTP entity data delivered to events. The amount of data ## Maximum number of HTTP entity data delivered to events.
## can be limited for better performance, zero disables truncation.
## ##
## .. bro:see:: http_entity_data skip_http_entity_data skip_http_data ## .. bro:see:: http_entity_data skip_http_entity_data skip_http_data
global http_entity_data_delivery_size = 1500 &redef; global http_entity_data_delivery_size = 1500 &redef;
@ -2731,6 +2737,7 @@ type ModbusRegisters: vector of count;
type ModbusHeaders: record { type ModbusHeaders: record {
tid: count; tid: count;
pid: count; pid: count;
len: count;
uid: count; uid: count;
function_code: count; function_code: count;
}; };
@ -3508,9 +3515,6 @@ const global_hash_seed: string = "" &redef;
## The maximum is currently 128 bits. ## The maximum is currently 128 bits.
const bits_per_uid: count = 96 &redef; const bits_per_uid: count = 96 &redef;
# Load BiFs defined by plugins.
@load base/bif/plugins
# Load these frameworks here because they use fairly deep integration with # Load these frameworks here because they use fairly deep integration with
# BiFs and script-land defined types. # BiFs and script-land defined types.
@load base/frameworks/logging @load base/frameworks/logging
@ -3519,3 +3523,7 @@ const bits_per_uid: count = 96 &redef;
@load base/frameworks/files @load base/frameworks/files
@load base/bif @load base/bif
# Load BiFs defined by plugins.
@load base/bif/plugins

7
scripts/base/protocols/dhcp/main.bro
View file

@ -47,13 +47,13 @@ redef record connection += {
const ports = { 67/udp, 68/udp }; const ports = { 67/udp, 68/udp };
redef likely_server_ports += { 67/udp }; redef likely_server_ports += { 67/udp };
event bro_init() event bro_init() &priority=5
{ {
Log::create_stream(DHCP::LOG, [$columns=Info, $ev=log_dhcp]); Log::create_stream(DHCP::LOG, [$columns=Info, $ev=log_dhcp]);
Analyzer::register_for_ports(Analyzer::ANALYZER_DHCP, ports); Analyzer::register_for_ports(Analyzer::ANALYZER_DHCP, ports);
} }
event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, router: dhcp_router_list, lease: interval, serv_addr: addr, host_name: string) event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, router: dhcp_router_list, lease: interval, serv_addr: addr, host_name: string) &priority=5
{ {
local info: Info; local info: Info;
info$ts = network_time(); info$ts = network_time();
@ -71,6 +71,9 @@ event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, router: dhcp_router_lis
info$assigned_ip = c$id$orig_h; info$assigned_ip = c$id$orig_h;
c$dhcp = info; c$dhcp = info;
}
event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, router: dhcp_router_list, lease: interval, serv_addr: addr, host_name: string) &priority=-5
{
Log::write(DHCP::LOG, c$dhcp); Log::write(DHCP::LOG, c$dhcp);
} }

5
scripts/base/protocols/ssl/consts.bro
View file

@ -30,6 +30,7 @@ export {
const HELLO_REQUEST = 0; const HELLO_REQUEST = 0;
const CLIENT_HELLO = 1; const CLIENT_HELLO = 1;
const SERVER_HELLO = 2; const SERVER_HELLO = 2;
const HELLO_VERIFY_REQUEST = 3; # RFC 6347
const SESSION_TICKET = 4; # RFC 5077 const SESSION_TICKET = 4; # RFC 5077
const CERTIFICATE = 11; const CERTIFICATE = 11;
const SERVER_KEY_EXCHANGE = 12; const SERVER_KEY_EXCHANGE = 12;
@ -40,6 +41,7 @@ export {
const FINISHED = 20; const FINISHED = 20;
const CERTIFICATE_URL = 21; # RFC 3546 const CERTIFICATE_URL = 21; # RFC 3546
const CERTIFICATE_STATUS = 22; # RFC 3546 const CERTIFICATE_STATUS = 22; # RFC 3546
const SUPPLEMENTAL_DATA = 23; # RFC 4680
## Mapping between numeric codes and human readable strings for alert ## Mapping between numeric codes and human readable strings for alert
## levels. ## levels.
@ -112,7 +114,8 @@ export {
[19] = "client_certificate_type", [19] = "client_certificate_type",
[20] = "server_certificate_type", [20] = "server_certificate_type",
[21] = "padding", # temporary till 2015-03-12 [21] = "padding", # temporary till 2015-03-12
[22] = "encrypt_then_mac", # temporary till 2015-06-05 [22] = "encrypt_then_mac",
[23] = "extended_master_secret", # temporary till 2015-09-26
[35] = "SessionTicket TLS", [35] = "SessionTicket TLS",
[40] = "extended_random", [40] = "extended_random",
[13172] = "next_protocol_negotiation", [13172] = "next_protocol_negotiation",

65
scripts/base/protocols/ssl/main.bro
View file

@ -25,9 +25,25 @@ export {
## indicates the server name that the client was requesting. ## indicates the server name that the client was requesting.
server_name: string &log &optional; server_name: string &log &optional;
## Session ID offered by the client for session resumption. ## Session ID offered by the client for session resumption.
session_id: string &log &optional; ## Not used for logging.
session_id: string &optional;
## Flag to indicate if the session was resumed reusing
## the key material exchanged in an earlier connection.
resumed: bool &log &default=F;
## Flag to indicate if we saw a non-empty session ticket being
## sent by the client using an empty session ID. This value
## is used to determine if a session is being resumed. It's
## not logged.
client_ticket_empty_session_seen: bool &default=F;
## Flag to indicate if we saw a client key exchange message sent
## by the client. This value is used to determine if a session
## is being resumed. It's not logged.
client_key_exchange_seen: bool &default=F;
## Last alert that was seen during the connection. ## Last alert that was seen during the connection.
last_alert: string &log &optional; last_alert: string &log &optional;
## Next protocol the server chose using the application layer
## next protocol extension, if present.
next_protocol: string &log &optional;
## The analyzer ID used for the analyzer instance attached ## The analyzer ID used for the analyzer instance attached
## to each connection. It is not used for logging since it's a ## to each connection. It is not used for logging since it's a
@ -149,8 +165,11 @@ event ssl_client_hello(c: connection, version: count, possible_ts: time, client_
set_session(c); set_session(c);
# Save the session_id if there is one set. # Save the session_id if there is one set.
if ( session_id != /^\x00{32}$/ ) if ( |session_id| > 0 && session_id != /^\x00{32}$/ )
{
c$ssl$session_id = bytestring_to_hexstr(session_id); c$ssl$session_id = bytestring_to_hexstr(session_id);
c$ssl$client_ticket_empty_session_seen = F;
}
} }
event ssl_server_hello(c: connection, version: count, possible_ts: time, server_random: string, session_id: string, cipher: count, comp_method: count) &priority=5 event ssl_server_hello(c: connection, version: count, possible_ts: time, server_random: string, session_id: string, cipher: count, comp_method: count) &priority=5
@ -159,6 +178,9 @@ event ssl_server_hello(c: connection, version: count, possible_ts: time, server_
c$ssl$version = version_strings[version]; c$ssl$version = version_strings[version];
c$ssl$cipher = cipher_desc[cipher]; c$ssl$cipher = cipher_desc[cipher];
if ( c$ssl?$session_id && c$ssl$session_id == bytestring_to_hexstr(session_id) )
c$ssl$resumed = T;
} }
event ssl_server_curve(c: connection, curve: count) &priority=5 event ssl_server_curve(c: connection, curve: count) &priority=5
@ -180,6 +202,45 @@ event ssl_extension_server_name(c: connection, is_orig: bool, names: string_vec)
} }
} }
event ssl_extension_application_layer_protocol_negotiation(c: connection, is_orig: bool, protocols: string_vec)
{
set_session(c);
if ( is_orig )
return;
if ( |protocols| > 0 )
c$ssl$next_protocol = protocols[0];
}
event ssl_handshake_message(c: connection, is_orig: bool, msg_type: count, length: count) &priority=5
{
set_session(c);
if ( is_orig && msg_type == SSL::CLIENT_KEY_EXCHANGE )
c$ssl$client_key_exchange_seen = T;
}
# Extension event is fired _before_ the respective client or server hello.
# Important for client_ticket_empty_session_seen.
event ssl_extension(c: connection, is_orig: bool, code: count, val: string) &priority=5
{
set_session(c);
if ( is_orig && SSL::extensions[code] == "SessionTicket TLS" && |val| > 0 )
# In this case, we might have an empty ID. Set back to F in client_hello event
# if it is not empty after all.
c$ssl$client_ticket_empty_session_seen = T;
}
event ssl_change_cipher_spec(c: connection, is_orig: bool) &priority=5
{
set_session(c);
if ( is_orig && c$ssl$client_ticket_empty_session_seen && ! c$ssl$client_key_exchange_seen )
c$ssl$resumed = T;
}
event ssl_alert(c: connection, is_orig: bool, level: count, desc: count) &priority=5 event ssl_alert(c: connection, is_orig: bool, level: count, desc: count) &priority=5
{ {
set_session(c); set_session(c);

4
scripts/base/utils/active-http.bro
View file

@ -65,12 +65,14 @@ function request2curl(r: Request, bodyfile: string, headersfile: string): string
cmd = fmt("%s -m %.0f", cmd, r$max_time); cmd = fmt("%s -m %.0f", cmd, r$max_time);
if ( r?$client_data ) if ( r?$client_data )
cmd = fmt("%s -d -", cmd); cmd = fmt("%s -d @-", cmd);
if ( r?$addl_curl_args ) if ( r?$addl_curl_args )
cmd = fmt("%s %s", cmd, r$addl_curl_args); cmd = fmt("%s %s", cmd, r$addl_curl_args);
cmd = fmt("%s \"%s\"", cmd, str_shell_escape(r$url)); cmd = fmt("%s \"%s\"", cmd, str_shell_escape(r$url));
# Make sure file will exist even if curl did not write one.
cmd = fmt("%s && touch %s", cmd, str_shell_escape(bodyfile));
return cmd; return cmd;
} }

9
scripts/base/utils/exec.bro
View file

@ -106,6 +106,15 @@ event Input::end_of_data(name: string, source:string)
local track_file = parts[2]; local track_file = parts[2];
# If the file is empty, still add it to the result$files table. This is needed
# because it is expected that the file was read even if it was empty.
local result = results[name];
if ( ! result?$files )
result$files = table();
if ( track_file !in result$files )
result$files[track_file] = vector();
Input::remove(name); Input::remove(name);
if ( name !in pending_files ) if ( name !in pending_files )

8
scripts/policy/protocols/ssl/weak-keys.bro
View file

@ -22,7 +22,7 @@ export {
## The minimal key length in bits that is considered to be safe. Any shorter ## The minimal key length in bits that is considered to be safe. Any shorter
## (non-EC) key lengths will trigger the notice. ## (non-EC) key lengths will trigger the notice.
const notify_minimal_key_length = 1024 &redef; const notify_minimal_key_length = 2048 &redef;
## Warn if the DH key length is smaller than the certificate key length. This is ## Warn if the DH key length is smaller than the certificate key length. This is
## potentially unsafe because it gives a wrong impression of safety due to the ## potentially unsafe because it gives a wrong impression of safety due to the
@ -56,7 +56,7 @@ event ssl_established(c: connection) &priority=3
NOTICE([$note=Weak_Key, NOTICE([$note=Weak_Key,
$msg=fmt("Host uses weak certificate with %d bit key", key_length), $msg=fmt("Host uses weak certificate with %d bit key", key_length),
$conn=c, $suppress_for=1day, $conn=c, $suppress_for=1day,
$identifier=cat(c$id$orig_h, c$id$orig_p, key_length) $identifier=cat(c$id$resp_h, c$id$resp_h, key_length)
]); ]);
} }
@ -71,7 +71,7 @@ event ssl_dh_server_params(c: connection, p: string, q: string, Ys: string) &pri
NOTICE([$note=Weak_Key, NOTICE([$note=Weak_Key,
$msg=fmt("Host uses weak DH parameters with %d key bits", key_length), $msg=fmt("Host uses weak DH parameters with %d key bits", key_length),
$conn=c, $suppress_for=1day, $conn=c, $suppress_for=1day,
$identifier=cat(c$id$orig_h, c$id$orig_p, key_length) $identifier=cat(c$id$resp_h, c$id$resp_p, key_length)
]); ]);
if ( notify_dh_length_shorter_cert_length && if ( notify_dh_length_shorter_cert_length &&
@ -86,7 +86,7 @@ event ssl_dh_server_params(c: connection, p: string, q: string, Ys: string) &pri
$msg=fmt("DH key length of %d bits is smaller certificate key length of %d bits", $msg=fmt("DH key length of %d bits is smaller certificate key length of %d bits",
key_length, c$ssl$cert_chain[0]$x509$certificate$key_length), key_length, c$ssl$cert_chain[0]$x509$certificate$key_length),
$conn=c, $suppress_for=1day, $conn=c, $suppress_for=1day,
$identifier=cat(c$id$orig_h, c$id$orig_p) $identifier=cat(c$id$resp_h, c$id$resp_p)
]); ]);
} }
} }

36
scripts/policy/tuning/logs-to-elasticsearch.bro
View file

@ -1,36 +0,0 @@
##! Load this script to enable global log output to an ElasticSearch database.
module LogElasticSearch;
export {
## An elasticsearch specific rotation interval.
const rotation_interval = 3hr &redef;
## Optionally ignore any :bro:type:`Log::ID` from being sent to
## ElasticSearch with this script.
const excluded_log_ids: set[Log::ID] &redef;
## If you want to explicitly only send certain :bro:type:`Log::ID`
## streams, add them to this set. If the set remains empty, all will
## be sent. The :bro:id:`LogElasticSearch::excluded_log_ids` option
## will remain in effect as well.
const send_logs: set[Log::ID] &redef;
}
event bro_init() &priority=-5
{
if ( server_host == "" )
return;
for ( stream_id in Log::active_streams )
{
if ( stream_id in excluded_log_ids ||
(|send_logs| > 0 && stream_id !in send_logs) )
next;
local filter: Log::Filter = [$name = "default-es",
$writer = Log::WRITER_ELASTICSEARCH,
$interv = LogElasticSearch::rotation_interval];
Log::add_filter(stream_id, filter);
}
}

3
scripts/test-all-policy.bro
View file

@ -98,7 +98,4 @@
@load tuning/defaults/packet-fragments.bro @load tuning/defaults/packet-fragments.bro
@load tuning/defaults/warnings.bro @load tuning/defaults/warnings.bro
@load tuning/json-logs.bro @load tuning/json-logs.bro
@load tuning/logs-to-elasticsearch.bro
@load tuning/track-all-assets.bro @load tuning/track-all-assets.bro
redef LogElasticSearch::server_host = "";

57
src/CMakeLists.txt
View file

@ -8,6 +8,7 @@ set(bro_ALL_GENERATED_OUTPUTS CACHE INTERNAL "automatically generated files" FO
# This collects bif inputs that we'll load automatically. # This collects bif inputs that we'll load automatically.
set(bro_AUTO_BIFS CACHE INTERNAL "BIFs for automatic inclusion" FORCE) set(bro_AUTO_BIFS CACHE INTERNAL "BIFs for automatic inclusion" FORCE)
set(bro_REGISTER_BIFS CACHE INTERNAL "BIFs for automatic registering" FORCE)
set(bro_BASE_BIF_SCRIPTS CACHE INTERNAL "Bro script stubs for BIFs in base distribution of Bro" FORCE) set(bro_BASE_BIF_SCRIPTS CACHE INTERNAL "Bro script stubs for BIFs in base distribution of Bro" FORCE)
set(bro_PLUGIN_BIF_SCRIPTS CACHE INTERNAL "Bro script stubs for BIFs in Bro plugins" FORCE) set(bro_PLUGIN_BIF_SCRIPTS CACHE INTERNAL "Bro script stubs for BIFs in Bro plugins" FORCE)
@ -117,8 +118,6 @@ include(BifCl)
set(BIF_SRCS set(BIF_SRCS
bro.bif bro.bif
logging.bif
input.bif
event.bif event.bif
const.bif const.bif
types.bif types.bif
@ -155,21 +154,25 @@ set(bro_SUBDIR_LIBS CACHE INTERNAL "subdir libraries" FORCE)
set(bro_PLUGIN_LIBS CACHE INTERNAL "plugin libraries" FORCE) set(bro_PLUGIN_LIBS CACHE INTERNAL "plugin libraries" FORCE)
add_subdirectory(analyzer) add_subdirectory(analyzer)
add_subdirectory(file_analysis)
add_subdirectory(probabilistic)
add_subdirectory(broxygen) add_subdirectory(broxygen)
add_subdirectory(file_analysis)
add_subdirectory(input)
add_subdirectory(iosource)
add_subdirectory(logging)
add_subdirectory(probabilistic)
set(bro_SUBDIRS set(bro_SUBDIRS
${bro_SUBDIR_LIBS} # Order is important here.
${bro_PLUGIN_LIBS} ${bro_PLUGIN_LIBS}
${bro_SUBDIR_LIBS}
) )
if ( NOT bro_HAVE_OBJECT_LIBRARIES ) if ( NOT bro_HAVE_OBJECT_LIBRARIES )
foreach (_plugin ${bro_PLUGIN_LIBS}) foreach (_plugin ${bro_PLUGIN_LIBS})
string(REGEX REPLACE "plugin-" "" _plugin "${_plugin}") string(REGEX REPLACE "plugin-" "" _plugin "${_plugin}")
string(REGEX REPLACE "-" "_" _plugin "${_plugin}") string(REGEX REPLACE "-" "_" _plugin "${_plugin}")
set(_decl "namespace plugin { namespace ${_plugin} { class Plugin; extern Plugin __plugin; } };") set(_decl "namespace plugin { namespace ${_plugin} { class Plugin; extern Plugin plugin; } };")
set(_use "i += (size_t)(&(plugin::${_plugin}::__plugin));") set(_use "i += (size_t)(&(plugin::${_plugin}::plugin));")
set(__BRO_DECL_PLUGINS "${__BRO_DECL_PLUGINS}${_decl}\n") set(__BRO_DECL_PLUGINS "${__BRO_DECL_PLUGINS}${_decl}\n")
set(__BRO_USE_PLUGINS "${__BRO_USE_PLUGINS}${_use}\n") set(__BRO_USE_PLUGINS "${__BRO_USE_PLUGINS}${_use}\n")
endforeach() endforeach()
@ -252,7 +255,6 @@ set(bro_SRCS
Anon.cc Anon.cc
Attr.cc Attr.cc
Base64.cc Base64.cc
BPF_Program.cc
Brofiler.cc Brofiler.cc
BroString.cc BroString.cc
CCL.cc CCL.cc
@ -277,14 +279,13 @@ set(bro_SRCS
EventRegistry.cc EventRegistry.cc
Expr.cc Expr.cc
File.cc File.cc
FlowSrc.cc Flare.cc
Frag.cc Frag.cc
Frame.cc Frame.cc
Func.cc Func.cc
Hash.cc Hash.cc
ID.cc ID.cc
IntSet.cc IntSet.cc
IOSource.cc
IP.cc IP.cc
IPAddr.cc IPAddr.cc
List.cc List.cc
@ -297,7 +298,7 @@ set(bro_SRCS
OSFinger.cc OSFinger.cc
PacketFilter.cc PacketFilter.cc
PersistenceSerializer.cc PersistenceSerializer.cc
PktSrc.cc Pipe.cc
PolicyFile.cc PolicyFile.cc
PrefixTable.cc PrefixTable.cc
PriorityQueue.cc PriorityQueue.cc
@ -346,24 +347,6 @@ set(bro_SRCS
threading/formatters/Ascii.cc threading/formatters/Ascii.cc
threading/formatters/JSON.cc threading/formatters/JSON.cc
logging/Manager.cc
logging/WriterBackend.cc
logging/WriterFrontend.cc
logging/writers/Ascii.cc
logging/writers/DataSeries.cc
logging/writers/SQLite.cc
logging/writers/ElasticSearch.cc
logging/writers/None.cc
input/Manager.cc
input/ReaderBackend.cc
input/ReaderFrontend.cc
input/readers/Ascii.cc
input/readers/Raw.cc
input/readers/Benchmark.cc
input/readers/Binary.cc
input/readers/SQLite.cc
3rdparty/sqlite3.c 3rdparty/sqlite3.c
plugin/Component.cc plugin/Component.cc
@ -371,7 +354,6 @@ set(bro_SRCS
plugin/TaggedComponent.h plugin/TaggedComponent.h
plugin/Manager.cc plugin/Manager.cc
plugin/Plugin.cc plugin/Plugin.cc
plugin/Macros.h
nb_dns.c nb_dns.c
digest.h digest.h
@ -387,22 +369,31 @@ else ()
target_link_libraries(bro ${bro_SUBDIRS} ${brodeps} ${CMAKE_THREAD_LIBS_INIT} ${CMAKE_DL_LIBS}) target_link_libraries(bro ${bro_SUBDIRS} ${brodeps} ${CMAKE_THREAD_LIBS_INIT} ${CMAKE_DL_LIBS})
endif () endif ()
if ( NOT "${bro_LINKER_FLAGS}" STREQUAL "" )
set_target_properties(bro PROPERTIES LINK_FLAGS "${bro_LINKER_FLAGS}")
endif ()
install(TARGETS bro DESTINATION bin) install(TARGETS bro DESTINATION bin)
set(BRO_EXE bro set(BRO_EXE bro
CACHE STRING "Bro executable binary" FORCE) CACHE STRING "Bro executable binary" FORCE)
set(BRO_EXE_PATH ${CMAKE_CURRENT_BINARY_DIR}/bro
CACHE STRING "Path to Bro executable binary" FORCE)
# Target to create all the autogenerated files. # Target to create all the autogenerated files.
add_custom_target(generate_outputs_stage1) add_custom_target(generate_outputs_stage1)
add_dependencies(generate_outputs_stage1 ${bro_ALL_GENERATED_OUTPUTS}) add_dependencies(generate_outputs_stage1 ${bro_ALL_GENERATED_OUTPUTS})
# Target to create the joint includes files that pull in the bif code. # Target to create the joint includes files that pull in the bif code.
bro_bif_create_includes(generate_outputs_stage2 ${CMAKE_CURRENT_BINARY_DIR} "${bro_AUTO_BIFS}") bro_bif_create_includes(generate_outputs_stage2a ${CMAKE_CURRENT_BINARY_DIR} "${bro_AUTO_BIFS}")
add_dependencies(generate_outputs_stage2 generate_outputs_stage1) bro_bif_create_register(generate_outputs_stage2b ${CMAKE_CURRENT_BINARY_DIR} "${bro_REGISTER_BIFS}")
add_dependencies(generate_outputs_stage2a generate_outputs_stage1)
add_dependencies(generate_outputs_stage2b generate_outputs_stage1)
# Global target to trigger creation of autogenerated code. # Global target to trigger creation of autogenerated code.
add_custom_target(generate_outputs) add_custom_target(generate_outputs)
add_dependencies(generate_outputs generate_outputs_stage2) add_dependencies(generate_outputs generate_outputs_stage2a generate_outputs_stage2b)
# Build __load__.bro files for standard *.bif.bro. # Build __load__.bro files for standard *.bif.bro.
bro_bif_create_loader(bif_loader "${bro_BASE_BIF_SCRIPTS}") bro_bif_create_loader(bif_loader "${bro_BASE_BIF_SCRIPTS}")

46
src/ChunkedIO.cc
View file

@ -210,6 +210,7 @@ bool ChunkedIOFd::WriteChunk(Chunk* chunk, bool partial)
else else
pending_head = pending_tail = q; pending_head = pending_tail = q;
write_flare.Fire();
return Flush(); return Flush();
} }
@ -232,6 +233,7 @@ bool ChunkedIOFd::PutIntoWriteBuffer(Chunk* chunk)
write_len += len; write_len += len;
delete chunk; delete chunk;
write_flare.Fire();
if ( network_time - last_flush > 0.005 ) if ( network_time - last_flush > 0.005 )
FlushWriteBuffer(); FlushWriteBuffer();
@ -269,6 +271,10 @@ bool ChunkedIOFd::FlushWriteBuffer()
if ( unsigned(written) == len ) if ( unsigned(written) == len )
{ {
write_pos = write_len = 0; write_pos = write_len = 0;
if ( ! pending_head )
write_flare.Extinguish();
return true; return true;
} }
@ -318,7 +324,12 @@ bool ChunkedIOFd::Flush()
} }
} }
return FlushWriteBuffer(); bool rval = FlushWriteBuffer();
if ( ! pending_head && write_len == 0 )
write_flare.Extinguish();
return rval;
} }
uint32 ChunkedIOFd::ChunkAvailable() uint32 ChunkedIOFd::ChunkAvailable()
@ -394,6 +405,9 @@ bool ChunkedIOFd::Read(Chunk** chunk, bool may_block)
#ifdef DEBUG_COMMUNICATION #ifdef DEBUG_COMMUNICATION
AddToBuffer("<false:read-chunk>", true); AddToBuffer("<false:read-chunk>", true);
#endif #endif
if ( ! ChunkAvailable() )
read_flare.Extinguish();
return false; return false;
} }
@ -402,9 +416,15 @@ bool ChunkedIOFd::Read(Chunk** chunk, bool may_block)
#ifdef DEBUG_COMMUNICATION #ifdef DEBUG_COMMUNICATION
AddToBuffer("<null:no-data>", true); AddToBuffer("<null:no-data>", true);
#endif #endif
read_flare.Extinguish();
return true; return true;
} }
if ( ChunkAvailable() )
read_flare.Fire();
else
read_flare.Extinguish();
#ifdef DEBUG #ifdef DEBUG
if ( *chunk ) if ( *chunk )
DBG_LOG(DBG_CHUNKEDIO, "read of size %d %s[%s]", DBG_LOG(DBG_CHUNKEDIO, "read of size %d %s[%s]",
@ -481,6 +501,9 @@ bool ChunkedIOFd::ReadChunk(Chunk** chunk, bool may_block)
read_pos = 0; read_pos = 0;
read_len = bytes_left; read_len = bytes_left;
if ( ! ChunkAvailable() )
read_flare.Extinguish();
// If allowed, wait a bit for something to read. // If allowed, wait a bit for something to read.
if ( may_block ) if ( may_block )
{ {
@ -607,6 +630,14 @@ bool ChunkedIOFd::IsFillingUp()
return stats.pending > MAX_BUFFERED_CHUNKS_SOFT; return stats.pending > MAX_BUFFERED_CHUNKS_SOFT;
} }
iosource::FD_Set ChunkedIOFd::ExtraReadFDs() const
{
iosource::FD_Set rval;
rval.Insert(write_flare.FD());
rval.Insert(read_flare.FD());
return rval;
}
void ChunkedIOFd::Clear() void ChunkedIOFd::Clear()
{ {
while ( pending_head ) while ( pending_head )
@ -618,6 +649,9 @@ void ChunkedIOFd::Clear()
} }
pending_head = pending_tail = 0; pending_head = pending_tail = 0;
if ( write_len == 0 )
write_flare.Extinguish();
} }
const char* ChunkedIOFd::Error() const char* ChunkedIOFd::Error()
@ -830,6 +864,7 @@ bool ChunkedIOSSL::Write(Chunk* chunk)
else else
write_head = write_tail = q; write_head = write_tail = q;
write_flare.Fire();
Flush(); Flush();
return true; return true;
} }
@ -935,6 +970,7 @@ bool ChunkedIOSSL::Flush()
write_state = LEN; write_state = LEN;
} }
write_flare.Extinguish();
return true; return true;
} }
@ -1104,6 +1140,13 @@ bool ChunkedIOSSL::IsFillingUp()
return false; return false;
} }
iosource::FD_Set ChunkedIOSSL::ExtraReadFDs() const
{
iosource::FD_Set rval;
rval.Insert(write_flare.FD());
return rval;
}
void ChunkedIOSSL::Clear() void ChunkedIOSSL::Clear()
{ {
while ( write_head ) while ( write_head )
@ -1114,6 +1157,7 @@ void ChunkedIOSSL::Clear()
write_head = next; write_head = next;
} }
write_head = write_tail = 0; write_head = write_tail = 0;
write_flare.Extinguish();
} }
const char* ChunkedIOSSL::Error() const char* ChunkedIOSSL::Error()

16
src/ChunkedIO.h
View file

@ -6,7 +6,8 @@
#include "config.h" #include "config.h"
#include "List.h" #include "List.h"
#include "util.h" #include "util.h"
#include "Flare.h"
#include "iosource/FD_Set.h"
#include <list> #include <list>
#ifdef NEED_KRB5_H #ifdef NEED_KRB5_H
@ -95,6 +96,11 @@ public:
// Returns underlying fd if available, -1 otherwise. // Returns underlying fd if available, -1 otherwise.
virtual int Fd() { return -1; } virtual int Fd() { return -1; }
// Returns supplementary file descriptors that become read-ready in order
// to signal that there is some work that can be performed.
virtual iosource::FD_Set ExtraReadFDs() const
{ return iosource::FD_Set(); }
// Makes sure that no additional protocol data is written into // Makes sure that no additional protocol data is written into
// the output stream. If this is activated, the output cannot // the output stream. If this is activated, the output cannot
// be read again by any of these classes! // be read again by any of these classes!
@ -177,6 +183,7 @@ public:
virtual void Clear(); virtual void Clear();
virtual bool Eof() { return eof; } virtual bool Eof() { return eof; }
virtual int Fd() { return fd; } virtual int Fd() { return fd; }
virtual iosource::FD_Set ExtraReadFDs() const;
virtual void Stats(char* buffer, int length); virtual void Stats(char* buffer, int length);
private: private:
@ -240,6 +247,8 @@ private:
ChunkQueue* pending_tail; ChunkQueue* pending_tail;
pid_t pid; pid_t pid;
bro::Flare write_flare;
bro::Flare read_flare;
}; };
// Chunked I/O using an SSL connection. // Chunked I/O using an SSL connection.
@ -262,6 +271,7 @@ public:
virtual void Clear(); virtual void Clear();
virtual bool Eof() { return eof; } virtual bool Eof() { return eof; }
virtual int Fd() { return socket; } virtual int Fd() { return socket; }
virtual iosource::FD_Set ExtraReadFDs() const;
virtual void Stats(char* buffer, int length); virtual void Stats(char* buffer, int length);
private: private:
@ -303,6 +313,8 @@ private:
// One SSL for all connections. // One SSL for all connections.
static SSL_CTX* ctx; static SSL_CTX* ctx;
bro::Flare write_flare;
}; };
#include <zlib.h> #include <zlib.h>
@ -328,6 +340,8 @@ public:
virtual bool Eof() { return io->Eof(); } virtual bool Eof() { return io->Eof(); }
virtual int Fd() { return io->Fd(); } virtual int Fd() { return io->Fd(); }
virtual iosource::FD_Set ExtraReadFDs() const
{ return io->ExtraReadFDs(); }
virtual void Stats(char* buffer, int length); virtual void Stats(char* buffer, int length);
void EnableCompression(int level) void EnableCompression(int level)

18
src/DNS_Mgr.cc
View file

@ -35,6 +35,7 @@
#include "Net.h" #include "Net.h"
#include "Var.h" #include "Var.h"
#include "Reporter.h" #include "Reporter.h"
#include "iosource/Manager.h"
extern "C" { extern "C" {
extern int select(int, fd_set *, fd_set *, fd_set *, struct timeval *); extern int select(int, fd_set *, fd_set *, fd_set *, struct timeval *);
@ -404,17 +405,17 @@ DNS_Mgr::~DNS_Mgr()
delete [] dir; delete [] dir;
} }
bool DNS_Mgr::Init() void DNS_Mgr::InitPostScript()
{ {
if ( did_init ) if ( did_init )
return true; return;
const char* cache_dir = dir ? dir : "."; const char* cache_dir = dir ? dir : ".";
if ( mode == DNS_PRIME && ! ensure_dir(cache_dir) ) if ( mode == DNS_PRIME && ! ensure_dir(cache_dir) )
{ {
did_init = 0; did_init = 0;
return false; return;
} }
cache_name = new char[strlen(cache_dir) + 64]; cache_name = new char[strlen(cache_dir) + 64];
@ -433,14 +434,12 @@ bool DNS_Mgr::Init()
did_init = 1; did_init = 1;
io_sources.Register(this, true); iosource_mgr->Register(this, true);
// We never set idle to false, having the main loop only calling us from // We never set idle to false, having the main loop only calling us from
// time to time. If we're issuing more DNS requests than we can handle // time to time. If we're issuing more DNS requests than we can handle
// in this way, we are having problems anyway ... // in this way, we are having problems anyway ...
idle = true; SetIdle(true);
return true;
} }
static TableVal* fake_name_lookup_result(const char* name) static TableVal* fake_name_lookup_result(const char* name)
@ -1217,9 +1216,10 @@ void DNS_Mgr::IssueAsyncRequests()
} }
} }
void DNS_Mgr::GetFds(int* read, int* write, int* except) void DNS_Mgr::GetFds(iosource::FD_Set* read, iosource::FD_Set* write,
iosource::FD_Set* except)
{ {
*read = nb_dns_fd(nb_dns); read->Insert(nb_dns_fd(nb_dns));
} }
double DNS_Mgr::NextTimestamp(double* network_time) double DNS_Mgr::NextTimestamp(double* network_time)

9
src/DNS_Mgr.h
View file

@ -12,7 +12,7 @@
#include "BroList.h" #include "BroList.h"
#include "Dict.h" #include "Dict.h"
#include "EventHandler.h" #include "EventHandler.h"
#include "IOSource.h" #include "iosource/IOSource.h"
#include "IPAddr.h" #include "IPAddr.h"
class Val; class Val;
@ -40,12 +40,12 @@ enum DNS_MgrMode {
// Number of seconds we'll wait for a reply. // Number of seconds we'll wait for a reply.
#define DNS_TIMEOUT 5 #define DNS_TIMEOUT 5
class DNS_Mgr : public IOSource { class DNS_Mgr : public iosource::IOSource {
public: public:
DNS_Mgr(DNS_MgrMode mode); DNS_Mgr(DNS_MgrMode mode);
virtual ~DNS_Mgr(); virtual ~DNS_Mgr();
bool Init(); void InitPostScript();
void Flush(); void Flush();
// Looks up the address or addresses of the given host, and returns // Looks up the address or addresses of the given host, and returns
@ -132,7 +132,8 @@ protected:
void DoProcess(bool flush); void DoProcess(bool flush);
// IOSource interface. // IOSource interface.
virtual void GetFds(int* read, int* write, int* except); virtual void GetFds(iosource::FD_Set* read, iosource::FD_Set* write,
iosource::FD_Set* except);
virtual double NextTimestamp(double* network_time); virtual double NextTimestamp(double* network_time);
virtual void Process(); virtual void Process();
virtual const char* Tag() { return "DNS_Mgr"; } virtual const char* Tag() { return "DNS_Mgr"; }

28
src/DebugLogger.cc
View file

@ -5,6 +5,7 @@
#include "DebugLogger.h" #include "DebugLogger.h"
#include "Net.h" #include "Net.h"
#include "plugin/Plugin.h"
DebugLogger debug_logger("debug"); DebugLogger debug_logger("debug");
@ -17,7 +18,8 @@ DebugLogger::Stream DebugLogger::streams[NUM_DBGS] = {
{ "dpd", 0, false }, { "tm", 0, false }, { "dpd", 0, false }, { "tm", 0, false },
{ "logging", 0, false }, {"input", 0, false }, { "logging", 0, false }, {"input", 0, false },
{ "threading", 0, false }, { "file_analysis", 0, false }, { "threading", 0, false }, { "file_analysis", 0, false },
{ "plugins", 0, false }, { "broxygen", 0, false } { "plugins", 0, false }, { "broxygen", 0, false },
{ "pktio", 0, false}
}; };
DebugLogger::DebugLogger(const char* filename) DebugLogger::DebugLogger(const char* filename)
@ -73,10 +75,12 @@ void DebugLogger::EnableStreams(const char* s)
{ {
if ( strcasecmp("verbose", tok) == 0 ) if ( strcasecmp("verbose", tok) == 0 )
verbose = true; verbose = true;
else else if ( strncmp(tok, "plugin-", 7) != 0 )
reporter->FatalError("unknown debug stream %s\n", tok); reporter->FatalError("unknown debug stream %s\n", tok);
} }
enabled_streams.insert(tok);
tok = strtok(0, ","); tok = strtok(0, ",");
} }
@ -105,4 +109,24 @@ void DebugLogger::Log(DebugStream stream, const char* fmt, ...)
fflush(file); fflush(file);
} }
void DebugLogger::Log(const plugin::Plugin& plugin, const char* fmt, ...)
{
string tok = string("plugin-") + plugin.Name();
tok = strreplace(tok, "::", "-");
if ( enabled_streams.find(tok) == enabled_streams.end() )
return;
fprintf(file, "%17.06f/%17.06f [plugin %s] ",
network_time, current_time(true), plugin.Name().c_str());
va_list ap;
va_start(ap, fmt);
vfprintf(file, fmt, ap);
va_end(ap);
fputc('\n', file);
fflush(file);
}
#endif #endif

15
src/DebugLogger.h
View file

@ -7,6 +7,8 @@
#ifdef DEBUG #ifdef DEBUG
#include <stdio.h> #include <stdio.h>
#include <string>
#include <set>
// To add a new debugging stream, add a constant here as well as // To add a new debugging stream, add a constant here as well as
// an entry to DebugLogger::streams in DebugLogger.cc. // an entry to DebugLogger::streams in DebugLogger.cc.
@ -27,8 +29,9 @@ enum DebugStream {
DBG_INPUT, // Input streams DBG_INPUT, // Input streams
DBG_THREADING, // Threading system DBG_THREADING, // Threading system
DBG_FILE_ANALYSIS, // File analysis DBG_FILE_ANALYSIS, // File analysis
DBG_PLUGINS, DBG_PLUGINS, // Plugin system
DBG_BROXYGEN, DBG_BROXYGEN, // Broxygen
DBG_PKTIO, // Packet sources and dumpers.
NUM_DBGS // Has to be last NUM_DBGS // Has to be last
}; };
@ -42,6 +45,10 @@ enum DebugStream {
#define DBG_PUSH(stream) debug_logger.PushIndent(stream) #define DBG_PUSH(stream) debug_logger.PushIndent(stream)
#define DBG_POP(stream) debug_logger.PopIndent(stream) #define DBG_POP(stream) debug_logger.PopIndent(stream)
#define PLUGIN_DBG_LOG(plugin, args...) debug_logger.Log(plugin, args)
namespace plugin { class Plugin; }
class DebugLogger { class DebugLogger {
public: public:
// Output goes to stderr per default. // Output goes to stderr per default.
@ -49,6 +56,7 @@ public:
~DebugLogger(); ~DebugLogger();
void Log(DebugStream stream, const char* fmt, ...); void Log(DebugStream stream, const char* fmt, ...);
void Log(const plugin::Plugin& plugin, const char* fmt, ...);
void PushIndent(DebugStream stream) void PushIndent(DebugStream stream)
{ ++streams[int(stream)].indent; } { ++streams[int(stream)].indent; }
@ -79,6 +87,8 @@ private:
bool enabled; bool enabled;
}; };
std::set<std::string> enabled_streams;
static Stream streams[NUM_DBGS]; static Stream streams[NUM_DBGS];
}; };
@ -89,6 +99,7 @@ extern DebugLogger debug_logger;
#define DBG_LOG_VERBOSE(args...) #define DBG_LOG_VERBOSE(args...)
#define DBG_PUSH(stream) #define DBG_PUSH(stream)
#define DBG_POP(stream) #define DBG_POP(stream)
#define PLUGIN_DBG_LOG(plugin, args...)
#endif #endif
#endif #endif

8
src/Event.cc
View file

@ -6,6 +6,7 @@
#include "Func.h" #include "Func.h"
#include "NetVar.h" #include "NetVar.h"
#include "Trigger.h" #include "Trigger.h"
#include "plugin/Manager.h"
EventMgr mgr; EventMgr mgr;
@ -77,6 +78,11 @@ EventMgr::~EventMgr()
void EventMgr::QueueEvent(Event* event) void EventMgr::QueueEvent(Event* event)
{ {
bool done = PLUGIN_HOOK_WITH_RESULT(HOOK_QUEUE_EVENT, HookQueueEvent(event), false);
if ( done )
return;
if ( ! head ) if ( ! head )
head = tail = event; head = tail = event;
else else
@ -115,6 +121,8 @@ void EventMgr::Drain()
SegmentProfiler(segment_logger, "draining-events"); SegmentProfiler(segment_logger, "draining-events");
PLUGIN_HOOK_VOID(HOOK_DRAIN_EVENTS, HookDrainEvents());
draining = true; draining = true;
while ( head ) while ( head )
Dispatch(); Dispatch();

2
src/Event.h
View file

@ -24,6 +24,8 @@ public:
SourceID Source() const { return src; } SourceID Source() const { return src; }
analyzer::ID Analyzer() const { return aid; } analyzer::ID Analyzer() const { return aid; }
TimerMgr* Mgr() const { return mgr; } TimerMgr* Mgr() const { return mgr; }
EventHandlerPtr Handler() const { return handler; }
val_list* Args() const { return args; }
void Describe(ODesc* d) const; void Describe(ODesc* d) const;

5
src/EventHandler.cc
View file

@ -13,6 +13,7 @@ EventHandler::EventHandler(const char* arg_name)
type = 0; type = 0;
error_handler = false; error_handler = false;
enabled = true; enabled = true;
generate_always = false;
} }
EventHandler::~EventHandler() EventHandler::~EventHandler()
@ -23,7 +24,9 @@ EventHandler::~EventHandler()
EventHandler::operator bool() const EventHandler::operator bool() const
{ {
return enabled && ((local && local->HasBodies()) || receivers.length()); return enabled && ((local && local->HasBodies())
|| receivers.length()
|| generate_always);
} }
FuncType* EventHandler::FType() FuncType* EventHandler::FType()

6
src/EventHandler.h
View file

@ -43,6 +43,11 @@ public:
void SetEnable(bool arg_enable) { enabled = arg_enable; } void SetEnable(bool arg_enable) { enabled = arg_enable; }
// Flags the event as interesting even if there is no body defined. In
// particular, this will then still pass the event on to plugins.
void SetGenerateAlways() { generate_always = true; }
bool GenerateAlways() { return generate_always; }
// We don't serialize the handler(s) itself here, but // We don't serialize the handler(s) itself here, but
// just the reference to it. // just the reference to it.
bool Serialize(SerialInfo* info) const; bool Serialize(SerialInfo* info) const;
@ -57,6 +62,7 @@ private:
bool used; // this handler is indeed used somewhere bool used; // this handler is indeed used somewhere
bool enabled; bool enabled;
bool error_handler; // this handler reports error messages. bool error_handler; // this handler reports error messages.
bool generate_always;
declare(List, SourceID); declare(List, SourceID);
typedef List(SourceID) receiver_list; typedef List(SourceID) receiver_list;

17
src/EventRegistry.cc
View file

@ -71,6 +71,23 @@ EventRegistry::string_list* EventRegistry::UsedHandlers()
return names; return names;
} }
EventRegistry::string_list* EventRegistry::AllHandlers()
{
string_list* names = new string_list;
IterCookie* c = handlers.InitForIteration();
HashKey* k;
EventHandler* v;
while ( (v = handlers.NextEntry(k, c)) )
{
names->append(v->Name());
delete k;
}
return names;
}
void EventRegistry::PrintDebug() void EventRegistry::PrintDebug()
{ {
IterCookie* c = handlers.InitForIteration(); IterCookie* c = handlers.InitForIteration();

2
src/EventRegistry.h
View file

@ -33,6 +33,8 @@ public:
string_list* UnusedHandlers(); string_list* UnusedHandlers();
string_list* UsedHandlers(); string_list* UsedHandlers();
string_list* AllHandlers();
void PrintDebug(); void PrintDebug();
private: private:

7
src/Expr.h
View file

@ -608,6 +608,10 @@ public:
CondExpr(Expr* op1, Expr* op2, Expr* op3); CondExpr(Expr* op1, Expr* op2, Expr* op3);
~CondExpr(); ~CondExpr();
const Expr* Op1() const { return op1; }
const Expr* Op2() const { return op2; }
const Expr* Op3() const { return op3; }
Expr* Simplify(SimplifyType simp_type); Expr* Simplify(SimplifyType simp_type);
Val* Eval(Frame* f) const; Val* Eval(Frame* f) const;
int IsPure() const; int IsPure() const;
@ -706,6 +710,7 @@ public:
~FieldExpr(); ~FieldExpr();
int Field() const { return field; } int Field() const { return field; }
const char* FieldName() const { return field_name; }
int CanDel() const; int CanDel() const;
@ -737,6 +742,8 @@ public:
HasFieldExpr(Expr* op, const char* field_name); HasFieldExpr(Expr* op, const char* field_name);
~HasFieldExpr(); ~HasFieldExpr();
const char* FieldName() const { return field_name; }
protected: protected:
friend class Expr; friend class Expr;
HasFieldExpr() { field_name = 0; } HasFieldExpr() { field_name = 0; }

74
src/Flare.cc Normal file
View file

@ -0,0 +1,74 @@
// See the file "COPYING" in the main distribution directory for copyright.
#include "Flare.h"
#include "Reporter.h"
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
using namespace bro;
Flare::Flare()
: pipe(FD_CLOEXEC, FD_CLOEXEC, O_NONBLOCK, O_NONBLOCK)
{
}
static void bad_pipe_op(const char* which)
{
char buf[256];
strerror_r(errno, buf, sizeof(buf));
reporter->FatalErrorWithCore("unexpected pipe %s failure: %s", which, buf);
}
void Flare::Fire()
{
char tmp;
for ( ; ; )
{
int n = write(pipe.WriteFD(), &tmp, 1);
if ( n > 0 )
// Success -- wrote a byte to pipe.
break;
if ( n < 0 )
{
if ( errno == EAGAIN )
// Success: pipe is full and just need at least one byte in it.
break;
if ( errno == EINTR )
// Interrupted: try again.
continue;
bad_pipe_op("write");
}
// No error, but didn't write a byte: try again.
}
}
void Flare::Extinguish()
{
char tmp[256];
for ( ; ; )
{
int n = read(pipe.ReadFD(), &tmp, sizeof(tmp));
if ( n >= 0 )
// Pipe may not be empty yet: try again.
continue;
if ( errno == EAGAIN )
// Success: pipe is now empty.
break;
if ( errno == EINTR )
// Interrupted: try again.
continue;
bad_pipe_op("read");
}
}

44
src/Flare.h Normal file
View file

@ -0,0 +1,44 @@
// See the file "COPYING" in the main distribution directory for copyright.
#ifndef BRO_FLARE_H
#define BRO_FLARE_H
#include "Pipe.h"
namespace bro {
class Flare {
public:
/**
* Create a flare object that can be used to signal a "ready" status via
* a file descriptor that may be integrated with select(), poll(), etc.
* Not thread-safe, but that should only require Fire()/Extinguish() calls
* to be made mutually exclusive (across all copies of a Flare).
*/
Flare();
/**
* @return a file descriptor that will become ready if the flare has been
* Fire()'d and not yet Extinguished()'d.
*/
int FD() const
{ return pipe.ReadFD(); }
/**
* Put the object in the "ready" state.
*/
void Fire();
/**
* Take the object out of the "ready" state.
*/
void Extinguish();
private:
Pipe pipe;
};
} // namespace bro
#endif // BRO_FLARE_H

228
src/FlowSrc.cc
View file

@ -1,228 +0,0 @@
// See the file "COPYING" in the main distribution directory for copyright.
//
// Written by Bernhard Ager, TU Berlin (2006/2007).
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <netdb.h>
#include "FlowSrc.h"
#include "Net.h"
#include "analyzer/protocol/netflow/netflow_pac.h"
#include <errno.h>
FlowSrc::FlowSrc()
{ // TODO: v9.
selectable_fd = -1;
idle = false;
data = 0;
pdu_len = -1;
exporter_ip = 0;
current_timestamp = next_timestamp = 0.0;
netflow_analyzer = new binpac::NetFlow::NetFlow_Analyzer();
}
FlowSrc::~FlowSrc()
{
delete netflow_analyzer;
}
void FlowSrc::GetFds(int* read, int* write, int* except)
{
if ( selectable_fd >= 0 )
*read = selectable_fd;
}
double FlowSrc::NextTimestamp(double* network_time)
{
if ( ! data && ! ExtractNextPDU() )
return -1.0;
else
return next_timestamp;
}
void FlowSrc::Process()
{
if ( ! data && ! ExtractNextPDU() )
return;
// This is normally done by calling net_packet_dispatch(),
// but as we don't have a packet to dispatch ...
network_time = next_timestamp;
expire_timers();
netflow_analyzer->downflow()->set_exporter_ip(exporter_ip);
// We handle exceptions in NewData (might have changed w/ new binpac).
netflow_analyzer->NewData(0, data, data + pdu_len);
data = 0;
}
void FlowSrc::Close()
{
safe_close(selectable_fd);
}
FlowSocketSrc::~FlowSocketSrc()
{
}
int FlowSocketSrc::ExtractNextPDU()
{
sockaddr_in from;
socklen_t fromlen = sizeof(from);
pdu_len = recvfrom(selectable_fd, buffer, NF_MAX_PKT_SIZE, 0,
(struct sockaddr*) &from, &fromlen);
if ( pdu_len < 0 )
{
reporter->Error("problem reading NetFlow data from socket");
data = 0;
next_timestamp = -1.0;
closed = 1;
return 0;
}
if ( fromlen != sizeof(from) )
{
reporter->Error("malformed NetFlow PDU");
return 0;
}
data = buffer;
exporter_ip = from.sin_addr.s_addr;
next_timestamp = current_time();
if ( next_timestamp < current_timestamp )
next_timestamp = current_timestamp;
else
current_timestamp = next_timestamp;
return 1;
}
FlowSocketSrc::FlowSocketSrc(const char* listen_parms)
{
int n = strlen(listen_parms) + 1;
char laddr[n], port[n], ident[n];
laddr[0] = port[0] = ident[0] = '\0';
int ret = sscanf(listen_parms, "%[^:]:%[^=]=%s", laddr, port, ident);
if ( ret < 2 )
{
snprintf(errbuf, BRO_FLOW_ERRBUF_SIZE,
"parsing your listen-spec went nuts: laddr='%s', port='%s'\n",
laddr[0] ? laddr : "", port[0] ? port : "");
closed = 1;
return;
}
const char* id = (ret == 3) ? ident : listen_parms;
netflow_analyzer->downflow()->set_identifier(id);
struct addrinfo aiprefs = {
0, PF_INET, SOCK_DGRAM, IPPROTO_UDP, 0, NULL, NULL, NULL
};
struct addrinfo* ainfo = 0;
if ( (ret = getaddrinfo(laddr, port, &aiprefs, &ainfo)) != 0 )
{
snprintf(errbuf, BRO_FLOW_ERRBUF_SIZE,
"getaddrinfo(%s, %s, ...): %s",
laddr, port, gai_strerror(ret));
closed = 1;
return;
}
if ( (selectable_fd = socket (PF_INET, SOCK_DGRAM, 0)) < 0 )
{
snprintf(errbuf, BRO_FLOW_ERRBUF_SIZE,
"socket: %s", strerror(errno));
closed = 1;
goto cleanup;
}
if ( bind (selectable_fd, ainfo->ai_addr, ainfo->ai_addrlen) < 0 )
{
snprintf(errbuf, BRO_FLOW_ERRBUF_SIZE,
"bind: %s", strerror(errno));
closed = 1;
goto cleanup;
}
cleanup:
freeaddrinfo(ainfo);
}
FlowFileSrc::~FlowFileSrc()
{
delete [] readfile;
}
int FlowFileSrc::ExtractNextPDU()
{
FlowFileSrcPDUHeader pdu_header;
if ( read(selectable_fd, &pdu_header, sizeof(pdu_header)) <
int(sizeof(pdu_header)) )
return Error(errno, "read header");
if ( pdu_header.pdu_length > NF_MAX_PKT_SIZE )
{
reporter->Error("NetFlow packet too long");
// Safely skip over the too-long PDU.
if ( lseek(selectable_fd, pdu_header.pdu_length, SEEK_CUR) < 0 )
return Error(errno, "lseek");
return 0;
}
if ( read(selectable_fd, buffer, pdu_header.pdu_length) <
pdu_header.pdu_length )
return Error(errno, "read data");
if ( next_timestamp < pdu_header.network_time )
{
next_timestamp = pdu_header.network_time;
current_timestamp = pdu_header.network_time;
}
else
current_timestamp = next_timestamp;
data = buffer;
pdu_len = pdu_header.pdu_length;
exporter_ip = pdu_header.ipaddr;
return 1;
}
FlowFileSrc::FlowFileSrc(const char* readfile)
{
int n = strlen(readfile) + 1;
char ident[n];
this->readfile = new char[n];
int ret = sscanf(readfile, "%[^=]=%s", this->readfile, ident);
const char* id = (ret == 2) ? ident : this->readfile;
netflow_analyzer->downflow()->set_identifier(id);
selectable_fd = open(this->readfile, O_RDONLY);
if ( selectable_fd < 0 )
{
closed = 1;
snprintf(errbuf, BRO_FLOW_ERRBUF_SIZE,
"open: %s", strerror(errno));
}
}
int FlowFileSrc::Error(int errlvl, const char* errmsg)
{
snprintf(errbuf, BRO_FLOW_ERRBUF_SIZE,
"%s: %s", errmsg, strerror(errlvl));
data = 0;
next_timestamp = -1.0;
closed = 1;
return 0;
}

84
src/FlowSrc.h
View file

@ -1,84 +0,0 @@
// See the file "COPYING" in the main distribution directory for copyright.
//
// Written by Bernhard Ager, TU Berlin (2006/2007).
#ifndef flowsrc_h
#define flowsrc_h
#include "IOSource.h"
#include "NetVar.h"
#include "binpac.h"
#define BRO_FLOW_ERRBUF_SIZE 512
// TODO: 1500 is enough for v5 - how about the others?
// 65536 would be enough for any UDP packet.
#define NF_MAX_PKT_SIZE 8192
struct FlowFileSrcPDUHeader {
double network_time;
int pdu_length;
uint32 ipaddr;
};
// Avoid including netflow_pac.h by explicitly declaring the NetFlow_Analyzer.
namespace binpac {
namespace NetFlow {
class NetFlow_Analyzer;
}
}
class FlowSrc : public IOSource {
public:
virtual ~FlowSrc();
// IOSource interface:
bool IsReady();
void GetFds(int* read, int* write, int* except);
double NextTimestamp(double* network_time);
void Process();
const char* Tag() { return "FlowSrc"; }
const char* ErrorMsg() const { return errbuf; }
protected:
FlowSrc();
virtual int ExtractNextPDU() = 0;
virtual void Close();
int selectable_fd;
double current_timestamp;
double next_timestamp;
binpac::NetFlow::NetFlow_Analyzer* netflow_analyzer;
u_char buffer[NF_MAX_PKT_SIZE];
u_char* data;
int pdu_len;
uint32 exporter_ip; // in network byte order
char errbuf[BRO_FLOW_ERRBUF_SIZE];
};
class FlowSocketSrc : public FlowSrc {
public:
FlowSocketSrc(const char* listen_parms);
virtual ~FlowSocketSrc();
int ExtractNextPDU();
};
class FlowFileSrc : public FlowSrc {
public:
FlowFileSrc(const char* readfile);
~FlowFileSrc();
int ExtractNextPDU();
protected:
int Error(int errlvl, const char* errmsg);
char* readfile;
};
#endif

80
src/Func.cc
View file

@ -46,6 +46,7 @@
#include "Event.h" #include "Event.h"
#include "Traverse.h" #include "Traverse.h"
#include "Reporter.h" #include "Reporter.h"
#include "plugin/Manager.h"
extern RETSIGTYPE sig_handler(int signo); extern RETSIGTYPE sig_handler(int signo);
@ -226,7 +227,7 @@ TraversalCode Func::Traverse(TraversalCallback* cb) const
HANDLE_TC_STMT_PRE(tc); HANDLE_TC_STMT_PRE(tc);
// FIXME: Traverse arguments to builtin functions, too. // FIXME: Traverse arguments to builtin functions, too.
if ( kind == BRO_FUNC ) if ( kind == BRO_FUNC && scope )
{ {
tc = scope->Traverse(cb); tc = scope->Traverse(cb);
HANDLE_TC_STMT_PRE(tc); HANDLE_TC_STMT_PRE(tc);
@ -244,6 +245,49 @@ TraversalCode Func::Traverse(TraversalCallback* cb) const
HANDLE_TC_STMT_POST(tc); HANDLE_TC_STMT_POST(tc);
} }
Val* Func::HandlePluginResult(Val* plugin_result, val_list* args, function_flavor flavor) const
{
// Helper function factoring out this code from BroFunc:Call() for better
// readability.
switch ( flavor ) {
case FUNC_FLAVOR_EVENT:
Unref(plugin_result);
plugin_result = 0;
break;
case FUNC_FLAVOR_HOOK:
if ( plugin_result->Type()->Tag() != TYPE_BOOL )
reporter->InternalError("plugin returned non-bool for hook");
break;
case FUNC_FLAVOR_FUNCTION:
{
BroType* yt = FType()->YieldType();
if ( (! yt) || yt->Tag() == TYPE_VOID )
{
Unref(plugin_result);
plugin_result = 0;
}
else
{
if ( plugin_result->Type()->Tag() != yt->Tag() )
reporter->InternalError("plugin returned wrong type for function call");
}
break;
}
}
loop_over_list(*args, i)
Unref((*args)[i]);
return plugin_result;
}
BroFunc::BroFunc(ID* arg_id, Stmt* arg_body, id_list* aggr_inits, BroFunc::BroFunc(ID* arg_id, Stmt* arg_body, id_list* aggr_inits,
int arg_frame_size, int priority) int arg_frame_size, int priority)
: Func(BRO_FUNC) : Func(BRO_FUNC)
@ -281,6 +325,17 @@ Val* BroFunc::Call(val_list* args, Frame* parent) const
#ifdef PROFILE_BRO_FUNCTIONS #ifdef PROFILE_BRO_FUNCTIONS
DEBUG_MSG("Function: %s\n", id->Name()); DEBUG_MSG("Function: %s\n", id->Name());
#endif #endif
SegmentProfiler(segment_logger, location);
if ( sample_logger )
sample_logger->FunctionSeen(this);
Val* plugin_result = PLUGIN_HOOK_WITH_RESULT(HOOK_CALL_FUNCTION, HookCallFunction(this, args), 0);
if ( plugin_result )
return HandlePluginResult(plugin_result, args, Flavor());
if ( bodies.empty() ) if ( bodies.empty() )
{ {
// Can only happen for events and hooks. // Can only happen for events and hooks.
@ -291,7 +346,6 @@ Val* BroFunc::Call(val_list* args, Frame* parent) const
return Flavor() == FUNC_FLAVOR_HOOK ? new Val(true, TYPE_BOOL) : 0; return Flavor() == FUNC_FLAVOR_HOOK ? new Val(true, TYPE_BOOL) : 0;
} }
SegmentProfiler(segment_logger, location);
Frame* f = new Frame(frame_size, this, args); Frame* f = new Frame(frame_size, this, args);
// Hand down any trigger. // Hand down any trigger.
@ -319,9 +373,6 @@ Val* BroFunc::Call(val_list* args, Frame* parent) const
Val* result = 0; Val* result = 0;
if ( sample_logger )
sample_logger->FunctionSeen(this);
for ( size_t i = 0; i < bodies.size(); ++i ) for ( size_t i = 0; i < bodies.size(); ++i )
{ {
if ( sample_logger ) if ( sample_logger )
@ -497,6 +548,11 @@ Val* BuiltinFunc::Call(val_list* args, Frame* parent) const
if ( sample_logger ) if ( sample_logger )
sample_logger->FunctionSeen(this); sample_logger->FunctionSeen(this);
Val* plugin_result = PLUGIN_HOOK_WITH_RESULT(HOOK_CALL_FUNCTION, HookCallFunction(this, args), 0);
if ( plugin_result )
return HandlePluginResult(plugin_result, args, FUNC_FLAVOR_FUNCTION);
if ( g_trace_state.DoTrace() ) if ( g_trace_state.DoTrace() )
{ {
ODesc d; ODesc d;
@ -550,18 +606,15 @@ void builtin_error(const char* msg, BroObj* arg)
} }
#include "bro.bif.func_h" #include "bro.bif.func_h"
#include "logging.bif.func_h"
#include "input.bif.func_h"
#include "reporter.bif.func_h" #include "reporter.bif.func_h"
#include "strings.bif.func_h" #include "strings.bif.func_h"
#include "bro.bif.func_def" #include "bro.bif.func_def"
#include "logging.bif.func_def"
#include "input.bif.func_def"
#include "reporter.bif.func_def" #include "reporter.bif.func_def"
#include "strings.bif.func_def" #include "strings.bif.func_def"
#include "__all__.bif.cc" // Autogenerated for compiling in the bif_target() code. #include "__all__.bif.cc" // Autogenerated for compiling in the bif_target() code.
#include "__all__.bif.register.cc" // Autogenerated for compiling in the bif_target() code.
void init_builtin_funcs() void init_builtin_funcs()
{ {
@ -572,16 +625,17 @@ void init_builtin_funcs()
gap_info = internal_type("gap_info")->AsRecordType(); gap_info = internal_type("gap_info")->AsRecordType();
#include "bro.bif.func_init" #include "bro.bif.func_init"
#include "logging.bif.func_init"
#include "input.bif.func_init"
#include "reporter.bif.func_init" #include "reporter.bif.func_init"
#include "strings.bif.func_init" #include "strings.bif.func_init"
#include "__all__.bif.init.cc" // Autogenerated for compiling in the bif_target() code.
did_builtin_init = true; did_builtin_init = true;
} }
void init_builtin_funcs_subdirs()
{
#include "__all__.bif.init.cc" // Autogenerated for compiling in the bif_target() code.
}
bool check_built_in_call(BuiltinFunc* f, CallExpr* call) bool check_built_in_call(BuiltinFunc* f, CallExpr* call)
{ {
if ( f->TheFunc() != BifFunc::bro_fmt ) if ( f->TheFunc() != BifFunc::bro_fmt )

5
src/Func.h
View file

@ -52,6 +52,7 @@ public:
Kind GetKind() const { return kind; } Kind GetKind() const { return kind; }
const char* Name() const { return name.c_str(); } const char* Name() const { return name.c_str(); }
void SetName(const char* arg_name) { name = arg_name; }
virtual void Describe(ODesc* d) const = 0; virtual void Describe(ODesc* d) const = 0;
virtual void DescribeDebug(ODesc* d, const val_list* args) const; virtual void DescribeDebug(ODesc* d, const val_list* args) const;
@ -69,6 +70,9 @@ public:
protected: protected:
Func(); Func();
// Helper function for handling result of plugin hook.
Val* HandlePluginResult(Val* plugin_result, val_list* args, function_flavor flavor) const;
DECLARE_ABSTRACT_SERIAL(Func); DECLARE_ABSTRACT_SERIAL(Func);
vector<Body> bodies; vector<Body> bodies;
@ -130,6 +134,7 @@ protected:
extern void builtin_error(const char* msg, BroObj* arg = 0); extern void builtin_error(const char* msg, BroObj* arg = 0);
extern void init_builtin_funcs(); extern void init_builtin_funcs();
extern void init_builtin_funcs_subdirs();
extern bool check_built_in_call(BuiltinFunc* f, CallExpr* call); extern bool check_built_in_call(BuiltinFunc* f, CallExpr* call);

2
src/ID.h
View file

@ -32,9 +32,11 @@ public:
void SetType(BroType* t) { Unref(type); type = t; } void SetType(BroType* t) { Unref(type); type = t; }
BroType* Type() { return type; } BroType* Type() { return type; }
const BroType* Type() const { return type; }
void MakeType() { is_type = 1; } void MakeType() { is_type = 1; }
BroType* AsType() { return is_type ? Type() : 0; } BroType* AsType() { return is_type ? Type() : 0; }
const BroType* AsType() const { return is_type ? Type() : 0; }
// If weak_ref is false, the Val is assumed to be already ref'ed // If weak_ref is false, the Val is assumed to be already ref'ed
// and will be deref'ed when the ID is deleted. // and will be deref'ed when the ID is deleted.

176
src/IOSource.cc
View file

@ -1,176 +0,0 @@
#include <sys/types.h>
#include <sys/time.h>
#include <unistd.h>
#include <assert.h>
#include <algorithm>
#include "util.h"
#include "IOSource.h"
IOSourceRegistry io_sources;
IOSourceRegistry::~IOSourceRegistry()
{
for ( SourceList::iterator i = sources.begin(); i != sources.end(); ++i )
delete *i;
sources.clear();
}
void IOSourceRegistry::RemoveAll()
{
// We're cheating a bit here ...
dont_counts = sources.size();
}
IOSource* IOSourceRegistry::FindSoonest(double* ts)
{
// Remove sources which have gone dry. For simplicity, we only
// remove at most one each time.
for ( SourceList::iterator i = sources.begin();
i != sources.end(); ++i )
if ( ! (*i)->src->IsOpen() )
{
delete *i;
sources.erase(i);
break;
}
// Ideally, we would always call select on the fds to see which
// are ready, and return the soonest. Unfortunately, that'd mean
// one select-call per packet, which we can't afford in high-volume
// environments. Thus, we call select only every SELECT_FREQUENCY
// call (or if all sources report that they are dry).
++call_count;
IOSource* soonest_src = 0;
double soonest_ts = 1e20;
double soonest_local_network_time = 1e20;
bool all_idle = true;
// Find soonest source of those which tell us they have something to
// process.
for ( SourceList::iterator i = sources.begin(); i != sources.end(); ++i )
{
if ( ! (*i)->src->IsIdle() )
{
all_idle = false;
double local_network_time = 0;
double ts = (*i)->src->NextTimestamp(&local_network_time);
if ( ts > 0 && ts < soonest_ts )
{
soonest_ts = ts;
soonest_src = (*i)->src;
soonest_local_network_time =
local_network_time ?
local_network_time : ts;
}
}
}
// If we found one and aren't going to select this time,
// return it.
int maxx = 0;
if ( soonest_src && (call_count % SELECT_FREQUENCY) != 0 )
goto finished;
// Select on the join of all file descriptors.
fd_set fd_read, fd_write, fd_except;
FD_ZERO(&fd_read);
FD_ZERO(&fd_write);
FD_ZERO(&fd_except);
for ( SourceList::iterator i = sources.begin();
i != sources.end(); ++i )
{
Source* src = (*i);
if ( ! src->src->IsIdle() )
// No need to select on sources which we know to
// be ready.
continue;
src->fd_read = src->fd_write = src->fd_except = 0;
src->src->GetFds(&src->fd_read, &src->fd_write, &src->fd_except);
FD_SET(src->fd_read, &fd_read);
FD_SET(src->fd_write, &fd_write);
FD_SET(src->fd_except, &fd_except);
maxx = max(src->fd_read, maxx);
maxx = max(src->fd_write, maxx);
maxx = max(src->fd_except, maxx);
}
// We can't block indefinitely even when all sources are dry:
// we're doing some IOSource-independent stuff in the main loop,
// so we need to return from time to time. (Instead of no time-out
// at all, we use a very small one. This lets FreeBSD trigger a
// BPF buffer switch on the next read when the hold buffer is empty
// while the store buffer isn't filled yet.
struct timeval timeout;
if ( all_idle )
{
// Interesting: when all sources are dry, simply sleeping a
// bit *without* watching for any fd becoming ready may
// decrease CPU load. I guess that's because it allows
// the kernel's packet buffers to fill. - Robin
timeout.tv_sec = 0;
timeout.tv_usec = 20; // SELECT_TIMEOUT;
select(0, 0, 0, 0, &timeout);
}
if ( ! maxx )
// No selectable fd at all.
goto finished;
timeout.tv_sec = 0;
timeout.tv_usec = 0;
if ( select(maxx + 1, &fd_read, &fd_write, &fd_except, &timeout) > 0 )
{ // Find soonest.
for ( SourceList::iterator i = sources.begin();
i != sources.end(); ++i )
{
Source* src = (*i);
if ( ! src->src->IsIdle() )
continue;
if ( FD_ISSET(src->fd_read, &fd_read) ||
FD_ISSET(src->fd_write, &fd_write) ||
FD_ISSET(src->fd_except, &fd_except) )
{
double local_network_time = 0;
double ts = src->src->NextTimestamp(&local_network_time);
if ( ts > 0.0 && ts < soonest_ts )
{
soonest_ts = ts;
soonest_src = src->src;
soonest_local_network_time =
local_network_time ?
local_network_time : ts;
}
}
}
}
finished:
*ts = soonest_local_network_time;
return soonest_src;
}
void IOSourceRegistry::Register(IOSource* src, bool dont_count)
{
Source* s = new Source;
s->src = src;
if ( dont_count )
++dont_counts;
return sources.push_back(s);
}

103
src/IOSource.h
View file

@ -1,103 +0,0 @@
// Interface for classes providing/consuming data during Bro's main loop.
#ifndef iosource_h
#define iosource_h
#include <list>
#include "Timer.h"
using namespace std;
class IOSource {
public:
IOSource() { idle = closed = false; }
virtual ~IOSource() {}
// Returns true if source has nothing ready to process.
bool IsIdle() const { return idle; }
// Returns true if more data is to be expected in the future.
// Otherwise, source may be removed.
bool IsOpen() const { return ! closed; }
// Returns select'able fds (leaves args untouched if we don't have
// selectable fds).
virtual void GetFds(int* read, int* write, int* except) = 0;
// The following two methods are only called when either IsIdle()
// returns false or select() on one of the fds indicates that there's
// data to process.
// Returns timestamp (in global network time) associated with next
// data item. If the source wants the data item to be processed
// with a local network time, it sets the argument accordingly.
virtual double NextTimestamp(double* network_time) = 0;
// Processes and consumes next data item.
virtual void Process() = 0;
// Returns tag of timer manager associated with last processed
// data item, nil for global timer manager.
virtual TimerMgr::Tag* GetCurrentTag() { return 0; }
// Returns a descriptual tag for debugging.
virtual const char* Tag() = 0;
protected:
// Derived classed are to set this to true if they have gone dry
// temporarily.
bool idle;
// Derived classed are to set this to true if they have gone dry
// permanently.
bool closed;
};
class IOSourceRegistry {
public:
IOSourceRegistry() { call_count = 0; dont_counts = 0; }
~IOSourceRegistry();
// If dont_count is true, this source does not contribute to the
// number of IOSources returned by Size(). The effect is that
// if all sources but the non-counting ones have gone dry,
// processing will shut down.
void Register(IOSource* src, bool dont_count = false);
// This may block for some time.
IOSource* FindSoonest(double* ts);
int Size() const { return sources.size() - dont_counts; }
// Terminate IOSource processing immediately by removing all
// sources (and therefore returning a Size() of zero).
void Terminate() { RemoveAll(); }
protected:
// When looking for a source with something to process,
// every SELECT_FREQUENCY calls we will go ahead and
// block on a select().
static const int SELECT_FREQUENCY = 25;
// Microseconds to wait in an empty select if no source is ready.
static const int SELECT_TIMEOUT = 50;
void RemoveAll();
unsigned int call_count;
int dont_counts;
struct Source {
IOSource* src;
int fd_read;
int fd_write;
int fd_except;
};
typedef list<Source*> SourceList;
SourceList sources;
};
extern IOSourceRegistry io_sources;
#endif

207
src/Net.cc
View file

@ -29,6 +29,10 @@
#include "Anon.h" #include "Anon.h"
#include "Serializer.h" #include "Serializer.h"
#include "PacketDumper.h" #include "PacketDumper.h"
#include "iosource/Manager.h"
#include "iosource/PktSrc.h"
#include "iosource/PktDumper.h"
#include "plugin/Manager.h"
extern "C" { extern "C" {
#include "setsignal.h" #include "setsignal.h"
@ -38,10 +42,7 @@ extern "C" {
extern int select(int, fd_set *, fd_set *, fd_set *, struct timeval *); extern int select(int, fd_set *, fd_set *, fd_set *, struct timeval *);
} }
PList(PktSrc) pkt_srcs; iosource::PktDumper* pkt_dumper = 0;
// FIXME: We should really merge PktDumper and PacketDumper.
PktDumper* pkt_dumper = 0;
int reading_live = 0; int reading_live = 0;
int reading_traces = 0; int reading_traces = 0;
@ -62,8 +63,8 @@ const u_char* current_pkt = 0;
int current_dispatched = 0; int current_dispatched = 0;
int current_hdr_size = 0; int current_hdr_size = 0;
double current_timestamp = 0.0; double current_timestamp = 0.0;
PktSrc* current_pktsrc = 0; iosource::PktSrc* current_pktsrc = 0;
IOSource* current_iosrc; iosource::IOSource* current_iosrc = 0;
std::list<ScannedFile> files_scanned; std::list<ScannedFile> files_scanned;
std::vector<string> sig_files; std::vector<string> sig_files;
@ -112,17 +113,21 @@ RETSIGTYPE watchdog(int /* signo */)
// saving the packet which caused the // saving the packet which caused the
// watchdog to trigger may be helpful, // watchdog to trigger may be helpful,
// so we'll save that one nevertheless. // so we'll save that one nevertheless.
pkt_dumper = new PktDumper("watchdog-pkt.pcap"); pkt_dumper = iosource_mgr->OpenPktDumper("watchdog-pkt.pcap", false);
if ( pkt_dumper->IsError() ) if ( ! pkt_dumper || pkt_dumper->IsError() )
{ {
reporter->Error("watchdog: can't open watchdog-pkt.pcap for writing\n"); reporter->Error("watchdog: can't open watchdog-pkt.pcap for writing");
delete pkt_dumper;
pkt_dumper = 0; pkt_dumper = 0;
} }
} }
if ( pkt_dumper ) if ( pkt_dumper )
pkt_dumper->Dump(current_hdr, current_pkt); {
iosource::PktDumper::Packet p;
p.hdr = current_hdr;
p.data = current_pkt;
pkt_dumper->Dump(&p);
}
} }
net_get_final_stats(); net_get_final_stats();
@ -141,121 +146,47 @@ RETSIGTYPE watchdog(int /* signo */)
return RETSIGVAL; return RETSIGVAL;
} }
void net_init(name_list& interfaces, name_list& readfiles, void net_update_time(double new_network_time)
name_list& netflows, name_list& flowfiles,
const char* writefile, const char* filter,
const char* secondary_filter, int do_watchdog)
{ {
init_net_var(); network_time = new_network_time;
PLUGIN_HOOK_VOID(HOOK_UPDATE_NETWORK_TIME, HookUpdateNetworkTime(new_network_time));
}
if ( readfiles.length() > 0 || flowfiles.length() > 0 ) void net_init(name_list& interfaces, name_list& readfiles,
const char* writefile, int do_watchdog)
{
if ( readfiles.length() > 0 )
{ {
reading_live = pseudo_realtime > 0.0; reading_live = pseudo_realtime > 0.0;
reading_traces = 1; reading_traces = 1;
for ( int i = 0; i < readfiles.length(); ++i ) for ( int i = 0; i < readfiles.length(); ++i )
{ {
PktFileSrc* ps = new PktFileSrc(readfiles[i], filter); iosource::PktSrc* ps = iosource_mgr->OpenPktSrc(readfiles[i], false);
assert(ps);
if ( ! ps->IsOpen() ) if ( ! ps->IsOpen() )
reporter->FatalError("%s: problem with trace file %s - %s\n", reporter->FatalError("problem with trace file %s (%s)",
prog, readfiles[i], ps->ErrorMsg()); readfiles[i],
else
{
pkt_srcs.append(ps);
io_sources.Register(ps);
}
if ( secondary_filter )
{
// We use a second PktFileSrc for the
// secondary path.
PktFileSrc* ps = new PktFileSrc(readfiles[i],
secondary_filter,
TYPE_FILTER_SECONDARY);
if ( ! ps->IsOpen() )
reporter->FatalError("%s: problem with trace file %s - %s\n",
prog, readfiles[i],
ps->ErrorMsg()); ps->ErrorMsg());
else
{
pkt_srcs.append(ps);
io_sources.Register(ps);
}
ps->AddSecondaryTablePrograms();
} }
} }
for ( int i = 0; i < flowfiles.length(); ++i ) else if ( interfaces.length() > 0 )
{
FlowFileSrc* fs = new FlowFileSrc(flowfiles[i]);
if ( ! fs->IsOpen() )
reporter->FatalError("%s: problem with netflow file %s - %s\n",
prog, flowfiles[i], fs->ErrorMsg());
else
{
io_sources.Register(fs);
}
}
}
else if ((interfaces.length() > 0 || netflows.length() > 0))
{ {
reading_live = 1; reading_live = 1;
reading_traces = 0; reading_traces = 0;
for ( int i = 0; i < interfaces.length(); ++i ) for ( int i = 0; i < interfaces.length(); ++i )
{ {
PktSrc* ps; iosource::PktSrc* ps = iosource_mgr->OpenPktSrc(interfaces[i], true);
ps = new PktInterfaceSrc(interfaces[i], filter); assert(ps);
if ( ! ps->IsOpen() ) if ( ! ps->IsOpen() )
reporter->FatalError("%s: problem with interface %s - %s\n", reporter->FatalError("problem with interface %s (%s)",
prog, interfaces[i], ps->ErrorMsg()); interfaces[i],
else
{
pkt_srcs.append(ps);
io_sources.Register(ps);
}
if ( secondary_filter )
{
PktSrc* ps;
ps = new PktInterfaceSrc(interfaces[i],
filter, TYPE_FILTER_SECONDARY);
if ( ! ps->IsOpen() )
reporter->Error("%s: problem with interface %s - %s\n",
prog, interfaces[i],
ps->ErrorMsg()); ps->ErrorMsg());
else
{
pkt_srcs.append(ps);
io_sources.Register(ps);
} }
ps->AddSecondaryTablePrograms();
}
}
for ( int i = 0; i < netflows.length(); ++i )
{
FlowSocketSrc* fs = new FlowSocketSrc(netflows[i]);
if ( ! fs->IsOpen() )
{
reporter->Error("%s: problem with netflow socket %s - %s\n",
prog, netflows[i], fs->ErrorMsg());
delete fs;
}
else
io_sources.Register(fs);
}
} }
else else
@ -267,12 +198,12 @@ void net_init(name_list& interfaces, name_list& readfiles,
if ( writefile ) if ( writefile )
{ {
// ### This will fail horribly if there are multiple pkt_dumper = iosource_mgr->OpenPktDumper(writefile, false);
// interfaces with different-lengthed media. assert(pkt_dumper);
pkt_dumper = new PktDumper(writefile);
if ( pkt_dumper->IsError() ) if ( ! pkt_dumper->IsOpen() )
reporter->FatalError("%s: can't open write file \"%s\" - %s\n", reporter->FatalError("problem opening dump file %s (%s)",
prog, writefile, pkt_dumper->ErrorMsg()); writefile, pkt_dumper->ErrorMsg());
ID* id = global_scope()->Lookup("trace_output_file"); ID* id = global_scope()->Lookup("trace_output_file");
if ( ! id ) if ( ! id )
@ -293,7 +224,7 @@ void net_init(name_list& interfaces, name_list& readfiles,
} }
} }
void expire_timers(PktSrc* src_ps) void expire_timers(iosource::PktSrc* src_ps)
{ {
SegmentProfiler(segment_logger, "expiring-timers"); SegmentProfiler(segment_logger, "expiring-timers");
TimerMgr* tmgr = TimerMgr* tmgr =
@ -307,7 +238,7 @@ void expire_timers(PktSrc* src_ps)
void net_packet_dispatch(double t, const struct pcap_pkthdr* hdr, void net_packet_dispatch(double t, const struct pcap_pkthdr* hdr,
const u_char* pkt, int hdr_size, const u_char* pkt, int hdr_size,
PktSrc* src_ps) iosource::PktSrc* src_ps)
{ {
if ( ! bro_start_network_time ) if ( ! bro_start_network_time )
bro_start_network_time = t; bro_start_network_time = t;
@ -315,7 +246,7 @@ void net_packet_dispatch(double t, const struct pcap_pkthdr* hdr,
TimerMgr* tmgr = sessions->LookupTimerMgr(src_ps->GetCurrentTag()); TimerMgr* tmgr = sessions->LookupTimerMgr(src_ps->GetCurrentTag());
// network_time never goes back. // network_time never goes back.
network_time = tmgr->Time() < t ? t : tmgr->Time(); net_update_time(tmgr->Time() < t ? t : tmgr->Time());
current_pktsrc = src_ps; current_pktsrc = src_ps;
current_iosrc = src_ps; current_iosrc = src_ps;
@ -363,11 +294,11 @@ void net_run()
{ {
set_processing_status("RUNNING", "net_run"); set_processing_status("RUNNING", "net_run");
while ( io_sources.Size() || while ( iosource_mgr->Size() ||
(BifConst::exit_only_after_terminate && ! terminating) ) (BifConst::exit_only_after_terminate && ! terminating) )
{ {
double ts; double ts;
IOSource* src = io_sources.FindSoonest(&ts); iosource::IOSource* src = iosource_mgr->FindSoonest(&ts);
#ifdef DEBUG #ifdef DEBUG
static int loop_counter = 0; static int loop_counter = 0;
@ -395,7 +326,7 @@ void net_run()
{ {
// Take advantage of the lull to get up to // Take advantage of the lull to get up to
// date on timers and events. // date on timers and events.
network_time = ct; net_update_time(ct);
expire_timers(); expire_timers();
usleep(1); // Just yield. usleep(1); // Just yield.
} }
@ -408,7 +339,7 @@ void net_run()
// date on timers and events. Because we only // date on timers and events. Because we only
// have timers as sources, going to sleep here // have timers as sources, going to sleep here
// doesn't risk blocking on other inputs. // doesn't risk blocking on other inputs.
network_time = current_time(); net_update_time(current_time());
expire_timers(); expire_timers();
// Avoid busy-waiting - pause for 100 ms. // Avoid busy-waiting - pause for 100 ms.
@ -465,16 +396,19 @@ void net_run()
void net_get_final_stats() void net_get_final_stats()
{ {
loop_over_list(pkt_srcs, i) const iosource::Manager::PktSrcList& pkt_srcs(iosource_mgr->GetPktSrcs());
for ( iosource::Manager::PktSrcList::const_iterator i = pkt_srcs.begin();
i != pkt_srcs.end(); i++ )
{ {
PktSrc* ps = pkt_srcs[i]; iosource::PktSrc* ps = *i;
if ( ps->IsLive() ) if ( ps->IsLive() )
{ {
struct PktSrc::Stats s; iosource::PktSrc::Stats s;
ps->Statistics(&s); ps->Statistics(&s);
reporter->Info("%d packets received on interface %s, %d dropped\n", reporter->Info("%d packets received on interface %s, %d dropped",
s.received, ps->Interface(), s.dropped); s.received, ps->Path().c_str(), s.dropped);
} }
} }
} }
@ -494,8 +428,6 @@ void net_finish(int drain_events)
sessions->Done(); sessions->Done();
} }
delete pkt_dumper;
#ifdef DEBUG #ifdef DEBUG
extern int reassem_seen_bytes, reassem_copied_bytes; extern int reassem_seen_bytes, reassem_copied_bytes;
// DEBUG_MSG("Reassembly (TCP and IP/Frag): %d bytes seen, %d bytes copied\n", // DEBUG_MSG("Reassembly (TCP and IP/Frag): %d bytes seen, %d bytes copied\n",
@ -516,29 +448,6 @@ void net_delete()
delete ip_anonymizer[i]; delete ip_anonymizer[i];
} }
// net_packet_match
//
// Description:
// - Checks if a packet matches a filter. It just wraps up a call to
// [pcap.h's] bpf_filter().
//
// Inputs:
// - fp: a BPF-compiled filter
// - pkt: a pointer to the packet
// - len: the original packet length
// - caplen: the captured packet length. This is pkt length
//
// Output:
// - return: 1 if the packet matches the filter, 0 otherwise
int net_packet_match(BPF_Program* fp, const u_char* pkt,
u_int len, u_int caplen)
{
// NOTE: I don't like too much un-const'ing the pkt variable.
return bpf_filter(fp->GetProgram()->bf_insns, (u_char*) pkt, len, caplen);
}
int _processing_suspended = 0; int _processing_suspended = 0;
static double suspend_start = 0; static double suspend_start = 0;
@ -556,8 +465,12 @@ void net_continue_processing()
if ( _processing_suspended == 1 ) if ( _processing_suspended == 1 )
{ {
reporter->Info("processing continued"); reporter->Info("processing continued");
loop_over_list(pkt_srcs, i)
pkt_srcs[i]->ContinueAfterSuspend(); const iosource::Manager::PktSrcList& pkt_srcs(iosource_mgr->GetPktSrcs());
for ( iosource::Manager::PktSrcList::const_iterator i = pkt_srcs.begin();
i != pkt_srcs.end(); i++ )
(*i)->ContinueAfterSuspend();
} }
--_processing_suspended; --_processing_suspended;

26
src/Net.h
View file

@ -5,27 +5,24 @@
#include "net_util.h" #include "net_util.h"
#include "util.h" #include "util.h"
#include "BPF_Program.h"
#include "List.h" #include "List.h"
#include "PktSrc.h"
#include "FlowSrc.h"
#include "Func.h" #include "Func.h"
#include "RemoteSerializer.h" #include "RemoteSerializer.h"
#include "iosource/IOSource.h"
#include "iosource/PktSrc.h"
#include "iosource/PktDumper.h"
extern void net_init(name_list& interfaces, name_list& readfiles, extern void net_init(name_list& interfaces, name_list& readfiles,
name_list& netflows, name_list& flowfiles, const char* writefile, int do_watchdog);
const char* writefile, const char* filter,
const char* secondary_filter, int do_watchdog);
extern void net_run(); extern void net_run();
extern void net_get_final_stats(); extern void net_get_final_stats();
extern void net_finish(int drain_events); extern void net_finish(int drain_events);
extern void net_delete(); // Reclaim all memory, etc. extern void net_delete(); // Reclaim all memory, etc.
extern void net_update_time(double new_network_time);
extern void net_packet_dispatch(double t, const struct pcap_pkthdr* hdr, extern void net_packet_dispatch(double t, const struct pcap_pkthdr* hdr,
const u_char* pkt, int hdr_size, const u_char* pkt, int hdr_size,
PktSrc* src_ps); iosource::PktSrc* src_ps);
extern int net_packet_match(BPF_Program* fp, const u_char* pkt, extern void expire_timers(iosource::PktSrc* src_ps = 0);
u_int len, u_int caplen);
extern void expire_timers(PktSrc* src_ps = 0);
extern void termination_signal(); extern void termination_signal();
// Functions to temporarily suspend processing of live input (network packets // Functions to temporarily suspend processing of live input (network packets
@ -82,13 +79,10 @@ extern const u_char* current_pkt;
extern int current_dispatched; extern int current_dispatched;
extern int current_hdr_size; extern int current_hdr_size;
extern double current_timestamp; extern double current_timestamp;
extern PktSrc* current_pktsrc; extern iosource::PktSrc* current_pktsrc;
extern IOSource* current_iosrc; extern iosource::IOSource* current_iosrc;
declare(PList,PktSrc); extern iosource::PktDumper* pkt_dumper; // where to save packets
extern PList(PktSrc) pkt_srcs;
extern PktDumper* pkt_dumper; // where to save packets
extern char* writefile; extern char* writefile;

4
src/NetVar.cc
View file

@ -245,8 +245,6 @@ bro_uint_t bits_per_uid;
#include "const.bif.netvar_def" #include "const.bif.netvar_def"
#include "types.bif.netvar_def" #include "types.bif.netvar_def"
#include "event.bif.netvar_def" #include "event.bif.netvar_def"
#include "logging.bif.netvar_def"
#include "input.bif.netvar_def"
#include "reporter.bif.netvar_def" #include "reporter.bif.netvar_def"
void init_event_handlers() void init_event_handlers()
@ -311,8 +309,6 @@ void init_net_var()
{ {
#include "const.bif.netvar_init" #include "const.bif.netvar_init"
#include "types.bif.netvar_init" #include "types.bif.netvar_init"
#include "logging.bif.netvar_init"
#include "input.bif.netvar_init"
#include "reporter.bif.netvar_init" #include "reporter.bif.netvar_init"
conn_id = internal_type("conn_id")->AsRecordType(); conn_id = internal_type("conn_id")->AsRecordType();

2
src/NetVar.h
View file

@ -255,8 +255,6 @@ extern void init_net_var();
#include "const.bif.netvar_h" #include "const.bif.netvar_h"
#include "types.bif.netvar_h" #include "types.bif.netvar_h"
#include "event.bif.netvar_h" #include "event.bif.netvar_h"
#include "logging.bif.netvar_h"
#include "input.bif.netvar_h"
#include "reporter.bif.netvar_h" #include "reporter.bif.netvar_h"
#endif #endif

4
src/Obj.cc
View file

@ -7,6 +7,7 @@
#include "Obj.h" #include "Obj.h"
#include "Serializer.h" #include "Serializer.h"
#include "File.h" #include "File.h"
#include "plugin/Manager.h"
Location no_location("<no location>", 0, 0, 0, 0); Location no_location("<no location>", 0, 0, 0, 0);
Location start_location("<start uninitialized>", 0, 0, 0, 0); Location start_location("<start uninitialized>", 0, 0, 0, 0);
@ -92,6 +93,9 @@ int BroObj::suppress_errors = 0;
BroObj::~BroObj() BroObj::~BroObj()
{ {
if ( notify_plugins )
PLUGIN_HOOK_VOID(HOOK_BRO_OBJ_DTOR, HookBroObjDtor(this));
delete location; delete location;
} }

5
src/Obj.h
View file

@ -92,6 +92,7 @@ public:
{ {
ref_cnt = 1; ref_cnt = 1;
in_ser_cache = false; in_ser_cache = false;
notify_plugins = false;
// A bit of a hack. We'd like to associate location // A bit of a hack. We'd like to associate location
// information with every object created when parsing, // information with every object created when parsing,
@ -151,6 +152,9 @@ public:
// extend compound objects such as statement lists. // extend compound objects such as statement lists.
virtual void UpdateLocationEndInfo(const Location& end); virtual void UpdateLocationEndInfo(const Location& end);
// Enable notification of plugins when this objects gets destroyed.
void NotifyPluginsOnDtor() { notify_plugins = true; }
int RefCnt() const { return ref_cnt; } int RefCnt() const { return ref_cnt; }
// Helper class to temporarily suppress errors // Helper class to temporarily suppress errors
@ -181,6 +185,7 @@ private:
friend inline void Ref(BroObj* o); friend inline void Ref(BroObj* o);
friend inline void Unref(BroObj* o); friend inline void Unref(BroObj* o);
bool notify_plugins;
int ref_cnt; int ref_cnt;
// If non-zero, do not print runtime errors. Useful for // If non-zero, do not print runtime errors. Useful for

83
src/Pipe.cc Normal file
View file

@ -0,0 +1,83 @@
// See the file "COPYING" in the main distribution directory for copyright.
#include "Pipe.h"
#include "Reporter.h"
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>
#include <cstdio>
using namespace bro;
static void pipe_fail(int eno)
{
char tmp[256];
strerror_r(eno, tmp, sizeof(tmp));
reporter->FatalError("Pipe failure: %s", tmp);
}
static void set_flags(int fd, int flags)
{
if ( flags )
fcntl(fd, F_SETFD, fcntl(fd, F_GETFD) | flags);
}
static void set_status_flags(int fd, int flags)
{
if ( flags )
fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | flags);
}
static int dup_or_fail(int fd, int flags)
{
int rval = dup(fd);
if ( rval < 0 )
pipe_fail(errno);
set_flags(fd, flags);
return rval;
}
Pipe::Pipe(int flags0, int flags1, int status_flags0, int status_flags1)
{
// pipe2 can set flags atomically, but not yet available everywhere.
if ( ::pipe(fds) )
pipe_fail(errno);
flags[0] = flags0;
flags[1] = flags1;
set_flags(fds[0], flags[0]);
set_flags(fds[1], flags[1]);
set_status_flags(fds[0], status_flags0);
set_status_flags(fds[1], status_flags1);
}
Pipe::~Pipe()
{
close(fds[0]);
close(fds[1]);
}
Pipe::Pipe(const Pipe& other)
{
fds[0] = dup_or_fail(other.fds[0], other.flags[0]);
fds[1] = dup_or_fail(other.fds[1], other.flags[1]);
flags[0] = other.flags[0];
flags[1] = other.flags[1];
}
Pipe& Pipe::operator=(const Pipe& other)
{
if ( this == &other )
return *this;
close(fds[0]);
close(fds[1]);
fds[0] = dup_or_fail(other.fds[0], other.flags[0]);
fds[1] = dup_or_fail(other.fds[1], other.flags[1]);
flags[0] = other.flags[0];
flags[1] = other.flags[1];
return *this;
}

56
src/Pipe.h Normal file
View file

@ -0,0 +1,56 @@
// See the file "COPYING" in the main distribution directory for copyright.
#ifndef BRO_PIPE_H
#define BRO_PIPE_H
namespace bro {
class Pipe {
public:
/**
* Create a pair of file descriptors via pipe(), or aborts if it cannot.
* @param flags0 file descriptor flags to set on read end of pipe.
* @param flags1 file descriptor flags to set on write end of pipe.
* @param status_flags0 descriptor status flags to set on read end of pipe.
* @param status_flags1 descriptor status flags to set on write end of pipe.
*/
Pipe(int flags0 = 0, int flags1 = 0, int status_flags0 = 0,
int status_flags1 = 0);
/**
* Close the pair of file descriptors owned by the object.
*/
~Pipe();
/**
* Make a copy of another Pipe object (file descriptors are dup'd).
*/
Pipe(const Pipe& other);
/**
* Assign a Pipe object by closing file descriptors and duping those of
* the other.
*/
Pipe& operator=(const Pipe& other);
/**
* @return the file descriptor associated with the read-end of the pipe.
*/
int ReadFD() const
{ return fds[0]; }
/**
* @return the file descriptor associated with the write-end of the pipe.
*/
int WriteFD() const
{ return fds[1]; }
private:
int fds[2];
int flags[2];
};
} // namespace bro
#endif // BRO_PIPE_H

804
src/PktSrc.cc
View file

@ -1,804 +0,0 @@
// See the file "COPYING" in the main distribution directory for copyright.
#include <errno.h>
#include <sys/stat.h>
#include "config.h"
#include "util.h"
#include "PktSrc.h"
#include "Hash.h"
#include "Net.h"
#include "Sessions.h"
// ### This needs auto-confing.
#ifdef HAVE_PCAP_INT_H
#include <pcap-int.h>
#endif
PktSrc::PktSrc()
{
interface = readfile = 0;
data = last_data = 0;
memset(&hdr, 0, sizeof(hdr));
hdr_size = 0;
datalink = 0;
netmask = 0xffffff00;
pd = 0;
idle = false;
next_sync_point = 0;
first_timestamp = current_timestamp = next_timestamp = 0.0;
first_wallclock = current_wallclock = 0;
stats.received = stats.dropped = stats.link = 0;
}
PktSrc::~PktSrc()
{
Close();
loop_over_list(program_list, i)
delete program_list[i];
BPF_Program* code;
IterCookie* cookie = filters.InitForIteration();
while ( (code = filters.NextEntry(cookie)) )
delete code;
delete [] interface;
delete [] readfile;
}
void PktSrc::GetFds(int* read, int* write, int* except)
{
if ( pseudo_realtime )
{
// Select would give erroneous results. But we simulate it
// by setting idle accordingly.
idle = CheckPseudoTime() == 0;
return;
}
if ( selectable_fd >= 0 )
*read = selectable_fd;
}
int PktSrc::ExtractNextPacket()
{
// Don't return any packets if processing is suspended (except for the
// very first packet which we need to set up times).
if ( net_is_processing_suspended() && first_timestamp )
{
idle = true;
return 0;
}
data = last_data = pcap_next(pd, &hdr);
if ( data && (hdr.len == 0 || hdr.caplen == 0) )
{
sessions->Weird("empty_pcap_header", &hdr, data);
return 0;
}
if ( data )
next_timestamp = hdr.ts.tv_sec + double(hdr.ts.tv_usec) / 1e6;
if ( pseudo_realtime )
current_wallclock = current_time(true);
if ( ! first_timestamp )
first_timestamp = next_timestamp;
idle = (data == 0);
if ( data )
++stats.received;
// Source has gone dry. If it's a network interface, this just means
// it's timed out. If it's a file, though, then the file has been
// exhausted.
if ( ! data && ! IsLive() )
{
closed = true;
if ( pseudo_realtime && using_communication )
{
if ( remote_trace_sync_interval )
remote_serializer->SendFinalSyncPoint();
else
remote_serializer->Terminate();
}
}
return data != 0;
}
double PktSrc::NextTimestamp(double* local_network_time)
{
if ( ! data && ! ExtractNextPacket() )
return -1.0;
if ( pseudo_realtime )
{
// Delay packet if necessary.
double packet_time = CheckPseudoTime();
if ( packet_time )
return packet_time;
idle = true;
return -1.0;
}
return next_timestamp;
}
void PktSrc::ContinueAfterSuspend()
{
current_wallclock = current_time(true);
}
double PktSrc::CurrentPacketWallClock()
{
// We stop time when we are suspended.
if ( net_is_processing_suspended() )
current_wallclock = current_time(true);
return current_wallclock;
}
double PktSrc::CheckPseudoTime()
{
if ( ! data && ! ExtractNextPacket() )
return 0;
if ( ! current_timestamp )
return bro_start_time;
if ( remote_trace_sync_interval )
{
if ( next_sync_point == 0 || next_timestamp >= next_sync_point )
{
int n = remote_serializer->SendSyncPoint();
next_sync_point = first_timestamp +
n * remote_trace_sync_interval;
remote_serializer->Log(RemoteSerializer::LogInfo,
fmt("stopping at packet %.6f, next sync-point at %.6f",
current_timestamp, next_sync_point));
return 0;
}
}
double pseudo_time = next_timestamp - first_timestamp;
double ct = (current_time(true) - first_wallclock) * pseudo_realtime;
return pseudo_time <= ct ? bro_start_time + pseudo_time : 0;
}
void PktSrc::Process()
{
if ( ! data && ! ExtractNextPacket() )
return;
current_timestamp = next_timestamp;
int pkt_hdr_size = hdr_size;
// Unfortunately some packets on the link might have MPLS labels
// while others don't. That means we need to ask the link-layer if
// labels are in place.
bool have_mpls = false;
int protocol = 0;
switch ( datalink ) {
case DLT_NULL:
{
protocol = (data[3] << 24) + (data[2] << 16) + (data[1] << 8) + data[0];
// From the Wireshark Wiki: "AF_INET6, unfortunately, has
// different values in {NetBSD,OpenBSD,BSD/OS},
// {FreeBSD,DragonFlyBSD}, and {Darwin/Mac OS X}, so an IPv6
// packet might have a link-layer header with 24, 28, or 30
// as the AF_ value." As we may be reading traces captured on
// platforms other than what we're running on, we accept them
// all here.
if ( protocol != AF_INET
&& protocol != AF_INET6
&& protocol != 24
&& protocol != 28
&& protocol != 30 )
{
sessions->Weird("non_ip_packet_in_null_transport", &hdr, data);
data = 0;
return;
}
break;
}
case DLT_EN10MB:
{
// Get protocol being carried from the ethernet frame.
protocol = (data[12] << 8) + data[13];
switch ( protocol )
{
// MPLS carried over the ethernet frame.
case 0x8847:
// Remove the data link layer and denote a
// header size of zero before the IP header.
have_mpls = true;
data += get_link_header_size(datalink);
pkt_hdr_size = 0;
break;
// VLAN carried over the ethernet frame.
case 0x8100:
data += get_link_header_size(datalink);
// Check for MPLS in VLAN.
if ( ((data[2] << 8) + data[3]) == 0x8847 )
have_mpls = true;
data += 4; // Skip the vlan header
pkt_hdr_size = 0;
// Check for 802.1ah (Q-in-Q) containing IP.
// Only do a second layer of vlan tag
// stripping because there is no
// specification that allows for deeper
// nesting.
if ( ((data[2] << 8) + data[3]) == 0x0800 )
data += 4;
break;
// PPPoE carried over the ethernet frame.
case 0x8864:
data += get_link_header_size(datalink);
protocol = (data[6] << 8) + data[7];
data += 8; // Skip the PPPoE session and PPP header
pkt_hdr_size = 0;
if ( protocol != 0x0021 && protocol != 0x0057 )
{
// Neither IPv4 nor IPv6.
sessions->Weird("non_ip_packet_in_pppoe_encapsulation", &hdr, data);
data = 0;
return;
}
break;
}
break;
}
case DLT_PPP_SERIAL:
{
// Get PPP protocol.
protocol = (data[2] << 8) + data[3];
if ( protocol == 0x0281 )
{
// MPLS Unicast. Remove the data link layer and
// denote a header size of zero before the IP header.
have_mpls = true;
data += get_link_header_size(datalink);
pkt_hdr_size = 0;
}
else if ( protocol != 0x0021 && protocol != 0x0057 )
{
// Neither IPv4 nor IPv6.
sessions->Weird("non_ip_packet_in_ppp_encapsulation", &hdr, data);
data = 0;
return;
}
break;
}
}
if ( have_mpls )
{
// Skip the MPLS label stack.
bool end_of_stack = false;
while ( ! end_of_stack )
{
end_of_stack = *(data + 2) & 0x01;
data += 4;
}
}
if ( pseudo_realtime )
{
current_pseudo = CheckPseudoTime();
net_packet_dispatch(current_pseudo, &hdr, data, pkt_hdr_size, this);
if ( ! first_wallclock )
first_wallclock = current_time(true);
}
else
net_packet_dispatch(current_timestamp, &hdr, data, pkt_hdr_size, this);
data = 0;
}
bool PktSrc::GetCurrentPacket(const struct pcap_pkthdr** arg_hdr,
const u_char** arg_pkt)
{
if ( ! last_data )
return false;
*arg_hdr = &hdr;
*arg_pkt = last_data;
return true;
}
int PktSrc::PrecompileFilter(int index, const char* filter)
{
// Compile filter.
BPF_Program* code = new BPF_Program();
if ( ! code->Compile(pd, filter, netmask, errbuf, sizeof(errbuf)) )
{
delete code;
return 0;
}
// Store it in hash.
HashKey* hash = new HashKey(HashKey(bro_int_t(index)));
BPF_Program* oldcode = filters.Lookup(hash);
if ( oldcode )
delete oldcode;
filters.Insert(hash, code);
delete hash;
return 1;
}
int PktSrc::SetFilter(int index)
{
// We don't want load-level filters for the secondary path.
if ( filter_type == TYPE_FILTER_SECONDARY && index > 0 )
return 1;
HashKey* hash = new HashKey(HashKey(bro_int_t(index)));
BPF_Program* code = filters.Lookup(hash);
delete hash;
if ( ! code )
{
safe_snprintf(errbuf, sizeof(errbuf),
"No precompiled pcap filter for index %d",
index);
return 0;
}
if ( pcap_setfilter(pd, code->GetProgram()) < 0 )
{
safe_snprintf(errbuf, sizeof(errbuf),
"pcap_setfilter(%d): %s",
index, pcap_geterr(pd));
return 0;
}
#ifndef HAVE_LINUX
// Linux doesn't clear counters when resetting filter.
stats.received = stats.dropped = stats.link = 0;
#endif
return 1;
}
void PktSrc::SetHdrSize()
{
int dl = pcap_datalink(pd);
hdr_size = get_link_header_size(dl);
if ( hdr_size < 0 )
{
safe_snprintf(errbuf, sizeof(errbuf),
"unknown data link type 0x%x", dl);
Close();
}
datalink = dl;
}
void PktSrc::Close()
{
if ( pd )
{
pcap_close(pd);
pd = 0;
closed = true;
}
}
void PktSrc::AddSecondaryTablePrograms()
{
BPF_Program* program;
loop_over_list(secondary_path->EventTable(), i)
{
SecondaryEvent* se = secondary_path->EventTable()[i];
program = new BPF_Program();
if ( ! program->Compile(snaplen, datalink, se->Filter(),
netmask, errbuf, sizeof(errbuf)) )
{
delete program;
Close();
return;
}
SecondaryProgram* sp = new SecondaryProgram(program, se);
program_list.append(sp);
}
}
void PktSrc::Statistics(Stats* s)
{
if ( reading_traces )
s->received = s->dropped = s->link = 0;
else
{
struct pcap_stat pstat;
if ( pcap_stats(pd, &pstat) < 0 )
{
reporter->Error("problem getting packet filter statistics: %s",
ErrorMsg());
s->received = s->dropped = s->link = 0;
}
else
{
s->dropped = pstat.ps_drop;
s->link = pstat.ps_recv;
}
}
s->received = stats.received;
if ( pseudo_realtime )
s->dropped = 0;
stats.dropped = s->dropped;
}
PktInterfaceSrc::PktInterfaceSrc(const char* arg_interface, const char* filter,
PktSrc_Filter_Type ft)
: PktSrc()
{
char tmp_errbuf[PCAP_ERRBUF_SIZE];
filter_type = ft;
// Determine interface if not specified.
if ( ! arg_interface && ! (arg_interface = pcap_lookupdev(tmp_errbuf)) )
{
safe_snprintf(errbuf, sizeof(errbuf),
"pcap_lookupdev: %s", tmp_errbuf);
return;
}
interface = copy_string(arg_interface);
// Determine network and netmask.
uint32 net;
if ( pcap_lookupnet(interface, &net, &netmask, tmp_errbuf) < 0 )
{
// ### The lookup can fail if no address is assigned to
// the interface; and libpcap doesn't have any useful notion
// of error codes, just error strings - how bogus - so we
// just kludge around the error :-(.
// sprintf(errbuf, "pcap_lookupnet %s", tmp_errbuf);
// return;
net = 0;
netmask = 0xffffff00;
}
// We use the smallest time-out possible to return almost immediately if
// no packets are available. (We can't use set_nonblocking() as it's
// broken on FreeBSD: even when select() indicates that we can read
// something, we may get nothing if the store buffer hasn't filled up
// yet.)
pd = pcap_open_live(interface, snaplen, 1, 1, tmp_errbuf);
if ( ! pd )
{
safe_snprintf(errbuf, sizeof(errbuf),
"pcap_open_live: %s", tmp_errbuf);
closed = true;
return;
}
// ### This needs autoconf'ing.
#ifdef HAVE_PCAP_INT_H
reporter->Info("pcap bufsize = %d\n", ((struct pcap *) pd)->bufsize);
#endif
#ifdef HAVE_LINUX
if ( pcap_setnonblock(pd, 1, tmp_errbuf) < 0 )
{
safe_snprintf(errbuf, sizeof(errbuf),
"pcap_setnonblock: %s", tmp_errbuf);
pcap_close(pd);
closed = true;
return;
}
#endif
selectable_fd = pcap_fileno(pd);
if ( PrecompileFilter(0, filter) && SetFilter(0) )
{
SetHdrSize();
if ( closed )
// Couldn't get header size.
return;
reporter->Info("listening on %s, capture length %d bytes\n", interface, snaplen);
}
else
closed = true;
}
PktFileSrc::PktFileSrc(const char* arg_readfile, const char* filter,
PktSrc_Filter_Type ft)
: PktSrc()
{
readfile = copy_string(arg_readfile);
filter_type = ft;
pd = pcap_open_offline((char*) readfile, errbuf);
if ( pd && PrecompileFilter(0, filter) && SetFilter(0) )
{
SetHdrSize();
if ( closed )
// Unknown link layer type.
return;
// We don't put file sources into non-blocking mode as
// otherwise we would not be able to identify the EOF.
selectable_fd = fileno(pcap_file(pd));
if ( selectable_fd < 0 )
reporter->InternalError("OS does not support selectable pcap fd");
}
else
closed = true;
}
SecondaryPath::SecondaryPath()
{
filter = 0;
// Glue together the secondary filter, if exists.
Val* secondary_fv = internal_val("secondary_filters");
if ( secondary_fv->AsTableVal()->Size() == 0 )
return;
int did_first = 0;
const TableEntryValPDict* v = secondary_fv->AsTable();
IterCookie* c = v->InitForIteration();
TableEntryVal* tv;
HashKey* h;
while ( (tv = v->NextEntry(h, c)) )
{
// Get the index values.
ListVal* index =
secondary_fv->AsTableVal()->RecoverIndex(h);
const char* str =
index->Index(0)->Ref()->AsString()->CheckString();
if ( ++did_first == 1 )
{
filter = copy_string(str);
}
else
{
if ( strlen(filter) > 0 )
{
char* tmp_f = new char[strlen(str) + strlen(filter) + 32];
if ( strlen(str) == 0 )
sprintf(tmp_f, "%s", filter);
else
sprintf(tmp_f, "(%s) or (%s)", filter, str);
delete [] filter;
filter = tmp_f;
}
}
// Build secondary_path event table item and link it.
SecondaryEvent* se =
new SecondaryEvent(index->Index(0)->Ref()->AsString()->CheckString(),
tv->Value()->AsFunc() );
event_list.append(se);
delete h;
Unref(index);
}
}
SecondaryPath::~SecondaryPath()
{
loop_over_list(event_list, i)
delete event_list[i];
delete [] filter;
}
SecondaryProgram::~SecondaryProgram()
{
delete program;
}
PktDumper::PktDumper(const char* arg_filename, bool arg_append)
{
filename[0] = '\0';
is_error = false;
append = arg_append;
dumper = 0;
open_time = 0.0;
// We need a pcap_t with a reasonable link-layer type. We try to get it
// from the packet sources. If not available, we fall back to Ethernet.
// FIXME: Perhaps we should make this configurable?
int linktype = -1;
if ( pkt_srcs.length() )
linktype = pkt_srcs[0]->LinkType();
if ( linktype < 0 )
linktype = DLT_EN10MB;
pd = pcap_open_dead(linktype, snaplen);
if ( ! pd )
{
Error("error for pcap_open_dead");
return;
}
if ( arg_filename )
Open(arg_filename);
}
bool PktDumper::Open(const char* arg_filename)
{
if ( ! arg_filename && ! *filename )
{
Error("no filename given");
return false;
}
if ( arg_filename )
{
if ( dumper && streq(arg_filename, filename) )
// Already open.
return true;
safe_strncpy(filename, arg_filename, FNBUF_LEN);
}
if ( dumper )
Close();
struct stat s;
int exists = 0;
if ( append )
{
// See if output file already exists (and is non-empty).
exists = stat(filename, &s); ;
if ( exists < 0 && errno != ENOENT )
{
Error(fmt("can't stat file %s: %s", filename, strerror(errno)));
return false;
}
}
if ( ! append || exists < 0 || s.st_size == 0 )
{
// Open new file.
dumper = pcap_dump_open(pd, filename);
if ( ! dumper )
{
Error(pcap_geterr(pd));
return false;
}
}
else
{
// Old file and we need to append, which, unfortunately,
// is not supported by libpcap. So, we have to hack a
// little bit, knowing that pcap_dumpter_t is, in fact,
// a FILE ... :-(
dumper = (pcap_dumper_t*) fopen(filename, "a");
if ( ! dumper )
{
Error(fmt("can't open dump %s: %s", filename, strerror(errno)));
return false;
}
}
open_time = network_time;
is_error = false;
return true;
}
bool PktDumper::Close()
{
if ( dumper )
{
pcap_dump_close(dumper);
dumper = 0;
is_error = false;
}
return true;
}
bool PktDumper::Dump(const struct pcap_pkthdr* hdr, const u_char* pkt)
{
if ( ! dumper )
return false;
if ( ! open_time )
open_time = network_time;
pcap_dump((u_char*) dumper, hdr, pkt);
return true;
}
void PktDumper::Error(const char* errstr)
{
safe_strncpy(errbuf, errstr, sizeof(errbuf));
is_error = true;
}
int get_link_header_size(int dl)
{
switch ( dl ) {
case DLT_NULL:
return 4;
case DLT_EN10MB:
return 14;
case DLT_FDDI:
return 13 + 8; // fddi_header + LLC
#ifdef DLT_LINUX_SLL
case DLT_LINUX_SLL:
return 16;
#endif
case DLT_PPP_SERIAL: // PPP_SERIAL
return 4;
case DLT_RAW:
return 0;
}
return -1;
}

258
src/PktSrc.h
View file

@ -1,258 +0,0 @@
// See the file "COPYING" in the main distribution directory for copyright.
#ifndef pktsrc_h
#define pktsrc_h
#include "Dict.h"
#include "Expr.h"
#include "BPF_Program.h"
#include "IOSource.h"
#include "RemoteSerializer.h"
#define BRO_PCAP_ERRBUF_SIZE PCAP_ERRBUF_SIZE + 256
extern "C" {
#include <pcap.h>
}
declare(PDict,BPF_Program);
// Whether a PktSrc object is used by the normal filter structure or the
// secondary-path structure.
typedef enum {
TYPE_FILTER_NORMAL, // the normal filter
TYPE_FILTER_SECONDARY, // the secondary-path filter
} PktSrc_Filter_Type;
// {filter,event} tuples conforming the secondary path.
class SecondaryEvent {
public:
SecondaryEvent(const char* arg_filter, Func* arg_event)
{
filter = arg_filter;
event = arg_event;
}
const char* Filter() { return filter; }
Func* Event() { return event; }
private:
const char* filter;
Func* event;
};
declare(PList,SecondaryEvent);
typedef PList(SecondaryEvent) secondary_event_list;
class SecondaryPath {
public:
SecondaryPath();
~SecondaryPath();
secondary_event_list& EventTable() { return event_list; }
const char* Filter() { return filter; }
private:
secondary_event_list event_list;
// OR'ed union of all SecondaryEvent filters
char* filter;
};
// Main secondary-path object.
extern SecondaryPath* secondary_path;
// {program, {filter,event}} tuple table.
class SecondaryProgram {
public:
SecondaryProgram(BPF_Program* arg_program, SecondaryEvent* arg_event)
{
program = arg_program;
event = arg_event;
}
~SecondaryProgram();
BPF_Program* Program() { return program; }
SecondaryEvent* Event() { return event; }
private:
// Associated program.
BPF_Program *program;
// Event that is run in case the program is matched.
SecondaryEvent* event;
};
declare(PList,SecondaryProgram);
typedef PList(SecondaryProgram) secondary_program_list;
class PktSrc : public IOSource {
public:
~PktSrc();
// IOSource interface
bool IsReady();
void GetFds(int* read, int* write, int* except);
double NextTimestamp(double* local_network_time);
void Process();
const char* Tag() { return "PktSrc"; }
const char* ErrorMsg() const { return errbuf; }
void ClearErrorMsg() { *errbuf ='\0'; }
// Returns the packet last processed; false if there is no
// current packet available.
bool GetCurrentPacket(const pcap_pkthdr** hdr, const u_char** pkt);
int HdrSize() const { return hdr_size; }
int DataLink() const { return datalink; }
void ConsumePacket() { data = 0; }
int IsLive() const { return interface != 0; }
pcap_t* PcapHandle() const { return pd; }
int LinkType() const { return pcap_datalink(pd); }
const char* ReadFile() const { return readfile; }
const char* Interface() const { return interface; }
PktSrc_Filter_Type FilterType() const { return filter_type; }
void AddSecondaryTablePrograms();
const secondary_program_list& ProgramTable() const
{ return program_list; }
// Signal packet source that processing was suspended and is now going
// to be continued.
void ContinueAfterSuspend();
// Only valid in pseudo-realtime mode.
double CurrentPacketTimestamp() { return current_pseudo; }
double CurrentPacketWallClock();
struct Stats {
unsigned int received; // pkts received (w/o drops)
unsigned int dropped; // pkts dropped
unsigned int link; // total packets on link
// (not always not available)
};
virtual void Statistics(Stats* stats);
// Precompiles a filter and associates the given index with it.
// Returns true on success, 0 if a problem occurred.
virtual int PrecompileFilter(int index, const char* filter);
// Activates the filter with the given index.
// Returns true on success, 0 if a problem occurred.
virtual int SetFilter(int index);
protected:
PktSrc();
static const int PCAP_TIMEOUT = 20;
void SetHdrSize();
virtual void Close();
// Returns 1 on success, 0 on time-out/gone dry.
virtual int ExtractNextPacket();
// Checks if the current packet has a pseudo-time <= current_time.
// If yes, returns pseudo-time, otherwise 0.
double CheckPseudoTime();
double current_timestamp;
double next_timestamp;
// Only set in pseudo-realtime mode.
double first_timestamp;
double first_wallclock;
double current_wallclock;
double current_pseudo;
struct pcap_pkthdr hdr;
const u_char* data; // contents of current packet
const u_char* last_data; // same, but unaffected by consuming
int hdr_size;
int datalink;
double next_sync_point; // For trace synchronziation in pseudo-realtime
char* interface; // nil if not reading from an interface
char* readfile; // nil if not reading from a file
pcap_t* pd;
int selectable_fd;
uint32 netmask;
char errbuf[BRO_PCAP_ERRBUF_SIZE];
Stats stats;
PDict(BPF_Program) filters; // precompiled filters
PktSrc_Filter_Type filter_type; // normal path or secondary path
secondary_program_list program_list;
};
class PktInterfaceSrc : public PktSrc {
public:
PktInterfaceSrc(const char* interface, const char* filter,
PktSrc_Filter_Type ft=TYPE_FILTER_NORMAL);
};
class PktFileSrc : public PktSrc {
public:
PktFileSrc(const char* readfile, const char* filter,
PktSrc_Filter_Type ft=TYPE_FILTER_NORMAL);
};
extern int get_link_header_size(int dl);
class PktDumper {
public:
PktDumper(const char* file = 0, bool append = false);
~PktDumper() { Close(); }
bool Open(const char* file = 0);
bool Close();
bool Dump(const struct pcap_pkthdr* hdr, const u_char* pkt);
pcap_dumper_t* PcapDumper() { return dumper; }
const char* FileName() const { return filename; }
bool IsError() const { return is_error; }
const char* ErrorMsg() const { return errbuf; }
// This heuristic will horribly fail if we're using packets
// with different link layers. (If we can't derive a reasonable value
// from the packet sources, our fall-back is Ethernet.)
int HdrSize() const
{ return get_link_header_size(pcap_datalink(pd)); }
// Network time when dump file was opened.
double OpenTime() const { return open_time; }
private:
void InitPd();
void Error(const char* str);
static const int FNBUF_LEN = 1024;
char filename[FNBUF_LEN];
bool append;
pcap_dumper_t* dumper;
pcap_t* pd;
double open_time;
bool is_error;
char errbuf[BRO_PCAP_ERRBUF_SIZE];
};
#endif

96
src/RemoteSerializer.cc
View file

@ -188,10 +188,11 @@
#include "File.h" #include "File.h"
#include "Conn.h" #include "Conn.h"
#include "Reporter.h" #include "Reporter.h"
#include "threading/SerialTypes.h"
#include "logging/Manager.h"
#include "IPAddr.h" #include "IPAddr.h"
#include "bro_inet_ntop.h" #include "bro_inet_ntop.h"
#include "iosource/Manager.h"
#include "logging/Manager.h"
#include "logging/logging.bif.h"
extern "C" { extern "C" {
#include "setsignal.h" #include "setsignal.h"
@ -284,10 +285,10 @@ struct ping_args {
\ \
if ( ! c ) \ if ( ! c ) \
{ \ { \
idle = io->IsIdle();\ SetIdle(io->IsIdle());\
return true; \ return true; \
} \ } \
idle = false; \ SetIdle(false); \
} }
static const char* msgToStr(int msg) static const char* msgToStr(int msg)
@ -533,7 +534,6 @@ RemoteSerializer::RemoteSerializer()
current_sync_point = 0; current_sync_point = 0;
syncing_times = false; syncing_times = false;
io = 0; io = 0;
closed = false;
terminating = false; terminating = false;
in_sync = 0; in_sync = 0;
last_flush = 0; last_flush = 0;
@ -558,7 +558,7 @@ RemoteSerializer::~RemoteSerializer()
delete io; delete io;
} }
void RemoteSerializer::Init() void RemoteSerializer::Enable()
{ {
if ( initialized ) if ( initialized )
return; return;
@ -571,7 +571,7 @@ void RemoteSerializer::Init()
Fork(); Fork();
io_sources.Register(this); iosource_mgr->Register(this);
Log(LogInfo, fmt("communication started, parent pid is %d, child pid is %d", getpid(), child_pid)); Log(LogInfo, fmt("communication started, parent pid is %d, child pid is %d", getpid(), child_pid));
initialized = 1; initialized = 1;
@ -1275,7 +1275,7 @@ bool RemoteSerializer::Listen(const IPAddr& ip, uint16 port, bool expect_ssl,
return false; return false;
listening = true; listening = true;
closed = false; SetClosed(false);
return true; return true;
} }
@ -1344,7 +1344,7 @@ bool RemoteSerializer::StopListening()
return false; return false;
listening = false; listening = false;
closed = ! IsActive(); SetClosed(! IsActive());
return true; return true;
} }
@ -1367,12 +1367,14 @@ void RemoteSerializer::Unregister(ID* id)
} }
} }
void RemoteSerializer::GetFds(int* read, int* write, int* except) void RemoteSerializer::GetFds(iosource::FD_Set* read, iosource::FD_Set* write,
iosource::FD_Set* except)
{ {
*read = io->Fd(); read->Insert(io->Fd());
read->Insert(io->ExtraReadFDs());
if ( io->CanWrite() ) if ( io->CanWrite() )
*write = io->Fd(); write->Insert(io->Fd());
} }
double RemoteSerializer::NextTimestamp(double* local_network_time) double RemoteSerializer::NextTimestamp(double* local_network_time)
@ -1382,7 +1384,7 @@ double RemoteSerializer::NextTimestamp(double* local_network_time)
if ( received_logs > 0 ) if ( received_logs > 0 )
{ {
// If we processed logs last time, assume there's more. // If we processed logs last time, assume there's more.
idle = false; SetIdle(false);
received_logs = 0; received_logs = 0;
return timer_mgr->Time(); return timer_mgr->Time();
} }
@ -1397,7 +1399,7 @@ double RemoteSerializer::NextTimestamp(double* local_network_time)
pt = timer_mgr->Time(); pt = timer_mgr->Time();
if ( packets.length() ) if ( packets.length() )
idle = false; SetIdle(false);
if ( et >= 0 && (et < pt || pt < 0) ) if ( et >= 0 && (et < pt || pt < 0) )
return et; return et;
@ -1452,7 +1454,7 @@ void RemoteSerializer::Process()
// FIXME: The following chunk of code is copied from // FIXME: The following chunk of code is copied from
// net_packet_dispatch(). We should change that function // net_packet_dispatch(). We should change that function
// to accept an IOSource instead of the PktSrc. // to accept an IOSource instead of the PktSrc.
network_time = p->time; net_update_time(p->time);
SegmentProfiler(segment_logger, "expiring-timers"); SegmentProfiler(segment_logger, "expiring-timers");
TimerMgr* tmgr = sessions->LookupTimerMgr(GetCurrentTag()); TimerMgr* tmgr = sessions->LookupTimerMgr(GetCurrentTag());
@ -1476,7 +1478,7 @@ void RemoteSerializer::Process()
} }
if ( packets.length() ) if ( packets.length() )
idle = false; SetIdle(false);
} }
void RemoteSerializer::Finish() void RemoteSerializer::Finish()
@ -1508,7 +1510,7 @@ bool RemoteSerializer::Poll(bool may_block)
} }
io->Flush(); io->Flush();
idle = false; SetIdle(false);
switch ( msgstate ) { switch ( msgstate ) {
case TYPE: case TYPE:
@ -1690,7 +1692,7 @@ bool RemoteSerializer::DoMessage()
case MSG_TERMINATE: case MSG_TERMINATE:
assert(terminating); assert(terminating);
io_sources.Terminate(); iosource_mgr->Terminate();
return true; return true;
case MSG_REMOTE_PRINT: case MSG_REMOTE_PRINT:
@ -1878,7 +1880,7 @@ void RemoteSerializer::RemovePeer(Peer* peer)
delete peer->cache_out; delete peer->cache_out;
delete peer; delete peer;
closed = ! IsActive(); SetClosed(! IsActive());
if ( in_sync == peer ) if ( in_sync == peer )
in_sync = 0; in_sync = 0;
@ -2723,8 +2725,8 @@ bool RemoteSerializer::ProcessLogCreateWriter()
fmt.EndRead(); fmt.EndRead();
id_val = new EnumVal(id, BifType::Enum::Log::ID); id_val = new EnumVal(id, internal_type("Log::ID")->AsEnumType());
writer_val = new EnumVal(writer, BifType::Enum::Log::Writer); writer_val = new EnumVal(writer, internal_type("Log::Writer")->AsEnumType());
if ( ! log_mgr->CreateWriter(id_val, writer_val, info, num_fields, fields, if ( ! log_mgr->CreateWriter(id_val, writer_val, info, num_fields, fields,
true, false, true) ) true, false, true) )
@ -2796,8 +2798,8 @@ bool RemoteSerializer::ProcessLogWrite()
} }
} }
id_val = new EnumVal(id, BifType::Enum::Log::ID); id_val = new EnumVal(id, internal_type("Log::ID")->AsEnumType());
writer_val = new EnumVal(writer, BifType::Enum::Log::Writer); writer_val = new EnumVal(writer, internal_type("Log::Writer")->AsEnumType());
success = log_mgr->Write(id_val, writer_val, path, num_fields, vals); success = log_mgr->Write(id_val, writer_val, path, num_fields, vals);
@ -2840,7 +2842,7 @@ void RemoteSerializer::GotEvent(const char* name, double time,
BufferedEvent* e = new BufferedEvent; BufferedEvent* e = new BufferedEvent;
// Our time, not the time when the event was generated. // Our time, not the time when the event was generated.
e->time = pkt_srcs.length() ? e->time = iosource_mgr->GetPktSrcs().size() ?
time_t(network_time) : time_t(timer_mgr->Time()); time_t(network_time) : time_t(timer_mgr->Time());
e->src = current_peer->id; e->src = current_peer->id;
@ -3085,7 +3087,7 @@ RecordVal* RemoteSerializer::GetPeerVal(PeerID id)
void RemoteSerializer::ChildDied() void RemoteSerializer::ChildDied()
{ {
Log(LogError, "child died"); Log(LogError, "child died");
closed = true; SetClosed(true);
child_pid = 0; child_pid = 0;
// Shut down the main process as well. // Shut down the main process as well.
@ -3184,7 +3186,7 @@ void RemoteSerializer::FatalError(const char* msg)
Log(LogError, msg); Log(LogError, msg);
reporter->Error("%s", msg); reporter->Error("%s", msg);
closed = true; SetClosed(true);
if ( kill(child_pid, SIGQUIT) < 0 ) if ( kill(child_pid, SIGQUIT) < 0 )
reporter->Warning("warning: cannot kill child pid %d, %s", child_pid, strerror(errno)); reporter->Warning("warning: cannot kill child pid %d, %s", child_pid, strerror(errno));
@ -3355,6 +3357,15 @@ SocketComm::~SocketComm()
static unsigned int first_rtime = 0; static unsigned int first_rtime = 0;
static void fd_vector_set(const std::vector<int>& fds, fd_set* set, int* max)
{
for ( size_t i = 0; i < fds.size(); ++i )
{
FD_SET(fds[i], set);
*max = ::max(fds[i], *max);
}
}
void SocketComm::Run() void SocketComm::Run()
{ {
first_rtime = (unsigned int) current_time(true); first_rtime = (unsigned int) current_time(true);
@ -3376,10 +3387,9 @@ void SocketComm::Run()
FD_ZERO(&fd_write); FD_ZERO(&fd_write);
FD_ZERO(&fd_except); FD_ZERO(&fd_except);
int max_fd = 0; int max_fd = io->Fd();
FD_SET(io->Fd(), &fd_read); FD_SET(io->Fd(), &fd_read);
max_fd = io->Fd(); max_fd = std::max(max_fd, io->ExtraReadFDs().Set(&fd_read));
loop_over_list(peers, i) loop_over_list(peers, i)
{ {
@ -3388,6 +3398,8 @@ void SocketComm::Run()
FD_SET(peers[i]->io->Fd(), &fd_read); FD_SET(peers[i]->io->Fd(), &fd_read);
if ( peers[i]->io->Fd() > max_fd ) if ( peers[i]->io->Fd() > max_fd )
max_fd = peers[i]->io->Fd(); max_fd = peers[i]->io->Fd();
max_fd = std::max(max_fd,
peers[i]->io->ExtraReadFDs().Set(&fd_read));
} }
else else
{ {
@ -3438,38 +3450,17 @@ void SocketComm::Run()
if ( ! io->IsFillingUp() && shutting_conns_down ) if ( ! io->IsFillingUp() && shutting_conns_down )
shutting_conns_down = false; shutting_conns_down = false;
// We cannot rely solely on select() as the there may
// be some data left in our input/output queues. So, we use
// a small timeout for select and check for data
// manually afterwards.
static long selects = 0; static long selects = 0;
static long canwrites = 0; static long canwrites = 0;
static long timeouts = 0;
++selects; ++selects;
if ( io->CanWrite() ) if ( io->CanWrite() )
++canwrites; ++canwrites;
// FIXME: Fine-tune this (timeouts, flush, etc.) int a = select(max_fd + 1, &fd_read, &fd_write, &fd_except, 0);
struct timeval small_timeout;
small_timeout.tv_sec = 0;
small_timeout.tv_usec =
io->CanWrite() || io->CanRead() ? 1 : 10;
#if 0
if ( ! io->CanWrite() )
usleep(10);
#endif
int a = select(max_fd + 1, &fd_read, &fd_write, &fd_except,
&small_timeout);
if ( a == 0 )
++timeouts;
if ( selects % 100000 == 0 ) if ( selects % 100000 == 0 )
Log(fmt("selects=%ld canwrites=%ld timeouts=%ld", selects, canwrites, timeouts)); Log(fmt("selects=%ld canwrites=%ld", selects, canwrites));
if ( a < 0 ) if ( a < 0 )
// Ignore errors for now. // Ignore errors for now.
@ -4211,6 +4202,7 @@ bool SocketComm::Listen()
safe_close(fd); safe_close(fd);
CloseListenFDs(); CloseListenFDs();
listen_next_try = time(0) + bind_retry_interval; listen_next_try = time(0) + bind_retry_interval;
freeaddrinfo(res0);
return false; return false;
} }

9
src/RemoteSerializer.h
View file

@ -6,7 +6,7 @@
#include "Dict.h" #include "Dict.h"
#include "List.h" #include "List.h"
#include "Serializer.h" #include "Serializer.h"
#include "IOSource.h" #include "iosource/IOSource.h"
#include "Stats.h" #include "Stats.h"
#include "File.h" #include "File.h"
#include "logging/WriterBackend.h" #include "logging/WriterBackend.h"
@ -22,13 +22,13 @@ namespace threading {
} }
// This class handles the communication done in Bro's main loop. // This class handles the communication done in Bro's main loop.
class RemoteSerializer : public Serializer, public IOSource { class RemoteSerializer : public Serializer, public iosource::IOSource {
public: public:
RemoteSerializer(); RemoteSerializer();
virtual ~RemoteSerializer(); virtual ~RemoteSerializer();
// Initialize the remote serializer (calling this will fork). // Initialize the remote serializer (calling this will fork).
void Init(); void Enable();
// FIXME: Use SourceID directly (or rename everything to Peer*). // FIXME: Use SourceID directly (or rename everything to Peer*).
typedef SourceID PeerID; typedef SourceID PeerID;
@ -140,7 +140,8 @@ public:
void Finish(); void Finish();
// Overidden from IOSource: // Overidden from IOSource:
virtual void GetFds(int* read, int* write, int* except); virtual void GetFds(iosource::FD_Set* read, iosource::FD_Set* write,
iosource::FD_Set* except);
virtual double NextTimestamp(double* local_network_time); virtual double NextTimestamp(double* local_network_time);
virtual void Process(); virtual void Process();
virtual TimerMgr::Tag* GetCurrentTag(); virtual TimerMgr::Tag* GetCurrentTag();

6
src/RuleAction.cc
View file

@ -65,11 +65,11 @@ RuleActionAnalyzer::RuleActionAnalyzer(const char* arg_analyzer)
void RuleActionAnalyzer::PrintDebug() void RuleActionAnalyzer::PrintDebug()
{ {
if ( ! child_analyzer ) if ( ! child_analyzer )
fprintf(stderr, "|%s|\n", analyzer_mgr->GetComponentName(analyzer)); fprintf(stderr, "|%s|\n", analyzer_mgr->GetComponentName(analyzer).c_str());
else else
fprintf(stderr, "|%s:%s|\n", fprintf(stderr, "|%s:%s|\n",
analyzer_mgr->GetComponentName(analyzer), analyzer_mgr->GetComponentName(analyzer).c_str(),
analyzer_mgr->GetComponentName(child_analyzer)); analyzer_mgr->GetComponentName(child_analyzer).c_str());
} }

24
src/Serializer.cc
View file

@ -19,6 +19,7 @@
#include "Conn.h" #include "Conn.h"
#include "Timer.h" #include "Timer.h"
#include "RemoteSerializer.h" #include "RemoteSerializer.h"
#include "iosource/Manager.h"
Serializer::Serializer(SerializationFormat* arg_format) Serializer::Serializer(SerializationFormat* arg_format)
{ {
@ -1045,7 +1046,7 @@ EventPlayer::EventPlayer(const char* file)
Error(fmt("event replayer: cannot open %s", file)); Error(fmt("event replayer: cannot open %s", file));
if ( ReadHeader() ) if ( ReadHeader() )
io_sources.Register(this); iosource_mgr->Register(this);
} }
EventPlayer::~EventPlayer() EventPlayer::~EventPlayer()
@ -1067,9 +1068,10 @@ void EventPlayer::GotFunctionCall(const char* name, double time,
// We don't replay function calls. // We don't replay function calls.
} }
void EventPlayer::GetFds(int* read, int* write, int* except) void EventPlayer::GetFds(iosource::FD_Set* read, iosource::FD_Set* write,
iosource::FD_Set* except)
{ {
*read = fd; read->Insert(fd);
} }
double EventPlayer::NextTimestamp(double* local_network_time) double EventPlayer::NextTimestamp(double* local_network_time)
@ -1085,7 +1087,7 @@ double EventPlayer::NextTimestamp(double* local_network_time)
{ {
UnserialInfo info(this); UnserialInfo info(this);
Unserialize(&info); Unserialize(&info);
closed = io->Eof(); SetClosed(io->Eof());
} }
if ( ! ne_time ) if ( ! ne_time )
@ -1142,7 +1144,7 @@ bool Packet::Serialize(SerialInfo* info) const
static BroFile* profiling_output = 0; static BroFile* profiling_output = 0;
#ifdef DEBUG #ifdef DEBUG
static PktDumper* dump = 0; static iosource::PktDumper* dump = 0;
#endif #endif
Packet* Packet::Unserialize(UnserialInfo* info) Packet* Packet::Unserialize(UnserialInfo* info)
@ -1188,7 +1190,7 @@ Packet* Packet::Unserialize(UnserialInfo* info)
p->hdr = hdr; p->hdr = hdr;
p->pkt = (u_char*) pkt; p->pkt = (u_char*) pkt;
p->tag = tag; p->tag = tag;
p->hdr_size = get_link_header_size(p->link_type); p->hdr_size = iosource::PktSrc::GetLinkHeaderSize(p->link_type);
delete [] tag; delete [] tag;
@ -1213,9 +1215,15 @@ Packet* Packet::Unserialize(UnserialInfo* info)
if ( debug_logger.IsEnabled(DBG_TM) ) if ( debug_logger.IsEnabled(DBG_TM) )
{ {
if ( ! dump ) if ( ! dump )
dump = new PktDumper("tm.pcap"); dump = iosource_mgr->OpenPktDumper("tm.pcap", true);
dump->Dump(p->hdr, p->pkt); if ( dump )
{
iosource::PktDumper::Packet dp;
dp.hdr = p->hdr;
dp.data = p->pkt;
dump->Dump(&dp);
}
} }
#endif #endif

7
src/Serializer.h
View file

@ -15,7 +15,7 @@
#include "SerialInfo.h" #include "SerialInfo.h"
#include "IP.h" #include "IP.h"
#include "Timer.h" #include "Timer.h"
#include "IOSource.h" #include "iosource/IOSource.h"
#include "Reporter.h" #include "Reporter.h"
class SerializationCache; class SerializationCache;
@ -350,12 +350,13 @@ public:
}; };
// Plays a file of events back. // Plays a file of events back.
class EventPlayer : public FileSerializer, public IOSource { class EventPlayer : public FileSerializer, public iosource::IOSource {
public: public:
EventPlayer(const char* file); EventPlayer(const char* file);
virtual ~EventPlayer(); virtual ~EventPlayer();
virtual void GetFds(int* read, int* write, int* except); virtual void GetFds(iosource::FD_Set* read, iosource::FD_Set* write,
iosource::FD_Set* except);
virtual double NextTimestamp(double* local_network_time); virtual double NextTimestamp(double* local_network_time);
virtual void Process(); virtual void Process();
virtual const char* Tag() { return "EventPlayer"; } virtual const char* Tag() { return "EventPlayer"; }

66
src/Sessions.cc
View file

@ -167,7 +167,7 @@ void NetSessions::Done()
void NetSessions::DispatchPacket(double t, const struct pcap_pkthdr* hdr, void NetSessions::DispatchPacket(double t, const struct pcap_pkthdr* hdr,
const u_char* pkt, int hdr_size, const u_char* pkt, int hdr_size,
PktSrc* src_ps) iosource::PktSrc* src_ps)
{ {
const struct ip* ip_hdr = 0; const struct ip* ip_hdr = 0;
const u_char* ip_data = 0; const u_char* ip_data = 0;
@ -184,10 +184,7 @@ void NetSessions::DispatchPacket(double t, const struct pcap_pkthdr* hdr,
// Blanket encapsulation // Blanket encapsulation
hdr_size += encap_hdr_size; hdr_size += encap_hdr_size;
if ( src_ps->FilterType() == TYPE_FILTER_NORMAL )
NextPacket(t, hdr, pkt, hdr_size); NextPacket(t, hdr, pkt, hdr_size);
else
NextPacketSecondary(t, hdr, pkt, hdr_size, src_ps);
} }
void NetSessions::NextPacket(double t, const struct pcap_pkthdr* hdr, void NetSessions::NextPacket(double t, const struct pcap_pkthdr* hdr,
@ -262,53 +259,6 @@ void NetSessions::NextPacket(double t, const struct pcap_pkthdr* hdr,
DumpPacket(hdr, pkt); DumpPacket(hdr, pkt);
} }
void NetSessions::NextPacketSecondary(double /* t */, const struct pcap_pkthdr* hdr,
const u_char* const pkt, int hdr_size,
const PktSrc* src_ps)
{
SegmentProfiler(segment_logger, "processing-secondary-packet");
++num_packets_processed;
uint32 caplen = hdr->caplen - hdr_size;
if ( caplen < sizeof(struct ip) )
{
Weird("truncated_IP", hdr, pkt);
return;
}
const struct ip* ip = (const struct ip*) (pkt + hdr_size);
if ( ip->ip_v == 4 )
{
const secondary_program_list& spt = src_ps->ProgramTable();
loop_over_list(spt, i)
{
SecondaryProgram* sp = spt[i];
if ( ! net_packet_match(sp->Program(), pkt,
hdr->len, hdr->caplen) )
continue;
val_list* args = new val_list;
StringVal* cmd_val =
new StringVal(sp->Event()->Filter());
args->append(cmd_val);
IP_Hdr ip_hdr(ip, false);
args->append(ip_hdr.BuildPktHdrVal());
// ### Need to queue event here.
try
{
sp->Event()->Event()->Call(args);
}
catch ( InterpreterException& e )
{ /* Already reported. */ }
delete args;
}
}
}
int NetSessions::CheckConnectionTag(Connection* conn) int NetSessions::CheckConnectionTag(Connection* conn)
{ {
if ( current_iosrc->GetCurrentTag() ) if ( current_iosrc->GetCurrentTag() )
@ -1440,14 +1390,24 @@ void NetSessions::DumpPacket(const struct pcap_pkthdr* hdr,
return; return;
if ( len == 0 ) if ( len == 0 )
pkt_dumper->Dump(hdr, pkt); {
iosource::PktDumper::Packet p;
p.hdr = hdr;
p.data = pkt;
pkt_dumper->Dump(&p);
}
else else
{ {
struct pcap_pkthdr h = *hdr; struct pcap_pkthdr h = *hdr;
h.caplen = len; h.caplen = len;
if ( h.caplen > hdr->caplen ) if ( h.caplen > hdr->caplen )
reporter->InternalError("bad modified caplen"); reporter->InternalError("bad modified caplen");
pkt_dumper->Dump(&h, pkt);
iosource::PktDumper::Packet p;
p.hdr = &h;
p.data = pkt;
pkt_dumper->Dump(&p);
} }
} }

10
src/Sessions.h
View file

@ -69,11 +69,11 @@ public:
~NetSessions(); ~NetSessions();
// Main entry point for packet processing. Dispatches the packet // Main entry point for packet processing. Dispatches the packet
// either through NextPacket() or NextPacketSecondary(), optionally // either through NextPacket(), optionally employing the packet
// employing the packet sorter first. // sorter first.
void DispatchPacket(double t, const struct pcap_pkthdr* hdr, void DispatchPacket(double t, const struct pcap_pkthdr* hdr,
const u_char* const pkt, int hdr_size, const u_char* const pkt, int hdr_size,
PktSrc* src_ps); iosource::PktSrc* src_ps);
void Done(); // call to drain events before destructing void Done(); // call to drain events before destructing
@ -221,10 +221,6 @@ protected:
void NextPacket(double t, const struct pcap_pkthdr* hdr, void NextPacket(double t, const struct pcap_pkthdr* hdr,
const u_char* const pkt, int hdr_size); const u_char* const pkt, int hdr_size);
void NextPacketSecondary(double t, const struct pcap_pkthdr* hdr,
const u_char* const pkt, int hdr_size,
const PktSrc* src_ps);
// Record the given packet (if a dumper is active). If len=0 // Record the given packet (if a dumper is active). If len=0
// then the whole packet is recorded, otherwise just the first // then the whole packet is recorded, otherwise just the first
// len bytes. // len bytes.

7
src/Stmt.cc
View file

@ -660,8 +660,13 @@ void Case::Describe(ODesc* d) const
TraversalCode Case::Traverse(TraversalCallback* cb) const TraversalCode Case::Traverse(TraversalCallback* cb) const
{ {
TraversalCode tc = cases->Traverse(cb); TraversalCode tc;
if ( cases )
{
tc = cases->Traverse(cb);
HANDLE_TC_STMT_PRE(tc); HANDLE_TC_STMT_PRE(tc);
}
tc = s->Traverse(cb); tc = s->Traverse(cb);
HANDLE_TC_STMT_PRE(tc); HANDLE_TC_STMT_PRE(tc);

1
src/Tag.cc
View file

@ -55,6 +55,7 @@ Tag& Tag::operator=(const Tag& other)
{ {
type = other.type; type = other.type;
subtype = other.subtype; subtype = other.subtype;
Unref(val);
val = other.val; val = other.val;
if ( val ) if ( val )

94
src/Type.cc
View file

@ -449,6 +449,11 @@ BroType* IndexType::YieldType()
return yield_type; return yield_type;
} }
const BroType* IndexType::YieldType() const
{
return yield_type;
}
void IndexType::Describe(ODesc* d) const void IndexType::Describe(ODesc* d) const
{ {
BroType::Describe(d); BroType::Describe(d);
@ -742,6 +747,11 @@ BroType* FuncType::YieldType()
return yield; return yield;
} }
const BroType* FuncType::YieldType() const
{
return yield;
}
int FuncType::MatchesIndex(ListExpr*& index) const int FuncType::MatchesIndex(ListExpr*& index) const
{ {
return check_and_promote_args(index, args) ? return check_and_promote_args(index, args) ?
@ -1371,6 +1381,11 @@ void OpaqueType::Describe(ODesc* d) const
d->Add(name.c_str()); d->Add(name.c_str());
} }
void OpaqueType::DescribeReST(ODesc* d, bool roles_only) const
{
d->Add(fmt(":bro:type:`%s` of %s", type_name(Tag()), name.c_str()));
}
IMPLEMENT_SERIAL(OpaqueType, SER_OPAQUE_TYPE); IMPLEMENT_SERIAL(OpaqueType, SER_OPAQUE_TYPE);
bool OpaqueType::DoSerialize(SerialInfo* info) const bool OpaqueType::DoSerialize(SerialInfo* info) const
@ -1393,6 +1408,23 @@ bool OpaqueType::DoUnserialize(UnserialInfo* info)
return true; return true;
} }
EnumType::EnumType(const string& name)
: BroType(TYPE_ENUM)
{
counter = 0;
SetName(name);
}
EnumType::EnumType(EnumType* e)
: BroType(TYPE_ENUM)
{
counter = e->counter;
SetName(e->GetName());
for ( NameMap::iterator it = e->names.begin(); it != e->names.end(); ++it )
names[copy_string(it->first)] = it->second;
}
EnumType::~EnumType() EnumType::~EnumType()
{ {
for ( NameMap::iterator iter = names.begin(); iter != names.end(); ++iter ) for ( NameMap::iterator iter = names.begin(); iter != names.end(); ++iter )
@ -1448,6 +1480,12 @@ void EnumType::CheckAndAddName(const string& module_name, const char* name,
broxygen_mgr->Identifier(id); broxygen_mgr->Identifier(id);
} }
else else
{
// We allow double-definitions if matching exactly. This is so that
// we can define an enum both in a *.bif and *.bro for avoiding
// cyclic dependencies.
if ( id->Name() != make_full_var_name(module_name.c_str(), name)
|| (id->HasVal() && val != id->ID_Val()->AsEnum()) )
{ {
Unref(id); Unref(id);
reporter->Error("identifier or enumerator value in enumerated type definition already exists"); reporter->Error("identifier or enumerator value in enumerated type definition already exists");
@ -1455,6 +1493,9 @@ void EnumType::CheckAndAddName(const string& module_name, const char* name,
return; return;
} }
Unref(id);
}
AddNameInternal(module_name, name, val, is_export); AddNameInternal(module_name, name, val, is_export);
set<BroType*> types = BroType::GetAliases(GetName()); set<BroType*> types = BroType::GetAliases(GetName());
@ -1473,9 +1514,9 @@ void EnumType::AddNameInternal(const string& module_name, const char* name,
names[copy_string(fullname.c_str())] = val; names[copy_string(fullname.c_str())] = val;
} }
bro_int_t EnumType::Lookup(const string& module_name, const char* name) bro_int_t EnumType::Lookup(const string& module_name, const char* name) const
{ {
NameMap::iterator pos = NameMap::const_iterator pos =
names.find(make_full_var_name(module_name.c_str(), name).c_str()); names.find(make_full_var_name(module_name.c_str(), name).c_str());
if ( pos == names.end() ) if ( pos == names.end() )
@ -1484,9 +1525,9 @@ bro_int_t EnumType::Lookup(const string& module_name, const char* name)
return pos->second; return pos->second;
} }
const char* EnumType::Lookup(bro_int_t value) const char* EnumType::Lookup(bro_int_t value) const
{ {
for ( NameMap::iterator iter = names.begin(); for ( NameMap::const_iterator iter = names.begin();
iter != names.end(); ++iter ) iter != names.end(); ++iter )
if ( iter->second == value ) if ( iter->second == value )
return iter->first; return iter->first;
@ -1494,6 +1535,16 @@ const char* EnumType::Lookup(bro_int_t value)
return 0; return 0;
} }
EnumType::enum_name_list EnumType::Names() const
{
enum_name_list n;
for ( NameMap::const_iterator iter = names.begin();
iter != names.end(); ++iter )
n.push_back(std::make_pair(iter->first, iter->second));
return n;
}
void EnumType::DescribeReST(ODesc* d, bool roles_only) const void EnumType::DescribeReST(ODesc* d, bool roles_only) const
{ {
d->Add(":bro:type:`enum`"); d->Add(":bro:type:`enum`");
@ -1644,6 +1695,23 @@ BroType* VectorType::YieldType()
return yield_type; return yield_type;
} }
const BroType* VectorType::YieldType() const
{
// Work around the fact that we use void internally to mark a vector
// as being unspecified. When looking at its yield type, we need to
// return any as that's what other code historically expects for type
// comparisions.
if ( IsUnspecifiedVector() )
{
BroType* ret = ::base_type(TYPE_ANY);
Unref(ret); // unref, because this won't be held by anyone.
assert(ret);
return ret;
}
return yield_type;
}
int VectorType::MatchesIndex(ListExpr*& index) const int VectorType::MatchesIndex(ListExpr*& index) const
{ {
expr_list& el = index->Exprs(); expr_list& el = index->Exprs();
@ -1742,7 +1810,7 @@ static int is_init_compat(const BroType* t1, const BroType* t2)
return 0; return 0;
} }
int same_type(const BroType* t1, const BroType* t2, int is_init) int same_type(const BroType* t1, const BroType* t2, int is_init, bool match_record_field_names)
{ {
if ( t1 == t2 || if ( t1 == t2 ||
t1->Tag() == TYPE_ANY || t1->Tag() == TYPE_ANY ||
@ -1798,7 +1866,7 @@ int same_type(const BroType* t1, const BroType* t2, int is_init)
if ( tl1 || tl2 ) if ( tl1 || tl2 )
{ {
if ( ! tl1 || ! tl2 || ! same_type(tl1, tl2, is_init) ) if ( ! tl1 || ! tl2 || ! same_type(tl1, tl2, is_init, match_record_field_names) )
return 0; return 0;
} }
@ -1807,7 +1875,7 @@ int same_type(const BroType* t1, const BroType* t2, int is_init)
if ( y1 || y2 ) if ( y1 || y2 )
{ {
if ( ! y1 || ! y2 || ! same_type(y1, y2, is_init) ) if ( ! y1 || ! y2 || ! same_type(y1, y2, is_init, match_record_field_names) )
return 0; return 0;
} }
@ -1825,7 +1893,7 @@ int same_type(const BroType* t1, const BroType* t2, int is_init)
if ( t1->YieldType() || t2->YieldType() ) if ( t1->YieldType() || t2->YieldType() )
{ {
if ( ! t1->YieldType() || ! t2->YieldType() || if ( ! t1->YieldType() || ! t2->YieldType() ||
! same_type(t1->YieldType(), t2->YieldType(), is_init) ) ! same_type(t1->YieldType(), t2->YieldType(), is_init, match_record_field_names) )
return 0; return 0;
} }
@ -1845,8 +1913,8 @@ int same_type(const BroType* t1, const BroType* t2, int is_init)
const TypeDecl* td1 = rt1->FieldDecl(i); const TypeDecl* td1 = rt1->FieldDecl(i);
const TypeDecl* td2 = rt2->FieldDecl(i); const TypeDecl* td2 = rt2->FieldDecl(i);
if ( ! streq(td1->id, td2->id) || if ( (match_record_field_names && ! streq(td1->id, td2->id)) ||
! same_type(td1->type, td2->type, is_init) ) ! same_type(td1->type, td2->type, is_init, match_record_field_names) )
return 0; return 0;
} }
@ -1862,7 +1930,7 @@ int same_type(const BroType* t1, const BroType* t2, int is_init)
return 0; return 0;
loop_over_list(*tl1, i) loop_over_list(*tl1, i)
if ( ! same_type((*tl1)[i], (*tl2)[i], is_init) ) if ( ! same_type((*tl1)[i], (*tl2)[i], is_init, match_record_field_names) )
return 0; return 0;
return 1; return 1;
@ -1870,7 +1938,7 @@ int same_type(const BroType* t1, const BroType* t2, int is_init)
case TYPE_VECTOR: case TYPE_VECTOR:
case TYPE_FILE: case TYPE_FILE:
return same_type(t1->YieldType(), t2->YieldType(), is_init); return same_type(t1->YieldType(), t2->YieldType(), is_init, match_record_field_names);
case TYPE_OPAQUE: case TYPE_OPAQUE:
{ {
@ -1880,7 +1948,7 @@ int same_type(const BroType* t1, const BroType* t2, int is_init)
} }
case TYPE_TYPE: case TYPE_TYPE:
return same_type(t1, t2, is_init); return same_type(t1, t2, is_init, match_record_field_names);
case TYPE_UNION: case TYPE_UNION:
reporter->Error("union type in same_type()"); reporter->Error("union type in same_type()");

27
src/Type.h
View file

@ -6,6 +6,7 @@
#include <string> #include <string>
#include <set> #include <set>
#include <map> #include <map>
#include <list>
#include "Obj.h" #include "Obj.h"
#include "Attr.h" #include "Attr.h"
@ -334,6 +335,7 @@ public:
TypeList* Indices() const { return indices; } TypeList* Indices() const { return indices; }
const type_list* IndexTypes() const { return indices->Types(); } const type_list* IndexTypes() const { return indices->Types(); }
BroType* YieldType(); BroType* YieldType();
const BroType* YieldType() const;
void Describe(ODesc* d) const; void Describe(ODesc* d) const;
void DescribeReST(ODesc* d, bool roles_only = false) const; void DescribeReST(ODesc* d, bool roles_only = false) const;
@ -396,6 +398,7 @@ public:
RecordType* Args() const { return args; } RecordType* Args() const { return args; }
BroType* YieldType(); BroType* YieldType();
const BroType* YieldType() const;
void SetYieldType(BroType* arg_yield) { yield = arg_yield; } void SetYieldType(BroType* arg_yield) { yield = arg_yield; }
function_flavor Flavor() const { return flavor; } function_flavor Flavor() const { return flavor; }
string FlavorString() const; string FlavorString() const;
@ -531,6 +534,7 @@ public:
const string& Name() const { return name; } const string& Name() const { return name; }
void Describe(ODesc* d) const; void Describe(ODesc* d) const;
void DescribeReST(ODesc* d, bool roles_only = false) const;
protected: protected:
OpaqueType() { } OpaqueType() { }
@ -542,7 +546,10 @@ protected:
class EnumType : public BroType { class EnumType : public BroType {
public: public:
EnumType() : BroType(TYPE_ENUM) { counter = 0; } typedef std::list<std::pair<string, bro_int_t> > enum_name_list;
EnumType(EnumType* e);
EnumType(const string& arg_name);
~EnumType(); ~EnumType();
// The value of this name is next internal counter value, starting // The value of this name is next internal counter value, starting
@ -555,12 +562,18 @@ public:
void AddName(const string& module_name, const char* name, bro_int_t val, bool is_export); void AddName(const string& module_name, const char* name, bro_int_t val, bool is_export);
// -1 indicates not found. // -1 indicates not found.
bro_int_t Lookup(const string& module_name, const char* name); bro_int_t Lookup(const string& module_name, const char* name) const;
const char* Lookup(bro_int_t value); // Returns 0 if not found const char* Lookup(bro_int_t value) const; // Returns 0 if not found
// Returns the list of defined names with their values. The names
// will be fully qualified with their module name.
enum_name_list Names() const;
void DescribeReST(ODesc* d, bool roles_only = false) const; void DescribeReST(ODesc* d, bool roles_only = false) const;
protected: protected:
EnumType() { counter = 0; }
DECLARE_SERIAL(EnumType) DECLARE_SERIAL(EnumType)
void AddNameInternal(const string& module_name, void AddNameInternal(const string& module_name,
@ -586,6 +599,7 @@ public:
VectorType(BroType* t); VectorType(BroType* t);
virtual ~VectorType(); virtual ~VectorType();
BroType* YieldType(); BroType* YieldType();
const BroType* YieldType() const;
int MatchesIndex(ListExpr*& index) const; int MatchesIndex(ListExpr*& index) const;
@ -625,9 +639,10 @@ inline BroType* base_type(TypeTag tag)
// Returns the BRO basic error type. // Returns the BRO basic error type.
inline BroType* error_type() { return base_type(TYPE_ERROR); } inline BroType* error_type() { return base_type(TYPE_ERROR); }
// True if the two types are equivalent. If is_init is true then the // True if the two types are equivalent. If is_init is true then the test is
// test is done in the context of an initialization. // done in the context of an initialization. If match_record_field_names is
extern int same_type(const BroType* t1, const BroType* t2, int is_init=0); // true then for record types the field names have to match, too.
extern int same_type(const BroType* t1, const BroType* t2, int is_init=0, bool match_record_field_names=true);
// True if the two attribute lists are equivalent. // True if the two attribute lists are equivalent.
extern int same_attrs(const Attributes* a1, const Attributes* a2); extern int same_attrs(const Attributes* a1, const Attributes* a2);

3
src/Val.cc
View file

@ -465,10 +465,7 @@ void Val::Describe(ODesc* d) const
d->SP(); d->SP();
} }
if ( d->IsReadable() )
ValDescribe(d); ValDescribe(d);
else
Val::ValDescribe(d);
} }
void Val::DescribeReST(ODesc* d) const void Val::DescribeReST(ODesc* d) const

39
src/Var.cc
View file

@ -9,6 +9,7 @@
#include "Serializer.h" #include "Serializer.h"
#include "RemoteSerializer.h" #include "RemoteSerializer.h"
#include "EventRegistry.h" #include "EventRegistry.h"
#include "Traverse.h"
static Val* init_val(Expr* init, const BroType* t, Val* aggr) static Val* init_val(Expr* init, const BroType* t, Val* aggr)
{ {
@ -392,6 +393,34 @@ void begin_func(ID* id, const char* module_name, function_flavor flavor,
} }
} }
class OuterIDBindingFinder : public TraversalCallback {
public:
OuterIDBindingFinder(Scope* s)
: scope(s) { }
virtual TraversalCode PreExpr(const Expr*);
Scope* scope;
vector<const NameExpr*> outer_id_references;
};
TraversalCode OuterIDBindingFinder::PreExpr(const Expr* expr)
{
if ( expr->Tag() != EXPR_NAME )
return TC_CONTINUE;
const NameExpr* e = static_cast<const NameExpr*>(expr);
if ( e->Id()->IsGlobal() )
return TC_CONTINUE;
if ( scope->GetIDs()->Lookup(e->Id()->Name()) )
return TC_CONTINUE;
outer_id_references.push_back(e);
return TC_CONTINUE;
}
void end_func(Stmt* body, attr_list* attrs) void end_func(Stmt* body, attr_list* attrs)
{ {
int frame_size = current_scope()->Length(); int frame_size = current_scope()->Length();
@ -429,6 +458,16 @@ void end_func(Stmt* body, attr_list* attrs)
} }
} }
if ( streq(id->Name(), "anonymous-function") )
{
OuterIDBindingFinder cb(scope);
body->Traverse(&cb);
for ( size_t i = 0; i < cb.outer_id_references.size(); ++i )
cb.outer_id_references[i]->Error(
"referencing outer function IDs not supported");
}
if ( id->HasVal() ) if ( id->HasVal() )
id->ID_Val()->AsFunc()->AddBody(body, inits, frame_size, priority); id->ID_Val()->AsFunc()->AddBody(body, inits, frame_size, priority);
else else

Some files were not shown because too many files have changed in this diff Show more