diff --git a/.gitmodules b/.gitmodules index 4998cc6b80..24375ce23d 100644 --- a/.gitmodules +++ b/.gitmodules @@ -19,3 +19,6 @@ [submodule "src/3rdparty"] path = src/3rdparty url = git://git.bro.org/bro-3rdparty +[submodule "aux/plugins"] + path = aux/plugins + url = git://git.bro.org/bro-plugins diff --git a/CHANGES b/CHANGES index 6746b0e86a..85de307c2a 100644 --- a/CHANGES +++ b/CHANGES @@ -1,4 +1,234 @@ +2.3-183 | 2014-09-24 10:08:04 -0500 + + * Add a "node" field to Intel::Seen struture and intel.log to + indicate which node discovered a hit on an intel item. (Seth Hall) + + * BIT-1261: Fixes to plugin quick start doc. (Jon Siwek) + +2.3-180 | 2014-09-22 12:52:41 -0500 + + * BIT-1259: Fix issue w/ duplicate TCP reassembly deliveries. + (Jon Siwek) + +2.3-178 | 2014-09-18 14:29:46 -0500 + + * BIT-1256: Fix file analysis events from coming after bro_done(). + (Jon Siwek) + +2.3-177 | 2014-09-17 09:41:27 -0500 + + * Documentation fixes. (Chris Mavrakis) + +2.3-174 | 2014-09-17 09:37:09 -0500 + + * Fixed some "make doc" warnings caused by reST formatting + (Daniel Thayer). + +2.3-172 | 2014-09-15 13:38:52 -0500 + + * Remove unneeded allocations for HTTP messages. (Jon Siwek) + +2.3-171 | 2014-09-15 11:14:57 -0500 + + * Fix a compile error on systems without pcap-int.h. (Jon Siwek) + +2.3-170 | 2014-09-12 19:28:01 -0700 + + * Fix incorrect data delivery skips after gap in HTTP Content-Range. + Addresses BIT-1247. (Jon Siwek) + + * Fix file analysis placement of data after gap in HTTP + Content-Range. Addresses BIT-1248. (Jon Siwek) + + * Fix issue w/ TCP reassembler not delivering some segments. + Addresses BIT-1246. (Jon Siwek) + + * Fix MIME entity file data/gap ordering and raise http_entity_data + in line with data arrival. Addresses BIT-1240. (Jon Siwek) + + * Implement file ID caching for MIME_Mail. (Jon Siwek) + + * Fix a compile error. (Jon Siwek) + +2.3-161 | 2014-09-09 12:35:38 -0500 + + * Bugfixes and test updates/additions. (Robin Sommer) + + * Interface tweaks and docs for PktSrc/PktDumper. (Robin Sommer) + + * Moving PCAP-related bifs to iosource/pcap.bif. (Robin Sommer) + + * Moving some of the BPF filtering code into base class. + This will allow packet sources that don't support BPF natively to + emulate the filtering via libpcap. (Robin Sommer) + + * Removing FlowSrc. (Robin Sommer) + + * Removing remaining pieces of the 2ndary path, and left-over + files of packet sorter. (Robin Sommer) + + * A bunch of infrastructure work to move IOSource, IOSourceRegistry + (now iosource::Manager) and PktSrc/PktDumper code into iosource/, + and over to a plugin structure. (Robin Sommer) + +2.3-137 | 2014-09-08 19:01:13 -0500 + + * Fix Broxygen's rendering of opaque types. (Jon Siwek) + +2.3-136 | 2014-09-07 20:50:46 -0700 + + * Change more http links to https. (Johanna Amann) + +2.3-134 | 2014-09-04 16:16:36 -0700 + + * Fixed a number of issues with OCSP reply validation. Addresses + BIT-1212. (Johanna Amann) + + * Fix null pointer dereference in OCSP verification code in case no + certificate is sent as part as the ocsp reply. Addresses BIT-1212. + (Johanna Amann) + +2.3-131 | 2014-09-04 16:10:32 -0700 + + * Make links in documentation templates protocol relative. (Johanna + Amann) + +2.3-129 | 2014-09-02 17:21:21 -0700 + + * Simplify a conditional with equivalent branches. (Jon Siwek) + + * Change EDNS parsing code to use rdlength more cautiously. (Jon + Siwek) + + * Fix a memory leak when bind() fails due to EADDRINUSE. (Jon Siwek) + + * Fix possible buffer over-read in DNS TSIG parsing. (Jon Siwek) + +2.3-124 | 2014-08-26 09:24:19 -0500 + + * Better documentation for sub_bytes (Jimmy Jones) + + * BIT-1234: Fix build on systems that already have ntohll/htonll + (Jon Siwek) + +2.3-121 | 2014-08-22 15:22:15 -0700 + + * Detect functions that try to bind variables from an outer scope + and raise an error saying that's not supported. Addresses + BIT-1233. (Jon Siwek) + +2.3-116 | 2014-08-21 16:04:13 -0500 + + * Adding plugin testing to Makefile's test-all. (Robin Sommer) + + * Converting log writers and input readers to plugins. + DataSeries and ElasticSearch plugins have moved to the new + bro-plugins repository, which is now a git submodule in the + aux/plugins directory. (Robin Sommer) + +2.3-98 | 2014-08-19 11:03:46 -0500 + + * Silence some doc-related warnings when using `bro -e`. + Closes BIT-1232. (Jon Siwek) + + * Fix possible null ptr derefs reported by Coverity. (Jon Siwek) + +2.3-96 | 2014-08-01 14:35:01 -0700 + + * Small change to DHCP documentation. In server->client messages the + host name may differ from the one requested by the client. + (Johanna Amann) + + * Split DHCP log writing from record creation. This allows users to + customize dhcp.log by changing the record in their own dhcp_ack + event. (Johanna Amann) + + * Update PATH so that documentation btests can find bro-cut. (Daniel + Thayer) + + * Remove gawk from list of optional packages in documentation. + (Daniel Thayer) + + * Fix for redefining built-in constants. (Robin Sommer) + +2.3-86 | 2014-07-31 14:19:58 -0700 + + * Fix for redefining built-in constants. (Robin Sommer) + + * Adding missing check that a plugin's API version matches what Bro + defines. (Robin Sommer) + + * Adding NEWS entry for plugins. (Robin Sommer) + +2.3-83 | 2014-07-30 16:26:11 -0500 + + * Minor adjustments to plugin code/docs. (Jon Siwek) + + * Dynamic plugin support. (Rpbin Sommer) + + Bro now supports extending core functionality, like protocol and + file analysis, dynamically with external plugins in the form of + shared libraries. See doc/devel/plugins.rst for an overview of the + main functionality. Changes coming with this: + + - Replacing the old Plugin macro magic with a new API. + + - The plugin API changed to generally use std::strings instead + of const char*. + + - There are a number of invocations of PLUGIN_HOOK_ + {VOID,WITH_RESULT} across the code base, which allow plugins + to hook into the processing at those locations. + + - A few new accessor methods to various classes to allow + plugins to get to that information. + + - network_time cannot be just assigned to anymore, there's now + function net_update_time() for that. + + - Redoing how builtin variables are initialized, so that it + works for plugins as well. No more init_net_var(), but + instead bifcl-generated code that registers them. + + - Various changes for adjusting to the now dynamic generation + of analyzer instances. + + - same_type() gets an optional extra argument allowing record type + comparision to ignore if field names don't match. (Robin Sommer) + + - Further unify file analysis API with the protocol analyzer API + (assigning IDs to analyzers; adding Init()/Done() methods; + adding subtypes). (Robin Sommer) + + - A new command line option -Q that prints some basic execution + time stats. (Robin Sommer) + + - Add support to the file analysis for activating analyzers by + MIME type. (Robin Sommer) + + - File::register_for_mime_type(tag: Analyzer::Tag, mt: + string): Associates a file analyzer with a MIME type. + + - File::add_analyzers_for_mime_type(f: fa_file, mtype: + string): Activates all analyzers registered for a MIME + type for the file. + + - The default file_new() handler calls + File::add_analyzers_for_mime_type() with the file's MIME + type. + +2.3-20 | 2014-07-22 17:41:02 -0700 + + * Updating submodule(s). + +2.3-19 | 2014-07-22 17:29:19 -0700 + + * Implement bytestring_to_coils() in Modbus analyzer so that coils + gets passed to the corresponding events. (Hui Lin) + + * Add length field to ModbusHeaders. (Hui Lin) + 2.3-12 | 2014-07-10 19:17:37 -0500 * Include yield of vectors in Broxygen's type descriptions. @@ -12,7 +242,7 @@ 2.3-7 | 2014-06-26 17:35:18 -0700 * Extending "make test-all" to include aux/bro-aux. (Robin Sommer) - + 2.3-6 | 2014-06-26 17:24:10 -0700 * DataSeries compilation issue fixed. (mlaterman) @@ -135,7 +365,7 @@ and the load balancing FAQ on the website. (Daniel Thayer) * Update some doc tests and line numbers (Daniel Thayer) - + 2.2-457 | 2014-05-16 14:38:31 -0700 * New script policy/protocols/ssl/validate-ocsp.bro that adds OSCP @@ -150,7 +380,7 @@ Amann) * Improved Heartbleed attack/scan detection. (Bernhard Amann) - + * Let TLS analyzer fail better when no longer in sync with the data stream. (Bernhard Amann) @@ -187,12 +417,12 @@ *Undelivered methods now use a uint64 in place of an int for the relative sequence space offset parameter. - Addresses BIT-348. + Addresses BIT-348. * Fixing compiler warnings. (Robin Sommer) - + * Update SNMP analyzer's DeliverPacket method signature. (Jon Siwek) - + 2.2-417 | 2014-05-07 10:59:22 -0500 * Change handling of atypical OpenSSL error case in x509 verification. (Jon Siwek) @@ -232,7 +462,7 @@ 2.2-397 | 2014-05-01 20:29:20 -0700 * Fix reference counting for lookup_ID() usages. (Jon Siwek) - + 2.2-395 | 2014-05-01 20:25:48 -0700 * Fix missing "irc-dcc-data" service field from IRC DCC connections. @@ -245,7 +475,7 @@ Siwek) * Improve file analysis manager shutdown/cleanup. (Jon Siwek) - + 2.2-388 | 2014-04-24 18:38:07 -0700 * Fix decoding of MIME quoted-printable. (Mareq) @@ -258,11 +488,11 @@ 2.2-381 | 2014-04-24 17:08:45 -0700 * Add Java version to software framework. (Brian Little) - + 2.2-379 | 2014-04-24 17:06:21 -0700 * Remove unused Val::attribs member. (Jon Siwek) - + 2.2-377 | 2014-04-24 16:57:54 -0700 * A larger set of SSL improvements and extensions. Addresses @@ -275,7 +505,7 @@ server_name, alpn, and ec-curves. - Adds support for the heartbeat events. - + - Add Heartbleed detector script. - Adds basic support for OCSP stapling. @@ -286,7 +516,7 @@ 2.2-353 | 2014-04-24 16:12:30 -0700 * Adapt HTTP partial content to cache file analysis IDs. (Jon Siwek) - + * Adapt SSL analyzer to generate file analysis handles itself. (Jon Siwek) @@ -302,11 +532,11 @@ Siwek) * Refactor file analysis file ID lookup. (Jon Siwek) - + 2.2-344 | 2014-04-22 20:13:30 -0700 * Refactor various hex escaping code. (Jon Siwek) - + 2.2-341 | 2014-04-17 18:01:41 -0500 * Fix duplicate DNS log entries. (Robin Sommer) @@ -314,9 +544,9 @@ 2.2-341 | 2014-04-17 18:01:01 -0500 * Refactor initialization of ASCII log writer options. (Jon Siwek) - + * Fix a memory leak in ASCII log writer. (Jon Siwek) - + 2.2-338 | 2014-04-17 17:48:17 -0500 * Disable input/logging threads setting their names on every @@ -328,7 +558,7 @@ 2.2-335 | 2014-04-10 15:04:57 -0700 * Small logic fix for main SSL script. (Bernhard Amann) - + * Update DPD signatures for detecting TLS 1.2. (Bernhard Amann) * Remove unused data member of SMTP_Analyzer to silence a Coverity @@ -352,7 +582,7 @@ 2.2-317 | 2014-04-03 10:51:31 -0400 - * Add a uid field to the signatures.log. Addresses BIT-1171 + * Add a uid field to the signatures.log. Addresses BIT-1171 (Anthony Verez) 2.2-315 | 2014-04-01 16:50:01 -0700 @@ -363,7 +593,7 @@ 2.2-313 | 2014-04-01 16:40:19 -0700 * Fix a couple nits reported by Coverity.(Jon Siwek) - + * Fix potential memory leak in IP frag reassembly reported by Coverity. (Jon Siwek) @@ -453,14 +683,14 @@ 2.2-271 | 2014-03-30 20:25:17 +0200 * Add unit tests covering vector/set/table ctors/inits. (Jon Siwek) - + * Fix parsing of "local" named table constructors. (Jon Siwek) * Improve type checking of records. Addresses BIT-1159. (Jon Siwek) - + 2.2-267 | 2014-03-30 20:21:43 +0200 - * Improve documentation of Bro clusters. Addresses BIT-1160. + * Improve documentation of Bro clusters. Addresses BIT-1160. (Daniel Thayer) 2.2-263 | 2014-03-30 20:19:05 +0200 @@ -487,7 +717,7 @@ 2.2-254 | 2014-03-30 19:55:22 +0200 * Update instructions on how to build Bro docs. (Daniel Thayer) - + 2.2-251 | 2014-03-28 08:37:37 -0400 * Quick fix to the ElasticSearch writer. (Seth Hall) @@ -517,7 +747,7 @@ formatter work. * Fixing compiler error. (Robin Sommer) - + * Fixing (very unlikely) double delete in HTTP analyzer when decapsulating CONNECTs. (Robin Sommer) @@ -543,7 +773,7 @@ Addresses BIT-1134. (Jon Siwek) * Enable fake DNS mode for test suites. - + * Improve analysis of TCP SYN/SYN-ACK reversal situations. (Jon Siwek) @@ -575,7 +805,7 @@ (Jon Siwek) * Silences some documentation warnings from Sphinx. (Jon Siwek) - + 2.2-215 | 2014-03-10 11:10:15 -0700 * Fix non-deterministic logging of unmatched DNS msgs. Addresses @@ -613,7 +843,7 @@ HTTP traffic. (Seth Hall) * Fixing removal of support analyzers. (Robin Sommer) - + 2.2-199 | 2014-03-03 16:34:20 -0800 * Allow iterating over bif functions with result type vector of any. @@ -628,11 +858,11 @@ 2.2-194 | 2014-02-28 14:50:53 -0800 * Remove packet sorter. Addresses BIT-700. (Bernhard Amann) - + 2.2-192 | 2014-02-28 09:46:43 -0800 * Update Mozilla root bundle. (Bernhard Amann) - + 2.2-190 | 2014-02-27 07:34:44 -0800 * Adjust timings of a few leak tests. (Bernhard Amann) @@ -662,7 +892,7 @@ 2.2-177 | 2014-02-20 17:27:46 -0800 * Update to libmagic version 5.17. Addresses BIT-1136. (Jon Siwek) - + 2.2-174 | 2014-02-14 12:07:04 -0800 * Support for MPLS over VLAN. (Chris Kanich) @@ -691,7 +921,7 @@ defined, but we see it being actively used. (Bernhard Amann) * Test baseline updates for DNS change. (Robin Sommer) - + 2.2-158 | 2014-02-09 23:45:39 -0500 * Change dns.log to include only standard DNS queries. (Jon Siwek) @@ -767,7 +997,7 @@ 2.2-128 | 2014-01-30 15:58:47 -0800 * Add leak test for Exec module. (Bernhard Amann) - + * Fix file_over_new_connection event to trigger when entire file is missed. (Jon Siwek) @@ -780,7 +1010,7 @@ "detect_filtered_trace". (Jon Siwek) * Improve TCP FIN retransmission handling. (Jon Siwek) - + 2.2-120 | 2014-01-28 10:25:23 -0800 * Fix and extend x509_extension() event, which now actually returns @@ -798,10 +1028,10 @@ 2.2-115 | 2014-01-22 12:11:18 -0800 * Add unit tests for new Bro Manual docs. (Jon Siwek) - + * New content for the "Using Bro" section of the manual. (Rafael Bonilla/Jon Siwek) - + 2.2-105 | 2014-01-20 12:16:48 -0800 * Support GRE tunnel decapsulation, including enhanced GRE headers. @@ -810,7 +1040,7 @@ Addresses BIT-867. (Jon Siwek) * Simplify FragReassembler memory management. (Jon Siwek) - + 2.2-102 | 2014-01-20 12:00:29 -0800 * Include file information (MIME type and description) into notice @@ -819,7 +1049,7 @@ 2.2-100 | 2014-01-20 11:54:58 -0800 * Fix caching of recently validated SSL certifcates. (Justin Azoff) - + 2.2-98 | 2014-01-20 11:50:32 -0800 * For notice suppresion, instead of storing the entire notice in @@ -847,7 +1077,7 @@ the city database instead of just the former. (Jon Siwek) * Broxygen init fixes. Addresses BIT-1110. (Jon Siwek) - + - Don't check mtime of bro binary if BRO_DISABLE_BROXYGEN env var set. - Fix failure to locate bro binary if invoking from a relative @@ -860,7 +1090,7 @@ len field being set to zero. (Seth Hall) * Canonify output of a unit test. (Jon Siwek) - + * A set of documentation updates. (Daniel Thayer) - Fix typo in Bro 2.2 NEWS on string indexing. @@ -907,9 +1137,9 @@ (Jon Siwek) * Close signature files after done parsing. (Jon Siwek) - + * Fix unlikely null ptr deref in broxygen::Manager. (Jon Siwek) - + * FreeBSD build fix addendum: unintended variable shadowing. (Jon Siwek) @@ -928,7 +1158,7 @@ were cleaned up. Addresses BIT-1103. (Bernhard Amann) * Minor Broxygen improvements. Addresses BIT-1098. (Jon Siwek) - + 2.2-51 | 2013-12-05 07:53:37 -0800 * Improve a unit test involving 'when' conditionals. (Jon Siwek) @@ -953,7 +1183,7 @@ 2.2-44 | 2013-12-04 12:41:51 -0800 * Fix string slice notation. Addresses BIT-1097. (Jon Siwek) - + Slice ranges were not correctly determined for negative indices and also off by one in general (included one more element at the end of the substring than what actually matched the index range). @@ -1008,7 +1238,7 @@ 2.2-11 | 2013-12-03 10:56:28 -0800 * Unit test for broccoli vector support. (Jon Siwek) - + * Changed ordering of Bro type tag enum, which was out of sync. (Jon Siwek) @@ -1020,14 +1250,14 @@ 2.2-6 | 2013-11-15 07:05:15 -0800 * Make "install-example-configs" target use DESTDIR. (Jon Siwek) - + 2.2-5 | 2013-11-11 13:47:54 -0800 * Fix the irc_reply event for certain server message types. (Seth Hall) * Fixed Segmentation fault in SQLite Writer. (Jon Crussell) - + 2.2 | 2013-11-07 10:25:50 -0800 * Release 2.2. diff --git a/CMakeLists.txt b/CMakeLists.txt index 87b99271c5..22d63a89d5 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,5 +1,9 @@ project(Bro C CXX) + +# When changing the minimum version here, also adapt +# aux/bro-aux/plugin-support/skeleton/CMakeLists.txt cmake_minimum_required(VERSION 2.6.3 FATAL_ERROR) + include(cmake/CommonCMakeConfig.cmake) ######################################################################## @@ -16,12 +20,18 @@ endif () get_filename_component(BRO_SCRIPT_INSTALL_PATH ${BRO_SCRIPT_INSTALL_PATH} ABSOLUTE) +set(BRO_PLUGIN_INSTALL_PATH ${BRO_ROOT_DIR}/lib/bro/plugins CACHE STRING "Installation path for plugins" FORCE) + configure_file(bro-path-dev.in ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev) + file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.sh "export BROPATH=`${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" + "export BRO_PLUGIN_PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src:${BRO_PLUGIN_INSTALL_PATH}\"\n" "export PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n") + file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.csh "setenv BROPATH `${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" + "setenv BRO_PLUGIN_PATH \"${CMAKE_CURRENT_BINARY_DIR}/src:${BRO_PLUGIN_INSTALL_PATH}\"\n" "setenv PATH \"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n") file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/VERSION" VERSION LIMIT_COUNT 1) @@ -117,33 +127,6 @@ if (GOOGLEPERFTOOLS_FOUND) endif () endif () -set(USE_DATASERIES false) -find_package(Lintel) -find_package(DataSeries) -find_package(LibXML2) - -if (NOT DISABLE_DATASERIES AND - LINTEL_FOUND AND DATASERIES_FOUND AND LIBXML2_FOUND) - set(USE_DATASERIES true) - include_directories(BEFORE ${Lintel_INCLUDE_DIR}) - include_directories(BEFORE ${DataSeries_INCLUDE_DIR}) - include_directories(BEFORE ${LibXML2_INCLUDE_DIR}) - list(APPEND OPTLIBS ${Lintel_LIBRARIES}) - list(APPEND OPTLIBS ${DataSeries_LIBRARIES}) - list(APPEND OPTLIBS ${LibXML2_LIBRARIES}) -endif() - -set(USE_ELASTICSEARCH false) -set(USE_CURL false) -find_package(LibCURL) - -if (NOT DISABLE_ELASTICSEARCH AND LIBCURL_FOUND) - set(USE_ELASTICSEARCH true) - set(USE_CURL true) - include_directories(BEFORE ${LibCURL_INCLUDE_DIR}) - list(APPEND OPTLIBS ${LibCURL_LIBRARIES}) -endif() - if (ENABLE_PERFTOOLS_DEBUG OR ENABLE_PERFTOOLS) # Just a no op to prevent CMake from complaining about manually-specified # ENABLE_PERFTOOLS_DEBUG or ENABLE_PERFTOOLS not being used if google @@ -165,6 +148,8 @@ set(brodeps include(TestBigEndian) test_big_endian(WORDS_BIGENDIAN) +include(CheckSymbolExists) +check_symbol_exists(htonll arpa/inet.h HAVE_BYTEORDER_64) include(OSSpecific) include(CheckTypes) @@ -174,6 +159,10 @@ include(MiscTests) include(PCAPTests) include(OpenSSLTests) include(CheckNameserCompat) +include(GetArchitecture) + +# Tell the plugin code that we're building as part of the main tree. +set(BRO_PLUGIN_INTERNAL_BUILD true CACHE INTERNAL "" FORCE) configure_file(${CMAKE_CURRENT_SOURCE_DIR}/config.h.in ${CMAKE_CURRENT_BINARY_DIR}/config.h) @@ -238,10 +227,6 @@ message( "\n tcmalloc: ${USE_PERFTOOLS_TCMALLOC}" "\n debugging: ${USE_PERFTOOLS_DEBUG}" "\njemalloc: ${ENABLE_JEMALLOC}" - "\ncURL: ${USE_CURL}" - "\n" - "\nDataSeries: ${USE_DATASERIES}" - "\nElasticSearch: ${USE_ELASTICSEARCH}" "\n" "\n================================================================\n" ) diff --git a/Makefile b/Makefile index 2b8e66503b..49d9a6173c 100644 --- a/Makefile +++ b/Makefile @@ -56,6 +56,7 @@ test-all: test test -d aux/broctl && ( cd aux/broctl && make test ) test -d aux/btest && ( cd aux/btest && make test ) test -d aux/bro-aux && ( cd aux/bro-aux && make test ) + test -d aux/plugins && ( cd aux/plugins && make test-all ) configured: @test -d $(BUILD) || ( echo "Error: No build/ directory found. Did you run configure?" && exit 1 ) diff --git a/NEWS b/NEWS index b6754e1389..6da13833c3 100644 --- a/NEWS +++ b/NEWS @@ -4,6 +4,32 @@ release. For an exhaustive list of changes, see the ``CHANGES`` file (note that submodules, such as BroControl and Broccoli, come with their own ``CHANGES``.) +Bro 2.4 (in progress) +===================== + +Dependencies +------------ + +New Functionality +----------------- + +- Bro now has support for external plugins that can extend its core + functionality, like protocol/file analysis, via shared libraries. + Plugins can be developed and distributed externally, and will be + pulled in dynamically at startup. Currently, a plugin can provide + custom protocol analyzers, file analyzers, log writers[TODO], input + readers[TODO], packet sources[TODO], and new built-in functions. A + plugin can furthermore hook into Bro's processing a number of places + to add custom logic. + + See https://www.bro.org/sphinx-git/devel/plugins.html for more + information on writing plugins. + +Changed Functionality +--------------------- + +- bro-cut has been rewritten in C, and is hence much faster. + Bro 2.3 ======= diff --git a/VERSION b/VERSION index e6cb320816..5e605d2618 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.3-12 +2.3-183 diff --git a/aux/binpac b/aux/binpac index ec1e052afd..3a4684801a 160000 --- a/aux/binpac +++ b/aux/binpac @@ -1 +1 @@ -Subproject commit ec1e052afd5a8cd3d1d2cbb28fcd688018e379a5 +Subproject commit 3a4684801aafa0558383199e9abd711650b53af9 diff --git a/aux/bro-aux b/aux/bro-aux index 31d011479a..9ea20c3905 160000 --- a/aux/bro-aux +++ b/aux/bro-aux @@ -1 +1 @@ -Subproject commit 31d011479a4e956e029d8b708446841a088dd7e3 +Subproject commit 9ea20c3905bd3fd5109849c474a2f2b4ed008357 diff --git a/aux/broccoli b/aux/broccoli index 1ee129f715..33d0ed4a54 160000 --- a/aux/broccoli +++ b/aux/broccoli @@ -1 +1 @@ -Subproject commit 1ee129f7159a2c32fe0cb0f44c9412486fb7a479 +Subproject commit 33d0ed4a54a6ecf08a0b5fe18831aa413b437066 diff --git a/aux/broctl b/aux/broctl index 8a13886f32..2f808bc854 160000 --- a/aux/broctl +++ b/aux/broctl @@ -1 +1 @@ -Subproject commit 8a13886f322f3b618832c0ca3976e07f686d14da +Subproject commit 2f808bc8541378b1a4953cca02c58c43945d154f diff --git a/aux/btest b/aux/btest index 4da1bd2403..1efa4d10f9 160000 --- a/aux/btest +++ b/aux/btest @@ -1 +1 @@ -Subproject commit 4da1bd24038d4977e655f2b210f34e37f0b73b78 +Subproject commit 1efa4d10f943351efea96def68e598b053fd217a diff --git a/aux/plugins b/aux/plugins new file mode 160000 index 0000000000..23055b473c --- /dev/null +++ b/aux/plugins @@ -0,0 +1 @@ +Subproject commit 23055b473c689a79da12b2825d8388f71f28c709 diff --git a/cmake b/cmake index 0f301aa08a..03de0cc467 160000 --- a/cmake +++ b/cmake @@ -1 +1 @@ -Subproject commit 0f301aa08a970150195a2ea5b3ed43d2d98b35b3 +Subproject commit 03de0cc467d2334dcb851eddd843d59fef217909 diff --git a/config.h.in b/config.h.in index 2d065f755e..755a9eee98 100644 --- a/config.h.in +++ b/config.h.in @@ -129,6 +129,9 @@ /* whether words are stored with the most significant byte first */ #cmakedefine WORDS_BIGENDIAN +/* whether htonll/ntohll is defined in */ +#cmakedefine HAVE_BYTEORDER_64 + /* ultrix can't hack const */ #cmakedefine NEED_ULTRIX_CONST_HACK #ifdef NEED_ULTRIX_CONST_HACK @@ -209,3 +212,14 @@ /* Common IPv6 extension structure */ #cmakedefine HAVE_IP6_EXT + +/* String with host architecture (e.g., "linux-x86_64") */ +#define HOST_ARCHITECTURE "@HOST_ARCHITECTURE@" + +/* String with extension of dynamic libraries (e.g., ".so") */ +#define DYNAMIC_PLUGIN_SUFFIX "@CMAKE_SHARED_MODULE_SUFFIX@" + +/* True if we're building outside of the main Bro source code tree. */ +#ifndef BRO_PLUGIN_INTERNAL_BUILD +#define BRO_PLUGIN_INTERNAL_BUILD @BRO_PLUGIN_INTERNAL_BUILD@ +#endif diff --git a/configure b/configure index 35095c333a..5747586db8 100755 --- a/configure +++ b/configure @@ -39,8 +39,6 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --disable-auxtools don't build or install auxiliary tools --disable-perftools don't try to build with Google Perftools --disable-python don't try to build python bindings for broccoli - --disable-dataseries don't use the optional DataSeries log writer - --disable-elasticsearch don't use the optional ElasticSearch log writer Required Packages in Non-Standard Locations: --with-openssl=PATH path to OpenSSL install root @@ -62,9 +60,6 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --with-ruby-lib=PATH path to ruby library --with-ruby-inc=PATH path to ruby headers --with-swig=PATH path to SWIG executable - --with-dataseries=PATH path to DataSeries and Lintel libraries - --with-xml2=PATH path to libxml2 installation (for DataSeries) - --with-curl=PATH path to libcurl install root (for ElasticSearch) Packaging Options (for developers): --binary-package toggle special logic for binary packaging @@ -183,12 +178,6 @@ while [ $# -ne 0 ]; do --enable-ruby) append_cache_entry DISABLE_RUBY_BINDINGS BOOL false ;; - --disable-dataseries) - append_cache_entry DISABLE_DATASERIES BOOL true - ;; - --disable-elasticsearch) - append_cache_entry DISABLE_ELASTICSEARCH BOOL true - ;; --with-openssl=*) append_cache_entry OpenSSL_ROOT_DIR PATH $optarg ;; @@ -243,16 +232,6 @@ while [ $# -ne 0 ]; do --with-swig=*) append_cache_entry SWIG_EXECUTABLE PATH $optarg ;; - --with-dataseries=*) - append_cache_entry DataSeries_ROOT_DIR PATH $optarg - append_cache_entry Lintel_ROOT_DIR PATH $optarg - ;; - --with-xml2=*) - append_cache_entry LibXML2_ROOT_DIR PATH $optarg - ;; - --with-curl=*) - append_cache_entry LibCURL_ROOT_DIR PATH $optarg - ;; --binary-package) append_cache_entry BINARY_PACKAGING_MODE BOOL true ;; diff --git a/doc/_templates/layout.html b/doc/_templates/layout.html index 2f8ea02aff..3df56a12ff 100644 --- a/doc/_templates/layout.html +++ b/doc/_templates/layout.html @@ -10,7 +10,7 @@ {% endblock %} {% block header %} - {% endblock %} @@ -108,6 +108,6 @@ {% endblock %} {% block footer %} - {% endblock %} diff --git a/doc/conf.py.in b/doc/conf.py.in index 9720d12ade..4faebed3b8 100644 --- a/doc/conf.py.in +++ b/doc/conf.py.in @@ -21,7 +21,7 @@ sys.path.insert(0, os.path.abspath('sphinx_input/ext')) # ----- Begin of BTest configuration. ----- btest = os.path.abspath("@CMAKE_SOURCE_DIR@/aux/btest") -brocut = os.path.abspath("@CMAKE_SOURCE_DIR@/aux/bro-aux/bro-cut") +brocut = os.path.abspath("@CMAKE_SOURCE_DIR@/build/aux/bro-aux/bro-cut") bro = os.path.abspath("@CMAKE_SOURCE_DIR@/build/src") os.environ["PATH"] += (":%s:%s/sphinx:%s:%s" % (btest, btest, bro, brocut)) diff --git a/doc/devel/plugins.rst b/doc/devel/plugins.rst new file mode 100644 index 0000000000..76f5c75a68 --- /dev/null +++ b/doc/devel/plugins.rst @@ -0,0 +1,438 @@ + +=================== +Writing Bro Plugins +=================== + +Bro is internally moving to a plugin structure that enables extending +the system dynamically, without modifying the core code base. That way +custom code remains self-contained and can be maintained, compiled, +and installed independently. Currently, plugins can add the following +functionality to Bro: + + - Bro scripts. + + - Builtin functions/events/types for the scripting language. + + - Protocol analyzers. + + - File analyzers. + + - Packet sources and packet dumpers. TODO: Not yet. + + - Logging framework backends. TODO: Not yet. + + - Input framework readers. TODO: Not yet. + +A plugin's functionality is available to the user just as if Bro had +the corresponding code built-in. Indeed, internally many of Bro's +pieces are structured as plugins as well, they are just statically +compiled into the binary rather than loaded dynamically at runtime. + +Quick Start +=========== + +Writing a basic plugin is quite straight-forward as long as one +follows a few conventions. In the following we walk a simple example +plugin that adds a new built-in function (bif) to Bro: we'll add +``rot13(s: string) : string``, a function that rotates every character +in a string by 13 places. + +Generally, a plugin comes in the form of a directory following a +certain structure. To get started, Bro's distribution provides a +helper script ``aux/bro-aux/plugin-support/init-plugin`` that creates +a skeleton plugin that can then be customized. Let's use that:: + + # mkdir rot13-plugin + # cd rot13-plugin + # init-plugin Demo Rot13 + +As you can see the script takes two arguments. The first is a +namespace the plugin will live in, and the second a descriptive name +for the plugin itself. Bro uses the combination of the two to identify +a plugin. The namespace serves to avoid naming conflicts between +plugins written by independent developers; pick, e.g., the name of +your organisation. The namespace ``Bro`` is reserved for functionality +distributed by the Bro Project. In our example, the plugin will be +called ``Demo::Rot13``. + +The ``init-plugin`` script puts a number of files in place. The full +layout is described later. For now, all we need is +``src/rot13.bif``. It's initially empty, but we'll add our new bif +there as follows:: + + # cat src/rot13.bif + module CaesarCipher; + + function rot13%(s: string%) : string + %{ + char* rot13 = copy_string(s->CheckString()); + + for ( char* p = rot13; *p; p++ ) + { + char b = islower(*p) ? 'a' : 'A'; + *p = (*p - b + 13) % 26 + b; + } + + BroString* bs = new BroString(1, reinterpret_cast(rot13), + strlen(rot13)); + return new StringVal(bs); + %} + +The syntax of this file is just like any other ``*.bif`` file; we +won't go into it here. + +Now we can already compile our plugin, we just need to tell the +configure script put in place by ``init-plugin`` where the Bro source +tree is located (Bro needs to have been built there first):: + + # ./configure --bro-dist=/path/to/bro/dist && make + [... cmake output ...] + +Now our ``rot13-plugin`` directory has everything that it needs +for Bro to recognize it as a dynamic plugin. Once we point Bro to it, +it will pull it in automatically, as we can check with the ``-N`` +option:: + + # export BRO_PLUGIN_PATH=/path/to/rot13-plugin + # bro -N + [...] + Plugin: Demo::Rot13 - (dynamic, version 1) + [...] + +That looks quite good, except for the dummy description that we should +replace with something nicer so that users will know what our plugin +is about. We do this by editing the ``config.description`` line in +``src/Plugin.cc``, like this:: + + [...] + plugin::Configuration Configure() + { + plugin::Configuration config; + config.name = "Demo::Rot13"; + config.description = "Caesar cipher rotating a string's characters by 13 places."; + config.version.major = 1; + config.version.minor = 0; + return config; + } + [...] + + # make + [...] + # bro -N | grep Rot13 + Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) + +Better. Bro can also show us what exactly the plugin provides with the +more verbose option ``-NN``:: + + # bro -NN + [...] + Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) + [Function] CaesarCipher::rot13 + [...] + +There's our function. Now let's use it:: + + # bro -e 'print CaesarCipher::rot13("Hello")' + Uryyb + +It works. We next install the plugin along with Bro itself, so that it +will find it directly without needing the ``BRO_PLUGIN_PATH`` +environment variable. If we first unset the variable, the function +will no longer be available:: + + # unset BRO_PLUGIN_PATH + # bro -e 'print CaesarCipher::rot13("Hello")' + error in , line 1: unknown identifier CaesarCipher::rot13, at or near "CaesarCipher::rot13" + +Once we install it, it works again:: + + # make install + # bro -e 'print CaesarCipher::rot13("Hello")' + Uryyb + +The installed version went into +``/lib/bro/plugins/Demo_Rot13``. + +We can distribute the plugin in either source or binary form by using +the Makefile's ``sdist`` and ``bdist`` target, respectively. Both +create corrsponding tarballs:: + + # make sdist + [...] + Source distribution in build/sdist/Demo_Rot13.tar.gz + + # make bdist + [...] + Binary distribution in build/Demo_Rot13-darwin-x86_64.tar.gz + +The source archive will contain everything in the plugin directory +except any generated files. The binary archive will contain anything +needed to install and run the plugin, i.e., just what ``make install`` +puts into place as well. As the binary distribution is +platform-dependent, its name includes the OS and architecture the +plugin was built on. + +Plugin Directory Layout +======================= + +A plugin's directory needs to follow a set of conventions so that Bro +(1) recognizes it as a plugin, and (2) knows what to load. While +``init-plugin`` takes care of most of this, the following is the full +story. We'll use ```` to represent a plugin's top-level +directory. + +``/__bro_plugin__`` + A file that marks a directory as containing a Bro plugin. The file + must exist, and its content must consist of a single line with the + qualified name of the plugin (e.g., "Demo::Rot13"). + +``/lib/--.so`` + The shared library containing the plugin's compiled code. Bro will + load this in dynamically at run-time if OS and architecture match + the current platform. + +``scripts/`` + A directory with the plugin's custom Bro scripts. When the plugin + gets activated, this directory will be automatically added to + ``BROPATH``, so that any scripts/modules inside can be + "@load"ed. + +``scripts``/__load__.bro + A Bro script that will be loaded immediately when the plugin gets + activated. See below for more information on activating plugins. + +``lib/bif/`` + Directory with auto-generated Bro scripts that declare the plugin's + bif elements. The files here are produced by ``bifcl``. + +By convention, a plugin should put its custom scripts into sub folders +of ``scripts/``, i.e., ``scripts//