diff --git a/.gitmodules b/.gitmodules index 4998cc6b80..24375ce23d 100644 --- a/.gitmodules +++ b/.gitmodules @@ -19,3 +19,6 @@ [submodule "src/3rdparty"] path = src/3rdparty url = git://git.bro.org/bro-3rdparty +[submodule "aux/plugins"] + path = aux/plugins + url = git://git.bro.org/bro-plugins diff --git a/CHANGES b/CHANGES index 6746b0e86a..60f6c515e2 100644 --- a/CHANGES +++ b/CHANGES @@ -1,4 +1,392 @@ +2.3-259 | 2014-10-27 10:04:04 -0500 + + * Documentation fixes. (Vicente Jimenez Aguilar and Stefano Azzalini) + +2.3-256 | 2014-10-24 15:33:45 -0700 + + * Adding missing test baseline. (Robin Sommer) + +2.3-255 | 2014-10-24 13:39:44 -0700 + + * Fixing unstable active-http test. (Robin Sommer) + +2.3-254 | 2014-10-24 11:40:51 -0700 + + * Fix active-http.bro to deal reliably with empty server responses, + which will now be passed back as empty files. (Christian Struck) + +2.3-248 | 2014-10-23 14:20:59 -0700 + + * Change order in which a plugin's scripts are loaded at startup. + (Robin Sommer) + +2.3-247 | 2014-10-21 13:42:38 -0700 + + * Updates to the SSL analyzer. (Johanna Amann) + + * Mark everything below 2048 bit as a weak key. + + * Fix notice suppression. + + * Add information about server-chosen protocol to ssl.log, if + provided by application_layer_next_protocol. + + * Add boolean flag to ssl.log signaling if a session was + resumed. Remove the (usually not really that useful) session + ID that the client sent. + +2.3-240 | 2014-10-21 13:36:33 -0700 + + * Fix Coverity-reported issues in DNP3 analyzer. (Seth Hall) + +2.3-238 | 2014-10-16 06:51:49 -0700 + + * Fix multipart HTTP/MIME entity file analysis so that (1) singular + CR or LF characters in multipart body content are no longer + converted to a full CRLF (thus corrupting the file) and (2) it + also no longer considers the CRLF before the multipart boundary as + part of the content. Addresses BIT-1235. (Jon Siwek) + +2.3-235 | 2014-10-15 10:20:47 -0500 + + * BIT-1273: Add error message for bad enum declaration syntax. + (Jon Siwek) + +2.3-234 | 2014-10-14 14:42:09 -0500 + + * Documentation fixes. (Steve Smoot) + +2.3-233 | 2014-10-09 16:00:27 -0500 + + * Change find-bro-logs unit test to follow symlinks. (Jon Siwek) + + * Add error checks and messages to a test script (Daniel Thayer) + +2.3-230 | 2014-10-08 08:15:17 -0700 + + * Further baseline normalization for plugin test portability. (Robin + Sommer) + +2.3-229 | 2014-10-07 20:18:11 -0700 + + * Fix for test portability. (Robin Sommer) + +2.3-228 | 2014-10-07 15:32:37 -0700 + + * Include plugin unit tests into the top-level btest configuration. (Robin Sommer) + + * Switching the prefix separator for packet source/dumper plugins + once more, now to "::". Addresses BIT-1267. (Robin Sommer) + + * Fix for allowing a packet source/dumper plugin to support multiple + prefixes with a colon. (Robin Sommer) + +2.3-225 | 2014-10-07 15:13:35 -0700 + + * Updating plugin documentation. (Robin Sommer) + +2.3-224 | 2014-10-07 14:32:17 -0700 + + * Improved the log file reference documentation. (Jeannette Dopheide + and Daniel Thayer) + + * Improves shockwave flash file signatures. (Seth Hall) + + - This moves the signatures out of the libmagic imported signatures + and into our own general.sig. + + - Expand the detection to LZMA compressed flash files. + + * Add new script language reference documentation on operators, + statements, and directives. Also improved the documentation on + types and attributes by splitting them into two docs, and + providing more examples and adding a chart on the top of each page + with links to each type and attribute for easier access to the + information. (Daniel Thayer) + + * Split the types and attributes reference doc into two docs. + (Daniel Thayer) + +2.3-208 | 2014-10-03 09:38:52 -0500 + + * BIT-1268: Fix uninitialized router_list argument in + dhcp_offer/dhcp_ack. (Jon Siwek) + +2.3-207 | 2014-10-02 16:39:17 -0700 + + * Updating plugin docs. (Robin Sommer) + + * Fix packet sources being treated as idle when a packet is + available. Addresses BIT-1266. (Jon Siwek) + + * Fix regression causing the main loop to spin more frequently. + Addresses BIT-1266. (Jon Siwek) + +2.3-203 | 2014-09-29 20:06:54 -0700 + + * Fix to use length parameter in DNP3 time conversion correctly now. + (Robin Sommer) + +2.3-202 | 2014-09-29 17:05:18 -0700 + + * New SSL extension type from IANA and a few other SSL const + changes. (Johanna Amann) + + * Make unexpected pipe errors fatal as precaution. Addresses + BIT-1260. (Jon Siwek) + + * Adding a function for DNP3 to translate the timestamp format. (Hui + Lin) + +2.3-197 | 2014-09-29 10:42:01 -0500 + + * Fix possible seg fault in TCP reassembler. (Jon Siwek) + +2.3-196 | 2014-09-25 17:53:27 -0700 + + * Changing prefix for packet sources/dumper from ':' to '%'. + Addresses BIT-1249. (Robin Sommer) + + * Remove timeouts from remote communication loop. The select() now + blocks until there's work to do instead of relying on a small + timeout value which can cause unproductive use of cpu cycles. (Jon + Siwek) + + * Improve error message when failing to activate a plugin. Also fix + a unit test helper script that checks plugin availability. (Jon + Siwek) + +2.3-183 | 2014-09-24 10:08:04 -0500 + + * Add a "node" field to Intel::Seen struture and intel.log to + indicate which node discovered a hit on an intel item. (Seth Hall) + + * BIT-1261: Fixes to plugin quick start doc. (Jon Siwek) + +2.3-180 | 2014-09-22 12:52:41 -0500 + + * BIT-1259: Fix issue w/ duplicate TCP reassembly deliveries. + (Jon Siwek) + +2.3-178 | 2014-09-18 14:29:46 -0500 + + * BIT-1256: Fix file analysis events from coming after bro_done(). + (Jon Siwek) + +2.3-177 | 2014-09-17 09:41:27 -0500 + + * Documentation fixes. (Chris Mavrakis) + +2.3-174 | 2014-09-17 09:37:09 -0500 + + * Fixed some "make doc" warnings caused by reST formatting + (Daniel Thayer). + +2.3-172 | 2014-09-15 13:38:52 -0500 + + * Remove unneeded allocations for HTTP messages. (Jon Siwek) + +2.3-171 | 2014-09-15 11:14:57 -0500 + + * Fix a compile error on systems without pcap-int.h. (Jon Siwek) + +2.3-170 | 2014-09-12 19:28:01 -0700 + + * Fix incorrect data delivery skips after gap in HTTP Content-Range. + Addresses BIT-1247. (Jon Siwek) + + * Fix file analysis placement of data after gap in HTTP + Content-Range. Addresses BIT-1248. (Jon Siwek) + + * Fix issue w/ TCP reassembler not delivering some segments. + Addresses BIT-1246. (Jon Siwek) + + * Fix MIME entity file data/gap ordering and raise http_entity_data + in line with data arrival. Addresses BIT-1240. (Jon Siwek) + + * Implement file ID caching for MIME_Mail. (Jon Siwek) + + * Fix a compile error. (Jon Siwek) + +2.3-161 | 2014-09-09 12:35:38 -0500 + + * Bugfixes and test updates/additions. (Robin Sommer) + + * Interface tweaks and docs for PktSrc/PktDumper. (Robin Sommer) + + * Moving PCAP-related bifs to iosource/pcap.bif. (Robin Sommer) + + * Moving some of the BPF filtering code into base class. + This will allow packet sources that don't support BPF natively to + emulate the filtering via libpcap. (Robin Sommer) + + * Removing FlowSrc. (Robin Sommer) + + * Removing remaining pieces of the 2ndary path, and left-over + files of packet sorter. (Robin Sommer) + + * A bunch of infrastructure work to move IOSource, IOSourceRegistry + (now iosource::Manager) and PktSrc/PktDumper code into iosource/, + and over to a plugin structure. (Robin Sommer) + +2.3-137 | 2014-09-08 19:01:13 -0500 + + * Fix Broxygen's rendering of opaque types. (Jon Siwek) + +2.3-136 | 2014-09-07 20:50:46 -0700 + + * Change more http links to https. (Johanna Amann) + +2.3-134 | 2014-09-04 16:16:36 -0700 + + * Fixed a number of issues with OCSP reply validation. Addresses + BIT-1212. (Johanna Amann) + + * Fix null pointer dereference in OCSP verification code in case no + certificate is sent as part as the ocsp reply. Addresses BIT-1212. + (Johanna Amann) + +2.3-131 | 2014-09-04 16:10:32 -0700 + + * Make links in documentation templates protocol relative. (Johanna + Amann) + +2.3-129 | 2014-09-02 17:21:21 -0700 + + * Simplify a conditional with equivalent branches. (Jon Siwek) + + * Change EDNS parsing code to use rdlength more cautiously. (Jon + Siwek) + + * Fix a memory leak when bind() fails due to EADDRINUSE. (Jon Siwek) + + * Fix possible buffer over-read in DNS TSIG parsing. (Jon Siwek) + +2.3-124 | 2014-08-26 09:24:19 -0500 + + * Better documentation for sub_bytes (Jimmy Jones) + + * BIT-1234: Fix build on systems that already have ntohll/htonll + (Jon Siwek) + +2.3-121 | 2014-08-22 15:22:15 -0700 + + * Detect functions that try to bind variables from an outer scope + and raise an error saying that's not supported. Addresses + BIT-1233. (Jon Siwek) + +2.3-116 | 2014-08-21 16:04:13 -0500 + + * Adding plugin testing to Makefile's test-all. (Robin Sommer) + + * Converting log writers and input readers to plugins. + DataSeries and ElasticSearch plugins have moved to the new + bro-plugins repository, which is now a git submodule in the + aux/plugins directory. (Robin Sommer) + +2.3-98 | 2014-08-19 11:03:46 -0500 + + * Silence some doc-related warnings when using `bro -e`. + Closes BIT-1232. (Jon Siwek) + + * Fix possible null ptr derefs reported by Coverity. (Jon Siwek) + +2.3-96 | 2014-08-01 14:35:01 -0700 + + * Small change to DHCP documentation. In server->client messages the + host name may differ from the one requested by the client. + (Johanna Amann) + + * Split DHCP log writing from record creation. This allows users to + customize dhcp.log by changing the record in their own dhcp_ack + event. (Johanna Amann) + + * Update PATH so that documentation btests can find bro-cut. (Daniel + Thayer) + + * Remove gawk from list of optional packages in documentation. + (Daniel Thayer) + + * Fix for redefining built-in constants. (Robin Sommer) + +2.3-86 | 2014-07-31 14:19:58 -0700 + + * Fix for redefining built-in constants. (Robin Sommer) + + * Adding missing check that a plugin's API version matches what Bro + defines. (Robin Sommer) + + * Adding NEWS entry for plugins. (Robin Sommer) + +2.3-83 | 2014-07-30 16:26:11 -0500 + + * Minor adjustments to plugin code/docs. (Jon Siwek) + + * Dynamic plugin support. (Rpbin Sommer) + + Bro now supports extending core functionality, like protocol and + file analysis, dynamically with external plugins in the form of + shared libraries. See doc/devel/plugins.rst for an overview of the + main functionality. Changes coming with this: + + - Replacing the old Plugin macro magic with a new API. + + - The plugin API changed to generally use std::strings instead + of const char*. + + - There are a number of invocations of PLUGIN_HOOK_ + {VOID,WITH_RESULT} across the code base, which allow plugins + to hook into the processing at those locations. + + - A few new accessor methods to various classes to allow + plugins to get to that information. + + - network_time cannot be just assigned to anymore, there's now + function net_update_time() for that. + + - Redoing how builtin variables are initialized, so that it + works for plugins as well. No more init_net_var(), but + instead bifcl-generated code that registers them. + + - Various changes for adjusting to the now dynamic generation + of analyzer instances. + + - same_type() gets an optional extra argument allowing record type + comparision to ignore if field names don't match. (Robin Sommer) + + - Further unify file analysis API with the protocol analyzer API + (assigning IDs to analyzers; adding Init()/Done() methods; + adding subtypes). (Robin Sommer) + + - A new command line option -Q that prints some basic execution + time stats. (Robin Sommer) + + - Add support to the file analysis for activating analyzers by + MIME type. (Robin Sommer) + + - File::register_for_mime_type(tag: Analyzer::Tag, mt: + string): Associates a file analyzer with a MIME type. + + - File::add_analyzers_for_mime_type(f: fa_file, mtype: + string): Activates all analyzers registered for a MIME + type for the file. + + - The default file_new() handler calls + File::add_analyzers_for_mime_type() with the file's MIME + type. + +2.3-20 | 2014-07-22 17:41:02 -0700 + + * Updating submodule(s). + +2.3-19 | 2014-07-22 17:29:19 -0700 + + * Implement bytestring_to_coils() in Modbus analyzer so that coils + gets passed to the corresponding events. (Hui Lin) + + * Add length field to ModbusHeaders. (Hui Lin) + 2.3-12 | 2014-07-10 19:17:37 -0500 * Include yield of vectors in Broxygen's type descriptions. @@ -12,7 +400,7 @@ 2.3-7 | 2014-06-26 17:35:18 -0700 * Extending "make test-all" to include aux/bro-aux. (Robin Sommer) - + 2.3-6 | 2014-06-26 17:24:10 -0700 * DataSeries compilation issue fixed. (mlaterman) @@ -135,7 +523,7 @@ and the load balancing FAQ on the website. (Daniel Thayer) * Update some doc tests and line numbers (Daniel Thayer) - + 2.2-457 | 2014-05-16 14:38:31 -0700 * New script policy/protocols/ssl/validate-ocsp.bro that adds OSCP @@ -150,7 +538,7 @@ Amann) * Improved Heartbleed attack/scan detection. (Bernhard Amann) - + * Let TLS analyzer fail better when no longer in sync with the data stream. (Bernhard Amann) @@ -187,12 +575,12 @@ *Undelivered methods now use a uint64 in place of an int for the relative sequence space offset parameter. - Addresses BIT-348. + Addresses BIT-348. * Fixing compiler warnings. (Robin Sommer) - + * Update SNMP analyzer's DeliverPacket method signature. (Jon Siwek) - + 2.2-417 | 2014-05-07 10:59:22 -0500 * Change handling of atypical OpenSSL error case in x509 verification. (Jon Siwek) @@ -232,7 +620,7 @@ 2.2-397 | 2014-05-01 20:29:20 -0700 * Fix reference counting for lookup_ID() usages. (Jon Siwek) - + 2.2-395 | 2014-05-01 20:25:48 -0700 * Fix missing "irc-dcc-data" service field from IRC DCC connections. @@ -245,7 +633,7 @@ Siwek) * Improve file analysis manager shutdown/cleanup. (Jon Siwek) - + 2.2-388 | 2014-04-24 18:38:07 -0700 * Fix decoding of MIME quoted-printable. (Mareq) @@ -258,11 +646,11 @@ 2.2-381 | 2014-04-24 17:08:45 -0700 * Add Java version to software framework. (Brian Little) - + 2.2-379 | 2014-04-24 17:06:21 -0700 * Remove unused Val::attribs member. (Jon Siwek) - + 2.2-377 | 2014-04-24 16:57:54 -0700 * A larger set of SSL improvements and extensions. Addresses @@ -275,7 +663,7 @@ server_name, alpn, and ec-curves. - Adds support for the heartbeat events. - + - Add Heartbleed detector script. - Adds basic support for OCSP stapling. @@ -286,7 +674,7 @@ 2.2-353 | 2014-04-24 16:12:30 -0700 * Adapt HTTP partial content to cache file analysis IDs. (Jon Siwek) - + * Adapt SSL analyzer to generate file analysis handles itself. (Jon Siwek) @@ -302,11 +690,11 @@ Siwek) * Refactor file analysis file ID lookup. (Jon Siwek) - + 2.2-344 | 2014-04-22 20:13:30 -0700 * Refactor various hex escaping code. (Jon Siwek) - + 2.2-341 | 2014-04-17 18:01:41 -0500 * Fix duplicate DNS log entries. (Robin Sommer) @@ -314,9 +702,9 @@ 2.2-341 | 2014-04-17 18:01:01 -0500 * Refactor initialization of ASCII log writer options. (Jon Siwek) - + * Fix a memory leak in ASCII log writer. (Jon Siwek) - + 2.2-338 | 2014-04-17 17:48:17 -0500 * Disable input/logging threads setting their names on every @@ -328,7 +716,7 @@ 2.2-335 | 2014-04-10 15:04:57 -0700 * Small logic fix for main SSL script. (Bernhard Amann) - + * Update DPD signatures for detecting TLS 1.2. (Bernhard Amann) * Remove unused data member of SMTP_Analyzer to silence a Coverity @@ -352,7 +740,7 @@ 2.2-317 | 2014-04-03 10:51:31 -0400 - * Add a uid field to the signatures.log. Addresses BIT-1171 + * Add a uid field to the signatures.log. Addresses BIT-1171 (Anthony Verez) 2.2-315 | 2014-04-01 16:50:01 -0700 @@ -363,7 +751,7 @@ 2.2-313 | 2014-04-01 16:40:19 -0700 * Fix a couple nits reported by Coverity.(Jon Siwek) - + * Fix potential memory leak in IP frag reassembly reported by Coverity. (Jon Siwek) @@ -453,14 +841,14 @@ 2.2-271 | 2014-03-30 20:25:17 +0200 * Add unit tests covering vector/set/table ctors/inits. (Jon Siwek) - + * Fix parsing of "local" named table constructors. (Jon Siwek) * Improve type checking of records. Addresses BIT-1159. (Jon Siwek) - + 2.2-267 | 2014-03-30 20:21:43 +0200 - * Improve documentation of Bro clusters. Addresses BIT-1160. + * Improve documentation of Bro clusters. Addresses BIT-1160. (Daniel Thayer) 2.2-263 | 2014-03-30 20:19:05 +0200 @@ -487,7 +875,7 @@ 2.2-254 | 2014-03-30 19:55:22 +0200 * Update instructions on how to build Bro docs. (Daniel Thayer) - + 2.2-251 | 2014-03-28 08:37:37 -0400 * Quick fix to the ElasticSearch writer. (Seth Hall) @@ -517,7 +905,7 @@ formatter work. * Fixing compiler error. (Robin Sommer) - + * Fixing (very unlikely) double delete in HTTP analyzer when decapsulating CONNECTs. (Robin Sommer) @@ -543,7 +931,7 @@ Addresses BIT-1134. (Jon Siwek) * Enable fake DNS mode for test suites. - + * Improve analysis of TCP SYN/SYN-ACK reversal situations. (Jon Siwek) @@ -575,7 +963,7 @@ (Jon Siwek) * Silences some documentation warnings from Sphinx. (Jon Siwek) - + 2.2-215 | 2014-03-10 11:10:15 -0700 * Fix non-deterministic logging of unmatched DNS msgs. Addresses @@ -613,7 +1001,7 @@ HTTP traffic. (Seth Hall) * Fixing removal of support analyzers. (Robin Sommer) - + 2.2-199 | 2014-03-03 16:34:20 -0800 * Allow iterating over bif functions with result type vector of any. @@ -628,11 +1016,11 @@ 2.2-194 | 2014-02-28 14:50:53 -0800 * Remove packet sorter. Addresses BIT-700. (Bernhard Amann) - + 2.2-192 | 2014-02-28 09:46:43 -0800 * Update Mozilla root bundle. (Bernhard Amann) - + 2.2-190 | 2014-02-27 07:34:44 -0800 * Adjust timings of a few leak tests. (Bernhard Amann) @@ -662,7 +1050,7 @@ 2.2-177 | 2014-02-20 17:27:46 -0800 * Update to libmagic version 5.17. Addresses BIT-1136. (Jon Siwek) - + 2.2-174 | 2014-02-14 12:07:04 -0800 * Support for MPLS over VLAN. (Chris Kanich) @@ -691,7 +1079,7 @@ defined, but we see it being actively used. (Bernhard Amann) * Test baseline updates for DNS change. (Robin Sommer) - + 2.2-158 | 2014-02-09 23:45:39 -0500 * Change dns.log to include only standard DNS queries. (Jon Siwek) @@ -767,7 +1155,7 @@ 2.2-128 | 2014-01-30 15:58:47 -0800 * Add leak test for Exec module. (Bernhard Amann) - + * Fix file_over_new_connection event to trigger when entire file is missed. (Jon Siwek) @@ -780,7 +1168,7 @@ "detect_filtered_trace". (Jon Siwek) * Improve TCP FIN retransmission handling. (Jon Siwek) - + 2.2-120 | 2014-01-28 10:25:23 -0800 * Fix and extend x509_extension() event, which now actually returns @@ -798,10 +1186,10 @@ 2.2-115 | 2014-01-22 12:11:18 -0800 * Add unit tests for new Bro Manual docs. (Jon Siwek) - + * New content for the "Using Bro" section of the manual. (Rafael Bonilla/Jon Siwek) - + 2.2-105 | 2014-01-20 12:16:48 -0800 * Support GRE tunnel decapsulation, including enhanced GRE headers. @@ -810,7 +1198,7 @@ Addresses BIT-867. (Jon Siwek) * Simplify FragReassembler memory management. (Jon Siwek) - + 2.2-102 | 2014-01-20 12:00:29 -0800 * Include file information (MIME type and description) into notice @@ -819,7 +1207,7 @@ 2.2-100 | 2014-01-20 11:54:58 -0800 * Fix caching of recently validated SSL certifcates. (Justin Azoff) - + 2.2-98 | 2014-01-20 11:50:32 -0800 * For notice suppresion, instead of storing the entire notice in @@ -847,7 +1235,7 @@ the city database instead of just the former. (Jon Siwek) * Broxygen init fixes. Addresses BIT-1110. (Jon Siwek) - + - Don't check mtime of bro binary if BRO_DISABLE_BROXYGEN env var set. - Fix failure to locate bro binary if invoking from a relative @@ -860,7 +1248,7 @@ len field being set to zero. (Seth Hall) * Canonify output of a unit test. (Jon Siwek) - + * A set of documentation updates. (Daniel Thayer) - Fix typo in Bro 2.2 NEWS on string indexing. @@ -907,9 +1295,9 @@ (Jon Siwek) * Close signature files after done parsing. (Jon Siwek) - + * Fix unlikely null ptr deref in broxygen::Manager. (Jon Siwek) - + * FreeBSD build fix addendum: unintended variable shadowing. (Jon Siwek) @@ -928,7 +1316,7 @@ were cleaned up. Addresses BIT-1103. (Bernhard Amann) * Minor Broxygen improvements. Addresses BIT-1098. (Jon Siwek) - + 2.2-51 | 2013-12-05 07:53:37 -0800 * Improve a unit test involving 'when' conditionals. (Jon Siwek) @@ -953,7 +1341,7 @@ 2.2-44 | 2013-12-04 12:41:51 -0800 * Fix string slice notation. Addresses BIT-1097. (Jon Siwek) - + Slice ranges were not correctly determined for negative indices and also off by one in general (included one more element at the end of the substring than what actually matched the index range). @@ -1008,7 +1396,7 @@ 2.2-11 | 2013-12-03 10:56:28 -0800 * Unit test for broccoli vector support. (Jon Siwek) - + * Changed ordering of Bro type tag enum, which was out of sync. (Jon Siwek) @@ -1020,14 +1408,14 @@ 2.2-6 | 2013-11-15 07:05:15 -0800 * Make "install-example-configs" target use DESTDIR. (Jon Siwek) - + 2.2-5 | 2013-11-11 13:47:54 -0800 * Fix the irc_reply event for certain server message types. (Seth Hall) * Fixed Segmentation fault in SQLite Writer. (Jon Crussell) - + 2.2 | 2013-11-07 10:25:50 -0800 * Release 2.2. diff --git a/CMakeLists.txt b/CMakeLists.txt index 87b99271c5..22d63a89d5 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,5 +1,9 @@ project(Bro C CXX) + +# When changing the minimum version here, also adapt +# aux/bro-aux/plugin-support/skeleton/CMakeLists.txt cmake_minimum_required(VERSION 2.6.3 FATAL_ERROR) + include(cmake/CommonCMakeConfig.cmake) ######################################################################## @@ -16,12 +20,18 @@ endif () get_filename_component(BRO_SCRIPT_INSTALL_PATH ${BRO_SCRIPT_INSTALL_PATH} ABSOLUTE) +set(BRO_PLUGIN_INSTALL_PATH ${BRO_ROOT_DIR}/lib/bro/plugins CACHE STRING "Installation path for plugins" FORCE) + configure_file(bro-path-dev.in ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev) + file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.sh "export BROPATH=`${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" + "export BRO_PLUGIN_PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src:${BRO_PLUGIN_INSTALL_PATH}\"\n" "export PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n") + file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.csh "setenv BROPATH `${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" + "setenv BRO_PLUGIN_PATH \"${CMAKE_CURRENT_BINARY_DIR}/src:${BRO_PLUGIN_INSTALL_PATH}\"\n" "setenv PATH \"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n") file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/VERSION" VERSION LIMIT_COUNT 1) @@ -117,33 +127,6 @@ if (GOOGLEPERFTOOLS_FOUND) endif () endif () -set(USE_DATASERIES false) -find_package(Lintel) -find_package(DataSeries) -find_package(LibXML2) - -if (NOT DISABLE_DATASERIES AND - LINTEL_FOUND AND DATASERIES_FOUND AND LIBXML2_FOUND) - set(USE_DATASERIES true) - include_directories(BEFORE ${Lintel_INCLUDE_DIR}) - include_directories(BEFORE ${DataSeries_INCLUDE_DIR}) - include_directories(BEFORE ${LibXML2_INCLUDE_DIR}) - list(APPEND OPTLIBS ${Lintel_LIBRARIES}) - list(APPEND OPTLIBS ${DataSeries_LIBRARIES}) - list(APPEND OPTLIBS ${LibXML2_LIBRARIES}) -endif() - -set(USE_ELASTICSEARCH false) -set(USE_CURL false) -find_package(LibCURL) - -if (NOT DISABLE_ELASTICSEARCH AND LIBCURL_FOUND) - set(USE_ELASTICSEARCH true) - set(USE_CURL true) - include_directories(BEFORE ${LibCURL_INCLUDE_DIR}) - list(APPEND OPTLIBS ${LibCURL_LIBRARIES}) -endif() - if (ENABLE_PERFTOOLS_DEBUG OR ENABLE_PERFTOOLS) # Just a no op to prevent CMake from complaining about manually-specified # ENABLE_PERFTOOLS_DEBUG or ENABLE_PERFTOOLS not being used if google @@ -165,6 +148,8 @@ set(brodeps include(TestBigEndian) test_big_endian(WORDS_BIGENDIAN) +include(CheckSymbolExists) +check_symbol_exists(htonll arpa/inet.h HAVE_BYTEORDER_64) include(OSSpecific) include(CheckTypes) @@ -174,6 +159,10 @@ include(MiscTests) include(PCAPTests) include(OpenSSLTests) include(CheckNameserCompat) +include(GetArchitecture) + +# Tell the plugin code that we're building as part of the main tree. +set(BRO_PLUGIN_INTERNAL_BUILD true CACHE INTERNAL "" FORCE) configure_file(${CMAKE_CURRENT_SOURCE_DIR}/config.h.in ${CMAKE_CURRENT_BINARY_DIR}/config.h) @@ -238,10 +227,6 @@ message( "\n tcmalloc: ${USE_PERFTOOLS_TCMALLOC}" "\n debugging: ${USE_PERFTOOLS_DEBUG}" "\njemalloc: ${ENABLE_JEMALLOC}" - "\ncURL: ${USE_CURL}" - "\n" - "\nDataSeries: ${USE_DATASERIES}" - "\nElasticSearch: ${USE_ELASTICSEARCH}" "\n" "\n================================================================\n" ) diff --git a/Makefile b/Makefile index 2b8e66503b..49d9a6173c 100644 --- a/Makefile +++ b/Makefile @@ -56,6 +56,7 @@ test-all: test test -d aux/broctl && ( cd aux/broctl && make test ) test -d aux/btest && ( cd aux/btest && make test ) test -d aux/bro-aux && ( cd aux/bro-aux && make test ) + test -d aux/plugins && ( cd aux/plugins && make test-all ) configured: @test -d $(BUILD) || ( echo "Error: No build/ directory found. Did you run configure?" && exit 1 ) diff --git a/NEWS b/NEWS index b6754e1389..6da13833c3 100644 --- a/NEWS +++ b/NEWS @@ -4,6 +4,32 @@ release. For an exhaustive list of changes, see the ``CHANGES`` file (note that submodules, such as BroControl and Broccoli, come with their own ``CHANGES``.) +Bro 2.4 (in progress) +===================== + +Dependencies +------------ + +New Functionality +----------------- + +- Bro now has support for external plugins that can extend its core + functionality, like protocol/file analysis, via shared libraries. + Plugins can be developed and distributed externally, and will be + pulled in dynamically at startup. Currently, a plugin can provide + custom protocol analyzers, file analyzers, log writers[TODO], input + readers[TODO], packet sources[TODO], and new built-in functions. A + plugin can furthermore hook into Bro's processing a number of places + to add custom logic. + + See https://www.bro.org/sphinx-git/devel/plugins.html for more + information on writing plugins. + +Changed Functionality +--------------------- + +- bro-cut has been rewritten in C, and is hence much faster. + Bro 2.3 ======= diff --git a/VERSION b/VERSION index e6cb320816..2ca780950f 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.3-12 +2.3-259 diff --git a/aux/binpac b/aux/binpac index ec1e052afd..c8e017b4b1 160000 --- a/aux/binpac +++ b/aux/binpac @@ -1 +1 @@ -Subproject commit ec1e052afd5a8cd3d1d2cbb28fcd688018e379a5 +Subproject commit c8e017b4b1893cf254fc2bc8eedd86b852a2e654 diff --git a/aux/bro-aux b/aux/bro-aux index 31d011479a..977654dc51 160000 --- a/aux/bro-aux +++ b/aux/bro-aux @@ -1 +1 @@ -Subproject commit 31d011479a4e956e029d8b708446841a088dd7e3 +Subproject commit 977654dc51ab08a2afde32241f108cdb4a581d8f diff --git a/aux/broccoli b/aux/broccoli index 1ee129f715..acb8fbe8e7 160000 --- a/aux/broccoli +++ b/aux/broccoli @@ -1 +1 @@ -Subproject commit 1ee129f7159a2c32fe0cb0f44c9412486fb7a479 +Subproject commit acb8fbe8e7bc6ace5135fb73dca8e29432cdc1ca diff --git a/aux/broctl b/aux/broctl index 8a13886f32..2b13bfcc94 160000 --- a/aux/broctl +++ b/aux/broctl @@ -1 +1 @@ -Subproject commit 8a13886f322f3b618832c0ca3976e07f686d14da +Subproject commit 2b13bfcc941018c76f74b81a6e74e5e4e723c747 diff --git a/aux/btest b/aux/btest index 4da1bd2403..1efa4d10f9 160000 --- a/aux/btest +++ b/aux/btest @@ -1 +1 @@ -Subproject commit 4da1bd24038d4977e655f2b210f34e37f0b73b78 +Subproject commit 1efa4d10f943351efea96def68e598b053fd217a diff --git a/aux/plugins b/aux/plugins new file mode 160000 index 0000000000..ad600b5bdc --- /dev/null +++ b/aux/plugins @@ -0,0 +1 @@ +Subproject commit ad600b5bdcd56a2723e323c0f2c8e1708956ca4f diff --git a/cmake b/cmake index 0f301aa08a..1316c07f70 160000 --- a/cmake +++ b/cmake @@ -1 +1 @@ -Subproject commit 0f301aa08a970150195a2ea5b3ed43d2d98b35b3 +Subproject commit 1316c07f7059647b6c4a496ea36e4b83bb5d8f0f diff --git a/config.h.in b/config.h.in index 2d065f755e..755a9eee98 100644 --- a/config.h.in +++ b/config.h.in @@ -129,6 +129,9 @@ /* whether words are stored with the most significant byte first */ #cmakedefine WORDS_BIGENDIAN +/* whether htonll/ntohll is defined in */ +#cmakedefine HAVE_BYTEORDER_64 + /* ultrix can't hack const */ #cmakedefine NEED_ULTRIX_CONST_HACK #ifdef NEED_ULTRIX_CONST_HACK @@ -209,3 +212,14 @@ /* Common IPv6 extension structure */ #cmakedefine HAVE_IP6_EXT + +/* String with host architecture (e.g., "linux-x86_64") */ +#define HOST_ARCHITECTURE "@HOST_ARCHITECTURE@" + +/* String with extension of dynamic libraries (e.g., ".so") */ +#define DYNAMIC_PLUGIN_SUFFIX "@CMAKE_SHARED_MODULE_SUFFIX@" + +/* True if we're building outside of the main Bro source code tree. */ +#ifndef BRO_PLUGIN_INTERNAL_BUILD +#define BRO_PLUGIN_INTERNAL_BUILD @BRO_PLUGIN_INTERNAL_BUILD@ +#endif diff --git a/configure b/configure index 35095c333a..5747586db8 100755 --- a/configure +++ b/configure @@ -39,8 +39,6 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --disable-auxtools don't build or install auxiliary tools --disable-perftools don't try to build with Google Perftools --disable-python don't try to build python bindings for broccoli - --disable-dataseries don't use the optional DataSeries log writer - --disable-elasticsearch don't use the optional ElasticSearch log writer Required Packages in Non-Standard Locations: --with-openssl=PATH path to OpenSSL install root @@ -62,9 +60,6 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --with-ruby-lib=PATH path to ruby library --with-ruby-inc=PATH path to ruby headers --with-swig=PATH path to SWIG executable - --with-dataseries=PATH path to DataSeries and Lintel libraries - --with-xml2=PATH path to libxml2 installation (for DataSeries) - --with-curl=PATH path to libcurl install root (for ElasticSearch) Packaging Options (for developers): --binary-package toggle special logic for binary packaging @@ -183,12 +178,6 @@ while [ $# -ne 0 ]; do --enable-ruby) append_cache_entry DISABLE_RUBY_BINDINGS BOOL false ;; - --disable-dataseries) - append_cache_entry DISABLE_DATASERIES BOOL true - ;; - --disable-elasticsearch) - append_cache_entry DISABLE_ELASTICSEARCH BOOL true - ;; --with-openssl=*) append_cache_entry OpenSSL_ROOT_DIR PATH $optarg ;; @@ -243,16 +232,6 @@ while [ $# -ne 0 ]; do --with-swig=*) append_cache_entry SWIG_EXECUTABLE PATH $optarg ;; - --with-dataseries=*) - append_cache_entry DataSeries_ROOT_DIR PATH $optarg - append_cache_entry Lintel_ROOT_DIR PATH $optarg - ;; - --with-xml2=*) - append_cache_entry LibXML2_ROOT_DIR PATH $optarg - ;; - --with-curl=*) - append_cache_entry LibCURL_ROOT_DIR PATH $optarg - ;; --binary-package) append_cache_entry BINARY_PACKAGING_MODE BOOL true ;; diff --git a/doc/_templates/layout.html b/doc/_templates/layout.html index 2f8ea02aff..3df56a12ff 100644 --- a/doc/_templates/layout.html +++ b/doc/_templates/layout.html @@ -10,7 +10,7 @@ {% endblock %} {% block header %} - {% endblock %} @@ -108,6 +108,6 @@ {% endblock %} {% block footer %} - {% endblock %} diff --git a/doc/conf.py.in b/doc/conf.py.in index 9720d12ade..4faebed3b8 100644 --- a/doc/conf.py.in +++ b/doc/conf.py.in @@ -21,7 +21,7 @@ sys.path.insert(0, os.path.abspath('sphinx_input/ext')) # ----- Begin of BTest configuration. ----- btest = os.path.abspath("@CMAKE_SOURCE_DIR@/aux/btest") -brocut = os.path.abspath("@CMAKE_SOURCE_DIR@/aux/bro-aux/bro-cut") +brocut = os.path.abspath("@CMAKE_SOURCE_DIR@/build/aux/bro-aux/bro-cut") bro = os.path.abspath("@CMAKE_SOURCE_DIR@/build/src") os.environ["PATH"] += (":%s:%s/sphinx:%s:%s" % (btest, btest, bro, brocut)) diff --git a/doc/devel/plugins.rst b/doc/devel/plugins.rst new file mode 100644 index 0000000000..c703345891 --- /dev/null +++ b/doc/devel/plugins.rst @@ -0,0 +1,447 @@ + +=================== +Writing Bro Plugins +=================== + +Bro is internally moving to a plugin structure that enables extending +the system dynamically, without modifying the core code base. That way +custom code remains self-contained and can be maintained, compiled, +and installed independently. Currently, plugins can add the following +functionality to Bro: + + - Bro scripts. + + - Builtin functions/events/types for the scripting language. + + - Protocol analyzers. + + - File analyzers. + + - Packet sources and packet dumpers. + + - Logging framework backends. + + - Input framework readers. + +A plugin's functionality is available to the user just as if Bro had +the corresponding code built-in. Indeed, internally many of Bro's +pieces are structured as plugins as well, they are just statically +compiled into the binary rather than loaded dynamically at runtime. + +Quick Start +=========== + +Writing a basic plugin is quite straight-forward as long as one +follows a few conventions. In the following we walk a simple example +plugin that adds a new built-in function (bif) to Bro: we'll add +``rot13(s: string) : string``, a function that rotates every character +in a string by 13 places. + +Generally, a plugin comes in the form of a directory following a +certain structure. To get started, Bro's distribution provides a +helper script ``aux/bro-aux/plugin-support/init-plugin`` that creates +a skeleton plugin that can then be customized. Let's use that:: + + # mkdir rot13-plugin + # cd rot13-plugin + # init-plugin Demo Rot13 + +As you can see the script takes two arguments. The first is a +namespace the plugin will live in, and the second a descriptive name +for the plugin itself. Bro uses the combination of the two to identify +a plugin. The namespace serves to avoid naming conflicts between +plugins written by independent developers; pick, e.g., the name of +your organisation. The namespace ``Bro`` is reserved for functionality +distributed by the Bro Project. In our example, the plugin will be +called ``Demo::Rot13``. + +The ``init-plugin`` script puts a number of files in place. The full +layout is described later. For now, all we need is +``src/rot13.bif``. It's initially empty, but we'll add our new bif +there as follows:: + + # cat src/rot13.bif + module CaesarCipher; + + function rot13%(s: string%) : string + %{ + char* rot13 = copy_string(s->CheckString()); + + for ( char* p = rot13; *p; p++ ) + { + char b = islower(*p) ? 'a' : 'A'; + *p = (*p - b + 13) % 26 + b; + } + + BroString* bs = new BroString(1, reinterpret_cast(rot13), + strlen(rot13)); + return new StringVal(bs); + %} + +The syntax of this file is just like any other ``*.bif`` file; we +won't go into it here. + +Now we can already compile our plugin, we just need to tell the +configure script put in place by ``init-plugin`` where the Bro source +tree is located (Bro needs to have been built there first):: + + # ./configure --bro-dist=/path/to/bro/dist && make + [... cmake output ...] + +Now our ``rot13-plugin`` directory has everything that it needs +for Bro to recognize it as a dynamic plugin. Once we point Bro to it, +it will pull it in automatically, as we can check with the ``-N`` +option:: + + # export BRO_PLUGIN_PATH=/path/to/rot13-plugin + # bro -N + [...] + Plugin: Demo::Rot13 - (dynamic, version 1) + [...] + +That looks quite good, except for the dummy description that we should +replace with something nicer so that users will know what our plugin +is about. We do this by editing the ``config.description`` line in +``src/Plugin.cc``, like this:: + + [...] + plugin::Configuration Configure() + { + plugin::Configuration config; + config.name = "Demo::Rot13"; + config.description = "Caesar cipher rotating a string's characters by 13 places."; + config.version.major = 1; + config.version.minor = 0; + return config; + } + [...] + + # make + [...] + # bro -N | grep Rot13 + Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) + +Better. Bro can also show us what exactly the plugin provides with the +more verbose option ``-NN``:: + + # bro -NN + [...] + Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) + [Function] CaesarCipher::rot13 + [...] + +There's our function. Now let's use it:: + + # bro -e 'print CaesarCipher::rot13("Hello")' + Uryyb + +It works. We next install the plugin along with Bro itself, so that it +will find it directly without needing the ``BRO_PLUGIN_PATH`` +environment variable. If we first unset the variable, the function +will no longer be available:: + + # unset BRO_PLUGIN_PATH + # bro -e 'print CaesarCipher::rot13("Hello")' + error in , line 1: unknown identifier CaesarCipher::rot13, at or near "CaesarCipher::rot13" + +Once we install it, it works again:: + + # make install + # bro -e 'print CaesarCipher::rot13("Hello")' + Uryyb + +The installed version went into +``/lib/bro/plugins/Demo_Rot13``. + +We can distribute the plugin in either source or binary form by using +the Makefile's ``sdist`` and ``bdist`` target, respectively. Both +create corrsponding tarballs:: + + # make sdist + [...] + Source distribution in build/sdist/Demo_Rot13.tar.gz + + # make bdist + [...] + Binary distribution in build/Demo_Rot13-darwin-x86_64.tar.gz + +The source archive will contain everything in the plugin directory +except any generated files. The binary archive will contain anything +needed to install and run the plugin, i.e., just what ``make install`` +puts into place as well. As the binary distribution is +platform-dependent, its name includes the OS and architecture the +plugin was built on. + +Plugin Directory Layout +======================= + +A plugin's directory needs to follow a set of conventions so that Bro +(1) recognizes it as a plugin, and (2) knows what to load. While +``init-plugin`` takes care of most of this, the following is the full +story. We'll use ```` to represent a plugin's top-level +directory. + +``/__bro_plugin__`` + A file that marks a directory as containing a Bro plugin. The file + must exist, and its content must consist of a single line with the + qualified name of the plugin (e.g., "Demo::Rot13"). + +``/lib/--.so`` + The shared library containing the plugin's compiled code. Bro will + load this in dynamically at run-time if OS and architecture match + the current platform. + +``scripts/`` + A directory with the plugin's custom Bro scripts. When the plugin + gets activated, this directory will be automatically added to + ``BROPATH``, so that any scripts/modules inside can be + "@load"ed. + +``scripts``/__load__.bro + A Bro script that will be loaded immediately when the plugin gets + activated. See below for more information on activating plugins. + +``lib/bif/`` + Directory with auto-generated Bro scripts that declare the plugin's + bif elements. The files here are produced by ``bifcl``. + +By convention, a plugin should put its custom scripts into sub folders +of ``scripts/``, i.e., ``scripts//