diff --git a/.gitmodules b/.gitmodules index 4998cc6b80..24375ce23d 100644 --- a/.gitmodules +++ b/.gitmodules @@ -19,3 +19,6 @@ [submodule "src/3rdparty"] path = src/3rdparty url = git://git.bro.org/bro-3rdparty +[submodule "aux/plugins"] + path = aux/plugins + url = git://git.bro.org/bro-plugins diff --git a/CHANGES b/CHANGES index 36a41ee656..1b084c7f19 100644 --- a/CHANGES +++ b/CHANGES @@ -1,4 +1,678 @@ +2.3-397 | 2015-01-27 10:13:10 -0600 + + * Handle guess_lexer exceptions in pygments reST directive (Jon Siwek) + +2.3-396 | 2015-01-23 10:49:15 -0600 + + * DNP3: fix reachable assertion and buffer over-read/overflow. + CVE number pending. (Travis Emmert, Jon Siwek) + + * Update binpac: Fix potential out-of-bounds memory reads in generated + code. CVE-2014-9586. (John Villamil and Chris Rohlf - Yahoo + Paranoids, Jon Siwek) + + * Fixing (harmless) Coverity warning. (Robin Sommer) + +2.3-392 | 2015-01-15 09:44:15 -0800 + + * Small changes to EC curve names in a newer draft. (Johanna Amann) + +2.3-390 | 2015-01-14 13:27:34 -0800 + + * Updating MySQL analyses. (Vlad Grigorescu) + - Use a boolean success instead of a result string. + - Change the affected_rows response detail string to a "rows" count. + - Fix the state tracking to log incomplete command. + + * Extend DNP3 to support communication over UDP. (Hui Lin) + + * Fix a bug in DNP3 determining the length of an object in some + cases. (Hui Lin) + +2.3-376 | 2015-01-12 09:38:10 -0600 + + * Improve documentation for connection_established event. (Jon Siwek) + +2.3-375 | 2015-01-08 13:10:09 -0600 + + * Increase minimum required CMake version to 2.8. (Jon Siwek) + +2.3-374 | 2015-01-07 10:03:17 -0600 + + * Improve documentation of the Intelligence Framework. (Daniel Thayer) + +2.3-371 | 2015-01-06 09:58:09 -0600 + + * Update/improve file mime type identification. (Seth Hall) + + - Change to the default BOF buffer size to 3000 (was 1024). + + - Reorganized MS signatures into a separate file. + + - Remove all of the x-c detections. Nearly all false positives. + + - Improve TAR detections, removing old, back up TAR detections. + + - Remove one of the x-elc detections that was too loose + and caused many false positives. + + - Improved lots of the signatures and added new ones. (Seth Hall) + + * Add support for file reassembly in the file analysis framework + (Seth Hall, Jon Siwek). + + - The reassembly behavior can be modified per-file by enabling or + disabling the reassembler and/or modifying the size of the + reassembly buffer. + + - Changed the file extraction analyzer to use stream-wise input to + avoid issues with the chunk-wise approach not immediately + triggering the file_new event due to mime-type detection delay. + Before, early chunks frequently ended up lost. Extraction also + will now explicitly NUL-fill gaps in the file instead of + implicitly relying on pwrite to do it. + +2.3-349 | 2015-01-05 15:21:13 -0600 + + * Fix race condition in unified2 file analyzer startup. (Jon siwek) + +2.3-348 | 2014-12-31 09:19:34 -0800 + + * Changing Makefile's test-all to run test-all for broctl, which now + executes trace-summary tests as well. (Robin Sommer) + +2.3-345 | 2014-12-31 09:06:15 -0800 + + * Correct a typo in the Notice framework doc. (Daniel Thayer) + +2.3-343 | 2014-12-12 12:43:46 -0800 + + * Fix PIA packet replay to deliver copy of IP header. This prevented + one from writing a packet-wise analyzer that needs access to IP + headers and can be attached to a connection via signature match. + Addresses BIT-1298 (Jon Siwek) + +2.3-338 | 2014-12-08 13:56:19 -0800 + + * Add man page for Bro. (Raúl Benencia) + + * Updating doc baselines. (Robin Sommer) + +2.3-334 | 2014-12-03 14:22:07 -0800 + + * Fix compound assignment to require proper L-value. Addresses + BIT-1295. (Jon Siwek) + +2.3-332 | 2014-12-03 14:14:11 -0800 + + * Make using local IDs in @if directives an error. Addresses + BIT-1296. (Jon Siwek) + +2.3-330 | 2014-12-03 14:10:39 -0800 + + * Fix some "make doc" warnings and update some doc tests. (Daniel + Thayer) + +2.3-328 | 2014-12-02 08:13:10 -0500 + + * Update windows-version-detection.bro to add support for + Windows 10. (Michal Purzynski) + +2.3-326 | 2014-12-01 12:10:27 -0600 + + * BIFScanner: fix invalid characters in generated preprocessor macros. + (Hilko Bengen) + + * BIT-1294: fix exec.bro from mutating Input::end_of_data event + parameters. (Johanna Amann) + + * Add/invoke "distclean" for testing directories. (Raúl Benencia) + + * Delete prebuilt python bytecode files from git. (Jon Siwek) + + * Add Windows detection based on CryptoAPI HTTP traffic as a software + framework policy script. (Vlad Grigorescu) + +2.3-316 | 2014-11-25 17:35:06 -0800 + + * Make the SSL analyzer skip further processing once encountering + situations which are very probably non-recoverable. (Johanna + Amann) + +2.3-313 | 2014-11-25 14:27:07 -0800 + + * Make SSL v2 protocol tests more strict. In its former state they + triggered on http traffic over port 443 sometimes. Found by Michał + Purzyński. (Johanna Amann) + + * Fix X509 analyzer to correctly return ECDSA as the key_type for + ECDSA certs. Bug found by Michał Purzyński. (Johanna Amann) + +2.3-310 | 2014-11-19 10:56:59 -0600 + + * Disable verbose bison output. (Jon Siwek) + +2.3-309 | 2014-11-18 12:17:53 -0800 + + * New decompose_uri() function in base/utils/urls that splits a URI + into its pieces. (Anthony Kasza). + +2.3-305 | 2014-11-18 11:09:04 -0800 + + * Improve coercion of &default expressions. Addresses BIT-1288. (Jon + Siwek) + +2.3-303 | 2014-11-18 10:53:04 -0800 + + * For DH key exchanges, use p as the parameter for weak key + exchanges. (Johanna Amann) + +2.3-301 | 2014-11-11 13:47:27 -0800 + + * Add builtin function enum_to_int() that converts an enum into a + integer. (Christian Struck) + +2.3-297 | 2014-11-11 11:50:47 -0800 + + * Removing method from SSL analyzer that's no longer used. (Robin + Sommer) + +2.3-296 | 2014-11-11 11:42:38 -0800 + + * A new analyzer parsing the MySQL wire protocol. Activity gets + logged into mysql.log. Supports protocol versions 9 and 10. (Vlad + Grigorescu) + +2.3-280 | 2014-11-05 09:46:33 -0500 + + * Add Windows detection based on CryptoAPI HTTP traffic as a + software framework policy script. (Vlad Grigorescu) + +2.3-278 | 2014-11-03 18:55:18 -0800 + + * Add new curves from draft-ietf-tls-negotiated-ff-dhe to SSL + analysis. (Johanna Amann) + +2.3-274 | 2014-10-31 17:45:25 -0700 + + * Adding call to new binpac::init() function. (Robin Sommer) + +2.3-272 | 2014-10-31 16:29:42 -0700 + + * Fix segfault if when statement's RHS is unitialized. Addresses + BIT-1176. (Jon Siwek) + + * Fix checking vector indices via "in". Addresses BIT-1280. (Jon + Siwek) + +2.3-268 | 2014-10-31 12:12:22 -0500 + + * BIT-1283: Fix crash when using &encrypt. (Jon Siwek) + +2.3-267 | 2014-10-31 10:35:02 -0500 + + * BIT-1284: Allow arbitrary when statement timeout expressions + (Jon Siwek) + +2.3-266 | 2014-10-31 09:21:28 -0500 + + * BIT-1166: Add configure options to fine tune local state dirs used + by BroControl. (Jon Siwek) + +2.3-264 | 2014-10-30 13:25:57 -0500 + + * Fix some minor Coverity Scan complaints. (Jon Siwek) + +2.3-263 | 2014-10-28 15:09:10 -0500 + + * Fix checking of fwrite return values (Johanna Amann) + +2.3-260 | 2014-10-27 12:54:17 -0500 + + * Fix errors/warnings when compiling with -std=c++11 (Jon Siwek) + +2.3-259 | 2014-10-27 10:04:04 -0500 + + * Documentation fixes. (Vicente Jimenez Aguilar and Stefano Azzalini) + +2.3-256 | 2014-10-24 15:33:45 -0700 + + * Adding missing test baseline. (Robin Sommer) + +2.3-255 | 2014-10-24 13:39:44 -0700 + + * Fixing unstable active-http test. (Robin Sommer) + +2.3-254 | 2014-10-24 11:40:51 -0700 + + * Fix active-http.bro to deal reliably with empty server responses, + which will now be passed back as empty files. (Christian Struck) + +2.3-248 | 2014-10-23 14:20:59 -0700 + + * Change order in which a plugin's scripts are loaded at startup. + (Robin Sommer) + +2.3-247 | 2014-10-21 13:42:38 -0700 + + * Updates to the SSL analyzer. (Johanna Amann) + + * Mark everything below 2048 bit as a weak key. + + * Fix notice suppression. + + * Add information about server-chosen protocol to ssl.log, if + provided by application_layer_next_protocol. + + * Add boolean flag to ssl.log signaling if a session was + resumed. Remove the (usually not really that useful) session + ID that the client sent. + +2.3-240 | 2014-10-21 13:36:33 -0700 + + * Fix Coverity-reported issues in DNP3 analyzer. (Seth Hall) + +2.3-238 | 2014-10-16 06:51:49 -0700 + + * Fix multipart HTTP/MIME entity file analysis so that (1) singular + CR or LF characters in multipart body content are no longer + converted to a full CRLF (thus corrupting the file) and (2) it + also no longer considers the CRLF before the multipart boundary as + part of the content. Addresses BIT-1235. (Jon Siwek) + +2.3-235 | 2014-10-15 10:20:47 -0500 + + * BIT-1273: Add error message for bad enum declaration syntax. + (Jon Siwek) + +2.3-234 | 2014-10-14 14:42:09 -0500 + + * Documentation fixes. (Steve Smoot) + +2.3-233 | 2014-10-09 16:00:27 -0500 + + * Change find-bro-logs unit test to follow symlinks. (Jon Siwek) + + * Add error checks and messages to a test script (Daniel Thayer) + +2.3-230 | 2014-10-08 08:15:17 -0700 + + * Further baseline normalization for plugin test portability. (Robin + Sommer) + +2.3-229 | 2014-10-07 20:18:11 -0700 + + * Fix for test portability. (Robin Sommer) + +2.3-228 | 2014-10-07 15:32:37 -0700 + + * Include plugin unit tests into the top-level btest configuration. (Robin Sommer) + + * Switching the prefix separator for packet source/dumper plugins + once more, now to "::". Addresses BIT-1267. (Robin Sommer) + + * Fix for allowing a packet source/dumper plugin to support multiple + prefixes with a colon. (Robin Sommer) + +2.3-225 | 2014-10-07 15:13:35 -0700 + + * Updating plugin documentation. (Robin Sommer) + +2.3-224 | 2014-10-07 14:32:17 -0700 + + * Improved the log file reference documentation. (Jeannette Dopheide + and Daniel Thayer) + + * Improves shockwave flash file signatures. (Seth Hall) + + - This moves the signatures out of the libmagic imported signatures + and into our own general.sig. + + - Expand the detection to LZMA compressed flash files. + + * Add new script language reference documentation on operators, + statements, and directives. Also improved the documentation on + types and attributes by splitting them into two docs, and + providing more examples and adding a chart on the top of each page + with links to each type and attribute for easier access to the + information. (Daniel Thayer) + + * Split the types and attributes reference doc into two docs. + (Daniel Thayer) + +2.3-208 | 2014-10-03 09:38:52 -0500 + + * BIT-1268: Fix uninitialized router_list argument in + dhcp_offer/dhcp_ack. (Jon Siwek) + +2.3-207 | 2014-10-02 16:39:17 -0700 + + * Updating plugin docs. (Robin Sommer) + + * Fix packet sources being treated as idle when a packet is + available. Addresses BIT-1266. (Jon Siwek) + + * Fix regression causing the main loop to spin more frequently. + Addresses BIT-1266. (Jon Siwek) + +2.3-203 | 2014-09-29 20:06:54 -0700 + + * Fix to use length parameter in DNP3 time conversion correctly now. + (Robin Sommer) + +2.3-202 | 2014-09-29 17:05:18 -0700 + + * New SSL extension type from IANA and a few other SSL const + changes. (Johanna Amann) + + * Make unexpected pipe errors fatal as precaution. Addresses + BIT-1260. (Jon Siwek) + + * Adding a function for DNP3 to translate the timestamp format. (Hui + Lin) + +2.3-197 | 2014-09-29 10:42:01 -0500 + + * Fix possible seg fault in TCP reassembler. (Jon Siwek) + +2.3-196 | 2014-09-25 17:53:27 -0700 + + * Changing prefix for packet sources/dumper from ':' to '%'. + Addresses BIT-1249. (Robin Sommer) + + * Remove timeouts from remote communication loop. The select() now + blocks until there's work to do instead of relying on a small + timeout value which can cause unproductive use of cpu cycles. (Jon + Siwek) + + * Improve error message when failing to activate a plugin. Also fix + a unit test helper script that checks plugin availability. (Jon + Siwek) + +2.3-183 | 2014-09-24 10:08:04 -0500 + + * Add a "node" field to Intel::Seen struture and intel.log to + indicate which node discovered a hit on an intel item. (Seth Hall) + + * BIT-1261: Fixes to plugin quick start doc. (Jon Siwek) + +2.3-180 | 2014-09-22 12:52:41 -0500 + + * BIT-1259: Fix issue w/ duplicate TCP reassembly deliveries. + (Jon Siwek) + +2.3-178 | 2014-09-18 14:29:46 -0500 + + * BIT-1256: Fix file analysis events from coming after bro_done(). + (Jon Siwek) + +2.3-177 | 2014-09-17 09:41:27 -0500 + + * Documentation fixes. (Chris Mavrakis) + +2.3-174 | 2014-09-17 09:37:09 -0500 + + * Fixed some "make doc" warnings caused by reST formatting + (Daniel Thayer). + +2.3-172 | 2014-09-15 13:38:52 -0500 + + * Remove unneeded allocations for HTTP messages. (Jon Siwek) + +2.3-171 | 2014-09-15 11:14:57 -0500 + + * Fix a compile error on systems without pcap-int.h. (Jon Siwek) + +2.3-170 | 2014-09-12 19:28:01 -0700 + + * Fix incorrect data delivery skips after gap in HTTP Content-Range. + Addresses BIT-1247. (Jon Siwek) + + * Fix file analysis placement of data after gap in HTTP + Content-Range. Addresses BIT-1248. (Jon Siwek) + + * Fix issue w/ TCP reassembler not delivering some segments. + Addresses BIT-1246. (Jon Siwek) + + * Fix MIME entity file data/gap ordering and raise http_entity_data + in line with data arrival. Addresses BIT-1240. (Jon Siwek) + + * Implement file ID caching for MIME_Mail. (Jon Siwek) + + * Fix a compile error. (Jon Siwek) + +2.3-161 | 2014-09-09 12:35:38 -0500 + + * Bugfixes and test updates/additions. (Robin Sommer) + + * Interface tweaks and docs for PktSrc/PktDumper. (Robin Sommer) + + * Moving PCAP-related bifs to iosource/pcap.bif. (Robin Sommer) + + * Moving some of the BPF filtering code into base class. + This will allow packet sources that don't support BPF natively to + emulate the filtering via libpcap. (Robin Sommer) + + * Removing FlowSrc. (Robin Sommer) + + * Removing remaining pieces of the 2ndary path, and left-over + files of packet sorter. (Robin Sommer) + + * A bunch of infrastructure work to move IOSource, IOSourceRegistry + (now iosource::Manager) and PktSrc/PktDumper code into iosource/, + and over to a plugin structure. (Robin Sommer) + +2.3-137 | 2014-09-08 19:01:13 -0500 + + * Fix Broxygen's rendering of opaque types. (Jon Siwek) + +2.3-136 | 2014-09-07 20:50:46 -0700 + + * Change more http links to https. (Johanna Amann) + +2.3-134 | 2014-09-04 16:16:36 -0700 + + * Fixed a number of issues with OCSP reply validation. Addresses + BIT-1212. (Johanna Amann) + + * Fix null pointer dereference in OCSP verification code in case no + certificate is sent as part as the ocsp reply. Addresses BIT-1212. + (Johanna Amann) + +2.3-131 | 2014-09-04 16:10:32 -0700 + + * Make links in documentation templates protocol relative. (Johanna + Amann) + +2.3-129 | 2014-09-02 17:21:21 -0700 + + * Simplify a conditional with equivalent branches. (Jon Siwek) + + * Change EDNS parsing code to use rdlength more cautiously. (Jon + Siwek) + + * Fix a memory leak when bind() fails due to EADDRINUSE. (Jon Siwek) + + * Fix possible buffer over-read in DNS TSIG parsing. (Jon Siwek) + +2.3-124 | 2014-08-26 09:24:19 -0500 + + * Better documentation for sub_bytes (Jimmy Jones) + + * BIT-1234: Fix build on systems that already have ntohll/htonll + (Jon Siwek) + +2.3-121 | 2014-08-22 15:22:15 -0700 + + * Detect functions that try to bind variables from an outer scope + and raise an error saying that's not supported. Addresses + BIT-1233. (Jon Siwek) + +2.3-116 | 2014-08-21 16:04:13 -0500 + + * Adding plugin testing to Makefile's test-all. (Robin Sommer) + + * Converting log writers and input readers to plugins. + DataSeries and ElasticSearch plugins have moved to the new + bro-plugins repository, which is now a git submodule in the + aux/plugins directory. (Robin Sommer) + +2.3-98 | 2014-08-19 11:03:46 -0500 + + * Silence some doc-related warnings when using `bro -e`. + Closes BIT-1232. (Jon Siwek) + + * Fix possible null ptr derefs reported by Coverity. (Jon Siwek) + +2.3-96 | 2014-08-01 14:35:01 -0700 + + * Small change to DHCP documentation. In server->client messages the + host name may differ from the one requested by the client. + (Johanna Amann) + + * Split DHCP log writing from record creation. This allows users to + customize dhcp.log by changing the record in their own dhcp_ack + event. (Johanna Amann) + + * Update PATH so that documentation btests can find bro-cut. (Daniel + Thayer) + + * Remove gawk from list of optional packages in documentation. + (Daniel Thayer) + + * Fix for redefining built-in constants. (Robin Sommer) + +2.3-86 | 2014-07-31 14:19:58 -0700 + + * Fix for redefining built-in constants. (Robin Sommer) + + * Adding missing check that a plugin's API version matches what Bro + defines. (Robin Sommer) + + * Adding NEWS entry for plugins. (Robin Sommer) + +2.3-83 | 2014-07-30 16:26:11 -0500 + + * Minor adjustments to plugin code/docs. (Jon Siwek) + + * Dynamic plugin support. (Rpbin Sommer) + + Bro now supports extending core functionality, like protocol and + file analysis, dynamically with external plugins in the form of + shared libraries. See doc/devel/plugins.rst for an overview of the + main functionality. Changes coming with this: + + - Replacing the old Plugin macro magic with a new API. + + - The plugin API changed to generally use std::strings instead + of const char*. + + - There are a number of invocations of PLUGIN_HOOK_ + {VOID,WITH_RESULT} across the code base, which allow plugins + to hook into the processing at those locations. + + - A few new accessor methods to various classes to allow + plugins to get to that information. + + - network_time cannot be just assigned to anymore, there's now + function net_update_time() for that. + + - Redoing how builtin variables are initialized, so that it + works for plugins as well. No more init_net_var(), but + instead bifcl-generated code that registers them. + + - Various changes for adjusting to the now dynamic generation + of analyzer instances. + + - same_type() gets an optional extra argument allowing record type + comparision to ignore if field names don't match. (Robin Sommer) + + - Further unify file analysis API with the protocol analyzer API + (assigning IDs to analyzers; adding Init()/Done() methods; + adding subtypes). (Robin Sommer) + + - A new command line option -Q that prints some basic execution + time stats. (Robin Sommer) + + - Add support to the file analysis for activating analyzers by + MIME type. (Robin Sommer) + + - File::register_for_mime_type(tag: Analyzer::Tag, mt: + string): Associates a file analyzer with a MIME type. + + - File::add_analyzers_for_mime_type(f: fa_file, mtype: + string): Activates all analyzers registered for a MIME + type for the file. + + - The default file_new() handler calls + File::add_analyzers_for_mime_type() with the file's MIME + type. + +2.3-20 | 2014-07-22 17:41:02 -0700 + + * Updating submodule(s). + +2.3-19 | 2014-07-22 17:29:19 -0700 + + * Implement bytestring_to_coils() in Modbus analyzer so that coils + gets passed to the corresponding events. (Hui Lin) + + * Add length field to ModbusHeaders. (Hui Lin) + +2.3-12 | 2014-07-10 19:17:37 -0500 + + * Include yield of vectors in Broxygen's type descriptions. + Addresses BIT-1217. (Jon Siwek) + +2.3-11 | 2014-07-10 14:49:27 -0700 + + * Fixing DataSeries output. It was using a now illegal value as its + default compression level. (Robin Sommer) + +2.3-7 | 2014-06-26 17:35:18 -0700 + + * Extending "make test-all" to include aux/bro-aux. (Robin Sommer) + +2.3-6 | 2014-06-26 17:24:10 -0700 + + * DataSeries compilation issue fixed. (mlaterman) + + * Fix a reference counting bug in ListVal ctor. (Jon Siwek) + +2.3-3 | 2014-06-26 15:41:04 -0500 + + * Support tilde expansion when Bro tries to find its own path. (Jon + Siwek) + +2.3-2 | 2014-06-23 16:54:15 -0500 + + * Remove references to line numbers in tutorial text. (Daniel Thayer) + +2.3 | 2014-06-16 09:48:25 -0500 + + * Release 2.3. + +2.3-beta-33 | 2014-06-12 11:59:28 -0500 + + * Documentation improvements/fixes. (Daniel Thayer) + +2.3-beta-24 | 2014-06-11 15:35:31 -0500 + + * Fix SMTP state tracking when server response is missing. + (Robin Sommer) + +2.3-beta-22 | 2014-06-11 12:31:38 -0500 + + * Fix doc/test that broke due to a Bro script change. (Jon Siwek) + + * Remove unused --with-libmagic configure option. (Jon Siwek) + +2.3-beta-20 | 2014-06-10 18:16:51 -0700 + + * Fix use-after-free in some cases of reassigning a table index. + Addresses BIT-1202. (Jon Siwek) + 2.3-beta-18 | 2014-06-06 13:11:50 -0700 * Add two more SSL events, one triggered for each handshake message @@ -82,7 +756,7 @@ and the load balancing FAQ on the website. (Daniel Thayer) * Update some doc tests and line numbers (Daniel Thayer) - + 2.2-457 | 2014-05-16 14:38:31 -0700 * New script policy/protocols/ssl/validate-ocsp.bro that adds OSCP @@ -97,7 +771,7 @@ Amann) * Improved Heartbleed attack/scan detection. (Bernhard Amann) - + * Let TLS analyzer fail better when no longer in sync with the data stream. (Bernhard Amann) @@ -134,12 +808,12 @@ *Undelivered methods now use a uint64 in place of an int for the relative sequence space offset parameter. - Addresses BIT-348. + Addresses BIT-348. * Fixing compiler warnings. (Robin Sommer) - + * Update SNMP analyzer's DeliverPacket method signature. (Jon Siwek) - + 2.2-417 | 2014-05-07 10:59:22 -0500 * Change handling of atypical OpenSSL error case in x509 verification. (Jon Siwek) @@ -179,7 +853,7 @@ 2.2-397 | 2014-05-01 20:29:20 -0700 * Fix reference counting for lookup_ID() usages. (Jon Siwek) - + 2.2-395 | 2014-05-01 20:25:48 -0700 * Fix missing "irc-dcc-data" service field from IRC DCC connections. @@ -192,7 +866,7 @@ Siwek) * Improve file analysis manager shutdown/cleanup. (Jon Siwek) - + 2.2-388 | 2014-04-24 18:38:07 -0700 * Fix decoding of MIME quoted-printable. (Mareq) @@ -205,11 +879,11 @@ 2.2-381 | 2014-04-24 17:08:45 -0700 * Add Java version to software framework. (Brian Little) - + 2.2-379 | 2014-04-24 17:06:21 -0700 * Remove unused Val::attribs member. (Jon Siwek) - + 2.2-377 | 2014-04-24 16:57:54 -0700 * A larger set of SSL improvements and extensions. Addresses @@ -222,7 +896,7 @@ server_name, alpn, and ec-curves. - Adds support for the heartbeat events. - + - Add Heartbleed detector script. - Adds basic support for OCSP stapling. @@ -233,7 +907,7 @@ 2.2-353 | 2014-04-24 16:12:30 -0700 * Adapt HTTP partial content to cache file analysis IDs. (Jon Siwek) - + * Adapt SSL analyzer to generate file analysis handles itself. (Jon Siwek) @@ -249,11 +923,11 @@ Siwek) * Refactor file analysis file ID lookup. (Jon Siwek) - + 2.2-344 | 2014-04-22 20:13:30 -0700 * Refactor various hex escaping code. (Jon Siwek) - + 2.2-341 | 2014-04-17 18:01:41 -0500 * Fix duplicate DNS log entries. (Robin Sommer) @@ -261,9 +935,9 @@ 2.2-341 | 2014-04-17 18:01:01 -0500 * Refactor initialization of ASCII log writer options. (Jon Siwek) - + * Fix a memory leak in ASCII log writer. (Jon Siwek) - + 2.2-338 | 2014-04-17 17:48:17 -0500 * Disable input/logging threads setting their names on every @@ -275,7 +949,7 @@ 2.2-335 | 2014-04-10 15:04:57 -0700 * Small logic fix for main SSL script. (Bernhard Amann) - + * Update DPD signatures for detecting TLS 1.2. (Bernhard Amann) * Remove unused data member of SMTP_Analyzer to silence a Coverity @@ -299,7 +973,7 @@ 2.2-317 | 2014-04-03 10:51:31 -0400 - * Add a uid field to the signatures.log. Addresses BIT-1171 + * Add a uid field to the signatures.log. Addresses BIT-1171 (Anthony Verez) 2.2-315 | 2014-04-01 16:50:01 -0700 @@ -310,7 +984,7 @@ 2.2-313 | 2014-04-01 16:40:19 -0700 * Fix a couple nits reported by Coverity.(Jon Siwek) - + * Fix potential memory leak in IP frag reassembly reported by Coverity. (Jon Siwek) @@ -400,14 +1074,14 @@ 2.2-271 | 2014-03-30 20:25:17 +0200 * Add unit tests covering vector/set/table ctors/inits. (Jon Siwek) - + * Fix parsing of "local" named table constructors. (Jon Siwek) * Improve type checking of records. Addresses BIT-1159. (Jon Siwek) - + 2.2-267 | 2014-03-30 20:21:43 +0200 - * Improve documentation of Bro clusters. Addresses BIT-1160. + * Improve documentation of Bro clusters. Addresses BIT-1160. (Daniel Thayer) 2.2-263 | 2014-03-30 20:19:05 +0200 @@ -434,7 +1108,7 @@ 2.2-254 | 2014-03-30 19:55:22 +0200 * Update instructions on how to build Bro docs. (Daniel Thayer) - + 2.2-251 | 2014-03-28 08:37:37 -0400 * Quick fix to the ElasticSearch writer. (Seth Hall) @@ -464,7 +1138,7 @@ formatter work. * Fixing compiler error. (Robin Sommer) - + * Fixing (very unlikely) double delete in HTTP analyzer when decapsulating CONNECTs. (Robin Sommer) @@ -490,7 +1164,7 @@ Addresses BIT-1134. (Jon Siwek) * Enable fake DNS mode for test suites. - + * Improve analysis of TCP SYN/SYN-ACK reversal situations. (Jon Siwek) @@ -522,7 +1196,7 @@ (Jon Siwek) * Silences some documentation warnings from Sphinx. (Jon Siwek) - + 2.2-215 | 2014-03-10 11:10:15 -0700 * Fix non-deterministic logging of unmatched DNS msgs. Addresses @@ -560,7 +1234,7 @@ HTTP traffic. (Seth Hall) * Fixing removal of support analyzers. (Robin Sommer) - + 2.2-199 | 2014-03-03 16:34:20 -0800 * Allow iterating over bif functions with result type vector of any. @@ -575,11 +1249,11 @@ 2.2-194 | 2014-02-28 14:50:53 -0800 * Remove packet sorter. Addresses BIT-700. (Bernhard Amann) - + 2.2-192 | 2014-02-28 09:46:43 -0800 * Update Mozilla root bundle. (Bernhard Amann) - + 2.2-190 | 2014-02-27 07:34:44 -0800 * Adjust timings of a few leak tests. (Bernhard Amann) @@ -609,7 +1283,7 @@ 2.2-177 | 2014-02-20 17:27:46 -0800 * Update to libmagic version 5.17. Addresses BIT-1136. (Jon Siwek) - + 2.2-174 | 2014-02-14 12:07:04 -0800 * Support for MPLS over VLAN. (Chris Kanich) @@ -638,7 +1312,7 @@ defined, but we see it being actively used. (Bernhard Amann) * Test baseline updates for DNS change. (Robin Sommer) - + 2.2-158 | 2014-02-09 23:45:39 -0500 * Change dns.log to include only standard DNS queries. (Jon Siwek) @@ -714,7 +1388,7 @@ 2.2-128 | 2014-01-30 15:58:47 -0800 * Add leak test for Exec module. (Bernhard Amann) - + * Fix file_over_new_connection event to trigger when entire file is missed. (Jon Siwek) @@ -727,7 +1401,7 @@ "detect_filtered_trace". (Jon Siwek) * Improve TCP FIN retransmission handling. (Jon Siwek) - + 2.2-120 | 2014-01-28 10:25:23 -0800 * Fix and extend x509_extension() event, which now actually returns @@ -745,10 +1419,10 @@ 2.2-115 | 2014-01-22 12:11:18 -0800 * Add unit tests for new Bro Manual docs. (Jon Siwek) - + * New content for the "Using Bro" section of the manual. (Rafael Bonilla/Jon Siwek) - + 2.2-105 | 2014-01-20 12:16:48 -0800 * Support GRE tunnel decapsulation, including enhanced GRE headers. @@ -757,7 +1431,7 @@ Addresses BIT-867. (Jon Siwek) * Simplify FragReassembler memory management. (Jon Siwek) - + 2.2-102 | 2014-01-20 12:00:29 -0800 * Include file information (MIME type and description) into notice @@ -766,7 +1440,7 @@ 2.2-100 | 2014-01-20 11:54:58 -0800 * Fix caching of recently validated SSL certifcates. (Justin Azoff) - + 2.2-98 | 2014-01-20 11:50:32 -0800 * For notice suppresion, instead of storing the entire notice in @@ -794,7 +1468,7 @@ the city database instead of just the former. (Jon Siwek) * Broxygen init fixes. Addresses BIT-1110. (Jon Siwek) - + - Don't check mtime of bro binary if BRO_DISABLE_BROXYGEN env var set. - Fix failure to locate bro binary if invoking from a relative @@ -807,7 +1481,7 @@ len field being set to zero. (Seth Hall) * Canonify output of a unit test. (Jon Siwek) - + * A set of documentation updates. (Daniel Thayer) - Fix typo in Bro 2.2 NEWS on string indexing. @@ -854,9 +1528,9 @@ (Jon Siwek) * Close signature files after done parsing. (Jon Siwek) - + * Fix unlikely null ptr deref in broxygen::Manager. (Jon Siwek) - + * FreeBSD build fix addendum: unintended variable shadowing. (Jon Siwek) @@ -875,7 +1549,7 @@ were cleaned up. Addresses BIT-1103. (Bernhard Amann) * Minor Broxygen improvements. Addresses BIT-1098. (Jon Siwek) - + 2.2-51 | 2013-12-05 07:53:37 -0800 * Improve a unit test involving 'when' conditionals. (Jon Siwek) @@ -900,7 +1574,7 @@ 2.2-44 | 2013-12-04 12:41:51 -0800 * Fix string slice notation. Addresses BIT-1097. (Jon Siwek) - + Slice ranges were not correctly determined for negative indices and also off by one in general (included one more element at the end of the substring than what actually matched the index range). @@ -955,7 +1629,7 @@ 2.2-11 | 2013-12-03 10:56:28 -0800 * Unit test for broccoli vector support. (Jon Siwek) - + * Changed ordering of Bro type tag enum, which was out of sync. (Jon Siwek) @@ -967,14 +1641,14 @@ 2.2-6 | 2013-11-15 07:05:15 -0800 * Make "install-example-configs" target use DESTDIR. (Jon Siwek) - + 2.2-5 | 2013-11-11 13:47:54 -0800 * Fix the irc_reply event for certain server message types. (Seth Hall) * Fixed Segmentation fault in SQLite Writer. (Jon Crussell) - + 2.2 | 2013-11-07 10:25:50 -0800 * Release 2.2. diff --git a/CMakeLists.txt b/CMakeLists.txt index 87b99271c5..c0ff6c09d4 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,5 +1,9 @@ project(Bro C CXX) -cmake_minimum_required(VERSION 2.6.3 FATAL_ERROR) + +# When changing the minimum version here, also adapt +# aux/bro-aux/plugin-support/skeleton/CMakeLists.txt +cmake_minimum_required(VERSION 2.8 FATAL_ERROR) + include(cmake/CommonCMakeConfig.cmake) ######################################################################## @@ -11,17 +15,28 @@ if (NOT BRO_SCRIPT_INSTALL_PATH) set(BRO_SCRIPT_INSTALL_PATH ${BRO_ROOT_DIR}/share/bro) endif () +if (NOT BRO_MAN_INSTALL_PATH) + # set the default Bro man page installation path (user did not specify one) + set(BRO_MAN_INSTALL_PATH ${BRO_ROOT_DIR}/share/man) +endif () + # sanitize the Bro script install directory into an absolute path # (CMake is confused by ~ as a representation of home directory) get_filename_component(BRO_SCRIPT_INSTALL_PATH ${BRO_SCRIPT_INSTALL_PATH} ABSOLUTE) +set(BRO_PLUGIN_INSTALL_PATH ${BRO_ROOT_DIR}/lib/bro/plugins CACHE STRING "Installation path for plugins" FORCE) + configure_file(bro-path-dev.in ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev) + file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.sh "export BROPATH=`${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" + "export BRO_PLUGIN_PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src:${BRO_PLUGIN_INSTALL_PATH}\"\n" "export PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n") + file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.csh "setenv BROPATH `${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" + "setenv BRO_PLUGIN_PATH \"${CMAKE_CURRENT_BINARY_DIR}/src:${BRO_PLUGIN_INSTALL_PATH}\"\n" "setenv PATH \"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n") file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/VERSION" VERSION LIMIT_COUNT 1) @@ -117,33 +132,6 @@ if (GOOGLEPERFTOOLS_FOUND) endif () endif () -set(USE_DATASERIES false) -find_package(Lintel) -find_package(DataSeries) -find_package(LibXML2) - -if (NOT DISABLE_DATASERIES AND - LINTEL_FOUND AND DATASERIES_FOUND AND LIBXML2_FOUND) - set(USE_DATASERIES true) - include_directories(BEFORE ${Lintel_INCLUDE_DIR}) - include_directories(BEFORE ${DataSeries_INCLUDE_DIR}) - include_directories(BEFORE ${LibXML2_INCLUDE_DIR}) - list(APPEND OPTLIBS ${Lintel_LIBRARIES}) - list(APPEND OPTLIBS ${DataSeries_LIBRARIES}) - list(APPEND OPTLIBS ${LibXML2_LIBRARIES}) -endif() - -set(USE_ELASTICSEARCH false) -set(USE_CURL false) -find_package(LibCURL) - -if (NOT DISABLE_ELASTICSEARCH AND LIBCURL_FOUND) - set(USE_ELASTICSEARCH true) - set(USE_CURL true) - include_directories(BEFORE ${LibCURL_INCLUDE_DIR}) - list(APPEND OPTLIBS ${LibCURL_LIBRARIES}) -endif() - if (ENABLE_PERFTOOLS_DEBUG OR ENABLE_PERFTOOLS) # Just a no op to prevent CMake from complaining about manually-specified # ENABLE_PERFTOOLS_DEBUG or ENABLE_PERFTOOLS not being used if google @@ -165,6 +153,8 @@ set(brodeps include(TestBigEndian) test_big_endian(WORDS_BIGENDIAN) +include(CheckSymbolExists) +check_symbol_exists(htonll arpa/inet.h HAVE_BYTEORDER_64) include(OSSpecific) include(CheckTypes) @@ -174,6 +164,10 @@ include(MiscTests) include(PCAPTests) include(OpenSSLTests) include(CheckNameserCompat) +include(GetArchitecture) + +# Tell the plugin code that we're building as part of the main tree. +set(BRO_PLUGIN_INTERNAL_BUILD true CACHE INTERNAL "" FORCE) configure_file(${CMAKE_CURRENT_SOURCE_DIR}/config.h.in ${CMAKE_CURRENT_BINARY_DIR}/config.h) @@ -186,6 +180,7 @@ include_directories(${CMAKE_CURRENT_BINARY_DIR}) add_subdirectory(src) add_subdirectory(scripts) add_subdirectory(doc) +add_subdirectory(man) include(CheckOptionalBuildSources) @@ -238,10 +233,6 @@ message( "\n tcmalloc: ${USE_PERFTOOLS_TCMALLOC}" "\n debugging: ${USE_PERFTOOLS_DEBUG}" "\njemalloc: ${ENABLE_JEMALLOC}" - "\ncURL: ${USE_CURL}" - "\n" - "\nDataSeries: ${USE_DATASERIES}" - "\nElasticSearch: ${USE_ELASTICSEARCH}" "\n" "\n================================================================\n" ) diff --git a/Makefile b/Makefile index f4b2104c73..207ce72780 100644 --- a/Makefile +++ b/Makefile @@ -48,13 +48,16 @@ bindist: distclean: rm -rf $(BUILD) + $(MAKE) -C testing $@ test: @( cd testing && make ) test-all: test - test -d aux/broctl && ( cd aux/broctl && make test ) + test -d aux/broctl && ( cd aux/broctl && make test-all ) test -d aux/btest && ( cd aux/btest && make test ) + test -d aux/bro-aux && ( cd aux/bro-aux && make test ) + test -d aux/plugins && ( cd aux/plugins && make test-all ) configured: @test -d $(BUILD) || ( echo "Error: No build/ directory found. Did you run configure?" && exit 1 ) diff --git a/NEWS b/NEWS index b6754e1389..5e2ef52ca1 100644 --- a/NEWS +++ b/NEWS @@ -4,6 +4,55 @@ release. For an exhaustive list of changes, see the ``CHANGES`` file (note that submodules, such as BroControl and Broccoli, come with their own ``CHANGES``.) +Bro 2.4 (in progress) +===================== + +Dependencies +------------ + +New Functionality +----------------- + +- Bro now has support for external plugins that can extend its core + functionality, like protocol/file analysis, via shared libraries. + Plugins can be developed and distributed externally, and will be + pulled in dynamically at startup. Currently, a plugin can provide + custom protocol analyzers, file analyzers, log writers[TODO], input + readers[TODO], packet sources[TODO], and new built-in functions. A + plugin can furthermore hook into Bro's processing a number of places + to add custom logic. + + See https://www.bro.org/sphinx-git/devel/plugins.html for more + information on writing plugins. + +- Bro now has supoprt for the MySQL wire protocol. Activity gets + logged into mysql.log. + +- Bro's file analysis now supports reassembly of files that are not + transferred/seen sequentially. + +Changed Functionality +--------------------- + +- bro-cut has been rewritten in C, and is hence much faster. + +- File analysis + + * Removed ``fa_file`` record's ``mime_type`` and ``mime_types`` + fields. The events ``file_mime_type`` and ``file_mime_types`` + have been added which contain the same information. The + ``mime_type`` field of ``Files::Info`` also still has this info. + + * Removed ``Files::add_analyzers_for_mime_type`` function. + + * Removed ``offset`` parameter of the ``file_extraction_limit`` + event. Since file extraction now internally depends on file + reassembly for non-sequential files, "offset" can be obtained + with other information already available -- adding together + ``seen_bytes`` and ``missed_bytes`` fields of the ``fa_file`` + record gives the how many bytes have been written so far (i.e. + the "offset"). + Bro 2.3 ======= diff --git a/VERSION b/VERSION index 5cd0dd2628..9a536ef2e7 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.3-beta-18 +2.3-397 diff --git a/aux/binpac b/aux/binpac index ec1e052afd..8d56b507b8 160000 --- a/aux/binpac +++ b/aux/binpac @@ -1 +1 @@ -Subproject commit ec1e052afd5a8cd3d1d2cbb28fcd688018e379a5 +Subproject commit 8d56b507b8b804fa83f6637f3b1f198e696cd603 diff --git a/aux/bro-aux b/aux/bro-aux index 5721df4f5f..0b713c027d 160000 --- a/aux/bro-aux +++ b/aux/bro-aux @@ -1 +1 @@ -Subproject commit 5721df4f5f6fa84de6257cca6582a28e45831786 +Subproject commit 0b713c027d3efaaca50e5df995c02656175573cd diff --git a/aux/broccoli b/aux/broccoli index c2f5dd2cb7..d43cc790e5 160000 --- a/aux/broccoli +++ b/aux/broccoli @@ -1 +1 @@ -Subproject commit c2f5dd2cb7876158fdf9721aebd22567db840db1 +Subproject commit d43cc790e5b8709b5e032e52ad0e00936494739b diff --git a/aux/broctl b/aux/broctl index 1e55bff2df..8c9b87bc73 160000 --- a/aux/broctl +++ b/aux/broctl @@ -1 +1 @@ -Subproject commit 1e55bff2df49fe7dc3bd54e21050b39530eeb714 +Subproject commit 8c9b87bc73e1ddaa304e3d89028c1e7b95d37a91 diff --git a/aux/btest b/aux/btest index 4da1bd2403..93d4989ed1 160000 --- a/aux/btest +++ b/aux/btest @@ -1 +1 @@ -Subproject commit 4da1bd24038d4977e655f2b210f34e37f0b73b78 +Subproject commit 93d4989ed1537e4d143cf09d44077159f869a4b2 diff --git a/aux/plugins b/aux/plugins new file mode 160000 index 0000000000..ad600b5bdc --- /dev/null +++ b/aux/plugins @@ -0,0 +1 @@ +Subproject commit ad600b5bdcd56a2723e323c0f2c8e1708956ca4f diff --git a/cmake b/cmake index 0f301aa08a..1316c07f70 160000 --- a/cmake +++ b/cmake @@ -1 +1 @@ -Subproject commit 0f301aa08a970150195a2ea5b3ed43d2d98b35b3 +Subproject commit 1316c07f7059647b6c4a496ea36e4b83bb5d8f0f diff --git a/config.h.in b/config.h.in index 2d065f755e..755a9eee98 100644 --- a/config.h.in +++ b/config.h.in @@ -129,6 +129,9 @@ /* whether words are stored with the most significant byte first */ #cmakedefine WORDS_BIGENDIAN +/* whether htonll/ntohll is defined in */ +#cmakedefine HAVE_BYTEORDER_64 + /* ultrix can't hack const */ #cmakedefine NEED_ULTRIX_CONST_HACK #ifdef NEED_ULTRIX_CONST_HACK @@ -209,3 +212,14 @@ /* Common IPv6 extension structure */ #cmakedefine HAVE_IP6_EXT + +/* String with host architecture (e.g., "linux-x86_64") */ +#define HOST_ARCHITECTURE "@HOST_ARCHITECTURE@" + +/* String with extension of dynamic libraries (e.g., ".so") */ +#define DYNAMIC_PLUGIN_SUFFIX "@CMAKE_SHARED_MODULE_SUFFIX@" + +/* True if we're building outside of the main Bro source code tree. */ +#ifndef BRO_PLUGIN_INTERNAL_BUILD +#define BRO_PLUGIN_INTERNAL_BUILD @BRO_PLUGIN_INTERNAL_BUILD@ +#endif diff --git a/configure b/configure index 5af2f25c8f..2b1c568b26 100755 --- a/configure +++ b/configure @@ -24,6 +24,13 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --prefix=PREFIX installation directory [/usr/local/bro] --scriptdir=PATH root installation directory for Bro scripts [PREFIX/share/bro] + --localstatedir=PATH when using BroControl, path to store log files + and run-time data (within log/ and spool/ subdirs) + [PREFIX] + --spooldir=PATH when using BroControl, path to store run-time data + [PREFIX/spool] + --logdir=PATH when using BroControl, path to store log file + [PREFIX/logs] --conf-files-dir=PATH config files installation directory [PREFIX/etc] Optional Features: @@ -39,8 +46,6 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --disable-auxtools don't build or install auxiliary tools --disable-perftools don't try to build with Google Perftools --disable-python don't try to build python bindings for broccoli - --disable-dataseries don't use the optional DataSeries log writer - --disable-elasticsearch don't use the optional ElasticSearch log writer Required Packages in Non-Standard Locations: --with-openssl=PATH path to OpenSSL install root @@ -50,7 +55,6 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --with-flex=PATH path to flex executable --with-bison=PATH path to bison executable --with-perl=PATH path to perl executable - --with-libmagic=PATH path to libmagic install root Optional Packages in Non-Standard Locations: --with-geoip=PATH path to the libGeoIP install root @@ -63,9 +67,6 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --with-ruby-lib=PATH path to ruby library --with-ruby-inc=PATH path to ruby headers --with-swig=PATH path to SWIG executable - --with-dataseries=PATH path to DataSeries and Lintel libraries - --with-xml2=PATH path to libxml2 installation (for DataSeries) - --with-curl=PATH path to libcurl install root (for ElasticSearch) Packaging Options (for developers): --binary-package toggle special logic for binary packaging @@ -150,6 +151,15 @@ while [ $# -ne 0 ]; do append_cache_entry BRO_ETC_INSTALL_DIR PATH $optarg user_set_conffilesdir="true" ;; + --localstatedir=*) + append_cache_entry BRO_LOCAL_STATE_DIR PATH $optarg + ;; + --spooldir=*) + append_cache_entry BRO_SPOOL_DIR PATH $optarg + ;; + --logdir=*) + append_cache_entry BRO_LOG_DIR PATH $optarg + ;; --enable-debug) append_cache_entry ENABLE_DEBUG BOOL true ;; @@ -184,12 +194,6 @@ while [ $# -ne 0 ]; do --enable-ruby) append_cache_entry DISABLE_RUBY_BINDINGS BOOL false ;; - --disable-dataseries) - append_cache_entry DISABLE_DATASERIES BOOL true - ;; - --disable-elasticsearch) - append_cache_entry DISABLE_ELASTICSEARCH BOOL true - ;; --with-openssl=*) append_cache_entry OpenSSL_ROOT_DIR PATH $optarg ;; @@ -211,9 +215,6 @@ while [ $# -ne 0 ]; do --with-perl=*) append_cache_entry PERL_EXECUTABLE PATH $optarg ;; - --with-libmagic=*) - append_cache_entry LibMagic_ROOT_DIR PATH $optarg - ;; --with-geoip=*) append_cache_entry LibGeoIP_ROOT_DIR PATH $optarg ;; @@ -247,16 +248,6 @@ while [ $# -ne 0 ]; do --with-swig=*) append_cache_entry SWIG_EXECUTABLE PATH $optarg ;; - --with-dataseries=*) - append_cache_entry DataSeries_ROOT_DIR PATH $optarg - append_cache_entry Lintel_ROOT_DIR PATH $optarg - ;; - --with-xml2=*) - append_cache_entry LibXML2_ROOT_DIR PATH $optarg - ;; - --with-curl=*) - append_cache_entry LibCURL_ROOT_DIR PATH $optarg - ;; --binary-package) append_cache_entry BINARY_PACKAGING_MODE BOOL true ;; diff --git a/doc/_templates/layout.html b/doc/_templates/layout.html index 2f8ea02aff..3df56a12ff 100644 --- a/doc/_templates/layout.html +++ b/doc/_templates/layout.html @@ -10,7 +10,7 @@ {% endblock %} {% block header %} - {% endblock %} @@ -108,6 +108,6 @@ {% endblock %} {% block footer %} - {% endblock %} diff --git a/doc/conf.py.in b/doc/conf.py.in index 9720d12ade..4faebed3b8 100644 --- a/doc/conf.py.in +++ b/doc/conf.py.in @@ -21,7 +21,7 @@ sys.path.insert(0, os.path.abspath('sphinx_input/ext')) # ----- Begin of BTest configuration. ----- btest = os.path.abspath("@CMAKE_SOURCE_DIR@/aux/btest") -brocut = os.path.abspath("@CMAKE_SOURCE_DIR@/aux/bro-aux/bro-cut") +brocut = os.path.abspath("@CMAKE_SOURCE_DIR@/build/aux/bro-aux/bro-cut") bro = os.path.abspath("@CMAKE_SOURCE_DIR@/build/src") os.environ["PATH"] += (":%s:%s/sphinx:%s:%s" % (btest, btest, bro, brocut)) diff --git a/doc/devel/plugins.rst b/doc/devel/plugins.rst new file mode 100644 index 0000000000..c703345891 --- /dev/null +++ b/doc/devel/plugins.rst @@ -0,0 +1,447 @@ + +=================== +Writing Bro Plugins +=================== + +Bro is internally moving to a plugin structure that enables extending +the system dynamically, without modifying the core code base. That way +custom code remains self-contained and can be maintained, compiled, +and installed independently. Currently, plugins can add the following +functionality to Bro: + + - Bro scripts. + + - Builtin functions/events/types for the scripting language. + + - Protocol analyzers. + + - File analyzers. + + - Packet sources and packet dumpers. + + - Logging framework backends. + + - Input framework readers. + +A plugin's functionality is available to the user just as if Bro had +the corresponding code built-in. Indeed, internally many of Bro's +pieces are structured as plugins as well, they are just statically +compiled into the binary rather than loaded dynamically at runtime. + +Quick Start +=========== + +Writing a basic plugin is quite straight-forward as long as one +follows a few conventions. In the following we walk a simple example +plugin that adds a new built-in function (bif) to Bro: we'll add +``rot13(s: string) : string``, a function that rotates every character +in a string by 13 places. + +Generally, a plugin comes in the form of a directory following a +certain structure. To get started, Bro's distribution provides a +helper script ``aux/bro-aux/plugin-support/init-plugin`` that creates +a skeleton plugin that can then be customized. Let's use that:: + + # mkdir rot13-plugin + # cd rot13-plugin + # init-plugin Demo Rot13 + +As you can see the script takes two arguments. The first is a +namespace the plugin will live in, and the second a descriptive name +for the plugin itself. Bro uses the combination of the two to identify +a plugin. The namespace serves to avoid naming conflicts between +plugins written by independent developers; pick, e.g., the name of +your organisation. The namespace ``Bro`` is reserved for functionality +distributed by the Bro Project. In our example, the plugin will be +called ``Demo::Rot13``. + +The ``init-plugin`` script puts a number of files in place. The full +layout is described later. For now, all we need is +``src/rot13.bif``. It's initially empty, but we'll add our new bif +there as follows:: + + # cat src/rot13.bif + module CaesarCipher; + + function rot13%(s: string%) : string + %{ + char* rot13 = copy_string(s->CheckString()); + + for ( char* p = rot13; *p; p++ ) + { + char b = islower(*p) ? 'a' : 'A'; + *p = (*p - b + 13) % 26 + b; + } + + BroString* bs = new BroString(1, reinterpret_cast(rot13), + strlen(rot13)); + return new StringVal(bs); + %} + +The syntax of this file is just like any other ``*.bif`` file; we +won't go into it here. + +Now we can already compile our plugin, we just need to tell the +configure script put in place by ``init-plugin`` where the Bro source +tree is located (Bro needs to have been built there first):: + + # ./configure --bro-dist=/path/to/bro/dist && make + [... cmake output ...] + +Now our ``rot13-plugin`` directory has everything that it needs +for Bro to recognize it as a dynamic plugin. Once we point Bro to it, +it will pull it in automatically, as we can check with the ``-N`` +option:: + + # export BRO_PLUGIN_PATH=/path/to/rot13-plugin + # bro -N + [...] + Plugin: Demo::Rot13 - (dynamic, version 1) + [...] + +That looks quite good, except for the dummy description that we should +replace with something nicer so that users will know what our plugin +is about. We do this by editing the ``config.description`` line in +``src/Plugin.cc``, like this:: + + [...] + plugin::Configuration Configure() + { + plugin::Configuration config; + config.name = "Demo::Rot13"; + config.description = "Caesar cipher rotating a string's characters by 13 places."; + config.version.major = 1; + config.version.minor = 0; + return config; + } + [...] + + # make + [...] + # bro -N | grep Rot13 + Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) + +Better. Bro can also show us what exactly the plugin provides with the +more verbose option ``-NN``:: + + # bro -NN + [...] + Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) + [Function] CaesarCipher::rot13 + [...] + +There's our function. Now let's use it:: + + # bro -e 'print CaesarCipher::rot13("Hello")' + Uryyb + +It works. We next install the plugin along with Bro itself, so that it +will find it directly without needing the ``BRO_PLUGIN_PATH`` +environment variable. If we first unset the variable, the function +will no longer be available:: + + # unset BRO_PLUGIN_PATH + # bro -e 'print CaesarCipher::rot13("Hello")' + error in , line 1: unknown identifier CaesarCipher::rot13, at or near "CaesarCipher::rot13" + +Once we install it, it works again:: + + # make install + # bro -e 'print CaesarCipher::rot13("Hello")' + Uryyb + +The installed version went into +``/lib/bro/plugins/Demo_Rot13``. + +We can distribute the plugin in either source or binary form by using +the Makefile's ``sdist`` and ``bdist`` target, respectively. Both +create corrsponding tarballs:: + + # make sdist + [...] + Source distribution in build/sdist/Demo_Rot13.tar.gz + + # make bdist + [...] + Binary distribution in build/Demo_Rot13-darwin-x86_64.tar.gz + +The source archive will contain everything in the plugin directory +except any generated files. The binary archive will contain anything +needed to install and run the plugin, i.e., just what ``make install`` +puts into place as well. As the binary distribution is +platform-dependent, its name includes the OS and architecture the +plugin was built on. + +Plugin Directory Layout +======================= + +A plugin's directory needs to follow a set of conventions so that Bro +(1) recognizes it as a plugin, and (2) knows what to load. While +``init-plugin`` takes care of most of this, the following is the full +story. We'll use ```` to represent a plugin's top-level +directory. + +``/__bro_plugin__`` + A file that marks a directory as containing a Bro plugin. The file + must exist, and its content must consist of a single line with the + qualified name of the plugin (e.g., "Demo::Rot13"). + +``/lib/--.so`` + The shared library containing the plugin's compiled code. Bro will + load this in dynamically at run-time if OS and architecture match + the current platform. + +``scripts/`` + A directory with the plugin's custom Bro scripts. When the plugin + gets activated, this directory will be automatically added to + ``BROPATH``, so that any scripts/modules inside can be + "@load"ed. + +``scripts``/__load__.bro + A Bro script that will be loaded immediately when the plugin gets + activated. See below for more information on activating plugins. + +``lib/bif/`` + Directory with auto-generated Bro scripts that declare the plugin's + bif elements. The files here are produced by ``bifcl``. + +By convention, a plugin should put its custom scripts into sub folders +of ``scripts/``, i.e., ``scripts//