diff --git a/.gitmodules b/.gitmodules index 24375ce23d..91f39e3d04 100644 --- a/.gitmodules +++ b/.gitmodules @@ -22,3 +22,6 @@ [submodule "aux/plugins"] path = aux/plugins url = git://git.bro.org/bro-plugins +[submodule "aux/broker"] + path = aux/broker + url = git://git.bro.org/broker diff --git a/CHANGES b/CHANGES index cad5e26988..f46a708870 100644 --- a/CHANGES +++ b/CHANGES @@ -1,4 +1,1164 @@ +2.4-406 | 2016-03-11 14:27:47 -0800 + + * Add NetControl and OpenFlow frameworks. (Johanna Amann) + +2.4-313 | 2016-03-08 07:47:57 -0800 + + * Remove old string functions in C++ code. This removes the + functions: strcasecmp_n, strchr_n, and strrchr_n. (Johanna Amann) + +2.4-307 | 2016-03-07 13:33:45 -0800 + + * Add "disable_analyzer_after_detection" and remove + "skip_processing_after_detection". Addresses BIT-1545. + (Aaron Eppert & Johanna Amann) + + * Add bad_HTTP_request_with_version weird (William Glodek) + +2.4-299 | 2016-03-04 12:51:55 -0800 + + * More detailed installation instructions for FreeBSD 9.X. (Johanna Amann) + + * Update CMake OpenSSL checks. (Johanna Amann) + + * "SUBSCRIBE" is a valid SIP. message per RFC 3265. Addresses + BIT-1529. (Johanna Amann) + + * Update documentation for connection log's RSTR. Addresses BIT-1535 + (Johanna Amann) + +2.4-284 | 2016-02-17 14:12:15 -0800 + + * Fix sometimes failing dump-events test. (Johanna Amann) + +2.4-282 | 2016-02-13 10:48:21 -0800 + + * Add missing break in in StartTLS case of IRC analyzer. Found by + Aaron Eppert. (Johanna Amann) + +2.4-280 | 2016-02-13 10:40:16 -0800 + + * Fix memory leaks in stats.cc and smb.cc. (Johanna Amann) + +2.4-278 | 2016-02-12 18:53:35 -0800 + + * Better multi-space separator handline. (Mark Taylor & Johanna Amann) + +2.4-276 | 2016-02-10 21:29:33 -0800 + + * Allow IRC commands to not have parameters. (Mark Taylor) + +2.4-272 | 2016-02-08 14:27:58 -0800 + + * fix memory leaks in find_all() and IRC analyzer. (Dirk Leinenbach) + +2.4-270 | 2016-02-08 13:00:57 -0800 + + * Removed duplicate parameter for IRC "QUIT" event handler. (Mark Taylor) + +2.4-267 | 2016-02-01 12:38:32 -0800 + + * Add testcase for CVE-2015-3194. (Johanna Amann) + + * Fix portability issue with use of mktemp. (Daniel Thayer) + +2.4-260 | 2016-01-28 08:05:27 -0800 + + * Correct irc_privmsg_message event handling bug. (Mark Taylor) + + * Update copyright year for Sphinx. (Johanna Amann) + +2.4-253 | 2016-01-20 17:41:20 -0800 + + * Support of RadioTap encapsulation for 802.11 (Seth Hall) + + Radiotap support should be fully functional with Radiotap + packets that include IPv4 and IPv6. Other radiotap packets are + silently ignored. + +2.4-247 | 2016-01-19 10:19:48 -0800 + + * Fixing C++11 compiler warnings. (Seth Hall) + + * Updating plugin documentation building. (Johanna Amann) + +2.4-238 | 2016-01-15 12:56:33 -0800 + + * Add HTTP version information to HTTP log file. (Aaron Eppert) + + * Add NOTIFY as a valid SIP message, per RFC 3265. (Aaron Eppert) + + * Improve HTTP parser's handling of requests that don't have a URI. + (William Glodek/Robin Sommer) + + * Fix crash when deleting non existing record member. Addresses + BIT-1519. (Johanna Amann) + +2.4-228 | 2015-12-19 13:40:09 -0800 + + * Updating BroControl submodule. + +2.4-227 | 2015-12-18 17:47:24 -0800 + + * Update host name in windows-version-detection.bro. (Aaron Eppert) + + * Update installation instructions to mention OpenSSL dependency for + newer OS X version. (Johanna Amann) + + * Change a stale bro-ids.org to bro.org. (Johanna Amann) + + * StartTLS support for IRC. (Johanna Amann) + + * Adding usage guard to canonifier script. (Robin Sommer) + +2.4-217 | 2015-12-04 16:50:46 -0800 + + * SIP scripts code cleanup. (Seth Hall) + + - Daniel Guerra pointed out a type issue for SIP request and + response code length fields which is now corrected. + + - Some redundant code was removed. + + - if/else tree modified to use switch instead. + +2.4-214 | 2015-12-04 16:40:15 -0800 + + * Delaying BinPAC initializaton until afte plugins have been + activated. (Robin Sommer) + +2.4-213 | 2015-12-04 15:25:48 -0800 + + * Use better data structure for storing BPF filters. (Robin Sommer) + +2.4-211 | 2015-11-17 13:28:29 -0800 + + * Making cluster reconnect timeout configurable. (Robin Sommer) + + * Bugfix for child process' communication loop. (Robin Sommer) + +2.4-209 | 2015-11-16 07:31:22 -0800 + + * Updating submodule(s). + +2.4-207 | 2015-11-10 13:34:42 -0800 + + * Fix to compile with OpenSSL that has SSLv3 disalbed. (Christoph + Pietsch) + + * Fix potential race condition when logging VLAN info to conn.log. + (Daniel Thayer) + +2.4-201 | 2015-10-27 16:11:15 -0700 + + * Updating NEWS. (Robin Sommer) + +2.4-200 | 2015-10-26 16:57:39 -0700 + + * Adding missing file. (Robin Sommer) + +2.4-199 | 2015-10-26 16:51:47 -0700 + + * Fix problem with the JSON Serialization code. (Aaron Eppert) + +2.4-188 | 2015-10-26 14:11:21 -0700 + + * Extending rexmit_inconsistency() event to receive an additional + parameter with the packet's TCP flags, if available. (Robin + Sommer) + +2.4-187 | 2015-10-26 13:43:32 -0700 + + * Updating NEWS for new plugins. (Robin Sommer) + +2.4-186 | 2015-10-23 15:07:06 -0700 + + * Removing pcap options for AF_PACKET support. Addresses BIT-1363. + (Robin Sommer) + + * Correct a typo in controller.bro documentation. (Daniel Thayer) + + * Extend SSL DPD signature to allow alert before server_hello. + (Johanna Amann) + + * Make join_string_vec work with vectors containing empty elements. + (Johanna Amann) + + * Fix support for HTTP CONNECT when server adds headers to response. + (Eric Karasuda). + + * Load static CA list for validation tests too. (Johanna Amann) + + * Remove cluster certificate validation script. (Johanna Amann) + + * Fix a bug in diff-remove-x509-names canonifier. (Daniel Thayer) + + * Fix test canonifiers in scripts/policy/protocols/ssl. (Daniel + Thayer) + +2.4-169 | 2015-10-01 17:21:21 -0700 + + * Fixed parsing of V_ASN1_GENERALIZEDTIME timestamps in x509 + certificates. (Yun Zheng Hu) + + * Improve X509 end-of-string-check code. (Johanna Amann) + + * Refactor X509 generalizedtime support and test. (Johanna Amann) + + * Fix case of offset=-1 (EOF) for RAW reader. Addresses BIT-1479. + (Johanna Amann) + + * Improve a number of test canonifiers. (Daniel Thayer) + + * Remove unnecessary use of TEST_DIFF_CANONIFIER. (Daniel Thayer) + + * Fixed some test canonifiers to read only from stdin + + * Remove unused test canonifier scripts. (Daniel Thayer) + + * A potpourri of updates and improvements across the documentation. + (Daniel Thayer) + + * Add configure option to disable Broker Python bindings. Also + improve the configure summary output to more clearly show whether + or not Broker Python bindings will be built. (Daniel Thayer) + +2.4-131 | 2015-09-11 12:16:39 -0700 + + * Add README.rst symlink. Addresses BIT-1413 (Vlad Grigorescu) + +2.4-129 | 2015-09-11 11:56:04 -0700 + + * hash-all-files.bro depends on base/files/hash (Richard van den Berg) + + * Make dns_max_queries redef-able, and bump default to 25. Addresses + BIT-1460 (Vlad Grigorescu) + +2.4-125 | 2015-09-03 20:10:36 -0700 + + * Move SIP analyzer to flowunit instead of datagram Addresses + BIT-1458 (Vlad Grigorescu) + +2.4-122 | 2015-08-31 14:39:41 -0700 + + * Add a number of out-of-bound checks to layer 2 code. Addresses + BIT-1463 (Johanna Amann) + + * Fix error in 2.4 release notes regarding SSH events. (Robin + Sommer) + +2.4-118 | 2015-08-31 10:55:29 -0700 + + * Fix FreeBSD build errors (Johanna Amann) + +2.4-117 | 2015-08-30 22:16:24 -0700 + + * Fix initialization of a pointer in RDP analyzer. (Daniel + Thayer/Robin Sommer) + +2.4-115 | 2015-08-30 21:57:35 -0700 + + * Enable Bro to leverage packet fanout mode on Linux. (Kris + Nielander). + + ## Toggle whether to do packet fanout (Linux-only). + const Pcap::packet_fanout_enable = F &redef; + + ## If packet fanout is enabled, the id to sue for it. This should be shared amongst + ## worker processes processing the same socket. + const Pcap::packet_fanout_id = 0 &redef; + + ## If packet fanout is enabled, whether packets are to be defragmented before + ## fanout is applied. + const Pcap::packet_fanout_defrag = T &redef; + + * Allow libpcap buffer size to be set via configuration. (Kris Nielander) + + ## Number of Mbytes to provide as buffer space when capturing from live + ## interfaces. + const Pcap::bufsize = 128 &redef; + + * Move the pcap-related script-level identifiers into the new Pcap + namespace. (Robin Sommer) + + snaplen -> Pcap::snaplen + precompile_pcap_filter() -> Pcap::precompile_pcap_filter() + install_pcap_filter() -> Pcap::install_pcap_filter() + pcap_error() -> Pcap::pcap_error() + + +2.4-108 | 2015-08-30 20:14:31 -0700 + + * Update Base64 decoding. (Jan Grashoefer) + + - A new built-in function, decode_base64_conn() for Base64 + decoding. It works like decode_base64() but receives an + additional connection argument that will be used for + reporting decoding errors into weird.log (instead of + reporter.log). + + - FTP, POP3, and HTTP analyzers now likewise log Base64 + decoding errors to weird.log. + + - The built-in functions decode_base64_custom() and + encode_base64_custom() are now deprecated. Their + functionality is provided directly by decode_base64() and + encode_base64(), which take an optional parameter to change + the Base64 alphabet. + + * Fix potential crash if TCP header was captured incompletely. + (Robin Sommer) + +2.4-103 | 2015-08-29 10:51:55 -0700 + + * Make ASN.1 date/time parsing more robust. (Johanna Amann) + + * Be more permissive on what characters we accept as an unquoted + multipart boundary. Addresses BIT-1459. (Johanna Amann) + +2.4-99 | 2015-08-25 07:56:57 -0700 + + * Add ``Q`` and update ``I`` documentation for connection history + field. Addresses BIT-1466. (Vlad Grigorescu) + +2.4-96 | 2015-08-21 17:37:56 -0700 + + * Update SIP analyzer. (balintm) + + - Allows space on both sides of ':'. + - Require CR/LF after request/reply line. + +2.4-94 | 2015-08-21 17:31:32 -0700 + + * Add file type detection support for video/MP2T. (Mike Freemon) + +2.4-93 | 2015-08-21 17:23:39 -0700 + + * Make plugin install honor DESTDIR= convention. (Jeff Barber) + +2.4-89 | 2015-08-18 07:53:36 -0700 + + * Fix diff-canonifier-external to use basename of input file. + (Daniel Thayer) + +2.4-87 | 2015-08-14 08:34:41 -0700 + + * Removing the yielding_teredo_decapsulation option. (Robin Sommer) + +2.4-86 | 2015-08-12 17:02:24 -0700 + + * Make Teredo DPD signature more precise. (Martina Balint) + +2.4-84 | 2015-08-10 14:44:39 -0700 + + * Add hook 'HookSetupAnalyzerTree' to allow plugins access to a + connection's initial analyzer tree for customization. (James + Swaro) + + * Plugins now look for a file "__preload__.bro" in the top-level + script directory. If found, they load it first, before any scripts + defining BiF elements. This can be used to define types that the + BiFs already depend on (like a custom type for an event argument). + (Robin Sommer) + +2.4-81 | 2015-08-08 07:38:42 -0700 + + * Fix a test that is failing very frequently. (Daniel Thayer) + +2.4-78 | 2015-08-06 22:25:19 -0400 + + * Remove build dependency on Perl (now requiring Python instad). + (Daniel Thayer) + + * CID 1314754: Fixing unreachable code in RSH analyzer. (Robin + Sommer) + + * CID 1312752: Add comment to mark 'case' fallthrough as ok. (Robin + Sommer) + + * CID 1312751: Removing redundant assignment. (Robin Sommer) + +2.4-73 | 2015-07-31 08:53:49 -0700 + + * BIT-1429: SMTP logs now include CC: addresses. (Albert Zaharovits) + +2.4-70 | 2015-07-30 07:23:44 -0700 + + * Updated detection of Flash and AdobeAIR. (Jan Grashoefer) + + * Adding tests for Flash version parsing and browser plugin + detection. (Robin Sommer) + +2.4-63 | 2015-07-28 12:26:37 -0700 + + * Updating submodule(s). + +2.4-61 | 2015-07-28 12:13:39 -0700 + + * Renaming config.h to bro-config.h. (Robin Sommer) + +2.4-58 | 2015-07-24 15:06:07 -0700 + + * Add script protocols/conn/vlan-logging.bro to record VLAN data in + conn.log. (Aaron Brown) + + * Add field "vlan" and "inner_vlan" to connection record. (Aaron + Brown) + + * Save the inner vlan in the Packet object for Q-in-Q setups. (Aaron + Brown) + + * Increasing plugin API version for recent packet source changes. + (Robin Sommer) + + * Slightly earlier protocol confirmation for POP3. (Johanna Amann) + +2.4-46 | 2015-07-22 10:56:40 -0500 + + * Fix broker python bindings install location to track --prefix. + (Jon Siwek) + +2.4-45 | 2015-07-21 15:19:43 -0700 + + * Enabling Broker by default. This means CAF is now a required + dependency, altjough for now at least, there's still a switch + --disable-broker to turn it off. + + * Requiring a C++11 compiler, and turning on C++11 support. (Robin + Sommer) + + * Tweaking the listing of hooks in "bro -NN" for consistency. (Robin + Sommer) + +2.4-41 | 2015-07-21 08:35:17 -0700 + + * Fixing compiler warning. (Robin Sommer) + + * Updates to IANA TLS registry. (Johanna Amann) + +2.4-38 | 2015-07-20 15:30:35 -0700 + + * Refactor code to use a common Packet type throught. (Jeff + Barber/Robin Sommer) + + * Extend parsing layer 2 and keeping track of layer 3 protoco. (Jeff Barber) + + * Add a raw_packet() event that generated for all packets and + include layer 2 information. (Jeff Barber) + +2.4-27 | 2015-07-15 13:31:49 -0700 + + * Fix race condition in intel test. (Johanna Amann) + +2.4-24 | 2015-07-14 08:04:11 -0700 + + * Correct Perl package name on FreeBSD in documentation.(Justin Azoff) + + * Adding an environment variable to BTest configuration for external + scripts. (Robin Sommer) + +2.4-20 | 2015-07-03 10:40:21 -0700 + + * Adding a weird for when truncated packets lead TCP reassembly to + ignore content. (Robin Sommer) + +2.4-19 | 2015-07-03 09:04:54 -0700 + + * A set of tests exercising IP defragmentation and TCP reassembly. + (Robin Sommer) + +2.4-17 | 2015-06-28 13:02:41 -0700 + + * BIT-1314: Add detection for Quantum Insert attacks. The TCP + reassembler can now keep a history of old TCP segments using the + tcp_max_old_segments option. An overlapping segment with different + data will then generate an rexmit_inconsistency event. The default + for tcp_max_old_segments is zero, which disabled any additional + buffering. (Yun Zheng Hu/Robin Sommer) + +2.4-14 | 2015-06-28 12:30:12 -0700 + + * BIT-1400: Allow '<' and '>' in MIME multipart boundaries. The spec + doesn't actually seem to permit these, but they seem to occur in + the wild. (Jon Siwek) + +2.4-12 | 2015-06-28 12:21:11 -0700 + + * BIT-1399: Trying to decompress deflated HTTP content even when + zlib headers are missing. (Seth Hall) + +2.4-10 | 2015-06-25 07:11:17 -0700 + + * Correct a name used in a header identifier (Justin Azoff) + +2.4-8 | 2015-06-24 07:50:50 -0700 + + * Restore the --load-seeds cmd-line option and enable the short + options -G/-H for --load-seeds/--save-seeds. (Daniel Thayer) + +2.4-6 | 2015-06-19 16:26:40 -0700 + + * Generate protocol confirmations for Modbus, making it appear as a + confirmed service in conn.log. (Seth Hall) + + * Put command line options in alphabetical order. (Daniel Thayer) + + * Removing dead code for no longer supported -G switch. (Robin + Sommer) (Robin Sommer) + +2.4 | 2015-06-09 07:30:53 -0700 + + * Release 2.4. + + * Fixing tiny thing in NEWS. (Robin Sommer) + +2.4-beta-42 | 2015-06-08 09:41:39 -0700 + + * Fix reporter errors with GridFTP traffic. (Robin Sommer) + +2.4-beta-40 | 2015-06-06 08:20:52 -0700 + + * PE Analyzer: Change how we calculate the rva_table size. (Vlad Grigorescu) + +2.4-beta-39 | 2015-06-05 09:09:44 -0500 + + * Fix a unit test to check for Broker requirement. (Jon Siwek) + +2.4-beta-38 | 2015-06-04 14:48:37 -0700 + + * Test for Broker termination. (Robin Sommer) + +2.4-beta-37 | 2015-06-04 07:53:52 -0700 + + * BIT-1408: Improve I/O loop and Broker IOSource. (Jon Siwek) + +2.4-beta-34 | 2015-06-02 10:37:22 -0700 + + * Add signature support for F4M files. (Seth Hall) + +2.4-beta-32 | 2015-06-02 09:43:31 -0700 + + * A larger set of documentation updates, fixes, and extentions. + (Daniel Thayer) + +2.4-beta-14 | 2015-06-02 09:16:44 -0700 + + * Add memleak btest for attachments over SMTP. (Vlad Grigorescu) + + * BIT-1410: Fix flipped tx_hosts and rx_hosts in files.log. Reported + by Ali Hadi. (Vlad Grigorescu) + + * Updating the Mozilla root certs. (Seth Hall) + + * Updates for the urls.bro script. Fixes BIT-1404. (Seth Hall) + +2.4-beta-6 | 2015-05-28 13:20:44 -0700 + + * Updating submodule(s). + +2.4-beta-2 | 2015-05-26 08:58:37 -0700 + + * Fix segfault when DNS is not available. Addresses BIT-1387. (Frank + Meier and Robin Sommer) + +2.4-beta | 2015-05-07 21:55:31 -0700 + + * Release 2.4-beta. + + * Update local-compat.test (Johanna Amann) + +2.3-913 | 2015-05-06 09:58:00 -0700 + + * Add /sbin to PATH in btest.cfg and remove duplicate default_path. + (Daniel Thayer) + +2.3-911 | 2015-05-04 09:58:09 -0700 + + * Update usage output and list of command line options. (Daniel + Thayer) + + * Fix to ssh/geo-data.bro for unset directions. (Vlad Grigorescu) + + * Improve SIP logging and remove reporter messages. (Seth Hall) + +2.3-905 | 2015-04-29 17:01:30 -0700 + + * Improve SIP logging and remove reporter messages. (Seth Hall) + +2.3-903 | 2015-04-27 17:27:59 -0700 + + * BIT-1350: Improve record coercion type checking. (Jon Siwek) + +2.3-901 | 2015-04-27 17:25:27 -0700 + + * BIT-1384: Remove -O (optimize scripts) command-line option, which + hadn't been working for a while already. (Jon Siwek) + +2.3-899 | 2015-04-27 17:22:42 -0700 + + * Fix the -J/--set-seed cmd-line option. (Daniel Thayer) + + * Remove unused -l, -L, and -Z cmd-line options. (Daniel Thayer) + +2.3-892 | 2015-04-27 08:22:22 -0700 + + * Fix typos in the Broker BIF documentation. (Daniel Thayer) + + * Update installation instructions and remove outdated references. + (Johanna Amann) + + * Easier support for systems with tcmalloc_minimal installed. (Seth + Hall) + +2.3-884 | 2015-04-23 12:30:15 -0500 + + * Fix some outdated documentation unit tests. (Jon Siwek) + +2.3-883 | 2015-04-23 07:10:36 -0700 + + * Fix -N option to work with builtin plugins as well. (Robin Sommer) + +2.3-882 | 2015-04-23 06:59:40 -0700 + + * Add missing .pac dependencies for some binpac analyzer targets. + (Jon Siwek) + +2.3-879 | 2015-04-22 10:38:07 -0500 + + * Fix compile errors. (Jon Siwek) + +2.3-878 | 2015-04-22 08:21:23 -0700 + + * Fix another compiler warning in DTLS. (Johanna Amann) + +2.3-877 | 2015-04-21 20:14:16 -0700 + + * Adding missing include. (Robin Sommer) + +2.3-876 | 2015-04-21 16:40:10 -0700 + + * Attempt at fixing a potential std::length_error exception in RDP + analyzer. Addresses BIT-1337. (Robin Sommer) + + * Fixing compile problem caused by overeager factorization. (Robin + Sommer) + +2.3-874 | 2015-04-21 16:09:20 -0700 + + * Change details of escaping when logging/printing. (Seth Hall/Robin + Sommer) + + - Log files now escape non-printable characters consistently + as "\xXX'. Furthermore, backslashes are escaped as "\\", + making the representation fully reversible. + + - When escaping via script-level functions (escape_string, + clean), we likewise now escape consistently with "\xXX" and + "\\". + + - There's no "alternative" output style anymore, i.e., fmt() + '%A' qualifier is gone. + + Addresses BIT-1333. + + * Remove several BroString escaping methods that are no longer + useful. (Seth Hall) + +2.3-864 | 2015-04-21 15:24:02 -0700 + + * A SIP protocol analyzer. (Vlad Grigorescu) + + Activity gets logged into sip.log. It generates the following + events: + + event sip_request(c: connection, method: string, original_URI: string, version: string); + event sip_reply(c: connection, version: string, code: count, reason: string); + event sip_header(c: connection, is_orig: bool, name: string, value: string); + event sip_all_headers(c: connection, is_orig: bool, hlist: mime_header_list); + event sip_begin_entity(c: connection, is_orig: bool); + event sip_end_entity(c: connection, is_orig: bool); + + The analyzer support SIP over UDP currently. + + * BIT-1343: Factor common ASN.1 code from RDP, SNMP, and Kerberos + analyzers. (Jon Siwek/Robin Sommer) + +2.3-838 | 2015-04-21 13:40:12 -0700 + + * BIT-1373: Fix vector index assignment reference count bug. (Jon Siwek) + +2.3-836 | 2015-04-21 13:37:31 -0700 + + * Fix SSH direction field being unset. Addresses BIT-1365. (Vlad + Grigorescu) + +2.3-835 | 2015-04-21 16:36:00 -0500 + + * Clarify Broker examples. (Jon Siwek) + +2.3-833 | 2015-04-21 12:38:32 -0700 + + * A Kerberos protocol analyzer. (Vlad Grigorescu) + + Activity gets logged into kerberos.log. It generates the following + events: + + event krb_as_request(c: connection, msg: KRB::KDC_Request); + event krb_as_response(c: connection, msg: KRB::KDC_Response); + event krb_tgs_request(c: connection, msg: KRB::KDC_Request); + event krb_tgs_response(c: connection, msg: KRB::KDC_Response); + event krb_ap_request(c: connection, ticket: KRB::Ticket, opts: KRB::AP_Options); + event krb_priv(c: connection, is_orig: bool); + event krb_safe(c: connection, is_orig: bool, msg: KRB::SAFE_Msg); + event krb_cred(c: connection, is_orig: bool, tickets: KRB::Ticket_Vector); + event krb_error(c: connection, msg: KRB::Error_Msg); + +2.3-793 | 2015-04-20 20:51:00 -0700 + + * Add decoding of PROXY-AUTHORIZATION header to HTTP analyze, + treating it the same as AUTHORIZATION. (Josh Liburdi) + + * Remove deprecated fields "hot" and "addl" from the connection + record. Remove the functions append_addl() and + append_addl_marker(). (Robin Sommer) + + * Removing the NetFlow analyzer, which hasn't been used anymore + since then corresponding command-line option went away. (Robin + Sommer) + +2.3-787 | 2015-04-20 19:15:23 -0700 + + * A file analyzer for Portable Executables. (Vlad Grigorescu/Seth + Hall). + + Activity gets logged into pe.log. It generates the following + events: + + event pe_dos_header(f: fa_file, h: PE::DOSHeader); + event pe_dos_code(f: fa_file, code: string); + event pe_file_header(f: fa_file, h: PE::FileHeader); + event pe_optional_header(f: fa_file, h: PE::OptionalHeader); + event pe_section_header(f: fa_file, h: PE::SectionHeader); + +2.3-741 | 2015-04-20 13:12:39 -0700 + + * API changes to file analysis mime type detection. Removed + "file_mime_type" and "file_mime_types" event, replacing them with + a new event called "file_metadata_inferred". Addresses BIT-1368. + (Jon Siwek) + + * A large series of improvements for file type identification. This + inludes a many signature updates (new types, cleanup, performance + improvments) and splitting out signatures into subfiles. (Seth + Hall) + + * Fix an issue with files having gaps before the bof_buffer is + filled, which could lead to file type identification not working + correctly. (Seth Hall) + + * Fix an issue with packet loss in HTTP file reporting for file type + identification wasn't working correctly zero-length bodies. (Seth + Hall) + + * X.509 certificates are now populating files.log with the mime type + application/pkix-cert. (Seth Hall) + + * Normalized some FILE_ANALYSIS debug messages. (Seth Hall) + +2.3-725 | 2015-04-20 12:54:54 -0700 + + * Updating submodule(s). + +2.3-724 | 2015-04-20 14:11:02 -0500 + + * Fix uninitialized field in raw input reader. (Jon Siwek) + +2.3-722 | 2015-04-20 12:59:03 -0500 + + * Remove unneeded documentation cross-referencing. (Jon Siwek) + +2.3-721 | 2015-04-20 12:47:05 -0500 + + * BIT-1380: Improve Broxygen output of &default expressions. + (Jon Siwek) + +2.3-720 | 2015-04-17 14:18:26 -0700 + + * Updating NEWS. + +2.3-716 | 2015-04-17 13:06:37 -0700 + + * Add seeking functionality to raw reader. One can now add an option + "offset" to the config map. Positive offsets are interpreted to be + from the beginning of the file, negative from the end of the file + (-1 is end of file). Only works for raw reader in streaming or + manual mode. Does not work with executables. Addresses BIT-985. + (Johanna Amann) + + * Allow setting packet and byte thresholds for connections. (Johanna Amann) + + This extends the ConnSize analyzer to be able to raise events when + each direction of a connection crosses a certain amount of bytes + or packets. + + Thresholds are set using: + - set_conn_bytes_threshold(c$id, [num-bytes], [direction]); + - set_conn_packets_threshold(c$id, [num-packets], [direction]); + + They raise the events, respectively: + - event conn_bytes_threshold_crossed(c: connection, threshold: count, is_orig: bool) + - event conn_packets_threshold_crossed(c: connection, threshold: count, is_orig: bool) + + Current thresholds can be examined using get_conn_bytes_threshold() + and get_conn_packets_threshold(). + + Only one threshold can be set per connection. + + * Add high-level API for packet/bytes thresholding in + base/protocols/conn/thresholds.bro that holds lists of thresholds + and raises an event for each threshold exactly once. (Johanna + Amann) + + * Fix a bug where child packet analyzers of the TCP analyzer + where not found using FindChild. + + * Update GridFTP analyzer to use connection thresholding instead of + polling. (Johanna Amann) + +2.3-709 | 2015-04-17 12:37:32 -0700 + + * Fix addressing the dreaded "internal error: unknown msg type 115 + in Poll()". (Jon Siwek) + + This patch removes the error handling code for overload conditions + in the main process that could cause trouble down the road. The + "chunked_io_buffer_soft_cap" script variable can now tune when the + client process begins shutting down peer connections, and the + default setting is now double what it used to be. Addresses + BIT-1376. + +2.3-707 | 2015-04-17 10:57:59 -0500 + + * Add more info about Broker to NEWS. (Jon Siwek) + +2.3-705 | 2015-04-16 08:16:45 -0700 + + * Update Mozilla CA list. (Johanna Amann) + + * Update tests to have them keep using older certificates where + appropiate. (Johanna Amann) + +2.3-699 | 2015-04-16 09:51:58 -0500 + + * Fix the to_count function to use strtoull versus strtoll. + (Jon Siwek) + +2.3-697 | 2015-04-15 09:51:15 -0700 + + * Removing error check verifying that an ASCII writer has been + properly finished. Instead of aborting, we now just clean up in + that case and proceed. Addresses BIT-1331. (Robin Sommer) + +2.3-696 | 2015-04-14 15:56:36 -0700 + + * Update sqlite to 3.8.9 + +2.3-695 | 2015-04-13 10:34:42 -0500 + + * Fix iterator invalidation in broker::Manager dtor. (Jon Siwek) + + * Add paragraph to plugin documentation. (Robin Sommer) + +2.3-693 | 2015-04-11 10:56:31 -0700 + + * BIT-1367: improve coercion of anonymous records in set constructor. + (Jon Siwek) + + * Allow to specify ports for sftp log rotator. (Johanna Amann) + +2.3-690 | 2015-04-10 21:51:10 -0700 + + * Make sure to always delete the remote serializer. Addresses + BIT-1306 and probably also BIT-1356. (Robin Sommer) + + * Cleaning up --help. -D and -Y/y were still listed, even though + they had no effect anymore. Removing some dead code along with -D. + Addresses BIT-1372. (Robin Sommer) + +2.3-688 | 2015-04-10 08:10:44 -0700 + + * Update SQLite to 3.8.8.3. + +2.3-687 | 2015-04-10 07:32:52 -0700 + + * Remove stale signature benchmarking code (-L command-line option). + (Jon Siwek) + + * BIT-844: fix UDP payload signatures to match packet-wise. (Jon + Siwek) + +2.3-682 | 2015-04-09 12:07:00 -0700 + + * Fixing input readers' component type. (Robin Sommer) + + * Tiny spelling correction. (Seth Hall) + +2.3-680 | 2015-04-06 16:02:43 -0500 + + * BIT-1371: remove CMake version check from binary package scripts. + (Jon Siwek) + +2.3-679 | 2015-04-06 10:16:36 -0500 + + * Increase some unit test timeouts. (Jon Siwek) + + * Fix Coverity warning in RDP analyzer. (Jon Siwek) + +2.3-676 | 2015-04-02 10:10:39 -0500 + + * BIT-1366: improve checksum offloading warning. + (Frank Meier, Jon Siwek) + +2.3-675 | 2015-03-30 17:05:05 -0500 + + * Add an RDP analyzer. (Josh Liburdi, Seth Hall, Johanna Amann) + +2.3-640 | 2015-03-30 13:51:51 -0500 + + * BIT-1359: Limit maximum number of DTLS fragments to 30. (Johanna Amann) + +2.3-637 | 2015-03-30 12:02:07 -0500 + + * Increase timeout duration in some broker tests. (Jon Siwek) + +2.3-636 | 2015-03-30 11:26:32 -0500 + + * Updates related to SSH analysis. (Jon Siwek) + + - Some scripts used wrong SSH module/namespace scoping on events. + - Fix outdated notice documentation related to SSH password guessing. + - Add a unit test for SSH pasword guessing notice. + +2.3-635 | 2015-03-30 11:02:45 -0500 + + * Fix outdated documentation unit tests. (Jon Siwek) + +2.3-634 | 2015-03-30 10:22:45 -0500 + + * Add a canonifier to a unit test's output. (Jon Siwek) + +2.3-633 | 2015-03-25 18:32:59 -0700 + + * Log::write in signature framework was missing timestamp. + (Andrew Benson/Michel Laterman) + +2.3-631 | 2015-03-25 11:03:12 -0700 + + * New SSH analyzer. (Vlad Grigorescu) + +2.3-600 | 2015-03-25 10:23:46 -0700 + + * Add defensive checks in code to calculate log rotation intervals. + (Pete Nelson). + +2.3-597 | 2015-03-23 12:50:04 -0700 + + * DTLS analyzer. (Johanna Amann) + + * Implement correct parsing of TLS record fragmentation. (Johanna + Amann) + +2.3-582 | 2015-03-23 11:34:25 -0700 + + * BIT-1313: In debug builds, "bro -B " now supports "all" and + "help" for "". "all" enables all debug streams. "help" prints a + list of available debug streams. (John Donnelly/Robin Sommer). + + * BIT-1324: Allow logging filters to inherit default path from + stream. This allows the path for the default filter to be + specified explicitly through $path="..." when creating a stream. + Adapted the existing Log::create_stream calls to explicitly + specify a path value. (Jon Siwek) + + * BIT-1199: Change the way the input framework deals with values it + cannot convert into BroVals, raising error messages instead of + aborting execution. (Johanna Amann) + + * BIT-788: Use DNS QR field to better identify flow direction. (Jon + Siwek) + +2.3-572 | 2015-03-23 13:04:53 -0500 + + * BIT-1226: Fix an example in quickstart docs. (Jon siwek) + +2.3-570 | 2015-03-23 09:51:20 -0500 + + * Correct a spelling error (Daniel Thayer) + + * Improvement to SSL analyzer failure mode. (Johanna Amann) + +2.3-565 | 2015-03-20 16:27:41 -0500 + + * BIT-978: Improve documentation of 'for' loop iterator invalidation. + (Jon Siwek) + +2.3-564 | 2015-03-20 11:12:02 -0500 + + * BIT-725: Remove "unmatched_HTTP_reply" weird. (Jon Siwek) + +2.3-562 | 2015-03-20 10:31:02 -0500 + + * BIT-1207: Add unit test to catch breaking changes to local.bro + (Jon Siwek) + + * Fix failing sqlite leak test (Johanna Amann) + +2.3-560 | 2015-03-19 13:17:39 -0500 + + * BIT-1255: Increase default values of + "tcp_max_above_hole_without_any_acks" and "tcp_max_initial_window" + from 4096 to 16384 bytes. (Jon Siwek) + +2.3-559 | 2015-03-19 12:14:33 -0500 + + * BIT-849: turn SMTP reporter warnings into weirds, + "smtp_nested_mail_transaction" and "smtp_unmatched_end_of_data". + (Jon Siwek) + +2.3-558 | 2015-03-18 22:50:55 -0400 + + * DNS: Log the type number for the DNS_RR_unknown_type weird. (Vlad Grigorescu) + +2.3-555 | 2015-03-17 15:57:13 -0700 + + * Splitting test-all Makefile target into Bro tests and test-aux. + (Robin Sommer) + +2.3-554 | 2015-03-17 15:40:39 -0700 + + * Deprecate &rotate_interval, &rotate_size, &encrypt. Addresses + BIT-1305. (Jon Siwek) + +2.3-549 | 2015-03-17 09:12:18 -0700 + + * BIT-1077: Fix HTTP::log_server_header_names. Before, it just + re-logged fields from the client side. (Jon Siwek) + +2.3-547 | 2015-03-17 09:07:51 -0700 + + * Update certificate validation script to cache valid intermediate + chains that it encounters on the wire and use those to try to + validate chains that might be missing intermediate certificates. + (Johanna Amann) + +2.3-541 | 2015-03-13 15:44:08 -0500 + + * Make INSTALL a symlink to doc/install/install.rst (Jon siwek) + + * Fix Broxygen coverage. (Jon Siwek) + +2.3-539 | 2015-03-13 14:19:27 -0500 + + * BIT-1335: Include timestamp in default extracted file names. + And add a policy script to extract all files. (Jon Siwek) + + * BIT-1311: Identify GRE tunnels as Tunnel::GRE, not Tunnel::IP. + (Jon Siwek) + + * BIT-1309: Add Connection class getter methods for flow labels. + (Jon Siwek) + +2.3-536 | 2015-03-12 16:16:24 -0500 + + * Fix Broker leak tests. (Jon Siwek) + +2.3-534 | 2015-03-12 10:59:49 -0500 + + * Update NEWS file. (Jon Siwek) + +2.3-533 | 2015-03-12 10:18:53 -0500 + + * Give broker python bindings default install path within --prefix. + (Jon Siwek) + +2.3-530 | 2015-03-10 13:22:39 -0500 + + * Fix broker data stores in absence of --enable-debug. (Jon Siwek) + +2.3-529 | 2015-03-09 13:14:27 -0500 + + * Fix format specifier in SSL protocol violation. (Jon Siwek) + +2.3-526 | 2015-03-06 12:48:49 -0600 + + * Fix build warnings, clarify broker requirements, update submodule. + (Jon Siwek) + + * Rename comm/ directories to broker/ (Jon Siwek) + + * Rename broker-related namespaces. (Jon Siwek) + + * Improve remote logging via broker by only sending fields w/ &log. + (Jon Siwek) + + * Disable a stream's remote logging via broker if it fails. (Jon Siwek) + + * Improve some broker communication unit tests. (Jon Siwek) + +2.3-518 | 2015-03-04 13:13:50 -0800 + + * Add bytes_recvd to stats.log recording the number of bytes + received, according to packet headers. (Mike Smiley) + +2.3-516 | 2015-03-04 12:30:06 -0800 + + * Extract most specific Common Name from SSL certificates (Johanna + Amann) + + * Send CN and SAN fields of SSL certificates to the Intel framework. + (Johanna Amann) + +2.3-511 | 2015-03-02 18:07:17 -0800 + + * Changes to plugin meta hooks for function calls. (Gilbert Clark) + + - Add frame argument. + + - Change return value to tuple unambigiously whether hook + returned a result. + +2.3-493 | 2015-03-02 17:17:32 -0800 + + * Extend the SSL weak-keys policy file to also alert when + encountering SSL connections with old versions as well as unsafe + cipher suites. (Johanna Amann) + + * Make the notice suppression handling of other SSL policy files a + tad more robust. (Johanna Amann) + +2.3-491 | 2015-03-02 17:12:56 -0800 + + * Updating docs for recent addition of local_resp. (Robin Sommer) + +2.3-489 | 2015-03-02 15:29:30 -0800 + + * Integrate Broker, Bro's new communication library. (Jon Siwek) + + See aux/broker/README for more information on Broker, and + doc/frameworks/comm.rst for the corresponding Bro script API. + + Broker support is by default off for now; it can be enabled at + configure time with --enable-broker. It requires CAF + (https://github.com/actor-framework/actor-framework); for now iot + needs CAF's "develop" branch. Broker also requires a C++11 + compiler. + + Broker will become a mandatory dependency in future Bro versions. + + * Add --enable-c++11 configure flag to compile Bro's source code in + C++11 mode with a corresponding compiler. (Jon Siwek) + 2.3-451 | 2015-02-24 16:37:08 -0800 * Updating submodule(s). @@ -245,7 +1405,7 @@ 2.3-328 | 2014-12-02 08:13:10 -0500 - * Update windows-version-detection.bro to add support for + * Update windows-version-detection.bro to add support for Windows 10. (Michal Purzynski) 2.3-326 | 2014-12-01 12:10:27 -0600 @@ -315,7 +1475,7 @@ 2.3-280 | 2014-11-05 09:46:33 -0500 - * Add Windows detection based on CryptoAPI HTTP traffic as a + * Add Windows detection based on CryptoAPI HTTP traffic as a software framework policy script. (Vlad Grigorescu) 2.3-278 | 2014-11-03 18:55:18 -0800 @@ -805,21 +1965,21 @@ 2.3-beta-18 | 2014-06-06 13:11:50 -0700 * Add two more SSL events, one triggered for each handshake message - and one triggered for the tls change cipherspec message. (Bernhard + and one triggered for the tls change cipherspec message. (Johanna Amann) * Small SSL bug fix. In case SSL::disable_analyzer_after_detection was set to false, the ssl_established event would fire after each - data packet once the session is established. (Bernhard Amann) + data packet once the session is established. (Johanna Amann) 2.3-beta-16 | 2014-06-06 13:05:44 -0700 * Re-activate notice suppression for expiring certificates. - (Bernhard Amann) + (Johanna Amann) 2.3-beta-14 | 2014-06-05 14:43:33 -0700 - * Add new TLS extension type numbers from IANA (Bernhard Amann) + * Add new TLS extension type numbers from IANA (Johanna Amann) * Switch to double hashing for Bloomfilters for better performance. (Matthias Vallentin) @@ -829,7 +1989,7 @@ (Matthias Vallentin) * Make buffer for X509 certificate subjects larger. Addresses - BIT-1195 (Bernhard Amann) + BIT-1195 (Johanna Amann) 2.3-beta-5 | 2014-05-29 15:34:42 -0500 @@ -851,19 +2011,19 @@ * Release 2.3-beta - * Clean up OpenSSL data structures on exit. (Bernhard Amann) + * Clean up OpenSSL data structures on exit. (Johanna Amann) - * Fixes for OCSP & x509 analysis memory leak issues. (Bernhard Amann) + * Fixes for OCSP & x509 analysis memory leak issues. (Johanna Amann) * Remove remaining references to BROMAGIC (Daniel Thayer) * Fix typos and formatting in event and BiF documentation (Daniel Thayer) * Update intel framework plugin for ssl server_name extension API - changes. (Bernhard Amann, Justin Azoff) + changes. (Johanna Amann, Justin Azoff) * Fix expression errors in SSL/x509 scripts when unparseable data - is in certificate chain. (Bernhard Amann) + is in certificate chain. (Johanna Amann) 2.2-478 | 2014-05-19 15:31:33 -0500 @@ -872,7 +2032,7 @@ 2.2-477 | 2014-05-19 14:13:00 -0500 - * Fix X509::Result record's "result" field to be set internally as type int instead of type count. (Bernhard Amann) + * Fix X509::Result record's "result" field to be set internally as type int instead of type count. (Johanna Amann) * Fix a couple of doc build warnings (Daniel Thayer) @@ -890,19 +2050,19 @@ * New script policy/protocols/ssl/validate-ocsp.bro that adds OSCP validation to ssl.log. The work is done by a new bif - x509_ocsp_verify(). (Bernhard Amann) + x509_ocsp_verify(). (Johanna Amann) * STARTTLS support for POP3 and SMTP. The SSL analyzer takes over when seen. smtp.log now logs when a connection switches to SSL. - (Bernhard Amann) + (Johanna Amann) - * Replace errors when parsing x509 certs with weirds. (Bernhard + * Replace errors when parsing x509 certs with weirds. (Johanna Amann) - * Improved Heartbleed attack/scan detection. (Bernhard Amann) + * Improved Heartbleed attack/scan detection. (Johanna Amann) * Let TLS analyzer fail better when no longer in sync with the data - stream. (Bernhard Amann) + stream. (Johanna Amann) 2.2-444 | 2014-05-16 14:10:32 -0500 @@ -921,7 +2081,7 @@ 2.2-427 | 2014-05-15 13:37:23 -0400 - * Fix dynamic SumStats update on clusters (Bernhard Amann) + * Fix dynamic SumStats update on clusters (Johanna Amann) 2.2-425 | 2014-05-08 16:34:44 -0700 @@ -973,11 +2133,11 @@ * Add DH support to SSL analyzer. When using DHE or DH-Anon, sever key parameters are now available in scriptland. Also add script to - alert on weak certificate keys or weak dh-params. (Bernhard Amann) + alert on weak certificate keys or weak dh-params. (Johanna Amann) - * Add a few more ciphers Bro did not know at all so far. (Bernhard Amann) + * Add a few more ciphers Bro did not know at all so far. (Johanna Amann) - * Log chosen curve when using ec cipher suite in TLS. (Bernhard Amann) + * Log chosen curve when using ec cipher suite in TLS. (Johanna Amann) 2.2-397 | 2014-05-01 20:29:20 -0700 @@ -989,7 +2149,7 @@ (Jon Siwek) * Correct a notice for heartbleed. The notice is thrown correctly, - just the message conteined wrong values. (Bernhard Amann) + just the message conteined wrong values. (Johanna Amann) * Improve/standardize some malloc/realloc return value checks. (Jon Siwek) @@ -1016,7 +2176,7 @@ 2.2-377 | 2014-04-24 16:57:54 -0700 * A larger set of SSL improvements and extensions. Addresses - BIT-1178. (Bernhard Amann) + BIT-1178. (Johanna Amann) - Fixes TLS protocol version detection. It also should bail-out correctly on non-tls-connections now @@ -1077,9 +2237,9 @@ 2.2-335 | 2014-04-10 15:04:57 -0700 - * Small logic fix for main SSL script. (Bernhard Amann) + * Small logic fix for main SSL script. (Johanna Amann) - * Update DPD signatures for detecting TLS 1.2. (Bernhard Amann) + * Update DPD signatures for detecting TLS 1.2. (Johanna Amann) * Remove unused data member of SMTP_Analyzer to silence a Coverity warning. (Jon Siwek) @@ -1108,7 +2268,7 @@ 2.2-315 | 2014-04-01 16:50:01 -0700 * Change logging's "#types" description of sets to "set". Addresses - BIT-1163 (Bernhard Amann) + BIT-1163 (Johanna Amann) 2.2-313 | 2014-04-01 16:40:19 -0700 @@ -1123,7 +2283,7 @@ (Jon Siwek) * Fix potential memory leak in x509 parser reported by Coverity. - (Bernhard Amann) + (Johanna Amann) 2.2-304 | 2014-03-30 23:05:54 +0200 @@ -1194,7 +2354,7 @@ from the certificates (e.g. elliptic curve information, subject alternative names, basic constraints). Certificate validation also was improved, should be easier to use and exposes information like - the full verified certificate chain. (Bernhard Amann) + the full verified certificate chain. (Johanna Amann) This update changes the format of ssl.log and adds a new x509.log with certificate information. Furthermore all x509 events and @@ -1232,7 +2392,7 @@ 2.2-256 | 2014-03-30 19:57:28 +0200 * For the summary statistics framewirk, change all &create_expire - attributes to &read_expire in the cluster part. (Bernhard Amann) + attributes to &read_expire in the cluster part. (Johanna Amann) 2.2-254 | 2014-03-30 19:55:22 +0200 @@ -1256,7 +2416,7 @@ 2.2-244 | 2014-03-17 08:24:17 -0700 * Fix compile errror on FreeBSD caused by wrong include file order. - (Bernhard Amann) + (Johanna Amann) 2.2-240 | 2014-03-14 10:23:54 -0700 @@ -1352,7 +2512,7 @@ * Improve SSL logging so that connections are logged even when the ssl_established event is not generated as well as other small SSL - fixes. (Bernhard Amann) + fixes. (Johanna Amann) 2.2-206 | 2014-03-03 16:52:28 -0800 @@ -1369,7 +2529,7 @@ * Allow iterating over bif functions with result type vector of any. This changes the internal type that is used to signal that a vector is unspecified from any to void. Addresses BIT-1144 - (Bernhard Amann) + (Johanna Amann) 2.2-197 | 2014-02-28 15:36:58 -0800 @@ -1377,37 +2537,37 @@ 2.2-194 | 2014-02-28 14:50:53 -0800 - * Remove packet sorter. Addresses BIT-700. (Bernhard Amann) + * Remove packet sorter. Addresses BIT-700. (Johanna Amann) 2.2-192 | 2014-02-28 09:46:43 -0800 - * Update Mozilla root bundle. (Bernhard Amann) + * Update Mozilla root bundle. (Johanna Amann) 2.2-190 | 2014-02-27 07:34:44 -0800 - * Adjust timings of a few leak tests. (Bernhard Amann) + * Adjust timings of a few leak tests. (Johanna Amann) 2.2-187 | 2014-02-25 07:24:42 -0800 - * More Google TLS extensions that are being actively used. (Bernhard + * More Google TLS extensions that are being actively used. Johanna( Amann) * Remove unused, and potentially unsafe, function - ListVal::IncludedInString. (Bernhard Amann) + ListVal::IncludedInString. (Johanna Amann) 2.2-184 | 2014-02-24 07:28:18 -0800 * New TLS constants from https://tools.ietf.org/html/draft-bmoeller-tls-downgrade-scsv-01. - (Bernhard Amann) + (Johanna Amann) 2.2-180 | 2014-02-20 17:29:14 -0800 * New SSL alert descriptions from https://tools.ietf.org/html/draft-ietf-tls-applayerprotoneg-04. - (Bernhard Amann) + (Johanna Amann) - * Update SQLite. (Bernhard Amann) + * Update SQLite. (Johanna Amann) 2.2-177 | 2014-02-20 17:27:46 -0800 @@ -1438,7 +2598,7 @@ 'modbus_read_fifo_queue_response' event handler. (Jon Siwek) * Add channel_id TLS extension number. This number is not IANA - defined, but we see it being actively used. (Bernhard Amann) + defined, but we see it being actively used. (Johanna Amann) * Test baseline updates for DNS change. (Robin Sommer) @@ -1480,7 +2640,7 @@ 2.2-147 | 2014-02-07 08:06:53 -0800 - * Fix x509-extension test sometimes failing. (Bernhard Amann) + * Fix x509-extension test sometimes failing. (Johanna Amann) 2.2-144 | 2014-02-06 20:31:18 -0800 @@ -1516,7 +2676,7 @@ 2.2-128 | 2014-01-30 15:58:47 -0800 - * Add leak test for Exec module. (Bernhard Amann) + * Add leak test for Exec module. (Johanna Amann) * Fix file_over_new_connection event to trigger when entire file is missed. (Jon Siwek) @@ -1534,7 +2694,7 @@ 2.2-120 | 2014-01-28 10:25:23 -0800 * Fix and extend x509_extension() event, which now actually returns - the extension. (Bernhard Amann) + the extension. (Johanna Amann) New event signauture: @@ -1649,7 +2809,7 @@ * Several improvements to input framework error handling for more robustness and more helpful error messages. Includes tests for - many cases. (Bernhard Amann) + many cases. (Johanna Amann) 2.2-66 | 2013-12-09 13:54:16 -0800 @@ -1675,7 +2835,7 @@ * Fix memory leak in input framework. If the input framework was used to read event streams and those streams contained records with more than one field, not all elements of the threading Values - were cleaned up. Addresses BIT-1103. (Bernhard Amann) + were cleaned up. Addresses BIT-1103. (Johanna Amann) * Minor Broxygen improvements. Addresses BIT-1098. (Jon Siwek) @@ -1719,7 +2879,7 @@ 2.2-40 | 2013-12-04 12:16:38 -0800 * ssl_client_hello() now receives a vector of ciphers, instead of a - set, to preserve their order. (Bernhard Amann) + set, to preserve their order. (Johanna Amann) 2.2-38 | 2013-12-04 12:10:54 -0800 @@ -1856,13 +3016,13 @@ 2.2-beta-157 | 2013-10-25 11:11:17 -0700 * Extend the documentation of the SQLite reader/writer framework. - (Bernhard Amann) + (Johanna Amann) * Fix inclusion of wrong example file in scripting tutorial. - Reported by Michael Auger @LM4K. (Bernhard Amann) + Reported by Michael Auger @LM4K. (Johanna Amann) * Alternative fix for the thrading deadlock issue to avoid potential - performance impact. (Bernhard Amann) + performance impact. (Johanna Amann) 2.2-beta-152 | 2013-10-24 18:16:49 -0700 @@ -1875,7 +3035,7 @@ 2.2-beta-150 | 2013-10-24 16:32:14 -0700 * Change temporary ASCII reader workaround for getline() on - Mavericks to permanent fix. (Bernhard Amann) + Mavericks to permanent fix. (Johanna Amann) 2.2-beta-148 | 2013-10-24 14:34:35 -0700 @@ -1889,7 +3049,7 @@ * Intel framework notes added to NEWS. (Seth Hall) * Temporary OSX Mavericks libc++ issue workaround for getline() - problem in ASCII reader. (Bernhard Amann) + problem in ASCII reader. (Johanna Amann) * Change test of identify_data BIF to ignore charset as it may vary with libmagic version. (Jon Siwek) @@ -1932,16 +3092,16 @@ 2.2-beta-80 | 2013-10-18 13:18:05 -0700 - * SQLite reader/writer documentation. (Bernhard Amann) + * SQLite reader/writer documentation. (Johanna Amann) * Check that the SQLite reader is only used in MANUAL reading mode. - (Bernhard Amann) + (Johanna Amann) * Rename the SQLite writer "dbname" configuration option to - "tablename". (Bernhard Amann) + "tablename". (Johanna Amann) * Remove the "dbname" configuration option from the SQLite reader as - it wasn't used there. (Bernhard Amann) + it wasn't used there. (Johanna Amann) 2.2-beta-73 | 2013-10-14 14:28:25 -0700 @@ -1973,9 +3133,9 @@ 2.2-beta-55 | 2013-10-10 13:36:38 -0700 - * A couple of new TLS extension numbers. (Bernhard Amann) + * A couple of new TLS extension numbers. (Johanna Amann) - * Suport for three more new TLS ciphers. (Bernhard Amann) + * Suport for three more new TLS ciphers. (Johanna Amann) * Removing ICSI notary from default site config. (Robin Sommer) @@ -2020,7 +3180,7 @@ 2.2-beta-18 | 2013-10-02 10:28:17 -0700 - * Add support for further TLS cipher suites. (Bernhard Amann) + * Add support for further TLS cipher suites. (Johanna Amann) 2.2-beta-13 | 2013-10-01 11:31:55 -0700 @@ -2070,7 +3230,7 @@ * Add links to Intelligence Framework documentation. (Daniel Thayer) - * Update Mozilla root CA list. (Bernhard Amann, Jon Siwek) + * Update Mozilla root CA list. (Johanna Amann, Jon Siwek) * Update documentation of required packages. (Daniel Thayer) @@ -2081,10 +3241,10 @@ 2.1-1357 | 2013-09-18 14:58:52 -0700 - * Update HLL API and its documentation. (Bernhard Amann) + * Update HLL API and its documentation. (Johanna Amann) * Fix case in HLL where hll_error_margin could be undefined. - (Bernhard Amann) + (Johanna Amann) 2.1-1352 | 2013-09-18 14:42:28 -0700 @@ -2145,7 +3305,7 @@ * Support for probabilistic set cardinality, using the HyperLogLog - algorithm. (Bernhard Amann, Soumya Basu) + algorithm. (Johanna Amann, Soumya Basu) Bro now provides the following BiFs: @@ -2184,7 +3344,7 @@ 2.1-1137 | 2013-08-27 13:26:44 -0700 * Add BiF hexstr_to_bytestring() that does exactly the opposite of - bytestring_to_hexstr(). (Bernhard Amann) + bytestring_to_hexstr(). (Johanna Amann) 2.1-1135 | 2013-08-27 12:16:26 -0700 @@ -2256,7 +3416,7 @@ 2.1-1078 | 2013-08-19 09:29:30 -0700 - * Moving sqlite code into new external 3rdparty submodule. (Bernhard + * Moving sqlite code into new external 3rdparty submodule. Johanna( Amann) 2.1-1074 | 2013-08-14 10:29:54 -0700 @@ -2356,12 +3516,12 @@ 2.1-1007 | 2013-08-01 15:41:54 -0700 - * More function documentation. (Bernhard Amann) + * More function documentation. (Johanna Amann) 2.1-1004 | 2013-08-01 14:37:43 -0700 * Adding a probabilistic data structure for computing "top k" - elements. (Bernhard Amann) + elements. (Johanna Amann) The corresponding functions are: @@ -2395,7 +3555,7 @@ 2.1-948 | 2013-07-31 20:08:28 -0700 * Fix segfault caused by merging an empty bloom-filter with a - bloom-filter already containing values. (Bernhard Amann) + bloom-filter already containing values. (Johanna Amann) 2.1-945 | 2013-07-30 10:05:10 -0700 @@ -2535,12 +3695,12 @@ 2.1-814 | 2013-07-15 18:18:20 -0700 * Fixing raw reader crash when accessing nonexistant file, and - memory leak when reading from file. Addresses #1038. (Bernhard + memory leak when reading from file. Addresses #1038. (Johanna Amann) 2.1-811 | 2013-07-14 08:01:54 -0700 - * Bump sqlite to 3.7.17. (Bernhard Amann) + * Bump sqlite to 3.7.17. (Johanna Amann) * Small test fixes. (Seth Hall) @@ -2590,7 +3750,7 @@ 2.1-780 | 2013-07-03 16:46:26 -0700 * Rewrite of the RAW input reader for improved robustness and new - features. (Bernhard Amann) This includes: + features. (Johanna Amann) This includes: - Send "end_of_data" event for all kind of streams. - Send "process_finished" event with exit code of child @@ -2719,12 +3879,12 @@ 2.1-656 | 2013-05-17 15:58:07 -0700 - * Fix mutex lock problem for writers. (Bernhard Amann) + * Fix mutex lock problem for writers. (Johanna Amann) 2.1-654 | 2013-05-17 13:49:52 -0700 * Tweaks to sqlite3 configuration to address threading issues. - (Bernhard Amann) + (Johanna Amann) 2.1-651 | 2013-05-17 13:37:16 -0700 @@ -2750,7 +3910,7 @@ 2.1-640 | 2013-05-15 17:24:09 -0700 - * Support for cleaning up threads that have terminated. (Bernhard + * Support for cleaning up threads that have terminated. (Johanna Amann and Robin Sommer). Includes: - Both logging and input frameworks now clean up threads once @@ -2767,14 +3927,14 @@ 2.1-626 | 2013-05-15 16:09:31 -0700 * Add "reservoir" sampler for SumStats framework. This maintains - a set of N uniquely distributed random samples. (Bernhard Amann) + a set of N uniquely distributed random samples. (Johanna Amann) 2.1-619 | 2013-05-15 16:01:42 -0700 * SQLite reader and writer combo. This allows to read/write persistent data from on disk SQLite databases. The current interface is quite low-level, we'll add higher-level abstractions - in the future. (Bernhard Amann) + in the future. (Johanna Amann) 2.1-576 | 2013-05-15 14:29:09 -0700 @@ -2795,7 +3955,7 @@ 2.1-500 | 2013-05-10 19:22:24 -0700 * Fix to prevent merge-hook of SumStat's unique plugin from damaging - source data. (Bernhard Amann) + source data. (Johanna Amann) 2.1-498 | 2013-05-03 17:44:08 -0700 @@ -2811,7 +3971,7 @@ 2.1-492 | 2013-05-02 12:46:26 -0700 * Work-around for sumstats framework not propagating updates after - intermediate check in cluster environments. (Bernhard Amann) + intermediate check in cluster environments. (Johanna Amann) * Always apply tcp_connection_attempt. Before this change it was only applied when a connection_attempt() event handler was @@ -2866,7 +4026,7 @@ 2.1-380 | 2013-03-18 12:18:10 -0700 * Fix gcc compile warnings in base64 encoder and benchmark reader. - (Bernhard Amann) + (Johanna Amann) 2.1-377 | 2013-03-17 17:36:09 -0700 @@ -2875,10 +4035,10 @@ 2.1-375 | 2013-03-17 13:14:26 -0700 * Add base64 encoding functionality, including new BiFs - encode_base64() and encode_base64_custom(). (Bernhard Amann) + encode_base64() and encode_base64_custom(). (Johanna Amann) * Replace call to external "openssl" in extract-certs-pem.bro with - that encode_base64(). (Bernhard Amann) + that encode_base64(). (Johanna Amann) * Adding a test for extract-certs-pem.pem. (Robin Sommer) @@ -2912,7 +4072,7 @@ 2.1-357 | 2013-03-08 09:18:35 -0800 - * Fix race-condition in table-event test. (Bernhard Amann) + * Fix race-condition in table-event test. (Johanna Amann) * s/bro-ids.org/bro.org/g. (Robin Sommer) @@ -2929,9 +4089,9 @@ 2.1-347 | 2013-03-06 16:48:44 -0800 - * Remove unused parameter from vector assignment method. (Bernhard Amann) + * Remove unused parameter from vector assignment method. (Johanna Amann) - * Remove the byte_len() and length() bifs. (Bernhard Amann) + * Remove the byte_len() and length() bifs. (Johanna Amann) 2.1-342 | 2013-03-06 15:42:52 -0800 @@ -2983,7 +4143,7 @@ 2.1-319 | 2013-02-04 09:45:34 -0800 - * Update input tests to use exit_only_after_terminate. (Bernhard + * Update input tests to use exit_only_after_terminate. (Johanna Amann) * New option exit_only_after_terminate to prevent Bro from exiting. @@ -3015,7 +4175,7 @@ 2.1-302 | 2013-01-23 16:17:29 -0800 * Refactoring ASCII formatting/parsing from loggers/readers into a - separate AsciiFormatter class. (Bernhard Amann) + separate AsciiFormatter class. (Johanna Amann) * Fix uninitialized locals in event/hook handlers from having a value. Addresses #932. (Jon Siwek) @@ -3046,7 +4206,7 @@ * Removing unused class member. (Robin Sommer) * Add opaque type-ignoring for the accept_unsupported_types input - framework option. (Bernhard Amann) + framework option. (Johanna Amann) 2.1-271 | 2013-01-08 10:18:57 -0800 @@ -3127,7 +4287,7 @@ 2.1-229 | 2012-12-14 14:46:12 -0800 * Fix memory leak in ASCII reader when encoutering errors in input. - (Bernhard Amann) + (Johanna Amann) * Improvements for the "bad checksums" detector to make it detect bad TCP checksums. (Seth Hall) @@ -3198,7 +4358,7 @@ yet. Addresses #66. (Jon Siwek) * Fix segfault: Delete correct entry in error case in input - framework. (Bernhard Amann) + framework. (Johanna Amann) * Bad record constructor initializers now give an error. Addresses #34. (Jon Siwek) @@ -3456,7 +4616,7 @@ * Rename the Input Framework's update_finished event to end_of_data. It will now not only fire after table-reads have been completed, but also after the last event of a whole-file-read (or - whole-db-read, etc.). (Bernhard Amann) + whole-db-read, etc.). (Johanna Amann) * Fix for DNS log problem when a DNS response is seen with 0 RRs. (Seth Hall) @@ -3471,7 +4631,7 @@ 2.1-61 | 2012-10-12 09:32:48 -0700 * Fix bug in the input framework: the config table did not work. - (Bernhard Amann) + (Johanna Amann) 2.1-58 | 2012-10-08 10:10:09 -0700 @@ -3506,7 +4666,7 @@ * Fix for the input framework: BroStrings were constructed without a final \0, which makes them unusable by basically all internal - functions (like to_count). (Bernhard Amann) + functions (like to_count). (Johanna Amann) * Remove deprecated script functionality (see NEWS for details). (Daniel Thayer) @@ -3558,7 +4718,7 @@ * Small change to non-blocking DNS initialization. (Jon Siwek) * Reorder a few statements in scan.l to make 1.5msecs etc work. - Adresses #872. (Bernhard Amann) + Adresses #872. (Johanna Amann) 2.1-6 | 2012-09-06 23:23:14 -0700 @@ -3587,11 +4747,11 @@ * Fix uninitialized value for 'is_partial' in TCP analyzer. (Jon Siwek) - * Parse 64-bit consts in Bro scripts correctly. (Bernhard Amann) + * Parse 64-bit consts in Bro scripts correctly. (Johanna Amann) - * Output 64-bit counts correctly on 32-bit machines (Bernhard Amann) + * Output 64-bit counts correctly on 32-bit machines (Johanna Amann) - * Input framework fixes, including: (Bernhard Amann) + * Input framework fixes, including: (Johanna Amann) - One of the change events got the wrong parameters. @@ -3632,7 +4792,7 @@ 2.1-beta-45 | 2012-08-22 16:11:10 -0700 * Add an option to the input framework that allows the user to chose - to not die upon encountering files/functions. (Bernhard Amann) + to not die upon encountering files/functions. (Johanna Amann) 2.1-beta-41 | 2012-08-22 16:05:21 -0700 @@ -3651,7 +4811,7 @@ 2.1-beta-35 | 2012-08-22 08:44:52 -0700 * Add testcase for input framework reading sets (rather than - tables). (Bernhard Amann) + tables). (Johanna Amann) 2.1-beta-31 | 2012-08-21 15:46:05 -0700 @@ -3710,9 +4870,9 @@ 2.1-beta-6 | 2012-08-10 12:22:52 -0700 - * Fix bug in input framework with an edge case. (Bernhard Amann) + * Fix bug in input framework with an edge case. (Johanna Amann) - * Fix small bug in input framework test script. (Bernhard Amann) + * Fix small bug in input framework test script. (Johanna Amann) 2.1-beta-3 | 2012-08-03 10:46:49 -0700 @@ -3761,13 +4921,13 @@ writers that don't have a postprocessor. (Seth Hall) * Update input framework documentation to reflect want_record - change. (Bernhard Amann) + change. (Johanna Amann) * Fix crash when encountering an InterpreterException in a predicate - in logging or input Framework. (Bernhard Amann) + in logging or input Framework. (Johanna Amann) * Input framework: Make want_record=T the default for events - (Bernhard Amann) + (Johanna Amann) * Changing the start/end markers in logs to open/close now reflecting wall clock. (Robin Sommer) @@ -3789,10 +4949,10 @@ * Add comprehensive error handling for close() calls. (Jon Siwek) - * Add more test cases for input framework. (Bernhard Amann) + * Add more test cases for input framework. (Johanna Amann) * Input framework: make error output for non-matching event types - much more verbose. (Bernhard Amann) + much more verbose. (Johanna Amann) 2.0-877 | 2012-07-25 17:20:34 -0700 @@ -3832,12 +4992,12 @@ * Fix initialization problem in logging class. (Jon Siwek) * Input framework now accepts escaped ASCII values as input (\x##), - and unescapes appropiately. (Bernhard Amann) + and unescapes appropiately. (Johanna Amann) * Make reading ASCII logfiles work when the input separator is - different from \t. (Bernhard Amann) + different from \t. (Johanna Amann) - * A number of smaller fixes for input framework. (Bernhard Amann) + * A number of smaller fixes for input framework. (Johanna Amann) 2.0-851 | 2012-07-24 15:04:14 -0700 @@ -3857,7 +5017,7 @@ * Reworking parts of the internal threading/logging/input APIs for thread-safety. (Robin Sommer) - * Bugfix for SSL version check. (Bernhard Amann) + * Bugfix for SSL version check. (Johanna Amann) * Changing a HTTP DPD from port 3138 to 3128. Addresses #857. (Robin Sommer) @@ -3877,7 +5037,7 @@ #763. (Robin Sommer) * Fix bug, where in dns.log rcode always was set to 0/NOERROR when - no reply package was seen. (Bernhard Amann) + no reply package was seen. (Johanna Amann) * Updating to Mozilla's current certificate bundle. (Seth Hall) @@ -3893,7 +5053,7 @@ * Remove baselines for some leak-detecting unit tests. (Jon Siwek) * Unblock SIGFPE, SIGILL, SIGSEGV and SIGBUS for threads, so that - they now propagate to the main thread. Adresses #848. (Bernhard + they now propagate to the main thread. Adresses #848. (Johanna Amann) 2.0-761 | 2012-07-12 08:14:38 -0700 @@ -3901,7 +5061,7 @@ * Some small fixes to further reduce SOCKS false positive logs. (Seth Hall) * Calls to pthread_mutex_unlock now log the reason for failures. - (Bernhard Amann) + (Johanna Amann) 2.0-757 | 2012-07-11 08:30:19 -0700 @@ -3932,11 +5092,11 @@ 2.0-733 | 2012-07-02 15:31:24 -0700 - * Extending the input reader DoInit() API. (Bernhard Amann). It now + * Extending the input reader DoInit() API. (Johanna Amann). It now provides a Info struct similar to what we introduced for log writers, including a corresponding "config" key/value table. - * Fix to make writer-info work when debugging is enabled. (Bernhard + * Fix to make writer-info work when debugging is enabled. (Johanna Amann) 2.0-726 | 2012-07-02 15:19:15 -0700 @@ -3975,7 +5135,7 @@ * Set input frontend type before starting the thread. This means that the thread type will be output correctly in the error - message. (Bernhard Amann) + message. (Johanna Amann) 2.0-719 | 2012-07-02 14:49:03 -0700 @@ -4064,7 +5224,7 @@ 2.0-622 | 2012-06-15 15:38:43 -0700 - * Input framework updates. (Bernhard Amann) + * Input framework updates. (Johanna Amann) - Disable streaming reads from executed commands. This lead to hanging Bros because pclose apparently can wait for eternity if @@ -4143,7 +5303,7 @@ * A new input framework enables scripts to read in external data dynamically on the fly as Bro is processing network traffic. - (Bernhard Amann) + (Johanna Amann) Currently, the framework supports reading ASCII input that's structured similar as Bro's log files as well as raw blobs of @@ -4310,7 +5470,7 @@ 2.0-315 | 2012-05-03 11:44:17 -0700 * Add two more TLS extension values that we see in live traffic. - (Bernhard Amann) + (Johanna Amann) * Fixed IPv6 link local unicast CIDR and added IPv6 loopback to private address space. (Seth Hall) @@ -4698,7 +5858,7 @@ 2.0-41 | 2012-02-03 04:10:53 -0500 - * Updates to the Software framework to simplify the API. (Bernhard + * Updates to the Software framework to simplify the API. (Johanna Amann) 2.0-40 | 2012-02-03 01:55:27 -0800 @@ -4841,7 +6001,7 @@ 2.0-beta-152 | 2012-01-03 14:51:34 -0800 - * Notices now record the transport-layer protocol. (Bernhard Amann) + * Notices now record the transport-layer protocol. (Johanna Amann) 2.0-beta-150 | 2012-01-03 14:42:45 -0800 @@ -4868,7 +6028,7 @@ assignments. Addresses #722. (Jon Siwek) * Make log headers include the type of data stored inside a set or - vector ("vector[string]"). (Bernhard Amann) + vector ("vector[string]"). (Johanna Amann) 2.0-beta-126 | 2011-12-18 15:18:05 -0800 @@ -5005,11 +6165,11 @@ * Fix order of include directories. (Jon Siwek) * Catch if logged vectors do not contain only atomic types. - (Bernhard Amann) + (Johanna Amann) 2.0-beta-47 | 2011-11-16 08:24:33 -0800 - * Catch if logged sets do not contain only atomic types. (Bernhard + * Catch if logged sets do not contain only atomic types. (Johanna Amann) * Promote libz and libmagic to required dependencies. (Jon Siwek) diff --git a/CMakeLists.txt b/CMakeLists.txt index 04ac197f74..374af64a18 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -61,7 +61,7 @@ if (NOT SED_EXE) endif () endif () -FindRequiredPackage(Perl) +FindRequiredPackage(PythonInterp) FindRequiredPackage(FLEX) FindRequiredPackage(BISON) FindRequiredPackage(PCAP) @@ -88,7 +88,7 @@ endif () include_directories(BEFORE ${PCAP_INCLUDE_DIR} - ${OpenSSL_INCLUDE_DIR} + ${OPENSSL_INCLUDE_DIR} ${BIND_INCLUDE_DIR} ${BinPAC_INCLUDE_DIR} ${ZLIB_INCLUDE_DIR} @@ -113,7 +113,7 @@ if (NOT DISABLE_PERFTOOLS) find_package(GooglePerftools) endif () -if (GOOGLEPERFTOOLS_FOUND) +if (GOOGLEPERFTOOLS_FOUND OR TCMALLOC_FOUND) set(HAVE_PERFTOOLS true) # Non-Linux systems may not be well-supported by gperftools, so # require explicit request from user to enable it in that case. @@ -141,7 +141,7 @@ endif () set(brodeps ${BinPAC_LIBRARY} ${PCAP_LIBRARY} - ${OpenSSL_LIBRARIES} + ${OPENSSL_LIBRARIES} ${BIND_LIBRARY} ${ZLIB_LIBRARY} ${JEMALLOC_LIBRARIES} @@ -165,18 +165,26 @@ include(PCAPTests) include(OpenSSLTests) include(CheckNameserCompat) include(GetArchitecture) +include(RequireCXX11) # Tell the plugin code that we're building as part of the main tree. set(BRO_PLUGIN_INTERNAL_BUILD true CACHE INTERNAL "" FORCE) -configure_file(${CMAKE_CURRENT_SOURCE_DIR}/config.h.in - ${CMAKE_CURRENT_BINARY_DIR}/config.h) +configure_file(${CMAKE_CURRENT_SOURCE_DIR}/bro-config.h.in + ${CMAKE_CURRENT_BINARY_DIR}/bro-config.h) include_directories(${CMAKE_CURRENT_BINARY_DIR}) ######################################################################## ## Recurse on sub-directories +if ( ENABLE_BROKER ) + add_subdirectory(aux/broker) + set(brodeps ${brodeps} broker) + add_definitions(-DENABLE_BROKER) + include_directories(BEFORE ${CMAKE_CURRENT_SOURCE_DIR}/aux/broker) +endif () + add_subdirectory(src) add_subdirectory(scripts) add_subdirectory(doc) @@ -224,6 +232,8 @@ message( "\nCXXFLAGS: ${CMAKE_CXX_FLAGS} ${CMAKE_CXX_FLAGS_${BuildType}}" "\nCPP: ${CMAKE_CXX_COMPILER}" "\n" + "\nBroker: ${ENABLE_BROKER}" + "\nBroker Python: ${BROKER_PYTHON_BINDINGS}" "\nBroccoli: ${INSTALL_BROCCOLI}" "\nBroctl: ${INSTALL_BROCTL}" "\nAux. Tools: ${INSTALL_AUX_TOOLS}" diff --git a/COPYING b/COPYING index 2c66f98113..5454660df2 100644 --- a/COPYING +++ b/COPYING @@ -1,4 +1,4 @@ -Copyright (c) 1995-2013, The Regents of the University of California +Copyright (c) 1995-2015, The Regents of the University of California through the Lawrence Berkeley National Laboratory and the International Computer Science Institute. All rights reserved. diff --git a/INSTALL b/INSTALL deleted file mode 100644 index 385dac93df..0000000000 --- a/INSTALL +++ /dev/null @@ -1,3 +0,0 @@ - -See doc/install/install.rst for installation instructions. - diff --git a/INSTALL b/INSTALL new file mode 120000 index 0000000000..95fcc60eda --- /dev/null +++ b/INSTALL @@ -0,0 +1 @@ +doc/install/install.rst \ No newline at end of file diff --git a/Makefile b/Makefile index 207ce72780..3efddc4dbc 100644 --- a/Makefile +++ b/Makefile @@ -51,13 +51,15 @@ distclean: $(MAKE) -C testing $@ test: - @( cd testing && make ) + -@( cd testing && make ) -test-all: test - test -d aux/broctl && ( cd aux/broctl && make test-all ) - test -d aux/btest && ( cd aux/btest && make test ) - test -d aux/bro-aux && ( cd aux/bro-aux && make test ) - test -d aux/plugins && ( cd aux/plugins && make test-all ) +test-aux: + -test -d aux/broctl && ( cd aux/broctl && make test-all ) + -test -d aux/btest && ( cd aux/btest && make test ) + -test -d aux/bro-aux && ( cd aux/bro-aux && make test ) + -test -d aux/plugins && ( cd aux/plugins && make test-all ) + +test-all: test test-aux configured: @test -d $(BUILD) || ( echo "Error: No build/ directory found. Did you run configure?" && exit 1 ) diff --git a/NEWS b/NEWS index 8af6c38201..b2310b0f62 100644 --- a/NEWS +++ b/NEWS @@ -4,11 +4,100 @@ release. For an exhaustive list of changes, see the ``CHANGES`` file (note that submodules, such as BroControl and Broccoli, come with their own ``CHANGES``.) -Bro 2.4 (in progress) +Bro 2.5 (in progress) ===================== -Dependencies ------------- +New Dependencies +---------------- + +- Bro now requires a compiler with C++11 support for building the + source code. + +- Bro now requires the C++ Actor Framework, CAF, which must be + installed first. See http://actor-framework.org. + +- Bro now requires Python instead of Perl to compile the source code. + +- The pcap buffer size can set through the new option Pcap::bufsize. + +New Functionality +----------------- + +- Bro now includes the NetControl framework. The framework allows for easy + interaction of Bro with hard- and software switches, firewalls, etc. + +- Bro now supports the Radiotap header for 802.11 frames. + +- Bro now tracks VLAN IDs. To record them inside the connection log, + load protocols/conn/vlan-logging.bro. + +- A new per-packet event raw_packet() provides access to layer 2 + information. Use with care, generating events per packet is + expensive. + +- A new built-in function, decode_base64_conn() for Base64 decoding. + It works like decode_base64() but receives an additional connection + argument that will be used for decoding errors into weird.log + (instead of reporter.log). + +- Two new built-in functions for handling set[subnet] and table[subnet]: + + - check_subnet(subnet, table) checks if a specific subnet is a member + of a set/table. This is different from the "in" operator, which always + performs a longest prefix match. + + - matching_subnets(subnet, table) returns all subnets of the set or table + that contain the given subnet. + +- Several built-in functions for handling IP addresses and subnets were added: + + - is_v4_subnet(subnet) checks whether a subnet specification is IPv4. + + - is_v6_subnet(subnet) checks whether a subnet specification is IPv6. + + - addr_to_subnet(addr) converts an IP address to a /32 subnet. + + - subnet_to_addr(subnet) returns the IP address part of a subnet. + + - subnet_width(subnet) returns the width of a subnet. + +- The IRC analyzer now recognizes StartTLS sessions and enable the SSL + analyzer for them. + +- New Bro plugins in aux/plugins: + + - af_packet: Native AF_PACKET support. + - kafka : Log writer interfacing to Kafka. + - myricom: Native Myricom SNF v3 support. + - pf_ring: Native PF_RING support. + - redis: An experimental log writer for Redis. + - tcprs: An TCP-level analyzer detecting retransmissions, reordering, and more. + +Changed Functionality +--------------------- + +- ``SSH::skip_processing_after_detection`` was removed. The functionality was + replaced by ``SSH::disable_analyzer_after_detection``. + +- Some script-level identifier have changed their names: + + snaplen -> Pcap::snaplen + precompile_pcap_filter() -> Pcap::precompile_pcap_filter() + install_pcap_filter() -> Pcap::install_pcap_filter() + pcap_error() -> Pcap::pcap_error() + + +Deprecated Functionality +------------------------ + + - The built-in functions decode_base64_custom() and + encode_base64_custom() are no longer needed and will be removed + in the future. Their functionality is now provided directly by + decode_base64() and encode_base64(), which take an optional + parameter to change the Base64 alphabet. + +Bro 2.4 +======= New Functionality ----------------- @@ -16,20 +105,51 @@ New Functionality - Bro now has support for external plugins that can extend its core functionality, like protocol/file analysis, via shared libraries. Plugins can be developed and distributed externally, and will be - pulled in dynamically at startup. Currently, a plugin can provide - custom protocol analyzers, file analyzers, log writers[TODO], input - readers[TODO], packet sources[TODO], and new built-in functions. A - plugin can furthermore hook into Bro's processing a number of places - to add custom logic. + pulled in dynamically at startup (the environment variables + BRO_PLUGIN_PATH and BRO_PLUGIN_ACTIVATE can be used to specify the + locations and names of plugins to activate). Currently, a plugin + can provide custom protocol analyzers, file analyzers, log writers, + input readers, packet sources and dumpers, and new built-in functions. + A plugin can furthermore hook into Bro's processing at a number of + places to add custom logic. See https://www.bro.org/sphinx-git/devel/plugins.html for more information on writing plugins. -- Bro now has supoprt for the MySQL wire protocol. Activity gets +- Bro now has support for the MySQL wire protocol. Activity gets logged into mysql.log. +- Bro now parses DTLS traffic. Activity gets logged into ssl.log. + +- Bro now has support for the Kerberos KRB5 protocol over TCP and + UDP. Activity gets logged into kerberos.log. + +- Bro now has an RDP analyzer. Activity gets logged into rdp.log. + +- Bro now has a file analyzer for Portable Executables. Activity gets + logged into pe.log. + +- Bro now has support for the SIP protocol over UDP. Activity gets + logged into sip.log. + +- Bro now features a completely rewritten, enhanced SSH analyzer. The + new analyzer is able to determine if logins failed or succeeded in + most circumstances, logs a lot more more information about SSH + sessions, supports v1, and introduces the intelligence type + ``Intel::PUBKEY_HASH`` and location ``SSH::IN_SERVER_HOST_KEY``. The + analayzer also generates a set of additional events + (``ssh_auth_successful``, ``ssh_auth_failed``, ``ssh_capabilities``, + ``ssh2_server_host_key``, ``ssh1_server_host_key``, + ``ssh_encrypted_packet``, ``ssh2_dh_server_params``, + ``ssh2_gss_error``, ``ssh2_ecc_key``). See next section for + incompatible SSH changes. + - Bro's file analysis now supports reassembly of files that are not - transferred/seen sequentially. + transferred/seen sequentially. The default file reassembly buffer + size is set with the ``Files::reassembly_buffer_size`` variable. + +- Bro's file type identification has been greatly improved (new file types, + bug fixes, and performance improvements). - Bro's scripting language now has a ``while`` statement:: @@ -39,6 +159,70 @@ New Functionality ``next`` and ``break`` can be used inside the loop's body just like with ``for`` loops. +- Bro now integrates Broker, a new communication library. See + aux/broker/README for more information on Broker, and + doc/frameworks/broker.rst for the corresponding Bro script API. + + With Broker, Bro has the similar capabilities of exchanging events and + logs with remote peers (either another Bro process or some other + application that uses Broker). It also includes a key-value store + API that can be used to share state between peers and optionally + allow data to persist on disk for longer-term storage. + + Broker support is by default off for now; it can be enabled at + configure time with --enable-broker. It requires CAF version 0.13+ + (https://github.com/actor-framework/actor-framework) as well as a + C++11 compiler (e.g. GCC 4.8+ or Clang 3.3+). + + Broker will become a mandatory dependency in future Bro versions and + replace the current communication and serialization system. + +- Add --enable-c++11 configure flag to compile Bro's source code in + C++11 mode with a corresponding compiler. Note that 2.4 will be the + last version of Bro that compiles without C++11 support. + +- The SSL analysis now alerts when encountering SSL connections with + old protocol versions or unsafe cipher suites. It also gained + extended reporting of weak keys, caching of already validated + certificates, and full support for TLS record defragmentation. SSL generally + became much more robust and added several fields to ssl.log (while + removing some others). + +- A new icmp_sent_payload event provides access to ICMP payload. + +- The input framework's raw reader now supports seeking by adding an + option "offset" to the config map. Positive offsets are interpreted + to be from the beginning of the file, negative from the end of the + file (-1 is end of file). + +- One can now raise events when a connection crosses a given size + threshold in terms of packets or bytes. The primary API for that + functionality is in base/protocols/conn/thresholds.bro. + +- There is a new command-line option -Q/--time that prints Bro's execution + time and memory usage to stderr. + +- BroControl now has a new command "deploy" which is equivalent to running + the "check", "install", "stop", and "start" commands (in that order). + +- BroControl now has a new option "StatusCmdShowAll" that controls whether + or not the broctl "status" command gathers all of the status information. + This option can be used to make the "status" command run significantly + faster (in this case, the "Peers" column will not be shown in the output). + +- BroControl now has a new option "StatsLogEnable" that controls whether + or not broctl will record information to the "stats.log" file. This option + can be used to make the "broctl cron" command run slightly faster (in this + case, "broctl cron" will also no longer send email about not seeing any + packets on the monitoring interfaces). + +- BroControl now has a new option "MailHostUpDown" which controls whether or + not the "broctl cron" command will send email when it notices that a host + in the cluster is up or down. + +- BroControl now has a new option "CommandTimeout" which specifies the number + of seconds to wait for a command that broctl ran to return results. + Changed Functionality --------------------- @@ -47,9 +231,17 @@ Changed Functionality - File analysis * Removed ``fa_file`` record's ``mime_type`` and ``mime_types`` - fields. The events ``file_mime_type`` and ``file_mime_types`` - have been added which contain the same information. The - ``mime_type`` field of ``Files::Info`` also still has this info. + fields. The event ``file_sniff`` has been added which provides + the same information. The ``mime_type`` field of ``Files::Info`` + also still has this info. + + * The earliest point that new mime type information is available is + in the ``file_sniff`` event which comes after the ``file_new`` and + ``file_over_new_connection`` events. Scripts which inspected mime + type info within those events will need to be adapted. (Note: for + users that worked w/ versions of Bro from git, for a while there was + also an event called ``file_mime_type`` which is now replaced with + the ``file_sniff`` event). * Removed ``Files::add_analyzers_for_mime_type`` function. @@ -58,15 +250,83 @@ Changed Functionality reassembly for non-sequential files, "offset" can be obtained with other information already available -- adding together ``seen_bytes`` and ``missed_bytes`` fields of the ``fa_file`` - record gives the how many bytes have been written so far (i.e. + record gives how many bytes have been written so far (i.e. the "offset"). -- has_valid_octets: now uses a string_vec parameter instead of +- The SSH changes come with a few incompatibilities. The following + events have been renamed: + + * ``SSH::heuristic_failed_login`` to ``ssh_auth_failed`` + * ``SSH::heuristic_successful_login`` to ``ssh_auth_successful`` + + The ``SSH::Info`` status field has been removed and replaced with + the ``auth_success`` field. This field has been changed from a + string that was previously ``success``, ``failure`` or + ``undetermined`` to a boolean. a boolean that is ``T``, ``F``, or + unset. + +- The has_valid_octets function now uses a string_vec parameter instead of string_array. - conn.log gained a new field local_resp that works like local_orig, just for the responder address of the connection. +- GRE tunnels are now identified as ``Tunnel::GRE`` instead of + ``Tunnel::IP``. + +- The default name for extracted files changed from extract-protocol-id + to extract-timestamp-protocol-id. + +- The weird named "unmatched_HTTP_reply" has been removed since it can + be detected at the script-layer and is handled correctly by the + default HTTP scripts. + +- When adding a logging filter to a stream, the filter can now inherit + a default ``path`` field from the associated ``Log::Stream`` record. + +- When adding a logging filter to a stream, the + ``Log::default_path_func`` is now only automatically added to the + filter if it has neither a ``path`` nor a ``path_func`` already + explicitly set. Before, the default path function would always be set + for all filters which didn't specify their own ``path_func``. + +- BroControl now establishes only one ssh connection from the manager to + each remote host in a cluster configuration (previously, there would be + one ssh connection per remote Bro process). + +- BroControl now uses SQLite to record state information instead of a + plain text file (the file "spool/broctl.dat" is no longer used). + On FreeBSD, this means that there is a new dependency on the package + "py27-sqlite3". + +- BroControl now records the expected running state of each Bro node right + before each start or stop. The "broctl cron" command uses this info to + either start or stop Bro nodes as needed so that the actual state matches + the expected state (previously, "broctl cron" could only start nodes in + the "crashed" state, and could never stop a node). + +- BroControl now sends all normal command output (i.e., not error messages) + to stdout. Error messages are still sent to stderr, however. + +- The capability of processing NetFlow input has been removed for the + time being. Therefore, the -y/--flowfile and -Y/--netflow command-line + options have been removed, and the netflow_v5_header and netflow_v5_record + events have been removed. + +- The -D/--dfa-size command-line option has been removed. + +- The -L/--rule-benchmark command-line option has been removed. + +- The -O/--optimize command-line option has been removed. + +- The deprecated fields "hot" and "addl" have been removed from the + connection record. Likewise, the functions append_addl() and + append_addl_marker() have been removed. + +- Log files now escape non-printable characters consistently as "\xXX'. + Furthermore, backslashes are escaped as "\\", making the + representation fully reversible. + Deprecated Functionality ------------------------ @@ -76,7 +336,7 @@ Deprecated Functionality concatenation/extraction functions. Note that the new functions use 0-based indexing, rather than 1-based. - The full list of now deprecation functions is: + The full list of now deprecated functions is: * split: use split_string instead. diff --git a/README.rst b/README.rst new file mode 120000 index 0000000000..100b93820a --- /dev/null +++ b/README.rst @@ -0,0 +1 @@ +README \ No newline at end of file diff --git a/VERSION b/VERSION index a8a700226f..3ddbdfa49b 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.3-451 +2.4-406 diff --git a/aux/binpac b/aux/binpac index 33cb1f8e6b..424d40c1e8 160000 --- a/aux/binpac +++ b/aux/binpac @@ -1 +1 @@ -Subproject commit 33cb1f8e6bf2e33c2773e86b157e1f343ee85dc6 +Subproject commit 424d40c1e8d5888311b50c0e5a9dfc9c5f818b66 diff --git a/aux/bro-aux b/aux/bro-aux index c9d340847c..105dfe4ad6 160000 --- a/aux/bro-aux +++ b/aux/bro-aux @@ -1 +1 @@ -Subproject commit c9d340847c668590a450f1881e6e3d763abe1138 +Subproject commit 105dfe4ad6c4ae4563b21cb0466ee350f0af0d43 diff --git a/aux/broccoli b/aux/broccoli index 1d55a0a84c..6ded82da49 160000 --- a/aux/broccoli +++ b/aux/broccoli @@ -1 +1 @@ -Subproject commit 1d55a0a84c5b1d0aa1727829300b388c92f92daa +Subproject commit 6ded82da498d805def6aa129cd7691d3b7287c37 diff --git a/aux/broctl b/aux/broctl index 76f99ea52c..583f3a3ff1 160000 --- a/aux/broctl +++ b/aux/broctl @@ -1 +1 @@ -Subproject commit 76f99ea52c3e021cade3d03eda7865d4f4d1793e +Subproject commit 583f3a3ff1847cf96a87f865d5cf0f36fae9dd67 diff --git a/aux/broker b/aux/broker new file mode 160000 index 0000000000..fe35cde8f0 --- /dev/null +++ b/aux/broker @@ -0,0 +1 @@ +Subproject commit fe35cde8f07ff7cf6decd2fb761cffc32e763d2d diff --git a/aux/btest b/aux/btest index 93d4989ed1..4bea8fa948 160000 --- a/aux/btest +++ b/aux/btest @@ -1 +1 @@ -Subproject commit 93d4989ed1537e4d143cf09d44077159f869a4b2 +Subproject commit 4bea8fa948be2bc86ff92399137131bc1c029b08 diff --git a/aux/plugins b/aux/plugins index 71d820e9d8..ab61be0c4f 160000 --- a/aux/plugins +++ b/aux/plugins @@ -1 +1 @@ -Subproject commit 71d820e9d8ca753fea8fb34ea3987993b28d79e4 +Subproject commit ab61be0c4f128c976f72dfa5a09a87cd842f387a diff --git a/config.h.in b/bro-config.h.in similarity index 100% rename from config.h.in rename to bro-config.h.in diff --git a/cmake b/cmake index ff08be5aa1..537e45afe1 160000 --- a/cmake +++ b/cmake @@ -1 +1 @@ -Subproject commit ff08be5aa1b8eaadbe2775cbc11b499c5f93349e +Subproject commit 537e45afe1006a10f73847fab5f13d28ce43fc4d diff --git a/configure b/configure index 2b1c568b26..a7a6f3b059 100755 --- a/configure +++ b/configure @@ -41,11 +41,13 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --enable-perftools-debug use Google's perftools for debugging --enable-jemalloc link against jemalloc --enable-ruby build ruby bindings for broccoli (deprecated) + --disable-broker disable use of the Broker communication library --disable-broccoli don't build or install the Broccoli library --disable-broctl don't install Broctl --disable-auxtools don't build or install auxiliary tools --disable-perftools don't try to build with Google Perftools --disable-python don't try to build python bindings for broccoli + --disable-pybroker don't try to build python bindings for broker Required Packages in Non-Standard Locations: --with-openssl=PATH path to OpenSSL install root @@ -54,19 +56,22 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --with-binpac=PATH path to BinPAC install root --with-flex=PATH path to flex executable --with-bison=PATH path to bison executable - --with-perl=PATH path to perl executable + --with-python=PATH path to Python executable + --with-libcaf=PATH path to C++ Actor Framework installation + (a required Broker dependency) Optional Packages in Non-Standard Locations: --with-geoip=PATH path to the libGeoIP install root --with-perftools=PATH path to Google Perftools install root --with-jemalloc=PATH path to jemalloc install root - --with-python=PATH path to Python interpreter --with-python-lib=PATH path to libpython --with-python-inc=PATH path to Python headers --with-ruby=PATH path to ruby interpreter --with-ruby-lib=PATH path to ruby library --with-ruby-inc=PATH path to ruby headers --with-swig=PATH path to SWIG executable + --with-rocksdb=PATH path to RocksDB installation + (an optional Broker dependency) Packaging Options (for developers): --binary-package toggle special logic for binary packaging @@ -88,7 +93,7 @@ Usage: $0 [OPTION]... [VAR=VALUE]... sourcedir="$( cd "$( dirname "$0" )" && pwd )" # Function to append a CMake cache entry definition to the -# CMakeCacheEntries variable +# CMakeCacheEntries variable. # $1 is the cache entry variable name # $2 is the cache entry variable type # $3 is the cache entry variable value @@ -96,6 +101,17 @@ append_cache_entry () { CMakeCacheEntries="$CMakeCacheEntries -D $1:$2=$3" } +# Function to remove a CMake cache entry definition from the +# CMakeCacheEntries variable +# $1 is the cache entry variable name +remove_cache_entry () { + CMakeCacheEntries="$CMakeCacheEntries -U $1" + + # Even with -U, cmake still warns by default if + # added previously with -D. + CMakeCacheEntries="$CMakeCacheEntries --no-warn-unused-cli" +} + # set defaults builddir=build prefix=/usr/local/bro @@ -105,10 +121,13 @@ append_cache_entry BRO_ROOT_DIR PATH $prefix append_cache_entry PY_MOD_INSTALL_DIR PATH $prefix/lib/broctl append_cache_entry BRO_SCRIPT_INSTALL_PATH STRING $prefix/share/bro append_cache_entry BRO_ETC_INSTALL_DIR PATH $prefix/etc +append_cache_entry BROKER_PYTHON_HOME PATH $prefix +append_cache_entry BROKER_PYTHON_BINDINGS BOOL false append_cache_entry ENABLE_DEBUG BOOL false append_cache_entry ENABLE_PERFTOOLS BOOL false append_cache_entry ENABLE_PERFTOOLS_DEBUG BOOL false -append_cache_entry ENABLE_JEMALLOC BOOL false +append_cache_entry ENABLE_JEMALLOC BOOL false +append_cache_entry ENABLE_BROKER BOOL true append_cache_entry BinPAC_SKIP_INSTALL BOOL true append_cache_entry BUILD_SHARED_LIBS BOOL true append_cache_entry INSTALL_AUX_TOOLS BOOL true @@ -142,6 +161,10 @@ while [ $# -ne 0 ]; do append_cache_entry CMAKE_INSTALL_PREFIX PATH $optarg append_cache_entry BRO_ROOT_DIR PATH $optarg append_cache_entry PY_MOD_INSTALL_DIR PATH $optarg/lib/broctl + + if [ -z "$user_disabled_broker" ]; then + append_cache_entry BROKER_PYTHON_HOME PATH $optarg + fi ;; --scriptdir=*) append_cache_entry BRO_SCRIPT_INSTALL_PATH STRING $optarg @@ -176,6 +199,11 @@ while [ $# -ne 0 ]; do --enable-jemalloc) append_cache_entry ENABLE_JEMALLOC BOOL true ;; + --disable-broker) + append_cache_entry ENABLE_BROKER BOOL false + remove_cache_entry BROKER_PYTHON_HOME + user_disabled_broker="true" + ;; --disable-broccoli) append_cache_entry INSTALL_BROCCOLI BOOL false ;; @@ -191,11 +219,14 @@ while [ $# -ne 0 ]; do --disable-python) append_cache_entry DISABLE_PYTHON_BINDINGS BOOL true ;; + --disable-pybroker) + append_cache_entry DISABLE_PYBROKER BOOL true + ;; --enable-ruby) append_cache_entry DISABLE_RUBY_BINDINGS BOOL false ;; --with-openssl=*) - append_cache_entry OpenSSL_ROOT_DIR PATH $optarg + append_cache_entry OPENSSL_ROOT_DIR PATH $optarg ;; --with-bind=*) append_cache_entry BIND_ROOT_DIR PATH $optarg @@ -212,9 +243,6 @@ while [ $# -ne 0 ]; do --with-bison=*) append_cache_entry BISON_EXECUTABLE PATH $optarg ;; - --with-perl=*) - append_cache_entry PERL_EXECUTABLE PATH $optarg - ;; --with-geoip=*) append_cache_entry LibGeoIP_ROOT_DIR PATH $optarg ;; @@ -248,6 +276,12 @@ while [ $# -ne 0 ]; do --with-swig=*) append_cache_entry SWIG_EXECUTABLE PATH $optarg ;; + --with-libcaf=*) + append_cache_entry LIBCAF_ROOT_DIR PATH $optarg + ;; + --with-rocksdb=*) + append_cache_entry ROCKSDB_ROOT_DIR PATH $optarg + ;; --binary-package) append_cache_entry BINARY_PACKAGING_MODE BOOL true ;; diff --git a/doc/components/bro-plugins/README.rst b/doc/components/bro-plugins/README.rst new file mode 120000 index 0000000000..8f96f50909 --- /dev/null +++ b/doc/components/bro-plugins/README.rst @@ -0,0 +1 @@ +../../../aux/plugins/README \ No newline at end of file diff --git a/doc/components/bro-plugins/af_packet/README.rst b/doc/components/bro-plugins/af_packet/README.rst new file mode 120000 index 0000000000..b8f745bed2 --- /dev/null +++ b/doc/components/bro-plugins/af_packet/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/af_packet/README \ No newline at end of file diff --git a/doc/components/bro-plugins/elasticsearch/README.rst b/doc/components/bro-plugins/elasticsearch/README.rst new file mode 120000 index 0000000000..8a5b78d689 --- /dev/null +++ b/doc/components/bro-plugins/elasticsearch/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/elasticsearch/README \ No newline at end of file diff --git a/doc/components/bro-plugins/myricom/README.rst b/doc/components/bro-plugins/myricom/README.rst new file mode 120000 index 0000000000..3bfabcdae3 --- /dev/null +++ b/doc/components/bro-plugins/myricom/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/myricom/README \ No newline at end of file diff --git a/doc/components/bro-plugins/netmap/README.rst b/doc/components/bro-plugins/netmap/README.rst new file mode 120000 index 0000000000..819a2bb0e9 --- /dev/null +++ b/doc/components/bro-plugins/netmap/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/netmap/README \ No newline at end of file diff --git a/doc/components/bro-plugins/pf_ring/README.rst b/doc/components/bro-plugins/pf_ring/README.rst new file mode 120000 index 0000000000..5ea666e8c9 --- /dev/null +++ b/doc/components/bro-plugins/pf_ring/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/pf_ring/README \ No newline at end of file diff --git a/doc/components/bro-plugins/redis/README.rst b/doc/components/bro-plugins/redis/README.rst new file mode 120000 index 0000000000..c42051828e --- /dev/null +++ b/doc/components/bro-plugins/redis/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/redis/README \ No newline at end of file diff --git a/doc/components/bro-plugins/tcprs/README.rst b/doc/components/bro-plugins/tcprs/README.rst new file mode 120000 index 0000000000..c0e84fd579 --- /dev/null +++ b/doc/components/bro-plugins/tcprs/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/tcprs/README \ No newline at end of file diff --git a/doc/components/broker/README.rst b/doc/components/broker/README.rst new file mode 120000 index 0000000000..eafa3b8e77 --- /dev/null +++ b/doc/components/broker/README.rst @@ -0,0 +1 @@ +../../../aux/broker/README \ No newline at end of file diff --git a/doc/components/broker/broker-manual.rst b/doc/components/broker/broker-manual.rst new file mode 120000 index 0000000000..90bf8f0833 --- /dev/null +++ b/doc/components/broker/broker-manual.rst @@ -0,0 +1 @@ +../../../aux/broker/broker-manual.rst \ No newline at end of file diff --git a/doc/components/index.rst b/doc/components/index.rst index fe05f13683..85527e9f9c 100644 --- a/doc/components/index.rst +++ b/doc/components/index.rst @@ -17,8 +17,11 @@ current, independent component releases. Broccoli - User Manual Broccoli Python Bindings Broccoli Ruby Bindings + Broker - Bro's (New) Messaging Library (README) + Broker - User Manual BroControl - Interactive Bro management shell Bro-Aux - Small auxiliary tools for Bro + Bro-Plugins - A collection of plugins for Bro BTest - A unit testing framework Capstats - Command-line packet statistic tool PySubnetTree - Python module for CIDR lookups diff --git a/doc/conf.py.in b/doc/conf.py.in index 4faebed3b8..ef9367483a 100644 --- a/doc/conf.py.in +++ b/doc/conf.py.in @@ -66,7 +66,7 @@ master_doc = 'index' # General information about the project. project = u'Bro' -copyright = u'2013, The Bro Project' +copyright = u'2016, The Bro Project' # The version info for the project you're documenting, acts as replacement for # |version| and |release|, also used in various other places throughout the diff --git a/doc/devel/plugins.rst b/doc/devel/plugins.rst index 5e488cfe01..dc1c9a3cd4 100644 --- a/doc/devel/plugins.rst +++ b/doc/devel/plugins.rst @@ -3,7 +3,7 @@ Writing Bro Plugins =================== -Bro internally provides plugin API that enables extending +Bro internally provides a plugin API that enables extending the system dynamically, without modifying the core code base. That way custom code remains self-contained and can be maintained, compiled, and installed independently. Currently, plugins can add the following @@ -32,7 +32,7 @@ Quick Start =========== Writing a basic plugin is quite straight-forward as long as one -follows a few conventions. In the following we walk a simple example +follows a few conventions. In the following we create a simple example plugin that adds a new built-in function (bif) to Bro: we'll add ``rot13(s: string) : string``, a function that rotates every character in a string by 13 places. @@ -81,7 +81,7 @@ The syntax of this file is just like any other ``*.bif`` file; we won't go into it here. Now we can already compile our plugin, we just need to tell the -configure script that ``init-plugin`` put in place where the Bro +configure script (that ``init-plugin`` created) where the Bro source tree is located (Bro needs to have been built there first):: # cd rot13-plugin @@ -99,7 +99,7 @@ option:: # export BRO_PLUGIN_PATH=/path/to/rot13-plugin/build # bro -N [...] - Plugin: Demo::Rot13 - (dynamic, version 1) + Demo::Rot13 - (dynamic, version 0.1) [...] That looks quite good, except for the dummy description that we should @@ -108,28 +108,30 @@ is about. We do this by editing the ``config.description`` line in ``src/Plugin.cc``, like this:: [...] - plugin::Configuration Configure() + plugin::Configuration Plugin::Configure() { plugin::Configuration config; config.name = "Demo::Rot13"; config.description = "Caesar cipher rotating a string's characters by 13 places."; - config.version.major = 1; - config.version.minor = 0; + config.version.major = 0; + config.version.minor = 1; return config; } [...] +Now rebuild and verify that the description is visible:: + # make [...] # bro -N | grep Rot13 - Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) + Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 0.1) -Better. Bro can also show us what exactly the plugin provides with the +Bro can also show us what exactly the plugin provides with the more verbose option ``-NN``:: # bro -NN [...] - Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) + Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 0.1) [Function] Demo::rot13 [...] @@ -157,10 +159,12 @@ The installed version went into ``/lib/bro/plugins/Demo_Rot13``. One can distribute the plugin independently of Bro for others to use. -To distribute in source form, just remove the ``build/`` (``make -distclean`` does that) and then tar up the whole ``rot13-plugin/`` +To distribute in source form, just remove the ``build/`` directory +(``make distclean`` does that) and then tar up the whole ``rot13-plugin/`` directory. Others then follow the same process as above after -unpacking. To distribute the plugin in binary form, the build process +unpacking. + +To distribute the plugin in binary form, the build process conveniently creates a corresponding tarball in ``build/dist/``. In this case, it's called ``Demo_Rot13-0.1.tar.gz``, with the version number coming out of the ``VERSION`` file that ``init-plugin`` put @@ -169,14 +173,14 @@ plugin, but no further source files. Optionally, one can include further files by specifying them in the plugin's ``CMakeLists.txt`` through the ``bro_plugin_dist_files`` macro; the skeleton does that for ``README``, ``VERSION``, ``CHANGES``, and ``COPYING``. To use the -plugin through the binary tarball, just unpack it and point -``BRO_PLUGIN_PATH`` there; or copy it into -``/lib/bro/plugins/`` directly. +plugin through the binary tarball, just unpack it into +``/lib/bro/plugins/``. Alternatively, if you unpack +it in another location, then you need to point ``BRO_PLUGIN_PATH`` there. Before distributing your plugin, you should edit some of the meta files that ``init-plugin`` puts in place. Edit ``README`` and ``VERSION``, and update ``CHANGES`` when you make changes. Also put a -license file in place as ``COPYING``; if BSD is fine, you find a +license file in place as ``COPYING``; if BSD is fine, you will find a template in ``COPYING.edit-me``. Plugin Directory Layout @@ -193,7 +197,7 @@ directory. With the skeleton, ```` corresponds to ``build/``. must exist, and its content must consist of a single line with the qualified name of the plugin (e.g., "Demo::Rot13"). -``/lib/--.so`` +``/lib/.-.so`` The shared library containing the plugin's compiled code. Bro will load this in dynamically at run-time if OS and architecture match the current platform. @@ -205,8 +209,15 @@ directory. With the skeleton, ```` corresponds to ``build/``. "@load"ed. ``scripts``/__load__.bro - A Bro script that will be loaded immediately when the plugin gets - activated. See below for more information on activating plugins. + A Bro script that will be loaded when the plugin gets activated. + When this script executes, any BiF elements that the plugin + defines will already be available. See below for more information + on activating plugins. + +``scripts``/__preload__.bro + A Bro script that will be loaded when the plugin gets activated, + but before any BiF elements become available. See below for more + information on activating plugins. ``lib/bif/`` Directory with auto-generated Bro scripts that declare the plugin's @@ -215,8 +226,8 @@ directory. With the skeleton, ```` corresponds to ``build/``. Any other files in ```` are ignored by Bro. By convention, a plugin should put its custom scripts into sub folders -of ``scripts/``, i.e., ``scripts//