diff --git a/.gitmodules b/.gitmodules index 24375ce23d..91f39e3d04 100644 --- a/.gitmodules +++ b/.gitmodules @@ -22,3 +22,6 @@ [submodule "aux/plugins"] path = aux/plugins url = git://git.bro.org/bro-plugins +[submodule "aux/broker"] + path = aux/broker + url = git://git.bro.org/broker diff --git a/CHANGES b/CHANGES index 85de307c2a..7bcf6a3ef4 100644 --- a/CHANGES +++ b/CHANGES @@ -1,4 +1,1588 @@ +2.4-228 | 2015-12-19 13:40:09 -0800 + + * Updating BroControl submodule. + +2.4-227 | 2015-12-18 17:47:24 -0800 + + * Update host name in windows-version-detection.bro. (Aaron Eppert) + + * Update installation instructions to mention OpenSSL dependency for + newer OS X version. (Johanna Amann) + + * Change a stale bro-ids.org to bro.org. (Johanna Amann) + + * StartTLS support for IRC. (Johanna Amann) + + * Adding usage guard to canonifier script. (Robin Sommer) + +2.4-217 | 2015-12-04 16:50:46 -0800 + + * SIP scripts code cleanup. (Seth Hall) + + - Daniel Guerra pointed out a type issue for SIP request and + response code length fields which is now corrected. + + - Some redundant code was removed. + + - if/else tree modified to use switch instead. + +2.4-214 | 2015-12-04 16:40:15 -0800 + + * Delaying BinPAC initializaton until afte plugins have been + activated. (Robin Sommer) + +2.4-213 | 2015-12-04 15:25:48 -0800 + + * Use better data structure for storing BPF filters. (Robin Sommer) + +2.4-211 | 2015-11-17 13:28:29 -0800 + + * Making cluster reconnect timeout configurable. (Robin Sommer) + + * Bugfix for child process' communication loop. (Robin Sommer) + +2.4-209 | 2015-11-16 07:31:22 -0800 + + * Updating submodule(s). + +2.4-207 | 2015-11-10 13:34:42 -0800 + + * Fix to compile with OpenSSL that has SSLv3 disalbed. (Christoph + Pietsch) + + * Fix potential race condition when logging VLAN info to conn.log. + (Daniel Thayer) + +2.4-201 | 2015-10-27 16:11:15 -0700 + + * Updating NEWS. (Robin Sommer) + +2.4-200 | 2015-10-26 16:57:39 -0700 + + * Adding missing file. (Robin Sommer) + +2.4-199 | 2015-10-26 16:51:47 -0700 + + * Fix problem with the JSON Serialization code. (Aaron Eppert) + +2.4-188 | 2015-10-26 14:11:21 -0700 + + * Extending rexmit_inconsistency() event to receive an additional + parameter with the packet's TCP flags, if available. (Robin + Sommer) + +2.4-187 | 2015-10-26 13:43:32 -0700 + + * Updating NEWS for new plugins. (Robin Sommer) + +2.4-186 | 2015-10-23 15:07:06 -0700 + + * Removing pcap options for AF_PACKET support. Addresses BIT-1363. + (Robin Sommer) + + * Correct a typo in controller.bro documentation. (Daniel Thayer) + + * Extend SSL DPD signature to allow alert before server_hello. + (Johanna Amann) + + * Make join_string_vec work with vectors containing empty elements. + (Johanna Amann) + + * Fix support for HTTP CONNECT when server adds headers to response. + (Eric Karasuda). + + * Load static CA list for validation tests too. (Johanna Amann) + + * Remove cluster certificate validation script. (Johanna Amann) + + * Fix a bug in diff-remove-x509-names canonifier. (Daniel Thayer) + + * Fix test canonifiers in scripts/policy/protocols/ssl. (Daniel + Thayer) + +2.4-169 | 2015-10-01 17:21:21 -0700 + + * Fixed parsing of V_ASN1_GENERALIZEDTIME timestamps in x509 + certificates. (Yun Zheng Hu) + + * Improve X509 end-of-string-check code. (Johanna Amann) + + * Refactor X509 generalizedtime support and test. (Johanna Amann) + + * Fix case of offset=-1 (EOF) for RAW reader. Addresses BIT-1479. + (Johanna Amann) + + * Improve a number of test canonifiers. (Daniel Thayer) + + * Remove unnecessary use of TEST_DIFF_CANONIFIER. (Daniel Thayer) + + * Fixed some test canonifiers to read only from stdin + + * Remove unused test canonifier scripts. (Daniel Thayer) + + * A potpourri of updates and improvements across the documentation. + (Daniel Thayer) + + * Add configure option to disable Broker Python bindings. Also + improve the configure summary output to more clearly show whether + or not Broker Python bindings will be built. (Daniel Thayer) + +2.4-131 | 2015-09-11 12:16:39 -0700 + + * Add README.rst symlink. Addresses BIT-1413 (Vlad Grigorescu) + +2.4-129 | 2015-09-11 11:56:04 -0700 + + * hash-all-files.bro depends on base/files/hash (Richard van den Berg) + + * Make dns_max_queries redef-able, and bump default to 25. Addresses + BIT-1460 (Vlad Grigorescu) + +2.4-125 | 2015-09-03 20:10:36 -0700 + + * Move SIP analyzer to flowunit instead of datagram Addresses + BIT-1458 (Vlad Grigorescu) + +2.4-122 | 2015-08-31 14:39:41 -0700 + + * Add a number of out-of-bound checks to layer 2 code. Addresses + BIT-1463 (Johanna Amann) + + * Fix error in 2.4 release notes regarding SSH events. (Robin + Sommer) + +2.4-118 | 2015-08-31 10:55:29 -0700 + + * Fix FreeBSD build errors (Johanna Amann) + +2.4-117 | 2015-08-30 22:16:24 -0700 + + * Fix initialization of a pointer in RDP analyzer. (Daniel + Thayer/Robin Sommer) + +2.4-115 | 2015-08-30 21:57:35 -0700 + + * Enable Bro to leverage packet fanout mode on Linux. (Kris + Nielander). + + ## Toggle whether to do packet fanout (Linux-only). + const Pcap::packet_fanout_enable = F &redef; + + ## If packet fanout is enabled, the id to sue for it. This should be shared amongst + ## worker processes processing the same socket. + const Pcap::packet_fanout_id = 0 &redef; + + ## If packet fanout is enabled, whether packets are to be defragmented before + ## fanout is applied. + const Pcap::packet_fanout_defrag = T &redef; + + * Allow libpcap buffer size to be set via configuration. (Kris Nielander) + + ## Number of Mbytes to provide as buffer space when capturing from live + ## interfaces. + const Pcap::bufsize = 128 &redef; + + * Move the pcap-related script-level identifiers into the new Pcap + namespace. (Robin Sommer) + + snaplen -> Pcap::snaplen + precompile_pcap_filter() -> Pcap::precompile_pcap_filter() + install_pcap_filter() -> Pcap::install_pcap_filter() + pcap_error() -> Pcap::pcap_error() + + +2.4-108 | 2015-08-30 20:14:31 -0700 + + * Update Base64 decoding. (Jan Grashoefer) + + - A new built-in function, decode_base64_conn() for Base64 + decoding. It works like decode_base64() but receives an + additional connection argument that will be used for + reporting decoding errors into weird.log (instead of + reporter.log). + + - FTP, POP3, and HTTP analyzers now likewise log Base64 + decoding errors to weird.log. + + - The built-in functions decode_base64_custom() and + encode_base64_custom() are now deprecated. Their + functionality is provided directly by decode_base64() and + encode_base64(), which take an optional parameter to change + the Base64 alphabet. + + * Fix potential crash if TCP header was captured incompletely. + (Robin Sommer) + +2.4-103 | 2015-08-29 10:51:55 -0700 + + * Make ASN.1 date/time parsing more robust. (Johanna Amann) + + * Be more permissive on what characters we accept as an unquoted + multipart boundary. Addresses BIT-1459. (Johanna Amann) + +2.4-99 | 2015-08-25 07:56:57 -0700 + + * Add ``Q`` and update ``I`` documentation for connection history + field. Addresses BIT-1466. (Vlad Grigorescu) + +2.4-96 | 2015-08-21 17:37:56 -0700 + + * Update SIP analyzer. (balintm) + + - Allows space on both sides of ':'. + - Require CR/LF after request/reply line. + +2.4-94 | 2015-08-21 17:31:32 -0700 + + * Add file type detection support for video/MP2T. (Mike Freemon) + +2.4-93 | 2015-08-21 17:23:39 -0700 + + * Make plugin install honor DESTDIR= convention. (Jeff Barber) + +2.4-89 | 2015-08-18 07:53:36 -0700 + + * Fix diff-canonifier-external to use basename of input file. + (Daniel Thayer) + +2.4-87 | 2015-08-14 08:34:41 -0700 + + * Removing the yielding_teredo_decapsulation option. (Robin Sommer) + +2.4-86 | 2015-08-12 17:02:24 -0700 + + * Make Teredo DPD signature more precise. (Martina Balint) + +2.4-84 | 2015-08-10 14:44:39 -0700 + + * Add hook 'HookSetupAnalyzerTree' to allow plugins access to a + connection's initial analyzer tree for customization. (James + Swaro) + + * Plugins now look for a file "__preload__.bro" in the top-level + script directory. If found, they load it first, before any scripts + defining BiF elements. This can be used to define types that the + BiFs already depend on (like a custom type for an event argument). + (Robin Sommer) + +2.4-81 | 2015-08-08 07:38:42 -0700 + + * Fix a test that is failing very frequently. (Daniel Thayer) + +2.4-78 | 2015-08-06 22:25:19 -0400 + + * Remove build dependency on Perl (now requiring Python instad). + (Daniel Thayer) + + * CID 1314754: Fixing unreachable code in RSH analyzer. (Robin + Sommer) + + * CID 1312752: Add comment to mark 'case' fallthrough as ok. (Robin + Sommer) + + * CID 1312751: Removing redundant assignment. (Robin Sommer) + +2.4-73 | 2015-07-31 08:53:49 -0700 + + * BIT-1429: SMTP logs now include CC: addresses. (Albert Zaharovits) + +2.4-70 | 2015-07-30 07:23:44 -0700 + + * Updated detection of Flash and AdobeAIR. (Jan Grashoefer) + + * Adding tests for Flash version parsing and browser plugin + detection. (Robin Sommer) + +2.4-63 | 2015-07-28 12:26:37 -0700 + + * Updating submodule(s). + +2.4-61 | 2015-07-28 12:13:39 -0700 + + * Renaming config.h to bro-config.h. (Robin Sommer) + +2.4-58 | 2015-07-24 15:06:07 -0700 + + * Add script protocols/conn/vlan-logging.bro to record VLAN data in + conn.log. (Aaron Brown) + + * Add field "vlan" and "inner_vlan" to connection record. (Aaron + Brown) + + * Save the inner vlan in the Packet object for Q-in-Q setups. (Aaron + Brown) + + * Increasing plugin API version for recent packet source changes. + (Robin Sommer) + + * Slightly earlier protocol confirmation for POP3. (Johanna Amann) + +2.4-46 | 2015-07-22 10:56:40 -0500 + + * Fix broker python bindings install location to track --prefix. + (Jon Siwek) + +2.4-45 | 2015-07-21 15:19:43 -0700 + + * Enabling Broker by default. This means CAF is now a required + dependency, altjough for now at least, there's still a switch + --disable-broker to turn it off. + + * Requiring a C++11 compiler, and turning on C++11 support. (Robin + Sommer) + + * Tweaking the listing of hooks in "bro -NN" for consistency. (Robin + Sommer) + +2.4-41 | 2015-07-21 08:35:17 -0700 + + * Fixing compiler warning. (Robin Sommer) + + * Updates to IANA TLS registry. (Johanna Amann) + +2.4-38 | 2015-07-20 15:30:35 -0700 + + * Refactor code to use a common Packet type throught. (Jeff + Barber/Robin Sommer) + + * Extend parsing layer 2 and keeping track of layer 3 protoco. (Jeff Barber) + + * Add a raw_packet() event that generated for all packets and + include layer 2 information. (Jeff Barber) + +2.4-27 | 2015-07-15 13:31:49 -0700 + + * Fix race condition in intel test. (Johanna Amann) + +2.4-24 | 2015-07-14 08:04:11 -0700 + + * Correct Perl package name on FreeBSD in documentation.(Justin Azoff) + + * Adding an environment variable to BTest configuration for external + scripts. (Robin Sommer) + +2.4-20 | 2015-07-03 10:40:21 -0700 + + * Adding a weird for when truncated packets lead TCP reassembly to + ignore content. (Robin Sommer) + +2.4-19 | 2015-07-03 09:04:54 -0700 + + * A set of tests exercising IP defragmentation and TCP reassembly. + (Robin Sommer) + +2.4-17 | 2015-06-28 13:02:41 -0700 + + * BIT-1314: Add detection for Quantum Insert attacks. The TCP + reassembler can now keep a history of old TCP segments using the + tcp_max_old_segments option. An overlapping segment with different + data will then generate an rexmit_inconsistency event. The default + for tcp_max_old_segments is zero, which disabled any additional + buffering. (Yun Zheng Hu/Robin Sommer) + +2.4-14 | 2015-06-28 12:30:12 -0700 + + * BIT-1400: Allow '<' and '>' in MIME multipart boundaries. The spec + doesn't actually seem to permit these, but they seem to occur in + the wild. (Jon Siwek) + +2.4-12 | 2015-06-28 12:21:11 -0700 + + * BIT-1399: Trying to decompress deflated HTTP content even when + zlib headers are missing. (Seth Hall) + +2.4-10 | 2015-06-25 07:11:17 -0700 + + * Correct a name used in a header identifier (Justin Azoff) + +2.4-8 | 2015-06-24 07:50:50 -0700 + + * Restore the --load-seeds cmd-line option and enable the short + options -G/-H for --load-seeds/--save-seeds. (Daniel Thayer) + +2.4-6 | 2015-06-19 16:26:40 -0700 + + * Generate protocol confirmations for Modbus, making it appear as a + confirmed service in conn.log. (Seth Hall) + + * Put command line options in alphabetical order. (Daniel Thayer) + + * Removing dead code for no longer supported -G switch. (Robin + Sommer) (Robin Sommer) + +2.4 | 2015-06-09 07:30:53 -0700 + + * Release 2.4. + + * Fixing tiny thing in NEWS. (Robin Sommer) + +2.4-beta-42 | 2015-06-08 09:41:39 -0700 + + * Fix reporter errors with GridFTP traffic. (Robin Sommer) + +2.4-beta-40 | 2015-06-06 08:20:52 -0700 + + * PE Analyzer: Change how we calculate the rva_table size. (Vlad Grigorescu) + +2.4-beta-39 | 2015-06-05 09:09:44 -0500 + + * Fix a unit test to check for Broker requirement. (Jon Siwek) + +2.4-beta-38 | 2015-06-04 14:48:37 -0700 + + * Test for Broker termination. (Robin Sommer) + +2.4-beta-37 | 2015-06-04 07:53:52 -0700 + + * BIT-1408: Improve I/O loop and Broker IOSource. (Jon Siwek) + +2.4-beta-34 | 2015-06-02 10:37:22 -0700 + + * Add signature support for F4M files. (Seth Hall) + +2.4-beta-32 | 2015-06-02 09:43:31 -0700 + + * A larger set of documentation updates, fixes, and extentions. + (Daniel Thayer) + +2.4-beta-14 | 2015-06-02 09:16:44 -0700 + + * Add memleak btest for attachments over SMTP. (Vlad Grigorescu) + + * BIT-1410: Fix flipped tx_hosts and rx_hosts in files.log. Reported + by Ali Hadi. (Vlad Grigorescu) + + * Updating the Mozilla root certs. (Seth Hall) + + * Updates for the urls.bro script. Fixes BIT-1404. (Seth Hall) + +2.4-beta-6 | 2015-05-28 13:20:44 -0700 + + * Updating submodule(s). + +2.4-beta-2 | 2015-05-26 08:58:37 -0700 + + * Fix segfault when DNS is not available. Addresses BIT-1387. (Frank + Meier and Robin Sommer) + +2.4-beta | 2015-05-07 21:55:31 -0700 + + * Release 2.4-beta. + + * Update local-compat.test (Johanna Amann) + +2.3-913 | 2015-05-06 09:58:00 -0700 + + * Add /sbin to PATH in btest.cfg and remove duplicate default_path. + (Daniel Thayer) + +2.3-911 | 2015-05-04 09:58:09 -0700 + + * Update usage output and list of command line options. (Daniel + Thayer) + + * Fix to ssh/geo-data.bro for unset directions. (Vlad Grigorescu) + + * Improve SIP logging and remove reporter messages. (Seth Hall) + +2.3-905 | 2015-04-29 17:01:30 -0700 + + * Improve SIP logging and remove reporter messages. (Seth Hall) + +2.3-903 | 2015-04-27 17:27:59 -0700 + + * BIT-1350: Improve record coercion type checking. (Jon Siwek) + +2.3-901 | 2015-04-27 17:25:27 -0700 + + * BIT-1384: Remove -O (optimize scripts) command-line option, which + hadn't been working for a while already. (Jon Siwek) + +2.3-899 | 2015-04-27 17:22:42 -0700 + + * Fix the -J/--set-seed cmd-line option. (Daniel Thayer) + + * Remove unused -l, -L, and -Z cmd-line options. (Daniel Thayer) + +2.3-892 | 2015-04-27 08:22:22 -0700 + + * Fix typos in the Broker BIF documentation. (Daniel Thayer) + + * Update installation instructions and remove outdated references. + (Johanna Amann) + + * Easier support for systems with tcmalloc_minimal installed. (Seth + Hall) + +2.3-884 | 2015-04-23 12:30:15 -0500 + + * Fix some outdated documentation unit tests. (Jon Siwek) + +2.3-883 | 2015-04-23 07:10:36 -0700 + + * Fix -N option to work with builtin plugins as well. (Robin Sommer) + +2.3-882 | 2015-04-23 06:59:40 -0700 + + * Add missing .pac dependencies for some binpac analyzer targets. + (Jon Siwek) + +2.3-879 | 2015-04-22 10:38:07 -0500 + + * Fix compile errors. (Jon Siwek) + +2.3-878 | 2015-04-22 08:21:23 -0700 + + * Fix another compiler warning in DTLS. (Johanna Amann) + +2.3-877 | 2015-04-21 20:14:16 -0700 + + * Adding missing include. (Robin Sommer) + +2.3-876 | 2015-04-21 16:40:10 -0700 + + * Attempt at fixing a potential std::length_error exception in RDP + analyzer. Addresses BIT-1337. (Robin Sommer) + + * Fixing compile problem caused by overeager factorization. (Robin + Sommer) + +2.3-874 | 2015-04-21 16:09:20 -0700 + + * Change details of escaping when logging/printing. (Seth Hall/Robin + Sommer) + + - Log files now escape non-printable characters consistently + as "\xXX'. Furthermore, backslashes are escaped as "\\", + making the representation fully reversible. + + - When escaping via script-level functions (escape_string, + clean), we likewise now escape consistently with "\xXX" and + "\\". + + - There's no "alternative" output style anymore, i.e., fmt() + '%A' qualifier is gone. + + Addresses BIT-1333. + + * Remove several BroString escaping methods that are no longer + useful. (Seth Hall) + +2.3-864 | 2015-04-21 15:24:02 -0700 + + * A SIP protocol analyzer. (Vlad Grigorescu) + + Activity gets logged into sip.log. It generates the following + events: + + event sip_request(c: connection, method: string, original_URI: string, version: string); + event sip_reply(c: connection, version: string, code: count, reason: string); + event sip_header(c: connection, is_orig: bool, name: string, value: string); + event sip_all_headers(c: connection, is_orig: bool, hlist: mime_header_list); + event sip_begin_entity(c: connection, is_orig: bool); + event sip_end_entity(c: connection, is_orig: bool); + + The analyzer support SIP over UDP currently. + + * BIT-1343: Factor common ASN.1 code from RDP, SNMP, and Kerberos + analyzers. (Jon Siwek/Robin Sommer) + +2.3-838 | 2015-04-21 13:40:12 -0700 + + * BIT-1373: Fix vector index assignment reference count bug. (Jon Siwek) + +2.3-836 | 2015-04-21 13:37:31 -0700 + + * Fix SSH direction field being unset. Addresses BIT-1365. (Vlad + Grigorescu) + +2.3-835 | 2015-04-21 16:36:00 -0500 + + * Clarify Broker examples. (Jon Siwek) + +2.3-833 | 2015-04-21 12:38:32 -0700 + + * A Kerberos protocol analyzer. (Vlad Grigorescu) + + Activity gets logged into kerberos.log. It generates the following + events: + + event krb_as_request(c: connection, msg: KRB::KDC_Request); + event krb_as_response(c: connection, msg: KRB::KDC_Response); + event krb_tgs_request(c: connection, msg: KRB::KDC_Request); + event krb_tgs_response(c: connection, msg: KRB::KDC_Response); + event krb_ap_request(c: connection, ticket: KRB::Ticket, opts: KRB::AP_Options); + event krb_priv(c: connection, is_orig: bool); + event krb_safe(c: connection, is_orig: bool, msg: KRB::SAFE_Msg); + event krb_cred(c: connection, is_orig: bool, tickets: KRB::Ticket_Vector); + event krb_error(c: connection, msg: KRB::Error_Msg); + +2.3-793 | 2015-04-20 20:51:00 -0700 + + * Add decoding of PROXY-AUTHORIZATION header to HTTP analyze, + treating it the same as AUTHORIZATION. (Josh Liburdi) + + * Remove deprecated fields "hot" and "addl" from the connection + record. Remove the functions append_addl() and + append_addl_marker(). (Robin Sommer) + + * Removing the NetFlow analyzer, which hasn't been used anymore + since then corresponding command-line option went away. (Robin + Sommer) + +2.3-787 | 2015-04-20 19:15:23 -0700 + + * A file analyzer for Portable Executables. (Vlad Grigorescu/Seth + Hall). + + Activity gets logged into pe.log. It generates the following + events: + + event pe_dos_header(f: fa_file, h: PE::DOSHeader); + event pe_dos_code(f: fa_file, code: string); + event pe_file_header(f: fa_file, h: PE::FileHeader); + event pe_optional_header(f: fa_file, h: PE::OptionalHeader); + event pe_section_header(f: fa_file, h: PE::SectionHeader); + +2.3-741 | 2015-04-20 13:12:39 -0700 + + * API changes to file analysis mime type detection. Removed + "file_mime_type" and "file_mime_types" event, replacing them with + a new event called "file_metadata_inferred". Addresses BIT-1368. + (Jon Siwek) + + * A large series of improvements for file type identification. This + inludes a many signature updates (new types, cleanup, performance + improvments) and splitting out signatures into subfiles. (Seth + Hall) + + * Fix an issue with files having gaps before the bof_buffer is + filled, which could lead to file type identification not working + correctly. (Seth Hall) + + * Fix an issue with packet loss in HTTP file reporting for file type + identification wasn't working correctly zero-length bodies. (Seth + Hall) + + * X.509 certificates are now populating files.log with the mime type + application/pkix-cert. (Seth Hall) + + * Normalized some FILE_ANALYSIS debug messages. (Seth Hall) + +2.3-725 | 2015-04-20 12:54:54 -0700 + + * Updating submodule(s). + +2.3-724 | 2015-04-20 14:11:02 -0500 + + * Fix uninitialized field in raw input reader. (Jon Siwek) + +2.3-722 | 2015-04-20 12:59:03 -0500 + + * Remove unneeded documentation cross-referencing. (Jon Siwek) + +2.3-721 | 2015-04-20 12:47:05 -0500 + + * BIT-1380: Improve Broxygen output of &default expressions. + (Jon Siwek) + +2.3-720 | 2015-04-17 14:18:26 -0700 + + * Updating NEWS. + +2.3-716 | 2015-04-17 13:06:37 -0700 + + * Add seeking functionality to raw reader. One can now add an option + "offset" to the config map. Positive offsets are interpreted to be + from the beginning of the file, negative from the end of the file + (-1 is end of file). Only works for raw reader in streaming or + manual mode. Does not work with executables. Addresses BIT-985. + (Johanna Amann) + + * Allow setting packet and byte thresholds for connections. (Johanna Amann) + + This extends the ConnSize analyzer to be able to raise events when + each direction of a connection crosses a certain amount of bytes + or packets. + + Thresholds are set using: + - set_conn_bytes_threshold(c$id, [num-bytes], [direction]); + - set_conn_packets_threshold(c$id, [num-packets], [direction]); + + They raise the events, respectively: + - event conn_bytes_threshold_crossed(c: connection, threshold: count, is_orig: bool) + - event conn_packets_threshold_crossed(c: connection, threshold: count, is_orig: bool) + + Current thresholds can be examined using get_conn_bytes_threshold() + and get_conn_packets_threshold(). + + Only one threshold can be set per connection. + + * Add high-level API for packet/bytes thresholding in + base/protocols/conn/thresholds.bro that holds lists of thresholds + and raises an event for each threshold exactly once. (Johanna + Amann) + + * Fix a bug where child packet analyzers of the TCP analyzer + where not found using FindChild. + + * Update GridFTP analyzer to use connection thresholding instead of + polling. (Johanna Amann) + +2.3-709 | 2015-04-17 12:37:32 -0700 + + * Fix addressing the dreaded "internal error: unknown msg type 115 + in Poll()". (Jon Siwek) + + This patch removes the error handling code for overload conditions + in the main process that could cause trouble down the road. The + "chunked_io_buffer_soft_cap" script variable can now tune when the + client process begins shutting down peer connections, and the + default setting is now double what it used to be. Addresses + BIT-1376. + +2.3-707 | 2015-04-17 10:57:59 -0500 + + * Add more info about Broker to NEWS. (Jon Siwek) + +2.3-705 | 2015-04-16 08:16:45 -0700 + + * Update Mozilla CA list. (Johanna Amann) + + * Update tests to have them keep using older certificates where + appropiate. (Johanna Amann) + +2.3-699 | 2015-04-16 09:51:58 -0500 + + * Fix the to_count function to use strtoull versus strtoll. + (Jon Siwek) + +2.3-697 | 2015-04-15 09:51:15 -0700 + + * Removing error check verifying that an ASCII writer has been + properly finished. Instead of aborting, we now just clean up in + that case and proceed. Addresses BIT-1331. (Robin Sommer) + +2.3-696 | 2015-04-14 15:56:36 -0700 + + * Update sqlite to 3.8.9 + +2.3-695 | 2015-04-13 10:34:42 -0500 + + * Fix iterator invalidation in broker::Manager dtor. (Jon Siwek) + + * Add paragraph to plugin documentation. (Robin Sommer) + +2.3-693 | 2015-04-11 10:56:31 -0700 + + * BIT-1367: improve coercion of anonymous records in set constructor. + (Jon Siwek) + + * Allow to specify ports for sftp log rotator. (Johanna Amann) + +2.3-690 | 2015-04-10 21:51:10 -0700 + + * Make sure to always delete the remote serializer. Addresses + BIT-1306 and probably also BIT-1356. (Robin Sommer) + + * Cleaning up --help. -D and -Y/y were still listed, even though + they had no effect anymore. Removing some dead code along with -D. + Addresses BIT-1372. (Robin Sommer) + +2.3-688 | 2015-04-10 08:10:44 -0700 + + * Update SQLite to 3.8.8.3. + +2.3-687 | 2015-04-10 07:32:52 -0700 + + * Remove stale signature benchmarking code (-L command-line option). + (Jon Siwek) + + * BIT-844: fix UDP payload signatures to match packet-wise. (Jon + Siwek) + +2.3-682 | 2015-04-09 12:07:00 -0700 + + * Fixing input readers' component type. (Robin Sommer) + + * Tiny spelling correction. (Seth Hall) + +2.3-680 | 2015-04-06 16:02:43 -0500 + + * BIT-1371: remove CMake version check from binary package scripts. + (Jon Siwek) + +2.3-679 | 2015-04-06 10:16:36 -0500 + + * Increase some unit test timeouts. (Jon Siwek) + + * Fix Coverity warning in RDP analyzer. (Jon Siwek) + +2.3-676 | 2015-04-02 10:10:39 -0500 + + * BIT-1366: improve checksum offloading warning. + (Frank Meier, Jon Siwek) + +2.3-675 | 2015-03-30 17:05:05 -0500 + + * Add an RDP analyzer. (Josh Liburdi, Seth Hall, Johanna Amann) + +2.3-640 | 2015-03-30 13:51:51 -0500 + + * BIT-1359: Limit maximum number of DTLS fragments to 30. (Johanna Amann) + +2.3-637 | 2015-03-30 12:02:07 -0500 + + * Increase timeout duration in some broker tests. (Jon Siwek) + +2.3-636 | 2015-03-30 11:26:32 -0500 + + * Updates related to SSH analysis. (Jon Siwek) + + - Some scripts used wrong SSH module/namespace scoping on events. + - Fix outdated notice documentation related to SSH password guessing. + - Add a unit test for SSH pasword guessing notice. + +2.3-635 | 2015-03-30 11:02:45 -0500 + + * Fix outdated documentation unit tests. (Jon Siwek) + +2.3-634 | 2015-03-30 10:22:45 -0500 + + * Add a canonifier to a unit test's output. (Jon Siwek) + +2.3-633 | 2015-03-25 18:32:59 -0700 + + * Log::write in signature framework was missing timestamp. + (Andrew Benson/Michel Laterman) + +2.3-631 | 2015-03-25 11:03:12 -0700 + + * New SSH analyzer. (Vlad Grigorescu) + +2.3-600 | 2015-03-25 10:23:46 -0700 + + * Add defensive checks in code to calculate log rotation intervals. + (Pete Nelson). + +2.3-597 | 2015-03-23 12:50:04 -0700 + + * DTLS analyzer. (Johanna Amann) + + * Implement correct parsing of TLS record fragmentation. (Johanna + Amann) + +2.3-582 | 2015-03-23 11:34:25 -0700 + + * BIT-1313: In debug builds, "bro -B " now supports "all" and + "help" for "". "all" enables all debug streams. "help" prints a + list of available debug streams. (John Donnelly/Robin Sommer). + + * BIT-1324: Allow logging filters to inherit default path from + stream. This allows the path for the default filter to be + specified explicitly through $path="..." when creating a stream. + Adapted the existing Log::create_stream calls to explicitly + specify a path value. (Jon Siwek) + + * BIT-1199: Change the way the input framework deals with values it + cannot convert into BroVals, raising error messages instead of + aborting execution. (Johanna Amann) + + * BIT-788: Use DNS QR field to better identify flow direction. (Jon + Siwek) + +2.3-572 | 2015-03-23 13:04:53 -0500 + + * BIT-1226: Fix an example in quickstart docs. (Jon siwek) + +2.3-570 | 2015-03-23 09:51:20 -0500 + + * Correct a spelling error (Daniel Thayer) + + * Improvement to SSL analyzer failure mode. (Johanna Amann) + +2.3-565 | 2015-03-20 16:27:41 -0500 + + * BIT-978: Improve documentation of 'for' loop iterator invalidation. + (Jon Siwek) + +2.3-564 | 2015-03-20 11:12:02 -0500 + + * BIT-725: Remove "unmatched_HTTP_reply" weird. (Jon Siwek) + +2.3-562 | 2015-03-20 10:31:02 -0500 + + * BIT-1207: Add unit test to catch breaking changes to local.bro + (Jon Siwek) + + * Fix failing sqlite leak test (Johanna Amann) + +2.3-560 | 2015-03-19 13:17:39 -0500 + + * BIT-1255: Increase default values of + "tcp_max_above_hole_without_any_acks" and "tcp_max_initial_window" + from 4096 to 16384 bytes. (Jon Siwek) + +2.3-559 | 2015-03-19 12:14:33 -0500 + + * BIT-849: turn SMTP reporter warnings into weirds, + "smtp_nested_mail_transaction" and "smtp_unmatched_end_of_data". + (Jon Siwek) + +2.3-558 | 2015-03-18 22:50:55 -0400 + + * DNS: Log the type number for the DNS_RR_unknown_type weird. (Vlad Grigorescu) + +2.3-555 | 2015-03-17 15:57:13 -0700 + + * Splitting test-all Makefile target into Bro tests and test-aux. + (Robin Sommer) + +2.3-554 | 2015-03-17 15:40:39 -0700 + + * Deprecate &rotate_interval, &rotate_size, &encrypt. Addresses + BIT-1305. (Jon Siwek) + +2.3-549 | 2015-03-17 09:12:18 -0700 + + * BIT-1077: Fix HTTP::log_server_header_names. Before, it just + re-logged fields from the client side. (Jon Siwek) + +2.3-547 | 2015-03-17 09:07:51 -0700 + + * Update certificate validation script to cache valid intermediate + chains that it encounters on the wire and use those to try to + validate chains that might be missing intermediate certificates. + (Johanna Amann) + +2.3-541 | 2015-03-13 15:44:08 -0500 + + * Make INSTALL a symlink to doc/install/install.rst (Jon siwek) + + * Fix Broxygen coverage. (Jon Siwek) + +2.3-539 | 2015-03-13 14:19:27 -0500 + + * BIT-1335: Include timestamp in default extracted file names. + And add a policy script to extract all files. (Jon Siwek) + + * BIT-1311: Identify GRE tunnels as Tunnel::GRE, not Tunnel::IP. + (Jon Siwek) + + * BIT-1309: Add Connection class getter methods for flow labels. + (Jon Siwek) + +2.3-536 | 2015-03-12 16:16:24 -0500 + + * Fix Broker leak tests. (Jon Siwek) + +2.3-534 | 2015-03-12 10:59:49 -0500 + + * Update NEWS file. (Jon Siwek) + +2.3-533 | 2015-03-12 10:18:53 -0500 + + * Give broker python bindings default install path within --prefix. + (Jon Siwek) + +2.3-530 | 2015-03-10 13:22:39 -0500 + + * Fix broker data stores in absence of --enable-debug. (Jon Siwek) + +2.3-529 | 2015-03-09 13:14:27 -0500 + + * Fix format specifier in SSL protocol violation. (Jon Siwek) + +2.3-526 | 2015-03-06 12:48:49 -0600 + + * Fix build warnings, clarify broker requirements, update submodule. + (Jon Siwek) + + * Rename comm/ directories to broker/ (Jon Siwek) + + * Rename broker-related namespaces. (Jon Siwek) + + * Improve remote logging via broker by only sending fields w/ &log. + (Jon Siwek) + + * Disable a stream's remote logging via broker if it fails. (Jon Siwek) + + * Improve some broker communication unit tests. (Jon Siwek) + +2.3-518 | 2015-03-04 13:13:50 -0800 + + * Add bytes_recvd to stats.log recording the number of bytes + received, according to packet headers. (Mike Smiley) + +2.3-516 | 2015-03-04 12:30:06 -0800 + + * Extract most specific Common Name from SSL certificates (Johanna + Amann) + + * Send CN and SAN fields of SSL certificates to the Intel framework. + (Johanna Amann) + +2.3-511 | 2015-03-02 18:07:17 -0800 + + * Changes to plugin meta hooks for function calls. (Gilbert Clark) + + - Add frame argument. + + - Change return value to tuple unambigiously whether hook + returned a result. + +2.3-493 | 2015-03-02 17:17:32 -0800 + + * Extend the SSL weak-keys policy file to also alert when + encountering SSL connections with old versions as well as unsafe + cipher suites. (Johanna Amann) + + * Make the notice suppression handling of other SSL policy files a + tad more robust. (Johanna Amann) + +2.3-491 | 2015-03-02 17:12:56 -0800 + + * Updating docs for recent addition of local_resp. (Robin Sommer) + +2.3-489 | 2015-03-02 15:29:30 -0800 + + * Integrate Broker, Bro's new communication library. (Jon Siwek) + + See aux/broker/README for more information on Broker, and + doc/frameworks/comm.rst for the corresponding Bro script API. + + Broker support is by default off for now; it can be enabled at + configure time with --enable-broker. It requires CAF + (https://github.com/actor-framework/actor-framework); for now iot + needs CAF's "develop" branch. Broker also requires a C++11 + compiler. + + Broker will become a mandatory dependency in future Bro versions. + + * Add --enable-c++11 configure flag to compile Bro's source code in + C++11 mode with a corresponding compiler. (Jon Siwek) + +2.3-451 | 2015-02-24 16:37:08 -0800 + + * Updating submodule(s). + +2.3-448 | 2015-02-23 16:58:10 -0800 + + * Updating NEWS. (Robin Sommer) + +2.3-447 | 2015-02-23 16:28:30 -0800 + + * Fix potential crash in logging framework when deserializing + WriterInfo from remote. where config is present. Testcase crashes + on unpatched versions of Bro. (Aaron Eppert) + + * Fix wrong value test in WriterBackend. (Aaron Eppert) + +2.3-442 | 2015-02-23 13:29:30 -0800 + + * Add a "local_resp" field to conn.log, along the lines of the + existing "local_orig". (Mike Smiley) + +2.3-440 | 2015-02-23 11:39:17 -0600 + + * Updating plugin docs to recent changes. (Robin Sommer) + + * Updating plugin tests to recent changes. (Robin Sommer) + + * Making plugin names case-insensitive for some internal comparisions. + Makes plugin system more tolerant against spelling inconsistencies + are hard to catch otherwise. (Robin Sommer) + + * Explicitly removing some old scripts on install that have moved + into plugins to prevent them causing confusion. (Robin Sommer) + + * BIT-1312: Removing setting installation plugin path from + bro-path-dev.sh. Also, adding to existing BRO_PLUGIN_PATH rather + than replacing. (Robin Sommer) + + * Creating the installation directory for plugins at install time. + (Robin Sommer) + +2.3-427 | 2015-02-20 13:49:33 -0800 + + * Removing dependency on PCAP_NETMASK_UNKNOWN to compile with + libpcap < 1.1.1. (Robin Sommer) + +2.3-426 | 2015-02-20 12:45:51 -0800 + + * Add 'while' statement to Bro language. Really. (Jon Siwek) + +2.3-424 | 2015-02-20 12:39:10 -0800 + + * Add the ability to remove surrounding braces from the JSON + formatter. (Seth Hall) + +2.3-419 | 2015-02-13 09:10:44 -0600 + + * BIT-1011: Update the SOCKS analyzer to support user/pass login. + (Nicolas Retrain, Seth Hall, Jon Siwek) + + - Add a new field to socks.log: "password". + - Two new events: "socks_login_userpass_request" and + "socks_login_userpass_reply". + - Two new weirds for unsupported SOCKS authentication method or + version. + - A new test for authenticated socks traffic. + +2.3-416 | 2015-02-12 12:18:42 -0600 + + * Submodule update - newest sqlite version (Johanna Amann) + + * Fix use of deprecated gperftools headers. (Jon Siwek) + +2.3-413 | 2015-02-08 18:23:05 -0800 + + * Fixing analyzer tag types for some Files::* functions. (Robin Sommer) + + * Changing load order for plugin scripts. (Robin Sommer) + +2.3-411 | 2015-02-05 10:05:48 -0600 + + * Fix file analysis of files with total size below the bof_buffer size + never delivering content to stream analyzers. (Seth Hall) + + * Add/fix log fields in x509 diff canonifier. (Jon Siwek) + + * "id" not defined for debug code when using -DPROFILE_BRO_FUNCTIONS + (Mike Smiley) + +2.3-406 | 2015-02-03 17:02:45 -0600 + + * Add x509 canonifier to a unit test. (Jon Siwek) + +2.3-405 | 2015-02-02 11:14:24 -0600 + + * Fix memory leak in new split_string* functions. (Jon Siwek) + +2.3-404 | 2015-01-30 14:23:27 -0800 + + * Update documentation (broken links, outdated tests). (Jon Siwek) + + * Deprecate split* family of BIFs. (Jon Siwek) + + These functions are now deprecated in favor of alternative versions that + return a vector of strings rather than a table of strings. + + Deprecated functions: + + - split: use split_string instead. + - split1: use split_string1 instead. + - split_all: use split_string_all instead. + - split_n: use split_string_n instead. + - cat_string_array: see join_string_vec instead. + - cat_string_array_n: see join_string_vec instead. + - join_string_array: see join_string_vec instead. + - sort_string_array: use sort instead instead. + - find_ip_addresses: use extract_ip_addresses instead. + + Changed functions: + + - has_valid_octets: uses a string_vec parameter instead of string_array. + + Addresses BIT-924. + + * Add a new attribute: &deprecated. While scripts are parsed, a + warning is raised for each usage of an identifier marked as + &deprecated. This also works for BIFs. Addresses BIT-924, + BIT-757. (Jon Siwek) + +2.3-397 | 2015-01-27 10:13:10 -0600 + + * Handle guess_lexer exceptions in pygments reST directive (Jon Siwek) + +2.3-396 | 2015-01-23 10:49:15 -0600 + + * DNP3: fix reachable assertion and buffer over-read/overflow. + CVE number pending. (Travis Emmert, Jon Siwek) + + * Update binpac: Fix potential out-of-bounds memory reads in generated + code. CVE-2014-9586. (John Villamil and Chris Rohlf - Yahoo + Paranoids, Jon Siwek) + + * Fixing (harmless) Coverity warning. (Robin Sommer) + +2.3-392 | 2015-01-15 09:44:15 -0800 + + * Small changes to EC curve names in a newer draft. (Johanna Amann) + +2.3-390 | 2015-01-14 13:27:34 -0800 + + * Updating MySQL analyses. (Vlad Grigorescu) + - Use a boolean success instead of a result string. + - Change the affected_rows response detail string to a "rows" count. + - Fix the state tracking to log incomplete command. + + * Extend DNP3 to support communication over UDP. (Hui Lin) + + * Fix a bug in DNP3 determining the length of an object in some + cases. (Hui Lin) + +2.3-376 | 2015-01-12 09:38:10 -0600 + + * Improve documentation for connection_established event. (Jon Siwek) + +2.3-375 | 2015-01-08 13:10:09 -0600 + + * Increase minimum required CMake version to 2.8. (Jon Siwek) + +2.3-374 | 2015-01-07 10:03:17 -0600 + + * Improve documentation of the Intelligence Framework. (Daniel Thayer) + +2.3-371 | 2015-01-06 09:58:09 -0600 + + * Update/improve file mime type identification. (Seth Hall) + + - Change to the default BOF buffer size to 3000 (was 1024). + + - Reorganized MS signatures into a separate file. + + - Remove all of the x-c detections. Nearly all false positives. + + - Improve TAR detections, removing old, back up TAR detections. + + - Remove one of the x-elc detections that was too loose + and caused many false positives. + + - Improved lots of the signatures and added new ones. (Seth Hall) + + * Add support for file reassembly in the file analysis framework + (Seth Hall, Jon Siwek). + + - The reassembly behavior can be modified per-file by enabling or + disabling the reassembler and/or modifying the size of the + reassembly buffer. + + - Changed the file extraction analyzer to use stream-wise input to + avoid issues with the chunk-wise approach not immediately + triggering the file_new event due to mime-type detection delay. + Before, early chunks frequently ended up lost. Extraction also + will now explicitly NUL-fill gaps in the file instead of + implicitly relying on pwrite to do it. + +2.3-349 | 2015-01-05 15:21:13 -0600 + + * Fix race condition in unified2 file analyzer startup. (Jon siwek) + +2.3-348 | 2014-12-31 09:19:34 -0800 + + * Changing Makefile's test-all to run test-all for broctl, which now + executes trace-summary tests as well. (Robin Sommer) + +2.3-345 | 2014-12-31 09:06:15 -0800 + + * Correct a typo in the Notice framework doc. (Daniel Thayer) + +2.3-343 | 2014-12-12 12:43:46 -0800 + + * Fix PIA packet replay to deliver copy of IP header. This prevented + one from writing a packet-wise analyzer that needs access to IP + headers and can be attached to a connection via signature match. + Addresses BIT-1298 (Jon Siwek) + +2.3-338 | 2014-12-08 13:56:19 -0800 + + * Add man page for Bro. (Raúl Benencia) + + * Updating doc baselines. (Robin Sommer) + +2.3-334 | 2014-12-03 14:22:07 -0800 + + * Fix compound assignment to require proper L-value. Addresses + BIT-1295. (Jon Siwek) + +2.3-332 | 2014-12-03 14:14:11 -0800 + + * Make using local IDs in @if directives an error. Addresses + BIT-1296. (Jon Siwek) + +2.3-330 | 2014-12-03 14:10:39 -0800 + + * Fix some "make doc" warnings and update some doc tests. (Daniel + Thayer) + +2.3-328 | 2014-12-02 08:13:10 -0500 + + * Update windows-version-detection.bro to add support for + Windows 10. (Michal Purzynski) + +2.3-326 | 2014-12-01 12:10:27 -0600 + + * BIFScanner: fix invalid characters in generated preprocessor macros. + (Hilko Bengen) + + * BIT-1294: fix exec.bro from mutating Input::end_of_data event + parameters. (Johanna Amann) + + * Add/invoke "distclean" for testing directories. (Raúl Benencia) + + * Delete prebuilt python bytecode files from git. (Jon Siwek) + + * Add Windows detection based on CryptoAPI HTTP traffic as a software + framework policy script. (Vlad Grigorescu) + +2.3-316 | 2014-11-25 17:35:06 -0800 + + * Make the SSL analyzer skip further processing once encountering + situations which are very probably non-recoverable. (Johanna + Amann) + +2.3-313 | 2014-11-25 14:27:07 -0800 + + * Make SSL v2 protocol tests more strict. In its former state they + triggered on http traffic over port 443 sometimes. Found by Michał + Purzyński. (Johanna Amann) + + * Fix X509 analyzer to correctly return ECDSA as the key_type for + ECDSA certs. Bug found by Michał Purzyński. (Johanna Amann) + +2.3-310 | 2014-11-19 10:56:59 -0600 + + * Disable verbose bison output. (Jon Siwek) + +2.3-309 | 2014-11-18 12:17:53 -0800 + + * New decompose_uri() function in base/utils/urls that splits a URI + into its pieces. (Anthony Kasza). + +2.3-305 | 2014-11-18 11:09:04 -0800 + + * Improve coercion of &default expressions. Addresses BIT-1288. (Jon + Siwek) + +2.3-303 | 2014-11-18 10:53:04 -0800 + + * For DH key exchanges, use p as the parameter for weak key + exchanges. (Johanna Amann) + +2.3-301 | 2014-11-11 13:47:27 -0800 + + * Add builtin function enum_to_int() that converts an enum into a + integer. (Christian Struck) + +2.3-297 | 2014-11-11 11:50:47 -0800 + + * Removing method from SSL analyzer that's no longer used. (Robin + Sommer) + +2.3-296 | 2014-11-11 11:42:38 -0800 + + * A new analyzer parsing the MySQL wire protocol. Activity gets + logged into mysql.log. Supports protocol versions 9 and 10. (Vlad + Grigorescu) + +2.3-280 | 2014-11-05 09:46:33 -0500 + + * Add Windows detection based on CryptoAPI HTTP traffic as a + software framework policy script. (Vlad Grigorescu) + +2.3-278 | 2014-11-03 18:55:18 -0800 + + * Add new curves from draft-ietf-tls-negotiated-ff-dhe to SSL + analysis. (Johanna Amann) + +2.3-274 | 2014-10-31 17:45:25 -0700 + + * Adding call to new binpac::init() function. (Robin Sommer) + +2.3-272 | 2014-10-31 16:29:42 -0700 + + * Fix segfault if when statement's RHS is unitialized. Addresses + BIT-1176. (Jon Siwek) + + * Fix checking vector indices via "in". Addresses BIT-1280. (Jon + Siwek) + +2.3-268 | 2014-10-31 12:12:22 -0500 + + * BIT-1283: Fix crash when using &encrypt. (Jon Siwek) + +2.3-267 | 2014-10-31 10:35:02 -0500 + + * BIT-1284: Allow arbitrary when statement timeout expressions + (Jon Siwek) + +2.3-266 | 2014-10-31 09:21:28 -0500 + + * BIT-1166: Add configure options to fine tune local state dirs used + by BroControl. (Jon Siwek) + +2.3-264 | 2014-10-30 13:25:57 -0500 + + * Fix some minor Coverity Scan complaints. (Jon Siwek) + +2.3-263 | 2014-10-28 15:09:10 -0500 + + * Fix checking of fwrite return values (Johanna Amann) + +2.3-260 | 2014-10-27 12:54:17 -0500 + + * Fix errors/warnings when compiling with -std=c++11 (Jon Siwek) + +2.3-259 | 2014-10-27 10:04:04 -0500 + + * Documentation fixes. (Vicente Jimenez Aguilar and Stefano Azzalini) + +2.3-256 | 2014-10-24 15:33:45 -0700 + + * Adding missing test baseline. (Robin Sommer) + +2.3-255 | 2014-10-24 13:39:44 -0700 + + * Fixing unstable active-http test. (Robin Sommer) + +2.3-254 | 2014-10-24 11:40:51 -0700 + + * Fix active-http.bro to deal reliably with empty server responses, + which will now be passed back as empty files. (Christian Struck) + +2.3-248 | 2014-10-23 14:20:59 -0700 + + * Change order in which a plugin's scripts are loaded at startup. + (Robin Sommer) + +2.3-247 | 2014-10-21 13:42:38 -0700 + + * Updates to the SSL analyzer. (Johanna Amann) + + * Mark everything below 2048 bit as a weak key. + + * Fix notice suppression. + + * Add information about server-chosen protocol to ssl.log, if + provided by application_layer_next_protocol. + + * Add boolean flag to ssl.log signaling if a session was + resumed. Remove the (usually not really that useful) session + ID that the client sent. + +2.3-240 | 2014-10-21 13:36:33 -0700 + + * Fix Coverity-reported issues in DNP3 analyzer. (Seth Hall) + +2.3-238 | 2014-10-16 06:51:49 -0700 + + * Fix multipart HTTP/MIME entity file analysis so that (1) singular + CR or LF characters in multipart body content are no longer + converted to a full CRLF (thus corrupting the file) and (2) it + also no longer considers the CRLF before the multipart boundary as + part of the content. Addresses BIT-1235. (Jon Siwek) + +2.3-235 | 2014-10-15 10:20:47 -0500 + + * BIT-1273: Add error message for bad enum declaration syntax. + (Jon Siwek) + +2.3-234 | 2014-10-14 14:42:09 -0500 + + * Documentation fixes. (Steve Smoot) + +2.3-233 | 2014-10-09 16:00:27 -0500 + + * Change find-bro-logs unit test to follow symlinks. (Jon Siwek) + + * Add error checks and messages to a test script (Daniel Thayer) + +2.3-230 | 2014-10-08 08:15:17 -0700 + + * Further baseline normalization for plugin test portability. (Robin + Sommer) + +2.3-229 | 2014-10-07 20:18:11 -0700 + + * Fix for test portability. (Robin Sommer) + +2.3-228 | 2014-10-07 15:32:37 -0700 + + * Include plugin unit tests into the top-level btest configuration. (Robin Sommer) + + * Switching the prefix separator for packet source/dumper plugins + once more, now to "::". Addresses BIT-1267. (Robin Sommer) + + * Fix for allowing a packet source/dumper plugin to support multiple + prefixes with a colon. (Robin Sommer) + +2.3-225 | 2014-10-07 15:13:35 -0700 + + * Updating plugin documentation. (Robin Sommer) + +2.3-224 | 2014-10-07 14:32:17 -0700 + + * Improved the log file reference documentation. (Jeannette Dopheide + and Daniel Thayer) + + * Improves shockwave flash file signatures. (Seth Hall) + + - This moves the signatures out of the libmagic imported signatures + and into our own general.sig. + + - Expand the detection to LZMA compressed flash files. + + * Add new script language reference documentation on operators, + statements, and directives. Also improved the documentation on + types and attributes by splitting them into two docs, and + providing more examples and adding a chart on the top of each page + with links to each type and attribute for easier access to the + information. (Daniel Thayer) + + * Split the types and attributes reference doc into two docs. + (Daniel Thayer) + +2.3-208 | 2014-10-03 09:38:52 -0500 + + * BIT-1268: Fix uninitialized router_list argument in + dhcp_offer/dhcp_ack. (Jon Siwek) + +2.3-207 | 2014-10-02 16:39:17 -0700 + + * Updating plugin docs. (Robin Sommer) + + * Fix packet sources being treated as idle when a packet is + available. Addresses BIT-1266. (Jon Siwek) + + * Fix regression causing the main loop to spin more frequently. + Addresses BIT-1266. (Jon Siwek) + +2.3-203 | 2014-09-29 20:06:54 -0700 + + * Fix to use length parameter in DNP3 time conversion correctly now. + (Robin Sommer) + +2.3-202 | 2014-09-29 17:05:18 -0700 + + * New SSL extension type from IANA and a few other SSL const + changes. (Johanna Amann) + + * Make unexpected pipe errors fatal as precaution. Addresses + BIT-1260. (Jon Siwek) + + * Adding a function for DNP3 to translate the timestamp format. (Hui + Lin) + +2.3-197 | 2014-09-29 10:42:01 -0500 + + * Fix possible seg fault in TCP reassembler. (Jon Siwek) + +2.3-196 | 2014-09-25 17:53:27 -0700 + + * Changing prefix for packet sources/dumper from ':' to '%'. + Addresses BIT-1249. (Robin Sommer) + + * Remove timeouts from remote communication loop. The select() now + blocks until there's work to do instead of relying on a small + timeout value which can cause unproductive use of cpu cycles. (Jon + Siwek) + + * Improve error message when failing to activate a plugin. Also fix + a unit test helper script that checks plugin availability. (Jon + Siwek) + 2.3-183 | 2014-09-24 10:08:04 -0500 * Add a "node" field to Intel::Seen struture and intel.log to diff --git a/CMakeLists.txt b/CMakeLists.txt index 22d63a89d5..b96923aa56 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -2,7 +2,7 @@ project(Bro C CXX) # When changing the minimum version here, also adapt # aux/bro-aux/plugin-support/skeleton/CMakeLists.txt -cmake_minimum_required(VERSION 2.6.3 FATAL_ERROR) +cmake_minimum_required(VERSION 2.8 FATAL_ERROR) include(cmake/CommonCMakeConfig.cmake) @@ -15,6 +15,11 @@ if (NOT BRO_SCRIPT_INSTALL_PATH) set(BRO_SCRIPT_INSTALL_PATH ${BRO_ROOT_DIR}/share/bro) endif () +if (NOT BRO_MAN_INSTALL_PATH) + # set the default Bro man page installation path (user did not specify one) + set(BRO_MAN_INSTALL_PATH ${BRO_ROOT_DIR}/share/man) +endif () + # sanitize the Bro script install directory into an absolute path # (CMake is confused by ~ as a representation of home directory) get_filename_component(BRO_SCRIPT_INSTALL_PATH ${BRO_SCRIPT_INSTALL_PATH} @@ -26,12 +31,12 @@ configure_file(bro-path-dev.in ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev) file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.sh "export BROPATH=`${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" - "export BRO_PLUGIN_PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src:${BRO_PLUGIN_INSTALL_PATH}\"\n" + "export BRO_PLUGIN_PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src\":${BRO_PLUGIN_PATH}\n" "export PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n") file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.csh "setenv BROPATH `${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" - "setenv BRO_PLUGIN_PATH \"${CMAKE_CURRENT_BINARY_DIR}/src:${BRO_PLUGIN_INSTALL_PATH}\"\n" + "setenv BRO_PLUGIN_PATH \"${CMAKE_CURRENT_BINARY_DIR}/src\":${BRO_PLUGIN_PATH}\n" "setenv PATH \"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n") file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/VERSION" VERSION LIMIT_COUNT 1) @@ -56,7 +61,7 @@ if (NOT SED_EXE) endif () endif () -FindRequiredPackage(Perl) +FindRequiredPackage(PythonInterp) FindRequiredPackage(FLEX) FindRequiredPackage(BISON) FindRequiredPackage(PCAP) @@ -108,7 +113,7 @@ if (NOT DISABLE_PERFTOOLS) find_package(GooglePerftools) endif () -if (GOOGLEPERFTOOLS_FOUND) +if (GOOGLEPERFTOOLS_FOUND OR TCMALLOC_FOUND) set(HAVE_PERFTOOLS true) # Non-Linux systems may not be well-supported by gperftools, so # require explicit request from user to enable it in that case. @@ -160,21 +165,30 @@ include(PCAPTests) include(OpenSSLTests) include(CheckNameserCompat) include(GetArchitecture) +include(RequireCXX11) # Tell the plugin code that we're building as part of the main tree. set(BRO_PLUGIN_INTERNAL_BUILD true CACHE INTERNAL "" FORCE) -configure_file(${CMAKE_CURRENT_SOURCE_DIR}/config.h.in - ${CMAKE_CURRENT_BINARY_DIR}/config.h) +configure_file(${CMAKE_CURRENT_SOURCE_DIR}/bro-config.h.in + ${CMAKE_CURRENT_BINARY_DIR}/bro-config.h) include_directories(${CMAKE_CURRENT_BINARY_DIR}) ######################################################################## ## Recurse on sub-directories +if ( ENABLE_BROKER ) + add_subdirectory(aux/broker) + set(brodeps ${brodeps} broker) + add_definitions(-DENABLE_BROKER) + include_directories(BEFORE ${CMAKE_CURRENT_SOURCE_DIR}/aux/broker) +endif () + add_subdirectory(src) add_subdirectory(scripts) add_subdirectory(doc) +add_subdirectory(man) include(CheckOptionalBuildSources) @@ -218,6 +232,8 @@ message( "\nCXXFLAGS: ${CMAKE_CXX_FLAGS} ${CMAKE_CXX_FLAGS_${BuildType}}" "\nCPP: ${CMAKE_CXX_COMPILER}" "\n" + "\nBroker: ${ENABLE_BROKER}" + "\nBroker Python: ${BROKER_PYTHON_BINDINGS}" "\nBroccoli: ${INSTALL_BROCCOLI}" "\nBroctl: ${INSTALL_BROCTL}" "\nAux. Tools: ${INSTALL_AUX_TOOLS}" diff --git a/COPYING b/COPYING index 2c66f98113..5454660df2 100644 --- a/COPYING +++ b/COPYING @@ -1,4 +1,4 @@ -Copyright (c) 1995-2013, The Regents of the University of California +Copyright (c) 1995-2015, The Regents of the University of California through the Lawrence Berkeley National Laboratory and the International Computer Science Institute. All rights reserved. diff --git a/INSTALL b/INSTALL deleted file mode 100644 index 385dac93df..0000000000 --- a/INSTALL +++ /dev/null @@ -1,3 +0,0 @@ - -See doc/install/install.rst for installation instructions. - diff --git a/INSTALL b/INSTALL new file mode 120000 index 0000000000..95fcc60eda --- /dev/null +++ b/INSTALL @@ -0,0 +1 @@ +doc/install/install.rst \ No newline at end of file diff --git a/Makefile b/Makefile index 49d9a6173c..3efddc4dbc 100644 --- a/Makefile +++ b/Makefile @@ -48,15 +48,18 @@ bindist: distclean: rm -rf $(BUILD) + $(MAKE) -C testing $@ test: - @( cd testing && make ) + -@( cd testing && make ) -test-all: test - test -d aux/broctl && ( cd aux/broctl && make test ) - test -d aux/btest && ( cd aux/btest && make test ) - test -d aux/bro-aux && ( cd aux/bro-aux && make test ) - test -d aux/plugins && ( cd aux/plugins && make test-all ) +test-aux: + -test -d aux/broctl && ( cd aux/broctl && make test-all ) + -test -d aux/btest && ( cd aux/btest && make test ) + -test -d aux/bro-aux && ( cd aux/bro-aux && make test ) + -test -d aux/plugins && ( cd aux/plugins && make test-all ) + +test-all: test test-aux configured: @test -d $(BUILD) || ( echo "Error: No build/ directory found. Did you run configure?" && exit 1 ) diff --git a/NEWS b/NEWS index 6da13833c3..5348826e04 100644 --- a/NEWS +++ b/NEWS @@ -4,11 +4,70 @@ release. For an exhaustive list of changes, see the ``CHANGES`` file (note that submodules, such as BroControl and Broccoli, come with their own ``CHANGES``.) -Bro 2.4 (in progress) +Bro 2.5 (in progress) ===================== -Dependencies ------------- +New Dependencies +---------------- + +- Bro now requires a compiler with C++11 support for building the + source code. + +- Bro now requires the C++ Actor Framework, CAF, which must be + installed first. See http://actor-framework.org. + +- Bro now requires Python instead of Perl to compile the source code. + +- The pcap buffer size can set through the new option Pcap::bufsize. + +New Functionality +----------------- + +- Bro now tracks VLAN IDs. To record them inside the connection log, + load protocols/conn/vlan-logging.bro. + +- A new per-packet event raw_packet() provides access to layer 2 + information. Use with care, generating events per packet is + expensive. + +- A new built-in function, decode_base64_conn() for Base64 decoding. + It works like decode_base64() but receives an additional connection + argument that will be used for decoding errors into weird.log + (instead of reporter.log). + +- The IRC analyzer now recognizes StartTLS sessions and enable the SSL + analyzer for them. + +- New Bro plugins in aux/plugins: + + - af_packet: Native AF_PACKET support. + - myricom: Native Myricom SNF v3 support. + - pf_ring: Native PF_RING support. + - redis: An experimental log writer for Redis. + - tcprs: An TCP-level analyzer detecting retransmissions, reordering, and more. + +Changed Functionality +--------------------- + +- Some script-level identifier have changed their names: + + snaplen -> Pcap::snaplen + precompile_pcap_filter() -> Pcap::precompile_pcap_filter() + install_pcap_filter() -> Pcap::install_pcap_filter() + pcap_error() -> Pcap::pcap_error() + + +Deprecated Functionality +------------------------ + + - The built-in functions decode_base64_custom() and + encode_base64_custom() are no longer needed and will be removed + in the future. Their functionality is now provided directly by + decode_base64() and encode_base64(), which take an optional + parameter to change the Base64 alphabet. + +Bro 2.4 +======= New Functionality ----------------- @@ -16,20 +75,257 @@ New Functionality - Bro now has support for external plugins that can extend its core functionality, like protocol/file analysis, via shared libraries. Plugins can be developed and distributed externally, and will be - pulled in dynamically at startup. Currently, a plugin can provide - custom protocol analyzers, file analyzers, log writers[TODO], input - readers[TODO], packet sources[TODO], and new built-in functions. A - plugin can furthermore hook into Bro's processing a number of places - to add custom logic. + pulled in dynamically at startup (the environment variables + BRO_PLUGIN_PATH and BRO_PLUGIN_ACTIVATE can be used to specify the + locations and names of plugins to activate). Currently, a plugin + can provide custom protocol analyzers, file analyzers, log writers, + input readers, packet sources and dumpers, and new built-in functions. + A plugin can furthermore hook into Bro's processing at a number of + places to add custom logic. See https://www.bro.org/sphinx-git/devel/plugins.html for more information on writing plugins. +- Bro now has support for the MySQL wire protocol. Activity gets + logged into mysql.log. + +- Bro now parses DTLS traffic. Activity gets logged into ssl.log. + +- Bro now has support for the Kerberos KRB5 protocol over TCP and + UDP. Activity gets logged into kerberos.log. + +- Bro now has an RDP analyzer. Activity gets logged into rdp.log. + +- Bro now has a file analyzer for Portable Executables. Activity gets + logged into pe.log. + +- Bro now has support for the SIP protocol over UDP. Activity gets + logged into sip.log. + +- Bro now features a completely rewritten, enhanced SSH analyzer. The + new analyzer is able to determine if logins failed or succeeded in + most circumstances, logs a lot more more information about SSH + sessions, supports v1, and introduces the intelligence type + ``Intel::PUBKEY_HASH`` and location ``SSH::IN_SERVER_HOST_KEY``. The + analayzer also generates a set of additional events + (``ssh_auth_successful``, ``ssh_auth_failed``, ``ssh_capabilities``, + ``ssh2_server_host_key``, ``ssh1_server_host_key``, + ``ssh_encrypted_packet``, ``ssh2_dh_server_params``, + ``ssh2_gss_error``, ``ssh2_ecc_key``). See next section for + incompatible SSH changes. + +- Bro's file analysis now supports reassembly of files that are not + transferred/seen sequentially. The default file reassembly buffer + size is set with the ``Files::reassembly_buffer_size`` variable. + +- Bro's file type identification has been greatly improved (new file types, + bug fixes, and performance improvements). + +- Bro's scripting language now has a ``while`` statement:: + + while ( i < 5 ) + print ++i; + + ``next`` and ``break`` can be used inside the loop's body just like + with ``for`` loops. + +- Bro now integrates Broker, a new communication library. See + aux/broker/README for more information on Broker, and + doc/frameworks/broker.rst for the corresponding Bro script API. + + With Broker, Bro has the similar capabilities of exchanging events and + logs with remote peers (either another Bro process or some other + application that uses Broker). It also includes a key-value store + API that can be used to share state between peers and optionally + allow data to persist on disk for longer-term storage. + + Broker support is by default off for now; it can be enabled at + configure time with --enable-broker. It requires CAF version 0.13+ + (https://github.com/actor-framework/actor-framework) as well as a + C++11 compiler (e.g. GCC 4.8+ or Clang 3.3+). + + Broker will become a mandatory dependency in future Bro versions and + replace the current communication and serialization system. + +- Add --enable-c++11 configure flag to compile Bro's source code in + C++11 mode with a corresponding compiler. Note that 2.4 will be the + last version of Bro that compiles without C++11 support. + +- The SSL analysis now alerts when encountering SSL connections with + old protocol versions or unsafe cipher suites. It also gained + extended reporting of weak keys, caching of already validated + certificates, and full support for TLS record defragmentation. SSL generally + became much more robust and added several fields to ssl.log (while + removing some others). + +- A new icmp_sent_payload event provides access to ICMP payload. + +- The input framework's raw reader now supports seeking by adding an + option "offset" to the config map. Positive offsets are interpreted + to be from the beginning of the file, negative from the end of the + file (-1 is end of file). + +- One can now raise events when a connection crosses a given size + threshold in terms of packets or bytes. The primary API for that + functionality is in base/protocols/conn/thresholds.bro. + +- There is a new command-line option -Q/--time that prints Bro's execution + time and memory usage to stderr. + +- BroControl now has a new command "deploy" which is equivalent to running + the "check", "install", "stop", and "start" commands (in that order). + +- BroControl now has a new option "StatusCmdShowAll" that controls whether + or not the broctl "status" command gathers all of the status information. + This option can be used to make the "status" command run significantly + faster (in this case, the "Peers" column will not be shown in the output). + +- BroControl now has a new option "StatsLogEnable" that controls whether + or not broctl will record information to the "stats.log" file. This option + can be used to make the "broctl cron" command run slightly faster (in this + case, "broctl cron" will also no longer send email about not seeing any + packets on the monitoring interfaces). + +- BroControl now has a new option "MailHostUpDown" which controls whether or + not the "broctl cron" command will send email when it notices that a host + in the cluster is up or down. + +- BroControl now has a new option "CommandTimeout" which specifies the number + of seconds to wait for a command that broctl ran to return results. + Changed Functionality --------------------- - bro-cut has been rewritten in C, and is hence much faster. +- File analysis + + * Removed ``fa_file`` record's ``mime_type`` and ``mime_types`` + fields. The event ``file_sniff`` has been added which provides + the same information. The ``mime_type`` field of ``Files::Info`` + also still has this info. + + * The earliest point that new mime type information is available is + in the ``file_sniff`` event which comes after the ``file_new`` and + ``file_over_new_connection`` events. Scripts which inspected mime + type info within those events will need to be adapted. (Note: for + users that worked w/ versions of Bro from git, for a while there was + also an event called ``file_mime_type`` which is now replaced with + the ``file_sniff`` event). + + * Removed ``Files::add_analyzers_for_mime_type`` function. + + * Removed ``offset`` parameter of the ``file_extraction_limit`` + event. Since file extraction now internally depends on file + reassembly for non-sequential files, "offset" can be obtained + with other information already available -- adding together + ``seen_bytes`` and ``missed_bytes`` fields of the ``fa_file`` + record gives how many bytes have been written so far (i.e. + the "offset"). + +- The SSH changes come with a few incompatibilities. The following + events have been renamed: + + * ``SSH::heuristic_failed_login`` to ``ssh_auth_failed`` + * ``SSH::heuristic_successful_login`` to ``ssh_auth_successful`` + + The ``SSH::Info`` status field has been removed and replaced with + the ``auth_success`` field. This field has been changed from a + string that was previously ``success``, ``failure`` or + ``undetermined`` to a boolean. a boolean that is ``T``, ``F``, or + unset. + +- The has_valid_octets function now uses a string_vec parameter instead of + string_array. + +- conn.log gained a new field local_resp that works like local_orig, + just for the responder address of the connection. + +- GRE tunnels are now identified as ``Tunnel::GRE`` instead of + ``Tunnel::IP``. + +- The default name for extracted files changed from extract-protocol-id + to extract-timestamp-protocol-id. + +- The weird named "unmatched_HTTP_reply" has been removed since it can + be detected at the script-layer and is handled correctly by the + default HTTP scripts. + +- When adding a logging filter to a stream, the filter can now inherit + a default ``path`` field from the associated ``Log::Stream`` record. + +- When adding a logging filter to a stream, the + ``Log::default_path_func`` is now only automatically added to the + filter if it has neither a ``path`` nor a ``path_func`` already + explicitly set. Before, the default path function would always be set + for all filters which didn't specify their own ``path_func``. + +- BroControl now establishes only one ssh connection from the manager to + each remote host in a cluster configuration (previously, there would be + one ssh connection per remote Bro process). + +- BroControl now uses SQLite to record state information instead of a + plain text file (the file "spool/broctl.dat" is no longer used). + On FreeBSD, this means that there is a new dependency on the package + "py27-sqlite3". + +- BroControl now records the expected running state of each Bro node right + before each start or stop. The "broctl cron" command uses this info to + either start or stop Bro nodes as needed so that the actual state matches + the expected state (previously, "broctl cron" could only start nodes in + the "crashed" state, and could never stop a node). + +- BroControl now sends all normal command output (i.e., not error messages) + to stdout. Error messages are still sent to stderr, however. + +- The capability of processing NetFlow input has been removed for the + time being. Therefore, the -y/--flowfile and -Y/--netflow command-line + options have been removed, and the netflow_v5_header and netflow_v5_record + events have been removed. + +- The -D/--dfa-size command-line option has been removed. + +- The -L/--rule-benchmark command-line option has been removed. + +- The -O/--optimize command-line option has been removed. + +- The deprecated fields "hot" and "addl" have been removed from the + connection record. Likewise, the functions append_addl() and + append_addl_marker() have been removed. + +- Log files now escape non-printable characters consistently as "\xXX'. + Furthermore, backslashes are escaped as "\\", making the + representation fully reversible. + +Deprecated Functionality +------------------------ + +- The split* family of functions are to be replaced with alternate + versions that return a vector of strings rather than a table of + strings. This also allows deprecation for some related string + concatenation/extraction functions. Note that the new functions use + 0-based indexing, rather than 1-based. + + The full list of now deprecated functions is: + + * split: use split_string instead. + + * split1: use split_string1 instead. + + * split_all: use split_string_all instead. + + * split_n: use split_string_n instead. + + * cat_string_array: see join_string_vec instead. + + * cat_string_array_n: see join_string_vec instead. + + * join_string_array: see join_string_vec instead. + + * sort_string_array: use sort instead. + + * find_ip_addresses: use extract_ip_addresses instead. + Bro 2.3 ======= diff --git a/README.rst b/README.rst new file mode 120000 index 0000000000..100b93820a --- /dev/null +++ b/README.rst @@ -0,0 +1 @@ +README \ No newline at end of file diff --git a/VERSION b/VERSION index 5e605d2618..dde094552f 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.3-183 +2.4-228 diff --git a/aux/binpac b/aux/binpac index 3a4684801a..214294c502 160000 --- a/aux/binpac +++ b/aux/binpac @@ -1 +1 @@ -Subproject commit 3a4684801aafa0558383199e9abd711650b53af9 +Subproject commit 214294c502d377bb7bf511eac8c43608e54c875a diff --git a/aux/bro-aux b/aux/bro-aux index 9ea20c3905..4e0d2bff4b 160000 --- a/aux/bro-aux +++ b/aux/bro-aux @@ -1 +1 @@ -Subproject commit 9ea20c3905bd3fd5109849c474a2f2b4ed008357 +Subproject commit 4e0d2bff4b2c287f66186c3654ef784bb0748d11 diff --git a/aux/broccoli b/aux/broccoli index 33d0ed4a54..959cc0a818 160000 --- a/aux/broccoli +++ b/aux/broccoli @@ -1 +1 @@ -Subproject commit 33d0ed4a54a6ecf08a0b5fe18831aa413b437066 +Subproject commit 959cc0a8181e7f4b07559a6aecca2a0d7d3d445c diff --git a/aux/broctl b/aux/broctl index 2f808bc854..1d0ca47534 160000 --- a/aux/broctl +++ b/aux/broctl @@ -1 +1 @@ -Subproject commit 2f808bc8541378b1a4953cca02c58c43945d154f +Subproject commit 1d0ca4753471cf822f612dc0d0e9bf9a439a994b diff --git a/aux/broker b/aux/broker new file mode 160000 index 0000000000..9a2e8ec7b3 --- /dev/null +++ b/aux/broker @@ -0,0 +1 @@ +Subproject commit 9a2e8ec7b365bde282edc7301c7936eed6b4fbbb diff --git a/aux/btest b/aux/btest index 1efa4d10f9..71a1e3efc4 160000 --- a/aux/btest +++ b/aux/btest @@ -1 +1 @@ -Subproject commit 1efa4d10f943351efea96def68e598b053fd217a +Subproject commit 71a1e3efc437aa9f981be71affa1c4615e8d98a5 diff --git a/aux/plugins b/aux/plugins index 23055b473c..35007df097 160000 --- a/aux/plugins +++ b/aux/plugins @@ -1 +1 @@ -Subproject commit 23055b473c689a79da12b2825d8388f71f28c709 +Subproject commit 35007df0974b566f75d7c82af5b4d5a022333d87 diff --git a/config.h.in b/bro-config.h.in similarity index 100% rename from config.h.in rename to bro-config.h.in diff --git a/cmake b/cmake index 03de0cc467..843cdf6a91 160000 --- a/cmake +++ b/cmake @@ -1 +1 @@ -Subproject commit 03de0cc467d2334dcb851eddd843d59fef217909 +Subproject commit 843cdf6a91f06e5407bffbc79a343bff3cf4c81f diff --git a/configure b/configure index 5747586db8..f94085f9d3 100755 --- a/configure +++ b/configure @@ -24,6 +24,13 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --prefix=PREFIX installation directory [/usr/local/bro] --scriptdir=PATH root installation directory for Bro scripts [PREFIX/share/bro] + --localstatedir=PATH when using BroControl, path to store log files + and run-time data (within log/ and spool/ subdirs) + [PREFIX] + --spooldir=PATH when using BroControl, path to store run-time data + [PREFIX/spool] + --logdir=PATH when using BroControl, path to store log file + [PREFIX/logs] --conf-files-dir=PATH config files installation directory [PREFIX/etc] Optional Features: @@ -34,11 +41,13 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --enable-perftools-debug use Google's perftools for debugging --enable-jemalloc link against jemalloc --enable-ruby build ruby bindings for broccoli (deprecated) + --disable-broker disable use of the Broker communication library --disable-broccoli don't build or install the Broccoli library --disable-broctl don't install Broctl --disable-auxtools don't build or install auxiliary tools --disable-perftools don't try to build with Google Perftools --disable-python don't try to build python bindings for broccoli + --disable-pybroker don't try to build python bindings for broker Required Packages in Non-Standard Locations: --with-openssl=PATH path to OpenSSL install root @@ -47,19 +56,22 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --with-binpac=PATH path to BinPAC install root --with-flex=PATH path to flex executable --with-bison=PATH path to bison executable - --with-perl=PATH path to perl executable + --with-python=PATH path to Python executable + --with-libcaf=PATH path to C++ Actor Framework installation + (a required Broker dependency) Optional Packages in Non-Standard Locations: --with-geoip=PATH path to the libGeoIP install root --with-perftools=PATH path to Google Perftools install root --with-jemalloc=PATH path to jemalloc install root - --with-python=PATH path to Python interpreter --with-python-lib=PATH path to libpython --with-python-inc=PATH path to Python headers --with-ruby=PATH path to ruby interpreter --with-ruby-lib=PATH path to ruby library --with-ruby-inc=PATH path to ruby headers --with-swig=PATH path to SWIG executable + --with-rocksdb=PATH path to RocksDB installation + (an optional Broker dependency) Packaging Options (for developers): --binary-package toggle special logic for binary packaging @@ -81,7 +93,7 @@ Usage: $0 [OPTION]... [VAR=VALUE]... sourcedir="$( cd "$( dirname "$0" )" && pwd )" # Function to append a CMake cache entry definition to the -# CMakeCacheEntries variable +# CMakeCacheEntries variable. # $1 is the cache entry variable name # $2 is the cache entry variable type # $3 is the cache entry variable value @@ -89,6 +101,17 @@ append_cache_entry () { CMakeCacheEntries="$CMakeCacheEntries -D $1:$2=$3" } +# Function to remove a CMake cache entry definition from the +# CMakeCacheEntries variable +# $1 is the cache entry variable name +remove_cache_entry () { + CMakeCacheEntries="$CMakeCacheEntries -U $1" + + # Even with -U, cmake still warns by default if + # added previously with -D. + CMakeCacheEntries="$CMakeCacheEntries --no-warn-unused-cli" +} + # set defaults builddir=build prefix=/usr/local/bro @@ -98,10 +121,13 @@ append_cache_entry BRO_ROOT_DIR PATH $prefix append_cache_entry PY_MOD_INSTALL_DIR PATH $prefix/lib/broctl append_cache_entry BRO_SCRIPT_INSTALL_PATH STRING $prefix/share/bro append_cache_entry BRO_ETC_INSTALL_DIR PATH $prefix/etc +append_cache_entry BROKER_PYTHON_HOME PATH $prefix +append_cache_entry BROKER_PYTHON_BINDINGS BOOL false append_cache_entry ENABLE_DEBUG BOOL false append_cache_entry ENABLE_PERFTOOLS BOOL false append_cache_entry ENABLE_PERFTOOLS_DEBUG BOOL false -append_cache_entry ENABLE_JEMALLOC BOOL false +append_cache_entry ENABLE_JEMALLOC BOOL false +append_cache_entry ENABLE_BROKER BOOL true append_cache_entry BinPAC_SKIP_INSTALL BOOL true append_cache_entry BUILD_SHARED_LIBS BOOL true append_cache_entry INSTALL_AUX_TOOLS BOOL true @@ -135,6 +161,10 @@ while [ $# -ne 0 ]; do append_cache_entry CMAKE_INSTALL_PREFIX PATH $optarg append_cache_entry BRO_ROOT_DIR PATH $optarg append_cache_entry PY_MOD_INSTALL_DIR PATH $optarg/lib/broctl + + if [ -z "$user_disabled_broker" ]; then + append_cache_entry BROKER_PYTHON_HOME PATH $optarg + fi ;; --scriptdir=*) append_cache_entry BRO_SCRIPT_INSTALL_PATH STRING $optarg @@ -144,6 +174,15 @@ while [ $# -ne 0 ]; do append_cache_entry BRO_ETC_INSTALL_DIR PATH $optarg user_set_conffilesdir="true" ;; + --localstatedir=*) + append_cache_entry BRO_LOCAL_STATE_DIR PATH $optarg + ;; + --spooldir=*) + append_cache_entry BRO_SPOOL_DIR PATH $optarg + ;; + --logdir=*) + append_cache_entry BRO_LOG_DIR PATH $optarg + ;; --enable-debug) append_cache_entry ENABLE_DEBUG BOOL true ;; @@ -160,6 +199,11 @@ while [ $# -ne 0 ]; do --enable-jemalloc) append_cache_entry ENABLE_JEMALLOC BOOL true ;; + --disable-broker) + append_cache_entry ENABLE_BROKER BOOL false + remove_cache_entry BROKER_PYTHON_HOME + user_disabled_broker="true" + ;; --disable-broccoli) append_cache_entry INSTALL_BROCCOLI BOOL false ;; @@ -175,6 +219,9 @@ while [ $# -ne 0 ]; do --disable-python) append_cache_entry DISABLE_PYTHON_BINDINGS BOOL true ;; + --disable-pybroker) + append_cache_entry DISABLE_PYBROKER BOOL true + ;; --enable-ruby) append_cache_entry DISABLE_RUBY_BINDINGS BOOL false ;; @@ -196,9 +243,6 @@ while [ $# -ne 0 ]; do --with-bison=*) append_cache_entry BISON_EXECUTABLE PATH $optarg ;; - --with-perl=*) - append_cache_entry PERL_EXECUTABLE PATH $optarg - ;; --with-geoip=*) append_cache_entry LibGeoIP_ROOT_DIR PATH $optarg ;; @@ -232,6 +276,12 @@ while [ $# -ne 0 ]; do --with-swig=*) append_cache_entry SWIG_EXECUTABLE PATH $optarg ;; + --with-libcaf=*) + append_cache_entry LIBCAF_ROOT_DIR PATH $optarg + ;; + --with-rocksdb=*) + append_cache_entry ROCKSDB_ROOT_DIR PATH $optarg + ;; --binary-package) append_cache_entry BINARY_PACKAGING_MODE BOOL true ;; diff --git a/doc/components/bro-plugins/README.rst b/doc/components/bro-plugins/README.rst new file mode 120000 index 0000000000..8f96f50909 --- /dev/null +++ b/doc/components/bro-plugins/README.rst @@ -0,0 +1 @@ +../../../aux/plugins/README \ No newline at end of file diff --git a/doc/components/bro-plugins/dataseries/README.rst b/doc/components/bro-plugins/dataseries/README.rst new file mode 120000 index 0000000000..3362e911fc --- /dev/null +++ b/doc/components/bro-plugins/dataseries/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/dataseries/README \ No newline at end of file diff --git a/doc/components/bro-plugins/elasticsearch/README.rst b/doc/components/bro-plugins/elasticsearch/README.rst new file mode 120000 index 0000000000..8a5b78d689 --- /dev/null +++ b/doc/components/bro-plugins/elasticsearch/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/elasticsearch/README \ No newline at end of file diff --git a/doc/components/bro-plugins/netmap/README.rst b/doc/components/bro-plugins/netmap/README.rst new file mode 120000 index 0000000000..819a2bb0e9 --- /dev/null +++ b/doc/components/bro-plugins/netmap/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/netmap/README \ No newline at end of file diff --git a/doc/components/bro-plugins/pf_ring/README.rst b/doc/components/bro-plugins/pf_ring/README.rst new file mode 120000 index 0000000000..5ea666e8c9 --- /dev/null +++ b/doc/components/bro-plugins/pf_ring/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/pf_ring/README \ No newline at end of file diff --git a/doc/components/bro-plugins/redis/README.rst b/doc/components/bro-plugins/redis/README.rst new file mode 120000 index 0000000000..c42051828e --- /dev/null +++ b/doc/components/bro-plugins/redis/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/redis/README \ No newline at end of file diff --git a/doc/components/broker/README.rst b/doc/components/broker/README.rst new file mode 120000 index 0000000000..eafa3b8e77 --- /dev/null +++ b/doc/components/broker/README.rst @@ -0,0 +1 @@ +../../../aux/broker/README \ No newline at end of file diff --git a/doc/components/broker/broker-manual.rst b/doc/components/broker/broker-manual.rst new file mode 120000 index 0000000000..90bf8f0833 --- /dev/null +++ b/doc/components/broker/broker-manual.rst @@ -0,0 +1 @@ +../../../aux/broker/broker-manual.rst \ No newline at end of file diff --git a/doc/components/index.rst b/doc/components/index.rst index fe05f13683..85527e9f9c 100644 --- a/doc/components/index.rst +++ b/doc/components/index.rst @@ -17,8 +17,11 @@ current, independent component releases. Broccoli - User Manual Broccoli Python Bindings Broccoli Ruby Bindings + Broker - Bro's (New) Messaging Library (README) + Broker - User Manual BroControl - Interactive Bro management shell Bro-Aux - Small auxiliary tools for Bro + Bro-Plugins - A collection of plugins for Bro BTest - A unit testing framework Capstats - Command-line packet statistic tool PySubnetTree - Python module for CIDR lookups diff --git a/doc/devel/plugins.rst b/doc/devel/plugins.rst index 76f5c75a68..dc1c9a3cd4 100644 --- a/doc/devel/plugins.rst +++ b/doc/devel/plugins.rst @@ -3,7 +3,7 @@ Writing Bro Plugins =================== -Bro is internally moving to a plugin structure that enables extending +Bro internally provides a plugin API that enables extending the system dynamically, without modifying the core code base. That way custom code remains self-contained and can be maintained, compiled, and installed independently. Currently, plugins can add the following @@ -17,11 +17,11 @@ functionality to Bro: - File analyzers. - - Packet sources and packet dumpers. TODO: Not yet. + - Packet sources and packet dumpers. - - Logging framework backends. TODO: Not yet. + - Logging framework backends. - - Input framework readers. TODO: Not yet. + - Input framework readers. A plugin's functionality is available to the user just as if Bro had the corresponding code built-in. Indeed, internally many of Bro's @@ -32,7 +32,7 @@ Quick Start =========== Writing a basic plugin is quite straight-forward as long as one -follows a few conventions. In the following we walk a simple example +follows a few conventions. In the following we create a simple example plugin that adds a new built-in function (bif) to Bro: we'll add ``rot13(s: string) : string``, a function that rotates every character in a string by 13 places. @@ -42,18 +42,17 @@ certain structure. To get started, Bro's distribution provides a helper script ``aux/bro-aux/plugin-support/init-plugin`` that creates a skeleton plugin that can then be customized. Let's use that:: - # mkdir rot13-plugin - # cd rot13-plugin - # init-plugin Demo Rot13 + # init-plugin ./rot13-plugin Demo Rot13 -As you can see the script takes two arguments. The first is a -namespace the plugin will live in, and the second a descriptive name -for the plugin itself. Bro uses the combination of the two to identify -a plugin. The namespace serves to avoid naming conflicts between -plugins written by independent developers; pick, e.g., the name of -your organisation. The namespace ``Bro`` is reserved for functionality -distributed by the Bro Project. In our example, the plugin will be -called ``Demo::Rot13``. +As you can see, the script takes three arguments. The first is a +directory inside which the plugin skeleton will be created. The second +is the namespace the plugin will live in, and the third is a descriptive +name for the plugin itself relative to the namespace. Bro uses the +combination of namespace and name to identify a plugin. The namespace +serves to avoid naming conflicts between plugins written by independent +developers; pick, e.g., the name of your organisation. The namespace +``Bro`` is reserved for functionality distributed by the Bro Project. In +our example, the plugin will be called ``Demo::Rot13``. The ``init-plugin`` script puts a number of files in place. The full layout is described later. For now, all we need is @@ -61,7 +60,7 @@ layout is described later. For now, all we need is there as follows:: # cat src/rot13.bif - module CaesarCipher; + module Demo; function rot13%(s: string%) : string %{ @@ -82,21 +81,25 @@ The syntax of this file is just like any other ``*.bif`` file; we won't go into it here. Now we can already compile our plugin, we just need to tell the -configure script put in place by ``init-plugin`` where the Bro source -tree is located (Bro needs to have been built there first):: +configure script (that ``init-plugin`` created) where the Bro +source tree is located (Bro needs to have been built there first):: + # cd rot13-plugin # ./configure --bro-dist=/path/to/bro/dist && make [... cmake output ...] -Now our ``rot13-plugin`` directory has everything that it needs -for Bro to recognize it as a dynamic plugin. Once we point Bro to it, -it will pull it in automatically, as we can check with the ``-N`` +This builds the plugin in a subdirectory ``build/``. In fact, that +subdirectory *becomes* the plugin: when ``make`` finishes, ``build/`` +has everything it needs for Bro to recognize it as a dynamic plugin. + +Let's try that. Once we point Bro to the ``build/`` directory, it will +pull in our new plugin automatically, as we can check with the ``-N`` option:: - # export BRO_PLUGIN_PATH=/path/to/rot13-plugin + # export BRO_PLUGIN_PATH=/path/to/rot13-plugin/build # bro -N [...] - Plugin: Demo::Rot13 - (dynamic, version 1) + Demo::Rot13 - (dynamic, version 0.1) [...] That looks quite good, except for the dummy description that we should @@ -105,34 +108,36 @@ is about. We do this by editing the ``config.description`` line in ``src/Plugin.cc``, like this:: [...] - plugin::Configuration Configure() + plugin::Configuration Plugin::Configure() { plugin::Configuration config; config.name = "Demo::Rot13"; config.description = "Caesar cipher rotating a string's characters by 13 places."; - config.version.major = 1; - config.version.minor = 0; + config.version.major = 0; + config.version.minor = 1; return config; } [...] +Now rebuild and verify that the description is visible:: + # make [...] # bro -N | grep Rot13 - Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) + Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 0.1) -Better. Bro can also show us what exactly the plugin provides with the +Bro can also show us what exactly the plugin provides with the more verbose option ``-NN``:: # bro -NN [...] - Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) - [Function] CaesarCipher::rot13 + Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 0.1) + [Function] Demo::rot13 [...] There's our function. Now let's use it:: - # bro -e 'print CaesarCipher::rot13("Hello")' + # bro -e 'print Demo::rot13("Hello")' Uryyb It works. We next install the plugin along with Bro itself, so that it @@ -141,36 +146,42 @@ environment variable. If we first unset the variable, the function will no longer be available:: # unset BRO_PLUGIN_PATH - # bro -e 'print CaesarCipher::rot13("Hello")' - error in , line 1: unknown identifier CaesarCipher::rot13, at or near "CaesarCipher::rot13" + # bro -e 'print Demo::rot13("Hello")' + error in , line 1: unknown identifier Demo::rot13, at or near "Demo::rot13" Once we install it, it works again:: # make install - # bro -e 'print CaesarCipher::rot13("Hello")' + # bro -e 'print Demo::rot13("Hello")' Uryyb The installed version went into ``/lib/bro/plugins/Demo_Rot13``. -We can distribute the plugin in either source or binary form by using -the Makefile's ``sdist`` and ``bdist`` target, respectively. Both -create corrsponding tarballs:: +One can distribute the plugin independently of Bro for others to use. +To distribute in source form, just remove the ``build/`` directory +(``make distclean`` does that) and then tar up the whole ``rot13-plugin/`` +directory. Others then follow the same process as above after +unpacking. - # make sdist - [...] - Source distribution in build/sdist/Demo_Rot13.tar.gz +To distribute the plugin in binary form, the build process +conveniently creates a corresponding tarball in ``build/dist/``. In +this case, it's called ``Demo_Rot13-0.1.tar.gz``, with the version +number coming out of the ``VERSION`` file that ``init-plugin`` put +into place. The binary tarball has everything needed to run the +plugin, but no further source files. Optionally, one can include +further files by specifying them in the plugin's ``CMakeLists.txt`` +through the ``bro_plugin_dist_files`` macro; the skeleton does that +for ``README``, ``VERSION``, ``CHANGES``, and ``COPYING``. To use the +plugin through the binary tarball, just unpack it into +``/lib/bro/plugins/``. Alternatively, if you unpack +it in another location, then you need to point ``BRO_PLUGIN_PATH`` there. - # make bdist - [...] - Binary distribution in build/Demo_Rot13-darwin-x86_64.tar.gz - -The source archive will contain everything in the plugin directory -except any generated files. The binary archive will contain anything -needed to install and run the plugin, i.e., just what ``make install`` -puts into place as well. As the binary distribution is -platform-dependent, its name includes the OS and architecture the -plugin was built on. +Before distributing your plugin, you should edit some of the meta +files that ``init-plugin`` puts in place. Edit ``README`` and +``VERSION``, and update ``CHANGES`` when you make changes. Also put a +license file in place as ``COPYING``; if BSD is fine, you will find a +template in ``COPYING.edit-me``. Plugin Directory Layout ======================= @@ -179,14 +190,14 @@ A plugin's directory needs to follow a set of conventions so that Bro (1) recognizes it as a plugin, and (2) knows what to load. While ``init-plugin`` takes care of most of this, the following is the full story. We'll use ```` to represent a plugin's top-level -directory. +directory. With the skeleton, ```` corresponds to ``build/``. ``/__bro_plugin__`` A file that marks a directory as containing a Bro plugin. The file must exist, and its content must consist of a single line with the qualified name of the plugin (e.g., "Demo::Rot13"). -``/lib/--.so`` +``/lib/.-.so`` The shared library containing the plugin's compiled code. Bro will load this in dynamically at run-time if OS and architecture match the current platform. @@ -198,16 +209,25 @@ directory. "@load"ed. ``scripts``/__load__.bro - A Bro script that will be loaded immediately when the plugin gets - activated. See below for more information on activating plugins. + A Bro script that will be loaded when the plugin gets activated. + When this script executes, any BiF elements that the plugin + defines will already be available. See below for more information + on activating plugins. + +``scripts``/__preload__.bro + A Bro script that will be loaded when the plugin gets activated, + but before any BiF elements become available. See below for more + information on activating plugins. ``lib/bif/`` Directory with auto-generated Bro scripts that declare the plugin's bif elements. The files here are produced by ``bifcl``. +Any other files in ```` are ignored by Bro. + By convention, a plugin should put its custom scripts into sub folders -of ``scripts/``, i.e., ``scripts//