diff --git a/CHANGES b/CHANGES index cc361fa4c0..de80e73dca 100644 --- a/CHANGES +++ b/CHANGES @@ -1,4 +1,556 @@ +2.4-188 | 2015-10-26 14:11:21 -0700 + + * Extending rexmit_inconsistency() event to receive an additional + parameter with the packet's TCP flags, if available. (Robin + Sommer) + +2.4-187 | 2015-10-26 13:43:32 -0700 + + * Updating NEWS for new plugins. (Robin Sommer) + +2.4-186 | 2015-10-23 15:07:06 -0700 + + * Removing pcap options for AF_PACKET support. Addresses BIT-1363. + (Robin Sommer) + + * Correct a typo in controller.bro documentation. (Daniel Thayer) + + * Extend SSL DPD signature to allow alert before server_hello. + (Johanna Amann) + + * Make join_string_vec work with vectors containing empty elements. + (Johanna Amann) + + * Fix support for HTTP CONNECT when server adds headers to response. + (Eric Karasuda). + + * Load static CA list for validation tests too. (Johanna Amann) + + * Remove cluster certificate validation script. (Johanna Amann) + + * Fix a bug in diff-remove-x509-names canonifier. (Daniel Thayer) + + * Fix test canonifiers in scripts/policy/protocols/ssl. (Daniel + Thayer) + +2.4-169 | 2015-10-01 17:21:21 -0700 + + * Fixed parsing of V_ASN1_GENERALIZEDTIME timestamps in x509 + certificates. (Yun Zheng Hu) + + * Improve X509 end-of-string-check code. (Johanna Amann) + + * Refactor X509 generalizedtime support and test. (Johanna Amann) + + * Fix case of offset=-1 (EOF) for RAW reader. Addresses BIT-1479. + (Johanna Amann) + + * Improve a number of test canonifiers. (Daniel Thayer) + + * Remove unnecessary use of TEST_DIFF_CANONIFIER. (Daniel Thayer) + + * Fixed some test canonifiers to read only from stdin + + * Remove unused test canonifier scripts. (Daniel Thayer) + + * A potpourri of updates and improvements across the documentation. + (Daniel Thayer) + + * Add configure option to disable Broker Python bindings. Also + improve the configure summary output to more clearly show whether + or not Broker Python bindings will be built. (Daniel Thayer) + +2.4-131 | 2015-09-11 12:16:39 -0700 + + * Add README.rst symlink. Addresses BIT-1413 (Vlad Grigorescu) + +2.4-129 | 2015-09-11 11:56:04 -0700 + + * hash-all-files.bro depends on base/files/hash (Richard van den Berg) + + * Make dns_max_queries redef-able, and bump default to 25. Addresses + BIT-1460 (Vlad Grigorescu) + +2.4-125 | 2015-09-03 20:10:36 -0700 + + * Move SIP analyzer to flowunit instead of datagram Addresses + BIT-1458 (Vlad Grigorescu) + +2.4-122 | 2015-08-31 14:39:41 -0700 + + * Add a number of out-of-bound checks to layer 2 code. Addresses + BIT-1463 (Johanna Amann) + + * Fix error in 2.4 release notes regarding SSH events. (Robin + Sommer) + +2.4-118 | 2015-08-31 10:55:29 -0700 + + * Fix FreeBSD build errors (Johanna Amann) + +2.4-117 | 2015-08-30 22:16:24 -0700 + + * Fix initialization of a pointer in RDP analyzer. (Daniel + Thayer/Robin Sommer) + +2.4-115 | 2015-08-30 21:57:35 -0700 + + * Enable Bro to leverage packet fanout mode on Linux. (Kris + Nielander). + + ## Toggle whether to do packet fanout (Linux-only). + const Pcap::packet_fanout_enable = F &redef; + + ## If packet fanout is enabled, the id to sue for it. This should be shared amongst + ## worker processes processing the same socket. + const Pcap::packet_fanout_id = 0 &redef; + + ## If packet fanout is enabled, whether packets are to be defragmented before + ## fanout is applied. + const Pcap::packet_fanout_defrag = T &redef; + + * Allow libpcap buffer size to be set via configuration. (Kris Nielander) + + ## Number of Mbytes to provide as buffer space when capturing from live + ## interfaces. + const Pcap::bufsize = 128 &redef; + + * Move the pcap-related script-level identifiers into the new Pcap + namespace. (Robin Sommer) + + snaplen -> Pcap::snaplen + precompile_pcap_filter() -> Pcap::precompile_pcap_filter() + install_pcap_filter() -> Pcap::install_pcap_filter() + pcap_error() -> Pcap::pcap_error() + + +2.4-108 | 2015-08-30 20:14:31 -0700 + + * Update Base64 decoding. (Jan Grashoefer) + + - A new built-in function, decode_base64_conn() for Base64 + decoding. It works like decode_base64() but receives an + additional connection argument that will be used for + reporting decoding errors into weird.log (instead of + reporter.log). + + - FTP, POP3, and HTTP analyzers now likewise log Base64 + decoding errors to weird.log. + + - The built-in functions decode_base64_custom() and + encode_base64_custom() are now deprecated. Their + functionality is provided directly by decode_base64() and + encode_base64(), which take an optional parameter to change + the Base64 alphabet. + + * Fix potential crash if TCP header was captured incompletely. + (Robin Sommer) + +2.4-103 | 2015-08-29 10:51:55 -0700 + + * Make ASN.1 date/time parsing more robust. (Johanna Amann) + + * Be more permissive on what characters we accept as an unquoted + multipart boundary. Addresses BIT-1459. (Johanna Amann) + +2.4-99 | 2015-08-25 07:56:57 -0700 + + * Add ``Q`` and update ``I`` documentation for connection history + field. Addresses BIT-1466. (Vlad Grigorescu) + +2.4-96 | 2015-08-21 17:37:56 -0700 + + * Update SIP analyzer. (balintm) + + - Allows space on both sides of ':'. + - Require CR/LF after request/reply line. + +2.4-94 | 2015-08-21 17:31:32 -0700 + + * Add file type detection support for video/MP2T. (Mike Freemon) + +2.4-93 | 2015-08-21 17:23:39 -0700 + + * Make plugin install honor DESTDIR= convention. (Jeff Barber) + +2.4-89 | 2015-08-18 07:53:36 -0700 + + * Fix diff-canonifier-external to use basename of input file. + (Daniel Thayer) + +2.4-87 | 2015-08-14 08:34:41 -0700 + + * Removing the yielding_teredo_decapsulation option. (Robin Sommer) + +2.4-86 | 2015-08-12 17:02:24 -0700 + + * Make Teredo DPD signature more precise. (Martina Balint) + +2.4-84 | 2015-08-10 14:44:39 -0700 + + * Add hook 'HookSetupAnalyzerTree' to allow plugins access to a + connection's initial analyzer tree for customization. (James + Swaro) + + * Plugins now look for a file "__preload__.bro" in the top-level + script directory. If found, they load it first, before any scripts + defining BiF elements. This can be used to define types that the + BiFs already depend on (like a custom type for an event argument). + (Robin Sommer) + +2.4-81 | 2015-08-08 07:38:42 -0700 + + * Fix a test that is failing very frequently. (Daniel Thayer) + +2.4-78 | 2015-08-06 22:25:19 -0400 + + * Remove build dependency on Perl (now requiring Python instad). + (Daniel Thayer) + + * CID 1314754: Fixing unreachable code in RSH analyzer. (Robin + Sommer) + + * CID 1312752: Add comment to mark 'case' fallthrough as ok. (Robin + Sommer) + + * CID 1312751: Removing redundant assignment. (Robin Sommer) + +2.4-73 | 2015-07-31 08:53:49 -0700 + + * BIT-1429: SMTP logs now include CC: addresses. (Albert Zaharovits) + +2.4-70 | 2015-07-30 07:23:44 -0700 + + * Updated detection of Flash and AdobeAIR. (Jan Grashoefer) + + * Adding tests for Flash version parsing and browser plugin + detection. (Robin Sommer) + +2.4-63 | 2015-07-28 12:26:37 -0700 + + * Updating submodule(s). + +2.4-61 | 2015-07-28 12:13:39 -0700 + + * Renaming config.h to bro-config.h. (Robin Sommer) + +2.4-58 | 2015-07-24 15:06:07 -0700 + + * Add script protocols/conn/vlan-logging.bro to record VLAN data in + conn.log. (Aaron Brown) + + * Add field "vlan" and "inner_vlan" to connection record. (Aaron + Brown) + + * Save the inner vlan in the Packet object for Q-in-Q setups. (Aaron + Brown) + + * Increasing plugin API version for recent packet source changes. + (Robin Sommer) + + * Slightly earlier protocol confirmation for POP3. (Johanna Amann) + +2.4-46 | 2015-07-22 10:56:40 -0500 + + * Fix broker python bindings install location to track --prefix. + (Jon Siwek) + +2.4-45 | 2015-07-21 15:19:43 -0700 + + * Enabling Broker by default. This means CAF is now a required + dependency, altjough for now at least, there's still a switch + --disable-broker to turn it off. + + * Requiring a C++11 compiler, and turning on C++11 support. (Robin + Sommer) + + * Tweaking the listing of hooks in "bro -NN" for consistency. (Robin + Sommer) + +2.4-41 | 2015-07-21 08:35:17 -0700 + + * Fixing compiler warning. (Robin Sommer) + + * Updates to IANA TLS registry. (Johanna Amann) + +2.4-38 | 2015-07-20 15:30:35 -0700 + + * Refactor code to use a common Packet type throught. (Jeff + Barber/Robin Sommer) + + * Extend parsing layer 2 and keeping track of layer 3 protoco. (Jeff Barber) + + * Add a raw_packet() event that generated for all packets and + include layer 2 information. (Jeff Barber) + +2.4-27 | 2015-07-15 13:31:49 -0700 + + * Fix race condition in intel test. (Johanna Amann) + +2.4-24 | 2015-07-14 08:04:11 -0700 + + * Correct Perl package name on FreeBSD in documentation.(Justin Azoff) + + * Adding an environment variable to BTest configuration for external + scripts. (Robin Sommer) + +2.4-20 | 2015-07-03 10:40:21 -0700 + + * Adding a weird for when truncated packets lead TCP reassembly to + ignore content. (Robin Sommer) + +2.4-19 | 2015-07-03 09:04:54 -0700 + + * A set of tests exercising IP defragmentation and TCP reassembly. + (Robin Sommer) + +2.4-17 | 2015-06-28 13:02:41 -0700 + + * BIT-1314: Add detection for Quantum Insert attacks. The TCP + reassembler can now keep a history of old TCP segments using the + tcp_max_old_segments option. An overlapping segment with different + data will then generate an rexmit_inconsistency event. The default + for tcp_max_old_segments is zero, which disabled any additional + buffering. (Yun Zheng Hu/Robin Sommer) + +2.4-14 | 2015-06-28 12:30:12 -0700 + + * BIT-1400: Allow '<' and '>' in MIME multipart boundaries. The spec + doesn't actually seem to permit these, but they seem to occur in + the wild. (Jon Siwek) + +2.4-12 | 2015-06-28 12:21:11 -0700 + + * BIT-1399: Trying to decompress deflated HTTP content even when + zlib headers are missing. (Seth Hall) + +2.4-10 | 2015-06-25 07:11:17 -0700 + + * Correct a name used in a header identifier (Justin Azoff) + +2.4-8 | 2015-06-24 07:50:50 -0700 + + * Restore the --load-seeds cmd-line option and enable the short + options -G/-H for --load-seeds/--save-seeds. (Daniel Thayer) + +2.4-6 | 2015-06-19 16:26:40 -0700 + + * Generate protocol confirmations for Modbus, making it appear as a + confirmed service in conn.log. (Seth Hall) + + * Put command line options in alphabetical order. (Daniel Thayer) + + * Removing dead code for no longer supported -G switch. (Robin + Sommer) (Robin Sommer) + +2.4 | 2015-06-09 07:30:53 -0700 + + * Release 2.4. + + * Fixing tiny thing in NEWS. (Robin Sommer) + +2.4-beta-42 | 2015-06-08 09:41:39 -0700 + + * Fix reporter errors with GridFTP traffic. (Robin Sommer) + +2.4-beta-40 | 2015-06-06 08:20:52 -0700 + + * PE Analyzer: Change how we calculate the rva_table size. (Vlad Grigorescu) + +2.4-beta-39 | 2015-06-05 09:09:44 -0500 + + * Fix a unit test to check for Broker requirement. (Jon Siwek) + +2.4-beta-38 | 2015-06-04 14:48:37 -0700 + + * Test for Broker termination. (Robin Sommer) + +2.4-beta-37 | 2015-06-04 07:53:52 -0700 + + * BIT-1408: Improve I/O loop and Broker IOSource. (Jon Siwek) + +2.4-beta-34 | 2015-06-02 10:37:22 -0700 + + * Add signature support for F4M files. (Seth Hall) + +2.4-beta-32 | 2015-06-02 09:43:31 -0700 + + * A larger set of documentation updates, fixes, and extentions. + (Daniel Thayer) + +2.4-beta-14 | 2015-06-02 09:16:44 -0700 + + * Add memleak btest for attachments over SMTP. (Vlad Grigorescu) + + * BIT-1410: Fix flipped tx_hosts and rx_hosts in files.log. Reported + by Ali Hadi. (Vlad Grigorescu) + + * Updating the Mozilla root certs. (Seth Hall) + + * Updates for the urls.bro script. Fixes BIT-1404. (Seth Hall) + +2.4-beta-6 | 2015-05-28 13:20:44 -0700 + + * Updating submodule(s). + +2.4-beta-2 | 2015-05-26 08:58:37 -0700 + + * Fix segfault when DNS is not available. Addresses BIT-1387. (Frank + Meier and Robin Sommer) + +2.4-beta | 2015-05-07 21:55:31 -0700 + + * Release 2.4-beta. + + * Update local-compat.test (Johanna Amann) + +2.3-913 | 2015-05-06 09:58:00 -0700 + + * Add /sbin to PATH in btest.cfg and remove duplicate default_path. + (Daniel Thayer) + +2.3-911 | 2015-05-04 09:58:09 -0700 + + * Update usage output and list of command line options. (Daniel + Thayer) + + * Fix to ssh/geo-data.bro for unset directions. (Vlad Grigorescu) + + * Improve SIP logging and remove reporter messages. (Seth Hall) + +2.3-905 | 2015-04-29 17:01:30 -0700 + + * Improve SIP logging and remove reporter messages. (Seth Hall) + +2.3-903 | 2015-04-27 17:27:59 -0700 + + * BIT-1350: Improve record coercion type checking. (Jon Siwek) + +2.3-901 | 2015-04-27 17:25:27 -0700 + + * BIT-1384: Remove -O (optimize scripts) command-line option, which + hadn't been working for a while already. (Jon Siwek) + +2.3-899 | 2015-04-27 17:22:42 -0700 + + * Fix the -J/--set-seed cmd-line option. (Daniel Thayer) + + * Remove unused -l, -L, and -Z cmd-line options. (Daniel Thayer) + +2.3-892 | 2015-04-27 08:22:22 -0700 + + * Fix typos in the Broker BIF documentation. (Daniel Thayer) + + * Update installation instructions and remove outdated references. + (Johanna Amann) + + * Easier support for systems with tcmalloc_minimal installed. (Seth + Hall) + +2.3-884 | 2015-04-23 12:30:15 -0500 + + * Fix some outdated documentation unit tests. (Jon Siwek) + +2.3-883 | 2015-04-23 07:10:36 -0700 + + * Fix -N option to work with builtin plugins as well. (Robin Sommer) + +2.3-882 | 2015-04-23 06:59:40 -0700 + + * Add missing .pac dependencies for some binpac analyzer targets. + (Jon Siwek) + +2.3-879 | 2015-04-22 10:38:07 -0500 + + * Fix compile errors. (Jon Siwek) + +2.3-878 | 2015-04-22 08:21:23 -0700 + + * Fix another compiler warning in DTLS. (Johanna Amann) + +2.3-877 | 2015-04-21 20:14:16 -0700 + + * Adding missing include. (Robin Sommer) + +2.3-876 | 2015-04-21 16:40:10 -0700 + + * Attempt at fixing a potential std::length_error exception in RDP + analyzer. Addresses BIT-1337. (Robin Sommer) + + * Fixing compile problem caused by overeager factorization. (Robin + Sommer) + +2.3-874 | 2015-04-21 16:09:20 -0700 + + * Change details of escaping when logging/printing. (Seth Hall/Robin + Sommer) + + - Log files now escape non-printable characters consistently + as "\xXX'. Furthermore, backslashes are escaped as "\\", + making the representation fully reversible. + + - When escaping via script-level functions (escape_string, + clean), we likewise now escape consistently with "\xXX" and + "\\". + + - There's no "alternative" output style anymore, i.e., fmt() + '%A' qualifier is gone. + + Addresses BIT-1333. + + * Remove several BroString escaping methods that are no longer + useful. (Seth Hall) + +2.3-864 | 2015-04-21 15:24:02 -0700 + + * A SIP protocol analyzer. (Vlad Grigorescu) + + Activity gets logged into sip.log. It generates the following + events: + + event sip_request(c: connection, method: string, original_URI: string, version: string); + event sip_reply(c: connection, version: string, code: count, reason: string); + event sip_header(c: connection, is_orig: bool, name: string, value: string); + event sip_all_headers(c: connection, is_orig: bool, hlist: mime_header_list); + event sip_begin_entity(c: connection, is_orig: bool); + event sip_end_entity(c: connection, is_orig: bool); + + The analyzer support SIP over UDP currently. + + * BIT-1343: Factor common ASN.1 code from RDP, SNMP, and Kerberos + analyzers. (Jon Siwek/Robin Sommer) + +2.3-838 | 2015-04-21 13:40:12 -0700 + + * BIT-1373: Fix vector index assignment reference count bug. (Jon Siwek) + +2.3-836 | 2015-04-21 13:37:31 -0700 + + * Fix SSH direction field being unset. Addresses BIT-1365. (Vlad + Grigorescu) + +2.3-835 | 2015-04-21 16:36:00 -0500 + + * Clarify Broker examples. (Jon Siwek) + +2.3-833 | 2015-04-21 12:38:32 -0700 + + * A Kerberos protocol analyzer. (Vlad Grigorescu) + + Activity gets logged into kerberos.log. It generates the following + events: + + event krb_as_request(c: connection, msg: KRB::KDC_Request); + event krb_as_response(c: connection, msg: KRB::KDC_Response); + event krb_tgs_request(c: connection, msg: KRB::KDC_Request); + event krb_tgs_response(c: connection, msg: KRB::KDC_Response); + event krb_ap_request(c: connection, ticket: KRB::Ticket, opts: KRB::AP_Options); + event krb_priv(c: connection, is_orig: bool); + event krb_safe(c: connection, is_orig: bool, msg: KRB::SAFE_Msg); + event krb_cred(c: connection, is_orig: bool, tickets: KRB::Ticket_Vector); + event krb_error(c: connection, msg: KRB::Error_Msg); + 2.3-793 | 2015-04-20 20:51:00 -0700 * Add decoding of PROXY-AUTHORIZATION header to HTTP analyze, diff --git a/CMakeLists.txt b/CMakeLists.txt index 8f60ab95ad..b96923aa56 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -61,7 +61,7 @@ if (NOT SED_EXE) endif () endif () -FindRequiredPackage(Perl) +FindRequiredPackage(PythonInterp) FindRequiredPackage(FLEX) FindRequiredPackage(BISON) FindRequiredPackage(PCAP) @@ -113,7 +113,7 @@ if (NOT DISABLE_PERFTOOLS) find_package(GooglePerftools) endif () -if (GOOGLEPERFTOOLS_FOUND) +if (GOOGLEPERFTOOLS_FOUND OR TCMALLOC_FOUND) set(HAVE_PERFTOOLS true) # Non-Linux systems may not be well-supported by gperftools, so # require explicit request from user to enable it in that case. @@ -165,22 +165,19 @@ include(PCAPTests) include(OpenSSLTests) include(CheckNameserCompat) include(GetArchitecture) +include(RequireCXX11) # Tell the plugin code that we're building as part of the main tree. set(BRO_PLUGIN_INTERNAL_BUILD true CACHE INTERNAL "" FORCE) -configure_file(${CMAKE_CURRENT_SOURCE_DIR}/config.h.in - ${CMAKE_CURRENT_BINARY_DIR}/config.h) +configure_file(${CMAKE_CURRENT_SOURCE_DIR}/bro-config.h.in + ${CMAKE_CURRENT_BINARY_DIR}/bro-config.h) include_directories(${CMAKE_CURRENT_BINARY_DIR}) ######################################################################## ## Recurse on sub-directories -if ( ENABLE_CXX11 ) - include(RequireCXX11) -endif () - if ( ENABLE_BROKER ) add_subdirectory(aux/broker) set(brodeps ${brodeps} broker) @@ -236,6 +233,7 @@ message( "\nCPP: ${CMAKE_CXX_COMPILER}" "\n" "\nBroker: ${ENABLE_BROKER}" + "\nBroker Python: ${BROKER_PYTHON_BINDINGS}" "\nBroccoli: ${INSTALL_BROCCOLI}" "\nBroctl: ${INSTALL_BROCTL}" "\nAux. Tools: ${INSTALL_AUX_TOOLS}" diff --git a/COPYING b/COPYING index 2c66f98113..5454660df2 100644 --- a/COPYING +++ b/COPYING @@ -1,4 +1,4 @@ -Copyright (c) 1995-2013, The Regents of the University of California +Copyright (c) 1995-2015, The Regents of the University of California through the Lawrence Berkeley National Laboratory and the International Computer Science Institute. All rights reserved. diff --git a/NEWS b/NEWS index c42145c9d2..cdb76383a1 100644 --- a/NEWS +++ b/NEWS @@ -4,19 +4,80 @@ release. For an exhaustive list of changes, see the ``CHANGES`` file (note that submodules, such as BroControl and Broccoli, come with their own ``CHANGES``.) -Bro 2.4 (in progress) +Bro 2.5 (in progress) ===================== +New Dependencies +---------------- + +- Bro now requires a compiler with C++11 support for building the + source code. + +- Bro now requires the C++ Actor Framework, CAF, which must be + installed first. See http://actor-framework.org. + +- Bro now requires Python instead of Perl to compile the source code. + +- The pcap buffer size can set through the new option Pcap::bufsize. + + +New Functionality +----------------- + +- Bro now tracks VLAN IDs. To record them inside the connection log, + load protocols/conn/vlan-logging.bro. + +- A new per-packet event raw_packet() provides access to layer 2 + information. Use with care, generating events per packet is + expensive. + +- A new built-in function, decode_base64_conn() for Base64 decoding. + It works like decode_base64() but receives an additional connection + argument that will be used for decoding errors into weird.log + (instead of reporter.log). + +- New Bro plugins in aux/plugins: + + - af_packet: Native AF_PACKET support. + - myricom: Native Myricom SNF v3 support. + - pf_ring: Native PF_RING support. + - redis: An experimental log writer for Redis. + +Changed Functionality +--------------------- + +- Some script-level identifier have changed their names: + + snaplen -> Pcap::snaplen + precompile_pcap_filter() -> Pcap::precompile_pcap_filter() + install_pcap_filter() -> Pcap::install_pcap_filter() + pcap_error() -> Pcap::pcap_error() + + +Deprecated Functionality +------------------------ + + - The built-in functions decode_base64_custom() and + encode_base64_custom() are no longer needed and will be removed + in the future. Their functionality is now provided directly by + decode_base64() and encode_base64(), which take an optional + parameter to change the Base64 alphabet. + +Bro 2.4 +======= + New Functionality ----------------- - Bro now has support for external plugins that can extend its core functionality, like protocol/file analysis, via shared libraries. Plugins can be developed and distributed externally, and will be - pulled in dynamically at startup. Currently, a plugin can provide - custom protocol analyzers, file analyzers, log writers, input - readers, packet sources and dumpers, and new built-in functions. A - plugin can furthermore hook into Bro's processing at a number of + pulled in dynamically at startup (the environment variables + BRO_PLUGIN_PATH and BRO_PLUGIN_ACTIVATE can be used to specify the + locations and names of plugins to activate). Currently, a plugin + can provide custom protocol analyzers, file analyzers, log writers, + input readers, packet sources and dumpers, and new built-in functions. + A plugin can furthermore hook into Bro's processing at a number of places to add custom logic. See https://www.bro.org/sphinx-git/devel/plugins.html for more @@ -27,21 +88,35 @@ New Functionality - Bro now parses DTLS traffic. Activity gets logged into ssl.log. +- Bro now has support for the Kerberos KRB5 protocol over TCP and + UDP. Activity gets logged into kerberos.log. + - Bro now has an RDP analyzer. Activity gets logged into rdp.log. - Bro now has a file analyzer for Portable Executables. Activity gets logged into pe.log. -- Bro now features a completely rewritten, enhanced SSH analyzer, with - a set of addedd events being generated. A lot more information about - SSH sessions is logged. The analyzer is able to determine if logins - failed or succeeded in most circumstances. +- Bro now has support for the SIP protocol over UDP. Activity gets + logged into sip.log. + +- Bro now features a completely rewritten, enhanced SSH analyzer. The + new analyzer is able to determine if logins failed or succeeded in + most circumstances, logs a lot more more information about SSH + sessions, supports v1, and introduces the intelligence type + ``Intel::PUBKEY_HASH`` and location ``SSH::IN_SERVER_HOST_KEY``. The + analayzer also generates a set of additional events + (``ssh_auth_successful``, ``ssh_auth_failed``, ``ssh_capabilities``, + ``ssh2_server_host_key``, ``ssh1_server_host_key``, + ``ssh_encrypted_packet``, ``ssh2_dh_server_params``, + ``ssh2_gss_error``, ``ssh2_ecc_key``). See next section for + incompatible SSH changes. - Bro's file analysis now supports reassembly of files that are not transferred/seen sequentially. The default file reassembly buffer size is set with the ``Files::reassembly_buffer_size`` variable. -- Bro's file type identification has been greatly improved. +- Bro's file type identification has been greatly improved (new file types, + bug fixes, and performance improvements). - Bro's scripting language now has a ``while`` statement:: @@ -67,7 +142,7 @@ New Functionality C++11 compiler (e.g. GCC 4.8+ or Clang 3.3+). Broker will become a mandatory dependency in future Bro versions and - replace the current communcation and serialization system. + replace the current communication and serialization system. - Add --enable-c++11 configure flag to compile Bro's source code in C++11 mode with a corresponding compiler. Note that 2.4 will be the @@ -75,10 +150,10 @@ New Functionality - The SSL analysis now alerts when encountering SSL connections with old protocol versions or unsafe cipher suites. It also gained - extended reporting of weak keys, caching of already valdidated - certificates, full support TLS record defragmentation. SSL generally + extended reporting of weak keys, caching of already validated + certificates, and full support for TLS record defragmentation. SSL generally became much more robust and added several fields to ssl.log (while - removing some other). + removing some others). - A new icmp_sent_payload event provides access to ICMP payload. @@ -91,6 +166,9 @@ New Functionality threshold in terms of packets or bytes. The primary API for that functionality is in base/protocols/conn/thresholds.bro. +- There is a new command-line option -Q/--time that prints Bro's execution + time and memory usage to stderr. + - BroControl now has a new command "deploy" which is equivalent to running the "check", "install", "stop", and "start" commands (in that order). @@ -139,10 +217,22 @@ Changed Functionality reassembly for non-sequential files, "offset" can be obtained with other information already available -- adding together ``seen_bytes`` and ``missed_bytes`` fields of the ``fa_file`` - record gives the how many bytes have been written so far (i.e. + record gives how many bytes have been written so far (i.e. the "offset"). -- has_valid_octets: now uses a string_vec parameter instead of +- The SSH changes come with a few incompatibilities. The following + events have been renamed: + + * ``SSH::heuristic_failed_login`` to ``ssh_auth_failed`` + * ``SSH::heuristic_successful_login`` to ``ssh_auth_successful`` + + The ``SSH::Info`` status field has been removed and replaced with + the ``auth_success`` field. This field has been changed from a + string that was previously ``success``, ``failure`` or + ``undetermined`` to a boolean. a boolean that is ``T``, ``F``, or + unset. + +- The has_valid_octets function now uses a string_vec parameter instead of string_array. - conn.log gained a new field local_resp that works like local_orig, @@ -186,12 +276,24 @@ Changed Functionality to stdout. Error messages are still sent to stderr, however. - The capability of processing NetFlow input has been removed for the - time being. + time being. Therefore, the -y/--flowfile and -Y/--netflow command-line + options have been removed, and the netflow_v5_header and netflow_v5_record + events have been removed. + +- The -D/--dfa-size command-line option has been removed. + +- The -L/--rule-benchmark command-line option has been removed. + +- The -O/--optimize command-line option has been removed. - The deprecated fields "hot" and "addl" have been removed from the connection record. Likewise, the functions append_addl() and append_addl_marker() have been removed. +- Log files now escape non-printable characters consistently as "\xXX'. + Furthermore, backslashes are escaped as "\\", making the + representation fully reversible. + Deprecated Functionality ------------------------ @@ -201,7 +303,7 @@ Deprecated Functionality concatenation/extraction functions. Note that the new functions use 0-based indexing, rather than 1-based. - The full list of now deprecation functions is: + The full list of now deprecated functions is: * split: use split_string instead. diff --git a/README.rst b/README.rst new file mode 120000 index 0000000000..100b93820a --- /dev/null +++ b/README.rst @@ -0,0 +1 @@ +README \ No newline at end of file diff --git a/VERSION b/VERSION index ed37397383..64502bd1aa 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.3-793 +2.4-188 diff --git a/aux/binpac b/aux/binpac index 544330932e..214294c502 160000 --- a/aux/binpac +++ b/aux/binpac @@ -1 +1 @@ -Subproject commit 544330932e7cd4615d6d19f63907e8aa2acebb9e +Subproject commit 214294c502d377bb7bf511eac8c43608e54c875a diff --git a/aux/bro-aux b/aux/bro-aux index 462e300bf9..4e0d2bff4b 160000 --- a/aux/bro-aux +++ b/aux/bro-aux @@ -1 +1 @@ -Subproject commit 462e300bf9c37dcc39b70a4c2d89d19f7351c804 +Subproject commit 4e0d2bff4b2c287f66186c3654ef784bb0748d11 diff --git a/aux/broccoli b/aux/broccoli index 45276b39a9..8046800085 160000 --- a/aux/broccoli +++ b/aux/broccoli @@ -1 +1 @@ -Subproject commit 45276b39a946d70095c983753cd321ad07dcf285 +Subproject commit 80468000859bcb7c3784c69280888fcfe89d8922 diff --git a/aux/broctl b/aux/broctl index d52d184bc9..921b0abcb9 160000 --- a/aux/broctl +++ b/aux/broctl @@ -1 +1 @@ -Subproject commit d52d184bc9aa976ee465914e95ff5c0274a18216 +Subproject commit 921b0abcb967666d8349c0c6c2bb8e41e1300579 diff --git a/aux/broker b/aux/broker index a9d74d9133..e7da54a3f4 160000 --- a/aux/broker +++ b/aux/broker @@ -1 +1 @@ -Subproject commit a9d74d91333b403be8d8c01f5aadb03a84968e9c +Subproject commit e7da54a3f40e71ca9020f9846256f60c0b885963 diff --git a/aux/btest b/aux/btest index d69df586c9..ce1d474859 160000 --- a/aux/btest +++ b/aux/btest @@ -1 +1 @@ -Subproject commit d69df586c91531db0c3abe838b10a429dda4fa87 +Subproject commit ce1d474859cc8a0f39d5eaf69fb1bb56eb1a5161 diff --git a/aux/plugins b/aux/plugins index 7a14085394..4354b330d9 160000 --- a/aux/plugins +++ b/aux/plugins @@ -1 +1 @@ -Subproject commit 7a14085394e54a950e477eb4fafb3827ff8dbdc3 +Subproject commit 4354b330d914a50f99da05cc78f830b5e86bd64e diff --git a/config.h.in b/bro-config.h.in similarity index 100% rename from config.h.in rename to bro-config.h.in diff --git a/cmake b/cmake index 2fd35ab6a6..843cdf6a91 160000 --- a/cmake +++ b/cmake @@ -1 +1 @@ -Subproject commit 2fd35ab6a6245a005828c32f0aa87eb21698c054 +Subproject commit 843cdf6a91f06e5407bffbc79a343bff3cf4c81f diff --git a/configure b/configure index b139ee2bec..f94085f9d3 100755 --- a/configure +++ b/configure @@ -41,14 +41,13 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --enable-perftools-debug use Google's perftools for debugging --enable-jemalloc link against jemalloc --enable-ruby build ruby bindings for broccoli (deprecated) - --enable-c++11 build using the C++11 standard - --enable-broker enable use of the Broker communication library - (requires C++ Actor Framework and C++11) + --disable-broker disable use of the Broker communication library --disable-broccoli don't build or install the Broccoli library --disable-broctl don't install Broctl --disable-auxtools don't build or install auxiliary tools --disable-perftools don't try to build with Google Perftools --disable-python don't try to build python bindings for broccoli + --disable-pybroker don't try to build python bindings for broker Required Packages in Non-Standard Locations: --with-openssl=PATH path to OpenSSL install root @@ -57,7 +56,7 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --with-binpac=PATH path to BinPAC install root --with-flex=PATH path to flex executable --with-bison=PATH path to bison executable - --with-perl=PATH path to perl executable + --with-python=PATH path to Python executable --with-libcaf=PATH path to C++ Actor Framework installation (a required Broker dependency) @@ -65,7 +64,6 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --with-geoip=PATH path to the libGeoIP install root --with-perftools=PATH path to Google Perftools install root --with-jemalloc=PATH path to jemalloc install root - --with-python=PATH path to Python interpreter --with-python-lib=PATH path to libpython --with-python-inc=PATH path to Python headers --with-ruby=PATH path to ruby interpreter @@ -95,7 +93,7 @@ Usage: $0 [OPTION]... [VAR=VALUE]... sourcedir="$( cd "$( dirname "$0" )" && pwd )" # Function to append a CMake cache entry definition to the -# CMakeCacheEntries variable +# CMakeCacheEntries variable. # $1 is the cache entry variable name # $2 is the cache entry variable type # $3 is the cache entry variable value @@ -103,6 +101,17 @@ append_cache_entry () { CMakeCacheEntries="$CMakeCacheEntries -D $1:$2=$3" } +# Function to remove a CMake cache entry definition from the +# CMakeCacheEntries variable +# $1 is the cache entry variable name +remove_cache_entry () { + CMakeCacheEntries="$CMakeCacheEntries -U $1" + + # Even with -U, cmake still warns by default if + # added previously with -D. + CMakeCacheEntries="$CMakeCacheEntries --no-warn-unused-cli" +} + # set defaults builddir=build prefix=/usr/local/bro @@ -112,10 +121,13 @@ append_cache_entry BRO_ROOT_DIR PATH $prefix append_cache_entry PY_MOD_INSTALL_DIR PATH $prefix/lib/broctl append_cache_entry BRO_SCRIPT_INSTALL_PATH STRING $prefix/share/bro append_cache_entry BRO_ETC_INSTALL_DIR PATH $prefix/etc +append_cache_entry BROKER_PYTHON_HOME PATH $prefix +append_cache_entry BROKER_PYTHON_BINDINGS BOOL false append_cache_entry ENABLE_DEBUG BOOL false append_cache_entry ENABLE_PERFTOOLS BOOL false append_cache_entry ENABLE_PERFTOOLS_DEBUG BOOL false -append_cache_entry ENABLE_JEMALLOC BOOL false +append_cache_entry ENABLE_JEMALLOC BOOL false +append_cache_entry ENABLE_BROKER BOOL true append_cache_entry BinPAC_SKIP_INSTALL BOOL true append_cache_entry BUILD_SHARED_LIBS BOOL true append_cache_entry INSTALL_AUX_TOOLS BOOL true @@ -150,8 +162,8 @@ while [ $# -ne 0 ]; do append_cache_entry BRO_ROOT_DIR PATH $optarg append_cache_entry PY_MOD_INSTALL_DIR PATH $optarg/lib/broctl - if [ -n "$user_enabled_broker" ]; then - append_cache_entry BROKER_PYTHON_HOME PATH $prefix + if [ -z "$user_disabled_broker" ]; then + append_cache_entry BROKER_PYTHON_HOME PATH $optarg fi ;; --scriptdir=*) @@ -187,14 +199,10 @@ while [ $# -ne 0 ]; do --enable-jemalloc) append_cache_entry ENABLE_JEMALLOC BOOL true ;; - --enable-c++11) - append_cache_entry ENABLE_CXX11 BOOL true - ;; - --enable-broker) - append_cache_entry ENABLE_CXX11 BOOL true - append_cache_entry ENABLE_BROKER BOOL true - append_cache_entry BROKER_PYTHON_HOME PATH $prefix - user_enabled_broker="true" + --disable-broker) + append_cache_entry ENABLE_BROKER BOOL false + remove_cache_entry BROKER_PYTHON_HOME + user_disabled_broker="true" ;; --disable-broccoli) append_cache_entry INSTALL_BROCCOLI BOOL false @@ -211,6 +219,9 @@ while [ $# -ne 0 ]; do --disable-python) append_cache_entry DISABLE_PYTHON_BINDINGS BOOL true ;; + --disable-pybroker) + append_cache_entry DISABLE_PYBROKER BOOL true + ;; --enable-ruby) append_cache_entry DISABLE_RUBY_BINDINGS BOOL false ;; @@ -232,9 +243,6 @@ while [ $# -ne 0 ]; do --with-bison=*) append_cache_entry BISON_EXECUTABLE PATH $optarg ;; - --with-perl=*) - append_cache_entry PERL_EXECUTABLE PATH $optarg - ;; --with-geoip=*) append_cache_entry LibGeoIP_ROOT_DIR PATH $optarg ;; diff --git a/doc/components/bro-plugins/README.rst b/doc/components/bro-plugins/README.rst new file mode 120000 index 0000000000..8f96f50909 --- /dev/null +++ b/doc/components/bro-plugins/README.rst @@ -0,0 +1 @@ +../../../aux/plugins/README \ No newline at end of file diff --git a/doc/components/bro-plugins/dataseries/README.rst b/doc/components/bro-plugins/dataseries/README.rst new file mode 120000 index 0000000000..3362e911fc --- /dev/null +++ b/doc/components/bro-plugins/dataseries/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/dataseries/README \ No newline at end of file diff --git a/doc/components/bro-plugins/elasticsearch/README.rst b/doc/components/bro-plugins/elasticsearch/README.rst new file mode 120000 index 0000000000..8a5b78d689 --- /dev/null +++ b/doc/components/bro-plugins/elasticsearch/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/elasticsearch/README \ No newline at end of file diff --git a/doc/components/bro-plugins/netmap/README.rst b/doc/components/bro-plugins/netmap/README.rst new file mode 120000 index 0000000000..819a2bb0e9 --- /dev/null +++ b/doc/components/bro-plugins/netmap/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/netmap/README \ No newline at end of file diff --git a/doc/components/bro-plugins/pf_ring/README.rst b/doc/components/bro-plugins/pf_ring/README.rst new file mode 120000 index 0000000000..5ea666e8c9 --- /dev/null +++ b/doc/components/bro-plugins/pf_ring/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/pf_ring/README \ No newline at end of file diff --git a/doc/components/bro-plugins/redis/README.rst b/doc/components/bro-plugins/redis/README.rst new file mode 120000 index 0000000000..c42051828e --- /dev/null +++ b/doc/components/bro-plugins/redis/README.rst @@ -0,0 +1 @@ +../../../../aux/plugins/redis/README \ No newline at end of file diff --git a/doc/components/index.rst b/doc/components/index.rst index c1feda4a61..85527e9f9c 100644 --- a/doc/components/index.rst +++ b/doc/components/index.rst @@ -21,6 +21,7 @@ current, independent component releases. Broker - User Manual BroControl - Interactive Bro management shell Bro-Aux - Small auxiliary tools for Bro + Bro-Plugins - A collection of plugins for Bro BTest - A unit testing framework Capstats - Command-line packet statistic tool PySubnetTree - Python module for CIDR lookups diff --git a/doc/devel/plugins.rst b/doc/devel/plugins.rst index 5c963a1552..dc1c9a3cd4 100644 --- a/doc/devel/plugins.rst +++ b/doc/devel/plugins.rst @@ -3,7 +3,7 @@ Writing Bro Plugins =================== -Bro internally provides plugin API that enables extending +Bro internally provides a plugin API that enables extending the system dynamically, without modifying the core code base. That way custom code remains self-contained and can be maintained, compiled, and installed independently. Currently, plugins can add the following @@ -32,7 +32,7 @@ Quick Start =========== Writing a basic plugin is quite straight-forward as long as one -follows a few conventions. In the following we walk a simple example +follows a few conventions. In the following we create a simple example plugin that adds a new built-in function (bif) to Bro: we'll add ``rot13(s: string) : string``, a function that rotates every character in a string by 13 places. @@ -81,7 +81,7 @@ The syntax of this file is just like any other ``*.bif`` file; we won't go into it here. Now we can already compile our plugin, we just need to tell the -configure script that ``init-plugin`` put in place where the Bro +configure script (that ``init-plugin`` created) where the Bro source tree is located (Bro needs to have been built there first):: # cd rot13-plugin @@ -99,7 +99,7 @@ option:: # export BRO_PLUGIN_PATH=/path/to/rot13-plugin/build # bro -N [...] - Plugin: Demo::Rot13 - (dynamic, version 1) + Demo::Rot13 - (dynamic, version 0.1) [...] That looks quite good, except for the dummy description that we should @@ -108,28 +108,30 @@ is about. We do this by editing the ``config.description`` line in ``src/Plugin.cc``, like this:: [...] - plugin::Configuration Configure() + plugin::Configuration Plugin::Configure() { plugin::Configuration config; config.name = "Demo::Rot13"; config.description = "Caesar cipher rotating a string's characters by 13 places."; - config.version.major = 1; - config.version.minor = 0; + config.version.major = 0; + config.version.minor = 1; return config; } [...] +Now rebuild and verify that the description is visible:: + # make [...] # bro -N | grep Rot13 - Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) + Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 0.1) -Better. Bro can also show us what exactly the plugin provides with the +Bro can also show us what exactly the plugin provides with the more verbose option ``-NN``:: # bro -NN [...] - Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) + Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 0.1) [Function] Demo::rot13 [...] @@ -157,10 +159,12 @@ The installed version went into ``/lib/bro/plugins/Demo_Rot13``. One can distribute the plugin independently of Bro for others to use. -To distribute in source form, just remove the ``build/`` (``make -distclean`` does that) and then tar up the whole ``rot13-plugin/`` +To distribute in source form, just remove the ``build/`` directory +(``make distclean`` does that) and then tar up the whole ``rot13-plugin/`` directory. Others then follow the same process as above after -unpacking. To distribute the plugin in binary form, the build process +unpacking. + +To distribute the plugin in binary form, the build process conveniently creates a corresponding tarball in ``build/dist/``. In this case, it's called ``Demo_Rot13-0.1.tar.gz``, with the version number coming out of the ``VERSION`` file that ``init-plugin`` put @@ -169,14 +173,14 @@ plugin, but no further source files. Optionally, one can include further files by specifying them in the plugin's ``CMakeLists.txt`` through the ``bro_plugin_dist_files`` macro; the skeleton does that for ``README``, ``VERSION``, ``CHANGES``, and ``COPYING``. To use the -plugin through the binary tarball, just unpack it and point -``BRO_PLUGIN_PATH`` there; or copy it into -``/lib/bro/plugins/`` directly. +plugin through the binary tarball, just unpack it into +``/lib/bro/plugins/``. Alternatively, if you unpack +it in another location, then you need to point ``BRO_PLUGIN_PATH`` there. Before distributing your plugin, you should edit some of the meta files that ``init-plugin`` puts in place. Edit ``README`` and ``VERSION``, and update ``CHANGES`` when you make changes. Also put a -license file in place as ``COPYING``; if BSD is fine, you find a +license file in place as ``COPYING``; if BSD is fine, you will find a template in ``COPYING.edit-me``. Plugin Directory Layout @@ -193,7 +197,7 @@ directory. With the skeleton, ```` corresponds to ``build/``. must exist, and its content must consist of a single line with the qualified name of the plugin (e.g., "Demo::Rot13"). -``/lib/--.so`` +``/lib/.-.so`` The shared library containing the plugin's compiled code. Bro will load this in dynamically at run-time if OS and architecture match the current platform. @@ -205,8 +209,15 @@ directory. With the skeleton, ```` corresponds to ``build/``. "@load"ed. ``scripts``/__load__.bro - A Bro script that will be loaded immediately when the plugin gets - activated. See below for more information on activating plugins. + A Bro script that will be loaded when the plugin gets activated. + When this script executes, any BiF elements that the plugin + defines will already be available. See below for more information + on activating plugins. + +``scripts``/__preload__.bro + A Bro script that will be loaded when the plugin gets activated, + but before any BiF elements become available. See below for more + information on activating plugins. ``lib/bif/`` Directory with auto-generated Bro scripts that declare the plugin's @@ -215,8 +226,8 @@ directory. With the skeleton, ```` corresponds to ``build/``. Any other files in ```` are ignored by Bro. By convention, a plugin should put its custom scripts into sub folders -of ``scripts/``, i.e., ``scripts//