diff --git a/.gitmodules b/.gitmodules index 24375ce23d..91f39e3d04 100644 --- a/.gitmodules +++ b/.gitmodules @@ -22,3 +22,6 @@ [submodule "aux/plugins"] path = aux/plugins url = git://git.bro.org/bro-plugins +[submodule "aux/broker"] + path = aux/broker + url = git://git.bro.org/broker diff --git a/CHANGES b/CHANGES index 3642c9c528..db09fdd340 100644 --- a/CHANGES +++ b/CHANGES @@ -1,4 +1,557 @@ +2.3-680 | 2015-04-06 16:02:43 -0500 + + * BIT-1371: remove CMake version check from binary package scripts. + (Jon Siwek) + +2.3-679 | 2015-04-06 10:16:36 -0500 + + * Increase some unit test timeouts. (Jon Siwek) + + * Fix Coverity warning in RDP analyzer. (Jon Siwek) + +2.3-676 | 2015-04-02 10:10:39 -0500 + + * BIT-1366: improve checksum offloading warning. + (Frank Meier, Jon Siwek) + +2.3-675 | 2015-03-30 17:05:05 -0500 + + * Add an RDP analyzer. (Josh Liburdi, Seth Hall, Johanna Amann) + +2.3-640 | 2015-03-30 13:51:51 -0500 + + * BIT-1359: Limit maximum number of DTLS fragments to 30. (Johanna Amann) + +2.3-637 | 2015-03-30 12:02:07 -0500 + + * Increase timeout duration in some broker tests. (Jon Siwek) + +2.3-636 | 2015-03-30 11:26:32 -0500 + + * Updates related to SSH analysis. (Jon Siwek) + + - Some scripts used wrong SSH module/namespace scoping on events. + - Fix outdated notice documentation related to SSH password guessing. + - Add a unit test for SSH pasword guessing notice. + +2.3-635 | 2015-03-30 11:02:45 -0500 + + * Fix outdated documentation unit tests. (Jon Siwek) + +2.3-634 | 2015-03-30 10:22:45 -0500 + + * Add a canonifier to a unit test's output. (Jon Siwek) + +2.3-633 | 2015-03-25 18:32:59 -0700 + + * Log::write in signature framework was missing timestamp. + (Andrew Benson/Michel Laterman) + +2.3-631 | 2015-03-25 11:03:12 -0700 + + * New SSH analyzer. (Vlad Grigorescu) + +2.3-600 | 2015-03-25 10:23:46 -0700 + + * Add defensive checks in code to calculate log rotation intervals. + (Pete Nelson). + +2.3-597 | 2015-03-23 12:50:04 -0700 + + * DTLS analyzer. (Johanna Amann) + + * Implement correct parsing of TLS record fragmentation. (Johanna + Amann) + +2.3-582 | 2015-03-23 11:34:25 -0700 + + * BIT-1313: In debug builds, "bro -B " now supports "all" and + "help" for "". "all" enables all debug streams. "help" prints a + list of available debug streams. (John Donnelly/Robin Sommer). + + * BIT-1324: Allow logging filters to inherit default path from + stream. This allows the path for the default filter to be + specified explicitly through $path="..." when creating a stream. + Adapted the existing Log::create_stream calls to explicitly + specify a path value. (Jon Siwek) + + * BIT-1199: Change the way the input framework deals with values it + cannot convert into BroVals, raising error messages instead of + aborting execution. (Johanna Amann) + + * BIT-788: Use DNS QR field to better identify flow direction. (Jon + Siwek) + +2.3-572 | 2015-03-23 13:04:53 -0500 + + * BIT-1226: Fix an example in quickstart docs. (Jon siwek) + +2.3-570 | 2015-03-23 09:51:20 -0500 + + * Correct a spelling error (Daniel Thayer) + + * Improvement to SSL analyzer failure mode. (Johanna Amann) + +2.3-565 | 2015-03-20 16:27:41 -0500 + + * BIT-978: Improve documentation of 'for' loop iterator invalidation. + (Jon Siwek) + +2.3-564 | 2015-03-20 11:12:02 -0500 + + * BIT-725: Remove "unmatched_HTTP_reply" weird. (Jon Siwek) + +2.3-562 | 2015-03-20 10:31:02 -0500 + + * BIT-1207: Add unit test to catch breaking changes to local.bro + (Jon Siwek) + + * Fix failing sqlite leak test (Johanna Amann) + +2.3-560 | 2015-03-19 13:17:39 -0500 + + * BIT-1255: Increase default values of + "tcp_max_above_hole_without_any_acks" and "tcp_max_initial_window" + from 4096 to 16384 bytes. (Jon Siwek) + +2.3-559 | 2015-03-19 12:14:33 -0500 + + * BIT-849: turn SMTP reporter warnings into weirds, + "smtp_nested_mail_transaction" and "smtp_unmatched_end_of_data". + (Jon Siwek) + +2.3-558 | 2015-03-18 22:50:55 -0400 + + * DNS: Log the type number for the DNS_RR_unknown_type weird. (Vlad Grigorescu) + +2.3-555 | 2015-03-17 15:57:13 -0700 + + * Splitting test-all Makefile target into Bro tests and test-aux. + (Robin Sommer) + +2.3-554 | 2015-03-17 15:40:39 -0700 + + * Deprecate &rotate_interval, &rotate_size, &encrypt. Addresses + BIT-1305. (Jon Siwek) + +2.3-549 | 2015-03-17 09:12:18 -0700 + + * BIT-1077: Fix HTTP::log_server_header_names. Before, it just + re-logged fields from the client side. (Jon Siwek) + +2.3-547 | 2015-03-17 09:07:51 -0700 + + * Update certificate validation script to cache valid intermediate + chains that it encounters on the wire and use those to try to + validate chains that might be missing intermediate certificates. + (Johanna Amann) + +2.3-541 | 2015-03-13 15:44:08 -0500 + + * Make INSTALL a symlink to doc/install/install.rst (Jon siwek) + + * Fix Broxygen coverage. (Jon Siwek) + +2.3-539 | 2015-03-13 14:19:27 -0500 + + * BIT-1335: Include timestamp in default extracted file names. + And add a policy script to extract all files. (Jon Siwek) + + * BIT-1311: Identify GRE tunnels as Tunnel::GRE, not Tunnel::IP. + (Jon Siwek) + + * BIT-1309: Add Connection class getter methods for flow labels. + (Jon Siwek) + +2.3-536 | 2015-03-12 16:16:24 -0500 + + * Fix Broker leak tests. (Jon Siwek) + +2.3-534 | 2015-03-12 10:59:49 -0500 + + * Update NEWS file. (Jon Siwek) + +2.3-533 | 2015-03-12 10:18:53 -0500 + + * Give broker python bindings default install path within --prefix. + (Jon Siwek) + +2.3-530 | 2015-03-10 13:22:39 -0500 + + * Fix broker data stores in absence of --enable-debug. (Jon Siwek) + +2.3-529 | 2015-03-09 13:14:27 -0500 + + * Fix format specifier in SSL protocol violation. (Jon Siwek) + +2.3-526 | 2015-03-06 12:48:49 -0600 + + * Fix build warnings, clarify broker requirements, update submodule. + (Jon Siwek) + + * Rename comm/ directories to broker/ (Jon Siwek) + + * Rename broker-related namespaces. (Jon Siwek) + + * Improve remote logging via broker by only sending fields w/ &log. + (Jon Siwek) + + * Disable a stream's remote logging via broker if it fails. (Jon Siwek) + + * Improve some broker communication unit tests. (Jon Siwek) + +2.3-518 | 2015-03-04 13:13:50 -0800 + + * Add bytes_recvd to stats.log recording the number of bytes + received, according to packet headers. (Mike Smiley) + +2.3-516 | 2015-03-04 12:30:06 -0800 + + * Extract most specific Common Name from SSL certificates (Johanna + Amann) + + * Send CN and SAN fields of SSL certificates to the Intel framework. + (Johanna Amann) + +2.3-511 | 2015-03-02 18:07:17 -0800 + + * Changes to plugin meta hooks for function calls. (Gilbert Clark) + + - Add frame argument. + + - Change return value to tuple unambigiously whether hook + returned a result. + +2.3-493 | 2015-03-02 17:17:32 -0800 + + * Extend the SSL weak-keys policy file to also alert when + encountering SSL connections with old versions as well as unsafe + cipher suites. (Johanna Amann) + + * Make the notice suppression handling of other SSL policy files a + tad more robust. (Johanna Amann) + +2.3-491 | 2015-03-02 17:12:56 -0800 + + * Updating docs for recent addition of local_resp. (Robin Sommer) + +2.3-489 | 2015-03-02 15:29:30 -0800 + + * Integrate Broker, Bro's new communication library. (Jon Siwek) + + See aux/broker/README for more information on Broker, and + doc/frameworks/comm.rst for the corresponding Bro script API. + + Broker support is by default off for now; it can be enabled at + configure time with --enable-broker. It requires CAF + (https://github.com/actor-framework/actor-framework); for now iot + needs CAF's "develop" branch. Broker also requires a C++11 + compiler. + + Broker will become a mandatory dependency in future Bro versions. + + * Add --enable-c++11 configure flag to compile Bro's source code in + C++11 mode with a corresponding compiler. (Jon Siwek) + +2.3-451 | 2015-02-24 16:37:08 -0800 + + * Updating submodule(s). + +2.3-448 | 2015-02-23 16:58:10 -0800 + + * Updating NEWS. (Robin Sommer) + +2.3-447 | 2015-02-23 16:28:30 -0800 + + * Fix potential crash in logging framework when deserializing + WriterInfo from remote. where config is present. Testcase crashes + on unpatched versions of Bro. (Aaron Eppert) + + * Fix wrong value test in WriterBackend. (Aaron Eppert) + +2.3-442 | 2015-02-23 13:29:30 -0800 + + * Add a "local_resp" field to conn.log, along the lines of the + existing "local_orig". (Mike Smiley) + +2.3-440 | 2015-02-23 11:39:17 -0600 + + * Updating plugin docs to recent changes. (Robin Sommer) + + * Updating plugin tests to recent changes. (Robin Sommer) + + * Making plugin names case-insensitive for some internal comparisions. + Makes plugin system more tolerant against spelling inconsistencies + are hard to catch otherwise. (Robin Sommer) + + * Explicitly removing some old scripts on install that have moved + into plugins to prevent them causing confusion. (Robin Sommer) + + * BIT-1312: Removing setting installation plugin path from + bro-path-dev.sh. Also, adding to existing BRO_PLUGIN_PATH rather + than replacing. (Robin Sommer) + + * Creating the installation directory for plugins at install time. + (Robin Sommer) + +2.3-427 | 2015-02-20 13:49:33 -0800 + + * Removing dependency on PCAP_NETMASK_UNKNOWN to compile with + libpcap < 1.1.1. (Robin Sommer) + +2.3-426 | 2015-02-20 12:45:51 -0800 + + * Add 'while' statement to Bro language. Really. (Jon Siwek) + +2.3-424 | 2015-02-20 12:39:10 -0800 + + * Add the ability to remove surrounding braces from the JSON + formatter. (Seth Hall) + +2.3-419 | 2015-02-13 09:10:44 -0600 + + * BIT-1011: Update the SOCKS analyzer to support user/pass login. + (Nicolas Retrain, Seth Hall, Jon Siwek) + + - Add a new field to socks.log: "password". + - Two new events: "socks_login_userpass_request" and + "socks_login_userpass_reply". + - Two new weirds for unsupported SOCKS authentication method or + version. + - A new test for authenticated socks traffic. + +2.3-416 | 2015-02-12 12:18:42 -0600 + + * Submodule update - newest sqlite version (Johanna Amann) + + * Fix use of deprecated gperftools headers. (Jon Siwek) + +2.3-413 | 2015-02-08 18:23:05 -0800 + + * Fixing analyzer tag types for some Files::* functions. (Robin Sommer) + + * Changing load order for plugin scripts. (Robin Sommer) + +2.3-411 | 2015-02-05 10:05:48 -0600 + + * Fix file analysis of files with total size below the bof_buffer size + never delivering content to stream analyzers. (Seth Hall) + + * Add/fix log fields in x509 diff canonifier. (Jon Siwek) + + * "id" not defined for debug code when using -DPROFILE_BRO_FUNCTIONS + (Mike Smiley) + +2.3-406 | 2015-02-03 17:02:45 -0600 + + * Add x509 canonifier to a unit test. (Jon Siwek) + +2.3-405 | 2015-02-02 11:14:24 -0600 + + * Fix memory leak in new split_string* functions. (Jon Siwek) + +2.3-404 | 2015-01-30 14:23:27 -0800 + + * Update documentation (broken links, outdated tests). (Jon Siwek) + + * Deprecate split* family of BIFs. (Jon Siwek) + + These functions are now deprecated in favor of alternative versions that + return a vector of strings rather than a table of strings. + + Deprecated functions: + + - split: use split_string instead. + - split1: use split_string1 instead. + - split_all: use split_string_all instead. + - split_n: use split_string_n instead. + - cat_string_array: see join_string_vec instead. + - cat_string_array_n: see join_string_vec instead. + - join_string_array: see join_string_vec instead. + - sort_string_array: use sort instead instead. + - find_ip_addresses: use extract_ip_addresses instead. + + Changed functions: + + - has_valid_octets: uses a string_vec parameter instead of string_array. + + Addresses BIT-924. + + * Add a new attribute: &deprecated. While scripts are parsed, a + warning is raised for each usage of an identifier marked as + &deprecated. This also works for BIFs. Addresses BIT-924, + BIT-757. (Jon Siwek) + +2.3-397 | 2015-01-27 10:13:10 -0600 + + * Handle guess_lexer exceptions in pygments reST directive (Jon Siwek) + +2.3-396 | 2015-01-23 10:49:15 -0600 + + * DNP3: fix reachable assertion and buffer over-read/overflow. + CVE number pending. (Travis Emmert, Jon Siwek) + + * Update binpac: Fix potential out-of-bounds memory reads in generated + code. CVE-2014-9586. (John Villamil and Chris Rohlf - Yahoo + Paranoids, Jon Siwek) + + * Fixing (harmless) Coverity warning. (Robin Sommer) + +2.3-392 | 2015-01-15 09:44:15 -0800 + + * Small changes to EC curve names in a newer draft. (Johanna Amann) + +2.3-390 | 2015-01-14 13:27:34 -0800 + + * Updating MySQL analyses. (Vlad Grigorescu) + - Use a boolean success instead of a result string. + - Change the affected_rows response detail string to a "rows" count. + - Fix the state tracking to log incomplete command. + + * Extend DNP3 to support communication over UDP. (Hui Lin) + + * Fix a bug in DNP3 determining the length of an object in some + cases. (Hui Lin) + +2.3-376 | 2015-01-12 09:38:10 -0600 + + * Improve documentation for connection_established event. (Jon Siwek) + +2.3-375 | 2015-01-08 13:10:09 -0600 + + * Increase minimum required CMake version to 2.8. (Jon Siwek) + +2.3-374 | 2015-01-07 10:03:17 -0600 + + * Improve documentation of the Intelligence Framework. (Daniel Thayer) + +2.3-371 | 2015-01-06 09:58:09 -0600 + + * Update/improve file mime type identification. (Seth Hall) + + - Change to the default BOF buffer size to 3000 (was 1024). + + - Reorganized MS signatures into a separate file. + + - Remove all of the x-c detections. Nearly all false positives. + + - Improve TAR detections, removing old, back up TAR detections. + + - Remove one of the x-elc detections that was too loose + and caused many false positives. + + - Improved lots of the signatures and added new ones. (Seth Hall) + + * Add support for file reassembly in the file analysis framework + (Seth Hall, Jon Siwek). + + - The reassembly behavior can be modified per-file by enabling or + disabling the reassembler and/or modifying the size of the + reassembly buffer. + + - Changed the file extraction analyzer to use stream-wise input to + avoid issues with the chunk-wise approach not immediately + triggering the file_new event due to mime-type detection delay. + Before, early chunks frequently ended up lost. Extraction also + will now explicitly NUL-fill gaps in the file instead of + implicitly relying on pwrite to do it. + +2.3-349 | 2015-01-05 15:21:13 -0600 + + * Fix race condition in unified2 file analyzer startup. (Jon siwek) + +2.3-348 | 2014-12-31 09:19:34 -0800 + + * Changing Makefile's test-all to run test-all for broctl, which now + executes trace-summary tests as well. (Robin Sommer) + +2.3-345 | 2014-12-31 09:06:15 -0800 + + * Correct a typo in the Notice framework doc. (Daniel Thayer) + +2.3-343 | 2014-12-12 12:43:46 -0800 + + * Fix PIA packet replay to deliver copy of IP header. This prevented + one from writing a packet-wise analyzer that needs access to IP + headers and can be attached to a connection via signature match. + Addresses BIT-1298 (Jon Siwek) + +2.3-338 | 2014-12-08 13:56:19 -0800 + + * Add man page for Bro. (Raúl Benencia) + + * Updating doc baselines. (Robin Sommer) + +2.3-334 | 2014-12-03 14:22:07 -0800 + + * Fix compound assignment to require proper L-value. Addresses + BIT-1295. (Jon Siwek) + +2.3-332 | 2014-12-03 14:14:11 -0800 + + * Make using local IDs in @if directives an error. Addresses + BIT-1296. (Jon Siwek) + +2.3-330 | 2014-12-03 14:10:39 -0800 + + * Fix some "make doc" warnings and update some doc tests. (Daniel + Thayer) + +2.3-328 | 2014-12-02 08:13:10 -0500 + + * Update windows-version-detection.bro to add support for + Windows 10. (Michal Purzynski) + +2.3-326 | 2014-12-01 12:10:27 -0600 + + * BIFScanner: fix invalid characters in generated preprocessor macros. + (Hilko Bengen) + + * BIT-1294: fix exec.bro from mutating Input::end_of_data event + parameters. (Johanna Amann) + + * Add/invoke "distclean" for testing directories. (Raúl Benencia) + + * Delete prebuilt python bytecode files from git. (Jon Siwek) + + * Add Windows detection based on CryptoAPI HTTP traffic as a software + framework policy script. (Vlad Grigorescu) + +2.3-316 | 2014-11-25 17:35:06 -0800 + + * Make the SSL analyzer skip further processing once encountering + situations which are very probably non-recoverable. (Johanna + Amann) + +2.3-313 | 2014-11-25 14:27:07 -0800 + + * Make SSL v2 protocol tests more strict. In its former state they + triggered on http traffic over port 443 sometimes. Found by Michał + Purzyński. (Johanna Amann) + + * Fix X509 analyzer to correctly return ECDSA as the key_type for + ECDSA certs. Bug found by Michał Purzyński. (Johanna Amann) + +2.3-310 | 2014-11-19 10:56:59 -0600 + + * Disable verbose bison output. (Jon Siwek) + +2.3-309 | 2014-11-18 12:17:53 -0800 + + * New decompose_uri() function in base/utils/urls that splits a URI + into its pieces. (Anthony Kasza). + +2.3-305 | 2014-11-18 11:09:04 -0800 + + * Improve coercion of &default expressions. Addresses BIT-1288. (Jon + Siwek) + +2.3-303 | 2014-11-18 10:53:04 -0800 + + * For DH key exchanges, use p as the parameter for weak key + exchanges. (Johanna Amann) + 2.3-301 | 2014-11-11 13:47:27 -0800 * Add builtin function enum_to_int() that converts an enum into a @@ -15,6 +568,11 @@ logged into mysql.log. Supports protocol versions 9 and 10. (Vlad Grigorescu) +2.3-280 | 2014-11-05 09:46:33 -0500 + + * Add Windows detection based on CryptoAPI HTTP traffic as a + software framework policy script. (Vlad Grigorescu) + 2.3-278 | 2014-11-03 18:55:18 -0800 * Add new curves from draft-ietf-tls-negotiated-ff-dhe to SSL diff --git a/CMakeLists.txt b/CMakeLists.txt index 22d63a89d5..8f60ab95ad 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -2,7 +2,7 @@ project(Bro C CXX) # When changing the minimum version here, also adapt # aux/bro-aux/plugin-support/skeleton/CMakeLists.txt -cmake_minimum_required(VERSION 2.6.3 FATAL_ERROR) +cmake_minimum_required(VERSION 2.8 FATAL_ERROR) include(cmake/CommonCMakeConfig.cmake) @@ -15,6 +15,11 @@ if (NOT BRO_SCRIPT_INSTALL_PATH) set(BRO_SCRIPT_INSTALL_PATH ${BRO_ROOT_DIR}/share/bro) endif () +if (NOT BRO_MAN_INSTALL_PATH) + # set the default Bro man page installation path (user did not specify one) + set(BRO_MAN_INSTALL_PATH ${BRO_ROOT_DIR}/share/man) +endif () + # sanitize the Bro script install directory into an absolute path # (CMake is confused by ~ as a representation of home directory) get_filename_component(BRO_SCRIPT_INSTALL_PATH ${BRO_SCRIPT_INSTALL_PATH} @@ -26,12 +31,12 @@ configure_file(bro-path-dev.in ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev) file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.sh "export BROPATH=`${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" - "export BRO_PLUGIN_PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src:${BRO_PLUGIN_INSTALL_PATH}\"\n" + "export BRO_PLUGIN_PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src\":${BRO_PLUGIN_PATH}\n" "export PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n") file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.csh "setenv BROPATH `${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" - "setenv BRO_PLUGIN_PATH \"${CMAKE_CURRENT_BINARY_DIR}/src:${BRO_PLUGIN_INSTALL_PATH}\"\n" + "setenv BRO_PLUGIN_PATH \"${CMAKE_CURRENT_BINARY_DIR}/src\":${BRO_PLUGIN_PATH}\n" "setenv PATH \"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n") file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/VERSION" VERSION LIMIT_COUNT 1) @@ -172,9 +177,21 @@ include_directories(${CMAKE_CURRENT_BINARY_DIR}) ######################################################################## ## Recurse on sub-directories +if ( ENABLE_CXX11 ) + include(RequireCXX11) +endif () + +if ( ENABLE_BROKER ) + add_subdirectory(aux/broker) + set(brodeps ${brodeps} broker) + add_definitions(-DENABLE_BROKER) + include_directories(BEFORE ${CMAKE_CURRENT_SOURCE_DIR}/aux/broker) +endif () + add_subdirectory(src) add_subdirectory(scripts) add_subdirectory(doc) +add_subdirectory(man) include(CheckOptionalBuildSources) @@ -218,6 +235,7 @@ message( "\nCXXFLAGS: ${CMAKE_CXX_FLAGS} ${CMAKE_CXX_FLAGS_${BuildType}}" "\nCPP: ${CMAKE_CXX_COMPILER}" "\n" + "\nBroker: ${ENABLE_BROKER}" "\nBroccoli: ${INSTALL_BROCCOLI}" "\nBroctl: ${INSTALL_BROCTL}" "\nAux. Tools: ${INSTALL_AUX_TOOLS}" diff --git a/INSTALL b/INSTALL deleted file mode 100644 index 385dac93df..0000000000 --- a/INSTALL +++ /dev/null @@ -1,3 +0,0 @@ - -See doc/install/install.rst for installation instructions. - diff --git a/INSTALL b/INSTALL new file mode 120000 index 0000000000..95fcc60eda --- /dev/null +++ b/INSTALL @@ -0,0 +1 @@ +doc/install/install.rst \ No newline at end of file diff --git a/Makefile b/Makefile index 49d9a6173c..3efddc4dbc 100644 --- a/Makefile +++ b/Makefile @@ -48,15 +48,18 @@ bindist: distclean: rm -rf $(BUILD) + $(MAKE) -C testing $@ test: - @( cd testing && make ) + -@( cd testing && make ) -test-all: test - test -d aux/broctl && ( cd aux/broctl && make test ) - test -d aux/btest && ( cd aux/btest && make test ) - test -d aux/bro-aux && ( cd aux/bro-aux && make test ) - test -d aux/plugins && ( cd aux/plugins && make test-all ) +test-aux: + -test -d aux/broctl && ( cd aux/broctl && make test-all ) + -test -d aux/btest && ( cd aux/btest && make test ) + -test -d aux/bro-aux && ( cd aux/bro-aux && make test ) + -test -d aux/plugins && ( cd aux/plugins && make test-all ) + +test-all: test test-aux configured: @test -d $(BUILD) || ( echo "Error: No build/ directory found. Did you run configure?" && exit 1 ) diff --git a/NEWS b/NEWS index 6de9bd8f3e..7129b293d5 100644 --- a/NEWS +++ b/NEWS @@ -28,11 +28,134 @@ New Functionality - Bro now has supoprt for the MySQL wire protocol. Activity gets logged into mysql.log. +- Bro now features a completely rewritten, enhanced SSH analyzer. A lot + more information about SSH sessions is logged. The analyzer is able to + determine if logins failed or succeeded in most circumstances. + +- Bro's file analysis now supports reassembly of files that are not + transferred/seen sequentially. + +- Bro's scripting language now has a ``while`` statement:: + + while ( i < 5 ) + print ++i; + + ``next`` and ``break`` can be used inside the loop's body just like + with ``for`` loops. + +- Bro now integrates Broker, it's new communication library. See + aux/broker/README for more information on Broker, and + doc/frameworks/comm.rst for the corresponding Bro script API. + + TODO: Extend with some more information on Broker. + + Broker support is by default off for now; it can be enabled at + configure time with --enable-broker. It requires CAF version 0.13+ + (https://github.com/actor-framework/actor-framework) as well as a + C++11 compiler (e.g. GCC 4.8+ or Clang 3.3+). + + Broker will become a mandatory dependency in future Bro versions. + +- Add --enable-c++11 configure flag to compile Bro's source code in + C++11 mode with a corresponding compiler. Note that 2.4 will be the + last version of Bro that compiles without C++11 support. + +- The SSL analysis now alert when encountering SSL connections with + old protocol versions or unsafe cipher suites. + +- [TODO] Add new BroControl features. + +- A new icmp_sent_payload event provides access to ICMP payload. + +- Bro now parses DTLS traffic. + +- Bro now has an RDP analyzer. + Changed Functionality --------------------- - bro-cut has been rewritten in C, and is hence much faster. +- File analysis + + * Removed ``fa_file`` record's ``mime_type`` and ``mime_types`` + fields. The events ``file_mime_type`` and ``file_mime_types`` + have been added which contain the same information. The + ``mime_type`` field of ``Files::Info`` also still has this info. + + * The earliest point that new mime type information is available is + in the ``file_mime_type`` event which comes after the ``file_new`` + and ``file_over_new_connection`` events. Scripts which inspected + mime type info within those events will need to be adapted. + + * Removed ``Files::add_analyzers_for_mime_type`` function. + + * Removed ``offset`` parameter of the ``file_extraction_limit`` + event. Since file extraction now internally depends on file + reassembly for non-sequential files, "offset" can be obtained + with other information already available -- adding together + ``seen_bytes`` and ``missed_bytes`` fields of the ``fa_file`` + record gives the how many bytes have been written so far (i.e. + the "offset"). + +- has_valid_octets: now uses a string_vec parameter instead of + string_array. + +- conn.log gained a new field local_resp that works like local_orig, + just for the responder address of the connection. + +- GRE tunnels are now identified as ``Tunnel::GRE`` instead of + ``Tunnel::IP``. + +- The default name for extracted files changed from extract-protocol-id + to extract-timestamp-protocol-id. + +- [TODO] Add changed BroControl features. + +- The weird named "unmatched_HTTP_reply" has been removed since it can + be detected at the script-layer and is handled correctly by the + default HTTP scripts. + +- When adding a logging filter to a stream, the filter can now inherit + a default ``path`` field from the associated ``Log::Stream`` record. + +- When adding a logging filter to a stream, the + ``Log::default_path_func`` is now only automatically added to the + filter if it has neither a ``path`` nor a ``path_func`` already + explicitly set. Before, the default path function would always be set + for all filters which didn't specify their own ``path_func``. + +- TODO: what SSH events got changed or removed? + +Deprecated Functionality +------------------------ + +- The split* family of functions are to be replaced with alternate + versions that return a vector of strings rather than a table of + strings. This also allows deprecation for some related string + concatenation/extraction functions. Note that the new functions use + 0-based indexing, rather than 1-based. + + The full list of now deprecation functions is: + + * split: use split_string instead. + + * split1: use split_string1 instead. + + * split_all: use split_string_all instead. + + * split_n: use split_string_n instead. + + * cat_string_array: see join_string_vec instead. + + * cat_string_array_n: see join_string_vec instead. + + * join_string_array: see join_string_vec instead. + + * sort_string_array: use sort instead. + + * find_ip_addresses: use extract_ip_addresses instead. + Bro 2.3 ======= diff --git a/VERSION b/VERSION index 5fa0f4af0c..937b335f4d 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.3-301 +2.3-680 diff --git a/aux/binpac b/aux/binpac index 77a86591dc..ab50e5115b 160000 --- a/aux/binpac +++ b/aux/binpac @@ -1 +1 @@ -Subproject commit 77a86591dcf89d7252d3676d3f1199d6c927d073 +Subproject commit ab50e5115bc0d217552a63f15382e45ed608f5fc diff --git a/aux/bro-aux b/aux/bro-aux index 977654dc51..462e300bf9 160000 --- a/aux/bro-aux +++ b/aux/bro-aux @@ -1 +1 @@ -Subproject commit 977654dc51ab08a2afde32241f108cdb4a581d8f +Subproject commit 462e300bf9c37dcc39b70a4c2d89d19f7351c804 diff --git a/aux/broccoli b/aux/broccoli index acb8fbe8e7..45276b39a9 160000 --- a/aux/broccoli +++ b/aux/broccoli @@ -1 +1 @@ -Subproject commit acb8fbe8e7bc6ace5135fb73dca8e29432cdc1ca +Subproject commit 45276b39a946d70095c983753cd321ad07dcf285 diff --git a/aux/broctl b/aux/broctl index 39e865dec9..e864a0949e 160000 --- a/aux/broctl +++ b/aux/broctl @@ -1 +1 @@ -Subproject commit 39e865dec9611b9b53b609cbc8df519cebae0a1e +Subproject commit e864a0949e52a797f4000194b5c2980cf3618deb diff --git a/aux/broker b/aux/broker new file mode 160000 index 0000000000..1a2ab9ee7c --- /dev/null +++ b/aux/broker @@ -0,0 +1 @@ +Subproject commit 1a2ab9ee7c80ca905e86a2a11283e7c0477341a9 diff --git a/aux/btest b/aux/btest index d67d89aaee..d69df586c9 160000 --- a/aux/btest +++ b/aux/btest @@ -1 +1 @@ -Subproject commit d67d89aaee32ad5edb9068db55d1310c2f36970a +Subproject commit d69df586c91531db0c3abe838b10a429dda4fa87 diff --git a/aux/plugins b/aux/plugins index ad600b5bdc..7a14085394 160000 --- a/aux/plugins +++ b/aux/plugins @@ -1 +1 @@ -Subproject commit ad600b5bdcd56a2723e323c0f2c8e1708956ca4f +Subproject commit 7a14085394e54a950e477eb4fafb3827ff8dbdc3 diff --git a/cmake b/cmake index 1316c07f70..2fd35ab6a6 160000 --- a/cmake +++ b/cmake @@ -1 +1 @@ -Subproject commit 1316c07f7059647b6c4a496ea36e4b83bb5d8f0f +Subproject commit 2fd35ab6a6245a005828c32f0aa87eb21698c054 diff --git a/configure b/configure index 2b1c568b26..b139ee2bec 100755 --- a/configure +++ b/configure @@ -41,6 +41,9 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --enable-perftools-debug use Google's perftools for debugging --enable-jemalloc link against jemalloc --enable-ruby build ruby bindings for broccoli (deprecated) + --enable-c++11 build using the C++11 standard + --enable-broker enable use of the Broker communication library + (requires C++ Actor Framework and C++11) --disable-broccoli don't build or install the Broccoli library --disable-broctl don't install Broctl --disable-auxtools don't build or install auxiliary tools @@ -55,6 +58,8 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --with-flex=PATH path to flex executable --with-bison=PATH path to bison executable --with-perl=PATH path to perl executable + --with-libcaf=PATH path to C++ Actor Framework installation + (a required Broker dependency) Optional Packages in Non-Standard Locations: --with-geoip=PATH path to the libGeoIP install root @@ -67,6 +72,8 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --with-ruby-lib=PATH path to ruby library --with-ruby-inc=PATH path to ruby headers --with-swig=PATH path to SWIG executable + --with-rocksdb=PATH path to RocksDB installation + (an optional Broker dependency) Packaging Options (for developers): --binary-package toggle special logic for binary packaging @@ -142,6 +149,10 @@ while [ $# -ne 0 ]; do append_cache_entry CMAKE_INSTALL_PREFIX PATH $optarg append_cache_entry BRO_ROOT_DIR PATH $optarg append_cache_entry PY_MOD_INSTALL_DIR PATH $optarg/lib/broctl + + if [ -n "$user_enabled_broker" ]; then + append_cache_entry BROKER_PYTHON_HOME PATH $prefix + fi ;; --scriptdir=*) append_cache_entry BRO_SCRIPT_INSTALL_PATH STRING $optarg @@ -176,6 +187,15 @@ while [ $# -ne 0 ]; do --enable-jemalloc) append_cache_entry ENABLE_JEMALLOC BOOL true ;; + --enable-c++11) + append_cache_entry ENABLE_CXX11 BOOL true + ;; + --enable-broker) + append_cache_entry ENABLE_CXX11 BOOL true + append_cache_entry ENABLE_BROKER BOOL true + append_cache_entry BROKER_PYTHON_HOME PATH $prefix + user_enabled_broker="true" + ;; --disable-broccoli) append_cache_entry INSTALL_BROCCOLI BOOL false ;; @@ -248,6 +268,12 @@ while [ $# -ne 0 ]; do --with-swig=*) append_cache_entry SWIG_EXECUTABLE PATH $optarg ;; + --with-libcaf=*) + append_cache_entry LIBCAF_ROOT_DIR PATH $optarg + ;; + --with-rocksdb=*) + append_cache_entry ROCKSDB_ROOT_DIR PATH $optarg + ;; --binary-package) append_cache_entry BINARY_PACKAGING_MODE BOOL true ;; diff --git a/doc/components/broker/README.rst b/doc/components/broker/README.rst new file mode 120000 index 0000000000..eafa3b8e77 --- /dev/null +++ b/doc/components/broker/README.rst @@ -0,0 +1 @@ +../../../aux/broker/README \ No newline at end of file diff --git a/doc/components/broker/broker-manual.rst b/doc/components/broker/broker-manual.rst new file mode 120000 index 0000000000..90bf8f0833 --- /dev/null +++ b/doc/components/broker/broker-manual.rst @@ -0,0 +1 @@ +../../../aux/broker/broker-manual.rst \ No newline at end of file diff --git a/doc/components/index.rst b/doc/components/index.rst index fe05f13683..c1feda4a61 100644 --- a/doc/components/index.rst +++ b/doc/components/index.rst @@ -17,6 +17,8 @@ current, independent component releases. Broccoli - User Manual Broccoli Python Bindings Broccoli Ruby Bindings + Broker - Bro's (New) Messaging Library (README) + Broker - User Manual BroControl - Interactive Bro management shell Bro-Aux - Small auxiliary tools for Bro BTest - A unit testing framework diff --git a/doc/devel/plugins.rst b/doc/devel/plugins.rst index c703345891..5e488cfe01 100644 --- a/doc/devel/plugins.rst +++ b/doc/devel/plugins.rst @@ -3,7 +3,7 @@ Writing Bro Plugins =================== -Bro is internally moving to a plugin structure that enables extending +Bro internally provides plugin API that enables extending the system dynamically, without modifying the core code base. That way custom code remains self-contained and can be maintained, compiled, and installed independently. Currently, plugins can add the following @@ -42,18 +42,17 @@ certain structure. To get started, Bro's distribution provides a helper script ``aux/bro-aux/plugin-support/init-plugin`` that creates a skeleton plugin that can then be customized. Let's use that:: - # mkdir rot13-plugin - # cd rot13-plugin - # init-plugin Demo Rot13 + # init-plugin ./rot13-plugin Demo Rot13 -As you can see the script takes two arguments. The first is a -namespace the plugin will live in, and the second a descriptive name -for the plugin itself. Bro uses the combination of the two to identify -a plugin. The namespace serves to avoid naming conflicts between -plugins written by independent developers; pick, e.g., the name of -your organisation. The namespace ``Bro`` is reserved for functionality -distributed by the Bro Project. In our example, the plugin will be -called ``Demo::Rot13``. +As you can see, the script takes three arguments. The first is a +directory inside which the plugin skeleton will be created. The second +is the namespace the plugin will live in, and the third is a descriptive +name for the plugin itself relative to the namespace. Bro uses the +combination of namespace and name to identify a plugin. The namespace +serves to avoid naming conflicts between plugins written by independent +developers; pick, e.g., the name of your organisation. The namespace +``Bro`` is reserved for functionality distributed by the Bro Project. In +our example, the plugin will be called ``Demo::Rot13``. The ``init-plugin`` script puts a number of files in place. The full layout is described later. For now, all we need is @@ -61,7 +60,7 @@ layout is described later. For now, all we need is there as follows:: # cat src/rot13.bif - module CaesarCipher; + module Demo; function rot13%(s: string%) : string %{ @@ -82,18 +81,22 @@ The syntax of this file is just like any other ``*.bif`` file; we won't go into it here. Now we can already compile our plugin, we just need to tell the -configure script put in place by ``init-plugin`` where the Bro source -tree is located (Bro needs to have been built there first):: +configure script that ``init-plugin`` put in place where the Bro +source tree is located (Bro needs to have been built there first):: + # cd rot13-plugin # ./configure --bro-dist=/path/to/bro/dist && make [... cmake output ...] -Now our ``rot13-plugin`` directory has everything that it needs -for Bro to recognize it as a dynamic plugin. Once we point Bro to it, -it will pull it in automatically, as we can check with the ``-N`` +This builds the plugin in a subdirectory ``build/``. In fact, that +subdirectory *becomes* the plugin: when ``make`` finishes, ``build/`` +has everything it needs for Bro to recognize it as a dynamic plugin. + +Let's try that. Once we point Bro to the ``build/`` directory, it will +pull in our new plugin automatically, as we can check with the ``-N`` option:: - # export BRO_PLUGIN_PATH=/path/to/rot13-plugin + # export BRO_PLUGIN_PATH=/path/to/rot13-plugin/build # bro -N [...] Plugin: Demo::Rot13 - (dynamic, version 1) @@ -127,12 +130,12 @@ more verbose option ``-NN``:: # bro -NN [...] Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) - [Function] CaesarCipher::rot13 + [Function] Demo::rot13 [...] There's our function. Now let's use it:: - # bro -e 'print CaesarCipher::rot13("Hello")' + # bro -e 'print Demo::rot13("Hello")' Uryyb It works. We next install the plugin along with Bro itself, so that it @@ -141,36 +144,40 @@ environment variable. If we first unset the variable, the function will no longer be available:: # unset BRO_PLUGIN_PATH - # bro -e 'print CaesarCipher::rot13("Hello")' - error in , line 1: unknown identifier CaesarCipher::rot13, at or near "CaesarCipher::rot13" + # bro -e 'print Demo::rot13("Hello")' + error in , line 1: unknown identifier Demo::rot13, at or near "Demo::rot13" Once we install it, it works again:: # make install - # bro -e 'print CaesarCipher::rot13("Hello")' + # bro -e 'print Demo::rot13("Hello")' Uryyb The installed version went into ``/lib/bro/plugins/Demo_Rot13``. -We can distribute the plugin in either source or binary form by using -the Makefile's ``sdist`` and ``bdist`` target, respectively. Both -create corrsponding tarballs:: +One can distribute the plugin independently of Bro for others to use. +To distribute in source form, just remove the ``build/`` (``make +distclean`` does that) and then tar up the whole ``rot13-plugin/`` +directory. Others then follow the same process as above after +unpacking. To distribute the plugin in binary form, the build process +conveniently creates a corresponding tarball in ``build/dist/``. In +this case, it's called ``Demo_Rot13-0.1.tar.gz``, with the version +number coming out of the ``VERSION`` file that ``init-plugin`` put +into place. The binary tarball has everything needed to run the +plugin, but no further source files. Optionally, one can include +further files by specifying them in the plugin's ``CMakeLists.txt`` +through the ``bro_plugin_dist_files`` macro; the skeleton does that +for ``README``, ``VERSION``, ``CHANGES``, and ``COPYING``. To use the +plugin through the binary tarball, just unpack it and point +``BRO_PLUGIN_PATH`` there; or copy it into +``/lib/bro/plugins/`` directly. - # make sdist - [...] - Source distribution in build/sdist/Demo_Rot13.tar.gz - - # make bdist - [...] - Binary distribution in build/Demo_Rot13-darwin-x86_64.tar.gz - -The source archive will contain everything in the plugin directory -except any generated files. The binary archive will contain anything -needed to install and run the plugin, i.e., just what ``make install`` -puts into place as well. As the binary distribution is -platform-dependent, its name includes the OS and architecture the -plugin was built on. +Before distributing your plugin, you should edit some of the meta +files that ``init-plugin`` puts in place. Edit ``README`` and +``VERSION``, and update ``CHANGES`` when you make changes. Also put a +license file in place as ``COPYING``; if BSD is fine, you find a +template in ``COPYING.edit-me``. Plugin Directory Layout ======================= @@ -179,7 +186,7 @@ A plugin's directory needs to follow a set of conventions so that Bro (1) recognizes it as a plugin, and (2) knows what to load. While ``init-plugin`` takes care of most of this, the following is the full story. We'll use ```` to represent a plugin's top-level -directory. +directory. With the skeleton, ```` corresponds to ``build/``. ``/__bro_plugin__`` A file that marks a directory as containing a Bro plugin. The file @@ -205,6 +212,8 @@ directory. Directory with auto-generated Bro scripts that declare the plugin's bif elements. The files here are produced by ``bifcl``. +Any other files in ```` are ignored by Bro. + By convention, a plugin should put its custom scripts into sub folders of ``scripts/``, i.e., ``scripts//