diff --git a/.gitmodules b/.gitmodules index 24375ce23d..91f39e3d04 100644 --- a/.gitmodules +++ b/.gitmodules @@ -22,3 +22,6 @@ [submodule "aux/plugins"] path = aux/plugins url = git://git.bro.org/bro-plugins +[submodule "aux/broker"] + path = aux/broker + url = git://git.bro.org/broker diff --git a/CHANGES b/CHANGES index d1031765cc..27b3020272 100644 --- a/CHANGES +++ b/CHANGES @@ -1,4 +1,459 @@ +2.3-720 | 2015-04-17 14:18:26 -0700 + + * Updating NEWS. + +2.3-716 | 2015-04-17 13:06:37 -0700 + + * Add seeking functionality to raw reader. One can now add an option + "offset" to the config map. Positive offsets are interpreted to be + from the beginning of the file, negative from the end of the file + (-1 is end of file). Only works for raw reader in streaming or + manual mode. Does not work with executables. Addresses BIT-985. + (Johanna Amann) + + * Allow setting packet and byte thresholds for connections. (Johanna Amann) + + This extends the ConnSize analyzer to be able to raise events when + each direction of a connection crosses a certain amount of bytes + or packets. + + Thresholds are set using: + - set_conn_bytes_threshold(c$id, [num-bytes], [direction]); + - set_conn_packets_threshold(c$id, [num-packets], [direction]); + + They raise the events, respectively: + - event conn_bytes_threshold_crossed(c: connection, threshold: count, is_orig: bool) + - event conn_packets_threshold_crossed(c: connection, threshold: count, is_orig: bool) + + Current thresholds can be examined using get_conn_bytes_threshold() + and get_conn_packets_threshold(). + + Only one threshold can be set per connection. + + * Add high-level API for packet/bytes thresholding in + base/protocols/conn/thresholds.bro that holds lists of thresholds + and raises an event for each threshold exactly once. (Johanna + Amann) + + * Fix a bug where child packet analyzers of the TCP analyzer + where not found using FindChild. + + * Update GridFTP analyzer to use connection thresholding instead of + polling. (Johanna Amann) + +2.3-709 | 2015-04-17 12:37:32 -0700 + + * Fix addressing the dreaded "internal error: unknown msg type 115 + in Poll()". (Jon Siwek) + + This patch removes the error handling code for overload conditions + in the main process that could cause trouble down the road. The + "chunked_io_buffer_soft_cap" script variable can now tune when the + client process begins shutting down peer connections, and the + default setting is now double what it used to be. Addresses + BIT-1376. + +2.3-707 | 2015-04-17 10:57:59 -0500 + + * Add more info about Broker to NEWS. (Jon Siwek) + +2.3-705 | 2015-04-16 08:16:45 -0700 + + * Update Mozilla CA list. (Johanna Amann) + + * Update tests to have them keep using older certificates where + appropiate. (Johanna Amann) + +2.3-699 | 2015-04-16 09:51:58 -0500 + + * Fix the to_count function to use strtoull versus strtoll. + (Jon Siwek) + +2.3-697 | 2015-04-15 09:51:15 -0700 + + * Removing error check verifying that an ASCII writer has been + properly finished. Instead of aborting, we now just clean up in + that case and proceed. Addresses BIT-1331. (Robin Sommer) + +2.3-696 | 2015-04-14 15:56:36 -0700 + + * Update sqlite to 3.8.9 + +2.3-695 | 2015-04-13 10:34:42 -0500 + + * Fix iterator invalidation in broker::Manager dtor. (Jon Siwek) + + * Add paragraph to plugin documentation. (Robin Sommer) + +2.3-693 | 2015-04-11 10:56:31 -0700 + + * BIT-1367: improve coercion of anonymous records in set constructor. + (Jon Siwek) + + * Allow to specify ports for sftp log rotator. (Johanna Amann) + +2.3-690 | 2015-04-10 21:51:10 -0700 + + * Make sure to always delete the remote serializer. Addresses + BIT-1306 and probably also BIT-1356. (Robin Sommer) + + * Cleaning up --help. -D and -Y/y were still listed, even though + they had no effect anymore. Removing some dead code along with -D. + Addresses BIT-1372. (Robin Sommer) + +2.3-688 | 2015-04-10 08:10:44 -0700 + + * Update SQLite to 3.8.8.3. + +2.3-687 | 2015-04-10 07:32:52 -0700 + + * Remove stale signature benchmarking code (-L command-line option). + (Jon Siwek) + + * BIT-844: fix UDP payload signatures to match packet-wise. (Jon + Siwek) + +2.3-682 | 2015-04-09 12:07:00 -0700 + + * Fixing input readers' component type. (Robin Sommer) + + * Tiny spelling correction. (Seth Hall) + +2.3-680 | 2015-04-06 16:02:43 -0500 + + * BIT-1371: remove CMake version check from binary package scripts. + (Jon Siwek) + +2.3-679 | 2015-04-06 10:16:36 -0500 + + * Increase some unit test timeouts. (Jon Siwek) + + * Fix Coverity warning in RDP analyzer. (Jon Siwek) + +2.3-676 | 2015-04-02 10:10:39 -0500 + + * BIT-1366: improve checksum offloading warning. + (Frank Meier, Jon Siwek) + +2.3-675 | 2015-03-30 17:05:05 -0500 + + * Add an RDP analyzer. (Josh Liburdi, Seth Hall, Johanna Amann) + +2.3-640 | 2015-03-30 13:51:51 -0500 + + * BIT-1359: Limit maximum number of DTLS fragments to 30. (Johanna Amann) + +2.3-637 | 2015-03-30 12:02:07 -0500 + + * Increase timeout duration in some broker tests. (Jon Siwek) + +2.3-636 | 2015-03-30 11:26:32 -0500 + + * Updates related to SSH analysis. (Jon Siwek) + + - Some scripts used wrong SSH module/namespace scoping on events. + - Fix outdated notice documentation related to SSH password guessing. + - Add a unit test for SSH pasword guessing notice. + +2.3-635 | 2015-03-30 11:02:45 -0500 + + * Fix outdated documentation unit tests. (Jon Siwek) + +2.3-634 | 2015-03-30 10:22:45 -0500 + + * Add a canonifier to a unit test's output. (Jon Siwek) + +2.3-633 | 2015-03-25 18:32:59 -0700 + + * Log::write in signature framework was missing timestamp. + (Andrew Benson/Michel Laterman) + +2.3-631 | 2015-03-25 11:03:12 -0700 + + * New SSH analyzer. (Vlad Grigorescu) + +2.3-600 | 2015-03-25 10:23:46 -0700 + + * Add defensive checks in code to calculate log rotation intervals. + (Pete Nelson). + +2.3-597 | 2015-03-23 12:50:04 -0700 + + * DTLS analyzer. (Johanna Amann) + + * Implement correct parsing of TLS record fragmentation. (Johanna + Amann) + +2.3-582 | 2015-03-23 11:34:25 -0700 + + * BIT-1313: In debug builds, "bro -B " now supports "all" and + "help" for "". "all" enables all debug streams. "help" prints a + list of available debug streams. (John Donnelly/Robin Sommer). + + * BIT-1324: Allow logging filters to inherit default path from + stream. This allows the path for the default filter to be + specified explicitly through $path="..." when creating a stream. + Adapted the existing Log::create_stream calls to explicitly + specify a path value. (Jon Siwek) + + * BIT-1199: Change the way the input framework deals with values it + cannot convert into BroVals, raising error messages instead of + aborting execution. (Johanna Amann) + + * BIT-788: Use DNS QR field to better identify flow direction. (Jon + Siwek) + +2.3-572 | 2015-03-23 13:04:53 -0500 + + * BIT-1226: Fix an example in quickstart docs. (Jon siwek) + +2.3-570 | 2015-03-23 09:51:20 -0500 + + * Correct a spelling error (Daniel Thayer) + + * Improvement to SSL analyzer failure mode. (Johanna Amann) + +2.3-565 | 2015-03-20 16:27:41 -0500 + + * BIT-978: Improve documentation of 'for' loop iterator invalidation. + (Jon Siwek) + +2.3-564 | 2015-03-20 11:12:02 -0500 + + * BIT-725: Remove "unmatched_HTTP_reply" weird. (Jon Siwek) + +2.3-562 | 2015-03-20 10:31:02 -0500 + + * BIT-1207: Add unit test to catch breaking changes to local.bro + (Jon Siwek) + + * Fix failing sqlite leak test (Johanna Amann) + +2.3-560 | 2015-03-19 13:17:39 -0500 + + * BIT-1255: Increase default values of + "tcp_max_above_hole_without_any_acks" and "tcp_max_initial_window" + from 4096 to 16384 bytes. (Jon Siwek) + +2.3-559 | 2015-03-19 12:14:33 -0500 + + * BIT-849: turn SMTP reporter warnings into weirds, + "smtp_nested_mail_transaction" and "smtp_unmatched_end_of_data". + (Jon Siwek) + +2.3-558 | 2015-03-18 22:50:55 -0400 + + * DNS: Log the type number for the DNS_RR_unknown_type weird. (Vlad Grigorescu) + +2.3-555 | 2015-03-17 15:57:13 -0700 + + * Splitting test-all Makefile target into Bro tests and test-aux. + (Robin Sommer) + +2.3-554 | 2015-03-17 15:40:39 -0700 + + * Deprecate &rotate_interval, &rotate_size, &encrypt. Addresses + BIT-1305. (Jon Siwek) + +2.3-549 | 2015-03-17 09:12:18 -0700 + + * BIT-1077: Fix HTTP::log_server_header_names. Before, it just + re-logged fields from the client side. (Jon Siwek) + +2.3-547 | 2015-03-17 09:07:51 -0700 + + * Update certificate validation script to cache valid intermediate + chains that it encounters on the wire and use those to try to + validate chains that might be missing intermediate certificates. + (Johanna Amann) + +2.3-541 | 2015-03-13 15:44:08 -0500 + + * Make INSTALL a symlink to doc/install/install.rst (Jon siwek) + + * Fix Broxygen coverage. (Jon Siwek) + +2.3-539 | 2015-03-13 14:19:27 -0500 + + * BIT-1335: Include timestamp in default extracted file names. + And add a policy script to extract all files. (Jon Siwek) + + * BIT-1311: Identify GRE tunnels as Tunnel::GRE, not Tunnel::IP. + (Jon Siwek) + + * BIT-1309: Add Connection class getter methods for flow labels. + (Jon Siwek) + +2.3-536 | 2015-03-12 16:16:24 -0500 + + * Fix Broker leak tests. (Jon Siwek) + +2.3-534 | 2015-03-12 10:59:49 -0500 + + * Update NEWS file. (Jon Siwek) + +2.3-533 | 2015-03-12 10:18:53 -0500 + + * Give broker python bindings default install path within --prefix. + (Jon Siwek) + +2.3-530 | 2015-03-10 13:22:39 -0500 + + * Fix broker data stores in absence of --enable-debug. (Jon Siwek) + +2.3-529 | 2015-03-09 13:14:27 -0500 + + * Fix format specifier in SSL protocol violation. (Jon Siwek) + +2.3-526 | 2015-03-06 12:48:49 -0600 + + * Fix build warnings, clarify broker requirements, update submodule. + (Jon Siwek) + + * Rename comm/ directories to broker/ (Jon Siwek) + + * Rename broker-related namespaces. (Jon Siwek) + + * Improve remote logging via broker by only sending fields w/ &log. + (Jon Siwek) + + * Disable a stream's remote logging via broker if it fails. (Jon Siwek) + + * Improve some broker communication unit tests. (Jon Siwek) + +2.3-518 | 2015-03-04 13:13:50 -0800 + + * Add bytes_recvd to stats.log recording the number of bytes + received, according to packet headers. (Mike Smiley) + +2.3-516 | 2015-03-04 12:30:06 -0800 + + * Extract most specific Common Name from SSL certificates (Johanna + Amann) + + * Send CN and SAN fields of SSL certificates to the Intel framework. + (Johanna Amann) + +2.3-511 | 2015-03-02 18:07:17 -0800 + + * Changes to plugin meta hooks for function calls. (Gilbert Clark) + + - Add frame argument. + + - Change return value to tuple unambigiously whether hook + returned a result. + +2.3-493 | 2015-03-02 17:17:32 -0800 + + * Extend the SSL weak-keys policy file to also alert when + encountering SSL connections with old versions as well as unsafe + cipher suites. (Johanna Amann) + + * Make the notice suppression handling of other SSL policy files a + tad more robust. (Johanna Amann) + +2.3-491 | 2015-03-02 17:12:56 -0800 + + * Updating docs for recent addition of local_resp. (Robin Sommer) + +2.3-489 | 2015-03-02 15:29:30 -0800 + + * Integrate Broker, Bro's new communication library. (Jon Siwek) + + See aux/broker/README for more information on Broker, and + doc/frameworks/comm.rst for the corresponding Bro script API. + + Broker support is by default off for now; it can be enabled at + configure time with --enable-broker. It requires CAF + (https://github.com/actor-framework/actor-framework); for now iot + needs CAF's "develop" branch. Broker also requires a C++11 + compiler. + + Broker will become a mandatory dependency in future Bro versions. + + * Add --enable-c++11 configure flag to compile Bro's source code in + C++11 mode with a corresponding compiler. (Jon Siwek) + +2.3-451 | 2015-02-24 16:37:08 -0800 + + * Updating submodule(s). + +2.3-448 | 2015-02-23 16:58:10 -0800 + + * Updating NEWS. (Robin Sommer) + +2.3-447 | 2015-02-23 16:28:30 -0800 + + * Fix potential crash in logging framework when deserializing + WriterInfo from remote. where config is present. Testcase crashes + on unpatched versions of Bro. (Aaron Eppert) + + * Fix wrong value test in WriterBackend. (Aaron Eppert) + +2.3-442 | 2015-02-23 13:29:30 -0800 + + * Add a "local_resp" field to conn.log, along the lines of the + existing "local_orig". (Mike Smiley) + +2.3-440 | 2015-02-23 11:39:17 -0600 + + * Updating plugin docs to recent changes. (Robin Sommer) + + * Updating plugin tests to recent changes. (Robin Sommer) + + * Making plugin names case-insensitive for some internal comparisions. + Makes plugin system more tolerant against spelling inconsistencies + are hard to catch otherwise. (Robin Sommer) + + * Explicitly removing some old scripts on install that have moved + into plugins to prevent them causing confusion. (Robin Sommer) + + * BIT-1312: Removing setting installation plugin path from + bro-path-dev.sh. Also, adding to existing BRO_PLUGIN_PATH rather + than replacing. (Robin Sommer) + + * Creating the installation directory for plugins at install time. + (Robin Sommer) + +2.3-427 | 2015-02-20 13:49:33 -0800 + + * Removing dependency on PCAP_NETMASK_UNKNOWN to compile with + libpcap < 1.1.1. (Robin Sommer) + +2.3-426 | 2015-02-20 12:45:51 -0800 + + * Add 'while' statement to Bro language. Really. (Jon Siwek) + +2.3-424 | 2015-02-20 12:39:10 -0800 + + * Add the ability to remove surrounding braces from the JSON + formatter. (Seth Hall) + +2.3-419 | 2015-02-13 09:10:44 -0600 + + * BIT-1011: Update the SOCKS analyzer to support user/pass login. + (Nicolas Retrain, Seth Hall, Jon Siwek) + + - Add a new field to socks.log: "password". + - Two new events: "socks_login_userpass_request" and + "socks_login_userpass_reply". + - Two new weirds for unsupported SOCKS authentication method or + version. + - A new test for authenticated socks traffic. + +2.3-416 | 2015-02-12 12:18:42 -0600 + + * Submodule update - newest sqlite version (Johanna Amann) + + * Fix use of deprecated gperftools headers. (Jon Siwek) + +2.3-413 | 2015-02-08 18:23:05 -0800 + + * Fixing analyzer tag types for some Files::* functions. (Robin Sommer) + + * Changing load order for plugin scripts. (Robin Sommer) + 2.3-411 | 2015-02-05 10:05:48 -0600 * Fix file analysis of files with total size below the bof_buffer size @@ -166,7 +621,7 @@ 2.3-328 | 2014-12-02 08:13:10 -0500 - * Update windows-version-detection.bro to add support for + * Update windows-version-detection.bro to add support for Windows 10. (Michal Purzynski) 2.3-326 | 2014-12-01 12:10:27 -0600 @@ -236,7 +691,7 @@ 2.3-280 | 2014-11-05 09:46:33 -0500 - * Add Windows detection based on CryptoAPI HTTP traffic as a + * Add Windows detection based on CryptoAPI HTTP traffic as a software framework policy script. (Vlad Grigorescu) 2.3-278 | 2014-11-03 18:55:18 -0800 diff --git a/CMakeLists.txt b/CMakeLists.txt index c0ff6c09d4..8f60ab95ad 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -31,12 +31,12 @@ configure_file(bro-path-dev.in ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev) file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.sh "export BROPATH=`${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" - "export BRO_PLUGIN_PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src:${BRO_PLUGIN_INSTALL_PATH}\"\n" + "export BRO_PLUGIN_PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src\":${BRO_PLUGIN_PATH}\n" "export PATH=\"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n") file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.csh "setenv BROPATH `${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" - "setenv BRO_PLUGIN_PATH \"${CMAKE_CURRENT_BINARY_DIR}/src:${BRO_PLUGIN_INSTALL_PATH}\"\n" + "setenv BRO_PLUGIN_PATH \"${CMAKE_CURRENT_BINARY_DIR}/src\":${BRO_PLUGIN_PATH}\n" "setenv PATH \"${CMAKE_CURRENT_BINARY_DIR}/src\":$PATH\n") file(STRINGS "${CMAKE_CURRENT_SOURCE_DIR}/VERSION" VERSION LIMIT_COUNT 1) @@ -177,6 +177,17 @@ include_directories(${CMAKE_CURRENT_BINARY_DIR}) ######################################################################## ## Recurse on sub-directories +if ( ENABLE_CXX11 ) + include(RequireCXX11) +endif () + +if ( ENABLE_BROKER ) + add_subdirectory(aux/broker) + set(brodeps ${brodeps} broker) + add_definitions(-DENABLE_BROKER) + include_directories(BEFORE ${CMAKE_CURRENT_SOURCE_DIR}/aux/broker) +endif () + add_subdirectory(src) add_subdirectory(scripts) add_subdirectory(doc) @@ -224,6 +235,7 @@ message( "\nCXXFLAGS: ${CMAKE_CXX_FLAGS} ${CMAKE_CXX_FLAGS_${BuildType}}" "\nCPP: ${CMAKE_CXX_COMPILER}" "\n" + "\nBroker: ${ENABLE_BROKER}" "\nBroccoli: ${INSTALL_BROCCOLI}" "\nBroctl: ${INSTALL_BROCTL}" "\nAux. Tools: ${INSTALL_AUX_TOOLS}" diff --git a/INSTALL b/INSTALL deleted file mode 100644 index 385dac93df..0000000000 --- a/INSTALL +++ /dev/null @@ -1,3 +0,0 @@ - -See doc/install/install.rst for installation instructions. - diff --git a/INSTALL b/INSTALL new file mode 120000 index 0000000000..95fcc60eda --- /dev/null +++ b/INSTALL @@ -0,0 +1 @@ +doc/install/install.rst \ No newline at end of file diff --git a/Makefile b/Makefile index 207ce72780..3efddc4dbc 100644 --- a/Makefile +++ b/Makefile @@ -51,13 +51,15 @@ distclean: $(MAKE) -C testing $@ test: - @( cd testing && make ) + -@( cd testing && make ) -test-all: test - test -d aux/broctl && ( cd aux/broctl && make test-all ) - test -d aux/btest && ( cd aux/btest && make test ) - test -d aux/bro-aux && ( cd aux/bro-aux && make test ) - test -d aux/plugins && ( cd aux/plugins && make test-all ) +test-aux: + -test -d aux/broctl && ( cd aux/broctl && make test-all ) + -test -d aux/btest && ( cd aux/btest && make test ) + -test -d aux/bro-aux && ( cd aux/bro-aux && make test ) + -test -d aux/plugins && ( cd aux/plugins && make test-all ) + +test-all: test test-aux configured: @test -d $(BUILD) || ( echo "Error: No build/ directory found. Did you run configure?" && exit 1 ) diff --git a/NEWS b/NEWS index af59858e06..93ffbea5a3 100644 --- a/NEWS +++ b/NEWS @@ -7,9 +7,6 @@ their own ``CHANGES``.) Bro 2.4 (in progress) ===================== -Dependencies ------------- - New Functionality ----------------- @@ -17,20 +14,98 @@ New Functionality functionality, like protocol/file analysis, via shared libraries. Plugins can be developed and distributed externally, and will be pulled in dynamically at startup. Currently, a plugin can provide - custom protocol analyzers, file analyzers, log writers[TODO], input - readers[TODO], packet sources[TODO], and new built-in functions. A - plugin can furthermore hook into Bro's processing a number of places - to add custom logic. + custom protocol analyzers, file analyzers, log writers, input + readers, packet sources and dumpers, and new built-in functions. A + plugin can furthermore hook into Bro's processing at a number of + places to add custom logic. See https://www.bro.org/sphinx-git/devel/plugins.html for more information on writing plugins. -- Bro now has supoprt for the MySQL wire protocol. Activity gets +- Bro now has support for the MySQL wire protocol. Activity gets logged into mysql.log. +- Bro now parses DTLS traffic. + +- Bro now has an RDP analyzer. + +- Bro now features a completely rewritten, enhanced SSH analyzer, with + a set of addedd events being generated. A lot more information about + SSH sessions is logged. The analyzer is able to determine if logins + failed or succeeded in most circumstances. + - Bro's file analysis now supports reassembly of files that are not transferred/seen sequentially. +- Bro's scripting language now has a ``while`` statement:: + + while ( i < 5 ) + print ++i; + + ``next`` and ``break`` can be used inside the loop's body just like + with ``for`` loops. + +- Bro now integrates Broker, a new communication library. See + aux/broker/README for more information on Broker, and + doc/frameworks/broker.rst for the corresponding Bro script API. + + With Broker, Bro has the similar capabilities of exchanging events and + logs with remote peers (either another Bro process or some other + application that uses Broker). It also includes a key-value store + API that can be used to share state between peers and optionally + allow data to persist on disk for longer-term storage. + + Broker support is by default off for now; it can be enabled at + configure time with --enable-broker. It requires CAF version 0.13+ + (https://github.com/actor-framework/actor-framework) as well as a + C++11 compiler (e.g. GCC 4.8+ or Clang 3.3+). + + Broker will become a mandatory dependency in future Bro versions and + replace the current communcation and serialization system. + +- Add --enable-c++11 configure flag to compile Bro's source code in + C++11 mode with a corresponding compiler. Note that 2.4 will be the + last version of Bro that compiles without C++11 support. + +- The SSL analysis now alerts when encountering SSL connections with + old protocol versions or unsafe cipher suites. It also gained + extended reporting of weak keys, caching of already valdidated + certificates, full support TLS record defragmentation. SSL generally + became much more robust and added several fields to ssl.log (while + removing some other). + +- A new icmp_sent_payload event provides access to ICMP payload. + +- The input framework's raw reader now supports seeking by adding an + option "offset" to the config map. Positive offsets are interpreted + to be from the beginning of the file, negative from the end of the + file (-1 is end of file). + +- One can now raise events when a connection crosses a given size + threshold in terms of packets or bytes. The primary API for that + functionality is in base/protocols/conn/thresholds.bro. + +- BroControl now has a new command "deploy" which is equivalent to running + the "check", "install", "stop", and "start" commands (in that order). + +- BroControl now has a new option "StatusCmdShowAll" that controls whether + or not the broctl "status" command gathers all of the status information. + This option can be used to make the "status" command run significantly + faster (in this case, the "Peers" column will not be shown in the output). + +- BroControl now has a new option "StatsLogEnable" that controls whether + or not broctl will record information to the "stats.log" file. This option + can be used to make the "broctl cron" command run slightly faster (in this + case, "broctl cron" will also no longer send email about not seeing any + packets on the monitoring interfaces). + +- BroControl now has a new option "MailHostUpDown" which controls whether or + not the "broctl cron" command will send email when it notices that a host + in the cluster is up or down. + +- BroControl now has a new option "CommandTimeout" which specifies the number + of seconds to wait for a command that broctl ran to return results. + Changed Functionality --------------------- @@ -43,6 +118,11 @@ Changed Functionality have been added which contain the same information. The ``mime_type`` field of ``Files::Info`` also still has this info. + * The earliest point that new mime type information is available is + in the ``file_mime_type`` event which comes after the ``file_new`` + and ``file_over_new_connection`` events. Scripts which inspected + mime type info within those events will need to be adapted. + * Removed ``Files::add_analyzers_for_mime_type`` function. * Removed ``offset`` parameter of the ``file_extraction_limit`` @@ -56,6 +136,46 @@ Changed Functionality - has_valid_octets: now uses a string_vec parameter instead of string_array. +- conn.log gained a new field local_resp that works like local_orig, + just for the responder address of the connection. + +- GRE tunnels are now identified as ``Tunnel::GRE`` instead of + ``Tunnel::IP``. + +- The default name for extracted files changed from extract-protocol-id + to extract-timestamp-protocol-id. + +- The weird named "unmatched_HTTP_reply" has been removed since it can + be detected at the script-layer and is handled correctly by the + default HTTP scripts. + +- When adding a logging filter to a stream, the filter can now inherit + a default ``path`` field from the associated ``Log::Stream`` record. + +- When adding a logging filter to a stream, the + ``Log::default_path_func`` is now only automatically added to the + filter if it has neither a ``path`` nor a ``path_func`` already + explicitly set. Before, the default path function would always be set + for all filters which didn't specify their own ``path_func``. + +- BroControl now establishes only one ssh connection from the manager to + each remote host in a cluster configuration (previously, there would be + one ssh connection per remote Bro process). + +- BroControl now uses SQLite to record state information instead of a + plain text file (the file "spool/broctl.dat" is no longer used). + On FreeBSD, this means that there is a new dependency on the package + "py27-sqlite3". + +- BroControl now records the expected running state of each Bro node right + before each start or stop. The "broctl cron" command uses this info to + either start or stop Bro nodes as needed so that the actual state matches + the expected state (previously, "broctl cron" could only start nodes in + the "crashed" state, and could never stop a node). + +- BroControl now sends all normal command output (i.e., not error messages) + to stdout. Error messages are still sent to stderr, however. + Deprecated Functionality ------------------------ diff --git a/VERSION b/VERSION index defa33cc31..4953236124 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.3-411 +2.3-720 diff --git a/aux/binpac b/aux/binpac index 77a86591dc..544330932e 160000 --- a/aux/binpac +++ b/aux/binpac @@ -1 +1 @@ -Subproject commit 77a86591dcf89d7252d3676d3f1199d6c927d073 +Subproject commit 544330932e7cd4615d6d19f63907e8aa2acebb9e diff --git a/aux/bro-aux b/aux/bro-aux index 0b713c027d..462e300bf9 160000 --- a/aux/bro-aux +++ b/aux/bro-aux @@ -1 +1 @@ -Subproject commit 0b713c027d3efaaca50e5df995c02656175573cd +Subproject commit 462e300bf9c37dcc39b70a4c2d89d19f7351c804 diff --git a/aux/broccoli b/aux/broccoli index d43cc790e5..45276b39a9 160000 --- a/aux/broccoli +++ b/aux/broccoli @@ -1 +1 @@ -Subproject commit d43cc790e5b8709b5e032e52ad0e00936494739b +Subproject commit 45276b39a946d70095c983753cd321ad07dcf285 diff --git a/aux/broctl b/aux/broctl index 8c9b87bc73..e864a0949e 160000 --- a/aux/broctl +++ b/aux/broctl @@ -1 +1 @@ -Subproject commit 8c9b87bc73e1ddaa304e3d89028c1e7b95d37a91 +Subproject commit e864a0949e52a797f4000194b5c2980cf3618deb diff --git a/aux/broker b/aux/broker new file mode 160000 index 0000000000..0c25c1daa7 --- /dev/null +++ b/aux/broker @@ -0,0 +1 @@ +Subproject commit 0c25c1daa7dcf885dd16cc1b725295dc36decafe diff --git a/aux/btest b/aux/btest index 93d4989ed1..d69df586c9 160000 --- a/aux/btest +++ b/aux/btest @@ -1 +1 @@ -Subproject commit 93d4989ed1537e4d143cf09d44077159f869a4b2 +Subproject commit d69df586c91531db0c3abe838b10a429dda4fa87 diff --git a/aux/plugins b/aux/plugins index ad600b5bdc..7a14085394 160000 --- a/aux/plugins +++ b/aux/plugins @@ -1 +1 @@ -Subproject commit ad600b5bdcd56a2723e323c0f2c8e1708956ca4f +Subproject commit 7a14085394e54a950e477eb4fafb3827ff8dbdc3 diff --git a/cmake b/cmake index 1316c07f70..2fd35ab6a6 160000 --- a/cmake +++ b/cmake @@ -1 +1 @@ -Subproject commit 1316c07f7059647b6c4a496ea36e4b83bb5d8f0f +Subproject commit 2fd35ab6a6245a005828c32f0aa87eb21698c054 diff --git a/configure b/configure index 2b1c568b26..b139ee2bec 100755 --- a/configure +++ b/configure @@ -41,6 +41,9 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --enable-perftools-debug use Google's perftools for debugging --enable-jemalloc link against jemalloc --enable-ruby build ruby bindings for broccoli (deprecated) + --enable-c++11 build using the C++11 standard + --enable-broker enable use of the Broker communication library + (requires C++ Actor Framework and C++11) --disable-broccoli don't build or install the Broccoli library --disable-broctl don't install Broctl --disable-auxtools don't build or install auxiliary tools @@ -55,6 +58,8 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --with-flex=PATH path to flex executable --with-bison=PATH path to bison executable --with-perl=PATH path to perl executable + --with-libcaf=PATH path to C++ Actor Framework installation + (a required Broker dependency) Optional Packages in Non-Standard Locations: --with-geoip=PATH path to the libGeoIP install root @@ -67,6 +72,8 @@ Usage: $0 [OPTION]... [VAR=VALUE]... --with-ruby-lib=PATH path to ruby library --with-ruby-inc=PATH path to ruby headers --with-swig=PATH path to SWIG executable + --with-rocksdb=PATH path to RocksDB installation + (an optional Broker dependency) Packaging Options (for developers): --binary-package toggle special logic for binary packaging @@ -142,6 +149,10 @@ while [ $# -ne 0 ]; do append_cache_entry CMAKE_INSTALL_PREFIX PATH $optarg append_cache_entry BRO_ROOT_DIR PATH $optarg append_cache_entry PY_MOD_INSTALL_DIR PATH $optarg/lib/broctl + + if [ -n "$user_enabled_broker" ]; then + append_cache_entry BROKER_PYTHON_HOME PATH $prefix + fi ;; --scriptdir=*) append_cache_entry BRO_SCRIPT_INSTALL_PATH STRING $optarg @@ -176,6 +187,15 @@ while [ $# -ne 0 ]; do --enable-jemalloc) append_cache_entry ENABLE_JEMALLOC BOOL true ;; + --enable-c++11) + append_cache_entry ENABLE_CXX11 BOOL true + ;; + --enable-broker) + append_cache_entry ENABLE_CXX11 BOOL true + append_cache_entry ENABLE_BROKER BOOL true + append_cache_entry BROKER_PYTHON_HOME PATH $prefix + user_enabled_broker="true" + ;; --disable-broccoli) append_cache_entry INSTALL_BROCCOLI BOOL false ;; @@ -248,6 +268,12 @@ while [ $# -ne 0 ]; do --with-swig=*) append_cache_entry SWIG_EXECUTABLE PATH $optarg ;; + --with-libcaf=*) + append_cache_entry LIBCAF_ROOT_DIR PATH $optarg + ;; + --with-rocksdb=*) + append_cache_entry ROCKSDB_ROOT_DIR PATH $optarg + ;; --binary-package) append_cache_entry BINARY_PACKAGING_MODE BOOL true ;; diff --git a/doc/components/broker/README.rst b/doc/components/broker/README.rst new file mode 120000 index 0000000000..eafa3b8e77 --- /dev/null +++ b/doc/components/broker/README.rst @@ -0,0 +1 @@ +../../../aux/broker/README \ No newline at end of file diff --git a/doc/components/broker/broker-manual.rst b/doc/components/broker/broker-manual.rst new file mode 120000 index 0000000000..90bf8f0833 --- /dev/null +++ b/doc/components/broker/broker-manual.rst @@ -0,0 +1 @@ +../../../aux/broker/broker-manual.rst \ No newline at end of file diff --git a/doc/components/index.rst b/doc/components/index.rst index fe05f13683..c1feda4a61 100644 --- a/doc/components/index.rst +++ b/doc/components/index.rst @@ -17,6 +17,8 @@ current, independent component releases. Broccoli - User Manual Broccoli Python Bindings Broccoli Ruby Bindings + Broker - Bro's (New) Messaging Library (README) + Broker - User Manual BroControl - Interactive Bro management shell Bro-Aux - Small auxiliary tools for Bro BTest - A unit testing framework diff --git a/doc/devel/plugins.rst b/doc/devel/plugins.rst index c703345891..5c963a1552 100644 --- a/doc/devel/plugins.rst +++ b/doc/devel/plugins.rst @@ -3,7 +3,7 @@ Writing Bro Plugins =================== -Bro is internally moving to a plugin structure that enables extending +Bro internally provides plugin API that enables extending the system dynamically, without modifying the core code base. That way custom code remains self-contained and can be maintained, compiled, and installed independently. Currently, plugins can add the following @@ -42,18 +42,17 @@ certain structure. To get started, Bro's distribution provides a helper script ``aux/bro-aux/plugin-support/init-plugin`` that creates a skeleton plugin that can then be customized. Let's use that:: - # mkdir rot13-plugin - # cd rot13-plugin - # init-plugin Demo Rot13 + # init-plugin ./rot13-plugin Demo Rot13 -As you can see the script takes two arguments. The first is a -namespace the plugin will live in, and the second a descriptive name -for the plugin itself. Bro uses the combination of the two to identify -a plugin. The namespace serves to avoid naming conflicts between -plugins written by independent developers; pick, e.g., the name of -your organisation. The namespace ``Bro`` is reserved for functionality -distributed by the Bro Project. In our example, the plugin will be -called ``Demo::Rot13``. +As you can see, the script takes three arguments. The first is a +directory inside which the plugin skeleton will be created. The second +is the namespace the plugin will live in, and the third is a descriptive +name for the plugin itself relative to the namespace. Bro uses the +combination of namespace and name to identify a plugin. The namespace +serves to avoid naming conflicts between plugins written by independent +developers; pick, e.g., the name of your organisation. The namespace +``Bro`` is reserved for functionality distributed by the Bro Project. In +our example, the plugin will be called ``Demo::Rot13``. The ``init-plugin`` script puts a number of files in place. The full layout is described later. For now, all we need is @@ -61,7 +60,7 @@ layout is described later. For now, all we need is there as follows:: # cat src/rot13.bif - module CaesarCipher; + module Demo; function rot13%(s: string%) : string %{ @@ -82,18 +81,22 @@ The syntax of this file is just like any other ``*.bif`` file; we won't go into it here. Now we can already compile our plugin, we just need to tell the -configure script put in place by ``init-plugin`` where the Bro source -tree is located (Bro needs to have been built there first):: +configure script that ``init-plugin`` put in place where the Bro +source tree is located (Bro needs to have been built there first):: + # cd rot13-plugin # ./configure --bro-dist=/path/to/bro/dist && make [... cmake output ...] -Now our ``rot13-plugin`` directory has everything that it needs -for Bro to recognize it as a dynamic plugin. Once we point Bro to it, -it will pull it in automatically, as we can check with the ``-N`` +This builds the plugin in a subdirectory ``build/``. In fact, that +subdirectory *becomes* the plugin: when ``make`` finishes, ``build/`` +has everything it needs for Bro to recognize it as a dynamic plugin. + +Let's try that. Once we point Bro to the ``build/`` directory, it will +pull in our new plugin automatically, as we can check with the ``-N`` option:: - # export BRO_PLUGIN_PATH=/path/to/rot13-plugin + # export BRO_PLUGIN_PATH=/path/to/rot13-plugin/build # bro -N [...] Plugin: Demo::Rot13 - (dynamic, version 1) @@ -127,12 +130,12 @@ more verbose option ``-NN``:: # bro -NN [...] Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) - [Function] CaesarCipher::rot13 + [Function] Demo::rot13 [...] There's our function. Now let's use it:: - # bro -e 'print CaesarCipher::rot13("Hello")' + # bro -e 'print Demo::rot13("Hello")' Uryyb It works. We next install the plugin along with Bro itself, so that it @@ -141,36 +144,40 @@ environment variable. If we first unset the variable, the function will no longer be available:: # unset BRO_PLUGIN_PATH - # bro -e 'print CaesarCipher::rot13("Hello")' - error in , line 1: unknown identifier CaesarCipher::rot13, at or near "CaesarCipher::rot13" + # bro -e 'print Demo::rot13("Hello")' + error in , line 1: unknown identifier Demo::rot13, at or near "Demo::rot13" Once we install it, it works again:: # make install - # bro -e 'print CaesarCipher::rot13("Hello")' + # bro -e 'print Demo::rot13("Hello")' Uryyb The installed version went into ``/lib/bro/plugins/Demo_Rot13``. -We can distribute the plugin in either source or binary form by using -the Makefile's ``sdist`` and ``bdist`` target, respectively. Both -create corrsponding tarballs:: +One can distribute the plugin independently of Bro for others to use. +To distribute in source form, just remove the ``build/`` (``make +distclean`` does that) and then tar up the whole ``rot13-plugin/`` +directory. Others then follow the same process as above after +unpacking. To distribute the plugin in binary form, the build process +conveniently creates a corresponding tarball in ``build/dist/``. In +this case, it's called ``Demo_Rot13-0.1.tar.gz``, with the version +number coming out of the ``VERSION`` file that ``init-plugin`` put +into place. The binary tarball has everything needed to run the +plugin, but no further source files. Optionally, one can include +further files by specifying them in the plugin's ``CMakeLists.txt`` +through the ``bro_plugin_dist_files`` macro; the skeleton does that +for ``README``, ``VERSION``, ``CHANGES``, and ``COPYING``. To use the +plugin through the binary tarball, just unpack it and point +``BRO_PLUGIN_PATH`` there; or copy it into +``/lib/bro/plugins/`` directly. - # make sdist - [...] - Source distribution in build/sdist/Demo_Rot13.tar.gz - - # make bdist - [...] - Binary distribution in build/Demo_Rot13-darwin-x86_64.tar.gz - -The source archive will contain everything in the plugin directory -except any generated files. The binary archive will contain anything -needed to install and run the plugin, i.e., just what ``make install`` -puts into place as well. As the binary distribution is -platform-dependent, its name includes the OS and architecture the -plugin was built on. +Before distributing your plugin, you should edit some of the meta +files that ``init-plugin`` puts in place. Edit ``README`` and +``VERSION``, and update ``CHANGES`` when you make changes. Also put a +license file in place as ``COPYING``; if BSD is fine, you find a +template in ``COPYING.edit-me``. Plugin Directory Layout ======================= @@ -179,7 +186,7 @@ A plugin's directory needs to follow a set of conventions so that Bro (1) recognizes it as a plugin, and (2) knows what to load. While ``init-plugin`` takes care of most of this, the following is the full story. We'll use ```` to represent a plugin's top-level -directory. +directory. With the skeleton, ```` corresponds to ``build/``. ``/__bro_plugin__`` A file that marks a directory as containing a Bro plugin. The file @@ -205,6 +212,8 @@ directory. Directory with auto-generated Bro scripts that declare the plugin's bif elements. The files here are produced by ``bifcl``. +Any other files in ```` are ignored by Bro. + By convention, a plugin should put its custom scripts into sub folders of ``scripts/``, i.e., ``scripts//