2.0-beta-pre | 2011-10-27 11:26:01 -0700 * Preliminary fix for SSH login detection: we need a counted measure of payload bytes (not ack tracking and not with the IP header which is what we have now). (Seth Hall) * Fixing send_id() problem. We no longer update &redef functions. Updating code on the fly isn't fully supported. (Robin Sommer) * Tuning the format of the pretty-printed alarm summaries. (Robin Sommer) 1.6-dev-1508 | 2011-10-26 17:24:50 -0700 * Updating submodule(s). (Robin Sommer) 1.6-dev-1507 | 2011-10-26 15:10:18 -0700 * Baseline updates. (Robin Sommer) 1.6-dev-1506 | 2011-10-26 14:48:43 -0700 * Updating submodule(s). (Robin Sommer) 1.6-dev-1505 | 2011-10-26 14:43:58 -0700 * A new base script that pretty-prints alarms in the regular summary. (Robin Sommer) * Adding a dummy log writer WRITER_NONE that just discards everything. (Robin Sommer) 1.6-dev-1498 | 2011-10-26 14:30:15 -0700 * Adding instructions to local.bro how to do ACTION_ALARM by default. (Seth Hall) 1.6-dev-1495 | 2011-10-26 10:15:58 -0500 * Updated unit test baselines. (Seth Hall) 1.6-dev-1491 | 2011-10-25 20:22:56 -0700 * Updating submodule(s). (Robin Sommer) 1.6-dev-1482 | 2011-10-25 19:08:32 -0700 * Fixing bug in log managers predicate evaluation. (Robin Sommer) 1.6-dev-1481 | 2011-10-25 18:17:03 -0700 * Fix a problem with DNS servers being logged that aren't actually servers. (Seth Hall) * Changed generated root cert DN format for RFC2253 compliance. (Jon Siwek) * Removed :bro doc directives from notice documentation. (Seth Hall) * New notice framework docs. (Seth Hall) * Adding sub messages to emails. (Seth Hall) * Adding extra fields to smtp and http to track transaction depth. (Seth Hall) * Fix for SSH login detection heuristic. (Seth Hall) * Removed some fields from http analysis that weren't commonly needed or were wrong. (Seth Hall) * Updated/fixed MSIE version parsing in the software framework. (Seth Hall) * Update Mozilla trust roots to index certs by subject distinguished name. (Jon Siwek) * weird.bro rewrite. (Seth Hall) * More notice email tuning. (Seth Hall) * Slightly restructured http file hashing to fix a bug. (Seth Hall) * Changed the notice name for interesting ssh logins to correctly reflect semantics of the notice. (Seth Hall) * Field name change to notice framwork. $result -> $action - $result is renamed to $action to reflect changes to the notice framework since there is already another result-like field ($suppress_for) and there may be more in the future. - Slipped in a change to add connection information to notice emails too. (Seth Hall) * Small script refinements and documentation updates. (Seth Hall) * Pass over upgrade guide. (Robin Sommer) 1.6-dev-1430 | 2011-10-21 10:39:09 -0700 * Fixing crash with unknown debug streams. Closes #643. (Robin Sommer) * Code to better handle interpreter errors, which can now be turned into non-fatal runtime errors rather than immediate aborts. (Robin Sommer). * Remove old make-src-packages script. (Jon Siwek) * Fixing a bunch of format strings. Closes #567. (Robin Sommer) * Cleaning up some distribution files. (Robin Sommer) * Various test, doc, and installation fixes/tweaks. (Seth Hall, Jon Siwek and Robin Sommer). * Varios smaller policy fixes and tweaks (Seth Hall). * Moving docs from web server into distribution. (Robin Sommer) * Fixing more (small) memory leaks. (Robin Sommer) * Profiling support for DNS_Mgr and triggers. With misc/profiling.bro, both now report a line in prof.log with some counters on usage. (Robin Sommer) * Fixing DNS memory leaks. Closes #534. (Robin Sommer) * Fix code for disabling analyzers. Closes #577. (Robin Sommer) * Changed communication option from listen_encrypted to listen_ssl. (Seth Hall) * Modification to the Communication framework API. (Seth Hall) - Simplified the communication API and made it easier to change to encrypted connections by not having separate variables to define encrypted and unencrypted ports. - Now, to enable listening without configuring nodes just load the frameworks/communication/listen script. - If encrypted listening is desired set the following: redef Communication::listen_encrypted=T; * Connection compressor now disabled by default. Addresses #559. (Robin Sommer) 1.6-dev-1372 | 2011-10-06 18:09:17 -0700 * Filtering some potentially high-volume DNS weirds. (Robin Sommer) * DNS now raises DPD events. Closes #577. (Robin Sommer) * Fixing a bunch of compiler warnings. (Robin Sommer) * Remote logs are auto-flushed if the last write was longer than a second ago. Addresses #498. (Robin Sommer) * Fix missing from previous MIME commit. (Robin Sommer) 1.6-dev-1366 | 2011-10-06 17:05:21 -0700 * Make CompHash computation/recovery for functions deterministic. Closes #636. (Jon Siwek) * Removing unnecessary @load in local.bro. (Robin Sommer) * Optimizing some MIME code. (Robin Sommer) * Speed improvements in logging code. (Robin Sommer) * Consolidating some node-specific functionality from scripts in broctl repo. (Jon Siwek) * Another fix the for 1xx script code. (Robin Sommer) 1.6-dev-1352 | 2011-10-05 16:20:51 -0700 * Fix for optional HTTP::Info status_code. (Jon Siwek) * Teaking some external testing scripts. (Jon Siwek) * HTTP bug fix reported by Martin Holste. (Seth Hall) * More script tuning. (Seth Hall) - Moved some of the weird events back to the base/ directory. - SSL fixes, updates, and performance optimization. * More adjustment to reduce Weird volumes. (Seth Hall) * Fixed an error when calculating x509 certificate hashes (reported by Martin Holste). (Seth Hall) * Clean up to cluster framework to make event handling clearer. (Seth Hall) * Fixed a bug in the notice framework. (Seth Hall) * Bug fix for FTP analysis script. (Seth Hall) 1.6-dev-1333 | 2011-09-29 22:29:51 -0700 * Fixing a number of memory leaks. (Robin Sommer) * Loaded_scripts.log is indented with spaces now and makes more sense to look at. (Seth Hall) * Teach HTTP parser to derive content length of multipart/byteranges bodies. Addresses #488. (Jon Siwek) * Change logging of HTTP 1xx responses to occur in their own columns. Addresses #411. (Jon Siwek) * Fix handling of HTTP 1xx response codes. Addresses #411). * Taking advantage of yet another trick to get installed browser plugins. (Seth Hall) - With the software-browser-plugins script you can watch for Omniture advertising servers to grab the list of installed plugins. - I reorganized the plugin detection a bit too to abstract it better. - Removed the WEB_ prefix from all of the Software::Type HTTP enums. They were essentially redundant due to the full name already being HTTP::SERVER (for example). 1.6-dev-1316 | 2011-09-28 16:50:05 -0700 * Unit test cleanup. Updated README and collected coverage-related tests in a common dir. (Jon Siwek) * Fixes for known-services. (Seth Hall) * Ported and 2.0ized the capture-loss script. (Seth Hall) * Communication fix and extension.(Robin Sommer) - Removing unnecessary log flushing. Closes #498. - Adding new BiF disconnect() that shuts a connection to a peer down. - terminate_connection() now first flushes any still buffered log messages. * Fix for high SSL memory usage by adding &transient attribute to top-level SSL pac array type. Closes #574. (Robin Sommer) * Fix a small bug in the metrics framework. (Seth Hall) * Temporarily removing scripts that aren't ready to be included. Will return before next release. (Seth Hall) * New SSL policy scripts. (Seth Hall) - protocols/ssl/expiring-certs uses time based information from certificates to determine if they will expire soon, have already expired, or haven't yet become valid. - protocols/ssl/extract-certs-pem is a script for taking certs off the line and converting them to PEM certificates with the openssl command line tool then dumping them to a file. * Notice::type_suppression_intervals: table[Notice::Type] of interval can be used to modify the suppression intervals for entire types of notices. (Seth Hall) * EOF SSL protocol violations are only generated a single time now. (Seth Hall) * Script level fixes. (Seth Hall) - Fixed a type name conflict in the Known namespace. - Fixed a DPD framework bug that was causing Reporter messages. - Fixed the notice_policy log. - Predicate functions are now logged. - Predicate functions are now optional. If not given, it's assumed that the result should always apply. (Seth Hall) - Fix a problem with accidental and mistaken HTTP log lines. 1.6-dev-1293 | 2011-09-22 19:44:37 -0700 * Smaller script tweaks. (Seth Hall) * Duplicate notice suppression. (Seth Hall) - Duplicate notices are discovered with the new Notice::Info field $identifier. It's a string that is left up to the notice implementor to define which would indicate a fundamentally duplicate notice. The field is optional and if it's not included it's not possible for notice suppression to take place. - Duplicate notices are suppressed by default for the interval defined by the Notice::default_suppression_interval variable (1 hour by default). - A new notice action was defined ACTION_NO_SUPPRESS to prevent suppression for a specific notice instance. A convenience set named not_suppressed_types was also created to not suppress entire notice types. - A new field was added to the PolicyItem type to modify the length of time a notice should be suppressed if the predicate matches. The field is named $suppress_for. This name makes the code more readable like this: $suppress_for = 1day - New events were created to give visibility into the notice framework's suppression activity. - event Notice::begin_suppression(n: Notice::Info) - event Notice::suppressed(n: Notice::Info) - event Notice::end_suppression(n: Notice::Info) - The suppression.bro script doesn't have a baseline because it is causing a segfault in Bro. This one test is the reason that this is being integrated into a branch instead of master. (Seth Hall) * Fix crash on exit. Addresses #607. (Jon Siwek) * Fix PktSrc setting next_timestamp even when no packet available. (Jon Siwek) * Fix lack of NUL-termination in to_upper/to_lower BIF's return val. (Jon Siwek) * Fixing unit tests and some minor bugs. (Jon Siwek) * Fix broctl cluster log rotation. Addresses #619. (Jon Siwek) * Added session ID to the SSL logging. (Seth Hall) * Adding "install-aux" target + updating bro-aux submodule. (Jon Siwek) * Cleaning up INSTALL and README. (Jon Siwek) * Remove $Id$ tags. (Jon Siwek) * Remove policy.old directory. Addresses #511. (Jon Siwek) * Small rework with ssl base script to reduce memory usage. (Seth Hall) * Updated the mozilla root certs. (Seth Hall) 1.6-dev-1261 | 2011-09-15 17:13:55 -0700 * Memory leak fixes. Addresses #574 (Jon Siwek) * Add configure options for ruby/bindings integration. (Jon Siwek) * Fix filter path_func to allow record argument as a subset of stream's columns. Addresses #600. (Jon Siwek) * Log rotation is now controlled directly through Filter records. (Jon Siwek) * Fix indexing for record types with optional fields. Addresses #378 (Jon Siwek) 1.6-dev-1248 | 2011-09-15 16:01:32 -0700 * Removed custom malloc() implementation for FreeBSD. Closes #557. (Jon Siwek) * Testing/external scripts no longer compute MD5 checksums for SMTP entities. (Robin Sommer) * External tests no longer include the full content of mismatching files in the diagnostics output. (Robin Sommer) 1.6-dev-1241 | 2011-09-14 22:51:52 -0400 * Fixing a major memory utilization issues with SSL analysis. (Seth Hall) * Enhancements to HTTP analysis: (Seth Hall) - More options for the header-names.bro script. - New script for logging header names and values. Closes #519. (Seth Hall) - HTTP body size measurement added to http.log. - The value of the content-length headers has now been removed in the default output but it could be added back locally at an installation by a user. - Added fields to indicate if some parsing interruption happened during the body transfer. Closes #581 (Seth Hall) * Misc smaller usability and correctness updates: (Seth Hall) - Removed an notice definition from the base SSL scripts. - Moved a logging stream ID into the export section for known-services and bumped priority for creating the stream. - Adding configuration knobs for the SQL injection attack detection script and renaming the HTTP::SQL_Injection_Attack notice to HTTP::SQL_Injection_Attack_Against - Bumped priority when creating Known::CERTS_LOG. - Fixing a warning from the cluster framework. (Seth Hall) * Bugfix for log writer, which didn't escape binary stuff in some situations. Closes #585. (Robin Sommer) * A larget set of changes to the testing/external infrastructure. The traces for external test-suites are no longer kept inside the repositories themselves but downloaded separately via curl. This is because git is pretty bad at dealing with large files. See the README for more information. (Robin Sommer) 1.6-dev-1221 | 2011-09-08 08:41:17 -0700 * Updates for documentation framework and script docs. (Jon Siwek) * The script level PF_RING support isn't working so removing it. (Seth Hall) * Delete SSL certificates from memory after ssl_established event. (Seth Hall) * Small fixes for SSL analysis. (Seth Hall) 1.6-dev-1212 | 2011-09-07 16:15:28 -0700 * Internally, the UID generation can now return values from different pool for better reproducability in testing mode. (Gilbert Clark). * Added new BiF unique_id_from(pool: string, prefix: string) that allows the user to specify a randomness pool. (Gilbert Clark) 1.6-dev-1198 | 2011-09-07 11:03:36 -0700 * Extended header for ASCII log that make it easier for scripts to parse Bro log files. (Gilbert Clark) * Potential fix for rotation crashes. Addresses #588. (Robin Sommer) * Added PF_RING load balancing support to the scripting layer, enabled by loading the misc/pf-ring-load-balancing script. (Seth Hall) * Added a BiF setenv() for setting environment variables. (Seth Hall) 1.6-dev-1184 | 2011-09-04 09:34:50 -0700 * FindPCAP now links against thread library when necessary (e.g. PF_RING's libpcap). (Jon Siwek) * Install binaries with an RPATH. (Jon Siwek) * Fix for a case where nested records weren't coerced even though possible. (Jon Siwek) * Changed ASCII writer to delay creation of log after rotation until next write. * Changed default snaplen to 65535 and added a -l/--snaplen command line option to set it explicitly. Addresses #447. (Jon Siwek) * Various updates to logging framework. (Seth Hall) * Changed presentation of enum labels to include namespace. (Jon Siwek) * HTTP analyzer is now enabled with any of the HTTP events. (Seth Hall) * Fixed missing format string that caused some segfaults. (Gregor Maier) * ASCII writer nows prints time interval with 6 decimal places. (Gregor Maier) * Added a Reporter::fatal BIF. (Jon Siwek) * Fixes for GeoIP support. Addresses #538. (Jon Siwek) * Fixed excessive memory usage of SSL analyzer on connections with gaps. (Gregor Maier) * Added a log postprocessing function that can SCP rotated logs to remote hosts. (Jon Siwek) * Added a BiF for getting the current Bro version string. (Jon Siwek) * Misc. doc/script/test cleanup. (Jon Siwek) * Fixed bare-mode @load dependency problems. (Jon Siwek) * Fixed check_for_unused_event_handlers option. (Jon Siwek) * Fixing some more bare-mode @load dependency issues (Jon Siwek) * Reorganizing btest/policy directory to match new scripts/ organization. Addresses #545 (Jon Siwek) * bro scripts generated from bifs now install to $prefix/share/bro/base. Addresses #545 (Jon Siwek) * Changeed/fixed some cluster script error reporting. (Jon Siwek) * Various script normalization. (Jon Siwek) * Add a test that checks each individual script can be loaded in bare-mode. Adressess #545. (Jon Siwek) * Tune when c$conn is set. Addresses #554. (Gregor Maier) * Add ConnSize_Analyzer's fields to conn.log. (Gregor Maier) * Fixing bug in "interesting hostnames" detection. (Seth Hall) * Adding metrics framework intermediate updates. (Seth Hall) 1.6-dev-1120 | 2011-08-19 19:00:15 -0700 * Fix for the CompHash fix. (Robin Sommer) 1.6-dev-1118 | 2011-08-18 14:11:55 -0700 * Fixing key size calculation in composite hash code. (Robin Sommer) 1.6-dev-1116 | 2011-08-18 10:05:07 -0700 * Remove the 'net' type from Bro (addresses #535). * Fix H3 assumption of an 8-bit byte/char. (Jon Siwek) * Allow reading from interface without additional script arguments. Explicitly passing in '-' as an additional command line argument still allows reading a script from stdin. (Jon Siwek) * SSH bruteforcing detection now done with metrics framework. (Seth Hall) * Updates for SQL injection attack detection to match the metrics framework updates. (Seth Hall) * Metrics framework now works on cluster setups. (Seth Hall) * Reclassifying more DNS manager errors as non-fatal errors. (Robin Sommer) * Fix ConnSize_Analyzer when used in conjunction with connection compressor. (Gregor Maier) * Fix reporter using part of the actual message as a format string. (Jon Siwek) 1.6-dev-1095 | 2011-08-13 11:59:07 -0700 * A larger number of script documentation updates. Closes #543. (Jon Siwek) * Workaround for FreeBSD CMake port missing debug flags. (Jon Siwek) * piped_exec() can now deal with null bytes. (Seth Hall) * Fix vector initialization for lists of records with optional types. Closes #485. (Jon Siwek) * Fix redef'ing records with &default empty set fields. Closes #460. (Jon Siwek) * Fix ConnSize_Analyzer when used in conjunction with the connection compressor. (Gregor Maier) * Fix reporter using part of the actual message as a format string. (Jon Siwek) * Fixing reporter's location tracking. Closes #492. (Robin Sommer) * Turning DNS errors into warnings. Closes #255. (Robin Sommer) * Logging's path_func now receives the log record as argument. Closes #555. (Robin Sommer) * Functions can now be logged; their full body gets recorded. Closes #506. (Robin Sommer) * Bugfix for hostname notice email extension. (Seth Hall) * Updates for notice framework. (Seth Hall) - New ACTION_ADD_GEODATA to add geodata to notices in an extension field named remote_location. - Loading extend-email/hostnames by default now that it only does anything when the ACTION_EMAIL action is applied (finally). * Updates to local.bro (Seth Hall) * Added the profiling script. (Seth Hall) * Updates for SSH scripts. (Seth Hall) * ConnSize analyzer is turned on by default now. (Seth Hall) * Updates for the build system and site local scripts for cluster. (Seth Hall) * HTTP now uses the extract_filename_from_content_disposition function. (Seth Hall) * Major SMTP script refactor. Closes #509. (Jon Siwek and Seth Hall) * New variable Site::local_nets_table in utils/site for mapping address to defined local subnet. * Metrics framework updates, more to come. (Seth Hall) 1.6-dev-1061 | 2011-08-08 18:25:27 -0700 * A set of new/changed tests regarding the new policy script organisation. (Robin Sommer) 1.6-dev-1058 | 2011-08-08 16:15:18 -0700 * Reorganisation of the scripts that Bro loads by default. (Seth Hall) - policy/ renamed to scripts/ - By default BROPATH now contains: - scripts/ - scripts/policy - scripts/site - The scripts in scripts/base/protocols/ only do logging and state building. - All of scripts/base/ is loaded by by default. This can however be disabled by switching Bro into "bare mode" using the new command-line option --bare-mode (or -b). The cripts in scripts/base/ don't use relative path loading to ease use of bare mode (to copy and paste that script). - The scripts in scripts/base/frameworks/ add functionality without causing any additional overhead. - All "detection" activity happens through scripts in scripts/policy/. - bro.init was renamed to base/init-bare.bro, and base/all.bro was renamed to init-default.bro. - local.bro now loads more functionality from policy/ and adds more documentation. (Seth Hall) * Adding default_path_func() to the logging framework that makes the default naming scheme script-level controlled. (Robin Sommer) * Reworking logging's postprocessor logic so that postprocessor commands are no longer run by the log writers themselves, but instead by a script level function. (Robin Sommer) * The communication subsystem is now by default off and must be enabled explicitly with a new BiF, enable_communication(). Closes #540. (Robin Sommer) * The hostname notice email extension now only add hostnames for emailed noticed. (Seth Hall) * Cleaning up doc generation. (Seth Hall) 1.6-dev-1044 | 2011-08-05 19:07:32 -0700 * Fixing memory (and CPU) leak in log writer. * Fixing crash in memory profiling. (Robin Sommer) * Fix compiler warning. (Robin Sommer) * Fixing missing sync in cluster setup. (Robin Sommer) 1.6-dev-1038 | 2011-08-05 18:25:44 -0700 * Smaller updates to script docs and their generation. (Jon Siwek) * When using a `print` statement to write to a file that has raw output enabled, NUL characters in string are no longer interpreted into "\0", no newline is appended afterwards, and each argument to `print` is written to the file without any additional separation. (Jon Siwek) * Test portatibility tweaks. (Jon Siwek) * Fixing PktSrc::Statistics() which retured bogus information offline mode. Closes #500. (Jon Siwek) * --with-perftools configure option now assumes --enable-perftools. Closes #527. (Jon Siwek) 1.6-dev-1018 | 2011-07-31 21:30:31 -0700 * Updating CHANGES. (Robin Sommer) 1.6-dev-1016 | 2011-07-30 18:34:28 -0700 * Install example config files dynamically. They'll only get installed when the distribution version differs from existing version on disk. (Jon Siwek) * Fixed memory leak in SSL analyzer. (Seth Hall) * Beginning rework of metrics interface. (Seth Hall) * New/updated unit tests for scripts. (Jon Siwek) * New/updated documentstion for scripts. (Jon Siwek) * A number of fixes for scripts in utils/. (Jon Siwek) 1.6-dev.244 Thu Jul 28 17:08:21 PDT 2011 - mask_addr() now returns subnet (addresses #512). (Jon Siwek) - Normalize Notice::Type identifiers per convention (closes #484). (Jon Siwek) - Fixing default-loaded-scripts test for BSD systems. (Jon Siwek) - New piped_exec() BiF for pipeing data into an external command. (Jon Siwek) 1.6-dev.242 Mon Jul 25 21:42:39 PDT 2011 - Adding a documentation coverage test. (Jon Siwek) - The CMake targets for generating reST docs from policy scripts are now automatically generated via the genDocSourcesList.sh script. (Jon Siwek) - Fixed a number of script error. (Jon Siwek) - Fixes to relative @load'ing. (Jon Siwek) - Fixes to tests. (Robin Sommer) 1.6-dev.240 Sun Jul 24 15:14:26 PDT 2011 - Updated tests and test baselines. (Jon Siwek) - ASCII log writer now prints time values w/ constant 6 digit precision. (Jon Siwek) - Many policy script updates acrsso the board (Seth Hall). - Moving devel-tools to bro-aux. (Robin Sommer) - BugFix for disable_analyzer(), which could cause crashes with some analyzers. (Robin Sommer) - Bugfix for potential segfault in DebugLogger. (Robin Sommer) 1.6-dev.226 Thu Jul 21 15:23:39 PDT 2011 - Extensions to the @load and @unload process. (Jon Siwek) * Make @load statements recognize relative paths. For example a script can do "@load ./foo" to load a script named foo.bro that lives in the same directory or "@load ../bar" to load a script named bar.bro in the parent directory, even if those directories are not contained in BROPATH. * Reimplementation of the @prefixes statement. (Closes #486) Any added prefixes are now used *after* all input files have been parsed to look for a prefixed, flattened version of the input file somewhere in BROPATH and, if found, load it. For example, if "lcl" is in @prefixes, and site.bro is loaded, then a file named "lcl.site.bro" that's in BROPATH would end up being automatically loaded as well. Packages work similarly, e.g. loading "protocols/http" means a file named "lcl.protocols.http.bro" in BROPATH gets loaded automatically. * Fix @unload'd files from generating bro_script_loaded event. * Updates to tests. 1.6-dev.225 Wed Jul 20 17:10:41 PDT 2011 - IRC improvements (Jon Siwek). Including: * Shorten what's displayed in the IRC's log mime_type column for DCC transfers. * Add IRC unit tests. * Fix IRC analyzer supplying wrong type to irc_dcc_message event. * Removed irc_client and irc_server events. * Added is_orig arguments to all other irc events. * Fix analyzer not recognizing Turbo DCC extension message format. * Fix analyzer not generating irc_dcc_message event when irc_privmsg_message event doesn't have a handler registered. - Fixing tests that need a diff canonifier. (Jon Siwek) 1.6-dev.223 Tue Jul 19 19:10:36 PDT 2011 - Adding a script to update CHANGES and VERSION. (Robin Sommer) 1.6-dev.218 Tue Jul 19 18:16:44 PDT 2011 - Comprehensive policy script overhaul/rewrite. (Seth Hall) Changes are too extensive to list individually. - Removing undocumented -H command line flag. (Robin Sommer) - Fixing many tests. (Everybody) - Fixing 0-chunk bug in remote logging. (Robin Sommer) - $PATH is now appropriately set by the bro-path-dev.(sh|csh) scripts. (Seth Hall) - Making valgrind a bit more happy. (Robin Sommer) - New BiF record_field_vals() that returns the fields of a record in a table with meta-information. (Robin Sommer) - Adding a script in aux/devel-tools that extracts a connection from a trace based on uid. (Robin Sommer) - Fixing bug causing crash when running without arguments. (Robin Sommer) - A new event bro_script_loaded() raised for each policy script loaded. Also removing the -l command-line option as that can now be done at the script-level. (Robin Sommer) - Fixing memory leaks. (Gilbert Clark, Seth Hall, Robin Sommer) - Many SSL analysis improvements and fixes. (Seth Hall) - Fixing bug with event priorities potentially being ignored for the handler. (Robin Sommer) - Overhauling the internal reporting of messages to the user. The new Reporter class is now in charge of reporting all errors, warnings, informational messages, weirds, and syslogs; and it passes everything through the script layer. (Robin Sommer) * Removed the alarm statement and the alarm_hook event. (Robin Sommer) - Adding new policy file test-all.bro that loads all other policies. This is for testing only. (Robin Sommer) - A new framework for doing regression testing with larger traces and more complex Bro configurations in testing/external. (Robin Sommer) - Many updates to script doc generation. (Jon Siwek) 1.6-dev.146 Sat Jun 25 18:12:27 PDT 2011 - DNS mapping are now becoming invalid when an entry's TTL expires. (Thomas Other) - Reworking how Bro tracks which scripts are already loaded. Rather than paths, Bro now tracks inode numbers. (Jon Siwek) - New BiF netstats() to query packet capture statistics. The netstats script now uses the new BiF to periocally report packets drops. The net_stats_update() event and the heartbeat_interval global went away. (Seth Hall) - Fixing bug with logging &optional records. Closes #476. (Robin Sommer) - Fixing istate.events-ssl test failing because of expired cert. (Jon Siwek) - A large number of improvements and fixes for Bro's doc mode. (Jon Siwek) - Significant updates for RPC and NFS analyzers (Gregor Maier) * Unify semantics for UDP and TCP connections. * RPC can now log to a log file if desired. * Portmapper can now log general activity to a log file and also log actual port mappings. * NFS analyzer now supports significantly more procedure calls as as file name tracking and file content extraction. - NetBIOS fixes. (Jon Siwek) - A number of unit tests are more robust and portable. (Jon Siwek) - A new BiF unique_id() that returns a string that's unique across Bro instaces with high probablity. (Robin Sommer) - Complete rewrite of the BinPAC SSL analyzer. (Seth Hall) * DER certificates are extracted as strings to be used with corresponding BiFs. * x509_verify function to verify single certs and/or full certificate chains. * Removed hand written SSL analyzer. * The ssl.bro script is just a place-holder for now. New version will come with the other new scripts. - New syslog analyzer. (Seth Hall) - @load now supports loading a directory. With a directory "foo" somewhere in BROPATH, "@load foo" now checks if there's a file "foo/__load__.bro". If so, it reads that file in. (Robin Sommer) - ASCII logger now escapes non-printable characters. Closes #450. (Robin Sommer) - Packaging tweaks and rewrite of 'dist' target. (Jon Siwek) - Changes to allow DEB packaging via CPack, addresses #458. (Jon Siwek) - An extension to the ICMP analyzer to handle redirects. Julien Sentier - Removing old istate test-suite. (Robin Sommer) - A hack to report missing GeoIP support only once. This closes #357, but #455 captures the need for a more general solution. (Robin Sommer) - Bugfix: vectors in records were not initalized. Closes #421. (Robin Sommer) - If IPv6 default is not compiled in, the default BPF filters now excludes IPv6 packets. (Robin Sommer) - New bif bro_has_ipv6() to check whether IPv6 support is compiled in. (Robin Sommer) - Updating btests and a Makefile. "make" now runs all the tests. (Robin Sommer) - Moving the test-scripts from the old test-suite over to btest. (Robin Sommer) - Fix for major bug in POP3 analyzer, which didn't recognize '.' terminators in multi-line replies if the terminator was bare (no newline). This caused it to ignore the rest of the session that it's analyzing. (Vern Paxson) - Fix compiler warning with gcc-4.4.4 (Gregor Maier) - Adding example documentation for a script's use of logging features. (Jon Siwek) - Adding &log attribute to static attr_names array. (Jon Siwek) - Bro can now track packet and byte counts per connection. (Gregor Maier) * If 'use_conn_size_analyzer' is true, the event engine tracks number of packets and raw IP bytes per connection. If report_conn_size_analyzer is true, these values are included as four new columns into conn.log * I changed conn.bro so that the value of report_conn_size_analyzer follows that of use_conn_size_analyzer. For the new conn.log, we probably want to get rid of report_conn_size_analyzer anyway. - Fixing numerous compiler warnings and portability issues. (All) - Switching vectors from being 1-based to 0-based. Note that this is a change that break backwards-compatibility. (Robin Sommer) - Increasing serialization format version for the recent 64-bit changes. (Robin Sommer) - Support for (mixed) MPLS and VLAN traffic, and a new default BPF filter. (Seth Hall and Robin Sommer) * Merging in the patch from #264, which provides support for mixed VLAN and MPLS traffic. * Changing Bro's default filter from being built dynamically to being a static "ip or not ip". To get the old behaviour back (i.e., the dynamically built filter), redef "all_packets" to false. * print-filter.bro now always prints the filter that Bro is actually using, even if overriden from the command line. (Robin Sommer) - Changing the HTTP's analyzers internals to use 64-bit integers. (Gregor Maier). - Fixing bug with deleting still unset record fields of table type. (Robin Sommer) 1.6-dev.99 Fri Apr 22 22:10:03 PDT 2011 - Extending the connection record with a unique identifier. (Robin Sommer) type connection: record { [...] id: string; }; These identifiers very likely unique even across independent Bro runs. - Delete operator for record fields. (Robin Sommer) "delete x$y" now resets record field "x" back to its original state if it is either &optional or has a &default. "delete" may not be used with non-optional/default fields. - Fixing bug with nested record coercions. (Robin Sommer) - Fixing a do_split() bug. (Seth Hall) 1.6-dev.94 Thu Apr 21 19:51:38 PDT 2011 - Fixing generation of config.h. (Jon Siwek) - Updates and tests for NetBIOS name BiF. (Seth Hall) - Fixing do_split bug(), and adding a test. (Seth Hall) - When Bro is given a PRNG seed, it now uses its own internal random number generator that produces consistent results across sytems. Note that this internal generator isn't very good, so it should only be used for testing purpses. (Robin Sommer) - The BTest configuration now sets the environemnt variables TZ=UTC and LANG=C to ensure consistent results. (Robin Sommer) - Logging fixes. (Robin Sommer) 1.6-dev.88 Wed Apr 20 20:43:48 PDT 2011 - Implementation of Bro's new logging framework. We will document this separately. (Robin Sommer) - Already defined record types can now be further extended via the '+=' operator. The added fields must be either &optional or have a &default value. (Robin Sommer) Example: type Foo: record { a: count; b: count &optional; }; redef record Foo += { c: count &default=42; d: count &optional; }; global f: Foo = [$a=21]; print f; Output: [a=21, b=, c=42, d=] - Enabling assignment of empty vectors ("vector()"). (Robin Sommer) - Fixing attributes to allow &default attributes to be associated with records fields of type tables/sets/vector. (Robin Sommer) - '[]' is now a valid record constructor. (Robin Sommer) - A instance of a record type A is now coercable into one of type B if the fields of type A are a subset of those of type B. (Robin Sommer) - A number of bug fixes and enhancements for record/set/table/vector coercion. (Robin Sommer) - Fixing a problem with records that have optional fields when used as table/set indices. Addresses #367. (Robin Sommer) - Fixing an off-by-one error in join_string_vec(). (Seth Hall) - Updating to_count() to cope with 64bit ints. (Seth Hall) - A new BiF count_to_v4_addr() to turn a count into an IPv4 address. (Seth Hall) 1.6-dev.80 Mon Apr 18 14:50:54 PDT 2011 - New framework for generating documentation from Bro scripts. (Jon Siwek) This includes: * Changes to Bro's scanner/parser to facilitate automatic generation of Bro policy script documentation in reStructuredText format. * New command line flags -Z/--doc-scripts to enable the new doc generation mode. * Changes to bifcl to pass comments starting with "##" through into the generated .bro script. * A "doc" build target for the top-level Makefile to first generate reStructuredText for a defined set of Bro policy scripts, and then run that through Sphinx to create HTML documentation. 1.6-dev.78 Mon Apr 18 12:52:55 PDT 2011 - Adding files to CMake build targets so they show up in generated IDE projects. This addresses #413. (Jon Siwek) - Fix unnecessary config.h preprocessor (re)definitions. This addresses #414. (Jon Siwek) - Updating istate tests. (Robin Sommer) - Adding files to CMake build targets so they show up in generated IDE projects. - Adding new environment variable BRO_SEED_FILE to set the seed file for the random number generator. (Robin Sommer) 1.6-dev.71 Fri Apr 1 16:06:33 PDT 2011 - Removing code for the following no longer supported functionality. * Trace rewriting. * DFA state expiration in regexp engine. * Active mapping. * Unused hash functions. (Robin Sommer) - Fixing crashes when SSL is not configured correctly. (Robin Sommer) 1.6-dev.66 Tue Mar 29 21:52:01 PDT 2011 - Initial btest setup (Don Appleman and Robin Sommer) - Porting the istate tests to btest (not finished) (Robin Sommer) 1.6-dev.63 Mon Mar 21 16:31:15 PDT 2011 - Changes to the way user-modifiable config files are installed (Jon Siwek) * Duplicates of the distribution's configuration files are now always installed with a .example suffix * Added --binary-package configure option to toggle configure logic specific to the creation of binary packages. * When not in binary packaging mode, `make install` never overwrites existing configure files in case they've been modified. The previous behavior (CMake's default) would only avoid overwriting modified files if one consistently uses the same build directory and doesn't reconfigure. - Fixed an issue with Mac package's pre-install script not preserving ACLs. (Jon Siwek) - Minor cleanup/refactor of the make-mac/rpm-packages scripts. (Jon Siwek) - Add explicit CMake check for compiler. (Jon Siwek) - Add alternative way to set BROPATH for running bro from build/ dir. (Jon Siwek) - Fixing compiler warnings (Gregor Maier) - Remvoing leftover local variables that caused compile error on Mac OS X. (Gregor Maier) 1.6-dev.53 Fri Feb 25 17:03:05 PST 2011 - Fixing file detector leak in remote communication module. (Scott Campbell) - Updating independent-state tests to work with new setup. (Robin Sommer) 1.6-dev.49 Fri Feb 25 15:37:28 PST 2011 - Enum IDs can have explicitly defined values. (Gregor Maier) - Extensions for the built-in function compiler, bifcl. (Gregor Maier) * Support for policy-layer namespaces. * Support for type declarations in bif files (with access them from C++) * Extended const declarations in bif files. See http://bro.icir.org/devel/bif-doc for more information. 1.6-dev.48 Fri Feb 25 10:53:04 PST 2011 - Preliminary TCP Reassembler fix: deliver data after 2GB by disabling the unused seq_to_skip feature. (Gregor Maier) 1.6-dev.47 Fri Feb 25 10:40:22 PST 2011 - Fixing endianess error in XDR when data is not 4-byte aligned. (Gregor Maier) - Fix for Val constructor with new int64 typedefs. (Gregor Maier) - Updated fix for OS X 10.5 compile error wrt llabs(). (Gregor Maier) - Fix more compiler warning wrt printf format strings. (Gregor Maier) 1.6-dev.45 Tue Feb 8 21:28:01 PST 2011 - Fixing a number of compiler warnings. (Seth Hall and Robin Sommer) 1.6-dev.44 Tue Feb 8 20:11:44 PST 2011 - A number of updates to the SSL analyzer, including support for new ciphers; SSL extensions; and bug fixes. The analyzer does not longer throw weird for exceeding a predefined cipherspec_size anymore. (Seth Hall and Rmkml). - The various split*() BiFs now handle strings containing null bytes correctly. (Seth Hall) - Adding new aux/btest submodule. This is a framework we will use in the future for doing unit tests. (Robin Sommer) 1.6-dev.41 Mon Feb 7 13:43:56 PST 2011 - Smarter way to increase the parent/child pipe's socket buffer. (Craig Leres). - Fixing bug with defining bro_int_t and bro_uint_t to be 64 bits wide on some platforms. (Robin Sommer) 1.6-dev.39 Mon Jan 31 16:42:23 PST 2011 - Login's confused messages now go through weird.bro. (Robin Sommer) 1.6-dev.36 Mon Jan 31 08:45:35 PST 2011 - Adding more configure options for finding dependencies, (Jon Siwek) --with-flex=PATH path to flex executable --with-bison=PATH path to bison executable --with-perl=PATH path to perl executable --with-python=PATH path to Python interpreter --with-python-lib=PATH path to libpython --with-python-inc=PATH path to Python headers --with-swig=PATH path to SWIG executable - Fixing typo in PCAPTests.cmake (Jon Siwek) 1.6-dev.33 Mon Jan 24 15:29:04 PST 2011 - Fixing bug in SMB analyzer. (Robin Sommer) - Configure wrapper now deletes previous CMake cache (Jon Siwek) - Fix for the --with-binpac configure option. (Jon Siwek) 1.6-dev.30 Thu Jan 20 16:32:43 PST 2011 - Changed configure wrapper to create config.status. (Jon Siwek) 1.6-dev.29 Thu Jan 20 16:29:56 PST 2011 - Fixing little problem with initialization of Bro-to-Bro event communication. (Christian Kreibich) 1.6-dev.27 Thu Jan 20 13:52:25 PST 2011 - Fine-tuning of the HTTP analyzer in terms of raising protocol violations and interrupted transfers. (Gregor Maier) 1.6-dev.21 Wed Jan 19 17:36:02 PST 2011 - Added 4 new BiFs and a new record type for testing the entropy of strings. (Seth Hall) find_entropy(data: string): entropy_test_result This is a one shot function that accepts a string and returns the result of the entropy calculations. entropy_test_init(index: any): bool This and the next two functions are for calculating entropy piece-wise. It only needs an index which can be any type of variable. It needs to be something that uniquely identifies the data stream that is currently having it's entropy calculated. entropy_test_add(index: any, data: string): bool This function is used to add data into the entropy calculation. It takes the index used in the function above and the data that you are adding and returns true if everything seemed to work, false otherwise. entropy_test_finish(index: any): entropy_test_result Calling this function indicates that all of the desired data has been inserted into the entropy_test_add function and the entropy should be calculated. This function *must* be called in order to clean up an internal state tracking variable. If this is never called on an index, it will result in a memory leak. The entropy_test_result values have several measures of the entropy, but a good one to work with is the "entropy" attribute. It's a double and as the value approaches 8.0 it can be considered more and more random. For example, a value of 7.832 would be quite random but a value of 4.671 is not very random. 1.6-dev.20 Wed Jan 19 17:30:11 PST 2011 - BRO_DNS_FAKE is now listed in the --help output. (Seth Hall) 1.6-dev.18 Wed Jan 19 16:37:13 PST 2011 - Removing unnecessary expire timer from http_sessions. (Gregor Maier) 1.6-dev.16 Sat Jan 15 14:14:21 PST 2011 - Updates to the build system. (Jonathan Siwek) * ``make dist`` is now available to be used with the top-level Makefile for creating source packages according to #344. * ``make-rpm-packages`` and ``make-mac-packages`` scripts can now generate binary packages according to #295. * Additional configure options to change packaging behavior. * OS X builds will now prefer to link static libraries of optional dependencies that don't come with the vanilla operating system. * Fix for OS X 10.5 compile error dealing with the llabs() function from stdlib. * Installing as a different user than the one that configured/built now works (although, a harmless error message about not being able to write the install manifest may occur). 1.6-dev.3 Wed Dec 8 04:09:38 PST 2010 - Merge with Subversion repository as of r7137. Incorporated change: * Fix for packet processing resumption when a remote Bro dies during state synchronization (Robin Sommer). 1.6-dev.2 Wed Dec 8 03:57:03 PST 2010 - Compatibility fix for OpenSSL 1.0.0 (Christian Kreibich, Gregor Maier). 1.6-dev.1 Sat Nov 27 12:19:47 PST 2010 - Merge with Subversion repository as of r7098. Incorporated changes: * Rotation post-processors are now passed an additional argument indicating whether Bro is terminating (Robin Sommer). * Bro now consistently generates a file_opened event for all fopen() calls. (Robin Sommer). * You can now redefine the email_notice_to function (Robin Sommer). 1.6-dev.0 Fri Nov 26 13:48:11 PST 2010 - The Bro source code is now developed in the new git repositories. See the developer pages at http://www.bro-ids.org for more information on the new development process. - Bro's build and installation setup has been moved from GNU autotools to CMake. As a result of that, layout and specifics of the distribution has changed significantly. - Lots of pieces have been removed from the distribution that are either now unnecessary or are no longer maintained. - As part of the cleanup, a numbef of Bro configure options and their corresponding functionality have been removed, including: * --disable-select-loop * --with-dag * --disable-nbdns * --enable-activemapping * --enable-activemapping * --enable-shippedpcap - The previous configure option --enable-int64 is now enabled by default, and can no longer be disabled. - ClamAV support has been removed, which has been non-functional for a while already. 1.5.2.7 Sun Sep 12 19:39:49 PDT 2010 - Addressed a number of lint nits (Vern Paxson). 1.5.2.6 Sun Sep 12 17:00:13 PDT 2010 - The SWIG file now explicitly lists those pieces from broccoli.h which it wants to wrap, rather than just including all of broccoli.h (Robin Sommer). This fixes the problem that the SWIG bindings depend on what configure finds out about the availability of libpcap even though the corresponding functions don't need to be wrapped anyway. - http-header.bro now includes a global include_header: set[string] (Robin Sommer). If it contains any strings, then only those headers will be processed. If left empty, then you continue to get the current behavior of processing all headers. - Several changes to drop.bro (Robin Sommer): * If true, the new flag Drop::dont_drop_locals indicates that local hosts should never be dropped. On by default. * If true, the new flag Drop::debugging activates extensive debugging output for the catch-and-release logic. Off by default. * The timeout for tracking dropping information is now 1 day rather than 7 days, to better echo the one-restart-a-day semantics used in the past. * Bug fix for hosts once dropped by later cleared; some state for them persisted. - Portability fix for Broccoli Python bindings on 64-bit platforms (Robin Sommer). - The HTTP analyzer no longer attempts to track Server/User-Agent versions, as these are hugely voluminous (Seth Hall). Ideally this would still be available as an option for someone who truly wants the full set. - HTTP and SMTP no longer have extra-short inactivity timeouts, as these were too often leading to premature expiration of a connection (Robin Sommer). - The "rst" tool (aux/rst/) now takes an optional "-I " argument that instructs it to inject as payload rather than sending a RST packet (Vern Paxson). must be NUL-terminated, and the NUL is not included. - Bug fix for crashes in the DNS analyzer when processing replies for which no request was seen (Robin Sommer). 1.5.2.5 Mon Jul 19 16:20:58 PDT 2010 - Removed now-quite-stale SSHv1 overflow detection, as it's more prone to false positives than useful detection (Vern Paxson). 1.5.2.4 Fri Jun 4 16:02:11 PDT 2010 - Bug fixes for terminating connections (Tyler Schoenke and Vern Paxson). 1.5.2.3 Wed Mar 24 18:23:57 PDT 2010 - Bug fixes for --enable-int64 and for avoiding bogus statistics / bad memory references when generating profiling information upon exit (Vern Paxson). 1.5.2.2 Tue Jan 12 12:33:42 PST 2010 - Broccoli compiler warning fixes (Kevin Lo). 1.5.2.1 Sun Jan 10 16:59:01 PST 2010 - Bug fix for Active Mapping support (Kevin Lo). 1.5.2 Sat Dec 26 18:38:37 PST 2009 - Portability fixes for --enable-int64 (Vern Paxson). 1.5.1 Fri Dec 18 15:17:12 PST 2009 - Due to a Python configuration problem, the original 1.5 distribution did not include the BroControl component, which also introduced a portability problem for CentOS. These issues have now been fixed (Robin Sommer and Vern Paxson). 1.5 Wed Dec 16 21:28:47 PST 2009 - Bro now comes with a new framework, BroControl, for managing an operational Bro setup, including support for installation, configuration, and maintainance tasks such a log archival and mail notification. The framework transparently supports both traditional standalone setups as well as cluster installations in which multiple Bro boxes coordinate to analyze a high-volume network link. See aux/broctl/README for more information about BroControl. Note, BroControl supersedes the older BroLite system, which is no longer supported and has been deprecated for a while now. - Numerous adjustments to DPD = dynamic protocol detection (Robin Sommer): o The Analyzer::ProtocolViolation?() method can now be passed the offending data (which POP3, SMTP, and FTP now do). This information is added to the "reason" string passed to the script level. o SMTP now more accurately reports violations. o FTP stops processing when client & server successfully negotiate an AUTH scheme (leading to subsequent encryption). o Analyzer::ProtocolViolation() is virtual, and TCP_ApplicationAnalyzer() overrides it to not report violations for any partial connections, because very likely these arise just due to the analyzer getting confused. o TCP::IsPartial() returns true if any side did not start with a SYN packet (used to be just be for the originator). o The connection_state_remove handler in conn.bro now has a higher &priority so that other handlers for the same event can use determine_service() and see any changes it performs. o DynDisable:max_volume specifies a volume limit (default 10K). Once a connection exceeds this limit, further protocol limitations will neither raise ProtocolViolation notices nor cause the analyzer to be disabled. o The event engine no longer raises protocol_violation events for TCP connections which had gaps, as these have proven too unreliable. (Note that, ideally, the *analyzers* should avoid reporting protocol_violations when they can't reliably parse a connection anymore after a gap; but many don't.) - A set of new script functions provide support for incrementally computing MD5 checksums (Seth Hall). md5_hash_init(index: any): bool Initializes an incremental hashing instance. "index" is a value of arbitrary type, used to identify this particular instance (you can have multiple concurrent instances by using different index values). Returns T on success, F on failure (such as the index is already in use). md5_hash_update(index: any, data: string): bool For the given hashing instance, updates the hash based on the given data. Returns T on success, F on failure (such as the index has not been initialized). md5_hash_finish(index: any): string Returns the MD5-printable hash for the given index and terminates the instance, or the string "" if the index was not active. - Bro now supports a believed-to-be-robust mechanism for estimating the proportion of traffic that it failed to capture ("measurement drops"), which can arise due to overload in either Bro itself, the kernel's packet filter, or problems with the link tapping mechanism (Vern Paxson). The event engine can generate estimates for either live traffic or what was previously recorded in a trace file, though traces subject to some forms of selective omission (such as skipping over parts of a connection to reduce storage) can lead to erroneous values. The estimates are based on observing gaps in TCP data streams, and come in two forms: the rate at which such gaps appear, and the relative volume of data missing due to the gaps. (We've found however that the volume-based estimator is not robust due to occasional packets with incorrect sequence numbers, so this estimator is off by default.) The easy way to get the estimates is to load capture-loss.bro. By default, it generates a CaptureLossSummary notice upon Bro's exit, which can look like: 1130222759.344066 CaptureLossSummary estimated rate = 0.00089124 / 0.000970997 (events/bytes) If the estimated loss is none, however, it suppresses this notice, unless you redef CaptureLoss::summary_if_none to T. You can also get finer-grained access by defining a "gap_report" event handler and redef'ing gap_report_freq to a non-zero interval (such as "10 sec"). This event allows you to pinpoint regions in time that exhibit significant capture loss. See capture-loss.bro for an example of a handler for this event. Finally, these changes include a number of fixes to Bro's ack_above_hole/content_gap analysis, which is now significantly more robust. - GeoIP support now supports ASN lookups via the built-in function lookup_asn(a: addr): count (Scott Campbell and Seth Hall). - The GeoIP built-in's lookup_location() and lookup_asn() now support IPv6 (Seth Hall). Note, the current GeoIP distribution doesn't include any IPv6 databases, so for now these won't succeed, but the hooks are in place for when databases become available. - lookup_location() now falls back back to the country database if the city database isn't available (Seth Hall). - The new SuccessfulPasswordGuessing Notice is generated when a host has been seen attempting password guessing (currently only for FTP sessions) and then successfully logs in (Royal Chan). You can control the threshold for such reports in terms of how many attempts the host must have made by redef'ing the variable password_guessing_success_threshhold, which defaults to 20. - The new script http-detect-passwd.bro analyzes the Web items returned for fetches that appear to be accessing the passwd file (Akhil Dhar). It generates a PasswordFullFetch Notice if it appears that the item includes a full password file, and PasswordShadowFetch if it looks like a shadowed password file. - The new built-in system_env(cmd: string, env: table[string] of string) works like system(), but puts the table entries into the environment before invoking the command (Robin Sommer). Each in the table creates an environment variable of the form "BRO_ARG_", whose value is the corresponding table entry. - The new script function execute_with_notice(cmd: string, notice_info) executes "cmd" with an environment containing the fields of the notice_info, i.e., the information associated with a Notice (Robin Sommer). Per the new system_env() function above, the environment variables appear as "BRO_ARG_", where is the field tag as it appears in notice.log when you enable use_tagging. - The new built-in enable_raw_output(file) acts the same as the attribute &raw_output (Seth Hall). - The new built-in file_opened(f: file) event is generated any time Bro opens a script-level file (Justin Azoff). You can use this, for example, if you want to ensure that a given file has a prelude in it such as human-readable headers, even when the file is rotated. - The notice_info record has a new field aux: table[string] of string &optional which you can use for information specific to a given type of notice (Robin Sommer). Entries in $aux appear as "aux_" tags in notice.log. - Another new notice_info record field is the boolean do_alarm (default=T), which, if set to F, overides a notice action otherwise specifying to generate an alarm (Robin Sommer). In other words, if do_alarm is F, no alarm will be generated independent of the notice action. This is a work-around for the fact that we can't specify more than one action. In particular, we couldn't NOTICE_DROP but then *not* alarm, which we now can by returning NOTICE_DROP yet setting do_alarm to F. - The notice_info record field $dropped now appears in the tagged output format if true (Robin Sommer). - NOTICEs relating to scan detection now no longer include the connection that triggered the notice, as it really doesn't contain any useful information, given that the particular trigger simply depends on the detection algorithm and its parameters (Robin Sommer). However, we do explicitly set $p (port number) in the notice, and also $n with the number of attempts. - drop.bro now hardwires a Catch-and-Release redrop after seeing one connection from a previously-dropped-but-already-released host (Robin Sommer). - drop.bro now provides some new hooks (Robin Sommer): event address_dropped(a: addr) Generated when an address has been dropped. event address_restored(a: addr) Generated when connectivity to an address has been restored, such as using the Catch-and-Release mechanism. event address_cleared(a: addr) Generated when an address that was dropped in the past is no longer being monitored looking for new connections (as part of the Catch-and-Release mechanism). - The new built-in function hexdump(data_str: string) : string returns a hex dump representation of the given input data (Christian Kreibich). The dump renders 16 bytes per line, with hex on the left and ASCII (where printable) on the right. - Bro's notion of when a TCP connection begins now dastes to the first instance of an initial SYN packet seen, rather than the last (Gregor Maier). - The Time Machine script tm-contents.bro now generates event contents_saved: event(c: connection, orig_file: string, resp_file: string) when the content of a connection has been completely saved to disk (Robin Sommer). - The mime.bro script now exports the MIME header callback table, and also marks it as &redef'able so you can modify its entries (Matthias Vallentin). The mime_log file is also now exported. - A new signature file, policy/sigs/http-bots.sig, contains signatures to detect some of the current HTTP based controlled bot families (Seth Hall). - The signature engine's HTTP pattern matching has been fixed (Seth Hall) to align with the documentation at: http://www.bro-ids.org/wiki/index.php/Reference_Manual:_Signatures#Content_conditions In particular, the content condition "http" is now referred to as "http-request" (though "http" still works for backward compatibility), "http-request-header" and "http-reply-header" now provide access to headers seen in only one direction, and similarly for "http-request-body" and "http-reply-body". (This latter is still accessible as "http-body" for backwards compatibility.) - The new script variable max_remote_events_processed: count (default 10) sets a limit on the number of remote events processed in each round, before tending to other inputs (Robin Sommer). - If you set the new script variable dump_used_event_handlers to T, then on startup Bro dumps out all of the event handlers that the loaded set of scripts can invoke (Matthias Vallenti). - Summaries for DNS PTR scanning now use a separate Notice, DNS_PTR_Scan_Summary, rather than overloading DNS_PTR_Scan (Robin Sommer). - scan.bro now provides a table skip_dest_server_ports: set[addr, port] which lists servers (defined as an address and a port) excluded from scan detection computations (Craig Leres and Jay Krous). - When redefining values on the command line directly (using var=value), quotation marks are now implicit only if "var" is a variable of type string (Christian Kreibich). This allows other string-like values (such as enum's) to be passed as well. - scan.bro now explicitly loads conn.bro so that it can itself be loaded independently (Robin Sommer). - login.bro depends on scan.bro (because of tracking authentication "scans"), so now it explicitly loads it (Vern Paxson). - UDP_datagram_length_mismatch is now by default flagged just once per originating host rather than once per connection, as it can generate tons of messages (Vern Paxson). - Removed now-long-boring flagging of access to Solaris "listen" service as "hot" (Vern Paxson). - Removal of libedit, since libreadline provides similar functionality (Christian Kreibich). - Added scripts missing from distribution: dce.bro, ncp.bro, and smb.bro (Vern Paxson). - ssh.bro now exports ssh_ports (Seth Hall) - A number of improvements to inter-Bro communication (Robin Sommer). (1) Remote communication now no longer includes location information for serialized objects; that removes quite a bit of redundacy from the network traffic. (2) The new option 'remote_check_sync_consistency" disables the cross-check on the receiving side of &synchronized state of whether the current value of a variable has the value expected by the sender. Transmitting the original values in addition to the updates generates quite a bit CPU & network load in some cases (in particular, a table of tables). The default for remote_check_sync_consistency is off, and so far that in particular seems to reduce the proxy's load quite a bit. (3) Complete overhaul of the internal caching of serialized objects. The objective of the caching is avoid retransmitting already sent values over and over again. It turns out, however, that some objects are very stable and hardly change or get replaced (e.g., Bro types); while other change all the time and are hardly reused some time later (e.g., Vals). Now we maintain *two* caches independently for these types of objects; one with a low turn-over one and another with a high one. This should reduce CPU load on both sender and receiver sides. The new scheme is only used if both communicating Bros support it; with older Bros, as well as with Broccoli, we continue using the old scheme. - Some reworking of remote printing (Robin Sommer), as follows. Bro now uses a new interprocess message rather than print_hook events, to better manage buffering and associated load (these can produce failures depending on system configuration; see remote.log). A number of timeouts and buffer sizes have been tuned. Internally, EINTR errors are now treated separately from EAGAIN. Finally, even with remote_check_sync_consistency=F, one type of consistency check was still being done; this is no longer the case. - The DNS analyzer now generates events (dns_query_reply/dns_rejected) for replies with zero questions (Robin Sommer). - Perftools support for incompatible changes in the 1.0 API (Robin Sommer). - Rearranged (generally reducing, though not always) some state timeouts associated with scan detection (Robin Sommer). In addition, when a scanning address crosses ignore_scanners_threshold (meaning that it will be ignored from now on anyway), it gets discarded from all state-tracking tables. Finally, the ignore_scanners_threshold now applies all kinds of scans, not just address scans. - Substantial Broccoli updates, including a new initialization requirement that breaks backward compatibility, support for enqueueing serialized event data for transmission, and OpenSSL threadsafe initialization. See aux/broccoli/ChangeLog for details (Christian Kreibich, Robin Sommer, and Matthias Vallentin). - Broccoli hashtable optimisation. See aux/broccoli/ChangeLog for details (Christian Kreibich & Matthias Vallentin). - Broccoli memory leak fixed, see aux/broccoli/ChangeLog for details (Christian Kreibich). - Broccoli: updates to bropipe tool (Steve Chan and Robin Sommer). - Bug fixes for Broccoli Python bindings (Robin Sommer and Matthias Vallentin). - Fixed nasty bug due to module scoping that completely kept stepping-stone detection from working (Vern Paxson). - A serious bug in the packet sorter has been fixed (Robin Sommer). - Bug fix for extra NULs getting embedded in escaped strings (Seth Hall). - Bug fix for HTTP messages that use "Connection: close" rather than length headers, which yielded erroneous reassembled messages with \r\n's when only \n's were present (Bernhard Ager). - Fix for reporting on ICMP flows that are expired from the flow table (Vern Paxson). Previously there was a race condition if the flow was flushed prior to its summary timer expiring. - The -l option (list the scripts that Bro loads) now correctly prints scripts loaded by the prefix mechanism, and uses indentation to indicate the load hierarchy (Robin Sommer). - A bug has been fixed (really, worked around) in drop.bro that prevented dropped addresses from being properly restored (Robin Sommer). - Fixes for deadlocking problems in the Broccoli protocol. See aux/broccoli/ChangeLog for details (Christian Kreibich & Robin Sommer). - Bug fix for DNS analyzer on 64-bit machines (Gregor Maier). - Bug fix for asynchronous DNS lookups to prevent some successful lookups being reported as timed out (Robin Sommer). - Bug fix for tracking line numbers associated with compound statements (Po-Ching Lin). - Fix for a rare condition in which the main Bro process couldn't kill its child process (Robin Sommer). - Fix for file rotation when the underlying file is deleted before the timer expires (Robin Sommer). - Fix for potential crash when communication connections break down, and also for releasing cached objects (Robin Sommer). - Fix for default table entries computed by function invocation to not cache previous results (Robin Sommer). - Fix for Bro's internal DNS resolution (Scott Campbell and Robin Sommer). - Portability fix for DAG packet capture (Gregor Maier). - Portability fix for --enable-brov6 (Robin Sommer). - Portability fixes for FreeBSD (Vern Paxson). - A work around for new_packet() crashing on IPv6 packets (Vern Paxson). For now, IPv6 packets are skipped. Also, for fragments the event handler is now only called for the fully reassembled packet. - The new configuration option --disable-nbdns supports disabling non-blocking DNS at configure time (Sean McCreary). Note, there are some known problems with it in some environments. - A number of configuration fixes and enhancements (Christian Kreibich and Robin Sommer). - Consistency nit for the configuration process (Seth Hall). - A number of reference-counting and other memory management fixes (Robin Sommer). - Bug fix for inter-Bro communication lockup (Seth Hall and Robin Sommer). - Bug fix for computing TCP payload length in new_packet event (Lothar Braun). - Bug fix for sending boolean True values via Broccoli (Seth Hall). - make distcheck fix to clean up .bif.bro files (Christian Kreibich). - Bug fix for DPD's recognition of SSLv2 connections (Seth Hall). - Bug fix for &default for tables indexed by subnets (Seth Hall). - A bug has been fixed that could crash Bro when you called get_event_peer() after a remote connection had already disppeared (Robin Sommer). - Introduced a work-around for crashes that occur when Bro exits due to handling a signal (Robin Sommer). - Bug fix for checkpoint.bro - don't schedule timers for times that aren't actually in the future (Robin Sommer). - Hostname formatting fix for anon.bro (Fabian Schneider). - Bug fix for redundant .log extension in Time Machine log file (reported by CS Lee). - Removed now-outdated special-casing of Linux reporting of packet filter statistics (Peter Wurzinger and Robin Sommer). - A number of memory leaks fixed (Robin Sommer). - Addressed warnings from newer versions of g++ (Robin Sommer and Vern Paxson). - Fixed an invocation issue in the ca-create script that prevented it from working with recent OpenSSL versions (Craig Leres & Christian Kreibich). - Comment fixed in drop-adapt (Justin Azoff). - Duplicate code removed from Val (Seth Hall). 1.4 Fri Oct 17 11:08:52 PDT 2008 - We are no longer supporting a previous Bro release as the "stable" version. Rather, the model now is that the current public release will aim for increasing stability (occasionally updated with fixes), and those who wish to use a "bleeding-edge" snapshot can do so via access to the public SVN source code repository, as explained at http://bro-ids.org/wiki/index.php/Subversion#Public_Access Note that all previous releases remain available from the download page; what is changing is that we no longer commit to support for the most recent of these. - We have clarified the copyright statement that covers most of the code to remove the "advertising clause" that derived from older BSD licenses, and we have removed copyright wording from most source code files. See COPYING for the current wording and a list of files that retain their own copyright notices. - Bro now supports analyzing NetFlow v5 data, i.e., from Cisco routers (Bernhard Ager). NetFlow can be useful for intrusion detection as it allows analysis of traffic from many different points in the network. Bro can now read NetFlow data from a UDP socket, as well as (mostly for debugging purposes) from a file in a specialized format. You can create these files with the programs given in aux/nftools. Command line switches: -Y|--netflow :[=] | read flow from socket This is the usual way of getting NetFlow data into Bro by opening a UDP socket on : and reading all incoming packets. Setting the to 0.0.0.0 should work on most platforms. Optionally you may set an identifier for the source - useful if there are many different sources you want to analyze in parallel. This might also be necessary if you want to use this feature with a clustered Bro. Examples: bro -Y 0.0.0.0:5555 netflow bro -i eth0 -Y 10.0.0.1:1234=src1 brolite netflow -y|--flowfile [=] Used to read from a file. You can optionally include an identifier for the source. Examples: bro -y myflowfile netflow bro -y myflowfile=src1 otherflowfile=src2 netflow Netflow Events: event netflow_v5_header(h: nf_v5_header) Generated upon reading a new NetFlow PDU, as summarized in the argument. The field h_id gives the flow source identifier and a serial number. You can use this field to associate subsequent netflow_v5_record events with their header. event netflow_v5_record (r: nf_v5_record) Every record within a NFv5 PDU generates a corresponding netflow_v5_record() event. The relatively complex timestamp format of NFv5 is already converted to Bro's time type, and the TCP header flags are separated into bools. The distribution includes an example analysis script, netflow.bro. It simply dumps received NetFlow records. If netflow_restitch is T (the default), then Bro performs flow restitching as well, and two script variables become relevant: global netflow_finished_conn_expire = 310 sec &redef; specifies how long to wait for additional flow records after a RST or FIN for const netflow_table_expire = 31 min; Its setting only affects table declarations, and therefore cannot be usefully redef'd. Auxiliary programs: Bro uses a custom format for flow data stored in files, to enable preserving timestamps of the PDU arrivals and the exporter's IP address. The tools nfcollector and ftwire2bro in aux/nftools/ provide ways to manipulate the Bro NF file format. The first dumps NetFlow data from a UDP socket to stdout or to a file in Bro format. The second converts NetFlow data in "wire" format to Bro format, and, while doing so, fakes up the exporter's IP address and timestamp. You can get "wire" format from normal flow-tools files, e.g., by using 'flow-export -f 4'. Please note that the Bro format is just a hack to allow for easier debugging. Therefore the format is not in fact platform independent, and not suitable for data storage. - A new DHCP analyzer generates the following events (Po-Ching Lin): event dhcp_discover(c: connection, msg: dhcp_msg, req_addr: addr) event dhcp_offer(c: connection, msg: dhcp_msg, mask: addr, event dhcp_request(c: connection, msg: dhcp_msg, event dhcp_decline(c: connection, msg: dhcp_msg) event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, event dhcp_nak(c: connection, msg: dhcp_msg) event dhcp_release(c: connection, msg: dhcp_msg) event dhcp_inform(c: connection, msg: dhcp_msg) where dhcp_msg values look like: type dhcp_msg: record { op: count; # 1 = BOOTREQUEST, 2 = BOOTREPLY m_type: count; # the type of DHCP message xid: count; # transaction ID of a DHCP session h_addr: string; # hardware address of the client ciaddr: addr; # original IP address of the client yiaddr: addr; # IP address assigned to the client }; See dhcp.bro for the corresponding analysis script (which could probably use some refinements). Note, this analyzer is implemented using BinPAC, so you will need to specify --use-binpac to activate it. - A BitTorrent analyzer is now available (Nadi Sarrar). See the policy scripts bittorrent.bro and bt-tracker.bro for the events generated for analyzing transfers and tracker dialogs, respectively. - The "Bro Lite" configuration is now deprecated and will not in general be supported (Robin Sommer & Vern Paxson). - "make install" now only installs a core set of files (Robin Sommer). Policy files are now installed in /share/bro/* (or whatever configure determines $datadir to be), which is now in Bro's default search path. It creates a directory /share/bro/site for local policy files, and the default BROPATH is extended to include this. The default path no longer includes policy/local. You can install the additional files used by the (now deprecated) "Bro Lite" configuration using "make install-brolite". - Substantial updates to Broccoli, including support for container types (tables and sets) as well as a new metadata structure for event callbacks, facilitating truly generic event handler implementations (Christian Kreibich, Seth Hall and Robin Sommer). See aux/broccoli/ChangeLog for details. - Extensive changes to allow Bro to process packets captured in the past intermingled with those captured in real-time (Matthias Vallentin and Robin Sommer). This operation reflects combining Bro with use of "Time Machine" functionality for packet capture. - We have unfortunately had to disable support for configuring Bro to use ClamAV, since it turns out that the key interface we need for processing blocks of memory directly rather than whole files is no longer supported by the package, and in fact was buggy even when it was (Robin Sommer). - The new signature option "http-body //" matches on the body data of HTTP entities (Robin Sommer). The matching is done after decompressing the body, if necessary. - The new built-in function identify_data(data: string, return_mime: bool) analyzes the string "data" and returns its type according to libmagic, if installed (Seth Hall). The second argument controls whether it should be returned as a MIME-type or just an identifying string. For example, identify_data("MZpofigu", F) returns the string "MS-DOS executable", and print identify_data("MZpofigu", T) returns "application/x-dosexec". - The new analysis script http-identified-files.bro identifies the type of items returned by Web servers using libMagic (if available) and generates notices for interesting types and mismatches between URLs and types (Seth Hall). You configure it using two variables. watched_mime_types is a pattern (default /application\/x-dosexec/ | /application\/x-executable/ ) for which any MIME type matching the pattern generates a HTTP_WatchedMIMEType notice. mime_types_extensions is a table mapping strings to patterns specifying how URLs for the given MIME type should appear. (Ideally, this would be a table mapping patterns to patterns, but Bro doesn't currently support that.) It defaults to: ["application/x-dosexec"] = /\.([eE][xX][eE]|[dD][lL][lL])/ i.e., do Windows executables end in .exe or .dll. You can also redef the pattern ignored_urls to specify URLs that should not generate complaints. It defaults to matching Windows Update. - The new script http-extract-items.bro extracts the items from HTTP traffic into individual files (Vern Paxson). Files are named: .._._. where is a redef'able prefix (default: "http-item"), is a number uniquely identifying the item, the next four are describe the connection tuple, and is "orig" if the item was transferred from the originator to the responder, "resp" otherwise. - The workings of how Bro interfaces to external programs for dropping/ restoring connectivity of misbehaving hosts has been significantly reworked (Brian Tierney and Robin Sommer). First, dropping decisions used to be made directly by analyzer scripts, such as scan.bro directly calling drop_address(). Now instead the scripts generate Notices and then the notice policy can have an action of NOTICE_DROP to codify that the response to the given Notice is to drop the source. The new notice_action_filter of drop_source drops the source of notices, and drop_source_and_terminate both drops the source and terminates the corresponding connection. So, to drop all sources triggering a specific notice, one can now, e.g., write: redef notice_action_filters += { [Hot::SSH_Overflow] = drop_source }; Related to this change, notice_info has a new field $dropped, set to true if the Notice triggered a (successful) drop. Second, by redef'ing Drop::use_catch_release to T (default F) you can activate "catch-and-release" logic. You use this mode when you need to manage a limited number of possible blocks, or to build in automatic "forgiveness" in situations where blocked sources might become benign (such as due to dynamic IP addresses). If a source has been idle for Drop::drop_time, then it is unblocked. However, if it is again seen as block-worthy, then it is blocked for an interval of Drop::long_drop_time. Third, ICMP scanning is now reported by its own notice, ICMPAddressScan, rather than Scan::AddressScan. - Google's perftools have replaced mpatrol for leak-checking and heap-profiling (Robin Sommer). If Bro is compiled with --enable-perftools and configure finds the perftools, there are two command-line options available: -m turns on leak checking of the main packet loop, with some uninteresting leaks are suppressed. Currently, with one exception (the RPC analyzer; problem not yet found), it reports no leaks when running the test suite. -M turns on heap profiling: Bro will take a snapshot of the heap before starting the main packet loop and another one when finished. These snapshots can then be analyzed with pprof. For more information about the perftools see http://code.google.com/p/google-perftools - Notice tags are now generated in a pseudo-unique fashion that, with high probability, ensures that tags generated by separate Bro processes don't clash when logged to a common location, such as for a Bro cluster (Robin Sommer). Tags are now string's rather than count's, and are associated with all notices, not just that are connection-related. You can however redef the string notice_tag_prefix or the function new_notice_tag to further control how such tags are generated. - Four new built-ins for type conversion (Robin Sommer): function double_to_interval(d: double): interval function addr_to_count(a: addr): count function port_to_count(p: port): count function count_to_port(c: count, t: transport_proto): port - Many policy scripts have been modified to use modules & scoping (Robin Sommer and Matthias Vallentin), which may require updates to existing scripts/refinements. - The new script variable dpd_conn_logs (default F), if true, changes the semantics of the service field in connection logs written to conn.log, as follows (Robin Sommer). It becomes a comma-separated list of analyzers confirmed by DPD to parse the connection's payload. If no analyzer could confirm its protocol, but the connection uses a well-known port, the service is the name of the port with "?" appended (e.g., "http?"), as long as the corresponding analyzer has not declined the connection. In addition, ftp-data sessions are labeled "ftp-data" and portmapper connections are labeled with the specific method-call (just as before). dpd_conn_logs defaults to F because the change in semantics may break scripts that parse conn.logs; but it will likely change to the default in the future. With dpd_conn_logs turned off, conn logs are generated as they used to be, with a few rare exceptions (with previous versions, the service field was sometimes determined while the connection was still alive; now it's always determined at the time when the conn.log entry is written out). - The SSL analyzer has been rewritten using BinPAC, with a number of robustness improvements (Tobias Kiesling). It currently is only used if you execute with --use-binpac. - Python bindings for Broccoli are now available in aux/broccoli/bindings/python/ (Robin Sommer). See README/README.html in that director for details. - The new "auth" option in remote.bro indicates whether a given side is considered "authoritative" for shared state, in which case it sends its initial state to &sync'ed peers (Robin Sommer). When two peers synchronize their state, one side sends its current set of state to the other as soon as the remote connection is established. The one sending the state used to be the one who has been running longer; now it can also be explicitly set via the "auth" flag in the Remote::Destination. - Two new tuning parameters for scan.bro (Robin Sommer): ignore_scanners_threshold (default 0): If a host has scanned more than this many hosts, it is completely excluded from further scan detection. 0 disables. addr_scan_trigger (default 0): A host is only tracked for address scanning once it has contacted this many different hosts. Primarily intended for using a two-stage scan detection with a Bro cluster: first, each node searches locally for scanners by looking for hosts contacting more than addr_scan_trigger destinations. Those hosts which do are then globally tracked throughout the cluster by &synchronizing the scan detector tables. - When Bro serializes functions, it now does so by default using only their name, rather than their full value (Robin Sommer). This prevents propagation of expiration functions associated with tables and sets. Note, currently there is no mechanism provided to switch from the default behavior, but the internal hooks are in place to do so. - The new built-in variable trace_output_file gives the name of the -w output trace file (Robin Sommer). - Bro no longer installs new file rotation timers when shutting down (Robin Sommer). - The new policy scripts remote-print-id{,-reply}.bro support convenient access to printing the identifiers of a remote Bro (Robin Sommer). You use the script remote-print-id.bro to request and receive the printing; the remote Bro must have loaded remote-print-id-reply.bro in order to process the request. Example use: bro -e 'redef PrintID::dst="" PrintID::id=""' remote-print-id - scan.bro has been heavily modified to better support distributed scan analysis (Matthias Vallentin and Robin Sommer). - The check for unused event handlers is now turned off by default (Robin Sommer). To enable, use "redef check_for_unused_event_handlers = T". - The new script drop.bro has been split off from scan.bro to isolate the logic concerning dropping addresses to block scans (Robin Sommer). - The new -l flag lists each script as it is loaded (Robin Sommer). - Textual descriptions of identifiers now include their attributes (Robin Sommer). - The new predefined function prefixed_id() returns a session identifier with its peer-ID prepended if it's associated with a remote Bro (Robin Sommer). This is now used when generating writing log files. - remote.bro now assigns a priority of -10 to its bro_init() event handler to allow others a chance to modify destinations (Robin Sommer). - A large number of BinPAC updates (Ruoming Pang and Robin Sommer). - The new built-in type_name(v): string returns the name of the type of the value v (Vern Paxson). For example, "typename(5.2)" returns "double". This function is mainly for internal debugging (i.e., finding mismatches between values generated by the event engine versus how their type is expected by the script layer). - The new built-in str_shell_escape() does some basic escaping on strings that will be passed to system() (Christian Kreibich). Note, this function isn't ready (robust enough) for routine use, however. - The new built-in disable_print_hook(file) acts the same as the attribute &disable_print_hook (Robin Sommer). - The new script terminate-connection.bro factors out the terminate_connection() functionality that used to be in conn.bro (Robin Sommer). - The new attribute &group= can be associated with event handlers to group them together into a set that can be manipulated as a whole (Robin Sommer). is a string reflecting the name given to the group. The built-in enable_event_group(group: string) turns on all the analyzers in a given group, and disable_event_group(group: string) deactivates them. - The new attribute &raw_output applies to variables of type file, disabling escaping of non-printable characters (Seth Hall). - You can now iterate over the characters in a string value using a "for" loop, e.g., "for ( c in str ) ..." (Robin Sommer). - The new built-in function cat_sep%(sep: string, def: string, ...%): string works similarly to cat(), except that it (a) separates the values by "sep" and (b) substitutes "def" for empty strings (Seth Hall). - The function string_escape() now takes a string of characters to escape rather than a single character (Robin Sommer). Each character in the string is preceded by '\' in the return value (also any embedded '\'s, as before). - The new built-in function global_ids() returns a table of all global identifiers along with associated information (Robin Sommer). The return value has type table[string] of script_id, indexed by the name of the identifier and yielding records with the following fields: type script_id: record { type_name: string; exported: bool; constant: bool; enum_constant: bool; redefinable: bool; value: any &optional; }; - The new script function find_last(str: string, re: pattern) returns the last occurrence of the given pattern in the given string, or an empty string if no match (Robin Sommer). Note that this function returns the match that starts at the largest index in the string, which is not necessarily the longest match. For example, a pattern of /.*/ will return just the final character in the string. - The new script variable record_all_packets, if redef'd to T (default F), instructs Bro to record every packet it processes (Robin Sommer). Prior to introducing this variable, Bro applied a few heuristics to reduce recording volume. Setting this variable also causes packets to be recorded very early in processing, which can be helpful for debugging crashes. - If the new script flag ssl_log_ciphers is set to T (default), ssl.bro logs the ciphers seen (Robin Sommer). - Much more expanded Time Machine support, now located in policy/time-machine/ (Robin Sommer), - The new command line option --status-file (alias -U) specifies the name of a file into which Bro will write an indicator of its current processing status (Robin Sommer). Possible values include "INITIALIZING", "RUNNING", "TERMINATING", "TERMINATED". - The new policy script targeted-scan.bro looks for repeated access from the same source to the same server, to detect things like SSH password-guessing attacks (Jim Mellander). - The "alternative" style for printing strings (i.e., a fmt() argument of "%As") now renders the raw string, other than escape-expanding embedded NULs (Vern Paxson). This change may be temporary, pending development of more fine-grained control over string rendering. - For now we have removed the %S functionality for fmt() (Robin Sommer). %S was meant to print "raw" strings, but later processing of such printing still introduces artifacts. - GeoIP information now includes latitude and longitude (Seth Hall). - ssh.bro now supports the variable skip_processing_after_handshake which directs the event engine to omit any further processing of an SSH connection after its initial handshake (Seth Hall and Robin Sommer). This can help with performance for large file transfers but precludes some kinds of analyses (e.g., tracking connection size). This change also adds a scope of "SSH". - Email notification of notices now allows for separate destinations depending on notice type (in particular, a regular mail destination versus a pager destination), and also escapes the notice to prevent injection attacks (Seth Hall and Robin Sommer). - The new policy script conn-flood.bro is a simple connection-flooding detector, mainly meant as a demonstration (Robin Sommer). - A large number of additions to the TLS/SSL known-ciphers suite (Seth Hall). - Serialization now uses 64-bit IDs to cache items rather than 32-bit, for robustness during long-running execution (Robin Sommer). - The new script variable tcp_max_initial_window specifies, for flows for which ACKs have never been seen, the maximum volume of initial data after which Bro will assume that it is seeing only one side of the connection and will not buffer data for consistency checking awaiting the later arrival of ACKs (Robin Sommer). It defaults to 4 KB. (Note, this used to be an internal value, so the behavior is not new.) Set to 0 to turn off this functionality and have Bro attempt to track all such flows. - The new script variable tcp_max_above_hole_without_any_acks specifies, for flows for which ACKs have never been seen, the maximum volume of data above a sequence hole that Bro will tolerate for a connection before giving up on tracking the flow (Robin Sommer). It defaults to 4 KB. (Note, this differs from tcp_max_initial_window in that this threshold applies to sequence holes rather than the beginning of flows. Like tcp_max_initial_window this used to be an internal value.) Set to 0 to turn off this functionality. - The new script variable tcp_excessive_data_without_further_acks specifies a threshold similar to tcp_max_above_hole_without_any_acks, but for flows for which Bro has seen ACKs (Robin Sommer). It defaults to 10 MB. Set to 0 to turn off the functionality. - Equal signs ("=") in text for notices are now escaped when using the tagged format to keep them unambiguous from the "=" delimiters (Robin Sommer). - The final tallies for notices are now processed as NoticeTally NOTICE's rather than directly alarm'd (Robin Sommer). - WeirdActivity notices now include an associated connection when appropriate (Robin Sommer). - Support for large (> 2^32 bytes) pcap trace files (Po-Ching Lin). - Scoped names ("...::...") are now allowed in signature "eval" constructs (Christian Kreibich). - scan.bro is now decoupled from conn.bro, i.e., you can @load the latter without getting the former (Vern Paxson). As part of this change, the logic to invoke TRW is now in scan.bro. - weird.bro has been updated with a number of missing Weird's (Vern Paxson). - If when using inter-Bro communication the child Bro process terminates, it now also terminates the parent process (Robin Sommer). - BinPAC analyzers now interoperate with DPD (Robin Sommer). - Some http.bro processing options are now exported so they can be accessed in other scripts (Robin Sommer). - SMTP analysis now applies to port 587/tcp as well as 25/tcp (Robin Sommer). - $conn is now set in ServerFound notices (Robin Sommer). - You can now create empty sets and tables using set() and table(), i.e., the usual set/table constructors with no arguments (Vern Paxson). By themselves, these have an unspecified type - you can't use them directly other than to assign them. For example, local bad_guys: set[addr]; ... bad_guys = set(); # start over assuming no bad guys - A number of scripts have been (slightly) simplified to use the new empty set()/table() constructors (Vern Paxson). Note that these still aren't usable for field assignments in record constructors, nor for attributes like &default = ... - Removed unused syntax for declaring sets based on a list of initial values (Vern Paxson). - set() and table() can now be used as arguments to function calls (Vern Paxson). - The vestigial &match attribute has been removed. - POP3 is now recognized using Dynamic Protocol Detection (Seth Hall). - The new event expected_connection_seen(c: connection, a: AnalyzerTag) is generated whenever a connection is seen for which we have previously scheduled an analyzer via expect_connection() (Robin Sommer). - The new built-in capture_state_updates logs all changes applied to &synchronized variables, in a fashion similar to the capture_events() built-in (Robin Sommer). An accompanying policy script, capture-state-updates.bro, turns this on to the file state-updates.bst. - If the new script variable suppress_local_output is set (default: F), Bro suppresses printing to local files if there's a receiver for print_hook events (Robin Sommer). This option is however ignored for files with a &disable_print_hook attribute. - The new notice action filter function file_if_remote specifies that notices from sent from remote source addresses should have an action NOTICE_FILE (Robin Sommer). - The new notice action filter function file_local_bro_notices specifies that notices generated by the local Bro instance (as opposed to a remote peer) should have an action NOTICE_FILE (Robin Sommer). - An arbitrary tag can now be past to post-processors for log rotation (Robin Sommer). - Default inactivity timeouts for interactive services shortened to 1 hour (Robin Sommer). - The scanning variables distinct_{peers,ports,low_ports} are now redef'able (Robin Sommer). - The new -S (--summary-only) option for site-report.pl directs to only generate connection summaries (Brian Tierney) - More useful default config file for edit-brorule.pl (Brian Tierney). - Bro now includes a test suite in testing/istate/ for its "independent state" functionality (Robin Sommer). - Support for parallel builds via make -j (Christian Kreibich). - Bro's default search path now includes includes policy/sigs/ and policy/time-machine/ (Robin Sommer). - Bro's internal processing of interprocess communication has been significantly overhauled to prevent potentially fatal race conditions (Robin Sommer). - Bro now checks calls to fmt() at compile-time to ensure that the correct number of arguments are present (Vern Paxson). This is useful in addition to Bro's run-time checking for arguments matching their corresponding format-specifiers in the case of rarely-executed statements that might not generate such run-time checks in routine testing. - The ports associated with Telnet and Rlogin are now redef'able (Robin Sommer). - MIME processing now removes leading whitespace from MIME headers (Sanmeet Bhatia and Robin Sommer). - TCP "weird" events reported by the connection compressor now match (other than a few rare corner-cases) those produced for normal TCP processing (rmkml and Robin Sommer). - Added Scan::suppress_UDP_scan_checks to control false positives on scan detection in environments with P2P protocols that use UDP (Vern Paxson). - The internal analyzer interface now includes an EndOfData() method that analyzers can use to report that all of a message has been delivered (Robin Sommer). - Fix for a significant memory leak in processing UDP when using -w (Robin Sommer). Note: this change turns off by default trace rewriting for generic UDP traffic. - Two serious regular expression bugs fixed (Vern Paxson). In the first, searching for a regular expression inside a string would fail if the pattern occurred only after an embedded newline. In the second, insufficient buffer was allocated when compiling regular expressions, leading to memory corruption. - Base64 decoding bug fixes (Christian Kreibich and Ruoming Pang). - Automatic rotation of files is now disabled for contents files written by the TCP reassembler, which otherwise leads to mangled files (Robin Sommer). - Bro now ships with an updated version of libpcap (0.9.8), which hopefully fixes problems managing trace files > 4 GB in size. - Significant bug fixes for gzip- and deflate-encoded Web items (Robin Sommer). - Bug fix for secondary-filter.bro (Vern Paxson). - Removed a naming ambiguity regarding TCP states (Vern Paxson). - Bug fix for signature scanner not matching all of its input (Vern Paxson). - Bug fix for using port values in signatures (Robin Sommer). - Minor policy script tweaks: state management for weird's, processing of Notice tags associated with connections, and dependencies for irc-bot.bro (Robin Sommer). - aux/ portability fixes (Vern Paxson). - Workarounds added for a BinPAC deficiency, which is that code in %cleanup clauses can also be executed during recovery from exceptions when parsing new data. This means that any delete's or Unref()'s need to also set the corresponding pointer to nil (Vern Paxson). - Bug fix for crashes with the non-BinPAC SSL analyzer (Robin Sommer). - Tweak to peer-status.bro since Bro now requires events to be declared prior to reference in a "schedule" statement (Robin Sommer). - The signature keyword "enable" now optionally accepts the syntax "foo:bar" to specify "activate analyzer bar as a child of analyzer foo" (Robin Sommer). This is used for example for an XML-over-HTTP analyzer that's in the works. - irc-bot-syslog.bro now uses open_log_file() for its log file (including the logging suffix) rather than a direct open (Vern Paxson). - Bug fix for tracking Blaster across a Bro Cluster (Robin Sommer). - Bug fix for the HTTP BinPAC analyzer chopping the trailing character off of HTTP headers when generating the http_all_headers event (Gregor Maier). - Bug fix for HTTP chunked items for which the chunk size line was terminated by CRLF but the CR and LF came in separate packets (Gregor Maier). - A bug has been fixed that would cause partial lines (for line-oriented protocols) to fail to be processed when a connection terminated (Robin Sommer). - Bro no longer treats a signal arriving before a previous signal has been processed as fatal, nor does it attempt processing of a termination signal if seemingly there are no race conditions to worry about (Robin Sommer). Both of these changes are an attempt to improve Bro's robustness. - Fix for attributes such as &encrypt not working in initial declarations but only in later redef's (Seth Hall and Robin Sommer). - Fixes for memory leaks in SSL processing (Seth Hall and Robin Sommer). - Fix for POP3 analyzer to not treat lines like "." as message terminators (Robin Sommer). - Bug fix for crashes arising from nil pointers in list expressions (Seth Hall and Robin Sommer). - Bug fix: a signature's "enable" would activate the corresponding analyzer even if no event handlers were defined for it (Robin Sommer). - Bug fixes to prevent crashes when mixing set_contents_file() with subsequent explicit close(), and to ensure all data written to file upon connection tear-down (Gert Doering and Robin Sommer). - Configuration support for MacPorts and Fink package management systems (Christian Kreibich & Vern Paxson). - Communication-only Bro's now send out email alarms (Robin Sommer). - Writes to a file that fail due are now run-time errors rather than fatal internal errors, since often these occur due to the disk being full (Robin Sommer). - Byte-order bug fix for lookup_location() (Robin Sommer). - BinPAC portability fix for 64-bit machines (Bernhard Ager and Robin Sommer). - Portability fixes for newer versions of gcc (Jan Gerrit Goebel and Robin Sommer). - Some support for porting to Solaris (Stephan Toggweiler). - Connection compressor bug fix for source and destination having the same IP address, such as when monitoring loopback (Robin Sommer). - Connection compressor bug fix for connections with multiple SYNs (Robin Sommer). - Bug fix for using already-declared local variables for looping over vectors in a "for" loop (Robin Sommer & Vern Paxson). - Bug fix for not processing truncated UDP packets (Tom Kho and Robin Sommer). - Bounds-check added to BinPAC-generated code (Tom Kho and Robin Sommer). - Bug fix for checking whether an IPv6 address is part of a subnet (Seth Hall). - Bug fixes for crashes relating to asynchronous DNS lookups performed at start-up (Robin Sommer). These changes also lowered the timeout before assuming failure from 20 seconds down to 5 seconds. - Portability and const-ness fixes (Kevin Lo and Robin Sommer). - Suppression of some content-gap complaints when running on traces that have been filtered down to only TCP control packets (Robin Sommer). - Removed unnecessary dependency in notice-action-filters.bro that led to errors when loading icmp.bro by itself (Vern Paxson). - Bug fix for potential infinite loop in client communiation (Robin Sommer). - Bug fix in reference counting that could eventually lead to roll-over (Robin Sommer). - Bug fix in communication initialization (Robin Sommer). - Internal documentation fix: timers are specified using absolute time, not relative (Robin Sommer). - Performance improvement for built-in find_all() function when running on large strings (Robin Sommer). - Memory leak fixes (Robin Sommer, Bernhard Ager, Christian Kreibich). - Bug fix for error recovery when encountering an unknown link layer (Bernhard Ager). - Bug fix for reversing client & server in a connection (Po-Ching Lin). - Bug fix for packet_contents when capture length exceeds the IP payload length due to Ethernet frame padding (Christian Kreibich). - Bug fix for tcp_packet event erroneously including Ethernet padding in its contents (Vern Paxson). - Bug fix for lookup_connection built-in (Seth Hall). - Portability nit for libedit tarball (Vern Paxson). - Broccoli portability fix for NetBSD (Christoph Leuzinger). - Type-checking for script-level event invocation was completedly broken - now fixed (Vern Paxson). - Portability fixes for different versions of g++/STL (Nicholas Weaver and Vern Paxson). - Fix for dynamic detection of SSL via DPD (Robin Sommer). - IPv6 portability fix for BinPAC-based DNS analyzer (Vern Paxson). Note, more portability work is needed for it. - Bug fix for bifcl error messages (Vern Paxson). - Minor bug fix for remote communication, plus some improved communication logging (Robin Sommer). - Bug fix for &printhook (Robin Sommer). - Bug fix for error message output (Robin Sommer). - Bug fix for termination cleanup (Robin Sommer). - Bug fix for some Rlogin corner cases (Robin Sommer & Vern Paxson). - Bug fix for bifcl generation of "interval" types (Vern Paxson). - Bug fix for getting connection memory statistics when Bro is exiting (Robin Sommer). - Config fix: --enable-debug now turns off -O2 for gcc (Robin Sommer). - Bug fixes for "heavy" analysis (Vern Paxson). - Broccoli bug fixes for types net and port (Robin Sommer). - Bug fixes for Telnet environment options (Robin Sommer). - Bug fix for accessing remote peer description (Robin Sommer). - A fix for the connection compressor generating new_connection too late (Robin Sommer). - Fixes for DAG support, including configuration and multiple interfaces (Robin Sommer). - Bug fix for serializing time-stamps of table entries (Robin Sommer). - Bug fix for dealing with peer IDs for remote communication (Robin Sommer). - Bug fix to avoid installing timers when timers have already been canceled (Robin Sommer). - Bug fix for interplay between serializing connections and connection compressor (Robin Sommer). - Memory leak fix for enum's (Robin Sommer). - Bug fix for files being closed prior to bro_done() (Vern Paxson). - aux/broccoli/contrib was not included in distribution (Robin Sommer). - Auto-configuration bug fix for BinPAC (Craig Leres). - Bug fix for dynamic protocol detection (Robin Sommer). - A number of configuration fixes for installation and portability (Christian Kreibich, Brian Tierney, Robin Sommer, Dan Kopecek). 1.3 Mon Jul 16 22:11:00 PDT 2007 - The Bro manual has been wikified at: http://www.bro-ids.org/wiki/index.php/User_Manual and this is the format in which it will evolve in the future (Christian Kreibich). - Much more extensive support for SMB, NetBIOS and NCP (Chris Grier). - The new attribute &priority=n defines the order of execution for handlers of the same event (Robin Sommer). Handlers with higher priority are executed first. n is an integer expression that must evaluate to a constant when the script is loaded. Example: > cat foo.bro event bro_init() &priority = -5 { print -5; } event bro_init() &priority = 5 { print 5; } event bro_init() { print 0; } # default priority=0 > ./bro foo.bro 5 0 -5 The connection_state_remove() handler in conn.bro now has priority -10 and therefore executes after all other handlers for this event. This fixes a long-standing problem of sometimes $addl fields not showing up in connection summaries. - The new expressions record(...), table(...), set(...) and vector(...) are constructors for the corresponding aggregate types (Vern Paxson). For example, record($foo = "hi", $bar = -6) is the same as the existing constructor [$foo = "hi", $bar = -6] For tables, sets, and vectors, the "..." values within the ()'s have the same syntax as those that you can list in variable initializations. For example, table([1, T] = "black", [4, F] = "red") returns a table of type "table[count, bool] of string". set(4, 3, -1) is a value of type "set[int]". - You can associate attributes with table() and set() constructors (Robin Sommer). For example: local s = set(1.2.3.4) &read_expire = 5 secs; associates a 5-second read expiration with the set assigned to s. - Bro now explicitly supports port numbers reflecting a transport protocol type of "unknown" (Christian Kreibich). Currently, this means "not TCP, UDP or ICMP". The numerical value of such a port is the IP protocol, so ranges from 0..255. For example: global p: port = 0/unknown; print fmt("%s", p); print fmt("p is TCP? %s", get_port_transport_proto(p) == tcp); print fmt("p is unknown? %s", get_port_transport_proto(p) == unknown_transport); yields 0/unknown p is TCP? F p is unknown? T In comparisons of different protocol types, the following holds: unknown < TCP < UDP < ICMP. - If your system supports "GeoIP" (see http://www.maxmind.com/app/geolitecity for a corresponding city database), then the new script function lookup_location(a: addr): geo_location returns a record of geographic information associated with an address (Seth Hall). The geo_location record has $country_code, $region and $city fields. If no information is available, each of these will be set to empty strings. If Bro hasn't been configured with GeoIP support, or if the address is IPv6 that cannot be directly converted to IPv4, then Bro produces a run-time error and likewise returns empty strings. - Signature-matching on HTTP components now processes the URI with escape sequences expanded (Robin Sommer). Ideally, there would be two signature keywords, one for decoded URIs (corresponding to this case) and one that allows matching against the URI as originally transmitted. - The connection compressor is no longer considered experimental, and is used by default (Robin Sommer). - The new function lookup_hostname(host: string): addr_set asychronously looks up the IPv4 address(es) of the given host via DNS (Robin Sommer). Like lookup_addr(), this function can only be used within a "when" statement. - The new built-in raw_bytes_to_v4_addr(s: string): addr takes a string that points to at least 4 bytes, and returns an address corresponding to interpreting these as being an IPv4 address in network order (Vern Paxson; suggested by Mike Dopheide). - Trace-rewriting support for DNS, SMB (Chris Grier). - The new script function find_all(str: string, re: pattern): string_set returns a string_set giving all occurrences of the pattern "re" in the string "str" (Robin Sommer). (Note that string_set's are unordered.) - The new policy script save-peer-status.bro generates a log to peer_status.$BRO_LOG_SUFFIX of updates received from communication peers (Robin Sommer). - The policy script print-filter.bro now includes two (scoped) variables, terminate_bro and to_file, which control whether to exit after printing the filter (default T) and whether to write to the log file pcap_filter.$BRO_LOG_SUFFIX or (default) to stdout (Robin Sommer). - The new script variable check_for_unused_event_handlers controls whether Bro checks for unused event handlers (Robin Sommer). It defaults to T, which was the past behavior (always report). - Bro now terminates if the only pending activity is future timers (Robin Sommer). It used to wait for those timers to expire, but this can cause fundamental problems if the timers are associated with table management (since these might never completely drain). - Tables and sets inside of records are now initialized to empty values rather than uninitialized (Vern Paxson). - A new variable allow_services_from (in hot.bro) complements the existing allow_service_to variable (Brian Tierney). It specifies that access to the given service from the given originator is allowed. - global_sizes() no longer reports internal variables (Robin Sommer). - The IRC analyzer is now activated if any of the (many) IRC event handlers are defined (Robin Sommer). - The default value for tcp_close_delay is now 5 sec rather than 0 sec (Robin Sommer). This prevents some spurious connection events. - Improved logic for dealing with "reversed" connections such as backscatter (Vern Paxson). - You can now left-justify fields when using fmt() with "%-" like in sprintf (Christian Kreibich). - Updates to DNS query types (Larry Leviton). - Added mechanism to http-header.bro to skip printing some HTTP headers (Larry Leviton). - The IrcHotWord notice now sets the associated connection (Robin Sommer). - If a notice has a tag, it's no longer overridden (Robin Sommer). - ServerFound notices now set the port field (Robin Sommer). - The built-in lookup_ID() now returns the string "" if the ID does not exist, rather than a run-time error (Robin Sommer). - The new tuning option ProtocolDetector::suppress_servers specifies a set of analyzers for which Bro generates ServerFound notices, but not ProtocolFound (Robin Sommer). This both reduces log file size and conserves memory. - A new notice_action_filter, tally_notice_type_and_ignore, works the same as tally_notice_type but returns IGNORE (Robin Sommer) - Setting summary_interval == 0 disables the creation of irc-bots.summary.log (Robin Sommer). - If you @load foo and a directory "foo" is in your path, Bro no longer tries to load it (Robin Sommer). - A number of BinPAC fixes and enhancements (Ruoming Pang, Chris Grier and Vern Paxson). - BinPAC now resides in aux/binpac rather than src/binpac (Ruoming Pang and Christian Kreibich). This reflects a decoupling of it from Bro so that it can be used to generate protocol analyzers for other projects too. - Removed example Inktomi entries from skip_scan_sources initialization, since they no longer exist (Vern Paxson). - The variable make notice_once_per_orig_tally_interval is now redef'able (Brian Tierney). - SIGPROF to the communication child process now logs resource stats to remote.log (Matthias Vallentin). - The new built-in getpid(): count returns Bro's process ID (Robin Sommer). - Patterns for detecting IRC-based bots updated (Robin Sommer). - irc-bot-syslog now logs just bots, not all IRC client/servers (Robin Sommer). - The new variable suppress_notice_actions in notice.bro suppresses notice_actions events for selected notice types (Robin Sommer). - Files opened during operation now rotate just like those opened at startup (Robin Sommer). - ResourceStats now also logs elapsed time and the reported number of packets-on-the-link (Mark Dedlow). - Printing a "file" value now produces its name (Robin Sommer). - Removed deliberate truncation of payload in port 80 FIN packets (Vern Paxson). - remote.log now includes received peer_descriptions (Robin Sommer). - Significant POP3 analyzer speed-ups (Vern Paxson). - Updated README (Vern Paxson). - Fix for "@load a" followed by "@load a.bro" not loading the same file twice (Robin Sommer). - Bug fixes for propagating state operations to uninitialized variables and for spurious state inconsistency messags (Robin Sommer). - Bug fix for sending final sync-points during pseudo-realtime mode (Robin Sommer). - Fix for possible buffer overflow (Christian Kreibich). - Bug fix for spurious end-of-file's during inter-Bro communication (Robin Sommer). - Bug fix for dpd_match_only_beginning=F (Robin Sommer). - Bug fix for updating timestamps (Christian Kreibich). - Bug fix for skipping ADU processing in adu.bro (Christian Kreibich and Zhichun Li). - Fix for ICMPs that carry ICMP headers (or non-TCP/UDP/ICMP headers) within them (Vern Paxson). - Fix for files being rotated after the timer queue has been deleted (Vern Paxson). - Bug fix for signature-matching with IPv6 subnets (Vern Paxson). - Bug fix for connection compressor setting connection origin (Robin Sommer). - Bug fix for interconn.bro when processing peculiar connections (Vern Paxson). - Fix for off-by-one buffer in sscanf call (Christian Kreibich). - Fixed inefficiency/warning flagged by g++ (Vern Paxson). - Bug fix for NUL string termination in SMB processing (Zhichun Li). - Fix for over-ref'ing of file Val's (Vern Paxson). - Fixes for some g++ warnings (Christian Kreibich, Vern Paxson). - gcc 3.4.2 portability fixes (Robin Sommer). - Minor build fixes for Broccoli, including a version bump to match that of Bro. See aux/broccoli/ChangeLog for details. - distcheck fixes (Christian Kreibich). - Configuration portability fixes (Matthias Vallentin, Jean-philippe Luiggi). - OpenBSD portability fixes (Jean-philippe Luiggi, Christian Kreibich). 1.2.1 Mon Dec 11 16:22:58 PST 2006 - Fixed delayed triggering of new_connection events when using the connection compressor. - Fixed tracking of first packet in TCP analyzer. (Reported by Guohan Lu) - The syslog built-in got lost during some previous merge. - Fixed crash if local variable is given as timeout value for table. (Reported by Mike Wood.) - Fixed using "time" values as table indices. - Added ssh to default brolite DPD configuration. - Fixed catching up to real-time in case of lull. - Fixed Broccoli "BRO_DATA_FORMAT_VERSION" to match version in Bro. - Fixed Makefile problem in doc directory. - Fixed Makefile dependency problem in binpac directory. - Added Linux tuning to brolite install script. - Modified Makefile to include broccoli/contrib. - Adding missing initialization to remote serializer. - Minor documentation updates for reference manual and Broccoli. 1.2 Tue Oct 17 12:09:49 PDT 2006 - Bro now supports DPD, dynamic protocol detection (Robin Sommer, Holger Dreger, and Michael Mai). With DPD, Bro can analyze protocols regardless of what port numbers they use: it infers the protocol based on which application analyzers can parse it without error. Adding this functionality involved extensive changes to Bro's internals, but also now enables multiple Bro analyzers to work on the same connection, either concurrently or one nested inside the other (we have not taken much advantage of this latter capability yet, but see the FTP events discussed below). There are a number of new policy scripts, events, and variables associated with DPD processing, as follows. Scripts: You activate DPD by @load'ing dpd.bro. It in turn instructs Bro to load the signature file policy/sigs/dpd.sig. Note that Bro uses signatures to expedite deciding which analyzers to try on a given connection; it does *not* simply use the signatures to make the determination of which protocol is in use, as this is insufficiently robust. (At this point, Bro provides signatures for FTP, IRC, HTTP, SMTP, and SSH. In the future we plan to add other protocols.) Along with dpd.bro, you need to @load detect-protocols.bro or detect-protocols-http.bro. The former enables general detection of application-layer protocols, while the latter does further inspection of HTTP sessions to characterize applications running on top of HTTP such as Gnutella or SOAP. (Loading dpd.bro is separate from loading one of these scripts because in principle Bro could use a different means than signatures to activate the analyzers, although currently it does not.) If you @load dyn-disable.bro, then once an analyzer determines that it does not match a given connection, it is deactivated (and a Notice is generated). Otherwise, it still proceeds to try its best to analyze the connection (to possibly be more robust against evasion). The scripts dce.bro and smb.bro enable DPD for the Windows DCE and SMB protocols, respectively. (Note that analysis of these protocols is undergoing a major expansion, not yet complete.) Events: event protocol_confirmation(c: connection, atype: count, aid: count) Generated when the given connection has been confirmed as conforming with the application type (protocol) specified by atype. aid is a globally unique analyzer ID that identifies a particular analyzer instance. The values for atype are symbolic names associated with each of Bro's analyzers, such as ANALYZER_IRC. See the initialization at the beginning of Analyzer.cc for the full set of names. The function analyzer_name(atype: count): string translates these symbolic names into text. For example, analyzer_name(ANALYZER_IRC) yields "IRC". event protocol_violation(c: connection, atype: count, aid: count, reason: string) Generated when the given connection has been found to violate the protocol of the given application type, with "reason" giving details. Variables: dpd_buffer_size: count (default 1024) Specifies how much pending data Bro keeps for connections that have not been classified yet. Once this fills, the data is deleted, though classification can still continue (see below). dpd_match_only_beginning: bool (default T) If set, specifies that Bro should stop signature matching if it has processed dpd_buffer_size bytes. dpd_ignore_ports: bool (default F) If set, then Bro does not take into consideration the port numbers associated with connections when attempting to classify them (which can otherwise help the process in some cases). dpd_reassemble_first_packets: bool (default T) If set, then Bro does TCP stream reassembly before applying signature-matching to detect protocols. likely_server_ports: set[port] Specifies a list of ports that Bro will consider as likely used by servers. For example, if Bro sees a connection that has already been established (so it does not know which side sent the initial SYN), and one side uses a port in this set, then it will assume that that side is the server (connection responder). The set is empty unless you populate it or @load server-ports.bro, which specifies a large number of values. dpd_config: table[AnalyzerTag] of dpd_protocol_config Specifies the DPD configuration associated with each tag. The type dpd_protocol_config is simply: type dpd_protocol_config: record { ports: set[port] &optional; }; i.e., an optional $ports field specifying a set of ports associatd with the tag. For example, ftp.bro now includes the equivalent of: redef dpd_config += { [ANALYZER_FTP] = [$ports = 21/tcp] }; Functions: The function expect_connection(orig: addr, resp: addr, resp_p: port, analyzer: count, tout: interval) is called to alert Bro that a new connection is expected, initiated by orig to a server running on resp's port resp_p (note: orig's port is not specified) which will correspond to the specified analyzer (e.g., "FILE", which is used to analyze files transferred by FTP - see next item). "tout" is a timeout to associate with the waiting. The function function disable_analyzer(cid: conn_id, aid: count) instructs Bro to disable the analyzer that generated the current event, assuming the analyzer is associated with the given connection ID. This is used by the dyn-disable.bro script discussed above. - A much more complete BinPAC compiler, along with new HTTP, DNS, and RPC/Portmap analyzers in binpac (Ruoming Pang). The flag "--use-binpac" activates the BinPAC-based analyzers (currently for HTTP and DNS). See www.cs.princeton.edu/~rpang/binpac-paper.pdf for a description of BinPAC, and let Ruoming know if you are interested in using BinPAC to build new analyzers. - A new type of analyzer, FILE, analyzes the contents of a connection as though it were a data file (Robin Sommer). Currently, it can generate two events: event file_transferred(c: connection, prefix: string, descr: string, mime_type: string) Indicates that the connection transferred a file. "prefix" is the beginning of the file's data; "descr" and "mime_type" are indicators of the file's type, as reported by the "libmagic" library. descr/mime_type are only set if Bro is configured on a system that includes the "libmagic" library. event file_virus(c: connection, virname: string) Indicates the connection transferred an executable corresponding to a known virus of the given name. This functionality is only available if Bro is configured on a system that includes the "libclamav" library. Note, this analyzer is enabled via a call to expect_connection by the FTP analyzer. - New events relating to IRC analysis (Robin Sommer): event irc_client(c: connection, prefix: string, data: string) Generated upon seing a client message sent over the given IRC connection. "prefix" is the command's prefix as defined by the IRC protocol. It is used by servers to indicate the true origin of the message; it may be empty. "data" contains the message. event irc_server(c: connection, prefix: string, data: string) Same for server messages. event irc_user_message(c: connection, user: string, host: string, server: string, real_name: string) Generated upon seeing an IRC "USER" command. event irc_password_message(c: connection, password: string) Generated upon seeing an IRC "PASS" command. event irc_channel_topic(c: connection, channel: string, topic: string) Generated upon seeing an IRC server reply that includes the channel topic. event irc_global_users(c: connection, prefix: string, msg: string) Generated upon seeing an IRC server reply that includes a count of the number of IRC users. - The new experimental script irc-bot.bro tracks IRC-based bots (Robin Sommer). The accompanying script irc-bot-syslog.bro syslog's the state of the bot analysis every IrcBot::summary_interval seconds (default 1 minute). - The new script proxy.bro looks for open Web proxies by matching incoming requests to a server with outgoing requests it makes (Robin Sommer). It generates HTTPProxyFound Notices when it finds one. - Changes to notices.bro (Robin Sommer): - notice_policy_item's now have a default $result of NOTICE_FILE and a default $priority of 1. - The new notice_action_filter, notice_alarm_per_orig, alarms on the first NoticeType from a specific source. Subsequent instances are tallied. - notice_action_filters now reside in the new script notice-action-filter.bro (automatically loaded by notice.bro). - The notice actions NOTICE_ALARM_PER_CONN, NOTICE_ALARM_PER_ORIG, and NOTICE_ALARM_ONCE have been removed, as they were never actually implemented. - If the notice_policy returns IGNORE or FILE, the action_filters filters are no longer consulted. - A new attribute for tables and sets, &mergeable, changes the semantics of assignments, as follows (Robin Sommer). Given two &mergeable tables/sets A and B, an assignment "A = B" becomes actually a join "A = A \cup B" (i.e., union). The envisoned use is to help avoid race conditions when doing remote state synchronization. - The semantics of &synchronized expire_funcs has changed (Robin Sommer). Now, when a table entry is expired and the operation is propagated to a a peer, the peer will call its expire_function. - TRW analysis now skips UDP traffic because it currently treats all UDP connections as failures (Robin Sommer). - trw.bro has been split into trw-impl.bro (the algorithm) and trw.bro (which simply activates the analysis), to facilitate writing scripts that have hooks into TRW analysis but don't presume it's active (Robin Sommer). - The option report_remote_notices in remote.bro has been replaced by a new script you include, remote-report-notices.bro (Robin Sommer). - The new function connect_peer() explicitly connects to a remote host (Robin Sommer). - The new script remote-send-id.bro sends the current value of an ID to a remote Bro and then terminates processing (Robin Sommer). It's intended for use from the command-line, as in bro -e "redef dst="" id="" remote-send-id The other scripts must set up the connection. is an index into Remote::destinations corresponding to the destination. - New built-ins {suspend,resume}_state_updates() can be called to temporarily avoid propagating updates to &sync'ed values (Robin Sommer). This can avoid duplicated activity. - The new function terminate_communication() instructs Bro to end its communication with remote peers (Robin Sommer). - The new event remote_state_access_performed is raised when remote state access has been performed (Robin Sommer). This is primarily for debugging. - The log() built-in has been renamed to ln() to avoid conflict (Vern Paxson). - bifcl now generates event generation wrapper functions from event.bif (Ruoming Pang). For example, to generate event http_reply, currently one writes: val_list* vl = new val_list; vl->append(BuildConnVal()); vl->append(new StringVal(fmt("%.1f", reply_version))); vl->append(new Val(reply_code, TYPE_COUNT)); if ( reply_reason_phrase ) vl->append(reply_reason_phrase); else vl->append(new StringVal("")); ConnectionEvent(http_reply, vl); In the future, one will be able to just call bro_event_http_reply(), and the code generated by bifcl looks like: void bro_event_http_reply(Connection* c, StringVal* version, bro_uint_t code, StringVal* reason) { val_list* vl = new val_list; vl->append(c->BuildConnVal()); vl->append(version); vl->append(new Val(code, TYPE_COUNT)); vl->append(reason); mgr.QueueEvent(http_reply, vl, SOURCE_LOCAL, c); } Accompanying this change is a semantic shift to types "string" and "port" in .bif files. They used to be translated to C++ types BroString* and uint32, respectively. Now they are translated to StringVal* and PortVal*. The functions in bro.bif are changed accordingly, and please be aware of this change when you write built-in functions in future. Also for this change, the parameter 'new' for rsh_request has been renamed 'new_session', as 'new' is a reserved word for C++. - Some ICMP "connections" now have services identified ("icmp-echo", "icmp-unreach") rather than just listing the service as "other" (Ruoming Pang). - The new option remote_trace_sync_interval specifies an interval after which each Bro will stop processing its trace and wait for all others to signal that they have reached the same time (Robin Sommer). The intent is support for operating Bro in a distributed cluster fashion (and in particular for debugging such clusters when running off-line on traces). This option only works in pseudo-realtime mode, and requires the new global remote_trace_sync_peers to give the total number of remote peers (not including self). Signaling is done via a new communication message type. - Extensions for DNS transformation/anonymization, including introduction of trace transformation for protocols other than TCP (Jason Lee). Not yet fully developed/debugged. - Extensions for HTTP transformation/anonymization (Martin Casado). Not yet fully developed/debugged. - The $conn field is now included in HTTPProxyFound notices (Robin Sommer). - Changed service inference algorithm to favor lower-numbered likely-servers over higher-numbered ones (Vern Paxson). - In pseudo-realtime mode, Bro now uses real-time for deciding which peer should send state (Robin Sommer). - Time synchronization for Bro's running on traces in pseudo-realtime mode added (Robin Sommer). - Avoidance of false content gaps improved when sorting packets with out-of-order timestamps (Ruoming Pang). - Packets from the packet sorter are now more robustly drained upon termination of input (Ruoming Pang). - Documentation for deep-copy updated (Christian Kreibich). - Nasty fragment reassembly bug fixed (Vern Paxson). - Serious bugs in EDNS0 processing fixed (Vern Paxson). - Fixed significant misfeature of interconn.bro that stopped all processing of a connection once it makes a detection (Vern Paxson). - Fixes for &read_expire operation across synchronizes tables (Robin Sommer). - Fixes for multiple peers exchanging initial &sync state simultaneously (Robin Sommer). - Improvements to graceful termination of Bro when communicating with remote peers (Robin Sommer). - Fix for ICMP analyzer not always generating icmp_sent events (Robin Sommer). This appears to still need some work, as now it generates redundant events. - Fix for initial exchange of &sync state which could lead to referencing unknown IDs (Robin Sommer). - Fix to scan detection for differing semantics of connection compressor vs. non-compressor (Robin Sommer). - Bug fix for distinguishing regular expression matches of length 0 from those of length 1 (Ruoming Pang). - Fix for SSH version parsing in the presence of content gaps (Robin Sommer). - Bug fix for IRC that could lead to crashes (Robin Sommer). - Bug fix to refrain from adding new timers when a connection has already been removed from the connection table (Robin Sommer). - Bug fix for packet_contents not including the transport-layer header (Robin Sommer). - Some memory leaks fixed (Robin Sommer). - A bunch of portability and distribution problems fixed (Christian Kreibich, Robin Sommer, Vern Paxson). 1.1 Mon May 15 10:50:33 PDT 2006 - Bro now supports a "when" statement for taking action upon something becoming true asynchronously (Robin Sommer). This provides a powerful new mechanism with numerous applications. Syntax: when '(' ')' [timeout '{ '}'] where the first can be a single statement or a block enclosed in {}'s, but the set associated with "timeout" must be enclosed in {}'s (to reduce ambiguities in Bro's grammar). Bro executes the first statement when becomes true. If you give a timeout and the condition has not been satisfied before it expires, Bro executes the second statement instead. A simple example: global t: table[addr] of count; event connection_established(c: connection) { local orig = c$id$orig_h; if ( orig !in t ) { t[orig] = 1; when ( t[orig] == 5 ) print fmt("%s has established 5 connections", orig); timeout 1 hr { print fmt("%s has NOT established 5 connections", orig); delete t[orig]; } } else ++t[orig]; } Notes: - The condition may be evaluated more than once, and at arbitrary times. - When the when-body is executed, the condition is guaranteed to be still satisfied. - Expression reevaluation is primarily triggered by modifications to globals. However, reevaluations do not take place immediately but potentially at a later point. This means that if we change a global to a value which would execute the trigger but then change it back, the change may go unnoticed. - Inside the condition you may introduce new locals. For example, when ( (local x = foo()) && x == 42 ) ... Such an assignment always yields true as its expression value (but the assignment might be delayed, for example if foo() is a delayed function call - see below). Delaying function calls ======================= Functions called inside the condition of a when-clause may delay their results until they're ready. This works for both script-level and built-in functions. For script-level functions, there is a new construct, "return ", to delay a function's result. When used, the function returns at the time the when-stmt's condition becomes true, and it yields the value that the when-stmt's body then returns. Toy example: global X: table[string] of count; function a() : count { # This delays until condition becomes true. return when ( "a" in X ) { return X["a"]; } timeout 5 min { return 0; } } event bro_init() { # Installs a trigger which fires if a() returns 42. when ( a() == 42 ) { print "Yippie!"; } X["a"] = 42; } There's also a new built-in function which can delay lookup_addr(host: addr) performs asynchronous DNS address->hostname lookups. Example: local h; addr; [...] when (local name = lookup_addr(h)) { print h, name; } See the function gen_hot_notice_with_hostnames() in conn.bro for a more worked-out example of using the "when" clause to translate the local address in SensitiveConnection notices to a hostname (contributed by Brian Tierney). This functionality is activated by redef'ing xlate_hot_local_addr to T. Here is the full evaluation model of a when's condition: - The condition may be evaluated more than once, at arbitrary times. - It is always fully evaluated, no matter whether some former evaluation has been suspended by a delaying function call. - All function calls which do not delay are always *fully* executed each time the condition is evaluated. - Function calls which delay are only executed *once*; their result is cached and re-used in the case the condition is evaluated again. - The condition is guaranteed to be true when the body is executed (potentially using cached function results) - By default Bro now uses a configuration similar to what used to be activated using reduce-memory.bro, along with some additional state timeouts that are new (Robin Sommer and Vern Paxson). This allows for better state management out-of-the-box, at the cost of some precision of analysis and resilience to evasion. In particular, the intent is to move towards being able to run Bro continuously without inexorably growing the amount of memory used until exhaustion. You can access a configuration similar to the previous default state management settings by loading heavy-analysis.bro. It turns on a load-prefix of "heavy", so when you load XXX.bro, a file heavy.XXX.bro will also be automatically loaded if present. Note that, as was the case for reduce-memory, you need to load heavy-analysis prior to other files for it to have effect. - The new module clear-passwords.bro monitors login/FTP/IRC/POP traffic for cleartext passwords (Jason Lee). - The new script service-probe.bro looks for remote hosts that repeatedly connect to the same service on local hosts (for a configurable set of services and connection sizes) in order to detect brute-forcing attacks such as password-guessing (Jim Mellander). - A new ARP analyzer generates three events: event arp_request(mac_src: string, mac_dst: string, SPA: addr, SHA: string, TPA: addr, THA: string); event arp_reply(mac_src: string, mac_dst: string, SPA: addr, SHA: string, TPA: addr, THA: string); event bad_arp(SPA: addr, SHA: string, TPA: addr, THA: string, explanation: string); with a corresponding policy script arp.bro (Chema Gonzalez and Vern Paxson). It writes logs to arp.$BRO_LOG_SUFFIX. It has not been tested much yet. - Bro Lite changes (Jason Lee): - default user for is now user 'bro' - now uses the correct sysctl on FreeBSD 6 - now uses the correct Perl path if site-report.pl not installed into '/usr/local/bro' - no longer prompts to encrypt email unless you pick to email reports - The default Bro Lite install now only checkpoints Bro once a week (Brian Tierney). - Implicit Bro file extensions (such as .bro for policy scripts and .sig for signatures) are now searched for first rather than only if the non-extension-version of the file doesn't exist (Vern Paxson). For example, running "bro -r trace mt" now first searches $BROPATH for "mt.bro" before searching for "mt", whereas it used to do these in the other order. - There's now a simpler mechanism for redef'ing variables on the command-line (Christian Kreibich). Any command line arguments of the form = are now expanded into policy code of the form "redef var=val;", where is wrapped in quotation marks if the value appears to be a string and doesn't have quotation marks already. This works with strings with whitespace such as foo="Hello World"; however, note that it means you can't use the mechanism to redef an enum value. - The Bro distribution now includes (and builds by default) Christian Kreibich's Broccoli library (Bro C Client Library), which enables programs to communicate with running Bro's (Christian Kreibich and Jason Lee). Configure with --disable-broccoli to turn this off. - Built-in functions log(x: double): double and exp(x: double): double which do natural logarithms and their inverses (Jaeyeon Jung). - The new built-in function gethostname() returns the local host's name (Jason Lee & Robin Sommer). - The new built-in function reading_traces() returns true if Bro is reading trace files (Robin Sommer). - The new built-ins suspend_processing() and continue_processing() provide script-level control for instructing the event engine to stop or resume processing packets (Robin Sommer). This is useful for coordinating simultaneous processing by multiple Bro's. - Email notices are now by default sent via /bin/mail, with "[Bro Alarm]" in the subject. - redef'ing a function now replaces the existing body rather than supplementing it (Robin Sommer), which was a bug. - You can now configure Bro to process encapsulated IP packets either by setting, as before, a fixed encap_hdr_size (for VLANs), or setting parse_udp_tunnels to T (Ruoming Pang). For the latter, you specify a UDP tunnel port using udp_tunnel_port (the previous variable "tunnel_port" has gone away); or you can leave it set to its default of 0/udp, in which case Bro will look for IP encapsulated in UDP packets on any port. - Added a simple form of profiling based on sampling the work done per-packet (Vern Paxson). The event engine generates a event load_sample(samples: load_sample_info, CPU: interval, dmem: int) event every load_sample_freq packets (roughly; it's randomized), where load_sample_freq defaults to 20. "samples" is simply a set[string]; it contains the names of the functions, event handlers, and their source files that were accessed during the processing of the sampled packet, along with an estimate of the CPU cost of processing the packet and (currently broken) memory allocated/freed. - Bro now includes experimental support for Endace DAG cards (Gregor Maier and Robin Sommer). To activate, configure with --with-DAG=/path/to/dagtool/installation and use "dag0" as the network interface. You may need to configure the card with the dagtools first. In general, if dagsnap works, Bro should work as well. - Log rotation has changed in a number of ways (Mark Dedlow & Robin Sommer): * The new variable log_rotate_base_time: string, if defined, specifies that logs should be rotated at log_rotate_base_time + i * rotate_interval intervals. Format is as a string in 24-hour time, "%H:%M", e.g, "12:00". This format may change in the future to instead be a Bro time type. * RotateLogs::date_format can be redefined to change format of timestamps in rotated files. * RotateLogs::build_name() can be redefined to implement an arbitrary naming scheme for rotated files. Note, this code has not been extensively tested. - Bro now by default builds a version of malloc bundled with its distribution (Vern Paxson & Brian Tierney). - The syntax for the clone operator now looks like a function call, "copy(x)" (Vern Paxson). - The new flag DNS::logging (default F), if T, disables generation of dns.log (which is often uninteresting and very large), though it still performs analysis leading to NOTICEs (Robin Sommer). - A new global, hostile_domain_list, has been added to dns.bro which lists domains to be flagged if A or MX records are queried (Scott Campbell). - Added globals dns_skip_all_{auth,addl} to skip all DNS AUTH/ADDL processing (Vern Paxson). Skipping these is on (true) by default, because such processing is quite expensive. - backdoor.bro now turns off by default some detectors that from experience have too many false positives, or (such as for HTTP) too many uninteresting true positives (Brian Tierney). In addition: - the module now generates a BackdoorFound notice for each backdoor - the new variable dump_backdoor_packets (default F) if set causes the packet that triggered the backdoor detection to be written to backdoor-packets/: