diff --git a/.gitmodules b/.gitmodules index 95053091cf..2ede715f49 100644 --- a/.gitmodules +++ b/.gitmodules @@ -16,3 +16,6 @@ [submodule "cmake"] path = cmake url = git://git.bro-ids.org/cmake +[submodule "magic"] + path = magic + url = git://git.bro.org/bromagic diff --git a/CHANGES b/CHANGES index 89d79e2f07..f9151efd79 100644 --- a/CHANGES +++ b/CHANGES @@ -1,4 +1,2512 @@ +2.1-1036 | 2013-08-05 17:29:11 -0400 + + * Fix the SSL infinite loop I just created. (Seth Hall) + +2.1-1035 | 2013-08-05 16:44:50 -0400 + + * Change to SSL log delay to cause the log to write even if delay times out. (Seth Hall) + +2.1-1034 | 2013-08-03 20:27:43 -0700 + + * A set of DHCP extensions. (Vlad Grigorescu) + + - Leases are logged to dhcp.log as they are seen. + - scripts/policy/protocols/dhcp/known-devices-and-hostnames.bro + - Added DPD sig. + +2.1-1027 | 2013-08-03 01:57:37 -0400 + + * Fix a major memory issue in the SumStats framework. + +2.1-1026 | 2013-08-02 22:35:09 -0400 + + * Fix the SumStats top-k plugin and test. (Seth Hall) + + * Rework of SumStats API to reduce high instantaneous memory + use on clusters. (Seth Hall) + + * Large update for the SumStats framework. + + - On-demand access to sumstats results through "return from" + functions named SumStats::request and Sumstats::request_key. + Both functions are tested in standalone and clustered modes. + + - $name field has returned to SumStats which simplifies cluster + code and makes the on-demand access stuff possible. + + - Clustered results can only be collected for 1 minute from their + time of creation now instead of time of last read. + + - Thresholds use doubles instead of counts everywhere now. + + - Calculation dependency resolution occurs at start up time now + instead of doing it at observation time which provide a minor + cpu performance improvement. A new plugin registration mechanism + was created to support this change. + + - AppStats now has a minimal doc string and is broken into hook-based + plugins. + + - AppStats and traceroute detection added to local.bro (Seth Hall) + +2.1-1009 | 2013-08-02 17:19:08 -0700 + + * A number of exec module and raw input reader fixes. (Jon Siwek) + +2.1-1007 | 2013-08-01 15:41:54 -0700 + + * More function documentation. (Bernhard Amann) + +2.1-1004 | 2013-08-01 14:37:43 -0700 + + * Adding a probabilistic data structure for computing "top k" + elements. (Bernhard Amann) + + The corresponding functions are: + + topk_init(size: count): opaque of topk + topk_add(handle: opaque of topk, value: any) + topk_get_top(handle: opaque of topk, k: count) + topk_count(handle: opaque of topk, value: any): count + topk_epsilon(handle: opaque of topk, value: any): count + topk_size(handle: opaque of topk): count + topk_sum(handle: opaque of topk): count + topk_merge(handle1: opaque of topk, handle2: opaque of topk) + topk_merge_prune(handle1: opaque of topk, handle2: opaque of topk) + +2.1-971 | 2013-08-01 13:28:32 -0700 + + * Fix some build errors. (Jon Siwek) + + * Internal refactoring of how plugin components are tagged/managed. + (Jon Siwek) + + * Fix various documentation, mostly related to file analysis. (Jon + Siwek) + + * Changing the Bloom filter hashing so that it's independent of + CompositeHash. (Robin Sommer) + +2.1-951 | 2013-08-01 11:19:23 -0400 + + * Small fix to deal with a bug in the SSL log delay mechanism. + +2.1-948 | 2013-07-31 20:08:28 -0700 + + * Fix segfault caused by merging an empty bloom-filter with a + bloom-filter already containing values. (Bernhard Amann) + +2.1-945 | 2013-07-30 10:05:10 -0700 + + * Make hashers serializable. (Matthias Vallentin) + + * Add docs and use default value for hasher names. (Matthias + Vallentin) + +2.1-939 | 2013-07-29 15:42:38 -0700 + + * Added Exec, Dir, and ActiveHTTP modules. (Seth Hall) + + base/utils/exec.bro provides a module to start external processes + asynchronously and retrieve their output on termination. + base/utils/dir.bro uses it to monitor a directory for changes, and + base/utils/active-http.bro for providing an interface for querying + remote web servers. + +2.1-930 | 2013-07-29 15:06:07 -0700 + + * Major file analysis overhaul in naming and appearance, along with + fixes and test updates. (Seth Hall and Jon Siwek) + + Includes: + + * Added protocol description functions that provide a super + compressed log representation. (Seth Hall) + + * Added mime types to http.log (Seth Hall) + + * Add jar files to the default MHR lookups. (Seth Hall) + + * Adding CAB files for MHR checking. (Seth Hall) + + * Improve malware hash registry script. + + - Include a link to a virustotal search in the notice sub message field. + - Give all information returned from Team Cymru in the notice message. + - Add more file types to match on to the default set. + + * Make the custom libmagic database a git submodule. + + * Add an is_orig parameter to file_over_new_connection event. + + * Recorrected the module name to Files. + + * Added Files::analyzer_name to get a more readable name for a + file analyzer. + + * Improved and just overall better handled multipart mime + transfers in HTTP and SMTP. HTTP now has orig_fuids and + resp_fuids log fields since multiple "files" can be transferred + with multipart mime in a single request/response pair. SMTP has + an fuids field which has file unique IDs for all parts + transferred. FTP and IRC have a log field named fuid added + because only a single file can be transferred per irc and ftp + log line. + +2.1-895 | 2013-07-29 14:07:35 -0700 + + * Adding a test for a DNSKEY RR. (Robin Sommer) + +2.1-894 | 2013-07-29 16:44:41 -0400 + + * Updates for the Intel Framework. (Seth Hall) + + - policy/frameworks/intel/seen is the new location for the + scripts that push data into the intel framework for checking. + + - The new policy/frameworks/intel/do_notice script adds an + example mechanism for data driven notices. + + - Remove the Intel insertion after heuristically detecting SSH + bruteforcing. + + - Intel importing format has changed (refer to docs). + + - All string matching is now case insensitive. + + - SMTP intel script has been updated to extract email + addresses correctly. + + - Small fix sneaking into the smtp base script to actually + extract individual email addresses in the To: field + correctly. + + +2.1-888 | 2013-07-25 12:02:41 -0700 + + * Protection about broken traces with empty pcap headers. (Matt + Thompson) + +2.1-887 | 2013-07-25 11:33:27 -0700 + + * Support for Bloom filter. (Matthias Vallentin) + + Bro now provides the following BiFs: + + bloomfilter_basic_init(fp: double, capacity: count, name: string &default=""): opaque of bloomfilter + bloomfilter_counting_init(k: count, cells: count, max: count, name: string &default=""): opaque of bloomfilter + bloomfilter_add(bf: opaque of bloomfilter, x: any) + bloomfilter_lookup(bf: opaque of bloomfilter, x: any): count + bloomfilter_merge(bf1: opaque of bloomfilter, bf2: opaque of bloomfilter): opaque of bloomfilter + bloomfilter_clear(bf: opaque of bloomfilter) + + Note that currently Bloom filters from separate Bro instances + (e.g., from different cluster nodes) cannot be merged. + +2.1-826 | 2013-07-25 10:12:26 -0700 + + * bif files declared with bif_target() are now automatically + compiled in. No more manual includes to pull them in. (Robin + Sommer) + + * Covenience make target in testing/btest to update the three + coverage tests that usually need tweaking when scripts get + added/removed. (Robin Sommer) + +2.1-824 | 2013-07-22 14:25:14 -0400 + + * Fixed a scriptland state issue that manifested especially badly on proxies. (Seth Hall) + + * Another test fix. (Robin Sommer) + + * Canonyfying the output of core.print-bpf-filters. (Robin Sommer) + +2.1-820 | 2013-07-18 12:30:04 -0700 + + * Extending external canonifier to remove fractional values from + capture_loss.log. (Robin Sommer) + + * Canonifying internal order for plugins and their components to + make it deterministic. (Robin Sommer) + + * Small raw reader tweaks that got left our earlier. (Robin Sommer) + +2.1-814 | 2013-07-15 18:18:20 -0700 + + * Fixing raw reader crash when accessing nonexistant file, and + memory leak when reading from file. Addresses #1038. (Bernhard + Amann) + +2.1-811 | 2013-07-14 08:01:54 -0700 + + * Bump sqlite to 3.7.17. (Bernhard Amann) + + * Small test fixes. (Seth Hall) + + * Fix a bug where the same analyzer tag was reused for two different + analyzers. (Seth Hall) + + * Moved DPD signatures into script specific directories. Left out + the BitTorrent signatures pending further updates to that + analyzer. (Seth Hall) + +2.1-802 | 2013-07-10 10:55:14 -0700 + + * Const adjustment for methods. (Jon Siwek) + +2.1-798 | 2013-07-08 13:05:37 -0700 + + * Rewrite of the packet filter framework. (Seth Hall) + + This includes: + + - Plugin interface for adding filtering mechanisms. + + - Integrated the packet filter framework with the analyzer + framework to retrieve well-known ports from there. + + - Support for BPF-based load balancing (IPv4 and IPv6). This will + tie in with upcoming BroControl support for configuring this. + + - Support for BPF-based connection sampling. + + - Support for "shunting" traffic with BPF filters. + + - Replaced PacketFilter::all_packets with + PacketFilter::enable_auto_protocol_capture_filters. + +2.1-784 | 2013-07-04 22:28:48 -0400 + + * Add a call to lookup_connection in SSH scripts to update connval. (Seth Hall) + + * Updating submodule(s). (Robin Sommer) + +2.1-782 | 2013-07-03 17:00:39 -0700 + + * Remove the SSL log queueing mechanism that was included with the + log delay mechanism. (Seth Hall) + +2.1-780 | 2013-07-03 16:46:26 -0700 + + * Rewrite of the RAW input reader for improved robustness and new + features. (Bernhard Amann) This includes: + + - Send "end_of_data" event for all kind of streams. + - Send "process_finished" event with exit code of child + process at process termination. + - Expose name of input stream to readers. + - Better error handling. + - New "force_kill" option which SIGKILLs processes on reader termination. + - Supports reading from stdout and stderr simultaneously. + - Support sending data to stdin of child process. + - Streaming reads from external commands work without blocking. + +2.1-762 | 2013-07-03 16:33:22 -0700 + + * Fix to correct support for TLS 1.2. Addresses #1020. (Seth Hall, + with help from Rafal Lesniak). + +2.1-760 | 2013-07-03 16:31:36 -0700 + + * Teach broxygen to generate protocol analyzer plugin reference. + (Jon Siwek) + + * Adding 'const' to a number of C++ methods. (Jon Siwek) + +2.1-757 | 2013-07-03 16:28:10 -0700 + + * Fix redef of table index from clearing table. + + `redef foo["x"] = 1` now acts like `redef foo += { ["x"] = 1 }` + instead of `redef foo = { ["x"] = 1 }`. + + Addresses #1013. (Jon Siwek) + + +2.1-755 | 2013-07-03 16:22:43 -0700 + + * Add a general file analysis overview/how-to document. (Jon Siwek) + + * Improve file analysis doxygen comments. (Jon Siwek) + + * Improve tracking of HTTP file extraction. http.log now has files + taken from request and response bodies in different fields for + each, and can now track multiple files per body. That is, the + "extraction_file" field is now "extracted_request_files" and + "extracted_response_files". Addresses #988. (Jon Siwek) + + * Fix HTTP multipart body file analysis. Each part now gets assigned + a different file handle/id. (Jon Siwek) + + * Remove logging of analyzers field of FileAnalysis::Info. (Jon + Siwek) + + * Remove extraction counter in default file extraction scripts. (Jon + Siwek) + + * Remove FileAnalysis::postpone_timeout. + FileAnalysis::set_timeout_interval can now perform same function. + (Jon Siwek) + + * Make default get_file_handle handlers &priority=5 so they're + easier to override. (Jon Siwek) + + * Add input interface to forward data for file analysis. The new + Input::add_analysis function is used to automatically forward + input data on to the file analysis framework. (Jon Siwek) + + * File analysis framework interface simplifications. (Jon Siwek) + + - Remove script-layer data input interface (will be managed directly + by input framework later). + + - Only track files internally by file id hash. Chance of collision + too small to justify also tracking unique file string. + + +2.1-741 | 2013-06-07 17:28:50 -0700 + + * Fixing typo that could cause an assertion to falsely trigger. + (Robin Sommer) + +2.1-740 | 2013-06-07 16:37:32 -0700 + + * Fix for CMake 2.6.x. (Robin Sommer) + +2.1-738 | 2013-06-07 08:38:13 -0700 + + * Remove invalid free on non-allocated pointer in hash function + object. Addresses #1018. (Matthias Vallentin) + +2.1-736 | 2013-06-06 10:05:20 -0700 + + * New "magic constants" @DIR and @FILENAME that expand to the + directory path of the current script and just the script file name + without path, respectively. (Jon Siwek) + +2.1-731 | 2013-06-04 21:19:08 -0700 + + * Reorginization of internal protocol analyzer code. We're moving + them to a modularized structure, based on a plugin model. Along + with this change comes generic plugin infrastructure that we'll + later extend to other Bro component as well. For now all plugins + are compiled in statically, but in the future we plan to also + enable dynamic loading at run time. (Robin Sommer) + + * Ignoring file ids in external tests. (Robin Sommer) + +2.1-675 | 2013-06-02 20:03:19 -0700 + + * Fix a compiler warning. (Robin Sommer) + + * Allow named vector/set/table/record constructors. Addresses #983. + (Jon Siwek) + + * Adding Makefile target test-all that also runs the BroControl test + suite. (Robin Sommer) + +2.1-664 | 2013-05-28 21:37:46 -0700 + + * Dangling pointer fix. Addresses #1004. (Jon Siwek) + +2.1-659 | 2013-05-24 17:24:18 -0700 + + * Fix broken/missing documentation. (Jon Siwek) + + * Fixing test that would fail without ES/curl support. (Robin + Sommer) + +2.1-656 | 2013-05-17 15:58:07 -0700 + + * Fix mutex lock problem for writers. (Bernhard Amann) + +2.1-654 | 2013-05-17 13:49:52 -0700 + + * Tweaks to sqlite3 configuration to address threading issues. + (Bernhard Amann) + +2.1-651 | 2013-05-17 13:37:16 -0700 + + * Fix uninitialized DPM member. (Jon Siwek) + + * Fix issue with transaction ID reuse in a single DNS connection. (Seth Hall) + + * New function added to the queue.bro script to support peeking at + the new gettable item in the queue without removing it. (Seth Hall) + +2.1-647 | 2013-05-17 07:47:14 -0700 + + * Fixing Broxygen generation to have BROMAGIC set. (Robin Sommer) + + * Fix for 'fchmod undeclared here' on FreeBSD. (Robin Sommer) + + * CMake policy fix to avoid errors with older versions. (Robin + Sommer) + +2.1-641 | 2013-05-15 18:15:09 -0700 + + * Test update. (Robin Sommer) + +2.1-640 | 2013-05-15 17:24:09 -0700 + + * Support for cleaning up threads that have terminated. (Bernhard + Amann and Robin Sommer). Includes: + + - Both logging and input frameworks now clean up threads once + they aren't further needed anymnore. + + - New function Log::remove_stream() that removes a logging + stream, stopping all writer threads that are associated with + it. Note, however, that removing a *filter* from a stream + still doesn't clean up any threads. The problem is that + because of the output paths potentially being created + dynamically it's unclear if the writer thread will still be + needed in the future. + +2.1-626 | 2013-05-15 16:09:31 -0700 + + * Add "reservoir" sampler for SumStats framework. This maintains + a set of N uniquely distributed random samples. (Bernhard Amann) + +2.1-619 | 2013-05-15 16:01:42 -0700 + + * SQLite reader and writer combo. This allows to read/write + persistent data from on disk SQLite databases. The current + interface is quite low-level, we'll add higher-level abstractions + in the future. (Bernhard Amann) + +2.1-576 | 2013-05-15 14:29:09 -0700 + + * Initial version of new file analysis framework. This moves most of + the processing of file content from script-land into the core, + where it belongs. Much of this is an internal change, and at this + point the new code has essentially feature-equality with the old + one. More script-level changes to come. (Jon Siwek) + +2.1-502 | 2013-05-10 19:29:37 -0700 + + * Allow default function/hook/event parameters. Addresses #972. (Jon + Siwek) + + * Change the endianness parameter of bytestring_to_count() BIF to + default to false (big endian). (Jon Siwek) + +2.1-500 | 2013-05-10 19:22:24 -0700 + + * Fix to prevent merge-hook of SumStat's unique plugin from damaging + source data. (Bernhard Amann) + +2.1-498 | 2013-05-03 17:44:08 -0700 + + * Table lookups return copy of non-const &default vals. This + prevents unintentional modifications to the &default value itself. + Addresses #981. (Jon Siwek) + +2.1-496 | 2013-05-03 15:54:47 -0700 + + * Fix memory leak and unnecessary allocations in OpaqueVal. + Addresses #986. (Matthias Vallentin) + +2.1-492 | 2013-05-02 12:46:26 -0700 + + * Work-around for sumstats framework not propagating updates after + intermediate check in cluster environments. (Bernhard Amann) + + * Always apply tcp_connection_attempt. Before this change it was + only applied when a connection_attempt() event handler was + defined. (Robin Sommer) + + * Fixing coverage.bare-mode-errors test. (Robin Sommer) + +2.1-487 | 2013-05-01 18:03:22 -0700 + + * Always apply tcp_connection_attempt timer, even if no + connection_attempt() event handler is defined. (Robin Sommer) + +2.1-486 | 2013-05-01 15:28:45 -0700 + + * New framework for computing summary statistics in + base/framework/sumstats. This replaces the metrics frameworks, and + comes with a number of applications build on top, see NEWS. More + documentation to follow. (Seth Hall) + +2.1-397 | 2013-04-29 21:19:00 -0700 + + * Fixing memory leaks in CompHash implementation. Addresses #987. + (Robin Sommer) + +2.1-394 | 2013-04-27 15:02:31 -0700 + + * Fixed a bug in the vulnerable software script and added a test. + (Seth Hall) + + * Fix schedule statements used outside event handlers. Addresses + #974. (Jon Siwek) + + * Fix record coercion for default inner record fields. Addresses + #973. (Jon Siwek) + + * Add bytestring_to_count function to bro.bif. Addresses #968. (Yun + Zheng Hu) + +2.1-386 | 2013-03-22 12:41:50 -0700 + + * Added reverse() function to strings.bif. (Yun Zheng Hu) + +2.1-384 | 2013-03-22 12:10:14 -0700 + + * Fix record constructors in table initializer indices. Addresses + #660. (Jon Siwek) + +2.1-382 | 2013-03-22 12:01:34 -0700 + + * Add support for 802.1ah (Q-in-Q). Addresses #641. (Seth Hall) + +2.1-380 | 2013-03-18 12:18:10 -0700 + + * Fix gcc compile warnings in base64 encoder and benchmark reader. + (Bernhard Amann) + +2.1-377 | 2013-03-17 17:36:09 -0700 + + * Fixing potential leak in DNS error case. (Vlad Grigorescu) + +2.1-375 | 2013-03-17 13:14:26 -0700 + + * Add base64 encoding functionality, including new BiFs + encode_base64() and encode_base64_custom(). (Bernhard Amann) + + * Replace call to external "openssl" in extract-certs-pem.bro with + that encode_base64(). (Bernhard Amann) + + * Adding a test for extract-certs-pem.pem. (Robin Sommer) + + * Renaming Base64Decoder to Base64Converter. (Robin Sommer) + +2.1-366 | 2013-03-17 12:35:59 -0700 + + * Correctly handle DNS lookups for software version ranges. (Seth + Hall) + + * Improvements to vulnerable software detection. (Seth Hall) + + - Add a DNS based updating method. This needs to be tested + still. + + - Vulnerable version ranges are used now instead of only single + versions. This can deal with software with multiple stable + major versions. + + * Update software version parsing and comparison to account for a + third numeric subversion. Also, $addl is now compared numerically + if the value is actually numeric. (Seth Hall) + +2.1-361 | 2013-03-13 07:18:22 -0700 + + * Add check for truncated link frames. Addresses #962. (Jacob + Baines) + + * Fix large memory allocation in IP fragment reassembly. Addresses + #961. (Jacob Baines) + +2.1-357 | 2013-03-08 09:18:35 -0800 + + * Fix race-condition in table-event test. (Bernhard Amann) + + * s/bro-ids.org/bro.org/g. (Robin Sommer) + +2.1-353 | 2013-03-07 13:31:37 -0800 + + * Fix function type-equivalence requiring same parameter names. + Addresses #957. (Jon Siwek) + +2.1-351 | 2013-03-07 13:27:29 -0800 + + * Fix new/delete mismatch. Addresses #958. (Jacob Baines) + + * Fix compiler warnings. (Jon Siwek) + +2.1-347 | 2013-03-06 16:48:44 -0800 + + * Remove unused parameter from vector assignment method. (Bernhard Amann) + + * Remove the byte_len() and length() bifs. (Bernhard Amann) + +2.1-342 | 2013-03-06 15:42:52 -0800 + + * Moved the Notice::notice event and Notice::policy table to both be + hooks. See documentation and NEWS for information. (Seth Hall). + +2.1-338 | 2013-03-06 15:10:43 -0800 + + * Fix init of local sets/vectors via curly brace initializer lists. + (Jon Siwek) + +2.1-336 | 2013-03-06 15:08:06 -0800 + + * Fix memory leaks resulting from 'when' and 'return when' + statements. Addresses #946. (Jon Siwek) + + * Fix three bugs with 'when' and 'return when' statements. Addresses + #946. (Jon Siwek) + +2.1-333 | 2013-03-06 14:59:47 -0800 + + * Add parsing for GTPv1 extension headers and control messages. (Jon Siwek) + + This includes: + + - A new generic gtpv1_message() event generated for any GTP + message type. + + - Specific events for the create/update/delete PDP context + request/response messages. + + Addresses #934. + +2.1-331 | 2013-03-06 14:54:33 -0800 + + * Fix possible null pointer dereference in identify_data BIF. Also + centralized libmagic calls for consistent error handling/output. + (Jon Siwek) + + * Fix build on OpenBSD 5.2. (Jon Siwek) + +2.1-328 | 2013-02-05 01:34:29 -0500 + + * New script to query the ICSI Certificate Notary + (http://notary.icsi.berkeley.edu/) over DNS and add information + to the SSL log at runtime. (Matthias Vallentin) + + * Add delayed logging to SSL base scripts. (Matthias Vallentin) + +2.1-319 | 2013-02-04 09:45:34 -0800 + + * Update input tests to use exit_only_after_terminate. (Bernhard + Amann) + + * New option exit_only_after_terminate to prevent Bro from exiting. + If set, the main loop won't terminate before somebody calls + terminate(). (Robin Sommer) + +2.1-311 | 2013-02-01 08:03:01 -0800 + + * Updating submodule(s). + +2.1-310 | 2013-01-30 20:09:27 -0800 + + * Add an error for record coercions that would orphan a field. (Jon + Siwek) + + * Fixing several scripts where a field in an inlined record was + never removed after a code refactor. (Jon Siwek) + +2.1-307 | 2013-01-25 13:50:57 -0800 + + * Fix runaway reference counting bug in record coercion. (Jon Siwek) + + * Fix memory leak in some reporter messaging cases. (Jon Siwek) + +2.1-304 | 2013-01-23 19:43:27 -0800 + + * Making a test portable. (Robin Sommer) + +2.1-302 | 2013-01-23 16:17:29 -0800 + + * Refactoring ASCII formatting/parsing from loggers/readers into a + separate AsciiFormatter class. (Bernhard Amann) + + * Fix uninitialized locals in event/hook handlers from having a + value. Addresses #932. (Jon Siwek) + + * Add a null value check in CompositeHash::ComputeHash. Addresses + #930. (Jon Siwek) + + * Change reporter messages to more reliably print to stderr. + Addressed #930 (and revisits #836). (Jon Siwek) + + * Changing test=suite's btest call to use "-j" instead of "-j 5". + (Robin Sommer) + + * Require "case" blocks to end with either "break", "return", or a + new "fallthrough" statement that passes control on to the + subsequent case. This gives us the best mix of safety, + readability, and flexibility. Addresses #754. (Jon Siwek) + +2.1-279 | 2013-01-18 17:18:22 -0800 + + * Revert "Trick for parallelizing input framework unit tests." The + old way of doing the tests seems more reliable for now. (Jon + Siwek) + + * Fixing variable size issues with http response code in + ElasticSearch writer. (Gilbert Clark) + + * Removing unused class member. (Robin Sommer) + + * Add opaque type-ignoring for the accept_unsupported_types input + framework option. (Bernhard Amann) + +2.1-271 | 2013-01-08 10:18:57 -0800 + + * Change substring index notation to use a colon. String slice + notation is now written as `s[1:2]`. Addresses #422. (Jon Siwek) + +2.1-268 | 2013-01-07 09:43:44 -0800 + + * Fix memory leak in OpaqueType::DoUnserialize. (Jon Siwek) + +2.1-265 | 2012-12-20 17:38:42 -0800 + + * Add array-style index accessor for strings. Addresses #422. (Jon + Siwek) + + The index expression can take up to two indices for the start and + end index of the substring to return (e.g. "mystring[1,3]"). + Negative indices are allowed, with -1 representing the last + character in the string. The indexing is not cyclic -- if the + starting index is >= the length of the string an empty string is + returned, and if the ending index is >= the length of the string + then it's interpreted as the last index of the string. Assigning + to substrings accessed like this isn't allowed. + +2.1-263 | 2012-12-20 16:22:09 -0800 + + * Bro's language now has a new set of types "opaque of X". (Matthias + Vallentin) + + Opaque values can be passed around like other values but they can + only be manipulated with BiF functions, not with other operators. + Currently, the following opaque types are supported: + + - opaque of md5 + - opaque of sha1 + - opaque of sha256 + - opaquey of entropy. + + They go along with the corrsponding BiF functions md5_*, sha1_*, + sha256_*, and entropy_*, respectively. Note that these functions + have changed their signatures to work with opaques types rather + than global state as it was before. + +2.1-240 | 2012-12-20 15:21:07 -0800 + + * Improve error for invalid use of types as values. Addresses #923. + (Jon Siwek) + +2.1-238 | 2012-12-20 15:11:25 -0800 + + * Finish implementation of script-layer switch statement. Addresses + #754. (Jon Siwek) + + They behave like C-style switches except case labels can be + comprised of multiple literal constants delimited by commas. Only + atomic types are allowed for now. Case label bodies that don't + execute a "return" or "break" statement will fall through to + subsequent cases. A default case label is allowed. + + * Fix a case where c$resp$size is misrepresented. Addresses #730. + (Jon Siwek) + +2.1-234 | 2012-12-20 12:12:19 -0800 + + * Fix return value of hook calls that have no handlers. For this + case, the return value is always true. (Jon Siwek) + + * Fix to_port() BIF for port strings with a port number of zero. + (Jon Siwek) + +2.1-231 | 2012-12-14 14:51:35 -0800 + + * Make const variables actually constant. Both local and global + variables declared with "const" could be modified, but now + expressions that would modify them generate an error message at + parse-time. Addresses #922. (Jon Siwek) + +2.1-229 | 2012-12-14 14:46:12 -0800 + + * Fix memory leak in ASCII reader when encoutering errors in input. + (Bernhard Amann) + + * Improvements for the "bad checksums" detector to make it detect + bad TCP checksums. (Seth Hall) + +2.1-223 | 2012-12-12 14:25:15 -0800 + + * Trick for parallelizing input framework unit tests. Instead of + loading listen.bro to block until files are read, just read a pcap + file in pseudo-realtime. (Jon Siwek) + + * Fix reliability of a unit test that relies on when statements. + (Jon Siwek) + + * Remove unused attributes. (Daniel Thayer) + - Removed attributes &postprocessor and &match from documentation and source code. + - Removed undocumented attribute &attr from source code. + - Removed internal attribute "(&tracked)" from documentation. + +2.1-218 | 2012-12-10 14:45:04 -0800 + + * Add GPRS Tunnelling Protocol (GTPv1) decapsulation. This currently + supports automatic decapsulation of GTP-U packets on UDP port 2152. + The GTPv1 headers for such tunnels can be inspected by handling + the "gtpv1_g_pdu_packet" event, which has a parameter of type + "gtpv1_hdr". Addresses #690. (Jon Siwek; derived from patch by + Carsten Langer) + + * Change BinPAC exceptions in AYIYA/GTP analyzers to do + "protocol_violation". (Jon Siwek) + +2.1-212 | 2012-12-07 19:42:03 -0800 + + * Changing the HTTP parser to accept request methods in alignment + with the RFC. (Robin Sommer) + +2.1-209 | 2012-12-05 16:44:04 -0800 + + * Adapting the HTTP request line parsing to only accept methods + consisting of letters [A-Za-z]. (Robin Sommer) + +2.1-207 | 2012-12-05 15:47:32 -0800 + + * Reporting warnings if kill/waitpid fail in communication system. + (Bill Parker) + + * Replace() bzero with memset(). (Bill Parker) + + * Merge remote-tracking branch 'vlad/topic/vladg/http-verbs' + + * vlad/topic/vladg/http-verbs: + A test for HTTP methods, including some horribly illegal requests. + Remove hardcoded HTTP verbs from the analyzer (#741) + + I added a "bad_HTTP_request" weird for HTTP request lines that don't + have more than a single word. + + Closes #741. (Robin Sommer) + + * A test for HTTP methods, including some horribly illegal requests. (Vlad Grigorescu) + + * Remove hardcoded HTTP verbs from the analyzer (#741) (Vlad Grigorescu) + + +2.1-203 | 2012-12-05 14:36:56 -0800 + + * Fix segfault: Synchronization of state between connecting peers + now skips over identifiers that aren't initialized with a value + yet. Addresses #66. (Jon Siwek) + + * Fix segfault: Delete correct entry in error case in input + framework. (Bernhard Amann) + + * Bad record constructor initializers now give an error. Addresses + #34. (Jon Siwek) + + * Invalid vector indices now generate error message. Addresses #24. + (Jon Siwek) + + * Bump CPack RPM package requirement to Python >= 2.6.0. (Jon Siwek) + + * Interpreter exceptions occurring in "when" blocks are now handled. + Addresses #779 (Jon Siwek) + +2.1-195 | 2012-12-03 14:50:33 -0800 + + * Catching out-of-memory in patricia tree code. (Bill Parker) + +2.1-194 | 2012-12-03 14:36:26 -0800 + + * Renaming ASCII writer filter option 'only_single_header_row' to + 'tsv'. Also clarifying usage. Closes #912. (Robin Sommer) + +2.1-193 | 2012-12-03 14:11:14 -0800 + + * Fix a set of bugs with table/set attributes. (Jon Siwek) + + - Identifiers that are initialized with set()/table() constructor + expressions now inherit attributes from the expression. Before, + statements like + + const i: set[string] = set() &redef; + + associated the attribute with the set() constructor, but not the + "i" identifier, preventing redefinition. Addresses #866. + + - Allow &default attribute to apply to tables initialized as empty + (via either "{ }" or "table()") or if the expression supplied to it + can evaluate to a type that's promotable to the same yield type as + the table. + +2.1-191 | 2012-12-03 14:08:56 -0800 + + * Add test of record() constructor to table initializer unit test. + (Jon Siwek) + + * Fix table(), set(), vector() constructors in table initializer + lists. Also adds type checking of yield values to table() + constructor and fixes the type checking of yield values in + vector() constructor. Addresses #5. (Jon Siwek) + +2.1-188 | 2012-12-03 14:04:29 -0800 + + * Hook functions now callable with "hook" expression (i.e., hook is + no longer a statement). The return value of the call is an + implicit boolean value of T if all hook handlers ran, or F if one + hook handler exited as a result of a break statement and + potentially prevented other handlers from running. + + Scripts don't need to declare hooks with an explicit return type of bool + (internally, that's assumed), and any values given to (optional) return + statements in handler definitions are just ignored. + + Addresses #918. (Jon Siwek) + + * Clarification in hook documentation. (Jon Siwek) + +2.1-184 | 2012-12-03 13:59:50 -0800 + + * Slightly fix up file name extraction from Content-Disposition + headers. (Seth Hall) + + * Adding -b flag to bro in unit tests so they run faster. + + * Fixed a DNS attribute issue. Reported by Matt Thompson. (Seth + Hall) + + * Adding NEWS placeholder for hooks and CSV mode. (Robin Sommer) + +2.1-178 | 2012-11-23 19:35:32 -0800 + + * The ASCII writer now supports a new filter config option + "only_single_header_row" that turns the output into CSV format + when set to "T". (Carsten Langer) + + * Add new function flavor called a "hook". This new flavor of + function behaves like a "synchronous event". See + doc/scripts/builtins.rst more details on usage. (Jon Siwek) + + * Improve auto-generated enum documentation. The names of enum types + are tracked so that variables holding a value of a given enum type + can generate a reference to it instead of just listing the type as + a generic "enum". (Jon Siwek) + +2.1-171 | 2012-11-23 18:24:15 -0800 + + * Fix ambiguity between composite table index and record ctor + expressions. If a table type is "global t = table[conn_id, bool] + of count", then checking membership like "[c$id, is_orig] in t" + now works. Addresses #80. (Jon Siwek) + +2.1-169 | 2012-11-23 18:21:32 -0800 + + * Fix some warnings from sphinx when building docs. (Jon Siwek) + +2.1-167 | 2012-11-14 13:19:17 -0800 + + * Add a new BIF "bytestring_to_double" for converting from a binary + representation of a double. Addresses #908. (Carsten Langer/Daniel + Thayer) + +2.1-162 | 2012-11-13 17:29:00 -0800 + + * Fix modbus register array parsing. (Jon Siwek) + + * Adjustments to modbus test cases. (Jon Siwek) + +2.1-157 | 2012-11-08 16:22:00 -0800 + + * Fix for lookup_hostname BIF. (Jon Siwek) + + * Fix for modbus test portability. (Robin Sommer) + +2.1-152 | 2012-11-05 16:52:34 -0800 + + * Initial version of a completely reworked intelligence framework. + See doc/intel.rst for more information. (Seth Hall) + + * Experimental Modbus analyzer. See policy/protocols/modbus/* for + example policies. (Dina Hadziosmanovic, Seth Hall) + +2.1-112 | 2012-11-05 13:58:20 -0800 + + * New base script for detecting cases of checksum offloading. + Reporter messages will now tell if one has bad checksums. (Seth + Hall) + + * Clarifying ownership rules for BroString constructors. (Robin + Sommer) + +2.1-109 | 2012-11-05 13:39:34 -0800 + + * Add detection rate threshold for MHR. (Vlad Grigorescu) + + * lookup_hostname_txt fixes. (Vlad Grigorescu) + +2.1-104 | 2012-11-01 10:37:50 -0700 + + * A new built-in function lookup_hostname_txt() provides support for + DNS TXT queries. (Vlad Grigorescu) + +2.1-101 | 2012-10-31 14:30:26 -0700 + + * Documentation reorg: The install info has been consolidated into a + single document (INSTALL), the upgrade info has been moved from + the FAQ to a section in the install doc, and the "upgrading from + 1.5 to 2.0" document has been updated (and renamed) to also + include 2.0 to 2.1 upgrade info. (Daniel Thayer) + +2.1-96 | 2012-10-31 14:23:50 -0700 + + * Renaming option defining the frequency of alarm summary mails to + 'Logging::default_alarm_mail_interval'. (Daniel Thayer) + +2.1-91 | 2012-10-24 16:04:47 -0700 + + * Adding PPPoE support to Bro. (Seth Hall) + +2.1-87 | 2012-10-24 15:40:06 -0700 + + * Adding missing &redef for some TCP options. Addresses #905, #906, + #907. (Carsten Langer) + +2.1-86 | 2012-10-24 15:37:11 -0700 + + * Add parsing rules for IPv4/IPv6 subnet literal constants. + Addresses #888. (Jon Siwek) + +2.1-84 | 2012-10-19 15:12:56 -0700 + + * Added a BiF strptime() to wrap the corresponding C function. (Seth + Hall) + +2.1-82 | 2012-10-19 15:05:40 -0700 + + * Add IPv6 support to signature header conditions. (Jon Siwek) + + - "src-ip" and "dst-ip" conditions can now use IPv6 addresses/subnets. + They must be written in colon-hexadecimal representation and enclosed + in square brackets (e.g. [fe80::1]). Addresses #774. + + - "icmp6" is now a valid protocol for use with "ip-proto" and "header" + conditions. This allows signatures to be written that can match + against ICMPv6 payloads. Addresses #880. + + - "ip6" is now a valid protocol for use with the "header" condition. + (also the "ip-proto" condition, but it results in a no-op in that + case since signatures apply only to the inner-most IP packet when + packets are tunneled). This allows signatures to match specifically + against IPv6 packets (whereas "ip" only matches against IPv4 packets). + + - "ip-proto" conditions can now match against IPv6 packets. Before, + IPv6 packets were just silently ignored which meant DPD based on + signatures did not function for IPv6 -- protocol analyzers would only + get attached to a connection over IPv6 based on the well-known ports + set in the "dpd_config" table. + +2.1-80 | 2012-10-19 14:48:42 -0700 + + * Change how "gridftp" gets added to service field of connection + records. In addition to checking for a finished SSL handshake over + an FTP connection, it now also requires that the SSL handshake + occurs after the FTP client requested AUTH GSSAPI, more + specifically identifying the characteristics of GridFTP control + channels. Addresses #891. (Jon Siwek) + + * Allow faster rebuilds in certain cases. Previously, when + rebuilding with a different "--prefix" or "--scriptdir", all Bro + source files were recompiled. With this change, only util.cc is + recompiled. (Daniel Thayer) + +2.1-76 | 2012-10-12 10:32:39 -0700 + + * Add support for recognizing GridFTP connections as an extension to + the standard FTP analyzer. (Jon Siwek) + + This is enabled by default and includes: + + - An analyzer for GSI mechanism of GSSAPI FTP AUTH method. GSI + authentication involves an encoded TLS/SSL handshake over the + FTP control session. For FTP sessions that attempt GSI + authentication, the *service* field of the connection log will + include "gridftp" (as well as also "ftp" and "ssl"). + + - Add an example of a GridFTP data channel detection script. It + relies on the heuristics of GridFTP data channels commonly + default to SSL mutual authentication with a NULL bulk cipher + and that they usually transfer large datasets (default + threshold of script is 1 GB). The script also defaults to + skip_further_processing() after detection to try to save + cycles analyzing the large, benign connection. + + For identified GridFTP data channels, the *services* fields of + the connection log will include "gridftp-data". + + * Add *client_subject* and *client_issuer_subject* as &log'd fields + to SSL::Info record. Also add *client_cert* and + *client_cert_chain* fields to track client cert chain. (Jon Siwek) + + * Add a script in base/protocols/conn/polling that generalizes the + process of polling a connection for interesting features. The + GridFTP data channel detection script depends on it to monitor + bytes transferred. (Jon Siwek) + +2.1-68 | 2012-10-12 09:46:41 -0700 + + * Rename the Input Framework's update_finished event to end_of_data. + It will now not only fire after table-reads have been completed, + but also after the last event of a whole-file-read (or + whole-db-read, etc.). (Bernhard Amann) + + * Fix for DNS log problem when a DNS response is seen with 0 RRs. + (Seth Hall) + +2.1-64 | 2012-10-12 09:36:41 -0700 + + * Teach --disable-dataseries/--disable-elasticsearch to ./configure. + Addresses #877. (Jon Siwek) + + * Add --with-curl option to ./configure. Addresses #877. (Jon Siwek) + +2.1-61 | 2012-10-12 09:32:48 -0700 + + * Fix bug in the input framework: the config table did not work. + (Bernhard Amann) + +2.1-58 | 2012-10-08 10:10:09 -0700 + + * Fix a problem with non-manager cluster nodes applying + Notice::policy. This could, for example, result in duplicate + emails being sent if Notice::emailed_types is redef'd in local.bro + (or any script that gets loaded on all cluster nodes). (Jon Siwek) + +2.1-56 | 2012-10-03 16:04:52 -0700 + + * Add general FAQ entry about upgrading Bro. (Jon Siwek) + +2.1-53 | 2012-10-03 16:00:40 -0700 + + * Add new Tunnel::delay_teredo_confirmation option that indicates + that the Teredo analyzer should wait until it sees both sides of a + connection using a valid Teredo encapsulation before issuing a + protocol_confirmation. Default is on. Addresses #890. (Jon Siwek) + +2.1-50 | 2012-10-02 12:06:08 -0700 + + * Fix a typing issue that prevented the ElasticSearch timeout to + work. (Matthias Vallentin) + + * Use second granularity for ElasticSearch timeouts. (Matthias + Vallentin) + + * Fix compile issues with older versions of libcurl, which don't + offer *_MS timeout constants. (Matthias Vallentin) + +2.1-47 | 2012-10-02 11:59:29 -0700 + + * Fix for the input framework: BroStrings were constructed without a + final \0, which makes them unusable by basically all internal + functions (like to_count). (Bernhard Amann) + + * Remove deprecated script functionality (see NEWS for details). + (Daniel Thayer) + +2.1-39 | 2012-09-29 14:09:16 -0700 + + * Reliability adjustments to istate tests with network + communication. (Jon Siwek) + +2.1-37 | 2012-09-25 14:21:37 -0700 + + * Reenable some tests that previously would cause Bro to exit with + an error. (Daniel Thayer) + + * Fix parsing of large integers on 32-bit systems. (Daniel Thayer) + + * Serialize language.when unit test with the "comm" group. (Jon + Siwek) + +2.1-32 | 2012-09-24 16:24:34 -0700 + + * Fix race condition in language/when.bro test. (Daniel Thayer) + +2.1-26 | 2012-09-23 08:46:03 -0700 + + * Add an item to FAQ page about broctl options. (Daniel Thayer) + + * Add more language tests. We now have tests of all built-in Bro + data types (including different representations of constant + values, and max./min. values), keywords, and operators (including + special properties of certain operators, such as short-circuit + evaluation and associativity). (Daniel Thayer) + + * Fix construction of ip6_ah (Authentication Header) record values. + + Authentication Headers with a Payload Len field set to zero would + cause a crash due to invalid memory allocation because the + previous code assumed Payload Len would always be great enough to + contain all mandatory fields of the header. (Jon Siwek) + + * Update compile/dependency docs for OS X. (Jon Siwek) + + * Adjusting Mac binary packaging script. Setting CMAKE_PREFIX_PATH + helps link against standard system libs instead of ones that come + from other package manager (e.g. MacPorts). (Jon Siwek) + + * Adjusting some unit tests that do cluster communication. (Jon Siwek) + + * Small change to non-blocking DNS initialization. (Jon Siwek) + + * Reorder a few statements in scan.l to make 1.5msecs etc work. + Adresses #872. (Bernhard Amann) + +2.1-6 | 2012-09-06 23:23:14 -0700 + + * Fixed a bug where "a -= b" (both operands are intervals) was not + allowed in Bro scripts (although "a = a - b" is allowed). (Daniel + Thayer) + + * Fixed a bug where the "!=" operator with subnet operands was + treated the same as the "==" operator. (Daniel Thayer) + + * Add sleeps to configuration_update test for better reliability. + (Jon Siwek) + + * Fix a segfault when iterating over a set when using malformed + index. (Daniel Thayer) + +2.1 | 2012-08-28 16:46:42 -0700 + + * Make bif.identify_magic robust against FreeBSD's libmagic config. + (Robin Sommer) + + * Remove automatic use of gperftools on non-Linux systems. + --enable-perftools must now explicity be supplied to ./configure + on non-Linux systems to link against the tcmalloc library. + + * Fix uninitialized value for 'is_partial' in TCP analyzer. (Jon + Siwek) + + * Parse 64-bit consts in Bro scripts correctly. (Bernhard Amann) + + * Output 64-bit counts correctly on 32-bit machines (Bernhard Amann) + + * Input framework fixes, including: (Bernhard Amann) + + - One of the change events got the wrong parameters. + + - Escape commas in sets and vectors that were unescaped before + tokenization. + + - Handling of zero-length-strings as last element in a set was + broken (sets ending with a ,). + + - Hashing of lines just containing zero-length-strings was broken. + + - Make set_separators different from , work for input framework. + + - Input framework was not handling counts and ints out of + 32-bit-range correctly. + + - Errors in single lines do not kill processing, but simply ignore + the line, log it, and continue. + + * Update documentation for builtin types. (Daniel Thayer) + + - Add missing description of interval "msec" unit. + + - Improved description of pattern by clarifying the issue of + operand order and difference between exact and embedded + matching. + + * Documentation fixes for signature 'eval' conditions. (Jon Siwek) + + * Remove orphaned 1.5 unit tests. (Jon Siwek) + + * Add type checking for signature 'eval' condition functions. (Jon + Siwek) + + * Adding an identifier to the SMTP blocklist notices for duplicate + suppression. (Seth Hall) + +2.1-beta-45 | 2012-08-22 16:11:10 -0700 + + * Add an option to the input framework that allows the user to chose + to not die upon encountering files/functions. (Bernhard Amann) + +2.1-beta-41 | 2012-08-22 16:05:21 -0700 + + * Add test serialization to "leak" unit tests that use + communication. (Jon Siwek) + + * Change to metrics/basic-cluster unit test for reliability. (Jon + Siwek) + + * Fixed ack tracking which could overflow quickly in some + situations. (Seth Hall) + + * Minor tweak to coverage.bare-mode-errors unit test to work with a + symlinked 'scripts' dir. (Jon Siwek) + +2.1-beta-35 | 2012-08-22 08:44:52 -0700 + + * Add testcase for input framework reading sets (rather than + tables). (Bernhard Amann) + +2.1-beta-31 | 2012-08-21 15:46:05 -0700 + + * Tweak to rotate-custom.bro unit test. (Jon Siwek) + + * Ignore small mem leak every rotation interval for dataseries logs. + (Jon Siwek) + +2.1-beta-28 | 2012-08-21 08:32:42 -0700 + + * Linking ES docs into logging document. (Robin Sommer) + +2.1-beta-27 | 2012-08-20 20:06:20 -0700 + + * Add the Stream record to Log:active_streams to make more dynamic + logging possible. (Seth Hall) + + * Fix portability of printing to files returned by + open("/dev/stderr"). (Jon Siwek) + + * Fix mime type diff canonifier to also skip mime_desc columns. (Jon + Siwek) + + * Unit test tweaks/fixes. (Jon Siwek) + + - Some baselines for tests in "leaks" group were outdated. + + - Changed a few of the cluster/communication tests to terminate + more explicitly instead of relying on btest-bg-wait to kill + processes. This makes the tests finish faster in the success case + and makes the reason for failing clearer in the that case. + + * Fix memory leak of serialized IDs when compiled with + --enable-debug. (Jon Siwek) + +2.1-beta-21 | 2012-08-16 11:48:56 -0700 + + * Installing a handler for running out of memory in "new". Bro will + now print an error message in that case rather than abort with an + uncaught exception. (Robin Sommer) + +2.1-beta-20 | 2012-08-16 11:43:31 -0700 + + * Fixed potential problems with ElasticSearch output plugin. (Seth + Hall) + +2.1-beta-13 | 2012-08-10 12:28:04 -0700 + + * Reporter warnings and error now print to stderr by default. New + options Reporter::warnings_to_stderr and + Reporter::errors_to_stderr to disable. (Seth Hall) + +2.1-beta-9 | 2012-08-10 12:24:29 -0700 + + * Add more BIF tests. (Daniel Thayer) + +2.1-beta-6 | 2012-08-10 12:22:52 -0700 + + * Fix bug in input framework with an edge case. (Bernhard Amann) + + * Fix small bug in input framework test script. (Bernhard Amann) + +2.1-beta-3 | 2012-08-03 10:46:49 -0700 + + * Merge branch 'master' of ssh://git.bro-ids.org/bro (Robin Sommer) + + * Fix configure script to exit with non-zero status on error (Jon + Siwek) + + * Improve ASCII output performance. (Robin Sommer) + +2.1-beta | 2012-07-30 11:59:53 -0700 + + * Improve log filter compatibility with remote logging. Addresses + #842. (Jon Siwek) + +2.0-907 | 2012-07-30 09:13:36 -0700 + + * Add missing breaks to switch cases in + ElasticSearch::HTTPReceive(). (Jon Siwek) + +2.0-905 | 2012-07-28 16:24:34 -0700 + + * Fix log manager hanging on waiting for pending file rotations, + plus writer API tweak for failed rotations. Addresses #860. (Jon + Siwek and Robin Sommer) + + * Tweaking logs-to-elasticsearch.bro so that it doesn't do anything + if ES server is unset. (Robin Sommer) + +2.0-902 | 2012-07-27 12:42:13 -0700 + + * New variable in logging framework Log::active_streams to indicate + Log:ID enums which are currently active. (Seth Hall) + + * Reworked how the logs-to-elasticsearch scripts works to stop + abusing the logging framework. (Seth Hall) + + * Fix input test for recent default change on fastpath. (Robin + Sommer) + +2.0-898 | 2012-07-27 12:22:03 -0700 + + * Small (potential performance) improvement for logging framework. (Seth Hall) + + * Script-level rotation postprocessor fix. This fixes a problem with + writers that don't have a postprocessor. (Seth Hall) + + * Update input framework documentation to reflect want_record + change. (Bernhard Amann) + + * Fix crash when encountering an InterpreterException in a predicate + in logging or input Framework. (Bernhard Amann) + + * Input framework: Make want_record=T the default for events + (Bernhard Amann) + + * Changing the start/end markers in logs to open/close now + reflecting wall clock. (Robin Sommer) + +2.0-891 | 2012-07-26 17:15:10 -0700 + + * Reader/writer API: preventing plugins from receiving further + messages after a failure. (Robin Sommer) + + * New test for input framework that fails to find a file. (Robin + Sommer) + + * Improving error handling for threads. (Robin Sommer) + + * Tweaking the custom-rotate test to produce stable output. (Robin + Sommer) + +2.0-884 | 2012-07-26 14:33:21 -0700 + + * Add comprehensive error handling for close() calls. (Jon Siwek) + + * Add more test cases for input framework. (Bernhard Amann) + + * Input framework: make error output for non-matching event types + much more verbose. (Bernhard Amann) + +2.0-877 | 2012-07-25 17:20:34 -0700 + + * Fix double close() in FilerSerializer class. (Jon Siwek) + + * Fix build warnings. (Daniel Thayer) + + * Fixes to ElasticSearch plugin to make libcurl handle http + responses correctly. (Seth Hall) + + * Fixing FreeBSD compiler error. (Robin Sommer) + + * Silencing compiler warnings. (Robin Sommer) + +2.0-871 | 2012-07-25 13:08:00 -0700 + + * Fix complaint from valgrind about uninitialized memory usage. (Jon + Siwek) + + * Fix differing log filters of streams from writing to same + writer/path (which now produces a warning, but is otherwise + skipped for the second). Addresses #842. (Jon Siwek) + + * Fix tests and error message for to_double BIF. (Daniel Thayer) + + * Compile fix. (Robin Sommer) + +2.0-866 | 2012-07-24 16:02:07 -0700 + + * Correct a typo in usage message. (Daniel Thayer) + + * Fix file permissions of log files (which were created with execute + permissions after a recent change). (Daniel Thayer) + +2.0-862 | 2012-07-24 15:22:52 -0700 + + * Fix initialization problem in logging class. (Jon Siwek) + + * Input framework now accepts escaped ASCII values as input (\x##), + and unescapes appropiately. (Bernhard Amann) + + * Make reading ASCII logfiles work when the input separator is + different from \t. (Bernhard Amann) + + * A number of smaller fixes for input framework. (Bernhard Amann) + +2.0-851 | 2012-07-24 15:04:14 -0700 + + * New built-in function to_double(s: string). (Scott Campbell) + +2.0-849 | 2012-07-24 11:06:16 -0700 + + * Adding missing include needed on some systems. (Robin Sommer) + +2.0-846 | 2012-07-23 16:36:37 -0700 + + * Fix WriterBackend::WriterInfo serialization, reenable ascii + start/end tags. (Jon Siwek) + +2.0-844 | 2012-07-23 16:20:59 -0700 + + * Reworking parts of the internal threading/logging/input APIs for + thread-safety. (Robin Sommer) + + * Bugfix for SSL version check. (Bernhard Amann) + + * Changing a HTTP DPD from port 3138 to 3128. Addresses #857. (Robin + Sommer) + + * ElasticSearch logging writer. See logging-elasticsearch.rst for + more information. (Vlad Grigorescu and Seth Hall). + + * Give configure a --disable-perftools option to disable Perftools + support even if found. (Robin Sommer) + + * The ASCII log writer now includes "#start " and "#end + lines in the each file. (Robin Sommer) + + * Renamed ASCII logger "header" options to "meta". (Robin Sommer) + + * ASCII logs now escape '#' at the beginning of log lines. Addresses + #763. (Robin Sommer) + + * Fix bug, where in dns.log rcode always was set to 0/NOERROR when + no reply package was seen. (Bernhard Amann) + + * Updating to Mozilla's current certificate bundle. (Seth Hall) + +2.0-769 | 2012-07-13 16:17:33 -0700 + + * Fix some Info:Record field documentation. (Vlad Grigorescu) + + * Fix overrides of TCP_ApplicationAnalyzer::EndpointEOF. (Jon Siwek) + + * Fix segfault when incrementing whole vector values. Also removed + RefExpr::Eval(Val*) method since it was never called. (Jon Siwek) + + * Remove baselines for some leak-detecting unit tests. (Jon Siwek) + + * Unblock SIGFPE, SIGILL, SIGSEGV and SIGBUS for threads, so that + they now propagate to the main thread. Adresses #848. (Bernhard + Amann) + +2.0-761 | 2012-07-12 08:14:38 -0700 + + * Some small fixes to further reduce SOCKS false positive logs. (Seth Hall) + + * Calls to pthread_mutex_unlock now log the reason for failures. + (Bernhard Amann) + +2.0-757 | 2012-07-11 08:30:19 -0700 + + * Fixing memory leak. (Seth Hall) + +2.0-755 | 2012-07-10 16:25:16 -0700 + + * Add sorting canonifier to rotate-custom unit test. Addresses #846. + (Jon Siwek) + + * Fix many compiler warnings. (Daniel Thayer) + + * Fix segfault when there's an error/timeout resolving DNS requests. + Addresses #846. (Jon Siwek) + + * Remove a non-portable test case. (Daniel Thayer) + + * Fix typos in input framework doc. (Daniel Thayer) + + * Fix typos in DataSeries documentation. (Daniel Thayer) + + * Bugfix making custom rotate functions work again. (Robin Sommer) + + * Tiny bugfix for returning writer name. (Robin Sommer) + + * Moving make target update-doc-sources from top-level Makefile to + btest Makefile. (Robin Sommer) + +2.0-733 | 2012-07-02 15:31:24 -0700 + + * Extending the input reader DoInit() API. (Bernhard Amann). It now + provides a Info struct similar to what we introduced for log + writers, including a corresponding "config" key/value table. + + * Fix to make writer-info work when debugging is enabled. (Bernhard + Amann) + +2.0-726 | 2012-07-02 15:19:15 -0700 + + * Extending the log writer DoInit() API. (Robin Sommer) + + We now pass in a Info struct that contains: + + - the path name (as before) + - the rotation interval + - the log_rotate_base_time in seconds + - a table of key/value pairs with further configuration options. + + To fill the table, log filters have a new field "config: table[string] + of strings". This gives a way to pass arbitrary values from + script-land to writers. Interpretation is left up to the writer. + + * Split calc_next_rotate() into two functions, one of which is + thread-safe and can be used with the log_rotate_base_time value + from DoInit(). + + * Updates to the None writer. (Robin Sommer) + + - It gets its own script writers/none.bro. + + - New bool option LogNone::debug to enable debug output. It then + prints out all the values passed to DoInit(). + + - Fixed a bug that prevented Bro from terminating. + +2.0-723 | 2012-07-02 15:02:56 -0700 + + * Extract ICMPv6 NDP options and include in ICMP events. This adds + a new parameter of type "icmp6_nd_options" to the ICMPv6 neighbor + discovery events. Addresses #833. (Jon Siwek) + + * Set input frontend type before starting the thread. This means + that the thread type will be output correctly in the error + message. (Bernhard Amann) + +2.0-719 | 2012-07-02 14:49:03 -0700 + + * Fix inconsistencies in random number generation. The + srand()/rand() interface was being intermixed with the + srandom()/random() one. The later is now used throughout. (Jon + Siwek) + + * Changed the srand() and rand() BIFs to work deterministically if + Bro was given a seed file. Addresses #825. (Jon Siwek) + + * Updating input framework unit tests to make them more reliable and + execute quicker. (Jon Siwek) + + * Fixed race condition in writer and reader initializations. (Jon + Siwek) + + * Small tweak to make test complete quicker. (Jon Siwek) + + * Drain events before terminating log/thread managers. (Jon Siwek) + + * Fix strict-aliasing warning in RemoteSerializer.cc. Addresses + #834. (Jon Siwek) + + * Fix typos in event documentation. (Daniel Thayer) + + * Fix typos in NEWS for Bro 2.1 beta. (Daniel Thayer) + +2.0-709 | 2012-06-21 10:14:24 -0700 + + * Fix exceptions thrown in event handlers preventing others from running. (Jon Siwek) + + * Add another SOCKS command. (Seth Hall) + + * Fixed some problems with the SOCKS analyzer and tests. (Seth Hall) + + * Updating NEWS in preparation for beta. (Robin Sommer) + + * Accepting different AF_INET6 values for loopback link headers. + (Robin Sommer) + +2.0-698 | 2012-06-20 14:30:40 -0700 + + * Updates for the SOCKS analyzer (Seth Hall). + + - A SOCKS log! + + - Now supports SOCKSv5 in the analyzer and the DPD sigs. + + - Added protocol violations. + + * Updates to the tunnels framework. (Seth Hall) + + - Make the uid field optional since it's conceptually incorrect + for proxies being treated as tunnels to have it. + + - Reordered two fields in the log. + + - Reduced the default tunnel expiration interface to something + more reasonable (1 hour). + + * Make Teredo bubble packet parsing more lenient. (Jon Siwek) + + * Fix a crash in NetSessions::ParseIPPacket(). (Jon Siwek) + +2.0-690 | 2012-06-18 16:01:33 -0700 + + * Support for decapsulating tunnels via the new tunnel framework in + base/frameworks/tunnels. + + Bro currently supports Teredo, AYIYA, IP-in-IP (both IPv4 and + IPv6), and SOCKS. For all these, it logs the outher tunnel + connections in both conn.log and tunnel.log, and proceeds to + analyze the inner payload as if it were not tunneled, including + also logging it in conn.log (with a new tunnel_parents column + pointing back to the outer connection(s)). (Jon Siwek, Seth Hall, + Gregor Maier) + + * The options "tunnel_port" and "parse_udp_tunnels" have been + removed. (Jon Siwek) + +2.0-623 | 2012-06-15 16:24:52 -0700 + + * Changing an error in the input framework to a warning. (Robin + Sommer) + +2.0-622 | 2012-06-15 15:38:43 -0700 + + * Input framework updates. (Bernhard Amann) + + - Disable streaming reads from executed commands. This lead to + hanging Bros because pclose apparently can wait for eternity if + things go wrong. + + - Automatically delete disabled input streams. + + - Documentation. + +2.0-614 | 2012-06-15 15:19:49 -0700 + + * Remove an old, unused diff canonifier. (Jon Siwek) + + * Improve an error message in ICMP analyzer. (Jon Siwek) + + * Fix a warning message when building docs. (Daniel Thayer) + + * Fix many errors in the event documentation. (Daniel Thayer) + +2.0-608 | 2012-06-11 15:59:00 -0700 + + * Add more error handling code to logging of enum vals. Addresses + #829. (Jon Siwek) + +2.0-606 | 2012-06-11 15:55:56 -0700 + + * Fix summary lines for BIF documentation and corrected the + description of "fmt" and "floor" BIFs. (Daniel Thayer) + + * Fix val_size BIF tests and improve docs. (Daniel Thayer) + +2.0-602 | 2012-06-07 15:06:19 -0700 + + * Include header for usleep(), caused compile failure on Archlinux. (Jon Siwek) + + * Revert "Fixed a bug with the MIME analyzer not removing whitespace + on wrapped headers." Needs discussion. (Robin Sommer) + +2.0-598 | 2012-06-06 11:47:00 -0700 + + * Add @load-sigs directive for loading signature files (addresses + #551). This can be used to load signatures relative to the current + scripts (e.g., "@load-sigs ./foo.sig"). (Jon Siwek) + + +2.0-596 | 2012-06-06 11:41:00 -0700 + + * Fixes for some BiFs and their documentation. (Daniel Thayer) + + * Many new unit tests for BiFs. (Daniel Thayer) + +2.0-579 | 2012-06-06 11:04:46 -0700 + + * Memory leak fixes for bad usages of VectorVal ctor. (Jon Siwek) + + * Fixed a bug with the MIME analyzer not removing whitespace on + wrapped headers. (Seth Hall) + + * Change Input::update_finished lookup to happen at init time. (Jon Siwek) + + * Fix going through the internal_handler() function which will now + set the event as "used" (i.e. it's marked as being raised + somewhere). Addresses #823. (Jon Siwek) + + * Fix format specifier on RemoteSerializer::Connect. This caused + 32-bit systems to show a warning at compile-time, and fail when + connecting to peers. (Jon Siwek) + + * Fixes for running tests in parallel. (Robin Sommer) + +2.0-571 | 2012-05-30 19:12:43 -0700 + + * Updating submodule(s). + +2.0-570 | 2012-05-30 19:08:18 -0700 + + * A new input framework enables scripts to read in external data + dynamically on the fly as Bro is processing network traffic. + (Bernhard Amann) + + Currently, the framework supports reading ASCII input that's + structured similar as Bro's log files as well as raw blobs of + data. Other formats will come in the future. + + See doc/input.rst for more information (this will be extended + further soon). + +2.0-395 | 2012-05-30 17:03:31 -0700 + + * Remove unnecessary assert in ICMP analyzer which could lead to + aborts. Addresses #822. + + * Improve script debugger backtrace and print commands. (Jon Siwek) + + * Switching default DS compression to gzip. (Robin Sommer) + + * Improve availability of IPv6 flow label in connection records. + This adds a "flow_label" field to the "endpoint" record type, + which is used for both the "orig" and "resp" fields of + "connection" records. The new "connection_flow_label_changed" + event also allows tracking of changes in flow labels: it's raised + each time one direction of the connection starts using a different + label. (Jon Siwek) + + * Add unit tests for Broccoli SSL and Broccoli IPv6 connectivity. + (Jon Siwek) + + * Remove AI_ADDRCONFIG getaddrinfo hints flag for listening sockets. + (Jon Siwek) + + * Undo unnecessary communication protocol version bump. (Jon Siwek) + + * Add support to Bro for connecting with peers over IPv6. (Jon Siwek) + + - Communication::listen_ipv6 needs to be redef'd to true in order + for IPv6 listening sockets to be opened. + + - Added Communication::listen_retry option as an interval at which + to retry binding to socket addresses that were already in use. + + - Added some explicit baselines to check in the istate.events and + istate.events-ssl tests -- the SSL test was incorrectly passing + because it compared two empty files. (The files being empty + because "http/base" was given as an argument to Bro which it + couldn't handle because that script doesn't exist anymore). + + - Support for communication over non-global IPv6 addresses. This + usually requires specifying an additional zone identifier (see + RFC 4007). The connect() and listen() BIFs have been changed to + accept this zone identifier as an argument. + + +2.0-377 | 2012-05-24 16:46:06 -0700 + + * Documentation fixes. (Jon Siwek and Daniel Thayer) + +2.0-372 | 2012-05-17 13:59:45 -0700 + + * Fix compile errors. (Jon Siwek) + + * Linking in the DS docs. (Robin Sommer) + + * Fix mobility checksums unit test. (Jon Siwek) + +2.0-367 | 2012-05-17 12:42:30 -0700 + + * Adding support for binary output via DataSeries. See + logging-dataseries.rst for more information. (Gilbert Clark and + Robin Sommer) + + * Adding target update-doc-sources to top-level Makefile that runs + genDocSourcesList.sh. (Robin Sommer) + + * Moving trace for rotation test into traces directory. (Robin Sommer) + + * Fixing a rotation race condition at termination. (Robin Sommer) + + * Extending log post-processor call to include the name of the + writer. (Robin Sommer) + + * In threads, an internal error now immediately aborts. Otherwise, + the error won't make it back to the main thread for a while and + subsequent code in the thread would still execute. (Robin Sommer) + + * DataSeries cleanup. (Robin Sommer) + + * Fixing threads' DoFinish() method. It wasn't called reliably. Now, + it's always called before the thread is destroyed (assuming + processing has went normally so far). (Robin Sommer) + +2.0-341 | 2012-05-17 09:54:30 -0700 + + * Add a comment to explain the ICMPv6 error message types. (Daniel Thayer) + + * Quieting external test output somehwat. (Robin Sommer) + +2.0-336 | 2012-05-14 17:15:44 -0700 + + * Don't print the various "weird" events to stderr. Address #805. + (Daniel Thayer) + + * Generate icmp_error_message event for ICMPv6 error msgs. + Previously, icmp_sent was being generated, but icmp_error_message + contains more info. + + * Improved documentation comments for icmp-related events. (Daniel + Thayer) + +2.0-330 | 2012-05-14 17:05:56 -0700 + + * Add `addr_to_uri` script-level function that adds brackets to an + address if it's IPv6 and will be included in a URI or when a + ":" needs to be appended to it. (Jon Siwek) + + * Also add a test case for content extraction. (Jon Siwek) + + * Fix typos and improve INSTALL document. (Daniel Thayer) + + * Switching to new btest command TEST-SERIALIZE for communication + tests. (Robin Sommer) + +2.0-323 | 2012-05-04 21:04:34 -0700 + + * Add SHA1 and SHA256 hashing BIFs. Addresses #542. + + * Refactor all internal MD5 stuff to use OpenSSL's. (Jon Siwek) + + * Changes to open-file caching limits and uncached file unserialization. (Jon Siwek) + + - Unserializing files that were previously kicked out of the open-file + cache would cause them to be fopen'd with the original access + permissions which is usually 'w' and causes truncation. They + are now opened in 'a' mode. (addresses #780) + + - Add 'max_files_in_cache' script option to manually set the maximum + amount of opened files to keep cached. Mainly this just helped + to create a simple test case for the above change. + + - Remove unused NO_HAVE_SETRLIMIT preprocessor switch. + + - On systems that don't enforce a limit on number of files opened for + the process, raise default max size of open-file cache from + 32 to 512. + +2.0-319 | 2012-05-03 13:24:44 -0700 + + * SSL bugfixes and cleanup. (Seth Hall) + + - SSL related files and classes renamed to remove the "binpac" term. + + - A small fix for DPD scripts to make the DPD log more helpful if + there are multiple continued failures. + + - Fixed the SSL analyzer to make it stop doing repeated violation + messages for some handshake failures. + + - Added a $issuer_subject to the SSL log. + + - Created a basic test for SSL. + + - Fixed parsing of TLS server extensions. (Seth Hall) + +2.0-315 | 2012-05-03 11:44:17 -0700 + + * Add two more TLS extension values that we see in live traffic. + (Bernhard Amann) + + * Fixed IPv6 link local unicast CIDR and added IPv6 loopback to + private address space. (Seth Hall) + + * Fixed a problem where cluster workers were still processing + notices in some cases. (Seth Hall) + + * Added a configure option to specify the 'etc' directory. Addresses + #801. (Daniel Thayer) + + +2.0-306 | 2012-04-24 14:37:00 -0700 + + * Add further TLS extension values "extended_random" and + "heartbeat". (Seth Hall) + + * Fix problem with extracting FTP passwords and add "ftpuser" as + another anonymous username. (Seth Hall, discovered by Patrik + Lundin). + +2.0-303 | 2012-04-19 10:01:06 -0700 + + * Changes related to ICMPv6 Neighbor Discovery messages. (Jon Siwek) + + - The 'icmp_conn' record now contains an 'hlim' field since hop limit + in the IP header is an interesting field for at least these ND + messages. + + - Fixed and extended 'icmp_router_advertisement' event parameters. + + - Changed 'icmp_neighbor_advertisement' event parameters to add + more of the known boolean flags. + +2.0-301 | 2012-04-17 17:58:55 -0700 + + * Bro now support ICMPv6. (Matti Mantere, Jon Siwek, Robin Sommer, + Daniel Thayer). + + Overall, Bro now raises the following ICMP events for v4 and v6 as + appropiate: + + event icmp_sent(c: connection, icmp: icmp_conn); + event icmp_echo_request(c: connection, icmp: icmp_conn, id: count, seq: count, payload: string); + event icmp_echo_reply(c: connection, icmp: icmp_conn, id: count, seq: count, payload: string); + event icmp_error_message(c: connection, icmp: icmp_conn, code: count, context: icmp_context); + event icmp_unreachable(c: connection, icmp: icmp_conn, code: count, context: icmp_context); + event icmp_packet_too_big(c: connection, icmp: icmp_conn, code: count, context: icmp_context); + event icmp_time_exceeded(c: connection, icmp: icmp_conn, code: count, context: icmp_context); + event icmp_parameter_problem(c: connection, icmp: icmp_conn, code: count, context: icmp_context); + event icmp_router_solicitation(c: connection, icmp: icmp_conn); + event icmp_router_advertisement(c: connection, icmp: icmp_conn, hop_limit: count, managed: bool, router_lifetime: count, reachable_time: interval, retrans_timer: interval); + event icmp_neighbor_solicitation(c: connection, icmp: icmp_conn, tgt:addr); + event icmp_neighbor_advertisement(c: connection, icmp: icmp_conn, tgt:addr); + event icmp_redirect(c: connection, icmp: icmp_conn, tgt: addr, dest: addr); + + The `icmp_conn` record got a new boolean field 'v6' that indicates + whether the ICMP message is v4 or v6. + + This change also includes further low-level work on existing IP + and ICMP code, including a reorganization of how ICMPv4 is + handled. + +2.0-281 | 2012-04-17 17:40:39 -0700 + + * Small updates for the bittorrent analyzer to support 64bit types + in binpac. (Seth Hall) + + * Removed the attempt at bittorrent resynchronization. (Seth Hall) + +2.0-276 | 2012-04-17 17:35:56 -0700 + + * Add more support for 's that lack some structure + definitions. (Jon Siwek) + +2.0-273 | 2012-04-16 18:08:56 -0700 + + * Removing QR flag from DNS log in response, which should not have + been there in the first place. (Seth Hall) + + * Sync up patricia.c/h with pysubnettree repo. (Daniel Thayer) + + * Adding missing leak groups to a couple tests. Also activating leak + checking for proxy in basic-cluster test. (Robin Sommer) + +2.0-267 | 2012-04-09 17:47:28 -0700 + + * Add support for mobile IPv6 Mobility Header (RFC 6275). (Jon + Siwek) + + - Enabled through a new --enable-mobile-ipv6 configure-time + option. If not enabled, the mobility header (routing type 2) and + Home Address Destination option are ignored. + + - Accessible at script-layer through 'mobile_ipv6_message' event. + + * Refactor IP_Hdr routing header handling, add MobileIPv6 Home + Address handling. Packets that use the Home Address Destination + option use that option's address as the connection's originator. + (Jon Siwek) + + * Revert TCP checksumming to cache common data, like it did before. + (Jon Siwek) + + * Improve handling of IPv6 routing type 0 extension headers. (Jon + Siwek) + + - flow_weird event with name argument value of "routing0_hdr" is raised + for packets containing an IPv6 routing type 0 header because this + type of header is now deprecated according to RFC 5095. + + - Packets with a routing type 0 header and non-zero segments left + now use the last address in that header in order to associate + with a connection/flow and for calculating TCP/UDP checksums. + + - Added a set of IPv4/IPv6 TCP/UDP checksum unit tests (Jon Siwek) + + * Fix table expiry for values assigned in bro_init() when reading + live. (Jon Siwek) + +2.0-257 | 2012-04-05 15:32:43 -0700 + + * Fix CMake from warning about unused ENABLE_PERFTOOLS_DEBUG + variable. (Jon Siwek) + + * Fix handling of IPv6 atomic fragments. (Jon Siwek) + + * Fix that prevents Bro processes that do neither local logging nor + request remote logs from spawning threads. (Robin Sommer) + + * Fixing perftools-debug support. (Robin Sommer) + + * Reverting SocketComm change tuning I/O behaviour. (Robin Sommer) + + * Adding notice_policy.log canonification for external tests. (Robin Sommer) + + +2.0-245 | 2012-04-04 17:25:20 -0700 + + * Internal restructuring of the logging framework: we now spawn + threads doing the I/O. From a user's perspective not much should + change, except that the OS may now show a bunch of Bro threads. + (Gilbert Clark and Robin Sommer). + + * When building Bro, we now always link in tcmalloc if it's found at + configure time. If it's installed but not picked up, + --with-perftools may help. (Robin Sommer) + + * Renaming the configure option --enable-perftools to + --enable-perftool-debug to indicate that the switch is only + relevant for debugging the heap. It's not needed to pick up + tcmalloc for better performance. (Robin Sommer) + +2.0-184 | 2012-03-28 15:11:11 -0700 + + * Improve handling of IPv6 Routing Type 0 headers. (Jon Siwek) + + - For RH0 headers with non-zero segments left, a + "routing0_segleft" flow_weird event is raised (with a + destination indicating the last address in the routing header), + and an "rh0_segleft" event can also be handled if the other + contents of the packet header are of interest. No further + analysis is done as the complexity required to correctly + identify destination endpoints of connections doesn't seem worth + it as RH0 has been deprecated by RFC 5095. + + - For RH0 headers without any segments left, a "routing0_header" + flow_weird event is raised, but further analysis still occurs as + normal. + +2.0-182 | 2012-03-28 15:01:57 -0700 + + * Remove dead tcp_checksum function from net_util. (Jon Siwek) + + * Change routing0_data_to_addrs BIF to return vector of addresses. + The order of addresses in type 0 routing headers is + interesting/important. (Jon Siwek) + + +2.0-179 | 2012-03-23 17:43:31 -0700 + + * Remove the default "tcp or udp or icmp" filter. In default mode, + Bro would load the packet filter script framework which installs a + filter that allows all packets, but in bare mode (the -b option), + this old filter would not follow IPv6 protocol chains and thus + filter out packets with extension headers. (Jon Siwek) + + * Update PacketFilter/Discarder code for IP version independence. + (Jon Siwek) + + * Fix some IPv6 header related bugs. (Jon Siwek) + + * Add IPv6 fragment reassembly. (Jon Siwek) + + * Add handling for IPv6 extension header chains. Addresses #531. + (Jon Siwek) + + - The script-layer 'pkt_hdr' type is extended with a new 'ip6' field + representing the full IPv6 header chain. + + - The 'new_packet' event is now raised for IPv6 packets. Addresses + #523. + + - A new event called 'ipv6_ext_header' is raised for any IPv6 + packet containing extension headers. + + - A new event called 'esp_packet' is raised for any packets using + ESP ('new_packet' and 'ipv6_ext_header' events provide + connection info, but that info can't be provided here since the + upper-layer payload is encrypted). + + - The 'unknown_protocol' weird is now raised more reliably when + Bro sees a transport protocol or IPv6 extension header it can't + handle. Addresses #522. + + * Add unit tests for IPv6 fragment reassembly, ipv6_ext_headers and + esp_packet events. (Jon Siwek) + + * Adapt FreeBSD's inet_ntop implementation for internal use. Now we + get consistent text representations of IPv6 addresses across + platforms. (Jon Siwek) + + * Update documentation for new syntax of IPv6 literals. (Jon Siwek) + + +2.0-150 | 2012-03-13 16:16:22 -0700 + + * Changing the regular expression to allow Site::local_nets in + signatures. (Julien Sentier) + + * Removing a line of dead code. Found by . Closes #786. (Julien + Sentier) + +2.0-146 | 2012-03-13 15:39:38 -0700 + + * Change IPv6 literal constant syntax to require encasing square + brackets. (Jon Siwek) + +2.0-145 | 2012-03-09 15:10:35 -0800 + + * Remove the match expression. 'match' and 'using' are no longer + keywords. Addressed #753. (Jon Siwek) + +2.0-143 | 2012-03-09 15:07:42 -0800 + + * Fix a BRO_PROFILER_FILE/mkstemp portability issue. Addresses #794. + (Jon Siwek) + +2.0-139 | 2012-03-02 09:33:04 -0800 + + * Changes to how script coverage integrates with test suites. (Jon Siwek) + + - BRO_PROFILER_FILE now passes .X* templated filenames to mkstemp + for generating unique coverage state files. + + - Rearranging Makefile targets. The general rule is that if the + all/brief target fails out due to a test failure, then the dependent + coverage target won't run, but can still be invoked directly later. + (e.g. make brief || make coverage) + + * Standardized on the &default function for SSL constants. (Seth + Hall) + + * Adding btest group "leaks" to leak tests. (Robin Sommer) + + * Adding btest group "comm" to communication tests for parallelizing + execution with new btest version. (Robin Sommer) + + * Sorting all output for diffing in the external tests. (Robin + Sommer) + + * Cleaned up dead code from the old SSL analyzers. Reported by + Julien Sentier. (Seth Hall) + + * Update/add tests for broccoli IPv6 addr/subnet support. Addresses + #448. (Jon Siwek) + + * Remove connection compressor. Addresses #559. (Jon Siwek) + + * Refactor IP_Hdr class ctors. Addresses #532. (Jon Siwek) + + +2.0-121 | 2012-02-24 16:34:17 -0800 + + * A number of smaller memory fixes and code cleanups. (Julien + Sentier) + + * Add to_subnet bif. Fixes #782). (Jon Siwek) + + * Fix IPAddr::Mask/ReverseMask not allowing argument of 0. (Jon + Siwek) + + * Refactor IPAddr v4 initialization from string. Fixes #775. (Jon Siwek) + + * Parse the dotted address string directly instead of canonicalizing + and passing to inet_pton. (Jon Siwek) + + +2.0-108 | 2012-02-24 15:21:07 -0800 + + * Refactoring a number of usages of new IPAddr class. (Jon Siwek) + + * Fixed a bug in remask_addr bif. (Jon Siwek) + +2.0-106 | 2012-02-24 15:02:20 -0800 + + * Raise minimum required CMake version to 2.6.3. (Jon Siwek) + +2.0-104 | 2012-02-24 14:59:12 -0800 + + * Add test case for FTP over IPv4. (Daniel Thayer) + + * Fix IPv6 URLs in ftp.log. (Daniel Thayer) + + * Add a test for FTP over IPv6 (Daniel Thayer) + + * Fix parsing of FTP EPRT command and EPSV response. (Daniel Thayer) + +2.0-95 | 2012-02-22 05:27:34 -0800 + + * GeoIP installation documentation update. (Seth Hall) + + * Decrease strictness of parsing IPv4 strings into addrs. Fixes #775. (Jon Siwek) + + * Fix memory leak in DNS manager. Fixes #777. (Jon Siwek) + + * Fix IPAddr/IPPrefix serialization bugs. (Jon Siwek) + + * Fix compile error. (Jon Siwek) + +2.0-86 | 2012-02-17 15:41:06 -0800 + + * Changing ARP detection to always kick in even if no analyzer is + activated. (Robin Sommer) + + * DNS name lookups performed by Bro now also query AAAA records. + DNS_Mgr handles combining the results of the A and AAAA queries + for a given hostname such that at the scripting layer, the name + resolution can yield a set with both IPv4 and IPv6 addresses. (Jon + Siwek) + + * Add counts_to_addr and addr_to_counts conversion BIFs. (Jon Siwek) + + * Change HashKey threshold for using H3 to 36 bytes. (Jon Siwek) + + * Remove mention of --enable-brov6 in docs. (Daniel Thayer) + + * Remove --enable-brov6 from configure usage text (Daniel Thayer) + + * Add a test and baseline for addr_to_ptr_name BiF. (Daniel Thayer) + + * Adding a test and baseline for ptr_name_to_addr BiF. (Seth Hall) + + * Fix the ptr_name_to_addr BiF to work with IPv6 (Daniel Thayer) + + * Fix a memory leak that perftools now complains about. (Jon Siwek) + + * Remove --enable-brov6 flag, IPv6 now supported by default. (Jon Siwek) + + Some script-layer changes of note: + + - dns_AAAA_reply event signature changed: the string representation + of an IPv6 addr is easily derived from the addr value, it doesn't + need to be another parameter. This event also now generated directly + by the DNS analyzer instead of being "faked" into a dns_A_reply event. + + - Removed addr_to_count BIF. It used to return the host-order + count representation of IPv4 addresses only. To make it more + generic, we might later add a BIF to return a vector of counts + in order to support IPv6. + + - Changed the result of enclosing addr variables in vertical pipes + (e.g. |my_addr|) to return the bit-width of the address type which + is 128 for IPv6 and 32 for IPv4. It used to function the same + way as addr_to_count mentioned above. + + - Remove bro_has_ipv6 BIF + +2.0-57 | 2012-02-10 00:02:35 -0800 + + * Fix typos in the documentation. (Daniel Thayer) + + * Fix compiler warning about Brofiler ctor init list order. (Jon Siwek) + + * Fix missing optional field access in webapp signature_match handler. (Jon Siwek) + +2.0-41 | 2012-02-03 04:10:53 -0500 + + * Updates to the Software framework to simplify the API. (Bernhard + Amann) + +2.0-40 | 2012-02-03 01:55:27 -0800 + + * Fix typos in documentation. (Daniel Thayer) + + * Fix sorting of lines in Brofiler coverage.log. (Daniel Thayer) + +2.0-38 | 2012-01-31 11:50:53 -0800 + + * Canonify sorting of lines in Brofiler coverage.log. (Daniel + Thayer) + +2.0-36 | 2012-01-27 10:38:14 -0800 + + * New "Brofiler" mode that tracks and records script statements + executed during runtime. (Jon Siwek) + + Use the BROFILER_FILE environment variable to point to a file in + which statement usage statistics from Bro script-layer can be + output. + + Script statements that should be ignored can be marked with a "# + @no-test" comment. For example: + + print "don't cover"; # @no-test + + if ( F ) + { # @no-test + ... + } + + * Integrated coverage measurement into test-suite. (Jon Siwek) + +2.0-20 | 2012-01-25 16:34:51 -0800 + + * BiF cleanup (Matthias Vallentin) + + - Rename NFS3::mode2string to a more generic file_mode(). + + - Unify do_profiling()/make_connection_persistent()/expect_connection() + to return any (i.e., nothing) instead of bools. + + - Perform type checking on count-to-port conversion. Related to #684. + + - Remove redundant connection_record() BiF. The same + functionality is provided by lookup_connection(). + + - Remove redundant active_connection() BiF. The same + functionality is provided by connection_exists(). + + - exit() now takes the exit code as argument. + + - to_port() now received a string instead of a count. + +2.0-9 | 2012-01-25 13:47:13 -0800 + + * Allow local table variables to be initialized with {} list + expressions. (Jon Siwek) + +2.0-7 | 2012-01-25 13:38:09 -0800 + + * Teach CompHash to allow indexing by records with vector/table/set + fields. Addresses #464. (Jon Siwek) + +2.0-5 | 2012-01-25 13:25:19 -0800 + + * Fixed a bug resulting in over-logging of detected webapps. (Seth Hall) + + * Make communication log baseline test more reliable. (Jon Siwek) + + * Fixed some broken links in documentation. (Daniel Thayer) + +2.0 | 2012-01-11 13:52:22 -0800 + + * Adding script reference documentation. (The Team). + +2.0-beta-194 | 2012-01-10 10:44:32 -0800 + + * Added an option for filtering out URLs before they are turned into + HTTP::Incorrect_File_Type notices. (Seth Hall) + + * Fix ref counting bug in BIFs that call internal_type. Addresses + #740. (Jon Siwek) + + * Adding back the stats.bro file. (Seth Hall) + + +2.0-beta-188 | 2012-01-10 09:49:29 -0800 + + * Change SFTP/SCP log rotators to use 4-digit year in filenames + Fixes #745. (Jon Siwek) + + * Adding back the stats.bro file. Addresses #656. (Seth Hall) + +2.0-beta-185 | 2012-01-09 18:00:50 -0800 + + * Tweaks for OpenBSD support. (Jon Siwek) + +2.0-beta-181 | 2012-01-08 20:49:04 -0800 + + * Add SFTP log postprocessor that transfers logs to remote hosts. + Addresses #737. (Jon Siwek) + + * Add FAQ entry about disabling NIC offloading features. (Jon Siwek) + + * Add a file NEWS with release notes. (Robin Sommer) + 2.0-beta-177 | 2012-01-05 15:01:07 -0800 * Replace the --snaplen/-l command line option with a @@ -11,22 +2519,22 @@ 2.0-beta-174 | 2012-01-04 12:47:10 -0800 * SSL improvements. (Seth Hall) - + - Added the ssl_session_ticket_handshake event back. - Fixed a few bugs. - + - Removed the SSLv2.cc file since it's not used. - + 2.0-beta-169 | 2012-01-04 12:44:39 -0800 * Tuning the pretty-printed alarm mails, which now include the covered time range into the subject. (Robin Sommer) * Adding top-level "test" target to Makefile. (Robin Sommer) - + * Adding SWIG as dependency to INSTALL. (Robin Sommer) - + 2.0-beta-155 | 2012-01-03 15:42:32 -0800 * Remove dead code related to record type inheritance. (Jon Siwek) @@ -40,7 +2548,7 @@ * CMake 2.6 top-level 'install' target compat. Fixes #729. (Jon Siwek) * Minor fixes to test process. Addresses #298. - + * Increase timeout interval of communication-related btests. (Jon Siwek) 2.0-beta-145 | 2011-12-19 11:37:15 -0800 @@ -69,16 +2577,16 @@ - Fixed some bugs with capturing data in the base DNS script. - Answers and TTLs are now vectors. - + - A warning that was being generated (dns_reply_seen_after_done) from transaction ID reuse is fixed. * SSL updates. (Seth Hall) - Added is_orig fields to the SSL events and adapted script. - + - Added a field named last_alert to the SSL log. - + - The x509_certificate function has an is_orig field now instead of is_server and its position in the argument list has moved. @@ -93,7 +2601,7 @@ parameter. (Jon Siwek) * Allow Broxygen markup "##<" for more general use. (Jon Siwek) - + 2.0-beta-116 | 2011-12-16 02:38:27 -0800 * Cleanup some misc Broxygen css/js stuff. (Jon Siwek) @@ -113,13 +2621,13 @@ HTTP::SQL_Injection_Victim. (Seth Hall). * Fixed DPD signatures for IRC. Fixes #311. (Seth Hall) - + * Removing Off_Port_Protocol_Found notice. (Seth Hall) * Teach Broxygen to more generally reference attribute values by name. (Jon Siwek) * SSH::Interesting_Hostname_Login cleanup. Fixes #664. (Seth Hall) - + * Fixed bug that was causing the malware hash registry script to break. (Seth Hall) @@ -158,7 +2666,7 @@ Addresses #704. (Jon Siwek) * Fix double-free of DNS_Mgr_Request object. Addresses #661. - + * Add a remote_log_peer event which comes with an event_peer record parameter. Addresses #493. (Jon Siwek) @@ -170,9 +2678,9 @@ Fixes #705. (Jon Siwek) * Turn some globals into constants. Addresses #633. (Seth Hall) - + * Rearrange packet filter and DPD documentation. (Jon Siwek) - + 2.0-beta-72 | 2011-11-30 20:16:09 -0800 * Fine-tuning the Sphinx layout to better match www. (Jon Siwek and @@ -205,14 +2713,14 @@ Amann) * Promote libz and libmagic to required dependencies. (Jon Siwek) - + * Fix parallel make from top-level to work on more platforms. (Jon Siwek) * Add decode_base64_custom(). Addresses #670 (Jon Siwek) - + * A bunch of Sphinx-doc reorgs and polishing. (Jon Siwek) - + 2.0-beta-28 | 2011-11-14 20:09:28 -0800 * Binary packaging script tweaks. We now require CMake 2.8.6. (Jon Siwek) @@ -221,7 +2729,10008 @@ Hall) * Tiny bugfix for http file extraction along with test. (Seth Hall) + +2.0-beta-21 | 2011-11-06 19:27:22 -0800 + + * Quickstart doc fixes. (Jon Siwek) + +2.0-beta-19 | 2011-11-03 17:41:00 -0700 + + * Fixing packet filter test. (Robin Sommer) + +2.0-beta-12 | 2011-11-03 15:21:08 -0700 + + * No longer write to the PacketFilter::LOG stream if not reading + traffic. (Seth Hall) + +2.0-beta-10 | 2011-11-03 15:17:08 -0700 + + * Notice framework documentation update. (Seth Hall) + + * Fixing compiler warnings (addresses #388) (Jon Siwek) + +2.0-beta | 2011-10-27 17:46:28 -0700 + + * Preliminary fix for SSH login detection: we need a counted measure + of payload bytes (not ack tracking and not with the IP header + which is what we have now). (Seth Hall) + + * Fixing send_id() problem. We no longer update &redef functions. + Updating code on the fly isn't fully supported. (Robin Sommer) + + * Tuning the format of the pretty-printed alarm summaries. (Robin + Sommer) + +1.6-dev-1508 | 2011-10-26 17:24:50 -0700 + + * Updating submodule(s). (Robin Sommer) + +1.6-dev-1507 | 2011-10-26 15:10:18 -0700 + + * Baseline updates. (Robin Sommer) + +1.6-dev-1506 | 2011-10-26 14:48:43 -0700 + + * Updating submodule(s). (Robin Sommer) + +1.6-dev-1505 | 2011-10-26 14:43:58 -0700 + + * A new base script that pretty-prints alarms in the regular + summary. (Robin Sommer) + + * Adding a dummy log writer WRITER_NONE that just discards + everything. (Robin Sommer) + +1.6-dev-1498 | 2011-10-26 14:30:15 -0700 + + * Adding instructions to local.bro how to do ACTION_ALARM by + default. (Seth Hall) + +1.6-dev-1495 | 2011-10-26 10:15:58 -0500 + + * Updated unit test baselines. (Seth Hall) + +1.6-dev-1491 | 2011-10-25 20:22:56 -0700 + + * Updating submodule(s). (Robin Sommer) + +1.6-dev-1482 | 2011-10-25 19:08:32 -0700 + + * Fixing bug in log managers predicate evaluation. (Robin Sommer) + +1.6-dev-1481 | 2011-10-25 18:17:03 -0700 + + * Fix a problem with DNS servers being logged that aren't actually + servers. (Seth Hall) + + * Changed generated root cert DN format for RFC2253 compliance. (Jon + Siwek) + + * Removed :bro doc directives from notice documentation. (Seth Hall) + + * New notice framework docs. (Seth Hall) + + * Adding sub messages to emails. (Seth Hall) + + * Adding extra fields to smtp and http to track transaction depth. + (Seth Hall) + + * Fix for SSH login detection heuristic. (Seth Hall) + + * Removed some fields from http analysis that weren't commonly + needed or were wrong. (Seth Hall) + + * Updated/fixed MSIE version parsing in the software framework. + (Seth Hall) + + * Update Mozilla trust roots to index certs by subject distinguished + name. (Jon Siwek) + + * weird.bro rewrite. (Seth Hall) + + * More notice email tuning. (Seth Hall) + + * Slightly restructured http file hashing to fix a bug. (Seth Hall) + + * Changed the notice name for interesting ssh logins to correctly + reflect semantics of the notice. (Seth Hall) + + * Field name change to notice framwork. $result -> $action + + - $result is renamed to $action to reflect changes to the notice + framework since there is already another result-like field + ($suppress_for) and there may be more in the future. + + - Slipped in a change to add connection information to notice + emails too. (Seth Hall) + + * Small script refinements and documentation updates. (Seth Hall) + + * Pass over upgrade guide. (Robin Sommer) + + +1.6-dev-1430 | 2011-10-21 10:39:09 -0700 + + * Fixing crash with unknown debug streams. Closes #643. (Robin + Sommer) + + * Code to better handle interpreter errors, which can now be turned + into non-fatal runtime errors rather than immediate aborts. (Robin + Sommer). + + * Remove old make-src-packages script. (Jon Siwek) + + * Fixing a bunch of format strings. Closes #567. (Robin Sommer) + + * Cleaning up some distribution files. (Robin Sommer) + + * Various test, doc, and installation fixes/tweaks. (Seth Hall, Jon + Siwek and Robin Sommer). + + * Varios smaller policy fixes and tweaks (Seth Hall). + + * Moving docs from web server into distribution. (Robin Sommer) + + * Fixing more (small) memory leaks. (Robin Sommer) + + * Profiling support for DNS_Mgr and triggers. With + misc/profiling.bro, both now report a line in prof.log with some + counters on usage. (Robin Sommer) + + * Fixing DNS memory leaks. Closes #534. (Robin Sommer) + + * Fix code for disabling analyzers. Closes #577. (Robin Sommer) + + * Changed communication option from listen_encrypted to listen_ssl. + (Seth Hall) + + * Modification to the Communication framework API. (Seth Hall) + + - Simplified the communication API and made it easier to change + to encrypted connections by not having separate variables to + define encrypted and unencrypted ports. + + - Now, to enable listening without configuring nodes just + load the frameworks/communication/listen script. + + - If encrypted listening is desired set the following: + redef Communication::listen_encrypted=T; + + * Connection compressor now disabled by default. Addresses #559. + (Robin Sommer) + + +1.6-dev-1372 | 2011-10-06 18:09:17 -0700 + + * Filtering some potentially high-volume DNS weirds. (Robin Sommer) + + * DNS now raises DPD events. Closes #577. (Robin Sommer) + + * Fixing a bunch of compiler warnings. (Robin Sommer) + + * Remote logs are auto-flushed if the last write was longer than a + second ago. Addresses #498. (Robin Sommer) + + * Fix missing from previous MIME commit. (Robin Sommer) + +1.6-dev-1366 | 2011-10-06 17:05:21 -0700 + + * Make CompHash computation/recovery for functions deterministic. + Closes #636. (Jon Siwek) + + * Removing unnecessary @load in local.bro. (Robin Sommer) + + * Optimizing some MIME code. (Robin Sommer) + + * Speed improvements in logging code. (Robin Sommer) + + * Consolidating some node-specific functionality from scripts in + broctl repo. (Jon Siwek) + + * Another fix the for 1xx script code. (Robin Sommer) + +1.6-dev-1352 | 2011-10-05 16:20:51 -0700 + + * Fix for optional HTTP::Info status_code. (Jon Siwek) + + * Teaking some external testing scripts. (Jon Siwek) + + * HTTP bug fix reported by Martin Holste. (Seth Hall) + + * More script tuning. (Seth Hall) + + - Moved some of the weird events back to the base/ directory. + + - SSL fixes, updates, and performance optimization. + + * More adjustment to reduce Weird volumes. (Seth Hall) + + * Fixed an error when calculating x509 certificate hashes (reported + by Martin Holste). (Seth Hall) + + * Clean up to cluster framework to make event handling clearer. + (Seth Hall) + + * Fixed a bug in the notice framework. (Seth Hall) + + * Bug fix for FTP analysis script. (Seth Hall) + +1.6-dev-1333 | 2011-09-29 22:29:51 -0700 + + * Fixing a number of memory leaks. (Robin Sommer) + + * Loaded_scripts.log is indented with spaces now and makes more + sense to look at. (Seth Hall) + + * Teach HTTP parser to derive content length of multipart/byteranges + bodies. Addresses #488. (Jon Siwek) + + * Change logging of HTTP 1xx responses to occur in their own + columns. Addresses #411. (Jon Siwek) + + * Fix handling of HTTP 1xx response codes. Addresses #411). + + * Taking advantage of yet another trick to get installed browser + plugins. (Seth Hall) + + - With the software-browser-plugins script you can watch for Omniture + advertising servers to grab the list of installed plugins. + + - I reorganized the plugin detection a bit too to abstract it better. + + - Removed the WEB_ prefix from all of the Software::Type HTTP enums. + They were essentially redundant due to the full name already being + HTTP::SERVER (for example). + +1.6-dev-1316 | 2011-09-28 16:50:05 -0700 + + * Unit test cleanup. Updated README and collected coverage-related + tests in a common dir. (Jon Siwek) + + * Fixes for known-services. (Seth Hall) + + * Ported and 2.0ized the capture-loss script. (Seth Hall) + + * Communication fix and extension.(Robin Sommer) + + - Removing unnecessary log flushing. Closes #498. + + - Adding new BiF disconnect() that shuts a connection to a peer down. + + - terminate_connection() now first flushes any still buffered log + messages. + + * Fix for high SSL memory usage by adding &transient attribute to + top-level SSL pac array type. Closes #574. (Robin Sommer) + + * Fix a small bug in the metrics framework. (Seth Hall) + + * Temporarily removing scripts that aren't ready to be included. + Will return before next release. (Seth Hall) + + * New SSL policy scripts. (Seth Hall) + + - protocols/ssl/expiring-certs uses time based information from + certificates to determine if they will expire soon, have already + expired, or haven't yet become valid. + + - protocols/ssl/extract-certs-pem is a script for taking certs off + the line and converting them to PEM certificates with the openssl + command line tool then dumping them to a file. + + * Notice::type_suppression_intervals: table[Notice::Type] of + interval can be used to modify the suppression intervals for + entire types of notices. (Seth Hall) + + * EOF SSL protocol violations are only generated a single time now. + (Seth Hall) + + * Script level fixes. (Seth Hall) + + - Fixed a type name conflict in the Known namespace. + + - Fixed a DPD framework bug that was causing Reporter messages. + + - Fixed the notice_policy log. + + - Predicate functions are now logged. + + - Predicate functions are now optional. If not given, it's assumed that + the result should always apply. (Seth Hall) + + - Fix a problem with accidental and mistaken HTTP log lines. + +1.6-dev-1293 | 2011-09-22 19:44:37 -0700 + + * Smaller script tweaks. (Seth Hall) + + * Duplicate notice suppression. (Seth Hall) + + - Duplicate notices are discovered with the new Notice::Info + field $identifier. It's a string that is left up to the + notice implementor to define which would indicate a + fundamentally duplicate notice. The field is optional and + if it's not included it's not possible for notice + suppression to take place. + + - Duplicate notices are suppressed by default for the interval + defined by the Notice::default_suppression_interval variable + (1 hour by default). + + - A new notice action was defined ACTION_NO_SUPPRESS to prevent + suppression for a specific notice instance. A convenience set + named not_suppressed_types was also created to not suppress + entire notice types. + + - A new field was added to the PolicyItem type to modify the length + of time a notice should be suppressed if the predicate matches. + The field is named $suppress_for. This name makes the code more + readable like this: $suppress_for = 1day + + - New events were created to give visibility into the notice + framework's suppression activity. + - event Notice::begin_suppression(n: Notice::Info) + - event Notice::suppressed(n: Notice::Info) + - event Notice::end_suppression(n: Notice::Info) + + - The suppression.bro script doesn't have a baseline because + it is causing a segfault in Bro. This one test is the + reason that this is being integrated into a branch instead + of master. (Seth Hall) + + * Fix crash on exit. Addresses #607. (Jon Siwek) + + * Fix PktSrc setting next_timestamp even when no packet available. + (Jon Siwek) + + * Fix lack of NUL-termination in to_upper/to_lower BIF's return val. + (Jon Siwek) + + * Fixing unit tests and some minor bugs. (Jon Siwek) + + * Fix broctl cluster log rotation. Addresses #619. (Jon Siwek) + + * Added session ID to the SSL logging. (Seth Hall) + + * Adding "install-aux" target + updating bro-aux submodule. (Jon + Siwek) + + * Cleaning up INSTALL and README. (Jon Siwek) + + * Remove $Id$ tags. (Jon Siwek) + + * Remove policy.old directory. Addresses #511. (Jon Siwek) + + * Small rework with ssl base script to reduce memory usage. (Seth + Hall) + + * Updated the mozilla root certs. (Seth Hall) + +1.6-dev-1261 | 2011-09-15 17:13:55 -0700 + + * Memory leak fixes. Addresses #574 (Jon Siwek) + + * Add configure options for ruby/bindings integration. (Jon Siwek) + + * Fix filter path_func to allow record argument as a subset of + stream's columns. Addresses #600. (Jon Siwek) + + * Log rotation is now controlled directly through Filter records. (Jon Siwek) + + * Fix indexing for record types with optional fields. Addresses #378 + (Jon Siwek) + +1.6-dev-1248 | 2011-09-15 16:01:32 -0700 + + * Removed custom malloc() implementation for FreeBSD. Closes #557. + (Jon Siwek) + + * Testing/external scripts no longer compute MD5 checksums for SMTP + entities. (Robin Sommer) + + * External tests no longer include the full content of mismatching + files in the diagnostics output. (Robin Sommer) + +1.6-dev-1241 | 2011-09-14 22:51:52 -0400 + + * Fixing a major memory utilization issues with SSL analysis. (Seth + Hall) + + * Enhancements to HTTP analysis: (Seth Hall) + + - More options for the header-names.bro script. + + - New script for logging header names and values. Closes #519. + (Seth Hall) + + - HTTP body size measurement added to http.log. + + - The value of the content-length headers has now been removed + in the default output but it could be added back locally at an + installation by a user. + + - Added fields to indicate if some parsing interruption happened + during the body transfer. Closes #581 (Seth Hall) + + * Misc smaller usability and correctness updates: (Seth Hall) + + - Removed an notice definition from the base SSL scripts. + + - Moved a logging stream ID into the export section for known-services + and bumped priority for creating the stream. + + - Adding configuration knobs for the SQL injection attack detection + script and renaming the HTTP::SQL_Injection_Attack notice to + HTTP::SQL_Injection_Attack_Against + + - Bumped priority when creating Known::CERTS_LOG. + + - Fixing a warning from the cluster framework. (Seth Hall) + + * Bugfix for log writer, which didn't escape binary stuff in some + situations. Closes #585. (Robin Sommer) + + * A larget set of changes to the testing/external infrastructure. + The traces for external test-suites are no longer kept inside the + repositories themselves but downloaded separately via curl. This + is because git is pretty bad at dealing with large files. See the + README for more information. (Robin Sommer) + +1.6-dev-1221 | 2011-09-08 08:41:17 -0700 + + * Updates for documentation framework and script docs. (Jon Siwek) + + * The script level PF_RING support isn't working so removing it. + (Seth Hall) + + * Delete SSL certificates from memory after ssl_established event. + (Seth Hall) + + * Small fixes for SSL analysis. (Seth Hall) + +1.6-dev-1212 | 2011-09-07 16:15:28 -0700 + + * Internally, the UID generation can now return values from + different pool for better reproducability in testing mode. + (Gilbert Clark). + + * Added new BiF unique_id_from(pool: string, prefix: string) that + allows the user to specify a randomness pool. (Gilbert Clark) + +1.6-dev-1198 | 2011-09-07 11:03:36 -0700 + + * Extended header for ASCII log that make it easier for scripts to + parse Bro log files. (Gilbert Clark) + + * Potential fix for rotation crashes. Addresses #588. (Robin Sommer) + + * Added PF_RING load balancing support to the scripting layer, + enabled by loading the misc/pf-ring-load-balancing script. (Seth + Hall) + + * Added a BiF setenv() for setting environment variables. (Seth + Hall) + +1.6-dev-1184 | 2011-09-04 09:34:50 -0700 + + * FindPCAP now links against thread library when necessary (e.g. + PF_RING's libpcap). (Jon Siwek) + + * Install binaries with an RPATH. (Jon Siwek) + + * Fix for a case where nested records weren't coerced even though + possible. (Jon Siwek) + + * Changed ASCII writer to delay creation of log after rotation until + next write. + + * Changed default snaplen to 65535 and added a -l/--snaplen command + line option to set it explicitly. Addresses #447. (Jon Siwek) + + * Various updates to logging framework. (Seth Hall) + + * Changed presentation of enum labels to include namespace. (Jon + Siwek) + + * HTTP analyzer is now enabled with any of the HTTP events. (Seth + Hall) + + * Fixed missing format string that caused some segfaults. (Gregor + Maier) + + * ASCII writer nows prints time interval with 6 decimal places. + (Gregor Maier) + + * Added a Reporter::fatal BIF. (Jon Siwek) + + * Fixes for GeoIP support. Addresses #538. (Jon Siwek) + + * Fixed excessive memory usage of SSL analyzer on connections with + gaps. (Gregor Maier) + + * Added a log postprocessing function that can SCP rotated logs to + remote hosts. (Jon Siwek) + + * Added a BiF for getting the current Bro version string. (Jon + Siwek) + + * Misc. doc/script/test cleanup. (Jon Siwek) + + * Fixed bare-mode @load dependency problems. (Jon Siwek) + + * Fixed check_for_unused_event_handlers option. (Jon Siwek) + + * Fixing some more bare-mode @load dependency issues (Jon Siwek) + + * Reorganizing btest/policy directory to match new scripts/ + organization. Addresses #545 (Jon Siwek) + + * bro scripts generated from bifs now install to + $prefix/share/bro/base. Addresses #545 (Jon Siwek) + + * Changeed/fixed some cluster script error reporting. (Jon Siwek) + + * Various script normalization. (Jon Siwek) + + * Add a test that checks each individual script can be loaded in + bare-mode. Adressess #545. (Jon Siwek) + + * Tune when c$conn is set. Addresses #554. (Gregor Maier) + + * Add ConnSize_Analyzer's fields to conn.log. (Gregor Maier) + + * Fixing bug in "interesting hostnames" detection. (Seth Hall) + + * Adding metrics framework intermediate updates. (Seth Hall) + +1.6-dev-1120 | 2011-08-19 19:00:15 -0700 + + * Fix for the CompHash fix. (Robin Sommer) + +1.6-dev-1118 | 2011-08-18 14:11:55 -0700 + + * Fixing key size calculation in composite hash code. (Robin Sommer) + +1.6-dev-1116 | 2011-08-18 10:05:07 -0700 + + * Remove the 'net' type from Bro (addresses #535). + + * Fix H3 assumption of an 8-bit byte/char. (Jon Siwek) + + * Allow reading from interface without additional script arguments. + Explicitly passing in '-' as an additional command line argument + still allows reading a script from stdin. (Jon Siwek) + + * SSH bruteforcing detection now done with metrics framework. (Seth + Hall) + + * Updates for SQL injection attack detection to match the metrics + framework updates. (Seth Hall) + + * Metrics framework now works on cluster setups. (Seth Hall) + + * Reclassifying more DNS manager errors as non-fatal errors. (Robin + Sommer) + + * Fix ConnSize_Analyzer when used in conjunction with connection + compressor. (Gregor Maier) + + * Fix reporter using part of the actual message as a format string. + (Jon Siwek) + +1.6-dev-1095 | 2011-08-13 11:59:07 -0700 + + * A larger number of script documentation updates. Closes #543. (Jon + Siwek) + + * Workaround for FreeBSD CMake port missing debug flags. (Jon Siwek) + + * piped_exec() can now deal with null bytes. (Seth Hall) + + * Fix vector initialization for lists of records with optional + types. Closes #485. (Jon Siwek) + + * Fix redef'ing records with &default empty set fields. Closes #460. + (Jon Siwek) + + * Fix ConnSize_Analyzer when used in conjunction with the connection + compressor. (Gregor Maier) + + * Fix reporter using part of the actual message as a format string. + (Jon Siwek) + + * Fixing reporter's location tracking. Closes #492. (Robin Sommer) + + * Turning DNS errors into warnings. Closes #255. (Robin Sommer) + + * Logging's path_func now receives the log record as argument. + Closes #555. (Robin Sommer) + + * Functions can now be logged; their full body gets recorded. + Closes #506. (Robin Sommer) + + * Bugfix for hostname notice email extension. (Seth Hall) + + * Updates for notice framework. (Seth Hall) + + - New ACTION_ADD_GEODATA to add geodata to notices in an extension + field named remote_location. + + - Loading extend-email/hostnames by default now that it only does + anything when the ACTION_EMAIL action is applied (finally). + + * Updates to local.bro (Seth Hall) + + * Added the profiling script. (Seth Hall) + + * Updates for SSH scripts. (Seth Hall) + + * ConnSize analyzer is turned on by default now. (Seth Hall) + + * Updates for the build system and site local scripts for cluster. + (Seth Hall) + + * HTTP now uses the extract_filename_from_content_disposition function. (Seth Hall) + + * Major SMTP script refactor. Closes #509. (Jon Siwek and Seth Hall) + + * New variable Site::local_nets_table in utils/site for mapping + address to defined local subnet. + + * Metrics framework updates, more to come. (Seth Hall) + + +1.6-dev-1061 | 2011-08-08 18:25:27 -0700 + + * A set of new/changed tests regarding the new policy script + organisation. (Robin Sommer) + +1.6-dev-1058 | 2011-08-08 16:15:18 -0700 + + * Reorganisation of the scripts that Bro loads by default. (Seth + Hall) + + - policy/ renamed to scripts/ + + - By default BROPATH now contains: + - scripts/ + - scripts/policy + - scripts/site + + - The scripts in scripts/base/protocols/ only do logging and state + building. + + - All of scripts/base/ is loaded by by default. This can however + be disabled by switching Bro into "bare mode" using the new + command-line option --bare-mode (or -b). The cripts in + scripts/base/ don't use relative path loading to ease use of + bare mode (to copy and paste that script). + + - The scripts in scripts/base/frameworks/ add functionality + without causing any additional overhead. + + - All "detection" activity happens through scripts in + scripts/policy/. + + - bro.init was renamed to base/init-bare.bro, and base/all.bro was + renamed to init-default.bro. + + - local.bro now loads more functionality from policy/ and adds + more documentation. (Seth Hall) + + * Adding default_path_func() to the logging framework that makes the + default naming scheme script-level controlled. (Robin Sommer) + + * Reworking logging's postprocessor logic so that postprocessor + commands are no longer run by the log writers themselves, but + instead by a script level function. (Robin Sommer) + + * The communication subsystem is now by default off and must be + enabled explicitly with a new BiF, enable_communication(). Closes + #540. (Robin Sommer) + + * The hostname notice email extension now only add hostnames for + emailed noticed. (Seth Hall) + + * Cleaning up doc generation. (Seth Hall) + +1.6-dev-1044 | 2011-08-05 19:07:32 -0700 + + * Fixing memory (and CPU) leak in log writer. + + * Fixing crash in memory profiling. (Robin Sommer) + + * Fix compiler warning. (Robin Sommer) + + * Fixing missing sync in cluster setup. (Robin Sommer) + + +1.6-dev-1038 | 2011-08-05 18:25:44 -0700 + + * Smaller updates to script docs and their generation. (Jon Siwek) + + * When using a `print` statement to write to a file that has raw output + enabled, NUL characters in string are no longer interpreted into "\0", + no newline is appended afterwards, and each argument to `print` is + written to the file without any additional separation. (Jon Siwek) + + * Test portatibility tweaks. (Jon Siwek) + + * Fixing PktSrc::Statistics() which retured bogus information + offline mode. Closes #500. (Jon Siwek) + + * --with-perftools configure option now assumes --enable-perftools. + Closes #527. (Jon Siwek) + +1.6-dev-1018 | 2011-07-31 21:30:31 -0700 + + * Updating CHANGES. (Robin Sommer) + +1.6-dev-1016 | 2011-07-30 18:34:28 -0700 + + * Install example config files dynamically. They'll only get + installed when the distribution version differs from existing + version on disk. (Jon Siwek) + + * Fixed memory leak in SSL analyzer. (Seth Hall) + + * Beginning rework of metrics interface. (Seth Hall) + + * New/updated unit tests for scripts. (Jon Siwek) + + * New/updated documentstion for scripts. (Jon Siwek) + + * A number of fixes for scripts in utils/. (Jon Siwek) + +1.6-dev.244 Thu Jul 28 17:08:21 PDT 2011 + +- mask_addr() now returns subnet (addresses #512). (Jon Siwek) + +- Normalize Notice::Type identifiers per convention (closes #484). + (Jon Siwek) + +- Fixing default-loaded-scripts test for BSD systems. (Jon Siwek) + +- New piped_exec() BiF for pipeing data into an external command. (Jon + Siwek) + +1.6-dev.242 Mon Jul 25 21:42:39 PDT 2011 + +- Adding a documentation coverage test. (Jon Siwek) + +- The CMake targets for generating reST docs from policy scripts are + now automatically generated via the genDocSourcesList.sh script. + (Jon Siwek) + +- Fixed a number of script error. (Jon Siwek) + +- Fixes to relative @load'ing. (Jon Siwek) + +- Fixes to tests. (Robin Sommer) + +1.6-dev.240 Sun Jul 24 15:14:26 PDT 2011 + +- Updated tests and test baselines. (Jon Siwek) + +- ASCII log writer now prints time values w/ constant 6 digit + precision. (Jon Siwek) + +- Many policy script updates acrsso the board (Seth Hall). + +- Moving devel-tools to bro-aux. (Robin Sommer) + +- BugFix for disable_analyzer(), which could cause crashes with some + analyzers. (Robin Sommer) + +- Bugfix for potential segfault in DebugLogger. (Robin Sommer) + +1.6-dev.226 Thu Jul 21 15:23:39 PDT 2011 + +- Extensions to the @load and @unload process. (Jon Siwek) + + * Make @load statements recognize relative paths. For example a + script can do "@load ./foo" to load a script named foo.bro that + lives in the same directory or "@load ../bar" to load a script + named bar.bro in the parent directory, even if those directories + are not contained in BROPATH. + + * Reimplementation of the @prefixes statement. (Closes #486) + + Any added prefixes are now used *after* all input files have + been parsed to look for a prefixed, flattened version of the + input file somewhere in BROPATH and, if found, load it. For + example, if "lcl" is in @prefixes, and site.bro is loaded, then + a file named "lcl.site.bro" that's in BROPATH would end up being + automatically loaded as well. Packages work similarly, e.g. + loading "protocols/http" means a file named + "lcl.protocols.http.bro" in BROPATH gets loaded automatically. + + * Fix @unload'd files from generating bro_script_loaded event. + + * Updates to tests. + +1.6-dev.225 Wed Jul 20 17:10:41 PDT 2011 + +- IRC improvements (Jon Siwek). Including: + + * Shorten what's displayed in the IRC's log mime_type column for + DCC transfers. + + * Add IRC unit tests. + + * Fix IRC analyzer supplying wrong type to irc_dcc_message event. + + * Removed irc_client and irc_server events. + + * Added is_orig arguments to all other irc events. + + * Fix analyzer not recognizing Turbo DCC extension message format. + + * Fix analyzer not generating irc_dcc_message event when irc_privmsg_message + event doesn't have a handler registered. + +- Fixing tests that need a diff canonifier. (Jon Siwek) + +1.6-dev.223 Tue Jul 19 19:10:36 PDT 2011 + +- Adding a script to update CHANGES and VERSION. (Robin Sommer) + +1.6-dev.218 Tue Jul 19 18:16:44 PDT 2011 + +- Comprehensive policy script overhaul/rewrite. (Seth Hall) + + Changes are too extensive to list individually. + +- Removing undocumented -H command line flag. (Robin Sommer) + +- Fixing many tests. (Everybody) + +- Fixing 0-chunk bug in remote logging. (Robin Sommer) + +- $PATH is now appropriately set by the bro-path-dev.(sh|csh) scripts. + (Seth Hall) + +- Making valgrind a bit more happy. (Robin Sommer) + +- New BiF record_field_vals() that returns the fields of a record in a + table with meta-information. (Robin Sommer) + +- Adding a script in aux/devel-tools that extracts a connection from a + trace based on uid. (Robin Sommer) + +- Fixing bug causing crash when running without arguments. (Robin Sommer) + +- A new event bro_script_loaded() raised for each policy script + loaded. Also removing the -l command-line option as that can now be + done at the script-level. (Robin Sommer) + +- Fixing memory leaks. (Gilbert Clark, Seth Hall, Robin Sommer) + +- Many SSL analysis improvements and fixes. (Seth Hall) + +- Fixing bug with event priorities potentially being ignored for the + handler. (Robin Sommer) + +- Overhauling the internal reporting of messages to the user. The new + Reporter class is now in charge of reporting all errors, warnings, + informational messages, weirds, and syslogs; and it passes + everything through the script layer. (Robin Sommer) + +* Removed the alarm statement and the alarm_hook event. (Robin Sommer) + +- Adding new policy file test-all.bro that loads all other policies. + This is for testing only. (Robin Sommer) + +- A new framework for doing regression testing with larger traces and + more complex Bro configurations in testing/external. (Robin Sommer) + +- Many updates to script doc generation. (Jon Siwek) + +1.6-dev.146 Sat Jun 25 18:12:27 PDT 2011 + +- DNS mapping are now becoming invalid when an entry's TTL expires. + (Thomas Other) + +- Reworking how Bro tracks which scripts are already loaded. Rather + than paths, Bro now tracks inode numbers. (Jon Siwek) + +- New BiF netstats() to query packet capture statistics. The netstats + script now uses the new BiF to periocally report packets drops. The + net_stats_update() event and the heartbeat_interval global went + away. (Seth Hall) + +- Fixing bug with logging &optional records. Closes #476. (Robin + Sommer) + +- Fixing istate.events-ssl test failing because of expired cert. (Jon + Siwek) + +- A large number of improvements and fixes for Bro's doc mode. (Jon + Siwek) + +- Significant updates for RPC and NFS analyzers (Gregor Maier) + + * Unify semantics for UDP and TCP connections. + + * RPC can now log to a log file if desired. + + * Portmapper can now log general activity to a log file and also log + actual port mappings. + + * NFS analyzer now supports significantly more procedure calls as + as file name tracking and file content extraction. + +- NetBIOS fixes. (Jon Siwek) + +- A number of unit tests are more robust and portable. (Jon Siwek) + +- A new BiF unique_id() that returns a string that's unique across Bro + instaces with high probablity. (Robin Sommer) + +- Complete rewrite of the BinPAC SSL analyzer. (Seth Hall) + + * DER certificates are extracted as strings to be used with + corresponding BiFs. + + * x509_verify function to verify single certs and/or full + certificate chains. + + * Removed hand written SSL analyzer. + + * The ssl.bro script is just a place-holder for now. New version + will come with the other new scripts. + +- New syslog analyzer. (Seth Hall) + +- @load now supports loading a directory. With a directory "foo" + somewhere in BROPATH, "@load foo" now checks if there's a file + "foo/__load__.bro". If so, it reads that file in. (Robin Sommer) + +- ASCII logger now escapes non-printable characters. Closes #450. + (Robin Sommer) + +- Packaging tweaks and rewrite of 'dist' target. (Jon Siwek) + +- Changes to allow DEB packaging via CPack, addresses #458. (Jon + Siwek) + +- An extension to the ICMP analyzer to handle redirects. Julien + Sentier + +- Removing old istate test-suite. (Robin Sommer) + +- A hack to report missing GeoIP support only once. This closes #357, + but #455 captures the need for a more general solution. (Robin + Sommer) + +- Bugfix: vectors in records were not initalized. Closes #421. (Robin + Sommer) + +- If IPv6 default is not compiled in, the default BPF filters now + excludes IPv6 packets. (Robin Sommer) + +- New bif bro_has_ipv6() to check whether IPv6 support is compiled in. + (Robin Sommer) + +- Updating btests and a Makefile. "make" now runs all the tests. + (Robin Sommer) + +- Moving the test-scripts from the old test-suite over to btest. + (Robin Sommer) + +- Fix for major bug in POP3 analyzer, which didn't recognize '.' + terminators in multi-line replies if the terminator was bare (no + newline). This caused it to ignore the rest of the session that it's + analyzing. (Vern Paxson) + +- Fix compiler warning with gcc-4.4.4 (Gregor Maier) + +- Adding example documentation for a script's use of logging features. + (Jon Siwek) + +- Adding &log attribute to static attr_names array. (Jon Siwek) + +- Bro can now track packet and byte counts per connection. (Gregor + Maier) + + * If 'use_conn_size_analyzer' is true, the event engine tracks + number of packets and raw IP bytes per connection. If + report_conn_size_analyzer is true, these values are included as + four new columns into conn.log + + * I changed conn.bro so that the value of + report_conn_size_analyzer follows that of + use_conn_size_analyzer. For the new conn.log, we probably want + to get rid of report_conn_size_analyzer anyway. + +- Fixing numerous compiler warnings and portability issues. (All) + +- Switching vectors from being 1-based to 0-based. Note that this is a + change that break backwards-compatibility. (Robin Sommer) + +- Increasing serialization format version for the recent 64-bit + changes. (Robin Sommer) + +- Support for (mixed) MPLS and VLAN traffic, and a new default BPF + filter. (Seth Hall and Robin Sommer) + + * Merging in the patch from #264, which provides support for mixed + VLAN and MPLS traffic. + + * Changing Bro's default filter from being built dynamically to + being a static "ip or not ip". To get the old behaviour back + (i.e., the dynamically built filter), redef "all_packets" to + false. + + * print-filter.bro now always prints the filter that Bro is + actually using, even if overriden from the command line. (Robin + Sommer) + +- Changing the HTTP's analyzers internals to use 64-bit integers. + (Gregor Maier). + +- Fixing bug with deleting still unset record fields of table type. + (Robin Sommer) + +1.6-dev.99 Fri Apr 22 22:10:03 PDT 2011 + +- Extending the connection record with a unique identifier. (Robin + Sommer) + + type connection: record { + [...] + id: string; + }; + + These identifiers very likely unique even across independent Bro + runs. + +- Delete operator for record fields. (Robin Sommer) + + "delete x$y" now resets record field "x" back to its original state + if it is either &optional or has a &default. "delete" may not be + used with non-optional/default fields. + +- Fixing bug with nested record coercions. (Robin Sommer) + +- Fixing a do_split() bug. (Seth Hall) + + +1.6-dev.94 Thu Apr 21 19:51:38 PDT 2011 + +- Fixing generation of config.h. (Jon Siwek) + +- Updates and tests for NetBIOS name BiF. (Seth Hall) + +- Fixing do_split bug(), and adding a test. (Seth Hall) + +- When Bro is given a PRNG seed, it now uses its own internal random + number generator that produces consistent results across sytems. + Note that this internal generator isn't very good, so it should only + be used for testing purpses. (Robin Sommer) + +- The BTest configuration now sets the environemnt variables TZ=UTC + and LANG=C to ensure consistent results. (Robin Sommer) + +- Logging fixes. (Robin Sommer) + +1.6-dev.88 Wed Apr 20 20:43:48 PDT 2011 + +- Implementation of Bro's new logging framework. We will document this + separately. (Robin Sommer) + +- Already defined record types can now be further extended via the + '+=' operator. The added fields must be either &optional or have a + &default value. (Robin Sommer) + + Example: + + type Foo: record { + a: count; + b: count &optional; + }; + + redef record Foo += { + c: count &default=42; + d: count &optional; + }; + + global f: Foo = [$a=21]; + + print f; + + Output: + + [a=21, b=, c=42, d=] + +- Enabling assignment of empty vectors ("vector()"). (Robin Sommer) + +- Fixing attributes to allow &default attributes to be associated with + records fields of type tables/sets/vector. (Robin Sommer) + +- '[]' is now a valid record constructor. (Robin Sommer) + +- A instance of a record type A is now coercable into one of type B if + the fields of type A are a subset of those of type B. (Robin Sommer) + +- A number of bug fixes and enhancements for record/set/table/vector + coercion. (Robin Sommer) + +- Fixing a problem with records that have optional fields when used as + table/set indices. Addresses #367. (Robin Sommer) + +- Fixing an off-by-one error in join_string_vec(). (Seth Hall) + +- Updating to_count() to cope with 64bit ints. (Seth Hall) + +- A new BiF count_to_v4_addr() to turn a count into an IPv4 address. + (Seth Hall) + +1.6-dev.80 Mon Apr 18 14:50:54 PDT 2011 + +- New framework for generating documentation from Bro scripts. (Jon + Siwek) + + This includes: + + * Changes to Bro's scanner/parser to facilitate automatic + generation of Bro policy script documentation in + reStructuredText format. + + * New command line flags -Z/--doc-scripts to enable the new doc + generation mode. + + * Changes to bifcl to pass comments starting with "##" through + into the generated .bro script. + + * A "doc" build target for the top-level Makefile to first + generate reStructuredText for a defined set of Bro policy + scripts, and then run that through Sphinx to create HTML + documentation. + +1.6-dev.78 Mon Apr 18 12:52:55 PDT 2011 + +- Adding files to CMake build targets so they show up in generated IDE + projects. This addresses #413. (Jon Siwek) + +- Fix unnecessary config.h preprocessor (re)definitions. This + addresses #414. (Jon Siwek) + +- Updating istate tests. (Robin Sommer) + +- Adding files to CMake build targets so they show up in generated IDE + projects. + +- Adding new environment variable BRO_SEED_FILE to set the seed file + for the random number generator. (Robin Sommer) + +1.6-dev.71 Fri Apr 1 16:06:33 PDT 2011 + +- Removing code for the following no longer supported functionality. + + * Trace rewriting. + * DFA state expiration in regexp engine. + * Active mapping. + * Unused hash functions. + + (Robin Sommer) + +- Fixing crashes when SSL is not configured correctly. (Robin Sommer) + +1.6-dev.66 Tue Mar 29 21:52:01 PDT 2011 + +- Initial btest setup (Don Appleman and Robin Sommer) + +- Porting the istate tests to btest (not finished) (Robin Sommer) + +1.6-dev.63 Mon Mar 21 16:31:15 PDT 2011 + +- Changes to the way user-modifiable config files are installed (Jon Siwek) + + * Duplicates of the distribution's configuration files are now + always installed with a .example suffix + + * Added --binary-package configure option to toggle configure + logic specific to the creation of binary packages. + + * When not in binary packaging mode, `make install` never + overwrites existing configure files in case they've been + modified. The previous behavior (CMake's default) would only + avoid overwriting modified files if one consistently uses the + same build directory and doesn't reconfigure. + +- Fixed an issue with Mac package's pre-install script not preserving + ACLs. (Jon Siwek) + +- Minor cleanup/refactor of the make-mac/rpm-packages scripts. (Jon + Siwek) + +- Add explicit CMake check for compiler. (Jon Siwek) + +- Add alternative way to set BROPATH for running bro from build/ dir. + (Jon Siwek) + +- Fixing compiler warnings (Gregor Maier) + +- Remvoing leftover local variables that caused compile error on Mac + OS X. (Gregor Maier) + +1.6-dev.53 Fri Feb 25 17:03:05 PST 2011 + +- Fixing file detector leak in remote communication module. (Scott + Campbell) + +- Updating independent-state tests to work with new setup. (Robin + Sommer) + +1.6-dev.49 Fri Feb 25 15:37:28 PST 2011 + +- Enum IDs can have explicitly defined values. (Gregor Maier) + +- Extensions for the built-in function compiler, bifcl. (Gregor Maier) + + * Support for policy-layer namespaces. + * Support for type declarations in bif files (with access them + from C++) + * Extended const declarations in bif files. + + See http://bro.icir.org/devel/bif-doc for more information. + +1.6-dev.48 Fri Feb 25 10:53:04 PST 2011 + +- Preliminary TCP Reassembler fix: deliver data after 2GB by disabling + the unused seq_to_skip feature. (Gregor Maier) + +1.6-dev.47 Fri Feb 25 10:40:22 PST 2011 + +- Fixing endianess error in XDR when data is not 4-byte aligned. + (Gregor Maier) + +- Fix for Val constructor with new int64 typedefs. (Gregor Maier) + +- Updated fix for OS X 10.5 compile error wrt llabs(). (Gregor Maier) + +- Fix more compiler warning wrt printf format strings. (Gregor Maier) + +1.6-dev.45 Tue Feb 8 21:28:01 PST 2011 + +- Fixing a number of compiler warnings. (Seth Hall and Robin Sommer) + +1.6-dev.44 Tue Feb 8 20:11:44 PST 2011 + +- A number of updates to the SSL analyzer, including support for new + ciphers; SSL extensions; and bug fixes. The analyzer does not longer + throw weird for exceeding a predefined cipherspec_size anymore. + (Seth Hall and Rmkml). + +- The various split*() BiFs now handle strings containing null bytes + correctly. (Seth Hall) + +- Adding new aux/btest submodule. This is a framework we will use in + the future for doing unit tests. (Robin Sommer) + +1.6-dev.41 Mon Feb 7 13:43:56 PST 2011 + +- Smarter way to increase the parent/child pipe's socket buffer. + (Craig Leres). + +- Fixing bug with defining bro_int_t and bro_uint_t to be 64 bits wide + on some platforms. (Robin Sommer) + +1.6-dev.39 Mon Jan 31 16:42:23 PST 2011 + +- Login's confused messages now go through weird.bro. (Robin Sommer) + +1.6-dev.36 Mon Jan 31 08:45:35 PST 2011 + +- Adding more configure options for finding dependencies, (Jon Siwek) + + --with-flex=PATH path to flex executable + --with-bison=PATH path to bison executable + --with-perl=PATH path to perl executable + --with-python=PATH path to Python interpreter + --with-python-lib=PATH path to libpython + --with-python-inc=PATH path to Python headers + --with-swig=PATH path to SWIG executable + +- Fixing typo in PCAPTests.cmake (Jon Siwek) + + +1.6-dev.33 Mon Jan 24 15:29:04 PST 2011 + +- Fixing bug in SMB analyzer. (Robin Sommer) + +- Configure wrapper now deletes previous CMake cache (Jon Siwek) + +- Fix for the --with-binpac configure option. (Jon Siwek) + +1.6-dev.30 Thu Jan 20 16:32:43 PST 2011 + +- Changed configure wrapper to create config.status. (Jon Siwek) + +1.6-dev.29 Thu Jan 20 16:29:56 PST 2011 + +- Fixing little problem with initialization of Bro-to-Bro event + communication. (Christian Kreibich) + + +1.6-dev.27 Thu Jan 20 13:52:25 PST 2011 + +- Fine-tuning of the HTTP analyzer in terms of raising protocol + violations and interrupted transfers. (Gregor Maier) + + +1.6-dev.21 Wed Jan 19 17:36:02 PST 2011 + +- Added 4 new BiFs and a new record type for testing the entropy of + strings. (Seth Hall) + + find_entropy(data: string): entropy_test_result + This is a one shot function that accepts a string and + returns the result of the entropy calculations. + + entropy_test_init(index: any): bool + This and the next two functions are for calculating entropy + piece-wise. It only needs an index which can be any type of + variable. It needs to be something that uniquely identifies + the data stream that is currently having it's entropy + calculated. + + entropy_test_add(index: any, data: string): bool + This function is used to add data into the entropy + calculation. It takes the index used in the function above + and the data that you are adding and returns true if + everything seemed to work, false otherwise. + + entropy_test_finish(index: any): entropy_test_result + Calling this function indicates that all of the desired data + has been inserted into the entropy_test_add function and the + entropy should be calculated. This function *must* be called + in order to clean up an internal state tracking variable. + If this is never called on an index, it will result in a + memory leak. + + The entropy_test_result values have several measures of the + entropy, but a good one to work with is the "entropy" attribute. + It's a double and as the value approaches 8.0 it can be considered + more and more random. For example, a value of 7.832 would be + quite random but a value of 4.671 is not very random. + +1.6-dev.20 Wed Jan 19 17:30:11 PST 2011 + +- BRO_DNS_FAKE is now listed in the --help output. (Seth Hall) + + +1.6-dev.18 Wed Jan 19 16:37:13 PST 2011 + +- Removing unnecessary expire timer from http_sessions. (Gregor + Maier) + + +1.6-dev.16 Sat Jan 15 14:14:21 PST 2011 + +- Updates to the build system. (Jonathan Siwek) + + * ``make dist`` is now available to be used with the top-level + Makefile for creating source packages according to #344. + + * ``make-rpm-packages`` and ``make-mac-packages`` scripts can + now generate binary packages according to #295. + + * Additional configure options to change packaging behavior. + + * OS X builds will now prefer to link static libraries of + optional dependencies that don't come with the vanilla + operating system. + + * Fix for OS X 10.5 compile error dealing with the llabs() + function from stdlib. + + * Installing as a different user than the one that + configured/built now works (although, a harmless error message + about not being able to write the install manifest may occur). + + +1.6-dev.3 Wed Dec 8 04:09:38 PST 2010 + +- Merge with Subversion repository as of r7137. Incorporated change: + + * Fix for packet processing resumption when a remote Bro dies + during state synchronization (Robin Sommer). + +1.6-dev.2 Wed Dec 8 03:57:03 PST 2010 + +- Compatibility fix for OpenSSL 1.0.0 (Christian Kreibich, Gregor + Maier). + +1.6-dev.1 Sat Nov 27 12:19:47 PST 2010 + +- Merge with Subversion repository as of r7098. Incorporated changes: + + * Rotation post-processors are now passed an additional argument + indicating whether Bro is terminating (Robin Sommer). + + * Bro now consistently generates a file_opened event for all + fopen() calls. (Robin Sommer). + + * You can now redefine the email_notice_to function (Robin + Sommer). + +1.6-dev.0 Fri Nov 26 13:48:11 PST 2010 + +- The Bro source code is now developed in the new git repositories. + See the developer pages at http://www.bro-ids.org for more + information on the new development process. + +- Bro's build and installation setup has been moved from GNU + autotools to CMake. As a result of that, layout and specifics of + the distribution has changed significantly. + +- Lots of pieces have been removed from the distribution that are + either now unnecessary or are no longer maintained. + +- As part of the cleanup, a numbef of Bro configure options and + their corresponding functionality have been removed, including: + + * --disable-select-loop + * --with-dag + * --disable-nbdns + * --enable-activemapping + * --enable-activemapping + * --enable-shippedpcap + +- The previous configure option --enable-int64 is now enabled by default, + and can no longer be disabled. + +- ClamAV support has been removed, which has been non-functional for + a while already. + +1.5.2.7 Sun Sep 12 19:39:49 PDT 2010 + +- Addressed a number of lint nits (Vern Paxson). + + +1.5.2.6 Sun Sep 12 17:00:13 PDT 2010 + +- The SWIG file now explicitly lists those pieces from broccoli.h which it + wants to wrap, rather than just including all of broccoli.h (Robin Sommer). + This fixes the problem that the SWIG bindings depend on what configure + finds out about the availability of libpcap even though the corresponding + functions don't need to be wrapped anyway. + +- http-header.bro now includes a global include_header: set[string] + (Robin Sommer). If it contains any strings, then only those headers + will be processed. If left empty, then you continue to get the current + behavior of processing all headers. + +- Several changes to drop.bro (Robin Sommer): + + * If true, the new flag Drop::dont_drop_locals indicates that + local hosts should never be dropped. On by default. + + * If true, the new flag Drop::debugging activates extensive debugging + output for the catch-and-release logic. Off by default. + + * The timeout for tracking dropping information is now 1 day + rather than 7 days, to better echo the one-restart-a-day semantics + used in the past. + + * Bug fix for hosts once dropped by later cleared; some state + for them persisted. + +- Portability fix for Broccoli Python bindings on 64-bit platforms (Robin + Sommer). + +- The HTTP analyzer no longer attempts to track Server/User-Agent + versions, as these are hugely voluminous (Seth Hall). Ideally this + would still be available as an option for someone who truly wants + the full set. + +- HTTP and SMTP no longer have extra-short inactivity timeouts, as + these were too often leading to premature expiration of a connection + (Robin Sommer). + +- The "rst" tool (aux/rst/) now takes an optional "-I " argument + that instructs it to inject as payload rather than sending a RST + packet (Vern Paxson). must be NUL-terminated, and the NUL is not + included. + +- Bug fix for crashes in the DNS analyzer when processing replies for + which no request was seen (Robin Sommer). + + +1.5.2.5 Mon Jul 19 16:20:58 PDT 2010 + +- Removed now-quite-stale SSHv1 overflow detection, as it's more prone + to false positives than useful detection (Vern Paxson). + + +1.5.2.4 Fri Jun 4 16:02:11 PDT 2010 + +- Bug fixes for terminating connections (Tyler Schoenke and Vern Paxson). + + +1.5.2.3 Wed Mar 24 18:23:57 PDT 2010 + +- Bug fixes for --enable-int64 and for avoiding bogus statistics / + bad memory references when generating profiling information upon + exit (Vern Paxson). + + +1.5.2.2 Tue Jan 12 12:33:42 PST 2010 + +- Broccoli compiler warning fixes (Kevin Lo). + + +1.5.2.1 Sun Jan 10 16:59:01 PST 2010 + +- Bug fix for Active Mapping support (Kevin Lo). + + +1.5.2 Sat Dec 26 18:38:37 PST 2009 + +- Portability fixes for --enable-int64 (Vern Paxson). + + +1.5.1 Fri Dec 18 15:17:12 PST 2009 + +- Due to a Python configuration problem, the original 1.5 distribution + did not include the BroControl component, which also introduced a + portability problem for CentOS. These issues have now been fixed (Robin + Sommer and Vern Paxson). + + +1.5 Wed Dec 16 21:28:47 PST 2009 + +- Bro now comes with a new framework, BroControl, for managing an + operational Bro setup, including support for installation, configuration, + and maintainance tasks such a log archival and mail notification. The + framework transparently supports both traditional standalone setups as + well as cluster installations in which multiple Bro boxes coordinate to + analyze a high-volume network link. + + See aux/broctl/README for more information about BroControl. + + Note, BroControl supersedes the older BroLite system, which is no longer + supported and has been deprecated for a while now. + +- Numerous adjustments to DPD = dynamic protocol detection (Robin Sommer): + + o The Analyzer::ProtocolViolation?() method can now be passed the + offending data (which POP3, SMTP, and FTP now do). This information + is added to the "reason" string passed to the script level. + + o SMTP now more accurately reports violations. + + o FTP stops processing when client & server successfully negotiate + an AUTH scheme (leading to subsequent encryption). + + o Analyzer::ProtocolViolation() is virtual, and + TCP_ApplicationAnalyzer() overrides it to not report violations + for any partial connections, because very likely these arise just + due to the analyzer getting confused. + + o TCP::IsPartial() returns true if any side did not start with + a SYN packet (used to be just be for the originator). + + o The connection_state_remove handler in conn.bro now has a higher + &priority so that other handlers for the same event can use + determine_service() and see any changes it performs. + + o DynDisable:max_volume specifies a volume limit (default 10K). + Once a connection exceeds this limit, further protocol + limitations will neither raise ProtocolViolation notices nor + cause the analyzer to be disabled. + + o The event engine no longer raises protocol_violation events for + TCP connections which had gaps, as these have proven too unreliable. + (Note that, ideally, the *analyzers* should avoid reporting + protocol_violations when they can't reliably parse a connection + anymore after a gap; but many don't.) + +- A set of new script functions provide support for incrementally computing + MD5 checksums (Seth Hall). + + md5_hash_init(index: any): bool + Initializes an incremental hashing instance. "index" is + a value of arbitrary type, used to identify this particular + instance (you can have multiple concurrent instances by + using different index values). Returns T on success, + F on failure (such as the index is already in use). + + md5_hash_update(index: any, data: string): bool + For the given hashing instance, updates the hash + based on the given data. Returns T on success, F on + failure (such as the index has not been initialized). + + md5_hash_finish(index: any): string + Returns the MD5-printable hash for the given index + and terminates the instance, or the string "" if the + index was not active. + +- Bro now supports a believed-to-be-robust mechanism for estimating the + proportion of traffic that it failed to capture ("measurement drops"), + which can arise due to overload in either Bro itself, the kernel's + packet filter, or problems with the link tapping mechanism (Vern Paxson). + The event engine can generate estimates for either live traffic or what + was previously recorded in a trace file, though traces subject to some + forms of selective omission (such as skipping over parts of a connection + to reduce storage) can lead to erroneous values. + + The estimates are based on observing gaps in TCP data streams, and + come in two forms: the rate at which such gaps appear, and the relative + volume of data missing due to the gaps. (We've found however that the + volume-based estimator is not robust due to occasional packets with + incorrect sequence numbers, so this estimator is off by default.) + + The easy way to get the estimates is to load capture-loss.bro. + By default, it generates a CaptureLossSummary notice upon Bro's exit, + which can look like: + + 1130222759.344066 CaptureLossSummary estimated rate = 0.00089124 / 0.000970997 (events/bytes) + + If the estimated loss is none, however, it suppresses this notice, + unless you redef CaptureLoss::summary_if_none to T. + + You can also get finer-grained access by defining a "gap_report" + event handler and redef'ing gap_report_freq to a non-zero interval + (such as "10 sec"). This event allows you to pinpoint regions in + time that exhibit significant capture loss. See capture-loss.bro + for an example of a handler for this event. + + Finally, these changes include a number of fixes to Bro's + ack_above_hole/content_gap analysis, which is now significantly + more robust. + +- GeoIP support now supports ASN lookups via the built-in + function lookup_asn(a: addr): count (Scott Campbell and Seth Hall). + +- The GeoIP built-in's lookup_location() and lookup_asn() now + support IPv6 (Seth Hall). Note, the current GeoIP distribution + doesn't include any IPv6 databases, so for now these won't succeed, + but the hooks are in place for when databases become available. + +- lookup_location() now falls back back to the country database if + the city database isn't available (Seth Hall). + +- The new SuccessfulPasswordGuessing Notice is generated when a host + has been seen attempting password guessing (currently only for FTP + sessions) and then successfully logs in (Royal Chan). You can control the + threshold for such reports in terms of how many attempts the host must + have made by redef'ing the variable password_guessing_success_threshhold, + which defaults to 20. + +- The new script http-detect-passwd.bro analyzes the Web items returned + for fetches that appear to be accessing the passwd file (Akhil Dhar). + It generates a PasswordFullFetch Notice if it appears that the item + includes a full password file, and PasswordShadowFetch if it looks like + a shadowed password file. + +- The new built-in + + system_env(cmd: string, env: table[string] of string) + + works like system(), but puts the table entries into the environment + before invoking the command (Robin Sommer). Each in the table + creates an environment variable of the form "BRO_ARG_", whose + value is the corresponding table entry. + +- The new script function + + execute_with_notice(cmd: string, notice_info) + + executes "cmd" with an environment containing the fields of the + notice_info, i.e., the information associated with a Notice (Robin Sommer). + Per the new system_env() function above, the environment variables appear + as "BRO_ARG_", where is the field tag as it appears in + notice.log when you enable use_tagging. + +- The new built-in enable_raw_output(file) acts the same as + the attribute &raw_output (Seth Hall). + +- The new built-in file_opened(f: file) event is generated any time Bro + opens a script-level file (Justin Azoff). You can use this, for example, + if you want to ensure that a given file has a prelude in it such as + human-readable headers, even when the file is rotated. + +- The notice_info record has a new field + + aux: table[string] of string &optional + + which you can use for information specific to a given type of notice + (Robin Sommer). Entries in $aux appear as "aux_" tags in notice.log. + +- Another new notice_info record field is the boolean do_alarm (default=T), + which, if set to F, overides a notice action otherwise specifying to + generate an alarm (Robin Sommer). In other words, if do_alarm is F, no + alarm will be generated independent of the notice action. + + This is a work-around for the fact that we can't specify more than one + action. In particular, we couldn't NOTICE_DROP but then *not* alarm, + which we now can by returning NOTICE_DROP yet setting do_alarm to F. + +- The notice_info record field $dropped now appears in the tagged output + format if true (Robin Sommer). + +- NOTICEs relating to scan detection now no longer include the connection + that triggered the notice, as it really doesn't contain any useful + information, given that the particular trigger simply depends on the + detection algorithm and its parameters (Robin Sommer). However, we do + explicitly set $p (port number) in the notice, and also $n with the + number of attempts. + +- drop.bro now hardwires a Catch-and-Release redrop after seeing one + connection from a previously-dropped-but-already-released host + (Robin Sommer). + +- drop.bro now provides some new hooks (Robin Sommer): + + event address_dropped(a: addr) + Generated when an address has been dropped. + + event address_restored(a: addr) + Generated when connectivity to an address has been restored, + such as using the Catch-and-Release mechanism. + + event address_cleared(a: addr) + Generated when an address that was dropped in the past is + no longer being monitored looking for new connections + (as part of the Catch-and-Release mechanism). + +- The new built-in function + + hexdump(data_str: string) : string + + returns a hex dump representation of the given input data (Christian + Kreibich). The dump renders 16 bytes per line, with hex on the left and + ASCII (where printable) on the right. + +- Bro's notion of when a TCP connection begins now dastes to the first + instance of an initial SYN packet seen, rather than the last (Gregor Maier). + +- The Time Machine script tm-contents.bro now generates + + event contents_saved: event(c: connection, orig_file: string, + resp_file: string) + + when the content of a connection has been completely saved to disk + (Robin Sommer). + +- The mime.bro script now exports the MIME header callback table, and also + marks it as &redef'able so you can modify its entries (Matthias Vallentin). + The mime_log file is also now exported. + +- A new signature file, policy/sigs/http-bots.sig, contains signatures + to detect some of the current HTTP based controlled bot families (Seth Hall). + +- The signature engine's HTTP pattern matching has been fixed (Seth Hall) + to align with the documentation at: + + http://www.bro-ids.org/wiki/index.php/Reference_Manual:_Signatures#Content_conditions + + In particular, the content condition "http" is now referred to as + "http-request" (though "http" still works for backward compatibility), + "http-request-header" and "http-reply-header" now provide access to + headers seen in only one direction, and similarly for "http-request-body" + and "http-reply-body". (This latter is still accessible as "http-body" + for backwards compatibility.) + +- The new script variable max_remote_events_processed: count (default 10) + sets a limit on the number of remote events processed in each round, + before tending to other inputs (Robin Sommer). + +- If you set the new script variable dump_used_event_handlers to T, + then on startup Bro dumps out all of the event handlers that the + loaded set of scripts can invoke (Matthias Vallenti). + +- Summaries for DNS PTR scanning now use a separate Notice, + DNS_PTR_Scan_Summary, rather than overloading DNS_PTR_Scan (Robin Sommer). + +- scan.bro now provides a table skip_dest_server_ports: set[addr, port] + which lists servers (defined as an address and a port) excluded from + scan detection computations (Craig Leres and Jay Krous). + +- When redefining values on the command line directly (using var=value), + quotation marks are now implicit only if "var" is a variable of type + string (Christian Kreibich). This allows other string-like values + (such as enum's) to be passed as well. + +- scan.bro now explicitly loads conn.bro so that it can itself + be loaded independently (Robin Sommer). + +- login.bro depends on scan.bro (because of tracking authentication + "scans"), so now it explicitly loads it (Vern Paxson). + +- UDP_datagram_length_mismatch is now by default flagged just once per + originating host rather than once per connection, as it can generate + tons of messages (Vern Paxson). + +- Removed now-long-boring flagging of access to Solaris "listen" + service as "hot" (Vern Paxson). + +- Removal of libedit, since libreadline provides similar functionality + (Christian Kreibich). + +- Added scripts missing from distribution: dce.bro, ncp.bro, and smb.bro + (Vern Paxson). + +- ssh.bro now exports ssh_ports (Seth Hall) + +- A number of improvements to inter-Bro communication (Robin Sommer). + + (1) Remote communication now no longer includes location information for + serialized objects; that removes quite a bit of redundacy from the network + traffic. + + (2) The new option 'remote_check_sync_consistency" disables the cross-check + on the receiving side of &synchronized state of whether the current value + of a variable has the value expected by the sender. Transmitting the + original values in addition to the updates generates quite a bit CPU & + network load in some cases (in particular, a table of tables). The default + for remote_check_sync_consistency is off, and so far that in particular + seems to reduce the proxy's load quite a bit. + + (3) Complete overhaul of the internal caching of serialized objects. The + objective of the caching is avoid retransmitting already sent values over + and over again. It turns out, however, that some objects are very stable + and hardly change or get replaced (e.g., Bro types); while other change + all the time and are hardly reused some time later (e.g., Vals). Now + we maintain *two* caches independently for these types of objects; one + with a low turn-over one and another with a high one. This should reduce + CPU load on both sender and receiver sides. + + The new scheme is only used if both communicating Bros support it; with + older Bros, as well as with Broccoli, we continue using the old scheme. + +- Some reworking of remote printing (Robin Sommer), as follows. Bro now + uses a new interprocess message rather than print_hook events, to better + manage buffering and associated load (these can produce failures depending + on system configuration; see remote.log). A number of timeouts and + buffer sizes have been tuned. Internally, EINTR errors are now treated + separately from EAGAIN. Finally, even with remote_check_sync_consistency=F, + one type of consistency check was still being done; this is no longer + the case. + +- The DNS analyzer now generates events (dns_query_reply/dns_rejected) + for replies with zero questions (Robin Sommer). + +- Perftools support for incompatible changes in the 1.0 API (Robin Sommer). + +- Rearranged (generally reducing, though not always) some state timeouts + associated with scan detection (Robin Sommer). In addition, when a + scanning address crosses ignore_scanners_threshold (meaning that it will + be ignored from now on anyway), it gets discarded from all state-tracking + tables. Finally, the ignore_scanners_threshold now applies all kinds + of scans, not just address scans. + +- Substantial Broccoli updates, including a new initialization requirement + that breaks backward compatibility, support for enqueueing serialized + event data for transmission, and OpenSSL threadsafe initialization. + See aux/broccoli/ChangeLog for details (Christian Kreibich, Robin + Sommer, and Matthias Vallentin). + +- Broccoli hashtable optimisation. See aux/broccoli/ChangeLog for + details (Christian Kreibich & Matthias Vallentin). + +- Broccoli memory leak fixed, see aux/broccoli/ChangeLog for details + (Christian Kreibich). + +- Broccoli: updates to bropipe tool (Steve Chan and Robin Sommer). + +- Bug fixes for Broccoli Python bindings (Robin Sommer and Matthias Vallentin). + +- Fixed nasty bug due to module scoping that completely kept stepping-stone + detection from working (Vern Paxson). + +- A serious bug in the packet sorter has been fixed (Robin Sommer). + +- Bug fix for extra NULs getting embedded in escaped strings (Seth Hall). + +- Bug fix for HTTP messages that use "Connection: close" rather than length + headers, which yielded erroneous reassembled messages with \r\n's when + only \n's were present (Bernhard Ager). + +- Fix for reporting on ICMP flows that are expired from the flow table + (Vern Paxson). Previously there was a race condition if the flow + was flushed prior to its summary timer expiring. + +- The -l option (list the scripts that Bro loads) now correctly prints + scripts loaded by the prefix mechanism, and uses indentation to indicate + the load hierarchy (Robin Sommer). + +- A bug has been fixed (really, worked around) in drop.bro that prevented + dropped addresses from being properly restored (Robin Sommer). + +- Fixes for deadlocking problems in the Broccoli protocol. See + aux/broccoli/ChangeLog for details (Christian Kreibich & Robin Sommer). + +- Bug fix for DNS analyzer on 64-bit machines (Gregor Maier). + +- Bug fix for asynchronous DNS lookups to prevent some successful lookups + being reported as timed out (Robin Sommer). + +- Bug fix for tracking line numbers associated with compound statements + (Po-Ching Lin). + +- Fix for a rare condition in which the main Bro process couldn't kill + its child process (Robin Sommer). + +- Fix for file rotation when the underlying file is deleted before the + timer expires (Robin Sommer). + +- Fix for potential crash when communication connections break down, + and also for releasing cached objects (Robin Sommer). + +- Fix for default table entries computed by function invocation to not + cache previous results (Robin Sommer). + +- Fix for Bro's internal DNS resolution (Scott Campbell and Robin Sommer). + +- Portability fix for DAG packet capture (Gregor Maier). + +- Portability fix for --enable-brov6 (Robin Sommer). + +- Portability fixes for FreeBSD (Vern Paxson). + +- A work around for new_packet() crashing on IPv6 packets (Vern Paxson). + For now, IPv6 packets are skipped. Also, for fragments the event handler + is now only called for the fully reassembled packet. + +- The new configuration option --disable-nbdns supports disabling non-blocking + DNS at configure time (Sean McCreary). Note, there are some known problems + with it in some environments. + +- A number of configuration fixes and enhancements (Christian Kreibich + and Robin Sommer). + +- Consistency nit for the configuration process (Seth Hall). + +- A number of reference-counting and other memory management fixes + (Robin Sommer). + +- Bug fix for inter-Bro communication lockup (Seth Hall and Robin Sommer). + +- Bug fix for computing TCP payload length in new_packet event (Lothar Braun). + +- Bug fix for sending boolean True values via Broccoli (Seth Hall). + +- make distcheck fix to clean up .bif.bro files (Christian Kreibich). + +- Bug fix for DPD's recognition of SSLv2 connections (Seth Hall). + +- Bug fix for &default for tables indexed by subnets (Seth Hall). + +- A bug has been fixed that could crash Bro when you called get_event_peer() + after a remote connection had already disppeared (Robin Sommer). + +- Introduced a work-around for crashes that occur when Bro exits + due to handling a signal (Robin Sommer). + +- Bug fix for checkpoint.bro - don't schedule timers for times that + aren't actually in the future (Robin Sommer). + +- Hostname formatting fix for anon.bro (Fabian Schneider). + +- Bug fix for redundant .log extension in Time Machine log file + (reported by CS Lee). + +- Removed now-outdated special-casing of Linux reporting of packet filter + statistics (Peter Wurzinger and Robin Sommer). + +- A number of memory leaks fixed (Robin Sommer). + +- Addressed warnings from newer versions of g++ (Robin Sommer and Vern Paxson). + +- Fixed an invocation issue in the ca-create script that prevented it from + working with recent OpenSSL versions (Craig Leres & Christian Kreibich). + +- Comment fixed in drop-adapt (Justin Azoff). + +- Duplicate code removed from Val (Seth Hall). + + +1.4 Fri Oct 17 11:08:52 PDT 2008 + +- We are no longer supporting a previous Bro release as the "stable" + version. Rather, the model now is that the current public release will + aim for increasing stability (occasionally updated with fixes), and those + who wish to use a "bleeding-edge" snapshot can do so via access to the + public SVN source code repository, as explained at + + http://bro-ids.org/wiki/index.php/Subversion#Public_Access + + Note that all previous releases remain available from the download page; + what is changing is that we no longer commit to support for the most + recent of these. + +- We have clarified the copyright statement that covers most of the + code to remove the "advertising clause" that derived from older + BSD licenses, and we have removed copyright wording from most source + code files. See COPYING for the current wording and a list of + files that retain their own copyright notices. + +- Bro now supports analyzing NetFlow v5 data, i.e., from Cisco routers + (Bernhard Ager). NetFlow can be useful for intrusion detection as it + allows analysis of traffic from many different points in the network. + Bro can now read NetFlow data from a UDP socket, as well as (mostly + for debugging purposes) from a file in a specialized format. You can + create these files with the programs given in aux/nftools. + + Command line switches: + + -Y|--netflow :[=] | read flow from socket + + This is the usual way of getting NetFlow data into Bro by + opening a UDP socket on : and reading all incoming + packets. Setting the to 0.0.0.0 should work on most + platforms. Optionally you may set an identifier for the + source - useful if there are many different sources you want + to analyze in parallel. This might also be necessary if you + want to use this feature with a clustered Bro. + + Examples: + bro -Y 0.0.0.0:5555 netflow + bro -i eth0 -Y 10.0.0.1:1234=src1 brolite netflow + + -y|--flowfile [=] + + Used to read from a file. You can optionally include an + identifier for the source. + + Examples: + bro -y myflowfile netflow + bro -y myflowfile=src1 otherflowfile=src2 netflow + + Netflow Events: + + event netflow_v5_header(h: nf_v5_header) + + Generated upon reading a new NetFlow PDU, as summarized in the + argument. The field h_id gives the flow source identifier and + a serial number. You can use this field to associate subsequent + netflow_v5_record events with their header. + + event netflow_v5_record (r: nf_v5_record) + + Every record within a NFv5 PDU generates a corresponding + netflow_v5_record() event. The relatively complex timestamp + format of NFv5 is already converted to Bro's time type, and + the TCP header flags are separated into bools. + + The distribution includes an example analysis script, netflow.bro. + It simply dumps received NetFlow records. If netflow_restitch is T + (the default), then Bro performs flow restitching as well, and two + script variables become relevant: + + global netflow_finished_conn_expire = 310 sec &redef; + + specifies how long to wait for additional flow records after + a RST or FIN for + + const netflow_table_expire = 31 min; + + Its setting only affects table declarations, and therefore + cannot be usefully redef'd. + + Auxiliary programs: + + Bro uses a custom format for flow data stored in files, + to enable preserving timestamps of the PDU arrivals and the + exporter's IP address. The tools nfcollector and ftwire2bro + in aux/nftools/ provide ways to manipulate the Bro NF file + format. The first dumps NetFlow data from a UDP socket to + stdout or to a file in Bro format. The second converts NetFlow + data in "wire" format to Bro format, and, while doing so, + fakes up the exporter's IP address and timestamp. You can get + "wire" format from normal flow-tools files, e.g., by using + 'flow-export -f 4'. Please note that the Bro format is just + a hack to allow for easier debugging. Therefore the format + is not in fact platform independent, and not suitable for data + storage. + +- A new DHCP analyzer generates the following events (Po-Ching Lin): + + event dhcp_discover(c: connection, msg: dhcp_msg, req_addr: addr) + event dhcp_offer(c: connection, msg: dhcp_msg, mask: addr, + event dhcp_request(c: connection, msg: dhcp_msg, + event dhcp_decline(c: connection, msg: dhcp_msg) + event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr, + event dhcp_nak(c: connection, msg: dhcp_msg) + event dhcp_release(c: connection, msg: dhcp_msg) + event dhcp_inform(c: connection, msg: dhcp_msg) + + where dhcp_msg values look like: + + type dhcp_msg: record { + op: count; # 1 = BOOTREQUEST, 2 = BOOTREPLY + m_type: count; # the type of DHCP message + xid: count; # transaction ID of a DHCP session + h_addr: string; # hardware address of the client + ciaddr: addr; # original IP address of the client + yiaddr: addr; # IP address assigned to the client + }; + + See dhcp.bro for the corresponding analysis script (which could + probably use some refinements). + + Note, this analyzer is implemented using BinPAC, so you will need + to specify --use-binpac to activate it. + +- A BitTorrent analyzer is now available (Nadi Sarrar). See the policy + scripts bittorrent.bro and bt-tracker.bro for the events generated for + analyzing transfers and tracker dialogs, respectively. + +- The "Bro Lite" configuration is now deprecated and will not in + general be supported (Robin Sommer & Vern Paxson). + +- "make install" now only installs a core set of files (Robin Sommer). + Policy files are now installed in /share/bro/* (or whatever + configure determines $datadir to be), which is now in Bro's default + search path. It creates a directory /share/bro/site for local + policy files, and the default BROPATH is extended to include this. The + default path no longer includes policy/local. You can install the + additional files used by the (now deprecated) "Bro Lite" configuration + using "make install-brolite". + +- Substantial updates to Broccoli, including support for container + types (tables and sets) as well as a new metadata structure for event + callbacks, facilitating truly generic event handler implementations + (Christian Kreibich, Seth Hall and Robin Sommer). See aux/broccoli/ChangeLog + for details. + +- Extensive changes to allow Bro to process packets captured in the + past intermingled with those captured in real-time (Matthias Vallentin + and Robin Sommer). This operation reflects combining Bro with use of + "Time Machine" functionality for packet capture. + +- We have unfortunately had to disable support for configuring Bro + to use ClamAV, since it turns out that the key interface we need + for processing blocks of memory directly rather than whole files + is no longer supported by the package, and in fact was buggy even + when it was (Robin Sommer). + +- The new signature option "http-body //" matches + on the body data of HTTP entities (Robin Sommer). The matching is + done after decompressing the body, if necessary. + +- The new built-in function identify_data(data: string, return_mime: bool) + analyzes the string "data" and returns its type according to libmagic, + if installed (Seth Hall). The second argument controls whether it should + be returned as a MIME-type or just an identifying string. For example, + identify_data("MZpofigu", F) returns the string "MS-DOS executable", and + print identify_data("MZpofigu", T) returns "application/x-dosexec". + +- The new analysis script http-identified-files.bro identifies the + type of items returned by Web servers using libMagic (if available) + and generates notices for interesting types and mismatches between + URLs and types (Seth Hall). + + You configure it using two variables. watched_mime_types is a pattern + (default /application\/x-dosexec/ | /application\/x-executable/ ) for + which any MIME type matching the pattern generates a HTTP_WatchedMIMEType + notice. + + mime_types_extensions is a table mapping strings to patterns specifying + how URLs for the given MIME type should appear. (Ideally, this would + be a table mapping patterns to patterns, but Bro doesn't currently support + that.) It defaults to: + + ["application/x-dosexec"] = /\.([eE][xX][eE]|[dD][lL][lL])/ + + i.e., do Windows executables end in .exe or .dll. + + You can also redef the pattern ignored_urls to specify URLs that should + not generate complaints. It defaults to matching Windows Update. + +- The new script http-extract-items.bro extracts the items from HTTP + traffic into individual files (Vern Paxson). Files are named: + + .._._. + + where is a redef'able prefix (default: "http-item"), is a + number uniquely identifying the item, the next four are describe the + connection tuple, and is "orig" if the item was transferred + from the originator to the responder, "resp" otherwise. + +- The workings of how Bro interfaces to external programs for dropping/ + restoring connectivity of misbehaving hosts has been significantly + reworked (Brian Tierney and Robin Sommer). + + First, dropping decisions used to be made directly by analyzer scripts, + such as scan.bro directly calling drop_address(). Now instead the + scripts generate Notices and then the notice policy can have an + action of NOTICE_DROP to codify that the response to the given Notice + is to drop the source. The new notice_action_filter of drop_source + drops the source of notices, and drop_source_and_terminate both + drops the source and terminates the corresponding connection. + + So, to drop all sources triggering a specific notice, one can now, e.g., + write: + + redef notice_action_filters += { [Hot::SSH_Overflow] = drop_source }; + + Related to this change, notice_info has a new field $dropped, set to + true if the Notice triggered a (successful) drop. + + Second, by redef'ing Drop::use_catch_release to T (default F) you can + activate "catch-and-release" logic. You use this mode when you need to + manage a limited number of possible blocks, or to build in automatic + "forgiveness" in situations where blocked sources might become benign + (such as due to dynamic IP addresses). If a source has been idle for + Drop::drop_time, then it is unblocked. However, if it is again seen as + block-worthy, then it is blocked for an interval of Drop::long_drop_time. + + Third, ICMP scanning is now reported by its own notice, ICMPAddressScan, + rather than Scan::AddressScan. + +- Google's perftools have replaced mpatrol for leak-checking and + heap-profiling (Robin Sommer). If Bro is compiled with --enable-perftools + and configure finds the perftools, there are two command-line options + available: + + -m turns on leak checking of the main packet loop, with some + uninteresting leaks are suppressed. Currently, with one + exception (the RPC analyzer; problem not yet found), it reports + no leaks when running the test suite. + + -M turns on heap profiling: Bro will take a snapshot of the heap + before starting the main packet loop and another one when + finished. These snapshots can then be analyzed with pprof. + + For more information about the perftools see + + http://code.google.com/p/google-perftools + +- Notice tags are now generated in a pseudo-unique fashion that, with high + probability, ensures that tags generated by separate Bro processes don't + clash when logged to a common location, such as for a Bro cluster (Robin + Sommer). Tags are now string's rather than count's, and are associated + with all notices, not just that are connection-related. You can however + redef the string notice_tag_prefix or the function new_notice_tag to + further control how such tags are generated. + +- Four new built-ins for type conversion (Robin Sommer): + + function double_to_interval(d: double): interval + function addr_to_count(a: addr): count + function port_to_count(p: port): count + function count_to_port(c: count, t: transport_proto): port + +- Many policy scripts have been modified to use modules & scoping + (Robin Sommer and Matthias Vallentin), which may require updates to + existing scripts/refinements. + +- The new script variable dpd_conn_logs (default F), if true, changes the + semantics of the service field in connection logs written to conn.log, + as follows (Robin Sommer). It becomes a comma-separated list of analyzers + confirmed by DPD to parse the connection's payload. If no analyzer could + confirm its protocol, but the connection uses a well-known port, the + service is the name of the port with "?" appended (e.g., "http?"), as + long as the corresponding analyzer has not declined the connection. + In addition, ftp-data sessions are labeled "ftp-data" and portmapper + connections are labeled with the specific method-call (just as before). + + dpd_conn_logs defaults to F because the change in semantics may break + scripts that parse conn.logs; but it will likely change to the default + in the future. With dpd_conn_logs turned off, conn logs are generated + as they used to be, with a few rare exceptions (with previous versions, + the service field was sometimes determined while the connection was still + alive; now it's always determined at the time when the conn.log entry + is written out). + +- The SSL analyzer has been rewritten using BinPAC, with a number of + robustness improvements (Tobias Kiesling). It currently is only used + if you execute with --use-binpac. + +- Python bindings for Broccoli are now available in + aux/broccoli/bindings/python/ (Robin Sommer). See README/README.html + in that director for details. + +- The new "auth" option in remote.bro indicates whether a given side is + considered "authoritative" for shared state, in which case it sends its + initial state to &sync'ed peers (Robin Sommer). When two peers synchronize + their state, one side sends its current set of state to the other as + soon as the remote connection is established. The one sending the state + used to be the one who has been running longer; now it can also be + explicitly set via the "auth" flag in the Remote::Destination. + +- Two new tuning parameters for scan.bro (Robin Sommer): + + ignore_scanners_threshold (default 0): + + If a host has scanned more than this many hosts, it is completely + excluded from further scan detection. 0 disables. + + addr_scan_trigger (default 0): + + A host is only tracked for address scanning once it has contacted + this many different hosts. Primarily intended for using a two-stage + scan detection with a Bro cluster: first, each node searches locally + for scanners by looking for hosts contacting more than + addr_scan_trigger destinations. Those hosts which do are then + globally tracked throughout the cluster by &synchronizing the scan + detector tables. + +- When Bro serializes functions, it now does so by default using only + their name, rather than their full value (Robin Sommer). This prevents + propagation of expiration functions associated with tables and sets. + Note, currently there is no mechanism provided to switch from the + default behavior, but the internal hooks are in place to do so. + +- The new built-in variable trace_output_file gives the name of the -w + output trace file (Robin Sommer). + +- Bro no longer installs new file rotation timers when shutting down + (Robin Sommer). + +- The new policy scripts remote-print-id{,-reply}.bro support convenient + access to printing the identifiers of a remote Bro (Robin Sommer). + You use the script remote-print-id.bro to request and receive the + printing; the remote Bro must have loaded remote-print-id-reply.bro + in order to process the request. + + Example use: + + bro -e 'redef PrintID::dst="" PrintID::id=""' + remote-print-id + +- scan.bro has been heavily modified to better support distributed scan + analysis (Matthias Vallentin and Robin Sommer). + +- The check for unused event handlers is now turned off by default + (Robin Sommer). To enable, use "redef check_for_unused_event_handlers = T". + +- The new script drop.bro has been split off from scan.bro to isolate + the logic concerning dropping addresses to block scans (Robin Sommer). + +- The new -l flag lists each script as it is loaded (Robin Sommer). + +- Textual descriptions of identifiers now include their attributes + (Robin Sommer). + +- The new predefined function prefixed_id() returns a session identifier with + its peer-ID prepended if it's associated with a remote Bro (Robin Sommer). + This is now used when generating writing log files. + +- remote.bro now assigns a priority of -10 to its bro_init() event handler + to allow others a chance to modify destinations (Robin Sommer). + +- A large number of BinPAC updates (Ruoming Pang and Robin Sommer). + +- The new built-in type_name(v): string returns the name of the type + of the value v (Vern Paxson). For example, "typename(5.2)" returns + "double". This function is mainly for internal debugging (i.e., + finding mismatches between values generated by the event engine + versus how their type is expected by the script layer). + +- The new built-in str_shell_escape() does some basic escaping on strings + that will be passed to system() (Christian Kreibich). Note, this function + isn't ready (robust enough) for routine use, however. + +- The new built-in disable_print_hook(file) acts the same as + the attribute &disable_print_hook (Robin Sommer). + +- The new script terminate-connection.bro factors out the terminate_connection() + functionality that used to be in conn.bro (Robin Sommer). + +- The new attribute &group= can be associated with event handlers + to group them together into a set that can be manipulated as a whole + (Robin Sommer). is a string reflecting the name given to the group. + + The built-in enable_event_group(group: string) turns on all the analyzers + in a given group, and disable_event_group(group: string) deactivates them. + +- The new attribute &raw_output applies to variables of type file, disabling + escaping of non-printable characters (Seth Hall). + +- You can now iterate over the characters in a string value using + a "for" loop, e.g., "for ( c in str ) ..." (Robin Sommer). + +- The new built-in + + function cat_sep%(sep: string, def: string, ...%): string + + works similarly to cat(), except that it (a) separates the values + by "sep" and (b) substitutes "def" for empty strings (Seth Hall). + +- The function string_escape() now takes a string of characters to escape + rather than a single character (Robin Sommer). Each character in the + string is preceded by '\' in the return value (also any embedded '\'s, + as before). + +- The new built-in function global_ids() returns a table of all global + identifiers along with associated information (Robin Sommer). The + return value has type table[string] of script_id, indexed by the name + of the identifier and yielding records with the following fields: + + type script_id: record { + type_name: string; + exported: bool; + constant: bool; + enum_constant: bool; + redefinable: bool; + value: any &optional; + }; + +- The new script function find_last(str: string, re: pattern) returns + the last occurrence of the given pattern in the given string, or + an empty string if no match (Robin Sommer). Note that this function + returns the match that starts at the largest index in the string, which + is not necessarily the longest match. For example, a pattern of /.*/ + will return just the final character in the string. + +- The new script variable record_all_packets, if redef'd to T (default F), + instructs Bro to record every packet it processes (Robin Sommer). + Prior to introducing this variable, Bro applied a few heuristics to + reduce recording volume. Setting this variable also causes packets + to be recorded very early in processing, which can be helpful for + debugging crashes. + +- If the new script flag ssl_log_ciphers is set to T (default), ssl.bro + logs the ciphers seen (Robin Sommer). + +- Much more expanded Time Machine support, now located in + policy/time-machine/ (Robin Sommer), + +- The new command line option --status-file (alias -U) specifies + the name of a file into which Bro will write an indicator of its current + processing status (Robin Sommer). Possible values include "INITIALIZING", + "RUNNING", "TERMINATING", "TERMINATED". + +- The new policy script targeted-scan.bro looks for repeated access from + the same source to the same server, to detect things like SSH + password-guessing attacks (Jim Mellander). + +- The "alternative" style for printing strings (i.e., a fmt() argument + of "%As") now renders the raw string, other than escape-expanding + embedded NULs (Vern Paxson). This change may be temporary, pending + development of more fine-grained control over string rendering. + +- For now we have removed the %S functionality for fmt() (Robin Sommer). + %S was meant to print "raw" strings, but later processing of such + printing still introduces artifacts. + +- GeoIP information now includes latitude and longitude (Seth Hall). + +- ssh.bro now supports the variable skip_processing_after_handshake + which directs the event engine to omit any further processing of an + SSH connection after its initial handshake (Seth Hall and Robin Sommer). + This can help with performance for large file transfers but precludes + some kinds of analyses (e.g., tracking connection size). This change + also adds a scope of "SSH". + +- Email notification of notices now allows for separate destinations + depending on notice type (in particular, a regular mail destination + versus a pager destination), and also escapes the notice to prevent + injection attacks (Seth Hall and Robin Sommer). + +- The new policy script conn-flood.bro is a simple connection-flooding + detector, mainly meant as a demonstration (Robin Sommer). + +- A large number of additions to the TLS/SSL known-ciphers suite (Seth Hall). + +- Serialization now uses 64-bit IDs to cache items rather than 32-bit, + for robustness during long-running execution (Robin Sommer). + +- The new script variable tcp_max_initial_window specifies, for flows + for which ACKs have never been seen, the maximum volume of initial + data after which Bro will assume that it is seeing only one side + of the connection and will not buffer data for consistency checking + awaiting the later arrival of ACKs (Robin Sommer). It defaults to 4 KB. + (Note, this used to be an internal value, so the behavior is not new.) + Set to 0 to turn off this functionality and have Bro attempt to + track all such flows. + +- The new script variable tcp_max_above_hole_without_any_acks specifies, + for flows for which ACKs have never been seen, the maximum volume of + data above a sequence hole that Bro will tolerate for a connection + before giving up on tracking the flow (Robin Sommer). It defaults to 4 KB. + (Note, this differs from tcp_max_initial_window in that this threshold + applies to sequence holes rather than the beginning of flows. Like + tcp_max_initial_window this used to be an internal value.) Set to 0 to + turn off this functionality. + +- The new script variable tcp_excessive_data_without_further_acks specifies + a threshold similar to tcp_max_above_hole_without_any_acks, but for + flows for which Bro has seen ACKs (Robin Sommer). It defaults to 10 MB. + Set to 0 to turn off the functionality. + +- Equal signs ("=") in text for notices are now escaped when using the + tagged format to keep them unambiguous from the "=" delimiters + (Robin Sommer). + +- The final tallies for notices are now processed as NoticeTally + NOTICE's rather than directly alarm'd (Robin Sommer). + +- WeirdActivity notices now include an associated connection when appropriate + (Robin Sommer). + +- Support for large (> 2^32 bytes) pcap trace files (Po-Ching Lin). + +- Scoped names ("...::...") are now allowed in signature "eval" + constructs (Christian Kreibich). + +- scan.bro is now decoupled from conn.bro, i.e., you can @load the + latter without getting the former (Vern Paxson). As part of this + change, the logic to invoke TRW is now in scan.bro. + +- weird.bro has been updated with a number of missing Weird's (Vern Paxson). + +- If when using inter-Bro communication the child Bro process terminates, + it now also terminates the parent process (Robin Sommer). + +- BinPAC analyzers now interoperate with DPD (Robin Sommer). + +- Some http.bro processing options are now exported so they can be + accessed in other scripts (Robin Sommer). + +- SMTP analysis now applies to port 587/tcp as well as 25/tcp (Robin Sommer). + +- $conn is now set in ServerFound notices (Robin Sommer). + +- You can now create empty sets and tables using set() and table(), + i.e., the usual set/table constructors with no arguments (Vern Paxson). + By themselves, these have an unspecified type - you can't use them + directly other than to assign them. For example, + + local bad_guys: set[addr]; + ... + bad_guys = set(); # start over assuming no bad guys + +- A number of scripts have been (slightly) simplified to use the + new empty set()/table() constructors (Vern Paxson). Note that + these still aren't usable for field assignments in record constructors, + nor for attributes like &default = ... + +- Removed unused syntax for declaring sets based on a list of initial + values (Vern Paxson). + +- set() and table() can now be used as arguments to function calls + (Vern Paxson). + +- The vestigial &match attribute has been removed. + +- POP3 is now recognized using Dynamic Protocol Detection (Seth Hall). + +- The new event expected_connection_seen(c: connection, a: AnalyzerTag) + is generated whenever a connection is seen for which we have previously + scheduled an analyzer via expect_connection() (Robin Sommer). + +- The new built-in capture_state_updates logs all changes applied to + &synchronized variables, in a fashion similar to the capture_events() + built-in (Robin Sommer). An accompanying policy script, + capture-state-updates.bro, turns this on to the file state-updates.bst. + +- If the new script variable suppress_local_output is set (default: F), + Bro suppresses printing to local files if there's a receiver for + print_hook events (Robin Sommer). This option is however ignored + for files with a &disable_print_hook attribute. + +- The new notice action filter function file_if_remote specifies + that notices from sent from remote source addresses should + have an action NOTICE_FILE (Robin Sommer). + +- The new notice action filter function file_local_bro_notices specifies + that notices generated by the local Bro instance (as opposed to a + remote peer) should have an action NOTICE_FILE (Robin Sommer). + +- An arbitrary tag can now be past to post-processors for log rotation + (Robin Sommer). + +- Default inactivity timeouts for interactive services shortened to + 1 hour (Robin Sommer). + +- The scanning variables distinct_{peers,ports,low_ports} are now + redef'able (Robin Sommer). + +- The new -S (--summary-only) option for site-report.pl directs to + only generate connection summaries (Brian Tierney) + +- More useful default config file for edit-brorule.pl (Brian Tierney). + +- Bro now includes a test suite in testing/istate/ for its "independent + state" functionality (Robin Sommer). + +- Support for parallel builds via make -j (Christian Kreibich). + +- Bro's default search path now includes includes policy/sigs/ and + policy/time-machine/ (Robin Sommer). + +- Bro's internal processing of interprocess communication has been + significantly overhauled to prevent potentially fatal race conditions + (Robin Sommer). + +- Bro now checks calls to fmt() at compile-time to ensure that the + correct number of arguments are present (Vern Paxson). This is useful + in addition to Bro's run-time checking for arguments matching their + corresponding format-specifiers in the case of rarely-executed statements + that might not generate such run-time checks in routine testing. + +- The ports associated with Telnet and Rlogin are now redef'able (Robin Sommer). + +- MIME processing now removes leading whitespace from MIME headers + (Sanmeet Bhatia and Robin Sommer). + +- TCP "weird" events reported by the connection compressor now match + (other than a few rare corner-cases) those produced for normal TCP + processing (rmkml and Robin Sommer). + +- Added Scan::suppress_UDP_scan_checks to control false positives + on scan detection in environments with P2P protocols that use UDP + (Vern Paxson). + +- The internal analyzer interface now includes an EndOfData() method that + analyzers can use to report that all of a message has been delivered + (Robin Sommer). + +- Fix for a significant memory leak in processing UDP when using -w + (Robin Sommer). Note: this change turns off by default trace rewriting + for generic UDP traffic. + +- Two serious regular expression bugs fixed (Vern Paxson). In the + first, searching for a regular expression inside a string would + fail if the pattern occurred only after an embedded newline. In + the second, insufficient buffer was allocated when compiling regular + expressions, leading to memory corruption. + +- Base64 decoding bug fixes (Christian Kreibich and Ruoming Pang). + +- Automatic rotation of files is now disabled for contents files written + by the TCP reassembler, which otherwise leads to mangled files + (Robin Sommer). + +- Bro now ships with an updated version of libpcap (0.9.8), which hopefully + fixes problems managing trace files > 4 GB in size. + +- Significant bug fixes for gzip- and deflate-encoded Web items (Robin Sommer). + +- Bug fix for secondary-filter.bro (Vern Paxson). + +- Removed a naming ambiguity regarding TCP states (Vern Paxson). + +- Bug fix for signature scanner not matching all of its input (Vern Paxson). + +- Bug fix for using port values in signatures (Robin Sommer). + +- Minor policy script tweaks: state management for weird's, processing + of Notice tags associated with connections, and dependencies for + irc-bot.bro (Robin Sommer). + +- aux/ portability fixes (Vern Paxson). + +- Workarounds added for a BinPAC deficiency, which is that code in %cleanup + clauses can also be executed during recovery from exceptions when parsing + new data. This means that any delete's or Unref()'s need to also set the + corresponding pointer to nil (Vern Paxson). + +- Bug fix for crashes with the non-BinPAC SSL analyzer (Robin Sommer). + +- Tweak to peer-status.bro since Bro now requires events to be + declared prior to reference in a "schedule" statement (Robin Sommer). + +- The signature keyword "enable" now optionally accepts the syntax + "foo:bar" to specify "activate analyzer bar as a child of analyzer foo" + (Robin Sommer). This is used for example for an XML-over-HTTP analyzer + that's in the works. + +- irc-bot-syslog.bro now uses open_log_file() for its log file (including + the logging suffix) rather than a direct open (Vern Paxson). + +- Bug fix for tracking Blaster across a Bro Cluster (Robin Sommer). + +- Bug fix for the HTTP BinPAC analyzer chopping the trailing character + off of HTTP headers when generating the http_all_headers event (Gregor Maier). + +- Bug fix for HTTP chunked items for which the chunk size line was terminated + by CRLF but the CR and LF came in separate packets (Gregor Maier). + +- A bug has been fixed that would cause partial lines (for line-oriented + protocols) to fail to be processed when a connection terminated + (Robin Sommer). + +- Bro no longer treats a signal arriving before a previous signal has + been processed as fatal, nor does it attempt processing of a termination + signal if seemingly there are no race conditions to worry about + (Robin Sommer). Both of these changes are an attempt to improve + Bro's robustness. + +- Fix for attributes such as &encrypt not working in initial declarations + but only in later redef's (Seth Hall and Robin Sommer). + +- Fixes for memory leaks in SSL processing (Seth Hall and Robin Sommer). + +- Fix for POP3 analyzer to not treat lines like "." as message + terminators (Robin Sommer). + +- Bug fix for crashes arising from nil pointers in list expressions + (Seth Hall and Robin Sommer). + +- Bug fix: a signature's "enable" would activate the corresponding analyzer + even if no event handlers were defined for it (Robin Sommer). + +- Bug fixes to prevent crashes when mixing set_contents_file() with + subsequent explicit close(), and to ensure all data written to + file upon connection tear-down (Gert Doering and Robin Sommer). + +- Configuration support for MacPorts and Fink package management systems + (Christian Kreibich & Vern Paxson). + +- Communication-only Bro's now send out email alarms (Robin Sommer). + +- Writes to a file that fail due are now run-time errors rather than + fatal internal errors, since often these occur due to the disk + being full (Robin Sommer). + +- Byte-order bug fix for lookup_location() (Robin Sommer). + +- BinPAC portability fix for 64-bit machines (Bernhard Ager and Robin Sommer). + +- Portability fixes for newer versions of gcc (Jan Gerrit Goebel and + Robin Sommer). + +- Some support for porting to Solaris (Stephan Toggweiler). + +- Connection compressor bug fix for source and destination having the + same IP address, such as when monitoring loopback (Robin Sommer). + +- Connection compressor bug fix for connections with multiple SYNs + (Robin Sommer). + +- Bug fix for using already-declared local variables for looping + over vectors in a "for" loop (Robin Sommer & Vern Paxson). + +- Bug fix for not processing truncated UDP packets (Tom Kho and Robin Sommer). + +- Bounds-check added to BinPAC-generated code (Tom Kho and Robin Sommer). + +- Bug fix for checking whether an IPv6 address is part of a subnet + (Seth Hall). + +- Bug fixes for crashes relating to asynchronous DNS lookups performed + at start-up (Robin Sommer). These changes also lowered the timeout + before assuming failure from 20 seconds down to 5 seconds. + +- Portability and const-ness fixes (Kevin Lo and Robin Sommer). + +- Suppression of some content-gap complaints when running on traces + that have been filtered down to only TCP control packets (Robin Sommer). + +- Removed unnecessary dependency in notice-action-filters.bro + that led to errors when loading icmp.bro by itself (Vern Paxson). + +- Bug fix for potential infinite loop in client communiation (Robin Sommer). + +- Bug fix in reference counting that could eventually lead to roll-over + (Robin Sommer). + +- Bug fix in communication initialization (Robin Sommer). + +- Internal documentation fix: timers are specified using absolute time, + not relative (Robin Sommer). + +- Performance improvement for built-in find_all() function when running + on large strings (Robin Sommer). + +- Memory leak fixes (Robin Sommer, Bernhard Ager, Christian Kreibich). + +- Bug fix for error recovery when encountering an unknown link layer + (Bernhard Ager). + +- Bug fix for reversing client & server in a connection (Po-Ching Lin). + +- Bug fix for packet_contents when capture length exceeds the IP payload + length due to Ethernet frame padding (Christian Kreibich). + +- Bug fix for tcp_packet event erroneously including Ethernet padding + in its contents (Vern Paxson). + +- Bug fix for lookup_connection built-in (Seth Hall). + +- Portability nit for libedit tarball (Vern Paxson). + +- Broccoli portability fix for NetBSD (Christoph Leuzinger). + +- Type-checking for script-level event invocation was completedly broken - + now fixed (Vern Paxson). + +- Portability fixes for different versions of g++/STL (Nicholas Weaver + and Vern Paxson). + +- Fix for dynamic detection of SSL via DPD (Robin Sommer). + +- IPv6 portability fix for BinPAC-based DNS analyzer (Vern Paxson). + Note, more portability work is needed for it. + +- Bug fix for bifcl error messages (Vern Paxson). + +- Minor bug fix for remote communication, plus some improved communication + logging (Robin Sommer). + +- Bug fix for &printhook (Robin Sommer). + +- Bug fix for error message output (Robin Sommer). + +- Bug fix for termination cleanup (Robin Sommer). + +- Bug fix for some Rlogin corner cases (Robin Sommer & Vern Paxson). + +- Bug fix for bifcl generation of "interval" types (Vern Paxson). + +- Bug fix for getting connection memory statistics when Bro is + exiting (Robin Sommer). + +- Config fix: --enable-debug now turns off -O2 for gcc (Robin Sommer). + +- Bug fixes for "heavy" analysis (Vern Paxson). + +- Broccoli bug fixes for types net and port (Robin Sommer). + +- Bug fixes for Telnet environment options (Robin Sommer). + +- Bug fix for accessing remote peer description (Robin Sommer). + +- A fix for the connection compressor generating new_connection too + late (Robin Sommer). + +- Fixes for DAG support, including configuration and multiple + interfaces (Robin Sommer). + +- Bug fix for serializing time-stamps of table entries (Robin Sommer). + +- Bug fix for dealing with peer IDs for remote communication (Robin Sommer). + +- Bug fix to avoid installing timers when timers have already + been canceled (Robin Sommer). + +- Bug fix for interplay between serializing connections and + connection compressor (Robin Sommer). + +- Memory leak fix for enum's (Robin Sommer). + +- Bug fix for files being closed prior to bro_done() (Vern Paxson). + +- aux/broccoli/contrib was not included in distribution (Robin Sommer). + +- Auto-configuration bug fix for BinPAC (Craig Leres). + +- Bug fix for dynamic protocol detection (Robin Sommer). + +- A number of configuration fixes for installation and portability + (Christian Kreibich, Brian Tierney, Robin Sommer, Dan Kopecek). + + +1.3 Mon Jul 16 22:11:00 PDT 2007 + +- The Bro manual has been wikified at: + + http://www.bro-ids.org/wiki/index.php/User_Manual + + and this is the format in which it will evolve in the future + (Christian Kreibich). + +- Much more extensive support for SMB, NetBIOS and NCP (Chris Grier). + +- The new attribute &priority=n defines the order of execution for handlers + of the same event (Robin Sommer). Handlers with higher priority are + executed first. n is an integer expression that must evaluate to a + constant when the script is loaded. + + Example: + > cat foo.bro + event bro_init() &priority = -5 { print -5; } + event bro_init() &priority = 5 { print 5; } + event bro_init() { print 0; } # default priority=0 + > ./bro foo.bro + 5 + 0 + -5 + + The connection_state_remove() handler in conn.bro now has priority + -10 and therefore executes after all other handlers for this event. + This fixes a long-standing problem of sometimes $addl fields not showing + up in connection summaries. + +- The new expressions record(...), table(...), set(...) and vector(...) + are constructors for the corresponding aggregate types (Vern Paxson). + For example, + + record($foo = "hi", $bar = -6) + + is the same as the existing constructor + + [$foo = "hi", $bar = -6] + + For tables, sets, and vectors, the "..." values within the ()'s have + the same syntax as those that you can list in variable initializations. + For example, + + table([1, T] = "black", [4, F] = "red") + + returns a table of type "table[count, bool] of string". + + set(4, 3, -1) + + is a value of type "set[int]". + +- You can associate attributes with table() and set() constructors + (Robin Sommer). For example: + + local s = set(1.2.3.4) &read_expire = 5 secs; + + associates a 5-second read expiration with the set assigned to s. + +- Bro now explicitly supports port numbers reflecting a transport protocol + type of "unknown" (Christian Kreibich). Currently, this means "not TCP, + UDP or ICMP". The numerical value of such a port is the IP protocol, + so ranges from 0..255. For example: + + global p: port = 0/unknown; + + print fmt("%s", p); + print fmt("p is TCP? %s", get_port_transport_proto(p) == tcp); + print fmt("p is unknown? %s", + get_port_transport_proto(p) == unknown_transport); + + yields + + 0/unknown + p is TCP? F + p is unknown? T + + In comparisons of different protocol types, the following holds: + unknown < TCP < UDP < ICMP. + +- If your system supports "GeoIP" (see http://www.maxmind.com/app/geolitecity + for a corresponding city database), then the new script function + + lookup_location(a: addr): geo_location + + returns a record of geographic information associated with an address + (Seth Hall). The geo_location record has $country_code, $region and + $city fields. If no information is available, each of these will be + set to empty strings. + + If Bro hasn't been configured with GeoIP support, or if the address is + IPv6 that cannot be directly converted to IPv4, then Bro produces a + run-time error and likewise returns empty strings. + +- Signature-matching on HTTP components now processes the URI with + escape sequences expanded (Robin Sommer). Ideally, there would be + two signature keywords, one for decoded URIs (corresponding to this + case) and one that allows matching against the URI as originally + transmitted. + +- The connection compressor is no longer considered experimental, and + is used by default (Robin Sommer). + +- The new function lookup_hostname(host: string): addr_set asychronously + looks up the IPv4 address(es) of the given host via DNS (Robin Sommer). + Like lookup_addr(), this function can only be used within a "when" + statement. + +- The new built-in + + raw_bytes_to_v4_addr(s: string): addr + + takes a string that points to at least 4 bytes, and returns an address + corresponding to interpreting these as being an IPv4 address in network + order (Vern Paxson; suggested by Mike Dopheide). + +- Trace-rewriting support for DNS, SMB (Chris Grier). + +- The new script function find_all(str: string, re: pattern): string_set + returns a string_set giving all occurrences of the pattern "re" in + the string "str" (Robin Sommer). (Note that string_set's are unordered.) + +- The new policy script save-peer-status.bro generates a log + to peer_status.$BRO_LOG_SUFFIX of updates received from + communication peers (Robin Sommer). + +- The policy script print-filter.bro now includes two (scoped) variables, + terminate_bro and to_file, which control whether to exit after printing + the filter (default T) and whether to write to the log file + pcap_filter.$BRO_LOG_SUFFIX or (default) to stdout (Robin Sommer). + +- The new script variable check_for_unused_event_handlers controls whether + Bro checks for unused event handlers (Robin Sommer). It defaults to T, + which was the past behavior (always report). + +- Bro now terminates if the only pending activity is future timers + (Robin Sommer). It used to wait for those timers to expire, but this + can cause fundamental problems if the timers are associated with table + management (since these might never completely drain). + +- Tables and sets inside of records are now initialized to empty + values rather than uninitialized (Vern Paxson). + +- A new variable allow_services_from (in hot.bro) complements the + existing allow_service_to variable (Brian Tierney). It specifies + that access to the given service from the given originator is + allowed. + +- global_sizes() no longer reports internal variables (Robin Sommer). + +- The IRC analyzer is now activated if any of the (many) IRC event + handlers are defined (Robin Sommer). + +- The default value for tcp_close_delay is now 5 sec rather than 0 sec + (Robin Sommer). This prevents some spurious connection events. + +- Improved logic for dealing with "reversed" connections such + as backscatter (Vern Paxson). + +- You can now left-justify fields when using fmt() with "%-" like + in sprintf (Christian Kreibich). + +- Updates to DNS query types (Larry Leviton). + +- Added mechanism to http-header.bro to skip printing some HTTP headers + (Larry Leviton). + +- The IrcHotWord notice now sets the associated connection (Robin Sommer). + +- If a notice has a tag, it's no longer overridden (Robin Sommer). + +- ServerFound notices now set the port field (Robin Sommer). + +- The built-in lookup_ID() now returns the string "" if the + ID does not exist, rather than a run-time error (Robin Sommer). + +- The new tuning option ProtocolDetector::suppress_servers specifies a + set of analyzers for which Bro generates ServerFound notices, but not + ProtocolFound (Robin Sommer). This both reduces log file size and + conserves memory. + +- A new notice_action_filter, tally_notice_type_and_ignore, works the same + as tally_notice_type but returns IGNORE (Robin Sommer) + +- Setting summary_interval == 0 disables the creation of irc-bots.summary.log + (Robin Sommer). + +- If you @load foo and a directory "foo" is in your path, Bro no longer + tries to load it (Robin Sommer). + +- A number of BinPAC fixes and enhancements (Ruoming Pang, Chris Grier + and Vern Paxson). + +- BinPAC now resides in aux/binpac rather than src/binpac (Ruoming Pang + and Christian Kreibich). This reflects a decoupling of it from Bro so + that it can be used to generate protocol analyzers for other projects too. + +- Removed example Inktomi entries from skip_scan_sources initialization, + since they no longer exist (Vern Paxson). + +- The variable make notice_once_per_orig_tally_interval is now + redef'able (Brian Tierney). + +- SIGPROF to the communication child process now logs resource stats to + remote.log (Matthias Vallentin). + +- The new built-in getpid(): count returns Bro's process ID (Robin Sommer). + +- Patterns for detecting IRC-based bots updated (Robin Sommer). + +- irc-bot-syslog now logs just bots, not all IRC client/servers (Robin Sommer). + +- The new variable suppress_notice_actions in notice.bro suppresses + notice_actions events for selected notice types (Robin Sommer). + +- Files opened during operation now rotate just like those opened at + startup (Robin Sommer). + +- ResourceStats now also logs elapsed time and the reported number of + packets-on-the-link (Mark Dedlow). + +- Printing a "file" value now produces its name (Robin Sommer). + +- Removed deliberate truncation of payload in port 80 FIN packets + (Vern Paxson). + +- remote.log now includes received peer_descriptions (Robin Sommer). + +- Significant POP3 analyzer speed-ups (Vern Paxson). + +- Updated README (Vern Paxson). + +- Fix for "@load a" followed by "@load a.bro" not loading the same file + twice (Robin Sommer). + +- Bug fixes for propagating state operations to uninitialized variables + and for spurious state inconsistency messags (Robin Sommer). + +- Bug fix for sending final sync-points during pseudo-realtime mode + (Robin Sommer). + +- Fix for possible buffer overflow (Christian Kreibich). + +- Bug fix for spurious end-of-file's during inter-Bro communication + (Robin Sommer). + +- Bug fix for dpd_match_only_beginning=F (Robin Sommer). + +- Bug fix for updating timestamps (Christian Kreibich). + +- Bug fix for skipping ADU processing in adu.bro (Christian Kreibich + and Zhichun Li). + +- Fix for ICMPs that carry ICMP headers (or non-TCP/UDP/ICMP headers) + within them (Vern Paxson). + +- Fix for files being rotated after the timer queue has been deleted + (Vern Paxson). + +- Bug fix for signature-matching with IPv6 subnets (Vern Paxson). + +- Bug fix for connection compressor setting connection origin (Robin Sommer). + +- Bug fix for interconn.bro when processing peculiar connections (Vern Paxson). + +- Fix for off-by-one buffer in sscanf call (Christian Kreibich). + +- Fixed inefficiency/warning flagged by g++ (Vern Paxson). + +- Bug fix for NUL string termination in SMB processing (Zhichun Li). + +- Fix for over-ref'ing of file Val's (Vern Paxson). + +- Fixes for some g++ warnings (Christian Kreibich, Vern Paxson). + +- gcc 3.4.2 portability fixes (Robin Sommer). + +- Minor build fixes for Broccoli, including a version bump to match that + of Bro. See aux/broccoli/ChangeLog for details. + +- distcheck fixes (Christian Kreibich). + +- Configuration portability fixes (Matthias Vallentin, Jean-philippe Luiggi). + +- OpenBSD portability fixes (Jean-philippe Luiggi, Christian Kreibich). + + +1.2.1 Mon Dec 11 16:22:58 PST 2006 + +- Fixed delayed triggering of new_connection events when using the + connection compressor. + +- Fixed tracking of first packet in TCP analyzer. (Reported by Guohan Lu) + +- The syslog built-in got lost during some previous merge. + +- Fixed crash if local variable is given as timeout value for table. + (Reported by Mike Wood.) + +- Fixed using "time" values as table indices. + +- Added ssh to default brolite DPD configuration. + +- Fixed catching up to real-time in case of lull. + +- Fixed Broccoli "BRO_DATA_FORMAT_VERSION" to match version in Bro. + +- Fixed Makefile problem in doc directory. + +- Fixed Makefile dependency problem in binpac directory. + +- Added Linux tuning to brolite install script. + +- Modified Makefile to include broccoli/contrib. + +- Adding missing initialization to remote serializer. + +- Minor documentation updates for reference manual and Broccoli. + + +1.2 Tue Oct 17 12:09:49 PDT 2006 + +- Bro now supports DPD, dynamic protocol detection (Robin Sommer, Holger + Dreger, and Michael Mai). With DPD, Bro can analyze protocols regardless + of what port numbers they use: it infers the protocol based on which + application analyzers can parse it without error. Adding this functionality + involved extensive changes to Bro's internals, but also now enables + multiple Bro analyzers to work on the same connection, either concurrently + or one nested inside the other (we have not taken much advantage of this + latter capability yet, but see the FTP events discussed below). + + There are a number of new policy scripts, events, and variables associated + with DPD processing, as follows. + + Scripts: + + You activate DPD by @load'ing dpd.bro. It in turn instructs Bro + to load the signature file policy/sigs/dpd.sig. Note that Bro + uses signatures to expedite deciding which analyzers to try on + a given connection; it does *not* simply use the signatures to + make the determination of which protocol is in use, as this is + insufficiently robust. (At this point, Bro provides signatures + for FTP, IRC, HTTP, SMTP, and SSH. In the future we plan to add + other protocols.) + + Along with dpd.bro, you need to @load detect-protocols.bro or + detect-protocols-http.bro. The former enables general detection + of application-layer protocols, while the latter does further + inspection of HTTP sessions to characterize applications running + on top of HTTP such as Gnutella or SOAP. (Loading dpd.bro + is separate from loading one of these scripts because in principle + Bro could use a different means than signatures to activate + the analyzers, although currently it does not.) + + If you @load dyn-disable.bro, then once an analyzer determines + that it does not match a given connection, it is deactivated + (and a Notice is generated). Otherwise, it still proceeds to try + its best to analyze the connection (to possibly be more robust + against evasion). + + The scripts dce.bro and smb.bro enable DPD for the Windows DCE and + SMB protocols, respectively. (Note that analysis of these protocols + is undergoing a major expansion, not yet complete.) + + Events: + + event protocol_confirmation(c: connection, atype: count, aid: count) + Generated when the given connection has been confirmed as + conforming with the application type (protocol) specified + by atype. aid is a globally unique analyzer ID that identifies + a particular analyzer instance. + + The values for atype are symbolic names associated with + each of Bro's analyzers, such as ANALYZER_IRC. See the + initialization at the beginning of Analyzer.cc for the + full set of names. + + The function analyzer_name(atype: count): string translates + these symbolic names into text. For example, + + analyzer_name(ANALYZER_IRC) + + yields "IRC". + + event protocol_violation(c: connection, atype: count, aid: count, + reason: string) + Generated when the given connection has been found to + violate the protocol of the given application type, with + "reason" giving details. + + Variables: + + dpd_buffer_size: count (default 1024) + Specifies how much pending data Bro keeps for connections + that have not been classified yet. Once this fills, the + data is deleted, though classification can still continue + (see below). + + dpd_match_only_beginning: bool (default T) + If set, specifies that Bro should stop signature matching + if it has processed dpd_buffer_size bytes. + + dpd_ignore_ports: bool (default F) + If set, then Bro does not take into consideration the port + numbers associated with connections when attempting to + classify them (which can otherwise help the process in + some cases). + + dpd_reassemble_first_packets: bool (default T) + If set, then Bro does TCP stream reassembly before applying + signature-matching to detect protocols. + + likely_server_ports: set[port] + Specifies a list of ports that Bro will consider as likely + used by servers. For example, if Bro sees a connection + that has already been established (so it does not know + which side sent the initial SYN), and one side uses a port + in this set, then it will assume that that side is the + server (connection responder). The set is empty unless + you populate it or @load server-ports.bro, which specifies + a large number of values. + + dpd_config: table[AnalyzerTag] of dpd_protocol_config + Specifies the DPD configuration associated with each tag. + The type dpd_protocol_config is simply: + + type dpd_protocol_config: record { + ports: set[port] &optional; + }; + + i.e., an optional $ports field specifying a set of ports + associatd with the tag. For example, ftp.bro now includes + the equivalent of: + + redef dpd_config += { + [ANALYZER_FTP] = [$ports = 21/tcp] + }; + + Functions: + + The function + + expect_connection(orig: addr, resp: addr, resp_p: port, + analyzer: count, tout: interval) + + is called to alert Bro that a new connection is expected, initiated + by orig to a server running on resp's port resp_p (note: orig's port + is not specified) which will correspond to the specified analyzer + (e.g., "FILE", which is used to analyze files transferred by FTP - + see next item). "tout" is a timeout to associate with the waiting. + + The function + + function disable_analyzer(cid: conn_id, aid: count) + + instructs Bro to disable the analyzer that generated the current + event, assuming the analyzer is associated with the given connection + ID. This is used by the dyn-disable.bro script discussed above. + +- A much more complete BinPAC compiler, along with new HTTP, DNS, and + RPC/Portmap analyzers in binpac (Ruoming Pang). The flag "--use-binpac" + activates the BinPAC-based analyzers (currently for HTTP and DNS). + See www.cs.princeton.edu/~rpang/binpac-paper.pdf for a description of + BinPAC, and let Ruoming know if you are interested in using BinPAC to build + new analyzers. + +- A new type of analyzer, FILE, analyzes the contents of a connection as + though it were a data file (Robin Sommer). Currently, it can generate + two events: + + event file_transferred(c: connection, prefix: string, descr: string, + mime_type: string) + Indicates that the connection transferred a file. "prefix" + is the beginning of the file's data; "descr" and "mime_type" + are indicators of the file's type, as reported by the + "libmagic" library. + + descr/mime_type are only set if Bro is configured on a + system that includes the "libmagic" library. + + event file_virus(c: connection, virname: string) + Indicates the connection transferred an executable + corresponding to a known virus of the given name. + + This functionality is only available if Bro is configured + on a system that includes the "libclamav" library. + + Note, this analyzer is enabled via a call to expect_connection by + the FTP analyzer. + +- New events relating to IRC analysis (Robin Sommer): + + event irc_client(c: connection, prefix: string, data: string) + Generated upon seing a client message sent over the given + IRC connection. "prefix" is the command's prefix as defined + by the IRC protocol. It is used by servers to indicate the + true origin of the message; it may be empty. "data" contains + the message. + + event irc_server(c: connection, prefix: string, data: string) + Same for server messages. + + event irc_user_message(c: connection, user: string, host: string, + server: string, real_name: string) + Generated upon seeing an IRC "USER" command. + + event irc_password_message(c: connection, password: string) + Generated upon seeing an IRC "PASS" command. + + event irc_channel_topic(c: connection, channel: string, topic: string) + Generated upon seeing an IRC server reply that includes + the channel topic. + + event irc_global_users(c: connection, prefix: string, msg: string) + Generated upon seeing an IRC server reply that includes + a count of the number of IRC users. + +- The new experimental script irc-bot.bro tracks IRC-based bots (Robin Sommer). + The accompanying script irc-bot-syslog.bro syslog's the state of the + bot analysis every IrcBot::summary_interval seconds (default 1 minute). + +- The new script proxy.bro looks for open Web proxies by matching incoming + requests to a server with outgoing requests it makes (Robin Sommer). It + generates HTTPProxyFound Notices when it finds one. + +- Changes to notices.bro (Robin Sommer): + + - notice_policy_item's now have a default $result of + NOTICE_FILE and a default $priority of 1. + + - The new notice_action_filter, notice_alarm_per_orig, alarms + on the first NoticeType from a specific source. Subsequent + instances are tallied. + + - notice_action_filters now reside in the new script + notice-action-filter.bro (automatically loaded by notice.bro). + + - The notice actions NOTICE_ALARM_PER_CONN, NOTICE_ALARM_PER_ORIG, + and NOTICE_ALARM_ONCE have been removed, as they were never + actually implemented. + + - If the notice_policy returns IGNORE or FILE, the action_filters + filters are no longer consulted. + +- A new attribute for tables and sets, &mergeable, changes the semantics + of assignments, as follows (Robin Sommer). Given two &mergeable tables/sets + A and B, an assignment "A = B" becomes actually a join "A = A \cup B" + (i.e., union). The envisoned use is to help avoid race conditions + when doing remote state synchronization. + +- The semantics of &synchronized expire_funcs has changed (Robin Sommer). + Now, when a table entry is expired and the operation is propagated to a + a peer, the peer will call its expire_function. + +- TRW analysis now skips UDP traffic because it currently treats + all UDP connections as failures (Robin Sommer). + +- trw.bro has been split into trw-impl.bro (the algorithm) and + trw.bro (which simply activates the analysis), to facilitate writing + scripts that have hooks into TRW analysis but don't presume it's + active (Robin Sommer). + +- The option report_remote_notices in remote.bro has been replaced + by a new script you include, remote-report-notices.bro (Robin Sommer). + +- The new function connect_peer() explicitly connects to a remote host + (Robin Sommer). + +- The new script remote-send-id.bro sends the current value of an ID + to a remote Bro and then terminates processing (Robin Sommer). It's + intended for use from the command-line, as in + + bro -e "redef dst="" id="" remote-send-id + + The other scripts must set up the connection. is an index into + Remote::destinations corresponding to the destination. + +- New built-ins {suspend,resume}_state_updates() can be called to + temporarily avoid propagating updates to &sync'ed values (Robin Sommer). + This can avoid duplicated activity. + +- The new function terminate_communication() instructs Bro to end its + communication with remote peers (Robin Sommer). + +- The new event remote_state_access_performed is raised when remote state + access has been performed (Robin Sommer). This is primarily for debugging. + +- The log() built-in has been renamed to ln() to avoid conflict (Vern Paxson). + +- bifcl now generates event generation wrapper functions from event.bif + (Ruoming Pang). For example, to generate event http_reply, currently + one writes: + + val_list* vl = new val_list; + vl->append(BuildConnVal()); + vl->append(new StringVal(fmt("%.1f", reply_version))); + vl->append(new Val(reply_code, TYPE_COUNT)); + if ( reply_reason_phrase ) + vl->append(reply_reason_phrase); + else + vl->append(new StringVal("")); + ConnectionEvent(http_reply, vl); + + In the future, one will be able to just call bro_event_http_reply(), and + the code generated by bifcl looks like: + + void bro_event_http_reply(Connection* c, StringVal* version, + bro_uint_t code, StringVal* reason) + { + val_list* vl = new val_list; + + vl->append(c->BuildConnVal()); + vl->append(version); + vl->append(new Val(code, TYPE_COUNT)); + vl->append(reason); + + mgr.QueueEvent(http_reply, vl, SOURCE_LOCAL, c); + } + + Accompanying this change is a semantic shift to types "string" and "port" + in .bif files. They used to be translated to C++ types BroString* and + uint32, respectively. Now they are translated to StringVal* and PortVal*. + The functions in bro.bif are changed accordingly, and please be aware + of this change when you write built-in functions in future. + + Also for this change, the parameter 'new' for rsh_request has been renamed + 'new_session', as 'new' is a reserved word for C++. + +- Some ICMP "connections" now have services identified ("icmp-echo", + "icmp-unreach") rather than just listing the service as "other" + (Ruoming Pang). + +- The new option remote_trace_sync_interval specifies an interval after + which each Bro will stop processing its trace and wait for all others + to signal that they have reached the same time (Robin Sommer). The + intent is support for operating Bro in a distributed cluster fashion + (and in particular for debugging such clusters when running off-line + on traces). + + This option only works in pseudo-realtime mode, and requires the new + global remote_trace_sync_peers to give the total number of remote peers + (not including self). Signaling is done via a new communication message + type. + +- Extensions for DNS transformation/anonymization, including introduction + of trace transformation for protocols other than TCP (Jason Lee). + Not yet fully developed/debugged. + +- Extensions for HTTP transformation/anonymization (Martin Casado). + Not yet fully developed/debugged. + +- The $conn field is now included in HTTPProxyFound notices (Robin Sommer). + +- Changed service inference algorithm to favor lower-numbered + likely-servers over higher-numbered ones (Vern Paxson). + +- In pseudo-realtime mode, Bro now uses real-time for deciding which + peer should send state (Robin Sommer). + +- Time synchronization for Bro's running on traces in pseudo-realtime mode + added (Robin Sommer). + +- Avoidance of false content gaps improved when sorting packets with + out-of-order timestamps (Ruoming Pang). + +- Packets from the packet sorter are now more robustly drained upon + termination of input (Ruoming Pang). + +- Documentation for deep-copy updated (Christian Kreibich). + +- Nasty fragment reassembly bug fixed (Vern Paxson). + +- Serious bugs in EDNS0 processing fixed (Vern Paxson). + +- Fixed significant misfeature of interconn.bro that stopped all processing + of a connection once it makes a detection (Vern Paxson). + +- Fixes for &read_expire operation across synchronizes tables (Robin Sommer). + +- Fixes for multiple peers exchanging initial &sync state simultaneously + (Robin Sommer). + +- Improvements to graceful termination of Bro when communicating with + remote peers (Robin Sommer). + +- Fix for ICMP analyzer not always generating icmp_sent events + (Robin Sommer). This appears to still need some work, as now + it generates redundant events. + +- Fix for initial exchange of &sync state which could lead to + referencing unknown IDs (Robin Sommer). + +- Fix to scan detection for differing semantics of connection compressor + vs. non-compressor (Robin Sommer). + +- Bug fix for distinguishing regular expression matches of length 0 from + those of length 1 (Ruoming Pang). + +- Fix for SSH version parsing in the presence of content gaps (Robin Sommer). + +- Bug fix for IRC that could lead to crashes (Robin Sommer). + +- Bug fix to refrain from adding new timers when a connection has + already been removed from the connection table (Robin Sommer). + +- Bug fix for packet_contents not including the transport-layer header + (Robin Sommer). + +- Some memory leaks fixed (Robin Sommer). + +- A bunch of portability and distribution problems fixed (Christian + Kreibich, Robin Sommer, Vern Paxson). + + +1.1 Mon May 15 10:50:33 PDT 2006 + +- Bro now supports a "when" statement for taking action upon something + becoming true asynchronously (Robin Sommer). This provides a powerful + new mechanism with numerous applications. + + Syntax: + + when '(' ')' [timeout '{ '}'] + + where the first can be a single statement or a block enclosed + in {}'s, but the set associated with "timeout" must be enclosed in + {}'s (to reduce ambiguities in Bro's grammar). + + Bro executes the first statement when becomes true. If you give + a timeout and the condition has not been satisfied before it expires, Bro + executes the second statement instead. + + A simple example: + + global t: table[addr] of count; + event connection_established(c: connection) + { + local orig = c$id$orig_h; + if ( orig !in t ) + { + t[orig] = 1; + + when ( t[orig] == 5 ) + print fmt("%s has established 5 connections", orig); + timeout 1 hr + { + print fmt("%s has NOT established 5 connections", orig); + delete t[orig]; + } + } + else + ++t[orig]; + } + + Notes: + - The condition may be evaluated more than once, and at arbitrary + times. + + - When the when-body is executed, the condition is guaranteed to be + still satisfied. + + - Expression reevaluation is primarily triggered by modifications + to globals. However, reevaluations do not take place immediately + but potentially at a later point. This means that if we change a + global to a value which would execute the trigger but then change + it back, the change may go unnoticed. + + - Inside the condition you may introduce new locals. For example, + + when ( (local x = foo()) && x == 42 ) ... + + Such an assignment always yields true as its expression value + (but the assignment might be delayed, for example if foo() is + a delayed function call - see below). + + Delaying function calls + ======================= + + Functions called inside the condition of a when-clause may delay their + results until they're ready. This works for both script-level and built-in + functions. + + For script-level functions, there is a new construct, "return ", + to delay a function's result. When used, the function returns at the + time the when-stmt's condition becomes true, and it yields the value + that the when-stmt's body then returns. Toy example: + + global X: table[string] of count; + + function a() : count + { + # This delays until condition becomes true. + return when ( "a" in X ) + { + return X["a"]; + } + timeout 5 min + { + return 0; + } + } + + event bro_init() + { + # Installs a trigger which fires if a() returns 42. + when ( a() == 42 ) { print "Yippie!"; } + + X["a"] = 42; + } + + There's also a new built-in function which can delay + + lookup_addr(host: addr) + + performs asynchronous DNS address->hostname lookups. Example: + + local h; addr; + [...] + when (local name = lookup_addr(h)) { print h, name; } + + See the function gen_hot_notice_with_hostnames() in conn.bro for + a more worked-out example of using the "when" clause to translate the + local address in SensitiveConnection notices to a hostname (contributed + by Brian Tierney). This functionality is activated by redef'ing + xlate_hot_local_addr to T. + + Here is the full evaluation model of a when's condition: + + - The condition may be evaluated more than once, at arbitrary times. + + - It is always fully evaluated, no matter whether some former + evaluation has been suspended by a delaying function call. + + - All function calls which do not delay are always *fully* executed + each time the condition is evaluated. + + - Function calls which delay are only executed *once*; their result is + cached and re-used in the case the condition is evaluated again. + + - The condition is guaranteed to be true when the body is executed + (potentially using cached function results) + +- By default Bro now uses a configuration similar to what used to be + activated using reduce-memory.bro, along with some additional state + timeouts that are new (Robin Sommer and Vern Paxson). This allows for + better state management out-of-the-box, at the cost of some precision + of analysis and resilience to evasion. In particular, the intent is to + move towards being able to run Bro continuously without inexorably growing + the amount of memory used until exhaustion. + + You can access a configuration similar to the previous default state + management settings by loading heavy-analysis.bro. It turns on a + load-prefix of "heavy", so when you load XXX.bro, a file heavy.XXX.bro + will also be automatically loaded if present. Note that, as was the + case for reduce-memory, you need to load heavy-analysis prior to other + files for it to have effect. + +- The new module clear-passwords.bro monitors login/FTP/IRC/POP traffic + for cleartext passwords (Jason Lee). + +- The new script service-probe.bro looks for remote hosts that repeatedly + connect to the same service on local hosts (for a configurable set of + services and connection sizes) in order to detect brute-forcing attacks + such as password-guessing (Jim Mellander). + +- A new ARP analyzer generates three events: + + event arp_request(mac_src: string, mac_dst: string, + SPA: addr, SHA: string, TPA: addr, THA: string); + + event arp_reply(mac_src: string, mac_dst: string, + SPA: addr, SHA: string, TPA: addr, THA: string); + + event bad_arp(SPA: addr, SHA: string, TPA: addr, THA: string, + explanation: string); + + with a corresponding policy script arp.bro (Chema Gonzalez and Vern Paxson). + It writes logs to arp.$BRO_LOG_SUFFIX. It has not been tested much yet. + +- Bro Lite changes (Jason Lee): + - default user for is now user 'bro' + - now uses the correct sysctl on FreeBSD 6 + - now uses the correct Perl path if site-report.pl not installed + into '/usr/local/bro' + - no longer prompts to encrypt email unless you pick to email reports + +- The default Bro Lite install now only checkpoints Bro once a week + (Brian Tierney). + +- Implicit Bro file extensions (such as .bro for policy scripts and .sig + for signatures) are now searched for first rather than only if the + non-extension-version of the file doesn't exist (Vern Paxson). For + example, running "bro -r trace mt" now first searches $BROPATH for + "mt.bro" before searching for "mt", whereas it used to do these in + the other order. + +- There's now a simpler mechanism for redef'ing variables on the command-line + (Christian Kreibich). Any command line arguments of the form = + are now expanded into policy code of the form "redef var=val;", where + is wrapped in quotation marks if the value appears to be a string + and doesn't have quotation marks already. This works with strings with + whitespace such as foo="Hello World"; however, note that it means you + can't use the mechanism to redef an enum value. + +- The Bro distribution now includes (and builds by default) Christian + Kreibich's Broccoli library (Bro C Client Library), which enables programs + to communicate with running Bro's (Christian Kreibich and Jason Lee). + Configure with --disable-broccoli to turn this off. + +- Built-in functions log(x: double): double and exp(x: double): double + which do natural logarithms and their inverses (Jaeyeon Jung). + +- The new built-in function gethostname() returns the local host's name + (Jason Lee & Robin Sommer). + +- The new built-in function reading_traces() returns true if Bro + is reading trace files (Robin Sommer). + +- The new built-ins suspend_processing() and continue_processing() provide + script-level control for instructing the event engine to stop or resume + processing packets (Robin Sommer). This is useful for coordinating + simultaneous processing by multiple Bro's. + +- Email notices are now by default sent via /bin/mail, with "[Bro Alarm]" + in the subject. + +- redef'ing a function now replaces the existing body rather than + supplementing it (Robin Sommer), which was a bug. + +- You can now configure Bro to process encapsulated IP packets either + by setting, as before, a fixed encap_hdr_size (for VLANs), or setting + parse_udp_tunnels to T (Ruoming Pang). For the latter, you specify a + UDP tunnel port using udp_tunnel_port (the previous variable "tunnel_port" + has gone away); or you can leave it set to its default of 0/udp, in which + case Bro will look for IP encapsulated in UDP packets on any port. + +- Added a simple form of profiling based on sampling the work done + per-packet (Vern Paxson). The event engine generates a + + event load_sample(samples: load_sample_info, CPU: interval, dmem: int) + + event every load_sample_freq packets (roughly; it's randomized), where + load_sample_freq defaults to 20. "samples" is simply a set[string]; it + contains the names of the functions, event handlers, and their source + files that were accessed during the processing of the sampled packet, + along with an estimate of the CPU cost of processing the packet and + (currently broken) memory allocated/freed. + +- Bro now includes experimental support for Endace DAG cards (Gregor Maier + and Robin Sommer). To activate, configure with + + --with-DAG=/path/to/dagtool/installation + + and use "dag0" as the network interface. You may need to configure the + card with the dagtools first. In general, if dagsnap works, Bro should + work as well. + +- Log rotation has changed in a number of ways (Mark Dedlow & Robin Sommer): + + * The new variable log_rotate_base_time: string, if defined, + specifies that logs should be rotated at log_rotate_base_time + + i * rotate_interval intervals. Format is as a string in + 24-hour time, "%H:%M", e.g, "12:00". This format may change + in the future to instead be a Bro time type. + + * RotateLogs::date_format can be redefined to change format of + timestamps in rotated files. + + * RotateLogs::build_name() can be redefined to implement an + arbitrary naming scheme for rotated files. + + Note, this code has not been extensively tested. + +- Bro now by default builds a version of malloc bundled with its + distribution (Vern Paxson & Brian Tierney). + +- The syntax for the clone operator now looks like a function call, + "copy(x)" (Vern Paxson). + +- The new flag DNS::logging (default F), if T, disables generation of + dns.log (which is often uninteresting and very large), though it + still performs analysis leading to NOTICEs (Robin Sommer). + +- A new global, hostile_domain_list, has been added to dns.bro which + lists domains to be flagged if A or MX records are queried (Scott Campbell). + +- Added globals dns_skip_all_{auth,addl} to skip all DNS AUTH/ADDL processing + (Vern Paxson). Skipping these is on (true) by default, because such + processing is quite expensive. + +- backdoor.bro now turns off by default some detectors that from experience + have too many false positives, or (such as for HTTP) too many uninteresting + true positives (Brian Tierney). In addition: + + - the module now generates a BackdoorFound notice for each backdoor + + - the new variable dump_backdoor_packets (default F) if set causes + the packet that triggered the backdoor detection to be written to + backdoor-packets/: