Merge branch 'master' of ssh://git.bro-ids.org/bro into topic/gilbert/plugin-api-tweak

CHANGES

@@ -1,4 +1,242 @@
 
+2.3-427 | 2015-02-20 13:49:33 -0800
+
+  * Removing dependency on PCAP_NETMASK_UNKNOWN to compile with
+    libpcap < 1.1.1. (Robin Sommer)
+
+2.3-426 | 2015-02-20 12:45:51 -0800
+
+  * Add 'while' statement to Bro language. Really. (Jon Siwek)
+
+2.3-424 | 2015-02-20 12:39:10 -0800
+
+  * Add the ability to remove surrounding braces from the JSON
+    formatter. (Seth Hall)
+
+2.3-419 | 2015-02-13 09:10:44 -0600
+
+  * BIT-1011: Update the SOCKS analyzer to support user/pass login.
+    (Nicolas Retrain, Seth Hall, Jon Siwek)
+
+    - Add a new field to socks.log: "password".
+    - Two new events: "socks_login_userpass_request" and
+      "socks_login_userpass_reply".
+    - Two new weirds for unsupported SOCKS authentication method or
+      version.
+    - A new test for authenticated socks traffic.
+
+2.3-416 | 2015-02-12 12:18:42 -0600
+
+  * Submodule update - newest sqlite version (Johanna Amann)
+
+  * Fix use of deprecated gperftools headers. (Jon Siwek)
+
+2.3-413 | 2015-02-08 18:23:05 -0800
+
+  * Fixing analyzer tag types for some Files::* functions. (Robin Sommer)
+
+  * Changing load order for plugin scripts. (Robin Sommer)
+
+2.3-411 | 2015-02-05 10:05:48 -0600
+
+  * Fix file analysis of files with total size below the bof_buffer size
+    never delivering content to stream analyzers. (Seth Hall)
+
+  * Add/fix log fields in x509 diff canonifier. (Jon Siwek)
+
+  * "id" not defined for debug code when using -DPROFILE_BRO_FUNCTIONS
+    (Mike Smiley)
+
+2.3-406 | 2015-02-03 17:02:45 -0600
+
+  * Add x509 canonifier to a unit test. (Jon Siwek)
+
+2.3-405 | 2015-02-02 11:14:24 -0600
+
+  * Fix memory leak in new split_string* functions. (Jon Siwek)
+
+2.3-404 | 2015-01-30 14:23:27 -0800
+
+  * Update documentation (broken links, outdated tests). (Jon Siwek)
+
+  * Deprecate split* family of BIFs. (Jon Siwek)
+
+    These functions are now deprecated in favor of alternative versions that
+    return a vector of strings rather than a table of strings.
+
+    Deprecated functions:
+
+    - split: use split_string instead.
+    - split1: use split_string1 instead.
+    - split_all: use split_string_all instead.
+    - split_n: use split_string_n instead.
+    - cat_string_array: see join_string_vec instead.
+    - cat_string_array_n: see join_string_vec instead.
+    - join_string_array: see join_string_vec instead.
+    - sort_string_array: use sort instead instead.
+    - find_ip_addresses: use extract_ip_addresses instead.
+
+    Changed functions:
+
+    - has_valid_octets: uses a string_vec parameter instead of string_array.
+
+    Addresses BIT-924.
+
+  * Add a new attribute: &deprecated. While scripts are parsed, a
+    warning is raised for each usage of an identifier marked as
+    &deprecated.  This also works for BIFs. Addresses BIT-924,
+    BIT-757. (Jon Siwek)
+
+2.3-397 | 2015-01-27 10:13:10 -0600
+
+  * Handle guess_lexer exceptions in pygments reST directive (Jon Siwek)
+
+2.3-396 | 2015-01-23 10:49:15 -0600
+
+  * DNP3: fix reachable assertion and buffer over-read/overflow.
+    CVE number pending. (Travis Emmert, Jon Siwek)
+
+  * Update binpac: Fix potential out-of-bounds memory reads in generated
+    code. CVE-2014-9586. (John Villamil and Chris Rohlf - Yahoo
+    Paranoids, Jon Siwek)
+
+  * Fixing (harmless) Coverity warning. (Robin Sommer)
+
+2.3-392 | 2015-01-15 09:44:15 -0800
+
+  * Small changes to EC curve names in a newer draft. (Johanna Amann)
+
+2.3-390 | 2015-01-14 13:27:34 -0800
+
+  * Updating MySQL analyses. (Vlad Grigorescu)
+     - Use a boolean success instead of a result string.
+     - Change the affected_rows response detail string to a "rows" count.
+     - Fix the state tracking to log incomplete command.
+
+  * Extend DNP3 to support communication over UDP. (Hui Lin)
+
+  * Fix a bug in DNP3 determining the length of an object in some
+    cases. (Hui Lin)
+
+2.3-376 | 2015-01-12 09:38:10 -0600
+
+  * Improve documentation for connection_established event. (Jon Siwek)
+
+2.3-375 | 2015-01-08 13:10:09 -0600
+
+  * Increase minimum required CMake version to 2.8. (Jon Siwek)
+
+2.3-374 | 2015-01-07 10:03:17 -0600
+
+  * Improve documentation of the Intelligence Framework. (Daniel Thayer)
+
+2.3-371 | 2015-01-06 09:58:09 -0600
+
+  * Update/improve file mime type identification. (Seth Hall)
+
+     - Change to the default BOF buffer size to 3000 (was 1024).
+
+     - Reorganized MS signatures into a separate file.
+
+     - Remove all of the x-c detections.  Nearly all false positives.
+
+     - Improve TAR detections, removing old, back up TAR detections.
+
+     - Remove one of the x-elc detections that was too loose
+       and caused many false positives.
+
+     - Improved lots of the signatures and added new ones. (Seth Hall)
+
+  * Add support for file reassembly in the file analysis framework
+    (Seth Hall, Jon Siwek).
+
+     - The reassembly behavior can be modified per-file by enabling or
+       disabling the reassembler and/or modifying the size of the
+       reassembly buffer.
+
+     - Changed the file extraction analyzer to use stream-wise input to
+       avoid issues with the chunk-wise approach not immediately
+       triggering the file_new event due to mime-type detection delay.
+       Before, early chunks frequently ended up lost.  Extraction also
+       will now explicitly NUL-fill gaps in the file instead of
+       implicitly relying on pwrite to do it.
+
+2.3-349 | 2015-01-05 15:21:13 -0600
+
+  * Fix race condition in unified2 file analyzer startup. (Jon siwek)
+
+2.3-348 | 2014-12-31 09:19:34 -0800
+
+  * Changing Makefile's test-all to run test-all for broctl, which now
+    executes trace-summary tests as well. (Robin Sommer)
+
+2.3-345 | 2014-12-31 09:06:15 -0800
+
+  * Correct a typo in the Notice framework doc. (Daniel Thayer)
+
+2.3-343 | 2014-12-12 12:43:46 -0800
+
+  * Fix PIA packet replay to deliver copy of IP header. This prevented
+    one from writing a packet-wise analyzer that needs access to IP
+    headers and can be attached to a connection via signature match.
+    Addresses BIT-1298 (Jon Siwek)
+
+2.3-338 | 2014-12-08 13:56:19 -0800
+
+  * Add man page for Bro. (Raúl Benencia)
+
+  * Updating doc baselines. (Robin Sommer)
+
+2.3-334 | 2014-12-03 14:22:07 -0800
+
+  * Fix compound assignment to require proper L-value. Addresses
+    BIT-1295. (Jon Siwek)
+
+2.3-332 | 2014-12-03 14:14:11 -0800
+
+  * Make using local IDs in @if directives an error. Addresses
+    BIT-1296. (Jon Siwek)
+
+2.3-330 | 2014-12-03 14:10:39 -0800
+
+  * Fix some "make doc" warnings and update some doc tests. (Daniel
+    Thayer)
+
+2.3-328 | 2014-12-02 08:13:10 -0500
+
+  * Update windows-version-detection.bro to add support for 
+    Windows 10. (Michal Purzynski)
+
+2.3-326 | 2014-12-01 12:10:27 -0600
+
+  * BIFScanner: fix invalid characters in generated preprocessor macros.
+    (Hilko Bengen)
+
+  * BIT-1294: fix exec.bro from mutating Input::end_of_data event
+    parameters. (Johanna Amann)
+
+  * Add/invoke "distclean" for testing directories. (Raúl Benencia)
+
+  * Delete prebuilt python bytecode files from git. (Jon Siwek)
+
+  * Add Windows detection based on CryptoAPI HTTP traffic as a software
+    framework policy script. (Vlad Grigorescu)
+
+2.3-316 | 2014-11-25 17:35:06 -0800
+
+  * Make the SSL analyzer skip further processing once encountering
+    situations which are very probably non-recoverable. (Johanna
+    Amann)
+
+2.3-313 | 2014-11-25 14:27:07 -0800
+
+  * Make SSL v2 protocol tests more strict. In its former state they
+    triggered on http traffic over port 443 sometimes. Found by Michał
+    Purzyński. (Johanna Amann)
+
+  * Fix X509 analyzer to correctly return ECDSA as the key_type for
+    ECDSA certs. Bug found by Michał Purzyński. (Johanna Amann)
+
 2.3-310 | 2014-11-19 10:56:59 -0600
 
   * Disable verbose bison output. (Jon Siwek)
@@ -34,6 +272,11 @@
     logged into mysql.log. Supports protocol versions 9 and 10. (Vlad
     Grigorescu)
 
+2.3-280 | 2014-11-05 09:46:33 -0500
+
+  * Add Windows detection based on CryptoAPI HTTP traffic as a 
+    software framework policy script. (Vlad Grigorescu)
+
 2.3-278 | 2014-11-03 18:55:18 -0800
 
   * Add new curves from draft-ietf-tls-negotiated-ff-dhe to SSL

CMakeLists.txt

@@ -2,7 +2,7 @@ project(Bro C CXX)
 
 # When changing the minimum version here, also adapt
 # aux/bro-aux/plugin-support/skeleton/CMakeLists.txt
-cmake_minimum_required(VERSION 2.6.3 FATAL_ERROR)
+cmake_minimum_required(VERSION 2.8 FATAL_ERROR)
 
 include(cmake/CommonCMakeConfig.cmake)
 
@@ -15,6 +15,11 @@ if (NOT BRO_SCRIPT_INSTALL_PATH)
     set(BRO_SCRIPT_INSTALL_PATH ${BRO_ROOT_DIR}/share/bro)
 endif ()
 
+if (NOT BRO_MAN_INSTALL_PATH)
+    # set the default Bro man page installation path (user did not specify one)
+    set(BRO_MAN_INSTALL_PATH ${BRO_ROOT_DIR}/share/man)
+endif ()
+
 # sanitize the Bro script install directory into an absolute path
 # (CMake is confused by ~ as a representation of home directory)
 get_filename_component(BRO_SCRIPT_INSTALL_PATH ${BRO_SCRIPT_INSTALL_PATH}
@@ -175,6 +180,7 @@ include_directories(${CMAKE_CURRENT_BINARY_DIR})
 add_subdirectory(src)
 add_subdirectory(scripts)
 add_subdirectory(doc)
+add_subdirectory(man)
 
 include(CheckOptionalBuildSources)
 

Makefile

@@ -48,12 +48,13 @@ bindist:
 
 distclean:
 	rm -rf $(BUILD)
+	$(MAKE) -C testing $@
 
 test:
 	@( cd testing && make )
 
 test-all: test
-	test -d aux/broctl && ( cd aux/broctl && make test )
+	test -d aux/broctl && ( cd aux/broctl && make test-all )
 	test -d aux/btest  && ( cd aux/btest && make test )
 	test -d aux/bro-aux && ( cd aux/bro-aux && make test )
 	test -d aux/plugins && ( cd aux/plugins && make test-all )

NEWS

@@ -28,11 +28,71 @@ New Functionality
 - Bro now has supoprt for the MySQL wire protocol. Activity gets
   logged into mysql.log.
 
+- Bro's file analysis now supports reassembly of files that are not
+  transferred/seen sequentially.
+
+- Bro's scripting language now has a ``while`` statement::
+
+        while ( i < 5 )
+            print ++i;
+
+  ``next`` and ``break`` can be used inside the loop's body just like
+  with ``for`` loops.
+
 Changed Functionality
 ---------------------
 
 - bro-cut has been rewritten in C, and is hence much faster.
 
+- File analysis
+
+    * Removed ``fa_file`` record's ``mime_type`` and ``mime_types``
+      fields.  The events ``file_mime_type`` and ``file_mime_types``
+      have been added which contain the same information.  The
+      ``mime_type`` field of ``Files::Info`` also still has this info.
+
+    * Removed ``Files::add_analyzers_for_mime_type`` function.
+
+    * Removed ``offset`` parameter of the ``file_extraction_limit``
+      event.  Since file extraction now internally depends on file
+      reassembly for non-sequential files, "offset" can be obtained
+      with other information already available -- adding together
+      ``seen_bytes`` and ``missed_bytes`` fields of the ``fa_file``
+      record gives the how many bytes have been written so far (i.e.
+      the "offset").
+
+- has_valid_octets: now uses a string_vec parameter instead of
+  string_array.
+
+Deprecated Functionality
+------------------------
+
+- The split* family of functions are to be replaced with alternate
+  versions that return a vector of strings rather than a table of
+  strings. This also allows deprecation for some related string
+  concatenation/extraction functions. Note that the new functions use
+  0-based indexing, rather than 1-based.
+
+  The full list of now deprecation functions is:
+
+    * split: use split_string instead.
+
+    * split1: use split_string1 instead.
+
+    * split_all: use split_string_all instead.
+
+    * split_n: use split_string_n instead.
+
+    * cat_string_array: see join_string_vec instead.
+
+    * cat_string_array_n: see join_string_vec instead.
+
+    * join_string_array: see join_string_vec instead.
+
+    * sort_string_array: use sort instead.
+
+    * find_ip_addresses: use extract_ip_addresses instead.
+
 Bro 2.3
 =======
 

VERSION

@@ -1 +1 @@
-2.3-310
+2.3-427

aux/bro-aux

@@ -1 +1 @@
-Subproject commit 977654dc51ab08a2afde32241f108cdb4a581d8f
+Subproject commit 0b713c027d3efaaca50e5df995c02656175573cd

aux/broccoli

@@ -1 +1 @@
-Subproject commit acb8fbe8e7bc6ace5135fb73dca8e29432cdc1ca
+Subproject commit d43cc790e5b8709b5e032e52ad0e00936494739b

aux/broctl

@@ -1 +1 @@
-Subproject commit 39e865dec9611b9b53b609cbc8df519cebae0a1e
+Subproject commit 8c9b87bc73e1ddaa304e3d89028c1e7b95d37a91

aux/btest

@@ -1 +1 @@
-Subproject commit d67d89aaee32ad5edb9068db55d1310c2f36970a
+Subproject commit 93d4989ed1537e4d143cf09d44077159f869a4b2

doc/ext/rst_directive.py

@@ -135,7 +135,10 @@ class Pygments(Directive):
                 # lexer not found, use default.
                 lexer = TextLexer()
         else:
-            lexer = guess_lexer(content)
+            try:
+                lexer = guess_lexer(content)
+            except:
+                lexer = TextLexer()
 
         # import sys
         # print >>sys.stderr, self.arguments, lexer.__class__

doc/frameworks/file_analysis_02.bro

@@ -1,7 +1,7 @@
-event file_new(f: fa_file)
+event file_mime_type(f: fa_file, mime_type: string)
     {
     print "new file", f$id;
-    if ( f?$mime_type && f$mime_type == "text/plain" )
+    if ( mime_type == "text/plain" )
         Files::add_analyzer(f, Files::ANALYZER_MD5);
     }
 

doc/frameworks/intel.rst

@@ -14,32 +14,35 @@ consume that data, make it available for matching, and provide
 infrastructure around improving performance, memory utilization, and
 generally making all of this easier.
 
-Data in the Intelligence Framework is the atomic piece of intelligence
+Data in the Intelligence Framework is an atomic piece of intelligence
 such as an IP address or an e-mail address along with a suite of
 metadata about it such as a freeform source field, a freeform
 descriptive field and a URL which might lead to more information about
 the specific item.  The metadata in the default scripts has been
 deliberately kept minimal so that the community can find the
-appropriate fields that need added by writing scripts which extend the
+appropriate fields that need to be added by writing scripts which extend the
 base record using the normal record extension mechanism.
 
 Quick Start
 -----------
 
-Load the package of scripts that sends data into the Intelligence
-Framework to be checked by loading this script in local.bro::
-
-	@load policy/frameworks/intel/seen
-
 Refer to the "Loading Intelligence" section below to see the format
 for Intelligence Framework text files, then load those text files with
 this line in local.bro::
 
 	redef Intel::read_files += { "/somewhere/yourdata.txt" };
 
-The data itself only needs to reside on the manager if running in a
+The text files need to reside only on the manager if running in a
 cluster.
 
+Add the following line to local.bro in order to load the scripts
+that send "seen" data into the Intelligence Framework to be checked against
+the loaded intelligence data::
+
+	@load policy/frameworks/intel/seen
+
+Intelligence data matches will be logged to the intel.log file.
+
 Architecture
 ------------
 
@@ -58,8 +61,10 @@ manager is the only node that needs the intelligence data.  The
 intelligence framework has distribution mechanisms which will push
 data out to all of the nodes that need it.
 
-Here is an example of the intelligence data format.  Note that all
+Here is an example of the intelligence data format (note that there will be
-whitespace field separators are literal tabs and fields containing only a
+additional fields if you are using CIF intelligence data or if you are
+using the policy/frameworks/intel/do_notice script).  Note that all fields
+must be separated by a single tab character and fields containing only a
 hyphen are considered to be null values. ::
 
 	#fields	indicator	indicator_type	meta.source	meta.desc	meta.url
@@ -69,8 +74,21 @@ hyphen are considered to be null values. ::
 For a list of all built-in `indicator_type` values, please refer to the
 documentation of :bro:see:`Intel::Type`.
 
-To load the data once files are created, use the following example
+Note that if you are using data from the Collective Intelligence Framework,
-code to define files to load with your own file names of course::
+then you will need to add the following line to your local.bro in order
+to support additional metadata fields used by CIF::
+
+	@load policy/integration/collective-intel
+
+There is a simple mechanism to raise a Bro notice (of type Intel::Notice)
+for user-specified intelligence matches.  To use this feature, add the
+following line to local.bro in order to support additional metadata fields
+(documented in the :bro:see:`Intel::MetaData` record)::
+
+	@load policy/frameworks/intel/do_notice
+
+To load the data once the files are created, use the following example
+to specify which files to load (with your own file names of course)::
 
 	redef Intel::read_files += {
 		"/somewhere/feed1.txt",
@@ -85,24 +103,23 @@ Seen Data
 
 When some bit of data is extracted (such as an email address in the
 "From" header in a message over SMTP), the Intelligence Framework
-needs to be informed that this data was discovered and it's presence
+needs to be informed that this data was discovered so that its presence
-should be checked within the intelligence data set.  This is
+will be checked within the loaded intelligence data.  This is
-accomplished through the :bro:see:`Intel::seen` function.
+accomplished through the :bro:see:`Intel::seen` function, however
+typically users won't need to work with this function due to the
+scripts included with Bro that will call this function.
 
-Typically users won't need to work with this function due to built in
+To load all of the scripts included with Bro for sending "seen" data to
-hook scripts that Bro ships with that will "see" data and send it into
+the intelligence framework, just add this line to local.bro::
-the intelligence framework.  A user may only need to load the entire
+
-package of hook scripts as a module or pick and choose specific
+	@load policy/frameworks/intel/seen
-scripts to load.  Keep in mind that as more data is sent into the
+
+Alternatively, specific scripts in that directory can be loaded.
+Keep in mind that as more data is sent into the
 intelligence framework, the CPU load consumed by Bro will increase
 depending on how many times the :bro:see:`Intel::seen` function is
 being called which is heavily traffic dependent.
 
-The full package of hook scripts that Bro ships with for sending this
-"seen" data into the intelligence framework can be loading by adding
-this line to local.bro::
-
-	@load policy/frameworks/intel/seen
 
 Intelligence Matches
 ********************
@@ -111,6 +128,7 @@ Against all hopes, most networks will eventually have a hit on
 intelligence data which could indicate a possible compromise or other
 unwanted activity.  The Intelligence Framework provides an event that
 is generated whenever a match is discovered named :bro:see:`Intel::match`.
+
 Due to design restrictions placed upon
 the intelligence framework, there is no assurance as to where this
 event will be generated.  It could be generated on the worker where
@@ -119,3 +137,7 @@ handled, only the data given as event arguments to the event can be
 assured since the host where the data was seen may not be where
 ``Intel::match`` is handled.
 
+Intelligence matches are logged to the intel.log file.  For a description of
+each field in that file, see the documentation for the :bro:see:`Intel::Info`
+record.
+

doc/frameworks/notice.rst

@@ -271,7 +271,7 @@ script that is generating the notice has indicated to the notice framework how
 to identify notices that are intrinsically the same. Identification of these
 "intrinsically duplicate" notices is implemented with an optional field in
 :bro:see:`Notice::Info` records named ``$identifier`` which is a simple string.
-If the ``$identifier`` and ``$type`` fields are the same for two notices, the
+If the ``$identifier`` and ``$note`` fields are the same for two notices, the
 notice framework actually considers them to be the same thing and can use that
 information to suppress duplicates for a configurable period of time.
 

doc/httpmonitor/file_extraction.bro

@@ -7,18 +7,15 @@ global mime_to_ext: table[string] of string = {
 	["text/html"] = "html",
 };
 
-event file_new(f: fa_file)
+event file_mime_type(f: fa_file, mime_type: string)
 	{
 	if ( f$source != "HTTP" )
 		return;
 
-	if ( ! f?$mime_type )
+	if ( mime_type !in mime_to_ext )
 		return;
 
-	if ( f$mime_type !in mime_to_ext )
+	local fname = fmt("%s-%s.%s", f$source, f$id, mime_to_ext[mime_type]);
-		return;
-
-	local fname = fmt("%s-%s.%s", f$source, f$id, mime_to_ext[f$mime_type]);
 	print fmt("Extracting file %s", fname);
 	Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=fname]);
-	}
+	}

doc/install/install.rst

@@ -35,7 +35,7 @@ before you begin:
 
 To build Bro from source, the following additional dependencies are required:
 
-    * CMake 2.6.3 or greater            (http://www.cmake.org)
+    * CMake 2.8 or greater              (http://www.cmake.org)
     * Make
     * C/C++ compiler
     * SWIG                              (http://www.swig.org)

doc/logs/index.rst

@@ -113,7 +113,7 @@ default, including:
 As you can see, some log files are specific to a particular protocol,
 while others aggregate information across different types of activity. 
 For a complete list of log files and a description of its purpose, 
-see :doc:`List of Log Files <../script-reference/list-of-log-files>`.
+see :doc:`Log Files <../script-reference/log-files>`.
 
 .. _bro-cut:
 

doc/script-reference/attributes.rst

@@ -49,6 +49,8 @@ The Bro scripting language supports the following attributes.
 +-----------------------------+-----------------------------------------------+
 | :bro:attr:`&type_column`    |Used by input framework for "port" type.       |
 +-----------------------------+-----------------------------------------------+
+| :bro:attr:`&deprecated`     |Marks an identifier as deprecated.             |
++-----------------------------+-----------------------------------------------+
 
 Here is a more detailed explanation of each attribute:
 
@@ -230,3 +232,9 @@ Here is a more detailed explanation of each attribute:
             msg: string;
         };
 
+.. bro:attr:: &deprecated
+
+    The associated identifier is marked as deprecated and will be
+    removed in a future version of Bro.  Look in the NEWS file for more
+    explanation and/or instructions to migrate code that uses deprecated
+    functionality.

doc/script-reference/statements.rst

@@ -45,8 +45,11 @@ Statements
 |                            | file                   |
 +----------------------------+------------------------+
 | :bro:keyword:`for`,        | Loop over each         |
-| :bro:keyword:`next`,       | element in a container |
+| :bro:keyword:`while`,      | element in a container |
-| :bro:keyword:`break`       | object                 |
+| :bro:keyword:`next`,       | object (``for``), or   |
+| :bro:keyword:`break`       | as long as a condition |
+|                            | evaluates to true      |
+|                            | (``while``).           |
 +----------------------------+------------------------+
 | :bro:keyword:`if`          | Evaluate boolean       |
 |                            | expression and if true,|
@@ -563,6 +566,36 @@ Here are the statements that the Bro scripting language supports.
     See the :bro:keyword:`return` statement for an explanation of how to
     create an asynchronous function in a Bro script.
 
+.. bro:keyword:: while
+
+    A "while" loop iterates over a body statement as long a given
+    condition remains true.
+
+    A :bro:keyword:`break` statement can be used at any time to immediately
+    terminate the "while" loop, and a :bro:keyword:`next` statement can be
+    used to skip to the next loop iteration.
+
+    Example::
+
+        local i = 0;
+
+        while ( i < 5 )
+            print ++i;
+
+        while ( some_cond() )
+            {
+            local finish_up = F;
+
+            if ( skip_ahead() )
+                next;
+
+            [...]
+
+            if ( finish_up )
+                break;
+
+            [...]
+            }
 
 .. _compound statement:
 

doc/scripting/index.rst

@@ -103,9 +103,9 @@ In the ``file_hash`` event handler, there is an ``if`` statement that is used
 to check for the correct type of hash, in this case
 a SHA1 hash.  It also checks for a mime type we've defined as
 being of interest as defined in the constant ``match_file_types``.
-The comparison is made against the expression ``f$mime_type``, which uses
+The comparison is made against the expression ``f$info$mime_type``, which uses
 the ``$`` dereference operator to check the value ``mime_type``
-inside the variable ``f``.  If the entire expression evaluates to true,
+inside the variable ``f$info``.  If the entire expression evaluates to true,
 then a helper function is called to do the rest of the work.  In that
 function, a local variable is defined to hold a string comprised of
 the SHA1 hash concatenated with ``.malware.hash.cymru.com``; this

man/CMakeLists.txt

@@ -0,0 +1,5 @@
+
+install(DIRECTORY . DESTINATION ${BRO_MAN_INSTALL_PATH}/man8 FILES_MATCHING
+        PATTERN "*.8"
+)
+

man/bro.8

@@ -0,0 +1,164 @@
+.TH BRO "8" "November 2014" "bro" "System Administration Utilities"
+.SH NAME
+bro \- passive network traffic analyzer
+.SH SYNOPSIS
+.B bro
+\/\fP [\fIoptions\fR] [\fIfile\fR ...]
+.SH DESCRIPTION
+Bro is primarily a security monitor that inspects all traffic on a link in
+depth for signs of suspicious activity. More generally, however, Bro
+supports a wide range of traffic analysis tasks even outside of the
+security domain, including performance measurements and helping with
+trouble-shooting.
+
+Bro comes with built-in functionality for a range of analysis and detection
+tasks, including detecting malware by interfacing to external registries,
+reporting vulnerable versions of software seen on the network, identifying
+popular web applications, detecting SSH brute-forcing, validating SSL
+certificate chains, among others.
+.SH OPTIONS
+.TP
+.B <file>
+policy file, or read stdin
+.TP
+\fB\-a\fR,\ \-\-parse\-only
+exit immediately after parsing scripts
+.TP
+\fB\-b\fR,\ \-\-bare\-mode
+don't load scripts from the base/ directory
+.TP
+\fB\-d\fR,\ \-\-debug\-policy
+activate policy file debugging
+.TP
+\fB\-e\fR,\ \-\-exec <bro code>
+augment loaded policies by given code
+.TP
+\fB\-f\fR,\ \-\-filter <filter>
+tcpdump filter
+.TP
+\fB\-g\fR,\ \-\-dump\-config
+dump current config into .state dir
+.TP
+\fB\-h\fR,\ \-\-help|\-?
+command line help
+.TP
+\fB\-i\fR,\ \-\-iface <interface>
+read from given interface
+.TP
+\fB\-p\fR,\ \-\-prefix <prefix>
+add given prefix to policy file resolution
+.TP
+\fB\-r\fR,\ \-\-readfile <readfile>
+read from given tcpdump file
+.TP
+\fB\-y\fR,\ \-\-flowfile <file>[=<ident>]
+read from given flow file
+.TP
+\fB\-Y\fR,\ \-\-netflow <ip>:<prt>[=<id>]
+read flow from socket
+.TP
+\fB\-s\fR,\ \-\-rulefile <rulefile>
+read rules from given file
+.TP
+\fB\-t\fR,\ \-\-tracefile <tracefile>
+activate execution tracing
+.TP
+\fB\-w\fR,\ \-\-writefile <writefile>
+write to given tcpdump file
+.TP
+\fB\-v\fR,\ \-\-version
+print version and exit
+.TP
+\fB\-x\fR,\ \-\-print\-state <file.bst>
+print contents of state file
+.TP
+\fB\-z\fR,\ \-\-analyze <analysis>
+run the specified policy file analysis
+.TP
+\fB\-C\fR,\ \-\-no\-checksums
+ignore checksums
+.TP
+\fB\-D\fR,\ \-\-dfa\-size <size>
+DFA state cache size
+.TP
+\fB\-F\fR,\ \-\-force\-dns
+force DNS
+.TP
+\fB\-I\fR,\ \-\-print\-id <ID name>
+print out given ID
+.TP
+\fB\-K\fR,\ \-\-md5\-hashkey <hashkey>
+set key for MD5\-keyed hashing
+.TP
+\fB\-L\fR,\ \-\-rule\-benchmark
+benchmark for rules
+.TP
+\fB\-N\fR,\ \-\-print\-plugins
+print available plugins and exit (\fB\-NN\fR for verbose)
+.TP
+\fB\-O\fR,\ \-\-optimize
+optimize policy script
+.TP
+\fB\-P\fR,\ \-\-prime\-dns
+prime DNS
+.TP
+\fB\-Q\fR,\ \-\-time
+print execution time summary to stderr
+.TP
+\fB\-R\fR,\ \-\-replay <events.bst>
+replay events
+.TP
+\fB\-S\fR,\ \-\-debug\-rules
+enable rule debugging
+.TP
+\fB\-T\fR,\ \-\-re\-level <level>
+set 'RE_level' for rules
+.TP
+\fB\-U\fR,\ \-\-status\-file <file>
+Record process status in file
+.TP
+\fB\-W\fR,\ \-\-watchdog
+activate watchdog timer
+.TP
+\fB\-X\fR,\ \-\-broxygen
+generate documentation based on config file
+.TP
+\fB\-\-pseudo\-realtime[=\fR<speedup>]
+enable pseudo\-realtime for performance evaluation (default 1)
+.TP
+\fB\-\-load\-seeds\fR <file>
+load seeds from given file
+.TP
+\fB\-\-save\-seeds\fR <file>
+save seeds to given file
+.SH ENVIRONMENT
+.TP
+.B BROPATH
+file search path
+.TP
+.B BRO_PLUGIN_PATH
+plugin search path
+.TP
+.B BRO_PLUGIN_ACTIVATE
+plugins to always activate
+.TP
+.B BRO_PREFIXES
+prefix list
+.TP
+.B BRO_DNS_FAKE
+disable DNS lookups
+.TP
+.B BRO_SEED_FILE
+file to load seeds from
+.TP
+.B BRO_LOG_SUFFIX
+ASCII log file extension
+.TP
+.B BRO_PROFILER_FILE
+Output file for script execution statistics
+.TP
+.B BRO_DISABLE_BROXYGEN
+Disable Broxygen documentation support
+.SH AUTHOR
+.B bro
+was written by The Bro Project <info@bro.org>.

scripts/base/files/unified2/main.bro

@@ -71,11 +71,50 @@ global classification_map: table[count] of string;
 global sid_map: table[count] of string;
 global gen_map: table[count] of string;
 
+global num_classification_map_reads = 0;
+global num_sid_map_reads = 0;
+global num_gen_map_reads = 0;
+global watching = F;
+
 # For reading in config files.
 type OneLine: record {
 	line: string;
 };
 
+function mappings_initialized(): bool
+	{
+	return num_classification_map_reads > 0 &&
+	       num_sid_map_reads > 0 &&
+	       num_gen_map_reads > 0;
+	}
+
+function start_watching()
+	{
+	if ( watching )
+		return;
+
+	watching = T;
+
+	if ( watch_dir != "" )
+		{
+		Dir::monitor(watch_dir, function(fname: string)
+			{
+			Input::add_analysis([$source=fname,
+			                     $reader=Input::READER_BINARY,
+			                     $mode=Input::STREAM,
+			                     $name=fname]);
+			}, 10secs);
+		}
+
+	if ( watch_file != "" )
+		{
+		Input::add_analysis([$source=watch_file,
+		                     $reader=Input::READER_BINARY,
+		                     $mode=Input::STREAM,
+		                     $name=watch_file]);
+		}
+	}
+
 function create_info(ev: IDSEvent): Info
 	{
 	local info = Info($ts=ev$ts,
@@ -113,34 +152,56 @@ redef record fa_file += {
 
 event Unified2::read_sid_msg_line(desc: Input::EventDescription, tpe: Input::Event, line: string)
 	{
-	local parts = split_n(line, / \|\| /, F, 100);
+	local parts = split_string_n(line, / \|\| /, F, 100);
-	if ( |parts| >= 2 && /^[0-9]+$/ in parts[1] )
+	if ( |parts| >= 2 && /^[0-9]+$/ in parts[0] )
-		sid_map[to_count(parts[1])] = parts[2];
+		sid_map[to_count(parts[0])] = parts[1];
 	}
 
 event Unified2::read_gen_msg_line(desc: Input::EventDescription, tpe: Input::Event, line: string)
 	{
-	local parts = split_n(line, / \|\| /, F, 3);
+	local parts = split_string_n(line, / \|\| /, F, 3);
-	if ( |parts| >= 2 && /^[0-9]+$/ in parts[1] )
+	if ( |parts| >= 2 && /^[0-9]+$/ in parts[0] )
-		gen_map[to_count(parts[1])] = parts[3];
+		gen_map[to_count(parts[0])] = parts[2];
 	}
 
 event Unified2::read_classification_line(desc: Input::EventDescription, tpe: Input::Event, line: string)
 	{
-	local parts = split_n(line, /: /, F, 2);
+	local parts = split_string_n(line, /: /, F, 2);
 	if ( |parts| == 2 )
 		{
-		local parts2 = split_n(parts[2], /,/, F, 4);
+		local parts2 = split_string_n(parts[1], /,/, F, 4);
 		if ( |parts2| > 1 )
-			classification_map[|classification_map|+1] = parts2[1];
+			classification_map[|classification_map|+1] = parts2[0];
 		}
 	}
 
+event Input::end_of_data(name: string, source: string)
+	{
+	if ( name == classification_config )
+		++num_classification_map_reads;
+	else if ( name == sid_msg )
+		++num_sid_map_reads;
+	else if ( name == gen_msg )
+		++num_gen_map_reads;
+	else
+		return;
+
+	if ( watching )
+		return;
+
+	if ( mappings_initialized() )
+		start_watching();
+	}
+
 event bro_init() &priority=5
 	{
 	Log::create_stream(Unified2::LOG, [$columns=Info, $ev=log_unified2]);
 
-	if ( sid_msg != "" )
+	if ( sid_msg == "" )
+		{
+		num_sid_map_reads = 1;
+		}
+	else
 		{
 		Input::add_event([$source=sid_msg,
 		                  $reader=Input::READER_RAW,
@@ -151,7 +212,11 @@ event bro_init() &priority=5
 		                  $ev=Unified2::read_sid_msg_line]);
 		}
 
-	if ( gen_msg != "" )
+	if ( gen_msg == "" )
+		{
+		num_gen_map_reads = 1;
+		}
+	else
 		{
 		Input::add_event([$source=gen_msg,
 		                  $name=gen_msg,
@@ -162,7 +227,11 @@ event bro_init() &priority=5
 		                  $ev=Unified2::read_gen_msg_line]);
 		}
 
-	if ( classification_config != "" )
+	if ( classification_config == "" )
+		{
+		num_classification_map_reads = 1;
+		}
+	else
 		{
 		Input::add_event([$source=classification_config,
 		                  $name=classification_config,
@@ -173,32 +242,16 @@ event bro_init() &priority=5
 		                  $ev=Unified2::read_classification_line]);
 		}
 
-	if ( watch_dir != "" )
+	if ( mappings_initialized() )
-		{
+		start_watching();
-		Dir::monitor(watch_dir, function(fname: string)
-			{
-			Input::add_analysis([$source=fname,
-			                     $reader=Input::READER_BINARY,
-			                     $mode=Input::STREAM,
-			                     $name=fname]);
-			}, 10secs);
-		}
-
-	if ( watch_file != "" )
-		{
-		Input::add_analysis([$source=watch_file,
-		                     $reader=Input::READER_BINARY,
-		                     $mode=Input::STREAM,
-		                     $name=watch_file]);
-		}
 	}
 
 event file_new(f: fa_file)
 	{
 	local file_dir = "";
-	local parts = split_all(f$source, /\/[^\/]*$/);
+	local parts = split_string_all(f$source, /\/[^\/]*$/);
 	if ( |parts| == 3 )
-		file_dir = parts[1];
+		file_dir = parts[0];
 
 	if ( (watch_file != "" && f$source == watch_file) || 
 	     (watch_dir != "" && compress_path(watch_dir) == file_dir) )

scripts/base/frameworks/files/magic/__load__.bro

@@ -1,2 +1,3 @@
 @load-sigs ./general
+@load-sigs ./msoffice
 @load-sigs ./libmagic

scripts/base/frameworks/files/magic/general.sig

@@ -1,16 +1,137 @@
 # General purpose file magic signatures.
 
 signature file-plaintext {
-    file-magic /([[:print:][:space:]]{10})/
+    file-magic /^([[:print:][:space:]]{10})/
     file-mime "text/plain", -20
 }
 
 signature file-tar {
-    file-magic /([[:print:]\x00]){100}(([[:digit:]\x00\x20]){8}){3}/
+    file-magic /^[[:print:]\x00]{100}([[:digit:]\x20]{7}\x00){3}([[:digit:]\x20]{11}\x00){2}([[:digit:]\x00\x20]{7}[\x20\x00])[0-7\x00]/
-    file-mime "application/x-tar", 150
+    file-mime "application/x-tar", 100
 }
 
+signature file-zip {
+	file-mime "application/zip", 10
+	file-magic /^PK\x03\x04.{2}/
+}
+
+signature file-jar {
+	file-mime "application/java-archive", 100
+	file-magic /^PK\x03\x04.{1,200}\x14\x00..META-INF\/MANIFEST\.MF/
+}
+
+signature file-java-applet {
+	file-magic /^\xca\xfe\xba\xbe...[\x2e-\x34]/
+	file-mime "application/x-java-applet", 71
+}
+
+# Shockwave flash
 signature file-swf {
-	file-magic /(F|C|Z)WS/
+	file-magic /^(F|C|Z)WS/
 	file-mime "application/x-shockwave-flash", 60
-}
+}
+
+# Microsoft Outlook's Transport Neutral Encapsulation Format
+signature file-tnef {
+	file-magic /^\x78\x9f\x3e\x22/
+	file-mime "application/vnd.ms-tnef", 100
+}
+
+# Mac OS X DMG files
+signature file-dmg {
+	file-magic /^(\x78\x01\x73\x0D\x62\x62\x60|\x78\xDA\x63\x60\x18\x05|\x78\x01\x63\x60\x18\x05|\x78\xDA\x73\x0D|\x78[\x01\xDA]\xED[\xD0-\xD9])/
+	file-mime "application/x-dmg", 100
+}
+
+# Mac OS X Mach-O executable
+signature file-mach-o {
+	file-magic /^[\xce\xcf]\xfa\xed\xfe/
+	file-mime "application/x-mach-o-executable", 100
+}
+
+# Mac OS X Universal Mach-O executable
+signature file-mach-o-universal {
+	file-magic /^\xca\xfe\xba\xbe..\x00[\x01-\x14]/
+	file-mime "application/x-mach-o-executable", 100
+}
+
+# XAR (eXtensible ARchive) format. 
+# Mac OS X uses this for the .pkg format.
+signature file-xar {
+	file-magic /^xar\!/
+	file-mime "application/x-xar", 100
+}
+
+signature file-pkcs7 {
+	file-magic /^MIME-Version:.*protocol=\"application\/pkcs7-signature\"/
+	file-mime "application/pkcs7-signature", 100
+}
+
+# Concatenated X.509 certificates in textual format.
+signature file-pem {
+	file-magic /^-----BEGIN CERTIFICATE-----/
+	file-mime "application/x-pem"
+}
+
+# Java Web Start file.
+signature file-jnlp {
+	file-magic /^\<jnlp\x20/
+	file-mime "application/x-java-jnlp-file", 100
+}
+
+signature file-ico {
+	file-magic /^\x00\x00\x01\x00/
+	file-mime "image/x-icon", 70
+}
+
+signature file-cur {
+	file-magic /^\x00\x00\x02\x00/
+	file-mime "image/x-cursor", 70
+}
+
+signature file-pcap {
+	file-magic /^(\xa1\xb2\xc3\xd4|\xd4\xc3\xb2\xa1)/
+	file-mime "application/vnd.tcpdump.pcap", 70
+}
+
+signature file-pcap-ng {
+	file-magic /^\x0a\x0d\x0d\x0a.{4}(\x1a\x2b\x3c\x4d|\x4d\x3c\x2b\x1a)/
+	file-mime "application/vnd.tcpdump.pcap", 100
+}
+
+signature file-shellscript {
+	file-mime "text/x-shellscript", 250
+	file-magic /^\x23\x21[^\n]{1,15}bin\/(env[[:space:]]+)?(ba|tc|c|z|fa|ae|k)?sh/
+}
+
+signature file-perl {
+	file-magic /^\x23\x21[^\n]{1,15}bin\/(env[[:space:]]+)?perl/
+	file-mime "text/x-perl", 60
+}
+
+signature file-ruby {
+	file-magic /^\x23\x21[^\n]{1,15}bin\/(env[[:space:]]+)?ruby/
+	file-mime "text/x-ruby", 60
+}
+
+signature file-python {
+	file-magic /^\x23\x21[^\n]{1,15}bin\/(env[[:space:]]+)?python/
+	file-mime "text/x-python", 60
+}
+
+signature file-php {
+	file-magic /^.*<\?php/
+	file-mime "text/x-php", 40
+}
+
+# Stereolithography ASCII format
+signature file-stl-ascii {
+	file-magic /^solid\x20/
+	file-mime "application/sla", 10
+}
+
+# Sketchup model file
+signature file-skp {
+	file-magic /^\xFF\xFE\xFF\x0E\x53\x00\x6B\x00\x65\x00\x74\x00\x63\x00\x68\x00\x55\x00\x70\x00\x20\x00\x4D\x00\x6F\x00\x64\x00\x65\x00\x6C\x00/
+	file-mime "application/skp", 100
+}

scripts/base/frameworks/files/magic/libmagic.sig

@@ -7,42 +7,18 @@
 # The instrumented version of the `file` command used to generate these
 # is located at: https://github.com/jsiwek/file/tree/bro-signatures.
 
-# >2080  string,=Foglio di lavoro Microsoft Exce (len=31), ["%s"], swap_endian=0
-signature file-magic-auto0 {
-	file-mime "application/vnd.ms-excel", 340
-	file-magic /(.{2080})(Foglio di lavoro Microsoft Exce)/
-}
-
 # >2  string,=---BEGIN PGP PUBLIC KEY BLOCK- (len=30), ["PGP public key block"], swap_endian=0
 signature file-magic-auto1 {
 	file-mime "application/pgp-keys", 330
 	file-magic /(.{2})(\x2d\x2d\x2dBEGIN PGP PUBLIC KEY BLOCK\x2d)/
 }
 
-# >2080  string,=Microsoft Excel 5.0 Worksheet (len=29), ["%s"], swap_endian=0
-signature file-magic-auto2 {
-	file-mime "application/vnd.ms-excel", 320
-	file-magic /(.{2080})(Microsoft Excel 5\x2e0 Worksheet)/
-}
-
 # >11  string,=must be converted with BinHex (len=29), ["BinHex binary text"], swap_endian=0
 signature file-magic-auto3 {
 	file-mime "application/mac-binhex40", 320
 	file-magic /(.{11})(must be converted with BinHex)/
 }
 
-# >2080  string,=Microsoft Word 6.0 Document (len=27), ["%s"], swap_endian=0
-signature file-magic-auto4 {
-	file-mime "application/msword", 300
-	file-magic /(.{2080})(Microsoft Word 6\x2e0 Document)/
-}
-
-# >2080  string,=Documento Microsoft Word 6 (len=26), ["Spanish Microsoft Word 6 document data"], swap_endian=0
-signature file-magic-auto5 {
-	file-mime "application/msword", 290
-	file-magic /(.{2080})(Documento Microsoft Word 6)/
-}
-
 # >0  string,=-----BEGIN PGP SIGNATURE- (len=25), ["PGP signature"], swap_endian=0
 signature file-magic-auto6 {
 	file-mime "application/pgp-signature", 280
@@ -92,36 +68,6 @@ signature file-magic-auto13 {
 	file-magic /(\x23\x21 ?\x2fusr\x2flocal\x2fbin\x2fgawk)/
 }
 
-# >0  string/wt,=#! /usr/local/bin/bash (len=22), ["Bourne-Again shell script text executable"], swap_endian=0
-signature file-magic-auto14 {
-	file-mime "text/x-shellscript", 250
-	file-magic /(\x23\x21 ?\x2fusr\x2flocal\x2fbin\x2fbash)/
-}
-
-# >0  string/wt,=#! /usr/local/bin/tcsh (len=22), ["Tenex C shell script text executable"], swap_endian=0
-signature file-magic-auto15 {
-	file-mime "text/x-shellscript", 250
-	file-magic /(\x23\x21 ?\x2fusr\x2flocal\x2fbin\x2ftcsh)/
-}
-
-# >0  string/wt,=#! /usr/local/bin/zsh (len=21), ["Paul Falstad's zsh script text executable"], swap_endian=0
-signature file-magic-auto16 {
-	file-mime "text/x-shellscript", 240
-	file-magic /(\x23\x21 ?\x2fusr\x2flocal\x2fbin\x2fzsh)/
-}
-
-# >0  string/wt,=#! /usr/local/bin/ash (len=21), ["Neil Brown's ash script text executable"], swap_endian=0
-signature file-magic-auto17 {
-	file-mime "text/x-shellscript", 240
-	file-magic /(\x23\x21 ?\x2fusr\x2flocal\x2fbin\x2fash)/
-}
-
-# >0  string/wt,=#! /usr/local/bin/ae (len=20), ["Neil Brown's ae script text executable"], swap_endian=0
-signature file-magic-auto18 {
-	file-mime "text/x-shellscript", 230
-	file-magic /(\x23\x21 ?\x2fusr\x2flocal\x2fbin\x2fae)/
-}
-
 # >0  string,=# PaCkAgE DaTaStReAm (len=20), ["pkg Datastream (SVR4)"], swap_endian=0
 signature file-magic-auto19 {
 	file-mime "application/x-svr4-package", 230
@@ -140,30 +86,12 @@ signature file-magic-auto21 {
 	file-magic /(\x5bKDE Desktop Entry\x5d)/
 }
 
-# >512  string,=R\000o\000o\000t\000 \000E\000n\000t\000r\000y (len=19), ["Microsoft Word Document"], swap_endian=0
-signature file-magic-auto22 {
-	file-mime "application/msword", 220
-	file-magic /(.{512})(R\x00o\x00o\x00t\x00 \x00E\x00n\x00t\x00r\x00y)/
-}
-
 # >0  string,=!<arch>\n__________E (len=19), ["MIPS archive"], swap_endian=0
 signature file-magic-auto23 {
 	file-mime "application/x-archive", 220
 	file-magic /(\x21\x3carch\x3e\x0a\x5f\x5f\x5f\x5f\x5f\x5f\x5f\x5f\x5f\x5fE)/
 }
 
-# >0  string/wt,=#! /usr/local/tcsh (len=18), ["Tenex C shell script text executable"], swap_endian=0
-signature file-magic-auto24 {
-	file-mime "text/x-shellscript", 210
-	file-magic /(\x23\x21 ?\x2fusr\x2flocal\x2ftcsh)/
-}
-
-# >0  string/wt,=#! /usr/local/bash (len=18), ["Bourne-Again shell script text executable"], swap_endian=0
-signature file-magic-auto25 {
-	file-mime "text/x-shellscript", 210
-	file-magic /(\x23\x21 ?\x2fusr\x2flocal\x2fbash)/
-}
-
 # >0  string/t,=# KDE Config File (len=17), ["KDE config file"], swap_endian=0
 signature file-magic-auto26 {
 	file-mime "application/x-kdelnk", 200
@@ -189,12 +117,6 @@ signature file-magic-auto29 {
 	file-magic /(\x23\x21 ?\x2fusr\x2fbin\x2fnawk)/
 }
 
-# >0  string/wt,=#! /usr/bin/tcsh (len=16), ["Tenex C shell script text executable"], swap_endian=0
-signature file-magic-auto30 {
-	file-mime "text/x-shellscript", 190
-	file-magic /(\x23\x21 ?\x2fusr\x2fbin\x2ftcsh)/
-}
-
 # >0  string/wt,=#! /usr/bin/gawk (len=16), ["GNU awk script text executable"], swap_endian=0
 signature file-magic-auto31 {
 	file-mime "text/x-gawk", 190
@@ -207,12 +129,6 @@ signature file-magic-auto32 {
 	file-magic /(.{369})(MICROSOFT PIFEX\x00)/
 }
 
-# >0  string/wt,=#! /usr/bin/bash (len=16), ["Bourne-Again shell script text executable"], swap_endian=0
-signature file-magic-auto33 {
-	file-mime "text/x-shellscript", 190
-	file-magic /(\x23\x21 ?\x2fusr\x2fbin\x2fbash)/
-}
-
 # >0  string/w,=#VRML V1.0 ascii (len=16), ["VRML 1 file"], swap_endian=0
 signature file-magic-auto34 {
 	file-mime "model/vrml", 190
@@ -334,12 +250,6 @@ signature file-magic-auto51 {
 	file-magic /(\x23\x21 ?\x2fusr\x2fbin\x2fawk)/
 }
 
-# >0  string/wt,=#! /usr/bin/zsh (len=15), ["Paul Falstad's zsh script text executable"], swap_endian=0
-signature file-magic-auto52 {
-	file-mime "text/x-shellscript", 180
-	file-magic /(\x23\x21 ?\x2fusr\x2fbin\x2fzsh)/
-}
-
 # >0  string,=MAS_UTrack_V00 (len=14), [""], swap_endian=0
 # >>14  string,>/0 (len=2), ["ultratracker V1.%.1s module sound data"], swap_endian=0
 signature file-magic-auto53 {
@@ -457,12 +367,6 @@ signature file-magic-auto70 {
 	file-magic /(\x3cmap ?version)/
 }
 
-# >0  string/wt,=#! /bin/tcsh (len=12), ["Tenex C shell script text executable"], swap_endian=0
-signature file-magic-auto71 {
-	file-mime "text/x-shellscript", 150
-	file-magic /(\x23\x21 ?\x2fbin\x2ftcsh)/
-}
-
 # >0  string/wt,=#! /bin/nawk (len=12), ["new awk script text executable"], swap_endian=0
 signature file-magic-auto72 {
 	file-mime "text/x-nawk", 150
@@ -475,12 +379,6 @@ signature file-magic-auto73 {
 	file-magic /(\x23\x21 ?\x2fbin\x2fgawk)/
 }
 
-# >0  string/wt,=#! /bin/bash (len=12), ["Bourne-Again shell script text executable"], swap_endian=0
-signature file-magic-auto74 {
-	file-mime "text/x-shellscript", 150
-	file-magic /(\x23\x21 ?\x2fbin\x2fbash)/
-}
-
 # >0  string/wt,=#! /bin/awk (len=11), ["awk script text executable"], swap_endian=0
 signature file-magic-auto75 {
 	file-mime "text/x-awk", 140
@@ -505,24 +403,6 @@ signature file-magic-auto78 {
 	file-magic /(d8\x3aannounce)/
 }
 
-# >0  string/wt,=#! /bin/csh (len=11), ["C shell script text executable"], swap_endian=0
-signature file-magic-auto79 {
-	file-mime "text/x-shellscript", 140
-	file-magic /(\x23\x21 ?\x2fbin\x2fcsh)/
-}
-
-# >0  string/wt,=#! /bin/ksh (len=11), ["Korn shell script text executable"], swap_endian=0
-signature file-magic-auto80 {
-	file-mime "text/x-shellscript", 140
-	file-magic /(\x23\x21 ?\x2fbin\x2fksh)/
-}
-
-# >0  string/wt,=#! /bin/zsh (len=11), ["Paul Falstad's zsh script text executable"], swap_endian=0
-signature file-magic-auto81 {
-	file-mime "text/x-shellscript", 140
-	file-magic /(\x23\x21 ?\x2fbin\x2fzsh)/
-}
-
 # >0  string/c,=BEGIN:VCARD (len=11), ["vCard visiting card"], swap_endian=0
 signature file-magic-auto82 {
 	file-mime "text/x-vcard", 140
@@ -545,12 +425,6 @@ signature file-magic-auto84 {
 	file-magic /(Forward to)/
 }
 
-# >0  string/wt,=#! /bin/sh (len=10), ["POSIX shell script text executable"], swap_endian=0
-signature file-magic-auto85 {
-	file-mime "text/x-shellscript", 130
-	file-magic /(\x23\x21 ?\x2fbin\x2fsh)/
-}
-
 # >0  string,=II*\000\020\000\000\000CR (len=10), ["Canon CR2 raw image data"], swap_endian=0
 signature file-magic-auto86 {
 	file-mime "image/x-canon-cr2", 130
@@ -585,12 +459,6 @@ signature file-magic-auto90 {
 	file-magic /(\x3cBookFile)/
 }
 
-# >2112  string,=MSWordDoc (len=9), ["Microsoft Word document data"], swap_endian=0
-signature file-magic-auto91 {
-	file-mime "application/msword", 120
-	file-magic /(.{2112})(MSWordDoc)/
-}
-
 # >0  string/t,=N#! rnews (len=9), ["mailed, batched news text"], swap_endian=0
 signature file-magic-auto92 {
 	file-mime "message/rfc822", 120
@@ -656,12 +524,6 @@ signature file-magic-auto100 {
 	file-magic /(MSCF\x00\x00\x00\x00)/
 }
 
-# >0  string/b,=\320\317\021\340\241\261\032\341 (len=8), ["Microsoft Office Document"], swap_endian=0
-signature file-magic-auto101 {
-	file-mime "application/msword", 110
-	file-magic /(\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1)/
-}
-
 # >21  string/c,=!SCREAM! (len=8), ["Screamtracker 2 module sound data"], swap_endian=0
 signature file-magic-auto102 {
 	file-mime "audio/x-mod", 110
@@ -754,10 +616,10 @@ signature file-magic-auto116 {
 }
 
 # >257  string,=ustar  \000 (len=8), ["GNU tar archive"], swap_endian=0
-signature file-magic-auto117 {
+#signature file-magic-auto117 {
-	file-mime "application/x-tar", 110
+#	file-mime "application/x-tar", 110
-	file-magic /(.{257})(ustar  \x00)/
+#	file-magic /(.{257})(ustar  \x00)/
-}
+#}
 
 # >0  string,=<MIFFile (len=8), ["FrameMaker MIF (ASCII) file"], swap_endian=0
 signature file-magic-auto118 {
@@ -771,12 +633,6 @@ signature file-magic-auto119 {
 	file-magic /(PK\x07\x08PK\x03\x04)/
 }
 
-# >0  string/b,=\t\004\006\000\000\000\020\000 (len=8), ["Microsoft Excel Worksheet"], swap_endian=0
-signature file-magic-auto120 {
-	file-mime "application/vnd.ms-excel", 110
-	file-magic /(\x09\x04\x06\x00\x00\x00\x10\x00)/
-}
-
 # >0  string/b,=WordPro\000 (len=8), ["Lotus WordPro"], swap_endian=0
 signature file-magic-auto121 {
 	file-mime "application/vnd.lotus-wordpro", 110
@@ -844,10 +700,10 @@ signature file-magic-auto130 {
 }
 
 # >257  string,=ustar\000 (len=6), ["POSIX tar archive"], swap_endian=0
-signature file-magic-auto131 {
+#signature file-magic-auto131 {
-	file-mime "application/x-tar", 90
+#	file-mime "application/x-tar", 90
-	file-magic /(.{257})(ustar\x00)/
+#	file-magic /(.{257})(ustar\x00)/
-}
+#}
 
 # >0  string,=AC1.40 (len=6), ["DWG AutoDesk AutoCAD Release 1.40"], swap_endian=0
 signature file-magic-auto132 {
@@ -994,12 +850,6 @@ signature file-magic-auto155 {
 	file-magic /(\x23 xmcd)/
 }
 
-# >0  string/b,=\333\245-\000\000\000 (len=6), ["Microsoft Office Document"], swap_endian=0
-signature file-magic-auto156 {
-	file-mime "application/msword", 90
-	file-magic /(\xdb\xa5\x2d\x00\x00\x00)/
-}
-
 # >2  string,=MMXPR3 (len=6), ["Motorola Quark Express Document (English)"], swap_endian=0
 signature file-magic-auto157 {
 	file-mime "application/x-quark-xpress-3", 90
@@ -1046,36 +896,6 @@ signature file-magic-auto162 {
 	file-magic /(\x3c\x3fxml)(.{15})(.*)( xmlns\x3d)(['"]http:\x2f\x2fwww.opengis.net\x2fkml)/
 }
 
-# >0  string,=PK\003\004 (len=4), [""], swap_endian=0
-# >>30  regex,=[Content_Types].xml|_rels/.rels (len=31), [""], swap_endian=0
-# >>>18 (lelong,+49), search/2000,=PK\003\004 (len=4), [""], swap_endian=0
-# >>>>&26  search/1000,=PK\003\004 (len=4), [""], swap_endian=0
-# >>>>>&26  string,=word/ (len=5), ["Microsoft Word 2007+"], swap_endian=0
-signature file-magic-auto163 {
-	file-mime "application/vnd.openxmlformats-officedocument.wordprocessingml.document", 80
-	file-magic /(PK\x03\x04)(.{26})(\[Content_Types\].xml|_rels\x2f.rels)(.*)(PK\x03\x04)(.{26})(.*)(PK\x03\x04)(.{26})(word\x2f)/
-}
-
-# >0  string,=PK\003\004 (len=4), [""], swap_endian=0
-# >>30  regex,=[Content_Types].xml|_rels/.rels (len=31), [""], swap_endian=0
-# >>>18 (lelong,+49), search/2000,=PK\003\004 (len=4), [""], swap_endian=0
-# >>>>&26  search/1000,=PK\003\004 (len=4), [""], swap_endian=0
-# >>>>>&26  string,=ppt/ (len=4), ["Microsoft PowerPoint 2007+"], swap_endian=0
-signature file-magic-auto164 {
-	file-mime "application/vnd.openxmlformats-officedocument.presentationml.presentation", 70
-	file-magic /(PK\x03\x04)(.{26})(\[Content_Types\].xml|_rels\x2f.rels)(.*)(PK\x03\x04)(.{26})(.*)(PK\x03\x04)(.{26})(ppt\x2f)/
-}
-
-# >0  string,=PK\003\004 (len=4), [""], swap_endian=0
-# >>30  regex,=[Content_Types].xml|_rels/.rels (len=31), [""], swap_endian=0
-# >>>18 (lelong,+49), search/2000,=PK\003\004 (len=4), [""], swap_endian=0
-# >>>>&26  search/1000,=PK\003\004 (len=4), [""], swap_endian=0
-# >>>>>&26  string,=xl/ (len=3), ["Microsoft Excel 2007+"], swap_endian=0
-signature file-magic-auto165 {
-	file-mime "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", 60
-	file-magic /(PK\x03\x04)(.{26})(\[Content_Types\].xml|_rels\x2f.rels)(.*)(PK\x03\x04)(.{26})(.*)(PK\x03\x04)(.{26})(xl\x2f)/
-}
-
 # >60  string,=RINEX (len=5), [""], swap_endian=0
 # >>80  search/256,=XXRINEXB (len=8), ["RINEX Data, GEO SBAS Broadcast"], swap_endian=0
 # >>>5  string,x, [", version %6.6s"], swap_endian=0
@@ -1229,30 +1049,12 @@ signature file-magic-auto187 {
 	file-magic /(\x00\x01\x00\x00\x00)/
 }
 
-# >0  string/b,=PO^Q` (len=5), ["Microsoft Word 6.0 Document"], swap_endian=0
-signature file-magic-auto188 {
-	file-mime "application/msword", 80
-	file-magic /(PO\x5eQ\x60)/
-}
-
 # >0  string,=%PDF- (len=5), ["PDF document"], swap_endian=0
 signature file-magic-auto189 {
 	file-mime "application/pdf", 80
 	file-magic /(\x25PDF\x2d)/
 }
 
-# >2114  string,=Biff5 (len=5), ["Microsoft Excel 5.0 Worksheet"], swap_endian=0
-signature file-magic-auto190 {
-	file-mime "application/vnd.ms-excel", 80
-	file-magic /(.{2114})(Biff5)/
-}
-
-# >2121  string,=Biff5 (len=5), ["Microsoft Excel 5.0 Worksheet"], swap_endian=0
-signature file-magic-auto191 {
-	file-mime "application/vnd.ms-excel", 80
-	file-magic /(.{2121})(Biff5)/
-}
-
 # >0  string/t,=Path: (len=5), ["news text"], swap_endian=0
 signature file-magic-auto192 {
 	file-mime "message/news", 80
@@ -1383,12 +1185,6 @@ signature file-magic-auto211 {
 	file-magic /(\x00\x00\x00\x01)([\x07\x27\x47\x67\x87\xa7\xc7\xe7])/
 }
 
-# >0  belong&,=-889275714 (0xcafebabe), [""], swap_endian=0
-signature file-magic-auto212 {
-	file-mime "application/x-java-applet", 71
-	file-magic /(\xca\xfe\xba\xbe)/
-}
-
 # >0  belong&ffffffffffffff00,=256 (0x00000100), [""], swap_endian=0
 # >>3  byte&,=0xba, ["MPEG sequence"], swap_endian=0
 signature file-magic-auto213 {
@@ -1706,46 +1502,6 @@ signature file-magic-auto245 {
 	file-magic /(PK\x03\x04)(.{22})(\x08\x00\x00\x00mimetypeapplication\x2f)(epub\x2bzip)/
 }
 
-# Seems redundant with other zip signature below.
-# >0  string,=PK\003\004 (len=4), [""], swap_endian=0
-# >>26  string,=\b\000\000\000mimetypeapplication/ (len=24), [""], swap_endian=0
-# >>>50  string,!epub+zip (len=8), [""], swap_endian=0
-# >>>>50  string,!vnd.oasis.opendocument. (len=23), [""], swap_endian=0
-# >>>>>50  string,!vnd.sun.xml. (len=12), [""], swap_endian=0
-# >>>>>>50  string,!vnd.kde. (len=8), [""], swap_endian=0
-# >>>>>>>38  regex,=[!-OQ-~]+ (len=9), ["Zip data (MIME type "%s"?)"], swap_endian=0
-#signature file-magic-auto246 {
-#	file-mime "application/zip", 39
-#	file-magic /(PK\x03\x04)(.{22})(\x08\x00\x00\x00mimetypeapplication\x2f)/
-#}
-
-# >0  string,=PK\003\004 (len=4), [""], swap_endian=0
-# >>26  string,=\b\000\000\000mimetype (len=12), [""], swap_endian=0
-# >>>38  string,!application/ (len=12), [""], swap_endian=0
-# >>>>38  regex,=[!-OQ-~]+ (len=9), ["Zip data (MIME type "%s"?)"], swap_endian=0
-signature file-magic-auto247 {
-	file-mime "application/zip", 39
-	file-magic /(PK\x03\x04)(.{22})(\x08\x00\x00\x00mimetype)/
-}
-
-# The indirect offset makes this difficult to convert.
-# The (.*) may be too generous.
-# >0  string,=PK\003\004 (len=4), [""], swap_endian=0
-# >>26 (leshort,+30), leshort&,=-13570 (0xcafe), ["Java archive data (JAR)"], swap_endian=0
-signature file-magic-auto248 {
-	file-mime "application/java-archive", 50
-	file-magic /(PK\x03\x04)(.*)(\xfe\xca)/
-}
-
-# The indeirect offset and string inequality make this difficult to convert.
-# >0  string,=PK\003\004 (len=4), [""], swap_endian=0
-# >>26 (leshort,+30), leshort&,!-13570 (0xcafe), [""], swap_endian=0
-# >>>26  string,!\b\000\000\000mimetype (len=12), ["Zip archive data"], swap_endian=0
-signature file-magic-auto249 {
-	file-mime "application/zip", 10
-	file-magic /(PK\x03\x04)(.{2})/
-}
-
 # >0  belong&,=442 (0x000001ba), [""], swap_endian=0
 # >>4  byte&,&0x40, [""], swap_endian=0
 signature file-magic-auto250 {
@@ -2065,18 +1821,6 @@ signature file-magic-auto299 {
 	file-magic /(PDN3)/
 }
 
-# >0  ulelong&,=2712847316 (0xa1b2c3d4), ["tcpdump capture file (little-endian)"], swap_endian=0
-signature file-magic-auto300 {
-	file-mime "application/vnd.tcpdump.pcap", 70
-	file-magic /(\xd4\xc3\xb2\xa1)/
-}
-
-# >0  ubelong&,=2712847316 (0xa1b2c3d4), ["tcpdump capture file (big-endian)"], swap_endian=0
-signature file-magic-auto301 {
-	file-mime "application/vnd.tcpdump.pcap", 70
-	file-magic /(\xa1\xb2\xc3\xd4)/
-}
-
 # >0  belong&,=-17957139 (0xfeedfeed), ["Java KeyStore"], swap_endian=0
 signature file-magic-auto302 {
 	file-mime "application/x-java-keystore", 70
@@ -2297,12 +2041,6 @@ signature file-magic-auto335 {
 	file-magic /(SIT\x21)/
 }
 
-# >0  lelong&,=574529400 (0x223e9f78), ["Transport Neutral Encapsulation Format"], swap_endian=0
-signature file-magic-auto336 {
-	file-mime "application/vnd.ms-tnef", 70
-	file-magic /(\x78\x9f\x3e\x22)/
-}
-
 # >0  string,=<ar> (len=4), ["System V Release 1 ar archive"], swap_endian=0
 signature file-magic-auto337 {
 	file-mime "application/x-archive", 70
@@ -2433,48 +2171,6 @@ signature file-magic-auto357 {
 	file-magic /(RIFF)(.{4})(AVI )/
 }
 
-# >0  belong&,=834535424 (0x31be0000), ["Microsoft Word Document"], swap_endian=0
-signature file-magic-auto358 {
-	file-mime "application/msword", 70
-	file-magic /(\x31\xbe\x00\x00)/
-}
-
-# >0  string/b,=\3767\000# (len=4), ["Microsoft Office Document"], swap_endian=0
-signature file-magic-auto359 {
-	file-mime "application/msword", 70
-	file-magic /(\xfe7\x00\x23)/
-}
-
-# >0  string/b,=\333\245-\000 (len=4), ["Microsoft WinWord 2.0 Document"], swap_endian=0
-signature file-magic-auto360 {
-	file-mime "application/msword", 70
-	file-magic /(\xdb\xa5\x2d\x00)/
-}
-
-# >0  string/b,=\333\245-\000 (len=4), ["Microsoft WinWord 2.0 Document"], swap_endian=0
-signature file-magic-auto361 {
-	file-mime "application/msword", 70
-	file-magic /(\xdb\xa5\x2d\x00)/
-}
-
-# >0  belong&,=6656 (0x00001a00), ["Lotus 1-2-3"], swap_endian=0
-signature file-magic-auto362 {
-	file-mime "application/x-123", 70
-	file-magic /(\x00\x00\x1a\x00)/
-}
-
-# >0  belong&,=512 (0x00000200), ["Lotus 1-2-3"], swap_endian=0
-signature file-magic-auto363 {
-	file-mime "application/x-123", 70
-	file-magic /(\x00\x00\x02\x00)/
-}
-
-# >0  string/b,=\000\000\001\000 (len=4), ["MS Windows icon resource"], swap_endian=0
-signature file-magic-auto364 {
-	file-mime "image/x-icon", 70
-	file-magic /(\x00\x00\x01\x00)/
-}
-
 # >0  lelong&,=268435536 (0x10000050), ["Psion Series 5"], swap_endian=0
 # >>4  lelong&,=268435565 (0x1000006d), ["database"], swap_endian=0
 # >>>8  lelong&,=268435588 (0x10000084), ["Agenda file"], swap_endian=0
@@ -2737,12 +2433,6 @@ signature file-magic-auto403 {
 	file-magic /(SBI)/
 }
 
-# >0  string/b,=\224\246. (len=3), ["Microsoft Word Document"], swap_endian=0
-signature file-magic-auto404 {
-	file-mime "application/msword", 60
-	file-magic /(\x94\xa6\x2e)/
-}
-
 # >0  string,=\004%! (len=3), ["PostScript document text"], swap_endian=0
 signature file-magic-auto405 {
 	file-mime "application/postscript", 60
@@ -2763,17 +2453,11 @@ signature file-magic-auto407 {
 	file-magic /(.*)([ \x09]*(class|module)[ \x09][A-Z])((modul|includ)e [A-Z]|def [a-z])(^[ \x09]*end([ \x09]*[;#].*)?$)/
 }
 
-# >512  string/b,=\354\245\301 (len=3), ["Microsoft Word Document"], swap_endian=0
-signature file-magic-auto408 {
-	file-mime "application/msword", 60
-	file-magic /(.{512})(\xec\xa5\xc1)/
-}
-
 # >0  regex/20,=^\.[A-Za-z0-9][A-Za-z0-9][ \t] (len=29), ["troff or preprocessor input text"], swap_endian=0
-signature file-magic-auto411 {
+#signature file-magic-auto411 {
-	file-mime "text/troff", 59
+#	file-mime "text/troff", 59
-	file-magic /(^\.[A-Za-z0-9][A-Za-z0-9][ \x09])/
+#	file-magic /(^\.[A-Za-z0-9][A-Za-z0-9][ \x09])/
-}
+#}
 
 # >0  search/4096,=\documentclass (len=14), ["LaTeX 2e document text"], swap_endian=0
 signature file-magic-auto412 {
@@ -2806,10 +2490,10 @@ signature file-magic-auto416 {
 }
 
 # >0  regex/20,=^\.[A-Za-z0-9][A-Za-z0-9]$ (len=26), ["troff or preprocessor input text"], swap_endian=0
-signature file-magic-auto417 {
+#signature file-magic-auto417 {
-	file-mime "text/troff", 56
+#	file-mime "text/troff", 56
-	file-magic /(^\.[A-Za-z0-9][A-Za-z0-9]$)/
+#	file-magic /(^\.[A-Za-z0-9][A-Za-z0-9]$)/
-}
+#}
 
 # >0  search/w/1,=#! /usr/bin/php (len=15), ["PHP script text executable"], swap_endian=0
 signature file-magic-auto418 {
@@ -2829,30 +2513,12 @@ signature file-magic-auto420 {
 	file-magic /(.*)(eval \x22exec \x2fusr\x2fbin\x2fperl)/
 }
 
-# >0  search/w/1,=#! /usr/local/bin/python (len=24), ["Python script text executable"], swap_endian=0
-signature file-magic-auto421 {
-	file-mime "text/x-python", 54
-	file-magic /(.*)(\x23\x21 ?\x2fusr\x2flocal\x2fbin\x2fpython)/
-}
-
 # >0  search/1,=Common subdirectories:  (len=23), ["diff output text"], swap_endian=0
 signature file-magic-auto422 {
 	file-mime "text/x-diff", 53
 	file-magic /(.*)(Common subdirectories\x3a )/
 }
 
-# >0  search/1,=#! /usr/bin/env python (len=22), ["Python script text executable"], swap_endian=0
-signature file-magic-auto423 {
-	file-mime "text/x-python", 52
-	file-magic /(.*)(\x23\x21 \x2fusr\x2fbin\x2fenv python)/
-}
-
-# >0  search/w/1,=#! /usr/local/bin/ruby (len=22), ["Ruby script text executable"], swap_endian=0
-signature file-magic-auto424 {
-	file-mime "text/x-ruby", 52
-	file-magic /(.*)(\x23\x21 ?\x2fusr\x2flocal\x2fbin\x2fruby)/
-}
-
 # >0  search/w/1,=#! /usr/local/bin/wish (len=22), ["Tcl/Tk script text executable"], swap_endian=0
 signature file-magic-auto425 {
 	file-mime "text/x-tcl", 52
@@ -2871,12 +2537,6 @@ signature file-magic-auto427 {
 	file-magic /(\xff\xd8)/
 }
 
-# >0  search/1,=#!/usr/bin/env python (len=21), ["Python script text executable"], swap_endian=0
-signature file-magic-auto428 {
-	file-mime "text/x-python", 51
-	file-magic /(.*)(\x23\x21\x2fusr\x2fbin\x2fenv python)/
-}
-
 # >0  search/1,=#!/usr/bin/env nodejs (len=21), ["Node.js script text executable"], swap_endian=0
 signature file-magic-auto429 {
 	file-mime "application/javascript", 51
@@ -3189,12 +2849,6 @@ signature file-magic-auto474 {
 	file-magic /(\x25\x21)/
 }
 
-# >0  search/1,=#! /usr/bin/env ruby (len=20), ["Ruby script text executable"], swap_endian=0
-signature file-magic-auto475 {
-	file-mime "text/x-ruby", 50
-	file-magic /(.*)(\x23\x21 \x2fusr\x2fbin\x2fenv ruby)/
-}
-
 # >0  regex/1,=(^[0-9]{5})[acdn][w] (len=20), ["MARC21 Classification"], swap_endian=0
 signature file-magic-auto476 {
 	file-mime "application/marc", 50
@@ -3228,10 +2882,10 @@ signature file-magic-auto480 {
 }
 
 # >0  string,=\n( (len=2), ["Emacs v18 byte-compiled Lisp data"], swap_endian=0
-signature file-magic-auto481 {
+#signature file-magic-auto481 {
-	file-mime "application/x-elc", 50
+#	file-mime "application/x-elc", 50
-	file-magic /(\x0a\x28)/
+#	file-magic /(\x0a\x28)/
-}
+#}
 
 # >0  string,=\021\t (len=2), ["Award BIOS Logo, 136 x 126"], swap_endian=0
 signature file-magic-auto482 {
@@ -3305,17 +2959,17 @@ signature file-magic-auto493 {
 	file-magic /(\xf7\x02)/
 }
 
-# >2  string,=\000\021 (len=2), ["TeX font metric data"], swap_endian=0
+## >2  string,=\000\021 (len=2), ["TeX font metric data"], swap_endian=0
-signature file-magic-auto494 {
+#signature file-magic-auto494 {
-	file-mime "application/x-tex-tfm", 50
+#	file-mime "application/x-tex-tfm", 50
-	file-magic /(.{2})(\x00\x11)/
+#	file-magic /(.{2})(\x00\x11)/
-}
+#}
-
+#
-# >2  string,=\000\022 (len=2), ["TeX font metric data"], swap_endian=0
+## >2  string,=\000\022 (len=2), ["TeX font metric data"], swap_endian=0
-signature file-magic-auto495 {
+#signature file-magic-auto495 {
-	file-mime "application/x-tex-tfm", 50
+#	file-mime "application/x-tex-tfm", 50
-	file-magic /(.{2})(\x00\x12)/
+#	file-magic /(.{2})(\x00\x12)/
-}
+#}
 
 # >0  beshort&,=-31486 (0x8502), ["GPG encrypted data"], swap_endian=0
 signature file-magic-auto496 {
@@ -3470,12 +3124,6 @@ signature file-magic-auto514 {
 	file-magic /(.*)(\x23\x21 \x2fusr\x2fbin\x2fenv lua)/
 }
 
-# >0  search/1,=#!/usr/bin/env ruby (len=19), ["Ruby script text executable"], swap_endian=0
-signature file-magic-auto515 {
-	file-mime "text/x-ruby", 49
-	file-magic /(.*)(\x23\x21\x2fusr\x2fbin\x2fenv ruby)/
-}
-
 # >0  search/1,=#! /usr/bin/env tcl (len=19), ["Tcl script text executable"], swap_endian=0
 signature file-magic-auto516 {
 	file-mime "text/x-tcl", 49
@@ -3493,12 +3141,6 @@ signature file-magic-auto519 {
 	file-magic /(.*)(\x23\x21\x2fusr\x2fbin\x2fenv lua)/
 }
 
-# >0  search/w/1,=#! /usr/bin/python (len=18), ["Python script text executable"], swap_endian=0
-signature file-magic-auto520 {
-	file-mime "text/x-python", 48
-	file-magic /(.*)(\x23\x21 ?\x2fusr\x2fbin\x2fpython)/
-}
-
 # >0  search/w/1,=#!/usr/bin/nodejs (len=17), ["Node.js script text executable"], swap_endian=0
 signature file-magic-auto521 {
 	file-mime "application/javascript", 47
@@ -3506,10 +3148,10 @@ signature file-magic-auto521 {
 }
 
 # >0  regex,=^class[ \t\n]+ (len=12), ["C++ source text"], swap_endian=0
-signature file-magic-auto522 {
+#signature file-magic-auto522 {
-	file-mime "text/x-c++", 47
+#	file-mime "text/x-c++", 47
-	file-magic /(.*)(class[ \x09\x0a]+[[:alnum:]_]+)(.*)(\x7b)(.*)(public:)/
+#	file-magic /(.*)(class[ \x09\x0a]+[[:alnum:]_]+)(.*)(\x7b)(.*)(public:)/
-}
+#}
 
 # >0  search/1,=This is Info file (len=17), ["GNU Info text"], swap_endian=0
 signature file-magic-auto528 {
@@ -3658,12 +3300,6 @@ signature file-magic-auto545 {
 	file-magic /(.*)(\x23\x21 ?\x2fusr\x2fbin\x2fwish)/
 }
 
-# >0  search/w/1,=#! /usr/bin/ruby (len=16), ["Ruby script text executable"], swap_endian=0
-signature file-magic-auto546 {
-	file-mime "text/x-ruby", 46
-	file-magic /(.*)(\x23\x21 ?\x2fusr\x2fbin\x2fruby)/
-}
-
 # >0  search/w/1,=#! /usr/bin/lua (len=15), ["Lua script text executable"], swap_endian=0
 signature file-magic-auto547 {
 	file-mime "text/x-lua", 45
@@ -3727,10 +3363,10 @@ signature file-magic-auto556 {
 }
 
 # >0  regex,=^extern[ \t\n]+ (len=13), ["C source text"], swap_endian=0
-signature file-magic-auto557 {
+#signature file-magic-auto557 {
-	file-mime "text/x-c", 43
+#	file-mime "text/x-c", 43
-	file-magic /(.*)(extern[ \x09\x0a]+)/
+#	file-magic /(.*)(extern[ \x09\x0a]+)/
-}
+#}
 
 # >0  search/4096,=% -*-latex-*- (len=13), ["LaTeX document text"], swap_endian=0
 signature file-magic-auto558 {
@@ -3746,10 +3382,10 @@ signature file-magic-auto558 {
 #}
 
 # >0  regex,=^struct[ \t\n]+ (len=13), ["C source text"], swap_endian=0
-signature file-magic-auto560 {
+#signature file-magic-auto560 {
-	file-mime "text/x-c", 43
+#	file-mime "text/x-c", 43
-	file-magic /(.*)(struct[ \x09\x0a]+)/
+#	file-magic /(.*)(struct[ \x09\x0a]+)/
-}
+#}
 
 # >0  search/w/1,=#!/bin/nodejs (len=13), ["Node.js script text executable"], swap_endian=0
 signature file-magic-auto561 {
@@ -3802,10 +3438,10 @@ signature file-magic-auto567 {
 }
 
 # >0  regex,=^char[ \t\n]+ (len=11), ["C source text"], swap_endian=0
-signature file-magic-auto568 {
+#signature file-magic-auto568 {
-	file-mime "text/x-c", 41
+#	file-mime "text/x-c", 41
-	file-magic /(.*)(char[ \x09\x0a]+)/
+#	file-magic /(.*)(char[ \x09\x0a]+)/
-}
+#}
 
 # >0  search/1,=#! (len=2), [""], swap_endian=0
 # >>0  regex,=^#!.*/bin/perl$ (len=15), ["Perl script text executable"], swap_endian=0
@@ -3887,23 +3523,11 @@ signature file-magic-auto578 {
 	file-magic /(^dnl )/
 }
 
-# >0  regex,=^all: (len=5), ["makefile script text"], swap_endian=0
-signature file-magic-auto579 {
-	file-mime "text/x-makefile", 40
-	file-magic /(^all:)/
-}
-
-# >0  regex,=^.PRECIOUS (len=10), ["makefile script text"], swap_endian=0
-signature file-magic-auto580 {
-	file-mime "text/x-makefile", 40
-	file-magic /(^.PRECIOUS)/
-}
-
 # >0  search/8192,=main( (len=5), ["C source text"], swap_endian=0
-signature file-magic-auto581 {
+#signature file-magic-auto581 {
-	file-mime "text/x-c", 40
+#	file-mime "text/x-c", 40
-	file-magic /(.*)(main\x28)/
+#	file-magic /(.*)(main\x28)/
-}
+#}
 
 # Not specific enough.
 # >0  search/1,=\" (len=2), ["troff or preprocessor input text"], swap_endian=0
@@ -3932,22 +3556,22 @@ signature file-magic-auto584 {
 #}
 
 # >0  regex,=^#include (len=9), ["C source text"], swap_endian=0
-signature file-magic-auto586 {
+#signature file-magic-auto586 {
-	file-mime "text/x-c", 39
+#	file-mime "text/x-c", 39
-	file-magic /(.*)(#include)/
+#	file-magic /(.*)(#include)/
-}
+#}
 
 # >0  search/1,=.\" (len=3), ["troff or preprocessor input text"], swap_endian=0
-signature file-magic-auto587 {
+#signature file-magic-auto587 {
-	file-mime "text/troff", 39
+#	file-mime "text/troff", 39
-	file-magic /(.*)(\x2e\x5c\x22)/
+#	file-magic /(.*)(\x2e\x5c\x22)/
-}
+#}
 
 # >0  search/1,='\" (len=3), ["troff or preprocessor input text"], swap_endian=0
-signature file-magic-auto588 {
+#signature file-magic-auto588 {
-	file-mime "text/troff", 39
+#	file-mime "text/troff", 39
-	file-magic /(.*)(\x27\x5c\x22)/
+#	file-magic /(.*)(\x27\x5c\x22)/
-}
+#}
 
 # >0  search/1,=<TeXmacs| (len=9), ["TeXmacs document text"], swap_endian=0
 signature file-magic-auto589 {
@@ -3974,10 +3598,10 @@ signature file-magic-auto592 {
 }
 
 # >0  search/1,=''' (len=3), ["troff or preprocessor input text"], swap_endian=0
-signature file-magic-auto593 {
+#signature file-magic-auto593 {
-	file-mime "text/troff", 39
+#	file-mime "text/troff", 39
-	file-magic /(.*)(\x27\x27\x27)/
+#	file-magic /(.*)(\x27\x27\x27)/
-}
+#}
 
 # >0  search/4096,=try: (len=4), [""], swap_endian=0
 # >>&0  regex,=^\s*except.*: (len=13), ["Python script text executable"], swap_endian=0
@@ -3999,12 +3623,6 @@ signature file-magic-auto596 {
 	file-magic /(.*)(\x22LIBHDR\x22)/
 }
 
-# >0  regex,=^SUBDIRS (len=8), ["automake makefile script text"], swap_endian=0
-signature file-magic-auto597 {
-	file-mime "text/x-makefile", 38
-	file-magic /(.*)(SUBDIRS)/
-}
-
 # >0  search/4096,=(defvar  (len=8), ["Lisp/Scheme program text"], swap_endian=0
 signature file-magic-auto598 {
 	file-mime "text/x-lisp", 38
@@ -4031,19 +3649,6 @@ signature file-magic-auto600 {
 #	file-magic /(.*)(\x2a\x2a\x2a )/
 #}
 
-# >0  search/1,='.\" (len=4), ["troff or preprocessor input text"], swap_endian=0
-signature file-magic-auto602 {
-	file-mime "text/troff", 38
-	file-magic /(.*)(\x27\x2e\x5c\x22)/
-}
-
-# LDFLAGS appears in other contexts, e.g. shell script.
-# >0  regex,=^LDFLAGS (len=8), ["makefile script text"], swap_endian=0
-#signature file-magic-auto603 {
-#	file-mime "text/x-makefile", 38
-#	file-magic /(.*)(LDFLAGS)/
-#}
-
 # >0  search/8192,="libhdr" (len=8), ["BCPL source text"], swap_endian=0
 signature file-magic-auto604 {
 	file-mime "text/x-bcpl", 38
@@ -4057,12 +3662,6 @@ signature file-magic-auto604 {
 #	file-magic /(^record)/
 #}
 
-# >0  regex,=^CFLAGS (len=7), ["makefile script text"], swap_endian=0
-signature file-magic-auto606 {
-	file-mime "text/x-makefile", 37
-	file-magic /(.*)(CFLAGS)/
-}
-
 # >0  search/4096,=(defun  (len=7), ["Lisp/Scheme program text"], swap_endian=0
 signature file-magic-auto607 {
 	file-mime "text/x-lisp", 37

scripts/base/frameworks/files/magic/msoffice.sig

@@ -0,0 +1,28 @@
+
+# This signature is non-specific and terrible but after
+# searching for a long time there doesn't seem to be a 
+# better option.  
+signature file-msword {
+	file-magic /^\xd0\xcf\x11\xe0\xa1\xb1\x1a\xe1/
+	file-mime "application/msword", 50
+}
+
+signature file-ooxml {
+	file-magic /^PK\x03\x04\x14\x00\x06\x00/
+	file-mime "application/vnd.openxmlformats-officedocument", 50
+}
+
+signature file-docx {
+	file-magic /^PK\x03\x04.{26}(\[Content_Types\]\.xml|_rels\x2f\.rels|word\x2f).*PK\x03\x04.{26}word\x2f/
+	file-mime "application/vnd.openxmlformats-officedocument.wordprocessingml.document", 80
+}
+
+signature file-xlsx {
+	file-magic /^PK\x03\x04.{26}(\[Content_Types\]\.xml|_rels\x2f\.rels|xl\2f).*PK\x03\x04.{26}xl\x2f/
+	file-mime "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet", 80
+}
+
+signature file-pptx {
+	file-magic /^PK\x03\x04.{26}(\[Content_Types\]\.xml|_rels\x2f\.rels|ppt\x2f).*PK\x03\x04.{26}ppt\x2f/
+	file-mime "application/vnd.openxmlformats-officedocument.presentationml.presentation", 80
+}

scripts/base/frameworks/files/main.bro

@@ -100,8 +100,9 @@ export {
 		## during the process of analysis e.g. due to dropped packets.
 		missing_bytes: count &log &default=0;
 
-		## The number of not all-in-sequence bytes in the file stream that
+		## The number of bytes in the file stream that were not delivered to
-		## were delivered to file analyzers due to reassembly buffer overflow.
+		## stream file analyzers.  This could be overlapping bytes or 
+		## bytes that couldn't be reassembled.
 		overflow_bytes: count &log &default=0;
 
 		## Whether the file analysis timed out at least once for the file.
@@ -124,6 +125,37 @@ export {
 	## generate two handles that would hash to the same file id.
 	const salt = "I recommend changing this." &redef;
 
+	## Decide if you want to automatically attached analyzers to 
+	## files based on the detected mime type of the file.
+	const analyze_by_mime_type_automatically = T &redef;
+
+	## The default setting for if the file reassembler is enabled for 
+	## each file.
+	const enable_reassembler = T &redef;
+
+	## The default per-file reassembly buffer size.
+	const reassembly_buffer_size = 1048576 &redef;
+
+	## Allows the file reassembler to be used if it's necessary because the
+	## file is transferred out of order.
+	##
+	## f: the file.
+	global enable_reassembly: function(f: fa_file);
+
+	## Disables the file reassembler on this file.  If the file is not 
+	## transferred out of order this will have no effect.
+	##
+	## f: the file.
+	global disable_reassembly: function(f: fa_file);
+
+	## Set the maximum size the reassembly buffer is allowed to grow
+	## for the given file.
+	##
+	## f: the file.
+	##
+	## max: Maximum allowed size of the reassembly buffer.
+	global set_reassembly_buffer_size: function(f: fa_file, max: count);
+
 	## Sets the *timeout_interval* field of :bro:see:`fa_file`, which is
 	## used to determine the length of inactivity that is allowed for a file
 	## before internal state related to it is cleaned up.  When used within
@@ -153,15 +185,6 @@ export {
 	                              tag: Files::Tag,
 	                              args: AnalyzerArgs &default=AnalyzerArgs()): bool;
 
-	## Adds all analyzers associated with a give MIME type to the analysis of
-	## a file.  Note that analyzers added via MIME types cannot take further
-	## arguments.
-	##
-	## f: the file.
-	##
-	## mtype: the MIME type; it will be compared case-insensitive.
-	global add_analyzers_for_mime_type: function(f: fa_file, mtype: string);
-
 	## Removes an analyzer from the analysis of a given file.
 	##
 	## f: the file.
@@ -244,7 +267,7 @@ export {
 	## mts: The set of MIME types, each in the form "foo/bar" (case-insensitive).
 	##
 	## Returns: True if the MIME types were successfully registered.
-	global register_for_mime_types: function(tag: Analyzer::Tag, mts: set[string]) : bool;
+	global register_for_mime_types: function(tag: Files::Tag, mts: set[string]) : bool;
 
 	## Registers a MIME type for an analyzer. If a future file with this type is seen,
 	## the analyzer will be automatically assigned to parsing it. The function *adds*
@@ -255,20 +278,20 @@ export {
 	## mt: The MIME type in the form "foo/bar" (case-insensitive).
 	##
 	## Returns: True if the MIME type was successfully registered.
-	global register_for_mime_type: function(tag: Analyzer::Tag, mt: string) : bool;
+	global register_for_mime_type: function(tag: Files::Tag, mt: string) : bool;
 
 	## Returns a set of all MIME types currently registered for a specific analyzer.
 	##
 	## tag: The tag of the analyzer.
 	##
 	## Returns: The set of MIME types.
-	global registered_mime_types: function(tag: Analyzer::Tag) : set[string];
+	global registered_mime_types: function(tag: Files::Tag) : set[string];
 
 	## Returns a table of all MIME-type-to-analyzer mappings currently registered.
 	##
 	## Returns: A table mapping each analyzer to the set of MIME types
 	##          registered for it.
-	global all_registered_mime_types: function() : table[Analyzer::Tag] of set[string];
+	global all_registered_mime_types: function() : table[Files::Tag] of set[string];
 
 	## Event that can be handled to access the Info record as it is sent on
 	## to the logging framework.
@@ -283,7 +306,8 @@ redef record fa_file += {
 global registered_protocols: table[Analyzer::Tag] of ProtoRegistration = table();
 
 # Store the MIME type to analyzer mappings.
-global mime_types: table[Analyzer::Tag] of set[string];
+global mime_types: table[Files::Tag] of set[string];
+global mime_type_to_analyzers: table[string] of set[Files::Tag];
 
 global analyzer_add_callbacks: table[Files::Tag] of function(f: fa_file, args: AnalyzerArgs) = table();
 
@@ -313,8 +337,6 @@ function set_info(f: fa_file)
 	f$info$overflow_bytes = f$overflow_bytes;
 	if ( f?$is_orig )
 		f$info$is_orig = f$is_orig;
-	if ( f?$mime_type )
-		f$info$mime_type = f$mime_type;
 	}
 
 function set_timeout_interval(f: fa_file, t: interval): bool
@@ -322,6 +344,21 @@ function set_timeout_interval(f: fa_file, t: interval): bool
 	return __set_timeout_interval(f$id, t);
 	}
 
+function enable_reassembly(f: fa_file)
+	{
+	__enable_reassembly(f$id);
+	}
+
+function disable_reassembly(f: fa_file)
+	{
+	__disable_reassembly(f$id);
+	}
+
+function set_reassembly_buffer_size(f: fa_file, max: count)
+	{
+	__set_reassembly_buffer(f$id, max);
+	}
+
 function add_analyzer(f: fa_file, tag: Files::Tag, args: AnalyzerArgs): bool
 	{
 	add f$info$analyzers[Files::analyzer_name(tag)];
@@ -337,15 +374,6 @@ function add_analyzer(f: fa_file, tag: Files::Tag, args: AnalyzerArgs): bool
 	return T;
 	}
 
-function add_analyzers_for_mime_type(f: fa_file, mtype: string)
-	{
-	local dummy_args: AnalyzerArgs;
-	local analyzers = __add_analyzers_for_mime_type(f$id, mtype, dummy_args);
-
-	for ( tag in analyzers )
-		add f$info$analyzers[Files::analyzer_name(tag)];
-	}
-
 function register_analyzer_add_callback(tag: Files::Tag, callback: function(f: fa_file, args: AnalyzerArgs))
 	{
 	analyzer_add_callbacks[tag] = callback;
@@ -366,17 +394,87 @@ function analyzer_name(tag: Files::Tag): string
 	return __analyzer_name(tag);
 	}
 
+function register_protocol(tag: Analyzer::Tag, reg: ProtoRegistration): bool
+	{
+	local result = (tag !in registered_protocols);
+	registered_protocols[tag] = reg;
+	return result;
+	}
+
+function register_for_mime_types(tag: Files::Tag, mime_types: set[string]) : bool
+	{
+	local rc = T;
+
+	for ( mt in mime_types )
+		{
+		if ( ! register_for_mime_type(tag, mt) )
+			rc = F;
+		}
+
+	return rc;
+	}
+
+function register_for_mime_type(tag: Files::Tag, mt: string) : bool
+	{
+	if ( tag !in mime_types )
+		{
+		mime_types[tag] = set();
+		}
+	add mime_types[tag][mt];
+
+	if ( mt !in mime_type_to_analyzers )
+		{
+		mime_type_to_analyzers[mt] = set();
+		}
+	add mime_type_to_analyzers[mt][tag];
+
+	return T;
+	}
+
+function registered_mime_types(tag: Files::Tag) : set[string]
+	{
+	return tag in mime_types ? mime_types[tag] : set();
+	}
+
+function all_registered_mime_types(): table[Files::Tag] of set[string]
+	{
+	return mime_types;
+	}
+
+function describe(f: fa_file): string
+	{
+	local tag = Analyzer::get_tag(f$source);
+	if ( tag !in registered_protocols )
+		return "";
+
+	local handler = registered_protocols[tag];
+	return handler$describe(f);
+	}
+
+event get_file_handle(tag: Files::Tag, c: connection, is_orig: bool) &priority=5
+	{
+	if ( tag !in registered_protocols )
+		return;
+
+	local handler = registered_protocols[tag];
+	set_file_handle(handler$get_file_handle(c, is_orig));
+	}
+
 event file_new(f: fa_file) &priority=10
 	{
 	set_info(f);
 
-	if ( f?$mime_type )
+	if ( enable_reassembler )
-		add_analyzers_for_mime_type(f, f$mime_type);
+		{
+		Files::enable_reassembly(f);
+		Files::set_reassembly_buffer_size(f, reassembly_buffer_size);
+		}
 	}
 
 event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) &priority=10
 	{
 	set_info(f);
+
 	add f$info$conn_uids[c$uid];
 	local cid = c$id;
 	add f$info$tx_hosts[f$is_orig ? cid$orig_h : cid$resp_h];
@@ -386,6 +484,24 @@ event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) &priori
 	add f$info$rx_hosts[f$is_orig ? cid$resp_h : cid$orig_h];
 	}
 
+event file_mime_type(f: fa_file, mime_type: string) &priority=10
+	{
+	set_info(f);
+
+	f$info$mime_type = mime_type;
+
+	if ( analyze_by_mime_type_automatically &&
+	     mime_type in mime_type_to_analyzers )
+		{
+		local analyzers = mime_type_to_analyzers[mime_type];
+		for ( a in analyzers )
+			{
+			add f$info$analyzers[Files::analyzer_name(a)];
+			Files::add_analyzer(f, a);
+			}
+		}
+	}
+
 event file_timeout(f: fa_file) &priority=10
 	{
 	set_info(f);
@@ -401,64 +517,3 @@ event file_state_remove(f: fa_file) &priority=-10
 	{
 	Log::write(Files::LOG, f$info);
 	}
-
-function register_protocol(tag: Analyzer::Tag, reg: ProtoRegistration): bool
-	{
-	local result = (tag !in registered_protocols);
-	registered_protocols[tag] = reg;
-	return result;
-	}
-
-function register_for_mime_types(tag: Analyzer::Tag, mime_types: set[string]) : bool
-	{
-	local rc = T;
-
-	for ( mt in mime_types )
-		{
-		if ( ! register_for_mime_type(tag, mt) )
-			rc = F;
-		}
-
-	return rc;
-	}
-
-function register_for_mime_type(tag: Analyzer::Tag, mt: string) : bool
-	{
-	if ( ! __register_for_mime_type(tag, mt) )
-		return F;
-
-	if ( tag !in mime_types )
-		mime_types[tag] = set();
-
-	add mime_types[tag][mt];
-	return T;
-	}
-
-function registered_mime_types(tag: Analyzer::Tag) : set[string]
-	{
-	return tag in mime_types ? mime_types[tag] : set();
-	}
-
-function all_registered_mime_types(): table[Analyzer::Tag] of set[string]
-	{
-	return mime_types;
-	}
-
-function describe(f: fa_file): string
-	{
-	local tag = Analyzer::get_tag(f$source);
-	if ( tag !in registered_protocols )
-		return "";
-
-	local handler = registered_protocols[tag];
-	return handler$describe(f);
-	}
-
-event get_file_handle(tag: Analyzer::Tag, c: connection, is_orig: bool) &priority=5
-	{
-	if ( tag !in registered_protocols )
-		return;
-
-	local handler = registered_protocols[tag];
-	set_file_handle(handler$get_file_handle(c, is_orig));
-	}

scripts/base/frameworks/intel/main.bro

@@ -67,6 +67,7 @@ export {
 		IN_ANYWHERE,
 	};
 
+	## Information about a piece of "seen" data.
 	type Seen: record {
 		## The string if the data is about a string.
 		indicator:       string        &log &optional;
@@ -124,7 +125,7 @@ export {
 		sources:  set[string]    &log &default=string_set();
 	};
 
-	## Intelligence data manipulation functions.
+	## Intelligence data manipulation function.
 	global insert: function(item: Item);
 
 	## Function to declare discovery of a piece of data in order to check
@@ -289,8 +290,8 @@ event Intel::match(s: Seen, items: set[Item]) &priority=5
 		if ( ! info?$fuid )
 			info$fuid = s$f$id;
 
-		if ( ! info?$file_mime_type && s$f?$mime_type )
+		if ( ! info?$file_mime_type && s$f?$info && s$f$info?$mime_type )
-			info$file_mime_type = s$f$mime_type;
+			info$file_mime_type = s$f$info$mime_type;
 
 		if ( ! info?$file_desc )
 			info$file_desc = Files::describe(s$f);

scripts/base/frameworks/logging/main.bro

@@ -405,30 +405,30 @@ function default_path_func(id: ID, path: string, rec: any) : string
 
 	local id_str = fmt("%s", id);
 
-	local parts = split1(id_str, /::/);
+	local parts = split_string1(id_str, /::/);
 	if ( |parts| == 2 )
 		{
 		# Example: Notice::LOG -> "notice"
-		if ( parts[2] == "LOG" )
+		if ( parts[1] == "LOG" )
 			{
-			local module_parts = split_n(parts[1], /[^A-Z][A-Z][a-z]*/, T, 4);
+			local module_parts = split_string_n(parts[0], /[^A-Z][A-Z][a-z]*/, T, 4);
 			local output = "";
-			if ( 1 in module_parts )
+			if ( 0 in module_parts )
-				output = module_parts[1];
+				output = module_parts[0];
+			if ( 1 in module_parts && module_parts[1] != "" )
+				output = cat(output, sub_bytes(module_parts[1],1,1), "_", sub_bytes(module_parts[1], 2, |module_parts[1]|));
 			if ( 2 in module_parts && module_parts[2] != "" )
-				output = cat(output, sub_bytes(module_parts[2],1,1), "_", sub_bytes(module_parts[2], 2, |module_parts[2]|));
+				output = cat(output, "_", module_parts[2]);
 			if ( 3 in module_parts && module_parts[3] != "" )
-				output = cat(output, "_", module_parts[3]);
+				output = cat(output, sub_bytes(module_parts[3],1,1), "_", sub_bytes(module_parts[3], 2, |module_parts[3]|));
-			if ( 4 in module_parts && module_parts[4] != "" )
-				output = cat(output, sub_bytes(module_parts[4],1,1), "_", sub_bytes(module_parts[4], 2, |module_parts[4]|));
 			return to_lower(output);
 			}
 
 		# Example: Notice::POLICY_LOG -> "notice_policy"
-		if ( /_LOG$/ in parts[2] )
+		if ( /_LOG$/ in parts[1] )
-			parts[2] = sub(parts[2], /_LOG$/, "");
+			parts[1] = sub(parts[1], /_LOG$/, "");
 
-		return cat(to_lower(parts[1]),"_",to_lower(parts[2]));
+		return cat(to_lower(parts[0]),"_",to_lower(parts[1]));
 		}
 	else
 		return to_lower(id_str);

scripts/base/frameworks/notice/main.bro

@@ -531,8 +531,8 @@ function create_file_info(f: fa_file): Notice::FileInfo
 	local fi: Notice::FileInfo = Notice::FileInfo($fuid = f$id,
 	                                              $desc = Files::describe(f));
 
-	if ( f?$mime_type )
+	if ( f?$info && f$info?$mime_type )
-		fi$mime = f$mime_type;
+		fi$mime = f$info$mime_type;
 
 	if ( f?$conns && |f$conns| == 1 )
 		for ( id in f$conns )

scripts/base/frameworks/software/main.bro

@@ -133,62 +133,62 @@ function parse(unparsed_version: string): Description
 		{
 		# The regular expression should match the complete version number
 		# and software name.
-		local version_parts = split_n(unparsed_version, /\/?( [\(])?v?[0-9\-\._, ]{2,}/, T, 1);
+		local version_parts = split_string_n(unparsed_version, /\/?( [\(])?v?[0-9\-\._, ]{2,}/, T, 1);
-		if ( 1 in version_parts )
+		if ( 0 in version_parts )
 			{
-			if ( /^\(/ in version_parts[1] )
+			if ( /^\(/ in version_parts[0] )
-				software_name = strip(sub(version_parts[1], /[\(]/, ""));
+				software_name = strip(sub(version_parts[0], /[\(]/, ""));
 			else
-				software_name = strip(version_parts[1]);
+				software_name = strip(version_parts[0]);
 			}
 		if ( |version_parts| >= 2 )
 			{
 			# Remove the name/version separator if it's left at the beginning
 			# of the version number from the previous split_all.
-			local sv = strip(version_parts[2]);
+			local sv = strip(version_parts[1]);
 			if ( /^[\/\-\._v\(]/ in sv )
-				sv = strip(sub(version_parts[2], /^\(?[\/\-\._v\(]/, ""));
+				sv = strip(sub(version_parts[1], /^\(?[\/\-\._v\(]/, ""));
-			local version_numbers = split_n(sv, /[\-\._,\[\(\{ ]/, F, 3);
+			local version_numbers = split_string_n(sv, /[\-\._,\[\(\{ ]/, F, 3);
-			if ( 5 in version_numbers && version_numbers[5] != "" )
+			if ( 4 in version_numbers && version_numbers[4] != "" )
-				v$addl = strip(version_numbers[5]);
+				v$addl = strip(version_numbers[4]);
-			else if ( 3 in version_parts && version_parts[3] != "" &&
+			else if ( 2 in version_parts && version_parts[2] != "" &&
-			          version_parts[3] != ")" )
+			          version_parts[2] != ")" )
 				{
-				if ( /^[[:blank:]]*\([a-zA-Z0-9\-\._[:blank:]]*\)/ in version_parts[3] )
+				if ( /^[[:blank:]]*\([a-zA-Z0-9\-\._[:blank:]]*\)/ in version_parts[2] )
 					{
-					v$addl = split_n(version_parts[3], /[\(\)]/, F, 2)[2];
+					v$addl = split_string_n(version_parts[2], /[\(\)]/, F, 2)[1];
 					}
 				else
 					{
-					local vp = split_n(version_parts[3], /[\-\._,;\[\]\(\)\{\} ]/, F, 3);
+					local vp = split_string_n(version_parts[2], /[\-\._,;\[\]\(\)\{\} ]/, F, 3);
-					if ( |vp| >= 1 && vp[1] != "" )
+					if ( |vp| >= 1 && vp[0] != "" )
+						{
+						v$addl = strip(vp[0]);
+						}
+					else if ( |vp| >= 2 && vp[1] != "" )
 						{
 						v$addl = strip(vp[1]);
 						}
-					else if ( |vp| >= 2 && vp[2] != "" )
+					else if ( |vp| >= 3 && vp[2] != "" )
 						{
 						v$addl = strip(vp[2]);
 						}
-					else if ( |vp| >= 3 && vp[3] != "" )
-						{
-						v$addl = strip(vp[3]);
-						}
 					else
 						{
-						v$addl = strip(version_parts[3]);
+						v$addl = strip(version_parts[2]);
 						}
 						
 					}
 				}
 			
-			if ( 4 in version_numbers && version_numbers[4] != "" )
-				v$minor3 = extract_count(version_numbers[4]);
 			if ( 3 in version_numbers && version_numbers[3] != "" )
-				v$minor2 = extract_count(version_numbers[3]);
+				v$minor3 = extract_count(version_numbers[3]);
 			if ( 2 in version_numbers && version_numbers[2] != "" )
-				v$minor = extract_count(version_numbers[2]);
+				v$minor2 = extract_count(version_numbers[2]);
 			if ( 1 in version_numbers && version_numbers[1] != "" )
-				v$major = extract_count(version_numbers[1]);
+				v$minor = extract_count(version_numbers[1]);
+			if ( 0 in version_numbers && version_numbers[0] != "" )
+				v$major = extract_count(version_numbers[0]);
 			}
 		}
 	
@@ -200,14 +200,14 @@ function parse_mozilla(unparsed_version: string): Description
 	{
 	local software_name = "<unknown browser>";
 	local v: Version;
-	local parts: table[count] of string;
+	local parts: string_vec;
 	
 	if ( /Opera [0-9\.]*$/ in unparsed_version )
 		{
 		software_name = "Opera";
-		parts = split_all(unparsed_version, /Opera [0-9\.]*$/);
+		parts = split_string_all(unparsed_version, /Opera [0-9\.]*$/);
-		if ( 2 in parts )
+		if ( 1 in parts )
-			v = parse(parts[2])$version;
+			v = parse(parts[1])$version;
 		}
 	else if ( / MSIE |Trident\// in unparsed_version )
 		{
@@ -222,28 +222,28 @@ function parse_mozilla(unparsed_version: string): Description
 			v = [$major=11,$minor=0];
 		else
 			{
-			parts = split_all(unparsed_version, /MSIE [0-9]{1,2}\.*[0-9]*b?[0-9]*/);
+			parts = split_string_all(unparsed_version, /MSIE [0-9]{1,2}\.*[0-9]*b?[0-9]*/);
-			if ( 2 in parts )
+			if ( 1 in parts )
-				v = parse(parts[2])$version;
+				v = parse(parts[1])$version;
 			}
 		}
 	else if ( /Version\/.*Safari\// in unparsed_version )
 		{
 		software_name = "Safari";
-		parts = split_all(unparsed_version, /Version\/[0-9\.]*/);
+		parts = split_string_all(unparsed_version, /Version\/[0-9\.]*/);
-		if ( 2 in parts )
+		if ( 1 in parts )
 			{
-			v = parse(parts[2])$version;
+			v = parse(parts[1])$version;
 			if ( / Mobile\/?.* Safari/ in unparsed_version )
 				v$addl = "Mobile";
 			}
 		}
 	else if ( /(Firefox|Netscape|Thunderbird)\/[0-9\.]*/ in unparsed_version )
 		{
-		parts = split_all(unparsed_version, /(Firefox|Netscape|Thunderbird)\/[0-9\.]*/);
+		parts = split_string_all(unparsed_version, /(Firefox|Netscape|Thunderbird)\/[0-9\.]*/);
-		if ( 2 in parts )
+		if ( 1 in parts )
 			{
-			local tmp_s = parse(parts[2]);
+			local tmp_s = parse(parts[1]);
 			software_name = tmp_s$name;
 			v = tmp_s$version;
 			}
@@ -251,48 +251,48 @@ function parse_mozilla(unparsed_version: string): Description
 	else if ( /Chrome\/.*Safari\// in unparsed_version )
 		{
 		software_name = "Chrome";
-		parts = split_all(unparsed_version, /Chrome\/[0-9\.]*/);
+		parts = split_string_all(unparsed_version, /Chrome\/[0-9\.]*/);
-		if ( 2 in parts )
+		if ( 1 in parts )
-			v = parse(parts[2])$version;
+			v = parse(parts[1])$version;
 		}
 	else if ( /^Opera\// in unparsed_version )
 		{
 		if ( /Opera M(ini|obi)\// in unparsed_version )
 			{
-			parts = split_all(unparsed_version, /Opera M(ini|obi)/);
+			parts = split_string_all(unparsed_version, /Opera M(ini|obi)/);
-			if ( 2 in parts )
+			if ( 1 in parts )
-				software_name = parts[2];
+				software_name = parts[1];
-			parts = split_all(unparsed_version, /Version\/[0-9\.]*/);
+			parts = split_string_all(unparsed_version, /Version\/[0-9\.]*/);
-			if ( 2 in parts )
+			if ( 1 in parts )
-				v = parse(parts[2])$version;
+				v = parse(parts[1])$version;
 			else
 				{
-				parts = split_all(unparsed_version, /Opera Mini\/[0-9\.]*/);
+				parts = split_string_all(unparsed_version, /Opera Mini\/[0-9\.]*/);
-				if ( 2 in parts )
+				if ( 1 in parts )
-					v = parse(parts[2])$version;
+					v = parse(parts[1])$version;
 				}
 			}
 		else
 			{
 			software_name = "Opera";
-			parts = split_all(unparsed_version, /Version\/[0-9\.]*/);
+			parts = split_string_all(unparsed_version, /Version\/[0-9\.]*/);
-			if ( 2 in parts )
+			if ( 1 in parts )
-				v = parse(parts[2])$version;
+				v = parse(parts[1])$version;
 			}
 		}
 	else if ( /AppleWebKit\/[0-9\.]*/ in unparsed_version )
 		{
 		software_name = "Unspecified WebKit";
-		parts = split_all(unparsed_version, /AppleWebKit\/[0-9\.]*/);
+		parts = split_string_all(unparsed_version, /AppleWebKit\/[0-9\.]*/);
-		if ( 2 in parts )
+		if ( 1 in parts )
-			v = parse(parts[2])$version;
+			v = parse(parts[1])$version;
 		}
 	else if ( / Java\/[0-9]\./ in unparsed_version )
 		{
 		software_name = "Java";
-		parts = split_all(unparsed_version, /Java\/[0-9\._]*/);
+		parts = split_string_all(unparsed_version, /Java\/[0-9\._]*/);
-		if ( 2 in parts )
+		if ( 1 in parts )
-			v = parse(parts[2])$version;
+			v = parse(parts[1])$version;
 		}
 
 	return [$version=v, $unparsed_version=unparsed_version, $name=software_name];

scripts/base/init-bare.bro

@@ -353,9 +353,10 @@ type connection: record {
 ## gives up and discards any internal state related to the file.
 const default_file_timeout_interval: interval = 2 mins &redef;
 
-## Default amount of bytes that file analysis will buffer before raising
+## Default amount of bytes that file analysis will buffer in order to use
-## :bro:see:`file_new`.
+## for mime type matching.  File analyzers attached at the time of mime type
-const default_file_bof_buffer_size: count = 1024 &redef;
+## matching or later, will receive a copy of this buffer.
+const default_file_bof_buffer_size: count = 4096 &redef;
 
 ## A file that Bro is analyzing.  This is Bro's type for describing the basic
 ## internal metadata collected about a "file", which is essentially just a
@@ -394,8 +395,10 @@ type fa_file: record {
 	## during the process of analysis e.g. due to dropped packets.
 	missing_bytes: count &default=0;
 
-	## The number of not all-in-sequence bytes in the file stream that
+	## The number of bytes in the file stream that were not delivered to
-	## were delivered to file analyzers due to reassembly buffer overflow.
+	## stream file analyzers.  Generally, this consists of bytes that
+	## couldn't be reassembled, either because reassembly simply isn't
+	## enabled, or due to size limitations of the reassembly buffer.
 	overflow_bytes: count &default=0;
 
 	## The amount of time between receiving new data for this file that
@@ -409,16 +412,6 @@ type fa_file: record {
 	## The content of the beginning of a file up to *bof_buffer_size* bytes.
 	## This is also the buffer that's used for file/mime type detection.
 	bof_buffer: string &optional;
-
-	## The mime type of the strongest file magic signature matches against
-	## the data chunk in *bof_buffer*, or in the cases where no buffering
-	## of the beginning of file occurs, an initial guess of the mime type
-	## based on the first data seen.
-	mime_type: string &optional;
-
-	## All mime types that matched file magic signatures against the data
-	## chunk in *bof_buffer*, in order of their strength value.
-	mime_types: mime_matches &optional;
 } &redef;
 
 ## Fields of a SYN packet.

scripts/base/protocols/dhcp/utils.bro

@@ -13,7 +13,7 @@ export {
 
 function reverse_ip(ip: addr): addr
 	{
-	local octets = split(cat(ip), /\./);
+	local octets = split_string(cat(ip), /\./);
-	return to_addr(cat(octets[4], ".", octets[3], ".", octets[2], ".", octets[1]));
+	return to_addr(cat(octets[3], ".", octets[2], ".", octets[1], ".", octets[0]));
 	}
 

scripts/base/protocols/dnp3/dpd.sig

@@ -5,5 +5,11 @@ signature dpd_dnp3_server {
 	ip-proto == tcp
 	payload /\x05\x64/
 	tcp-state responder
- 	enable "dnp3"
+ 	enable "dnp3_tcp"
+}
+
+signature dpd_dnp3_server_udp {
+	ip-proto == udp
+	payload /\x05\x64/
+	enable "dnp3_udp"
 }

scripts/base/protocols/dnp3/main.bro

@@ -31,16 +31,16 @@ redef record connection += {
 	dnp3: Info &optional;
 };
 
-const ports = { 20000/tcp };
+const ports = { 20000/tcp , 20000/udp };
 redef likely_server_ports += { ports };
 
 event bro_init() &priority=5
 	{
 	Log::create_stream(DNP3::LOG, [$columns=Info, $ev=log_dnp3]);
-	Analyzer::register_for_ports(Analyzer::ANALYZER_DNP3, ports);
+	Analyzer::register_for_ports(Analyzer::ANALYZER_DNP3_TCP, ports);
 	}
 
-event dnp3_application_request_header(c: connection, is_orig: bool, fc: count)
+event dnp3_application_request_header(c: connection, is_orig: bool, application_control: count, fc: count)
 	{
 	if ( ! c?$dnp3 )
 		c$dnp3 = [$ts=network_time(), $uid=c$uid, $id=c$id];
@@ -49,7 +49,7 @@ event dnp3_application_request_header(c: connection, is_orig: bool, fc: count)
 	c$dnp3$fc_request = function_codes[fc];
 	}
 
-event dnp3_application_response_header(c: connection, is_orig: bool, fc: count, iin: count)
+event dnp3_application_response_header(c: connection, is_orig: bool, application_control: count, fc: count, iin: count)
 	{
 	if ( ! c?$dnp3 )
 		c$dnp3 = [$ts=network_time(), $uid=c$uid, $id=c$id];

scripts/base/protocols/ftp/files.bro

@@ -17,6 +17,10 @@ export {
 
 	## Describe the file being transferred.
 	global describe_file: function(f: fa_file): string;
+
+	redef record fa_file += { 
+		ftp: FTP::Info &optional;
+	};
 }
 
 function get_file_handle(c: connection, is_orig: bool): string
@@ -48,7 +52,6 @@ event bro_init() &priority=5
 	                          $describe        = FTP::describe_file]);
 	}
 
-
 event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) &priority=5
 	{
 	if ( [c$id$resp_h, c$id$resp_p] !in ftp_data_expected ) 
@@ -56,6 +59,14 @@ event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) &priori
 
 	local ftp = ftp_data_expected[c$id$resp_h, c$id$resp_p];
 	ftp$fuid = f$id;
-	if ( f?$mime_type )
+
-		ftp$mime_type = f$mime_type;
+	f$ftp = ftp;
+	}
+
+event file_mime_type(f: fa_file, mime_type: string) &priority=5
+	{
+	if ( ! f?$ftp )
+		return;
+
+	f$ftp$mime_type = mime_type;
 	}

scripts/base/protocols/ftp/main.bro

@@ -274,7 +274,7 @@ event file_transferred(c: connection, prefix: string, descr: string,
 	if ( [id$resp_h, id$resp_p] in ftp_data_expected )
 		{
 		local s = ftp_data_expected[id$resp_h, id$resp_p];
-		s$mime_type = split1(mime_type, /;/)[1];
+		s$mime_type = split_string1(mime_type, /;/)[0];
 		}
 	}
 

scripts/base/protocols/http/entities.bro

@@ -35,6 +35,10 @@ export {
 		## body.
 		resp_mime_depth: count            &default=0;
 	};
+
+	redef record fa_file += {
+		http: HTTP::Info &optional;
+	};
 }
 
 event http_begin_entity(c: connection, is_orig: bool) &priority=10
@@ -67,6 +71,8 @@ event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) &priori
 	{
 	if ( f$source == "HTTP" && c?$http ) 
 		{
+		f$http = c$http;
+
 		if ( c$http?$current_entity && c$http$current_entity?$filename )
 			f$info$filename = c$http$current_entity$filename;
 
@@ -76,14 +82,6 @@ event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) &priori
 				c$http$orig_fuids = string_vec(f$id);
 			else
 				c$http$orig_fuids[|c$http$orig_fuids|] = f$id;
-
-			if ( f?$mime_type )
-				{
-				if ( ! c$http?$orig_mime_types )
-					c$http$orig_mime_types = string_vec(f$mime_type);
-				else
-					c$http$orig_mime_types[|c$http$orig_mime_types|] = f$mime_type;
-				}
 			}
 		else
 			{
@@ -91,17 +89,29 @@ event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) &priori
 				c$http$resp_fuids = string_vec(f$id);
 			else
 				c$http$resp_fuids[|c$http$resp_fuids|] = f$id;
-
-			if ( f?$mime_type )
-				{
-				if ( ! c$http?$resp_mime_types )
-					c$http$resp_mime_types = string_vec(f$mime_type);
-				else
-					c$http$resp_mime_types[|c$http$resp_mime_types|] = f$mime_type;
-				}
 			}
 		}
+	}
 
+event file_mime_type(f: fa_file, mime_type: string) &priority=5
+	{
+	if ( ! f?$http || ! f?$is_orig )
+		return;
+
+	if ( f$is_orig )
+		{
+		if ( ! f$http?$orig_mime_types )
+			f$http$orig_mime_types = string_vec(mime_type);
+		else
+			f$http$orig_mime_types[|f$http$orig_mime_types|] = mime_type;
+		}
+	else
+		{
+		if ( ! f$http?$resp_mime_types )
+			f$http$resp_mime_types = string_vec(mime_type);
+		else
+			f$http$resp_mime_types[|f$http$resp_mime_types|] = mime_type;
+		}
 	}
 
 event http_end_entity(c: connection, is_orig: bool) &priority=5

scripts/base/protocols/http/main.bro

@@ -242,7 +242,7 @@ event http_header(c: connection, is_orig: bool, name: string, value: string) &pr
 
 		else if ( name == "HOST" )
 			# The split is done to remove the occasional port value that shows up here.
-			c$http$host = split1(value, /:/)[1];
+			c$http$host = split_string1(value, /:/)[0];
 
 		else if ( name == "RANGE" )
 			c$http$range_request = T;
@@ -262,12 +262,12 @@ event http_header(c: connection, is_orig: bool, name: string, value: string) &pr
 			if ( /^[bB][aA][sS][iI][cC] / in value )
 				{
 				local userpass = decode_base64(sub(value, /[bB][aA][sS][iI][cC][[:blank:]]/, ""));
-				local up = split(userpass, /:/);
+				local up = split_string(userpass, /:/);
 				if ( |up| >= 2 )
 					{
-					c$http$username = up[1];
+					c$http$username = up[0];
 					if ( c$http$capture_password )
-						c$http$password = up[2];
+						c$http$password = up[1];
 					}
 				else
 					{

scripts/base/protocols/http/utils.bro

@@ -42,12 +42,12 @@ function extract_keys(data: string, kv_splitter: pattern): string_vec
 	{
 	local key_vec: vector of string = vector();
 	
-	local parts = split(data, kv_splitter);
+	local parts = split_string(data, kv_splitter);
 	for ( part_index in parts )
 		{
-		local key_val = split1(parts[part_index], /=/);
+		local key_val = split_string1(parts[part_index], /=/);
-		if ( 1 in key_val )
+		if ( 0 in key_val )
-			key_vec[|key_vec|] = key_val[1];
+			key_vec[|key_vec|] = key_val[0];
 		}
 	return key_vec;
 	}

scripts/base/protocols/irc/files.bro

@@ -12,6 +12,10 @@ export {
 
 	## Default file handle provider for IRC.
 	global get_file_handle: function(c: connection, is_orig: bool): string;
+
+	redef record fa_file += {
+		irc: IRC::Info &optional;
+	};
 }
 
 function get_file_handle(c: connection, is_orig: bool): string
@@ -34,6 +38,12 @@ event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) &priori
 	irc$fuid = f$id;
 	if ( irc?$dcc_file_name )
 		f$info$filename = irc$dcc_file_name;
-	if ( f?$mime_type )
+
-		irc$dcc_mime_type = f$mime_type;
+	f$irc = irc;
 	}
+
+event file_mime_type(f: fa_file, mime_type: string) &priority=5
+	{
+	if ( f?$irc )
+		f$irc$dcc_mime_type = mime_type;
+	}

scripts/base/protocols/mysql/main.bro

@@ -18,8 +18,10 @@ export {
 		cmd:	string	&log;
 		## The argument issued to the command
 		arg:	string	&log;
-		## The result (error, OK, etc.) from the server
+		## Did the server tell us that the command succeeded?
-		result: string &log &optional;
+		success: bool &log &optional;
+		## The number of affected rows, if any
+		rows: count &log &optional;
 		## Server message, if any
 		response: string &log &optional;
 	};
@@ -57,16 +59,21 @@ event mysql_handshake(c: connection, username: string)
 
 event mysql_command_request(c: connection, command: count, arg: string) &priority=5
 	{
-	if ( ! c?$mysql )
+	if ( c?$mysql )
 		{
-		local info: Info;
+		# We got a request, but we haven't logged our
-		info$ts = network_time();
+		# previous request yet, so let's do that now.
-		info$uid = c$uid;
+		Log::write(mysql::LOG, c$mysql);
-		info$id = c$id;
+		delete c$mysql;
-		info$cmd = commands[command];
-		info$arg = sub(arg, /\0$/, "");
-		c$mysql = info;
 		}
+
+	local info: Info;
+	info$ts = network_time();
+	info$uid = c$uid;
+	info$id = c$id;
+	info$cmd = commands[command];
+	info$arg = sub(arg, /\0$/, "");
+	c$mysql = info;
 	}
 
 event mysql_command_request(c: connection, command: count, arg: string) &priority=-5
@@ -83,7 +90,7 @@ event mysql_error(c: connection, code: count, msg: string) &priority=5
 	{
 	if ( c?$mysql )
 		{
-		c$mysql$result = "error";
+		c$mysql$success = F;
 		c$mysql$response = msg;
 		}
 	}
@@ -101,8 +108,8 @@ event mysql_ok(c: connection, affected_rows: count) &priority=5
 	{
 	if ( c?$mysql )
 		{
-		c$mysql$result = "ok";
+		c$mysql$success = T;
-		c$mysql$response = fmt("Affected rows: %d", affected_rows);
+		c$mysql$rows = affected_rows;
 		}
 	}
 
@@ -114,3 +121,12 @@ event mysql_ok(c: connection, affected_rows: count) &priority=-5
 		delete c$mysql;
 		}
 	}
+
+event connection_state_remove(c: connection) &priority=-5
+	{
+	if ( c?$mysql )
+		{
+		Log::write(mysql::LOG, c$mysql);
+		delete c$mysql;
+		}
+	}

scripts/base/protocols/smtp/main.bro

@@ -98,7 +98,7 @@ event bro_init() &priority=5
 
 function find_address_in_smtp_header(header: string): string
 {
-	local ips = find_ip_addresses(header);
+	local ips = extract_ip_addresses(header);
 	# If there are more than one IP address found, return the second.
 	if ( |ips| > 1 )
 		return ips[1];
@@ -163,7 +163,7 @@ event smtp_request(c: connection, is_orig: bool, command: string, arg: string) &
 		{
 		if ( ! c$smtp?$rcptto )
 			c$smtp$rcptto = set();
-		add c$smtp$rcptto[split1(arg, /:[[:blank:]]*/)[2]];
+		add c$smtp$rcptto[split_string1(arg, /:[[:blank:]]*/)[1]];
 		c$smtp$has_client_activity = T;
 		}
 
@@ -172,8 +172,8 @@ event smtp_request(c: connection, is_orig: bool, command: string, arg: string) &
 		# Flush last message in case we didn't see the server's acknowledgement.
 		smtp_message(c);
 
-		local partially_done = split1(arg, /:[[:blank:]]*/)[2];
+		local partially_done = split_string1(arg, /:[[:blank:]]*/)[1];
-		c$smtp$mailfrom = split1(partially_done, /[[:blank:]]?/)[1];
+		c$smtp$mailfrom = split_string1(partially_done, /[[:blank:]]?/)[0];
 		c$smtp$has_client_activity = T;
 		}
 	}
@@ -234,14 +234,14 @@ event mime_one_header(c: connection, h: mime_header_rec) &priority=5
 		if ( ! c$smtp?$to )
 			c$smtp$to = set();
 
-		local to_parts = split(h$value, /[[:blank:]]*,[[:blank:]]*/);
+		local to_parts = split_string(h$value, /[[:blank:]]*,[[:blank:]]*/);
 		for ( i in to_parts )
 			add c$smtp$to[to_parts[i]];
 		}
 
 	else if ( h$name == "X-ORIGINATING-IP" )
 		{
-		local addresses = find_ip_addresses(h$value);
+		local addresses = extract_ip_addresses(h$value);
 		if ( 1 in addresses )
 			c$smtp$x_originating_ip = to_addr(addresses[1]);
 		}

scripts/base/protocols/socks/main.bro

@@ -16,8 +16,10 @@ export {
 		id:          conn_id         &log;
 		## Protocol version of SOCKS.
 		version:     count           &log;
-		## Username for the proxy if extracted from the network.
+		## Username used to request a login to the proxy.
 		user:        string          &log &optional;
+		## Password used to request a login to the proxy.
+		password:    string          &log &optional;
 		## Server status for the attempt at using the proxy.
 		status:      string          &log &optional;
 		## Client requested SOCKS address. Could be an address, a name
@@ -91,3 +93,21 @@ event socks_reply(c: connection, version: count, reply: count, sa: SOCKS::Addres
 	if ( "SOCKS" in c$service )
 		Log::write(SOCKS::LOG, c$socks);
 	}
+
+event socks_login_userpass_request(c: connection, user: string, password: string) &priority=5
+	{
+	# Authentication only possible with the version 5.
+	set_session(c, 5); 
+
+	c$socks$user = user;
+	c$socks$password = password;
+	}
+
+event socks_login_userpass_reply(c: connection, code: count) &priority=5
+	{
+	# Authentication only possible with the version 5.
+	set_session(c, 5);
+
+	c$socks$status = v5_status[code];
+	}
+

scripts/base/protocols/ssl/consts.bro

@@ -158,12 +158,11 @@ export {
 		[26] = "brainpoolP256r1",
 		[27] = "brainpoolP384r1",
 		[28] = "brainpoolP512r1",
-		# draft-ietf-tls-negotiated-ff-dhe-02
+		# draft-ietf-tls-negotiated-ff-dhe-05
-		[256] = "ffdhe2432",
+		[256] = "ffdhe2048",
 		[257] = "ffdhe3072",
 		[258] = "ffdhe4096",
-		[259] = "ffdhe6144",
+		[259] = "ffdhe8192",
-		[260] = "ffdhe8192",
 		[0xFF01] = "arbitrary_explicit_prime_curves",
 		[0xFF02] = "arbitrary_explicit_char2_curves"
 	} &default=function(i: count):string { return fmt("unknown-%d", i); };

scripts/base/utils/active-http.bro

@@ -105,21 +105,21 @@ function request(req: Request): ActiveHTTP::Response
 			# The reply is the first line.
 			if ( i == 0 )
 				{
-				local response_line = split_n(headers[0], /[[:blank:]]+/, F, 2);
+				local response_line = split_string_n(headers[0], /[[:blank:]]+/, F, 2);
 				if ( |response_line| != 3 )
 					return resp;
 
-				resp$code = to_count(response_line[2]);
+				resp$code = to_count(response_line[1]);
-				resp$msg = response_line[3];
+				resp$msg = response_line[2];
 				resp$body = join_string_vec(result$files[bodyfile], "");
 				}
 			else
 				{
 				local line = headers[i];
-				local h = split1(line, /:/);
+				local h = split_string1(line, /:/);
 				if ( |h| != 2 )
 					next;
-				resp$headers[h[1]] = sub_bytes(h[2], 0, |h[2]|-1);
+				resp$headers[h[0]] = sub_bytes(h[1], 0, |h[1]|-1);
 				}
 			}
 		return resp;

scripts/base/utils/addrs.bro

@@ -32,7 +32,7 @@ const ip_addr_regex =
 ## octets: an array of strings to check for valid octet values.
 ##
 ## Returns: T if every element is between 0 and 255, inclusive, else F.
-function has_valid_octets(octets: string_array): bool
+function has_valid_octets(octets: string_vec): bool
 	{
 	local num = 0;
 	for ( i in octets )
@@ -51,10 +51,10 @@ function has_valid_octets(octets: string_array): bool
 ## Returns: T if the string is a valid IPv4 or IPv6 address format.
 function is_valid_ip(ip_str: string): bool
 	{
-	local octets: string_array;
+	local octets: string_vec;
 	if ( ip_str == ipv4_addr_regex )
 		{
-		octets = split(ip_str, /\./);
+		octets = split_string(ip_str, /\./);
 		if ( |octets| != 4 )
 			return F;
 
@@ -67,13 +67,13 @@ function is_valid_ip(ip_str: string): bool
 			{
 			# the regexes for hybrid IPv6-IPv4 address formats don't for valid
 			# octets within the IPv4 part, so do that now
-			octets = split(ip_str, /\./);
+			octets = split_string(ip_str, /\./);
 			if ( |octets| != 4 )
 				return F;
 
 			# get rid of remaining IPv6 stuff in first octet
-			local tmp = split(octets[1], /:/);
+			local tmp = split_string(octets[0], /:/);
-			octets[1] = tmp[|tmp|];
+			octets[0] = tmp[|tmp| - 1];
 
 			return has_valid_octets(octets);
 			}
@@ -92,14 +92,32 @@ function is_valid_ip(ip_str: string): bool
 ## input: a string that may contain an IP address anywhere within it.
 ##
 ## Returns: an array containing all valid IP address strings found in *input*.
-function find_ip_addresses(input: string): string_array
+function find_ip_addresses(input: string): string_array &deprecated
 	{
-	local parts = split_all(input, ip_addr_regex);
+	local parts = split_string_all(input, ip_addr_regex);
 	local output: string_array;
 
 	for ( i in parts )
 		{
-		if ( i % 2 == 0 && is_valid_ip(parts[i]) )
+		if ( i % 2 == 1 && is_valid_ip(parts[i]) )
+			output[|output|] = parts[i];
+		}
+	return output;
+	}
+
+## Extracts all IP (v4 or v6) address strings from a given string.
+##
+## input: a string that may contain an IP address anywhere within it.
+##
+## Returns: an array containing all valid IP address strings found in *input*.
+function extract_ip_addresses(input: string): string_vec
+	{
+	local parts = split_string_all(input, ip_addr_regex);
+	local output: string_vec;
+
+	for ( i in parts )
+		{
+		if ( i % 2 == 1 && is_valid_ip(parts[i]) )
 			output[|output|] = parts[i];
 		}
 	return output;

scripts/base/utils/exec.bro

@@ -82,9 +82,9 @@ event Exec::line(description: Input::EventDescription, tpe: Input::Event, s: str
 
 event Exec::file_line(description: Input::EventDescription, tpe: Input::Event, s: string)
 	{
-	local parts = split1(description$name, /_/);
+	local parts = split_string1(description$name, /_/);
-	local name = parts[1];
+	local name = parts[0];
-	local track_file = parts[2];
+	local track_file = parts[1];
 
 	local result = results[name];
 	if ( ! result?$files )
@@ -96,15 +96,16 @@ event Exec::file_line(description: Input::EventDescription, tpe: Input::Event, s
 		result$files[track_file][|result$files[track_file]|] = s;
 	}
 
-event Input::end_of_data(name: string, source:string)
+event Input::end_of_data(orig_name: string, source:string)
 	{
-	local parts = split1(name, /_/);
+	local name = orig_name;
-	name = parts[1];
+	local parts = split_string1(name, /_/);
+	name = parts[0];
 
 	if ( name !in pending_commands || |parts| < 2 )
 		return;
 
-	local track_file = parts[2];
+	local track_file = parts[1];
 
 	# If the file is empty, still add it to the result$files table. This is needed
 	# because it is expected that the file was read even if it was empty.

scripts/base/utils/files.bro

@@ -23,7 +23,7 @@ function extract_filename_from_content_disposition(data: string): string
 
 	# Remove quotes around the filename if they are there.
 	if ( /^\"/ in filename )
-		filename = split_n(filename, /\"/, F, 2)[2];
+		filename = split_string_n(filename, /\"/, F, 2)[1];
 
 	# Remove the language and encoding if it's there.
 	if ( /^[a-zA-Z0-9\!#$%&+-^_`{}~]+'[a-zA-Z0-9\!#$%&+-^_`{}~]*'/ in filename )

scripts/base/utils/numbers.bro

@@ -2,9 +2,9 @@
 ## If no integer can be found, 0 is returned.
 function extract_count(s: string): count
 	{
-	local parts = split_n(s, /[0-9]+/, T, 1);
+	local parts = split_string_n(s, /[0-9]+/, T, 1);
-	if ( 2 in parts )
+	if ( 1 in parts )
-		return to_count(parts[2]);
+		return to_count(parts[1]);
 	else
 		return 0;
-	}
+	}

scripts/base/utils/paths.bro

@@ -13,12 +13,12 @@ const absolute_path_pat = /(\/|[A-Za-z]:[\\\/]).*/;
 function extract_path(input: string): string
 	{
 	const dir_pattern = /(\/|[A-Za-z]:[\\\/])([^\"\ ]|(\\\ ))*/;
-	local parts = split_all(input, dir_pattern);
+	local parts = split_string_all(input, dir_pattern);
 
 	if ( |parts| < 3 )
 		return "";
 
-	return parts[2];
+	return parts[1];
 	}
 
 ## Compresses a given path by removing '..'s and the parent directory it
@@ -31,27 +31,27 @@ function compress_path(dir: string): string
 	{
 	const cdup_sep = /((\/)*([^\/]|\\\/)+)?((\/)+\.\.(\/)*)/;
 
-	local parts = split_n(dir, cdup_sep, T, 1);
+	local parts = split_string_n(dir, cdup_sep, T, 1);
 	if ( |parts| > 1 )
 		{
 		# reaching a point with two parent dir references back-to-back means
 		# we don't know about anything higher in the tree to pop off
-		if ( parts[2] == "../.." )
+		if ( parts[1] == "../.." )
-			return cat_string_array(parts);
+			return join_string_vec(parts, "");
-		if ( sub_bytes(parts[2], 0, 1) == "/" )
+		if ( sub_bytes(parts[1], 0, 1) == "/" )
-			parts[2] = "/";
+			parts[1] = "/";
 		else
-			parts[2] = "";
+			parts[1] = "";
-		dir = cat_string_array(parts);
+		dir = join_string_vec(parts, "");
 		return compress_path(dir);
 		}
 
 	const multislash_sep = /(\/\.?){2,}/;
-	parts = split_all(dir, multislash_sep);
+	parts = split_string_all(dir, multislash_sep);
 	for ( i in parts )
-		if ( i % 2 == 0 )
+		if ( i % 2 == 1 )
 			parts[i] = "/";
-	dir = cat_string_array(parts);
+	dir = join_string_vec(parts, "");
 
 	# remove trailing slashes from path
 	if ( |dir| > 1 && sub_bytes(dir, |dir|, 1) == "/" )

scripts/base/utils/patterns.bro

@@ -50,11 +50,11 @@ type PatternMatchResult: record {
 ## Returns: a record indicating the match status.
 function match_pattern(s: string, p: pattern): PatternMatchResult
 	{
-	local a = split_n(s, p, T, 1);
+	local a = split_string_n(s, p, T, 1);
 
 	if ( |a| == 1 )
 		# no match
 		return [$matched = F, $str = "", $off = 0];
 	else
-		return [$matched = T, $str = a[2], $off = |a[1]| + 1];
+		return [$matched = T, $str = a[1], $off = |a[0]| + 1];
 	}

scripts/base/utils/urls.bro

@@ -48,7 +48,7 @@ function find_all_urls_without_scheme(s: string): string_set
 
 function decompose_uri(s: string): URI
 	{
-	local parts: string_array;
+	local parts: string_vec;
 	local u: URI = [$netlocation="", $path="/"];
 
 	if ( /\?/ in s)
@@ -56,55 +56,55 @@ function decompose_uri(s: string): URI
 		# Parse query.
 		u$params = table();
 
-		parts = split1(s, /\?/);
+		parts = split_string1(s, /\?/);
-		s = parts[1];
+		s = parts[0];
-		local query: string = parts[2];
+		local query: string = parts[1];
 
 		if ( /&/ in query )
 			{
-			local opv: table[count] of string = split(query, /&/);
+			local opv = split_string(query, /&/);
 
 			for ( each in opv )
 				{
 				if ( /=/ in opv[each] )
 					{
-					parts = split1(opv[each], /=/);
+					parts = split_string1(opv[each], /=/);
-					u$params[parts[1]] = parts[2];
+					u$params[parts[0]] = parts[1];
 					}
 				}
 			}
 		else
 			{
-			parts = split1(query, /=/);
+			parts = split_string1(query, /=/);
-			u$params[parts[1]] = parts[2];
+			u$params[parts[0]] = parts[1];
 			}
 		}
 
 	if ( /:\/\// in s )
 		{
 		# Parse scheme and remove from s.
-		parts = split1(s, /:\/\//);
+		parts = split_string1(s, /:\/\//);
-		u$scheme = parts[1];
+		u$scheme = parts[0];
-		s = parts[2];
+		s = parts[1];
 		}
 
 	if ( /\// in s )
 		{
 		# Parse path and remove from s.
-		parts = split1(s, /\//);
+		parts = split_string1(s, /\//);
-		s = parts[1];
+		s = parts[0];
-		u$path = fmt("/%s", parts[2]);
+		u$path = fmt("/%s", parts[1]);
 
 		if ( |u$path| > 1 && u$path[|u$path| - 1] != "/" )
 			{
 			local last_token: string = find_last(u$path, /\/.+/);
-			local full_filename = split1(last_token, /\//)[2];
+			local full_filename = split_string1(last_token, /\//)[1];
 
 			if ( /\./ in full_filename )
 				{
 				u$file_name = full_filename;
-				u$file_base = split1(full_filename, /\./)[1];
+				u$file_base = split_string1(full_filename, /\./)[0];
-				u$file_ext = split1(full_filename, /\./)[2];
+				u$file_ext = split_string1(full_filename, /\./)[1];
 				}
 			else
 				{
@@ -117,9 +117,9 @@ function decompose_uri(s: string): URI
 	if ( /:/ in s )
 		{
 		# Parse location and port.
-		parts = split1(s, /:/);
+		parts = split_string1(s, /:/);
-		u$netlocation = parts[1];
+		u$netlocation = parts[0];
-		u$portnum = to_count(parts[2]);
+		u$portnum = to_count(parts[1]);
 		}
 	else
 		u$netlocation = s;

scripts/policy/frameworks/files/detect-MHR.bro

@@ -42,15 +42,15 @@ function do_mhr_lookup(hash: string, fi: Notice::FileInfo)
 	when ( local MHR_result = lookup_hostname_txt(hash_domain) )
 		{
 		# Data is returned as "<dateFirstDetected> <detectionRate>"
-		local MHR_answer = split1(MHR_result, / /);
+		local MHR_answer = split_string1(MHR_result, / /);
 
 		if ( |MHR_answer| == 2 )
 			{
-			local mhr_detect_rate = to_count(MHR_answer[2]);
+			local mhr_detect_rate = to_count(MHR_answer[1]);
 
 			if ( mhr_detect_rate >= notice_threshold )
 				{
-				local mhr_first_detected = double_to_time(to_double(MHR_answer[1]));
+				local mhr_first_detected = double_to_time(to_double(MHR_answer[0]));
 				local readable_first_detected = strftime("%Y-%m-%d %H:%M:%S", mhr_first_detected);
 				local message = fmt("Malware Hash Registry Detection rate: %d%%  Last seen: %s", mhr_detect_rate, readable_first_detected);
 				local virustotal_url = fmt(match_sub_url, hash);
@@ -66,6 +66,7 @@ function do_mhr_lookup(hash: string, fi: Notice::FileInfo)
 
 event file_hash(f: fa_file, kind: string, hash: string)
 	{
-	if ( kind == "sha1" && f?$mime_type && match_file_types in f$mime_type )
+	if ( kind == "sha1" && f?$info && f$info?$mime_type && 
+	     match_file_types in f$info$mime_type )
 		do_mhr_lookup(hash, Notice::create_file_info(f));
 	}

scripts/policy/frameworks/intel/seen/http-headers.bro

@@ -31,7 +31,7 @@ event http_header(c: connection, is_orig: bool, name: string, value: string)
 			case "X-FORWARDED-FOR":
 			if ( is_valid_ip(value) )
 				{
-				local addrs = find_ip_addresses(value);
+				local addrs = extract_ip_addresses(value);
 				for ( i in addrs )
 					{
 					Intel::seen([$host=to_addr(addrs[i]),

scripts/policy/frameworks/intel/seen/smtp.bro

@@ -30,10 +30,10 @@ event mime_end_entity(c: connection)
 
 		if ( c$smtp?$mailfrom )
 			{
-			local mailfromparts = split_n(c$smtp$mailfrom, /<.+>/, T, 1);
+			local mailfromparts = split_string_n(c$smtp$mailfrom, /<.+>/, T, 1);
 			if ( |mailfromparts| > 2 )
 				{
-				Intel::seen([$indicator=mailfromparts[2][1:-2],
+				Intel::seen([$indicator=mailfromparts[1][1:-2],
 				             $indicator_type=Intel::EMAIL,
 				             $conn=c,
 				             $where=SMTP::IN_MAIL_FROM]);
@@ -44,10 +44,10 @@ event mime_end_entity(c: connection)
 			{
 			for ( rcptto in c$smtp$rcptto )
 				{
-				local rcpttoparts = split_n(rcptto, /<.+>/, T, 1);
+				local rcpttoparts = split_string_n(rcptto, /<.+>/, T, 1);
 				if ( |rcpttoparts| > 2 )
 					{
-					Intel::seen([$indicator=rcpttoparts[2][1:-2],
+					Intel::seen([$indicator=rcpttoparts[1][1:-2],
 					             $indicator_type=Intel::EMAIL,
 					             $conn=c,
 					             $where=SMTP::IN_RCPT_TO]);
@@ -57,10 +57,10 @@ event mime_end_entity(c: connection)
 
 		if ( c$smtp?$from )
 			{
-			local fromparts = split_n(c$smtp$from, /<.+>/, T, 1);
+			local fromparts = split_string_n(c$smtp$from, /<.+>/, T, 1);
 			if ( |fromparts| > 2 )
 				{
-				Intel::seen([$indicator=fromparts[2][1:-2],
+				Intel::seen([$indicator=fromparts[1][1:-2],
 				             $indicator_type=Intel::EMAIL,
 				             $conn=c,
 				             $where=SMTP::IN_FROM]);
@@ -71,10 +71,10 @@ event mime_end_entity(c: connection)
 			{
 			for ( email_to in c$smtp$to )
 				{
-				local toparts = split_n(email_to, /<.+>/, T, 1);
+				local toparts = split_string_n(email_to, /<.+>/, T, 1);
 				if ( |toparts| > 2 )
 					{
-					Intel::seen([$indicator=toparts[2][1:-2],
+					Intel::seen([$indicator=toparts[1][1:-2],
 					             $indicator_type=Intel::EMAIL,
 					             $conn=c,
 					             $where=SMTP::IN_TO]);
@@ -84,10 +84,10 @@ event mime_end_entity(c: connection)
 
 		if ( c$smtp?$reply_to )
 			{
-			local replytoparts = split_n(c$smtp$reply_to, /<.+>/, T, 1);
+			local replytoparts = split_string_n(c$smtp$reply_to, /<.+>/, T, 1);
 			if ( |replytoparts| > 2 )
 				{
-				Intel::seen([$indicator=replytoparts[2][1:-2],
+				Intel::seen([$indicator=replytoparts[1][1:-2],
 				             $indicator_type=Intel::EMAIL,
 				             $conn=c,
 				             $where=SMTP::IN_REPLY_TO]);

scripts/policy/frameworks/software/vulnerable.bro

@@ -55,18 +55,18 @@ function decode_vulnerable_version_range(vuln_sw: string): VulnerableVersionRang
 		return vvr;
 		}
 
-	local versions = split1(vuln_sw, /\x09/);
+	local versions = split_string1(vuln_sw, /\x09/);
 
 	for ( i in versions )
 		{
-		local field_and_ver = split1(versions[i], /=/);
+		local field_and_ver = split_string1(versions[i], /=/);
 		if ( |field_and_ver| != 2 )
 			return vvr; #failure!
 
-		local ver = Software::parse(field_and_ver[2])$version;
+		local ver = Software::parse(field_and_ver[1])$version;
-		if ( field_and_ver[1] == "min" )
+		if ( field_and_ver[0] == "min" )
 			vvr$min = ver;
-		else if ( field_and_ver[1] == "max" )
+		else if ( field_and_ver[0] == "max" )
 			vvr$max = ver;
 		}
 
@@ -84,15 +84,15 @@ event grab_vulnerable_versions(i: count)
 
 	when ( local result = lookup_hostname_txt(cat(i,".",vulnerable_versions_update_endpoint)) )
 		{
-		local parts = split1(result, /\x09/);
+		local parts = split_string1(result, /\x09/);
 		if ( |parts| != 2 ) #failure or end of list!
 			{
 			schedule vulnerable_versions_update_interval { grab_vulnerable_versions(1) };
 			return;
 			}
 
-		local sw = parts[1];
+		local sw = parts[0];
-		local vvr = decode_vulnerable_version_range(parts[2]);
+		local vvr = decode_vulnerable_version_range(parts[1]);
 		if ( sw !in internal_vulnerable_versions )
 			internal_vulnerable_versions[sw] = set();
 		add internal_vulnerable_versions[sw][vvr];

scripts/policy/frameworks/software/windows-version-detection.bro

@@ -0,0 +1,69 @@
+##! Windows systems access a Microsoft Certificate Revocation List (CRL) periodically. The
+##! user agent for these requests reveals which version of Crypt32.dll installed on the system,
+##! which can uniquely identify the version of Windows that's running.
+##!
+##! This script will log the version of Windows that was identified to the Software framework.
+
+@load base/protocols/http
+@load base/frameworks/software
+
+module OS;
+
+export {
+        redef enum Software::Type += {
+                ## Identifier for Windows operating system versions
+                WINDOWS,
+        };
+
+        type Software::name_and_version: record {
+                name   : string;
+                version: Software::Version;
+        };
+
+        const crypto_api_mapping: table[string] of Software::name_and_version = {
+                ["Microsoft-CryptoAPI/5.131.2195.6661"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2195, $minor3=6661, $addl="2000 SP4"]],
+                ["Microsoft-CryptoAPI/5.131.2195.6824"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2195, $minor3=6824, $addl="2000 with MS04-11"]],
+                ["Microsoft-CryptoAPI/5.131.2195.6926"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2195, $minor3=6926, $addl="2000 with Hotfix 98830"]],
+
+                ["Microsoft-CryptoAPI/5.131.2600.0"]    = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2600, $minor3=0,    $addl="XP SP0"]],
+                ["Microsoft-CryptoAPI/5.131.2600.1106"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2600, $minor3=1106, $addl="XP SP1"]],
+                ["Microsoft-CryptoAPI/5.131.2600.2180"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2600, $minor3=2180, $addl="XP SP2"]],
+                ["Microsoft-CryptoAPI/5.131.2600.3180"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2600, $minor3=3180, $addl="XP SP3 Beta 1"]],
+                ["Microsoft-CryptoAPI/5.131.2600.3205"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2600, $minor3=3205, $addl="XP SP3 Beta 2"]],
+                ["Microsoft-CryptoAPI/5.131.2600.3249"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2600, $minor3=3249, $addl="XP SP3 RC Beta"]],
+                ["Microsoft-CryptoAPI/5.131.2600.3264"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2600, $minor3=3264, $addl="XP SP3 RC1"]],
+                ["Microsoft-CryptoAPI/5.131.2600.3282"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2600, $minor3=3282, $addl="XP SP3 RC1 Update"]],
+                ["Microsoft-CryptoAPI/5.131.2600.3300"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2600, $minor3=3300, $addl="XP SP3 RC2"]],
+                ["Microsoft-CryptoAPI/5.131.2600.3311"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2600, $minor3=3311, $addl="XP SP3 RC2 Update"]],
+                ["Microsoft-CryptoAPI/5.131.2600.5508"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2600, $minor3=5508, $addl="XP SP3 RC2 Update 2"]],
+                ["Microsoft-CryptoAPI/5.131.2600.5512"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=2600, $minor3=5512, $addl="XP SP3"]],
+
+                ["Microsoft-CryptoAPI/5.131.3790.0"]    = [$name="Windows", $version=[$major=5, $minor=131, $minor2=3790, $minor3=0,    $addl="XP x64 or Server 2003 SP0"]],
+                ["Microsoft-CryptoAPI/5.131.3790.1830"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=3790, $minor3=1830, $addl="XP x64 or Server 2003 SP1"]],
+                ["Microsoft-CryptoAPI/5.131.3790.3959"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=3790, $minor3=3959, $addl="XP x64 or Server 2003 SP2"]],
+                ["Microsoft-CryptoAPI/5.131.3790.5235"] = [$name="Windows", $version=[$major=5, $minor=131, $minor2=3790, $minor3=5235, $addl="XP x64 or Server 2003 with MS13-095"]],
+
+                ["Microsoft-CryptoAPI/6.0"]             = [$name="Windows", $version=[$major=6, $minor=0, $addl="Vista or Server 2008"]],
+                ["Microsoft-CryptoAPI/6.1"]             = [$name="Windows", $version=[$major=6, $minor=1, $addl="7 or Server 2008 R2"]],
+                ["Microsoft-CryptoAPI/6.2"]             = [$name="Windows", $version=[$major=6, $minor=2, $addl="8 or Server 2012"]],
+                ["Microsoft-CryptoAPI/6.3"]             = [$name="Windows", $version=[$major=6, $minor=3, $addl="8.1 or Server 2012 R2"]],
+                ["Microsoft-CryptoAPI/6.4"]             = [$name="Windows", $version=[$major=6, $minor=4, $addl="10 Technical Preview"]],
+        } &redef;
+}
+
+event HTTP::log_http(rec: HTTP::Info) &priority=5
+        {
+        if ( rec?$host && rec?$user_agent && rec$host == "crl.microsoft.com" &&
+             /Microsoft-CryptoAPI\// in rec$user_agent )
+                {
+                if ( rec$user_agent !in crypto_api_mapping )
+                        {
+                        Software::found(rec$id, [$unparsed_version=sub(rec$user_agent, /Microsoft-CryptoAPI/, "Unknown CryptoAPI Version"), $host=rec$id$orig_h, $software_type=WINDOWS]);
+                        }
+                else
+                        {
+                        local result = crypto_api_mapping[rec$user_agent];
+                        Software::found(rec$id, [$version=result$version, $name=result$name, $host=rec$id$orig_h, $software_type=WINDOWS]);
+                        }
+                }
+        }

scripts/policy/misc/detect-traceroute/main.bro

@@ -74,10 +74,10 @@ event bro_init() &priority=5
 	                  $threshold=icmp_time_exceeded_threshold,
 	                  $threshold_crossed(key: SumStats::Key, result: SumStats::Result) =
 	                  	{
-	                  	local parts = split_n(key$str, /-/, F, 2);
+	                  	local parts = split_string_n(key$str, /-/, F, 2);
-	                  	local src = to_addr(parts[1]);
+	                  	local src = to_addr(parts[0]);
-	                  	local dst = to_addr(parts[2]);
+	                  	local dst = to_addr(parts[1]);
-	                  	local proto = parts[3];
+	                  	local proto = parts[2];
 	                  	Log::write(LOG, [$ts=network_time(), $src=src, $dst=dst, $proto=proto]);
 	                  	NOTICE([$note=Traceroute::Detected,
 	                  	        $msg=fmt("%s seems to be running traceroute using %s", src, proto),

scripts/policy/protocols/http/software-browser-plugins.bro

@@ -45,13 +45,13 @@ event log_http(rec: Info)
 	if ( rec$omniture && rec?$uri )
 		{
 		# We do {5,} because sometimes we see p=6 in the urls.
-		local parts = split_n(rec$uri, /&p=([^&]{5,});&/, T, 1);
+		local parts = split_string_n(rec$uri, /&p=([^&]{5,});&/, T, 1);
-		if ( 2 in parts )
+		if ( 1 in parts )
 			{
 			# We do sub_bytes here just to remove the extra extracted 
 			# characters from the regex split above.
-			local sw = sub_bytes(parts[2], 4, |parts[2]|-5);
+			local sw = sub_bytes(parts[1], 4, |parts[1]|-5);
-			local plugins = split(sw, /[[:blank:]]*;[[:blank:]]*/);
+			local plugins = split_string(sw, /[[:blank:]]*;[[:blank:]]*/);
 			
 			for ( i in plugins )
 				Software::found(rec$id, [$unparsed_version=plugins[i], $host=rec$id$orig_h, $software_type=BROWSER_PLUGIN]);

scripts/policy/protocols/smtp/blocklists.bro

@@ -47,7 +47,7 @@ event smtp_reply(c: connection, is_orig: bool, code: count, cmd: string,
 			local message = fmt("%s received an error message mentioning an SMTP block list", c$id$orig_h);
 
 			# Determine if the originator's IP address is in the message.
-			local ips = find_ip_addresses(msg);
+			local ips = extract_ip_addresses(msg);
 			local text_ip = "";
 			if ( |ips| > 0 && to_addr(ips[0]) == c$id$orig_h )
 				{

scripts/policy/protocols/ssl/notary.bro

@@ -70,23 +70,23 @@ event ssl_established(c: connection) &priority=3
 			clear_waitlist(digest);
 			return;
 			}
-		local fields = split(str, / /);
+		local fields = split_string(str, / /);
 		if ( |fields| != 5 ) # version 1 has 5 fields.
 			{
 			clear_waitlist(digest);
 			return;
 			}
-		local version = split(fields[1], /=/)[2];
+		local version = split_string(fields[0], /=/)[1];
 		if ( version != "1" )
 			{
 			clear_waitlist(digest);
 			return;
 			}
 		local r = notary_cache[digest];
-		r$first_seen = to_count(split(fields[2], /=/)[2]);
+		r$first_seen = to_count(split_string(fields[1], /=/)[1]);
-		r$last_seen = to_count(split(fields[3], /=/)[2]);
+		r$last_seen = to_count(split_string(fields[2], /=/)[1]);
-		r$times_seen = to_count(split(fields[4], /=/)[2]);
+		r$times_seen = to_count(split_string(fields[3], /=/)[1]);
-		r$valid = split(fields[5], /=/)[2] == "1";
+		r$valid = split_string(fields[4], /=/)[1] == "1";
 
 		# Assign notary answer to all records waiting for this digest.
 		if ( digest in waitlist )

scripts/test-all-policy.bro

@@ -32,6 +32,7 @@
 @load frameworks/packet-filter/shunt.bro
 @load frameworks/software/version-changes.bro
 @load frameworks/software/vulnerable.bro
+@load frameworks/software/windows-version-detection.bro
 @load integration/barnyard2/__load__.bro
 @load integration/barnyard2/main.bro
 @load integration/barnyard2/types.bro

src/Attr.cc

@@ -18,7 +18,7 @@ const char* attr_name(attr_tag t)
 		"&encrypt",
 		"&raw_output", "&mergeable", "&priority",
 		"&group", "&log", "&error_handler", "&type_column",
-		"(&tracked)",
+		"(&tracked)", "&deprecated",
 	};
 
 	return attr_names[int(t)];
@@ -212,6 +212,7 @@ void Attributes::DescribeReST(ODesc* d) const
 void Attributes::CheckAttr(Attr* a)
 	{
 	switch ( a->Tag() ) {
+	case ATTR_DEPRECATED:
 	case ATTR_OPTIONAL:
 	case ATTR_REDEF:
 		break;

src/Attr.h

@@ -34,7 +34,8 @@ typedef enum {
 	ATTR_ERROR_HANDLER,
 	ATTR_TYPE_COLUMN,	// for input framework
 	ATTR_TRACKED,	// hidden attribute, tracked by NotifierRegistry
-#define NUM_ATTRS (int(ATTR_TRACKED) + 1)
+	ATTR_DEPRECATED,
+#define NUM_ATTRS (int(ATTR_DEPRECATED) + 1)
 } attr_tag;
 
 class Attr : public BroObj {

src/Expr.cc

@@ -1438,7 +1438,7 @@ bool AddExpr::DoUnserialize(UnserialInfo* info)
 	}
 
 AddToExpr::AddToExpr(Expr* arg_op1, Expr* arg_op2)
-: BinaryExpr(EXPR_ADD_TO, arg_op1, arg_op2)
+: BinaryExpr(EXPR_ADD_TO, arg_op1->MakeLvalue(), arg_op2)
 	{
 	if ( IsError() )
 		return;
@@ -1562,7 +1562,7 @@ bool SubExpr::DoUnserialize(UnserialInfo* info)
 	}
 
 RemoveFromExpr::RemoveFromExpr(Expr* arg_op1, Expr* arg_op2)
-: BinaryExpr(EXPR_REMOVE_FROM, arg_op1, arg_op2)
+: BinaryExpr(EXPR_REMOVE_FROM, arg_op1->MakeLvalue(), arg_op2)
 	{
 	if ( IsError() )
 		return;
@@ -3213,6 +3213,10 @@ FieldExpr::FieldExpr(Expr* arg_op, const char* arg_field_name)
 			{
 			SetType(rt->FieldType(field)->Ref());
 			td = rt->FieldDecl(field);
+
+			if ( td->FindAttr(ATTR_DEPRECATED) )
+				reporter->Warning("deprecated (%s$%s)", rt->GetName().c_str(),
+				                  field_name);
 			}
 		}
 	}
@@ -3333,6 +3337,9 @@ HasFieldExpr::HasFieldExpr(Expr* arg_op, const char* arg_field_name)
 
 		if ( field < 0 )
 			ExprError("no such field in record");
+		else if ( rt->FieldDecl(field)->FindAttr(ATTR_DEPRECATED) )
+			reporter->Warning("deprecated (%s?$%s)", rt->GetName().c_str(),
+			                  field_name);
 
 		SetType(base_type(TYPE_BOOL));
 		}
@@ -4147,16 +4154,28 @@ RecordCoerceExpr::RecordCoerceExpr(Expr* op, RecordType* r)
 			}
 
 		for ( i = 0; i < map_size; ++i )
-			if ( map[i] == -1 &&
+			{
-			     ! t_r->FieldDecl(i)->FindAttr(ATTR_OPTIONAL) )
+			if ( map[i] == -1 )
 				{
-				char buf[512];
+				if ( ! t_r->FieldDecl(i)->FindAttr(ATTR_OPTIONAL) )
-				safe_snprintf(buf, sizeof(buf),
+					{
-					      "non-optional field \"%s\" missing", t_r->FieldName(i));
+					char buf[512];
-				Error(buf);
+					safe_snprintf(buf, sizeof(buf),
-				SetError();
+					              "non-optional field \"%s\" missing",
-				break;
+					              t_r->FieldName(i));
+					Error(buf);
+					SetError();
+					break;
+					}
 				}
+			else
+				{
+				if ( t_r->FieldDecl(i)->FindAttr(ATTR_DEPRECATED) )
+					reporter->Warning("deprecated (%s$%s)",
+					                  t_r->GetName().c_str(),
+					                  t_r->FieldName(i));
+				}
+			}
 		}
 	}
 

src/Frag.cc

@@ -28,7 +28,7 @@ void FragTimer::Dispatch(double t, int /* is_expire */)
 FragReassembler::FragReassembler(NetSessions* arg_s,
 			const IP_Hdr* ip, const u_char* pkt,
 			HashKey* k, double t)
-	: Reassembler(0, REASSEM_IP)
+	: Reassembler(0)
 	{
 	s = arg_s;
 	key = k;

src/Func.cc

@@ -335,7 +335,7 @@ int BroFunc::IsPure() const
 Val* BroFunc::Call(val_list* args, Frame* parent) const
 	{
 #ifdef PROFILE_BRO_FUNCTIONS
-	DEBUG_MSG("Function: %s\n", id->Name());
+	DEBUG_MSG("Function: %s\n", Name());
 #endif
 
 	SegmentProfiler(segment_logger, location);

src/ID.cc

@@ -248,6 +248,16 @@ void ID::UpdateValAttrs()
 		}
 	}
 
+void ID::MakeDeprecated()
+	{
+	if ( IsDeprecated() )
+		return;
+
+	attr_list* attr = new attr_list;
+	attr->append(new Attr(ATTR_DEPRECATED));
+	AddAttrs(new Attributes(attr, Type(), false));
+	}
+
 void ID::AddAttrs(Attributes* a)
 	{
 	if ( attrs )

src/ID.h

@@ -80,6 +80,11 @@ public:
 	Attr* FindAttr(attr_tag t) const
 		{ return attrs ? attrs->FindAttr(t) : 0; }
 
+	bool IsDeprecated() const
+		{ return FindAttr(ATTR_DEPRECATED) != 0; }
+
+	void MakeDeprecated();
+
 	void Error(const char* msg, const BroObj* o2 = 0);
 
 	void Describe(ODesc* d) const;

src/IP.cc

@@ -636,3 +636,51 @@ VectorVal* IPv6_Hdr_Chain::BuildVal() const
 
 	return rval;
 	}
+
+IP_Hdr* IP_Hdr::Copy() const
+	{
+	char* new_hdr = new char[HdrLen()];
+
+	if ( ip4 )
+		{
+		memcpy(new_hdr, ip4, HdrLen());
+		return new IP_Hdr((const struct ip*) new_hdr, true);
+		}
+
+	memcpy(new_hdr, ip6, HdrLen());
+	const struct ip6_hdr* new_ip6 = (const struct ip6_hdr*)new_hdr;
+	IPv6_Hdr_Chain* new_ip6_hdrs = ip6_hdrs->Copy(new_ip6);
+	return new IP_Hdr(new_ip6, true, 0, new_ip6_hdrs);
+	}
+
+IPv6_Hdr_Chain* IPv6_Hdr_Chain::Copy(const ip6_hdr* new_hdr) const
+	{
+	IPv6_Hdr_Chain* rval = new IPv6_Hdr_Chain;
+	rval->length = length;
+
+#ifdef ENABLE_MOBILE_IPV6
+	if ( homeAddr )
+		rval->homeAddr = new IPAddr(*homeAddr);
+#endif
+
+	if ( finalDst )
+		rval->finalDst = new IPAddr(*finalDst);
+
+	if ( chain.empty() )
+		{
+		reporter->InternalWarning("empty IPv6 header chain");
+		delete rval;
+		return 0;
+		}
+
+	const u_char* new_data = (const u_char*)new_hdr;
+	const u_char* old_data = chain[0]->Data();
+
+	for ( size_t i = 0; i < chain.size(); ++i )
+		{
+		int off = chain[i]->Data() - old_data;
+		rval->chain.push_back(new IPv6_Hdr(chain[i]->Type(), new_data + off));
+		}
+
+	return rval;
+	}

src/IP.h

@@ -157,6 +157,12 @@ public:
 		delete finalDst;
 		}
 
+	/**
+	 * @return a copy of the header chain, but with pointers to individual
+	 * IPv6 headers now pointing within \a new_hdr.
+	 */
+	IPv6_Hdr_Chain* Copy(const struct ip6_hdr* new_hdr) const;
+
 	/**
 	 * Returns the number of headers in the chain.
 	 */
@@ -264,6 +270,14 @@ protected:
 	// point to a fragment
 	friend class FragReassembler;
 
+	IPv6_Hdr_Chain() :
+		length(0),
+#ifdef ENABLE_MOBILE_IPV6
+		homeAddr(0),
+#endif
+		finalDst(0)
+		{}
+
 	/**
 	 * Initializes the header chain from an IPv6 header structure, and replaces
 	 * the first next protocol pointer field that points to a fragment header.
@@ -353,6 +367,13 @@ public:
 		{
 		}
 
+	/**
+	 * Copy a header.  The internal buffer which contains the header data
+	 * must not be truncated.  Also note that if that buffer points to a full
+	 * packet payload, only the IP header portion is copied.
+	 */
+	IP_Hdr* Copy() const;
+
 	/**
 	 * Destructor.
 	 */
@@ -554,6 +575,7 @@ public:
 	RecordVal* BuildPktHdrVal() const;
 
 private:
+
 	const struct ip* ip4;
 	const struct ip6_hdr* ip6;
 	bool del;

src/Reassem.cc

@@ -31,7 +31,7 @@ DataBlock::DataBlock(const u_char* data, uint64 size, uint64 arg_seq,
 
 uint64 Reassembler::total_size = 0;
 
-Reassembler::Reassembler(uint64 init_seq, ReassemblerType arg_type)
+Reassembler::Reassembler(uint64 init_seq)
 	{
 	blocks = last_block = 0;
 	trim_seq = last_reassem_seq = init_seq;

src/Reassem.h

@@ -22,11 +22,10 @@ public:
 };
 
 
-enum ReassemblerType { REASSEM_IP, REASSEM_TCP };
 
 class Reassembler : public BroObj {
 public:
-	Reassembler(uint64 init_seq, ReassemblerType arg_type);
+	Reassembler(uint64 init_seq);
 	virtual ~Reassembler();
 
 	void NewBlock(double t, uint64 seq, uint64 len, const u_char* data);

src/SerialTypes.h

@@ -87,6 +87,7 @@ SERIAL_TCP_CONTENTS(TCP_NVT, 3)
 #define SERIAL_REASSEMBLER(name, val) SERIAL_CONST(name, val, REASSEMBLER)
 SERIAL_REASSEMBLER(REASSEMBLER, 1)
 SERIAL_REASSEMBLER(TCP_REASSEMBLER, 2)
+SERIAL_REASSEMBLER(FILE_REASSEMBLER, 3)
 
 #define SERIAL_VAL(name, val) SERIAL_CONST(name, val, VAL)
 SERIAL_VAL(VAL, 1)
@@ -180,6 +181,7 @@ SERIAL_STMT(INIT_STMT, 17)
 SERIAL_STMT(NULL_STMT, 18)
 SERIAL_STMT(WHEN_STMT, 19)
 SERIAL_STMT(FALLTHROUGH_STMT, 20)
+SERIAL_STMT(WHILE_STMT, 21)
 
 #define SERIAL_TYPE(name, val) SERIAL_CONST(name, val, BRO_TYPE)
 SERIAL_TYPE(BRO_TYPE, 1)

src/Stmt.cc

@@ -23,7 +23,7 @@ const char* stmt_name(BroStmtTag t)
 		"print", "event", "expr", "if", "when", "switch",
 		"for", "next", "break", "return", "add", "delete",
 		"list", "bodylist",
-		"<init>", "fallthrough",
+		"<init>", "fallthrough", "while",
 		"null",
 	};
 
@@ -1127,6 +1127,126 @@ bool EventStmt::DoUnserialize(UnserialInfo* info)
 	return event_expr != 0;
 	}
 
+WhileStmt::WhileStmt(Expr* arg_loop_condition, Stmt* arg_body)
+	: loop_condition(arg_loop_condition), body(arg_body)
+	{
+	if ( ! loop_condition->IsError() &&
+	     ! IsBool(loop_condition->Type()->Tag()) )
+		loop_condition->Error("while conditional must be boolean");
+	}
+
+WhileStmt::~WhileStmt()
+	{
+	Unref(loop_condition);
+	Unref(body);
+	}
+
+int WhileStmt::IsPure() const
+	{
+	return loop_condition->IsPure() && body->IsPure();
+	}
+
+void WhileStmt::Describe(ODesc* d) const
+	{
+	Stmt::Describe(d);
+
+	if ( d->IsReadable() )
+		d->Add("(");
+
+	loop_condition->Describe(d);
+
+	if ( d->IsReadable() )
+		d->Add(")");
+
+	d->SP();
+	d->PushIndent();
+	body->AccessStats(d);
+	body->Describe(d);
+	d->PopIndent();
+	}
+
+TraversalCode WhileStmt::Traverse(TraversalCallback* cb) const
+	{
+	TraversalCode tc = cb->PreStmt(this);
+	HANDLE_TC_STMT_PRE(tc);
+
+	tc = loop_condition->Traverse(cb);
+	HANDLE_TC_STMT_PRE(tc);
+
+	tc = body->Traverse(cb);
+	HANDLE_TC_STMT_PRE(tc);
+
+	tc = cb->PostStmt(this);
+	HANDLE_TC_STMT_POST(tc);
+	}
+
+Val* WhileStmt::Exec(Frame* f, stmt_flow_type& flow) const
+	{
+	RegisterAccess();
+	flow = FLOW_NEXT;
+	Val* rval = 0;
+
+	for ( ; ; )
+		{
+		Val* cond = loop_condition->Eval(f);
+
+		if ( ! cond )
+			break;
+
+		bool cont = cond->AsBool();
+		Unref(cond);
+
+		if ( ! cont )
+			break;
+
+		flow = FLOW_NEXT;
+		rval = body->Exec(f, flow);
+
+		if ( flow == FLOW_BREAK || flow == FLOW_RETURN )
+			break;
+		}
+
+	if ( flow == FLOW_LOOP || flow == FLOW_BREAK )
+		flow = FLOW_NEXT;
+
+	return rval;
+	}
+
+Stmt* WhileStmt::Simplify()
+	{
+	loop_condition = simplify_expr(loop_condition, SIMPLIFY_GENERAL);
+
+	if ( loop_condition->IsConst() && loop_condition->IsZero() )
+		return new NullStmt();
+
+	body = simplify_stmt(body);
+	return this;
+	}
+
+IMPLEMENT_SERIAL(WhileStmt, SER_WHILE_STMT);
+
+bool WhileStmt::DoSerialize(SerialInfo* info) const
+	{
+	DO_SERIALIZE(SER_WHILE_STMT, Stmt);
+
+	if ( ! loop_condition->Serialize(info) )
+		return false;
+
+	return body->Serialize(info);
+	}
+
+bool WhileStmt::DoUnserialize(UnserialInfo* info)
+	{
+	DO_UNSERIALIZE(Stmt);
+	loop_condition = Expr::Unserialize(info);
+
+	if ( ! loop_condition )
+		return false;
+
+	body = Stmt::Unserialize(info);
+	return body != 0;
+	}
+
 ForStmt::ForStmt(id_list* arg_loop_vars, Expr* loop_expr)
 : ExprStmt(STMT_FOR, loop_expr)
 	{

src/Stmt.h

@@ -310,6 +310,33 @@ protected:
 	EventExpr* event_expr;
 };
 
+class WhileStmt : public Stmt {
+public:
+
+	WhileStmt(Expr* loop_condition, Stmt* body);
+	~WhileStmt();
+
+	int IsPure() const;
+
+	void Describe(ODesc* d) const;
+
+	TraversalCode Traverse(TraversalCallback* cb) const;
+
+protected:
+	friend class Stmt;
+
+	WhileStmt()
+		{ loop_condition = 0; body = 0; }
+
+	Val* Exec(Frame* f, stmt_flow_type& flow) const;
+	Stmt* Simplify();
+
+	DECLARE_SERIAL(WhileStmt);
+
+	Expr* loop_condition;
+	Stmt* body;
+};
+
 class ForStmt : public ExprStmt {
 public:
 	ForStmt(id_list* loop_vars, Expr* loop_expr);

src/StmtEnums.h

@@ -17,6 +17,7 @@ typedef enum {
 	STMT_LIST, STMT_EVENT_BODY_LIST,
 	STMT_INIT,
 	STMT_FALLTHROUGH,
+	STMT_WHILE,
 	STMT_NULL
 #define NUM_STMTS (int(STMT_NULL) + 1)
 } BroStmtTag;

src/Type.cc

@@ -1434,7 +1434,7 @@ EnumType::~EnumType()
 // Note, we use reporter->Error() here (not Error()) to include the current script
 // location in the error message, rather than the one where the type was
 // originally defined.
-void EnumType::AddName(const string& module_name, const char* name, bool is_export)
+void EnumType::AddName(const string& module_name, const char* name, bool is_export, bool deprecated)
 	{
 	/* implicit, auto-increment */
 	if ( counter < 0)
@@ -1443,11 +1443,11 @@ void EnumType::AddName(const string& module_name, const char* name, bool is_expo
 		SetError();
 		return;
 		}
-	CheckAndAddName(module_name, name, counter, is_export);
+	CheckAndAddName(module_name, name, counter, is_export, deprecated);
 	counter++;
 	}
 
-void EnumType::AddName(const string& module_name, const char* name, bro_int_t val, bool is_export)
+void EnumType::AddName(const string& module_name, const char* name, bro_int_t val, bool is_export, bool deprecated)
 	{
 	/* explicit value specified */
 	if ( counter > 0 )
@@ -1457,11 +1457,11 @@ void EnumType::AddName(const string& module_name, const char* name, bro_int_t va
 		return;
 		}
 	counter = -1;
-	CheckAndAddName(module_name, name, val, is_export);
+	CheckAndAddName(module_name, name, val, is_export, deprecated);
 	}
 
 void EnumType::CheckAndAddName(const string& module_name, const char* name,
-                               bro_int_t val, bool is_export)
+                               bro_int_t val, bool is_export, bool deprecated)
 	{
 	if ( Lookup(val) )
 		{
@@ -1477,6 +1477,10 @@ void EnumType::CheckAndAddName(const string& module_name, const char* name,
 		id = install_ID(name, module_name.c_str(), true, is_export);
 		id->SetType(this->Ref());
 		id->SetEnumConst();
+
+		if ( deprecated )
+			id->MakeDeprecated();
+
 		broxygen_mgr->Identifier(id);
 		}
 	else

src/Type.h

@@ -554,12 +554,12 @@ public:
 
 	// The value of this name is next internal counter value, starting
 	// with zero. The internal counter is incremented.
-	void AddName(const string& module_name, const char* name, bool is_export);
+	void AddName(const string& module_name, const char* name, bool is_export, bool deprecated);
 
 	// The value of this name is set to val. Once a value has been
 	// explicitly assigned using this method, no further names can be
 	// added that aren't likewise explicitly initalized.
-	void AddName(const string& module_name, const char* name, bro_int_t val, bool is_export);
+	void AddName(const string& module_name, const char* name, bro_int_t val, bool is_export, bool deprecated);
 
 	// -1 indicates not found.
 	bro_int_t Lookup(const string& module_name, const char* name) const;
@@ -580,7 +580,8 @@ protected:
 			const char* name, bro_int_t val, bool is_export);
 
 	void CheckAndAddName(const string& module_name,
-	                     const char* name, bro_int_t val, bool is_export);
+	                     const char* name, bro_int_t val, bool is_export,
+	                     bool deprecated);
 
 	typedef std::map< const char*, bro_int_t, ltstr > NameMap;
 	NameMap names;

src/Var.cc

@@ -435,6 +435,10 @@ void end_func(Stmt* body, attr_list* attrs)
 		loop_over_list(*attrs, i)
 			{
 			Attr* a = (*attrs)[i];
+
+			if ( a->Tag() == ATTR_DEPRECATED )
+				continue;
+
 			if ( a->Tag() != ATTR_PRIORITY )
 				{
 				a->Error("illegal attribute for function body");

src/analyzer/protocol/dnp3/DNP3.cc

@@ -97,7 +97,6 @@
 //                                            Binpac DNP3 Analyzer
 
 #include "DNP3.h"
-#include "analyzer/protocol/tcp/TCP_Reassembler.h"
 #include "events.bif.h"
 
 using namespace analyzer::dnp3;
@@ -109,12 +108,14 @@ const unsigned int PSEUDO_APP_LAYER_INDEX = 11;		// index of first DNP3 app-laye
 const unsigned int PSEUDO_TRANSPORT_LEN = 1;		// length of DNP3 Transport Layer
 const unsigned int PSEUDO_LINK_LAYER_LEN = 8;		// length of DNP3 Pseudo Link Layer
 
-bool DNP3_Analyzer::crc_table_initialized = false;
+bool DNP3_Base::crc_table_initialized = false;
-unsigned int DNP3_Analyzer::crc_table[256];
+unsigned int DNP3_Base::crc_table[256];
 
-DNP3_Analyzer::DNP3_Analyzer(Connection* c) : TCP_ApplicationAnalyzer("DNP3", c)
+
+DNP3_Base::DNP3_Base(analyzer::Analyzer* arg_analyzer)
 	{
-	interp = new binpac::DNP3::DNP3_Conn(this);
+	analyzer = arg_analyzer;
+	interp = new binpac::DNP3::DNP3_Conn(analyzer);
 
 	ClearEndpointState(true);
 	ClearEndpointState(false);
@@ -123,49 +124,12 @@ DNP3_Analyzer::DNP3_Analyzer(Connection* c) : TCP_ApplicationAnalyzer("DNP3", c)
 		PrecomputeCRCTable();
 	}
 
-DNP3_Analyzer::~DNP3_Analyzer()
+DNP3_Base::~DNP3_Base()
 	{
 	delete interp;
 	}
 
-void DNP3_Analyzer::Done()
+bool DNP3_Base::ProcessData(int len, const u_char* data, bool orig)
-	{
-	TCP_ApplicationAnalyzer::Done();
-
-	interp->FlowEOF(true);
-	interp->FlowEOF(false);
-	}
-
-void DNP3_Analyzer::DeliverStream(int len, const u_char* data, bool orig)
-	{
-	TCP_ApplicationAnalyzer::DeliverStream(len, data, orig);
-
-	try
-		{
-		if ( ! ProcessData(len, data, orig) )
-			SetSkip(1);
-		}
-
-	catch ( const binpac::Exception& e )
-		{
-		SetSkip(1);
-		throw;
-		}
-	}
-
-void DNP3_Analyzer::Undelivered(uint64 seq, int len, bool orig)
-	{
-	TCP_ApplicationAnalyzer::Undelivered(seq, len, orig);
-	interp->NewGap(orig, len);
-	}
-
-void DNP3_Analyzer::EndpointEOF(bool is_orig)
-	{
-	TCP_ApplicationAnalyzer::EndpointEOF(is_orig);
-	interp->FlowEOF(is_orig);
-	}
-
-bool DNP3_Analyzer::ProcessData(int len, const u_char* data, bool orig)
 	{
 	Endpoint* endp = orig ? &orig_state : &resp_state;
 
@@ -174,25 +138,30 @@ bool DNP3_Analyzer::ProcessData(int len, const u_char* data, bool orig)
 		if ( endp->in_hdr )
 			{
 			// We're parsing the DNP3 header and link layer, get that in full.
-			if ( ! AddToBuffer(endp, PSEUDO_APP_LAYER_INDEX, &data, &len) )
+			int res = AddToBuffer(endp, PSEUDO_APP_LAYER_INDEX, &data, &len);
+
+			if ( res == 0 )
 				return true;
 
+			if ( res < 0 )
+				return false;
+
 			// The first two bytes must always be 0x0564.
 			if( endp->buffer[0] != 0x05 || endp->buffer[1] != 0x64 )
 				{
-				Weird("dnp3_header_lacks_magic");
+				analyzer->Weird("dnp3_header_lacks_magic");
 				return false;
 				}
 
 			// Make sure header checksum is correct.
 			if ( ! CheckCRC(PSEUDO_LINK_LAYER_LEN, endp->buffer, endp->buffer + PSEUDO_LINK_LAYER_LEN, "header") )
 				{
-				ProtocolViolation("broken_checksum");
+				analyzer->ProtocolViolation("broken_checksum");
 				return false;
 				}
 
 			// If the checksum works out, we're pretty certainly DNP3.
-			ProtocolConfirmation();
+			analyzer->ProtocolConfirmation();
 
 			// DNP3 packets without transport and application
 			// layers can happen, we ignore them.
@@ -207,7 +176,7 @@ bool DNP3_Analyzer::ProcessData(int len, const u_char* data, bool orig)
 			u_char ctrl = endp->buffer[PSEUDO_CONTROL_FIELD_INDEX];
 
 			if ( orig != (bool)(ctrl & 0x80) )
-				Weird("dnp3_unexpected_flow_direction");
+				analyzer->Weird("dnp3_unexpected_flow_direction");
 
 			// Update state.
 			endp->pkt_length = endp->buffer[PSEUDO_LENGTH_INDEX];
@@ -222,7 +191,11 @@ bool DNP3_Analyzer::ProcessData(int len, const u_char* data, bool orig)
 
 		if ( ! endp->in_hdr )
 			{
-			assert(endp->pkt_length);
+			if ( endp->pkt_length <= 0 )
+				{
+				analyzer->Weird("dnp3_negative_or_zero_length_link_layer");
+				return false;
+				}
 
 			// We're parsing the DNP3 application layer, get that
 			// in full now as well. We calculate the number of
@@ -230,11 +203,17 @@ bool DNP3_Analyzer::ProcessData(int len, const u_char* data, bool orig)
 			// the packet length by determining how much 16-byte
 			// chunks fit in there, and then add 2 bytes CRC for
 			// each.
-			int n = PSEUDO_APP_LAYER_INDEX + (endp->pkt_length - 5) + ((endp->pkt_length - 5) / 16) * 2 + 2 - 1;
+			int n = PSEUDO_APP_LAYER_INDEX + (endp->pkt_length - 5) + ((endp->pkt_length - 5) / 16) * 2
+					+ 2 * ( ((endp->pkt_length - 5) % 16 == 0) ? 0 : 1) - 1 ;
 
-			if ( ! AddToBuffer(endp, n, &data, &len) )
+			int res = AddToBuffer(endp, n, &data, &len);
+
+			if ( res == 0 )
 				return true;
 
+			if ( res < 0 )
+				return false;
+
 			// Parse the the application layer data.
 			if ( ! ParseAppLayer(endp) )
 				return false;
@@ -248,22 +227,45 @@ bool DNP3_Analyzer::ProcessData(int len, const u_char* data, bool orig)
 	return true;
 	}
 
-bool DNP3_Analyzer::AddToBuffer(Endpoint* endp, int target_len, const u_char** data, int* len)
+int DNP3_Base::AddToBuffer(Endpoint* endp, int target_len, const u_char** data, int* len)
 	{
 	if ( ! target_len )
-		return true;
+		return 1;
+
+	if ( *len < 0 )
+		{
+		reporter->AnalyzerError(analyzer, "dnp3 negative input length: %d", *len);
+		return -1;
+		}
+
+	if ( target_len < endp->buffer_len )
+		{
+		reporter->AnalyzerError(analyzer, "dnp3 invalid target length: %d - %d",
+		                        target_len, endp->buffer_len);
+		return -1;
+		}
 
 	int to_copy = min(*len, target_len - endp->buffer_len);
 
+	if ( endp->buffer_len + to_copy > MAX_BUFFER_SIZE )
+		{
+		reporter->AnalyzerError(analyzer, "dnp3 buffer length exceeded: %d + %d",
+		                        endp->buffer_len, to_copy);
+		return -1;
+		}
+
 	memcpy(endp->buffer + endp->buffer_len, *data, to_copy);
 	*data += to_copy;
 	*len -= to_copy;
 	endp->buffer_len += to_copy;
 
-	return endp->buffer_len == target_len;
+	if ( endp->buffer_len == target_len )
+		return 1;
+
+	return 0;
 	}
 
-bool DNP3_Analyzer::ParseAppLayer(Endpoint* endp)
+bool DNP3_Base::ParseAppLayer(Endpoint* endp)
 	{
 	bool orig = (endp == &orig_state);
 	binpac::DNP3::DNP3_Flow* flow = orig ? interp->upflow() : interp->downflow();
@@ -291,8 +293,15 @@ bool DNP3_Analyzer::ParseAppLayer(Endpoint* endp)
 		if ( ! CheckCRC(n, data, data + n, "app_chunk") )
 			return false;
 
+		if ( data + n >= endp->buffer + endp->buffer_len )
+			{
+			reporter->AnalyzerError(analyzer,
+			                        "dnp3 app layer parsing overflow %d - %d",
+			                        endp->buffer_len, n);
+			return false;
+			}
+
 		// Pass on to BinPAC.
-		assert(data + n < endp->buffer + endp->buffer_len);
 		flow->flow_buffer()->BufferData(data + transport, data + n);
 		transport = 0;
 
@@ -306,7 +315,7 @@ bool DNP3_Analyzer::ParseAppLayer(Endpoint* endp)
 	if ( ! is_first && ! endp->encountered_first_chunk )
 		{
 		// We lost the first chunk.
-		Weird("dnp3_first_application_layer_chunk_missing");
+		analyzer->Weird("dnp3_first_application_layer_chunk_missing");
 		return false;
 		}
 
@@ -320,7 +329,7 @@ bool DNP3_Analyzer::ParseAppLayer(Endpoint* endp)
 	return true;
 	}
 
-void DNP3_Analyzer::ClearEndpointState(bool orig)
+void DNP3_Base::ClearEndpointState(bool orig)
 	{
 	Endpoint* endp = orig ? &orig_state : &resp_state;
 	binpac::DNP3::DNP3_Flow* flow = orig ? interp->upflow() : interp->downflow();
@@ -333,18 +342,18 @@ void DNP3_Analyzer::ClearEndpointState(bool orig)
 	endp->pkt_cnt = 0;
 	}
 
-bool DNP3_Analyzer::CheckCRC(int len, const u_char* data, const u_char* crc16, const char* where)
+bool DNP3_Base::CheckCRC(int len, const u_char* data, const u_char* crc16, const char* where)
 	{
 	unsigned int crc = CalcCRC(len, data);
 
 	if ( crc16[0] == (crc & 0xff) && crc16[1] == (crc & 0xff00) >> 8 )
 		return true;
 
-	Weird(fmt("dnp3_corrupt_%s_checksum", where));
+	analyzer->Weird(fmt("dnp3_corrupt_%s_checksum", where));
 	return false;
 	}
 
-void DNP3_Analyzer::PrecomputeCRCTable()
+void DNP3_Base::PrecomputeCRCTable()
 	{
 	for( unsigned int i = 0; i < 256; i++)
 		{
@@ -362,7 +371,7 @@ void DNP3_Analyzer::PrecomputeCRCTable()
 		}
 	}
 
-unsigned int DNP3_Analyzer::CalcCRC(int len, const u_char* data)
+unsigned int DNP3_Base::CalcCRC(int len, const u_char* data)
 	{
 	unsigned int crc = 0x0000;
 
@@ -374,3 +383,76 @@ unsigned int DNP3_Analyzer::CalcCRC(int len, const u_char* data)
 
 	return ~crc & 0xFFFF;
 	}
+
+DNP3_TCP_Analyzer::DNP3_TCP_Analyzer(Connection* c)
+	: DNP3_Base(this), TCP_ApplicationAnalyzer("DNP3_TCP", c)
+	{
+	}
+
+DNP3_TCP_Analyzer::~DNP3_TCP_Analyzer()
+	{
+	}
+
+void DNP3_TCP_Analyzer::Done()
+	{
+	TCP_ApplicationAnalyzer::Done();
+
+	Interpreter()->FlowEOF(true);
+	Interpreter()->FlowEOF(false);
+	}
+
+void DNP3_TCP_Analyzer::DeliverStream(int len, const u_char* data, bool orig)
+	{
+	TCP_ApplicationAnalyzer::DeliverStream(len, data, orig);
+
+	try
+		{
+		if ( ! ProcessData(len, data, orig) )
+			SetSkip(1);
+		}
+
+	catch ( const binpac::Exception& e )
+		{
+		SetSkip(1);
+		throw;
+		}
+	}
+
+void DNP3_TCP_Analyzer::Undelivered(uint64 seq, int len, bool orig)
+	{
+	TCP_ApplicationAnalyzer::Undelivered(seq, len, orig);
+	Interpreter()->NewGap(orig, len);
+	}
+
+void DNP3_TCP_Analyzer::EndpointEOF(bool is_orig)
+	{
+	TCP_ApplicationAnalyzer::EndpointEOF(is_orig);
+	Interpreter()->FlowEOF(is_orig);
+	}
+
+DNP3_UDP_Analyzer::DNP3_UDP_Analyzer(Connection* c)
+	: DNP3_Base(this), Analyzer("DNP3_UDP", c)
+	{
+	}
+
+DNP3_UDP_Analyzer::~DNP3_UDP_Analyzer()
+	{
+	}
+
+void DNP3_UDP_Analyzer::DeliverPacket(int len, const u_char* data, bool orig, uint64 seq, const IP_Hdr* ip, int caplen)
+	{
+	Analyzer::DeliverPacket(len, data, orig, seq, ip, caplen);
+
+	try
+		{
+		if ( ! ProcessData(len, data, orig) )
+			SetSkip(1);
+		}
+
+	catch ( const binpac::Exception& e )
+		{
+		SetSkip(1);
+		throw;
+		}
+	}
+

src/analyzer/protocol/dnp3/DNP3.h

@@ -3,24 +3,20 @@
 #define ANALYZER_PROTOCOL_DNP3_DNP3_H
 
 #include "analyzer/protocol/tcp/TCP.h"
+#include "analyzer/protocol/udp/UDP.h"
+
 #include "dnp3_pac.h"
 
 namespace analyzer { namespace dnp3 {
 
-class DNP3_Analyzer : public tcp::TCP_ApplicationAnalyzer {
+class DNP3_Base {
 public:
-	DNP3_Analyzer(Connection* conn);
+	DNP3_Base(analyzer::Analyzer* analyzer);
-	virtual ~DNP3_Analyzer();
+	virtual ~DNP3_Base();
 
-	virtual void Done();
+	binpac::DNP3::DNP3_Conn* Interpreter()	{ return interp; }
-	virtual void DeliverStream(int len, const u_char* data, bool orig);
-	virtual void Undelivered(uint64 seq, int len, bool orig);
-	virtual void EndpointEOF(bool is_orig);
 
-	static Analyzer* Instantiate(Connection* conn)
+protected:
-		{ return new DNP3_Analyzer(conn); }
-
-private:
 	static const int MAX_BUFFER_SIZE = 300;
 
 	struct Endpoint	{
@@ -35,22 +31,64 @@ private:
 
 	bool ProcessData(int len, const u_char* data, bool orig);
 	void ClearEndpointState(bool orig);
-	bool AddToBuffer(Endpoint* endp, int target_len, const u_char** data, int* len);
+
+	/**
+	 * Buffers packet data until it reaches a specified length.
+	 * @param endp an endpoint speaking DNP3 to which data will be buffered.
+	 * @param target_len the required length of the buffer
+	 * @param data source buffer to copy bytes from.  Will be incremented
+	 * by the number of bytes copied by this function.
+	 * @param len the number of bytes available in \a data.  Will be decremented
+	 * by the number of bytes copied by this function.
+	 * @return -1 if invalid input parameters were supplied, 0 if the endpoint's
+	 * buffer is not yet \a target_len bytes in size, or 1 the buffer is the
+	 * required size.
+	 */
+	int AddToBuffer(Endpoint* endp, int target_len, const u_char** data, int* len);
+
 	bool ParseAppLayer(Endpoint* endp);
 	bool CheckCRC(int len, const u_char* data, const u_char* crc16, const char* where);
 	unsigned int CalcCRC(int len, const u_char* data);
 
-	binpac::DNP3::DNP3_Conn* interp;
-
-	Endpoint orig_state;
-	Endpoint resp_state;
-
 	static void PrecomputeCRCTable();
 
 	static bool crc_table_initialized;
 	static unsigned int crc_table[256];
+
+	analyzer::Analyzer* analyzer;
+	binpac::DNP3::DNP3_Conn* interp;
+
+	Endpoint orig_state;
+	Endpoint resp_state;
 };
 
-} } // namespace analyzer::* 
+class DNP3_TCP_Analyzer : public DNP3_Base, public tcp::TCP_ApplicationAnalyzer {
+public:
+	DNP3_TCP_Analyzer(Connection* conn);
+	virtual ~DNP3_TCP_Analyzer();
+
+	virtual void Done();
+	virtual void DeliverStream(int len, const u_char* data, bool orig);
+	virtual void Undelivered(uint64 seq, int len, bool orig);
+	virtual void EndpointEOF(bool is_orig);
+
+	static Analyzer* Instantiate(Connection* conn)
+		{ return new DNP3_TCP_Analyzer(conn); }
+};
+
+class DNP3_UDP_Analyzer : public DNP3_Base, public analyzer::Analyzer {
+public:
+	DNP3_UDP_Analyzer(Connection* conn);
+	virtual ~DNP3_UDP_Analyzer();
+
+	virtual void DeliverPacket(int len, const u_char* data, bool orig,
+                    uint64 seq, const IP_Hdr* ip, int caplen);
+
+	static analyzer::Analyzer* Instantiate(Connection* conn)
+		{ return new DNP3_UDP_Analyzer(conn); }
+};
+
+
+} } // namespace analyzer::*
 
 #endif

src/analyzer/protocol/dnp3/Plugin.cc

@@ -12,11 +12,12 @@ class Plugin : public plugin::Plugin {
 public:
 	plugin::Configuration Configure()
 		{
-		AddComponent(new ::analyzer::Component("DNP3", ::analyzer::dnp3::DNP3_Analyzer::Instantiate));
+		AddComponent(new ::analyzer::Component("DNP3_TCP", ::analyzer::dnp3::DNP3_TCP_Analyzer::Instantiate));
+		AddComponent(new ::analyzer::Component("DNP3_UDP", ::analyzer::dnp3::DNP3_UDP_Analyzer::Instantiate));
 
 		plugin::Configuration config;
 		config.name = "Bro::DNP3";
-		config.description = "DNP3 analyzer";
+		config.description = "DNP3 UDP/TCP analyzers";
 		return config;
 		}
 } plugin;

src/analyzer/protocol/dnp3/dnp3-analyzer.pac

@@ -38,7 +38,7 @@ flow DNP3_Flow(is_orig: bool) {
 		return true;
 		%}
 
-	function get_dnp3_application_request_header(fc: uint8): bool
+	function get_dnp3_application_request_header(application_control: uint8, fc: uint8): bool
 		%{
 		if ( ::dnp3_application_request_header )
 			{
@@ -46,13 +46,14 @@ flow DNP3_Flow(is_orig: bool) {
 				connection()->bro_analyzer(),
 				connection()->bro_analyzer()->Conn(),
 				is_orig(),
+				application_control, 
 				fc
 				);
 			}
 		return true;
 		%}
 
-	function get_dnp3_application_response_header(fc: uint8, iin: uint16): bool
+	function get_dnp3_application_response_header(application_control: uint8, fc: uint8, iin: uint16): bool
 		%{
 		if ( ::dnp3_application_response_header )
 			{
@@ -60,6 +61,7 @@ flow DNP3_Flow(is_orig: bool) {
 				connection()->bro_analyzer(),
 				connection()->bro_analyzer()->Conn(),
 				is_orig(),
+				application_control,
 				fc,
 				iin
 				);
@@ -743,11 +745,11 @@ refine typeattr Header_Block += &let {
 };
 
 refine typeattr DNP3_Application_Request_Header += &let {
-	process_request: bool =  $context.flow.get_dnp3_application_request_header(function_code);
+	process_request: bool =  $context.flow.get_dnp3_application_request_header(application_control, function_code);
 };
 
 refine typeattr DNP3_Application_Response_Header += &let {
-	process_request: bool =  $context.flow.get_dnp3_application_response_header(function_code, internal_indications);
+	process_request: bool =  $context.flow.get_dnp3_application_response_header(application_control, function_code, internal_indications);
 };
 
 refine typeattr Object_Header += &let {

src/analyzer/protocol/dnp3/dnp3-protocol.pac

@@ -90,7 +90,7 @@ type DNP3_Application_Response_Header = record {
 type Request_Objects(function_code: uint8) = record {
 	object_header: Object_Header(function_code);
 	data: case (object_header.object_type_field) of {
-		0x0c03 -> bocmd_PM: Request_Data_Object(function_code, object_header.qualifier_field, object_header.object_type_field )[ ( object_header.number_of_item / 8 ) + 1 ];
+		0x0c03 -> bocmd_PM: Request_Data_Object(function_code, object_header.qualifier_field, object_header.object_type_field )[ ( object_header.number_of_item / 8 ) + 1*( object_header.number_of_item > ( (object_header.number_of_item / 8)*8 ) ) ];
 		0x3202 -> time_interval_ojbects: Request_Data_Object(function_code, object_header.qualifier_field, object_header.object_type_field )[ object_header.number_of_item]
 							&check( object_header.qualifer_field == 0x0f && object_header.number_of_item == 0x01);
 		default -> ojbects: Request_Data_Object(function_code, object_header.qualifier_field, object_header.object_type_field )[ object_header.number_of_item];
@@ -112,10 +112,10 @@ type Request_Objects(function_code: uint8) = record {
 type Response_Objects(function_code: uint8) = record {
 	object_header: Object_Header(function_code);
 	data: case (object_header.object_type_field) of {
-		0x0101 -> biwoflag: Response_Data_Object(function_code, object_header.qualifier_field, object_header.object_type_field )[ ( object_header.number_of_item / 8 ) + 1 ];
+		0x0101 -> biwoflag: Response_Data_Object(function_code, object_header.qualifier_field, object_header.object_type_field )[ ( object_header.number_of_item / 8 ) + 1*( object_header.number_of_item > ( (object_header.number_of_item / 8)*8 ) ) ];
-		0x0301 -> diwoflag: Response_Data_Object(function_code, object_header.qualifier_field, object_header.object_type_field )[ ( object_header.number_of_item / 8 ) + 1 ];
+		0x0301 -> diwoflag: Response_Data_Object(function_code, object_header.qualifier_field, object_header.object_type_field )[ ( object_header.number_of_item / 8 ) + 1*( object_header.number_of_item > ( (object_header.number_of_item / 8)*8 ) ) ];
-		0x0a01 -> bowoflag: Response_Data_Object(function_code, object_header.qualifier_field, object_header.object_type_field )[ ( object_header.number_of_item / 8 ) + 1 ];
+		0x0a01 -> bowoflag: Response_Data_Object(function_code, object_header.qualifier_field, object_header.object_type_field )[ ( object_header.number_of_item / 8 ) + 1*( object_header.number_of_item > ( (object_header.number_of_item / 8)*8 ) )];
-		0x0c03 -> bocmd_PM: Response_Data_Object(function_code, object_header.qualifier_field, object_header.object_type_field )[ ( object_header.number_of_item / 8 ) + 1 ];
+		0x0c03 -> bocmd_PM: Response_Data_Object(function_code, object_header.qualifier_field, object_header.object_type_field )[ ( object_header.number_of_item / 8 ) + 1*( object_header.number_of_item > ( (object_header.number_of_item / 8)*8 ) )];
 		default -> ojbects: Response_Data_Object(function_code, object_header.qualifier_field, object_header.object_type_field )[ object_header.number_of_item];
 	};
 };

src/analyzer/protocol/dnp3/events.bif

@@ -7,7 +7,7 @@
 ##
 ## fc: function code.
 ##
-event dnp3_application_request_header%(c: connection, is_orig: bool, fc: count%);
+event dnp3_application_request_header%(c: connection, is_orig: bool, application: count, fc: count%);
 
 ## Generated for a DNP3 response header.
 ##
@@ -19,7 +19,7 @@ event dnp3_application_request_header%(c: connection, is_orig: bool, fc: count%)
 ##
 ## iin: internal indication number.
 ##
-event dnp3_application_response_header%(c: connection, is_orig: bool, fc: count, iin: count%);
+event dnp3_application_response_header%(c: connection, is_orig: bool, application: count, fc: count, iin: count%);
 
 ## Generated for the object header found in both DNP3 requests and responses.
 ##

src/analyzer/protocol/mysql/events.bif

@@ -9,7 +9,7 @@
 ##
 ## arg: The argument for the command (empty string if not provided).
 ##
-## .. bro:see:: mysql_error mysql_ok mysql_server_version mysql_handshake_response
+## .. bro:see:: mysql_error mysql_ok mysql_server_version mysql_handshake
 event mysql_command_request%(c: connection, command: count, arg: string%);
 
 ## Generated for an unsuccessful MySQL response.
@@ -23,7 +23,7 @@ event mysql_command_request%(c: connection, command: count, arg: string%);
 ##
 ## msg: Any extra details about the error (empty string if not provided).
 ##
-## .. bro:see:: mysql_command_request mysql_ok mysql_server_version mysql_handshake_response
+## .. bro:see:: mysql_command_request mysql_ok mysql_server_version mysql_handshake
 event mysql_error%(c: connection, code: count, msg: string%);
 
 ## Generated for a successful MySQL response.
@@ -35,7 +35,7 @@ event mysql_error%(c: connection, code: count, msg: string%);
 ##
 ## affected_rows: The number of rows that were affected.
 ##
-## .. bro:see:: mysql_command_request mysql_error mysql_server_version mysql_handshake_response
+## .. bro:see:: mysql_command_request mysql_error mysql_server_version mysql_handshake
 event mysql_ok%(c: connection, affected_rows: count%);
 
 ## Generated for the initial server handshake packet, which includes the MySQL server version.
@@ -47,7 +47,7 @@ event mysql_ok%(c: connection, affected_rows: count%);
 ##
 ## ver: The server version string.
 ##
-## .. bro:see:: mysql_command_request mysql_error mysql_ok mysql_handshake_response
+## .. bro:see:: mysql_command_request mysql_error mysql_ok mysql_handshake
 event mysql_server_version%(c: connection, ver: string%);
 
 ## Generated for a client handshake response packet, which includes the username the client is attempting

src/analyzer/protocol/pia/PIA.cc

@@ -22,6 +22,7 @@ void PIA::ClearBuffer(Buffer* buffer)
 	for ( DataBlock* b = buffer->head; b; b = next )
 		{
 		next = b->next;
+		delete b->ip;
 		delete [] b->data;
 		delete b;
 		}
@@ -31,7 +32,7 @@ void PIA::ClearBuffer(Buffer* buffer)
 	}
 
 void PIA::AddToBuffer(Buffer* buffer, uint64 seq, int len, const u_char* data,
-			bool is_orig)
+			bool is_orig, const IP_Hdr* ip)
 	{
 	u_char* tmp = 0;
 
@@ -42,6 +43,7 @@ void PIA::AddToBuffer(Buffer* buffer, uint64 seq, int len, const u_char* data,
 		}
 
 	DataBlock* b = new DataBlock;
+	b->ip = ip ? ip->Copy() : 0;
 	b->data = tmp;
 	b->is_orig = is_orig;
 	b->len = len;
@@ -59,9 +61,10 @@ void PIA::AddToBuffer(Buffer* buffer, uint64 seq, int len, const u_char* data,
 	buffer->size += len;
 	}
 
-void PIA::AddToBuffer(Buffer* buffer, int len, const u_char* data, bool is_orig)
+void PIA::AddToBuffer(Buffer* buffer, int len, const u_char* data, bool is_orig,
+                      const IP_Hdr* ip)
 	{
-	AddToBuffer(buffer, -1, len, data, is_orig);
+	AddToBuffer(buffer, -1, len, data, is_orig, ip);
 	}
 
 void PIA::ReplayPacketBuffer(analyzer::Analyzer* analyzer)
@@ -69,7 +72,7 @@ void PIA::ReplayPacketBuffer(analyzer::Analyzer* analyzer)
 	DBG_LOG(DBG_ANALYZER, "PIA replaying %d total packet bytes", pkt_buffer.size);
 
 	for ( DataBlock* b = pkt_buffer.head; b; b = b->next )
-		analyzer->DeliverPacket(b->len, b->data, b->is_orig, -1, 0, 0);
+		analyzer->DeliverPacket(b->len, b->data, b->is_orig, -1, b->ip, 0);
 	}
 
 void PIA::PIA_Done()
@@ -96,7 +99,7 @@ void PIA::PIA_DeliverPacket(int len, const u_char* data, bool is_orig, uint64 se
 	if ( (pkt_buffer.state == BUFFERING || new_state == BUFFERING) &&
 	     len > 0 )
 		{
-		AddToBuffer(&pkt_buffer, seq, len, data, is_orig);
+		AddToBuffer(&pkt_buffer, seq, len, data, is_orig, ip);
 		if ( pkt_buffer.size > dpd_buffer_size )
 			new_state = dpd_match_only_beginning ?
 						SKIPPING : MATCHING_ONLY;

src/analyzer/protocol/pia/PIA.h

@@ -49,6 +49,7 @@ protected:
 	// Buffers one chunk of data.  Used both for packet payload (incl.
 	// sequence numbers for TCP) and chunks of a reassembled stream.
 	struct DataBlock {
+		IP_Hdr* ip;
 		const u_char* data;
 		bool is_orig;
 		int len;
@@ -66,9 +67,9 @@ protected:
 	};
 
 	void AddToBuffer(Buffer* buffer, uint64 seq, int len,
-				const u_char* data, bool is_orig);
+				const u_char* data, bool is_orig, const IP_Hdr* ip = 0);
 	void AddToBuffer(Buffer* buffer, int len,
-				const u_char* data, bool is_orig);
+				const u_char* data, bool is_orig, const IP_Hdr* ip = 0);
 	void ClearBuffer(Buffer* buffer);
 
 	DataBlock* CurrentPacket()	{ return &current_packet; }

src/analyzer/protocol/socks/SOCKS.cc

@@ -57,8 +57,7 @@ void SOCKS_Analyzer::DeliverStream(int len, const u_char* data, bool orig)
 		// with the rest of the conneciton.
 		//
 		// Note that we assume that no payload data arrives before both endpoints
-		// are done with there part of the SOCKS protocol.
+		// are done with their part of the SOCKS protocol.
-
 		if ( ! pia )
 			{
 			pia = new pia::PIA_TCP(Conn());

src/analyzer/protocol/socks/events.bif

@@ -27,3 +27,19 @@ event socks_request%(c: connection, version: count, request_type: count, sa: SOC
 ## p: The destination port for the proxied traffic.
 event socks_reply%(c: connection, version: count, reply: count, sa: SOCKS::Address, p: port%);
 
+## Generated when a SOCKS client performs username and password based login.
+##
+## c: The parent connection of the proxy.
+##
+## user: The given username.
+##
+## password: The given password.
+event socks_login_userpass_request%(c: connection, user: string, password: string%);
+
+## Generated when a SOCKS server replies to a username/password login attempt.
+##
+## c: The parent connection of the proxy.
+## 
+## code: The response code for the attempted login.
+event socks_login_userpass_reply%(c: connection, code: count%);
+

src/analyzer/protocol/socks/socks-analyzer.pac

@@ -148,6 +148,37 @@ refine connection SOCKS_Conn += {
 		return true;
 		%}
 
+	function socks5_auth_request_userpass(request: SOCKS5_Auth_Request_UserPass_v1): bool
+		%{
+		StringVal* user = new StringVal(${request.username}.length(), (const char*) ${request.username}.begin());
+		StringVal* pass = new StringVal(${request.password}.length(), (const char*) ${request.password}.begin());
+		
+		BifEvent::generate_socks_login_userpass_request(bro_analyzer(),
+		                                                bro_analyzer()->Conn(),
+		                                                user, pass);
+		return true;
+		%}
+
+	function socks5_unsupported_authentication_method(auth_method: uint8): bool
+		%{
+		reporter->Weird(bro_analyzer()->Conn(), fmt("socks5_unsupported_authentication_method_%d", auth_method));
+		return true;
+		%}
+
+	function socks5_unsupported_authentication_version(auth_method: uint8, version: uint8): bool
+		%{
+		reporter->Weird(bro_analyzer()->Conn(), fmt("socks5_unsupported_authentication_%d_%d", auth_method, version));
+		return true;
+		%}
+	
+	function socks5_auth_reply_userpass(reply: SOCKS5_Auth_Reply_UserPass_v1): bool
+		%{
+		BifEvent::generate_socks_login_userpass_reply(bro_analyzer(),
+		                                              bro_analyzer()->Conn(),
+		                                              ${reply.code});
+		return true;
+		%}
+
 	function version_error(version: uint8): bool
 		%{
 		bro_analyzer()->ProtocolViolation(fmt("unsupported/unknown SOCKS version %d", version));
@@ -176,3 +207,22 @@ refine typeattr SOCKS5_Request += &let {
 refine typeattr SOCKS5_Reply += &let {
 	proc: bool = $context.connection.socks5_reply(this);
 };
+
+refine typeattr SOCKS5_Auth_Negotiation_Reply += &let {
+};
+
+refine typeattr SOCKS5_Auth_Request_UserPass_v1 += &let {
+	proc: bool = $context.connection.socks5_auth_request_userpass(this);
+};
+
+refine typeattr SOCKS5_Auth_Reply_UserPass_v1 += &let {
+	proc: bool = $context.connection.socks5_auth_reply_userpass(this);
+};
+
+refine typeattr SOCKS5_Unsupported_Authentication_Method += &let {
+	proc: bool = $context.connection.socks5_unsupported_authentication_method($context.connection.v5_auth_method());
+};
+
+refine typeattr SOCKS5_Unsupported_Authentication_Version += &let {
+	proc: bool = $context.connection.socks5_unsupported_authentication_version($context.connection.v5_auth_method(), version);
+};

src/analyzer/protocol/socks/socks-protocol.pac

@@ -1,10 +1,15 @@
 
+type SOCKS_Message(is_orig: bool) = case $context.connection.v5_in_auth_sub_negotiation() of {
+	true  -> auth: SOCKS5_Auth_Message(is_orig);
+	false -> msg:  SOCKS_Version(is_orig);
+};
+
 type SOCKS_Version(is_orig: bool) = record {
 	version: uint8;
 	msg:     case version of {
-		4       -> socks4_msg:     SOCKS4_Message(is_orig);
+		4       -> socks4_msg:      SOCKS4_Message(is_orig);
-		5       -> socks5_msg:     SOCKS5_Message(is_orig);
+		5       -> socks5_msg:      SOCKS5_Message(is_orig);
-		default -> socks_msg_fail: SOCKS_Version_Error(version);
+		default -> socks_msg_fail:  SOCKS_Version_Error(version);
 	};
 };
 
@@ -14,10 +19,11 @@ type SOCKS_Version_Error(version: uint8) = record {
 
 # SOCKS5 Implementation
 type SOCKS5_Message(is_orig: bool) = case $context.connection.v5_past_authentication() of {
-	true  -> msg:  SOCKS5_Real_Message(is_orig);
 	false -> auth: SOCKS5_Auth_Negotiation(is_orig);
+	true  -> msg:  SOCKS5_Real_Message(is_orig);
 };
 
+
 type SOCKS5_Auth_Negotiation(is_orig: bool) = case is_orig of {
 	true  -> req:  SOCKS5_Auth_Negotiation_Request;
 	false -> rep:  SOCKS5_Auth_Negotiation_Reply;
@@ -31,7 +37,61 @@ type SOCKS5_Auth_Negotiation_Request = record {
 type SOCKS5_Auth_Negotiation_Reply = record {
 	selected_auth_method: uint8;
 } &let {
+	in_auth_sub_neg = $context.connection.set_v5_in_auth_sub_negotiation(selected_auth_method == 0 || selected_auth_method == 0xff ? false : true);
 	past_auth = $context.connection.set_v5_past_authentication();
+	set_auth = $context.connection.set_v5_auth_method(selected_auth_method);
+};
+
+type SOCKS5_Auth_Message(is_orig: bool) = case is_orig of {
+	true  -> req: SOCKS5_Auth_Request;
+	false -> rep: SOCKS5_Auth_Reply;
+};
+
+type SOCKS5_Auth_Request = case $context.connection.v5_auth_method() of {
+	0x02    -> userpass    : SOCKS5_Auth_Request_UserPass;
+	default -> unsupported : SOCKS5_Unsupported_Authentication_Method;
+};
+
+type SOCKS5_Unsupported_Authentication_Method = record {
+	crap: bytestring &restofdata;
+};
+
+type SOCKS5_Unsupported_Authentication_Version(version: uint8) = record {
+	crap: bytestring &restofdata;
+};
+
+type SOCKS5_Auth_Request_UserPass = record {
+	version: uint8;
+	msg:     case version of {
+		1       -> v1:           SOCKS5_Auth_Request_UserPass_v1;
+		default -> unsupported:  SOCKS5_Unsupported_Authentication_Version(version);
+	};
+};
+
+type SOCKS5_Auth_Request_UserPass_v1 = record {
+	ulen     : uint8;
+	username : bytestring &length=ulen;
+	plen     : uint8;
+	password : bytestring &length=plen;
+};
+
+type SOCKS5_Auth_Reply = case $context.connection.v5_auth_method() of {
+	0x02    -> userpass    : SOCKS5_Auth_Reply_UserPass;
+	default -> unsupported : SOCKS5_Unsupported_Authentication_Method;
+} &let {
+	in_auth_sub_neg = $context.connection.set_v5_in_auth_sub_negotiation(false);
+};
+
+type SOCKS5_Auth_Reply_UserPass = record {
+	version: uint8;
+	msg:     case version of {
+		1       -> v1:           SOCKS5_Auth_Reply_UserPass_v1;
+		default -> unsupported:  SOCKS5_Unsupported_Authentication_Version(version);
+	};
+};
+
+type SOCKS5_Auth_Reply_UserPass_v1 = record {
+	code : uint8;
 };
 
 type SOCKS5_Real_Message(is_orig: bool) = case is_orig of {
@@ -55,10 +115,10 @@ type SOCKS5_Address = record {
 } &byteorder = bigendian;
 
 type SOCKS5_Request = record {
-	command: uint8;
+	command     : uint8;
-	reserved: uint8;
+	reserved    : uint8;
-	remote_name: SOCKS5_Address;
+	remote_name : SOCKS5_Address;
-	port: uint16;
+	port        : uint16;
 } &byteorder = bigendian;
 
 type SOCKS5_Reply = record {
@@ -98,13 +158,28 @@ type SOCKS4_Reply = record {
 
 refine connection SOCKS_Conn += {
 	%member{
+		bool v5_in_auth_sub_negotiation_;
 		bool v5_authenticated_;
+		uint8 selected_auth_method_;
 	%}
 
 	%init{
+		v5_in_auth_sub_negotiation_ = false;
 		v5_authenticated_ = false;
+		selected_auth_method_ = 255;
 	%}
 
+	function v5_in_auth_sub_negotiation(): bool
+		%{
+		return v5_in_auth_sub_negotiation_;
+		%}
+
+	function set_v5_in_auth_sub_negotiation(b: bool): bool
+		%{
+		v5_in_auth_sub_negotiation_ = b;
+		return true;
+		%}
+
 	function v5_past_authentication(): bool
 		%{
 		return v5_authenticated_;
@@ -115,5 +190,16 @@ refine connection SOCKS_Conn += {
 		v5_authenticated_ = true;
 		return true;
 		%}
+
+	function set_v5_auth_method(method: uint8): bool
+		%{
+		selected_auth_method_ = method;
+		return true;
+		%}
+
+	function v5_auth_method(): uint8
+		%{
+		return selected_auth_method_;
+		%}
 };
 

src/analyzer/protocol/socks/socks.pac

@@ -20,7 +20,7 @@ connection SOCKS_Conn(bro_analyzer: BroAnalyzer) {
 %include socks-protocol.pac
 
 flow SOCKS_Flow(is_orig: bool) {
-	datagram = SOCKS_Version(is_orig) withcontext(connection, this);
+	datagram = SOCKS_Message(is_orig) withcontext(connection, this);
 };
 
 %include socks-analyzer.pac

src/analyzer/protocol/ssl/ssl-analyzer.pac

@@ -112,7 +112,10 @@ refine connection SSL_Conn += {
 					cipher_suites24 : uint24[]) : bool
 		%{
 		if ( ! version_ok(version) )
+			{
 			bro_analyzer()->ProtocolViolation(fmt("unsupported client SSL version 0x%04x", version));
+			bro_analyzer()->SetSkip(true);
+			}
 		else
 			bro_analyzer()->ProtocolConfirmation();
 
@@ -152,7 +155,10 @@ refine connection SSL_Conn += {
 					comp_method : uint8) : bool
 		%{
 		if ( ! version_ok(version) )
+			{
 			bro_analyzer()->ProtocolViolation(fmt("unsupported server SSL version 0x%04x", version));
+			bro_analyzer()->SetSkip(true);
+			}
 
 		if ( ssl_server_hello )
 			{
@@ -202,6 +208,7 @@ refine connection SSL_Conn += {
 			// This should be impossible due to the binpac parser
 			// and protocol description
 			bro_analyzer()->ProtocolViolation(fmt("Impossible extension length: %lu", length));
+			bro_analyzer()->SetSkip(true);
 			return true;
 			}
 
@@ -392,7 +399,11 @@ refine connection SSL_Conn += {
 	function proc_check_v2_server_hello_version(version: uint16) : bool
 		%{
 		if ( version != SSLv20 )
+			{
 			bro_analyzer()->ProtocolViolation(fmt("Invalid version in SSL server hello. Version: %d", version));
+			bro_analyzer()->SetSkip(true);
+			return false;
+			}
 
 		return true;
 		%}
@@ -479,13 +490,13 @@ refine typeattr ServerHello += &let {
 };
 
 refine typeattr V2ServerHello += &let {
-	proc : bool = $context.connection.proc_server_hello(rec, server_version, 0,
-				conn_id_data, 0, 0, ciphers, 0);
-
 	check_v2 : bool = $context.connection.proc_check_v2_server_hello_version(server_version);
 
+	proc : bool = $context.connection.proc_server_hello(rec, server_version, 0,
+				conn_id_data, 0, 0, ciphers, 0) &requires(check_v2) &if(check_v2 == true);
+
 	cert : bool = $context.connection.proc_v2_certificate(rec, cert_data)
-		&requires(proc);
+		&requires(proc) &requires(check_v2) &if(check_v2 == true);
 };
 
 refine typeattr Certificate += &let {

src/analyzer/protocol/ssl/ssl-protocol.pac

@@ -36,7 +36,7 @@ type SSLRecord(is_orig: bool) = record {
 } &length = length+5, &byteorder=bigendian,
 	&let {
 	version : int =
-		$context.connection.determine_ssl_record_layer(head0, head1, head2, head3, head4);
+		$context.connection.determine_ssl_record_layer(head0, head1, head2, head3, head4, is_orig);
 
 	content_type : int = case version of {
 		SSLv20 -> head2+300;
@@ -748,7 +748,7 @@ refine connection SSL_Conn += {
 		%}
 
 	function determine_ssl_record_layer(head0 : uint8, head1 : uint8,
-					head2 : uint8, head3: uint8, head4: uint8) : int
+					head2 : uint8, head3: uint8, head4: uint8, is_orig: bool) : int
 		%{
 		// re-check record layer version to be sure that we still are synchronized with
 		// the data stream
@@ -759,6 +759,7 @@ refine connection SSL_Conn += {
 			     version != TLSv11 && version != TLSv12 )
 				{
 				bro_analyzer()->ProtocolViolation(fmt("Invalid version late in TLS connection. Packet reported version: %d", version));
+				bro_analyzer()->SetSkip(true);
 				return UNKNOWN_VERSION;
 				}
 			}
@@ -768,13 +769,14 @@ refine connection SSL_Conn += {
 
 		if ( head0 & 0x80 )
 			{
-			if ( head2 == 0x01 ) // SSLv2 client hello.
+			if ( head2 == 0x01 && is_orig ) // SSLv2 client hello.
 				{
 				uint16 version = (head3 << 8) | head4;
 				if ( version != SSLv20 && version != SSLv30 && version != TLSv10 &&
 				     version != TLSv11 && version != TLSv12 )
 					{
 					bro_analyzer()->ProtocolViolation(fmt("Invalid version in SSL client hello. Version: %d", version));
+					bro_analyzer()->SetSkip(true);
 					return UNKNOWN_VERSION;
 					}
 
@@ -782,7 +784,7 @@ refine connection SSL_Conn += {
 					return SSLv20;
 				}
 
-			else if ( head2 == 0x04 ) // SSLv2 server hello. This connection will continue using SSLv2.
+			else if ( head2 == 0x04 && head4 < 2 && ! is_orig ) // SSLv2 server hello. This connection will continue using SSLv2.
 				{
 				record_layer_version_ = SSLv20;
 				return SSLv20;
@@ -791,6 +793,7 @@ refine connection SSL_Conn += {
 			else // this is not SSL or TLS.
 				{
 				bro_analyzer()->ProtocolViolation(fmt("Invalid headers in SSL connection. Head1: %d, head2: %d, head3: %d", head1, head2, head3));
+				bro_analyzer()->SetSkip(true);
 				return UNKNOWN_VERSION;
 				}
 			}
@@ -800,6 +803,7 @@ refine connection SSL_Conn += {
 		     version != TLSv11 && version != TLSv12 )
 			{
 			bro_analyzer()->ProtocolViolation(fmt("Invalid version in TLS connection. Version: %d", version));
+			bro_analyzer()->SetSkip(true);
 			return UNKNOWN_VERSION;
 			}
 
@@ -810,6 +814,7 @@ refine connection SSL_Conn += {
 			}
 
 		bro_analyzer()->ProtocolViolation(fmt("Invalid type in TLS connection. Version: %d, Type: %d", version, head0));
+		bro_analyzer()->SetSkip(true);
 		return UNKNOWN_VERSION;
 		%}
 

src/analyzer/protocol/tcp/TCP_Reassembler.cc

@@ -28,7 +28,7 @@ TCP_Reassembler::TCP_Reassembler(analyzer::Analyzer* arg_dst_analyzer,
 				TCP_Analyzer* arg_tcp_analyzer,
 				TCP_Reassembler::Type arg_type,
 				TCP_Endpoint* arg_endp)
-	: Reassembler(1, REASSEM_TCP)
+	: Reassembler(1)
 	{
 	dst_analyzer = arg_dst_analyzer;
 	tcp_analyzer = arg_tcp_analyzer;