Update NEWS (finalizations/formatting)

CHANGES

@@ -1,4 +1,8 @@
 
+2.5-952 | 2018-08-31 17:30:21 -0500
+
+  * Update NEWS (finalizations/formatting) (Jon Siwek, Corelight)
+
 2.5-951 | 2018-08-31 15:33:31 -0500
 
   * Improve `make dist` (Jon Siwek, Corelight)

NEWS

@@ -4,9 +4,8 @@ release. For an exhaustive list of changes, see the ``CHANGES`` file
 (note that submodules, such as BroControl and Broccoli, come with
 their own ``CHANGES``.)
 
-
+Bro 2.6
-Bro 2.6 (in progress)
+=======
-=====================
 
 New Functionality
 -----------------
@@ -16,22 +15,26 @@ New Functionality
   to the version in 2.5), and much of its implementation has been
   redone. There's a new script-level "broker" framework that
   supersedes the old "communication" framework, which is now
-  deprecated. The "cluster" and "control" frameworks have been ported
+  deprecated.  All scripts that ship with Bro have been ported to use
-  to Broker; same for BroControl. For more about the new Broker
+  to Broker.  BroControl has likewise been ported to use Broker.
-  framework, see doc/frameworks/broker.rst (there's also a guide there
-  for porting existing Bro scripts to Broker). For more about Broker
-  itself, including its API for external applications, see
-  aux/broker/doc.
 
-  TODO: Replace documentation paths with URLs once these are available
+  For more about the new Broker framework, see
-  online.
+  https://www.bro.org/sphinx-git/frameworks/broker.html.  There's also
+  a guide there for porting existing Bro scripts to Broker. For more
+  about Broker itself, including its API for external applications,
+  see https://bro-broker.readthedocs.io/en/stable
 
-  When using BroControl, the meaning of proxies has changed with
+  When using BroControl, the function of proxies has changed with
   Broker. If you are upgrading and have configured more than one proxy
   currenty, we recommend going back down to a single proxy node now.
-  Unless you are using custom scripts doing significant data
+  That should be fine unless you are using custom scripts doing
-  distribution themselves through the new cluster framework, that
+  significant data distribution through the new cluster framework.
-  should be fine.
+
+  A side effect of the switch to using Broker is a reduced number of
+  file descriptors being polled in Bro's main event loop (1 per worker
+  versus 5).  This should increase the number of workers one can
+  use before reaching the common 1024 file descriptor limitation of
+  "select()".
 
 - Bro now has new "is" and "as" script operators for dynamic
   type-checking and casting.
@@ -71,7 +74,7 @@ New Functionality
         s=Foo
         s=default
 
-- The existing "switch" got extended to now also support switching by
+- The existing "switch" statement got extended to now also support switching by
   type rather than value. The new syntax supports two type-based versions
   of "case":
 
@@ -114,23 +117,22 @@ New Functionality
   - Option variables: The new "option" keyword allows variables to be
     declared as runtime options. Such variables cannot be changed
     using normal assignments. Instead, they can be changed using the
-    new function Config::set_value.  This function will automatically
+    new function "Config::set_value".  This function will automatically
     apply the change to all nodes in a cluster.  Note that options can also
-    be changed using the new function Option::set, but this function will
+    be changed using the new function "Option::set", but this function will
     not send the change to any other nodes, so Config::set_value should
     typically be used instead of Option::set.
 
     Various redef-able constants in the standard Bro scripts have
     been converted to runtime options.  This change will not affect any
     user scripts because the initial value of runtime options can still be
-    redefined with a "redef" declaration.  Example:
+    redefined with a "redef" declaration.  Example::
 
         option testvar = "old value";
-
         redef testvar = "new value";
 
     It is possible to "subscribe" to an option through
-    Option::set_change_handler, which will trigger a handler callback
+    "Option::set_change_handler", which will trigger a handler callback
     when an option changes. Change handlers can optionally modify
     values before they are applied by returning the desired value, or
     reject updates by returning the old value. Priorities can be
@@ -157,12 +159,12 @@ New Functionality
   - Script-level configuration framework: The new script framework
     base/framework/config facilitates reading in new option values
     from external files at runtime. The format for these files looks
-    like this:
+    like this::
 
         [option name][tab/spaces][new variable value]
 
     Configuration files to read can be specified by adding them to
-    Config::config_files.
+    "Config::config_files".
 
     Usage example::
 
@@ -175,10 +177,10 @@ New Functionality
         }
 
     The specified file will now be monitored continuously for changes, so
-    that writing "TestConfig::testbool T" into /path/to/config.dat will
+    that writing "TestConfig::testbool T" into ``/path/to/config.dat`` will
     automatically update the option's value accordingly.
 
-    The configuration framework creates a config.log that shows all
+    The configuration framework creates a ``config.log`` that shows all
     value changes that took place.
 
   - Config reader: Internally, the configuration framework uses a new
@@ -210,14 +212,23 @@ New Functionality
 - Support for OCSP and Signed Certificate Timestamp. This adds the
   following events and BIFs:
 
-  - Events: ocsp_request, ocsp_request_certificate,
+  - Events:
-    ocsp_response_status, ocsp_response_bytes
-    ocsp_response_certificate ocsp_extension
-    x509_ocsp_ext_signed_certificate_timestamp
-    ssl_extension_signed_certificate_timestamp
 
-  - Functions: sct_verify, x509_subject_name_hash,
+    - ocsp_request
-    x509_issuer_name_hash x509_spki_hash
+    - ocsp_request_certificate
+    - ocsp_response_status
+    - ocsp_response_bytes
+    - ocsp_response_certificate
+    - ocsp_extension
+    - x509_ocsp_ext_signed_certificate_timestamp
+    - ssl_extension_signed_certificate_timestamp
+
+  - Functions
+
+    - sct_verify
+    - x509_subject_name_hash
+    - x509_issuer_name_hash
+    - x509_spki_hash
 
 - The SSL scripts provide a new hook "ssl_finishing(c: connection)"
   to trigger actions after the handshake has concluded.
@@ -226,21 +237,28 @@ New Functionality
   events. These events mostly extract information from the server and client
   key exchange messages. The new events are:
 
-  ssl_ecdh_server_params, ssl_dh_server_params, ssl_server_signature,
+  - ssl_ecdh_server_params
-  ssl_ecdh_client_params, ssl_dh_client_params, ssl_rsa_client_pms
+  - ssl_dh_server_params
+  - ssl_server_signature
+  - ssl_ecdh_client_params
+  - ssl_dh_client_params
+  - ssl_rsa_client_pms
 
-  Since ssl_ecdh_server_params contains more information than the old
+  Since "ssl_ecdh_server_params" contains more information than the old
-  ssl_server_curve event, ssl_server_curve is now marked as deprecated.
+  "ssl_server_curve" event, "ssl_server_curve" is now marked as deprecated.
 
-- The ssl_application_data event was retired and replaced with ssl_plaintext_data.
+- The "ssl_application_data" event was retired and replaced with
+  "ssl_plaintext_data".
 
 - Some SSL events were changed and now provide additional data. These events
   are:
 
-  ssl_client_hello, ssl_server_hello, ssl_encrypted_data
+  - ssl_client_hello
+  - ssl_server_hello
+  - ssl_encrypted_data
 
   If you use these events, you can make your scripts work on old and new versions
-  of Bro by wrapping the event definition in an @if, for example:
+  of Bro by wrapping the event definition in an "@if", for example::
 
     @if ( Version::at_least("2.6") || ( Version::number == 20500 && Version::info$commit >= 944 ) )
     event ssl_client_hello(c: connection, version: count, record_version: count, possible_ts: time, client_random: string, session_id: string, ciphers: index_vec, comp_methods: index_vec)
@@ -249,33 +267,50 @@ New Functionality
     @endif
 
 - Functions for retrieving files by their ID have been added:
-  Files::file_exists, Files::lookup_File
 
-- New functions in the logging API: Log::get_filter_names, Log::enable_stream
+  - Files::file_exists
+  - Files::lookup_File
+
+- New functions in the logging API
+
+  - Log::get_filter_names
+  - Log::enable_stream
 
 - HTTP now recognizes and skips upgraded/websocket connections.  A new event,
-  http_connection_upgrade, is raised in such cases.
+  "http_connection_upgrade", is raised in such cases.
 
 - Added a MOUNT3 protocol parser
 
   - This is not enabled by default (no ports are registered and no
     DPD signatures exist, so no connections will end up attaching the
     new Mount analyzer).  If it were to be activated by users, the
-    following events are available: mount_proc_null, mount_proc_mnt,
+    following events are available
-    mount_proc_umnt, mount_proc_umnt_all, mount_proc_not_implemented,
-    mount_reply_status.
 
-- Added new NFS events: nfs_proc_symlink, nfs_proc_link, nfs_proc_sattr.
+    - mount_proc_null
+    - mount_proc_mnt
+    - mount_proc_umnt
+    - mount_proc_umnt_all
+    - mount_proc_not_implemented
+    - mount_reply_status
 
-- The SMB scripts in policy/protocols/smb are now moved into
+- Added new NFS events
-  base/protocols/smb and loaded/enabled by default.  If you previously
+
-  loaded these scripts from their policy/ location (in local.bro or
+  - nfs_proc_symlink
+  - nfs_proc_link
+  - nfs_proc_sattr
+
+- The SMB scripts in ``policy/protocols/smb`` are now moved into
+  ``base/protocols/smb`` and loaded/enabled by default.  If you previously
+  loaded these scripts from their ``policy/ location`` (in local.bro or
   other custom scripts) you may now remove/change those although they
-  should still work since policy/protocols/smb is simply a placeholder
+  should still work since ``policy/protocols/smb`` is simply a placeholder
-  script that redirects to the new base/ location.
+  script that redirects to the new ``base/`` location.
 
-- Added new SMB events: smb1_transaction_secondary_request,
+- Added new SMB events
-  smb1_transaction2_secondary_request, smb1_transaction_response.
+
+  - smb1_transaction_secondary_request
+  - smb1_transaction2_secondary_request
+  - smb1_transaction_response
 
 - Bro can now decrypt Kerberos tickets, and retrieve the authentication from
   them, given a suitable keytab file.
@@ -303,7 +338,7 @@ New Functionality
   match "Foo", but it will match "foo".
 
 - "make install" now installs Bro's include headers (and more) into
-  --prefix so that compiling plugins no longer needs access to a
+  "--prefix" so that compiling plugins no longer needs access to a
   source/build tree. For OS distributions, this also facilitates
   creating "bro-devel" packages providing all files necessary to build
   plugins.
@@ -326,9 +361,14 @@ New Functionality
 
 - The above connection history behaviors occurring multiple times
   (i.e., starting at 10 instances, than again for 100 instances,
-  etc.) generate corresponding events: tcp_multiple_checksum_errors,
+  etc.) generate corresponding events:
-  udp_multiple_checksum_errors, tcp_multiple_zero_windows, and
+
-  tcp_multiple_retransmissions.  Each has the same form, e.g.
+  - tcp_multiple_checksum_errors
+  - udp_multiple_checksum_errors
+  - tcp_multiple_zero_windows
+  - tcp_multiple_retransmissions
+
+  Each has the same form, e.g.::
 
       event tcp_multiple_retransmissions(c: connection, is_orig: bool,
 				                         threshold: count);
@@ -348,7 +388,7 @@ New Functionality
   with &redef by appending the result of expressions "a", "b", and "c" to
   the vector at initialization-time.
 
-- A new @deprecated directive was added. It marks a script-file as
+- A new "@deprecated" directive was added. It marks a script-file as
   deprecated.
 
 Changed Functionality
@@ -365,10 +405,10 @@ Changed Functionality
   - The DHCP log now represents DHCP sessions based on transaction ID
     and works on Bro cluster deployments.
 
-  - Removed the policy/protocols/dhcp/known-devices-and-hostnames.bro
+  - Removed the ``policy/protocols/dhcp/known-devices-and-hostnames.bro``
     script since it's generally less relevant now with the updated log.
 
-  - Removed the base/protocols/dhcp/utils.bro script and thus the
+  - Removed the ``base/protocols/dhcp/utils.bro`` script and thus the
     "reverse_ip" function.
 
   - Replaced all DHCP events with the single "dhcp_message" event.
@@ -383,17 +423,17 @@ Changed Functionality
     - dhcp_release
     - dhcp_inform
 
-  - A new script, policy/protocols/dhcp/deprecated_events.bro, may be loaded
+  - A new script, ``policy/protocols/dhcp/deprecated_events.bro``, may be
-    to aid those transitioning away from the list of "removed" events above.
+    loaded to aid those transitioning away from the list of "removed"
-    The script provides definitions for the old events and automatically
+    events above.  The script provides definitions for the old events
-    generates them from a dhcp_message handler, thus providing equivalent
+    and automatically generates them from a "dhcp_message" handler, thus
-    functionality to the previous Bro release.  Such usage emits deprecation
+    providing equivalent functionality to the previous Bro release.
-    warnings.
+    Such usage emits deprecation warnings.
 
-- Removed policy/misc/known-devices.bro script and thus
+- Removed ``policy/misc/known-devices.bro`` script and thus
-  known_devices.log will no longer be created.
+  ``known_devices.log`` will no longer be created.
 
-- The --with-binpac= configure option has changed to mean "path
+- The "--with-binpac" configure option has changed to mean "path
   to the binpac executable" instead of "path to binpac installation root".
 
 - The MIME types used to identify X.509 certificates in SSL
@@ -401,46 +441,49 @@ Changed Functionality
   "application/x-x509-user-cert" for host certificates and
   "application/x-x509-ca-cert" for CA certificates.
 
-- With the new ssl_ecdh_server_params event, the ssl_server_curve
+- The "ssl_server_curve" event is considered deprecated and will be removed
-  event is considered deprecated and will be removed in a future
+  in in the future.  See the new "ssl_ecdh_server_params" event for a
-  version of Bro.
+  replacement.
 
 - The Socks analyzer no longer logs passwords by default. This
   brings its behavior in line with the FTP/HTTP analyzers which also
   do not log passwords by default.
 
-  To restore the previous behavior and log Socks passwords, use:
+  To restore the previous behavior and log Socks passwords, use::
 
       redef SOCKS::default_capture_password = T;
 
 - The DNS base scripts no longer generate some noisy and annoying
-  weirds (dns_unmatched_msg, dns_unmatched_msg_quantity, dns_unmatched_reply).
+  weirds:
 
-- The "tunnel_parents" field of conn.log is now marked &optional, so, for
+  - dns_unmatched_msg
-  the default configuration of logs, this field will show "-" instead of
+  - dns_unmatched_msg_quantity
-  "(empty)" for connections that lack any tunneling.
+  - dns_unmatched_reply
+
+- The "tunnel_parents" field of ``conn.log`` is now marked ``&optional``, so,
+  in the default configuration of logs, this field will show "-"
+  instead of "(empty)" for connections that lack any tunneling.
 
 - SMB event argument changes:
 
-  - smb1_transaction_request now has two additional arguments, "parameters"
+  - "smb1_transaction_request" now has two additional arguments, "parameters"
     and "data" strings
 
-  - smb1_transaction2_request now has an additional "args" record argument
+  - "smb1_transaction2_request" now has an additional "args" record argument
 
-- The SMB::write_cmd_log option has been removed and the corresponding
+- The "SMB::write_cmd_log" option has been removed and the corresponding
-  logic moving to policy/protocols/smb/log-cmds.bro which can simply
+  logic moving to ``policy/protocols/smb/log-cmds.bro`` which can simply
   be loaded to produce the same effect of toggling the old flag on.
 
 - SSL event argument changes:
 
-  - event ssl_server_signature now has an additional argument
+  - "ssl_server_signature" now has an additional argument
     "signature_and_hashalgorithm".
 
 - The "dnp3_header_block" event no longer has the "start" parameter.
 
-- The string_to_pattern() built-in (and the now-deprecated merge_pattern()
+- The "string_to_pattern()" and now-deprecated "merge_pattern()"
-  built-in) is no longer restricted to only be called at initialization time.
+  built-ins are no longer restricted to only be called at initialization time.
-
 
 - GeoIP Legacy Database support has been replaced with GeoIP2 MaxMind DB
   format support.
@@ -461,10 +504,15 @@ Changed Functionality
 
   Those options can be changed if one needs the previous behavior of
   a "net_weird", "flow_weird", or "conn_weird" event being raised for
-  every single event.  Otherwise, there is a new weird_stats.log which
+  every single event.
-  contains concise summaries of weird counts per type per time period
+
-  and the original weird.log may not differ much either, except in
+  The original ``weird.log`` may not differ much with these changes,
-  the cases where a particular weird type exceeds the sampling threshold.
+  except in the cases where a particular weird type exceeds the
+  sampling threshold.
+
+  Otherwise, there is a new ``weird_stats.log`` generated via
+  ``policy/misc/weird-stats.bro`` which contains concise summaries
+  of weird counts per type per time period.
 
 - Improved DCE-RPC analysis via tracking of context identifier mappings
 
@@ -479,6 +527,9 @@ Changed Functionality
     - dce_rpc_alter_context
     - dce_rpc_alter_context_resp
 
+- The default value of ``Pcap::snaplen`` changed from 8192 to 9216 bytes
+  to better accommodate jumbo frames.
+
 Removed Functionality
 ---------------------
 
@@ -495,28 +546,25 @@ Removed Functionality
 
 - The node-specific ``site/local-*.bro`` scripts have been removed.
 
-- The default value of ``Pcap::snaplen`` changed from 8192 to 9216 bytes
-  to better accommodate jumbo frames.
-
 Deprecated Functionality
 ------------------------
 
 - The old communication system is now deprecated and scheduled for
   removal with the next Bro release. This includes the "communication"
-  framework, the &sychronized attributes, and the existing
+  framework, the ``&sychronized`` attributes, and the existing
   communication-related BiFs. Use Broker instead.
 
 - The infrastructure for serializing Bro values into a binary
   representation is now deprecated and scheduled for removal with the
-  next Bro release. This includes the &persistent attribute, as well
+  next Bro release. This includes the ``&persistent`` attribute, as well
-  as BiFs like send_id(). Use Broker data stores and the new
+  as BIFs like "send_id()". Use Broker data stores and the new
   configuration framework instead.
 
 - Mixing of scalars and vectors, such as "v + e" yielding a vector
   corresponding to the vector v with the scalar e added to each of
   its elements, has been deprecated.
 
-- The built-in function merge_pattern() has been deprecated.  It will
+- The built-in function "merge_pattern()" has been deprecated.  It will
   be replaced by the '&' operator for patterns.
 
 - The undocumented feature of using "&&" and "||" operators for patterns
@@ -526,6 +574,93 @@ Deprecated Functionality
   removal with the next Bro release. Bro's new configuration framework
   is taking its place.
 
+Bro 2.5.5
+=========
+
+Bro 2.5.5 primarily addresses security issues.
+
+- Fix array bounds checking in BinPAC: for arrays that are fields within
+  a record, the bounds check was based on a pointer to the start of the
+  record rather than the start of the array field, potentially resulting
+  in a buffer over-read.
+
+- Fix SMTP command string comparisons: the number of bytes compared was
+  based on the user-supplied string length and can lead to incorrect
+  matches.  e.g. giving a command of "X" incorrectly matched
+  "X-ANONYMOUSTLS" (and an empty commands match anything).
+
+The following changes address potential vectors for Denial of Service
+reported by Christian Titze & Jan Grashöfer of Karlsruhe Institute of
+Technology:
+
+- "Weird" events are now generally suppressed/sampled by default according
+   to some tunable parameters:
+
+  - Weird::sampling_whitelist
+  - Weird::sampling_threshold
+  - Weird::sampling_rate
+  - Weird::sampling_duration
+
+  Those options can be changed if one needs the previous behavior of
+  a "net_weird", "flow_weird", or "conn_weird" event being raised for
+  every single event.  Otherwise, there is a new weird_stats.log which
+  contains concise summaries of weird counts per type per time period
+  and the original weird.log may not differ much either, except in
+  the cases where a particular weird type exceeds the sampling threshold.
+  These changes help improve performance issues resulting from excessive
+  numbers of weird events.
+
+- Improved handling of empty lines in several text protocol analyzers
+  that can cause performance issues when seen in long sequences.
+
+- Add 'smtp_excessive_pending_cmds' weird which serves as a notification
+  for when the "pending command" queue has reached an upper limit and
+  been cleared to prevent one from attempting to slowly exhaust memory.
+
+Bro 2.5.4
+=========
+
+Bro 2.5.4 primarily fixes security issues:
+
+* Multiple fixes and improvements to BinPAC generated code related to
+  array parsing, with potential impact to all Bro's BinPAC-generated
+  analyzers in the form of buffer over-reads or other invalid memory
+  accesses depending on whether a particular analyzer incorrectly
+  assumed that the evaulated-array-length expression is actually the
+  number of elements that were parsed out from the input.
+
+* The NCP analyzer (not enabled by default and also updated to actually
+  work with newer Bro APIs in the release) performed a memory allocation
+  based directly on a field in the input packet and using signed integer
+  storage.  This could result in a signed integer overflow and memory
+  allocations of negative or very large size, leading to a crash or
+  memory exhaustion.  The new NCP::max_frame_size tuning option now
+  limits the maximum amount of memory that can be allocated.
+
+There's also the following bug fixes:
+
+* A memory leak in the SMBv1 analyzer.
+
+* The MySQL analyzer was generally not working as intended, for example,
+  it now is able to parse responses that contain multiple results/rows.
+
+Bro 2.5.3
+=========
+
+Bro 2.5.3 fixes a security issue in Binpac generated code. In some cases
+the code generated by binpac could lead to an integer overflow which can
+lead to out of bound reads and allow a remote attacker to crash Bro; there
+is also a possibility that this can be exploited in other ways.
+
+Bro 2.5.2
+=========
+
+Bro 2.5.2 fixes a security issue in the ContentLine analyzer. In rare cases
+a bug in the ContentLine analyzer can lead to an out of bound write of a single
+byte. This allows a remote attacker to crash Bro; there also is a possibility
+this can be exploited in other ways. CVE-2017-1000458 has been assigned to this
+issue.
+
 Bro 2.5.1
 =========
 

VERSION

@@ -1 +1 @@
-2.5-951
+2.5-952