Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-02 06:38:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Update NEWS (finalizations/formatting)

Browse source
This commit is contained in:
Jon Siwek 2018-08-31 17:15:44 -05:00
parent d1e4dbe5e3
commit 56c14fb6d5
3 changed files with 237 additions and 98 deletions
Download patch file Download diff file Expand all files Collapse all files

329
NEWS
View file

@ -4,9 +4,8 @@ release. For an exhaustive list of changes, see the ``CHANGES`` file
(note that submodules, such as BroControl and Broccoli, come with
their own ``CHANGES``.)
Bro 2.6 (in progress)
=====================
Bro 2.6
=======
New Functionality
-----------------
@ -16,22 +15,26 @@ New Functionality
to the version in 2.5), and much of its implementation has been
redone. There's a new script-level "broker" framework that
supersedes the old "communication" framework, which is now
deprecated. The "cluster" and "control" frameworks have been ported
to Broker; same for BroControl. For more about the new Broker
framework, see doc/frameworks/broker.rst (there's also a guide there
for porting existing Bro scripts to Broker). For more about Broker
itself, including its API for external applications, see
aux/broker/doc.
deprecated. All scripts that ship with Bro have been ported to use
to Broker. BroControl has likewise been ported to use Broker.
TODO: Replace documentation paths with URLs once these are available
online.
For more about the new Broker framework, see
https://www.bro.org/sphinx-git/frameworks/broker.html. There's also
a guide there for porting existing Bro scripts to Broker. For more
about Broker itself, including its API for external applications,
see https://bro-broker.readthedocs.io/en/stable
When using BroControl, the meaning of proxies has changed with
When using BroControl, the function of proxies has changed with
Broker. If you are upgrading and have configured more than one proxy
currenty, we recommend going back down to a single proxy node now.
Unless you are using custom scripts doing significant data
distribution themselves through the new cluster framework, that
should be fine.
That should be fine unless you are using custom scripts doing
significant data distribution through the new cluster framework.
A side effect of the switch to using Broker is a reduced number of
file descriptors being polled in Bro's main event loop (1 per worker
versus 5). This should increase the number of workers one can
use before reaching the common 1024 file descriptor limitation of
"select()".
- Bro now has new "is" and "as" script operators for dynamic
type-checking and casting.
@ -71,7 +74,7 @@ New Functionality
s=Foo
s=default
- The existing "switch" got extended to now also support switching by
- The existing "switch" statement got extended to now also support switching by
type rather than value. The new syntax supports two type-based versions
of "case":
@ -114,23 +117,22 @@ New Functionality
- Option variables: The new "option" keyword allows variables to be
declared as runtime options. Such variables cannot be changed
using normal assignments. Instead, they can be changed using the
new function Config::set_value. This function will automatically
new function "Config::set_value". This function will automatically
apply the change to all nodes in a cluster. Note that options can also
be changed using the new function Option::set, but this function will
be changed using the new function "Option::set", but this function will
not send the change to any other nodes, so Config::set_value should
typically be used instead of Option::set.
Various redef-able constants in the standard Bro scripts have
been converted to runtime options. This change will not affect any
user scripts because the initial value of runtime options can still be
redefined with a "redef" declaration. Example:
redefined with a "redef" declaration. Example::
option testvar = "old value";
redef testvar = "new value";
It is possible to "subscribe" to an option through
Option::set_change_handler, which will trigger a handler callback
"Option::set_change_handler", which will trigger a handler callback
when an option changes. Change handlers can optionally modify
values before they are applied by returning the desired value, or
reject updates by returning the old value. Priorities can be
@ -157,12 +159,12 @@ New Functionality
- Script-level configuration framework: The new script framework
base/framework/config facilitates reading in new option values
from external files at runtime. The format for these files looks
like this:
like this::
[option name][tab/spaces][new variable value]
Configuration files to read can be specified by adding them to
Config::config_files.
"Config::config_files".
Usage example::
@ -175,10 +177,10 @@ New Functionality
}
The specified file will now be monitored continuously for changes, so
that writing "TestConfig::testbool T" into /path/to/config.dat will
that writing "TestConfig::testbool T" into ``/path/to/config.dat`` will
automatically update the option's value accordingly.
The configuration framework creates a config.log that shows all
The configuration framework creates a ``config.log`` that shows all
value changes that took place.
- Config reader: Internally, the configuration framework uses a new
@ -210,14 +212,23 @@ New Functionality
- Support for OCSP and Signed Certificate Timestamp. This adds the
following events and BIFs:
- Events: ocsp_request, ocsp_request_certificate,
ocsp_response_status, ocsp_response_bytes
ocsp_response_certificate ocsp_extension
x509_ocsp_ext_signed_certificate_timestamp
ssl_extension_signed_certificate_timestamp
- Events:
- Functions: sct_verify, x509_subject_name_hash,
x509_issuer_name_hash x509_spki_hash
- ocsp_request
- ocsp_request_certificate
- ocsp_response_status
- ocsp_response_bytes
- ocsp_response_certificate
- ocsp_extension
- x509_ocsp_ext_signed_certificate_timestamp
- ssl_extension_signed_certificate_timestamp
- Functions
- sct_verify
- x509_subject_name_hash
- x509_issuer_name_hash
- x509_spki_hash
- The SSL scripts provide a new hook "ssl_finishing(c: connection)"
to trigger actions after the handshake has concluded.
@ -226,21 +237,28 @@ New Functionality
events. These events mostly extract information from the server and client
key exchange messages. The new events are:
ssl_ecdh_server_params, ssl_dh_server_params, ssl_server_signature,
ssl_ecdh_client_params, ssl_dh_client_params, ssl_rsa_client_pms
- ssl_ecdh_server_params
- ssl_dh_server_params
- ssl_server_signature
- ssl_ecdh_client_params
- ssl_dh_client_params
- ssl_rsa_client_pms
Since ssl_ecdh_server_params contains more information than the old
ssl_server_curve event, ssl_server_curve is now marked as deprecated.
Since "ssl_ecdh_server_params" contains more information than the old
"ssl_server_curve" event, "ssl_server_curve" is now marked as deprecated.
- The ssl_application_data event was retired and replaced with ssl_plaintext_data.
- The "ssl_application_data" event was retired and replaced with
"ssl_plaintext_data".
- Some SSL events were changed and now provide additional data. These events
are:
ssl_client_hello, ssl_server_hello, ssl_encrypted_data
- ssl_client_hello
- ssl_server_hello
- ssl_encrypted_data
If you use these events, you can make your scripts work on old and new versions
of Bro by wrapping the event definition in an @if, for example:
of Bro by wrapping the event definition in an "@if", for example::
@if ( Version::at_least("2.6") || ( Version::number == 20500 && Version::info$commit >= 944 ) )
event ssl_client_hello(c: connection, version: count, record_version: count, possible_ts: time, client_random: string, session_id: string, ciphers: index_vec, comp_methods: index_vec)
@ -249,33 +267,50 @@ New Functionality
@endif
- Functions for retrieving files by their ID have been added:
Files::file_exists, Files::lookup_File
- New functions in the logging API: Log::get_filter_names, Log::enable_stream
- Files::file_exists
- Files::lookup_File
- New functions in the logging API
- Log::get_filter_names
- Log::enable_stream
- HTTP now recognizes and skips upgraded/websocket connections. A new event,
http_connection_upgrade, is raised in such cases.
"http_connection_upgrade", is raised in such cases.
- Added a MOUNT3 protocol parser
- This is not enabled by default (no ports are registered and no
DPD signatures exist, so no connections will end up attaching the
new Mount analyzer). If it were to be activated by users, the
following events are available: mount_proc_null, mount_proc_mnt,
mount_proc_umnt, mount_proc_umnt_all, mount_proc_not_implemented,
mount_reply_status.
following events are available
- Added new NFS events: nfs_proc_symlink, nfs_proc_link, nfs_proc_sattr.
- mount_proc_null
- mount_proc_mnt
- mount_proc_umnt
- mount_proc_umnt_all
- mount_proc_not_implemented
- mount_reply_status
- The SMB scripts in policy/protocols/smb are now moved into
base/protocols/smb and loaded/enabled by default. If you previously
loaded these scripts from their policy/ location (in local.bro or
- Added new NFS events
- nfs_proc_symlink
- nfs_proc_link
- nfs_proc_sattr
- The SMB scripts in ``policy/protocols/smb`` are now moved into
``base/protocols/smb`` and loaded/enabled by default. If you previously
loaded these scripts from their ``policy/ location`` (in local.bro or
other custom scripts) you may now remove/change those although they
should still work since policy/protocols/smb is simply a placeholder
script that redirects to the new base/ location.
should still work since ``policy/protocols/smb`` is simply a placeholder
script that redirects to the new ``base/`` location.
- Added new SMB events: smb1_transaction_secondary_request,
smb1_transaction2_secondary_request, smb1_transaction_response.
- Added new SMB events
- smb1_transaction_secondary_request
- smb1_transaction2_secondary_request
- smb1_transaction_response
- Bro can now decrypt Kerberos tickets, and retrieve the authentication from
them, given a suitable keytab file.
@ -303,7 +338,7 @@ New Functionality
match "Foo", but it will match "foo".
- "make install" now installs Bro's include headers (and more) into
--prefix so that compiling plugins no longer needs access to a
"--prefix" so that compiling plugins no longer needs access to a
source/build tree. For OS distributions, this also facilitates
creating "bro-devel" packages providing all files necessary to build
plugins.
@ -326,9 +361,14 @@ New Functionality
- The above connection history behaviors occurring multiple times
(i.e., starting at 10 instances, than again for 100 instances,
etc.) generate corresponding events: tcp_multiple_checksum_errors,
udp_multiple_checksum_errors, tcp_multiple_zero_windows, and
tcp_multiple_retransmissions. Each has the same form, e.g.
etc.) generate corresponding events:
- tcp_multiple_checksum_errors
- udp_multiple_checksum_errors
- tcp_multiple_zero_windows
- tcp_multiple_retransmissions
Each has the same form, e.g.::
event tcp_multiple_retransmissions(c: connection, is_orig: bool,
threshold: count);
@ -348,7 +388,7 @@ New Functionality
with &redef by appending the result of expressions "a", "b", and "c" to
the vector at initialization-time.
- A new @deprecated directive was added. It marks a script-file as
- A new "@deprecated" directive was added. It marks a script-file as
deprecated.
Changed Functionality
@ -365,10 +405,10 @@ Changed Functionality
- The DHCP log now represents DHCP sessions based on transaction ID
and works on Bro cluster deployments.
- Removed the policy/protocols/dhcp/known-devices-and-hostnames.bro
- Removed the ``policy/protocols/dhcp/known-devices-and-hostnames.bro``
script since it's generally less relevant now with the updated log.
- Removed the base/protocols/dhcp/utils.bro script and thus the
- Removed the ``base/protocols/dhcp/utils.bro`` script and thus the
"reverse_ip" function.
- Replaced all DHCP events with the single "dhcp_message" event.
@ -383,17 +423,17 @@ Changed Functionality
- dhcp_release
- dhcp_inform
- A new script, policy/protocols/dhcp/deprecated_events.bro, may be loaded
to aid those transitioning away from the list of "removed" events above.
The script provides definitions for the old events and automatically
generates them from a dhcp_message handler, thus providing equivalent
functionality to the previous Bro release. Such usage emits deprecation
warnings.
- A new script, ``policy/protocols/dhcp/deprecated_events.bro``, may be
loaded to aid those transitioning away from the list of "removed"
events above. The script provides definitions for the old events
and automatically generates them from a "dhcp_message" handler, thus
providing equivalent functionality to the previous Bro release.
Such usage emits deprecation warnings.
- Removed policy/misc/known-devices.bro script and thus
known_devices.log will no longer be created.
- Removed ``policy/misc/known-devices.bro`` script and thus
``known_devices.log`` will no longer be created.
- The --with-binpac= configure option has changed to mean "path
- The "--with-binpac" configure option has changed to mean "path
to the binpac executable" instead of "path to binpac installation root".
- The MIME types used to identify X.509 certificates in SSL
@ -401,46 +441,49 @@ Changed Functionality
"application/x-x509-user-cert" for host certificates and
"application/x-x509-ca-cert" for CA certificates.
- With the new ssl_ecdh_server_params event, the ssl_server_curve
event is considered deprecated and will be removed in a future
version of Bro.
- The "ssl_server_curve" event is considered deprecated and will be removed
in in the future. See the new "ssl_ecdh_server_params" event for a
replacement.
- The Socks analyzer no longer logs passwords by default. This
brings its behavior in line with the FTP/HTTP analyzers which also
do not log passwords by default.
To restore the previous behavior and log Socks passwords, use:
To restore the previous behavior and log Socks passwords, use::
redef SOCKS::default_capture_password = T;
redef SOCKS::default_capture_password = T;
- The DNS base scripts no longer generate some noisy and annoying
weirds (dns_unmatched_msg, dns_unmatched_msg_quantity, dns_unmatched_reply).
weirds:
- The "tunnel_parents" field of conn.log is now marked &optional, so, for
the default configuration of logs, this field will show "-" instead of
"(empty)" for connections that lack any tunneling.
- dns_unmatched_msg
- dns_unmatched_msg_quantity
- dns_unmatched_reply
- The "tunnel_parents" field of ``conn.log`` is now marked ``&optional``, so,
in the default configuration of logs, this field will show "-"
instead of "(empty)" for connections that lack any tunneling.
- SMB event argument changes:
- smb1_transaction_request now has two additional arguments, "parameters"
- "smb1_transaction_request" now has two additional arguments, "parameters"
and "data" strings
- smb1_transaction2_request now has an additional "args" record argument
- "smb1_transaction2_request" now has an additional "args" record argument
- The SMB::write_cmd_log option has been removed and the corresponding
logic moving to policy/protocols/smb/log-cmds.bro which can simply
- The "SMB::write_cmd_log" option has been removed and the corresponding
logic moving to ``policy/protocols/smb/log-cmds.bro`` which can simply
be loaded to produce the same effect of toggling the old flag on.
- SSL event argument changes:
- event ssl_server_signature now has an additional argument
- "ssl_server_signature" now has an additional argument
"signature_and_hashalgorithm".
- The "dnp3_header_block" event no longer has the "start" parameter.
- The string_to_pattern() built-in (and the now-deprecated merge_pattern()
built-in) is no longer restricted to only be called at initialization time.
- The "string_to_pattern()" and now-deprecated "merge_pattern()"
built-ins are no longer restricted to only be called at initialization time.
- GeoIP Legacy Database support has been replaced with GeoIP2 MaxMind DB
format support.
@ -461,10 +504,15 @@ Changed Functionality
Those options can be changed if one needs the previous behavior of
a "net_weird", "flow_weird", or "conn_weird" event being raised for
every single event. Otherwise, there is a new weird_stats.log which
contains concise summaries of weird counts per type per time period
and the original weird.log may not differ much either, except in
the cases where a particular weird type exceeds the sampling threshold.
every single event.
The original ``weird.log`` may not differ much with these changes,
except in the cases where a particular weird type exceeds the
sampling threshold.
Otherwise, there is a new ``weird_stats.log`` generated via
``policy/misc/weird-stats.bro`` which contains concise summaries
of weird counts per type per time period.
- Improved DCE-RPC analysis via tracking of context identifier mappings
@ -479,6 +527,9 @@ Changed Functionality
- dce_rpc_alter_context
- dce_rpc_alter_context_resp
- The default value of ``Pcap::snaplen`` changed from 8192 to 9216 bytes
to better accommodate jumbo frames.
Removed Functionality
---------------------
@ -495,28 +546,25 @@ Removed Functionality
- The node-specific ``site/local-*.bro`` scripts have been removed.
- The default value of ``Pcap::snaplen`` changed from 8192 to 9216 bytes
to better accommodate jumbo frames.
Deprecated Functionality
------------------------
- The old communication system is now deprecated and scheduled for
removal with the next Bro release. This includes the "communication"
framework, the &sychronized attributes, and the existing
framework, the ``&sychronized`` attributes, and the existing
communication-related BiFs. Use Broker instead.
- The infrastructure for serializing Bro values into a binary
representation is now deprecated and scheduled for removal with the
next Bro release. This includes the &persistent attribute, as well
as BiFs like send_id(). Use Broker data stores and the new
next Bro release. This includes the ``&persistent`` attribute, as well
as BIFs like "send_id()". Use Broker data stores and the new
configuration framework instead.
- Mixing of scalars and vectors, such as "v + e" yielding a vector
corresponding to the vector v with the scalar e added to each of
its elements, has been deprecated.
- The built-in function merge_pattern() has been deprecated. It will
- The built-in function "merge_pattern()" has been deprecated. It will
be replaced by the '&' operator for patterns.
- The undocumented feature of using "&&" and "||" operators for patterns
@ -526,6 +574,93 @@ Deprecated Functionality
removal with the next Bro release. Bro's new configuration framework
is taking its place.
Bro 2.5.5
=========
Bro 2.5.5 primarily addresses security issues.
- Fix array bounds checking in BinPAC: for arrays that are fields within
a record, the bounds check was based on a pointer to the start of the
record rather than the start of the array field, potentially resulting
in a buffer over-read.
- Fix SMTP command string comparisons: the number of bytes compared was
based on the user-supplied string length and can lead to incorrect
matches. e.g. giving a command of "X" incorrectly matched
"X-ANONYMOUSTLS" (and an empty commands match anything).
The following changes address potential vectors for Denial of Service
reported by Christian Titze & Jan Grashöfer of Karlsruhe Institute of
Technology:
- "Weird" events are now generally suppressed/sampled by default according
to some tunable parameters:
- Weird::sampling_whitelist
- Weird::sampling_threshold
- Weird::sampling_rate
- Weird::sampling_duration
Those options can be changed if one needs the previous behavior of
a "net_weird", "flow_weird", or "conn_weird" event being raised for
every single event. Otherwise, there is a new weird_stats.log which
contains concise summaries of weird counts per type per time period
and the original weird.log may not differ much either, except in
the cases where a particular weird type exceeds the sampling threshold.
These changes help improve performance issues resulting from excessive
numbers of weird events.
- Improved handling of empty lines in several text protocol analyzers
that can cause performance issues when seen in long sequences.
- Add 'smtp_excessive_pending_cmds' weird which serves as a notification
for when the "pending command" queue has reached an upper limit and
been cleared to prevent one from attempting to slowly exhaust memory.
Bro 2.5.4
=========
Bro 2.5.4 primarily fixes security issues:
* Multiple fixes and improvements to BinPAC generated code related to
array parsing, with potential impact to all Bro's BinPAC-generated
analyzers in the form of buffer over-reads or other invalid memory
accesses depending on whether a particular analyzer incorrectly
assumed that the evaulated-array-length expression is actually the
number of elements that were parsed out from the input.
* The NCP analyzer (not enabled by default and also updated to actually
work with newer Bro APIs in the release) performed a memory allocation
based directly on a field in the input packet and using signed integer
storage. This could result in a signed integer overflow and memory
allocations of negative or very large size, leading to a crash or
memory exhaustion. The new NCP::max_frame_size tuning option now
limits the maximum amount of memory that can be allocated.
There's also the following bug fixes:
* A memory leak in the SMBv1 analyzer.
* The MySQL analyzer was generally not working as intended, for example,
it now is able to parse responses that contain multiple results/rows.
Bro 2.5.3
=========
Bro 2.5.3 fixes a security issue in Binpac generated code. In some cases
the code generated by binpac could lead to an integer overflow which can
lead to out of bound reads and allow a remote attacker to crash Bro; there
is also a possibility that this can be exploited in other ways.
Bro 2.5.2
=========
Bro 2.5.2 fixes a security issue in the ContentLine analyzer. In rare cases
a bug in the ContentLine analyzer can lead to an out of bound write of a single
byte. This allows a remote attacker to crash Bro; there also is a possibility
this can be exploited in other ways. CVE-2017-1000458 has been assigned to this
issue.
Bro 2.5.1
=========