Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-02 06:38:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1

Merge remote-tracking branch 'origin/topic/timw/seven-two-news'

Browse source 
* origin/topic/timw/seven-two-news:
  Updates for the various Broker changes
  Add versions of bundled dependencies
  Fix a few typos.
  Additional user contributions for NEWS
  NEWS addition for cluster backends
  NEWS additions for 7.2
  Reformat 7.2 NEWS entries for consistent line lengths
This commit is contained in:
Tim Wojtulewicz 2025-04-25 10:25:01 -07:00
commit 4f65b89edf
3 changed files with 220 additions and 75 deletions
Download patch file Download diff file Expand all files Collapse all files

42
CHANGES
View file

@ -1,3 +1,45 @@
7.2.0-dev.657 | 2025-04-25 10:25:01 -0700
* Updates for the various Broker changes (Christian Kreibich, Corelight)
* Add versions of bundled dependencies (Tim Wojtulewicz, Corelight)
* Lower listen/connect retry intervals in Broker and the cluster framework to 1sec (Christian Kreibich, Corelight)
The former defaults (30sec, 1min) can slow down cluster startup and recovery
considerably, and other systems have more aggressive intervals still.
* Bump cluster testsuite (Christian Kreibich, Corelight)
This pulls in an update for the backpressure disconnect tests, which now need to
set the policy explicitly.
* Switch Broker's default backpressure policy to drop_oldest, bump buffer sizes (Christian Kreibich, Corelight)
At every site where we've dug into backpressure disconnect findings, it has been
the case that the default values were too small. 8192, so 4x the old default,
suffices at every site to drown out premature disconnects.
With metrics now available for the send buffers regardless of backpressure
overflow policy, this also switches the default from "disconnect" to
"drop_oldest" (for both peers and websockets), meaning that peerings remain
untouched but the oldest queued message simply gets dropped when a new message
is enqueued. With this policy, the number of backpressure overflows is then
simply the count of discarded messages, something that users can tune to see
drop to zero in everyday use. Another benefit is that marginal overflows cause
less message loss than when an entire buffer's worth (plus potentially more
in-flight messages) gets thrown out with a disconnect.
* Deprecate Broker::congestion_queue_size and stop using it internally (Christian Kreibich, Corelight)
Since a reorg in the Broker library (commit b04195183) that revamped flow
control and that we pulled in with Zeek 5.0, this setting hasn't done
anything. Broker's endpoint::make_subscriber() and
endpoint::make_status_subscriber() take a queue size argument (with a default
value) that simply gets dropped in the eventual subscriber::make() call. See:
https://github.com/zeek/broker/commit/b04195183515a1db8eb493011a744ba787239e98#diff-5c0d2baa7981caeb6a4080708ddca6ad929746d10c73d66598e46d7c2c03c8deL34-R178
7.2.0-dev.649 | 2025-04-25 10:23:30 -0700
* Lower listen/connect retry intervals in Broker and the cluster framework to 1sec (Christian Kreibich, Corelight)

251
NEWS
View file

@ -6,31 +6,48 @@ release. For an exhaustive list of changes, see the ``CHANGES`` file
Zeek 7.2.0
==========
We would like to thank Aashish Sharma (@initconf), Anthony Verez (@netantho),
Anthony Kasza (@anthonykasza), @biswajitutil, Brendan Kapp (@BrendanKapp),
Carlos Lopez, Chris Hinshaw (@MMChrisHinshaw), Faan Rossouw (@faanross),
@FishyFluffer, Fupeng Zhao (@AmazingPP), Herbert (@Herbert-Karl), @jbaggs, Jan
Grashöfer (@J-Gras), Julian Krieger (@juliankrieger), Justin Azoff
(@JustinAzoff), @Laotree, Mark Overholser (@markoverholser), Mike Dopheide
(@dopheide-esnet), @mnhsrj, Mohan Dhawan (@Mohan-Dhawan), @philipp-tg, Seth Hall
(@sethhall), and @timo-mue for their contributions to this release.
Breaking Changes
----------------
- The ``is_remote_event()``, ``current_analyzer()`` and ``current_event_time()``
builtin functions do not return the previous event's values anymore when event
draining has completed. The same applies to the corresponding C++ accessors on
the ``EventMgr`` class. The functions now return false, 0 or the zero time instead.
- The ``is_remote_event()``, ``current_analyzer()`` and ``current_event_time()`` builtin
functions do not return the previous event's values anymore when event draining has
completed. The same applies to the corresponding C++ accessors on the ``EventMgr``
class. The functions now return false, 0 or the zero time instead.
- The ``to_int()`` built-in function was changed to match the return behavior of
``to_count()``. Previously, ``to_int()`` would silently ignore invalid inputs and return a
``0``. It now returns an error instead.
New Functionality
-----------------
- Some DNS events are not raised when ``dns_skip_all_addl`` is set to true.
Zeek now raises a warning when a script declares these events while this
option is set to true.
- The following dependencies have had updates:
- Types can now be used as constants in Zeek script. This allows types to be
directly passed into BIFs without aliasing.
- The bundled version of c-ares has been updated to v1.34.5.
- A new ``enc_part`` field was added to the Kerberos ``KRB_Response`` record
passed as part of the ``krb_as_response`` event. This field contains the
encrypted session information from a Kerberos response, including the cipher
and encrypted data.
- The bundled version of ZeekJS has been updated to v0.17.0.
- Geneve tunnel options of the current packet can be extracted from scripts
using the new PacketAnalyzer::Geneve::get_options() builtin function.
- Some DNS events are not raised when ``dns_skip_all_addl`` is set to true. Zeek now
raises a warning when a script declares these events while this option is set to true.
- Types can now be used as constants in Zeek script. This allows types to be directly
passed into BIFs without aliasing.
- A new ``enc_part`` field was added to the Kerberos ``KRB_Response`` record passed as
part of the ``krb_as_response`` event. This field contains the encrypted session
information from a Kerberos response, including the cipher and encrypted data.
- Geneve tunnel options of the current packet can be extracted from scripts using the new
``PacketAnalyzer::Geneve::get_options()`` builtin function.
- The new ``is_valid_subnet()`` function mirrors ``is_valid_ip()``, for subnets.
@ -47,7 +64,7 @@ New Functionality
erasure of elements are available.
- Backends can support both asynchronous mode (using ``when`` statements) and
synchronous mode (blocking until the operation copmletes). BIF methods were added
synchronous mode (blocking until the operation completes). BIF methods were added
under new ``Storage::Async`` and ``Storage::Sync`` modules for these two modes. The
modes can be used interchangeably with the same backend handle.
@ -55,12 +72,51 @@ New Functionality
backend for NATS that will be available as an external plugin, but it is not quite
ready yet. Both of the existing backends support usage in a cluster environment.
- Broker now exposes more information through ``broker.log``. Broker generated
log messages are now propagated as events to Zeek. This allows exposing more
information for debugging and operational behavior of Broker via Zeek logs.
Two new script-level options ``Broker::log_severity_level`` and
``Broker::log_stderr_severity_level`` have been introduced to control
the which events to expose by default.
- Improved alternative cluster backend support.
The ZeroMQ cluster backend added in Zeek 7.1 has received various correctness,
performance and robustness fixes, particularly concerning shutdown and high-load
scenarios.
Initial performance testing indicates less CPU time used on a large single node
instance with high logging and eventing rates.
We're evaluating switching the default cluster backend from Broker to ZeroMQ With
Zeek 8.1. Therefore, we welcome early adopters and testers to validate ZeroMQ as an
alternative to Broker. If you're not using Broker specific integrations (e.g. Broker's
Python or C++ bindings) and run a single-node Zeek cluster, switching to ZeroMQ
should be as simple as loading the following script on each of cluster node.
@load frameworks/cluster/backend/zeromq/connect
A proof-of-concept plugin for the open-source NATS messaging system is available at
https://github.com/zeek/zeek-cluster-backend-nats for testing and experimentation.
- Broker now exposes more information through ``broker.log``. Broker generated log
messages are now propagated as events to Zeek. This allows exposing more information for
debugging and operational behavior of Broker via Zeek logs. Two new script-level
options ``Broker::log_severity_level`` and ``Broker::log_stderr_severity_level`` have
been introduced to control the which events to expose by default.
- Broker's new per-peer send buffer backpressure handling, introduced in 7.1,
has received several updates. We've increased the default buffer sizes to 8192
messages for both peers and websockets, and switched the default overflow
handling policy to "drop_oldest", meaning that in a full buffer the oldest
message enqueued gets dropped to allow enqueuing a new one. Three additional
metrics are available to understand the health of each peering's buffer,
regardless of the overflow policy active. These are:
- zeek_broker_peer_buffer_levels: a gauge of the current buffer fill level,
- zeek_broker_peer_buffer_recent_max_levels: a gauge that tracks the maximum
buffer fill level seen over the last ``Broker::buffer_stats_reset_interval`.
- zeek_broker_peer_buffer_overflows_total: a counter that tracks the number
of times a given peering's send buffer has overflowed. For the "drop_oldest"
and "drop_newest" policies, this is the count of messages dropped.
Each of these is labeled with the current endpoint and the peer's, as provided
by the cluster topology.
- New WebSocket functionality was added to Zeek's cluster component.
@ -86,88 +142,135 @@ New Functionality
that client may still be in transit and later executed, even on the node running
the WebSocket server.
- Vectors containing ``pattern`` values can now be compared using ``==`` and ``!=`` in
scripts. This previously resulted in a fatal error.
- The set of non-routable subnets defined in ``Site::private_address_space`` was expanded
to include ``239.0.0.0/8``, ``224.0.0.0/24`, ``[2002:e000::]/40``, ``[2002:ef00::]/24``,
and ``[fec0::]/10`. These addresses come from RFCs 2365, 3058, 3879, and 5771. This may
result in traffic being considered as local traffic that wasn't previously.
- The ``to_count()`` and ``to_int()`` built-in functions now trim trailing spaces passed
in the argument. They were already trimming leading spaces.
- The ``ip_proto`` field is now populated for a connection encapsulated in a tunnel.
- The documentation for ZeekJS is now included in the main Zeek documentation (as seen on
https://docs.zeek.org) by default.
- Searching for the headers for libkrb5 was made more robust. Additionally, the
restrictions on using libkrb5 only on Linux platforms was removed. CMake will now search
for it on all platforms as expected.
Changed Functionality
---------------------
- The ``service`` field in the connection log is now sorted in the order that
protocol analyzers raise their confirmation events.
Since the time at which the protocol confirmation is raised depends on the
individual implementation of each analyzer, there is no specific meaning
to the order that the services appear. However, the order should be
deterministic between runs. It also will in many cases represent
the order in which layered protocols are parsed (e.g. "quic,ssl").
- The ``service`` field in the connection log is now sorted in the order that protocol
analyzers raise their confirmation events. Since the time at which the protocol
confirmation is raised depends on the individual implementation of each analyzer, there
is no specific meaning to the order that the services appear. However, the order should
be deterministic between runs. It also will in many cases represent the order in which
layered protocols are parsed (e.g. "quic,ssl").
- The way that protocol violations are handled by the dynamic protocol
detection (DPD) changed. Now, a violation that is raised by an analyzer
before it is confirmed will immediately disable the analyzer. This adjusts
the behavior back to the historically desired state, and aligns it with
the treatment of confirmed analyzers.
- The way that protocol violations are handled by the dynamic protocol detection (DPD)
changed. Now, a violation that is raised by an analyzer before it is confirmed will
immediately disable the analyzer. This adjusts the behavior back to the historically
desired state, and aligns it with the treatment of confirmed analyzers.
As a consequence of this, the option ``DPD::max_violations`` is no longer used.
It will be retained till Zeek 8.1 to prevent script errors, and raises a
deprecation warning.
To extend the visibility of protocol violations, a new option
``DPD::track_removed_services_in_connection`` was added. Enabling it causes
failed analyzers to no longer be removed from the ``service`` field of the
connection log. Instead, analyzers are never removed after they are
confirmed. Instead, failed analyzers are logged by additionally adding an
entry with a prepended "-". So a connection that attached the ``ssl``
analyzer which later failed due to a protocol error will be logged as
``ssl,-ssl``.
``DPD::track_removed_services_in_connection`` was added. Enabling it causes failed
analyzers to no longer be removed from the ``service`` field of the connection
log. Instead, analyzers are never removed after they are confirmed. Instead, failed
analyzers are logged by additionally adding an entry with a prepended "-". So a
connection that attached the ``ssl`` analyzer which later failed due to a protocol error
will be logged as ``ssl,-ssl``.
This change also adds a new policy script,
``protocols/conn/failed-service-logging.zeek``. Loading this script adds the
column ``failed_service`` to the connection.log. This column contains the
list of protocol analyzers that failed due to a protocol error.
``protocols/conn/failed-service-logging.zeek``. Loading this script adds the column
``failed_service`` to the connection.log. This column contains the list of protocol
analyzers that failed due to a protocol error.
- Command line options processing will no longer print usage whenever there
is an error. Instead, issues in command line processing will print an error,
then prompt to use --help. The --help usage will now print to standard output
rather than standard error.
- Command line options processing will no longer print usage whenever there is an
error. Instead, issues in command line processing will print an error, then prompt to
use --help. The --help usage will now print to standard output rather than standard
error.
- Saving seeds with ``--save-seeds`` will now put Zeek into deterministic mode.
A subsequent ``--load-seeds`` run with the same scripts and traces will produce
identical UID values as the original ``--save-seeds` run.
- Saving seeds with ``--save-seeds`` will now put Zeek into deterministic mode. A
subsequent ``--load-seeds`` run with the same scripts and traces will produce identical
UID values as the original ``--save-seeds` run.
- The `policy/protocols/dns/detect-external-names.zeek` script now no longer logs
names that were found in mDNS broadcasts by default. This is configurable with
the new `DNS::skip_resp_host_port_pairs` option.
- The `policy/protocols/dns/detect-external-names.zeek` script now no longer logs names
that were found in mDNS broadcasts by default. This is configurable with the new
`DNS::skip_resp_host_port_pairs` option.
Furthermore, the script now supports and logs IPv6 results.
- The ``mkdir()``, ``rmdir()``, ``unlink()``, and ``rename()`` functions now
trigger reporter warnings instead of builtin errors when hitting trouble. This
allows Zeek to continue gracefully in case of such problems, particularly
during ``zeek_init()``.
- The ``mkdir()``, ``rmdir()``, ``unlink()``, and ``rename()`` functions now trigger
reporter warnings instead of builtin errors when hitting trouble. This allows Zeek to
continue gracefully in case of such problems, particularly during ``zeek_init()``.
- The RDP analyzer now also parses connections that do not contain the cookie
field, which were previously rejected.
- The RDP analyzer now also parses connections that do not contain the cookie field, which
were previously rejected.
- An enum's zeek::detail::ID instance now holds its ``EnumVal``. For example,
looking up the "Conn::LOG" identifier allows to directly query the ``EnumVal``
using ``ID::GetVal()``.
- An enum's zeek::detail::ID instance now holds its ``EnumVal``. For example, looking up
the "Conn::LOG" identifier allows to directly query the ``EnumVal`` using
``ID::GetVal()``.
- When the send buffer to a Broker peer overflows and the "disconnect" overflow
policy is in use, Zeek now only attempts to re-establish peerings when the
node observing the overflow originally established the peering. That is,
re-peering is now only attempted in consistency with the underlying Broker
peering topology. This avoids pointless connection attempts to ephemeral TCP
client-side ports, which clould clutter the Broker logs.
- When the send buffer to a Broker peer overflows and the "disconnect" overflow policy is
in use, Zeek now only attempts to re-establish peerings when the node observing the
overflow originally established the peering. That is, re-peering is now only attempted
in consistency with the underlying Broker peering topology. This avoids pointless
connection attempts to ephemeral TCP client-side ports, which could clutter the Broker
logs.
- The connect and listen retry intervals of Broker and the Cluster framework
have all been reduced to one second, from previously 30s/60s.
- The protocol confirmation for IRC was made more robust. It now checks for valid commands
before confirming a connection as IRC.
- Packet dumping now properly handles both the inner and outer packets of a tunneled
connection, ensuring that the outer packets are always dumped correctly alongside the
inner packets.
- SSH banner parsing was previously a bit too strict in some ways and too permissive in
others. This has been changed to be more robust, now accepting text before the SSH
banner starts. This was previously a protocol violation but is actually allowed by the
spec. This should help prevent non-ssh traffic on port 22 from causing an ssh.log to be
created. A new event called ``ssh_server_pre_banner_data`` was added, and is set When
this kind of text data is encountered.
- The SNAP analyzer now uses both the OUI and protocol identifier in forwarding
decisions. Previously it only used the identifier, which lead to some packets not being
handled at all and also not being logged in ``unknown_protocols.log``.
- The BIND library is no longer required for building Zeek. It hasn't been required since
our switch to use the C-Ares library back in the 5.0 release, but we never removed the
requirement from CMake.
Removed Functionality
---------------------
- Broker's broker_buffered_messages metric has been removed, since the
backpressure handling introduced in 7.1 rendered it obsolete. Use the new
per-peering metrics described above instead.
Deprecated Functionality
------------------------
- Support for DNS resolution of hostname literals in Zeek scripts has been
deprecated. If you've used this feature, use the new ``blocking_lookup_hostname()``
builtin function to populate sets or tables in a ``zeek_init()`` handler,
or with top-level statements.
- Support for DNS resolution of hostname literals in Zeek scripts has been deprecated. If
you've used this feature, use the new ``blocking_lookup_hostname()`` builtin function to
populate sets or tables in a ``zeek_init()`` handler, or with top-level statements.
- ``Broker::listen_websocket()`` was deprecated in favor of ``Cluster::listen_websocket()`.
- The ``Broker::congestion_queue_size`` tunable has had no effect since Zeek 5.0
and is slated for removal without replacement.
Zeek 7.1.0
==========
@ -364,7 +467,7 @@ New Functionality
- Zeek now ships with an experimental Spicy-based SSL analyzer, which is
disabled by default. This analyzer can be enabled using the
``--enable-spicy-ssl`` conifgure-time option. The Spicy-based analyzer has
``--enable-spicy-ssl`` configure-time option. The Spicy-based analyzer has
full support for SSL and TLS, just like the current binpac analyzer. It does,
however, not support any version of DTLS. Enabling it will disable DTLS
parsing in Zeek.

2
VERSION
View file

@ -1 +1 @@
7.2.0-dev.649
7.2.0-dev.657