8.1.0-dev.50 | 2025-08-12 17:42:46 -0700

  * Add NEWS entry for field length limiting (Tim Wojtulewicz, Corelight)

  * Tag truncated values with a flag, plus pack threading::Value better (Tim Wojtulewicz, Corelight)

  * Remove length limiting on string fields for HTTP (Tim Wojtulewicz, Corelight)

  * Make total_size counter a member in logging::Manager (Tim Wojtulewicz, Corelight)

  * Remove using numeric_limits and just check for zero instead (Tim Wojtulewicz, Corelight)

  * Expand the size of the log-size filters for x509 (Tim Wojtulewicz, Corelight)

  * Add options to filter at the stream level as well as globally (Tim Wojtulewicz, Corelight)

  * Add a weird that gets emitted when strings/containers are over the limits (Tim Wojtulewicz, Corelight)

  * Add metrics to track string and container fields limited by length (Tim Wojtulewicz, Corelight)

  * Replace unused stream argument from RecordToLogRecord with WriterInfo (Tim Wojtulewicz, Corelight)

    This also adds a WriterInfo argument to ValToLogVal and passes the one from
    RecordToLogRecord into it.

  * Implement string- and container-length filtering at the log record level (Tim Wojtulewicz, Corelight)

8.1.0-dev.38 | 2025-08-12 12:38:24 -0700

  * Bump pre-commit hooks (Benjamin Bannier, Corelight)

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

8.1.0-dev.35 | 2025-08-12 11:01:12 -0700

  * Pass DNS complete_flag along as a uint8_t instead of a String (Tim Wojtulewicz, Corelight)

  * Update docs submodule with 8.1 deprecation removals (Tim Wojtulewicz, Corelight)

  * Update zeekjs submodule with 8.1 deprecation fixes (Tim Wojtulewicz, Corelight)

  * Remove deprecations tagged for v8.1 (Tim Wojtulewicz, Corelight)

8.1.0-dev.29 | 2025-08-11 11:37:22 -0700

  * smb2/read: Parse only 1 byte for data_offset, ignore reserved1 (Arne Welzel, Corelight)

    A user provided a SMB2 pcap with the reserved1 field of a ReadResponse
    set to 1 instead of 0. This confused the padding computation due to
    including this byte into the offset. Properly split data_offset and
    reserved1 into individual byte fields.

    Closes #4730

8.1.0-dev.27 | 2025-08-10 21:28:33 -0700

  * GH-4176: cluster: Add on_subscribe() and on_unsubscribe() hooks (Arne Welzel, Corelight)

    Closes #4176

8.1.0-dev.24 | 2025-08-08 14:23:51 +0200

  * GH-4176: cluster: Add on_subscribe() and on_unsubscribe() hooks (Arne Welzel, Corelight)

8.1.0-dev.22 | 2025-08-07 08:39:27 -0700

  * Update zeek-aux to remove BRO_DIST from plugin skeleton (Tim Wojtulewicz, Corelight)

  * cmake_minimum_required() should come before project() (Tim Wojtulewicz, Corelight)

8.1.0-dev.19 | 2025-08-07 10:48:40 +0200

  * btest/tap-analyzer: Update existing test and add new one for UpdateConnVal() (Arne Welzel, Corelight)

    This also changes the output of connection UIDs from the tap analyzer to be
    prefixed with C for easier correlation with other logs.

  * SessionAdapter: Keep tap_analyzers until destruction (Arne Welzel, Corelight)

    connection_state_remove() is invoked after Done(), so it's not a good
    idea to remove the tap analyzers before in case they have up-to-date
    information for the connection val.

  * tcp,udp,icmp adapters: Move TapPacket() to earlier (Arne Welzel, Corelight)

    Writing a test, the packet was tapped after protocol analysis at least
    for TCP. Ensure tapping happens before. The adapter->Process() moving
    after pkt->session made me a bit wondering if things are underspecified
    here, but seems reasonable to set the session on pkt before adapter->Process().

  * tcp,udp,icmp adapters: Fix UpdateConnVal() superclass call (Arne Welzel, Corelight)

    Now that SessionAdapter implements UpdateConnVal(), the individual
    adapters need to call that instead of Analyzer::UpdateConnVal()

    Thanks clang-tidy.

8.1.0-dev.14 | 2025-08-06 14:37:50 +0100

  * Add proto to analyzer.log (Johanna Amann, Corelight)

    The analyzer.log file was missing the protocol field to distinguish
    tcp/udp connections.

8.1.0-dev.12 | 2025-08-06 09:10:08 -0400

  * GH-4722: Fix record coercion with compatible types (Evan Typanski, Corelight)

    Fixes #4722

8.1.0-dev.10 | 2025-08-06 14:27:32 +0200

  * TapAnalyzer: Fix docstring (Arne Welzel, Corelight)

    Relates to #4337 #4725 #4734

  * btest/plugins/tap-analyzer: Update baseline (Arne Welzel, Corelight)

    Relates to #4337 #4725 #4734

8.1.0-dev.7 | 2025-08-05 20:00:19 +0200

  * TapAnalyzer: More verdict to action rename (Arne Welzel, Corelight)

    Relates to #4725 #4337

8.1.0-dev.5 | 2025-08-05 19:48:50 +0200

  * IPBasedAnalyzer: Call TapPacket() when skipping (Arne Welzel, Corelight)

    When skip_further_processing() is called, a TapAnalyzer should still see
    the packets as skipped with SkipReason "skipping".

  * SessionAdapter: Introduce TapAnalyzer for session adapter (Arne Welzel, Corelight)

    This commit introduces a mechanism to attach light weight analyzers to
    the root analyzer of sessions in order to tap into the packets delivered
    to child analyzer.

8.1.0-dev.2 | 2025-08-04 09:35:41 -0700

  * Compile contributors for Zeek 8.0 in the NEWS file (Christian Kreibich, Corelight)

  * Start of 8.1.0 development (Tim Wojtulewicz, Corelight)

8.0.0-dev.827 | 2025-08-01 17:10:13 +0200

  * ci/windows: No ZeroMQ cluster backend (Arne Welzel, Corelight)

    Doesn't seems there's libzmq available, so just skip building.

  * cluster/zeromq: Bail on missing ZeroMQ by default (Arne Welzel, Corelight)

8.0.0-dev.824 | 2025-08-01 14:49:35 +0200

  * Remove zeek::Span and use std::span instead (Dominik Charousset, Corelight)

8.0.0-dev.821 | 2025-07-31 14:32:00 -0700

  * Update CT/CA lists to versions from NSS 3.114 (Tim Wojtulewicz, Corelight)

8.0.0-dev.817 | 2025-07-31 10:34:43 -0700

  * Fix use-after-move reported by clang-tidy (Tim Wojtulewicz, Corelight)

    This was introduced by 9eb94ee151ec437fd076ef16d016af372391af65.

8.0.0-dev.815 | 2025-07-31 16:10:33 +0100

  * Add tests for the deprecated-dpd-log.zeek policy script (Johanna Amann, Corelight)

    This re-adds baselines for the old dpd.log to check functionality until
    its removal in 8.1

  * Move c$service_violation to deprecated-dpd-log.zeek (Johanna Amann, Corelight)

    This moves c$service_violation to the deprecated-dpd-log policy script.

    This is the only script in the distribution that uses the field, and it
    is unlikely to be used externally. It is also responsible for a
    significant amount of memory use by itself.

8.0.0-dev.812 | 2025-07-31 07:57:52 -0700

  * Bump `auxil/spicy` to latest development snapshot (Benjamin Bannier, Corelight)

8.0.0-dev.809 | 2025-07-30 10:28:06 +0200

  * RecordType: Ensure &default fields are always re-initialized (Arne Welzel, Corelight)

    This started working partly after the deferral logic introduced with
    Zeek 6.0 so this finishes it :-)

  * Attr: Deprecate using &default and &optional together on record fields (Arne Welzel, Corelight)

    If &default implies re-initialization of the field, using them together
    doesn't make much sense.

  * RecordType: Allow deferring &default=vector(), set(), table() fields (Arne Welzel, Corelight)

8.0.0-dev.805 | 2025-07-30 10:07:50 +0200

  * fixes for specialized ZAM operations needing to check whether record fields exist (Vern Paxson, Corelight)

8.0.0-dev.802 | 2025-07-29 21:00:52 +0100

  * DNS-fuzzer: raise new_connection event (Johanna Amann, Corelight)

    The conn protocol scripts now assume that new_connection is run before
    connection_state_remove. Update the DNS fuzzer to raise the
    new_connection event.

  * Move Conn::set_conn() from connection_state_remove to new_connection (Johanna Amann, Corelight)

    c$conn is often needed for connection events, but it being established
    in connection_state_removed can be a problem because event handlers have
    to remember to call Con::set_conn().

    This commit moves to call Conn::set_conn() in new_connection.

8.0.0-dev.797 | 2025-07-29 18:23:35 +0200

  * cluster::Event: Move implementation into cluster/Event.{h,cc} (Arne Welzel, Corelight)

  * cluster: Move cluster::detail::Event to cluster::Event (Arne Welzel, Corelight)

    This class is a parameter of virtual methods of the Backend API for users
    to implement and also a parameter to the HookPublishEvent() API. Seems it
    shouldn't be in detail and instead we should own it.

    Alternatively, could mark the cluster APIs as not-stable-yet, but I
    think we can move forward and make it non-detail for 8.0.

8.0.0-dev.794 | 2025-07-29 07:41:13 -0700

  * Fix ConnKey deprecation warnings from generic fuzzer (Tim Wojtulewicz, Corelight)

8.0.0-dev.792 | 2025-07-29 07:19:25 -0700

  * Fix types passed to some of the IRC analyzer events (Tim Wojtulewicz, Corelight)

8.0.0-dev.790 | 2025-07-29 07:18:39 -0700

  * Fix build failure with std::ranges on Debian 11 (Tim Wojtulewicz, Corelight)

8.0.0-dev.788 | 2025-07-29 11:29:11 +0200

  * cluster.bif: Improve Cluster::publish() docstring (Arne Welzel, Corelight)

  * btest/cluster/zeromq: Add tests for overload behavior (Arne Welzel, Corelight)

    The overload-drop.zeek and overload-no-drop.zeek tests have proxy,
    worker-1 and worker-2 publish to the manager topic. For the drop
    case, we verify that both, the senders, but also the manager drops
    events. For the no-drop test, the HWMs are set such that all events
    are buffered.

    The overload-worker-proxy-topic*.zeek tests are similar, but instead
    of publishing to the manager topic, proxy, worker-1 and worker-2 publish
    to the proxy and worker topics to overload each other. This had
    previously resulted in lockups and these tests verify that this doesn't
    happen anymore.

  * cluster/zeromq: Metric for msg errors (Arne Welzel, Corelight)

  * cluster/zeromq: Drop events when overloaded (Arne Welzel, Corelight)

    When either the XPUB socket's hwm is reached, or the onloop queue is
    full, drop the events. Users can set ths xpub_sndhwm and
    onloop_queue_hwm to 0 to avoid these drops at the risk of unbounded
    memory growth.

  * cluster/zeromq: Comments and move lookups to InitPostScript() (Arne Welzel, Corelight)

  * cluster/zeromq: Rework lambdas to member functions (Arne Welzel, Corelight)

  * cluster/zeromq: Support local XPUB/XSUB hwm and buf configurability (Arne Welzel, Corelight)

  * cluster/OnLoop: Support DontBlock and Force flags for queueing (Arne Welzel, Corelight)

    Also allow max_queue_size to be 0 for unlimited queueing.

  * cluster/ThreadedBackend: Injectable OnLoopProcess instance (Arne Welzel, Corelight)

    This allows injecting a custom onloop process to configure the
    max_queue_size at instantiation time. Also allow access to the
    instance directly and deprecate the QueueForProcessing() helper

8.0.0-dev.778 | 2025-07-29 10:12:18 +0200

  * ConnStats: Expose num_packets_unprocessed (Arne Welzel, Corelight)

    Not sure it's the best place to put, but we don't have packet analysis stats
    bif and also num_packets is already there, so seems reasonable to put the
    num_packets_unprocessed into

  * packet_analysis/Manager: Rename GetUnprocessedCount() to PacketsUnprocessed() (Arne Welzel, Corelight)

    Mostly to align with PacketsProcessed()

8.0.0-dev.775 | 2025-07-29 10:05:20 +0200

  * Update .git-blame-ignore-revs (Arne Welzel, Corelight)

  * analyzer/protocol: Reformat with spicy-format (Arne Welzel, Corelight)

  * pre-commit-config: Bump spicy-format to 0.26.0 (Arne Welzel, Corelight)

8.0.0-dev.770 | 2025-07-28 14:18:15 -0700

  * dce-rpc: Make named_pipe filed docs extensive (Arne Welzel, Corelight)

    Closes #3935

8.0.0-dev.768 | 2025-07-28 14:16:16 -0700

  * Fix parsing of EDNS rcode (Johanna Amann, Corelight)

    The EDNS rcode was incorrectly calculated. The extended rcode is formed
    by taking the upper 8 bits of the extended rcode field, plus the lower 4
    bits of the existing rcode.

    This also adds a new trace with an extended rcode, and a testcase
    parsing it.

    Reported by dwhitemv25.

    Fixes GH-4656

8.0.0-dev.766 | 2025-07-28 14:15:19 -0700

  * Expand coverage of IRC analyzer with more commands (Tim Wojtulewicz, Corelight)

  * Add support for ircv3 tags (Tim Wojtulewicz, Corelight)

    This also replaces the irc-dcc-send.trace file in our pcap library with
    a cleaner one that includes ircv3 tags.

8.0.0-dev.763 | 2025-07-28 13:57:18 -0700

  * Move ID::type in structure to fill memory padding (Tim Wojtulewicz, Corelight)

  * Use sorted forward_list instead of multimap for ID option change handlers (Tim Wojtulewicz, Corelight)

8.0.0-dev.760 | 2025-07-28 13:12:47 -0700

  * Remove intermediate cipher vectors in ssl-analyzer.pac (Tim Wojtulewicz, Corelight)

  * Reduce the size of Func::Body quite a bit with some extra packing (Tim Wojtulewicz, Corelight)

  * Switch to using std::ranges algorithms (Tim Wojtulewicz, Corelight)

  * Enable modernize-std-numbers clang-tidy checker, fix findings (Tim Wojtulewicz, Corelight)

  * Switch to using c++20 constraints instead of std::enable_if (Tim Wojtulewicz, Corelight)

8.0.0-dev.754 | 2025-07-28 12:24:41 -0400

  * Deprecate `record_type_to_vector` (Evan Typanski, Corelight)

    After #4682, record_fields is ordered so this is redundant. Furthermore,
    it is incorrectly documented, as it says nested records are flattened.

8.0.0-dev.751 | 2025-07-25 09:13:51 -0700

  * Use size_t for byte sizes in SerializationFormat (Tim Wojtulewicz, Corelight)

  * Add ODesc::Size() that returns size_t, deprecate ODesc::Len() (Tim Wojtulewicz, Corelight)

8.0.0-dev.748 | 2025-07-25 10:58:16 +0200

  * RecordType: Allow field init deferral of deferrable record constructors (Arne Welzel, Corelight)

    The ctx: conn_id_ctx &default=conn_id_ctx() field was not optimized
    as deferrable even though by default its an empty record and so deferring
    initialization seems safe. Open-code the special record constructor
    expression case in ExprFieldInit so that the ctx field is not default
    initialized at record construction anymore. I am wondering a bit if the
    same applies to &default=set() and &default=table().

  * Conn: Use conn_id_ctx singleton (Arne Welzel, Corelight)

  * Conn: Add InitPostScript() and conn_id_ctx singleton (Arne Welzel, Corelight)

  * ID: Add conn_id_ctx (Arne Welzel, Corelight)

    Seemed a bit unfortunate to use id::connection and id::conn_id, but
    then do something different for conn_id_ctx.

8.0.0-dev.742 | 2025-07-24 12:46:16 -0700

  * Update broker submodule to pull in clang 20 fix [nomail] (Tim Wojtulewicz, Corelight)

8.0.0-dev.740 | 2025-07-24 10:39:42 -0700

  * Add process_start_time_seconds and process_pid metrics (Tim Wojtulewicz, Corelight)

8.0.0-dev.738 | 2025-07-24 08:00:57 -0700

  * CI: Only run weekly tasks as part of cron (Tim Wojtulewicz, Corelight)

8.0.0-dev.736 | 2025-07-24 09:48:54 +0200

  * GH-4645: conn_key/fivetuple: Handle one-way ICMP conns in DoConnKeyFromVal() (Arne Welzel, Corelight)

    When a conn_id represents a ICMP "connection", we need to determine the
    is_one_way flag for InitTuple() in order to skip any flipping of address
    and ports for one-way ICMP connections.

8.0.0-dev.734 | 2025-07-24 07:55:31 +0100

  * PPPoE: add session id logging (Johanna Amann, Corelight)

    This adds a new PacketAnalyzer::PPPoE::session_id bif, which extracts
    the PPPoE session ID from the current packet.

    Furthermore, a new policy script is added which adds the pppoe session
    id to the connection log.

8.0.0-dev.730 | 2025-07-23 15:26:08 -0700

  * Fix a few other minor issues reported by Coverity (Tim Wojtulewicz, Corelight)

  * Add a few extra null checks, plus a missing initialization that led to a bad null check (Tim Wojtulewicz, Corelight)

  * Fix some integer overflow issues reported by Coverity (Tim Wojtulewicz, Corelight)

  * Ignore a couple of known-unused results reported by Coverity (Tim Wojtulewicz, Corelight)

  * Fix some bit-shifting overflow/UB issues reported by Coverity (Tim Wojtulewicz, Corelight)

  * Reset the value of a status variable in SQLite backend before using it in a loop (Tim Wojtulewicz, Corelight)

  * Fix a potential memory leak reported by Coverity (Tim Wojtulewicz, Corelight)

  * Avoid some string copies in IRC analyzer (Tim Wojtulewicz, Corelight)

  * Add some additional std::moves reported by Coverity (Tim Wojtulewicz, Corelight)

  * Fix an unsigned integer comparison reported by Coverity (Tim Wojtulewicz, Corelight)

  * Fix uninitialized class member Coverity findings (Tim Wojtulewicz, Corelight)

  * Handle uncaught exception during setup (Tim Wojtulewicz, Corelight)

  * Update gen-zam submodule for Coverity findings (Tim Wojtulewicz, Corelight)

8.0.0-dev.716 | 2025-07-23 14:03:47 -0700

  * Fix swapped storage metrics names (Tim Wojtulewicz, Corelight)

  * Add flag to force synchronous mode when calling storage script-land functions (Tim Wojtulewicz, Corelight)

8.0.0-dev.713 | 2025-07-23 19:31:13 +0200

  * NEWS: ZeekControl, ZeroMQ and WebSocket (Arne Welzel, Corelight)

  * Update zeekctl module for ClusterBackend and UseWebSocket (Arne Welzel, Corelight)

  * control: Use Cluster::publish() for replying (Arne Welzel, Corelight)

    Switching to ZeroMQ as cluster backend and dabbling with zeekctl
    and WebSocket, replies didn't arrive due to the usage of
    Broker::publish() rather than Cluster::publish(). Additionally,
    add the node name to the topic on which we reply so that the
    receiver can figure out which node sent the reply. It could've
    been a separate event parameter, but the topic appears just fine.

  * Fix clang-tidy findings in embedded C++ from bif files (Tim Wojtulewicz, Corelight)

8.0.0-dev.707 | 2025-07-23 08:21:17 -0700

  * CI: Add weekly task for running builds with newest compilers (Tim Wojtulewicz, Corelight)

8.0.0-dev.705 | 2025-07-23 08:20:30 -0700

  * Output more information when cloning repos for include_plugins CI task (Tim Wojtulewicz, Corelight)

8.0.0-dev.703 | 2025-07-23 13:37:56 +0200

  * Type/RecordType: Make table returned by GetRecordFieldsVal() ordered (Arne Welzel, Corelight)

    Seems only reasonable to provide that guarantee as pointed out in #4674.

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

8.0.0-dev.700 | 2025-07-22 11:40:19 -0700

  * Update zeek-af_packet-plugin submodule to fix initialization [nomail] (Tim Wojtulewicz, Corelight)

8.0.0-dev.698 | 2025-07-22 08:04:59 -0700

  * Return weird if a log line is over a configurable size limit (Tim Wojtulewicz, Corelight)

8.0.0-dev.693 | 2025-07-18 14:28:29 -0700

  * Add SQLite page_count and file_size metrics (Tim Wojtulewicz, Corelight)

  * Add btests to cover storage metrics (Tim Wojtulewicz, Corelight)

  * Add storage metrics for operations, expirations, data transferred (Tim Wojtulewicz, Corelight)

  * Fix ordering of telemtry metrics when running under test (Tim Wojtulewicz, Corelight)

  * Make RunPragma take an optional value parser to return data (Tim Wojtulewicz, Corelight)

  * Make SQLite::Step take a callback function for parsing result data (Tim Wojtulewicz, Corelight)

8.0.0-dev.684 | 2025-07-17 12:58:57 -0700

  * Update libkqueue submodule [nomail] (Tim Wojtulewicz, Corelight)

8.0.0-dev.682 | 2025-07-17 09:09:28 -0700

  * Use std::string/string_view versions of starts_with/ends_with where appropriate (Tim Wojtulewicz, Corelight)

    The util:: versions of these methods remain as a thin wrapper around them so
    they can be used with const char* arguments. Otherwise callers have to manually
    make string_view objects from the input.
    s Please enter the commit message for your changes. Lines starting

8.0.0-dev.680 | 2025-07-17 08:40:49 -0700

  * Add nolint for enum size for HILTI_RT_ENUM use (Tim Wojtulewicz, Corelight)

8.0.0-dev.677 | 2025-07-16 17:46:48 +0200

  * Spicy: Fix missing include. (Robin Sommer, Corelight)

  * Bump Spicy. (Robin Sommer, Corelight)

  * GH-4481: Spicy: Add functions to check if Zeek provides an analyzer of a given name. (Robin Sommer, Corelight)

    ```
    ## Checks if there is a Zeek analyzer of a given name.
    ##
    ## analyzer: the Zeek-side name of the analyzer to check for
    ## if_enabled: if true, only checks for analyzers that are enabled
    ##
    ## Returns the type of the analyzer if it exists, or ``Undef`` if it does not.
    public function has_analyzer(analyzer: string, if_enabled: bool = True): bool &cxxname="zeek::spicy::rt::has_analyzer";

    ## Differentiates between the types of analyzers Zeek provides.
    public type AnalyzerType = enum { Protocol, File, Packet, };

    ## Returns the type of a Zeek analyzer of a given name.
    ##
    ## analyzer: the Zeek-side name of the analyzer to check
    ## if_enabled: if true, only checks for analyzers that are enabled
    ##
    ## Returns the type of the analyzer if it exists, or ``Undef`` if it does not.
    public function analyzer_type(analyzer: string, if_enabled: bool = True): AnalyzerType &cxxname="zeek::spicy::rt::analyzer_type";

    ```

8.0.0-dev.673 | 2025-07-16 12:38:09 +0200

  * NEWS: Add entry about SMTP::enable_rfc822_msg_file_analysis (Arne Welzel, Corelight)

  * btest: Add tests for full email extraction (cccs-jsjm)

  * SMTP: Adapt scripts for enable_rfc822_msg_file_analysis (Arne Welzel, Corelight)

    Specifically, set a MIME part's parent_id to the rfc822_msg_fuid if it
    is set and take into account the current rfc822_msg_fuid for describe_file()
    to avoid fuid collisions of the top-level RFC822 message and the first
    MIME part.

  * SMTP: Add missing Undelivered() call (Arne Welzel, Corelight)

  * SMTP: Add enable_rfc822_msg_file_analysis (Arne Welzel, Corelight)

    Enabling this option will instantiate a new fa_file instance for every
    top-level RFC 822 message in an SMTP transaction.

8.0.0-dev.667 | 2025-07-15 17:37:41 -0700

  * GH-4559: Prefer explicit construction to coercion in record initialization (Benjamin Bannier, Corelight)

8.0.0-dev.665 | 2025-07-15 17:36:31 -0700

  * Add missing header to allow std::sort() on GCC 15.1 (Christian Kreibich, Corelight)

  * Make PUT on SQLite backend implicitly overwrite expired entries (Benjamin Bannier, Corelight)

  * Prevent SQLite storage backend from serving expired entries (Benjamin Bannier, Corelight)

  * Update package-manager submodule [nomail] (Tim Wojtulewicz, Corelight)

  * Update binpac submodule [nomail] (Tim Wojtulewicz, Corelight)

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

8.0.0-dev.656 | 2025-07-14 13:09:11 -0700

  * Fix some missing #includes resulting from removal of ghc::filesystem (Tim Wojtulewicz, Corelight)

  * Remove ghc::filesystem submodule, switch to std::filesystem (Tim Wojtulewicz, Corelight)

8.0.0-dev.653 | 2025-07-14 11:26:06 -0700

  * Documentation: connection_SYN_packet (Johanna Amann, Corelight)

    Slightly clarify documentation related to the connection_SYN_packet
    event.

8.0.0-dev.651 | 2025-07-14 09:31:19 -0700

  * Use ranges::reverse_view to fix a few reverse ranged-for loops (Tim Wojtulewicz)

  * Disable a few new modernize clang-tidy checkers, enabled by C++20 (Tim Wojtulewicz)

  * btest: Switch putty.org usage in tests to official URL (Arne Welzel, Corelight)

    Closes #4655

8.0.0-dev.646 | 2025-07-13 19:22:52 -0700

  * Move util::Deferred into util-types.h (Tim Wojtulewicz, Corelight)

8.0.0-dev.643 | 2025-07-11 15:35:33 -0400

  * Cleanup unused baselines (Evan Typanski, Corelight)

8.0.0-dev.641 | 2025-07-11 12:13:45 -0700

  * Add NEWS entry about C++ 20 [nomail] [skip ci] (Tim Wojtulewicz, Corelight)

8.0.0-dev.639 | 2025-07-11 12:58:06 -0400

  * crash fixes for constructs that expect add/delete expressions to return values (Vern Paxson, Corelight)

8.0.0-dev.636 | 2025-07-10 15:42:52 -0700

  * Reduce some false-positive warnings from GCC to warnings with -Werror (Tim Wojtulewicz, Corelight)

    This also works around some of the same warnings. These are known bugs
    in GCC 11+ and GCC 13.x.

  * Add zeek-namespaced alias for jthread (Tim Wojtulewicz, Corelight)

    Apple Clang (as of version 17.0.0) does not support jthread unless
    you enable experimental libcpp features by defining
    _LIBCPP_ENABLE_EXPERIMENTAL or passing -fexperimental_library to
    the compiler. Even if you don't do those, our 3rdparty jthread
    library fails to build because of a bunch of namespace/naming
    collisions. I moved our 3rdparty one to the `nonstd` namespace, and
    am using an alias under __APPLE__ to avoid the collisions.

  * Include RequireCXXStd.cmake from main CMakeLists.txt (Tim Wojtulewicz, Corelight)

  * Update submodules to C++20-enabled versions (Tim Wojtulewicz, Corelight)

8.0.0-dev.631 | 2025-07-10 08:55:39 -0700

  * Modify known-services policy script to add storage framework support (Tim Wojtulewicz, Corelight)

    This adds a flag to enable using the storage framework instead of Broker stores,
    plus a btest for both broker and the the storage framework.

8.0.0-dev.629 | 2025-07-10 09:38:32 +0200

  * Spicy: Add linker scope to debugging output. (Robin Sommer, Corelight)

  * Spicy: Remove declarations for unimplemented functions. (Benjamin Bannier, Corelight)

  * Spicy: Bump `auxil/spicy/` to latest development snapshot. (Benjamin Bannier & Robin Sommer, Corelight)

  * Spicy: Let `zeek::protocol_handle_close()` send a TCP EOF. (Benjamin Bannier, Corelight)

    Zeek's analyzer API makes it hard to determine during analyzer
    shutdown whether a regular end-of-data has been reached, or if we're
    aborting in the middle of a session (e.g., because Zeek missed the
    remaining packets): the corresponding analyzer method, `EndOfData()`
    gets called in both cases.

    In an earlier change, we had stopped signaling Spicy analyzers a
    regular finish when that `EndOfData()` method executes, because doing
    so could trigger a parse error if it wasn't a regular shutdown—-which
    isn't desired, a user request was to just silently stop processing in
    this case.

    However, that behavior now seems unfortunate in the case that one
    deliberately calls `zeek::protocol_handle_close()` to terminate an
    analyzer: this feels like a regular shutdown that should just
    immediately happen. We achieve this now in this function by
    additionally signaling the shutdown at the TCP layer as an "end of
    file", which, for Spicy analyzers, happens to run the final, orderly
    tear-down.

    Not exactly great, but ti seems to thread the needle to achieve the
    desired semantics in both cases.

8.0.0-dev.622 | 2025-07-09 15:10:03 -0700

  * ZAM optimizer fix for += / -= set operations (Vern Paxson, Corelight)

  * Bump `spicy-format` pre-commit hook (Benjamin Bannier, Corelight)

    pre-commit ignores Cargo.lock files for Rust projects, so any movement
    in a Rust project's dependencies can break a hook, even if no code in
    the hook changed. I have tried to work with upstream on a fix, but they
    basically told me they weren't interested and to get lost.

    This bumps the `spicy-format` pre-commit hook to a new version which
    explicitly deals with bumps of its dependencies. Having to do this
    semi-regularly is not fun, and ideally somebody interested in using this
    hook would help set up infrastructure in the hook so it just pulls
    pre-built binaries. This is not directly supported by pre-commit, but
    many projects work around this by declaring a Python module which then
    pulls pre-build binaries which already exist for spicy-format.

8.0.0-dev.618 | 2025-07-08 17:45:16 -0700

  * Clarify the cookie field's origin in the RDP log. (Christian Kreibich, Corelight)

8.0.0-dev.615 | 2025-07-08 17:53:56 +0200

  * Add NEWS entry for generic packet thresholds (Jan Grashoefer, Corelight)

  * Allow for multiple generic packet thresholds (Jan Grashoefer, Corelight)

    Co-authored-by: Arne Welzel <arne.welzel@corelight.com>

  * Add btest for conn_generic_packet_threshold_crossed event (Jan Grashoefer, Corelight)

  * Update dump-events btest baseline (Jan Grashoefer, Corelight)

    Changes in endpoint stats are a side-effect caused by the ConnSize
    analyzer updating the conn record triggering the threshold event. The
    phenomenon is described in https://github.com/zeek/zeek/issues/4214.

  * Add conn_generic_packet_threshold_crossed event (Jan Grashoefer, Corelight)

8.0.0-dev.609 | 2025-07-08 11:42:05 +0100

  * PPPoE: don't forward more bytes than header indicates (Johanna Amann, Corelight)

    This changes the PPPoE parser so that it doesn't forward extra bytes
    that might be appended after the payload. Instead, it raises a weird if
    the payload size doesn't match the size indicated by the header.

    This is in line with what other protocol parsers (like UDP) are doing.

    Addresses GH-4602

8.0.0-dev.605 | 2025-07-08 10:40:45 +0200

  * Bump ZeekJS to work with new Location constructor (Arne Welzel, Corelight)

  * remove non-functional column information from Location objects (Vern Paxson, Corelight)

  * isolate Location specifics to private class variables to enforce correct line number ordering (Vern Paxson, Corelight)

8.0.0-dev.601 | 2025-07-07 20:07:30 +0200

  * script_opt/IDOptInfo: Add NOLINT(bugprone-branch-clone) (Arne Welzel, Corelight)

8.0.0-dev.599 | 2025-07-07 18:17:15 +0200

  * Bump zeekjs to v0.18.0 (Arne Welzel, Corelight)

8.0.0-dev.597 | 2025-07-07 18:13:44 +0200

  * fix for error in ZAM's constant propagation logic (Vern Paxson, Corelight)

8.0.0-dev.594 | 2025-07-03 20:43:51 +0200

  * scripts: Use tpe instead of type_, again (Arne Welzel, Corelight)

    The .rst generation doesn't escape the trailing `_` and the docs build
    gets upset due to using `type` as a reference target then.

    For the better or worse, revert to using tpe. Though I acknowledge this
    means we need to be careful with trailing underscores because our docs
    build is so fragile.

    Partly reverts b9eabbabbab912c8f45c49013865a88c1256f342.

8.0.0-dev.592 | 2025-07-03 18:41:29 +0200

  * NEWS: Adapt for conn_id$ctx introduction (Arne Welzel, Corelight)

  * conn_key/fivetuple: Drop support for non conn_id records (Arne Welzel, Corelight)

    Previously, we supported any records that happened to have orig_h,
    resp_h, etc. fields, but it's not exactly clear why we ever did. Users
    that relied on this can instantiate an explicit conn_id instance, too.

  * Conn: Move conn_id init and flip to IPBasedConnKey (Arne Welzel, Corelight)

    This loosens the coupling of the script-layer conn_id record and
    the code in Conn a bit, moving more into the IPBasedConnKey class.

    I'm not quite sure whether moving the flipping logic is worth it,
    but assuming Conn could become non-IP in the future, it might.

  * IPBasedConnKey: Add GetTransportProto() helper (Arne Welzel, Corelight)

  * input/Manager: Ignore empty record types (Arne Welzel, Corelight)

    Somewhere record types with zero fields get the optional attribute
    apparently. The input/sqlite/basic test failed due to complaining
    that ctx is optional. It isn't optional and when it has zero fields
    we can just ignore it, too.

    Also adds a input framework test with an explicit empty record type

  * external: Bump commit hashes for external suites (Arne Welzel, Corelight)

    get_file_handle() may include c$id and perturbs their values when adding new
    fields. I think that's reasonable, as files transferred in one VLAN should
    be treated separate from files transferred in a different VLAN.

  * ip/vlan_fivetuple: Populate nested conn_id_context, not conn_id (Arne Welzel, Corelight)

    This also enforces conn_id and conn_id_ctx types instead of being
    able to handle any conn_id-like record.

  * ConnKey: Extend DoPopulateConnIdVal() with ctx (Arne Welzel, Corelight)

    This prepares the move where ConnKey implementations should fill out
    ctx rather than filling conn_id directly. The API continues to receive
    both, conn_id and ctx, as adding fields to `conn_id` is reasonable
    use-case even if it's just for logging purposes.

  * btest: Update tests and baselines after adding ctx to conn_id (Arne Welzel, Corelight)

  * init-bare: Add conn_id_ctx to conn_id (Arne Welzel, Corelight)

    This nested record can be used to discriminate orig_h or resp_h being
    observed in different "contexts". A context can be based on VLAN tags,
    but any custom ConnKey implementation should populate the ctx field,
    allowing to write context-aware Zeek scripts without needing to know
    what the context really is.

8.0.0-dev.581 | 2025-07-02 07:47:32 +0100

  * Spicy SSL analyzer: non-functional tweaks (Johanna Amann, Corelight).

    Move the negotiated version outside of the possible range of protocol values,
    and move exceptions to &requires & throw instead of directly calling
    spicy::decline. This is stylistic with no functional changes.

8.0.0-dev.577 | 2025-07-01 14:19:26 -0400

  * Touchup TODOs in the Redis analyzer (Evan Typanski, Corelight)

    Also renames `KnownCommand` to `RedisCommand` to avoid conflicts.

  * Handle more Redis RESP3 protocol pieces (Evan Typanski, Corelight)

    This passes the "minimum protocol version" along in the reply and adds
    support for attributes, which were added relatively recently.

  * Stringify all Redis-RESP serialized data (Evan Typanski, Corelight)

  * GH-4504: Handle Redis protocol `message` separately (Evan Typanski, Corelight)

    Closes #4504

    Messages are not typical responses, so they need special handling. This
    is different between RESP2 and 3, so this is the first instance where
    the script layer needs to tell the difference.

8.0.0-dev.571 | 2025-07-01 11:03:23 +0200

  * Bump pre-commit hooks (Benjamin Bannier, Corelight)

8.0.0-dev.569 | 2025-07-01 09:20:43 +0200

  * SQLite: Reset expiration time on overwrite (Tim Wojtulewicz, Corelight)

8.0.0-dev.566 | 2025-06-30 14:19:26 -0700

  * Make clang-tidy warnings report as errors to cause CI build to fail (Tim Wojtulewicz, Corelight)

  * Fix a clang-tidy finding in cluster telemetry code (Tim Wojtulewicz, Corelight)

  * Fix clang-tidy readability-isolate-declaration warnings (Benjamin Bannier, Corelight)

    I missed one of these in review so a machine is probably better at
    catching them.

    I fixed the existing instances which where largely in code which look
    dated. Where possible I slightly reorganized the code so we do not have
    to leave values uninitialized, but did not touch up anything else.

8.0.0-dev.562 | 2025-06-30 17:55:58 +0200

  * cluster/WebSocket: Include X-Application-Name in cluster.log (Arne Welzel, Corelight)

    A bit ad-hoc formatting for the log, but that's mostly because cluster.log
    only has message field and I don't think having a dedicated application_name
    column is worth it. That could also be added by custom scripts if it's really
    wanted for a given deployment.

8.0.0-dev.560 | 2025-06-30 13:29:07 +0200

  * cluster/Telemetry: Cache CallExpr locations (Arne Welzel, Corelight)

  * cluster/Telemetry: Avoid unneeded StringVal() construction (Arne Welzel, Corelight)

  * Val: Switch TablePatternMatcher to std::string_view (Arne Welzel, Corelight)

    ...and add TableVal::LookupPattern(std::string_view sv).

  * RE: Add MatchAll() and MatchSet() for std::string_view (Arne Welzel, Corelight)

  * cluster/websocket: Fix and test for invalid X-Application-Name (Arne Welzel, Corelight)

  * cluster/telemetry: Move topic_normalization redef to zeromq (Arne Welzel, Corelight)

8.0.0-dev.551 | 2025-06-26 20:21:31 +0100

  * Spicy SSL analyzer: make record layer version parsing more strict (Johanna Amann, Corelight)

    The Spicy based SSL analyzer was, so far, more permissive with the
    record layer versions that it would accept.

    This change brings the parsing of record layer versions in line with the
    binpac based analyzer. This behavioral difference was discovered due to
    a test that changed with the recent dpd log changes.

8.0.0-dev.548 | 2025-06-26 18:49:31 +0200

  * extend script coverage profiling to track whether conditionals evaluate to true/false (Vern Paxson, Corelight)

8.0.0-dev.546 | 2025-06-26 14:30:37 +0200

  * btest/cluster/telemetry: Add smoke testing for telemetry (Arne Welzel, Corelight)

  * cluster/WebSocket: Fetch X-Application-Name header as app label (Arne Welzel, Corelight)

  * cluster/WebSocket: Pass X-Application-Name to dispatcher (Arne Welzel, Corelight)

    This is going to be used to add labels to telemetry if the
    X-Application-Name header is set.

  * broker/WebSocketShim: Add calls to Telemetry hooks (Arne Welzel, Corelight)

    WebSocket clients with Broker do not use the normal Broker backend, so
    we need to add the telemetry invocations explicitly.

  * cluster/WebSocket: Configure telemetry for WebSocket backends (Arne Welzel, Corelight)

  * broker: Hook up generic cluster telemetry (Arne Welzel, Corelight)

  * cluster: Introduce telemetry component (Arne Welzel, Corelight)

8.0.0-dev.538 | 2025-06-26 09:58:39 +0100

  * Only pass session ticket data in ssl_session_ticket_handshake event (Johanna Amann, Corelight)

    This commit fixes the parsing of the data field in the SSL analyzer. So
    far, this field contained two extra bytes at the beginning, which
    contain the length of the following data.

8.0.0-dev.535 | 2025-06-25 19:27:01 +0200

  * Update zeek.bif (bhaskarbhar)

  * Update init-bare.zeek (bhaskarbhar)

  * Remove `list` from Zeek grammar (Evan Typanski, Corelight)

    Using it would make Zeek segfault (at least in locals). It has been an
    error for 15 years.

8.0.0-dev.530 | 2025-06-25 10:35:53 -0400

  * Remove `list` from Zeek grammar (Evan Typanski, Corelight)

    Using it would make Zeek segfault (at least in locals). It has been an
    error for 15 years.

8.0.0-dev.528 | 2025-06-25 14:17:35 +0200

  * btest/plugins: Add test for custom ConnKey factory (Arne Welzel, Corelight)

    This just counts DoInits() and adds that information to the conn_id
    record, but without including it into the hash. Mostly for smoke
    testing.

  * NEWS updates for pluggable connection tuples. (Christian Kreibich, Corelight)

  * Add a VLAN-aware flow tuple implementation. (Christian Kreibich, Corelight)

    This is a first "real" implementation of a custom tuple, adding additional
    fields over the standard five-tuple.

    Includes test cases.

  * Deprecate ConnTuple and related APIs. (Christian Kreibich, Corelight)

    Given IP-aware ConnKeys, ConnTuples aren't really required any more. ConnTuple
    had two benefits:

    - It preserved the original src/dst orientation from the packet headers it was
    based on, which IPBasedConnKey now tracks and provides accessor methods for.

    - In IPBasedAnalyzer::AnalyzePacket() its instance survived past the std:move()
    of the key into NewConn(), which we sidestep by keeping the original src address
    and port around until we need after the connection is obtained.

  * Deprecate the old Connection constructor and detail::ConnKey class. (Christian Kreibich, Corelight)

    The new key-based Connection constructor replaces the former, and the new
    ConnKey class tree replaces the latter.

  * Switch to virtualized use of new zeek::ConnKey class tree (Christian Kreibich, Corelight)

    This touches quite a few places, but each just swaps out existing
    APIs and/or zeek::detail::ConnKey instances.

  * Provide a connkey factory for Zeek's default five-tuples. (Christian Kreibich, Corelight)

    Since the base factory is pure virtual this is now the first full
    implementation, but still a bit of a special case because it implements Zeek's
    default behavior and doesn't add "custom" content to the tuple.

  * Add IP-specific ConnKey implementation. (Christian Kreibich, Corelight)

    The InitTuple() implementation here is a placeholder for a fuller one following
    later, when we do away with the need for ConnTuple.

  * Establish plugin infrastructure for ConnKey factories. (Christian Kreibich, Corelight)

    ConnKey factories are intermediaries that encapsulate the details of how to
    instantiate ConnKeys, which codify the hash input for connection lookups.

  * Add new ConnKey abstraction. (Christian Kreibich, Corelight)

8.0.0-dev.517 | 2025-06-25 09:33:46 +0200

  * telemetry: Rename endpoint label to node label (Arne Welzel, Corelight)

    Using a label named "endpoint" is not intuitive and requires explaining to
    users that it's really just the Cluster::node value. Change the label to
    "node", so that we don't need to do the explaining.

    This probably breaks some existing users of the Prometheus metrics, but after
    looking more at metrics recently, "endpoint" really is a thorn in my eye.

8.0.0-dev.514 | 2025-06-24 15:38:18 -0700

  * Add get_tags_by_category BIF method (bhaskarbhar)

  * Add NEWS entry about deprecation of --with-binpac and --with-bifcl (Tim Wojtulewicz, Corelight)

8.0.0-dev.511 | 2025-06-24 12:51:08 -0700

  * Deprecate --with-binpac/--with-bifcl configure options (Tim Wojtulewicz, Corelight)

  * Remove deprecated --disable-archiver configure argument (Tim Wojtulewicz, Corelight)

8.0.0-dev.508 | 2025-06-24 19:07:14 +0200

  * btest/broker/publish-errors: Avoid exit(0) (Arne Welzel, Corelight)

    Calling exit() doesn't properly shutdown the manager instances and TSAN reports a
    leaked thread. Just avoid this for now by using terminate() instead.

8.0.0-dev.506 | 2025-06-24 17:44:06 +0200

  * DNS: Implement NAPTR RR support (Arne Welzel, Corelight)

    My phone is sending NAPTR queries and we reported an unknown RR type 35
    in weird.log for the response, so figured I'd just add it.

  * DNS: Move extract_char_string() helper around (Arne Welzel, Corelight)

8.0.0-dev.503 | 2025-06-24 17:14:28 +0200

  * cluster/zeromq: Short-circuit DoPublishLogWrite() when not initialized (Arne Welzel, Corelight)

    After moving the log_push initialization from the constructor to the
    DoInit() method, it's now possible that DoPublishLogWrites() is invoked
    even if DoInit() was never called. Handle this by short-circuiting. This
    is sort of an error, but can happen during tests if scripts are loaded
    somewhat arbitrarily.

  * cluster/zeromq: Hook up and enable IPV6 by default (Arne Welzel, Corelight)

    ZeroMQ's IPv6 support isn't enabled by default, resulting in
    "No such device" errors when attempting to listen on an IPv6
    address. This change adds a ipv6 option to the ZeroMQ module
    and enables it by default. Further, adds a test configuring
    everything to listen on IPv6 ::1 as well, and one test to provoke
    the original error. This also regularizes some error messages.

    The addr_to_uri() calls weren't actually needed, but they apparently do
    not hurt and the result is easier on the eyes, so use them :-)

  * cluster/zeromq/connect: Make failures fatal (Arne Welzel, Corelight)

    The cluster is borked if the initialization fails, so may as well just
    completely abort Zeek at that point with a fatal error. There's no real
    point in continuing to run.

  * cluster/zeromq: Move log_push creation to DoInit() (Arne Welzel, Corelight)

    The log_push socket should be affected by the IPV6 option, so need to
    delay its creation a bit.

8.0.0-dev.498 | 2025-06-24 16:30:52 +0200

  * GH-4587: Build builtin Spicy analyzers in debug mode if debug mode is enabled (Benjamin Bannier, Corelight)

8.0.0-dev.496 | 2025-06-24 14:00:39 +0100

  * Default canonifier change to only remove first timestamp in line (Johanna Amann, Corelight)

    In the past, we used a default canonifier, which removes everything that
    looks like a timestamp from log files. The goal of this is to prevent
    logs from changing, e.g., due to local system times ending up in log
    files.

    This update changes the default canonifier, so that only the first
    timestamp in a line is removed. This should skip timestamps that are
    likely to change while keeping timestamps that are parsed
    from protocol information.

  * Align SMB timestamp calculation between operating systems (Johanna Amann, Corelight)

    The way that SMB timestamps were calculated used a "long double" for
    the conversion calculation.

    OS-X and Linux seem to have different sizes for long doubles. On the
    Linux distributions that I have available, a long double is 128bits, vs
    64 bits in OS-X.

    This leads to slight discrepancies in the timestamps. This commit
    changes this calculation to use a double on both systems, which is 64
    bits and eliminates this difference.

8.0.0-dev.493 | 2025-06-23 10:52:22 -0700

  * Switch IPv6_Hdr_Chain to a vector of objects instead of pointers (Tim Wojtulewicz, Corelight)

8.0.0-dev.491 | 2025-06-23 10:39:31 -0700

  * Remove unneeded include dirs in zeromq CMakeLists.txt (Tim Wojtulewicz, Corelight)

8.0.0-dev.489 | 2025-06-23 10:36:38 -0700

  * Deprecate BRO_PLUGIN_INSTALL_PATH constant (Tim Wojtulewicz, Corelight)

  * Make constants in IP::ParseResult uppercase, deprecate the old ones (Tim Wojtulewicz, Corelight)

  * Fix comparison against CapLen results in IPTunnel (Tim Wojtulewicz, Corelight)

  * Fix clang-tidy warnings in headers (Tim Wojtulewicz, Corelight)

    This entry is a large collection of commits.

  * Change clang-tidy configuration to also scan headers (Tim Wojtulewicz, Corelight)

  * Don't pass local source/binary directory to zeek_add_plugin for Redis backend (Tim Wojtulewicz, Corelight)

  * Update bifcl/binpac/gen-zam submodules with clang-tidy fixes (Tim Wojtulewicz, Corelight)

  * Avoid linting headers in external paths (Tim Wojtulewicz, Corelight)

8.0.0-dev.447 | 2025-06-22 15:49:11 +0200

  * GH-4594: Align WebSocket error in cluster with one in Broker (Benjamin Bannier, Corelight)

    The schema of cluster WebSocket error messages deviated from the
    existing one in Broker which breaks seamless migration from the Broker
    WebSocket bindings.

    This patch aligns the serialization in cluster with the one in Broker.
    This is technically a breaking change of the cluster schema, but since
    it never worked like documented and is still experimental this is
    probably fine.

8.0.0-dev.445 | 2025-06-22 15:48:21 +0200

  * GH-4571: broker: Handle Broker::publish() with non Broker::Event (Arne Welzel, Corelight)

8.0.0-dev.443 | 2025-06-18 13:27:06 +0100

  * Change x509 not_before/not_after to not be based on local timezone (Johanna Amann, Corelight)

    Not the not_before/not_after fields output GMT based times.

    Also adds a new btest diff canonifier which only removes the first
    timestamp in a line.

  * Mark ZeekString vector helper methods deprecated (Tim Wojtulewicz, Corelight)

8.0.0-dev.439 | 2025-06-18 13:12:59 +0200

  * Enable Spicy SSL analyzer in nightly CI job against Spicy `HEAD` (Benjamin Bannier, Corelight)

  * Clean up use of references in Spicy SSL analyzer (Benjamin Bannier, Corelight)

    The Spicy SSL analyzer was using references pretty heavily, probably to
    work around now fixed issues with `inout` parameters in older Spicy
    versions. At least for units this seems not needed anymore, and was also
    partially incorrect, e.g., the Spicy docs call out that when using
    `inout` parameters, passed and expected types should match exactly so
    passing a reference as an `inout` value seems incorrect. Additionally,
    one use case for references in Spicy is to use their interior
    mutability, i.e., a reference never needs to be passed `inout` since
    their can always be mutated.

    Internally units are stored as reference-counted values, and references
    to units are not much cheaper since they also need to be refcounted.
    With that there seems litle reason to use references in this analyzer at
    all, and this patch drops their use completely; instead we either pass
    values, or values declared `inout`.

    We leave the use of references for sharing sinks in place.

  * GH-2091: Add workaround for zeek/spicy#2091 (Benjamin Bannier, Corelight)

8.0.0-dev.435 | 2025-06-17 09:25:30 +0200

  * GH-4573: IOSource: Remove IsPacketSource (Arne Welzel, Corelight)

    This wasn't used in tree and even the PktSrc class doesn't override this
    to return true, so just remove it outright without deprecation.

8.0.0-dev.433 | 2025-06-16 14:58:12 +0200

  * btest/logging: Fly-by cleanup (Arne Welzel, Corelight)

  * GH-4562: logging/Ascii: Fix abort() for non-existing postrotation functions (Arne Welzel, Corelight)

    When looking up the postprocessor function from shadow files, id::find_func()
    would abort() if the function wasn't available instead of falling back
    to the default postprocessor.

    Fix by using id::find() and checking the type explicitly and also adding a
    strict type check while at it.

    This issue was tickled by loading the json-streaming-logs package,
    Zeek creating shadow files containing its custom postprocessor function,
    then restarting Zeek without the package loaded.

  * Bump `auxil/spicy` to latest development snapshot (Benjamin Bannier, Corelight)

8.0.0-dev.428 | 2025-06-16 13:47:54 +0200

  * btest/cluster/websocket: Add cert-less test (Arne Welzel, Corelight)

8.0.0-dev.425 | 2025-06-11 13:18:36 -0700

  * Add missing #include to packet_analysis/Component.h (Tim Wojtulewicz, Corelight)

8.0.0-dev.423 | 2025-06-11 11:12:18 -0700

  * Remove some unused #includes from spicy code (Tim Wojtulewicz, Corelight)

  * Remove using util.h in various headers in favor of util-types.h (Tim Wojtulewicz, Corelight)

  * Move type definitions/aliases from util.h to a separate file (Tim Wojtulewicz, Corelight)

8.0.0-dev.419 | 2025-06-11 09:00:46 -0700

  * Bugfix: AsString() on an EnumVal will segfault (Christian Kreibich, Corelight)

8.0.0-dev.417 | 2025-06-11 17:24:11 +0200

  * GH-4522: smtp: Fix last_reply column in smtp.log for BDAT LAST (Arne Welzel, Corelight)

    The response to BDAT LAST was never recognized, resulting in the
    BDAT LAST commands not being logged in a timely fashion and receiving
    the wrong status.

    This likely doesn't handle complex pipeline scenarios, but it fixes
    the wrong behavior for smtp_reply() not handling simple BDAT commands
    responses.

    Thanks @cccs-jsjm for the report!

8.0.0-dev.414 | 2025-06-10 12:08:36 -0700

  * Move netbios_ssn_session_timeout to a script-level constant (Tim Wojtulewicz, Corelight)

8.0.0-dev.412 | 2025-06-10 18:50:39 +0200

  * btest/opt/ZAM-bif-tracking: Disable by default (Arne Welzel, Corelight)

8.0.0-dev.410 | 2025-06-09 18:00:33 -0700

  * Btests: don't use -C in Zeek invocations that don't actually need it (Christian Kreibich, Corelight)

  * Remove executable file permission bits from a bunch of our pcaps (Christian Kreibich, Corelight)

8.0.0-dev.406 | 2025-06-06 11:45:33 -0700

  * Move initialization of RandTest members to header (Tim Wojtulewicz, Corelight)

  * Update .clang-tidy to have modernize-* enabled with some exclusions (Tim Wojtulewicz, Corelight)

  * Fix clang-tidy modernize findings (Tim Wojtulewicz, Corelight)

  * Update bifcl submodule with clang-tidy fixes [nomail] (Tim Wojtulewicz, Corelight)

8.0.0-dev.389 | 2025-06-06 10:43:07 -0700

  * Make Zeekygen docs generation (-X) imply parse-only (-a) (Christian Kreibich, Corelight)

8.0.0-dev.382 | 2025-06-06 17:33:39 +0200

  * Fix `&ordered` attribute not preserved in table initializer assignments (赵富鹏)

8.0.0-dev.380 | 2025-06-06 10:26:17 +0200

  * docker: Add `net-tools` and `procps` dependencies (Edoardo Mich)

    Add `net-tools` and `procps` to the final.Dockerfile, to avoid warning in
    zeekctl commands like: "failed to find local IP addresses [...]" and the
    error in `zeekctl top` command caused respectively by missing `ifconfig`
    and `top` commands.

8.0.0-dev.378 | 2025-06-05 20:31:00 -0700

  * Follow-ups to configure output reformatting (Tim Wojtulewicz, Corelight)

8.0.0-dev.375 | 2025-06-05 12:43:26 -0700

  * Add busy_timeout script-level option, override any busy_timeout pragma (Tim Wojtulewicz, Corelight)

  * Handle potential contention when running sqlite expiration (Tim Wojtulewicz, Corelight)

  * Add expiration to sqlite-cluster.btest (Tim Wojtulewicz, Corelight)

  * Use unique_ptr to avoid needing to call sqlite3_reset manually (Tim Wojtulewicz, Corelight)

  * Move Deferred class from ZeroMQ to util (Tim Wojtulewicz, Corelight)

  * Temporarily disable pppoe-over-qinq-test for spicy-ssl (Johanna Amann, Corelight)

    The analyzer.log changes exposed a new bug in the Spicy SSL
    implemenataion.

    Relates to GH-4547

8.0.0-dev.367 | 2025-06-05 08:22:25 -0700

  * Silence -Wnontrivial-memcall warning in ConnKey methods (Tim Wojtulewicz, Corelight)

8.0.0-dev.363 | 2025-06-05 07:15:59 +0100

  * dpd.log to analyzer.log changes:

    Logging of failed analyzers has been overhauled. `dpd.log` was replaced by
    a new `analyzer.log` that presents a more unified and consistent view of
    failed analyzers. The previous `analyzer.log` was renamed to
    `analyzer-debug.log`.

    For protocol analyzers, `analyzer.log` now reports initially confirmed
    analyzers that Zeek subsequently removed from the connection due to a
    protocol violation.

    For file and packet analyzers, all errors will be logged to `analyzer.log`.

    As part of this work, a new `analyzer_failed` event has been introduced.
    This event is raised when an analyzer is removed because of raising a
    violation.

    `analyzer.log` was renamed to `analyzer_debug.log`, and is no longer
    created by default. The log file will be created if the
    `frameworks/analyzer/debug-logging.zeek` policy script is loaded.

    The namespace for options in the script changed to
    `Analyzer::DebugLogging`. Furthermore the default options changed to enable
    more detailed output by default.

    The `service_violation` field of the connection record was marked as
    deprecated.  A new `failed_analyzers` field was inroduced instead, with
    slightly different semantics.  Where c$failed_analyzers tracks analyzers
    that were removed due to a violation, c$service_violation tracks violations
    - and doesn't care if an analyzer was actually removed due to it.
    (Johanna Amann, Corelight)

  * Move detect-protocol from frameworks/dpd to frameworks/analyzer

    detect-protocol.zeek was the last non-deprecated script left in
    policy/frameworks/dpd. It was moved to policy/frameworks/analyzer. A
    script that loads the script from the new location with a deprecation
    warning was added. (Johanna Amann, Corelight)

8.0.0-dev.353 | 2025-06-04 09:38:20 -0700

  * Add some notes about missing/disabled cppcoreguildlines clang-tidy checkers (Tim Wojtulewicz, Corelight)

  * Fix clang-tidy cppcoreguidelines-macro-usage findings (macro functions) (Tim Wojtulewicz, Corelight)

  * Fix clang-tidy cppcoreguidelines-macro-usage findings (macros as constants) (Tim Wojtulewicz, Corelight)

  * script_opt: Add missing virtual destructor (cppcoreguidelines-virtual-class-destructor) (Tim Wojtulewicz, Corelight)

  * Parallelize coverage/bare-mode-errors (Johanna Amann, Corelight)

    Currently, coverage/bare-mode-errors is one of the slowest tests in the
    entire test suite. This is caused by the fact that it has to repeatedly
    launch Zeek for every script that we ship. This is done sequentially.

    This commit changes this test to use xargs to spawn 20 parallell
    processes.

8.0.0-dev.346 | 2025-06-04 08:39:54 -0400

  * Fix Spicy re-enable builtin analyzer debug message (Evan Typanski, Corelight)

8.0.0-dev.341 | 2025-06-03 11:54:04 -0700

  * Redis: Add support for sending AUTH commands during connection (Tim Wojtulewicz, Corelight)

  * Redis: disconnect cleanly if INFO request fails (Tim Wojtulewicz, Corelight)

  * Fix segfault if storage sync open_backend returns bad code (Tim Wojtulewicz, Corelight)

  * Add ToStdString and ToStdStringView to ZeekString (Tim Wojtulewicz, Corelight)

8.0.0-dev.335 | 2025-06-02 14:52:44 -0700

  * Update libunistd submodule [nomail] (Tim Wojtulewicz, Corelight)

  * Update Windows CI image to Visual Studio 2022 (Tim Wojtulewicz, Corelight)

8.0.0-dev.332 | 2025-06-02 14:31:29 -0700

  * CI: Force rebuild of opensuse tumbleweed docker image (Tim Wojtulewicz, Corelight)

    Something is off with the libhiredis package that's installed
    as part of the current image, and 'zypper patch' is failing to
    update it during the prepare step. Force a rebuild so the
    updated package is part of the image to start with.

8.0.0-dev.330 | 2025-06-02 14:30:41 -0700

  * Bump Spicy to pull in fix. (Robin Sommer, Corelight)

    Includes a new regression test.

    Closes #4501.

8.0.0-dev.328 | 2025-06-02 11:45:03 -0700

  * Bump zeekjs to v0.17.1 (Arne Welzel, Corelight)

        b340648 version: 0.17.1
        ccd32e1 ci/docker: Use Fedora 42, retire Fedora 40
        6893431 Nodejs: Use argv.data() over &argv[0]

8.0.0-dev.326 | 2025-06-02 11:43:56 -0700

  * Fix incorrectly copied comment [skip CI] (Benjamin Bannier, Corelight)

8.0.0-dev.324 | 2025-06-02 17:32:39 +0200

  * Event: Bail on add_missing_remote_network_timestamp without add_network_timestamp (Arne Welzel, Corelight)

  * btest/plugin: Test custom metadata publish (Arne Welzel, Corelight)

    Usage demo for plugin writers to add custom event metadata and access in
    in Zeek scripts.

  * NEWS: Add note about generic event metadata (Arne Welzel, Corelight)

  * cluster: Remove deprecated Event constructor (Arne Welzel, Corelight)

    It is now unused, ditch it. This wasn't available in an LTS release yet
    and anyhow is in the detail namespace.

  * cluster: Remove some explicit timestamp handling (Arne Welzel, Corelight)

    Backend::MakeClusterEvent() for now is the only place to add implicit
    network timestamp metadata within the cluster component.

  * broker/Manager: Fetch and forward all metadata from events (Arne Welzel, Corelight)

    Also use the generic metadata version for publishing, keep the
    ts-based API for now, but only add timestamps when
    EventMetadata::add_network_timestamp is T. I'm not sure what the
    right way forward here is, maybe deprecating Broker's publish event
    variations and funneling through cluster.

  * Event/init-bare: Add add_missing_remote_network_timestamp logic (Arne Welzel, Corelight)

    Make defaulting to the local network timestamp for remote events opt-in.

  * cluster/Backend/DoProcessEvent: Use generic metadata, not just timestamps (Arne Welzel, Corelight)

  * cluster/Event: Support moving args and metadata from event (Arne Welzel, Corelight)

  * cluster/serializer/broker: Support generic metadata (Arne Welzel, Corelight)

    Instead of handling just the network timestamp, support extraction of
    the whole metadata vector that broker events hold.

  * cluster/Event: Generic metadata support (Arne Welzel, Corelight)

    Instead of a timestamp attribute, switch to holding a EventMetadataVectorPtr
    like zeek::Event instances do. Keep the old constructor until the end of
    the patch series.

  * Event: Use -1.0 for undefined/unset timestamps (Arne Welzel, Corelight)

    This can happen if either there's no network timestamp associated with
    an event, or there's currently no event being dispatched. Using 0.0
    isn't great as it's the normal start timestamp before reading a network
    packet. Using -1.0 gives the caller a chance to check and realize what's
    going on.

  * cluster: Use shorter obj_desc versions (Arne Welzel, Corelight)

  * Desc: Add obj_desc() / obj_desc_short() overloads for IntrusivePtr (Arne Welzel, Corelight)

    When using these helpers in code, one barely has raw pointers and the
    repeated `.get()` call cumbersome and noisy.

8.0.0-dev.309 | 2025-06-02 10:17:14 +0200

  * Prefer `std::move` over copy (Benjamin Bannier, Corelight)

8.0.0-dev.307 | 2025-06-02 10:12:14 +0200

  * minor BTest maintenance updates for -O gen-C++ (Vern Paxson, Corelight)

  * fix for more robustly finding BTests to assess for -O gen-C++ (Vern Paxson, Corelight)

  * fix for -O gen-C++ dealing with type constants of unnamed compound types (Vern Paxson, Corelight)

8.0.0-dev.303 | 2025-06-02 09:51:53 +0200

  * intel/seen/manage-event-groups: Policy script for toggling intel event groups (Arne Welzel, Corelight)

    Co-authored-by: Mohan Dhawan <mohan@corelight.com>

  * intel: Add indicator_inserted and indicator_removed hooks (Arne Welzel, Corelight)

    This change adds two new hooks to the Intel framework that can be used
    to intercept added and removed indicators and their type.

    These hooks are fairly low-level. One immediate use-case is to count the
    number of indicators loaded per Intel::Type and enable and disable the
    corresponding event groups of the intel/seen scripts.

    I attempted to gauge the overhead and while it's definitely there, loading
    a file with ~500k DOMAIN entries takes somewhere around ~0.5 seconds hooks
    when populated via the min_data_store store mechanism. While that
    doesn't sound great, it actually takes the manager on my system 2.5
    seconds to serialize and Cluster::publish() the min_data_store alone
    and its doing that serially for every active worker. Mostly to say that
    the bigger overhead in that area on the manager doing redundant work
    per worker.

    Co-authored-by: Mohan Dhawan <mohan@corelight.com>

8.0.0-dev.299 | 2025-05-30 13:06:42 -0700

  * fix for crash when interpreting transformed ASTs that include multi-field record assignments/additions (Vern Paxson, Corelight)

  * Remove unused ZAM compiler method (Vern Paxson, Corelight)

8.0.0-dev.296 | 2025-05-30 12:16:27 -0700

  * Add utility methods to make CMake summary output nicer (Tim Wojtulewicz, Corelight)

8.0.0-dev.294 | 2025-05-30 10:29:29 -0700

  * CI: Add PR label for skipping all CI jobs (Tim Wojtulewicz, Corelight)

  * CI: Add PR label for running cluster tests (Tim Wojtulewicz, Corelight)

8.0.0-dev.291 | 2025-05-30 08:39:28 -0700

  * CI: Add new task to run clang-tidy as part of nightly builds (Tim Wojtulewicz, Corelight)

  * CI: Update to clang 19 on ubuntu 24.04, add clang-tidy package (Tim Wojtulewicz, Corelight)

8.0.0-dev.288 | 2025-05-30 17:24:05 +0200

  * GH-4494: logging/ascii/json: Make TS_MILLIS signed, add TS_MILLIS_UNSIGNED (Arne Welzel, Corelight)

    It seems TS_MILLIS is specifically for Elasticsearch and starting with
    Elasticsearch 8.2 epoch_millis does (again?) support negative epoch_millis,
    so make Zeek produce that by default.

    If this breaks a given deployment, they can switch Zeek back to TS_MILLIS_UNSIGNED.

    https://discuss.elastic.co/t/migration-from-es-6-8-to-7-17-issues-with-negative-date-epoch-timestamp/335259
    https://github.com/elastic/elasticsearch/pull/80208

    Thanks for @timo-mue for reporting!

8.0.0-dev.286 | 2025-05-30 08:12:43 -0700

  * Add move operations for LogWriteHeader (Tim Wojtulewicz, Corelight)

  * Add missing setting of type in session::Key move operations (Tim Wojtulewicz, Corelight)

  * Update .clang-tidy to have performance-* enabled with some exclusions (Tim Wojtulewicz, Corelight)

  * Fix clang-tidy performance-* warnings (Tim Wojtulewicz, Corelight)

8.0.0-dev.271 | 2025-05-30 16:48:43 +0200

  * Update doc submodule [nomail] [skip ci] (Arne Welzel, Corelight)

8.0.0-dev.270 | 2025-05-30 16:44:23 +0200

  * Bump doc submodule for ext/zeek.py modifications (Arne Welzel, Corelight)

  * RecordType:DescribeReST: Render RecordType using zeek:field directive (Arne Welzel, Corelight)

8.0.0-dev.267 | 2025-05-30 11:43:57 +0200

  * IXWebsocket: Bump to version with memset() sock addr fix (Arne Welzel, Corelight)

  * cluster/websocket: Deprecate $listen_host, introduce $listen_addr (Arne Welzel, Corelight)

  * GH-4474: cluster/websocket-ixwebsocket: Determine proper address_family (Arne Welzel, Corelight)

8.0.0-dev.262 | 2025-05-28 08:29:33 -0700

  * Remove obsolete --with-bind configure flag (Tim Wojtulewicz, Corelight)

8.0.0-dev.259 | 2025-05-27 12:49:05 -0700

  * Add a few std::moves to fix Coverity warnings (Tim Wojtulewicz, Corelight)

  * Fix possible memory leak in broker::Manager (Tim Wojtulewicz, Corelight)

  * fixup! Use SCN* constants when reading into sized integer types (Tim Wojtulewicz, Corelight)

  * from_json: Statically lookup err index in result type (Tim Wojtulewicz, Corelight)

  * Use SCN* constants when reading into sized integer types (Tim Wojtulewicz, Corelight)

  * Remove setting unused variable value in FTP::parse_eftp (Tim Wojtulewicz, Corelight)

8.0.0-dev.252 | 2025-05-27 11:59:00 -0700

  * Update .clang-tidy to have bugprone-* enabled with some exclusions (Tim Wojtulewicz, Corelight)

  * Fix clang-tidy bugprone-* warnings (Tim Wojtulewicz, Corelight)

8.0.0-dev.229 | 2025-05-27 17:46:35 +0100

  * Add explicit TLS support for FTP (Johanna Amann, Corelight)

    This is defined in RFC 4217; TLS is initialized by the client sending an
    `AUTH TLS` command.

8.0.0-dev.227 | 2025-05-27 10:18:49 -0400

  * spicy-redis: Separate error replies from success (Evan Typanski, Corelight)

  * spicy-redis: Cleanup scripts and tests (Evan Typanski, Corelight)

    - Recomputes checksums for pcaps to keep clean
    - Removes some tests that had big pcaps or weren't necessary
    - Cleans up scripting names and minor points
    - Comments out Spicy code that causes a build failure now with a TODO to
      uncomment it

  * spciy-redis: Bring Redis analyzer into Zeek proper (Evan Typanski, Corelight)

  * spicy-redis: Abort parsing if server data comes first (Evan Typanski, Corelight)

    Redis seems to only want client data first to request server data. The
    DPD signature seems to pick up on some cases where server data comes
    first, but is otherwise "valid" RESP. See if this helps lower FP rates.

  * spicy-redis: Add recursion depth to server data (Evan Typanski, Corelight)

  * spicy-redis: Make client data only accept bulk strings (Evan Typanski, Corelight)

  * spicy-redis: Add dpd signature and clean pcaps (Evan Typanski, Corelight)

  * spicy-redis: Add some commands and touch up parsing (Evan Typanski, Corelight)

  * spicy-redis: Add some script logic for logging (Evan Typanski, Corelight)

  * spicy-redis: Separate client/server (Evan Typanski, Corelight)

    This makes the parser more official and splits the client/server out
    from each other.

  * spicy-redis: Touchup logging and Spicy issues (Evan Typanski, Corelight)

  * spicy-redis: Add synchronization and pipeline support (Evan Typanski, Corelight)

  * spicy-redis: Begin Spicy Redis analyzer (Evan Typanski, Corelight)

8.0.0-dev.209 | 2025-05-26 16:08:44 +0200

  * btest: Add test for Cluster::hello zero-timestamp (Arne Welzel, Corelight)

  * EventMgr/Enqueue: Add automatic timestamp metadata to local events, only (Arne Welzel, Corelight)

    It seems less surprising if only local events receive automatic network
    timestamp metadata. For remote events the automatic value will most
    likely be misleading.

  * cluster and broker: Propagate zero-timestamp as metadata, too. (Arne Welzel, Corelight)

    This will be cleaned up later to just pass all contained metadata from
    a cluster event to the queued event, but for now do this here, otherwise
    we break some internal tests.

8.0.0-dev.204 | 2025-05-23 12:13:41 -0700

  * Redis: bump version of hiredis required (Tim Wojtulewicz, Corelight)

  * Redis: return proper error if connection fails (Tim Wojtulewicz, Corelight)

8.0.0-dev.201 | 2025-05-23 20:57:54 +0200

  * Add generic event metadata, part 1 (Arne Welzel, Corelight)

    The ``ts`` member of ``Event`` was removed in favor of a more generic
    approach that can hold any event metadata. Further, the Zeek script-layer
    API for registering event metadata types was added.

    Event instances do not network timestamp metadata by default anymore.
    This feature can be re-enabled via EventMetadata::add_network_timestamp=T.

8.0.0-dev.191 | 2025-05-23 08:32:00 -0700

  * Remove FindClangTidy.cmake, update cmake submodule (Tim Wojtulewicz, Corelight)

8.0.0-dev.188 | 2025-05-22 16:54:58 -0700

  * SQLite: Add TODO note about possibly using sqlite3_busy_timeout (Tim Wojtulewicz, Corelight)

  * SQLite: Fix typo in variable name causing pragmas not to retry on busy (Tim Wojtulewicz, Corelight)

  * SQLite: Use tableval iteration instead of ToMap for pragmas (Tim Wojtulewicz, Corelight)

  * SQLite: Fix logging/error messages around executing pragmas (Tim Wojtulewicz, Corelight)

  * Disable sqlite-cluster btest (Tim Wojtulewicz, Corelight)

    This test is being flaky on some platforms and still having problems
    with executing pragmas at startup. Disable it for now until it can be
    fixed.

8.0.0-dev.181 | 2025-05-21 12:47:01 -0700

  * Update baseline of sqlite-basic btest after recent changes (Tim Wojtulewicz, Corelight)

8.0.0-dev.179 | 2025-05-21 09:40:31 -0700

  * SQLite: Move integrity_check to pragma table (Tim Wojtulewicz, Corelight)

  * SQLite: Add backend option for pragma timeout (Tim Wojtulewicz, Corelight)

  * SQLite: Rename tuning_params to pragma_commands, move running pragmas to utility method (Tim Wojtulewicz, Corelight)

  * SQLite: Retry pragma statements at startup to avoid contention (Tim Wojtulewicz, Corelight)

  * SQLite: Check for locked database as well as busy databases (Tim Wojtulewicz, Corelight)

  * SQLite: Fix some string-sizing issues (Tim Wojtulewicz, Corelight)

  * SQLite: Run pragmas on connection before creating table (Tim Wojtulewicz, Corelight)

  * SQLite: Add busy_timeout pragma to default options (Tim Wojtulewicz, Corelight)

  * Prefix sqlite-based btests with sqlite- to match redis tests (Tim Wojtulewicz, Corelight)

  * Add sqlite cluster storage btest (Tim Wojtulewicz, Corelight)

8.0.0-dev.168 | 2025-05-21 13:38:46 +0200

  * http/detect-sql-injection: Fix zeekygen comment (Arne Welzel, Corelight)

        Discarded extraneous Zeekygen comment: $src field; and always provides a victim IP address in the $dst field.

  * btest/coverage/test-all-policy-zeekygen: Load test-all-policy with zeekygen enabled (Arne Welzel, Corelight)

    There should not be warnings produced. The default ZEEK_DISABLE_ZEEKYGEN=1
    setting in the btest configuration hid some issues previously.

8.0.0-dev.164 | 2025-05-20 12:02:09 -0700

  * Add extra input files to ftp fuzzer corpus (Tim Wojtulewicz, Corelight)

  * Use bool instead of int flag in FTP analyzer's parse_eftp method (Tim Wojtulewicz, Corelight)

  * Fix undefined behavior in FTP analyzer's parse_port method (Tim Wojtulewicz, Corelight)

  * Fix invalid-read in FTP analyzer's parse_port method (Tim Wojtulewicz, Corelight)

8.0.0-dev.159 | 2025-05-20 20:30:18 +0200

  * Update cluster-layout.zeek usage in btests (Arne Welzel, Corelight)

  * testing/btest/Files: Add generic broker/cluster-layout.zeek (Arne Welzel, Corelight)

    This is a cluster-layout.zeek template that can be copied into a testing
    directory if needed. The idea is that a developer sets environment
    variables within their btest and the Cluster::nodes variable is
    implicitly extended by appropriate nodes.

    For example, using @TEST-PORT BROKER_LOGGER1_PORT will add an appropriate
    logger-1 node to Cluster::nodes, based on the existence of the
    BROKER_LOGGER1_PORT environment variable.

8.0.0-dev.146 | 2025-05-20 16:25:58 +0200

  * site/local: Switch to detect-sql-injection (Arne Welzel, Corelight)

  * Add a revised script for detecting HTTP SQL injection, deprecate original (Vern Paxson, Corelight)

8.0.0-dev.142 | 2025-05-19 12:32:56 -0700

  * EventTrace: Fix operator==() UBSAN downcast error (Arne Welzel, Corelight)

    New test triggered the following error:

        runtime error: downcast of address 0x57021a323ea0 which does not point to an object of type 'const FileVal' 0x57021a323ea0: note: object is of type 'zeek::FuncVal'

8.0.0-dev.140 | 2025-05-19 11:04:29 -0700

  * Change from_json to return an error rather than print it. (Steve Smoot, Corelight)

8.0.0-dev.136 | 2025-05-19 20:01:30 +0200

  * btest/core: Add event-trace test (Arne Welzel, Corelight)

  * zeek-setup: Free event_trace_mgr after generating trace (Arne Welzel, Corelight)

  * EventTraceMgr: Rename etm to event_trace_mgr (Arne Welzel, Corelight)

  * EventTraceMgr: Move fclose() to destructor (Arne Welzel, Corelight)

  * Update cmake submodule (Tim Wojtulewicz, Corelight)

8.0.0-dev.130 | 2025-05-19 10:25:34 -0700

  * Remove unnecessary #includes across the repo (Tim Wojtulewicz, Corelight)

8.0.0-dev.121 | 2025-05-19 09:43:59 -0700

  * Rename CI-related github labels for PRs (Tim Wojtulewicz, Corelight)

8.0.0-dev.118 | 2025-05-16 10:27:27 -0700

  * Remove including <cinttypes> from util.h (Tim Wojtulewicz, Corelight)

  * Remove telemetry #includes from OpaqueVal.h (Tim Wojtulewicz, Corelight)

  * Reduce includes in plugin/Component.h (Tim Wojtulewicz, Corelight)

  * Remove zeek/Stats.h include from NetVar.h (Tim Wojtulewicz, Corelight)

  * Include StmtBase/StmtEnums in Func.h instead of Stmt.h (Tim Wojtulewicz, Corelight)

    This requires changes in lots of other files that were depending on Func.h
    to provide that include for them.

  * Use modern names for standard headers (Tim Wojtulewicz, Corelight)

  * Remove fix for CentOS 7 from TCP_Flags.h (Tim Wojtulewicz, Corelight)

  * Fix usage of std::string in http analyzer (Tim Wojtulewicz, Corelight)

  * Reorder top section of net_util.h to batch includes together (Tim Wojtulewicz, Corelight)

  * Use quotes instead of <> for zeek includes (Tim Wojtulewicz, Corelight)

  * Fix Obj.h include in IntrusivePtr.h to have full path (Tim Wojtulewicz, Corelight)

8.0.0-dev.106 | 2025-05-16 13:39:56 +0200

  * Generate --event-trace output explicitly rather than in EventTraceMgr destructor (Vern Paxson, Corelight)

8.0.0-dev.104 | 2025-05-15 12:15:42 -0700

  * Require non-system version of libkrb5 on macOS (Tim Wojtulewicz, Corelight)

8.0.0-dev.102 | 2025-05-15 16:44:16 +0200

  * IXWebSocket: Point at upstream, bump to v11.4.6 (Arne Welzel, Corelight)

8.0.0-dev.100 | 2025-05-15 10:21:53 +0100

  * Add STLS command to POP3 DPD signature.

    Pop3 connections that use StartTLS now should be detected on non-default ports.
	  (Jan Grashoefer, Corelight)

8.0.0-dev.96 | 2025-05-13 19:50:47 +0200

  * fix for ZAM optimization when an aggregate is modified inside of a loop (Vern Paxson, Corelight)

8.0.0-dev.94 | 2025-05-13 10:03:46 -0700

  * Allow macOS to search for libkrb5, but disallow system version of library (Tim Wojtulewicz, Corelight)

8.0.0-dev.92 | 2025-05-13 18:26:22 +0200

  * btest/cluster/websocket: Update tests for new event signature (Arne Welzel, Corelight)

  * GH-4440: cluster/websocket: Propagate code and reason to websocket_client_lost() (Arne Welzel, Corelight)

    This allows to get visibility into the reason why ixwebsocket or the
    client decided to disconnect.

  * cluster/websocket: Support configurable ping interval (Arne Welzel, Corelight)

    Primarily for testing purposes and maybe the hard-coded 5 seconds is too
    aggressive for some deployments, so makes sense for it to be
    configurable.

  * GH-4440: IXWebSocket: Bump to improve O^2 rxbuf erase() (Arne Welzel, Corelight)

8.0.0-dev.87 | 2025-05-12 11:33:38 -0700

  * Fix clang-tidy `bugprone-inc-dec-in-conditions` report in Spicy plugins glue compiler (Benjamin Bannier, Corelight)

  * Fix clang-tidy `performance-enum-size` reports in Spicy plugin's glue compiler (Benjamin Bannier, Corelight)

  * Remove unneeded copies in Spicy plugin glue compiler (Benjamin Bannier, Corelight)

  * Adjust for renamed function flavor in Spicy development version (Benjamin Bannier, Corelight)

    With zeek/spicy#2048 the flavor for functions was renamed from
    `Standard` to `Function`. This patch adapts the code for that while
    still allowing using earlier Spicy versions.

8.0.0-dev.82 | 2025-05-09 17:25:38 +0200

  * speed up file analysis, remove IncrementByteCount (Justin Azoff)

    Avoid creating and recreating count objects for each chunk of file
    analyzed.  This replaces counts inside of records with c++ uint64_ts.

8.0.0-dev.80 | 2025-05-09 09:51:48 +0200

  * Updating submodules binpac, broker and zeek-aux [nomail] (Arne Welzel, Corelight)

8.0.0-dev.78 | 2025-05-08 15:14:25 -0700

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

8.0.0-dev.76 | 2025-05-08 15:07:09 -0700

  * Bugfix: accurately track Broker buffer overflows w/ multiple peerings (Christian Kreibich, Corelight)

8.0.0-dev.74 | 2025-05-08 13:46:54 -0700

  * Downgrade broker clone FatalError to an Error (Tim Wojtulewicz, Corelight)

8.0.0-dev.72 | 2025-05-08 13:18:52 +0200

  * probabilistic/BitVector: Add include <cstdint> (Arne Welzel, Corelight)

  * Bump spicy to fix build with GCC 15.1 (Arne Welzel, Corelight)

  * CI: Drop fedora-40 (Arne Welzel, Corelight)

  * CI: Add fedora-42 (Arne Welzel, Corelight)

8.0.0-dev.67 | 2025-05-08 09:25:19 +0200

  * Include cstdint header to support uint8_t/uint64_t in GCC 15+ (i2z1)

8.0.0-dev.64 | 2025-05-07 15:39:17 -0700

  * Move options to redis backend options instead of module-level options (Tim Wojtulewicz, Corelight)

  * Add operation_timeout and command_timeout storage backend options (Tim Wojtulewicz, Corelight)

8.0.0-dev.61 | 2025-05-07 14:09:42 -0700

  * Fix Broker metrics naming bugs that snuck in w/ last-minute renaming [skip ci] (Christian Kreibich, Corelight)

8.0.0-dev.59 | 2025-05-07 10:29:37 -0700

  * Fix policy/protocols/conn/failed-service-logging.zeek (Johanna Amann, Corelight)

    In GH-4422 it was pointed out that the protocols/conn/failed-service-logging.zeek
    policy script only works when
    `DPD::track_removed_services_in_connection=T` is set.

    This was caused by a logic error in the script. This commit fixes this
    logic error and introduces an additional test that checks that
    failed-service-logging works even when the option is not set to true.

  * Update opt.ZAM-bif-tracking baseline (Tim Wojtulewicz, Corelight)

8.0.0-dev.56 | 2025-05-07 17:23:54 +0200

  * btest/cluster/generic/publish-any: Apply Christian's fix from broker/publish-any (Arne Welzel, Corelight)

  * wstest/terminate-while-queueing: Patch close_socket() (Arne Welzel, Corelight)

    I believe there's a bug/usage issue in the websockets library
    where during send(), EOF is detected and stored, but the receiving
    thread is then discarding the last received frame. Avoid the bug
    by replacing the close_socket() implementation of the websockets
    library just for that test and leave detecting the EOF condition
    to the receiving thread.

8.0.0-dev.53 | 2025-05-07 08:16:46 -0700

  * Add Storage::is_connected BIF (Tim Wojtulewicz, Corelight)

  * Move cast_handle from storage-async.bif to BackendHandleVal static method (Tim Wojtulewicz, Corelight)

8.0.0-dev.50 | 2025-05-07 13:21:39 +0200

  * cluster/websocket: Stop and wait for reply thread during Terminate() (Arne Welzel, Corelight)

    The terminate-while-queueing test added for #4428 failed spuriously
    indicating that sometimes WebSocket clients receive code 1000 instead of 1001.
    This happens if the ixwebsocket server is shutdown before the reply thread had a
    chance to process queued close messages.

    Fix by signaling and waiting for the dispatcher's reply thread to terminate
    before returning from Terminate().

8.0.0-dev.46 | 2025-05-06 14:20:54 +0200

  * testing/btest: Fix double commented @TEST- lines (Arne Welzel, Corelight)

        sed -i 's/^# # @/# @/g'

8.0.0-dev.44 | 2025-05-06 14:20:22 +0200

  * GH-4420: Websocket: Close onloop during Terminate() (Arne Welzel, Corelight)

    Terminate() is called when Zeek shuts down. If WebSocket client threads
    were blocked in QueueForProcessing() due to reaching queue limits, these
    previously would not exit QueueForProcessing() and instead block
    indefinitely, resulting in the ixwebsocket library blocking and its
    garbage collection thread running at 100%. Not great.

    Closing the onloop instance will unblock the WebSocket client threads
    for a timely shutdown.

    Closes #4420

  * OnLoop: notify_all() instead of notify_one() (Arne Welzel, Corelight)

    There might be more than one thread blocked waiting for room in the
    queue, ensure they all wake up when shutting down.

8.0.0-dev.40 | 2025-05-05 13:40:20 -0700

  * Add baseline for find_first test, update comments, and reorder function imports (yexiaochuan)

  * Add find_first string function (yexiaochuan)

8.0.0-dev.37 | 2025-05-05 11:11:57 -0700

  * Add commands to the static methods for the Redis implementation (Tim Wojtulewicz, Corelight)

  * Redis: Check server version when connecting (Tim Wojtulewicz, Corelight)

8.0.0-dev.34 | 2025-05-05 10:44:27 -0700

  * Use std::string_view in Redis::DoExpire to avoid copies (Tim Wojtulewicz, Corelight)

8.0.0-dev.32 | 2025-05-05 14:40:11 +0200

  * QUIC: Extract reset_crypto() function (Arne Welzel, Corelight)

  * QUIC: Rename ConnectionIDInfo to Context (Arne Welzel, Corelight)

    Lets just call it what it is given that it contains more than just
    connection IDs.

  * QUIC: Switch initial_destination_conn_id to optional (Arne Welzel, Corelight)

  * QUIC: Use initial destination conn_id for decryption (Arne Welzel, Corelight)

    Ensure the client side also uses the initial destination connection ID
    for decryption purposes instead of the one from the current long header
    packet. PCAP from local WiFi hotspot.

  * QUIC: Handle CRYPTO frames across multiple INITIAL packets (Arne Welzel, Corelight)

    Instead of sending the accumulated CRYPTO frames after processing an
    INITIAL packet, add logic to determine the total length of the TLS
    Client or Server Hello (by peeking into the first 4 byte). Once all
    CRYPTO frames have arrived, flush the reassembled data to the TLS
    analyzer at once.

  * QUIC: Do not consume EncryptedLongPacketPayload (Arne Welzel, Corelight)

    The payload is already consumed within the InitialPacket unit. Consuming
    it again resulted in UDP datagrams with multiple packets to ignore
    the remaining packets in the same UDP datagram. The baseline changes
    showing I being followed by a new H indicates that the INITIAL packet
    was followed by a HANDSHAKE packet, but previously Zeek discarded
    these.

  * QUIC: Fix ACK frame parsing (Arne Welzel, Corelight)

8.0.0-dev.22 | 2025-04-30 17:56:03 +0200

  * Add a new HookPublishEvent() plugin hook (Arne Welzel, Corelight)

    This hook can be implemented and enabled by plugins in order to
    intercept Cluster::publish() calls.

8.0.0-dev.17 | 2025-04-30 17:23:06 +0200

  * ZAM-bif-tracking: Remove ZeroMQ dependency (Arne Welzel, Corelight)

8.0.0-dev.15 | 2025-04-29 17:50:31 -0700

  * fixed incorrect ZAM optimization of expressions seen in single-statement inlined functions (Vern Paxson, Corelight)

8.0.0-dev.13 | 2025-04-29 08:52:55 -0700

  * External tests: add removed logs to CT list to prevent baseline changes (Johanna Amann, Corelight)

  * Update Mozilla CA list and CT list to NSS 3.110 (Tim Wojtulewicz, Corelight)

8.0.0-dev.10 | 2025-04-29 15:00:44 +0200

  * coalesce smtp handlers for ADDR (Mohan Dhawan, Corelight)

  * Add fine-grained groups for Intel events (Mohan Dhawan, Corelight)

8.0.0-dev.7 | 2025-04-28 19:41:01 +0200

  * broker/WebSocketShim: Check RegisterFd() return (Arne Welzel, Corelight)

  * cluster/OnLoop: Fix coverity report about proc accessed without lock (Arne Welzel, Corelight)

    Coverity complains proc is set under a lock, but accessed in Process()
    without a lock. Fix this by setting it in Close() also without locking.
    The proc member should only ever be accessed my the main thread.

8.0.0-dev.4 | 2025-04-28 18:24:09 +0200

  * GH-3045: broker/Data/data_to_val: Fail on vectors/lists with holes (Arne Welzel, Corelight)

    Instead of simply removing holes from vectors or lists when converting
    from Val to Broker format, error out as the receiver has no chance to
    reconstruct where the hole might have been.

    We could encode holes with broker::none, but this will put unnecessary
    burden on language bindings and users due to the potential optionality.
    Think a std::vector<uint64_t> that technically needs to be a
    std::vector<std::optional<uint64_t>> to represent optional elements
    properly.

    Closes #3045

8.0.0-dev.2 | 2025-04-25 16:11:36 -0700

  * Fix use-after-move in recent broker changes (Tim Wojtulewicz, Corelight)

  * Start of 8.0.0 development (Tim Wojtulewicz, Corelight)

7.2.0-dev.659 | 2025-04-25 10:33:32 -0700

  * Ignore case when matching prefix in http analyzer (Kshitiz Bartariya)

7.2.0-dev.657 | 2025-04-25 10:25:01 -0700

  * Updates for the various Broker changes (Christian Kreibich, Corelight)

  * Add versions of bundled dependencies (Tim Wojtulewicz, Corelight)

  * Lower listen/connect retry intervals in Broker and the cluster framework to 1sec (Christian Kreibich, Corelight)

    The former defaults (30sec, 1min) can slow down cluster startup and recovery
    considerably, and other systems have more aggressive intervals still.

  * Bump cluster testsuite (Christian Kreibich, Corelight)

    This pulls in an update for the backpressure disconnect tests, which now need to
    set the policy explicitly.

  * Switch Broker's default backpressure policy to drop_oldest, bump buffer sizes (Christian Kreibich, Corelight)

    At every site where we've dug into backpressure disconnect findings, it has been
    the case that the default values were too small. 8192, so 4x the old default,
    suffices at every site to drown out premature disconnects.

    With metrics now available for the send buffers regardless of backpressure
    overflow policy, this also switches the default from "disconnect" to
    "drop_oldest" (for both peers and websockets), meaning that peerings remain
    untouched but the oldest queued message simply gets dropped when a new message
    is enqueued. With this policy, the number of backpressure overflows is then
    simply the count of discarded messages, something that users can tune to see
    drop to zero in everyday use.  Another benefit is that marginal overflows cause
    less message loss than when an entire buffer's worth (plus potentially more
    in-flight messages) gets thrown out with a disconnect.

  * Deprecate Broker::congestion_queue_size and stop using it internally (Christian Kreibich, Corelight)

    Since a reorg in the Broker library (commit b04195183) that revamped flow
    control and that we pulled in with Zeek 5.0, this setting hasn't done
    anything. Broker's endpoint::make_subscriber() and
    endpoint::make_status_subscriber() take a queue size argument (with a default
    value) that simply gets dropped in the eventual subscriber::make() call. See:

    https://github.com/zeek/broker/commit/b04195183515a1db8eb493011a744ba787239e98#diff-5c0d2baa7981caeb6a4080708ddca6ad929746d10c73d66598e46d7c2c03c8deL34-R178

7.2.0-dev.649 | 2025-04-25 10:23:30 -0700

  * Lower listen/connect retry intervals in Broker and the cluster framework to 1sec (Christian Kreibich, Corelight)

  * Bump cluster testsuite (Christian Kreibich, Corelight)

  * Switch Broker's default backpressure policy to drop_oldest, bump buffer sizes (Christian Kreibich, Corelight)

  * Deprecate Broker::congestion_queue_size and stop using it internally (Christian Kreibich, Corelight)

7.2.0-dev.644 | 2025-04-25 10:02:58 -0700

  * Add basic btest to verify that Broker peering telemetry is available. (Christian Kreibich, Corelight)

  * Add cluster framework telemetry for Broker's send-buffer use (Christian Kreibich, Corelight)

    See NEWS or scripts/base/frameworks/cluster/broker-telemetry.zeek for details.

  * Add peer buffer update tracking to the Broker manager's event_observer (Christian Kreibich, Corelight)

  * Rename the Broker manager's LoggerAdapter (Christian Kreibich, Corelight)

  * Avoid race in the cluster/broker/publish-any btest (Christian Kreibich, Corelight)

7.2.0-dev.638 | 2025-04-25 06:41:06 -0700

  * Skip linting on highwayhash and src/3rdparty files (Tim Wojtulewicz, Corelight)

7.2.0-dev.636 | 2025-04-25 06:40:46 -0700

  * Minor changes to storage framework script docs (Tim Wojtulewicz, Corelight)

7.2.0-dev.634 | 2025-04-25 08:40:29 -0400

  * Bump Spicy (Evan Typanski, Corelight)

7.2.0-dev.632 | 2025-04-25 12:03:44 +0200

  * btest/cluster/websocket: Move no-subscriptions test (Arne Welzel, Corelight)

    ...and also add one for broker.

  * cluster/websocket: Leverage ReadyToPublishCallback() (Arne Welzel, Corelight)

    Change WebSocket client handling to return only when the ready to
    publish callback has been invoked.

  * cluster/zeromq: Implement DoReadyToPublishCallback() (Arne Welzel, Corelight)

    The ZeroMQ heuristic for "ready to publish" is to create an unique and
    ephemeral subscription using the XSUB socket and observe it arrive on the
    XPUB socket. At this point, visibility into other node's subscriptions
    is provided.

  * cluster/Backend: Add ReadyToPublishCallback() API (Arne Welzel, Corelight)

    Provide a mechanism to allow a cluster backend report when it is ready
    for publish operations. This is primarily useful for ZeroMQ which has
    sender-side filtering and is only really ready for publishing when it
    has learned about subscriptions from other nodes.

7.2.0-dev.627 | 2025-04-25 09:03:01 +0200

  * broker/WebSocketShim/tests: Comment out two endpoint tests (Arne Welzel, Corelight)

    Running the remote tests on a loaded system results in timeouts, even
    after bumping the tiemouts to 10 seconds. Comment them out for now.

  * broker/WebSocketShim/tests: Replace hard-coded timeout values with vars (Arne Welzel, Corelight)

7.2.0-dev.624 | 2025-04-24 14:03:52 +0200

  * cluster/websocket: Raise websocket_client_lost() after terminate (Arne Welzel, Corelight)

    Just in case events are created during backend->Terminate(). These
    should come before the Cluster::websocket_client_lost() event.

  * cluster/ThreadedBackend: Invoke onloop->Process() during DoTerminate() (Arne Welzel, Corelight)

    Also, document how to use ThreadedBackend's DoTerminate()

  * cluster/ThreadedBackend: Remove Process() (Arne Welzel, Corelight)

    This must have been left-over from before OnLoopProcess existed. It
    wasn't called or used anymore.

  * zeromq: Call super class DoTerminate() after stopping thread (Arne Welzel, Corelight)

    The internal ZeroMQ thread would call QueueForProcessing() thereby
    accessing the onloop member. As ThreadedBackend::DoTerminate() unsets it,
    this was a) reported as a data race by TSAN and b) potentially caused
    missed events that were still to be queued.

7.2.0-dev.617 | 2025-04-24 08:17:08 +0200

  * cluster/websocket: Short-circuit clients without subscriptions (Arne Welzel, Corelight)

  * cluster/websocket: Factor out active subscription handling (Arne Welzel, Corelight)

7.2.0-dev.613 | 2025-04-23 12:14:48 -0700

  * Statically lookup field offsets for connection values in UDP and ICMP analyzers (Tim Wojtulewicz, Corelight)

  * Skip calling islower before toupper (Tim Wojtulewicz, Corelight)

7.2.0-dev.608 | 2025-04-23 17:18:20 +0200

  * COPYING-3rdparty: Add some missing submodules [skip ci] (Arne Welzel, Corelight)

  * Bump Spicy. (Robin Sommer, Corelight)

7.2.0-dev.604 | 2025-04-23 08:08:00 -0700

  * Fix crash related to Broker stores (Dominik Charousset, Corelight)

7.2.0-dev.602 | 2025-04-23 15:55:46 +0200

  * cluster/cluster.bif: Fix CountVal casted to PortVal (Arne Welzel, Corelight)

    Thanks UBSAN!

7.2.0-dev.600 | 2025-04-23 15:07:47 +0200

  * CI: No more Ubuntu 20.04 [skip ci] (Arne Welzel, Corelight)

    Ubuntu 20.04's default Python doesn't deal well with the type annotations
    used in btest/Files/wstest.py. Given that it's about to be EOL, just remove it.

7.2.0-dev.598 | 2025-04-23 14:28:15 +0200

  * Add NEWS entry for WebSocket functionality (Arne Welzel, Corelight)

  * btest/cluster: Add broker logging test for sanity (Arne Welzel, Corelight)

    Not very related to the PR, but created to help provoke an issue
    with the broker changes.

  * broker/WebSocketShim: Add tests for endpoint, hubs and subscribers (Arne Welzel, Corelight)

    These are really testing broker functionality, mostly added to ensure
    the behavior is as expected by the WebSocketShim functionality.

  * broker: Deprecate NodeID(), use SetNodeId() (Arne Welzel, Corelight)

  * cluster/Backend: Do not inline Init() (Arne Welzel, Corelight)

  * cluster: Add Backend::SetNodeId() (Arne Welzel, Corelight)

  * cluster/websocket: Make websocket dispatcher queue size configurable (Arne Welzel, Corelight)

    Limit the number WebSocket events queued from external clients to
    dispatcher instances to produce back pressure to the clients if
    Zeek's IO loop is overloaded.

  * cluster/zeromq: Attempt publish during termination (Arne Welzel, Corelight)

    Explicitly notify the internal thread about the shutdown via the
    inproc socket pair. This ensures that the internal thread processes
    all previous messages on the inproc socket before terminating.

    This fixes the scenario where a backend is created, a few messages published
    and then immediately terminated as can be done with WebSocket clients.
    Previously, some of the messages published might have still been in the
    inproc socket's queue and were simply discarded.

    Adds the same test for Broker and ZeroMQ backends.

  * broker/main: Reference Cluster::publish() for auto_publish() deprecation (Arne Welzel, Corelight)

    In hindsight, this is the better thing to do and with Zeek 7.2 we should
    be confident enough that it'll work.

  * broker/main: Deprecate Broker::listen_websocket() (Arne Welzel, Corelight)

    Optimistically deprecate Broker::listen_websocket() and promote
    Cluster::listen_websocket() instead.

  * cluster/websocket: Remove comment about broker (Arne Welzel, Corelight)

  * btest/cluster/websocket: Add tests using broker (Arne Welzel, Corelight)

    Add tests to verify Cluster::listen_websocket() with the Broker backend
    is functional.

  * btest/cluster/websocket: Move ZeroMQ test and use wstest.py (Arne Welzel, Corelight)

    Adapt the test to be the same as Broker, to have "expected" behavior.

  * btest/files: Introduce wstest.py (Arne Welzel, Corelight)

    This adds a minimal helper library for reusing some of the code to
    test WebSocket client access to Zeek using Python.

  * cluster/websocket: Special case broker backend for shim usage (Arne Welzel, Corelight)

    When Cluster::backend is configured with CLUSTER_BACKEND_BROKER, switch
    WebSocketClients to CLUSTER_BACKEND_BROKER_WEBSOCKET_SHIM instead.

    Instead of the special case, we could also add something to Backend
    called NewWebSocketBackend(), but if it only affects broker, I think
    the special case is okay for now.

  * broker: Add WebSocketShim backend (Arne Welzel, Corelight)

    This adds a cluster backend implementation using broker's hub primitive
    to connect WebSocket clients with the local broker endpoint for pub/sub
    functionality.

  * broker/Manager: Add MakeHub() and ReleaseHub() (Arne Welzel, Corelight)

    These are used by WebSocket clients to create broker::hub instances

  * Bump broker submodule (Arne Welzel, Corelight)

  * cluster/Backend: Add ProcessError() (Arne Welzel, Corelight)

    Allow backends to pass errors to a strategy. Locally, these raise
    Cluster::Backend::error() events that are logged to the reporter
    as errors.

  * cluster/Backend: Rename EnqueueLocalEvent() to ProcessLocalEvent() (Arne Welzel, Corelight)

  * cluster/Backend: Rename HandleRemoteEvent() to ProcessEvent() (Arne Welzel, Corelight)

    ...also add Backend::ProcessEvent() for backends to trigger event
    processing without needing to use ProcessEventMessage().

  * cluster/websocket: Fix websocket_client_added id value (Arne Welzel, Corelight)

  * broker: Include log event identifier in stderr output (Arne Welzel, Corelight)

  * cluster/websocket: Debug output fixes (Arne Welzel, Corelight)

  * Update zeek-aux submodule [nomail] (Tim Wojtulewicz, Corelight)

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

7.2.0-dev.571 | 2025-04-22 08:44:28 -0700

  * Update c-ares to v1.34.5 and vcpkg to a version that includes it (Tim Wojtulewicz, Corelight)

7.2.0-dev.569 | 2025-04-21 17:14:44 -0700

  * Use Broker peering directionality when re-peering after backpressure overflows (Christian Kreibich, Corelight)

  * Expand Broker APIs to allow tracking directionality of peering establishment (Christian Kreibich, Corelight)

7.2.0-dev.565 | 2025-04-18 11:36:17 -0700

  * Use longer path when including krb5.h to match the cmake lookup (Tim Wojtulewicz, Corelight)

  * Update cmake submodule for krb5 fixes [nomail] (Tim Wojtulewicz, Corelight)

7.2.0-dev.562 | 2025-04-17 17:48:45 -0700

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

7.2.0-dev.560 | 2025-04-17 09:28:50 -0700

  * Update src/3rdparty submodule to disable clang-format (Tim Wojtulewicz, Corelight)

  * Disable linting for files generated by bison (Tim Wojtulewicz, Corelight)

    These files will report lots of findings in the code that we have no
    control over.

  * Make sure clang-tidy and iwyu are added to all targets (Tim Wojtulewicz, Corelight)

7.2.0-dev.556 | 2025-04-17 08:59:11 -0700

  * Add libnode-dev to docs github runner, update docs to include ZeekJS (Tim Wojtulewicz, Corelight)

7.2.0-dev.554 | 2025-04-17 16:56:34 +0200

  * pre-commit: Ensure testing files have @TEST lines commented (Arne Welzel, Corelight)

  * testing/btest/*js: Comment all @TEST lines (Arne Welzel, Corelight)

  * testing/btest/*test: Comment all @TEST lines (Arne Welzel, Corelight)

  * testing/btest/*evt: Comment all @TEST lines (Arne Welzel, Corelight)

  * testing/btest/*zeek: Comment all @TEST lines (Arne Welzel, Corelight)

7.2.0-dev.548 | 2025-04-17 15:20:27 +0200

  * Remove finish_run() (Jan Grashoefer, Corelight)

  * Deprecate session manager's Done() (Jan Grashoefer, Corelight)

  * Clear sessions when session manager is done (Jan Grashoefer, Corelight)

    It looks like there is no reason to keep sessions in the map beyond the
    point where the session manager is considered done. This hopefully
    simplifies the shutdown control flow a tiny bit.

7.2.0-dev.544 | 2025-04-16 15:40:52 -0700

  * Suppress warnings from compilation of external ixwebsocket dependency (Benjamin Bannier, Corelight)

    Clang warns about declared but unused parameters somewhere in the guts of
    IXWebSocket (internal code, not its headers). We are not interested in
    this or similar warnings since we do not control this code, so suppress
    all warnings for this target.

7.2.0-dev.541 | 2025-04-16 09:34:09 +0200

  * ldap: Clean up from code review (Arne Welzel, Corelight)

  * ldap: Add Sicily Authentication constants (Arne Welzel, Corelight)

  * GH-4275: ldap: Only switch into MS_KRB5 mode if responseToken exists (Arne Welzel, Corelight)

7.2.0-dev.536 | 2025-04-15 18:31:28 +0200

  * IXWebSocket: Bump to latest upstream master (Arne Welzel, Corelight)

    The PR for the threading issue was merged. Still keeping the submodule
    to point at our own fork as I have a hunch there might be more.

7.2.0-dev.534 | 2025-04-15 08:57:48 -0700

  * CI: Add krb5 to FreeBSD (Tim Wojtulewicz, Corelight)

  * Switch libkrb5 check to exclude only Darwin (Tim Wojtulewicz, Corelight)

7.2.0-dev.530 | 2025-04-14 10:11:59 -0700

  * Add STORAGE_ prefixes for backends and serializers (Tim Wojtulewicz, Corelight)

  * Add versioning to JSON serializer (Tim Wojtulewicz, Corelight)

  * Remove unnecessary includes in Val.h (Tim Wojtulewicz, Corelight)

  * Move byte_buffer types from cluster and storage into util (Tim Wojtulewicz, Corelight)

  * Remove unnecessary <array> and <memory> includes from util.h (Tim Wojtulewicz, Corelight)

  * Mark storage classes as final where appropriate (Tim Wojtulewicz, Corelight)

  * Add JSON storage serializer, use with existing backends/tests (Tim Wojtulewicz, Corelight)

  * Make ValFromJSON return zeek::expected instead of a variant (Tim Wojtulewicz, Corelight)

  * Ground work for pluggable storage serializers (Tim Wojtulewicz, Corelight)

7.2.0-dev.520 | 2025-04-14 18:58:55 +0200

  * Bump zeekjs to v0.17.0 (Arne Welzel, Corelight)

  * Update bifcl submodule [nomail] (Tim Wojtulewicz, Corelight)

7.2.0-dev.517 | 2025-04-13 17:11:40 +0200

  * cluster/Backend: Add name and lookup component tag (Arne Welzel, Corelight)

    This adds two new accessors on Backend, Name() and Tag() that can
    be used for introspection of a Backend instance.

  * cluster/Event: Hide members behind accessors (Arne Welzel, Corelight)

  * cluster/PublishEvent:: Make event non-const (Arne Welzel, Corelight)

    We want to introduce a hook that can modify the cluster event instances, so
    need to pass around a non-const version of it.

  * broker/Manager: Re-use broker serializer for conversion (Arne Welzel, Corelight)

  * EventMgr: Add Dispatch() with handler and args (Arne Welzel, Corelight)

    Allow users to call event_mgr.Dispatch(handler, args) instead of
    constructing the Event instance themselves. Deprecate the old API
    and replace users.

    There's a subtle change that net_done() may be propagated via
    auto_publish() now, but that still needs opt-in from script land
    and likely no one did that, or else they'd expected to have it
    work anyhow.

  * plugin/Manager: Fix MetaHookPre and MetaHookPost using HOOK_CALL_FUNCTION (Arne Welzel, Corelight)

7.2.0-dev.510 | 2025-04-11 15:16:53 +0200

  * Prevent event timestamps set to future (Jan Grashoefer, Corelight)

    For scheduled events, the event timestamp is the intended timestamp. If
    we force timer expiration, the timestamp might be in the future. Today,
    this happens on shutdown. This change guarantees that event timestamps
    are never set beyond network time.

7.2.0-dev.508 | 2025-04-11 12:59:53 +0200

  * GH-4301: Fix `--with-spicy`. (Robin Sommer, Corelight)

  * GH-3522: Spicy: Document lifetime semantics of Zeek analyzers created from Spicy. (Robin Sommer, Corelight)

  * Bump Spicy. (Robin Sommer, Corelight)

7.2.0-dev.503 | 2025-04-10 15:47:27 +0200

  * btest/files/x509/files: Sort analyzers in baseline (Arne Welzel, Corelight)

7.2.0-dev.501 | 2025-04-09 11:11:57 -0700

  * Revert addition of std::unreachable (Tim Wojtulewicz, Corelight)

    After further testing with the fuzzer corpus, std::unreachable isn't
    necessary here. It's fine to just let the default case break to fix
    the Coverity warning.

7.2.0-dev.499 | 2025-04-09 18:11:08 +0200

  * btest/brokerstore-attr-persistence-clone: Add BTEST_BG_RUN_SLEEP=1 (Arne Welzel, Corelight)

7.2.0-dev.497 | 2025-04-08 11:30:53 -0700

  * Add move assignment operator to ZeekString, which already had a move constructor (Tim Wojtulewicz, Corelight)

  * Don't initialize telemetry listener at all if ZEEKCTL_CHECK_CONFIG is set (Tim Wojtulewicz, Corelight)

  * Avoid extra temporary strings in concatenation (Tim Wojtulewicz, Corelight)

  * Use const auto references to avoid some copies (Tim Wojtulewicz, Corelight)

  * Clean up initialization of TunnelEncapsulation to use default member values (Tim Wojtulewicz, Corelight)

  * Remove some unnecessary std::move calls in broker::Manager (Tim Wojtulewicz, Corelight)

  * Avoid divide-by-zero in CardinalityCounter::Size (Tim Wojtulewicz, Corelight)

  * Minor fixes in POP3 analyzer based on flycheck warnings (Tim Wojtulewicz, Corelight)

    This also adds an implementation of unreachable() to use for default
    cases where it shouldn't ever actually cause the default.

  * More std::move changes based on Coverity findings (Tim Wojtulewicz, Corelight)

  * Fix a potential memory leak in Debug function (Tim Wojtulewicz, Corelight)

7.2.0-dev.486 | 2025-04-08 11:00:46 -0700

  * Bump pre-commit hooks (Benjamin Bannier, Corelight)

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

7.2.0-dev.482 | 2025-04-08 10:59:15 -0700

  * Make SNAP analyzer use both OUI and protocol for forwarding (Tim Wojtulewicz, Corelight)

  * Change packet analyzer identifiers to be 64-bit (Tim Wojtulewicz, Corelight)

7.2.0-dev.478 | 2025-04-04 18:36:37 +0200

  * broker/main: Adapt enum values to agree with comm.bif (Arne Welzel, Corelight)

    Logic to detect this error already existed, but due to enum identifiers
    not having a value set, it never triggered before.

  * btest/plugin: Add plugin testing enum identifiers (Arne Welzel, Corelight)

    This plugin has a generic name, but for now just tests the API around
    enum types and enum vals at InitPostScript() time.

  * Var/add_type: Do not clone EnumType when declared for the first time (Arne Welzel, Corelight)

    EnumType receives the name into its constructor. Even for the  first declaration
    the name is not empty and instead the same as the identifier's name. Due to that,
    add_type() previously took the else path and created a shallow clone of the
    initial type instead of using it. This lead to buggy behavior where enum value
    identifiers declared within an enum's first body have a different TypePtr
    associated than the one that is found via `zeek::id::find_type()`. It also
    means that enum identifiers added via redef later would have a different
    TypePtr than those in the initial pointer.

  * type/id/zeekygen: Add EnumVal to enum identifiers (Arne Welzel, Corelight)

    Provide a direct way to go from a zeek:id::ID value to EnumVal without
    needing to go through the type.

7.2.0-dev.473 | 2025-04-03 13:10:43 +0200

  * GH-4323: Traverse: Do not short-circuit traverse_all() if stmts is nullptr (Arne Welzel, Corelight)

    Since commit 0a813a53c7a5326d91fc5aa979429d04972d664b,
    zeek::detail::stmts is set to a nullptr when before it was usually
    an empty list of statements. This caused traverse_all() to be
    short-circuited unless global statements were available.

7.2.0-dev.471 | 2025-04-03 10:53:00 +0100

  * Remove unnecessary check for bind library. (Johanna Amann, Corelight)

    This has not been required since we switched to c-ares and breaks builds
    on Fedora Rawhide.

7.2.0-dev.469 | 2025-04-02 16:36:00 -0700

  * Some minor c++ modernization in EnumType methods (Tim Wojtulewicz, Corelight)

  * Avoid O(n) search in EnumType::Lookup (Tim Wojtulewicz, Corelight)

7.2.0-dev.466 | 2025-04-02 18:49:50 +0200

  * Spicy: Rework code for converting Spicy values to Zeek values. (Robin Sommer, Corelight)

    The logic was template-based so far, which wasn't great because: (1)
    conceptually, it models the Spicy types at the wrong layer (C++ rather
    than HILTI types), and (2) stopped working with some recent Spicy
    updates (which we have temporarily reverted in the meantime to keep
    Zeek working).

    The new code is based on HILTI's runtime type information and the
    corresponding introspection API, pretty much like `spicy-dump` works
    as well. This is the recommended approach for working with HILTI
    values, and generally much cleaner.

7.2.0-dev.464 | 2025-04-02 14:52:44 +0200

  * Spicy: Port over to Spicy's new tuple representation. (Robin Sommer, Corelight)

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

7.2.0-dev.461 | 2025-04-01 12:58:19 -0700

  * Add comment annotation to disable copying redef value into docs (Tim Wojtulewicz, Corelight)

7.2.0-dev.458 | 2025-03-31 18:59:11 +0200

  * CI: Upgrade zeekctl task to Debian 12 (Arne Welzel, Corelight)

  * GH-3535: CI: Add external plugins to builtin plugins task (Arne Welzel, Corelight)

    This clones a few external plugins to include them in a build. Except
    for zeek-kafka, the others are maintained by the Zeek project.

7.2.0-dev.454 | 2025-03-31 18:44:59 +0200

  * Hook into Broker logs via its new API (Dominik Charousset)

    The new Broker API allows us to provide a custom logger to Broker that
    pulls previously unattainable context information out of Broker to put
    them into broker.log for users of Zeek.

    Since Broker log events happen asynchronously, we cache them in a queue
    and use a flare to notify Zeek of activity. Furthermore, the Broker
    manager now implements the `ProcessFd` function to avoid unnecessary
    polling of the new log queue. As a side effect, data stores are polled
    less as well.

7.2.0-dev.451 | 2025-03-31 09:37:02 -0700

  * GH-3526: Add "U" to QUIC history docstrings and expand version string docs (Christian Kreibich, Corelight)

7.2.0-dev.449 | 2025-03-31 15:08:05 +0200

  * Use one-based index in warning message (Benjamin Bannier, Corelight)

    This was surfacing a zero-based counter to users which was confusing.

7.2.0-dev.447 | 2025-03-31 14:21:51 +0200

  * Event/EventMgr: protected to private (Arne Welzel, Corelight)

    These classes are final, so deriving isn't possible. No reason to have
    protected members.

  * EventHandler: Deprecate Call(args, no_remote, ts), add Call(args) (Arne Welzel, Corelight)

    The ts parameter was only added to Call() for the Broker::auto_publish()
    functionality and propagating the network timestamp. By now, the auto-publish
    functionality is deprecated, so it'd be good to cleanup that signature. There
    won't be any need for no_remote in the future either.

    Allow users to just use Call() instead.

  * EventHandler: Header cleanup (Arne Welzel, Corelight)

  * EventMgr: Do not cache current event attributes (Arne Welzel, Corelight)

    Avoid proliferation of various members on EventMgr by storing the
    pointer of the current event instead.

    This subtly changes the behavior of some builtin functions as they would
    have returned the prior event's data when executed outside of event
    draining (e.g. C++ level hook invocations), but I think that's actually
    for the better.

  * EventMgr: Drop src_val (Arne Welzel, Corelight)

    This is a left over and hasn't been used since a while.

  * Event: Header cleanup (Arne Welzel, Corelight)

7.2.0-dev.439 | 2025-03-27 16:12:44 -0700

  * Make storage events take a tag for the backend instead of a string (Tim Wojtulewicz, Corelight)

  * Add move constructor to Tag class (Tim Wojtulewicz, Corelight)

7.2.0-dev.436 | 2025-03-27 14:07:11 -0700

  * Fix handling of timeout conditions from storage backends (Tim Wojtulewicz, Corelight)

  * Reformat plugin.storage btest to be more consistent with other storage tests (Tim Wojtulewicz, Corelight)

7.2.0-dev.433 | 2025-03-27 12:40:40 -0700

  * Remove "experimental" from the QUIC history field's comment string (Christian Kreibich, Corelight)

7.2.0-dev.431 | 2025-03-26 14:23:51 +0100

  * cluster/zeromq: Improve XPUB stall behavior, add a metric (Arne Welzel, Corelight)

    Instead of fprintf, track the number of occurrences via a metric and
    change the sleep loop to a blocking send instead.

7.2.0-dev.429 | 2025-03-26 13:59:23 +0100

  * GH-4309: telemetry: Run callbacks at collect time (Arne Welzel, Corelight)

    Calling collect_metrics() from a script would not invoke metric
    callbacks, resulting in most of the process metrics to be zero
    when a Zeek process isn't scraped via Prometheus.

7.2.0-dev.427 | 2025-03-24 19:45:24 +0100

  * cluster/OnLoop: Add metric for queue stalling instead of fprintf (Arne Welzel, Corelight)

  * btest/cluster/websocket: ZeroMQ backend test (Arne Welzel, Corelight)

  * cluster/OnLoop: Switch to condition variable (Arne Welzel, Corelight)

    The busy polling wasn't clever and usually resulted in delays. For now,
    switch to mutex/condition variable and log an error if the timeouts are
    immense.

  * cluster/zeromq: Fix node_topic() and nodeid_topic() (Arne Welzel, Corelight)

    Due to prefix matching, worker-1's node_topic() also matched worker-10,
    worker-11, etc. Suffix the node topic with a `.`. The original implementation
    came from NATS, where subjects are separated by `.`.

  * cluster/websocket: Fix null deref at WebSocket server shutdown (Arne Welzel, Corelight)

  * btest/cluster/websocket: Harden multi-client tests (Arne Welzel, Corelight)

  * cluster/zeromq: Fix unsubscription visibility (Arne Welzel, Corelight)

    When two workers connect to zeek.cluster.worker, the central ZeroMQ
    proxy would not propagate unsubscription information to other nodes
    once they both left. Set ZMQ_XPUB_VERBOSER on the proxies XPUB socket
    for visibility.

7.2.0-dev.418 | 2025-03-21 11:56:40 -0700

  * Redis: Handle other errors from requests, fix KEY_EXISTS for put operations (Tim Wojtulewicz, Corelight)

  * SQLite: handle existing keys when overwrite=F correctly (Tim Wojtulewicz, Corelight)

  * Remove unnecessary type aliases from storage btests (Tim Wojtulewicz, Corelight)

  * Avoid thread-leak in scripts.base.frameworks.file-analysis.bifs.enable-disable btest (Tim Wojtulewicz, Corelight)

    This btest uses the exit() BIF to shut down, which immediately calls
    ::exit() and kills Zeek without doing any shutdown. This will sometimes
    leave the thread running the storage manager, which causes TSan to
    complain about a thread leak. Switch to use the terminate() BIF instead
    which cleanly shuts down all of Zeek.

  * Fix data-race with calling DBG_LOG from a separate thread (Tim Wojtulewicz, Corelight)

  * Fix data-race with ReturnCode objects in Sqlite::DoExpire (Tim Wojtulewicz, Corelight)

  * Fix data race with calling run_state::network_time from a separate thread (Tim Wojtulewicz, Corelight)

  * Add NEWS entry for Storage, reduce CHANGES spam (Tim Wojtulewicz, Corelight)

  * Fix Coverity findings in the SQLite backend (Tim Wojtulewicz, Corelight)

  * Remove inclusion of non-existent expected-lite path during dynamic plugin builds (Tim Wojtulewicz, Corelight)

  * Squash code from OperationResultCallback into ResultCallback (Tim Wojtulewicz, Corelight)

  * Add hiredis to generate-docs workflow to enable Redis backend (Tim Wojtulewicz, Corelight)

7.2.0-dev.405 | 2025-03-21 09:43:18 +0100

  * testing/btest: Remove btest-bg-run sleep 1 (Arne Welzel, Corelight)

    Bump to the latest btest master version and set BTEST_BG_RUN_SLEEP
    environment variable to 0. This makes btest-bg-run return
    immediately instead of delaying by 1 second.

7.2.0-dev.402 | 2025-03-19 12:47:37 -0700

  * util: Fix terminate_processing() called from non-main thread (Arne Welzel, Corelight)

    The current raise(SIGTERM) sends a signal to the calling thread. If
    terminate_processing() is called from a non-main thread and that thread
    was created at InitPostScript() time when SIGTERM is blocked, or has
    blocked SIGTERM manually, terminate_processing() has no effect.

    Switching to kill(getpid(), SIGTERM) guarantees that a thread that doesn't
    block the signal (minimally the main thread) will run the handler.

7.2.0-dev.400 | 2025-03-19 12:46:41 -0700

  * -O gen-C++ support for pattern-to-pattern comparison (Vern Paxson, Corelight)

  * -O gen-C++ support for pattern vector comparisons (Vern Paxson, Corelight)

  * -O gen-C++ support for type expressions (Vern Paxson, Corelight)

7.2.0-dev.395 | 2025-03-19 11:58:41 -0700

  * CI: Run btests on macOS under sudo (Tim Wojtulewicz, Corelight)

7.2.0-dev.393 | 2025-03-19 19:56:42 +0100

  * Bump zeekjs to v0.16.0 (Arne Welzel, Corelight)

7.2.0-dev.391 | 2025-03-19 10:10:32 -0700

  * Add is_valid_subnet BiF (Yacin Nadji, Corelight)

  * Add btests for has_valid_octets() and normalize_mac() (Christian Kreibich, Corelight)

7.2.0-dev.387 | 2025-03-19 15:28:08 +0000

  * SSH: make banner parsing more robust (Johanna Amann, Corelight)

    This change revamps SSH banner parsing.  The previous behavior was both
    a bit too strict in some regards, and too permissive in other.

    Specifically, clients are now required to send a line starting with
    "SSH-" as the first line.  This is in line with the RFC, as well with
    observed behavior. This also prevents the creation of `ssh.log` for
    non-SSH traffic on port 22.

    For the server side, we now accept text before the SSH banner. This
    previously led to a protocol violation but is allowed by the spec.

    New tests are added to cover these cases.

7.2.0-dev.382 | 2025-03-18 11:52:03 -0700

  * Add analyzer registration from VLAN to VNTAG (Tim Wojtulewicz, Corelight)

7.2.0-dev.380 | 2025-03-18 11:46:08 -0700

  * Update src/3rdparty submodule [nomail] (Tim Wojtulewicz, Corelight)

7.2.0-dev.378 | 2025-03-18 11:43:48 -0700

  * Merged new Storage framework (Tim Wojtulewicz, Corelight)

    See the NEWS for more details about this new feature.

7.2.0-dev.325 | 2025-03-18 09:07:40 +0100

  * GH-4250: Spicy: Fix passing vectors to Zeek. (Robin Sommer, Corelight)

    We missed the allocator argument for the C++-side conversion
    template, which let vector's of certain types not compile.

  * [Spicy] Add missing `to_string` overload. (Robin Sommer, Corelight)

  * Bump Spicy. (Robin Sommer, Corelight)

    This includes a fix to adapt to Spicy-side changes.

  * Bump `cmake` submodule. (Robin Sommer, Corelight)

7.2.0-dev.318 | 2025-03-17 11:55:11 -0700

  * ci/init-external-repo.sh: Use regex to match macos cirrus task (Arne Welzel, Corelight)

  * CI: Change macOS runner to Sequoia (Tim Wojtulewicz, Corelight)

7.2.0-dev.314 | 2025-03-12 13:10:58 +0100

  * github/generate-docs: Run pre-commit (Arne Welzel, Corelight)

    ...it's a bit unfortunate that we cannot use the action for this. It's
    because ./doc is a submodule and not the main repo.

  * cluster/backends/zeromq: Fix rst link in docs (Arne Welzel, Corelight)

7.2.0-dev.310 | 2025-03-11 10:50:49 +0100

  * ci/opensuse-tumpleweed: Bust cache (Arne Welzel, Corelight)

    Got a build failure because the old container images had python3-devel
    for Python3.11 or something older, but then prepare.sh would install
    Python 3.13 and Zeek's configure failing due to trying to find the
    devel headers from python313-devel which wasn't installed by prepare.sh

  * ci/macos/prepare: Install python@3 explicitly (Arne Welzel, Corelight)

    It seems Homebrew's Python 3.12 doesn't install default symlinks or
    python3 symlinks[1]. I believe this results in prepare.sh using the
    system's Python rather than Homebrew's. Install python@3 explicitly
    to put the symlinks in place.

    [1] https://stackoverflow.com/a/77655631

  * cluster/websocket: Implement WebSocket server (Arne Welzel, Corelight)

  * cluster/websocket: Add IXWebsocket submodule (Arne Welzel, Corelight)

  * ci/alpine: Install openssl package for testing (Arne Welzel, Corelight)

  * ci: Install websockets from pip for all distros (Arne Welzel, Corelight)

    The cluster/websocket tests were developed against websockets 14.2,
    but Ubuntu and Alpine ship too old versions. Switch to installing
    the latest version from pip instead, so we don't need to bother making
    tests compatible with very old Python packages shipped by distributions.

  * auxil/libunistd: Bump for ssize_t typedef (Arne Welzel, Corelight)

  * auxil/broker: Bump to latest master version (Arne Welzel, Corelight)

  * cluster/zeromq: Catch log_push.send() exception (Arne Welzel, Corelight)

  * cluster/zeromq: Catch exceptions as const zmq::error_t& (Arne Welzel, Corelight)

  * cluster/zeromq: No assert on inproc handling (Arne Welzel, Corelight)

    This might happen if we didn't succeed in completely sending a multipart
    message and stop early.

  * cluster/zeromq: Support configuring IO threads for proxy thread (Arne Welzel, Corelight)

  * cluster/zeromq: Move variable lookups from DoInit() to DoInitPostScript() (Arne Welzel, Corelight)

  * cluster/zeromq: Handle EINTR at shutdown (Arne Welzel, Corelight)

    Read ::signal_val and early exit a DoPublish() in case termination
    happened while blocked in inproc.send()

  * cluster/zeromq: Queue one message at a time (Arne Welzel, Corelight)

    Queueing multiple messages can easily overload the IO loop without
    creating any backpressure.

  * cluster/Backend: Queue a single message only (Arne Welzel, Corelight)

    The ZeroMQ backend would accumulate multiple messages and enqueue them
    all at once. However, as this could potentially result in huge batches
    of events being queued into the event loop at once, switch to a one
    message at a time model. If there's too many messages queued already,
    OnLoop::QueueForProcessing() will block the ZeroMQ thread until
    there's room available again.

  * cluster/zeromq: Adapt for OnLoopProcess changes (Arne Welzel, Corelight)

  * cluster/ThreadedBackend: Switch to OnLoopProcess (Arne Welzel, Corelight)

  * cluster/OnLoop: Introduce helper template class (Arne Welzel, Corelight)

  * serializer/broker: Expose to_broker_event() and to_zeek_event() (Arne Welzel, Corelight)

    This is useful for reuse by WebSocket clients that use
    the JSON v1 encoding.

7.2.0-dev.288 | 2025-03-10 19:16:57 +0100

  * btest/javascript: Add file_sniff() and file_state_remove() test (Arne Welzel, Corelight)

    Using file_state_remove() or file_sniff() would previously crash with the
    spicy-zip file analyzer. We don't have spicy-zip here, so it's not a
    proper reproducer, but still makes sense to test these events.

  * GH-4239: Bump zeekjs to v0.15.0 (Arne Welzel, Corelight)

        f5ff7c4 version: 0.15.0
        2a48143 docker/fedora: Bust cache and ensure sqlite-libs is updated
        05e5f57 Executor: Unblock SIGTERM in thread
        46496b9 Instance: Remove unsued variable
        c3b3c86 Instance: Offload JavaScript invocations to executor thread
        09e311f Introduce Executor helper

    Mainly, run JavaScript on a separate thread for interoperability with
    Spicy fiber stacks.

    Closes #4239

7.2.0-dev.285 | 2025-03-10 08:28:54 -0700

  * CI: Unconditionally upgrade pip on macOS (Tim Wojtulewicz, Corelight)

7.2.0-dev.283 | 2025-03-10 06:11:40 -0700

  * Fix return types in docs for double_to_count and double_to_int (Tim Wojtulewicz, Corelight)

7.2.0-dev.280 | 2025-03-07 13:13:39 -0500

  * GH-4268: Fix printing type constants like `double` (Evan Typanski, Corelight)

    Fixes #4268

7.2.0-dev.278 | 2025-03-07 11:02:12 -0700

  * Fix indentation in .clang-tidy (Tim Wojtulewicz, Corelight)

  * Remove existing hand-written clang-tidy setup (Tim Wojtulewicz, Corelight)

  * Add ability to enable iwyu and clang-tidy at configure time (Tim Wojtulewicz, Corelight)

7.2.0-dev.273 | 2025-03-06 16:41:43 -0700

  * Set ip_proto when creating new tunnel encapsulations (Tim Wojtulewicz, Corelight)

  * Add code of conduct and contributing to repo. (Johanna Amann, Corelight)

7.2.0-dev.269 | 2025-03-06 13:12:31 +0000

  * Remove violating analyzer from services field again (Johanna Amann, Corelight)

    This reverts some of the recent DPD changes; specifically violations
    trigger removal from the services field, again, by default.

    Discussion in GH-4521

7.2.0-dev.265 | 2025-03-05 19:23:53 +0100

  * ZAM: bif-tracking updates for blocking_lookup_hostname() (Arne Welzel, Corelight)

  * GH-4216 GH-4219: scan.l: Deprecate DNS resolutions of hostname literals (Arne Welzel, Corelight)

    This also skips DNS lookups when running with zeek --parse-only.

  * input/zeek-setup: Track parse_only globally (Arne Welzel, Corelight)

    For easier access in scan.l

  * zeek.bif: Introduce blocking_lookup_hostname() (Arne Welzel, Corelight)

    As a replacement for host literal DNS resolutions.

  * btest/dns_mgr: Update run-dnsmasq, use --host-record (Arne Welzel, Corelight)

    Using dnsmasq --address covers an entire domain, so dns.example.com
    AAAA queries would also return IPv6 addresses for example.com

    Instead, this was always meant to use host entries.

7.2.0-dev.258 | 2025-03-04 12:52:38 -0800

  * Update zeekctl submodule to pull in testsuite fix for metrics port (Christian Kreibich, Corelight)

7.2.0-dev.257 | 2025-03-04 09:33:54 -0700

  * Reformat embedded C++ code in bison/flex files (Tim Wojtulewicz, Corelight)

7.2.0-dev.255 | 2025-03-04 08:52:13 -0700

  * Fix Coverity findings from recent IPTunnel dumping changes (Tim Wojtulewicz, Corelight)

7.2.0-dev.253 | 2025-03-04 13:38:32 +0000

  * RDP: cookie is optional (Johanna Amann, Corelight)

    RDP connections are now parsed even if the optional cookie value
    is not present.

    Fixes GH-4237

7.2.0-dev.250 | 2025-03-04 12:07:01 +0100

  * Update zeek.bif documentation for to_count and to_int (Mark Overholser)

  * CI: Add release-build ZAM task for baseline checks during PRs (Tim Wojtulewicz, Corelight)

  * Fix language/assert btest baselines for to_count/to_int changes (Tim Wojtulewicz, Corelight)

  * Update the bifs.to_count baseline for ZAM (Tim Wojtulewicz, Corelight)

7.2.0-dev.244 | 2025-03-04 09:40:41 +0100

  * GH-2311: scan.l: Fix @load-plugin scripts loading (Arne Welzel, Corelight)

    For a plugin loaded via @load-plugin, create a YY_BUFFER_STATE holding
    the required loads for the implicitly loaded files. In loaded scripts,
    this generated file will show up with a path of the shared object file
    of the plugin with the __preload__.zeek and __load__.zeek files loaded
    by it.

    Closes #2311

  * scan.l: Extract switch_to() from load_files() (Arne Welzel, Corelight)

  * ScannedFile: Allow skipping canonicalization (Arne Welzel, Corelight)

  * Bump pre-commit hooks (Benjamin Bannier, Corelight)

  * Always break lines when formatting `spicy_add_analyzer` (Benjamin Bannier, Corelight)

  * Fix formatting of `zeek_add_plugin` (Benjamin Bannier, Corelight)

7.2.0-dev.232 | 2025-02-25 07:29:43 -0700

  * CI: Use the right variable for the number of test jobs for zeekctl_debian11_task tasks (Tim Wojtulewicz, Corelight)

  * CI: Use fewer btest jobs for ZAM tasks (Tim Wojtulewicz, Corelight)

7.2.0-dev.228 | 2025-02-24 11:37:42 -0700

  * Fix incorrect syntax for static_cast in to_count that I introduced in a previous merge (Mark Overholser)

  * Update baselines after changes to to_count and to_int (Mark Overholser)

  * fix to_count in zeek.bif to resolve conflict from bbanier's adjustment to static casting of the return from strtoull (Mark Overholser)

  * Fix tests in to_count to reflect appropriate function; add tests for 0x-formatted hex values for to_count and to_int (Mark Overholser)

  * update BIFs to_int() and to_count() to accept optional 'base' argument; allows more more exotic conversions from hex, octal, binary (Mark Overholser)

  * Fixes #4076; update BIFs to_int() and to_count() to accept optional 'base' argument for more easy conversions of strings representing (for example) hexadecimal, octal, or binary numbers (Mark Overholser)

  * geneve/functions.bif: Fix :zeek:see directive, take two (Arne Welzel, Corelight)

  * geneve/functions.bif: Fix :zeek:see directive (Arne Welzel, Corelight)

7.2.0-dev.217 | 2025-02-22 12:33:25 -0800

  * PacketAnalyzer::Geneve: Add get_options() (Arne Welzel, Corelight)

    Allow to extract Geneve options on-demand, for example during a
    new_connection() event.

  * packet_analysis: Track data spans of packet analyzers (Arne Welzel, Corelight)

    Do not just track the analyzer instance in the stack, but also the
    data span it is given. This allows to extract more information on-demand
    during event processing.

    TrackAnalyzer() is technically a public API, but no one should use it
    outside of the Analyzer's Forward methods itself.

7.2.0-dev.214 | 2025-02-21 12:24:21 +0100

  * Fix bifs.to_count and bifs.to_int btests under ZAM (Tim Wojtulewicz, Corelight)

7.2.0-dev.212 | 2025-02-20 16:40:19 -0700

  * Set original/outer packet flags to reflect inner packet results (mnhsrj)

    Propagate inner packet flags such as 'processed', 'dump_packet', 'dump_size'
    to outer packet for packets involving tunneled data.

7.2.0-dev.210 | 2025-02-20 15:35:21 -0700

  * Also trim trailing spaces in `to_count`/`to_int` inputs (Benjamin Bannier, Corelight)

    Previously we would already trim leading spaces in inputs to `to_count`
    and `to_int`, effectively by just passing the behavior of the low-level
    functions used in their implementations to the user. While this was
    useful it was also inconsistent in that we did not allow trailing
    spaces which we enable with this patch.

  * Align error handling in `to_int` with existing behavior of `to_count` (Benjamin Bannier, Corelight)

    Previously `to_int` would silently ignore invalid inputs and simply
    return `0` while `to_count` would return an error; this patch changes
    `to_int` to behave like `to_count`.

    This introduces a breaking change in that `to_int` now raises an error
    for trailing spaces (but still accepts leading spaces) where it
    previously would have silently accepted it. This is consistent with
    the behavior of `to_count`, but one could also argue that both of
    these should only accept properly trimmed input; I did not go that route
    since that would introduce breaking changes for both these functions
    instead of for just one of them.

  * Baseline handling of leading/trailing spaces in `to_count`/`to_int`. (Benjamin Bannier, Corelight)

    Currently `to_count` reports an error for trailing spaces (but not for
    leading ones) while `to_int` silently accepts them. This patch adds
    baselines capture the current behavior.

  * Add error messages to `to_count`/`to_int` baselines (Benjamin Bannier, Corelight)

    This captures error messages produced by `to_count`, but will also
    baseline future error messages from `to_int` once we introduce them.

7.2.0-dev.203 | 2025-02-18 08:51:23 -0700

  * cirrus: Bump FreeBSD 14 task to 14.2 (Arne Welzel, Corelight)

  * Fix dead links in README (Brendan Kapp)

7.2.0-dev.199 | 2025-02-14 11:28:48 -0800

  * Do not export locals (Evan Typanski, Corelight)

  * DPD changes - ZAM baseline updates (Johanna Amann, Corelight)

7.2.0-dev.194 | 2025-02-07 07:31:26 +0000

  * Allow to track service violations in conn.log. (Johanna Amann, Corelight)

    This introduces ian options, DPD::track_removed_services_in_connection.
    It adds failed services to the services column, prefixed with a
    "-".

    Alternatively, we also introduce the
    policy/protocols/conn/failed-services-logging.zeek, which provides the same
    information in a new column in conn.log.

  * Make conn.log service field ordered (Johanna Amann, Corelight)

    This changes service set in the connection record, and thus also the
    conn.log service field to being ordered. Speficically, the order of the
    entries in the service field will be the same order in which protocols
    will be confirmed. This means that it now is possible to see which
    protocols were layered over each other in which order by looking at the
    respective conn.log entry.

  * DPD: change handling of pre-confirmation violations, remove max_violations (Johanna Amann, Corelight)

    This commit revamps the handling of analyzer violations that happen
    before an analyzer confirms the protocol.

    This change reinstates the originally intended behavior of DPD. When an
    analyzer that has not been confirmed raises a protocol violation, it is
    immediately removed from the connection. This also makes a lot of sense
    - this allows the analyzer to be in a "tasting" phase at the beginning
    of the connection, and to error out quickly once it realizes that it was
    attached to a connection not containing the desired protocol.

    This change also removes the DPD::max_violations option, as it no longer
    serves any purpose after this change. (In practice, the option remains
    with an &deprecated warning, but it is no longer used for anything).

  * DPD: log analyzers that have confirmed (Johanna Amann, Corelight)

    This switches the DPD logic to always log analyzers that raised a
    protocol confirmation.

    The logic is that, once a protocol has been confirmed - and thus there
    probably is log output - it does not make sense to later remove it from
    the log. It does make sense to somehow flag it as failed - but that
    seems like a secondary step.

  * IRC analyzer - make protocol confirmation more robust. (Johanna Amann, Corelight)

    This commit changes the logic of the IRC analyzer to check for valid
    commands before confirming. Before that it nearly unconditionally confirmed
    connections.

  * CI: bump FreeBSD 14 to 14.2, which came out in Dec'24 (Christian Kreibich, Corelight)

  * CI: remove OpenSuSE Leap 15.5, EOL. (Christian Kreibich, Corelight)

7.2.0-dev.177 | 2025-02-06 09:44:15 +0100

  * GH-3595: Downgrade internal errors to reporter warnings in file/directory BiFs (Christian Kreibich, Corelight)

    This allows Zeek execution to continue gracefully in the presence of such
    errors, particularly at zeek_init() time. Includes a tweak to expand the
    bifs.directory_operations test to check continuation after errors.

  * Add NEWS entry for recent kerberos change [nomail] [skip ci] (Tim Wojtulewicz, Corelight)

7.2.0-dev.173 | 2025-02-05 16:40:41 +0100

  * cluster/zeromq: Call DoTerminate() in destructor (Arne Welzel, Corelight)

    Normal life-cycle is that Terminate() / DoTerminate() is called
    by zeek-setup code. If that doesn't happen, shutdown and join
    threads during destructor.

    try { } catch (...) suggested by Benjamin.

  * cluster/zeromq: Use lambda for thread trampoline (Arne Welzel, Corelight)

  * cluster/zeromq: Do not call util::fmt() from thread (Arne Welzel, Corelight)

    ...util::fmt() uses a static buffer, so this is problematic.

    I've dabbled a bit replacing std::thread with using threading::BasicThread
    which would offer Fmt(), but this makes things more complicated. Primarily
    as BasicThread is registered with the thread manager and the shutdown
    interactions become entangled. The thread might be terminated before the
    backend, or vice-versa. Seems nicer for the thread to be owned by the backend.

7.2.0-dev.169 | 2025-02-05 11:10:21 +0100

  * cluster/zeromq: Fix Unsubscribe() bug caused by \x00 prefix (Arne Welzel, Corelight)

  * cluster: Add SubscribeCallback support (Arne Welzel, Corelight)

    This allows callers of Subscribe() to pass in a callback that will be invoked
    once the subscription is established or failed to establish. It is the
    backend's responsibility to execute the callback on the main thread either
    synchronously, or preferably asynchronously at a later point, by
    scheduling a task on the IO main loop.

    This turns on ZMQ_XPUB_VERBOSE for ZeroMQ so that notifications about
    subscriptions are raised even if the subscriptions has previously been
    observed.

  * cluster/zeromq: Fix XSUB threading issues (Arne Welzel, Corelight)

    It is not safe to use the same socket from different threads, but the
    current code used the xsub socket directly from the main thread (to setup
    subscriptions) and from the internal thread for polling and reading.

    Leverage the PAIR socket already in use for forwarding publish operations
    to the internal thread also for subscribe and unsubscribe.

    The failure mode is/was a bit annoying. Essentially, closing of the
    context would hang indefinitely in zmq_ctx_term().

  * cluster/zeromq: Use NodeId(), drop my_node_id (Arne Welzel, Corelight)

  * cluster/Backend: Pass node_id via Init() (Arne Welzel, Corelight)

  * cluster/Backend: Make backend event processing customizable (Arne Welzel, Corelight)

    This allows configurability at the code level to decide what to do with
    a received remote events and events produced by a backend. For now, only
    enqueue events into the process's script layer, but for the WebSocket
    interface, the action would be to send out the event on a WebSocket
    connection instead.

  * cluster/broker/Serializer: Fix adaptor to adapter (Arne Welzel, Corelight)

  * cluster/Backend: Do not use const std::string_view& (Arne Welzel, Corelight)

  * cluster/serializer/broker: Fix handler lookup (Arne Welzel, Corelight)

    Handler overwrites operator bool, so need to explicitly test for nullptr
    rather than not having any handlers defined.

  * broker/Manager: Move name in PublishEvent() (Arne Welzel, Corelight)

  * btest/zeromq/test-bootstrap: Fix port parsing (Arne Welzel, Corelight)

    to_port() will produce an error on empty strings which extract_count()
    does not.

  * EventHandler: Support operator!= (Arne Welzel, Corelight)

7.2.0-dev.156 | 2025-02-05 09:05:00 +0000

  * IPv6 support for detect-external-names and testcase (Johanna Amann, Corelight)

  * Add  `skip_resp_host_port_pairs` option to policy/protocols/dns/detect-external-names.zeek.
    This option allows to skip the detection of external names for certain addresses.
    By default this is set so that mDNS broadcasts no longer are logged as extenal. (jbaggs)

7.2.0-dev.152 | 2025-02-04 14:50:36 -0800

  * Bump cluster testsuite to pull in resilience to agent connection timing [skip ci] (Christian Kreibich, Corelight)

7.2.0-dev.150 | 2025-02-04 16:50:44 +0100

  * GH-4209: util/init_random_seed: write_file implies deterministic (Johanna Amann, Corelight)

    This makes Zeek run in deterministic mode with --save-seeds usage
    and reworks all the extra indirections used in init_random_seed()
    to make it easier to follow the control flow.

7.2.0-dev.148 | 2025-02-04 09:52:29 +0100

  * external/subdir-btest.cfg: Set OPENSSL_ENABLE_SHA1_SIGNATURES=1 (Arne Welzel, Corelight)

    We already do something similar for OPENSSL_ENABLE_MD5_VERIFY=1

  * btest/x509_verify: Drop OpenSSL 1.0 hack (Arne Welzel, Corelight)

    We do not have a distro in CI anymore that ships OpenSSL 1.0,
    drop the hack.

  * GH-4035: testing/btest: Use OPENSSL_ENABLE_SHA1_SIGNATURES (Arne Welzel, Corelight)

    This reverts the call to update-crypto-policies in the Fedora 41 image
    and instead sets OPENSSL_ENABLE_SHA1_SIGNATURES in the individual tests.
    This allows RHEL 10 or Fedora 41 users to run the tests in question
    without needing to fiddle with system settings.

    Fixes #4035

7.2.0-dev.144 | 2025-02-04 09:18:25 +0100

  * Add ZAM baseline for new scripts.base.protocols.quic.analyzer-confirmations btest (Tim Wojtulewicz, Corelight)

7.2.0-dev.142 | 2025-02-03 18:41:40 -0700

  * Make enc_part value from kerberos response available to scripts (Philipp Tekeser-Glasz)

7.2.0-dev.140 | 2025-02-03 18:34:00 -0700

  * Add note to Traces/README about possible malware in pe/pe.trace (Tim Wojtulewicz, Corelight)

  * Fix formatting of Traces/README entry for modbus-eit.trace (Tim Wojtulewicz, Corelight)

7.2.0-dev.137 | 2025-02-03 16:53:04 -0800

  * Remove unused SupervisedNode::InitCluster declaration (Christian Kreibich, Corelight)

7.2.0-dev.135 | 2025-02-03 17:50:31 +0100

  * QUIC/decrypt_crypto: Rename all_data to data (Arne Welzel, Corelight)

    ...for consistency.

  * GH-4201: QUIC: Confirm before forwarding data to SSL (Arne Welzel, Corelight)

    Fixes #4201

  * GH-4198: QUIC: Parse all QUIC packets in a UDP datagram (Arne Welzel, Corelight)

    A UDP datagram may contain multiple QUIC packets, but the parser so far
    handled only the very first packet, ignoring any subsequent packets.

    Fixes #4198

  * QUIC: Only slurp till packet end, not till &eod (Arne Welzel, Corelight)

    This doesn't change behavior, but avoids slurping in more data than
    needed. A UDP packet an contain multiple QUIC packets and we'd read
    all following ones instead just the one we're interested in.

7.2.0-dev.129 | 2025-01-31 15:58:37 -0800

  * Management framework:

    - Bump cluster testsuite to pull in updated Prometheus tests (Christian Kreibich, Corelight)
    - move up addition of agent IPs into deployable cluster configs (Christian Kreibich, Corelight)
    - Support multiple instances per host addr in auto metrics generation (Michael Dopheide)
    - When auto-generating metrics ports for worker nodes, get them more uniform across instances. (Michael Dopheide)

7.2.0-dev.124 | 2025-01-30 10:51:14 -0700

  * Pack some classes for better memory usages (Tim Wojtulewicz, Corelight)

    - Analyzer: Reduce from 208 bytes to 192 bytes, remove one cache line
    - EventGroup: Reduce from 104 bytes to 96 bytes
    - Packet: Reduce from 200 bytes to 184 bytes, remove one cache line
    - threading::Value: Reduce from 48 bytes to 40 bytes
    - ConnTuple: push hole to the end of struct
    - TCP_Reassembler: Reduce from 240 bytes to 232 bytes

7.2.0-dev.122 | 2025-01-24 15:47:15 -0700

  * Make types into constants (Evan Typanski, Corelight)

    This allows types to be used in expressions, but they can't be
    reassigned. Note that this was meant to be a special "type expression" -
    but that is unnecessary complexity.

    Type expressions would allow access to the type without going through
    its constant value, but the constant value is never changed, so it's
    simply a few more checks if necessary when functionality gets expanded.
    This way, ZAM and other code will not need updates, so the potential for
    increased work in the future is probably not worth caring about.

7.2.0-dev.116 | 2025-01-24 11:42:14 -0700

  * Fix errors from rst linting on the generated docs (Tim Wojtulewicz, Corelight)

7.2.0-dev.113 | 2025-01-23 11:49:23 -0700

  * Fix options parsing segfault (Evan Typanski, Corelight)

    A command like this would segfault:

    zeek -b test.zeek --debug

    The issue was that `getopt_long` was using a null element to determine
    what the end of the options array is. If it saw a non-null element after
    `--debug` it would say it's the argument for optarg, even if it's beyond
    `zeek_args.size()`. Instead, just make sure the array is
    null-terminated.

7.2.0-dev.111 | 2025-01-23 11:21:21 -0700

  * Minor whitespace cleanup in init-bare.zeek (Tim Wojtulewicz, Corelight)

  * Add missing export blocks to init-bare.zeek (Tim Wojtulewicz, Corelight)

  * Merge some GLOBAL module sections in init-bare (Tim Wojtulewicz, Corelight)

  * Merge Tunnel module sections in init-bare (Tim Wojtulewicz, Corelight)

7.2.0-dev.106 | 2025-01-23 16:46:33 +0100

  * script_opt: Use emit_builtin_error() for Lookup-Conn (Arne Welzel, Corelight)

    Still create a separate baseline file because ZAM doesn't provide
    detailed location information.

    Patch for ZBI.op provided by @vpax

7.2.0-dev.104 | 2025-01-21 16:48:56 -0700

  * session/Manager: Emit explicit errors for FindConnection() with proto=65535 (Arne Welzel, Corelight)

    We silently broke users constructing conn_id records manually and
    subsequently using them with lookup_connection() or connection_exists().

    This is an attempt to at least report a runtime error about the situation
    so it doesn't go completely unnoticed.

  * IPAddr/ConnKey: Protect from uninitialized conn_id (Arne Welzel, Corelight)

    Check if the non-default fields exist using HasField()
    and use GetField() for proto such that it'll initialize
    the default value which GetFieldAs<> doesn't do.
    default

  * IPAddr/ConnKey: Promote transport to uint16_t (Arne Welzel, Corelight)

    Instead of a separate bool field which is also stored in the session
    table, promote the transport field to uint16_t and encode an invalid
    ConnKey as transport 2**16-2

  * session/Manager: Header cleanup (Arne Welzel, Corelight)

7.2.0-dev.99 | 2025-01-20 10:27:32 +0100

  * fixes for -O gen-standalone-C++ generation of lambdas (Vern Paxson, Corelight)

7.2.0-dev.95 | 2025-01-14 21:39:01 +0100

  * logging: Fix reporter message (Arne Welzel, Corelight)

  * logging: Avoid repeated writer name lookups for plugin hooks (Arne Welzel, Corelight)

    If a plugin provides a write hook, the invocation for HookLogWrite() would
    redo looking up the writer's name from the enum value and instantiating
    a new std::string instance for every write. Avoid doing this.

  * logging: Fix HookLogInit() and HookLogWrite() info usage (Arne Welzel, Corelight)

    There's two instances of WriterBackend::WriterInfo for a given
    writer. One in Manager::WriterInfo that's accessible via
    stream.writers and a copy within WriterFrontend.

    Commit 78999d147d7a98a064d25854eea38468c1af7c5e switched to use the
    address of the frontend's info instance for HookLogWrite() invocations,
    breaking users using the address for identification purposes.

7.2.0-dev.91 | 2025-01-14 18:20:04 +0100

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

  * Prevent unneeded copies in QUIC C++ helper code (Benjamin Bannier, Corelight)

7.2.0-dev.88 | 2025-01-14 14:39:14 +0000

  * Raise warnings when for DNS events that are not raised due to dns_skip_all_addl (Johanna Amann, Corelight)

    By default, dns_skip_all_addl is set to false. This causes several
    events to not be raised. This change emits warnings when a user defines
    event handlers for events that will not be raised.

    Furthermore, it adds notes about this behavior to the documentation. We
    also introduce a new BIF, `is_event_handled`, which checks if an event
    is handled.

    Fixes GH-4061

7.2.0-dev.84 | 2025-01-14 11:12:52 +0100

  * support for record extensions when using -O gen-standalone-C++ (Vern Paxson, Corelight)

7.2.0-dev.82 | 2025-01-14 08:42:27 +0100

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

  * Prevent copies in various places (Benjamin Bannier, Corelight)

7.2.0-dev.79 | 2025-01-13 08:30:54 -0700

  * Fix incomplete includes in Spicy SSL analyer C++ code (Benjamin Bannier, Corelight)

    This appears to have been broken by
    feec451bcee7c459bc9a93e39ae18dc41515ac17.

7.2.0-dev.77 | 2025-01-13 08:21:51 -0700

  * Add SECURITY.md, pointing at the website (Tim Wojtulewicz, Corelight)

7.2.0-dev.72 | 2025-01-11 17:36:31 +0100

  * Mark `swap` specialization `noexcept` (Benjamin Bannier, Corelight)

  * Clean up some includes (Benjamin Bannier, Corelight)

  * Prevent exception in `noexcept` function. (Benjamin Bannier, Corelight)

  * Prevent exception escape. (Benjamin Bannier, Corelight)

  * Prevent unnecessary copies in Spicy bindings (Benjamin Bannier, Corelight)

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

7.2.0-dev.64 | 2025-01-10 11:15:24 -0800

  * GH-4102: Harden flaky test based on creating a file (Evan Typanski, Corelight)

7.2.0-dev.62 | 2025-01-09 22:17:41 -0700

  * Update btests for new local-only subnets (Tim Wojtulewicz, Corelight)

  * Add recommended non-routable subnets (JW-Corelight, Corelight)

    224.0.0.0/24 (and 6to4 conversion 2002:e000::/40) from RFC5771	"Multicast Local Network Control Block" defined as non-routable.

    239.0.0.0/8 (and 6to4 conversion 2002:ef00::/24) from RFC2365 "Administratively Scoped IP Multicast"

    fec0::/10 from RFC3879 "Deprecated Site Local Addresses"
    (cherry picked from commit 821ab2dbed0bc24c14b35ed22be6ba754646ca24)

  * Fix another typo in the Coverity workflow script (Tim Wojtulewicz, Corelight)

7.2.0-dev.57 | 2025-01-09 13:47:29 -0700

  * Add missing include for <vector> (Tim Wojtulewicz, Corelight)

  * Extended ZAM validation to include macros (Vern Paxson, Corelight)

7.2.0-dev.54 | 2025-01-09 12:09:07 -0700

  * factoring of some ZAM header files for better modularity (Vern Paxson, Corelight)

7.2.0-dev.52 | 2025-01-09 12:02:45 -0700

  * fixes for initializing globals when using -O gen-standalone-C++ (Vern Paxson, Corelight)

7.2.0-dev.50 | 2025-01-09 16:26:00 +0100

  * Update COPYING date to now and fix some [skip CI] (Evan Typanski, Corelight)

7.2.0-dev.48 | 2025-01-08 21:44:21 -0800

  * Try to be more careful with `--help` printing (Evan Typanski, Corelight)
  * Update doc submodule [nomail] [skip ci] (zeek-bot)
  * Add CECPQ2 curve number to SSL consts (Johanna Amann, Corelight)

7.2.0-dev.43 | 2025-01-08 08:38:25 -0700

  * CI: Fix escaping in coverity workflow (Tim Wojtulewicz, Corelight)

7.2.0-dev.40 | 2025-01-07 11:41:31 -0700

  * Add 7.1 contributors to NEWS file [skip ci] (Christian Kreibich, Corelight)

7.2.0-dev.38 | 2025-01-07 10:05:15 -0700

  * More updates to the SSL consts from recent protocol additions (Johanna Amann, Corelight)

7.2.0-dev.35 | 2025-01-06 15:51:03 -0700

  * Update ssl consts with more post-quantum curves (Johanna Amann, Corelight)

7.2.0-dev.33 | 2025-01-06 10:37:33 -0700

  * CI: Fix coverity workflow to support large uploads (Tim Wojtulewicz, Corelight)

    The previous setup only allowed uploads of up to 500MB, which we
    eclipsed a long time ago. They recently started enforcing it, so
    this switches over to use a call-and-response API for uploading
    larger files.

  * CI: Remove use of wget from coverity workflow (Tim Wojtulewicz, Corelight)

  * CI: Upgrade coverity workflow to ubuntu24, add jq (Tim Wojtulewicz, Corelight)

7.2.0-dev.28 | 2025-01-05 14:19:15 -0700

  * Fix incorrect uses of `zeek:see` (Benjamin Bannier)

    This fixes instances where `zeek:see` was used incorrectly so it was not
    rendered correctly. All these instances have been found by looking for
    `zeek:see` in the generated HTML where it should not be visible anymore.

    I also removed a doc reference to `paraglob_add` which never existed.

7.2.0-dev.25 | 2024-12-20 15:31:19 -0700

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

7.2.0-dev.23 | 2024-12-20 15:14:33 -0700

  * Require spicy for spicy.analyzer-id btest (Tim Wojtulewicz, Corelight)

  * Document `get_tag` to ensure that `name` exists (Evan Typanski, Corelight)

    This caused confusion and I don't think it's very intuitive. If called
    with a name that does not exist, this returns without a value, not even
    an error value. Changing that seems like it could be more deprecation
    work.

7.2.0-dev.18 | 2024-12-18 15:44:40 -0700

  * Add spicy runtime-support current_analyzer_id(), use it to set id in events (Tim Wojtulewicz, Corelight)

7.2.0-dev.16 | 2024-12-18 09:28:33 +0100

  * pre-commit: Bump spicy-format to 0.23 (Arne Welzel, Corelight)

  * fix for ZAM optimization of assigning a record field to result of "in" operation (Vern Paxson, Corelight)

7.2.0-dev.13 | 2024-12-17 12:15:42 -0700

  * Return a warning and fail if creating a store at global scope (Tim Wojtulewicz, Corelight)

7.2.0-dev.11 | 2024-12-16 12:37:25 -0700

  * ZAM/relexpr-op NE for patterns (Vern Paxson, Corelight)

  * Expr: Fix folding of pattern values to support == and != (Arne Welzel, Corelight)

    The fatal error is actually triggered at runtime, so it's a bit
    dangerous for users, but not sure there's many use-cases to
    compare vectors of patterns.

7.2.0-dev.8 | 2024-12-16 10:19:18 -0700

  * QUIC/decrypt_crypto: Actually check if decryption was successful (Arne Welzel, Corelight)

    ...and bail if it wasn't.

    PCAP was produced using OSS-Fuzz input from issue 383379789.

  * QUIC/decrypt_crypto: Limit payload_length to 10k (Arne Welzel, Corelight)

    Given we dynamically allocate memory for decryption, employ a limit
    that is unlikely to be hit, but allows for large payloads produced
    by the fuzzer or jumbo frames.

  * QUIC/decrypt_crypto: Fix decrypting into too small stack buffer (Arne Welzel, Corelight)

    A QUIC initial packet larger than 1500 bytes could lead to crashes
    due to the usage of a fixed size stack buffer for decryption.

    Allocate the necessary memory dynamically on the heap instead.

  * Fix naming of zeromq package in Coverity workflow (Tim Wojtulewicz, Corelight)

  * Fix naming of cppzmq-dev package in Coverity workflow (Tim Wojtulewicz, Corelight)

7.2.0-dev.2 | 2024-12-13 16:46:18 -0700

  * CI: Add missing packages to coverity workflow (Tim Wojtulewicz, Corelight)

  * Start of 7.2.0 development (Tim Wojtulewicz, Corelight)

7.1.0-dev.829 | 2024-12-13 13:15:41 -0700

  * Updating submodule(s) [nomail] (Tim Wojtulewicz, Corelight)

7.1.0-dev.827 | 2024-12-13 12:17:25 -0700

  * btest/supervisor: Bump one timeout to 30 seconds (Arne Welzel, Corelight)

  * btest/cluster: Bump timeouts to 30 seconds (Arne Welzel, Corelight)

    ZAM startup may take a long time, particularly in CI environments, so
    bump it up from 10 to 30 seconds.

  * btest/validate-ZAM: Update baseline, run always (Arne Welzel, Corelight)

7.1.0-dev.823 | 2024-12-13 12:14:46 -0700

  * simplified ZAM instructions for executing "assert" statements (Vern Paxson, Corelight)

7.1.0-dev.821 | 2024-12-13 12:14:17 -0700

  * cirrus/container_image_manifest: match RC tags, too (Arne Welzel, Corelight)

    The rest should just work, assuming VERSION files contain an
    appropriate value. Add a check for that, too.

  * cirrus: Move additional tag "computation" into separate script (Arne Welzel, Corelight)

    This allows for easier testing locally and maybe re-usability.

7.1.0-dev.818 | 2024-12-13 08:04:17 -0700

  * fix for memory management associated with ZAM table iteration (Vern Paxson, Corelight)

7.1.0-dev.816 | 2024-12-13 12:59:08 +0100

  * zeekygen: Disable Cluster::Experimental module (Arne Welzel, Corelight)

    The nodes-experimental/manager.zeek file ends up calling Broker::publish()
    unconditionally, resulting in a warning. Skip running that code when
    generating documentation.

7.1.0-dev.814 | 2024-12-13 10:57:19 +0100

  * BTest updates in support of compiling-scripts-to-C++ (Vern Paxson, Corelight)

  * -O gen-standalone-C++ fixes for recent more aggressive AST profiling (Vern Paxson, Corelight)

7.1.0-dev.811 | 2024-12-12 20:11:39 -0800

  * removed a bunch of no-longer used BTest alternatives and their associated baselines (Vern Paxson, Corelight)

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

7.1.0-dev.808 | 2024-12-12 13:17:51 -0700

  * netcontrol: Move to Cluster::publish() (Arne Welzel, Corelight)

  * openflow: Move to Cluster::publish() (Arne Welzel, Corelight)

  * netcontrol/catch-and-release: Move to Cluster::publish() (Arne Welzel, Corelight)

  * config: Move to Cluster::publish() (Arne Welzel, Corelight)

  * ssl/validate-certs: Move to Cluster::publish() (Arne Welzel, Corelight)

  * irc: Move to Cluster::publish() (Arne Welzel, Corelight)

  * ftp: Move to Cluster::publish() (Arne Welzel, Corelight)

  * dhcp: Move to cluster publish (Arne Welzel, Corelight)

  * notice: Move to Cluster::publish() (Arne Welzel, Corelight)

  * intel: Move to Cluster::publish() (Arne Welzel, Corelight)

  * sumstats: Move to Cluster::publish() (Arne Welzel, Corelight)

7.1.0-dev.796 | 2024-12-12 13:16:57 -0700

  * cluster/Backend: Handle unspecified table/set (Arne Welzel, Corelight)

    Same as what we do in Broker. Use the expected type if publishing
    a table() or set() parameter.

    This fixes issues when switching sumstats to Cluster::publish()

  * cluster: Fix Cluster::publish() of Broker::Data (Arne Welzel, Corelight)

    The broker serializer leverages the existing data_to_val() function.
    During unserialization, if the destination type is any, the logic
    simply wraps the broker::data value into a Broker::Data record.
    Therefore, events with any parameters are currently exposed to
    the Broker::Data type.

    There is a bigger issue in that re-publishing such Broker::Data
    instances would encode them as a normal record. Explicitly prevent
    this by serializing the contained data value directly instead, similar
    to what Broker already did when publishing a record.

  * cluster: Be noisy when attempting to connect to an unknown node (Arne Welzel, Corelight)

    Mostly due to spending too much time wondering why nodes didn't connect
    when there was a mismatch between "manager" and "manager-1" in the
    cluster layout. Remove manager from test-all-policy-cluster test to
    avoid connection attempts in this test.

7.1.0-dev.792 | 2024-12-12 11:35:08 -0700

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

7.1.0-dev.790 | 2024-12-12 11:33:01 -0700

  * Add note about various dependency updates [nomail] [skip ci] (Tim Wojtulewicz, Corelight)

  * NEWS additions for v7.1 [nomail] [skip ci] (Tim Wojtulewicz, Corelight)

7.1.0-dev.787 | 2024-12-12 14:45:39 +0100

  * Pre-compute the node topics for all pool entries. (Justin Azoff, Corelight)

    A zeek script profile showed a small percentage of time spent in
    Cluster::node_topic, but this never changes and can be cached.

  * Fix ZAM's implementation of Analyzer::name() BiF (Christian Kreibich, Corelight)

    ZAM's implementation deviates from Zeek's in consideration of remappings, which
    means that running ZAM produced conn.log service names prefixed with "spicy_"
    whereas regular Zeek doesn't.

7.1.0-dev.782 | 2024-12-11 15:33:06 -0700

  * Update zeekctl submodule to remove BroControl dir (Tim Wojtulewicz, Corelight)

  * Run Coverity workflow on ubuntu 22.04 (Tim Wojtulewicz, Corelight)

7.1.0-dev.779 | 2024-12-11 12:59:17 -0700

  * Add blurb about Spicy SSL to NEWS (Johanna Amann, Corelight)

7.1.0-dev.777 | 2024-12-11 12:38:48 -0700

  * Add Spicy analyzers to Coverity builds (Tim Wojtulewicz, Corelight)

7.1.0-dev.775 | 2024-12-11 11:45:31 -0700

  * Swap pre-commit yapf for ruff/ruff-format, fix findings (Tim Wojtulewicz, Corelight)

  * Upgrade package-manager submodule to get python upgrade (Tim Wojtulewicz, Corelight)

  * Upgrade broker submodule to get python upgrade (Tim Wojtulewicz, Corelight)

  * Upgrade zeekctl submodule to get python upgrade (Tim Wojtulewicz, Corelight)

  * Upgrade zeek-client submodule to get python upgrade (Tim Wojtulewicz, Corelight)

  * Upgrade btest submodule to get python upgrade (Tim Wojtulewicz, Corelight)

  * Require Python 3.9 in CMakeLists.txt (Tim Wojtulewicz, Corelight)

  * CI: Install python 3.9 on ubuntu 20 (Tim Wojtulewicz, Corelight)

7.1.0-dev.766 | 2024-12-11 11:07:53 -0700

  * btest: Avoid loading ZeroMQ if not compiled in (Arne Welzel, Corelight)

    ...at the same time, add some `TEST-REQUIRES: have-zeromq` which
    unfortunately means that developers will usually want libzmq
    installed on their system.

  * btest/coverage: Avoid warnings in test-all-policy-cluster (Arne Welzel, Corelight)

  * Broker::publish: Warn on using Broker::publish() when inactive (Arne Welzel, Corelight)

    This is mostly for transitioning base scripts to Cluster::publish() and
    avoid silent surprises why certain things don't work when using ZeroMQ.

  * Update zeek-testing and zeek-testing-cluster commit hashes (Tim Wojtulewicz, Corelight)

7.1.0-dev.760 | 2024-12-11 09:28:04 +0100

  * Update ZAM BiF-tracking (Arne Welzel, Corelight)

  * configure: Add --disable-cluster-backend-zeromq (Arne Welzel, Corelight)

  * CMakeLists: Cluster backends output (Arne Welzel, Corelight)

  * zeromq: Conditionally enable by default (Arne Welzel, Corelight)

    Instead of having ZeroMQ as a new dependency, enable the ZeroMQ backend only
    if ZeroMQ is available on the system as suggested by Tim.

  * btest/generic: Add publish_hrw(), publish_rr() and logging tests (Arne Welzel, Corelight)

    They currently use zeromq, but technically they should be valid for
    any other backend, too, even broker.

  * generate-docs: Run on Ubuntu 24.04, add cppzmq (Arne Welzel, Corelight)

  * docker: Add cppzmq/libzmq dependencies (Arne Welzel, Corelight)

  * tsan_suppressions: Add called_from_lib: libzmq (Arne Welzel, Corelight)

  * ci: Add cppzmq and libzmq to most platforms (Arne Welzel, Corelight)

  * cluster/backend/zeromq: Add ZeroMQ based cluster backend (Arne Welzel, Corelight)

    This is a cluster backend implementation using a central XPUB/XSUB proxy
    that by default runs on the manager node. Logging is implemented leveraging
    PUSH/PULL sockets between logger and other nodes, rather than going
    through XPUB/XSUB.

    The test-all-policy-cluster baseline changed: Previously, Broker::peer()
    would be called from setup-connections.zeek, causing the IO loop to be
    alive. With the ZeroMQ backend, the IO loop is only alive when
    Cluster::init() is called, but that doesn't happen anymore.

  * cluster/backend/zeromq: Add cppzmq submodule (Arne Welzel, Corelight)

    Not all supported platforms provide a recent enough cppzmq version,
    add a fallback as submodule. cppzmq is a header-only library, so there's
    no build step involved.

  * cluster: Add Cluster::node_id(), allow redef of node_topic(), nodeid_topic() (Arne Welzel, Corelight)

    This provides a way for non-broker cluster backends to override a
    node's identifier and its own topics that it listens on by default.

  * cluster: Move publish_hrw() and publish_rr() to cluster.bif (Arne Welzel, Corelight)

    From this point on, Cluster::publish_hrw() and Cluster::publish_rr()
    go through cluster/Backend.cc code.

7.1.0-dev.745 | 2024-12-10 16:15:57 -0700

  * Add btest for unknown_protocols.log (Jan Grashoefer, Corelight)

  * Add protocol_id count to unknown protocol record (Jan Grashoefer, Corelight)

    The count representation is not logged and added for access to the value
    in log policy hooks without converting the logged hex representation.

7.1.0-dev.742 | 2024-12-10 14:41:10 -0700

  * NEWS tweaks [skip ci] (Christian Kreibich, Corelight)

    - Switch list items back to "-" from "*" -- we hadn't used "*" since
      2.5.4 but started in 7.1, probably by accident? :-)

    - Fix a typo.

7.1.0-dev.740 | 2024-12-10 20:31:36 +0100

  * Update BiF-tracking, add get_current_packet_ts() (Arne Welzel, Corelight)

    Also, run the ZAM-bif-tracking test in non-ZAM environments so
    failures are caught immediately. There's nothing overly ZAM specific about
    running this test.

    I'm not sure I like the fact that any new contributor adding a BiF
    will need to dig into this... it might be a bit intimidating.

7.1.0-dev.738 | 2024-12-10 18:26:31 +0100

  * Introduce get_packet_lag() (Jan Grashoefer, Corelight)

  * Add btest for get_current_packet_ts() (Jan Grashoefer, Corelight)

  * Introduce get_current_packet_ts to fix packet lag (Jan Grashoefer, Corelight)

    Using network_time to calculate packet lag will produce wrong results
    when there is no packet available but network time does not (yet) fall
    back to wall clock.

7.1.0-dev.734 | 2024-12-10 09:56:46 +0100

  * DNS/dns_binds_rr: Fix complte to complete typo, switch to count (Arne Welzel, Corelight)

    From my reading in the docs the complete_flag should only ever be a
    single byte, so add a weird for when it is longer, but use count
    as the new type.

7.1.0-dev.732 | 2024-12-09 23:28:30 -0800

  * Support for Broker I/O backpressure overflow policies (Christian Kreibich, Corelight, and Dominik Charousset)

    - Add sleep() BiF
    - Add backpressure disconnect notification to cluster.log and via telemetry
    - Remove unneeded @loads from base/misc/version.zeek
    - Add Cluster::nodeid_to_node() helper function
    - Support re-peering with Broker peers that fall behind
    - Add Zeek-level configurability of Broker slow-peer disconnects
    - Bump Broker to pull in disconnect feature and infinite-loop fix
    - No need to namespace Cluster:: functions in their own namespace

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

7.1.0-dev.720 | 2024-12-09 12:22:44 -0700

  * Add missing copyright line to headers and cc files (Arne Welzel, Corelight)

  * pre-commit: Add license-header check inspired by Spicy (Arne Welzel, Corelight)

  * Add missing "COPYING" in file comments (Arne Welzel, Corelight)

    This was just done via sed. There's a number of files that don't
    have a license entry at all.

7.1.0-dev.716 | 2024-12-09 12:15:46 -0700

  * BTest baseline updates for compile-to-C++ (Vern Paxson, Corelight)

  * mark ZAM regression BTests as not suitable for compile-to-C++ (Vern Paxson, Corelight)

  * fix for -O gen-C++ maintenance helper to skip BTest intermediary files (Vern Paxson, Corelight)

  * introduced simplified initialization for non-standalone -O gen-C++ code (Vern Paxson, Corelight)
    tied -O gen-standalone-C++ to use of --optimize-files

  * streamline generated -O C++ code by relying on per-function profiles rather than aggregate profile (Vern Paxson, Corelight)

  * when reporting available/unavailble C++ script bodies, flag those that are skipped (Vern Paxson, Corelight)

  * modified AST profiling to mark (and fully skip) non-optimizable functions (Vern Paxson, Corelight)

  * modified merge_types() to skip work if given identical types, which (Vern Paxson, Corelight)
    also preserves type names (useful for -O gen-C++)

7.1.0-dev.707 | 2024-12-09 12:08:21 -0700

  * Move python signatures to a separate file (Tim Wojtulewicz, Corelight)

  * Add signatures for Python bytecode for 3.8-3.14 (Tim Wojtulewicz, Corelight)

  * Bump pre-commit hooks (Benjamin Bannier, Corelight)

  * Bump typos pre-commit hooks (Benjamin Bannier, Corelight)

    This now picks up additional typical misspellings, but also triggers on
    more identifiers we use. I opted for fixing the obvious misspellings and
    updated the allowlist for anything else.

  * Bump clang-format pre-commit hooks (Benjamin Bannier, Corelight)

    This brings in slightly better formatting around uniform initialization,
    and comments after blocks not surrounded by `{ .. }`.

7.1.0-dev.700 | 2024-12-09 14:03:29 +0100

  * ScriptOpt: Fail compilation if known exprs/stmts is outdated (Arne Welzel, Corelight)

  * skip optimization of functions with AST nodes unknown to script optimization (Vern Paxson, Corelight)

  * Disable CTU-SME test under TSAN (Arne Welzel, Corelight)

7.1.0-dev.695 | 2024-12-06 17:33:19 +0100

  * btest/http: Demo StreamEvent analyzer with HTTP::upgrade_analyzers (Arne Welzel, Corelight)

  * protocol: Add StreamEvent analyzer (Arne Welzel, Corelight)

    This analyzer can be used to transport raw stream data for a given
    connection to the script layer. For example, adding this analyzer into
    the HTTP::upgrade_analyzer or using it to configure a child WebSocket
    analyzer allows to get access to the raw stream data in script land
    when no more appropriate protocol analyzer is available.

7.1.0-dev.692 | 2024-12-06 10:22:07 +0100

  * fix for ZAM "assert" statements potentially evaluating invalid expressions (Vern Paxson, Corelight)

7.1.0-dev.689 | 2024-12-05 13:48:46 -0700

  * init-bare: Deprecate tcp_reassembler_ports (Arne Welzel, Corelight)

    ...and remove from NetVar, seems left-over.

7.1.0-dev.687 | 2024-12-05 21:43:49 +0100

  * ZAM documentation updates for asserts and event handler run-time errors (Vern Paxson, Corelight)

  * BTest updates for ZAM support of (optionally) keeping "assert" statements (Vern Paxson, Corelight)

  * command-line options for controlling script optimization: keeping asserts, (Vern Paxson, Corelight)
    avoiding event handler coalescence

  * ZAM support for option to not coalesce event handlers (Vern Paxson, Corelight)

  * ZAM support for keeping "assert" statements (Vern Paxson, Corelight)

  * internal support for script optimization options for keeping asserts, not (Vern Paxson, Corelight)
    consolidating event handlers

  * ZAM operations to support asserts (Vern Paxson, Corelight)

  * simplified "assert" by not trying to catch messages that themselves have errors (Vern Paxson, Corelight)

  * ZAM optimization now removes hook calls to hooks without any bodies (Vern Paxson, Corelight)

7.1.0-dev.676 | 2024-12-05 11:08:55 -0700

  * update of BTest that tracks number of (and validates) ZAM operations (Vern Paxson, Corelight)

  * ZAM support for pattern equality/inequality operations (Vern Paxson, Corelight)

  * expanded ZAM operations for bit-shifting to allow for int/count shift values (Vern Paxson, Corelight)

  * added type coercion for bit-shifting expressions (Vern Paxson, Corelight)

7.1.0-dev.671 | 2024-12-05 14:51:01 +0100

  * zeek-testing: Add Experiment-VM-Microsoft-Windows7AD-1 PCAP and baselines (Arne Welzel, Corelight)

    The pcap comes from the following dataset [1]:

        CTU-SME-11: a labeled dataset with real benign and malicious network
        traffic mimicking a small medium-size enterprise environment

    [1] https://zenodo.org/records/7958259

7.1.0-dev.668 | 2024-12-04 13:47:10 +0100

  * init-bare/zeek-setup: Groundwork for instantiating cluster backends (Arne Welzel, Corelight)

  * cluster/serializer: Add binary-serialization-format (Arne Welzel, Corelight)

    This is a serializer for log records that is using SerialTypes
    for serializing and un-serializing rather. Essentially, this is
    similar to what broker does except for the envelope.

  * logging/WriterFrontend: Add logic for non-broker cluster backends (Arne Welzel, Corelight)

    If cluster::backend isn't broker_mgr, use the WriterFrontend's buffering
    logic and send a whole batch of log writes during FlushWriteBuffer().
    This is a different path than broker's own logging logic.

    Preferably we adapt broker to a model where it isn't
    buffering either.

  * logging/WriterBackend: Include logging/Types.h (Arne Welzel, Corelight)

  * logging/Manager: Implement new WriteBatchFromRemote() (Arne Welzel, Corelight)

  * logging/WriterFrontend: Add LogWriteHeader as member (Arne Welzel, Corelight)

    The header captures the enum values as well as the fields

  * logging: Add filter_name to WriterInfo (Arne Welzel, Corelight)

    ...with this change, it'll be possible to identify WriterFrontend's
    based on (stream, filter_name, path) pairs in addition to (stream,
    writer, path) pairs.

7.1.0-dev.660 | 2024-12-04 11:13:58 +0100

  * GH-4077: btest/spicy.type-converter: Drop ASAN_OPTIONS from test (Arne Welzel, Corelight)

7.1.0-dev.658 | 2024-12-04 11:07:34 +0100

  * Remove obsolete c_str_safe utility (Dominik Charousset, Corelight)

    The old `c_str_safe` utility function allowed Zeek to operator on
    `broker::data` and `broker::variant`. The former grants access to actual
    `std::string` objects while the latter only provides access to fields
    via `std::string_view`. Since the Zeek formatting functions need null
    terminated strings, we need to copy the characters into a
    null-terminated container first.

    After removing support for `broker::data` and `broker::variant` from the
    same code paths, we can drop `c_str_safe` and always do the copying
    (since we are always dealing with `broker::variant` now).

  * Remove obsolete Broker compatibility layer (Dominik Charousset, Corelight)

    Since the transition to broker::variant has been long finalized, there
    is no more need to be able to go back to a pre-variant version of
    Broker. Hence, we can drop various utilities that allow Zeek to run with
    older Broker releases.

7.1.0-dev.654 | 2024-12-03 10:10:15 -0700

  * Add interval_as_double argument to control how intervals are converted to JSON (Tim Wojtulewicz, Corelight)

  * Add btest for round-trip JSON conversion (Tim Wojtulewicz, Corelight)

  * Allow comparing two PatternVals (Tim Wojtulewicz, Corelight)

  * Handle conversion between data from Val::ToJSON and ValFromJSON better (Tim Wojtulewicz, Corelight)

7.1.0-dev.649 | 2024-12-02 13:43:26 +0100

  * added new Cluster:: BiFs to script optimization tracking (Vern Paxson, Corelight)

  * added locations to ZAM run-time errors about vector size mismatches (Vern Paxson, Corelight)

7.1.0-dev.645 | 2024-12-02 10:59:04 +0100

  * fixed -O dump-final-ZAM to work in conjunction with -O no-ZAM-opt (Vern Paxson, Corelight)

7.1.0-dev.643 | 2024-12-02 09:51:10 +0100

  * Remove support for building against Spicy versions before 1.8 (Benjamin Bannier, Corelight)

    zeek-6.0 shipped with spicy-1.8, but we still have dedicated handling
    for earlier versions of Spicy in the code. This patch cleans them up
    since these versions are unsupported.

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

7.1.0-dev.639 | 2024-11-27 12:31:40 -0700

  * Add TEST-REQUIRES: which jq to two new tests that are using jq (Johanna Amann, Corelight)

  * Explain why modbus test does not work with the Spicy SSL analyzer (Johanna Amann, Corelight)

    This took a _long_ time to figure out.

7.1.0-dev.635 | 2024-11-27 08:29:19 +0000

  * Support setting the synchronous flag and the journal_mode for the SQLite log writer.

    These options allow for a significant speed-up of SQLite logging, at the expense
    of some data security. To change the default settings, redef the LogSQLite::synchronous
    and LogSQLite::journal_mode consts. (Mymaqn)

7.1.0-dev.629 | 2024-11-26 17:45:08 +0100

  * ci/test.sh: Run doctest with TZ=UTC (Arne Welzel, Corelight)

    Broker's JSON serialization is TZ dependent (which seems a bug). For now
    do the same as we do in btest.cfg and run doctests with TZ set to UTC.

    Reported in zeek/broker#434.

  * cluster/setup-connections: Switch to Cluster::subscribe(), short-circuit broker (Arne Welzel, Corelight)

    For the time being, this is easiest, otherwise we'd need to
    conditionally load a broker-specific policy script based on
    Cluster::backend being set.

  * cluster/serializer: Add Broker based event serializers (Arne Welzel, Corelight)

    This adds the first event serializers that use
    broker functionality. Binary and JSON formats.

  * cluster: Add Cluster scoped bifs (Arne Welzel, Corelight)

    ... and a broker based test using Cluster::publish() and
    Cluster::subscribe().

  * Reporter: Add ScriptLocationScope helper (Arne Welzel, Corelight)

  * init-bare/zeek-setup: Add Cluster::backend const &redef (Arne Welzel, Corelight)

  * broker: Implement cluster::Backend interface (Arne Welzel, Corelight)

  * Broker: Fix some error messages (Arne Welzel, Corelight)

  * broker: Remove MakeEvent(ArgsSpan) (Arne Welzel, Corelight)

    This was added previously in the 7.1 cycle. Now that MakeEvent() was
    removed from cluster::Backend, there's no need for Broker to provide
    this version.

  * Update 3dparty submodule (Johanna Amann, Corelight)

7.1.0-dev.618 | 2024-11-26 17:17:00 +0100

  * GH-4052: More robust memory management for ZAM execution - fixes #4052 (Vern Paxson, Corelight)

  * Update SQLite to 3.47.1 (Johanna Amann, Corelight)

7.1.0-dev.614 | 2024-11-26 15:25:09 +0100

  * ZeekArgs: Deprecate val_list_to_args() (Arne Welzel, Corelight)

    Fly-by deprecation for something that isn't used in-tree anymore.

7.1.0-dev.611 | 2024-11-22 18:37:37 +0100

  * btest/plugins: Add a plugin testing Init and Done hooks (Arne Welzel, Corelight)

7.1.0-dev.609 | 2024-11-22 15:29:29 +0100

  * GH-4052: Disable core.expr-execption btest under ZAM to fix CI builds (Tim Wojtulewicz, Corelight)

7.1.0-dev.607 | 2024-11-22 12:32:21 +0100

  * btest: Test Broker::make_event() together with Cluster::publish_hrw() (Arne Welzel, Corelight)

  * btest: Add cluster dir, minimal test for enum value (Arne Welzel, Corelight)

  * broker: Add shim plugin adding a backend component (Arne Welzel, Corelight)

    For broker, this isn't really functioning, but at least makes the
    CLUSTER_BACKEND_BROKER enum available.

  * zeek-setup: Instantiate backend::manager (Arne Welzel, Corelight)

    Required to allow registration of cluster components.

  * cluster: Add to src/CMakeLists.txt (Arne Welzel, Corelight)

  * cluster: Add Components and ComponentManager for new components (Arne Welzel, Corelight)

  * cluster/Backend: Interface for cluster backends (Arne Welzel, Corelight)

  * cluster/Serializer: Interface for event and log serializers (Arne Welzel, Corelight)

  * logging: Introduce logging/Types.h (Arne Welzel, Corelight)

    Header and implementation for types shared between the cluster and
    logging framework. The logging framework will be adapted later to
    use these. For now, the new cluster components will simply reference
    them.

  * SerialTypes/Field: Allow default construction and add move constructor (Arne Welzel, Corelight)

    This is in preparation of using SerialTypes to serialize and
    unserialize complete log batches which include Field instances
    and not just Value's.

  * DebugLogger: Add cluster debugging stream (Arne Welzel, Corelight)

  * plugin: Add component enums for pluggable cluster backends (Arne Welzel, Corelight)

  * broker: Pass frame to MakeEvent() (Arne Welzel, Corelight)

    This was lost in a prior change.

7.1.0-dev.592 | 2024-11-21 16:38:55 +0100

  * sumstat/non-cluster: Move last epoch processing to zeek_done() (Arne Welzel, Corelight)

    @Sheco reported that standalone epoch processing may exclude scheduled
    events when the final sumstat epoch runs before. For example, this easily
    happens when attempting to do sumstat observations within connection_state_remove().

    Delay final epoch processing to zeek_done() instead.

    This doesn't deal with the clustered version - this would need something
    more elaborate and potentially a mechanism to delay the shutdown of
    other cluster nodes until/after sumstat processing completed.

7.1.0-dev.590 | 2024-11-21 14:40:53 +0100

  * plugin: Add InitPreExecution() (Arne Welzel, Corelight)

    Currently, plugins do not have a way to further inspect or even mutate
    script functions after ZAM optimization ran. One use-case here is
    zeek-perf-support [1]. This plugin wraps Stmt instances of functions,
    events and hooks hooks with a small assembly stub to support JIT map
    files [2] and for integration with perf tools.

    This change introduces a new InitPreExecution() hook that runs after
    ZAM optimization completed, just before the zeek_init() event is enqueued.

    Additionally, remove the existing CPP_activation_hook. It doesn't seem
    to be used. If it becomes necessary in the future, the new
    InitPreExecution() hook can be leveraged instead.

    [1] https://github.com/zeek/zeek-perf-support
    [2] https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/jit-interface.txt

7.1.0-dev.588 | 2024-11-21 10:22:21 +0100

  * Bump zeekjs to v0.14.0 (Arne Welzel, Corelight)

        79b0c21 version: 0.14.0
        b75b384 docker: Use Fedora 41 packages
        d65cbc8 Minor header cleanup
        4fd7e8b scripts: Remove __preload__.zeek and types.zeek
        93a8113 CreateEnvironment: Unset kOwnsProcessState and kOwnsInspector
        3b1f5d3 Instance::Init(): Replace parameters with Options struct

  * Fix documentation for `have_spicy` [skip CI] (Benjamin Bannier, Corelight)

    The continued line was incorrectly marked up (as a plain comment instead
    of a Zeekygen comment) which caused only half of the docstring to be
    rendered by Zeekygen.

7.1.0-dev.583 | 2024-11-19 19:16:51 +0100

  * Bump zeekjs to 0.13.2 (Arne Welzel, Corelight)

        41ec748 cmake: Use foreach range
        2b90a56 tests: Update new-event to run with 7.1+, remove ip_proto elsewhere
        a596f6f Makefile: Update outdated commands
        b954661 Enable `cmake/FindNodejs.cmake` to find more libnode versions

7.1.0-dev.581 | 2024-11-19 10:52:56 -0700

  * CI: Use test.sh script for running ZAM tests, but disable parts of it (Tim Wojtulewicz, Corelight)

    This ensures that the normal test setup is run, which enables greedy
    mode on Cirrus allowing us to over-provision the testing. This should
    help speed up the ZAM builds a bit.

7.1.0-dev.579 | 2024-11-18 10:13:00 -0800

  * Order rule traversal in RuleMatcher::Match() operations by Rule index (Christian Kreibich, Corelight)

7.1.0-dev.577 | 2024-11-18 09:39:05 -0800

  * Define ordering on Rule instances and use on sets in RuleMatcher (Christian Kreibich, Corelight)

7.1.0-dev.575 | 2024-11-18 17:28:18 +0100

  * communityid: Use GetField() over GetFieldAs() (Arne Welzel, Corelight)

    GetFieldAs() does not initialize an optional or default field. Apparently,
    for ZAM that makes a difference, possibly the [] record construction is
    optimized, not initializing proto until actually accessed. Not quite
    sure why that's not happening for classic script interpretation
    though.

  * btest/Baseline.zam: Updates for proto addition (Arne Welzel, Corelight)

7.1.0-dev.572 | 2024-11-16 09:34:27 -0700

  * Make debug stream names use "-", and handle "_" transparently (Christian Kreibich, Corelight)

    This looks consistent, and doesn't break any existing uses with underscores.

  * Make "-B all" apply to plugin streams as well. (Christian Kreibich, Corelight)

  * Sort streams in "-B help" output, and match case-insensitively throughout (Christian Kreibich, Corelight)

    Matching of plugins' debug streams was still case-sensitive. Also contains some
    minor output tweaks.

    It'd be nice to only list plugin debug streams actually _used_ by plugins. I
    didn't see a quick way to do that so that's for another time.

7.1.0-dev.568 | 2024-11-15 19:08:02 -0800

  * Don't forward out of the unknown IP transport analyzer (Tim Wojtulewicz, Corelight)

  * The cisco-fabric-path pcap used in tests doesn't need the -C argument (Tim Wojtulewicz, Corelight)

  * Add btests for ip_proto policy files (Tim Wojtulewicz, Corelight)

7.1.0-dev.563 | 2024-11-15 17:39:49 +0100

  * zeek-setup: Remove some unused headers (Arne Welzel, Corelight)

    The zeek_binpac.h one isn't used directly, but keeping it over
    <binpac.h> include. Also do some std prefixing. binpac.h has
    'using namespace std', so these slip through :-(

  * clang-format: Sort doctest header at the bottom (Arne Welzel, Corelight)

  * RuleMatcher: Move plugin/Manager.h include from .h to .cc (Arne Welzel, Corelight)

  * iosource/Manager: Remove superflous includes (Arne Welzel, Corelight)

  * telemetry/Manager: Remove broker header include, add fnmatch.h (Arne Welzel, Corelight)

  * logging/Manager: Fix using filename from input.h in debug log (Arne Welzel, Corelight)

    ...and remove network_time, it's always included.

  * Update external commit hashes (Arne Welzel, Corelight)

  * communityid: Do not include ports for non TCP, UDP, ICMP (Arne Welzel, Corelight)

    Checked against the result of pycommunityid. The SCTP case
    isn't quite right, because Zeek's core will not have extracted
    any ports for SCTP.

7.1.0-dev.552 | 2024-11-14 14:15:24 +0100

  * sumstats: Remove copy() for Broker::publish() calls (Arne Welzel, Corelight)

    Serialization happens immediately at Broker::publish() time, there
    should be no caching issues.

  * broker/Publish: Use event time instead of network time (Arne Welzel, Corelight)

    Discussed with @J-Gras, calling Broker::publish() within a scheduled
    should use the "intended timestamp" implicitly.

    This is subtle, but supposedly more expected when running
    a pcap replay cluster.

  * GH-3637: broker/Eventhandler: Deprecate Broker::auto_publish() for v8.1 (Arne Welzel, Corelight)

    Relates to #3637

  * btest: Remove Broker::auto_publish() usages (Arne Welzel, Corelight)

    The ones that seemed to test Broker::auto_publish() were annotated
    for removal.

  * frameworks/control: Remove Broker::auto_publish() (Arne Welzel, Corelight)

  * catch-and-release: Remove Broker::auto_publish() (Arne Welzel, Corelight)

  * ssl/validate-certs: Remove Broker::auto_publish() (Arne Welzel, Corelight)

  * sumstats: Remove Broker::auto_publish() (Arne Welzel, Corelight)

  * cluster_started: No Broker::auto_publish() use (Arne Welzel, Corelight)

  * openflow: Remove Broker::auto_publish() (Arne Welzel, Corelight)

  * dhcp: Remove Broker::auto_publish() (Arne Welzel, Corelight)

    This isn't prettier, but neither worse IMO. A test would be good.

  * frameworks/notice: Remove Broker::auto_publish() (Arne Welzel, Corelight)

  * netcontrol: Replace Broker::auto_publish() (Arne Welzel, Corelight)

    I'd think we could drop the cluster.zeek and non-cluster.zeek and
    just unconditionally do the publish(), but keeping it for now.

  * intel: Switch to Cluster::publish() (Arne Welzel, Corelight)

    This isn't quite making things a lot nicer, but more explicit.

  * broker: Support publish() of unspecified set() / table() (Arne Welzel, Corelight)

    Calling Broker::make_event(..., table()) or Broker::publish(..., table())
    caused runtime type errors as same_type() would result in false.

  * types: Fix table() resulting in table_type->IsSet() == true (Arne Welzel, Corelight)

    typename(table()) apparently always resulted in a set[] type
    being rendered. Make the yield type of an unspecified table
    ANY so that type->IsSet() ends up false.

    While at it, also render unspecified types as table(), set() and
    vector() rather than vector of void, set[] or table[] of any which
    IMO should help to figure out what's going.

7.1.0-dev.535 | 2024-11-13 17:28:09 -0800

  * CI: Use FEDORA40 crypto policy in Fedora 41 (Johanna Amann, Corelight)

  * Bump zeekjs to 0.13.0 (Arne Welzel, Corelight)

  * CI: bump FreeBSD 13 to 13.4, released in September (Christian Kreibich, Corelight)

  * CI: drop Fedora 39, add 41 (Christian Kreibich, Corelight)

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

7.1.0-dev.529 | 2024-11-13 14:36:04 -0700

  * Add NEWS entry for ip_proto feature (Tim Wojtulewicz, Corelight)

  * Move IP protocol names table out of policy script to init-bare (Tim Wojtulewicz, Corelight)

  * Minor review nits (Tim Wojtulewicz, Corelight)

  * Fixes for community ID hashing with new proto values (Tim Wojtulewicz, Corelight)

  * Use new_connection instead of connection_state_remove (Tim Wojtulewicz, Corelight)

  * Add policy script to remove ip_proto field, rename protocol naming script (Tim Wojtulewicz, Corelight)

  * Rename protocol_id field to ip_proto and similar renaming for name field (Tim Wojtulewicz, Corelight)

  * Increase size of proto fields to uint16_t, add common default value (Tim Wojtulewicz, Corelight)

  * Disable part of core/dict-iteration-expire5 btest to avoid iteration bug (Tim Wojtulewicz)

    The second set of seeds in this test trip the bug reported in #3538

  * Add conn.log entries for connections with unhandled IP protocols (Tim Wojtulewicz, Corelight)

  * CI: Add Ubuntu 24.10 (Johanna Amann, Corelight)

7.1.0-dev.516 | 2024-11-12 12:26:41 -0700

  * diff-remove-abspath: Add separate handling of Windows paths (Tim Wojtulewicz)

  * diff-remove-abspath: Remove capture of windows drive letters from POSIX regex (Tim Wojtulewicz)

7.1.0-dev.513 | 2024-11-12 16:16:11 +0100

  * Bump Spicy to current `main`. (Robin Sommer, Corelight)

7.1.0-dev.511 | 2024-11-12 16:00:00 +0100

  * PktSrc: Remove first_timestamp condition check (Arne Welzel, Corelight)

    The comment is stale and first_timestamp is only relevant/available
    in pseudo_realtime.

  * PktSrc: Fix includes (Arne Welzel, Corelight)

  * PktSrc/RunState: Scale on first_wallclock and move pseudo realtime logic to RunState (Arne Welzel, Corelight)

    check_pseudo_time() used zeek_start_time which skews things sufficiently
    around being in the past when ZAM compilation takes multiple seconds. Switch
    to using first_wallclock instead.

    Further, move setting of first_timestamp and first_wallclock from PktSrc
    into RunState's dispatch_packet(), so it's more centralized now.

    The only pseudo_realtime piece left in PktSrc() is in GetNextTimeout() to
    determine how long the PktSrc is idle until the next packet is ready.

  * RunState.h: Deprecate misleadingly named current_packet_timestamp() (Arne Welzel, Corelight)

    This returns current_pseudo, naming it current_packet_timestamp()
    is actively misleading.

  * debug: Add processing suspended/continued to debug.log (Arne Welzel, Corelight)

7.1.0-dev.505 | 2024-11-12 10:26:15 +0100

  * ZAM fixes for assignments involving "any" record fields (Vern Paxson, Corelight)

  * fixes for (mostly ZAM) vector operation issues found by ASAN (Vern Paxson, Corelight)

7.1.0-dev.501 | 2024-11-11 21:05:41 +0100

  * GH-4006: Fix nullptr deref in Spicy accept/decline input (Evan Typanski, Corelight)

    Seems like this is a continuation of #4006

  * Spicy: Improve error messages reporting malformed unit names in EVT files. (Robin Sommer, Corelight)

  * GH-3988: Spicy:: Remove unhelpful assertion. (Robin Sommer, Corelight)

    In cases of a malformed event definition, this could fire instead of a
    more helpful error message coming later.

    Closes #3988.

  * GH-4007: Spicy: Do not raise an analyzer error when a connection is missing a regular tear-down. (Robin Sommer, Corelight)

    So far, when Zeek didn't see a connection's regular tear-down (e.g.,
    because its state timed-out before we got to the end), we'd still
    signal a regular end-of-data to Spicy parsers. As a result, they would
    then typically raise a parse error because they were probably still
    expecting data and would now declare it missing. That's not very
    useful because semantically it's not really a protocol issue if the
    data just doesn't make it over to us; it's a transport-layer issue
    that Zeek already handles elsewhere. So we now switch to signaling
    end-of-data to Spicy analyzers only if the connection indeed shuts
    down regularly. This is also matches how BinPAC handles it.

    This also comes with a test exercising various combinations of
    end-of-data behavior so that we ensure consistent/desired behavior.

    Closes #4007.

7.1.0-dev.494 | 2024-11-11 10:25:43 +0100

  * ci: Run ZAM CI if src/script_opt is modified (Arne Welzel, Corelight)

    ...and rework && to || conditions.

7.1.0-dev.492 | 2024-11-11 09:52:41 +0100

  * DNS_Mgr: Remove processing of dns aliases in general (Tim Wojtulewicz, Corelight)

  * ci: Add dnsmasq to a few platforms for testing (Arne Welzel, Corelight)

  * DNS_Mgr: Fix aliases memory issues (Arne Welzel, Corelight)

  * btest: Add integration test for DNS_Mgr (Arne Welzel, Corelight)

    This makes use of an ephemeral dnsmasq instance

  * DNS_Mgr: Remove usage of ares_getsock from Lookup (Tim Wojtulewicz, Corelight)

  * DNS_Mgr: Remove usage of ares_getsock from GetNextTimeout (Tim Wojtulewicz, Corelight)

  * DNS_Mgr: Switch to ares_set_servers_csv (Tim Wojtulewicz, Corelight)

  * DNS_Mgr: Use ares_dns_record methods for queries (Tim Wojtulewicz, Corelight)

  * Update vcpkg submodule to pick up c-ares v1.34.2 (Tim Wojtulewicz, Corelight)

  * Update c-ares submodule to v1.34.2 (Tim Wojtulewicz, Corelight)

7.1.0-dev.481 | 2024-11-11 09:34:10 +0100

  * policy/community-id: Populate conn$community_id in new_connection() (Arne Welzel, Corelight)

    This wasn't possible before #3028 was fixed, but now it's safe to set
    the value in new_connection() and allow other users access to the
    field much earlier. We do not have to deal with connection_flipped()
    because the community-id hash is symmetric.

7.1.0-dev.478 | 2024-11-08 18:36:07 +0100

  * Fixed ZAM memory leak when coercing values to "any" (Vern Paxson, Corelight)

  * ci: Remove -b from test_script lines for zam tasks (Arne Welzel, Corelight)

7.1.0-dev.475 | 2024-11-08 15:35:00 +0100

  * fixes for ZAM's propagation of control flow information for some degenerate constructs (Vern Paxson, Corelight)

7.1.0-dev.473 | 2024-11-08 15:31:39 +0100

  * fixed ZAM memory leak when looping over vectors of records (Vern Paxson, Corelight)

7.1.0-dev.471 | 2024-11-08 15:22:48 +0100

  * fixed access to uninitialized memory in ZAM's "cat" built-in (Vern Paxson, Corelight)

  * GH-3906: ci: Add asan and ubsan sanitizer tasks for ZAM (Arne Welzel, Corelight)

7.1.0-dev.467 | 2024-11-06 11:50:12 +0100

  * Make Spicy hook execution order deterministic in test (Benjamin Bannier, Corelight)

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

7.1.0-dev.464 | 2024-11-05 17:38:41 +0100

  * Fix potential nullptr deref in Spicy plugin runtime (Benjamin Bannier, Corelight)

    If we added a file but the other side of the connection had already run
    into a protocol violation and shut down we could previously have
    dereferenced a null cookie. This patch fixes the code so it now throws
    in such scenarios.

  * Assume no Spicy cookie in most places (Benjamin Bannier, Corelight)

    We would previously assert that it was available which could have lead
    to aborts since when the analyzer for either side of a connection shuts
    down the connection cookie could get cleared and become nil. This patch
    reworks the code slightly so we now never assume it is available. We do
    this by either throwing or by making the whole operation requesting the
    cookie a noop.

7.1.0-dev.461 | 2024-11-04 19:35:21 +0100

  * btest: Add detect-sqli test script (Arne Welzel, Corelight)

    ...I added the baselines during the merge, just not the test itself.

7.1.0-dev.459 | 2024-11-04 18:47:02 +0100

  * switched CPPCompile::Canonicalize() to take std::string instead of const char* (Vern Paxson, Corelight)

7.1.0-dev.457 | 2024-11-04 16:33:14 +0100

  * minor ZAM BTest updates for recently added fnv1a64 BiF (Vern Paxson, Corelight)

7.1.0-dev.455 | 2024-11-04 16:32:37 +0100

  * RuleMatcher: Use a single list for tracking pattern_matches (Arne Welzel, Corelight)

  * ZeekString: Implement move constructor (Arne Welzel, Corelight)

  * Bump and reenable spicy-format pre-commit hook (Benjamin Bannier, Corelight)

  * Add test for TCP over 802.3/SNAP (Johanna Amann, Corelight)

    This admittedly is a quite esoteric combination of protocols. But - as
    we do correctly support them, it seems nice to have a slightly more
    complete testcase that covers this.

7.1.0-dev.447 | 2024-10-30 14:52:40 +0100

  * CI: Upgrade FreeBSD 14 to 14.1 (Evan Typanski, Corelight)

  * CI: Add Ubuntu Clang task (Evan Typanski, Corelight)

    This feels more like a band-aid solution - it still requires very manual
    intervention to get the next builds, but at least this way we don't rely
    on whatever Clang/libc++ happens to ship with FreeBSD for the only
    libc++ testing. It seems like sanitizer builds don't use libc++, but I'd
    rather a dedicated Linux/Clang/libc++ task at least.

  * Fix up minor warnings in touched files (Evan Typanski, Corelight)

  * GH-3994: Fix Clang 19 deprecation failure (Evan Typanski, Corelight)

    Fixes #3994

    Clang 19 with libc++ started failing to compile because the default
    implementation of `std::char_traits` was removed, making uses of
    `std::char_traits<unsigned char>` invalid. This was more of used for
    convenience before, but it should be roughly the same behavior with
    `char`.

    See relevant LLVM commits:

    https://github.com/llvm/llvm-project/commit/aeecef08c385b1e4955155dd649a2d3724463849

    https://github.com/llvm/llvm-project/commit/08a0faf4cd32bce6c51027ea9b5ec351747995b4

7.1.0-dev.442 | 2024-10-30 14:43:18 +0100

  * pre-commit-config: Disable spicy-format (Arne Welzel, Corelight)

    This fails to install currently: bbannier/spicy-format#157

7.1.0-dev.441 | 2024-10-30 14:36:57 +0100

  * GH-3984: Fix issues with Modbus message logging (Emmanuele Zambon)

    This commit fixes three issues with Zeek's Modbus message logging:

    1 - Some exception responses (e.g., READ_COILS_EXCEPTION) are logged
        twice: once without and once with the exception message.
    2 - Some exception responses (e.g., PROGRAM_484_EXCEPTION) are not
        logged.
    3 - Some known but reserved function codes (e.g., PROGRAM_UNITY) are
        logged as unk-xxx (e.g., unk-90), while it would be possible to
        log their known name.

    To address these inconsistencies, the modbus parser has been updated
    to parse all exception responses (i.e., all responses where the MSB
    of the function code is set) using the already defined Exception
    message.
    Also, the Modbus main.zeek script has been updated to consistently
    demand logging exception responses to the specialized
    modbus_exception event, rather than logging some exception responses
    in the modbus_message event and others in the modbus_exception event.
    Finally, the main.zeek script has been updated to make sure that
    for every known function code, the corresponding exception code was
    also present, and the enumeration of known function codes in
    consts.zeek has been expanded.

    Closes #3984

7.1.0-dev.439 | 2024-10-30 14:33:48 +0100

  * signatures: Add data_end_offset to signature_match() and custom events (Arne Welzel, Corelight)

    This change tracks the current offset (number of bytes fed into matchers)
    on the top-level RuleEndpointState such that we can compute the relative ending
    for matched texts individually.

    Additionally, it adds the data_end_offset as a new optional parameter to
    signature_match().

  * Add pattern_end_offset to signature_state (ronny8360988)

7.1.0-dev.436 | 2024-10-30 11:47:53 +0000

  * modbus: Skip non-modbus traffic testing with Spicy SSL (Arne Welzel, Corelight)

    With Spicy SSL, there's no SSL conn.log produced.

  * testing: Add have-spicy-ssl helper and update tests (Arne Welzel, Corelight)

  * Improvements in FNV1A hash functions (Martin van Hensbergen)

    - Add FNV1a64 hash function
    - make hash function type aware
    - add unit tests

7.1.0-dev.430 | 2024-10-29 14:08:25 +0100

  * account for spaces encoding to plus signs in sqli regex detection (Cooper Grill)

7.1.0-dev.428 | 2024-10-29 13:50:05 +0100

  * fixed "-O gen-C++" naming of "when" captures to avoid ambiguities due to inlining (Vern Paxson, Corelight)

    	Previously there was logic for doing this, but it was ineffectual
    	due to the order in which canonicalization was done. This problem
    	manifested non-deterministically depending on order-of-generation
    	of "when" lambdas, which is why previous testing didn't catch it.

  * Simplified & made more robust maintenance helper script for "-O gen-C++" testing (Vern Paxson, Corelight)

  * "-a cpp" baseline updates to reflect recent BTest changes (Vern Paxson, Corelight)

7.1.0-dev.423 | 2024-10-24 07:08:45 -0700

  * Telemetry framework: move BIFs to the primary-bif stage (Christian Kreibich, Corelight)

  * Minor comment tweaks for init-frameworks-and-bifs.zeek (Christian Kreibich, Corelight)

7.1.0-dev.420 | 2024-10-23 05:49:31 -0700

  * Update broker submodule (Christian Kreibich, Corelight)

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

7.1.0-dev.418 | 2024-10-22 19:04:22 +0200

  * btest/telemetry: Fix "Note compilable" typo (Arne Welzel, Corelight)

  * misc/stats: Add zeek_net_timestamp_seconds (Arne Welzel, Corelight)

    Adding a metric for the network time value itself should make it
    possible to observe it stopping or growing slowly as compared to
    realtime when Zeek isn't able to keep up.

    Also, modify the telemetry/log.zeek test to include misc/stats and
    log at a higher frequency with a more interesting pcap.

  * telemetry/Manager: Remove variant include (Arne Welzel, Corelight)

  * GH-3947: telemetry: Invoke Telemetry::sync() only at scrape/collection time (Arne Welzel, Corelight)

    This stops invoking Telemetry::sync() via a scheduled event and instead
    only invokes it on-demand. This makes metric collection network time
    independent and lazier, too.

    With Prometheus scrape requests being processed on Zeek's main thread
    now, we can safely invoke the script layer Telemetry::sync() hook.

7.1.0-dev.413 | 2024-10-22 11:24:50 +0200

  * GH-3978: Bump zeekjs to 0.12.1 (Arne Welzel, Corelight)

        7ec0953 ci: Bump actions/upload-artifact
        8e5914d ci/docker: Bump distros, update some OBS repos
        129b737 Fix crash during shutdown

7.1.0-dev.410 | 2024-10-21 15:51:05 +0200

  * GH-3962: Prevent non-Modbus on port 502 to be reported as Modbus (Emmanuele Zambon)

    This commit prevents most non-Modbus TCP traffic on port 502 to be
    reported as Modbus in conn.log as well as in modbus.log.
    To do so, we have introduced two &enforce checks in the Modbus
    protocol definition that checks that some specific fields of the
    (supposedly) Modbus header are compatible with values specified in
    the specs.

    To ensure non-regression, with this commit we also introduce a
    new btest.

    Closes #3962

7.1.0-dev.408 | 2024-10-18 17:51:04 -0700

  * factored CPP source's main header into collection of per-source-file headers (Vern Paxson, Corelight)

  * renamed script optimization Attrs.h header to prepare for factoring large Compile.h (Vern Paxson, Corelight)

  * factored ZAM source's main header into collection of per-source-file headers (Vern Paxson, Corelight)

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

7.1.0-dev.403 | 2024-10-18 09:55:20 -0700

  * Fix installation of Python packages in generate docs CI job again (Benjamin Bannier, Corelight)

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

7.1.0-dev.400 | 2024-10-15 17:11:57 +0200

  * Fix invalid Sphinx directive in docstring (Benjamin Bannier, Corelight)

  * Bump GH actions for generate-docs job (Benjamin Bannier, Corelight)

  * Fix installation of Python packages in generate docs CI job (Benjamin Bannier, Corelight)

  * Bump GH actions for pre-commit job (Benjamin Bannier, Corelight)

  * Pin distribution for pre-commit CI job (Benjamin Bannier, Corelight)

7.1.0-dev.393 | 2024-10-15 09:56:19 +0200

  * Test `.evt` file `&priority` (Evan Typanski, Corelight)

    This was not documented nor tested, so this tests the behavior.
    Documentation should be added later.

7.1.0-dev.390 | 2024-10-09 15:36:41 -0700

  * removed specialized ZAM instructions for GTPv1 and Teredo cleanup BiFs (Vern Paxson, Corelight)

7.1.0-dev.388 | 2024-10-09 15:36:09 -0700

  * -a zam baseline updates reflecting recent changes to main baselines (Vern Paxson, Corelight)

7.1.0-dev.386 | 2024-10-09 14:26:35 -0700

  * GH-3957: input/Raw: Rework GetLine() (Arne Welzel, Corelight)

  * GH-215: POP3: Rework unbounded pending command fix (Arne Welzel, Corelight)

7.1.0-dev.383 | 2024-10-04 12:45:05 -0700

  * Bump zeek-testing-cluster to pull in a baseline update (Christian Kreibich, Corelight)

7.1.0-dev.381 | 2024-10-04 19:31:36 +0200

  * GH-3949: Fix flaky `missing-file-initially` test (Evan Typanski, Corelight)

    That test got flaky probably from #3949 on centosstream9 CI. You can
    replicate that behavior by increasing the sleep time when waiting for
    the file such that the test will attempt to read the missing file again.
    Since the one second wait for file is glacially slow for this, speeding
    it up should mean that the file gets created sooner and so the test
    won't try to open the file again. But, it's always still technically
    possible, since the test will wait for 10 seconds and the heartbeat
    seems to be 1 second. At least if that happens, it's probably a bug or
    massive slowdown of some kind.

7.1.0-dev.379 | 2024-10-04 13:34:31 +0200

  * Add missing newline in `spicyz` usage message (Benjamin Bannier, Corelight)

7.1.0-dev.377 | 2024-10-02 20:17:28 +0200

  * Fix input error test nondeterminism (Evan Typanski, Corelight)

    It seems like other similar tests get by because they have more "stuff"
    before they call `terminate()` most likely. But, to be safe, just
    removing the "received termination signal" line seems like the best
    approach.

7.1.0-dev.375 | 2024-10-01 11:40:16 +0200

  * Report suppressed warnings count (Evan Typanski, Corelight)

    This also triggers if there is one warning, which seems a little weird,
    but it seems mostly reasonable.

  * GH-3692: Avoid duplicating warnings when reading table (Evan Typanski, Corelight)

    Invalid lines in a file was the one case that would not suppress future
    warnings. Just make it suppress warnings too, but clear that suppression
    if there is a field in between that doesn't error.

    Fixes #3692

7.1.0-dev.371 | 2024-09-27 15:17:04 -0700

  * A round of ZAM/gen-C++-related updates and fixes (Vern Paxson, Corelight)

    - factoring of logic used by ZAM's low-level optimizer when adjusting control flow info
    - BTest baseline update for more complete function/lambda names
    - tweak to -O gen-C++ maintenance script to avoid treating plugins as BTests
    - fixed lambda hash collision bug due to function descriptions lacking full parameter information
    - fixes (to avoid collisions) for AST profiling's function hash computations
    - removed unused ZAM cast-to-any operation
    - fixes for ZAM tracking the return type associated with function calls
    - ZAM control-flow tracking now explicitly includes the ends of loops
    - fix for ZAM identification of common subexpressions
    - "-O dump-final-ZAM" option similar to "dump-ZAM" only prints final version of functions
    - fix for setting object locations to avoid use-after-free situation
    - extended "-O allow-cond" to apply to both gen-C++ and gen-standalone-C++
    - -O gen-C++ fix for run-time warnings for "when" lambdas
    - fix to -O gen-C++ for recent AST profiling changes for identifying function parameters
    - fix to -O gen-C++ for dealing with "hidden" parameters
    - tweak to prevent an incorrect warning for scripts compiled to C++
    - fixed overly narrow Spicy test for manipulating packet analyzers
    - fixed memory leak for recursive ZAM functions that exit via an exception
    - remove unnecessary header include

7.1.0-dev.351 | 2024-09-27 16:47:39 +0200

  * Honor no plaintext password logging for PASS cmd (Martin van Hensbergen)

7.1.0-dev.349 | 2024-09-27 15:37:13 +0200

  * NEWS: Update (Arne Welzel, Corelight)

  * scripts/base/cluster: Move active node management into node_down() (Arne Welzel, Corelight)

    With the idea of an alternative cluster backend, we should
    not maintain Cluster state within low-level Broker events.

  * logging/Manager: Extract another CreateWriter() helper (Arne Welzel, Corelight)

    For other cluster backends, CreateWriter() will use a logger's filter
    configuration rather than receiving all configuration through CreateLog.
    Extract a helper out from WriteToFilters() for reuse.

  * logging/Manager: Extract path_func invocation into helper (Arne Welzel, Corelight)

  * logging: Dedicated log flush timer (Arne Welzel, Corelight)

    Log flushing is currently triggered based on the threading heartbeat timer
    of WriterBackends and the hard-coded WRITE_BUFFER_SIZE 1000.

    This change introduces a separate timer that is managed by the logger
    manager instead of piggy-backing on the heartbeat timer, as well as a
    const &redef for the buffer size.

    This allows to modify the log flush frequency and batch size independently
    of the threading heartbeat interval. Later, this will allow to re-use the
    buffering and flushing logic of writer frontends for non-Broker cluster
    backends, too.

    One change here is that even frontends that do not have a backend will
    be flushed regularly. This is wanted for non-Broker backends and should be
    very cheap. Possibly, Broker can piggy back on this timer down the road, too,
    rather than using its own script-level timer (see Broker::log_flush()).

  * all: Change to use Func::GetName() (Arne Welzel, Corelight)

  * script_opt: Use Func::GetName() (Arne Welzel, Corelight)

  * Func: Add std::string name accessors, deprecate const char* versions (Arne Welzel, Corelight)

  * plugin/ComponentManager: Support lookup by EnumValPtr (Arne Welzel, Corelight)

7.1.0-dev.339 | 2024-09-27 14:59:45 +0200

  * zeek-setup: Support enabling debug streams through env variable (Arne Welzel, Corelight)

    For debugging btests, it can be convenient to enable debug streams
    by setting an environment variable rather than editing zeek invocations
    and adding -B selectively.

    Sample use case:

        $ export ZEEK_DEBUG_LOG_STREAMS=all
        $ btest -d core/failing-test.zeek
        $ less .tmp/core/failing-test/debug.log

    This change makes Zeek's -B option and ZEEK_DEBUG_LOG_STREAMS are additive.

  * Install `procps` in OpenSuse Leap images (Benjamin Bannier, Corelight)

    The tests `core.sigterm-regular` and `core.sigterm-stdin` rely on `ps`
    to be present which is not the case anymore on OpenSuse Leap; install it
    explicitly there.

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

7.1.0-dev.332 | 2024-09-23 10:59:51 -0700

  * pop3: Remove unused headers (Arne Welzel, Corelight)

  * pop3: Prevent unbounded state growth (Arne Welzel, Corelight)

    The cmds list may grow unbounded due to the POP3 analyzer being in
    multiLine mode after seeing `AUTH` in a Redis connection, but never
    a `.` terminator. This can easily be provoked by the Redis ping
    command.

    This adds two heuristics: 1) Forcefully process the oldest commands in
    the cmds list and cap it at max_pending_commands. 2) Start raising
    analyzer violations if the client has been using more than
    max_unknown_client_commands commands (default 10).

    Closes #3936

  * btest/pop3: Add somewhat more elaborate testing (Arne Welzel, Corelight)

    PCAP taken from here: https://tranalyzer.com/tutorial/pop and reference
    added to Traces/README.

7.1.0-dev.328 | 2024-09-18 19:10:43 +0200

  * Add enum value negative check (Evan Typanski, Corelight)

    There was one already at parse time, this adds a check later so that
    cases like overflows or internal enums with negative values get caught.

  * Fix port/enum values `SizeOf` not being a count (Evan Typanski, Corelight)

    Really, they both should be count. But, they were getting provided as an
    integer. Port is easy since it is backed by an unsigned value. Enums
    *should* be unsigned, but aren't. This doesn't address that, it just
    takes the other name for this operator (absolute value) and makes the
    enum value positive if it's negative.

    This fixes a case where using the size of operator on enum/port values
    in certain contexts (like the default parameter of a struct) would cause
    an internal error.

7.1.0-dev.324 | 2024-09-17 18:51:34 +0200

  * btest: Update baselines for removal-hooks addition (Arne Welzel, Corelight)

    The removal_hooks field exists in bare mode (seems fine) and moved within the
    connection record to earlier, so a bunch of baselines changed

  * gtpv1: Replace connection_state_remove() with RemovalHook (Arne Welzel, Corelight)

  * teredo: Replace connection_state_remove() with RemovalHook (Arne Welzel, Corelight)

    Remove overhead of unconditionally calling remove_teredo_connection()
    for *every* connection by installing a connection removal hook for only
    when state was allocated.

  * teredo: Move conn member from analyzer to encapsulation (Arne Welzel, Corelight)

    There's only a single instance of the Teredo analyzer. Mutating the conn
    member for every new packet and leaving it set after processing the
    packet is confusing. Move conn into TeredoEncapsulation instead, or pass
    it explicitly.

7.1.0-dev.319 | 2024-09-17 18:50:40 +0200

  * Update broker submodule (Arne Welzel, Corelight)

    ...to pin prometheus to 1.2.4

  * telemetry: Move callbacks to Zeek (Arne Welzel, Corelight)

    Now that we run callbacks on the main loop, we can move callback support
    for Counter and Gauge instances directly into Zeek and don't need to patch
    prometheus-cpp anymore.

  * auxil/prometheus-cpp: Pin to 1.2.4 (Arne Welzel, Corelight)

7.1.0-dev.315 | 2024-09-12 09:18:35 +0200

  * Update Dockerfile to avoid LegacyKeyValueFormat warning (MP-Corelight)

  * Fix potential endless loop in util::strreplace (Tim Wojtulewicz, Corelight)

7.1.0-dev.311 | 2024-09-12 08:52:02 +0200

  * Remove core.negative-time btest (Tim Wojtulewicz, Corelight)

    This test is failing with libpcap libpcap 1.10; for more information
    see https://github.com/zeek/zeek/issues/3921

7.1.0-dev.309 | 2024-09-12 08:51:08 +0200

  * Bump spicy-format to 0.16.2 (Johanna Amann, Corelight)

  * Spicy SSL: reformat with new version of spicy format (Johanna Amann, Corelight)

7.1.0-dev.305 | 2024-09-11 16:55:55 +0200

  * Spicy SSL analyzer:

    This commit adds an alternative Spicy-based SSL analyzer. It supports
    nearly the entire functionality of the current binpac analyzer, with
    the exception of DTLS and decryption.

    This currently is mostly for internal tests, or for Spicy testing purposes.
    There is no functional advantage to use the Spicy based analyzer - it does
    not have any additional features.

    It is currently gated behind a configure-time flag that needs to be provided
    to enable it (--enable-spicy-ssl).

7.1.0-dev.259 | 2024-09-09 13:24:45 +0200

  * script_opt/ZAM/IterInfo.h: Add missing Dict.h dependency (Arne Welzel, Corelight)

    Triggered when trying to remove some other includes elsewhere.

  * script_opt/ZAM: ZBody.h / Support.h: Cleanup includes, use forward declarations (Arne Welzel, Corelight)

    Not sure the IsAny() convenience functions were really worth the
    include zeek/Expr.h.

  * script_opt/ZAM/Profile: Remove Zeek header includes (Arne Welzel, Corelight)

    This can stand by itself.

  * script_opt: Extend Support.h to break include dependencies (Arne Welzel, Corelight)

    Rebuilding ZBody.cc is annoyingly slow with UBSAN or ASAN enabled. A rebuild
    is currently triggered when touching Manager.h files like logging/Manager.h,
    broker/Manager.h or file_analysis/Manager.h and various other headers that are
    included from there or directly in ZBody.h.

    Add tiny wrapper functions to Support.h that encapsulate the actual Zeek
    functionality and move the includes to Support.cc instead. This will
    cause frequent Support.cc rebuilds, but that is decently fast.

    This comes at the expense of one indirect function call for some ops, but
    its overhead should be negligible in comparison to a lookup for a
    connection or component or a log_mgr->Write() call.

  * script_opt: Do not include ZBody.h in ZAM/Validate.cc (Arne Welzel, Corelight)

    Touching broker/Manager.h or logging/Manager.h or anything that's
    included in ZBody.h results in a rebuild of a bunch of files in ZAM.
    Particularly, ZAM/Validate.cc takes much too long to recompile for
    me (45+ seconds) and apparently that's mostly by accident.

    Clean up the ZAM/Validate.cc includes to prevent this.

7.1.0-dev.252 | 2024-09-06 19:17:29 +0200

  * btest/ldap: Add regression test for #3919 (Arne Welzel, Corelight)

    This works as expected in master, it's just that we forgot to backport
    PR #3845 to 7.0.1. Add the PCAP from Martin anyhow.

    Closes #3919.

7.1.0-dev.250 | 2024-09-06 17:47:36 +0200

  * postgresql: Simplify SSL buffering and forwarding (Arne Welzel, Corelight)

    The ssl_sink can only be connected form the backend side, so don't
    overcomplicate it.

  * postgresql: Initial parser implementation (Arne Welzel, Corelight)

    This adds a protocol parser for the PostgreSQL protocol and a new
    postgresql.log similar to the existing mysql.log.

    This should be considered preliminary and hopefully during 7.1 and 7.2
    with feedback from the community, we can improve on the events and logs.
    Even if most PostgreSQL communication is encrypted in the real-world, this
    will minimally allow monitoring of the SSLRequest and hand off further
    analysis to the SSL analyzer.

    This originates from github.com/awelzel/spicy-postgresql, with lots of
    polishing happening in the past two days.

7.1.0-dev.247 | 2024-09-05 19:19:31 +0200

  * testing/external: Update private baselines (Arne Welzel, Corelight)

  * analyzer/syslog: Reformat with spicy-format (Arne Welzel, Corelight)

  * analyzer/finger: Reformat with spicy-format (Arne Welzel, Corelight)

  * scripts/spicy: Reformat with spicy-format (Arne Welzel, Corelight)

  * pre-commit: Add spicy-format (Arne Welzel, Corelight)

    This excludes the larger protocol analyzers. These should be formatted
    during the 7.2 cycle, shortly before 8.0.

7.1.0-dev.241 | 2024-09-04 14:53:19 -0700

  * Check for netbios to avoid reporting extra bad DNS opcodes (Tim Wojtulewicz, Corelight)

  * Add weird for unhandled opcodes in DNS analyzer (Tim Wojtulewicz, Corelight)

7.1.0-dev.238 | 2024-09-03 09:03:24 -0700

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

7.1.0-dev.236 | 2024-09-03 10:18:32 +0200

  * Remove pre-commit exclusions for clang-format (Benjamin Bannier, Corelight)

    We previously would ignore any files under `auxil/`. Since pre-commit
    only formats files tracked in this repo, and `auxil/` today contains
    only submodules this exclusion did not exclude anything; if however we
    ever end up adding C++ files under `auxil/` we would like them to be
    formatted consistently.

  * Bump clang-format (Benjamin Bannier, Corelight)

    This patch contains a bump of the configured clang-format version from
    17.0.3 to 18.1.8 and automatically generated C++ source updates. The
    main difference we are seeing from this is fixes for previously
    incomplete reformats.

7.1.0-dev.233 | 2024-09-03 10:02:59 +0200

  * Bump zeek-aux for zeek/zeek-aux#57 (Arne Welzel, Corelight)

7.1.0-dev.232 | 2024-08-30 18:56:24 +0200

  * http: fix password capture when enabled (Pierre Lalet)

    The current implementation would only log, if the password contains a
    colon, the part before the first colon (e.g., the password
    `password:password` would be logged as `password`).

    A test has been added to confirm the expected behaviour.

7.1.0-dev.230 | 2024-08-30 15:38:52 +0200

  * RunState: Drop broker_mgr->Active() usage (Arne Welzel, Corelight)

    ...and deprecate it. The communication_enabled flag doesn't really
    make a difference in how the IO loop works.

  * logging/WriterFrontend: No need for explicit CleanupWriteBuffer() (Arne Welzel, Corelight)

    Any pending records will be cleaned in the destructor of WriterFrontend
    and WriteBuffer, no need to do this explicitly.

  * logging: Switch index-assignment of raw pointers to emplace_back() (Arne Welzel, Corelight)

  * broker/logging: Change threading::Value** usage std::vector instead (Arne Welzel, Corelight)

    This allows to leverage automatic memory management, less allocations
    and using move semantics for expressing ownership.

    This breaks the existing logging and broker API, but keeps the plugin
    DoWrite() and HookLogWrite() methods functioning.

    It further changes ValToLogVal to return a threading::Value rather than
    a threading::Value*. The vector_val and set_val fields unfortunately
    use the same pointer-to-array-of-pointers approach. this can'tbe changed
    as it'd break backwards compatibility for plugin provided input readers
    and log writers.

  * threading/Value: Support move and copy constructors (Arne Welzel, Corelight)

7.1.0-dev.222 | 2024-08-27 13:29:12 +0100

  * Fix parsing of version field in SSLv2 client hello (Johanna Amann, Corelight)

    It turns out that, for probably a long time, we have reported an
    incorrect version when parsing an SSLv2 client hello. We always reported
    this as SSLv2, no matter which version the client hello actually
    contained.

    This bug probably went unnoticed for a long time, as SSLv2 is
    essentially unused nowadays, and as this field does not show up in the
    default logs.

    This was found due to a baseline difference when writing the Spicy SSL
    analyzer.

7.1.0-dev.219 | 2024-08-23 14:18:35 +0200

  * Analyzer: Do not add child analyzers when finished (Arne Welzel, Corelight)

    Depending on an analyzer's implementation, its Done() method may
    attempt to access analyzer or connection state when executing.
    When this happens in the destructor of the parent analyzer during
    the process of destructing a connection, this state may have been
    deleted, resulting in use-after-free crashes or worse memory
    corruption.

    The following cases have been observed in the wild for when this happens.

      * PIA matching during Done() for undelivered TCP data enables a Spicy
        based analyzer which in turn attempts to raise an analyzer violation
        during Done()->EndOfData().

      * Spicy analyzers attaching new analyzers during their Done() processing
        which in turn attempt to use TCP() (to call FindChild()) during Done()
        while the analyzer tree / connection is being destructed.

    The second scenario was previously found to happen in the HTTP analyzer
    and fixed with 6ef9423f3cff13e6c73f97eb6a3a27d6f64cc320.

    Plug these scenarios by short-circuiting AddChildAnalyzer() if the analyzer
    or connection have finished or are being finished.

7.1.0-dev.217 | 2024-08-23 09:40:53 +0200

  * TCP_Reassembler: Fix IsOrig() position in Match() call (Arne Welzel, Corelight)

    Found during a debug session with @rsmmr. Undelivered TCP data
    would only be matched for the responder and eol set to IsOrig().

  * Spicy: Register well-known ports through an event handler. (Robin Sommer, Corelight)

    This avoids the earlier problem of not tracking ports correctly in
    scriptland, while still supporting `port` in EVT files and `%port` in
    Spicy files.

    As it turns out we are already following the same approach for file
    analyzers' MIME types, so I'm applying the same pattern: it's one
    event per port, without further customization points. That leaves the
    patch pretty small after all while fixing the original issue.

  * Revert "Remove deprecated port/ports fields for spicy analyzers" (Robin Sommer, Corelight)

    This reverts commit 15d404dd191a723960e4efd956eec22739d3f1c2.

7.1.0-dev.211 | 2024-08-20 11:38:48 +0100

  * Make ssl_history work for SSLv2 handshakes/connections (Johanna Amann, Corelight)

    It turns out that the ssl_history field never was populated with C/S for
    SSLv2 connections, or connections using the SSLv2 handshake. In our
    testcases, the latter is especially common - with connections up to TLS1
    using the old SSLv2 client hello for backwards compatibility.

    This change resolves this issue. As the history is not by default
    enabled in a lot of locations, baseline impact is minor.

7.1.0-dev.209 | 2024-08-20 10:10:46 +0200

  * broker: Deprecate MakeEvent(ValPList*) (Arne Welzel, Corelight)

    The variadic broker messaging BIFs currently convert @ARGS@ into a
    ValPList before passing it on to MakeEvent(). This appears historic
    plumbing. Implement the same functionality using Span<const ValPtr>
    and do the extra copying in the now deprecated MakeEvent().

    Further, make passing a frame optional as not all callers may
    have one available.

  * Span: Remove deduction guideline for Iter, Iter, include cleanup (Arne Welzel, Corelight)

    We don't have a constructor for that, so that's confusing, also
    <array> isn't used.

  * ScriptOpt: Ensure global statements have non-null scope (Arne Welzel, Corelight)

    The ProfileFunc() logic assumed that GetScope() returned a non-nullptr.
    This holds except for the synthetic global statements function.

    Fix the latter and add an assert, also add a name to the type so it's
    easier to recognize in a debugger what's going on, otherwise the name
    is "".

    This was found by UBSAN due to it seeing the ->OrderedVars() call on a
    nullptr. Elsewhere, num_params == 0 shielded from that access and so
    didn't lead to crashes.

  * Update gen-zam submodule [nomail] [skip ci] (Tim Wojtulewicz, Corelight)

7.1.0-dev.202 | 2024-08-16 09:06:41 -0700

  * Remove unused wrapper packet analyzer (Jan Grashoefer, Corelight)

    This is a leftover from the migration to the packet analysis framework.
    The analyzer wrapped the original packet analysis code for comparison.

7.1.0-dev.200 | 2024-08-16 16:50:47 +0200

  * Add DNS TKEY event (Evan Typanski, Corelight)

7.1.0-dev.198 | 2024-08-16 12:10:00 +0200

  * simpler and more robust identification of function parameters for AST profiling (Vern Paxson, Corelight)

  * fixes to limit AST traversal in the face of recursive types (Vern Paxson, Corelight)

  * address some script optimization compiler warnings under Linux (Vern Paxson, Corelight)

  * fix for -O C++ construction of variable names that use multiple module namespaces (Vern Paxson, Corelight)

  * fix for script optimization of "opaque" values that are run-time constants (Vern Paxson, Corelight)

  * fix for script optimization of nested switch statements (Vern Paxson, Corelight)

  * script optimization fix for complex "in" expressions in conditionals (Vern Paxson, Corelight)

  * updates to typos allow-list reflecting ZAM regularization changes (Vern Paxson, Corelight)

  * BTest updates for ZAM regularization changes (Vern Paxson, Corelight)

  * convert new ZAM operations to use typed operands (Vern Paxson, Corelight)

  * complete migration of ZAM to use only public ZVal methods (Vern Paxson, Corelight)

  * "-O validate-ZAM" option to validate generated ZAM instructions (Vern Paxson, Corelight)

  * internal option to suppress control-flow optimization (Vern Paxson, Corelight)

  * exposing some functionality for greater flexibility in structuring run-time execution (Vern Paxson, Corelight)

  * rework ZAM compilation of type switches to leverage value switches (Vern Paxson, Corelight)

  * add tracking of control flow information (Vern Paxson, Corelight)

  * factoring of ZAM operation specifications into separate files (Vern Paxson, Corelight)

  * updates to ZAM operations / gen-zam regularization, other than the operations themselves (Vern Paxson, Corelight)

  * type-checking fix for vector-of-string operations (Vern Paxson, Corelight)

  * ZVal constructor for booleans (Vern Paxson, Corelight)

  * fix for nit in base/protocols/krb/main.zeek (Vern Paxson, Corelight)

  * mark functions skipped by ZAM compilation as such (Vern Paxson, Corelight)

  * fix for avoiding inadvertent interpreter errors in CallExpr::IsPure() (Vern Paxson, Corelight)

  * support for traversing ZAM code similar to AST traversal (Vern Paxson, Corelight)

  * run-time warnings for scripts compiled to C++ (Vern Paxson, Corelight)

  * allow C++ script compiler access to type internals (Vern Paxson, Corelight)

  * fixes for script optimization of coerce-to-any expressions (Vern Paxson, Corelight)

  * fix to correctly track whether a capture needs deep-copying (Vern Paxson, Corelight)

  * fix for -O report-C++ (Vern Paxson, Corelight)

  * support for more in-depth AST profiling (Vern Paxson, Corelight)

  * allow profiling without updating of hash values (Vern Paxson, Corelight)

  * ListVal method to clear the list to allow reusing w/o new construction (Vern Paxson, Corelight)

  * accessor for smart-pointer version of FileVal's value (Vern Paxson, Corelight)

7.1.0-dev.164 | 2024-08-15 10:30:37 +0200

  * Func: Add SetCapturesVec() (Arne Welzel, Corelight)

    Add an API to directly set captures_vec for use by C++ compilation. The
    current code keys off or asserts on ZAM stmts, making it difficult to
    leverage captures_vec in other contexts.

  * marked some recently added BTests as not suitable for -O gen-C++ testing (Vern Paxson, Corelight)

  * robustness improvements for -O gen-C++ generation of lambdas / "when"s (Vern Paxson, Corelight)

  * speedups for compilation of initializers in -O gen-C++ generated code (Vern Paxson, Corelight)

  * fixes for -O gen-C++ generation of floating point constants (Vern Paxson, Corelight)

  * -O gen-C++ fix for dealing with use of more than one module qualifier (Vern Paxson, Corelight)

  * header tweaks to provide gen-C++ script optimization with more flexibility (Vern Paxson, Corelight)

  * fix for script optimization of constants of type "opaque" (Vern Paxson, Corelight)

  * fix for script optimization of "in" operations (Vern Paxson, Corelight)

  * some minor tidying of -O gen-C++ sources (Vern Paxson, Corelight)

7.1.0-dev.152 | 2024-08-14 20:08:14 +0200

  * mysql: Implement and test COM_CHANGE_USER (Arne Welzel, Corelight)

    This reworks the parser such that COM_CHANGE_USER switches the
    connection back into the CONNECTION_PHASE so that we can remove the
    EXPECT_AUTH_SWITCH special case in the COMMAND_PHASE. Adds two pcaps
    produced with Python that actually do COM_CHANGE_USER as it seems
    not possible from the MySQL CLI.

  * mysql: Remove auth plugin state (Arne Welzel, Corelight)

    Now that we use CONNECTION_PHASE and interpret AuthSwitchRequest
    correctly, we can remove reliance on the plugin discrepancy to
    determine expectations.

  * GH-3880: mysql: Handle server connection phase separately from command phase (Arne Welzel, Corelight)

    This avoids interpreting an AuthSwitchRequest (0xfe) during the command
    phase as EOF_Packet.

    Thanks @AmazingPP.

    Closes #3880

7.1.0-dev.147 | 2024-08-13 12:37:17 +0200

  * Deprecate old unknown_protocol event signature (Jan Grashoefer, Corelight)

  * Use raw pointer for packet analyzer history (Jan Grashoefer, Corelight)

  * Update external baselines for analyzer history (Jan Grashoefer, Corelight)

  * Update btest baselines for analyzer history (Jan Grashoefer, Corelight)

  * Add packet analyzer history (Jan Grashoefer, Corelight)

7.1.0-dev.141 | 2024-08-12 11:07:32 +0200

  * spicyz: Add back message about removed support for port / ports in evt (Arne Welzel, Corelight)

    spicy-dhcp, spicy-http and spicy-dns all have this still in their .evt files,
    so it seems popular. Be more helpful than "unexpected token" to users.

7.1.0-dev.139 | 2024-08-09 20:57:23 +0200

  * rule-parse: Remove id_to_str() lookup to squelch coverity warning (Arne Welzel, Corelight)

    Coverity didn't like that id_to_str() allocates memory and we didn't
    free it. Remote its usage wholesale.

  * GH-3774: btest: Skip core.script-args under TSAN (Arne Welzel, Corelight)

    TSAN may re-execute the executable when the memory layout doesn't
    fullfill requirements, causing argument confusion when that happens.

    Closes #3774.

  * Update zeekctl submodule [nomail] (Tim Wojtulewicz, Corelight)

7.1.0-dev.133 | 2024-08-08 10:44:31 +0200

  * Update zeekctl submodule (Tim Wojtulewicz)

  * Add note to NEWS about the removal of OpaqueVal::DoSerialize and OpaqueVal::DoUnserialize (Tim Wojtulewicz, Corelight)

7.1.0-dev.129 | 2024-08-07 12:20:22 -0700

  * Remove deprecated port/ports fields for spicy analyzers (Tim Wojtulewicz, Corelight)

  * Remove deprecated Cluster::Node::interface field (Tim Wojtulewicz, Corelight)

  * Remove deprecated signature definition format (Tim Wojtulewicz, Corelight)

  * Return an error if GLOBAL:: prefix is used (Tim Wojtulewicz, Corelight)

  * Remove deprecated BloomFilter serialization methods (Tim Wojtulewicz, Corelight)

  * Remove deprecated OpaqueVal serialization methods (Tim Wojtulewicz, Corelight)

  * Remove deprecated DECLARE_OPAQUE_VALUE macro (Tim Wojtulewicz, Corelight)

  * Make TypePtr::Capture member variables private (Tim Wojtulewicz, Corelight)

    The public versions were marked as deprecated for 7.0, and accessors
    should be used to manage them now.

  * Remove deprecated Trigger constructor (Tim Wojtulewicz, Corelight)

  * Remove deprecated Controller::auto_assign_ports and Controller::auto_assign_start_port (Tim Wojtulewicz, Corelight)

  * Remove deprecated load-balacing policy script (Tim Wojtulewicz, Corelight)

  * Remove deprecated prometheus telemetry policy script (Tim Wojtulewicz, Corelight)

  * Remove deprecated policy/tuning/default package (Tim Wojtulewicz, Corelight)

  * Remove deprecated time machine settings (Tim Wojtulewicz, Corelight)

  * Remove deprecated json NullDoubleWriter class (Tim Wojtulewicz, Corelight)

  * Remove deprecated modbus event definitions (Tim Wojtulewicz, Corelight)

  * Remove Connection::AppendAddl (Tim Wojtulewicz, Corelight)

  * Remove STMT_ANY statement type (Tim Wojtulewicz, Corelight)

  * Remove EventRegistry::Used and EventRegistry::SetUsed (Tim Wojtulewicz, Corelight)

7.1.0-dev.109 | 2024-08-07 14:10:54 +0200

  * ldap: Promote uint8 to uint64 before shifting (Arne Welzel, Corelight)

    Relates to zeek/spicy#1829

7.1.0-dev.107 | 2024-08-07 11:43:15 +0200

  * ci/ubuntu-24.04: Use ccache 4.10.2 (Arne Welzel, Corelight)

    The ccache version shipped with Ubuntu 24.04 does not yet recognize
    --fprofile-update=atomic, install one that does.

    Now that the asan_sanitizer build also includes building Spicy and
    running the spicyz test suite, ccache is quite important.

    Reference ccache/ccache#1408 and zeek/zeek#3777.

7.1.0-dev.105 | 2024-08-07 10:54:10 +0200

  * telemetry/Manager: Check RegisterFd() return value (Arne Welzel, Corelight)

    Please coverity.

  * telemetry/Manager: Track sent_in and sent_out totals without callback (Arne Welzel, Corelight)

    For terminated threads, the totals would go down once the threads are
    removed, which isn't great. Move tracking of sent in and sent out
    messages from callback to explicit `Inc()` calls.

    Also fixes total_messages_in_metric being initialized twice rather
    than total_messages_out_metric.

  * threading/Manager: Switch inf bucket from infinity() to max() (Arne Welzel, Corelight)

    For uint64_t, std::numeric_limits<T>::has_infinity is false and infinity()
    actually returns 0. Use uint64_t's max() instead. We could cast to double
    and use the double infinity, but this seems reasonable, too.

    This was found while trying to provoke some pending messages and being
    confused why all but the "inf" bucket increased.

  * threading/Manager: "lt" to "le" and do not break (Arne Welzel, Corelight)

    The buckets are specified as lower-equal (changed from lower-than now),
    which means we shouldn't break: The larger "le" bucket contains all previous
    buckets, too. The "inf" bucket represents the current number of threads.

    For example, with a total of 10 threads, 5 threads with 0 messages pending,
    another 4 threads with 50 messages, and on with 2000 messages, the metrics
    would end end up as follows:

        pending_buckets{le=1}      = 5
        pending_buckets{le=10}     = 5
        pending_buckets{le=100}    = 9
        pending_buckets{le=1000}   = 9
        pending_buckets{le=10000}  = 10
        pending_buckets{le=inf}    = 10

    This might be strange initially, but aligns with the Prometheus
    histogram approach (though we're using gauges here).

7.1.0-dev.99 | 2024-08-06 20:08:37 +0200

  * Bump auxil/spicy to latest development snapshot (Arne Welzel, Corelight)

  * spicy/runtime-support: Switch ParameterMismatch::_fmt to static (Arne Welzel, Corelight)

    UBSAN's vptr sanitize isn't happy with the call to _fmt()
    in its member initializer list.

        $ zeek -r Traces/ssh/single-conn.trace .tmp/spicy.event-args-mismatch/test.hlto .tmp/spicy.event-args-mismatch/event-args-mismatch.zeek
        <...>/src/include/zeek/spicy/runtime-support.h:80:29: runtime error: member call on address 0x511000369540 which does not point to an object of type 'zeek::spicy::rt::ParameterMismatch'
        0x511000369540: note: object has invalid vptr
         00 00 00 00  be be be be be be be be  be be be be be be be be  be be be be be be be be  be be be be
                      ^~~~~~~~~~~~~~~~~~~~~~~
                      invalid vptr
            #0 0x7f9c9977b019 in zeek::spicy::rt::ParameterMismatch::ParameterMismatch(std::basic_string_view<char, std::char_traits<char>>, zeek::IntrusivePtr<zeek::Type> const&, std::basic_string_view<char, std::char_traits<char>>) <...>/src/include/zeek/spicy/runtime-support.h:80:29
            #1 0x7f9c9977a6a2 in zeek::spicy::rt::to_val(hilti::rt::Bytes const&, zeek::IntrusivePtr<zeek::Type> const&) <...>/src/include/zeek/spicy/runtime-support.h:562:15

  * coverage/lcov_html: Ignore testing/btest/.tmp (Arne Welzel, Corelight)

    gcda/gcno files in the btest/.tmp directory are from .htlo files
    referencing ephemeral cc files. No need to include these.

  * cirrus: Do not disable Spicy for sanitizer builds (Arne Welzel, Corelight)

  * ldap: Avoid unset m$opcode (Arne Welzel, Corelight)

    Initial fuzzing caused a bind response to arrive before a bind request,
    resulting in an unset field expression error:

        expression error in base/protocols/ldap/main.zeek, line 270: field value missing (LDAP::m$opcode)

    Prevent this by ensuring m$opcode is set and raising instead.

  * GH-3860: fuzzers: Add LDAP fuzzing (Arne Welzel, Corelight)

    LDAP supports both, UDP and TCP as separate analyzers. The corpus
    is identical, however. Started to hit the TLS analyzer fairly
    quickly, too.

    Closes #3860

7.1.0-dev.92 | 2024-08-06 09:01:40 -0700

  * Don't install empty ZAM directories (Tim Wojtulewicz, Corelight)

  * btest/spicy: Make replaces-conflicts trigger replaces code path (Arne Welzel, Corelight)

    The current test attempts to instantiate two spicy::SSH_1 protocol
    analyzers in the .evt file. The intention likely was to use two
    distinct protocol analyzer both trying to replace the builtin SSH
    analyzer.

    Coincidentally, fixing this happens to workaround TSAN errors tickled
    by the FatalError() call while loading the .hlto with two identically
    named analyzers.

        $ cat .tmp/spicy.replaces-conflicts/output
        error: redefinition of protocol analyzer spicy::SSH_1
        ThreadSanitizer: main thread finished with ignores enabled
          One of the following ignores was not ended (in order of probability)
          Ignore was enabled at:
            #0 __llvm_gcov_init __linker___d192e45c25d5ee23-484d3e0fc2caf5b4.cc (ssh.hlto+0x34036) (BuildId: 091934ca4da885e7)
            #1 __llvm_gcov_init __linker___d192e45c25d5ee23-484d3e0fc2caf5b4.cc (ssh.hlto+0x34036) (BuildId: 091934ca4da885e7)
            ...

    I was tempted to replace FatalError() with Error() and rely on
    zeek-setup.cc's early exiting on any reporter errors, but this
    seems easier for now.

    Relates to #3865.

7.1.0-dev.87 | 2024-08-05 14:39:56 -0700

  * Remove some unnecessary #includes (Tim Wojtulewicz, Corelight)

  * Avoid capturing 'this' for callback in telemetry::Manager (Tim Wojtulewicz, Corelight)

  * Make telemetry metrics out of MsgThread statistics (Tim Wojtulewicz, Corelight)

  * Split cpu time metric into user/system components like prof.log (Tim Wojtulewicz, Corelight)

    The total can be calculated from the two parts via Prometheus/Grafana
    if desired, so it's more informative to pass them as separate parts.

  * Move broker statistics to be telemetry metrics (Tim Wojtulewicz, Corelight)

  * Add timer counts as telemetry metrics (Tim Wojtulewicz, Corelight)

  * Move thread manager stats to telemetry metric (Tim Wojtulewicz, Corelight)

  * Add extra metrics to session_mgr (Tim Wojtulewicz, Corelight)

    - Sessions killed by activity
    - Current number of sessions across all types

  * Move dns_mgr stats to telemetry instruments (Tim Wojtulewicz, Corelight)

  * Move trigger stats to telemetry instruments (Tim Wojtulewicz, Corelight)

  * Move pulling of global state inside 'expensive' check for stats (Tim Wojtulewicz, Corelight)

7.1.0-dev.74 | 2024-08-02 15:49:40 -0700

  * Process metric callbacks from the main-loop thread (Tim Wojtulewicz, Corelight)

    This avoids the callbacks from being processed on the worker thread
    spawned by Civetweb. It fixes data race issues with lookups involving
    global variables, amongst other threading issues.

  * CI: Use 16GB of memory for FreeBSD builds (Tim Wojtulewicz, Corelight)

7.1.0-dev.68 | 2024-07-26 10:09:48 -0700

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

    This in particular pulls in a fix for zeek/spicy#1808.

7.1.0-dev.66 | 2024-07-26 15:14:36 +0200

  * GH-3853: ldap: Recognize SASL+SPNEGO+NTLMSSP (Arne Welzel, Corelight)

    The ctu-sme-11-win7ad-1-ldap-tcp-50041.pcap file was harvested
    from the CTU-SME-11 (Experiment-VM-Microsoft-Windows7AD-1) dataset
    at https://zenodo.org/records/7958259 (DOI 10.5281/zenodo.7958258).

    Closes #3853

7.1.0-dev.63 | 2024-07-25 09:19:20 -0700

  * Use accessor to reach into X509_ALGOR (Theo Buehler)

    Despite already having an accessor, X509_ALGOR wasn't made opaque
    during OpenSSL 1.1.0 development. It would be nice if this could be
    fixed at some point, so avoid reaching into that struct by using the
    accessor

7.1.0-dev.61 | 2024-07-25 12:02:51 +0200

  * mysql: Simplify length computation (Arne Welzel, Corelight)

    Thanks Tim!

  * mysql: Improve date and time parsing (Fupeng Zhao)

  * btest/mysql: Clean query-attr.pcapng (Arne Welzel, Corelight)

    It contained some unrelated IntelliJ traffic.

  * mysql: Support non-string query attributes (Arne Welzel, Corelight)

    The query attributes aren't exposed to script layer right now, but this
    should at least parse over them once encountered and some fixups.

  * btest/mysql: Add pcap with non-string query attributes (Arne Welzel, Corelight)

    Pcap was generated as follows. Doesn't seem wireshark even parses
    this properly right now.

        with common.get_connection() as c:
            with c.cursor() as cur:
                date1 = datetime.date(1987, 10, 18)
                datetime1 = datetime.datetime(1990, 9, 26, 12, 13, 14)
                cur.add_attribute("number1", 42)
                cur.add_attribute("string1", "a string")
                cur.add_attribute("date1", date1)
                cur.add_attribute("datetime1", datetime1)
                cur.execute("SELECT version()")
                result = cur.fetchall()
                print("result", result)

  * mysql: Introduce mysql_ssl_request event (Arne Welzel, Corelight)

    This should've been added with fa48c885 for completion. Do it now.
    The MySQL spec calls it SSLRequest packet, so keep SSL in the name for
    consistency.

  * mysql: Fix EOFIfLegacyThenResultSet (Arne Welzel, Corelight)

    Only expect a result next if an EOF was consumed.

  * mysql: Add data parameter to mysql_auth_plugin (Arne Welzel, Corelight)

    This may contain salt from the server or a hashed password from the client.

  * mysql: Add mysql_auth_plugin, mysql_auth_more_data and mysql_auth_switch_request events (Arne Welzel, Corelight)

    Remove caching_sha2_password parsing/state from the analyzer and implement
    the generic events. If we actually want to peak into the authentication
    mechanism, we could write a separate analyzer for it. For now, treat it
    as opaque values that are exposed to script land.

    The added tests show the --get-server-public-key in use where
    mysql_auth_more_data contains an RSA public key.

  * mysql: AuthSwitchRequest: &enforce a 0xfe / 254 status (Arne Welzel, Corelight)

  * mysql: Make auth_plugin_ a std::string (Arne Welzel, Corelight)

  * mysql: Fix auth_plugin_data_part2 length computation (Arne Welzel, Corelight)

  * Refactored connection phase state handling (Fupeng Zhao)

    Added `ConnectionExpected` enum for expected packet types during the connection phase.

  * Add support for "auth switch" and "query attrs" (Fupeng Zhao)

    Also fix the issue where Resultset could not correctly distinguish between EOF_Packet and OK_Packet.

  * Add support for parsing the "caching_sha2_password" auth plugin (Fupeng Zhao)

7.1.0-dev.45 | 2024-07-24 15:28:59 -0700

  * Update binpac submodule to better format output code [nomail] (Tim Wojtulewicz, Corelight)

7.1.0-dev.43 | 2024-07-24 13:25:52 -0700

  * Update 7.0 NEWS with blurb about multi-PDU parsing causing increased load [nomail] [skip ci] (Tim Wojtulewicz, Corelight)

  * Fix handling of zero-length SMB2 error responses (Tim Wojtulewicz, Corelight)

7.1.0-dev.40 | 2024-07-24 11:18:03 +0200

  * minor optimization of boolean comparisons (Vern Paxson, Corelight)

  * GH-3839: fix & regression test for GH-3839 (spurious warnings for "when" constructs) (Vern Paxson, Corelight)

7.1.0-dev.37 | 2024-07-23 19:18:37 -0700

  * Bump zeek-testing-cluster to reflect deprecation of prometheus.zeek (Christian Kreibich, Corelight)

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

7.1.0-dev.34 | 2024-07-23 08:54:23 -0700

  * Update Mozilla CA list and CT list (Johanna Amann, Corelight)

7.1.0-dev.32 | 2024-07-23 08:51:11 -0700

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

7.1.0-dev.30 | 2024-07-23 12:38:54 +0200

  * ldap: Add heuristic for wrap tokens (Arne Welzel, Corelight)

    Instead of dissecting the GSSAPI handshake, add another heuristic
    into MaybeEncrypted to check for the WRAP token identifier.

    After this change, the pcap on the following ticket is processed
    nicely: https://gitlab.com/wireshark/migration-test/-/issues/9398

  * ldap: Ignore ec/rrc for sealed wrap tokens (Arne Welzel, Corelight)

    It shouldn't matter for the encrypted payload that we'll
    just consume and ignore.

  * ldap: Add LDAP sample with SASL-SRP mechanism (Arne Welzel, Corelight)

    This is what @dopheide-esnet actually saw. Produced with a custom
    cyrus-sasl and openldap build :-(

  * ldap: Reintroduce encryption after SASL heuristic (Arne Welzel, Corelight)

    @dopheide-esnet provided sample captures where SASL SRP is used as
    a SASL mechanism and the follow-up LDAP messages are encrypted. It's
    not clear how to determine whether encryption will or will not happen,
    so re-add a heuristic to determine this based on the first byte of
    the first message *after* the successful bindResponse handshake. If
    that byte is 0x30, assume cleartext.

    I haven't been able to produce such pcaps, unfortunately, but the
    cleartext path is tested via the existing sasl-ntlm.pcap.

  * ldap: Fix assuming GSS-SPNEGO for all bindResponses (Arne Welzel, Corelight)

    In retrospect that's an obvious bug.

  * ldap: Implement extended request/response and StartTLS support (Arne Welzel, Corelight)

    PCAP was produced with a local OpenLDAP server configured to support StartTLS.

    This puts the Zeek calls into a separate ldap_zeek.spicy file/module
    to separate it from LDAP.

7.1.0-dev.23 | 2024-07-23 10:02:52 +0200

  * telemetry: Deprecate prometheus.zeek policy script (Arne Welzel, Corelight)

    With Cluster::Node$metrics_port being optional, there's not really
    a need for the extra script. New rule, if a metrics_port is set, the
    node will attempt to listen on it.

    Users can still redef Telemetry::metrics_port *after*
    base/frameworks/telemetry was loaded to change the port defined
    in cluster-layout.zeek.

  * Update broker submodule [nomail] (Tim Wojtulewicz, Corelight)

7.1.0-dev.20 | 2024-07-19 19:51:12 +0200

  * GH-3836: input/Manager: Improve type checks of record fields with type any (Arne Welzel, Corelight)

    Calling AsRecordType() or AsFunc() on a Val of type any isn't safe.

    Closes #3836

7.1.0-dev.18 | 2024-07-17 15:37:12 -0700

  * Bump zeek-testing-cluster to pull in tee SIGPIPE fix (Christian Kreibich, Corelight)

7.1.0-dev.16 | 2024-07-17 16:45:13 +0200

  * ldap: Remove MessageWrapper with magic 0x30 searching (Arne Welzel, Corelight)

    This unit implements a heuristic to search for the 0x30 sequence
    byte if Message couldn't readily be parsed. Remove it with the
    idea of explicit and predictable support for SASL mechanisms.

  * ldap: Harden parsing a bit (Arne Welzel, Corelight)

    ASN1Message(True) may go off parsing arbitrary input data as
    "something ASN.1" This could be GBs of octet strings or just very
    long sequences. Avoid this by open-coding some top-level types expected.

    This also tries to avoid some of the &parse-from usages that result
    in unnecessary copies of data.

    Adds a locally generated PCAP with addRequest/addResponse that we
    don't currently handle.

  * ldap: Handle integrity-only KRB wrap tokens (Arne Welzel, Corelight)

    Mostly staring at the PCAPs and opened a few RFCs. For now, only if the
    MS_KRB5 OID is used and accepted in a bind response, start stripping
    KRB5 wrap tokens for both, client and server traffic.

    Would probably be nice to forward the GSS-API data to the analyzer...

    Closes zeek/spicy-ldap#29.

7.1.0-dev.12 | 2024-07-16 10:16:02 -0700

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

    This patch bump Spicy to the latest development snapshot. This
    introduces a backwards-incompatible change in that it removes support
    for a never officially supported syntax to specify unit fields (so I
    would argue: not strictly a breaking change).

7.1.0-dev.10 | 2024-07-12 16:02:22 -0700

  * CI: Set FETCH_CONTENT_FULLY_DISCONNECTED flag for configure (Tim Wojtulewicz, Corelight)

  * Fix a broken merge (Tim Wojtulewicz, Corelight)

    I merged an old version of the branch on accident and then merged the right
    one over top of it, but git ended up including both versions. This fixes
    that mistake.

7.1.0-dev.6 | 2024-07-12 09:51:39 -0700

  * Do not emit hook files for builtin modules (Benjamin Bannier, Corelight)

    We would previously emit a C++ file with hooks for at least the builtin
    `spicy` module even though that module like any other builtin module
    never contains implementations of hooks for types in user code.

    This patch prevents modules with skipped implementations (such as our
    builtin modules) from being added to the compilation which prevents
    generating their hook files.

7.1.0-dev.2 | 2024-07-12 09:46:34 -0700

  * Fix warning about grealpath when running 'make dist' on Linux (Tim Wojtulewicz, Corelight)

7.0.0-dev.467 | 2024-07-11 12:14:52 -0700

  * Update the scripts.base.frameworks.telemetry.internal-metrics test (Christian Kreibich, Corelight)

  * Revert "Temporarily disable the scripts/base/frameworks/telemetry/internal-metrics btest" (Christian Kreibich, Corelight)

  * Bump Broker to pull in new Prometheus support and pass in Zeek's registry (Dominik Charousset and Christian Kreibich, Corelight)

7.0.0-dev.461 | 2024-07-10 18:45:36 +0200

  * Extend btest for logging of disabled analyzers (Jan Grashoefer, Corelight)

  * Add logging of disabled analyzers to analyzer.log (Jan Grashoefer, Corelight)

7.0.0-dev.458 | 2024-07-10 11:05:19 +0200

  * testing/btest: Default to HILTI_JIT_PARALLELISM=1 (Arne Welzel, Corelight)

    This is a rework of b59bed9d06f2257a6b4f6c6408b8363a277eebac moving
    HILTI_JIT_PARALLELISM=1 into btest.cfg to make it default applicable to
    btest -j users (and CI).

    The background for this change is that spicyz may spawn up to nproc compiler
    instances by default. Combined with btest -j, this may be nproc x nproc
    instances worst case. Particularly with gcc, this easily overloads CI or
    local systems, putting them into hard-to-recover-from thrashing/OOM states.

    Exporting HILTI_JIT_PARALLELISM in the shell allows overriding.

  * Revert "CI: Use ccache and a single CPU when building spicy analyzers for btests" (Arne Welzel, Corelight)

    This reverts commit b59bed9d06f2257a6b4f6c6408b8363a277eebac.

7.0.0-dev.454 | 2024-07-09 15:12:19 -0700

  * CI: Use ccache and a single CPU when building spicy analyzers for btests (Tim Wojtulewicz, Corelight)

7.0.0-dev.452 | 2024-07-09 14:53:41 -0700

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

7.0.0-dev.450 | 2024-07-09 10:15:13 -0700

  * minor script optimization updates to reflect recent changes, Coverity findings (Vern Paxson, Corelight)

7.0.0-dev.447 | 2024-07-09 17:10:16 +0200

  * Fix for --display-cmake in configure (cknill)
    Moved build directory creation further down in the script so that --display-cmake
    has a chance to happen before build tree setup.

7.0.0-dev.445 | 2024-07-09 00:07:25 -0700

  * Management framework: bump cluster testsuite to pull in telemetry tests (Christian Kreibich, Corelight)

  * Management framework: bump zeek-client (Christian Kreibich, Corelight)

  * Management framework: augment deployed configs with instance IP addresses (Christian Kreibich, Corelight)

  * Management framework: add auto-enumeration of metrics ports (Christian Kreibich, Corelight)

  * Management framework: propagate metrics port from agent (Christian Kreibich, Corelight)

  * Management framework: add metrics port in management & Supervisor node records (Christian Kreibich, Corelight)

  * Harden the telemetry manager against unset Telemetry::metrics_address (Christian Kreibich, Corelight)

  * Comment-only tweaks for telemetry-related settings. (Christian Kreibich, Corelight)

  * Update submodules [nomail] (Tim Wojtulewicz, Corelight)

7.0.0-dev.434 | 2024-07-04 17:34:23 +0200

  * CMakeLists: Ensure Threads::Threads target exists (Arne Welzel, Corelight)

    Fix failure on OSX with CMake 3.30 complaining about missing
    Threads::Threads target.

  * ci/macos: Only use sonoma image (Arne Welzel, Corelight)

7.0.0-dev.431 | 2024-07-04 17:22:42 +0200

  * Bump auxil/zeek-aux (Benjamin Bannier, Corelight)

7.0.0-dev.429 | 2024-07-04 15:15:56 +0200

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

7.0.0-dev.427 | 2024-07-04 13:35:25 +0200

  * GH-3789: HTTP: Implement FlipRoles() (Arne Welzel, Corelight)

    When Zeek flips roles of a HTTP connection subsequent to the HTTP analyzer
    being attached, that analyzer would not update its own ContentLine analyzer
    state, resulting in the wrong ContentLine analyzer being switched into
    plain delivery mode.

    In debug builds, this would result in assertion failures, in production
    builds, the HTTP analyzer would receive HTTP bodies as individual header
    lines, or conversely, individual header lines would be delivered as a
    large chunk from the ContentLine analyzer.

    PCAPs were generated locally using tcprewrite to select well-known-http ports
    for both endpoints, then editcap to drop the first SYN packet.

    Kudos to @JordanBarnartt for keeping at it.

    Closes #3789

  * ContentLineAnalyzer: Add getter for skip_partial (Arne Welzel, Corelight)

7.0.0-dev.423 | 2024-07-02 15:35:36 -0700

  * Remove the Supervisor's internal ClusterEndpoint struct. (Christian Kreibich, Corelight)

  * Provide a script-layer equivalent to Supervisor::__init_cluster(). (Christian Kreibich, Corelight)

7.0.0-dev.420 | 2024-07-02 14:46:29 -0700

  * Update NEWS file to cover JSON enhancements (Christian Kreibich, Corelight)

  * Support JSON roundtripping via to_json()/from_json() for patterns (Christian Kreibich, Corelight)

  * Support table deserialization in from_json() (Christian Kreibich, Corelight)

  * Support map-based definition of ports in from_json() (Christian Kreibich, Corelight)

  * Document the field_escape_pattern in the to_json() BiF (Christian Kreibich, Corelight)

7.0.0-dev.414 | 2024-07-02 19:42:10 +0200

  * Bump cmake submodule [nomail] (Arne Welzel, Corelight)

7.0.0-dev.413 | 2024-07-02 14:41:27 +0200

  * threading/Manager: Warn if threads are added after termination (Arne Welzel, Corelight)

    The core.file-analyzer-violation test showed that it's possible to
    create new threads (log writers) when Zeek is in the process of
    terminating. This can result in the IO manager's deconstructor
    deleting IO sources for threads that are still running.

    This is sort of a scripting issue, so for now log a reporter warning
    when it happens to have a bit of a bread-crumb what might be
    going on. In the future it might make sense to plug APIs with
    zeek_is_terminating().

  * iosource/Manager: Reap dry sources while computing timeout (Arne Welzel, Corelight)

    Avoids looping over the sources vector twice and should result
    in the same behavior.

  * GH-3682: threading/MsgThread: Decouple IO source and thread lifetimes (Arne Welzel, Corelight)

    MsgThread acting as an IO source can result in the situation where the
    threading manager's heartbeat timer deletes a finished MsgThread instance,
    but at the same time this thread is in the list of ready IO sources the
    main loop is currently processing.

    Fix this by decoupling the lifetime of the IO source part and properly
    registering as lifetime managed IO sources with the IO manager.

    Fixes #3682

  * iosource/Manager: Do not manage lifetime of pkt_src (Arne Welzel, Corelight)

    Now that dry sources are properly reaped and freed, an offline packet
    source would be deleted once dry, resulting in GetPktSrc() returning
    a wild pointer. Don't manage the packet source lifetime and instead
    free it during Manager destruction.

  * iosource/Manager: Honor manage_lifetime and dont_count for short-lived IO sources (Arne Welzel, Corelight)

    If an IO source is registered and becomes dry at runtime, the IO
    manager would not honor its manage_lifetime or dont_count attribute
    during collection, resulting in memory leaks.

    This probably hasn't mattered so far as there's no IO sources registered
    in-tree at runtime using manage_lifetime=true.

7.0.0-dev.407 | 2024-06-26 13:16:10 +0200

  * coverage/lcov_html: Allow missing coveralls token (Arne Welzel, Corelight)

    This is a fixup for 0cd023b83919fca7bfac55e75a41f724e820fd26 which
    currently causes ASAN coverage builds to fail for non-master branches
    when due to a missing COVERALLS_REPO_TOKEN.

    Instead of bailing out for non-master branches, pass `--dry-run` to the
    coveralls-lcov invocation to test more of the script.

7.0.0-dev.405 | 2024-06-25 13:35:41 +0200

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

7.0.0-dev.403 | 2024-06-25 11:29:58 +0200

  * NEWS: Add entry about FileExtractDir (Arne Welzel, Corelight)

  * Update zeekctl submodule (Arne Welzel, Corelight)

  * zeek-testing-private: Update baseline, after merge (Arne Welzel, Corelight)

7.0.0-dev.399 | 2024-06-24 11:19:14 +0200

  * script optimization for record operations sourced (in part) from other records (Vern Paxson, Corelight)

7.0.0-dev.397 | 2024-06-20 18:16:51 -0700

  * zeek-testing-private: Update baseline (Arne Welzel, Corelight)

  * ssh: Revert half-duplex robustness (Arne Welzel, Corelight)

    This reverts part of commit a0888b7e36308d241f4c62b42715a94d499aab23 due
    to inhibiting analyzer violations when parsing non SSH traffic when
    the &restofdata path is entered.

    See additional discussion here:
    https://github.com/zeek/zeek/pull/3781
    https://github.com/zeek/zeek/pull/3720/files#r1636314375

7.0.0-dev.394 | 2024-06-20 17:28:11 -0700

  * Add "zeek/" prefixes to includes in spicy/runtime-support.h (Michael Dopheide)

7.0.0-dev.392 | 2024-06-20 15:51:42 +0200

  * Spicy: Provide runtime API to access Zeek-side globals. (Robin Sommer, Corelight)

    This allows to read Zeek global variables from inside Spicy code. The
    main challenge here is supporting all of Zeek's data type in a
    type-safe manner.

    The most straight-forward API is a set of functions
    `get_<type>(<id>)`, where `<type>` is the Zeek-side type
    name (e.g., `count`, `string`, `bool`) and `<id>` is the fully scoped
    name of the Zeek-side global (e.g., `MyModule::Boolean`). These
    functions then return the corresponding Zeek value, converted in an
    appropriate Spicy type. Example:

        Zeek:
            module Foo;

            const x: count = 42;
            const y: string = "xxx";

        Spicy:
            import zeek;

            assert zeek::get_count("Foo::x") == 42;
            assert zeek::get_string("Foo::y") == b"xxx"; # returns bytes(!)

    For container types, the `get_*` function returns an opaque types that
    can be used to access the containers' values. An additional set of
    functions `as_<type>` allows converting opaque values of atomic
    types to Spicy equivalents. Example:

        Zeek:
            module Foo;

            const s: set[count] = { 1, 2 };
            const t: table[count] of string = { [1] = "One", [2] = "Two" }

        Spicy:

            # Check set membership.
            local set_ = zeek::get_set("Foo::s");
            assert zeek::set_contains(set_, 1) == True

            # Look up table element.
            local table_ = zeek::get_table("Foo::t");
            local value = zeek::table_lookup(t, 1);
            assert zeek::as_string(value) == b"One"

    There are also functions for accessing elements of Zeek-side vectors
    and records.

    If any of these `zeek::*` conversion functions fails (e.g., due to a
    global of that name not existing), it will throw an exception.

    The documentation has more information on all of this.

    Design considerations:

        - We support only reading Zeek variables, not writing. This is
          both to simplify the API, and also conceptually to avoid
          offering backdoors into Zeek state that could end up with a very
          tight coupling of Spicy and Zeek code.

        - We accept that a single access might be relatively slow due to
          name lookup and data conversion. This is primarily meant for
          configuration-style data, not for transferring lots of dynamic
          state over.

        - In that spirit, we don't support deep-copying complex data types
          from Zeek over to Spicy. This is (1) to avoid performance
          problems when accidentally copying large containers over,
          potentially even at every access; and (2) to avoid the two sides
          getting out of sync if one ends up modifying a container without
          the other being able to see it.

  * Spicy: Reformat `zeek.spicy` with `spicy-format`. (Robin Sommer, Corelight)

  * Spicy: Extend exception hierarchy. (Robin Sommer, Corelight)

    We move the current `TypeMismatch` into a  new `ParameterMismatch`
    exception that's derived from a more general `TypeMismatch` now that
    can also be used for other, non-parameter mismatches.

7.0.0-dev.387 | 2024-06-18 10:52:25 +0200

  * CMakeLists: Disable -Werror for 3rdparty/sqlite3.c (Arne Welzel, Corelight)

    We package vanilla sqlite from upstream and on Fedora 40 with sqlite 3.46
    there's the following compiler warning:

        In function 'sqlite3Strlen30',
            inlined from 'sqlite3ColumnSetColl' at
            ../../src/3rdparty/sqlite3.c:122105:10:
            ../../src/3rdparty/sqlite3.c:35003:28: error: 'strlen' reading 1 or more bytes from a region of size 0 [-Werror=stringop-overread]
        35003 |   return 0x3fffffff & (int)strlen(z);
              |                            ^~~~~~~~~
        In function 'sqlite3ColumnSetColl':

    Disabling -Werror on sqlite3.c seems sensible given we have little
    control over that code.

  * Bump zeek-3rdparty to pull in sqlite move to 3.46 (Christian Kreibich, Corelight)

    This avoids a compiler warning/error on Fedora 40.

  * CI: drop Fedora 38, add 40 (Christian Kreibich, Corelight)

7.0.0-dev.383 | 2024-06-14 13:50:05 +0200

  * GH-3783: Spicy: Disallow repeating replacements of the same analyzer. (Robin Sommer, Corelight)

    We now reject EVT files that attempt to replace the same built-in
    analyzer multiple times as doing so would be ill-defined and not very
    intuitive in what exactly it means.

  * Bump Spicy. (Robin Sommer, Corelight)

7.0.0-dev.380 | 2024-06-11 15:11:43 +0200

  * Drop EOL centos8-stream in CI (Benjamin Bannier, Corelight)

7.0.0-dev.378 | 2024-06-11 12:01:03 +0200

  * Suppress a known data race during civetweb shutdown (Tim Wojtulewicz)

7.0.0-dev.376 | 2024-06-11 11:00:46 +0200

  * Bump cmake for -fprofile-update=atomic usage (Arne Welzel, Corelight)

  * cirrus: Unset CCACHE_BASEDIR for asan/coverage build (Arne Welzel, Corelight)

    When CCACHE_BASEDIR is set, ccache will rewrite absolute paths to
    relative paths in order to allow compilation in different source
    directories. We do not need this feature on Cirrus (the checkout
    is always in /zeek) and using absolute paths avoids
    confusion/normalization needs for the gcov -p results.

    We could consider removing the global CCACHE_BASEDIR, but it'd
    bust the ccache of every other task, too.

7.0.0-dev.372 | 2024-06-07 09:28:52 -0700

  * Add Telemetry::metrics_address option (Tim Wojtulewicz, Corelight)

7.0.0-dev.368 | 2024-06-06 18:08:29 -0700

  * Change prometheus test to check for require jq (Tim Wojtulewicz, Corelight)

7.0.0-dev.366 | 2024-06-06 18:06:42 -0700

  * Check for 'zeekctl check' before trying to start up prometheus (Tim Wojtulewicz, Corelight)

7.0.0-dev.362 | 2024-06-04 14:16:54 -0700

  * Switch to zeek fork of prometheus-cpp (Tim Wojtulewicz, Corelight)

  * Remove unnecessary shared_from_this on instrument classes (Tim Wojtulewicz, Corelight)

  * Restore label_names field in MetricOpts record (Tim Wojtulewicz)

  * Change how we count FDs on Linux to fix zeekctl stop issues (Tim Wojtulewicz)

  * Update zeekctl tests for telemetry rework (Tim Wojtulewicz, Corelight)

  * Use forward declarations of prometheus-cpp types in telemetry::Manager (Tim Wojtulewicz, Corelight)

  * Add prometheus-cpp files to install set for plugins to use (Tim Wojtulewicz, Corelight)

  * Fix a memory leak with the CivetWeb callbacks in telemetry (Tim Wojtulewicz, Corelight)

  * Fix a bunch of copy-instead-of-move findings from Coverity (Tim Wojtulewicz, Corelight)

  * Move telmetry label names out of opts records, into main metric records (Tim Wojtulewicz, Corelight)

  * Ensure the order of label values matches the label names (Tim Wojtulewicz, Corelight)

  * Remove prefix column from telemetry.log (Tim Wojtulewicz, Corelight)

  * Fix race condition by pre-building the cluster json data for services.json (Tim Wojtulewicz)

  * Set running_under_test for scripts.base.frameworks.logging.telemetry test (Tim Wojtulewicz, Corelight)

7.0.0-dev.347 | 2024-06-04 11:36:13 -0700

  * Update reporter.bif to describe special case of errors in init (Smoot)

    Originally proposed in zeek/zeek-docs#257, but reverted via
    9f9ebde62380a3012a1471d9ff1c1c91c7aa69da.

7.0.0-dev.345 | 2024-06-04 10:36:46 -0700

  * script optimization baseline tweaks due to recent minor changes (Vern Paxson, Corelight)

  * updated list of BiFs for script optimization (Vern Paxson, Corelight)

  * addressed some Coverity nits (Vern Paxson, Corelight)

  * improved error cascade for invalid attributes (Vern Paxson, Corelight)

7.0.0-dev.338 | 2024-05-31 14:18:15 -0700

  * Add type aliases for instrument and family shared_ptrs (Tim Wojtulewicz, Corelight)

  * Update NEWS for double and is_sum changes (Tim Wojtulewicz, Corelight)

  * Remove is_sum arguments from counters and gauges (Tim Wojtulewicz, Corelight)

  * Change all instruments to only handle doubles (Tim Wojtulewicz, Corelight)

  * Add comment to telemetry::Manager::InitPostScript (Tim Wojtulewicz, Corelight)

  * Remove all of the ZEEK_METRICS_ environment variables (Tim Wojtulewicz, Corelight)

  * Fix header comments in scripts/policy/frameworks/telemetry/prometheus.zeek (Tim Wojtulewicz, Corelight)

  * Change all prometheus #includes to use angle brackets (Tim Wojtulewicz, Corelight)

  * Update zeekctl submodule for metrics_port node.cfg option (Tim Wojtulewicz, Corelight)

  * Remove the is_sum argument from BIF histogram creation methods (Tim Wojtulewicz, Corelight)

  * Update NEWS for Telemetry rework (Tim Wojtulewicz, Corelight)

  * Remove Telemetry::metrics_export_prefixes option (Tim Wojtulewicz, Corelight)

  * Validate that label names are constant in non-zeek metrics (Tim Wojtulewicz, Corelight)

  * Avoid calling Collect() in counter/gauge Value() methods if not needed (Tim Wojtulewicz, Corelight)

  * Fix some determinism issues with btests (Tim Wojtulewicz, Corelight)

  * Temporarily disable the scripts/base/frameworks/telemetry/internal-metrics btest (Tim Wojtulewicz, Corelight)

  * Fix the scripts.policy.frameworks.telemetry.prometheus btest to use the service discovery endpoint (Tim Wojtulewicz, Corelight)

  * Btest updates due to recent changes (Tim Wojtulewicz, Corelight)

  * Update telemetry log policy due to the fact that unit will not be filled in anymore (Tim Wojtulewicz, Corelight)

  * Rework everything to access the prometheus-cpp objects more directly (Tim Wojtulewicz, Corelight)

  * Move telemetry opaques out of OpaqueVal.h (Tim Wojtulewicz, Corelight)

  * Add GetRegistry() method for retrieving access to prometheus-cpp registry (Tim Wojtulewicz, Corelight)

  * Remove Broker metrics configuration values and methods (Tim Wojtulewicz, Corelight)

  * Add support for callbacks for gauges/counters, restore process stat metrics (Tim Wojtulewicz, Corelight)

  * Add a services.json endpoint for Prometheus service discovery (Tim Wojtulewicz, Corelight)

  * Don't include endpoint label/value when collecting metrics internally (Tim Wojtulewicz, Corelight)

  * Remove everything related to aggregation (Tim Wojtulewicz, Corelight)

  * Make all options const-redefs, remove all of the change handler code (Tim Wojtulewicz, Corelight)

  * Move base types from telemetry framework to init-bare (Tim Wojtulewicz, Corelight)

  * Update test baselines due to underscore changes (Tim Wojtulewicz, Corelight)

  * Convert telemetry code to use prometheus-cpp (Tim Wojtulewicz, Corelight)

  * Add necessary script-land changes (Tim Wojtulewicz, Corelight)

  * Add prometheus-cpp submodule, FindPrometheusCpp cmake module (Tim Wojtulewicz, Corelight)

7.0.0-dev.302 | 2024-05-31 09:37:21 -0700

  * More precise error reporting for the disable_analyzer() BiF (Christian Kreibich, Corelight)

  * Add btests for the lookup_connection_analyzer_id() BiF. (Christian Kreibich, Corelight)

  * Add BiF for looking up a connection's numeric protocol analyzer IDs (Christian Kreibich, Corelight)

7.0.0-dev.298 | 2024-05-29 13:49:00 -0700

  * removing now-vestigial "add" and "delete" statements (Vern Paxson, Corelight)

  * script optimization support for "add" and "delete" being expressions (Vern Paxson, Corelight)

  * "add" and "delete" are now expressions rather than statements (Vern Paxson, Corelight)

  * bug fix for dealing with illegal types of initializers (Vern Paxson, Corelight)

  * added a space when rendering some expressions so they're more readable (Vern Paxson, Corelight)

7.0.0-dev.292 | 2024-05-29 14:11:57 +0100

  * Copy timestamp from file object (mvhensbergen)

		In some cases, e.g. running zeek on short pcaps as opposed to continuous
		packet streams, network_time() may not equal the time that was used when
		generating the file object.  This results in the pe.log entry having a
		different timestamp than its corresponding files.log entry which is strange
		as they refer to the exact same file.

  * CI: Add coveralls-lcov gem to ubuntu 24, remove from ubuntu 20 (Tim Wojtulewicz, Corelight)

7.0.0-dev.287 | 2024-05-28 13:05:45 -0700

  * CI: Revert part of 2bde82ffa2f5fe9161e3322fee82eb45699b9164 to fix coverage builds (Tim Wojtulewicz, Corelight)

7.0.0-dev.284 | 2024-05-24 15:04:27 -0700

  * Fix Zeekygen warning for QUIC::unrecognized_version event (Christian Kreibich, Corelight)

7.0.0-dev.282 | 2024-05-24 10:49:31 -0700

  * Update TLS consts, mainly new named curves. (Johanna Amann, Corelight)

    Add test for X25519Kyber768Draft00 (post-quantum key agreement)

7.0.0-dev.280 | 2024-05-21 16:22:57 -0700

  * CI: Remove --enable-werror for asan builds (Tim Wojtulewicz, Corelight)

    There's a bug in GCC (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=105562) that
    causes std::regex to emit some warnings when built with -fsanitize=address. Any
    warnings that actually mean anything would be caught by the other sanitizer
    builds or by the regular ubuntu24 build.

  * CI: Run coverage builds for PRs, but only upload on master (Tim Wojtulewicz, Corelight)

  * Coverage: Ignore a few errors during generation (Tim Wojtulewicz, Corelight)

  * Coverage: don't bother ignoring non-existent bro.dir files (Tim Wojtulewicz, Corelight)

  * CI: Re-enable coverage (Tim Wojtulewicz, Corelight)

  * CI: Switch asan build back to gcc (Tim Wojtulewicz, Corelight)

    The tools used for coverage (gcov, lcov) work better with GCC. We could switch
    over to similar tools for llvm, but they way they store files during coverage
    builds don't work as well for us (mostly the places they store the files).

  * CI: Avoid divide by zero error when generating coverage files (Tim Wojtulewicz, Corelight)

7.0.0-dev.270 | 2024-05-20 15:54:09 -0700

  * Fix for suppressing SMB logging of previously-logged files (Vern Paxson, Corelight)

7.0.0-dev.268 | 2024-05-17 08:56:31 -0700

  * Bump Spicy to latest dev snapshot (Benjamin Bannier, Corelight)

  * CI: Disable coverage temporarily, broken with clang-18 (Tim Wojtulewicz, Corelight)

7.0.0-dev.265 | 2024-05-16 11:09:23 -0700

  * Fix for ZAM inlining of nested function calls with the same parameter names (Vern Paxson, Corelight)

  * Fixed ZAM logic error in canonicalizing specialized min/max instructions (Vern Paxson, Corelight)

  * Fixed order-of-evaluation bug in ZAM Subnet-To-Addr instruction (Vern Paxson, Corelight)

  * "-a zam" BTest baseline update reflecting recent Spicy baseline change (Vern Paxson, Corelight)

7.0.0-dev.260 | 2024-05-16 09:34:55 -0700

  * CI: Disable spicy_head task for release branches (Tim Wojtulewicz, Corelight)

7.0.0-dev.258 | 2024-05-15 20:00:41 -0700

  * CI: Use llvm-cov-18 on Cirrus for building coverage data (Tim Wojtulewicz, Corelight)

  * CI: Add missing lcov package to ubuntu 24 Dockerfile (Tim Wojtulewicz)

7.0.0-dev.254 | 2024-05-13 17:40:37 -0700

  * CI: Use clang-18 on Ubuntu 24 for sanitizers builds, enable TSan builds (Tim Wojtulewicz)

  * CI: Add clang-18 to Ubuntu 24.04 VM (Tim Wojtulewicz)

7.0.0-dev.251 | 2024-05-13 16:38:33 +0200

  * Spicy: Apply missing fix for recent Spicy bump. (Robin Sommer, Corelight)

7.0.0-dev.249 | 2024-05-13 11:51:17 +0200

  * Bump Spicy, with small tweaks for API changes. (Robin Sommer, Corelight)

7.0.0-dev.247 | 2024-05-10 10:19:43 -0700

  * CI: Update asan and ubsan tasks to use Ubuntu 22.04 VM (Tim Wojtulewicz, Corelight)

  * CI: Replace Ubuntu 23.10 with Ubuntu 24.04 (Tim Wojtulewicz, Corelight)

7.0.0-dev.244 | 2024-05-10 11:08:47 +0200

  * GH-3725: Spicy: Fix service reporting for replaced analyzers. (Robin Sommer, Corelight)

    We accidentally applied analyzer mappings when looking up an
    analyzer's name from scriptland.

  * GH-3561: Spicy: Support UDP in Spicy's `protocol_*` runtime functions. (Robin Sommer, Corelight)

    This extends the ability to feed new payload back into Zeek's analyzer
    pipeline from TCP to now also UDP.

    Note: We don't extend this further to ICMP because the ICMP analyzer
    cannot be dynamically instantiated (Zeek aborts when trying so). As
    ICMP isn't very interesting from use case perspective anyways, that
    seems fine.

  * Add method to analyzer to retrieve direct child by name. (Robin Sommer, Corelight)

  * Add explicit children life-cycle management method to analyzers. (Robin Sommer, Corelight)

  * Extend PIA's `FirstPacket` API. (Robin Sommer, Corelight)

    `FirstPacket()` so far supported only TCP. To extend this to UDP, we
    move the method into the PIA base class; give it a protocol parameter
    for the case that there's no actual packet is available; and add the
    ability to create fake UDP packets as well, not just TCP.

    This whole thing is pretty ugly to begin with, and this doesn't make
    it nicer, but we need this extension that so we can feed UDP data into
    the signature engine that's tunneled over other protocols. Without the
    fake packets, DPD signatures in particular wouldn't have anything to
    match on.

  * Spicy: Prepare for supporting forwarding to protocols other than TCP. (Robin Sommer, Corelight)

    So far the Spicy runtime supported forwarding data into other
    analyzers only for TCP analyzers. This puts branching logic in place
    that let the relevant runtime functions dispatch differently based on
    the target transport-layer protocol. We don't implement anything else
    than TCP yet; that will come next.

    Along with the internal changes, this also updates the user-visible
    runtime function to pass protocol information in. For now, this
    likewise remains limited to TCP. The function signatures are chosen so
    that they stay backwards-compatible to previous Spicy version. In
    particular, they default to TCP where not otherwise specified.

  * Fix include of private file in public header (Benjamin Bannier, Corelight)

  * Update package-manager submodule [nomail] (Tim Wojtulewicz, Corelight)

  * Update zeekctl submodule [nomail] (Tim Wojtulewicz, Corelight)

7.0.0-dev.231 | 2024-05-08 13:09:25 -0700

  * enable ZAM operation specifications to reside in multiple files (not yet used) (Vern Paxson, Corelight)

7.0.0-dev.229 | 2024-05-08 09:55:01 -0700

  * Move packet_analysis::Dispatcher to detail namespace (Tim Wojtulewicz, Corelight)

.0.0-dev.226 | 2024-05-07 12:02:28 -0700

  * make SSH analyzer robust to half-duplex connections (Vern Paxson, Corelight)

7.0.0-dev.224 | 2024-05-07 10:26:40 -0700

  * Clarify membership in the Site::private_address_space prefix list. [skip ci] (Christian Kreibich, Corelight)

    IANA's IPv6 special-purpose address registry now has members that technically
    meet the definition of not being globally reachable, but don't imply operating
    locally. An example: https://datatracker.ietf.org/doc/draft-ietf-6man-sids/06/

    This change just explains that distinction.

7.0.0-dev.222 | 2024-05-07 10:25:55 -0700

  * Avoid segfault when generating Zeekygen docs on Zeek-internal identifiers. (Christian Kreibich, Corelight)

    Identifiers that Zeek generates explicitly in-core for example during component management
    ("Analyzer::ANALYZER_FOO", "Input::READER_FOO", etc), aren't associated with any
    scripts and would lead to nullpointer derefs when trying to identify
    modification time.

    This fix returns a zero timestamp in such cases, ensuring it never triggers docs
    refreshes. We do the same e.g. in PackageInfo instances.

  * Add btest for Zeekygen docs extraction on identifiers defined by the Zeek core. (Christian Kreibich, Corelight)

    Mainly, this should not crash. :-) The baseline output file also should remain
    empty, since Zeekygen's staleness test should conclude that it's sufficiently
    fresh.

7.0.0-dev.218 | 2024-05-07 09:45:25 +0200

  * Fix packet analyzer replacement. (Robin Sommer, Corelight)

    Also reworking the existing test, which was pretty odd/broken.

  * GH-3573: Spicy: Wenn replacing an analyzer add a component mapping. (Robin Sommer, Corelight)

    This uses the new API to replace components internally.

    With these changes in place, replacing protocol analyzers now don't
    need to register their ports anymore if they match what the original
    analyzer was using (because the old one's registrations will map
    over).

    Packet analyzer replacement doesn't quite work yet but will be fixed
    in next commit.

  * Add component API to transparently remap one component to another one. (Robin Sommer, Corelight)

    When a specific component is requested through its tag or name, one
    can now have the component manager transparently return a different
    one that has been registered to replace the original one. We limit
    this to disabled components to avoid unnecessary confusion. That also
    means that remappings are currently only supported for analyzers
    (because other types of components cannot be disabled for now, per the
    previous change).

  * Move enabled/disabled functionality from analyzers into `Component` base class API. (Robin Sommer, Corelight)

    The different analyzers types all had their own methods for
    enabling/disabling their availability. This change abstracts that into
    a new API inside their base class (`plugin::Component`) so that they
    can be toggled in a unified way.

    In principle, other types of components could/should use this as well
    now, so that, e.g., an input reader's availability could be toggled at
    runtime. The code doesn't make that broader change for now because it
    would requires a series of changes wherever these other component
    types are being used. However, that means that one now could try
    toggling some other component through the new API without that having
    any effect. To catch that, there's a runtime check in place that turns
    any such attempt into an internal error.

  * GH-3440: Spicy: Query Zeek scriptland for file handles. (Robin Sommer, Corelight)

    Like traditional file analyzers, we now query Zeek's
    `get_file_handle()` event for handles when a connection begins
    analyzing an embedded file. That means that Spicy-side protocol
    analyzers that are forwarding data into file analysis now need to call
    Zeek's `Files::register_protocol()` and provide a callback for
    computing file handles. If that's missing, Zeek will now issue a
    warning. This aligns with the requirements Zeek's traditional protocol
    analyzers. (If the EVT file defines a protocol analyzer to `replace`
    an existing one, that one's `register_protocol()` will be consulted.)

    Because Zeek's `get_file_handle()` event requires a current
    connection, if a Spicy file analyzer isn't directly part of a
    connection context (e.g., with nested files), we continue to use
    hardcoded, built-in file handle. Scriptland won't be consulted in
    that case, just like before.

  * Spicy: Cleanup some runtime code. (Robin Sommer, Corelight)

  * GH-3606: Spicy: Map Spicy's `Protocol` enum to Zeek's `transport_proto`. (Robin Sommer, Corelight)

    We now hardcode this specific enum mapping to make Zeek-side usage
    more convenient.

7.0.0-dev.207 | 2024-05-06 16:59:31 -0700

  * Update broker submodule [nomail] (Tim Wojtulewicz, Corelight)

7.0.0-dev.205 | 2024-05-06 11:16:15 -0700

  * Move the options from policy/tuning/defaults to actual Zeek defaults, deprecate that package (Tim Wojtulewicz, Corelight)

7.0.0-dev.200 | 2024-05-02 19:00:53 -0700

  * refine Val "footprint" to equate long strings with multiple objects (Vern Paxson, Corelight)

7.0.0-dev.198 | 2024-05-02 10:21:43 -0700

  * Constify classes in RuleMatcher, fixes c++20 build failure (Tim Wojtulewicz, Corelight)

7.0.0-dev.196 | 2024-04-29 09:09:28 -0700

  * Fix cid propagation into files.log (Johanna Amann, Corelight)

    Changes to the connection id were not propagated to files.log in all
    cases.

    Fixes GH-3700

7.0.0-dev.194 | 2024-04-29 09:08:20 -0700

  * factored BTest constants into globals to prevent premature ZAM optimization when testing (Vern Paxson, Corelight)

  * addressed minor issues flagged by Coverity (Vern Paxson, Corelight)

7.0.0-dev.191 | 2024-04-29 09:06:24 -0700

  * Avoid calling typecasts in Val when we have direct access to the underlying value object (Tim Wojtulewicz, Corelight)

7.0.0-dev.187 | 2024-04-26 12:29:18 -0700

  * Gracefully handle empty/missing shadow file (Peter Cullen, Corelight)

    When a shadow file is empty/missing during rotation, Zeek aborts
    with an error message, but if the shadow file was empty, it'll still
    be there after the restart, causing an endless restart loop. This
    solution gracefully handles the rotation in such cases using the
    default file extension and post processing function.

7.0.0-dev.184 | 2024-04-26 11:17:52 -0700

  * GH-3671: Factor in caplens in ICMPAnalyzer::DeliverPacket length calculations (Christian Kreibich, Corelight)

  * Update binpac submodule [nomail] (Tim Wojtulewicz, Corelight)

  * Update gen-zam submodule [nomail] (Tim Wojtulewicz, Corelight)

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

7.0.0-dev.179 | 2024-04-25 09:15:30 -0700

  * rewrite of ZAM optimization of BiFs and script functions/idioms (Vern Paxson, Corelight)

  * flag base script as being known to ZAM optimization (Vern Paxson, Corelight)

  * ZAM optimization for constructing records w/ fields initialized to network_time (Vern Paxson, Corelight)

  * factor some functionality used by BiFs to make accessible to ZAM instructions (Vern Paxson, Corelight)

  * factor script optimization Expr AST nodes out of main AST header (Vern Paxson, Corelight)

  * new internal AST node for expression idioms with corresponding ZAM built-ins (Vern Paxson, Corelight)

  * streamlining of expressions only used for script optimization (Vern Paxson, Corelight)

  * reordering of expressions to match main usage vs. script-optimization extras (Vern Paxson, Corelight)

  * support for conditional expressions transforming into interal has-elements expr (Vern Paxson, Corelight)

  * make same_expr() functionality accessible as a utility (Vern Paxson, Corelight)

  * unify functionality across EqExpr and RelExpr classes (Vern Paxson, Corelight)

  * introduce notions of script functions/BiFs that are (1) foldable, (2) replaceable (Vern Paxson, Corelight)

  * optimization of scripting idioms - min/max, has-elements (Vern Paxson, Corelight)

  * hooks for ZAM optimization of calls to particular functions (Vern Paxson, Corelight)

  * shift ZAM maintenance monitoring of new BiFs to a BTest so it automatically runs (Vern Paxson, Corelight)

  * make externally available the expression used for initializating a record field (Vern Paxson, Corelight)

  * record field initialization optimization: do coercions at compile-time if applicable (Vern Paxson, Corelight)

  * $ZAM_PROFILE_ALL set tells ZAM to profile function bodies even if not executed (Vern Paxson, Corelight)

  * performance speed-up for SMB base scripts (Vern Paxson, Corelight)

  * fix for ZAM optimization pruning of unnecessary instructions (Vern Paxson, Corelight)

  * bug fix for ZAM code generation (Vern Paxson, Corelight)

  * optimize generation of "-O gen-C++" code for table/vector size operations (Vern Paxson, Corelight)

  * removing vestigial same_val() function (Vern Paxson, Corelight)

  * fix for ZAM optimization of '!' expressions (Vern Paxson, Corelight)

  * fix for ZAM tracking of return values when in-lining (Vern Paxson, Corelight)

  * fix for ZAM profiling potentially accessing freed memory (Vern Paxson, Corelight)

  * Fix typo in hook name (Benjamin Bannier, Corelight)

    While in Spicy code a hook priority is spelled `priority=4711` the
    attribute is still called `&priority` (like in HILTI) and we rely on
    exactly that name when e.g., extracting hook priorities for scheduling.

    This change was introduced as part of
    db98dc419334667e9299ef00670f4d435f3e408c and caused the default hook
    priority for hooks defined in EVT files (intended to be -1000 to likely
    schedule after e.g., hooks in the Spicy grammars) to be ignored. This
    could then e.g., introduce issue when a `%done` hook would mutate state
    exposed in an EVT hook (which now might not have seen the updated state
    due to different scheduling).

7.0.0-dev.147 | 2024-04-22 17:08:25 -0700

  * Don't attempt to stop or flush disabled writer frontends (Tim Wojtulewicz, Corelight)

7.0.0-dev.145 | 2024-04-22 15:55:53 -0700

  * Allow SMB_TCP record to contain multiple protocol identifiers/headers (Tim Wojtulewicz, Corelight)

7.0.0-dev.143 | 2024-04-22 15:15:55 -0700

  * Check for valid identifier in WhenStmt::Describe (Tim Wojtulewicz, Corelight)

  * Bump zeek-aux submodule to pull in zeek-archiver addition (Christian Kreibich, Corelight)

  * Remove zeek-archiver from build config, deprecate --disable-archiver in configure (Christian Kreibich, Corelight)

  * Drop zeek-archiver submodule (Christian Kreibich, Corelight)

    zeek-archiver now lives in the zeek-aux submodule.

7.0.0-dev.136 | 2024-04-18 09:02:14 -0700

  * Update core.pcap.filter-warning baseline (Peter Cullen, Corelight)

    Only the line numbers changed.

  * Add PacketFilter::remove_exclude function (Peter Cullen, Corelight)

    This adds a public function to the PacketFilter framework that
    allows the caller to remove an exclude filter by filter id.

7.0.0-dev.132 | 2024-04-17 14:59:43 -0700

  * Parse and store localversion string (Peter Cullen, Corelight)

  * Update version string btests for localversion (Peter Cullen, Corelight)

  * Support for configurable localversion (Peter Cullen, Corelight)

7.0.0-dev.123 | 2024-04-17 14:33:55 -0700

  * fix for ZAM optimization of "while" loops (Vern Paxson, Corelight)

7.0.0-dev.121 | 2024-04-17 09:20:53 -0700

  * Add Opensuse leap 15.6 to CI (Johanna Amann, Corelight)

7.0.0-dev.119 | 2024-04-15 17:52:44 +0200

  * Bump Spicy. (Robin Sommer, Corelight)

    Includes a couple of updates for the QUIC analyzer.

  * Remove support for old Spicy versions from QUIC analyzer. (Robin Sommer, Corelight)

7.0.0-dev.116 | 2024-04-11 15:26:05 -0700

  * Make sure that vcpkg isn't preferred if pcap_root_dir is passed in (Tim Wojtulewicz)

7.0.0-dev.114 | 2024-04-10 11:02:32 +0200

  * Bump Spicy, including necessary changes to work with the current
    Spicy version. (Robin Sommer, Corelight)

  * Remove some unused Spicy state. (Robin Sommer, Corelight)

7.0.0-dev.111 | 2024-04-09 15:12:13 -0700

  * ZAM fix for concretizing vectors in record constructors (Vern Paxson, Corelight)

  * improve ZAM's estimation of profiling overheads (Vern Paxson, Corelight)

7.0.0-dev.108 | 2024-04-08 12:02:03 -0700

  * CI: Remove commented openssl 1.1 workaround from macOS preparation script (Tim Wojtulewicz, Corelight)

  * CI: Fix installation of python package on macOS (Tim Wojtulewicz, Corelight)

  * fix ZAM "cat" of doubles/times to include trailing ".0" per normal BiF behavior (Vern Paxson, Corelight)

7.0.0-dev.102 | 2024-04-02 08:17:08 -0700

  * fix ZAM "cat" of doubles/times to include trailing ".0" per normal BiF behavior (Vern Paxson, Corelight)

7.0.0-dev.100 | 2024-03-29 11:15:24 -0700

  * CI: Specify the xcode version of the macOS Sonoma instance (Tim Wojtulewicz, Corelight)

7.0.0-dev.98 | 2024-03-28 11:09:48 -0700

  * Remove vestigial Conan bit in CMakeLists.txt (Christian Kreibich, Corelight)

  * When configuring Spicy, be prepated for zeek_lib or zeek_exe targets. (Christian Kreibich, Corelight)

  * Fix a typo in CMakeLists.txt when building Zeek as a library (Christian Kreibich, Corelight)

7.0.0-dev.94 | 2024-03-25 11:25:47 +0100

  * Bump Spicy to current `main`. (Robin Sommer, Corelight)

7.0.0-dev.92 | 2024-03-19 14:39:26 -0700

  * ZAM documentation updated to reflect finer-grained profiling (Vern Paxson, Corelight)

  * ZAM-specific BTest baseline changes for tweak to how ZAM bodies print (Vern Paxson, Corelight)

  * BTest baseline changes for tweak to how ZAM bodies print (Vern Paxson, Corelight)

  * ZAM execution changes to support richer profiling (Vern Paxson, Corelight)

  * use richer block-aware location information for ZAM instructions (Vern Paxson, Corelight)

  * better isolation of ZAM instruction elements (Vern Paxson, Corelight)
    hooks for tracking extended ZAM profiling location

  * framework for --enable-ZAM-profiling configuration (Vern Paxson, Corelight)
    profiles go to zprof.log rather than stdout

  * ZAM classes in support of finer-grained profiling (Vern Paxson, Corelight)

  * refined ZAM function profiling to include (correct) statement line number blocks (Vern Paxson, Corelight)

  * avoid potentially expensive mallinfo() call if result won't be used (Vern Paxson, Corelight)

7.0.0-dev.80 | 2024-03-19 10:59:22 -0700

  * Remove variant from StdFunctionStmt (Tim Wojtulewicz, Corelight)

    The variant ended up conflicting with std::bind, which resulted in failures
    on the btest invoking it. Change back to a single function that takes a
    flow, and default it to a value in Exec.

7.0.0-dev.78 | 2024-03-18 16:37:46 -0700

  * Add a method to register an event handler to a std::function via C++ (Tim Wojtulewicz, Corelight)

7.0.0-dev.76 | 2024-03-18 16:15:44 -0700

  * Update Broker submodule (Dominik Charousset, Corelight)

  * cirrus: Remove misleading and always empty AWS_ECR_USERNAME (Arne Welzel, Corelight)

7.0.0-dev.72 | 2024-03-18 09:55:51 +0100

  * btest/spicy: Remove most port usages (Arne Welzel, Corelight)

  * spicy: Deprecate port/ports in .evt files (Arne Welzel, Corelight)

  * Update doc submodule (Arne Welzel, Corelight)

    For spicy-pygments.py sync.

  * github/generate-docs: Only commit if there are staged changes (Arne Welzel, Corelight)

    git diff-index by default includes staged and non-staged changes.
    The autogen-spicy-docs script copies over spicy-pygments.py from
    the Spicy tree into doc/ext.

    The job would attempt to commit scripts and script-reference even
    though nothing actually is staged when a spurious diff existed in
    ext/spicy-pygments.py. Guard from this by using --cached.

7.0.0-dev.66 | 2024-03-16 11:14:25 +0100

  * Bump Spicy to latest version reworking AST memory management. (Robin Sommer, Corelight)

    Includes the necessary Zeek-side changes.

    Goes with https://github.com/zeek/spicy/pull/1691.

  * GH-3647: Add a CI test with the same options as we used for binary builds. (Johanna Amann, Corelight)

    We already do have a test that check static builds, but none that tests
    the exact same build flags we use for binary builds on OBS. This commit
    fixes this oversight.

  * GH-3647: Fix binary build of `spicyz`. (Robin Sommer, Corelight)

    We need to link against the Spicy toolchain's object libraries but
    were linking against static libraries instead.

7.0.0-dev.61 | 2024-03-15 13:14:47 -0700

  * CI: Remove CentOS 7 and Debian 10 builds (Tim Wojtulewicz, Corelight)

7.0.0-dev.59 | 2024-03-15 13:36:30 +0100

  * testing/external/cluster: Switch to docker-compose v2 (Arne Welzel, Corelight)

    The Python docker-compose executable in the docker_builder image is not
    functional anymore (TypeError: kwargs_from_env() got an unexpected
    keyword argument 'ssl_version').

    docker-compose v1 has been sun-setted for quite some time. Do the
    minimal to switch to docker compose v2.

7.0.0-dev.57 | 2024-03-13 10:17:48 -0700

  * Add community contributions for the 6.2 timeframe to NEWS. [skip ci] (Christian Kreibich, Corelight)

7.0.0-dev.54 | 2024-03-12 10:31:19 +0100

  * GH-3640: Expr: Handle TYPE_VOID in SizeExpr and AssignExpr::Typecheck() (Arne Welzel, Corelight)

    @vpax reported surprising behavior when working with "void values".
    While these are not exposed to script land, plumb the places he
    pointed out are causing confusing behavior.

    Closes #3640.

7.0.0-dev.52 | 2024-03-08 19:10:06 +0100

  * AST location fixes for -O gen-C++ (Vern Paxson, Corelight)

  * script optimization support for clearing tables/vectors using "delete" (Vern Paxson, Corelight)

  * fix for ZAM printing of instructions that modify slots other than slot 1 (Vern Paxson, Corelight)

  * fixed bug where appending an empty vector to a vector-of-vector's had no effect (Vern Paxson, Corelight)

7.0.0-dev.47 | 2024-03-07 09:52:29 +0100

  * Bump Spicy. (Robin Sommer, Corelight)

7.0.0-dev.45 | 2024-03-06 10:38:55 +0100

  * Make worker restart in cluster btest race-proof (Jan Grashoefer, Corelight)

7.0.0-dev.43 | 2024-03-05 19:23:09 +0100

  * highwayhash: Point to github.com/google/highwayhash, bump to master (Arne Welzel, Corelight)

  * RE/Specific_RE_Matcher: Make Compile() return false for syntax errors (Arne Welzel, Corelight)

    When creating RE_Matcher instances at runtime and verifying the pattern
    compiles via Compile(), the syntax_error flag wasn't respected and
    Compile() would return true even for some invalid regular expressions.

    For example, compiling /a{1,b}/, Compile() would return true even though
    it produced a reporter error while parsing complaining about b not
    being valid.

    This patch improves the error handling, so that calling Compile() returns
    false whenever zeek::detail::synerr() was called while a pattern was
    parsed. The use-case is creation of patterns at runtime based on
    JavaScript strings. These might be entered or received at runtime via
    an API. This change allows to be a bit more robust to detect invalid
    input and raising exceptions to notify the user.

    This also move syntax_error and csize out of global scope.

    If RE_Matcher was to be used as an actual API, we likely should squelch
    the reporter errors and mark it as not thread safe, but this is a small
    step forward.

7.0.0-dev.38 | 2024-03-04 12:32:59 -0700

  * CI: Bump FreeBSD 13 build to 13.3 (Tim Wojtulewicz, Corelight)

7.0.0-dev.36 | 2024-03-04 18:22:34 +0100

  * Stmt: Fix assert evaluating cond twice (Arne Welzel, Corelight)

    Since 81a9745fb36fb6705873921423364bd2c438bfc4, the assert condition is
    evaluated twice. This leads to unexpected behavior when cond has a side
    effect like publishing a message or creating a log stream or filter.

    Found while using the following in ad-hoc testing code and wondering
    why two messages were published.

        assert publish(Cluster::worker_topic, hello, "abc")

7.0.0-dev.34 | 2024-03-04 15:49:19 +0100

  * Port Spicy integration to new AST API. (Robin Sommer, Corelight)

7.0.0-dev.32 | 2024-02-28 17:10:26 +0100

  * Bump zeekjs to 0.12.0 (Arne Welzel, Corelight)

      0a89ca6 doc: Expand zeek.as() description and add an example
      7e814d7 Types: Implement basic pattern support
      43df9d2 Update docs to provide example of shared Node.js openSSL configuration
      3bf2ea5 lsan suppressions: Add some for 21.11
      640affa zeek.global_vars: Remove leftover internal field usage
      3ee53c7 zeek.global_vars: Implement setter
      8144061 zeek.as: Support more Zeek types
      b453483 zeek.as: Fix crash for non-atomic types

7.0.0-dev.29 | 2024-02-27 15:18:49 +0100

  * GH-3619: spicy/zeekygen: Remove mtime from generated code (Arne Welzel, Corelight)

    Zeekygen implements its own make-style update logic to prevent
    re-creation of files that have not changed. To fulfill this, we
    currently encode the current time into spicyz generated .cc files.

    This degrades ccache efficiency for built-in analyzers and also
    for all .evt files compiled during testing. Switch SpicyModuleInfo
    to return current time instead. This results in the re-generation
    of documentation files unconditionally when running Zeekygen, but
    that seems more acceptable IMO.

    Generally wonder if Zeekygen should produce output unconditionally
    and if we need to clobber prevention, compare with the content of
    the existing file.

7.0.0-dev.26 | 2024-02-26 21:15:59 +0100

  * signatures/iso-9660: Add \x01 suffix to CD001 (Arne Welzel, Corelight)

    As discussed with Tim, that should make it a bit more robust against
    false positives.

  * test-all-policy: Do not load iso-9660.zeek (Arne Welzel, Corelight)

    Changing the default_file_bof_buffer_size has subtle impact on
    MIME type detection and changed the zeek-testing baseline. Do
    not load this new script via test-all-policy to avoid this.

    The new test was mainly an aid to understand what is actually going on.
    In short, if default_file_bof_buffer_size is larger than the file MIME
    detection only runs when the buffer is full, or when the file is removed.
    When a file transfer happens over multiple HTTP connections, only
    some or one of the http.log entries will have a proper response MIME type.

    PCAP extracted from 2009-M57-day11-18.trace.gz.

  * signatures: Move ISO 9660 signature to policy (Arne Welzel, Corelight)

    The previous "fix" caused significant performance degradation without
    the signature ever having a chance to trigger. Moving it to policy
    seems the best compromise, the alternative being outright removing it.

  * GH-3490: global_ids: Align script_id$type_name field with type_name() (Arne Welzel, Corelight)

    Populate script_id$type_name with what the type_name() bif
    would produce for the same identifier.

    Closes #3490

  * utils: Introduce packages.zeek with can_load() helper (Arne Welzel, Corelight)

  * GH-3594: zeek.bif: Add find_in_zeekpath() helper (Arne Welzel, Corelight)

    Relates to #3594. This helper can be used to determine the path that
    will be used by @load, if at all.

7.0.0-dev.16 | 2024-02-26 11:09:57 -0700

  * Update Mozilla CA and Google CT list and related tests. (Johanna Amann, Corelight)

7.0.0-dev.13 | 2024-02-22 17:00:29 +0100

  * GH-3139: spicy: Remove Zeek version from spicyz/config.h.in (Arne Welzel, Corelight)

    It seems that Zeek's version number and string only need to be
    available at runtime, so this change removes it from spicyz/configh.in
    to avoid needlessly busting ccache for the src/spicyz tree for on a
    Zeek version bump.

  * spicy: Drop ZEEK_VERSION_NUMBER ifdef'ery (Arne Welzel, Corelight)

    Given src/spicy is part of the Zeek tree, these are not necessary and
    are probably just remains from the spicy-plugin era.

7.0.0-dev.10 | 2024-02-22 12:38:38 +0100

  * signatures: Fix ISO 9960 signature (Arne Welzel, Corelight)

    This signature only really works when default_file_bof_buffer_size is bumped
    to a sufficient value (40k).

7.0.0-dev.8 | 2024-02-22 12:22:56 +0100

  * testing: Update zeek-testing-private commit hash (Arne Welzel, Corelight)

  * ftp: Reset fuid after logging (Arne Welzel, Corelight)

    A user reported being confused about the fuid association of subsequent
    FTP commands when a data transfer has completed. It seems reasonable to
    unset fuid upon logging a FTP command which had a fuid.

    The current behavior results in the PORT or PASV commands after a RETR or STOR
    to have the fuid of the prior file transfer. Similarly, any CWD or DEL commands
    following a file transfer will unnecessarily be logged with the fuid of the
    prior file transfer.

    This tickles the baselines for the private testing PCAP a lot, primarily
    because there data connections in that pcap are never established properly.
    E.g, the fuids FzDzid1Dxm9srVKHXf and FEfYX73q5C6GEQZXX9 have been re-used
    for multiple commands.

    This may look like we're losing information, but the fuids vanishing
    in the normal btests belong to a LIST command that isn't logged by
    default into ftp.log. If it was, the fuid would be attached to it.

7.0.0-dev.5 | 2024-02-22 12:01:33 +0100

  * GH-3472: Allow delete statement for tables, sets and vectors (Arne Welzel, Corelight)

    Relates to #3472. This allow "delete tbl" as an alternative for
    clear_table(tbl). Also works for vectors.

  * Update cmake submodule [nomail] (Tim Wojtulewicz, Corelight)

  * Update cmake submodule [nomail] (Tim Wojtulewicz, Corelight)

  * Point cmake submodule at master branch (Tim Wojtulewicz, Corelight)

  * Start of 7.0.0 development (Tim Wojtulewicz, Corelight)

  * Update zkg submodule to v3.0.1 (Tim Wojtulewicz, Corelight)

6.2.0-dev.545 | 2024-02-20 10:57:30 -0700

  * Pull in variant-enabled Broker (Dominik Charousset, Corelight)

    Switch the Broker submodule to a version with `broker::variant` enabled
    and provide new overloads for `broker::variant` in the Broker manager.

6.2.0-dev.543 | 2024-02-15 11:08:16 -0500

  * Bump zkg to v3.0.0 (Tim Wojtulewicz, Corelight)

6.2.0-dev.541 | 2024-02-15 11:07:17 -0500

  * Bump Spicy to latest dev snapshot. (Benjamin Bannier, Corelight)

6.2.0-dev.539 | 2024-02-13 20:10:06 +0100

  * zeek.bif: Booleanate String constructor calls (Arne Welzel, Corelight)

  * Conn/net_utils/fmt_mac: Avoid snprintf(), memcpy() and allocation (Arne Welzel, Corelight)

    The fmt_mac() function returning a std::string means the resulting mac
    is copied at least once upon returning. Then, the Assign() in GetVal()
    taking a std::string internally allocates a new zeek::String which
    hits a malloc (no short-string optimization for zeek::String) and then
    also copies the content from the std::string into the malloced memory.

    Save a few cycles by directly using the allocated memory with the
    String instance. This change improves runtime for a SYN-only pcap
    with just base/protocols/conn loaded by some 1-2%.

  * bytestring_to_hexstr: Use bytetohex() instead of snprintf() (Arne Welzel, Corelight)

    ...also avoid the extra memcpy and memory allocation.

  * digest/digest_print: Use bytetohex() instead of snprintf() (Arne Welzel, Corelight)

6.2.0-dev.533 | 2024-02-08 15:12:16 +0100

  * Bump zeekjs for STMT_ANY deprecation (Arne Welzel, Corelight)

  * GH-3529: Stmt: Deprecate STMT_ANY (Arne Welzel, Corelight)

    This isn't used in-tree and has been misused by some external plugins
    of mine (zeekjs and zeek-perf-support) for their own Stmt subclasses.
    These plugins should be updated to use the new STMT_EXTERN statement.

    Handle STMT_ANY explicitly in stmt_name() for the time being
    to fix #3529 until we remove STMT_ANY for good.

  * Stmt: Introduce STMT_EXTERN (Arne Welzel, Corelight)

    It's currently possible for plugin's to implement their own statement
    subclasses and override the Exec() implementation. This has been leveraged
    by ZeekJS [1] and zeek-perf-support [2] as well as a private WASM plugin.
    All of these used STMT_ANY as the tag of their own statement subclasses.

    With STMT_EXTERN, we make the possibility to add external code into the AST
    somewhat more supported. It's all in detail space and plugin authors have
    no guarantee for stability, but it seems such a powerful extension point
    that IMO we should keep it.

    I'm conscious there's the broader topic how this interacts with ZAM
    optimization like in-lining or rewriting of statements. However, this
    already applies to the STMT_ANY usage of the mentioned plugins.

    [1] https://github.com/corelight/zeekjs
    [2] https://github.com/zeek/zeek-perf-support

6.2.0-dev.528 | 2024-02-07 12:45:30 -0700

  * Update .gitignore to add Emacs and Vim temp files (Tim Wojtulewicz)

  * Update vcpkg submodule to pull in c-ares 1.26.0 to match other platforms (Tim Wojtulewicz)

6.2.0-dev.525 | 2024-02-07 09:36:18 -0800

  * Some NEWS updates for 6.2 (Christian Kreibich, Corelight)

6.2.0-dev.523 | 2024-02-07 13:14:40 +0100

  * websocket: Implement Spicy's unmask in C++ (Arne Welzel, Corelight)

    This improves runtime for the test pcap I have from 22 seconds to
    12 seconds, about 1.8x times faster.

  * websocket: Add Spicy parser version, too. (Arne Welzel, Corelight)

    The Spicy analyzer is added as a child analyzer when enabled and the
    WebSocket.cc logic dispatches between the BinPac and Spicy version.

    It substantially slower when tested against a somewhat artificial
    2.4GB PCAP. The first flamegraph indicates that the unmask() function
    stands out with 35% of all samples, and above it shared_ptr samples.

6.2.0-dev.519 | 2024-02-06 11:51:26 +0000

  * Netcontrol: add rule_added_policy (Johanna Amann, Corelight)

    rule_added_policy allows the modification of rules just after they have
    been added. This allows the implementation of some more complex features
    - like changing rule states depending on insertion in other plugins.

  * Netcontrol: more logging in catch-and-release (Johanna Amann, Corelight)

    Catch-and-release logs now include the plugin that is responsible for an
    action. Furthermore, the catch-and-release log also includes instances
    where a rule already existed, and where an error occurred during an
    operation.

  * Netcontrol: allow supplying explicit name to Debug plugin (Johanna Amann, Corelight)

    This change extends the arguments of NetControl::create_debug, and
    allows the specification of an optional name argument, which can be used
    instead of the default-generated name.

    This is helpful when one wants to attach several plugins to verify
    behavior in those cases.

6.2.0-dev.515 | 2024-02-06 11:22:26 +0100

  * Obj: Implement with_location_of() as template (Arne Welzel, Corelight)

  * fixes & enhancements to location information associated w/ AST nodes & ZAM optimization (Vern Paxson, Corelight)

6.2.0-dev.512 | 2024-02-06 09:55:48 +0100

  * Do not default PoolSpec topics to the empty string. (Christian Kreibich, Corelight)

    Similar to `node_topic`, we already spell out a topic in the existing use and
    there's no obviously meaningful default value.

  * Do not default to proxy nodes in Broker::PoolSpec (Christian Kreibich, Corelight)

    This requires pool creation to spell out a spec explicitly, which the only code
    using these types already does. There's no reason for pools to automatically
    refer to proxies.

  * Add missing include for int32_t in DebugCmds.h (Tim Wojtulewicz)

  * CI: Remove emacs backup file of fedora-38 Dockerfile (Tim Wojtulewicz)

  * CI: Fix building of alpine image with regards to python packages (Tim Wojtulewicz)

6.2.0-dev.505 | 2024-02-05 17:18:41 -0800

  * Bump cluster testsuite to latest main (Christian Kreibich, Corelight)

6.2.0-dev.504 | 2024-02-05 12:50:48 -0700

  * minor updates and fixes for script optimization maintenance (Vern Paxson, Corelight)

6.2.0-dev.500 | 2024-02-02 15:27:23 -0700

  * Fix searching for libpcap installed by vcpkg (Tim Wojtulewicz)

  * Switch all of the conan configuration to vcpkg (Tim Wojtulewicz)

6.2.0-dev.497 | 2024-02-02 13:50:05 -0800

  * Fix markup typo in the Intel::seen_policy hook docstring [skip ci] (Christian Kreibich, Corelight)

6.2.0-dev.495 | 2024-02-01 11:22:03 -0700

  * Configure c-ares before libkqueue (Tim Wojtulewicz, Corelight)

    On platforms without a native libkqueue, c-ares is using the existing
    value for HAVE_KQUEUE that was set during the libkqueue setup. We don't
    pass the libkqueue information down to the c-ares cmake run so it won't
    have the paths or library when it builds.

  * Update 3rdparty submodule to update sqlite to 3.45.0 (Tim Wojtulewicz, Corelight)

  * Upgrade rapidjson to current upstream master (Tim Wojtulewicz, Corelight)

  * Upgrade c-ares to 1.26.0 (Tim Wojtulewicz, Corelight)

6.2.0-dev.490 | 2024-01-31 14:27:46 +0100

  * spicy/diff-remove-timestamp: Fix missing -e (Arne Welzel, Corelight)

    This got lost when converting to sed -E.

6.2.0-dev.487 | 2024-01-30 21:47:06 +0100

  * quic: Log client's source connection id, too. (Arne Welzel, Corelight)

    Seem reasonable give we log the server SCID. Interestingly, the Chromium
    examples actually have zero length (empty) source connection IDs. I wonder
    if that's part of their "protocol ossification avoidance" effort.

  * quic: Continue decryption attempts for later INITIAL packets (Arne Welzel, Corelight)

    The original logic stopped decrypting any INITIAL packets after the
    first. The Firefox/cloudflare pcaps actually show that the server
    replies with a QUIC INITAL packet containing just ACK frames and no
    CRYPTO frames. Only the second QUIC INITIAL packet from the server
    then contains the CRYPTO frames.

    There's no good reason to stop decryption attempts, either we succeed
    down the road and then stop, or we fail and raise analyzer violations.

  * quic: Rename set_conn() to set_session() (Arne Welzel, Corelight)

    set_conn() should be about the c$conn record. Most other base scripts
    for protocols use set_session(), so do the same.

6.2.0-dev.483 | 2024-01-29 10:30:06 +0100

  * Bump Spicy to current `main`. (Robin Sommer, Corelight)

6.2.0-dev.481 | 2024-01-26 17:13:53 -0800

  * Move GeoIP availability test in btests to `zeek-config --have-geoip` (Christian Kreibich, Corelight)

  * Fix MMDB::Lookup() to check result status correctly (Christian Kreibich, Corelight)

  * Add btest for succeeding/failing IPv4/IPv6 lookups (Christian Kreibich, Corelight)

  * Add an IPv6 range to the test MMDB DBs (Christian Kreibich, Corelight)

6.2.0-dev.476 | 2024-01-26 15:08:37 -0700

  * Force Windows CI to use OpenSSL 3.1.1 (Tim Wojtulewicz)

    There's something wrong with chocolatey's OpenSSL 3.2.0 package that
    causes cmake to not be able to find libcrypto even though it's clearly
    in the directory. Pinning to 3.1.1 fixes the build issue.

6.2.0-dev.473 | 2024-01-25 20:54:56 +0100

  * ZAM optimizations for record creation (Vern Paxson, Corelight)

    includes reworking of managing "auxiliary" information for ZAM instructions

  * testing/external: Revert commit hash for zeek-testing (Arne Welzel, Corelight)

6.2.0-dev.470 | 2024-01-25 12:25:57 +0100

  * GH-3256: Intel: Introduce Intel::seen_policy() hook (Arne Welzel, Corelight)

    This introduces a new hook into the Intel::seen() function that allows
    users to directly interact with the result of a find() call via external
    scripts.

    This should solve the use-case brought up by @chrisanag1985 in
    discussion #3256: Recording and acting on "no intel match found".

    @Canon88 was recently asking on Slack about enabling HTTP logging for a
    given connection only when an Intel match occurred and found that the
    Intel::match() event would only occur on the manager. The
    Intel::match_remote() event might be a workaround, but possibly running a
    bit too late and also it's just an internal "detail" event that might not
    be stable.

    Another internal use case revolved around enabling packet recording
    based on Intel matches which necessarily needs to happen on the worker
    where the match happened. The proposed workaround is similar to the above
    using Intel::match_remote().

    This hook also provides an opportunity to rate-limit heavy hitter intel
    items locally on the worker nodes, or even replacing the event approach
    currently used with a customized approach.

6.2.0-dev.468 | 2024-01-25 12:21:24 +0100

  * websocket: Fix opcode for continuation frames (Arne Welzel, Corelight)

    A continuation frame has the same type as the first frame, but that
    information wasn't used nor kept, resulting payload of continuation
    frames not being forwarded. The pcap was created with a fake Python
    server and a bit of message crafting.

6.2.0-dev.465 | 2024-01-24 15:25:16 -0800

  * Prepare Broker manager for broker::variant (Dominik Charousset)

6.2.0-dev.463 | 2024-01-24 23:04:38 +0100

  * testing/scripts: Use sed -E everywhere (Arne Welzel, Corelight)

    I'm always a bit worried to use sed -E anywhere, because the canonifiers
    give the impression it won't work everywhere consistently. My manpage says
    sed -E should be preferred for portability, so lets remove the
    sed -r / sed -E differentiation assuming it's just a thing from the past.

6.2.0-dev.461 | 2024-01-24 23:04:12 +0100

  * HTTP: Coverity std::move suggestion (Arne Welzel, Corelight)

  * WebSocket: Coverity std::move suggestion (Arne Welzel, Corelight)

  * SMTP/BDAT: Fix SonarQube reported issues (Arne Welzel, Corelight)

6.2.0-dev.456 | 2024-01-23 21:53:07 +0100

  * btest/smtp/bdat: Move tests into proper directory (Arne Welzel, Corelight)

  * BDAT: Harden parse_bdat_arg() (Arne Welzel, Corelight)

    There implementation assumed that arg is null terminated. Due to
    the ContentLineAnalyzer wrongly being in plain delivery mode, this
    assumption was violated. It shouldn't happen anymore, but protect
    from this anyhow.

  * SMTP: Reset ContentLineAnalyzer plain delivery on EndData() (Arne Welzel, Corelight)

    When resetting the BDAT state, we also need to switch the ContentLine
    analyzer back into line mode, otherwise we're feeding plain delivery
    data through ProcessLine(), possibly violating some assumptions about
    null termination.

    Do it for both ContentLineAnalyzers - only one of them will be in plain
    delivery mode anyhow, but we don't keep state which one it was.

  * SMTP: Add SMTP_IN_BDAT state (Arne Welzel, Corelight)

    Initially this reused SMTP_IN_DATA, but separating into SMTP_IN_BDAT
    to avoid spurious EndData() calls upon a server's reply. The client
    should usually continue to send the full in-flight chunk still.

6.2.0-dev.451 | 2024-01-23 12:40:53 -0700

  * Remove setting non-existent session history for IPTunnel (Tim Wojtulewicz, Corelight)

6.2.0-dev.449 | 2024-01-23 10:41:16 -0700

  * Change to use ToStdStringView() in a few other BIFs (Tim Wojtulewicz, Corelight)

  * Convert remove_prefix/suffix BIFs to use std::string_view (Tim Wojtulewicz, Corelight)

  * Rework starts_with BIF similarly to ends_with changes in 1649e3e7cc61d77645ca99a6068124e6fc11df2f (Tim Wojtulewicz, Corelight)

6.2.0-dev.445 | 2024-01-23 18:01:50 +0100

  * websocket: Handle breaking from WebSocket::configure_analyzer() (Arne Welzel, Corelight)

    ...and various nits from the review.

  * websocket: Address review feedback for BinPac code (Arne Welzel, Corelight)

    * Rename mask_ to masking_key_
    * Fold FrameHeaderFixed into FrameHeader directly
    * Drop WebSocket_FramePayloadUnmask type

    Thanks a bunch @ckreibich!

  * fuzzers: Add WebSocket fuzzer (Arne Welzel, Corelight)

    It immediately found an issue with &transient, but fairly stable thereafter.

    This is a separate fuzzer implementation as there's a custom Configure()
    call for the analyzer as well as disabling all other analyzers so we
    don't fuzz unrelated protocols.

  * websocket: Fix crash for fragmented messages (Arne Welzel, Corelight)

    The &transient attribute does not work well with $element as that won't
    be available within &until anymore apparently.

    Found after a few seconds building out the fuzzer.

  * websocket: Verify Sec-WebSocket-Key/Accept headers and review feedback (Arne Welzel, Corelight)

    Don't log them, they are random and arbitrary in the normal case. Users
    can do the following to log them if wanted.

        redef += WebSocket::Info$client_key += { &log };
        redef += WebSocket::Info$server_accept += { &log };

  * btest/websocket: Test for coalesced reply-ping (Arne Welzel, Corelight)

    Add a constructed PCAP where the HTTP/websocket server send a WebSocket
    ping message directly with the packet of the HTTP reply. Ensure this is
    interpreted the same as if the WebSocket message is in a separate packet
    following the HTTP reply.

    For the server side this should work, for the client side we'd need to
    synchronize suspend parsing the client side as we currently cannot quite
    know whether it's a pipelined HTTP request following, or upgraded protocol
    data and we don't have "suspend parsing" functionality here.

  * HTTP/CONNECT: Also weird on extra data in reply (Arne Welzel, Corelight)

  * HTTP/Upgrade: Weird when more data is available (Arne Welzel, Corelight)

    After an HTTP upgrade to another protocol, create a weird if the packet
    that contains the HTTP reply *also* contains some additional data
    belonging to the upgraded to protocol already.

  * ContentLine: Add GetDeliverStreamRemainingLength() accessor (Arne Welzel, Corelight)

    Helper to get information from the ContentLine analyzer about
    bytes still pending to be delivered. In certain cases this can
    be a signal for weirdness.

  * HTTP: Drain event queue after instantiating upgrade analyzer (Arne Welzel, Corelight)

    With configurability through script-land comes the draw back
    that we actually need to execute event handlers in the middle
    of the parsing process: This might not be the best model, but
    the script-side configurability it enables is kind of nice.

    This explicit call only matters here when the HTTP reply is
    directly followed by some WebSocket message data within the
    same network packet, otherwise the queue is drained once the
    packet has been completely processed anyhow.

  * btest/http: Explain switching-protocols test change as comment (Arne Welzel, Corelight)

    DPD enables HTTP based on the content of the WebSocket frames. However,
    it's not HTTP, the protocol is x-kaazing-handshake and the server sends
    some form of status/acknowledge to the client first, so the HTTP and the
    HTTP analyzer receives that as the first bytes of the response and
    bails, oh well.

  * GH-3424: WebSocket: Introduce new analyzer and log (Arne Welzel, Corelight)

    This adds a new WebSocket analyzer that is enabled with the HTTP upgrade
    mechanism introduced previously. It is a first implementation in BinPac with
    manual chunking of frame payload. Configuration of the analyzer is sketched
    via the new websocket_handshake() event and a configuration BiF called
    WebSocket::__configure_analyzer(). In short, script land collects WebSocket
    related HTTP headers and can forward these to the analyzer to change its
    parsing behavior at websocket_handshake() time. For now, however, there's
    no actual logic that would change behavior based on agreed upon extensions
    exchanged via HTTP headers (e.g. frame compression). WebSocket::Configure()
    simply attaches a PIA_TCP analyzer to the WebSocket analyzer for dynamic
    protocol detection (or a custom analyzer if set). The added pcaps show this
    in action for tunneled ssh, http and https using wstunnel. One test pcap is
    Broker's WebSocket traffic from our own test suite, the other is the
    Jupyter websocket traffic from the ticket/discussion.

    This commit further adds a basic websocket.log that aggregates the WebSocket
    specific headers (Sec-WebSocket-*) headers into a single log.

    Closes #3424

  * HTTP: Add mechanism to instantiate Upgrade analyzer (Arne Welzel, Corelight)

    When a HTTP upgrade request/reply is detected, lookup an analyzer tag
    from HTTP::upgrade_analyzers, or if nothing is found, attach PIA_TCP.

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

6.2.0-dev.430 | 2024-01-22 10:36:11 -0700

  * Make BIFs just return ValPtr directly instead of BifReturnVal (Tim Wojtulewicz, Corelight)

  * Drop support for subscribing to bro/ topics (Tim Wojtulewicz, Corelight)

6.2.0-dev.426 | 2024-01-21 19:31:02 -0700

  * MIME: Cap nested MIME analysis depth to 100 (Arne Welzel, Corelight)

    OSS-Fuzz managed to produce a MIME multipart message construction with
    thousands of nested entities (or that's what Zeek makes out of it anyhow).
    Prevent such deep analysis by capping at a nesting depth of 100,
    preventing unnecessary resource usage. A new weird named exceeded_mime_max_depth
    is reported when this limit is reached.

    This change reduces the runtime of the OSS-Fuzz reproducer from ~45 seconds
    to ~2.5 seconds.

    The test PCAP was produced from a Python script using the email package
    and sending the rendered version via POST to a HTTP server.

6.2.0-dev.423 | 2024-01-19 21:18:44 +0100

  * SMTP/BDAT: Use strtoull and bail on UULONG_MAX values (Arne Welzel, Corelight)

  * SMTP/BDAT: Fix int/int64_t/uint64_t confusion (Arne Welzel, Corelight)

    The BDAT analyzer should be supporting uint64_t sized chunks reasonably well,
    but the ContentLine analyzer does not, And also, I totally got types for
    RemainingChunkSize() and in DeliverStream() wrong, resulting in overflows
    and segfaults when very large chunk sizes were used.

    Tickled by OSS-Fuzz. Actually running the fuzzer locally only took a
    few minutes to find the crash, too. Embarrassing.

  * SMTP: Reject BDAT chunks larger than int64_t's max value (Arne Welzel, Corelight)

    The ContentLine analyzer does not support uint64_t.

6.2.0-dev.419 | 2024-01-19 10:25:52 -0800

  * Default to setting ZEEK_HAVE_JAVASCRIPT=no in CMakeLists.txt (Christian Kreibich, Corelight)
  * Show --disable-javascript in `configure --help` (Christian Kreibich, Corelight)
  * Show AF_PACKET support status in cmake output, and sort features (Christian Kreibich, Corelight)
  * Add feature tests for AF_PACKET, GeoIP, and JavaScript to zeek-config (Christian Kreibich, Corelight)

6.2.0-dev.414 | 2024-01-19 12:27:39 +0100

  * GH-3177: Merge branch 'topic/bbannier/issue-3177' (Benjamin Bannier, Corelight)

  * GH-3177: Make sure Spicy symbols are available. (Benjamin Bannier, Corelight)

6.2.0-dev.412 | 2024-01-18 16:16:50 -0800

  * Fix a zeek_init -> zeek_done confusion in a docstring (Christian Kreibich, Corelight)
  * Fix typo in docstring (Christian Kreibich, Corelight)

6.2.0-dev.408 | 2024-01-17 10:00:06 -0700

  * Enable darwin builds for zeek-security repo (Tim Wojtulewicz, Corelight)

6.2.0-dev.405 | 2024-01-16 12:07:11 -0700

  * Squeeze a bit more performance out of the ends_with bif (Tim Wojtulewicz)

6.2.0-dev.403 | 2024-01-16 18:03:54 +0100

  * SMTP: No state update for bad BDAT commands (Arne Welzel, Corelight)

    OSS-Fuzz found that providing an invalid BDAT line would tickle an
    assert in UpdateState(). The BDAT state was never initialized, but
    within UpdateState() that was expected.

    This also removes the AnalyzerViolation() call for bad BDAT commands
    and instead raises a weird. The SMTP analyzer is very lax and not triggering
    the violation allows to parse the server's response to such an invalid
    command.

    PCAP files produced by a custom Python SMTP client against Postfix.

  * SMTP/BDAT: Harden BDAT argument parsing a bit (Arne Welzel, Corelight)

    If the size is followed by something and it's not " LAST", treat it
    as an error.

6.2.0-dev.400 | 2024-01-16 13:08:32 +0000

  * Fix ssl-log-ext omitting data in some cases (Johanna Amann, Corelight)

    ssl-log-ext had a bug that caused data present in the SSL connection to
    not be logged in some cases. Specifically, the script relied on the base
    ssl script to initialize some data structures; however, this means that
    protocol messages that arrive before a message is handled by the base
    ssl script are not logged.

    This commit changes the ssl-log-ext script to also initialize the data
    structures; now messages are correctly included in the log in all cases.

6.2.0-dev.398 | 2024-01-15 15:18:32 +0100

  * Session/Conn: Follow-up fix for hist_seen and history lift (Arne Welzel, Corelight)

    Borked in commit b4e86f28b8adcbd45b9008dfa1818c6eb2fbb4d0.

    We don't need to initialize std::string explicitly either.

    Thanks coverity!

6.2.0-dev.396 | 2024-01-15 15:17:45 +0100

  * ZAM speedup for constructing empty vectors (Vern Paxson, Corelight)

  * fixes for ZAM optimization of "switch" statements (Vern Paxson, Corelight)

  * BTests to catch regressions for recent ZAM fixes (Vern Paxson, Corelight)

  * "-a zam" BTest baseline update for recent changes (Vern Paxson, Corelight)

  * fix for needing to always flush optimization information for identifiers (Vern Paxson, Corelight)

  * fix for logic bug in ldap base script (Vern Paxson, Corelight)

  * better name for key variable in script optimization (Vern Paxson, Corelight)

  * ZAM fix for tracking variable usage (Vern Paxson, Corelight)

  * ZAM fixes for "for" loops that are only used to choose an element from a table/set (Vern Paxson, Corelight)

  * ZAM fixes for loops indexed with variables not used in the loop body (Vern Paxson, Corelight)

  * fix for ZAM location tracking - more extensive changes are pending (Vern Paxson, Corelight)

  * fixes for ZAM's special-casing of that "cat" BiF (Vern Paxson, Corelight)

  * some fixes for ZAM memory management (Vern Paxson, Corelight)

  * streamlining of some script optimization APIs (Vern Paxson, Corelight)

  * fixes for initializations of "-O gen-C++" script compilations (Vern Paxson, Corelight)

  * script optimization fixes for "concretizing" vector-of-any's (Vern Paxson, Corelight)

6.2.0-dev.378 | 2024-01-12 10:48:33 +0100

  * btest/smtp: Test with smtp-bdat-pipeline-8bitmime.pcap (Arne Welzel, Corelight)

    Not sure about the origin of this pcap, so adding it in a separate
    commit, but it seems a nice real-world test case.

  * GH-3264: SMTP: Add BDAT support (Arne Welzel, Corelight)

6.2.0-dev.375 | 2024-01-12 09:27:58 +0100

  * Modernize various C++/Zeek-isms in the MMDB code. (Christian Kreibich, Corelight)

  * Fix MMDB code to re-open explicitly opened DBs correctly (Christian Kreibich, Corelight)

    The filename from which a DB first gets opened (either via an explicitly
    specified filename, or via the path sequence now configurable at the script
    layer) is now "sticky", meaning re-opening won't switch to a different file.

    This was easiest by moving most state into the MMDB class itself. The previous
    approach of tracking the two DB instances via a smart pointer and blowing the
    pointed-to objects away as needed is now instead one of two objects fixed over
    the lifetime of Zeek, able to open/close/reopen their underlying Maxmind DBs.

    The MMDB class now only has one Lookup() method since there was no need to break
    them apart -- it saves the return of a MMDB_lookup_result_s over the stack and
    there's no need for throwing an exception.

  * Add btest to verify behavior of re-opened MMDBs opened directly via BIFs (Christian Kreibich, Corelight)

    The mmdb_open_location_db() and mmdb_open_asn_db() BiFs were untested, and Zeek
    has a bug that makes any DBs opened that way fall back to looking up DBs via the
    existing script-level config mechanism (via mmdb_dir), which is at least
    unexpected and might well be unconfigured if somebody uses the direct BiFs.

  * Simplify MMDB code by moving more lookup functionality into MMDB class (Christian Kreibich, Corelight)

  * Move MMDB logic out of mmdb.bif and into MMDB.cc/h. (Christian Kreibich, Corelight)

    This does not change the implementation except for some light renaming where
    things are now naturally scoped within MMDB.cc.

  * Fix mmdb.temporary-error testcase when MMDBs are installed on system (Christian Kreibich, Corelight)

    The test would previously fail in settings where the user has Maxmind DBs
    installed in the hardwired system locations, because the fallback logic still
    picked those up.

  * Adapt MMDB BiF code to new script-layer variables (Christian Kreibich, Corelight)

  * Update btest baselines to reflect introduction of mmdb.bif (Christian Kreibich, Corelight)

  * Move MaxMind/GeoIP BiF functionality into separate file (Christian Kreibich, Corelight)

  * Provide script-level configurability of MaxMind DB placement on disk (Christian Kreibich, Corelight)

    This lifts the list of fallback directories in which Zeek will look for Maxmind
    DBs into the script layer, and makes the names of the DB files themselves
    (previously hardwired) configurable as well.

    This does not yet change the in-core code; that commit follows.

  * Sort toplevel .bif list in CMakeLists (Christian Kreibich, Corelight)

    Not important, but here it is safe to do so and the list is getting lengthy.

6.2.0-dev.362 | 2024-01-11 20:17:45 +0100

  * GH-3540: Known: Keep &create_expire on local tables/sets valid (Arne Welzel, Corelight)

    After switching the known scripts away from broker stores, the
    &create_expire value of the local tables/sets of the known scripts
    wasn't in effect due to Cluster::node_up() and Cluster::node_down()
    re-assigning these without keeping the &create_expire attribute
    intact. This broke the "log hosts every 24h" behavior.

6.2.0-dev.360 | 2024-01-11 12:36:11 +0100

  * Document upcoming breaking change for OpaqueVal (Dominik Charousset, Corelight)

  * Integrate review feedback (Dominik Charousset, Corelight)

  * Fix formatting (Dominik Charousset, Corelight)

  * Backward compatibility for OpaqueVal serialization (Dominik Charousset, Corelight)

    External plugins depend on the API for `OpaqueVal`. This set of changes
    brings back the previous signature for the `Serialize` and `Unserialize`
    member functions. The new set of functions that operate on the recently
    added `BrokerData` API were renamed accordingly and use a `Data` suffix to
    distinguish between the old and new interface.

    For the transition period, `OpaqueVal` now has two "sets" of
    serialization functions: old and new (using the suffix). By default, the
    new functions call the old API and then convert to the new types. Hence,
    plugins that override the old set of member functions will continue to
    work. New code should only override the new set of functions.

    Since the macro `DECLARE_OPAQUE_VALUE` (a convenience macro for adding a
    default set of member functions to a subtype of `OpaqueVal`) might be
    used by 3rd parties, the macro has been "restored" to its previous
    behavior, i.e., it will override the old set of member functions. The
    new macro `DECLARE_OPAQUE_VALUE_V2` is similar but overrides the new set
    of functions instead.

    The class `BloomFilter` uses the same member function signatures as
    `OpaqueVal` for serialization. Hence, the same old/new split was
    implemented to keep the APIs consistent.

6.2.0-dev.355 | 2024-01-11 10:59:05 +0100

  * NEWS: Update news for tunnel depth changes (Arne Welzel, Corelight)

  * tunnels: Add 'X' to history when reaching Tunnel::max_depth (Arne Welzel, Corelight)

  * Session/TCP/UDP: Reserve HIST_UNKNOWN_PKT mask (Arne Welzel, Corelight)

    This is meant to be used for a new 'X' code in the history in scenarios when
    packets are knowingly not processed or an unexpected unknown situation
    is recognized.

    Usually, these situations are currently reported via weirds or analyzer violations,
    but being able to include it in the history field allows them to be more visible.

    Will be used for exceeding tunnel depths first.

  * Conn: Deprecated AppendAddl (Arne Welzel, Corelight)

  * Conn/Session: Lift history logic into Session (Arne Welzel, Corelight)

    This should allow to mangle a Session's history also from packet
    analyzers without necessarily knowing the concrete connection type.

    Given Connection is a subclass of Session, I don't think this
    changes much.

  * GH-3439: init-bare: Default Tunnel::max_depth to 4 (Arne Welzel, Corelight)

    In AWS GLB environments, the max_depth of 2 is easily reached due to packets
    being encapsulated with GENEVE and VXLAN [1]. Any additional encapsulation
    layer causes Zeek raise a weird and ignore the inner traffic. Bump the default
    maximum depth to 4, while not common it's not unusual either to observe
    this in the wild.

    [1] https://docs.aws.amazon.com/vpc/latest/mirroring/traffic-mirroring-packet-formats.html

6.2.0-dev.347 | 2024-01-10 12:32:43 -0700

  * Use std::move in return values from bif methods to avoid copies (Tim Wojtulewicz, Corelight)

  * Use bool return values instead of int in a couple zeek.bif static methods (Tim Wojtulewicz, Corelight)

6.2.0-dev.344 | 2024-01-10 14:07:18 +0100

  * quic: Handle and log unhandled_version (Arne Welzel, Corelight)

  * quic: Support decryption of a few more versions (Arne Welzel, Corelight)

  * parse: Support @if conditionals in record definitions (Arne Welzel, Corelight)

6.2.0-dev.339 | 2024-01-09 09:15:39 +0100

  * Dict: Invalidate iterators during Clear() (Arne Welzel, Corelight)

    If we don't do this, a RobustDictIterator owned by someone else holds
    dangling pointers to the Dictionary that is being cleared.

    Closes #3523

6.2.0-dev.337 | 2024-01-08 14:10:44 -0700

  * Merge branch 'topic/timw/werror' (Tim Wojtulewicz, Corelight)

    * topic/timw/werror:
      CI: Remove unused openssl30_config
      CPP-gen: Don't emit extra braces if only one element
      Use <poll.h> instead of <sys/poll.h>
      ZAM: Create ListValPtr directly instead of a stack object
      Bump zeekjs to pick up dprintf warning fix
      Avoid unused-result warning in Supervisor
      Update src/3rdparty submodule to fix sprintf warning in modp
      Fix warning with attribute string lookup
      Set -Werror / /WX via target_compile_options to force warnings as errors

  * CI: Remove unused openssl30_config (Tim Wojtulewicz, Corelight)

  * CPP-gen: Don't emit extra braces if only one element (Tim Wojtulewicz, Corelight)

  * Use <poll.h> instead of <sys/poll.h> (Tim Wojtulewicz, Corelight)

  * ZAM: Create ListValPtr directly instead of a stack object (Tim Wojtulewicz, Corelight)

  * Bump zeekjs to pick up dprintf warning fix (Tim Wojtulewicz, Corelight)

  * Avoid unused-result warning in Supervisor (Tim Wojtulewicz, Corelight)

  * Update src/3rdparty submodule to fix sprintf warning in modp (Tim Wojtulewicz, Corelight)

  * Fix warning with attribute string lookup (Tim Wojtulewicz, Corelight)

  * Set -Werror / /WX via target_compile_options to force warnings as errors (Tim Wojtulewicz, Corelight)

6.2.0-dev.327 | 2024-01-08 10:04:49 -0700

  * Update broker to pick up Windows 2-minute exception fix (Tim Wojtulewicz, Corelight)

6.2.0-dev.322 | 2024-01-05 16:07:04 +0100

  * ldap: Fix substring filter parsing and rendering (Arne Welzel, Corelight)

    The initial (prefix) and final (suffix) strings are specified individually
    with a variable number of "any" matches that can occur between these.
    The previous implementation assumed a single string and rendered it
    as *<string>*.

    Reported and PCAP provided by @martinvanhensbergen, thanks!

    Closes zeek/spicy-ldap#27

6.2.0-dev.320 | 2024-01-05 14:43:59 +0100

  * quic: tests: Require have-spicy (Arne Welzel, Corelight)

  * quic: analyzer: Recognize and report unknown versions better (Arne Welzel, Corelight)

    This makes the analyzer.log entry more informative by including the
    actual version and also allows to handle this scenario in script land
    if needed.

  * quic: tests: Add QUIC v2 test cases (Arne Welzel, Corelight)

    Produced using examples from the go-quic project, patching the clients
    to force QUIC v2.

  * quic: analyzer: Support QUIC v2 (Arne Welzel, Corelight)

    QUIC v2 changed the version *and* the packet type enumeration to prevent
    protocol ossification. Use an intermediary unit to handle the difference.

  * quic: decrypt_crypto: Support QUIC v2 (Arne Welzel, Corelight)

    Attempt to refactor in order to re-use common code between the two
    versions.

6.2.0-dev.314 | 2024-01-04 16:40:00 +0100

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

    This bump mostly brings in small incremental performance improvements
    for runtime throughput for Spicy-generated parsers. With a large
    internal analyzer at Corelight these changes improve throughput by
    around (3.8±1.2)%.

6.2.0-dev.311 | 2024-01-03 12:29:23 +0100

  * Update external baselines (Arne Welzel, Corelight)

  * ldap: Use scalar values in logs where appropriate (Arne Welzel, Corelight)

    Skimming through the RFC, the previous approach of having containers for most
    fields seems unfounded for normal protocol operation. The new weirds could just
    as well be considered protocol violations. Outside of duplicated or missed data
    they just shouldn't happen for well-behaved client/server behavior.
    Additionally, with non-conformant traffic it would be trivial to cause
    unbounded state growth and immense log record sizes.

    Unfortunately, things have become a bit clunky now.

    Closes #3504

  * ldap: Rename LDAP::search_result to LDAP::search_result_entry (Arne Welzel, Corelight)

    To avoid confusion potential confusion between SearchResultEntry
    and SearchResultDone.

6.2.0-dev.307 | 2024-01-03 12:25:03 +0100

  * segment_profiling: Remove SegmentProfiler and load_sample event (Arne Welzel, Corelight)

    While it seems interesting functionality, this hasn't been documented,
    maintained or knowingly leveraged for many years.

    There are various other approaches today, too:

    * We track the number of event handler invocations regardless of
      profiling. It's possible to approximate a load_sample event by
      comparing the result of two get_event_stats() calls. Or, visualize
      the corresponding counters in a Prometheus setup to get an idea of
      event/s broken down by event names.

    * HookCallFunction() allows to intercept script execution, including
      measuring the time execution takes.

    * The global call_stack and g_frame_stack can be used from plugins
      (and even external processes) to walk the Zeek script stack at certain
      points to implement a sampling profiler.

    * USDT probes or more plugin hooks will likely be preferred over Zeek
      builtin functionality in the future.

    Relates to #3458

6.2.0-dev.305 | 2024-01-03 09:38:17 +0000

  * Avoid extra copies in threading_field_to_data (Dominik Charousset)

    Passing the `broker::data` paramter for `threading_field_to_data` by
    value forces copies. However, the function only needs const access to
    its parameter. Hence, taking the parameter as const-ref is the logical
    choice and avoids unnecessary copies of individual data fields.

6.2.0-dev.302 | 2024-01-02 11:13:17 -0700

  * smb: Fix &read_expire not in effect due to &default=string_set() usage (Arne Welzel, Corelight)

    The SMB::State$recent_files field is meant to have expiring entries.
    However, due to usage of &default=string_set(), the &read_expire
    attribute is not respected causing unbounded state growth. Replace
    &default=string_set() with &default=set().

    Thanks to ya-sato on Slack for reporting!

    Related: zeek/zeek-docs#179, #3513.

6.2.0-dev.300 | 2023-12-16 17:57:49 +0100

  * TableType: Convert table_hash unique_ptr (Arne Welzel, Corelight)

    Cannot remove the destructor as otherwise the compiler attempts to create
    its implementation in Type.h where CompositeHash isn't a complete type
    yet and std::unique_ptr's delete fails to be instantiated.

  * streamlining of constructing script-level tables (Vern Paxson, Corelight)

6.2.0-dev.297 | 2023-12-15 14:59:34 -0700

  * Update broker submodule (Dominik Charousset, Corelight)

6.2.0-dev.295 | 2023-12-15 13:10:27 -0700

  * improve search-ability in  zeek.spicy (Smoot)

6.2.0-dev.292 | 2023-12-15 18:58:00 +0100

  * ci/btest: Remove ZEEK_PROFILER_FILE from btest.cfg, set in ci/test.sh explicitly (Arne Welzel, Corelight)

    The produced coverage files are of little use in current local workflows
    and usually just end-up taking up disk space. ZEEK_PROFILER_FILE can be
    set explicitly if there's a one-off need to produce these locally, too.

  * ci: Remove ZEEK_CI_DISABLE_SCRIPT_PROFILING logic (Arne Welzel, Corelight)

    To my knowledge this doesn't make a huge difference anymore and locally
    I'm only using ASAN builds. It's not actually too slow.

6.2.0-dev.288 | 2023-12-14 20:03:07 -0800

  * Bump binpac, zeek-archiver, and zeek-aux submodules. (Christian Kreibich, Corelight)

6.2.0-dev.287 | 2023-12-14 19:55:28 -0800

  * CI updates (Christian Kreibich, Corelight)

    - Move Debian variations from 11 to 12
    - Bump Ubuntu 23.04 to 23.10
    - Drop openSUSE Leap 15.4, about to EOL.
    - FreeBSD 14 is out now, 12 is about to EOL.
    - distro EOL comment tweaks
    - drop Fedora 37, add Fedora 39

6.2.0-dev.279 | 2023-12-15 00:11:59 +0000

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

6.2.0-dev.278 | 2023-12-14 10:28:27 +0100

  * rule-parse: Remove [event_name] syntax, deprecate msg as identifier (Arne Welzel, Corelight)

    As suggested by Robin. Thanks.

  * Bump zeekctl submodule to pull in Python 3.12 ConfigParser fix. (Christian Kreibich, Corelight)

6.2.0-dev.275 | 2023-12-12 09:52:00 +0100

  * recent BTests that should be skipped when using -O gen-C++ (Vern Paxson, Corelight)

  * expanded ZAM maintenance notes & support scripts (Vern Paxson, Corelight)

  * script optimization tracking of functions called by event engine or indirectly (Vern Paxson, Corelight)

  * memory-handling fixes for information associated with low-level ZAM instructions (Vern Paxson, Corelight)

  * fix for -O C++ lambda functions reporting errors/warnings (Vern Paxson, Corelight)

  * revert problems with profiling attributes introduced by recent script-opt PR (Vern Paxson, Corelight)

  * script optimization fixes for pattern tables (Vern Paxson, Corelight)

  * regularized (some) types of pointers used in script optimization (Vern Paxson, Corelight)

  * splitting off script optimization CSE into its own source files (Vern Paxson, Corelight)

  * some very minor tidying of script optimization code/documentation (Vern Paxson, Corelight)

  * fix for Trigger's whose termination leads to deleting other Trigger's (Vern Paxson, Corelight)

  * bug fix for delayed logging (Vern Paxson, Corelight)

6.2.0-dev.262 | 2023-12-11 13:11:09 +0100

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

6.2.0-dev.259 | 2023-12-08 11:54:03 -0700

  * Audit creation of temporary strings in Spicy bindings (Benjamin Bannier, Corelight)

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

6.2.0-dev.256 | 2023-12-08 10:01:10 -0700

  * incorporate latest version of gen-zam to correctly generate indirect calls (Vern Paxson, Corelight)

  * added sub-directory for tracking ZAM maintenance issues (Vern Paxson, Corelight)

  * BTest to stress-test AST optimizer's assessment of side effects (Vern Paxson, Corelight)

  * reworked AST optimizers analysis of side effects during aggregate operations & calls (Vern Paxson, Corelight)

  * script optimization support for tracking information associated with BiFs/functions (Vern Paxson, Corelight)

  * fix for AST analysis of inlined functions (Vern Paxson, Corelight)

  * improved AST optimizer's analysis of variable usage in inlined functions (Vern Paxson, Corelight)

  * new method for Stmt nodes to report whether they could execute a "return" (Vern Paxson, Corelight)

  * bug fixes for indirect function calls when using ZAM (Vern Paxson, Corelight)

  * minor fixes for script optimization, exporting of attr_name, script layout tweak (Vern Paxson, Corelight)

6.2.0-dev.245 | 2023-12-06 18:42:25 +0100

  * Bump cmake submodule (Arne Welzel, Corelight)

  * iosource: Specify libkqueue dependency for subdir lib (Arne Welzel, Corelight)

    Since a while my build has been spilling the following warnings:

        [18/1687] Building C object auxil/c-ares/src/lib/CMakeFiles/c-ares.dir/ares__addrinfo2hostent.c.o
        cc1: warning: zeek/prod-build/libkqueue-build/include: No such file or directory [-Wmissing-include-dirs]

    My take is that FindKqueue extends the include directories globally and
    tickles this warning because c-ares is built first. Grepping around,
    <sys/event.h> is only included in iosource/Manager.cc, so we should
    be able to reduce the exposure just to the iosource subdir.

  * Update zeek-aux submodule (Arne Welzel, Corelight)

6.2.0-dev.241 | 2023-12-06 11:41:29 +0100

  * spicy/runtime-support: Use Vector::unsafeBegin() for args iteration (Arne Welzel, Corelight)

  * Bump spicy submodule (Arne Welzel, Corelight)

6.2.0-dev.238 | 2023-12-05 16:00:56 +0100

  * SegmentProfiler: Do not initialize initial_rusage (Arne Welzel, Corelight)

    We use the SegmentProfiler in quite a few hot places and the memset of
    the rusage structure (144bytes here) can show up significantly even if
    the segment profiler itself isn't used.

    Relates to #3485.

  * EventMgr: Remove queue_flare, use GetNextTimeout() instead (Arne Welzel, Corelight)

    It can be visible overhead to call write() on the underlying pipe of the
    EventMgr's flare whenever the first event is enqueued during an IO loop
    iteration. Particularly in scenarios where there's about 1 event per packet
    for long lived connections and script-side event processing is fast.

    Given the event manager is drained anyhow at the end of the main loop, this
    shouldn't be needed. In fact, the EventMgr.Process() method is basically
    a stub. The one reason it is needed is when more events are enqueued during
    a drain. That, however, can be dealt with by implementing GetNextTimeout()
    to return 0.0 when there's more events queued. This way the main-loop's poll
    timeout is 0.0 and it'll continue immediately.

    This also allows to removes some extra code and drop the recently introduced
    InitPostFork() addition: Without a pipe, there's no need to recreate it.

  * UpdateConnVal: Avoid FieldOffset() calls (Arne Welzel, Corelight)

    These can be significant if a lot of new connections and or events
    are created for which an existing conn val needs updating and otherwise
    things are very fast.

  * RuleActionMIME: Switch to std::string (Arne Welzel, Corelight)

    And return const std::string& from GetMIME(). Probably not at all performance
    relevant, but while I'm already here.

  * signatures: Support custom event via [event_name] syntax (Arne Welzel, Corelight)

    This change allows to specify a per signature specific event, overriding
    the default signature_match event. It further removes the message
    parameter from such events if not provided in the signature.

    This also tracks the message as StringValPtr directly to avoid
    allocating the same StringVal for every DoAction() call.

    Closes #3403

  * zeek-setup: Exit when rule loading tickles reporter errors (Arne Welzel, Corelight)

    With custom events for signatures, Reporter::error() may be invoked
    while loading them. Early exit in case that happens. We could continue
    and either disable the signatures or fallback to the default
    signature_match() event, but not sure that would be obviously better.

  * rule-scan: Copy yytext strings (Arne Welzel, Corelight)

    When trying to use TOK_IDENT and TOK_STRING in a single rule, that
    resulted in "corrupt" strings.

    https://www.gnu.org/software/bison/manual/html_node/Strings-are-Destroyed.html

6.2.0-dev.229 | 2023-12-04 12:32:17 -0800

  * Add facade types to avoid using raw Broker types (Dominik Charousset, Corelight)

6.2.0-dev.226 | 2023-12-04 18:57:29 +0100

  * logging: Fix coverity std::move suggestions (Arne Welzel, Corelight)

  * logging/Manager: Fix coverity null-deref (Arne Welzel, Corelight)

    Prior code assumed non-null stream given the active_write_ctx matches,
    but please coverity.

  * Fix some compiler warnings in logging::Manager (Tim Wojtulewicz, Corelight)

  * logging/Manager: Fix token_val->AsCount() in debug logging (Arne Welzel, Corelight)

    Second UBSAN error triggered from log delay merge.

  * logging/Manager: Fix AsTime() to AsInterval() (Arne Welzel, Corelight)

    Found by UBSAN after merge of log delay branch.

6.2.0-dev.218 | 2023-12-01 12:04:32 +0100

  * logging: ref() to record_ref() renaming (Arne Welzel, Corelight)

  * logging: Fix typos from review (Arne Welzel, Corelight)

  * logging/Manager: Make LogDelayExpiredTimer an implementation detail (Arne Welzel, Corelight)

    The only reason this was a private component of Manager was to access
    the Stream's function. Use a generic callback and a lambda to avoid
    that exposure.

  * logging/WriteToFilters: Use range-based for loop (Arne Welzel, Corelight)

  * testing/btest: Log::delay() from JavaScript (Arne Welzel, Corelight)

  * NEWS: Entry for delayed log writes (Arne Welzel, Corelight)

  * Bump doc submodule to branch (Arne Welzel, Corelight)

  * logging: Do not keep delay state persistent (Arne Welzel, Corelight)

    If Log::remove_stream() and Log::create_stream() is called for a stream,
    do not restore the previously used max delay or max queue size.

  * logging: delay documentation polishing (Arne Welzel, Corelight)

    Based on PR feedback.

  * logging: Better error messages for invalid Log::delay() calls (Arne Welzel, Corelight)

    Add a test for Log::delay() usage within filter policy hooks, too.

  * logging/Manager: Implement DelayTokenType as an actual opaque (Arne Welzel, Corelight)

    With a bit of tweaking in the JavaScript plugin to support opaque types, this
    will allow the delay functionality to work there, too.

    Making the LogDelayToken an actual opaque seems reasonable, too. It's not
    supposed to be user inspected.

  * logging: Implement get_delay_queue_size() (Arne Welzel, Corelight)

    Primarily for introspection given that re-delaying may exceed
    queue sizes.

  * logging: Introduce Log::delay() and Log::delay_finish() (Arne Welzel, Corelight)

    This is a verbose, opinionated and fairly restrictive version of the log delay idea.
    Main drivers are explicitly, foot-gun-avoidance and implementation simplicity.

    Calling the new Log::delay() function is only allowed within the execution
    of a Log::log_stream_policy() hook for the currently active log write.

    Conceptually, the delay is placed between the execution of the global stream
    policy hook and the individual filter policy hooks. A post delay callback
    can be registered with every Log::delay() invocation. Post delay callbacks
    can (1) modify a log record as they see fit, (2) veto the forwarding of the
    log record to the log filters and (3) extend the delay duration by calling
    Log::delay() again. The last point allows to delay a record by an indefinite
    amount of time, rather than a fixed maximum amount. This should be rare and
    is therefore explicit.

    Log::delay() increases an internal reference count and returns an opaque
    token value to be passed to Log::delay_finish() to release a delay reference.
    Once all references are released, the record is forwarded to all filters
    attached to a stream when the delay completes.

    This functionality separates Log::log_stream_policy() and individual filter
    policy hooks. One consequence is that a common use-case of filter policy hooks,
    removing unproductive log records, may run after a record was delayed. Users
    can lift their filtering logic to the stream level (or replicate the condition
    before the delay decision). The main motivation here is that deciding on a
    stream-level delay in per-filter hooks is too late. Attaching multiple filters
    to a stream can additionally result in hard to understand behavior.

    On the flip side, filter policy hooks are guaranteed to run after the delay
    and can be used for further mangling or filtering of a delayed record.

  * logging/Manager: zeek::detail'ify (Arne Welzel, Corelight)

    Introducing zeek::logging::detail requires detail:: references to be
    qualified as preparation.

  * logging/Manager: Split Write() (Arne Welzel, Corelight)

    If we delay in the stream policy hook, we'll need to resume writing
    to the attached filters later on. Prepare for that by splitting out
    the filter processing.

  * Timer: Add LOG_DELAY_EXPIRE timer type (Arne Welzel, Corelight)

  * Ascii: Remove extra include (Arne Welzel, Corelight)

    Should not be needed, not sure what I was thinking.

6.2.0-dev.200 | 2023-11-30 10:16:13 +0100

  * GH-3473: OpaqueVal: Register TelemetryVals, too (Arne Welzel, Corelight)

    Even if they are not serializable, OpaqueMgr::TypeID() is called
    during BuildJSON() for them and that previously just aborted.

6.2.0-dev.198 | 2023-11-29 11:51:56 +0100

  * Update zeekjs submodule (Arne Welzel, Corelight)

    Contains the following fixes:

        2da4abe Types: Add support for opaque types
        1f1093f Types: Cast internal field to v8::Value
        67e225c Plugin: Avoid creating Exprs at runtime

6.2.0-dev.196 | 2023-11-28 13:40:54 -0700

  * Add some uses of std::move in constructors and simple functions for pass-by-value arguments (Tim Wojtulewicz, Corelight)

  * Avoid creating a few temporary values to avoid copy operations (Tim Wojtulewicz, Corelight)

  * Change function return types to more concise types where possible (Tim Wojtulewicz, Corelight)

    This helps avoid needing to call std::move on return values when we know the
    type of value is fixed.

6.2.0-dev.192 | 2023-11-28 15:11:42 +0000

  * TLS: Update cipher consts and keyexchange parsing (Johanna Amann, Corelight)

    Update cipher consts.

    Furthermore some past updates have been applied to scriptland, but it
    was not considered that some of these also have to be applied to binpac
    code, to be able to correcly parse the ServerKeyExchange message.

6.2.0-dev.189 | 2023-11-27 21:35:26 +0100

  * OCSP: Open-code unknown revoke reason strings (Arne Welzel, Corelight)

    OpenSSL 3.2.0 knows about more reasons. Add some backwards compatibility.

    Reference: https://github.com/openssl/openssl/commit/1c8a7f5091e2c5aebc043be86bcbedc6947e1c6f

6.2.0-dev.187 | 2023-11-23 17:19:51 +0000

  * Spicy: allow providing file id in zeek::file_begin (Johanna Amann, Corelight)

    Allow spicy parsers to generate their own file IDs and provide them to
    Zeek. This duplicates functionality that is currently possible (and
    used) by some binpac-based analyzers. One example for an analyzer
    creating its own file IDs is the SSL analyzer.

  * Bump cmake submodule for INCLUDE_DIRS fix (Arne Welzel, Corelight)

6.2.0-dev.180 | 2023-11-22 10:34:18 +0100

  * Integrate review feedback (Dominik Charousset, Corelight)

  * Avoid OpenSSL header dependencies (Dominik Charousset, Corelight)

6.2.0-dev.176 | 2023-11-21 14:31:12 +0100

  * Add jq to final.Dockerfile (Matti Bispham, Corelight)

6.2.0-dev.174 | 2023-11-21 12:00:20 +0100

  * TableVal: Replace raw subnets/pattern_matcher with unique_ptr (Arne Welzel, Corelight)

  * TablePatternMatcher: Drop Insert()/Remove(), use Clear() (Arne Welzel, Corelight)

    Also move Clear() when assigning into more generic Assign() function.

  * TableType: Simplify and inline Is...Index tests (Arne Welzel, Corelight)

  * NEWS: Add small table[pattern] section (Arne Welzel, Corelight)

  * Expr/Val: Add support for in set[pattern] (Arne Welzel, Corelight)

  * zeek.bif: Implement table_pattern_matcher_stats() bif for introspection (Arne Welzel, Corelight)

    Provide a script accessible way to introspect the DFA stats that can be
    leveraged to gather runtime statistics of the underlying DFA. This
    re-uses the existing MatcherStats used by ``get_matcher_stats()``.

  * DFA: Extract inner Stats struct from DFA_State_Cache (Arne Welzel, Corelight)

    This makes it possible to forward declare the class in Val.h which
    otherwise seems difficult.

  * Expr: Implement string in table[pattern] of X (Arne Welzel, Corelight)

    Not sure how useful this is (and the implementation isn't optimized in
    any way), but seems reasonable for consistency.

    Vern suggested that set[pattern] can already be achieved via
    set_to_regex(), so left out any set[pattern] variants.

  * RE: Remove RE_DisjunctiveMatcher and re-use MatchAll() (Arne Welzel, Corelight)

    Seems we can just open code the CompileSet() usage in the TablePatternMatcher
    helper without indirecting through another class. Further, add the collection
    of indices into MatchAll() rather than duplicating its code in
    MatchDisjunction(). Doesn't seem like MatchAll() is used widely.

  * TablePatternMatcher: Use const StringValPtr& instead of const StringVal* (Arne Welzel, Corelight)

  * Val: Move TablePatternMatcher into detail namespace (Arne Welzel, Corelight)

    There's anyway only prototype in the headers, so detail seems better
    than the public zeek namespace.

  * TablePatternMatcher: Use unique_ptr (Arne Welzel, Corelight)

  * IndexType: Add IsPatternIndex(), like IsSubNetIndex() (Arne Welzel, Corelight)

  * btest: Add test for pattern tables and when (Arne Welzel, Corelight)

  * Reuse CompileSet() instead of || string formatting (Arne Welzel, Corelight)

  * script optimization support for using strings to index table[pattern] values (Vern Paxson, Corelight)

  * BTests for indexing "table[pattern] of T" with strings (Vern Paxson, Corelight)

  * support for indexing "table[pattern] of T" with strings to get multi-matches (Vern Paxson, Corelight)

6.2.0-dev.155 | 2023-11-21 10:08:14 +0100

  * GH-3453: DNS: Add Ed25519 and Ed448 enum values to parser (Arne Welzel, Corelight)

    We already had these declared in dns/const.zeek, so extend the parser
    as well to avoid raising weirds and add some test pcaps:

       $ dig @8.8.8.8 DNSKEY ed448.no
       $ dig @8.8.8.8 ed448.no +dnssec

    And the same for the ed25519.no domain.

  * RE_Match_State: Do not reset current_pos for every Match() call (Arne Welzel, Corelight)

    This seems like a bug: If one feeds one byte at a time, current_pos
    is being reset for every byte, possibly reporting the wrong offsets
    in the accepted_matches map.

  * frameworks/software: Fix stale value used for stripping (Arne Welzel, Corelight)

    There was some confusion around which value was used subsequent to a strip(),
    but sub not respecting anchors make it appear to work. Also seems that the
    `\(?` part seems redundant.

  * GH-3455: strings.bif/sub,gsub: Respect anchors in pattern (Arne Welzel, Corelight)

    Anchors within pattern passed to sub() or gsub() were previously ignored,
    replacing any occurrence of '<text>' even when '^<text>' was used as a
    pattern.

    This is a pretty user-visible change (and we even have anchored patterns
    within the base scripts), but seems "the right thing to do".

    Relates to #3455

6.2.0-dev.147 | 2023-11-17 18:00:32 +0100

  * BTest updates to accommodate event handler coalescence differences (Vern Paxson, Corelight)

  * BTests for testing that event handler coalescence operates as expected (Vern Paxson, Corelight)

  * coalescing of event handlers (ZAM optimization) (Vern Paxson, Corelight)

6.2.0-dev.143 | 2023-11-17 09:24:36 -0700

  * CI: Install missing packages on opensuse (Tim Wojtulewicz, Corelight)

6.2.0-dev.141 | 2023-11-17 13:01:19 +0100

  * GH-3455: strings.bif/do_split_string: Pass bol and eol to MatchPrefix() (Arne Welzel, Corelight)

    This allows better control of BOL and EOL. MatchPrefix() / LongestMatch()
    always start with BOL.

  * RE_Matcher: Add MatchPrefix with bol/eol control (Arne Welzel, Corelight)

6.2.0-dev.137 | 2023-11-11 17:45:51 +0100

  * Bind scan_path to the scope; avoid heap allocation (Dominik Charousset, Corelight)

6.2.0-dev.134 | 2023-11-10 12:42:57 +0100

  * retention of superseded AST elements to prevent pointer mis-aliasing (Vern Paxson, Corelight)

  * BTest updates for latest ZAM maintenance (Vern Paxson, Corelight)

  * greater ZAM optimization of inlined function calls (Vern Paxson, Corelight)

  * some minor ZAM optimization improvements (Vern Paxson, Corelight)

  * added "-O noinline" option to turn off ZAM inlining, to help with diagnosing optimization problems (Vern Paxson, Corelight)

  * fixes for a number of ZAM optimization bugs (Vern Paxson, Corelight)

  * allow explicitly marking an identifier as equivalent to special '_' identifier (Vern Paxson, Corelight)

  * fixed some warnings about mixing signed & unsigned integers (Vern Paxson, Corelight)

  * descriptions of "for" statements now include their "value variable" if present (Vern Paxson, Corelight)

6.2.0-dev.123 | 2023-11-09 17:22:29 +0100

  * GH-3443: Spicy: Provide `zeek::skip_input()` to disable deliver to
    current analyzer. (Robin Sommer, Corelight)

6.2.0-dev.121 | 2023-11-09 10:42:16 +0100

  * btest/opt: Update pure-inlining baseline (Arne Welzel, Corelight)

    I suspect this as just stale due to not being executed regularly.

  * btest/opt: Update basic baseline (Arne Welzel, Corelight)

  * Expr: Fix alloc-dealloc-mismatch (Arne Welzel, Corelight)

    I *thought* Vern had fixed this up, but maybe that was another
    occurrence.

  * btest: Move zam/basic.test into opt/basic.test, update TestDirs (Arne Welzel, Corelight)

    It seems "opt" can and should run unconditionally. The "misc" dir was
    definitely an oversight.

  * Add deprecations to NEWS [nomail] [skipci] (Arne Welzel, Corelight)

6.2.0-dev.114 | 2023-11-07 19:27:58 +0100

  * Bump zeekctl (Arne Welzel, Corelight)

  * GH-3187: EventHandler: Deprecate SetUsed() and Used() as well. (Arne Welzel, Corelight)

    Seems the latter isn't used outside of the functions that were deprecated
    in the previous commit and with UsageAnalyzer not making use of this
    information unclear why we should keep it around.

    Relates to #3187.

  * GH-3187: EventRegistry: Deprecate UsedHandlers() and UnusedHandlers() (Arne Welzel, Corelight)

    and check_for_unused_event_handlers: UsageAnalyzer is more thorough
    and the previous ones weren't extended to work with &is_used and
    should probably be considered superseded by the UsageAnalyzer even
    if that currently does not provide a public API and just prints
    out deprecation warnings.

    I'm also tempted to deprecate SetUsed() and Used() of EventHandler
    for the same reason.

    Closes #3187.

  * time machine: Mark leftovers for removal in v7.1 (Arne Welzel, Corelight)

    I suspect we could just drop these directly, but lets follow the
    deprecation cycle.

  * policy/misc/load-balancing: Deprecate script (Arne Welzel, Corelight)

  * GH-2877: cluster: Deprecate the Cluster::Node$interface field (Arne Welzel, Corelight)

    This field isn't required by a worker and it's certainly not used by a
    worker to listen on that specific interface. It also isn't required to
    be set consistently and its use in-tree limited to the old load-balancing
    script.

    There's a bif called packet_source() which on a worker will provide
    information about the actually used packet source.

    Relates to zeek/zeek#2877.

6.2.0-dev.107 | 2023-11-07 10:03:24 +0100

  * Update binpac and bifcl submodules [nomail] (Tim Wojtulewicz, Corelight)

  * CI: Use other base64 options on macOS Sonoma too (Tim Wojtulewicz, Corelight)

  * CI: Add macOS Sonoma build, remove macOS Monterey build (Tim Wojtulewicz, Corelight)

6.2.0-dev.102 | 2023-11-07 09:58:25 +0100

  * Merge branch 'topic/xb-anssi/http_signature_body_end_match' of https://github.com/xb-anssi/zeek (Arne Welzel, Corelight)

    * 'topic/xb-anssi/http_signature_body_end_match' of https://github.com/xb-anssi/zeek:
      Let signature framework match HTTP body end
      Test how the signature framework matches HTTP body

  * Let signature framework match HTTP body end (xb-anssi)

    The HTTP analyzer never tells the signature framework when the body of a
    request or a response ends, so any signature regex ending in a '$' used
    in an 'http-request-body' or in an 'http-reply-body' condition will
    never match.

    This made it impossible to write a signature which could distinguish an
    HTTP body consisting only of something from an HTTP body prefixed by
    that same something.

    - Fix:

    The fix notifies the signature framework on EndOfData() that there will
    be no further data to match for this body by giving it an empty buffer
    of length 0 with the eol parameter set to true and all others set to
    false. This lets it reach the '$' state in its DFA, and doesn't affect
    other documented HTTP match behaviours.

    - Limitation:

    Since the signature framework doesn't appear to keep previously consumed
    data on hand, any match of an http-*-body condition whose patterns ends
    with a '$' will lead to an empty data parameter being passed to the
    signature_match() event because the body data is no longer available
    when EndOfData() happens.

    Due to segmentation there is anyway no guarantee the data parameter
    would have held the entire match even without the '$', since the data
    parameter only receives the last chunk of data which completed the match
    condition, as can be seen on prefix matches in the btest cases where the
    matching data spans multiple segments (the event gives 'B' and not
    'AB'), so this is only an extreme case of partial data being given to
    that event.

  * Test how the signature framework matches HTTP body (xb-anssi)

    This adds a signatures/http-body-match btest to verify how the signature
    framework matches HTTP body in requests and responses.

    It currently fails because the 'http-request-body' and 'http-reply-body'
    clauses never match anything when there is a '$' in their regular
    expressions.

    The other pattern clauses such as the 'payload' clause do not suffer
    from that restriction and it is not documented as a limitation of HTTP
    body pattern clauses either, so it is probably a bug.

    The "http-body-match" btest shows that without a fix any signatures
    which ends with a '$' in a http-request-body or http-reply-body rule
    will never raise a signature_match() event, and that signatures which do
    not end with a '$' cannot distinguish an HTTP body prefixed by the
    matching pattern (ex: ABCD) from an HTTP body consisting entirely of the
    matching pattern (ex: AB).

    Test cases by source port:
    - 13579:
      - GET without body, plain res body (CD, only)
    - 13578:
      - GET without body, plain res body (CDEF, prefix)
    - 24680:
      - POST plain req body (AB, only), plain res body (CD, only)
    - 24681:
      - POST plain req body (ABCD, prefix), plain res body (CDEF, prefix)
    - 24682:
      - POST gzipped req body (AB, only), gzipped res body (CD, only)
      - POST plain req body (CD, only), plain res body (EF, only)
    - 33210:
      - POST multipart plain req body (AB;CD;EF, prefix)
      - plain res body (CD, only)
    - 33211:
      - POST multipart plain req body (ABCD;EF, prefix)
      - plain res body (CDEF, prefix)
    - 34527:
      - POST chunked gzipped req body (AB, only)
      - chunked gzipped res body (CD, only)
    - 34528:
      - POST chunked gzipped req body (ABCD, prefix)
      - chunked gzipped res body (CDEF, prefix)

    The tests with source ports 24680, 24682 and 34527 should
    match the signature http_request_body_AB_only and the signature
    http_request_body_AB_prefix, but they only match the latter.

    The tests with source ports 13579, 24680, 24682, 33210 and 34527 should
    match the signature http_response_body_CD_only and the signature
    http_response_body_CD_prefix, but they only match the latter.

    The tests with source ports 24680, 24681, 33210 and 33211 show how the
    http_request_body_AB_then_CD signature with two http-request-body
    conditions match either on one or multiple requests (documented
    behaviour).

    The test cases with other source ports show where the
    http_request_body_AB_only and http_response_body_CD_only signatures
    should not match because their bodies include more than the searched
    patterns.

6.2.0-dev.99 | 2023-11-07 09:56:55 +0100

  * Fix unsafe and inefficient uses of copy_string (Dominik Charousset, Corelight)

    Add a new overload to `copy_string` that takes the input characters plus
    size. The new overload avoids inefficient scanning of the input for the
    null terminator in cases where we know the size beforehand. Furthermore,
    this overload *must* be used when dealing with input character sequences
    that may have no null terminator, e.g., when the input is from a
    `std::string_view` object.

6.2.0-dev.97 | 2023-11-06 09:29:49 -0700

  * Update c-ares in Windows conanfile (Tim Wojtulewicz, Corelight)

  * Upgrade c-ares submodule to 1.21.0 (Tim Wojtulewicz, Corelight)

6.2.0-dev.93 | 2023-11-01 12:04:16 +0100

  * build_inner_connection: Use the outer packet's timestamp (Arne Welzel, Corelight)

    Don't construct the timeval based on run_state, just use the timestamp
    of the outer packet to avoid the extra int/double conversions required.

  * build_inner_connection: Avoid one extra Init() (Arne Welzel, Corelight)

    Packet::Init() is not so cheap as one might think: It computes a
    timestamp from { 0, 0 } using double division. Just avoid this
    by not initializing an empty Packet.

  * packet_analysis: Do not run DetectProtocol() on disabled analyzers (Arne Welzel, Corelight)

    This came up when disabling the TEREDO analyzer but still seeing its
    DetectProtocol() method prominently in flame graphs.

  * packet_analysis/Dispatcher: Do not index table twice (Arne Welzel, Corelight)

    It's okay to return the nullptr that's in the table, no need to check
    for != nullptr before dereferencing again.

  * GH-3379: packet_analysis: Avoid shared_ptr copying for analyzer lookups (Arne Welzel, Corelight)

    For deeply encapsulated connections (think AWS traffic mirroring format
    like IP,UDP,GENEVE,IP,UDP,VXLAN,ETH,IP,TCP), the Dispatcher::Lookup()
    method is fairly visible in profiles when running in bare mode.

    This changes the Analyzer::Lookup() and Dispatcher::Lookup() return value
    breaking the API in favor of the performance improvement.

    Relates to zeek/zeek#3379.

6.2.0-dev.86 | 2023-10-31 16:17:33 +0000

  * SSL: Add new extension types and ECH test (Johanna Amann, Corelight)

    This commit adds a multitude of new extension types that were added in
    the last few years; it also adds grease values to extensions, curves,
    and ciphersuites.

    Furthermore, it adds a test that contains a encrypted-client-hello
    key-exchange (which uses several extension types that we do not have in
    our baseline so far).

6.2.0-dev.83 | 2023-10-30 12:34:26 -0700

  * Format JSON with clang-format (Benjamin Bannier, Corelight)

    We do not activate support for JavaScript at this time since most of our
    JavaScript code is in BTest files to test zeekjs, but these files also
    contain other languages which leads to largely misformated files.

  * Reformat Zeek in Spicy style (Benjamin Bannier, Corelight)

    This largely copies over Spicy's `.clang-format` configuration file. The
    one place where we deviate is header include order since Zeek depends on
    headers being included in a certain order.

6.2.0-dev.78 | 2023-10-30 11:47:23 +0100

  * GH-3421: Normalize version strings in test (Benjamin Bannier, Corelight)

6.2.0-dev.74 | 2023-10-27 11:04:16 -0700

  * IEEE802_11: Fix buffer-overflow due to not accounting for the amsdu header (Arne Welzel, Corelight)

    The fuzzer generated input where data length left was 14 and the amsdu_len
    field set to 14. That caused buffer overread due not taking into account
    the amsdu header length of 14.

  * ssl: Prevent unbounded ssl_history growth (Arne Welzel, Corelight)

    The ssl_history field may grow unbounded (e.g., ssl_alert event). Prevent this
    by capping using a configurable limit (default 100) and raise a weird once reached.

  * ssl: Cap number of alerts parsed from SSL record (Arne Welzel, Corelight)

    Limit the number of events raised from an SSL record with content_type
    alert (21) to a configurable maximum number (default 10). For TLS 1.3,
    the limit is set to 1 as specified in the RFC. Add a new weird cases
    where the limit is exceeded.

    OSS-Fuzz managed to generate a reproducer that raised ~660k ssl_plaintext
    and ssl_alert events given ~810kb of input data. This change prevents this
    with hopefully no negative side-effect in the real-world.

  * ftp: Do not base seq on number of pending commands (Arne Welzel, Corelight)

    Previously, seq was computed as the result of |pending_commands|+1. This
    opened the possibility to override queued commands, as well as logging
    the same pending ftp reply multiple times.

    For example, when commands 1, 2, 3 are pending, command 1 may be dequeued,
    but the incoming command then receives seq 3 and overrides the already
    pending command 3. The second scenario happens when ftp_reply() selected
    command 3 as pending for logging, but is then followed by many ftp_request()
    events. This resulted in command 3's response being logged for every
    following ftp_request() over and over again.

    Avoid both scenarios by tracking the command sequence as an absolute counter.

  * X509Common/SCT parsing: Fix memory leak in error case (Arne Welzel, Corelight)

    Failed to free the copy of the OCSP extension's value the inner
    octet string couldn't be parsed.

6.2.0-dev.64 | 2023-10-27 19:19:55 +0200

  * fuzzers: Add DTLS fuzzer (Arne Welzel, Corelight)

    Not sure it adds a lot over the SSL analyzer as they share code, but it
    provides a distinct entry point at least.

  * GH-3398: generic-analyzer-fuzzer: Support NextPacket() fuzzing (Arne Welzel, Corelight)

    A number of analyzers that we've been fuzzing with the generic-analyzer-fuzzer
    setup do not implement DeliverStream() and instead only work with DeliverPacket()
    (ntp, syslog, sip, radius, ...). Calling DeliverStream() on those is
    pretty much a noop and fuzzing not effective.

    This change adds support to fuzz DeliverPacket(). Whether to use packet
    or stream fuzzing is configured through a define via CMake.

    This is still a bit limited in that for analyzers that support both,
    DeliverPacket() and DeliverStream(), only one code path is fuzzed.

  * Require `truncate` for a test using it (Benjamin Bannier, Corelight)

    It looks like older versions of macOS like Monterey do not ship with
    `truncate`. Make a recently added test require it to suppress spurious
    failures.

  * Bump outdated baseline (Benjamin Bannier, Corelight)

    This fixes an outdated baseline added with
    caaffd0324ac0557ba4c5ce0290d07a4d013a735.

  * Bump auxil/spicy to latest release (Benjamin Bannier, Corelight)

6.2.0-dev.56 | 2023-10-26 15:34:17 +0200

  * Fix tests so they work both with GNU and BSD tools (Benjamin Bannier, Corelight)

    The GNU and BSD versions of `touch` and `truncate` allow slightly
    different arguments, change the tests so they work in both versions.

  * Install libmaxminddb in macOS CI (Benjamin Bannier, Corelight)

6.2.0-dev.53 | 2023-10-25 17:01:57 +0200

  * fuzzer-setup: Allow customization without recompiling (Arne Welzel, Corelight)

    This change allows to invoke a reproducer with different script options
    and extra scripts to ease debugging and investigation. For example, enabling
    the DPD debug stream and adding misc/dump-events can be done as follows after
    this change:

        ZEEK_DEBUG_LOG_STDERR=1 ../../build/src/fuzzers/zeek-pop3-fuzzer <test-case> -- -B dpd misc/dump-events

  * subdir-btest: Allow setting build_dir (Arne Welzel, Corelight)

    We have that functionality in btest.cfg and it's actually nice to use
    it for toggling between a debug/asan and release build.

6.2.0-dev.49 | 2023-10-25 12:55:41 +0200

  * GH-3043: Supervisor: Handle EAGAIN error on stem pipe (Arne Welzel, Corelight)

    util::safe_write() calls abort() in case of EAGAIN errors. This is
    easily observed when starting clusters with 32 workers or more.

    Add a custom write_message() function handling EAGAIN by retrying
    after a small sleep. It's not clear a more complicated poll() would be
    much better: The pipe might be ready for writing, but then our message
    might not actually fit in, resulting in another EAGAIN error. And even
    poll() would introduce blocking/sleeping code.

    Take some precautions against the stem and the supervisor dead-locking
    when both pipes are full by draining the other end on EAGAIN errors.

6.2.0-dev.47 | 2023-10-25 09:15:20 +0200

  * CI: Pass -A flag to btest for cluster-testing builds (Tim Wojtulewicz, Corelight)

6.2.0-dev.44 | 2023-10-24 13:05:55 -0700

  * Update CMake submodule for caching fix (Dominik Charousset, Corelight)

6.2.0-dev.41 | 2023-10-24 19:35:20 +0200

  * GH-3190: event: Reinitialize EventMgr's flare after fork() from stem (Arne Welzel, Corelight)

    Because EventMgr is defined globally as an object (rather than a global
    pointer to an EventMgr object), its pipe is created even before main()
    is entered. This further means that in the fork-based supervisor setup,
    all Zeek processes created from the top-level supervisor process share
    the same pipe object for the EventMgr. In turn, whenever any of the
    processes enqueued an event, the flare was fired and ready for reading
    on all other processes in the cluster, causing much contention and
    unneeded overhead.

  * btest/Files/mmdb: golang-ci suggested fixes (Arne Welzel, Corelight)

6.2.0-dev.38 | 2023-10-24 13:57:03 +0200

  * base/frameworks/spicy: Do not load base/misc/version (Arne Welzel, Corelight)

    Unsure what it's used for today and also results in the situation that on
    some platforms we generate a reporter.log in bare mode, while on others
    where spicy is disabled, we do not.

    If we want base/frameworks/version loaded by default, should put it into
    init-bare.zeek and possibly remove the loading of the reporter framework
    from it - Reporter::error() would still work and be visible on stderr,
    just not create a reporter.log.

  * btest/core/mmdb: Basic lookup_autonomous_system / lookup_location tests (Arne Welzel, Corelight)

  * btest/files: Add mmdb testing databases and generator code (Arne Welzel, Corelight)

  * zeek.bif: Switch mmdb stale check to network_time (Arne Welzel, Corelight)

    Makes testing easier and aligns better with log rotation and timer
    expiration. Should not have an effect in practice. Also, log detail
    about whether inode or modification time changed, too.

  * Fix mmdb pointer destruction (peter.cullen, Corelight)

    The mmdb references needs to use reset() instead of release()
    so that the destructor gets called and the stale DB is actually
    cleaned up. Otherwise, the pointer is leaked and the stale DB
    remains in memory.

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

6.2.0-dev.29 | 2023-10-23 11:25:15 +0200

  * GH-3370: Spicy: Avoid creating Zeek types multiple times. (Robin Sommer, Corelight)

    We now cache the first time we create a (named) Zeek type so that we
    can reuse it later when needed again.

    Closes #3370.

  * Spicy: Unroll Zeek type registrations. (Robin Sommer, Corelight)

    For records and enums, the type registration calls could get pretty
    large, and then slow to compile. We now build up their vectors
    entry by entry instead of all at once.

  * quic: Skip new test if have-quic is false (Arne Welzel, Corelight)

6.2.0-dev.24 | 2023-10-20 11:44:15 -0700

  * quic: Bump maximum history length, make configurable (Arne Welzel, Corelight)

    From zeek/spicy-quic#15

6.2.0-dev.21 | 2023-10-19 19:59:49 +0200

  * fuzzers: Add SSL fuzzer (Arne Welzel, Corelight)

6.2.0-dev.19 | 2023-10-19 16:25:34 +0200

  * GH-3383: Fix missing version number in `spicyz`. (Robin Sommer, Corelight)

6.2.0-dev.17 | 2023-10-19 11:08:07 +0200

  * ldap: Use longer event names (Arne Welzel, Corelight)

    It's unusual to compress and shorten event names of protocol analyzers,
    switch to a slightly longer name instead.

  * ldap: Add spicy-events.zeek (Arne Welzel, Corelight)

  * quic: Add spicy-events.zeek (Arne Welzel, Corelight)

6.2.0-dev.13 | 2023-10-18 12:05:24 -0700

  * Add new SSH key exchange algorithms reported by OpenSSH 9.0 (Tim Wojtulewicz, Corelight)

6.2.0-dev.11 | 2023-10-17 19:33:47 +0200

  * gre-over-udp: Update testing pcap with both endpoints (Arne Welzel, Corelight)

    The first pcap only contained packets from the originator, not the responder.

    What stands out here is that the Linux kernel doesn't seem to use a symmetric
    flow hash for the tunneled connection, resulting in a total of four tunnel
    connections for the two inner connections. Sigh.

6.2.0-dev.9 | 2023-10-17 15:54:34 +0200

  * performance improvements for the LDAP ASN.1 analyzer (Paul Dokas, Corelight)

6.2.0-dev.6 | 2023-10-16 11:49:33 -0700

  * Add registration for GRE-over-UDP (Tim Wojtulewicz, Corelight)

  * Update c-ares to v1.20.1 [nomail] (Tim Wojtulewicz, Corelight)

6.2.0-dev.2 | 2023-10-13 14:36:17 -0700

  * Broker: Adapt to upstream API adjustments (Dominik Charousset)

6.1.0-dev.561 | 2023-10-13 13:15:59 +0200

  * GH-14: protocol/quic/decrypt_crypto: Reuse OpenSSL context objects (Arne Welzel, Corelight)

    It is not necessary to allocate and free the context objects used for
    HKDF and AES all the time, they can be re-used. The main assumption here
    is no cross-thread usage, but this should be guaranteed even with the
    fibers: QUIC_decrypt_crypto_payload() always runs to completion.

    A pcap with ~12k QUIC connections had ~15% samples in
    QUIC_decrypt_crypto_payload. After this change it is down to 5%
    of samples. The improvement in runtime is ~16%, 12.2 seconds
    to 10.2 seconds.

    From zeek/spicy-quic#14

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

6.1.0-dev.557 | 2023-10-13 08:16:24 +0200

  * CI: Add more logging during docker builds (Tim Wojtulewicz, Corelight)

  * CI: Add timeouts for apt commands in docker builder Dockerfiles (Tim Wojtulewicz, Corelight)

6.1.0-dev.553 | 2023-10-12 17:11:18 +0200

  * Fix signed-unsigned comparison. (Benjamin Bannier, Corelight)

    We compared the return value of `sizeof` which returns a `std::size_t`
    with an `int` which GCC warned about. This patch changes the `int` to a
    `size_t`.

6.1.0-dev.551 | 2023-10-12 15:53:45 +0200

  * quic: Avoid bytes for VariableLengthInteger (Arne Welzel, Corelight)

    Allocation of bytes objects due to parsing and usage of pack and the
    invocation of to_uint() showed significantly in profiles (3.3% sample
    matches). Switch to a more procedural approach to avoid the allocation
    overhead.

    From zeek/spicy-quic/pull/13

  * protocol/quic: Add doc-id/doc-description to .evt file (Arne Welzel, Corelight)

    Without this, the QUIC enums end up under Zeek::Spicy and we don't
    have an extra Zeek::QUIC section in the docs.

6.1.0-dev.547 | 2023-10-11 18:02:25 +0200

  * ci/btest: Remove spicy-quic helper, disable Spicy on CentOS 7 (Arne Welzel, Corelight)

    The have-quic pattern wasn't great and it wouldn't scale.

  * btest/core/ppp: Run test in bare mode (Arne Welzel, Corelight)

  * btest/quic: Update other tests (Arne Welzel, Corelight)

  * testing/quic: Fixups and simplification after Zeek integration (Arne Welzel, Corelight)

  * quic: Integrate as default analyzer (Arne Welzel, Corelight)

  * quic: Include Copyright lines to the analyzer's source code contributed by Fox-IT (Arne Welzel, Corelight)

    This is primarily such that they stay intact when importing into the
    Zeek project. Also move LICENSE to COPYING.

  * quic: Squashed follow-ups: quic.log, tests, various fixes, performance (Arne Welzel, Corelight)

  * quic: Initial implementation (Joost)

6.1.0-dev.538 | 2023-10-11 15:20:27 +0200

  * ci/update-zeekygen-docs.sh: Do output stderr by default (Arne Welzel, Corelight)

    stderr was only produced when there was a real failure as it had been a
    bit noisy previously, but has hidden actual problems. Now that there is
    less noise, just output stderr all the time.

  * zeekygen: Disable zeek_init() handlers causing error output (Arne Welzel, Corelight)

    These modules have zeek_init() handlers with actual logic that has side-effects
    and error outputs. Just disable them during zeekygen runs.

  * zeekygen/example: Squelch unused function warning (Arne Welzel, Corelight)

  * iosource/pcap: Support configurable buffer size (Arne Welzel, Corelight)

    On Linux with a default ext4 or tmpfs filesystem, the default buffer size for
    reading a pcap is chosen as 4k (strace/gdb validated). When reading large pcaps
    containing raw data transfers, the syscall overhead for read becomes visible
    in profiles. Support configurability of the buffer size and default to 128kb.

    When processing a ~830M PCAP (16 UDP connections, each transferring ~50MB) in
    bare mode, this change improves runtime from 1.39 sec to 1.29 sec. Increasing
    the buffer further didn't provide a noticeable boost.

  * util/setvbuf: Respect buf argument (Arne Welzel, Corelight)

6.1.0-dev.531 | 2023-10-11 14:09:22 +0200

  * protocol/ldap: Add doc-id/doc-description to .evt file (Arne Welzel, Corelight)

  * spicy/manager: Ensure Zeekygen knows identifier for registered types (Arne Welzel, Corelight)

    Without this, Zeekygen won't generate documentation about exported
    enum types as it can not resolve the identifier. Also, only register a
    type as item with the Spicy plugin if there's no _module_info currently
    active.

6.1.0-dev.528 | 2023-10-10 20:03:31 +0200

  * Introduce dedicated `LDAP::Info` (Benjamin Bannier, Corelight)

  * Remove redundant storing of protocol in LDAP logs (Benjamin Bannier, Corelight)

  * Use LDAP `RemovalHook` instead of implementing `connection_state_remove` (Benjamin Bannier, Corelight)

  * Tidy up LDAP code by using local references (Benjamin Bannier, Corelight)

  * Pluralize container names in LDAP types (Benjamin Bannier, Corelight)

  * Move LDAP script constants to their own file (Benjamin Bannier, Corelight)

  * Name `LDAP::Message` and `LDAP::Search` `*Info` (Benjamin Bannier, Corelight)

  * Make ports for LDAP analyzers fully configurable (Benjamin Bannier, Corelight)

    This moves the ports the LDAP analyzers should be triggered on from the
    EVT file to the Zeek module. This gives users full control over which
    ports the analyzers are registered for while previously they could only
    register them for additional ports (there is no Zeek script equivalent
    of `Manager::UnregisterAnalyzerForPort`).

    The analyzers could still be triggered via DPD, but this is intentional.
    To fully disable analyzers users can use e.g.,

    ```zeek
    event zeek_init()
        {
        Analyzer::disable_analyzer(Analyzer::ANALYZER_LDAP_TCP);
        }
    ```

  * Require have-spicy for tests which log spicy-ldap information (Benjamin Bannier, Corelight)

  * Fix LDAP analyzer setup for when Spicy analyzers are disabled (Benjamin Bannier, Corelight)

  * Bump zeek-testing-private (Benjamin Bannier, Corelight)

  * Integrate spicy-ldap test suite (Benjamin Bannier, Corelight)

  * Move spicy-ldap into Zeek protocol analyzer tree (Benjamin Bannier, Corelight)

  * Explicitly use all of spicy-ldap's modules (Benjamin Bannier, Corelight)

    This fixes building of this multi-module analyzer.

  * Explicitly list `asn1.spicy` as spicy-ldap source (Benjamin Bannier, Corelight)

    This will lead to the file being installed so it can be consumed by
    other parsers.

  * Remove uses of `zeek` module in spicy-ldap (Benjamin Bannier, Corelight)

  * Fix typos in spicy-ldap (Benjamin Bannier, Corelight)

  * Remove project configuration files in spicy-ldap (Benjamin Bannier, Corelight)

  * Integrate spicy-ldap into build (Benjamin Bannier, Corelight)

  * Import zeek/spicy-ldap@57b5eff9883d1c43896f4278218351a132de9ce1 (Benjamin Bannier, Corelight)

  * Merge branch 'topic/timw/arm-build-failure' (Tim Wojtulewicz, Corelight)

    * topic/timw/arm-build-failure:
      CI: Disable spicy on arm_debian11 and opensuse_leap_15_4 builds
      CI: Remove sanitizers resource template

  * CI: Disable spicy on arm_debian11 and opensuse_leap_15_4 builds (Tim Wojtulewicz, Corelight)

    These two builds are struggling to complete due to OOM issues. Reducing the
    number of CPUs didn't help, so this temporarily disables Spicy from
    building on those hosts to reduce the memory usage.

  * CI: Remove sanitizers resource template (Tim Wojtulewicz, Corelight)

    This template was added when the sanitizer builds needed more memory than all
    of the other builds. We've since increased the amount of memory that the other
    builds use beyond what's requested in this template, so it doesn't make sense
    to keep it around.

  * Updating zeek-aux submodule [nomail] (Arne Welzel, Corelight)

  * NEWS: Polish external plugins message [nomail] [skip ci] (Arne Welzel, Corelight)

6.1.0-dev.502 | 2023-10-09 16:26:58 +0200

  * btest/plugins/hooks: Run in bare mode (Arne Welzel, Corelight)

    Motivation is basically the same as in 88bb5270263934f0004fd9fdeebf5ac25015d213.
    For plugin.hooks, one example is that adding a new option in the default script
    changes the baseline due registration of change handlers. Also, the connection
    record is printed in various places, resulting in churn when the default
    scripts change.

  * Bump cmake submodule for 3.15 requirement (Arne Welzel, Corelight)

6.1.0-dev.499 | 2023-10-09 14:15:41 +0200

  * btest/coverage: Add record-fields test (Arne Welzel, Corelight)

    Justin pointed out that the misc/dump-events test shows added fields to
    the connection record. Add a new test that prints the connection record
    recursively in bare and default mode to cover that use-case
    specifically.

  * btest/dump-events: Run in bare mode, load conn and smtp scripts only (Arne Welzel, Corelight)

    The dump-events baseline changes are pure noise and have spurred confusion
    for internal and external contributors. For example, adding new
    analyzers have perturbed orderings of sets holding analyzer tags.

    Running in non-bare mode, the baselines change almost whenever any of the
    record types attached to connections change in the default scripts. This
    causes continuous and seemingly little useful updates to the baselines.

    This change switches the test to run in bare mode and explicitly loads
    just base/protocols/conn and base/protocols/smtp. The primary intention
    of the test should be testing the functionality of the misc/dump-events
    script, not the raised events of all loaded default scripts (for that the
    used PCAP is too narrow).

    Protocol specific scripts that do want to leverage misc/dump-events for
    baseline creation of their or their analyzer's events can add additional
    specific tests with suitable PCAP files.

6.1.0-dev.495 | 2023-10-06 16:08:56 +0200

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

6.1.0-dev.493 | 2023-10-04 22:46:38 -0700

  * Btest for normalized reporter path reporting (Christian Kreibich, Corelight)

  * Normalize file paths in find_file_in_path() (Christian Kreibich, Corelight)

6.1.0-dev.490 | 2023-10-04 20:15:54 +0200

  * Var/Func: Render function parameters using comma, not semicolon (Arne Welzel, Corelight)

    A bit larger follow-up to what Tim pointed out: Function prototype descriptions
    previously used semicolons to separate parameters.

    Switch to use commas when a RecordType is used as function parameter.
    Use existing "func_args" naming for consistency.

  * GH-3314: Var: Fix null-pointer deref on redefinition of lambdas (Arne Welzel, Corelight)

6.1.0-dev.486 | 2023-10-02 11:50:56 -0700

  * get_current_packet: Add note/warning to docs (Arne Welzel, Corelight)

6.1.0-dev.484 | 2023-10-02 11:44:21 -0700

  * zeek-setup: Early exit when parsing failed (Arne Welzel, Corelight)

    When there are errors reported during yyparse(), Zeek still continued
    running initialization functions like init_general_global_var(), init_net_var()
    and run_bif_initializers(). These usually call abort() in unexpected
    situations causing misleading and confusing errors. This patch prevents
    this by exiting earlier.

    Closes #3316

6.1.0-dev.482 | 2023-10-02 11:42:58 -0700

  * broker/Manager: Fix deletion of composite keys (Arne Welzel, Corelight)

    Follow-up for commit 44ae8f9e8f25ff403707026657e000693550873f.

  * Add btest for brokerstore delete with complex index. (Jan Grashoefer, Corelight)

6.1.0-dev.479 | 2023-10-02 11:09:13 -0700

  * minor updates to -O gen-C++ maintenance notes and scripts (Vern Paxson, Corelight)

  * ZAM fix for inlining bug when a local is assigned to a function parameter (Vern Paxson, Corelight)

  * ZAM fixes for record creation and table indexing potentially having side-effects (Vern Paxson, Corelight)

  * removed order-of-evaluation ambiguity from new BTest (Vern Paxson, Corelight)

  * introduce internal "mask" expression node to avoid mixed-type overloading of "/" (Vern Paxson, Corelight)

  * fix bug in failing to concretize empty vectors (Vern Paxson, Corelight)

  * ZAM BTest update to match recent change to one of the tests (Vern Paxson, Corelight)

  * fixed ZAM reduction bug in += / -= operators (Vern Paxson, Corelight)

6.1.0-dev.470 | 2023-10-02 10:13:04 -0700

  * generate-docs: Bump ccache from 500M to 2000M (Arne Welzel, Corelight)

    Since enabling Spicy, 500M (default) of ccache max-size size is not
    sufficient for efficient cache usage of a Zeek build anymore, so we
    end up thrashing the configured ccache. Build times have increased
    from 5-10 minutes to 1+ hour after enabling Spicy in the docs build.

6.1.0-dev.468 | 2023-10-02 10:10:22 -0700

  * Enable ccache-backed Spicy JIT for users of developer scripts. (Benjamin Bannier, Corelight)

    For CI and more general BTest execution we already enabled ccache
    previously, but developers using a Spicy build out of the build
    directory still needed to manually set up caching.

    With this patch we enable it for such cases as well via
    `zeek-path-dev.*sh`.

6.1.0-dev.466 | 2023-10-02 10:08:38 -0700

  * DNS_Mgr: Fix | vs & for pollfd.revents (Arne Welzel, Corelight)

    Follow-up for c4c4a23bfd19c258beb7254be9d96716914ad03f.

6.1.0-dev.464 | 2023-10-02 14:38:18 +0200

  * Bump auxil/spicy to latest development snapshot (Benjamin Bannier, Corelight)

6.1.0-dev.462 | 2023-09-28 20:29:36 +0200

  * cirrus/ci: Run ubuntu22_task on benchmark-nightly, too (Arne Welzel, Corelight)

6.1.0-dev.460 | 2023-09-28 10:00:55 +0200

  * Fix GCC warnings about unintended pointer comparisions (Benjamin Bannier, Corelight)

6.1.0-dev.458 | 2023-09-27 16:53:02 +0200

  * cirrus/ci: Add ubuntu22_spicy_task and ubuntu22_spicy_head_task (Arne Welzel, Corelight)

    These tasks are meant to run nightly on the master branch. Currently,
    the external dns, http and dhcp Spicy analyzers are installed via zkg
    post building. The build artifact is uploaded to Cirrus and the benchmarker
    API triggered.

    For the spicy_head task, the auxil/spicy submodule is pulled to the latest
    commit. This also provides a bit of a nightly integration test.

  * cirrus/ci: Add pre_build_script mechanism (Arne Welzel, Corelight)

    Allow tasks to run a bit of code just before ci/build.sh.

6.1.0-dev.455 | 2023-09-26 18:42:20 +0200

  * auxil/libunistd: Bump for poll() inclusion (Arne Welzel, Corelight)

  * GH-3309: DNS_Mgr: Replace ares_fds() with ares_getsock() (Arne Welzel, Corelight)

    On Slack, a user reported "fortify source" aborts within ares_fds() due
    to the FDs used by c-ares exceeding 1024 and thereby larger than the
    maximum fd value that a fd_set can hold.

    Switch to ares_get_socks() and poll() to avoid this.

    Closes #3309.

6.1.0-dev.451 | 2023-09-26 16:14:24 +0200

  * Use unique port for test `scripts.base.utils.active-http` (Benjamin Bannier, Corelight)

  * Do not automatically load Spicy modules in dot paths. (Benjamin Bannier, Corelight)

  * Exclude dot directories when searching ZEEK_PLUGIN_PATH (Benjamin Bannier, Corelight)

    Dot directories rarely contain anything we would want to load as a
    dynamic plugin. Even worse, they likely contain files with externally
    controlled lifetimes which might be removed while we are using them
    (see e.g., zeek/btest#98).

    With this patch we do not search _discovered_ dot directories anymore.
    We continue to load from a user-specified `ZEEK_PLUGIN_PATH`, even if
    its name starts with a dot.

    Since this patch changes previous behavior it is a **BREAKING CHANGE**.

6.1.0-dev.447 | 2023-09-26 11:07:12 +0200

  * VectorVal: Embed vector_val (Arne Welzel, Corelight)

    Similar motivation as for RecordVal, save an extra malloc/free
    and pointer indirection.

    This breaks the `auto& RawVec()` API which previously returned
    a reference to the std::vector*. It now returns a reference
    to the vector instead. It's commented as intended for internal
    and compiled code, so even though it's public API,

    The previous `std::vector<std::optional<ZVal>>*&` return type was also very
    likely not intended (all consumers just dereference it anyhow). I'm certain
    this API was never meant to modify the actual pointer value.

    I've switched to explicit typing, too.

  * RecordVal: Embed record_val (Arne Welzel, Corelight)

    This should remove one malloc/free per created and destroyed record instance
    and avoid one extra pointer indirection to access fields.

6.1.0-dev.444 | 2023-09-25 14:59:19 +0200

  * Do not require cookie for `reject_protocol` during teardown. (Benjamin Bannier, Corelight)

    A user reported that they ran into a situation where a parse error
    caused an assertion failure in `reject_protocol`. printf debugging points
    to the `CookieSetter`s in the `try`/`catch` blocks during processing
    already clearing the cookie RAII-style; since their `catch` blocks
    already send an analyzer violation no further sending of one from
    `reject_protocol` is required.

    This patch replaces the assert in `reject_protocol` with a runtime check
    and an early return if the cookie is already cleared.

6.1.0-dev.442 | 2023-09-25 13:50:15 +0200

  * ci/benchmark: Also send CIRRUS_TASK_NAME to benchmarker API (Arne Welzel, Corelight)

    We otherwise cannot distinguish between different benchmarking jobs when
    we introduce them (think ubuntu22_jemalloc or ubuntu22_spicy).

6.1.0-dev.441 | 2023-09-22 14:38:26 -0700

  * CI: Disable tsan task until it can be fixed (Tim Wojtulewicz, Corelight)

  * Update af_packet plugin submodule [nomail] (Tim Wojtulewicz)

6.1.0-dev.436 | 2023-09-22 15:06:39 +0200

  * Bump Spicy (Benjamin Bannier, Corelight)

6.1.0-dev.434 | 2023-09-22 12:13:23 +0200

  * Fix formatting of `spicy_add_analyzer` (Benjamin Bannier, Corelight)

  * Update cmake submodule [nomail] (Arne Welzel, Corelight)

6.1.0-dev.430 | 2023-09-21 10:55:12 +0200

  * Add check to docs CI whether Spicy docs are up to date. (Robin Sommer, Corelight)

  * Bump zeek-docs. (Robin Sommer, Corelight)

  * Enable Spicy when generating docs. (Robin Sommer, Corelight)

  * GH-3306: Zeekygen: Merge in Spicy analyzer documentation. (Robin Sommer, Corelight)

  * Zeekygen: Refine location tracking for Spicy events. (Robin Sommer, Corelight)

  * Zeekygen: Refactor code documenting analyzers. (Robin Sommer, Corelight)

  * Add `%doc-*` tags to Syslog and Finger analyzers. (Robin Sommer, Corelight)

  * Spicy: Support additional documentation tags inside EVT files. (Robin Sommer, Corelight)

    So far we had trouble documenting Spicy analyzers through Zeekygen
    because they would show up as components belonging to the
    `Zeek::Spicy` plugin; whereas traditional analyzers would be their own
    plugins and hence documented individually on their own. This commit
    teaches Zeekygen to track Spicy analyzers separately inside their own
    `Info` instances. This information isn't further used in this commit
    yet, but will be merged with the plugin output in a subsequent change
    to get the expected joint output.

    To pass additional information to Zeekygen, EVT files now also support
    two new tags for Zeekygen purposes:

    - `%doc-id = ID;` defines the global ID under which everything inside
      the EVT file will be documented by Zeekygen, conceptually comparable
      to plugin names (e.g., `Zeek::Syslog`).

    - `%doc-description = "text" provides additional text to go into the
      documentation (comparable to plugin descriptions).

    This information is carried through into the HLTO runtime
    initialization code, from where it's registered with Zeekygen.

    This commit also removes a couple of previous hacks of how Spicy
    integrated with Zeekygen which (1) ended up generating broken doc output
    for Spicy components, and (2) don't seem to be necessary anymore
    anyways.

  * Fix lack of namespace qualification inside an include file. (Robin Sommer, Corelight)

  * Exclude script from documentation that interferes with other locations. (Robin Sommer, Corelight)

6.1.0-dev.419 | 2023-09-21 09:55:51 +0200

  * GH-3298: Spicy: Fix support for exporting bitfields. (Robin Sommer, Corelight)

    Anonymous bitfields now have their fields lifted into the surrounding
    record.

  * Bump Spicy. (Robin Sommer, Corelight)

6.1.0-dev.416 | 2023-09-20 13:04:21 +0200

  * Correctly compute name for spicyz export with debug log. (Benjamin Bannier, Corelight)

    In 36a6770e9891114b4f42dac3728040c67cc9411e we changed the way Spicy
    fields exported as
    record fields are represented. This broke the `zeek` debug log which
    hardcoded a different representation.

    This patch brings the generation of the debug log in line with the
    actual AST structure. We also enable debug logging in one test to
    validate that it works, at least for the case we hit here.

  * Updating submodule(s) [nomail] (Benjamin Bannier, Corelight)

6.1.0-dev.413 | 2023-09-19 19:56:44 +0200

  * ci/benchmark: Send more Cirrus, GitHub and Zeek version information (Arne Welzel, Corelight)

    I wonder if we hit a query string limitation and should instead switch
    to post body here. It's the `curl -G -X POST` part making this a POST
    with a long query string.

6.1.0-dev.411 | 2023-09-18 10:46:34 -0700

  * Remove some unnecessary includes of telemetry headers (Tim Wojtulewicz, Corelight)

6.1.0-dev.409 | 2023-09-15 11:58:40 +0200

  * Update zeekjs submodule (Arne Welzel, Corelight)

    Primarily for the following fixes:

        4dfc0dd Nodejs: Do not use UV_RUN_DEFAULT during InitPostScript()
        ade764e Nodejs: Unset kNoInitializeCppgc for Node 20.6 or higher
        a38df85 Fix type check of parameters array with zeek.event
        deddadd Fix type check of parameters array with zeek.invoke

  * CMakeLists: Set Python3_EXECUTABLE hint for spicy (Arne Welzel, Corelight)

  * Avoid assertion/tag error on invalid table constructor index expression (Christian Kreibich, Corelight)

    If an index expression in a table constructor isn't a list, don't abort with an
    assertion failure (on debug builds) or a bad tag check. Instead, mark the
    constructor expression erroneous and return gracefully. The following...

      global a_table: table[subnet] of string = {
            1.2.3.4/24 = "unspecified",    # should have been [1.2.3.4/24] = "unspecified",
      };

    ...now yields:

      $ zeek ./test.zeek
      error in ././test.zeek, line 2: table constructor index is not a list (1.2.3.0/24 = unspecified)
      error in ././test.zeek, line 2: type clash in assignment (a_table = table(1.2.3.0/24 = unspecified))

  * GH-3279: Fix handling of module scope when checking exported Spicy types for collisions (Benjamin Bannier, Corelight)

    When checking exported Spicy types for collisions with existing Zeek
    types we previously would also check whether they collide with names in
    global scope, i.e., we didn't provide a `no_global` arg to
    `detail::lookup_ID` which defaulted to false (since we also provided a
    module name I'd argue that the behavior of that function is confusing
    and probably error-prone -- like seen here).

    This meant that e.g., a Spicy enum `foo::Direction` (automatically in
    implicit Spicy module scope) would be detected to collide with the
    existing Zeek `Direction` enum.

    With this patch we use the `lookup_ID` API correctly and do not check
    against potential collisions with globals anymore since it is not
    needed.

  * File extraction: use fseek (Johanna Amann, Corelight)

    In the past, we allocated a buffer with zeroes and wrote that with
    fwrite. Now, instead we just fseek to the correct offset.

    This changes the way in which the file extract limit is counted a bit;
    skipped bytes do no longer count against the file size limit.

    (cherry picked from commit 5071592e9b7105090a1d9de19689c499070749d4)

6.1.0-dev.397 | 2023-09-12 17:00:26 -0700

  * Update dump-events test baseline after latest master changes (Tim Wojtulewicz)

  * Add extract_limit_includes_missing option for file extraction (Johanna Amann, Corelight)

    Setting this option to false does not count missing bytes in files towards the
    extraction limits, and allows to extract data up to the desired limit,
    even when partial files are written.

    When missing bytes are encountered, files are now written as sparse
    files.

    Using this option requires the underlying storage and utilities to support
    sparse files.

    (cherry picked from commit afa6f3a0d3b8db1ec5b5e82d26225504c2891089)

  * ftp: Do not log non-pending commands (Arne Welzel, Corelight)

    OSS Fuzz generated a CWD request and reply followed by very many EPRT
    requests. This caused Zeek to re-log the CWD request and invoke `build_url_ftp()`
    over and over again resulting in long processing times.

    Avoid this scenario by not logging commands that aren't pending anymore.

    (cherry picked from commit b05dd31667ff634ec7d017f09d122f05878fdf65)

  * http/smtp: Fix wrong character class usage (Arne Welzel, Corelight)

    A call to `extract_filename_from_content_disposition()` is only
    efficient if the string is guaranteed to contain the pattern that
    is removed by `sub()`. Due to missing brackets around the `[:blank:]`
    character class, an overly long string (756kb) ending in
    "Type:dtanameaa=" matched the wrong pattern causing `sub()` to
    exhibit quadratic runtime. Besides that, we may have potentially
    extracted wrong information from a crafted header value.

    (cherry picked from commit 6d385b1ca724a10444865e4ad38a58b31a2e2288)

  * VLAN: Fix length checking for non-Ethernet type 2 frames (Tim Wojtulewicz, Corelight)

    (cherry picked from commit c579eb7b7cf3ca1d9008e74d7934155866c7e4f7)

  * Make sure that all sessions/connections are done before deleting plugins (Tim Wojtulewicz, Corelight)

    (cherry picked from commit e29b499a211137cf3cf0c24a4ff82db1b806f132)

6.1.0-dev.390 | 2023-09-12 20:52:01 +0200

  * CreationInitsOptimizer: Use PreTypedef() instead of PreType() (Arne Welzel, Corelight)

    PreTypedef() does not require the PreID() trampoline and ensures
    it's only called for IDs that are types. Also allows dropping
    the const_cast<> due to id->GetType() returning a const TypePtr which
    is different from a `const Type*`...

  * GH-3260: Fix deferred record initialization (Arne Welzel, Corelight)

    Put RecordFieldInit instances into creation_inits during parsing and
    determine their deferrability in an InitPostScript step. Any
    RecordFieldInits can be deferred are moved into deferred_inits.

  * testing/btest: Un-deferred record initalization tests (Arne Welzel, Corelight)

    These baselines should align with what Zeek 5.2 or before did.

6.1.0-dev.385 | 2023-09-11 10:15:51 -0700

  * Move allocation of analyzer_hash to file_mgr (Tim Wojtulewicz, Corelight)

6.1.0-dev.382 | 2023-09-08 16:46:53 +0200

  * fix for ZAM statement-level profiling (broken by GH-3199) (Vern Paxson, Corelight)

  * GH-3249: ZAM fixes for compatibility with GH-3249 changes (Vern Paxson, Corelight)

  * GH-3249: -O gen-C++ fixes for compatibility with GH-3249 changes (Vern Paxson, Corelight)

  * minor BTest reordering to diminish differences with script optimization (Vern Paxson, Corelight)

6.1.0-dev.377 | 2023-09-08 13:48:26 +0200

  * ID/Stmt: Introduce INIT_SKIP and use in ForStmt (Arne Welzel, Corelight)

    Currently, loop vars are added to a function scope's inits and
    initialized upon entering a function with default values. This
    applies to vector, record and table types.

    This is unnecessary for variables used in for loops as they are
    guaranteed to be initialized while iterating.

  * GH-3267: CompositeHash: Skip record initialization when recovering vals (Arne Welzel, Corelight)

    Initializing fields of recovered records caused running &default expression
    of fields just so that they are re-assigned in the next step with the
    recovered fields. The second test case still shows that the loop var
    is initialized as well even though that's not needed.

    Add tests for iterating over records with &default attributes for both,
    tables and vectors.

    Fixes #3267

6.1.0-dev.374 | 2023-09-07 15:58:22 +0200

  * GH-3276: Propagate failure reason in `spicy::decline_input`/`zeek::reject_protocol`. (Benjamin Bannier, Corelight)

    Closes #3276.

6.1.0-dev.372 | 2023-09-07 14:26:27 +0200

  * ci: update-alternative to have python3 be python3.9 (Arne Welzel, Corelight)

    Tweak so that python3 is correct for the Python_FIND_UNVERSIONED_NAMES
    change. python3 on 15.4 was pointing at Python3.6 still and for that
    there are no dev headers installed.

    A normal user could use --with-python instead, but in CI this is easier.

  * Add OpenSUSE Leap 15.5 to ci (Johanna Amann, Corelight)

    This also adds 15.4 to the list of distributions that are skipped by
    default - let me know if this is not a good idea. I assume that by now
    most people will be running 15.5 (15.4 will be EOL in ~3 months).

  * GH-3275: configure/CMakelists: Use Python_EXECUTABLE as hint for interpreter (Arne Welzel, Corelight)

    The PYTHON_EXECUTABLE variable was only interpreted by our own FindPythonDev.
    cmake's FindPython uses Python_EXECUTABLE instead.

    Also, if --with-python is not used, prefer unversioned executables that
    match the version requirement. Further, remove Python_ADDITIONAL_VERSIONS
    that is interpreted by FindPythonInterp which we're not using anymore.

    https://cmake.org/cmake/help/latest/module/FindPython.html#artifacts-specification
    https://cmake.org/cmake/help/latest/module/FindPythonInterp.html

    Closes #3275

6.1.0-dev.368 | 2023-09-07 13:28:42 +0200

  * GH-3278: spicy: Do not register port N+1 for port N in .evt file (Arne Welzel, Corelight)

  * Updating auxil/zeek-client submodule [nomail] (Christian Kreibich, Corelight)

6.1.0-dev.364 | 2023-09-05 19:56:59 +0200

  * DNS_Mgr: Use Process() for timeout expiration (Arne Welzel, Corelight)

    DNS_Mgr has a GetNextTimeout() implementation that may return 0.0. When
    that is the case, its IO source is enqueued as ready with an fd of -1.
    This in turn results in Process() being called instead of ProcessFd()
    in RunState.cc.

    Ensure timeouts behavior is properly handled by actually forwarding
    timeout indications to c-ares via DNS_Mgr::Process(). This results
    in pending DNS queries for which a timeout happened to actually
    timeout (when there's no other connectivity).

  * DNS_Mgr: Fix GetNextTimeout() returning absolute values (Arne Welzel, Corelight)

    Not sure, must have been some sort of left-over, but wasn't really
    effective due to Process() not being implemented.

6.1.0-dev.361 | 2023-09-04 14:00:37 +0200

  * Fix check for emailed notices (Justin Azoff, Corelight)

    This bug was introduced in 9d0bd96

6.1.0-dev.358 | 2023-09-01 11:53:55 +0200

  * GH-1705: http: Prevent request/response de-synchronization and unbounded state growth (Arne Welzel, Corelight)

    When http_reply events are received before http_request events, either
    through faking traffic or possible re-ordering, it is possible to trigger
    unbounded state growth due to later http_requests never being matched
    again with responses.

    Prevent this by synchronizing request/response counters when late
    requests come in.

    Also forcefully flush pending requests when http_replies are never
    observed either due to the analyzer having been disabled or because
    half-duplex traffic.

6.1.0-dev.356 | 2023-08-30 11:24:51 +0200

  * testing/external: Exclude telemetry.log from baselines (Arne Welzel, Corelight)

    The diffs produced by telemetry.log when introducing a weird or
    removing/adding protocol specific logs is overwhelming and distracting
    without providing value. Exclude telemetry.log similar to how we already
    exclude stats.log.

    Some more targeted telemetry.log tests exists in the normal testing/btest
    suite and that appears more sensible.

6.1.0-dev.354 | 2023-08-29 12:13:51 -0700

  * Forward OpenSSL include path to plugins (Dominik Charousset, Corelight)

6.1.0-dev.349 | 2023-08-29 09:11:21 -0700

  * CI: Add ubuntu 23.04 build (Tim Wojtulewicz, Corelight)

  * CI: Remove EOL ubuntu 22.10 build (Tim Wojtulewicz, Corelight)

6.1.0-dev.346 | 2023-08-29 09:39:10 +0200

  * pre-commit: autoupdate (Arne Welzel, Corelight)

    ...except for clang-format, because versions after v13.0.0 have
    borked the Whitesmith formatting. Also moves yapf from
    pre-commit/mirrors-yapf to google/yapf.

6.1.0-dev.343 | 2023-08-28 20:07:16 +0200

  * cmake: Bump submodule for removal of NEED_KRB5_H (Arne Welzel, Corelight)

  * zeek-config.h: Drop NEED_KRB5_H (Arne Welzel, Corelight)

    This is unused in the Zeek tree after the previous commit, remove it
    from zeek-config.h

  * File: Drop krb5.h include (Arne Welzel, Corelight)

    It looks as if krb5.h was only ever needed together with OpenSSL, then
    OpenSSL includes were removed, but the krb5.h ones stayed around.

6.1.0-dev.339 | 2023-08-28 10:53:37 -0700

  * change Trigger constructor to not potentially auto-delete itself (Vern Paxson, Corelight)

6.1.0-dev.337 | 2023-08-28 12:56:14 +0200

  * Update doc, cmake submodules for installing .spicy files (Arne Welzel, Corelight)

6.1.0-dev.336 | 2023-08-28 12:19:45 +0200

  * testing: Bump external test suite (Arne Welzel, Corelight)

  * dhcp: Handle is_orig=T for connections from server to 255.255.255.255 (Arne Welzel, Corelight)

    This works around the new semantics of is_orig=T for "connections"
    from DHCP servers to broadcast addresses. IMO, having the server address
    as originator in the conn.log is still more intuitive.

  * GH-3235: IPBasedAnalyzer: Don't flip connections when destination is broadcast (Arne Welzel, Corelight)

6.1.0-dev.331 | 2023-08-25 21:38:40 +0200

  * BTests for any/vector-of-any fixes (Vern Paxson, Corelight)

  * fixes for vector assignments involving "any"/"vector of any" types (Vern Paxson, Corelight)

6.1.0-dev.327 | 2023-08-24 12:17:42 -0700

  * updated notes regarding "-O gen-C++" maintenance (Vern Paxson, Corelight)

  * "-O gen-C++" support for "assert" statements (Vern Paxson, Corelight)

  * addressed some nits re "-O gen-C++" script optimization (Vern Paxson, Corelight)

  * fixes for compiling lambdas to C++ (Vern Paxson, Corelight)

  * fixes to avoid ambiguities in analyzing captures for script optimization (Vern Paxson, Corelight)

  * disambiguate lambdas by adding scoping and consideration of captures (Vern Paxson, Corelight)

  * addressed performance and correctness issues flagged by Coverity (Vern Paxson, Corelight)

6.1.0-dev.319 | 2023-08-24 12:26:00 +0200

  * PPP: Add PPP analyzer to handle LINKTYPE_PPP (0x9) (Arne Welzel, Corelight)

    Using pcaps from https://interop.seemann.io/ as samples for QUIC protocol
    data didn't produce a conn.log for the contained data. `tcpdump -r`
    and Wireshark do show the contained IP/UDP packets. Teach Zeek how
    to handle link type DLT_PPP 0x09 using a new PPP analyzer based on the
    PPPSerial analyzer code.

    Usual update to files/x509 baseline after adding new analyzer due
    to enum values changing.

6.1.0-dev.316 | 2023-08-23 13:04:59 -0700

  * CI: Force alpine image to rebuild (Tim Wojtulewicz, Corelight)

  * CI: Install python3.8 on older distros (Tim Wojtulewicz, Corelight)

  * CI: Remove Ubuntu 18 build (EOL) (Tim Wojtulewicz, Corelight)

6.1.0-dev.312 | 2023-08-23 12:31:16 +0100

  * Community-id: load main script in notice script, fix notice script (Johanna Amann, Corelight)

    This change makes the community-id script that adds the community id to
    notice.log automatically load the main script if this was not already
    loaded.

    In the past, the script just did not perform any action if the main
    script was not loaded.

    This change also makes the notice script respect the seed/base64
    settings that were set in the main script.

    Fixes GH-3242

6.1.0-dev.310 | 2023-08-22 14:59:28 +0200

  * GH-3218/GH-3219: Spicy: Extend functionality of `export` in EVT
    files. (Robin Sommer, Corelight)

    We now support selecting which fields of a unit type get exported into
    the automatically created Zeek record; as well as selecting which
    fields get a `&log` attribute added automatically to either all fields
    or to selected fields.

    Syntax:

    - To export only selected fields:

        export Foo::X with { field1, field3 };

    - To export all but selected fields:

        export Foo::X without { field2, field3 };

    - To `&log` all fields:

        export Foo::X &log;

    - To `&log` only selected fields:

        export Foo::X with { field1 &log, field3 }; # exports (only) field1 and field3, and marks field1 for logging

  * pre-commit: Pin to latest shfmt-py version (Arne Welzel, Corelight)

    This allows users to run shfmt-py with Python > 3.9. Also drop
    the explicit Python version for the setup-python action.

6.1.0-dev.303 | 2023-08-15 17:33:51 +0100

  * Raw reader: use posix_spawn instead of fork + exec (Johanna Amann, Corelight)

    This commit switchexisd the Raw reader to use posix_spawn, instead of
    the combination of fork + exec. This should be much more efficient, and
    also makes the code smaller, and easier to read and understand.

6.1.0-dev.300 | 2023-08-15 09:19:57 -0700

  * Force pre-commit to use python 3.9 (Tim Wojtulewicz, Corelight)

6.1.0-dev.298 | 2023-08-14 11:18:14 -0700

  * CI: update freebsd to 13.2 and 12.4 (Tim Wojtulewicz, Corelight)

6.1.0-dev.293 | 2023-08-11 10:34:52 -0700

  * Fix conversion with record types (Tim Wojtulewicz, Corelight)

  * Add conversion between set and vector using 'as' keyword (Tim Wojtulewicz, Corelight)

  * Add std::move for a couple of variables passed by value (Tim Wojtulewicz, Corelight)

6.1.0-dev.289 | 2023-08-11 09:52:24 -0700

  * Modbus: Add early return in case of data being too short to parse (Tim Wojtulewicz, Corelight)

6.1.0-dev.287 | 2023-08-09 09:08:33 -0700

  * Explicitly link Zeek executable against Spicy libraries in binary packaging mode. (Benjamin Bannier, Corelight)

6.1.0-dev.284 | 2023-08-08 10:55:03 -0700

  * Pass parsed file record information with ReadFile/WriteFile events (Tim Wojtulewicz, Corelight)

  * Add length field from header to ModbusHeaders record type (Tim Wojtulewicz, Corelight)

  * Modbus: Add support for Encapsulation Interface Transport (FC=2B) requests and responses (Tim Wojtulewicz, Corelight)

  * Modbus: Add support for Diagnostics (FC=8) requests and responses (Tim Wojtulewicz, Corelight)

6.1.0-dev.272 | 2023-08-07 09:41:34 -0700

  * Change VERSION variable in CMake config to ZEEK_VERSION_FULL (Tim Wojtulewicz, Corelight)

  * Update bifcl and binpac for FindPackage fixes (Tim Wojtulewicz, Corelight)

  * Remove usage of FindRequiredPackage (Tim Wojtulewicz, Corelight)

  * Update zeekctl and cmake to deprecate FindRequiredPackage (Tim Wojtulewicz, Corelight)

  * Update broker submodule for pybind11 + cmake changes (Tim Wojtulewicz, Corelight)

  * ci: Bump centos-7 (Arne Welzel, Corelight)

6.1.0-dev.264 | 2023-08-07 10:22:18 +0200

  * Suppress new tsan findings from Ubuntu 22 upgrade (Tim Wojtulewicz)

  * Update tsan build to ubuntu22 (Tim Wojtulewicz, Corelight)

6.1.0-dev.261 | 2023-08-04 12:31:51 +0200

  * TableVal: Unify &default and &default_insert lookups (Arne Welzel, Corelight)

    Introduce DefaultAttr() helper to avoid a bit of duplicated code.

  * GH-2668: Add &default_insert attribute for tables (Arne Welzel, Corelight)

    This is based on the discussion in zeek/zeek#2668. Using &default with tables
    can be confusing as the default value is not inserted. The following example
    prints an empty table at the end even new Service records was instantiated.

        type Service: record {
            occurrences: count &default=0;
            last_seen: time &default=network_time();
        };

        global services: table[string] of Service &default=Service();

        event zeek_init()
            {
            services["http"]$occurrences += 1;
            services["http"]$last_seen = network_time();

            print services;
            }

    Changing above &default to &default_insert will insert the newly created
    default value upon a missed lookup and act less surprising.

    Other examples that caused confusion previously revolved around table of sets
     or table of vectors and `add` or `+=` not working as expected.

        tbl_of_vector["http"] += 1
        add tbl_of_set["http"][1];

6.1.0-dev.258 | 2023-08-03 07:18:21 +0100

  * Do not forward padding to downstream TCP packet analyzer (Johanna Amann, Corelight)

    This is similar to GH-3206. There do not seem to be practical
    consequences - but we should still fix it.

    This also includes the udp-testcase that was forgotten in GH-3206.

6.1.0-dev.256 | 2023-08-02 10:11:22 -0700

  * Rename magic __bro_plugin__ file to __zeek_plugin__ (Tim Wojtulewicz, Corelight)

6.1.0-dev.254 | 2023-08-02 09:23:18 +0200

  * GH-3122: UsageAnalyzer: Collect identifiers found in attributes as seeds (Arne Welzel, Corelight)

    This marks every identifier used within an attribute as seeds. The scenario
    this avoids is functions referenced through attributes on unused tables or
    record types (&default, &expire_func, ...) being dinged as unused as
    that's rather confusing.

    Also adds test for the above and a light smoke test into language/ as it
    doesn't appear we had coverage here.

  * Force refresh of all CI docker images (Tim Wojtulewicz, Corelight)

    There's something going on with the image cache on Cirrus where the images
    are sometimes vanishing from the cache, thus causing builds to fail
    because it can't load them. This forces a rebuild of all of the images,
    thus refreshing the cached version of all of them.

6.1.0-dev.250 | 2023-08-01 09:58:04 -0700

  * Fix memory leak in script_opt's Expr code (Tim Wojtulewicz, Corelight)

6.1.0-dev.248 | 2023-07-31 15:59:41 -0700

  * Do not forward more than the remaining data to downstream UDP analyzer (Johanna Amann, Corelight)

    This fixes a bug introduced in 2b9de839b0948c7de3eb5ed4a397194f96aae6b5
    / GH-3080, which causes UDP padding to be sent to UDP based analyzers.

    Fixes GH-3205.

  * Force rebuild of windows CI image (Tim Wojtulewicz, Corelight)

6.1.0-dev.243 | 2023-07-26 14:46:06 -0700

  * increase BTest wait time to abide ZAM compilation times (Vern Paxson, Corelight)

  * avoid script coverage overhead (especially memory) when using ZAM (Vern Paxson, Corelight)

  * fixes for correctly tracking which functions have been fully inlined (Vern Paxson, Corelight)

  * support for discarding ASTs once compiled via ZAM script optimization (Vern Paxson, Corelight)

  * some code simplifications and streamlining (Vern Paxson, Corelight)

6.1.0-dev.236 | 2023-07-25 10:23:47 -0700

  * fix for installing identifiers for lambdas into the global scope (Vern Paxson, Corelight)

6.1.0-dev.234 | 2023-07-25 09:36:27 -0700

  * Better input framework error messages for unset non-optionals (Johanna Amann, Corelight)

    The input framework currently gives a rather opaque error message when
    encountering a line in which a required value is not provided. This
    change updates this behavior; the error message now provides the record
    element (or the name or the index element) which was not set in the
    input data, even though it is required to be set by the underlying Zeek
    type.

6.1.0-dev.232 | 2023-07-25 09:34:54 -0700

  * Fix script_opt/cpp test scripts to work correctly on all platforms (Tim Wojtulewicz)

  * CPP code-gen: Use std::move for constructor arguments (Tim Wojtulewicz)

  * CPP code-gen: Mark child classes final, remove final from function defs (Tim Wojtulewicz)

6.1.0-dev.225 | 2023-07-20 09:23:33 -0700

  * Print type of opaque instead of 'no value description' (Tim Wojtulewicz, Corelight)

  * Add compiler flag to disable unknown pragma warnings on Windows (Tim Wojtulewicz, Corelight)

    Without this flag you get a warning from every use of Type.h since it
    contains a '#pramga GCC' for disabling a deprecation warning.

6.1.0-dev.220 | 2023-07-17 16:35:06 -0700

  * allow signature actions to be dynamically updated (Vern Paxson, Corelight)

6.1.0-dev.218 | 2023-07-17 16:33:22 -0700

  * more extensive ZAM inlining & compilation of lambdas (Vern Paxson, Corelight)

  * fixes for ZAM compilation of switch statements (Vern Paxson, Corelight)

  * fix for ZAM inlining failing to inline function call arguments (Vern Paxson, Corelight)

  * avoid constructing TypeList's on-the-fly for ListVal's with fixed types (Vern Paxson, Corelight)

  * -O gen-C++ fix for recent smart pointer changes (Vern Paxson, Corelight)

  * minor BTest generalizations & simplifications (Vern Paxson, Corelight)

  * fixes for compiling function calls that return non-error nil ValPtr's (Vern Paxson, Corelight)

6.1.0-dev.207 | 2023-07-14 09:15:16 -0700

  * -O gen-C++ fix for recent smart pointer changes (Vern Paxson, Corelight)

  * -a cpp Btest baseline updates (Vern Paxson, Corelight)

  * minor BTest generalizations & simplifications (Vern Paxson, Corelight)

  * fixes for compiling function calls that return non-error nil ValPtr's (Vern Paxson, Corelight)

6.1.0-dev.202 | 2023-07-12 17:29:26 +0200

  * ci: Bust Ubuntu 22.10 image cache (Arne Welzel, Corelight)

    Cirrus jobs failing to pull the image, try rebuilding.

  * GH-2709: Support :: prefix to reference global namespace, deprecate GLOBAL:: (Arne Welzel, Corelight)

6.1.0-dev.198 | 2023-07-11 16:16:56 -0700

  * dce-rpc: Test cases for unbounded state growth (Arne Welzel, Corelight)

    Pcaps produced as shown in #3145 using a samba container and rpcclient.

  * dce-rpc: Handle smb2_close_request() in scripts (Arne Welzel, Corelight)

    If there's a request to close a fid and it's in the dce_rpc_backing
    table, remove it from there.

  * smb/dce-rpc: Cleanup DCE-RPC analyzers when fid is closed and limit them (Arne Welzel, Corelight)

    This patch does two things:

    1) For SMB close requests, tear down any associated DCE-RPC
       analyzer if one exists.

    2) Protect from fid_to_analyzer_map growing unbounded by introducing a
       new SMB::max_dce_rpc_analyzers limit and forcefully wipe the
       analyzers if exceeded. Propagate this to script land as event
       smb_discarded_dce_rpc_analyzers() for additional cleanup.

    This is mostly to fix how the binpac SMB analyzer tracks individual
    DCE-RPC analyzers per open fid. Connections that re-open the same or
    different pipe may currently allocate unbounded number of analyzers.

    Closes #3145.

  * dce-rpc: Do not repeatedly register removal hooks (Arne Welzel, Corelight)

    ...once should be enough.

  * removed use of unnecessary member variable accessors for StmtList methods (Vern Paxson, Corelight)

  * migrated some raw pointers to smart pointers (Vern Paxson, Corelight)

6.1.0-dev.190 | 2023-07-07 09:56:56 -0700

  * Remove unused state_label() method from ssl analyzer binpac files (Tim Wojtulewicz)

  * Mark some overridden functions with override keyword (Tim Wojtulewicz)

  * Use emplace_back over push_back where appropriate (Tim Wojtulewicz)

  * packet_analysis: Define all plugin type as final (Tim Wojtulewicz)

  * Use std::make_unique in one place instead of declaring unique_ptr directly (Tim Wojtulewicz)

  * Avoid unnecessary type names in return statements (Tim Wojtulewicz)

  * Simplify type trait usage (remove ::value usage) (Tim Wojtulewicz)

  * A handful of int-to-bool conversions (Tim Wojtulewicz)

  * Replace empty destructor bodies with =default definitions (Tim Wojtulewicz)

  * Reduce amount of files passed to clang-tidy (Tim Wojtulewicz)

6.1.0-dev.179 | 2023-07-07 11:45:53 +0200

  * GH-3157: [Spicy] Support `switch` fields when exporting Spicy types to Zeek. (Robin Sommer, Corelight)

6.1.0-dev.177 | 2023-07-06 11:34:38 +0200

  * Fix for nested "when" statements leading to confusion over scoping (Vern Paxson, Corelight)

6.1.0-dev.175 | 2023-07-06 09:27:03 +0200

  * Define early_shutdown lambda earlier in zeek-setup, avoids build failure with gperftools (Tim Wojtulewicz)

6.1.0-dev.172 | 2023-07-05 13:32:42 -0700

  * Set `SPICY_BUILD_DIRECTORY` in `zeek-path-dev`. (Robin Sommer, Corelight)

    Having this set allows running Spicy tests from inside
    `auxil/spicy/tests`.

6.1.0-dev.170 | 2023-07-05 13:32:21 -0700

  * Update dump-events baseline, not running with OpenSSL 3 (Arne Welzel, Corelight)

  * Conn: In-place val flip and connection_flipped() (Arne Welzel, Corelight)

    Avoids loosing state on a connection value when a connection is flipped.

    Fixes up the NTP baseline as well where this was visible: analyzer_confirmation_info()
    was raised for a connection value which was immediately forgotten due to
    the subsequent connection flipping.

    Closed #3028

  * Conn: Remove is_version_sep() (Arne Welzel, Corelight)

    Leftover from 2655a653314237f5354fdb644bf642d0d5867f63.

  * Remove icmp_conn leftovers (Arne Welzel, Corelight)

    Roughly 2.5 years ago all events taking the ``icmp_conn`` parameter were
    removed with 44ad614094db3c57fef7fb226c539f6f090a5421 and the NetVar.cc
    type not populated anymore.

    Remove the left-overs in script land, too.

6.1.0-dev.164 | 2023-07-05 10:23:27 -0700

  * Bump Spicy to latest release. (Benjamin Bannier, Corelight)

    This is the `master` branch companion patch for `release/6.0`'s
    d6ad7f56c56925c0b0b3f27e98eb13b66e956c8f.

    Closes #3101.

6.1.0-dev.161 | 2023-07-05 08:33:36 -0700

  * Update binpac, bifcl, and broker submodules for funsigned-char change (Tim Wojtulewicz, Corelight)

  * Add /J flag on Windows to force unsigned char (Tim Wojtulewicz, Corelight)

  * Use -funsigned-char flag when building Zeek (Tim Wojtulewicz, Corelight)

6.1.0-dev.157 | 2023-07-05 15:24:57 +0200

  * GH-3136: global_ids(): Include module names (Arne Welzel, Corelight)

    Ad-hoc include module names in the global_ids() table. Table values will
    have the type_name field set to "module" and their key in the table is
    prefixed with "module " to avoid clashes with existing global identifiers
    shadowing module names (Management::Node being an existing example).

  * Update package-manager submodule [nomail] (Tim Wojtulewicz, Corelight)

6.1.0-dev.154 | 2023-06-30 17:32:17 +0200

  * Update zeekjs submodule to 0.9.4 (Arne Welzel, Corelight)

  * Fixup external hashes modified during merge (Arne Welzel, Corelight)

6.1.0-dev.152 | 2023-06-30 10:19:19 +0200

  * documentation updates (Vern Paxson, Corelight)

  * updates to ZAM-specific BTest baseline (Vern Paxson, Corelight)

  * Address PR review feedback on zam-feature-complete (Vern Paxson, Corelight)

  * Updates to BTest baselines due to previous BTest tweaks (Vern Paxson, Corelight)

  * tweaks to BTests for ZAM feature completeness; new ZAM-only btest (Vern Paxson, Corelight)

  * removal of unused functionality and some follow-on simplifications (Vern Paxson, Corelight)

  * feature completeness for ZAM (Vern Paxson, Corelight)

  * -O gen-C++ tweaks to be compatible with ZAM changes (Vern Paxson, Corelight)

  * ZAM support for "when" statements (Vern Paxson, Corelight)

  * ZAM changes intermixed with lambda and "when" support (Vern Paxson, Corelight)

  * WhenStmt/WhenInfo restructuring in support of ZAM "when" statements (Vern Paxson, Corelight)

  * ZAM support for lambdas (Vern Paxson, Corelight)

  * ZAM internals have a notion of "captures" as global-like variables (Vern Paxson, Corelight)

  * AST profiling enhnacements in support of script optimization for lambdas/whens (Vern Paxson, Corelight)

  * refinements to LambdaExpr's to provide flexibility, support for ZVal captures (Vern Paxson, Corelight)

  * support in ScriptFunc class for ZVal-oriented vector of captures (Vern Paxson, Corelight)

  * simplifications to the Frame class now that it no longer has to support old-style captures (Vern Paxson, Corelight)

  * use Ingredients directly for constructing functions (Vern Paxson, Corelight)

  * the "Capture" struct is now a class (Vern Paxson, Corelight)

  * more debugging information when dumping script optimization data structures (Vern Paxson, Corelight)

  * bug fixes for script optimization intermediate forms (Vern Paxson, Corelight)

  * clarifying comments, interface tightening (Vern Paxson, Corelight)

  * added some class accessors/set-ers (Vern Paxson, Corelight)

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

6.1.0-dev.127 | 2023-06-29 11:22:58 -0700

  * Move CMake template files to separate directory (Tim Wojtulewicz, Corelight)

6.1.0-dev.125 | 2023-06-29 18:33:41 +0200

  * Add test with MUD example (Arne Welzel, Corelight)

    This came up as a use-case on GitHub showing the limitations of the
    opinionated approach. Add it as an example.

    Reference https://www.rfc-editor.org/rfc/rfc8520#section-9

  * GH-3142: from_json: Support function to normalize key names (Arne Welzel, Corelight)

    When a JSON document contains key names containing colons or other
    special characters that are not valid in Zeek identifiers, from_json()
    cannot be used to parse such input.

    This change allows a customizable normalization function.

6.1.0-dev.121 | 2023-06-27 12:18:38 -0700

  * Prefer Spicy include directories of this build over accidental ones. (Benjamin Bannier, Corelight)

    Closes #3153.

6.1.0-dev.118 | 2023-06-27 18:17:40 +0200

  * fix http AUTHORIZATION base64 decode failed (progmboy)

6.1.0-dev.115 | 2023-06-21 17:45:35 -0700

  * formatters/JSON: Make JSON::NullDoubleWriter use zeek::json::detail version (Arne Welzel, Corelight)

    Not using inheritance and preferring composition to avoid including the
    detail/json.h header do an indirection via a unique_ptr and then just
    re-use the Double() implementation.

  * formatters/JSON: Remove surrounding_braces (Arne Welzel, Corelight)

    This seems to have become unused 4 years ago with 9b76e8faf44e90c41f33f24b18900a50f0840c5a,
    remove it.

  * formatters/JSON: Prepare to remove rapidjson from installed Zeek headers (Arne Welzel, Corelight)

    threading/formatters/JSON.h currently includes rapidjson headers for declaring
    the NullDoubleWriter. This appears mostly an internal detail, but
    results in the situation that 1) we need to ship rapidjson headers with
    the Zeek install tree and 2) taking care that external plugins are able
    to find these headers should they include formatters/JSON.h.

    There are currently no other Zeek headers that include rapidjson, so this
    seems very unfortunate and self-inflicted given it's not actually required.

    Attempt to hide this implementation detail with the goal to remove the
    rapidjson includes with v7.1 and then also stop bundling and exposing
    the include path to external plugins.

    The NullDoubleWriter implementation moves into a new formatters/detail/json.h
    header which is not installed.

    Closes #3128

6.1.0-dev.110 | 2023-06-21 15:36:32 -0700

  * Update zeekjs submodule (Tim Wojtulewicz, Corelight)

  * Update broker submodule (Tim Wojtulewicz, Corelight)

6.1.0-dev.108 | 2023-06-21 15:33:50 -0700

  * telemetry: Disable metrics centralization by default (Arne Welzel, Corelight)

    Move the telemetry/cluster.zeek file over into policy/frameworks/telemetry/prometheus.zeek.

    Mention it in local.zeek.

    Relates to zeek/broker#366.

6.1.0-dev.106 | 2023-06-21 15:33:24 -0700

  * GSSAPI: basic support for MIC/WRAP tokens (Johanna Amann, Corelight)

    When MIC/WRAP tokens are encountered, we now skip the message, instead
    of raising a parse error. The data in the messages is encrypted - so it
    does not seem work to raise an event at the moment.

6.1.0-dev.104 | 2023-06-20 10:15:24 -0700

  * CI: Force rebuild of tumbleweed VM to pick up newer version of python (Tim Wojtulewicz, Corelight)

    The version of python included in the existing VM doesn't have the sqlite
    module included for some reason. Forcing the VM to rebuild installs python311
    which does include it, fixing a build failure.

  * Stmt/Assert: Use std::move() for backtrace element (Arne Welzel, Corelight)

6.1.0-dev.100 | 2023-06-15 21:13:46 -0700

  * Update external testing hashes for 6.1 deprecation changes (Tim Wojtulewicz, Corelight)

  * Remove leftover baseline files from 6.1 deprecation removal (Tim Wojtulewicz, Corelight)

6.1.0-dev.96 | 2023-06-15 16:27:36 -0700

  * Update dce-rpc constants (Tim Wojtulewicz, Corelight)

  * Update external testing repo hashes to current master commits (Tim Wojtulewicz, Corelight)

6.1.0-dev.93 | 2023-06-15 16:25:50 -0700

  * -O gen-C++ refinements for BiF failures, negative vector indices, boolean vector operations (Vern Paxson, Corelight)

6.1.0-dev.91 | 2023-06-15 14:21:03 -0700

  * ZAM bug fix for incorrect management of global variables (Vern Paxson, Corelight)

  * ZAM bug fix (simplification) for nested inline functions that don't do anything (Vern Paxson, Corelight)

  * ZAM support for negative vector indices (Vern Paxson, Corelight)

  * ZAM ignores new "assert" statement (Vern Paxson, Corelight)

  * fixes for doing "script validation" to check for ZAM compile-ability (Vern Paxson, Corelight)

  * tweak for Broker BTest that needs a delay for ZAM (Vern Paxson, Corelight)

  * BTest to check for invalid negative vector indices (Vern Paxson, Corelight)

6.1.0-dev.80 | 2023-06-15 12:15:29 -0700

  * Force -std=c++17 mode for plugin targets, remove use of RequireCXX17.cmake (Tim Wojtulewicz, Corelight)

  * Add Spicy updates to NEWS for 6.0. (Robin Sommer, Corelight)

6.1.0-dev.76 | 2023-06-15 18:36:25 +0200

  * plugin/Manager: Warn when plugin with the same name is found (Arne Welzel, Corelight)

    This was brought up on Slack as a potential source of confusion during
    development as it's not visible when plugin directories are ignored outside
    of looking into the plugin debug stream. I'd actually prefer to just
    FatalError() this, but a warning seems reasonably visible for interactive
    usage.

  * Options: Do not output full usage on --test error (Arne Welzel, Corelight)

    ...mostly because the usage output is very long and the actual useful
    error message scrolls off the screen.

  * GH-3090: CMakeLists: Propagate DOCTEST defines to external plugins (Arne Welzel, Corelight)

    If Zeek has not been built with doctest enabled then it's not easy
    to run a plugin's tests (and if they really wanted to they could
    again undef the DOCTEST_CONFIG_DISABLE and provide their own doctest
    main() implementation.

  * GH-3090: CMakeLists: Add rapidjson/include to zeek_dynamic_plugin_base (Arne Welzel, Corelight)

    threading/formatters/JSON.h has a rapidjson include. Extend the
    include directories of external plugins so they are setup to find
    these in Zeek's install tree.

  * GH-3090: ZeekPluginBootstrap: Encode Zeek's CMAKE_BUILD_TYPE (Arne Welzel, Corelight)

    ...and bump cmake to have plugin's make use of it.

6.1.0-dev.66 | 2023-06-14 10:09:46 -0700

  * Change table initialization deprecation to error (Tim Wojtulewicz, Corelight)

  * Remove 5.2 deprecation we missed (Tim Wojtulewicz, Corelight)

  * Remove Supervisor::NodeConfig (6.1 deprecation) (Tim Wojtulewicz, Corelight)

  * Remove LogAscii::logdir (6.1 deprecation) (Tim Wojtulewicz, Corelight)

  * Make break/next/continue outside loops an error (6.1 deprecation) (Tim Wojtulewicz, Corelight)

  * Remove analyzer_confirmation/analyzer_violation events (6.1 deprecation) (Tim Wojtulewicz, Corelight)

  * Remove full scripts marked as 6.1 deprecations (Tim Wojtulewicz, Corelight)

  * Remove script functions marked as unused (6.1 deprecations) (Tim Wojtulewicz, Corelight)

  * Remove deprecations tagged for v6.1 (Tim Wojtulewicz, Corelight)

6.1.0-dev.54 | 2023-06-14 18:55:27 +0200

  * docker: Add libnode to enable JavaScript support (Arne Welzel, Corelight)

  * docker: Bump images to Debian 12 (Arne Welzel, Corelight)

6.1.0-dev.50 | 2023-06-14 09:25:58 -0700

  * Fix usage of realpath on macOS, instead preferring grealpath (Tim Wojtulewicz, Corelight)

  * Don't generate minimal tarball anymore (Tim Wojtulewicz, Corelight)

  * NEWS: Fix enumeration in Breaking Changes (Arne Welzel, Corelight)

6.1.0-dev.46 | 2023-06-14 12:59:41 +0200

  * all: Fix typos identified by typos pre-commit hook (Arne Welzel, Corelight)

  * NEWS typo fixes (Arne Welzel, Corelight)

  * Start with a typos pre-commit file (Arne Welzel, Corelight)

    Inspired by Spicy and me creating typos everywhere. Some of the
    suggestions are also very reasonable.

6.1.0-dev.42 | 2023-06-14 12:51:08 +0200

  * NEWS: Small section about assert statement (Arne Welzel, Corelight)

  * Stmt: Rework assertion hooks break semantics (Arne Welzel, Corelight)

    Using break in either of the hooks allows to suppress the default reporter
    error message rather than suppressing solely based on the existence of an
    assertion_failure() handler.

  * Stmt: Introduce assert statement and related hooks (Arne Welzel, Corelight)

    including two hooks called assertion_failure() and assertion_result() for
    customization and tracking of assertion results.

  * ZeekArgs: Helper for empty arguments (Arne Welzel, Corelight)

  * Reporter: Allow AssertStmt to throw InterpreterException (Arne Welzel, Corelight)

  * Lift backtrace() code into Func.{h,cc} (Arne Welzel, Corelight)

    This is to be re-used by the assertion facility.

6.1.0-dev.32 | 2023-06-13 11:29:36 -0700

  * Clarify subitem relationship in CMake configure summary. (Benjamin Bannier, Corelight)

6.1.0-dev.30 | 2023-06-13 11:28:26 -0700

  * tests: Do not use archiver_rotation_format_func as postprocessor (Arne Welzel, Corelight)

    This test triggered ubsan by putting a function with the wrong type
    as a post-processor into the .shadow file. Don't do that.

    Likely Zeek should provide a better error message, but hand-crafting
    .shadow files isn't what is normally done and this is to fix the
    master build for now.

6.1.0-dev.28 | 2023-06-13 17:33:31 +0200

  * GH-3112: cluster/logger: Fix leftover-log-rotation in multi-logger setups (Arne Welzel, Corelight)

    Populating log_metadata during zeek_init() is too late for the
    leftover-log-rotation functionality, so do it at script parse time.

    Also, prepend archiver_ to the log_metadata table and encoding function
    due to being in the global namespace and to align with the
    archiver_rotation_format_func. This hasn't been in a released
    version yet, so fine to rename still.

    Closes #3112

  * cluster/logger: Fix global var reference (Arne Welzel, Corelight)

6.1.0-dev.25 | 2023-06-12 15:27:20 -0700

  * Update broker submodule [nomail] (Tim Wojtulewicz, Corelight)

6.1.0-dev.23 | 2023-06-12 11:06:34 -0700

  * Bump `auxil/spicy`. (Benjamin Bannier, Corelight)

6.1.0-dev.19 | 2023-06-12 11:04:42 -0700

  * test-all-policy: Do not load nodes-experimental/manager.zeek (Arne Welzel, Corelight)

    Turns out loading this script in non-cluster mode uses Cluster::log()
    and creates cluster.log in the external baselines saying "cluster
    started". Do not load it into test-all-policy.zeek and instead rely
    on the new test-all-policy-cluster.test to load it transitively
    when running as manager for basic checking.

  * cluster/main: Remove extra @if ( Cluster::is_enabled() ) (Arne Welzel, Corelight)

    These have been discussed in the context of "@if &analyze" [1] and
    am much in favor for not disabling/removing ~100 lines (more than
    fits on a single terminal) out from the middle of a file. There's no
    performance impact for having these handlers enabled unconditionally.
    Also, any future work on "@if &analyze" will look at them again which
    we could also skip.

    This also reverts back to the behavior where the Cluster::LOG stream
    is created even in non cluster setups like in previous Zeek versions.
    As long as no one writes to it there's essentially no difference. If
    someone does write to Cluster::LOG, I'd argue not black holing these
    messages is better. Schema generators using Log::active_streams will
    continue to discover Cluster::LOG even if they run in non-cluster
    mode.

    https://github.com/zeek/zeek/pull/3062#discussion_r1200498905

6.1.0-dev.16 | 2023-06-12 10:56:55 -0700

  * socks/dpd: Add newer auth methods (Arne Welzel, Corelight)

    The IANA has allocated a few more authentication methods, minimally
    extend the signature to catch those.

  * socks/dpd: Fix socks5_server side signature (Arne Welzel, Corelight)

    The server replies with \x05 and identifier for the chosen method.
    Not quite sure what the previous signature meant capture.

    See also: https://datatracker.ietf.org/doc/html/rfc1928#section-3

    Closes #3099.

6.1.0-dev.13 | 2023-06-09 11:03:54 +0200

  * ci/windows: choco --localonly is gone (Arne Welzel, Corelight)

    choco 2.0 is now used after some caching changes on the Cirrus side [1]
    and the --localonly flag is gone from choco [2], remove its usage.

    [1] https://github.com/cirruslabs/cirrus-ci-docs/issues/1174#issuecomment-1580928673
    [2] https://docs.chocolatey.org/en-us/guides/upgrading-to-chocolatey-v2-v6#the-list-command-now-lists-local-packages-only-and-the-local-only-and-lo-options-have-been-removed

6.1.0-dev.8 | 2023-06-05 14:16:53 +0200

  * Update zeekctl for pysubnetree/patricia upgrade (Tim Wojtulewicz, Corelight)

  * Update link to slack in README.md (Tim Wojtulewicz, Corelight)

6.1.0-dev.4 | 2023-05-31 13:48:49 -0700

  * Update bifcl, binpac, and broker repos for cmake changes (Tim Wojtulewicz)

  * bison_target argument changes from HEADER to DEFINES_FILE (Tim Wojtulewicz, Corelight)

  * Update cmake submodule [nomail] (Tim Wojtulewicz, Corelight)

  * Start of 6.1.0 development (Tim Wojtulewicz, Corelight)

6.0.0-dev.683 | 2023-05-31 09:50:46 +0200

  * Simplify code generated for Spicy analyzer port ranges. (Benjamin Bannier, Corelight)

    We previously would reprent port ranges from EVT files element-wise.
    This can potentially generate a lot of code (all on a single line
    though) which some versions of GCC seem to have trouble with, and which
    also causes JIT overhead.

    With this patch we switch to directly representing ranges. Single ports
    are represented as ranges `[start, start]`.

    Closes #3094.

  * catch-and-release: Mark cr_check_rule as is_used (Arne Welzel, Corelight)

  * Add test-all-policy-cluster (Arne Welzel, Corelight)

    After the introduction of @if ... analyze, a lot of warnings were
    triggered due to nested @if and @if .. analyze usage.

    Add a test for coverage of all policy scripts in cluster mode
    for the usual node types so this does not happen again.

6.0.0-dev.662 | 2023-05-26 20:51:43 +0200

  * Update ZeekJS submodule to 0.9.1 (Arne Welzel, Corelight)

6.0.0-dev.660 | 2023-05-26 10:03:29 +0200

  * CMakeLists: Skip zeek-version.h include for zeek_objs, too (Arne Welzel, Corelight)

    I've continued to see somewhat slower builds after Zeek version bumps. It
    appears files covered by zeek_objs didn't have -DZEEK_CONFIG_SKIP_VERSION_H
    set causing ccache invalidation after a version bump.

  * Bump cluster testsuite to latest main (Arne Welzel, Corelight)

6.0.0-dev.657 | 2023-05-25 19:49:20 -0700

  * Fix minor type-clash warning on Windows (Tim Wojtulewicz)

  * Fix Coverity warning involving object copy in TLS binpac code (Tim Wojtulewicz, Corelight)

6.0.0-dev.654 | 2023-05-25 20:01:37 +0200

  * Address wire/capture length feedback (Arne Welzel, Corelight)

  * packet_analysis/TCP: Do not use untrusted len for DeliverPacket() (Arne Welzel, Corelight)

    We should not be passing the untrusted TCP header length into
    DeliverPacket(). Also, DeliverPacket() cap len parameter should
    be the capture length of the packet, not remaining data.

  * GH-2683: Add regression test using pcap from GH-2683 (Arne Welzel, Corelight)

  * Add btest to test Geneve->VXLAN->Truncated inner packet (Tim Wojtulewicz, Corelight)

  * IP: Update packet->len with accumulated fragment size (Arne Welzel, Corelight)

    With packet->len representing the wire length and other places
    relying on it, ensure it's updated for fragments as well. This
    assumes non-truncated fragments right now. Otherwise we'd need
    to teach the FragmentReassembler to somehow track this independently
    but it would be a mess.

  * UDP: Forward any remaining data (also empty) to session-analysis (Arne Welzel, Corelight)

    The protocol analyzers are prepared to receive truncated data and
    this way we give analyzers a chance to look at data. We previously
    allowed empty data being passed: When len ended up 0 and remaining
    was 0 too.

  * IPTunnel: Compute inner wire length based on cap_len differences. (Arne Welzel, Corelight)

  * IP: fix weird name to not be ipv6 specific (Tim Wojtulewicz, Corelight)

  * UDP: don't validate checksum if caplen < len (Tim Wojtulewicz, Corelight)

    This may happen with truncated packets and will cause asan builds to bail out
    before the packet can be forwarded along. The TCP analyzer already has this
    check, but it's missing for UDP.

  * PIA: Modernize how struct initialization is done (Tim Wojtulewicz, Corelight)

6.0.0-dev.643 | 2023-05-25 09:03:40 -0700

  * btest.cfg: Set HILTI_CXX_COMPILER_LAUNCHER based on build/CMakeCache.txt (Arne Welzel, Corelight)

    If CMakeCache.txt indicates the Zeek build was done with ccache,
    take the CCACHE_PROGRAM:FILEPATH= line and populate the
    HILTI_CXX_COMPILER_LAUNCHER environment.

    For local development with ccache, this speeds up the spicy tests
    significantly after the initial run without knowing or needing know
    that setting HILTI_CXX_COMPILER_LAUNCHER would've helped.

6.0.0-dev.641 | 2023-05-25 09:03:08 -0700

  * Speed up Spicy-related tests. (Benjamin Bannier, Corelight)

    This patch changes invocations of `spicyz` and similar Spicy tools in
    tests which perform compilation to use debug mode via passing `-d`. This
    in turn leads to Spicy compiling generated C++ code in debug as opposed
    to release mode which typically seems to require less CPU time and RAM.
    For a local test running with `btest -j 16` and no caching via
    `HILTI_CXX_COMPILER_LAUNCER` this sped up running of BTests under
    `spicy/` by about 40s on my machine (120s vs 160s).

6.0.0-dev.636 | 2023-05-24 09:41:01 +0200

  * stats: Add zeek-net-packet-lag-seconds metric (Arne Welzel, Corelight)

  * GH-3060: Support negative index lookups for vectors (Tim Wojtulewicz, Corelight)

6.0.0-dev.628 | 2023-05-23 19:32:05 +0200

  * ci: Disable openh264 repository on tumbleweed (Arne Welzel, Corelight)

6.0.0-dev.626 | 2023-05-23 17:34:43 +0200

  * Bump cmake to fix build of multiple Spicy analyzers in a package. (Robin Sommer, Corelight)

6.0.0-dev.624 | 2023-05-23 15:45:34 +0200

  * GH-3071: Fix crashing when disable_analyzer() called on root analyzers (Arne Welzel, Corelight)

6.0.0-dev.622 | 2023-05-22 16:38:56 -0700

  * Revert part of previous CMake debug type fix (Tim Wojtulewicz)

6.0.0-dev.620 | 2023-05-22 16:34:06 -0700

  * btest/input/raw: Fix reread test (Arne Welzel, Corelight)

    This seems to have relied on the reading file twice behavior simply
    testing that 16 lines are observed. Switch to using two separate
    files and doing a system("mv ...") to trigger the REREAD logic, there's
    not force_update() needed and it wouldn't do anything if the file
    hadn't changed anyway.

  * input/Raw: Avoid reading file twice in MODE_REREAD (Arne Welzel, Corelight)

    Found while writing documentation and being confused why
    all lines and end_of_data() arrive twice during startup.

    The test is a bit fuzzy, but does fail reliably without
    the changes to Raw.cc

    Also fix not checking dev in the MODE_REREAD path.

    Closes #3053

6.0.0-dev.617 | 2023-05-22 10:14:27 +0200

  * Fix CMAKE_BUILD_TYPE_LOWER definition-vs-use ordering problem (Christian Kreibich, Corelight)

    It needs to be defined by the time we create zeek-config, which happens before
    its current definition. To avoid a redundant TOLOWER when we check for presence
    of --enable-debug at the beginning, this also switches this to a case-unadjusted
    comparison to "Debug", which we use elsewhere in the file too.

6.0.0-dev.614 | 2023-05-19 10:51:35 -0700

  * Allow plugins to call find_package(Zeek) (Dominik Charousset)

6.0.0-dev.611 | 2023-05-19 09:37:39 -0700

  * files: Warn once for missing get_file_handle() (Arne Welzel, Corelight)

    Repeating the message for every new call to get_file_handle() is not
    very useful. It's pretty much an analyzer configuration issue so logging
    it once should be enough.

  * MIME: Re-use cur_entity_id for EndOfFile() (Arne Welzel, Corelight)

    If DataIn() was called and a cur_entity_id (file_id) has been produced
    previously, re-use it for calls to EndOfFile(). This avoids a costly
    event_mgr.Drain() when we already have that information. It should be safer,
    too, as `get_file_handle()` in script may generate a different ID and
    thereby de-synchronizing.

  * zeek-fuzzer-setup: Configure fake DNS (Arne Welzel, Corelight)

    I'm not sure if we somehow set this for oss-fuzz through the environment,
    but didn't find anything obvious.

    Running oss-fuzz reproducers locally can triggers lookups to malware.hash.cymru.com
    and potentially other domains due to loading local.zeek.

  * SupportAnalyzer: Stop delivering to disabled parent analyzer (Arne Welzel, Corelight)

    When the parent of a support analyzer has been disabled, short-circuit
    delivering stream or packet data to it.

    The specific scenario this avoids is the Content-Line analyzer continuing
    to feed data lines into an disabled SMTP analyzer in turn creating more
    events.

    This is primarily useful for our fuzzing setup where data chunks up to 1MB
    are generated and fed into the analyzer pipeline. In the real-world, chunk
    sizes are usually bounded to packet size. Certain TCP reassembly constellations
    may trigger these scenarios, however.

    Closes #168

  * Add length checking to ToRawPktHdrVal for truncated packets (Tim Wojtulewicz, Corelight)

  * ftp: No unbounded directory command re-use (Arne Welzel, Corelight)

    OSS-Fuzz generated traffic containing a CWD command with a single very large
    path argument (427kb) starting with ".___/` \x00\x00...", This is followed
    by a large number of ftp replies with code 250. The directory logic in
    ftp_reply() would match every incoming reply with the one pending CWD command,
    triggering path buildup ending with something 120MB in size.

    Protect from re-using a directory command by setting a flag in the
    CmdArg record when it was consumed for the path traversal logic.

    This doesn't prevent unbounded path build-up generally, but does prevent the
    amplification of a single large command with very many small ftp_replies.
    Re-using a pending path command seems like a bug as well.

6.0.0-dev.605 | 2023-05-18 08:54:41 -0700

  * Fix CMake ordering issue leaving configuration paths unset. (Robin Sommer, Corelight)

6.0.0-dev.602 | 2023-05-17 16:10:57 +0200

  * Add license header to zeek-config*.h and zeek-version.h (Arne Welzel, Corelight)

  * Rename util-config.h to zeek-config-paths.h and install it (Arne Welzel, Corelight)

    The util-config.h has never been installed previously. Skimming the history,
    it was only meant for inclusion from util.cc, hence the name. Now that it's
    included from some other headers, rename it to align with what it
    contains and install it, too.

  * Ensure spicyz/config.h is installed (Arne Welzel, Corelight)

6.0.0-dev.598 | 2023-05-17 12:46:23 +0200

  * Introduce environment variables to override more paths configured
    into `spicyz` (`ZEEK_SPICY_DATA_PATH`, `ZEEK_SPICY_MODULE_PATH`).
    (Robin Sommer, Corelight).

  * Fix an ordering issue in Spicy support's CMake config. This led to
    variables left unset inside the subdirectory. (Robin Sommer,
    Corelight)

6.0.0-dev.595 | 2023-05-17 10:36:22 +0200

  * Fix get_active_node_count for node types not present. (Jan Grashoefer, Corelight)

6.0.0-dev.592 | 2023-05-17 09:02:09 +0200

  * zeekygen: Render function parameters as :param x: instead of :x: (Arne Welzel, Corelight)

    We're currently rendering parameter descriptions from .bif file into
    the .rst as follows:

        :cid: The connection identifier.

        :aid: The analyzer ID.

    Switch this to :param cid: instead so that we can have Sphinx deal with
    this as param docfield and group all parameters into a single section.

    Currently, having the bare :cid: style causes sphinx to treat it as an
    unknown field type, capitalize it and render it.

6.0.0-dev.589 | 2023-05-16 12:15:07 +0200

  * Move Spicy submodule from `auxil/spicy/spicy` to `auxil/spicy`.
    (Robin Sommer, Corelight)

6.0.0-dev.587 | 2023-05-16 11:40:40 +0200

  * Integrate Spicy plugin into Zeek proper. (Robin Sommer, Corelight)

    The plugin code is now a part of Zeek, located inside the standard
    `src/` tree. Going forward, we will maintain it here and phase out
    the external plugin.

    The integration reflects the `spicy-plugin` code as of
    `d8c296b81cc2a11`.

    In addition to moving the code into Zeek's source tree, this comes
    with a couple small functional changes:

    - `spicyz` no longer tries to infer if it's running from the build
      directory. Instead `ZEEK_SPICY_LIBRARY` can be set to a custom
      location. `zeek-set-path.sh` does that now.

    - ZEEK_CONFIG can be set to change what `spicyz -z` print out. This is
      primarily for backwards compatibility.

    The minimum Spicy version is now 1.8 (i.e., current `main` branch
    at the time of merge).

    For now, this all remains backwards compatible with the current
    `zkg` analyzer templates so that they work with both external and
    integrated Spicy support. Later, once we don't need to support any
    external Spicy plugin versions anymore, we can clean up the
    templates as well.

  * Add `zkg_provides` to `--build-info`. (Robin Sommer, Corelight)

    This makes dependencies explicit that Zeek provides built-in for
    `zkg`. It's in support of
    https://github.com/zeek/package-manager/pull/157.

    For now, `zkg_provides` contains the same data as `included_plugins`
    plus an entry for `spicy-plugin`.

  * Fix Spicy tuple type conversion. (Robin Sommer, Corelight)

    With an anonymous Spicy-side tuple type, we'd be missing an ID to
    create a Zeek-side record, leading to undefined behavior. To still
    support this case, we now make up an ID. In addition, we also could
    end up not correctly tracking type ID during conversion; using a stack
    now to handle recursion correctly.

  * Fix generation of Spicy's file IDs. (Robin Sommer, Corelight)

    They weren't stable, and potentially repetitive.
  * Modernize plugin test. (Robin Sommer, Corelight)

    Not using the `zeek/` include style could lead to path problems.

6.0.0-dev.570 | 2023-05-12 23:29:09 +0200

  * Introduce ZEEK_SEED_VALUES environment variable (Arne Welzel, Corelight)

    For "individually different but deterministic" runs specifying Zeek's
    seed as an environment variable eases setups as one can avoid creating
    extra seed files for each of the individual processes.

    It is an error to specify the new ZEEK_SEED_VALUES variable together
    with the existing ZEEK_SEED_FILE and -G. ZEEK_SEED takes precedence over
    deterministic mode (-D) like ZEEK_SEED_FILE does today already.

6.0.0-dev.568 | 2023-05-12 13:41:55 -0700

  * enhancements for event-tracing: (Vern Paxson, Corelight)
      - reporting of potentially sensitive constants
      - tracking of unsupported types enabling hand-editing to fix them
      - fixed generation of "unspecified" aggregates
      - fixed generation of IPv6 constants
      - fixed generation when running without a packet source

6.0.0-dev.563 | 2023-05-12 10:01:42 -0700

  * Fix memory leak caused by pattern compilation failure (mAsk°)

6.0.0-dev.559 | 2023-05-11 15:01:44 +0100

  * Add DTLSv1.3 support (Johanna Amann, Corelight)

    DTLSv1.3 changes the DTLS record format, introducing a completely new
    unified header - which is a first for DTLS.

    In case connection IDs are used, parsing of the unified header
    is skipped. This is due to the fact, that the header then contains a
    variable length element, with the length of the element not given in the
    header. Instead, the length is given in the client/server hello message
    of the opposite side of the connection (which we might have missed).

    Furthermore, parsing is not of a high importance, since we are not
    passing the connection ID, or any of the other parsed values of the
    unified header into scriptland.

  * SSL: Fix logging of Hello Retry Requests (Johanna Amann, Corelight)

    It turns out that we never logged hello retry requests correctly in the
    ssl_history field.

    Hello retry requests are (in their final version) signaled by a specific
    random value in the server random.

    This commit fixes this oversight, and hello retry requests are now
    correctly logged as such.

  * remote_event_ts_compat: Fix NETWORK_TIMESTAMP to NetworkTimestamp (Arne Welzel, Corelight)

6.0.0-dev.551 | 2023-05-11 14:00:31 +0200

  * Add compatibility tests for timestamped events. (Jan Grashoefer, Corelight)

    This adds compatibility tests for receiving non-timestamped events as
    well as providing timestamps via broker websockets.

  * Add timestamps to auto published broker events. (Jan Grashoefer, Corelight)

  * Add timestamps to manually published broker events. (Jan Grashoefer, Corelight)

  * Annotate scheduled events with intended timestamp. (Jan Grashoefer, Corelight)

  * Add timestamp to events. (Jan Grashoefer, Corelight)

6.0.0-dev.544 | 2023-05-11 00:01:20 +0200

  * GH-3028: policy/community-id: Do not use new_connection() (Arne Welzel, Corelight)

    Issue #3028 tracks how a flipped connections reset a connection's value
    including any state set during new_connection(). For the time being,
    update community-id functionality back to the original connection_state_remove()
    approach to avoid missing community_ids on flipped connections.

6.0.0-dev.541 | 2023-05-10 23:16:24 +0200

  * testing/zeek-version-link: Assume nm is there (Arne Welzel, Corelight)

  * Drop dependency for zeek_dynamic_plugin_base (Dominik Charousset, Corelight)

  * Fixup ifdef check in Plugin.h (Arne Welzel, Corelight)

  * Update cmake_minimum_required() in test plugins (Arne Welzel, Corelight)

  * testing: Add zeek-version-link tests (Arne Welzel, Corelight)

  * Clean up ZEEK_CONFIG_SKIP_VERSION_H issues (Dominik Charousset, Corelight)

6.0.0-dev.533 | 2023-05-09 13:38:37 -0700

  * core.network_time.broker: Test reliability improvement (Arne Welzel, Corelight)

    I wasn't able to reproduce this locally, but after looking at
    -B main-loop,tm for a bit it dawned that if the manager is sending
    ticks too fast, the Broker IO source may consume two ticks in one go
    before expiring timers and that would explain the observed baseline
    differences.

    Solve this by removing the reliance on realtime delays and switch to
    a request-reply pattern instead.

6.0.0-dev.531 | 2023-05-09 13:38:15 -0700

  * ci: Add public-ecr-vacuum.sh (Arne Welzel, Corelight)

6.0.0-dev.529 | 2023-05-09 12:54:28 -0700

  * Updating submodule(s) [nomail] (Tim Wojtulewicz, Corelight)

6.0.0-dev.527 | 2023-05-09 19:11:00 +0200

  * GH-2930: zeek.bif: Add log2() and ceil() (Arne Welzel, Corelight)

6.0.0-dev.525 | 2023-05-09 09:05:51 -0700

  * Use the same rules as cmake submodule to reformat Zeek (Tim Wojtulewicz, Corelight)

  * Update cmake submodule after reformat (Tim Wojtulewicz, Corelight)

6.0.0-dev.522 | 2023-05-09 15:19:43 +0200

  * cluster/supervisor: Multi-logger awareness (Arne Welzel, Corelight)

    When multiple loggers are configured in a Supervisor controlled cluster
    configuration, encode extra information into the rotated filename to
    identify which logger produced the log.

    This is similar to the approach taken for ZeekControl, re-using the
    log_suffix terminology, but as there's only a single zeek-archiver
    process and no postprocessors and no other side-channel for additional
    information, we encode extra metadata into the filename. zeek-archiver
    is extended to recognize the special metadata part of the filename.

    This also solves the issue that multiple loggers in a supervisor setup
    overwrite each others log files within a single log-queue directory.

  * Bump zeek-archiver submodule (Arne Welzel, Corelight)

6.0.0-dev.519 | 2023-05-09 11:03:32 +0200

  * Fixup Val.h/Val.cc: Actually move ValFromJSON into zeek::detail (Arne Welzel, Corelight)

6.0.0-dev.518 | 2023-05-09 10:19:46 +0200

  * Implement from_json bif (Fupeng Zhao)

6.0.0-dev.516 | 2023-05-05 14:08:15 -0700

  * BTest baseline updates for -O gen-C++ (Vern Paxson, Corelight)

  * updates to C++ maintenance scripts to better handle uncompilable BTests (Vern Paxson, Corelight)

  * added ZEEK_REPORT_UNCOMPILABLE environment variable for "-O report-uncompilable" (Vern Paxson, Corelight)

  * BTest baseline updates for ZAM (Vern Paxson, Corelight)

6.0.0-dev.511 | 2023-05-05 21:04:11 +0200

  * Revert "Skip version.h by default for Zeek sources" (Arne Welzel, Corelight)

    This reverts commit 8246baf25c692b658e7a39527f6652e37f5b5095.

    Actually fails the include_plugins CI test and I had just
    opened another related issue.

6.0.0-dev.510 | 2023-05-05 20:14:21 +0200

  * Skip version.h by default for Zeek sources (Dominik Charousset, Corelight)

6.0.0-dev.508 | 2023-05-05 08:48:10 -0700

  * generate-docs: Only update submodule pointer during scheduled builds (Tim Wojtulewicz, Corelight)

6.0.0-dev.505 | 2023-05-04 20:08:33 +0200

  * GH-2998: NTP: Detect out-of-order packets (Arne Welzel, Corelight)

    The NTP mode provides us with the identity of the endpoints. For the
    simple CLIENT / SERVER modes, flip the connection if we detect
    orig/resp disagreeing with what the message says. This mainly
    results in the history getting a ^ and the ntp.log / conn.log
    showing the corrected endpoints.

6.0.0-dev.503 | 2023-05-04 10:56:33 -0700

  * Include compiler in --show-config output (Dominik Charousset, Corelight)

  * Fix CMake option defaults on Windows (Dominik Charousset, Corelight)

  * Move build defaults from configure to CMake (Dominik Charousset, Corelight)

    Moving the defaults for build variables from the `configure` script to
    `CMakeLists.txt` gives the same default behavior on platforms where the
    `configure` script is not available (Windows) and also allows a pure
    CMake-based work flow (e.g., the standard `cmake -S . -B build`) without
    having to manually adjust the defaults.

    The `configure` script also becomes much simpler as a result.

6.0.0-dev.498 | 2023-05-04 09:30:18 +0200

  * scripts/smb2-main: Reset script-level state upon smb2_discarded_messages_state() (Arne Welzel, Corelight)

    This is similar to what the external corelight/zeek-smb-clear-state script
    does, but leverages the smb2_discarded_messages_state() event instead of
    regularly checking on the state of SMB connections.

    The pcap was created using the dperson/samba container image and mounting
    a share with Linux's CIFS filesystem, then copying the content of a
    directory with 100 files. The test uses a BPF filter to imitate mostly
    "half-duplex" traffic.

  * smb2: Limit per-connection read/ioctl/tree state (Arne Welzel, Corelight)

    Users on Slack observed memory growth in an environment with a lot of
    SMB traffic. jeprof memory profiling pointed at the offset and fid maps
    kept per-connection for smb2 read requests.

    These maps can grow unbounded if responses are seen before requests, there's
    packet drops, just one side of the connection is visible, or we fail to parse
    responses properly.

    Forcefully wipe out these maps when they grow too large and raise
    smb2_discarded_messages_state() to notify script land about this.

6.0.0-dev.493 | 2023-05-03 11:13:15 -0700

  * Update Mozilla CA and Google CT lists (Johanna Amann, Corelight)

6.0.0-dev.491 | 2023-05-03 09:31:36 -0700

  * Surround string assignments in zeek-config in quotes (Tim Wojtulewicz, Corelight)

6.0.0-dev.489 | 2023-05-03 09:31:19 -0700

  * Remove parser error message for bro_init, et al (Tim Wojtulewicz, Corelight)

6.0.0-dev.487 | 2023-05-03 09:30:55 -0700

  * Add additional length check to IEEE 802.11 analyzer (Tim Wojtulewicz, Corelight)

6.0.0-dev.484 | 2023-05-03 14:18:03 +0100

  * SSL: do not try to disable failed analyzer (Johanna Amann, Corelight)

    Currently, if a TLS/DTLS analyzer fails with a protocol violation, we
    will still try to remove the analyzer later, which results in the
    following error message:

    error: connection does not have analyzer specified to disable

    Now, instead we don't try removing the analyzer anymore, after a
    violation occurred.

6.0.0-dev.480 | 2023-05-02 20:28:55 +0200

  * ip4_hdr: Add DF, MF, offset and sum fields (Arne Welzel, Corelight)

    For low-level packet analysis use-cases, these fields are currently
    not script-land accessible via raw_packet() or so. They are accessible
    on the icmp_context record, but not on the actual ip4_hdr record, so
    add them.

  * GH-2991: unload: Fix unloading of packages (Arne Welzel, Corelight)

    @ynadji found that unloading packages doesn't work due to @unload not
    resolving the __load__.zeek file within a directory like @load does.

    Fixes #2991

6.0.0-dev.476 | 2023-05-02 11:12:44 -0700

  * Use workaround for setvbuf on Windows in DebugLogger/Extract file analyzer (Tim Wojtulewicz)

  * Add ifdef'd implementation of setvbuf to zeek::util (Tim Wojtulewicz, Corelight)

6.0.0-dev.473 | 2023-05-02 09:42:19 -0700

  * Fix smith-waterman sorting to follow correct Compare semantics (Tim Wojtulewicz)

6.0.0-dev.470 | 2023-04-28 16:33:46 -0700

  * CI: Automatically update opensuse-tumbleweed (Tim Wojtulewicz, Corelight)

  * CI: Remove Fedora 36, add Fedora 38 (Tim Wojtulewicz, Corelight)

6.0.0-dev.467 | 2023-04-28 10:01:13 +0200

  * GH-2791: pcap/Source: Allow more than 32bit for link and dropped stats (Arne Welzel, Corelight)

    The PktSrc::Stats object works with 64bit unsigned integers. Unfortunately,
    libpcap's struct pcap_stat is using 32bit values and users have reported
    the wrapping of these values being visible in their stats.log roughly every
    7.5 hours (~160kpps).

    This change moves tracking of link and drop counters into the PktSrc::Stats
    object (like is done for received and bytes_received) and updates them
    on a call to PcapSource::Statistics() with the difference to the
    previous stats values to prevent the wrap from becoming visible to
    script land.

    This doesn't cover the case of the stats counters wrapping around multiple
    times between two invocations of PktSrc::Statistics(). With the default
    interval of 5 minutes for the stats script, this seems acceptable.

    Closes #2791.

  * record_fields: Include information about optionality of fields (Arne Welzel, Corelight)

    This was reported as a wish for log schema generation, so add it...

  * Fix a few warnings from recent changes (Tim Wojtulewicz)

6.0.0-dev.461 | 2023-04-28 09:37:08 +0200

  * Simplify btests using cluster_started event. (Jan Grashoefer, Corelight)

6.0.0-dev.457 | 2023-04-27 11:25:45 -0700

  * Modify Windows test cmd file to actually run tests (Tim Wojtulewicz)

  * Add template file and cmake call for zeek-path setup on Windows (Tim Wojtulewicz)

6.0.0-dev.453 | 2023-04-27 13:22:43 +0200

  * Revert putting plugins into a fresh scope for now (Dominik Charousset, Corelight)

  * Propagate zeek-version.h skip via CMake properties (Dominik Charousset, Corelight)

    Add a define for `ZEEK_CONFIG_SKIP_VERSION_H` to static plugins as well
    as to dynamic plugins that we build alongside Zeek. When including
    `zeek-config.h` with this macro defined, the header skips including
    `zeek-version.h`.

6.0.0-dev.449 | 2023-04-27 12:16:46 +0200

  * Bump zeekctl to multi-logger version (Arne Welzel, Corelight)

  * logging: Support rotation_postprocessor_command_env (Arne Welzel, Corelight)

    This new table provides a mechanism to add environment variables to the
    postprocessor execution. Use case is from ZeekControl to inject a suffix
    to be used when running with multiple logger.

6.0.0-dev.445 | 2023-04-27 09:08:45 +0200

  * Update ZeekJS submodule to 0.9.0 (Arne Welzel, Corelight)

    * Performance improvements for the Redis Log::log_stream_policy example
      around PortVal wrapping and caching field offsets for property lookups.
    * Debug output is now channeled through PLUGIN_DBG_LOG() and available via
      zeek -B plugin-Zeek-JavaScript instead of unconditionally on stderr.
    * Reduced CMake output when Node.js isn't found

6.0.0-dev.443 | 2023-04-26 12:42:48 -0700

  * extend ZEEK_PROFILER_FILE profiling to include summaries for functions/hooks/event handlers (Vern Paxson, Corelight)

6.0.0-dev.441 | 2023-04-26 15:37:59 +0200

  * Update AF-Packet submodule (Arne Welzel, Corelight)

    Include Tim's cleanup and modernization fixes, too.

  * Update AF-Packet submodule (Arne Welzel, Corelight)

    * Mask VLAN ID from tp_vlan_tci field to fix vlan > 4095 reported by Zeek
      when PCP and/or DEI bits are set.
    * Descriptive error message when interface is down. Instead of
      "Invalid argument", Zeek now reports "interface is down".

6.0.0-dev.437 | 2023-04-25 13:07:57 -0700

  * Update docs and NEWS to include LLC, SNAP, and Novell packet analyzers (Tim Wojtulewicz)

  * Fix length checks in VLAN/Ethernet analyzers for non-ethertype protocols (Tim Wojtulewicz, Corelight)

  * Add forwarding from VLAN analyzer into LLC, SNAP, and Novell 802.3 analyzers (Tim Wojtulewicz, Corelight)

  * Remove non-standard way of forwarding out of the Ethernet analyzer (Tim Wojtulewicz, Corelight)

  * Add basic LLC, SNAP, and Novell 802.3 packet analyzers (Tim Wojtulewicz, Corelight)

  * ARP: add support for IEEE802 hardware type (Tim Wojtulewicz, Corelight)

6.0.0-dev.430 | 2023-04-25 11:37:44 -0700

  * Merge branch 'topic/timw/2167-aruba-expansion' (Tim Wojtulewicz)

    * topic/timw/2167-aruba-expansion:
      Add NEWS entry about 802.11 and ARUBA changes
      Remove workaround for tunnels from IEEE 802.11 analyzer
      PPP in the GRE analyzer is actually PP*T*P
      Add support for 802.11 A-MSDU aggregates
      Fix IEEE 802.11 analyzer to properly forward tunneled packets
      Fix IEEE 802.11 analyzer to skip packets with the Protected bit enabled
      Expand support for Aruba protocol types in GRE analyzer
      Default tunnel_type in iosource::Packet to NONE

  * Add NEWS entry about 802.11 and ARUBA changes (Tim Wojtulewicz)

  * Remove workaround for tunnels from IEEE 802.11 analyzer (Tim Wojtulewicz, Corelight)

  * PPP in the GRE analyzer is actually PP*T*P (Tim Wojtulewicz, Corelight)

  * Add support for 802.11 A-MSDU aggregates (Tim Wojtulewicz, Corelight)

  * Fix IEEE 802.11 analyzer to properly forward tunneled packets (Tim Wojtulewicz, Corelight)

    This mostly happens with Aruba, but could possibly happen with other tunnels too.

  * Fix IEEE 802.11 analyzer to skip packets with the Protected bit enabled (Tim Wojtulewicz, Corelight)

  * Expand support for Aruba protocol types in GRE analyzer (Tim Wojtulewicz, Corelight)

    This also fixes the GRE analyzer to forward into the IEEE 802.11 analyzer
    if it encounters Aruba packets with the proper protocol types. This way
    the QoS header can be handled correctly.

  * Default tunnel_type in iosource::Packet to NONE (Tim Wojtulewicz, Corelight)

6.0.0-dev.421 | 2023-04-25 12:39:01 +0200

  * Given the -C flag, set script-layer ignore_checksums to true. (Christian Kreibich, Corelight)

  * Remove references to bro_broker in broker/Manager.h (Tim Wojtulewicz, Corelight)

6.0.0-dev.416 | 2023-04-24 18:22:27 +0200

  * Add irc_dcc_send_ack event and fix missing fields (Fupeng Zhao)

6.0.0-dev.414 | 2023-04-24 14:36:32 +0200

  * cmake: Fixup BRO_PLUGIN_INSTALL_PATH references (Arne Welzel, Corelight)

    Commit 19cbb328a9b4598b71e0aa092bf8a03732da8d63 removed setting of
    BRO_PLUGIN_INSTALL_PATH in favor of ZEEK_PLUGIN_DIR. Replace left-over
    references the former.

6.0.0-dev.412 | 2023-04-24 10:33:29 +0200

  * Fix install directory for plugins (Dominik Charousset, Corelight)

6.0.0-dev.410 | 2023-04-24 09:46:59 +0200

  * policy: Import zeek-community-id scripts into protocols/conn frameworks/notice (Christian Kreibich, Corelight)

  * Add community_id_v1() based on corelight/zeek-community-id (Christian Kreibich, Corelight)

    "Community ID" has become an established flow hash for connection correlation
    across different monitoring and storage systems. Other NSMs have had native
    and built-in support for Community ID since late 2018. And even though the
    roots of "Community ID" are very close to Zeek, Zeek itself has never provided
    out-of-the-box support and instead required users to install an external plugin.

    While we try to make that installation as easy as possible, an external plugin
    always sets the bar higher for an initial setup and can be intimidating.
    It also requires a rebuild operation of the plugin during upgrades. Nothing
    overly complicated, but somewhat unnecessary for such popular functionality.

    This isn't a 1:1 import. The options are parameters and the "verbose"
    functionality  has been removed. Further, instead of a `connection`
    record, the new bif works with `conn_id`, allowing computation of the
    hash with little effort on the command line:

        $ zeek -e 'print community_id_v1([$orig_h=1.2.3.4, $orig_p=1024/tcp, $resp_h=5.6.7.8, $resp_p=80/tcp])'
        1:RcCrCS5fwYUeIzgDDx64EN3+okU

    Reference: https://github.com/corelight/zeek-community-id/

6.0.0-dev.405 | 2023-04-24 09:23:01 +0200

  * file_analysis/Manager: Remove RunState.h include (Arne Welzel, Corelight)

    The file_analysis/Manager.h header only needs run_state::terminating, so
    declare just that similar to what Val.h does. This breaks ZBody compilation
    due to OP_READING_LIVE_TRAFFIC_V and OP_READING_TRACES_V accessing
    run-state::reading_live and run_state::reading_traces. Add the include to
    ZBody.cc as a fix.

    This may break external plugins, too, but then they didn't have the
    right includes setup to begin with.

6.0.0-dev.402 | 2023-04-21 14:04:52 -0700

  * Update NEWS to cover cluster framework changes. (Jan Grashoefer, Corelight)

  * Add cluster_started restart tests. (Jan Grashoefer, Corelight)

  * Add basic cluster_started tests. (Jan Grashoefer, Corelight)

  * Add cluster_started and node_fully_connected events. (Jan Grashoefer, Corelight)

  * Add hook into cluster connection setup. (Jan Grashoefer, Corelight)

  * Add broadcast_topics set. (Jan Grashoefer, Corelight)

  * Generalize Cluster::worker_count. (Jan Grashoefer, Corelight)

6.0.0-dev.394 | 2023-04-20 16:03:45 -0700

  * Edit pass over the current 6.0 NEWS entries. (Christian Kreibich, Corelight)

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

6.0.0-dev.391 | 2023-04-19 19:48:50 +0200

  * zeek.bif: Remove cat_sep() fully var_arg changes (Arne Welzel, Corelight)

    These were introduced to better catch type violations at runtime. With
    bifcl doing these checks, revert to a better documented version.

  * GH-2935: broker/messaging: Runtime type checks for pool (Arne Welzel, Corelight)

    publish_hrw() and publish_rr() are excluded from type checking due to their
    variadic nature. Passing a wrong type for the pool argument previously triggered
    an abort, now the result is runtime errors. This isn't great, but it's
    better than crashing Zeek.

    Closes #2935

  * bifcl: Bump for runtime-type checks in var_arg bifs (Arne Welzel, Corelight)

6.0.0-dev.387 | 2023-04-19 09:01:32 -0700

  * Add call stacks to script profiler output (Tim Wojtulewicz, Corelight)

6.0.0-dev.385 | 2023-04-19 11:58:09 +0200

  * different fix for MSVC compiler issues (Vern Paxson, Corelight)

  * more general approach for addressing MSVC compiler issues with IntrusivePtr (Vern Paxson, Corelight)

  * restored RecordType::Create, now marked as deprecated (Vern Paxson, Corelight)
    tidying of namespaces and private class members
    simplification of flagging record field initializations that should be skipped
    address peculiar MSVC compilation complaint for IntrusivePtr's

  * clarifications and tidying for record field initializations (Vern Paxson, Corelight)

  * optimize record construction by deferring initializations of aggregates (Vern Paxson, Corelight)

  * compile-scripts-to-C++ speedups by switching to raw record access (Vern Paxson, Corelight)

  * logging speedup by switching to raw record access (Vern Paxson, Corelight)

  * remove redundant record coercions (Vern Paxson, Corelight)

6.0.0-dev.376 | 2023-04-19 10:14:02 +0200

  * Improve CMake variables, update cmake submodule (Dominik Charousset, Corelight)

  * Fix builds with plugins that use zeek-config (Dominik Charousset, Corelight)

    When building plugins externally with `zeek-config` (as opposed to using
    `ZEEK_DIST`), they point into the install prefix. There, we add a new
    file `ZeekPluginBootstrap.cmake` that helps `ZeekPlugin.cmake` to find
    everything else it needs from there.

    Our template for plugins generates a `configure` script that sets
    various variables with values from `zeek-config`. We only need
    `BROKER_ROOT_DIR` with the new bootstrapping logic. Everything else, we
    can get from the new bootstrapping file and from the CMake package file
    for Zeek.

6.0.0-dev.373 | 2023-04-18 11:11:44 -0700

  * log-caching-cluster: Wait for X509::known_log_certs to populate (Arne Welzel, Corelight)

    The known_log_certs table is populated asynchronously via broker after a
    Broker::peer_added. It may take a variable amount of time depending on where
    we run this test and it has been observed flaky specifically for the
    arm_debian11 task. Instead of racing, give worker-2 3 seconds for receiving
    the expected table content before continuing.

    Fixes #2885

6.0.0-dev.371 | 2023-04-14 13:02:35 +0200

  * ci/debian-11: Install libnode-dev, too (Arne Welzel, Corelight)

    Debian 11 doesn't have a new enough libnode version, so JavaScript
    should not be attempted to be built.

  * CMakeLists: Convert string append to list append (Arne Welzel, Corelight)

    Seems the builtin plugins started with string(APPEND ...) and that
    was copied over. Make it list(APPEND ...) instead.

  * Add experimental JavaScript support when libnode is available (Arne Welzel, Corelight)

        zeek.on('zeek_init', () => {
            console.log('Hello, Zeek!');
        });

    For interaction with external systems and HTTP APIs, JavaScript and the
    Node.js ecosystem beat Zeek script. Make it more easily accessible by
    including ZeekJS with Zeek directly.

    When a recent enough libnode version is found on the build system, ZeekJS is
    added as a builtin plugin. This behavior can be disabled via
    ``--disable-javascript``. Linux distributions providing such a package are
    Ubuntu (22.10) and Debian (testing/bookworm) as libnode-dev.
    Fedora provides it as nodejs-devel.

    This plugin takes over loading of .js or .cjs files. When no such files
    are provided to Zeek, Node and the V8 engine are not initialized and
    should not get into the way.

    This should be considered experimental.

6.0.0-dev.367 | 2023-04-14 10:32:17 +0200

  * Revert "Type: Add TypeManager->TypeList() and use for ListVal()" (Arne Welzel, Corelight)

    This reverts commit 24c606b4df92f9871964c5bcb2fe90e43a177b1f.

    This commit introduced a memory leak due to ListVal::Append() modifying
    the cached TYPE_ANY type list.

6.0.0-dev.364 | 2023-04-13 15:54:00 -0700

  * Add NEWS entry for the CMake changes (Dominik Charousset, Corelight)

  * Redesign subdir-libs and plugin scaffolding (Dominik Charousset, Corelight)

6.0.0-dev.361 | 2023-04-13 09:05:31 -0700

  * GH-2920: Don't warn on uninitialized container options (Tim Wojtulewicz, Corelight)

6.0.0-dev.359 | 2023-04-13 08:57:22 -0700

  * Reimplement fixes to handle commas at the end of config parser lines (Tim Wojtulewicz, Corelight)

  * Revert "Convert config framework to use std::regex" (Tim Wojtulewicz, Corelight)

    This reverts commit 65ee2287e9b74f861872d9e16b9c11fb300cfabd.

  * Revert "GH-636: Fix regex to handle commas at the end of config parser lines" (Tim Wojtulewicz, Corelight)

    This reverts commit 05bb50978905a6c3132b20eb2cfd246715f91356.

6.0.0-dev.355 | 2023-04-13 09:24:19 +0200

  * logging/Manager: Fix crash for rotation format function not returning (Arne Welzel, Corelight)

    While working on a rotation format function, ran into Zeek crashing
    when not returning a value from it, fix and recover the same way as
    for scripting errors.

6.0.0-dev.353 | 2023-04-12 13:23:29 -0700

  * iosource/Manager: Fix poll_interval updating using not-yet valid IsLive() (Arne Welzel, Corelight)

    Testing io_poll_interval_live tweaks with @dopheide-esnet on a Myricom based
    system to reduce CPU usage showed no visible effect.

    It turns out, the pkt_src->IsLive() call used to update poll_interval is only
    valid *after* calling ->Register() with the source. The conditional updating
    of the poll_interval introduced in 4fa3e4b9b4d78e6f20f42b9dbd85216403be07db
    never worked out how it was intended to.

    The fix ensures that

    * we actually use a poll_interval of 10 in the live case
    * changing io_poll_interval_live does have an effect

    This is a bit of a major change due to lowering the default poll_interval
    by a magnitude, but that seemed to have been the intention always. It's also
    tunable via redef, so worst case it can be adapted via configuration.

    As reference, with the default a Pcap::non_fd_timeout of 20usec *and* a
    poll_interval of 100, theoretically we'd be trying to ask a non-selectable
    packet source 500000 per second for a new packet. This is not a likely packet
    rate that a single worker would currently observe or manage to process.

6.0.0-dev.350 | 2023-04-11 15:41:31 -0700

  * Only use -Wno-changes-meaning if it exists (Tim Wojtulewicz, Corelight)

6.0.0-dev.348 | 2023-04-11 15:30:45 -0700

  * file_analysis/File: Report overflowing chunks as weird and discard/truncate (Arne Welzel, Corelight)

    This is one level above the Reassembler where we still have information
    about the file and source. A weird entry may looks as follows:

        1679759398.237353  ... file_offset_overflow    FXPLGt4SeMmlMKahJc: offset=fffffffffffffff7 len=10      F       zeek    HTTP

  * Reassem: Reject blocks overflowing 64bit upper (Arne Welzel, Corelight)

    The reassembler logic isn't wrap around safe, so just truncate or
    reject such blocks. For files specifically, a byte offset in the
    2**64 bytes represents 16EiB which is the maximum size supported
    by BTRFS or NTFS (and probably nothing we'd ever see in practice).

  * zeek-setup: Load scrips before running unit tests (Arne Welzel, Corelight)

    It is currently not possible to call a->Conn()->GetVal() or construct a
    zeek/file_analysis/File object from within doctests, as these quickly
    reference the unpopulated zeek::id namespace to construct Val objects
    of various types, making it hard write basic tests without completely
    re-organizing.

    Move running of the unit tests after parsing the scripts, so it is possible
    for some basic exercising of File objects within tests.

6.0.0-dev.344 | 2023-04-11 15:23:42 -0700

  * RDP: Instantiate SSL analyzer instead of PIA (Tim Wojtulewicz, Corelight)

  * RDP: add some enforcement to required values based on MS-RDPBCGR docs (Tim Wojtulewicz, Corelight)

6.0.0-dev.341 | 2023-04-11 15:16:39 -0700

  * Stop skipping expiration of empty DNS mappings (Tim Wojtulewicz, Corelight)

6.0.0-dev.339 | 2023-04-11 15:16:15 -0700

  * smtp: Validate mail transaction and disable SMTP analyzer if excessive (Arne Welzel, Corelight)

    An invalid mail transaction is determined as

    * RCPT TO command without a preceding MAIL FROM
    * a DATA command without a preceding RCPT TO

    and logged as a weird.

    The testing pcap for invalid mail transactions was produced with a Python
    script against a local exim4 configured to accept more errors and unknown
    commands than 3 by default:

        # exim4.conf.template
        smtp_max_synprot_errors = 100
        smtp_max_unknown_commands = 100

    See also: https://www.rfc-editor.org/rfc/rfc5321#section-3.3

  * generic-analyzer-fuzzer: Detect disable_analyzer() from scripts (Arne Welzel, Corelight)

    Test if the analyzer is removed from the TCPSessionAdapter during
    event processing. If we don't do this, we continue feeding the analyzer
    even if scripts decided to disable the analyzer.

    The analyzer instance isn't flagged as disabled itself, so we need
    to look at the parent's children.

6.0.0-dev.336 | 2023-04-11 14:50:47 -0700

  * ftp/main: Special case for intermediate reply lines (Arne Welzel, Corelight)

    The medium.trace in the private external test suite contains one
    session/server that violates the multi-line reply protocol and
    happened to work out fairly well regardless due to how we looked
    up the pending commands unconditionally before.

    Continue to match up reply lines that "look like they contain status codes"
    even if cont_resp = T. This still improves runtime for the OSS-Fuzz
    generated test case and keeps the external baselines valid.

    The affected session can be extracted as follows:

        zcat Traces/medium.trace.gz | tcpdump -r  - 'port 1491 and port 21'

    We could push this into the analyzer, too, minimally the RFC says:

        > If an intermediary line begins with a 3-digit number, the Server
        > must pad the front  to avoid confusion.

  * ftp/main: Skip get_pending_command() for intermediate reply lines (Arne Welzel, Corelight)

    Intermediate lines of multiline replies usually do not contain valid status
    codes (even if servers may opt to include them). Their content may be anything
    and likely unrelated to the original command. There's little reason for us
    trying to match them with a corresponding command.

    OSS-Fuzz generated a large command reply with very many intermediate lines
    which caused long processing times due to matching every line with all
    currently pending commands.
    This is a DoS vector against Zeek. The new ipv6-multiline-reply.trace and
    ipv6-retr-samba.trace files have been extracted from the external ipv6.trace.

6.0.0-dev.333 | 2023-04-11 12:05:29 -0700

  * Add cstdint to WeirdState.h to fix compilation error on gcc13 (Tim Wojtulewicz, Corelight)

  * Ignore -Wchanges-meaning warning in Spicy (Tim Wojtulewicz, Corelight)

    This fixes a build failure with GCC 13. It's meant as a short-term fix to avoid
    the failure in Zeek, but probably should be fixed more correctly upstream in
    the Spicy repo.

  * Update broker submodule to pull in GCC 13 fix (Tim Wojtulewicz, Corelight)

  * CI: Force rebuild of OpenSUSE Tumbleweed VM to pick up GCC 13 (Tim Wojtulewicz, Corelight)

6.0.0-dev.327 | 2023-04-08 11:22:57 -0700

  * Fix a long-standing bug in the math around continue_processing (Tim Wojtulewicz, Corelight)

  * Add is_processing_suspended BIF (Tim Wojtulewicz, Corelight)

6.0.0-dev.323 | 2023-04-06 13:46:28 -0700

  * Add NEWS entries for filtered packet statistics and telemetry (Tim Wojtulewicz, Corelight)

  * Report packet statistics via the telemetry framework (Tim Wojtulewicz, Corelight)

  * Add optional packet filtered statistics for packet sources (Tim Wojtulewicz, Corelight)

  * ci: Add debian-12 / bookworm (Arne Welzel, Corelight)

    The next version of Debian (bookworm) had a hard-freeze on
    2023-03-16. Seems reasonable to have it in CI now.

6.0.0-dev.317 | 2023-04-05 10:39:22 +0200

  * scan.l: Support @pragma, specifically push/pop ignore-deprecations (Arne Welzel, Corelight)

    This allows to ignore-deprecations as follows:

        @pragma push ignore-deprecations
        print r$deprecated;
        @pragma pop

    A bit of infrastructure is put in place for the push and pop, but
    nothing overly generalized for future pragmas.

  * Switch deprecations to reporter->Deprecation() (Arne Welzel, Corelight)

    Removes a bit of reliance around the magic DoLog() rendering at the
    cost of needing to open-code some of it. The new obj_desc_short()
    helper makes that acceptable, though.

  * Reporter: Add dedicated Deprecation() method (Arne Welzel, Corelight)

    Minimally, provide a way to funnel all deprecations through
    reporter->Deprecation() instead of various Warning() invocations.

  * Desc: Add obj_desc_short() (Arne Welzel, Corelight)

    Same as obj_desc() but use the short version and do not include the
    location information by default. New method instead of bool parameters
    for readability.

6.0.0-dev.312 | 2023-04-04 09:05:34 -0700

  * Update gen-zam submodule for std::move warnings (Tim Wojtulewicz, Corelight)

6.0.0-dev.310 | 2023-04-04 10:00:24 +0200

  * test suite update for minor change in "when" error messages (Vern Paxson, Corelight)

  * removed skeletal (non-functioning) "when" support from ZAM (Vern Paxson, Corelight)

  * simplify WhenInfo and Trigger classes given removal of old capture semantics (Vern Paxson, Corelight)

  * introduced notion of light-weight Frame clones (Vern Paxson, Corelight)

  * changed function_ingredients struct to FunctionIngredients class with accessors (Vern Paxson, Corelight)

6.0.0-dev.303 | 2023-04-03 16:37:02 +0200

  * addressed static analysis concern about possible null pointer (Vern Paxson, Corelight)

  * tweaks for "-O C++" of BTest's with conditional code (Vern Paxson, Corelight)

6.0.0-dev.300 | 2023-04-03 13:32:40 +0200

  * minor ZAM BTest baseline updates (Vern Paxson, Corelight)

  * fixed type mismatch for ssl_certificate_request event (Vern Paxson, Corelight)

  * skip ZAM optimization of invalid scripts (Vern Paxson, Corelight)

  * extended script validation to be call-able on a per-function basis (Vern Paxson, Corelight)

6.0.0-dev.295 | 2023-04-03 13:19:20 +0200

  * GH-2296: PktSrc: Introduce Pcap::non_fd_timeout (Arne Welzel, Corelight)

    Increasing this value 10x has lowered CPU usage on a Myricom based
    deployment significantly with reportedly no adverse side-effects.

    After reviewing the Zeek 3 IO loop, my hunch is that previously when
    no packets were available, we'd sleep 20usec every loop iteration after
    calling ->Process() on the packet source. With current master ->Process()
    is called 10 times on a packet source before going to sleep just once
    for 20 usec. Likely this explains the increased CPU usage reported.

    It's probably too risky to increase the current value, so introduce
    a const &redef value for advanced users to tweak it. A middle ground
    might be to lower ``io_poll_interval_live`` to 5 and increase the new
    ``Pcap::non_fd_timeout`` setting to 100usec.

    While this doesn't really fix #2296, we now have enough knobs for tweaking.

    Closes #2296.

6.0.0-dev.293 | 2023-03-31 10:38:45 +0200

  * Add test cases for Geneve. (Jan Grashoefer, Corelight)

  * Fix forwarding of tunnelled packets. (Jan Grashoefer, Corelight)

    This fixes a bug for AYIYA, Geneve and VXLAN forwarding encapsulated
    content only if it's longer than their header. A new weird is introduced
    to indicate empty tunnels.

6.0.0-dev.290 | 2023-03-31 10:33:53 +0200

  * Type: Add TypeManager->TypeList() and use for ListVal() (Arne Welzel, Corelight)

    It turns out that for every ListVal we construct, we also allocate
    and construct a new TypeList instance, even though they are all the
    same. Pre-create and cache the type instances in a new TypeManager.

    The following script runs ~10% faster for me after this change.

        global tbl: table[string] of string;
        global i = 0;
        while ( ++i < 10000000 )
            tbl["a"] = "a";

6.0.0-dev.288 | 2023-03-30 13:09:18 -0700

  * Add reporter framework to core.packet-analyzer-override btest (Tim Wojtulewicz, Corelight)

6.0.0-dev.286 | 2023-03-30 09:52:26 -0700

  * Consider cap len when forwarding into packet analysis. (Jan Grashoefer, Corelight)

    When forwarding into packet analysis from TCP or UDP, the protocol's
    length fields were trusted. This might be dangerous in case of truncated
    packets.

6.0.0-dev.284 | 2023-03-30 09:50:57 -0700

  * Reintroduce info when overriding packet analyzer mappings. (Jan Grashoefer, Corelight)

    In #2464 the warning when overriding a packet analyzer mapping was
    removed. While a warning seems indeed excessive, some info would still
    be nice to have.

  * Update Cirrus memory limits to avoid timeouts when building (Tim Wojtulewicz)

6.0.0-dev.280 | 2023-03-30 09:35:48 +0200

  * Analyzer: Add GetChildAnalyzer() and IsPreventedChildAnalyzer() (Arne Welzel, Corelight)

    GetChildAnalyzer() has the same semantics as HasChildAnalyzer(), but returns
    the raw pointer to the child analyzer. Main issue is memory management: That
    pointer is not guaranteed to stay valid. It might be disabled from script
    land or otherwise removed from the analyzer tree and subsequent
    deleted in one of the Forward* methods.

    IsPreventedChildAnalyzer() provides minimal introspection for prevented
    child analyzer tags and allows to remove some duplicated code.

6.0.0-dev.278 | 2023-03-29 14:34:26 -0700

  * Bump Spicy and spicy-plugin to their latest development snapshots. (Benjamin Bannier, Corelight)

6.0.0-dev.276 | 2023-03-28 09:19:59 -0700

  * Fix a bunch of -Wunqualified-std-cast-call warnings from clang 15 (Tim Wojtulewicz)

6.0.0-dev.274 | 2023-03-27 21:32:53 +0200

  * btest/broker: Add test using Python bindings and zeek -r (Arne Welzel, Corelight)

    This came up in community.zeek.org as a use-case and there
    were a few gaps and holes, so add a test showing minimally
    the current behavior.

  * Broker: Remove network time initialization (Arne Welzel, Corelight)

    Remove the special case of initializing network time if it hasn't
    happened yet. The argument about broker.log containing 0.0 timestamps
    is more a problem of the log, not something that would justify modifying
    network time globally. For broker.log and possibly cluster.log, it might
    be more reasonable to use current time, anyway.

    I was a bit wary about tables backed by broker stores being populated
    with network_time set to 0.0, but there seems to exist logic and assumptions
    that this is okay: It should be the same as if one populates a table with
    expirations set within zeek_init().

    In fact, staring a bit more, *not setting* network time might be more correct
    as workers that don't see packets would never set zeek_start_network_time
    which is used within the expiration computation.

6.0.0-dev.271 | 2023-03-24 09:12:04 +0100

  * cirrus: No container images on PRs (Arne Welzel, Corelight)

6.0.0-dev.268 | 2023-03-23 13:40:26 +0100

  * Ensure master/release builds run for zeek-security (Tim Wojtulewicz, Corelight)

6.0.0-dev.266 | 2023-03-23 13:35:28 +0100

  * iosource: Remove non-existing ManagerBase friend (Arne Welzel, Corelight)

  * broker::Manager: use_realtime_false when allow_network_time_forward=F (Arne Welzel, Corelight)

    Run the broker in non-realtime mode when allow_network_time_forward=F.
    This may need an extra option for really advanced use-cases, but for
    now this seems reasonable.

  * A set of tests around network time handling (Arne Welzel, Corelight)

  * broker::Manager: No more network_time forwarding (Arne Welzel, Corelight)

    network_time forwarding will happen in the main-loop before draining the
    EventMgr so timers/events scheduled based on broker messages should
    behave similarly. This also keeps network_time unaffected during
    non pseudo-realtime trace processing.

  * TimerMgr: No network_time updates in Process() (Arne Welzel, Corelight)

    network_time forwarding will now happen centrally in the main loop.
    The TimerMgr returns a valid timeout that can be waited for and will
    trigger network_time advancement, so we don't need to do it.

  * Event: No more network_time updates (Arne Welzel, Corelight)

    The whole docs read like this was only used to update the
    network_time, so there may be a follow-up to ditch EventMgr
    even being an IOSource (which could be argued it's not IO).

  * RunState: Implement forward_network_time_if_applicable() (Arne Welzel, Corelight)

    Add a central place where the decision when it's okay to update network time
    to the current time (wallclock) is. It checks for pseudo_realtime and packet
    source existence as well as packet source idleness.

    A new const &redef allows to completely disable forwarding of network time.

  * PktSrc: Add HasBeenIdleFor() method (Arne Welzel, Corelight)

    This method will be used by the main loop to determine if an interface
    has become idle. Initially this will be used to determine when it is
    acceptable to update network_time to the current time (wallclock).

  * PktSrc: Move termination pseduo_realtime special case to RunState (Arne Welzel, Corelight)

    This also removes setting pseduo_realtime to 0.0 in the main loop
    when the packet source has been closed. I had tried to understand
    the implications it actually seems, if we shutdown the iosource::Manager
    anyway, it shouldn't and it's just confusing.

  * Bump zeekctl and doc submodule for updates regarding Site::local_nets (Christian Kreibich, Corelight)

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

6.0.0-dev.250 | 2023-03-21 19:17:36 +0100

  * Expr: Factor out type tag extraction (Arne Welzel, Corelight)

  * Var: Add version to deprecated initialization (Arne Welzel, Corelight)

    This has been added for Zeek 5.0, so mark it for removal in Zeek 6.1.

  * Stmt: Error on deprecated when/local usage (Arne Welzel, Corelight)

    This has been around since Zeek v4.1, so it was warned about in Zeek 5.0
    LTS and we could've removed it with 5.1.

    Also removed merge_top_scope() from the zeek::detail namespace, it's
    unused now.

    Updated the when-aggregates test somehow. I'm not quite sure what had
    been tested there :-/

  * Expr: Remove vector scalar operations (Arne Welzel, Corelight)

    These seem to have been deprecated since 2018, so do it now.
    Unfortunately the message didn't contain a version when it'll
    be removed, but it's been long enough.

  * parse.y: Make out-of-scope use errors (Arne Welzel, Corelight)

    This was marked to be done in Zeek 5.1, so do it now. The message
    didn't include a version, unfortunately, but minimally there was
    a comment when it should happen.

  * scan.l: Remove unused deprecated_attr (Arne Welzel, Corelight)

6.0.0-dev.243 | 2023-03-21 16:40:35 +0100

  * Add pcap_file option to supervised nodes. (Jan Grashoefer, Corelight)

    This allows to start Supervised nodes with a pcap_file argument
    rather than interface.

6.0.0-dev.241 | 2023-03-21 10:10:50 +0100

  * Add a new plugin test with verbose IO source output (Arne Welzel, Corelight)

    This is mostly for documentation/verification purposes of how the IO loop
    currently does draining and when it picks up FD based (non packet) IO
    source. For example, it shows that currently FD based sources are processed
    fairly delayed and that we now also process two timeout sources that are ready.

  * iosource: Make poll intervals configurable (Arne Welzel, Corelight)

    This probably should not be changed by users, but it's useful for
    testing and experimentation rather than needing to recompile.

    Processing 100 packets without checking an FD based IO source can
    actually mean that FD based sources are never checked during a read
    of a very small pcap...

  * iomanager/Poll: Add zero-timeout timeout_src also when there's other events ready (Arne Welzel, Corelight)

    This would generally happen the next loop iteration around anyway, but
    seems nice to ensure a zero timeout source will be processed at the same
    time as sources with ready FDs.

  * iomanager: Collect all sources with zero timeouts as ready (Arne Welzel, Corelight)

    Previously, if two iosources returned 0.0 as their timeout, only
    one of them would be considered ready. An always ready source
    therefore may starve other ready ones due to this and minimally
    this behavior seems surprising.

    Offline pcap sources are always ready and return 0.0 for
    GetNextTimeout() (unless in pseudo-realtime), so we can
    also remove the offline source special case.

    One subtle side-effect of this change is that if an IO source
    returns a 0.0 timeout *and* it's file descriptor is ready in
    the same loop iteration, it may be processed twice.

6.0.0-dev.235 | 2023-03-20 10:04:55 -0700

  * ci/centos-stream-8: Add which command (Arne Welzel, Corelight)

  * Update zeekctl test baseline for command.peerstatus test (Tim Wojtulewicz)

  * Update zeekctl test baseline for changes to Site::local_nets (Tim Wojtulewicz)

6.0.0-dev.231 | 2023-03-18 17:10:33 -0700

  * Update Broker for fixing #2853 (Dominik Charousset, Corelight)

6.0.0-dev.229 | 2023-03-17 14:39:03 -0700

  * Fix integer signedness comparison warning with GCC 12.2 (Christian Kreibich, Corelight)

  * Update doc submodule (zeek-bot)

6.0.0-dev.226 | 2023-03-16 10:18:43 +0100

  * dns: Remove AD and CD flags from log (Arne Welzel, Corelight)

    There was a misunderstanding whether to include them by default in
    the dns.log, so remove them again.

    There had also been a discussion and quirk that AD of a request would
    always be overwritten by reply in the dns.log unless the reply is
    missing. For now, let users extend dns.log themselves for what best
    fits their requirements, rather than adding these flags by default.

    Add a btest to print AD and CD flags for smoke testing still.

6.0.0-dev.224 | 2023-03-15 20:02:13 -0700

  * Treat private address space as site-local by default (Christian Kreibich, Corelight)

    - Update NEWS to cover new Site::local_nets behavior
    - Bump external testsuites to reflect updated baselines
    - Add btests for new Site::local_nets behavior
    - Update plugins.hooks baseline to reflect added config framework activity
    - Update scripts.policy.misc.dump-events baseline
    - Update btest baselines to reflect the use of local address ranges
    - Tighten local-nets filtering in the logging framework's path-func-column-demote test
    - Fix scripts.policy.frameworks.intel.removal test given address locality info
    - Provide a mechanism to suppress logging of internal config framework activity

6.0.0-dev.212 | 2023-03-16 00:41:33 +0000

  * Move port_masked variable inside #else block (Tim Wojtulewicz, Corelight)

6.0.0-dev.210 | 2023-03-15 10:47:48 -0700

  * Add configure option for preallocating PortVal objects (Tim Wojtulewicz, Corelight)

6.0.0-dev.208 | 2023-03-15 14:10:07 +0100

  * Fix the x509-invalid-extension test on OpenSSL 3.1 (Johanna Amann, Corelight)

6.0.0-dev.206 | 2023-03-15 10:02:52 +0100

  * SSL/TLS: Parse CertificateRequest message (Johanna Amann, Corelight)

    This commit introduces parsing of the CertificateRequest message in the
    TLS handshake. It introduces a new event ssl_certificate_request, as
    well as a new function parse_distinguished_name, which can be used to
    parse part of the ssl_certificate_request event parameters.

    This commit also introduces a new policy script, which appends
    information about the CAs a TLS server requests in the
    CertificateRequest message, if it sends it.

6.0.0-dev.202 | 2023-03-14 10:33:32 +0100

  * GH-2672: Parse DNSSEC AD and CD bits (Michael R. Torres)

    Parse authentic data (AD) and checking disabled (CD) bits according to
    RFC 2535. Leaves the Z field as-is, in case users are already handling
    this elsewhere and depend on the value being the integer for all 3 bits.

    https://www.rfc-editor.org/rfc/rfc2535#section-6.1

    Fixes #2672

6.0.0-dev.199 | 2023-03-13 18:40:15 +0100

  * GH-2851: HTTP: Make Content-Range parsing more robust (Arne Welzel, Corelight)

    This was exposed by OSS-Fuzz after the HTTP/0.9 changes in zeek/zeek#2851:
    We do not check the result of parsing the from and last bytes of a
    Content-Range header and would reference uninitialized values on the stack
    if these were not valid.

    This doesn't seem as bad as it sounds outside of yielding non-sensible values:
    If the result was negative, we weird/bailed. If the result was positive, we
    already had to treat it with suspicion anyway and the SetPlainDelivery()
    logic accounts for that.

6.0.0-dev.197 | 2023-03-13 17:54:02 +0100

  * GH-2851: HTTP: Reset reply_message for HTTP/0.9 (Arne Welzel, Corelight)

    OSS-Fuzz tickled an assert when sending a HTTP response before a HTTP/0.9
    request. Avoid this by resetting reply_message upon seeing a HTTP/0.9 request.

    PCAP was generated artificially: Server sending a reply providing a
    Content-Length. Because HTTP/0.9 processing would remove the ContentLine
    support analyzer, more data was delivered to the HTTP_Message than
    expected, triggering an assert.

6.0.0-dev.195 | 2023-03-13 09:02:49 -0700

  * dlclose() handle if the library loaded wasn't a plugin (Tim Wojtulewicz, Corelight)

  * Fix uninitialized variables in Ascii writer (Tim Wojtulewicz, Corelight)

  * Fix use-after-move bug in Scope constructor (Tim Wojtulewicz, Corelight)

  * Add missing field initializations for ConnTuple (Tim Wojtulewicz, Corelight)

  * Fix possible resource leak in analyzer unit tests (Tim Wojtulewicz, Corelight)

6.0.0-dev.189 | 2023-03-13 09:55:56 +0100

  * Allow offline packet sources to register FDs. (Jan Grashoefer, Corelight)

  * GH-2039: GH-2842: PktSrc: Avoid calling ExtractNextPacketInternal() in GetNextTimeout() (Arne Welzel, Corelight)

    This reworks 2aec7640dd60c3ea3cffd82461588567e406db34 (zeek/zeek#2039) to
    avoid calling ExtractNextPacketInternal() within GetNextTimeout() for
    the non-pseudo-realtime case. Also relates to zeek/zeek#2842.

    The intention of the referenced change was to avoid a 0.00002 timeout when
    a non-selectable packet source has more packets queued. This was implemented
    by checking for a new packet within GetNextTimeout().

    The proposed change switches to an predictive approach: Use the result of
    the previous ExtractNextPacket() call (stored as had_packet) as an indication
    whether more packets are to be expected.

    Calling ExtractNextPacketInternal() within GetNextTimeout() may cause
    surprising behavior as some packet source may block [1] or spent a significant
    amount of time (e.g. applying BPF filters [2]) within ExtractNextPacket().
    The result of GetNextTimeout() should be available immediately as guidance
    for the main-loop and the actual work should happen within the ->Process()
    method.

    This change also attempts to separate the pseudo-realtime logic from the
    non-pseudo-realtime in an attempt show pseudo-realtime as special.

    [1] https://github.com/hosom/bro-napatech/blob/00c4d657e034927301d5f8d9bc03eca81e619699/src/Napatech.cc#L116
    [2] https://github.com/sethhall/bro-myricom/blob/58b25c8ebac6d184ff8ff0c27f8da2b603694dab/src/Myricom.cc#L250

6.0.0-dev.185 | 2023-03-13 09:38:34 +0100

  * GH-2837: cirrus: Add smoke testing for builtin plugins (Arne Welzel, Corelight)

  * ci/collect-repo-info: Make plugin VERSION reading more robust (Arne Welzel, Corelight)

  * configure: Quote --include-plugins argument (Arne Welzel, Corelight)

6.0.0-dev.179 | 2023-03-12 13:52:15 -0700

  * Force rebuild of CentOS 7 CI image to pick up git install (Tim Wojtulewicz, Corelight)

6.0.0-dev.177 | 2023-03-12 13:09:36 -0700

  * get_dns_stats: Expose total cache size and cached text entries (Arne Welzel, Corelight)

    It wasn't possible from script land to determine the total size
    of the cache table held by the DNS_Mgr. Add the total and also
    also the TEXT entries count.

6.0.0-dev.175 | 2023-03-10 15:50:04 -0700

  * Add weird for unknown HTTP/0.9 request method (Tim Wojtulewicz, Corelight)

  * Special case HTTP 0.9 early on (Tim Wojtulewicz, Corelight)

    Mostly, treat HTTP0.9 completely separate. Because we're doing raw
    delivery of a body directly, fake enough (connection_close=1, and finish
    headers manually) so that the MIME infrastructure thinks it is seeing a
    body.

    This deals better with the body due to accounting for the first line. Also
    it avoids the content line analyzer to strip CRLF/LF and the analyzer
    then adding CRLF unconditionally by fully bypassing the content line
    analyzer.

    Concretely, the vlan-mpls test case contains a HTTP response with LF only,
    but the previous implementation would use CRLF, accounting for two many bytes.
    Same for the http.no-version test which would previously report a body
    length of 280 and now is at 323 (which agrees with wireshark).

    Further, the mime_type detection for the http-09 test case works because
    it's now seeing the full body.

    Drawback: We don't extract headers when a server actually replies with
    a HTTP/1.1 message, but grrr, something needs to give I guess.

  * Remove a couple unnecessary break statements (Tim Wojtulewicz, Corelight)

6.0.0-dev.171 | 2023-03-10 15:38:04 -0700

  * Pin conan to 1.58 on Windows until conan.cmake supports 2.0 (Tim Wojtulewicz, Corelight)

6.0.0-dev.169 | 2023-03-10 09:05:03 +0100

  * ScriptValidation: Make break/next a warning until Zeek 6.1 (Arne Welzel, Corelight)

    The ja3 package uses next instead of return and triggers the new
    errors with Zeek 5.2. That seems somewhat bad.

6.0.0-dev.167 | 2023-03-08 13:21:36 +0100

  * Bump Spicy to latest main (Arne Welzel, Corelight)

6.0.0-dev.165 | 2023-03-08 12:07:45 +0100

  * Revert "Merge branch 'topic/jgras/iosource-offline-fd' of https://github.com/J-Gras/zeek" (Arne Welzel, Corelight)

    This reverts commit 957825441a2ed32421fd9955b1cd9d3cbf561da0, reversing
    changes made to c8cdc75f2bf56a369904ad54d9a9d05477fb44a3.

    Caused spurious CI failures in the external testing baselines. See zeek/zeek#2842.

6.0.0-dev.164 | 2023-03-08 11:05:07 +0100

  * Allow offline packet sources to register FDs. (Jan Grashoefer, Corelight)

6.0.0-dev.162 | 2023-03-08 10:43:00 +0100

  * extend BTest "path" canonicalization to include compiled-to-C++ variable names (Vern Paxson, Corelight)

  * use a subclass of TableType for incremental construction of compiled scripts (Vern Paxson, Corelight)

  * script_opt/CPP: errors, recursive type fixes, fix embedded comments (Vern Paxson, Corelight)

    better (than nothing) run-time errors for compiled scripts
    fixes for dealing with recursive types in compiled scripts
    fix for values in compiled scripts containing embedded comment markers

  * Fix for EnumVal's returning their underlying value (Vern Paxson, Corelight)

    Change EnumVal()->AsEnum() to zeek_int_t.

  * Manually prune CI ccache. (Benjamin Bannier, Corelight)

    Since ccache in Cirrus currently does not preserve timestamps the
    timestamp-based cache pruning implemented by ccache likely would just
    remove random files. To work around this, this patch implements a manual
    pruning step. This step runs after the build so that at least the files
    used in the build should have their timestamps updated. We can then
    force eviction of some of the unused files from the cache by cleaning
    the cache with a size less than the maximum cache size.

6.0.0-dev.155 | 2023-03-07 09:28:27 -0700

  * Add trigger_mgr to iosource_mgr later during startup (Tim Wojtulewicz, Corelight)

    This fixes a potential crash due to trigger_mgr getting shutdown earlier
    than dns_mgr, and dns_mgr then trying to use it after it's been deleted.
    This change forces the order of initialization/destruction in
    iosource_mgr to cause dns_mgr to be deleted first.

6.0.0-dev.150 | 2023-03-02 12:49:16 -0700

  * Default known stores to false to avoid Broker stores (Tim Wojtulewicz, Corelight)

6.0.0-dev.148 | 2023-03-02 10:05:06 +0100

  * logging/Manager: Remove unused variable (Arne Welzel, Corelight)

  * logging: Add telemetry for streams and log writers (Arne Welzel, Corelight)

    This adds one metric per log stream and one metric per log writer (path based)
    to track the number of writes on a stream level as well as on a writer level.

        $ curl -sSf localhost:8181/metrics | grep Conn
        zeek_log_writer_writes_total{endpoint="",filter-name="default",module="HTTP",path="http",stream="HTTP::LOG",writer="Log::WRITER_SQLITE"} 1 1677497572770
        zeek_log_stream_writes_total{endpoint="",module="HTTP",stream="HTTP::LOG"} 1 1677497572770

    The initial version of this change also included metrics around log
    write vetoes, but given no log policies exist in the default configuration
    and they are mostly interesting for a few streams/writers only, skip this
    for now. These can always be added by the script writer, too.

    The difference between the stream level writes and concrete writers can
    be used to deduce the number of vetoes (or errors) as a starting point.

6.0.0-dev.143 | 2023-03-01 10:40:36 +0000

  * X.509: expose the signature type inside the tbs certificate (Johanna Amann, Corelight)

    This change exposes the signature type inside the signed portion of an
    X.509 certificate. In the past, we only exposed the signature type that
    is contained inside the signature, which is outside the signed portion
    of the X.509 certificate.

    In theory, both signature fields should have the same value; it is,
    however, possible to encode differing values in both fields. The new
    field is not logged by default.

6.0.0-dev.141 | 2023-03-01 10:20:59 +0100

  * cirrus: Add zeekctl CI task (Arne Welzel, Corelight)

    Run the zeekctl testing suite within a Debian 11 container. This runs on
    pushes to master and release branches. Also, tagging PRs with fullci or
    zeekctlci triggers it.

6.0.0-dev.136 | 2023-02-28 09:24:34 -0700

  * Add build status, slack, and discourse badges to readme (Tim Wojtulewicz, Corelight)

6.0.0-dev.134 | 2023-02-27 14:48:16 -0700

  * Configure apt to retry a few times on Docker image builder tasks (Tim Wojtulewicz, Corelight)

6.0.0-dev.132 | 2023-02-27 13:47:41 -0700

  * Bump zeekctl and add NEWS entries (Arne Welzel, Corelight)

    These are added to the 5.2 section of NEWS. This is for backporting
    into release/5.2.

6.0.0-dev.130 | 2023-02-27 09:36:31 -0700

  * Update bifcl submodule [nomail] (Tim Wojtulewicz, Corelight)

6.0.0-dev.128 | 2023-02-27 12:36:09 +0100

  * GH-2801: Fix re-instantiation of analyzers that were just removed. (Robin Sommer, Corelight)

6.0.0-dev.126 | 2023-02-27 09:52:31 +0100

  * Update cmake submodule (Arne Welzel, Corelight)

    To pull in zeek/cmake#62.

  * cirrus: Also add ccache limits to container builds (Arne Welzel, Corelight)

  * cirrus: Add an epoch to the ccache fingerprint, bump to 1000M (Arne Welzel, Corelight)

  * cirrus: Cap ccache max size to 500M and max files to 20000 (Arne Welzel, Corelight)

6.0.0-dev.117 | 2023-02-23 08:27:12 -0700

  * GH-2730: Use binary mode when opening files in Extract file analyzer (Tim Wojtulewicz)

6.0.0-dev.115 | 2023-02-23 07:40:22 -0700

  * Include stdint.h early in flex files to prevent redefinition of int type macros (Tim Wojtulewicz)

  * SMB: clarify a confusing line in one of the pac files (Tim Wojtulewicz)

  * module_util: Change streq helper method to just return bool (Tim Wojtulewicz)

  * NTLM analyzer: Don't bother checking boolean values for > 0 (Tim Wojtulewicz)

  * FTP analyzer: Remove unnecessary lines from constructor (Tim Wojtulewicz)

  * IP_Hdr: take length value as uint64_t instead of int (Tim Wojtulewicz)

  * EventTrace: take size_t as a constructor argument instead of int (Tim Wojtulewicz)

  * Change argument to Tempvar constructor to size_t, avoids conversion warning in template construction (Tim Wojtulewicz)

6.0.0-dev.104 | 2023-02-23 15:18:50 +0100

  * ci/collect-repo-info: No hard dependency on git, take two (Arne Welzel, Corelight)

    Collecting information about builtin external plugins also
    tried to use `git` unconditionally.

6.0.0-dev.100 | 2023-02-22 17:05:56 -0700

  * Use btest -A on CI to get more information about stuck tests (Tim Wojtulewicz, Corelight)

6.0.0-dev.98 | 2023-02-22 15:53:12 -0700

  * Use ensurepip to install pip on FreeBSD (Tim Wojtulewicz, Corelight)

    The py39-pip package is sometimes not available on FreeBSD, and using
    ensurepip is the recommended way to install pip according to the pip
    documentation. ensurepip should be available in every python
    installation.

6.0.0-dev.96 | 2023-02-22 14:26:29 -0700

  * Update benchmarker Cirrus secrets to use org-level values (Tim Wojtulewicz, Corelight)

6.0.0-dev.94 | 2023-02-22 19:29:52 +0100

  * ci/collect-repo-info: No hard dependency on git (Arne Welzel, Corelight)

    The OBS builds run in an environment where no git is installed and
    release tarballs assembled manually.

    Drop the hard-requirement on git from ci/collect-repo-info.json.

6.0.0-dev.92 | 2023-02-22 16:49:36 +0100

  * GH-2776: zeek-config: Move ZEEK_VERSION* out of zeek-config.h (Arne Welzel, Corelight)

    When bumping the VERSION file, everything that includes the auto-generated
    zeek/zeek-config.h needs to rebuild and ccache usage is voided due the file
    changing. Split the mutable version information into a new zeek-version.h
    file to avoid this.

    Further, do not include zeek-version.h within Plugin.h and avoid the reference
    to ZEEK_VERSION_FUNCTION unless we're building an external plugin.

    Closes #2776.

6.0.0-dev.89 | 2023-02-22 15:51:44 +0100

  * Update Spicy plugin. (Robin Sommer, Corelight)

6.0.0-dev.87 | 2023-02-22 12:58:05 +0100

  * external/testing-setup: Less telemetry.log volume (Arne Welzel, Corelight)

    For traces spanning a long time period, don't produce immense
    amounts of logs. Also filter down the event handler invocation
    entries to connection related events in telemetry.log.

  * EventHandler: Use telemetry framework for EventHandler.call_count (Arne Welzel, Corelight)

    Put the IntCounter into a std::optional rather than initializing
    it at EventHandler construction time as that will currently expose
    a time series per event handler through the Prometheus API.

6.0.0-dev.82 | 2023-02-21 16:48:03 -0700

  * parse.y: Improve error reporting of type redef (Arne Welzel, Corelight)

    It's happening regularly to me that I forget the type specifier when redef'ing
    records or enums and usually it takes me a while to figure out what's going
    on as the errors are not descriptive. Improve the error reporting and just
    bail as there's no sensible way to continue.

    Closes #2777

6.0.0-dev.79 | 2023-02-21 12:28:07 -0700

  * ftp: Limit user, password, arg and reply_msg column sizes in log (Arne Welzel, Corelight)

    The user and password fields are replicated to each of the ftp.log
    entries. Using a very large username (100s of KBs) allows to bloat
    the log without actually sending much traffic. Further, limit the
    arg and reply_msg columns to large, but not unbounded values.

  * Allow async DNS responses with zero TTLs to be considered valid after expriation (Tim Wojtulewicz, Corelight)

    This fixes a potential memory leak when getting responses for asnyc DNS requests
    where the TTL value on the response is zero. We were immediately considering the
    request as expired and never removing it from the map of requests. This lead to
    the DNS_Mgr eventually stopping processing of async requests.

  * Add max_size argument for find_all/find_all_ordered BIFs (Tim Wojtulewicz, Corelight)

6.0.0-dev.75 | 2023-02-19 19:23:50 -0700

  * Update seemingly-unrelated btests (Eldon Koyle, Corelight)

    For some reason, the plugin order appears to have changed in the files btests.

  * Use a default analyzer (Eldon Koyle, Corelight)

    Use a default analyzer instead of hardcoding a protocol number.

  * Simplify PBB analyzer by using Ethernet analyzer (Eldon Koyle, Corelight)

    After the first 4 bytes, this traffic actually just looks like Ethernet.
    Rather than try to re-implement the ethernet analyzer, just check the
    length, skip 4 bytes, and pass it on.

  * Add btest for PBB and update baselines (Eldon Koyle, Corelight)

  * Use constexpr instead of #define (Eldon Koyle, Corelight)

  * Cleanup and add customer MAC addresses (Eldon Koyle, Corelight)

     * Put c-dst/c-src in l2_dst/l2_src
     * use #define instead of const int and move to PBB.h

  * Add PBB (802.1ah) support (Eldon Koyle, Corelight)

  * Bump Spicy and spicy-plugin to latest releases. (Benjamin Bannier, Corelight)

    (cherry picked from commit f27a9a62648e642cc2f6aa267d067d529487eb21)

  * Trim diffed output in test `spicy.spicy-dump`. (Benjamin Bannier, Corelight)

    We previously would include any and all output from stderr during
    compilation in the test baseline. Depending on the used compiler this
    output may contain C++ compilation warnings which are uninteresting for
    the behavior under test.

    (cherry picked from commit 5221edf474e6a8cb05a1470318c7d8007857f202)

  * Update cmake and zeek-aux submodules [nomail] (Tim Wojtulewicz, Corelight)

6.0.0-dev.64 | 2023-02-17 17:36:40 +0100

  * TableVal: Propagate &on_change attribute through copy() (Arne Welzel, Corelight)

    Mostly for consistency with &default, &expire_func and other attributes
    being propagated through a copy(). Seems this was just missed during
    the implementation and/or was never tested for.

  * GH-2793: TableVal: Propagate &ordered through copy() (Arne Welzel, Corelight)

    Copying an &ordered table or set would result in a copy that is not ordered.
    This seems rather surprising behavior, so propagate the &ordered attribute.

  * GH-2690: Type/is_supported_index_type: Deal with recursive record types (Arne Welzel, Corelight)

    This plugs the issue reported in #2690, there might be more though.

6.0.0-dev.57 | 2023-02-16 15:10:33 -0700

  * Move get_relative_seq and get_segment_len to public static methods (Tim Wojtulewicz, Corelight)

  * CI: remove no longer needed workaround for GITHUB_ACTION env var in cluster tests (Christian Kreibich, Corelight)

    This got fixed in the testsuite via zeek/zeek-testing-cluster#24.

  * CI: directly invoke btest in the cluster testsuite (Christian Kreibich, Corelight)

    This resembles the way we also invoke it in ci/test.sh, and "-d"'s direct
    console output saves a roundtrip through uploaded artifacts when tests fail.
    This skips test retries for now -- not sure we really need it for this
    testsuite.

6.0.0-dev.52 | 2023-02-15 11:12:28 -0700

  * Fix linking of zeek_build_info on Windows (Tim Wojtulewicz)

  * CI: Enable Windows builds for PRs (Tim Wojtulewicz)

  * Call python explicitly from cmake for collecting repo info on Windows (Tim Wojtulewicz)

  * Rework zeek-inet-ntop snprintf return value handling (Tim Wojtulewicz)

6.0.0-dev.47 | 2023-02-15 19:10:35 +0100

  * zeek-setup: Ensure telemetry_mgr is created before other managers (Arne Welzel, Corelight)

    It's difficult to initialize metrics families in the constructor
    of other Managers if the telemetry_mgr isn't around yet.

6.0.0-dev.45 | 2023-02-15 13:30:29 +0100

  * added to C++ script compiler maintainer notes utility of starting with full base script compile (Vern Paxson, Corelight)

  * fixes for order-of-initialization in scripts compiled to C++ (Vern Paxson, Corelight)
    annotations of such initializations to tie them to the original Zeek script

6.0.0-dev.41 | 2023-02-14 13:18:11 -0800

  * Bump external zeek-testing commit hash to bring it up to master (Christian Kreibich, Corelight)

6.0.0-dev.40 | 2023-02-14 07:39:06 -0700

  * HashKey: Do not call Describe() unconditionally in DEBUG mode (Arne Welzel, Corelight)

    An unnecessary overhead of the Hash() method was uncovered for DEBUG builds
    due to computing a description of every HashKey() even when the DBG_HASHKEY
    stream is not enabled. Squelch it.

6.0.0-dev.38 | 2023-02-14 07:35:50 -0700

  * Call AnalyzerConfirmation earlier in VXLAN/Geneve analysis (Tim Wojtulewicz, Corelight)

  * Add validation of session to start of AYIYA/VXLAN/Geneve analysis (Tim Wojtulewicz, Corelight)

    This mimics how the Teredo analyzer is already doing it, including
    sending a weird if the session is invalid and bailing out if the
    protocol was already violated.

  * ci/collect-repo-info.py: Recognize git worktrees, too (Arne Welzel, Corelight)

    Benjamin reported ci/collect-repo-info.py failing for him in
    Git worktrees, use `git` to actually check if we're in a repo.

6.0.0-dev.33 | 2023-02-13 12:56:01 +0100

  * GH-1405: Add zeek -V/--build-info (Arne Welzel, Corelight)

    This adds a new utility called ci/collect-repo-info.py to produce a JSON
    document that is then baked into the Zeek executable file. Further, when
    creating a tarball via `make dist`, put a top-level repo-info.json file
    in place that is picked when no .git directory exists.

6.0.0-dev.30 | 2023-02-13 12:10:50 +0100

  * docker/Makefile: Remove stray quotes, unused DOCKER (Arne Welzel, Corelight)

  * cirrus/container: Polish, fix and automated pushing of images (Arne Welzel, Corelight)

    * Add ci/find-current-version.sh to figure out versions of lts and
      feature branch based on repository state.

    * Do not use ZEEK_IMAGE_REPO for the local images that
      are expected to exist.

    * Only use IMAGE_NAME/IMAGE_TAG instead of the ARCH and MANIFEST
      versions that contained the very same values

    * Support and push additional tags

      In addition to the lts and feature tags, also push the base version
      tags. 5.0 or 5.1 such that someone pulling zeek/zeek:5.0 will get
      the most recent available 5.0 patch release.

6.0.0-dev.27 | 2023-02-11 22:07:31 -0700

  * better error reporting when ZAM code calls a function (Vern Paxson, Corelight)

  * fix for ZAM compilation of "in" expressions (Vern Paxson, Corelight)

  * Update src/3rdparty submodule to get rid of some sprintf warnings (Tim Wojtulewicz, Corelight)

6.0.0-dev.22 | 2023-02-10 10:25:37 +0100

  * CI: Call 'brew install' for macOS to ensure we're using openssl@3 on all hosts (Tim Wojtulewicz, Corelight)

  * Remove files in build/src/3rdparty from coverage reports (Tim Wojtulewicz, Corelight)

6.0.0-dev.14 | 2023-02-07 10:29:28 -0700

  * Update binpac submodule for modernization fixes [nomail] (Tim Wojtulewicz, Corelight)

6.0.0-dev.11 | 2023-02-06 11:13:08 -0700

  * krb/smb2_krb_nokeytab: Register get_file_handle() to avoid warnings (Arne Welzel, Corelight)

    Now that the common event handler logs a warning, ensure there's one
    in place, even if it's just returning stub data.

  * fuzzer-setup: Do not always generate new_event (Arne Welzel, Corelight)

    new_event should never be used on production systems, so don't turn
    it on for fuzzing either as it showed up as bottlenecks in flamegraphs.

  * EventHandler: Support unsetting generate_always (Arne Welzel, Corelight)

  * files/main: No empty file_ids (Arne Welzel, Corelight)

    When an analyzer calls DataIn(), there's a costly callback construct
    going through the event queue. If an analyzer does not have a
    get_file_handle() handler installed, the produced file_id would
    end up empty and ignored. Consequently, the get_file_handle() callback
    was invoked for every new DataIn() invocations.

    This is surprising and costly. Log a warning when this happens and
    instead set a generically generated file handle value instead to
    prevent the repeated get_file_handle() invocations.

6.0.0-dev.6 | 2023-02-06 10:33:24 -0700

  * CI: Fix the ThreadSanitizer build (Tim Wojtulewicz, Corelight)

6.0.0-dev.3 | 2023-02-03 10:50:24 -0700

  * Start of 6.0.0 development (Tim Wojtulewicz, Corelight)

5.2.0-dev.637 | 2023-02-03 09:05:15 -0700

  * Bump spicy-plugin to latest development snapshot. (Benjamin Bannier, Corelight)

5.2.0-dev.635 | 2023-02-03 16:48:18 +0100

  * Add BIF `have_spicy_analyzers()`. (Robin Sommer, Corelight)

    We previously used the Spicy plugin's `Spicy::available` to test for
    Spicy support. However, having Spicy support does not necessarily mean that we
    have built Zeek with its in-tree Spicy analyzers: the Spicy plugin
    could have been pulled in from external. The new BIF now reliably
    tells us whether the Spicy analyzers are available; its result
    corresponds to what `zeek-config --have-spicy-analyzers` returns as
    well.

    We also move the two current checks over to use this BIF.

    (Note: I refrained from renaming the CMake-side `USE_SPICY_ANALYERS`
    to `HAVE_SPICY_ANALYZERS`. We should do this eventually for
    consistency, but I didn't want to make more changes than necessary
    right now.)

  * Update package-manager submodule. (Tim Wojtulewicz, Corelight)

5.2.0-dev.632 | 2023-02-02 20:20:27 +0100

  * btest/decode_base_errors: Avoid binary output in baseline (Arne Welzel, Corelight)

5.2.0-dev.630 | 2023-02-02 18:35:21 +0100

  * cirrus: No container images builds or cluster testing on branches (Arne Welzel, Corelight)

  * Bump private testsuite for char handling fixes (Arne Welzel, Corelight)

  * GH-2742: Base64: report byte as positive integer (Arne Welzel, Corelight)

    A baseline difference between arm64 and x86 showed up. We would
    print a wrong character as negative value on x86 due to chars
    being signed by default. Force an unsigned interpretation which
    is also more reasonable because we'd have never indexed the
    base64 table with -112

        -XXXXXXXXXX.XXXXXX      XXXXXXXXXXX     131.243.99.154  3288 193.159.183.138 80      base64_illegal_encoding character -112 ignored by Base64 decoding       F       zeek    -
        +XXXXXXXXXX.XXXXXX      XXXXXXXXXXX     131.243.99.154  3288 193.159.183.138 80      base64_illegal_encoding character 144 ignored by Base64 decoding        F       zeek    -

    Fixes more of #2742

  * GH-2742: netbios_decode: use unsigned char for result (Arne Welzel, Corelight)

    The buf[i] < 3 condition in use previously allowed all chars (signed on x86)
    through that had the 0x80 high-bit set after reconstructing from the two
    bytes of the netbios name, resulting in escaped non-ascii content in
    the logs.

    Fixes more of #2742

  * GH-2742: analyzer/http: Do not assume char is signed (Arne Welzel, Corelight)

    On aarch64, char is unsigned, so is_HTTP_token_char() allowed
    non-ASCII stuff with the high-bit set.

    Fixes part of #2742

  * cirrus: Run tests in a Debian 11 container, too (Arne Welzel, Corelight)

5.2.0-dev.620 | 2023-02-02 12:52:40 +0100

  * cirrus: There's no zeekurity repository on AWS ECR (Arne Welzel, Corelight)

5.2.0-dev.619 | 2023-02-02 11:25:38 +0100

  * cirrus: Also push images to the zeekurity account (Arne Welzel, Corelight)

    This was mistakenly removed during f9b0681c98de3dd68f05bd1a8f36fcf8a6ca9068.

5.2.0-dev.617 | 2023-02-01 16:38:41 -0800

  * Bump cluster testsuite to pull in fix for test baseline race condition (Christian Kreibich, Corelight)

5.2.0-dev.615 | 2023-02-01 16:05:42 -0800

  * Update btest submodule to v1.0 (Christian Kreibich, Corelight)

5.2.0-dev.614 | 2023-02-01 15:51:15 -0800

  * Tighten the scripts.base.frameworks.logging.hooks test (Christian Kreibich, Corelight)

  * Fix two btest-diff checks that couldn't fail (Christian Kreibich, Corelight)

5.2.0-dev.611 | 2023-02-01 15:34:22 -0700

  * Update broker submodule [nomail] (Tim Wojtulewicz, Corelight)

5.2.0-dev.609 | 2023-02-01 20:55:45 +0100

  * docker: Extend tests to run a very basic Zeek script (Arne Welzel, Corelight)

    Mostly because a `-v` shortcuts quite, while a zeek -e 'print zeek_version()'
    will also parse all base scripts, etc.

  * cirrus: No broker-tests and no cpp-tests for images (Arne Welzel, Corelight)

  * docker: Update Makefile/README to align with the Cirrus CI setup (Arne Welzel, Corelight)

  * cirrus: Add back testing of container images (Arne Welzel, Corelight)

5.2.0-dev.604 | 2023-02-01 10:48:03 -0700

  * Restore/rename field in SMB2::Fscontrol record type (Tim Wojtulewicz, Corelight)

    b41a4bf06dc2f3a4218cfa0a95b536ec9d253917 removed a field from this record
    because it had a duplicate name as another field. The field does need to
    exist, but it needs the correct name.

5.2.0-dev.602 | 2023-02-01 10:47:41 -0700

  * testing/http: http-11-request-then-cruft (Arne Welzel, Corelight)

    A client sends a "proper" HTTP/1.1 request and afterwards a few T /\n\n sequences.
    The latter ones aren't logged.

  * testing/http: Add pcap extracted from m5-long external test-suite (Arne Welzel, Corelight)

    This tests that the HTTP version is now updated if it changes in the
    course of a connection.

  * testing/external: m57-long baseline update (Arne Welzel, Corelight)

    There's a HTTP server that first replies with HTTP/1.1, then HTTP/1.0.
    Seems actually nicer to have the real value within the log/event.

  * analyzers/http: Update request_version on subsequent SetVersion() calls (Arne Welzel, Corelight)

    The #124 PR introduced special treatment when HTTP version 0.9
    was set. With #127, a reproducer that set HTTP/1.0 in the first
    request was created and subsequent requests wouldn't reset to
    HTTP version 0.9.

    This is subtle, but doesn't seem like things fall apart.

    Improves runtime from 20 seconds to 2 seconds for the given
    reproducer.

5.2.0-dev.597 | 2023-02-01 10:47:21 -0700

  * testing/ftp: Add tests and pcaps with invalid reply lines (Arne Welzel, Corelight)

    These have been created artificially. The tests show that for an
    invalid reply line without a numeric code, with a numeric code < 100
    or a numeric code not followed by a space we now raise an analyzer
    violation and disable the analyzer.

  * ftp: Harden reply handing a bit and don't raise bad replies to script-land (Arne Welzel, Corelight)

    This improves runtime of the oss-fuzz generated traffic in #125.
    Specifically, that reproducers included a 064- reply code that was
    interpreted as needing to be continued.

    Also, return after AnalyzerViolations() for server replies rather than
    propagating bad replies them to script-land. This trusts server's to
    generally behave according to specification.

  * ftp: ignore invalid commands (Arne Welzel, Corelight)

    Do not propagate wrong FTP commands to script land.

5.2.0-dev.593 | 2023-02-01 11:39:15 +0100

  * Provide infrastructure to migrate legacy analyzers to in-tree
    Spicy analyzers. As initial (trivial) examples, we port the Syslog
    and Finger analyzers over to Spicy. The only user-visible change
    is that we now detect Finger as service in `conn.log`. (Robin
    Sommer, Corelight)

  * GH-2675: Switch to building shared Spicy toolchain libraries by
    default. Furthermore, Zeek itself no longer link against the
    toolchain libraries itself because the Spicy plugin has removed
    JIT support. But the other Spicy tools continue to do so, and
    shrink in size now. (Robin Sommer, Corelight)

  * Extend Zeekygen manager with method to check if it's active. (Robin Sommer, Corelight)

5.2.0-dev.589 | 2023-02-01 09:02:00 +0100

  * cirrus/containers: Do not install btest pcaps (Arne Welzel, Corelight)

  * cirrus: Replace curl with _cache (Arne Welzel, Corelight)

  * cirrus: Add cluster testing (Arne Welzel, Corelight)

  * cirrus: Polish container_image_manifest_docker_builder (Arne Welzel, Corelight)

  * cirrus: Use ccache for faster container builds (Arne Welzel, Corelight)

  * cirrus: Add docker_builder tasks to build and push images (Arne Welzel, Corelight)

  * github: No more docker workflow (Arne Welzel, Corelight)

5.2.0-dev.579 | 2023-01-31 14:49:29 +0100

  * mysql: Recognize when client/server negotiate SSL (Arne Welzel, Corelight)

    This instantiates the SSL analyzer when the client requests SSL
    so that Zeek now has a bit more visibility into encrypted MySQL
    connections.

    The pattern used is the same as in the IMAP, POP or XMPP analyzer.

  * analyzer: Move disabling_analyzer() hook into Analyzer module (Arne Welzel, Corelight)

    When disabling_analyzer() was introduced, it was added to the GLOBAL
    module. The awkward side-effect is that implementing a hook handler
    in another module requires to prefix it with GLOBAL. Alternatively, one
    can re-open the GLOBAL module and implement the handler in that scope.

    Both are not great, and prefixing with GLOBAL is ugly, so move the
    identifier to the Analyzer module and ask users to prefix with Analyzer.

  * Skip some error reporting when the record type has errors (Arne Welzel, Corelight)

5.2.0-dev.571 | 2023-01-30 13:39:01 -0700

  * Remove setting osx-sysroot on macOS CI builds (Tim Wojtulewicz, Corelight)

5.2.0-dev.569 | 2023-01-30 09:19:23 -0700

  * Whitespace cleanup in 5.2 NEWS block (Tim Wojtulewicz, Corelight)

  * Updates to the NEWS for upcoming release/5.2 branching (Tim Wojtulewicz, Corelight)

5.2.0-dev.566 | 2023-01-30 11:05:40 +0100

  * GH-2698: broker/store: Extend SQLiteOptions around data safety and performance (Arne Welzel, Corelight)

    Add configurability of synchronous and journal_mode for SQLite backed
    Broker data stores. Setting these to synchronous=normal and journal_mode=wal
    can significantly improve throughput at the cost of some durability in
    the presence of power loss or OS crash. In the context of Zeek, this is
    likely more than acceptable.

    Additionally, add integrity_check and failure_mode options to support deleting
    and re-opening a corrupted SQLite database at store creation.

5.2.0-dev.563 | 2023-01-28 09:46:59 -0700

  * Enable Debian 10 builds for PRs (Tim Wojtulewicz, Corelight)

5.2.0-dev.561 | 2023-01-28 08:52:32 -0700

  * Update smb1-OSS-fuzz-54883.pcap to have correct snaplen (Tim Wojtulewicz, Corelight)

  * ci-notification: No notifications for "skipped" runs [skip ci] (Arne Welzel, Corelight)

    It is not useful to receive email notifications for CI runs
    that were skipped.

5.2.0-dev.557 | 2023-01-27 13:24:58 -0700

  * smb1: Prevent accessing uninitialized referenced_tree (Arne Welzel, Corelight)

    The added pcap was created from an OSS Fuzz test case and is borderline
    valid SMB traffic, but it triggered a scripting error.

    Closes #2726

5.2.0-dev.553 | 2023-01-27 13:23:09 -0700

  * Add tooling section to README (Tim Wojtulewicz, Corelight)

  * A bunch of other minor issues reported by PVS-Studio (Tim Wojtulewicz, Corelight)

  * Simplify setup of IPv6 DNS resolvers (Tim Wojtulewicz, Corelight)

  * Add missing assignment operators to RobustDictIterator (Tim Wojtulewicz, Corelight)

  * Fix some member-variable shadowing issues (Tim Wojtulewicz, Corelight)

  * Fix/simplify some if statement comparisons (Tim Wojtulewicz, Corelight)

  * Fix a few minor int pointer value initializations (Tim Wojtulewicz, Corelight)

  * Fix a bunch of missing class member initializations (Tim Wojtulewicz, Corelight)

  * Remove unnecessary virtual specifiers from final classes (Tim Wojtulewicz, Corelight)

5.2.0-dev.543 | 2023-01-27 12:08:28 -0700

  * Add a few extra length checks to Teredo::DetectProtocol (Tim Wojtulewicz, Corelight)

5.2.0-dev.541 | 2023-01-27 18:44:43 +0100

  * github: cat the check_suite payload (Arne Welzel, Corelight)

  * github: Send emails for success conclusion, too (Arne Welzel, Corelight)

5.2.0-dev.538 | 2023-01-27 11:24:35 +0100

  * testing/mysql: Add traces recorded with a free-tier MySQL instance (Arne Welzel, Corelight)

  * MySQL: Fix endianness, introduce mysql_eof() event (Arne Welzel, Corelight)

    We were parsing MySQL using bigendian even though the protocol is
    specified as with "least significant byte first" [1]. This is most
    problematic when parsing length encoded strings with 2 byte length
    fields...

    Further, I think, the EOF_Packet parsing was borked, either due to
    testing the CLIENT_DEPRECATE_EOF with the wrong endianness, or due to
    the workaround in Resultset processing raising mysql_ok(). Introduce a
    new mysql_eof() that triggers for EOF_Packet's and remove the fake
    mysql_ok() Resultset invocation to fix. Adapt the mysql script and tests
    to account for the new event.

    This is a quite backwards incompatible change on the event level, but
    due to being quite buggy in general, doubt this matters to many.

    [1] https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_dt_integers.html

5.2.0-dev.534 | 2023-01-26 20:06:24 +0100

  * Tame error reporting and abort() for undefined types (Arne Welzel, Corelight)

5.2.0-dev.532 | 2023-01-26 10:55:02 -0700

  * Bump spicy to v1.6.1. (Benjamin Bannier, Corelight)

5.2.0-dev.530 | 2023-01-26 09:44:36 -0700

  * Update external testing repo commit hashes for Teredo changes (Tim Wojtulewicz, Corelight)

  * Update scripts.base.frameworks.analyzer.logging btest to use a different trace file (Tim Wojtulewicz, Corelight)

  * Update btest baselines due to reduced calls to UID::Set (Tim Wojtulewicz, Corelight)

    UID::Set is called by AnalyzerViolation, and since Teredo isn't calling nearly
    as many of those anymore the UIDs used by other calls changed as well.

  * Rework TeredoAnalzyer::DetectProtocol to not catch as many packets (Tim Wojtulewicz, Corelight)

5.2.0-dev.525 | 2023-01-26 09:11:45 -0700

  * GH-2669: Give script location for type clash in for-loop construction (Tim Wojtulewicz, Corelight)

5.2.0-dev.523 | 2023-01-26 14:37:35 +0100

  * cirrus: Introduce SKIP_TASK_ON_PR (Arne Welzel, Corelight)

    Limit the number of tasks we run on PRs to only a subset instead of
    the full set to reduce the amount of resources we use for PRs. Labeling
    a PR with "fullci" can be used enable running all tasks.

    This removes a total of 14 task execution from PRs by default at the cost of
    needing to monitor master builds more closely for distro specific breakage.

5.2.0-dev.521 | 2023-01-25 14:11:03 -0700

  * removed -O add-C++ option and updated documentation (Vern Paxson, Corelight)

5.2.0-dev.519 | 2023-01-25 11:12:15 -0700

  * GH-2684: Stop violating VXLAN for forwarding failures (Tim Wojtulewicz, Corelight)

5.2.0-dev.515 | 2023-01-25 08:19:02 -0700

  * Fix CI benchmark script to properly urlencode arguments (Tim Wojtulewicz, Corelight)

5.2.0-dev.513 | 2023-01-24 09:01:42 -0700

  * Add test for new handling of unknown RDP keyboards (Tim Wojtulewicz, Corelight)

5.2.0-dev.510 | 2023-01-23 14:35:14 -0700

  * Maintenance updates for -O gen-C++ / -O gen-standalone-C++ (Vern Paxson, Corelight)
      fixes for using BiFs in standalone global initializations
      avoiding redundant global initializations
      updates to maintenance scripts and notes
      removal of an unused member variable

5.2.0-dev.508 | 2023-01-23 12:48:06 -0700

  * Log raw keyboard value on best guess (jeff-bb)

  * Avoid excessive fmt calls, return default behavior on unknown (jeff-bb)

    Using "in" to query the language const. This also handles the case of not
    having a best guess and continue using the existing behavior.

    Given
    keyboard_layout = 1033 (0x0409), "keyboard-English - United States"
    keyboard_layout = 66569 (0x00010409), "keyboard-English - United States (Best Guess)"
    keyboard_layout = 12345 (0x3039), "keyboard-12345"

  * "Best Guess" unknown keyboard / language variants (jeff-bb)

    If the lookup table does not have an entry, it will just log as the raw
    decimal language/keyboard code. With this change, if we do not have an entry
    in the lookup table, we'll look at the low order / 4 least significant bits
    to see if we have a match. The high order / 4 most significant bits are
    flags/modifiers to the base language/keyboard code. We'll append that it is
    a "Best Guess"

5.2.0-dev.504 | 2023-01-23 09:37:47 -0700

  * Update RDP Keyboard Languages (jeff-bb)

    Using additional sources to add more languages / locales. Changed sort order to match other sections.

5.2.0-dev.499 | 2023-01-19 09:14:07 -0700

  * Reduce startup time on Windows by using std::filesystem::canonical (Tim Wojtulewicz)

    realpath() apparently doesn't handle Windows symlinks very well. This
    causes plugin::Manager and ScannedFile to rescan a bunch of extra
    paths that they should be skipping. This commit reduces the startup
    time on Windows by 3-4 seconds (~8.5s to ~5s).

  * Fix diff-remove-abspath to ignore Windows drive letters (Tim Wojtulewicz)

  * Re-enable the float/double versions of hton{d,f} on Windows (Tim Wojtulewicz)

  * Make sure the DEBUG flag is defined when building in debug mode (Tim Wojtulewicz)

    MSVC uses _DEBUG, unlike all of the other compilers, so we aren't
    properly enabling/disabling varying bits of code that are dependent
    on DEBUG being defined.

  * Don't close the kqueue manually on shutdown on Windows (Tim Wojtulewicz)

  * Fix MSVC compiler warning due to a type mismatch (Tim Wojtulewicz)

  * GH-2565: Update libunistd submodule to fix builds on VS2022 (Tim Wojtulewicz)

  * GH-2641: Lazy-load syslog to avoid creating bro.log on Windows (Tim Wojtulewicz)

  * Use newer diff on Alpine for --strip-trailing-ci (Tim Wojtulewicz, Corelight)

  * Update btest submodule to version that supports Windows (Tim Wojtulewicz)

  * Use pathsep btest value in btest.cfg (Tim Wojtulewicz)

  * Use build_dir btest value in btest.cfg (Tim Wojtulewicz)

  * Fix zeek-path-dev.sh creation to better support Windows (Tim Wojtulewicz)

  * Remove outdated distributions from CI. (Johanna Amann, Corelight)

    Fedora 35: CI folder was forgotten during removal

    Opensuse Leap 15.3: EOL was 2022

    Openssl-3: No longer necessary, since part of distributions (e.g. Ubuntu
    22.10)

5.2.0-dev.483 | 2023-01-18 17:12:57 +0100

  * Fix ZAM memory leak as reported in #2634 (Vern Paxson, Corelight)

  * bifs/parse_eftp: Prevent reporter warnings/errors on invalid input (Arne Welzel, Corelight)

    When passing invalid IPs or an out-of range port to parse_eftp()
    a warning or error was generated on stderr (in addition to setting
    the $valid field to F). Prevent the output by adding safe-guarding
    and using IPAddr::ConvertString() instead.

5.2.0-dev.478 | 2023-01-12 09:41:11 +0100

  * fix for crashes when record definitions repeat a field name (Vern Paxson, Corelight)

5.2.0-dev.476 | 2023-01-11 17:00:32 -0800

  * CI updates (Christian Kreibich, Corelight)

    - remove Fedora 35, now EOL
    - avoid deprecated ::set-output use
    - move action/checkout use to v3

  * Bump zeek-archiver, zeek-af_packet-plugin, and doc submodules (Christian Kreibich, Corelight)

  * Fixes to support the Npcap library on Windows (Tim Wojtulewicz, Corelight)

  * GH-2645: fix for crash when specifying an unwriteable file to --profile-scripts (GH-2645) (Vern Paxson, Corelight)

  * Fix a docstring typo (Christian Kreibich, Corelight)

5.2.0-dev.464 | 2023-01-10 10:25:28 +0100

  * Expand docstrings of file_new and file_over_new_connection events (Christian Kreibich, Corelight)

  * GH-2031: analyzer: Add analyzer.log for logging violations/confirmations (Arne Welzel, Corelight)

    By default this only logs all the violations, regardless of the
    confirmation state (for which there's still dpd.log). It includes
    packet, protocol and file analyzers.

    This uses options, change handlers and event groups for toggling
    the functionality at runtime.

  * GH-2647: intel/seen/file-names: Use file_over_new_connection() (Arne Welzel, Corelight)

    The seen/file-names script relies on f$info$filename to be populated.
    For HTTP and other network protocols, however, this field is only
    populated during file_over_new_connection() that's running after
    file_new().

    Use the file_new() event only for files without connections and
    file_over_new_connection() implies that f$conns is populated, anyway.

    Special case SMB to avoid finding files twice, because there's a
    custom implementation in seen/smb-filenames.zeek.

  * scripts/analyzer: Introduce Analyzer::requested_analyzers (Arne Welzel, Corelight)

    In certain deployment scenarios, all analyzers are disabled by default.
    However, conditionally/optionally loaded scripts may rely on analyzers
    functioning and declare a request for them.

    Add a global set set to the Analyzer module where external scripts can record
    their requirement/request for a certain analyzer. Analyzers found in this
    set are enabled at zeek_init() time.

5.2.0-dev.453 | 2023-01-05 09:53:47 -0700

  * Add error and abort if fuzzer requested is not found (Tim Wojtulewicz, Corelight)

  * Add a bunch of new fuzzers and corpus data (Tim Wojtulewicz, Corelight)

5.2.0-dev.450 | 2023-01-05 12:12:13 +0100

  * EventRegistry/Func: Disable events when all bodies are disabled (Arne Welzel, Corelight)

    Add state to Func for tracking if any enabled bodies exist which
    allows us to propagate it up to the EventHandler::bool() operator.
    In turn, when all bodies of an event's Func have been runtime disabled,
    the event itself will not be invoked anymore.

5.2.0-dev.447 | 2023-01-03 15:44:39 -0700

  * Fix sprintf warnings on macOS (Tim Wojtulewicz, Corelight)

    The most recent compiler update for macOS marked sprintf as
    deprecated, so we started getting warnings from all of the places
    that use it.

5.2.0-dev.445 | 2023-01-03 12:10:20 -0700

  * Update COPYING to 2023 (Tim Wojtulewicz, Corelight)

  * Rename COPYING.3rdparty to COPYING-3rdparty (Tim Wojtulewicz, Corelight)

    This keeps GitHub from being confused about this file being part of
    the repository's license, and thus not displaying the proper linkage
    to our actual COPYING file.

5.2.0-dev.442 | 2023-01-03 12:09:10 -0700

  * CI: avoid doing Windows builds with every push (Tim Wojtulewicz, Corelight)

  * CI: drop macOS Big Sur, add macOS Ventura (Tim Wojtulewicz, Corelight)

5.2.0-dev.437 | 2022-12-16 08:16:30 -0700

  * GH-2637: Re-remove the bro symlink wrapper (Tim Wojtulewicz, Corelight)

5.2.0-dev.435 | 2022-12-13 21:51:24 -0800

  * Docker Hub: Push images to zeek/zeek and zeek/zeek-dev (Arne Welzel, Corelight)

5.2.0-dev.432 | 2022-12-13 10:25:21 -0700

  * Bump Spicy and spicy-plugin to latest releases. (Benjamin Bannier, Corelight)

    This bumps Spicy to 1.6.0 and spicy-plugin to 1.4.2.

5.2.0-dev.430 | 2022-12-13 11:50:47 +0100

  * Func: Use std::stable_sort() for sorting bodies by priority (Arne Welzel, Corelight)

5.2.0-dev.427 | 2022-12-13 11:37:47 +0100

  * logging: Add event_groups to Stream (Arne Welzel, Corelight)

    This commit adds an optional event_groups field to the Logging::Stream record
    to associated event groups with logging streams.

    This can be used to disable all event groups of a logging stream when it is
    disabled. It does require making an explicit connection between the
    logging stream and the involved groups, however.

  * zeek.bif: Add has_event_group() / has_module_events() (Arne Welzel, Corelight)

    Introduce helpers to determine if a given attribute or module event
    group exists given a string.

5.2.0-dev.422 | 2022-12-09 16:07:47 +0100

  * maintenance updates for -O C++ (Vern Paxson, Corelight)

  * maintenance updates for -O ZAM (Vern Paxson, Corelight)

5.2.0-dev.418 | 2022-12-08 10:28:01 +0100

  * dfa: Remove transition_counter (Arne Welzel, Corelight)

5.2.0-dev.414 | 2022-12-06 12:21:44 +0100

  * analyzer/files: handle non-analyzer names in describe_file() (Arne Welzel, Corelight)

    When a fa_file object is created through the use of Input::add_analysis(),
    the fa_file's source is likely not valid representation of an analyzer's
    tag and a Files::describe() should not error and instead return an empty
    description.

  * frameworks/notice: Handle fa_file with no or more than a single connection better (Arne Welzel, Corelight)

    * When a file is transferred over multiple connection, have
      create_file_info() just pick the first one instead of none.

    * Do not unconditionally assume cid and cuid as set on a
      Notice::FileInfo object.

5.2.0-dev.411 | 2022-12-06 12:15:42 +0100

  * GH-2547: ContentLine: Fix spelling of "suppress", deprecate SupressWeirds() (Arne Welzel, Corelight)

  * GH-2577: zeekygen/normalize_script_path: Special case plugin dirnames without _ (Arne Welzel, Corelight)

    In normal installations, a plugin's basename has an underscore in it
    to separate the namespace from the plugin name. E.g Zeek_Spicy. When
    there is no underscore, this is most likely due to ./build being
    picked up when using ZEEK_PLUGIN_PATH. The basename ends-up "build"
    and is susceptible to collisions.

    Prepend one parent directory as a heuristic to make this scenario less
    likely, assuming ./build is usually below a repository checkout that
    uniquely identifies the plugin.

5.2.0-dev.407 | 2022-12-06 10:01:45 +0100

  * Added several events for detailed info on the SSH2 key init directions (Joost Jansen)

5.2.0-dev.404 | 2022-12-05 10:33:53 +0100

  * Expose PA_ENC_TIMESTAMP to script land (nadavkluger)

  * Update btest submodule [nomail] [skip ci] (Christian Kreibich, Corelight)

5.2.0-dev.401 | 2022-12-04 18:36:56 -0800

  * ID: Add AddAttr() helper (Arne Welzel, Corelight)

  * GH-2628: Attr: Duplicated &is_used is allowed (Arne Welzel, Corelight)

5.2.0-dev.398 | 2022-12-02 18:39:01 -0800

  * parse.y: Allow trailing commas for table, set, vector and record construction (Arne Welzel, Corelight)

  * ci/alpine: Use musl-fts-dev rather than fts-dev (Arne Welzel, Corelight)

5.2.0-dev.393 | 2022-11-30 09:42:00 -0800

  * GH-846: Expand Conn::Info$duration comment to clarify TCP end-of-connection handling (Christian Kreibich, Corelight)

5.2.0-dev.391 | 2022-11-30 15:37:57 +0000

  * script optimization fixes: (Vern Paxson, Corelight)
     new initialization model for standalone C++ scripts
     type coercion fix
     ZAM fix for compiling using C++ optimizer
     disambiguate empty constructors

  * TimerMgr: Add back max_timer_expires=0 special case (Arne Welzel, Corelight)

    Fixes #2514.

  * Add btest for expiration of all pending timers. (Jan Grashoefer, Corelight)

  * input: Add location info for invalid enums (Arne Welzel, Corelight)

    Fixes #2613

  * mqtt: Move from policy/ into base/ (Arne Welzel, Corelight)

    Register dpd signatures and the analyzer when running in default mode.

    Closes #2583

  * Expr: Warn on count underflow for c -= 1 and c = c - 1 (Arne Welzel, Corelight)

    Closes #2486

  * Reporter: Add ExprRuntimeWarning() (Arne Welzel, Corelight)

5.2.0-dev.376 | 2022-11-29 14:53:41 -0800

  * smb: Drop references to uid_map in state. (Arne Welzel, Corelight)

  * smb: Drop AUTH_LOG (Arne Welzel, Corelight)

5.2.0-dev.373 | 2022-11-29 12:12:34 -0800

  * CI: add Fedora 37 (Christian Kreibich, Corelight)

5.2.0-dev.371 | 2022-11-28 17:37:28 -0800

  * provide deprecated version of merge_type_list() (Vern Paxson, Corelight)

  * keep merge_types() externally available; address reviewing suggestion (Vern Paxson, Corelight)

  * change vector constructors to require direct type equivalence for non-arithmetics (Vern Paxson, Corelight)

5.2.0-dev.365 | 2022-11-22 21:15:42 -0800

  * Type coercion fix: transform +=/-= operators with arithmetic targets to explict assignments (Vern Paxson, Corelight)

5.2.0-dev.363 | 2022-11-22 15:49:10 -0800

  * zeekygen.bif: Drop using namespace zeekygen (Arne Welzel, Corelight)

  * zeekygen: Add BIFs to access declaring scripts (Arne Welzel, Corelight)

  * Update broker submodule [nomail] (Tim Wojtulewicz, Corelight)

5.2.0-dev.359 | 2022-11-22 12:27:54 -0700

  * Add additional length checking to IPv6::ToVal (Tim Wojtulewicz, Corelight)

5.2.0-dev.357 | 2022-11-22 12:27:26 -0700

  * ftp: Introduce FTP::max_command_length (Arne Welzel, Corelight)

    oss-fuzz produced FTP traffic with a ~550KB long FTP command. Cap FTP command
    length at 100 bytes, log a weird if a command is larger than that and move
    on to the next. Likely it's not actual FTP traffic, but raising an
    analyzer violation would allow clients an easy way to disable the analyzer
    by sending an overly long command.

    The added test PCAP was generated using a fake Python socket server/client.

5.2.0-dev.355 | 2022-11-22 12:26:50 -0700

  * http: Heuristic around rejecting malformed HTTP/0.9 traffic (Arne Welzel, Corelight)

    oss-fuzz generated "HTTP traffic" containing 250k+ sequences of "T<space>\r\r"
    which Zeek then logged as individual HTTP requests. Add a heuristic to bail
    on such request lines. It's a bit specific to the test case, but should work.

5.2.0-dev.352 | 2022-11-21 09:10:28 -0700

  * Fix logic error in vfmt() when growing the buffer (Michael Dopheide)

5.2.0-dev.350 | 2022-11-21 09:09:09 -0700

  * Added NTLM challenge and response (nadavkluger)

5.2.0-dev.347 | 2022-11-18 09:41:44 -0700

  * docker/ecr: Push into zeek/zeek, not zeekurity/zeek (Arne Welzel, Corelight)

5.2.0-dev.345 | 2022-11-18 09:41:08 -0700

  * Fix warning in broker.web-socket-events btest (Dominik Charousset, Corelight)

5.2.0-dev.343 | 2022-11-18 09:40:51 -0700

  * Fix the strcasestr unit test (Tim Wojtulewicz)

5.2.0-dev.341 | 2022-11-18 09:40:30 -0700

  * spelling: repetitions (Josh Soref)

5.2.0-dev.339 | 2022-11-18 11:49:47 +0100

  * GH-2426: Fixing productive connections with missing SYN still considered partial after flipping direction. (Robin Sommer, Corelight)

    In https://github.com/zeek/zeek/pull/2191, we added endpoint flipping
    for cases where a connection starts with a SYN/ACK followed by ACK or
    data. The goal was to treat the connection as productive and go ahead
    and parse it. But the TCP analyzer could continue to consider it
    partial after flipping, meaning that app layers would bail out. #2426
    shows such a case: HTTP gets correctly activated after flipping
    through content inspection, but it won't process anything because
    `IsPartial()` returns true. As the is-partial state reflects
    whether we saw the first packets each in direction, this patch now
    overrides that state for the originally missing SYN after flipping.

    We actually had the same problem at a couple of other locations already
    as well. One of that only happened to work because of the originally
    inconsistent state flipping that was fixed in the previous commit. The
    corresponding unit test now broke after that change. This commit
    updates that logic as well to override the state.

5.2.0-dev.336 | 2022-11-17 16:28:40 -0800

  * Publish container images to ECR in addition to docker.io. (Benjamin Bannier, Corelight)

5.2.0-dev.334 | 2022-11-17 11:34:41 -0700

  * Make dependency of `zeek` on spicy-plugin's driver object file explicit. (Benjamin Bannier, Corelight)

5.2.0-dev.332 | 2022-11-17 11:31:23 -0700

  * Spelling NEWS & CMakeLists.txt (Josh Soref)

5.2.0-dev.330 | 2022-11-17 11:30:19 -0700

  * Spelling testing (Josh Soref)

5.2.0-dev.327 | 2022-11-16 14:55:02 -0700

  * spelling: organization (Josh Soref)

  * spelling: invalidate (Josh Soref)

5.2.0-dev.324 | 2022-11-16 14:53:35 -0700

  * spelling: github (Josh Soref)

5.2.0-dev.322 | 2022-11-16 14:51:35 -0700

  * smb1: Ensure existence of dialect_index in offered dialects (Arne Welzel, Corelight)

5.2.0-dev.320 | 2022-11-16 14:50:58 -0700

  * ssh: Test for c$ssh$analyzer_id existence (Arne Welzel, Corelight)

5.2.0-dev.317 | 2022-11-15 11:34:47 -0700

  * init-bare: Fix zeek:see after bif renaming (Arne Welzel, Corelight)

5.2.0-dev.315 | 2022-11-14 13:19:33 -0700

  * Update broker submodule [nomail] (Tim Wojtulewicz, Corelight)

5.2.0-dev.313 | 2022-11-14 09:41:06 -0700

  * Re-enable strcasestr on Windows, fix linking error (Tim Wojtulewicz)

  * Minor renaming changes to event handler stats bif, plus a test (Tim Wojtulewicz, Corelight)

  * Remove unneeded forward-declaration of select() (Tim Wojtulewicz, Corelight)

  * Rework setting adding ports to map slightly (Tim Wojtulewicz, Corelight)

5.2.0-dev.307 | 2022-11-11 15:13:47 -0700

  * Merged support for Microsoft Windows (Tomer Lev, Elad Solomon, Microsoft)

5.2.0-dev.234 | 2022-11-11 12:48:20 -0700

  * Spelling src (Josh Soref)

    Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

5.2.0-dev.232 | 2022-11-09 18:23:11 -0700

  * Script optimization maintenance and updates: (Vern Paxson, Corelight)
      maintenance fixes for variadic run-time checks, '_' placeholder identifier
      "-O allow-cond" permits compiling scripts to C++ when influenced by @if conditionals
      more robust standalone compile-to-C++ properties
      fix for nested "when" statements
      test suite updates

  * spelling: Fixes in pac files and functional changes in src directory (Josh Soref)

    Signed-off-by: Josh Soref <2119212+jsoref@users.noreply.github.com>

5.2.0-dev.215 | 2022-11-09 11:03:43 -0700

  * Configure script: drop --with-caf, add -D option (Dominik Charousset, Corelight)

5.2.0-dev.213 | 2022-11-09 08:51:08 -0700

  * Update cmake submodule to remove STATIC_KQUEUE variable (Tim Wojtulewicz)

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

  * Update zeek-testing-private commit hash (Tim Wojtulewicz, Corelight)

5.2.0-dev.209 | 2022-11-08 16:44:48 -0700

  * {http,smtp}/entities: Align header regexes with extract_filename_from_content_disposition() (Arne Welzel, Corelight)

  * analyzer/HTTP: Call TCP_ApplicationAnalyzer::Done() after RequestMade()/ReplyMade() (Arne Welzel, Corelight)

  * file_analysis/Analyzer: Limit maximum number of violations (Arne Welzel, Corelight)

    Just the same as for protocol analyzers, prevent from violation event overload.

  * test: Add btest verifying max_analyzer_violations functionality (Arne Welzel, Corelight)

    The pcap has been generated roughly based on the example found on wikipedia
    with some added garbled response after the STAT command from the client.

  * analyzer: Cap analyzer violations at 1000 per analyzer instance (Arne Welzel, Corelight)

  * Report an error if a key is too big for a DictEntry (Tim Wojtulewicz, Corelight)

  * Use larger size variable for DictEntry key_size to avoid overflows with big keys (Tim Wojtulewicz, Corelight)

  * ftp: Limit pending commands to FTP::max_pending_commands (default 20) (Arne Welzel, Corelight)

  * Update baselines after AnalyzePacket changes (Arne Welzel, Corelight)

    Mainly, for bad headers we do not (can not) put addresses into
    the log anymore. Some were rather obviously bogus previously
    already.

  * protocol/ip: Only attach IP_Hdr to Packet if valid (Arne Welzel, Corelight)

    Ensure packet->ip_hdr is not set (so no one can assume it's valid)
    when AnalyzePacket() found something weird with the header.

5.2.0-dev.198 | 2022-11-07 11:47:53 +0100

  * Bump documentation.

  * GH-2528: cluster: Add warning about cluster-layout.zeek content
    (Arne Welzel, Corelight)

5.2.0-dev.195 | 2022-11-07 11:22:31 +0100

  * zeek-setup: Load builtin-plugins/__preload__.zeek before initializing bifs (Arne Welzel, Corelight)

    Prevent errors as follows with the bro-http2 plugin.

        error in /mitrecnd_HTTP2.events.bif.zeek, line 95: identifier not defined: http2_stream_stat
        error in /mitrecnd_HTTP2.events.bif.zeek, line 363: identifier not defined: http2_settings
        internal error in /mitrecnd_HTTP2.events.bif.zeek, line 460: Failed to find type named: http2_settings_unrecognized_table

  * GH-2483: cmake: Introduce bro_PLUGIN_LINK_LIBS and add to link command. (Arne Welzel, Corelight)

    When using --include-plugins with plugins that depend other libraries
    using zeek_plugin_link_library(), these libraries were stashed into
    bro_SUBDIR_LIBS that is used in add_executable(). However, that does not
    work for shared libraries.

    Explicitly track library dependencies in bro_PLUGIN_LINK_LIBS now and
    use target_link_libraries() with Zeek (there's a change in cmake to
    make zeek_plugin_link_library() populate bro_PLUGIN_LINK_LIBS instead).

    Currently, zeekdeps needs to come after bro_PLUGIN_LINK_LIBS due to how
    spicy-plugin (and other external plugins) fiddle with zeekdeps in order
    to stash dependencies into the PARENT_SCOPE.

    bro_PLUGIN_LINK_LIBS will continue to work with static libraries.

    The failure occurred when attempting to include bro-http2 [1].

    Fixes #2483

    [1] https://github.com/MITRECND/bro-http2

  * cmake: Do not use include_directories() for included plugins (Arne Welzel, Corelight)

    There's a cmake repo change that does the include_directories() in
    zeek_plugin_begin() unconditionally instead.

5.2.0-dev.189 | 2022-11-04 09:15:57 -0700

  * fuzzer-setup: Update ZEEKPATH to align with DEFAULT_ZEEKPATH (Arne Welzel, Corelight)

    The util-config.h file uses @DEFAULT_ZEEKPATH@ which includes the
    builtin-plugins directory. Do the same change for the fuzzers so
    that scripts from builtin plugins can be found.

    Fixes #2540

5.2.0-dev.186 | 2022-11-03 09:21:35 -0700

  * Add a missing deletion in ZAM code (Tim Wojtulewicz, Corelight)

  * Add a couple of mising assignment operators to match copy constructors (Tim Wojtulewicz, Corelight)

  * Fix a bunch of variable shadowing issues from LGTM (Tim Wojtulewicz, Corelight)

5.2.0-dev.182 | 2022-11-03 08:57:39 -0700

  * Add configure --display-cmake argument (Tim Wojtulewicz, Corelight)

5.2.0-dev.180 | 2022-11-03 08:57:07 -0700

  * Update external test hashes (Tim Wojtulewicz, Corelight)

  * Update DNS type strings to match correct mappings (Tim Wojtulewicz, Corelight)

5.2.0-dev.177 | 2022-11-03 08:56:24 -0700

  * Add Ubuntu 22.10 to ci (Johanna Amann, Corelight)

5.2.0-dev.175 | 2022-11-02 16:54:57 -0700

  * Spelling fixes: scripts (Josh Soref)

5.2.0-dev.173 | 2022-11-02 10:45:34 +0100

  * GH-2440: parse.y: Traverse AST post parsing to detect break/next
    usage issues (Arne Welzel, Corelight)

5.2.0-dev.171 | 2022-11-01 07:47:41 -0700

  * Func: Do not crash on va_args confusion for script funcs (Arne Welzel, Corelight)

    Script and BIF functions with a single any parameter are excluded from
    type checking regarding arguments. This makes it possible to call a
    ScriptFunc with more arguments than it actually has parameters and frame
    space for, causing heap-buffer-overflows.

    This change runtime checks expected parameters and provided arguments
    and short-circuits execution as well as logging runtime expression errors.

    Fixes #2446

5.2.0-dev.169 | 2022-10-31 15:18:04 -0700

  * bifs/record_fields: Include actual enum name in type_name (Arne Welzel, Corelight)

    One more from @stevesmoot. The record_fields() BIF produced "enum" as
    type_name for fields of type enum.

    Extend container_type_name() to append the actual name of the enum.

    This is changing the format and may break consumers, but those are
    likely in a category that are happy to adapt. Not having the actual
    enum name available wasn't very helpful.

    We could alternatively render only the actual type_name without the
    prefixed "enum", but that isn't how it's done for record types currently
    and it would make it more difficult to decide which subsequent BIFs to
    use for further introspection, like enum_names().

5.2.0-dev.167 | 2022-10-31 14:57:21 -0700

  * IPBasedAnalyzer/TCPSessionAdapter: Fix TCP reassembly decision for known port analyzers (Arne Welzel, Corelight)

    This seems to be an age-old bug. Reported by mchen on discourse [1].

    The TCPSessionAdapter decides in AddExtraAnalyzers() whether to enable
    reassembly or not. When dpd_reassemble_first_packets is F, this boils down to
    ! GetChildren().empty(). The intention being that if any analyzers have been
    added to the connection based on known ports, reassembly is to be enabled.

    However, GetChildren() does not take into account new_children and so
    ! GetChildren().empty() is always false here and reassembly solely
    based on dpd_reassemble_first_packets=F (or the tcp_content... options).
    Ouch.

    Call AppendNewChildren() before AddExtraAnalyzers() as a fix. Without this,
    the new test does not produce an http.log and service "http" isn't in conn.log.

    [1] https://community.zeek.org/t/how-to-activate-an-application-layer-analyzer-when-signature-dpd-reassemble-first-packets-is-off/6763

5.2.0-dev.162 | 2022-10-28 16:54:23 -0700

  * GH-2425: cat_sep: Make fully vararg and do explicit runtime type checks (Arne Welzel, Corelight)

  * Update cmake submodule (Tim Wojtulewicz, Corelight)

5.2.0-dev.159 | 2022-10-28 15:44:57 -0700

  * Update Spicy to the latest development snapshot. (Benjamin Bannier, Corelight)

  * Bump spicy-plugin to v1.4.0. (Benjamin Bannier, Corelight)

    spicy-plugin-1.4.0 changes behavior, so it is not backwards-compatible
    with the spicy-plugin-1.3 series. This patch should not be backported to
    LTS release branches.

5.2.0-dev.156 | 2022-10-27 17:11:13 -0700

  * Bump spicy-plugin to v1.3.21. (Benjamin Bannier, Corelight)

5.2.0-dev.154 | 2022-10-27 17:09:55 -0700

  * egrep reported as obsolete by opensuse-tumbleweed builds (Tim Wojtulewicz, Corelight)

5.2.0-dev.150 | 2022-10-26 16:48:18 -0700

  * Include in Jan's AF_PACKET plugin as builtin plugin (Arne Welzel, Corelight)

  * Update doc submodule. (Robin Sommer, Corelight)

5.2.0-dev.147 | 2022-10-26 11:37:24 +0200

  * GH-2272: Use `json_escape_utf8` in broker debug output (AmazingPP)

5.2.0-dev.145 | 2022-10-26 10:51:05 +0200

  * Reintroduce event groups. (Arne Welzel, Corelight)

    Allow the ``&group`` attribute on event and hook handlers for
    annotating them with one or more event groups. These groups can be
    disabled and enable during runtime. Disabling an event group
    implies disabling all event and hook handlers that are part of it.

    Additionally, event and hook handlers are implicitly placed into
    event module groups based on the module they are implemented in.
    All events implemented in a given module can be toggled with
    ``disable_module_events()`` and ``enable_module_events()``.

  * Update doc submodule [nomail] [skip ci] (zeek-bot)

5.2.0-dev.142 | 2022-10-25 12:37:16 +0200

  * enum_names: Support naming types with a string (Arne Welzel, Corelight)

    Add support to enum_names() to transparently lookup the type if a string
    is provided. This is similar in how record_fields() behaves when being
    passed a string.

5.2.0-dev.140 | 2022-10-25 12:34:39 +0200

  * scripts: Migrate table iteration to blank identifiers (Arne Welzel, Corelight)

  * Introduce special treatment for the blank identifier _ (Arne Welzel, Corelight)

    Mostly: Do not instantiate variables within for loops and allow
    reusing differently typed blanks which previously wasn't possible.

  * Management framework: bump zeek-client and cluster testsuite (Christian Kreibich, Corelight)

  * Add new zeek-client dependency to Dockerfile: python3-websocket (Christian Kreibich, Corelight)

    This is a required dependency of zeek-client, and available as a .deb on Debian.

  * Management framework: add websocket support to controller (Christian Kreibich, Corelight)

    The controller now listens on an additional port, defaulting to 2149, for Broker
    connections via websockets. Configuration works as for the existing traditional
    Broker port (2150), via ZEEK_CONTROLLER_WEBSOCKET_ADDR and
    ZEEK_CONTROLLER_WEBSOCKET_PORT environment variables, as well as corresponding
    redef'able constants.

    To disable the websockets feature, leave ZEEK_CONTROLLER_WEBSOCKET_PORT unset
    and redefine Management::Controller::default_port_websocket to 0/unknown.

5.2.0-dev.134 | 2022-10-24 08:10:06 -0700

  * Re-enable ci-based benchmarking (Tim Wojtulewicz, Corelight)

5.2.0-dev.131 | 2022-10-21 16:29:14 -0700

  * Remove external CAF build for Coverity setup (Christian Kreibich, Corelight)

5.2.0-dev.129 | 2022-10-21 12:24:25 -0700

  * Remove cmake tests for LIBPCAP_PCAP_COMPILE_NOPCAP_HAS_ERROR_PARAMETER (Tim Wojtulewicz, Corelight)

  * Rework the non-pcap version of BPF_Program::Compile (Tim Wojtulewicz, Corelight)

  * Add btest that exercises the pcap filter warnings (Tim Wojtulewicz, Corelight)

  * Restore reporting messages for pcap filter issues (Tim Wojtulewicz, Corelight)

  * Add column to packet_filter.log for failure reason (Tim Wojtulewicz, Corelight)

  * Store error message from BPF compilation (Tim Wojtulewicz, Corelight)

  * Minor cleanup in BPF filtering code (Tim Wojtulewicz, Corelight)

5.2.0-dev.120 | 2022-10-21 10:09:11 -0700

  * Reorder includes in Options.cc (Tim Wojtulewicz, Corelight)

  * Remove some leftover nb_dns stuff from the DNS_Mgr rework (Tim Wojtulewicz, Corelight)

5.2.0-dev.116 | 2022-10-20 13:03:45 -0700

  * Update bifcl, binpac, broker, cmake, gen-zam, and paraglob submodules (Tim Wojtulewicz, Corelight)

5.2.0-dev.113 | 2022-10-18 08:30:45 -0700

  * reset performance metrics every SizeUp() (Jason Lu)

  * reduce space_distance_samples from int64_6 to uint32_t (Jason Lu)

  * add performance based growth strategy (Jason Lu)

5.2.0-dev.108 | 2022-10-17 16:18:55 -0700

  * -O C++ maintenance updates to test suite (Vern Paxson, Corelight)

  * -O gen-C++ maintenance: fixes for attributes & initializing globals (Vern Paxson, Corelight)

5.2.0-dev.105 | 2022-10-17 08:48:20 -0700

  * Surface a better CMake error if the user passes an invalid sanitizer name (Tim Wojtulewicz, Corelight)

5.2.0-dev.103 | 2022-10-16 17:49:24 -0700

  * plugins: Reject dynamic plugins matching names of built-in ones (Arne Welzel, Corelight)

    This goes the hard-exit on conflicts route as IMO it provides better
    messaging that something is wrong, rather than defaulting to something
    the user may not expect.

    Fixes #2403

5.2.0-dev.101 | 2022-10-16 17:48:26 -0700

  * Add &ordered attribute for tables/sets (Tim Wojtulewicz, Corelight)

5.2.0-dev.99 | 2022-10-16 15:30:25 -0700

  * Update zeek-3rdparty submodule for doctest/sqlite updates (Tim Wojtulewicz, Corelight)

  * Update libkqueue submodule to latest upstream master (Tim Wojtulewicz, Corelight)

  * Update rapidjson submodule to latest upstream master (Tim Wojtulewicz, Corelight)

  * Update highwayhash submodule to latest upstream master (Tim Wojtulewicz, Corelight)

5.2.0-dev.94 | 2022-10-13 15:12:19 -0500

  * Add toplevel build dir to `PATH` for dev helper scripts. (Benjamin Bannier, Corelight)

    We already added the toplevel build dir to the paths exposed by
    `build/zeek-path-dev`, but never made the matching change for
    `build/zeek-path-dev.[sh,csh]`. Due to that a shell would have never
    found `zeek-config` from the build env like it would have been found for
    installations, and could potentially even have picked up a different
    one.

5.2.0-dev.92 | 2022-10-13 15:10:51 -0500

  * Update bifcl submodule to get fixes for DWARF information for lldb (Tim Wojtulewicz, Corelight)

5.2.0-dev.90 | 2022-10-13 08:57:45 -0500

  * ZAM maintenance for recent test suite changes (and fixups missed last time) (Vern Paxson, Corelight)

5.2.0-dev.86 | 2022-10-12 15:56:23 -0500

  * Update gen-zam and paraglob submodules for Coverity/clang-tidy fixes (Tim Wojtulewicz, Corelight)

  * Fix some compiler warnings in script_opt (Tim Wojtulewicz, Corelight)

  * Fix recent Coverity findings (Tim Wojtulewicz, Corelight)

    1491334: AUTO_CAUSES_COPY due to a for loop using auto& instead of const auto&
    1498652: AUTO_CAUSES_COPY due to a for loop using auto& instead of const auto&
    1498745: AUTO_CAUSES_COPY due to a for loop using auto& instead of const auto&

5.2.0-dev.82 | 2022-10-12 20:02:36 +0200

  * Move API for disabling packet analyzers into component. (Robin Sommer)

    Before, that API was part of the analyzers themselves, which meant
    we couldn't disable a packet analyzer before it had been
    instantiated. That was different from protocol/file analyzers,
    where we disable them through the corresponding component.

  * Suppress warning on an existing packet analyzer being remapped. (Robin Sommer)

5.2.0-dev.79 | 2022-10-12 09:32:38 -0700

  * smtp: Prevent script errors when smtp$entity is not set (Tim Wojtulewicz, Corelight)

    This is the same issue presented in 38e226bf75772dad8769fa4b5ae61fabb5fb34b9 but
    for SMTP instead of HTTP.

5.2.0-dev.77 | 2022-10-10 11:10:31 -0700

  * Disable robust iteration for ordered dictionaries (Tim Wojtulewicz, Corelight)

    This also includes some minor commenting cleanup in that class

  * Add support for itertors with ordered dictionaries (Tim Wojtulewicz, Corelight)

  * Add equality, inequality, copy, and move operators to HashKey (Tim Wojtulewicz, Corelight)

  * Disable annoying bugprone-easily-swappable-parameters clang-tidy check [skip ci] (Tim Wojtulewicz, Corelight)

5.2.0-dev.70 | 2022-10-06 14:18:18 -0700

  * Remove stray `/` at the end of directory name. (Benjamin Bannier, Corelight)

    This variable is used to construct a path where the trailing `/` lead to
    full paths like `/foo/spicy-plugin//spicyz`.

  * Remove commented out code. (Benjamin Bannier, Corelight)

  * Bump spicy-plugin to v1.3.19. (Benjamin Bannier, Corelight)

5.2.0-dev.66 | 2022-10-06 14:15:01 -0700

  * Update CT log list. (Johanna Amann, Corelight)

    This uses the v3 json as a source for the first time. The test needed
    some updating because Google removed a couple more logs - in the future
    this should hopefully not be neccessary anymore because I think v3
    should retain all logs.

    In theory this might be neat in 5.1.

5.2.0-dev.51 | 2022-10-05 13:27:08 +0200

  * analyzer: Add file_analyzer support to enable_analyzer()/disable_analyzer() (Arne Welzel, Corelight)

5.2.0-dev.46 | 2022-10-03 09:45:05 -0700

  * use dynamic rather than static initialization of globals for scripts compiled to C++ (Vern Paxson, Corelight)

  * safety checking for initializing scripts compiled to C++ (Vern Paxson, Corelight)

  * fixes for initializing scripts compiled to C++ (Vern Paxson, Corelight)

  * restructured tracking of initializations of globals for script compilation (Vern Paxson, Corelight)

5.2.0-dev.40 | 2022-10-03 09:44:40 -0700

  * http: Prevent script errors when http$current_entity is not set (Arne Welzel, Corelight)

    The current_entity tracking in HTTP assumes that client/server never
    send HTTP entities at the same time. The attached pcap (generated
    artificially) violates this and triggers:

        1663698249.307259 expression error in <...>base/protocols/http/./entities.zeek, line 89: field value missing (HTTP::c$http$current_entity)

    For the http-no-crlf test, include weird.log as baseline. Now that weird is
    @load'ed from http, it is actually created and seems to make sense
    to btest-diff it, too.

5.2.0-dev.38 | 2022-10-03 09:08:42 -0700

  * Fix some Coverity issues in the recent ZAM updates (Tim Wojtulewicz, Corelight)

5.2.0-dev.35 | 2022-09-30 16:40:22 -0700

  * fixes for compiling "standalone" C++ scripts (Vern Paxson, Corelight)

5.2.0-dev.33 | 2022-09-30 12:36:34 -0700

  * fix for deprecated when's where the inner frame is larger than the outer frame (Vern Paxson, Corelight)

5.2.0-dev.31 | 2022-09-30 10:03:00 +0200

  * const: Deprecate tunnel-specific enable_* flags (Arne Welzel, Corelight)

    With packet analyzers being toggle-able at runtime these can go.
    They hadn't been consistently implemented either (VXLAN, Geneve).

  * packet_analysis: Introduce PacketAnalyzer::__disable_analyzer() (Arne Welzel, Corelight)

    This adds machinery to the packet_analysis manager for disabling
    and enabling packet analyzers and implements two low-level bifs
    to use it.

    Extend Analyzer::enable_analyzer() and Analyzer::disable_analyzer()
    to transparently work with packet analyzers, too. This also allows
    to add packet analyzers to Analyzer::disabled_analyzers.

5.2.0-dev.27 | 2022-09-28 10:34:28 +0200

  * Bump `bifcl` and `doc` submodules. (Robin Sommer)

5.2.0-dev.25 | 2022-09-28 10:00:40 +0200

  * Introduce generic analyzer_confirmation_info and
    analyzer_violation_info (Arne Welzel, Corelight)

    Introduce two new events for analyzer confirmation and analyzer violation
    reporting. The current analyzer_confirmation and analyzer_violation
    events assume connection objects and analyzer ids are available which
    is not always the case. We're already passing aid=0 for packet analyzers
    and there's not currently a way to report violations from file analyzers
    using analyzer_violation, for example.

    These new events use an extensible Info record approach so that additional
    (optional) information can be added later without changing the signature.
    It would allow for per analyzer extensions to the info records to pass
    analyzer specific info to script land. It's not clear that this would be
    a good idea, however.

    The previous analyzer_confirmation and analyzer_violation events
    continue to exist, but are deprecated and will be removed with Zeek 6.1.

  * GH-133: packet_analysis: Protect from nullptr sessions (Arne
    Welzel, Corelight) Passing nullptr sessions to
    AnalyzerConfirmation and AnalyzerViolation of protocol analyzers
    previously blew up - protect from that.

  * packet_analysis: Raise violations once (and don't raise
    confirmations after violations) (Arne Welzel, Corelight) This is
    mostly to avoid per-packet violations for packet analyzers that
    have sessions attached to them.

  * Introduce basic test triggering when zeek -r triggers script
    errors. (Arne Welzel, Corelight)

  * file_analysis: Implement AnalyzerViolation() for
    file_analysis/Analyzer (Arne Welzel, Corelight)

5.2.0-dev.19 | 2022-09-27 13:32:20 +0200

  * GH-2422: cmake: Exclude empty directories due to headers install
    (Arne Welzel, Corelight)

5.2.0-dev.17 | 2022-09-27 13:30:27 +0200

  * packet_analysis: Do not raise analyzer_confirmation per-packet for
    tunnels (Arne Welzel, Corelight)

5.2.0-dev.15 | 2022-09-23 13:33:16 -0700

  * Skip darwin builds on zeek-security repo (Tim Wojtulewicz, Corelight)

5.2.0-dev.12 | 2022-09-20 21:11:02 -0700

  * scripts/conn: Open-code determine_service() (Arne Welzel, Corelight)

  * Make connection$service_violation a set[string] (Arne Welzel, Corelight)

  * strings: Implement join_string_set() as bif (Arne Welzel, Corelight)

  * GH-2388: dpd: Replace negated service fmt() magic with dedicated field (Arne Welzel, Corelight)

5.2.0-dev.6 | 2022-09-20 13:04:41 -0700

  * Dict: Mark iterator move operator/constructors as noexcept (Tim Wojtulewicz, Corelight)

  * Remove unused headers from Dict.cc (Tim Wojtulewicz, Corelight)

  * Dict: Reduce some math required in repeatedly-called functions (Tim Wojtulewicz)

  * Dict: use std::log2, remove custom Log2 method (Tim Wojtulewicz)

  * Fix some typos in the 5.1.0 NEWS (Tim Wojtulewicz, Corelight)

  * Start of 5.2.0 development (Tim Wojtulewicz, Corelight)

  * Replace all the Warning() calls after IsFieldDeprecated() over to Warn() (AmazingPP)

  * Fix deprecation not flagged and incorrect line number in record (AmazingPP)

5.1.0-dev.535 | 2022-09-19 13:51:11 -0700

  * updates to notes for compile-to-C++ maintenance (Vern Paxson, Corelight)

  * fix deprecated "local" scoping in test scripts (Vern Paxson, Corelight)

  * C++ script generation fix for lambdas that have identical bodies (Vern Paxson, Corelight)

  * fix for C++ scripts that refer to "opaque" types (Vern Paxson, Corelight)

  * C++ compilation support for 2-valued vector "for" loops (Vern Paxson, Corelight)

  * C++ compilation support for RE /s operator (Vern Paxson, Corelight)

  * run-time checking of vector operations for overflows and division-by-zero (Vern Paxson, Corelight)

  * error propagation fix to avoid a crash (Vern Paxson, Corelight)

5.1.0-dev.517 | 2022-09-19 13:44:13 -0700

  * specialized ZAM operators for speeding up cat() operations (Vern Paxson, Corelight)

  * ZAM support for two-valued "for" loops over vectors (Vern Paxson, Corelight)

  * make it non-fatal for optimized scripts to encounter an empty "for" body (Vern Paxson, Corelight)

  * script optimization fix for new-style table constructors (Vern Paxson, Corelight)

5.1.0-dev.511 | 2022-09-19 12:41:31 -0700

  * Add better length check to BuildOptionsVal() (Tim Wojtulewicz, Corelight)

  * smb: Unify transaction_data_to_val() usage (Arne Welzel, Corelight)

    the data/string handling in smb1-com-transaction.pcac was improved
    with c75519ca8869448efc488598090f96d63352312f, re-use the added
    functionality also for smb1-com-transaction-secondary.pac to avoid
    buffer overflows.

  * Add some additional checks for caplen in ICMP analyzer (Tim Wojtulewicz, Corelight)

  * irc: Check parts.size() before removing the nick (Arne Welzel, Corelight)

    ...also, don't remove the nick twice.

  * Check IP payload length before casting to transport structure (Tim Wojtulewicz, Corelight)

5.1.0-dev.506 | 2022-09-15 11:11:57 -0400

  * Use NextStream instead of ForwardStream in fuzzers (Tim Wojtulewicz, Corelight)

5.1.0-dev.504 | 2022-09-13 08:09:27 -0400

  * Bump spicy-plugin to v1.3.18. (Benjamin Bannier, Corelight)

5.1.0-dev.500 | 2022-09-09 09:37:48 -0700

  * CI: remove FreeBSD 13.0, update FreeBSD 12 to 12.3 (Tim Wojtulewicz, Corelight)

5.1.0-dev.497 | 2022-09-07 14:31:07 +0200

  * GH-2289: parse.y/directives: Reject directives as statements (Arne Welzel, Corelight)

    Conditional directives (``@if``, ``@ifdef``, ``@ifndef``, ``@else`` and ``@endif``)
    can not be placed directly following ``if``, ``for`` or ``while`` statements
    anymore. This was interpreted non-intuitively and could lead to subtle bugs.
    The statement following the directive was placed outside of its intended block.
    Placing braces after ``if``, ``for`` or ``while`` should result in the
    intended behavior.

5.1.0-dev.495 | 2022-09-07 14:29:32 +0200

  * frameworks/dpd: Move to frameworks/analyzer/dpd, load by default
    (Arne Welzel, Corelight)

5.1.0-dev.489 | 2022-09-06 07:44:18 -0700

  * Add is_used attribute to an ID if used in a signature eval statement (Tim Wojtulewicz, Corelight)

5.1.0-dev.486 | 2022-09-02 13:57:31 -0700

  * Update Management framework to new Supervisor::NodeConfig script fields (Christian Kreibich, Corelight)

  * Suppress deprecation warnings for the Supervisor's own NodeConfig::script access (Christian Kreibich, Corelight)

  * Expand Supervisor to support loading additional scripts before user scripts (Christian Kreibich, Corelight)

5.1.0-dev.481 | 2022-09-01 00:52:02 +0000

  * Fix warning in modbus analyzer (Tim Wojtulewicz, Corelight)

5.1.0-dev.479 | 2022-08-31 10:50:05 -0700

  * telemetry: Use dynamic metrics port, remove broker topic mentioning from NEWS entry (Arne Welzel, Corelight)

  * telemetry: In a cluster, open port 9911 for Prometheus by default (Arne Welzel, Corelight)

5.1.0-dev.476 | 2022-08-31 08:16:53 -0700

  * bifs/to_port: Avoid ASAN errors when calling to_port("") (Arne Welzel, Corelight)

    Ran into this when using to_port(getenv(...)) for an undefined/empty
    environment variable. ASAN doesn't like that the slash variable ends
    up being access behind the string.

5.1.0-dev.473 | 2022-08-29 09:26:56 -0700

  * Add type checking to string_cat arguments (Tim Wojtulewicz, Corelight)

5.1.0-dev.471 | 2022-08-26 14:49:04 -0700

  * gtpv1: Do not register for protocol detection (Arne Welzel, Corelight)

5.1.0-dev.469 | 2022-08-26 14:36:03 -0700

  * Move fuzzer corpus files to another directory (Tim Wojtulewicz, Corelight)

  * Add a way to create generic fuzzers without creating new files (Tim Wojtulewicz, Corelight)

5.1.0-dev.466 | 2022-08-26 11:00:06 -0700

  * Check for valid ip_hdr length before trying to make a Val out of it (Tim Wojtulewicz, Corelight)

  * Fix potential overflow in modbus analyzer's bytestring_to_coils (Tim Wojtulewicz, Corelight)

  * Add some sanity checking to BadARPEvent method (Tim Wojtulewicz, Corelight)

  * Swap DNS EDNS field order to match script-land type (Tim Wojtulewicz, Corelight)

  * Reset packet cap_len before returning from IP::AnalyzePacket (Tim Wojtulewicz, Corelight)

5.1.0-dev.461 | 2022-08-26 09:09:23 -0700

  * Revert to regular peering fn for ssl_auth_failure (Dominik Charousset, Corelight)

    With the fixed Broker version, the `ssl_auth_failure` btest can use the
    regular `peer` function again.

  * Bump Spicy dependencies to spicy-1.5.1 and spicy-plugin-1.3.17. (Benjamin Bannier, Corelight)

    (cherry picked from commit 7e1840f5fed47ddaa9111f265bb647f16c49663a)

5.1.0-dev.457 | 2022-08-25 15:15:29 -0700

  * Switch macOS Monterey build to new m1-based VM (Tim Wojtulewicz, Corelight)

5.1.0-dev.455 | 2022-08-25 12:46:00 -0700

  * Avoid race conditions in broker.ssl_auth_failure btest (Tim Wojtulewicz, Corelight)

5.1.0-dev.452 | 2022-08-24 09:33:58 -0700

  * Use correct variable to pass existing Spicy root to spicy-plugin. (Benjamin Bannier, Corelight)

    spicy-plugin can find Spicy in paths given by `SPICY_ROOT_DIR` while
    `./configure` instead set `SPICY_ROOT`. With this patch we now set the
    correct variable.

    We also adjust variations of the previous variable name with different
    capitalization which caused us to not properly configure spicy-plugin
    (which triggers finding Spicy in its given prefix).

    Closes #2363.

5.1.0-dev.450 | 2022-08-24 09:22:47 -0700

  * Skip test based on preprocessor flag set by cmake (Simeon Miteff, Corelight)

    Relies on change in d42dcb2d55029975a6a6b2e6378fc49a268631ec

  * Set flag for libpcap without DLT_LINUX_SLL2 (Simeon Miteff, Corelight)

    Requires
    https://github.com/zeek/cmake/commit/6fd82a7e1d626f68ebf616b45f9bec11ca49d295

    Submodule edited until that can be merged.

  * Force event order in core/init-error btest (Simeon Miteff, Corelight)

    See https://github.com/zeek/zeek/pull/2340#issuecomment-1218131444

  * Add support for DLT_LINUX_SLL2 PCAP link-type (Simeon Miteff, Corelight)

5.1.0-dev.442 | 2022-08-24 13:22:17 +0100

  * Add Broker::metrics_import_topics (Arne Welzel & Dominik Charousset, both Corelight)

5.1.0-dev.439 | 2022-08-23 20:28:05 -0700

  * parse.y/anonymous_function: Allow conditionals between begin_lambda and lambda_body (Arne Welzel, Corelight)

    This is to allow conditional statements following a lambda header
    (begin_lambda) just as is done with func_hdr conditional_list func_body.

5.1.0-dev.436 | 2022-08-23 10:19:23 -0700

  * Deprecate misc/scan.zeek (Arne Welzel, Corelight)

5.1.0-dev.434 | 2022-08-23 09:33:42 -0700

  * stmt: Support iterating over vector values (Arne Welzel, Corelight)

5.1.0-dev.432 | 2022-08-22 17:34:10 -0700

  * Truncate Docker image artifact at end of Docker CI workflow (Christian Kreibich, Corelight)

5.1.0-dev.430 | 2022-08-22 16:12:28 -0700

  * Fix signed comparison warnings in bytestring_to_count (Tim Wojtulewicz)

5.1.0-dev.428 | 2022-08-22 14:16:59 -0700

  * Bump doc to pull in Sphinx exclusion for autogenerated files (Christian Kreibich, Corelight)

  * Remove long-unused BRO_DISABLE_BROXYGEN env var from btests (Christian Kreibich, Corelight)

  * Add Sphinx label to Zeekygen-generated plugin docs (Christian Kreibich, Corelight)

5.1.0-dev.424 | 2022-08-22 10:56:47 -0700

  * Merge branch 'topic/timw/parse-packet-results' (Tim Wojtulewicz, Corelight)

    * topic/timw/parse-packet-results:
      Make result of IP::ParsePacket easier to understand
      Remove reporter warning for bad IP protocols

  * Make result of IP::ParsePacket easier to understand (Tim Wojtulewicz, Corelight)

  * Remove reporter warning for bad IP protocols (Tim Wojtulewicz, Corelight)

    It turns out that this can be *very* spammy on networks where we're receiving
    lots of these packets, and can fill up the reporter log very quickly. Weirds are
    already reported in all of these cases anyways, so it doesn't make sense to log
    a reporter warning too.

5.1.0-dev.421 | 2022-08-19 15:23:22 -0700

  * Remove unified2 file analyzer (Arne Welzel, Corelight)

5.1.0-dev.419 | 2022-08-19 14:57:12 -0700

  * Remove barnyard2 integration scripts (Arne Welzel, Corelight)

5.1.0-dev.417 | 2022-08-18 15:56:49 -0700

  * Retroactively cover log filter predicate deprecation/removal in NEWS (Christian Kreibich, Corelight)

5.1.0-dev.414 | 2022-08-17 15:46:05 -0700

  * Support other byte lengths in bytestring_to_count (Tim Wojtulewicz, Corelight)

5.1.0-dev.412 | 2022-08-17 13:31:13 -0700

  * Alignment tweak in help output (Christian Kreibich, Corelight)

5.1.0-dev.410 | 2022-08-17 13:29:01 -0700

  * GH-163: Use ID name (including module name) to create EventExpr when possible (Tim Wojtulewicz, Corelight)

    The change to the capture-loss test is actually a fix for a bug exposed by the
    code change. Previously it wasn't firing the scheduled event because of a failed
    name lookup. Now that the lookup has been fixed, the event happens twice.

5.1.0-dev.408 | 2022-08-17 09:00:50 -0700

  * Use ForwardStream in fuzzers instead of DeliverStream (Tim Wojtulewicz, Corelight)

5.1.0-dev.406 | 2022-08-17 10:38:07 +0200

  * Fix code formatting. (Robin Sommer, Corelight)

5.1.0-dev.405 | 2022-08-17 09:07:33 +0200

  * Add support for parsing TCP option 27, and validate lengths for
    TCP options 28, 29, & 34. (Fatema BW)

5.1.0-dev.393 | 2022-08-17 08:59:11 +0200

  * files.log: Unroll and introduce uid and id fields (Arne Welzel, Corelight)

    This is a script-only change that unrolls File::Info records into
    multiple files.log entries if the same file was seen over different
    connections by a single worker. Consequently, the File::Info record
    gets the commonly used uid and id fields added. These fields are
    optional for File::Info - a file may be analyzed without relation
    to a network connection (e.g by using Input::add_analysis()).

    The existing tx_hosts, rx_hosts and conn_uids fields of Files::Info
    are not meaningful after this change and removed by default. Therefore,
    files.log will have them removed, too.

    The tx_hosts, rx_hosts and conn_uids fields can be revived by using the
    policy script frameworks/files/deprecated-txhosts-rxhosts-connuids.zeek
    included in the distribution. However, with v6.1 this script will be
    removed.

5.1.0-dev.391 | 2022-08-16 09:17:06 -0700

  * Fix a swath of AUTO_CAUSES_COPY issues reported by Coverity (Tim Wojtulewicz, Corelight)

5.1.0-dev.389 | 2022-08-15 09:22:30 -0700

  * Update zeek-testing-private commit hash after email regex changes (Tim Wojtulewicz, Corelight)

    Related to https://github.com/zeek/zeek/pull/2221

5.1.0-dev.386 | 2022-08-11 11:56:55 -0700

  * Add a field to Modbus/TCP log to indicate the Modbus PDU type (Michael Torres)

    Add the `pdu_type` field to Modbus over TCP logs to indicate whether the Modbus
    message was a request or a response. Due to the client/server nature of Modbus
    over TCP/IP, all messages from the TCP session originator are requests, while
    all messages from the TCP session responder are responses.

    Adding this information to the default log surfaces protocol metadata in a way
    that doesn't require users to understand the Modbus over TCP protocol.

  * Add modbus transaction and unit ids to logs (Michael Torres)

    Add transaction IDs and unit IDs to default modbus over TCP/IP logs.
    Update the relevant testing baselines to account for the extra fields.

  * Enable modbus logging for requests (Michael Torres)

5.1.0-dev.382 | 2022-08-11 10:41:08 -0700

  * UPDATED: improving email address splitting for common comma case (TheAvgJojo)

5.1.0-dev.380 | 2022-08-11 10:30:33 -0700

  * Fix a crash related to a broken IPv6 chain (Tim Wojtulewicz)

  * Add a couple of ICMP files to corpus for packet-fuzzer (Tim Wojtulewicz, Corelight)

  * Trick event handlers into returning that they exist during fuzzing (Tim Wojtulewicz, Corelight)

  * Add http, ftp, imap, and smtp fuzzers and corpora (Tim Wojtulewicz, Corelight)

  * Add section to fuzzer README about generating corpus from pcaps (Tim Wojtulewicz, Corelight)

  * Rename fuzzers/README to README.rst so github renders it (Tim Wojtulewicz, Corelight)

5.1.0-dev.373 | 2022-08-11 08:49:22 -0700

  * Add table_keys function (AmazingPP)

  * Add table_values function (AmazingPP)

5.1.0-dev.368 | 2022-08-11 16:01:32 +0200

  * GH-1678: Introduce global disabling_analyzer() hook to veto disable_analyzer() (Arne Welzel, Corelight)

    This hook can be used to coordinate disabling an analyzer for a
    given connection. The contract is simple: Any script can veto a
    disable_analyzer() call by breaking from this hook. The decision
    is local to the script taking into account any state attached to
    the connection object or script specific state stored elsewhere. A
    script breaking from the hook takes over the responsibility to
    call disable_analyzer() at a later point when it finds the
    condition due to which it vetoed fulfilled (which may be never).

    Signature:

        disabling_analyzer: hook(c: connection, atype: AllAnalyzers::Tag, aid: count);

    Example use-cases are keeping the SSL analyzer enabled for
    finger-printing until a certain amount of bytes or packets have
    been transferred or similarly the connection duration exceed a
    certain threshold.

    Other example use-cases might be keeping analyzers for SSH, RDP or SSL
    enabled for connections from specific subnets.

  * ssl: Only delete `c$ssl$analyzer_id` when disabling the analyzer
    was successful (Arne Welzel, Corelight)

5.1.0-dev.364 | 2022-08-11 11:59:00 +0200

  * GH-2000: Support redef'ing the &log attribute of record fields (Arne Welzel, Corelight)

    Add new syntax for adding and removing attributes from record fields:

        redef RecordType$field_name += { &log };
        redef RecordType$field_name -= { &log };

    For now this only allowed for the &log attribute as the semantics are clear.
    For &default and &optional the semantics aren't obvious and no use-cases have
    been identified where those would make sense to change.

    This enables a mechanism to add potentially interesting fields to the typical
    Info records in base scripts, but letting users opt-into actually including
    them into their log. At the same time, users that find specific fields in a
    standard log uninteresting can opt-out without using `Log::Filter$exclude`
    which can be difficult to use correctly. Patching or forking external packages
    to remove columns from a log can also be avoided with this mechanism.

    Closes #2000.

  * GH-2262: telemetry: Remove singleton BIFs and the C++ pieces (Arne Welzel, Corelight)

    The low-level singleton Telemetry BIFs have been removed with the that there
    haven't been any users. Singleton metrics can be instantiated by providing
    an empty label vector instead and aren't in any way a special concept.

    Closes #2262.

5.1.0-dev.360 | 2022-08-11 11:52:26 +0200

  * telemetry: Switch to histogram family bucket and bound accessors
    (Arne Welzel, Corelight)

5.1.0-dev.357 | 2022-08-10 14:16:33 -0700

  * Management framework: minor bugfixes and tweaks (Christian Kreibich, Corelight)

    - bump cluster testsuite
    - log node set in dispatch requests cleanly
    - log additional node events
    - upon deployment, make agent log multiple node results
    - fix early return condition for get-id-value

5.1.0-dev.351 | 2022-08-09 09:50:13 -0700

  * Fix module-scoped type definitions that conflict with existing global ones (Tim Wojtulewicz, Corelight)

5.1.0-dev.349 | 2022-08-08 11:09:04 -0700

  * lower priority for change handlers (Yacin Nadji, Corelight)

  * split update_zones_regex into two functions (Yacin Nadji, Corelight)

5.1.0-dev.344 | 2022-08-08 11:03:24 -0700

  * Squelch the zeekygen warnings for command line (AmazingPP)

5.1.0-dev.341 | 2022-08-05 16:21:13 +0000

  * Added test case with back-to-back notices (Annie Bryan)

  * Fix notice suppression atomicity bug (Annie Bryan)

5.1.0-dev.337 | 2022-08-05 16:19:59 +0000

  * GH-2034: Store module names and use them in lookups for ifdef (Tim Wojtulewicz, Corelight)

5.1.0-dev.335 | 2022-08-05 16:50:18 +0200

  * Introduce telemetry framework (Arne Welzel, Corelight)

    Adds base/frameworks/telemetry with wrappers around telemetry.bif
    and updates telemetry/Manager to support collecting metrics from
    script land.

    Adds policy/frameworks/telemetry/log for logging of metrics data
    into a new telemetry.log and telemetry_histogram.log and add into
    local.zeek by default.

5.1.0-dev.331 | 2022-08-03 10:27:28 -0700

  * input/config: Use thread-safe Fmt() rather than util::fmt() for regex formatting (Arne Welzel, Corelight)

    Calling util::fmt() from DoUpdate() of a thread is not safe as it is
    using a statically allocated buffer and other threads or the main
    thread may concurrently modify this buffer.

    This was found by observing the scripts.base.frameworks.config.several-files
    failing once in a blue moon (1/250 sometimes 1/1000 runs) with messages like
    "Failed to compile regex: Parenthesis is not closed.":

        scripts.base.frameworks.config.several-files ...
          > btest-bg-run zeek zeek -b %INPUT
          > btest-bg-wait 10
        ... scripts.base.frameworks.config.several-files failed
          % 'btest-bg-wait 10' failed unexpectedly (exit code 1)
          % cat .stderr
          The following processes did not terminate:
          zeek -b /home/awelzel/corelight-oss/zeek/testing/btest/.tmp/scripts.base.frameworks.config.several-files/several-files.zeek
          -----------
          <<< [3667265] zeek -b /home/awelzel/corelight-oss/zeek/testing/btest/.tmp/scripts.base.frameworks.config.several-files/several-files.zeek
          error: ../configfile1/Input::READER_CONFIG: Failed to compile regex: Parenthesis is not closed.
          received termination signal
          >>>

5.1.0-dev.329 | 2022-08-03 10:26:45 -0700

  * testing/missing-enum-value: redef exit_only_after_terminate=T (Arne Welzel, Corelight)

    Seems this was the intention all along as the corresponding terminate()
    call is there.

  * option.bif: Short-circuit option changes when terminating (Arne Welzel, Corelight)

    Due to the asynchronous behavior of the input framework and broker
    communication, change handlers were previously called even after
    zeek_done() event processing completed and also broker shutdown.

    Accessing broker store handles within change handlers this late
    triggered invalid Broker store handle messages:

        error in ././my_option_store.zeek, line 13: invalid Broker store handle (Broker::put(Test::store, to_any_coercemy_option, to_any_coerceTest::new_value, 0 secs) and broker::store::{})

    Fixes #2010

5.1.0-dev.326 | 2022-08-03 09:56:37 -0700

  * updates for gen-C++ maintenance, including skipping some inappropriate tests (Vern Paxson, Corelight)

  * fix for profiling "when" statements (Vern Paxson, Corelight)

  * gen-C++ support for vector bit-shift operations (Vern Paxson, Corelight)

  * corrected wording in some btest comments (Vern Paxson, Corelight)

  * make gen-C++ maintenance scripts directly executable (Vern Paxson, Corelight)

  * ZAM support for bit-shifting (Vern Paxson, Corelight)

  * don't allow deprecated-style mixing of vectors and scaling for shifting (Vern Paxson, Corelight)
    leverage restrictions placed on shifting (RHS is always unsigned)
    split deprecated vector operations into separate test, with separate ZAM baseline

  * ZAM fix for vector "in" operator (Vern Paxson, Corelight)

  * ensure that language tests pay attention to .stderr (Vern Paxson, Corelight)

  * fix vector tests, including checking for errors (Vern Paxson, Corelight)

5.1.0-dev.312 | 2022-08-02 12:37:51 -0700

  * Update plugins.hooks baseline with new DHCP options (peter.cullen, Corelight)

  * Prevent large dhcp log entries (Peter Cullen, Corelight)

    A flood of DHCP traffic can result if very large log entries consisting
    of many uids and/or msg_types. Such large log entries can disrupt a SIEM
    ingestion pipeline. This change forcing a log entry to be written when
    the number of uids or the number of msg_Types exceed a certain value.
    The values are treated as options for easy configuration.

5.1.0-dev.309 | 2022-08-02 11:33:22 -0700

  * Add support for /s modifier to RE matcher and parser (Tim Wojtulewicz, Corelight)

  * Code cleanup in RE_Matcher code (Tim Wojtulewicz, Corelight)

    - Use std::string in Specific_RE_Matcher instead of char*
    - Change a couple of ints-as-bools to bools

  * Add basic unit tests for RE_Matcher (Tim Wojtulewicz, Corelight)

  * Add /s modifier to parser for patterns (Tim Wojtulewicz, Corelight)

  * Update gen-zam submodule [nomail] (Tim Wojtulewicz, Corelight)

5.1.0-dev.303 | 2022-08-01 09:56:45 -0700

  * GH-1344: Give better warning when using a type that doesn't exist (Tim Wojtulewicz, Corelight)

5.1.0-dev.301 | 2022-07-29 12:10:20 -0700

  * Add btest for vector bit-shift operators (Tim Wojtulewicz, Corelight)

  * Handle error cases for bit-shift operators more cleanly (Tim Wojtulewicz, Corelight)

5.1.0-dev.298 | 2022-07-29 12:00:57 -0700

  * Update 3rdparty submodule to get patricia reformat (Tim Wojtulewicz, Corelight)

5.1.0-dev.295 | 2022-07-28 11:21:59 -0700

  * Re-enable sending coverage data to Coveralls via Cirrus (Tim Wojtulewicz, Corelight)

    This was lost in 903f4bcc8ea2bc260f7ca9bf8d9fff445cb22f26

5.1.0-dev.292 | 2022-07-25 09:43:44 -0700

  * Return an error if redef'ing a variable without redef attribute (Tim Wojtulewicz, Corelight)

5.1.0-dev.289 | 2022-07-21 11:54:12 -0700

  * Add comments for specialized versions of util::split (Tim Wojtulewicz, Corelight)

  * Make char*/wchar_t* split methods use string_view/wstring_view (Tim Wojtulewicz, Corelight)

  * GH-1628: Return an error for duplicate record field names (Tim Wojtulewicz, Corelight)

  * Fix a bug with splitting strings in Ascii input reader (Tim Wojtulewicz, Corelight)

    The previous way of splitting strings would break if the last string in
    the line was an empty string, and it would return one fewer fields than
    it should have. This was breaking the last line in the
    scripts.base.framework.input.ascii.setspecialcases once the bug fixed in
    GH #1628 was fixed.

  * Iterator and looping cleanup in Ascii input reader (Tim Wojtulewicz, Corelight)

  * Add util::split methods for splitting strings (Tim Wojtulewicz, Corelight)

5.1.0-dev.282 | 2022-07-21 09:06:31 -0700

  * Add bitshift operators (AmazingPP)

5.1.0-dev.279 | 2022-07-19 15:21:10 +0200

  * GH-2267: Add btest that runs the doctest-based unit tests. (Robin
    Sommer, Corelight)

  * Fix local const variables can be modified via loops. (AmazingPP)

5.1.0-dev.273 | 2022-07-19 12:57:11 +0100

  * Remove use of fallible `get_conn_transport_proto` in `analyzer_violation`. (Benjamin Bannier, Corelight)

    When setting up the DPD info we previously would get the
    `transport_proto` for the connection with `get_conn_transport_proto`.
    This function takes a `conn_id` and would fail fatally if the connection
    for the given ID was unknown. It seems it was possible to run into such
    scenarios when the `analyzer_violation` event was processed after the
    connection had been cleaned up.

    We now get the `transport_proto` directly from the ports in the
    `connection` passed into `analyzer_violation` via
    `get_port_transport_proto` which cannot fail.

5.1.0-dev.271 | 2022-07-19 12:01:41 +0200

  * GH-2276: Raise smb2_file_delete for CREATE with
    FILE_DELETE_ON_CLOSE. (Arne Welzel, Corelight)

    When a CREATE request contains the FILE_DELETE_ON_CLOSE option and
    the subsequent CREATE response indicates success, we now raise the
    smb2_file_delete event to log a delete action in smb_files.log and
    also give users a way to handle this scenario.

5.1.0-dev.269 | 2022-07-18 08:57:32 -0700

  * DebugLogger: "auto const" to "const auto" (Arne Welzel, Corelight)

    This was pointed out in a separate PR so fixing the occurrence I
    had managed to sneak in already.

5.1.0-dev.267 | 2022-07-18 08:55:38 -0700

  * Fix some warnings from bro_version deprecation (Tim Wojtulewicz)

5.1.0-dev.265 | 2022-07-18 08:54:30 -0700

  * GH-636: Fix regex to handle commas at the end of config parser lines (Tim Wojtulewicz, Corelight)

  * Convert config framework to use std::regex (Tim Wojtulewicz, Corelight)

5.1.0-dev.261 | 2022-07-14 15:46:29 -0700

  * Management framework: fix an agent boot-time race condition plus minor tweaks (Christian Kreibich, Corelight)

      - bump cluster testsuite
      - undo the testsuite result mask
      - additional logging tweaks
      - additional context in a few log messages
      - await Supervisor peering before sending agent's hello
      - remove outdated comment

5.1.0-dev.254 | 2022-07-14 09:48:20 -0700

  * Add pow function (AmazingPP)

5.1.0-dev.252 | 2022-07-14 08:58:01 -0700

  * GH-2258: Add skip decorator to DNS unit tests (Tim Wojtulewicz, Corelight)

5.1.0-dev.250 | 2022-07-14 08:57:38 -0700

  * GH-2261: Fix type used for analyzer_confirmation/violation events (Tim Wojtulewicz, Corelight)

5.1.0-dev.248 | 2022-07-13 18:17:13 -0700

  * Fix requirement check in a logging framework / sqlite btest (Christian Kreibich, Corelight)

5.1.0-dev.245 | 2022-07-13 11:57:18 -0700

  * Stop signal-masking upon running unit tests (Christian Kreibich, Corelight)

  * Pause signal-masking during script parsing (Christian Kreibich, Corelight)

  * Add btests to verify Zeek's handling of SIGTERM and reading stdin (Christian Kreibich, Corelight)

  * Add procps/procps-ng to several CI Docker images (Christian Kreibich, Corelight)

5.1.0-dev.240 | 2022-07-13 11:23:38 -0700

  * Use clang-format for all files in `testing/btest/plugins`. (Benjamin Bannier, Corelight)

    This is a fixup commit for dc65b6248c017b947e60afa4992decab04772111.

5.1.0-dev.238 | 2022-07-13 11:11:48 -0700

  * GH-696: Add bytestring_to_float BIF (Tim Wojtulewicz, Corelight)

5.1.0-dev.236 | 2022-07-13 09:30:59 +0100

  * Avoid race conditions in broker.ssl-mismatch test (Dominik Charousset, Corelight)

  * Raw reader: Fix tail -F semantics when want_record=F and add tests for the new features.
    See GH-2266. (Craig Leres)

5.1.0-dev.229 | 2022-07-12 14:17:45 -0700

  * Rename bro_version and BRO_PLUGIN_BRO_VERSION to zeek names (Tim Wojtulewicz, Corelight)

  * Update binpac, bifcl, and gen-zam submodules for renaming changes (Tim Wojtulewicz, Corelight)

  * Deprecate BRO_PLUGIN_API_VERSION, create zeek::PLUGIN_API_VERSION (Tim Wojtulewicz, Corelight)

  * Deprecate HOOK_BRO_OBJ_DTOR and related methods (Tim Wojtulewicz, Corelight)

  * Miscellaneous deprecations and renaming (Tim Wojtulewicz, Corelight)

  * Rename all of bro lexer methods and such (Tim Wojtulewicz, Corelight)

  * Stop installing the bro symlink, it's been long enough (Tim Wojtulewicz, Corelight)

  * Deprecate bro_inet_ntop.{c,h} (Tim Wojtulewicz, Corelight)

  * Deprecate bro_int_t and bro_uint_t (Tim Wojtulewicz, Corelight)

5.1.0-dev.219 | 2022-07-12 12:00:10 -0700

  * Run clang-format on all of our plugin test c++ files (Tim Wojtulewicz, Corelight)

5.1.0-dev.217 | 2022-07-11 17:29:19 -0700

  * Fix race condition in scripts.base.utils.dir test (Christian Kreibich, Corelight)

  * Remove needless "touch" in scripts.base.utils.dir test (Christian Kreibich, Corelight)

5.1.0-dev.214 | 2022-07-11 09:46:52 -0700

  * Fix default build with `--disable-cpp-tests`. (Benjamin Bannier, Corelight)

    For a build configured with `--disable-cpp-tests` we globally set
    `-DDOCTEST_CONFIG_DISABLE`. Spicy does not expect this define to be used
    and its unit tests fail to build with it.

    This patch adds a Zeek-side workaround for that. We now do not build the
    Spicy unit test targets as part of `ALL` anymore.

    Closes #2238.

5.1.0-dev.212 | 2022-07-11 09:45:18 -0700

  * fix for script profiling of lambda functions (Vern Paxson, Corelight)

5.1.0-dev.210 | 2022-07-11 09:43:49 -0700

  * Fix for ZAM optimization of '?' conditional with vector-valued result (Vern Paxson, Corelight)

5.1.0-dev.208 | 2022-07-11 12:51:21 +0200

  * policy: Use literal dots for patterns used against `content-type`
    and `hostname`. (Arne Welzel, Corelight)

5.1.0-dev.204 | 2022-07-08 10:13:00 -0700

  * Fix usage of broker::convert(double,timespan) (Tim Wojtulewicz, Corelight)

5.1.0-dev.202 | 2022-07-08 09:55:35 -0700

  * Avoid calling safe_realloc unnecessarily from ODesc::Grow() (Tim Wojtulewicz, Corelight)

5.1.0-dev.200 | 2022-07-08 09:29:04 +0200

  * GH-913: zeek-setup: Validate plugin debug streams during startup.
    (Arne Welzel, Corelight)

5.1.0-dev.198 | 2022-07-07 14:19:23 -0700

  * Bump broker and zeek-archiver submodules to pull in CI updates (Christian Kreibich, Corelight)

  * CI updates  (Christian Kreibich, Corelight)

    - add FreeBSD 13.1 and fix broken link
    - Add OpenSUSE Tumbleweed
    - drop Ubuntu 21.10, EOL 2022-07-14
    - drop Debian 9, EOL 2022-06-30

5.1.0-dev.192 | 2022-07-07 07:51:04 +0200

  * sqlite default-logdir test: Remove ls ./logs baseline (Arne Welzel, Corelight)

    Observed .sqlite-journal files and missing reporter.sqlite files
    in CI runs. Subsequently reading the ./test.sqlite file is more
    reliable and should be good enough.

  * logging/sqlite: Recognize Log::default_logdir and place files there if set (Arne Welzel, Corelight)

  * logging: Introduce Log::default_logdir deprecate LogAscii::logdir and per writer logdir (Arne Welzel, Corelight)

    Also modify FormatRotationPath to keep rotated logs within
    Log::default_logdir unless the rotation function explicitly
    set dir, e.g. by when the user redef'ed default_rotation_interval.

  * logging/ascii: Fix .shadow paths when using LogAscii::logdir (Arne Welzel, Corelight)

    With the introduction of LogAscii::logdir, log filenames can now include
    parent directories rather than being plain basenames. Enabling log rotation,
    leftover log rotation and setting LogAscii::logdir broke due to not
    handling this situation.

5.1.0-dev.186 | 2022-07-05 22:18:52 +0000

  * Remove unnecssary includes of Dict.h (Tim Wojtulewicz, Corelight)

  * Turn PDict into a type alias (Tim Wojtulewicz, Corelight)

  * Convert Dictionary types to be templated classes (Tim Wojtulewicz, Corelight)

    This has the fortunate side-effect of also making it so we can store
    the value objects as typed pointers, instead of void*.

  * Fix signedness comparison warning in ssl analyzer (Tim Wojtulewicz, Corelight)

  * Management framework: make artifact filenames Github-compatible (Christian Kreibich, Corelight)

    The upload-artifact action refuses to process certain filenames:

      "Invalid characters include: Double quote ", Colon :, Less than <,
       Greater than >, Vertical bar |, Asterisk *, Question mark ?, Carriage
       return \r, Line feed \n"

    On failure, this iterates over the btest artifacts in the testsuite's .tmp
    directory and renames any of these to dots.

  * Management framework: mask testsuite result (Christian Kreibich, Corelight)

    There's currently a race condition in the agent-supervisor interaction that can
    cause requests to the Supervisor to get lost. This temporarily masks the
    testsuite result until I've had a chance to investigate.

5.1.0-dev.177 | 2022-07-05 15:03:12 +0100

  * SSL Analyzer: track connection direction by messages (Johanna Amann)

    This PR changes the way in which the SSL analyzer tracks the direction
    of connections. So far, the SSL analyzer assumed that the originator of
    a connection would send the client hello (and other associated
    client-side events), and that the responder would be the SSL servers.

    In some circumstances this is not true, and the initiator of a
    connection is the server, with the responder being the client. So far
    this confused some of the internal statekeeping logic and could lead to
    mis-parsing of extensions.

    This reversal of roles can happen in DTLS, if a connection uses STUN -
    and potentially in some StartTLS protocols.

    This PR tracks the direction of a TLS connection using the hello
    request, client hello and server hello handshake messages. Furthermore,
    it changes the SSL events from providing is_orig to providing is_client,
    where is_client is true for the client_side of a connection. Since the
    argument positioning in the event has not changed, old scripts will
    continue to work seamlessly - the new semantics are what everyone
    writing SSL scripts will have expected in any case.

    There is a new event that is raised when a connection is flipped. A
    weird is raised if a flip happens repeatedly.

    Addresses GH-2198.

5.1.0-dev.174 | 2022-07-05 10:26:14 +0200

  * Add `gulrak/filesystem` as a compiler-independent
    `std::filesystem` replacement. The `ghc::filesystem` namespace is
    exposed as `zeek::filesystem` in `util.h`. (Arne Welzel,
    Corelight)

5.1.0-dev.172 | 2022-07-04 10:22:45 +0100

  * The STREAM mode of the ASCII reader now behaves like `tail -F`: when file is
    removed/replaced, it will start tracking the new file. See
    https://github.com/zeek/zeek/pull/2097 for more detail. (Craig Leres)

  * Remove loops from Dict iterator invalidation unit test (Tim Wojtulewicz, Corelight)

    This fixes Coverity finding 1490366

5.1.0-dev.154 | 2022-07-01 14:10:58 -0700

  * Remove unused util::detail::rand64bit method (Tim Wojtulewicz, Corelight)

5.1.0-dev.151 | 2022-07-01 11:11:29 -0700

  * Management framework: bump zeek-client and cluster testsuite (Christian Kreibich, Corelight)

5.1.0-dev.149 | 2022-07-01 09:23:33 -0700

  * GH-2229: Fix some typos in weak-keys.zeek (Tim Wojtulewicz, Corelight)

5.1.0-dev.146 | 2022-06-30 13:38:33 -0700

  * Bump Spicy and spicy-plugin to their latest release. (Benjamin Bannier, Corelight)

5.1.0-dev.144 | 2022-06-30 13:37:40 -0700

  * Optimize 64 bit random number generation (Johanna Amann)

    rand64bit called random 4 times to generate one 64 bit number. There is
    no reason to do this - random() is basically guaranteed to return a 32
    bit number.

    This also adds a static check to make sure that it does.

5.1.0-dev.142 | 2022-06-30 12:27:42 -0700

  * Remove other general deprecations (Tim Wojtulewicz, Corelight)

  * Remove some deprecated ocsp/ssl base scripts (Tim Wojtulewicz, Corelight)

  * Remove deprecated DESC_PORTABLE ODesc mode and ODesc::IsPortable() (Tim Wojtulewicz, Corelight)

  * Remove deprecated UDP, ICMP, and TCP analyzers (Tim Wojtulewicz, Corelight)

  * Remove deprecated protocol_confirmation/violation events and methods (Tim Wojtulewicz, Corelight)

  * Remove deprecated IterCookie-based dictionary iteration (Tim Wojtulewicz, Corelight)

  * Remove deprecated MemoryAllocation() methods and related code (Tim Wojtulewicz, Corelight)

  * Remove deprecated --enable-mobile-ipv6 configure argument (Tim Wojtulewicz, Corelight)

5.1.0-dev.133 | 2022-06-30 09:56:20 -0700

  * Add helpers for syncing commit files with external testsuites (Christian Kreibich, Corelight)

  * Fix typo in update-timing target for external testsuites (Christian Kreibich, Corelight)

  * Update broker submodule [nomail] (Tim Wojtulewicz, Corelight)

5.1.0-dev.128 | 2022-06-27 13:04:47 -0700

  * removed deprecated capture-by-reference closures (Vern Paxson, Corelight)

5.1.0-dev.126 | 2022-06-27 11:43:27 -0700

  * GH-2183: Rework Packet checksummed variable naming (Tim Wojtulewicz, Corelight)

5.1.0-dev.124 | 2022-06-27 10:37:06 -0700

  * Code modernization: Convert from deprecated C standard library headers (Tim Wojtulewicz)

5.1.0-dev.121 | 2022-06-27 09:26:54 -0700

  * Mark bools in BasicThread as atomic to avoid data races (Tim Wojtulewicz)

  * Avoid calling DBG_LOG during signal handling (Tim Wojtulewicz)

  * Fixes for iosource::Manager for deadlocks during shutdown (Tim Wojtulewicz, Corelight)

5.1.0-dev.116 | 2022-06-24 14:18:40 -0700

  * Bump broker submodule to pick up two fixes (Tim Wojtulewicz, Corelight)

    - https://github.com/zeek/broker/pull/244
    - https://github.com/zeek/broker/pull/245

  * Add NEWS entry for 5.0 management changes [nomail] [skip ci] (Tim Wojtulewicz, Corelight)

5.1.0-dev.113 | 2022-06-24 09:08:00 -0700

  * News entry about Bloom Filter changes in 5.0. (Johanna Amann, Corelight)

5.1.0-dev.110 | 2022-06-23 13:03:55 -0700

  * Bump Spicy to the latest development version. (Benjamin Bannier, Corelight)

5.1.0-dev.108 | 2022-06-23 12:26:26 -0700

  * Management framework: support for cluster node restarts (Christian Kreibich, Corelight)

    - bump external cluster testsuite
    - bump zeek-client
    - edit pass over docstrings
    - node restart support
    - more consistent Supervisor interaction in the agent
    - log the controller's startup deployment attempt
    - bugfix for a get_id_value corner case
    - minor timeout bugfix
    - make "result" argument plural in multi-result response events

5.1.0-dev.98 | 2022-06-22 22:39:32 -0700

  * Management framework: separate config staging and deployment (Christian Kreibich, Corelight)

    - bump external cluster testsuite
    - bump zeek-client
    - rename set_configuration events to stage_configuration
    - trigger deployment upon when instances are ready
    - more resilient node shutdown upon deployment
    - re-trigger deployment upon controller launch
    - move most deployment handling to internal function
    - distinguish internally and externally requested deployments
    - track instances by their Broker IDs
    - tweak Supervisor event logging
    - make helper function a local
    - rename "log_level" to "level"
    - add "finish" callback to requests
    - add a helper for rendering result vectors to a string
    - agents now skip re-deployment of current config
    - suppress notify_agent_hello upon Supervisor peering
    - introduce state machine for configs and persist them
    - introduce deployment API in controller
    - rename agent "set_configuration" to "deploy"
    - consistency fixes to the Result record

5.1.0-dev.75 | 2022-06-22 12:06:00 -0700

  * Provide zeek-client by default (Christian Kreibich, Corelight)

5.1.0-dev.73 | 2022-06-22 09:40:05 -0700

  * GH-1991: Add option to limit the number of tunnel_changed events (Tim Wojtulewicz, Corelight)

5.1.0-dev.71 | 2022-06-21 16:58:25 -0700

  * Management framework: config validation (Christian Kreibich, Corelight)

    - bump external cluster testsuite
    - bump zeek-client
    - add config validation
    - improvements to port auto-enumeration
    - control output-to-console in Supervisor
    - handle no-instances corner case in set-config correctly

5.1.0-dev.64 | 2022-06-21 12:23:20 -0700

  * CI: Add Fedora 36, remove Fedora 34 (Johanna Amann, Corelight)

  * Update broker submodule to fix put_unique issues from 5.0RC1 (Tim Wojtulewicz, Corelight)

5.1.0-dev.61 | 2022-06-17 09:21:44 -0700

  * CI: Add OpenSuSE Leap 15.4, fix 15.3 (Johanna Amann, Corelight)

    15.3 included the wrong update repo in the Dockerfile

5.1.0-dev.59 | 2022-06-17 09:09:58 -0700

  * Bump Spicy and spicy-plugin to latest dev snapshot. (Benjamin Bannier, Corelight)

    This bump brings in fixes for the following issues:

    - zeek/spicy#1204
    - zeek/spicy#1208
    - zeek/spicy#1210
    - zeek/spicy-plugin#112
    - zeek/spicy-plugin#117

5.1.0-dev.56 | 2022-06-16 15:57:19 -0700

  * flip connections that begin with SYN-ACKs if subsequent acks or data seen (Vern Paxson, Corelight)

5.1.0-dev.53 | 2022-06-16 14:27:54 -0700

  * Revert "Merge remote-tracking branch 'turrisxyz/Dependabot-GitHub-Actions'" (Tim Wojtulewicz, Corelight)

    This reverts commit 910e3b302406b838c1c12e016d4a05306a3bad46, reversing
    changes made to 64198655204bee20cc6b933d07909bd30869d7e8.

5.1.0-dev.52 | 2022-06-16 11:11:33 -0700

  * Management framework: bump external cluster testsuite (Christian Kreibich, Corelight)

  * Management framework: make agents support zeek-archiver invocations (Christian Kreibich, Corelight)

  * Management framework: fix module naming typo (Christian Kreibich, Corelight)

5.1.0-dev.48 | 2022-06-16 10:08:14 -0700

  * Use ccache for docs gen github workflow (Tim Wojtulewicz, Corelight)

  * CI: Add Ubuntu 22.04 (LTS Release, EOL 04/2027) (Johanna Amann, Corelight)

5.1.0-dev.38 | 2022-06-14 15:43:44 -0700

  * Fix pattern matching in Cirrus dependabot check (Tim Wojtulewicz, Corelight)

5.1.0-dev.35 | 2022-06-14 14:39:02 -0700

  * Ignore dependabot PRs on Cirrus (Tim Wojtulewicz, Corelight)

5.1.0-dev.33 | 2022-06-14 14:34:31 -0700

  * Merge PQ_Timer into base TimerMgr class (Tim Wojtulewicz, Corelight)

5.1.0-dev.30 | 2022-06-14 12:02:46 -0700

  * Management framework: bump external cluster testsuite (Christian Kreibich, Corelight)

  * Management framework: switch default network visibilities (Christian Kreibich, Corelight)

5.1.0-dev.27 | 2022-06-14 11:30:28 -0700

  * chore: Set permissions for GitHub actions (naveen)

     Restrict the GitHub token permissions only to the required ones; this way, even if the attackers will succeed in compromising your workflow, they won’t be able to do much.

    - Included permissions for the action. https://github.com/ossf/scorecard/blob/main/docs/checks.md#token-permissions

    https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#permissions

    https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs

    [Keeping your GitHub Actions and workflows secure Part 1: Preventing pwn requests](https://securitylab.github.com/research/github-actions-preventing-pwn-requests/)

    Signed-off-by: naveen <172697+naveensrinivasan@users.noreply.github.com>

5.1.0-dev.25 | 2022-06-14 10:07:32 -0700

  * chore: Included githubactions in the dependabot config (naveen)

    This should help with keeping the GitHub actions updated on new releases. This will also help with keeping it secure.

    Dependabot helps in keeping the supply chain secure https://docs.github.com/en/code-security/dependabot

    GitHub actions up to date https://docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot

    https://github.com/ossf/scorecard/blob/main/docs/checks.md#dependency-update-tool
    Signed-off-by: naveen <172697+naveensrinivasan@users.noreply.github.com>

  * Sort package names in Dockerfiles (Tim Wojtulewicz, Corelight)

  * Dockerfile changes for consistency with Broker files (Tim Wojtulewicz, Corelight)

  * Make some of the CI Dockerfiles more consistent (Tim Wojtulewicz, Corelight)

5.1.0-dev.16 | 2022-06-08 13:42:41 -0700

  * Management framework: port auto-assignment (Christian Kreibich, Corelight)

    - support auto-assignment of ports in cluster nodes
    - also use send_set_configuration_response_error elsewhere
    - minor log formatting tweak, for consistency
    - bump zeek-client to pull in relaxed port handling
    - bump external cluster testsuite

5.1.0-dev.10 | 2022-06-07 10:02:39 -0700

  * Keep make dist from deleting all paths containing 'build' [skip ci] (Tim Wojtulewicz)

5.1.0-dev.8 | 2022-06-06 09:38:59 -0700

  * Remove duplicate entry about websockets from 5.0 NEWS (Tim Wojtulewicz, Corelight)

5.1.0-dev.7 | 2022-06-06 09:36:32 -0700

  * NEWS updates for 5.0 (Vern Paxson, Corelight)

  * ZAM test suite updates due to recent changes (Vern Paxson, Corelight)

5.1.0-dev.1 | 2022-06-03 12:38:20 -0700

  * Update local-compat test for 5.1 (Tim Wojtulewicz, Corelight)

  * Start of 5.1.0 development (Tim Wojtulewicz, Corelight)

5.0.0-dev.575 | 2022-06-03 10:55:47 -0700

  * Management framework: bump external cluster testsuite (Christian Kreibich, Corelight)

  * Management framework: bump zeek-client to bring back to master (Christian Kreibich, Corelight)

5.0.0-dev.572 | 2022-06-03 10:31:51 -0700

  * Fix hang in DNS analyzer with malformed packet (Tim Wojtulewicz, Corelight)

    (cherry picked from commit dfc9f77b68f6d02d059ade322b9c132e32c26872)

5.0.0-dev.571 | 2022-06-03 09:50:45 -0700

  * Additional &is_used tags in the Netcontrol and Sumstats frameworks (Christian Kreibich, Corelight)

    When running a cluster, these functions only get called in select node types and
    could trigger no-caller warnings on stderr.

5.0.0-dev.569 | 2022-06-03 09:50:01 -0700

  * Management framework: bump zeek-client to pull in rendering tweaks (Christian Kreibich, Corelight)

  * Management framework: bump external cluster testsuite (Christian Kreibich, Corelight)

  * Mark lookup_asn() BIF as deprecated in v6.1 (Phil Rzewski)

  * Define geo_autonomous_system record type (Phil Rzewski)

  * Add lookup_autonomous_system() BIF that returns AS number and org (Phil Rzewski)

5.0.0-dev.559 | 2022-06-02 16:58:58 -0700

  * Mark lookup_asn() BIF as deprecated in v6.1 (Phil Rzewski)

  * Define geo_autonomous_system record type (Phil Rzewski)

  * Add lookup_autonomous_system() BIF that returns AS number and org (Phil Rzewski)

5.0.0-dev.553 | 2022-06-02 13:16:44 -0700

  * Make broker.web-socket-events test require the Python websockets package (Christian Kreibich, Corelight)

5.0.0-dev.551 | 2022-06-02 12:19:32 -0700

  * Expand scripts.base.frameworks.intel.cluster-transparency test (Christian Kreibich, Corelight)

  * Fix races in scripts.base.frameworks.intel.cluster-transparency-with-proxy test (Christian Kreibich, Corelight)

  * Add Intel::send_store_on_node_up boolean to control min_data_store delivery (Christian Kreibich, Corelight)

  * Fix a -Wstringop-overflow memset size warning with GCC 12 (Christian Kreibich, Corelight)

5.0.0-dev.545 | 2022-06-02 12:00:53 +0200

  * Add WebSocket support for exchanging events with external clients.
    (Robin Sommer, Corelight)

    This exposes Broker's new WebSocket support in Zeek. To enable it,
    call `Broker::listen_websocket()`. Zeek will then start listening on
    port 9997 for incoming WebSocket connections.

    See the Broker documentation for a description of the message format
    expected over these WebSocket connections.

5.0.0-dev.540 | 2022-06-01 11:08:42 -0700

  * GH-2136: Clean up DNS_Mgr before shutting down (Tim Wojtulewicz, Corelight)

5.0.0-dev.538 | 2022-06-01 09:20:21 -0700

  * GH-2101: Update cmake submodule to fix c-ares include path problem (Tim Wojtulewicz, Corelight)

5.0.0-dev.535 | 2022-05-31 12:58:32 -0700

  * Management framework updates (Christian Kreibich, Corelight)

    - bump external cluster testsuite
    - bump zeek-client to pull in set-config rendering
    - enable stdout/stderr reporting
    - Supervisor extensions for stdout/stderr handling
    - disambiguate redef field names in agent and controller
    - move to ResultVec in agent's set_configuration response
    - tune request timeout granularity and interval
    - verify node starts when deploying a configuration
    - a bit of debug-level logging for troubleshooting

5.0.0-dev.525 | 2022-05-31 12:53:01 -0700

  * Add Supervisor::node_status notification event (Christian Kreibich, Corelight)

5.0.0-dev.523 | 2022-05-31 11:13:47 -0700

  * Update spicy-plugin with change that checks for zeek version (Tim Wojtulewicz, Corelight)

  * deprecation messages for unused base script functions (Vern Paxson, Corelight)

  * clearer messages for warning about unused functions (Vern Paxson, Corelight)

  * annotate orphan base script components with &deprecated (Vern Paxson, Corelight)

  * annotate base scripts with &is_used as needed (Vern Paxson, Corelight)

  * --no-usage-warnings flag to suppress analysis (Vern Paxson, Corelight)

  * support for associating &is_used attributes with functions (Vern Paxson, Corelight)

  * classes for evaluating function/hook/event usage (Vern Paxson, Corelight)

  * broader support for AST traversal, including Attr and Attributes objects (Vern Paxson, Corelight)

  * include attributes in descriptions of sets and tables (Vern Paxson, Corelight)

5.0.0-dev.508 | 2022-05-27 14:33:47 -0700

  * Update zeek-aux submodule to fix a compiler warning (Tim Wojtulewicz, Corelight)

5.0.0-dev.505 | 2022-05-26 16:08:42 -0700

  * Management framework updates (Christian Kreibich, Corelight)

    - bump zeek-client to pull in instance serialization fixes
    - bump external cluster testsuite
    - update agent-checkin test to reflect recent changes
    - place each Zeek process in its own working dir
    - set defaults for log rotation and persistent state
    - add spool and state directory config settings
    - establish stdout/stderr files also for cluster nodes
    - default to having agents check in with the (local) controller
    - move role variable from logging into framework-wide config
    - distinguish supervisor/supervisee when loading agent/controller
    - simplify agent and controller stdout/stderr files
    - prefix the management logs with "management-"
    - comment and layouting tweaks, no functional change
    - rename env var that labels agents/controllers
    - increase robustness of agent/controller naming

  * Add some missing NEWS entries (Tim Wojtulewicz, Corelight)

5.0.0-dev.488 | 2022-05-26 08:23:42 -0700

  * GH-2054: Allow nulls as separators for join_string_vec (Tim Wojtulewicz, Corelight)

5.0.0-dev.485 | 2022-05-25 12:09:42 -0700

  * Propagate BPF_Program error message to script land (Tim Wojtulewicz, Corelight)

  * Allow pcap pktsrc to use other BPF_Program::Compile method (Tim Wojtulewicz, Corelight)

5.0.0-dev.482 | 2022-05-25 09:40:18 -0700

  * Bump Spicy and spicy-plugin. (Benjamin Bannier, Corelight)

  * Clean up test `spicy.spicyz-jit`. (Benjamin Bannier, Corelight)

    This test was created from a copy of `spicy.spicyz-aot`, but updated
    incompletly. This patch removes the last ahead-of-time compilation bits.

5.0.0-dev.479 | 2022-05-25 09:38:20 -0700

  * GH-2125: Fix compiler warnings in ScriptProfile (Tim Wojtulewicz, Corelight)

5.0.0-dev.477 | 2022-05-25 09:37:58 -0700

  * Use ZEEK_DICT_DEBUG more in Dict.cc instead of DEBUG (Tim Wojtulewicz)

5.0.0-dev.475 | 2022-05-25 09:37:19 -0700

  * Move fuzzer setup in CMake below builtin plugin setup (Tim Wojtulewicz, Corelight)

  * Remove flex/bison flags from CI build (Tim Wojtulewicz, Corelight)

5.0.0-dev.470 | 2022-05-24 11:55:37 -0700

  * Ensure presence of Zeek-related directories in toplevel CMakeLists.txt (Christian Kreibich, Corelight)

  * Add base/misc/installation.zeek, with Zeek installation directories (Christian Kreibich, Corelight)

  * Add scripts.base.misc.installation btest (Christian Kreibich, Corelight)

5.0.0-dev.466 | 2022-05-23 17:47:08 -0700

  * Add DOCKERFILE_VERSION variable to all CI dockerfiles (Tim Wojtulewicz, Corelight)

5.0.0-dev.462 | 2022-05-19 11:45:38 -0700

  * speed up ZAM compilation by capping function size when inlining (Vern Paxson, Corelight)

5.0.0-dev.460 | 2022-05-19 11:24:50 -0700

  * Zeekify the scripts.base.utils.paths test (Christian Kreibich, Corelight)

  * Expand build_path() function to handle empty dir arguments gracefully (Christian Kreibich, Corelight)

5.0.0-dev.457 | 2022-05-19 10:51:45 -0700

  * Rework .cirrus.yml a bit to commonize some parts (Tim Wojtulewicz)

  * Enable 'greedy' mode for other CI builds (Tim Wojtulewicz, Corelight)

  * Enable ccache for Cirrus builds (Tim Wojtulewicz, Corelight)

  * Fix SSL tests on Centos Stream 9 (Johanna Amann, Corelight)

    SHA1 signatures are only allowed in the legacy policy. I have no idea
    where this is documented - I found it in a patch for the OpenSSL source
    package.

    Also adding tar as a specific dependency, since we have a test that
    requires it, and at least for me it was not present in a minimal
    installation.

    Fixes #2115

5.0.0-dev.451 | 2022-05-19 09:13:13 -0700

  * Disable Spicy in Zeek Coverity builds (Tim Wojtulewicz, Corelight)

5.0.0-dev.449 | 2022-05-18 11:11:58 -0700

  * Always generate helper scripts for Spicy. (Benjamin Bannier, Corelight)

    We previously only generated these files if configuring a build which
    included Spicy. This meant that e.g., `zeek-path-dev.*` scripts would
    emit errors for non-existing paths if Spicy was not included.

    We now always generate the helper scripts so we can reference them
    regardless of build configuration. If Spicy is not included the files
    will still be sourced, but have no actual effect.

5.0.0-dev.447 | 2022-05-18 11:07:30 -0700

  * Partially revert OpenSSL hashing changes (Johanna Amann, Corelight)

    This commit partially reverts the changes that we made in
    6217851d6db3859b2add34773be5a6b3ecba49f0. It turns out that reverting to
    the legacy OpenSSL API means that Zeek can no longer be run in FIPS mode
    for old (but still used) versions of OpenSSL.

    This commit thus uses a bunch of #ifdefs to support both implementations
    simultaneously.

5.0.0-dev.445 | 2022-05-18 11:07:02 -0700

  * GH-1672: Keep a reference to a table being expired, just in case (Tim Wojtulewicz, Corelight)

5.0.0-dev.442 | 2022-05-17 09:15:27 -0700

  * Remove unnecessary uses of util::fmt() (Tim Wojtulewicz, Corelight)

5.0.0-dev.440 | 2022-05-17 08:50:15 -0700

  * Add NEWS item for new requirements due to default-enabled Spicy support. (Benjamin Bannier, Corelight)

  * Include headers in binary dir before other headers. (Benjamin Bannier, Corelight)

    If e.g., optional dependencies are installed in the same prefix as a
    previous Zeek installation, the build would have previously found
    `zeek-config.h` from the external prefix instead of from the current
    build.

    With this patch we now include the current binary dir before any
    previously configured include directories (here in particular: optional
    dependencies). With that the correct `zeek-config.h` is found.

  * Increase memory in CI. (Benjamin Bannier, Corelight)

    Spicy requires substantial amounts of RAM to compile. In this patch we
    bump the amount of RAM available in CI so a build including Spicy can be
    performed.

  * Add tests for bundled Spicy infrastructure. (Benjamin Bannier, Corelight)

  * Include spicy in build. (Benjamin Bannier, Corelight)

  * Add `bare_mode` bif. (Benjamin Bannier, Corelight)

  * Change test so included plugins can load notice framework. (Benjamin Bannier, Corelight)

  * Do not log function arguments in test. (Benjamin Bannier, Corelight)

    This was excercising hooks to validate that they were called. For that
    it uses pretty verbose logging. Since version numbers are not stable and
    hard to canonify this script already unloads the version module (there
    is e.g., code in `base/misc/version` which splits the version strings
    and works on single components which might appear like pretty ordinary
    floating point numbers in the log).

    This test however worked under the assumption that nothing in bare mode
    loads `base/misc/version` which is hard to guarantee, especially
    considering that plugins can be embedded and might load that module
    themself.

    With this patch we now make the logging slightly less verbose so that
    function call arguments are not logged anymore.

5.0.0-dev.431 | 2022-05-16 17:00:24 -0700

  * Store location during Ascii input parsing for use in error messages (Tim Wojtulewicz, Corelight)

  * Include file information in input reader error messages (Tim Wojtulewicz, Corelight)

5.0.0-dev.427 | 2022-05-16 16:59:31 -0700

  * NEWS items for script profiling, "footprint" BiFs, and Trigger method deprecations (Vern Paxson, Corelight)

  * make deprecation advice a bit more helpful (Vern Paxson, Corelight)

  * add deprecation for Trigger Cache/Lookup interfaces (Vern Paxson, Corelight)

  * helper scripts for -O C++ maintenance (Vern Paxson, Corelight)

  * "-O use-C++" now fails if no C++ bodies wound up being applicable (Vern Paxson, Corelight)

  * update environemnt variable names to match command-line arguments (Vern Paxson, Corelight)

  * final changes for -O C++ feature completeness, mainly "when" statement support (Vern Paxson, Corelight)

  * Change "when" statements that don't require closures to use new implementation. (Vern Paxson, Corelight)
    Provide hooks for script optimization access to "when" statements.
    Regularize treatment of naming and timeouts for Triggers.

  * switch cached Trigger values to be opaque-and-generic, rather than assuming CallExpr's (Vern Paxson, Corelight)

  * tracking of when statements/expressions occur in a "when" context (Vern Paxson, Corelight)

  * fix bug that sometimes associated the wrong location with global statements (Vern Paxson, Corelight)

  * fix incomplete AST traversal of "when" statements (Vern Paxson, Corelight)

  * add some missing "when" closures (Vern Paxson, Corelight)

  * regularize test suite names to avoid custom/outdated suffices (Vern Paxson, Corelight)

5.0.0-dev.410 | 2022-05-16 13:14:31 -0700

  * Call pip using python -m flag during FreeBSD CI prep (Tim Wojtulewicz, Corelight)

5.0.0-dev.408 | 2022-05-16 09:10:53 -0700

  * Store hash analyzer kind StringVals for later reuse (Tim Wojtulewicz, Corelight)

  * Add Component::GetCanonicalNameVal to avoid extra StringVal allocations (Tim Wojtulewicz, Corelight)

  * Avoid type unnecessary type conversion in ComponentManager::GetComponentName (Tim Wojtulewicz, Corelight)

  * Only set connection history if strings differ (Tim Wojtulewicz, Corelight)

5.0.0-dev.403 | 2022-05-16 09:10:15 -0700

  * Explicitly define copy and move operators/constructors for ScriptProfileStats (Tim Wojtulewicz, Corelight)

    Fixes Coverity 1488768

5.0.0-dev.401 | 2022-05-16 09:09:37 -0700

  * Add --retry to curl invocations in update-traces (Tim Wojtulewicz, Corelight)

5.0.0-dev.399 | 2022-05-16 09:09:05 -0700

  * GH-2103: Disable leak detection for bifs.footprint in another way (Tim Wojtulewicz, Corelight)

5.0.0-dev.396 | 2022-05-13 16:50:53 -0700

  * Fix UB during early shutdown on OpenSSL state (Dominik Charousset, Corelight)

5.0.0-dev.393 | 2022-05-13 16:48:56 -0700

  * Correcly typify bloom filters in all cases during merge (Johanna Amann)

    Before, there were edge-cases where the type was not transferred if one
    of the involved bloom-filters was untypified.

    Fixes GH-1938

  * Add intersect operation for bloom filters (Johanna Amann)

    Intersecting two bloom filters yields a bloom filter that returns true
    when an element was contained in both bloom filters. The false positive
    rate is potentially a bit higher than in the original bloom filters.

    This operation also works for counting bloom filters, however the
    counters are discarded and the bloomfilters are converted to basic bloom
    filters. The reason is that there is no obvious meaning to the counters
    when two bloom filters are intersected - besides the fact if an element
    was inserted at all.

  * Add bloomfilter_decrement bif (Johanna Amann)

    This bif implements the decrement operation for counting bloom filters.

    It also clarifies some of the documentation.

5.0.0-dev.385 | 2022-05-12 11:03:25 +0100

  * Include certificate information in SSL::Weak_Key notice (Johanna Amann, Corelight)

5.0.0-dev.382 | 2022-05-11 12:56:13 -0700

  * tidy up after generating profile (Vern Paxson, Corelight)

  * fix for coverage reporting for functions that use "when" statements (Vern Paxson, Corelight)

  * new global_options() BiF to speed up startup, plus a micro-preen (Vern Paxson, Corelight)

  * hooks for new --profile-scripts option (Vern Paxson, Corelight)

  * classes for managing script profiles (Vern Paxson, Corelight)

  * address some holes in script coverage (Vern Paxson, Corelight)

  * fix for script coverage missing on-exit activity (Vern Paxson, Corelight)

  * memory management fixes for loggers (Vern Paxson, Corelight)

  * make curr_CPU_time() broadly available rather than just isolated to ZAM (Vern Paxson, Corelight)

  * Add exact name of the Top-k algorithm. (Johanna Amann)

    I needed to figure out which exact algorithm we use for our
    probabilistic top-k measurements. It turns out that we do not mention
    this in our source tree at all so far.

5.0.0-dev.368 | 2022-05-09 10:37:44 +0100

  * when printing a description of a type, use its name if available (Vern Paxson, Corelight)

5.0.0-dev.363 | 2022-05-06 15:28:34 -0700

  * Management framework: consistency tweaks in zeek-client, plus test updates (Christian Kreibich, Corelight)

5.0.0-dev.361 | 2022-05-06 12:47:49 -0700

  * Update doctest to v2.4.8 (Tim Wojtulewicz, Corelight)

5.0.0-dev.359 | 2022-05-06 11:32:13 -0700

  * to avoid recursion, track all aggregates, not just records (Vern Paxson, Corelight)
    isolate the internal methods

  * simpler public calling interface for computing footprint (Vern Paxson, Corelight)

  * use stack-based set to prevent infinite recursion rather than a static one (Vern Paxson, Corelight)

  * change value_footprint() to val_footprint() to be more similar to val_size() (Vern Paxson, Corelight)

  * make including count of container elements non-optional (Vern Paxson, Corelight)

  * fix for tracking footprints of mutually-recursive records (Vern Paxson, Corelight)

  * added value_footprint() and global_container_footprints() BiFs (Vern Paxson, Corelight)

5.0.0-dev.347 | 2022-05-05 18:09:44 -0700

  * Management framework: add get_configuration_request/response transaction (Christian Kreibich, Corelight)

  * Update zeek-archiver submodule (Christian Kreibich, Corelight)

5.0.0-dev.344 | 2022-05-04 11:57:50 -0700

  * fix for coverage reporting for functions that use "when" statements (Vern Paxson, Corelight)

  * ZAM maintenance for recent changes and some newly exercised corner cases (Vern Paxson, Corelight)

5.0.0-dev.341 | 2022-05-04 09:23:55 -0700

  * Fixes Coverity issues reported from DNS_Mgr merge (Tim Wojtulewicz, Corelight)

    Reports fixed: 1488295, 1488292, 1488281, 1488279

5.0.0-dev.338 | 2022-05-03 14:40:57 -0700

  * fix for sets containing recursive records (Vern Paxson, Corelight)

5.0.0-dev.336 | 2022-05-03 17:09:54 +0100

  * Update X.509 and CT trust anchors (Johanna Amann, Corelight)

5.0.0-dev.332 | 2022-04-28 19:52:04 +0000

  * Initialize OpenSSL on startup (Dominik Charousset, Corelight)

  * Avoid double-initialization of OpenSSL (Dominik Charousset, Corelight)

  * Canonify intel.log in read-file-dist-cluster test (Dominik Charousset, Corelight)

  * Port Zeek to latest Broker API (Dominik Charousset)

5.0.0-dev.322 | 2022-04-27 21:00:29 +0000

  * Disable OpenSSL initialization starting with 1.1.0 (Johanna Amann, Corelight)

    Starting with OpenSSL 1.1.0, library initialization is no longer
    required - and might even be harmful.

    See https://wiki.openssl.org/index.php/Library_Initialization for
    details.

5.0.0-dev.319 | 2022-04-27 17:42:42 +0000

  * Wrap call to doctest's MESSAGE() method in Reporter in try/catch block (Tim Wojtulewicz, Corelight)

    Also check whether doctest is even enabled before trying to use it.

  * Pre-initialize c-ares channel object. Fixes Coverity 1488318 (Tim Wojtulewicz, Corelight)

  * Ask c-ares for the next timeout instead of passing a fixed value (Tim Wojtulewicz, Corelight)

  * Remove obsolete DNS_Mgr::asyncs_timeouts (Tim Wojtulewicz, Corelight)

5.0.0-dev.314 | 2022-04-27 09:43:23 -0700

  * Management framework: consistency fixes around event() vs Broker::publish() (Christian Kreibich, Corelight)

5.0.0-dev.312 | 2022-04-26 09:52:34 -0700

  * Rework FindCAres.cmake to not use ExternalProject, fixing OBS builds (Tim Wojtulewicz, Corelight)

5.0.0-dev.309 | 2022-04-22 13:11:12 -0700

  * Add DNS fuzzing corpus from c-ares (Tim Wojtulewicz, Corelight)

  * Set larger UDP buffer to avoid TCP fallback if possible (Tim Wojtulewicz, Corelight)

    This commit sets the UDP buffer to a larger size, as well as adds
    an EDNS block to the DNS request passing this size. This allows
    DNS servers to return larger responses, and in turn allow c-ares
    to avoid TCP fallback due to requests failing because of the lack
    of buffer size.

  * Add new features to IOSource::Manager, used by DNS_Mgr (Tim Wojtulewicz, Corelight)

    - iosource_mgr can now track write events to file descriptors as well
      as read events. This adds an argument to both RegisterFd() and
      UnregisterFd() for setting the mode, defaulting to read.
    - IOSources can now implement a ProcessFd() method that allows them to
      handle events to single file descriptors instead of of having to
      loop through/track sets of them at processing time.

  * Add out_ptr, use for c-ares interface calls (Tim Wojtulewicz, Corelight)

  * Store all mappings in a single map instead of split by type (Tim Wojtulewicz, Corelight)

    This opens up the possibility of storing other request types outside
    of T_A, T_PTR and T_TXT without requiring redoing the caching. It
    also fixes the caching code in DNS_Mapping, adding a version number
    to the start of the cache file so the cache structure can be modified
    and old caches invalidated more easily.

  * Add merging to DNS_Mgr::AddResult() to support both ipv4 and ipv6 responses simultaneously (Tim Wojtulewicz, Corelight)

  * Rework DNS_Mgr API to be more consistent and to support more request types (Tim Wojtulewicz, Corelight)

  * Replace nb_dns library with C-Ares (Tim Wojtulewicz, Corelight)

  * Add unit testing for DNS_Mgr and related classes (Tim Wojtulewicz, Corelight)

  * Update doc gen VM to ubuntu-latest, output cmake version during configure (Tim Wojtulewicz, Corelight)

  * Use doctest macro to tie Reporter output to test cases (Tim Wojtulewicz, Corelight)

  * Add const versions of dereference operators for DictEntry (Tim Wojtulewicz, Corelight)

  * Add DNS fuzzer (Tim Wojtulewicz, Corelight)

5.0.0-dev.288 | 2022-04-22 07:00:56 -0700

  * Fix generate-docs action for running on forks. (Benjamin Bannier, Corelight)

    The generate-docs action previously always required secrets to run so
    that it could possibly perform a push (if run from a schedule), and to
    send out an email on failure. Since secrets are unavailable for forks
    this meant that this action would always fail for PRs from forks.

    In this patch we use an unauthenticated clone unless running from a
    schedule. This is fine as for PRs this action would just regenerate the
    docs to check for errors, but not to actually update them (no push
    performed). We also change the failure notification step to only execute
    for scheduled runs.

5.0.0-dev.286 | 2022-04-21 13:34:34 -0700

  * Suppress progress dots in zkg's output in Docker package-install check (Christian Kreibich, Corelight)

5.0.0-dev.284 | 2022-04-21 09:17:28 -0700

  * Enable vptr undefined behavior check (Tim Wojtulewicz, Corelight)

5.0.0-dev.282 | 2022-04-20 17:17:55 -0700

  * Update libkqueue for Coverity and build warning fixes (Tim Wojtulewicz, Corelight)

5.0.0-dev.280 | 2022-04-19 09:42:28 -0700

  * Escape special characters in paths before using them as regexes (Tim Wojtulewicz, Corelight)

5.0.0-dev.277 | 2022-04-18 16:38:27 -0700

  * Management framework updates (Christian Kreibich, Corelight)

    - bump external testsuite
    - allow selecting cluster nodes in get_id_value
    - minor tweaks to logging component
    - bump zeek-client to pull in get-id-value command
    - improve handling of node run states
    - add get_id_value dispatch
    - allow dispatching "actions" on cluster nodes.
    - some renaming to avoid the term "data cluster"
    - allow agents to communicate with cluster nodes

  * Avoid whitespace around function type strings in JSON rendering (Christian Kreibich, Corelight)

  * Disable TSan CI task temporarily while we sort out some intermittent test failures (Tim Wojtulewicz, Corelight)

5.0.0-dev.265 | 2022-04-18 12:45:08 -0700

  * state-holding fix: track unique identifiers for Func's in CompHash's, not Func's themselves (Vern Paxson, Corelight)

5.0.0-dev.263 | 2022-04-18 09:22:30 -0700

  * Add "Reporter" entry to fix plugin hook_name() vs HookType imbalance (Christian Kreibich, Corelight)

    The hook_name() list was missing an entry corresponding to HOOK_REPORTER.

    Co-authored-by: Peter Cullen <peter.cullen@corelight.com>

5.0.0-dev.259 | 2022-04-14 10:26:29 -0700

  * GH-2038: Don't sleep when non-selectable PktSrc has data available (Anthony Coddington)

    PktSrc::GetNextTimeout always returned a fixed timeout of 20 microseconds for non-selectable packet sources regardless of whether they have packets available. This adds unnecessary delay every FindReadySources poll_interval when packets are available to be read.

    Instead, for non-selectable packet sources, check whether packets are available and return a timeout of 0 to indicate data is available. This is closer to the behaviour of the old capture loop.

    This was mitigated somewhat by the fact FindReadySources poll interval defaults to 100 packets, and live sources are added to the ready list regardless of whether they have packets available (unless it is time to force a poll).

5.0.0-dev.257 | 2022-04-14 10:13:28 -0700

  * Re-instantiate providing location information to `LoadFile` hooks. (Robin Sommer, Corelight)

    #1835 subtly changed the semantics of the `LoadFile` plugin hook to no
    longer have the current script location available for signature files
    being loaded through `@load-sigs`. This was undocumented behavior, so
    it's technically not a regression, but since at least one external
    plugin is depending on it, this change restores the old behavior.

5.0.0-dev.255 | 2022-04-14 10:12:49 -0700

  * Fix another crash during dictionary iteration. (Robin Sommer, Corelight)

    Closes #2017.

  * Fix assertions in dictionary that can trigger for benign reasons. (Robin Sommer, Corelight)

    These assertions were checking for a situation that I believe can
    happen legitimately: a robust iterator pointing to an index that,
    after some table resizing, happens to be inside the overflow area and
    hence empty. We'll now move it to the end of the table in the case.

  * Fix robust iterators when modifying dictionary during iteration. (Robin Sommer, Corelight)

    When inserting/deleting elements, we now remove their `DictEntries`
    from any robust iterators' bookkeeping. First, we don't need that
    information anymore, and second the `DictEntries` contain pointers
    that may become invalid.

    I don't know how to write a unit test for this unfortunately because
    it depends on where exactly things land in the hash table.

    Btw, memory mgmt for DictEntries is pretty fragile: They contain
    pointers to both memory they own (`key`) and memory they don't own
    (`value`). The former type of pointers is shallow-copied on
    assignment/copy-construction, meaning that there can be multiple
    instances seemingly owning the same memory. That only works because
    deletion is manual, and not part of standard destruction. The second
    type of pointer has a similar problem, except that it's managed
    externally. It's important to not end up with multiple `DictEntries`
    pointing to the same value (which is actually what that iterator
    bookkeeping did).

    Addresses #2032.

5.0.0-dev.250 | 2022-04-14 09:51:23 -0700

  * Split asan/ubsan CI builds, add tsan build (Tim Wojtulewicz, Corelight)

5.0.0-dev.248 | 2022-04-14 08:59:34 -0700

  * Disable object-size analysis if optimization set to -O0 (Tim Wojtulewicz, Corelight)

5.0.0-dev.246 | 2022-04-14 10:48:19 +0200

  * Allow analyzer violations to explicitly set tag. (Robin Sommer, Corelight)

5.0.0-dev.244 | 2022-04-13 10:52:58 -0700

  * Add test to ensure enum_to_int's return values are ordered (Yacin Nadji, Corelight)

5.0.0-dev.242 | 2022-04-13 10:51:21 -0700

  * Add unit test for other get_word() version (Tim Wojtulewicz, Corelight)

5.0.0-dev.240 | 2022-04-11 12:46:51 -0700

  * Mask our signal handlers' triggering signals around thread creation (Christian Kreibich, Corelight)

5.0.0-dev.238 | 2022-04-11 12:40:02 -0700

  * GH-2026: Ensure both protocol and analyzer confirmation and violation events can be called (Tim Wojtulewicz, Corelight)

5.0.0-dev.235 | 2022-04-09 00:08:50 +0000

  * Update libkqueue to 2.6.0 release [skip ci] [nomail] (Tim Wojtulewicz)

5.0.0-dev.233 | 2022-04-08 11:30:52 -0700

  * Bump submodules to pull in InstallSymlink fix (Christian Kreibich, Corelight)

5.0.0-dev.231 | 2022-04-05 18:04:47 -0700

  * fix for ill-formed (complex) &default function (Vern Paxson, Corelight)

  * type-checking for use of empty table constructors in expressions (Vern Paxson, Corelight)

  * catch empty constructors used for type inference (Vern Paxson, Corelight)
    suppress repeated error messages

  * factoring to make checking of &default attributes externally accessible (Vern Paxson, Corelight)

  * bug fix for empty table constructors with &default attributes (plus a typo) (Vern Paxson, Corelight)

5.0.0-dev.222 | 2022-04-05 18:04:15 -0700

  * reduce interpreter frames for compiled function bodies (Vern Paxson, Corelight)

5.0.0-dev.219 | 2022-04-05 16:07:48 -0700

  * Correct origin documentation of the version field in the HTTP log. (Christian Kreibich, Corelight)

5.0.0-dev.217 | 2022-04-04 13:27:32 -0700

  * Move new TLS decryption capabilities up to Zeek 5 in NEWS file (Christian Kreibich, Corelight)

  * Update NEWS to reflect recent updates (Christian Kreibich, Corelight)

5.0.0-dev.214 | 2022-04-04 10:52:41 -0700

  * fix & btest for ZAM bug with inlined nested loop (Vern Paxson, Corelight)

5.0.0-dev.212 | 2022-04-04 10:51:20 -0700

  * GH-2009: Use auto to fix ZIP analyzer failure on some platforms (Tim Wojtulewicz, Corelight)

5.0.0-dev.210 | 2022-03-28 17:04:51 -0700

  * Add cmake-time reporting of bifcl, binpac, and gen-zam used for build (Christian Kreibich, Corelight)

  * Build Gen-ZAM from a submodule and support use of pre-existing executable (Christian Kreibich, Corelight)

5.0.0-dev.204 | 2022-03-25 15:31:21 -0700

  * --event-trace / -E option to generate event trace (Vern Paxson, Corelight)

  * hooks to support event tracing (Vern Paxson, Corelight)

  * classes providing event-tracing/dumping functionality (Vern Paxson, Corelight)

  * provide access to Val internals for event tracing purposes (Vern Paxson, Corelight)

  * set_network_time() BiF in support of event replaying (Vern Paxson, Corelight)

5.0.0-dev.195 | 2022-03-24 11:01:28 -0700

  * switch variable initialization over to being expression-based (Vern Paxson, Corelight)

  * simplification of Val classes now that they don't have to support initialization (Vern Paxson, Corelight)

  * rework type inference due to switch from separate initializers to expressions (Vern Paxson, Corelight)

  * avoid evaluating calls to determine whether an expression value is ignored (Vern Paxson, Corelight)

  * reworking of expressions to unify =/+=/-= with initialization (Vern Paxson, Corelight)

  * allow {} expression lists for =/+=/-= RHS (Vern Paxson, Corelight)

5.0.0-dev.177 | 2022-03-23 13:05:51 +0100

  * Improve the formatting of the SSL::Info::ssl_history documentation (Johanna Amann, Corelight)

5.0.0-dev.173 | 2022-03-16 15:06:05 -0700

  * Fix document generation (Christian Kreibich, Corelight)

5.0.0-dev.169 | 2022-03-10 11:09:37 -0700

  * add raw_bytes_to_v6_addr in docs when raw_bytes_to_v4_addr is present (Yacin Nadji, Corelight)

  * Zero out bytes by default for consistent return value on error (Yacin Nadji, Corelight)

  * Add tests for raw_bytes_to_v6_addr (Yacin Nadji, Corelight)

  * Add raw_bytes_to_v6_addr function (Yacin Nadji, Corelight)

5.0.0-dev.164 | 2022-03-08 09:30:37 -0700

  * Update 3rdparty submodule for bsd-getopt-long fix (Tim Wojtulewicz)

5.0.0-dev.162 | 2022-03-07 12:36:37 +0100

  * Improve error message when receiving unexpected record content via
    Broker. (Robin Sommer, Corelight)

5.0.0-dev.160 | 2022-03-02 13:48:07 +0000

  * restored record constructor checking for missing-but-mandatory fields. This includea a new btest
    as well as a fix to the base-scrpts. (Vern Paxson, Corelight)

5.0.0-dev.156 | 2022-03-02 08:23:50 +0000

  * The is_num(), is_alpha(), and is_alnum() BiFs now return F on empty string.
    The testcases for these functions, and for is_ascii() were expanded. The documentation of is_ascii()
    concerning behavior of an empty string was clarified (Christian Kreibich, Corelight)

5.0.0-dev.151 | 2022-03-02 08:09:28 +0000

  * SSL: rudimentary decryption for TLS 1.2 (Florian Wilkens, Johanna Amann)

    With this version, we support rudimentary decryption of TLS 1.2 connections, if the key material
    of the connection (in our case the pre-master secret) is available. Note that this functionality
    only works for TLS 1.2 connections using the TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 cipher suite.
    No other combinations are currently supported.

    For more information, see the NEWS entry and the TLS Decryption documentation.

5.0.0-dev.121 | 2022-02-24 09:11:03 -0700

  * GH-1980: Deprecate and return warning for zeek-config's caf-root option (Tim Wojtulewicz, Corelight)

5.0.0-dev.118 | 2022-02-23 10:51:57 -0700

  * GH-1949: Remove unused timer_mgr_inactivity_timeout global (Tim Wojtulewicz, Corelight)

5.0.0-dev.116 | 2022-02-21 18:17:13 -0700

  * remove deprecated union and timer types, addressing #1898 (Matthew Luckie)

5.0.0-dev.114 | 2022-02-11 09:30:04 -0800

  * Minor modernizations to Github workflows (Christian Kreibich, Corelight)

5.0.0-dev.112 | 2022-02-10 17:56:27 -0800

  * Reorg of the cluster controller to new "Management framework" layout (Christian Kreibich, Corelight)

  * Bump external cluster testsuite to reflect Management framework reorg (Christian Kreibich, Corelight)

  * Bump zeek-client to reflect Management framework reorg (Christian Kreibich, Corelight)

5.0.0-dev.108 | 2022-02-10 10:35:02 -0700

  * Fixing a big pile of Coverity issues (Tim Wojtulewicz, Corelight)

5.0.0-dev.106 | 2022-02-09 15:15:21 -0800

  * Expand generate-docs Github workflow to test docs build on PRs (Christian Kreibich, Corelight)

5.0.0-dev.104 | 2022-02-09 13:14:04 -0800

  * Updates to the cluster controller scripts to fix the docs build (Christian Kreibich, Corelight)

  * Bump zeek-client for Broker enum fix/workaround (Christian Kreibich, Corelight)

5.0.0-dev.100 | 2022-02-07 14:18:50 -0800

  * Add capture to a Sumstats when-statement to fix deprecation warning (Christian Kreibich, Corelight)

5.0.0-dev.97 | 2022-02-07 16:24:06 +0100

  * Update to latest Broker without public CAF dependencies. (Dominik
    Charousset, Corelight)

  * Fix GCC builds and string output for Broker errors (Dominik
    Charousset, Corelight)

5.0.0-dev.94 | 2022-02-07 08:14:47 -0700

  * String/StringVal: Replace char*/string constructors with string_view (Tim Wojtulewicz, Corelight)

5.0.0-dev.92 | 2022-02-04 10:33:47 -0700

  * fix existing checks for looking to use C++ when it's not available (Vern Paxson, Corelight)

5.0.0-dev.90 | 2022-02-04 10:32:41 -0700

  * fixes for ZAM profiling, which didn't get fully integrated originally (Vern Paxson, Corelight)

  * minor enhancements for ZAM inlining (Vern Paxson, Corelight)

5.0.0-dev.87 | 2022-02-03 13:17:25 -0800

  * Expansion of cluster controller functionality (Christian Kreibich, Corelight)

    - Bump external cluster testsuite
    - Bump zeek-client for the get-nodes command
    - Add ClusterController::API::get_nodes_request/response event pair
    - Support optional listening ports for cluster nodes
    - Don't auto-publish Supervisor response events in the cluster agent
    - Make members of the ClusterController::Types::State enum all-caps
    - Be more conservative with triggering request timeout events
    - Move redefs of ClusterController::Request::Request to their places of use
    - Simplify ClusterController::API::set_configuration_request/response

5.0.0-dev.77 | 2022-02-03 11:20:16 +0000

  * Match DPD TLS signature on one-sided connections. (Johanna Amann, Corelight)

    This commit changes DPD matching for TLS connections. A one-sided match
    is enough to enable DPD now.

    This commit also removes DPD for SSLv2 connections. SSLv2 connections do
    basically no longer happen in the wild. SSLv2 is also really finnicky to
    identify correctly - there is very little data required to match it, and
    basically all matches today will be false positives. If DPD for SSLv2 is
    still desired, the optional signature in policy/protocols/ssl/dpd-v2.sig
    can be loaded.

5.0.0-dev.74 | 2022-02-02 09:46:00 +0100

  * GH-1890: Consistently warn about mixing vector and scalar operand
    depreciaton (Zeke Medley, Corelight)

5.0.0-dev.72 | 2022-02-02 09:36:30 +0100

  * Let TCP-based application analyzers operate without any TCP parent
    analyzer. (Robin Sommer, Corelight)

5.0.0-dev.70 | 2022-01-25 13:52:00 -0700

  * bug fix for vector slice assignment (Vern Paxson, Corelight)

5.0.0-dev.67 | 2022-01-25 12:25:48 +0000

  * updated Bro->Zeek in comments in the source tree (Vern Paxson, Corelight)

5.0.0-dev.65 | 2022-01-24 13:41:25 -0800

  * CI updates (Christian Kreibich, Corelight)

    - add Ubuntu 21.10
    - remove OpenSUSE Leap 15.2 (EOL)
    - add CentOS Stream 9

5.0.0-dev.61 | 2022-01-17 10:35:15 +0000

  * fix for adding a non-managed type to an empty vector (Vern Paxson, Corelight)

5.0.0-dev.58 | 2022-01-14 14:41:15 -0700

  * explicitly provide the frame for evaluating a "when" timeout expression (Vern Paxson, Corelight)

  * tests for new "when" semantics/errors (Vern Paxson, Corelight)

  * captures for "when" statements (Vern Paxson, Corelight)
    update Triggers to IntrusivePtr's and simpler AST traversal
    introduce IDSet type, migrate associated "ID*" types to "const ID*"

  * logic (other than in profiling) for assignments that yield separate values (Vern Paxson, Corelight)

  * option for internal use to mark a function type as allowing non-expression returns (Vern Paxson, Corelight)

  * removed some now-obsolete profiling functionality (Vern Paxson, Corelight)

5.0.0-dev.47 | 2022-01-14 12:22:11 -0700

  * Remove Fedora 33 from CI. (Johanna Amann, Corelight)

    EOL date was 2021‑11‑30

5.0.0-dev.45 | 2022-01-14 12:21:25 -0700

  * Have `make dist` cleanup a few more wayward files before tarring (Tim Wojtulewicz, Corelight)

5.0.0-dev.43 | 2022-01-14 15:59:53 +0000

  * Make more StringVal member functions const (Johanna Amann, Corelight)

5.0.0-dev.41 | 2022-01-12 10:16:38 -0700

  * Fix --with-jemalloc configure option on FreeBSD (Tim Wojtulewicz, Corelight)

5.0.0-dev.39 | 2022-01-11 15:02:37 -0700

  * support for compiling type-based switches to C++ (Vern Paxson, Corelight)

  * make encountering uncompilable "standalone" functions to be a hard error (Vern Paxson, Corelight)

  * fix: ZAM could misinterpret a "type" switch that starts with a "default" (Vern Paxson, Corelight)

  * fix: don't treat pseudo-identifiers in type cases as local variables (Vern Paxson, Corelight)

  * removed development helper scripts now obsolete with --optimize-files= (Vern Paxson, Corelight)

5.0.0-dev.33 | 2022-01-11 14:57:58 -0700

  * Bump CI sanitizer task to Ubuntu 20.04 (Tim Wojtulewicz, Corelight)

  * Fix uninitialized variable warning (Tim Wojtulewicz, Corelight)

  * Enable null check for UBSan (Tim Wojtulewicz, Corelight)

5.0.0-dev.28 | 2022-01-10 10:30:39 -0700

  * fix for avoiding de-ref of nil pointer (Vern Paxson, Corelight)

5.0.0-dev.25 | 2022-01-10 14:42:50 +0100

  * GH-1844: Fix host header normalization in intel framework. (Robin
    Sommer, Corelight)

  * GH-1844: Switch to recording unmodified HTTP header. (Robin
    Sommer, Corelight)

5.0.0-dev.22 | 2022-01-07 12:34:25 -0700

  * GH-1624: Migrate check_and_promote and a few Expr methods to IntrusivePtr (Tim Wojtulewicz, Corelight)

5.0.0-dev.19 | 2022-01-06 15:06:59 -0700

  * bug fix for reporting poorly formed record constructors (Vern Paxson, Corelight)

5.0.0-dev.17 | 2022-01-06 14:04:48 -0700

  * Remove CentOS 8 from CI. (Johanna Amann, Corelight)

    EOL was 2021-12-31

5.0.0-dev.15 | 2022-01-06 13:54:38 -0700

  * test suite alternative baseline fixes for recent test renaming (Vern Paxson, Corelight)

  * addressed a couple of memory leaks in ZAM execution (Vern Paxson, Corelight)

5.0.0-dev.10 | 2022-01-05 09:13:45 -0700

  * Make FreeBSD more flexible, don't look for jemalloc unless we were (Craig Leres)
    given a specific path for it.

  * Set JEMALLOC_FOUND when on FreeBSD (suggested by Tim Wojtulewicz) (Craig Leres)

  * FreeBSD: don't look for jemalloc as a package, it's in the base system (Craig Leres)

5.0.0-dev.6 | 2022-01-05 09:12:33 -0700

  * Fix return type of double_to_int (Johanna Amann, Corelight)

    Fixes GH-1919

4.2.0-dev.514 | 2022-01-03 13:56:12 -0700

  * deprecation warning on use of out-of-scope local (Vern Paxson, Corelight)

4.2.0-dev.510 | 2022-01-03 13:54:52 -0700

  * Switch BitTorrent analyzer to Zeek's regex engine (Avinal Kumar)

    - Removes dependency on <regex.h>
    - Replaces regex function with Zeek's standard regex functions
    - Some replacements are workaround, may be improved later via an
    appropiate API
    - Update test baseline to fix what seems to be capturing on a bug in the
    existing code.

    Edit pass by Robin Sommer. Note that our test doesn't cover all the code
    paths, but it does go through the one with the most substantial change.

  * Adding test for BitTorrent tracker. (Robin Sommer, Corelight)

    Our test trace is extracted from https://www.cloudshark.org/captures/b9089aac6eee.

    There actually seems to be a bug in the existing code: the URI passed to
    bt_tracker_request() includes a partial HTTP version. This commits
    includes the baseline as the current code produces it, we'll fix that in
    a subsequent comment.

4.2.0-dev.506 | 2022-01-03 09:33:43 -0800

  * Expansion of the emerging cluster controller framework (Christian Kreibich, Corelight)

    - Controller/agent connectivity is now controlled by pushed configurations
    - The Request module now supports state timeouts
    - Use Result records consistently for responses to the client
    - Track successful config deployment in cluster controller
    - Add ClusterController::API::notify_agents_ready event
    - Make all globals start with a "g_" prefix
    - Add missing debug() log function to log module's API
    - Add separate utility module for controller and agent
    - Additional infrastructure for printing types
    - Bump zeek-client to v0.2.0
    - Add Github action job for cluster tests
    - Tweak Docker image configure invocation to include zeek-client
    - Zeekygen documentation pass

4.2.0-dev.477 | 2021-12-14 16:53:57 -0700

  * fixes for double-delete and reducing '?' operator with constant alternatives (Vern Paxson, Corelight)

  * correct usage info for -u flag; -uu no longer supported (Vern Paxson, Corelight)

4.2.0-dev.468 | 2021-12-14 11:34:47 -0700

  * factoring of generating C++ initializations, no semantic changes (Vern Paxson, Corelight)

  * restored support for incremental compilation of scripts to C++ (Vern Paxson, Corelight)

  * fixes for -O gen-standalone-C++ (Vern Paxson, Corelight)

  * new ZEEK_FILE_ONLY and ZEEK_FUNC_ONLY environment variables for debugging script optimization - replaces ZEEK_ONLY (Vern Paxson, Corelight)

  * fix for compiling record constructors to C++ (Vern Paxson, Corelight)

  * fixes for compiling vector operations to C++ (Vern Paxson, Corelight)

  * fixed for profiling missing some profile elements (Vern Paxson, Corelight)

  * minor efficiency tweak for ZAM record construction (Vern Paxson, Corelight)

4.2.0-dev.456 | 2021-12-14 09:23:47 -0700

  * GH-1860: Add double_to_int() bif (Tim Wojtulewicz, Corelight)

4.2.0-dev.454 | 2021-12-13 09:41:32 -0700

  * Check for sets before attempting to check for same Yield types (Tim Wojtulewicz)

  * Add early bail-outs to same_type() (Tim Wojtulewicz)

  * Fix types for Analyzer::register_for_port(s) to be the same (Tim Wojtulewicz)

  * Update cmake submodule across all other submodules (Tim Wojtulewicz, Corelight)

4.2.0-dev.448 | 2021-12-10 15:35:34 -0700

  * update btest to no longer use (unsupported) %S formatting, no longer needed (Vern Paxson, Corelight)

  * replace --optimize-only with --optimize-funcs and --optimize-files (Vern Paxson, Corelight)

4.2.0-dev.444 | 2021-12-10 13:13:13 -0700

  * reintroduction of "-O add-C++" option (Vern Paxson, Corelight)

4.2.0-dev.442 | 2021-12-10 13:12:43 -0700

  * fixes for vector operations (Vern Paxson, Corelight)

  * flag globals initialized to opaque values as non-compilable (Vern Paxson, Corelight)

  * skip type signatures for lambdas (Vern Paxson, Corelight)

  * fix for translating filenames beginning with numbers to C++ variable names (Vern Paxson, Corelight)

4.2.0-dev.436 | 2021-12-10 13:11:36 -0700

  * update script-to-C++ compilation for new record constructor internals (Vern Paxson, Corelight)

4.2.0-dev.434 | 2021-12-10 13:11:10 -0700

  * updates to ZAM to track recent changes in script semantics (Vern Paxson, Corelight)

4.2.0-dev.432 | 2021-12-10 09:28:23 -0700

  * GH-1741: Print error if calling a non-hook with hook keyword (Tim Wojtulewicz, Corelight)

  * GH-1740: Report a better error message if table key is not a list (Tim Wojtulewicz, Corelight)

4.2.0-dev.428 | 2021-12-09 14:58:53 -0700

  * GH-1125: Support GRE ARUBA headers (Tim Wojtulewicz, Corelight)

  * Fix ethertype for ARP in Geneve forwarding rules (Tim Wojtulewicz, Corelight)

4.2.0-dev.425 | 2021-12-09 13:45:17 -0800

  * Add LogAscii::json_include_unset_fields flag to control unset field rendering (Christian Kreibich, Corelight)

4.2.0-dev.423 | 2021-12-09 19:56:43 +0000

  * Improve error message for clash between variable and function name (Johanna Amann, Corelight)
    Fixes GH-1832

  * Restore --disable-zeekctl configure argument (Tim Wojtulewicz, Corelight)

  * Update plugin.hooks baseline for recent Geneve change (Tim Wojtulewicz, Corelight)

4.2.0-dev.419 | 2021-12-07 09:34:45 -0700

  * GH-1764: Update mappings for Geneve analyzer to IP4/IP6/ARP (Tim Wojtulewicz, Corelight)

4.2.0-dev.417 | 2021-12-06 17:00:16 -0800

  * Flip C++ unit tests to being enabled by default (Christian Kreibich, Corelight)

    To disable them, configure with --disable-cpp-tests.

  * Support for unit tests in plugins (Christian Kreibich, Corelight)

4.2.0-dev.410 | 2021-12-06 11:29:32 -0700

  * Remove separate Tag types, note breaking change in NEWS (Tim Wojtulewicz, Corelight)

4.2.0-dev.408 | 2021-12-06 09:15:24 -0700

  * GH-1768: Properly cleanup existing log stream when recreated on with the same ID (Tim Wojtulewicz, Corelight)

4.2.0-dev.406 | 2021-12-01 10:32:34 -0700

  * Format Python scripts with yapf. (Benjamin Bannier, Corelight)

    We also add a very basic yapf configuration file. Most of the changes in
    this patch were performed automatically, but we broke one overly long
    string into multiple components on `src/make_dbg_constants.py`.

  * Format shell scripts with shfmt. (Benjamin Bannier, Corelight)

    All changes in this patch were performed automatically with `shfmt` with
    configuration flags specified in `.pre-commit-config.yaml`.

4.2.0-dev.403 | 2021-12-01 10:25:32 -0700

  * fix btest comment to more accurately describe the test (Vern Paxson, Corelight)

  * btests for erroneous script conditionals (Vern Paxson, Corelight)

  * avoid compiling-to-C++ for functions potentially influenced by conditionals (Vern Paxson, Corelight)

  * track the use of conditionals in functions and files (Vern Paxson, Corelight)

  * AST profiles track the associated function/body/expression (Vern Paxson, Corelight)

4.2.0-dev.396 | 2021-12-01 09:44:03 -0700

  * GH-1873: Deprecate the tag types differently to avoid type clashes (Tim Wojtulewicz, Corelight)

4.2.0-dev.394 | 2021-11-30 11:53:35 -0700

  * Fix for the recent patch that allows segment offloaded packets. (Johanna Amann, Corelight)

    We recently added support for segment offloaded packets. It turns out
    that this can lead to problems in UDP/ICMP based parsers since I missed
    correctly also updating the payloadlength there, and using the capture
    length instead when segment offloading is enabled.

    Credit to OSS-Fuzz for discovery
    https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41391
    https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41394
    https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=41395
    (Link to details becomes public 30 days after patch release)

4.2.0-dev.393 | 2021-11-29 13:46:59 -0700

  * Fix a number of Coverity findings (Tim Wojtulewicz, Corelight)

    1466460: Uninitialized field in gtp-analyzer.pac
    1462465: Null pointer dereference in CompositeHash::SingleValHash
    1462463: Copy/paste error in TCPSessionAdapter::build_syn_packet_val
    1462067: Uninitialized fields in Zinst

4.2.0-dev.391 | 2021-11-29 13:44:11 -0700

  * suppress unneeded initializations (Vern Paxson, Corelight)

4.2.0-dev.387 | 2021-11-24 13:32:33 -0700

  * fixes for constructing and assigning records with fields that are empty vectors (Vern Paxson, Corelight)

4.2.0-dev.385 | 2021-11-23 19:43:48 -0700

  * Changes to speed up compilation of Compiled-to-C++ Zeek Scripts (Vern Paxson, Corelight)

  * removing unused SubNetType class (Vern Paxson, Corelight)

4.2.0-dev.371 | 2021-11-23 19:41:10 -0700

  * Add new tunnel packet analyzers, remove old ones (Tim Wojtulewicz, Corelight)

  * Add PacketAnalyzer::register_for_port(s) functions (Tim Wojtulewicz, Corelight)

    These allow packet analyzers to register ports as identifiers to forward from
    parent analyzers, while also adding those ports to the now-global
    Analyzer::ports table at the same time.

  * Add analyzer_confirmation and analyzer_violation events (Tim Wojtulewicz, Corelight)

  * Add utility function for tunnel analyzers to setup encapsulation (Tim Wojtulewicz, Corelight)

  * Store some additional information in the packet during processing (Tim Wojtulewicz, Corelight)

    - Session related to the packet
    - is_orig information if a UDP header was found

  * Minor fix in UDP to avoid duplicating tunnels (Tim Wojtulewicz, Corelight)

  * Fix error text in IPTunnel analyzer (Tim Wojtulewicz, Corelight)

  * Change Packet::ip_hdr to be a shared_ptr so it can be copied into EncapsulatingConn (Tim Wojtulewicz, Corelight)

  * Add method for packet analyzers to register for protocol detection (Tim Wojtulewicz, Corelight)

  * Add concept of "parent" tag namespaces (Tim Wojtulewicz, Corelight)

    This allows us to create an EnumType that groups all of the analyzer
    tag values into a single type, while still having the existing types
    that split them up. We can then use this for certain events that benefit
    from taking all of the tag types at once.

  * Unify plugin::Component and plugin::TaggedComponent into a single class (Tim Wojtulewicz, Corelight)

    These two are almost always used in conjunction with each other, and
    TaggedComponent is never used by itself. Combining them together into
    a single class will help simplify some of the code around managing
    the mapping between Tags and Components.

  * Remove uses of deprecated Tag types (Tim Wojtulewicz, Corelight)

  * Unify all of the Tag types into one type (Tim Wojtulewicz, Corelight)

    - Remove tag types for each component type (analyzer, etc)
    - Add deprecated versions of the old types
    - Remove unnecessary tag element from templates for TaggedComponent and ComponentManager
    - Enable TaggedComponent to pass an EnumType when initializing Tag objects
    - Update some tests that are affected by the tag enum values changing order

4.2.0-dev.350 | 2021-11-23 15:35:06 +0000

  * Add testcase for TCP segment offloading (GH-1829). (Johanna Amann, Corelight)

4.2.0-dev.348 | 2021-11-23 13:45:39 +0000

  * OpenSSL 3 compatibility (Johanna Amann, Corelight)

    Zeek is now compatible with OpenSSL 3.0, our test baselines pass cleanly, and
    we have a CI run for OpenSSL 3.0. This has a certain amount of new code for
    X.509 certificate parsing. Apart from that, the main chainge is that we
    use an older, legacy, API for OpaqueVal hashing, since the newer API
    does not allow us to serialize data anymore. For details see ticket 1379.

4.2.0-dev.340 | 2021-11-23 10:10:13 +0000

  * Accept packets that use tcp segment offloading. (Johanna Amann, Corelight)

    When checksum offloading is enabled, we now forward packets that
    have 0 header lengths set - and assume that they have TSO enabled.

    If checksum offloading is not enabled, we drop the packets (GH-1829)

  * Updates to NEWS to cover recent additions. [nomail] [skip ci] (Christian Kreibich, Corelight)

  * Update doc and auxil/zeek-aux submodules [nomail] [skip ci] (Christian Kreibich, Corelight)

  * Update cmake and aux/zeek-aux submodules [nomail] [skip ci] (Christian Kreibich, Corelight)

4.2.0-dev.333 | 2021-11-17 11:57:04 -0800

  * Clean up fully after successful Docker btests (Christian Kreibich, Corelight)

4.2.0-dev.331 | 2021-11-15 10:10:52 -0800

  * Fix ref-naming typo in the Github Docker workflow (Christian Kreibich, Corelight)

4.2.0-dev.328 | 2021-11-12 13:46:32 -0700

  * Update libkqueue submodule (Tim Wojtulewicz, Corelight)

4.2.0-dev.326 | 2021-11-12 09:30:54 -0700

  * Added plugin.unprocessed_packet_hook btest (Tim Wojtulewicz, Corelight)

  * Fix whitespace in help output (Tim Wojtulewicz, Corelight)

  * Add command-line option to write unprocessed packets to a file (Tim Wojtulewicz, Corelight)

    This commit also changes the PcapDumper to automatically flush after
    every called to Dump(). This is because pcap_dump has an internal buffer
    of some sort that only writes to the file after a set amount of bytes.
    When using the new option on a low-traffic network, it might be a while
    before you see any packets written since it has to overcome that buffer
    limit first.

  * GH-1620: Add event and plugin hook to track packets not processed (Tim Wojtulewicz, Corelight)

4.2.0-dev.319 | 2021-11-10 10:20:01 -0700

  * Install include headers from `src/3rdparty/`. (Benjamin Bannier, Corelight)

    This is a fixup commit for 72cbc7cd13b7c1bda98658104431c3b530ff68d6
    where we move some header files from `src/` to `src/3rdparty/` but
    missed adding install rules for these header. Since some of these
    headers are exposed in installed headers they need to be installed as
    well.

4.2.0-dev.317 | 2021-11-10 11:33:29 +0000

  * Add case-insensitive search for find_str and rfind_str (Abdel)

4.2.0-dev.314 | 2021-11-10 11:16:28 +0100

  * GH-1757: Add new hook `HookLoadFileExtended` that allows plugins
    to supply Zeek script and signature code to parse. (Robin Sommer)

    The new hook works similar to the existing `HookLoadFile` but,
    additionally, allows the plugin to return a string that contains
    the code to be used for the file being loaded. If the plugin does
    so, the content of any actual file on disk will be ignored. This
    works for both Zeek scripts and signatures.

  * Fix an issue where signature files supplied on the command line
    wouldn't pass through the file loading hooks. (Robin Sommer,
    Corelight)

4.2.0-dev.310 | 2021-11-09 10:29:59 -0700

  * Add Github action exercising pre-commit (Benjamin Bannier, Corelight)

    This patch adds a Github action which exercises pre-commit linters for
    commits to the `master` branch or for pull requests.  We adds this task
    as a Github action since we expect it to finish quickly; running outside
    of Cirrus makes it possible provide feedback quickly.

  * Add pre-commit config. (Benjamin Bannier, Corelight)

    This patch adds `clang-format` as only linter for now. This replaces the
    previously used script from `auxil/run-clang-format` which we remove.

    This requires the Python program `pre-commit`
    (https://pypi.org/project/pre-commit/). With that one can then run
    `clang-format` on the whole codebase with

        $ pre-commit run -a clang-format

    or on just the staged files

        # Explicitly selecting linter.
        $ pre-commit run clang-format

        # Run all linters (currently just `clang-format`).
        $ pre-commit

    `pre-commit` supports managing Git commit hooks so that linters are run
    on commit. Linters can be installed with

        $ pre-commit install

    The documentation at https://pre-commit.com/ covers these topics in
    addition to more information.

  * Format code with `clang-format` (Benjamin Bannier, Corelight)

    This patch formats files not conforming to the C++ formatting with
    `clang-format`.

  * Remove stale files `src/DebugCmdInfoConstants.*` (Benjamin Bannier, Corelight)

    The files generated from `src/DebugCmdInfoConstants.in` are placed in
    `build/src/` by the build setup, and generated file in `src/` removed
    here were unused and possibly out-of-date.

  * Disable formatting for files in `testing/btest/plugins` (Benjamin Bannier, Corelight)

    Files in that folder were previously not formatted. With this patch we
    now disable formatting in that folder explicitly by adding a dedicated
    `clang-format` config which deactivates any formatting changes.

  * Move 3rdparty source files to `3rdparty/` (Benjamin Bannier, Corelight)

    This patch moves in-tree 3rdparty source files to `3rdparty/`. With that
    we can remove special treatment of these files for `run-clang-format`.

4.2.0-dev.303 | 2021-11-09 09:45:57 -0700

  * GH-1819: Handle recursive types when describing type in binary mode (Tim Wojtulewicz, Corelight)

4.2.0-dev.301 | 2021-11-09 09:28:18 -0700

  * Remove no-op false-teredo test (Tim Wojtulewicz, Corelight)

4.2.0-dev.297 | 2021-11-05 12:49:55 -0700

  * Only push CI's Docker images when we're on the main repo (Christian Kreibich, Corelight)

  * Add macOS Monterey and drop Catalina in CI (Christian Kreibich, Corelight)

  * Add Fedora 35 to CI (Christian Kreibich, Corelight)

4.2.0-dev.292 | 2021-11-04 14:28:35 -0700

  * Fix C++ set intersection code (Yacin Nadji, Corelight)

4.2.0-dev.286 | 2021-11-03 09:36:41 -0700

  * GH-693: use pcap_dump_open_append where supported (Tim Wojtulewicz, Corelight)

4.2.0-dev.284 | 2021-11-03 09:35:10 -0700

  * GH-1781: Add .git-blame-ignore-revs file (Tim Wojtulewicz, Corelight)

4.2.0-dev.280 | 2021-11-01 09:20:16 -0700

  * Fix issue with broken libpcaps that return repeat packets (Tim Wojtulewicz, Corelight)

    This is apparently a problem with the Myricom version of libpcap, where
    instead of returning a null or a zero if no packets are available, it
    returns the previous packet. This causes Zeek to improperly parse the
    packet and crash. We thought we had fixed this previously with a check
    for a null packet but that fix was not enough.

4.2.0-dev.277 | 2021-10-21 17:23:46 -0700

  * Apply some missing clang-format changes (Tim Wojtulewicz, Corelight)

4.2.0-dev.274 | 2021-10-20 11:13:16 -0700

  * Remove trailing whitespace from script files (Tim Wojtulewicz, Corelight)

4.2.0-dev.271 | 2021-10-19 14:54:56 +0200

  * Add parsing of DNS SVCB/HTTPS records (FlyingWithJerome)

4.2.0-dev.260 | 2021-10-15 09:45:45 +0100

  * logging/writers/ascii: shadow files: Add fsync() before rename(). This
    prevents potential problems with leftover files after unclean shutdowns.
    (Arne Welzel, Corelight)

  * Fix typo in typedef changes that broke tests on 32-bit Debian 9 (Tim Wojtulewicz, Corelight)

4.2.0-dev.255 | 2021-10-12 09:22:37 -0700

  * Replace most uses of typedef with using for type aliasing (Tim Wojtulewicz, Corelight)

4.2.0-dev.253 | 2021-10-11 11:38:30 -0700

  * Don't initialize private testsuite in CI when not running out of Zeek project (Christian Kreibich, Corelight)

  * Drop FreeBSD 11.4 (now EOL), add FreeBSD 14 as informational build (Christian Kreibich, Corelight)

4.2.0-dev.249 | 2021-10-04 12:47:42 -0700

  * Expand table/set tests to cover new index types (Christian Kreibich, Corelight)

  * Expression tweaks to get complex types to work as container indices (Christian Kreibich, Corelight)

4.2.0-dev.245 | 2021-10-01 14:19:19 -0700

  * Enable customization of the Docker image's configure invocation (Christian Kreibich, Corelight)

  * Add Python Broker module load test to Docker image btests (Christian Kreibich, Corelight)

  * Set PYTHONPATH in Docker to automatically find Zeek's modules (Christian Kreibich, Corelight)

  * Docker image fix: libpython needs to be 3.9 on Debian Bullseye (Christian Kreibich, Corelight)

4.2.0-dev.240 | 2021-10-01 13:28:39 -0700

  * Bump highwayhash submodule to pull in another FreeBSD header fix (Christian Kreibich, Corelight)

  * CI tweak: upgrade curl on FreeBSD to avoid cert expiration (Christian Kreibich, Corelight)

  * CI tweak: add a datestamp to Debian 9 Dockerfiles to invalidate Cirrus build cache (Christian Kreibich, Corelight)

4.2.0-dev.235 | 2021-09-28 10:04:46 -0700

  * GHI-1766: Remove address from Site::private_address_space that converts into 0.0.0.0/0 (Tim Wojtulewicz, Corelight)

4.2.0-dev.233 | 2021-09-27 12:06:27 -0700

  * Fix a number of issues with the initial pass of clang-format (Tim Wojtulewicz, Corelight)

4.2.0-dev.224 | 2021-09-26 10:27:05 -0700

  * Fix space issue in docker tag computation (Benjamin Bannier, Corelight)

4.2.0-dev.222 | 2021-09-25 11:47:46 -0700

  * Fixing homebrew, again. (Tim Wojtulewicz, Corelight)

    Homebrew apparently requires you to now specify the version of the package
    you're updating. We previously could just run `brew update openssl` and it
    would work. Now we must run `brew update openssl@1.1` for the command to
    succeed.

4.2.0-dev.220 | 2021-09-24 20:49:08 -0700

  * Add .dockerignore to suppress btest artifacts (Christian Kreibich, Corelight)

    We run btest in CI between image build and upload, and since its artifacts
    weren't suppressed, the discrepancy caused the Dockerfile COPY to cache-miss.

4.2.0-dev.218 | 2021-09-24 11:10:30 -0700

  * Add a minimal containerized Docker environment (Benjamin Bannier, Corelight)

    This patch adds a minimal Zeek environment packaged as a container.
    Since this is intended both as a base layer for other images and as a
    quick way to explore Zeek we install only zeek and zkg as basic
    functionality.

4.2.0-dev.214 | 2021-09-24 10:31:34 -0700

  * script simplification that removes an unnecessary &is_assigned (Vern Paxson, Corelight)

  * removing -uu functionality and associated script analysis now no longer needed (Vern Paxson, Corelight)

4.2.0-dev.208 | 2021-09-23 17:48:13 +0200

  * Fix and extend protocol forwarding/logging in dce_rpc-auth.
    (FOX-DS)

4.2.0-dev.205 | 2021-09-23 12:24:06 +0200

  * Avoid allocation of duplicate zero-length strings for new
    connections. (Justin Azoff, Corelight)

  * Sanity-check the method passed into ActiveHTTP. Reported by Pierre
    Gaulon. (Robin Sommer, Corelight)

4.2.0-dev.203 | 2021-09-21 15:16:49 -0700

  * Sanitize log files names before they go into system(). (Robin Sommer, Corelight)

    In principle, an attacker who's controlling the Zeek scripts being
    loaded could have set log paths to include non-safe characters leading
    to arbitrary command execution during log rotation. This fix avoids that
    be sanitizing the file names / command lines.

    Note, though, that this isn't a problem that we can really solve:
    somebody controlling scripts can just as well inject custom
    `system()` calls to begin with.

    Closes #54.

4.2.0-dev.201 | 2021-09-21 15:15:57 -0700

  * PIA - switch size to int64_t (Johanna Amann, Corelight)

    This brings the PIA size counter in line with the actual datatype used
    on the scripting layer - both now use an int64_t.

  * Introduce dpd_max_packets (Johanna Amann, Corelight)

    dpd_max_packets is an additional setting that limits the maximum amount
    of packets that dpd will cache; before dpd was only limited by buffer
    size (but could cache a limitless amount of data-less packets).

4.2.0-dev.198 | 2021-09-21 14:13:21 -0700

  * Use `brew update` instead of `brew update-reset` in CI macOS prepare script (Tim Wojtulewicz, Corelight)

    Using `brew update-reset` causes homebrew to reset to homebrew's HEAD commit, which may
    be buggy and broken. It appears whatever Cirrus was doing previously on their Catalina
    VM is no longer a problem, and so update-reset isn't required anymore. Switch to
    `brew update` to make sure we still get newer versions of the packages, but is
    actually a versioned release of homebrew.

4.2.0-dev.196 | 2021-09-21 09:44:26 -0700

  * Disable script profiling/coverage on sanitizer builds (Tim Wojtulewicz, Corelight)

4.2.0-dev.194 | 2021-09-21 17:56:14 +0200

  * Optimize software framework version parsing by adding a small
    cache in front of the parse method, and moving the parsing itself
    to the proxies where the caching can be more efficient. (Justin
    Azoff, Corelight)

  * Add further mappings for MS-OAUT IDispatch methods. (FOX-DS)

4.2.0-dev.189 | 2021-09-21 07:45:11 -0700

  * Add btests for new functionality (Christian Kreibich, Corelight)

    - Expand language.set to cover sets of sets
    - Expand language.table to cover tables indexed with tables
    - Add language.table-nested-set-ordering to capture the reproducer from GHI-1753

  * Remove unused HashKey constructor and reorder for consistency (Christian Kreibich, Corelight)

    One of the HashKey constructors was only used in the old CompHash code.
    This aso reorders some constructors and the destructor for readability.

  * Refactor CompHash class to use new HashKey buffering features (Christian Kreibich, Corelight)

    This preserves the previous hash key buffer layout (so the testsuite still
    passes) and overall approach but gets rid of the codepath for writing singleton
    serializations. This code path required a fourth switch block over all types
    (besides reads, writes, and size computation) and was inconsistent with the one
    for writing non-atomic types.

  * Add debug string and ODesc support to HashKey class (Christian Kreibich, Corelight)

    This allows tracing of hash key buffer reservations, reads, and writes via a new
    debug stream, and supports printing a summary of a HashKey object via
    Describe(). The latter comes in handy e.g. in TableVal::Describe() (where
    including the hash key is now available but commented out).

  * Refactor HashKey class to support read/write operations (Christian Kreibich, Corelight)

    This preserves the optimization of storing values directly in the key_u member
    union when feasible, and using a variable size buffer otherwise. It also adds
    bounds-checking for that buffer, moves size arguments to size_t, decouples
    construction from hash computation, emulates the tagging feature found in
    SerializationFormat to assist troubleshooting, and switches feasible
    reinterpret_casts to static_casts.

  * Add unit tests for memory helpers (Christian Kreibich, Corelight)

  * Add memory sizing/alignment helpers to util.cc/h (Christian Kreibich, Corelight)

    This functionality previously lived in the CompHash class, with one difference:
    this removes a discrepancy between the offset aligner and the memory pointer
    aligner/padder. The size aligner used to align the provided offset and then add an
    additional alignment size (for example, 1 aligned to 4 wouldn't yield 4 but 8).
    Like the memory aligners it now only rounds up as needed.

    Includes unit tests.

  * Ensure table/set HashKey buffer reservation and writes happen in same order (Christian Kreibich, Corelight)

    This takes the existing sorting for table index hashkeys we had in place during
    hash key writes and applies it also during buffer size reservation. It changes
    the approach slightly: the underlying map now points to the TableVal entry index
    vals directly, rather than to the numerical index into an additional list that
    gets built up to store those indexes. Doing so removes the need for that list.

4.2.0-dev.179 | 2021-09-20 11:20:50 +0200

  * Add TSval and TSecr to "SYN_packet" record. (gpotter2)

4.2.0-dev.174 | 2021-09-16 15:36:52 -0700

  * Minor fixes for build problems after reformatting (Tim Wojtulewicz, Corelight)

  * Reformat the world (Tim Wojtulewicz, Corelight)

  * Fixes for includes ahead of reformatting (Tim Wojtulewicz, Corelight)

  * GH-386: Add clang-format config and scripts (Tim Wojtulewicz, Corelight)

4.2.0-dev.169 | 2021-09-16 11:15:14 +0200

  * Added four new opnum mappings for MS-OAUT IDispatch methods. (FOX-DS)

  * Reformat docs in addrs.zeek to fix doc generation. (Tim
    Wojtulewicz, Corelight)

4.2.0-dev.165 | 2021-09-14 18:39:34 -0700

  * Addressed a number of low-level issues found by Coverity and compiler warnings (Vern Paxson, Corelight)

  * Fixed https://github.com/zeek/zeek/issues/1776 (Vern Paxson, Corelight)

4.2.0-dev.159 | 2021-09-14 13:00:50 -0700

  * ignore_checksums_nets: Add test for multiple subnets (Arne Welzel, Corelight)

    This test fails with Zeek 4.1 and Zeek 4.0.3.

4.2.0-dev.157 | 2021-09-08 12:04:45 -0700

  * fix for looping over vectors with holes per https://github.com/zeek/zeek/issues/1763 (Vern Paxson, Corelight)

  * btest for https://github.com/zeek/zeek/issues/1763 (Vern Paxson, Corelight)

4.2.0-dev.154 | 2021-09-08 12:03:51 -0700

  * Fix mis-usage of string::append that leads to an overflow (Tim Wojtulewicz, Corelight)

  * Use json_escape_utf8 for all utf8 data in ODesc (Tim Wojtulewicz, Corelight)

  * Switch the TSV Zeek logs to be UTF8 by default. (Seth Hall, Corelight)

    There is a paired zeek-testing branch for some updates there.

4.2.0-dev.150 | 2021-09-08 11:44:15 -0700

  * Fixes and btests for vector coercion overflows, typing, and holes (Vern Paxson, Corelight)

  * Improve -u reporting, distinguishing between maybe-vs-definite problems (Vern Paxson, Corelight)

  * "-a ZAM" testing baseline alternative (Vern Paxson, Corelight)

  * The main ZAM code, now available via -O ZAM (Vern Paxson, Corelight)

  * Rewrote tracking of variable definition ranges to be much more performant (Vern Paxson, Corelight)

  * Extend ID/Expr/Stmt classes to have associated optimization information (Vern Paxson, Corelight)

4.2.0-dev.112 | 2021-09-03 18:12:12 +0000

  * Add btests for DNS WKS and BINDS (Vlad Grigorescu)

  * Add btest for DNS WKS RR. (Vlad Grigorescu)

  * Add btest for DNS NSEC3PARAM RR. (Vlad Grigorescu)

4.2.0-dev.106 | 2021-09-03 18:10:31 +0000

  * Code modernization: use ranged-based for loop where possible (Vlad Grigorescu)

  * Improve performance of asn1_oid_to_val string conversions. (Vlad Grigorescu)

4.2.0-dev.103 | 2021-09-03 18:08:57 +0000

  * Disable the scripts.base.frameworks.logging.sqlite.simultaneous-writes test under TSan (Tim Wojtulewicz, Corelight)

    Due to a bug (or intentional code) in SQLite, we disabled enabling the shared cache
    in sqlite3 if running under ThreadSanitizer (see cf1fefbe0b0a6163b389cc92b5a6878c7fc95f1f).
    Unfortunately, this has the side-effect of breaking the simultaneous-writes test because
    the shared cache is disabled. This is hopefully a temporary fix until SQLite fixes the
    issue on their side.

  * Mark MsgThread::cnt_sent_{in,out} as atomic to avoid a data race (Tim Wojtulewicz, Corelight)

  * Disable call to sqlite3_enable_shared_cache under ThreadSanitizer (Tim Wojtulewicz, Corelight)

    See https://sqlite.org/forum/forumpost/54424d80ee for details.

4.2.0-dev.99 | 2021-09-03 17:36:09 +0000

  * GH-1589: Avoid extracting IP-like strings from SMTP headers (Tim Wojtulewicz, Corelight)

  * Minor updates to the external-testsuite scripts (Christian Kreibich, Corelight)

    Delete the unused Baseline folder, remove a Bro-era name from gitignore,
    change create-new-repo to work with a second argument that is actually
    a URL, and tweak whitespace. Expand description in README and update the
    explanation of OPENSSL_ENABLE_MD5_VERIFY in subdir-btest.cfg.

4.2.0-dev.94 | 2021-08-31 15:58:00 +0200

  * GH-1709: Fix signed integer overflow in PIA on big sequence number
    holes. (Johanna Amann, Corelight)

4.2.0-dev.92 | 2021-08-31 15:55:29 +0200

  * Allow `-B` flag in non-debug builds, but ignore it (unless it's
    `help`). (Benjamin Bannier, Corelight)

4.2.0-dev.89 | 2021-08-26 14:35:28 -0700

  * CI support refresh. (Christian Kreibich, Corelight)
    - Add Debian 11 (Bullseye)
    - Drop Ubuntu 16.04

  * Remove unneccessary >= 0 check in a UTF32 comparison Resolves
    Coverity CID 1461523. (Christian Kreibich, Corelight)

  * Trivial signedness warning fix. (Christian Kreibich, Corelight)

  * Fix addr/string type confusion in Broker::peers(). (Christian
    Kreibich, Corelight)

  * Simplify the supervisor's listen() on default address/port.
    (Christian Kreibich, Corelight)

4.2.0-dev.78 | 2021-08-19 09:39:23 -0700

  * Return fully-escaped string if utf8 conversion fails (Tim Wojtulewicz, Corelight)

    This adds a new function for validating UTF-8 sequences by converting to
    UTF-32. This allows us to also check for various blocks of codepointsi
    that we consider invalid while checking for valid sequences in general.

4.2.0-dev.76 | 2021-08-18 08:40:41 -0700

  * Fix option length computation in Geneve analyzer. (Benjamin Bannier, Corelight)

    We previously computed the length of the Geneve options field
    incorrectly which lead to us passing data at an incorrect offset to
    inner analyzers. With this patch we now interpret the length field
    correctly, according the the spec
    https://datatracker.ietf.org/doc/html/rfc8926#section-3.4.

    Closes #1726.

4.2.0-dev.73 | 2021-08-12 09:57:59 -0700

  * GH-1713: Avoid unneeded reallocs in SerializationFormat::WriteData (Tim Wojtulewicz, Corelight)

4.2.0-dev.70 | 2021-08-06 13:29:30 -0700

  * Do not lookup ignore_checksums_nets for every packet (Johanna Amann, Corelight)

    This could lead to a noticeable (single-percent) performance
    improvement.

    Most of the functionality for this is in the packet analyzers that now
    cache ignore_chesksums_nets.

    Based on a patch by Arne Welzel (Corelight).

4.2.0-dev.68 | 2021-08-04 09:57:32 +0100

  * Use unordered_map to store sessions for performance reasons. This might lead to an 8-9% speedup of Zeek.
    See GH-1706 for details. (Tim Wojtulewicz, Corelight)

4.2.0-dev.64 | 2021-08-03 10:23:41 +0100

  * Under certain circumstances, Zeek processes could get into an infinite looping state inside RotationTimer.
    This is fixed by handling special cases of rotation happening exactly on the time boundary, and fixing
    a special case of timer expiration. Fixes GH-1689. (Sowmya Ramapatruni, Corelight)

4.2.0-dev.62 | 2021-08-03 10:21:18 +0100

  * Fix some HTTP evasions. Now HTTP packets are correctly parsed, when CRLF is missing on a multipart
    boundary / at packet boundaries. Fixes GH-1598 (jerome Grandvalet)

4.2.0-dev.59 | 2021-08-02 11:10:04 -0700

  * Fix failing test due to patricia change (Tim Wojtulewicz, Corelight)

  * Rebase patricia code on upstream version (Tim Wojtulewicz, Corelight)

4.2.0-dev.56 | 2021-07-30 08:52:49 -0700

  * GH-1654: Exclude the .git directory when installing cmake files (Tim Wojtulewicz, Corelight)

4.2.0-dev.54 | 2021-07-29 13:23:51 -0700

  * GH-1692: Add some safety to ASN's binary_to_int64 to avoid bad shifts (Tim Wojtulewicz, Corelight)

4.2.0-dev.50 | 2021-07-27 09:36:13 -0700

  * Fix a use-after-free during shutdown (Tim Wojtulewicz, Corelight)

4.2.0-dev.48 | 2021-07-26 13:03:01 -0700

  * GH-1693: Fix potential crash with elements being modified during robust iteration (Tim Wojtulewicz, Corelight)

  * Update HMAC key used for benchmarking service (Tim Wojtulewicz, Corelight)

4.2.0-dev.45 | 2021-07-23 09:28:49 -0700

  * GH-1684: Ensure that the time gets updated every pass if we're reading live traffic (Tim Wojtulewicz, Corelight)

    This is necessary for e.g. packet sources that don't have a selectable
    file descriptor. They'll always be ready on a very short timeout, but
    won't necessarily have a packet to process. In these case, sometimes
    the time won't get updated for a long time and timers don't function
    correctly.

4.2.0-dev.43 | 2021-07-21 11:41:19 -0700

  * Fix handling of timers when cloning TableVals (Johanna Amann, Corelight)

    When cloning TableVals, a new timer was created for the wrong object
    (the existing TableVal, not the clone). This lead to the already
    existing timer being no longer accessible. Which, in turn, leads to an
    abandoned timer reading into no longer allocated data when the original
    TableVal is deleted.

    Fixes GH-1687

4.2.0-dev.40 | 2021-07-20 09:58:14 -0700

  * Cover in NEWS the inclusion of btest tooling in the installation (Christian Kreibich, Corelight)

    [skip ci]

4.2.0-dev.38 | 2021-07-20 09:55:59 -0700

  * Fix generate-docs github action to send email when it fails (Tim Wojtulewicz, Corelight)

  * Use Cirrus's new greedy mode for parallelizing builds and tests (Christian Kreibich, Corelight)

    This oversubscribes our cores 2x, which testing shows we actually
    run with at times: speedup is around a third on average for builds,
    and a bit more than that for testing.

    Also some light Bashification in ci/build.sh, for consistency.

4.2.0-dev.34 | 2021-07-19 08:56:45 -0700

  * Bump highwayhash to pull in FreeBSD 14 fix (Christian Kreibich, Corelight)

4.2.0-dev.32 | 2021-07-19 08:55:06 -0700

  * Update 3rdparty submodule to pull in doctest to 2.4.6. (Robin Sommer, Corelight)

    Our old version didn't support compilation on Apple's M1 yet.

4.2.0-dev.30 | 2021-07-19 08:54:30 -0700

  * Fix registration of protocol analyzers from inside plugins. (Robin Sommer, Corelight)

    With the recent packet manager work, it broke to register a protocol
    analyzer for a specific port from inside a plugin's initialization code.
    That's because that registration now depends on the packet manager being
    set up, which isn't case at that time a plugin's `InitPostInit()` runs.
    This fix contains two parts:

        - Initialize the packet manager before the analyzer manager, so that
          the latter's `InitPostScript()` can rely on the former being
          ready.

        - Change the analyzer manager to (only) record port registrations
          happening before it's fully initialized. Its `InitPostScript()`
          then performs the actual registrations, knowing it can use the
          packet manager now.

    This comes with a `cmake/` to add a missing include directory.

4.2.0-dev.28 | 2021-07-16 22:43:35 -0700

  * Update broker submodule for new CAF version (Tim Wojtulewicz, Corelight)

  * Setup generate-docs workflow to run on push to test branch (Tim Wojtulewicz, Corelight)

4.2.0-dev.24 | 2021-07-15 11:33:31 -0700

  * Add a TODO to return a correct status for ParseRR_WKS (Vlad Grigorescu)

  * Update scripts/site/local-compat test for 4.2 cycle (Tim Wojtulewicz)

  * Remove tests for deprecated operators. (Robin Sommer, Corelight)

  * Add tests for operations on vectors of string. (Robin Sommer, Corelight)

4.2.0-dev.18 | 2021-07-15 09:50:25 -0700

  * Remove tests for deprecated operators. (Robin Sommer, Corelight)

  * Add tests for operations on vectors of string. (Robin Sommer, Corelight)

4.2.0-dev.14 | 2021-07-14 15:01:20 -0700

  * Fix race conditions in scripts.base.frameworks.input.reread (Christian Kreibich, Corelight)

    On FreeBSD, this test showed two problems: (1) reordering problems
    based on writing the predicate, event, and end-of-data updates into a
    single file, (2) a race condition based on printing the entirety of
    the table description argument in update events. The description
    contains the destination table, and its content at the time an update
    event gets processed isn't deterministic: depending on the number
    of updates the reader thread has sent, the table will contain a
    varying number of entries.

  * Fix race condition in scripts.base.frameworks.input.invalid* (Christian Kreibich, Corelight)

    The invalidset and invalidtext tests loaded an input file via table
    and event reads, in parallel. On FreeBSD this triggers an occasional
    reordering of messages coming out of the reader thread vs the input
    managers. This commit makes the table and event reads sequential,
    avoiding the race.

4.2.0-dev.11 | 2021-07-14 15:00:11 -0700

  * Reorder fields in some classes for more compact memory layout (Tim Wojtulewicz)

  * Redo how reassembled flag is accessed in IP_Hdr, filling in a memory hole (Tim Wojtulewicz)

4.2.0-dev.7 | 2021-07-13 12:23:14 -0700

  * Zeekygen doesn't support comments on set members (Vlad Grigorescu)

  * Add 6to4 addresses, if the matching IPv4 address is private (Vlad Grigorescu)

  * Update Site::private_address_space to latest IANA networks (Vlad Grigorescu)

4.1.0-dev.921 | 2021-07-09 12:01:40 -0700

  * docs: escape * in documentation (Vlad Grigorescu)

  * Update the mail_dest documentation with the new email_dest field (Vlad Grigorescu)

4.1.0-dev.918 | 2021-07-08 18:00:26 -0700

  * Fix a compile warning on FreeBSD (Tim Wojtulewicz, Corelight)


4.1.0-dev.916 | 2021-07-08 16:49:55 -0700

  * Add a cluster controller testcase for agent-controller checkin (Christian Kreibich, Corelight)

    This verifies that in a setup with the supervisor creating both controller and
    agent, the agent successfully checks in with the controller.

  * Add zeek-client via new submodule (Christian Kreibich, Corelight)

    The new module resides in auxil/zeek-client. It does not get installed unless
    one configures with --enable-zeek-client.

  * Introduce cluster controller and cluster agent scripting (Christian Kreibich, Corelight)

    This is a preliminary implementation of a subset of the functionality set out in
    our cluster controller architecture. The controller is the central management
    node, existing once in any Zeek cluster. The agent is a node that runs once per
    instance, where an instance will commonly be a physical machine. The agent in
    turn manages the "data cluster", i.e. the traditional notion of a Zeek cluster
    with manager, worker nodes, etc.

    Agent and controller live in the policy folder, and are activated when loading
    policy/frameworks/cluster/agent and policy/frameworks/cluster/controller,
    respectively. Both run in nodes forked by the supervisor. When Zeek doesn't use
    the supervisor, they do nothing. Otherwise, boot.zeek instructs the supervisor
    to create the respective node, running main.zeek.

    Both controller and agent have their own config.zeek with relevant knobs. For
    both, controller/types.zeek provides common data types, and controller/log.zeek
    provides basic logging (without logger communication -- no such node might
    exist).

    A primitive request-tracking abstraction can be found in controller/request.zeek
    to track outstanding request events and their subsequent responses.

  * Establish a separate init script when using the supervisor (Christian Kreibich, Corelight)

    The supervisor does not require the full weight of scripts that
    init-default.zeek brings with it. The new file, init-supervisor.zeek, contains
    only what's required by the supervisor in addition to the other always-loaded
    init files.

  * Add optional bare-mode boolean flag to Supervisor's node configuration (Christian Kreibich, Corelight)

    When omitted, the node inherits the Supervisor's bare-mode
    status. When true/false, the new Zeek node will enable/disable bare
    mode, respectively. It continues to load any scripts passed at the
    command line and in the additional scripts list already provided in
    the node configuration.

    Includes testcase.

  * Add support for making the supervisor listen for requests (Christian Kreibich, Corelight)

    The supervisor now starts listening on the configured Broker default address and
    port when the new boolean SupervisorControl::enable_listen is T. Listening
    remains disabled by default. Listening allows nodes to communicate with the
    supervisor via the events laid out in control.zeek, to conduct further node
    management.

  * Add support for setting environment variables via supervisor (Christian Kreibich, Corelight)

    The NodeConfig record now has a table for specifying environment variable names
    and values, which the supervisor sets in the created node.

    This also repositions the cpu_affinity member to keep the order the same in
    the corresponding script-layer and in-core types.

    Includes testcase.

4.1.0-dev.907 | 2021-07-08 16:00:06 -0700

  * Fix reference in the logging framework docs re. Log::StreamPolicyHook (Christian Kreibich, Corelight)

4.1.0-dev.905 | 2021-07-08 12:59:01 -0700

  * Update broker submodule for process metrics (Tim Wojtulewicz, Corelight)

    This also updates CAF to 0.18.4

4.1.0-dev.903 | 2021-07-07 10:10:26 -0700

  * GH-1644: Pass a larger structure to be calloc'd in patricia.c (Tim Wojtulewicz, Corelight)

  * Fix multiple spelling errors in debug output (Tim Wojtulewicz, Corelight)

4.1.0-dev.900 | 2021-07-06 08:58:48 -0700

  * Removing a fix from another branch (Seth Hall, Corelight)

  * Update binpac pointer (Seth Hall, Corelight)

  * Update bifcl and binpac for building Zeek as a submodule (Seth Hall, Corelight)

  * Updated the cmake pointer (Seth Hall, Corelight)

  * Fix the builtin plugin preload mechanism (Seth Hall, Corelight)

  * Update submodule pointer for cmake to point at the appropriate branch (Seth Hall, Corelight)

  * Switch to more accurate source and binary references in cmake (Seth Hall, Corelight)

    This is mostly involved with moving CMAKE_SOURCE_DIR and
    CMAKE_BINARY_DIR to PROJECT_SOURCE_DIR and PROJECT_BINARY_DIR because it
    enables zeek to be built as a subproject of another.

    There is another branch in the cmake repository with the same name that
    is also required for this to fully work.

4.1.0-dev.891 | 2021-07-06 08:49:50 -0700

  * Fix a number of Coverity findings (Tim Wojtulewicz, Corelight)

    - 1458048: Use-after-free in the SQLite logger
    - 1457823: Missing a break statement in script-opt reduction
    - 1453966: Dead code in CompHash
    - 1445417: Unintialized variable in StaticHash64
    - 1437716: Unintialized variables in FileInfo in scan.l

4.1.0-dev.889 | 2021-07-06 08:48:50 -0700

  * Add a few DNS query types. (Johanna Amann, Corelight)

4.1.0-dev.887 | 2021-07-06 08:47:22 +0200

  * Add some more items to NEWS. (Robin Sommer, Corelight)

4.1.0-dev.884 | 2021-07-05 10:06:59 +0100

  * Add ability to check if hostname is valid for a specific cert (Johanna Amann, Corelight)

    This commit adds two new bifs, x509_check_hostname and
    x509_check_cert_hostname. These bifs can be used to check if a given
    hostname which can, e.g., be sent in a SNI is valid for a specific
    certificate.

    This PR furthermore modifies the ssl logs again, and adds information
    about this to the log-file. Furthermore we now by default remove the
    server certificate information from ssl.log - I doubt that this is often
    looked at, it is not present in TLS 1.3, we do still have the SNI, and
    if you need it you have the information in x509.log.

    This also fixes a small potential problem in X509.cc assuming there
    might be SAN-entries that contain null-bytes.

  * Add ssl_history field to ssl.log (Johanna Amann)

    This is the equivalent to a connection history for SSL - and contains
    information about which protocol messages were exchanged in which order.

  * Add policy script suppressing certificate events (Johanna Amann, Corelight)

    The added disable-certificate-events-known-certs.zeek disables repeated
    X509 events in SSL connections, given that the connection terminates at
    the same server and used the samt SNI as a previously seen connection
    with the same certificate.

    For people that see significant amounts of TLS 1.2 traffic, this could
    reduce the amount of raised events significantly - especially when a
    lot of connections are repeat connections to the same servers.

    The practical impact of not raising these events is actually very little
    - unless a script directly interacts with the x509 events, everything
    works as before - the x509 variables in the connection records are still
    being set (from the cache).

  * Add new ssl-log-ext policy script (Johanna Amann, Corelight)

    This policy script significantly extends the details that are logged
    about SSL/TLS handshakes.

  * Deprecate extract-certs-pem.zeek and add log-certs-base64.zeek (Johanna Amann, Corelight)

    Extract-certs-pem writes pem files to a dedicated file; since it does
    not really work in cluster-environments it was never super helpful.

    This commit deprecates this file and, instead, adds
    log-certs-base64.zeek, which adds the base64-encoded certificate (which
    is basically equivalent with a PEM) to the log-file. Since, nowadays,
    the log-files are deduplicates this should not add a huge overhead.

  * Implement X509 certificate log caching (Johanna Amann, Corelight)

    By default, each certificate is now output only once per hour. This also
    should work in cluster mode, where we use the net broker-table-syncing
    feature to distribute the information about already seen certificates
    across the entire cluster.

    Log caching is also pretty configureable and can be changed using a
    range of confiuration options and hooks.

    Note that this is currently completely separate from X509 events
    caching, which prevents duplicate parsing of X509 certificates.

  * Deprecate ICSI SSL notary script. (Johanna Amann, Corelight)

    The ICSI notary is pretty much inactive. Furthermore - this approach
    does no longer make much sense at this point of time - performing, e.g.,
    signed certificate timestamp validation is much more worthwhile.

  * Change SSL and X.509 logging format (Johanna Amann, Corelight)

    This commit changes the SSL and X.509 logging formats. Logs are
    now indexed by their hash and no longer by the file ID.

    This commit introduces two new options, which determine if certificate
    issuers and subjects are still logged in ssl.log. The default is to have
    the host subject/issuer logged, but to remove client-certificate
    information. Client-certificates are not a typically used feature
    nowadays.

  * Enable OCSP logging by default. (Johanna Amann, Corelight)

    It turns out that this can actually contain a slew of interesting
    information - like operating systems querying for the revocation of
    software signing certificates, e.g.

  * Split the code that handles X509 event hashing into its own file (Johanna Amann, Corelight)

    This also improves documentation of this feature.

4.1.0-dev.864 | 2021-07-02 12:50:40 -0700

  * Add a global log policy hook to the logging framework (Christian Kreibich, Corelight)

    This addresses the need for a central hook on any log write, which
    wasn't previously doable without a lot of effort. The log manager
    invokes the new Log::log_stream_policy hook prior to any filter-specific
    hooks. Like filter-level hooks, it may veto a log write. Even when
    it does, filter-level hooks still get invoked, but cannot "un-veto".

    Includes test cases.

4.1.0-dev.861 | 2021-07-02 11:58:01 -0700

  * Use mallinfo2() instead of mallinfo() when available (Christian Kreibich, Corelight)

    glibc 2.33 deprecates mallinfo in favor of a struct that returns
    its members as size_ts instead of ints.

4.1.0-dev.859 | 2021-07-02 10:16:25 -0700

  * Update Broker submodule (Dominik Charousset, Corelight)

4.1.0-dev.856 | 2021-07-02 13:01:05 +0100

  * Update Mozilla CA list and CT anchors. (Johanna Amann, Corelight)

    This commit switches to only allowing the CT logs that are currently
    accepted by Google Chrome - which makes much more sense for us since
    this is (potentially) used for validation.

    Additional CT logs can be added in user-scripts.

  * Minor NEWS formatting fixes (Tim Wojtulewicz, Corelight)

4.1.0-dev.852 | 2021-07-01 08:46:41 -0700

  * low-level coding style fixes (Vern Paxson, Corelight)

  * support for standalone compiled scripts to export globals with module qualifiers (Vern Paxson, Corelight)

  * updates for documentation of functionality for compiling scripts to C++ (Vern Paxson, Corelight)

  * fixes for standalone C++ scripts making types & variables/functions available (Vern Paxson, Corelight)

  * fixed bug limiting availability of load_CPP() BiF (Vern Paxson, Corelight)

  * updates to development helper scripts to support new workflow (Vern Paxson, Corelight)

  * simpler workflow for -O gen-C++ ; also some hooks for -O gen-standalone-C++ (Vern Paxson, Corelight)

  * ReplaceBody now deletes a body if the replacement is nil (Vern Paxson, Corelight)

  * removal of can't-actually-be-executed code (Vern Paxson, Corelight)

4.1.0-dev.842 | 2021-06-30 20:32:37 -0700

  * Skip input framework entries with missing but non-optional fields (Christian Kreibich, Corelight)

    The framework so far populated data structures with missing fields
    even when those fields are defined without the &optional
    attribute. When using the attribute, such entries continue to get
    populated.

    Update tests to reflect focus on unset fields.

  * Fix segfault in input framework when reading unset fields (Christian Kreibich, Corelight)

    Unset fields could trigger unexpected null pointers in the input
    manager. This also adds a warning such fields come up.

4.1.0-dev.837 | 2021-06-30 14:10:58 -0700

  * Fix tests (Seth Hall, Corelight)

  * Removed a non-functional builtin plugin preload loading mechanism (Seth Hall, Corelight)

  * Load the builtin-plugin preload files and fix the dev path. (Seth Hall, Corelight)

4.1.0-dev.833 | 2021-06-30 08:50:39 -0700

  * GH-1500: Mark BasicThread::Done to be ignored by ThreadSanitizer (Tim Wojtulewicz, Corelight)

    There's a known false positive with the atomic variables in this method
    that triggers a complaint from ThreadSanitizer. Marking it as ignored
    avoids the warning.

4.1.0-dev.831 | 2021-06-30 13:17:01 +0100

  * Update the bundled SQLite to 3.36.0 (Johanna Amann, Corelight)

4.1.0-dev.830 | 2021-06-30 07:51:08 +0200

  * GH-1406: Fix SMB tests on Apple M1. (Robin Sommer, Corelight)

4.1.0-dev.828 | 2021-06-29 11:11:22 -0700

  * Remove remaining vestigial IDMEF support code (Christian Kreibich, Corelight)

4.1.0-dev.826 | 2021-06-29 10:39:18 -0700

  * made RecordVal::AppendField protected: it's low-level & requires knowledge of internals (Vern Paxson, Corelight)

4.1.0-dev.821 | 2021-06-29 08:38:37 -0700

  * Fix package name for CMake on CentOS 8 (Dominik Charousset, Corelight)

4.1.0-dev.818 | 2021-06-28 13:50:13 -0700

  * GH-1216: Enable Mobile IPv6 support by default (Tim Wojtulewicz, Corelight)

    This removes the ENABLE_MOBILE_IPV6 #define variable. It also marks the
    --enable-mobile-ipv6 configure argument as deprecated.

4.1.0-dev.816 | 2021-06-28 11:08:29 -0700

  * GH-572: Mark MemoryAllocation() and related methods deprecated (Tim Wojtulewicz, Corelight)

4.1.0-dev.814 | 2021-06-28 11:06:39 -0700

  * Check for -1 return from FieldOffset() in Val::HasField()

    Fixes Coverity 1457804 (Tim Wojtulewicz, Corelight)

4.1.0-dev.812 | 2021-06-28 11:02:46 -0700

  * whoops overlooked the need to canonicalize filenames (Vern Paxson, Corelight)

  * another set of tweaks per review comments (Vern Paxson, Corelight)

  * addressed a number of code review comments (Vern Paxson, Corelight)

  * baseline updates for merge (Vern Paxson, Corelight)

  * Merge remote-tracking branch 'origin/master' into topic/vern/ZAM-prep (Vern Paxson, Corelight)

  * support "any" coercions for "-O gen-C++" (Vern Paxson, Corelight)

  * better descriptions for named record constructors (Vern Paxson, Corelight)

  * test suite baseline updates for "-a opt" optimize-AST alternative (Vern Paxson, Corelight)

  * test suite baseline updates for "-a xform" alternative / AST transformation (Vern Paxson, Corelight)

  * error propagation fix for AST reduction (Vern Paxson, Corelight)

  * updates to "-a inline" test suite alternative baseline (Vern Paxson, Corelight)

  * updates for the main test suite baseline (Vern Paxson, Corelight)

  * updates to test suite tests for compatibility with upcoming ZAM functionality (Vern Paxson, Corelight)

  * "-O compile-all" option to specify compilation of inlined functions (Vern Paxson, Corelight)

  * compile inlined functions if they're also used indirectly (Vern Paxson, Corelight)

  * provide ZAM-generated code with low-level access to record fields (Vern Paxson, Corelight)

  * fix for cloning records with fields of type "any" (Vern Paxson, Corelight)

  * direct access for ZAM to VectorVal internal vector (Vern Paxson, Corelight)

  * ZVal constructors, accessors & methods in support of ZAM (Vern Paxson, Corelight)

  * switch ZVal representation of types from Type objects to TypeVal's (Vern Paxson, Corelight)

  * revised error-reporting interface for ZVal's, to accommodate ZAM inner loop (Vern Paxson, Corelight)

  * faster construction of records by factoring static decisions into RecordType's (Vern Paxson, Corelight)

  * make "switch" internals accessible to ZAM; tidying of same (Vern Paxson, Corelight)

  * factor out "cast" functionality to make available to lower-level ZAM access (Vern Paxson, Corelight)

  * tidying for check_and_promote_expr (Vern Paxson, Corelight)

  * employ explicit conversions to/from "any" and "vector of any" types (Vern Paxson, Corelight)

  * more robust treatment of arithmetic coercions (Vern Paxson, Corelight)

  * support for constructing VectorVal's directly from underlying ZVal vectors (Vern Paxson, Corelight)

  * support for ensuring that a vector can be treated as having a homogeneous type (Vern Paxson, Corelight)

  * factoring out of low-level vector indexing to make available to ZAM (Vern Paxson, Corelight)

  * minor changes for more robust behavior in the face of errors (Vern Paxson, Corelight)

  * gracefully deal with "eval" exceptions that occur during AST reduction (Vern Paxson, Corelight)

  * directly construct records of known types, rather than requiring coercion (Vern Paxson, Corelight)

  * fixes for treating WhileStmt's "loop_cond_pred_stmt" as a first-class citizen (Vern Paxson, Corelight)

  * support for profiling function bodies w/o needing accompanying ScriptFunc object (Vern Paxson, Corelight)

  * support for Frame's having call locations even if no associated CallExpr (Vern Paxson, Corelight)

  * fix for AST optimization altering top-level body statement (Vern Paxson, Corelight)

  * fix for analyzing variable usage inside of table initializers (Vern Paxson, Corelight)

  * fix for inlining type-based switch statements (Vern Paxson, Corelight)

  * fix for computing |size| of files and subnets (Vern Paxson, Corelight)

  * fix for tracking the effects of += operations (Vern Paxson, Corelight)

  * Expr method to invert the sense of a relational (Vern Paxson, Corelight)

  * Trigger constructor (and factoring) to support lower-level constructions (Vern Paxson, Corelight)

  * fixed / removed out-of-date comments, tidied check_and_promote_args() interface (Vern Paxson, Corelight)

  * convert scopes to be managed using IntrusivePtr's (Vern Paxson, Corelight)

  * various accessors used by ZAM compiler (Vern Paxson, Corelight)

  * Update submodule(s) [nomail] (Tim Wojtulewicz, Corelight)

4.1.0-dev.764 | 2021-06-27 10:50:19 -0700

  * Remove unnecessary -B arguments from Zeek invocations in testsuite

    Now that Zeek no longer silently accepts -B when not compiled in debug
    mode, these tests were failing. (Christian Kreibich, Corelight)

  * Fix perftools-enabled build (Christian Kreibich, Corelight)

  * Minor tweaks to single-character command line option handling

    - Use of "-B" now triggers usage output and errors out when Zeek
      wasn't built with debugging support.

    - Always keep the perftools options (-m, -M) in the optparse string,
      for consistency with other flags dependent on configuration. We
      still fall through to usage and erroring out as before.

    - Minor indentation fix. (Christian Kreibich, Corelight)

4.1.0-dev.760 | 2021-06-27 10:46:01 -0700

  * Add btest test case (Luke Cesarz)

  * Fix typo (Luke Cesarz)

  * Fix segfault with incomplete connection

    Add required HasField check before GetFieldAs call (Luke Cesarz)

4.1.0-dev.755 | 2021-06-23 13:53:54 -0700

  * Call brew update-reset in ci/macos/prepare.sh

    This fixes some issues with the Catalina builds when it prepares the image
    and the base Cirrus image has old recipes for Homebrew. The VM then has
    to build a bunch of packages it shouldn't need to. (Tim Wojtulewicz, Corelight)

  * GH-1368: Use --osx-sysroot for macOS CI builds (Tim Wojtulewicz, Corelight)

4.1.0-dev.750 | 2021-06-21 16:14:03 -0700

  * Remove the Stepping Stone analyzer

    This commit removes the stepping stone analyzer. It has been deactivated
    by default since at least Zeek 2.0, is dysfunctional in cluster settings
    and has a bunch of other issued.

    Relates to GH-1573 (Johanna Amann, Corelight)

4.1.0-dev.748 | 2021-06-21 15:41:29 -0700

  * Drop Fedora 32 from CI, now past EOL (Christian Kreibich, Corelight)

  * Remove a double-defined TLS cert from a Broker btest (Christian Kreibich, Corelight)

4.1.0-dev.744 | 2021-06-21 09:17:36 +0200

  * Add a new field `email_dest` to NOTICEs, which defines where to
    send email to. The email-related NOTICE actions fill this now, and
    then emails will be sent to all recorded addresses at the end of
    NOTICE processing. This makes email generation more consistent and
    extensible. (Vlad Grigorescu)

  * Add page and email administrator to mails processed by hostnames
    extension. (Vlad Grigorescu)

4.1.0-dev.731 | 2021-06-17 10:40:58 +0100

  * Change SSH version field to be `&optional`.

    In version 3.3.0-dev.537 we added handling for SSH version 1.99 which
    used a SSH version of 0 to indicate weird cases where no version could be
    determined.

    This patch is a fixup for that patch. We now use an `&optional` version value.
    If no SSH version can be eixtracted the version will be unset; additionally a
    `conn_weird` event will be raised. See GH-1590. (Benjamin Bannier, Corelight)

4.1.0-dev.727 | 2021-06-14 16:19:34 -0700

  * Bump Highwayhash submodule to pull in fix for FreeBSD (Christian Kreibich, Corelight)

4.1.0-dev.725 | 2021-06-11 11:54:30 -0700

  * Fixes for the builtin plugin functionality (Seth Hall, Corelight)

4.1.0-dev.722 | 2021-06-10 10:42:57 -0700

  * Added --include-plugins configure argument (Seth Hall, Corelight)

4.1.0-dev.720 | 2021-06-10 11:29:19 +0100

  * Introduce script-land option LogAscii::logdir that can be used to set
    the logging directory. (Henrik Kramselund Jereminsen)

4.1.0-dev.715 | 2021-06-09 09:12:26 -0700

  * Fix macOS Big Sur builds on Cirrus

    - Upgrade the Big Sur VM to use the Xcode 12.5 version. This has a newer
      version of brew installed on it that fixes an issue with an EOL package host
      that finally shut down for good recently.
    - Use 'brew upgrade' for openssl and cmake, since those are both present on the
      base VM. This prevents 'brew install' from printing an error if the package
      exists but is out of date. (Tim Wojtulewicz, Corelight)

4.1.0-dev.713 | 2021-06-08 13:54:28 -0700

  * Add OpenSUSE Leap 15.3 to testing (Johanna Amann, Corelight)

4.1.0-dev.709 | 2021-06-07 09:41:28 +0200

  * Improve assignment operators for IntrusivePtr. (Dominik Charousset, Corelight)

  * Fix docs for `ProcStats`: `mem` is in bytes, not KB. (Arne Welzel, Corelight)

4.1.0-dev.704 | 2021-06-04 08:29:18 -0700

  * Add deprecated headers for UDP and ICMP analyzers (Tim Wojtulewicz, Corelight)

  * Fix handling of IP packets with bogus IP header lengths

    Credit to OSS-Fuzz for discovery
    https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=34711
    (Link to details becomes public 30 days after patch release) (Tim Wojtulewicz, Corelight)

4.1.0-dev.700 | 2021-06-03 09:27:57 -0700

  * Make update-traces fail when the curl invocation fails (Christian Kreibich, Corelight)

4.1.0-dev.697 | 2021-06-02 15:08:12 -0700

  * Add FreeBSD 13 to CI (Christian Kreibich, Corelight)

  * Add Fedora 34 to CI (Christian Kreibich, Corelight)

4.1.0-dev.693 | 2021-06-02 13:22:09 -0700

  * Label session adapters in the output of zeek -NN (Tim Wojtulewicz, Corelight)

  * Split session adapter code into separate files from the analyzers (Tim Wojtulewicz, Corelight)

  * Move adapter-specific code back into the adapter (Tim Wojtulewicz, Corelight)

  * Move ICMP counterpart methods outside of ICMPAnalyzer class

    These were previously global methods in the old analyzer, and moving them
    to be private members of ICMPAnalyzer broke the usage of them by at least
    one external plugin. (Tim Wojtulewicz, Corelight)

  * Remove obsolete Skipping()/SetSkip() from Connection (Tim Wojtulewicz, Corelight)

  * Remove some code from IPBasedAnalyzer and children that was waiting for TCP to be implemented (Tim Wojtulewicz, Corelight)

  * Move TCPStateStats object out of session_mgr (Tim Wojtulewicz, Corelight)

  * Move analyzer-to-port mapping out of analyzer::Manager into packet analyzers (Tim Wojtulewicz, Corelight)

  * Move packet parsing code out of adapter into analyzer (Tim Wojtulewicz, Corelight)

  * Move old TCP analyzer into analyzer adapter in packet analysis tree (Tim Wojtulewicz, Corelight)

4.1.0-dev.681 | 2021-06-02 09:57:41 -0700

  * Add some extra length checking when parsing mobile ipv6 packets

    Credit to OSS-Fuzz for discovery
    https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=34263
    (Link to details becomes public 30 days after patch release) (Tim Wojtulewicz, Corelight)

4.1.0-dev.679 | 2021-06-01 19:11:40 -0700

  * Replace toupper() usages in netbios decoding BIFs (Jon Siwek, Corelight)

    This avoids potential for locale-dependent results of toupper() by
    instead using a function that simply maps ASCII characters a-z to A-Z.

4.1.0-dev.676 | 2021-06-01 10:19:19 -0700

  * Integrate review feedback (Dominik Charousset, Corelight)

  * Sync new broker options, fix name inconsistencies (Dominik Charousset, Corelight)

  * Integrate new Broker metric exporter parameters (Dominik Charousset, Corelight)

4.1.0-dev.671 | 2021-06-01 09:51:38 -0700

  * Update detect-MHR.zeek (Chris C)

    Update Virustotal URL to current
    option match_sub_url = "https://www.virustotal.com/gui/search/%s"

4.1.0-dev.669 | 2021-06-01 09:39:30 -0700

  * GH-839: Fix use of &optional sub-records within table/set indices (Jon Siwek, Corelight)

4.1.0-dev.666 | 2021-05-26 10:51:51 -0700

  * Ensure SessionAdapter members are initialized (Tim Wojtulewicz, Corelight)

    Fixes Coverity #1453273

4.1.0-dev.665 | 2021-05-26 08:07:26 +0200

  * Extend the file analyzer API to set source manually. (Robin
    Sommer, Corelight)

4.1.0-dev.661 | 2021-05-24 15:03:54 -0700

  * Update Broker submodule for bump of embedded CAF to 0.18.3 (Jon Siwek, Corelight)

4.1.0-dev.660 | 2021-05-24 12:38:44 -0700

  * Add type field to session::Key to help avoid collisions in map (Tim Wojtulewicz, Corelight)

  * Move bad UDP checksum handling into adapter object (Tim Wojtulewicz, Corelight)

  * Rename IPBasedTransportAnalyzer to SessionAdapter (Tim Wojtulewicz, Corelight)

    This also also combines the old TransportLayerAnalyzer class into
    SessionAdapter, and removes the old class. This requires naming changes
    in a few places but no functionality changes.

  * Move building session analyzer tree out of analyzer::Manager (Tim Wojtulewicz, Corelight)

  * Rework the packet flow through the IP-based analyzers (Tim Wojtulewicz, Corelight)

  * Add new UDP packet analyzer, remove old one (Tim Wojtulewicz, Corelight)

  * Add new ICMP packet analyzer, remove old one (Tim Wojtulewicz, Corelight)

  * Add base class for IP-based packet analyzers (Tim Wojtulewicz, Corelight)

  * Move SessionManager::ParseIPPacket to IP analyzer's namespace (Tim Wojtulewicz, Corelight)

  * Added skeletons for TCP/UDP/ICMP packet analysis plugins. (Tim Wojtulewicz, Corelight)

    This includes integration into the IP plugin and calling of the sessions code from each plugin.

4.1.0-dev.646 | 2021-05-18 11:47:25 -0700

  * Omit unneeded decimal points in modp_dtoa2() scientific notation output (Jon Siwek, Corelight)

    For example, "1e-13" is now used instead of "1.e-13".

  * GH-1244: Change modp_dtoa2() to use scientific notation for small values (Jon Siwek, Corelight)

    This fixes problems where printing floating point numbers less than
    10^-6 output as "0.0".  Such numbers now use using scientific notation
    and preserve the value's actual floating point representation.

4.1.0-dev.643 | 2021-05-17 11:57:58 -0700

  * GH-1546: Make DictIterator() public, add copy/move operators (Tim Wojtulewicz, Corelight)

4.1.0-dev.641 | 2021-05-17 11:28:11 -0700

  * GH-1558: Fix reading `vector of enum` types from config files (Jon Siwek, Corelight)

  * GH-1555: Fix reading empty set[enum] values from config files (Jon Siwek, Corelight)

4.1.0-dev.638 | 2021-05-17 13:08:28 +0100

  * Manual page updates (Henrik Kramselund Jereminsen)

4.1.0-dev.631 | 2021-05-11 09:26:37 -0700

  * Add unit tests to ZeekString.cc (Tim Wojtulewicz)

4.1.0-dev.628 | 2021-05-10 12:44:25 -0700

  * Add experimental support for translating Zeek scripts to equivalent C++ (Vern Paxson, Corelight)

    The generated C++ can then be compiled directly into the `zeek` binary,
    replacing use of the interpreter and producing better runtime performance.
    See `src/script_opt/CPP/README.md` for a guide on how to use this feature.

  * Add new "-a cpp" btest alternative (Vern Paxson, Corelight)

  * Add VectorVal methods to leverage ZVal representation (Vern Paxson, Corelight)

  * Fix backtrace BiF to avoid iterator invalidation & support compiled code (Vern Paxson, Corelight)

4.1.0-dev.593 | 2021-05-10 10:17:34 +0100

  * Explain zeek-config options in help output (Christian Kreibich, Corelight)

  * Sort variables at top of zeek-config alphabetically (Christian Kreibich, Corelight)

  * Install Zeek's btest tooling with the distribution

    This creates $PREFIX/share/btest in the install tree, with the
    following folders:

    - scripts/ for the canonifiers
    - data/ for random.seed
    - data/pcaps for the test pcaps

    The pcaps can be skipped by configuring with --disable-btest-pcaps. (Christian Kreibich, Corelight)

4.1.0-dev.587 | 2021-05-05 14:05:51 +0000

  * Merge remote-tracking branch 'origin/topic/timw/session-coverity'

    * origin/topic/timw/session-coverity:
      Minor cleanup in IPAddr.h
      Fix a few Coverity warnings from the session manager work (Tim Wojtulewicz)

  * Minor cleanup in IPAddr.h (Tim Wojtulewicz, Corelight)

  * Fix a few Coverity warnings from the session manager work

    - Be explicit about setting the copied flag in session::Key. Coverity seems
      confused about when that flag is set if it gets set by default
      initialization. This should fix 1452757 and 1452759.
    - Explicitly copy the fields in ConnKey instead of using memcpy. Fixes
      1452758. (Tim Wojtulewicz, Corelight)

4.1.0-dev.583 | 2021-05-03 18:21:33 -0700

  * switched RecordVal's to use std::optional for tracking missing fields (Vern Paxson, Corelight)

  * added constructors for directly building ZVal's (Vern Paxson, Corelight)

4.1.0-dev.580 | 2021-04-30 18:29:22 -0700

  * Add missing zeek/ prefix to a telemetry header's includes (Jon Siwek, Corelight)

4.1.0-dev.578 | 2021-04-30 09:42:26 -0700

  * GH-1534: Fix excessive coredump for duplicate enum definitions

    An adequate error message was previously reported for duplicate enum
    definitions, this just now prevents trying to access it as a constant in
    subsequent parsing and further generating a coredump. (Jon Siwek, Corelight)

4.1.0-dev.576 | 2021-04-30 09:40:18 -0700

  * Fixes to `decode_netbios_name` and `decode_netbios_name_type` BIFs

    Fixes to `decode_netbios_name`:

    * Improve validation that input string is a NetBIOS encoding
      (32 bytes, with characters ranging from 'A' to 'P').  This helps
      prevent Undefined Behavior of left-shifting negative values.
      Invalid encodings now cause a return-value of an empty string.

    * More liberal in what decoded characters are allowed.  Namely,
      spaces are now allowed (but any trailing null-bytes and spaces
      are trimmed, similar to before).

    Fixes to `decode_netbios_name_type`:

    * Improve validation that input string is a NetBIOS encoding
      (32 bytes, with characters ranging from 'A' to 'P').  This helps
      prevent Undefined Behavior of left-shifting negative values and
      a heap-buffer-overread when the input string is too small.
      Invalid encodings now cause a return-value of 256. (Jon Siwek, Corelight)

4.1.0-dev.573 | 2021-04-29 11:29:39 -0700

  * Rename ConnID and ConnIDKey (Tim Wojtulewicz, Corelight)

  * Remove Session prefix from some session-related classes and files (Tim Wojtulewicz, Corelight)

  * Move session code into new directory and into zeek::session namespace (Tim Wojtulewicz, Corelight)

  * Move SessionKey into a separate file, added comments (Tim Wojtulewicz, Corelight)

  * Review cleanup

    - Add constructors for ConnIDKey, remove BuildConnIDKey()
    - Rename protocol stats classes and move to implementation file
    - Rename "num" field of protocol stats to "active"
    - Explicitly delete copy operations for SessionKey
    - Change argument for ProtocolStats methods to const-reference
    - Make key validity methods in Session not be virtual
    - Rename Session::ClearKey and Session::IsKeyValid (Tim Wojtulewicz, Corelight)

  * Rework stats to store handles to the counters (Tim Wojtulewicz, Corelight)

  * Rename some connection-specific methods in Session and SessionManager (Tim Wojtulewicz, Corelight)

  * Add deprecated version of Sessions.h (Tim Wojtulewicz, Corelight)

  * Rename NetSessions to SessionManager

    This also includes:
    - Deprecating the NetSessions name.
    - Renaming the zeek::sessions global to zeek::session_mgr and deprecating the old name.
    - Renaming Sessions.{h,cc} to SessionManager.{h,cc}. (Tim Wojtulewicz, Corelight)

  * Store a single map of Sessions instead of split maps of Connections.

    This commit also includes:
    - Storing the transport protocol in ConnID and ConnIDKey to allow tcp and
      udp connections from the same IP/Port combinations. This happens in the
      core.cisco-fabric-path test, for example.
    - Lots of test updates. The reasons for these are two fold. First, with
      the change to only store a single map means that TCP, UDP, and ICMP
      connections are now mixed. When Zeek drains the map at shutdown, it drains
      each of those protocols together instead of separately. The second is
      because of how Sessions are stored in the map. We're now storing them
      keyed by the hash of the key stored by the Session objects, which causes
      them to again be in the map in a different order. (Tim Wojtulewicz, Corelight)

  * Convert session stats to use the new telemetry API (Tim Wojtulewicz, Corelight)

  * Add test for get_conn_stats BIF before reworking session stats (Tim Wojtulewicz, Corelight)

  * Add new Session base class

    This is mostly code copied from the existing Connection class, as that class now
    inherits from Session. (Tim Wojtulewicz, Corelight)

  * Move packet filter out of NetSessions (Tim Wojtulewicz, Corelight)

  * Move some code out of NetSessions

    - TCPStateStats update when a session is removed was moved to Connection
    - Stepping Stone manager moved to a singleton object in SteppingStoneManager (Tim Wojtulewicz, Corelight)

4.1.0-dev.557 | 2021-04-29 09:00:51 -0700

  * Add basic testing for Geneve protocol analyzer

    The added pcap file was downloaded from an attachment to
    https://gitlab.com/wireshark/wireshark/-/issues/10193 without explicit
    license. (Benjamin Bannier, Corelight)

  * GH-1517: Add Geneve decap support

    This patch adds the ability to decap Geneve packets to process the inner
    payload. The structure of the analyzer borrows heavily from the VXLAN
    analyzer. (Benjamin Bannier, Corelight)

4.1.0-dev.554 | 2021-04-28 13:45:53 -0700

  * Add a fatal error condition for invalid Dictionary insertion distances (Jon Siwek, Corelight)

    When choosing poor/aggressive values for `table_expire_interval`,
    `table_expire_delay`, and/or `table_incremental_step` that tend to
    leave tables in state of constant table-expiry-iteration, the underlying
    Dictionary is never allowed the chance to complete remapping operations
    which re-position entries to more ideal locations (e.g. after
    reallocating the table to be able to store more entries).

    That situation not only leads to the Dictionary generally having a less
    efficient structure, but eventually, the lack of re-positioning may
    cause an insertion to calculate the new entry's
    distance-from-ideal-position to be a value requiring a full 16-bits or
    more (>=65535), but an entry only allows storing 16-bit distance values,
    with 65535 being a sentinel value that is supposed to indicate an empty
    entry.  Dictionary operations may start misbehaving if that's allowed to
    happen.

  * Fix using clear_table() within an &expire_func (Jon Siwek, Corelight)

    This previously crashed since clear_table()/TableVal::RemoveAll() left
    behind a stale iterator to the old table causing a heap-use-after-free
    when resuming table expiry iteration in TableVal::DoExpire().

  * Remove saving/restoring of value pointer after calling expire_func (Jon Siwek, Corelight)

    It's no longer used for anything.  Previously, it was used to detect
    whether the expiry batch finished iterating the entire table or not, but
    that's now determined by directly checking if the iterator itself
    signifies the end of the table.

  * Avoid allocating a HashKey for no-op table expiry iterations (Jon Siwek, Corelight)

4.1.0-dev.549 | 2021-04-28 13:09:30 -0700

  * Fix -Wsign-compare warnings in Debug{Cmds}.cc (Jon Siwek, Corelight)

4.1.0-dev.547 | 2021-04-28 09:27:15 -0700

  * GH-1528: Remove broken Queue/PQueue class, replace with std::deque (Tim Wojtulewicz, Corelight)

4.1.0-dev.545 | 2021-04-26 11:14:49 -0700

  * GH-1483: Improve error for mismatched container initialization types (Jon Siwek, Corelight)

    For example, trying to assign a vector, table, set, or record
    constructor expression to a global variable of a different type now
    provides a more explanatory error message than the previous
    "Val::CONVERTER" fatal-error and coredump.

4.1.0-dev.543 | 2021-04-26 09:53:53 -0700

  * Update btest baselines for --enable-mobile-ipv6 builds (Tim Wojtulewicz, Corelight)

  * Build ubuntu18 on CI with --enable-mobile-ipv6 (Tim Wojtulewicz, Corelight)

4.1.0-dev.539 | 2021-04-20 15:21:03 -0700

  * Fix LGTM finding in zeek-cut with uses of localtime and gmtime (Tim Wojtulewicz, Corelight)

4.1.0-dev.537 | 2021-04-20 14:34:48 -0700

  * Restore support for vectors with holes and improve test cases (Vern Paxson, Corelight)

    There's no longer breaking behavior changes to how vector-holes work from
    previous Zeek versions.

  * Fix cloning/copying vectors that contain holes (Vern Paxson, Corelight)

  * fix using ++/-- to vectors that contain holes (Vern Paxson, Corelight)

4.1.0-dev.526 | 2021-04-16 16:03:06 -0700

  * GH-1506: Fix Broker unserialization of set/table function indices (Jon Siwek, Corelight)

    Zeek function types are serialized as a broker::vector, but the
    unserialization logic for Zeek set/table types that use a function for
    as an index incorrectly identified it as a composite-index, which also
    use broker::vector, rather than a singleton-index, and makes such
    unserialization fail.

    A general example where this failure can happen in practice is when
    trying to unserialize a connection record for which there's a
    Conn::RemovalHook, since that's a set[function], and a specific case of
    that is use of the Intel Framework in a Zeek cluster.

4.1.0-dev.524 | 2021-04-16 08:08:38 -0700

  * Move an assert() in input/Manager.cc to account for ValueToVal errors (Jon Siwek, Corelight)

  * Add test for config framework (Tim Wojtulewicz, Corelight)

  * Fix similar issues with ValueTo* methods in the input framework (Tim Wojtulewicz, Corelight)

  * GH-1487: Handle error from ValueToVal instead of ignoring it (Tim Wojtulewicz, Corelight)

4.1.0-dev.519 | 2021-04-15 17:44:16 -0700

  * Fix indexing of set/table types with a vector (Jon Siwek, Corelight)

    That previously caused an internal error/crash

  * GH-1507: Tolerate junk data before SIP requests (Jon Siwek, Corelight)

    This allows for data that won't match a SIP request method to precede an
    actual request and generates a new 'sip_junk_before_request' weird when
    encountering such a situation.

  * GH-1503: Improve &expire_func compatibility type-checking (Jon Siwek, Corelight)

    Previously, incompatible &expire_funcs could mistakenly be used, such as
    when using that attribute on the unspecified table()/set()
    initializations/assignments, resulting in invalid function calls that
    eventually crash Zeek.

4.1.0-dev.512 | 2021-04-15 11:54:44 -0700

  * GH-1496: Fix build on armv7 architecture (Jon Siwek, Corelight)

4.1.0-dev.510 | 2021-04-14 16:05:55 -0700

  * GH-1450: Improve printing/logging of large double/interval/time values

    The modp_dtoa/modp_dtoa2 functions aren't capable of handling double
    values larger than INT_MAX and fallback on using sprintf() in that
    situation.  Previously, the format string to that sprintf() was "%e",
    defaulting to a precision of 6, which is already too few digits to
    represent a number known to be larger than INT_MAX.  Now, an sprintf()
    is still performed for values larger than INT_MAX and still uses a
    scientific notation format, but in a way that uses as many decimal
    digits as needed to preserve information. (Jon Siwek, Corelight)

4.1.0-dev.508 | 2021-04-14 15:54:03 -0700

  * Add btest for unterminated pattern parsing error behavior

    And clarified the error message that it's more about finding an
    unterminated pattern than knowing for sure there's remaining pattern
    text spanning multiple lines. (Jon Siwek, Corelight)

  * GH-1497: Support CRLF line-endings in Zeek scripts and signature files (Jon Siwek, Corelight)

  * Escape non-printables in "unrecognized character" parser error messages (Jon Siwek, Corelight)

4.1.0-dev.504 | 2021-04-14 15:51:39 -0700

  * Add a check for null packet data in pcap IOSource

    Some libpcaps (observed in Myricom's) may claim to have read a packet,
    but either did not really read a packet or at least provide no way
    to access its contents, so this adds a check for null-data to
    handle those cases. (Jon Siwek, Corelight)

4.1.0-dev.501 | 2021-04-12 18:03:53 -0700

  * Add explanation of vector holes/in-operator changes to NEWS (Jon Siwek, Corelight)

4.1.0-dev.500 | 2021-04-09 12:18:06 -0700

  * Reference the manual's explanation of originator and responder from conn_id (Christian Kreibich, Corelight)

4.1.0-dev.498 | 2021-04-07 18:30:58 -0700

  * Disable Broker tests in Coverity Scan cronjob build (Jon Siwek, Corelight)

4.1.0-dev.497 | 2021-04-07 15:54:21 -0700

  * GH-1493: Fix build with -DENABLE_MOBILE_IPV6 (Tim Wojtulewicz, Corelight)

4.1.0-dev.495 | 2021-04-07 11:12:13 -0700

  * Change TableVal::ToMap() to return ValPtr-indexed maps (Jon Siwek, Corelight)

  * Add std::hash specialization for IntrusivePtr (Jon Siwek, Corelight)

  * Move IntrusivePtr relational operators to zeek namespace

    Otherwise some cases relying on argument-dependent lookup (ADL) fail. (Jon Siwek, Corelight)

4.1.0-dev.490 | 2021-04-05 14:13:48 -0700

  * Add Telemetry API (C++/BIFs) for gathering runtime metrics (Dominik Charousset, Corelight)

    This relies on the CAF metrics API/implementation and allows potential
    export to Prometheus.  These typical metric types are supported:
    counters, gauges, histograms, timers.

4.1.0-dev.475 | 2021-04-03 09:39:10 -0700

  * Increase timeout of plugins.reader btest (Jon Siwek, Corelight)

  * optionally hash original, not extended, form of records for profiling (Vern Paxson, Corelight)

    profile types associated with global initializations
    incorporate type name, if present, into type hash

  * fixed out-of-range enum constant: OpenFlow::INVALID_COOKIE (Vern Paxson, Corelight)

    (Though it's not currently an enum value)

  * Add method to add a fully-qualified enum name (module name included) (Vern Paxson, Corelight)

  * enum types track whether they've had values added via "redef" (Vern Paxson, Corelight)

  * tracking of original size of records (pre redef'ing) (Vern Paxson, Corelight)

  * Added TableVal::ToMap to retrieve a table's entire contents as a unordered_map (Vern Paxson, Corelight)

  * factoring out some replicated code in the parser (Vern Paxson, Corelight)

  * some micro-preening of parse.y; no semantic changes (Vern Paxson, Corelight)

  * for readability, removed explicit scoping in parse.y; no semantic changes (Vern Paxson, Corelight)

4.1.0-dev.461 | 2021-04-01 14:11:44 -0700

  * function profiling rewritten - more detailed info, supports global profiling (Vern Paxson, Corelight)

    Hashes for Zeek script types are now done globally rather than
    per-function-body, which can save considerable time due to the complexity
    of some commonly used types (such as connection records).

    Hashing has been expanded to provide more robust distinctness
    (lack of collisions in practice) and determinism (consistently computing
    the same hash across compilations).

  * track whether a given function/body should be included/skipped for optimization (Vern Paxson, Corelight)

4.1.0-dev.451 | 2021-03-31 11:58:08 -0700

  * Add ssh to Alpine Dockerfile for retrieving external test repos (Jon Siwek, Corelight)

4.1.0-dev.449 | 2021-03-31 10:47:22 -0700

  * Change ci/init-external-repos.sh to use `base64 -d` (Jon Siwek, Corelight)

    The BusyBox version of `base64` does not have `--decode`.

4.1.0-dev.448 | 2021-03-31 10:29:17 -0700

  * Teach ci/init-external-repos.sh to consider user permissions (Jon Siwek, Corelight)

    Any errors while setting up external/private test repo should fail the
    task for PRs submitted by a user with write/admin permission.

4.1.0-dev.447 | 2021-03-31 09:55:05 -0700

  * CI: Add OpenSUSE Leap 15.2 (Johanna Amann, Corelight)

  * Add CI task for Alpine Linux (Jon Siwek, Corelight)

  * Separate stdout from stderr in btest baselines (Jon Siwek, Corelight)

    Redirecting both to the same file can show platform-specific differences
    (e.g. Alpine), likely due to different buffering defaults.

  * Remove newline-eof canonification attempt in diff-remove-timestamps (Jon Siwek, Corelight)

  * Change a <sys/errno.h> include to <errno.h> (Jon Siwek, Corelight)

    Some systems (e.g. Alpine) may warn that that the former is incorrect.

4.1.0-dev.440 | 2021-03-30 14:57:07 -0700

  * Fix incomplete-type for struct timeval (Andrew Benson)

    This fixes building on musl (e.g. Void, Alpine, etc.)

4.1.0-dev.438 | 2021-03-29 15:42:25 -0700

  * "balance" tests with multiple Zeek scripts to load the same elements (Vern Paxson, Corelight)

  * put global statements into a quasi-function to support script optimization (Vern Paxson, Corelight)

4.1.0-dev.434 | 2021-03-29 13:18:18 -0700

  * Fix sign-compare compiler warning in coerce_to_record() (Jon Siwek, Corelight)

  * Fix maybe-uninitialized warning in ZVal::ToVal() (Jon Siwek, Corelight)

  * Change RecordVal::GetFieldAs() to use std::vector::operator[] (Jon Siwek, Corelight)

    Since the method claims it's up to the user to ensure the field exists
    before calling, the extra bounds-checking done by std::vector::at()
    isn't needed.

  * Add RecordVal::AssignField() and use it in supervisor code (Jon Siwek, Corelight)

    This is a convenience method to assign a known record field value by
    field name.  May also be useful to reduce warnings from static analysis
    (e.g. Coverity) about not checking for negative return values before
    assigning since that now flows through a [[noreturn]] error path.

  * GH-960: Fix include order of bundled header files (Jon Siwek, Corelight)

    Previously, a system-wide installation of any bundled auxil/ software
    (like CAF) may get found/included rather than the bundled version and
    possibly break the build.

4.1.0-dev.427 | 2021-03-27 14:18:16 -0700

  * Update bundled CAF to 0.18.2 (Jon Siwek, Corelight)

4.1.0-dev.426 | 2021-03-26 17:17:45 -0700

  * GH-1463: Rename supervisor_rotation_format_func to archiver_rotation_format_func (Vlad Grigorescu)

    And expose it for non-supervised setups.  Also deprecates the old name.

4.1.0-dev.423 | 2021-03-26 16:44:26 -0700

  * GH-1454: Heartbleed: fix substraction order. (Johanna Amann)

    Fixes incorrect/overflowed `n` value for `SSL_Heartbeat_Many_Requests`
    notices where number of server heartbeats is greater than number of client
    heartbeats.

    The larger number was substracted from the smaller one leading to an
    integer overflow. However, no information was lost due to everything
    also being present in the notice message.

4.1.0-dev.421 | 2021-03-26 16:39:22 -0700

  * Fix crash in Analyzer::ForwardPacket due to recursive analyzer calls. (Tim Wojtulewicz, Corelight)

    The change in 44f558df7b5a85bae40945de653bcb2448e0a7f4 that made analyzer_list
    a std::vector instead of a std::list doesn't take into account that in some
    cases an analyzer may chain back into itself, such as with UDP-in-UDP tunnels.
    In these cases, the second call to ForwardPacket may cause iterator
    invalidation, leading to a crash, so this reverts back to using an std::list.

  * Include git sha in request to benchmark host (Tim Wojtulewicz, Corelight)

4.1.0-dev.417 | 2021-03-25 11:37:55 -0700

  * test suite update due to factoring out coerce_to_record() (Vern Paxson, Corelight)

  * removal of vestigial #include's (breaking an include loop in the process) (Vern Paxson, Corelight)

  * better method name: HasCopySemantics() (Vern Paxson, Corelight)

  * Use STL functionality to initialize coercion map (Vern Paxson, Corelight)

  * comments for factored-out index slice functions (Vern Paxson, Corelight)

  * support for subclassing ScriptFunc's, esp. for alternate lambda closures (Vern Paxson, Corelight)

  * factor out record coercion; modernize management of coercion "map" (Vern Paxson, Corelight)

  * lower-level method for adding fields to records (Vern Paxson, Corelight)

  * factor out "print" statement's execution functionality (Vern Paxson, Corelight)

  * functions for indexing slices and strings (Vern Paxson, Corelight)

  * new function for getting the location, if any, associated with the current call (Vern Paxson, Corelight)

4.1.0-dev.404 | 2021-03-24 16:58:50 -0700

  * Fix missing `user_agent` existence check in smtp/software.zeek (Michael Dopheide)

    Without the check there can be extra reporter.log error noise, but
    no actual difference in functionality.

4.1.0-dev.402 | 2021-03-24 15:49:52 -0700

  * Add `bin_dir` setting for default zkg config file (Jon Siwek, Corelight)

    Which defaults to `${zeek_install_prefix}/bin`, same as what a
    standalone `zkg autoconfig` would use.

4.1.0-dev.400 | 2021-03-23 20:44:19 -0700

  * whitespace tweaks (Vern Paxson, Corelight)

  * resolved some TODO comments (Vern Paxson, Corelight)

  * remove unnecessary casts, and change necessary ones to use static_cast<> (Vern Paxson, Corelight)

  * explain cmp_func default (Vern Paxson, Corelight)

  * change functions for ZVal type management to static members (Vern Paxson, Corelight)

  * fix some unsigned/signed integer warnings (Vern Paxson, Corelight)

  * address lint concern about uninitialized variable (Vern Paxson, Corelight)

  * Remove use of obsolete forward-declaration macros (Tim Wojtulewicz, Corelight)

  * Merge branch 'topic/vern/zval' of github.com:zeek/zeek into topic/vern/zval (Vern Paxson, Corelight)

  * fix #include's that lack zeek/ prefixes (Vern Paxson, Corelight)

  * overlooked another way in which vector holes can be created (Vern Paxson, Corelight)

  * initialize vector holes to the correct corresponding type (Vern Paxson, Corelight)

  * explicitly populate holes created in vectors (Vern Paxson, Corelight)

  * fix other instances of GetField().get() assuming long-lived ValPtr's (Vern Paxson, Corelight)

  * fix for now-incorrect assumption that GetField always returns an existing ValPtr (Vern Paxson, Corelight)

  * ... and porting over memory management for assignment to vector elements (Vern Paxson, Corelight)

  * ... porting over memory management for assignment to record fields (Vern Paxson, Corelight)

  * dagnabbit, forgot to port over destructor cleanup from ZAM_vector/ZAM_record (Vern Paxson, Corelight)

  * fix #include's that lack zeek/ prefixes (Vern Paxson, Corelight)

  * explicitly populate holes created in vectors (Vern Paxson, Corelight)

  * fixes for now-incorrect assumption that GetField always returns an existing ValPtr (Vern Paxson, Corelight)

  * memory management for assignment to vector elements (Vern Paxson, Corelight)

  * memory management for assignment to record fields (Vern Paxson, Corelight)

  * destructor cleanup from ZAM_vector/ZAM_record (Vern Paxson, Corelight)

  * test suite update for Smith-Waterman now using 0-based vectors (Vern Paxson, Corelight)

  * test suite updates for expanded sort() BiF semantics (bools, doubles) (Vern Paxson, Corelight)

  * test suite updates reflecting that vectors no longer support arbitrary "holes" (Vern Paxson, Corelight)

  * migration to using new differentiated methods for setting record fields (Vern Paxson, Corelight)

  * RecordVal methods for differentiated setting of record fields (Vern Paxson, Corelight)

  * fix for fencepost error when looping over vectors (Vern Paxson, Corelight)

  * migrate to using RecordVal's new Remove() method (Vern Paxson, Corelight)

  * new RecordVal Remove() method to clear a field (Vern Paxson, Corelight)

  * use differentiated record field accessors (Vern Paxson, Corelight)

  * base support for differentiated record field accessors (Vern Paxson, Corelight)

  * migrate to differentiated vector "At" accessors to support future efficiency (Vern Paxson, Corelight)

  * fix for subtle bug due to GetField now potentially returning a new ValPtr (Vern Paxson, Corelight)

  * use HasField() rather than GetField() where appropriate (Vern Paxson, Corelight)

  * moving sort()/order() functionality into VectorVal (Vern Paxson, Corelight)

  * convert VectorVal's to use vector<ZVal> as internal representation (Vern Paxson, Corelight)

  * convert RecordVal's to use vector<ZVal> as internal representation (Vern Paxson, Corelight)

  * added ZVal for low-level representations of Zeek script values (Vern Paxson, Corelight)

  * fix mis-typed record fields that happened to work in original RecordVal impl. (Vern Paxson, Corelight)

  * fix off-by-one assumption regarding vector indexing dating to 2011 (Vern Paxson, Corelight)

4.1.0-dev.356 | 2021-03-23 19:33:44 -0700

  * Increase timeout of openflow.log-cluster btest (Jon Siwek, Corelight)

4.1.0-dev.355 | 2021-03-23 19:05:10 -0700

  * add option for deterministic descriptions of sets & tables (Vern Paxson, Corelight)

  * determinism for concurrent Zeek test suite invocations; split out deprecations (Vern Paxson, Corelight)

  * disambiguate descriptions of enum types; include attributes when describing record types (Vern Paxson, Corelight)

  * more liberal view of attribute equality; allow suppressing attr type-checking (Vern Paxson, Corelight)

  * support for operations on sets that return new values (Vern Paxson, Corelight)

  * add low-level method for adding new enum values (Vern Paxson, Corelight)

  * sundry accessors/cast-ers; RE_Matcher's track their construction values (Vern Paxson, Corelight)

  * add convenience functions for comparing IP addresses (Vern Paxson, Corelight)

4.1.0-dev.346 | 2021-03-23 17:25:02 -0700

  * Add new ``./configure --plugindir`` option (Vlad Grigorescu)

    Useful for customizing the default installation path for plugins.

4.1.0-dev.344 | 2021-03-23 16:29:27 -0700

  * Clean up iterator usage in Analyzer (Tim Wojtulewicz, Corelight)

    The ``zeek::analyzer::analyzer_list`` type-alias changed from an
    ``std::list`` to ``std::vector`` which, in practice, is not expected to be
    used from plugins in API-incompatible way and may result in ~1-2% overall
    performance benefit.

4.1.0-dev.341 | 2021-03-18 12:45:04 -0700

  * fix for associating current scope with the name of enums; name tidying (Vern Paxson, Corelight)

  * avoid infinite recursion in same_type() if it is analyzing recursive types (Vern Paxson, Corelight)

  * remove iffy reliance on type punning that relies on interpreter's behavior (Vern Paxson, Corelight)

  * fixes for propagating optimization options, and pruning script function analysis (Vern Paxson, Corelight)

  * tidying with respect to "const", and streamlining OuterIDBindingFinder (Vern Paxson, Corelight)

  * making Exec methods non-const, so execution can manage side-information (Vern Paxson, Corelight)

  * removing unneeded BoolVal constructor that can cause overload ambiguity issues (Vern Paxson, Corelight)

  * lint fixes: ensuring functions return values, robustness to nil Val's (Vern Paxson, Corelight)

  * correcting some simple typos (Vern Paxson, Corelight)

4.1.0-dev.330 | 2021-03-17 13:48:34 -0700

  * Prevent use of LeakSanitizer on FreeBSD (Jon Siwek, Corelight)

4.1.0-dev.328 | 2021-03-17 10:35:59 -0700

  * Add tests for ERSPAN Type I patch

    Co-authored-by: Markel Elorza Alvarez <melorzaalvarez@gmail.com>
    Co-authored-by: Ivan Arrizabalaga Cupido <ivanarrcup@gmail.com> (Gorka Olalde Mendia)

  * Add ERSPAN Type I patch

    Co-authored-by: Markel Elorza Alvarez <melorzaalvarez@gmail.com>
    Co-authored-by: Ivan Arrizabalaga Cupido <ivanarrcup@gmail.com> (Gorka Olalde Mendia)

4.1.0-dev.323 | 2021-03-11 12:43:21 -0800

  * ts fields in SMB logs now default to network_time() (Seth Hall, Corelight)

    This avoids a problem identified by amanbansal2709 in pull
    request #1288. I fixed it in a different way than that pull request
    by making sure the ts field is always set so that this isssue doesn't
    return in the future.

4.1.0-dev.320 | 2021-03-10 12:29:13 -0800

  * GH-1432: Use buffered IO for file extraction (Jon Siwek, Corelight)

    This can improve performance significantly: ~3.5x faster when tested on
    a large file passing data to the file analysis framework in small chunks
    of 20 bytes.

4.1.0-dev.318 | 2021-03-10 12:05:47 -0800

  * Avoid searching a directory for dynamic plugins multiple times (Jon Siwek, Corelight)

4.1.0-dev.316 | 2021-03-10 13:00:27 +0000

  * Fix potential mime type detection bug in IRC/FTP file_transferred event

    The files framework uses strncpy to copy data into the buffer that is
    used for IRC/FTP mime type detection. From all I can tell that means
    that, in these cases, currently mime type detection will be messed up if
    the data being passed in contains zero bytes. (Johanna Amann, Corelight)

4.1.0-dev.314 | 2021-03-08 18:28:22 -0800

  * new "opt" btest alternative (Vern Paxson, Corelight)

  * update to "xform" test suite baseline due to recent line number changes (Vern Paxson, Corelight)

  * logic for driving AST optimization when requested (Vern Paxson, Corelight)

  * methods implementing AST optimization (aliasing, constant propagation, CSE) (Vern Paxson, Corelight)

  * helper class checking if common-subexpression elimination opportunity is valid (Vern Paxson, Corelight)

  * helper function for comparing collections of definition points (Vern Paxson, Corelight)

  * track more information about temporary variables (Vern Paxson, Corelight)

  * simplify and tidy up some script-opt interfaces (Vern Paxson, Corelight)

  * enable setting "-O optimize-AST" option (Vern Paxson, Corelight)

  * fix for reducing operations on boolean vectors (Vern Paxson, Corelight)

  * Add flag to FuncType::CheckArgs() to control type-equivalence warning output (Vern Paxson, Corelight)

  * remove unneeded IndexExpr::Traverse() virtual method (same as inherited method) (Vern Paxson, Corelight)

  * accessors for additional Expr subclasses (Vern Paxson, Corelight)

  * convenience functions to check for aggregate types (Vern Paxson, Corelight)

  * fix header includes to have proper zeek/ prefixes (Vern Paxson, Corelight)

4.1.0-dev.288 | 2021-03-08 12:29:14 -0800

  * GH-1426: Improve handling of Broker data store creation failures (Jon Siwek, Corelight)

    Broker::create_master() and Broker::create_clone() now return
    a valid value even when there's a failure to open the backend database
    (e.g. SQLite filesystem error).  In that case, the returned value can
    still be passed into other data store operations, but they'll fail
    immediately with an error.  Broker::is_closed() can now also be used to
    determine whether the data store creation calls failed.

4.1.0-dev.286 | 2021-03-08 11:24:38 +0000

  * Allow non-TCP based protocols to use SSL analyzer. (Keith Jones)

4.1.0-dev.284 | 2021-03-05 17:37:03 -0800

  * Simplify some copy_string() usages in threading::Field (Jon Siwek, Corelight)

  * Use const-reference in plugin::Manager::MetaHookPost for minor performance gain (Tim Wojtulewicz, Corelight)

  * Fix missing assigmnent operator/copy constructor pairings reported by LGTM (Tim Wojtulewicz, Corelight)

  * Fix variable shadowing issues reported by LGTM (Tim Wojtulewicz, Corelight)

4.1.0-dev.277 | 2021-03-05 16:14:23 -0800

  * Avoid reporting paths repeatedly in zeek-config --include_dir (Christian Kreibich, Corelight)

  * Fix comment in IP analyzer (Tim Wojtulewicz, Corelight)

4.1.0-dev.272 | 2021-03-01 11:36:06 -0800

  * Change update-zeekygen-docs.sh to set release branch in Sphinx config (Jon Siwek, Corelight)

    Allows release branches to generate correct links to GitHub source code.

  * Teach Zeekygen to produce source-code-range information (Jon Siwek, Corelight)

    Related to https://github.com/zeek/zeek-docs/issues/56

  * Add normalize_script_path() zeek::zeekygen::detail namespace (Jon Siwek, Corelight)

  * Add starts_with()/ends_with() to zeek::util namespace (Jon Siwek, Corelight)

4.1.0-dev.266 | 2021-02-25 08:45:49 -0700

  * Avoid superfluous string copies when adding to zeek::detail::sig_files

    Coverity 1387015 (Jon Siwek, Corelight)

  * Initialize an RD_Decorate member via std::move (Jon Siwek, Corelight)

  * Fix invalid iterator comparison in UseDefs::FindSuccUsage()

    Coverity 1447018 (Jon Siwek, Corelight)

4.1.0-dev.260 | 2021-02-24 15:52:31 -0800

  * Remove obsolete ZEEK_FORWARD_DECLARE_NAMESPACED macros (Tim Wojtulewicz, Corelight)

4.1.0-dev.258 | 2021-02-24 15:10:51 -0700

  * Minor Fedora 32/33 updates in CI

    - Switch Fedora package management from yum to dnf, the default
    - Minimize set of requested RPMs (Christian Kreibich, Corelight)

  * Add CentOS Stream 8 to CI

    There's no official Docker image on the Docker Hub, it's on quay.io:
    https://quay.io/repository/centos/centos?tab=tags

    At the moment, "Stream" and "Stream 8" are synonymous. (Christian Kreibich, Corelight)

4.1.0-dev.255 | 2021-02-24 12:35:47 -0800

  * Support explicit disabling of file analyzers (Jon Siwek, Corelight)

    This adds various methods/BIFs the enable/disable file analyzers
    or query whether they're currently enabled.

4.1.0-dev.252 | 2021-02-23 12:01:52 -0800

  * "xform" btest alternative baseline update (Vern Paxson, Corelight)

  * Test baselines updates related to "unused assignment" warnings (Vern Paxson, Corelight)

  * USE ZEEK_USAGE_ISSUES environment variable in tests rather than -u/-uu (Vern Paxson, Corelight)

    Since it has precedence over the flag arguments and don't want btest
    alternatives to override it.

  * splitting out "usage" test suite alternative into -u/-uu versions (Vern Paxson, Corelight)

  * adding &is_used attribute for base scripts (Vern Paxson, Corelight)

    Not actually needed yet, but will be once optimization is added.

  * removing unused assignments from base scripts (Vern Paxson, Corelight)

  * enhancements/changes to the Reduce class in preparation for use-defs (Vern Paxson, Corelight)

  * Add "-Odump-uds" and "ZEEK_DUMP_UDS" flag/environment variable for dumping use-defs (Vern Paxson, Corelight)

  * Add &is_used attribute to suppress set-but-not-used usage warnings (Vern Paxson, Corelight)

  * Support for "unused assignment" dead-code detection/warning via -u/-uu analysis (Vern Paxson, Corelight)

4.1.0-dev.229 | 2021-02-16 12:32:23 -0800

  * Update Broker submodule for performance/credit-tracking fixes (Jon Siwek, Corelight)

4.1.0-dev.228 | 2021-02-16 11:10:40 -0800

  * Fix `major_subsys_version` field in `pe_optional_header` event

    It was incorrectly set the same as the `minor_subsys_version` field
    of the `PE::OptionalHeader` record. (Jon Siwek, Corelight)

4.1.0-dev.225 | 2021-02-12 14:40:14 -0800

  * Fix CentOS 8 CI Dockerfile (Jon Siwek, Corelight)

    The "PowerTools" repoid changed to "powertools":
    https://bugs.centos.org/view.php?id=17920

4.1.0-dev.224 | 2021-02-12 11:18:32 -0800

  * GH-1398: Fix buffer overread in ascii formatter (Johanna Amann, Corelight)

    When a text with an (escaped) zero byte was passed to ParseValue, only
    the part of the string up to the zero byte was copied, but the length of
    the full string was passed to the input framework.

    This leads to the input manager reading over the end of the buffer.

4.1.0-dev.220 | 2021-02-11 11:10:46 -0800

  * GH-1399: Remove RocksDB configure script options (Jon Siwek, Corelight)

4.1.0-dev.219 | 2021-02-11 10:56:16 -0800

  * Add a script to cleanup the file listing before sending to Coveralls (Tim Wojtulewicz)

    GCC has a "feature" where it doesn't realize that files in the
    src/zeek/... tree are the same as files in the normal 'src/' tree. This
    leads the coverage script to send duplicates to Coveralls and pollute
    the display with them. The new script scrapes the intermediate output
    from lcov and de-duplicates anything in src/zeek/ to be just from src/.

4.1.0-dev.216 | 2021-02-08 13:06:07 -0800

  * Allow --parse-only to work with --usage-issues flag (Jon Siwek, Corelight)

    Use of --parse-only would previously exit before --usage-issues had a
    chance to analyze scripts and report any discovered issues and it can be
    useful to combine both flags for checking scripts for mistakes without
    actually executing any code.

    This also improves the behavior of --parse-only in combination with
    reporting problems in signature files (previously, it exited before
    even reading them).

4.1.0-dev.212 | 2021-02-05 10:56:23 -0800

  * Support for Zeek script "Reaching Definitions" (RD) analysis (Vern Paxson, Corelight)

    RD analysis invovles tracking the extent to which a given variable
    definition (assignment) can be visible elsewhere in the Zeek script.  The
    analysis works on local variables in function/event/hook bodies, but not
    across them.

    The code tracks two forms of RDs, "minimal" (what's guaranteed to reach a
    given point in a function body) and "maximal" (what possibly could reach).
    Upcoming script optimization will use the latter, but the former currently
    allows identification of places for which a value is used where it
    does not appear that it will necessarily be defined. Specifying the
    ``zeek -u`` option will generate warnings for instances where this holds
    for local variables. Specifying ``zeek -uu`` turns on additional
    expensive analysis to report instances where record fields might be used
    without having previously been set.

    The base scripts have some places where static analysis lacks sufficient
    power to tell that values are being used safely (guaranteed to have been
    identified). In order to enable users to employ ``zeek -u`` on their own
    scripts without being distracted by these instances, this change also
    includes a new attribute, ``&is_assigned``, which can be associated with a
    variable or a record field to inform Zeek's analysis that the script writer
    asserts the value will be set, suppressing the associated warnings.

    For the test suite, there's new ``btest -a usage`` alternative to exercise
    both ``zeek -uu`` and ``&is_assigned`` functionality.

4.1.0-dev.173 | 2021-02-03 11:16:04 -0800

  * GH-1389: Add VN-Tag packet analyzer that skips VN-Tag headers (Tim Wojtulewicz, Corelight)

4.1.0-dev.171 | 2021-02-03 09:53:12 -0700

  * Prefix #includes of .bif.h files with zeek/

    This enables locating the headers within the install-tree using the
    dirs provided by `zeek-config --include_dir`.

    To enable locating these headers within the build-tree, this change also
    creates a 'build/src/include/zeek -> ..' symlink. (Jon Siwek, Corelight)

4.1.0-dev.169 | 2021-02-03 09:50:33 -0700

  * Bump macOS jobs in CI from 4 to 12 cores

    A recent Cirrus upgrade enables this and saves 4-5 minutes for the
    build and test jobs (combined), for both Catalina and Big Sur. (Christian Kreibich, Corelight)

4.1.0-dev.165 | 2021-02-01 15:53:56 -0800

  * Update documentation to include "Book of Zeek" revisions (Jon Siwek, Corelight)

4.1.0-dev.164 | 2021-02-01 12:13:38 -0800

  * Deprecate zeekenv() and use getenv() directly (Jon Siwek, Corelight)

4.1.0-dev.162 | 2021-01-29 21:05:41 -0800

  * GH-1377: Add missing zeek/ to header includes (Jon Siwek, Corelight)

4.1.0-dev.158 | 2021-01-29 16:37:56 -0800

  * Fix language.init-in-anon-function btest due to changes to log filter predicates (Tim Wojtulewicz, Corelight)

  * Remove deprecated log filter predicates for 4.1 (Christian Kreibich, Corelight)

    Update the logging framework tests: since hooks operate
    by name, they cannot be anonymous. I'm also dropping the &optional
    attribute from the status field, since here know that the values are
    actually defined, and access to an optional status field should
    normally be guarded by the existence test operator.

  * Remove deprecated Plugin::HookCallFunction (Tim Wojtulewicz, Corelight)

  * Remove deprecated support for .bro script extension and BRO_ environment variables (Tim Wojtulewicz, Corelight)

  * Remove deprecated ICMP events (Tim Wojtulewicz, Corelight)

  * Remove some deprected methods/events from bif files (Tim Wojtulewicz, Corelight)

  * Remove deprecated TYPE_COUNTER (Tim Wojtulewicz, Corelight)

  * Remove all of the random single-file deprecations (Tim Wojtulewicz, Corelight)

  * Remove all fully-deprecated files (Tim Wojtulewicz, Corelight)

4.1.0-dev.146 | 2021-01-29 10:20:28 -0800

  * Skip negative integers in input framework where not type-permissible (Christian Kreibich, Corelight)

    Script-layer counts, when provided as negative integers in an input
    file, got cast to unsigned values because strtoull() does not complain
    about negative values. For example, input string "-1" would lead to
    value 18446744073709551615 (an all-ones 64-bit int) on x86_64. This is
    more likely to be an error than an intent to get very large,
    platform-dependent values, so these input lines are now skipped with
    according messaging in the reporter.log/stderr.

    This also affected ports: -1/tcp got cast to unsigned and only thrown
    out because PortVal rejects values > 65535, mapping them to 0. We now
    skip such inputs as well.

4.1.0-dev.144 | 2021-01-29 09:54:01 -0700

  * Fix various broken links in script documentation (Jon Siwek, Corelight)

4.1.0-dev.141 | 2021-01-25 12:32:48 -0800

  * Update broker submodule, embedded CAF to 0.18 (Jon Siwek, Corelight)

4.1.0-dev.137 | 2021-01-21 17:29:40 -0800

  * Update SSL consts (Johanna Amann)

    There have been a bunch of mostly TLS 1.3 related changes.

4.1.0-dev.135 | 2021-01-21 16:57:01 -0800

  * GH-1352: Added flag to stop processing SMTP headers in attached messages (Jon Oakley)

    Previously, header values from sub-entities could overwrite the top-level
    MIME entity values, causing the SMTP log to contain misleading information.

4.1.0-dev.132 | 2021-01-21 13:59:35 -0800

  * GH-1296: fix type-checks related to list-type equality (Jon Siwek, Corelight)

    List-types as used in composite table/set indices, for example,
    previously had incorrect same_type() comparisons due to flattening
    of the list-type into a single type without checking whether the
    number and kind of types all match.

    This patch simply removes the flatten_type() call from same_type() since
    it was already contradicting/preventing a subsequent full-comparison
    between elements of two TYPE_LISTs.

    There was also a superfluous special-case of the `in` operator's
    type-checking for testing whether a record is in a table/set.  It's
    superfluous because the general case will already do the type-checking
    from MatchesIndex() after first wrapping the record operand in a
    ListExpr.  The previous logic was incorrectly relying on the
    flatten_type() for testing equality of a record-type against a
    list-type, whereas the general case correctly normalizes to testing
    equality of two list-types.

    The special-cased type-checking logic for assigning a record value to a
    table index during its initialization similarly needed minor
    re-organization in order to maintain the same error messages as before.

  * Guard against nullptr deref after AST-reduction consistency checks (Jon Siwek, Corelight)

4.1.0-dev.128 | 2021-01-21 13:37:10 -0800

  * Install BTest with Zeek (Christian Kreibich, Corelight)

    Configuring with --disable-btest skips the installation. The main
    reason for doing this is that many Zeek packages come with testsuites
    that nearly always require btest, so providing btest out of the box
    makes sense. Note that zkg's recent additions to its PATH environment
    variable management mean that this btest instance is automatically
    found also when the Zeek installation's binary folder isn't in the
    path.

4.1.0-dev.125 | 2021-01-21 11:27:14 +0000

  * More precise type information in a config framework error message

    When an option's value is a reader-incompatible table or set, Zeek now
    renders the type as expressed in the script layer (e.g. "set[addr,addr]")
    as opposed to the internal type tag (which'd here be "table", including
    for sets). (Christian Kreibich, Corelight)

  * Explicitly don't support sets with multiple index types in input/config frameworks

    The input framework's Manager::IsCompatibleType() already rejected
    sets with multiple index types that aren't all the same (i.e. that are
    not pure). Pure ones (e.g. "set[addr,addr]") slipped through and could
    cause Zeek to segfault elsewhere in the config framework due to type
    comparison subtleties. Note that the ASCII reader can't read such sets
    anyway, so this method now rejects sets with any kind of index-type
    tuple.

    In the config framework, the script-level change handler has a risky
    conversion from any to set[bool], which can trigger segfaults when the
    underlying set's index is a type tuple. We now prevent this code path
    by ensuring it only applies to sets with a single index type.
    (Christian Kreibich, Corelight)

  * Fix logo/website link in README (Jon Siwek, Corelight)

4.1.0-dev.120 | 2021-01-19 21:08:11 -0800

  * Update Cirrus CI naming for MacOS images (Christian Kreibich, Corelight)

    The docs on https://cirrus-ci.org/guide/macOS/ changed from
    "osx_instance" to "macos_instance", so let's reflect that.

4.1.0-dev.118 | 2021-01-18 19:14:30 -0700

  * Remove superfluous nullptr checks in Val/RecordVal methods (Jon Siwek, Corelight)

4.1.0-dev.116 | 2021-01-14 19:29:58 -0800

  * Implement standard-library-compatible iterators for Dictionary (Tim Wojtulewicz, Corelight)

4.1.0-dev.114 | 2021-01-14 15:00:48 -0800

  * Add support for rewriting the ASTs for script functions in "reduced" form

    In reduced ASTs, expressions have only variables or constants for operands
    (no subexpressions - those are replaced by temporary variables). In
    addition, some expressions are transformed into alternatives (new specialized
    expressions, or in some cases new internal types of statements) to
    more directly encapsulate their operation.

    The functionality is enabled by using -O xform. You can dump the
    before-and-after ASTs using -O dump-xform (very large output!). The new
    command-line option --optimize-only=X restricts the processing to only the
    function named X, and also automatically turns on dump-xform.

  * Add new btest alternative for script transformation (xform) (Vern Paxson, Corelight)

  * split bifs.string_utils into a non-error test and an only-errors test (Vern Paxson, Corelight)

    To help control for differing error propagation

  * Add convenience function, obj_desc(), for accessing object descriptions (Vern Paxson, Corelight)

  * removed unused EventBodyList subclass (Vern Paxson, Corelight)

4.1.0-dev.82 | 2021-01-14 13:27:23 -0700

  * Use static_cast instead of dynamic_cast for Val conversions (Tim Wojtulewicz)

  * Remove GET_FIELD_AS macro, replace with template methods (Tim Wojtulewicz, Corelight)

  * Add templated As() method to Val, use in various places we were using dynamic_cast (Tim Wojtulewicz, Corelight)

  * Move ValImpl classes into detail namespace (Tim Wojtulewicz)

  * Some review cleanup in Val.cc (Tim Wojtulewicz, Corelight)

  * Changes addressing pending issues per PR discussion (Vern Paxson, Corelight)

  * Remove BroValUnion by hoisting underlying Val subclass values into subclasses (Vern Paxson, Corelight)

4.1.0-dev.74 | 2021-01-14 11:57:48 -0700

  * Use xcode 11 on the Catalina CI image (Tim Wojtulewicz, Corelight)

4.1.0-dev.71 | 2021-01-11 18:03:25 -0800

  * Add []-style variable-capture-list for Zeek lambda functions (Vern Paxson, Corelight)

    The previous behavior of automatically capturing references to variables
    outside a lambda's scope is now deprecated. An explicit capture list which
    also specifies the desired copy-semantics will be required when
    writing lambda functions that refer to local variables of an outer scope.

    Examples of the new capture-list syntax are described at
    https://docs.zeek.org/en/master/script-reference/types.html#type-function

  * nit: fixed some 0/1 values that should instead be false/true (Vern Paxson, Corelight)

  * factored some complexity of begin_func() into static functions for clarity (Vern Paxson, Corelight)

  * error propagation fix: don't complain about "unused" values that themselves are due to errors (Vern Paxson, Corelight)

  * corrected & reflowed some comments, plus a whitespace tweak (Vern Paxson, Corelight)

4.1.0-dev.52 | 2021-01-11 11:11:13 -0800

  * Remove unusable/broken RocksDB code and options (Jon Siwek, Corelight)

    The Broker RockSDB data store backend was previously unusable
    and broken, so all code and options related to it are now removed.

4.1.0-dev.51 | 2021-01-07 17:02:28 -0800

  * Virtualize Obj::GetLocationInfo() (Vern Paxson, Corelight)

    Cleaner approach for localizing errors associated with duplicated ASTs.

  * Add support for inlining of Zeek script functions (Vern Paxson, Corelight)

  * Add support for duplicating Zeek ASTS (Vern Paxson, Corelight)

  * Update COPYING to 2021 (Johanna Amann, Corelight)

4.1.0-dev.27 | 2021-01-06 20:42:35 -0800

  * GH-1347: Update cmake module to fix ZeekPluginDynamic's find_package(CAF) (Jon Siwek, Corelight)

4.1.0-dev.25 | 2020-12-28 15:37:20 -0700

  * Add LGTM config file (Jon Siwek, Corelight)

4.1.0-dev.23 | 2020-12-23 10:36:22 -0800

  * Fix subtle race on data store initialization (Dominik Charousset, Corelight)

4.1.0-dev.19 | 2020-12-22 10:58:13 -0800

  * Add new TLS 1.3 constants. (Johanna Amann)

  * Fix TLS 1.3 session resumption detection. (Johanna Amann)

    Now we detect TLS 1.3 session resumption by looking if both sides have
    the PSK extension set, which is much more exact than the previous
    approach.

  * Introduce ssl_probable_encrypted_handshake_message event (Johanna Amann)

    This event is raised for messages that (probably) are TLS 1.3 handshake
    messages, including finished. This allows scripts to examine handshake
    messages without having to handle all encrypted messages.

  * SSL Analyzer: ignore CCS for TLS 1.3 (Johanna Amann)

    In TLS 1.3, the ChangeCipherSpec message is meaningless; it only is
    included to convince middleboxes that the devices actually are speaking
    TLS 1.2. Nowadays some TLS 1.3 implementations also just don't send the
    packet.

    In a push to unify our handling of TLS 1.3 connections - ignore CCS and
    always go with application data packet counting to determine if
    connections are or are not encrypted.

  * GH-1323: TLS analyzer: change logic to track TLS 1.3 connection establishment (Johanna Amann)

    This commit changes the logic that is used to tracks connection
    establishment - and moves it from scriptland into the core.

    TLS 1.3 connection establishment is much more finnicky for us than the
    establishment of earlier versions - since we cannot rely on the CCS
    message anymore (which is meaningless and not sent in a lot of cases).

    With this commit, the ssl_encrypted_data message gets raised for
    encrypted TLS 1.3 handshake messages - which is much more correct than
    the behavior before that just interpreted them as plaintext messages.

4.1.0-dev.12 | 2020-12-21 10:55:13 -0800

  * Add MacOS Big Sur to CI (Christian Kreibich, Corelight)

4.1.0-dev.10 | 2020-12-17 16:11:10 -0800

  * Remove explicit use of config mode for finding CAF (Dominik Charousset, Corelight)

4.1.0-dev.7 | 2020-12-16 15:10:04 -0800

  * logging/ascii: Atomically create .shadow files using rename() (Arne Welzel, Corelight)

    A logger process being terminated/killed while in the process of creating
    a new .shadow file may leave an empty (invalid) one around. This in turn
    causes the logger to error and exit during startup.

4.1.0-dev.4 | 2020-12-15 08:00:44 -0800

  * Add test case to cover weird EDNS ECS parsing situations (Jon Siwek, Corelight)

  * Fix EDNS ECS option parsing bugs (Jon Siwek, Corelight)

    * The parsing of IPv6 addresses tried to fill a stack-buffer with as
      much data as supplied in the Option even if it was in excess of the
      desired prefix or maximum IPv6 address size.  This could result in an
      overflow of that stack-buffer.

    * The parsing of IPv4 addresses would overwrite the storage used for
      that address as many times as there were bytes in the Option in excess
      of the desired prefix length or maximum IPv4 address size.  This could
      cause the resulting IPv4 address to be derived from the incorrect
      data.

    * Upon encountering unexpected/excessive option-length or source-prefix
      parameters, the data pointer used for parsing was also not always
      advanced to the start of the next alleged option's data.  Assuming all
      other parsing code correctly guards against invalid input, there's no
      further harm from that other than the subsequent parsing being more
      likely to encounter unexpected values and emitting more Weirds.

    Credit to OSS-Fuzz for discovery
    https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=28336
    (Link to details becomes public 30 days after patch release)

4.1.0-dev.1 | 2020-12-14 22:27:57 -0800

  * Fix local.zeek compatibility test (Jon Siwek, Corelight)

3.3.0-dev.664 | 2020-12-14 21:23:01 -0800

  * Update submodules (Jon Siwek, Corelight)

3.3.0-dev.663 | 2020-12-14 14:27:45 -0800

  * Rename a 'do_net_run' variable to 'do_run_loop'

    For clarity, since the net_run() function was renamed to run_loop(). (Jon Siwek, Corelight)

  * GH-1329: call Zeek's cleanup function from standalone fuzzer driver (Jon Siwek, Corelight)

    Otherwise, the global Broker manager object containing CAF/threading
    logic is never destructed and can result in a heap-use-after-free if it
    tries to access other global objects after they're cleaned up from
    __cxa_finalize().

3.3.0-dev.660 | 2020-12-14 10:55:15 -0800

  * Fix typo in table iterator invalidation test comment (Tim Wojtulewicz, Corelight)

  * GH-1328: Improve behavior of Dictionary iterator invalidation (Jon Siwek, Corelight)

    Previously, an assertion was triggered in debug builds upon any attempt
    to insert or remove a Dictionary entry while any iteration of that
    Dictionary is underway and also even in cases where Dictionary membership
    was not actually modified (and thus invalidates a loop).

    Now, it emits run-time warnings regardless of build-type and only when
    insert/remove operations truly change the Dictionary membership.  In the
    context of a Zeek script causing an invalidation, the warning message
    also now helps pinpoint the exact expression that causes it.

  * Add a note about aligning .cirrus.yml to platform support matrix wiki page (Johanna Amann, Corelight)

3.3.0-dev.650 | 2020-12-12 20:24:19 -0800

  * Install zkg as part of the Zeek distribution. (Christian Kreibich, Corelight)

    - Add auxil/package-manager submodule as an optional build
      source. When the submodule is present, zkg gets installed into the
      Zeek installation's bin directory, its config file into etc/zkg, and
      its state into var/lib/zkg. Like zeekctl, zkg finds its own module
      independently of any PYTHONPATH. Installation via pip remains
      supported. You can skip zkg explicitly via --disable-zkg. See the
      NEWS update for details.

    - Establish a "zeek/python" subdirectory under libdir as the common place
      for Python modules in the Zeek distribution. This now separates out
      the Broker Python bindings, ZeekControl, and zkg's Python module.

    - Add configure flags to allow customizing this Python folder, in
      three ways: --python-dir, --python-prefix, and --python-home. These
      differ in the logic they automatically add to the path, and build on
      the logic already used in Broker.

    - Include a (commented-out) @load for zkg's packages folder in local.zeek.

3.3.0-dev.648 | 2020-12-12 13:39:28 -0800

  * Update Coverity Scan GitHub Action (Jon Siwek, Corelight)

    * Remove old Python 2 vs. 3 workaround
    * Build CAF separately to prevent internal errors in Coverity tools

3.3.0-dev.645 | 2020-12-10 14:14:12 -0800

  * Fix message ordering of Broker messages (Dominik Charousset, Corelight)

    Using two separate Broker subscribers for status events and regular
    messages introduces a race on the two objects. Even if Broker sends all
    messages in a particular (deterministic) order, Zeek may still process
    them in a different order as a result. Since several tests rely on a
    strict ordering of Broker events, these tests could fail sporadically.

    Using only a single subscriber for all Broker messages makes sure that
    Zeek observes all messages in the same order as Broker emits them.

  * Fix UB in shutdown of Broker manager (Dominik Charousset, Corelight)

  * Migrate to CAF 0.18 (Dominik Charousset, Corelight)

3.3.0-dev.638 | 2020-12-09 17:29:03 -0800

  * Update Mozilla Root Store (Johanna Amann, Corelight)

3.3.0-dev.634 | 2020-12-09 10:51:49 -0800

  * GH-1321: Prevent compounding of `connection_status_update` event timers (Jon Siwek, Corelight)

    Particularly for ICMP connections, a new timer got added every time a
    `connection` record was updated even if there was still a pending timer
    for that connection.

  * Update Debug.cc to use setsignal() wrapper (Jon Siwek, Corelight)

    So as to prefer sigaction() over either sigset() or signal(), which are
    less favorable due to underspecification issues and some treating them
    as obsolete/deprecated.

3.3.0-dev.630 | 2020-12-07 16:52:52 -0800

  * Fix narrowing conversion compiler error in SMB error-response logic (Jon Siwek, Corelight)

3.3.0-dev.629 | 2020-12-07 16:03:25 -0800

  * Add tests for new SMB3 multichannel support (Vlad Grigorescu)

  * Fix SMB2 response status parsing. Fixes #1286 (Vlad Grigorescu)

3.3.0-dev.624 | 2020-12-07 18:17:34 +0000

  * Switch test baselines to canonified baselines

    All baselines are now using the new btest feature to save baselines in the canonified
    form (after the canonifiers are run). This should, hopefully, lead to less changes in
    the future - that are easier to diff.

    In addition, this cleans up a few small problems of the canonifiers which were over-eager
    in identifying some numerical and path values.

    We also use the new btest-diff --binary flag to identify binary files on which no
    canonifier (not even the default canonifier) is applied.

    (Christian Kreibich, Corelight with tiny additions by Johanna Amann, Corelight)

3.3.0-dev.611 | 2020-12-07 09:42:37 -0800

  * Fix misassociation of Zeekygen-style comments within function scopes (Jon Siwek, Corelight)

    All Zeekygen-style comments relate to entities at global scope, so those
    found within functions are now ignored instead of misassociated.

3.3.0-dev.609 | 2020-12-04 19:20:39 -0800

  * Remove an extraneous Zeekygen-style comment (Jon Siwek, Corelight)

3.3.0-dev.607 | 2020-12-04 11:16:09 -0800

  * Fix the CMake 'dist' target of Zeek plugins to only run when outdated (Benjamin Bannier, Corelight)

3.3.0-dev.604 | 2020-12-04 18:40:03 +0000

  * Sumstats: allow users to manage epoch manually

    This change allows users to specify an epoch length of 0, which means
    that the user manually has to finish the epochs. A new next_epoch
    function is introduced to allow users to manually end epochs.

    Addresses GH-348 (Johanna Amann, Corelight)

  * Sumstats: epoch_finished was not called under certain circumstances

    In non-clustered mode, epoch_finished was not called when there was no
    data during the epoch.

    This behavior does not fit the documentation, and also is different in
    cluster-mode, where epoch_finished is, indeed, called after every epoch.

    This small change fixes this behavior. (Johanna Amann, Corelight)

3.3.0-dev.600 | 2020-12-03 18:02:22 -0800

  * Add a CI task for compiling with static broker/binpac (Johanna Amann, Corelight)

3.3.0-dev.596 | 2020-12-03 09:35:42 -0700

  * Fix a couple of life-time issues when plugin loading fails.

    Reported by Coverity.

    Follow-up to #1179. (Robin Sommer, Corelight)

3.3.0-dev.593 | 2020-12-02 12:53:04 -0800

  * Add `count_to_double` and `int_to_double` bif functions (Yacin Nadji, Corelight)

3.3.0-dev.590 | 2020-12-02 11:11:26 -0800

  * Update minimum required CMake to 3.5 (Jon Siwek, Corelight)

    Also now uses CMake's ENABLE_EXPORTS target property for the zeek
    executable to ensure symbols are visible to plugins.  Prior to CMake
    3.4, the policy was to export symbols by default for certain platforms,
    but later versions need either the explicit target property or policy.

3.3.0-dev.587 | 2020-12-01 10:17:42 -0700

  * GH-1184: Add 'source' field to weird log denoting where the weird was reported (Tim Wojtulewicz, Corelight)

3.3.0-dev.585 | 2020-12-01 14:42:54 +0000

  * Retry loading plugins on failure to resolve to dependencies.
    Closes #1179. (Robin Sommer, Corelight)

3.3.0-dev.580 | 2020-11-30 14:07:39 -0700

  * Find correct zeek namespace in debug logger macros.

    These macros forward to functionality in `zeek::detail::debug_logger`
    and are not intended for customization. This patch fixes the macros to
    always use `::zeek::detail::debug_logger` as without the leading `::`
    lookup could happen in any potentially local namespace `zeek` which does
    not need to provide this symbol.

    This closes zeek/spicy#597. (Benjamin Bannier, Corelight)

3.3.0-dev.576 | 2020-11-26 18:16:07 +0000

  * Remove Python2 compatibility logic. We now require at least Python 3.5.
    This includes script changes, improves the cmake logic to find python3,
    makes scripts explicitly call python3 and documentation updates.

    (Jon Siwek, Corelight)

  * Remove Fedora 31 (EOL) from CI (Jon Siwek, Corelight)

3.3.0-dev.564 | 2020-11-24 15:23:50 -0800

  * Improve support for custom libdir locations (Christian Kreibich, Corelight)

    - Remove hardwiring of $ZEEK_ROOT/lib throughout the three and
      defaults the name of Zeek's library directory to the default on the
      given platform (e.g. lib64), via GNUInstallDirs.

    - Consistently use that lib directory, instead of two lib folders
      resulting when using a custom libdir.

    - Remove the old lib directory in the installation prefix, if one exists

    - Add --lib_dir to zeek-config (and sort its options a bit).

3.3.0-dev.561 | 2020-11-23 21:50:19 -0800

  * Move implementation of internal_{type,var,etc} methods back into global namespace.
    (Tim Wojtulewicz, Corelight)

    This fixes an unknown symbol error if using those methods. They're defined
    as extern in the global namespace in Var.h, but Var.cc had their
    implementations defined in the zeek::detail namespace.

3.3.0-dev.559 | 2020-11-23 21:39:29 -0800

  * Simplify Debian/Ubuntu CI dependencies and setup (Dominik Charousset, Corelight)

  * Update .gitignore to ignore pyenv .python-version (Otto Fowler)

3.3.0-dev.554 | 2020-11-19 18:09:01 -0800

  * Reverts the SMTP regex change in dead3226a545e264072ced40284f86ac41528ba8. (Tim Wojtulewicz, Corelight)

    The regex change broke some of the external tests. I added some more cases
    to the regular email btest to hopefully cover all of the cases better.

3.3.0-dev.551 | 2020-11-17 15:01:04 -0700

  * Added unit tests for regex fix (christina23)

  * Improved regex for SMTP parsing (christina23)

3.3.0-dev.548 | 2020-11-17 13:32:57 -0800

  * GH-1033: BrokerStore <-> Zeek Tables: support complex indices (Johanna Amann, Corelight)

    This change adds support for complex indexes for sets and tables. With
    this change, sets with indexes like:

    set[string, count, count]

    will function. Before this change, Zeek raised an error message in these
    cases.

3.3.0-dev.546 | 2020-11-17 11:50:47 +0000

  * Fix memory leak in deprecated Analyzer::ConnectionEvent() (Jon Siwek, Corelight)

3.3.0-dev.543 | 2020-11-16 11:50:49 -0800

  * GH-352: Improve HTTP::match_sql_injection_uri regex (Jon Siwek, Corelight)

    Changes \x00-\x37 ranges to \x00-\x1f with assumption that the former
    was attempting to match ASCII control characters, but mistook an octal
    range for hex.  This change reduces some false positives.

3.3.0-dev.541 | 2020-11-16 11:22:00 -0800

  * Simplify ssh/main.zeek by using "ssh_server_host_key" for fingerprinting (Jon Siwek, Corelight)

  * Deprecate "ssh1_server_host_key" parameters *e* and *p* (Jon Siwek, Corelight)

    They are named such that *e* is actually the modulus, not the exponent.
    The replacement parameters are named *exponent* and *modulus* for
    clarity.

  * GH-1264: Implement "ssh_server_host_key" event (Jon Siwek, Corelight)

    This event provides host key fingerprints for both SSH1 and SSH2.

3.3.0-dev.537 | 2020-11-16 11:03:05 +0000

  * [SSH] Handle SSH version 1.99
    SSH can set in its identification a version 1.99 (SSH-1.99-xxx).
    That means the client/server is compatible with SSHv1 and SSHv2.
    So the version choice depends of the both side.

    1.99 : 1.99 => 2.0
    1.99 : 1.x  => 1.x
    1.99 : 2.0  => 2.O

    (see "Compatibility With Old SSH Versions" in RFC 4253) (Brevet Vivien)

3.3.0-dev.534 | 2020-11-12 14:31:10 -0800

  * Move UnknownProtocol options to init-bare.zeek (Jon Siwek, Corelight)

    Otherwise the `unknown_protocol` event cannot be used independently
    from `policy/mic/unknown-protocols.zeek`.

  * Coverity 1436183: Initialize packet_analysis::Manager fields (Jon Siwek, Corelight)

  * GH-1273: Change SizeExpr to yield "any" type when operating on "any" (Jon Siwek, Corelight)

  * Add enum_names() BIF to return names of an enum type's values (Jon Siwek, Corelight)

  * Add type_aliases() BIF for introspecting type-names of types/values (Jon Siwek, Corelight)

  * Change Type::type_aliases map to store IntrusivePtr (Jon Siwek, Corelight)

    And deprecate Type::GetAliases() and Type::AddAlias() since they
    took raw pointers.  Now replaced with Type::Aliases() and
    Type::RegisterAlias().

  * Fix lookup_ID() BIF to return enum values (Jon Siwek, Corelight)

    Looking up an enum value from a string equal to its name previously
    returned "<no ID value>".

3.3.0-dev.524 | 2020-11-12 12:16:00 -0700

  * Move 'using namespace' declaration after other includes (Tim Wojtulewicz, Corelight)

  * Use binpac.h as if it was an external dependency (Tim Wojtulewicz, Corelight)

  * Fix includes of bif.h and _pac.h files to use full paths inside build directory (Tim Wojtulewicz, Corelight)

  * Remove unnecessary include of NetVar.h from packet analysis plugins (Tim Wojtulewicz, Corelight)

  * GH-1079: Use full paths starting with zeek/ when including files (Tim Wojtulewicz, Corelight)

  * Fix type clash fatal error with log filters that use $ext_func and $include/$exclude

    The logging manager's Manager::TraverseRecord(), called when adding a
    log filter to a stream, skipped any fields intoduced by a filter's
    $ext_func when such fields weren't mentioned in a $include restriction
    or mentioned in an $exclude restriction. This was inconsistent with
    Manager::RecordToFilterVals, used when actually writing log entries,
    which does include those values.

    The result was that the record indices descent in Manager::RecordToFilterVals
    expects to find only record values, when in fact only the record
    provided by ext_func is present. This leads to type mismatches and
    hard Zeek exits like this one:

    1300475173.475401 fatal error in zeek/share/zeek//base/init-bare.zeek, line 4810: Val::CONVERTER (string/record) (zeek)

    The fix makes ext_func's field additions decisive, meaning the
    filter's include/exclude lists don't apply to it. If a user really
    wants to override this, they can reset the filter's ext_func back to
    our no-op default.

    The included btest produces the above error when the fix is not present. (Christian Kreibich, Corelight)

3.3.0-dev.516 | 2020-11-11 12:53:02 -0700

  * Support for additional DNS RR Type: LOC[29], SSHFP[44], NSEC3PARAM[51], custom BIND9 signaling[65534] (Fatema BW)

3.3.0-dev.514 | 2020-11-10 14:05:51 -0700

  * GH-1269: Fix LogAscii::enable_leftover_log_rotation crash in bad dirs

    Running with that option enabled inside a bad directory (e.g. lack of
    permissions) crashed due to not checking for failure of opendir(). (Jon Siwek, Corelight)

3.3.0-dev.512 | 2020-11-10 13:55:33 -0700

  * Change Debian 10 CI config to use GCC (Jon Siwek, Corelight)

  * Update external testing repo hashes for canonification updates (Tim Wojtulewicz, Corelight)

3.3.0-dev.509 | 2020-11-10 08:41:54 -0700

  * GH-1221: Add unknown_protocols.log for logging packet analyzer lookup failures (Tim Wojtulewicz, Corelight)

  * Remove default_analyzer for Ethernet packet analzyer (Tim Wojtulewicz, Corelight)

3.3.0-dev.506 | 2020-11-09 16:27:04 -0800

  * Add CI support for Debian 10, Fedora 31 & 33, FreeBSD 11, Ubuntu 20.04 (Christian Kreibich, Corelight)

    This also adds two environment variables to the Dockerfiles for the
    Debian/Ubuntu flavors to avoid stalling on user input when configuring
    tzdata.

3.3.0-dev.504 | 2020-11-09 10:56:09 -0700

  * Remove now-unused Packet::l2_valid field (Tim Wojtulewicz, Corelight)

  * GH-1186: Remove Packet::hdr_size and uses of it.

    This change also removes Packet::IP(), since Packet now contains an ip_hdr member
    that points at the IP header if it exists. (Tim Wojtulewicz, Corelight)

3.3.0-dev.501 | 2020-11-04 10:41:56 -0800

  * GH-1252: rename files with colons for Windows compatibility (Jon Siwek, Corelight)

    A short-term measure so that the Zeek source tree can simply exist on a
    Windows filesystem.  For true support, the logic that decides/generates
    the filename format will need to change.

  * Use std::function instead of a function pointer in packet_analysis::Component (Tim Wojtulewicz, Corelight)

3.3.0-dev.495 | 2020-11-02 12:03:52 -0700

  * GH-1215: Remove dispatch_map from packet analysis, replace with BIF methods for registering dispatches (Tim Wojtulewicz, Corelight)

3.3.0-dev.493 | 2020-11-02 09:51:03 -0800

  * GH-1256: Write out strerror when writing errno during safe_write (Tim Wojtulewicz, Corelight)

3.3.0-dev.490 | 2020-10-27 13:41:27 -0700

  * updated to mention notice_alarm.log by name. That has taken the place of the
    file formerly named alarm.log (Duffy O'Craven)

3.3.0-dev.488 | 2020-10-26 11:03:51 -0700

  * Improve how Zeekygen generated record/enum redefinition docs

    It now provides a summary of the new fields/enums added by any given
    redefinition along with associated commentary. (Jon Siwek, Corelight)

3.3.0-dev.486 | 2020-10-26 10:41:48 -0700

  * GH-1245: require TLD of hostname literals to start with a letter (Jon Siwek, Corelight)

3.3.0-dev.484 | 2020-10-23 14:47:44 -0700

  * Add coveralls badge to README.md (Tim Wojtulewicz)

  * Remove the entire auxil directory from the results (Tim Wojtulewicz)

  * Prevent non-master builds, add cirrus information to upload (Tim Wojtulewicz)

  * Add support for sending data to coveralls with lcov_html.sh via Cirrus (Tim Wojtulewicz, Corelight)

3.3.0-dev.479 | 2020-10-23 12:25:15 -0700

  * GH-251 (revert): remove coercion-to-signed-integer for |x| expressions (Jon Siwek, Corelight)

    For `|x|`, where `x` is an expression with an integral result, an
    implicit coercion of that result into signed `int` type no longer takes
    place.

    This was actually the behavior before Zeek 3.0 as well, but the attempt
    to prevent mistakes that easily result from integer literals in Zeek
    being unsigned like `|5 - 9|` causing an overflow/wraparound and
    yielding a very large number is not generally consistent since overflows
    are still generally able to happen in other ways and also in other
    contexts besides just absolute-values.  So the preference was to revert
    to a behavior that favors consistency.  For reference, see
    https://github.com/zeek/zeek/pull/251#issuecomment-713956976

3.3.0-dev.476 | 2020-10-22 15:59:56 -0400

  * Add an option to ignore packets sourced from particular subnets.

    It's implemented with a new set[subnet] option named ignore_checksums_nets.
    If you populate this set with subnets, any packet with a src address within
    that set of subnets will not have it's checksum validated. (Seth Hall, Corelight)

  * Update submodule(s) [nomail] (Jon Siwek, Corelight)


3.3.0-dev.467 | 2020-10-21 11:06:18 -0700

  * Fix a couple of Coverity findings (1433618, 1433619) (Tim Wojtulewicz, Corelight)

3.3.0-dev.465 | 2020-10-20 13:34:20 +0000

  * Bump BTest to 0.64.

3.3.0-dev.464 | 2020-10-20 08:43:40 +0000

  * Update BTest to 0.63. New test baselines will now be store
    canonified. (Robin Sommer, Corelight)

3.3.0-dev.463 | 2020-10-19 18:57:00 -0700

  * Add length checks for IP header values before parsing TCP/UDP (Tim Wojtulewicz, Corelight)

3.3.0-dev.461 | 2020-10-19 11:54:23 -0700

  * Fix deprecation warning in POP3 fuzzer (Jon Siwek, Corelight)

  * Add header length check to GRE packet analyzer (Tim Wojtulewicz, Corelight)

3.3.0-dev.456 | 2020-10-16 12:41:58 -0700

  * Avoid passing null pointers to memcmp in ZeekString comparison methods (Jon Siwek, Corelight)

3.3.0-dev.454 | 2020-10-16 10:34:53 -0700

  * Change ICMP Neighbor Discovery option length storage to a uint16 (Vlad Grigorescu)

    This fixes an overflow in the calculation of option lengths in
    ICMP Neighbor Discovery messages.

3.3.0-dev.451 | 2020-10-16 07:09:43 +0000

  * Make event ordering deterministic

    NetControl::init and filter_change_tracking could basically be raised in
    random order. (Johanna Amann, Corelight)

  * Introduce generate_all_events bif and add option to misc/dump-events

    generate_all_events causes all events to be raised internally; this
    makes it possible for dump_events to really capture all events (and not
    just those that were handled).

    Addresses GH-169 (Johanna Amann, Corelight)

3.3.0-dev.444 | 2020-10-15 13:25:12 -0700

  * Rework Sessions::Weird (Tim Wojtulewicz, Corelight)

  * Prevent String::Set from crashing if passed a nullptr (Tim Wojtulewicz, Corelight)

  * Store packet's ip header as unique_ptr (Tim Wojtulewicz, Corelight)

  * Remove some unused includes from Packet.h (Tim Wojtulewicz, Corelight)

  * Use shared_ptr for encapsulation data instead of raw pointer (Tim Wojtulewicz, Corelight)

  * Review cleanup (Tim Wojtulewicz, Corelight)

  * Update external testing repo hashes (Tim Wojtulewicz, Corelight)

  * Reorder Packet member variables slightly for better packing (Tim Wojtulewicz, Corelight)

  * Store the ip header in the packet after processing, reuse other places (Tim Wojtulewicz, Corelight)

  * Change to store data in packet directly instead of keystore (Tim Wojtulewicz, Corelight)

  * Don't always insert data into keystore for tunnels (Tim Wojtulewicz, Corelight)

  * Add comment about packet header size and session analysis (Tim Wojtulewicz, Corelight)

  * Set data to ip header's payload instead of advancing the pointer (Tim Wojtulewicz, Corelight)

  * Move packet dumping to packet_mgr (Tim Wojtulewicz, Corelight)

  * Make Sessions::NextPacket call packet_mgr, fix fuzzer code to do the same (Tim Wojtulewicz, Corelight)

  * Move IP and IP tunnel code from Sessions into packet analyzers (Tim Wojtulewicz, Corelight)

  * Prep work for IP changes

    - Move all of the time handling code out of PktSrc into RunState
    - Call packet_mgr->ProcessPacket() from various places to setup layer 2 data in packets (Tim Wojtulewicz, Corelight)

3.3.0-dev.426 | 2020-10-14 10:47:37 -0700

  * GH-1211: Improve error message for already-defined functions (Jon Siwek, Corelight)

3.3.0-dev.422 | 2020-10-13 16:26:24 -0700

  * GH-1208: Use Dictionary validity assertions only during CI (Jon Siwek, Corelight)

3.3.0-dev.420 | 2020-10-13 15:34:02 -0700

  * Change "Cluster::Node$p" field to use "&default=0/unknown" (Seth Hall, Corelight)

    This effectively makes defining the listening port optional in the cluster
    node layout config, with "0/unknown" meaning the node is not pre-configured
    to listen for incoming connections from other cluster nodes.

3.3.0-dev.414 | 2020-10-13 13:49:05 -0700

  * Update cmake submodule to pull in fix for building libkqueue (Tim Wojtulewicz, Corelight)

  * GH-1063: Update libkqueue to fix pf_ring-zc failures (Tim Wojtulewicz, Corelight)

3.3.0-dev.411 | 2020-10-13 13:38:38 -0700

  * Remove unused LoginConn type and variable in Conn.h (Tim Wojtulewicz, Corelight)

3.3.0-dev.409 | 2020-10-13 13:32:18 -0700

  * Make it possible to pass command line options through to scripts. (Seth Hall, Corelight)

    A new ``zeek_script_args`` variable contains a list of arguments passed
    to a script.  E.g. either when explicitly executing Zeek like
    ``zeek -- myscript.zeek -arg1 -arg2``, or when using Zeek to interpret
    executable scripts that contain a hashbang line at the top like::

      #!/usr/local/zeek/bin/zeek --

3.3.0-dev.403 | 2020-10-13 10:50:12 -0700

  * Add new Pcap::findalldevs() BIF (Seth Hall, Corelight)

  * Remove superfluous RuleCondition destructors (Jon Siwek, Corelight)

  * Silence Clang's warning about ignoring GCC's maybe-uninitialized warning (Jon Siwek, Corelight)

  * Add reference to network_time_init from zeek_init docs (Jon Siwek, Corelight)

3.3.0-dev.390 | 2020-10-12 17:43:15 -0700

  * Improve documentation for zeek_init event scheduling pitfalls (Jon Siwek, Corelight)

3.3.0-dev.388 | 2020-10-12 17:02:20 -0700

  * Add CaptureLoss::Too_Little_Traffic notice (Vlad Grigorescu)

  * Add CaptureLoss::initial_watch_interval for a quick read on cluster health after startup. (Vlad Grigorescu)

  * Improve capture-loss.zeek documentation. (Vlad Grigorescu)

  * Fix whitespace in capture-loss.zek (Vlad Grigorescu)

3.3.0-dev.381 | 2020-10-12 11:15:29 -0700

  * GH-779: Add "udp-state" signature condition (Jon Siwek, Corelight)

    It accepts "originator" or "responder" states as a way to enforce that
    the signature only matches packets in the associated direction.
    The "established" state is rejected as an error since it doesn't
    have a useful meaning like it does for the "tcp-state" condition.

  * Rename RuleConditionTCPState::TCPState enum values (Jon Siwek, Corelight)

  * Rename "tcp-state" signature parser tokens to not be TCP-specific (Jon Siwek, Corelight)

3.3.0-dev.377 | 2020-10-08 12:07:42 -0700

  * GH-1200: ignore a maybe-uninitialized warning (Jon Siwek, Corelight)

3.3.0-dev.374 | 2020-10-07 10:44:11 -0700

  * Fix multipart MIME leak of sub-part found after closing-boundary (Jon Siwek, Corelight)

    After detecting a closing-boundary for a given multipart MIME entity, it
    enters into an "end of data" state, however any subsequent boundary
    delimiter could still cause the allocation of a sub-entity object that
    is never released due to cleanup logic being bypassed upon finding the
    "end of data" state already reached.

    This change prevents allocation/processing of sub-entities after the
    "end of data" state is reached (e.g. from detecting a multipart
    closing-boundary).  This new behavior still aligns with RFC 2046
    expectations:

    "There appears to be room for additional information prior to the first
    boundary delimiter line and following the final boundary delimiter line.
    These areas should generally be left blank, and implementations must
    ignore anything that appears before the first boundary delimiter line or
    after the last one."

    Credit to OSS-Fuzz for discovery
    https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26027
    (Link to details becomes public 30 days after patch release)

  * Btests for log filter policy hooks (Christian Kreibich, Corelight)

  * Btest baseline updates to reflect new logging policy hooks (Christian Kreibich, Corelight)

  * Migrate existing use of filter predicates to policy hooks (Christian Kreibich, Corelight)

  * Support for log filter policy hooks (Christian Kreibich, Corelight)

    This adds a "policy" hook into the logging framework's streams and
    filters to replace the existing log filter predicates. The hook
    signature is as follows:

        hook(rec: any, id: Log::ID, filter: Log::Filter);

    The logging manager invokes hooks on each log record. Hooks can veto
    log records via a break, and modify them if necessary. Log filters
    inherit the stream-level hook, but can override or remove the hook as
    needed.

    The distribution's existing log streams now come with pre-defined
    hooks that users can add handlers to. Their name is standardized as
    "log_policy" by convention, with additional suffixes when a module
    provides multiple streams. The following adds a handler to the Conn
    module's default log policy hook:

        hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter)
                {
                if ( some_veto_reason(rec) )
                    break;
                }

    By default, this handler will get invoked for any log filter
    associated with the Conn::LOG stream.

    The existing predicates are deprecated for removal in 4.1 but continue
    to work.

3.3.0-dev.365 | 2020-10-06 12:19:49 -0700

  * GH-425: Avoid temporary value while coercing records already of the right type. (Robin Sommer, Corelight)

    The combination of this commit with the previous one now lets the examples
    in GH-425 all execute with the same performance.

  * GH-425: Optimize record constructor expression. (Robin Sommer, Corelight)

    We remove the inheritance from UnaryExpr because we know the type of the
    operand precisely and can skip a temporary when evaluating the expression.

  * Unify type comparisions for records. (Robin Sommer, Corelight)

    For records, same_type(r1, r2) would not check if the fields'
    attributes match as well. That seems like an oversight, and some
    callers of same_type() did indeed add that check on their end. This
    commit moves the check into same_type() itself. That generally doesn't
    seem make any differences except for a couple of places validating
    code, which we update a bit. That in turn leans to slightly different
    (better?) error messages for a couple of test cases.

3.3.0-dev.361 | 2020-10-06 10:13:37 -0700

  * logging/ascii: Support leftover log rotation in non-supervisor setups (Arne Welzel, Corelight)

    Allow enabling leftover log rotation through
    LogAscii::enable_leftover_log_rotation and redef this for the
    logger node in a supervisor setup individually.

3.3.0-dev.359 | 2020-10-02 10:49:35 -0700

  * GH-1180: Add external dependency include paths to `zeek-config --include_dir` (Jon Siwek, Corelight)

    This helps ensure successful compilation of a plugin that includes a
    Zeek header which includes an external dependency header even if the
    plugin itself doesn't necessarily depend on that external library.

3.3.0-dev.357 | 2020-10-02 10:26:11 -0700

  * Switch one's complement checksum implementation (Jon Siwek, Corelight)

    Borrows the `in_cksum` code from tcpdump, which borrowed from FreeBSD.
    It handles unaligned data better and also unrolls the inner loop to
    process 16 two-byte values at a time versus 2 one-byte values at a time
    in the previous version.  Generally measured as ~1.5x faster in a
    release build.  The new API should generally be more amenable to any
    future optimization explorations since all relevant data blocks are
    available within a single call rather than spread across multiple.

3.3.0-dev.355 | 2020-09-29 17:08:44 -0700

  * Improve broker/ssl_auth_failure.zeek test (Jon Siwek, Corelight)

    There was a race that previously may cause it to not truly test
    whether the connecting-process exited due to SSL authentication
    failure or just because the listening-process wasn't available yet
    (e.g. due to process scheduling variability).

3.3.0-dev.352 | 2020-09-25 15:09:29 -0700

  * GH-160: change find-filtered-trace to ignore pcaps with any non-TCP traffic (Robin Sommer, Corelight)

3.3.0-dev.350 | 2020-09-25 14:37:58 -0700

  * Add dce_rpc_request_stub and dce_rpc_response_stub events (Yacin Nadji, Corelight)

  * Fix namespace warning in fuzzer code (Tim Wojtulewicz, Corelight)

3.3.0-dev.346 | 2020-09-24 16:03:28 -0700

  * Fix a Sphinx warning about misformatted packet analyzer comment (Jon Siwek, Corelight)

  * Add Zeekygen documentation support for packet analyzers (Jon Siwek, Corelight)

  * Move packet_mgr to the zeek namespace (Tim Wojtulewicz, Corelight)

3.3.0-dev.341 | 2020-09-24 08:16:45 -0700

  * Fix negative-value-left-shift undefined behavior in patricia trie (Jon Siwek, Corelight)

  * Improve negation of ConstExpr (Jon Siwek, Corelight)

    * Instead of creating a NegExpr for negation of a literal/constant,
      a ConstExpr is now created directly.

    * For negation of integer literals, there's now an additional check
      for whether the integer would be outside the range of possible 'int'
      values.  This can also help prevent the undefined behavior due to
      overflow as a result of trying to represent the minimum 'int' value of
      -9223372036854775808 as a literal in a script -- the unsigned value is
      cast to signed yielding INT64_MIN, then INT64_MIN is negated.

  * Avoid signed integer overflow when combining SMB header PID bits (Jon Siwek, Corelight)

    Such an overflow invokes undefined behavior.

  * Avoid unary negation of INT64_MIN in modp_litoa10 (Jon Siwek, Corelight)

    Overlow can occur in that case, which is undefined behavior.

  * Avoid double-to-int conversion overflows in modp_dtoa functions (Jon Siwek, Corelight)

    Those methods already had a fallback to use sprintf() for large values
    except:

    * The check-for-large-value was unnecessarily done after many
      operations that aren't relevant to the check and those operations can
      result in a conversion overflow (undefined behavior).

    * The check-for-large-value was using the literal value for a
      32-bit INT_MAX instead of just using INT_MAX.  For a platform where
      `int` is less than 32-bits, the same conversion overflow from the
      previous point could still occur (undefined behavior).

    * The check-for-large-value was not inclusive of INT_MAX.
      In a case where the conversion of INT_MAX itself to a double
      can't be represented exactly, it's implementation-defined whether
      the closest higher or closest lower representable-value is selected.
      If the higher value is selected, then a `double` value comparing equal
      to INT_MAX-as-converted-to-double would cause an overflow of an `int`
      upon conversion (undefined behavior).

  * Fix divide-by-zero in Entropy analyzer (Jon Siwek, Corelight)

  * Fix divide-by-zero in stats/profiling memory usage calculation (Jon Siwek, Corelight)

  * Fix uninitialized field in POP3 fuzzer (Jon Siwek, Corelight)

  * Add framework for running UndefinedBehaviorSanitizer in CI (Jon Siwek, Corelight)

3.3.0-dev.329 | 2020-09-23 11:32:06 -0700

  * Update NEWS (Tim Wojtulewicz, Corelight)

  * Fix build on FreeBSD, which was missing full definition of sockaddr for ARP (Tim Wojtulewicz, Corelight)

  * Update hashes for external testing repos (Tim Wojtulewicz, Corelight)

  * Fix handling of session analysis.

    This reverts commit 0e51be18ac59d7a176f18780f1f8df85561449b9. (Jan Grashoefer)

  * Fix Sessions::DoNextPacket to ignore ARP packets and not report weirds for them (Tim Wojtulewicz, Corelight)

  * Packet analysis cleanup. (Jan Grashoefer)

  * Minor cleanup (Tim Wojtulewicz, Corelight)

  * Simplify packet analyzer config. (Jan Grashoefer)

  * Make default packet analyzer definition explicit. (Jan Grashoefer)

  * Add explicit root analyzer for packet analysis. (Jan Grashoefer)

  * Extend packet analysis test. (Jan Grashoefer)

  * Simplify MPLS analysis. (Jan Grashoefer)

  * Improve packet analysis data flow. (Jan Grashoefer)

  * Improve packet analyzer API. (Jan Grashoefer)

  * Rename DefaultAnalyzer to IP. (Jan Grashoefer)

  * Move ARP analysis into packet analyzer. (Jan Grashoefer)

  * Small cleanup of packet analysis. (Jan Grashoefer)

  * Fix some build issues from the rebase (Tim Wojtulewicz, Corelight)

  * Remove encap_hdr_size (replaced by skip analyzer). (Jan Grashoefer)

  * Add skip analyzer test and update baselines. (Jan Grashoefer)

  * Allow to overwrite packet analysis mappings. (Jan Grashoefer)

  * Add SkipAnalyzer.

    This is WIP: The test case would require a new pcap or the possibility
    to overwrite analyzer mappings. The CustomEncapsulationSkip method and
    the corresponding options need to be removed. (Jan Grashoefer)

  * Suggested code improvements for packet analysis. (Jan Grashoefer)

  * Improve naming in packet analysis. (Jan Grashoefer)

  * Migrate all packet analyzers to new API. (Jan Grashoefer)

  * Remove magic identifiers from Ethernet analyzer. (Jan Grashoefer)

  * Bring back default packet analysis.

    Default analyzers can be configured per packet analyzer by omitting the
    identifier in the ConfigEntry. (Jan Grashoefer)

  * Further simplified the packet analysis API.

    This is still WIP and includes the following changes:
    * Dispatchers are now part of analyzers (moving dispatching logic from
      the manager to the analyzers)
    * All available analyzers are instantiated on start up
    * Removal of configuration class (Jan Grashoefer)

  * Move dispatching into packet analyzers.

    WIP that updates only the Ethernet analyzer. (Jan Grashoefer)

  * Move cur_pos from packet into packet manager loop. (Jan Grashoefer)

  * Remove packet_analysis/Defines.h

    - Replace uses of identifier_t with uint32_t
    - Replace repeated usage of tuple type for Analysis results with type alias (Tim Wojtulewicz, Corelight)

  * Move VectorDispatcher to be the only dispatcher (Tim Wojtulewicz, Corelight)

  * Remove Manager::Reset() method (Tim Wojtulewicz, Corelight)

  * Remove enabled state from Components, ability to enable/disable from Manager (Tim Wojtulewicz, Corelight)

  * Use shared_ptr instead of raw pointers in packet_analysis for analyzers and dispatchers (Tim Wojtulewicz, Corelight)

  * Whitespace fixes from review (Tim Wojtulewicz, Corelight)

  * Reorganize some pointer handling (Tim Wojtulewicz, Corelight)

  * Fix ordering of debug logger strings to match the enum (Tim Wojtulewicz, Corelight)

  * Fixes for various btest issues

    - Fix handling of truncated ethernet headers, fix core.truncation test output
    - Update commit hashes for external private test repo (Tim Wojtulewicz, Corelight)

  * Merge ProtocolAnalyzerSet into Manager, remove AnalyzerSet base class (Tim Wojtulewicz, Corelight)

  * Minor fixes for packet analyzer renaming. (Jan Grashoefer)

  * Renamed LL-Analyzers to Packet Analyzers. (Jan Grashoefer)

  * Initial implementation of Lower-Level analyzers (Peter Oettig)

3.3.0-dev.285 | 2020-09-22 16:01:51 -0700

  * Improve fa_file/fa_metadata documentation (Duffy O'Craven, Corelight)

3.3.0-dev.284 | 2020-09-22 14:10:36 -0700

  * Avoid race condition in control/id_value btest (Jon Siwek, Corelight)

3.3.0-dev.282 | 2020-09-22 11:08:56 -0700

  * GH-927: Fix circumvention of evaluation order in 'when' conditions (Jon Siwek, Corelight)

3.3.0-dev.279 | 2020-09-21 15:22:45 -0700

  * CI: Move to Fedora 32 (Christian Kreibich, Corelight)

    Fedora 30 is EOL.

3.3.0-dev.275 | 2020-09-17 13:39:46 -0700

  * GH-1164: Fix incorrect RSTOS0 conn_state determinations

    The RSTOS0 `conn_state` label is documented as "Originator sent a SYN
    followed by a RST, never saw SYN-ACK from responder", but was previously
    applied to cases where no originator SYN exists, like a single RST-only
    packet. (Jon Siwek, Corelight)

3.3.0-dev.273 | 2020-09-17 13:11:44 -0700

  * GH-1119: add base/protcols/conn/removal-hooks.zeek

    This adds two new functions: `Conn::register_removal_hook()` and
    `Conn::unregister_removal_hook()` for registering a hook function to be
    called back during `connection_state_remove`.  The benefit of using hook
    callback approach is better scalability: the overhead of unrelated
    protocols having to dispatch no-op `connection_state_remove` handlers is
    avoided. (Jon Siwek, Corelight)

  * Avoid run-time cost of find-filtered-trace.zeek for live traffic (Jon Siwek, Corelight)

  * Remove connection_successful and successful_connection_remove events

    Related to https://github.com/zeek/zeek/issues/1119 (Jon Siwek, Corelight)

3.3.0-dev.269 | 2020-09-17 11:42:38 -0700

  * GH-1155: Recursively check table index for unsupported types

    Previously, container types used within a table/set index were not
    deeply checked to ensure all constituents could be part of an index. (Jon Siwek, Corelight)

  * GH-1159: Fix vector-of-interval multiplication/division arithmetic

    Those operations done between a vector-of-interval and a
    vector-of-arithmetic-type previously threw a runtime expression error
    due to an incorrect coercion being used internally. (Jon Siwek, Corelight)

3.3.0-dev.265 | 2020-09-17 11:24:42 -0700

  * Avoid passing address of member in packed struct #1074

    This appeases -Waddress-of-packed-member warnings in some
    compiler/platform combinations via use of local variables. (Christian Kreibich, Corelight)

3.3.0-dev.262 | 2020-09-14 12:27:52 -0700

  * Improve notice/suppression-cluster btest to avoid race potential (Jon Siwek, Corelight)

3.3.0-dev.260 | 2020-09-10 11:53:03 -0700

  * Fix generate-docs Action to stage autogenerated script-reference files (Jon Siwek, Corelight)

3.3.0-dev.258 | 2020-09-09 10:46:37 -0700

  * Improve Reporter weird-sampling-whitelist getters/setters (Jon Siwek, Corelight)

    - getter methods return const-ref
    - setter methods pass by value and std::move()
    - ranged-for loops over the whitelists access by const-ref

  * Fix memory leak caused by re-entering GTPv1 parsing (Jon Siwek, Corelight)

    If the inner packet of a decapsulated GTPv1 frame maps to the same
    Connection as the outer packet and also contains another GTPv1
    encapsulation, the same GTPv1 analyzer attempts to parse the inner
    packet.  In the process, the inner parsing method overwrites pointers to
    memory resources the outer parsing method has not yet released and
    results in a memory leak.

  * Fix memory leak caused by re-entering AYIYA parsing (Jon Siwek, Corelight)

    If the inner packet of a decapsulated AYIYA frame maps to the same
    Connection as the outer packet and also contains another AYIYA
    encapsulation, the same AYIYA analyzer attempts to parse the inner
    packet.  In the process, the inner parsing method overwrites pointers to
    memory resources the outer parsing method has not yet released and
    results in a memory leak.

    Credit to OSS-Fuzz for discovery
    https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=25256

3.3.0-dev.254 | 2020-09-08 17:21:16 -0700

  * GH-343: Change HTTP DPD signatures to trigger analyzer independent of peer state. (Robin Sommer, Corelight)

    This is to avoid missing large sessions where a single side exceeds
    the DPD buffer size. It comes with the trade-off that now the analyzer
    can be triggered by anybody controlling one of the endpoints (instead
    of both).

3.3.0-dev.252 | 2020-09-08 17:04:19 -0700

  * GH-623: Extend weird sampling with option to track selected weirds globally. (Robin Sommer, Corelight)

    The new set "sampling_global_list" lists weirds to rate-limit
    globally instead of per connection/flow.

3.3.0-dev.249 | 2020-09-04 18:30:19 -0700

  * Exclude installing "zeek -> ." include dir symlink (Jon Siwek, Corelight)

    The symlink only exists for use within the source-tree and isn't needed
    for the install-tree.

3.3.0-dev.247 | 2020-09-04 17:24:28 -0700

  * GH-165: Fix global initializations that indirectly use builtin types (Jon Siwek, Corelight)

  * Improve how primary/top-level BIFs get initialized (Jon Siwek, Corelight)

  * GH-1122: Allow initializing globals with calls to subdir BIFs (Jon Siwek, Corelight)

  * GH-1122: Improve error for global record initialization exceptions (Jon Siwek, Corelight)

3.3.0-dev.241 | 2020-09-03 15:02:29 -0700

  * Change a script comment to use "code-block" directive (Jon Siwek, Corelight)

3.3.0-dev.240 | 2020-09-02 20:51:13 -0700

  * Improve CI benchmark script's error handling/messaging (Jon Siwek, Corelight)

3.3.0-dev.238 | 2020-09-02 11:39:25 -0700

  * GH-955: Prohibit `fallthrough` in typecasting `switch` cases (Jon Siwek, Corelight)

3.3.0-dev.236 | 2020-09-01 22:33:05 -0700

  * Fix/finalize the generate-docs Github Action (Jon Siwek, Corelight)

3.3.0-dev.230 | 2020-09-01 20:32:01 -0700

  * Fix generate-docs action to not fail when there's no changes (Jon Siwek, Corelight)

3.3.0-dev.229 | 2020-09-01 19:49:36 -0700

  * Update generate-docs action to use access token for submodule update (Jon Siwek, Corelight)

3.3.0-dev.228 | 2020-09-01 18:37:33 -0700

  * Update generate-docs action to install Sphinx dependency (Jon Siwek, Corelight)

3.3.0-dev.227 | 2020-09-01 17:58:22 -0700

  * Add missing configure step for doc-generation action (Jon Siwek, Corelight)

3.3.0-dev.226 | 2020-09-01 17:53:55 -0700

  * Improve documentation for rfind_str() (Jon Siwek, Corelight)

  * GH-1149: Add GitHub Action to automate generation of zeek-docs (Jon Siwek, Corelight)

3.3.0-dev.224 | 2020-09-01 11:57:44 -0700

  * Reformat strings.bif Zeekygen comments to fix Sphinx warnings (Jon Siwek, Corelight)

3.3.0-dev.221 | 2020-08-31 17:31:21 -0700

  * GH-174: Treat ambiguous attribute duplication as an error (Tim Wojtulewicz, Corelight)

    For example, a &default=1 and a &default=2 attribute are not valid when
    used together, but two duplicate &log attributes together are acceptable
    although redundant.

3.3.0-dev.216 | 2020-08-31 14:57:57 -0700

  * Simplify a broker btest (Jon Siwek, Corelight)

3.3.0-dev.215 | 2020-08-28 16:51:40 -0700

  * GH-926: Improve type-checking for ternary conditional operator (Jon Siwek, Corelight)

  * Unbreak build on power architectures by fixing small cmake issue (Johanna Amann, Corelight)

3.3.0-dev.211 | 2020-08-27 18:00:55 -0700

  * Potentially improve intel/remove-item-cluster btest (Jon Siwek, Corelight)

3.3.0-dev.210 | 2020-08-27 15:34:14 -0700

  * GH-158: Ensure packet-filter framework is loaded if -f option is passed (Tim Wojtulewicz, Corelight)

3.3.0-dev.208 | 2020-08-27 11:12:22 -0700

  * GH-1145: Fix a umask issue for processing-status file (Vlad Grigorescu)

3.3.0-dev.206 | 2020-08-26 11:20:58 -0700

  * Rename bro-bif.h to zeek-bif.h (Tim Wojtulewicz, Corelight)

  * Fix uses of bro.pac to use zeek.pac (Tim Wojtulewicz, Corelight)

  * Rename the bro-related binpac files to zeek (Tim Wojtulewicz, Corelight)

3.3.0-dev.202 | 2020-08-26 10:59:59 -0700

  * GH-1145: Have mkdir in ensure_dir respect umask. (Vlad Grigorescu)

    This also aligns with the mkdir BIF.

3.3.0-dev.200 | 2020-08-25 19:47:02 -0700

  * Update plugin btests for namespace changes (Tim Wojtulewicz, Corelight)

  * Remove namespace qualifications where unnecessary. (Tim Wojtulewicz, Corelight)

3.3.0-dev.196 | 2020-08-25 17:04:12 -0700

  * Fix closing timestamp of rotated log files in supervised-cluster mode (Jon Siwek, Corelight)

3.3.0-dev.195 | 2020-08-25 14:32:47 -0700

  * Improve an Intel framework btest (Jon Siwek, Corelight)

    Relying on non-atomic `cp` operation may not cooperate well with the
    Input framework file re-reading logic.

3.3.0-dev.194 | 2020-08-24 19:20:29 -0700

  * Mark Continuation.h and PacketDumper.h as deprecated (Tim Wojtulewicz, Corelight)

  * Remove vector iterator type aliases from SmithWaterman code, fix uses of them (Tim Wojtulewicz, Corelight)

  * Fix warning with usage of fmt() (Tim Wojtulewicz, Corelight)

3.3.0-dev.190 | 2020-08-24 14:43:06 -0700

  * Make set_processing_status() signal-safe.

    Closes #574. (Robin Sommer, Corelight)

3.3.0-dev.188 | 2020-08-24 14:30:43 -0700

  * GH-710: Check that &expire_func is a function and not hook/event (Jon Siwek, Corelight)

3.3.0-dev.186 | 2020-08-24 14:28:25 -0700

  * Fix a case where PktSrc gets processed twice in one runloop iteration

    For a non-live PktSrc, it had a special-case to be considered "ready"
    every iteration, but additionally every 1 in 100 iterations (the polling
    frequency), if there were no other "ready" IOSources, it would get added
    to the "ready" set a 2nd time.

    This commit completely excludes PktSrc from being processed during the
    1/100 runloop iteration where a Poll() happens.  That exclusion is
    desirable for a second reason: if reading a pcap happens to do its final
    Process() during that 1/100 polling-iteration and there's other
    IOSources ready to process like EventMgr/TimerMgr, those sources have
    logic to advance network-time to current-time if a PktSrc is no longer
    open.  So in such a case, PktSrc::Process() closes, then
    EventMgr::Process() sees there's no longer an active PktSrc and advances
    to current-time, then EventMgr::Drain() happens and may dispatch
    various events that were previous scheduled, with those events now
    unexpectedly seeing a network_time() returning current-time. (Jon Siwek, Corelight)

3.3.0-dev.184 | 2020-08-24 14:27:31 -0700

  * GH-594: Improve table initialization type-check error messages (Jon Siwek, Corelight)

3.3.0-dev.182 | 2020-08-21 13:46:16 -0700

  * Fix `zeek -NN` not printing canonical names for file analyzers (Tim Wojtulewicz, Corelight)

3.3.0-dev.180 | 2020-08-21 13:28:23 -0700

  * Remove definition-less ThreadVal<->BrokerData conversion functions (Jon Siwek, Corelight)

    Their presence/deprecation may cause "undefined reference" build
    failures for some compilers (GCC 9.3.0 in my case).

3.3.0-dev.177 | 2020-08-21 08:36:48 -0700

  * Add missing namespace to usage of get_exe_path in fuzzer (Tim Wojtulewicz, Corelight)

  * Rename methods in RunState to remove 'net' from their names (Tim Wojtulewicz, Corelight)

  * Move zeek::net namespace to zeek::run_state namespace.

    This also moves all of the code from Net.{h,cc} to RunState.{h,cc} and marks Net.h as deprecated (Tim Wojtulewicz, Corelight)

  * Move ScannedFile class and associated globals into ScannedFile.h and out of Net.h and scan.l (Tim Wojtulewicz, Corelight)

  * Rename types in ZeekList.h to be consistent with the style guide (Tim Wojtulewicz, Corelight)

  * Move NetVar from zeek to zeek::detail namespace (Tim Wojtulewicz, Corelight)

  * Remove PRI_PTR_COMPAT macros (Tim Wojtulewicz, Corelight)

  * Fix indentation of namespaced aliases (Tim Wojtulewicz, Corelight)

  * Move zeek-setup code into namespaces (Tim Wojtulewicz, Corelight)

  * Move ZeekList types to zeek namespace (Tim Wojtulewicz, Corelight)

  * Move __RegisterBif from zeek::detail::plugin to zeek::plugin::detail (Tim Wojtulewicz, Corelight)

  * Remove unimplemented zeek_magic_path/bro_magic_path method (Tim Wojtulewicz, Corelight)

  * Move all plugin classes into zeek::plugin::detail namespaces (Tim Wojtulewicz, Corelight)

  * Rename BroList.h to ZeekList.h (Tim Wojtulewicz, Corelight)

  * Move a few smaller files to zeek namespaces (Tim Wojtulewicz, Corelight)

  * Tag the end of some namespaces for consistency (Tim Wojtulewicz, Corelight)

  * Move a few of the zeek::util methods and variables to zeek::util::detail (Tim Wojtulewicz, Corelight)

  * Move zeekygen code to zeek::zeekygen::detail namespace (Tim Wojtulewicz, Corelight)

  * Mark zeek::util::pad_size as constexpr, which provides a small performance improvement (Tim Wojtulewicz, Corelight)

  * Move everything in util.h to zeek::util namespace.

    This commit includes renaming a number of methods prefixed with bro_ to be prefixed with zeek_. (Tim Wojtulewicz, Corelight)

  * Deprecate ptr_compat_uint and ptr_compat_int in util.h (Tim Wojtulewicz, Corelight)

  * Move NetVar.h to zeek namespace (Tim Wojtulewicz, Corelight)

  * Move the functions and variables in Net.h to the zeek::net namespace. This includes moving network_time out of util.h. (Tim Wojtulewicz, Corelight)

  * Add namespaced version of trigger_mgr, fix a couple uses of event_registry (Tim Wojtulewicz, Corelight)

  * Move the remainder of the analyzers to zeek namespaces (Tim Wojtulewicz, Corelight)

  * Move arp, tcp, udp, pia, and stepping stone analyzers (Tim Wojtulewicz, Corelight)

  * Move bro_broker code to zeek::Broker namespace (Tim Wojtulewicz, Corelight)

  * Move notifier code to zeek::notifier::detail namespace (Tim Wojtulewicz, Corelight)

  * Move threading classes to zeek namespaces (Tim Wojtulewicz, Corelight)

  * Move probabilistic code into zeek namespaces (Tim Wojtulewicz, Corelight)

  * Move iosource code to zeek namespaces (Tim Wojtulewicz, Corelight)

  * Move logging code to zeek namespaces (Tim Wojtulewicz, Corelight)

  * Move input code to zeek namespaces (Tim Wojtulewicz, Corelight)

  * Move file_analysis code to zeek namespaces (Tim Wojtulewicz, Corelight)


3.3.0-dev.142 | 2020-08-20 16:26:06 -0700

  * Improve stability of config/cluster_resend btest (Jon Siwek, Corelight)

3.3.0-dev.141 | 2020-08-20 14:56:15 -0700

  * Mark FileAssembler destructor as default, remove implementation (Tim Wojtulewicz, Corelight)

  * Remove a few protected or private no-op constructors (Tim Wojtulewicz, Corelight)

  * Properly initialize Timer::type. Fixes Coverity 1431144. (Tim Wojtulewicz, Corelight)

  * Properly initialize members of Reassembler. Fixes Coverity 1413348. (Tim Wojtulewicz, Corelight)

  * Fix a number of Coverity issues in Dict (Tim Wojtulewicz, Corelight)

    1431186: Asserting that an unsigned value is >= 0 is pointless
    1431188/1431189/1431191: Side effect of using an assignment operator in a call to ASSERT()
    1431193: Dereference after null check could lead to null being used
    1431195: Use of rand() replaced with random()

3.3.0-dev.135 | 2020-08-20 10:23:29 -0700

  * remove variables in netvar, use tabs in DNS.h and polish comments in dns events. (FlyingWithJerome)

  * add new line at the end of the test baseline (FlyingWithJerome)

  * add data+=option_len to skip unknown bytes (FlyingWithJerome)

  * better explanation to server cookie (FlyingWithJerome)

  * add edns-cookie testcase (FlyingWithJerome)

  * remove data+=option_len error in cookie and keepalive (FlyingWithJerome)

  * add EDNS cookie parsing (FlyingWithJerome)

  * add testcases (FlyingWithJerome)

  * add units (100ms) to comments (FlyingWithJerome)

  * initial adding EDNS TCP keepalive (FlyingWithJerome)

3.3.0-dev.119 | 2020-08-19 14:35:51 -0700

  * Misc strings.bif adjustments

    * Declare rfind_str() with correct return type
    * Fix compiler warnings for signed/unsigned comparisons (Jon Siwek, Corelight)

3.3.0-dev.117 | 2020-08-19 14:12:06 -0700

  * Use constexpr for IPAddr::v4_mapped_prefix declaration

    The inconsistent use of constexpr in the definition but not declaration
    causes an unrecoverable error for Coverity as seen in issue 1431151 (Jon Siwek, Corelight)

3.3.0-dev.115 | 2020-08-19 11:49:24 -0700

  * Update docs for configuring CI email notifications (Jon Siwek, Corelight)

    Secrets are now stored organization-wide rather than per-repository.

3.3.0-dev.113 | 2020-08-18 12:10:26 -0700

  * GH-467: Catch use of &add_func/&delete_func on IDs without values. (Robin Sommer, Corelight)

  * Fix compiler warning from missing namepsace qualification (Tim Wojtulewicz, Corelight)

3.3.0-dev.109 | 2020-08-14 10:00:07 -0700

  * GH-178: Add new string bif methods based on python string utilities (Tim Wojtulewicz, Corelight)

3.3.0-dev.107 | 2020-08-12 20:01:37 +0000

  * Accept bracketed ipv6 addresses (Justin Azoff, Corelight)

  * Fix minimize_info in ftp/main not returning a value.
    Fixes GH-1120 (Johanna Amann, Corelight)

  * Fix config reader regular expression for MUSL (Seth Hall, Corelight)

  * Moved verb ACTION_DROP to base/frameworks/notice/main.zeek.
    ACTION_DROP is not only part of catch-n-release subsystem and can be useful in other cases (Aashish Sharma)

  * Added new acld verbs of filter and nofilter (Aashish Sharma)

  * Fix some printf warnings with size_t values (Tim Wojtulewicz, Corelight)

3.3.0-dev.97 | 2020-08-11 11:34:17 -0700

  * Move Dict constants to detail namespace (Tim Wojtulewicz, Corelight)

  * Add a few missing deprecation fixes (Tim Wojtulewicz, Corelight)

  * Adjust Dict whitespace/style (Jon Siwek, Corelight)

  * Adjust more btest timings (Jon Siwek, Corelight)

  * Improve termination reliability/speed for brokerstore btests (Jon Siwek, Corelight)

  * General btest cleanup

    - Use `-b` most everywhere, it will save time.

    - Start some intel tests upon the input file being fully read instead of
      at an arbitrary time.

    - Improve termination condition for some sumstats/cluster tests.

    - Filter uninteresting output from some supervisor tests.

    - Test for `notice_policy.log` is no longer needed. (Jon Siwek, Corelight)

  * Update NEWS about change in Dictionary implementation (Jon Siwek, Corelight)

  * Improve Intel expire-item btest to be less time-sensitive (Jon Siwek, Corelight)

  * Improve btests with unstable table/set output ordering

    Particularly, the final output order of a table/set is sensitive to
    order of input/insertions and some tests were converting
    std::unordered_{set,map} to Zeek table/set and iteration over those
    standard containers may not always loop through elements in the same
    order across all platforms. (Jon Siwek, Corelight)

  * Update doc submodule (Jon Siwek, Corelight)

  * Adjust a few btests that were unstable due to time-sensitivity (Jon Siwek, Corelight)

  * Fix DNS script deleting a table element while iterating

    Doesn't typically cause any problems since the loop breaks after
    deleting, except there's now an assert in debug builds catching
    potential problems like this. (Jon Siwek, Corelight)

  * Improve a brokerstore btest to filter out Broker connection messages (Jon Siwek, Corelight)

  * Sort output of a few SumStats cluster tests

    The order of $epoch_result() function calls among all keys within a
    single epoch isn't consequential. (Jon Siwek, Corelight)

  * Fix extract_first_email_addr() to really return the first email

    The use of find_all() in extract_email_addrs_vec() extracted occurrences
    to an intermediate set and thus lost any sense of ordering.

    This changes extract_email_addrs_vec() to use find_all_ordered() and
    return all occurrences of email addresses found in the argument,
    included duplicates, with their order of occurrence preserved. (Jon Siwek, Corelight)

  * Add find_all_ordered() BIF

    Operates similar to find_all(), except returns a vector instead of
    set to allow preservation of order/duplicates. (Jon Siwek, Corelight)

  * Extend external test suite canonifier with set-sorting logic

    Two new canonifiers: one to sort the contents of conn.log "service"
    field and another to sort the contents of any field of type "set". (Jon Siwek, Corelight)

  * Update btests/baselines for OpenDict compat

    Haven't checked different build configurations yet, but all except
    a few SumStats tests are stable for me now.  The external tests
    are also completely failing, but haven't looked at those yet. (Jon Siwek, Corelight)

  * Fix new/malloc/delete/free mismatches in Dictionary code (Jon Siwek, Corelight)

  * Add explanation for a Dict TODO item (Jon Siwek, Corelight)

  * Fix compiler warning in Dictionary debug/dump printf format string (Jon Siwek, Corelight)

  * Overwrite old Dict.h/cc with new ones (Tim Wojtulewicz, Corelight)

  * Fix issue with sumstats script and fix baselines that were crashing previously (Tim Wojtulewicz, Corelight)

  * Extend the timeouts on a few intel tests, update baselines (Tim Wojtulewicz, Corelight)

  * Updating test baselines for new dictionary code due to changes in ordering of fields in the dictionary (Tim Wojtulewicz, Corelight)

  * Add namespaces for OpenDict files (Tim Wojtulewicz, Corelight)

  * Add unit tests from Dict into OpenDict files (Tim Wojtulewicz, Corelight)

  * Build fixups

    - Fix some clang-tidy warnings
    - Add std:: namespaces for ofstream and ios
    - Remove some unused methods (Tim Wojtulewicz, Corelight)

  * Massive formatting cleanup (jasonlue)

  * Remove key reference after insertion because the key may become invalid afterwards. (jasonlue)

  * add Clustered Hashing based Open Addressing Dict. To replace the existing dict, #define USE_OPEN_DICT (jasonlue)

3.3.0-dev.55 | 2020-08-10 09:57:36 -0700

  * GH-1100: Fix reported body-length of HTTP messages w/ sub-entities (Jon Siwek, Corelight)

    The body-lengths of sub-entities, like multipart messages, got counted
    twice by mistake: once upon the end of the sub-entity and then again
    upon the end of the top-level entity that contains all sub-entities.
    The size of just the top-level entity is the correct one to use.

3.3.0-dev.53 | 2020-08-05 12:35:50 -0700

  * Fix deprecation warnings in fuzz targets (Jon Siwek, Corelight)

  * Remove dbg_read_internal_state() (Jon Siwek, Corelight)

    It's never had a definition anyway and the namespacing/deprecation
    process was now causing a compile error in debug/fuzzing builds w/ GCC

3.3.0-dev.50 | 2020-08-04 20:22:48 +0000

  * Fix incorrect conflict detection of namespaced-enum-names

    E.g. defining a `Foo::RED` enum name when a `GLOBAL::RED` identifier
    already exists would previously be treated as an error, even though the
    names don't truly conflict. (Jon Siwek, Corelight)

  * GH-1080: Treat enum name re-use across different enum types as an error (Jon Siwek, Corelight)

  * GH-1080: Rename conflicting NetControl::DROP enum definitions

    ``NetControl::DROP`` had 3 conflicting definitions that could potentially
    be used incorrectly without any warnings or type-checking errors.
    Such enum redefinition conflicts are now caught and treated as errors,
    so the ``NetControl::DROP`` enums had to be renamed:

    * The use as enum of type ``Log::ID`` is renamed to ``NetControl::DROP_LOG``

    * The use as enum of type ``NetControl::CatchReleaseInfo`` is renamed to
      ``NetControl::DROP_REQUESTED``

    * The use as enum of type ``NetControl::RuleType`` is unchanged and still
      named ``NetControl::DROP`` (Jon Siwek, Corelight)

3.3.0-dev.44 | 2020-08-04 08:47:08 -0700

  * Silence compiler warnings related to deprecated TYPE_COUNTER (Jon Siwek, Corelight)

3.3.0-dev.42 | 2020-08-03 10:37:38 -0700

  * Remove counter type, only leaving compatibility enum. The type was unused/non-functional.
    See GH-1095 (Johanna Amann, Corelight).

3.3.0-dev.38 | 2020-08-01 09:21:17 -0700

  * Properly forward-declare zeek::plugin::plugin in DebugLogger (Tim Wojtulewicz, Corelight)

  * Rename Brofiler to ScriptCoverageManager (Tim Wojtulewicz, Corelight)

  * Move Conn and related types to zeek namespace (Tim Wojtulewicz, Corelight)

  * Move reassembler code to namespaces (Tim Wojtulewicz, Corelight)

  * Move Session code to namespaces (Tim Wojtulewicz, Corelight)

  * Move ODesc to zeek namespace (Tim Wojtulewicz, Corelight)

  * Move Stats and related classes to zeek::detail (Tim Wojtulewicz, Corelight)

  * Move a few low-use classes to namespaces (Tim Wojtulewicz, Corelight)

  * Move DebugLogger to zeek namespaces (Tim Wojtulewicz, Corelight)

  * Move all of the hashing classes/functions to zeek::detail namespace (Tim Wojtulewicz, Corelight)

  * Move all of the Packet-related classes to namespaces (Tim Wojtulewicz, Corelight)

  * Move EventMgr, EventHandler, and EventRegistry code to zeek namespace. Rename mgr to event_mgr. (Tim Wojtulewicz, Corelight)

  * Move Timer and PriorityQueue classes to namespaces (Tim Wojtulewicz, Corelight)

  * Move BroFile to zeek namespace, rename to File (Tim Wojtulewicz, Corelight)

  * Move Reporter to zeek namespace (Tim Wojtulewicz, Corelight)

  * Move DNS_Mgr to zeek::detail namespace (Tim Wojtulewicz, Corelight)

  * Move BroSubstring to zeek::detail, rename to Substring (Tim Wojtulewicz)

  * Move regex matching code to zeek namespaces (Tim Wojtulewicz, Corelight)

  * Move all of the rule matching code to zeek::detail (Tim Wojtulewicz, Corelight)

  * Move IP Addr/Prefix/Header classes into namespaces (Tim Wojtulewicz, Corelight)

  * Move all debugger code into the zeek::detail namespace (Tim Wojtulewicz, Corelight)

  * Restore globally-namespaced plugin_mgr variable, move zeek::plugin::plugin_mgr to zeek::plugin_mgr (Tim Wojtulewicz, Corelight)

  * Move all base analyzer classes to zeek::analyzer namespace (Tim Wojtulewicz, Corelight)

  * Move base Tag class to zeek namespace (Tim Wojtulewicz, Corelight)

  * Update submodule

     [nomail] (Johanna Amann, Corelight)

  * Update submodule(s)

     [nomail] (Jon Siwek, Corelight)


3.3.0-dev.11 | 2020-07-30 17:47:44 +0000

  * Use .empty() instead of checking size against zero (Tim Wojtulewicz, Corelight)

  * Use properly-sized loop variables or convert to ranged-for (bugprone-too-small-loop-variable) (Tim Wojtulewicz, Corelight)

  * Fix potential heap-use-after-free access (Dominik Charousset, Corelight)

3.3.0-dev.4 | 2020-07-28 19:56:35 +0000

  * GH-1083: Fix Input Framework 'change' events for 'set' destinations (Jon Siwek, Corelight)

3.3.0-dev.1 | 2020-07-27 12:04:48 -0700

  * Update site/local.zeek compatibility test (Jon Siwek, Corelight)

3.3.0-dev | 2020-07-27 11:49:28 -0700

  * Reserve spot in NEWS for 4.0.0 (Jon Siwek, Corelight)

3.2.0-dev.992 | 2020-07-27 11:36:53 -0700

  * Fix potential NetbiosSSN analyzer stack overflow (Jon Siwek, Corelight)

    The Contents_NetbiosSSN analyzer used a recursive message parsing
    function that determined the size of the next message from the input
    packet-data itself.  A packet containing a sequence of many small
    messages could cause a stack overflow since a recursion happened after
    processing each message.

  * Fix potential DNS analyzer stack overflow (Jon Siwek, Corelight)

    The Contents_DNS analyzer used a recursive message parsing function that
    determined the size of the next message from the input packet-data
    itself.  A packet containing a sequence of many small messages could
    cause a stack overflow since a recursion happened after processing
    each message.

    Credit to OSS-Fuzz for discovery
    https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=24272

3.2.0-dev.988 | 2020-07-27 11:19:20 -0700

  * Update submodules to release versions in prep for release/3.2 branch (Jon Siwek, Corelight)

3.2.0-dev.986 | 2020-07-24 13:57:19 -0700

  * Misc/minor cleanup after reviewing new Broker table store code (Jon Siwek, Corelight)

    Just some naming/style + minor memory management tweaks (avoid
    some copies; use move semantics).

3.2.0-dev.985 | 2020-07-24 11:40:22 -0700

  * MySQL: Fix EOF parsing logic bug. (Vlad Grigorescu)

    We were correctly NOT expecting an EOF, but because we were parsing the
    header and then not parsing the rest, we would get out of sync

3.2.0-dev.979 | 2020-07-24 09:03:06 -0700

  * Implement EDNS Client Subnet Option (Ron Wellman)

3.2.0-dev.974 | 2020-07-23 13:02:13 -0700

    * origin/topic/jsiwek/gh-1076-fix-random:
      Deprecate bro_srandom(), replace with zeek::seed_random().
      Add zeek::max_random() & fix misuse of RAND_MAX w/ zeek::random_number()
      Deprecate bro_random(), replace with zeek::random_number()
      Deprecate bro_prng(), replace with zeek::prng()
      GH-1076: Fix bro_srandom() to replace 0 seeds with 1
      GH-1076: Fix bro_prng() implementation
      GH-1076: Fix use of getrandom() (Tim Wojtulewicz, Corelight)

  * Deprecate bro_srandom(), replace with zeek::seed_random().

    Avoiding zeek::srandom() to avoid potential for confusion with srandom() (Jon Siwek, Corelight)

  * Add zeek::max_random() & fix misuse of RAND_MAX w/ zeek::random_number()

    In deterministic mode, RAND_MAX is not related to the result of
    zeek::random_number() (formerly bro_random()), but some logic was
    using RAND_MAX as indication of the possible range of values.  The
    new zeek::max_random() will give the correct upper-bound regardless
    of whether deterministic-mode is used. (Jon Siwek, Corelight)

  * Deprecate bro_random(), replace with zeek::random_number()

    Avoiding the use of zeek::random() due to potential for confusion
    with random(). (Jon Siwek, Corelight)

  * Deprecate bro_prng(), replace with zeek::prng()

    The type used for storing the state of the RNG is changed from
    `unsigned int` to `long int` since the former has a minimal range
    of [0, 65,535] while the RNG function itself has a range of
    [1, 2147483646].  A `long int` must be capable of
    [−2147483647, +2147483647] and is also the return type of `random()`,
    which is what zeek::prng() aims to roughly parity. (Jon Siwek, Corelight)

  * GH-1076: Fix bro_srandom() to replace 0 seeds with 1

    The bro_prng() implementation cannot generate 0 as a result since it
    causes every subsequent number from the PRNG to also be 0, so use the
    number 1 instead of 0. (Jon Siwek, Corelight)

  * GH-1076: Fix bro_prng() implementation

    The intermediate result of the PRNG used unsigned storage, preventing
    the ( result < 0 ) branch from ever being evaluated.  This could cause
    return values to exceed the modulus as well as RAND_MAX.

    One interesting effect of this is potential for the rand() BIF to
    return values outside the requested maximum limit.

    Another interesting effect of this is that a PacketFilter may start
    randomly dropping packets even if it was not configured for
    random-packet-drops. (Jon Siwek, Corelight)

  * GH-1076: Fix use of getrandom()

    The availability and use of getrandom() actually caused unrandom and
    deterministic results in terms of Zeek's random number generation. (Jon Siwek, Corelight)

3.2.0-dev.965 | 2020-07-23 08:31:41 -0700

  * GH-1068: Add zeek symlink to allow "zeek/Foo.h" include style

    Also changes `zeek-config --include_dir` to now output
    `$prefix/include:$prefix/include/zeek` instead of only
    `$prefix/include/zeek` (Jon Siwek, Corelight)

3.2.0-dev.962 | 2020-07-21 12:39:34 -0700

  * Add zeek-archiver tool as submodule

    This is added to the default Zeek build as a convenience since it's the
    new suggested method for archiving the logs produced by a Supervised
    Zeek Cluster. (Jon Siwek, Corelight)

3.2.0-dev.959 | 2020-07-21 15:34:59 +0000

  * Broker Store table synchronizatio, (Johanna Amann, Corelight)

    Zeek now supports synchronizing tables/sets across clusters using
    a backing Broker store. The same feature also allows persistent
    storage of data in tables/sets over Zeek restarts. This feature is
    implemented using the new ``&backend`` attribute.

    To synchronize a table over a cluster, you can, e.g., use:

        global t: table[string] of count &backend=Broker::MEMORY;

    This feature is documented in detail here:

        https://docs.zeek.org/en/current/frameworks/broker.html#broker-store-backed-zeek-tables-for-data-synchronization-and-persistence

    This feature is experimental and the syntax/specifics can change in the future.

3.2.0-dev.919 | 2020-07-17 16:37:11 -0700

  * Use namespaced version of Location to silence warnings (Tim Wojtulewicz, Corelight)

3.2.0-dev.915 | 2020-07-17 16:10:46 -0700

  * Various deprecation fixes, reported by failed Spicy builds (Tim Wojtulewicz, Corelight)

    - Add deprecated version of ID::SetType() that takes Type*
    - Add deprecated versions of zeek::set_location in the global namespace
    - Fix global namespace version of lookup_ID to return ID*

  * Add deprecated version of EnumType::GetVal() to return EnumVal* (Tim Wojtulewicz, Corelight)

    Rename IntrusivePtr version to GetEnumVal

3.2.0-dev.911 | 2020-07-17 22:32:42 +0000

  * Fix race condition in ensure_dir()

    If something else created the dir between the stat() and mkdir(),
    it previously reported that as a failure. (Jon Siwek, Corelight)

  * Fix tokenize_string() to work with delimiters of length > 1. (Jon Siwek, Corelight)

3.2.0-dev.908 | 2020-07-17 10:19:07 +0000

  * GH-734: Improve handling of lambdas that escape enclosing frame.
    (Jon Siwek, Corelight)

3.2.0-dev.906 | 2020-07-17 10:08:31 +0000

  * GH-837: Add test cases for mismatched Broker SSL configs. (Jon Siwek, Corelight)

  * GH-837: Emit reporter errors for Broker errors. (Jon Siwek,
    Corelight)

  * Fix incorrect/missing Broker error status code numbers. (Jon Siwek, Corelight)

3.2.0-dev.902 | 2020-07-14 19:10:36 -0700

  * GH-1062: fix integer conversion errors related to Tag subtypes (Jon Siwek, Corelight)

3.2.0-dev.899 | 2020-07-14 00:02:05 +0000

  * Improve Broker store API's handling of invalid arguments

    * Some methods mistakenly returned a bool instead of QueryResult
      when passed an invalid `opaque of Broker::Store` handle.

    * Now generates a runtime exception for store_name() and is_closed()
      calls that pass an invalid `opaque of Broker::Store` handle as any
      returned value can't be reasonably used in any subsequent logic.

    * Descriptions of any invalid arguments are now given in the error
      message. (Jon Siwek, Corelight)

  * Add zeek::detail::emit_builtin_exception() functions

    These work like zeek::emit_builtin_error(), but also throw an InterpreterException (Jon Siwek, Corelight)

  * GH-1024: fix crash on passing wrong types to Broker store API (Jon Siwek, Corelight)

3.2.0-dev.894 | 2020-07-13 12:12:17 -0700

  * GH-1019: deprecate icmp_conn params for ICMP events

    Previously, a single `icmp_conn` record was built per ICMP "connection"
    and re-used for all events generated from it.  This may have been a
    historical attempt at performance optimization, but:

      * By default, Zeek does not load any scripts that handle ICMP events.

      * The one script Zeek ships with that does handle ICMP events,
        "detect-traceroute", is already noted as being disabled due to
        potential performance problems of doing that kind of analysis.

      * Re-use of the original `icmp_conn` record tends to misreport
        TTL and length values since they come from original packet instead
        of the current one.

      * Even if we chose to still re-use `icmp_conn` records and just fill
        in a new TTL and length value each packet, a user script could have
        stored a reference to the record and not be expecting those values
        to be changed out from underneath them.

    Now, a new `icmp_info` record is created/populated in all ICMP events
    and should be used instead of `icmp_conn`.  It also removes the
    orig_h/resp_h fields as those are redundant with what's already
    available in the connection record. (Jon Siwek, Corelight)

3.2.0-dev.892 | 2020-07-13 12:10:22 -0700

  * Fix wrong frame offsets for locals of alternate event/hook prototypes

    Local frame offsets were being assigned based on number of the alternate
    prototype's parameters, which may end up having less total parameters
    than the canonical prototype, causing the local value to incorrectly
    overwrite an event/hook argument value. (Jon Siwek, Corelight)

  * Add deprecation expression to deprecated prototype/parameter messages (Jon Siwek, Corelight)

  * Improve "use of deprecated prototype" warning message

    The location information now points out the place of the deprecated
    prototype instead of the location where the ID was initially declared
    (which may not itself be a deprecated prototype). (Jon Siwek, Corelight)

  * Emit deprecation warning for use of &deprecated function parameters

    Particularly, this is meant for using &deprecated on canonical
    event/hook prototype parameters to encourage users to create handlers
    to another, non-deprecated prototype.  i.e. for canonical prototypes,
    we may not always want to put &deprecated directly on the prototype
    itself since that signals deprecation of the ID entirely. (Jon Siwek, Corelight)

3.2.0-dev.885 | 2020-07-10 11:20:41 -0700

  * Add more error checks to shadow log parsing (Jon Siwek, Corelight)

    i.e. Coverity warns about possible use of ftell() negative return value

3.2.0-dev.884 | 2020-07-09 14:09:58 -0700

  * Add Supervisor::{stdout,stderr}_hook (Jon Siwek, Corelight)

    These allow capturing/handling the stdout/stderr of child processes
    via Zeek scripts.

  * Add Log::rotation_format_func and Log::default_rotation_dir options (Jon Siwek, Corelight)

    These may be redefined to customize log rotation path prefixes,
    including use of a directory.  File extensions are still up to
    individual log writers to add themselves during the actual rotation.

    These new also allow for some simplication to the default
    ASCII postprocessor function: it eliminates the need for it doing an
    extra/awkward rename() operation that only changes the timestamp format.

    This also teaches the supervisor framework to use these new options
    to rotate ascii logs into a log-queue/ directory with a specific
    file name format (intended for an external archiver process to
    monitor separately).

  * Allow Zeek function equality comparisons (Jon Siwek, Corelight)

    Equality between two functions acts like a pointer comparison.

  * Implement leftover log rotation/archival for supervised nodes (Jon Siwek, Corelight)

    This helps prevent a node from being killed/crashing in the middle
    of writing a log, restarting, and eventually clobbering that log
    file that never underwent the rotation/archival process.

    The old `archive-log` and `post-terminate` scripts as used by
    ZeekControl previously implemented this behavior, but the new logic is
    entirely in the ASCII writer.  It uses ".shadow" log files stored
    alongside the real log to help detect such scenarios and rotate them
    correctly upon the next startup of the Zeek process.

  * Move some supervisor structures to detail namespace (Jon Siwek, Corelight)

    * StemHandle -> detail::SupervisorStemHandle
    * SupervisedNode -> detail::SupervisedNode
    * Supervisor::Node -> detail::SupervisorNode

  * Improve default supervisor stdout/stderr handling (Jon Siwek, Corelight)

    The stdout/stderr of child processes is now redirected over a pipe back
    to the supervisor process so that it can prefix the output with
    the name of the emitting node.

  * Factor Supervisor initialization data to separate struct from Stem (Jon Siwek, Corelight)

  * Move ParentProcessCheckTimer to detail namespace (Jon Siwek, Corelight)

3.2.0-dev.874 | 2020-07-08 17:46:20 -0700

  * scan.l: Remove "constant" did_module_restore logic / dead code (Arne Welzel)

3.2.0-dev.872 | 2020-07-08 12:25:11 -0700

  * Fix FreeBSD CI script to install right SWIG package (Jon Siwek, Corelight)

3.2.0-dev.870 | 2020-07-08 15:41:15 +0000

  * Fix WhileStmt to call Stmt(Tag) constructor. (Jon Siwek,
    Corelight)

3.2.0-dev.868 | 2020-07-08 15:39:46 +0000

  * GH-928: use realpath() instead of inode to de-duplicate scripts.
    (Jon Siwek, Corelight)

3.2.0-dev.864 | 2020-07-06 21:06:41 -0700

  * GH-1040: Add zero-indexed version of str_split (Tim Wojtulewicz, Corelight)

3.2.0-dev.862 | 2020-07-06 20:40:44 -0700

  * GH-1041: Move compress_path to a bif that uses normalize_path (Tim Wojtulewicz, Corelight)

3.2.0-dev.859 | 2020-07-06 14:55:00 +0000

  * Add backtrace() and print_backtrace() BIF functions. (Jon Siwek,
    Corelight)

3.2.0-dev.855 | 2020-07-03 14:29:55 -0700

  * Fix --enable-mobile-ipv6 build (Jon Siwek, Corelight)

3.2.0-dev.854 | 2020-07-03 14:17:42 -0700

  * Fix namespace of GetCurrentLocation() to zeek::detail (Jon Siwek, Corelight)

3.2.0-dev.853 | 2020-07-02 19:24:09 -0700

  * Rename BroString files to ZeekString (Tim Wojtulewicz, Corelight)

  * Rename BroObj to Obj (Tim Wojtulewicz, Corelight)

  * Rename BroString to zeek::String (Tim Wojtulewicz, Corelight)

  * Move Func up to zeek namespace, rename BroFunc to ScriptFunc (Tim Wojtulewicz, Corelight)

  * Mark global val_mgr as deprecated and fix uses of it to use namespaced version (Tim Wojtulewicz, Corelight)

  * Update binpac and bifcl submodules with review changes (Tim Wojtulewicz, Corelight)

  * Move Location to zeek::detail and BroObj to zeek (Tim Wojtulewicz, Corelight)

  * Move BroString to zeek namespace (Tim Wojtulewicz, Corelight)

  * Move Dictionary/PDict, List/PList, and Queue/PQueue to zeek namespace (Tim Wojtulewicz, Corelight)

  * Remove typedef that should have been removed in 3.1 (Tim Wojtulewicz, Corelight)

  * Move Func and associated classes into zeek::detail namespace (Tim Wojtulewicz, Corelight)

  * Move Frame and Scope to zeek::detail namespace (Tim Wojtulewicz)

  * Move all Val classes to the zeek namespaces (Tim Wojtulewicz, Corelight)

  * Use type aliases for IntrusivePtr definitions (Tim Wojtulewicz, Corelight)

  * Move deprecation macro to zeek-config.h.in to avoid having to over-include util.h (Tim Wojtulewicz, Corelight)

  * Move IntrusivePtr and utility methods to the zeek namespace (Tim Wojtulewicz, Corelight)

3.2.0-dev.834 | 2020-07-02 15:57:09 -0700

  * GH-786: fix Val::As methods casting incomplete types (Jon Siwek, Corelight)

  * GH-786: fix Stmt::As methods casting incomplete types (Jon Siwek, Corelight)

  * GH-786: fix Expr::As methods casting incomplete types (Jon Siwek, Corelight)

3.2.0-dev.830 | 2020-07-02 11:36:28 -0700

  * GH-786: Move Type::As methods to cc file so they have type info for casting safely (Tim Wojtulewicz, Corelight)

3.2.0-dev.828 | 2020-07-01 09:57:23 -0700

  * Fix a deprecation warning compiling with GCC (Jon Siwek, Corelight)

3.2.0-dev.827 | 2020-06-30 16:54:22 -0700

  * Deprecate plugin::HookType and plugin::component::Type in a different way (Tim Wojtulewicz, Corelight)

  * Deprecate init_class and IDScope in another way. (Tim Wojtulewicz, Corelight)

    This also renamed init_class to InitClass for consistency.

  * Deprecate TypeTag and friends in a different way (Tim Wojtulewicz, Corelight)

  * Deprecate attr_tag in a different way, rename to AttrTag (Tim Wojtulewicz, Corelight)

3.2.0-dev.822 | 2020-06-30 15:54:38 -0700

  * Revert Attributes::Attrs back to return an attr_list and mark it deprecated (Tim Wojtulewicz, Corelight)

  * GH-1034: Revert TypeList::Types() back to return a type_list* and mark it deprecated (Tim Wojtulewicz, Corelight)

3.2.0-dev.817 | 2020-06-29 19:23:21 +0000

  * GH-1036: change print.log to log network time instead of current (Jon Siwek, Corelight)

3.2.0-dev.815 | 2020-06-29 10:25:05 -0700

  * reduce memory usage of ConnPolling (Justin Azoff)

    Instead of scheduling the event with the full 'connection' record,
    schedule it with the smaller 'conn_id' record.

3.2.0-dev.813 | 2020-06-26 16:25:34 +0000

  * Fix several issues with command-line option redefs

    * Variables of `string` type can now be set to an empty string

    * Trying to set a variable with non-`string` type to an empty value
      now emits an error instead of silently doing nothing

    * Providing an invalid identifier now emits an "unknown identifier"
      error instead of silently doing nothing (Jon Siwek, Corelight)

  * Fix "possibly-truncated" compiler warning in BuildJSON snprintf() (Jon Siwek, Corelight)

  * GH-1025: allow copying/cloning of `opaque of Broker::Store`

    Implemented simply as a reference count increment of the
    data store handle. (Jon Siwek, Corelight)

  * Fix shadowed variable that breaks lookup_hostname(). (Jon Siwek, Corelight)

3.2.0-dev.804 | 2020-06-25 23:58:59 -0700

  * Extend dns_request, dns_reject, and dns_query_reply events with original_query param (Ryan Victory)

    Where the "query" parameter is downcased, the "original_query" is
    passed unmodified to these events.

    Added a policy/protocols/dns/log-original-query-case.zeek script that adds
    the original_query to the dns.log file as well.

  * Convert to using permissions to check for access to cirrus variables in benchmark script (Tim Wojtulewicz, Corelight)

3.2.0-dev.798 | 2020-06-19 13:25:02 -0700

  * Remove last_access_time from TableEntryVal. (Johanna Amann, Corelight)

    Turns out - this was no longer used. And it takes up at least 8 bytes
    of space for every single table/set entry.

3.2.0-dev.796 | 2020-06-18 20:40:59 +0000

  * cmake: Make musl support more distro agnostic (Andrew Benson)

3.2.0-dev.794 | 2020-06-18 20:04:06 +0000

  * Minimize data published for expected FTP data channel analysis

    Previously, more data than could effectively be utilized by any remote
    Zeek was published (e.g. full list of pending commands or other
    transient state that may add up to non-trivial amount of bytes). (Jon Siwek, Corelight)

  * GH-998: Fix Reporter::conn_weird() to handle expired connections

    This introduces a new sampling state-map for expired connections to fix
    segfaults that previously occured when passing in a `connection` record
    to `Reporter::conn_weird()` for which the internal `Connection` object
    had already been expired and deleted.  This also introduces a new event
    called `expired_conn_weird`, which is similar to `conn_weird`, except
    the full `connection` record is no longer available, just the `conn_id`
    and UID string. (Jon Siwek, Corelight)

  * Place build file in explicit location for benchmarking to work correctly (Tim Wojtulewicz, Corelight)

  * Update highwayhash submodule to upstream.
    This includes a patch by us, which fixes the compile on AARCH64.
    Fixes GH-1006. (Johanna Amann, Corelight)

3.2.0-dev.788 | 2020-06-15 11:04:20 -0700

  * Fix location where CI places build.tgz (Jon Siwek, Corelight)

3.2.0-dev.787 | 2020-06-15 10:27:48 -0700

  * Add new CI task for running benchmarks on a remote host (Tim Wojtulewicz, Corelight)

3.2.0-dev.783 | 2020-06-11 23:21:41 -0700

  * Compare pcap_next_ex() result to PCAP_ERROR/PCAP_ERROR_BREAK (Jon Siwek, Corelight)

  * GH-977: Improve pcap error handling (Jon Siwek, Corelight)

    Switches from pcap_next() to pcap_next_ex() to better handle all error
    conditions.  This allows, for example, to have a non-zero exit code for
    a Zeek process that fails to fully process all packets in a pcap file.

  * Remove not-useful code in iosource::Manager::OpenPktSrc (Jon Siwek, Corelight)

3.2.0-dev.779 | 2020-06-11 23:17:46 -0700

  * Rename BroType to zeek::Type (Tim Wojtulewicz, Corelight)

  * Move Type types to zeek namespace (Tim Wojtulewicz, Corelight)

  * Move Flare/Pipe from the bro namespace to zeek::detail (Tim Wojtulewicz, Corelight)

  * Move Attr to the zeek::detail namespace (Tim Wojtulewicz, Corelight)

  * Move Trigger into the zeek::detail namespace (Tim Wojtulewicz, Corelight)

  * Move ID to the zeek::detail namespace (Tim Wojtulewicz, Corelight)

  * Move Anon.h into zeek::detail namespace (Tim Wojtulewicz, Corelight)

  * Mark all of the aliased classes in plugin/Plugin.h deprecated (Tim Wojtulewicz, Corelight)

    And fix all of the plugins that were using them

  * Move all of the base plugin classes into the zeek::plugin namespace (Tim Wojtulewicz, Corelight)

  * Expr: move all classes into zeek::detail (Tim Wojtulewicz, Corelight)

  * Stmt: move Stmt classes into zeek::detail namespace (Tim Wojtulewicz, Corelight)

  * Add utility macro for creating namespaced aliases for classes (Tim Wojtulewicz, Corelight)

3.2.0-dev.763 | 2020-06-10 16:34:31 -0700

  * Optimize Connection::RemovalEvent() for bare-mode usage (Jon Siwek, Corelight)

    It was creating RecordVals even if they wouldn't be used by any event
    handler and that situation is common/expected for `zeek -b` mode.
    Normally, there's at least the tunnel scripts with a `new_connection`
    handler causing the connection RecordVals to be built.

3.2.0-dev.762 | 2020-06-09 15:19:15 -0700

  * Restrict Cirrus CI to only zeek repo's branches (Jon Siwek, Corelight)

3.2.0-dev.761 | 2020-06-09 09:39:31 -0700

  * Prevent IP fragment reassembly on packets without minimal IP header (Jon Siwek, Corelight)

    The IP fragment reassembly process assumes a packet contains at least
    the minimum IP header, but such a check did not previously occur,
    resulting in a heap buffer over-read.  For example, a self-reported
    IPv4 IHL field with a value less than minimum IPv4 header length of
    20 bytes.  Such packets likely aren't routable on their own, but one
    can create an artifical pcap like that or possibly encapsulate it
    within another protocol to trigger this bug.

  * Fix potential leak of Analyzers added to tree during Analyzer::Done (Jon Siwek, Corelight)

    Credit to OSS-Fuzz for discovery
    https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22630

  * Fix multiple content-transfer-encoding headers causing a memory leak (Jon Siwek, Corelight)

    The MIME analyzer may also leak if it sees many multipart boundary
    parameters.

    Credit to OSS-Fuzz for discovery
    https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22871

  * Fix potential stack overflow in NVT analyzer (Jon Siwek, Corelight)

    The NVT_Analyzer (e.g. as instantiated to support the FTP analyzer)
    uses a recursive parsing function that may only advance one byte at a
    time and can easily cause a stack overflow as a result.  This change
    replaces the recursive calls with equivalent iterative logic.

    Credit to OSS-Fuzz for discovery
    https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22898
    https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=22972

  * Fix NVT analyzer memory leak from multiple telnet authn name options (Jon Siwek, Corelight)

    Credit to OSS-Fuzz for discovery
    https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=23069

3.2.0-dev.751 | 2020-06-08 22:30:18 +0000

  * GH-999: Stop formatting DHCP Client ID Hardware Type 0 as MAC

    For `DHCP::ClientID$hwtype` fields equal to 0, the `hwaddr` field is
    no longer misformatted as a MAC and instead just contains the raw bytes
    seen in the DHCP Client ID Option. (Jon Siwek, Corelight)

3.2.0-dev.749 | 2020-06-08 11:22:45 -0700

  * Remove inline from some static KeyedHash members (Jon Siwek, Corelight)

    Coverity Scan builds currently encounter catastrophic error, claiming
    alignas requires use on both declaration and definition, so appears to
    actually not understand "static inline" in combo with alignas.

  * Improve Func.h inclusion (Jon Siwek, Corelight)

    Now forward declares some Broker types since Broker/CAF headers
    generally slow things down and also Coverity Scan currently has a
    catastrophic error on some CAF headers.

    Also a few other changes to EventHandler/BifReturnVal to reduce number
    of places that depend on Func.h.

  * Rename aux/ to auxil/ (Jon Siwek, Corelight)

    Since "aux" is not an allowed file/dir name on Windows.

3.2.0-dev.744 | 2020-06-04 15:11:56 -0700

  * Fix use-after-move of proc_status_file breaking -U flag (Jon Siwek, Corelight)

  * Change Cirrus CI config to use org-level secured variable (Jon Siwek, Corelight)

3.2.0-dev.742 | 2020-06-03 13:29:12 -0700

  * GH-973: Fix the return type of topk_get_top()

    e.g. when using `string` elements it should return `vector of string`,
    not `vector of list of string` like it did before. (Jon Siwek, Corelight)

3.2.0-dev.740 | 2020-06-03 13:17:26 -0700

  * Deprecate Val(double, TypeTag) ctor, add TimeVal/DoubleVal subclasses

    This also updates all usages of the deprecated Val ctor to use
    either IntervalVal, TimeVal, or DoubleVal ctors.  The reason for
    doing away with the old constructor is that using it with TYPE_INTERVAL
    isn't strictly correct since there exists a more specific subclass,
    IntervalVal, with overriden ValDescribe() method that ought to be used
    to print such values in a more descriptive way. (Jon Siwek, Corelight)

3.2.0-dev.738 | 2020-06-02 18:13:50 -0700

  * GH-985: Fix descriptions of double_to_interval() return values (Jon Siwek, Corelight)

    The BIF was not returning an IntervalVal which has an overriden
    ValDescribe() method that allows for prettier printing like "6.0 secs"
    instead of just "6.0".

3.2.0-dev.736 | 2020-06-02 12:37:56 -0700

  * Decrease number of CPUs/memory for Cirrus CI tasks (Jon Siwek, Corelight)

3.2.0-dev.735 | 2020-06-02 10:55:43 -0700

  * Change CI script to compile from build/Makefile (Jon Siwek, Corelight)

3.2.0-dev.734 | 2020-06-01 20:44:29 -0700

  * Add missing include in util.cc (Jon Siwek, Corelight)

3.2.0-dev.733 | 2020-06-01 19:25:37 -0700

  * Remove Analyzer.h from bro-bif.h (Tim Wojtulewicz, Corelight)

  * Remove IPAddr.h from Reporter.h (Tim Wojtulewicz, Corelight)

  * Remove the inclusion of Func.h from NetVar.h (Tim Wojtulewicz, Corelight)

3.2.0-dev.727 | 2020-06-01 10:57:16 -0700

  * Integrate review feedback

    * Add deprecation for MIME_Entity::ContentType(), use GetContentType()

    * Add deprecation for MIME_Entity::ContentSubType(), use GetContentSubType()

    * Add deprecation for MIME_Message::BuildHeaderVal(), use ToHeaderVal()

    * Add deprecation for MIME_Message::BuildHeaderTable(), use ToHeaderTable()

    * Add deprecation for mime::new_string_val(), use mime::to_stringval()

    * Add deprecation for ARP_Analyzer::ConstructAddrVal(), use ToAddrVal()

    * Add deprecation for ARP_Analyzer::EthAddrToStr(), use ToEthAddrStr()

    * Change the Func::Call() replacement to be named Func::Invoke() (Jon Siwek, Corelight)

  * Switch Broker Val converter visitor to return IntrusivePtr (Jon Siwek, Corelight)

  * Change BroFunc ctor to take const-ref IntrusivePtr<ID> (Jon Siwek, Corelight)

  * Add version of Frame::SetElement() taking IntrusivePtr<ID>

    Expect the version using raw ID* could go away eventually, but this is
    convenience for the meantime. (Jon Siwek, Corelight)

  * Change Scope/Func inits from id_list* to vector<IntrusivePtr<ID>> (Jon Siwek, Corelight)

  * Change Scope::GenerateTemporary() to return IntrusivePtr (Jon Siwek, Corelight)

  * Deprecate Scope::ReturnType(), replace with GetReturnType() (Jon Siwek, Corelight)

  * Deprecate Scope::ScopeID(), replace with GetID() (Jon Siwek, Corelight)

  * Switch parsing to use vector<IntrusivePtr<Attr>> from attr_list

    This allows improved passing/storing of Attr references to Exprs,
    TypeDecl, Scope, etc. (Jon Siwek, Corelight)

  * Deprecate TableVal::FindAttr(), replace with GetAttr() (Jon Siwek, Corelight)

  * Deprecate TypeDecl::FindAttr(), replace with GetAttr() (Jon Siwek, Corelight)

  * Deprecate ID::FindAttr(), replace with GetAttr() (Jon Siwek, Corelight)

  * Deprecate Attributes::FindAttr(), replace with Find() (Jon Siwek, Corelight)

  * Deprecate Attributes::AddAttrs(Attributes*)

    Replaced with version taking an IntrusivePtr parameter (Jon Siwek, Corelight)

  * Add Attributes ctor that takes IntrusivePtrs (Jon Siwek, Corelight)

  * Change Attributes to store std:vector<IntrusivePtr<Attr>>

    This also changes the return type of Attributes::Attrs() from attr_list* (Jon Siwek, Corelight)

  * Change Attr::SetAttrExpr() to non-template (Jon Siwek, Corelight)

  * Deprecate Attr::AttrExpr(), replace with GetExpr() (Jon Siwek, Corelight)

  * Deprecate ID::Attrs(), replace with GetAttrs() (Jon Siwek, Corelight)

  * Remove weak_ref param from ID::SetVal()

    It was not used anywhere. (Jon Siwek, Corelight)

  * Store weak ref boolean along with Frame element Vals (Jon Siwek, Corelight)

  * Deprecate Frame::GetElement(ID*), replace with GetElementByID() (Jon Siwek, Corelight)

  * Deprecate Frame::NthElement(), replace with GetElement() (Jon Siwek, Corelight)

  * Switch Frame::SetElement() to use IntrusivePtr (Jon Siwek, Corelight)

  * Store IntrusivePtrs in Frame (Jon Siwek, Corelight)

  * Deprecate Plugin::HookCallFunction(), replace with HookFunctionCall()

    This also changes the argument type of Func::operator() to zeek::Args*
    to allow plugins to be able to alter function arguments in place as
    was previously documented. (Jon Siwek, Corelight)

  * Switch plugin::Manager::HookCallFunction() to return IntrusivePtr

    The plugin::Plugin side of things is not (yet) changed. (Jon Siwek, Corelight)

  * Port remaining file analysis API to use IntrusivePtr (Jon Siwek, Corelight)

  * Deprecate file analyzer construction methods taking raw RecordVal*

    Replaced with versions that instead take IntrusivePtr (Jon Siwek, Corelight)

  * Deprecate file_analysis::Analyzer::Args(), replace with GetArgs() (Jon Siwek, Corelight)

  * Deprecate file_analysis::File::GetVal(), replace with ToVal() (Jon Siwek, Corelight)

  * Change file_analysis::File::GetID() to return const-ref (Jon Siwek, Corelight)

  * Fix build: some don't like IntrusivePtr default arg w/ incomplete type (Jon Siwek, Corelight)

  * Change bro_broker::DataVal::ScriptDataType() to IntrusivePtr (Jon Siwek, Corelight)

  * Change find_nested_record_types() to take IntrusivePtr (Jon Siwek, Corelight)

  * Deprecate TypeType::Type(), replace with GetType() (Jon Siwek, Corelight)

  * Add is_assignable() overload taking TypeTag (Jon Siwek, Corelight)

  * Add is_atomic_type() overloads for IntrusivePtr (Jon Siwek, Corelight)

  * Add same_type() overloads for IntrusivePtr args (Jon Siwek, Corelight)

  * Change merge_types() to take IntrusivePtrs (Jon Siwek, Corelight)

  * Deprecate IndexType::Indices(), replace with GetIndices() (Jon Siwek, Corelight)

  * Add is_vector() methods taking const-ref IntrusivePtr (Jon Siwek, Corelight)

  * Factor static-local nil IntrusivePtrs to global locations

    Minor optimization to remove any run-time impact. (Jon Siwek, Corelight)

  * Deprecate VectorVal::Lookup(), replace with At() (Jon Siwek, Corelight)

  * Switch VectorVal BroValUnion to store std::vector<IntrusivePtr<Val>>

    This changes the return type of AsVector() from std::vector<Val*>* (Jon Siwek, Corelight)

  * Deprecate VectorVal::Insert() taking raw Val*, use IntrusivePtr (Jon Siwek, Corelight)

  * Deprecate VectorVal::Assign methods taking raw Val*

    And adapt usages to pass in to alternate method taking IntrusivePtr (Jon Siwek, Corelight)

  * Merge remote-tracking branch 'origin/master' into topic/jsiwek/gh-893-intrusive-ptr-migration (Jon Siwek, Corelight)

  * Change EventHandler to store IntrusivePtr<Func>

    Also deprecates the LocalHandler() and SetLocalHandler() methods,
    replaced with GetFunc() and SetFunc(). (Jon Siwek, Corelight)

  * Add Val::AsFuncPtr() convenience method

    Since it's not trivial to store IntrusivePtr in BroValUnion and also
    not satisfying to store IntrusivePtr*. (Jon Siwek, Corelight)

  * Deprecate StringVal::Substitute(), replace with Replace() (Jon Siwek, Corelight)

  * Switch a TableVal::CallChangeFunc param to IntrusivePtr (Jon Siwek, Corelight)

  * Fix ambiguous ODesc::Add() call (Jon Siwek, Corelight)

  * Minor TableVal::Assign() ref-counting optimization (Jon Siwek, Corelight)

  * Improve TableVal HashKey management

    * Deprecated ComputeHash() methods and replaced with MakeHashKey()
      which returns std::unique_ptr<HashKey>

    * Deprecated RecoverIndex() and replaced with RecreateIndex()
      which takes HashKey& and returns IntrusivePtr.

    * Updated the new TableVal Assign()/Remove() methods to take either
      std::unique_ptr<HashKey> or HashKey& as appropriate for clarity of
      ownership expectations. (Jon Siwek, Corelight)

  * Add back in a deprecated TableVal ctor taking raw pointers (Jon Siwek, Corelight)

  * Deprecate TableVal::Attrs(), replace with GetAttrs() (Jon Siwek, Corelight)

  * Deprecate TableVal::Delete(), replace with Remove() (Jon Siwek, Corelight)

  * Switch Func::Call(val_list*) back to returning Val*

    And renamed the method returning IntrusivePtr to operator().
    This corrects the deprecation process for Func::Call(val_list*). (Jon Siwek, Corelight)

  * Deprecate TableVal::Lookup(), replace with Find()/FindOrDefault() (Jon Siwek, Corelight)

  * Deprecate TableVal IsSubsetOf and EqualTo taking Val*, use Val& (Jon Siwek, Corelight)

  * Deprecate ComputeHash(Val*) methods, replace with ComputeHash(Val&) (Jon Siwek, Corelight)

  * Deprecate TableVal::Intersect(), replace with Intersection() (Jon Siwek, Corelight)

  * Deprecate TableVal::Assign methods with Val*, add IntrusivePtr overloads (Jon Siwek, Corelight)

  * Deprecate RecordVal::Lookup(const char*, bool)

    Replace with GetField(const char*) and GetFieldOrDefault(const char*). (Jon Siwek, Corelight)

  * Deprecate RecordVal::LookupWithDefault() replace with GetFieldOrDefault()

    (The former was previously changed during this release cycle to return
    Intrusive pointer, but this just changes it back to return Val* and
    deprecates it). (Jon Siwek, Corelight)

  * Deprecate RecordVal::Lookup(int), replace with GetField(int) (Jon Siwek, Corelight)

  * Change BroValUnion to use IntrusivePtr for record field storage

    This also changes the AsRecord() and AsNonConstRecord() accessors
    to return std::vector<IntrusivePtr<Val>>* instead of val_list* (Jon Siwek, Corelight)

  * Switch ASN1 Val conversion functions to return IntrusivePtr (Jon Siwek, Corelight)

  * Deprecate RecordVal::Assign(int, Val*)

    And adapt all usages to the existing overload taking IntrusivePtr. (Jon Siwek, Corelight)

  * Switch RPC analyzers to use IntrusivePtr (Jon Siwek, Corelight)

  * Switch RecordVal::CoerceTo() to use IntrusivePtr (Jon Siwek, Corelight)

  * Deprecate TableEntryVal::Value(), replace with GetVal() (Jon Siwek, Corelight)

  * Switch OpaqueVal::SerializeType() to IntrusivePtr (Jon Siwek, Corelight)

  * Switch BlommFilterVal/CardinalityVal to use IntrusivePtr<BroType> (Jon Siwek, Corelight)

  * Switch some TopkVal methods to use IntrusivePtr (Jon Siwek, Corelight)

  * Switch TopkVal to store IntrusivePtr<BroType> (Jon Siwek, Corelight)

  * Switch OpaqueVal::UnserializeType() to return IntrusivePtr (Jon Siwek, Corelight)

  * Deprecate FuncType::ArgTypes(), replace with ParamList() (Jon Siwek, Corelight)

  * Deprecate FuncType::Args(), replace with Params() (Jon Siwek, Corelight)

  * Deprecate EventHandler::FType(), replace with GetType() (Jon Siwek, Corelight)

  * Deprecate BroFile::FType(), replace with GetType() (Jon Siwek, Corelight)

  * Deprecate Func::FType(), replace with Func::GetType() (Jon Siwek, Corelight)

  * Change {Get,Set}ContentsFile() to use IntrusivePtr (Jon Siwek, Corelight)

  * Deprecate BroFile::GetFile(), replace with BroFile::Get() (Jon Siwek, Corelight)

  * Deprecate Val(BroFile*) ctor, replace with one using IntrusivePtr (Jon Siwek, Corelight)

  * Deprecate Val(Func*) ctor, replace with one using IntrusivePtr (Jon Siwek, Corelight)

  * Deprecate OpaqueVal/HashVal ctors that take OpaqueType*

    Replaced with ones that take IntrusivePtr (Jon Siwek, Corelight)

  * fixup! Deprecate Scope::Lookup(), replace with Scope::Find() (Jon Siwek, Corelight)

  * Switch EnumType::GetVal() to return const-ref (Jon Siwek, Corelight)

  * Change protected EnumVal ctor to use IntrusivePtr (Jon Siwek, Corelight)

  * Deprecate RecordVal(RecordType*) ctor

    Replaced with one that takes IntrusivePtr (Jon Siwek, Corelight)

  * Switch zeek::id::lookup to zeek::id::find

    For parity with Scope since it now uses Find instead of Lookup (Jon Siwek, Corelight)

  * Use zeek::BifEvent:: for enqueue_ functions instead of BifEvent:: (Jon Siwek, Corelight)

  * Replace deprecated usage of BifFunc:: with zeek::BifFunc::

    Names of functions also changed slightly, like bro_fmt -> fmt_bif.

    Should generally be unusual/unexpected to see somone calling these
    directly from C++ in their plugin, but since technically possible in
    previous versions, I also removed the "private" restriction on accessing
    the BifReturnVal member. (Jon Siwek, Corelight)

  * Deprecate names in BifConst, replace with zeek::BifConst

    Some Val* types are also replaced with IntrusivePtr at the new location (Jon Siwek, Corelight)

  * Deprecate all BroType* in BifType:: namespace

    Replaced with equivalently named IntrusivePtr in zeek::BifType:: (Jon Siwek, Corelight)

  * Change EventRegistry/EventHandler methods to use std::string{_view} (Jon Siwek, Corelight)

  * Deprecate internal_handler(), replace with EventRegistry::Register()

    Added a couple explicit event declarations that were missing: "net_done"
    and "dns_mapping_name_changed". (Jon Siwek, Corelight)

  * Deprecate remaining "opt_internal" functions in Var.h (Jon Siwek, Corelight)

  * Remove signal_val declaration from Var.h (Jon Siwek, Corelight)

  * Change zeek::id::lookup functions to use std::string_view (Jon Siwek, Corelight)

  * Change Scope::Find() and Scope::Remove() to use std::string_view (Jon Siwek, Corelight)

  * Change lookup_ID() to return a const-reference (Jon Siwek, Corelight)

  * Deprecate Scope::Lookup(), replace with Scope::Find() (Jon Siwek, Corelight)

  * Move various elements into ID.h and zeek::id namespace

    * A handful of generic/useful/common global type pointers that used
      to be in NetVar.h

    * Lookup functions that used to be Var.h (Jon Siwek, Corelight)

  * Trim the list of "global type pointers" from NetVar.h further

    Most of them are deprecated now, with usage sites now doing the lookup
    themselves. (Jon Siwek, Corelight)

  * Deprecate global Val pointers in NetVar.h

    All of these have fairly niche uses, so better maintained as
    lookup/static closer to the usage site. (Jon Siwek, Corelight)

  * Deprecate global type pointers in NetVar.h

    There's analogous IntrusivePtrs in zeek::vars (Jon Siwek, Corelight)

  * Add RecordVal ctor that takes IntrusivePtr (Jon Siwek, Corelight)

  * Remove unused FlattenExpr (Jon Siwek, Corelight)

  * Add missing "vector_coerce" to expr_name() (Jon Siwek, Corelight)

  * Deprecate Expr::Type(), replace with GetType() (Jon Siwek, Corelight)

  * Deprecate Val::Type(), replace with GetType() (Jon Siwek, Corelight)

  * Change Val to store IntrusivePtr (Jon Siwek, Corelight)

  * Remove VectorVal::vector_type member (Jon Siwek, Corelight)

  * Deprecate VectorVal(VectorType*) ctora

    Adds a new one taking an IntrusivePtr. (Jon Siwek, Corelight)

  * Deprecate internal_list_val() (Jon Siwek, Corelight)

  * Deprecate opt_internal_val() (Jon Siwek, Corelight)

  * Deprecate internal_func(), replace with zeek::lookup_func() (Jon Siwek, Corelight)

  * Deprecate internal_val() and internal_const_val()

    Replaced with zeek::lookup_val() and zeek::lookup_const() (Jon Siwek, Corelight)

  * Deprecate internal_type(), replace with zeek::lookup_type() (Jon Siwek, Corelight)

  * Deprecate ID::ID_Val(), replace with ID::GetVal() (Jon Siwek, Corelight)

  * Use std::move in some zeekygen::Manager methods (Jon Siwek, Corelight)

  * Add Val TypeType constructor taking an IntrusivePtr (Jon Siwek, Corelight)

  * Deprecate ID::Type(), replace with GetType() (Jon Siwek, Corelight)

  * Deprecate ID::AsType(), add ID::IsType() and ID::GetType() (Jon Siwek, Corelight)

  * Deprecate BroType::YieldType(), replace with Yield() (Jon Siwek, Corelight)

  * Change base_type() to return const-ref, deprecate base_type_no_ref() (Jon Siwek, Corelight)

  * Deprecate BroType::GetField() and BroType::HasField() (Jon Siwek, Corelight)

  * Deprecate RecordType::FieldType(), replace with GetFieldType() (Jon Siwek, Corelight)

  * Migrate TypeList to store IntrusivePtrs

    This changes return types of TypeList::Types() and
    IndexType::IndexTypes() to return std::vector instead of type_list* (Jon Siwek, Corelight)

  * Deprecate TypeList::PureType(), replace with TypeList::GetPureType() (Jon Siwek, Corelight)

  * Deprecate SetType::SetElements(), replace with SetType::Elements() (Jon Siwek, Corelight)

  * Remove unused TableType::ExpandRecordIndex() (Jon Siwek, Corelight)

  * Add cast_intrusive() and make use of it in two spots (Jon Siwek, Corelight)

  * Give make_intrusive() access to protected EnumVal ctor (Jon Siwek, Corelight)

  * Remove two superfluous IntrusivePtr NewRefs in Type.cc (Jon Siwek, Corelight)

  * Change BroType::ShallowClone() to return IntrusivePtr (Jon Siwek, Corelight)

  * Migrate IP.cc to use IntrusivePtr (Jon Siwek, Corelight)

  * Migrate SMB analyzer to use IntrusivePtr

    Deprecates the utf16_bytestring_to_utf8_val() function with replacement
    being utf16_to_utf8_val(). (Jon Siwek, Corelight)

  * Migrate ARP analyzer to use IntrusivePtr (Jon Siwek, Corelight)

  * Migrate HTTP/MIME analyzers to use IntrusivePtr (Jon Siwek, Corelight)

  * Migrate ICMP analyzer to use IntrusivePtr (Jon Siwek, Corelight)

  * Migrate DNS analyzer to use IntrusivePtr (Jon Siwek, Corelight)

  * Migrate Tag classes to use IntrusivePtr

    Deprecates various methods that previously took raw pointers (Jon Siwek, Corelight)

  * Deprecate TableVal::ConvertToList() and TableVal::ConvertToPureList()

    Replaced with ToListVal() and ToPureListVal() that return IntrusivePtr (Jon Siwek, Corelight)

  * Switch ListVal to store IntrusivePtrs

    * Deprecates ListVal::Index() methods and replaces with ListVal::Idx()

    * Replaces ListVal::Vals() method with one that returns
      std::vector<IntrusivePtr<Val>> rather than val_list (Jon Siwek, Corelight)

  * Deprecant ListVal::Append(Val*) and add IntrusivePtr version (Jon Siwek, Corelight)

  * Deprecate ListVal::ConvertToSet(), add ListVal::ToSetVal() (Jon Siwek, Corelight)

  * Deprecate TunnelEncapsulation BuildRecordVal/BuildVectorVal methods

    Replaced with ToVal methods that return IntrusivePtr (Jon Siwek, Corelight)

  * Deprecate various IP/packet header Val-building methods

    And supply new alternatives that use IntrusivePtr (Jon Siwek, Corelight)

  * Update submodule(s)

     [nomail] (Jon Siwek, Corelight)


3.2.0-dev.573 | 2020-05-29 17:13:36 -0700

  * Add "Known::service_udp_requires_response" option (Jon Siwek, Corelight)

    Determines whether to require UDP server response before considering
    an active service to log in known_services.log.

  * Update known-services.zeek logic for "active" TCP services (Jon Siwek, Corelight)

    To better check for only endpoints that either have observed a TCP handshake
    or else assumed to have done one in the past (partial connections, missed
    the handshake, but now see data/acks from server)

  * Log services with unknown protocol names (Michael Dopheide)

  * GH-989: Fix crash on using some deprecated environment variables (Jon Siwek, Corelight)

3.2.0-dev.570 | 2020-05-28 14:49:10 -0700

  * Add negotiation flags parameter to some RDP events (Anthony Kasza, Corelight)

    Namely rdp_connect_request, rdp_negotiation_response, and rdp_negotiation_failure

  * Update test baselines for new Broker connection status/error strings (Jon Siwek, Corelight)

3.2.0-dev.565 | 2020-05-26 21:55:54 +0000

  * Add DCE-RPC constants from BZAR project (V)

3.2.0-dev.562 | 2020-05-26 11:06:34 -0700

  * GH-979: Update libkqueue to fix use on CentOS 6 (Jon Siwek, Corelight)

3.2.0-dev.561 | 2020-05-21 13:05:39 -0700

  * GH-983: Fix opaque Broker types lacking a Type after (de)serialization (Jon Siwek, Corelight)

3.2.0-dev.559 | 2020-05-21 13:04:19 -0700

  * Make SendEvent callable from all threads

    This commit refactors the SendEvent call and moves it from the Input
    ReaderBackend to to MsgThread. This allows all other types of threads
    to access this functionality.

    This necessitated a few more changes. Most importantly, one of the
    ValueToVal methods was moved over to SerialTypes. Whereit arguably
    belongs - there was nothing that was input-framework specific in
    that method - and the functionality could come in useful in a number
    of cases. (Johanna Amann, Corelight)

3.2.0-dev.557 | 2020-05-21 11:41:12 -0700

  * Speed up FuzzBuffer ChunkCount validity check (Justin Azoff, Corelight)

3.2.0-dev.555 | 2020-05-20 11:19:08 -0700

  * Disable output of Reporter messages to stderr in fuzz targets (Jon Siwek, Corelight)

3.2.0-dev.554 | 2020-05-20 10:56:46 -0700

  * Improve standalone fuzz driver timing output (Jon Siwek, Corelight)

  * Skip fuzz inputs that have more than 64 chunks (Justin Azoff, Corelight)

3.2.0-dev.550 | 2020-05-19 10:50:42 -0700

  * Upgrade to latest Broker changes for CAF 0.18 (Dominik Charousset, Corelight)

  * Include pcap.h instead of pcap/dlt.h in packet-fuzzer (Tim Wojtulewicz, Corelight)

3.2.0-dev.547 | 2020-05-18 10:47:54 -0700

  * add packet fuzzer (Justin Azoff)

  * Fix building fuzz targets on macOS (Jon Siwek, Corelight)

  * Highwayhash: small build fix (Johanna Amann, Corelight)

    Turns out that hh_neon should not be compiled on generic arm CPUs.
    That one is only for aarch64.

3.2.0-dev.539 | 2020-05-15 19:47:55 +0000

  * Replace bzero() with memset() (Noah Treuhaft)

  * Change Coverity action to output build log (Jon Siwek, Corelight)

3.2.0-dev.530 | 2020-05-13 15:05:31 -0700

  * GH-906: Fix the regex in url.zeek to better match for find_all_urls (Tim Wojtulewicz, Corelight)

3.2.0-dev.526 | 2020-05-13 13:49:29 -0700

  * Hash unification: address PR feedback (Johanna Amann, Corelight)

  * Update architecture checks for highwayhash

    Now we use cmake to determine which compiler flags are set to
    determine which implementation of highwayhash to compile. This should
    be much more robust.

    Also - fix missing include on old ubuntus. (Johanna Amann, Corelight)

  * Highwayhas: unbreak 32 bit compiles (Johanna Amann, Corelight)

  * Update external test suites for hash changes. (Johanna Amann, Corelight)

  * Hashing: Remove unnecessary include (Johanna Amann, Corelight)

  * Merge remote-tracking branch 'origin/master' into topic/johanna/hash-unification (Johanna Amann, Corelight)

  * Add highwayhash in a more cmake-y way (Johanna Amann, Corelight)

  * Remove highwayhash submodule.

    It is moving to 3rdparty... (Johanna Amann, Corelight)

  * Merge remote-tracking branch 'origin/master' into topic/johanna/hash-unification (Johanna Amann, Corelight)

  * Switch file UID hashing from md5 to highwayhash.

    This commit switches UID hashing from md5 to a highway hash. It also
    moves the salt value out of the file plugin - and makes it
    installation-specific instead - it is moved to the global namespace.

    There now are digest hash functions to make "static"
    installation-specific hashes that are stable over workers available to
    everyone; hashes can be 64, 128 or 256 bits in size.

    Due to the fact that we switch the file hashing algorithm, all file
    hashes change.

    The underlyigng algorithm that is used for hashing is highwayhash-128,
    which is significantly faster than md5. (Johanna Amann, Corelight)

  * Switch most internal md5 calls to digest calls.

    The places that used md5 basically already used it as a digest
    algorithm. Switching to a digest just means that the internal values
    used to not change between runs - which is actually wanted in these
    cases.

    This commit also removes our special cmake subdirectory. We don't expose
    highwayhash in headers anymore - so we can just treat it as an internal
    implementation choice that is not directly exposed to plugins. (Johanna Amann, Corelight)

  * Start refactoring hashing.

    This commit moves some of the hash datastructures and code from
    util.cc into Hash.cc - where it seems more appropriate.

    It also starts to make more Keyed hash functions available - still
    using siphash as the default 64 bit keyed hash, but also making
    128 and 256 bit highway hashes available.

    There already are a few other functions that are defined but not
    yet implemented - these will be "static" keyed hashes - which use
    an installation specific key. These will be used to, e.g., get
    rid of md5 hashing for the generation of file UIDs. (Johanna Amann, Corelight)

  * Introduce calculate_digest convenience function

    This function just calculates the chosen digest and returns the result
    in either the passed buffer, or in a static buffer. Basically a superset
    to the surprisingly popular internal_md5. (Johanna Amann, Corelight)

  * Remove the siphash->hmac-md5 switch after 36 bytes.

    Currently, siphash is used for strings up to 36 bytes. hmac-md5 is used
    for longer strings.

    This switch-over is a remnant of the previous hash-function that was
    used, which apparently was slower with longer input strings.

    This change serves no purpose anymore. I performed a few performance tests
    on strings of varying sizes:

    For a 40 byte string with 10 million iterations:

    siphash: 0.31 seconds
    hmac-md5: 3.8 seconds

    For a 1080 byte string with 10 million iterations:

    siphash: 4.2 seconds
    hmac-md5: 17 seconds

    For a 18360 byte string with 10 million iterations:

    siphash: 69 seconds
    hmac-md5: 240 seconds

    Hence, this commit removes the use of hmac-md5.

    This change causes reordering of lines in a few logs.

    This commit also changes the datastructure for the seed in probabilistic/Hasher
    to get rid of a type-punning warning. (Johanna Amann, Corelight)

  * Fix plugin compile problems with highwayhash

    The way this is included is still hacky as hell - but now all tests pass. (Johanna Amann, Corelight)

  * Replace siphash with Google implementation

    This adds the entirety of the highwayhash implementation of Google.
    This includes siphash as well as severl highwayhash variants - which
    are faster.

    This first commit only switches out the siphash implementation. All
    hashes that are generated are exactly the same as before. However, this
    does make all other hashes available to be used by us.

    I did some performance tests vs the previous siphash implementation by
    running the 2009-M57-day11-18 trace 100x through both cases. The average
    runtime was virtually the same (within 0.014 seconds of each other).

    Note that the way that I included the highwayhash implementation in our
    cmake setup is... well, let's say hacky. This definitely needs to be
    changed a bit before including this in a real build. (Johanna Amann, Corelight)


3.2.0-dev.508 | 2020-05-13 11:16:05 -0700

  * Disable broker tests from building on the CI (Tim Wojtulewicz, Corelight)

    This results in a ~10% reduction in build time.

  * Disable broker documentation examples if disabling broker tests (Tim Wojtulewicz, Corelight)

3.2.0-dev.505 | 2020-05-12 18:52:19 -0700

  * Use zeek::detail namespace for fuzzer utils (Jon Siwek, Corelight)

  * Set terminating flag during fuzzer cleanup (Jon Siwek, Corelight)

  * Add missing include to standalone fuzzer driver (Jon Siwek, Corelight)

  * Improve standalone fuzzer driver error messages (Jon Siwek, Corelight)

  * Merge branch 'master' into topic/jsiwek/fuzzing (Jon Siwek, Corelight)

  * Test fuzzers against seed corpus under CI ASan build (Jon Siwek, Corelight)

  * Update fuzzing README with OSS-Fuzz integration notes (Jon Siwek, Corelight)

  * Link fuzzers against shared library to reduce executable sizes (Jon Siwek, Corelight)

  * Improve FuzzBuffer chunking

    Now allocates a new buffer for each chunk to better detect over-reads (Jon Siwek, Corelight)

  * Fix compiler warning in standalone fuzzer driver (Jon Siwek, Corelight)

  * Adjust minor fuzzing documentation (Jon Siwek, Corelight)

  * Exit immediately after running unit tests (Jon Siwek, Corelight)

  * Add OSS-Fuzz Zeek script search path to fuzzers (Jon Siwek, Corelight)

  * Assume libFuzzer when LIB_FUZZING_ENGINE file doesn't exist

    i.e. environment variable may be set, but not point to existing file (Jon Siwek, Corelight)

  * Change handling of LIB_FUZZING_ENGINE

    Should better support OSS-Fuzz, which may set it to either
    be path to library to link against or linker flag like
    "-fsanitize=fuzzer" (Jon Siwek, Corelight)

  * Change --enable-fuzzing to --enable-fuzzers

    Since it controls whether to build the fuzzer targets, not whether those
    fuzzer targets actually use a fuzzing engine. (Jon Siwek, Corelight)

  * Add standalone driver for fuzz targets

    Useful for cases that don't need to use a fuzzing engine, but just run
    the fuzz targets over some set of inputs, like for regression/CI tests.

    Also added a POP3 fuzzer dictionary, seed corpus, and README with
    examples. (Jon Siwek, Corelight)

  * Add basic structure for fuzzing targets

    General changes:

    * Add -D/--deterministic command line option as
      convenience/alternative to -G/--load-seeds (i.e. no file needed, it just
      uses zero-initialized random seeds).  It also changes Broker data
      stores over to using deterministic timing rather than real time.

    * Add option to make Reporter abort on runtime scripting errors (Jon Siwek, Corelight)


3.2.0-dev.486 | 2020-05-11 11:11:51 -0700

  * Limit rate at which MMDB error/status messages are emitted

    If there's some bad state we can be in where MMDB lookup/open operations
    consistently fail, then the volume of associated reporter messages can
    get overwhelmingly large especially if a lookup operation is being done
    for each network connection.

    This adds a limit of an arbitrary 20 messages every 5 minutes, which
    should be enough information to understand the overall
    open/close/lookup-failure pattern. (Jon Siwek, Corelight)

3.2.0-dev.484 | 2020-05-08 11:50:54 -0700

  * Change timer_list in BroList to be an unordered list. (Tim Wojtulewicz, Corelight)

    This type is used by Conn and Analyzer to hold onto timers being added and
    removed. We don't expect the elements in those lists to maintain an order
    as the list is being modified.

  * Add ability for List to be ordered/unordered (Tim Wojtulewicz, Corelight)

    This fixes a "bug" with List where remove_nth() can be an O(n) operation
    when it doesn't need to be. remove_nth for lists that don't necessarily
    need to keep an order can be an O(1) operation instead.

  * Add unit testing for List (Tim Wojtulewicz, Corelight)

  * Fix bug with List where replace() doesn't work with non-pointer types (Tim Wojtulewicz, Corelight)

3.2.0-dev.478 | 2020-05-08 11:47:38 -0700

  * Added examples to set_to_regex comments (James Lagermann, Corelight)

  * Unbreak build on Fedora 32 (gcc 10.0.1) (Johanna Amann, Corelight)

    It requires cstdint in a few more headers.

3.2.0-dev.475 | 2020-05-07 17:15:23 -0700

  * GH-958: Fix crash when trying to redef non-existing enum (Johanna Amann, Corelight)

3.2.0-dev.473 | 2020-05-06 10:40:09 -0700

  * Revert addition of final modifier to JSON formatter (Tim Wojtulewicz, Corelight)

3.2.0-dev.471 | 2020-05-06 10:00:58 -0700

  * Fix global buffer over-read in POP3 analyzer (Justin Azoff, Corelight)

  * Fix SSL scripting error leading to access of unitialized field (Jon Siwek, Corelight)

    Reported by Justin Azoff

  * Remove outdated comment on set_to_regex. (Johanna Amann, Corelight)

    We can add patterns at runtime since 2.6.

3.2.0-dev.467 | 2020-05-04 18:00:35 -0700

  * GH-952: Correct spelling of DCE/RPC operation string NetrLogonSameLogonWithFlags
    (Jon Siwek, Corelight)

3.2.0-dev.466 | 2020-05-04 17:50:14 -0700

  * Add network_time_init() event. (Jan Grashoefer)

    This event is generated upon first initialization of network_time.

3.2.0-dev.461 | 2020-05-04 17:08:46 -0700

  * Avoid scheduling multiple inactivity timers (Justin Azoff and Jon Siwek, Corelight)

    Also updated language.expire_subnet btest which is unduly sensitive to
    timer-related changes

3.2.0-dev.459 | 2020-05-01 17:46:20 -0700

  * Extend CI config to cover building with libmaxminddb support (Jon Siwek, Corelight)

  * Ensure time continues moving forward if a pcap source is suspended (Tim Wojtulewicz, Corelight)

3.2.0-dev.455 | 2020-05-01 09:44:30 -0700

  * GH-938: fix IO loop iterations sometimes skipping offline pcap sources (Jon Siwek, Corelight)

3.2.0-dev.451 | 2020-04-29 16:28:34 -0700

  * Organized and added to the shipped file identification signatures. (Seth Hall, Corelight)

     - Added ISO 9660 disk image

     - Created new files for categorizing signatures better.
       - executable.sig - Executable (and bytecode) files.
       - java.sig - Java related files (class/jar, etc).
       - programming.sig - Mostly scripting language identification

3.2.0-dev.447 | 2020-04-29 15:55:03 -0700

  * GH-713: Fixed misc/stats.zeek skipping a log entry on termination (Brittany Donowho)

3.2.0-dev.445 | 2020-04-29 15:25:03 -0700

  * Add warning message for unknown Broker statuses (Jon Siwek, Corelight)

    There's now a couple placeholder/unimplemented status values in Broker
    related to upcoming routing features that we don't want to handle
    explicitly for compatibility reasons, but also don't want the compiler
    warning about unhandled values in the switch.

3.2.0-dev.443 | 2020-04-28 17:10:38 -0700

  * GH-941: Fix build when configured to use libmaxminddb (Jon Siwek, Corelight)

3.2.0-dev.441 | 2020-04-27 13:34:22 -0700

  * Fix a few more IntrusivePtr deprecation warnings (Tim Wojtulewicz, Corelight)

  * Fix cloning of TypeType values (Vern Paxson, Corelight)

3.2.0-dev.437 | 2020-04-27 19:30:24 +0000

  * GH-854: provide access to original HTTP/MIME header names

    The "http_header" event now has an "original_name" parameter that allows
    access to the original header name (the "name" parameter reamins the
    same as before: it's the uppercased header name).

    The "mime_header_rec" record type now also includes an "original_name"
    field to similarly provide access to original header name in the
    following events: "http_all_headers", "mime_one_header", and
    "mime_all_headers". (Jon Siwek, Corelight)

  * Remove error message from empty bloomfilter lookups

    If a bloomfilter doesn't have a type, that just means no
    bloomfilter_add() has been called yet, so seems undesirable to emit an
    error for a lookup against something that's known to be empty. (Jon Siwek, Corelight)

  * unused variables found via use-def analysis (plus an indentation micro-nit) (Vern Paxson, Corelight)

3.2.0-dev.431 | 2020-04-27 12:09:30 -0700

  * Update various BIFs to return IntrusivePtr (Jon Siwek, Corelight)

3.2.0-dev.428 | 2020-04-24 16:19:45 -0700

  * Deprecate returning Val* from BIFs (Jon Siwek, Corelight)

  * Deprecate binpac::string_to_val (Jon Siwek, Corelight)

  * Deprecate binpac::bytestring_to_val, replace with binpac::to_stringval (Jon Siwek, Corelight)

  * Update deprecated BifEvent::generate_* usages (Jon Siwek, Corelight)

  * Deprecate Connection::Event and Analyzer::Event methods

    And update usages to the "EnqueueEvent" methods. (Jon Siwek, Corelight)

  * Deprecate BuildConnVal() methods and update usages to ConnVal()

    The later being a new method that returns IntrusivePtr (Jon Siwek, Corelight)

  * Update all BIFs to return IntrusivePtr instead of Val* (Jon Siwek, Corelight)

  * Update deprecated ValManager::GetPort usages (Jon Siwek, Corelight)

  * Update deprecated ValManager::GetEmptyString usages (Jon Siwek, Corelight)

  * Update deprecated ValManager::GetCount usages (Jon Siwek, Corelight)

  * Update deprecated ValManager::GetInt usages (Jon Siwek, Corelight)

  * Update deprecated ValManager::GetBool usages (Jon Siwek, Corelight)

  * Update deprecated ValManager GetTrue/GetFalse usages (Jon Siwek, Corelight)

  * Deprecate all ValManager "Get" methods

    Alternate methods that return IntrusivePtr are available in similarly
    named methods that omit the "Get" prefix. (Jon Siwek, Corelight)

  * Change BIFs to return a wrapper object

    That allows returning either Val* or IntrusivePtr<T>.  The former could
    eventually be deprecated, but it's used extensively at the moment. (Jon Siwek, Corelight)


3.2.0-dev.412 | 2020-04-22 10:43:39 -0700

  * Fix buffer over-read in Ident analyzer (Max Kellermann)

3.2.0-dev.410 | 2020-04-21 10:25:29 -0700

  * Remove "connection_external" event (Johanna Amann, Corelight)

    This event was forgotten in our broccoli cleanup. It cannot be
    raised by anything anymore

3.2.0-dev.405 | 2020-04-16 15:38:46 -0700

  * Remove ineffective &default in netcontrol cluster event handler args (Jon Siwek, Corelight)

3.2.0-dev.404 | 2020-04-16 14:12:20 -0700

  * GH-895: Remove use of Variable-Length-Arrays (Jon Siwek, Corelight)

3.2.0-dev.402 | 2020-04-16 13:00:59 -0700

  * Add default function for Kerberos constant-lookup-tables (Jon Siwek, Corelight)

  * check for the existance of f?$conns in file_sniff event in policy/protocols/ssl/log-hostcerts-only.zeek
	  (SG)

  * Update README.md to fix the logo and one of the links (Tim Wojtulewicz, Corelight)

3.2.0-dev.396 | 2020-04-14 15:45:54 -0700

  * A few minor cleanups in Dict (Tim Wojtulewicz, Corelight)

3.2.0-dev.394 | 2020-04-14 15:29:00 -0700

  * Fix a confusing variable name shadowing (Jon Siwek, Corelight)

3.2.0-dev.392 | 2020-04-14 11:06:04 -0700

  * Fix Stack Overflow in POP3_Analyzer::ProcessRequest. (Johanna Amann, Corelight)

    The VLA can overflow given a large enough string. As a small fix, this
    commit gets rid of the VLA and assigns the password directly to the
    target string.

    This was reported by Matteo Rizzo (Google).

3.2.0-dev.391 | 2020-04-13 14:59:50 -0700

  * Add warning for ineffective &default arguments in handlers

    For event/hook handlers that had a previous declaration, any &default
    arguments are ineffective.  Only &default uses in the initial
    prototype's arguments have an effect (that includes if the handler
    is actually the site at which the declaration occurs). (Jon Siwek, Corelight)

  * Fix frame size allocation of alternate event/hook handlers

    When the only existing handler doesn't use the full argument list, still
    have to ensure the full frame is allocated because all arguments get set
    regardless. (Jon Siwek, Corelight)

  * Emit error for alternate event/hook prototype args with attributes

    Argument attributes are only allowed in the canonical prototype. (Jon Siwek, Corelight)

  * Improve alternate event/hook prototype matching

    This fixes it to again allow the old behavior of matching a handler
    against the canonical prototype as long as all argument types, but not
    necessarily names, match. (Jon Siwek, Corelight)

  * Allow alternate event/hook prototype declarations

    The alternates must be some subset of the canonical prototype (the one
    that's first declared) and allows users to define handlers for any
    such prototype.  Example:

        # Prototype declarations
        global my_event: event(s: string, c: count);
        global my_event: event(c: count);
        global my_event: event();

        # Handler definitions
        event my_event(s: string, c: count) { print s, c; }
        event my_event(c: count) { print c; }
        event my_event() { }

    This allows handlers to consume a subset of the arguments or even
    re-order them.  This makes it easier to either extend an existing
    event/hook's arguments and/or deprecate usages of certain prototypes. (Jon Siwek, Corelight)

  * Fix Broker topics used to uniquely identify cluster nodes

    Node-specific topic prefix subscriptions/publications now add a trailing
    slash like "zeek/cluster/node/<name>/".  Without the trailing slash,
    messages attempting to target "proxy-10" may also be sent to "proxy-1"
    since subscription matching is prefix-based. (Jon Siwek, Corelight)

  * Update submodule(s)

     [nomail] (Jon Siwek, Corelight)

  * Fix include in X509Common.h

    Give full path to Analyzer.h, so that X509Common.h is include-able by
    plugins. (Johanna Amann, Corelight)


3.2.0-dev.382 | 2020-04-09 13:17:03 -0700

  * Remove 'using namespace std' as well as other using statements from headers.

    This unfortunately cuases a ton of flow-down changes because a lot of other
    code was depending on that definition existing. This has a fairly large chance
    to break builds of external plugins, considering how many internal ones it broke. (Tim Wojtulewicz, Corelight)

3.2.0-dev.378 | 2020-04-09 08:47:44 -0700

  * Replace most of the uses of 0 or NULL to indicate null pointers with nullptr.
    This change does not change any calls to syscalls, in the interest of passing
    what the API for those calls says to pass. (Tim Wojtulewicz, Corelight)

3.2.0-dev.372 | 2020-04-08 14:00:28 -0700

  * Lazy-initalize some of the fields in Frame to reduce size (Tim Wojtulewicz, Corelight)

  * Set InternalHashTag to a uint16_t so CompositeHash doesn't have a gap in it.

    Resizes CompositeHash from 32 bytes to 24 bytes. (Tim Wojtulewicz, Corelight)

  * Mark constants in List constexpr so they don't actually take up space in created objects

    This resizes List from 24 bytes to 16 bytes. (Tim Wojtulewicz, Corelight)

  * Reorder some class variables to fill in gaps in structure packing (Tim Wojtulewicz, Corelight)

  * Mark classes as final, where appropriate. (Tim Wojtulewicz, Corelight)

    These classes were recommended by gcc's -Wsugggest-final-types as places where
    large numbers of virtual functions could be optimized away.

3.2.0-dev.360 | 2020-04-07 14:12:27 -0700

  * Add "udp_content_ports" option

    Any port in that set found as either source or destination port
    of a UDP packet will cause the "udp_contents" event to be raised. (Jon Siwek, Corelight)

  * Add new "udp_content_delivery_ports_use_resp" option

    This controls whether ports given by "udp_content_delivery_ports_orig" and
    "udp_content_delivery_ports_orig" are in terms of the UDP packet's
    destination port or by the Connection's "responder" port (the former is
    the unchanged default behavior). (Jon Siwek, Corelight)

3.2.0-dev.357 | 2020-04-07 13:43:38 -0700

  * Fix connection duration thresholding

    Connection durations were being compared "less than" instead of "greater
    than" the desired duration threshold as provided to
    set_current_conn_bytes_threshold() or
    ConnThreshold::set_duration_threshold() (Jon Siwek, Corelight)

3.2.0-dev.355 | 2020-04-07 12:05:17 -0700

  * Fix using patterns stored as table/set indices

    Pattern values used as table/set indices are stored in serialized form
    using just the pattern text, but re-creating the value from that didn't
    fully initialize/compile the regex matcher after (re-)creating it from
    that pattern text. (Jon Siwek, Corelight)

3.2.0-dev.353 | 2020-04-06 17:27:35 -0700

  * Improve parsing of RDPEUDP/RDPEUDP2 (Anthony Kasza, Corelight)

3.2.0-dev.338 | 2020-04-02 18:29:23 -0700

  * Add RDP over UDP analyzer (Anthony Kasza, Corelight)

3.2.0-dev.336 | 2020-04-02 15:03:04 -0700

  * Fix uses of GetBool in bifs to use GetTrue/GetFalse (Tim Wojtulewicz, Corelight)

  * Change Sessions::dump_this_packet to a bool (Tim Wojtulewicz, Corelight)

3.2.0-dev.332 | 2020-03-31 15:34:52 -0700

  * GH-725: fix logic for finding a lambda's usage of outer IDs (Jon Siwek, Corelight)

  * Change record field anonymous functions to use lambda expressions

    There was an alternate syntax to assign anonymous functions to record
    fields that was never migrated to use the new lambda expression
    machinery (and so didn't allow referencing variables in outer scope):

        type myrec: record {
            foo: function(a: string);
        };

        local o = "o";
        local mr = myrec($foo(a: string) = { print a + o; });

    (Jon Siwek, Corelight)

3.2.0-dev.329 | 2020-03-31 08:48:10 -0700

  * Mark input/output message classes as final, since nothing should be inheriting from them (Tim Wojtulewicz, Corelight)

  * Add missing override specifier to a number of methods, remove virtual from some as well (Tim Wojtulewicz, Corelight)

  * Add override specifier to Configure() method in almost all of the internal plugins (Tim Wojtulewicz, Corelight)

3.2.0-dev.325 | 2020-03-31 07:04:59 +0000

  * Fix bug in intel framework letting deletes of mixed-cased entries
    fail. (Antonio Nappa, Corelight)

3.2.0-dev.322 | 2020-03-31 06:44:57 +0000

  * A large number of functions had return values and/or arguments
    changed to use ``bool`` types instead of ``int``. (Tim
    Wojtulewicz, Corelight)

3.2.0-dev.320 | 2020-03-30 14:09:23 -0700

  * GH-887: improve GRE/ERSPAN parsing of non-IPv4/IPv6 inner payload

    This changes the decapsulation logic for GRE/ERSPAN payloads to re-use
    existing Layer 2 parsing logic that already handles things like 802.1Q
    tags correctly before going on to process the inner IPv4/IPv6 payload. (Jon Siwek, Corelight)

  * Fix ERSPAN III optional sub-header parsing (Jon Siwek, Corelight)

  * Many small cleanups of Expr.cc (Tim Wojtulewicz, Corelight)

  * Remove unused cq queue code (Tim Wojtulewicz, Corelight)

  * Add a "reserved" field to tcp_hdr record

    This record is available by handling either raw_packet or new_packet
    events. (Jon Siwek, Corelight)

  * GH-878: Make RocksDB usage opt-in and add --enable-rocksdb configure flag (Jon Siwek, Corelight)

3.2.0-dev.300 | 2020-03-27 11:48:46 -0700

  * Func::DescribeDebug: move a NumFields() call out of loop (Jon Siwek, Corelight)

  * Use const-ref parameter for zeek::val_list_to_args()

    It ended up being used a bit more than initially expected and this
    is closer to the style we're generally aiming for. (Jon Siwek, Corelight)

  * Fix missing IntrusivePtr.h include and ambiguous ODesc::Add call (Jon Siwek, Corelight)

  * Remove TimerMgr arg from event queuing/scheduling methods

    It's not useful for anything since there's only ever a single TimerMgr. (Jon Siwek, Corelight)

  * Deprecate Analyzer::ConnectionEvent()

    And update usages to Analyzer::EnqueueConnEvent() (Jon Siwek, Corelight)

  * Deprecate file_analysis::File::FileEvent methods using val_list args

    And update usages to the overload that takes a zeek::Args instead. (Jon Siwek, Corelight)

  * Deprecate Connection::ConnectionEvent methods

    And update usages to Connection::EnqueueEvent (Jon Siwek, Corelight)

  * Deprecate EventMgr::QueueEventFast() and update usages to Enqueue() (Jon Siwek, Corelight)

  * Deprecate EventMgr::QueueEvent() and update usages to Enqueue() (Jon Siwek, Corelight)

  * Deprecate Func::Call(val_list*, ...)

    The version taking a vector of intrusive pointers should be used
    instead.  A variadic version is also added that forwards all arguments. (Jon Siwek, Corelight)

  * Use vector<IntrusivePtr<Val>> for Func::Call and Event queuing args

    This change may break BIFs that use @ARGS@, @ARG@, or @ARGC@ since their
    types have changed. (Jon Siwek, Corelight)

  * Fix memory leak in Zeek when-statement bodies with runtime errors (Jon Siwek, Corelight)

  * Change TableVal::RecoverIndex() to return IntrusivePtr (Jon Siwek, Corelight)

  * Use IntrusivePtr in TableVal::CallExpireFunc (Jon Siwek, Corelight)

  * Fix memory leak when runtime error occurs in a Zeek for-loop (Jon Siwek, Corelight)

  * Enable leak checks for btests that produce runtime exceptions

    These were previously reporting leaks due to various allocations not
    getting cleaned up during the stack unwind, but at the current state of
    the transition toward IntrusivePtr usage, theses tests no longer leak. (Jon Siwek, Corelight)

3.2.0-dev.280 | 2020-03-24 13:49:43 -0700

  * Add test case for binpac flowbuffer frame length parsing bug (Jon Siwek, Corelight)

3.2.0-dev.277 | 2020-03-20 11:29:36 -0700

  * Initialize RecordVal default fields when redef'd (Jon Siwek, Corelight)

    If a RecordVal had been created, but later its RecordType redef'd to
    contain fields with &default, those fields were incorrectly left
    uninitialized.

  * GH-857: fix redefining record types used to index tables (Jon Siwek, Corelight)

    This change tracks all TableVals created at parse-time whose index
    depends on a given RecordType.  Should that RecordType be redef'd, those
    TableVals are immediately rebuilt such that they are valid to
    subsequently use in either parse-time initializations or eventually in
    any arbitrary run-time expression.

  * Change RecordVals to get resized at time of RecordType redef (Jon Siwek, Corelight)

    Opposed to unconditionally checking all RecordVals whether they need to
    be resized after parsing ends.

3.2.0-dev.273 | 2020-03-20 10:05:21 -0700

  * GH-865: fix parsing of SMB NegotiateContextList

    * The compression capability was incorrectly set to 0x0004 instead of 0x0003

    * The padding was 4-byte instead of 8-byte aligned and also the spec.
      does not strictly require the padding for the last item in the list.

    * Add a default case to handle parsing of unknown context types. (Jon Siwek, Corelight)

3.2.0-dev.271 | 2020-03-17 22:52:40 -0700

  * Scope: store IntrusivePtr in `local` (Max Kellermann)

  * Scope: pass IntrusivePtr to AddInit() (Max Kellermann)

  * DNS_Mgr: use class IntrusivePtr (Max Kellermann)

  * Scope: use class IntrusivePtr (Max Kellermann)

  * Attr: use class IntrusivePtr (Max Kellermann)

  * Expr: check_and_promote_expr() returns IntrusivePtr (Max Kellermann)

    Instead of returning a pseudo-boolean integer, it now returns a
    referenced object or nullptr on error.  The old API was very error
    prone because of its obscure reference counting semantics.

  * Frame: use class IntrusivePtr (Max Kellermann)

  * Val: RecordVal::LookupWithDefault() returns IntrusivePtr (Max Kellermann)

  * Type: RecordType::FieldDefault() returns IntrusivePtr (Max Kellermann)

    Fixes memory leak in EventHandler::NewEvent().

  * Val: TableVal::Delete() returns IntrusivePtr (Max Kellermann)

  * Type: base_type() returns IntrusivePtr (Max Kellermann)

  * Type: init_type() returns IntrusivePtr (Max Kellermann)

  * Type: merge_types() returns IntrusivePtr (Max Kellermann)

  * Type: use class IntrusivePtr in VectorType (Max Kellermann)

  * Type: use class IntrusivePtr in EnumType (Max Kellermann)

  * Type: use class IntrusivePtr in FileType (Max Kellermann)

  * Type: use class IntrusivePtr in TypeDecl (Max Kellermann)

  * Type: make TypeDecl `final` and the dtor non-`virtual` (Max Kellermann)

  * Type: use class IntrusivePtr in TypeType (Max Kellermann)

  * Type: use class IntrusivePtr in FuncType (Max Kellermann)

  * Type: use class IntrusivePtr in TypeList (Max Kellermann)

  * Type: use class IntrusivePtr in IndexType (Max Kellermann)

  * Val: use class IntrusivePtr in class TableVal (Max Kellermann)

  * Val: use class IntrusivePtr in class TableEntryVal (Max Kellermann)

  * CompHash: return IntrusivePtr (Max Kellermann)

  * Type: return IntrusivePtr (Max Kellermann)

  * Val: add TableVal::Assign() overload with IntrusivePtr (Max Kellermann)

  * zeekygen/IdentifierInfo: use class IntrusivePtr more (Max Kellermann)

  * Func: use class IntrusivePtr (Max Kellermann)

  * GH-845: reference `id_list` before passing to new `BroFunc` (Max Kellermann)

    Passing the `id_list` pointer to `BroFunc` transfers ownership of the
    contained `ID` instances, because `~BroFunc()` unreferences them.
    Therefore, we need to increase the reference counters for each
    `BroFunc` instance to fix the use-after-free bug.

  * input/Manager: fix three use-after-free bugs (Max Kellermann)

3.2.0-dev.237 | 2020-03-17 16:54:35 -0700

  * Cleanup in iosource/Packet (Tim Wojtulewicz, Corelight)

  * Add ability for packet sources to flag a packet's l2 or l3 checksum as valid.

    This lets packet source plugins implement handling of hardware checksum
    offloading, if available. Setting the flags will skip the internal checksumming
    for either layer 2 and/or layer 3. (Tim Wojtulewicz, Corelight)

3.2.0-dev.234 | 2020-03-16 12:37:17 -0700

  * Remove an unused smb2_negotiate_response event handler. (M.Shirk)

3.2.0-dev.231 | 2020-03-13 13:41:23 -0700

  * Add branch whitelist for CI notifications (Jon Siwek, Corelight)

3.2.0-dev.230 | 2020-03-12 13:33:09 -0700

  * Introduce X509 certificate caching.

    This change introduces caching of X509 certificates. If a certificate is
    encountered more than 10 times in a minute, it will be cached for the future
    (until it is not seen for more than a minute). The caching means that the
    base scripts cache the certificate information and use cached information
    to raise events.

    This means certificates do not have to be repeatedly parsed; furthermore
    the events are passed "recycled" Vals (so data is not repeatedly copied around).

    This new feature is highly configureable - see the x509 main.zeek for full
    information. You can both change the timeouts, disable the feature, or
    even stop X509 events from being raised (by hooking X509::x509_certificate_cache_replay).

    (Johanna Amann, Corelight)

3.2.0-dev.223 | 2020-03-12 12:03:48 -0700

  * Update pointer to external test repo (Jon Siwek, Corelight)

3.2.0-dev.222 | 2020-03-11 16:39:53 -0700

  * Fix whitespace in connection record comments (Jon Siwek, Corelight)

3.2.0-dev.221 | 2020-03-11 11:21:20 -0700

  * Made additional MySQL fixes. (Vlad Grigorescu)

    1) There are a couple more places where the new protocol uses and OK
    packet instead of the deprecated EOF.

    2) With > 255 results, we could end up in an situation where the uint8
    sequence number would wrap, and we'd naively think it'd be a new
    handshake.

    Now, we track the previous sequence number, and assume overflow if it
    was 255 previously and 0 now.

    We also reset the previous sequence number to 0 in various packets
    that we'd expect at the end of other commands.

  * Add support to MySQL for deprecation of EOF packets. (Vlad Grigorescu)

    From the docs: "As of MySQL 5.7.5, OK packes are also used to indicate
    EOF, and EOF packets are deprecated."

    The client sets a capability flag (CLIENT_DEPRECATE_EOF) to indicate
    that it expects an OK instead of an EOF after the resultset rows.

  * MySQL analyzer whitespace cleanup (Vlad Grigorescu)

  * Fix EOF detection in the MySQL protocol analyzer. (Vlad Grigorescu)

    The MySQL documentation
    (https://dev.mysql.com/doc/dev/mysql-server/latest/page_protocol_basic_eof_packet.html)
    warns us that "You must check whether the packet length is less than 9 to
    make sure that it is a EOF_Packet packet."

    While we were doing this in two places, we were comparing the total
    packet length, which includes the 4-byte header. Changed to compare to
    13 instead.

3.2.0-dev.214 | 2020-03-09 13:35:26 -0700

  * Stop running GitHub Actions in forked repos (Jon Siwek, Corelight)

  * Remove Travis CI configuration (Jon Siwek, Corelight)

  * Add GitHub Action for Coverity Scan (Jon Siwek, Corelight)

3.2.0-dev.209 | 2020-03-06 16:27:29 -0800

  * Add urgent flag to tcp_flags documentation (Tim Wojtulewicz, Corelight)

3.2.0-dev.205 | 2020-03-06 07:59:08 -0800

  * Move libkqueue submodule to the aux directory (Jon Siwek, Corelight)

  * Move rapidjson submodule to aux directory

    This also now installs the full rapidjson include/ tree in order to
    allow external plugins to properly build if they include
    threading/formatters/JSON.h (Jon Siwek, Corelight)

  * GH-831: Fix the "bro" symlink for binary packaging mode. (Jon Siwek, Corelight)

3.2.0-dev.198 | 2020-03-05 14:18:10 -0800

  * Fix a btest to order zeek_init() events via &priority (Jon Siwek, Corelight)

3.2.0-dev.192 | 2020-02-28 11:36:12 -0800

  * threading::Queue::WakeUp(): lock mutex before notifying condition_variable (Max Kellermann)

    Not locking the associated mutex can create race conditions and lockups.

3.2.0-dev.190 | 2020-02-28 00:42:17 -0800

  * Stmt: use class IntrusivePtr (Max Kellermann)

  * Stmt: remove unused default constructors and `friend` declarations (Max Kellermann)

  * Val: remove unimplemented prototype recover_val() (Max Kellermann)

  * Val: cast_value_to_type() returns IntrusivePtr (Max Kellermann)

  * Val: use IntrusivePtr in check_and_promote() (Max Kellermann)

  * Val: use nullptr instead of 0 (Max Kellermann)

  * zeekygen: use class IntrusivePtr (Max Kellermann)

  * ID: use class IntrusivePtr (Max Kellermann)

  * Expr: use class IntrusivePtr (Max Kellermann)

  * Var: copy Location to stack, to fix use-after-free crash bug (Max Kellermann)

  * Scope: lookup_ID() and install_ID() return IntrusivePtr<ID> (Max Kellermann)

  * Scope: delete duplicate locals (Max Kellermann)

  * EventRegistry: automatically delete EventHandlers (Max Kellermann)

  * main: destroy event_registry after iosource_mgr (Max Kellermann)

    Fixes use-after-free bugs because PcapSource::Close() queues an event.

  * zeekygen/IdentifierInfo: delete duplicate fields (Max Kellermann)

  * main: free the global scope in terminate_bro() (Max Kellermann)

    Make valgrind a bit happier.

  * Scope: pop_scope() returns IntrusivePtr<> (Max Kellermann)

  * Scope: unref all inits in destructor (Max Kellermann)

  * Var: pass IntrusivePtr to add_global(), add_local() etc. (Max Kellermann)

  * plugin/ComponentManager: hold a reference to the EnumType (Max Kellermann)

    Use class IntrusivePtr<>.

  * Type: fix use-after-free bug in VectorType::ShallowClone() (Max Kellermann)

    The new VectorType instance needs a new reference to the `yield_type`.

  * Var: fix crash when redeclaring a function with different args (Max Kellermann)

3.2.0-dev.167 | 2020-02-27 14:24:55 -0800

  * Expr: use fmt instead of sprintf (Tim Wojtulewicz, Corelight)

  * Expr: other minor initialization cleanup (Tim Wojtulewicz, Corelight)

  * Expr: use List::empty() (Tim Wojtulewicz, Corelight)

  * Expr: Convert a bunch of methods returning ints to return bools (Tim Wojtulewicz, Corelight)

  * IPAddr: minor cleanup (Tim Wojtulewicz, Corelight)

    - Mark empty constructors/destructors as default
    - Initialization cleanup
    - Remove unnecessary elses from before returns

  * PriorityQueue: initialization cleanup (Tim Wojtulewicz, Corelight)

  * IP: Cleanup initialization, make a few functions consistent (Tim Wojtulewicz, Corelight)

3.2.0-dev.159 | 2020-02-26 19:51:24 -0800

  * Pop global frame stack on exception. (Johanna Amann, Corelight)

    When changing this code for 3.0.0, the code-flow changed significantly.
    One of the changes lead to exception being re-thrown from Call - which
    leads to the function cleanup not always being run. While the Unref of
    the framepointer was (correctly) moved into the catch block, the global
    frame stack was not cleaned up - and contains an invalid pointer after
    encountering an interpreter exception.

    This small commit pops the stack and removes the framepointer that was
    pushed onto it before re-throwing the exception.

    I also think that this change (re-throwing the exception) breaks the
    Zeek debugger in significant ways (you cannot debug a script that causes
    a Runtime error anymore).

3.2.0-dev.157 | 2020-02-26 10:04:32 -0800

  * IntrusivePtr: overload std::swap() (Max Kellermann)

  * IntrusivePtr: eliminate setPtr() (Max Kellermann)

    There are only two call sites, and those hard-code the `add_ref`
    parameter.

  * IntrusivePtr: optimize release() using std::exchange() (Max Kellermann)

  * Expr: remove unused default constructors and `friend` declarations (Max Kellermann)

  * Remove useless override: RecordAssignExpr::Eval() (Max Kellermann)

  * Type: don't pass reference to pointer to MatchesIndex() (Max Kellermann)

    No implementation modifies the pointer value.  To guard against this,
    this commit changes `&` to `const`.

  * Expr: don't pass reference to pointer to check_and_promote*() (Max Kellermann)

    The function never modifies the pointer value.

    check_and_promote_expr() is left untouched because it really does
    modify the pointer.

  * ID: remove unused default constructor (Max Kellermann)

  * Scope: simplify PList access, use pop_back() and back() (Max Kellermann)

  * List: add method empty() (Max Kellermann)

3.2.0-dev.146 | 2020-02-26 09:40:18 -0800

  * Type: fix double free bug in SetType::ShallowClone() (Max Kellermann)

3.2.0-dev.144 | 2020-02-25 19:52:57 -0800

  * Add a faster siphash24 implementation than the reference one (Tim Wojtulewicz, Corelight)

    - Average of 10 runs of 2009-M57-day11-18.trace (release build at -O3):
      - Master: 6.027s  93650 bytes max RSS
      - Commit: 5.950s  93271 bytes max RSS

    - Hashing a fixed 32-byte payload 10 million times with a fixed key:
      - Master: 1.397411s
      - Commit: 0.998211s

3.2.0-dev.142 | 2020-02-25 19:27:28 -0800

  * Updates to Broker to build CAF as sub-project (Dominik Charousset, Corelight)

    The embedded CAF now builds directly as a CMake subdirectory rather
    than an external project.

3.2.0-dev.137 | 2020-02-24 20:41:43 -0800

  * parse.y: fix memory leak in FieldAssignExpr call (Max Kellermann)

  * parse.y: fix use-after-free bug in open-ended index_slice (Max Kellermann)

  * Type: fix use-after-free bug in init_type() (Max Kellermann)

  * Expr: fix potential memory leak in RecordCoerceExpr::Fold() (Max Kellermann)

  * Expr: fix memory leak in RecordCoerceExpr::InitVal() (Max Kellermann)

  * zeekygen/IdentifierInfo: fix memory leak in operator=() (Max Kellermann)

  * Func: fix memory leaks in get_func_priority() (Max Kellermann)

  * parse.y: fix several memory leaks after lookup_ID() (Max Kellermann)

  * Func: fix memory leaks in check_built_in_call() (Max Kellermann)

  * Var: fix memory leaks in add_global() and add_local() (Max Kellermann)

  * Var: add missing references to `init` in add{,_and_assign}_local() (Max Kellermann)

  * parse.y: hold reference on init_expr for zeekygen::Manager::Redef() (Max Kellermann)

  * Expr: fix two memory leaks in AssignExpr::InitVal() (Max Kellermann)

  * parse.y: fix memory leak after "&deprecated" without string (Max Kellermann)

  * RuleMatcher: delete PatternSet instances in destructor (Max Kellermann)

  * Fix reference counting in Option::set_change_handler() (Max Kellermann)

3.2.0-dev.120 | 2020-02-24 18:13:04 -0800

  * Update zeek-testing commit (Jon Siwek, Corelight)

3.2.0-dev.119 | 2020-02-24 18:02:10 -0800

  * Use IntrusivePtr in TableVal::ExpandAndInit() to clarify ownership (Max Kellermann)

3.2.0-dev.117 | 2020-02-24 18:53:03 -0700

  * Switch to using [[deprecated]] attribute (Jon Siwek, Corelight)

3.2.0-dev.115 | 2020-02-24 18:48:50 -0700

  * Improve formatting of doubles that are close to integers

    Now checks for approximate floating point equality so that more doubles
    get properly disambiguated from integers (Jon Siwek, Corelight)

  * Improve HTTP version number comparisons

    Previous use of floating point comparisons was not always stable. (Jon Siwek, Corelight)

  * Add a 32-bit task to Cirrus CI config (Jon Siwek, Corelight)

  * Replace va_list fmt() overload with vfmt()

    Using an overload that takes a va_list argument potentially causes
    accidental misuse on platforms (e.g. 32-bit) where va_list is
    implemented as a type that may collide with commonly-used argument
    types.

    For example:

        char* c = copy_string("hi");
        fmt("%s", (const char*)c);
        fmt("%s", c);

    The first fmt() call correctly goes through fmt(const char*, ...) first,
    but the second mistakenly goes through fmt(const char*, va_list) first
    because variadic function overloads have lower priority during overload
    resolution and va_list on a 32-bit system happens to be defined as a
    pointer type that can match with "char*" but not "const char*". (Jon Siwek, Corelight)

  * Format tables indexed by patterns consistently across 32-bit/64-bit

    Uses a full 64 bit integer for length values regardless of actual size_t
    to get consistent results between either 32-bit and 64-bit platforms. (Jon Siwek, Corelight)

  * Format interval values consistently across 32-bit/64-bit platforms (Jon Siwek, Corelight)

3.2.0-dev.108 | 2020-02-24 17:24:07 -0800

  * Change OpaqueVal/HashVal APIs to use IntrusivePtr (Max Kellermann)

3.2.0-dev.106 | 2020-02-24 17:07:10 -0800

  * ScannedFile: Identify already scanned files by device and inode (Arne Welzel)

    Zeek scripts located on separate filesystems, but sharing the same inode
    number leads to scripts not being loaded. The reason is that a `ScannedFile`
    is only identified by `st_ino` which is not enough to uniquely identify a
    file in a system.

  * GH-808: Add ZEEK_VERSION_NUMBER definition to zeek-config.h (Jon Siwek, Corelight)

    This is the result of (major * 10000 + minor * 100 + patch), for example
    3.1.2 becomes 30102.  This definition may be helpful for external code
    that requires conditional compilation to support multiple Zeek
    versions with differing APIs.

  * Fix CI to checkout right commit of zeek-testing-private (Jon Siwek, Corelight)

3.2.0-dev.99 | 2020-02-21 21:23:52 -0800

  * Update a URL in CI README (Jon Siwek, Corelight)

3.2.0-dev.98 | 2020-02-21 20:04:30 -0800

  * Val: use C++ initializers (Max Kellermann)

  * Val: add BroValUnion constructors (Max Kellermann)

  * Val: reduce duplicate code by using delegating constructors (Max Kellermann)

  * Val: remove unused default constructors and `friend` declarations (Max Kellermann)

  * Val: remove the unnecessary BroValUnion typedef (Max Kellermann)

  * Type: remove unnecessary enum typedefs (Max Kellermann)

  * Type: use C++ initializers (Max Kellermann)

  * Type: move code from BroType::BroType() to constexpr functions (Max Kellermann)

    Prepare to inline the constructor, which will one day be `constexpr`
    (requires moving the `std::string name` field somewhere else).

  * Type: remove useless BroType destructor (Max Kellermann)

  * Obj: disallow copying BroObj (Max Kellermann)

    Copying a BroObj is dangerous, and should only be done with dedicated
    (virtual) methods which are implemented by all derived classes.  This
    commit avoids unintentional copies.

  * Obj: use C++ initializers (Max Kellermann)

  * Obj: make `no_location` constexpr (Max Kellermann)

    This ensures that the variable is initialized at compile time and may
    allow the compiler to apply more initializations.

3.2.0-dev.85 | 2020-02-21 15:29:45 -0800

  * threading/MsgThread: add [[noreturn]] to InternalError() (Max Kellermann)

  * Flare: add [[noreturn]] to bad_pipe_op() (Max Kellermann)

  * Obj: add [[noreturn]] attributes to Internal(),  bad_ref() (Max Kellermann)

  * Reporter: add [[noreturn]] attribute to several methods (Max Kellermann)

3.2.0-dev.80 | 2020-02-21 10:14:05 -0800

  * OpaqueVal: remove misplaced `virtual` keywords (Max Kellermann)

    These methods are not meant to be overridden.

  * CompHash: use class IntrusivePtr for the `type` field (Max Kellermann)

  * IntrusivePtr: replace the "add_ref" parameter with tag structs (Max Kellermann)

    Using a runtime parameter is obscure and error-prone.  Avoiding
    error-prone code and getting reference counting right is the whole
    point of this class.

  * IntrusivePtr: remove reset(), nobody uses it (Max Kellermann)

    This method mimicks std::unique_ptr::reset(), but adds an obscure
    "add_ref" parameter which is error prone.  Since nobody uses this
    method, and this method is all about dealing with raw pointers which
    we shouldn't be doing, let's remove it.

  * IntrusivePtr: remove ordering operators (Max Kellermann)

    These violate the C++ standard because comparing pointers to unrelated
    objects is undefined behavior.

  * IntrusivePtr: rename detach() to release() (Max Kellermann)

    Follow the C++ standard library conventions (here: `std::unique_ptr`).

  * IntrusivePtr: move nullptr initializer to field declaration (Max Kellermann)

    This allows "defaulting" the default constructor, and guarantees that
    all constructors really initialize the field to a legal value.

3.2.0-dev.71 | 2020-02-20 14:57:58 -0800

  * Fix missing reference count incrment in AssignExpr::InitVal() (Max Kellermann)

3.2.0-dev.69 | 2020-02-19 18:40:58 -0800

  * Fix various reference counting issues in Assign() implementations/callers (Jon Siwek, Corelight)

  * Fix memory leak in Input stream creation when using &type_column (Max Kellermann)

  * Fix two memory leaks in BloomFilterVal::Merge() (Max Kellermann)

  * Fix memory leaks in script debugger (Max Kellermann)

  * scan.l: fix missing Unref() in do_atif() (Max Kellermann)

  * Fix parse-time memory leak in RecordConstructorExpr::InitVal() (Max Kellermann)

  * Fix memory leaks in AssignExpr::EvalIntoAggregate() error conditions (Max Kellermann)

  * Fix memory leaks in CondExpr::Eval() error conditions (Max Kellermann)

    Also fixes reference counting issue for vector-based conditionals.

  * Fix memory leaks in BoolExpr::Eval() error conditions (Max Kellermann)

  * Fix various memory leaks in Assign() error conditions (Max Kellermann)

  * Fix memory leaks in BinaryExpr::Eval() error conditions (Max Kellermann)

  * Fix potential future memory leak in HTTP analyzer (Max Kellermann)

    This isn't really a memory leak because ParseRequest() never fails,
    but if it one day "learns" to fail, the `request_method` allocation
    will leak.

3.2.0-dev.53 | 2020-02-18 12:12:28 -0800

  * Make DNS NSEC3 parsing more resilient to introducing a memory leak
    when no dns_NSEC3 handler exists (Max Kellermann)

  * analyzer/protocol/dns: change runtime check to assert() (Max Kellermann)

    If it were legal to call SendReplyOrRejectEvent() without an
    EventHandlerPtr, then this would leak the `question_name` object.  But
    this method has just one caller, and it verifies the EventHandlerPtr.

  * Fix memory leak when no dns_TSIG_addl event handler exists (Max Kellermann)

3.2.0-dev.46 | 2020-02-14 22:02:50 -0800

  * Fix code format of various reporter btests (Jon Siwek, Corelight)

3.2.0-dev.45 | 2020-02-14 21:14:49 -0800

  * Update libkqueue for a linux bugfix (Jon Siwek, Corelight)

3.2.0-dev.44 | 2020-02-14 16:16:54 -0700

  * RE: make the RE_Matcher destructor non-virtual (Max Kellermann)

  * analyzer/protocol/smtp: remove unnecessary nullptr check (Max Kellermann)

  * analyzer/Manager: remove unnecessary clear() calls from destructor (Max Kellermann)

  * analyzer/protocol/http: remove unnecessary empty destructor (Max Kellermann)

  * iosource/pcap/Source: remove unused fields `last_hdr`, `last_data` (Max Kellermann)

  * iosource/Manager: eliminate two std::string copies (Max Kellermann)


3.2.0-dev.37 | 2020-02-14 11:09:50 -0800

  * Fix various format specifiers (Jon Siwek, Corelight)

3.2.0-dev.36 | 2020-02-14 10:29:36 -0700

  * broker: include cleanup (Max Kellermann)

  * file_analysis: include cleanup (Max Kellermann)

  * file_analysis/Analyzer: eliminate duplicate constructor (Max Kellermann)

  * probabilistic/Topk: include cleanup (Max Kellermann)

  * digest: eliminate the "Reporter.h" include (Max Kellermann)

  * Val: eliminate the "RE.h" include (Max Kellermann)

  * Val: eliminate the "BroString.h" include (Max Kellermann)

  * Val: eliminate the "CompHash.h" include (Max Kellermann)

  * Val: forward-declare class PDict, reduce includes (Max Kellermann)

  * Val: eliminate the "Scope.h" include (Max Kellermann)


3.2.0-dev.25 | 2020-02-13 19:05:56 -0800

  * Reset the number of entries in a dict when calling Clear() (Tim Wojtulewicz, Corelight)

  * Code cleanup in Dict.h (Tim Wojtulewicz, Corelight)

  * Add unit testing for the public Dictionary API (Tim Wojtulewicz, Corelight)

3.2.0-dev.21 | 2020-02-13 17:14:26 -0800

  * Check for failure when registering event manager with iosource manager
    (Tim Wojtulewicz, Corelight)

3.2.0-dev.18 | 2020-02-12 00:04:40 -0800

  * Fix SSH analyzer assertion/leak after duplicate KEX packet (Max Kellermann)

3.2.0-dev.16 | 2020-02-11 23:44:06 -0800

  * Add missing #includes (clang-diagnostic-error) (Tim Wojtulewicz, Corelight)

  * Remove unnecessary const from a return value
    (readability-const-return-type) (Tim Wojtulewicz, Corelight)

  * Create SegmentProfiler objects with names (bugprone-unused-raii)
    (Tim Wojtulewicz, Corelight)

    The reason behind this one is that without a real variable name, the
    profile objects are immediately desctructed and the profiling only happens
    for the small window when they were valid. If the intention is to profile
    the method where they were defined, this doesn't actually happen.

  * Mark a few clang-tidy findings as false-positive (Tim Wojtulewicz, Corelight)

  * Use std::move in a few places instead of copying argument
    (performance-unnecessary-value-param) (Tim Wojtulewicz, Corelight)

  * Use const-references in lots of places (preformance-unnecessary-value-param)
    (Tim Wojtulewicz, Corelight)

  * Use string_view for a couple of Dbg methods (Tim Wojtulewicz, Corelight)

  * Use const references over copying variables
    (performance-unnecessary-copy-initialization, performance-for-range-copy)
    (Tim Wojtulewicz, Corelight)

  * Use single-character version of string find() (performance-faster-string-find)
    (Tim Wojtulewicz, Corelight)

  * Move constructors/operators should be marked noexcept to avoid the compiler
    picking the copy constructor instead (performance-noexcept-move-constructor)
    (Tim Wojtulewicz, Corelight)

3.2.0-dev.1 | 2020-02-08 16:20:20 -0800

  * Update Broker submodule (Jon Siwek, Corelight)

3.2.0-dev | 2020-02-08 16:01:42 -0800

  * Update to start of 3.2.0 development (Jon Siwek, Corelight)

3.1.0-dev.603 | 2020-02-08 15:47:48 -0800

  * Fix broken links in documentation (Jon Siwek, Corelight)

3.1.0-dev.602 | 2020-02-08 12:44:40 -0800

  * Update submodules to release versions (Jon Siwek, Corelight)

3.1.0-dev.601 | 2020-02-08 09:40:15 -0800

  * Update embedded CAF to 0.17.4 (Jon Siwek, Corelight)

3.1.0-dev.600 | 2020-02-07 21:10:00 -0800

  * Note new CMake 3.0+ and C++17 requirements in NEWS (Jon Siwek, Corelight)

3.1.0-dev.599 | 2020-02-07 19:58:35 -0800

  * Improve an input framework test (Jon Siwek, Corelight)

3.1.0-dev.598 | 2020-02-07 15:06:56 -0800

  * Improve stability of a &expire_func btest (Jon Siwek, Corelight)

3.1.0-dev.597 | 2020-02-07 15:25:56 -0700

  * plugin/Manager: migrate to std::string_view (Max Kellermann)

  * util: optimize the normal_path() common case

    Speeds up Zeek startup by 2%. (Max Kellermann)

  * util: pass string_view to without_bropath_component() (Max Kellermann)

  * module_util: make GLOBAL_MODULE_NAME constexpr

    This allows the compiler to hard-code pointers to the string without
    looking up a global variable at runtime. (Max Kellermann)

  * Scope: convert Scope::Lookup() and others to template

    Allows passing rvalue references which eliminates unnecessary
    std::string copies.

    This speeds up Zeek startup by 1-2%. (Max Kellermann)

  * Scope: Vars() returns const reference

    No caller wants to modify the container. (Max Kellermann)

  * Anon: remove unnecessary {map,vector}::clear() calls

    The destructor will do this automatically. (Max Kellermann)

  * Dict: make the destructor non-virtual (Max Kellermann)

  * Obj: make the Location constructors `constexpr` (Max Kellermann)

  * Obj: remove unused fields Location::{timestamp,text} (Max Kellermann)

  * Obj: remove Location::delete_data, nobody ever sets it

    This allows removing the destructor completely, making the class
    trivially-destructible. (Max Kellermann)

  * Obj: make the Location destructor non-virtual

    Nobody ever derives from this class.  This removes an indirect call
    and allows inlining the destructor.

    This speeds up Zeek startup by 1-2%. (Max Kellermann)


3.1.0-dev.584 | 2020-02-07 14:12:17 -0800

  * Improve kerberos analyzer address and event handling

    Adds a weird, "invalid_kerberos_addr_len", for invalid kerberos host
    address lengths and also fixes a memory leak when processing KRB_KDC_REQ
    and KRB_KDC_REP messages for message types that do not match a
    known/expected type. (Jon Siwek, Corelight)

3.1.0-dev.582 | 2020-02-07 09:46:00 -0800

  * Add environment variable to disable supervisor SIGKILL

    For use in btests since SIGKILL can otherwise bypass the usual
    btest-bg-wait timeout mechanism and make reason for test failures less
    obvious. (Jon Siwek, Corelight)

  * Fix a btest's race condition by handling Pcap::file_done (Jon Siwek, Corelight)

  * Add Pcap::file_done event

    It signals when a pcap file is done being processed. (Jon Siwek, Corelight)

  * Improve btest timeouts

    * Generally increase timeouts for tests that have recent transient
      failures

    * Change any test that relied on `btest-bg-wait -k` since that's never
      going to play with with CI systems.  Instead, we always need to have
      a well-defined termination condition in the test itself (and most
      already did, so didn't really need the `-k` flag anyway). (Jon Siwek, Corelight)

3.1.0-dev.577 | 2020-02-07 09:43:48 -0800

  * Use 2 btest retries for CI (Jon Siwek, Corelight)

3.1.0-dev.575 | 2020-02-07 09:31:50 -0800

  * GH-780: Prevent log batches from indefinite buffering

    Logs that got sent sparsely or burstily would get buffered for long
    periods of time since the logic to flush them only does so on the next
    log write.  In the worst case, a subsequent log write could never happen
    and cause a log entry to be indefinitely buffered.

    This fix introduces a recurring event/timer to simply flush all pending
    logs at frequency of Broker::log_batch_interval. (Jon Siwek, Corelight)

3.1.0-dev.571 | 2020-02-07 10:33:16 +0000

  * Update NEWS to point out Broker wireformat change. (Robin Sommer,
    Corelight)

3.1.0-dev.570 | 2020-02-06 14:58:34 -0800

  * add &on_change attribute for sets/tables

    The &on_change attribute allows users to specify a function that is
    called when a set or table is changed via a direct insertion/delete/change
    or element expiry.

    Example:

    function set_change(t: set[string], tpe: TableChange, idx: string)
        {
        print "set_change", idx, tpe;
        }

    global s: set[string] &on_change=set_change;

    (Johanna Amann, Corelight)

3.1.0-dev.554 | 2020-02-06 13:05:39 -0800

  * Switch GitHub Action for CI emails to use zeek organization (Jon Siwek, Corelight)

  * Restore accidentally reverted change to fix FreeBSD 11 build (Tim Wojtulewicz, Corelight)

3.1.0-dev.552 | 2020-02-05 21:47:08 -0800

  * Update CAF to current 0.17.4 release branch (Jon Siwek, Corelight)

3.1.0-dev.551 | 2020-02-05 17:50:56 -0800

  * Remove presumed ownership of HeartbeatTimer (Jon Siwek, Corelight)

    threading::Manager is currently never deleted, but if that ever changes,
    deleting the stored HeartbeatTimer pointer would be a double-free
    since TimerMgr owns it.

  * Set threading::Manager terminating state (Jon Siwek, Corelight)

    Coverity CID 1417429

  * Remove unused HeartbeatTimer member: do_expire (Jon Siwek, Corelight)

    Coverity CID 1417434

3.1.0-dev.548 | 2020-02-05 14:57:08 -0800

  * GH-773: Make EventMgr an IOSource so that events interrupt kevent waits (Tim Wojtulewicz, Corelight)

3.1.0-dev.546 | 2020-02-05 10:51:35 -0800

  * Change threading formatter errors to warnings. (Johanna Amann, Corelight)

    This change standardizes threading formatter error handling and moves
    the remaining error calls to be warnings instead.

    This is in line with already existing code - in most cases warnings were
    raised, only a few cases raised errors. These cases do not differ
    significantly from other cases in which warnings are raised.

    This also fixes GH-692, in which misformatted lines prevent future file
    parsing.

    This commit also moves the FailWarn method that is used by both the
    config and the ascii reader up to the ReaderBackend. Furthermore it
    makes the Warning method of ReaderBackend respect the warning
    suppression that is introduced by the FailWarn method.

  * Update NEWS about plugin breakage from the include-file changes (Tim Wojtulewicz, Corelight)

3.1.0-dev.542 | 2020-02-04 14:05:19 -0700

  * UID, ..: un-inline methods to reduce header dependencies

    Only 1% build time speedup, but still, it declutters the headers a bit.

    Before this patch:

     2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
     72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps

    After this patch:

     2537.19user 142.94system 2:26.90elapsed 1824%CPU (0avgtext+0avgdata 1434268maxresident)k
     16240inputs+8887152outputs (1931major+48728888minor)pagefaults 0swaps (Max Kellermann)

  * include cleanup

    The Zeek code base has very inconsistent #includes.  Many sources
    included a few headers, and those headers included other headers, and
    in the end, nearly everything is included everywhere, so missing
    #includes were never noticed.  Another side effect was a lot of header
    bloat which slows down the build.

    First step to fix it: in each source file, its own header should be
    included first to verify that each header's includes are correct, and
    none is missing.

    After adding the missing #includes, I replaced lots of #includes
    inside headers with class forward declarations.  In most headers,
    object pointers are never referenced, so declaring the function
    prototypes with forward-declared classes is just fine.

    This patch speeds up the build by 19%, because each compilation unit
    gets smaller.  Here are the "time" numbers for a fresh build (with a
    warm page cache but without ccache):

    Before this patch:

     3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
     760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps

    After this patch:

     2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
     72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps (Max Kellermann)

  * Updating submodule(s).

     [nomail] (Jon Siwek, Corelight)


3.1.0-dev.538 | 2020-02-04 11:57:35 +0000

  * Updating submodules.

3.1.0-dev.536 | 2020-02-03 15:44:10 -0700

  * Change various functions to by-value std::string_view args (Jon Siwek, Corelight)

3.1.0-dev.534 | 2020-02-03 15:42:24 -0700

  * Change packet source fields of Options to std::optional (Jon Siwek, Corelight)

3.1.0-dev.531 | 2020-02-03 13:23:22 -0800

  * Update sqlite to 3.31.1 (Johanna Amann, Corelight)

	* Update certificate list to state of NSS 3.49.2. (Johanna Amann, Corelight)

3.1.0-dev.530 | 2020-02-03 13:21:16 -0800

  * Disable Travis leak test (Jon Siwek, Corelight)

    Travis environment doesn't offer enough memory for running ASan very
    well, but Cirrus has it covered anyway.

3.1.0-dev.529 | 2020-02-03 13:04:30 -0800

  * CMakeLists.txt: fix cross-compilation with binpac (Fabrice Fontaine)

3.1.0-dev.527 | 2020-02-03 10:11:00 -0800

  * Change requested memory in Cirrus CI config (Jon Siwek, Corelight)

    The ASan build does require quite a bit more, but also the default
    release builds should do ok with slightly less.

3.1.0-dev.526 | 2020-02-03 10:04:00 -0800

  * Increase timeout value for live interfaces without file descriptors (Tim Wojtulewicz, Corelight)

  * Use ranged-for loops in a few places in iosource Manager (Tim Wojtulewicz, Corelight)

  * Change order of includes in iosource Manager, fixes build on FreeBSD 11 (Tim Wojtulewicz, Corelight)

3.1.0-dev.521 | 2020-01-31 14:18:17 -0800

  * util: optimize expand_escape() by avoiding sscanf() (Max Kellermann)

    sscanf() is notoriously slow, and the default scripts have lots of hex
    escapes.  This patch can reduce Zeek's startup time by 3-9%.

3.1.0-dev.519 | 2020-01-31 13:19:09 -0800

  * util: optimize tokenize_string() and normalize_path() (Max Kellermann)

    This patch can speed up Zeek startup by 10-25%.

    Adds a new tokenize_string() overload which returns string_views and
    changes existing tokenize_string() and normalize_path() to use string_view
    arguments.

3.1.0-dev.510 | 2020-01-31 11:20:28 -0800

  * Remove extra fmt() in a reporter->Error() call (Jon Siwek, Corelight)

  * parse.y: Properly set location info for functions (Arne Welzel)

    When defining a function, remember the location where the function header
    was and restore it before calling `end_func()`. Inside `end_func()`, a
    `BroFunc` object is created using the current global location information.

    This came up while experimenting with zeek script profiling and wondering
    why the locations set for `BroFunc` were "somewhere" in the middle of
    functions instead of spanning them.

  * Desc: move realloc() call out of the loop (Max Kellermann)

  * SerializationFormat: move realloc() call out of the loop (Max Kellermann)

  * PacketDumper: remove unused types (Max Kellermann)

3.1.0-dev.503 | 2020-01-31 11:00:32 -0800

  * Show percentage of packets dropped in the final process output (Tim Wojtulewicz, Corelight)

  * Remove concept of multiple timer managers (Tim Wojtulewicz, Corelight)

    - All timers are now handled by a single global timer manager, which
      simplifies how they handled by the IOSource manager.

    - This change flows down a number of changes to other parts of the code.
      The timer manager tag field is removed, which means that matching
      connections to a timer manager is also removed. This removes the ability to
      tag a connection as internal or external, since that's how the connections
      where differentiated. This in turn removes the `current_conns_extern` field
      from the `ConnStats` record type in the script layer.

  * Have terminate_processing() raise SIGTERM instead of calling the signal handler directly (Tim Wojtulewicz, Corelight)

  * PktSrc iosource changes to match the new IOSource API (Tim Wojtulewicz, Corelight)

  * Broker manager changes to match the new IOSource API and loop architecture (Tim Wojtulewicz, Corelight)

  * DNS_Mgr changes to match the new IOSource API and loop architecture (Tim Wojtulewicz, Corelight)

  * Threading changes for the new loop architecture (Tim Wojtulewicz, Corelight)

    - threading::Manager is no longer an IOSource.

    - threading::MsgThread is now an IOSource. This allows threads themselves
      to signal when they have data to process instead of continually checking
      each of the threads on every loop pass.

    - Make the thread heartbeat timer an actual timer and let it fire as
      necessary instead of checking to see if it should fire

  * Add Trigger manager for managing triggers created by things like 'when' statements

    - Adds new trigger namespace (Tim Wojtulewicz, Corelight)

    - Adds trigger::Manager class as a new IOSource for keeping track of
      triggers and integrating them into the loop. Previously the loop relied on
      the event manager Drain() method to process all triggers on every loop, but
      now that the loop actively waits for events to occur, triggers would not
      fire when they needed to. Adding them as part of the loop ensures they're
      checked.

  * Make TimerMgr an IOSource (Tim Wojtulewicz, Corelight)

    - This allows the loop to check what the next timeout is and use that as
      the basis for the timeout of the poll

    - This commit also removes the TimerMgr::Tag type, since it causes a name
      clash with other code in IOSource

  * Modify IOSource Manager to implement new loop architecture (Tim Wojtulewicz, Corelight)

    - Removes entire FindSoonest method that includes all of the calls to select() for checking for ready sources

    - Removes FD_Set checking against IOSources

    - Adds system for registering and unregistering file descriptors from
      IOSources. This allows individual sources to mark themselves as ready to be
      checked by the loop as they become available.

    - Adds entirely new loop architecture based on checking the IOSources for
      when their next timeout is, and then waiting for either that timeout or
      when the next source is ready. This also implements the polling based on
      what the OS supports, instead of just calling select() on all platforms.
      Currently it supports kqueue, epoll, and plain poll.

    - Adds system for pinging the loop to force it to wake up

  * Minor amount of code cleanup in Pcap IO source (Tim Wojtulewicz, Corelight)

  * IOSource API changes for new loop architecture (Tim Wojtulewicz, Corelight)

    - Removed GetFD and NextTimestamp methods

    - Removed concept of idle sources

    - Renamed Init to InitSource to avoid a name clash with the thread code

    - Added GetNextTimeout method

    This commit also removes the FD_Set file since it's no longer used

  * Type and variable usage cleanup in Net.h (Tim Wojtulewicz, Corelight)

  * Remove #include of some iosource files from Net.h (Tim Wojtulewicz, Corelight)

  * Only allow a single trace file (-r) or interface (-i) option on the command-line (Tim Wojtulewicz, Corelight)

  * Remove CQ_TimerMgr in favor of PQ_TimerMgr (Tim Wojtulewicz, Corelight)

3.1.0-dev.482 | 2020-01-30 19:16:03 -0800

  * Update NEWS for naming changes. (Robin Sommer, Corelight)

  * Remove test for legacy plugin. (Robin Sommer, Corelight)

    We no longer support creating new plugins using the old naming
    scheme.

  * Remove legancy symlinks in aux/. (Robin Sommer, Corelight)

    That's aux/bro-aux and aux/broctl.

  * Add warnings when loading scripts ending in ".bro", or using legacy environment variables. (Robin Sommer, Corelight)

  * Fix missing rename to .zeek-dns-cache. (Robin Sommer, Corelight)

  * No longer symlink local.zeek to local.bro. (Robin Sommer, Corelight)

  * Update notice user agent. (Robin Sommer, Corelight)

  * Remove old_comm_usage_is_ok. (Robin Sommer, Corelight)

  * Remove bro-config.h.in and bro-path-dev.in. (Robin Sommer, Corelight)

  * Change Bro wrapper script to now abort when old executable names are still used. (Robin Sommer, Corelight)

  * Remove APIs that were explicitly deprecated to be removed in 3.1. (Robin Sommer, Corelight)

    Special handling for bro_{init,done,script_loaded} events: if still
    used, they cause Zeek to abort at startup.

3.1.0-dev.469 | 2020-01-30 08:44:15 -0700

  * Fix leaked FDs in redirecting supervised node stdout/stderr (Jon Siwek, Corelight)

  * Add checks for failed fcntl calls (Jon Siwek, Corelight)

  * Set Pipe file descriptor flags correctly (Jon Siwek, Corelight)

3.1.0-dev.465 | 2020-01-29 11:45:15 -0800

  * Scope: fix memory leak by removing duplicate copy_string() call (Max Kellermann)

    Note this is a parse-time memory leak, not a run-time leak.

  * util, nb_dns: fix off-by-one bugs in strncpy() calls (Max Kellermann)

    Fortunately, these bugs had no effect because the following lines
    overwrote the last character with a null byte.

  * Type, util: add `constexpr` to static variables (Max Kellermann)

  * Net: remove unused variable (Max Kellermann)

3.1.0-dev.460 | 2020-01-29 11:16:15 -0800

  * Skip file analysis for zero-length SSL/TLS data (Jon Siwek, Corelight)

3.1.0-dev.458 | 2020-01-29 12:53:32 +0000

  * Add a new supervisor framework that enables Zeek to operate
    clusters of processes itself without any external help. (Jon
    Siwek, Corelight)

    The Supervisor framework provides an entirely new deployment mode
    for Zeek, one that supervises a set of Zeek processes that are
    meant to be persistent. A Supervisor automatically revives any
    process that dies or exits prematurely and also arranges for an
    ordered shutdown of the entire process tree upon its own
    termination. This Supervisor mode for Zeek provides the basic
    foundation for process configuration/management that could be used
    to deploy a Zeek cluster similar to what ZeekControl does, but is
    also simpler to integrate as a standard system service.

    This mode is still experimental and will evolve over time. The
    command-line argument of ``-j`` toggles Zeek to run in "Supervisor
    mode" to allow for creation and management of child processes. If
    you're going to test this, please note that you will need some
    custom script code to configure the processes you want Zeek to
    run. See the documentation for more information:
    https://docs.zeek.org/en/stable/frameworks/supervisor.html

3.1.0-dev.408 | 2020-01-28 17:56:02 -0800

  * Update Cirrus CI config to use macOS Catalina (Jon Siwek, Corelight)

3.1.0-dev.406 | 2020-01-28 12:42:25 -0800

  * util: use getrandom() on Linux if available (Max Kellermann)

    Unlike /dev/urandom, getrandom() doesn't need a file descriptor and
    works when there is no /dev.  It requires Linux 3.17 and glibc 2.25,
    but there is a fallback to the old code.

3.1.0-dev.403 | 2020-01-24 15:15:04 -0800

  * Fixed decompose_uri() errors on URIs with empty port component (Frerich Raabe)

3.1.0-dev.398 | 2020-01-23 18:08:43 -0800

  * Skip check for outdated docs in Cirrus CI for PRs (Jon Siwek, Corelight)

3.1.0-dev.397 | 2020-01-23 15:30:33 -0800

  * Use consistent zeek_init priority for Log::create_stream calls

    Typically in base scripts, Log::create_stream() is called in zeek_init()
    handler with &priority=5 such that it will have already been created
    in the default zeek_init() &priority=0. (Jon Siwek, Corelight)

3.1.0-dev.395 | 2020-01-23 13:50:42 +0000

  * Update Broker submodule.

3.1.0-dev.394 | 2020-01-23 13:18:54 +0000

  * Improve creation of SMB transaction data strings. (Jon Siwek,
    Corelight)

3.1.0-dev.392 | 2020-01-23 13:14:28 +0000

  * Improve FTP word/whitespace handling. (Jon Siwek, Corelight)

3.1.0-dev.389 | 2020-01-18 10:49:15 +0000

  * GHI-595: Convert from nlohmann/json to RapidJSON for performance
    reasons. (Tim Wojtulewicz, Corelight)

  * Optimize json_escape_utf8() and expand its unit tests. (Tim
    Wojtulewicz, Corelight)

  * Convert type-checking macros to actual functions. (Tim
    Wojtulewicz, Corelight)

  * Use the list of files from clang-tidy when searching for unit
    tests. (Tim Wojtulewicz, Corelight)

3.1.0-dev.383 | 2020-01-17 11:51:01 +0000

  * Various code modernization cleanup. (Tim Wojtulewicz, Corelight)

3.1.0-dev.378 | 2020-01-16 13:18:13 +0000

  * Handle invalid Base64 encodings in FTP ADAT analyzer (Jon Siwek,
    Corelight)

3.1.0-dev.376 | 2020-01-14 09:45:45 -0800

  * Fix warning when reading files from non-network sources (Seth Hall, Corelight)

    If files are being read from non-network sources, there was a warning in
    the SSL base scripts about missing the f$conns field.

3.1.0-dev.372 | 2020-01-13 12:10:42 +0000

  * Fix method returning a reference to a temporary. Found by
    cppcheck, via by Arthur Wong. (Robin Sommer, Corelight)

3.1.0-dev.370 | 2020-01-09 19:04:59 -0800

  * Update timeouts for a btest (Jon Siwek, Corelight)

3.1.0-dev.368 | 2020-01-09 13:39:38 -0800

  * Extend ASAN_OPTIONS in btests instead of overwrite (Jon Siwek, Corelight)

3.1.0-dev.366 | 2020-01-09 12:28:32 -0800

  * Update external test repo commit hash (Jon Siwek, Corelight)

3.1.0-dev.365 | 2020-01-09 11:38:06 -0800

  * Enable LeakSanitizer for unit tests run via doctest (Jon Siwek, Corelight)

3.1.0-dev.363 | 2020-01-09 10:25:24 -0800

  * fix NTLM field value access

    The fields being checked for existence were not the same as the fields
    being accessed. (Justin Azoff, Corelight)

3.1.0-dev.361 | 2020-01-08 10:29:59 -0800

  * Increase timeout for a few btests (Jon Siwek, Corelight)

3.1.0-dev.360 | 2020-01-07 20:58:19 -0800

  * Preserve optimization setting for sanitizer build (Jon Siwek, Corelight)

3.1.0-dev.359 | 2020-01-06 16:40:55 -0800

  * Fix build warning in X509 plugin (Tim Wojtulewicz, Corelight)

3.1.0-dev.357 | 2020-01-06 14:10:54 -0800

  * GH-412: Enable Patterns as Table index (Dev Bali, Corelight)

3.1.0-dev.354 | 2020-01-06 14:16:29 -0700

  * Fix snprintf compiler warning in hexdump BIF (Jon Siwek, Corelight)

  * Updating submodule(s).

     [nomail] (Jon Siwek, Corelight)


3.1.0-dev.351 | 2020-01-06 11:20:09 -0800

  * Increase a timeout for a btest (Jon Siwek, Corelight)

3.1.0-dev.350 | 2020-01-06 10:51:07 -0800

  * Fix Cirrus CI FreeBSD python dependencies installation (Jon Siwek, Corelight)

3.1.0-dev.348 | 2020-01-06 11:19:36 -0700

  * Increase timeout for a btest

    May be more timing-sensitive with slower ASAN/LSAN configuration in
    addition to already-slower CI environment. (Jon Siwek, Corelight)

  * Switch CI scripts to use ASAN/LSAN instead of gperftools

    Using AddressSanitizer/LeakSanitizer is better.  It covers the full
    unit/baseline test suite by default without requiring one to write
    specific memory leak tests.  It also covers other types of memory errors
    besides just leaks. (Jon Siwek, Corelight)

  * Remove redundant memory leak btests

    Or otherwise convert into a regular btest if it didn't already seem to
    be covered.

    There's no need for a separate memory leak test group since compiling
    with LeakSanitizer now covers leak checking for the full btest suite. (Jon Siwek, Corelight)

  * Fix reference counting issues related to lambdas/closures

    For example, circular references between a lambda function the frame
    it's stored within and/or its closure could cause memory leaks.

    This also fixes other various reference-count ownership issues that
    could lead to memory errors.

    There may still be some potential/undiscovered issues because the "outer
    ID" finding logic doesn't look quite right as the AST traversal descends
    within nested lambdas and considers their locals as "outer", but
    possibly the other logic for locating values in closures or cloning
    closures just works around that behavior. (Jon Siwek, Corelight)

  * Disable LeakSanitizer for btests that have known leaks

    E.g. ones that throw interpreter exceptions,  as those are currently
    known to potentially cause leaks.  Fixing the underlying leaks involves
    the larger task of more IntrusivePtr usage.

    Reference cycles may also cause leaks. (Jon Siwek, Corelight)

  * Rewrite the btest for when-statement timeouts

    To avoid a memory leak in DNS lookups that's hard to work around and
    does not otherwise effect typical operation when Zeek is allowed to
    continue to run after zeek_init(). (Jon Siwek, Corelight)

  * Fix memory leak when a logging plugin hook prevents a write (Jon Siwek, Corelight)

  * Fix memory leaks in various input framework error-handling cases (Jon Siwek, Corelight)

  * Fix memory leak in Reporter::get_weird_sampling_whitelist() BIF (Jon Siwek, Corelight)

  * Fix reference counting of Log::Filter "config" field

    Which can potentially be a minor memory leak if there's a lot of dynamic
    adding/removing of logging filters. (Jon Siwek, Corelight)

  * Fix memory leak in system_env() BIF (Jon Siwek, Corelight)

  * Clean up triggers awaiting global state modification at shutdown

    Otherwise they can be reported as memory leaks since no more global
    state modifications will take place to notify the trigger to clean
    itself up. (Jon Siwek, Corelight)

  * Fix memory leak in initializing log writers with no local backend (Jon Siwek, Corelight)

  * Fix packet filter memory leaks (Jon Siwek, Corelight)

  * Skip sending thread heartbeat if it alread asked to be finished

    Otherwise the heartbeat message may fail to be processed and show up as
    leaked memory. (Jon Siwek, Corelight)

  * Fix memory leak of sqlite input reader prepared statement (Jon Siwek, Corelight)

  * Prevent duplicate "finish" threading messages

    As they don't get processed and may show up as a memory leak. (Jon Siwek, Corelight)

  * Fix memory leak when table-based input stream overwrites old entries (Jon Siwek, Corelight)

  * Fix scripting error in a DHCP btest (Jon Siwek, Corelight)

  * Fix memory leaks in Kerberos ticket decryption

    Memory allocated to the decrypted ticket data as well as the server
    principal were not freed.

    Also fixed potential leaks in error cases that called
    krb5_get_error_message() without freeing the returned value. (Jon Siwek, Corelight)

  * Fix scripting error in an ftp btest (Jon Siwek, Corelight)

  * Update paraglob submodule (Jon Siwek, Corelight)

  * Fix malloc/delete mismatch in JSON formatting

    ODesc allocated with malloc() and BroString deallocated with delete[],
    but really the intermediate BroString wasn't even needed when copying
    into std::string. (Jon Siwek, Corelight)

  * Delete/timeout pending DNS requests during shutdown

    Primarily, this change prevents the pending requests showing up as
    memory leaks. (Jon Siwek, Corelight)

  * Fix memory leak in OCSP parsing functions

    Various OCSP parsing functions used in presence of OpenSSL 1.1 used
    "d2i_ASN1_SEQUENCE_ANY" which returns a "STACK_OF(ASN1_TYPE)", but used
    "sk_ASN1_TYPE_free" instead of "sk_ASN1_TYPE_pop_free" to free it.  The
    former only frees the stack structure while the later frees both the
    structure and the elements. (Jon Siwek, Corelight)

  * Free the global X509 certificate root store on shutdown

    Otherwise LeakSanitizer reports its contents as leaked. (Jon Siwek, Corelight)

  * Add general LeakSanitizer macros/instrumentation (Jon Siwek, Corelight)

  * Improve --sanitizers configure option

    * Rename SANITIZERS CMake variable to ZEEK_SANITIZERS for clarity

    * Use -O1 by default to improve speed (set NO_OPTIMIZATIONS env. var.
      to override and use -O0).  Uses -fno-optimize-sibling-calls with -O1
      to still get "perfect stack traces".

    * Updates various sub-projects with sanitizer improvements:
      binpac and bifcl, by default, now ignore leaks reported by LeakSanitizer
      so that it doesn't interfere with the Zeek build (Jon Siwek, Corelight)


3.1.0-dev.319 | 2020-01-06 09:44:11 -0800

  * Mark safe_snprintf and safe_vsnprintf as deprecated, remove uses of them (Tim Wojtulewicz, Corelight)

  * Add unit tests to util.cc and module_util.cc (Tim Wojtulewicz, Corelight)

3.1.0-dev.314 | 2019-12-18 13:36:07 -0800

  * Add GitHub Action for CI notification emails (Jon Siwek, Corelight)

3.1.0-dev.313 | 2019-12-18 13:23:51 -0800

  * Add Cirrus CI config (Jon Siwek, Corelight)

3.1.0-dev.309 | 2019-12-16 09:40:01 -0800

  * GHI-599: avoid memory leak with default pattern matching and json formatting (Tim Wojtulewicz, Corelight)

3.1.0-dev.307 | 2019-12-16 08:20:58 -0800

  * Update URL for Malware Hash Registry website (Jon Siwek, Corelight)

3.1.0-dev.300 | 2019-12-05 12:34:41 -0700

  * GH-700: add zeek_args() BIF

    Provides access to all zeek command-line arguments ("argv"). (Jon Siwek, Corelight)

3.1.0-dev.297 | 2019-12-05 11:57:12 -0700

  * GH-700: add packet_sources() BIF

    Provides access to properties of packet sources, like interface or pcap
    file name. (Jon Siwek, Corelight)

3.1.0-dev.295 | 2019-12-04 14:43:27 -0700

  * Use new Zeek Logo instead of Bro Eyes on README.md (Dev Bali, Corelight)

3.1.0-dev.292 | 2019-12-02 13:37:19 -0800

  * GH-619: Allow "print" statements to be redirected to a Log (Dev Bali, Corelight)

3.1.0-dev.286 | 2019-11-21 08:47:32 -0800

  * GH-684: Fix parsing of RPC calls with non-AUTH_UNIX flavors

    The parsing logic that should be specific to the AUTH_UNIX credential
    flavor was previously applied unconditionally to other flavors. (Jon Siwek, Corelight)

3.1.0-dev.284 | 2019-11-21 08:29:36 -0800

  * Fix ZEEK_PROFILER_FILE file format/parsing

    Some Zeek script statement descriptions were exceeding the hardcoded
    maximum length and also could contain tab characters which were
    supposed to be reserved for use as a delimiter in the file format. (Jon Siwek, Corelight)

3.1.0-dev.282 | 2019-11-18 12:06:13 +0000

  * GH-646: Add new "successful_connection_remove" event. (Jon Siwek, Corelight)

    Includes:

        - Switch Zeek's base scripts over to using it in place of
          "connection_state_remove". The difference between the two is
          that "connection_state_remove" is raised for all events
          while "successful_connection_remove" excludes TCP
          connections that were never established (just SYN packets).
          There can be performance benefits to this change for some
          use-cases.

        - Add new event called ``connection_successful`` and a new
          ``connection`` record field named "successful" to help
          indicate this new property of connections.

3.1.0-dev.280 | 2019-11-15 18:27:27 -0800

  * Run doctest unit tests in Travis CI (Jon Siwek, Corelight)

  * Fix indents/whitespace in Travis CI script (Jon Siwek, Corelight)

  * Adjust btests for OpenBSD portability (Jon Siwek, Corelight)

  * Convert pcapng test suite files to pcap format for OpenBSD compat (Jon Siwek, Corelight)

  * Fix undefined symbols loading libbroker on OpenBSD (Jon Siwek, Corelight)

  * Fix compile warnings on OpenBSD (Jon Siwek, Corelight)

3.1.0-dev.271 | 2019-11-14 19:16:31 -0800

  * Add initial scaffold for unit testing via doctest (Dominik Charousset, Corelight)

3.1.0-dev.266 | 2019-11-14 17:29:00 -0800

  * Add hint to run `make distclean` if configure fails (Simon Hardy-Francis, Corelight)

3.1.0-dev.264 | 2019-11-14 16:57:13 -0800

  * Install test-all-policy.zeek script (Jon Siwek, Corelight)

    The zeekygen script, which is already installed, refers to it, and
    could also generally be useful for "test everything" type scenarios.

3.1.0-dev.263 | 2019-11-13 13:43:16 -0800

  * Improve record_fields() BIF to recursively give full container type names (Henri DF)

3.1.0-dev.257 | 2019-11-11 13:40:11 -0800

  * Update embedded CAF to 0.17.3 (Jon Siwek, Corelight)

3.1.0-dev.256 | 2019-11-07 13:42:11 -0800

  * Allow record_fields() string arguments that name a record type (Jon Siwek, Corelight)

3.1.0-dev.253 | 2019-11-05 09:54:01 -0800

  * Add and use new IntrusivePtr type (Dominik Charousset, Corelight)

    Manual memory management via Ref/Unref is verbose and prone to error. An
    intrusive smart pointer automates the reference counting, makes code
    more robust (in particular w.r.t. to exceptions) and reduces boilerplate
    code. A big benefit of the intrusive smart pointers for Zeek is that
    they can co-exist with the manual memory management. Rather than having
    to port the entire code base at once, we can migrate components
    one-by-one. In this first step, we add the new template
    `IntrusivePtr<T>` and start using it in the Broker Manager. This makes
    the previous `unref_guard` obsolete.

3.1.0-dev.247 | 2019-11-04 17:26:32 -0800

  * Cluster-ize FTP data channel analysis (Jeff Barber)

    The ftp_data_expected table is now synchronized across Zeek
    worker nodes.  Note there's still a possible race condition where
    the traffic for the ftp-data connection proceeds before the Zeek
    cluster has a change to inform workers to expect it.

3.1.0-dev.245 | 2019-11-04 16:57:11 -0800

  * Use CMake object libraries for subdir libs and plugins (Dominik Charousset, Corelight)

3.1.0-dev.242 | 2019-11-04 09:58:53 -0800

  * Fix error handling in decode_base64() (Jon Siwek, Corelight)

3.1.0-dev.241 | 2019-11-04 09:44:20 -0800

  * Revert recent changes to Base64.h API. (Jon Siwek, Corelight)

    This reverts commit 705210a035d4128ecdf249c7db98607c570bfcd2.
    The original changes broke the Base64.h API which may be used by
    external plugins.

3.1.0-dev.240 | 2019-11-01 15:58:55 -0700

  * Use #pragma once in Base64.h (Jon Siwek, Corelight)

3.1.0-dev.239 | 2019-10-31 12:18:16 -0700

  * Fix Travis CI script to run Coverity build in container (Jon Siwek, Corelight)

3.1.0-dev.237 | 2019-10-31 06:37:23 +0100

  * Remove "using namespace std" from header files (Jon Siwek, Corelight)

3.1.0-dev.235 | 2019-10-30 15:04:13 -0700

  * GH-649: define missing DLT_NFLOG to support OpenBSD (Jon Siwek, Corelight)

3.1.0-dev.234 | 2019-10-30 13:33:08 -0700

  * Add --libdir convenience flag to configure. (Johanna Amann, Corelight)

3.1.0-dev.231 | 2019-10-30 11:02:28 -0700

  * Add missing ipv6 support in patricia.c local_inet_pton() (Tim Shelton)

3.1.0-dev.229 | 2019-10-30 07:57:19 +0000

  * GH-664: fix signature matching for cases where the first TCP
    packet seen contained payload data, regardless of its TCP flags.
    (Jon Siwek, Corelight)

3.1.0-dev.227 | 2019-10-29 09:39:10 -0700

  * Replace build_unique with make_unique (Johanna Amann, Corelight)

    This was a rarely used convenience function from when we did not yet
    have c++17 support.

  * GH-626: Revert "Fix compilation on OS-X." (Johanna Amann, Corelight)

    Reverts workaround in cde28074a169212aa8f38fdac225ecbeac4e642d
    which depended on C++14 features at a time when we used only C++11.

3.1.0-dev.222 | 2019-10-28 20:18:15 -0700

  * Install cmake3 from EPEL on CentOS CI system (Jon Siwek, Corelight)

3.1.0-dev.221 | 2019-10-28 18:30:24 -0700

  * Require CMake 3.0+ (Dominik Charousset, Corelight)

  * Move CMake project() after cmake_minimum_required() (Dominik Charousset, Corelight)

    The call to `project` must come after `cmake_minimum_required` in CMake
    in order to get the correct policy settings.

3.1.0-dev.219 | 2019-10-28 20:59:12 +0100

  * GH-654: allow table() in function &default expressions

    Table parameters of functions previously did not coerce unspecified
    tables used in their &default attribute to the correct type. (Jon Siwek, Corelight)

  * GH-654: allow table() in record &default expressions

    Table fields of records previously did not coerce unspecified tables
    used in their &default attribute to the correct type. (Jon Siwek, Corelight)

  * Fix up some documentation. (Keith J. Jones)

3.1.0-dev.214 | 2019-10-28 10:58:43 +0000

  * GH-585: Require C++17 (Jon Siwek, Corelight)

  * Update Travis CI config for C++17 (Jon Siwek, Corelight)

3.1.0-dev.209 | 2019-10-28 09:17:15 +0000

  * Fix HTTP::build_url. (Alexey Mokeev)

  * Free memory allocated by glob() in plugin manager. (Arne Welzel)

  * Remove always-false condition in plugin manager. (Arne Welzel)

3.1.0-dev.203 | 2019-10-25 10:41:10 -0700

  * Change usage of old bro:see directive to zeek:see (Jon Siwek, Corelight)

  * Use explicit path name for NTP log stream (Jon Siwek, Corelight)

    For consistency (we do this for all other logs) and just to avoid
    the extra path function calls.

3.1.0-dev.201 | 2019-10-25 10:12:49 -0700

  * Fixed typo. (Keith J. Jones)

3.1.0-dev.197 | 2019-10-21 09:34:18 -0700

  * Update embedded CAF in Broker to 0.17.2 (Jon Siwek, Corelight)

3.1.0-dev.195 | 2019-10-21 11:42:18 +0200

  * Prefer cmake3 command, add --cmake=PATH option (Dominik Charousset, Corelight)

3.1.0-dev.192 | 2019-10-17 16:32:16 -0700

   * Zeekctl: change gzip compression level from 9 to default #614 (JC Connell)

3.1.0-dev.191 | 2019-10-16 10:02:45 -0700

  * Update broker submodule to use pybind11 v2.4.3 (Jon Siwek, Corelight)

  * Improve &default validation for global vars: exclude sets (Jon Siwek, Corelight)

  * Remove unnecessary &optional from a Config::set_value() parameter (Jon Siwek, Corelight)

  * Remove check for redundant attributes (Jon Siwek, Corelight)

3.1.0-dev.185 | 2019-10-15 16:46:22 -0700

  * Add weirds for invalid TCP option lengths (Jon Siwek, Corelight)

  * GH-618: add "tcp_options" event containing TCP option values (Jon Siwek, Corelight)

  * Fix tcp_option event

    It was not being raised in all the cases it should have been due to
    an incorrect/unnecessary truncation check. (Jon Siwek, Corelight)

3.1.0-dev.181 | 2019-10-14 22:05:01 -0700

  * Fix redef'ing a table with a new &default attribute (Jon Siwek, Corelight)

3.1.0-dev.179 | 2019-10-14 21:33:15 -0700

  * Support whitespace at end of line for config reader (Seth Hall, Corelight)

3.1.0-dev.177 | 2019-10-14 20:13:24 -0700

  * Proof of Concept on Bugprone Narrowing Conversions: Base64 (Dev Bali, Corelight)

3.1.0-dev.175 | 2019-10-12 10:39:49 -0700

  * Use Ubuntu 14.04 for Travis CI environment (Jon Siwek, Corelight)

3.1.0-dev.172 | 2019-10-08 15:14:35 -0700

  * Add new BIF: print_raw()

    This is a convenience function to make it easier to print literal byte
    sequences to stdout without additional escaping like what may be added
    by the default `print` statement behavior.

    For example, related to GH-596, `print` currently escapes even valid
    UTF-8 byte sequences and makes it difficult to output valid JSON strings
    containing such. (Jon Siwek, Corelight)

  * Fix compilation on OS-X. (Johanna Amann)

3.1.0-dev.169 | 2019-10-07 10:50:35 -0700

  * GH-589: improve printing of sub-microsecond intervals (Jon Siwek, Corelight)

3.1.0-dev.167 | 2019-10-07 10:42:15 -0700

  * GH-591: allow Config::set_value() to use empty/unspecified table/sets (Jon Siwek, Corelight)

  * GH-591: fix reading set[enum] values from input files (Jon Siwek, Corelight)

3.1.0-dev.164 | 2019-10-02 11:04:13 -0700

  * Fix build with external CAF but bundled Broker (Dominik Charousset)

3.1.0-dev.161 | 2019-10-01 16:41:35 -0700

  * Update COPYING.3rdparty (Tim Wojtulewicz, Corelight)

  * Use json::emplace to avoid some extra calls to operator[] (Tim Wojtulewicz, Corelight)

  * Use tessil/unordered-map instead of nlohmann/fifo-map to improve JSON
    logging performance (Tim Wojtulewicz, Corelight)

3.1.0-dev.156 | 2019-10-01 09:05:49 +0000

  * Improve RecordVal JSON formatting to no longer create a record
    introspection table each time when all the required information
    can be obtained directly in the RecordVal and RecordType objects.
    (Jon Siwek, Corelight)

3.1.0-dev.153 | 2019-09-30 21:09:55 -0700

  * GH-602: fix paraglob header inclusion, installation, and search paths (Jon Siwek, Corelight)

3.1.0-dev.151 | 2019-09-30 17:39:50 -0700

  * GH-606: Output nulls into json data if a field isn't set (Tim Wojtulewicz, Corelight)

  * GH-598: handle multi-key sets correctly when outputting json (Tim Wojtulewicz, Corelight)

  * Fix Xcode deprecation warning for std::ptr_fun (Jon Siwek, Corelight)

3.1.0-dev.143 | 2019-09-24 10:23:24 -0700

  * Simplify reassembly code using DataBlockList::LastBlock() (Jon Siwek, Corelight)

  * Fix DataBlockList::DataSize() (Jon Siwek, Corelight)

    Parameters got filled opposite to what they were supposed to be and
    also didn't consider cutoffs that land in the middle of a block.

3.1.0-dev.141 | 2019-09-24 10:31:50 +0000

  * Reorganize reassembly data structures. This replaces the previous
    linked list-based implementation with std::map to avoid O(n) worst
    case performance. (Jon Siwek, Corelight)

3.1.0-dev.131 | 2019-09-23 13:07:09 -0700

  * Add --build-dir as alias for --builddir (Dominik Charousset, Corelight)

  * Fix overly specific include_directories usage in CMake script (Dominik Charousset, Corelight)

3.1.0-dev.127 | 2019-09-20 10:40:54 -0700

  * Fix uncaught InterpreterException in Frame cloning (Robin Sommer, Corelight)

    Fix for CIDs 1402823 and 1394050.

3.1.0-dev.125 | 2019-09-19 16:40:10 -0700

  * Update header include guard in zeek-config.h/bro-config.h (Jon Siwek, Corelight)

3.1.0-dev.124 | 2019-09-17 11:16:47 -0700

  * Add speculative service script. (Jan Grashoefer)

    The speculative service script handles dpd_late_match events to extend
    conn.log with infos about potential protocol identifications.

  * Allow to handle late DPD matches. (Jan Grashoefer)

    If "dpd_match_only_beginning" is disabled, matches of protocol signatures
    can be handeld using protocol_late_match. To prevent further matching in
    this case, the new option "dpd_late_match_stop" may be activated.

3.1.0-dev.118 | 2019-09-17 17:21:58 +0000

  * GH-566: Fix cases where ssh_encrypted_packet event wasn't raised.
    When encrypted data was bundled within the same segment as the
    NewKeys message, it wasn't not reported via a
    ssh_encrypted_package event as it should have been. (Jon Siwek,
    Corelight)

3.1.0-dev.116 | 2019-09-17 10:08:38 -0700

  * Switch from header guards to pragma once (Dominik Charousset, Corelight)

3.1.0-dev.114 | 2019-09-17 09:32:34 -0700

  * GH-580: Add additional 3rd party headers to includes (Derek Ditch)

    Adds FIFO Map and JSON headers to include distribution

3.1.0-dev.112 | 2019-09-16 17:11:41 -0700

  * Add move assignment operator to logging::Tag (Tim Wojtulewicz, Corelight)

    Coverity 1357732

  * Throw an error during input::Manager::CreateTableStream if val is null (Tim Wojtulewicz, Corelight)

    Coverity 1402730

  * Use ntohl instead of manually swapping bytes in extract_XDR_uint32 (Tim Wojtulewicz, Corelight)

    Coverity 1375796 and others

  * Add check for null init variable during make_var. (Tim Wojtulewicz, Corelight)

    Coverity 1403419

  * Remove dead check in Val::check_and_promote. (Tim Wojtulewicz, Corelight)

    Coverity 1401810

  * Handle failure of fcntl in Pipe. (Tim Wojtulewicz, Corelight)

    Coverity 1241934, 1241935

  * Initialize missing field in File. (Tim Wojtulewicz, Corelight)

    Coverity 1057851, 1057852, 1057853

  * Add null check for results of dynamic_cast in AssignExpr::TypeCheck. (Tim Wojtulewicz, Corelight)

    Coverity 1403416 and 1403417

  * Add null check when getting key size for List types. (Tim Wojtulewicz, Corelight)

    Coverity 1058242

3.1.0-dev.102 | 2019-09-16 10:50:47 -0700

  * Stop loading scan detection in local.zeek by default (Jon Siwek, Corelight)

3.1.0-dev.98 | 2019-09-04 16:28:01 -0700

  * Update Broker include dir search path (Jon Siwek, Corelight)

3.1.0-dev.96 | 2019-09-04 13:41:42 -0700

  * Update embedded CAF to 0.17.1 (plus cherry-picked memory leak fix) (Jon Siwek, Corelight)

3.1.0-dev.93 | 2019-09-03 10:35:16 -0700

  * Fix minor typos in code comments (The Alchemist)

3.1.0-dev.89 | 2019-08-29 13:05:09 -0700

  * CID 1404734: fix NetSessions::MemoryAllocation() (Jon Siwek, Corelight)

3.1.0-dev.88 | 2019-08-28 12:48:04 -0700

  * GH-545: add "addl" parameter to flow_weird and net_weird events (Jon Siwek, Corelight)

  * GH-554: don't init PIA endpoint matchers if there's only file-magic

    The logic for initializing PIA endpoint matchers was previously
    skipped if "there's no global rule matcher", and that's only true
    when no signature files get loaded.

    But when using `zeek -b`, some file-magic signatures still get loaded
    by default, so the PIA endpoint matchers still get initialized even
    though they don't need to be -- file-magic patterns play no part
    in PIA.

    For typical use-cases (not using the `-b` flag), this change won't
    help any, but we do at least use `-b` often within the test suite. (Jon Siwek, Corelight)

  * GH-554: remove use of file magic in protocol-based signature logic

    This can be a significant performance/memory improvement since
    otherwise the protocol-based rule matching logic ends up superfluously
    creating file-matching state per file-matcher per connection/endpoint. (Jon Siwek, Corelight)

  * GH-541: add test cases for NTLM AV Pair sequence handling (Jon Siwek, Corelight)

  * GH-541: fix handling of NTLM AV Pair sequences

    Empty AV Pair sequences or AV Pair sequences that lack a terminator
    could cause accesses past the end of the parsed vector. (Jon Siwek, Corelight)

  * Make Tag::Error values constant (Jon Siwek, Corelight)

  * Simplify operator bool()'s used for Tag error checks (Jon Siwek, Corelight)

  * Add/use unspecified IPAddr constants

    The analyzer-scheduling code was otherwise frequently converting the
    unspecified v4/v6 addresses from strings. (Jon Siwek, Corelight)

3.1.0-dev.75 | 2019-08-27 10:04:43 -0700

  * Fix parsing of smb3 negotiate context (Pavel Ershov)

3.1.0-dev.73 | 2019-08-26 14:57:45 -0700

  * Close down SSL/sqlite at shutdown even if net_run not used (Tim Wojtulewicz, Corelight)

  * Finish processing program arguments before setting up SSL/sqlite (Tim Wojtulewicz, Corelight)

3.1.0-dev.70 | 2019-08-23 14:31:17 -0700

  * Make github identify our Flex source correctly. (Seth Hall, Corelight)

3.1.0-dev.68 | 2019-08-23 06:34:50 -0400

  * Fix unit tests for new ordering from NetSessions::Drain (Tim Wojtulewicz, Corelight)

  * Change FragReassembler to use a tuple as a key and use std::map for fragments in Sessions (Tim Wojtulewicz, Corelight)

  * Rework Session/Connection tracking to use a std::map instead of PDict (Tim Wojtulewicz, Corelight)

  * Improve GitHub language identification/stats (Seth Hall, Corelight)

  * Zeekify thread naming. (Seth Hall, Corelight)

    I copied the same style that caf uses ("zk" with single dot and no space).
    This gives some consistency with caf and avoids us wasting more
    space beyond "bro: ".  OSs only give 16 characters for thread names
    so anything we can gain here is nice.

3.1.0-dev.58 | 2019-08-16 18:45:39 +0000

  * GH-532: Improve disable_analyzer BIF. (Jon Siwek, Corelight)

    - Add an extra "prevent" parameter (default value of false), which
      helps prevent the same analyzer type from being attached in the
      future.

    - Fixes disable_analyzer() to work when called even earlier, like
      within the protocol_confirmation event.

    - Fixes disable_analyzer() when called on an analyzer added to the
      tree via TCP_Analyzer::AddChildPacketAnalyzer.

3.1.0-dev.55 | 2019-08-14 16:18:44 -0700

  * Fix misc. Coverity warnings (Jon Siwek, Corelight)

3.1.0-dev.54 | 2019-08-14 15:38:02 -0700

  * Deprecate int/uint{8,16,32,64} typedefs, replace with actual cstdint types (Tim Wojtulewicz, Corelight)

3.1.0-dev.52 | 2019-08-14 13:46:40 -0700

  * Change file_analysis::Manager::ignored to use std::set (Jon Siwek, Corelight)

3.1.0-dev.50 | 2019-08-14 12:32:56 -0700

  * Update documentation for string_to_pattern BIF (Jon Siwek, Corelight)

3.1.0-dev.48 | 2019-08-13 20:15:17 -0700

  * Cleanups related to PDict -> std::map replacements (Jon Siwek, Corelight)

  * Replace various uses of PDict with std::map (Tim Wojtulewicz, Corelight)

3.1.0-dev.40 | 2019-08-13 23:44:45 +0000

  * Change over to whitelisting clang-tidy options instead of
    blacklisting. (Tim Wojtulewicz, Corelight)

  * Use FindClangTidy from the cmake submodule. (Tim Wojtulewicz,
    Corelight)

3.1.0-dev.36 | 2019-08-13 22:42:54 +0000

  * Replace use of deprecated pcap_lookupdev(). (Jon Siwek, Corelight)

  * Use _exit() in Reporter::FatalError. (Jon Siwek, Corelight)

  * GH-533: Use consistent "lib" install dir. (Jon Siwek, Corelight)

3.1.0-dev.30 | 2019-08-13 13:48:47 -0700

  * Add new LogAscii::gzip_file_extension option. (Tim Wojtulewicz, Corelight)

    This can be used with the LogAscii::gzip_level option to set the file
    extension of log files when they are compressed at creation time.

3.1.0-dev.28 | 2019-08-13 12:01:44 -0700

  * Remove redundant buffering in ContentLine analyzer (Justin Azoff)

    The contentline analyzer has two code paths that buffer data:
     * right at the top of DeliverStream
     * later in DoDeliverOnce

    However, contentline can be in plain delivery mode, and if so, the
    buffer resize in DeliverStream does not need to be done just because
    DeliverStream was passed an 8K data chunk.

    This was causing contentline to resize it's buffer to fit chunks of HTTP
    response data.  Additionally, the buffer was sized to be 3/2 of the
    chunk, so an 8K chunk would result in a 12K allocation.

3.1.0-dev.26 | 2019-08-13 11:25:20 -0700

  * Add tests for {http,mime}_all_headers events (Jon Siwek, Corelight)

    And remove unnecessary check for mime_all_headers in HTTP entities
    (they ony raise the http_all_headers event, never mime_all_headers).

3.1.0-dev.24 | 2019-08-12 19:30:26 -0700

  * Avoid buffering all http/mime headers (Justin Azoff)

    Only buffer all http/mime headers if the http_all_headers or
    mime_all_headers events are in use.

3.1.0-dev.22 | 2019-08-12 13:31:12 -0700

  * GH-535: fix typo of "C_HESIOD" in DNS::classes (Jon Siwek, Corelight)

3.1.0-dev.21 | 2019-08-12 13:00:21 -0700

  * Add new distro to Travis CI configuration for running leak tests (Tim Wojtulewicz, Corelight)

3.1.0-dev.18 | 2019-08-09 10:43:28 -0700

  * GH-419: improve multi-protocol logging in known_services.log (Mauro Palumbo)

    Previously, when multiple protocols were detected on a given addr/port
    pair, not all protocols were always logged.

3.1.0-dev.7 | 2019-08-09 09:56:06 -0700

  * Remove empty services from known_services.log (Mauro Palumbo)

  * Add check to log to known_services.log when removing active udp connections (Mauro Palumbo)

3.1.0-dev.5 | 2019-08-09 09:33:22 -0700

  * Add Intel::read_error event to allow custom error handling (Mauro Palumbo)

  * Export Intel::read_entry event for general user access (Mauro Palumbo)

3.1.0-dev | 2019-08-08 16:08:50 -0700

  * Change version to 3.1.0-dev (Jon Siwek, Corelight)

2.6-767 | 2019-08-07 12:27:21 -0700

  * Add memory leak test of closure (un)serialization (Jon Siwek, Corelight)

2.6-765 | 2019-08-07 08:05:35 -0700

  * GH-527: fix parsing of MQTT Remaining Length field

    Packet length is encoded in up to four bytes, with MSB (0x80)
    indicating if there's more bytes in the representation still to follow.
    The comparison/bitwise-mask wasn't correctly testing the MSB.

    Coverity CID 1403964 (Jon Siwek, Corelight)

  * GH-527: fix LambdaExpr::Traverse

    Coverity CID 1403966 (Jon Siwek, Corelight)

  * GH-527: fix ref-counting issues in Frame unserialization

    Coverity CIDs 1403968, 1403967 (Jon Siwek, Corelight)

2.6-760 | 2019-08-05 21:01:16 -0700

  * GH-474: change MQTT::max_payload_size to be a runtime option (Jon Siwek, Corelight)

2.6-758 | 2019-08-05 18:59:53 -0700

  * Fix malformed SMB documentation (Jon Siwek, Corelight)

  * Fix documentation warnings for MQTT identifiers (Jon Siwek, Corelight)

2.6-756 | 2019-08-05 17:32:33 -0700

  * Disable MQTT by default (Johanna Amann, Corelight)

    To enable MQTT, one has to load policy/scripts/mqtt. Like with smb in
    2.5, the consts are loaded by default.

2.6-754 | 2019-08-05 10:12:51 -0700

  * GH-474: add MQTT::max_payload_size option

    This caps size of payload strings within mqtt_publish events and
    mqtt_publish.log files.  A new "payload_len" field in the log file
    shows the real payload size in cases where it may have been truncated. (Jon Siwek, Corelight)

  * GH-474: use topic vectors for MQTT (un)subscribe events/logs (Jon Siwek, Corelight)

  * Update Certificate Transparency list (Johanna Amann)

  * Update CA store to NSS 3.45 (Johanna Amann)

2.6-748 | 2019-08-02 11:55:46 -0700

  * GH-517: fix MQTT suback/unsuback accessing non-existent index (Jon Siwek, Corelight)

2.6-747 | 2019-08-02 11:30:52 -0700

  * Fix how Broker/CAF sleep duration options are set (Jon Siwek, Corelight)

  * Add duration thresholding to the conn-size analyzer. (Johanna Amann, Corelight)

    Now, in addition to setting thresholds for bytes and packet, one can set
    a threshold for connection duration. Note that the threshold event is
    only raised once the next packet in the connection is seen.

    This also fixes a small pre-existing bug, in which a bunch of warnings
    were raised if someone just used the lower-level functions without going
    through the higher-level scripting API.

2.6-744 | 2019-08-01 13:33:12 -0700

  * Guarantee unique internal name for each lambda function
    By dealing with hash collisions. (Jon Siwek, Corelight)

  * Use consistent hashing method for internal lambda function names

    The results of std::hash<std::string> may vary depending on platform.
    E.g. test suite failed on macOS due to Linux generating different lambda
    function names. (Jon Siwek, Corelight)

  * Improve error messages from to_addr and to_subnet BIFs. (Jon Siwek, Corelight)

2.6-739 | 2019-08-01 12:28:25 -0700

  * Fix a test that used a hardcoded Broker port (Jon Siwek, Corelight)

2.6-737 | 2019-08-01 11:22:57 -0700

  * GH-512: add --mandir configure option (Jon Siwek, Corelight)

2.6-735 | 2019-07-31 21:29:58 -0700

  * Fix the link to "good first issue" tickets. (Seth Hall, Corelight)

2.6-733 | 2019-07-31 21:23:37 -0700

  * Add MQTT analyzer, updated/ported from original version by Supriya Kumar (Seth Hall, Corelight)

    This analyzer generates three logs to fully display what is happening over the MQTT connection.
      - mqtt_connect.log
      - mqtt_subscribe.log
      - mqtt_publish.log

    At this time it only supports MQTT 3.1 and 3.1.1

2.6-729 | 2019-07-31 14:24:44 -0700

  * Rename a broxygen unit test to zeekygen (Jon Siwek, Corelight)

2.6-728 | 2019-07-31 14:15:29 -0700

  * Fix hello world script in READMEs (Seth Hall, Corelight & Jon Siwek, Corelight)

  * Fixes a tiny Bro->Zeek renaming issue (Seth Hall, Corelight)

2.6-725 | 2019-07-31 10:44:05 -0700

  * GH-506: fix NTP script errors (Seth Hall, Corelight)

2.6-723 | 2019-07-30 19:36:56 -0700

  * Add LogAscii::enable_utf_8 option (Dev Bali, Corelight)

    This option allows valid utf8 sequences to be written directly
    into the ASCII logs without any escaping.

2.6-713 | 2019-07-30 18:12:49 +0000

  * Fix memory leaks in expire_func introduced by recent changes (Jon
    Siwek, Corelight)

2.6-711 | 2019-07-29 20:15:27 -0700

  * Fix duplicate TCP packets not being detected as retransmissions (Jeff Barber)

2.6-708 | 2019-07-30 02:46:39 +0000

  * Add an additional license file, COPYING.3rdparty, that collects
    all third party software licenses. (Johanna Amann, Corelight)

2.6-706 | 2019-07-30 02:30:44 +0000

  * Zeek's anonymous functions now capture their closures by
    reference. This means that they can use and modify variables from
    the scope that they were generated in. For example:

        local n = 3;
        local f = function() { n += 1; };
        f();
        print n; # prints 4

    See NEWS and the documentations for more details. (Zeke Medley, Corelight)

2.6-674 | 2019-07-29 16:11:42 -0400

  * added more options to SMB2 set-info command (Mauro Palumbo)

2.6-664 | 2019-07-29 10:01:12 -0700

  * Add release branches to Travis CI whitelist (Jon Siwek, Corelight)

  * GH-488: teach the Version module to parse new version scheme

    Such as the new -rc format for release candidates (replacing "beta") and
    -dev.X for development versions in the master branch. (Jon Siwek, Corelight)

  * Improve type inference for vector-of-enum constructor (Jon Siwek, Corelight)

  * GHI-486: Switch over to using LLVM utf8-checking code to better validate characters (Tim Wojtulewicz, Corelight)

  * Fix undefined behavior via casting file analyzers to protocol analyzers

    When generating some events for PE and X509 file analyzers, there's
    an invalid cast from file_analysis::Analyzer to analyzer::Analyzer
    and subsequent invalid member access via analyzer::Analyzer::GetID()
    called on what is really a pointer to a file analyzer. (Jon Siwek, Corelight)

  * Fix undefined behavior via hrw_weight BIF signed int overflow (Jon Siwek, Corelight)

  * Fix undefined behavior via invalid TCP analyzer cast

    A connection's root analyzer isn't necessarily TCP and an unchecked
    C-style cast is undefined behavior in those cases. (Jon Siwek, Corelight)

  * GH-485: fix cases where DHCP log omits MAC field

    The field is populated in this order of preference:

      (1) Use a client-identifier option sent by client
      (2) Use the server's CHADDR field
      (3) Use the client's CHADDR field

    Case (3) did not exist before this patch. (Jon Siwek, Corelight)

2.6-649 | 2019-07-23 09:37:05 -0700

  * Remove --disable-perftools from coverity builds (Jon Siwek, Corelight)

2.6-647 | 2019-07-22 12:46:04 -0700

  * Ignore abs-path in test. (Zeke Medley, Corelight)

  * Report argument # type check failed on. (Zeke Medley, Corelight)

  * Update test baseline. (Zeke Medley, Corelight)

  * Improve func arg type checking. (Zeke Medley, Corelight)

  * &expire_func(table, arg1, arg2, ...) + type checking. (Zeke Medley, Corelight)

2.6-640 | 2019-07-22 12:22:08 -0700

  * Mark List::insert deprecated in favor of push_front (Tim Wojtulewicz, Corelight)

  * Mark List::sort as deprecated, remove List::sortedinsert (Tim Wojtulewicz, Corelight)

  * Simplify container iterators to just use pointers directly (Tim Wojtulewicz, Corelight)

2.6-635 | 2019-07-22 10:58:56 -0700

  * Add DPD::max_violations option

    This allows one to tune the number of protocol violations to tolerate
    from any given analyzer type before just disabling a given instance
    of it.

    Also removes the "disabled_aids" field from the DPD::Info record
    since it serves no purpose: in this case, calling disable_analyzer
    multiple times for the same analyzer is a no-op. (Jon Siwek, Corelight)

  * Abort when --enable-jemalloc is given, but JeMalloc is not found
    (Johanna Amann, Corelight)

  * GH-475: Require --enable-perftools to link in tcmalloc. (Jon Siwek, Corelight)

2.6-630 | 2019-07-19 18:01:44 -0700

  * Fix typo in NEWS (Jon Siwek, Corelight)

2.6-628 | 2019-07-19 15:48:58 +0000

  * Support old plugins that may still reference "bro-config.h",
    "bro-path-dev", or "bro" inside the build tree through
    compatibility symlinks/wrappers. (Jon Siwek, Corelight)

2.6-626 | 2019-07-18 19:44:31 -0700

  * Remove unused expression type tag: EXPR_MATCH (Jon Siwek, Corelight)

  * Add missing expr_name: vector_coerce (Jon Siwek, Corelight)

2.6-624 | 2019-07-16 16:07:59 +0000

  * Redo README. We now have separate plain text and Markdown versions.
    (Zeke Medley, Corelight)

2.6-616 | 2019-07-16 15:21:37 +0000

  * Fix referecne counting bug in EnumType copy constructor. (Jon
    Siwek, Corelight)

  * Remove unused BroObj::in_ser_cache member. (Jon Siwek, Corelight)

2.6-612 | 2019-07-15 19:46:04 -0700

  * Fix a potential usage of List::remove_nth(-1) (Jon Siwek, Corelight)

  * Change List::remote(const T&) to return a bool (Jon Siwek, Corelight)

    It now indicates whether the removal took place or not, depending
    on whether a matching element was found in the list.

  * Fix debug build due to old int_list usage within assert (Jon Siwek, Corelight)

  * Convert uses of loop_over_list to ranged-for loops (Tim Wojtulewicz, Corelight)

  * Remove loop_over_queue (as an example for later removing loop_over_list) (Tim Wojtulewicz, Corelight)

  * Change int_list in CCL.h to be a vector, fix uses of int_list to match (Tim Wojtulewicz, Corelight)

  * Remove List<> usage from strings.bif (Tim Wojtulewicz, Corelight)

  * Replace uses of the old Queue/PQueue generation code with new template versions (Tim Wojtulewicz, Corelight)

  * Convert BaseQueue/Queue/PQueue into templates, including iterator support (Tim Wojtulewicz, Corelight)

  * Replace uses of the old Dict generation code with new template versions (Tim Wojtulewicz, Corelight)

  * Convert PDict into template (Tim Wojtulewicz, Corelight)

  * Replace uses of the old List generation code with new template versions (Tim Wojtulewicz, Corelight)

  * Convert BaseList/List/PList into templates, including iterator support (Tim Wojtulewicz, Corelight)

2.6-598 | 2019-07-12 18:20:12 -0700

  * Fix canonification of timestamps with a decisecond multiple (Jon Siwek, Corelight)

2.6-597 | 2019-07-12 15:01:56 -0700

  * Fix a couple of resource leaks from JSON changes (Tim Wojtulewicz, Corelight)

2.6-595 | 2019-07-12 13:34:08 -0700

  * GH-157: Mark some attributes as not allowed for global variables (Tim Wojtulewicz, Corelight)

    This disallows &default for global values that are not tables, and &optional for all globals.

  * Fix uncaught exceptions from Val cloning failures (Jon Siwek, Corelight)

2.6-591 | 2019-07-11 13:29:28 -0700

  * Fix potential thread safety issue with zeekenv util function

    Observed segfault accessing the local static std::map of zeekenv() from
    a logging thread, but only in non-debug builds using Apple/Clang
    compiler, not in a debug build or GCC.  Don't quite get this behavior
    since static local variable initialization is supposed to be thread-safe
    since C++11, but moving to a global static works and is "more efficient"
    anyway since there's no longer any run-time overhead. (Jon Siwek, Corelight)

2.6-589 | 2019-07-11 13:14:52 -0700

  * GH-421: fix bugs/regressions in DNP3 analyzer (Hui Lin)

2.6-587 | 2019-07-11 12:13:48 -0700

  * Fix a sign-compare compiler warning (Jon Siwek, Corelight)

2.6-586 | 2019-07-11 11:15:40 -0700

  * Convert all JSON output to use an external library for better consistency (Tim Wojtulewicz, Corelight)

    See NEWS for more details; this makes to_json a bif and causes slight changes in its
    output, as well as the output of the JSON logger.

2.6-576 | 2019-07-10 18:38:54 -0700

  * Remove unused option: chunked_io_buffer_soft_cap (Jon Siwek, Corelight)

2.6-575 | 2019-07-09 18:28:03 -0700

  * Avoid a null dereference (Coverity-1402816) (Tim Wojtulewicz, Corelight)

  * Avoid resource leaks (Coverity-1402818, Coverity-1402812) (Tim Wojtulewicz, Corelight)

  * Avoid null dereference in broker (Coverity-1402824, Coverity-1402814) (Tim Wojtulewicz, Corelight)

  * Improve stability of a unit test (Jon Siwek, Corelight)

2.6-569 | 2019-07-03 13:03:22 -0700

  * Improve stability of a unit test (Jon Siwek, Corelight)

2.6-568 | 2019-07-03 11:50:56 -0700

  * Add clang-tidy rule to CMake including a base configuration (Tim Wojtulewicz, Corelight)

2.6-566 | 2019-07-03 11:08:24 -0700

  * Improve Zeekygen output for long attribute expressions (Jon Siwek, Corelight)

2.6-565 | 2019-07-03 09:32:34 -0700

  * GH-446: Deprecate rfb_event. (Johanna Amann, Corelight)

2.6-563 | 2019-07-03 01:57:40 -0700

  * Fix CIF integration and add logging options to intel.log and added comments to code (sfinlon)

2.6-558 | 2019-07-01 01:27:50 -0700

  * GH-443: fix uses of timestamp 0 in cluster diagnostic logs

    For broker.log and cluster.log: there was a race condition.  A worker's
    first IOSource that it processes is potentially Broker if there were
    no packets available yet and thread scheduling happens to work out
    such that network connections (inside CAF threads) become established
    before we enter the main I/O loop.  Such peering establishments would
    generate logs with timestamp 0 as there was not yet any code path
    taken that would update network_time.

    For reporter.log: any non-worker (packet-processing) node would just
    unnecessarily use a timestamp of 0 for their reporter messages. (Jon Siwek, Corelight)

2.6-556 | 2019-07-01 00:56:13 -0700

  * Improve deprecation warning messages (Jon Siwek, Corelight)

  * Remove deprecated DNS events

    - dns_full_request
    - non_dns_request (Jon Siwek, Corelight)

  * Remove BackDoor analyzer (Jon Siwek, Corelight)

  * Remove InterConn analyzer (Jon Siwek, Corelight)

  * Remove deprecated/unused irc_servers option (Jon Siwek, Corelight)

  * Remove deprecated print_hook event (Jon Siwek, Corelight)

  * Remove dead code: dump_used_event_handlers (Jon Siwek, Corelight)

  * Remove unused software_version_found events

    - software_version_found
    - software_unparsed_version_found
    - software_parse_error (Jon Siwek, Corelight)

  * Remove deprecated open_log_file and log_file_name functions (Jon Siwek, Corelight)

  * Remove deprecated/unused "packet" type (Jon Siwek, Corelight)

  * Un-deprecate anonymizer BIFs (Jon Siwek, Corelight)

  * Un-deprecate file rotation functions

    - rotate_file
    - rotate_file_by_name
    - calc_next_rotate

    These still have use-cases even though no longer used for our logging
    functionality.  E.g. rotate_file_by_name may be used to rotate
    pcap dump files.

    Also the log_rotate_base_time option was marked deprecated, but still
    used in the new logging framework. (Jon Siwek, Corelight)

  * Switch default CAF scheduler policy to work sharing

    It may generally be better for our default use-case, as workers may
    save a few percent cpu utilization as this policy does not have to
    use any polling like the stealing policy does.

    This also helps avoid a potential issue with the implementation of
    spinlocks used in the work-stealing policy in current CAF versions,
    where there's some conditions where lock contention causes a thread
    to spin for long periods without relinquishing the cpu to others. (Jon Siwek, Corelight)

  * Update sqlite to 3.28.0.  (Johanna Amann, Corelight)

  * GH-320: Improve RFB (VNC) protocol parsing

    Parsing now stops for both client and server if either encounters
    any parsing error or invalid state.

    After a complete handshake, server messages are no longer parsed.
    Support for that is incomplete and not sure it's that useful anyway
    since it mostly contains pixel data. (Jon Siwek, Corelight)

2.6-536 | 2019-06-28 12:10:55 -0700

  * Add Windows Minidump file signature (Alexander Bolshakov)

2.6-534 | 2019-06-28 11:48:41 -0700

  * Change notices to be processed on worker. (Johanna Amann, Corelight)

    In the past they were processed on the manager - which requires big
    records to be sent around.

    This has a potential of incompatibilities if someone relied on global
    state for notice processing.

    Also may prevent notice de-duplication due to expected race
    condition of suppression messages taking time to proaogate out
    to all cluster nodes.

2.6-531 | 2019-06-27 12:09:08 -0700

  * GH-375: Remove the BroFile cache (Johanna Amann, Corelight)

2.6-529 | 2019-06-27 10:12:34 -0700

  * Fix creating a StringVal from std::string. (Johanna Amann, Corelight)

    Currently, creating a StringVal from a std::string did not work with
    data that contains \0 characters. This easy fix changes this - and
    should also lead to a small speed increase for code using this
    constructor.

    This obviously means that more data might copied now in some cases that
    were previously cut off at the first 0-byte. Our test-suite did not
    reveal any such cases.

2.6-526 | 2019-06-25 12:45:31 -0700

  * Make a paraglob unit test parallelizable (Jon Siwek, Corelight)

2.6-523 | 2019-06-25 10:38:24 -0700

  * GH-427: improve default ID values shown by Zeekygen

    The default value of an ID is now truly the one used to initialize it,
    unaltered by any subsequent redefs.

    Redefs are now shown separately, along with the expression that
    modifies the ID's value. (Jon Siwek, Corelight)

  * Unbreak build on Linux (Johanna Amann, Corelight)

2.6-519 | 2019-06-24 15:25:08 -0700

  * GH-435: fix null pointer deref in RPC analyzer. (Jon Siwek, Corelight)

2.6-517 | 2019-06-24 15:20:39 -0700

  * Add paraglob, a fairly quick data structure for matching a string against a large list of patterns.
	 (Zeke Medley, Corelight)

  * GH-171: support warning messages alongside deprecated attributes (Tim Wojtulewicz, Corelight)

2.6-503 | 2019-06-21 11:17:58 -0700

  * GH-417: Remove old, unmaintained p0f support. (Johanna Amann, Corelight)

2.6-500 | 2019-06-20 20:54:15 -0700

  * Add new RDP event: rdp_client_cluster_data (Jeff Atkinson)

  * Added "options" field to RDP::ClientChannelDef (Jeff Atkinson)

2.6-494 | 2019-06-20 20:24:38 -0700

  * Renaming src/StateAccess.{h,cc} to src/Notifier.{h,cc}.

    The old names did not reflect the content of the files anymore. (Robin Sommer, Corelight)

  * Remove MutableVal, StateAccess classes, enum Opcode. (Robin Sommer, Corelight)

  * Redo API for notifiers.

    There's now an notifier::Modifiable interface class that class
    supposed to signal modifications are to be derived from. This takes
    the place of the former MutableValue class and also unifies how Val
    and IDs signal modifications. (Robin Sommer, Corelight)

  * Redo NotfifierRegistry to no longer rely on StateAccess.

    We simplify the API to a simple Modified() operation. (Robin Sommer, Corelight)

  * Add new test for when-statement watching global variables. (Robin Sommer, Corelight)

2.6-482 | 2019-06-20 19:57:20 -0700

  * Make configure complain if submodules are not checked out. (Johanna Amann, Corelight)

  * Improve C++ header includes to improve build time (Jon Siwek, Corelight)

2.6-479 | 2019-06-20 18:31:58 -0700

  * Fix TableVal::DoClone to use CloneState cache (Jon Siwek, Corelight)

2.6-478 | 2019-06-20 14:19:11 -0700

  * Remove old Broccoli SSL options (Jon Siwek, Corelight)

    - ssl_ca_certificate
    - ssl_private_key
    - ssl_passphrase

2.6-477 | 2019-06-20 14:00:22 -0700

  * Remove unused SerialInfo.h and SerialTypes.h headers (Jon Siwek, Corelight)

2.6-476 | 2019-06-20 13:23:22 -0700

  * Remove opaque of ocsp_resp. (Johanna Amann, Corelight)

    Only used in one event, without any way to use the opaque for anything
    else. At this point this just seems like a complication that has no
    reason to be there.

  * Remove remnants of event serializer. (Johanna Amann, Corelight)

  * Reimplement serialization infrastructure for OpaqueVals.
    (Robin Sommer, Corelight & Johanna Amann, Corelight)

    We need this to sender through Broker, and we also leverage it for
    cloning opaques. The serialization methods now produce Broker data
    instances directly, and no longer go through the binary formatter.

    Summary of the new API for types derived from OpaqueVal:

        - Add DECLARE_OPAQUE_VALUE(<class>) to the class declaration
        - Add IMPLEMENT_OPAQUE_VALUE(<class>) to the class' implementation file
        - Implement these two methods (which are declated by the 1st macro):
            - broker::data DoSerialize() const
            - bool DoUnserialize(const broker::data& data)

    This machinery should work correctly from dynamic plugins as well.

    OpaqueVal provides a default implementation of DoClone() as well that
    goes through serialization. Derived classes can provide a more
    efficient version if they want.

    The declaration of the "OpaqueVal" class has moved into the header
    file "OpaqueVal.h", along with the new serialization infrastructure.
    This is breaking existing code that relies on the location, but
    because the API is changing anyways that seems fine.

  * Implement a Shallow Clone operation for types. (Johanna Amann, Corelight)

    This is needed to track name changes for the documentation.

  * Remove old serialization infrastrucutre. (Johanna Amann, Corelight)

2.6-454 | 2019-06-19 09:39:06 -0700

  * GH-393: Add slice notation for vectors (Tim Wojtulewicz, Corelight & Jon Siwek, Corelight)

    Example Syntax:

    local v = vector(1, 2, 3, 4, 5);
    v[2:4] = vector(6, 7, 8);        # v is now [1, 2, 6, 7, 8, 5]
    print v[:4];                     # prints [1, 2, 6, 7]

2.6-446 | 2019-06-17 20:26:49 -0700

  * Rename bro to zeek in error messages (Daniel Thayer)

2.6-444 | 2019-06-15 19:09:03 -0700

  * Add/rewrite NTP support (Vlad Grigorescu and Mauro Palumbo)

2.6-416 | 2019-06-14 20:57:57 -0700

  * DNS: Add support for SPF response records (Vlad Grigorescu)

2.6-413 | 2019-06-14 19:51:28 -0700

  * GH-406: rename bro.bif to zeek.bif (Jon Siwek, Corelight)

2.6-412 | 2019-06-14 19:26:21 -0700

  * GH-387: update Broker topic names to use "zeek/" prefix (Jon Siwek, Corelight)

  * GH-323: change builtin plugin namespaces to Zeek (Jon Siwek, Corelight)

2.6-408 | 2019-06-13 11:19:50 -0700

  * Fix potential null-dereference in current_time() (Tim Wojtulewicz, Corelight)

  * Add --sanitizers configure script to enable Clang sanitizers (Tim Wojtulewicz, Corelight)

2.6-404 | 2019-06-12 15:10:19 -0700

  * Rename directories from bro to zeek (Daniel Thayer)

    The new default installation prefix is /usr/local/zeek

2.6-400 | 2019-06-07 20:06:33 -0700

  * Adapt bro_plugin CMake macros to use zeek_plugin (Jon Siwek, Corelight)

2.6-399 | 2019-06-07 14:02:18 -0700

  * Update SSL documentation. (Johanna Amann)

  * Support the newer TLS 1.3 key_share extension. (Johanna Amann)

  * Include all data of the server-hello random (Johanna Amann)

    Before we cut the first 4 bytes, which makes it impossible to recognize
    several newer packets (like the hello retry).

  * Parse TLS 1.3 pre-shared-key extension. (Johanna Amann)


    Adds new events:

    - ssl_extension_pre_shared_key_client_hello
    - ssl_extension_pre_shared_key_server_hello

2.6-391 | 2019-06-07 17:29:28 +1000

  * GH-209: replace "remote_ip" field of radius.log with "tunnel_client".
    Also changes type from addr to string. (Jon Siwek, Corelight)

2.6-389 | 2019-06-06 20:02:19 -0700

  * Update plugin unit tests to use --zeek-dist (Jon Siwek, Corelight)

2.6-388 | 2019-06-06 19:48:55 -0700

  * Change default value of peer_description "zeek" (Jon Siwek, Corelight)

2.6-387 | 2019-06-06 18:51:09 -0700

  * Rename Bro to Zeek in Zeekygen-generated documentation (Jon Siwek, Corelight)

2.6-386 | 2019-06-06 17:17:55 -0700

  * Add new RDP event: rdp_native_encrytped_data (Anthony Kasza, Corelight)

2.6-384 | 2019-06-06 16:49:14 -0700

  * Add new RDP event: rdp_client_security_data (Jeff Atkinson)

2.6-379 | 2019-06-06 11:56:58 -0700

  * Improve sqlite logging unit tests (Jon Siwek, Corelight)

2.6-378 | 2019-06-05 16:23:04 -0700

  * Rename BRO_DEPRECATED macro to ZEEK_DEPRECATED (Jon Siwek, Corelight)

2.6-377 | 2019-06-05 16:15:58 -0700

  * Deprecate functions with "bro" in them. (Jon Siwek, Corelight)

    * "bro_is_terminating" is now "zeek_is_terminating"

    * "bro_version" is now "zeek_version"

    The old functions still exist for now, but are deprecated.

2.6-376 | 2019-06-05 13:29:57 -0700

  * GH-379: move catch-and-release and unified2 scripts to policy/ (Jon Siwek, Corelight)

    These are no longer loaded by default due to the performance impact they
    cause simply by being loaded (they have event handlers for commonly
    generated events) and they aren't generally useful enough to justify it.

2.6-375 | 2019-06-04 19:28:06 -0700

  * Simplify threading::Value destructor (Jon Siwek, Corelight)

  * Add pattern support to input framework. (Zeke Medley, Corelight)

2.6-369 | 2019-06-04 17:53:10 -0700

  * GH-155: Improve coercion of expression lists to vector types (Tim Wojtulewicz, Corelight)

  * GH-159: Allow coercion of numeric record field values to other types (Tim Wojtulewicz, Corelight)

  * Allow passing a location to BroObj::Warning and BroObj::Error. (Tim Wojtulewicz, Corelight)

    This allows callers (such as check_and_promote) to pass an expression
    location to be logged if the location doesn't exist in the value being
    promoted.

  * Add CLion directories to gitignore (Tim Wojtulewicz, Corelight)

  * Move #define outside of max_type for clarity (Tim Wojtulewicz, Corelight)

2.6-361 | 2019-06-04 10:30:21 -0700

  * GH-293: Protect copy() against reference cycles. (Robin Sommer, Corelight)

    Reference cycles shouldn't occur but there's nothing really preventing
    people from creating them, so may just as well be safe and deal with
    them when cloning values.

2.6-359 | 2019-05-31 13:37:17 -0700

  * Remove old documentation reference to rotate_interval (Jon Siwek, Corelight)

2.6-357 | 2019-05-30 10:57:54 -0700

  * Tweak to ASCII reader warning suppression (Christian Kreibich, Corelight)

    Warnings in the ASCII reader so far remained suppressed even
    when an input file changed. It's helpful to learn about problems
    in the data when putting in place new data files, so this change
    maintains the existing warning suppression while processing a file,
    but re-enables warnings after updates to a file.

2.6-354 | 2019-05-29 09:46:19 -0700

  * Add weird: "RDP_channels_requested_exceeds_max" (Vlad Grigorescu)

2.6-352 | 2019-05-28 17:57:36 -0700

  * Reduce data copying in Broker message processing (Jon Siwek, Corelight)

  * Improve Broker I/O loop integration: less mutex locking (Jon Siwek, Corelight)

    Checking a subscriber for available messages required locking a mutex,
    but we should never actually need to do that in the main-loop to check
    for Broker readiness since we can rely on file descriptor polling.

  * Improve processing of broker data store responses (Jon Siwek, Corelight)

    Now retrieves and processes all N available responses at once instead
    of one-by-one-until-empty.

2.6-345 | 2019-05-28 11:32:16 -0700

  * RDP: Add parsing and logging of channels requested by the client. (Vlad Grigorescu)

    Can determine capabilities requested by the client, as well as attacks such
    as CVE-2019-0708.

2.6-342 | 2019-05-28 10:48:37 -0700

  * GH-168: Improve type-checking for table/set list assignment. (Zeke Medley and Jon Siwek, Corelight)

2.6-340 | 2019-05-24 18:02:43 -0700

  * Add support for parsing additional DHCP options (Jay Wren)

    The following optional fields were added to the DHCP::Options record:

      - time_offset  (Option 2)
      - time_servers (Option 4)
      - name_servers (Option 5)
      - ntp_servers  (Option 42)

2.6-338 | 2019-05-24 17:06:08 -0700

  * Add input file name to additional ASCII reader warning messages (Christian Kreibich, Corelight)

2.6-336 | 2019-05-24 10:23:20 -0700

  * GH-378: check validity of missing 'val' field in Input::add_table (Jon Siwek, Corelight)

2.6-335 | 2019-05-24 08:58:59 -0700

  * Fix memory leak when no protocol_violation event handler exists (Jon Siwek, Corelight)

2.6-334 | 2019-05-23 20:40:03 -0700

  * Add an internal getenv wrapper function: zeekenv (Jon Siwek, Corelight)

    It maps newer environment variable names starting with ZEEK to the
    legacy names starting with BRO.

  * Rename all BRO-prefixed environment variables (Daniel Thayer)

    For backward compatibility when reading values, we first check
    the ZEEK-prefixed value, and if not set, then check the corresponding
    BRO-prefixed value.

2.6-331 | 2019-05-23 18:03:42 -0700

  * Update broker unit test output. (Jon Siwek, Corelight)

    Due to string representation of Broker vectors changing (they now
    use parentheses instead of square brackets).

2.6-330 | 2019-05-23 13:04:26 -0700

  * GH-173: Support ranges of values for value_list elements in the signature parser
    (Tim Wojtulewicz, Corelight)

  * GH-173: Modify the signature parser so ID components can't start with numbers
    (Tim Wojtulewicz, Corelight)

2.6-327 | 2019-05-23 11:56:11 -0700

  * Remove redundant RecordVal::record_type member (Jon Siwek, Corelight)

2.6-326 | 2019-05-23 10:49:38 -0700

  * Fix parse-time RecordVal tracking containing duplicates (Jon Siwek, Corelight)

2.6-325 | 2019-05-22 23:56:23 -0700

  * Add leak-checks for new copy operations (Johanna Amann, Corelight)

  * Finish implementation of new copy method. (Johanna Amann, Corelight)

    All types (besides EntropyVal) now support a native copy operation,
    which uses primitives of the underlying datatypes to perform a quick
    copy, without serialization.

    EntropyVal is the one exception - since that type is rather complex
    (many members) and will probably not be copied a lot, if at all, it
    makes sense to just use the serialization function.

    This will have to be slightly re-written in the near-term-future to use
    the new serialization function for that opaque type.

    This change also introduces a new x509_from_der bif, which allows to
    parse a der into an opaque of x509.

    This change removes the d2i_X509_ wrapper function; this was a remnant
    when d2i_X509 took non-const arguments. We directly use d2i_X509 at
    several places assuming const-ness, so there does not seem to ba a
    reason to keep the wrapper.

    This change also exposed a problem in the File cache - cases in which an
    object was brought back into the cache, and writing occurred in the
    file_open event were never correctly handeled as far as I can tell.

  * Reimplement copy(). (Robin Sommer, Corelight)

    The old implementation used the serialization framework, which is
    going away. This is a new standalone implementation that should also
    be quite a bit faster.

2.6-318 | 2019-05-21 09:17:53 -0700

  * Remove state_dir and state_write_delay options (Jon Siwek, Corelight)

  * Remove a reference to &synchronized from docs (Jon Siwek, Corelight)

2.6-316 | 2019-05-20 20:56:46 -0700

  * Additional Bro to Zeek renaming (Daniel Thayer)

  * Added a new unit test for legacy Bro Plugins (Daniel Thayer)

  * Added a symlink bro-path-dev.in for use by legacy Bro packages (Daniel Thayer)

2.6-314 | 2019-05-20 16:20:33 -0700

  * Remove deprecated attributes. (Johanna Amann, Corelight)
    To be more exact: &encrypt, &mergeable, &rotate_interval, &rotate_size

    Also removes no longer used redef-able constants:
    log_rotate_interval, log_max_size, log_encryption_key

2.6-311 | 2019-05-20 09:07:58 -0700

  * Add missing &optional attr to KRB record fields; also add existence
    checks to scripts (Jon Siwek, Corelight).

2.6-308 | 2019-05-17 14:13:46 -0700

  * Always emit scripting errors to stderr during zeek_init (Jon Siwek, Corelight)

2.6-307 | 2019-05-16 13:37:24 -0700

  * More bro-to-zeek renaming in scripts and other files (Daniel Thayer)

  * More bro-to-zeek renaming in the unit tests (Daniel Thayer)

2.6-303 | 2019-05-15 15:03:11 -0700

  * Changes needed due to bro-to-zeek renaming in broker (Daniel Thayer)

2.6-301 | 2019-05-15 10:05:53 -0700

  * Fix potential race in openflow broker plugin (Jon Siwek, Corelight)

2.6-300 | 2019-05-15 09:00:57 -0700

  * Fixes to DNS lookup, including ref-counting bugs, preventing starvation
    of the DNS_Mgr in the I/O loop, dead code removal, and a fix that
    prevents the timeout of already resolved DNS lookups (Jon Siwek, Corelight)

2.6-292 | 2019-05-14 19:01:05 -0700

  * Fix maybe-uninitialized compiler warning (Jon Siwek, Corelight)

2.6-290 | 2019-05-14 18:35:25 -0700

  * Update btest.cfg path to use zeek-aux (Jon Siwek, Corelight)

2.6-288 | 2019-05-14 17:47:55 -0700

  * Update CMake to use aux/zeekctl and aux/zeek-aux submodules (Jon Siwek, Corelight)

2.6-287 | 2019-05-14 17:40:40 -0700

  * Rename broctl submodule to zeekctl (Jon Siwek, Corelight)

2.6-286 | 2019-05-14 13:19:12 -0700

  * Undo an unintentional change to btest.cfg from a recent commit (Daniel Thayer)

  * Fix zeek-wrapper and improve error messages (Daniel Thayer)

    The script was not passing command-line arguments to the new program.

  * Update for renaming BroControl to ZeekControl. (Robin Sommer, Corelight)

  * GH-239: Rename bro to zeek, bro-config to zeek-config, and bro-path-dev to zeek-path-dev.
    (Robin Sommer, Corelight)

    This also installs symlinks from "zeek" and "bro-config" to a wrapper
    script that prints a deprecation warning.

2.6-279 | 2019-05-13 20:02:59 -0700

  * GH-365: improve un-indexable type error message (Jon Siwek, Corelight)

2.6-277 | 2019-05-08 12:42:18 -0700

  * Allow tuning Broker log batching via scripts (Jon Siwek, Corelight)

    Via redefining "Broker::log_batch_size" or "Broker::log_batch_interval"

2.6-276 | 2019-05-08 09:03:27 -0700

  * Force the Broker IOSource to idle periodically, preventing packet
	  IOSource starvation. (Jon Siwek, Corelight).

2.6-274 | 2019-05-08 08:58:25 -0700

  * GH-353: Add `/<re>/i` case-insensitive signature syntax (Jon Siwek, Corelight)

2.6-272 | 2019-05-06 18:43:13 -0700

  * Remove support for using && and || with patterns. (Johanna Amann, Corelight)

    This was never documented and previously deprecated.

  * Remove RemoteSerializer and related code/types. (Johanna Amann, Corelight)

    Also removes broccoli from the source tree.

  * Remove PersistenceSerializer. (Johanna Amann, Corelight)

  * Remove &synchronized and &persistent attributes. (Johanna Amann, Corelight)

2.6-264 | 2019-05-03 11:16:38 -0700

  * Fix sporadic openflow/broker test failure (Jon Siwek, Corelight)

2.6-263 | 2019-05-02 22:49:40 -0700

  * Install local.zeek as symlink to pre-existing local.bro (Jon Siwek, Corelight)

    This a convenience for those that are upgrading.  If we didn't do
    this, then deployments can silently break until the user intervenes
    since BroControl now prefers to load the initially-vanilla local.zeek
    instead of the formerly-customized local.bro.

2.6-262 | 2019-05-02 21:39:01 -0700

  * Rename Zeexygen to Zeekygen (Jon Siwek, Corelight)

2.6-261 | 2019-05-02 20:49:23 -0700

  * Remove previously deprecated policy/protocols/smb/__load__ (Jon Siwek, Corelight)

2.6-260 | 2019-05-02 19:16:48 -0700

  * GH-243: Remove deprecated functions/events from 2.6 and earlier (Johanna Amann, Corelight)

2.6-258 | 2019-05-02 12:26:54 -0700

  * GH-340: Improve IPv4/IPv6 regexes, extraction, and validity functions.

    is_valid_ip() is not a BIF, the IP regular expressions are improved and
    extract_ip_addresses should give better results due to this.
    (Jon Siwek, Corelight)

2.6-255 | 2019-05-01 08:38:49 -0700

  * Add methods to queue events without handler existence check

    Added ConnectionEventFast() and QueueEventFast() methods to avoid
    redundant event handler existence checks.

    It's common practice for caller to already check for event handler
    existence before doing all the work of constructing the arguments, so
    it's desirable to not have to check for existence again.

    E.g. going through ConnectionEvent() means 3 existence checks:
    one you do yourself before calling it, one in ConnectionEvent(), and then
    another in QueueEvent().

    The existence check itself can be more than a few operations sometimes
    as it needs to check a few flags that determine if it's enabled, has
    a local body, or has any remote receivers in the old comm. system or
    has been flagged as something to publish in the new comm. system. (Jon Siwek, Corelight)

  * Cleanup/improve PList usage and Event API

    Majority of PLists are now created as automatic/stack objects,
    rather than on heap and initialized either with the known-capacity
    reserved upfront or directly from an initializer_list (so there's no
    wasted slack in the memory that gets allocated for lists containing
    a fixed/known number of elements).

    Added versions of the ConnectionEvent/QueueEvent methods that take
    a val_list by value.

    Added a move ctor/assign-operator to Plists to allow passing them
    around without having to copy the underlying array of pointers. (Jon Siwek, Corelight)

2.6-250 | 2019-04-29 18:09:29 -0700

  * Remove 'dns_resolver' option, replace w/ ZEEK_DNS_RESOLVER env. var. (Jon Siwek, Corelight)

2.6-249 | 2019-04-26 19:26:44 -0700

  * Fix parsing of hybrid IPv6-IPv4 addr literals with no zero compression (Jon Siwek, Corelight)

2.6-246 | 2019-04-25 10:22:11 -0700

  * Add Zeexygen cross-reference links for some events (Jon Siwek, Corelight)

2.6-245 | 2019-04-23 18:42:02 -0700

  * Expose TCP analyzer utility functions to derived classes (Vern Paxson, Corelight)

2.6-243 | 2019-04-22 19:42:52 -0700

  * GH-234: rename Broxygen to Zeexygen along with roles/directives (Jon Siwek, Corelight)

    * All "Broxygen" usages have been replaced in
      code, documentation, filenames, etc.

    * Sphinx roles/directives like ":bro:see" are now ":zeek:see"

    * The "--broxygen" command-line option is now "--zeexygen"

2.6-242 | 2019-04-22 22:43:09 +0200

  * update SSL consts from TLS 1.3 (Johanna Amann)

2.6-241 | 2019-04-22 12:38:06 -0700

  * Add 'g' character to conn.log history field to flag content gaps (Vern Paxson, Corelight)

    There's also a small change to TCP state machine that distrusts ACKs
    appearing at the end of connections (in FIN or RST) such that they won't
    count towards revealing a true content gap.

2.6-237 | 2019-04-19 12:00:37 -0700

  * GH-236: Add zeek_script_loaded event, deprecate bro_script_loaded (Jon Siwek, Corelight)

    Existing handlers for bro_script_loaded automatically alias to the new
    zeek_script_loaded event, but emit a deprecation warning.

2.6-236 | 2019-04-19 11:16:35 -0700

  * Add zeek_init/zeek_done events and deprecate bro_init/bro_done (Seth Hall, Corelight)

    Any existing handlers for bro_init and bro_done will automatically alias
    to the new zeek_init and zeek_done events such that code will not break,
    but will emit a deprecation warning.

2.6-232 | 2019-04-18 09:34:13 +0200

  * Prevent topk_merge from crashing when second argument is empty set (Jeff Barber)

2.6-230 | 2019-04-17 16:44:16 -0700

  * Fix unit test failures on case-insensitive file systems (Jon Siwek, Corelight)

2.6-227 | 2019-04-16 17:44:31 -0700

  * GH-237: add `@load foo.bro` -> foo.zeek fallback (Jon Siwek, Corelight)

    When failing to locate a script with explicit .bro suffix, check for
    whether one with a .zeek suffix exists and use it instead.

2.6-225 | 2019-04-16 16:07:49 -0700

  * Use .zeek file suffix in unit tests (Jon Siwek, Corelight)

2.6-223 | 2019-04-16 11:56:00 -0700

  * Update tests and baselines due to renaming all scripts (Daniel Thayer)

  * Rename all scripts to have ".zeek" file extension (Daniel Thayer)

  * Add test cases to verify new file extension is recognized (Daniel Thayer)

  * Fix the core/load-duplicates.bro test (Daniel Thayer)

  * Update script search logic for new .zeek file extension (Daniel Thayer)

    When searching for script files, look for both the new and old file
    extensions.  If a file with ".zeek" can't be found, then search for
    a file with ".bro" as a fallback.

  * Remove unnecessary ".bro" from @load directives (Daniel Thayer)

2.6-212 | 2019-04-12 10:12:31 -0700

  * smb2_write_response event added (Mauro Palumbo)

2.6-210 | 2019-04-10 09:54:27 -0700

  * Add options to tune BinPAC flowbuffer policy (Jon Siwek, Corelight)

2.6-208 | 2019-04-10 11:36:17 +0000

  * Improve PE file analysis (Jon Siwek, Corelight)

  * Set PE analyzer CMake dependencies correctly (Jon Siwek, Corelight)

2.6-205 | 2019-04-05 17:06:26 -0700

  * Add script to update external test repo commit pointers (Jon Siwek, Corelight)

2.6-203 | 2019-04-04 16:35:52 -0700

  * Update DTLS error handling (Johanna Amann, Corelight)

    - Adds tuning options: SSL::dtls_max_version_errors and
      SSL::dtls_max_reported_version_errors

2.6-200 | 2019-04-03 09:44:53 -0700

  * Fix reporter net_weird API usage for unknown_mobility_type
    (Jon Siwek, Corelight)

  * Remove variable content from weird names

    This changes many weird names to move non-static content from the
    weird name into the "addl" field to help ensure the total number of
    weird names is reasonably bounded.  Note the net_weird and flow_weird
    events do not have an "addl" parameter, so information may no longer
    be available in those cases -- to make it available again we'd need
    to either (1) define new events that contain such a parameter, or
    (2) change net_weird/flow_weird event signature (which is a breaking
    change for user-code at the moment).

    Also, the generic handling of binpac exceptions for analyzers which
    to not otherwise catch and handle them has been changed from a Weird
    to a ProtocolViolation.

    Finally, a new "file_weird" event has been added for reporting
    weirdness found during file analysis. (Jon Siwek, Corelight)

2.6-197 | 2019-04-03 09:08:58 -0700

  * Make Syslog analyzer accept non-conformant messages that omit Priority.
    (Jon Siwek, Corelight)

2.6-195 | 2019-03-27 12:36:34 -0700

  * Reduce weird-stats overhead (Justin Azoff, Corelight)

2.6-193 | 2019-03-27 10:53:01 -0700

  * Update now-broken Broker API usages (Jon Siwek, Corelight)

    Related to https://github.com/zeek/broker/pull/38, see Broker's NEWS file
    for C++ code migration hints.

2.6-192 | 2019-03-25 17:49:18 -0700

  * Deprecate str_shell_escape, add safe_shell_quote replacement (Jon Siwek, Corelight)

2.6-191 | 2019-03-25 16:43:10 -0700

  * Add support for SMB filenames to the intel framework (Stephen Hosom)

2.6-186 | 2019-03-25 09:41:57 -0700

  * Added policy script for intel removal. (Jan Grashoefer)

  * Added Intel::filter_item hook to filter intelligence items. (Jan Grashoefer)

2.6-178 | 2019-03-21 14:10:44 -0700

  * Add support for parsing SMB 3.1.1 NegotiateContextList response values (Mauro Palumbo)

2.6-175 | 2019-03-20 19:25:11 -0700

  * Parse SMB2 TRANSFORM_HEADER messages and generate new smb2_transform_header event (Mauro Palumbo)

2.6-172 | 2019-03-20 17:59:30 -0700

  * Fix smb_files.log missing FUID field in read/write actions (Mauro Palumbo)

2.6-169 | 2019-03-19 19:12:47 -0700

  * Add support for NFLOG link-layer type (Ryan Denniston)

2.6-167 | 2019-03-18 13:58:28 -0700

  * GH-307: Build binpac as a shared lib, not static by default (Jon Siwek, Corelight)

2.6-166 | 2019-03-18 11:45:35 -0700

  * Add source file path control options for Input and Intel frameworks (Christian Kreibich, Corelight)

    This introduces the following redefinable string constants, empty by
    default:

    - InputAscii::path_prefix
    - InputBinary::path_prefix
    - Intel::path_prefix

2.6-164 | 2019-03-15 19:45:48 -0700

  * Migrate table-based for-loops to key-value iteration (Jon Siwek, Corelight)

  * GH-154: Extend for-loops to allow iteration over a table's key-value pairs (Zeke Medley)

2.6-161 | 2019-03-15 12:59:31 -0700

  * Fix SSH remote_location geo-data not being logged for successful authNs. (Michael Dopheide)

2.6-159 | 2019-03-14 16:39:52 -0700

  * Move NEWS file back into main repo from zeek-docs (Jon Siwek, Corelight)

2.6-158 | 2019-03-14 16:23:30 -0700

  * Fix signed/unsigned comparison compiler warning (Jon Siwek, Corelight)

2.6-157 | 2019-03-14 16:18:13 +0000

  * GH-250: Add VXLAN decapsulation support (Henrik Lund Kramshoej; Jon Siwek, Corelight)

    Zeek now automatically decapsulates VXLAN traffic on UDP port
    4789. It will log such sessions as Tunnel::VXLAN in tunnel.log and
    proceed to analyze the inner payload. Two options allow to tune
    the analysis:

        * "Tunnel::vxlan_ports" allows to tune the set of VXLAN ports
          to analyze/decapsulate.

        * "Tunnel::validate_vxlan_checksums" allows for tuning of how
          checksums associated with the outer UDP header of a possible
          VXLAN tunnel are handled.

    A new "vxlan_packet" event also provides per-packet access to
    VXLAN traffic.

2.6-154 | 2019-03-13 17:28:26 -0700

  * Decrease memory usage via deferred list/dict initialization (Justin Azoff, Corelight)

2.6-152 | 2019-03-13 13:46:17 -0700

  * Add field to the default http.log for the Origin header (Nate Guagenti)

2.6-149 | 2019-03-13 18:21:59 +0000

  * GH-289: Add options to limit entries in http.log file fields. The
    "orig_fuids", "orig_filenames", "orig_mime_types" http.log fields
    as well as their "resp" counterparts are now limited to having
    "HTTP::max_files_orig" or "HTTP::max_files_resp" entries, which
    are 15 by default.  The limit can also be ignored case-by-case via
    the "HTTP::max_files_policy" hook. (Jon Siwek, Corelight)

  * GH-282: Remove JSON formatter's range restriction on numbers. It
    now produces numbers as large as is required to match the data it
    needs to represent. (Jon Siwek, Corelight)

  * GH-281: Improve parsing of Google Pixel user agent. (Jon Siwek,
    Corelight)

  * GH-286: Check for record type mismatch in ternary operator. (Jon
    Siwek, Corelight)

2.6-141 | 2019-03-08 18:36:25 -0800

  * Improve DNS query queuing logic (Jon Siwek, Corelight)

2.6-140 | 2019-03-08 16:21:42 -0800

  * Improve performance of DNS policy scripts (Justin Azoff, Corelight)

2.6-135 | 2019-03-07 13:14:00 -0800

  * Fix typos in dnp3-protocol.pac (g0nzu1)

2.6-132 | 2019-03-06 15:30:58 -0800

  * GH-219: revert a breaking change to |x| operator for interval/time (Jon Siwek, Corelight)

2.6-130 | 2019-02-22 14:56:41 -0600

  * Make input framework parse whitespace around various data types. (Johanna Amann, Corelight)

2.6-128 | 2019-02-22 14:32:48 -0600

  * Add missing libkrb5 include dir to CMake config (Jon Siwek, Corelight)

2.6-127 | 2019-02-15 17:51:51 -0600

  * Skip autogenerated doc coverage test for Travis pull requests (Jon Siwek, Corelight)

  * Add rstrip and lstrip BIFs (Zeke Medley)

  * Improve format of conn_state docs (Jon Siwek, Corelight)

2.6-117 | 2019-02-13 16:14:50 -0800

  * Improve format of conn_state docs (Jon Siwek, Corelight)

2.6-116 | 2019-02-07 10:32:01 -0600

  * GH-208: change invalid subnet expressions to a runtime error (Jon Siwek, Corelight)

  * GH-211: improve consistency of how scripting errors are handled (Jon Siwek, Corelight)

    Scripting errors/mistakes now consistently generate a runtime error
    which have the behavior of unwinding the call stack all the way out of
    the current event handler.

    This also changes the behavior of the startup/initialization process
    to abort if there's errors during bro_init() rather than continue on
    to the main run loop.

2.6-113 | 2019-02-06 13:17:39 -0600

  * Add validity checking/warnings for Broker messages (Jon Siwek, Corelight)

  * Fix crash when using debug.log. (Johanna Amann, Corelight)

2.6-111 | 2019-01-29 18:17:35 -0600

  * Fix memory leak due to enum type/val circular references (Jon Siwek, Corelight)

2.6-110 | 2019-01-29 14:49:10 -0800

  * Add fuid to SSL:Invalid_Server_Cert notice (Stephen Hosom)

2.6-108 | 2019-01-28 14:11:19 -0600

  * GH-210: improve call stack tracking w/ argument info (Jon Siwek, Corelight)

2.6-106 | 2019-01-24 17:53:03 -0600

  * Fix building with LibreSSL again (Jon Siwek, Corelight)

2.6-105 | 2019-01-24 15:22:31 -0800

  * GH-167: improve error message for unclosed function at EOF (Jon Siwek, Corelight)

2.6-103 | 2019-01-24 17:09:05 -0600

  * Change digest.h functions to use EVP_MD_CTX interface (Johanna Amann)

  * Improve support for FIPS systems (Robert Clark)

2.6-98 | 2019-01-24 12:52:18 -0800

  * Added ERSPAN III testing (Stu H)

2.6-95 | 2019-01-23 09:49:35 -0800

  * GH-219: fix |x| operator int overflow / floating point type inconsistency
    (Jon Siwek, Corelight)

2.6-92 | 2019-01-22 08:53:36 -0800

  * GH-151: fix hash calculation for nested sets

    Hash key construction of nested sets depended on the order in
    which their elements are iterated, which varied even between sets
    containing equivalent elements.  The iteration order is now sorted
    by each element's hash value (or, on collision, by full key) such
    that equivalent sets no longer hash differently. (Jon Siwek, Corelight)

2.6-89 | 2019-01-18 15:17:34 -0800

  * Pre-allocate and re-use Vals for bool, int, count, enum and empty string (Jon Siwek, Corelight)

  * Preallocate booleans and small counts < 4096 (Justin Azoff, Corelight)

2.6-86 | 2019-01-17 18:03:10 -0600

  * Improve ERSPAN Type III support (Jon Siwek, Corelight)

  * Implement ERSPAN type II and ERSPAN type III support (Stu H)

2.6-82 | 2019-01-17 14:09:29 -0600

  * Change doc/ subdir into a git submodule (Jon Siwek, Corelight)

    The docs now live at https://github.com/zeek/zeek-docs

2.6-81 | 2019-01-16 19:03:07 -0600

  * Add Broker::peer_counts_as_iosource option (Jon Siwek, Corelight)

2.6-80 | 2019-01-16 11:14:47 -0600

  * Patch to recognized the Revoked bit in DNSKEY Flag (Fatema BW)

2.6-77 | 2019-01-15 14:24:55 -0600

  * GH-170: fix segfault triggered by invalid pattern symbols (Jon Siwek, Corelight)

2.6-76 | 2019-01-15 12:12:09 -0600

  * GH-172: fix broxygen not merging bif and script identifier comments (Jon Siwek, Corelight)

2.6-75 | 2019-01-15 10:30:06 -0600

  * GH-213: change type of vector for-loop index to a count (Jon Siwek, Corelight)

2.6-71 | 2019-01-14 16:11:58 -0600

  * GH-205: prioritize use of sigaction() over sigset() (Jon Siwek, Corelight)

2.6-70 | 2019-01-14 15:34:18 -0600

  * GH-188: fix crash when shutting down with pending reporter errors(Jon Siwek, Corelight)

2.6-69 | 2019-01-14 14:49:49 -0600

  * Fix compiler warning in DNS analyzer (Jon Siwek, Corelight)

2.6-68 | 2019-01-14 14:18:46 -0600

  * GH-162: fix segfault when &expire_func is missing a return value (Jon Siwek, Corelight)

2.6-67 | 2019-01-14 14:01:00 -0600

  * GH-161: fix segfault in &default type checking for sets (Jon Siwek, Corelight)

2.6-66 | 2019-01-14 10:26:47 -0600

  * Fix performance issue due to variable reuse in table expiration (Justin Azoff, Corelight)

2.6-62 | 2019-01-10 15:45:04 -0600

  * Reorganize documentation index (Jon Siwek, Corelight)

2.6-61 | 2019-01-10 13:40:04 -0600

  * Add RTD yaml config file (Jon Siwek, Corelight)

  * Remove some Bro usages in main TOC entries (Jon Siwek, Corelight)

  * Remove "contents" Sphinx directive usages (Jon Siwek, Corelight)

  * Add a `make livehtml` target (Jon Siwek, Corelight)

  * Use sourcecode Sphinx directive more widely (Jon Siwek, Corelight)

  * Use Sphinx RTD theme for user manual (Jon Siwek, Corelight)

  * Remove unused Sphinx extensions (Jon Siwek, Corelight)

  * Remove broxygen Sphinx integration (Jon Siwek, Corelight)

  * Remove Sphinx btest integrations and tests (Jon Siwek, Corelight)

2.6-46 | 2019-01-10 09:10:08 -0800

  * improve performance of catch and release script (Justin Azoff, Corelight)

2.6-43 | 2019-01-07 09:50:43 -0800

  * GH-227: Improve LibreSSL support (Jon Siwek, Corelight)

2.6-41 | 2019-01-04 17:50:00 -0600

  * Replace some bro.org usages with zeek.org (Jon Siwek, Corelight)

2.6-39 | 2019-01-02 11:26:27 -0600

  * Add BIF: Reporter::fatal_error_with_core (Stephen Hosom)

2.6-27 | 2018-12-10 11:53:41 -0600

  * GH-216: Add FTS dependency when building on Alpine (Jon Siwek, Corelight)

  * Remove unnecessary header include (Jon Siwek, Corelight)

  * GH-216: Improve default DNS resolution support for Alpine/musl (Jon Siwek, Corelight)

    /etc/resolv.conf now gets parsed for the first IPv4 nameserver that works.

  * Add dns_resolver option (Jon Siwek, Corelight)

2.6-22 | 2018-12-10 11:16:53 -0600

  * Introduce --enable-static-broker configuration option. (Johanna Amann)

    This option builds the bundled broker (and caf) statically.

2.6-20 | 2018-12-07 16:36:35 -0600

  * Update github/download links (Jon Siwek, Corelight)

2.6-19 | 2018-12-07 07:26:51 -0600

  * Fix Travis git clone command (Jon Siwek, Corelight)

2.6-18 | 2018-12-06 20:11:01 -0600

  * Update external test suite locations (Jon Siwek, Corelight)

  * Update submodules to use github.com/zeek (Jon Siwek, Corelight)

2.6-16 | 2018-11-29 17:05:44 -0600

  * Parallelize communication tests using btest TEST-PORT (Jon Siwek, Corelight)

2.6-14 | 2018-11-29 16:27:38 -0600

  * Improve introspection of Record and TypeType values (Jon Siwek, Corelight)

    * TypeType values are now printable and yield the type name/alias
    * Fix record_fields BIF to return correct type name for fields
    * Allow TypeType values that point to a RecordType to be used with
      record_fields BIF

  * Bro plugins should support a patch version (x.y.z) (Jon Zeolla)

  * GH-148: add priority to DNSSEC event handlers (Jon Siwek, Corelight)

  * DNSSEC support (Fatema Bannat Wala)

2.6 | 2018-11-29 10:03:33 -0600

  * Release 2.6.

2.6-beta3-2 | 2018-11-22 07:56:17 -0600

  * GH-218: Add missing ICMP router advertisement counterpart (Jon Siwek, Corelight)

2.6-beta3 | 2018-11-14 17:09:42 -0600

  * Release 2.6-beta3

2.6-beta2-83 | 2018-11-08 12:25:21 -0600

  * Fix SumStats "last" plugin in cluster mode (Jon Siwek, Corelight)

2.6-beta2-82 | 2018-11-08 09:38:52 -0600

  * Remove unnecessary Bloom filter empty check (Matthias Vallentin)

2.6-beta2-80 | 2018-11-07 11:46:34 -0600

  * Support appending to vector of any (Jon Siwek, Corelight)

2.6-beta2-79 | 2018-11-07 10:27:00 -0600

  * Fix coding conventions nits/typos (Vern Paxson, Corelight)

2.6-beta2-77 | 2018-11-06 09:32:17 -0600

  * Switch GridFTP options from redef to option (Vlad Grigorescu)

  * Improve error handling in x509_ocsp_verify function (Jon Siwek, Corelight)

2.6-beta2-68 | 2018-11-02 18:30:01 -0500

  * Fix a unit test relying on a bash-ism (Jon Siwek, Corelight)

2.6-beta2-67 | 2018-11-02 17:41:46 -0500

  * Add script-layer call stack to internal errors messages that abort (Jon Siwek, Corelight)

  * Improve error message of index assignment expression failures (Jon Siwek, Corelight)

2.6-beta2-65 | 2018-11-02 09:36:30 -0500

  * Improve Travis script to show multiple core dump stacks (Jon Siwek, Corelight)

2.6-beta2-64 | 2018-11-02 08:56:59 -0500

  * Improve a weird stats unit test (Jon Siwek, Corelight)

2.6-beta2-62 | 2018-11-01 20:39:07 -0500

  * Fix Travis script typo (Jon Siwek, Corelight)

2.6-beta2-61 | 2018-11-01 19:57:32 -0500

  * Add more debug output to Travis script (Jon Siwek, Corelight)

2.6-beta2-60 | 2018-11-01 18:35:26 -0500

  * Add core file search and stack trace output for Travis builds (Jon Siwek, Corelight)

  * Update license year for 2018 (Vlad Grigorescu)

2.6-beta2-57 | 2018-10-31 22:26:24 -0500

  * GH-199: change `bro --help` exit status from 1 to 0 (Jon Siwek, Corelight)

2.6-beta2-55 | 2018-10-30 09:59:44 -0500

  * Add a test with an encrypted MySQL connection (Vlad Grigorescu)

  * Fix parsing of MySQL NUL Strings (Vlad Grigorescu)

2.6-beta2-51 | 2018-10-26 10:41:42 -0500

  * Add missing record field comment (Jon Siwek, Corelight)

2.6-beta2-50 | 2018-10-26 10:23:57 -0500

  * Add missing record field comments (Jon Siwek, Corelight)

2.6-beta2-49 | 2018-10-25 18:56:02 -0500

  * Fix minor documentation mistakes (Jon Siwek, Corelight)

2.6-beta2-46 | 2018-10-23 13:01:28 -0500

  * GH-192: Generate ssh_auth_attempted for the 'none' authentication method.
    (Vlad Grigorescu)

2.6-beta2-43 | 2018-10-19 11:15:44 -0500

  * Improve scripts/base/utils/dir unit test (Jon Siwek, Corelight)

2.6-beta2-42 | 2018-10-18 10:21:01 -0500

  * Fix documentation link for notice_alarm.log fields (Jon Siwek, Corelight)

2.6-beta2-40 | 2018-10-16 15:37:49 -0500

  * Change DNP3::function_codes name for request 0x21 (Dale Lakes)

  * Fix resource record type names in DNS::query_types for 41 and 100 (Dale Lakes)

  * Add missing DNS resource record types to DNS::query_types (Dale Lakes)

  * Refactor DCE_RPC constants to be specified in hex instead of decimal (Dale Lakes)

2.6-beta2-35 | 2018-10-16 13:41:15 -0500

  * Update baselines for SSH capabilities fix (Vlad Grigorescu)

  * Fix SSH analyzer bug where is_server in capabilities is wrong. (Vlad Grigorescu)

2.6-beta2-32 | 2018-10-16 09:22:54 -0700

  * Fix typo in Sessions.h (Eiji Yanagi (Cisco))

2.6-beta2-31 | 2018-10-15 16:42:36 -0500

  * Add DCE_RPC exchange_mapi operations to relevant consts.bro file (Dale Lakes)

2.6-beta2-29 | 2018-10-12 21:30:19 +0000

  * GH-186: fix JSON formatting of timestamps before Unix epoch (Jon Siwek, Corelight)

2.6-beta2-28 | 2018-10-12 12:48:33 -0400

  * Fix test baseline for plugin skeleton update (Jon Siwek, Corelight)

2.6-beta2-27 | 2018-10-12 12:18:02 -0400

  * Convert site::local_nets, etc. into options. (Johanna Amann)

2.6-beta2-25 | 2018-10-12 08:33:32 -0400

  * Fix crash when modifying a table from within its &expire_func (Jon Siwek, Corelight)

2.6-beta2-24 | 2018-10-05 14:24:34 -0500

  * GH-184: add `bro-config --build_type`, outputs CMake build type (Jon Siwek, Corelight)

2.6-beta2-22 | 2018-10-04 11:31:48 -0500

  * Add return value checks for some RPC parsing functions (Jon Siwek, Corelight)

  * Add 'fallthrough' comment to a switch/case block (Jon Siwek, Corelight)

2.6-beta2-20 | 2018-10-03 15:47:26 -0500

  * Improve broker.remote_id unit test (Jon Siwek, Corelight)

  * Increase broker unit test timeout intervals (Jon Siwek, Corelight)

2.6-beta2-18 | 2018-10-03 11:09:04 -0500

  * Fix memory leak in broker type checking (Jon Siwek, Corelight)

2.6-beta2-17 | 2018-10-02 16:05:10 -0500

  * Update testing/btest/README (Jon Siwek, Corelight)

2.6-beta2-14 | 2018-09-25 16:38:29 -0500

  * Add some missing @TEST-REQUIRES to a few tests (Daniel Thayer)

2.6-beta2-12 | 2018-09-24 10:56:09 -0500

  * Fix BasicThread::SetOSName on FreeBSD (Dominik Charousset)

2.6-beta2-10 | 2018-09-21 13:29:15 -0500

  * Fix some broken @TEST-REQUIRES in unit tests (Daniel Thayer)

2.6-beta2-8 | 2018-09-21 13:25:50 -0500

  * Emit missing GeoIP database errors only once at startup (Jon Siwek, Corelight)

2.6-beta2-7 | 2018-09-21 10:18:55 -0500

  * Fix compile error in MMDB GeoIP code (Jon Siwek, Corelight)

2.6-beta2-6 | 2018-09-20 13:15:15 -0500

  * Add a missing "break" in OSFinger.cc (Daniel Thayer)

  * Fix buffer sizes in the rotate_file function (Daniel Thayer)

2.6-beta2-3 | 2018-09-19 15:21:00 -0500

  * Add HTTP::sqli_policy hook to ignore counting a request as a SQL injection
    (Justin Azoff)

2.6-beta2 | 2018-09-18 16:52:34 -0500

  * Release 2.6-beta2

2.6-beta | 2018-09-18 15:05:24 -0500

  * Release 2.6-beta

2.5-1001 | 2018-09-12 19:47:57 -0500

  * Fix IRC names command parsing (Jon Siwek, Corelight)

2.5-996 | 2018-09-11 13:04:20 -0500

  * Fix raw input reader not removing streams for dead processes. (Seth Hall, Corelight)

2.5-994 | 2018-09-10 19:47:03 -0500

  * Try to fix a rare broker test instability (Jon Siwek, Corelight)

  * Fix invalid memory free when using Log::default_field_name_map (Jon Siwek, Corelight)

2.5-992 | 2018-09-10 18:34:09 -0500

  * Stabilize a unit test. (Jon Siwek, Corelight)

  * Fix potential memory leak in Kerberos scripts
    (reported by Maksim Shudrak and fixed by Jon Siwek, Corelight)

2.5-990 | 2018-09-10 14:55:13 -0500

  * Fix recursive type checks/casts of broker data into type 'any' (Jon Siwek, Corelight)

  * Fix is/as operators on vector values (Jon Siwek, Corelight)

2.5-988 | 2018-09-07 17:49:34 -0500

  * Update default Broker/CAF thread tuning (Jon Siwek, Corelight)

2.5-987 | 2018-09-07 11:24:34 -0500

  * Update NEWS explaining Bro runs as 1 process instead of 2 (Daniel Thayer)

  * Update NEWS for changes to broctl "top" command output (Daniel Thayer)

2.5-984 | 2018-09-07 09:57:52 -0500

  * Give Cluster::rr_topic "key" argument a default value (Jon Siwek, Corelight)

2.5-983 | 2018-09-06 18:26:20 -0500

  * Disable broker message forwarding by default (Jon Siwek, Corelight)

2.5-982 | 2018-09-06 08:58:09 -0500

  * Documentation updates (Daniel Thayer)

  * Fix a typo and indentation in the configure script (Daniel Thayer)

  * Add krb5 devel package to Travis docker containers (Daniel Thayer)

2.5-975 | 2018-09-05 16:52:32 -0500

  * Allow weird sampling settings to be updateable at runtime (Johanna Amann, Corelight)

  * Permit weird sampling rate of 0, which suppresses all weirds (Johanna Amann, Corelight)

  * Switch packet stats to uint64. (Robin Sommer, Corelight)

2.5-969 | 2018-09-05 15:11:48 -0500

  * BIT-1208: remove unused weirds from Weird::actions table (Jon Siwek, Corelight)

  * BIT-1779: use BRO_LOG_SUFFIX env var in ascii log rotation function (Jon Siwek, Corelight)

2.5-967 | 2018-09-05 19:30:48 +0000

  * Fix printf format specification for reporting packet stats. (Robin Sommer, Corelight)

2.5-965 | 2018-09-04 17:17:36 -0500

  * Updates to NTLM script handling. (Seth Hall, Corelight)

     - This separates NTLM handling away from SMB.

     - It logs more accurately when logins are succeed or fail
       or even if the resulting status of an authentication is
       unknown.

     - Adds some new fields where the server is indicating information
       about itself (server_nb_computer_name, server_dns_computer_name,
       and server_tree_name)

2.5-962 | 2018-09-04 12:11:14 -0500

  * Improve update-changes output (Jon Siwek, Corelight)

2.5-961 | 2018-09-04 12:07:54 -0500

  * Sort output of a coverage unit test (Jon Siwek, Corelight)

  * Remove non-ascii char from rdp/consts.bro (Jon Siwek, Corelight)

2.5-957 | 2018-09-04 09:28:47 -0500

  * Fix/improve the find-bro-logs.test (Daniel Thayer)

  * Fix typos/formatting in NEWS (Daniel Thayer)

  * Clarify 'old_comm_usage_is_ok' error message (Jon Siwek, Corelight)

  * Update Mozilla CA list to NSS 3.39 (Johanna Amann, Corelight)

2.5-952 | 2018-08-31 17:30:21 -0500

  * Update NEWS (finalizations/formatting) (Jon Siwek, Corelight)

2.5-951 | 2018-08-31 15:33:31 -0500

  * Improve `make dist` (Jon Siwek, Corelight)

2.5-950 | 2018-08-31 08:54:36 -0700

  * Add @deprecated directive and deprecate policy/protocols/smb/__load__.bro

    @deprecated a warning stating that the script is deprecated. (Jon Siwek, Corelight)

2.5-947 | 2018-08-30 16:05:36 -0500

  * Allow loading policy/protocols/smb once again (Jon Siwek, Corelight)

2.5-946 | 2018-08-30 09:51:16 -0500

  * Update NEWS with more info about runtime options (Daniel Thayer)

2.5-944 | 2018-08-30 09:28:41 -0500

  * Introduce ssl_plaintext_data event, replacing ssl_application_data event.
    (Johanna Amann)

  * Add record layer version to event ssl_encrypted_data. (Johanna Amann)

  * Add compression methods to ssl_client_hello event. (Johanna Amann)

2.5-932 | 2018-08-30 00:08:58 +0000

  * Add Broker::forward() function. This enables explicit forwarding
    of events matching a given topic prefix.  Even if a receiving node
    has an event handler, it will not be raised if the event was sent
    along a topic that matches a previous call to Broker::forward().
    (Jon Siwek, Corelight)

  * Enable implicit Broker message forwarding by default. (Jon Siwek,
    Corelight)

  * Remove Cluster::broadcast_topic. As enabling Broker forwarding
    would cause routing loops with messages sent to such a topic (one
    subscribed to on all nodes). (Jon Siwek, Corelight)

  * Remove Intel Broker topics, re-use existing Cluster topics. (Jon
    Siwek, Corelight)

  * Update broker docs to reflect best-practice/convention for
    declaring new topics.

  * Remove "relay" family of Broker functions.  (Jon Siwek, Corelight)

    Namely these are now removed:

        - Broker::relay
        - Broker::publish_and_relay
        - Cluster::relay_rr
        - Cluster::relay_hrw

    The idea being that Broker may eventually implement the necessary
    routing (plus load balancing) functionality.  For now, code that
    used these should "manually" handle and re-publish events as
    needed.

2.5-924 | 2018-08-29 18:21:37 -0500

  * Allow event/function headers to be wrapped in directives. (Johanna Amann)

    For example:

        @if ( conditions )
            event a(...)
        @else
            event b(...)
        @endif
                { ... }

2.5-922 | 2018-08-29 17:22:20 -0500

  * Fix unit tests (Jon Siwek, Corelight)

  * Fix strict-aliasing compiler warning (Jon Siwek, Corelight)

2.5-919 | 2018-08-29 14:58:06 -0500

  * Update unit test baseline for new BinPAC output (Jon Siwek, Corelight)

  * CT List update - a few more logs. (Johanna Amann)

  * Update certificate list to NSS 3.38 (Johanna Amann)

2.5-915 | 2018-08-28 14:22:25 -0700

  * Improve input framework re-read logic

    Changed from checking for "has newer modification time" to "has
    different modification time or inode number". (Jon Siwek, Corelight)

  * Convert more redef-able constants to runtime options (Daniel Thayer)

2.5-911 | 2018-08-24 17:47:03 -0700

  * Add a missing initializer to a runtime option (Daniel Thayer)

  * Convert more redef-able constants to runtime options (Daniel Thayer)

2.5-907 | 2018-08-24 17:23:46 -0700

  * Fix base/misc/version.bro version parsing. (Johanna Amann)

2.5-906 | 2018-08-24 14:57:55 -0500

  * Stabilize a cluster logging unit test (Jon Siwek, Corelight)

2.5-905 | 2018-08-24 10:21:35 -0500

  * Detect MaxMind DB changes and auto-reload (Jonathan Perkins, Corelight)

2.5-903 | 2018-08-23 16:54:24 -0500

  * Fix finding of kerberos and libmaxminddb CMake < 3.3 (Daniel Thayer)

  * BIT-1885: fix "kill" threading message (Jon Siwek, Corelight)

2.5-900 | 2018-08-23 15:18:48 -0500

  * Improve readability of the Travis job log (Daniel Thayer)

  * Fix tracking of DCE-RPC context identifier mappings

    This adds previously-missing support for "Alter Context"
    request/response PDUs (initial patch contributed by Mark Fernandez).

    Also, context ID arguments were added to dce_rpc_bind, dce_rpc_request,
    and dce_rpc_response in order to properly track what endpoint/operation
    a given opnum maps to. (Jon Siwek, Corelight)

2.5-897 | 2018-08-23 15:53:16 +0000

 * BIT-1885: Fix input framework memory leak. For input threads that
   get joined during run-time, messages could remain in the thread's
   queue and leak. (Jon Siwek, Corelight)

 * Increase timeout for a memleak test. (Jon Siwek, Corelight)

2.5-894 | 2018-08-22 12:05:19 -0500

  * Ensure external test repo hashes track origin/master (Jon Siwek, Corelight)

2.5-892 | 2018-08-22 11:49:12 -0500

  * Fix "unused CMake variable" configuration warnings (Jon Siwek, Corelight)

2.5-890 | 2018-08-21 16:47:52 -0500

  * Fix Travis CI script to checkout particular commits of external tests
    (Jon Siwek, Corelight)

  * Fix signed/unsigned comparison warning (Jon Siwek, Corelight)

2.5-888 | 2018-08-21 15:54:56 -0500

  * Add --with-broker configure option (Jon Siwek, Corelight)

2.5-887 | 2018-08-21 14:54:12 -0500

  * Change default snaplen to 9216 bytes to better accommodate jumbo frames
    (Justin Azoff)

2.5-884 | 2018-08-20 15:39:21 -0500

  * Fix outdated documentation test baselines (Jon Siwek, Corelight)

  * Add 'smtp_excessive_pending_cmds' weird (Jon Siwek, Corelight)

  * Fix SMTP command string comparisons (Jon Siwek, Corelight)

  * Improve handling of empty lines in several text protocol analyzers
    (Jon Siwek, Corelight)

  * Add rate-limiting sampling mechanism for weird events

    The generation of weird events, by default, are now rate-limited
    according to these tunable options:

      - Weird::sampling_whitelist
      - Weird::sampling_threshold
      - Weird::sampling_rate
      - Weird::sampling_duration

    The new get_reporter_stats() BIF also allows one to query the
    total number of weirds generated (pre-sampling) which the new
    policy/misc/weird-stats.bro script uses periodically to populate
    a weird_stats.log.

    There's also new reporter BIFs to allow generating weirds from the
    script-layer such that they go through the same, internal
    rate-limiting/sampling mechanisms:

      - Reporter::conn_weird
      - Reporter::flow_weird
      - Reporter::net_weird

    Some of the code was adapted from previous work by Johanna Amann.
    (Jon Siwek, Corelight)

  * Teach timestamp canonifier about timestamps before ~2001
    (Jon Siwek, Corelight)

2.5-877 | 2018-08-20 14:58:58 -0500

  * Remove the node-specific local-*.bro scripts (Daniel Thayer)

2.5-875 | 2018-08-20 12:45:32 -0500

  * Improve diff-remove-abspath canonifier: collapse '/' sequences
    (Jon Siwek, Corelight)

  * Remove unused redef-able constants (Daniel Thayer)

  * Convert some redef-able constants to runtime options (Daniel Thayer)

2.5-870 | 2018-08-17 17:07:57 -0500

  * Documentation improvements (Daniel Thayer)

2.5-855 | 2018-08-17 16:34:51 -0500

  * Add script to support the old DHCP events (Vlad Grigorescu)

2.5-852 | 2018-08-17 15:15:55 -0500

  * BIT-466: add redef += support to vectors (Jon Siwek, Corelight)

2.5-850 | 2018-08-17 11:12:53 -0500

  * BIT-1815: move SMB::write_cmd_log functionality into policy/ script

    The option is removed, but same functionality is now enabled simply
    by loading policy/protocols/smb/log-cmds.bro (Jon Siwek, Corelight)

2.5-849 | 2018-08-17 10:29:58 -0500

  * Fix possible race in netcontrol acld/broker plugins (Jon Siwek, Corelight)

2.5-848 | 2018-08-16 17:21:28 -0500

  * Enable SMB by default by moving scripts from policy/ to base/
    (Jon Siwek, Corelight)

2.5-847 | 2018-08-16 16:07:14 -0500

  * BIT-1924: add DHCP port to software.log for completeness
    (Jon Siwek, Corelight)

2.5-846 | 2018-08-16 14:11:02 -0500

  * BIT-1858: fix logged-names for DNS RR types 44 and 45 (Jon Siwek, Corelight)

  * BIT-1850: add missing DCE/RPC PDU type enum values (Jon Siwek, Corelight)

2.5-844 | 2018-08-16 12:13:16 -0500

  * Add env. variables to override Broker listen/connect retry intervals
    And use them to default retries to 1sec for all unit tests.
    (Jon Siwek, Corelight)

2.5-843 | 2018-08-15 18:01:56 -0500

  * BIT-1544: allow NULs in file analysis handles (Jon Siwek, Corelight)

2.5-842 | 2018-08-15 11:00:20 -0500

  * Fix seg fault on trying to type-cast invalid/nil Broker::Data
    (Jon Siwek, Corelight)

2.5-841 | 2018-08-14 16:45:09 -0500

  * BIT-1798: fix PPTP GRE tunnel decapsulation (Jon Siwek, Corelight)

2.5-840 | 2018-08-13 17:40:06 -0500

  * Fix SumStats::observe key normalization logic
    (reported by Jim Mellander and fixed by Jon Siwek, Corelight)

2.5-839 | 2018-08-13 10:51:43 -0500

  * Make options redef-able by default. (Johanna Amann, Corelight)

  * Fix incorrect input framework warnings when parsing ports.
    (Johanna Amann, Corelight)

  * Allow input framework to accept 0 and 1 as valid boolean values.
    (Johanna Amann, Corelight)

  * Improve the travis-job script to work outside of Travis (Daniel Thayer)

  * Fix validate-certs.bro comments (Jon Siwek, Corelight)

2.5-831 | 2018-08-10 17:12:53 -0500

  * Immediately apply broker subscriptions made during bro_init()
    (Jon Siwek, Corelight)

  * Update default broker threading configuration to use 4 threads and allow
    tuning via BRO_BROKER_MAX_THREADS env. variable (Jon Siwek, Corelight)

  * Misc. unit test improvements (Jon Siwek, Corelight)

2.5-826 | 2018-08-08 13:09:27 -0700

  * Add support for code coverage statistics for bro source files after running btest
    test suite

    This adds --enable-coverage flag to configure Bro with gcov.
    A new directory named /testing/coverage/ contains a new
    coverage target. By default a coverage.log is created; running
    make html in testing/coverage creates a HTML report.
    (Chung Min Kim, Corelight)

2.5-819 | 2018-08-08 13:03:22 -0500

  * Fix cluster layout graphic and doc warnings (Jon Siwek, Corelight)

  * Added missing tcp-state for signature dpd_rfb_server (Zhongjie Wang)

2.5-815 | 2018-08-06 17:07:56 -0500

  * Fix an "uninitialized" compiler warning (Jon Siwek, Corelight)

  * Fix (non)suppression of proxy-bound events in known-*.bro scripts
    (Jon Siwek, Corelight)

2.5-811 | 2018-08-03 11:33:57 -0500

  * Update scripts to use vector "+=" append operation (Vern Paxson, Corelight)

  * Add vector "+=" append operation (Vern Paxson, Corelight)

  * Improve a travis output message in pull request builds (Daniel Thayer)

  * Use default version of OpenSSL on all travis docker containers
    (Daniel Thayer)

2.5-802 | 2018-08-02 10:40:36 -0500

  * Add set operations: union, intersection, difference, comparison
    (Vern Paxson, Corelight)

2.5-796 | 2018-08-01 16:31:25 -0500

  * Add 'W' connection history indicator for zero windows
    (Vern Paxson, Corelight)

  * Allow logarithmic 'T'/'C'/'W' connection history repetitions, which
    also now raise their own events (Vern Paxson, Corelight)

2.5-792 | 2018-08-01 12:15:31 -0500

  * fix NTLM NegotiateFlags field offsets (Jeffrey Bencteux)

2.5-790 | 2018-08-01 11:25:27 -0500

  * Fix --with-binpac configure option (Jon Siwek, Corelight)

  * Update CAF-finding logic (Jon Siwek, Corelight)

2.5-787 | 2018-07-31 16:50:55 -0500

  * Add Cisco FabricPath support (Damani Wade, Corelight)

  * Replace GeoIP Legacy DB support with MaxMind DB support
    (Jonathan Perkins, Corelight)

2.5-782 | 2018-07-31 11:53:22 +0200

  * Update install instructions for OpenSSL 1.1 compat (Jon Siwek, Corelight)

  * Remove requestorName parameter of ocsp_request event

    This field isn't publicly available via the OpenSSL 1.1 API, not used
    in the base scripts, and has no example in the test suit, so removing
    it is simpler than trying to support manually parsing it out of the
    raw data. (Jon Siwek, Corelight)

  * Adjust x509 unit tests to work around OpenSSL 1.0 vs. 1.1 differences (Jon Siwek, Corelight)

  * Fixes for OpenSSL 1.1 support (Jon Siwek, Corelight)

2.5-775 | 2018-07-24 16:39:34 -0500

  * Add broker/binpac/caf dirs to bro-config script (Jon Siwek, Corelight)

  * Exclude CMakeFiles from header installation path (Jon Siwek, Corelight)

2.5-773 | 2018-07-24 15:04:41 +0000

  * BIT-1950: Support PPPoE over QinQ (Jon Siwek, Corelight)

2.5-771 | 2018-07-24 02:26:17 +0000

  * Support building plugins from Bro installation prefix so that it
    does no longer need access to a Bro source/build tree. This
    required installing various Bro headers, BinPAC and it's headers,
    bifcl, and Bro's custom CMake modules. (Jon Siwek, Corelight)

  * Add binpac to install process. (Jon Siwek, Corelight)

  * Move bifcl to a separate repo. (Jon Siwek, Corelight)

2.5-766 | 2018-07-24 01:39:07 +0000

  * Clusterization of configureation framework. (Johanna Amann, Corelight)

  * Fix special-case-bug for vectors in UnaryExpr. (Johanna Amann, Corelight)

  * Teach Option::set to unwrap Broker::Data values (Jon Siwek, Corelight)

  * Fix some compiler warnings. (Robin Sommer, Corelight)

2.5-749 | 2018-07-20 12:08:06 -0500

  * Make Broker congestion queue size tunable and increase default
    (Jon Siwek, Corelight)

  * Improve control framework id-update/test output (Jon Siwek, Corelight)

2.5-747 | 2018-07-18 09:51:13 -0500

  * Improve some netcontrol unit tests (Jon Siwek, Corelight)

2.5-746 | 2018-07-17 17:51:13 -0500

  * Improve an input framework unit test (Jon Siwek, Corelight)

2.5-745 | 2018-07-17 16:46:16 -0500

  * Add explicit key in Travis known_hosts (Jon Siwek, Corelight)

2.5-743 | 2018-07-17 14:20:19 -0500

  * Port broker::data variant usages to use CAF API directly
    (Jon Siwek, Corelight)

2.5-741 | 2018-07-16 16:06:02 -0500

  * Improve Specific_RE_Matcher::CompileSet() error condition cleanup
    (Jon Siwek, Corelight)

2.5-740 | 2018-07-16 16:01:31 -0500

  * Add support for case-insensitive patterns (Vern Paxson, Corelight)

2.5-730 | 2018-07-16 10:39:33 -0500

  * de-restrict pattern-oriented BiFs to no longer require only running at init
    (Vern Paxson)

  * Add option to toggle extraction of subject alternate names from X509 SAN
    DNS field (Liviu Valsan)

2.5-725 | 2018-07-03 14:56:10 -0500

  * BIT-1941: improve unit test stability (Corelight)

2.5-723 | 2018-07-03 09:34:10 -0500

  * Fix unstable config framework test (Corelight)

2.5-722 | 2018-07-03 09:16:37 -0500

  * BIT-1941: teach diff-remove-timestamps about time 0 (Corelight)

2.5-721 | 2018-07-02 16:29:21 -0500

  * BIT-1941: improve reliability of broker.disconnect unit test (Corelight)

2.5-719 | 2018-06-27 20:02:52 -0500

  * Fix some typos and formatting in NEWS and other documentation
    (Daniel Thayer)

  * Add documentation for type-based switch statment, as/is operators,
    bitwise operators, and pattern operators (Daniel Thayer)

2.5-711 | 2018-06-27 19:11:58 -0500

  * Prevent double-wrapping Broker::Data in published event args (Corelight)

2.5-710 | 2018-06-26 18:06:22 -0500

  * Add memory leak unit test for pattern operations (Corelight)

  * fixed 3 leaks in creating pattern values (Vern Paxson)

  * add & and | operators for patterns (Vern Paxson)

  * deprecate merge_patterns() (Vern Paxson)

  * deprecate boolean scalar+vector operations (Vern Paxson)

  * deprecate mixing scalars and vectors (Vern Paxson)

  * deprecate && / || operators for patterns (Vern Paxson)

2.5-690 | 2018-06-26 15:05:23 -0500

  * Fix deprecated actor_system_config field usages (Corelight)

2.5-689 | 2018-06-26 11:45:52 -0500

  * Remove header self-inclusions (Corelight)

  * Fix travis-job script to not fail when all tests succeed (Daniel Thayer)

2.5-687 | 2018-06-25 16:35:25 -0500

  * Reorganize internal + private broker/Manager.h bits (Corelight)

  * Reduce proliferation of including broker header files (Corelight)

2.5-684 | 2018-06-25 11:26:55 -0500

  * Use docker containers to run Bro tests on Travis CI (Daniel Thayer)

  * Travis CI fewer failures and improved output messages (Daniel Thayer)

2.5-681 | 2018-06-22 20:17:06 -0500

  * Fix null pointer deref in AST traversal (Corelight)

  * Fix for ancient reference-counting bug in NFA.cc (Vern Paxson)

2.5-679 | 2018-06-21 16:00:48 -0500

  * Add support for bitwise operations (&, |, ^, ~) on "count" values.
    (Vern Paxson)

2.5-671 | 2018-06-21 11:55:39 -0500

  * Add ability for BroControl to skip cluster setup (Corelight)

  * BIT-1938: fix crash in Broker manager shutdown (Corelight)

  * Disable broxygen when running unit tests (Daniel Thayer)

2.5-668 | 2018-06-15 17:14:33 -0500

  * Make old comm. system usages an error unless old_comm_usage_is_ok is set
    (Corelight)

2.5-667 | 2018-06-15 15:30:11 -0500

  * Add --disable-broker-tests configure option (Corelight)

2.5-663 | 2018-06-14 12:51:28 -0500

  * Add Broker::max_threads and Broker::max_sleep tuning options,
    remove Broker::max_live_threads and Broker::max_pcap threads (Corelight)

  * Minor optimization to bro_broker::Manager::FlushPendingQueries (Corelight)

2.5-660 | 2018-06-12 13:49:39 -0500

  * Add Broker::max_live_threads and Broker::max_pcap_threads tunables
    (Corelight)

2.5-658 | 2018-06-08 16:41:07 +0000

  * Allow BRO_DEFAULT_LISTEN_ADDRESS to control broker listen address.
    This environment variable is now set to listen only on IPv4
    loopback when running unit tests (instead of using the default
    INADDR_ANY). (Corelight)

  * Move some of the @loads out from init-bare.bro into a new
    init-frameworks-and-bifs.bro in order to better support calling BIFs
    (like `getenv`) from variable initializations in those particular
    frameworks. (Corelight)

2.5-655 | 2018-06-08 10:43:03 -0500

  * Correct conn history field documentation (Corelight)

2.5-652 | 2018-06-07 13:57:23 -0500

  * GH-131: disable krb ticket decryption on non-Linux (Corelight)

2.5-651 | 2018-06-07 09:57:29 -0500

  * Fix signed/unsigned comparison compiler warning (Corelight)

2.5-650 | 2018-06-06 16:20:18 -0500

  * Improve Broker performance (Corelight)

2.5-648 | 2018-06-05 17:32:47 -0500

  * BIT-1936: improve Broxygen warnings (Corelight)

2.5-647 | 2018-06-05 15:19:16 -0500

  * Update `make doc`: don't copy broker docs (Corelight)

2.5-646 | 2018-06-05 11:31:43 -0500

  * Add NCP::max_frame_size tuning option (Corelight)

  * Migrate NCP analyzer to use latest analyzer API (Corelight)

  * Fix read at invalid address in X509 extension parser (Johanna Amann)

2.5-642 | 2018-06-04 13:52:46 -0500

  * Make 0 be a valid packet source timestamp (Corelight)

2.5-641 | 2018-06-04 09:18:59 -0700

  * Add Broker::publish_and_relay BIF

    Like Broker::relay, except the relaying-node also calls event handlers. (Corelight)

  * Document variable argument list BIFs using ellipsis. (Corelight).

  * Support unserializing broker data into type 'any'

    The receiver side will wrap the data as a Broker::Data value, which
    can then be type-checked/cast via 'is' or 'as' operators to a specific
    Bro type. For example:

    Sender:

        Broker::publish("topic", my_event, "hello")

    Receiver:

        event my_event(arg: any)
            {
            if ( arg is string )
                print arg as string;
            }

     (Corelight)

  * Fix a bug in broker data type-casting check (Corelight)

  * Remove dead code in broker data/val conversion function (Corelight)

  * SSH protocol now assesses the packet length at an earlier stage within binpac
    (Andrew Woodford).

  * Remove some UTF-8 characters that snuck into a few scripts. (Corelight)

  * Decrypt the krb ticket and extract authentication data. (Julien Wallior)

2.5-619 | 2018-06-01 11:29:15 -0500

  * Relocate temporary script coverage files (Corelight)

2.5-618 | 2018-06-01 10:03:24 -0500

  * BIT-1635: fix `make doc` warnings (Corelight)

  * Add smb2_file_sattr event (Devin Trejo)

  * Add bad ARP tests (Pierre LATET)

  * Fix SCT validation when invalid certificates are in chain. (Johanna Amann)

2.5-611 | 2018-05-29 10:13:17 -0500

  * Fix NEWS file formatting (Corelight)

  * Improve Broker docs with reminder about modules and event namespace
    scoping interactions. (Michael Dopheide)

  * Change Intel framework to round-robin insertion events across proxies
    (Corelight)

  * Add a counter for number of alive nodes within a given cluster pool
    (Corelight)

  * Fix how cluster framework tracks worker count (Corelight)

2.5-599 | 2018-05-23 16:50:12 -0500

  * Documentation improvements/fixes (Corelight)

2.5-598 | 2018-05-22 15:05:24 -0500

  * Fixes for MySQL and SMB protocol parsers (Corelight)

    * MySQL: the parser for this was generally broken (not following
      the specification well) and needed many changes.  One addition is a
      new "mysql_result_row" event that provides access to the results of
      queries.

    * SMB: the spec seems to explitly call out the omission of the
      PrimaryDomain field on SMB_COM_SESSION_SETUP_ANDX responses (and I
      don't see that field in pcaps either), so this may have just been a
      typo that used to work fine in the past only due to faulty array
      parsing behavior in binpac.

  * BIT-1829: add unit test for modbus parser issue (Corelight)

2.5-591 | 2018-05-22 09:19:59 -0500

  * Make Reassembler::TotalSize a constant time operation (Corelight)

2.5-589 | 2018-05-21 21:37:54 +0000

  * Switch Bro's communication over to Broker; deprecate the old
    communication system, including Broccoli. See NEWS for more.

    (Many people contributed to this effort. Broker library: Jon
    Siwek, Matthias Vallentin, Robin Sommer, Dominik Charousset.
    Porting Bro to Broker: Daniel Thayer, Robin Sommer, Jon Siwek.
    Further contributions by: Johanna Amann, Justin Azoff, Matthias
    Fischer, Jan Grashoefer, and Seth Hall. The final integration was
    supported by Corelight.)

  * Extend switch statement to branch by type of the operand. See NEWS
    for more. (Robin Sommer)

  * Add new operators "is" and "as" for dynamic type casting and type
    checking. See NEWS for more. (Robin Sommer)

2.5-582 | 2018-05-21 13:34:16 -0500

  * Update link to flex pattern docs (Corelight)

  * Add non-standard experimental Google post-quantum ciphers (Johanna Amann)

  * ARP: fix the l2 source address check for ARP over Wi-Fi (Pierre LALET)

  * Support 802.11 monitor mode (Pierre LALET)

2.5-569 | 2018-05-10 11:24:07 -0500

  * BIT-1927: relocate notice/extend-email/ scripts to policy/ dir and
    load it from local.bro to allow users to control whether it is used.
    (Stephen Hosom)

  * Sort output of the missing-file-initially.bro test (Daniel Thayer)

2.5-565 | 2018-05-08 15:29:53 -0500

  * BIT-1926: add unit tests for misc. HTTP patches (Corelight)

  * Fix case insensitive HTTP/MIME header name comparisons
    (Jeffrey Bencteux)

  * Don't use chunked mode Transfer-Encoding with HTTP/1.0 (Jeffrey Bencteux)

  * Fix handling of HTTP body length when Content-Range length differs
    from Content-Length. (Jeffrey Bencteux)

  * Decode 'x-gzip' HTTP Content-Encoding the same as 'gzip'
    (Jeffrey Bencteux)

2.5-559 | 2018-05-08 11:23:28 -0700

  * Add test for dump_current_packet bif. (Johanna Amann)

	* Fix dump_packet & dump_current_packet to work with several filenames.
    (Assaf Morami)

2.5-553 | 2018-05-03 14:59:53 -0500

  * Make BinPAC exception handling more consistent (Vlad Grigorescu)

2.5-551 | 2018-05-01 18:27:38 -0500

  * Fix the ip-broken-header.bro test on macOS due to missing 'xzcat'
    (Daniel Thayer)

  * Improve reliability of the logging rotate.bro test (Daniel Thayer)

  * Improve reliability of missing-file-initially.bro test (Daniel thayer)

2.5-547 | 2018-05-01 18:17:14 -0500

  * Update install instructions for Ubuntu 18.04 (Daniel Thayer)

2.5-545 | 2018-05-01 18:09:30 -0500

  * Improve canonicalization of build dir path in a coverage unit test
    (Corelight)

2.5-544 | 2018-05-01 17:57:15 -0500

  * Rewrite the DHCP analyzer and accompanying script-layer API.
    (Valerio G, Corelight)

     * Reduced all DHCP events into a single dhcp_message event.
       (removed legacy events since they weren't widely used anyway)
     - Support many more DHCP options.

     - DHCP log is completely reworked and now represents DHCP sessions
       based on the transaction ID (and works on clusters).

     - Removed the known-devices-and-hostnames.bro and known-devices.bro
       scripts since it's generally less relevant now with the updated log.

  * Change include directory search order to better support --with-openssl.
    (Johanna Amann)

2.5-535 | 2018-04-30 16:22:30 -0500

  * Improve how coverage unit tests handle name of build dir (Corelight)

2.5-534 | 2018-04-27 19:59:46 -0400

  * Fix subnet expiration in the intel framework. (Seth Hall)

  * BIT-1909: fix invalid redef'd record field accesses (Jon Siwek)

2.5-527 | 2018-04-27 11:01:03 -0500

  * BIT-1430: Improve cross compilation support (Corelight)

    * Add --toolchain= configure option

    * Add --with-bifcl= configure option

    * Change --with-binpac= configure option to mean "path to binpac
      executable"

2.5-526 | 2018-04-25 11:06:50 -0500

  * BIT-1914: comment out &check usages now that they emit warnings
    and convert some to &enforce (Corelight)

  * Removed the "start" parameter of the dnp3_header_block event
    since it's always the same value. (Corelight)

2.5-522 | 2018-04-25 10:48:38 -0500

  * Improve dce-rpc/consts.bro operations table organization (Luciano Mammino)

2.5-519 | 2018-04-20 07:46:07 -0700

  * Reduce number of btest threads running tests on Travis CI. (Daniel Thayer)

2.5-515 | 2018-04-18 11:44:36 -0500

  * Improve std::map usages in SMB code. (Corelight)

2.5-514 | 2018-04-18 10:54:24 -0500

  * Improve HLL cardinality estimate unit test. (Corelight)

2.5-513 | 2018-04-18 10:38:41 -0500

  * Updating the defined SMB2 dialects to match Microsofts current docs.
    (Corelight)

  * BIT-1862: Improve handling SMB pending commands and read response tree id.
    (Stefano Rinaldi, Corelight)

    On rare occasions, SMB server doesn't return the tree id on read responses.

  * Better reporter for Brostring with embedded NUL (Philippe Antoine)

  * Fix config input reader on systems with gcc 4.8 (Daniel Thayer)

2.5-504 | 2018-04-06 10:51:19 -0700

  * Trim the Travis CI build log output. (Daniel Thayer)

  * Remove unneeded lines from .travis.yml. (Daniel Thayer)

2.5-501 | 2018-04-06 10:49:54 -0700

  * Fix NETBIOSSSN analyzer name (Vladimir Ruzanov)

  * Additional fix for Kerberos in GSSAPI. (Seth Hall)

2.5-498 | 2018-04-03 01:59:46 -0400

  * Improvements to GSSAPI handling of Kerberos messages (John E. Rollinson, Seth Hall, juno0812, Justin Oursler)

  * Improve SMB2 Create command events and add newly parsed data. (Julien Wallior)

2.5-483 | 2018-03-29 14:10:48 -0700

  * Source code clean up (Johanna Amann)
      - Mark one-parameter constructors as 'explicit' & use 'override' where possible
      - Remove unimplemented & unused functions from header files.
      - Make some data flows more explicit for compilers.

2.5-478 | 2018-03-29 12:59:49 -0700

  * Recognize TLS 1.3 negotiation correctly. The way in which TLS 1.3
    is negotiated was changed slightly in later revisions of the
    standard. (Johanna Amann)

  * Fix the travis-job script to always run external tests. (Daniel
    Thayer)

  * Fix information leak in the update-traces script. (Daniel Thayer)

  * Add Coverity scan and private testing to Travis CI. (Daniel Thayer)

2.5-471 | 2018-03-21 13:56:57 -0700

  * Fix a memory leak in SMBv1 share mapping. (Corelight)

  * Fix one new minor typo in the config framework docs. (Daniel Thayer)

2.5-467 | 2018-03-15 14:58:40 -0700

  * Configure Travis CI email recipients and build branches. (Daniel Thayer)

  * Add documentation of the configuration framework, and improve
    existing script comments.  (Daniel Thayer)

2.5-459 | 2018-03-07 12:46:57 -0600

  * Update a doc test/baseline (Corelight)

  * Add removed root certificate back to test that requires it.

    Test has a trace that contains a WoSign certificate - they are no longer
    recognized by pretty much anyone. (Johanna Amann)

2.5-457 | 2018-02-18 17:35:50 -0600

  * Fix another warning when building the documentation (Daniel Thayer)

  * Fix a warning when building documentation (Daniel Thayer)

  * Fix the config framework several-files.bro test (Daniel Thayer)

  * Update Mozilla CA list to state of NSS 3.35. (Johanna Amann)

  * Update list of Certificate Transparency logs. (Johanna Amann)

2.5-449 | 2018-02-14 08:49:27 -0800

  * Patch in Binpac submodule that fixes an integer overflow
    (Philippe Antoine/Catena cyber).

2.5-448 | 2018-02-12 11:09:00 -0600

  * Fix pessimizing-move compiler warning. (Corelight)

2.5-447 | 2018-02-12 11:00:44 -0600

  * Add limit to number of auth flavors parsed out of MNT replies (Corelight)

  * Treat LibreSSL as an older OpenSSL (Xiaogrill)

2.5-445 | 2018-02-07 14:20:59 -0800

  * Add new configuration framework for dynamically changing script
    options at runtime. See NEWS for more. (Corelight)

  * Allow the empty field separator to be empty when reading through
    the input frameworkk. (Corelight)

2.5-435 | 2018-02-06 08:40:38 -0800

  * BIT-1854: Improve reassembly overlap checking. (Corelight)

  * BIT-1854: Fix the 'tcp_excessive_data_without_further_acks'
    option. (Corelight)

  * Make parsing of ServerKeyExchange work for D(TLS) < 1.2. (Johanna
    Amann)

  * Add more details to ssl_server_signature. (Johanna Amann)

2.5-427 | 2018-02-05 15:09:14 -0800

  * BIT-1898: Fix problems with SumStats non-cluster.bro script.
    Reported by Jim Mellander. (Corelight)

2.5-424 | 2018-02-05 15:07:20 -0800

  * Add a .travis.yml. file (Daniel Thayer)

2.5-422 | 2018-02-05 16:28:25 -0600

  * fix setup field handling in smb1_com_transaction_request messages

    This field is an array of 16 bit words and was parsed as an array of
    32 bit words. Moreover, one can not assume the format is going to be a
    16 bits opcode followed by a 16 bit file ID, the content of the setup
    field is different according to its first 16 bits word that defines
    the subcommand code. See MS-CIFS section 2.2.4.33.1 :

    Setup (variable): An array of two-byte words that provides transaction
    context to the server. The size and content of the array are specific
    to individual subcommands. (Jeffrey Bencteux)

  * add smb1_transaction2_secondary_request event

    parse and expose SMB_COM_TRANSACTION2_SECONDARY (0x33) message to
    script level. See MS-CIFS section 2.2.4.47.1. (Jeffrey Bencteux)

  * add smb1_transaction_secondary_request event

    expose SMB_COM_TRANSACTION_SECONDARY (0x26) message to script
    language. See MS-CIFS section 2.2.4.34.1. (Jeffrey Bencteux)

  * add parameters and data to smb1_transaction_request/response messages

    expose SMB_Data.Trans_Parameters and SMB_Data.Trans_Data fields of
    SMB_COM_TRANSACTION (0x25) message type. See MS-CIFS section
    2.2.4.33.1.

    These fields are exposed to the script level as Bro strings. Note that
    this commit also expose a new event smb1_transaction_response.
    (Jeffrey Bencteux)

  * add SMB_Parameters.Words to smb1_transaction2_request event

    expose the fields contained in SMB_Parameters.Words of the
    SMB_COM_TRANSACTION2 (0x32) message to the script language. See
    MS-CIFS section 2.2.46.1. (Jeffrey Bencteux)

2.5-410 | 2018-02-05 15:18:41 -0600

  * Fix warnings when building sphinx docs (Corelight)

2.5-409 | 2018-02-05 14:12:21 -0600

  * Bug fix: nfs3_writeargs didn't properly return filehandle. (Devin Trejo)

  * Add NFS events and unit tests: nfs_proc_symlink, nfs_proc_link,
    nfs_proc_sattr. (Devin Trejo)

2.5-405 | 2018-02-05 13:29:39 -0600

  * Add MOUNT3 protocol parser.

    It's not activated by default.  New events available: mount_proc_null,
    mount_proc_mnt, mount_proc_umnt, mount_proc_umnt_all,
    mount_proc_not_implemented, mount_reply_status. (Devin Trejo)

2.5-402 | 2018-02-05 10:43:59 -0600

  * Fix (unlikely) memory leak in nb_dns.c (Corelight)

  * Remove bro-plugins submodule from .gitmodules file (Daniel Thayer)

2.5-399 | 2018-01-30 14:31:45 -0800

  * Adapt the X509 analyzer to partially support OpenSSL 1.1.
    (Hilgo Bengen, Johanna Amann)

2.5-395 | 2018-01-26 15:46:05 -0600

  * BIT-1894: fix bad integer casts in BIFs: sort, rand, order, to_int
    (Corelight)

2.5-392 | 2018-01-19 11:39:34 -0600

  * Make nearly all bool conversion operators explicit. (Corelight)

2.5-390 | 2018-01-17 16:09:55 -0600

  * Logging: implement get_filter_names and small fixes.

    get_filter_names(id: ID) : set[string] returns the names of the current
    list of filters for a specified log stream.

    Furthermore this commit makes a number of logging functions more robust
    by checking existence of values before trying to modify them. This
    commit also really implements (and tests) the enable_stream function.
    (Corelight)

2.5-388 | 2018-01-16 15:35:21 -0600

  * HTTP: Recognize and skip upgrade/websocket connections.

    This patches the HTTP analyzerto recognize when a connections is upgraded
    to a different protocol (e.g. client uses the Upgrade header and gets a
    101 reply).  In this case, the analyzer stops further processing (which
    would have resulted in DPD errors) and raises a new event:

      event http_connection_upgrade(c: connection, protocol: string);

    The protocol parameter contains the name of the protocol that is being
    upgraded to, as specified in one of the header values. (Johanna Amann)

2.5-385 | 2018-01-16 14:51:07 -0600

  * Make tunnel_parents in conn.log optional.

    This makes conn.logs a bit prettier (and smaller) because all lines tha
    do not use a tunnel will now have a "-" instead of the "(empty)" for
    tunnel_parents. (Johanna Amann)

  * Correct include-path in x509Common.h (Johanna Amann)

2.5-381 | 2018-01-12 10:03:21 -0800

  * Preallocate all possible PortVals, mark PortVal ctors deprecated.

    The performance benefit is small (maybe ~1% at most), however, it's a
    trivial change without downsides. (Jon Siwek)

  * Add BRO_DEPRECATED macro. (Jon Siwek)

  * Add functions for retrieving files by their id.

    There are two new script level functions to query and lookup files
    from the core by their IDs.  These are adding feature parity for
    similarly named functions for files.  The function prototypes are
    as follows:

      Files::file_exists(fuid: string): bool
      Files::lookup_File(fuid: string): fa_file (Seth Hall)

2.5-375 | 2018-01-11 11:47:01 -0600

  * Fix a test that fails in some environments (Daniel Thayer)

  * Add CVE ID for BIT-1856. (Johanna Amann)

2.5-372 | 2017-12-15 15:08:51 -0600

  * Remove some DNS weirds that caused volume and are generally not useful:
    dns_unmatched_msg, dns_unmatched_msg_quantity, dns_unmatched_reply.
    (Corelight)

2.5-369 | 2017-12-13 14:22:47 -0600

  * Fix typo in analyzer::Manager API docs (Corelight)

2.5-368 | 2017-12-08 13:09:25 -0600

  * Improve for-loop iteration performance over empty tables. (Justin Azoff)

  * Fix gcc7 warnings. (Johanna Amann)

2.5-363 | 2017-12-05 11:00:09 -0600

  * Fix documentation for ReassemblerStats. (Corelight)

2.5-362 | 2017-12-02 09:45:04 -0600

  * BIT-1791: Do not log SOCKS passwords by default and add
    SOCKS::default_capture_password option.  (Johanna Amann)

  * Add missing ; in SSL binpac parser, found by Luke Valenta. (Johanna Amann)

2.5-359 | 2017-11-29 14:01:37 -0600

  * Add --ccache option to configure script (requires CMake 3.10+). (Corelight)

2.5-358 | 2017-11-28 12:28:14 -0800

  * Extend the TLS analyzer with several events containing cryptographic
    parameters from the client and server key exchanges.

   The new events are:

    ssl_ecdh_server_params, ssl_dh_server_params, ssl_server_signature,
    ssl_ecdh_client_params, ssl_dh_client_params, ssl_rsa_client_pms

    Since ssl_ecdh_server_params contains more information than the old
    ssl_server_curve event, ssl_server_curve is now marked as deprecated.

    (Luke Valenta)

2.5-352 | 2017-11-21 13:21:51 -0600

  * Fix assignments to event arguments becoming visible to subsequent
    handlers. (Robin Sommer)

2.5-350 | 2017-11-21 12:19:28 -0600

  * Add HookReporter plugin hook function.

    This hook gives access to basically all information that is available in
    the function in Reporter.cc that performs the logging. The hook is
    called each time when anything passes through the reporter in the cases
    in which an event usually would be called. This includes weirds. The
    hook can return false to prevent the normal reporter events from being
    raised. (Corelight)

2.5-348 | 2017-11-21 11:30:55 -0600

  * Fix a nb_dns.c compile error (older OSs) due to C90 vs C99. (Corelight)

2.5-347 | 2017-11-20 14:00:37 -0600

  * Fix and extend behavior of HookLoadFile. (Corelight)

2.5-345 | 2017-11-20 11:28:59 -0600

  * BIT-1827: fix error on initializing DNS w/ IPv6 nameserver. (Corelight)

  * Add --build-type flag to configure wrapper. (Corelight)

2.5-343 | 2017-11-17 15:27:04 -0800

  * Fix ASCII logging of very large values of type "double".
    Previously, the nonsensical "NAN.0" would be written to ASCII logs
    for any value >= 1e248). (Daniel Thayer)

  * Add more test cases to ascii-double.bro (Daniel Thayer)

  * Enforce a maximum line length in ContentLine analyzer. (Justin Azoff)

  * Fix OOB read with IP packets that have a header length greater than the total
    length of their packet. (Johanna Amann)

  * Verify version field of IP packets read from tunnels. (Johanna Amann)

2.5-332 | 2017-10-27 13:27:16 -0700

  * Bro docs tweaks for correctness and readability. (Christian Kreibich)

  * Fix use-after-free in Trigger.cc. (Johanna Amann)

2.5-328 | 2017-10-16 13:13:41 -0700

  * Patch OOB write in content-line analyzer.

    A combination of packets can trigger an out of bound write of '0' byte
    in the content-line analyzer. Addresses BIT-1856 / CVE-2017-1000458.
    (Frank Meier/Johanna Amann)

2.5-327 | 2017-10-16 12:21:01 -0700

  * Updating submodule(s).

2.5-326 | 2017-10-05 14:34:20 -0700

  * Update the SSH analyzer to support the "curve25519-sha256" KEX.
    (Vlad Grigorescu)

2.5-321 | 2017-10-03 12:00:29 -0500

  * Add "-B scripts" flag to allow debug output of script load order.
    (Corelight)

  * Fix segmentation fault on eval condition with no return value. (Corelight)

2.5-317 | 2017-09-29 09:54:50 -0400

  * BIT-1853 - Fix an issue with broctl triggering reporter error in the
    intel framework. (Justin Azoff)

  * BIT-1845 - Make "in" keyword work with binary data. (Johanna Amann)

  * Add TLS 1.3 fix and testcase due to Google Chrome's use of TLS 1.3.

    It turns out that Chrome supports an experimental mode to support TLS
    1.3, which uses a non-standard way to negotiate TLS 1.3 with a server.
    This non-standard way to negotiate TLS 1.3 breaks the current draft RFC
    and re-uses an extension on the server-side with a different binary
    formatting, causing us to throw a binpac exception.

    This patch ignores the extension when sent by the server, continuing to
    correctly parse the server_hello reply (as far as possible).

    From what I can tell this seems to be google working around the fac
    that MITM equipment cannot deal with TLS 1.3 server hellos; this change
    makes the fact that TLS 1.3 is used completely opaque unless one looks
    into a few extensions.

    We currently log this as TLS 1.2. (Johanna Amann)

2.5-310 | 2017-09-21 09:10:21 -0700

  * fix interaction of gridftp scripts with other thresholds. (Justin Azoff)

2.5-307 | 2017-09-20 10:51:09 -0500

  * BIT-1846: Updating broctl submodule to include fix for symlinking
    issue (Jon Siwek)

2.5-306 | 2017-09-18 14:43:42 -0700

  * Make strerror_r portable, supporting XSI/gnu versions. (Thomas Petersen)

  * Prevent crash when calling bro -U. (Thomas Petersen)

  * Remove annoying error message from connsize bifs. (Johanna Amann)

  * Add test to verify that log rotation works with gzipped logs (Daniel Thayer)

  * Fix ascii writer to not discard a ".gz" file extension. (Daniel Thayer)

    When Bro writes a compressed log, it uses a file extension of ".gz".
    However, upon log rotation the ascii writer script function
    "default_rotation_postprocessor_func" was discarding the ".gz"
    file extension.  Fixed so that the correct file extension is
    preserved after rotation. (Daniel Thayer)

2.5-297 | 2017-09-11 09:26:33 -0700

  * Fix small OCSP parser bug; serial numbers were not passed to events
    (Johanna Amann)

  * Fix expire-redef.bro test. (Daniel Thayer)

2.5-294 | 2017-08-11 13:51:49 -0500

  * Fix core.truncation unit test on macOS. (Jon Siwek)

  * Fix a netcontrol test that often fails (Daniel Thayer)

  * Update install instructions for Fedora 26 (Daniel Thayer)

2.5-288 | 2017-08-04 14:17:10 -0700

  * Fix field not being populated, which resulted in a reporter
    messsage. Addresses BIT-1831. Reported by Chris Herdt. (Seth Hall)

  * Support for OCSP and Signed Certificate Timestamp. (Liang
    Zhu/Johanna Amann)

      - OCSP parsing is added to the X.509 module.

      - Signed Certificate Timestamp extraction, parsing, & validation
        is added to the SSL, X.509, and OCSP analyzers. Validation is
        added to the X.509 BIFs.

    This adds the following events and BIFs:

      - event ocsp_request(f: fa_file, version: count, requestorName: string);
      - event ocsp_request_certificate(f: fa_file, hashAlgorithm: string, issuerNameHash: string, issuerKeyHash: string, serialNumber: string);
      - event ocsp_response_status(f: fa_file, status: string);
      - event ocsp_response_bytes(f: fa_file, resp_ref: opaque of ocsp_resp, status: string, version: count, responderId: string, producedAt: time, signatureAlgorithm: string, certs: x509_opaque_vector);
      - event ocsp_response_certificate(f: fa_file, hashAlgorithm: string, issuerNameHash: string, issuerKeyHash: string, serialNumber: string, certStatus: string, revokeTime: time, revokeReason: string, thisUpdate: time, nextUpdate: time);
      - event ocsp_extension(f: fa_file, ext: X509::Extension, global_resp: bool);
      - event x509_ocsp_ext_signed_certificate_timestamp(f: fa_file, version: count, logid: string, timestamp: count, hash_algorithm: count, signature_algorithm: count, signature: string);
      - event ssl_extension_signed_certificate_timestamp(c: connection, is_orig: bool, version: count, logid: string, timestamp: count, signature_and_hashalgorithm: SSL::SignatureAndHashAlgorithm, signature: string);
      - function sct_verify(cert: opaque of x509, logid: string, log_key: string, signature: string, timestamp: count, hash_algorithm: count, issuer_key_hash: string &default=""): bool
      - function x509_subject_name_hash(cert: opaque of x509, hash_alg: count): string
      - function x509_issuer_name_hash(cert: opaque of x509, hash_alg: count): string
      - function x509_spki_hash(cert: opaque of x509, hash_alg: count): string

    This also changes the MIME types that we use to identify X.509
    certificates in SSL connections from "application/pkix-cert" to
    "application/x-x509-user-cert" for host certificates and
    "application/x-x509-ca-cert" for CA certificates.

  * The SSL scripts provide a new hook "ssl_finishing(c: connection)"
    to trigger actions after the handshake has concluded. (Johanna
    Amann)

  * Add an internal API for protocol analyzers to provide the MIME
    type of file data directly, disabling automatic inferrence.
    (Johanna Amann).

2.5-186 | 2017-07-28 12:22:20 -0700

  * Improved handling of '%' at end of line in HTTP analyzer. (Johanna
    Amann)

  * Add canonifier to catch and release test that should fix test
    failures. (Johanna Amann)

2.5-181 | 2017-07-25 16:02:41 -0700

  * Extend plugin infrastructure to catch Bro version mismatches at link
    time.

    The version number used for the function name is slightly normalized
    to skip any git revision postfixes (i.e., "2.5-xxx" is always treated
    as "2.5-git") so that one doesn't need to recompile all plugins after
    every master commit. That seems good enough, usually people run into
    this when upgrading to a new release. The Plugin API version is also
    part of the version number.

    If one loads an old plugin into a new Bro, the error message looks
    like this:

        $ bro -NN Demo::Foo
        fatal error in /home/robin/bro/master/scripts/base/init-bare.bro, line 1:
        cannot load plugin library /home/robin/tmp/p/build//lib/Demo-Foo.linux-x86_64.so:
        /home/robin/tmp/p/build//lib/Demo-Foo.linux-x86_64.so: undefined symbol: bro_version_2_5_git_debug

    (Robin Sommer)

  * Several fixes and improvements for software version parsing.

     - Addresses Philip Romero's question from the Bro mailing list.
     - Adds Microsoft Edge as a detected browser.
     - We are now unescaping encoded characters in software names. (Seth Hall)

  * Remove another reference to now removed bro-plugins. (Johanna Amann)

2.5-175 | 2017-07-07 14:35:11 -0700

  * Removing aux/plugins. Most of the plugins are now Bro packages.
    (Robin Sommer)

  * Update install instructions for Debian 9. (Daniel Thayer)

2.5-170 | 2017-07-07 12:20:19 -0700

  * Update krb-protocol.pac (balintm)
    This fixes parsing of KRB_AP_Options where the padding and flags were reversed.

  * Add new cipher suites from draft-ietf-tls-ecdhe-psk-aead-05 (Johanna Amann)

  * Test changes: remove loading of listen.bro in tests that do not use it,
    serialize tests that load listen.bro, fix race conditions in some tests.
    (Daniel Thayer)

  * The broccoli-v6addrs "-r" option was renamed to "-R" (Daniel Thayer)

2.5-156 | 2017-06-13 11:01:56 -0700

  * Add 2.5.1 news file to master. (Johanna Amann)

  * Remove link to no longer existing myricom plugin. (Johanna Amann)

2.5-152 | 2017-06-05 15:16:49 -0700

  * Remove non-existing links; this broke documentation build. (Johanna Amann)

  * Fix at_least in Version.bro - it did exactly the oposite of the documented
    behavior. (Johanna Amann)

2.5-147 | 2017-05-22 20:32:32 -0500

  * Add nfs unittest. (Julien Wallior)

  * Added nfs_proc_rename event to rpc/nfs protocol analyzer.
    (Roberto Del Valle Rodriguez)

  * Expand parsing of RPC Call packets to add Uid, Gid, Stamp, MachineName
    and AuxGIDs (Julien Wallior)

  * Fix NFS protocol parser. (Julien Wallior)

2.5-142 | 2017-05-22 00:08:52 -0500

  * Add gzip log writing to the ascii writer.

    This feature can be enabled globally for all logs by setting
    LogAscii::gzip_level to a value greater than 0.

    This feature can be enabled on a per-log basis by setting gzip-level in
    $config to a value greater than 0. (Corelight)

2.5-140 | 2017-05-12 15:31:32 -0400

  * Lessen cluster load due to notice suppression.
    (Johanna Amann, Justin Azoff)

2.5-137 | 2017-05-04 11:37:48 -0500

  * Add plugin hooks for log init and writing: HookLogInit and HookLogWrite.
    (Corelight)

  * TLS: Fix compile warning (comparison between signed/unsigned).

    This was introduced with the addition of new TLS1.3 extensions. (Johanna Amann)


2.5-134 | 2017-05-01 10:34:34 -0500

  * Add rename, unlink, and rmdir bifs. (Corelight)

2.5-131 | 2017-04-21 14:27:16 -0700

  * Guard more format strings with __attribute__((format)). (Johanna Amann)

  * Add support for two TLS 1.3 extensions.

    New events:
        - event ssl_extension_supported_versions(c: connection, is_orig: bool, versions: index_vec)
        - event ssl_extension_psk_key_exchange_modes(c: connection, is_orig: bool, modes: index_vec) (Johanna Amann)

2.5-125 | 2017-04-17 22:02:39 +0200

  * Documentation updates for loading Bro scripts. (Seth Hall)

2.5-123 | 2017-04-10 13:30:14 -0700

  * Fix some failing tests by increasing delay times. (Daniel Thayer)

  * Threading Types: add a bit of documentation to subnet type. (Johanna Amann)

  * Fixing couple issues reported by Coverity. (Robin Sommer)

2.5-119 | 2017-04-07 10:30:09 -0700

  * Fix the test group name in some broker test files. (Daniel Thayer)

  * NetControl: small rule_error changes (test, call fix). (Johanna Amann)

  * SSL: update dpd signature for TLS1.3. (Johanna Amann)

2.5-115 | 2017-03-23 07:25:41 -0700

  * Fix a test that was failing on some platforms. (Daniel Thayer)

  * Remove test for cluster catch and release. This test keeps failing
    intermittently because of timing issues that are surprisingly hard
    to fix. (Johanna Amann)

  * Fix some Coverity warnings. (Daniel Thayer)

2.5-106 | 2017-03-13 11:19:03 -0700

  * print version string to stdout on --version, instead
    of printing it to stderr. Output is not an error output.  (Pete)

  * Fix compiler warning raised by llvm8. (Johanna Amann)

  * Fix coverity warning in Ascii reader. (Johanna Amann)

2.5-101 | 2017-03-09 12:20:11 -0500

  * Input's framework's ascii reader is now more resilient.

    By default, the ASCII reader does not fail on errors anymore.
    If there is a problem parsing a line, a reporter warning is
    written and parsing continues.  If the file is missing or can't
    be read, the input thread just tries again on the next heartbeat.
    (Seth Hall, Johanna Amann)

2.5-92 | 2017-03-03 10:44:14 -0800

  * Move most threading to C++11 primitives (mostly). (Johanna Amann)

  * Fix a test that sometimes fails on FreeBSD. (Daniel Thayer)

  * Remove build time warnings. (Seth Hall)

2.5-84 | 2017-02-27 15:08:55 -0500

  * Change semantics of Broker's remote logging to match old communication
    framework. (Robin Sommer)

  * Add and fix documentation for HookSetupAnalyzerTree (Johanna Amann)


2.5-76 | 2017-02-23 10:19:57 -0800

  * Kerberos ciphertext had some additional ASN.1 content being lumped
    in. (Vlad Grigorescu)

  * Updated Windows version detection to include Windows 10. (Fatema
    Bannatwala, Keith Lehigh, Mike, Seth Hall).

2.5-70 | 2017-02-20 00:20:02 -0500

  * Rework the RADIUS base script.

    Fixes BIT-1769 which improves logging behavior when replies aren't
    seen.  Also added a `framed_addr` field to indicate if the radius
    server is hinting at an address for the client and a `ttl` field to
    show how quickly the server is responding. (Seth Hall)

2.5-68 | 2017-02-18 13:59:05 -0500

  * Refactored base krb scripts. (Seth Hall)

  * New script to log ticket hashes in krb log
    (policy/protocols/krb/ticket-logging.bro).  Also, add
    ciphertext to ticket data structure. (John E. Rollinson)

2.5-62 | 2017-02-15 15:56:38 -0800

  * Fix case in which scripts were able to access unitialized variables
    in certain cases. Addresses BIT-1785. (Jon Siwek)

2.5-60 | 2017-02-15 15:19:20 -0800

  * Implement ERSPAN support.

    There is a small caveat to this implementation.  The ethernet
    header that is carried over the tunnel is ignored.  If a user
    tries to do MAC address logging, it will only show the MAC
    addresses for the outer tunnel and the inner MAC addresses
    will be stripped and not available anywhere. (Seth Hall)

  * Tiny mime-type fix from Dan Caselden. (Seth Hall)

  * Update failing intel framework test. (Johanna Amann)

2.5-55 | 2017-02-10 09:50:43 -0500

  * Fixed intel expiration reset. Reinserting the same indicator did not reset
    the expiration timer for the indicator in the underlying data store.
    Addresses BIT-1790. (Jan Grashoefer)

2.5-51 | 2017-02-06 10:15:56 -0500

  * Fix memory leak in file analyzer. (Johanna Amann)

  * Fix a series of problems with the to_json function.
    Addresses BIT-1788. (Daniel Thayer)

2.5-44 | 2017-02-03 16:38:10 -0800

  * Change snap lengths of some tests. (Johanna Amann)

  * Fix layer 2 connection flipping. If connection flipping occured in
    Sessions.cc code (invoked e.g. when the original SYN is missing),
    layer 2 flipping was not performed. (Johanna Amann)

2.5-39 | 2017-02-01 14:03:08 -0800

  * Fix file analyzer memory management, and a delay in disabling file analyzers.
    File analyzers are no longer deleted immediately; this is delayed until
    a file opject is destroyed. Furthermore, no data is sent to file analyzers
    anymore after they have been disabled.

2.5-33 | 2017-02-01 10:07:47 -0500

  * New file types sigs. (Keith Lehigh)

  * Change snaplen of test trace from 1,000,000 to 10,000

    Recent versions of libpcap are unhappy with values bigger than 262,144
    and will refuse reading the file. (Johanna Amann)


2.5-30 | 2017-01-26 13:24:36 -0800

  * Extend file extraction log, adding extracted_cutoff and extracted_size
    fields. (Seth Hall)

  * Add new TLS extension type (cached_info) (Johanna Amann)

  * Remove brocon event; it caused test failures. (Johanna Amann)

  * Add missing paths to SMB Log::create_streams calls. (Johanna Amann)

  * Tiny xlsx file signature fix. (Dan Caselden)

  * Allow access to global variables using GLOBAL:: namespace.
    Addresses BIT-1758. (Francois Pennaneac)

2.5-17 | 2016-12-07 14:51:37 -0800

  * Broxygen no longer attempts to do tilde expansion of PATH, giving
    an error message instead if bro is located in a PATH component
    that starts with a tilde. Broxygen also no longer attempts to get
    the mtime of the bro executable when bro is not invoced with the
    "-X" option. (Daniel Thayer)

  * Fix failing tests, compiler warnings and build issues on OpenBSD.
    (Daniel Thayer)

2.5-9 | 2016-12-05 11:39:54 -0800

  * Fix validation of OCSP replies inside of Bro. (Johanna Amann)

    At one place in the code, we did not check the correct return
    code. This makes it possible for a reply to get a response of
    "good", when the OCSP reply is not actually signed by the
    responder in question.

    This also instructs OCSP verication to skip certificate chain
    validation, which we do ourselves earlier because the OCSP verify
    function cannot do it correctly (no way to pass timestamp).

2.5-6 | 2016-11-29 12:51:04 -0800

  * Fix a build failure on OpenBSD relating to pcap_pkthdr. Also fixes
    an include issue on OpenBSD. (Daniel Thayer)

  * Fix compile error in krb-types.pac. (Johanna Amann)

  * Update krb-types.pac: KerberosString formatting for the principal
    principal name is now compliant with RFC 4120 section 5.2.2. (jamesecorrenti)

2.5 | 2016-11-16 14:51:59 -0800

  * Release 2.5.

2.5-beta2-17 | 2016-11-14 17:59:19 -0800

  * Add missing '@load ./pubkey-hashes' to
    policy/frameworks/intel/seen. (Robin Sommer)

2.5-beta2-15 | 2016-11-14 17:52:55 -0800

  * Remove unused "bindist" make target. (Daniel Thayer)

  * Improve the "How to Upgrade" page in the Bro docs. (Daniel Thayer)

  * Update the quickstart guide for the deploy command. (Daniel Thayer)

  * Improved installation instructions for Mac OS X. (Daniel Thayer)

  * Lots of more small updates to documentation. (Daniel Thayer)

2.5-beta2 | 2016-11-02 12:13:11 -0700

  * Release 2.5-beta2.

2.5-beta-135 | 2016-11-02 09:47:20 -0700

  * SMB fixes and cleanup. Includes better SMB error handling, improved DCE_RPC
    handling in edge cases where drive_mapping is not seen. The concept of unknown
    shares has been removed with this change. Also fixes SMB tree connect handling and
    removes files that are not parsed. SMB2 error parsing is disabled because it never
    was implemented correctly. (Seth Hall)

  * Including a test for raw NTLM in SMB (Seth Hall)

  * Updates for SMB auth handling from Martin van Hensbergen.

     - Raw NTLM (not in GSSAPI) over SMB is now handled correctly.
     - The encrypted NTLM session key is now passed into scriptland
       through the ntlm_authenticate event. (Seth Hall)

  * Add a files framework signature for VIM tmp files. (Seth Hall)

  * Version parsing scripts now supports several beta versions. (Johanna Amann)

2.5-beta-123 | 2016-11-01 09:40:49 -0700

  * Add a new site policy script local-logger.bro. (Daniel Thayer)

2.5-beta-121 | 2016-10-31 14:24:33 -0700

  * Python 3 compatibility fixes for documentation building. (Daniel Thayer)

2.5-beta-114 | 2016-10-27 09:00:24 -0700

  * Fix for Sphinx >= 1.4 compability. (Robin Sommer)

2.5-beta-113 | 2016-10-27 07:44:25 -0700

  * XMPP: Fix detection of StartTLS when using namespaces. (Johanna
    Amann)

2.5-beta-110 | 2016-10-26 09:42:11 -0400

  * Improvements DCE_RPC analyzer to make it perform fragment handling correctly
    and generally be more resistent to unexpected traffic. (Seth Hall)

2.5-beta-102 | 2016-10-25 09:43:45 -0700

  * Update number of bytes in request/response of smb1-com-open-andx.pac. (balintm)

  * Fix a IPv4 CIDR specifications and payload-size condition of signature matching.
    (Robin Sommer)

  * Python 3 compatibility fix for coverage-calc script. (Daniel Thayer)

2.5-beta-93 | 2016-10-24 11:11:07 -0700

  * Fix alignment issue of ones_complement_checksum. This error
    occured reproducibly newer compilers when called from
    icmp6_checksum. (Johanna Amann)

2.5-beta-91 | 2016-10-20 11:40:37 -0400

  * Fix istate.pybroccoli test on systems using Python 3. (Daniel Thayer)

2.5-beta-89 | 2016-10-18 21:50:51 -0400

  * SSH analyzer changes: the events are now restructured a bit. There is a new
    event ssh_auth_attempted, that is raised each time authentication is tried.
    ssh_auth_failed is still only being raised once per connection. There also
    is an additional event ssh_auth_result giving more information about the
    number of times that authentication was tried and if it succeded/failed in
    the end. The number of authentication attemps are now part of ssh.log.
    Addresses BIT-1641. (Vlad Grigorescu)

2.5-beta-79 | 2016-10-13 15:58:48 -0700

  * Fix MD5 problem with FreeBSD 11.0 and clang 3.8. The apparent
    cause is some confusion in clang when using a static char inside a
    static inline function that is refered to several compilation
    units. (Johanna Amann)

  * Initial TLS 1.3 support, as of draft-16. (Johanna Amann)

2.5-beta-73 | 2016-10-13 14:03:04 -0700

  * Added missing README files for documentation. (Daniel Thayer)

  * List new log files in the log-files.rst document. (Daniel Thayer)

2.5-beta-67 | 2016-10-10 08:28:38 -0700

  * Fixes for DCE_RPC analyzer. (Seth Hall)

      - DCE_RPC fragmentation handling returns.

      - Fixed some general parsing issues.

      - Fixed an issue with the DCE_RPC signature not working for IPv6
        connections.

2.5-beta-64 | 2016-10-10 08:20:42 -0700

  * Fix httpd.py test script for Py3 compatibility. (Daniel Thayer)

  * Tiny fix for a DCE_RPC script issue. Fixes BIT-1688. (Seth Hall)

  * Fix for plugins/hooks test. (Johanna Amann)

  * Update a TLS constants in preparation for TLS 1.3, and rename a
    few names that had never been formally assigned yet. (Johanna
    Amann)

  * Fixing Broxygen indexing confusion for plugins. Broxygen now
    indexes plugin scripts as, e.g., "Bro_Netmap/scripts/init.bro".
    Addresses BIT-1693. (Robin Sommer)

2.5-beta-54 | 2016-10-06 14:24:01 -0700

  * Fixing documentation piece on the interesting-hostname script.
    (Robin Sommer)

  * Improve the SMB documentation. (Vlad Grigorescu)

2.5-beta-46 | 2016-10-06 14:11:03 -0700

  * Fixing Broxygen indexing confusion for plugins. Scripts in plugins now
    get an artificial index prefix: "plugin_name/", followed by the script's
    relative path inside the plugin's top-level directory. For example,
    "/opt/bro/lib/bro/plugins/Bro_Netmap/scripts/init.bro" now turns into
    "Bro_Netmap/scripts/init.bro" for Broxygen purposes (whereas it used to
    be just "init.bro"). (Robin Sommer)

  * Fix a couple of tests, addressing issues of the newly introduced version.bro
    as well as small FreeBSD 11.0 issues. (Johanna Amann)

  * Update documentation license. (Johanna Amann)

  * Add a convenient way to access version information to Bro.

      @if ( Version::number >= 20500 )
      or
      @if ( Version::at_least("2.5") )

      Version::info contains detailed information about the running version of
      Bro, including beta flags, etc. (Johanna Amann)

2.5-beta-35 | 2016-10-02 17:38:31 -0400

  * Normalize http host in seen script.  (Johanna Amann)

2.5-beta-33 | 2016-10-02 14:42:22 -0400

  * Handle removing non-existent intel items. (Jan Grashoefer)

2.5-beta-29 | 2016-09-28 18:18:35 -0700

  * Prettifying reporter output in case no epxression is associated
    with runtime error. (Robin Sommer)

2.5-beta-28 | 2016-09-27 11:44:33 -0700

  * Check if the number of fields in a log write are equal to the
    number of fields required. Addresses BIT-1683. (Johanna Amann)

  * Fix a small memory leak for disabled log writers. (Johanna Amann)

  * Fix loaded-scripts.bro to work with arbitrary indentation levels.
    Addresses BIT-1691 (Johanna Amann)

  * In interesting-hostnames.bro, move lookup_addr() outside of when
    statement to avoid expensive cloning of full connection record.
    Addresses BIT-1670. (Justin Azoff)

2.5-beta-21 | 2016-09-26 10:15:23 -0700

  * Fix a debugger bug where it would not support statements like print(3).
    Addresses BIT-1703. (Vlad Grigorescu)

2.5-beta-19 | 2016-09-19 17:16:40 -0700

  * Kerberos updates (Vlad Grigorescu):

    * Make PA_EncType_Info salt field optional.

    * Add support for parsing ENCTYPE_INFO pre-auth data (same as
      ENCTYPE_INFO2).

2.5-beta-17 | 2016-09-19 15:26:14 -0700

  * Clarified string and fa_file documentation (Moshe Kaplan)

2.5-beta-12 | 2016-09-06 07:35:38 -0700

  * Added a missing fclose in scan.l. Addresses BIT-1690.
    (Daniel Thayer).

  * Fix issue with file_extraction_limit event. (Seth Hall)

  * Fix a crash when a user disables DCE_RPC while enabling SMB.
    (Seth Hall)

2.5-beta-6 | 2016-08-19 07:50:10 -0700

  * Clarify explanation of mime_entity_data event. (Moshe Kaplan)

  * Update NEWS, correcting typos formatting and adding more
    information. (Daniel Thayer)

  * Remove old ack_above_hole event from scripts.
    Fixes BIT-1673. (Johanna Amann)

2.5-beta | 2016-08-17 10:37:49 -0700

  * Release 2.5-beta.

  * Fix sphinx build errors (Johanna Amann)

  * Change failure in utf16_bytestring_to_utf8_val to be a conn weird. (Seth Hall)

  * Fix test failure caused by uninitialized memory. (Johanna Amann)

  * SMB: fix rounding error due to value truncation when converting timestamps. (Johanna Amann)

2.4-947 | 2016-08-16 12:10:02 -0700

  * Fix issues with handling of indermediate sumstats updates. (Justin Azoff)

  * Address coverity errors. (Johanna Amann)

2.4-943 | 2016-08-15 17:03:14 -0700

  * Add 'bro-config' script. (Jon Siwek)

  * add certificate to external list for debian 8 to fix test. (Johanna Amann)

  * KRB: fix field value missing error for msg$client_name. (Johanna Amann)

  * Tiny SMB cleanup in pipe handling as well as NTLM fixes. (Seth Hall).

  * Fix a number of format specifier errors. (Johanna Amann)

  * Make several tests more stable. (Johanna Amann)

2.4-927 | 2016-08-11 21:49:06 -0700

  * Make component tags generated during component initialization stable.
    Before, it was dependent on the order a compiler called constructors.
    This makes a few tests work with gcc. (Johanna Amann)

  * Make x509 intel seen script more robust (Johanna Amann)

  * Input: DisableFrontend was not called upon Init failure. (Johanna Amann)

  * DCE_RPC code simplification. (Seth Hall)

2.4-921 | 2016-08-10 20:29:48 -0700

  * Add logging framework ext-data mechanism. It is now possible to
    extend logs by adding new data columns by them - either to specific
    ones, or globally to all logs. This can, e.g., be used to add node
    names to all logs. (Seth Hall)

  * Add unrolling separator & field name map to logging framework.
    One can now use logging separators other than ".", as well as
    change specific column names in logs. (Seth Hall)

  * Fix memory leak in EnumType. (Johanna Amann)

  * Fix configure warning when compiling with --enable-broker. (Johanna Amann)

  * Add netcontrol-connectors to aux directory. (Johanna Amann)

  * Update Mozilla CA list. (Johanna Amann)

  * update scripts loaded by default in local.bro. Traceroute is now disabled
    by default, stats and capture-loss enabled by default. (Johanna Amann)

2.4-907 | 2016-08-09 15:42:17 -0400

  * Updating NEWS.

2.4-905 | 2016-08-09 08:19:37 -0700

  * GSSAPI analyzer now forwards authentication blobs more correctly.
    (Seth Hall)

  * The KRB analyzer now includes support for the PA_ENCTYPE_INFO2
    pre-auth data type. (Seth Hall)

  * Add an argument to "disable_analyzer" function to not do a
    reporter message by default. (Seth Hall)

2.4-902 | 2016-08-08 16:50:35 -0400

  * Adding SMB analyzer. (Seth Hall, Vlad Grigorescu and many others)

  * NetControl: allow reasons in remove_rule calls. Addresses BIT-1655
    (Johanna Amann)

2.4-893 | 2016-08-05 15:43:04 -0700

  * Remove -z/--analysis option. (Johanna Amann)

  * Remove already defunct code for XML serialization. (Johanna Amann)

2.4-885 | 2016-08-05 15:03:59 -0700

  * Reverting SMB analyzer merge. (Robin Sommer)

2.4-883 | 2016-08-05 12:57:26 -0400

  * Add a new node type for logging with the cluster framework scripts by
    adding a new Bro node type for doing logging (this is intended to
    reduce the load on the manager). If a user chooses not to specify a
    logger node in the cluster configuration, then the manager will
    write logs locally as usual. (Daniel Thayer)

2.4-874 | 2016-08-05 12:43:06 -0400

  * SMB analyzer (Seth Hall, Vlad Grigorescu and many others)

2.4-759 | 2016-08-05 09:32:42 -0400

  * Intel framework improvements (Jan Grashoefer)
    * Added expiration for intelligence items.
    * Improved intel notices.
    * Added hook to allow extending the intel log.
    * Added support for subnets to intel-framework.

2.4-742 | 2016-08-02 15:28:31 -0700

  * Fix duplicate SSH authentication failure events. Addresses BIT-1641.
    (Robin Sommer)

  * Remove OpenSSL dependency for plugins. (Robin Sommer)

2.4-737 | 2016-08-02 11:38:07 -0700

  * Fix some Coverity warnings. (Robin Sommer)

2.4-735 | 2016-08-02 11:05:36 -0700

  * Added string slicing examples to documentation. (Moshe Kaplan)

2.4-733 | 2016-08-01 09:09:29 -0700

  * Fixing a CMake dependency issue for the pcap bifs. (Robin Sommer)

2.4-732 | 2016-08-01 08:33:00 -0700

  * Removing pkg/make-*-packages scripts. BIT-1509 #closed (Robin
    Sommer)

2.4-731 | 2016-08-01 08:14:06 -0700

  * Correct endianness of IP addresses in SNMP. Addresses BIT-1644.
    (Anony Mous)

2.4-729 | 2016-08-01 08:00:54 -0700

  * Fix behavior of connection_pending event. It is now really only
    raised when Bro is terminating. Also adds a test-case that raises
    the event. (Johanna Amann)

  * Retired remove -J/-K options (set md5/hash key) from the manpage.
    They had already been removed from the code. (Johanna Amann)

  * NetControl: Add catch-and-release event when IPs are forgotten.
    This adds an event catch_release_forgotten() that is raised once
    Catch & Release ceases block management for an IP address because
    the IP has not been seen in traffic during the watch interval.
    (Johanna Amann)

2.4-723 | 2016-07-26 15:04:26 -0700

  * Add error events to input framework. (Johanna Amann)

    This change introduces error events for Table and Event readers.
    Users can now specify an event that is called when an info,
    warning, or error is emitted by their input reader. This can,
    e.g., be used to raise notices in case errors occur when reading
    an important input stream.

    Example:

        event error_event(desc: Input::TableDescription, msg: string, level: Reporter::Level)
            {
            ...
            }

        event bro_init()
            {
            Input::add_table([$source="a", $error_ev=error_event, ...]);
            }

     Addresses BIT-1181.

  * Calling Error() in an input reader now automatically will disable
    the reader and return a failure in the Update/Heartbeat calls.
    (Johanna Amann)

  * Convert all errors in the ASCII formatter into warnings (to show
    that they are non-fatal. (Johanna Amann)

  * Enable SQLite shared cache mode. This allows all threads accessing
    the same database to share sqlite objects. See
    https://www.sqlite.org/sharedcache.html. Addresses BIT-1325.
    (Johanna Amann)

  * NetControl: Adjust default priority of ACTION_DROP hook to standad
    level. (Johanna Amann)

  * Fix types when constructing SYN_packet record. Fixes BIT-1650.
    (Grant Moyer).

2.4-715 | 2016-07-23 07:27:05 -0700

  * SQLite writer: Remove unused string formatting function. (Johanna Amann)

  * Deprecated the ElasticSearch log writer. (Johanna Amann)

2.4-709 | 2016-07-15 09:05:20 -0700

  * Change Bro's hashing for short inputs and Bloomfilters from H3 to
    Siphash, which produces much better results for HLL in particular.
    (Johanna Amann)

  * Fix a long-standing bug which truncated hash values to 32-bit on
    most machines. (Johanna Amann)

  * Fixes to HLL. Addresses BIT-1612. (Johanna Amann)

  * Add test checking the quality of HLL. (Johanna Amann)

  * Remove the -K/-J options for setting keys. (Johanna Amann)

  * SSL: Fix memory management problem. (Johanna Amann)

2.4-693 | 2016-07-12 11:29:17 -0700

  * Change TCP analysis to process connections without the initial SYN as
    non-partial connections. Addresses BIT-1492. (Robin Sommer).

2.4-691 | 2016-07-12 09:58:38 -0700

  * SSL: add support for signature_algorithms extension. (Johanna
    Amann)

2.4-688 | 2016-07-11 11:10:33 -0700

  * Disable broker by default. To enable it, use --enable-broker.
    Addresses BIT-1645. (Daniel Thayer)

2.4-686 | 2016-07-08 19:14:43 -0700

  * Added flagging of retransmission to the connection history.
    Addresses BIT-977. (Robin Sommer)

2.4-683 | 2016-07-08 14:55:04 -0700

  * Extending connection history field to flag with '^' when Bro flips
    a connection's endpoints. Addresses BIT-1629. (Robin Sommer)

2.4-680 | 2016-07-06 09:18:21 -0700

  * Remove ack_above_hole() event, which was a subset of content_gap
    and led to plenty noise. Addresses BIT-688. (Robin Sommer)

2.4-679 | 2016-07-05 16:35:53 -0700

  * Fix segfault when an existing enum identifier is added again with
    a different value. Addresses BIT-931. (Robin Sommer)

  * Escape the empty indicator in logs if it occurs literally as a
    field's actual content. Addresses BIT-931. (Robin Sommer)

2.4-676 | 2016-06-30 17:27:54 -0700

  * A larger series of NetControl updates. (Johanna Amann)

        * Add NetControl framework documentation to the Bro manual.

        * Use NetControl for ACTION_DROP of notice framework. So far,
          this action did nothing by default.

        * Rewrite of catch-and-release.

        * Fix several small logging issues.

        * find_rules_subnet() now works in cluster mode. This
          introduces two new events, NetControl::rule_new and
          NetControl::rule_destroyed, which are raised when rules are
          first added and then deleted from the internal state
          tracking.

        * Fix acld whitelist command.

        * Add rule existance as a state besides added and failure.

        * Suppress duplicate "plugin activated" messages.

        * Make new Broker plugin options accessible.

        * Add predicates to Broker plugin.

  * Tweak SMTP scripts to not to pull in the notice framework.

2.4-658 | 2016-06-30 16:55:32 -0700

  * Fix a number of documentation building errors. (Johanna Amann)

  * Input/Logging: Make bool conversion operator explicit. (Johanna Amann)

  * Add new TLS ciphers from RFC 7905. (Johanna Amann)

2.4-648 | 2016-06-21 18:33:22 -0700

  * Fix memory leaks. Reported by Dk Jack. (Johanna Amann)

2.4-644 | 2016-06-21 13:59:05 -0400

  * Fix an off-by-one error when grabbing x-originating-ip header in
    email. (Seth Hall, Aashish Sharma)

2.4-642 | 2016-06-18 13:18:23 -0700

  * Fix potential mismatches when ignoring duplicate weirds. (Johanna Amann)

  * Weird: Rewrite internals of weird logging. (Johanna Amann)

        - "flow weirds" now actually log information about the flow
        that they occur in.

        - weirds can now be generated by calling Weird::weird() with
        the info record directly, allowing more fine-granular passing
        of information. This is e.g. used for DNS weirds.

    Addresses BIT-1578 (Johanna Amann)

  * Exec: fix reader cleanup when using read_files, preventing file
    descriptors from leaking every time it was used. (Johanna Amann)

  * Raw Writer: Make code more c++11-y, remove raw pointers. (Johanna
    Amann)

  * Add separate section with logging changes to NEWS. (Seth Hall)

2.4-635 | 2016-06-18 01:40:17 -0400

  * Add some documentation for modbus data types. Addresses
    BIT-1216. (Seth Hall)

  * Removed app-stats scripts. Addresses BIT-1171. (Seth Hall)

2.4-631 | 2016-06-16 16:45:10 -0400

  * Fixed matching mail address intel and added test (Jan Grashoefer)

  * A new utilities script named email.bro with some utilities
    for parsing out email addresses from strings. (Seth Hall)

  * SMTP "rcptto" and "mailfrom" fields now do some minimal
    parsing to clean up email addresses. (Seth Hall)

  * Added "cc" to the SMTP log and feed it into the Intel framework
    with the policy/frameworks/intel/seen/smtp.bro script. (Seth Hall)

2.4-623 | 2016-06-15 17:31:12 -0700

  * &default values are no longer overwritten with uninitialized
    by the input framework. (Jan Grashoefer)

2.4-621 | 2016-06-15 09:18:02 -0700

  * Fixing memory leak in changed table expiration code. (Robin
    Sommer)

  * Fixing test portability. (Robin Sommer)

  * Move the HTTP "filename" field (which was never filled out
    anyways) to "orig_filenames" and "resp_filenames".  (Seth Hall)

  * Add a round trip time (rtt) field to dns.log. (Seth Hall)

  * Add ACE archive files to the identified file types. Addresses
    BIT-1609. (Stephen Hosom)

2.4-613 | 2016-06-14 18:10:37 -0700

  * Preventing the event processing from looping endlessly when an
    event reraised itself during execution of its handlers. (Robin
    Sommer)

2.4-612 | 2016-06-14 17:42:52 -0700

  * Improved handling of 802.11 headers. (Jan Grashoefer)

2.4-609 | 2016-06-14 17:15:28 -0700

  * Fixed table expiration evaluation. The expiration attribute
    expression is now evaluated for every use. Thus later adjustments
    of the value (e.g. by redefining a const) will now take effect.
    Values less than 0 will disable expiration. (Jan Grashoefer)

2.4-606 | 2016-06-14 16:11:07 -0700

  * Fix parsing precedence of "hook" expression. Addresses BIT-1619
    (Johanna Amann)

  * Update the "configure" usage message for --with-caf (Daniel
    Thayer)

2.4-602 | 2016-06-13 08:16:34 -0700

  * Fixing Covertity warning (CID 1356391). (Robin Sommer)

  * Guarding against reading beyond packet data when accessing L2
    address in Radiotap header. (Robin Sommer)

2.4-600 | 2016-06-07 15:53:19 -0700

  * Fixing typo in BIF macros. Reported by Jeff Barber. (Robin Sommer)

2.4-599 | 2016-06-07 12:37:32 -0700

  * Add new functions haversine_distance() and haversine_distance_ip()
    for calculating geographic distances. They requires that Bro be
    built with libgeoip. (Aashish Sharma/Daniel Thayer).

2.4-597 | 2016-06-07 11:46:45 -0700

  * Fixing memory leak triggered by new MAC address logging. (Robin
    Sommer)

2.4-596 | 2016-06-07 11:07:29 -0700

  * Don't create debug.log immediately upon startup (BIT-1616).
    (Daniel Thayer)

2.4-594 | 2016-06-06 18:11:16 -0700

  * ASCII Input: Accept DOS/Windows newlines. Addresses BIT-1198
    (Johanna Amann)

  * Fix BinPAC exception in RFB analyzer. (Martin van Hensbergen)

  * Add URL decoding for the unofficial %u00AE style of encoding. (Seth Hall)

  * Remove the unescaped_special_char HTTP weird. (Seth Hall)

2.4-588 | 2016-06-06 17:59:34 -0700

  * Moved link-layer addresses into endpoints. The link-layer
    addresses are now part of the connection endpoints following the
    originator/responder pattern. (Jan Grashoefer)

  * Link-layer addresses are extracted for 802.11 plus RadioTap. (Jan
    Grashoefer)

  * Fix coverity error (uninitialized variable) (Johanna Amann)

  * Use ether_ntoa instead of ether_ntoa_r

    The latter is thread-safe, but a GNU addition which does not exist on
    OS-X. Since the function only is called in the main thread, it should
    not matter if it is or is not threadsafe. (Johanna Amann)

  * Fix FreeBSD/OSX compile problem due to headers (Johanna Amann)

2.4-581 | 2016-05-30 10:58:19 -0700

  * Adding missing new script file mac-logging.bro. (Robin Sommer)

2.4-580 | 2016-05-29 13:41:10 -0700

  * Add Ethernet MAC addresses to connection record. c$eth_src and
    c$eth_dst now contain the Ethernet address if available. A new
    script protocols/conn/mac-logging.bro adds these to conn.log when
    loaded. (Robin Sommer)

2.4-579 | 2016-05-29 08:54:57 -0700

  * Fixing Coverity warning. Addresses CID 1356116. (Robin Sommer)

  * Fixing FTP cwd getting overlue long. (Robin Sommer)

  * Clarifying notice documentation. Addresses BIT-1405. (Robin
    Sommer)

  * Changing protocol_{confirmation,violation} events to queue like
    any other event. Addresses BIT-1530. (Robin Sommer)

  * Normalizing test baseline. (Robin Sommer)

  * Do not use scientific notations when printing doubles in logs.
    Addresses BIT-1558. (Robin Sommer)

2.4-573 | 2016-05-23 13:21:03 -0700

  * Ignoring packets with negative timestamps. Addresses BIT-1562 and
    BIT-1443. (Robin Sommer)

2.4-572 | 2016-05-23 12:45:23 -0700

  * Fix for a table refering to a expire function that's not defined.
    Addresses BIT-1597. (Robin Sommer)

2.4-571 | 2016-05-23 08:26:43 -0700

  * Fixing a few Coverity warnings. (Robin Sommer)

2.4-569 | 2016-05-18 07:39:35 -0700

  * DTLS: Use magix constant from RFC 5389 for STUN detection.
    (Johanna Amann)

  * DTLS: Fix binpac bug with DTLSv1.2 client hellos. (Johanna Amann)

  * DTLS: Fix interaction with STUN. Now the DTLS analyzer cleanly
    skips all STUN messages. (Johanna Amann)

  * Fix the way that child analyzers are added. (Johanna Amann)

2.4-563 | 2016-05-17 16:25:21 -0700

  * Fix duplication of new_connection_contents event. Addresses
    BIT-1602 (Johanna Amann)

  * SMTP: Support SSL upgrade via X-ANONYMOUSTLS This seems to be a
    non-standardized microsoft extension that, besides having a
    different name, works pretty much the same as StartTLS. We just
    treat it as such. (Johanna Amann)

  * Fixing control framework's net_stats and peer_status commands. For
    the latter, this removes most of the values returned, as we don't
    have access to them anymore. (Robin Sommer)

2.4-555 | 2016-05-16 20:10:15 -0700

  * Fix failing plugin tests on OS X 10.11. (Daniel Thayer)

  * Fix failing test on Debian/FreeBSD. (Johanna Amann)

2.4-552 | 2016-05-12 08:04:33 -0700

  * Fix a bug in receiving remote logs via broker. (Daniel Thayer)

  * Fix Bro and unit tests when broker is not enabled. (Daniel Thayer)

  * Added interpreter error for local event variables. (Jan Grashoefer)

2.4-544 | 2016-05-07 12:19:07 -0700

  * Switching all use of gmtime and localtime to use reentrant
    variants. (Seth Hall)

2.4-541 | 2016-05-06 17:58:45 -0700

  * A set of new built-in function for gathering execution statistics:

      get_net_stats(), get_conn_stats(), get_proc_stats(),
      get_event_stats(), get_reassembler_stats(), get_dns_stats(),
      get_timer_stats(), get_file_analysis_stats(), get_thread_stats(),
      get_gap_stats(), get_matcher_stats().

    net_stats() resource_usage() have been superseded by these. (Seth
    Hall)

  * New policy script misc/stats.bro that records Bro execution
    statistics in a standard Bro log file. (Seth Hall)

  * A series of documentation improvements. (Daniel Thayer)

  * Rudimentary XMPP StartTLS analyzer. It parses certificates out of
    XMPP connections using StartTLS. It aborts processing if StartTLS
    is not found. (Johanna Amann)

2.4-507 | 2016-05-03 11:18:16 -0700

  * Fix incorrect type tags in Bro broker source code. These are just
    used for error reporting. (Daniel Thayer)

  * Update docs and tests of the fmt() function. (Daniel Thayer)

2.4-500 | 2016-05-03 11:16:50 -0700

  * Updating submodule(s).

2.4-498 | 2016-04-28 11:34:52 -0700

  * Rename Broker::print to Broker::send_print and Broker::event to
    Broker::send_event to avoid using reserved keywords as function
    names. (Daniel Thayer)

  * Add script wrapper functions for Broker BIFs. This faciliates
    documenting them through Broxygen. (Daniel Thayer)

  * Extend, update, and clean up Broker tests. (Daniel Thayer)

  * Intel: Allow to provide uid/fuid instead of conn/file. (Johanna
    Amann)

  * Provide file IDs for hostname matches in certificates. (Johanna
    Amann)

  * Rudimentary IMAP StartTLS analyzer. It parses certificates out of
    IMAP connections using StartTLS. It aborts processing if StartTLS
    is not found. (Johanna Amann)

2.4-478 | 2016-04-28 09:56:24

  * Fix parsing of x509 pre-y2k dates. (Johanna Amann)

  * Fix small error in bif documentation. (Johanna Amann)

  * Fix unknown data link type error message. (Vitaly Repin)

  * Correcting spelling errors. (Jeannette Dopheide)

  * Minor cleanup in ARP analyzer. (Johanna Amann)

  * Fix parsing of pre-y2k dates in X509 certificates. (Johanna Amann)

  * Fix small error in get_current_packet documentation. (Johanna Amann)

2.4-471 | 2016-04-25 15:37:15 -0700

  * Add DNS tests for huge TLLs and CAA. (Johanna Amann)

  * Add DNS "CAA" RR type and event. (Mark Taylor)

  * Fix DNS response parsing: TTLs are unsigned. (Mark Taylor)

2.4-466 | 2016-04-22 16:25:33 -0700

  * Rename BrokerStore and BrokerComm to Broker. Also split broker main.bro
    into two scripts. (Daniel Thayer)

  * Add get_current_packet_header bif. (Jan Grashoefer)

2.4-457 | 2016-04-22 08:36:27 -0700

  * Fix Intel framework not checking the CERT_HASH indicator type. (Johanna Amann)

2.4-454 | 2016-04-14 10:06:58 -0400

  * Additional mime types for file identification and a few fixes. (Seth Hall)

    New file mime types:
     - .ini files
     - MS Registry policy files
     - MS Registry files
     - MS Registry format files (e.g. DESKTOP.DAT)
     - MS Outlook PST files
     - Apple AFPInfo files

    Mime type fixes:
     - MP3 files with ID3 tags.
     - JSON and XML matchers were extended

  * Avoid a macro name conflict on FreeBSD. (Seth Hall, Daniel Thayer)

2.4-452 | 2016-04-13 01:15:20 -0400

  * Add a simple file entropy analyzer. (Seth Hall)

  * Analyzer and bro script for RFB/VNC protocol (Martin van Hensbergen)

    This analyzer parses the Remote Frame Buffer
    protocol, usually referred to as the 'VNC protocol'.

    It supports several dialects (3.3, 3.7, 3.8) and
    also handles the Apple Remote Desktop variant.

    It will log such facts as client/server versions,
    authentication method used, authentication result,
    height, width and name of the shared screen.


2.4-430 | 2016-04-07 13:36:36 -0700

  * Fix regex literal in scripting documentation. (William Tom)

2.4-428 | 2016-04-07 13:33:08 -0700

  * Confirm protocol in SNMP/SIP only if we saw a response SNMP/SIP
    packet. (Vlad Grigorescu)

2.4-424 | 2016-03-24 13:38:47 -0700

  * Only load openflow/netcontrol if compiled with broker. (Johanna Amann)

  * Adding canonifier to test. (Robin Sommer)

2.4-422 | 2016-03-21 19:48:30 -0700

  * Adapt to recent change in CAF CMake script. (Matthias Vallentin)

  * Deprecate --with-libcaf in favor of --with-caf, as already done in
    Broker. (Matthias Vallentin)

2.4-418 | 2016-03-21 12:22:15 -0700

  * Add protocol confirmation to MySQL analyzer. (Vlad Grigorescu)

  * Check that there is only one of &read_expire, &write_expire,
    &create_expire. (Johanna Amann)

  * Fixed &read_expire for subnet-indexed tables, plus test case. (Jan
    Grashoefer)

  * Add filter_subnet_table() that works similar to matching_subnet()
    but returns a filtered view of the original set/table only
    containing the changed subnets. (Jan Grashoefer)

  * Fix bug in tablue values' tracking read operations. (Johanna
    Amann)

  * Update TLS constants and extensions from IANA. (Johanna Amann)

2.4-406 | 2016-03-11 14:27:47 -0800

  * Add NetControl and OpenFlow frameworks.  (Johanna Amann)

2.4-313 | 2016-03-08 07:47:57 -0800

  * Remove old string functions in C++ code. This removes the
    functions: strcasecmp_n, strchr_n, and strrchr_n. (Johanna Amann)

2.4-307 | 2016-03-07 13:33:45 -0800

  * Add "disable_analyzer_after_detection" and remove
    "skip_processing_after_detection". Addresses BIT-1545.
    (Aaron Eppert & Johanna Amann)

  * Add bad_HTTP_request_with_version weird (William Glodek)

2.4-299 | 2016-03-04 12:51:55 -0800

  * More detailed installation instructions for FreeBSD 9.X. (Johanna Amann)

  * Update CMake OpenSSL checks. (Johanna Amann)

  * "SUBSCRIBE" is a valid SIP. message per RFC 3265. Addresses
     BIT-1529. (Johanna Amann)

  * Update documentation for connection log's RSTR. Addresses BIT-1535
    (Johanna Amann)

2.4-284 | 2016-02-17 14:12:15 -0800

  * Fix sometimes failing dump-events test. (Johanna Amann)

2.4-282 | 2016-02-13 10:48:21 -0800

  * Add missing break in in StartTLS case of IRC analyzer. Found by
    Aaron Eppert. (Johanna Amann)

2.4-280 | 2016-02-13 10:40:16 -0800

  * Fix memory leaks in stats.cc and smb.cc. (Johanna Amann)

2.4-278 | 2016-02-12 18:53:35 -0800

  * Better multi-space separator handline. (Mark Taylor & Johanna Amann)

2.4-276 | 2016-02-10 21:29:33 -0800

  * Allow IRC commands to not have parameters. (Mark Taylor)

2.4-272 | 2016-02-08 14:27:58 -0800

  * fix memory leaks in find_all() and IRC analyzer. (Dirk Leinenbach)

2.4-270 | 2016-02-08 13:00:57 -0800

  * Removed duplicate parameter for IRC "QUIT" event handler. (Mark Taylor)

2.4-267 | 2016-02-01 12:38:32 -0800

  * Add testcase for CVE-2015-3194. (Johanna Amann)

  * Fix portability issue with use of mktemp. (Daniel Thayer)

2.4-260 | 2016-01-28 08:05:27 -0800

  * Correct irc_privmsg_message event handling bug. (Mark Taylor)

  * Update copyright year for Sphinx. (Johanna Amann)

2.4-253 | 2016-01-20 17:41:20 -0800

  * Support of RadioTap encapsulation for 802.11 (Seth Hall)

    Radiotap support should be fully functional with Radiotap
    packets that include IPv4 and IPv6. Other radiotap packets are
    silently ignored.

2.4-247 | 2016-01-19 10:19:48 -0800

  * Fixing C++11 compiler warnings. (Seth Hall)

  * Updating plugin documentation building. (Johanna Amann)

2.4-238 | 2016-01-15 12:56:33 -0800

  * Add HTTP version information to HTTP log file. (Aaron Eppert)

  * Add NOTIFY as a valid SIP message, per RFC 3265. (Aaron Eppert)

  * Improve HTTP parser's handling of requests that don't have a URI.
    (William Glodek/Robin Sommer)

  * Fix crash when deleting non existing record member. Addresses
    BIT-1519. (Johanna Amann)

2.4-228 | 2015-12-19 13:40:09 -0800

  * Updating BroControl submodule.

2.4-227 | 2015-12-18 17:47:24 -0800

  * Update host name in windows-version-detection.bro. (Aaron Eppert)

  * Update installation instructions to mention OpenSSL dependency for
    newer OS X version. (Johanna Amann)

  * Change a stale bro-ids.org to bro.org. (Johanna Amann)

  * StartTLS support for IRC. (Johanna Amann)

  * Adding usage guard to canonifier script. (Robin Sommer)

2.4-217 | 2015-12-04 16:50:46 -0800

  * SIP scripts code cleanup. (Seth Hall)

     - Daniel Guerra pointed out a type issue for SIP request and
       response code length fields which is now corrected.

     - Some redundant code was removed.

     - if/else tree modified to use switch instead.

2.4-214 | 2015-12-04 16:40:15 -0800

  * Delaying BinPAC initializaton until afte plugins have been
    activated. (Robin Sommer)

2.4-213 | 2015-12-04 15:25:48 -0800

  * Use better data structure for storing BPF filters. (Robin Sommer)

2.4-211 | 2015-11-17 13:28:29 -0800

  * Making cluster reconnect timeout configurable. (Robin Sommer)

  * Bugfix for child process' communication loop. (Robin Sommer)

2.4-209 | 2015-11-16 07:31:22 -0800

  * Updating submodule(s).

2.4-207 | 2015-11-10 13:34:42 -0800

  * Fix to compile with OpenSSL that has SSLv3 disalbed. (Christoph
    Pietsch)

  * Fix potential race condition when logging VLAN info to conn.log.
    (Daniel Thayer)

2.4-201 | 2015-10-27 16:11:15 -0700

  * Updating NEWS. (Robin Sommer)

2.4-200 | 2015-10-26 16:57:39 -0700

  * Adding missing file. (Robin Sommer)

2.4-199 | 2015-10-26 16:51:47 -0700

  * Fix problem with the JSON Serialization code. (Aaron Eppert)

2.4-188 | 2015-10-26 14:11:21 -0700

  * Extending rexmit_inconsistency() event to receive an additional
    parameter with the packet's TCP flags, if available. (Robin
    Sommer)

2.4-187 | 2015-10-26 13:43:32 -0700

  * Updating NEWS for new plugins. (Robin Sommer)

2.4-186 | 2015-10-23 15:07:06 -0700

  * Removing pcap options for AF_PACKET support. Addresses BIT-1363.
    (Robin Sommer)

  * Correct a typo in controller.bro documentation. (Daniel Thayer)

  * Extend SSL DPD signature to allow alert before server_hello.
    (Johanna Amann)

  * Make join_string_vec work with vectors containing empty elements.
    (Johanna Amann)

  * Fix support for HTTP CONNECT when server adds headers to response.
    (Eric Karasuda).

  * Load static CA list for validation tests too. (Johanna Amann)

  * Remove cluster certificate validation script. (Johanna Amann)

  * Fix a bug in diff-remove-x509-names canonifier. (Daniel Thayer)

  * Fix test canonifiers in scripts/policy/protocols/ssl. (Daniel
    Thayer)

2.4-169 | 2015-10-01 17:21:21 -0700

  * Fixed parsing of V_ASN1_GENERALIZEDTIME timestamps in x509
    certificates. (Yun Zheng Hu)

  * Improve X509 end-of-string-check code. (Johanna Amann)

  * Refactor X509 generalizedtime support and test. (Johanna Amann)

  * Fix case of offset=-1 (EOF) for RAW reader. Addresses BIT-1479.
    (Johanna Amann)

  * Improve a number of test canonifiers. (Daniel Thayer)

  * Remove unnecessary use of TEST_DIFF_CANONIFIER. (Daniel Thayer)

  * Fixed some test canonifiers to read only from stdin

  * Remove unused test canonifier scripts. (Daniel Thayer)

  * A potpourri of updates and improvements across the documentation.
    (Daniel Thayer)

  * Add configure option to disable Broker Python bindings. Also
    improve the configure summary output to more clearly show whether
    or not Broker Python bindings will be built. (Daniel Thayer)

2.4-131 | 2015-09-11 12:16:39 -0700

  * Add README.rst symlink. Addresses BIT-1413 (Vlad Grigorescu)

2.4-129 | 2015-09-11 11:56:04 -0700

  * hash-all-files.bro depends on base/files/hash (Richard van den Berg)

  * Make dns_max_queries redef-able, and bump default to 25. Addresses
    BIT-1460 (Vlad Grigorescu)

2.4-125 | 2015-09-03 20:10:36 -0700

  * Move SIP analyzer to flowunit instead of datagram Addresses
    BIT-1458 (Vlad Grigorescu)

2.4-122 | 2015-08-31 14:39:41 -0700

  * Add a number of out-of-bound checks to layer 2 code. Addresses
    BIT-1463 (Johanna Amann)

  * Fix error in 2.4 release notes regarding SSH events. (Robin
    Sommer)

2.4-118 | 2015-08-31 10:55:29 -0700

  * Fix FreeBSD build errors (Johanna Amann)

2.4-117 | 2015-08-30 22:16:24 -0700

  * Fix initialization of a pointer in RDP analyzer. (Daniel
    Thayer/Robin Sommer)

2.4-115 | 2015-08-30 21:57:35 -0700

  * Enable Bro to leverage packet fanout mode on Linux. (Kris
    Nielander).

       ## Toggle whether to do packet fanout (Linux-only).
       const Pcap::packet_fanout_enable = F &redef;

       ## If packet fanout is enabled, the id to sue for it. This should be shared amongst
       ## worker processes processing the same socket.
       const Pcap::packet_fanout_id = 0 &redef;

       ## If packet fanout is enabled, whether packets are to be defragmented before
       ## fanout is applied.
       const Pcap::packet_fanout_defrag = T &redef;

  * Allow libpcap buffer size to be set via configuration. (Kris Nielander)

       ## Number of Mbytes to provide as buffer space when capturing from live
       ## interfaces.
       const Pcap::bufsize = 128 &redef;

  * Move the pcap-related script-level identifiers into the new Pcap
    namespace. (Robin Sommer)

        snaplen                  -> Pcap::snaplen
        precompile_pcap_filter() -> Pcap::precompile_pcap_filter()
        install_pcap_filter()    -> Pcap::install_pcap_filter()
        pcap_error()             -> Pcap::pcap_error()


2.4-108 | 2015-08-30 20:14:31 -0700

   * Update Base64 decoding.  (Jan Grashoefer)

        - A new built-in function, decode_base64_conn() for Base64
          decoding. It works like decode_base64() but receives an
          additional connection argument that will be used for
          reporting decoding errors into weird.log (instead of
          reporter.log).

        - FTP, POP3, and HTTP analyzers now likewise log Base64
          decoding errors to weird.log.

        - The built-in functions decode_base64_custom() and
          encode_base64_custom() are now deprecated. Their
          functionality is provided directly by decode_base64() and
          encode_base64(), which take an optional parameter to change
          the Base64 alphabet.

  * Fix potential crash if TCP header was captured incompletely.
    (Robin Sommer)

2.4-103 | 2015-08-29 10:51:55 -0700

  * Make ASN.1 date/time parsing more robust. (Johanna Amann)

  * Be more permissive on what characters we accept as an unquoted
    multipart boundary. Addresses BIT-1459. (Johanna Amann)

2.4-99 | 2015-08-25 07:56:57 -0700

  * Add ``Q`` and update ``I`` documentation for connection history
    field. Addresses BIT-1466. (Vlad Grigorescu)

2.4-96 | 2015-08-21 17:37:56 -0700

  * Update SIP analyzer. (balintm)

        - Allows space on both sides of ':'.
        - Require CR/LF after request/reply line.

2.4-94 | 2015-08-21 17:31:32 -0700

  * Add file type detection support for video/MP2T. (Mike Freemon)

2.4-93 | 2015-08-21 17:23:39 -0700

  * Make plugin install honor DESTDIR= convention. (Jeff Barber)

2.4-89 | 2015-08-18 07:53:36 -0700

  * Fix diff-canonifier-external to use basename of input file.
  (Daniel Thayer)

2.4-87 | 2015-08-14 08:34:41 -0700

  * Removing the yielding_teredo_decapsulation option. (Robin Sommer)

2.4-86 | 2015-08-12 17:02:24 -0700

  * Make Teredo DPD signature more precise. (Martina Balint)

2.4-84 | 2015-08-10 14:44:39 -0700

  * Add hook 'HookSetupAnalyzerTree' to allow plugins access to a
    connection's initial analyzer tree for customization. (James
    Swaro)

  * Plugins now look for a file "__preload__.bro" in the top-level
    script directory. If found, they load it first, before any scripts
    defining BiF elements. This can be used to define types that the
    BiFs already depend on (like a custom type for an event argument).
    (Robin Sommer)

2.4-81 | 2015-08-08 07:38:42 -0700

  * Fix a test that is failing very frequently. (Daniel Thayer)

2.4-78 | 2015-08-06 22:25:19 -0400

  * Remove build dependency on Perl (now requiring Python instad).
    (Daniel Thayer)

  * CID 1314754: Fixing unreachable code in RSH analyzer. (Robin
    Sommer)

  * CID 1312752: Add comment to mark 'case' fallthrough as ok. (Robin
    Sommer)

  * CID 1312751: Removing redundant assignment. (Robin Sommer)

2.4-73 | 2015-07-31 08:53:49 -0700

  * BIT-1429: SMTP logs now include CC: addresses. (Albert Zaharovits)

2.4-70 | 2015-07-30 07:23:44 -0700

  * Updated detection of Flash and AdobeAIR. (Jan Grashoefer)

  * Adding tests for Flash version parsing and browser plugin
    detection. (Robin Sommer)

2.4-63 | 2015-07-28 12:26:37 -0700

  * Updating submodule(s).

2.4-61 | 2015-07-28 12:13:39 -0700

  * Renaming config.h to bro-config.h. (Robin Sommer)

2.4-58 | 2015-07-24 15:06:07 -0700

  * Add script protocols/conn/vlan-logging.bro to record VLAN data in
    conn.log. (Aaron Brown)

  * Add field "vlan" and "inner_vlan" to connection record. (Aaron
    Brown)

  * Save the inner vlan in the Packet object for Q-in-Q setups. (Aaron
    Brown)

  * Increasing plugin API version for recent packet source changes.
    (Robin Sommer)

  * Slightly earlier protocol confirmation for POP3. (Johanna Amann)

2.4-46 | 2015-07-22 10:56:40 -0500

  * Fix broker python bindings install location to track --prefix.
    (Jon Siwek)

2.4-45 | 2015-07-21 15:19:43 -0700

  * Enabling Broker by default. This means CAF is now a required
    dependency, altjough for now at least, there's still a switch
    --disable-broker to turn it off.

  * Requiring a C++11 compiler, and turning on C++11 support. (Robin
    Sommer)

  * Tweaking the listing of hooks in "bro -NN" for consistency. (Robin
    Sommer)

2.4-41 | 2015-07-21 08:35:17 -0700

  * Fixing compiler warning. (Robin Sommer)

  * Updates to IANA TLS registry. (Johanna Amann)

2.4-38 | 2015-07-20 15:30:35 -0700

  * Refactor code to use a common Packet type throught. (Jeff
    Barber/Robin Sommer)

  * Extend parsing layer 2 and keeping track of layer 3 protoco. (Jeff Barber)

  * Add a raw_packet() event that generated for all packets and
    include layer 2 information. (Jeff Barber)

2.4-27 | 2015-07-15 13:31:49 -0700

  * Fix race condition in intel test. (Johanna Amann)

2.4-24 | 2015-07-14 08:04:11 -0700

  * Correct Perl package name on FreeBSD in documentation.(Justin Azoff)

  * Adding an environment variable to BTest configuration for external
    scripts. (Robin Sommer)

2.4-20 | 2015-07-03 10:40:21 -0700

  * Adding a weird for when truncated packets lead TCP reassembly to
    ignore content. (Robin Sommer)

2.4-19 | 2015-07-03 09:04:54 -0700

  * A set of tests exercising IP defragmentation and TCP reassembly.
    (Robin Sommer)

2.4-17 | 2015-06-28 13:02:41 -0700

  * BIT-1314: Add detection for Quantum Insert attacks. The TCP
    reassembler can now keep a history of old TCP segments using the
    tcp_max_old_segments option. An overlapping segment with different
    data will then generate an rexmit_inconsistency event. The default
    for tcp_max_old_segments is zero, which disabled any additional
    buffering. (Yun Zheng Hu/Robin Sommer)

2.4-14 | 2015-06-28 12:30:12 -0700

  * BIT-1400: Allow '<' and '>' in MIME multipart boundaries. The spec
    doesn't actually seem to permit these, but they seem to occur in
    the wild. (Jon Siwek)

2.4-12 | 2015-06-28 12:21:11 -0700

  * BIT-1399: Trying to decompress deflated HTTP content even when
    zlib headers are missing. (Seth Hall)

2.4-10 | 2015-06-25 07:11:17 -0700

  * Correct a name used in a header identifier (Justin Azoff)

2.4-8 | 2015-06-24 07:50:50 -0700

  * Restore the --load-seeds cmd-line option and enable the short
    options -G/-H for --load-seeds/--save-seeds. (Daniel Thayer)

2.4-6 | 2015-06-19 16:26:40 -0700

  * Generate protocol confirmations for Modbus, making it appear as a
    confirmed service in conn.log. (Seth Hall)

  * Put command line options in alphabetical order. (Daniel Thayer)

  * Removing dead code for no longer supported -G switch. (Robin
    Sommer) (Robin Sommer)

2.4 | 2015-06-09 07:30:53 -0700

  * Release 2.4.

  * Fixing tiny thing in NEWS. (Robin Sommer)

2.4-beta-42 | 2015-06-08 09:41:39 -0700

  * Fix reporter errors with GridFTP traffic. (Robin Sommer)

2.4-beta-40 | 2015-06-06 08:20:52 -0700

  * PE Analyzer: Change how we calculate the rva_table size. (Vlad Grigorescu)

2.4-beta-39 | 2015-06-05 09:09:44 -0500

  * Fix a unit test to check for Broker requirement. (Jon Siwek)

2.4-beta-38 | 2015-06-04 14:48:37 -0700

  * Test for Broker termination. (Robin Sommer)

2.4-beta-37 | 2015-06-04 07:53:52 -0700

  * BIT-1408: Improve I/O loop and Broker IOSource. (Jon Siwek)

2.4-beta-34 | 2015-06-02 10:37:22 -0700

  * Add signature support for F4M files. (Seth Hall)

2.4-beta-32 | 2015-06-02 09:43:31 -0700

  * A larger set of documentation updates, fixes, and extentions.
    (Daniel Thayer)

2.4-beta-14 | 2015-06-02 09:16:44 -0700

  * Add memleak btest for attachments over SMTP. (Vlad Grigorescu)

  * BIT-1410: Fix flipped tx_hosts and rx_hosts in files.log. Reported
    by Ali Hadi. (Vlad Grigorescu)

  * Updating the Mozilla root certs. (Seth Hall)

  * Updates for the urls.bro script. Fixes BIT-1404. (Seth Hall)

2.4-beta-6 | 2015-05-28 13:20:44 -0700

  * Updating submodule(s).

2.4-beta-2 | 2015-05-26 08:58:37 -0700

  * Fix segfault when DNS is not available. Addresses BIT-1387. (Frank
    Meier and Robin Sommer)

2.4-beta | 2015-05-07 21:55:31 -0700

  * Release 2.4-beta.

  * Update local-compat.test (Johanna Amann)

2.3-913 | 2015-05-06 09:58:00 -0700

  * Add /sbin to PATH in btest.cfg and remove duplicate default_path.
    (Daniel Thayer)

2.3-911 | 2015-05-04 09:58:09 -0700

  * Update usage output and list of command line options. (Daniel
    Thayer)

  * Fix to ssh/geo-data.bro for unset directions. (Vlad Grigorescu)

  * Improve SIP logging and remove reporter messages. (Seth Hall)

2.3-905 | 2015-04-29 17:01:30 -0700

  * Improve SIP logging and remove reporter messages. (Seth Hall)

2.3-903 | 2015-04-27 17:27:59 -0700

  * BIT-1350: Improve record coercion type checking. (Jon Siwek)

2.3-901 | 2015-04-27 17:25:27 -0700

  * BIT-1384: Remove -O (optimize scripts) command-line option, which
    hadn't been working for a while already. (Jon Siwek)

2.3-899 | 2015-04-27 17:22:42 -0700

  * Fix the -J/--set-seed cmd-line option. (Daniel Thayer)

  * Remove unused -l, -L, and -Z cmd-line options. (Daniel Thayer)

2.3-892 | 2015-04-27 08:22:22 -0700

  * Fix typos in the Broker BIF documentation. (Daniel Thayer)

  * Update installation instructions and remove outdated references.
    (Johanna Amann)

  * Easier support for systems with tcmalloc_minimal installed. (Seth
    Hall)

2.3-884 | 2015-04-23 12:30:15 -0500

  * Fix some outdated documentation unit tests. (Jon Siwek)

2.3-883 | 2015-04-23 07:10:36 -0700

  * Fix -N option to work with builtin plugins as well. (Robin Sommer)

2.3-882 | 2015-04-23 06:59:40 -0700

  * Add missing .pac dependencies for some binpac analyzer targets.
    (Jon Siwek)

2.3-879 | 2015-04-22 10:38:07 -0500

  * Fix compile errors. (Jon Siwek)

2.3-878 | 2015-04-22 08:21:23 -0700

  * Fix another compiler warning in DTLS. (Johanna Amann)

2.3-877 | 2015-04-21 20:14:16 -0700

  * Adding missing include. (Robin Sommer)

2.3-876 | 2015-04-21 16:40:10 -0700

  * Attempt at fixing a potential std::length_error exception in RDP
    analyzer. Addresses BIT-1337. (Robin Sommer)

  * Fixing compile problem caused by overeager factorization. (Robin
    Sommer)

2.3-874 | 2015-04-21 16:09:20 -0700

  * Change details of escaping when logging/printing. (Seth Hall/Robin
    Sommer)

        - Log files now escape non-printable characters consistently
          as "\xXX'. Furthermore, backslashes are escaped as "\\",
          making the representation fully reversible.

        - When escaping via script-level functions (escape_string,
          clean), we likewise now escape consistently with "\xXX" and
          "\\".

        - There's no "alternative" output style anymore, i.e., fmt()
          '%A' qualifier is gone.

    Addresses BIT-1333.

  * Remove several BroString escaping methods that are no longer
    useful. (Seth Hall)

2.3-864 | 2015-04-21 15:24:02 -0700

  * A SIP protocol analyzer. (Vlad Grigorescu)

	Activity gets logged into sip.log. It generates the following
    events:

	    event sip_request(c: connection, method: string, original_URI: string, version: string);
        event sip_reply(c: connection, version: string, code: count, reason: string);
        event sip_header(c: connection, is_orig: bool, name: string, value: string);
        event sip_all_headers(c: connection, is_orig: bool, hlist: mime_header_list);
        event sip_begin_entity(c: connection, is_orig: bool);
        event sip_end_entity(c: connection, is_orig: bool);

    The analyzer support SIP over UDP currently.

  * BIT-1343: Factor common ASN.1 code from RDP, SNMP, and Kerberos
    analyzers. (Jon Siwek/Robin Sommer)

2.3-838 | 2015-04-21 13:40:12 -0700

  * BIT-1373: Fix vector index assignment reference count bug. (Jon Siwek)

2.3-836 | 2015-04-21 13:37:31 -0700

  * Fix SSH direction field being unset. Addresses BIT-1365. (Vlad
    Grigorescu)

2.3-835 | 2015-04-21 16:36:00 -0500

  * Clarify Broker examples. (Jon Siwek)

2.3-833 | 2015-04-21 12:38:32 -0700

  * A Kerberos protocol analyzer. (Vlad Grigorescu)

	Activity gets logged into kerberos.log. It generates the following
    events:

        event krb_as_request(c: connection, msg: KRB::KDC_Request);
        event krb_as_response(c: connection, msg: KRB::KDC_Response);
        event krb_tgs_request(c: connection, msg: KRB::KDC_Request);
        event krb_tgs_response(c: connection, msg: KRB::KDC_Response);
        event krb_ap_request(c: connection, ticket: KRB::Ticket, opts: KRB::AP_Options);
        event krb_priv(c: connection, is_orig: bool);
        event krb_safe(c: connection, is_orig: bool, msg: KRB::SAFE_Msg);
        event krb_cred(c: connection, is_orig: bool, tickets: KRB::Ticket_Vector);
        event krb_error(c: connection, msg: KRB::Error_Msg);

2.3-793 | 2015-04-20 20:51:00 -0700

  * Add decoding of PROXY-AUTHORIZATION header to HTTP analyze,
    treating it the same as AUTHORIZATION. (Josh Liburdi)

  * Remove deprecated fields "hot" and "addl" from the connection
    record. Remove the functions append_addl() and
    append_addl_marker(). (Robin Sommer)

  * Removing the NetFlow analyzer, which hasn't been used anymore
    since then corresponding command-line option went away. (Robin
    Sommer)

2.3-787 | 2015-04-20 19:15:23 -0700

  * A file analyzer for Portable Executables. (Vlad Grigorescu/Seth
    Hall).

	Activity gets logged into pe.log. It generates the following
	events:

        event pe_dos_header(f: fa_file, h: PE::DOSHeader);
        event pe_dos_code(f: fa_file, code: string);
        event pe_file_header(f: fa_file, h: PE::FileHeader);
        event pe_optional_header(f: fa_file, h: PE::OptionalHeader);
        event pe_section_header(f: fa_file, h: PE::SectionHeader);

2.3-741 | 2015-04-20 13:12:39 -0700

  * API changes to file analysis mime type detection. Removed
    "file_mime_type" and "file_mime_types" event, replacing them with
    a new event called "file_metadata_inferred". Addresses BIT-1368.
    (Jon Siwek)

  * A large series of improvements for file type identification. This
    inludes a many signature updates (new types, cleanup, performance
    improvments) and splitting out signatures into subfiles. (Seth
    Hall)

  * Fix an issue with files having gaps before the bof_buffer is
    filled, which could lead to file type identification not working
    correctly. (Seth Hall)

  * Fix an issue with packet loss in HTTP file reporting for file type
    identification wasn't working correctly zero-length bodies. (Seth
    Hall)

  * X.509 certificates are now populating files.log with the mime type
    application/pkix-cert. (Seth Hall)

  * Normalized some FILE_ANALYSIS debug messages. (Seth Hall)

2.3-725 | 2015-04-20 12:54:54 -0700

  * Updating submodule(s).

2.3-724 | 2015-04-20 14:11:02 -0500

  * Fix uninitialized field in raw input reader. (Jon Siwek)

2.3-722 | 2015-04-20 12:59:03 -0500

  * Remove unneeded documentation cross-referencing. (Jon Siwek)

2.3-721 | 2015-04-20 12:47:05 -0500

  * BIT-1380: Improve Broxygen output of &default expressions.
    (Jon Siwek)

2.3-720 | 2015-04-17 14:18:26 -0700

  * Updating NEWS.

2.3-716 | 2015-04-17 13:06:37 -0700

  * Add seeking functionality to raw reader. One can now add an option
    "offset" to the config map. Positive offsets are interpreted to be
    from the beginning of the file, negative from the end of the file
    (-1 is end of file). Only works for raw reader in streaming or
    manual mode. Does not work with executables. Addresses BIT-985.
    (Johanna Amann)

  * Allow setting packet and byte thresholds for connections. (Johanna Amann)

    This extends the ConnSize analyzer to be able to raise events when
    each direction of a connection crosses a certain amount of bytes
    or packets.

    Thresholds are set using:
        - set_conn_bytes_threshold(c$id, [num-bytes], [direction]);
        - set_conn_packets_threshold(c$id, [num-packets], [direction]);

    They raise the events, respectively:
        - event conn_bytes_threshold_crossed(c: connection, threshold: count, is_orig: bool)
        - event conn_packets_threshold_crossed(c: connection, threshold: count, is_orig: bool)

    Current thresholds can be examined using get_conn_bytes_threshold()
    and get_conn_packets_threshold().

    Only one threshold can be set per connection.

  * Add high-level API for packet/bytes thresholding in
    base/protocols/conn/thresholds.bro that holds lists of thresholds
    and raises an event for each threshold exactly once. (Johanna
    Amann)

  * Fix a bug where child packet analyzers of the TCP analyzer
    where not found using FindChild.

  * Update GridFTP analyzer to use connection thresholding instead of
    polling. (Johanna Amann)

2.3-709 | 2015-04-17 12:37:32 -0700

  * Fix addressing the dreaded "internal error: unknown msg type 115
  in Poll()". (Jon Siwek)

    This patch removes the error handling code for overload conditions
    in the main process that could cause trouble down the road. The
    "chunked_io_buffer_soft_cap" script variable can now tune when the
    client process begins shutting down peer connections, and the
    default setting is now double what it used to be. Addresses
    BIT-1376.

2.3-707 | 2015-04-17 10:57:59 -0500

  * Add more info about Broker to NEWS. (Jon Siwek)

2.3-705 | 2015-04-16 08:16:45 -0700

  * Update Mozilla CA list. (Johanna Amann)

  * Update tests to have them keep using older certificates where
    appropiate. (Johanna Amann)

2.3-699 | 2015-04-16 09:51:58 -0500

  * Fix the to_count function to use strtoull versus strtoll.
    (Jon Siwek)

2.3-697 | 2015-04-15 09:51:15 -0700

  * Removing error check verifying that an ASCII writer has been
    properly finished. Instead of aborting, we now just clean up in
    that case and proceed. Addresses BIT-1331. (Robin Sommer)

2.3-696 | 2015-04-14 15:56:36 -0700

  * Update sqlite to 3.8.9

2.3-695 | 2015-04-13 10:34:42 -0500

  * Fix iterator invalidation in broker::Manager dtor. (Jon Siwek)

  * Add paragraph to plugin documentation. (Robin Sommer)

2.3-693 | 2015-04-11 10:56:31 -0700

  * BIT-1367: improve coercion of anonymous records in set constructor.
    (Jon Siwek)

  * Allow to specify ports for sftp log rotator. (Johanna Amann)

2.3-690 | 2015-04-10 21:51:10 -0700

  * Make sure to always delete the remote serializer. Addresses
    BIT-1306 and probably also BIT-1356. (Robin Sommer)

  * Cleaning up --help. -D and -Y/y were still listed, even though
    they had no effect anymore. Removing some dead code along with -D.
    Addresses BIT-1372. (Robin Sommer)

2.3-688 | 2015-04-10 08:10:44 -0700

  * Update SQLite to 3.8.8.3.

2.3-687 | 2015-04-10 07:32:52 -0700

  * Remove stale signature benchmarking code (-L command-line option).
    (Jon Siwek)

  * BIT-844: fix UDP payload signatures to match packet-wise. (Jon
    Siwek)

2.3-682 | 2015-04-09 12:07:00 -0700

  * Fixing input readers' component type. (Robin Sommer)

  * Tiny spelling correction. (Seth Hall)

2.3-680 | 2015-04-06 16:02:43 -0500

  * BIT-1371: remove CMake version check from binary package scripts.
    (Jon Siwek)

2.3-679 | 2015-04-06 10:16:36 -0500

  * Increase some unit test timeouts. (Jon Siwek)

  * Fix Coverity warning in RDP analyzer. (Jon Siwek)

2.3-676 | 2015-04-02 10:10:39 -0500

  * BIT-1366: improve checksum offloading warning.
    (Frank Meier, Jon Siwek)

2.3-675 | 2015-03-30 17:05:05 -0500

  * Add an RDP analyzer. (Josh Liburdi, Seth Hall, Johanna Amann)

2.3-640 | 2015-03-30 13:51:51 -0500

  * BIT-1359: Limit maximum number of DTLS fragments to 30. (Johanna Amann)

2.3-637 | 2015-03-30 12:02:07 -0500

  * Increase timeout duration in some broker tests. (Jon Siwek)

2.3-636 | 2015-03-30 11:26:32 -0500

  * Updates related to SSH analysis. (Jon Siwek)

    - Some scripts used wrong SSH module/namespace scoping on events.
    - Fix outdated notice documentation related to SSH password guessing.
    - Add a unit test for SSH pasword guessing notice.

2.3-635 | 2015-03-30 11:02:45 -0500

  * Fix outdated documentation unit tests. (Jon Siwek)

2.3-634 | 2015-03-30 10:22:45 -0500

  * Add a canonifier to a unit test's output. (Jon Siwek)

2.3-633 | 2015-03-25 18:32:59 -0700

  * Log::write in signature framework was missing timestamp.
    (Andrew Benson/Michel Laterman)

2.3-631 | 2015-03-25 11:03:12 -0700

  * New SSH analyzer. (Vlad Grigorescu)

2.3-600 | 2015-03-25 10:23:46 -0700

  * Add defensive checks in code to calculate log rotation intervals.
    (Pete Nelson).

2.3-597 | 2015-03-23 12:50:04 -0700

  * DTLS analyzer. (Johanna Amann)

  * Implement correct parsing of TLS record fragmentation. (Johanna
    Amann)

2.3-582 | 2015-03-23 11:34:25 -0700

  * BIT-1313: In debug builds, "bro -B <x>" now supports "all" and
    "help" for "<x>". "all" enables all debug streams. "help" prints a
    list of available debug streams. (John Donnelly/Robin Sommer).

  * BIT-1324: Allow logging filters to inherit default path from
    stream. This allows the path for the default filter to be
    specified explicitly through $path="..." when creating a stream.
    Adapted the existing Log::create_stream calls to explicitly
    specify a path value. (Jon Siwek)

  * BIT-1199: Change the way the input framework deals with values it
    cannot convert into BroVals, raising error messages instead of
    aborting execution. (Johanna Amann)

  * BIT-788: Use DNS QR field to better identify flow direction. (Jon
    Siwek)

2.3-572 | 2015-03-23 13:04:53 -0500

  * BIT-1226: Fix an example in quickstart docs. (Jon siwek)

2.3-570 | 2015-03-23 09:51:20 -0500

  * Correct a spelling error (Daniel Thayer)

  * Improvement to SSL analyzer failure mode. (Johanna Amann)

2.3-565 | 2015-03-20 16:27:41 -0500

  * BIT-978: Improve documentation of 'for' loop iterator invalidation.
    (Jon Siwek)

2.3-564 | 2015-03-20 11:12:02 -0500

  * BIT-725: Remove "unmatched_HTTP_reply" weird. (Jon Siwek)

2.3-562 | 2015-03-20 10:31:02 -0500

  * BIT-1207: Add unit test to catch breaking changes to local.bro
    (Jon Siwek)

  * Fix failing sqlite leak test (Johanna Amann)

2.3-560 | 2015-03-19 13:17:39 -0500

  * BIT-1255: Increase default values of
    "tcp_max_above_hole_without_any_acks" and "tcp_max_initial_window"
    from 4096 to 16384 bytes. (Jon Siwek)

2.3-559 | 2015-03-19 12:14:33 -0500

  * BIT-849: turn SMTP reporter warnings into weirds,
    "smtp_nested_mail_transaction" and "smtp_unmatched_end_of_data".
    (Jon Siwek)

2.3-558 | 2015-03-18 22:50:55 -0400

  * DNS: Log the type number for the DNS_RR_unknown_type weird. (Vlad Grigorescu)

2.3-555 | 2015-03-17 15:57:13 -0700

  * Splitting test-all Makefile target into Bro tests and test-aux.
    (Robin Sommer)

2.3-554 | 2015-03-17 15:40:39 -0700

  * Deprecate &rotate_interval, &rotate_size, &encrypt. Addresses
    BIT-1305. (Jon Siwek)

2.3-549 | 2015-03-17 09:12:18 -0700

  * BIT-1077: Fix HTTP::log_server_header_names. Before, it just
    re-logged fields from the client side. (Jon Siwek)

2.3-547 | 2015-03-17 09:07:51 -0700

  * Update certificate validation script to cache valid intermediate
    chains that it encounters on the wire and use those to try to
    validate chains that might be missing intermediate certificates.
    (Johanna Amann)

2.3-541 | 2015-03-13 15:44:08 -0500

  * Make INSTALL a symlink to doc/install/install.rst (Jon siwek)

  * Fix Broxygen coverage. (Jon Siwek)

2.3-539 | 2015-03-13 14:19:27 -0500

  * BIT-1335: Include timestamp in default extracted file names.
    And add a policy script to extract all files. (Jon Siwek)

  * BIT-1311: Identify GRE tunnels as Tunnel::GRE, not Tunnel::IP.
    (Jon Siwek)

  * BIT-1309: Add Connection class getter methods for flow labels.
    (Jon Siwek)

2.3-536 | 2015-03-12 16:16:24 -0500

  * Fix Broker leak tests. (Jon Siwek)

2.3-534 | 2015-03-12 10:59:49 -0500

  * Update NEWS file. (Jon Siwek)

2.3-533 | 2015-03-12 10:18:53 -0500

  * Give broker python bindings default install path within --prefix.
    (Jon Siwek)

2.3-530 | 2015-03-10 13:22:39 -0500

  * Fix broker data stores in absence of --enable-debug. (Jon Siwek)

2.3-529 | 2015-03-09 13:14:27 -0500

  * Fix format specifier in SSL protocol violation. (Jon Siwek)

2.3-526 | 2015-03-06 12:48:49 -0600

  * Fix build warnings, clarify broker requirements, update submodule.
    (Jon Siwek)

  * Rename comm/ directories to broker/ (Jon Siwek)

  * Rename broker-related namespaces. (Jon Siwek)

  * Improve remote logging via broker by only sending fields w/ &log.
    (Jon Siwek)

  * Disable a stream's remote logging via broker if it fails. (Jon Siwek)

  * Improve some broker communication unit tests. (Jon Siwek)

2.3-518 | 2015-03-04 13:13:50 -0800

  * Add bytes_recvd to stats.log recording the number of bytes
    received, according to packet headers. (Mike Smiley)

2.3-516 | 2015-03-04 12:30:06 -0800

  * Extract most specific Common Name from SSL certificates (Johanna
    Amann)

  * Send CN and SAN fields of SSL certificates to the Intel framework.
    (Johanna Amann)

2.3-511 | 2015-03-02 18:07:17 -0800

  * Changes to plugin meta hooks for function calls. (Gilbert Clark)

        - Add frame argument.

        - Change return value to tuple unambigiously whether hook
          returned a result.

2.3-493 | 2015-03-02 17:17:32 -0800

  * Extend the SSL weak-keys policy file to also alert when
    encountering SSL connections with old versions as well as unsafe
    cipher suites. (Johanna Amann)

  * Make the notice suppression handling of other SSL policy files a
    tad more robust. (Johanna Amann)

2.3-491 | 2015-03-02 17:12:56 -0800

  * Updating docs for recent addition of local_resp. (Robin Sommer)

2.3-489 | 2015-03-02 15:29:30 -0800

  * Integrate Broker, Bro's new communication library. (Jon Siwek)

    See aux/broker/README for more information on Broker, and
    doc/frameworks/comm.rst for the corresponding Bro script API.

    Broker support is by default off for now; it can be enabled at
    configure time with --enable-broker. It requires CAF
    (https://github.com/actor-framework/actor-framework); for now iot
    needs CAF's "develop" branch. Broker also requires a C++11
    compiler.

    Broker will become a mandatory dependency in future Bro versions.

  * Add --enable-c++11 configure flag to compile Bro's source code in
    C++11 mode with a corresponding compiler. (Jon Siwek)

2.3-451 | 2015-02-24 16:37:08 -0800

  * Updating submodule(s).

2.3-448 | 2015-02-23 16:58:10 -0800

  * Updating NEWS. (Robin Sommer)

2.3-447 | 2015-02-23 16:28:30 -0800

  * Fix potential crash in logging framework when deserializing
    WriterInfo from remote. where config is present. Testcase crashes
    on unpatched versions of Bro. (Aaron Eppert)

  * Fix wrong value test in WriterBackend. (Aaron Eppert)

2.3-442 | 2015-02-23 13:29:30 -0800

  * Add a "local_resp" field to conn.log, along the lines of the
    existing "local_orig". (Mike Smiley)

2.3-440 | 2015-02-23 11:39:17 -0600

  * Updating plugin docs to recent changes. (Robin Sommer)

  * Updating plugin tests to recent changes. (Robin Sommer)

  * Making plugin names case-insensitive for some internal comparisions.
    Makes plugin system more tolerant against spelling inconsistencies
    are hard to catch otherwise. (Robin Sommer)

  * Explicitly removing some old scripts on install that have moved
    into plugins to prevent them causing confusion. (Robin Sommer)

  * BIT-1312: Removing setting installation plugin path from
    bro-path-dev.sh.  Also, adding to existing BRO_PLUGIN_PATH rather
    than replacing. (Robin Sommer)

  * Creating the installation directory for plugins at install time.
    (Robin Sommer)

2.3-427 | 2015-02-20 13:49:33 -0800

  * Removing dependency on PCAP_NETMASK_UNKNOWN to compile with
    libpcap < 1.1.1. (Robin Sommer)

2.3-426 | 2015-02-20 12:45:51 -0800

  * Add 'while' statement to Bro language. Really. (Jon Siwek)

2.3-424 | 2015-02-20 12:39:10 -0800

  * Add the ability to remove surrounding braces from the JSON
    formatter. (Seth Hall)

2.3-419 | 2015-02-13 09:10:44 -0600

  * BIT-1011: Update the SOCKS analyzer to support user/pass login.
    (Nicolas Retrain, Seth Hall, Jon Siwek)

    - Add a new field to socks.log: "password".
    - Two new events: "socks_login_userpass_request" and
      "socks_login_userpass_reply".
    - Two new weirds for unsupported SOCKS authentication method or
      version.
    - A new test for authenticated socks traffic.

2.3-416 | 2015-02-12 12:18:42 -0600

  * Submodule update - newest sqlite version (Johanna Amann)

  * Fix use of deprecated gperftools headers. (Jon Siwek)

2.3-413 | 2015-02-08 18:23:05 -0800

  * Fixing analyzer tag types for some Files::* functions. (Robin Sommer)

  * Changing load order for plugin scripts. (Robin Sommer)

2.3-411 | 2015-02-05 10:05:48 -0600

  * Fix file analysis of files with total size below the bof_buffer size
    never delivering content to stream analyzers. (Seth Hall)

  * Add/fix log fields in x509 diff canonifier. (Jon Siwek)

  * "id" not defined for debug code when using -DPROFILE_BRO_FUNCTIONS
    (Mike Smiley)

2.3-406 | 2015-02-03 17:02:45 -0600

  * Add x509 canonifier to a unit test. (Jon Siwek)

2.3-405 | 2015-02-02 11:14:24 -0600

  * Fix memory leak in new split_string* functions. (Jon Siwek)

2.3-404 | 2015-01-30 14:23:27 -0800

  * Update documentation (broken links, outdated tests). (Jon Siwek)

  * Deprecate split* family of BIFs. (Jon Siwek)

    These functions are now deprecated in favor of alternative versions that
    return a vector of strings rather than a table of strings.

    Deprecated functions:

    - split: use split_string instead.
    - split1: use split_string1 instead.
    - split_all: use split_string_all instead.
    - split_n: use split_string_n instead.
    - cat_string_array: see join_string_vec instead.
    - cat_string_array_n: see join_string_vec instead.
    - join_string_array: see join_string_vec instead.
    - sort_string_array: use sort instead instead.
    - find_ip_addresses: use extract_ip_addresses instead.

    Changed functions:

    - has_valid_octets: uses a string_vec parameter instead of string_array.

    Addresses BIT-924.

  * Add a new attribute: &deprecated. While scripts are parsed, a
    warning is raised for each usage of an identifier marked as
    &deprecated.  This also works for BIFs. Addresses BIT-924,
    BIT-757. (Jon Siwek)

2.3-397 | 2015-01-27 10:13:10 -0600

  * Handle guess_lexer exceptions in pygments reST directive (Jon Siwek)

2.3-396 | 2015-01-23 10:49:15 -0600

  * DNP3: fix reachable assertion and buffer over-read/overflow.
    CVE number pending. (Travis Emmert, Jon Siwek)

  * Update binpac: Fix potential out-of-bounds memory reads in generated
    code. CVE-2014-9586. (John Villamil and Chris Rohlf - Yahoo
    Paranoids, Jon Siwek)

  * Fixing (harmless) Coverity warning. (Robin Sommer)

2.3-392 | 2015-01-15 09:44:15 -0800

  * Small changes to EC curve names in a newer draft. (Johanna Amann)

2.3-390 | 2015-01-14 13:27:34 -0800

  * Updating MySQL analyses. (Vlad Grigorescu)
     - Use a boolean success instead of a result string.
     - Change the affected_rows response detail string to a "rows" count.
     - Fix the state tracking to log incomplete command.

  * Extend DNP3 to support communication over UDP. (Hui Lin)

  * Fix a bug in DNP3 determining the length of an object in some
    cases. (Hui Lin)

2.3-376 | 2015-01-12 09:38:10 -0600

  * Improve documentation for connection_established event. (Jon Siwek)

2.3-375 | 2015-01-08 13:10:09 -0600

  * Increase minimum required CMake version to 2.8. (Jon Siwek)

2.3-374 | 2015-01-07 10:03:17 -0600

  * Improve documentation of the Intelligence Framework. (Daniel Thayer)

2.3-371 | 2015-01-06 09:58:09 -0600

  * Update/improve file mime type identification. (Seth Hall)

     - Change to the default BOF buffer size to 3000 (was 1024).

     - Reorganized MS signatures into a separate file.

     - Remove all of the x-c detections.  Nearly all false positives.

     - Improve TAR detections, removing old, back up TAR detections.

     - Remove one of the x-elc detections that was too loose
       and caused many false positives.

     - Improved lots of the signatures and added new ones. (Seth Hall)

  * Add support for file reassembly in the file analysis framework
    (Seth Hall, Jon Siwek).

     - The reassembly behavior can be modified per-file by enabling or
       disabling the reassembler and/or modifying the size of the
       reassembly buffer.

     - Changed the file extraction analyzer to use stream-wise input to
       avoid issues with the chunk-wise approach not immediately
       triggering the file_new event due to mime-type detection delay.
       Before, early chunks frequently ended up lost.  Extraction also
       will now explicitly NUL-fill gaps in the file instead of
       implicitly relying on pwrite to do it.

2.3-349 | 2015-01-05 15:21:13 -0600

  * Fix race condition in unified2 file analyzer startup. (Jon siwek)

2.3-348 | 2014-12-31 09:19:34 -0800

  * Changing Makefile's test-all to run test-all for broctl, which now
    executes trace-summary tests as well. (Robin Sommer)

2.3-345 | 2014-12-31 09:06:15 -0800

  * Correct a typo in the Notice framework doc. (Daniel Thayer)

2.3-343 | 2014-12-12 12:43:46 -0800

  * Fix PIA packet replay to deliver copy of IP header. This prevented
    one from writing a packet-wise analyzer that needs access to IP
    headers and can be attached to a connection via signature match.
    Addresses BIT-1298 (Jon Siwek)

2.3-338 | 2014-12-08 13:56:19 -0800

  * Add man page for Bro. (Raúl Benencia)

  * Updating doc baselines. (Robin Sommer)

2.3-334 | 2014-12-03 14:22:07 -0800

  * Fix compound assignment to require proper L-value. Addresses
    BIT-1295. (Jon Siwek)

2.3-332 | 2014-12-03 14:14:11 -0800

  * Make using local IDs in @if directives an error. Addresses
    BIT-1296. (Jon Siwek)

2.3-330 | 2014-12-03 14:10:39 -0800

  * Fix some "make doc" warnings and update some doc tests. (Daniel
    Thayer)

2.3-328 | 2014-12-02 08:13:10 -0500

  * Update windows-version-detection.bro to add support for
    Windows 10. (Michal Purzynski)

2.3-326 | 2014-12-01 12:10:27 -0600

  * BIFScanner: fix invalid characters in generated preprocessor macros.
    (Hilko Bengen)

  * BIT-1294: fix exec.bro from mutating Input::end_of_data event
    parameters. (Johanna Amann)

  * Add/invoke "distclean" for testing directories. (Raúl Benencia)

  * Delete prebuilt python bytecode files from git. (Jon Siwek)

  * Add Windows detection based on CryptoAPI HTTP traffic as a software
    framework policy script. (Vlad Grigorescu)

2.3-316 | 2014-11-25 17:35:06 -0800

  * Make the SSL analyzer skip further processing once encountering
    situations which are very probably non-recoverable. (Johanna
    Amann)

2.3-313 | 2014-11-25 14:27:07 -0800

  * Make SSL v2 protocol tests more strict. In its former state they
    triggered on http traffic over port 443 sometimes. Found by Michał
    Purzyński. (Johanna Amann)

  * Fix X509 analyzer to correctly return ECDSA as the key_type for
    ECDSA certs. Bug found by Michał Purzyński. (Johanna Amann)

2.3-310 | 2014-11-19 10:56:59 -0600

  * Disable verbose bison output. (Jon Siwek)

2.3-309 | 2014-11-18 12:17:53 -0800

  * New decompose_uri() function in base/utils/urls that splits a URI
    into its pieces. (Anthony Kasza).

2.3-305 | 2014-11-18 11:09:04 -0800

  * Improve coercion of &default expressions. Addresses BIT-1288. (Jon
    Siwek)

2.3-303 | 2014-11-18 10:53:04 -0800

  * For DH key exchanges, use p as the parameter for weak key
    exchanges. (Johanna Amann)

2.3-301 | 2014-11-11 13:47:27 -0800

  * Add builtin function enum_to_int() that converts an enum into a
    integer. (Christian Struck)

2.3-297 | 2014-11-11 11:50:47 -0800

  * Removing method from SSL analyzer that's no longer used. (Robin
    Sommer)

2.3-296 | 2014-11-11 11:42:38 -0800

  * A new analyzer parsing the MySQL wire protocol. Activity gets
    logged into mysql.log. Supports protocol versions 9 and 10. (Vlad
    Grigorescu)

2.3-280 | 2014-11-05 09:46:33 -0500

  * Add Windows detection based on CryptoAPI HTTP traffic as a
    software framework policy script. (Vlad Grigorescu)

2.3-278 | 2014-11-03 18:55:18 -0800

  * Add new curves from draft-ietf-tls-negotiated-ff-dhe to SSL
    analysis. (Johanna Amann)

2.3-274 | 2014-10-31 17:45:25 -0700

  * Adding call to new binpac::init() function. (Robin Sommer)

2.3-272 | 2014-10-31 16:29:42 -0700

  * Fix segfault if when statement's RHS is unitialized. Addresses
    BIT-1176. (Jon Siwek)

  * Fix checking vector indices via "in". Addresses BIT-1280.  (Jon
    Siwek)

2.3-268 | 2014-10-31 12:12:22 -0500

  * BIT-1283: Fix crash when using &encrypt. (Jon Siwek)

2.3-267 | 2014-10-31 10:35:02 -0500

  * BIT-1284: Allow arbitrary when statement timeout expressions
    (Jon Siwek)

2.3-266 | 2014-10-31 09:21:28 -0500

  * BIT-1166: Add configure options to fine tune local state dirs used
    by BroControl. (Jon Siwek)

2.3-264 | 2014-10-30 13:25:57 -0500

  * Fix some minor Coverity Scan complaints. (Jon Siwek)

2.3-263 | 2014-10-28 15:09:10 -0500

  * Fix checking of fwrite return values (Johanna Amann)

2.3-260 | 2014-10-27 12:54:17 -0500

  * Fix errors/warnings when compiling with -std=c++11 (Jon Siwek)

2.3-259 | 2014-10-27 10:04:04 -0500

  * Documentation fixes. (Vicente Jimenez Aguilar and Stefano Azzalini)

2.3-256 | 2014-10-24 15:33:45 -0700

  * Adding missing test baseline. (Robin Sommer)

2.3-255 | 2014-10-24 13:39:44 -0700

  * Fixing unstable active-http test. (Robin Sommer)

2.3-254 | 2014-10-24 11:40:51 -0700

  * Fix active-http.bro to deal reliably with empty server responses,
    which will now be passed back as empty files. (Christian Struck)

2.3-248 | 2014-10-23 14:20:59 -0700

  * Change order in which a plugin's scripts are loaded at startup.
    (Robin Sommer)

2.3-247 | 2014-10-21 13:42:38 -0700

  * Updates to the SSL analyzer. (Johanna Amann)

        * Mark everything below 2048 bit as a weak key.

        * Fix notice suppression.

        * Add information about server-chosen protocol to ssl.log, if
          provided by application_layer_next_protocol.

        * Add boolean flag to ssl.log signaling if a session was
          resumed. Remove the (usually not really that useful) session
          ID that the client sent.

2.3-240 | 2014-10-21 13:36:33 -0700

  * Fix Coverity-reported issues in DNP3 analyzer. (Seth Hall)

2.3-238 | 2014-10-16 06:51:49 -0700

  * Fix multipart HTTP/MIME entity file analysis so that (1) singular
    CR or LF characters in multipart body content are no longer
    converted to a full CRLF (thus corrupting the file) and (2) it
    also no longer considers the CRLF before the multipart boundary as
    part of the content. Addresses BIT-1235. (Jon Siwek)

2.3-235 | 2014-10-15 10:20:47 -0500

  * BIT-1273: Add error message for bad enum declaration syntax.
    (Jon Siwek)

2.3-234 | 2014-10-14 14:42:09 -0500

  * Documentation fixes. (Steve Smoot)

2.3-233 | 2014-10-09 16:00:27 -0500

  * Change find-bro-logs unit test to follow symlinks. (Jon Siwek)

  * Add error checks and messages to a test script (Daniel Thayer)

2.3-230 | 2014-10-08 08:15:17 -0700

  * Further baseline normalization for plugin test portability. (Robin
    Sommer)

2.3-229 | 2014-10-07 20:18:11 -0700

  * Fix for test portability. (Robin Sommer)

2.3-228 | 2014-10-07 15:32:37 -0700

  * Include plugin unit tests into the top-level btest configuration. (Robin Sommer)

  * Switching the prefix separator for packet source/dumper plugins
    once more, now to "::". Addresses BIT-1267. (Robin Sommer)

  * Fix for allowing a packet source/dumper plugin to support multiple
    prefixes with a colon. (Robin Sommer)

2.3-225 | 2014-10-07 15:13:35 -0700

  * Updating plugin documentation. (Robin Sommer)

2.3-224 | 2014-10-07 14:32:17 -0700

  * Improved the log file reference documentation. (Jeannette Dopheide
    and Daniel Thayer)

  * Improves shockwave flash file signatures. (Seth Hall)

     - This moves the signatures out of the libmagic imported signatures
       and into our own general.sig.

     - Expand the detection to LZMA compressed flash files.

  * Add new script language reference documentation on operators,
    statements, and directives.  Also improved the documentation on
    types and attributes by splitting them into two docs, and
    providing more examples and adding a chart on the top of each page
    with links to each type and attribute for easier access to the
    information. (Daniel Thayer)

  * Split the types and attributes reference doc into two docs.
    (Daniel Thayer)

2.3-208 | 2014-10-03 09:38:52 -0500

  * BIT-1268: Fix uninitialized router_list argument in
    dhcp_offer/dhcp_ack. (Jon Siwek)

2.3-207 | 2014-10-02 16:39:17 -0700

  * Updating plugin docs. (Robin Sommer)

  * Fix packet sources being treated as idle when a packet is
    available. Addresses BIT-1266. (Jon Siwek)

  * Fix regression causing the main loop to spin more frequently.
    Addresses BIT-1266. (Jon Siwek)

2.3-203 | 2014-09-29 20:06:54 -0700

  * Fix to use length parameter in DNP3 time conversion correctly now.
    (Robin Sommer)

2.3-202 | 2014-09-29 17:05:18 -0700

  * New SSL extension type from IANA and a few other SSL const
    changes. (Johanna Amann)

  * Make unexpected pipe errors fatal as precaution. Addresses
    BIT-1260. (Jon Siwek)

  * Adding a function for DNP3 to translate the timestamp format. (Hui
    Lin)

2.3-197 | 2014-09-29 10:42:01 -0500

  * Fix possible seg fault in TCP reassembler. (Jon Siwek)

2.3-196 | 2014-09-25 17:53:27 -0700

  * Changing prefix for packet sources/dumper from ':' to '%'.
    Addresses BIT-1249. (Robin Sommer)

  * Remove timeouts from remote communication loop. The select() now
    blocks until there's work to do instead of relying on a small
    timeout value which can cause unproductive use of cpu cycles. (Jon
    Siwek)

  * Improve error message when failing to activate a plugin. Also fix
    a unit test helper script that checks plugin availability. (Jon
    Siwek)

2.3-183 | 2014-09-24 10:08:04 -0500

  * Add a "node" field to Intel::Seen struture and intel.log to
    indicate which node discovered a hit on an intel item. (Seth Hall)

  * BIT-1261: Fixes to plugin quick start doc. (Jon Siwek)

2.3-180 | 2014-09-22 12:52:41 -0500

  * BIT-1259: Fix issue w/ duplicate TCP reassembly deliveries.
    (Jon Siwek)

2.3-178 | 2014-09-18 14:29:46 -0500

  * BIT-1256: Fix file analysis events from coming after bro_done().
    (Jon Siwek)

2.3-177 | 2014-09-17 09:41:27 -0500

  * Documentation fixes. (Chris Mavrakis)

2.3-174 | 2014-09-17 09:37:09 -0500

  * Fixed some "make doc" warnings caused by reST formatting
    (Daniel Thayer).

2.3-172 | 2014-09-15 13:38:52 -0500

  * Remove unneeded allocations for HTTP messages. (Jon Siwek)

2.3-171 | 2014-09-15 11:14:57 -0500

  * Fix a compile error on systems without pcap-int.h. (Jon Siwek)

2.3-170 | 2014-09-12 19:28:01 -0700

  * Fix incorrect data delivery skips after gap in HTTP Content-Range.
    Addresses BIT-1247. (Jon Siwek)

  * Fix file analysis placement of data after gap in HTTP
    Content-Range. Addresses BIT-1248. (Jon Siwek)

  * Fix issue w/ TCP reassembler not delivering some segments.
    Addresses BIT-1246. (Jon Siwek)

  * Fix MIME entity file data/gap ordering and raise http_entity_data
    in line with data arrival. Addresses BIT-1240. (Jon Siwek)

  * Implement file ID caching for MIME_Mail. (Jon Siwek)

  * Fix a compile error. (Jon Siwek)

2.3-161 | 2014-09-09 12:35:38 -0500

  * Bugfixes and test updates/additions. (Robin Sommer)

  * Interface tweaks and docs for PktSrc/PktDumper. (Robin Sommer)

  * Moving PCAP-related bifs to iosource/pcap.bif. (Robin Sommer)

  * Moving some of the BPF filtering code into base class.
    This will allow packet sources that don't support BPF natively to
    emulate the filtering via libpcap. (Robin Sommer)

  * Removing FlowSrc. (Robin Sommer)

  * Removing remaining pieces of the 2ndary path, and left-over
    files of packet sorter. (Robin Sommer)

  * A bunch of infrastructure work to move IOSource, IOSourceRegistry
    (now iosource::Manager) and PktSrc/PktDumper code into iosource/,
    and over to a plugin structure. (Robin Sommer)

2.3-137 | 2014-09-08 19:01:13 -0500

  * Fix Broxygen's rendering of opaque types. (Jon Siwek)

2.3-136 | 2014-09-07 20:50:46 -0700

  * Change more http links to https. (Johanna Amann)

2.3-134 | 2014-09-04 16:16:36 -0700

  * Fixed a number of issues with OCSP reply validation. Addresses
    BIT-1212. (Johanna Amann)

  * Fix null pointer dereference in OCSP verification code in case no
    certificate is sent as part as the ocsp reply. Addresses BIT-1212.
    (Johanna Amann)

2.3-131 | 2014-09-04 16:10:32 -0700

  * Make links in documentation templates protocol relative. (Johanna
    Amann)

2.3-129 | 2014-09-02 17:21:21 -0700

  * Simplify a conditional with equivalent branches. (Jon Siwek)

  * Change EDNS parsing code to use rdlength more cautiously. (Jon
    Siwek)

  * Fix a memory leak when bind() fails due to EADDRINUSE. (Jon Siwek)

  * Fix possible buffer over-read in DNS TSIG parsing. (Jon Siwek)

2.3-124 | 2014-08-26 09:24:19 -0500

  * Better documentation for sub_bytes (Jimmy Jones)

  * BIT-1234: Fix build on systems that already have ntohll/htonll
    (Jon Siwek)

2.3-121 | 2014-08-22 15:22:15 -0700

  * Detect functions that try to bind variables from an outer scope
    and raise an error saying that's not supported. Addresses
    BIT-1233. (Jon Siwek)

2.3-116 | 2014-08-21 16:04:13 -0500

  * Adding plugin testing to Makefile's test-all. (Robin Sommer)

  * Converting log writers and input readers to plugins.
    DataSeries and ElasticSearch plugins have moved to the new
    bro-plugins repository, which is now a git submodule in the
    aux/plugins directory. (Robin Sommer)

2.3-98 | 2014-08-19 11:03:46 -0500

  * Silence some doc-related warnings when using `bro -e`.
    Closes BIT-1232. (Jon Siwek)

  * Fix possible null ptr derefs reported by Coverity. (Jon Siwek)

2.3-96 | 2014-08-01 14:35:01 -0700

  * Small change to DHCP documentation. In server->client messages the
    host name may differ from the one requested by the client.
    (Johanna Amann)

  * Split DHCP log writing from record creation. This allows users to
    customize dhcp.log by changing the record in their own dhcp_ack
    event. (Johanna Amann)

  * Update PATH so that documentation btests can find bro-cut. (Daniel
    Thayer)

  * Remove gawk from list of optional packages in documentation.
    (Daniel Thayer)

  * Fix for redefining built-in constants. (Robin Sommer)

2.3-86 | 2014-07-31 14:19:58 -0700

  * Fix for redefining built-in constants. (Robin Sommer)

  * Adding missing check that a plugin's API version matches what Bro
    defines. (Robin Sommer)

  * Adding NEWS entry for plugins. (Robin Sommer)

2.3-83 | 2014-07-30 16:26:11 -0500

  * Minor adjustments to plugin code/docs. (Jon Siwek)

  * Dynamic plugin support. (Rpbin Sommer)

    Bro now supports extending core functionality, like protocol and
    file analysis, dynamically with external plugins in the form of
    shared libraries. See doc/devel/plugins.rst for an overview of the
    main functionality. Changes coming with this:

        - Replacing the old Plugin macro magic with a new API.

        - The plugin API changed to generally use std::strings instead
          of const char*.

        - There are a number of invocations of PLUGIN_HOOK_
          {VOID,WITH_RESULT} across the code base, which allow plugins
          to hook into the processing at those locations.

        - A few new accessor methods to various classes to allow
          plugins to get to that information.

        - network_time cannot be just assigned to anymore, there's now
          function net_update_time() for that.

        - Redoing how builtin variables are initialized, so that it
          works for plugins as well. No more init_net_var(), but
          instead bifcl-generated code that registers them.

        - Various changes for adjusting to the now dynamic generation
          of analyzer instances.

    - same_type() gets an optional extra argument allowing record type
      comparision to ignore if field names don't match. (Robin Sommer)

    - Further unify file analysis API with the protocol analyzer API
      (assigning IDs to analyzers; adding Init()/Done() methods;
      adding subtypes). (Robin Sommer)

    - A new command line option -Q that prints some basic execution
      time stats. (Robin Sommer)

    - Add support to the file analysis for activating analyzers by
      MIME type. (Robin Sommer)

          - File::register_for_mime_type(tag: Analyzer::Tag, mt:
            string): Associates a file analyzer with a MIME type.

          - File::add_analyzers_for_mime_type(f: fa_file, mtype:
            string): Activates all analyzers registered for a MIME
            type for the file.

          - The default file_new() handler calls
            File::add_analyzers_for_mime_type() with the file's MIME
            type.

2.3-20 | 2014-07-22 17:41:02 -0700

  * Updating submodule(s).

2.3-19 | 2014-07-22 17:29:19 -0700

  * Implement bytestring_to_coils() in Modbus analyzer so that coils
    gets passed to the corresponding events. (Hui Lin)

  * Add length field to ModbusHeaders. (Hui Lin)

2.3-12 | 2014-07-10 19:17:37 -0500

  * Include yield of vectors in Broxygen's type descriptions.
    Addresses BIT-1217. (Jon Siwek)

2.3-11 | 2014-07-10 14:49:27 -0700

  * Fixing DataSeries output. It was using a now illegal value as its
    default compression level. (Robin Sommer)

2.3-7 | 2014-06-26 17:35:18 -0700

  * Extending "make test-all" to include aux/bro-aux. (Robin Sommer)

2.3-6 | 2014-06-26 17:24:10 -0700

  * DataSeries compilation issue fixed. (mlaterman)

  * Fix a reference counting bug in ListVal ctor. (Jon Siwek)

2.3-3 | 2014-06-26 15:41:04 -0500

  * Support tilde expansion when Bro tries to find its own path. (Jon
    Siwek)

2.3-2 | 2014-06-23 16:54:15 -0500

  * Remove references to line numbers in tutorial text. (Daniel Thayer)

2.3 | 2014-06-16 09:48:25 -0500

  * Release 2.3.

2.3-beta-33 | 2014-06-12 11:59:28 -0500

  * Documentation improvements/fixes. (Daniel Thayer)

2.3-beta-24 | 2014-06-11 15:35:31 -0500

  * Fix SMTP state tracking when server response is missing.
    (Robin Sommer)

2.3-beta-22 | 2014-06-11 12:31:38 -0500

  * Fix doc/test that broke due to a Bro script change. (Jon Siwek)

  * Remove unused --with-libmagic configure option. (Jon Siwek)

2.3-beta-20 | 2014-06-10 18:16:51 -0700

  * Fix use-after-free in some cases of reassigning a table index.
    Addresses BIT-1202. (Jon Siwek)

2.3-beta-18 | 2014-06-06 13:11:50 -0700

  * Add two more SSL events, one triggered for each handshake message
    and one triggered for the tls change cipherspec message. (Johanna
    Amann)

  * Small SSL bug fix. In case SSL::disable_analyzer_after_detection
    was set to false, the ssl_established event would fire after each
    data packet once the session is established. (Johanna Amann)

2.3-beta-16 | 2014-06-06 13:05:44 -0700

  * Re-activate notice suppression for expiring certificates.
    (Johanna Amann)

2.3-beta-14 | 2014-06-05 14:43:33 -0700

  * Add new TLS extension type numbers from IANA (Johanna Amann)

  * Switch to double hashing for Bloomfilters for better performance.
    (Matthias Vallentin)

  * Bugfix to use full digest length instead of just one byte for
    Bloomfilter's universal hash function. Addresses BIT-1140.
    (Matthias Vallentin)

  * Make buffer for X509 certificate subjects larger. Addresses
    BIT-1195 (Johanna Amann)

2.3-beta-5 | 2014-05-29 15:34:42 -0500

  * Fix misc/load-balancing.bro's reference to
    PacketFilter::sampling_filter (Jon Siwek)

2.3-beta-4 | 2014-05-28 14:55:24 -0500

  * Fix potential mem leak in remote function/event unserialization.
    (Jon Siwek)

  * Fix reference counting bug in table coercion expressions (Jon Siwek)

  * Fix an "unused value" warning. (Jon Siwek)

  * Remove a duplicate unit test baseline dir. (Jon Siwek)

2.3-beta | 2014-05-19 16:36:50 -0500

  * Release 2.3-beta

  * Clean up OpenSSL data structures on exit. (Johanna Amann)

  * Fixes for OCSP & x509 analysis memory leak issues. (Johanna Amann)

  * Remove remaining references to BROMAGIC (Daniel Thayer)

  * Fix typos and formatting in event and BiF documentation (Daniel Thayer)

  * Update intel framework plugin for ssl server_name extension API
    changes. (Johanna Amann, Justin Azoff)

  * Fix expression errors in SSL/x509 scripts when unparseable data
    is in certificate chain. (Johanna Amann)

2.2-478 | 2014-05-19 15:31:33 -0500

  * Change record ctors to only allow record-field-assignment
    expressions. (Jon Siwek)

2.2-477 | 2014-05-19 14:13:00 -0500

  * Fix X509::Result record's "result" field to be set internally as type int instead of type count. (Johanna Amann)

  * Fix a couple of doc build warnings (Daniel Thayer)

2.2-470 | 2014-05-16 15:16:32 -0700

  * Add a new section "Cluster Configuration" to the docs that is
    intended as a how-to for configuring a Bro cluster.  Most of this
    content was moved here from the BroControl doc (which is now
    intended as more of a reference guide for more experienced users)
    and the load balancing FAQ on the website. (Daniel Thayer)

  * Update some doc tests and line numbers (Daniel Thayer)

2.2-457 | 2014-05-16 14:38:31 -0700

  * New script policy/protocols/ssl/validate-ocsp.bro that adds OSCP
    validation to ssl.log. The work is done by a new bif
    x509_ocsp_verify(). (Johanna Amann)

  * STARTTLS support for POP3 and SMTP. The SSL analyzer takes over
    when seen. smtp.log now logs when a connection switches to SSL.
    (Johanna Amann)

  * Replace errors when parsing x509 certs with weirds. (Johanna
    Amann)

  * Improved Heartbleed attack/scan detection. (Johanna Amann)

  * Let TLS analyzer fail better when no longer in sync with the data
    stream. (Johanna Amann)

2.2-444 | 2014-05-16 14:10:32 -0500

  * Disable all default AppStat plugins except facebook. (Jon Siwek)

  * Update for the active http test to force it to use ipv4. (Seth Hall)

2.2-441 | 2014-05-15 11:29:56 -0700

  * A new RADIUS analyzer. (Vlad Grigorescu)

    It produces a radius.log and generates two events:

        event radius_message(c: connection, result: RADIUS::Message);
        event radius_attribute(c: connection, attr_type: count, value: string);

2.2-427 | 2014-05-15 13:37:23 -0400

  * Fix dynamic SumStats update on clusters (Johanna Amann)

2.2-425 | 2014-05-08 16:34:44 -0700

  * Fix reassembly of data w/ sizes beyond 32-bit capacities. (Jon Siwek)

    Reassembly code (e.g. for TCP) now uses int64/uint64 (signedness
    is situational) data types in place of int types in order to
    support delivering data to analyzers that pass 2GB thresholds.
    There's also changes in logic that accompany the change in data
    types, e.g. to fix TCP sequence space arithmetic inconsistencies.

    Another significant change is in the Analyzer API: the *Packet and
    *Undelivered methods now use a uint64 in place of an int for the
    relative sequence space offset parameter.

    Addresses BIT-348.

  * Fixing compiler warnings. (Robin Sommer)

  * Update SNMP analyzer's DeliverPacket method signature. (Jon Siwek)

2.2-417 | 2014-05-07 10:59:22 -0500

  * Change handling of atypical OpenSSL error case in x509 verification. (Jon Siwek)

  * Fix memory leaks in X509 certificate parsing/verification. (Jon Siwek)

  * Fix new []/delete mismatch in input::reader::Raw::DoClose(). (Jon Siwek)

  * Fix buffer over-reads in file_analysis::Manager::Terminate() (Jon Siwek)

  * Fix buffer overlows in IP address masking logic. (Jon Siwek)

    That could occur either in taking a zero-length mask on an IPv6 address
    (e.g. [fe80::]/0) or a reverse mask of length 128 on any address (e.g.
    via the remask_addr BuiltIn Function).

  * Fix new []/delete mismatch in ~Base64Converter. (Jon Siwek)

2.2-410 | 2014-05-02 12:49:53 -0500

  * Replace an unneeded OPENSSL_malloc call. (Jon Siwek)

2.2-409 | 2014-05-02 12:09:06 -0500

  * Clean up and documentation for base SNMP script. (Jon Siwek)

  * Update base SNMP script to now produce a snmp.log. (Seth Hall)

  * Add DH support to SSL analyzer.  When using DHE or DH-Anon, sever
    key parameters are now available in scriptland.  Also add script to
    alert on weak certificate keys or weak dh-params. (Johanna Amann)

  * Add a few more ciphers Bro did not know at all so far. (Johanna Amann)

  * Log chosen curve when using ec cipher suite in TLS. (Johanna Amann)

2.2-397 | 2014-05-01 20:29:20 -0700

  * Fix reference counting for lookup_ID() usages. (Jon Siwek)

2.2-395 | 2014-05-01 20:25:48 -0700

  * Fix missing "irc-dcc-data" service field from IRC DCC connections.
    (Jon Siwek)

  * Correct a notice for heartbleed. The notice is thrown correctly,
    just the message conteined wrong values. (Johanna Amann)

  * Improve/standardize some malloc/realloc return value checks. (Jon
    Siwek)

  * Improve file analysis manager shutdown/cleanup. (Jon Siwek)

2.2-388 | 2014-04-24 18:38:07 -0700

  * Fix decoding of MIME quoted-printable. (Mareq)

2.2-386 | 2014-04-24 18:22:29 -0700

  * Do a Intel::ADDR lookup for host field if we find an IP address
    there. (jshlbrd)

2.2-381 | 2014-04-24 17:08:45 -0700

  * Add Java version to software framework. (Brian Little)

2.2-379 | 2014-04-24 17:06:21 -0700

  * Remove unused Val::attribs member. (Jon Siwek)

2.2-377 | 2014-04-24 16:57:54 -0700

  * A larger set of SSL improvements and extensions. Addresses
    BIT-1178. (Johanna Amann)

        - Fixes TLS protocol version detection. It also should
          bail-out correctly on non-tls-connections now

        - Adds support for a few TLS extensions, including
          server_name, alpn, and ec-curves.

        - Adds support for the heartbeat events.

        - Add Heartbleed detector script.

        - Adds basic support for OCSP stapling.

  * Fix parsing of DNS TXT RRs w/ multiple character-strings.
    Addresses BIT-1156. (Jon Siwek)

2.2-353 | 2014-04-24 16:12:30 -0700

  * Adapt HTTP partial content to cache file analysis IDs. (Jon Siwek)

  * Adapt SSL analyzer to generate file analysis handles itself. (Jon
    Siwek)

  * Adapt more of HTTP analyzer to use cached file analysis IDs. (Jon
    Siwek)

  * Adapt IRC/FTP analyzers to cache file analysis IDs. (Jon Siwek)

  * Refactor regex/signature AcceptingSet data structure and usages.
    (Jon Siwek)

  * Enforce data size limit when checking files for MIME matches. (Jon
    Siwek)

  * Refactor file analysis file ID lookup. (Jon Siwek)

2.2-344 | 2014-04-22 20:13:30 -0700

  * Refactor various hex escaping code. (Jon Siwek)

2.2-341 | 2014-04-17 18:01:41 -0500

  * Fix duplicate DNS log entries. (Robin Sommer)

2.2-341 | 2014-04-17 18:01:01 -0500

  * Refactor initialization of ASCII log writer options. (Jon Siwek)

  * Fix a memory leak in ASCII log writer. (Jon Siwek)

2.2-338 | 2014-04-17 17:48:17 -0500

  * Disable input/logging threads setting their names on every
    heartbeat. (Jon Siwek)

  * Fix bug when clearing Bloom filter contents. Reported by
    @colonelxc. (Matthias Vallentin)

2.2-335 | 2014-04-10 15:04:57 -0700

  * Small logic fix for main SSL script. (Johanna Amann)

  * Update DPD signatures for detecting TLS 1.2. (Johanna Amann)

  * Remove unused data member of SMTP_Analyzer to silence a Coverity
    warning. (Jon Siwek)

  * Fix missing @load dependencies in some scripts. Also update the
    unit test which is supposed to catch such errors. (Jon Siwek)

2.2-326 | 2014-04-08 15:21:51 -0700

  * Add SNMP datagram parsing support.This supports parsing of SNMPv1
    (RFC 1157), SNMPv2 (RFC 1901/3416), and SNMPv2 (RFC 3412).  An
    event is raised for each SNMP PDU type, though there's not
    currently any event handlers for them and not a default snmp.log
    either.  However, simple presence of SNMP is currently visible now
    in conn.log service field and known_services.log. (Jon Siwek)

2.2-319 | 2014-04-03 15:53:25 -0700

  * Improve __load__.bro creation for .bif.bro stubs. (Jon Siwek)

2.2-317 | 2014-04-03 10:51:31 -0400

  * Add a uid field to the signatures.log.  Addresses BIT-1171
    (Anthony Verez)

2.2-315 | 2014-04-01 16:50:01 -0700

  * Change logging's "#types" description of sets to "set". Addresses
    BIT-1163 (Johanna Amann)

2.2-313 | 2014-04-01 16:40:19 -0700

  * Fix a couple nits reported by Coverity.(Jon Siwek)

  * Fix potential memory leak in IP frag reassembly reported by
    Coverity. (Jon Siwek)

2.2-310 | 2014-03-31 18:52:22 -0700

  * Fix memory leak and unchecked dynamic cast reported by Coverity.
    (Jon Siwek)

  * Fix potential memory leak in x509 parser reported by Coverity.
    (Johanna Amann)

2.2-304 | 2014-03-30 23:05:54 +0200

  * Replace libmagic w/ Bro signatures for file MIME type
    identification. Addresses BIT-1143. (Jon Siwek)

    Includes:

    - libmagic is no longer used at all.  All MIME type detection is
      done through new Bro signatures, and there's no longer a means
      to get verbose file type descriptions. The majority of the
      default file magic signatures are derived from the default magic
      database of libmagic ~5.17.

    - File magic signatures consist of two new constructs in the
      signature rule parsing grammar: "file-magic" gives a regular
      expression to match against, and "file-mime" gives the MIME type
      string of content that matches the magic and an optional strength
      value for the match.

    - Modified signature/rule syntax for identifiers: they can no
      longer start with a '-', which made for ambiguous syntax when
      doing negative strength values in "file-mime".  Also brought
      syntax for Bro script identifiers in line with reality (they
      can't start with numbers or include '-' at all).

    - A new built-in function, "file_magic", can be used to get all
      file magic matches and their corresponding strength against a
      given chunk of data.

    - The second parameter of the "identify_data" built-in function
      can no longer be used to get verbose file type descriptions,
      though it can still be used to get the strongest matching file
      magic signature.

    - The "file_transferred" event's "descr" parameter no longer
      contains verbose file type descriptions.

    - The BROMAGIC environment variable no longer changes any behavior
      in Bro as magic databases are no longer used/installed.

    - Removed "binary" and "octet-stream" mime type detections. They
      don' provide any more information than an uninitialized
      mime_type field which implicitly means no magic signature
      matches and so the media type is unknown to Bro.

    - The "fa_file" record now contains a "mime_types" field that
      contains all magic signatures that matched the file content
      (where the "mime_type" field is just a shortcut for the
      strongest match).

    - Reverted back to minimum requirement of CMake 2.6.3 from 2.8.0.

  * The logic for adding file ids to {orig,resp}_fuids fields of the
    http.log incorrectly depended on the state of
    {orig,resp}_mime_types fields, so sometimes not all file ids
    associated w/ the session were logged. (Jon Siwek)

  * Fix MHR script's use of fa_file$mime_type before checking if it's
    initialized. (Jon Siwek)

2.2-294 | 2014-03-30 22:08:25 +0200

  * Rework and move X509 certificate processing from the SSL protocol
    analyzer to a dedicated file analyzer. This will allow us to
    examine X509 certificates from sources other than SSL in the
    future. Furthermore, Bro now parses more fields and extensions
    from the certificates (e.g. elliptic curve information, subject
    alternative names, basic constraints). Certificate validation also
    was improved, should be easier to use and exposes information like
    the full verified certificate chain. (Johanna Amann)

    This update changes the format of ssl.log and adds a new x509.log
    with certificate information. Furthermore all x509 events and
    handling functions have changed.

2.2-271 | 2014-03-30 20:25:17 +0200

  * Add unit tests covering vector/set/table ctors/inits. (Jon Siwek)

  * Fix parsing of "local" named table constructors. (Jon Siwek)

  * Improve type checking of records. Addresses BIT-1159. (Jon Siwek)

2.2-267 | 2014-03-30 20:21:43 +0200

  * Improve documentation of Bro clusters. Addresses BIT-1160.
    (Daniel Thayer)

2.2-263 | 2014-03-30 20:19:05 +0200

  * Don't include locations into serialization when cloning values.
    (Robin Sommer)

2.2-262 | 2014-03-30 20:12:47 +0200

  * Refactor SerializationFormat::EndWrite and ChunkedIO::Chunk memory
    management. (Jon Siwek)

  * Improve SerializationFormat's write buffer growth strategy. (Jon
    Siwek)

  * Add --parse-only option to exit after parsing scripts. May be
    useful for syntax-checking tools. (Jon Siwek)

2.2-256 | 2014-03-30 19:57:28 +0200

  * For the summary statistics framewirk, change all &create_expire
    attributes to &read_expire in the cluster part. (Johanna Amann)

2.2-254 | 2014-03-30 19:55:22 +0200

  * Update instructions on how to build Bro docs. (Daniel Thayer)

2.2-251 | 2014-03-28 08:37:37 -0400

  * Quick fix to the ElasticSearch writer. (Seth Hall)

2.2-250 | 2014-03-19 17:20:55 -0400

  * Improve performance of MHR script by reducing cloned Vals in
    a "when" scope. (Jon Siwek)

2.2-248 | 2014-03-19 14:47:40 -0400

  * Make SumStats work incrementally and non-blocking in non-cluster
    mode, but force it to operate by blocking if Bro is shutting
    down. (Seth Hall)

2.2-244 | 2014-03-17 08:24:17 -0700

  * Fix compile errror on FreeBSD caused by wrong include file order.
    (Johanna Amann)

2.2-240 | 2014-03-14 10:23:54 -0700

  * Derive results of DNS lookups from from input when in BRO_DNS_FAKE
    mode. Addresses BIT-1134. (Jon Siwek)

  * Fixing a few cases of undefined behaviour introduced by recent
    formatter work.

  * Fixing compiler error. (Robin Sommer)

  * Fixing (very unlikely) double delete in HTTP analyzer when
    decapsulating CONNECTs. (Robin Sommer)

2.2-235 | 2014-03-13 16:21:19 -0700

  * The Ascii writer has a new option LogAscii::use_json for writing
	out logs as JSON. (Seth Hall)

  * Ascii input reader now supports all config options as per-input
    stream "config" values. (Seth Hall)

  * Refactored formatters and updated the the writers a bit. (Seth
	Hall)

2.2-229 | 2014-03-13 14:58:30 -0700

  * Refactoring analyzer manager code to reuse
    ApplyScheduledAnalyzers(). (Robin Sommer)

2.2-228 | 2014-03-13 14:25:53 -0700

  * Teach async DNS lookup builtin-functions about BRO_DNS_FAKE.
    Addresses BIT-1134. (Jon Siwek)

  * Enable fake DNS mode for test suites.

  * Improve analysis of TCP SYN/SYN-ACK reversal situations. (Jon
    Siwek)

    - Since it's just the handshake packets out of order, they're no
      longer treated as partial connections, which some protocol analyzers
      immediately refuse to look at.

    - The TCP_Reassembler "is_orig" state failed to change, which led to
      protocol analyzers sometimes using the wrong value for that.

    - Add a unit test which exercises the Connection::FlipRoles() code
      path (i.e. the SYN/SYN-ACK reversal situation).

    Addresses BIT-1148.

  * Fix bug in Connection::FlipRoles. It didn't swap address values
    right and also didn't consider that analyzers might be scheduled
    for the new connection tuple. Reported by Kevin McMahon. Addresses
    BIT-1148. (Jon Siwek)

2.2-221 | 2014-03-12 17:23:18 -0700

  * Teach configure script --enable-jemalloc, --with-jemalloc.
    Addresses BIT-1128. (Jon Siwek)

2.2-218 | 2014-03-12 17:19:45 -0700

  * Improve DBG_LOG macro (perf. improvement for --enable-debug mode).
    (Jon Siwek)

  * Silences some documentation warnings from Sphinx. (Jon Siwek)

2.2-215 | 2014-03-10 11:10:15 -0700

  * Fix non-deterministic logging of unmatched DNS msgs. Addresses
    BIT-1153 (Jon Siwek)

2.2-213 | 2014-03-09 08:57:37 -0700

  * No longer accidentally attempting to parse NBSTAT RRs as SRV RRs
    in DNS analyzer. (Seth Hall)

  * Fix DNS SRV responses and a small issue with NBNS queries and
    label length. (Seth Hall)

     - DNS SRV responses never had the code written to actually
       generate the dns_SRV_reply event.  Adding this required
       extending the event a bit to add extra information.  SRV responses
       now appear in the dns.log file correctly.

     - Fixed an issue where some Microsoft NetBIOS Name Service lookups
       would exceed the max label length for DNS and cause an incorrect
       "DNS_label_too_long" weird.

2.2-210 | 2014-03-06 22:52:36 -0500

  * Improve SSL logging so that connections are logged even when the
    ssl_established event is not generated as well as other small SSL
    fixes. (Johanna Amann)

2.2-206 | 2014-03-03 16:52:28 -0800

  * HTTP CONNECT proxy support. The HTTP analyzer now supports
    handling HTTP CONNECT proxies. (Seth Hall)

  * Expanding the HTTP methods used in the DPD signature to detect
    HTTP traffic. (Seth Hall)

  * Fixing removal of support analyzers. (Robin Sommer)

2.2-199 | 2014-03-03 16:34:20 -0800

  * Allow iterating over bif functions with result type vector of any.
    This changes the internal type that is used to signal that a
    vector is unspecified from any to void. Addresses BIT-1144
    (Johanna Amann)

2.2-197 | 2014-02-28 15:36:58 -0800

  * Remove test code. (Robin Sommer)

2.2-194 | 2014-02-28 14:50:53 -0800

  * Remove packet sorter. Addresses BIT-700. (Johanna Amann)

2.2-192 | 2014-02-28 09:46:43 -0800

  * Update Mozilla root bundle. (Johanna Amann)

2.2-190 | 2014-02-27 07:34:44 -0800

  * Adjust timings of a few leak tests. (Johanna Amann)

2.2-187 | 2014-02-25 07:24:42 -0800

  * More Google TLS extensions that are being actively used. Johanna(
    Amann)

  * Remove unused, and potentially unsafe, function
    ListVal::IncludedInString. (Johanna Amann)

2.2-184 | 2014-02-24 07:28:18 -0800

  * New TLS constants from
    https://tools.ietf.org/html/draft-bmoeller-tls-downgrade-scsv-01.
    (Johanna Amann)

2.2-180 | 2014-02-20 17:29:14 -0800

  * New SSL alert descriptions from
    https://tools.ietf.org/html/draft-ietf-tls-applayerprotoneg-04.
    (Johanna Amann)

  * Update SQLite. (Johanna Amann)

2.2-177 | 2014-02-20 17:27:46 -0800

  * Update to libmagic version 5.17. Addresses BIT-1136. (Jon Siwek)

2.2-174 | 2014-02-14 12:07:04 -0800

  * Support for MPLS over VLAN. (Chris Kanich)

2.2-173 | 2014-02-14 10:50:15 -0800

  * Fix misidentification of SOCKS traffic that in particiular seemed
    to happen a lot with DCE/RPC traffic. (Vlad Grigorescu)

2.2-170 | 2014-02-13 16:42:07 -0800

  * Refactor DNS script's state management to improve performance.
    (Jon Siwek)

  * Revert "Expanding the HTTP methods used in the signature to detect
    HTTP traffic." (Robin Sommer)

2.2-167 | 2014-02-12 20:17:39 -0800

  * Increase timeouts of some unit tests. (Jon Siwek)

  * Fix memory leak in modbus analyzer. Would happen if there's a
    'modbus_read_fifo_queue_response' event handler. (Jon Siwek)

  * Add channel_id TLS extension number. This number is not IANA
    defined, but we see it being actively used. (Johanna Amann)

  * Test baseline updates for DNS change. (Robin Sommer)

2.2-158 | 2014-02-09 23:45:39 -0500

  * Change dns.log to include only standard DNS queries. (Jon Siwek)

  * Improve DNS analysis. (Jon Siwek)

    - Fix parsing of empty question sections (when QDCOUNT == 0).  In this
      case, the DNS parser would extract two 2-byte fields for use in either
      "dns_query_reply" or "dns_rejected" events (dependent on value of
      RCODE) as qclass and qtype parameters.  This is not correct, because
      such fields don't actually exist in the DNS message format when
      QDCOUNT is 0.  As a result, these events are no longer raised when
      there's an empty question section.  Scripts that depends on checking
      for an empty question section can do that in the "dns_message" event.

    - Add a new "dns_unknown_reply" event, for when Bro does not know how
      to fully parse a particular resource record type.  This helps fix a
      problem in the default DNS scripts where the logic to complete
      request-reply pair matching doesn't work because it's waiting on more
      RR events to complete the reply.  i.e. it expects ANCOUNT number of
      dns_*_reply events and will wait until it gets that many before
      completing a request-reply pair and logging it to dns.log.  This could
      cause bogus replies to match a previous request if they happen to
      share a DNS transaction ID. (Jon Siwek)

    - The previous method of matching queries with replies was still
      unreliable in cases where the reply contains no answers.  The new code
      also takes extra measures to avoid pending state growing too large in
      cases where the condition to match a query with a corresponding reply is
      never met, but yet DNS messages continue to be exchanged over the same
      connection 5-tuple (preventing cleanup of the pending state). (Jon Siwek)

  * Updates to httpmonitor and mimestats documentation. (Jeannette Dopheide)

  * Updates to Logs and Cluster documentation (Jeannette Dopheide)

2.2-147 | 2014-02-07 08:06:53 -0800

  * Fix x509-extension test sometimes failing. (Johanna Amann)

2.2-144 | 2014-02-06 20:31:18 -0800

  * Fixing bug in POP3 analyzer. With certain input the analyzer could
    end up trying to write to non-writable memory. (Robin Sommer)

2.2-140 | 2014-02-06 17:58:04 -0800

  * Fixing memory leaks in input framework. (Robin Sommer)

  * Add script to detect filtered TCP traces. Addresses BIT-1119. (Jon
    Siwek)

2.2-137 | 2014-02-04 09:09:55 -0800

  * Minor unified2 script documentation fix. (Jon Siwek)

2.2-135 | 2014-01-31 11:09:36 -0800

  * Added some grammar and spelling corrections to Installation and
    Quick Start Guide. (Jeannette Dopheide)

2.2-131 | 2014-01-30 16:11:11 -0800

  * Extend file analysis API to allow file ID caching. This allows an
    analyzer to either provide file IDs associated with some file
    content or to cache a file ID that was already determined by
    script-layer logic so that subsequent calls to the file analysis
    interface can bypass costly detours through script-layer.  This
    can yield a decent performance improvement for analyzers that are
    able to take advantage of it and deal with streaming content (like
    HTTP, which has been adapted accordingly). (Jon Siwek)

2.2-128 | 2014-01-30 15:58:47 -0800

  * Add leak test for Exec module. (Johanna Amann)

  * Fix file_over_new_connection event to trigger when entire file is
    missed. (Jon Siwek)

  * Improve TCP connection size reporting for half-open connections.
    (Jon Siwek)

  * Improve gap reporting in TCP connections that never see data. We
    no longer accomodate SYN/FIN/RST-filtered traces by not reporting
    missing data. The behavior can be reverted by redef'ing
    "detect_filtered_trace". (Jon Siwek)

  * Improve TCP FIN retransmission handling. (Jon Siwek)

2.2-120 | 2014-01-28 10:25:23 -0800

  * Fix and extend x509_extension() event, which now actually returns
    the extension. (Johanna Amann)

    New event signauture:

        event x509_extension(c: connection, is_orig: bool, cert:X509, extension: X509_extension_info)

2.2-117 | 2014-01-23 14:18:19 -0800

  * Fixing initialization context in anonymous functions. (Robin
    Sommer)

2.2-115 | 2014-01-22 12:11:18 -0800

  * Add unit tests for new Bro Manual docs. (Jon Siwek)

  * New content for the "Using Bro" section of the manual. (Rafael
    Bonilla/Jon Siwek)

2.2-105 | 2014-01-20 12:16:48 -0800

  * Support GRE tunnel decapsulation, including enhanced GRE headers.
    GRE tunnels are treated just like IP-in-IP tunnels by parsing past
    the GRE header in between the delivery and payload IP packets.
    Addresses BIT-867. (Jon Siwek)

  * Simplify FragReassembler memory management. (Jon Siwek)

2.2-102 | 2014-01-20 12:00:29 -0800

  * Include file information (MIME type and description) into notice
    emails if available. (Justin Azoff)

2.2-100 | 2014-01-20 11:54:58 -0800

  * Fix caching of recently validated SSL certifcates. (Justin Azoff)

2.2-98 | 2014-01-20 11:50:32 -0800

  * For notice suppresion, instead of storing the entire notice in
    Notice::suppressing, just store the time the notice should be
    suppressed until. This saves significant memory but can no longer
    raise end_suppression, which has been removed. (Justin Azoff)

2.2-96 | 2014-01-20 11:41:07 -0800

  * Integrate libmagic 5.16. Bro now now always relies on
    builtin/shipped magic library/database. (Jon Siwek)

  * Bro now requires a CMake 2.8.x, but no longer a pre-installed
    libmagic. (Jon Siwek)

2.2-93 | 2014-01-13 09:16:51 -0800

  * Fixing compile problems with some versions of libc++. Reported by
    Craig Leres. (Robin Sommer)

2.2-91 | 2014-01-13 01:33:28 -0800

  * Improve GeoIP City database support. When trying to open a city
    database, it now considers both the "REV0" and "REV1" versions of
    the city database instead of just the former. (Jon Siwek)

  * Broxygen init fixes. Addresses BIT-1110. (Jon Siwek)

    - Don't check mtime of bro binary if BRO_DISABLE_BROXYGEN env var set.

    - Fix failure to locate bro binary if invoking from a relative
      path and '.' isn't in PATH.

  * Fix for packet writing to make it use the global snap length.
    (Seth Hall)

  * Fix for traffic with TCP segmentation offloading with IP header
    len field being set to zero. (Seth Hall)

  * Canonify output of a unit test. (Jon Siwek)

  * A set of documentation updates. (Daniel Thayer)

      - Fix typo in Bro 2.2 NEWS on string indexing.
      - Fix typo in the Quick Start Guide, and clarified the
        instructions about modifying crontab.
      - Add/fix documentation for missing/misnamed event parameters.
      - Fix typos in BIF documentation of hexstr_to_bytestring.
      - Update the documentation of types and attributes.
      - Documented the new substring extraction functionality.
      - Clarified the description of "&priority" and "void".

2.2-75 | 2013-12-18 08:36:50 -0800

  * Fixing segfault with mismatching set &default in record fields.
    (Robin Sommer)

2.2-74 | 2013-12-16 08:49:55 -0800

  * Improve warnings emitted from raw/execute input reader. (Jon
    Siwek)

  * Further improve core.when-interpreter-exceptions unit test. (Jon
    Siwek)

2.2-72 | 2013-12-12 07:12:47 -0800

  * Improve the core.when-interpreter-exceptions unit test to prevent
    it from occasionally timing out. (Jon Siwek)

2.2-70 | 2013-12-10 15:02:50 -0800

  * Fix (harmless) uninitialized field in basename/dirname util
    wrapper. (Jon Siwek)

2.2-68 | 2013-12-09 15:19:37 -0800

  * Several improvements to input framework error handling for more
    robustness and more helpful error messages. Includes tests for
    many cases. (Johanna Amann)

2.2-66 | 2013-12-09 13:54:16 -0800

  * Fix table &default reference counting for record ctor expressions.
    (Jon Siwek)

  * Close signature files after done parsing. (Jon Siwek)

  * Fix unlikely null ptr deref in broxygen::Manager. (Jon Siwek)

  * FreeBSD build fix addendum: unintended variable shadowing. (Jon
    Siwek)

  * Fix build on FreeBSD. basename(3)/dirname(3) const-ness may vary
    w/ platform. (Jon Siwek)

  * Updated software framework to support parsing IE11 user-agent
    strings. (Seth Hall)

  * Fix the irc_reply event for several server message types. (Seth
    Hall)

  * Fix memory leak in input framework. If the input framework was
    used to read event streams and those streams contained records
    with more than one field, not all elements of the threading Values
    were cleaned up. Addresses BIT-1103. (Johanna Amann)

  * Minor Broxygen improvements. Addresses BIT-1098. (Jon Siwek)

2.2-51 | 2013-12-05 07:53:37 -0800

  * Improve a unit test involving 'when' conditionals. (Jon Siwek)

2.2-48 | 2013-12-04 13:45:47 -0800

  * Support omission of string slice low/high indices, BIT-1097.

    Omission of the low index defaults to 0:

        s = "12345"; s[:3] == "123"

    Omission of the high index defaults to length of the string:

        s = "12345"; s[3:] == "45" (Jon Siwek)

  * Tweak to SMTP script to adjust for new string slicing behaviour.
    (Robin Sommer)

  * Test updates. (Robin Sommer)

2.2-44 | 2013-12-04 12:41:51 -0800

  * Fix string slice notation. Addresses BIT-1097. (Jon Siwek)

    Slice ranges were not correctly determined for negative indices
    and also off by one in general (included one more element at the
    end of the substring than what actually matched the index range).
    It's now equivalent to Python slice notation.  Accessing a string
    at a single index is also the same as Python except that an
    out-of-range index returns an empty string instead of throwing an
    expection.

2.2-41 | 2013-12-04 12:40:51 -0800

  * Updating tests. (Robin Sommer)

2.2-40 | 2013-12-04 12:16:38 -0800

  * ssl_client_hello() now receives a vector of ciphers, instead of a
    set, to preserve their order. (Johanna Amann)

2.2-38 | 2013-12-04 12:10:54 -0800

  * New script misc/dump-events.bro, along with core support, that
    dumps events Bro is raising in an easily readable form for
    debugging. (Robin Sommer)

  * Prettyfing Describe() for record types. If a record type has a
    name and ODesc is set to short, we now print the name instead of
    the full field list. (Robin Sommer)

2.2-35 | 2013-12-04 10:10:32 -0800

  * Rework the automated script-reference documentation generation
    process, broxygen. Addresses BIT-701 and BIT-751. (Jon Siwek)

    Highlights:

        - Remove --doc-scripts and -Z options to toggle documentation
          mode. The parser is now always instrumented to gather
          documentation from comments of the form "##", "##!", or
          "##<".

        - Raw comments are available at runtime through several BIF
          functions: get_*_comments;

        - Add --broxygen and -X options to toggle generating
          reST-format documentation output, driven by a config file
          argument.

        - Add a "broxygen" Sphinx extension domain, allowing certain
          pieces of documentation to be generated on-the-fly via
          invoking a Bro process. Re-organized/cleaned up the Sphinx
          source tree in doc/ to use this in some places.

2.2-11 | 2013-12-03 10:56:28 -0800

  * Unit test for broccoli vector support. (Jon Siwek)

  * Changed ordering of Bro type tag enum, which was out of sync. (Jon
    Siwek)

2.2-9 | 2013-11-18 14:03:21 -0800

  * Update local.bro for Bro >= 2.2. The commented out Notice::policy
    example didn't work anymore. (Daniel Thayer)

2.2-6 | 2013-11-15 07:05:15 -0800

  * Make "install-example-configs" target use DESTDIR. (Jon Siwek)

2.2-5 | 2013-11-11 13:47:54 -0800

  * Fix the irc_reply event for certain server message types. (Seth
    Hall)

  * Fixed Segmentation fault in SQLite Writer. (Jon Crussell)

2.2 | 2013-11-07 10:25:50 -0800

  * Release 2.2.

  * Removing location information from ssh.log in external tests.
    (Robin Sommer)

2.2-beta-199 | 2013-11-07 00:36:46 -0800

  * Fixing warnings during doc build. (Robin Sommer)

2.2-beta-198 | 2013-11-06 22:54:30 -0800

  * Update docs and tests for a recent change to detect-MHR.bro
    (Daniel Thayer)

  * Update tests and baselines for sumstats docs. (Daniel Thayer)

2.2-beta-194 | 2013-11-06 14:39:50 -0500

  * Remove resp_size from the ssh log. Refactor when we write out to
    the log a bit. Geodata now works reliably. (Vlad Grigorescu)

  * Update VirusTotal URL to work with changes to their website and
    changed it to a redef. (Vlad Grigorescu)

  * Added a document for the SumStats framework. (Seth Hall)

2.2-beta-184 | 2013-11-03 22:53:42 -0800

  * Remove swig-ruby from required packages section of install doc.
    (Daniel Thayer)

2.2-beta-182 | 2013-11-01 05:26:05 -0700

  * Adding source and original copyright statement to Mozilla cert
    list. (Robin Sommer)

  * Canonfying an intel test to not depend on output order. (Robin
    Sommer)

2.2-beta-177 | 2013-10-30 04:54:54 -0700

  * Fix thread processing/termination conditions. (Jon Siwek)

2.2-beta-175 | 2013-10-29 09:30:09 -0700

  * Return the Dir module to file name tracking instead of inode
    tracking to avoid missing files that reuse a formerly seen inode.
    (Seth Hall)

  * Deprecate Broccoli Ruby bindings and no longer build them by
    default; use --enable-ruby to do so. (Jon Siwek)

2.2-beta-167 | 2013-10-29 06:02:38 -0700

  * Change percent_lost in capture-loss from a string to a double.
    (Vlad Grigorescu)

  * New version of the threading queue deadlock fix. (Robin Sommer)

  * Updating README with download/git information. (Robin Sommer)

2.2-beta-161 | 2013-10-25 15:48:15 -0700

  * Add curl to list of optional dependencies. It's used by the
    active-http.bro script. (Daniel Thayer)

  * Update test and baseline for a recent doc test fix. (Daniel
    Thayer)

2.2-beta-158 | 2013-10-25 15:05:08 -0700

  * Updating README with download/git information. (Robin Sommer)

2.2-beta-157 | 2013-10-25 11:11:17 -0700

  * Extend the documentation of the SQLite reader/writer framework.
    (Johanna Amann)

  * Fix inclusion of wrong example file in scripting tutorial.
    Reported by Michael Auger @LM4K. (Johanna Amann)

  * Alternative fix for the thrading deadlock issue to avoid potential
    performance impact. (Johanna Amann)

2.2-beta-152 | 2013-10-24 18:16:49 -0700

  * Fix for input readers occasionally dead-locking. (Robin Sommer)

2.2-beta-151 | 2013-10-24 16:52:26 -0700

  * Updating submodule(s).

2.2-beta-150 | 2013-10-24 16:32:14 -0700

  * Change temporary ASCII reader workaround for getline() on
    Mavericks to permanent fix. (Johanna Amann)

2.2-beta-148 | 2013-10-24 14:34:35 -0700

  * Add gawk to list of optional packages. (Daniel Thayer)

  * Add more script package README files. (Daniel Thayer)

  * Add NEWS about new features of BroControl and upgrade info.
    (Daniel Thayer)

  * Intel framework notes added to NEWS. (Seth Hall)

  * Temporary OSX Mavericks libc++ issue workaround for getline()
    problem in ASCII reader. (Johanna Amann)

  * Change test of identify_data BIF to ignore charset as it may vary
    with libmagic version. (Jon Siwek)

  * Ensure that the starting BPF filter is logged on clusters. (Seth
    Hall)

  * Add UDP support to the checksum offload detection script. (Seth
    Hall)

2.2-beta-133 | 2013-10-23 09:50:16 -0700

  * Fix record coercion tolerance of optional fields. (Jon Siwek)

  * Add NEWS about incompatible local.bro changes, addresses BIT-1047.
    (Jon Siwek)

  * Fix minor formatting problem in NEWS. (Jon Siwek)

2.2-beta-129 | 2013-10-23 09:47:29 -0700

  * Another batch of documentation fixes and updates. (Daniel Thayer)

2.2-beta-114 | 2013-10-18 14:17:57 -0700

  * Moving the SQLite examples into separate Bro files to turn them
    into sphinx-btest tests. (Robin Sommer)

2.2-beta-112 | 2013-10-18 13:47:13 -0700

  * A larger chunk of documentation fixes and cleanup. (Daniel Thayer)

    Apart from many smaller improves this includes in particular:

        * Add README files for most Bro frameworks and base/protocols.
        * Add README files for base/protocols.
        * Update installation instructions.
        * Improvements to file analysis docs and conversion to using
          btest sphinx.

2.2-beta-80 | 2013-10-18 13:18:05 -0700

  * SQLite reader/writer documentation. (Johanna Amann)

  * Check that the SQLite reader is only used in MANUAL reading mode.
    (Johanna Amann)

  * Rename the SQLite writer "dbname" configuration option to
    "tablename". (Johanna Amann)

  * Remove the "dbname" configuration option from the SQLite reader as
    it wasn't used there. (Johanna Amann)

2.2-beta-73 | 2013-10-14 14:28:25 -0700

  * Fix misc. Coverity-reported issues (leaks, potential null pointer
    deref, dead code, uninitialized values,
    time-of-check-time-of-use). (Jon Siwek)

  * Add check for sqlite3 command to tests that require it. (Daniel
    Thayer)

2.2-beta-68 | 2013-10-14 09:26:09 -0700

  * Add check for curl command to active-http.test. (Daniel Thayer)

2.2-beta-64 | 2013-10-14 09:20:04 -0700

  * Review usage of Reporter::InternalError, addresses BIT-1045.

    Replaced some with InternalWarning or AnalyzerError, the later
    being a new method which signals the analyzer to not process
    further input. (Jon Siwek)

  * Add new event for TCP content file write failures:
    "contents_file_write_failure". (Jon Siwek)

2.2-beta-57 | 2013-10-11 17:23:25 -0700

  * Improve Broxygen end-of-sentence detection. (Jon Siwek)

2.2-beta-55 | 2013-10-10 13:36:38 -0700

  * A couple of new TLS extension numbers. (Johanna Amann)

  * Suport for three more new TLS ciphers. (Johanna Amann)

  * Removing ICSI notary from default site config. (Robin Sommer)

2.2-beta-51 | 2013-10-07 17:33:56 -0700

  * Polishing the reference and scripting sections of the manual.
    (Robin Sommer)

  * Fixing the historical CHANGES record. (Robin Sommer)

  * Updating copyright notice. (Robin Sommer)

2.2-beta-38 | 2013-10-02 11:03:29 -0700

  * Fix uninitialized (or unused) fields. (Jon Siwek)

  * Remove logically dead code. (Jon Siwek)

  * Remove dead/unfinished code in unary not expression.  (Jon Siwek)

  * Fix logic for failed DNS TXT lookups. (Jon Siwek)

  * A couple null ptr checks. (Jon Siwek)

  * Improve return value checking and error handling. (Jon Siwek)

  * Remove unused variable assignments. (Jon Siwek)

  * Prevent division/modulo by zero in scripts. (Jon Siwek)

  * Fix unintentional always-false condition. (Jon Siwek)

  * Fix invalidated iterator usage. (Jon Siwek)

  * Fix DNS_Mgr iterator mismatch. (Jon Siwek)

  * Set safe umask when creating script profiler tmp files. (Jon Siwek)

  * Fix nesting/indent level whitespace mismatch. (Jon Siwek)

  * Add checks to avoid improper negative values use. (Jon Siwek)

2.2-beta-18 | 2013-10-02 10:28:17 -0700

  * Add support for further TLS cipher suites. (Johanna Amann)

2.2-beta-13 | 2013-10-01 11:31:55 -0700

  * Updating bifcl usage message. (Robin Sommer)

  * Fix bifcl getopt() usage. (Jon Siwek)

2.2-beta-8 | 2013-09-28 11:16:29 -0700

  * Fix a "make doc" warning. (Daniel Thayer)

2.2-beta-4 | 2013-09-24 13:23:30 -0700

  * Fix for setting REPO in Makefile. (Robin Sommer)

  * Whitespace fix. (Robin Sommer)

  * Removing :doc: roles so that we can render this with docutils
    directly. (Robin Sommer)

2.2-beta | 2013-09-23 20:57:48 -0700

  * Update 'make dist' target. (Jon Siwek)

2.1-1387 | 2013-09-23 11:54:48 -0700

  * Change submodules to fixed URL. (Jon Siwek)

  * Updating NEWS. (Robin Sommer)

  * Fixing an always false condition. (Robin Sommer)

  * Fix required for compiling with clang 3.3. (Robin Sommer)

2.1-1377 | 2013-09-20 14:38:15 -0700

  * Updates to the scripting introduction. (Scott Runnels)

  * Kill raw input reader's child by process group to reliably clean
    it up. (Jon Siwek)

2.1-1368 | 2013-09-19 20:07:57 -0700

  * Add more links in the GeoLocation document (Daniel Thayer)

2.1-1364 | 2013-09-19 15:12:08 -0700

  * Add links to Intelligence Framework documentation. (Daniel Thayer)

  * Update Mozilla root CA list. (Johanna Amann, Jon Siwek)

  * Update documentation of required packages. (Daniel Thayer)

2.1-1359 | 2013-09-18 15:01:50 -0700

  * Make client and server random available on script-level. Addresses
    BIT-950. (Eric Wustrow)

2.1-1357 | 2013-09-18 14:58:52 -0700

  * Update HLL API and its documentation. (Johanna Amann)

  * Fix case in HLL where hll_error_margin could be undefined.
    (Johanna Amann)

2.1-1352 | 2013-09-18 14:42:28 -0700

  * Fix a number of compiler warnings. (Daniel Thayer)

  * Fix cmake warning about ENABLE_PERFTOOLS not being used. (Daniel
    Thayer)

2.1-1344 | 2013-09-16 16:20:55 -0500

  * Refactor Analyzer::AddChildAnalyzer and usages. (Jon Siwek)

  * Minor refactor to SSL BinPAC grammer. (Jon Siwek)

  * Minor refactor to Broxygen enum comments. (Jon Siwek)

  * Fix possible (unlikely) use of uninitialized value. (Jon Siwek)

  * Fix/improve dereference-before-null-checks. (Jon Siwek)

  * Fix out-of-bounds memory accesses, and remove a
    variable-length-array usage. (Jon Siwek)

  * Fix potential mem leak. (Jon Siwek)

  * Fix double-free and deallocator mismatch. (Jon Siwek)

  * Fix another function val reference counting bug. (Jon Siwek)

2.1-1335 | 2013-09-12 16:13:53 -0500

  * Documentation fixes (Daniel Thayer, Jon Siwek)

  * Fix various potential memory leaks. (Jon Siwek)

  * Fix significant memory leak in function unserialization. (Jon Siwek)

  * Fix use-after-free and invalid/mismatch deallocator bugs. (Jon Siwek)

  * Fixed an issue with the HLL_UNIQUE SumStats plugin that caused a reporter error. (Seth Hall)

  * Make the notice $actions field have a default empty set to avoid having to check for it's presence. (Seth Hall)

  * Fix signatures that use identifiers of type table. (Jon Siwek)

  * Fix memory leak if a DNS request fails to be made. (Jon Siwek)

  * Fix memory leak in DNS TXT lookups. (Jon Siwek)

  * Fix raw execution input reader's signal blocking which resulted in lingering processes. (Jon Siwek)

2.1-1306 | 2013-08-31 16:06:05 -0700

  * Reorganized and signifcantly extended documentation. This includes
    two new chapters contributed by Scott Runnels.

2.1-1216 | 2013-08-31 10:39:40 -0700


  * Support for probabilistic set cardinality, using the HyperLogLog
    algorithm. (Johanna Amann, Soumya Basu)

    Bro now provides the following BiFs:

        hll_cardinality_init(err: double, confidence: double): opaque of cardinality
        hll_cardinality_add(handle: opaque of cardinality, elem: any): bool
        hll_cardinality_merge_into(handle1: opaque of cardinality, handle2: opaque of cardinality): bool
        hll_cardinality_estimate(handle: opaque of cardinality): double
        hll_cardinality_copy(handle: opaque of cardinality): opaque of cardinality

2.1-1154 | 2013-08-30 08:27:45 -0700

  * Fix global opaque val segfault. Addresses BIT-1071. (Jon Siwek)

  * Fix malloc/delete mismatch. (Jon Siwek)

  * Fix invalid pointer dereference in AsciiFormatter. (Jon Siwek)

2.1-1150 | 2013-08-29 13:43:01 -0700

  * Fix input framework memory leaks. (Jon Siwek)

  * Fix memory leak in SOCKS analyzer for bad addr types. (Jon Siwek)

  * Fix Bloom filter memory leaks. (Jon Siwek)

2.1-1144 | 2013-08-28 18:51:06 -0700

  * Add bits_per_uid unit test. Addresses BIT-1016. (Jon Siwek)

  * UID optimizations. Addresses BIT-1016. (Jon Siwek)

  * Added a $unique_max field to Reducers for the SumStats::UNIQUE
    calculation, and using the new option in scan.bro and the FTP
    bruteforce detection. (Seth Hall)

2.1-1137 | 2013-08-27 13:26:44 -0700

  * Add BiF hexstr_to_bytestring() that does exactly the opposite of
    bytestring_to_hexstr(). (Johanna Amann)

2.1-1135 | 2013-08-27 12:16:26 -0700

  * More SumStats fixes. (Seth Hall)

  * Increase UIDs to 96 bits.  (Jon Siwek)

    - The bit-length is adjustable via redef'ing bits_per_uid.

    - Prefix 'C' is added to connection UIDS (including IP tunnels)
      and 'F' to files.

    Addresses BIT-1016.

2.1-1128 | 2013-08-24 10:27:29 -0700

  * Remove code relict in input framework. (Jon Siwek)

  * Fix documentation for mkdir BIF. (Jon Siwek)

  * File extraction tweaks. (Jon Siwek)

    - Default extraction limit of 100MB now provided via a tuning
      script loaded in local.bro so that command-line Bro is unlimited
      by default.

    - Extraction directory is now created on request of file
      extraction rather than unconditionally in bro_init(). (Jon
      Siwek)

2.1-1124 | 2013-08-23 16:33:52 -0700

  * Fixed a number of object bugs DNP3 analyzer. (Hui Lin)

2.1-1122 | 2013-08-22 16:52:27 -0700

  * Use macros to create file analyzer plugin classes. (Jon Siwek)

  * Add options to limit extracted file sizes w/ 100MB default. (Jon
    Siwek)

2.1-1117 | 2013-08-22 08:44:12 -0700

  * A number of input framework fixes and corresponding test stability
    improvements. (Jon Siwek)

  * Make memory leak tests able to time out.  (Jon Siwek)

  * Fix a compiler warning regarding strncat misuse. (Jon Siwek)

2.1-1103 | 2013-08-21 19:11:34 -0400

  * A number of sumstats fixes. (Seth Hall, Vlad Grigorescu)

  * Fix memory leak w/ when statements. Addresses BIT-1058. (Jon
    Siwek)

  * Switching to relative submodule paths (Robin Sommer)

2.1-1089 | 2013-08-19 11:25:11 -0700

  * Fix bloom filters' dependence on size_t. (Jon Siwek, Matthias
    Vallentin).

2.1-1081 | 2013-08-19 11:19:33 -0700

  * New BiF levenshtein_distance() to compute the Levenshtein distance
    between two strings. (Anthony Kasza)

2.1-1078 | 2013-08-19 09:29:30 -0700

  * Moving sqlite code into new external 3rdparty submodule. Johanna(
    Amann)

2.1-1074 | 2013-08-14 10:29:54 -0700

  * Fix timer type enum and timer name array mismatch. (Jon Siwek)

2.1-1072 | 2013-08-14 10:28:51 -0700

  * Adding the unified2 analyzer that reads unified2 files from disk,
    turning them into events. (Seth Hall)

  * Fixing intel framework tests. (Seth Hall)

2.1-1059 | 2013-08-13 23:52:41 -0400

  * Add file name support to intel framework. (Seth Hall)

  * Add file support to intel framework and slightly restructure
    intel http handling. (Seth Hall)

2.1-1052 | 2013-08-12 14:38:14 -0700

  * Fixing bug in DNP3 analyzer flagged by compiler warning. (Robin
    Sommer)

2.1-1050 | 2013-08-12 11:37:44 -0700

  * Experimental DNP3 analyzer. This includes only very basic
    script-level support at the moment, but quite a number of events
    are provided. (Hui Lin, Robin Sommer)

2.1-1041 | 2013-08-09 15:32:22 -0700

  * Update coverage baselines for canonical load order of scripts.
    (Jon Siwek)

2.1-1039 | 2013-08-09 15:30:15 -0700

  * Fix mem leak in DHCP analyzer. (Jon Siwek)

  * Fix a unit test outdated by recent sumstats changes. (Jon Siwek)

2.1-1036 | 2013-08-05 17:29:11 -0400

  * Fix the SSL infinite loop I just created. (Seth Hall)

2.1-1035 | 2013-08-05 16:44:50 -0400

  * Change to SSL log delay to cause the log to write even if delay times out. (Seth Hall)

2.1-1034 | 2013-08-03 20:27:43 -0700

  * A set of DHCP extensions. (Vlad Grigorescu)

     - Leases are logged to dhcp.log as they are seen.
     - scripts/policy/protocols/dhcp/known-devices-and-hostnames.bro
     - Added DPD sig.

2.1-1027 | 2013-08-03 01:57:37 -0400

  * Fix a major memory issue in the SumStats framework.

2.1-1026 | 2013-08-02 22:35:09 -0400

  * Fix the SumStats top-k plugin and test. (Seth Hall)

  * Rework of SumStats API to reduce high instantaneous memory
    use on clusters. (Seth Hall)

  * Large update for the SumStats framework.

     - On-demand access to sumstats results through "return from"
       functions named SumStats::request and Sumstats::request_key.
       Both functions are tested in standalone and clustered modes.

     - $name field has returned to SumStats which simplifies cluster
       code and makes the on-demand access stuff possible.

     - Clustered results can only be collected for 1 minute from their
       time of creation now instead of time of last read.

     - Thresholds use doubles instead of counts everywhere now.

     - Calculation dependency resolution occurs at start up time now
       instead of doing it at observation time which provide a minor
       cpu performance improvement.  A new plugin registration mechanism
       was created to support this change.

     - AppStats now has a minimal doc string and is broken into hook-based
       plugins.

     - AppStats and traceroute detection added to local.bro (Seth Hall)

2.1-1009 | 2013-08-02 17:19:08 -0700

  * A number of exec module and raw input reader fixes. (Jon Siwek)

2.1-1007 | 2013-08-01 15:41:54 -0700

  * More function documentation. (Johanna Amann)

2.1-1004 | 2013-08-01 14:37:43 -0700

  * Adding a probabilistic data structure for computing "top k"
    elements. (Johanna Amann)

    The corresponding functions are:

        topk_init(size: count): opaque of topk
        topk_add(handle: opaque of topk, value: any)
        topk_get_top(handle: opaque of topk, k: count)
        topk_count(handle: opaque of topk, value: any): count
        topk_epsilon(handle: opaque of topk, value: any): count
        topk_size(handle: opaque of topk): count
        topk_sum(handle: opaque of topk): count
        topk_merge(handle1: opaque of topk, handle2: opaque of topk)
        topk_merge_prune(handle1: opaque of topk, handle2: opaque of topk)

2.1-971 | 2013-08-01 13:28:32 -0700

  * Fix some build errors. (Jon Siwek)

  * Internal refactoring of how plugin components are tagged/managed.
    (Jon Siwek)

  * Fix various documentation, mostly related to file analysis. (Jon
    Siwek)

  * Changing the Bloom filter hashing so that it's independent of
    CompositeHash. (Robin Sommer)

2.1-951 | 2013-08-01 11:19:23 -0400

  * Small fix to deal with a bug in the SSL log delay mechanism.

2.1-948 | 2013-07-31 20:08:28 -0700

  * Fix segfault caused by merging an empty bloom-filter with a
    bloom-filter already containing values. (Johanna Amann)

2.1-945 | 2013-07-30 10:05:10 -0700

  * Make hashers serializable. (Matthias Vallentin)

  * Add docs and use default value for hasher names. (Matthias
    Vallentin)

2.1-939 | 2013-07-29 15:42:38 -0700

  * Added Exec, Dir, and ActiveHTTP modules. (Seth Hall)

    base/utils/exec.bro provides a module to start external processes
    asynchronously and retrieve their output on termination.
    base/utils/dir.bro uses it to monitor a directory for changes, and
    base/utils/active-http.bro for providing an interface for querying
    remote web servers.

2.1-930 | 2013-07-29 15:06:07 -0700

  * Major file analysis overhaul in naming and appearance, along with
    fixes and test updates. (Seth Hall and Jon Siwek)

    Includes:

    * Added protocol description functions that provide a super
      compressed log representation. (Seth Hall)

    * Added mime types to http.log (Seth Hall)

    * Add jar files to the default MHR lookups. (Seth Hall)

    * Adding CAB files for MHR checking. (Seth Hall)

    * Improve malware hash registry script.

        - Include a link to a virustotal search in the notice sub message field.
        - Give all information returned from Team Cymru in the notice message.
        - Add more file types to match on to the default set.

    * Make the custom libmagic database a git submodule.

    * Add an is_orig parameter to file_over_new_connection event.

    * Recorrected the module name to Files.

    * Added Files::analyzer_name to get a more readable name for a
      file analyzer.

    * Improved and just overall better handled multipart mime
      transfers in HTTP and SMTP.  HTTP now has orig_fuids and
      resp_fuids log fields since multiple "files" can be transferred
      with multipart mime in a single request/response pair.  SMTP has
      an fuids field which has file unique IDs for all parts
      transferred. FTP and IRC have a log field named fuid added
      because only a single file can be transferred per irc and ftp
      log line.

2.1-895 | 2013-07-29 14:07:35 -0700

  * Adding a test for a DNSKEY RR. (Robin Sommer)

2.1-894 | 2013-07-29 16:44:41 -0400

  * Updates for the Intel Framework. (Seth Hall)

      - policy/frameworks/intel/seen is the new location for the
        scripts that push data into the intel framework for checking.

      - The new policy/frameworks/intel/do_notice script adds an
        example mechanism for data driven notices.

      - Remove the Intel insertion after heuristically detecting SSH
        bruteforcing.

      - Intel importing format has changed (refer to docs).

      - All string matching is now case insensitive.

      - SMTP intel script has been updated to extract email
        addresses correctly.

      - Small fix sneaking into the smtp base script to actually
        extract individual email addresses in the To: field
        correctly.


2.1-888 | 2013-07-25 12:02:41 -0700

  * Protection about broken traces with empty pcap headers. (Matt
    Thompson)

2.1-887 | 2013-07-25 11:33:27 -0700

  * Support for Bloom filter. (Matthias Vallentin)

    Bro now provides the following BiFs:

        bloomfilter_basic_init(fp: double, capacity: count, name: string &default=""): opaque of bloomfilter
        bloomfilter_counting_init(k: count, cells: count, max: count, name: string &default=""): opaque of bloomfilter
        bloomfilter_add(bf: opaque of bloomfilter, x: any)
        bloomfilter_lookup(bf: opaque of bloomfilter, x: any): count
        bloomfilter_merge(bf1: opaque of bloomfilter, bf2: opaque of bloomfilter): opaque of bloomfilter
        bloomfilter_clear(bf: opaque of bloomfilter)

    Note that currently Bloom filters from separate Bro instances
    (e.g., from different cluster nodes) cannot be merged.

2.1-826 | 2013-07-25 10:12:26 -0700

  * bif files declared with bif_target() are now automatically
    compiled in. No more manual includes to pull them in. (Robin
    Sommer)

  * Covenience make target in testing/btest to update the three
    coverage tests that usually need tweaking when scripts get
    added/removed. (Robin Sommer)

2.1-824 | 2013-07-22 14:25:14 -0400

  * Fixed a scriptland state issue that manifested especially badly on proxies. (Seth Hall)

  * Another test fix. (Robin Sommer)

  * Canonyfying the output of core.print-bpf-filters. (Robin Sommer)

2.1-820 | 2013-07-18 12:30:04 -0700

  * Extending external canonifier to remove fractional values from
    capture_loss.log. (Robin Sommer)

  * Canonifying internal order for plugins and their components to
    make it deterministic. (Robin Sommer)

  * Small raw reader tweaks that got left our earlier. (Robin Sommer)

2.1-814 | 2013-07-15 18:18:20 -0700

  * Fixing raw reader crash when accessing nonexistant file, and
    memory leak when reading from file. Addresses #1038. (Johanna
    Amann)

2.1-811 | 2013-07-14 08:01:54 -0700

  * Bump sqlite to 3.7.17. (Johanna Amann)

  * Small test fixes. (Seth Hall)

  * Fix a bug where the same analyzer tag was reused for two different
    analyzers. (Seth Hall)

  * Moved DPD signatures into script specific directories. Left out
    the BitTorrent signatures pending further updates to that
    analyzer. (Seth Hall)

2.1-802 | 2013-07-10 10:55:14 -0700

  * Const adjustment for methods. (Jon Siwek)

2.1-798 | 2013-07-08 13:05:37 -0700

  * Rewrite of the packet filter framework. (Seth Hall)

    This includes:

    - Plugin interface for adding filtering mechanisms.

    - Integrated the packet filter framework with the analyzer
      framework to retrieve well-known ports from there.

    - Support for BPF-based load balancing (IPv4 and IPv6).  This will
      tie in with upcoming BroControl support for configuring this.

    - Support for BPF-based connection sampling.

    - Support for "shunting" traffic with BPF filters.

    - Replaced PacketFilter::all_packets with
      PacketFilter::enable_auto_protocol_capture_filters.

2.1-784 | 2013-07-04 22:28:48 -0400

  * Add a call to lookup_connection in SSH scripts to update connval. (Seth Hall)

  * Updating submodule(s). (Robin Sommer)

2.1-782 | 2013-07-03 17:00:39 -0700

  * Remove the SSL log queueing mechanism that was included with the
    log delay mechanism. (Seth Hall)

2.1-780 | 2013-07-03 16:46:26 -0700

  * Rewrite of the RAW input reader for improved robustness and new
    features. (Johanna Amann) This includes:

        - Send "end_of_data" event for all kind of streams.
        - Send "process_finished" event with exit code of child
          process at process termination.
        - Expose name of input stream to readers.
        - Better error handling.
        - New "force_kill" option which SIGKILLs processes on reader termination.
        - Supports reading from stdout and stderr simultaneously.
        - Support sending data to stdin of child process.
        - Streaming reads from external commands work without blocking.

2.1-762 | 2013-07-03 16:33:22 -0700

  * Fix to correct support for TLS 1.2. Addresses #1020. (Seth Hall,
    with help from Rafal Lesniak).

2.1-760 | 2013-07-03 16:31:36 -0700

  * Teach broxygen to generate protocol analyzer plugin reference.
    (Jon Siwek)

  * Adding 'const' to a number of C++ methods. (Jon Siwek)

2.1-757 | 2013-07-03 16:28:10 -0700

  * Fix redef of table index from clearing table.

    `redef foo["x"] = 1` now acts like `redef foo += { ["x"] = 1 }`
    instead of `redef foo = { ["x"] = 1 }`.

    Addresses #1013. (Jon Siwek)


2.1-755 | 2013-07-03 16:22:43 -0700

  * Add a general file analysis overview/how-to document. (Jon Siwek)

  * Improve file analysis doxygen comments. (Jon Siwek)

  * Improve tracking of HTTP file extraction. http.log now has files
    taken from request and response bodies in different fields for
    each, and can now track multiple files per body. That is, the
    "extraction_file" field is now "extracted_request_files" and
    "extracted_response_files". Addresses #988. (Jon Siwek)

  * Fix HTTP multipart body file analysis. Each part now gets assigned
    a different file handle/id. (Jon Siwek)

  * Remove logging of analyzers field of FileAnalysis::Info. (Jon
    Siwek)

  * Remove extraction counter in default file extraction scripts. (Jon
    Siwek)

  * Remove FileAnalysis::postpone_timeout.
    FileAnalysis::set_timeout_interval can now perform same function.
    (Jon Siwek)

  * Make default get_file_handle handlers &priority=5 so they're
    easier to override. (Jon Siwek)

  * Add input interface to forward data for file analysis. The new
    Input::add_analysis function is used to automatically forward
    input data on to the file analysis framework. (Jon Siwek)

  * File analysis framework interface simplifications. (Jon Siwek)

    - Remove script-layer data input interface (will be managed directly
      by input framework later).

    - Only track files internally by file id hash.  Chance of collision
      too small to justify also tracking unique file string.


2.1-741 | 2013-06-07 17:28:50 -0700

  * Fixing typo that could cause an assertion to falsely trigger.
    (Robin Sommer)

2.1-740 | 2013-06-07 16:37:32 -0700

  * Fix for CMake 2.6.x. (Robin Sommer)

2.1-738 | 2013-06-07 08:38:13 -0700

  * Remove invalid free on non-allocated pointer in hash function
    object. Addresses #1018. (Matthias Vallentin)

2.1-736 | 2013-06-06 10:05:20 -0700

  * New "magic constants" @DIR and @FILENAME that expand to the
    directory path of the current script and just the script file name
    without path, respectively. (Jon Siwek)

2.1-731 | 2013-06-04 21:19:08 -0700

  * Reorginization of internal protocol analyzer code. We're moving
    them to a modularized structure, based on a plugin model. Along
    with this change comes generic plugin infrastructure that we'll
    later extend to other Bro component as well. For now all plugins
    are compiled in statically, but in the future we plan to also
    enable dynamic loading at run time. (Robin Sommer)

  * Ignoring file ids in external tests. (Robin Sommer)

2.1-675 | 2013-06-02 20:03:19 -0700

  * Fix a compiler warning. (Robin Sommer)

  * Allow named vector/set/table/record constructors. Addresses #983.
    (Jon Siwek)

  * Adding Makefile target test-all that also runs the BroControl test
    suite.  (Robin Sommer)

2.1-664 | 2013-05-28 21:37:46 -0700

  * Dangling pointer fix. Addresses #1004. (Jon Siwek)

2.1-659 | 2013-05-24 17:24:18 -0700

  * Fix broken/missing documentation. (Jon Siwek)

  * Fixing test that would fail without ES/curl support. (Robin
    Sommer)

2.1-656 | 2013-05-17 15:58:07 -0700

  * Fix mutex lock problem for writers. (Johanna Amann)

2.1-654 | 2013-05-17 13:49:52 -0700

  * Tweaks to sqlite3 configuration to address threading issues.
    (Johanna Amann)

2.1-651 | 2013-05-17 13:37:16 -0700

  * Fix uninitialized DPM member. (Jon Siwek)

  * Fix issue with transaction ID reuse in a single DNS connection. (Seth Hall)

  * New function added to the queue.bro script to support peeking at
    the new gettable item in the queue without removing it. (Seth Hall)

2.1-647 | 2013-05-17 07:47:14 -0700

  * Fixing Broxygen generation to have BROMAGIC set. (Robin Sommer)

  * Fix for 'fchmod undeclared here' on FreeBSD. (Robin Sommer)

  * CMake policy fix to avoid errors with older versions. (Robin
    Sommer)

2.1-641 | 2013-05-15 18:15:09 -0700

  * Test update. (Robin Sommer)

2.1-640 | 2013-05-15 17:24:09 -0700

  * Support for cleaning up threads that have terminated. (Johanna
    Amann and Robin Sommer). Includes:

      - Both logging and input frameworks now clean up threads once
        they aren't further needed anymnore.

      - New function Log::remove_stream() that removes a logging
        stream, stopping all writer threads that are associated with
        it. Note, however, that removing a *filter* from a stream
        still doesn't clean up any threads. The problem is that
        because of the output paths potentially being created
        dynamically it's unclear if the writer thread will still be
        needed in the future.

2.1-626 | 2013-05-15 16:09:31 -0700

  * Add "reservoir" sampler for SumStats framework. This maintains
    a set of N uniquely distributed random samples. (Johanna Amann)

2.1-619 | 2013-05-15 16:01:42 -0700

  * SQLite reader and writer combo. This allows to read/write
    persistent data from on disk SQLite databases. The current
    interface is quite low-level, we'll add higher-level abstractions
    in the future. (Johanna Amann)

2.1-576 | 2013-05-15 14:29:09 -0700

  * Initial version of new file analysis framework. This moves most of
    the processing of file content from script-land into the core,
    where it belongs. Much of this is an internal change, and at this
    point the new code has essentially feature-equality with the old
    one. More script-level changes to come. (Jon Siwek)

2.1-502 | 2013-05-10 19:29:37 -0700

  * Allow default function/hook/event parameters. Addresses #972. (Jon
    Siwek)

  * Change the endianness parameter of bytestring_to_count() BIF to
    default to false (big endian). (Jon Siwek)

2.1-500 | 2013-05-10 19:22:24 -0700

  * Fix to prevent merge-hook of SumStat's unique plugin from damaging
    source data. (Johanna Amann)

2.1-498 | 2013-05-03 17:44:08 -0700

  * Table lookups return copy of non-const &default vals. This
    prevents unintentional modifications to the &default value itself.
    Addresses #981.  (Jon Siwek)

2.1-496 | 2013-05-03 15:54:47 -0700

  * Fix memory leak and unnecessary allocations in OpaqueVal.
    Addresses #986. (Matthias Vallentin)

2.1-492 | 2013-05-02 12:46:26 -0700

  * Work-around for sumstats framework not propagating updates after
    intermediate check in cluster environments. (Johanna Amann)

  * Always apply tcp_connection_attempt. Before this change it was
    only applied when a connection_attempt() event handler was
    defined. (Robin Sommer)

  * Fixing coverage.bare-mode-errors test. (Robin Sommer)

2.1-487 | 2013-05-01 18:03:22 -0700

  * Always apply tcp_connection_attempt timer, even if no
    connection_attempt() event handler is defined. (Robin Sommer)

2.1-486 | 2013-05-01 15:28:45 -0700

  * New framework for computing summary statistics in
    base/framework/sumstats. This replaces the metrics frameworks, and
    comes with a number of applications build on top, see NEWS. More
    documentation to follow. (Seth Hall)

2.1-397 | 2013-04-29 21:19:00 -0700

  * Fixing memory leaks in CompHash implementation. Addresses #987.
    (Robin Sommer)

2.1-394 | 2013-04-27 15:02:31 -0700

  * Fixed a bug in the vulnerable software script and added a test.
    (Seth Hall)

  * Fix schedule statements used outside event handlers. Addresses
    #974. (Jon Siwek)

  * Fix record coercion for default inner record fields. Addresses
    #973. (Jon Siwek)

  * Add bytestring_to_count function to bro.bif. Addresses #968. (Yun
    Zheng Hu)

2.1-386 | 2013-03-22 12:41:50 -0700

  * Added reverse() function to strings.bif. (Yun Zheng Hu)

2.1-384 | 2013-03-22 12:10:14 -0700

  * Fix record constructors in table initializer indices.  Addresses
    #660. (Jon Siwek)

2.1-382 | 2013-03-22 12:01:34 -0700

  * Add support for 802.1ah (Q-in-Q). Addresses #641. (Seth Hall)

2.1-380 | 2013-03-18 12:18:10 -0700

  * Fix gcc compile warnings in base64 encoder and benchmark reader.
    (Johanna Amann)

2.1-377 | 2013-03-17 17:36:09 -0700

  * Fixing potential leak in DNS error case. (Vlad Grigorescu)

2.1-375 | 2013-03-17 13:14:26 -0700

  * Add base64 encoding functionality, including new BiFs
	encode_base64() and encode_base64_custom(). (Johanna Amann)

  * Replace call to external "openssl" in extract-certs-pem.bro with
	that encode_base64(). (Johanna Amann)

  * Adding a test for extract-certs-pem.pem. (Robin Sommer)

  * Renaming Base64Decoder to Base64Converter. (Robin Sommer)

2.1-366 | 2013-03-17 12:35:59 -0700

  * Correctly handle DNS lookups for software version ranges. (Seth
    Hall)

  * Improvements to vulnerable software detection. (Seth Hall)

     - Add a DNS based updating method.  This needs to be tested
       still.

     - Vulnerable version ranges are used now instead of only single
       versions.  This can deal with software with multiple stable
       major versions.

  * Update software version parsing and comparison to account for a
    third numeric subversion. Also, $addl is now compared numerically
    if the value is actually numeric. (Seth Hall)

2.1-361 | 2013-03-13 07:18:22 -0700

  * Add check for truncated link frames. Addresses #962. (Jacob
    Baines)

  * Fix large memory allocation in IP fragment reassembly. Addresses
    #961. (Jacob Baines)

2.1-357 | 2013-03-08 09:18:35 -0800

  * Fix race-condition in table-event test. (Johanna Amann)

  * s/bro-ids.org/bro.org/g. (Robin Sommer)

2.1-353 | 2013-03-07 13:31:37 -0800

  * Fix function type-equivalence requiring same parameter names.
    Addresses #957. (Jon Siwek)

2.1-351 | 2013-03-07 13:27:29 -0800

  * Fix new/delete mismatch. Addresses #958. (Jacob Baines)

  * Fix compiler warnings. (Jon Siwek)

2.1-347 | 2013-03-06 16:48:44 -0800

  * Remove unused parameter from vector assignment method. (Johanna Amann)

  * Remove the byte_len() and length() bifs. (Johanna Amann)

2.1-342 | 2013-03-06 15:42:52 -0800

  * Moved the Notice::notice event and Notice::policy table to both be
    hooks. See documentation and NEWS for information. (Seth Hall).

2.1-338 | 2013-03-06 15:10:43 -0800

  * Fix init of local sets/vectors via curly brace initializer lists.
    (Jon Siwek)

2.1-336 | 2013-03-06 15:08:06 -0800

  * Fix memory leaks resulting from 'when' and 'return when'
    statements. Addresses #946. (Jon Siwek)

  * Fix three bugs with 'when' and 'return when' statements. Addresses
    #946. (Jon Siwek)

2.1-333 | 2013-03-06 14:59:47 -0800

  * Add parsing for GTPv1 extension headers and control messages.  (Jon Siwek)

    This includes:

      - A new generic gtpv1_message() event generated for any GTP
        message type.

      - Specific events for the create/update/delete PDP context
        request/response messages.

    Addresses #934.

2.1-331 | 2013-03-06 14:54:33 -0800

  * Fix possible null pointer dereference in identify_data BIF. Also
    centralized libmagic calls for consistent error handling/output.
    (Jon Siwek)

  * Fix build on OpenBSD 5.2. (Jon Siwek)

2.1-328 | 2013-02-05 01:34:29 -0500

  * New script to query the ICSI Certificate Notary
    (http://notary.icsi.berkeley.edu/) over DNS and add information
    to the SSL log at runtime. (Matthias Vallentin)

  * Add delayed logging to SSL base scripts. (Matthias Vallentin)

2.1-319 | 2013-02-04 09:45:34 -0800

  * Update input tests to use exit_only_after_terminate. (Johanna
    Amann)

  * New option exit_only_after_terminate to prevent Bro from exiting.
    If set, the main loop won't terminate before somebody calls
    terminate(). (Robin Sommer)

2.1-311 | 2013-02-01 08:03:01 -0800

  * Updating submodule(s).

2.1-310 | 2013-01-30 20:09:27 -0800

  * Add an error for record coercions that would orphan a field. (Jon
    Siwek)

  * Fixing several scripts where a field in an inlined record was
    never removed after a code refactor. (Jon Siwek)

2.1-307 | 2013-01-25 13:50:57 -0800

  * Fix runaway reference counting bug in record coercion. (Jon Siwek)

  * Fix memory leak in some reporter messaging cases. (Jon Siwek)

2.1-304 | 2013-01-23 19:43:27 -0800

  * Making a test portable. (Robin Sommer)

2.1-302 | 2013-01-23 16:17:29 -0800

  * Refactoring ASCII formatting/parsing from loggers/readers into a
    separate AsciiFormatter class. (Johanna Amann)

  * Fix uninitialized locals in event/hook handlers from having a
    value. Addresses #932. (Jon Siwek)

  * Add a null value check in CompositeHash::ComputeHash. Addresses
    #930. (Jon Siwek)

  * Change reporter messages to more reliably print to stderr.
    Addressed #930 (and revisits #836). (Jon Siwek)

  * Changing test=suite's btest call to use "-j" instead of "-j 5".
    (Robin Sommer)

  * Require "case" blocks to end with either "break", "return", or a
    new "fallthrough" statement that passes control on to the
    subsequent case. This gives us the best mix of safety,
    readability, and flexibility. Addresses #754. (Jon Siwek)

2.1-279 | 2013-01-18 17:18:22 -0800

  * Revert "Trick for parallelizing input framework unit tests." The
    old way of doing the tests seems more reliable for now. (Jon
    Siwek)

  * Fixing variable size issues with http response code in
    ElasticSearch writer. (Gilbert Clark)

  * Removing unused class member. (Robin Sommer)

  * Add opaque type-ignoring for the accept_unsupported_types input
    framework option. (Johanna Amann)

2.1-271 | 2013-01-08 10:18:57 -0800

  * Change substring index notation to use a colon.  String slice
    notation is now written as `s[1:2]`. Addresses #422. (Jon Siwek)

2.1-268 | 2013-01-07 09:43:44 -0800

  * Fix memory leak in OpaqueType::DoUnserialize. (Jon Siwek)

2.1-265 | 2012-12-20 17:38:42 -0800

  * Add array-style index accessor for strings.  Addresses #422. (Jon
    Siwek)

    The index expression can take up to two indices for the start and
    end index of the substring to return (e.g. "mystring[1,3]").
    Negative indices are allowed, with -1 representing the last
    character in the string.  The indexing is not cyclic -- if the
    starting index is >= the length of the string an empty string is
    returned, and if the ending index is >= the length of the string
    then it's interpreted as the last index of the string.  Assigning
    to substrings accessed like this isn't allowed.

2.1-263 | 2012-12-20 16:22:09 -0800

  * Bro's language now has a new set of types "opaque of X". (Matthias
    Vallentin)

    Opaque values can be passed around like other values but they can
    only be manipulated with BiF functions, not with other operators.
    Currently, the following opaque types are supported:

        - opaque of md5
        - opaque of sha1
        - opaque of sha256
        - opaquey of entropy.

    They go along with the corrsponding BiF functions md5_*, sha1_*,
    sha256_*, and entropy_*, respectively. Note that these functions
    have changed their signatures to work with opaques types rather
    than global state as it was before.

2.1-240 | 2012-12-20 15:21:07 -0800

  * Improve error for invalid use of types as values. Addresses #923.
    (Jon Siwek)

2.1-238 | 2012-12-20 15:11:25 -0800

  * Finish implementation of script-layer switch statement. Addresses
    #754. (Jon Siwek)

    They behave like C-style switches except case labels can be
    comprised of multiple literal constants delimited by commas.  Only
    atomic types are allowed for now.  Case label bodies that don't
    execute a "return" or "break" statement will fall through to
    subsequent cases.  A default case label is allowed.

  * Fix a case where c$resp$size is misrepresented.  Addresses #730.
    (Jon Siwek)

2.1-234 | 2012-12-20 12:12:19 -0800

  * Fix return value of hook calls that have no handlers. For this
    case, the return value is always true. (Jon Siwek)

  * Fix to_port() BIF for port strings with a port number of zero.
    (Jon Siwek)

2.1-231 | 2012-12-14 14:51:35 -0800

  * Make const variables actually constant. Both local and global
    variables declared with "const" could be modified, but now
    expressions that would modify them generate an error message at
    parse-time. Addresses #922. (Jon Siwek)

2.1-229 | 2012-12-14 14:46:12 -0800

  * Fix memory leak in ASCII reader when encoutering errors in input.
    (Johanna Amann)

  * Improvements for the "bad checksums" detector to make it detect
    bad TCP checksums. (Seth Hall)

2.1-223 | 2012-12-12 14:25:15 -0800

  * Trick for parallelizing input framework unit tests. Instead of
    loading listen.bro to block until files are read, just read a pcap
    file in pseudo-realtime. (Jon Siwek)

  * Fix reliability of a unit test that relies on when statements.
    (Jon Siwek)

  * Remove unused attributes. (Daniel Thayer)
        - Removed attributes &postprocessor and &match from documentation and source code.
        - Removed undocumented attribute &attr from source code.
        - Removed internal attribute "(&tracked)" from documentation.

2.1-218 | 2012-12-10 14:45:04 -0800

  * Add GPRS Tunnelling Protocol (GTPv1) decapsulation. This currently
    supports automatic decapsulation of GTP-U packets on UDP port 2152.
    The GTPv1 headers for such tunnels can be inspected by handling
    the "gtpv1_g_pdu_packet" event, which has a parameter of type
    "gtpv1_hdr". Addresses #690. (Jon Siwek; derived from patch by
    Carsten Langer)

  * Change BinPAC exceptions in AYIYA/GTP analyzers to do
    "protocol_violation". (Jon Siwek)

2.1-212 | 2012-12-07 19:42:03 -0800

  * Changing the HTTP parser to accept request methods in alignment
    with the RFC. (Robin Sommer)

2.1-209 | 2012-12-05 16:44:04 -0800

  * Adapting the HTTP request line parsing to only accept methods
    consisting of letters [A-Za-z]. (Robin Sommer)

2.1-207 | 2012-12-05 15:47:32 -0800

  * Reporting warnings if kill/waitpid fail in communication system.
    (Bill Parker)

  * Replace() bzero with memset(). (Bill Parker)

  * Merge remote-tracking branch 'vlad/topic/vladg/http-verbs'

    * vlad/topic/vladg/http-verbs:
      A test for HTTP methods, including some horribly illegal requests.
      Remove hardcoded HTTP verbs from the analyzer (#741)

    I added a "bad_HTTP_request" weird for HTTP request lines that don't
    have more than a single word.

    Closes #741. (Robin Sommer)

  * A test for HTTP methods, including some horribly illegal requests. (Vlad Grigorescu)

  * Remove hardcoded HTTP verbs from the analyzer (#741) (Vlad Grigorescu)


2.1-203 | 2012-12-05 14:36:56 -0800

  * Fix segfault: Synchronization of state between connecting peers
    now skips over identifiers that aren't initialized with a value
    yet. Addresses #66. (Jon Siwek)

  * Fix segfault: Delete correct entry in error case in input
    framework. (Johanna Amann)

  * Bad record constructor initializers now give an error. Addresses
    #34. (Jon Siwek)

  * Invalid vector indices now generate error message. Addresses #24.
    (Jon Siwek)

  * Bump CPack RPM package requirement to Python >= 2.6.0. (Jon Siwek)

  * Interpreter exceptions occurring in "when" blocks are now handled.
    Addresses #779 (Jon Siwek)

2.1-195 | 2012-12-03 14:50:33 -0800

  * Catching out-of-memory in patricia tree code. (Bill Parker)

2.1-194 | 2012-12-03 14:36:26 -0800

  * Renaming ASCII writer filter option 'only_single_header_row' to
    'tsv'. Also clarifying usage. Closes #912. (Robin Sommer)

2.1-193 | 2012-12-03 14:11:14 -0800

  * Fix a set of bugs with table/set attributes. (Jon Siwek)

    - Identifiers that are initialized with set()/table() constructor
      expressions now inherit attributes from the expression.  Before,
      statements like

         const i: set[string] = set() &redef;

      associated the attribute with the set() constructor, but not the
      "i" identifier, preventing redefinition.  Addresses #866.

    - Allow &default attribute to apply to tables initialized as empty
      (via either "{ }" or "table()") or if the expression supplied to it
      can evaluate to a type that's promotable to the same yield type as
      the table.

2.1-191 | 2012-12-03 14:08:56 -0800

  * Add test of record() constructor to table initializer unit test.
    (Jon Siwek)

  * Fix table(), set(), vector() constructors in table initializer
    lists. Also adds type checking of yield values to table()
    constructor and fixes the type checking of yield values in
    vector() constructor. Addresses #5. (Jon Siwek)

2.1-188 | 2012-12-03 14:04:29 -0800

  * Hook functions now callable with "hook" expression (i.e., hook is
    no longer a statement). The return value of the call is an
    implicit boolean value of T if all hook handlers ran, or F if one
    hook handler exited as a result of a break statement and
    potentially prevented other handlers from running.

    Scripts don't need to declare hooks with an explicit return type of bool
    (internally, that's assumed), and any values given to (optional) return
    statements in handler definitions are just ignored.

    Addresses #918. (Jon Siwek)

  * Clarification in hook documentation. (Jon Siwek)

2.1-184 | 2012-12-03 13:59:50 -0800

  * Slightly fix up file name extraction from Content-Disposition
    headers. (Seth Hall)

  * Adding -b flag to bro in unit tests so they run faster.

  * Fixed a DNS attribute issue. Reported by Matt Thompson. (Seth
    Hall)

  * Adding NEWS placeholder for hooks and CSV mode. (Robin Sommer)

2.1-178 | 2012-11-23 19:35:32 -0800

  * The ASCII writer now supports a new filter config option
    "only_single_header_row" that turns the output into CSV format
    when set to "T". (Carsten Langer)

  * Add new function flavor called a "hook". This new flavor of
    function behaves like a "synchronous event". See
    doc/scripts/builtins.rst more details on usage. (Jon Siwek)

  * Improve auto-generated enum documentation. The names of enum types
    are tracked so that variables holding a value of a given enum type
    can generate a reference to it instead of just listing the type as
    a generic "enum". (Jon Siwek)

2.1-171 | 2012-11-23 18:24:15 -0800

  * Fix ambiguity between composite table index and record ctor
    expressions. If a table type is "global t = table[conn_id, bool]
    of count", then checking membership like "[c$id, is_orig] in t"
    now works. Addresses #80. (Jon Siwek)

2.1-169 | 2012-11-23 18:21:32 -0800

  * Fix some warnings from sphinx when building docs. (Jon Siwek)

2.1-167 | 2012-11-14 13:19:17 -0800

  * Add a new BIF "bytestring_to_double" for converting from a binary
    representation of a double. Addresses #908. (Carsten Langer/Daniel
    Thayer)

2.1-162 | 2012-11-13 17:29:00 -0800

  * Fix modbus register array parsing. (Jon Siwek)

  * Adjustments to modbus test cases. (Jon Siwek)

2.1-157 | 2012-11-08 16:22:00 -0800

  * Fix for lookup_hostname BIF. (Jon Siwek)

  * Fix for modbus test portability. (Robin Sommer)

2.1-152 | 2012-11-05 16:52:34 -0800

  * Initial version of a completely reworked intelligence framework.
    See doc/intel.rst for more information. (Seth Hall)

  * Experimental Modbus analyzer. See policy/protocols/modbus/* for
    example policies. (Dina Hadziosmanovic, Seth Hall)

2.1-112 | 2012-11-05 13:58:20 -0800

  * New base script for detecting cases of checksum offloading.
    Reporter messages will now tell if one has bad checksums. (Seth
    Hall)

  * Clarifying ownership rules for BroString constructors. (Robin
    Sommer)

2.1-109 | 2012-11-05 13:39:34 -0800

  * Add detection rate threshold for MHR. (Vlad Grigorescu)

  * lookup_hostname_txt fixes. (Vlad Grigorescu)

2.1-104 | 2012-11-01 10:37:50 -0700

  * A new built-in function lookup_hostname_txt() provides support for
    DNS TXT queries. (Vlad Grigorescu)

2.1-101 | 2012-10-31 14:30:26 -0700

  * Documentation reorg: The install info has been consolidated into a
    single document (INSTALL), the upgrade info has been moved from
    the FAQ to a section in the install doc, and the "upgrading from
    1.5 to 2.0" document has been updated (and renamed) to also
    include 2.0 to 2.1 upgrade info. (Daniel Thayer)

2.1-96 | 2012-10-31 14:23:50 -0700

  * Renaming option defining the frequency of alarm summary mails to
    'Logging::default_alarm_mail_interval'. (Daniel Thayer)

2.1-91 | 2012-10-24 16:04:47 -0700

  * Adding PPPoE support to Bro. (Seth Hall)

2.1-87 | 2012-10-24 15:40:06 -0700

  * Adding missing &redef for some TCP options. Addresses #905, #906,
    #907. (Carsten Langer)

2.1-86 | 2012-10-24 15:37:11 -0700

  * Add parsing rules for IPv4/IPv6 subnet literal constants.
    Addresses #888. (Jon Siwek)

2.1-84 | 2012-10-19 15:12:56 -0700

  * Added a BiF strptime() to wrap the corresponding C function. (Seth
    Hall)

2.1-82 | 2012-10-19 15:05:40 -0700

  * Add IPv6 support to signature header conditions. (Jon Siwek)

    - "src-ip" and "dst-ip" conditions can now use IPv6 addresses/subnets.
      They must be written in colon-hexadecimal representation and enclosed
      in square brackets (e.g. [fe80::1]).  Addresses #774.

    - "icmp6" is now a valid protocol for use with "ip-proto" and "header"
      conditions.  This allows signatures to be written that can match
      against ICMPv6 payloads.  Addresses #880.

    - "ip6" is now a valid protocol for use with the "header" condition.
      (also the "ip-proto" condition, but it results in a no-op in that
      case since signatures apply only to the inner-most IP packet when
      packets are tunneled).  This allows signatures to match specifically
      against IPv6 packets (whereas "ip" only matches against IPv4 packets).

    - "ip-proto" conditions can now match against IPv6 packets.  Before,
      IPv6 packets were just silently ignored which meant DPD based on
      signatures did not function for IPv6 -- protocol analyzers would only
      get attached to a connection over IPv6 based on the well-known ports
      set in the "dpd_config" table.

2.1-80 | 2012-10-19 14:48:42 -0700

  * Change how "gridftp" gets added to service field of connection
    records. In addition to checking for a finished SSL handshake over
    an FTP connection, it now also requires that the SSL handshake
    occurs after the FTP client requested AUTH GSSAPI, more
    specifically identifying the characteristics of GridFTP control
    channels. Addresses #891. (Jon Siwek)

  * Allow faster rebuilds in certain cases. Previously, when
    rebuilding with a different "--prefix" or "--scriptdir", all Bro
    source files were recompiled.  With this change, only util.cc is
    recompiled. (Daniel Thayer)

2.1-76 | 2012-10-12 10:32:39 -0700

  * Add support for recognizing GridFTP connections as an extension to
    the standard FTP analyzer. (Jon Siwek)

    This is enabled by default and includes:

      - An analyzer for GSI mechanism of GSSAPI FTP AUTH method. GSI
        authentication involves an encoded TLS/SSL handshake over the
        FTP control session. For FTP sessions that attempt GSI
        authentication, the *service* field of the connection log will
        include "gridftp" (as well as also "ftp" and "ssl").

      - Add an example of a GridFTP data channel detection script. It
        relies on the heuristics of GridFTP data channels commonly
        default to SSL mutual authentication with a NULL bulk cipher
        and that they usually transfer large datasets (default
        threshold of script is 1 GB).  The script also defaults to
        skip_further_processing() after detection to try to save
        cycles analyzing the large, benign connection.

        For identified GridFTP data channels, the *services* fields of
        the connection log will include "gridftp-data".

  * Add *client_subject* and *client_issuer_subject* as &log'd fields
    to SSL::Info record.  Also add *client_cert* and
    *client_cert_chain* fields to track client cert chain. (Jon Siwek)

  * Add a script in base/protocols/conn/polling that generalizes the
    process of polling a connection for interesting features. The
    GridFTP data channel detection script depends on it to monitor
    bytes transferred. (Jon Siwek)

2.1-68 | 2012-10-12 09:46:41 -0700

  * Rename the Input Framework's update_finished event to end_of_data.
    It will now not only fire after table-reads have been completed,
    but also after the last event of a whole-file-read (or
    whole-db-read, etc.). (Johanna Amann)

  * Fix for DNS log problem when a DNS response is seen with 0 RRs.
    (Seth Hall)

2.1-64 | 2012-10-12 09:36:41 -0700

  * Teach --disable-dataseries/--disable-elasticsearch to ./configure.
    Addresses #877. (Jon Siwek)

  * Add --with-curl option to ./configure. Addresses #877. (Jon Siwek)

2.1-61 | 2012-10-12 09:32:48 -0700

  * Fix bug in the input framework: the config table did not work.
    (Johanna Amann)

2.1-58 | 2012-10-08 10:10:09 -0700

  * Fix a problem with non-manager cluster nodes applying
    Notice::policy. This could, for example, result in duplicate
    emails being sent if Notice::emailed_types is redef'd in local.bro
    (or any script that gets loaded on all cluster nodes). (Jon Siwek)

2.1-56 | 2012-10-03 16:04:52 -0700

  * Add general FAQ entry about upgrading Bro. (Jon Siwek)

2.1-53 | 2012-10-03 16:00:40 -0700

  * Add new Tunnel::delay_teredo_confirmation option that indicates
    that the Teredo analyzer should wait until it sees both sides of a
    connection using a valid Teredo encapsulation before issuing a
    protocol_confirmation. Default is on. Addresses #890. (Jon Siwek)

2.1-50 | 2012-10-02 12:06:08 -0700

  * Fix a typing issue that prevented the ElasticSearch timeout to
    work. (Matthias Vallentin)

  * Use second granularity for ElasticSearch timeouts. (Matthias
    Vallentin)

  * Fix compile issues with older versions of libcurl, which don't
    offer *_MS timeout constants. (Matthias Vallentin)

2.1-47 | 2012-10-02 11:59:29 -0700

  * Fix for the input framework: BroStrings were constructed without a
    final \0, which makes them unusable by basically all internal
    functions (like to_count). (Johanna Amann)

  * Remove deprecated script functionality (see NEWS for details).
    (Daniel Thayer)

2.1-39 | 2012-09-29 14:09:16 -0700

  * Reliability adjustments to istate tests with network
    communication. (Jon Siwek)

2.1-37 | 2012-09-25 14:21:37 -0700

  * Reenable some tests that previously would cause Bro to exit with
    an error. (Daniel Thayer)

  * Fix parsing of large integers on 32-bit systems. (Daniel Thayer)

  * Serialize language.when unit test with the "comm" group. (Jon
    Siwek)

2.1-32 | 2012-09-24 16:24:34 -0700

  * Fix race condition in language/when.bro test. (Daniel Thayer)

2.1-26 | 2012-09-23 08:46:03 -0700

  * Add an item to FAQ page about broctl options. (Daniel Thayer)

  * Add more language tests. We now have tests of all built-in Bro
    data types (including different representations of constant
    values, and max./min. values), keywords, and operators (including
    special properties of certain operators, such as short-circuit
    evaluation and associativity). (Daniel Thayer)

  * Fix construction of ip6_ah (Authentication Header) record values.

    Authentication Headers with a Payload Len field set to zero would
    cause a crash due to invalid memory allocation because the
    previous code assumed Payload Len would always be great enough to
    contain all mandatory fields of the header. (Jon Siwek)

  * Update compile/dependency docs for OS X. (Jon Siwek)

  * Adjusting Mac binary packaging script. Setting CMAKE_PREFIX_PATH
    helps link against standard system libs instead of ones that come
    from other package manager (e.g. MacPorts). (Jon Siwek)

  * Adjusting some unit tests that do cluster communication. (Jon Siwek)

  * Small change to non-blocking DNS initialization. (Jon Siwek)

  * Reorder a few statements in scan.l to make 1.5msecs etc work.
    Adresses #872. (Johanna Amann)

2.1-6 | 2012-09-06 23:23:14 -0700

  * Fixed a bug where "a -= b" (both operands are intervals) was not
    allowed in Bro scripts (although "a = a - b" is allowed). (Daniel
    Thayer)

  * Fixed a bug where the "!=" operator with subnet operands was
    treated the same as the "==" operator. (Daniel Thayer)

  * Add sleeps to configuration_update test for better reliability.
    (Jon Siwek)

  * Fix a segfault when iterating over a set when using malformed
    index. (Daniel Thayer)

2.1 | 2012-08-28 16:46:42 -0700

  * Make bif.identify_magic robust against FreeBSD's libmagic config.
    (Robin Sommer)

  * Remove automatic use of gperftools on non-Linux systems.
    --enable-perftools must now explicity be supplied to ./configure
    on non-Linux systems to link against the tcmalloc library.

  * Fix uninitialized value for 'is_partial' in TCP analyzer. (Jon
    Siwek)

  * Parse 64-bit consts in Bro scripts correctly. (Johanna Amann)

  * Output 64-bit counts correctly on 32-bit machines (Johanna Amann)

  * Input framework fixes, including:  (Johanna Amann)

    - One of the change events got the wrong parameters.

    - Escape commas in sets and vectors that were unescaped before
      tokenization.

    - Handling of zero-length-strings as last element in a set was
      broken (sets ending with a ,).

    - Hashing of lines just containing zero-length-strings was broken.

    - Make set_separators different from , work for input framework.

    - Input framework was not handling counts and ints out of
      32-bit-range correctly.

    - Errors in single lines do not kill processing, but simply ignore
      the line, log it, and continue.

  * Update documentation for builtin types. (Daniel Thayer)

    - Add missing description of interval "msec" unit.

    - Improved description of pattern by clarifying the issue of
      operand order and difference between exact and embedded
      matching.

  * Documentation fixes for signature 'eval' conditions. (Jon Siwek)

  * Remove orphaned 1.5 unit tests. (Jon Siwek)

  * Add type checking for signature 'eval' condition functions. (Jon
    Siwek)

  * Adding an identifier to the SMTP blocklist notices for duplicate
    suppression. (Seth Hall)

2.1-beta-45 | 2012-08-22 16:11:10 -0700

  * Add an option to the input framework that allows the user to chose
    to not die upon encountering files/functions. (Johanna Amann)

2.1-beta-41 | 2012-08-22 16:05:21 -0700

  * Add test serialization to "leak" unit tests that use
    communication. (Jon Siwek)

  * Change to metrics/basic-cluster unit test for reliability. (Jon
    Siwek)

  * Fixed ack tracking which could overflow quickly in some
    situations. (Seth Hall)

  * Minor tweak to coverage.bare-mode-errors unit test to work with a
    symlinked 'scripts' dir. (Jon Siwek)

2.1-beta-35 | 2012-08-22 08:44:52 -0700

  * Add testcase for input framework reading sets (rather than
    tables). (Johanna Amann)

2.1-beta-31 | 2012-08-21 15:46:05 -0700

  * Tweak to rotate-custom.bro unit test. (Jon Siwek)

  * Ignore small mem leak every rotation interval for dataseries logs.
    (Jon Siwek)

2.1-beta-28 | 2012-08-21 08:32:42 -0700

  * Linking ES docs into logging document. (Robin Sommer)

2.1-beta-27 | 2012-08-20 20:06:20 -0700

  * Add the Stream record to Log:active_streams to make more dynamic
    logging possible. (Seth Hall)

  * Fix portability of printing to files returned by
    open("/dev/stderr"). (Jon Siwek)

  * Fix mime type diff canonifier to also skip mime_desc columns. (Jon
    Siwek)

  * Unit test tweaks/fixes. (Jon Siwek)

    - Some baselines for tests in "leaks" group were outdated.

    - Changed a few of the cluster/communication tests to terminate
      more explicitly instead of relying on btest-bg-wait to kill
      processes.  This makes the tests finish faster in the success case
      and makes the reason for failing clearer in the that case.

  * Fix memory leak of serialized IDs when compiled with
    --enable-debug. (Jon Siwek)

2.1-beta-21 | 2012-08-16 11:48:56 -0700

  * Installing a handler for running out of memory in "new". Bro will
    now print an error message in that case rather than abort with an
    uncaught exception. (Robin Sommer)

2.1-beta-20 | 2012-08-16 11:43:31 -0700

  * Fixed potential problems with ElasticSearch output plugin. (Seth
    Hall)

2.1-beta-13 | 2012-08-10 12:28:04 -0700

  * Reporter warnings and error now print to stderr by default. New
	options Reporter::warnings_to_stderr and
	Reporter::errors_to_stderr to disable. (Seth Hall)

2.1-beta-9 | 2012-08-10 12:24:29 -0700

  * Add more BIF tests. (Daniel Thayer)

2.1-beta-6 | 2012-08-10 12:22:52 -0700

  * Fix bug in input framework with an edge case. (Johanna Amann)

  * Fix small bug in input framework test script. (Johanna Amann)

2.1-beta-3 | 2012-08-03 10:46:49 -0700

  * Merge branch 'master' of ssh://git.bro-ids.org/bro (Robin Sommer)

  * Fix configure script to exit with non-zero status on error (Jon
    Siwek)

  * Improve ASCII output performance. (Robin Sommer)

2.1-beta | 2012-07-30 11:59:53 -0700

  * Improve log filter compatibility with remote logging. Addresses
    #842. (Jon Siwek)

2.0-907 | 2012-07-30 09:13:36 -0700

  * Add missing breaks to switch cases in
    ElasticSearch::HTTPReceive(). (Jon Siwek)

2.0-905 | 2012-07-28 16:24:34 -0700

  * Fix log manager hanging on waiting for pending file rotations,
    plus writer API tweak for failed rotations. Addresses #860. (Jon
    Siwek and Robin Sommer)

  * Tweaking logs-to-elasticsearch.bro so that it doesn't do anything
    if ES server is unset. (Robin Sommer)

2.0-902 | 2012-07-27 12:42:13 -0700

  * New variable in logging framework Log::active_streams to indicate
    Log:ID enums which are currently active. (Seth Hall)

  * Reworked how the logs-to-elasticsearch scripts works to stop
    abusing the logging framework. (Seth Hall)

  * Fix input test for recent default change on fastpath. (Robin
    Sommer)

2.0-898 | 2012-07-27 12:22:03 -0700

  * Small (potential performance) improvement for logging framework. (Seth Hall)

  * Script-level rotation postprocessor fix. This fixes a problem with
    writers that don't have a postprocessor. (Seth Hall)

  * Update input framework documentation to reflect want_record
    change. (Johanna Amann)

  * Fix crash when encountering an InterpreterException in a predicate
    in logging or input Framework. (Johanna Amann)

  * Input framework: Make want_record=T the default for events
    (Johanna Amann)

  * Changing the start/end markers in logs to open/close now
    reflecting wall clock. (Robin Sommer)

2.0-891 | 2012-07-26 17:15:10 -0700

  * Reader/writer API: preventing plugins from receiving further
    messages after a failure. (Robin Sommer)

  * New test for input framework that fails to find a file. (Robin
    Sommer)

  * Improving error handling for threads. (Robin Sommer)

  * Tweaking the custom-rotate test to produce stable output. (Robin
    Sommer)

2.0-884 | 2012-07-26 14:33:21 -0700

  * Add comprehensive error handling for close() calls. (Jon Siwek)

  * Add more test cases for input framework. (Johanna Amann)

  * Input framework: make error output for non-matching event types
    much more verbose. (Johanna Amann)

2.0-877 | 2012-07-25 17:20:34 -0700

  * Fix double close() in FilerSerializer class. (Jon Siwek)

  * Fix build warnings. (Daniel Thayer)

  * Fixes to ElasticSearch plugin to make libcurl handle http
    responses correctly. (Seth Hall)

  * Fixing FreeBSD compiler error. (Robin Sommer)

  * Silencing compiler warnings. (Robin Sommer)

2.0-871 | 2012-07-25 13:08:00 -0700

  * Fix complaint from valgrind about uninitialized memory usage. (Jon
    Siwek)

  * Fix differing log filters of streams from writing to same
    writer/path (which now produces a warning, but is otherwise
    skipped for the second). Addresses #842. (Jon Siwek)

  * Fix tests and error message for to_double BIF. (Daniel Thayer)

  * Compile fix. (Robin Sommer)

2.0-866 | 2012-07-24 16:02:07 -0700

  * Correct a typo in usage message. (Daniel Thayer)

  * Fix file permissions of log files (which were created with execute
    permissions after a recent change). (Daniel Thayer)

2.0-862 | 2012-07-24 15:22:52 -0700

  * Fix initialization problem in logging class. (Jon Siwek)

  * Input framework now accepts escaped ASCII values as input (\x##),
    and unescapes appropiately. (Johanna Amann)

  * Make reading ASCII logfiles work when the input separator is
    different from \t. (Johanna Amann)

  * A number of smaller fixes for input framework. (Johanna Amann)

2.0-851 | 2012-07-24 15:04:14 -0700

  * New built-in function to_double(s: string). (Scott Campbell)

2.0-849 | 2012-07-24 11:06:16 -0700

  * Adding missing include needed on some systems. (Robin Sommer)

2.0-846 | 2012-07-23 16:36:37 -0700

  * Fix WriterBackend::WriterInfo serialization, reenable ascii
    start/end tags. (Jon Siwek)

2.0-844 | 2012-07-23 16:20:59 -0700

  * Reworking parts of the internal threading/logging/input APIs for
    thread-safety. (Robin Sommer)

  * Bugfix for SSL version check. (Johanna Amann)

  * Changing a HTTP DPD from port 3138 to 3128. Addresses #857. (Robin
    Sommer)

  * ElasticSearch logging writer. See logging-elasticsearch.rst for
    more information. (Vlad Grigorescu and Seth Hall).

  * Give configure a --disable-perftools option to disable Perftools
    support even if found. (Robin Sommer)

  * The ASCII log writer now includes "#start <timestamp>" and "#end
    <timestamp> lines in the each file. (Robin Sommer)

  * Renamed ASCII logger "header" options to "meta". (Robin Sommer)

  * ASCII logs now escape '#' at the beginning of log lines. Addresses
    #763. (Robin Sommer)

  * Fix bug, where in dns.log rcode always was set to 0/NOERROR when
    no reply package was seen. (Johanna Amann)

  * Updating to Mozilla's current certificate bundle. (Seth Hall)

2.0-769 | 2012-07-13 16:17:33 -0700

  * Fix some Info:Record field documentation. (Vlad Grigorescu)

  * Fix overrides of TCP_ApplicationAnalyzer::EndpointEOF. (Jon Siwek)

  * Fix segfault when incrementing whole vector values. Also removed
    RefExpr::Eval(Val*) method since it was never called. (Jon Siwek)

  * Remove baselines for some leak-detecting unit tests. (Jon Siwek)

  * Unblock SIGFPE, SIGILL, SIGSEGV and SIGBUS for threads, so that
    they now propagate to the main thread. Adresses #848. (Johanna
    Amann)

2.0-761 | 2012-07-12 08:14:38 -0700

  * Some small fixes to further reduce SOCKS false positive logs. (Seth Hall)

  * Calls to pthread_mutex_unlock now log the reason for failures.
    (Johanna Amann)

2.0-757 | 2012-07-11 08:30:19 -0700

  * Fixing memory leak. (Seth Hall)

2.0-755 | 2012-07-10 16:25:16 -0700

  * Add sorting canonifier to rotate-custom unit test. Addresses #846.
    (Jon Siwek)

  * Fix many compiler warnings. (Daniel Thayer)

  * Fix segfault when there's an error/timeout resolving DNS requests.
    Addresses #846. (Jon Siwek)

  * Remove a non-portable test case. (Daniel Thayer)

  * Fix typos in input framework doc. (Daniel Thayer)

  * Fix typos in DataSeries documentation. (Daniel Thayer)

  * Bugfix making custom rotate functions work again. (Robin Sommer)

  * Tiny bugfix for returning writer name. (Robin Sommer)

  * Moving make target update-doc-sources from top-level Makefile to
    btest Makefile. (Robin Sommer)

2.0-733 | 2012-07-02 15:31:24 -0700

  * Extending the input reader DoInit() API. (Johanna Amann). It now
    provides a Info struct similar to what we introduced for log
    writers, including a corresponding "config" key/value table.

  * Fix to make writer-info work when debugging is enabled. (Johanna
    Amann)

2.0-726 | 2012-07-02 15:19:15 -0700

  * Extending the log writer DoInit() API. (Robin Sommer)

    We now pass in a Info struct that contains:

        - the path name (as before)
        - the rotation interval
        - the log_rotate_base_time in seconds
        - a table of key/value pairs with further configuration options.

    To fill the table, log filters have a new field "config: table[string]
    of strings". This gives a way to pass arbitrary values from
    script-land to writers. Interpretation is left up to the writer.

  * Split calc_next_rotate() into two functions, one of which is
    thread-safe and can be used with the log_rotate_base_time value
    from DoInit().

  * Updates to the None writer. (Robin Sommer)

        - It gets its own script writers/none.bro.

        - New bool option LogNone::debug to enable debug output. It then
          prints out all the values passed to DoInit().

        - Fixed a bug that prevented Bro from terminating.

2.0-723 | 2012-07-02 15:02:56 -0700

  * Extract ICMPv6 NDP options and include in ICMP events.  This adds
    a new parameter of type "icmp6_nd_options" to the ICMPv6 neighbor
    discovery events. Addresses #833. (Jon Siwek)

  * Set input frontend type before starting the thread. This means
    that the thread type will be output correctly in the error
    message. (Johanna Amann)

2.0-719 | 2012-07-02 14:49:03 -0700

  * Fix inconsistencies in random number generation. The
    srand()/rand() interface was being intermixed with the
    srandom()/random() one.  The later is now used throughout. (Jon
    Siwek)

  * Changed the srand() and rand() BIFs to work deterministically if
    Bro was given a seed file. Addresses #825. (Jon Siwek)

  * Updating input framework unit tests to make them more reliable and
    execute quicker. (Jon Siwek)

  * Fixed race condition in writer and reader initializations. (Jon
    Siwek)

  * Small tweak to make test complete quicker. (Jon Siwek)

  * Drain events before terminating log/thread managers. (Jon Siwek)

  * Fix strict-aliasing warning in RemoteSerializer.cc. Addresses
    #834. (Jon Siwek)

  * Fix typos in event documentation. (Daniel Thayer)

  * Fix typos in NEWS for Bro 2.1 beta. (Daniel Thayer)

2.0-709 | 2012-06-21 10:14:24 -0700

  * Fix exceptions thrown in event handlers preventing others from running. (Jon Siwek)

  * Add another SOCKS command. (Seth Hall)

  * Fixed some problems with the SOCKS analyzer and tests. (Seth Hall)

  * Updating NEWS in preparation for beta. (Robin Sommer)

  * Accepting different AF_INET6 values for loopback link headers.
    (Robin Sommer)

2.0-698 | 2012-06-20 14:30:40 -0700

  * Updates for the SOCKS analyzer (Seth Hall).

    - A SOCKS log!

    - Now supports SOCKSv5 in the analyzer and the DPD sigs.

    - Added protocol violations.

  * Updates to the tunnels framework. (Seth Hall)

    - Make the uid field optional since it's conceptually incorrect
      for proxies being treated as tunnels to have it.

    - Reordered two fields in the log.

    - Reduced the default tunnel expiration interface to something
      more reasonable (1 hour).

  * Make Teredo bubble packet parsing more lenient. (Jon Siwek)

  * Fix a crash in NetSessions::ParseIPPacket(). (Jon Siwek)

2.0-690 | 2012-06-18 16:01:33 -0700

  * Support for decapsulating tunnels via the new tunnel framework in
    base/frameworks/tunnels.

    Bro currently supports Teredo, AYIYA, IP-in-IP (both IPv4 and
    IPv6), and SOCKS. For all these, it logs the outher tunnel
    connections in both conn.log and tunnel.log, and proceeds to
    analyze the inner payload as if it were not tunneled, including
    also logging it in conn.log (with a new tunnel_parents column
    pointing back to the outer connection(s)). (Jon Siwek, Seth Hall,
    Gregor Maier)

  * The options "tunnel_port" and "parse_udp_tunnels" have been
    removed. (Jon Siwek)

2.0-623 | 2012-06-15 16:24:52 -0700

  * Changing an error in the input framework to a warning. (Robin
    Sommer)

2.0-622 | 2012-06-15 15:38:43 -0700

  * Input framework updates. (Johanna Amann)

    - Disable streaming reads from executed commands. This lead to
      hanging Bros because pclose apparently can wait for eternity if
      things go wrong.

     - Automatically delete disabled input streams.

     - Documentation.

2.0-614 | 2012-06-15 15:19:49 -0700

  * Remove an old, unused diff canonifier. (Jon Siwek)

  * Improve an error message in ICMP analyzer. (Jon Siwek)

  * Fix a warning message when building docs. (Daniel Thayer)

  * Fix many errors in the event documentation. (Daniel Thayer)

2.0-608 | 2012-06-11 15:59:00 -0700

  * Add more error handling code to logging of enum vals. Addresses
    #829. (Jon Siwek)

2.0-606 | 2012-06-11 15:55:56 -0700

  * Fix summary lines for BIF documentation and corrected the
    description of "fmt" and "floor" BIFs. (Daniel Thayer)

  * Fix val_size BIF tests and improve docs. (Daniel Thayer)

2.0-602 | 2012-06-07 15:06:19 -0700

  * Include header for usleep(), caused compile failure on Archlinux. (Jon Siwek)

  * Revert "Fixed a bug with the MIME analyzer not removing whitespace
    on wrapped headers." Needs discussion. (Robin Sommer)

2.0-598 | 2012-06-06 11:47:00 -0700

  * Add @load-sigs directive for loading signature files (addresses
    #551). This can be used to load signatures relative to the current
    scripts (e.g., "@load-sigs ./foo.sig"). (Jon Siwek)


2.0-596 | 2012-06-06 11:41:00 -0700

    * Fixes for some BiFs and their documentation. (Daniel Thayer)

    * Many new unit tests for BiFs. (Daniel Thayer)

2.0-579 | 2012-06-06 11:04:46 -0700

  * Memory leak fixes for bad usages of VectorVal ctor. (Jon Siwek)

  * Fixed a bug with the MIME analyzer not removing whitespace on
    wrapped headers. (Seth Hall)

  * Change Input::update_finished lookup to happen at init time. (Jon Siwek)

  * Fix going through the internal_handler() function which will now
    set the event as "used" (i.e. it's marked as being raised
    somewhere). Addresses #823. (Jon Siwek)

  * Fix format specifier on RemoteSerializer::Connect. This caused
    32-bit systems to show a warning at compile-time, and fail when
    connecting to peers. (Jon Siwek)

  * Fixes for running tests in parallel. (Robin Sommer)

2.0-571 | 2012-05-30 19:12:43 -0700

  * Updating submodule(s).

2.0-570 | 2012-05-30 19:08:18 -0700

  * A new input framework enables scripts to read in external data
    dynamically on the fly as Bro is processing network traffic.
    (Johanna Amann)

    Currently, the framework supports reading ASCII input that's
    structured similar as Bro's log files as well as raw blobs of
    data. Other formats will come in the future.

    See doc/input.rst for more information (this will be extended
    further soon).

2.0-395 | 2012-05-30 17:03:31 -0700

  * Remove unnecessary assert in ICMP analyzer which could lead to
    aborts. Addresses #822.

  * Improve script debugger backtrace and print commands. (Jon Siwek)

  * Switching default DS compression to gzip. (Robin Sommer)

  * Improve availability of IPv6 flow label in connection records.
    This adds a "flow_label" field to the "endpoint" record type,
    which is used for both the "orig" and "resp" fields of
    "connection" records.  The new "connection_flow_label_changed"
    event also allows tracking of changes in flow labels: it's raised
    each time one direction of the connection starts using a different
    label. (Jon Siwek)

  * Add unit tests for Broccoli SSL and Broccoli IPv6 connectivity.
    (Jon Siwek)

  * Remove AI_ADDRCONFIG getaddrinfo hints flag for listening sockets.
    (Jon Siwek)

  * Undo unnecessary communication protocol version bump. (Jon Siwek)

  * Add support to Bro for connecting with peers over IPv6. (Jon Siwek)

    - Communication::listen_ipv6 needs to be redef'd to true in order
      for IPv6 listening sockets to be opened.

    - Added Communication::listen_retry option as an interval at which
      to retry binding to socket addresses that were already in use.

    - Added some explicit baselines to check in the istate.events and
      istate.events-ssl tests -- the SSL test was incorrectly passing
      because it compared two empty files.  (The files being empty
      because "http/base" was given as an argument to Bro which it
      couldn't handle because that script doesn't exist anymore).

    - Support for communication over non-global IPv6 addresses. This
      usually requires specifying an additional zone identifier (see
      RFC 4007). The connect() and listen() BIFs have been changed to
      accept this zone identifier as an argument.


2.0-377 | 2012-05-24 16:46:06 -0700

  * Documentation fixes. (Jon Siwek and Daniel Thayer)

2.0-372 | 2012-05-17 13:59:45 -0700

  * Fix compile errors. (Jon Siwek)

  * Linking in the DS docs. (Robin Sommer)

  * Fix mobility checksums unit test. (Jon Siwek)

2.0-367 | 2012-05-17 12:42:30 -0700

  * Adding support for binary output via DataSeries. See
    logging-dataseries.rst for more information. (Gilbert Clark and
    Robin Sommer)

  * Adding target update-doc-sources to top-level Makefile that runs
    genDocSourcesList.sh. (Robin Sommer)

  * Moving trace for rotation test into traces directory. (Robin Sommer)

  * Fixing a rotation race condition at termination. (Robin Sommer)

  * Extending log post-processor call to include the name of the
    writer. (Robin Sommer)

  * In threads, an internal error now immediately aborts. Otherwise,
    the error won't make it back to the main thread for a while and
    subsequent code in the thread would still execute. (Robin Sommer)

  * DataSeries cleanup. (Robin Sommer)

  * Fixing threads' DoFinish() method. It wasn't called reliably. Now,
    it's always called before the thread is destroyed (assuming
    processing has went normally so far). (Robin Sommer)

2.0-341 | 2012-05-17 09:54:30 -0700

  * Add a comment to explain the ICMPv6 error message types. (Daniel Thayer)

  * Quieting external test output somehwat. (Robin Sommer)

2.0-336 | 2012-05-14 17:15:44 -0700

  * Don't print the various "weird" events to stderr. Address #805.
    (Daniel Thayer)

  * Generate icmp_error_message event for ICMPv6 error msgs.
    Previously, icmp_sent was being generated, but icmp_error_message
    contains more info.

  * Improved documentation comments for icmp-related events. (Daniel
    Thayer)

2.0-330 | 2012-05-14 17:05:56 -0700

  * Add `addr_to_uri` script-level function that adds brackets to an
    address if it's IPv6 and will be included in a URI or when a
    ":<port>" needs to be appended to it. (Jon Siwek)

  * Also add a test case for content extraction. (Jon Siwek)

  * Fix typos and improve INSTALL document. (Daniel Thayer)

  * Switching to new btest command TEST-SERIALIZE for communication
    tests. (Robin Sommer)

2.0-323 | 2012-05-04 21:04:34 -0700

  * Add SHA1 and SHA256 hashing BIFs. Addresses #542.

  * Refactor all internal MD5 stuff to use OpenSSL's. (Jon Siwek)

  * Changes to open-file caching limits and uncached file unserialization. (Jon Siwek)

    - Unserializing files that were previously kicked out of the open-file
      cache would cause them to be fopen'd with the original access
      permissions which is usually 'w' and causes truncation.  They
      are now opened in 'a' mode.  (addresses #780)

    - Add 'max_files_in_cache' script option to manually set the maximum
      amount of opened files to keep cached.  Mainly this just helped
      to create a simple test case for the above change.

    - Remove unused NO_HAVE_SETRLIMIT preprocessor switch.

    - On systems that don't enforce a limit on number of files opened for
      the process, raise default max size of open-file cache from
      32 to 512.

2.0-319 | 2012-05-03 13:24:44 -0700

  * SSL bugfixes and cleanup. (Seth Hall)

    - SSL related files and classes renamed to remove the "binpac" term.

    - A small fix for DPD scripts to make the DPD log more helpful if
      there are multiple continued failures.

    - Fixed the SSL analyzer to make it stop doing repeated violation
      messages for some handshake failures.

    - Added a $issuer_subject to the SSL log.

    - Created a basic test for SSL.

    - Fixed parsing of TLS server extensions. (Seth Hall)

2.0-315 | 2012-05-03 11:44:17 -0700

  * Add two more TLS extension values that we see in live traffic.
    (Johanna Amann)

  * Fixed IPv6 link local unicast CIDR and added IPv6 loopback to
    private address space. (Seth Hall)

  * Fixed a problem where cluster workers were still processing
    notices in some cases. (Seth Hall)

  * Added a configure option to specify the 'etc' directory. Addresses
    #801. (Daniel Thayer)


2.0-306 | 2012-04-24 14:37:00 -0700

  * Add further TLS extension values "extended_random" and
    "heartbeat". (Seth Hall)

  * Fix problem with extracting FTP passwords and add "ftpuser" as
    another anonymous username.  (Seth Hall, discovered by Patrik
    Lundin).

2.0-303 | 2012-04-19 10:01:06 -0700

  * Changes related to ICMPv6 Neighbor Discovery messages. (Jon Siwek)

    - The 'icmp_conn' record now contains an 'hlim' field since hop limit
      in the IP header is an interesting field for at least these ND
      messages.

    - Fixed and extended 'icmp_router_advertisement' event parameters.

    - Changed 'icmp_neighbor_advertisement' event parameters to add
      more of the known boolean flags.

2.0-301 | 2012-04-17 17:58:55 -0700

  * Bro now support ICMPv6. (Matti Mantere, Jon Siwek, Robin Sommer,
	Daniel Thayer).

	Overall, Bro now raises the following ICMP events for v4 and v6 as
	appropiate:

        event icmp_sent(c: connection, icmp: icmp_conn);
        event icmp_echo_request(c: connection, icmp: icmp_conn, id: count, seq: count, payload: string);
        event icmp_echo_reply(c: connection, icmp: icmp_conn, id: count, seq: count, payload: string);
        event icmp_error_message(c: connection, icmp: icmp_conn, code: count, context: icmp_context);
        event icmp_unreachable(c: connection, icmp: icmp_conn, code: count, context: icmp_context);
        event icmp_packet_too_big(c: connection, icmp: icmp_conn, code: count, context: icmp_context);
        event icmp_time_exceeded(c: connection, icmp: icmp_conn, code: count, context: icmp_context);
        event icmp_parameter_problem(c: connection, icmp: icmp_conn, code: count, context: icmp_context);
        event icmp_router_solicitation(c: connection, icmp: icmp_conn);
        event icmp_router_advertisement(c: connection, icmp: icmp_conn, hop_limit: count, managed: bool, router_lifetime: count, reachable_time: interval, retrans_timer: interval);
        event icmp_neighbor_solicitation(c: connection, icmp: icmp_conn, tgt:addr);
        event icmp_neighbor_advertisement(c: connection, icmp: icmp_conn, tgt:addr);
        event icmp_redirect(c: connection, icmp: icmp_conn, tgt: addr, dest: addr);

    The `icmp_conn` record got a new boolean field 'v6' that indicates
    whether the ICMP message is v4 or v6.

    This change also includes further low-level work on existing IP
	and ICMP code, including a reorganization of how ICMPv4 is
	handled.

2.0-281 | 2012-04-17 17:40:39 -0700

  * Small updates for the bittorrent analyzer to support 64bit types
    in binpac. (Seth Hall)

  * Removed the attempt at bittorrent resynchronization. (Seth Hall)

2.0-276 | 2012-04-17 17:35:56 -0700

  * Add more support for <netinet/ip6.h>'s that lack some structure
    definitions. (Jon Siwek)

2.0-273 | 2012-04-16 18:08:56 -0700

  * Removing QR flag from DNS log in response, which should not have
    been there in the first place. (Seth Hall)

  * Sync up patricia.c/h with pysubnettree repo. (Daniel Thayer)

  * Adding missing leak groups to a couple tests. Also activating leak
    checking for proxy in basic-cluster test. (Robin Sommer)

2.0-267 | 2012-04-09 17:47:28 -0700

  * Add support for mobile IPv6 Mobility Header (RFC 6275).  (Jon
    Siwek)

    - Enabled through a new --enable-mobile-ipv6 configure-time
      option. If not enabled, the mobility header (routing type 2) and
      Home Address Destination option are ignored.

    - Accessible at script-layer through 'mobile_ipv6_message' event.

  * Refactor IP_Hdr routing header handling, add MobileIPv6 Home
    Address handling. Packets that use the Home Address Destination
    option use that option's address as the connection's originator.
    (Jon Siwek)

  * Revert TCP checksumming to cache common data, like it did before.
    (Jon Siwek)

  * Improve handling of IPv6 routing type 0 extension headers. (Jon
    Siwek)

    - flow_weird event with name argument value of "routing0_hdr" is raised
      for packets containing an IPv6 routing type 0 header because this
      type of header is now deprecated according to RFC 5095.

    - Packets with a routing type 0 header and non-zero segments left
      now use the last address in that header in order to associate
      with a connection/flow and for calculating TCP/UDP checksums.

    - Added a set of IPv4/IPv6 TCP/UDP checksum unit tests (Jon Siwek)

  * Fix table expiry for values assigned in bro_init() when reading
    live. (Jon Siwek)

2.0-257 | 2012-04-05 15:32:43 -0700

  * Fix CMake from warning about unused ENABLE_PERFTOOLS_DEBUG
    variable. (Jon Siwek)

  * Fix handling of IPv6 atomic fragments. (Jon Siwek)

  * Fix that prevents Bro processes that do neither local logging nor
    request remote logs from spawning threads. (Robin Sommer)

  * Fixing perftools-debug support. (Robin Sommer)

  * Reverting SocketComm change tuning I/O behaviour. (Robin Sommer)

  * Adding notice_policy.log canonification for external tests. (Robin Sommer)


2.0-245 | 2012-04-04 17:25:20 -0700

  * Internal restructuring of the logging framework: we now spawn
    threads doing the I/O. From a user's perspective not much should
    change, except that the OS may now show a bunch of Bro threads.
    (Gilbert Clark and Robin Sommer).

  * When building Bro, we now always link in tcmalloc if it's found at
    configure time. If it's installed but not picked up,
    --with-perftools may help. (Robin Sommer)

  * Renaming the configure option --enable-perftools to
    --enable-perftool-debug to indicate that the switch is only
    relevant for debugging the heap. It's not needed to pick up
    tcmalloc for better performance. (Robin Sommer)

2.0-184 | 2012-03-28 15:11:11 -0700

  * Improve handling of IPv6 Routing Type 0 headers.  (Jon Siwek)

    - For RH0 headers with non-zero segments left, a
      "routing0_segleft" flow_weird event is raised (with a
      destination indicating the last address in the routing header),
      and an "rh0_segleft" event can also be handled if the other
      contents of the packet header are of interest. No further
      analysis is done as the complexity required to correctly
      identify destination endpoints of connections doesn't seem worth
      it as RH0 has been deprecated by RFC 5095.

    - For RH0 headers without any segments left, a "routing0_header"
      flow_weird event is raised, but further analysis still occurs as
      normal.

2.0-182 | 2012-03-28 15:01:57 -0700

  * Remove dead tcp_checksum function from net_util. (Jon Siwek)

  * Change routing0_data_to_addrs BIF to return vector of addresses.
    The order of addresses in type 0 routing headers is
    interesting/important. (Jon Siwek)


2.0-179 | 2012-03-23 17:43:31 -0700

  * Remove the default "tcp or udp or icmp" filter. In default mode,
    Bro would load the packet filter script framework which installs a
    filter that allows all packets, but in bare mode (the -b option),
    this old filter would not follow IPv6 protocol chains and thus
    filter out packets with extension headers. (Jon Siwek)

  * Update PacketFilter/Discarder code for IP version independence.
    (Jon Siwek)

  * Fix some IPv6 header related bugs. (Jon Siwek)

  * Add IPv6 fragment reassembly. (Jon Siwek)

  * Add handling for IPv6 extension header chains. Addresses #531.
    (Jon Siwek)

    - The script-layer 'pkt_hdr' type is extended with a new 'ip6' field
      representing the full IPv6 header chain.

    - The 'new_packet' event is now raised for IPv6 packets. Addresses
      #523.

    - A new event called 'ipv6_ext_header' is raised for any IPv6
      packet containing extension headers.

    - A new event called 'esp_packet' is raised for any packets using
      ESP ('new_packet' and 'ipv6_ext_header' events provide
      connection info, but that info can't be provided here since the
      upper-layer payload is encrypted).

    - The 'unknown_protocol' weird is now raised more reliably when
      Bro sees a transport protocol or IPv6 extension header it can't
      handle. Addresses #522.

  * Add unit tests for IPv6 fragment reassembly, ipv6_ext_headers and
    esp_packet events. (Jon Siwek)

  * Adapt FreeBSD's inet_ntop implementation for internal use. Now we
    get consistent text representations of IPv6 addresses across
    platforms. (Jon Siwek)

  * Update documentation for new syntax of IPv6 literals. (Jon Siwek)


2.0-150 | 2012-03-13 16:16:22 -0700

  * Changing the regular expression to allow Site::local_nets in
    signatures. (Julien Sentier)

  * Removing a line of dead code. Found by . Closes #786. (Julien
    Sentier)

2.0-146 | 2012-03-13 15:39:38 -0700

  * Change IPv6 literal constant syntax to require encasing square
    brackets. (Jon Siwek)

2.0-145 | 2012-03-09 15:10:35 -0800

  * Remove the match expression. 'match' and 'using' are no longer
    keywords. Addressed #753. (Jon Siwek)

2.0-143 | 2012-03-09 15:07:42 -0800

  * Fix a BRO_PROFILER_FILE/mkstemp portability issue. Addresses #794.
    (Jon Siwek)

2.0-139 | 2012-03-02 09:33:04 -0800

  * Changes to how script coverage integrates with test suites. (Jon Siwek)

    - BRO_PROFILER_FILE now passes .X* templated filenames to mkstemp
      for generating unique coverage state files.

    - Rearranging Makefile targets.  The general rule is that if the
      all/brief target fails out due to a test failure, then the dependent
      coverage target won't run, but can still be invoked directly later.
      (e.g. make brief || make coverage)

  * Standardized on the &default function for SSL constants. (Seth
    Hall)

  * Adding btest group "leaks" to leak tests. (Robin Sommer)

  * Adding btest group "comm" to communication tests for parallelizing
    execution with new btest version. (Robin Sommer)

  * Sorting all output for diffing in the external tests. (Robin
    Sommer)

  * Cleaned up dead code from the old SSL analyzers. Reported by
    Julien Sentier. (Seth Hall)

  * Update/add tests for broccoli IPv6 addr/subnet support. Addresses
    #448. (Jon Siwek)

  * Remove connection compressor. Addresses #559. (Jon Siwek)

  * Refactor IP_Hdr class ctors. Addresses #532. (Jon Siwek)


2.0-121 | 2012-02-24 16:34:17 -0800

  * A number of smaller memory fixes and code cleanups. (Julien
    Sentier)

  * Add to_subnet bif. Fixes #782). (Jon Siwek)

  * Fix IPAddr::Mask/ReverseMask not allowing argument of 0. (Jon
    Siwek)

  * Refactor IPAddr v4 initialization from string. Fixes #775. (Jon Siwek)

  * Parse the dotted address string directly instead of canonicalizing
    and passing to inet_pton. (Jon Siwek)


2.0-108 | 2012-02-24 15:21:07 -0800

  * Refactoring a number of usages of new IPAddr class. (Jon Siwek)

  * Fixed a bug in remask_addr bif. (Jon Siwek)

2.0-106 | 2012-02-24 15:02:20 -0800

  * Raise minimum required CMake version to 2.6.3. (Jon Siwek)

2.0-104 | 2012-02-24 14:59:12 -0800

  * Add test case for FTP over IPv4. (Daniel Thayer)

  * Fix IPv6 URLs in ftp.log. (Daniel Thayer)

  * Add a test for FTP over IPv6 (Daniel Thayer)

  * Fix parsing of FTP EPRT command and EPSV response. (Daniel Thayer)

2.0-95 | 2012-02-22 05:27:34 -0800

  * GeoIP installation documentation update. (Seth Hall)

  * Decrease strictness of parsing IPv4 strings into addrs. Fixes #775. (Jon Siwek)

  * Fix memory leak in DNS manager. Fixes #777. (Jon Siwek)

  * Fix IPAddr/IPPrefix serialization bugs. (Jon Siwek)

  * Fix compile error. (Jon Siwek)

2.0-86 | 2012-02-17 15:41:06 -0800

  * Changing ARP detection to always kick in even if no analyzer is
    activated. (Robin Sommer)

  * DNS name lookups performed by Bro now also query AAAA records.
    DNS_Mgr handles combining the results of the A and AAAA queries
    for a given hostname such that at the scripting layer, the name
    resolution can yield a set with both IPv4 and IPv6 addresses. (Jon
    Siwek)

  * Add counts_to_addr and addr_to_counts conversion BIFs. (Jon Siwek)

  * Change HashKey threshold for using H3 to 36 bytes. (Jon Siwek)

  * Remove mention of --enable-brov6 in docs. (Daniel Thayer)

  * Remove --enable-brov6 from configure usage text (Daniel Thayer)

  * Add a test and baseline for addr_to_ptr_name BiF. (Daniel Thayer)

  * Adding a test and baseline for ptr_name_to_addr BiF. (Seth Hall)

  * Fix the ptr_name_to_addr BiF to work with IPv6 (Daniel Thayer)

  * Fix a memory leak that perftools now complains about. (Jon Siwek)

  * Remove --enable-brov6 flag, IPv6 now supported by default. (Jon Siwek)

    Some script-layer changes of note:

    - dns_AAAA_reply event signature changed: the string representation
      of an IPv6 addr is easily derived from the addr value, it doesn't
      need to be another parameter.  This event also now generated directly
      by the DNS analyzer instead of being "faked" into a dns_A_reply event.

    - Removed addr_to_count BIF.  It used to return the host-order
      count representation of IPv4 addresses only.  To make it more
      generic, we might later add a BIF to return a vector of counts
      in order to support IPv6.

    - Changed the result of enclosing addr variables in vertical pipes
      (e.g. |my_addr|) to return the bit-width of the address type which
      is 128 for IPv6 and 32 for IPv4.  It used to function the same
      way as addr_to_count mentioned above.

    - Remove bro_has_ipv6 BIF

2.0-57 | 2012-02-10 00:02:35 -0800

  * Fix typos in the documentation. (Daniel Thayer)

  * Fix compiler warning about Brofiler ctor init list order. (Jon Siwek)

  * Fix missing optional field access in webapp signature_match handler. (Jon Siwek)

2.0-41 | 2012-02-03 04:10:53 -0500

  * Updates to the Software framework to simplify the API. (Johanna
    Amann)

2.0-40 | 2012-02-03 01:55:27 -0800

  * Fix typos in documentation. (Daniel Thayer)

  * Fix sorting of lines in Brofiler coverage.log. (Daniel Thayer)

2.0-38 | 2012-01-31 11:50:53 -0800

  * Canonify sorting of lines in Brofiler coverage.log. (Daniel
    Thayer)

2.0-36 | 2012-01-27 10:38:14 -0800

  * New "Brofiler" mode that tracks and records script statements
    executed during runtime. (Jon Siwek)

    Use the BROFILER_FILE environment variable to point to a file in
    which statement usage statistics from Bro script-layer can be
    output.

    Script statements that should be ignored can be marked with a "#
    @no-test" comment. For example:

        print "don't cover"; # @no-test

        if ( F )
            { # @no-test
            ...
            }

  * Integrated coverage measurement into test-suite. (Jon Siwek)

2.0-20 | 2012-01-25 16:34:51 -0800

  * BiF cleanup (Matthias Vallentin)

      - Rename NFS3::mode2string to a more generic file_mode().

      - Unify do_profiling()/make_connection_persistent()/expect_connection()
        to return any (i.e., nothing) instead of bools.

      - Perform type checking on count-to-port conversion. Related to #684.

      - Remove redundant connection_record() BiF. The same
        functionality is provided by lookup_connection().

      - Remove redundant active_connection() BiF. The same
        functionality is provided by connection_exists().

      - exit() now takes the exit code as argument.

      - to_port() now received a string instead of a count.

2.0-9 | 2012-01-25 13:47:13 -0800

  * Allow local table variables to be initialized with {} list
    expressions. (Jon Siwek)

2.0-7 | 2012-01-25 13:38:09 -0800

  * Teach CompHash to allow indexing by records with vector/table/set
    fields. Addresses #464. (Jon Siwek)

2.0-5 | 2012-01-25 13:25:19 -0800

  * Fixed a bug resulting in over-logging of detected webapps. (Seth Hall)

  * Make communication log baseline test more reliable. (Jon Siwek)

  * Fixed some broken links in documentation. (Daniel Thayer)

2.0 | 2012-01-11 13:52:22 -0800

  * Adding script reference documentation. (The Team).

2.0-beta-194 | 2012-01-10 10:44:32 -0800

  * Added an option for filtering out URLs before they are turned into
    HTTP::Incorrect_File_Type notices. (Seth Hall)

  * Fix ref counting bug in BIFs that call internal_type. Addresses
    #740. (Jon Siwek)

  * Adding back the stats.bro file. (Seth Hall)


2.0-beta-188 | 2012-01-10 09:49:29 -0800

  * Change SFTP/SCP log rotators to use 4-digit year in filenames
    Fixes #745. (Jon Siwek)

  * Adding back the stats.bro file. Addresses #656. (Seth Hall)

2.0-beta-185 | 2012-01-09 18:00:50 -0800

  * Tweaks for OpenBSD support. (Jon Siwek)

2.0-beta-181 | 2012-01-08 20:49:04 -0800

  * Add SFTP log postprocessor that transfers logs to remote hosts.
    Addresses #737. (Jon Siwek)

  * Add FAQ entry about disabling NIC offloading features. (Jon Siwek)

  * Add a file NEWS with release notes. (Robin Sommer)

2.0-beta-177 | 2012-01-05 15:01:07 -0800

  * Replace the --snaplen/-l command line option with a
    scripting-layer option called "snaplen" (which can also be
    redefined on the command line, e.g. `bro -i eth0 snaplen=65535`).

  * Reduce snaplen default from 65535 to old default of 8192. Fixes
    #720. (Jon Siwek)

2.0-beta-174 | 2012-01-04 12:47:10 -0800

  * SSL improvements. (Seth Hall)

    - Added the ssl_session_ticket_handshake event back.

    - Fixed a few bugs.

    - Removed the SSLv2.cc file since it's not used.

2.0-beta-169 | 2012-01-04 12:44:39 -0800

  * Tuning the pretty-printed alarm mails, which now include the
    covered time range into the subject. (Robin Sommer)

  * Adding top-level "test" target to Makefile. (Robin Sommer)

  * Adding SWIG as dependency to INSTALL. (Robin Sommer)

2.0-beta-155 | 2012-01-03 15:42:32 -0800

  * Remove dead code related to record type inheritance. (Jon Siwek)

2.0-beta-152 | 2012-01-03 14:51:34 -0800

  * Notices now record the transport-layer protocol. (Johanna Amann)

2.0-beta-150 | 2012-01-03 14:42:45 -0800

  * CMake 2.6 top-level 'install' target compat. Fixes #729. (Jon Siwek)

  * Minor fixes to test process. Addresses #298.

  * Increase timeout interval of communication-related btests. (Jon Siwek)

2.0-beta-145 | 2011-12-19 11:37:15 -0800

  * Empty fields are now logged as "(empty)" by default. (Robin
    Sommer)

  * In log headers, only escape information when necessary. (Robin
    Sommer)

2.0-beta-139 | 2011-12-19 07:06:29 -0800

  * The hostname notice email extension works now, plus a general
    mechanism for adding delayed information to notices. (Seth Hall)

  * Fix &default fields in records not being initialized in coerced
    assignments. Addresses #722. (Jon Siwek)

  * Make log headers include the type of data stored inside a set or
    vector ("vector[string]"). (Johanna Amann)

2.0-beta-126 | 2011-12-18 15:18:05 -0800

  * DNS updates.  (Seth Hall)

    - Fixed some bugs with capturing data in the base DNS script.

    - Answers and TTLs are now vectors.

    - A warning that was being generated (dns_reply_seen_after_done)
      from transaction ID reuse is fixed.

  * SSL updates. (Seth Hall)

    - Added is_orig fields to the SSL events and adapted script.

    - Added a field named last_alert to the SSL log.

    - The x509_certificate function has an is_orig field now instead
      of is_server and its position in the argument list has moved.

    - A bit of reorganization and cleanup in the core analyzer. (Seth
      Hall)

2.0-beta-121 | 2011-12-18 15:10:15 -0800

  * Enable warnings for malformed Broxygen xref roles. (Jon Siwek)

  * Fix Broxygen confusing scoped IDs at start of line as function
    parameter. (Jon Siwek)

  * Allow Broxygen markup "##<" for more general use. (Jon Siwek)

2.0-beta-116 | 2011-12-16 02:38:27 -0800

  * Cleanup some misc Broxygen css/js stuff. (Jon Siwek)

  * Add search box to Broxygen docs. Fixes #726. (Jon Siwek)

  * Fixed major bug with cluster synchronization, which was not
    working. (Seth Hall)

  * Fix missing action in notice policy for looking up GeoIP data.
    (Jon Siwek)

  * Better persistent state configuration warning messages (fixes
    #433). (Jon Siwek)

  * Renaming HTTP::SQL_Injection_Attack_Against to
    HTTP::SQL_Injection_Victim. (Seth Hall).

  * Fixed DPD signatures for IRC. Fixes #311. (Seth Hall)

  * Removing Off_Port_Protocol_Found notice. (Seth Hall)

  * Teach Broxygen to more generally reference attribute values by name. (Jon Siwek)

  * SSH::Interesting_Hostname_Login cleanup.  Fixes #664. (Seth Hall)

  * Fixed bug that was causing the malware hash registry script to
    break. (Seth Hall)

  * Remove remnant of libmagic optionality. (Jon Siwek)

2.0-beta-98 | 2011-12-07 08:12:08 -0800

  * Adapting test-suite's diff-all so that it expands globs in both
    current and baseline directory. Closes #677. (Robin Sommer)

2.0-beta-97 | 2011-12-06 11:49:29 -0800

  * Omit loading local-<node>.bro scripts from base cluster framework.
    Addresses #663 (Jon Siwek)

2.0-beta-94 | 2011-12-03 15:57:19 -0800

  * Adapting attribute serialization when talking to Broccoli. (Robin
    Sommer)

2.0-beta-92 | 2011-12-03 15:56:03 -0800

  * Changes to Broxygen master script package index.  (Jon Siwek)

    - Now only lists packages as those directories in the script hierarchy
      that contain an __load__.bro file.

    - Script packages (dirs with a __load__.bro file), can now include
      a README (in reST format) that will automatically be appended
      under the link to a specific package in the master package
      index.

2.0-beta-88 | 2011-12-02 17:00:58 -0800

  * Teach LogWriterAscii to use BRO_LOG_SUFFIX environemt variable.
    Addresses #704. (Jon Siwek)

  * Fix double-free of DNS_Mgr_Request object. Addresses #661.

  * Add a remote_log_peer event which comes with an event_peer record
    parameter. Addresses #493. (Jon Siwek)

  * Remove example redef of SMTP::entity_excerpt_len from local.bro.
    Fixes error emitted when loading local.bro in bare mode. (Jon
    Siwek)

  * Add missing doc targets to top Makefile; remove old doc/Makefile.
    Fixes #705. (Jon Siwek)

  * Turn some globals into constants. Addresses #633. (Seth Hall)

  * Rearrange packet filter and DPD documentation. (Jon Siwek)

2.0-beta-72 | 2011-11-30 20:16:09 -0800

  * Fine-tuning the Sphinx layout to better match www. (Jon Siwek and
    Robin Sommer)

2.0-beta-69 | 2011-11-29 16:55:31 -0800

  * Fixing ASCII logger to escape the unset-field place holder if
    written out literally. (Robin Sommer)

2.0-beta-68 | 2011-11-29 15:23:12 -0800

  * Lots of documentation polishing. (Jon Siwek)

  * Teach Broxygen the ".. bro:see::" directive. (Jon Siwek)

  * Teach Broxygen :bro:see: role for referencing any identifier in
    the Bro domain. (Jon Siwek)

  * Teach Broxygen to generate an index of Bro notices. (Jon Siwek)

  * Fix order of include directories. (Jon Siwek)

  * Catch if logged vectors do not contain only atomic types.
    (Johanna Amann)

2.0-beta-47 | 2011-11-16 08:24:33 -0800

  * Catch if logged sets do not contain only atomic types. (Johanna
    Amann)

  * Promote libz and libmagic to required dependencies. (Jon Siwek)

  * Fix parallel make from top-level to work on more platforms. (Jon
    Siwek)

  * Add decode_base64_custom(). Addresses #670 (Jon Siwek)

  * A bunch of Sphinx-doc reorgs and polishing. (Jon Siwek)

2.0-beta-28 | 2011-11-14 20:09:28 -0800

  * Binary packaging script tweaks. We now require CMake 2.8.6. (Jon Siwek)

  * More default "weird" tuning for the "SYN_with_data" notice. (Seth
    Hall)

  * Tiny bugfix for http file extraction along with test. (Seth Hall)

2.0-beta-21 | 2011-11-06 19:27:22 -0800

  * Quickstart doc fixes. (Jon Siwek)

2.0-beta-19 | 2011-11-03 17:41:00 -0700

  * Fixing packet filter test. (Robin Sommer)

2.0-beta-12 | 2011-11-03 15:21:08 -0700

  * No longer write to the PacketFilter::LOG stream if not reading
    traffic. (Seth Hall)

2.0-beta-10 | 2011-11-03 15:17:08 -0700

  * Notice framework documentation update. (Seth Hall)

  * Fixing compiler warnings (addresses #388) (Jon Siwek)

2.0-beta | 2011-10-27 17:46:28 -0700

  * Preliminary fix for SSH login detection: we need a counted measure
    of payload bytes (not ack tracking and not with the IP header
    which is what we have now). (Seth Hall)

  * Fixing send_id() problem. We no longer update &redef functions.
    Updating code on the fly isn't fully supported. (Robin Sommer)

  * Tuning the format of the pretty-printed alarm summaries. (Robin
    Sommer)

1.6-dev-1508 | 2011-10-26 17:24:50 -0700

  * Updating submodule(s). (Robin Sommer)

1.6-dev-1507 | 2011-10-26 15:10:18 -0700

  * Baseline updates. (Robin Sommer)

1.6-dev-1506 | 2011-10-26 14:48:43 -0700

  * Updating submodule(s). (Robin Sommer)

1.6-dev-1505 | 2011-10-26 14:43:58 -0700

  * A new base script that pretty-prints alarms in the regular
    summary. (Robin Sommer)

  * Adding a dummy log writer WRITER_NONE that just discards
    everything. (Robin Sommer)

1.6-dev-1498 | 2011-10-26 14:30:15 -0700

  * Adding instructions to local.bro how to do ACTION_ALARM by
  default. (Seth Hall)

1.6-dev-1495 | 2011-10-26 10:15:58 -0500

  * Updated unit test baselines. (Seth Hall)

1.6-dev-1491 | 2011-10-25 20:22:56 -0700

  * Updating submodule(s). (Robin Sommer)

1.6-dev-1482 | 2011-10-25 19:08:32 -0700

  * Fixing bug in log managers predicate evaluation. (Robin Sommer)

1.6-dev-1481 | 2011-10-25 18:17:03 -0700

  * Fix a problem with DNS servers being logged that aren't actually
    servers. (Seth Hall)

  * Changed generated root cert DN format for RFC2253 compliance. (Jon
    Siwek)

  * Removed :bro doc directives from notice documentation. (Seth Hall)

  * New notice framework docs. (Seth Hall)

  * Adding sub messages to emails. (Seth Hall)

  * Adding extra fields to smtp and http to track transaction depth.
    (Seth Hall)

  * Fix for SSH login detection heuristic. (Seth Hall)

  * Removed some fields from http analysis that weren't commonly
    needed or were wrong. (Seth Hall)

  * Updated/fixed MSIE version parsing in the software framework.
    (Seth Hall)

  * Update Mozilla trust roots to index certs by subject distinguished
    name. (Jon Siwek)

  * weird.bro rewrite. (Seth Hall)

  * More notice email tuning. (Seth Hall)

  * Slightly restructured http file hashing to fix a bug. (Seth Hall)

  * Changed the notice name for interesting ssh logins to correctly
    reflect semantics of the notice.  (Seth Hall)

  * Field name change to notice framwork.  $result -> $action

    - $result is renamed to $action to reflect changes to the notice
      framework since there is already another result-like field
      ($suppress_for) and there may be more in the future.

    - Slipped in a change to add connection information to notice
      emails too. (Seth Hall)

  * Small script refinements and documentation updates. (Seth Hall)

  * Pass over upgrade guide. (Robin Sommer)


1.6-dev-1430 | 2011-10-21 10:39:09 -0700

  * Fixing crash with unknown debug streams. Closes #643. (Robin
    Sommer)

  * Code to better handle interpreter errors, which can now be turned
    into non-fatal runtime errors rather than immediate aborts. (Robin
    Sommer).

  * Remove old make-src-packages script. (Jon Siwek)

  * Fixing a bunch of format strings. Closes #567. (Robin Sommer)

  * Cleaning up some distribution files. (Robin Sommer)

  * Various test, doc, and installation fixes/tweaks. (Seth Hall, Jon
    Siwek and Robin Sommer).

  * Varios smaller policy fixes and tweaks (Seth Hall).

  * Moving docs from web server into distribution. (Robin Sommer)

  * Fixing more (small) memory leaks. (Robin Sommer)

  * Profiling support for DNS_Mgr and triggers. With
    misc/profiling.bro, both now report a line in prof.log with some
    counters on usage. (Robin Sommer)

  * Fixing DNS memory leaks. Closes #534. (Robin Sommer)

  * Fix code for disabling analyzers. Closes #577. (Robin Sommer)

  * Changed communication option from listen_encrypted to listen_ssl.
    (Seth Hall)

  * Modification to the Communication framework API. (Seth Hall)

    - Simplified the communication API and made it easier to change
      to encrypted connections by not having separate variables to
      define encrypted and unencrypted ports.

    - Now, to enable listening without configuring nodes just
      load the frameworks/communication/listen script.

    - If encrypted listening is desired set the following:
    	redef Communication::listen_encrypted=T;

  * Connection compressor now disabled by default. Addresses #559.
    (Robin Sommer)


1.6-dev-1372 | 2011-10-06 18:09:17 -0700

  * Filtering some potentially high-volume DNS weirds. (Robin Sommer)

  * DNS now raises DPD events. Closes #577. (Robin Sommer)

  * Fixing a bunch of compiler warnings. (Robin Sommer)

  * Remote logs are auto-flushed if the last write was longer than a
    second ago. Addresses #498. (Robin Sommer)

  * Fix missing from previous MIME commit. (Robin Sommer)

1.6-dev-1366 | 2011-10-06 17:05:21 -0700

  * Make CompHash computation/recovery for functions deterministic.
    Closes #636. (Jon Siwek)

  * Removing unnecessary @load in local.bro. (Robin Sommer)

  * Optimizing some MIME code. (Robin Sommer)

  * Speed improvements in logging code. (Robin Sommer)

  * Consolidating some node-specific functionality from scripts in
    broctl repo. (Jon Siwek)

  * Another fix the for 1xx script code. (Robin Sommer)

1.6-dev-1352 | 2011-10-05 16:20:51 -0700

  * Fix for optional HTTP::Info status_code. (Jon Siwek)

  * Teaking some external testing scripts. (Jon Siwek)

  * HTTP bug fix reported by Martin Holste. (Seth Hall)

  * More script tuning. (Seth Hall)

    - Moved some of the weird events back to the base/ directory.

    - SSL fixes, updates, and performance optimization.

  * More adjustment to reduce Weird volumes. (Seth Hall)

  * Fixed an error when calculating x509 certificate hashes (reported
    by Martin Holste). (Seth Hall)

  * Clean up to cluster framework to make event handling clearer.
    (Seth Hall)

  * Fixed a bug in the notice framework. (Seth Hall)

  * Bug fix for FTP analysis script. (Seth Hall)

1.6-dev-1333 | 2011-09-29 22:29:51 -0700

  * Fixing a number of memory leaks. (Robin Sommer)

  * Loaded_scripts.log is indented with spaces now and makes more
     sense to look at. (Seth Hall)

  * Teach HTTP parser to derive content length of multipart/byteranges
     bodies. Addresses #488. (Jon Siwek)

  * Change logging of HTTP 1xx responses to occur in their own
    columns. Addresses #411. (Jon Siwek)

  * Fix handling of HTTP 1xx response codes. Addresses #411).

  * Taking advantage of yet another trick to get installed browser
  plugins. (Seth Hall)

    - With the software-browser-plugins script you can watch for Omniture
      advertising servers to grab the list of installed plugins.

    - I reorganized the plugin detection a bit too to abstract it better.

    - Removed the WEB_ prefix from all of the Software::Type HTTP enums.
      They were essentially redundant due to the full name already being
      HTTP::SERVER (for example).

1.6-dev-1316 | 2011-09-28 16:50:05 -0700

  * Unit test cleanup. Updated README and collected coverage-related
    tests in a common dir. (Jon Siwek)

  * Fixes for known-services. (Seth Hall)

  * Ported and 2.0ized the capture-loss script. (Seth Hall)

  * Communication fix and extension.(Robin Sommer)

    - Removing unnecessary log flushing. Closes #498.

    - Adding new BiF disconnect() that shuts a connection to a peer down.

    - terminate_connection() now first flushes any still buffered log
      messages.

  * Fix for high SSL memory usage by adding &transient attribute to
    top-level SSL pac array type. Closes #574. (Robin Sommer)

  * Fix a small bug in the metrics framework. (Seth Hall)

  * Temporarily removing scripts that aren't ready to be included.
    Will return before next release.  (Seth Hall)

  * New SSL policy scripts.  (Seth Hall)

    - protocols/ssl/expiring-certs uses time based information from
      certificates to determine if they will expire soon, have already
      expired, or haven't yet become valid.

    - protocols/ssl/extract-certs-pem is a script for taking certs off
      the line and converting them to PEM certificates with the openssl
      command line tool then dumping them to a file.

  * Notice::type_suppression_intervals: table[Notice::Type] of
    interval can be used to modify the suppression intervals for
    entire types of notices. (Seth Hall)

  * EOF SSL protocol violations are only generated a single time now.
    (Seth Hall)

  * Script level fixes.  (Seth Hall)

    - Fixed a type name conflict in the Known namespace.

    - Fixed a DPD framework bug that was causing Reporter messages.

    - Fixed the notice_policy log.

    - Predicate functions are now logged.

    - Predicate functions are now optional.  If not given, it's assumed that
      the result should always apply. (Seth Hall)

    - Fix a problem with accidental and mistaken HTTP log lines.

1.6-dev-1293 | 2011-09-22 19:44:37 -0700

  * Smaller script tweaks. (Seth Hall)

  * Duplicate notice suppression. (Seth Hall)

    - Duplicate notices are discovered with the new Notice::Info
      field $identifier.  It's a string that is left up to the
      notice implementor to define which would indicate a
      fundamentally duplicate notice.  The field is optional and
      if it's not included it's not possible for notice
      suppression to take place.

    - Duplicate notices are suppressed by default for the interval
      defined by the Notice::default_suppression_interval variable
      (1 hour by default).

    - A new notice action was defined ACTION_NO_SUPPRESS to prevent
      suppression for a specific notice instance.  A convenience set
      named not_suppressed_types was also created to not suppress
      entire notice types.

    - A new field was added to the PolicyItem type to modify the length
      of time a notice should be suppressed if the predicate matches.
      The field is named $suppress_for.  This name makes the code more
      readable like this: $suppress_for = 1day

    - New events were created to give visibility into the notice
      framework's suppression activity.
      - event Notice::begin_suppression(n: Notice::Info)
      - event Notice::suppressed(n: Notice::Info)
      - event Notice::end_suppression(n: Notice::Info)

    - The suppression.bro script doesn't have a baseline because
      it is causing a segfault in Bro.  This one test is the
      reason that this is being integrated into a branch instead
      of master. (Seth Hall)

  * Fix crash on exit. Addresses #607. (Jon Siwek)

  * Fix PktSrc setting next_timestamp even when no packet available.
    (Jon Siwek)

  * Fix lack of NUL-termination in to_upper/to_lower BIF's return val.
    (Jon Siwek)

  * Fixing unit tests and some minor bugs. (Jon Siwek)

  * Fix broctl cluster log rotation. Addresses #619. (Jon Siwek)

  * Added session ID to the SSL logging. (Seth Hall)

  * Adding "install-aux" target + updating bro-aux submodule. (Jon
    Siwek)

  * Cleaning up INSTALL and README. (Jon Siwek)

  * Remove $Id$ tags. (Jon Siwek)

  * Remove policy.old directory. Addresses #511. (Jon Siwek)

  * Small rework with ssl base script to reduce memory usage. (Seth
    Hall)

  * Updated the mozilla root certs. (Seth Hall)

1.6-dev-1261 | 2011-09-15 17:13:55 -0700

  * Memory leak fixes. Addresses #574 (Jon Siwek)

  * Add configure options for ruby/bindings integration. (Jon Siwek)

  * Fix filter path_func to allow record argument as a subset of
    stream's columns. Addresses #600. (Jon Siwek)

  * Log rotation is now controlled directly through Filter records. (Jon Siwek)

  * Fix indexing for record types with optional fields. Addresses #378
    (Jon Siwek)

1.6-dev-1248 | 2011-09-15 16:01:32 -0700

  * Removed custom malloc() implementation for FreeBSD. Closes #557.
    (Jon Siwek)

  * Testing/external scripts no longer compute MD5 checksums for SMTP
    entities. (Robin Sommer)

  * External tests no longer include the full content of mismatching
    files in the diagnostics output. (Robin Sommer)

1.6-dev-1241 | 2011-09-14 22:51:52 -0400

  * Fixing a major memory utilization issues with SSL analysis. (Seth
    Hall)

  * Enhancements to HTTP analysis: (Seth Hall)

      - More options for the header-names.bro script.

      - New script for logging header names and values. Closes #519.
        (Seth Hall)

      - HTTP body size measurement added to http.log.

      - The value of the content-length headers has now been removed
        in the default output but it could be added back locally at an
        installation by a user.

      - Added fields to indicate if some parsing interruption happened
        during the body transfer. Closes #581 (Seth Hall)

  * Misc smaller usability and correctness updates: (Seth Hall)

    - Removed an notice definition from the base SSL scripts.

    - Moved a logging stream ID into the export section for known-services
      and bumped priority for creating the stream.

    - Adding configuration knobs for the SQL injection attack detection
      script and renaming the HTTP::SQL_Injection_Attack notice to
      HTTP::SQL_Injection_Attack_Against

    - Bumped priority when creating Known::CERTS_LOG.

    - Fixing a warning from the cluster framework. (Seth Hall)

  * Bugfix for log writer, which didn't escape binary stuff in some
    situations. Closes #585. (Robin Sommer)

  * A larget set of changes to the testing/external infrastructure.
    The traces for external test-suites are no longer kept inside the
    repositories themselves but downloaded separately via curl. This
    is because git is pretty bad at dealing with large files. See the
    README for more information. (Robin Sommer)

1.6-dev-1221 | 2011-09-08 08:41:17 -0700

  * Updates for documentation framework and script docs. (Jon Siwek)

  * The script level PF_RING support isn't working so removing it.
    (Seth Hall)

  * Delete SSL certificates from memory after ssl_established event.
    (Seth Hall)

  * Small fixes for SSL analysis. (Seth Hall)

1.6-dev-1212 | 2011-09-07 16:15:28 -0700

  * Internally, the UID generation can now return values from
    different pool for better reproducability in testing mode.
    (Gilbert Clark).

  * Added new BiF unique_id_from(pool: string, prefix: string) that
    allows the user to specify a randomness pool. (Gilbert Clark)

1.6-dev-1198 | 2011-09-07 11:03:36 -0700

  * Extended header for ASCII log that make it easier for scripts to
    parse Bro log files. (Gilbert Clark)

  * Potential fix for rotation crashes. Addresses #588. (Robin Sommer)

  * Added PF_RING load balancing support to the scripting layer,
    enabled by loading the misc/pf-ring-load-balancing script. (Seth
    Hall)

  * Added a BiF setenv() for setting environment variables. (Seth
    Hall)

1.6-dev-1184 | 2011-09-04 09:34:50 -0700

  * FindPCAP now links against thread library when necessary (e.g.
    PF_RING's libpcap). (Jon Siwek)

  * Install binaries with an RPATH. (Jon Siwek)

  * Fix for a case where nested records weren't coerced even though
    possible. (Jon Siwek)

  * Changed ASCII writer to delay creation of log after rotation until
    next write.

  * Changed default snaplen to 65535 and added a -l/--snaplen command
    line option to set it explicitly. Addresses #447. (Jon Siwek)

  * Various updates to logging framework. (Seth Hall)

  * Changed presentation of enum labels to include namespace. (Jon
    Siwek)

  * HTTP analyzer is now enabled with any of the HTTP events. (Seth
    Hall)

  * Fixed missing format string that caused some segfaults. (Gregor
    Maier)

  * ASCII writer nows prints time interval with 6 decimal places.
    (Gregor Maier)

  * Added a Reporter::fatal BIF. (Jon Siwek)

  * Fixes for GeoIP support. Addresses #538. (Jon Siwek)

  * Fixed excessive memory usage of SSL analyzer on connections with
    gaps. (Gregor Maier)

  * Added a log postprocessing function that can SCP rotated logs to
    remote hosts. (Jon Siwek)

  * Added a BiF for getting the current Bro version string. (Jon
    Siwek)

  * Misc. doc/script/test cleanup. (Jon Siwek)

  * Fixed bare-mode @load dependency problems. (Jon Siwek)

  * Fixed check_for_unused_event_handlers option. (Jon Siwek)

  * Fixing some more bare-mode @load dependency issues (Jon Siwek)

  * Reorganizing btest/policy directory to match new scripts/
    organization. Addresses #545 (Jon Siwek)

  * bro scripts generated from bifs now install to
    $prefix/share/bro/base. Addresses #545 (Jon Siwek)

  * Changeed/fixed some cluster script error reporting. (Jon Siwek)

  * Various script normalization. (Jon Siwek)

  * Add a test that checks each individual script can be loaded in
    bare-mode. Adressess #545. (Jon Siwek)

  * Tune when c$conn is set. Addresses #554. (Gregor Maier)

  * Add ConnSize_Analyzer's fields to conn.log. (Gregor Maier)

  * Fixing bug in "interesting hostnames" detection. (Seth Hall)

  * Adding metrics framework intermediate updates. (Seth Hall)

1.6-dev-1120 | 2011-08-19 19:00:15 -0700

  * Fix for the CompHash fix. (Robin Sommer)

1.6-dev-1118 | 2011-08-18 14:11:55 -0700

  * Fixing key size calculation in composite hash code. (Robin Sommer)

1.6-dev-1116 | 2011-08-18 10:05:07 -0700

  * Remove the 'net' type from Bro (addresses #535).

  * Fix H3 assumption of an 8-bit byte/char. (Jon Siwek)

  * Allow reading from interface without additional script arguments.
    Explicitly passing in '-' as an additional command line argument
    still allows reading a script from stdin. (Jon Siwek)

  * SSH bruteforcing detection now done with metrics framework. (Seth
    Hall)

  * Updates for SQL injection attack detection to match the metrics
    framework updates. (Seth Hall)

  * Metrics framework now works on cluster setups. (Seth Hall)

  * Reclassifying more DNS manager errors as non-fatal errors. (Robin
    Sommer)

  * Fix ConnSize_Analyzer when used in conjunction with connection
    compressor. (Gregor Maier)

  * Fix reporter using part of the actual message as a format string.
    (Jon Siwek)

1.6-dev-1095 | 2011-08-13 11:59:07 -0700

  * A larger number of script documentation updates. Closes #543. (Jon
    Siwek)

  * Workaround for FreeBSD CMake port missing debug flags. (Jon Siwek)

  * piped_exec() can now deal with null bytes. (Seth Hall)

  * Fix vector initialization for lists of records with optional
    types. Closes #485. (Jon Siwek)

  * Fix redef'ing records with &default empty set fields. Closes #460.
    (Jon Siwek)

  * Fix ConnSize_Analyzer when used in conjunction with the connection
    compressor. (Gregor Maier)

  * Fix reporter using part of the actual message as a format string.
    (Jon Siwek)

  * Fixing reporter's location tracking. Closes #492. (Robin Sommer)

  * Turning DNS errors into warnings. Closes #255. (Robin Sommer)

  * Logging's path_func now receives the log record as argument.
    Closes #555. (Robin Sommer)

  * Functions can now be logged; their full body gets recorded.
    Closes #506. (Robin Sommer)

  * Bugfix for hostname notice email extension. (Seth Hall)

  * Updates for notice framework. (Seth Hall)

    - New ACTION_ADD_GEODATA to add geodata to notices in an extension
      field named remote_location.

    - Loading extend-email/hostnames by default now that it only does
      anything when the ACTION_EMAIL action is applied (finally).

  * Updates to local.bro (Seth Hall)

  * Added the profiling script. (Seth Hall)

  * Updates for SSH scripts. (Seth Hall)

  * ConnSize analyzer is turned on by default now. (Seth Hall)

  * Updates for the build system and site local scripts for cluster.
    (Seth Hall)

  * HTTP now uses the extract_filename_from_content_disposition function. (Seth Hall)

  * Major SMTP script refactor. Closes #509. (Jon Siwek and Seth Hall)

  * New variable Site::local_nets_table in utils/site for mapping
    address to defined local subnet.

  * Metrics framework updates, more to come. (Seth Hall)


1.6-dev-1061 | 2011-08-08 18:25:27 -0700

  * A set of new/changed tests regarding the new policy script
    organisation. (Robin Sommer)

1.6-dev-1058 | 2011-08-08 16:15:18 -0700

  * Reorganisation of the scripts that Bro loads by default. (Seth
    Hall)

    - policy/ renamed to scripts/

    - By default BROPATH now contains:
    	- scripts/
    	- scripts/policy
    	- scripts/site

    - The scripts in scripts/base/protocols/ only do logging and state
      building.

    - All of scripts/base/ is loaded by by default. This can however
      be disabled by switching Bro into "bare mode" using the new
      command-line option --bare-mode (or -b). The cripts in
      scripts/base/ don't use relative path loading to ease use of
      bare mode (to copy and paste that script).

    - The scripts in scripts/base/frameworks/ add functionality
      without causing any additional overhead.

    - All "detection" activity happens through scripts in
      scripts/policy/.

    - bro.init was renamed to base/init-bare.bro, and base/all.bro was
      renamed to init-default.bro.

    - local.bro now loads more functionality from policy/ and adds
      more documentation. (Seth Hall)

  * Adding default_path_func() to the logging framework that makes the
    default naming scheme script-level controlled. (Robin Sommer)

  * Reworking logging's postprocessor logic so that postprocessor
    commands are no longer run by the log writers themselves, but
    instead by a script level function. (Robin Sommer)

  * The communication subsystem is now by default off and must be
    enabled explicitly with a new BiF, enable_communication(). Closes
    #540. (Robin Sommer)

  * The hostname notice email extension now only add hostnames for
    emailed noticed. (Seth Hall)

  * Cleaning up doc generation. (Seth Hall)

1.6-dev-1044 | 2011-08-05 19:07:32 -0700

  * Fixing memory (and CPU) leak in log writer.

  * Fixing crash in memory profiling. (Robin Sommer)

  * Fix compiler warning. (Robin Sommer)

  * Fixing missing sync in cluster setup. (Robin Sommer)


1.6-dev-1038 | 2011-08-05 18:25:44 -0700

  * Smaller updates to script docs and their generation. (Jon Siwek)

  * When using a `print` statement to write to a file that has raw output
    enabled, NUL characters in string are no longer interpreted into "\0",
    no newline is appended afterwards, and each argument to `print` is
    written to the file without any additional separation. (Jon Siwek)

  * Test portatibility tweaks. (Jon Siwek)

  * Fixing PktSrc::Statistics() which retured bogus information
    offline mode. Closes #500. (Jon Siwek)

  * --with-perftools configure option now assumes --enable-perftools.
    Closes #527. (Jon Siwek)

1.6-dev-1018 | 2011-07-31 21:30:31 -0700

  * Updating CHANGES. (Robin Sommer)

1.6-dev-1016 | 2011-07-30 18:34:28 -0700

  * Install example config files dynamically. They'll only get
    installed when the distribution version differs from existing
    version on disk. (Jon Siwek)

  * Fixed memory leak in SSL analyzer. (Seth Hall)

  * Beginning rework of metrics interface. (Seth Hall)

  * New/updated unit tests for scripts. (Jon Siwek)

  * New/updated documentstion for scripts. (Jon Siwek)

  * A number of fixes for scripts in utils/. (Jon Siwek)

1.6-dev.244 Thu Jul 28 17:08:21 PDT 2011

- mask_addr() now returns subnet (addresses #512). (Jon Siwek)

- Normalize Notice::Type identifiers per convention (closes #484).
  (Jon Siwek)

- Fixing default-loaded-scripts test for BSD systems. (Jon Siwek)

- New piped_exec() BiF for pipeing data into an external command. (Jon
  Siwek)

1.6-dev.242 Mon Jul 25 21:42:39 PDT 2011

- Adding a documentation coverage test. (Jon Siwek)

- The CMake targets for generating reST docs from policy scripts are
  now automatically generated via the genDocSourcesList.sh script.
  (Jon Siwek)

- Fixed a number of script error. (Jon Siwek)

- Fixes to relative @load'ing.  (Jon Siwek)

- Fixes to tests. (Robin Sommer)

1.6-dev.240 Sun Jul 24 15:14:26 PDT 2011

- Updated tests and test baselines. (Jon Siwek)

- ASCII log writer now prints time values w/ constant 6 digit
  precision. (Jon Siwek)

- Many policy script updates acrsso the board (Seth Hall).

- Moving devel-tools to bro-aux. (Robin Sommer)

- BugFix for disable_analyzer(), which could cause crashes with some
  analyzers. (Robin Sommer)

- Bugfix for potential segfault in DebugLogger. (Robin Sommer)

1.6-dev.226 Thu Jul 21 15:23:39 PDT 2011

- Extensions to the @load and @unload process. (Jon Siwek)

    * Make @load statements recognize relative paths. For example a
      script can do "@load ./foo" to load a script named foo.bro that
      lives in the same directory or "@load ../bar" to load a script
      named bar.bro in the parent directory, even if those directories
      are not contained in BROPATH.

    * Reimplementation of the @prefixes statement. (Closes #486)

      Any added prefixes are now used *after* all input files have
      been parsed to look for a prefixed, flattened version of the
      input file somewhere in BROPATH and, if found, load it. For
      example, if "lcl" is in @prefixes, and site.bro is loaded, then
      a file named "lcl.site.bro" that's in BROPATH would end up being
      automatically loaded as well. Packages work similarly, e.g.
      loading "protocols/http" means a file named
      "lcl.protocols.http.bro" in BROPATH gets loaded automatically.

    * Fix @unload'd files from generating bro_script_loaded event.

    * Updates to tests.

1.6-dev.225 Wed Jul 20 17:10:41 PDT 2011

- IRC improvements (Jon Siwek). Including:

    * Shorten what's displayed in the IRC's log mime_type column for
      DCC transfers.

    * Add IRC unit tests.

    * Fix IRC analyzer supplying wrong type to irc_dcc_message event.

    * Removed irc_client and irc_server events.

    * Added is_orig arguments to all other irc events.

    * Fix analyzer not recognizing Turbo DCC extension message format.

    * Fix analyzer not generating irc_dcc_message event when irc_privmsg_message
      event doesn't have a handler registered.

- Fixing tests that need a diff canonifier. (Jon Siwek)

1.6-dev.223 Tue Jul 19 19:10:36 PDT 2011

- Adding a script to update CHANGES and VERSION. (Robin Sommer)

1.6-dev.218 Tue Jul 19 18:16:44 PDT 2011

- Comprehensive policy script overhaul/rewrite. (Seth Hall)

  Changes are too extensive to list individually.

- Removing undocumented -H command line flag. (Robin Sommer)

- Fixing many tests. (Everybody)

- Fixing 0-chunk bug in remote logging. (Robin Sommer)

- $PATH is now appropriately set by the bro-path-dev.(sh|csh) scripts.
  (Seth Hall)

- Making valgrind a bit more happy. (Robin Sommer)

- New BiF record_field_vals() that returns the fields of a record in a
  table with meta-information. (Robin Sommer)

- Adding a script in aux/devel-tools that extracts a connection from a
  trace based on uid. (Robin Sommer)

- Fixing bug causing crash when running without arguments. (Robin Sommer)

- A new event bro_script_loaded() raised for each policy script
  loaded. Also removing the -l command-line option as that can now be
  done at the script-level. (Robin Sommer)

- Fixing memory leaks. (Gilbert Clark, Seth Hall, Robin Sommer)

- Many SSL analysis improvements and fixes. (Seth Hall)

- Fixing bug with event priorities potentially being ignored for the
  handler. (Robin Sommer)

- Overhauling the internal reporting of messages to the user. The new
  Reporter class is now in charge of reporting all errors, warnings,
  informational messages, weirds, and syslogs; and it passes
  everything through the script layer. (Robin Sommer)

* Removed the alarm statement and the alarm_hook event. (Robin Sommer)

- Adding new policy file test-all.bro that loads all other policies.
  This is for testing only. (Robin Sommer)

- A new framework for doing regression testing with larger traces and
  more complex Bro configurations in testing/external. (Robin Sommer)

- Many updates to script doc generation.  (Jon Siwek)

1.6-dev.146 Sat Jun 25 18:12:27 PDT 2011

- DNS mapping are now becoming invalid when an entry's TTL expires.
  (Thomas Other)

- Reworking how Bro tracks which scripts are already loaded. Rather
  than paths, Bro now tracks inode numbers. (Jon Siwek)

- New BiF netstats() to query packet capture statistics. The netstats
  script now uses the new BiF to periocally report packets drops. The
  net_stats_update() event and the heartbeat_interval global went
  away. (Seth Hall)

- Fixing bug with logging &optional records. Closes #476. (Robin
  Sommer)

- Fixing istate.events-ssl test failing because of expired cert. (Jon
  Siwek)

- A large number of improvements and fixes for Bro's doc mode. (Jon
  Siwek)

- Significant updates for RPC and NFS analyzers (Gregor Maier)

    * Unify semantics for UDP and TCP connections.

    * RPC can now log to a log file if desired.

    * Portmapper can now log general activity to a log file and also log
      actual port mappings.

    * NFS analyzer now supports significantly more procedure calls as
      as file name tracking and file content extraction.

- NetBIOS fixes. (Jon Siwek)

- A number of unit tests are more robust and portable.  (Jon Siwek)

- A new BiF unique_id() that returns a string that's unique across Bro
  instaces with high probablity.  (Robin Sommer)

- Complete rewrite of the BinPAC SSL analyzer. (Seth Hall)

    * DER certificates are extracted as strings to be used with
      corresponding BiFs.

    * x509_verify function to verify single certs and/or full
      certificate chains.

    * Removed hand written SSL analyzer.

    * The ssl.bro script is just a place-holder for now. New version
      will come with the other new scripts.

- New syslog analyzer. (Seth Hall)

- @load now supports loading a directory. With a directory "foo"
  somewhere in BROPATH, "@load foo" now checks if there's a file
  "foo/__load__.bro". If so, it reads that file in. (Robin Sommer)

- ASCII logger now escapes non-printable characters. Closes #450.
  (Robin Sommer)

- Packaging tweaks and rewrite of 'dist' target. (Jon Siwek)

- Changes to allow DEB packaging via CPack, addresses #458. (Jon
  Siwek)

- An extension to the ICMP analyzer to handle redirects. Julien
  Sentier

- Removing old istate test-suite. (Robin Sommer)

- A hack to report missing GeoIP support only once. This closes #357,
  but #455 captures the need for a more general solution. (Robin
  Sommer)

- Bugfix: vectors in records were not initalized. Closes #421. (Robin
  Sommer)

- If IPv6 default is not compiled in, the default BPF filters now
  excludes IPv6 packets. (Robin Sommer)

- New bif bro_has_ipv6() to check whether IPv6 support is compiled in.
  (Robin Sommer)

- Updating btests and a Makefile. "make" now runs all the tests.
  (Robin Sommer)

- Moving the test-scripts from the old test-suite over to btest.
  (Robin Sommer)

- Fix for major bug in POP3 analyzer, which didn't recognize '.'
  terminators in multi-line replies if the terminator was bare (no
  newline). This caused it to ignore the rest of the session that it's
  analyzing. (Vern Paxson)

- Fix compiler warning with gcc-4.4.4 (Gregor Maier)

- Adding example documentation for a script's use of logging features.
  (Jon Siwek)

- Adding &log attribute to static attr_names array. (Jon Siwek)

- Bro can now track packet and byte counts per connection. (Gregor
  Maier)

    * If 'use_conn_size_analyzer' is true, the event engine tracks
      number of packets and raw IP bytes per connection. If
      report_conn_size_analyzer is true, these values are included as
      four new columns into conn.log

    * I changed conn.bro so that the value of
      report_conn_size_analyzer follows that of
      use_conn_size_analyzer. For the new conn.log, we probably want
      to get rid of report_conn_size_analyzer anyway.

- Fixing numerous compiler warnings and portability issues. (All)

- Switching vectors from being 1-based to 0-based. Note that this is a
  change that break backwards-compatibility. (Robin Sommer)

- Increasing serialization format version for the recent 64-bit
  changes. (Robin Sommer)

- Support for (mixed) MPLS and VLAN traffic, and a new default BPF
  filter. (Seth Hall and Robin Sommer)

    * Merging in the patch from #264, which provides support for mixed
      VLAN and MPLS traffic.

    * Changing Bro's default filter from being built dynamically to
      being a static "ip or not ip". To get the old behaviour back
      (i.e., the dynamically built filter), redef "all_packets" to
      false.

    * print-filter.bro now always prints the filter that Bro is
      actually using, even if overriden from the command line. (Robin
      Sommer)

- Changing the HTTP's analyzers internals to use 64-bit integers.
  (Gregor Maier).

- Fixing bug with deleting still unset record fields of table type.
  (Robin Sommer)

1.6-dev.99 Fri Apr 22 22:10:03 PDT 2011

- Extending the connection record with a unique identifier. (Robin
  Sommer)

    type connection: record {
        [...]
        id: string;
   };

  These identifiers very likely unique even across independent Bro
  runs.

- Delete operator for record fields. (Robin Sommer)

  "delete x$y" now resets record field "x" back to its original state
  if it is either &optional or has a &default. "delete" may not be
  used with non-optional/default fields.

- Fixing bug with nested record coercions. (Robin Sommer)

- Fixing a do_split() bug. (Seth Hall)


1.6-dev.94 Thu Apr 21 19:51:38 PDT 2011

- Fixing generation of config.h. (Jon Siwek)

- Updates and tests for NetBIOS name BiF. (Seth Hall)

- Fixing do_split bug(), and adding a test. (Seth Hall)

- When Bro is given a PRNG seed, it now uses its own internal random
  number generator that produces consistent results across sytems.
  Note that this internal generator isn't very good, so it should only
  be used for testing purpses. (Robin Sommer)

- The BTest configuration now sets the environemnt variables TZ=UTC
  and LANG=C to ensure consistent results. (Robin Sommer)

- Logging fixes. (Robin Sommer)

1.6-dev.88 Wed Apr 20 20:43:48 PDT 2011

- Implementation of Bro's new logging framework. We will document this
  separately. (Robin Sommer)

- Already defined record types can now be further extended via the
  '+=' operator. The added fields must be either &optional or have a
  &default value. (Robin Sommer)

  Example:

        type Foo: record {
            a: count;
            b: count &optional;
        };

        redef record Foo += {
            c: count &default=42;
            d: count &optional;
        };

        global f: Foo = [$a=21];

        print f;

  Output:

        [a=21, b=<uninitialized>, c=42, d=<uninitialized>]

- Enabling assignment of empty vectors ("vector()"). (Robin Sommer)

- Fixing attributes to allow &default attributes to be associated with
  records fields of type tables/sets/vector. (Robin Sommer)

- '[]' is now a valid record constructor. (Robin Sommer)

- A instance of a record type A is now coercable into one of type B if
  the fields of type A are a subset of those of type B. (Robin Sommer)

- A number of bug fixes and enhancements for record/set/table/vector
  coercion. (Robin Sommer)

- Fixing a problem with records that have optional fields when used as
  table/set indices. Addresses #367. (Robin Sommer)

- Fixing an off-by-one error in join_string_vec(). (Seth Hall)

- Updating to_count() to cope with 64bit ints. (Seth Hall)

- A new BiF count_to_v4_addr() to turn a count into an IPv4 address.
  (Seth Hall)

1.6-dev.80 Mon Apr 18 14:50:54 PDT 2011

- New framework for generating documentation from Bro scripts. (Jon
  Siwek)

  This includes:

    * Changes to Bro's scanner/parser to facilitate automatic
      generation of Bro policy script documentation in
      reStructuredText format.

    * New command line flags -Z/--doc-scripts to enable the new doc
      generation mode.

    * Changes to bifcl to pass comments starting with "##" through
      into the generated .bro script.

    * A "doc" build target for the top-level Makefile to first
      generate reStructuredText for a defined set of Bro policy
      scripts, and then run that through Sphinx to create HTML
      documentation.

1.6-dev.78 Mon Apr 18 12:52:55 PDT 2011

- Adding files to CMake build targets so they show up in generated IDE
  projects. This addresses #413. (Jon Siwek)

- Fix unnecessary config.h preprocessor (re)definitions. This
  addresses #414. (Jon Siwek)

- Updating istate tests. (Robin Sommer)

- Adding files to CMake build targets so they show up in generated IDE
  projects.

- Adding new environment variable BRO_SEED_FILE to set the seed file
  for the random number generator. (Robin Sommer)

1.6-dev.71 Fri Apr  1 16:06:33 PDT 2011

- Removing code for the following no longer supported functionality.

    * Trace rewriting.
    * DFA state expiration in regexp engine.
    * Active mapping.
    * Unused hash functions.

  (Robin Sommer)

- Fixing crashes when SSL is not configured correctly. (Robin Sommer)

1.6-dev.66 Tue Mar 29 21:52:01 PDT 2011

- Initial btest setup (Don Appleman and Robin Sommer)

- Porting the istate tests to btest (not finished) (Robin Sommer)

1.6-dev.63 Mon Mar 21 16:31:15 PDT 2011

- Changes to the way user-modifiable config files are installed  (Jon Siwek)

    * Duplicates of the distribution's configuration files are now
      always installed with a .example suffix

    * Added --binary-package configure option to toggle configure
      logic specific to the creation of binary packages.

    * When not in binary packaging mode, `make install` never
      overwrites existing configure files in case they've been
      modified. The previous behavior (CMake's default) would only
      avoid overwriting modified files if one consistently uses the
      same build directory and doesn't reconfigure.

- Fixed an issue with Mac package's pre-install script not preserving
  ACLs. (Jon Siwek)

- Minor cleanup/refactor of the make-mac/rpm-packages scripts. (Jon
  Siwek)

- Add explicit CMake check for compiler. (Jon Siwek)

- Add alternative way to set BROPATH for running bro from build/ dir.
  (Jon Siwek)

- Fixing compiler warnings (Gregor Maier)

- Remvoing leftover local variables that caused compile error on Mac
  OS X. (Gregor Maier)

1.6-dev.53 Fri Feb 25 17:03:05 PST 2011

- Fixing file detector leak in remote communication module. (Scott
  Campbell)

- Updating independent-state tests to work with new setup. (Robin
  Sommer)

1.6-dev.49 Fri Feb 25 15:37:28 PST 2011

- Enum IDs can have explicitly defined values. (Gregor Maier)

- Extensions for the built-in function compiler, bifcl. (Gregor Maier)

    * Support for policy-layer namespaces.
    * Support for type declarations in bif files (with access them
      from C++)
    * Extended const declarations in bif files.

  See http://bro.icir.org/devel/bif-doc for more information.

1.6-dev.48 Fri Feb 25 10:53:04 PST 2011

- Preliminary TCP Reassembler fix: deliver data after 2GB by disabling
  the unused seq_to_skip feature. (Gregor Maier)

1.6-dev.47 Fri Feb 25 10:40:22 PST 2011

- Fixing endianess error in XDR when data is not 4-byte aligned.
  (Gregor Maier)

- Fix for Val constructor with new int64 typedefs. (Gregor Maier)

- Updated fix for OS X 10.5 compile error wrt llabs(). (Gregor Maier)

- Fix more compiler warning wrt printf format strings.  (Gregor Maier)

1.6-dev.45 Tue Feb  8 21:28:01 PST 2011

- Fixing a number of compiler warnings. (Seth Hall and Robin Sommer)

1.6-dev.44 Tue Feb  8 20:11:44 PST 2011

- A number of updates to the SSL analyzer, including support for new
  ciphers; SSL extensions; and bug fixes. The analyzer does not longer
  throw weird for exceeding a predefined cipherspec_size anymore.
  (Seth Hall and Rmkml).

- The various split*() BiFs now handle strings containing null bytes
  correctly. (Seth Hall)

- Adding new aux/btest submodule. This is a framework we will use in
  the future for doing unit tests. (Robin Sommer)

1.6-dev.41 Mon Feb  7 13:43:56 PST 2011

- Smarter way to increase the parent/child pipe's socket buffer.
  (Craig Leres).

- Fixing bug with defining bro_int_t and bro_uint_t to be 64 bits wide
  on some platforms. (Robin Sommer)

1.6-dev.39 Mon Jan 31 16:42:23 PST 2011

- Login's confused messages now go through weird.bro. (Robin Sommer)

1.6-dev.36 Mon Jan 31 08:45:35 PST 2011

- Adding more configure options for finding dependencies, (Jon Siwek)

    --with-flex=PATH       path to flex executable
    --with-bison=PATH      path to bison executable
    --with-perl=PATH       path to perl executable
    --with-python=PATH     path to Python interpreter
    --with-python-lib=PATH path to libpython
    --with-python-inc=PATH path to Python headers
    --with-swig=PATH       path to SWIG executable

- Fixing typo in PCAPTests.cmake  (Jon Siwek)


1.6-dev.33 Mon Jan 24 15:29:04 PST 2011

- Fixing bug in SMB analyzer. (Robin Sommer)

- Configure wrapper now deletes previous CMake cache (Jon Siwek)

- Fix for the --with-binpac configure option. (Jon Siwek)

1.6-dev.30 Thu Jan 20 16:32:43 PST 2011

- Changed configure wrapper to create config.status. (Jon Siwek)

1.6-dev.29 Thu Jan 20 16:29:56 PST 2011

- Fixing little problem with initialization of Bro-to-Bro event
  communication. (Christian Kreibich)


1.6-dev.27 Thu Jan 20 13:52:25 PST 2011

- Fine-tuning of the HTTP analyzer in terms of raising protocol
  violations and interrupted transfers. (Gregor Maier)


1.6-dev.21 Wed Jan 19 17:36:02 PST 2011

- Added 4 new BiFs and a new record type for testing the entropy of
  strings. (Seth Hall)

    find_entropy(data: string): entropy_test_result
        This is a one shot function that accepts a string and
        returns the result of the entropy calculations.

    entropy_test_init(index: any): bool
        This and the next two functions are for calculating entropy
        piece-wise. It only needs an index which can be any type of
        variable. It needs to be something that uniquely identifies
        the data stream that is currently having it's entropy
        calculated.

    entropy_test_add(index: any, data: string): bool
        This function is used to add data into the entropy
        calculation. It takes the index used in the function above
        and the data that you are adding and returns true if
        everything seemed to work, false otherwise.

     entropy_test_finish(index: any): entropy_test_result
        Calling this function indicates that all of the desired data
        has been inserted into the entropy_test_add function and the
        entropy should be calculated. This function *must* be called
        in order to clean up an internal state tracking variable.
        If this is never called on an index, it will result in a
        memory leak.

  The entropy_test_result values have several measures of the
  entropy, but a good one to work with is the "entropy" attribute.
  It's a double and as the value approaches 8.0 it can be considered
  more and more random.  For example, a value of 7.832 would be
  quite random but a value of 4.671 is not very random.

1.6-dev.20 Wed Jan 19 17:30:11 PST 2011

- BRO_DNS_FAKE is now listed in the --help output. (Seth Hall)


1.6-dev.18 Wed Jan 19 16:37:13 PST 2011

- Removing unnecessary expire timer from http_sessions. (Gregor
  Maier)


1.6-dev.16 Sat Jan 15 14:14:21 PST 2011

- Updates to the build system. (Jonathan Siwek)

    * ``make dist`` is now available to be used with the top-level
      Makefile for creating source packages according to #344.

    * ``make-rpm-packages`` and ``make-mac-packages`` scripts can
      now generate binary packages according to #295.

    * Additional configure options to change packaging behavior.

    * OS X builds will now prefer to link static libraries of
      optional dependencies that don't come with the vanilla
      operating system.

    * Fix for OS X 10.5 compile error dealing with the llabs()
      function from stdlib.

    * Installing as a different user than the one that
      configured/built now works (although, a harmless error message
      about not being able to write the install manifest may occur).


1.6-dev.3 Wed Dec  8 04:09:38 PST 2010

- Merge with Subversion repository as of r7137. Incorporated change:

    * Fix for packet processing resumption when a remote Bro dies
      during state synchronization (Robin Sommer).

1.6-dev.2 Wed Dec  8 03:57:03 PST 2010

- Compatibility fix for OpenSSL 1.0.0 (Christian Kreibich, Gregor
  Maier).

1.6-dev.1 Sat Nov 27 12:19:47 PST 2010

- Merge with Subversion repository as of r7098. Incorporated changes:

    * Rotation post-processors are now passed an additional argument
      indicating whether Bro is terminating (Robin Sommer).

    * Bro now consistently generates a file_opened event for all
      fopen() calls. (Robin Sommer).

    * You can now redefine the email_notice_to function (Robin
      Sommer).

1.6-dev.0 Fri Nov 26 13:48:11 PST 2010

- The Bro source code is now developed in the new git repositories.
  See the developer pages at http://www.bro-ids.org for more
  information on the new development process.

- Bro's build and installation setup has been moved from GNU
  autotools to CMake. As a result of that, layout and specifics of
  the distribution has changed significantly.

- Lots of pieces have been removed from the distribution that are
  either now unnecessary or are no longer maintained.

- As part of the cleanup, a numbef of Bro configure options and
  their corresponding functionality have been removed, including:

    * --disable-select-loop
    * --with-dag
    * --disable-nbdns
    * --enable-activemapping
    * --enable-activemapping
    * --enable-shippedpcap

- The previous configure option --enable-int64 is now enabled by default,
  and can no longer be disabled.

- ClamAV support has been removed, which has been non-functional for
  a while already.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

1.5.3 Thu Mar  3 08:55:11 PST 2011

- Removing aux/broctl/policy/cluster-addrs.hot.bro from the
  distribution. The script is no longer needed and could in fact break
  an installation because it redefines an old variable that has went
  away. (Robin Sommer)

- Smarter way to increase the communication module's pipe's socket
  buffer size, resulting in a value closer to the allowed maximum.
  (Craig Leres)

- BroControl now also maintains links from the log archive to the
  current set of logs when running in standalone mode. (Robin Sommer)

- Bug fix for a file descriptor leak in the remote communication
  module. (Scott Campbell)

- Bug fix for BroControl to now activate trace-summary's sampling in
  cluster mode, but not anymore in standalone mode. (Robin Sommer)

- Broccoli updates:

    * Accept empty strings ("") as values in the configuration file.
      (Craig Leres)

    * Support for specifying a separate host key for SSL-enabled
      operation, with documentation update. (Craig Leres)

1.5.2 Wed Jan 12 17:34:55 PST 2011

- Portability fixes for --enable-int64 (Vern Paxson).

- Bug fix for Active Mapping support (Kevin Lo).

- Broccoli compiler warning fixes (Kevin Lo).

- Bug fixes for --enable-int64 and for avoiding bogus statistics /
  bad memory references when generating profiling information upon
  exit (Vern Paxson).

- Bug fixes for terminating connections (Tyler Schoenke and Vern Paxson).

- Removed now-quite-stale SSHv1 overflow detection, as it's more prone
  to false positives than useful detection (Vern Paxson).

- The SWIG file now explicitly lists those pieces from broccoli.h which it
  wants to wrap, rather than just including all of broccoli.h (Robin Sommer).

- http-header.bro now includes a global "include_header: set[string]" If it
  contains any strings, then only those headers will be processed.  If left
  empty, then you continue to get the current behavior of processing all
  headers. (Robin Sommer).

- Several changes to drop.bro (Robin Sommer):

	* If True, the new flag Drop::dont_drop_locals indicates that
	  local hosts should never be dropped.  On by default.

	* If True, the new flag Drop::debugging activates extensive debugging
	  output for the catch-and-release logic.  Off by default.

	* The timeout for tracking dropping information is now 1 day
	  rather than 7 days, to better echo the one-restart-a-day semantics
	  used in the past.

	* Bug fix for hosts once dropped by later cleared; some state
	  for them persisted.

- Portability fix for Broccoli Python bindings on 64-bit platforms (Robin
  Sommer).

- The HTTP analyzer no longer attempts to track Server/User-Agent
  versions, as these are hugely voluminous (Seth Hall).

- HTTP and SMTP no longer have extra-short inactivity timeouts, as
  these were too often leading to premature expiration of a connection
  (Robin Sommer).

- Tracking of HTTP refer[r]er's by setting log_referrer. (Vern Paxson).

- The "rst" tool (aux/rst/) now takes an optional "-I <text>" argument
  that instructs it to inject <text> as payload rather than sending a RST
  packet (Vern Paxson).  <text> must be NUL-terminated, and the NUL is not
  included.

- Bug fix for crashes in the DNS analyzer when processing replies for
  which no request was seen (Robin Sommer).

- Addressed a number of lint nits (Vern Paxson).

- Rotation post-processors are now passed an additional argument
  indicating whether Bro is terminating (Robin Sommer).

- Bro now consistently generates a file_opened event for all fopen() calls.
  (Robin Sommer).

- The "cf" utility now ignores a leading "t=" prefix, for compatibility
  with Bro's "tagged" logging format (Robin Sommer).

- You can now redefine the email_notice_to function (Robin Sommer).

- Fix for packet processing resumption when a remote Bro dies during
  state synchronization (Robin Sommer).

- OpenSSL/X509 portability fix, at long last (Gregor Maier & Christian
  Kreibich).

- Fix for compatibility with newer versions of autoconf (Gregor Maier).

- A larger BroControl update (Robin Sommer, if not marked otherwise):

  o Increasing default timeouts for scan detector significantly.

  o Increasing the manager's max_remote_events_processed to
    something large, as it would slow down the process too much
    otherwise and there's no other work to be interleaved with it
    anyway.

  o Adding debug output to cluster's part of catch-and-release
    (extends the debugging already present in policy/debug.bro)

  o Fixing typo in util.py. Closes #223.

  o Added note to README pointing to HTML version.

  o Disabling print_hook for proxies' remote.log.

  o broctl's capstats now reports a total as well, and stats.log
    tracks these totals. Closes #160.

  o Avoiding spurious "waiting for lock" messages in cron mode.
    Closes #206.

  o Bug fixes for installation on NFS.

  o Bug fix for top command on FreeBSD 8.

  o crash-diag now checks whether gdb is available.

  o trace-summary reports the sample factor in use in its output,
    and now also applies it to the top-local-networks output (not
    doing the latter was a bug).

  o Removed the default twice-a-day rotation for conn.log. The
    default rotation for conn.log now is now once every 24h, just
    like for all other logs with the exception of mail.log (which is
    still rotated twice a day, and thus the alarms are still mailed
    out twice a day).

  o Fixed the problem of logs sometimes being filed into the wrong
    directory (see the (now gone) FAQ entry in the README).

  o One can now customize the archive naming scheme. See the
    corresponding FAQ entry in the README.

  o Cleaned up, and extended, collection of cluster statistics.

    ${logdir}/stats now looks like this:

      drwxr-xr-x   4 bro  wheel      59392 Apr  5 17:55 .
      drwxr-xr-x  96 bro  wheel       2560 Apr  6 12:00 ..
      -rw-r--r--   1 bro  wheel        576 Apr  6 16:40 meta.dat
      drwxr-xr-x   2 bro  wheel       2048 Apr  6 16:40 profiling
      -rw-r--r--   1 bro  wheel  771834825 Apr  6 16:40 stats.log
      drwxr-xr-x   2 bro  wheel       2048 Apr  6 16:25 www

    stats.log accumulates cluster statistics collected every time
    "cron" is called.

    - profiling/ keeps the nodes' prof.logs.

    - www/ keeps a subset of stats.log in CSV format for easy plotting.

    - meta.dat contains meta information about the current cluster
    state (in particular which nodes we have, and when the last
    stats update was done).

    Note that there is no Web setup yet to actually visualize the data in
	www/.

  o BroControl now automatically maintains links inside today's log
    archive directory pointing to the current live version of the
    corresponding log file (if Bro is running). For example:

        smtp.log.11:52:18-current -> /usr/local/cluster/spool/manager/smtp.log

  o Alarms mailed out by BroControl now (1) have the notice msg in the
    subject; and (2) come with the full mail.log entry in the body.

  o Fixing broctl's top output. (Seth Hall).

  o Fixing broctl's df output in certain situations.

  o BroControl fix for dealing with large vsize values reported by
    "top" (Craig Leres).

1.5.1 Fri Dec 18 15:17:12 PST 2009

- Due to a Python configuration problem, the original 1.5 distribution
  did not include the BroControl component, which also introduced a
  portability problem for CentOS.  These issues have now been fixed (Robin
  Sommer and Vern Paxson).


1.5 Wed Dec 16 21:28:47 PST 2009

- Bro now comes with a new framework, BroControl, for managing an
  operational Bro setup, including support for installation, configuration,
  and maintainance tasks such a log archival and mail notification.  The
  framework transparently supports both traditional standalone setups as
  well as cluster installations in which multiple Bro boxes coordinate to
  analyze a high-volume network link.

  See aux/broctl/README for more information about BroControl.

  Note, BroControl supersedes the older BroLite system, which is no longer
  supported and has been deprecated for a while now.

- Numerous adjustments to DPD = dynamic protocol detection (Robin Sommer):

    o The Analyzer::ProtocolViolation?() method can now be passed the
      offending data (which POP3, SMTP, and FTP now do).  This information
      is added to the "reason" string passed to the script level.

    o SMTP now more accurately reports violations.

    o FTP stops processing when client & server successfully negotiate
      an AUTH scheme (leading to subsequent encryption).

    o Analyzer::ProtocolViolation() is virtual, and
      TCP_ApplicationAnalyzer() overrides it to not report violations
      for any partial connections, because very likely these arise just
      due to the analyzer getting confused.

    o TCP::IsPartial() returns true if any side did not start with
      a SYN packet (used to be just be for the originator).

    o The connection_state_remove handler in conn.bro now has a higher
      &priority so that other handlers for the same event can use
      determine_service() and see any changes it performs.

    o DynDisable:max_volume specifies a volume limit (default 10K).
      Once a connection exceeds this limit, further protocol
      limitations will neither raise ProtocolViolation notices nor
      cause the analyzer to be disabled.

    o The event engine no longer raises protocol_violation events for
      TCP connections which had gaps, as these have proven too unreliable.
      (Note that, ideally, the *analyzers* should avoid reporting
      protocol_violations when they can't reliably parse a connection
      anymore after a gap; but many don't.)

- A set of new script functions provide support for incrementally computing
  MD5 checksums (Seth Hall).

	md5_hash_init(index: any): bool
		Initializes an incremental hashing instance.  "index" is
		a value of arbitrary type, used to identify this particular
		instance (you can have multiple concurrent instances by
		using different index values).  Returns T on success,
		F on failure (such as the index is already in use).

	md5_hash_update(index: any, data: string): bool
		For the given hashing instance, updates the hash
		based on the given data.  Returns T on success, F on
		failure (such as the index has not been initialized).

	md5_hash_finish(index: any): string
		Returns the MD5-printable hash for the given index
		and terminates the instance, or the string "" if the
		index was not active.

- Bro now supports a believed-to-be-robust mechanism for estimating the
  proportion of traffic that it failed to capture ("measurement drops"),
  which can arise due to overload in either Bro itself, the kernel's
  packet filter, or problems with the link tapping mechanism (Vern Paxson).
  The event engine can generate estimates for either live traffic or what
  was previously recorded in a trace file, though traces subject to some
  forms of selective omission (such as skipping over parts of a connection
  to reduce storage) can lead to erroneous values.

  The estimates are based on observing gaps in TCP data streams, and
  come in two forms: the rate at which such gaps appear, and the relative
  volume of data missing due to the gaps.  (We've found however that the
  volume-based estimator is not robust due to occasional packets with
  incorrect sequence numbers, so this estimator is off by default.)

  The easy way to get the estimates is to load capture-loss.bro.
  By default, it generates a CaptureLossSummary notice upon Bro's exit,
  which can look like:

	1130222759.344066 CaptureLossSummary estimated rate = 0.00089124 / 0.000970997 (events/bytes)

  If the estimated loss is none, however, it suppresses this notice,
  unless you redef CaptureLoss::summary_if_none to T.

  You can also get finer-grained access by defining a "gap_report"
  event handler and redef'ing gap_report_freq to a non-zero interval
  (such as "10 sec").  This event allows you to pinpoint regions in
  time that exhibit significant capture loss.  See capture-loss.bro
  for an example of a handler for this event.

  Finally, these changes include a number of fixes to Bro's
  ack_above_hole/content_gap analysis, which is now significantly
  more robust.

- GeoIP support now supports ASN lookups via the built-in
  function lookup_asn(a: addr): count (Scott Campbell and Seth Hall).

- The GeoIP built-in's lookup_location() and lookup_asn() now
  support IPv6 (Seth Hall).  Note, the current GeoIP distribution
  doesn't include any IPv6 databases, so for now these won't succeed,
  but the hooks are in place for when databases become available.

- lookup_location() now falls back back to the country database if
  the city database isn't available (Seth Hall).

- The new SuccessfulPasswordGuessing Notice is generated when a host
  has been seen attempting password guessing (currently only for FTP
  sessions) and then successfully logs in (Royal Chan).  You can control the
  threshold for such reports in terms of how many attempts the host must
  have made by redef'ing the variable password_guessing_success_threshhold,
  which defaults to 20.

- The new script http-detect-passwd.bro analyzes the Web items returned
  for fetches that appear to be accessing the passwd file (Akhil Dhar).
  It generates a PasswordFullFetch Notice if it appears that the item
  includes a full password file, and PasswordShadowFetch if it looks like
  a shadowed password file.

- The new built-in

	system_env(cmd: string, env: table[string] of string)

  works like system(), but puts the table entries into the environment
  before invoking the command (Robin Sommer).  Each <index> in the table
  creates an environment variable of the form "BRO_ARG_<index>", whose
  value is the corresponding table entry.

- The new script function

	execute_with_notice(cmd: string, notice_info)

  executes "cmd" with an environment containing the fields of the
  notice_info, i.e., the information associated with a Notice (Robin Sommer).
  Per the new system_env() function above, the environment variables appear
  as "BRO_ARG_<tag>", where <tag> is the field tag as it appears in
  notice.log when you enable use_tagging.

- The new built-in enable_raw_output(file) acts the same as
  the attribute &raw_output (Seth Hall).

- The new built-in file_opened(f: file) event is generated any time Bro
  opens a script-level file (Justin Azoff).  You can use this, for example,
  if you want to ensure that a given file has a prelude in it such as
  human-readable headers, even when the file is rotated.

- The notice_info record has a new field

	aux: table[string] of string &optional

  which you can use for information specific to a given type of notice
  (Robin Sommer).  Entries in $aux appear as "aux_<index>" tags in notice.log.

- Another new notice_info record field is the boolean do_alarm (default=T),
  which, if set to F, overides a notice action otherwise specifying to
  generate an alarm (Robin Sommer).  In other words, if do_alarm is F, no
  alarm will be generated independent of the notice action.

  This is a work-around for the fact that we can't specify more than one
  action.  In particular, we couldn't NOTICE_DROP but then *not* alarm,
  which we now can by returning NOTICE_DROP yet setting do_alarm to F.

- The notice_info record field $dropped now appears in the tagged output
  format if true (Robin Sommer).

- NOTICEs relating to scan detection now no longer include the connection
  that triggered the notice, as it really doesn't contain any useful
  information, given that the particular trigger simply depends on the
  detection algorithm and its parameters (Robin Sommer).  However, we do
  explicitly set $p (port number) in the notice, and also $n with the
  number of attempts.

- drop.bro now hardwires a Catch-and-Release redrop after seeing one
  connection from a previously-dropped-but-already-released host
  (Robin Sommer).

- drop.bro now provides some new hooks (Robin Sommer):

	event address_dropped(a: addr)
		Generated when an address has been dropped.

	event address_restored(a: addr)
		Generated when connectivity to an address has been restored,
		such as using the Catch-and-Release mechanism.

	event address_cleared(a: addr)
		Generated when an address that was dropped in the past is
		no longer being monitored looking for new connections
		(as part of the Catch-and-Release mechanism).

- The new built-in function

	hexdump(data_str: string) : string

  returns a hex dump representation of the given input data (Christian
  Kreibich).  The dump renders 16 bytes per line, with hex on the left and
  ASCII (where printable) on the right.

- Bro's notion of when a TCP connection begins now dastes to the first
  instance of an initial SYN packet seen, rather than the last (Gregor Maier).

- The Time Machine script tm-contents.bro now generates

	event contents_saved: event(c: connection, orig_file: string,
					resp_file: string)

  when the content of a connection has been completely saved to disk
  (Robin Sommer).

- The mime.bro script now exports the MIME header callback table, and also
  marks it as &redef'able so you can modify its entries (Matthias Vallentin).
  The mime_log file is also now exported.

- A new signature file, policy/sigs/http-bots.sig, contains signatures
  to detect some of the current HTTP based controlled bot families (Seth Hall).

- The signature engine's HTTP pattern matching has been fixed (Seth Hall)
  to align with the documentation at:

  http://www.bro-ids.org/wiki/index.php/Reference_Manual:_Signatures#Content_conditions

  In particular, the content condition "http" is now referred to as
  "http-request" (though "http" still works for backward compatibility),
  "http-request-header" and "http-reply-header" now provide access to
  headers seen in only one direction, and similarly for "http-request-body"
  and "http-reply-body".  (This latter is still accessible as "http-body"
  for backwards compatibility.)

- The new script variable max_remote_events_processed: count (default 10)
  sets a limit on the number of remote events processed in each round,
  before tending to other inputs (Robin Sommer).

- If you set the new script variable dump_used_event_handlers to T,
  then on startup Bro dumps out all of the event handlers that the
  loaded set of scripts can invoke (Matthias Vallenti).

- Summaries for DNS PTR scanning now use a separate Notice,
  DNS_PTR_Scan_Summary, rather than overloading DNS_PTR_Scan (Robin Sommer).

- scan.bro now provides a table skip_dest_server_ports: set[addr, port]
  which lists servers (defined as an address and a port) excluded from
  scan detection computations (Craig Leres and Jay Krous).

- When redefining values on the command line directly (using var=value),
  quotation marks are now implicit only if "var" is a variable of type
  string (Christian Kreibich).  This allows other string-like values
  (such as enum's) to be passed as well.

- scan.bro now explicitly loads conn.bro so that it can itself
  be loaded independently (Robin Sommer).

- login.bro depends on scan.bro (because of tracking authentication
  "scans"), so now it explicitly loads it (Vern Paxson).

- UDP_datagram_length_mismatch is now by default flagged just once per
  originating host rather than once per connection, as it can generate
  tons of messages (Vern Paxson).

- Removed now-long-boring flagging of access to Solaris "listen"
  service as "hot" (Vern Paxson).

- Removal of libedit, since libreadline provides similar functionality
  (Christian Kreibich).

- Added scripts missing from distribution: dce.bro, ncp.bro, and smb.bro
  (Vern Paxson).

- ssh.bro now exports ssh_ports (Seth Hall)

- A number of improvements to inter-Bro communication (Robin Sommer).

 (1) Remote communication now no longer includes location information for
 serialized objects; that removes quite a bit of redundacy from the network
 traffic.

 (2) The new option 'remote_check_sync_consistency" disables the cross-check
 on the receiving side of &synchronized state of whether the current value
 of a variable has the value expected by the sender. Transmitting the
 original values in addition to the updates generates quite a bit CPU &
 network load in some cases (in particular, a table of tables). The default
 for remote_check_sync_consistency is off, and so far that in particular
 seems to reduce the proxy's load quite a bit.

 (3) Complete overhaul of the internal caching of serialized objects.  The
 objective of the caching is avoid retransmitting already sent values over
 and over again. It turns out, however, that some objects are very stable
 and hardly change or get replaced (e.g., Bro types); while other change
 all the time and are hardly reused some time later (e.g., Vals).  Now
 we maintain *two* caches independently for these types of objects; one
 with a low turn-over one and another with a high one.  This should reduce
 CPU load on both sender and receiver sides.

 The new scheme is only used if both communicating Bros support it; with
 older Bros, as well as with Broccoli, we continue using the old scheme.

- Some reworking of remote printing (Robin Sommer), as follows.  Bro now
  uses a new interprocess message rather than print_hook events, to better
  manage buffering and associated load (these can produce failures depending
  on system configuration; see remote.log).  A number of timeouts and
  buffer sizes have been tuned.  Internally, EINTR errors are now treated
  separately from EAGAIN.  Finally, even with remote_check_sync_consistency=F,
  one type of consistency check was still being done; this is no longer
  the case.

- The DNS analyzer now generates events (dns_query_reply/dns_rejected)
  for replies with zero questions (Robin Sommer).

- Perftools support for incompatible changes in the 1.0 API (Robin Sommer).

- Rearranged (generally reducing, though not always) some state timeouts
  associated with scan detection (Robin Sommer).  In addition, when a
  scanning address crosses ignore_scanners_threshold (meaning that it will
  be ignored from now on anyway), it gets discarded from all state-tracking
  tables.  Finally, the ignore_scanners_threshold now applies all kinds
  of scans, not just address scans.

- Substantial Broccoli updates, including a new initialization requirement
  that breaks backward compatibility, support for enqueueing serialized
  event data for transmission, and OpenSSL threadsafe initialization.
  See aux/broccoli/ChangeLog for details (Christian Kreibich, Robin
  Sommer, and Matthias Vallentin).

- Broccoli hashtable optimisation. See aux/broccoli/ChangeLog for
  details (Christian Kreibich & Matthias Vallentin).

- Broccoli memory leak fixed, see aux/broccoli/ChangeLog for details
  (Christian Kreibich).

- Broccoli: updates to bropipe tool (Steve Chan and Robin Sommer).

- Bug fixes for Broccoli Python bindings (Robin Sommer and Matthias Vallentin).

- Fixed nasty bug due to module scoping that completely kept stepping-stone
  detection from working (Vern Paxson).

- A serious bug in the packet sorter has been fixed (Robin Sommer).

- Bug fix for extra NULs getting embedded in escaped strings (Seth Hall).

- Bug fix for HTTP messages that use "Connection: close" rather than length
  headers, which yielded erroneous reassembled messages with \r\n's when
  only \n's were present (Bernhard Ager).

- Fix for reporting on ICMP flows that are expired from the flow table
  (Vern Paxson).  Previously there was a race condition if the flow
  was flushed prior to its summary timer expiring.

- The -l option (list the scripts that Bro loads) now correctly prints
  scripts loaded by the prefix mechanism, and uses indentation to indicate
  the load hierarchy (Robin Sommer).

- A bug has been fixed (really, worked around) in drop.bro that prevented
  dropped addresses from being properly restored (Robin Sommer).

- Fixes for deadlocking problems in the Broccoli protocol. See
  aux/broccoli/ChangeLog for details (Christian Kreibich & Robin Sommer).

- Bug fix for DNS analyzer on 64-bit machines (Gregor Maier).

- Bug fix for asynchronous DNS lookups to prevent some successful lookups
  being reported as timed out (Robin Sommer).

- Bug fix for tracking line numbers associated with compound statements
  (Po-Ching Lin).

- Fix for a rare condition in which the main Bro process couldn't kill
  its child process (Robin Sommer).

- Fix for file rotation when the underlying file is deleted before the
  timer expires (Robin Sommer).

- Fix for potential crash when communication connections break down,
  and also for releasing cached objects (Robin Sommer).

- Fix for default table entries computed by function invocation to not
  cache previous results (Robin Sommer).

- Fix for Bro's internal DNS resolution (Scott Campbell and Robin Sommer).

- Portability fix for DAG packet capture (Gregor Maier).

- Portability fix for --enable-brov6 (Robin Sommer).

- Portability fixes for FreeBSD (Vern Paxson).

- A work around for new_packet() crashing on IPv6 packets (Vern Paxson).
  For now, IPv6 packets are skipped.  Also, for fragments the event handler
  is now only called for the fully reassembled packet.

- The new configuration option --disable-nbdns supports disabling non-blocking
  DNS at configure time (Sean McCreary).  Note, there are some known problems
  with it in some environments.

- A number of configuration fixes and enhancements (Christian Kreibich
  and Robin Sommer).

- Consistency nit for the configuration process (Seth Hall).

- A number of reference-counting and other memory management fixes
  (Robin Sommer).

- Bug fix for inter-Bro communication lockup (Seth Hall and Robin Sommer).

- Bug fix for computing TCP payload length in new_packet event (Lothar Braun).

- Bug fix for sending boolean True values via Broccoli (Seth Hall).

- make distcheck fix to clean up .bif.bro files (Christian Kreibich).

- Bug fix for DPD's recognition of SSLv2 connections (Seth Hall).

- Bug fix for &default for tables indexed by subnets (Seth Hall).

- A bug has been fixed that could crash Bro when you called get_event_peer()
  after a remote connection had already disppeared (Robin Sommer).

- Introduced a work-around for crashes that occur when Bro exits
  due to handling a signal (Robin Sommer).

- Bug fix for checkpoint.bro - don't schedule timers for times that
  aren't actually in the future (Robin Sommer).

- Hostname formatting fix for anon.bro (Fabian Schneider).

- Bug fix for redundant .log extension in Time Machine log file
  (reported by CS Lee).

- Removed now-outdated special-casing of Linux reporting of packet filter
  statistics (Peter Wurzinger and Robin Sommer).

- A number of memory leaks fixed (Robin Sommer).

- Addressed warnings from newer versions of g++ (Robin Sommer and Vern Paxson).

- Fixed an invocation issue in the ca-create script that prevented it from
  working with recent OpenSSL versions (Craig Leres & Christian Kreibich).

- Comment fixed in drop-adapt (Justin Azoff).

- Duplicate code removed from Val (Seth Hall).


1.4 Fri Oct 17 11:08:52 PDT 2008

- We are no longer supporting a previous Bro release as the "stable"
  version.  Rather, the model now is that the current public release will
  aim for increasing stability (occasionally updated with fixes), and those
  who wish to use a "bleeding-edge" snapshot can do so via access to the
  public SVN source code repository, as explained at

	  http://bro-ids.org/wiki/index.php/Subversion#Public_Access

  Note that all previous releases remain available from the download page;
  what is changing is that we no longer commit to support for the most
  recent of these.

- We have clarified the copyright statement that covers most of the
  code to remove the "advertising clause" that derived from older
  BSD licenses, and we have removed copyright wording from most source
  code files.  See COPYING for the current wording and a list of
  files that retain their own copyright notices.

- Bro now supports analyzing NetFlow v5 data, i.e., from Cisco routers
  (Bernhard Ager).  NetFlow can be useful for intrusion detection as it
  allows analysis of traffic from many different points in the network.
  Bro can now read NetFlow data from a UDP socket, as well as (mostly
  for debugging purposes) from a file in a specialized format.  You can
  create these files with the programs given in aux/nftools.

  Command line switches:

	  -Y|--netflow <ip>:<prt>[=<id>] | read flow from socket

	    This is the usual way of getting NetFlow data into Bro by
	    opening a UDP socket on <ip>:<prt> and reading all incoming
	    packets.  Setting the <ip> to 0.0.0.0 should work on most
	    platforms.  Optionally you may set an identifier <id> for the
	    source - useful if there are many different sources you want
	    to analyze in parallel. This might also be necessary if you
	    want to use this feature with a clustered Bro.

	    Examples:
		      bro -Y 0.0.0.0:5555 netflow
		      bro -i eth0 -Y 10.0.0.1:1234=src1 brolite netflow

	  -y|--flowfile <file>[=<ident>]

	    Used to read from a file. You can optionally include an
	    identifier for the source.

	    Examples:
		      bro -y myflowfile netflow
		      bro -y myflowfile=src1 otherflowfile=src2 netflow

  Netflow Events:

	  event netflow_v5_header(h: nf_v5_header)

	    Generated upon reading a new NetFlow PDU, as summarized in the
	    argument.  The field h_id gives the flow source identifier and
	    a serial number. You can use this field to associate subsequent
	    netflow_v5_record events with their header.

	  event netflow_v5_record (r: nf_v5_record)

	    Every record within a NFv5 PDU generates a corresponding
	    netflow_v5_record() event.  The relatively complex timestamp
	    format of NFv5 is already converted to Bro's time type, and
	    the TCP header flags are separated into bools.

  The distribution includes an example analysis script, netflow.bro.
  It simply dumps received NetFlow records.  If netflow_restitch is T
  (the default), then Bro performs flow restitching as well, and two
  script variables become relevant:

	  global netflow_finished_conn_expire = 310 sec &redef;

	    specifies how long to wait for additional flow records after
	    a RST or FIN for

	  const netflow_table_expire = 31 min;

	    Its setting only affects table declarations, and therefore
	    cannot be usefully redef'd.

  Auxiliary programs:

	    Bro uses a custom format for flow data stored in files,
	    to enable preserving timestamps of the PDU arrivals and the
	    exporter's IP address.  The tools nfcollector and ftwire2bro
	    in aux/nftools/ provide ways to manipulate the Bro NF file
	    format.  The first dumps NetFlow data from a UDP socket to
	    stdout or to a file in Bro format.  The second converts NetFlow
	    data in "wire" format to Bro format, and, while doing so,
	    fakes up the exporter's IP address and timestamp.  You can get
	    "wire" format from normal flow-tools files, e.g., by using
	    'flow-export -f 4'.  Please note that the Bro format is just
	    a hack to allow for easier debugging.  Therefore the format
	    is not in fact platform independent, and not suitable for data
	    storage.

- A new DHCP analyzer generates the following events (Po-Ching Lin):

	event dhcp_discover(c: connection, msg: dhcp_msg, req_addr: addr)
	event dhcp_offer(c: connection, msg: dhcp_msg, mask: addr,
	event dhcp_request(c: connection, msg: dhcp_msg,
	event dhcp_decline(c: connection, msg: dhcp_msg)
	event dhcp_ack(c: connection, msg: dhcp_msg, mask: addr,
	event dhcp_nak(c: connection, msg: dhcp_msg)
	event dhcp_release(c: connection, msg: dhcp_msg)
	event dhcp_inform(c: connection, msg: dhcp_msg)

  where dhcp_msg values look like:

	type dhcp_msg: record {
		op: count;	# 1 = BOOTREQUEST, 2 = BOOTREPLY
		m_type: count;	# the type of DHCP message
		xid: count;	# transaction ID of a DHCP session
		h_addr: string;	# hardware address of the client
		ciaddr: addr;	# original IP address of the client
		yiaddr: addr;	# IP address assigned to the client
	};

  See dhcp.bro for the corresponding analysis script (which could
  probably use some refinements).

  Note, this analyzer is implemented using BinPAC, so you will need
  to specify --use-binpac to activate it.

- A BitTorrent analyzer is now available (Nadi Sarrar).  See the policy
  scripts bittorrent.bro and bt-tracker.bro for the events generated for
  analyzing transfers and tracker dialogs, respectively.

- The "Bro Lite" configuration is now deprecated and will not in
  general be supported (Robin Sommer & Vern Paxson).

- "make install" now only installs a core set of files (Robin Sommer).
  Policy files are now installed in <prefix>/share/bro/* (or whatever
  configure determines $datadir to be), which is now in Bro's default
  search path.  It creates a directory <prefix>/share/bro/site for local
  policy files, and the default BROPATH is extended to include this.  The
  default path no longer includes policy/local.  You can install the
  additional files used by the (now deprecated) "Bro Lite" configuration
  using "make install-brolite".

- Substantial updates to Broccoli, including support for container
  types (tables and sets) as well as a new metadata structure for event
  callbacks, facilitating truly generic event handler implementations
  (Christian Kreibich, Seth Hall and Robin Sommer). See aux/broccoli/ChangeLog
  for details.

- Extensive changes to allow Bro to process packets captured in the
  past intermingled with those captured in real-time (Matthias Vallentin
  and Robin Sommer).  This operation reflects combining Bro with use of
  "Time Machine" functionality for packet capture.

- We have unfortunately had to disable support for configuring Bro
  to use ClamAV, since it turns out that the key interface we need
  for processing blocks of memory directly rather than whole files
  is no longer supported by the package, and in fact was buggy even
  when it was (Robin Sommer).

- The new signature option "http-body /<regexp>/" matches <regexp>
  on the body data of HTTP entities (Robin Sommer).  The matching is
  done after decompressing the body, if necessary.

- The new built-in function identify_data(data: string, return_mime: bool)
  analyzes the string "data" and returns its type according to libmagic,
  if installed (Seth Hall).  The second argument controls whether it should
  be returned as a MIME-type or just an identifying string.  For example,
  identify_data("MZpofigu", F) returns the string "MS-DOS executable", and
  print identify_data("MZpofigu", T) returns "application/x-dosexec".

- The new analysis script http-identified-files.bro identifies the
  type of items returned by Web servers using libMagic (if available)
  and generates notices for interesting types and mismatches between
  URLs and types (Seth Hall).

  You configure it using two variables.  watched_mime_types is a pattern
  (default /application\/x-dosexec/ | /application\/x-executable/ ) for
  which any MIME type matching the pattern generates a HTTP_WatchedMIMEType
  notice.

  mime_types_extensions is a table mapping strings to patterns specifying
  how URLs for the given MIME type should appear.  (Ideally, this would
  be a table mapping patterns to patterns, but Bro doesn't currently support
  that.)  It defaults to:

		["application/x-dosexec"] = /\.([eE][xX][eE]|[dD][lL][lL])/

  i.e., do Windows executables end in .exe or .dll.

  You can also redef the pattern ignored_urls to specify URLs that should
  not generate complaints.  It defaults to matching Windows Update.

- The new script http-extract-items.bro extracts the items from HTTP
  traffic into individual files (Vern Paxson).  Files are named:

	<prefix>.<n>.<orig-addr>_<orig-port>.<resp-addr>_<resp-port>.<is-orig>

  where <prefix> is a redef'able prefix (default: "http-item"), <n> is a
  number uniquely identifying the item, the next four are describe the
  connection tuple, and <is-orig> is "orig" if the item was transferred
  from the originator to the responder, "resp" otherwise.

- The workings of how Bro interfaces to external programs for dropping/
  restoring connectivity of misbehaving hosts has been significantly
  reworked (Brian Tierney and Robin Sommer).

  First, dropping decisions used to be made directly by analyzer scripts,
  such as scan.bro directly calling drop_address().  Now instead the
  scripts generate Notices and then the notice policy can have an
  action of NOTICE_DROP to codify that the response to the given Notice
  is to drop the source.  The new notice_action_filter of drop_source
  drops the source of notices, and drop_source_and_terminate both
  drops the source and terminates the corresponding connection.

  So, to drop all sources triggering a specific notice, one can now, e.g.,
  write:

	redef notice_action_filters += { [Hot::SSH_Overflow] = drop_source };

  Related to this change, notice_info has a new field $dropped, set to
  true if the Notice triggered a (successful) drop.

  Second, by redef'ing Drop::use_catch_release to T (default F) you can
  activate "catch-and-release" logic.  You use this mode when you need to
  manage a limited number of possible blocks, or to build in automatic
  "forgiveness" in situations where blocked sources might become benign
  (such as due to dynamic IP addresses).  If a source has been idle for
  Drop::drop_time, then it is unblocked.  However, if it is again seen as
  block-worthy, then it is blocked for an interval of Drop::long_drop_time.

  Third, ICMP scanning is now reported by its own notice, ICMPAddressScan,
  rather than Scan::AddressScan.

- Google's perftools have replaced mpatrol for leak-checking and
  heap-profiling (Robin Sommer).  If Bro is compiled with --enable-perftools
  and configure finds the perftools, there are two command-line options
  available:

	-m turns on leak checking of the main packet loop, with some
	   uninteresting leaks are suppressed.  Currently, with one
	   exception (the RPC analyzer; problem not yet found), it reports
	   no leaks when running the test suite.

	-M turns on heap profiling: Bro will take a snapshot of the heap
	   before starting the main packet loop and another one when
	   finished. These snapshots can then be analyzed with pprof.

  For more information about the perftools see

	http://code.google.com/p/google-perftools

- Notice tags are now generated in a pseudo-unique fashion that, with high
  probability, ensures that tags generated by separate Bro processes don't
  clash when logged to a common location, such as for a Bro cluster (Robin
  Sommer).  Tags are now string's rather than count's, and are associated
  with all notices, not just that are connection-related.  You can however
  redef the string notice_tag_prefix or the function new_notice_tag to
  further control how such tags are generated.

- Four new built-ins for type conversion (Robin Sommer):

	function double_to_interval(d: double): interval
	function addr_to_count(a: addr): count
	function port_to_count(p: port): count
	function count_to_port(c: count, t: transport_proto): port

- Many policy scripts have been modified to use modules & scoping
  (Robin Sommer and Matthias Vallentin), which may require updates to
  existing scripts/refinements.

- The new script variable dpd_conn_logs (default F), if true, changes the
  semantics of the service field in connection logs written to conn.log,
  as follows (Robin Sommer).  It becomes a comma-separated list of analyzers
  confirmed by DPD to parse the connection's payload.  If no analyzer could
  confirm its protocol, but the connection uses a well-known port, the
  service is the name of the port with "?" appended (e.g., "http?"), as
  long as the corresponding analyzer has not declined the connection.
  In addition, ftp-data sessions are labeled "ftp-data" and portmapper
  connections are labeled with the specific method-call (just as before).

  dpd_conn_logs defaults to F because the change in semantics may break
  scripts that parse conn.logs; but it will likely change to the default
  in the future. With dpd_conn_logs turned off, conn logs are generated
  as they used to be, with a few rare exceptions (with previous versions,
  the service field was sometimes determined while the connection was still
  alive; now it's always determined at the time when the conn.log entry
  is written out).

- The SSL analyzer has been rewritten using BinPAC, with a number of
  robustness improvements (Tobias Kiesling).  It currently is only used
  if you execute with --use-binpac.

- Python bindings for Broccoli are now available in
  aux/broccoli/bindings/python/ (Robin Sommer).  See README/README.html
  in that director for details.

- The new "auth" option in remote.bro indicates whether a given side is
  considered "authoritative" for shared state, in which case it sends its
  initial state to &sync'ed peers (Robin Sommer).  When two peers synchronize
  their state, one side sends its current set of state to the other as
  soon as the remote connection is established.  The one sending the state
  used to be the one who has been running longer; now it can also be
  explicitly set via the "auth" flag in the Remote::Destination.

- Two new tuning parameters for scan.bro (Robin Sommer):

  ignore_scanners_threshold (default 0):

	If a host has scanned more than this many hosts, it is completely
	excluded from further scan detection.  0 disables.

  addr_scan_trigger (default 0):

	A host is only tracked for address scanning once it has contacted
	this many different hosts.  Primarily intended for using a two-stage
	scan detection with a Bro cluster: first, each node searches locally
	for scanners by looking for hosts contacting more than
	addr_scan_trigger destinations.  Those hosts which do are then
	globally tracked throughout the cluster by &synchronizing the scan
	detector tables.

- When Bro serializes functions, it now does so by default using only
  their name, rather than their full value (Robin Sommer).  This prevents
  propagation of expiration functions associated with tables and sets.
  Note, currently there is no mechanism provided to switch from the
  default behavior, but the internal hooks are in place to do so.

- The new built-in variable trace_output_file gives the name of the -w
  output trace file (Robin Sommer).

- Bro no longer installs new file rotation timers when shutting down
  (Robin Sommer).

- The new policy scripts remote-print-id{,-reply}.bro support convenient
  access to printing the identifiers of a remote Bro (Robin Sommer).
  You use the script remote-print-id.bro to request and receive the
  printing; the remote Bro must have loaded remote-print-id-reply.bro
  in order to process the request.

  Example use:

	  bro -e 'redef PrintID::dst="<dst>" PrintID::id="<name-of-id>"'
			<other scripts> remote-print-id

- scan.bro has been heavily modified to better support distributed scan
  analysis (Matthias Vallentin and Robin Sommer).

- The check for unused event handlers is now turned off by default
  (Robin Sommer).  To enable, use "redef check_for_unused_event_handlers = T".

- The new script drop.bro has been split off from scan.bro to isolate
  the logic concerning dropping addresses to block scans (Robin Sommer).

- The new -l flag lists each script as it is loaded (Robin Sommer).

- Textual descriptions of identifiers now include their attributes
  (Robin Sommer).

- The new predefined function prefixed_id() returns a session identifier with
  its peer-ID prepended if it's associated with a remote Bro (Robin Sommer).
  This is now used when generating writing log files.

- remote.bro now assigns a priority of -10 to its bro_init() event handler
  to allow others a chance to modify destinations (Robin Sommer).

- A large number of BinPAC updates (Ruoming Pang and Robin Sommer).

- The new built-in type_name(v): string returns the name of the type
  of the value v (Vern Paxson).  For example, "typename(5.2)" returns
  "double".  This function is mainly for internal debugging (i.e.,
  finding mismatches between values generated by the event engine
  versus how their type is expected by the script layer).

- The new built-in str_shell_escape() does some basic escaping on strings
  that will be passed to system() (Christian Kreibich).  Note, this function
  isn't ready (robust enough) for routine use, however.

- The new built-in disable_print_hook(file) acts the same as
  the attribute &disable_print_hook (Robin Sommer).

- The new script terminate-connection.bro factors out the terminate_connection()
  functionality that used to be in conn.bro (Robin Sommer).

- The new attribute &group=<tag> can be associated with event handlers
  to group them together into a set that can be manipulated as a whole
  (Robin Sommer).  <tag> is a string reflecting the name given to the group.

  The built-in enable_event_group(group: string) turns on all the analyzers
  in a given group, and disable_event_group(group: string) deactivates them.

- The new attribute &raw_output applies to variables of type file, disabling
  escaping of non-printable characters (Seth Hall).

- You can now iterate over the characters in a string value using
  a "for" loop, e.g., "for ( c in str ) ..." (Robin Sommer).

- The new built-in

      function cat_sep%(sep: string, def: string, ...%): string

  works similarly to cat(), except that it (a) separates the values
  by "sep" and (b) substitutes "def" for empty strings (Seth Hall).

- The function string_escape() now takes a string of characters to escape
  rather than a single character (Robin Sommer).  Each character in the
  string is preceded by '\' in the return value (also any embedded '\'s,
  as before).

- The new built-in function global_ids() returns a table of all global
  identifiers along with associated information (Robin Sommer).  The
  return value has type table[string] of script_id, indexed by the name
  of the identifier and yielding records with the following fields:

	type script_id: record {
		type_name: string;
		exported: bool;
		constant: bool;
		enum_constant: bool;
		redefinable: bool;
		value: any &optional;
	};

- The new script function find_last(str: string, re: pattern) returns
  the last occurrence of the given pattern in the given string, or
  an empty string if no match (Robin Sommer).  Note that this function
  returns the match that starts at the largest index in the string, which
  is not necessarily the longest match.  For example, a pattern of /.*/
  will return just the final character in the string.

- The new script variable record_all_packets, if redef'd to T (default F),
  instructs Bro to record every packet it processes (Robin Sommer).
  Prior to introducing this variable, Bro applied a few heuristics to
  reduce recording volume.  Setting this variable also causes packets
  to be recorded very early in processing, which can be helpful for
  debugging crashes.

- If the new script flag ssl_log_ciphers is set to T (default), ssl.bro
  logs the ciphers seen (Robin Sommer).

- Much more expanded Time Machine support, now located in
  policy/time-machine/ (Robin Sommer),

- The new command line option --status-file <file> (alias -U) specifies
  the name of a file into which Bro will write an indicator of its current
  processing status (Robin Sommer).  Possible values include "INITIALIZING",
  "RUNNING", "TERMINATING", "TERMINATED".

- The new policy script targeted-scan.bro looks for repeated access from
  the same source to the same server, to detect things like SSH
  password-guessing attacks (Jim Mellander).

- The "alternative" style for printing strings (i.e., a fmt() argument
  of "%As") now renders the raw string, other than escape-expanding
  embedded NULs (Vern Paxson).  This change may be temporary, pending
  development of more fine-grained control over string rendering.

- For now we have removed the %S functionality for fmt() (Robin Sommer).
  %S was meant to print "raw" strings, but later processing of such
  printing still introduces artifacts.

- GeoIP information now includes latitude and longitude (Seth Hall).

- ssh.bro now supports the variable skip_processing_after_handshake
  which directs the event engine to omit any further processing of an
  SSH connection after its initial handshake (Seth Hall and Robin Sommer).
  This can help with performance for large file transfers but precludes
  some kinds of analyses (e.g., tracking connection size).  This change
  also adds a scope of "SSH".

- Email notification of notices now allows for separate destinations
  depending on notice type (in particular, a regular mail destination
  versus a pager destination), and also escapes the notice to prevent
  injection attacks (Seth Hall and Robin Sommer).

- The new policy script conn-flood.bro is a simple connection-flooding
  detector, mainly meant as a demonstration (Robin Sommer).

- A large number of additions to the TLS/SSL known-ciphers suite (Seth Hall).

- Serialization now uses 64-bit IDs to cache items rather than 32-bit,
  for robustness during long-running execution (Robin Sommer).

- The new script variable tcp_max_initial_window specifies, for flows
  for which ACKs have never been seen, the maximum volume of initial
  data after which Bro will assume that it is seeing only one side
  of the connection and will not buffer data for consistency checking
  awaiting the later arrival of ACKs (Robin Sommer).  It defaults to 4 KB.
  (Note, this used to be an internal value, so the behavior is not new.)
  Set to 0 to turn off this functionality and have Bro attempt to
  track all such flows.

- The new script variable tcp_max_above_hole_without_any_acks specifies,
  for flows for which ACKs have never been seen, the maximum volume of
  data above a sequence hole that Bro will tolerate for a connection
  before giving up on tracking the flow (Robin Sommer).  It defaults to 4 KB.
  (Note, this differs from tcp_max_initial_window in that this threshold
  applies to sequence holes rather than the beginning of flows.  Like
  tcp_max_initial_window this used to be an internal value.)  Set to 0 to
  turn off this functionality.

- The new script variable tcp_excessive_data_without_further_acks specifies
  a threshold similar to tcp_max_above_hole_without_any_acks, but for
  flows for which Bro has seen ACKs (Robin Sommer).  It defaults to 10 MB.
  Set to 0 to turn off the functionality.

- Equal signs ("=") in text for notices are now escaped when using the
  tagged format to keep them unambiguous from the "=" delimiters
  (Robin Sommer).

- The final tallies for notices are now processed as NoticeTally
  NOTICE's rather than directly alarm'd (Robin Sommer).

- WeirdActivity notices now include an associated connection when appropriate
  (Robin Sommer).

- Support for large (> 2^32 bytes) pcap trace files (Po-Ching Lin).

- Scoped names ("...::...") are now allowed in signature "eval"
  constructs (Christian Kreibich).

- scan.bro is now decoupled from conn.bro, i.e., you can @load the
  latter without getting the former (Vern Paxson).  As part of this
  change, the logic to invoke TRW is now in scan.bro.

- weird.bro has been updated with a number of missing Weird's (Vern Paxson).

- If when using inter-Bro communication the child Bro process terminates,
  it now also terminates the parent process (Robin Sommer).

- BinPAC analyzers now interoperate with DPD (Robin Sommer).

- Some http.bro processing options are now exported so they can be
  accessed in other scripts (Robin Sommer).

- SMTP analysis now applies to port 587/tcp as well as 25/tcp (Robin Sommer).

- $conn is now set in ServerFound notices (Robin Sommer).

- You can now create empty sets and tables using set() and table(),
  i.e., the usual set/table constructors with no arguments (Vern Paxson).
  By themselves, these have an unspecified type - you can't use them
  directly other than to assign them.  For example,

	local bad_guys: set[addr];
	...
	bad_guys = set();	# start over assuming no bad guys

- A number of scripts have been (slightly) simplified to use the
  new empty set()/table() constructors (Vern Paxson).  Note that
  these still aren't usable for field assignments in record constructors,
  nor for attributes like &default = ...

- Removed unused syntax for declaring sets based on a list of initial
  values (Vern Paxson).

- set() and table() can now be used as arguments to function calls
  (Vern Paxson).

- The vestigial &match attribute has been removed.

- POP3 is now recognized using Dynamic Protocol Detection (Seth Hall).

- The new event expected_connection_seen(c: connection, a: AnalyzerTag)
  is generated whenever a connection is seen for which we have previously
  scheduled an analyzer via expect_connection() (Robin Sommer).

- The new built-in capture_state_updates logs all changes applied to
  &synchronized variables, in a fashion similar to the capture_events()
  built-in (Robin Sommer).  An accompanying policy script,
  capture-state-updates.bro, turns this on to the file state-updates.bst.

- If the new script variable suppress_local_output is set (default: F),
  Bro suppresses printing to local files if there's a receiver for
  print_hook events (Robin Sommer).  This option is however ignored
  for files with a &disable_print_hook attribute.

- The new notice action filter function file_if_remote specifies
  that notices from sent from remote source addresses should
  have an action NOTICE_FILE (Robin Sommer).

- The new notice action filter function file_local_bro_notices specifies
  that notices generated by the local Bro instance (as opposed to a
  remote peer) should have an action NOTICE_FILE (Robin Sommer).

- An arbitrary tag can now be past to post-processors for log rotation
  (Robin Sommer).

- Default inactivity timeouts for interactive services shortened to
  1 hour (Robin Sommer).

- The scanning variables distinct_{peers,ports,low_ports} are now
  redef'able (Robin Sommer).

- The new -S (--summary-only) option for site-report.pl directs to
  only generate connection summaries (Brian Tierney)

- More useful default config file for edit-brorule.pl (Brian Tierney).

- Bro now includes a test suite in testing/istate/ for its "independent
  state" functionality (Robin Sommer).

- Support for parallel builds via make -j (Christian Kreibich).

- Bro's default search path now includes includes policy/sigs/ and
  policy/time-machine/ (Robin Sommer).

- Bro's internal processing of interprocess communication has been
  significantly overhauled to prevent potentially fatal race conditions
  (Robin Sommer).

- Bro now checks calls to fmt() at compile-time to ensure that the
  correct number of arguments are present (Vern Paxson).  This is useful
  in addition to Bro's run-time checking for arguments matching their
  corresponding format-specifiers in the case of rarely-executed statements
  that might not generate such run-time checks in routine testing.

- The ports associated with Telnet and Rlogin are now redef'able (Robin Sommer).

- MIME processing now removes leading whitespace from MIME headers
  (Sanmeet Bhatia and Robin Sommer).

- TCP "weird" events reported by the connection compressor now match
  (other than a few rare corner-cases) those produced for normal TCP
  processing (rmkml and Robin Sommer).

- Added Scan::suppress_UDP_scan_checks to control false positives
  on scan detection in environments with P2P protocols that use UDP
  (Vern Paxson).

- The internal analyzer interface now includes an EndOfData() method that
  analyzers can use to report that all of a message has been delivered
  (Robin Sommer).

- Fix for a significant memory leak in processing UDP when using -w
  (Robin Sommer).  Note: this change turns off by default trace rewriting
  for generic UDP traffic.

- Two serious regular expression bugs fixed (Vern Paxson).  In the
  first, searching for a regular expression inside a string would
  fail if the pattern occurred only after an embedded newline.  In
  the second, insufficient buffer was allocated when compiling regular
  expressions, leading to memory corruption.

- Base64 decoding bug fixes (Christian Kreibich and Ruoming Pang).

- Automatic rotation of files is now disabled for contents files written
  by the TCP reassembler, which otherwise leads to mangled files
  (Robin Sommer).

- Bro now ships with an updated version of libpcap (0.9.8), which hopefully
  fixes problems managing trace files > 4 GB in size.

- Significant bug fixes for gzip- and deflate-encoded Web items (Robin Sommer).

- Bug fix for secondary-filter.bro (Vern Paxson).

- Removed a naming ambiguity regarding TCP states (Vern Paxson).

- Bug fix for signature scanner not matching all of its input (Vern Paxson).

- Bug fix for using port values in signatures (Robin Sommer).

- Minor policy script tweaks: state management for weird's, processing
  of Notice tags associated with connections, and dependencies for
  irc-bot.bro (Robin Sommer).

- aux/ portability fixes (Vern Paxson).

- Workarounds added for a BinPAC deficiency, which is that code in %cleanup
  clauses can also be executed during recovery from exceptions when parsing
  new data.  This means that any delete's or Unref()'s need to also set the
  corresponding pointer to nil (Vern Paxson).

- Bug fix for crashes with the non-BinPAC SSL analyzer (Robin Sommer).

- Tweak to peer-status.bro since Bro now requires events to be
  declared prior to reference in a "schedule" statement (Robin Sommer).

- The signature keyword "enable" now optionally accepts the syntax
  "foo:bar" to specify "activate analyzer bar as a child of analyzer foo"
  (Robin Sommer).  This is used for example for an XML-over-HTTP analyzer
  that's in the works.

- irc-bot-syslog.bro now uses open_log_file() for its log file (including
  the logging suffix) rather than a direct open (Vern Paxson).

- Bug fix for tracking Blaster across a Bro Cluster (Robin Sommer).

- Bug fix for the HTTP BinPAC analyzer chopping the trailing character
  off of HTTP headers when generating the http_all_headers event (Gregor Maier).

- Bug fix for HTTP chunked items for which the chunk size line was terminated
  by CRLF but the CR and LF came in separate packets (Gregor Maier).

- A bug has been fixed that would cause partial lines (for line-oriented
  protocols) to fail to be processed when a connection terminated
  (Robin Sommer).

- Bro no longer treats a signal arriving before a previous signal has
  been processed as fatal, nor does it attempt processing of a termination
  signal if seemingly there are no race conditions to worry about
  (Robin Sommer).  Both of these changes are an attempt to improve
  Bro's robustness.

- Fix for attributes such as &encrypt not working in initial declarations
  but only in later redef's (Seth Hall and Robin Sommer).

- Fixes for memory leaks in SSL processing (Seth Hall and Robin Sommer).

- Fix for POP3 analyzer to not treat lines like "<space>." as message
  terminators (Robin Sommer).

- Bug fix for crashes arising from nil pointers in list expressions
  (Seth Hall and Robin Sommer).

- Bug fix: a signature's "enable" would activate the corresponding analyzer
  even if no event handlers were defined for it (Robin Sommer).

- Bug fixes to prevent crashes when mixing set_contents_file() with
  subsequent explicit close(), and to ensure all data written to
  file upon connection tear-down (Gert Doering and Robin Sommer).

- Configuration support for MacPorts and Fink package management systems
  (Christian Kreibich & Vern Paxson).

- Communication-only Bro's now send out email alarms (Robin Sommer).

- Writes to a file that fail due are now run-time errors rather than
  fatal internal errors, since often these occur due to the disk
  being full (Robin Sommer).

- Byte-order bug fix for lookup_location() (Robin Sommer).

- BinPAC portability fix for 64-bit machines (Bernhard Ager and Robin Sommer).

- Portability fixes for newer versions of gcc (Jan Gerrit Goebel and
  Robin Sommer).

- Some support for porting to Solaris (Stephan Toggweiler).

- Connection compressor bug fix for source and destination having the
  same IP address, such as when monitoring loopback (Robin Sommer).

- Connection compressor bug fix for connections with multiple SYNs
  (Robin Sommer).

- Bug fix for using already-declared local variables for looping
  over vectors in a "for" loop (Robin Sommer & Vern Paxson).

- Bug fix for not processing truncated UDP packets (Tom Kho and Robin Sommer).

- Bounds-check added to BinPAC-generated code (Tom Kho and Robin Sommer).

- Bug fix for checking whether an IPv6 address is part of a subnet
  (Seth Hall).

- Bug fixes for crashes relating to asynchronous DNS lookups performed
  at start-up (Robin Sommer).  These changes also lowered the timeout
  before assuming failure from 20 seconds down to 5 seconds.

- Portability and const-ness fixes (Kevin Lo and Robin Sommer).

- Suppression of some content-gap complaints when running on traces
  that have been filtered down to only TCP control packets (Robin Sommer).

- Removed unnecessary dependency in notice-action-filters.bro
  that led to errors when loading icmp.bro by itself (Vern Paxson).

- Bug fix for potential infinite loop in client communiation (Robin Sommer).

- Bug fix in reference counting that could eventually lead to roll-over
  (Robin Sommer).

- Bug fix in communication initialization (Robin Sommer).

- Internal documentation fix: timers are specified using absolute time,
  not relative (Robin Sommer).

- Performance improvement for built-in find_all() function when running
  on large strings (Robin Sommer).

- Memory leak fixes (Robin Sommer, Bernhard Ager, Christian Kreibich).

- Bug fix for error recovery when encountering an unknown link layer
  (Bernhard Ager).

- Bug fix for reversing client & server in a connection (Po-Ching Lin).

- Bug fix for packet_contents when capture length exceeds the IP payload
  length due to Ethernet frame padding (Christian Kreibich).

- Bug fix for tcp_packet event erroneously including Ethernet padding
  in its contents (Vern Paxson).

- Bug fix for lookup_connection built-in (Seth Hall).

- Portability nit for libedit tarball (Vern Paxson).

- Broccoli portability fix for NetBSD (Christoph Leuzinger).

- Type-checking for script-level event invocation was completedly broken -
  now fixed (Vern Paxson).

- Portability fixes for different versions of g++/STL (Nicholas Weaver
  and Vern Paxson).

- Fix for dynamic detection of SSL via DPD (Robin Sommer).

- IPv6 portability fix for BinPAC-based DNS analyzer (Vern Paxson).
  Note, more portability work is needed for it.

- Bug fix for bifcl error messages (Vern Paxson).

- Minor bug fix for remote communication, plus some improved communication
  logging (Robin Sommer).

- Bug fix for &printhook (Robin Sommer).

- Bug fix for error message output (Robin Sommer).

- Bug fix for termination cleanup (Robin Sommer).

- Bug fix for some Rlogin corner cases (Robin Sommer & Vern Paxson).

- Bug fix for bifcl generation of "interval" types (Vern Paxson).

- Bug fix for getting connection memory statistics when Bro is
  exiting (Robin Sommer).

- Config fix: --enable-debug now turns off -O2 for gcc (Robin Sommer).

- Bug fixes for "heavy" analysis (Vern Paxson).

- Broccoli bug fixes for types net and port (Robin Sommer).

- Bug fixes for Telnet environment options (Robin Sommer).

- Bug fix for accessing remote peer description (Robin Sommer).

- A fix for the connection compressor generating new_connection too
  late (Robin Sommer).

- Fixes for DAG support, including configuration and multiple
  interfaces (Robin Sommer).

- Bug fix for serializing time-stamps of table entries (Robin Sommer).

- Bug fix for dealing with peer IDs for remote communication (Robin Sommer).

- Bug fix to avoid installing timers when timers have already
  been canceled (Robin Sommer).

- Bug fix for interplay between serializing connections and
  connection compressor (Robin Sommer).

- Memory leak fix for enum's (Robin Sommer).

- Bug fix for files being closed prior to bro_done() (Vern Paxson).

- aux/broccoli/contrib was not included in distribution (Robin Sommer).

- Auto-configuration bug fix for BinPAC (Craig Leres).

- Bug fix for dynamic protocol detection (Robin Sommer).

- A number of configuration fixes for installation and portability
  (Christian Kreibich, Brian Tierney, Robin Sommer, Dan Kopecek).


1.3 Mon Jul 16 22:11:00 PDT 2007

- The Bro manual has been wikified at:

	http://www.bro-ids.org/wiki/index.php/User_Manual

  and this is the format in which it will evolve in the future
  (Christian Kreibich).

- Much more extensive support for SMB, NetBIOS and NCP (Chris Grier).

- The new attribute &priority=n defines the order of execution for handlers
  of the same event (Robin Sommer).  Handlers with higher priority are
  executed first.  n is an integer expression that must evaluate to a
  constant when the script is loaded.

  Example:
          > cat foo.bro
          event bro_init() &priority = -5 { print -5; }
          event bro_init() &priority =  5 { print 5; }
          event bro_init()                { print 0; }	# default priority=0
          > ./bro foo.bro
          5
          0
          -5

  The connection_state_remove() handler in conn.bro now has priority
  -10 and therefore executes after all other handlers for this event.
  This fixes a long-standing problem of sometimes $addl fields not showing
  up in connection summaries.

- The new expressions record(...), table(...), set(...) and vector(...)
  are constructors for the corresponding aggregate types (Vern Paxson).
  For example,

	record($foo = "hi", $bar = -6)

  is the same as the existing constructor

	[$foo = "hi", $bar = -6]

  For tables, sets, and vectors, the "..." values within the ()'s have
  the same syntax as those that you can list in variable initializations.
  For example,

	table([1, T] = "black", [4, F] = "red")

  returns a table of type "table[count, bool] of string".

	set(4, 3, -1)

  is a value of type "set[int]".

- You can associate attributes with table() and set() constructors
  (Robin Sommer).  For example:

         local s = set(1.2.3.4) &read_expire = 5 secs;

  associates a 5-second read expiration with the set assigned to s.

- Bro now explicitly supports port numbers reflecting a transport protocol
  type of "unknown" (Christian Kreibich).  Currently, this means "not TCP,
  UDP or ICMP".  The numerical value of such a port is the IP protocol,
  so ranges from 0..255.  For example:

    global p: port = 0/unknown;

    print fmt("%s", p);
    print fmt("p is TCP? %s", get_port_transport_proto(p) == tcp);
    print fmt("p is unknown? %s",
		get_port_transport_proto(p) == unknown_transport);

  yields

    0/unknown
    p is TCP? F
    p is unknown? T

  In comparisons of different protocol types, the following holds:
  unknown < TCP < UDP < ICMP.

- If your system supports "GeoIP" (see http://www.maxmind.com/app/geolitecity
  for a corresponding city database), then the new script function

	  lookup_location(a: addr): geo_location

  returns a record of geographic information associated with an address
  (Seth Hall).  The geo_location record has $country_code, $region and
  $city fields.  If no information is available, each of these will be
  set to empty strings.

  If Bro hasn't been configured with GeoIP support, or if the address is
  IPv6 that cannot be directly converted to IPv4, then Bro produces a
  run-time error and likewise returns empty strings.

- Signature-matching on HTTP components now processes the URI with
  escape sequences expanded (Robin Sommer).  Ideally, there would be
  two signature keywords, one for decoded URIs (corresponding to this
  case) and one that allows matching against the URI as originally
  transmitted.

- The connection compressor is no longer considered experimental, and
  is used by default (Robin Sommer).

- The new function lookup_hostname(host: string): addr_set asychronously
  looks up the IPv4 address(es) of the given host via DNS (Robin Sommer).
  Like lookup_addr(), this function can only be used within a "when"
  statement.

- The new built-in

	raw_bytes_to_v4_addr(s: string): addr

  takes a string that points to at least 4 bytes, and returns an address
  corresponding to interpreting these as being an IPv4 address in network
  order (Vern Paxson; suggested by Mike Dopheide).

- Trace-rewriting support for DNS, SMB (Chris Grier).

- The new script function find_all(str: string, re: pattern): string_set
  returns a string_set giving all occurrences of the pattern "re" in
  the string "str" (Robin Sommer).  (Note that string_set's are unordered.)

- The new policy script save-peer-status.bro generates a log
  to peer_status.$BRO_LOG_SUFFIX of updates received from
  communication peers (Robin Sommer).

- The policy script print-filter.bro now includes two (scoped) variables,
  terminate_bro and to_file, which control whether to exit after printing
  the filter (default T) and whether to write to the log file
  pcap_filter.$BRO_LOG_SUFFIX or (default) to stdout (Robin Sommer).

- The new script variable check_for_unused_event_handlers controls whether
  Bro checks for unused event handlers (Robin Sommer).  It defaults to T,
  which was the past behavior (always report).

- Bro now terminates if the only pending activity is future timers
  (Robin Sommer).  It used to wait for those timers to expire, but this
  can cause fundamental problems if the timers are associated with table
  management (since these might never completely drain).

- Tables and sets inside of records are now initialized to empty
  values rather than uninitialized (Vern Paxson).

- A new variable allow_services_from (in hot.bro) complements the
  existing allow_service_to variable (Brian Tierney).  It specifies
  that access to the given service from the given originator is
  allowed.

- global_sizes() no longer reports internal variables (Robin Sommer).

- The IRC analyzer is now activated if any of the (many) IRC event
  handlers are defined (Robin Sommer).

- The default value for tcp_close_delay is now 5 sec rather than 0 sec
  (Robin Sommer).  This prevents some spurious connection events.

- Improved logic for dealing with "reversed" connections such
  as backscatter (Vern Paxson).

- You can now left-justify fields when using fmt() with "%-" like
  in sprintf (Christian Kreibich).

- Updates to DNS query types (Larry Leviton).

- Added mechanism to http-header.bro to skip printing some HTTP headers
  (Larry Leviton).

- The IrcHotWord notice now sets the associated connection (Robin Sommer).

- If a notice has a tag, it's no longer overridden (Robin Sommer).

- ServerFound notices now set the port field (Robin Sommer).

- The built-in lookup_ID() now returns the string "<unknown id>" if the
  ID does not exist, rather than a run-time error (Robin Sommer).

- The new tuning option ProtocolDetector::suppress_servers specifies a
  set of analyzers for which Bro generates ServerFound notices, but not
  ProtocolFound (Robin Sommer).  This both reduces log file size and
  conserves memory.

- A new notice_action_filter, tally_notice_type_and_ignore, works the same
  as tally_notice_type but returns IGNORE (Robin Sommer)

- Setting summary_interval == 0 disables the creation of irc-bots.summary.log
  (Robin Sommer).

- If you @load foo and a directory "foo" is in your path, Bro no longer
  tries to load it (Robin Sommer).

- A number of BinPAC fixes and enhancements (Ruoming Pang, Chris Grier
  and Vern Paxson).

- BinPAC now resides in aux/binpac rather than src/binpac (Ruoming Pang
  and Christian Kreibich).  This reflects a decoupling of it from Bro so
  that it can be used to generate protocol analyzers for other projects too.

- Removed example Inktomi entries from skip_scan_sources initialization,
  since they no longer exist (Vern Paxson).

- The variable make notice_once_per_orig_tally_interval is now
  redef'able (Brian Tierney).

- SIGPROF to the communication child process now logs resource stats to
  remote.log (Matthias Vallentin).

- The new built-in getpid(): count returns Bro's process ID (Robin Sommer).

- Patterns for detecting IRC-based bots updated (Robin Sommer).

- irc-bot-syslog now logs just bots, not all IRC client/servers (Robin Sommer).

- The new variable suppress_notice_actions in notice.bro suppresses
  notice_actions events for selected notice types (Robin Sommer).

- Files opened during operation now rotate just like those opened at
  startup (Robin Sommer).

- ResourceStats now also logs elapsed time and the reported number of
  packets-on-the-link (Mark Dedlow).

- Printing a "file" value now produces its name (Robin Sommer).

- Removed deliberate truncation of payload in port 80 FIN packets
  (Vern Paxson).

- remote.log now includes received peer_descriptions (Robin Sommer).

- Significant POP3 analyzer speed-ups (Vern Paxson).

- Updated README (Vern Paxson).

- Fix for "@load a" followed by "@load a.bro" not loading the same file
  twice (Robin Sommer).

- Bug fixes for propagating state operations to uninitialized variables
  and for spurious state inconsistency messags (Robin Sommer).

- Bug fix for sending final sync-points during pseudo-realtime mode
  (Robin Sommer).

- Fix for possible buffer overflow (Christian Kreibich).

- Bug fix for spurious end-of-file's during inter-Bro communication
  (Robin Sommer).

- Bug fix for dpd_match_only_beginning=F (Robin Sommer).

- Bug fix for updating timestamps (Christian Kreibich).

- Bug fix for skipping ADU processing in adu.bro (Christian Kreibich
  and Zhichun Li).

- Fix for ICMPs that carry ICMP headers (or non-TCP/UDP/ICMP headers)
  within them (Vern Paxson).

- Fix for files being rotated after the timer queue has been deleted
  (Vern Paxson).

- Bug fix for signature-matching with IPv6 subnets (Vern Paxson).

- Bug fix for connection compressor setting connection origin (Robin Sommer).

- Bug fix for interconn.bro when processing peculiar connections (Vern Paxson).

- Fix for off-by-one buffer in sscanf call (Christian Kreibich).

- Fixed inefficiency/warning flagged by g++ (Vern Paxson).

- Bug fix for NUL string termination in SMB processing (Zhichun Li).

- Fix for over-ref'ing of file Val's (Vern Paxson).

- Fixes for some g++ warnings (Christian Kreibich, Vern Paxson).

- gcc 3.4.2 portability fixes (Robin Sommer).

- Minor build fixes for Broccoli, including a version bump to match that
  of Bro.  See aux/broccoli/ChangeLog for details.

- distcheck fixes (Christian Kreibich).

- Configuration portability fixes (Matthias Vallentin, Jean-philippe Luiggi).

- OpenBSD portability fixes (Jean-philippe Luiggi, Christian Kreibich).


1.2.1 Mon Dec 11 16:22:58 PST 2006

- Fixed delayed triggering of new_connection events when using the
  connection compressor.

- Fixed tracking of first packet in TCP analyzer. (Reported by Guohan Lu)

- The syslog built-in got lost during some previous merge.

- Fixed crash if local variable is given as timeout value for table.
  (Reported by Mike Wood.)

- Fixed using "time" values as table indices.

- Added ssh to default brolite DPD configuration.

- Fixed catching up to real-time in case of lull.

- Fixed Broccoli "BRO_DATA_FORMAT_VERSION" to match version in Bro.

- Fixed Makefile problem in doc directory.

- Fixed Makefile dependency problem in binpac directory.

- Added Linux tuning to brolite install script.

- Modified Makefile to include broccoli/contrib.

- Adding missing initialization to remote serializer.

- Minor documentation updates for reference manual and Broccoli.


1.2 Tue Oct 17 12:09:49 PDT 2006

- Bro now supports DPD, dynamic protocol detection (Robin Sommer, Holger
  Dreger, and Michael Mai).  With DPD, Bro can analyze protocols regardless
  of what port numbers they use: it infers the protocol based on which
  application analyzers can parse it without error.  Adding this functionality
  involved extensive changes to Bro's internals, but also now enables
  multiple Bro analyzers to work on the same connection, either concurrently
  or one nested inside the other (we have not taken much advantage of this
  latter capability yet, but see the FTP events discussed below).

  There are a number of new policy scripts, events, and variables associated
  with DPD processing, as follows.

  Scripts:

	You activate DPD by @load'ing dpd.bro.  It in turn instructs Bro
	to load the signature file policy/sigs/dpd.sig.  Note that Bro
	uses signatures to expedite deciding which analyzers to try on
	a given connection; it does *not* simply use the signatures to
	make the determination of which protocol is in use, as this is
	insufficiently robust.  (At this point, Bro provides signatures
	for FTP, IRC, HTTP, SMTP, and SSH.  In the future we plan to add
	other protocols.)

	Along with dpd.bro, you need to @load detect-protocols.bro or
	detect-protocols-http.bro. The former enables general detection
	of application-layer protocols, while the latter does further
	inspection of HTTP sessions to characterize applications running
	on top of HTTP such as Gnutella or SOAP.   (Loading dpd.bro
	is separate from loading one of these scripts because in principle
	Bro could use a different means than signatures to activate
	the analyzers, although currently it does not.)

	If you @load dyn-disable.bro, then once an analyzer determines
	that it does not match a given connection, it is deactivated
	(and a Notice is generated).  Otherwise, it still proceeds to try
	its best to analyze the connection (to possibly be more robust
	against evasion).

	The scripts dce.bro and smb.bro enable DPD for the Windows DCE and
	SMB protocols, respectively.  (Note that analysis of these protocols
	is undergoing a major expansion, not yet complete.)

  Events:

	event protocol_confirmation(c: connection, atype: count, aid: count)
		Generated when the given connection has been confirmed as
		conforming with the application type (protocol) specified
		by atype. aid is a globally unique analyzer ID that identifies
		a particular analyzer instance.

		The values for atype are symbolic names associated with
		each of Bro's analyzers, such as ANALYZER_IRC.  See the
		initialization at the beginning of Analyzer.cc for the
		full set of names.

		The function analyzer_name(atype: count): string translates
		these symbolic names into text.  For example,

			analyzer_name(ANALYZER_IRC)

		yields "IRC".

	event protocol_violation(c: connection, atype: count, aid: count,
				reason: string)
		Generated when the given connection has been found to
		violate the protocol of the given application type, with
		"reason" giving details.

  Variables:

	dpd_buffer_size: count (default 1024)
		Specifies how much pending data Bro keeps for connections
		that have not been classified yet.  Once this fills, the
		data is deleted, though classification can still continue
		(see below).

	dpd_match_only_beginning: bool (default T)
		If set, specifies that Bro should stop signature matching
		if it has processed dpd_buffer_size bytes.

	dpd_ignore_ports: bool (default F)
		If set, then Bro does not take into consideration the port
		numbers associated with connections when attempting to
		classify them (which can otherwise help the process in
		some cases).

	dpd_reassemble_first_packets: bool (default T)
		If set, then Bro does TCP stream reassembly before applying
		signature-matching to detect protocols.

	likely_server_ports: set[port]
		Specifies a list of ports that Bro will consider as likely
		used by servers.  For example, if Bro sees a connection
		that has already been established (so it does not know
		which side sent the initial SYN), and one side uses a port
		in this set, then it will assume that that side is the
		server (connection responder).  The set is empty unless
		you populate it or @load server-ports.bro, which specifies
		a large number of values.

	dpd_config: table[AnalyzerTag] of dpd_protocol_config
		Specifies the DPD configuration associated with each tag.
		The type dpd_protocol_config is simply:

			type dpd_protocol_config: record {
				 ports: set[port] &optional;
			};

		i.e., an optional $ports field specifying a set of ports
		associatd with the tag.  For example, ftp.bro now includes
		the equivalent of:

			redef dpd_config += {
				[ANALYZER_FTP] = [$ports = 21/tcp]
			};

  Functions:

	The function

		expect_connection(orig: addr, resp: addr, resp_p: port,
					analyzer: count, tout: interval)

	is called to alert Bro that a new connection is expected, initiated
	by orig to a server running on resp's port resp_p (note: orig's port
	is not specified) which will correspond to the specified analyzer
	(e.g., "FILE", which is used to analyze files transferred by FTP -
	see next item).  "tout" is a timeout to associate with the waiting.

	The function

		function disable_analyzer(cid: conn_id, aid: count)

	instructs Bro to disable the analyzer that generated the current
	event, assuming the analyzer is associated with the given connection
	ID.  This is used by the dyn-disable.bro script discussed above.

- A much more complete BinPAC compiler, along with new HTTP, DNS, and
  RPC/Portmap analyzers in binpac (Ruoming Pang). The flag "--use-binpac"
  activates the BinPAC-based analyzers (currently for HTTP and DNS).
  See www.cs.princeton.edu/~rpang/binpac-paper.pdf for a description of
  BinPAC, and let Ruoming know if you are interested in using BinPAC to build
  new analyzers.

- A new type of analyzer, FILE, analyzes the contents of a connection as
  though it were a data file (Robin Sommer).  Currently, it can generate
  two events:

	event file_transferred(c: connection, prefix: string, descr: string,
				mime_type: string)
		Indicates that the connection transferred a file. "prefix"
		is the beginning of the file's data; "descr" and "mime_type"
		are indicators of the file's type, as reported by the
		"libmagic" library.

		descr/mime_type are only set if Bro is configured on a
		system that includes the "libmagic" library.

	event file_virus(c: connection, virname: string)
		Indicates the connection transferred an executable
		corresponding to a known virus of the given name.

		This functionality is only available if Bro is configured
		on a system that includes the "libclamav" library.

  Note, this analyzer is enabled via a call to expect_connection by
  the FTP analyzer.

- New events relating to IRC analysis (Robin Sommer):

	event irc_client(c: connection, prefix: string, data: string)
		Generated upon seing a client message sent over the given
		IRC connection.  "prefix" is the command's prefix as defined
		by the IRC protocol.  It is used by servers to indicate the
		true origin of the message; it may be empty.  "data" contains
		the message.

	event irc_server(c: connection, prefix: string, data: string)
		Same for server messages.

	event irc_user_message(c: connection, user: string, host: string,
				server: string, real_name: string)
		Generated upon seeing an IRC "USER" command.

	event irc_password_message(c: connection, password: string)
		Generated upon seeing an IRC "PASS" command.

	event irc_channel_topic(c: connection, channel: string, topic: string)
		Generated upon seeing an IRC server reply that includes
		the channel topic.

	event irc_global_users(c: connection, prefix: string, msg: string)
		Generated upon seeing an IRC server reply that includes
		a count of the number of IRC users.

- The new experimental script irc-bot.bro tracks IRC-based bots (Robin Sommer).
  The accompanying script irc-bot-syslog.bro syslog's the state of the
  bot analysis every IrcBot::summary_interval seconds (default 1 minute).

- The new script proxy.bro looks for open Web proxies by matching incoming
  requests to a server with outgoing requests it makes (Robin Sommer).  It
  generates HTTPProxyFound Notices when it finds one.

- Changes to notices.bro (Robin Sommer):

	- notice_policy_item's now have a default $result of
	  NOTICE_FILE and a default $priority of 1.

	- The new notice_action_filter, notice_alarm_per_orig, alarms
	  on the first NoticeType from a specific source.  Subsequent
	  instances are tallied.

	- notice_action_filters now reside in the new script
	  notice-action-filter.bro (automatically loaded by notice.bro).

	- The notice actions NOTICE_ALARM_PER_CONN, NOTICE_ALARM_PER_ORIG,
	  and NOTICE_ALARM_ONCE have been removed, as they were never
	  actually implemented.

	- If the notice_policy returns IGNORE or FILE, the action_filters
	  filters are no longer consulted.

- A new attribute for tables and sets, &mergeable, changes the semantics
  of assignments, as follows (Robin Sommer).  Given two &mergeable tables/sets
  A and B, an assignment "A = B" becomes actually a join "A = A \cup B"
  (i.e., union).  The envisoned use is to help avoid race conditions
  when doing remote state synchronization.

- The semantics of &synchronized expire_funcs has changed (Robin Sommer).
  Now, when a table entry is expired and the operation is propagated to a
  a peer, the peer will call its expire_function.

- TRW analysis now skips UDP traffic because it currently treats
  all UDP connections as failures (Robin Sommer).

- trw.bro has been split into trw-impl.bro (the algorithm) and
  trw.bro (which simply activates the analysis), to facilitate writing
  scripts that have hooks into TRW analysis but don't presume it's
  active (Robin Sommer).

- The option report_remote_notices in remote.bro has been replaced
  by a new script you include, remote-report-notices.bro (Robin Sommer).

- The new function connect_peer() explicitly connects to a remote host
  (Robin Sommer).

- The new script remote-send-id.bro sends the current value of an ID
  to a remote Bro and then terminates processing (Robin Sommer).  It's
  intended for use from the command-line, as in

	bro -e "redef dst="<dst>" id="<name-of-id>" <scripts> remote-send-id

  The other scripts must set up the connection. <dst> is an index into
  Remote::destinations corresponding to the destination.

- New built-ins {suspend,resume}_state_updates() can be called to
  temporarily avoid propagating updates to &sync'ed values (Robin Sommer).
  This can avoid duplicated activity.

- The new function terminate_communication() instructs Bro to end its
  communication with remote peers (Robin Sommer).

- The new event remote_state_access_performed is raised when remote state
  access has been performed (Robin Sommer).  This is primarily for debugging.

- The log() built-in has been renamed to ln() to avoid conflict (Vern Paxson).

- bifcl now generates event generation wrapper functions from event.bif
  (Ruoming Pang).  For example, to generate event http_reply, currently
  one writes:

		val_list* vl = new val_list;
		vl->append(BuildConnVal());
		vl->append(new StringVal(fmt("%.1f", reply_version)));
		vl->append(new Val(reply_code, TYPE_COUNT));
		if ( reply_reason_phrase )
			vl->append(reply_reason_phrase);
		else
			vl->append(new StringVal("<empty>"));
		ConnectionEvent(http_reply, vl);

  In the future, one will be able to just call bro_event_http_reply(), and
  the code generated by bifcl looks like:

	void bro_event_http_reply(Connection* c, StringVal* version,
					bro_uint_t code, StringVal* reason)
		{
		val_list* vl = new val_list;

		vl->append(c->BuildConnVal());
		vl->append(version);
		vl->append(new Val(code, TYPE_COUNT));
		vl->append(reason);

		mgr.QueueEvent(http_reply, vl, SOURCE_LOCAL, c);
		}

  Accompanying this change is a semantic shift to types "string" and "port"
  in .bif files.  They used to be translated to C++ types BroString* and
  uint32, respectively.  Now they are translated to StringVal* and PortVal*.
  The functions in bro.bif are changed accordingly, and please be aware
  of this change when you write built-in functions in future.

  Also for this change, the parameter 'new' for rsh_request has been renamed
  'new_session', as 'new' is a reserved word for C++.

- Some ICMP "connections" now have services identified ("icmp-echo",
  "icmp-unreach") rather than just listing the service as "other"
  (Ruoming Pang).

- The new option remote_trace_sync_interval specifies an interval after
  which each Bro will stop processing its trace and wait for all others
  to signal that they have reached the same time (Robin Sommer).  The
  intent is support for operating Bro in a distributed cluster fashion
  (and in particular for debugging such clusters when running off-line
  on traces).

  This option only works in pseudo-realtime mode, and requires the new
  global remote_trace_sync_peers to give the total number of remote peers
  (not including self).  Signaling is done via a new communication message
  type.

- Extensions for DNS transformation/anonymization, including introduction
  of trace transformation for protocols other than TCP (Jason Lee).
  Not yet fully developed/debugged.

- Extensions for HTTP transformation/anonymization (Martin Casado).
  Not yet fully developed/debugged.

- The $conn field is now included in HTTPProxyFound notices (Robin Sommer).

- Changed service inference algorithm to favor lower-numbered
  likely-servers over higher-numbered ones (Vern Paxson).

- In pseudo-realtime mode, Bro now uses real-time for deciding which
  peer should send state (Robin Sommer).

- Time synchronization for Bro's running on traces in pseudo-realtime mode
  added (Robin Sommer).

- Avoidance of false content gaps improved when sorting packets with
  out-of-order timestamps (Ruoming Pang).

- Packets from the packet sorter are now more robustly drained upon
  termination of input (Ruoming Pang).

- Documentation for deep-copy updated (Christian Kreibich).

- Nasty fragment reassembly bug fixed (Vern Paxson).

- Serious bugs in EDNS0 processing fixed (Vern Paxson).

- Fixed significant misfeature of interconn.bro that stopped all processing
  of a connection once it makes a detection (Vern Paxson).

- Fixes for &read_expire operation across synchronizes tables (Robin Sommer).

- Fixes for multiple peers exchanging initial &sync state simultaneously
  (Robin Sommer).

- Improvements to graceful termination of Bro when communicating with
  remote peers (Robin Sommer).

- Fix for ICMP analyzer not always generating icmp_sent events
  (Robin Sommer).  This appears to still need some work, as now
  it generates redundant events.

- Fix for initial exchange of &sync state which could lead to
  referencing unknown IDs (Robin Sommer).

- Fix to scan detection for differing semantics of connection compressor
  vs. non-compressor (Robin Sommer).

- Bug fix for distinguishing regular expression matches of length 0 from
  those of length 1 (Ruoming Pang).

- Fix for SSH version parsing in the presence of content gaps (Robin Sommer).

- Bug fix for IRC that could lead to crashes (Robin Sommer).

- Bug fix to refrain from adding new timers when a connection has
  already been removed from the connection table (Robin Sommer).

- Bug fix for packet_contents not including the transport-layer header
  (Robin Sommer).

- Some memory leaks fixed (Robin Sommer).

- A bunch of portability and distribution problems fixed (Christian
  Kreibich, Robin Sommer, Vern Paxson).


1.1 Mon May 15 10:50:33 PDT 2006

- Bro now supports a "when" statement for taking action upon something
  becoming true asynchronously (Robin Sommer).  This provides a powerful
  new mechanism with numerous applications.

  Syntax:

	when '(' <expr> ')' <stmt> [timeout <interval> '{ <stmts> '}']

  where the first <stmt> can be a single statement or a block enclosed
  in {}'s, but the set associated with "timeout" must be enclosed in
  {}'s (to reduce ambiguities in Bro's grammar).

  Bro executes the first statement when <expr> becomes true. If you give
  a timeout and the condition has not been satisfied before it expires, Bro
  executes the second statement instead.

  A simple example:

	global t: table[addr] of count;
	event connection_established(c: connection)
	    {
	    local orig = c$id$orig_h;
	    if ( orig !in t )
		{
		t[orig] = 1;

		when ( t[orig] == 5 )
		    print fmt("%s has established 5 connections", orig);
		timeout 1 hr
		    {
		    print fmt("%s has NOT established 5 connections", orig);
		    delete t[orig];
		    }
		}
	    else
		++t[orig];
	    }

  Notes:
	- The condition may be evaluated more than once, and at arbitrary
	  times.

	- When the when-body is executed, the condition is guaranteed to be
	  still satisfied.

	- Expression reevaluation is primarily triggered by modifications
	  to globals.  However, reevaluations do not take place immediately
	  but potentially at a later point.  This means that if we change a
	  global to a value which would execute the trigger but then change
	  it back, the change may go unnoticed.

	- Inside the condition you may introduce new locals.  For example,

	    when ( (local x = foo()) && x == 42 ) ...

	  Such an assignment always yields true as its expression value
	  (but the assignment might be delayed, for example if foo() is
	  a delayed function call - see below).

  Delaying function calls
  =======================

  Functions called inside the condition of a when-clause may delay their
  results until they're ready. This works for both script-level and built-in
  functions.

  For script-level functions, there is a new construct, "return <when-stmt>",
  to delay a function's result. When used, the function returns at the
  time the when-stmt's condition becomes true, and it yields the value
  that the when-stmt's body then returns. Toy example:

      global X: table[string] of count;

      function a() : count
	    {
	    # This delays until condition becomes true.
	    return when ( "a" in X )
		  {
		  return X["a"];
		  }
	    timeout 5 min
		  {
		  return 0;
		  }
	    }

      event bro_init()
	    {
	    # Installs a trigger which fires if a() returns 42.
	    when ( a() == 42 ) { print "Yippie!"; }

	    X["a"] = 42;
	    }

  There's also a new built-in function which can delay

	lookup_addr(host: addr)

  performs asynchronous DNS address->hostname lookups. Example:

	    local h; addr;
	    [...]
	    when (local name = lookup_addr(h)) { print h, name; }

  See the function gen_hot_notice_with_hostnames() in conn.bro for
  a more worked-out example of using the "when" clause to translate the
  local address in SensitiveConnection notices to a hostname (contributed
  by Brian Tierney).  This functionality is activated by redef'ing
  xlate_hot_local_addr to T.

  Here is the full evaluation model of a when's condition:

       - The condition may be evaluated more than once, at arbitrary times.

       - It is always fully evaluated, no matter whether some former
	 evaluation has been suspended by a delaying function call.

       - All function calls which do not delay are always *fully* executed
	 each time the condition is evaluated.

       - Function calls which delay are only executed *once*; their result is
	 cached and re-used in the case the condition is evaluated again.

       - The condition is guaranteed to be true when the body is executed
	 (potentially using cached function results)

- By default Bro now uses a configuration similar to what used to be
  activated using reduce-memory.bro, along with some additional state
  timeouts that are new (Robin Sommer and Vern Paxson).  This allows for
  better state management out-of-the-box, at the cost of some precision
  of analysis and resilience to evasion.  In particular, the intent is to
  move towards being able to run Bro continuously without inexorably growing
  the amount of memory used until exhaustion.

  You can access a configuration similar to the previous default state
  management settings by loading heavy-analysis.bro.  It turns on a
  load-prefix of "heavy", so when you load XXX.bro, a file heavy.XXX.bro
  will also be automatically loaded if present.  Note that, as was the
  case for reduce-memory, you need to load heavy-analysis prior to other
  files for it to have effect.

- The new module clear-passwords.bro monitors login/FTP/IRC/POP traffic
  for cleartext passwords (Jason Lee).

- The new script service-probe.bro looks for remote hosts that repeatedly
  connect to the same service on local hosts (for a configurable set of
  services and connection sizes) in order to detect brute-forcing attacks
  such as password-guessing (Jim Mellander).

- A new ARP analyzer generates three events:

	event arp_request(mac_src: string, mac_dst: string,
			SPA: addr, SHA: string, TPA: addr, THA: string);

	event arp_reply(mac_src: string, mac_dst: string,
			SPA: addr, SHA: string, TPA: addr, THA: string);

	event bad_arp(SPA: addr, SHA: string, TPA: addr, THA: string,
			explanation: string);

  with a corresponding policy script arp.bro (Chema Gonzalez and Vern Paxson).
  It writes logs to arp.$BRO_LOG_SUFFIX.  It has not been tested much yet.

- Bro Lite changes (Jason Lee):
	- default user for is now user 'bro'
	- now uses the correct sysctl on FreeBSD 6
	- now uses the correct Perl path if site-report.pl not installed
	  into '/usr/local/bro'
	- no longer prompts to encrypt email unless you pick to email reports

- The default Bro Lite install now only checkpoints Bro once a week
  (Brian Tierney).

- Implicit Bro file extensions (such as .bro for policy scripts and .sig
  for signatures) are now searched for first rather than only if the
  non-extension-version of the file doesn't exist (Vern Paxson).  For
  example, running "bro -r trace mt" now first searches $BROPATH for
  "mt.bro" before searching for "mt", whereas it used to do these in
  the other order.

- There's now a simpler mechanism for redef'ing variables on the command-line
  (Christian Kreibich).  Any command line arguments of the form <var>=<val>
  are now expanded into policy code of the form "redef var=val;", where
  <val> is wrapped in quotation marks if the value appears to be a string
  and doesn't have quotation marks already.  This works with strings with
  whitespace such as foo="Hello World"; however, note that it means you
  can't use the mechanism to redef an enum value.

- The Bro distribution now includes (and builds by default) Christian
  Kreibich's Broccoli library (Bro C Client Library), which enables programs
  to communicate with running Bro's (Christian Kreibich and Jason Lee).
  Configure with --disable-broccoli to turn this off.

- Built-in functions log(x: double): double and exp(x: double): double
  which do natural logarithms and their inverses (Jaeyeon Jung).

- The new built-in function gethostname() returns the local host's name
  (Jason Lee & Robin Sommer).

- The new built-in function reading_traces() returns true if Bro
  is reading trace files (Robin Sommer).

- The new built-ins suspend_processing() and continue_processing() provide
  script-level control for instructing the event engine to stop or resume
  processing packets (Robin Sommer).  This is useful for coordinating
  simultaneous processing by multiple Bro's.

- Email notices are now by default sent via /bin/mail, with "[Bro Alarm]"
  in the subject.

- redef'ing a function now replaces the existing body rather than
  supplementing it (Robin Sommer), which was a bug.

- You can now configure Bro to process encapsulated IP packets either
  by setting, as before, a fixed encap_hdr_size (for VLANs), or setting
  parse_udp_tunnels to T (Ruoming Pang).  For the latter, you specify a
  UDP tunnel port using udp_tunnel_port (the previous variable "tunnel_port"
  has gone away); or you can leave it set to its default of 0/udp, in which
  case Bro will look for IP encapsulated in UDP packets on any port.

- Added a simple form of profiling based on sampling the work done
  per-packet (Vern Paxson).  The event engine generates a

	event load_sample(samples: load_sample_info, CPU: interval, dmem: int)

  event every load_sample_freq packets (roughly; it's randomized), where
  load_sample_freq defaults to 20.  "samples" is simply a set[string]; it
  contains the names of the functions, event handlers, and their source
  files that were accessed during the processing of the sampled packet,
  along with an estimate of the CPU cost of processing the packet and
  (currently broken) memory allocated/freed.

- Bro now includes experimental support for Endace DAG cards (Gregor Maier
  and Robin Sommer).  To activate, configure with

	--with-DAG=/path/to/dagtool/installation

  and use "dag0" as the network interface. You may need to configure the
  card with the dagtools first. In general, if dagsnap works, Bro should
  work as well.

- Log rotation has changed in a number of ways (Mark Dedlow & Robin Sommer):

	  * The new variable log_rotate_base_time: string, if defined,
	    specifies that logs should be rotated at log_rotate_base_time +
	    i * rotate_interval intervals. Format is as a string in
	    24-hour time, "%H:%M", e.g, "12:00".  This format may change
	    in the future to instead be a Bro time type.

	  * RotateLogs::date_format can be redefined to change format of
	    timestamps in rotated files.

	  * RotateLogs::build_name() can be redefined to implement an
	    arbitrary naming scheme for rotated files.

  Note, this code has not been extensively tested.

- Bro now by default builds a version of malloc bundled with its
  distribution (Vern Paxson & Brian Tierney).

- The syntax for the clone operator now looks like a function call,
  "copy(x)" (Vern Paxson).

- The new flag DNS::logging (default F), if T, disables generation of
  dns.log (which is often uninteresting and very large), though it
  still performs analysis leading to NOTICEs (Robin Sommer).

- A new global, hostile_domain_list, has been added to dns.bro which
  lists domains to be flagged if A or MX records are queried (Scott Campbell).

- Added globals dns_skip_all_{auth,addl} to skip all DNS AUTH/ADDL processing
  (Vern Paxson).  Skipping these is on (true) by default, because such
  processing is quite expensive.

- backdoor.bro now turns off by default some detectors that from experience
  have too many false positives, or (such as for HTTP) too many uninteresting
  true positives (Brian Tierney).  In addition:

	- the module now generates a BackdoorFound notice for each backdoor

	- the new variable dump_backdoor_packets (default F) if set causes
	  the packet that triggered the backdoor detection to be written to
	  backdoor-packets/<tag>:<time>

	- the new variable backdoor_ignore_host_port_pairs is a set[addr, port]
	  specify host/port combinations to ignore

	- 587/tcp is now recognized as another SMTP port, and 7000/tcp as
	  a popular IRC port ignored by default

	- brolite-backdoor.bro is a sample of using backdoor.bro

- A bunch of enhancements and fixes for the IRC backdoor detector
  (Vern Paxson).

- The cf utility in aux/cf/ now gets the format to use (unless you specify
  -f fmt) from $CFTIMEFMT in the environment.  You can now specify -f
  without a format to revert to the default format.  This change also
  includes a significant performance improvement when processing large
  files (Mark Dedlow and Craig Leres).

- Cleanups for brolite.bro and brolite-backdoor.bro (Brian Tierney).
  brolite.bro now uses rotate-logs by default.

- backdoor.bro now enables analysis of partial connections (Vern Paxson).

- brolite config cleanup: removed smtp.bro from default, increased
  max_timer_expires, changed default BROPATH to look at site dir
  first (Brian Tierney).

- The reference manual has been updated for the terminology changes
  of log -> alarm, alert -> notice, and rule -> signature (Vern Paxson).
  Some vestiges of the older terminology remain, in part because they're
  still present in some facets of Bro.

- The new script function get_current_packet(): pcap_packet returns
  the current packet as a "pcap_packet" record with fields $ts_sec,
  $ts_usec, $caplen, $len (all of type count) and $data (a string)
  reflecting the corresponding libpcap values (Christian Kreibich).
  You can write this packet to a dump file using the new function
  dump_packet(pkt: pcap_packet, file_name: string): bool, which writes
  (or appends) the packet to a file of the given name, returning T
  on success and F on error.

- The new fmt() specifier 'T'  converts values of type "time" to ISO
  format timestamps, analogous to how 'D' does this for ISO dates
  (Mark Dedlow).  fmt("%T", <time>) is equivalent to
  fmt("%s", strftime("%F-%T.%N", <time>)), except that strftime
  does not (yet) offer "%N" for nanoseconds (but see 'date +%F-%T.%N').

- The new %S format for fmt() inserts a "raw" version of the given string -
  that is, embedded NULs, control characters, etc., are present without
  any escaping (Christian Kreibich).

- Zero-padding and field widths now work for all fmt() formats rather than
  just %e/%f/%g (Christian Kreibich).   For example, you can now say:

	local filename = fmt("log-%04.txt", ++counter);

  and get logfiles log-0001.txt, log-0002.txt, ..., log-0999.txt, etc.

- The 'x' format specifier now supports values of type "addr", converting
  them t hex (Mark Dedlow).  For example,

	  fmt("str=%s hex=%x", 1.2.3.4, 1.2.3.4)

  produces

	str=1.2.3.4 hex=01020304

  The field designation is either %08x (if compiled for IPv4 only) or
  %08x%08x%08x%08x (if compiled with IPv6 support).

- firewall.bro has been extended to support multiple independent
  rule-sets (by calling begin() for the start of the next one),
  specifying sets of addresses, being FTP-aware, and with a more
  streamlined Notice message (Robin Sommer).

- The HTTP script variables maintain_http_sessions and http_sessions
  are now exported so they can be redefined or, for the latter, have
  timeouts added/adjusted (Robin Sommer).

- You can load the new policy script log-append.bro to change Bro's
  behavior so that when it runs appends to existing log files rather
  than overwriting them (Mark Dedlow).

- New &disable_print_hook attribute for files (Robin Sommer).  If set,
  print statements to the file don't trigger the print_hook event.  This
  is useful to keep the output of certain files from being propagated to
  peers.

- You can now associate "classes" with remote peers (Robin Sommer).  When
  connecting, a node may send a specific class to which it considers itself
  belonging. The accepting side can then tune its configuration based on
  the received class.

  This is primarily for the having multiple unrelated Broccolis running on the
  same host, all connecting to the same remote Bro (e.g., sshd and syslog
  sensors).

  To use this, on the Bro side the record Remote::Destination now has a
  field "class: string" (default: unset).  If set, the given config entry
  only applies for connecting remote peers that send the given class.
  If it is set and we're connecting to another peer, we propagate the class.

  Example:

      On the listening Bro:

	    redef Remote::destinations += {
		["peer-1"] =
			[$host = 127.0.0.1, $class="ftp", $events = /ftp.*/],
	        ["peer-2"] =
			[$host = 127.0.0.1, $class="http", $events = /http.*/]
	    };

      On peer 1:

	    redef Remote::destinations += {
		  ["master"] =
			[$host = 127.0.0.1, $class="ftp",
			 $events = /.*/, $connect=T]
	    };

      On peer 2:

	    redef Remote::destinations += {
		  ["master"] =
			[$host = 127.0.0.1, $class="http",
			 $events = /.*/, $connect=T]
	    };

  All of these may run on the same host.

- A bunch of changes to adu.bro (Christian Kreibich):

	- New ADU_MAX_DEPTH limits depth (at ADU granularity) into a
	  flow up to which ADUs are reported.

	- Handles UDP.

	- New event adu_done(c: connection) signals that no further ADUs
	  will be delivered for a connection.  This is useful since adu.bro
	  relies on event connection_state_remove() to remove state, and
	  if a policy using adu.bro likewise uses this event type then
	  event sequencing can cause adu_tx/rx events to occur after
	  connection_state_remove() has been processed.

	- Now correctly clips ADU to maximum allowed size.  (Note, this
	  has been temporarily commented out because it relies on a new
	  string function that has not yet been integrated into the
	  main distribution.)

	- Now can ignore specific connections dynamically.

	- TCP content gaps are now recognized and ADU delivery is for now
	  stopped for such flows, unless explicitly requested.

	- No longer logs to file in test mode.

- The new function add_notice_tag() explicitly adds a unique notice tag
  to a connection's $addl field (Robin Sommer).  This is sometimes necessary
  to ensure that the tag appears in the connection summary.

- Bro now performs serialization (such as when checkpointing &persistent
  tables or communicating them between Bro's) in an incremental fashion,
  intermingling transfers of large tables with ongoing packet processing
  (Robin Sommer).  Doing so helps avoid packet drops for large items.
  This has not yet been implemented for the initial handshake done
  for &synchronized items.

- ssl.bro now stores certificates by default in the subdirectory "certs/"
  (Robin Sommer).

- Analysis of weak/unknown ciphersuites in ssl.bro reworked (Holger Dreger).

- New cipher for SSL analysis, SSL_CK_RC4_64_WITH_MD5 (Holger Dreger).

- load-levels and cpu-adapt now log their adaptations to the log file
  rather than generating alarms (Robin Sommer).

- The default adaptation levels in cpu-adapt have been tweaked for better
  behavior (Robin Sommer).

- A new structure of the event loop (implemented by Robin Sommer) is now
  enabled during configuration by default (Christian Kreibich).  You can
  revert to the previous structure using --disable-select-loop.

- When configuring Bro, the version of pcap that comes with the Bro
  distribution is no longer used by default (Jason Lee).  Instead,
  the system one is used, or one at the same directory level as Bro.
  To use the Bro distribution version, configure with --enable-shippedpcap.

- backdoor.bro now has comments clarifying that it does not itself
  alter capture_filters (Vern Paxson).

- If you set backdoor_stat_period to 0 sec, then this now turns off
  the periodic component of backdoor analysis (Holger Dreger).

- The filters specified in notice_action_filters now take an additional
  argument specifying the action that has been determined so far (Robin
  Sommer).  This allows the filter to decide to not change the current
  action, if it so wishes.

- The new event notice_alarm(n: notice_info, action: NoticeAction) is
  generated for every notice that results in an alarm (Robin Sommer).

- Tallying of notices is now done using a notice, which has type NoticeTally
  (Robin Sommer).

- The new notice action filter alarm_always_notice specifies an action
  of NOTICE_ALARM_ALWAYS (Vern Paxson).

- If the watchdog expires and Bro isn't generating a packet trace file,
  the current packet is saved to "watchdog-pkt.pcap" (Robin Sommer).

- New boolean globals tcp_contents_deliver_all_{orig,resp} allow easy
  requesting of content delivery for all TCP traffic in orig/resp directions
  (Christian Kreibich).

- The new event udp_contents(u: connection, is_orig: bool, contents: string)
  delivers the contents of UDP packets analogous to tcp_contents (Christian
  Kreibich).  The boolean globals udp_content_deliver_all_{orig,resp} and
  tables udp_content_delivery_ports_{orig,resp} control for which ports
  content is delivered, analogous to the globals that control tcp_contents.

- New option --set-seed=n sets the random number seed to n (Vern Paxson).

- Notices now report current time for remotely-received notices rather
  than network time (Brian Tierney).

- Notices now include a tag es=<peer_description> any time a peer
  description is defined, not just for remote notices (Robin Sommer).

- The global log_as_connection has been removed from icmp.bro, which now
  only logs ICMP flows via the usual connection logging (Vern Paxson).

- The Destination variable $accept_state has been renamed $accept_input
  to better reflect its meaning (Vern Paxson).

- A remote destination's $sync field now indicates whether to accept
  ongoing state changes from peers, rather than just upon start-up
  (Robin Sommer).  The variable $accept_state controls whether we
  accept events.

- Logging of forms of Bro communication has been unified (Robin Sommer).

- Updates for packet filtering documentation (Christian Kreibich).

- A new global, stp_skip_src, lists sources that should be skipped for
  stepping-stone analysis (Vern Paxson).  ssh-stepping.bro adds sources to
  this list if they've instantiated more than src_fanout_no_stp_analysis_thresh
  connections, keeping them blocked until they've been idle for 15 seconds.

- Added a default notice-policy.bro as an example (Brian Tierney).

- Expanded on descriptive text in notice-policy.bro (Vern Paxson).

- ef removed from aux/hf/, as it's of little use and a headache to
  maintain for portability (Vern Paxson).

- The version of libpcap bundled with the distribution has been
  elevated to 0.8.3 (Jason Lee).

- Bro now compiles again if non-blocking DNS is not available (Robin Sommer).

- Resource statistics logging now differentiates between offline
  processing vs. remote-communication-only (Mark Dedlow and Robin Sommer).

- The script variable ICMP::distinct_pairs now times out its state,
  with a default of 15 minutes after creation (Robin Sommer).

- The Bro version reported now includes "-debug" if Bro was configured
  with --enable-debug (Robin Sommer).

- scan.bro now defaults "shut_down_all_scans" to T, meaning it by
  default detects scans on all ports, not just those in the set
  shut_down_scans (Vern Paxson).  Please note, this variable is
  misnamed - it should be "detect_all_scans" - but that change is
  waiting on reworking the basic structure of scan detection.

- Major bug fix for signature matcher missing matches on analyzer data
  (Robin Sommer).  For example, a condition "http /foo/" would only have
  match with the first URL in a connection, not subsequent ones.  Fixing
  this changes the calling sequence of the match_signatures() built-in to
  take an additional final parameter, "clear", which, if set, resets the
  matcher to its starting state prior to matching.

- Serious bug in regular expression matching - and hence signature engine -
  fixed (Robin Sommer).

- Bug fix for formatting (via fmt()) of very long strings (Vern Paxson).

- Fixed mail_reports.sh to correctly find sendmail binary on various systems
  (Brian Tierney).

- Numerous changes to Bro's internal string representation, and more
  flexibility in how strings are rendered for display (Christian Kreibich).

- Pseudo-real-time now can be initialized using an optional argument
  that corresponds to the degree of time compression (Robin Sommer).
  For example, --pseudo-realtime=0.5 causes time to advance half as fast
  as it would in real-time.  The default value is 1.0; any value > 0 is
  allowed.

- The SSH analyzer now looks for just linefeeds as line terminators when
  extracting version strings, rather than carriage-return-line-feeds, to
  match actual implementations rather than the RFC (suggested by Chema
  Gonzalez).

- Playing back events from files now working again (Robin Sommer).

- Bro now uses current_time() rather than network_time to track the
  modification time of variables, since network_time doesn't advance
  when only receiving events (Robin Sommer).

- Bug fixes for IPv6 support, including processing UDP traffic
  (which had been completely broken) and subtle interactions (actually,
  lack thereof) between the connection compressor and IPv6 that
  could lead to crashes (Vern Paxson).

- Portability tweaks for NetBSD, 64-bit Linux SuSe and FreeBSD 5.4
  (Christian Kreibich, Jason Lee and Vern Paxson).

- Bug fix for IPv6 "::" constants that start with hex digits specified
  using 0x (Vern Paxson).

- Calling the built-in terminate() function twice now has no additional
  effect (Christian Kreibich).  It used to terminate Bro abruptly, without
  cleanly shutting down.

- Removed active.bro; use active_connection() + connection_record() instead
  (Vern Paxson).

- Bro lite reports now work with rotated logs files (Brian Tierney)

- Bug fix for conditions such as "payload /^user/", which now work equivalent
  to "payload /user/" (Robin Sommer).

- Tweaks to sensitive patterns in HTTP request URIs to reduce false
  positives (Brian Tierney).

- Bug fixes for strip() built-in function (Holger Dreger).

- Memory leak in built-in function to_addr() fixed (Ruoming Pang).

- Bug fix for "hot" connections sometimes not having their notice tag
  appearing in connection summaries (Robin Sommer).

- Bug fixes for IRC analysis (Vern Paxson and Robin Sommer).

- Syslogging now works if Bro is running in communication-only mode
  i.e., live, but not reading a network interface (Robin Sommer).

- Bug fix to allow tuning of TRW parameters (Vern Paxson).

- Bug fixes for SSL analysis (Holger Dreger).

- Removed logic that inverted orig/resp in some scans (Vern Paxson).

- Lint & memory allocation tweaks (Vern Paxson).

- Bug fixes for inactivity timeouts (Robin Sommer).

- Bug fix for Bro Lite cron job (Jason Lee).

- When binding to a listening port for remote communication fails,
  the port number is now reported (Robin Sommer).

- Some spurious reporting removed from configure output (Jason Lee).

- Fix for "weird"'s generated by connection compressor but not
  recognized at the policy script level (Vern Paxson).

- Fixes for detecting content gaps and not matching previously delivered
  data (Ruoming Pang).

- Bug fixes for TCP rewriter (Ruoming Pang).

- Bug fixes for crashes in SSL analyzer (Vern Paxson).

- Bug fix for avoiding busy-waiting when a communication child dies
  (Robin Sommer).

- Bug fix for BiF's that use 'T' and 'F' in character constants
  (Vern Paxson).

- Memory leak fixes (Robin Sommer, Christian Kreibich, Vern Paxson and
  Ruoming Pang).

- The peer table for inter-Bro communication is now correctly indexed by a
  peer_id (Robin Sommer).

- Bug fix for exchange of initial &synchronized state which could
  prevent communication from entering main phase (Robin Sommer).

- Bug fix for propagating incremented table values derived from
  a table's &default (Robin Sommer).

- Bug fixes for the POP3 analyzer when analyzing non-NUL-terminated strings
  or bad base64 encodings (Vern Paxson).

- Updates for Bro's internal hash functions (Ruoming Pang).

- The debug and communication log files now comply with $BRO_LOG_SUFFIX
  (Robin Sommer).

- Some internal debugging additions (Ruoming Pang).

- Internal cleanup regarding "const" strings (Ruoming Pang).

- A number of casts changed to use modern C++-style pointer casting
  such as reinterpret_cast and static_cast (Ruoming Pang).

- Bug fixes for inter-Bro communication on 64-bit systems (Robin Sommer).

- Bug fixes for detecting errors for SSL connections (Robin Sommer).

- Potential null pointer dereference fixed (Robin Sommer).

- Inter-Bro communication is now more reliable in the presence of errors
  (Robin Sommer).

- Performance enhancement for tracking values whose elements might
  change (Robin Sommer).

- Fixes for peers having differing enum lists (Robin Sommer).  This can
  occur because they're running different scripts and which do different
  redef +='s to add enum values.

- += now works for interval types (Vern Paxson).

- Bug fix for exchanging peer descriptions (Robin Sommer).

- Bug fix for processing multipart-MIME HTTP messages with content-length
  headers (Ruoming Pang).

- Bug fix for failing to escape "'s in HTTP server replies (Robin Sommer).

- Bug fix for propagating increment operations on tables (Robin Sommer).

- Bug fixes for files (Robin Sommer): set open time to current time if
  network time is not initialized; when deserializing files, prevent them
  from being closed immediately due to reference-counting interaction.

- Bug fix to prevent reporting some scans twice (Robin Sommer).

- Bug fix for printing enum's (Christian Kreibich).

- When not configured with --enable debug, Bro now still accepts (yet ignores)
  option -B (Robin Sommer).

- Serialization enhancements and fixes, including a change of the
  protocol version number (Robin Sommer).

- Bug fix for logging inter-Bro communication (Robin Sommer).

- Bug fixes for enumerating attributes and timers (Robin Sommer).

- Bug fix for signatures matching first on one side of the connection,
  and then on the other, being reported twice (Robin Sommer).

- Inter-Bro communication now continues to work even when packet processing
  has been suspended (Robin Sommer).

- Fix for running multiple Bro's together in pseudo-realtime (Robin Sommer).

- Tweak to print-resources.bro so it can be loaded standalone (Vern Paxson).

- Bug fix for &persistent state not being save if Bro wasn't running
  with an input source (Robin Sommer).

- Bug fix for which process ID to check to see if children are still alive
  (Robin Sommer).

- Bug fix for no longer crashing if the expiration function associated
  with a table deletes the element from the table rather than returning
  an interval of 0 secs to indicate it should be deleted (Chema Gonzalez).

- Bug fix for OutboundTFTP notice: now checks to ensure that not only is
  the source local, but the destination is not local (Vern Paxson).

- Bug fix for a subtle interaction mediated by errno, which could cause a
  failed read() to later confuse pcap_dispatch() (Chema Gonzalez).

- Bug fix for TCP contents assertion checking (Ruoming Pang).

- Bug fix for error output on small RPC fragments (Ruoming Pang).

- Fix for connection compressor bug in tracking connection history
  (Robin Sommer).

- Bug fix for potential floating point exception in signature engine's
  resource-profiling code (Robin Sommer).

- Bug fix for low-level List data structure when replacing a list element
  beyond the end of a list (Robin Sommer).

- Bug fix in initializing capabilities when setting up communication between
  Bro peers (Robin Sommer).

- A number of connection compressor bug fixes: weird's for spontaneous
  FINs and RSTs, consistent processing of "connections" that begin with
  RSTs, correct checksum computations, and weird's printed to stderr if
  no event handler defined (Robin Sommer).

- load_sample_freq is now &redef (Vern Paxson).

- Bug fix for backdoor detector incorrectly matching substrings (Vern Paxson).

- Bug fix for canceling timers sometimes failing to cancel all of
  them (Robin Sommer).

- Error handling during un-serialization now handled more robustly
  (Robin Sommer).

- Bug fix for division by zero if expensive_profiling_multiple
  set to zero (Robin Sommer).

- Bug fix for connection logs failing to track all of the annotation
  ($addl) associated with a connection (Vern Paxson).

- Portability fix for BinPAC (Ruoming Pang).

- Fix to NFS analyzer for missing values in events reporting failed requests
  (Vern Paxson).

- autogen.sh now aborts as soon as one of the tools it invokes fails
  (Christian Kreibich).

- Fixed bug where not having SSL would cause bro to not compile (Jason Lee).

- State-holding fix for adu.bro (Christian Kreibich).

- A number of configuration tweaks (Craig Leres & Christian Kreibich).

- Fix for sig-functions.bro: checks isApache* functions, which ensure
  that Apache is indeed in the software set before accessing the index
  (Brian Tierney and Robin Sommer).

- Smith-Waterman fixes and test suite script (Christian Kreibich).


1.0 Sun Oct 23 17:27:45 PDT 2005

- Bro now includes BinPAC (Binary Protocol Analyzer Compiler), a language
  and compiler for automating the construction of analyzers for binary
  protocols (Ruoming Pang).

- Ruoming has used BinPAC to rewrite the analyzers for DCE/RPC (with
  significant enhancements, including adding the endpoing mapper) and SMB
  (likewise enhanced and bug-fixed), and creating a new analyzer for NCP
  (Netware Core Protocol).  The NCP analyzer generates two events:

	ncp_request(c: connection, frame_type: count, length: count,
			func: count)

	ncp_reply(c: connection, frame_type: count, length: count,
			req_frame: count, req_func: count,
			completion_code: count)

- The beginnings of an analyzer for NetBIOS name service (Ruoming Pang).
  It generates the following events:

	event nbns_standard_name_query(c: connection)
	event nbns_nbstat_query(c: connection)
	event nbns_name_reg_request(c: connection)
	event nbns_nb_name_query(c: connection)

- New IRC analyzer (Roland Gruber).  It generates a lot of events; see
  policy/irc.bro.  Note, the formatting of the log file will at some point
  be changed to be more uniform and streamlined.

- ICMP events now include an initial parameter of type "connection",
  the same as for TCP & UDP flows (Ruoming Pang).  This facilitates
  traffic analysis by associating generic connection events such as
  connection_state_remove with ICMP events.  This affects:

	event icmp_sent(c: connection, icmp: icmp_conn)
	event icmp_echo_request(c: connection, icmp: icmp_conn,
			id: count, seq: count, payload: string)
	event icmp_echo_reply(c: connection, icmp: icmp_conn, id: count,
			seq: count, payload: string)
	event icmp_unreachable(c: connection, icmp: icmp_conn,
			code: count, context: icmp_context)
	event icmp_time_exceeded(c: connection, icmp: icmp_conn,
			code: count, context: icmp_context)

- New POP3 analyzer (Florian Schimandl, Hugh Dollman and Robin Sommer).
  Loading pop3.bro analyzes the protocol messages, and loading mime-pop.bro
  also extracts the email headers and content.

- New events (Ruoming Pang):

	connection_first_ACK(c: connection)
		generated upon the ACK completing a TCP handshake. Useful
		in detecting "blink scans" (a FIN coming from the client
		right after the ACK)

	tcp_rexmit(c: connection, is_orig: bool, seq: count, len: count,
			data_in_flight: count, window: count)
		generated when a TCP sender retransmits data

	rpc_call(c: connection, prog: count, ver: count, proc: count,
			status: count, start_time: time,
			call_len: count, reply_len: count)
		can be used to process RPC calls in a generic fashion

	nfs_reply_status(n: connection, status: count)
		supplies the status of NFS server replies

	netbios_session_raw_message(c: connection, is_orig: bool, msg: string)
		access to a NetBIOS SSN message in raw terms

	smb_get_dfs_referral(c: connection, max_referral_level: count,
				file_name: string)
		generated for SMB DFS referal requests

	dns_message(c: connection, is_orig: bool, msg: dns_msg, len: count)
		low-level event generated for each DNS request/reponse

	dce_rpc_bind(c: connection, uuid: string)
		generated for DCE RPC binds

	dce_rpc_message(c: connection, is_orig: bool, ptype: dce_rpc_ptype,
			msg: string)
		low-level access to DCE RPC messages; see const.bif for
		dce_rpc_ptype values

	epm_map_response(c: connection, uuid: string, p: port, h: addr)
		reply from DCE portmapper

- New operator |x|, a sizeof operator (Christian Kreibich).  It yields
  absolute values for numerical values, file size for files, the number
  of enums for an enum type, the number of addresses in subnets, number
  of fields in records, numeric equivalent for addresses, the number of
  elements in vectors/sets/tables, and the length of strings.

- A new clone operator, "* <expr>", produces deep copies of aggregate
  values and the usual duplicates for atomic ones (Christian Kreibich).
  For example, while:

	   1	type foo: record {
	   2		c: count;
	   3		s: string;
	   4	};
	   5
	   6	f1$c = 10;
	   7	f1$s = "Hello";
	   8
	   9	f2 = f1;
	   10	f1$c = 20;
	   11	f1$s = "World";
	   12
	   13	print fmt("%d/%s %d/%s", f1$c, f1$s, f2$c, f2$s);

  yields "20/World 20/World", changing line 9 to:

	f2 = *f1;

  yields "20/World 10/Hello".

- New operators "+=" and "-=", which work on both numerical values and
  strings (Christian Kreibich).

- "+" now works on strings: s1 + s2 yields the concatenation of both
  (Christian Kreibich).

- You can now express the equivalent of ICMP "port numbers" using
  <number>/icmp, where <number> is the ICMP type (Ruoming Pang).

- Bro now accepts long versions of options, such as --readfile for -r
  (Christian Kreibich).

- Bro now has a "pseudo-realtime" mode, activated by --pseudo-realtime,
  that causes it to mimic real-time operation when executing against
  a trace (Robin Sommer).  This is useful for evaluating performance in
  a controlled fashion.

- SMTP analyzer changes (Roger Winslow): support for 554 code in RCPT
  responses; logging when the server refuses the argument to RCPT;
  support for 502 code in response to a HELP command.

- Addition of two universal hash functions: H3 (from David Moore, based
  on code by Ken Keys) and TwoWise (Dietzfelbinger, from Yin Zhang)
  (Ruoming Pang).  Use --enable-h3 and --enable-dietzfelbinger to enable
  them (used as the hash function for short data).  H3 is used by default.

- The "bif" compiler for compiling Bro built-in functions now supports
  an "enum" type (Ruoming Pang).  The syntax is:

	enum dce_rpc_ptype
	%{
		DCE_RPC_REQUEST,
		DCE_RPC_PING,
		DCE_RPC_RESPONSE,
	%}

  which is translated to an enum declaration of "dce_rpc_ptype" in
  Bro, an EnumType* enum_dce_rpc_ptype in NetVar.{h,cc} and a C++ enum
  BroEnum::dce_rpc_ptype {...}.

  One limitation is that redef's on enum types cannot be taken into
  account because the bif is parsed at compile time.

- 64-bit integer support via --enable-int64 (Ruoming Pang).

- The new, experimental policy script adu.bro provides a generic way to
  extract application-layer ADUs (Christian Kreibich).  It heuristicly
  groups blocks of content sent from one side to another, uninterrupted
  by any data in the opposite direction, into an approximate ADU (request
  or reply).  These then generate adu_tx (originator -> responder) and
  adu_rx (responder -> originator) events.  You can control on which ports
  it does this analysis, as well as the amount of data inspected nad
  grouped, using variables documented in the script.

- The new built-in function

	function str_smith_waterman(s1: string, s2: string, params: sw_params)
	: sw_substring_vec

  computes the Smith-Waterman overlap between two strings (Christian Kreibich).
  The third parameter is a record with two fields, $min_toklen (minimum
  length for common tokens) and $sw_variant, which takes a value of 0
  for single-matching and 1 for multiple-matching.

  The return value is a vector of sw_substring records, which hold the
  following fields:

	str: string;	# the common subsequence
	index1: count;	# where it occurs in input string 1
	index2: count;	# where it occurs in input string 2
	new: bool;	# true if start of new alignment

- If you set the new control variable record_state_history to T, then
  connections recorded to the conn.$BRO_LOG_SUFFIX log file will include
  a field that shows the different states encountered during the connection
  (Mark Allman):

	Symbol	State
	------	-----
	S	Initial SYN seen for TCP connection.
	H	SYN-ACK seen for TCP connection.
	D	Data packet seen (TCP or UDP).
	A	Pure ACK seen for TCP connection.
	F	FIN seen for TCP connection.
	R	RST seen for TCP connection.
	I	TCP connection included a FIN+RST packet.
	Q	TCP connection included a packet with multiple connection
		control flags other than FIN+RST (e.g., SYN+RST).
	C	Connection included one or more packets with failed checksums
		(TCP or UDP).

  The symbols are printed in upper-case for connection originators and
  lower-case for responders.  The S/H/F/R symbols are also repeated if
  Bro sees the corresponding control packet subsequently with a different
  sequence number.

  For example, a TCP connection which Bro saw from the beginning, i.e.,
  a normal establishment, followed by the client (originator) first sending
  data, then the server responding, followed by the server initiating a
  normal close which the client then completes, will be annotated as
  "ShADadfF".

- The "for" looping construct now can be used to iterate over the non-empty
  indices of a vector (Christian Kreibich).

- If you set the new variable skip_http_data to T (default: F), then the
  HTTP analyzer will attempt to not reassemble the data portions of HTTP
  request/responses (Ruoming Pang).  This can be a performance benefit
  in environments with high volumes of HTTP traffic, though it may not be
  a large win if the processing is dominated by executing the policy script.

- The new built-in

	remask_addr(a1: addr, a2: addr, top_bits_from_a1: count): addr

  take some top bits (e.g. subnet address) from a1 and remaining bits
  (intra-subnet part) from a2 and merge them to get a new address (Ruoming
  Pang).  Useful for anonymizing at the subnet level while preserving
  serial scans.

- The new built-in

	decode_netbios_name(name: string): string

  takes a string in NetBIOS encoding and returns its original form
  (Ruoming Pang).

- The new variable ignore_keep_alive_rexmit controls whether to
  include keep-alives when counting retransmitted packets (Ruoming Pang).
  It defaults to F (i.e., do count them).

- The calling sequence of dce_rpc_request and dce_rpc_reply have changed to:

	event dce_rpc_request(c: connection, opnum: count, stub: string)
	event dce_rpc_response(c: connection, opnum: count, stub: string)

  (Ruoming Pang).  Use dce_rpc_message to get access to the RPC type and
  the raw message.

- The calling sequence of the netbios_session_message event has changed to:

	netbios_ssn_message(c: connection, is_orig: bool,
				msg_type: count, data_len: count)

  (Ruoming Pang).  Previously it was parameterized with the connection
  and the raw message (now available via netbios_session_raw_message).

- The calling sequences of smb_com_{read,write}_andx have changed to
  no longer include the is_orig parameter because it is in fact fixed for
  these events (Ruoming Pang).

- The calling sequence of smb_message has changed (Ruoming Pang) to:

	smb_message(c: connection, is_orig: bool, cmd: string,
			body_length: count)

- Bug fix specifying the &default value for tables that yield function
  values (Ruoming Pang).  For example:

	type tcp_content_handler_func:
		function (c: connection, is_orig: bool, seq: count,
				contents: string);

	function default_tcp_content_handler(c: connection, is_orig: bool,
						seq: count, contents: string)
		{
		# do something ...
		}

	const tcp_content_orig_handlers: table[port] of
		tcp_content_handler_func = {} &redef &default =
			default_tcp_content_handler;

  Previously, Bro would take the function given with &default as the default
  function to call when accessing a missing element, rather than a default
  *value* to directly return. Bro now checks the value type against the
  function type to see if they match in type.

- The new variables forward_remote_events and forward_remote_state_changes
  specify whether to broadcast events/state received from one peer to other
  peers (Robin Sommer).  Both default to F.  Note, these options are temporary;
  they will disappear when we add a more sophisticated script-level
  communication framework.

- Vectors can now be initialized using the syntax such as

	global foo: vector of string = ["foo","bar"];

  (Robin Sommer).

- Bug fixes for &synchronize'ing vectors (Robin Sommer).

- The internal implementation of strings in the policy language has
  been heavily revamped (Christian Kreibich).

- String built-in functions are now in strings.bif rather than bro.bif
  (Christian Kreibich).  This includes two new built-ins:

	str_split(s: string, idx: index_vec): string_vec
	strstr(big: string, little: string): count

  string_vec is a new policy script type that is an alias for
  "vector of string".

- The new options --load-seeds <file> and --save-seeds <file> let you
  record Bro's seeds to a file and then re-use these seeds in a later
  invocation (Christian Kreibich). The primary intended usage is to
  provide determinism in hash table iterations etc. for debugging purposes.

- Communication protocol changes (Robin Sommer):

  * Internal PING/PONG messages to measure round-trip times.  The new script
    remote-ping.bro issues PINGs every second and logs to remote.log.

  * Optional data compression if libz is available.  Remote::Peer$compression
    specifies compression level, with no compression being the default.

  * Inter-Bro communication is now performed in four explicit phases:

    //  Setup:
    //      Initial phase.
    //      VERSION messages must be exchanged.
    //      Ends when both peers have sent VERSION.
    //  Handshake:
    //      REQUEST_EVENTS/REQUEST_SYNC/CAPTURE_FILTER/CAPS/selected SERIALs
    //        may be exchanged.
    //      Phase ends when both peers have sent PHASE_DONE.
    //  State synchronization:
    //      Entered iff at least one of the peers has sent REQUEST_SYNC.
    //      The peer with the smallest runtime (incl. in VERSION msg) sends
    //        SERIAL messages comprising all of its state.
    //      Phase ends when peer sends another PHASE_DONE.
    //  Running:
    //      Peers exchange SERIAL (and PING/PONG) messages.
    //      Phase ends with connection tear-down by one of the peers.

  * Serializing network packets includes textual tags for identification.

  * Serializing files includes the state of buffering.

- Pending events for remote peers are now flushed when Bro terminates,
  and the net_done event is *not* propagated to peers (Robin Sommer).

- Makefile.am cleanups (Christian Kreibich).

- libpcap portability fix for OpenBSD (Gordon Willem Klok).

- Performance bug fix for SMTP relay detection (Vern Paxson).

- sprintf -> snprintf tweak (Vern Paxson).

- Bug fix for serializer regular-expression matchers (Robin Sommer).

- Some fixes for access to uninitialized variables/state (Christian Kreibich
  and Vern Paxson).

- More informative messages for some internal errors (Christian Kreibich).

- Bug fixes for implementation of vectors (Christian Kreibich).

- Fixes for FreeBSD 5 installs (Jason Lee).

- gcc 4.0 compatibility (Christian Kreibich).

- Bug fix for correctly propagating libpcap failures (Chema Gonzalez).

- Bug fixes for prefix-preserving IP address anonymization (Chema Gonzalez).

- The MIME analyzer in mime.bro is now in "module MIME" (Vern Paxson).

- Bug fix for the IRC backdoor detector (Scott Campbell).

- The capture filter used for NFS traffic now includes UDP fragments,
  since NFS UDP traffic is often fragmented (Ruoming Pang).

- New internal mechanisms to suspend/resume processing to enable a Bro
  receiving synchronized state to put its own packet processing on hold
  (Robin Sommer).

- A bug with the serialization cache not being used for modified objects
  has been fixed (Robin Sommer).

- A number of enhancements to inter-Bro communication performance and error
  handling improved (Robin Sommer).

- Internal restructuring to fix problems with dispatching packets when
  using the packet sorter (Ruoming Pang).

- Christian Kreibich has contributed a number of fixes for code flaws
  such as potentially unsafe library calls.


0.9a10 Tue Sep  6 10:41:53 PDT 2005

- Fixes for portability to 64-bit architectures (Christian Kreibich).

- Bug fix for broken syslog'ing of alarms (Scott Campbell).

- The manual has been updated to clarify that aggregate values in events
  are passed as shallow copies, so that modifications to elements of the
  values after posting the event but before it's handled will be visible
  to the handlers for the events (Christian Kreibich).

- HTTP logging now includes the host from the Host header in the request
  (Craig Leres).  Note, currenty this only is done when using http-reply.bro,
  not if you only analyze requests.

- You can now specify a passphrase for the SSL cert used for inter-Bro
  communication by redef'ing the variaable "ssl_passphrase" (Christian
  Kreibich).  Leaving it unchanged causes the passphrase to be read
  interactively.

- Certificates created using ca-issue now have 2-year lifetimes rather
  than the default of 30 days (Christian Kreibich).

- A problem with handshaking between Bro peers has been fixed (Christian
  Kreibich).

- A bug has been fixed in scanning false positives due to backscatter
  in the form of SYN ACKs (Vern Paxson).

- Alerts sent via email now use a From address configured from bro.cfg
  (Randy Mcclelland-Bane).  Also, if sending an alert via gpg fails,
  it's sent instead as plaintext.

- Scan notices now include information about the connection that
  triggered the scan detection decision (Vern Paxson).

- Exported some TRW variables so the user can adjust their associated
  timers (Vern Paxson).

- The new script variable dns_max_queries sets a maximum on the number of
  queries that can appear in a DNS request (Scott Campbell & Vern Paxson).
  If more queries appear, the request is treated as non-DNS traffic and
  ignored.  The variable defaults to a value of 5.  Setting it to 0 turns
  off this functionality, so Bro processes all apparent requests.

- The "weird" messages generated by the DNS analyzer now have a more
  regular naming structure and processing (Scott Campbell and Vern Paxson).

- Tweaked bif_arg.cc to pass gcc4.0 and bro.bif to not collide with
  uuid in OSX 10.4 (Jason Lee).  Now works on OSX 10.4, though use
  --disable-localpcap when compiling.

- Bro now compiles cleanly under OpenBSD (Jason Lee).

- NOTE: the connection compressor has a known serious bug and should
  not be used at present.  Since it is an experimental feature, fixing it
  is deferred to the next release.

- Some bugs fixed in the management of hash keys when using the
  connection compressor (Robin Sommer).

- Tweak for the connection compressor to generate truncated_header weird's
  (Robin Sommer).

- Temporary bug fix for type clash in SSL version numbers (Vern Paxson)
  by making them consistently of type int.  The correct fix is probably
  for them to be consistently of type count, depending on how Bro's notion
  of general version processing, and its SSL analyzer, both evolve.

- Bug fix for trace rewriting failing if Bro was not compiled to check
  assertions (Martin Casado).

- Fixed logic bug in signal handling regarding whether we're currently
  idle waiting for input vs. processing a packet or the event queue
  (Vern Paxson).  Note, this change has not been heavily tested.

- Some bug fixes for correct operation when DNS names fail to resolve
  (Vern Paxson).  It's not clear that these fixes are complete, however.

- Fixed to not compile libpcap when --disable-localpcap is given to configure
  (Jason Lee).

- Fixed configuration of local pcap for IPv6 if --enable-brov6 is specified
  (Jason Lee).

- A problem with "make install" when building from the libpcap included
  in the sources has been fixed (Christian Kreibich).


0.9a9 Thu May 19 23:31:33 PDT 2005

- First cut at analyzer for NFS (Vern Paxson).  It generates the following
  events:
	event nfs_request_null(n: connection)
	event nfs_request_getattr(n: connection, fh: string, attrs: nfs3_attrs)
	event nfs_request_lookup(n: connection, req: nfs3_lookup_args,
				rep: nfs3_lookup_reply)
	event nfs_request_fsstat(n: connection, root_fh: string,
				stat: nfs3_fsstat)
	event nfs_attempt_null(n: connection, status: count)
	event nfs_attempt_getattr(n: connection, status: count, fh: string)
	event nfs_attempt_lookup(n: connection, status: count,
				req: nfs3_lookup_args,
				dir_attrs: nfs3_opt_attrs)
	event nfs_attempt_fsstat(n: connection, status: count,
				root_fh: string, obj_attrs: nfs3_opt_attrs)

- The new script OS-fingerprint.bro integrates Bro's new passive OS
  fingerprinting mechanism with the software.bro framework (Vern Paxson).

- You can now operate on patterns using && and || (Vern Paxson).
  If p1 and p2 are patterns, then p1 && p2 yields a pattern that matches
  their concatenation and p1 || p2 yields a pattern that matches either.
  Note that the syntax for this may change in the future to a single '&'
  or '|', which would be more consistent with the use of '|' in
  constructing pattern constants.

- An experimental "connection compressor" tracks not-yet-established
  connections using much less memory than Bro normally does (Robin Sommer).
  This is potentially a major win during flooding attacks and high-speed
  scans.  You activate it by setting use_connection_compressor to T.  You
  can then control the granularity of its processing using the variables
  cc_handle_resets, cc_handle_only_syns, and cc_instantiate_on_data.  See
  bro.init for brief discussion of these.

- The experimental new script firewall.bro supports firewall-rule-like
  processing of connections in terms of allow/deny (Robin Sommer).  It is
  not particularly efficient.

- sensor-sshd.bro provides an experimental interface for receiving
  events from instrumented SSH servers that communicate with Bro via
  the Broccoli client library (Christian Kreibich and Robin Sommer).
  Supporting this also entailed extensions to login.bro so it can
  process the events even though they don't correspond to a connection
  known to Bro's event engine.

- The new built-in function match_signatures() can be used in a policy
  script to send text directly into the signature engine (Robin Sommer).

- Correction: the 0.9a8 CHANGES states that the mail_script variable used
  for NOTICE_EMAIL defaults to mail_script.sh.  The correct value is instead
  "mail_notice.sh".

- The scripts rsh.bro and passwords.bro, and the passive-fingerprinting
  signatures policy/sigs/p0fsyn.osf were inadvertantly left out of the
  0.9a8 distribution.

- Added s2b (snort to bro) files into the distribution. (Jason Lee)

- Non-blocking packet capture under Linux has been fixed (Robin Sommer).

- Fixed printing of DNS replies, which used to work but was broken
  a number of months ago (Vern Paxson).

- The new script brolite-sigs separates out how signatures are configured
  in Bro Lite so the functionality can be enabled/disabled with a simple
  load statement (Roger Winslow).  That is, to use signatures with Bro
  lite, simply add "@load brolite-sigs".

- The new script variable enable_syslog (default T) controls whether
  alarm's are syslog'd (Robin Sommer).  As before, syslogs can only happen
  when Bro is reading from live network traffic (this should be changed
  at some point, to accommodate real-time Bro's that don't read the network
  but collect events from other sensors).  Previously, in that case syslog's
  always happened; now, you can turn them off using this variable.

- The new script variable expensive_profiling_multiple controls how
  often, when doing profiling, to perform more expensive forms of
  profiling, in particular, memory consumption profiling (Robin Sommer).
  If profiling_interval is set to 15 sec and expensive_profiling_multiple
  is set to 20, then expensive profiling will be done every 5 minutes
  (these are the defaults now in profiling.bro).  Also, the profiling_update
  event now includes a second argument, expensive: bool, which indicates
  whether the update corresponds to one of these expensive profiling
  intervals.

- First cut at parsing DNS AAAA replies (Scott Campbell).  This is quite
  incomplete - currently, the replies are turned into fake A record replies,
  due to the difficulty of dealing with IPv6 addresses if Bro wasn't built
  to analyze IPv6 traffic.

- software.bro has been tweaked to have a new control variable,
  "only_report_local" (default F).  If true, then only software versions
  for local addresses (as determined by is_local_addr()) will be
  reported.

- synflood.bro now has a script variable max_sources (default 100) that
  specifies the maximum number of sources to track for a given victim
  (Robin Sommer).

- Remote peers now negotiate their versions of the serialization format
  (Robin Sommer). If they don't agree then the connection is terminated.

- Generic UDP request/response processing has been moved into the new
  policy script udp-common.bro, which, unlike udp.bro, does *not* set the
  packet filter to capture all UDP traffic (Robin Sommer).  A number
  of UDP-based policy scripts have been modified to use udp-common.bro
  rather than udp.bro.

- When printing serialized/independent state, access times are now
  again included (Robin Sommer).

- Bro's implementation of timers has been switched (reverted) to using
  priority queues (Vern Paxson).

- The http-request.bro script variables skip_remote_sensitive_URIs and
  const sensitive_post_URIs are now exported so they can be accessed
  externally (Robin Sommer).

- Some new rootkit filenames have been added to ftp.bro and
  http-request.bro (Brian Tierney).  The plan is to eventually
  merge these lists so there's only one main list.

- trw.bro is now scoped as a module "TRW" (Brian Tierney).

- Better support of the '--disable-localpcap' flag to configure, and
  consolidated all the pcap checks in configure.in (Jason Lee).

- A bug in processing bare carriage-returns in Telnet input/output
  has been fixed (Vern Paxson).

- The Bro Lite bro.rc script has been tweaked to use the 'ax' flags
  instead of '-ax' (Jason Lee).

- A bug with reporting ICMP "ports" (i.e., type + code) has been fixed
  (Vern Paxson).

- Bug fix for excessively large RPC messages (Ruoming Pang).

- A bug with /0 subnet prefixes has been fixed (Robin Sommer).

- The function record_connection() now takes the file to write to
  as its first argument (Robin Sommer).

- remote.bro now tracks whether a given Destination is connected
  (Robin Sommer).

- mail_notice.sh is now installed as part of installing a distribution
  (Jason Lee).

- Fixed bug where the sort order for the test suite changed depending
  on locale. (Jason Lee)

- Bug fix for email_notice() when notice_action_filters not defined for
  given notice (Vern Paxson).

- The test suite test for rare-events fixed to not give false positives
  (Jason Lee).

- Date added for 0.9a8 release.


0.9a8 Wed Feb 16 17:09:34 PST 2005

- aux/rst/ contains the source for the "rst" tool used by Bro (via the
  policy script function terminate_connection() in conn.bro) to tear
  down established connections by forging RST packets.

- Bro's main event loop has been reworked (Robin Sommer).  This should
  (1) not cause any visible differences in most cases, (2) improve
  performance in some cases, (3) fixed problems running Bro without
  a network input (but still receiving asynchronous input from remote
  event sources).  There are some more changes coming to this soon.

- Passive OS fingerprinting has been added, based on Michal Zalewski's
  "p0f" tool (Holger Dreger).  Currently, it's limited to fingerprinting
  clients based on the initial SYNs they send.  To use it, define
  an event handler:

	OS_version_found(c: connection, host: addr, OS: OS_version)

  OS_version is a record containing a string $genre (e.g., "Solaris"),
  a string $detail (e.g., "2.0.27"), a count $dist (hop-count distance
  from monitor to host), and $match_type, which specifies via an
  enumerated type whether the match was direct from a signature,
  generic to the genre, or "fuzzy".

  The match is done against a passive fingerprinting signature file,
  which is specified by the variable passive_fingerprint_file.
  It defaults to "sigs/p0fsyn", which is found using $BROPATH
  and has an "osf" suffix added.

  You can restrict the matching to only be performed for hosts from
  particular subnets by adding those subnets to the variable
  generate_OS_version_event.  If it's empty (default), then all subnets
  are analyzed.

  Note, the passive fingerprinting should be integrated with the
  version-tracking in software.bro, but this hasn't been done yet.

- Support for IPv6 has been repaired and brought up to date.  Note, however,
  that inter-Bro communication currently only works over IPv4.

- Signature-matching is now off by default in brolite.bro.  If you want
  to use it, define use_signatures = T prior to @load'ing it.

- Notices are now tied to their corresponding connections (Scott Campbell).

- New backdoor detectors for IRC, SMTP, Gaobot (Scott Campbell).

- Signature matches now have a connection associated with them (Scott Campbell).

- Bro scripts that set initial timers (via "schedule" statements in a
  bro_init handler) but don't have any source of network input (trace
  files or live interfaces) now execute in real-time, with network_time
  set to the current time, rather than having their timers expire immediately.

- Default timeouts have been added to tables in trw.bro and http.bro, which
  have been found operationally to potentially grow very large (Scott Campbell).

- The new policy script large-conns.bro can be included in order to
  track the size of TCP connections (each direction is referred to
  as a "flow") using a secondary packet filter (Chema Gonzalez).
  This method is completely separate from Bro's usual size accounting,
  and offers the advantages that it tracks sizes even for connections
  that don't terminate (or for which Bro misses their establishment)
  and for connections with sizes > 4 GB.

  The interface is via the function:

	function estimate_flow_size_and_remove(cid: conn_id, orig: bool)

  If $orig=T, then an estimate of the size of the forward (originator)
  direction is returned.  If $orig=F, then the reverse (responder) direction
  is returned.  In both cases, what's returned is a "flow_size_est" record,
  which includes a flag, $have_est, indicating whether there was any
  estimate formed. If $have_est is T, then the record also includes
  an estimate in bounded by $lower (lower bound) and $upper (upper bound).
  The estimate also includes $num_inconsistent, which, if > 0, means that
  the estimates came from sequence numbers that were inconsistent, and
  thus something is wrong - perhaps packet drops by the secondary filter).
  Finally, calling the function causes the flow's record to be deleted.

- An RSH analyzer has been contributed by Manu (ManuX@rstack.org).
  It generates rsh_request and rsh_reply events, and the following notices:

	  DifferentRSH_Usernames
		Client and server username differ.

	  FailedRSH_Authentication
		Attempt to authenticate via RSH failed.

	  InteractiveRSH
		The RSH session appears to be interactive (multiple
		lines of user commands).

	  SensitiveRSH_Input
	  SensitiveRSH_Output
		RSH client input or server output match input_trouble/
		full_input_trouble or output_trouble/full_output_trouble.

- The new notice action NOTICE_EMAIL indicates that in addition to
  logging an alarm, it should also be sent out as email (Scott Campbell).
  By default, email is only sent if Bro is running on live traffic;
  you can override this via redef'ing the script variable mail_notification.
  Mail is sent using the script specified by the mail_script variable
  (default: "mail_script.sh", which is now included in the distribution,
  but at present is not installed), which must be in $PATH.  The mail
  is sent to the username specified in mail_dest (default: the local
  "bro" user, though you can change this to name@domain).

  Note that specifying email as a separate notice action may change
  in the future, to instead be an attribute that's associated with
  other notice actions.  For example, it may make sense to want
  to specify both NOTICE_ALARM_PER_CONN and NOTICE_EMAIL; currently,
  however, you can't do this.

- A similar notice action NOTICE_PAGE does the same thing as NOTICE_EMAIL
  except it send the mail to mail_page_dest (Scott Campbell).

- You can now use the attribute &rotate_size for file objects to
  specify the maximum file size in bytes (Robin Sommer). If the limit
  is reached, the file is rotated similiarly as is already done with
  &rotate (which, for consistency, has been renamed to &rotate_interval).

  For both &rotate_size and &rotate_interval, when they trigger they
  now generate events (rotate_size and rotate_interval, respectively;
  each takes the file as the sole argument) rather than invoking
  &postprocessor, which has been removed.

  There's also a new variable log_rotate_size to set a global size maximum.

  Related to log rotation are the following new built-in functions:

	rotate_file(f: file) closes the file, moves it to a temporary
	name, and opens a new one. It returns the new "rotate_info"
	record, which gives the temporary name and the open/close times.

	rotate_file_by_name(s: string): similar, but call by the name
	of the file rather than a Bro script value.  This is needed
	because some files are not represented by file objects but need
	to be rotated nevertheless (most importantly, the tcpdump save
	file and the dump files for dump_current_packet()). This function
	rotates the file with the given name.

  Finally, you can load the new policy script rotate-logs.bro to
  get default behavior of rotating all log files every hour.

- The new "@unload <script>" directive specifies that future @load's of
  <script> should be skipped.  This is useful for overriding analyzers
  loaded by scripts that pull in a bunch of analysis.  For example,

	@unload ntp
	@load mt

  would load all of the "mt" analyzers *except* ntp.bro.

- The new built-in function get_file_name(f: file): string returns
  the filename associated with a file (John McNicholas).

- The new built-in function get_contents_file(id: conn_id, direction: count)
  returns the contents file (set using set_contents_file()) for the given
  direction (John McNicholas).

- The new built-ins time_to_double() and double_to_time() convert between
  double values and time values (Robin Sommer).  The new built-in floor()
  returns the floor of a double value; this returned value is also a double.
  Thus, floor(-3.4) returns -4.0.

- Support for sending packets between Bro's (Robin Sommer).

- Bro now has a geneal mechanism internal for traversing policy scripts
  (Umesh Shankar).  Various script analyses can be specified using the
  new -z flag.

  Currently, the one supported form of analysis is "-z notice", which
  prints all of the different types of notices that the script you've
  loaded can generate.  For example, "bro -z notice ftp" will generate:

	  Found NOTICE: BackscatterSeen
	  Found NOTICE: FTP_PrivPort
	  Found NOTICE: FTP_BadPort
	  Found NOTICE: PortScan
	  Found NOTICE: FTP_ExcessiveFilename
	  Found NOTICE: ScanSummary
	  Found NOTICE: AddressDropped
	  Found NOTICE: DroppedPackets
	  Found NOTICE: SensitiveConnection
	  Found NOTICE: FTP_UnexpectedConn
	  Found NOTICE: SSH_Overflow
	  Found NOTICE: FTP_Sensitive
	  Found NOTICE: TerminatingConnection
	  Found NOTICE: PasswordGuessing
	  Found NOTICE: AddressDropIgnored
	  Found NOTICE: AddressScan

- The signature rule language now supports an "active" keyword,
  which can be set to "true" or "false", with the latter turning
  off the rule (Roger Winslow).  If set to false the signature will
  not be loaded into the rule matcher, otherwise it is.

- The signature rule language now supports meta data of the form
  ".MMM<whitespace>XXX", where MMM is arbitrary text which makes up the name
  of the meta data option and where XXX is arbitrary text up to the end
  of the current line (Roger Winslow).  The intent is that some forms of
  meta-data will be regularized/standardized in the future - information
  such as date modifed, category/class, weighting, etc.  For now, it
  provides a way to annotate rules with nominally more structure than just
  using comments (though it is currently treated the same, i.e., everything
  is ignored).

- The following meta data option names are now reserved: .version, .revision,
  .version-date, .revision-date, .date-created, .location

- The new enumerated type "transport_proto" is used to specify different
  types of transport protocols: "tcp", "udp", "icmp", and "unknown_transport".
  Associated with it are new built-in functions: is_udp_prot(), is_icmp_port(),
  get_conn_transport_proto, and get_port_transport_proto.  The latter two
  map a given connection and a given "port" value to their corresponding
  transport_proto value.

- A bunch of tuning (regular expressions for sensitive login sessions,
  scan detection thresholds, forbidden/hot usernames and filenames,
  sensitive URIs, "weird" actions) have been incorporated from
  operational configurations.

- Serious bugs in managing large numbers of files fixed.

- A serious bug with negative DNS TTL settings (and, more generally,
  with negative timer values) fixed.

- The traditional connection logging format is no longer supported.

- The SMTP analyzer's state machine processing has been modified to
  correctly deal with clients that (incorrectly) pipeline their commands
  (Ruoming Pang).

- A bug fixed in detecting SMTP relays for connections w/o message bodies
  (Ruoming Pang).

- A bunch of bugs in recording connection summaries for UDP flows
  have been fixed.

- A new script module, passwords.bro, generates PasswordExposed notices
  for activity (currently just rlogin/telnet logins) that expose passwords.

- A new script module, file-flush.bro, can be loaded to cause all log
  files to be flushed every file_flush_interval (default: 10) seconds.
  This is handy if you like to watch the files in real time.

- Zone transfers now generate a ZoneTransfer notice, unless the host
  making the request is in DNS::zone_transfers_okay.

- Bro's DNS cache (generated using -P and accessed using -F) is
  now kept in the .state/ subdirectory rather than in the user's
  home directory (Roger Winslow).

- Some changes to remote propagation of events/values and detection of
  state inconsistencies (Robin Sommer).

- A fix for avoiding delays on low-volume links for some systems for
  which it can take a long time to fill up the pcap buffer, and pcap doesn't
  return partial buffers (Robin Sommer).

- A bug in table expiration timers has been fixed (Robin Sommer).

- A bug in comparing subnets has been fixed.

- A bug in using a non-constant value for a &write_expire attribute
  has been fixed.

- A bug in using CONTENTS_BOTH for writing reassembled streams to
  files has been fixed (John McNicholas).

- A subtle but potentially damaging bug in fragment reassembly has
  been fixed.

- A bug with using local variables of vector types has been fixed.

- A bug with comparing strings has been fixed.

- Bro no longer generates the RST_with_data "weird", as with modern
  stacks it's no longer any sort of strange occurrence.

- Related to this, the signature rule matcher no longer matches
  against the payload of RST packets.  (Note, this is an imcompatibility
  with Snort.)

- Portmapper mappings are now written in the connection log in
  alphabetical order.

- The event engine variable frag_timeout now defaults to 5 minutes if you
  load frag.bro, and is accessed via redef rather than by defining the
  global directly.

- The interval that signatures.bro waits for until generating a signature
  summary can now be set using the new script variable sig_summary_interval,
  and a bug in generating the summaries has been fixed (Robin Sommer).

- The new script peer-status.bro generates periodic "update" events regarding
  a remote peer's status (Robin Sommer).  These take the form:

	type peer_status: record {
		res: bro_resources;
		stats: net_stats;
		current_time: time;
		cpu: double;		# average CPU load since last update
		default_filter: string;	# default capture filter
	};

- The bro_resources record returned by resource_usage() now includes
  three additional fields, $version (the version of Bro), $debug
  (T if Bro was compiled with debugging information), and $start_time
  (the time Bro began executing - clock time, not network time).

- The new built-in function same_object(o1: any, o2: any): bool
  returns true if its arguments refer to the same object, false
  otherwise.  This can be useful for comparing tables, for example
  in calls to table element expiration functions.

- The new built-in function bro_is_terminating(): bool returns true if
  Bro is done reading from its network input source(s) and is now
  in its final termination cleanup (Robin Sommer).

- A new built-in strftime() formats a timestamp, returning a string
  (Robin Sommer).

- A new built-in file_size() returns the size of the file with a
  given name (Robin Sommer - note: *not* a Bro file value).

- A potential deadlock with inter-Bro communication has been fixed
  (Robin Sommer).

- Bro now always forks a copy of itself when executing, as this
  can save considerable memory when using inter-Bro communication
  (Robin Sommer).

- The Bro interconnection protocol now includes explicit handshaking
  during session establishment to mark that a peer is ready (Robin Sommer).
  Implementing this includes a change in the wire protocol that is
  incompatible with the protocol used in the past.

- The TCP inactivity timer is now started whenever a connection
  transitions from a pre-establishment state (including "inactive")
  to some sort of established state (Robin Sommer).  Prior to this
  fix, connections for which a proper SYN handshake was not seen would
  not be timed out as inactive.

- The --disable-openssl configure option has been removed; now
  the only option is --with-openssl, and --with-openssl=no disables
  use of OpenSSL (Gregor Maier).

- A bug in invoking &expire_func functions has been fixed (Robin Sommer).

- A bug in logfile rotation has been fixed (Robin Sommer).

- A bug in recognizing negative floating point values has been fixed.

- worm.bro now suppresses the default signature action for worms
  it knows about, since it generates events for them (Robin Sommer).
  The list of worms detected via signatures now includes Bagle-BC.

- Signatures for known worms are now skipped when doing signature
  summaries and scan detection, if worm.bro is loaded (Robin Sommer).

- request_remote_events and request_remote_sync now implicitly
  do set_accept_state, too.

- Better error handling for SSL connections (Robin Sommer).

- Bug fixed which caused diagnostic messages to be lost when using
  inter-Bro communication (Robin Sommer).

- gcc 3.4 portability fixes (Brian Lindauer).

- Solaris portability fixes (Robin Sommer).

- The Bro distribution now includes and uses its own version of libpcap
  for portability reasons (Jason Lee).

- Some minor bug fixes to handling of tcpdump save files (Robin Sommer).

- Detection added for a (now quite old) SSHv1 overflow attack.

- A bug in skipping processing of connections for large chunks of
  data has been fixed (Chema Gonzalez).

- Some memory leaks fixed (Robin Sommer).

- fmt()'s "%d" format now accepts values of enum types.


0.9a7 Mon Nov  1 13:21:05 PST 2004

- New terminology:

	o We've found that the term "log" has been too overloaded,
	  sometimes meaning "something to record for audit purposes" and
	  other times meaning "something worthy of getting the operator's
	  attention right now, for example via syslog".

	  We are now using "log" to only refer to the first of these,
	  and refer to the second as "alarm".

	o We've found that "alert" (and the ALERT() function, etc) is
	  likewise confusing.  Some expect it to mean something alarm-worthy
	  (to use the new name from the previous item), while others
	  expect it to mean the output from a sensor, which might not
	  be worth getting the operator's attention.

	  We are now using "notice" to refer to what had previously
	  been called "alert".  So, for example, rather than call ALERT()
	  you call NOTICE() to enter something into the I've-seen-something-
	  maybe-it's-worth-an-alarm framework.  The first field associated
	  with such a call was $alert, which is now $note.  These notifications
	  are logged to notice.$BRO_LOG_SUFFIX rather than "alert."...

  These changes show up in many places in the policy scripts.  Some
  globals log_XXX are now alarm_XXX or notice_XXX.  However, uses of
  "log" that refer to "something to record for audit purposes" remain
  using that name.

- Bro now uses the "automake" suite of tools for its configuration
  (Jason Lee).  This includes major reworking of its Makefile's
  and configuration/installation scripts.

- Bro now flags event handlers that are never invoked (Umesh Shankar).
  This catches typos like:

	event bro_initt() { ... }

- The scripts directory now includes scripts for generating nightly
  reports (Roger Winslow).  Note that these are not presently a supported
  part of the public Bro distribution, but will become so in the
  not-too-distant future.

- policy/brolite.bro contains a first version of the "Bro Lite" configuration
  (Scott Campbell, Roger Winslow, et al).  This also is not presently
  supported for the public Bro distribution, but will become so.

- Major overhaul of the internal serialization framework (Robin Sommer).
  Some more changes are forthcoming, though not on this scale.

- Changed packet capture/drop statistics to (1) explicitly track received
  packets, (2) work better under Linux, (3) report link counts when available
  (Robin Sommer).  The net_stats record field "interface_drops" has been
  renamed "pkts_link".  It reports the number of packets captured by the
  NIC (if available), rather than the number dropped by the NIC (which never
  actually worked, anyway).

- The DNS analysis has been extended to deal with TSIG and to better deal
  with EDNS0 records, and its overall structure reworked somewhat
  (Scott Campbell & Roger Winslow).  The scripts in dns.bro are now
  inside "module DNS" scope.

- Improved logging of ICMP flows (Scott Campbell).  This includes the
  addition of a "len" field in icmp_conn.  Some significant additional
  changes/improvements to ICMP processing will be coming very soon.

- The Bro README and "quick start" documentation has been updated
  (Brian Tierney and Jim Rothfuss).

- Some significant state-holding problems in the presence of packet filtering
  or packet drops have been fixed.

- You can now instruct the signature engine to constrain the number of
  regular expressions it groups together into a single matcher using the
  new script variable "sig_max_group_size" (Robin Sommer).  This can result
  in significant memory savings for large sets of signatures.  It can
  increase the CPU processing required; however, in our testing so far
  this does not appear to be the case.

- The signature engine now provides better location information in its
  error messages (Robin Sommer).

- statistics.bro has been renamed profiling.bro.  Along with this change,
  the following script variables have been renamed:

	do_statistics => do_profiling
	statistics_file => profiling_file
	statistics_interval => profiling_interval

- A new script, stats.bro, can be included to generate light-weight
  running statistics on memory and CPU use.

- A new script, print-sig-states.bro, can be included to generate
  periodic dumps of signature-matching statistics (Robin Sommer).

- The connect_clear()/connect_ssl() built-ins have been replaced
  by a single connect() function (Robin Sommer).

- Remote Bro's (more generally, remote event sources) are now represented
  in policy scripts using the new "event_peer" type rather than as an
  addr/port pair, and get_event_source() has been renamed get_event_peer()
  (Robin Sommer).  The new function get_local_event_peer() returns an
  event_peer corresponding to the local Bro.

- The new script remote-print.bro can be used to print all events
  received from remote sources (Robin Sommer).

- When you call send_state(), an event finished_send_state() is now
  generated when all of the state has been sent (Robin Sommer).

- If you define a handler for the new print_hook() event, then it
  will be invoked every time a "print" statement executes (Robin Sommer).

- The -g flag no longer takes a directory as an argument, but always
  uses the .state directory (Robin Sommer).

- The new -I flag prints the value of a given identifier and exits
  (Robin Sommer).  This flag may go away in the future.  For now,
  its role is that it prints timestamps of things like table elements,
  which are no longer printed by a regular "print" statement.

- If a connection proceeds as SYN ->, <- RST, RST ->, this is now
  reported as a regular rejected ("REJ") connection, rather than
  one reset by the originator ("RSTO").

- The TCP analyzer now attempts to detect connections for which
  one side's packets are being filtered out.  It does this using
  two C++ variables (not presently accessible at the script level),

	const int max_initial_window = 4096;
	const int max_above_hole_without_any_acks = 4096;

  If more than max_initial_window data has been sent by one side and no
  ACKs have been seen by the other side, then it's assumed that no ACKs
  will ever show up, and the buffer for the data should be reclaimed
  immediately after it's reassembled.  max_above_hole_without_any_acks
  plays a similar role, but for the case when measurement drops have
  lead to Bro holding data for a sender for which it will never see
  an ACK (which would be an "ACK above hole"), so it should give up
  on buffering it.

- A portability bug in Bro's parser code has been fixed (Jason Lee).

- STARTTLS for SMTP now causes the SMTP session to be ignored, rather
  than generating a slew of error messages because the session becomes
  unparseable.

- Bro's "watchdog" handler is now less prone to crashing due to
  using non-reentrant library calls (Jason Lee).

- Better error message when reading an input file encounters an error.

- adtrace now prints its usage when invoked without the required
  arguments (Jason Lee).

- A bug in using recursive script functions has been fixed.

- A bunch of whitespace/code layout tweaks.


0.9a4 Wed Sep  8 17:33:54 PDT 2004

- The directory structure of the Bro distribution has changed (Jason Lee).
  The source code is now in a subdirectory, src/, and the scripts
  snort2bro (and snort2bro.cfg) and make-ftp-safe-vocabulary.awk have
  been moved into scripts/.

- "make install" has been revamped (Jason Lee).

- The format of the alert log file has changed.  Fields in it are
  colon-separated.  THIS WILL LIKELY CHANGE SOON.

- The policy for formatting signature matches has been revamped,
  including colon-separated fields in the signature log file
  (Roger Winslow).  THIS WILL LIKELY CHANGE SOON.

- The BRO_ID environment variable has been renamed BRO_LOG_SUFFIX.

- A new flag, -e, lets you specify Bro code to execute via the command
  line (Christian Kreibich).  So, for example,

	bro -r mytrace.tcpdump -e 'redef traditional_conn_format = T' tcp

  will run tcp.bro on the trace "mytrace.tcpdump", but with
  traditional_conn_format redefined to be true.  Note that statements
  have an implicit ';' added to them for convenience.

- A new signature alert, "MultipleSigResponders", is generated if a
  host triggers the same signature on multiple responders.

- Bro now supports "packet profiling", which provides fairly fine-grained
  statistics on number of packets processed, volume, elapsed real/user/system
  time, and change in memory consumption (Holger Dreger).  Three variables
  control the output.  The double pkt_profile_freq controls the frequency
  of output.  The units in which it's interpreted depends on the setting
  of the pkt_profile_mode variable (which is of type pkt_profile_modes,
  an enum).  A value of PKT_PROFILE_MODE_SECS means that statistics
  are generated every pkt_profile_freq seconds; PKT_PROFILE_MODE_PKTS
  means every pkt_profile_freq packets; and PKT_PROFILE_MODE_BYTES, every
  pkt_profile_freq bytes.  The default (PKT_PROFILE_MODE_NONE) means
  to not generate packet profiling.

  Packet profiling is written to the new log file, pkt_profile_file.
  If you "@load pkt-profile", you can turn on packet profiling using
  some handy defaults.

- statistics.bro now reports on how many TCP connections are in
  <originator-state, responder-state> for the different TCP endpoint
  states (SYN sent, SYN ack'd, connection established, etc.).
  Contributed by Holger Dreger.

- tcp_content_delivery_ports_{orig,resp} are now table's of bool rather
  than set's (Ruoming Pang).  The semantics are that if you have a
  tcp_contents event handler, then if the orig/resp port is in the given
  table *and the yield value is T*, then the event will be invoked.  This
  allows you to now explicitly skip over some ports.

- The processing of default values in tables has been changed internally
  (Ruoming Pang).  It's possible this has introduced some subtle bugs
  (as some of these came up during testing).

- A serious bug in Base64 processing has been fixed (Ruoming Pang).

- The NetBIOS and SMB analyzers have been updated in minor ways
  (Ruoming Pang).

- statistics.bro now reports a "lag" figure indicating the elasped
  time between the last expired timer's target expiration time and
  the current packet timestamp (Robin Sommer).  Lag can grow if Bro
  is getting behind in timer expiration due to the setting of
  max_timer_expires.

- Bro's default filter is now "tcp or udp or icmp" rather than
  "tcp or udp".

- alert_info records now have an optional port associated with them
  (for example, to be used to describe scan activity).

- A bug has been fixed in which deleting a table element with an
  associated timer could crash Bro (Robin Sommer).

- A bug that would cause a crash for malformed EPASV directives
  has been fixed (Robin Sommer).

- A bug with inactivity timeouts not being generated for partial
  connections has been fixed (Robin Sommer).

- A bug in synflood.bro has been fixed (Robin Sommer).

- Some tuning adjustments to incremental expiration of table entries
  (Robin Sommer).

- Improved portability to Darwin (Christian Kreibich).

- alert_info records now have additional optional fields, "iconn"
  (associated ICMP connection), "dst" (destination address), and
  "p" (associated port).  The source_is_responder fields has been
  removed.

- The default packet filter now includes "icmp".

- Some memory allocation/free mismatches & minor leaks (Robin Sommer).

- Minor tweaks to ssl.bro (Robin Sommer).

- Bro now supports "null" link layers (Christian Kreibich).

- aux/adtrace contains a program that spits out MAC/IP information
  from traces (Holger Dreger).

- A bug in handling malformed RPCs has been fixed (Scott Campbell).

- The formatting of "weird" messages that have additional parameters
  has been changed to be more regularized with other "weird" messages.

- The new "weird" type "base64_illegal_encoding" takes the place of
  some previously unstructured Base64 "weird" errors.

- A tweak to ftp.bro will give it slightly more consistent results
  for some forms of unusual traffic.


0.9a3 Wed Jul  7 22:06:26 PDT 2004

- Improved ICMP processing, including scan detection (Scott Campbell).

- ICMP "connections" are now considered unidirectional.

- Fixed broken VLAN support (integration of original patch was incomplete).

- Fixed a bug in erroneously generating additional "ContentGap"
  alerts after an initial one.

- Connection durations are now always reported as floating-point decimal,
  never in exponential notation.

- Removed unused time parameter from a bunch of internal calls.

- Fixed some compilation warnings.

- "make clean" now removes generated policy/*.bif.bro files (Christian
  Kreibich).


0.9a2 Fri Jun 11 00:07:04 PDT 2004

- NetBIOS analysis has been extended with a CIFS/SMB analyzer (Ruoming Pang).
  While this is incomplete, it has many important elements.  The corresponding
  events:

	smb_message(c: connection, is_orig: bool, cmd: string, msg: string)
	smb_com_tree_connect_andx(c: connection, path: string, service: string)
	smb_com_nt_create_andx(c: connection, name: string)
	smb_com_transaction(c: connection, is_orig: bool, subcmd: count,
				name: string, data: string)
	smb_com_transaction2(c: connection, is_orig: bool, subcmd: count,
				name: string, data: string)
	smb_com_read_andx(c: connection, is_orig: bool, data: string)
	smb_com_write_andx(c: connection, is_orig: bool, data: string)

  This analyzer is still experimental.

- Greater support for vectors (Umesh Shankar), much of it taken from
  the 'S' language.

  You can use a boolean vector as an index into another vector (providing
  both are the same length) and each 'T' value in the index extracts the
  corresponding element from the indexed vector.  For example, "x[x > 3]"
  returns a vector whose elements are those elements of x that are greater
  than, while if y is a vector of the same length as x then "y[x > 3]"
  extracts those elements of y that have the same position as the elements
  in x that are greater than 3.

  You can also use an arithmetic vector to index another vector.  Each
  element present in the index is extracted.  So, for example:

	global a: vector of count;
	global b: vector of string;

	a[1] = 3;
	a[2] = 3;
	a[3] = 1;

	b[1] = "foo";
	b[2] = "bar";
	b[3] = "bletch";

	print b[a];

  prints:

	[bletch, bletch, foo]

- The new built-ins any_set() and all_set() return true if for a given
  boolean vector any element is true or all of the elements is true
  (Umesh Shankar).  So, for example, "any_set(x < 0)" returns T if
  an element of x is less than zero.

- The new built-in sort() takes a vector as an argument and sorts it
  *in place* (Umesh Shankar).  (The in-place semantics may change in the
  future.)  An optional second argument can be used to specify a
  function to call for comparing elements, and is required for non-arithmetic
  vectors.  For example, the following could be used to sort a vector
  of strings based solely on the length of the strings:

	function string_compare(a : string, b: string): int
		{
		local la = byte_len(a);
		local lb = byte_len(b);

		return (la < lb) ? -1 : ((lb > la) ? 1 : 0);
		}

- The new function order() has the same calling sequence as sort(),
  but instead of returning (and altering in place) the sorted vector,
  it returns a "vector of count" giving the *indices* that if used
  to index the vector will return it sorted.  So, for example,
  given two vectors x and y of the same length (but not necessarily
  of the same type),

	local x_sort_indices = order(x);
	x = x[x_sort_indices];
	y = y[x_sort_indices];

  will assign x to a sorted version of itself and also rearrange y such
  that elements of y that were paired with elements of x originally
  remain paired after the sorting.

- The ICMP analyzer now has a general notion of "context", i.e., the packet
  associated with ICMP status messages such as Unreachable or Time Exceeded
  (Ruoming Pang).  This changes the parameters to the icmp_unreachable
  event.  A new event, icmp_time_exceed, is now also available.

- The tcp_segment even has been replaced by a pair of new events (Ruoming Pang):

	tcp_packet(c: connection, is_orig: bool, flags: string, seq: count,
			ack: count, len: count, payload: string)

  is invoked for each TCP packet.  "flags" is a string containing "SFAPU"
  for the SYN/FIN/etc TCP flags.

	  tcp_contents(c: connection, is_orig: bool, seq: count,
			contents: string)

  is invoked for each chunk of the byte-stream that has been reassembled
  in sequence, providing it satisfies tcp_content_delivery_ports_{orig,resp},
  per the next item.

- You can specify the set of ports for which contents should be reassembled
  for the originator (responder, respectively) stream using the new sets
  tcp_content_delivery_ports_{orig,resp} (Ruoming Pang).  This can be
  useful for user-level stream analysis for protocols not known to Bro's
  event engine.  These controls may change to a "table of bool" in the future,
  in order to support an &default attribute.

- New built-in functions (Ruoming Pang):

	function interval_to_double(i: interval): double
		Converts a value of type "interval" to "double".

	function write_file(f: file, data: string): bool
		Writes the given string to the given file, returning
		F on error.

	function is_ascii(str: string): bool
		Returns T if the given string consists entirely of
		ASCII characters (i.e., in the range 0..127).

	function sqrt(x: double): double
		Returns the square-root of x, or -1 and a run-time error
		if x is < 0.

	function uuid_to_string(uuid: string): string
		Takes a UUID and returns its string representation, where
		UUID = Universal Unique Identifier as defined per
		http://www.opengroup.org/onlinepubs/9629399/apdxa.htm#tagcjh_20

	function string_to_ascii_hex(s: string): string
		Returns the ASCII hex representation of the given string.
		For example, string_to_ascii_hex("foo") returns "666f6f".

	function match_pattern(s: string, p:pattern): pattern_match_result
		Matches the given pattern against the given string, returning
		a record with three fields:

			matched: bool;	# T if a match was found, F otherwise
			str: string;	# portion of string that first matched
			off: count;	# 1-based offset where match starts

		For example,
			match_pattern("foobar", /o*[a-k]/)
		returns
			[matched=T, str=f, off=1]
		because the *first* match is for zero o's followed by an [a-k],
		while
			match_pattern("foobar", /o+[a-k]/)
		returns
			[matched=T, str=oob, off=2]

- Functions that terminate without returning a value when they were declared
  to do so now generate a run-time warning (Christian Kreibich).  Functions
  in the standard set of policy scripts that did this have been fixed.

- The new event non_dns_request(c: connection, msg: string) is generate
  to make the contents of malformed DNS requests available for analysis,
  with the assumption that these are actually some other protocol entirely
  (Ruoming Pang).

- If you redef truncate_http_URI to have a value >= 0, then any HTTP
  URIs generated by the event engine will be truncated to this length
  (Ruoming Pang).  This can be convenient when analyzing traffic that
  generates huge URIs (as do some automated attacks).

- "SEARCH" is now recognized as a standard HTTP method (Ruoming Pang).

- The new event connection_EOF(c: connection, is_orig: bool) is generated
  when one side of a connection closes (Ruoming Pang).

- synflood.bro and the corresponding event engine internals now works
  in terms of probabilities (0.0-1.0) instead of percentages (0-100)
  (Robin Sommer).  The script has had several tweaks, including using
  new_connection() rather than connection_attempt(), which gives it
  quicker response and broader coverage (it'll detect non-TCP flooding,
  too, so "synflood" is now a bit of a misnomer), at the cost of perhaps
  more CPU load.

- A signature for Witty has been added to policy/sigs/worm.sig (Ruoming Pang).

- Makefile now has a "test" target.  Currently this only works for internal
  development (we haven't put together a public test suite yet; that will
  take some time due to the need to make sure no sensitive information leaks).

- The built-in function generator now knows about "double" as a built-in type
  (Ruoming Pang).

- Some generated files have been removed from the Bro distribution since
  they're redundant (Ruoming Pang).

- A bug has been fixed in which contents files might not be correctly
  written upon termination of Bro (Ruoming Pang).

- A bug has been fixed in which UDP connections didn't generate
  new_connection events (Ruoming Pang).

- Support for the Linux "cooked capture" link layer (Ruoming Pang).

- BPF support has been factored into a separate class, BPF_Program, which
  makes for easier portability (Christian Kreibich).

- A serious low-level Dictionary bug has been fixed (Christian Kreibich).

- A bug that could cause Bro to crash if it receives an event from another
  Bro that it isn't able to process has been fixed (Christian Kreibich).

- A bug in set file descriptors non-blocking has been fixed
  (Christian Kreibich).

- A bug that could cause some error messages to generate crashes has
  been fixed.

- The global skip_event_tcp_option has been removed.


0.9a1 Mon Jun  7 01:33:00 PDT 2004

- 0.8a85 is the new STABLE release, to be updated only for bug (and
  portability) fixes.  0.9a1 is the new CURRENT release, which will have
  both bug fixes and new functionality.

- Support for FTP EPRT, EPSV (Holger Dreger).

- Change to timer management to recover memory more quickly (Robin Sommer).

- Tweaks to eliminate a number of compiler warnings (Robin Sommer).

- Statistics now report number of connections terminated due
  to inactivity (Robin Sommer).

- New Makefile target, pub-tar-no-doc builds a public tarball but
  without the (large) doc/ subdirectory.

- Bug fix for identd requests with illegal port numbers.

- The example of a scanning source to skip in scan.bro has been changed
  from an AltaVista robot (now obsolete) to a Google robot.

- Some previously fatal internal errors have now been turned into
  "internal warnings", which Bro is able to continue operating
  in their presence.


0.8a84 Wed May 26 23:33:39 PDT 2004

- autoconf tweaks for portability to Darwin (Christian Kreibich).

- Fixed subtle bug in chunked-IO reads (Christian Kreibich).

- Fixed bug for the "discarder" framework in which specifying an
  IP discarder would cause other TCP/UDP/ICMP discarders not being
  invoked (Christian Kreibich).

- Fatal bug in signature matching fixed (Robin Sommer).

- Missing member variable initialization fixed (Robin Sommer).  (Needed for
  compilation with new versions of gcc.)

- Makefile bug for "make install" fixed.

- Fixed bug that could lead to Bro crashing if an SMTP session
  had data sent after a RST.

- Removed some out-of-date SMTP analysis warning messages.


0.8a82 Tue Apr 27 11:53:24 PDT 2004

- Fixed inactivity timer loop when a packet arrives exactly when
  the timer is set to expire.


0.8a81 Mon Apr 26 22:46:37 PDT 2004

- A bunch of memory leaks fixed (Chema Gonzalez).

- A new HTTP analyzer variable, content_truncation_limit, controls
  how much of an HTTP request/reply contents element (i.e., what's
  passed to http_entity_data - this is *not* the entire content, but
  the next chunk's worth) is logged in the log file.  It defaults
  to 40 bytes.  Setting it to 0 means "log all of it".

- Fix to avoid crashing for malformed RPC requests.

- Improved OpenSSL auto-configuration (Robin Sommer).

- Fix for compiling without OpenSSL.

- A new built-in, double_to_count(), converts a value of type "double"
  to the corresponding "count" (Chema Gonzalez).  We should probably
  add floor(), ceil(), etc.

- Parameterization of trw.bro tweaked (Jaeyeon Jung).


0.8a79 Wed Mar 24 22:02:53 PST 2004

- Bro now has an SSL analyzer, written by Michael Kuhn and Benedikt Ostermaier,
  with further development by Scott Campbell.  It generates the following
  events:

	event process_X509_extensions(c: connection, ex: X509_extension)
	event ssl_X509_error(c: connection, err: int, err_string: string)
	event ssl_certificate(c: connection, cert: X509, is_server: bool)
	event ssl_certificate_seen(c: connection, is_server: bool)
	event ssl_conn_alert(c: connection, version: count, level: count,
				description: count)
	event ssl_conn_attempt(c: connection, version: int)
	event ssl_conn_established(c: connection, version: int,
				cipher_suite: count)
	event ssl_conn_reused(c: connection, session_id: sessionID)
	event ssl_conn_server_reply(c: connection, version: int)
	event ssl_conn_weak(name: string, c: connection)
	event ssl_session_insertion(c: connection, id: sessionID)

  Note, it still has a lot of rough edges; particularly, handling
  non-conformant input.  It also generates unnecessary ContentGap alerts
  due to the way it runs multiple analyzers (SSLv2 and SSLv3) on a single
  connection.  This will be fixed in the fairly near-term future.

- The manual has been updated with chapters on signatures (Robin Sommer)
  and using the interactive debugger (Umesh Shankar), along with a partial
  description of the new SSL analyzer (Michael Kuhn and Benedikt Ostermaier)
  and a number of updates to the documentation of built-in functions (Umesh
  Shankar), though this latter is still not complete since Umesh actually
  contributed this quite a while ago.

- Ruoming Pang has contributed a crude analyzer for DCE/RPC (used for Windows).
  It generates simple dce_rpc_request and dce_rpc_reply events.  It should
  not be considered stable.

- The traditional connection logging format (traditional_conn_format)
  is no longer the default.  The 0.8a70 release notes stated that this
  was the case but this time it really is :-).

- An experimental "vector" type has been added (Umesh Shankar).  A vector
  is an aggregate type.  For example:

	local suspect_hosts: vector of addr;

  You can randomly access elements starting with the first as 1, e.g.,

	suspect_hosts[1] = 10.0.0.8;

  and can also add elements at later postions even if there are gaps:

	suspect_hosts[31337] = 10.0.0.9;

  *The semantics and capabilities of vectors will be changing considerably.*

- Umesh Shankar has developed a framework for generating IDMEF messages.
  Currently it needs a modified version of libidmef, which is not included
  in this distribution.  Contact me or Umesh if you want a copy.

- A new attribute &synchronized causes the given global variable to
  be *synchronized* across concurrent instances of Bro (which are
  intercommunicating via remote.bro).  Any change made by one of them
  to the variable will be reflected (soon after) in the copy at
  the others.  A new event remote_state_inconsistency is generated
  if two instances both change the value before they're synchronized.
  (Contributed by Robin Sommer.)

- trw.bro implements a new scan detection algorithm, Threshold Random Walk
  (Jaeyeon Jung).  It's described in an upcoming IEEE S&P symposium paper.
  The analyzer generates two events:

	TRWAddressScan, # source flagged as scanner by TRW algorithm
	TRWScanSummary, # summary of scanning activities reported by TRW

  TRW is generally much more sensitive than Bro's regular detection algorithm.

- vlan.bro provides support for VLAN encapsulation.  More generally, Bro
  now has support for any sort of constant-offset encapsulation (Vinod
  Yegneswaran).  You specify the header size by redef'ing encap_hdr_size.
  You can also redef tunnel_port to be a UDP port which Bro treats as being
  the encapsulation (in the packet stream - not addressed to it) rather
  than all traffic.

- If you turn on statistics (i.e., via @load statistics) and also redef
  segment_profiling to T, then Bro will generate to the statistics file
  a trace of its "segment" processing.  A segment is a unit of internal
  execution.  Profiles look like:

	1058517499.615430 segment-processing-packet dt=0.000013 dmem=0
	1058517499.615430 segment-draining-events dt=0.000012 dmem=0
	1058517499.615671 segment-expiring-timers dt=0.000010 dmem=0
	1058517499.615671 segment-processing-packet dt=0.000010 dmem=0
	1058517499.615671 segment-draining-events dt=0.000012 dmem=0
	1058517499.615671 segment-policy/conn.bro:282 dt=0.000011 dmem=0
	1058517499.615671 segment-policy/conn.bro:253 dt=0.000012 dmem=0

  The first line states that at the given (packet) timestamp, the event
  engine processed a packet, taking 13 usec of CPU time to do so, and
  not consuming any memory (from the kernel's perspective; this is *not*
  fine-grained memory consumption).  The next lines indicate 12 usec were
  spent draining events and 10 usec expiring timers.  The last two lines
  indicate that the functions at lines 282 and 253 in conn.bro were
  executed, requiring 11 usec and 12 usec, respectively.

  Note #1: timings are just what rusage() reports, so not necessarily
  very accurate for small times.

  Note #2: there's a bug in tracking function line numbers that hasn't
  been ferreted out yet, so they're only roughly correct.

- The inactivity_timeout global has been split into tcp_inactivity_timeout/
  udp_inactivity_timeout/icmp_inactivity_timeout (Robin Sommer).  Using
  this, the default inactivity timeouts for UDP and ICMP have been changed
  from "no timeout" to 10 seconds.  This is needed because otherwise
  analyzing a stream of UDP or ICMP traffic generally gobbles up memory
  quickly and never recovers it; and there's seems little point in trying
  to consolidate long-lived-but-often-inactive UDP/ICMP streams.

- The new policy script cpu-adapt.bro is an extension to load-levels.bro
  (see CHANGES for 0.8a37 below) to adapt the packet filter based on the
  current CPU load. If the load is below cpu_lower_limit (default 40%),
  the load-level is decreased.  If it's above cpu_upper_limit (default
  90%), it's increased.  (Robin Sommer)

- The new policy script hand-over.bro can be used for a new running
  instance of Bro to smoothly take over operation from an old instance,
  i.e., it implements hand-over of state between two Bro instances when
  checkpointing (Robin Sommer). First, all persistent data (i.e. variables
  declared &persistent and connections for which make_connection_persistent()
  has been called) is transferred from the old instance to the new instance.
  Then the old instance terminates itself and the new one starts processing.

  The host from which we want to take over the state has to be added to
  remote_peers_clear (or remote_peers_ssl), setting hand_over to T. The
  host which is allowed to perform a hand-over with us has to be added
  with a port of 0/tcp and hand_over=T. An example for a handover between
  two instances on the same machine:

  @load hand-over
  redef remote_peers_clear += {
        [127.0.0.1, 47756/tcp] = [$hand_over = T],
        [127.0.0.1, 0/tcp] = [$hand_over = T]
        };

  (This interface may be subject to change in the future.)

- New script functions (Robin Sommer):

      function terminate()
          Terminates Bro via SIGTERM.

      function dump_config()
          Dumps Bro's full configuration into state_dir (one file per
          variable/type/function, etc.)

      function send_state(ip: addr, p: port)
          Send all of persistent state to the remote host.

      function set_accept_state(ip: addr, p: port, accept: bool)
          If accept is true, state sent by the given host will be
          accepted (default: false)

      function make_connection_persistent(c: connection)
          Declare the given connection state to be persistent (i.e.
          to be saved upon termination and exchanged by send_state).
          checkpoint.bro uses this to declare some services to be
          persistent by default.

      function is_local_interface(ip: addr): bool
          Returns true if the given address is assigned to a local interface.

- Printing of sets and tables now includes timestamps indicating when the
  element was added (Robin Sommer):

         ID did_ssh_version = {
            [129.187.20.9, F] = 1 @11/01-15:55,
            [212.144.77.26, T] = 2 @11/01-15:55,
            [141.84.116.26, T] = 10 @11/01-15:55,
            [217.232.245.249, T] = 1 @11/01-15:55,
            [217.235.217.149, T] = 1 @11/01-15:55,
            [129.187.39.13, F] = 2 @11/01-15:55,
            [129.187.208.139, F] = 1 @11/01-15:55,
            }

  The format may change in the future, and will probably be made an option.

- Similarly, you can print functions to get both a timestamp of the last
  time the given block was executed and a count of how often (Robin Sommer):

	  ID record_connection = record_connection
	  (@11/01-16:03 #6549)
	  {
	  id = c$id;
	  local_init = is_local_addr(id$orig_h);
	  local_addr = local_init ? id$orig_h : id$resp_h;
		    remote_addr = local_init ? id$resp_h : id$orig_h;
	  flags = local_init ? "L" : "";
	  if (remote_addr in neighbor_nets)
		  (@<never> #0)
		  flags = cat(flags, "U");

	  if ("" == flags)
		  (@11/01-16:03 #2110)
		  flags = "X";

	  is_tcp = is_tcp_port(id$orig_p);
	  ;
	  if (is_tcp)
		  (@11/01-16:03 #6549)
		  {
		  if (c$orig$state in conn_closed || c$resp$state in conn_closed
)
			  (@11/01-16:03 #4739)
			  duration = fmt("%.6g", c$duration);
		  else
			  (@11/01-16:03 #1810)
			  duration = "?";
	  [...]

- You can now specify numbers using hex constants, e.g., 0xabcd = 43981
  (Michael Kuhn and Benedikt Ostermaier).

- A new function, join_string_array(sep: string, a: string_array) concatenates
  strings in 'a' and inserts 'sep' between every two adjacent elements
  (Ruoming Pang).  E.g., join_string_array("", {"a", "b", "c"}) returns
  "a b c", and join_string_array("", a) is the same as cat_string_array(a).

- checkpoint.bro now makes some services persistent by default
  (Robin Sommer).

- The new_packet event now includes both the associated connection
  and a pkt_hdr describing the packet (Robin Sommer).

- The functions functions connect_ssl() and connect_clear() have been replaced
  by a single connect() function taking an additional parameter to
  differentiate the types (Robin Sommer).

- The new function stop_listening() unbinds the listen port (Robin Sommer).

- A new flag packet_filter_default says whether the Bro-level packet-filter
  will by default accept all or reject everything (Robin Sommer).

- Function calls can now be sent to remote Bro's, though there isn't yet
  an interface for accessing this from the script level (Robin Sommer).

- Bro now has an generalized internal framework for serializing objects
  and monitoring access to state (Robin Sommer).

- Better memory allocation accounting (Robin Sommer).

- A minor tweak to the output generated by statistics.bro.

- Improved localization of source code for functions in messages (but
  there are still some bug).

- Autoconf looks for -ltermcap (Robin Sommer).

- Fixes for bugs in the management of table expiration values (Chema Gonzalez).

- A bug in printing "void" values has been fixed (Chema Gonzalez).

- -t bug fixed (Chema Gonzalez).

- A bug has been fixed in which sometimes "expression value ignored"
  was erroneously generated.

- A bug with packet_contents and UDP packets with checksum errors
  has been fixed (Ruoming Pang).

- A memory leak in packet timestamp sorting via packet_sort_window
  has been fixed (Ruoming Pang).

- A bug has been fixed in expanding NULs when printing strings (Ruoming Pang).

- Bug fixes for extracting connection contents via contents.bro (Ruoming Pang).

- Bogus error message "Can't install default pcap filter" when using -F
  removed.


0.8a70 Sun Feb  8 14:19:45 PST 2004

- Bro has a new home page at

	http://www-nrg.ee.lbl.gov/bro.html

  It includes a "wish list" of Bro development projects:

	http://www-nrg.ee.lbl.gov/bro-wishlist.html

- The "match" expression has been completely overhauled (Umesh Shankar).
  It now has the syntax:

	match EXPR1 using EXPR2

  Its semantics are complicated, but it's very powerful (see its use for
  alert filtering below).  EXPR1 can have any type T.  EXPR2 must be of
  type "set[R]", where R is a record type.  R must have the following fields:

	$pred	- type is "function(T): bool".  This is the predicate
		  associated with the record.  It is passed in EXPR1's
		  value and returns true or false.

	$result	- can have any type T'.  This is the value to use when
		  if $pred returns true for EXPR1.

	$priority - type must be arithmetic (count, int, double).  This
		  is the priority associated with the match of EXPR1
		  if $pred returns true.

  The way the expression works is that EXPR1 is evaluated yielding a
  value V.  EXPR2 is then evaluated yielding a set of records whose
  type includes the above fields.  Bro then spins through each of the
  records in the set and tests whether its $pred predicate holds for V.
  If so, it records the given $result and the associated $priority.
  It then returns for the value of the entire expression the $result
  with the highest $priority.

  Here's an example.  The following script:

	global match_stuff = {
		[$pred = function(a: count): bool { return a > 5; },
		 $result = "it's big",
		 $priority = 2],

		[$pred = function(a: count): bool { return a > 15; },
		 $result = "it's really big",
		 $priority = 3],

		[$pred = function(a: count): bool { return T; },
		 $result = "default",
		 $priority = 0],
	};

	print match 0 using match_stuff;
	print match 10 using match_stuff;
	print match 20 using match_stuff;

  when executed will print:

	default
	it's big
	it's really big

  (Note that Bro actually will first evalute $priority before evaluating
  $pred, and if it already has a better (higher) priority result, it
  will not bother calling $pred.)

- There's a new syntax for designating function values (Umesh Shankar).
  It currently only works when associating a function value with a
  record field in a record constructor:

	[$foo(bad_guy: addr) = { launch_counter_DDOS(bad_guy); return 3.14; }]

  is equivalent to:

	[$foo = function(bad_guy: addr): double = {
			launch_counter_DDOS(bad_guy);
			return 3.14;
	}]

  The goal is to make such functions easier on the eye to express.
  The changes are (1) no "function" keywork necessary, (2) no function
  return type necessary (note, it is inferred from the "return" statement
  in the function body; eventually this will work for all functions, not
  just those in record constructors), (3) the '=' sign comes after the ')'
  rather than before the keyword "function".

  Given this syntax, we can rewrite the initialization of match_stuff
  in the example above as:

	global match_stuff = {
		[$pred(a: count) = { return a > 5; },
		 $result = "it's big",
		 $priority = 2],

		[$pred(a: count) = { return a > 15; },
		 $result = "it's really big",
		 $priority = 3],

		[$pred(a: count) = { return T; },
		 $result = "default",
		 $priority = 0],
	};

- The motivation behind these elaborate new mechanisms is to provide a
  powerful and streamlined way to filter alerts.  According, alert.bro
  now processes any alerts generated via ALERT() through a new global,
  alert_policy.  alert_policy's type is set[alert_policy_item], where
  alert_policy_item is:

	type alert_policy_item: record {
		result: AlertAction;
		pred: function(a: alert_info): bool;
		priority: count;
	};

  The idea is that you specify your alert post-filtering by redef'ing
  new elements into alert_policy.  For example, here are two post-filtering
  rules used at LBL to weed out uninteresting alerts:

	# Ignore connections marked as sensitive because they're
	# NTP to otherwise-sensitive hosts (first clause) or they happen
	# to involve 2766/tcp (Solaris Listen), which happens frequently
	# to ftp.ee.lbl.gov if Bro misses the PORT negotiation.
	[$pred(a: alert_info) =
		{
		return a$alert == SensitiveConnection &&
			(a$conn$id$resp_p == 123/udp || # NTP
			 a$msg == /Solaris listen service/);
		},
	 $result = ALERT_FILE,
	 $priority = 1],


	# Ignore sensitive URIs if the request was unsuccessful (code 404,
	# or not answered.)
	[$pred(a: alert_info) =
		{
		return a$alert == HTTP::HTTP_SensitiveURI &&
			a$msg == /.*((\(404)|(no reply)).*/;
		},
	 $result = ALERT_FILE,
	 $priority = 1],

  These rules are part of:

	redef alert_policy += {
		... these records and others ...
	};

  The idea behind them is to demote certain alerts that would ordinarily
  be syslog'd (i.e., the associated action is ALERT_LOG_ALWAYS) to instead
  just be recorded in the alert.$BRO_ID file.  Naturally, there are
  many other possibilities.  For example:

	[$pred(a: alert_info) = {
		if ( a$alert == FTP::FTP_Sensitive &&
		     a$msg == /.*crown_jewels.*/ )
			{
			system("page_the_duty_officer \"crown jewels theft!\"");
			return T;
			}
		else
			return F;
		},
	$result = ALERT_LOG_ALWAYS,
	$priority = 1000],

  would run the program page_the_duty_officer with the argument "crown
  jewels theft!" if an FTP_Sensitive alert was generated and the log message
  included the text "crown_jewels".  More generally, post-filtering needn't
  just be about deciding on how the alert is logged; the processing can
  run programs, update tables, etc., just like any other function call might.

- You can use the new function tally_alert_type in an alert_action_filters
  initialization in order to suppress immediate logging of an alert and
  instead have Bro generate a summary of how many times the given alert
  was seen when it exits.  You can use another new function, file_alert,
  to specify an alert_action_filters initialization that causes the alerts
  to just be written to the alert.$BRO_ID file but not otherwise logged.

  For example:

	redef alert_action_filters += {
		# Just summarize various packet capture glitches.
		[[RetransmissionInconsistency, ContentGap, DroppedPackets,
		  AckAboveHole]] =
			tally_alert_type,

		[RemoteWorm] = file_alert,
	};

  would specify that RetransmissionInconsistency (etc.) alerts should just
  be reported in the log file (log.$BRO_ID) as a total count, and
  RemoteWorm should only be put in the alert.$BRO_ID file, but not
  otherwise logged or counted.

  You could get the same functionality by writing alert_policy records,
  but they're quite a bit bulkier than the above.  Note that
  alert_action_filters entries take precedence over alert_policy
  records, but are evaluated *after* the "match" on alert_policy,
  so if you have predicates in alert_policy with side-effects (like the
  invocation of page_the_duty_officer in the example above), those
  will still happen.

- The alert_info record (which is used in calls to ALERT) now has
  slots for some more additional information:

	user: string;	# can hold an assocaited username
	filename: string;	# an associated filename
	method: string;	# associated HTTP method
	URL: string;	# associated URL
	n: count;	# any associated count/number/status code

  (These are all &optional, so you don't need to specify them if they're
  not appropriate.)  A number of ALERT's in the default policy scripts
  have been changed to include these.  The intent is to add more such
  information in the future.  Ideally, alert_policy records shouldn't
  be doing checks like "a$msg == /.*((\(404)|(no reply)).*/" but instead
  extracting the status code directly from a field of 'a' (which is an
  alert_info record).

- ALERT now fills in the '$id' field in the alert_info record with
  the $id of the associated connection, if the caller didn't suppy
  a $id but did supply a $conn.  Likewise, it will fill in $src with
  the $orig_h from $id (after first filling in $id).  The net result
  is that you can rely on $id and $src being set for any alert that
  has an associated connection.

- The HTTP analysis scripts (policy/http*.bro) have been converted to
  use the "module" facility, similar to how ftp.bro was converted for
  0.8a48.  This may require changing some of your own scripts, generally
  just to add "HTTP::" qualifiers.

- Now that the variables associated with FTP analysis are part of an
  "FTP" module, the "ftp_" prefix associated with:

	ftp_excessive_filename_len
	ftp_excessive_filename_trunc_len
	ftp_guest_ids
	ftp_hot_cmds
	ftp_hot_files
	ftp_hot_guest_files
	ftp_ignore_invalid_PORT
	ftp_ignore_privileged_PASVs
	ftp_log
	ftp_skip_hot

  has been removed, and these are now called:

	excessive_filename_len
	excessive_filename_trunc_len
	guest_ids
	hot_cmds
	hot_files
	hot_guest_files
	ignore_invalid_PORT
	ignore_privileged_PASVs
	log_file
	skip_hot

  To get to them from other scripts, you specify, for example,

	redef FTP::guest_ids = { .... };

  whereas before you had to use:

	redef FTP::ftp_guest_ids = { .... };

- The new connection logging format introduced in 0.8a57 is now the
  default, unless you redef the new variable "traditional_conn_format"
  to be T (Robin Sommer).  Connections using unidentified ephemeral
  ports now have a service of simply "other" rather than other-XXXX.
  The 'U' connection status flag has been removed (unless you're using
  traditional_conn_format).

- Tables can now be directly indexed by records, and indexing using records
  is no longer interchangeable with using a corresponding list of indices
  (Umesh Shankar).  This may require adjustments to existing policy
  scripts.

- Hostnames such as www.google.com now have type set[addr] rather than
  a funky internal list type.

- The new function dump_current_packet(file_name: string) dumps a copy of
  the current packet to the file with the given name, appending it if the
  file already exists (Robin Sommer).  The file is in tcpdump format.
  A handy use for this is in an event handler for signature_match(),
  to record packets that match given signatures.

- The event new_packet() is invoked for each new packet (Robin Sommer).
  It currently doesn't provide the packet contents but soon will in
  a fashion similar to secondary-filter.bro.

- "cf -f fmt" specifies a strtime() format.  -u specifics UTC time rather
  than local time (Mark Delow and Craig Leres).  cf now has a man page
  (Craig Leres).

- Two new variables, backdoor_ignore_local and backdoor_ignore_remote,
  can be used to specify backdoor signatures that should be ignored
  if the server is local/remote.

- A bug has been fixed in which a "next" executed in the final iteration
  of a for loop would mangle the subsequent processing of the outer
  statements (Chema Gonzalez).

- Bug fixes for MIME and Base64 processing (Ruoming Pang).

- pcap.bro now builds its filter in the opposite order (restrict_filters
  first), which can improve packet filtering performance (Robin Sommer).

- A bug in &default has been fixed.

- More SSL autoconf tweaks (Robin Sommer).

- Portability for different pcap_compile_nopcap() calling sequences
  (Chema Gonzalez).

- Some tweaks for a minor reduction in memory consumption.

- A memory leak for secondary packet filters has been fixed.

- The localization of error messages (what script line they correspond to)
  has been improved.


0.8a58 Tue Dec 16 08:55:47 PST 2003

- Compatibility with older versions of libpcap (Chema Gonzalez).


0.8a57 Tue Dec  9 10:14:30 PST 2003

- The format of Bro's connection summaries is changing.  The new format
  looks like

	  1069437569.904605 0.230644 1.2.3.4 5.6.7.8 http 59377 80 tcp 610 275 S3 L

  That is, <timestamp>, <duration>, <originator address>, <responder address>,
  <service>, <originator port>, <responder port>, <originator bytes>,
  <responder bytes>, <connection state>, <flags>.  (Robin Sommer)

  The script variable traditional_conn_format=T specifies to use the old
  format rather than this new one.  This is *currently* the default, but
  will change soon to default to F instead.  If you have comments on this
  new format, we'd like to hear them.

- The SigAction's available in signatures.bro have been extended (Robin Sommer).
  SIG_FILE_BUT_NO_SCAN is like SIG_FILE but without any horizontal/vertical
  processing; SIG_LOG_ONCE logs only an alert only the first time it occurs;
  SIG_LOG_PER_ORIG logs only the first instance of an alert generated by a
  particular originator; SIG_COUNT has been renamed SIG_COUNT_PER_RESP; and
  SIG_SUMMARY suppresses logging of individual alerts but generates a
  per-originator summary.

- A new -p option for snort2bro tells it to only process signatures that
  include matching on payload (Robin Sommer).

- You can now explicitly include or exclude particular SIDs when
  running snort2bro by specifying a configuration file via -c (Robin
  Sommer).  The format is simple, just "include" or "ignore" followed
  by the SID number:

	# sid-526 BAD TRAFFIC data in TCP SYN packet
	ignore	526

	# sid-623 matches a null-flags stealth scan.  Include it even
	# if we build with -p, since it doesn't tend to generate any
	# false positives.
	include	623

  The new "snort2bro.cfg" file gives examples (i.e., the above).

- Bro can now serialize functions and event handlers, meaning that these
  can be passed as values between Bro's and dumped using -g (Robin Sommer).
  One of the main goals in supporting this is to allow in situ alteration
  of the Bro's configuration (e.g., you can edit a function and change its
  functioning and have a running Bro pick up the change without having to
  stop and be restarted).  Such dynamic reconfiguration is experimentally
  supported via -g <dir> (see below).

- &persistent state is now stored in the *directory* given by state_dir
  (default: "./.state"), one file per variable, rather than a single file
  (Robin Sommer).

- Storing &persistent state to disk is now done incrementally: after writing
  each file, there's a delay of state_write_delay (default: 0.1 secs) before
  working on the next file (Robin Sommer). This may introduce small
  inconsistencies, but prevents load spikes that can lead to packet drops.
  Currently, there is no mechanism to incrementally store a single variable
  (like a large table), although there is already some framework in place
  to eventually support this.

- The *experimental* new -g <dir> option dumps the script-level configuration
  (excluding things defined in internal default scripts like bro.init)
  into the directory <dir>. These files may be printed with "bro -x <file>",
  or copied into the state_dir of a running Bro, which will then pick up
  the change if it has loaded checkpoint.bro.  (When picking up changes,
  event handlers are always added, while functions, types, and variables
  replace the current ones).

- Table values are now incrementally expired rather than all at once
  (Robin Sommer).  That is, if the expiration timer goes off and hundreds
  of values should now be expired, the work of doing so is spread over
  chunks of table_expire_size (default: 50) at a time, separated by a
  delay of table_expire_delay (default: 0.1 secs).  This change aims to
  prevent large processing spikes that can lead to packet drops.

- New built-ins sub() and gsub() act like awk's functions of the same
  name, changing substrings (either first, or all) that match a given
  regular expression to a given target string.  (Note, the calling sequence
  differs from the order used by awk.)

- The new auxiliary script aux/scripts/mvlog is a handy way to manage
  checkpointed logs.  See the script for documentation.

- The &expire_func function now takes two arguments.  The second is
  of type "any" and corresponds to the index(es) of the element being
  expired.  To access the individual indices, you use a new assignment form:

	[a, b, c] = index_val;

  (where index_val is the second argument of type "any").  This assigns
  a to the first index, b to the second, and c to the third.  NOTE: the
  use of "any" types here is *temporary* and will be changing in the
  future to a general "tuple" notion.  (Robin Sommer)

- scan.bro and conn.bro have been reworked to consume less memory and to
  support more flexible state expiration (Robin Sommer).

- The new builtin rescan_state() causes Bro to re-read any persistent
  data values (Robin Sommer).

- snort2bro now supports continued lines ("\<newline>") (Robin Sommer).

- The calling sequences of the software_version_found() and
  software_parse_error() events has changed, and a new event,
  software_unparsed_version_found(), is invoked for raw version
  strings (i.e., the version string prior to the event engine
  attempting to parse it into version/major/minor) (Robin Sommer).

- Software version tracking for clients now tracks all versions, not just
  the latest version (Robin Sommer).

- alert_info records now include an optional field event_src, which is the
  source of the event if it was received from an external Bro (Robin Sommer).

- Regular expressions now support {} iteration values of 0, and generate
  better error messages.

- Output generated by icmp.bro is now redirected into an "icmp" log file
  (Robin Sommer).

- autoconf tweaks for configuring OpenSSL on Linux (Ruoming Pang, Robin Sommer).
  Tested on RedHat (thanks to Anton Chuvakin), Debian, FreeBSD, Solaris.

- You can now turn off using OpenSSL even if the OS supports it, via
  configuring with --disable-openssl (Robin Sommer).

- Variable size computations (per global_sizes()) are now more accurate
  (Robin Sommer).

- A bug with combining file encryption and log rotation has been
  fixed (Robin Sommer).

- A problem tracking directionality in signatures fixed (Robin Sommer).

- Bro now continues running if DNS is not functioning (Robin Sommer).

- Rewriter memory use has been significantly reduced (Ruoming Pang).

- Some bugs with -A/-w interaction have been fixed (Ruoming Pang).


0.8a48 Tue Oct 21 15:56:13 PDT 2003

- There is now a mechanism in place for multiple Bro's to communicate with
  one another via sockets (Robin Sommer).  *This is quite experimental at
  this point* and may have significant bugs and/or need significant
  enhancements.

  By loading listen-clear.bro or listen-ssl.bro, an instance of Bro starts
  to listen on a TCP port.  The first of these listens for unencrypted
  connections and the second for connections encrypted via OpenSSL.  To
  connect to a listening Bro, you load remote-clear.bro or remote-ssl.bro.
  For this connection, you specify which events you want to receive by
  giving a regular expression (e.g.  "/http_*/" for all HTTP events),
  although only those events for which you have defined a local handler
  are actually requested.  Communication is uni-directional in the sense
  that for a certain connection the events go only from Bro-A to Bro-B but
  not from B to A (you could set up a second connection for the other
  direction, though).

  The OpenSSL certificate used to authorize remote  Bro's is specified in
  the script variable "ssl_ca_certificate" and the private key and certificate
  for the local Bro via "ssl_private_key".

  If Bro A connects to Bro B, by default it sends over its capture_filter.
  But Bro B uses it only if it has loaded remote-pcap.bro.  This is the
  beginning of structuring inter-Bro trust mechanisms.  It is done via two
  new events, remote_capture_filter(ip: addr, p: port, filter: string) and
  send_capture_filter(ip: addr, p: port, s: string) : bool.

  The implementation forks a second process which does the socket
  communication, so that the main process should not be affected too much.
  The two processes communicate via a pipe.

  You can call is_remote_event() to determine whether the event currently
  being handled was originated by a remote Bro (if T) or the local Bro
  (if F).

  If a connection with a remote Bro terminates (for whatever reason), Bro
  may try to reconnect automatically.

  A new function, get_event_source(), returns a record event_source
  describing the source that raised the last event.

  See doc/ssl.txt for an explanation of how to create the keys/certificates.

- A fledgling Gnutella analyzer has been contributed (Mark Allman).
  It generates the following events:

	event gnutella_text_msg(c: connection, orig: bool, headers: string)
	event gnutella_binary_msg(c: connection, orig: bool, msg_type: count,
					ttl: count, hops: count, msg_len: count,
					payload: string, payload_len: count,
					trunc: bool, complete: bool)
	event gnutella_partial_binary_msg(c: connection, orig: bool,
						msg: string, len: count)
	event gnutella_establish(c: connection)
	event gnutella_not_establish(c: connection)
	event gnutella_http_notify(c: connection)

- Bro now supports a secondary channel for acquiring packets (Chema Gonzalez).
  You access it by redef'ing the new global "secondary_filters", adding
  table[string] of event(filter: string, pkt: pkt_hdr).  The string
  specifies a tcpdump filter; anytime a packet matches the filter
  (including packets that would *not* otherwise be matched by
  capture_filter), then the given event handler is invoked.

  For example,

	  redef secondary_filters += {
		  ["tcp[13] & 7 != 0"] = rst_syn_fin_flag,
	  }

  will invoke rst_syn_fin_flag() anytime a TCP packet is seen for
  which the SYN/FIN/RST bits are non-zero.  The event handler will
  be passed the string "tcp[13] & 7 != 0" (so it can tell which
  of possibly multiple filters matched) and a pkt_hdr value, which
  includes the IP header and, if present, the TCP, UDP or ICMP header.

  Another example, and what motivated the addition, is:

	  redef secondary_filters += {
		["ip[10:2] & 0xffc == 0x398"] = sampled_1_in_1024_packet,
	  }

  which will invoke sampled_1_in_1024_packet() any time the given
  10 bits in the IP checksum match the pattern 0x398.  If the checksum
  field is uniformly distributed then this roughly corresponds to
  1-in-1024 random sampling.  (Chema has also developed BPF additions
  to support true random sampling.)

  See policy/secondary-filter.bro for examples.

- Bro now does a much better job of keeping track of how much memory
  has been allocated for different structures (Robin Sommer).

  This includes more accurate computations for global_size().

  In addition, if you redef "statistics_interval" to be a non-zero time
  interval, then with that periodicity a summary of memory usage (including
  memory used by event engine components) is dumped to the file
  "statistics_file".  In addition, at this point a "do_statistics" event
  is generated.  You can also call the new built-in statistics_update()
  to generate memory statistics on demand.

  The above structure is likely to change in the future.  statistics_interval
  will probably go away, to be replaced by either explicit calls to
  statistics_update() (which you can do on a timer if you like by using
  "schedule"), or by a built-in function that returns a record of all
  the statistics, that you can then format however you want.

- A major memory leak in HTTP analysis has been fixed (Ruoming Pang).

- New attributes &rotate = <interval expression> and
  &postprocessor = <string expression> can be associated with a file
  variable in order to specify how often the file should be rotated to a
  new filename on disk, and, when rotation occurs, the name of a shell
  program to run on the now-older version as a postprocessor (Robin Sommer).

- Similarly, log_postprocessor and log_rotate_interval specify the default
  values for files.  Unless redef'd, these themselves default to the empty
  string (= no postprocessing) and 0.0 seconds (no rotation).  (Robin Sommer)

- A new attribute, &encrypt, applies to variables of "file" type.  It specifies
  that the version on disk should be encrypted, using either the key specified
  as the value of the attribute, or, if no value is specified, using the
  value of the new script variable log_encryption_key.  The key is an OpenSSL
  public key; it's used to then embed a Blowfish session key.  (Robin Sommer)

  A new utility, aux/bdcat/bdcat ("Bro decrypt cat") can be used to decrypt
  the files.

- The internal structure of TCP analysis has been significantly altered.
  Previously, TCP_Endpoint tracked endpoint state and TCP_EndpointContents
  (derived from it) did stream reassembly.  These have now been separated;
  TCP_Endpoint still tracks endpoint state, but TCP_EndointContents has
  become TCP_Contents, and is only loosely coupled with TCP_Endpoint.
  The reason for this change is to lay the groundwork for (1) applying
  an application analyzer to a connection after several packets for
  the connection have already been read, and (2) applying *multiple*
  application analyzers to a single connection.

- Bro now supports the universal hashing if USE_UHASH is defined
  (Ruoming Pang).  Universal hashing is a lighter-weight alternative
  to MD5 hashing that retains the property of being very difficult
  for an attacker to guess.  It comes in two flavors, a 32-bit
  version (which you get with USE_UHASH) and a faster 16-bit version
  (which you get if you also define USE_NH).  Bro will likely switch
  to using these by default in the near future, as their performance
  gain over MD5 is significant.

- New built-in functions srand() and rand() provide access to the
  usual C-style random number seeding & generation (Chema Gonzalez).

- You can now specify server/client addresses to leave in the clear in
  IP address anonymization (via the built-in variables preserve_orig_addr
  and preserve_resp_addr). Correspondingly, the built-in function for
  IP anonymization now takes a parameter to specify the type of the address
  (orig, resp, or other), instead of the method of anonymization
  (seq, random, prefix-preserving).  (Ruoming Pang)

- Trace anonymization now has prelminary support for handling TCP options
  via the new event "tcp_option" (Ruoming Pang).  It is only invoked
  if skip_event_tcp_option is F (it defaults to T).

- A new event, tcp_segment, is similar to the event "packet_content"
  but provides more information: is_orig (directionality), sequence
  number, and acknowledgement number (Ruoming Pang).

- ./configure finds OpenSSL if it's in some standard location.  Otherwise,
  you may specify it --with-openssl=<path>.  If OpenSSL is not available,
  Bro compiles cleanly without and gives warnings if a script tries use SSL.
  (Robin Sommer)

- The internal links in manual/entire.html have been fixed so they
  now work (Chema Gonzalez).

- A new policy script, blaster.bro, detects instances of the W32.Blaster
  worm (Robin Sommer).

- Signature files (for input to the signature engine) now reside in
  policy/sigs/*.sig.  This directory is now on the default $BROPATH.

- sig.ex.ssl-worm.bro and sig.ex.web-rules.bro have been updated
  to reflect changes in keywords (Robin Sommer).  They've been
  renamed to ex.ssl-worm.sig and ex.web-rules.sig and reside
  in policy/sigs/, per the above.

- The module facility has been changed to have its scope limited to
  the current file plus any additional files that are automatically
  processed based on its name plus $BRO_PREFIXES.

- As an experiment, ftp.bro has been modified to a style that includes
  using "module FTP".  Likely other policy scripts will be converted
  in the near future, and their variable names changed accordingly
  (e.g., "ftp_hot_files" will become "FTP::hot_files").

- The new "match" expression has been modified to allow any yield type
  rather than just strings.  It is likely to change significantly again
  soon.

- Iterating over multi-dimensional tables/sets now works (Robin Sommer).
  For example:

    const remote_peers_ssl : table[addr, port] of Peer &redef;
    [...]
    for ( [ip, p] in remote_peers_ssl )
        connect_ssl(ip, p, remote_peers_ssl[ip, p]$retry);

- Checkpointing of persistent state on SIGHUP now happens via bro.init
  (Robin Sommer).  Not tested.

- fmt() now supports %x for hexadecimal formatting (Chema Gonzalez).

- alert.bro logs the source for remote alerts; by redefining the new
  "event_source_description: string" locally, you can specify how a
  Bro is to refered to on the remote side.  (Robin Sommer)

- software.bro now tracks HTTP clients, too (Robin Sommer).  This
  will be extended in the near future.

- Some FreeBSD 5.X porting tweaks (Sergey Osokin).


0.8a37 Wed Sep  3 23:20:21 PDT 2003

- A new attribute, "&persistent", indicates that a global variable's
  state should persist on disk (Robin Sommer).  Currently, they
  reside in the file "state.bst".

  For example, given the following script:

	global a = 0 &persistent;

	event bro_init()
		{
		print ++a;
		}

  then every time you run it, a increases by one.

  You can dump the state file using "bro -x state.bst <scriptname>".
  (There's also a partially-implemented XML dumper which you invoke via
  "bro -X state.bst <scriptname>".)

  If you send Bro a HUP signal, then it will synchronize all persistent
  state to disk (checkpoint) at that point.  More generally, the policy
  script can do so at any time by calling the new built-in checkpoint_state().

  By including the new policy script "checkpoint", you can have Bro
  automatically checkpoint every time checkpoint_interval elapses
  (default 15 minutes).

- You can also record events to disk by loading the "capture-events"
  policy script, or calling the new capture_events() built-in (Robin Sommer).
  The events are written to the file events.bst.  You can dump these
  using bro -x (or -X), as above.  You can also replay them using
  "bro -R <file>".  By replaying them using the same policy script as
  generated them, you should get the same output.  For example:

	bro -r tracefile myscript capture-events

	...

	bro -R events.log myscript

- An experimental module facility has been added (Umesh Shankar).

  The module facility implements namespaces. Everything is in some namespace
  or other. The default namespace is called "GLOBAL" and is searched by
  default when doing name resolution. The scoping operator is "::" as in
  C++. You can only access things in the current namespace, things in the
  GLOBAL namespace, or things that have been explicitly exported from a
  different namespace. Exported variables and functions still require
  fully-qualified names. The syntax is as follows:

  module foo;  # Sets the current namespace to "foo"
  export {
	int i;
	int j;
  }
  int k;

  module bar;
  int i;

  foo::i = 1;
  bar::i = 2;
  print i;    # bar::i (since we're currently in module bar)
  j = 3;      # ERROR: j is exported, but the fully qualified name
              #        foo::j is required
  foo::k = 4; # ERROR: k is not exported

  The same goes for calling functions.

  One restriction currently in place is that variables not in the "GLOBAL"
  namespace can't shadow those in GLOBAL, so you can't have

    module GLOBAL;
    global i: int;

    module other_module;
    global i: int;

  It is a little confusing that the "global" declaration really only means
  that the variable i is global to the current module, not that it is truly
  global and thus visible everywhere (that would require that it be in
  GLOBAL, or if using the full name is ok, that it be exported).  Perhaps
  there will be a change to the syntax in the future to address this.

  The "module" statement cuts across @load commands, so that if you say:

	module foo;
	@load other_script;

  then other_script will be in module foo. Likewise if other_script changes
  to module bar, then the current module will be module bar even after
  other_script is done.  However, this functionality may change in the future
  if it proves problematic.

  The policy scripts in the Bro distribution have not yet been updated to
  use it, but there is a backward-compatibility feature so that existing
  scripts should work without modification. In particular, everything is
  put in GLOBAL by default.

- The hooks are now in place for communicating events between running
  Bro's.  An experimental implementation of doing so (written by Robin
  Sommer) will be integrated shortly.

- A side-effect of getting those hooks in place is that event handlers must
  now be directly specified (by naming them) rather than indirectly
  (for example, by indexing a table whose value yields an event handler).
  This may be fixed soon.

- An experimental "match" expression scans a list of predicates to find
  the first one that's both true and has the highest priority (Umesh Shankar).

  The general form is:

	match some_record on {
		[ priority, class_name, predicate ],
		...
	}

  where "predicate" is evaluated in the context of the value (and type) of
  some_record.  For example, if some_record has a field "num_alerts"
  then predicate could be "$num_alerts > 5".

  "priority" is a non-negative integer (i.e., of type "count"), and,
  for now, "class_name" is a string.

  For example,

	global c: conn_id;

	c = [ $orig_h = 0.0.0.0, $orig_p = 0/tcp,
	      $resp_h = 1.1.1.1, $resp_p = 1/tcp ];

	print match c on {
		[ 2, "emptyweb", $orig_h == 0.0.0.0 && $resp_p == 80/tcp ],
		[ 1, "emptyhost", $orig_h == 0.0.0.0 ],
		[ 0, "should not match", 1 == 0 ]
	};

  will print "emptyhost".

  The initial intent behind this is to eventually provide more flexible
  customization of alert processing, though it clearly has broader
  applicable.  *It is very likely that the semantics and perhaps the syntax
  of "match" will change in the near future.*

- Bro's packet filter is now computed via pcap.bro (which is automatically
  included via bro.init).  It uses two new built-ins:

	precompile_pcap_filter(id: PcapFilterID, s: string):  bool
	install_pcap_filter(id: PcapFilterID): bool

  The first is for precompiling a pcap filter so it can be installed
  or turned off dynamically. Associating an ID with the filter, you can
  then activate the filter by calling the second function (installing a
  new filter replaces the current one).  (Robin Sommer)

  Be default, pcap.bro is responsible for building a pcap string based on
  the capture/restrict_filters defined by the various analyzers. It compiles
  and installs this filter, so there is no observable difference in usage
  to the old implementation, except capture/restrict_filter are now *tables*
  rather than strings, and are written as plural rather than singular.
  So the analyzers need to define something like this:

	[finger.bro]
	redef capture_filters += { ["finger"] = "port finger" };

  This then allows "finger" to be used as the name for the corresponding
  filter element (see the next item).

- load-level.bro is an experimental policy script for allowing Bro to
  shed or add on load (in terms of which protocols it analyzes).  It
  provides three interface functions for setting the current loadlevel:

	# level is LoadLevel1, ..., LoadLevel10
	function set_load_level(level: PcapFilterID): bool

	function increase_load_level()
	function decrease_load_level()

  load-levels.bro defines ten different load levels (from 1 to 10, with 10
  being the one with the most load imposed on the system), which are
  configured by defining capture_load_level/restrict_load_levels: Example:

	redef capture_load_levels += {
		["dns"]          = LoadLevel1,
		["smtp"]         = LoadLevel2,
		["http-reply"]   = LoadLevel3,
		["http-request"] = LoadLevel8,
	};

  This means for example: "include the capture_filter associated with
  'http-reply' if the current load level is 3 or below".  There's a similar
  mechanism for restrict_filters:

	redef restrict_filters += {
		["cs-only"] = "net 131.159.0.0/16",
	};

	redef restrict_load_levels += {
		["cs-only"] = LoadLevel7,
	};

  This applies the given restrict_filter if the current load level is 7
  or *above*.

  The pcap filters belonging to the ten load levels are built and pre-compiled
  on startup. The three functions shown above just call install_pcap_filter()
  then.  (Robin Sommer)

- drop-adapt.bro tries to adjust the load level based on the current
  packet drop rate (Robin Sommer).

- synflood.bro is an experimental policy script for detecting SYN floods.
  It is not yet documented, other than brief comments in the script.
  (Robin Sommer)

- Subnet lookups now use Patricia trees instead of brute-force, which should
  make them significantly more efficient for large lists, e.g., local nets.
  (Robin Sommer)

- Due to the addition of modules, which use a C++-style "::" scope
  designator, you now need to use "0x" to introduce IPv6 address constants
  that use "::" to specify a series of empty octets.  For example, you
  used to be able to specify "deadbeef::cafe", but now this needs to be
  "0xdeadbeef::cafe".  Note that "1::2" can still be written without needing
  a "0x" prefix; it's just hex constants that start with letters that need
  the prefix.

- A new built-in, escape_string(), takes a string and returns a copy of
  it that uses escape sequences for any problematic characters (Robin Sommer).

- A number of low-level bug fixes and portability tweaks (Robin Sommer,
  Ruoming Pang, Christian Kreibich, Chema Gonzalez).

- A new timer, status_update_timer, fires for each ongoing connection
  every connection_status_update_interval seconds (default = 0 seconds,
  which means "don't fire).  (Robin Sommer)

- An additional Bro-level packet filter can filter/sample packets
  based on their src/dest ip/subnet (using a Patricia tree for
  efficiency; Robin Sommer). install_src_addr_filter(ip, flags, p) drops
  packets originating from ip with probability p (0..100) if none of the
  given TCP flags is set. install_src_net_filter, install_dst_addr_filter
  and install_dst_net_filter" work similarly.  The corresponding "uninstall_*"
  functions remove the filters again.

- The @if/@else/@endif mechanisms have been made more powerful (Robin Sommer).

- New configure option --enable-debug to compile without optimization
  (Robin Sommer).

- Small tweaks to the mpatrol support (Robin Sommer).

- SMTP is now one of the services which can use a small inactivity timeout
  in inactivity.bro (Robin Sommer).

- Alerts for signatures which are triggered by worms may be suppressed if
  we already know that the triggering host is indeed infected by a worm
  (Robin Sommer).

- Matches of a signature can now be counted per destination host
  (Robin Sommer).

- snort2bro now ignores sid-526 ("BAD TRAFFIC data in TCP SYN packet";
  Robin Sommer). Due to different semantics of Bro (matching stream-wise)
  and Snort (matching packet-wise) this signature generates a lot of
  false positives.


0.8a34 Sun Jul 13 09:11:32 PDT 2003

- The new "subnet" type corresponds to a CIDR prefix (Robin Sommer).
  You can use the '/' operator to convert an address to a subnet.  The
  "in" operator tests whether an address matches a subnet (e.g., "1.2.3.4
  in 1.2.255.255/16" yields T).

  You can index tables with index type "subnet" using addresses to retrieve
  the table entry with the longest-matching prefix for that address, and
  a number of tables/sets in the default scripts have been converted to
  this form.  So for example the local_16_nets and local_24_nets variables
  have been replaced by local_nets, which has type "set[subnet]",
  is_local_addr() now refers to it, and skip_scan_nets_{16,24} have
  likewise been consolidated into skip_scan_nets.

  One present deficiency is that subnets can't be used in tables/sets
  with multiple indices.  Fixing this is going to take some time ...
  Another deficiency is that the longest-match lookup is not very
  efficient.

  Caution: this feature has been only briefly tested with IPv6.

- ALERT now generates an event "alert_action" which includes the alert_info
  and AlertAction associated with an alert.  You can define your own handler
  to further customize alert processing.

- The "snort2bro" conversion utility has been extended to recognize some
  of the new Snort options (depth, distance, within; Robin Sommer).  For
  example:

      alert .... ( ... content:"|00 01 86 B8|";
		       content:"|00 00 00 02|"; distance:4; within:4;
		       content:"%x %x"; distance:16; within:256;
		       ... )

  is converted to:

      signature sid-1891 {
	  ...
	  payload /.*\x00\x01\x86\xB8.{4}\x00\x00\x00\x02.{16}.{0,251}%x %x/
	  }

  Note that not all of the new Snort options are supported yet.

- You can refer to script variables within signatures (Robin Sommer).
  For example, the following is now possible:

	signature sid-547 {
	  ip-proto == tcp
	  src-ip != local_nets
	  dst-ip == local_nets
	  dst-port == 21
	  event "FTP \"MKD  \" possible warez site"
	  tcp-state established
	  payload /[mM][kK][dD]  /
	  }

  This makes the signatures independent of the environment ("local_nets"
  in the above examle).  snort2bro now converts some well-known Snort
  variables into corresponding Bro variables, a number of which are listed
  in policy/site.bro.

- The default action for signature matches is now logging insted of only
  writing it to a file (Robin Sommer).

- You can now use the '^' and '$' regular expression operators inside
  subpatterns.  For example, /foo$/ | /^bar/ works now.

- You can now use "msec" for milliseconds (in addition to "usec" for
  microseconds, "sec" for seconds, etc).

- The log_file_name and open_log_file functions are now redef'able
  if you want to change their behavior.

- Bro now exits more cleanly upon exhaustion of memory (Robin Sommer).

- A bug was fixed for the case of getrlimit() returning RLIM_INFINITY for
  the maximum number of open files.

- Numerous additions of std:: scoping to address porting issues (Robin Sommer).

- gcc 3.X and Solaris portability fixes.

- A new event RemoteWorm is the complement of LocalWorm (Robin Sommer).

- A bug in which the FTP analyzer would complain about failing to be able
  to look up a connection has been fixed (Robin Sommer).

- You can now configure Bro using "--with-mpatrol" to activate MPatrol
  debugging hooks.  When built in this mode, -m dumps the leak table
  and -M shows unfreed parts of the heap.  These dumps can also be
  triggered using SIGUSR1 / SIGUSR2, respectively.  (Robin Sommmer)

- A script function get_contents_type() which returns the type of reassembling
  being done for a connection (none/orig-side/resp-side/both; Robin Sommer).

- A minor bug fix for the regular expression matcher (Robin Sommer).


0.8a32 Thu Jun 12 23:33:21 PDT 2003

- The low-level hash functions have been reimplemented to use HMAC-MD5 to
  counter the "algorithmic complexity attacks" discussed in the USENIX
  Security 2003 paper by Scott Crosby & Dan Wallach
  (http://www.cs.rice.edu/~scrosby/hash/) (Ruoming Pang and Vern Paxson)

- Ruoming Pang has made extensive changes to the anonymization/transformation
  framework.  A paper describing it will appear in SIGCOMM 2003, and will
  be included with the Bro distribution once the final copy is ready.

- Internal hash tables now resize incrementally, and more quickly.
  This makes a big difference in avoiding lengthy processing pauses when
  processing large volumes of traffic. (Craig Leres)

- gcc 3.1 and Linux portability tweaks.

- The calling sequence of http_request() has changed.  It now includes both
  the original URI and the escaped URI.  This was made explicit because
  we found that if only the original was passed in, it was too easy to forget
  to expand the escapes in it; and if only the escaped was passed in,
  some attacks could be missed. (Ruoming Pang)

- Signature rules can now refer to "finger" payload as well as HTTP and FTP
  (Robin Sommer).

- The signature engine now includes keywords "dst-ip", "dst-port",
  "ip-proto", "src-ip", "src-port".  (Robin Sommer)

- Packet sorting now defaults to off.

- The FTP analysis now attempts to track the current directory (Ruoming Pang).

- A number of scan detection additions have been added (Scott Campbell):

	activate_landmine_check = F
		if T, then access to more than landmine_thresh_trigger
		addresses in landmine_address (a set of addresses)
		constitutes a scan

	activate_priv_port_check = T
		if T, then inbound access to more than priv_scan_trigger
		privileged ports (i.e., ports < 1024) is considered a port
		scan.  You can exclude particular services from the count
		via troll_skip_service, which defaults to { smtp, ftp,
		ssh, 20/tcp, http }.

- The SMTP analysis now includes a new alert, HotEmailRecipient, which
  is triggered by mail sent to any of the addresses specified in
  the pattern hot_recipients (which defaults to a bare /@/).

- The new built-in cat_string_array_n() is like cat_string_array() except
  it takes starting and ending indices as additional arguments (Ruoming Pang).

- The new built-in sort_string_array() takes a string array and returns
  a sorted version (Ruoming Pang).  It currently uses an N^2 algorithm
  so shouldn't be used on large arrays.

- The new built-in subst() substitutes all instances of a given pattern
  in a given string with another string (Ruoming Pang).  For example,
  subst("fooboo", /oo/, "xxx") returns "fxxxbxxx".

- The new built-in cut_tail() trims characters from the end of a string
  (Ruoming Pang).  For example, cut_tail("fooboo", 2) returns "foob".

- sub_bytes() can now take a negative argument, which is with respect to
  the end of the string rather than the beginning (Ruoming Pang).

- The new built-in md5_hmac() returns an HMAC-MD5 hash of the given string
  (Ruoming Pang).  The HMAC secret key is generated from available entropy
  when Bro starts up, or it can be specified for repeatability using
  the new -K flag.

- The new built-in split_all() is like split() except that the returned
  array also includes the parts of the string that match the pattern
  used for the splitting (Ruoming Pang).

- The new built-in split_n() splits up to a given number of instances,
  optionally returning the parts matching the split pattern (Ruoming Pang).

- The new built-in split_complete() is the most general of the split functions
  (the others are written in terms of it).  It splits a given string
  with separators that appear in either a given pattern or a given set
  of strings, optionally including the text matching the separators in
  its return value, and optionally limiting the number of matches to
  a specified maximum.  (Ruoming Pang)

- The new built-in to_string_literal() returns an escaped version of a string
  suitable to feeding into Bro's parser.  For example, calling it on
  "foo\nbar" (where '\n' is an embedded newline) returns "foo\x0abar".
  (Ruoming Pang)

- subst_substring() has been renamed subst_string (Ruoming Pang).

- unescape_URI() no longer takes the connection as an argument (Ruoming Pang).

- config.guess and config.sub updated

- String escape expansion has been refined (Ruoming Pang) and some bugs
  fixed.  It now supports a format that's consistent with Bro string literals.

- Scanning of octal and hex escape sequences now stops after 3 or 2
  characters, respectively.  For example, "\0007" now expands to a
  NUL followed by the character '7'.  (Ruoming Pang)

- Bug fixes for handling of bare <CR> and <LF>'s when <CRLF> expected
  (Ruoming Pang), and associated "weird" events.

- A bug in signature matching reassembly has been fixed (Robin Sommer).

- A bug in reporting "bad connection size" for connection sizes > 2GB
  has been fixed.

- A bug in computing sizes for large partial connections has been fixed.

- A bug in delayed generation of connection closed events has been fixed.

- A framework has been added for compression of some "weird" events
  to only be generated once-per-connection (Ruoming Pang).

- Some of the "weird"'s generated by the HTTP analyzer have been regularized.

- Some memory management fixes.

- A performance problem in allocating List objects was fixed.

- The copyright dates have been updated (Craig Leres).


0.8a22 Wed Jan 15 16:47:18 PST 2003

- There is now a "conditional compilation" feature similar to C's preprocessor
  (contributed by Robin Sommer).  "@if (expr)" will include the remaining
  input lines up to "@endif" only if the given expression evaluates to true.
  "@ifdef (id)" will do so only if the given identifier has been defined,
  and "@ifndef (id)" only if it has not been defined.  There's currently
  no "@else" directive, and conditionals cannot be nested.  Both of these
  will be added in the future.

- New built-in functions (contributed by Ruoming Pang): parse_dotted_addr
  takes a string in A1.A2.A3.A4 form an returns a corresponding "addr" value.
  unescape_URI takes a URI that possibly has embedded escape sequences
  ("%61" for 'a') and expands the sequences.

- The URIs in HTTP requests are no longer automaticaly escaped.  Instead,
  they need to be manually escaped using unescape_URI.  As this is likely
  error-prone (users forgetting to do so), the plan is to change the
  calling sequence of http_request in the near future so that *both* the
  escaped and the unescaped URI are passed in.

- A number of g++ 3.0 porting tweaks have been added (thanks to Sean Irvine).

- The term "rule" has been systematically changed to "signature" (Robin Sommer).

- The functionality of monitoring packet drops has been moved into its
  own policy script, netstats.bro (Robin Sommer).

- A number of rewriter and location bugs have been fixed.

- Some missing HTTP "weird"'s have now been included in weird.bro.


0.8a21 Thu Nov 28 23:31:38 PST 2002

- A new mechanism will reorder packets within a window of packet_sort_window
  if due to their sequence numbers they appear to be out of order.  This
  can help a great deal when reading from multiple NICs.  (Contributed by
  Ruoming Pang.)

- A bug in regular-expression matching in which for example
  "2a1" == /[0-9]+/ would evaluate as true was fixed (Ruoming Pang).

- There's now a rewriter/anonymizer for FTP (Ruoming Pang).

- The rewriter/transformation framework now supports a notion of
  delaying transformation decisions until later (Ruoming Pang).

- An incompatibility with bison 1.75 has been identified.  The problem
  is that bison 1.75 generates incomplete location information for empty
  grammar productions (those for which the RHS is empty).  This will
  be fixed soon.

- Some bugs in the signature engine have been fixed (Robin Sommer).

- The sources no longer use the not-fully-portable hash_map template
  (Umesh Shankar).

- Some bugs with the debugger getting confused about source line number
  locations, and also with the -t option, have been fixed (Umesh Shankar).

- If a content gap occurs wholly inside an HTTP entity, then the analyzer
  will skip over the entity and continue processing any subseqeuent HTTP
  items, rather than giving up on processing the connection (Ruoming Pang).

- The following new built-in functions have been contributed by
  Ruoming Pang:

	function cat_string_array(a: string_array): string
	function split_all(str: string, re: pattern): string_array
	function strstr(big: string, little: string): count
	function subst_substring(s: string, from: string, to: string): string
	function int_to_count(n: int): count
	function fmt_ftp_port(a: addr, p: port): string


0.8a20 Sun Nov 17 20:09:31 PST 2002

- This is the first "public" Bro release (one accessible directly via
  the Web).  The README has been correspondingly updated.

- The user manual has been significantly expanded, and source for it is
  now included in the release.

- Some "active mapping" bug fixes (contributed by Umesh Shankar).

- The configuration script now checks for the presence of the necessary
  BIND libraries/headers, and also for bogus --enable options (contributed
  by Craig Leres).

- backdoor.bro now includes a tcpdump filter for detecting (some) KaZaA
  traffic.

- http-reply.bro now tracks the sizes of request/response entities.

- http-request.bro now treats an URL accessing variants of /etc/netconfig
  as sensitive.


0.8a18 Sun Oct 27 15:28:23 PST 2002

- Improvements to the performance of the SMTP analyzer (Ruoming Pang).
  A new function, skip_smtp_data, skips over the content of an SMTP
  message.

- If you're doing trace rewriting and specify -w but don't specify -A,
  then the rewritten traffic goes to the -w save file, along with the
  usual non-transformed traffic (Ruoming Pang).  If a connection is
  being transformed (which is specified by the new rewriting_smtp_trace
  and rewriting_http_trace globals), then only its transformed packets
  are written to the -w file; not the original packets.  This allows
  a potentially large degree of trace compression.  There's also a
  new variable omit_rewrite_place_holder, which directs that when rewriting
  a trace, packets that are placeholders for now-missing packets (because
  you've excised content) are *not* generated.  This diminishes the
  timing fidelity of the transformed trace, but also can save a large
  amount of space.

- SMTP relay analysis is now standalone from regular SMTP analysis
  (Ruoming Pang).

- Some memory management and error propagation fixes (Ruoming Pang and
  Vern Paxson).


0.8a16 Wed Oct 23 23:48:40 PDT 2002

- "--enable-brov6" was broken by some recent changes.  It now works again.

- Some "make distclean" tweaks.

- Error checking for "redef enum".


0.8a15 Tue Oct 22 00:02:51 PDT 2002

- Fixed Makefile bug which left out wide-spread dependencies on
  event.bif.netvar_h.


0.8a14 Mon Oct 21 01:16:46 PDT 2002

- The "add" statement has been implemented.  This allows you to add
  elements to sets.  A bunch of policy scripts that used to use
  "table [] of bool"'s for this purpose have been modified to instead
  use sets.

- You can now extend the elements of an enum using "redef enum foo += { ... }"
  where the additional names are listed in the braces (contributed by
  Umesh Shankar).  A number of policy scripts have been tweaked to use
  this for the (fairly) new Alert enum.  This allows you to create new
  Alert's without alert.bro having to know about them.

- Some bugs in identifying error locations have been fixed.

- -A now supports anonymizing IP addresses (contributed by Ruoming Pang).
  This includes four new functions: preserve_prefix, preserve_subnet, and
  preserve_net (which direct that the corresponding prefix/net/subnet
  not be anonymized) and anonymize_addr (which returns the anonymized
  version of the given address.

- Some bugs in HTTP -A have been fixed (thanks to Ruoming Pang).

- The beginnings of support for CIDR prefixes has been added by
  Ruoming Pang.

- Porting tweaks (use of map's rather than hash_map's, courtesy of
  Umesh Shankar; libedit comments fixed).

- http-detail.bro has gone away.

- Some more copyright and $Header -> $Id fixes.

- There is now a function string_cat() which concatenates two strings
  and returns the result (contributed by Ruoming Pang).  This function
  is useful in place of the regular cat() because the latter will
  expand escape sequences etc.  It probably shouldn't, and that may
  change in the future.


0.8a11 Sun Oct 13 10:53:07 PDT 2002

- The framework for defining built-in functions has been extended
  so it's also now the way to specify the interface to event handlers.
  See event.bif.  (Contributed by Ruoming Pang)

- A new policy script, http-abstract.bro, can be loaded when doing HTTP
  transformation (via http-rewriter.bro), contributed by Ruoming Pang.
  It trims HTTP text responses to the first "http_abstract_max_length" bytes.

- A new built-in, skip_http_entity_data, specifies that the entity
  associated with the current HTTP request/response should be skipped
  (Ruoming Pang).

- More changes have been made to the mechanisms for tracking source
  file locations in order to pinpoint error messages.  If you encounter
  problems, please let me know.

- If you try to configure but your resolve library lacks res_mkquery(),
  this is now a fatal error (Craig Leres).  Ideally, someone will modify
  DNS_Mgr.cc to work even if only blocking DNS is available ...

- In most (not all, yet) of the sources, the copyright notices have
  been updated, $Header has been changed to $Id, and config.h is included
  (Craig Leres).


0.8a10 Tue Oct  8 16:05:42 PDT 2002

- The way that Bro tracks the source-file location associated with
  different objects has been tweaked, with a resulting savings of
  about 10% in Bro's memory use and a smidgen of CPU time, too.

- Built-in functions now are better about identifying source-file locations
  associated with errors.

- The http.$BRO_ID log file format has changed to no longer track
  individual request streams.  (So it is now closer to how it used
  to operate in 0.7.)

- The autoconf setup has been tweaked so that you use:

	--enable-brov6              enable IPV6 processing
	--enable-activemapping      enable active mapping processing
	--enable-expire-dfa-states  enable DFA state expiration

  rather than --with-XXX as was the case in 0.8a9.


0.8a9 Mon Oct  7 10:15:12 PDT 2002

- A bunch of configuration/autoconf/portability tweaks (Craig Leres).
  These include compiling under Solaris.

  When running ../configure, you can now specify:

	--with-brov6              enable IPV6 processing
	--with-activemapping      enable active mapping processing
	--with-expire-dfa-states  enable DFA state expiration

  You no longer set these via editing the Makefile.

- Some bugs with HTTP 1.0 keep-alive connections fixed (Ruoming Pang).

- The "hf" suite of utilities is no longer distributed in aux/.  Instead,
  get it directly from ftp://ftp.ee.lbl.gov/hf.tar.gz.

- bro_logchk has been renamed bro-logchk.pl and is now distributed in aux/
  (it was supposed to be before, but was inadvertantly left out of the list
  of distribution files).


0.8a7 Fri Oct  4 22:24:30 PDT 2002

- HTTP reply analysis has been split into a number of policy script files:

	http-body.bro
	http-detail.bro
	http-entity.bro
	http-event.bro
	http-header.bro
	http-reply.bro
	http-rewriter.bro

  so you can readily trade off how detailed the HTTP processing is vs.
  the CPU/memory it consumes (contributed by Ruoming Pang).

- Bro now generates login_prompt events when the $TTYPROMPT environment
  variable is passed during a login session, in order to detect the
  recently publicized Solaris remote /bin/login exploit.

- Ruoming Pang has extended the framework for defining "rewriter" functions
  to now also serve as the way to define general built-in functions.

- bro.init has been rearranged to have a more logical structure (courtesy
  of Ruoming Pang).

- Craig Leres contributed a number of portability & autoconf tweaks.

- Craig Leres has extended nb_dns.{h,c} to support IPv6 DNS lookups.
  Bro does not yet take advantage of these extensions.

- The beginnings of portability to gcc 3.2 were added.  There unfortunately
  is more work to do here!

- The README has finally been updated to have more correspondence with
  the 0.8 release.


0.8a6 Wed Oct  2 18:58:12 PDT 2002

- Upgrade to autoconf 2.53 (Craig Leres).


0.8a5 Tue Oct  1 19:04:53 PDT 2002

- The regular expression matcher how has a mechanism to stop scanning when
  no further match is possible (Robin Sommer).  If you find problems with
  regular expression matching, especially if you're using EXPIRE_DFA_STATES,
  please let us know!

- Rule/signature files are now searched for using $BROPATH (Robin Sommer).
  In addition, you can define a list of signature files to incorporate
  using the new global "signature_files".  For example,

	redef signature_files += "web-stuff"

  will look for web-stuff and web-stuff.sig along $BROPATH.

- The tcp_deliver_undelivered variable is now tcp_match_undelivered and
  only applies to delivering otherwise-undelivered data to the signature
  matcher; such data is *not* delivered to any associated analyzer.
  (Robin Sommer)

- The framework for tracking version numbers now allows negative as
  well as positive versions (Robin Sommer).


0.8a4 Tue Oct  1 15:54:58 PDT 2002

- Support for extracting the contents of HTTP replies (and POST's),
  and for transforming/anonymizing HTTP traffic, contributed by Ruoming Pang.

- Some minor internal tweaks to the timer management to help track patterns
  of timer expiration.


0.8a3 Mon Sep 23 22:48:07 PDT 2002

- HTTP reply handling refined in policy scripts.

- New built-in functions to_int(), to_count() convert strings to
  integers/counts.

- Bug fixes for DNS_Mgr and IPv6.

- AckAboveHole alerts now ignored (just written to alert.$BRO_ID) by default.

- Packets belong to ignored connections (for example, partial connections if
  partial_connection_ok is false) are no longer recorded to the save file.

- Some minor formatting/naming tweaks.


0.8a1 Sat Sep 21 22:09:23 PDT 2002

- IPv6 support enabled if you build using -DBROv6.  Deficiencies: Bro
  doesn't yet look up hostnames for AAAA records; no handling of extension
  headers (if you have traces of these, please send them to me!); no
  handling of FTP PORT/PASV w/ IPv6 addresses (again, if you have traces,
  please send them!); DNS analyzer doesn't understand AAAA yet (again,
  please send me traces!); you have to change the capture_filter line
  in tcp.bro (as indicated in the script) in order to process TCP traffic,
  due to deficiencies in libpcap's support for IPv6.

- Bro is migrating towards a more structured way of handling log messages /
  alerts.  Analyzers now @load alert.bro, which has a function ALERT()
  for processing alerts.  Soon this function will provide a variety of
  filtering/processing hooks; expect changes.

- Bro now has an HTTP response analyzer (contributed by Ruoming Pang).
  The HTTP policy scripts have been split up into http.bro (just general
  definitions), http-request.bro (handles requests; loaded by http.bro),
  http-reply.bro (handles replies; you need to explicitly load this), and
  http-detail.bro (handles individual headers).  http-reply.bro will be
  undergoing some significant reworking in the near future; probably the
  scripts will be merged back into a single http.bro plus http-detail.bro.

- ssl-worm.bro contains a prototype policy script for detecting the
  Scalper SSL worm (contributed by Robin Sommer).  It uses the signature
  file sig.ex.ssl-worm.bro.  If someone has traces of Scalper in action
  to send us, that would be great.

- A new policy script, contents.bro, extracts the contents of each
  Bro connection into its own pair of files (one file for each
  direction).  Use in conjunction with -f or discarder_XXX() to
  extract specific connections.

- A new built-in function, strcmp(), returns the usual comparison between
  two strings (contributed by Robin Sommer).

- A new event, content_gap(), is generated when Bro detects that it is
  forced to skip over data in a reconstructed TCP stream because it is
  missing from the packet input.

- BIND8 is no longer included with the distribution.  If this causes you
  problems, let me know.

- aux/scripts/bro_logchk is a Perl script for parsing Bro HTTP & FTP logs
  (contributed by Jim Barlow).

- You can now compare addresses to see which is larger.  a < b means
  that in network order, the octets making up 'a' are ordered before
  those for 'b'.  E.g., 128.2.3.4 < 128.2.3.5 < 129.0.0.1.  Note that
  IPv4 addresses are all < IPv6 addresses (other than IPv4 addresses
  that are embedded in IPv6 addresses, e.g., ::128.2.3.4 < 128.2.3.5).

- Serious bug in TCP option handling fixed.

- Some bugs in CRLF handling fixed (courtesy Ruoming Pang).

- Bug in the implementation of &optional fixed.

- Bug in computing memory statistics when not reading packets (from
  an interface or the trace file) fixed.

- You can now include a trailing comma after the last item in an
  "enum" enumeration list.

- port-name.bro now maps 389/tcp to "ldap".

- A bug has been fixed in loading files multiple times


v0.7a175 Thu Aug 29 21:14:34 PDT 2002

- bro -s <file> reads in a *signature* file to search for regular expressions
  in packet payloads or TCP byte streams (written by Robin Sommer).
  See policy/rules.bro for an example of a policy script for processing
  the matches.

  Note that this feature is experimental, and will be evolving in the
  near future.

- The python script "snort2bro" reads in Snort signatures and translates
  them into Bro signature rules, suitable for processing using -s.

  An example of its operation is seen by running

	python snort2bro < sig.ex.web-rules.snort

  which, after reading in sig.ex.classification.config and
  sig.ex.reference.config, generates the output given in
  sig.ex.web-rules.bro, which is suitable to use as input to
  bro -s.

- bro -d invokes a gdb-like debugger (written by Umesh Shankar).  You can
  set breakpoints and watchpoints, examine tracebacks, print Bro expressions,
  and the like.  Type "help" for on-line help.

- bro -t <tracefile> turns on tracing of the policy script execution,
  written to the given file.

- Bro now includes an SMTP analyzer, which includes processing MIME
  message bodies (written by Ruoming Pang).  See smtp.bro and mime.bro
  for related policy scripts.  smtp.bro includes several experimental
  techniques for detecting mail relaying.

- You can now define enumerated types, such as

	type scanner_type: enum {
		SCANNER_STEALTH, SCANNER_HIGH_SPEED, SCANNER_AMBIGUOUS,
	};

  Enumerated types can be compared for equality with one another, and used
  as table indices, but cannot be converted to/from integers.

- bro -A <file> invokes an experimental, general trace transformation/
  anonymization framework (written by Ruoming Pang) which writes a modified
  tcpdump trace file from the input (which can be the network or another
  trace file) with potentially extensive modifications to the recorded
  packets.

  Transformers are built from .rw files (currently, {finger,ftp,ident,smtp}.rw),
  which are processed by the utility "rwcl" to generate both event engine
  analyzer components and rewriter policy scripts (for example, after
  configuring and building Bro, you'll find the scripts
  policy/{finger,ftp,ident,smtp}.rw.bro).

  See policy/smtp-rewriter.bro for an example of a policy script that
  performs transformation/anonymization.

- New built-ins:

	split(s: string, p: pattern): string_array;

	  takes a string and splits it into pieces at each occurrence of
	  the regular expression pattern p.  (The functionality is like
	  that in awk.)  It returns a string_array, which is a table[count]
	  of string that is indexed starting at 1, giving the different
	  pieces.

	  For example,

		split("foobar", /o/)

	  returns a 3-element table, for which [1] is the string "f",
	  [2] is the string "" (empty), and [3] is the string "bar".

	split1(s: string, p: pattern): string_array;

	  split1() does the same thing as split(), but only performs splitting
	  at the first occurrence, so it returns either a one-element table
	  (if the pattern doesn't appear in the string) or a two-element
	  table.  split1("foobar", /o/) returns a 2-element table for which
	  [1] is "f" and [2] is "obar".

	md5_hash(s: string): string

	  returns (in human-readable form) the MD5 hash of a given string.

	  So, for example,

		md5_hash("foobar")

	  yields "3858f62230ac3c915f300c664312c63f".

	to_addr(s: string): addr

	  takes a string representing an address in "dotted quad" format
	  and returns the correponding "addr" value.

	set_buf(f: file, buffered: bool)

	  sets the given file to have its writes buffered or unbuffered
	  depending on the value of "buffered".  It does not return a value.

	connection_exists: function(c: conn_id): bool

	  returns T if the given connection identifier corresponds to a
	  currently instantiated connection (one for which the event engine
	  has state), F otherwise.

	lookup_connection(c: conn_id): connection

	  returns the "connection" record associated with the given
	  connection identifier, or a fatal run-time error if there
	  isn't one.

	set_inactivity_timeout(c: conn_id, timeout: interval): interval

	  sets the inactivity timeout for the given connection to the
	  given interval, returning the old interval.

	  If the interval is non-zero, then when no packets have been
	  processed for a connection after that much time has elapsed,
	  the connection is deleted, and an "inactivity_timeout" event
	  generated.

	get_matcher_stats(): matcher_stats

	  used for gathering statistics about the signature matcher

	rewriting_trace(): bool

	  returns T if -A was specified (anonymize/rewrite a trace),
	  F otherwise.

- New events:

	connection_state_remove(c: connection);

	  Invoked when the event engine has removed the connection from
	  its state.

	connection_SYN_packet(c: connection, pkt: SYN_packet);

	  Invoked for each SYN/SYN-ACK packet.

	connection_timeout(c: connection);

	  Invoked when the event engine times out a connection - for
	  example, because the originator sent a SYN that was never
	  answered, so the connection was never established.

	connection_reused: event(c: connection);

	  Invoked when the event engine decides that a new SYN for
	  an existing connection reflects a reuse of the connection
	  four-tuple, rather than belonging to the existing connection.

- New globals:

	const ignore_checksums = F &redef;

	  If true, then the event engine does not verify checksums (and
	  hence will not discard packets with bad checksums).

	const tcp_deliver_undelivered = F &redef;

	  If true, then when the event engine closes a connection, if
	  that connection has a chunk of data not yet delivered to its
	  analyzer (which only happens if the data is above a sequence
	  hole, indicating either a packet filter glitch or a protocol
	  botch), then the undelivered data will at that point be delivered
	  to the connection's analyzer.

	const tcp_reassembler_ports_orig: set[port] = {} &redef;
	const tcp_reassembler_ports_resp: set[port] = {} &redef;

	  Sets of ports for which, if a connection has the corresponding
	  originator/responder port, then the event engine will reassemble
	  the byte stream of the connection.

	  Normally, the event engine reassembles byte streams for any
	  connection for which there's an analyzer, and otherwise doesn't.
	  These variables can be used to force reassembly for the originator
	  or responder side (respectively) of connections for which there
	  isn't otherwise an analyzer.  This is useful when doing signature
	  matching on reassembled byte streams, for protocols that are
	  not otherwise analyzed by the event engine.

	const table_expire_interval = 1 min &redef;

	  How often to check table entries to see whether they've expired
	  (see &read_expire, etc., below).

	const requires_trace_commitment = F;

	  If true, then when rewriting/anonymizing traces, nothing will
	  actually be written to the edited trace file unless you call:

		rewrite_commit_trace(c: connection, commit: bool, future: bool)

	  If "future" is true, then future rewritten packets will be
	  automatically commited; otherwise, writing them to the trace
	  file requires another explicit rewrite_commit_trace() call.

	const inactivity_timeout = 0 secs &redef;

	  As noted above, when a connection becomes inactive, time it out
	  after this interval.  If 0 secs, then don't time it out.

- An SSH analyzer extracts SSH client/server version information.  See
  ssh.bro for the related policy script.

- There's now a (very) simple TFTP analyzer available in tftp.bro.

- You can now set the global "frag_timeout" to an interval which controls
  how long fragments are kept before discarding them (contributed by Ashley
  Thomas).  If you don't set the global, or set it to 0.0 sec, then fragments
  are kept around indefinitely.

- An implementation of an experimental anti-evasion technique, "active
  mapping", has been written by Umesh Shankar.  It is not yet ready for
  general use, and isn't compiled in unless -DACTIVE_MAPPING.

- Four new attributes can now be associated with tables (implemented
  by Robin Sommer): &read_expire, &write_expire, and &create_expire
  will delete table entries after a given interval has elapsed since
  the table entry was last read, written, or created.  For example:

	global a: table[addr] of count &default=0 &create_expire = 5 sec;

  will delete each entry added to it 5 seconds after the entry was added,
  regardless of subsequent read/write activity to the element.

  &expire_func allows you to associate a function with the table such that
  whenever an entry expires, the function is invoked.  It's passed the
  value of the table entry (not the index - perhaps this should be changed),
  and returns an "interval" value.  If the interval is <= 0.0 seconds, then
  the table entry is immediately deleted.  Otherwise, it is deleted after
  the given interval has elapsed.

- When listing multiple attributes, you no longer separate them with
  commas.  For example, if you used to have:

	global a: table[string] of count &default=0, &redef;

  you now need to use:

	global a: table[string] of count &default=0 &redef;

- You can now construct records using

	[$field1 = <expression>, $field2 = <expression>, ...]

  Such record values can be assigned to other records providing that the
  target value's type includes all of the fields (same name and type)
  present in the record value, and that any missing fields have the
  &optional or &default attribute (see next item).

  You can also include a record value inside the record constructor, and
  all of its fields will be included in the constructed record value.

- Record fields can now be annotated with &optional, to indicate
  that the field needn't be present, or &default, which indicates
  a default value to provide if the field is missing.

- You can query as to whether a record has a value for a given field
  using the new "?$" operator.  So for example:

	type my_rec: record {
		num: count &default = 0;
		msg: string;	# mandatory, since no &optional/&default
	};

	global r: my_rec;

	r = [$msg = "hello"];

	print r?$num, r?$msg, r$num;

  will print "F, T, 0," because even though 'r' has a default value
  for $num (which shows up when printing r$num), that field is missing,
  hence r?$num is F.

- An experimental scheme has been added (by Umesh Shankar) for managing
  general attributes associated either with all values ("global attributes")
  or particular particular values.  This scheme is likely to change in
  the near future, and hence isn't explained here further.

- The DNS analysis now includes ADDL and AUTH records, and much richer
  policy script analysis (per policy/dns.bro).

- You can now "redef" a function or event handler to override its
  previous definition.  For a function, this looks like:

	redef log_hook = function(msg: string): bool
		{
		...
		}

  For an event handler, it's just the usual definition preceded by "redef.
  For example,

	redef event ack_above_hole(c: connection) { }

  would replace the default ack_above_hole handler with one that does nothing.

- HTTP server and HTTP proxy backdoor detectors have been added,
  generating http_signature_found and http_proxy_signature_found,
  respectively (contributed by Ruoming Pang).

- A KaZaA backdoor detector has been added, which generates
  kazaa_signature_found for likely KaZaA connections.

- The new policy scripts flag-irc.bro and flag-warez.bro provide
  hooks for defining site policies for detecting IRC and access
  to warez.

- portmapper.bro now tracks the services it sees, and the names are
  used in connection summaries rather than generic services like port-656.

- bro -C (or redef'ing the "ignore_checksums" global to T) instructs
  Bro to ignore any checksum errors and go ahead and analyze such packets.

- The (trivial) policy script print-globals.bro dumps out all of the policy
  script global variables and the amount of memory they consume.

- The policy script code-red.bro has been renamed worm.bro and generalized
  to detect Nimda as well as Code Red 1 & 2.

- A bunch of additional default sensitive URIs have been added to http.bro.
  http.bro also now doesn't report worm-related URIs.

- A bunch of less common portnames were removed from port-names.bro.

- Empty regular expressions are now allowed.

- The finger_request event now has a third parameter, the additional
  text after the username.

- More systematic handling of NULs and CRLF by the event engine.

- Hex escape sequences now must have exactly two hexadecimal characters.

- FYI - work has begun on significantly altering the way that policy
  scripts generate alerts.

- Work has begun (by Robin Sommer) on a general framework for tracking
  client/server versions.  See software.bro.

- Work has begun on a NETBIOS analyzer (see NetbiosSSN.cc).  Contributions
  (e.g., finishing it :-) welcome.

- Work has begun on migrating the internals to process IPv6 in addition
  to IPv4.

- A number of bug fixes, leaks, and memory allocation lint tweaks.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.7a90 Thu Sep 06 00:50:43 PDT 2001

- Better state management, especially for use in high-volume sites.  There
  are now a number of ways to track the resources consumed by Bro while it
  runs, and to reduce the amount of memory Bro consumes.  Adding

	@load print-resources

  will log a summary of the resources consumed by Bro when it exits, and

	@load reduce-memory

  will change a number of default values in order to significantly diminish
  the amount of memory Bro requires.

  Other ways to lower the required memory discussed in the next three items.

- The global "maintain_http_sessions" now controls whether http.bro tracks
  multiple HTTP connections between the same hosts as a single HTTP session.
  Doing so costs a considerable amount of state (so set to F for reducing
  the memory impact of HTTP analysis).

- The global "suppress_scan_checks", if true, turns off address and port
  scan detection.  (You can't achieve this by simply not @load'ing scan.bro,
  because it's loaded by some of the default policy scripts.)  Turning it
  off can save a lot of memory.

- Note, the ident.bro is also expensive in terms of state-holding.

- New library functions:

	- resource_usage() returns a record detailing real time,
	  CPU time, memory, other getrusage info, and the current
	  and maximum number of TCP/UDP/ICMP connections, and
	  timers and fragments.

	- val_size() returns the size in bytes needed to represent
	  a given value (which can be a record, a table, or a
	  simple constant, for example).  It's not fully accurate
	  but is in the right ballpark.

	- global_sizes() return a table mapping every global
	  variable to its size (per val_size()).  Useful for
	  tracking which ones are growing large over time.

- You can now control a number of timers related to Bro's connection
  management.  Setting them to lower values generally decreases the
  amount of state Bro has to keep (see reduce-memory.bro), though
  this can also make it easier for an attacker to evade detection:

	tcp_SYN_timeout: interval
		Controls how long Bro waits after seeing the
		beginning of a connection (whether due to a SYN
		or not; the timer is misnamed) before checking
		whether it elicited any reply.

	tcp_session_timer: interval
		After a connection has closed, wait this long for
		further activity before checking whether to time
		out its state.

	tcp_connection_linger: interval
		When checking a closed connection for further
		activity, Bro should consider it inactive if there
		hasn't been any for this long.  It also complains
		if the connection is reused before this much time
		has elapsed.

	tcp_attempt_delay: interval
		Bro waits this long upon seeing an initial SYN
		before timing out the connection attempt.

	tcp_close_delay: interval
		Upon seeing a normal connection close, Bro flushes
		state after this much time.

	tcp_reset_delay: interval
		Upon seeing a RST, Bro flushes state after this
		much time.

	tcp_partial_close_delay: interval
		Bro generates a connection_partial_close event this
		much time after one half of a partial connection
		closes, assuming there has been no subsequent
		activity.

	non_analyzed_lifetime: interval
		If a connection belongs to an application that you
		aren't analyzing, Bro times it out after this
		interval.  If 0 secs (default), then it doesn't
		time it out.

	dns_session_timeout: interval
	ntp_session_timeout: interval
	rpc_timeout: interval
		Bro waits this long before timing out a DNS/NTP/RPC
		request.

	max_timer_expires: count
		The maximum number of timers to expire after
		processing each new packet.  The value trades off
		spreading out the timer expiration load with
		possibly having to hold state longer.  A value of 0
		means "process all expired timers with each new
		packet".

- Two new flags control other facets of Bro's connection management,
  and thus state-holding:

	partial_connection_ok: bool
		Whether Bro should analyze connections for which
		it doesn't see the beginning, only the middle.
		This can be very expensive to do in the face of
		stealth-scanning, which looks like a bunch of
		partial connections.

		Note, the HTTP analyzer has been modified to
		now always skip partial connections.  This should
		instead be user controllable.

	tcp_SYN_ack_ok: bool
		If true, Bro instantiates connection state when
		it sees a SYN ack but not the initial SYN (even
		if partial_connection_ok is false).  The intent
		behind this knob (which is not well tested) is
		to allow you to filter out initial SYNs and only
		react to SYN acks.  This keeps Bro from holding
		state during SYN scans and SYN flooding, except
		for when the destination responds.

- Some other miscellaneous thresholds that you can now modify from your
  policy script:

	heartbeat_interval: count
		How often to generate net_stats_update() events.
		This timer really isn't needed any more, since
		you can use "schedule" to achieve the same effect.

	tcp_storm_thresh: count
		If Bro sees this many FINs/RSTs in a row, it
		flags them as a "storm".

	tcp_storm_interarrival_thresh: interval
		The FINs/RSTs must come with this much time or less
		between them.

- The state management for line-oriented applications like HTTP requests
  has been improved.

- The HTTP analyzer now expands %hex sequences.  If anyone has a Unicode
  expander to contribute, that'd be terrific.

- The Code Red detection is more robust (fewer false positives).

- A new redefinable variable, skip_services, lists applications that should
  not be analyzed for purposes of detecting scans.  (Default:  ident)
  The point of having this (poorly named) hook is so that code-red.bro
  can add "http" to it, to avoid reporting the zillions of Code Red scans
  that a site can see.

- Bro now matches regular expressions using lazy DFA evaluation.  The upshot
  of this is (1) it no longer maintains .bro-RE-cache.v1, (2) it starts
  up fast regardless of whether you've added new regular expressions, (3)
  you can afford to add lots of regular expressions, and (4) it's actually
  a bit faster.

- The list of "hot_ids" has been expanded with a number of other common
  root-privileged accounts.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.7a61 Fri Apr 06 14:51:47 PDT 2001

- An NTP analyzer has been added.  See policy/ntp.bro for a corresponding
  policy file that detects the newly discovered NTP remote buffer overflow
  attack.

- example-attacks/ is a new directory that contains trace files of attacks.
  Currently, there are just two to play with:

	bro -r example-attacks/ftp-site-exec.trace mt

  will run on a trace of a "site exec" overflow attack, and

	bro -r example-attacks/ntp-attack.trace mt ntp

  will run on an example of the NTP overflow.

- The doc/ directory includes the postscript and HTML versions of the
  first draft of the Bro manual.

- A new policy file, icmp.bro, has preliminary (and only partially developed)
  policy for analyzing ICMP.

- The file libpcap.bufsize.patch includes the patch necessary on some systems
  to increase the maximum libpcap buffer size.

- You can now use anonymous functions in &default expressions, so for
  example you can do:

	global foo: table[count] of string = {
		[1] = "1st", [2] = "2nd", [3] = "3rd",
	} &default = function(n: count): string { return fmt("%dth", n); };

  and then referring to foo[5] will yield "5th".

- There's now a "for" statement to iterate over the indices of a table
  or the members of a set:

	for ( i in foo )

  for the above "foo" will iterate with i assigned to 1, 2, and 3; *but
  not in general in that order*.

- The function contains_string() has been removed, and now you can instead
  use an expression like

	"bar" in "foobar"

  which will yield T.

- The scan detection now has a mechanism for attempting to detect SYN flooding
  backscatter and flagging it as different from a stealth scan.

- New event handlers:

	new_connection_contents()
		like new_connection(), but reassembles the
		stream so you can use set_content_files() to
		write it to a file

	udp_session_done()
		invoked when a UDP session (which is defined on
		a per-protocol basis; currently only for NTP)
		finishes.

	ntp_message()
		invoked for each NTP message

- UDP processing now does accounting for detecting scans.

- UDP processing now tracks numbers of requests/replies for sessions that
  support that notion.  The connections are annotated by udp_session_done()
  with "[m,n]" for "m" requests and "n" replies, providing either m or n > 1.

- New variable accessible from policy:

	watchdog_interval
		how often the watchdog should check for whether
		Bro is making internal progress

- A bunch of functions no longer have a first argument of the current time;
  get it instead from network_time() if you need it:

	authentication_accepted
	authentication_rejected
	conn_weird
	conn_weird_addl
	flow_weird
	net_weird

- A bunch of functions now return bool rather than int values:

	set_contents_file
	set_login_state
	set_record_packets
	skip_further_processing

- The variable "hot_dests" has been renamed to "hot_dsts".

- 111/tcp is now identified as "portmap" rather than "rpc".

- Connections flagged as hot for some types of characteristics are now
  annotated with the reason associated with the decision.  (I think a lot
  more of this is needed.)

- Portmapper dumps are annotated with the results of the mapping.  This
  will be streamlined in the future.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.7a48 Wed Sep 13 14:37:30 PDT 2000

- Changes between this release and v0.6 missing :-(

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.6 Wed Jul 21 17:02:50 PDT 1999

- Support for regular expressions added.  You specify lex-style regular
  expressions between '/'s, for example "/\/etc\/(passwd|shadow)/" has
  the type "pattern" and matches /etc/passwd or /etc/shadow (the slashes
  in the pattern need to be escaped or else they'd delimit the end of the
  pattern).  Pattern-matching is via the "in" operator, so for example:

	if ( filename in /\/etc\/(passwd|shadow)/ )
		sensitive_file_access(filename);

  or

	const sensitive_files = /\/etc\/(passwd|shadow)/;

	...

	if ( filename in sensitive_files )
		sensitive_file_access(filename);

  Presently the "in" operator requires that the entire left-hand side
  be matched by the pattern.  So, for example, if you want to find the
  string "eggdrop" anywhere inside the string "line", you would use

	if ( line in /.*eggdrop.*/ )

  If you leave off either of the .*'s, then eggdrop will only be matched
  at the beginning or end of the line.

  In the future, there will be mechanisms for specifying whether you
  want to match anywhere in a line, or anchored; accordingly, *the above
  syntax is subject to change*.

  Bro compiles regular expressions into DFAs for fast matching.  This can take
  quite a bit of time for complicated patterns.  Consequently, it maintains a
  cache of compiled regular expressions in $HOME/.bro-RE-cache-v1.  You can
  always safely remove this file; Bro will recreate/repopulate it as needed.
  It does not clean up unused entries from it, so if you change your patterns
  frequently, you will accumulated lots of old ones and should delete the
  file to garbage collect them.

- An rlogin analysis module has been added and the telnet analysis
  generalized to generic "login" analysis, with the following events:

	login_failure(c: connection, user: string, client_user: string,
			password: string, line: string)
		Generated on a failed attempt to log in.  client_user is
		the remote user name, if the login is via the rlogin
		protocol.
	login_success(c: connection, user: string, client_user: string,
					password: string, line: string)
		Generated on a successful attempt to log in.

	login_input_line(c: connection, line: string)
		Generated per line of input typed by the user.
	login_output_line(c: connection, line: string)
		Generated per line of output generated by the server.

	login_confused(c: connection, msg: string, line: string)
		Generated when a login dialog confuses the heuristic
		analyzer.  msg is a tag for the state mismatch that
		was unexpected, line is the corresponding dialog text.

	login_confused_text(c: connection, line: string)
		Once a connection is in the confused state, then this
		is generated for each subsequent line.

	login_terminal(c: connection, terminal: string)
		Generated if the terminal type associated with the
		connection is seen.

	login_display(c: connection, display: string)
		Generated if the display associated with the connection
		is seen.

	excessive_line(c: connection)
		Generated when the connection has produced an excessively
		long line.

  login_input_line() and login_output_line() are very powerful for
  detecting intrusions, when coupled with regular-expression matching.

  login_terminal() is used to detect backdoors that are triggered
  by the terminal environment variable.

- An ident analysis module has been added (port 113).  It generates
  ident_request, ident_reply, and ident_error events.  Port 113 used
  to be referred to as "auth"; now it's referred to as "ident".

- A new type of scan detection has been added, which is triggered
  by a remote host trying a large number of username/password
  combinations.  See the account_tried() function in scan.bro.

- The default search path for .bro files is now

	  .:priv-policy:policy:pub-policy:/usr/local/lib/bro

  where priv-policy/ is intended for private policy and pub-policy/
  for public policy.  The Bro alpha distribution ships with a
  sample set of pub-policy scripts.

- New built-ins:

	system(s: string): int
		executes the given shell command using system()
		and returns its status.

	set_contents_file(c: conn_id, direction: count, f: file)
		copies connection c's reassembled byte stream in
		either the originator-to-responder direction (if
		direction is CONTENTS_ORIG) or the responder-to-
		originator direction (CONTENTS_RESP) to the file f.

	reading_live_traffic(): bool
		returns true if Bro is running on live traffic (read
		from a network interface), false if it's reading from
		a save file.

	mkdir(f: string): bool
		creates the given directory, returning true if it
		was able to, false if not.

	get_orig_seq(c: conn_id): count;
		returns the highest sequence number sent by the
		originator of connection c.
	get_resp_seq(c: conn_id): count;
		same for c's responder.

- Additional new events (other than those related to the new analyzers):

	new_connection(c: connection)
		is generated whenever a new connection is seen.

	partial_connection(c: connection)
		is generated whenever a new partial connection (one
		that doesn't begin with a SYN handshake) is seen.

	pm_bad_port(r: connection, bad_p: count)
		is generated when a portmapper response contains
		a bad port number.

- Functions, tables and sets can now be assigned.  Assignment is
  made by reference to the underlying object.

- Bro no longer looks up identifiers using getservbyname() to see if they
  should be interpreted as port numbers, since this led to portability
  problems.  Instead, a number of constants are defined in bro.init:
  bgp, domain, finger, ftp, gopher, http, ident, rlogin, smtp, ssh and telnet,

- Bro now supports an arbitrary number of open files (not bound by
  the system's limit on file descriptors).

- There's now a finger_reply event to go with finger_request.

- A bunch more RPC service names have been added, thanks to Job de Haas
  and others.

- A bug has been fixed in the watchdog handling that caused it to
  sometimes expire after a period of network inactivity.

- The Bro paper in doc/ has been revised (it isn't quite up-to-date,
  but considerably closer than the USENIX version).

- There has been a large amount of reworking of the internals, both
  to Bro itself and in the policy scripts.  If you find something you're
  wondering about, feel free to send me mail asking about it.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.5 Sun Oct  4 00:19:35 PDT 1998

- Added Linux support.

- Major autoconf changes.

- Some tweaks to suppress g++ warnings.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.4 Fri Sep 11 00:08:04 PDT 1998

- The new aux/ directory has some utility programs and scripts.
  See aux/README for details.

- Documentation (though exceedingly limited) describing the connection
  log summaries generated by policy/tcp.bro now available in doc/conn-logs.

- The Telnet username recognition code has been heavily reworked and is
  now significantly more robust.

- A new policy file, policy/scan.bro, detects port and address scanning.

- Bro now detects the "Land" attack (a trivial addition to its spoof
  detection).

- The distribution now comes with BIND 8.1.2.

- A new flavor of "weird" event has been added, flow_weird, for unusual
  events associated with <src IP addr, dst IP addr> flows (i.e., a coarser
  granularity than TCP or UDP connections).

- Bro now reassembles fragments, checking for overlaps and consistency.
  Associated with fragments are the following flow_weird's:

	excessively_large_fragment
	excessively_small_fragment
	fragment_inconsistency
	fragment_overlap
	fragment_protocol_inconsistency
	fragment_size_inconsistency
	fragment_with_DF
	incompletely_captured_fragment

- The TCP stream reassembly code was rewritten to share functionality
  with the new fragment reassembly code.

- If a handler for it is present, then Bro will generate "packet_contents"
  events containing the contents of each packet it receives.  This is just
  for experimenting with, and, in particular, this event does *not* reflect
  TCP stream reassembly.

- The handling of "conn_weird" events in tcp.bro now has more options.
  In the weird_action table, WEIRD_LOG_ALWAYS means that the weird event
  should always be logged; WEIRD_LOG_PER_CONN means that it should be
  logged once per connection; and WEIRD_LOG_PER_ORIG that it should be
  launched once per originator IP address.

- The example hot.bro now includes provisions for flagging sensitive
  inbound and outbound services.

- Bro now limits the number of events processed when draining the event
  queue after processing a packet.  This makes Bro less prone to packet
  loss during high-speed scans.

- The line-oriented TCP endpoint class was split into TCP_EndpointLine,
  which supports the line-oriented functionality, and TCP_NVT (derived
  from TCP_EndpointLine), which implements the Network Virtual Terminal
  used by Telnet and FTP.

- The TCP_NVT class now understands the Telnet Environment option.

- Escape sequences are now '\' followed by 1 or more octal digits,
  instead of excatly three octal digits (which is error prone).

- If the watchdog timer expires, it now reports the number of events
  processed in the current batch of packets, as well as other timing
  information.

- Bro now should not report packet drops that occur after it has begun
  to exit (these can occur when draining the pending event queue takes
  a while).

- Bro now detects TCP acknowledgements that occur above a sequence
  hole, generating an ack_above_hole event.  Nominally, this indicates
  packet filter drops, but in fact some buggy TCPs manage to do this :-(.

- Fledgling HTTP support added.  An http_request event is generated when
  a new HTTP request is seen, and http_stats is generated when an HTTP
  connection terminates, giving (uninteresting to most people) statistics
  concerning the request(s).  A lot more is needed: parsing persistent
  connections and HTTP replies, for one.  This is just a start.  A stub
  for http_reply exists but these events are not presently generated.

- Ported to Linux (thanks to Pascal Bouchareine).

- A bug in to_lower() and to_upper() was fixed.

- The reporting for unexpected FTP connections now more directly
  identifies the corresponding FTP session.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.3 Fri Feb 13 19:59:39 PST 1998

- Timers are now implemented using calendar queues rather than priority
  queues (with thanks to Craig Leres).

- The new byte_len() built-in returns the length of a string interpreted
  as a set of bytes (including any NUL bytes, especially the final one
  for a typical string).

- The new sub_bytes() built-in extracts a subset of a string interpreted
  as a set of bytes (i.e., immune to any embedded NULs).

- Fixed bad interaction with the latest version of libpcap that on some
  BPF systems would cause Bro to exit any time a little bit of time went by
  without any traffic matching its filter.

- A bug with constant-folding of the ?: operator has been fixed.

- A new "conn_stats" event delivers statistical analysis (number of packets
  transmitted, retransmitted, out-of-order, replicated) of each connection.
  If you define a handler for it, then *no* other TCP processing is done.
  This was added for off-line analysis of traces with large numbers of
  connections in them.

- Some minor portability tweaks.

-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

v0.2 Sat Oct 25 11:22:37 PDT 1997

- Added autoconf support, thanks to Scott Denton.

- Ported to FreeBSD, Solaris.

- Fixed a bug in which partial connections were ignored even for
  protocols (e.g., FTP) that can make use of partial dialogs.

- Included BIND version 8 sources directly in the distribution.

- Better usage() information (again thanks to Scott), -h and -v flags.

- README, CHANGES files created.
