Arne Welzel
8fc86bb4b6
Event: Bail on add_missing_remote_network_timestamp without add_network_timestamp
2025-06-02 17:31:36 +02:00
Arne Welzel
0ab53c75cd
btest/plugin: Test custom metadata publish
...
Usage demo for plugin writers to add custom event metadata and access in
in Zeek scripts.
2025-06-02 17:31:36 +02:00
Arne Welzel
7db03a8c77
NEWS: Add note about generic event metadata
2025-06-02 17:31:36 +02:00
Arne Welzel
8e87dcbdb2
cluster: Remove deprecated Event constructor
...
It is now unused, ditch it. This wasn't available in an LTS release yet
and anyhow is in the detail namespace.
2025-06-02 17:31:36 +02:00
Arne Welzel
e3a83addce
cluster: Remove some explicit timestamp handling
...
Backend::MakeClusterEvent() for now is the only place to add implicit
network timestamp metadata within the cluster component.
2025-06-02 17:31:36 +02:00
Arne Welzel
e1f70164e0
broker/Manager: Fetch and forward all metadata from events
...
Also use the generic metadata version for publishing, keep the
ts-based API for now, but only add timestamps when
EventMetadata::add_network_timestamp is T. I'm not sure what the
right way forward here is, maybe deprecating Broker's publish event
variations and funneling through cluster.
2025-06-02 17:31:36 +02:00
Arne Welzel
96f2d5d369
Event/init-bare: Add add_missing_remote_network_timestamp logic
...
Make defaulting to the local network timestamp for remote events opt-in.
2025-06-02 17:31:36 +02:00
Arne Welzel
b87109fcf1
cluster/Backend/DoProcessEvent: Use generic metadata, not just timestamps
2025-06-02 17:31:36 +02:00
Arne Welzel
4996ba88fb
cluster/Event: Support moving args and metadata from event
2025-06-02 17:31:36 +02:00
Arne Welzel
46d4b5825b
cluster/serializer/broker: Support generic metadata
...
Instead of handling just the network timestamp, support extraction of
the whole metadata vector that broker events hold.
2025-06-02 17:31:36 +02:00
Arne Welzel
71412f35b7
cluster/Event: Generic metadata support
...
Instead of a timestamp attribute, switch to holding a EventMetadataVectorPtr
like zeek::Event instances do. Keep the old constructor until the end of
the patch series.
2025-06-02 17:31:36 +02:00
Arne Welzel
7b4b1779bf
Event: Use -1.0 for undefined/unset timestamps
...
This can happen if either there's no network timestamp associated with
an event, or there's currently no event being dispatched. Using 0.0
isn't great as it's the normal start timestamp before reading a network
packet. Using -1.0 gives the caller a chance to check and realize what's
going on.
2025-06-02 17:31:36 +02:00
Benjamin Bannier
0dae8b8d2d
Fix incorrectly copied comment [skip CI]
2025-06-02 13:24:24 +02:00
Arne Welzel
31f51f7a87
Merge remote-tracking branch 'origin/topic/bbannier/coverity-fixes'
...
* origin/topic/bbannier/coverity-fixes:
Prefer `std::move` over copy
2025-06-02 10:17:24 +02:00
Arne Welzel
e5bb6317fa
Merge remote-tracking branch 'origin/topic/vern/CPP-maint.May25'
...
* origin/topic/vern/CPP-maint.May25:
minor BTest maintenance updates for -O gen-C++
fix for more robustly finding BTests to assess for -O gen-C++
fix for -O gen-C++ dealing with type constants of unnamed compound types
2025-06-02 10:12:27 +02:00
Arne Welzel
41f04eda72
Merge remote-tracking branch 'origin/topic/awelzel/intel-indicator-hooks'
...
* origin/topic/awelzel/intel-indicator-hooks:
intel/seen/manage-event-groups: Policy script for toggling intel event groups
intel: Add indicator_inserted and indicator_removed hooks
2025-06-02 09:52:07 +02:00
Arne Welzel
0619fe2f4f
intel/seen/manage-event-groups: Policy script for toggling intel event groups
...
Co-authored-by: Mohan Dhawan <mohan@corelight.com>
2025-06-02 09:51:14 +02:00
Arne Welzel
7eb849ddf4
intel: Add indicator_inserted and indicator_removed hooks
...
This change adds two new hooks to the Intel framework that can be used
to intercept added and removed indicators and their type.
These hooks are fairly low-level. One immediate use-case is to count the
number of indicators loaded per Intel::Type and enable and disable the
corresponding event groups of the intel/seen scripts.
I attempted to gauge the overhead and while it's definitely there, loading
a file with ~500k DOMAIN entries takes somewhere around ~0.5 seconds hooks
when populated via the min_data_store store mechanism. While that
doesn't sound great, it actually takes the manager on my system 2.5
seconds to serialize and Cluster::publish() the min_data_store alone
and its doing that serially for every active worker. Mostly to say that
the bigger overhead in that area on the manager doing redundant work
per worker.
Co-authored-by: Mohan Dhawan <mohan@corelight.com>
2025-06-02 09:50:48 +02:00
Benjamin Bannier
1760d99c49
Prefer std::move over copy
2025-06-02 08:45:32 +02:00
Vern Paxson
614eb8d343
minor BTest maintenance updates for -O gen-C++
2025-05-31 12:52:44 -07:00
Vern Paxson
9117ccab12
fix for more robustly finding BTests to assess for -O gen-C++
2025-05-31 12:50:14 -07:00
Vern Paxson
e165e64fa5
fix for -O gen-C++ dealing with type constants of unnamed compound types
2025-05-31 12:49:37 -07:00
zeek-bot
224519c11a
Update doc submodule [nomail] [skip ci]
2025-05-31 00:26:58 +00:00
Tim Wojtulewicz
3282bbc429
Merge remote-tracking branch 'origin/topic/vern/ZAM-maint.May25'
...
* origin/topic/vern/ZAM-maint.May25:
fix for crash when interpreting transformed ASTs that include multi-field record assignments/additions
Remove unused ZAM compiler method
2025-05-30 13:07:01 -07:00
Tim Wojtulewicz
70bc0d9deb
Merge remote-tracking branch 'origin/topic/timw/cleanup-cmake-summary-output'
...
* origin/topic/timw/cleanup-cmake-summary-output:
Add utility methods to make CMake summary output nicer
2025-05-30 12:16:35 -07:00
Tim Wojtulewicz
e93242726b
Add utility methods to make CMake summary output nicer
2025-05-30 11:57:43 -07:00
Tim Wojtulewicz
dc5dd8be45
Merge remote-tracking branch 'origin/topic/timw/new-ci-pr-labels'
...
* origin/topic/timw/new-ci-pr-labels:
CI: Add PR label for skipping all CI jobs
CI: Add PR label for running cluster tests
2025-05-30 10:29:37 -07:00
Tim Wojtulewicz
bc4cf14237
CI: Add PR label for skipping all CI jobs
2025-05-30 10:29:02 -07:00
Tim Wojtulewicz
e9544386fe
CI: Add PR label for running cluster tests
2025-05-30 10:27:52 -07:00
Vern Paxson
dc68a62a1e
fix for crash when interpreting transformed ASTs that include multi-field record assignments/additions
2025-05-30 09:44:26 -07:00
Vern Paxson
ba0b7492a7
Remove unused ZAM compiler method
2025-05-30 09:38:42 -07:00
Tim Wojtulewicz
9c290df47f
Merge remote-tracking branch 'origin/topic/timw/ci-clang-tidy'
...
* origin/topic/timw/ci-clang-tidy:
CI: Add new task to run clang-tidy as part of nightly builds
CI: Update to clang 19 on ubuntu 24.04, add clang-tidy package
2025-05-30 08:39:36 -07:00
Tim Wojtulewicz
bf9813a7c6
CI: Add new task to run clang-tidy as part of nightly builds
2025-05-30 08:39:14 -07:00
Tim Wojtulewicz
dbd787a81f
CI: Update to clang 19 on ubuntu 24.04, add clang-tidy package
2025-05-30 08:39:14 -07:00
Arne Welzel
f4cd92e24a
Merge remote-tracking branch 'origin/topic/awelzel/4494-ts-millis-signed'
...
* origin/topic/awelzel/4494-ts-millis-signed:
logging/ascii/json: Make TS_MILLIS signed, add TS_MILLIS_UNSIGNED
2025-05-30 17:24:17 +02:00
Arne Welzel
93813a5079
logging/ascii/json: Make TS_MILLIS signed, add TS_MILLIS_UNSIGNED
...
It seems TS_MILLIS is specifically for Elasticsearch and starting with
Elasticsearch 8.2 epoch_millis does (again?) support negative epoch_millis,
so make Zeek produce that by default.
If this breaks a given deployment, they can switch Zeek back to TS_MILLIS_UNSIGNED.
https://discuss.elastic.co/t/migration-from-es-6-8-to-7-17-issues-with-negative-date-epoch-timestamp/335259
https://github.com/elastic/elasticsearch/pull/80208
Thanks for @timo-mue for reporting!
Closes #4494
2025-05-30 17:23:29 +02:00
Tim Wojtulewicz
c387ec87be
Merge remote-tracking branch 'origin/topic/timw/clang-tidy-performance-fixes'
...
* origin/topic/timw/clang-tidy-performance-fixes:
Add move operations for LogWriteHeader
Add missing setting of type in session::Key move operations
Update .clang-tidy to have performance-* enabled with some exclusions
Fix clang-tidy performance-inefficient-string-concatenation warnings
Fix clang-tidy performance-unnecessary-copy-initialization warnings
Fix clang-tidy performance-move-const-argument warnings (not move assignable/copyable)
Fix clang-tidy performance-move-const-argument warnings (passing move to const argument)
Fix clang-tidy performance-move-const-argument warnings (moving trivially copyable)
Fix clang-tidy performance-move-const-argument warnings (moving const variables)
Fix clang-tidy performance-inefficient-vector-operation warnings
Fix clang-tidy performance-for-range-copy warnings
Fix clang-tidy performance-faster-string-find warnings
Fix clang-tidy performance-enum-size warnings
Fix clang-tidy performance-avoid-endl warnings
2025-05-30 08:13:19 -07:00
Tim Wojtulewicz
6eb49a10cc
Add move operations for LogWriteHeader
2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
df852255c6
Add missing setting of type in session::Key move operations
2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
db69773d23
Update .clang-tidy to have performance-* enabled with some exclusions
2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
c609d5c90a
Fix clang-tidy performance-inefficient-string-concatenation warnings
2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
cb8c35748a
Fix clang-tidy performance-unnecessary-copy-initialization warnings
2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
909413838c
Fix clang-tidy performance-move-const-argument warnings (not move assignable/copyable)
2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
57c10a6ace
Fix clang-tidy performance-move-const-argument warnings (passing move to const argument)
2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
ad4694f529
Fix clang-tidy performance-move-const-argument warnings (moving trivially copyable)
2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
144a3dee3a
Fix clang-tidy performance-move-const-argument warnings (moving const variables)
2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
6196950567
Fix clang-tidy performance-inefficient-vector-operation warnings
2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
b8e28abb97
Fix clang-tidy performance-for-range-copy warnings
2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
178d7f4cd0
Fix clang-tidy performance-faster-string-find warnings
2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
f4c47d0357
Fix clang-tidy performance-enum-size warnings
2025-05-30 08:12:29 -07:00
