Commit graph

18389 commits

Author SHA1 Message Date
Arne Welzel
4996ba88fb cluster/Event: Support moving args and metadata from event 2025-06-02 17:31:36 +02:00
Arne Welzel
46d4b5825b cluster/serializer/broker: Support generic metadata
Instead of handling just the network timestamp, support extraction of
the whole metadata vector that broker events hold.
2025-06-02 17:31:36 +02:00
Arne Welzel
71412f35b7 cluster/Event: Generic metadata support
Instead of a timestamp attribute, switch to holding a EventMetadataVectorPtr
like zeek::Event instances do. Keep the old constructor until the end of
the patch series.
2025-06-02 17:31:36 +02:00
Arne Welzel
7b4b1779bf Event: Use -1.0 for undefined/unset timestamps
This can happen if either there's no network timestamp associated with
an event, or there's currently no event being dispatched. Using 0.0
isn't great as it's the normal start timestamp before reading a network
packet. Using -1.0 gives the caller a chance to check and realize what's
going on.
2025-06-02 17:31:36 +02:00
Arne Welzel
e2e13902f3 cluster: Use shorter obj_desc versions 2025-05-26 17:53:23 +02:00
Arne Welzel
0c18ec5e5d Desc: Add obj_desc() / obj_desc_short() overloads for IntrusivePtr
When using these helpers in code, one barely has raw pointers and the
repeated `.get()` call cumbersome and noisy.
2025-05-26 16:10:48 +02:00
Arne Welzel
cef63e871e Merge remote-tracking branch 'origin/topic/awelzel/fix-no-zero-timestamp-metadata'
* origin/topic/awelzel/fix-no-zero-timestamp-metadata:
  btest: Add test for Cluster::hello zero-timestamp
  EventMgr/Enqueue: Add automatic timestamp metadata to local events, only
  cluster and broker: Propagate zero-timestamp as metadata, too.
2025-05-26 16:08:58 +02:00
Arne Welzel
277c3f5245 btest: Add test for Cluster::hello zero-timestamp 2025-05-26 16:08:27 +02:00
Arne Welzel
a9d22611d0 EventMgr/Enqueue: Add automatic timestamp metadata to local events, only
It seems less surprising if only local events receive automatic network
timestamp metadata. For remote events the automatic value will most
likely be misleading.
2025-05-26 16:08:27 +02:00
Arne Welzel
d828e08a9e cluster and broker: Propagate zero-timestamp as metadata, too.
This will be cleaned up later to just pass all contained metadata from
a cluster event to the queued event, but for now do this here, otherwise
we break some internal tests.
2025-05-26 13:37:42 +02:00
zeek-bot
ffb07ce2bd Update doc submodule [nomail] [skip ci] 2025-05-24 00:24:08 +00:00
Tim Wojtulewicz
f996f661b6 Merge remote-tracking branch 'origin/topic/timw/redis-connection-failure'
* origin/topic/timw/redis-connection-failure:
  Redis: bump version of hiredis required
  Redis: return proper error if connection fails
2025-05-23 12:13:51 -07:00
Tim Wojtulewicz
85e27afaa8 Redis: bump version of hiredis required 2025-05-23 12:13:13 -07:00
Tim Wojtulewicz
0fb4548ff0 Redis: return proper error if connection fails 2025-05-23 12:13:13 -07:00
Arne Welzel
d929392a76 Merge remote-tracking branch 'origin/topic/awelzel/4177-4178-custom-event-metadata-part-1'
* origin/topic/awelzel/4177-4178-custom-event-metadata-part-1:
  Event: Move meta after args
  Event: Use IntrusivePtr to manage obj refcount
  btest/zam: Update for new EventMetadata bifs
  broker and cluster: Switch to new Enqueue() API
  Event/zeek.bif: Add EventMetadata current() and current_values() accessors
  Event: Deprecate default network timestamp metadata
  Event: Store timestamp in metadata vector
  EventRegistry/zeek.bif/init-bare: Add event metadata infrastructure
  EventMgr: Add CurrentEvent() accessor
2025-05-23 21:02:28 +02:00
Arne Welzel
bfcb68f8e7 Event: Move meta after args 2025-05-23 19:32:50 +02:00
Arne Welzel
89402fcc1b Event: Use IntrusivePtr to manage obj refcount
Fly-by cleanup.
2025-05-23 19:32:50 +02:00
Arne Welzel
e4e9ec3e80 btest/zam: Update for new EventMetadata bifs 2025-05-23 19:32:50 +02:00
Arne Welzel
506fea3335 broker and cluster: Switch to new Enqueue() API
This is a bit intermediary. In part 2 this will deal with any metadata,
not just timestamps.
2025-05-23 19:32:50 +02:00
Arne Welzel
75aa6588fe Event/zeek.bif: Add EventMetadata current() and current_values() accessors
...and basic smoke testing.
2025-05-23 19:32:50 +02:00
Arne Welzel
53b0f0ad64 Event: Deprecate default network timestamp metadata
This deprecates the Event constructor and the ``ts`` parameter of Enqueue()
Instead, versions are introduced that take a detail::MetadataVectorPtr which
can hold the network timestamp metadata and is meant to be allocated by the
caller instead of automatically during Enqueue() or within the Event
constructor.

This also introduces a BifConst ``EventMetadata::add_network_timestamp`` to
opt-in adding network timestamps to events globally. It's disabled by
default as there are not a lot of known use cases that need this.
2025-05-23 19:32:23 +02:00
Arne Welzel
12c523f3f7 Event: Store timestamp in metadata vector
This removes the ts attribute from Event and instead allocates a vector for
storing metadata. By default, adds the network time as a TimeVal. Later
patches will make the allocation of the vector optional by introducing a
different constructor so that users that are not interested in network
timestamp metadata do not take the allocation hit.

Moving the explicit ``ts`` out of the event is done in order to treat it
just as generic metadata, too. However, the Time() accessor is adapted to
lookup the value from the metadata vector instead.
2025-05-23 19:32:23 +02:00
Arne Welzel
cc7dc60c1e EventRegistry/zeek.bif/init-bare: Add event metadata infrastructure
Introduce a new EventMetadata module and members on EventMgr to register
event metadata types.
2025-05-23 19:31:58 +02:00
Arne Welzel
fc96c81c70 EventMgr: Add CurrentEvent() accessor
Avoid proliferation of accessors on EventMgr.
2025-05-23 19:31:06 +02:00
Tim Wojtulewicz
57b618ee4a Merge remote-tracking branch 'origin/topic/timw/remove-findclangtidy'
* origin/topic/timw/remove-findclangtidy:
  Remove FindClangTidy.cmake, update cmake submodule
2025-05-23 08:32:13 -07:00
Tim Wojtulewicz
b091f5b5c3 Remove FindClangTidy.cmake, update cmake submodule 2025-05-23 08:30:48 -07:00
zeek-bot
7110a8b838 Update doc submodule [nomail] [skip ci] 2025-05-23 00:16:11 +00:00
Tim Wojtulewicz
bb17d9e4d6 Merge remote-tracking branch 'origin/topic/timw/sqlite-cluster-test-followup'
* origin/topic/timw/sqlite-cluster-test-followup:
  SQLite: Add TODO note about possibly using sqlite3_busy_timeout
  SQLite: Fix typo in variable name causing pragmas not to retry on busy
  SQLite: Use tableval iteration instead of ToMap for pragmas
  SQLite: Fix logging/error messages around executing pragmas
2025-05-22 16:55:20 -07:00
Tim Wojtulewicz
8a27b894b6 SQLite: Add TODO note about possibly using sqlite3_busy_timeout 2025-05-22 10:23:17 -07:00
Tim Wojtulewicz
25f144381c SQLite: Fix typo in variable name causing pragmas not to retry on busy 2025-05-22 10:23:17 -07:00
Tim Wojtulewicz
57d797e087 SQLite: Use tableval iteration instead of ToMap for pragmas 2025-05-22 10:22:40 -07:00
Tim Wojtulewicz
384ea046b3 SQLite: Fix logging/error messages around executing pragmas 2025-05-22 10:22:40 -07:00
zeek-bot
0864048c3f Update doc submodule [nomail] [skip ci] 2025-05-22 00:15:56 +00:00
Tim Wojtulewicz
25dd1a2702 Disable sqlite-cluster btest
This test is being flaky on some platforms and still having problems
with executing pragmas at startup. Disable it for now until it can be
fixed.
2025-05-21 15:42:29 -07:00
Tim Wojtulewicz
6b104f3e22 Merge remote-tracking branch 'origin/topic/timw/fix-sqlite-btest-output'
* origin/topic/timw/fix-sqlite-btest-output:
  Update baseline of sqlite-basic btest after recent changes
2025-05-21 12:47:16 -07:00
Tim Wojtulewicz
0c7ad126d6 Update baseline of sqlite-basic btest after recent changes 2025-05-21 11:01:54 -07:00
Tim Wojtulewicz
1862e66097 Merge remote-tracking branch 'origin/topic/timw/sqlite-cluster-test'
* origin/topic/timw/sqlite-cluster-test:
  SQLite: Move integrity_check to pragma table
  SQLite: Add backend option for pragma timeout
  SQLite: Rename tuning_params to pragma_commands, move running pragmas to utility method
  SQLite: Retry pragma statements at startup to avoid contention
  SQLite: Check for locked database as well as busy databases
  SQLite: Fix some string-sizing issues
  SQLite: Run pragmas on connection before creating table
  SQLite: Add busy_timeout pragma to default options
  Prefix sqlite-based btests with sqlite- to match redis tests
  Add sqlite cluster storage btest
2025-05-21 09:41:15 -07:00
Tim Wojtulewicz
a58128a45c SQLite: Move integrity_check to pragma table 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
78dffb1d6f SQLite: Add backend option for pragma timeout 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
f0e7b78554 SQLite: Rename tuning_params to pragma_commands, move running pragmas to utility method 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
53cb3c3681 SQLite: Retry pragma statements at startup to avoid contention 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
227d24b64d SQLite: Check for locked database as well as busy databases 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
1bff4c9537 SQLite: Fix some string-sizing issues 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
ac064bd6af SQLite: Run pragmas on connection before creating table 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
26cc6d4e7b SQLite: Add busy_timeout pragma to default options 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
e91421a8de Prefix sqlite-based btests with sqlite- to match redis tests 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
41bddae59f Add sqlite cluster storage btest 2025-05-21 09:38:27 -07:00
Arne Welzel
891338194f Merge remote-tracking branch 'origin/topic/awelzel/fixup-detect-sql-injection-zeekygen-comment'
* origin/topic/awelzel/fixup-detect-sql-injection-zeekygen-comment:
  http/detect-sql-injection: Fix zeekygen comment
  btest/coverage/test-all-policy-zeekygen: Load test-all-policy with zeekygen enabled
2025-05-21 13:39:07 +02:00
Arne Welzel
8d588a10a9 http/detect-sql-injection: Fix zeekygen comment
Discarded extraneous Zeekygen comment: $src field; and always provides a victim IP address in the $dst field.
2025-05-21 11:49:48 +02:00
Arne Welzel
aabd1e3825 btest/coverage/test-all-policy-zeekygen: Load test-all-policy with zeekygen enabled
There should not be warnings produced. The default ZEEK_DISABLE_ZEEKYGEN=1
setting in the btest configuration hid some issues previously.
2025-05-21 11:49:12 +02:00