Commit graph

18522 commits

Author SHA1 Message Date
Arne Welzel
e4e9ec3e80 btest/zam: Update for new EventMetadata bifs 2025-05-23 19:32:50 +02:00
Arne Welzel
506fea3335 broker and cluster: Switch to new Enqueue() API
This is a bit intermediary. In part 2 this will deal with any metadata,
not just timestamps.
2025-05-23 19:32:50 +02:00
Arne Welzel
75aa6588fe Event/zeek.bif: Add EventMetadata current() and current_values() accessors
...and basic smoke testing.
2025-05-23 19:32:50 +02:00
Arne Welzel
53b0f0ad64 Event: Deprecate default network timestamp metadata
This deprecates the Event constructor and the ``ts`` parameter of Enqueue()
Instead, versions are introduced that take a detail::MetadataVectorPtr which
can hold the network timestamp metadata and is meant to be allocated by the
caller instead of automatically during Enqueue() or within the Event
constructor.

This also introduces a BifConst ``EventMetadata::add_network_timestamp`` to
opt-in adding network timestamps to events globally. It's disabled by
default as there are not a lot of known use cases that need this.
2025-05-23 19:32:23 +02:00
Arne Welzel
12c523f3f7 Event: Store timestamp in metadata vector
This removes the ts attribute from Event and instead allocates a vector for
storing metadata. By default, adds the network time as a TimeVal. Later
patches will make the allocation of the vector optional by introducing a
different constructor so that users that are not interested in network
timestamp metadata do not take the allocation hit.

Moving the explicit ``ts`` out of the event is done in order to treat it
just as generic metadata, too. However, the Time() accessor is adapted to
lookup the value from the metadata vector instead.
2025-05-23 19:32:23 +02:00
Arne Welzel
cc7dc60c1e EventRegistry/zeek.bif/init-bare: Add event metadata infrastructure
Introduce a new EventMetadata module and members on EventMgr to register
event metadata types.
2025-05-23 19:31:58 +02:00
Arne Welzel
fc96c81c70 EventMgr: Add CurrentEvent() accessor
Avoid proliferation of accessors on EventMgr.
2025-05-23 19:31:06 +02:00
Tim Wojtulewicz
57b618ee4a Merge remote-tracking branch 'origin/topic/timw/remove-findclangtidy'
* origin/topic/timw/remove-findclangtidy:
  Remove FindClangTidy.cmake, update cmake submodule
2025-05-23 08:32:13 -07:00
Tim Wojtulewicz
b091f5b5c3 Remove FindClangTidy.cmake, update cmake submodule 2025-05-23 08:30:48 -07:00
zeek-bot
7110a8b838 Update doc submodule [nomail] [skip ci] 2025-05-23 00:16:11 +00:00
Tim Wojtulewicz
bb17d9e4d6 Merge remote-tracking branch 'origin/topic/timw/sqlite-cluster-test-followup'
* origin/topic/timw/sqlite-cluster-test-followup:
  SQLite: Add TODO note about possibly using sqlite3_busy_timeout
  SQLite: Fix typo in variable name causing pragmas not to retry on busy
  SQLite: Use tableval iteration instead of ToMap for pragmas
  SQLite: Fix logging/error messages around executing pragmas
2025-05-22 16:55:20 -07:00
Tim Wojtulewicz
8a27b894b6 SQLite: Add TODO note about possibly using sqlite3_busy_timeout 2025-05-22 10:23:17 -07:00
Tim Wojtulewicz
25f144381c SQLite: Fix typo in variable name causing pragmas not to retry on busy 2025-05-22 10:23:17 -07:00
Tim Wojtulewicz
57d797e087 SQLite: Use tableval iteration instead of ToMap for pragmas 2025-05-22 10:22:40 -07:00
Tim Wojtulewicz
384ea046b3 SQLite: Fix logging/error messages around executing pragmas 2025-05-22 10:22:40 -07:00
zeek-bot
0864048c3f Update doc submodule [nomail] [skip ci] 2025-05-22 00:15:56 +00:00
Tim Wojtulewicz
25dd1a2702 Disable sqlite-cluster btest
This test is being flaky on some platforms and still having problems
with executing pragmas at startup. Disable it for now until it can be
fixed.
2025-05-21 15:42:29 -07:00
Tim Wojtulewicz
6b104f3e22 Merge remote-tracking branch 'origin/topic/timw/fix-sqlite-btest-output'
* origin/topic/timw/fix-sqlite-btest-output:
  Update baseline of sqlite-basic btest after recent changes
2025-05-21 12:47:16 -07:00
Tim Wojtulewicz
0c7ad126d6 Update baseline of sqlite-basic btest after recent changes 2025-05-21 11:01:54 -07:00
Tim Wojtulewicz
1862e66097 Merge remote-tracking branch 'origin/topic/timw/sqlite-cluster-test'
* origin/topic/timw/sqlite-cluster-test:
  SQLite: Move integrity_check to pragma table
  SQLite: Add backend option for pragma timeout
  SQLite: Rename tuning_params to pragma_commands, move running pragmas to utility method
  SQLite: Retry pragma statements at startup to avoid contention
  SQLite: Check for locked database as well as busy databases
  SQLite: Fix some string-sizing issues
  SQLite: Run pragmas on connection before creating table
  SQLite: Add busy_timeout pragma to default options
  Prefix sqlite-based btests with sqlite- to match redis tests
  Add sqlite cluster storage btest
2025-05-21 09:41:15 -07:00
Tim Wojtulewicz
a58128a45c SQLite: Move integrity_check to pragma table 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
78dffb1d6f SQLite: Add backend option for pragma timeout 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
f0e7b78554 SQLite: Rename tuning_params to pragma_commands, move running pragmas to utility method 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
53cb3c3681 SQLite: Retry pragma statements at startup to avoid contention 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
227d24b64d SQLite: Check for locked database as well as busy databases 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
1bff4c9537 SQLite: Fix some string-sizing issues 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
ac064bd6af SQLite: Run pragmas on connection before creating table 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
26cc6d4e7b SQLite: Add busy_timeout pragma to default options 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
e91421a8de Prefix sqlite-based btests with sqlite- to match redis tests 2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
41bddae59f Add sqlite cluster storage btest 2025-05-21 09:38:27 -07:00
Arne Welzel
891338194f Merge remote-tracking branch 'origin/topic/awelzel/fixup-detect-sql-injection-zeekygen-comment'
* origin/topic/awelzel/fixup-detect-sql-injection-zeekygen-comment:
  http/detect-sql-injection: Fix zeekygen comment
  btest/coverage/test-all-policy-zeekygen: Load test-all-policy with zeekygen enabled
2025-05-21 13:39:07 +02:00
Arne Welzel
8d588a10a9 http/detect-sql-injection: Fix zeekygen comment
Discarded extraneous Zeekygen comment: $src field; and always provides a victim IP address in the $dst field.
2025-05-21 11:49:48 +02:00
Arne Welzel
aabd1e3825 btest/coverage/test-all-policy-zeekygen: Load test-all-policy with zeekygen enabled
There should not be warnings produced. The default ZEEK_DISABLE_ZEEKYGEN=1
setting in the btest configuration hid some issues previously.
2025-05-21 11:49:12 +02:00
zeek-bot
7d54a58a76 Update doc submodule [nomail] [skip ci] 2025-05-21 00:15:25 +00:00
Tim Wojtulewicz
c596556036 Merge remote-tracking branch 'origin/topic/timw/3913-parse-port-invalid-read'
* origin/topic/timw/3913-parse-port-invalid-read:
  Add extra input files to ftp fuzzer corpus
  Use bool instead of int flag in FTP analyzer's parse_eftp method
  Fix undefined behavior in FTP analyzer's parse_port method
  Fix invalid-read in FTP analyzer's parse_port method
2025-05-20 12:02:18 -07:00
Arne Welzel
ec744024d3 Merge remote-tracking branch 'origin/topic/awelzel/btest-cluster-layout-replacement'
* origin/topic/awelzel/btest-cluster-layout-replacement:
  btest remaining: Use generic cluster-layout.zeek
  btest/frameworks/logging: Use generic cluster-layout.zeek
  btest/files/x509: Use generic cluster-layout.zeek
  btest/frameworks/notice: Use generic cluster-layout.zeek
  btest/policy: Use generic cluster-layout.zeek
  btest/frameworks/intel: Use generic cluster-layout.zeek
  btest/frameworks/sumstats: Use generic cluster-layout.zeek
  btest/frameworks/cluster: Use generic cluster-layout.zeek
  btest/frameworks/config: Use generic cluster-layout.zeek
  btest/broker: Use generic cluster-layout.zeek
  btest/cluster: Use generic cluster-layout.zeek
  testing/btest/Files: Add generic broker/cluster-layout.zeek
2025-05-20 20:30:47 +02:00
Arne Welzel
00eabb6cbb btest remaining: Use generic cluster-layout.zeek 2025-05-20 20:30:01 +02:00
Arne Welzel
9365f71965 btest/frameworks/logging: Use generic cluster-layout.zeek 2025-05-20 20:30:01 +02:00
Arne Welzel
6c00c05249 btest/files/x509: Use generic cluster-layout.zeek 2025-05-20 20:30:01 +02:00
Arne Welzel
d7b5955e5e btest/frameworks/notice: Use generic cluster-layout.zeek 2025-05-20 20:30:01 +02:00
Arne Welzel
2a3f2d0004 btest/policy: Use generic cluster-layout.zeek
Catch-all for cluster scripts in policy directory.
2025-05-20 20:30:01 +02:00
Arne Welzel
00a12a4cc5 btest/frameworks/intel: Use generic cluster-layout.zeek 2025-05-20 20:30:01 +02:00
Arne Welzel
4dec63936e btest/frameworks/sumstats: Use generic cluster-layout.zeek 2025-05-20 20:30:01 +02:00
Arne Welzel
0a06a77c69 btest/frameworks/cluster: Use generic cluster-layout.zeek 2025-05-20 20:30:01 +02:00
Arne Welzel
e114b0e371 btest/frameworks/config: Use generic cluster-layout.zeek 2025-05-20 20:30:01 +02:00
Arne Welzel
f9ff396acf btest/broker: Use generic cluster-layout.zeek 2025-05-20 20:30:01 +02:00
Arne Welzel
a15df5fc11 btest/cluster: Use generic cluster-layout.zeek 2025-05-20 20:30:01 +02:00
Arne Welzel
b3f53dc455 testing/btest/Files: Add generic broker/cluster-layout.zeek
This is a cluster-layout.zeek template that can be copied into a testing
directory if needed. The idea is that a developer sets environment
variables within their btest and the Cluster::nodes variable is
implicitly extended by appropriate nodes.

For example, using @TEST-PORT BROKER_LOGGER1_PORT will add an appropriate
logger-1 node to Cluster::nodes, based on the existence of the
BROKER_LOGGER1_PORT environment variable.
2025-05-20 20:30:01 +02:00
Tim Wojtulewicz
9ae16a3db3 Add extra input files to ftp fuzzer corpus 2025-05-20 10:58:32 -07:00
Tim Wojtulewicz
517dfff529 Use bool instead of int flag in FTP analyzer's parse_eftp method 2025-05-20 10:58:32 -07:00