Commit graph

216 commits

Author SHA1 Message Date
Tim Wojtulewicz
d95affde4d Remove deprecations tagged for v8.1 2025-08-12 10:19:03 -07:00
Tim Wojtulewicz
54d67c3322 Merge remote-tracking branch 'origin/topic/timw/cleanup-warnings-from-plugin-btest-builds'
* origin/topic/timw/cleanup-warnings-from-plugin-btest-builds:
  Update zeek-aux to remove BRO_DIST from plugin skeleton
  cmake_minimum_required() should come before project()
2025-08-07 08:39:40 -07:00
Tim Wojtulewicz
162ecc022e Update zeek-aux to remove BRO_DIST from plugin skeleton 2025-08-07 08:39:08 -07:00
Tim Wojtulewicz
3c535ec215 cmake_minimum_required() should come before project() 2025-08-06 12:10:41 -07:00
Arne Welzel
f98508bbb0 btest/tap-analyzer: Update existing test and add new one for UpdateConnVal()
This also changes the output of connection UIDs from the tap analyzer to be
prefixed with C for easier correlation with other logs.

Relates to #4337 #4725 #4734 #4737
2025-08-06 17:22:59 +02:00
Arne Welzel
b4925fbd16 TapAnalyzer: More verdict to action rename
Relates to #4725 #4337
2025-08-05 19:59:06 +02:00
Arne Welzel
4bc7f9532c IPBasedAnalyzer: Call TapPacket() when skipping
When skip_further_processing() is called, a TapAnalyzer should still see
the packets as skipped with SkipReason "skipping".
2025-08-05 19:47:04 +02:00
Arne Welzel
dc904b2216 SessionAdapter: Introduce TapAnalyzer for session adapter
This commit introduces a mechanism to attach light weight analyzers to
the root analyzer of sessions in order to tap into the packets delivered
to child analyzer.
2025-08-05 19:47:02 +02:00
Arne Welzel
bda70067ec cluster: Move cluster::detail::Event to cluster::Event
This class is a parameter of virtual methods of the Backend API for users
to implement and also a parameter to the HookPublishEvent() API. Seems it
shouldn't be in detail and instead we should own it.

Alternatively, could mark the cluster APIs as not-stable-yet, but I
think we can move forward and make it non-detail for 8.0.
2025-07-29 18:13:59 +02:00
Tim Wojtulewicz
a0ffe7f748 Add storage metrics for operations, expirations, data transferred 2025-07-18 14:28:04 -07:00
Vern Paxson
5c63133226 isolate Location specifics to private class variables to enforce correct line number ordering 2025-07-08 10:39:28 +02:00
Arne Welzel
b7a22a87c6 ConnKey: Extend DoPopulateConnIdVal() with ctx
This prepares the move where ConnKey implementations should fill out
ctx rather than filling conn_id directly. The API continues to receive
both, conn_id and ctx, as adding fields to `conn_id` is reasonable
use-case even if it's just for logging purposes.
2025-07-03 18:19:46 +02:00
Arne Welzel
e7b1b174f0 btest/plugins: Add test for custom ConnKey factory
This just counts DoInits() and adds that information to the conn_id
record, but without including it into the hash. Mostly for smoke
testing.
2025-06-25 13:19:26 +02:00
Johanna Amann
0c875220e9 Default canonifier change to only remove first timestamp in line
In the past, we used a default canonifier, which removes everything that
looks like a timestamp from log files. The goal of this is to prevent
logs from changing, e.g., due to local system times ending up in log
files.

This, however, also has the side-effect of removing information that is
parsed from protocols which probably should be part of our tests.
There is at least one test (1999 certificates) where the entire test
output was essentially removed by the canonifier.

GH-4521 was similarly masked by this.

This commit changes the default canonifier, so that only the first
timestamp in a line is removed. This should skip timestamps that are
likely to change while keeping timestamps that are parsed
from protocol information.

A pass has been made over the tests, with some additional adjustments
for cases which require the old canonifier.

There are some cases in which we probably could go further and not
remove timestamps at all - that, however, seems like a follow-up
project.
2025-06-18 15:41:48 +01:00
Johanna Amann
130c89a0a7 dpd->analyzer.log change - rename files
To address review feedback in GH-4362: rename analyzer-failed-log.zeek
to loggig.zeek, analyzer-debug-log.zeek to debug-logging.zeek and
dpd-log.zeek to deprecated-dpd-log.zeek.

Includes respective test, NEWS, etc updates.
2025-06-03 16:32:52 +01:00
Johanna Amann
af77a7a83b Analyzer failure logging: tweaks and test fixes
The main part of this commit are changes in tests. A lot of the tests
that previously relied on analyzer.log or dpd.log now use the new
analyzer-failed.log.

I verified all the changes and, as far as I can tell, everything
behaves as it should. This includes the external test baselines.

This change also enables logging of file and packet analyzer to
analyzer_failed.log and fixes some small behavior issues.

The analyzer_failed event is no longer raised when the removal of an
analyzer is vetoed.

If an analyzer is no longer active when an analyzer violation is raised,
currently the analyzer_failed event is raised. This can, e.g., happen
when an analyzer error happens at the very end of the connection. This
makes the behavior more similar to what happened in the past, and also
intuitively seems to make sense.

A bug introduced in the failed service logging was fixed.
2025-06-03 15:56:42 +01:00
Arne Welzel
0ab53c75cd btest/plugin: Test custom metadata publish
Usage demo for plugin writers to add custom event metadata and access in
in Zeek scripts.
2025-06-02 17:31:36 +02:00
Tim Wojtulewicz
456c1fa42c Remove unnecessary #includes in analyzer/packet analyzer/file analyzer headers 2025-05-19 10:25:05 -07:00
Tim Wojtulewicz
e618d00326 Remove including <cinttypes> from util.h 2025-05-16 10:14:37 -07:00
Tim Wojtulewicz
c659592773 Reduce includes in plugin/Component.h 2025-05-16 10:14:37 -07:00
Tim Wojtulewicz
648f0f0623 Use quotes instead of <> for zeek includes 2025-05-16 10:14:36 -07:00
Arne Welzel
0bf3417d4c btest/plugin: Test for PublishEventHook() 2025-04-30 17:26:33 +02:00
Arne Welzel
85b8c8866b testing/btest/*zeek: Comment all @TEST lines 2025-04-17 16:30:23 +02:00
Tim Wojtulewicz
cb1ef47a31 Add STORAGE_ prefixes for backends and serializers 2025-04-14 10:11:13 -07:00
Tim Wojtulewicz
1169fcf2a2 Move byte_buffer types from cluster and storage into util 2025-04-14 10:11:13 -07:00
Tim Wojtulewicz
88786a28a2 Add JSON storage serializer, use with existing backends/tests 2025-04-14 10:11:13 -07:00
Arne Welzel
6d97d5526a plugin/Manager: Fix MetaHookPre and MetaHookPost using HOOK_CALL_FUNCTION 2025-04-11 09:58:07 +02:00
Arne Welzel
cb610bdea2 btest/plugin: Add plugin testing enum identifiers
This plugin has a generic name, but for now just tests the API around
enum types and enum vals at InitPostScript() time.
2025-04-04 15:36:42 +02:00
Tim Wojtulewicz
32ae8f4eaa Make storage events take a tag for the backend instead of a string 2025-03-27 16:12:24 -07:00
Tim Wojtulewicz
85701e4514 Fix handling of timeout conditions from storage backends 2025-03-27 13:52:11 -07:00
Tim Wojtulewicz
046f32a6df Reformat plugin.storage btest to be more consistent with other storage tests 2025-03-27 13:52:11 -07:00
Tim Wojtulewicz
ba9cf1e4db Remove unnecessary type aliases from storage btests 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
e5b06367f7 Squash code from OperationResultCallback into ResultCallback 2025-03-21 11:44:39 -07:00
Tim Wojtulewicz
e6f1eea1b7 Remove default argument for callbacks, reorder function arguments 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
9ed3e33f97 Completely rework return values from storage operations 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
e766af7322 Split sync/async handling into the BIF methods 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
a485b1d237 Make backend options a record, move actual options to be sub-records 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
28951dccf1 Split sync and async into separate script-land namespaces 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
f1a7376e0a Return generic result for get operations that includes error messages 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
4695060d75 Allow opening and closing backends to be async 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
7ad6a05f5b Add infrastructure for asynchronous storage operations 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
d07d27453a Add infrastructure for automated expiration of storage entries
This is used for backends that don't support expiration natively.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
8dee733a7d Change args to Storage::put to be a record
The number of args being passed to the put() methods was getting to be
fairly long, with more on the horizon. Changing to a record means simplifying
things a little bit.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
69d940533d Pass key/value types for validation when opening backends 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
2ea0f3e70a Lay out initial parts for the Storage framework
This includes a manager, component manager, BIF and script code, and
parts to support new storage backend plugins.
2025-03-18 09:32:34 -07:00
Arne Welzel
ab99f8e233 scan.l: Fix @load-plugin scripts loading
For a plugin loaded via @load-plugin, create a YY_BUFFER_STATE holding
the required loads for the implicitly loaded files. In loaded scripts,
this generated file will show up with a path of the shared object file
of the plugin with the __preload__.zeek and __load__.zeek files loaded
by it.

Closes #2311
2025-03-04 09:35:00 +01:00
Arne Welzel
927a06b9ab logging: Fix HookLogInit() and HookLogWrite() info usage
There's two instances of WriterBackend::WriterInfo for a given
writer. One in Manager::WriterInfo that's accessible via
stream.writers and a copy within WriterFrontend.

Commit 78999d147d switched to use the
address of the frontend's info instance for HookLogWrite() invocations,
breaking users using the address for identification purposes.
2025-01-14 10:44:50 +01:00
Arne Welzel
85189ca918 btest: Avoid loading ZeroMQ if not compiled in
...at the same time, add some `TEST-REQUIRES: have-zeromq` which
unfortunately means that developers will usually want libzmq
installed on their system.
2024-12-11 18:08:43 +01:00
Arne Welzel
39b43e8dd6 btest/plugins: Add a plugin testing Init and Done hooks
Follow-up for #4047.
2024-11-22 13:37:03 +01:00
Christian Kreibich
b1e0e26484 Make debug stream names use "-", and handle "_" transparently
This looks consistent, and doesn't break any existing uses with underscores.
2024-11-15 17:53:29 -08:00