Commit graph

202 commits

Author SHA1 Message Date
Johanna Amann
130c89a0a7 dpd->analyzer.log change - rename files
To address review feedback in GH-4362: rename analyzer-failed-log.zeek
to loggig.zeek, analyzer-debug-log.zeek to debug-logging.zeek and
dpd-log.zeek to deprecated-dpd-log.zeek.

Includes respective test, NEWS, etc updates.
2025-06-03 16:32:52 +01:00
Johanna Amann
af77a7a83b Analyzer failure logging: tweaks and test fixes
The main part of this commit are changes in tests. A lot of the tests
that previously relied on analyzer.log or dpd.log now use the new
analyzer-failed.log.

I verified all the changes and, as far as I can tell, everything
behaves as it should. This includes the external test baselines.

This change also enables logging of file and packet analyzer to
analyzer_failed.log and fixes some small behavior issues.

The analyzer_failed event is no longer raised when the removal of an
analyzer is vetoed.

If an analyzer is no longer active when an analyzer violation is raised,
currently the analyzer_failed event is raised. This can, e.g., happen
when an analyzer error happens at the very end of the connection. This
makes the behavior more similar to what happened in the past, and also
intuitively seems to make sense.

A bug introduced in the failed service logging was fixed.
2025-06-03 15:56:42 +01:00
Arne Welzel
0ab53c75cd btest/plugin: Test custom metadata publish
Usage demo for plugin writers to add custom event metadata and access in
in Zeek scripts.
2025-06-02 17:31:36 +02:00
Tim Wojtulewicz
456c1fa42c Remove unnecessary #includes in analyzer/packet analyzer/file analyzer headers 2025-05-19 10:25:05 -07:00
Tim Wojtulewicz
e618d00326 Remove including <cinttypes> from util.h 2025-05-16 10:14:37 -07:00
Tim Wojtulewicz
c659592773 Reduce includes in plugin/Component.h 2025-05-16 10:14:37 -07:00
Tim Wojtulewicz
648f0f0623 Use quotes instead of <> for zeek includes 2025-05-16 10:14:36 -07:00
Arne Welzel
0bf3417d4c btest/plugin: Test for PublishEventHook() 2025-04-30 17:26:33 +02:00
Arne Welzel
85b8c8866b testing/btest/*zeek: Comment all @TEST lines 2025-04-17 16:30:23 +02:00
Tim Wojtulewicz
cb1ef47a31 Add STORAGE_ prefixes for backends and serializers 2025-04-14 10:11:13 -07:00
Tim Wojtulewicz
1169fcf2a2 Move byte_buffer types from cluster and storage into util 2025-04-14 10:11:13 -07:00
Tim Wojtulewicz
88786a28a2 Add JSON storage serializer, use with existing backends/tests 2025-04-14 10:11:13 -07:00
Arne Welzel
6d97d5526a plugin/Manager: Fix MetaHookPre and MetaHookPost using HOOK_CALL_FUNCTION 2025-04-11 09:58:07 +02:00
Arne Welzel
cb610bdea2 btest/plugin: Add plugin testing enum identifiers
This plugin has a generic name, but for now just tests the API around
enum types and enum vals at InitPostScript() time.
2025-04-04 15:36:42 +02:00
Tim Wojtulewicz
32ae8f4eaa Make storage events take a tag for the backend instead of a string 2025-03-27 16:12:24 -07:00
Tim Wojtulewicz
85701e4514 Fix handling of timeout conditions from storage backends 2025-03-27 13:52:11 -07:00
Tim Wojtulewicz
046f32a6df Reformat plugin.storage btest to be more consistent with other storage tests 2025-03-27 13:52:11 -07:00
Tim Wojtulewicz
ba9cf1e4db Remove unnecessary type aliases from storage btests 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
e5b06367f7 Squash code from OperationResultCallback into ResultCallback 2025-03-21 11:44:39 -07:00
Tim Wojtulewicz
e6f1eea1b7 Remove default argument for callbacks, reorder function arguments 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
9ed3e33f97 Completely rework return values from storage operations 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
e766af7322 Split sync/async handling into the BIF methods 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
a485b1d237 Make backend options a record, move actual options to be sub-records 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
28951dccf1 Split sync and async into separate script-land namespaces 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
f1a7376e0a Return generic result for get operations that includes error messages 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
4695060d75 Allow opening and closing backends to be async 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
7ad6a05f5b Add infrastructure for asynchronous storage operations 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
d07d27453a Add infrastructure for automated expiration of storage entries
This is used for backends that don't support expiration natively.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
8dee733a7d Change args to Storage::put to be a record
The number of args being passed to the put() methods was getting to be
fairly long, with more on the horizon. Changing to a record means simplifying
things a little bit.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
69d940533d Pass key/value types for validation when opening backends 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
2ea0f3e70a Lay out initial parts for the Storage framework
This includes a manager, component manager, BIF and script code, and
parts to support new storage backend plugins.
2025-03-18 09:32:34 -07:00
Arne Welzel
ab99f8e233 scan.l: Fix @load-plugin scripts loading
For a plugin loaded via @load-plugin, create a YY_BUFFER_STATE holding
the required loads for the implicitly loaded files. In loaded scripts,
this generated file will show up with a path of the shared object file
of the plugin with the __preload__.zeek and __load__.zeek files loaded
by it.

Closes #2311
2025-03-04 09:35:00 +01:00
Arne Welzel
927a06b9ab logging: Fix HookLogInit() and HookLogWrite() info usage
There's two instances of WriterBackend::WriterInfo for a given
writer. One in Manager::WriterInfo that's accessible via
stream.writers and a copy within WriterFrontend.

Commit 78999d147d switched to use the
address of the frontend's info instance for HookLogWrite() invocations,
breaking users using the address for identification purposes.
2025-01-14 10:44:50 +01:00
Arne Welzel
85189ca918 btest: Avoid loading ZeroMQ if not compiled in
...at the same time, add some `TEST-REQUIRES: have-zeromq` which
unfortunately means that developers will usually want libzmq
installed on their system.
2024-12-11 18:08:43 +01:00
Arne Welzel
39b43e8dd6 btest/plugins: Add a plugin testing Init and Done hooks
Follow-up for #4047.
2024-11-22 13:37:03 +01:00
Christian Kreibich
b1e0e26484 Make debug stream names use "-", and handle "_" transparently
This looks consistent, and doesn't break any existing uses with underscores.
2024-11-15 17:53:29 -08:00
Christian Kreibich
7563af4906 Make "-B all" apply to plugin streams as well. 2024-11-15 13:53:06 -08:00
Christian Kreibich
5e4942bcc0 Sort streams in "-B help" output, and match case-insensitively throughout
Matching of plugins' debug streams was still case-sensitive. Also contains some
minor output tweaks.

It'd be nice to only list plugin debug streams actually _used_ by plugins. I
didn't see a quick way to do that so that's for another time.
2024-11-15 12:50:13 -08:00
Arne Welzel
d18045ee16 testing: Add have-spicy-ssl helper and update tests 2024-10-29 09:30:01 +01:00
Johanna Amann
cf1074518e Spicy SSL analyzer - address feedback
Minor proposed changes from https://github.com/zeek/zeek/pull/3765,
mostly cosmetic.

Changes CI to be part of an already existing spicy CI job.
2024-09-11 11:30:16 +02:00
Johanna Amann
84c4d53a4e Spicy TLS - full test suite pass
With this commit, the entire Zeek test suite passes using spicy TLS.
Tests that either use a SSLv2 handshake, or DTLS are skipped, as the
parser currently does not support either.

Similarly, tests that rely on behavior we cannot replicate (baseline,
hooks, exact error messages) are passed. Other than that, all the
TLS-based tests pass with 100% the exact same baseline results.

This necessitated a couple of small tweaks to the spicy file - the
testcases uncovered several small problems.

This commit also enables cirrus tests for Spicy SSL/TLS.
2024-08-13 14:41:37 +01:00
Tim Wojtulewicz
9de1dd16d5 Remove variant from StdFunctionStmt
The variant ended up conflicting with std::bind, which resulted in failures
on the btest invoking it. Change back to a single function that takes a
flow, and default it to a value in Exec.
2024-03-19 09:07:45 -07:00
Tim Wojtulewicz
39890fda5e Add a method to register an event handler to a std::function via C++ 2024-03-18 16:36:25 -07:00
Dominik Charousset
cebb85b1e8 Fix unsafe and inefficient uses of copy_string
Add a new overload to `copy_string` that takes the input characters plus
size. The new overload avoids inefficient scanning of the input for the
null terminator in cases where we know the size beforehand. Furthermore,
this overload *must* be used when dealing with input character sequences
that may have no null terminator, e.g., when the input is from a
`std::string_view` object.
2023-11-03 15:25:38 +01:00
Benjamin Bannier
f5a76c1aed Reformat Zeek in Spicy style
This largely copies over Spicy's `.clang-format` configuration file. The
one place where we deviate is header include order since Zeek depends on
headers being included in a certain order.
2023-10-30 09:40:55 +01:00
Arne Welzel
07ac6fa074 btest/plugins/hooks: Run in bare mode
Motivation is basically the same as in 88bb527026.
For plugin.hooks, one example is that adding a new option in the default script
changes the baseline due registration of change handlers. Also, the connection
record is printed in various places, resulting in churn when the default
scripts change.
2023-10-09 16:13:59 +02:00
Benjamin Bannier
0896f09081 Exclude dot directories when searching ZEEK_PLUGIN_PATH
Dot directories rarely contain anything we would want to load as a
dynamic plugin. Even worse, they likely contain files with externally
controlled lifetimes which might be removed while we are using them
(see e.g., zeek/btest#98).

With this patch we do not search _discovered_ dot directories anymore.
We continue to load from a user-specified `ZEEK_PLUGIN_PATH`, even if
its name starts with a dot.

Since this patch changes previous behavior it is a **BREAKING CHANGE**.
2023-09-26 11:56:47 +02:00
Arne Welzel
5a9c6d4633 Merge remote-tracking branch 'origin/topic/awelzel/duplicate-plugin-no-debug'
* origin/topic/awelzel/duplicate-plugin-no-debug:
  plugin/Manager: Warn when plugin with the same name is found
2023-06-15 18:39:49 +02:00
Tim Wojtulewicz
4229af6820 Remove deprecations tagged for v6.1 2023-06-14 10:07:22 -07:00
Arne Welzel
fdd6dfd096 plugin/Manager: Warn when plugin with the same name is found
This was brought up on Slack as a potential source of confusion during
development as it's not visible when plugin directories are ignored outside
of looking into the plugin debug stream. I'd actually prefer to just
FatalError() this, but a warning seems reasonably visible for interactive
usage.
2023-06-14 17:36:18 +02:00