Commit graph

5128 commits

Author SHA1 Message Date
Evan Typanski
efe1ad2581 Make types into constants
Closes #4173

This allows types to be used in expressions, but they can't be
reassigned. Note that this was meant to be a special "type expression" -
but that is unnecessary complexity.

Type expressions would allow access to the type without going through
its constant value, but the constant value is never changed, so it's
simply a few more checks if necessary when functionality gets expanded.
This way, ZAM and other code will not need updates, so the potential for
increased work in the future is probably not worth caring about.
2025-01-24 11:16:10 -05:00
Tim Wojtulewicz
9b75de4e09 Merge some GLOBAL module sections in init-bare 2025-01-23 11:21:08 -07:00
Arne Welzel
3b4b379950 script_opt: Use emit_builtin_error() for Lookup-Conn
Still create a separate baseline file because ZAM doesn't provide
detailed location information.

Patch for ZBI.op provided by @vpax
2025-01-23 09:35:22 +01:00
Arne Welzel
17836ef7d9 session/Manager: Emit explicit errors for FindConnection() with proto=65535
We silently broke users constructing conn_id records manually and
subsequently using them with lookup_connection() or connection_exists().

This is an attempt to at least report a runtime error about the situation
so it doesn't go completely unnoticed.
2025-01-17 17:57:49 +01:00
Arne Welzel
868eb129bf Merge remote-tracking branch 'origin/topic/awelzel/fix-writer-info-in-logging-hooks'
* origin/topic/awelzel/fix-writer-info-in-logging-hooks:
  logging: Fix reporter message
  logging: Avoid repeated writer name lookups for plugin hooks
  logging: Fix HookLogInit() and HookLogWrite() info usage
2025-01-14 21:39:31 +01:00
Johanna Amann
b2222e97a1 Merge remote-tracking branch 'origin/topic/johanna/gh-4061'
* origin/topic/johanna/gh-4061:
  Update BiF-tracking, add is_event_handled
  Address review comments and small updates for DNS warnings
  Raise warnings when for DNS events that are not raised due to dns_skip_all_addl
2025-01-14 14:39:56 +00:00
Arne Welzel
927a06b9ab logging: Fix HookLogInit() and HookLogWrite() info usage
There's two instances of WriterBackend::WriterInfo for a given
writer. One in Manager::WriterInfo that's accessible via
stream.writers and a copy within WriterFrontend.

Commit 78999d147d switched to use the
address of the frontend's info instance for HookLogWrite() invocations,
breaking users using the address for identification purposes.
2025-01-14 10:44:50 +01:00
Johanna Amann
6bfa55904c Update BiF-tracking, add is_event_handled 2025-01-14 09:33:49 +00:00
Johanna Amann
13f042cc27 Address review comments and small updates for DNS warnings
This commit addresses review feedback for DH-4155. Furthermore it fixes
test failures, and adds a new test for the is_event_handled bif.
2025-01-14 09:33:48 +00:00
Christian Kreibich
f2d54db694 Merge remote-tracking branch 'origin/topic/etyp/harden-flaky-test'
* origin/topic/etyp/harden-flaky-test:
  Harden flaky test based on creating a file
2025-01-10 11:15:45 -08:00
Tim Wojtulewicz
0fcbc8546e Update btests for new local-only subnets 2025-01-09 22:16:42 -07:00
Evan Typanski
34f13e7291 Harden flaky test based on creating a file
Closes #4102

Surely it won't take over 10 seconds to create the file
2025-01-09 11:06:04 -05:00
Johanna Amann
9f72353a41 Raise warnings when for DNS events that are not raised due to dns_skip_all_addl
By default, dns_skip_all_addl is set to false. This causes several
events to not be raised. This change emits warnings when a user defines
event handlers for events that will not be raised.

Furthermore, it adds notes about this behavior to the documentation. We
also introduce a new BIF, `is_event_handled`, which checks if an event
is handled.

Fixes GH-4061
2025-01-07 17:46:27 +00:00
Vern Paxson
7a908a2876 Extended ZAM validation to include macros 2025-01-02 15:32:30 -08:00
Benjamin Bannier
4278ab2b04 Bump auxil/spicy to latest development snapshot 2024-12-20 15:18:20 -07:00
Tim Wojtulewicz
eb8f99da92 Require spicy for spicy.analyzer-id btest 2024-12-20 14:25:04 -07:00
Tim Wojtulewicz
fd4f25965d Add spicy runtime-support current_analyzer_id(), use it to set id in events 2024-12-18 15:44:09 -07:00
Arne Welzel
991bc9644d Merge remote-tracking branch 'origin/topic/vern/ZAM-field-assign-in-op'
* origin/topic/vern/ZAM-field-assign-in-op:
  pre-commit: Bump spicy-format to 0.23
  fix for ZAM optimization of assigning a record field to result of "in" operation
2024-12-18 09:28:44 +01:00
Vern Paxson
6e549554b8 fix for ZAM optimization of assigning a record field to result of "in" operation 2024-12-17 19:49:43 -08:00
Tim Wojtulewicz
6b5d0491aa Return a warning and fail if creating a store at global scope 2024-12-17 11:32:43 -07:00
Tim Wojtulewicz
33eaa5ccda Merge remote-tracking branch 'origin/topic/awelzel/4084-vector-of-pattern-compare'
* origin/topic/awelzel/4084-vector-of-pattern-compare:
  ZAM/relexpr-op NE for patterns
  Expr: Fix folding of pattern values to support == and !=
2024-12-16 12:37:46 -07:00
Vern Paxson
30de8dbea1 ZAM/relexpr-op NE for patterns 2024-12-16 19:56:57 +01:00
Arne Welzel
cd8adb3da5 Expr: Fix folding of pattern values to support == and !=
The fatal error is actually triggered at runtime, so it's a bit
dangerous for users, but not sure there's many use-cases to
compare vectors of patterns.

Closes #4084
2024-12-16 10:56:02 +01:00
Arne Welzel
7bdc856f0d QUIC/decrypt_crypto: Actually check if decryption was successful
...and bail if it wasn't.

PCAP was produced using OSS-Fuzz input from issue 383379789.
2024-12-15 20:39:26 -07:00
Tim Wojtulewicz
db3dcd5ff9 Start of 7.2.0 development 2024-12-13 13:38:53 -07:00
Tim Wojtulewicz
ee6d1374ba Merge remote-tracking branch 'origin/topic/vern/ZAM-assert-streamlining'
* origin/topic/vern/ZAM-assert-streamlining:
  btest/supervisor: Bump one timeout to 30 seconds
  btest/cluster: Bump timeouts to 30 seconds
  btest/validate-ZAM: Update baseline, run always
2024-12-13 12:17:45 -07:00
Arne Welzel
a675aea6bb btest/supervisor: Bump one timeout to 30 seconds 2024-12-13 18:35:39 +01:00
Arne Welzel
21e33fdcd9 btest/cluster: Bump timeouts to 30 seconds
ZAM startup may take a long time, particularly in CI environments, so
bump it up from 10 to 30 seconds.
2024-12-13 18:28:43 +01:00
Arne Welzel
48b26f74f7 btest/validate-ZAM: Update baseline, run always 2024-12-13 18:25:28 +01:00
Arne Welzel
9c5c8b55cc Merge remote-tracking branch 'origin/topic/vern/CPP-standalone-fixes'
* origin/topic/vern/CPP-standalone-fixes:
  BTest updates in support of compiling-scripts-to-C++
  -O gen-standalone-C++ fixes for recent more aggressive AST profiling
2024-12-13 10:57:28 +01:00
Vern Paxson
85ef6e0cb8 removed a bunch of no-longer used BTest alternatives and their associated baselines 2024-12-12 19:16:34 -08:00
Vern Paxson
8876ec6f2d BTest updates in support of compiling-scripts-to-C++ 2024-12-12 14:30:23 -08:00
Tim Wojtulewicz
25554fa668 Merge remote-tracking branch 'origin/topic/awelzel/fix-cluster-publish-any'
* origin/topic/awelzel/fix-cluster-publish-any:
  cluster/Backend: Handle unspecified table/set
  cluster: Fix Cluster::publish() of Broker::Data
  cluster: Be noisy when attempting to connect to an unknown node
2024-12-12 13:17:08 -07:00
Arne Welzel
fdf783df65 cluster/Backend: Handle unspecified table/set
Same as what we do in Broker. Use the expected type if publishing
a table() or set() parameter.

This fixes issues when switching sumstats to Cluster::publish()
2024-12-12 17:54:42 +01:00
Arne Welzel
d9a74cf32d cluster: Fix Cluster::publish() of Broker::Data
The broker serializer leverages the existing data_to_val() function.
During unserialization, if the destination type is any, the logic
simply wraps the broker::data value into a Broker::Data record.
Therefore, events with any parameters are currently exposed to
the Broker::Data type.

There is a bigger issue in that re-publishing such Broker::Data
instances would encode them as a normal record. Explicitly prevent
this by serializing the contained data value directly instead, similar
to what Broker already did when publishing a record.
2024-12-12 17:54:37 +01:00
Arne Welzel
271fc15041 cluster: Be noisy when attempting to connect to an unknown node
Mostly due to spending too much time wondering why nodes didn't connect
when there was a mismatch between "manager" and "manager-1" in the
cluster layout. Remove manager from test-all-policy-cluster test to
avoid connection attempts in this test.
2024-12-12 13:01:04 +01:00
Christian Kreibich
235638815f Fix ZAM's implementation of Analyzer::name() BiF
ZAM's implementation deviates from Zeek's in consideration of remappings, which
means that running ZAM produced conn.log service names prefixed with "spicy_"
whereas regular Zeek doesn't.
2024-12-11 23:16:47 -08:00
Tim Wojtulewicz
49f82b325b Swap pre-commit yapf for ruff/ruff-format, fix findings 2024-12-11 11:08:37 -07:00
Arne Welzel
85189ca918 btest: Avoid loading ZeroMQ if not compiled in
...at the same time, add some `TEST-REQUIRES: have-zeromq` which
unfortunately means that developers will usually want libzmq
installed on their system.
2024-12-11 18:08:43 +01:00
Arne Welzel
c6c6d88b4c btest/coverage: Avoid warnings in test-all-policy-cluster 2024-12-11 17:20:45 +01:00
Arne Welzel
0ad3210177 Broker::publish: Warn on using Broker::publish() when inactive
This is mostly for transitioning base scripts to Cluster::publish() and
avoid silent surprises why certain things don't work when using ZeroMQ.
2024-12-11 17:20:42 +01:00
Arne Welzel
f674fc2634 Merge remote-tracking branch 'origin/topic/timw/update-zeek-testing'
* origin/topic/timw/update-zeek-testing:
  Update zeek-testing and zeek-testing-cluster commit hashes
2024-12-11 13:19:33 +01:00
Arne Welzel
4ee2f9256b Merge remote-tracking branch 'origin/topic/awelzel/pluggable-cluster-backends-part4'
* origin/topic/awelzel/pluggable-cluster-backends-part4:
  Update ZAM BiF-tracking
  configure: Add --disable-cluster-backend-zeromq
  CMakeLists: Cluster backends output
  zeromq: Conditionally enable by default
  btest/generic: Add publish_hrw(), publish_rr() and logging tests
  generate-docs: Run on Ubuntu 24.04, add cppzmq
  docker: Add cppzmq/libzmq dependencies
  tsan_suppressions: Add called_from_lib: libzmq
  ci: Add cppzmq and libzmq to most platforms
  cluster/backend/zeromq: Add ZeroMQ based cluster backend
  cluster/backend/zeromq: Add cppzmq submodule
  cluster: Add Cluster::node_id(), allow redef of node_topic(), nodeid_topic()
  cluster: Move publish_hrw() and publish_rr() to cluster.bif
2024-12-11 09:52:41 +01:00
Tim Wojtulewicz
0701895132 Merge remote-tracking branch 'J-Gras/topic/jgras/add-unknown-proto-count'
* J-Gras/topic/jgras/add-unknown-proto-count:
  Add btest for unknown_protocols.log
  Add protocol_id count to unknown protocol record
2024-12-10 16:16:12 -07:00
Tim Wojtulewicz
f61ba4df29 Update zeek-testing and zeek-testing-cluster commit hashes 2024-12-10 16:13:00 -07:00
Arne Welzel
759281dabd Update ZAM BiF-tracking 2024-12-10 20:35:41 +01:00
Arne Welzel
d816bfb249 btest/generic: Add publish_hrw(), publish_rr() and logging tests
They currently use zeromq, but technically they should be valid for
any other backend, too, even broker.
2024-12-10 20:33:02 +01:00
Arne Welzel
35c79ab2e3 cluster/backend/zeromq: Add ZeroMQ based cluster backend
This is a cluster backend implementation using a central XPUB/XSUB proxy
that by default runs on the manager node. Logging is implemented leveraging
PUSH/PULL sockets between logger and other nodes, rather than going
through XPUB/XSUB.

The test-all-policy-cluster baseline changed: Previously, Broker::peer()
would be called from setup-connections.zeek, causing the IO loop to be
alive. With the ZeroMQ backend, the IO loop is only alive when
Cluster::init() is called, but that doesn't happen anymore.
2024-12-10 20:33:02 +01:00
Arne Welzel
210b54799e cluster: Move publish_hrw() and publish_rr() to cluster.bif
From this point on, Cluster::publish_hrw() and Cluster::publish_rr()
go through cluster/Backend.cc code.
2024-12-10 20:33:02 +01:00
Arne Welzel
7449b050b3 Update BiF-tracking, add get_current_packet_ts()
Also, run the ZAM-bif-tracking test in non-ZAM environments so
failures are caught immediately. There's nothing overly ZAM specific about
running this test.

I'm not sure I like the fact that any new contributor adding a BiF
will need to dig into this... it might be a bit intimidating.
2024-12-10 20:07:56 +01:00