Commit graph

3724 commits

Author SHA1 Message Date
Tim Wojtulewicz
cb1ef47a31 Add STORAGE_ prefixes for backends and serializers 2025-04-14 10:11:13 -07:00
Tim Wojtulewicz
e545fe8256 Ground work for pluggable storage serializers 2025-04-14 10:02:35 -07:00
Robin Sommer
9db73415cd
Spicy: Document lifetime semantics of Zeek analyzers created from Spicy.
Closes #3522.
2025-04-10 12:17:05 +02:00
Arne Welzel
6bc36e8cf8 broker/main: Adapt enum values to agree with comm.bif
Logic to detect this error already existed, but due to enum identifiers
not having a value set, it never triggered before.

Should probably backport this one.
2025-04-04 15:36:42 +02:00
Robin Sommer
94ddd7f411
Spicy: Port over to Spicy's new tuple representation.
Includes a fix for supporting CMake 4.0.
2025-04-02 14:14:26 +02:00
Tim Wojtulewicz
55e458c5f7 Add comment annotation to disable copying redef value into docs 2025-04-01 10:23:55 -07:00
Arne Welzel
14697ea6ba Merge remote-tracking branch 'origin/topic/neverlord/broker-logging'
* origin/topic/neverlord/broker-logging:
  Integrate review feedback
  Hook into Broker logs via its new API
2025-03-31 18:53:43 +02:00
Christian Kreibich
98c203b8cb Add "U" to QUIC history docstrings and expand version string docs
Looks like we overlooked documenting "U" in zeek/zeek#3526 .
2025-03-27 13:29:40 -07:00
Christian Kreibich
2199cb1ddd Remove "experimental" from the QUIC history field's comment string [skip ci]
We're unlikely to fundamentally change (or remove) this field at this point, and
some users wondered whether we might do so, given the labeling.
2025-03-26 14:03:52 -07:00
Arne Welzel
2963c49f27 cluster/zeromq: Fix node_topic() and nodeid_topic()
Due to prefix matching, worker-1's node_topic() also matched worker-10,
worker-11, etc. Suffix the node topic with a `.`. The original implementation
came from NATS, where subjects are separated by `.`.

Adapt nodeid_topic() for consistency.
2025-03-24 18:36:26 +01:00
Tim Wojtulewicz
43faea880b Add analyzer registration from VLAN to VNTAG 2025-03-18 11:51:27 -07:00
Tim Wojtulewicz
c7015e8250 Split storage.bif file into events/sync/async, add more comments 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
f40947f6ac Update comments in script files, run zeek-format on all of them 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
a40db844eb Redis: Handle disconnection correctly via callback 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
c7503654e8 Add IN_PROGRESS return code, handle for async backends 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
9ed3e33f97 Completely rework return values from storage operations 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
c247de8ec3 Redis: Rework everything to only use async mode 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
40f60f26b3 Run expiration on a separate thread 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
a485b1d237 Make backend options a record, move actual options to be sub-records 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
28951dccf1 Split sync and async into separate script-land namespaces 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
f1a7376e0a Return generic result for get operations that includes error messages 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
4695060d75 Allow opening and closing backends to be async 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
52d94b781a Redis: Force storage sync mode when reading pcaps, default to async mode 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
31e146b16d Redis: Add new backend 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
3e8ff836aa SQLite: Add tuning options to configuration 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
9d1eef3fbc Add basic SQLite storage backend 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
7ad6a05f5b Add infrastructure for asynchronous storage operations 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
d07d27453a Add infrastructure for automated expiration of storage entries
This is used for backends that don't support expiration natively.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
8dee733a7d Change args to Storage::put to be a record
The number of args being passed to the put() methods was getting to be
fairly long, with more on the horizon. Changing to a record means simplifying
things a little bit.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
69d940533d Pass key/value types for validation when opening backends 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
2ea0f3e70a Lay out initial parts for the Storage framework
This includes a manager, component manager, BIF and script code, and
parts to support new storage backend plugins.
2025-03-18 09:32:34 -07:00
Arne Welzel
cc0c48423d cluster/backends/zeromq: Fix rst link in docs 2025-03-12 10:11:25 +01:00
Arne Welzel
6032741868 cluster/websocket: Implement WebSocket server 2025-03-10 17:07:30 +01:00
Arne Welzel
aad512c616 cluster/zeromq: Support configuring IO threads for proxy thread 2025-03-10 17:07:30 +01:00
Arne Welzel
ba7b605a97 cluster/zeromq: Move variable lookups from DoInit() to DoInitPostScript() 2025-03-10 17:07:30 +01:00
Johanna Amann
b8c135d7cb Remove violating analyzer from services field again
This reverts some of the recent DPD changes; specifically violations
trigger removal from the services field, again, by default.

Discussion in GH-4521
2025-03-04 15:10:49 +00:00
Benjamin Bannier
5d44073b94 Bump pre-commit hooks 2025-03-04 08:14:26 +01:00
Arne Welzel
776c003033 PacketAnalyzer::Geneve: Add get_options()
Allow to extract Geneve options on-demand, for example during a
new_connection() event.
2025-02-22 12:19:42 -08:00
Dominik Charousset
20b3eca257 Integrate review feedback 2025-02-15 16:37:24 +01:00
Dominik Charousset
30615f425e Hook into Broker logs via its new API
The new Broker API allows us to provide a custom logger to Broker that
pulls previously unattainable context information out of Broker to put
them into broker.log for users of Zeek.

Since Broker log events happen asynchronously, we cache them in a queue
and use a flare to notify Zeek of activity. Furthermore, the Broker
manager now implements the `ProcessFd` function to avoid unnecessary
polling of the new log queue. As a side effect, data stores are polled
less as well.
2025-02-08 16:28:02 +01:00
Johanna Amann
fc233fd8d0 Merge remote-tracking branch 'origin/topic/johanna/dpd-changes'
* origin/topic/johanna/dpd-changes:
  DPD: failed services logging alignment
  DPD: update test baselines; change options for external tests.
  DPD: change policy script for service violation logging; add NEWS
  DPD changes - small script fixes and renames.
  Update public and private test suite for DPD changes.
  Allow to track service violations in conn.log.
  Make conn.log service field ordered
  DPD: change handling of pre-confirmation violations, remove max_violations
  DPD: log analyzers that have confirmed
  IRC analyzer - make protocol confirmation more robust.
2025-02-07 07:35:30 +00:00
Johanna Amann
66d07c31fa DPD: failed services logging alignment
There were some special cases in which the failed-service-logging policy
script might log a service being removed that was not removed due to an
analyzer violation. This change should fix these cases.
2025-02-06 18:56:30 +00:00
Johanna Amann
0fa1ecce8f DPD: change policy script for service violation logging; add NEWS
This commit renames the `service_violation` column that can be added via
a policy script to `failed_service`. This expresses the intent of it
better - the column contains services that failed and were removed after
confirmation.

Furthermore, the script is fixed so it actually does this - before it
would sometimes add services to the list that were not actually removed.
In the course of this, the type of the column was changed from a vector
to an ordered set.

Due to the column rename, the policy script itself is also renamed.

Also adds a NEWS entry for the DPD changes.
2025-02-06 18:56:30 +00:00
Johanna Amann
c402c28f7e Merge remote-tracking branch 'origin/topic/johanna/sslindentation'
* origin/topic/johanna/sslindentation:
  SSL main.zeek - fix indentation
2025-02-06 17:00:40 +00:00
Johanna Amann
3caa49023f SSL main.zeek - fix indentation
No functional changes
2025-02-06 16:15:58 +00:00
Johanna Amann
e3493bc110 DPD changes - small script fixes and renames.
This addresses review feedback of GH-4200. No functional changes.
2025-02-05 13:55:43 +00:00
Arne Welzel
fa22f91ca4 cluster/zeromq: Fix XSUB threading issues
It is not safe to use the same socket from different threads, but the
current code used the xsub socket directly from the main thread (to setup
subscriptions) and from the internal thread for polling and reading.

Leverage the PAIR socket already in use for forwarding publish operations
to the internal thread also for subscribe and unsubscribe.

The failure mode is/was a bit annoying. Essentially, closing of the
context would hang indefinitely in zmq_ctx_term().
2025-02-05 10:39:56 +01:00
Arne Welzel
769044e8e1 cluster/Backend: Pass node_id via Init() 2025-02-05 10:39:56 +01:00
Johanna Amann
6324445d62 Merge remote-tracking branch 'origin/master' into topic/johanna/dpd-changes
This also includes some test baseline updates, due to recent QUIC
changes.

* origin/master: (39 commits)
  Update doc submodule [nomail] [skip ci]
  Bump cluster testsuite to pull in resilience to agent connection timing [skip ci]
  IPv6 support for detect-external-names and testcase
  Add  `skip_resp_host_port_pairs` option.
  util/init_random_seed: write_file implies deterministic
  external/subdir-btest.cfg: Set OPENSSL_ENABLE_SHA1_SIGNATURES=1
  btest/x509_verify: Drop OpenSSL 1.0 hack
  testing/btest: Use OPENSSL_ENABLE_SHA1_SIGNATURES
  Add ZAM baseline for new scripts.base.protocols.quic.analyzer-confirmations btest
  QUIC/decrypt_crypto: Rename all_data to data
  QUIC: Confirm before forwarding data to SSL
  QUIC: Parse all QUIC packets in a UDP datagram
  QUIC: Only slurp till packet end, not till &eod
  Remove unused SupervisedNode::InitCluster declaration
  Update doc submodule [nomail] [skip ci]
  Bump cluster testsuite to pull in updated Prometheus tests
  Make enc_part value from kerberos response available to scripts
  Management framework: move up addition of agent IPs into deployable cluster configs
  Support multiple instances per host addr in auto metrics generation
  When auto-generating metrics ports for worker nodes, get them more uniform across instances.
  ...
2025-02-05 09:31:16 +00:00
Johanna Amann
7a1a2c8d63 IPv6 support for detect-external-names and testcase
This commit builds on top of GH-4183 and adds IPv6 support for
policy/protocols/dns/detect-external-names.

Additionally it adds a test-case for this file testing it with mDNS
queries.
2025-02-04 17:34:43 +00:00