Commit graph

1691 commits

Author SHA1 Message Date
Christian Kreibich
68fadd0464 Lower listen/connect retry intervals in Broker and the cluster framework to 1sec
The former defaults (30sec, 1min) can slow down cluster startup and recovery
considerably, and other systems have more aggressive intervals still.
2025-04-25 10:22:35 -07:00
Arne Welzel
86249db2a3 testing/btest/*test: Comment all @TEST lines 2025-04-17 16:30:23 +02:00
Arne Welzel
85b8c8866b testing/btest/*zeek: Comment all @TEST lines 2025-04-17 16:30:23 +02:00
Arne Welzel
ff58be2f36 ldap: Only switch into MS_KRB5 mode if responseToken exists
If the server doesn't include a responseToken within negTokenResp,
assume there won't be signing or sealing happening on the
connection. Don't switch into MS_KRB5 mode.

Closes #4275
2025-04-15 20:10:52 +02:00
Tim Wojtulewicz
cb1ef47a31 Add STORAGE_ prefixes for backends and serializers 2025-04-14 10:11:13 -07:00
Tim Wojtulewicz
88786a28a2 Add JSON storage serializer, use with existing backends/tests 2025-04-14 10:11:13 -07:00
Arne Welzel
2f6c31df89 btest/files/x509/files: Sort analyzers in baseline
This test is very sensitive to new enum values that perturb the final
order of the output. Listing the commits that modify that file in the
past clearly shows changes to the baseline happen when a new component
with an enum value is introduced.

    40f04d4ccf broker: Add WebSocketShim backend
    51836d08ae protocol: Add StreamEvent analyzer
    fdde1e9841 cluster/serializer: Add binary-serialization-format
    6fb73aa9da broker: Add shim plugin adding a backend component
    85ca59484b postgresql: Initial parser implementation
    efc2681152 WebSocket: Introduce new analyzer and log
    ...

That's just distraction. Point fix the test to sort the analyzers before
writing the final log line.

I've previously attempted this in a more complete fashion in PR #3365
(issue #3364), but seems this direct fix might just make more sense.
2025-04-10 15:02:44 +02:00
Tim Wojtulewicz
30d2642272 Merge remote-tracking branch 'origin/topic/timw/4281-unconditionally-forward-out-of-snap'
* origin/topic/timw/4281-unconditionally-forward-out-of-snap:
  Make SNAP analyzer use both OUI and protocol for forwarding
  Change packet analyzer identifiers to be 64-bit
2025-04-08 10:59:15 -07:00
Tim Wojtulewicz
32ae8f4eaa Make storage events take a tag for the backend instead of a string 2025-03-27 16:12:24 -07:00
Arne Welzel
c3c6ee5a2b telemetry: Run callbacks at collect time
Calling collect_metrics() from a script would not invoke metric
callbacks, resulting in most of the process metrics to be zero
when a Zeek process isn't scraped via Prometheus.

Fixes #4309
2025-03-26 12:07:27 +01:00
Tim Wojtulewicz
195b87b873 Make SNAP analyzer use both OUI and protocol for forwarding 2025-03-24 15:20:50 -07:00
Tim Wojtulewicz
855c530b64 Redis: Handle other errors from requests, fix KEY_EXISTS for put operations 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
3d7fcfb428 SQLite: handle existing keys when overwrite=F correctly 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
ba9cf1e4db Remove unnecessary type aliases from storage btests 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
d5ebaf476d Avoid thread-leak in scripts.base.frameworks.file-analysis.bifs.enable-disable btest
This btest uses the exit() BIF to shut down, which immediately calls
::exit() and kills Zeek without doing any shutdown. This will sometimes
leave the thread running the storage manager, which causes TSan to
complain about a thread leak. Switch to use the terminate() BIF instead
which cleanly shuts down all of Zeek.
2025-03-21 11:56:27 -07:00
Christian Kreibich
637baefd2c Merge branch 'topic/christian/is_valid_subnet'
* topic/christian/is_valid_subnet:
  Minor typo/markup fixes in NEWS.
  Add is_valid_subnet BiF
  Add btests for has_valid_octets() and normalize_mac()
2025-03-19 10:11:02 -07:00
Johanna Amann
7626039635 Merge remote-tracking branch 'origin/topic/johanna/ssh-server-banners-can-be-wild'
* origin/topic/johanna/ssh-server-banners-can-be-wild:
  SSH analyzer - tiny aesthetic fixes
  SSH: make banner parsing more robust
  SSH: split banner into client/server parts
2025-03-19 15:31:35 +00:00
Yacin Nadji
8b83c2995a Add is_valid_subnet BiF
Also includes consistency tweak for is_valid_ip() plus test cases.
2025-03-18 20:25:32 -07:00
Christian Kreibich
7ba9609992 Add btests for has_valid_octets() and normalize_mac()
These functions reside in base/utils/addrs.zeek and were not tested. We don't
use has_valid_octets() anywhere right now, normalize_mac() is used in the RADIUS
analyzer, and addr_to_uri() is used in the signatures/dpd.zeek tests.
2025-03-18 20:21:43 -07:00
Tim Wojtulewicz
d0741c8001 Allow sync methods to be called from when conditions, add related btest 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
b067a6e588 Redis: Fix sync erase, add btest for it 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
cc7b2dc890 Implement Storage::backend_opened and Storage::backend_lost events 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
a99a13dc4c SQLite: expand expiration test 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
cca1d4f988 Redis: Fix thread-contention issues with Expire(), add more tests 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
b81e876ec8 Change how redis-server is run during btests, removing redis.conf 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
9ed3e33f97 Completely rework return values from storage operations 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
e766af7322 Split sync/async handling into the BIF methods 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
c247de8ec3 Redis: Rework everything to only use async mode 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
a485b1d237 Make backend options a record, move actual options to be sub-records 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
28951dccf1 Split sync and async into separate script-land namespaces 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
42ad5bbf7d Add btest that uses a Redis backend in a cluster 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
f1a7376e0a Return generic result for get operations that includes error messages 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
4695060d75 Allow opening and closing backends to be async 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
ea87c773cd Redis: Support non-native expiration when reading traces 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
08bebaa426 Redis: Add btests for the redis backend 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
6289eb8e15 SQLite: Fix some issues with expiration, including in the btest 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
6bc5f70236 SQLite: Add additional btests, which also cover general storage functionality
- New erase/overwrite tests
- Change existing sqlite-basic test to use async
- Test passing bad keys to validate backend type checking
- New test for compound keys and values
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
ec49f5d550 SQLite: Handle automated expiration 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
9d1eef3fbc Add basic SQLite storage backend 2025-03-18 09:32:34 -07:00
Johanna Amann
6023c8b906 SSH: make banner parsing more robust
This change revamps SSH banner parsing.  The previous behavior was both
a bit too strict in some regards, and too permissive in other.

Specifically, clients are now required to send a line starting with
"SSH-" as the first line.  This is in line with the RFC, as well with
observed behavior. This also prevents the creation of `ssh.log` for
non-SSH traffic on port 22.

For the server side, we now accept text before the SSH banner. This
previously led to a protocol violation but is allowed by the spec.

New tests are added to cover these cases.
2025-03-18 16:19:33 +00:00
Johanna Amann
2daf692c95 Add two protocol mismatch testcases
These traces contain different protocols being used by originator/responder.

Traces from GH-4251
2025-03-04 15:38:20 +00:00
Johanna Amann
72ea7fc06c RDP: cookie is optional
Fixes GH-4237
2025-03-04 13:38:01 +00:00
Johanna Amann
0fa1ecce8f DPD: change policy script for service violation logging; add NEWS
This commit renames the `service_violation` column that can be added via
a policy script to `failed_service`. This expresses the intent of it
better - the column contains services that failed and were removed after
confirmation.

Furthermore, the script is fixed so it actually does this - before it
would sometimes add services to the list that were not actually removed.
In the course of this, the type of the column was changed from a vector
to an ordered set.

Due to the column rename, the policy script itself is also renamed.

Also adds a NEWS entry for the DPD changes.
2025-02-06 18:56:30 +00:00
Johanna Amann
e3493bc110 DPD changes - small script fixes and renames.
This addresses review feedback of GH-4200. No functional changes.
2025-02-05 13:55:43 +00:00
Johanna Amann
6324445d62 Merge remote-tracking branch 'origin/master' into topic/johanna/dpd-changes
This also includes some test baseline updates, due to recent QUIC
changes.

* origin/master: (39 commits)
  Update doc submodule [nomail] [skip ci]
  Bump cluster testsuite to pull in resilience to agent connection timing [skip ci]
  IPv6 support for detect-external-names and testcase
  Add  `skip_resp_host_port_pairs` option.
  util/init_random_seed: write_file implies deterministic
  external/subdir-btest.cfg: Set OPENSSL_ENABLE_SHA1_SIGNATURES=1
  btest/x509_verify: Drop OpenSSL 1.0 hack
  testing/btest: Use OPENSSL_ENABLE_SHA1_SIGNATURES
  Add ZAM baseline for new scripts.base.protocols.quic.analyzer-confirmations btest
  QUIC/decrypt_crypto: Rename all_data to data
  QUIC: Confirm before forwarding data to SSL
  QUIC: Parse all QUIC packets in a UDP datagram
  QUIC: Only slurp till packet end, not till &eod
  Remove unused SupervisedNode::InitCluster declaration
  Update doc submodule [nomail] [skip ci]
  Bump cluster testsuite to pull in updated Prometheus tests
  Make enc_part value from kerberos response available to scripts
  Management framework: move up addition of agent IPs into deployable cluster configs
  Support multiple instances per host addr in auto metrics generation
  When auto-generating metrics ports for worker nodes, get them more uniform across instances.
  ...
2025-02-05 09:31:16 +00:00
Johanna Amann
7a1a2c8d63 IPv6 support for detect-external-names and testcase
This commit builds on top of GH-4183 and adds IPv6 support for
policy/protocols/dns/detect-external-names.

Additionally it adds a test-case for this file testing it with mDNS
queries.
2025-02-04 17:34:43 +00:00
Arne Welzel
bb2e20d353 testing/btest: Use OPENSSL_ENABLE_SHA1_SIGNATURES
This reverts the call to update-crypto-policies in the Fedora 41 image
and instead sets OPENSSL_ENABLE_SHA1_SIGNATURES in the individual tests.
This allows RHEL 10 or Fedora 41 users to run the tests in question
without needing to fiddle with system settings.

Fixes #4035
2025-02-04 09:43:26 +01:00
Tim Wojtulewicz
c8eab363fb Merge remote-tracking branch 'philipp-tg/asrep_enc_part'
* philipp-tg/asrep_enc_part:
  Make enc_part value from kerberos response available to scripts
2025-02-03 18:41:53 -07:00
Arne Welzel
5d9d3921a6 QUIC: Confirm before forwarding data to SSL
Fixes #4201
2025-02-03 17:36:41 +01:00
Arne Welzel
6a14e64a17 QUIC: Parse all QUIC packets in a UDP datagram
A UDP datagram may contain multiple QUIC packets, but the parser so far
handled only the very first packet, ignoring any subsequent packets.

Fixes #4198
2025-02-03 17:36:37 +01:00