Commit graph

5238 commits

Author SHA1 Message Date
Tim Wojtulewicz
ba9cf1e4db Remove unnecessary type aliases from storage btests 2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
d5ebaf476d Avoid thread-leak in scripts.base.frameworks.file-analysis.bifs.enable-disable btest
This btest uses the exit() BIF to shut down, which immediately calls
::exit() and kills Zeek without doing any shutdown. This will sometimes
leave the thread running the storage manager, which causes TSan to
complain about a thread leak. Switch to use the terminate() BIF instead
which cleanly shuts down all of Zeek.
2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
e5b06367f7 Squash code from OperationResultCallback into ResultCallback 2025-03-21 11:44:39 -07:00
Arne Welzel
32d7cec549 testing/btest: Remove btest-bg-run sleep 1
Bump to the latest btest master version and set BTEST_BG_RUN_SLEEP
environment variable and to 0. This makes btest-bg-run return
immediately instead of delaying by 1 second.
2025-03-20 10:11:12 +01:00
Tim Wojtulewicz
38484b6dc6 Merge remote-tracking branch 'origin/topic/vern/CPP-maint.Mar25'
* origin/topic/vern/CPP-maint.Mar25:
  minor baseline updates for -O gen-C++
  -O gen-C++ support for pattern-to-pattern comparison
  -O gen-C++ support for pattern vector comparisons
  -O gen-C++ support for type expressions
2025-03-19 12:46:52 -07:00
Vern Paxson
fb43bc63cd minor baseline updates for -O gen-C++ 2025-03-19 12:46:15 -07:00
Christian Kreibich
637baefd2c Merge branch 'topic/christian/is_valid_subnet'
* topic/christian/is_valid_subnet:
  Minor typo/markup fixes in NEWS.
  Add is_valid_subnet BiF
  Add btests for has_valid_octets() and normalize_mac()
2025-03-19 10:11:02 -07:00
Johanna Amann
7626039635 Merge remote-tracking branch 'origin/topic/johanna/ssh-server-banners-can-be-wild'
* origin/topic/johanna/ssh-server-banners-can-be-wild:
  SSH analyzer - tiny aesthetic fixes
  SSH: make banner parsing more robust
  SSH: split banner into client/server parts
2025-03-19 15:31:35 +00:00
Yacin Nadji
8b83c2995a Add is_valid_subnet BiF
Also includes consistency tweak for is_valid_ip() plus test cases.
2025-03-18 20:25:32 -07:00
Christian Kreibich
7ba9609992 Add btests for has_valid_octets() and normalize_mac()
These functions reside in base/utils/addrs.zeek and were not tested. We don't
use has_valid_octets() anywhere right now, normalize_mac() is used in the RADIUS
analyzer, and addr_to_uri() is used in the signatures/dpd.zeek tests.
2025-03-18 20:21:43 -07:00
Tim Wojtulewicz
43faea880b Add analyzer registration from VLAN to VNTAG 2025-03-18 11:51:27 -07:00
Tim Wojtulewicz
c7015e8250 Split storage.bif file into events/sync/async, add more comments 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
d0741c8001 Allow sync methods to be called from when conditions, add related btest 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
b067a6e588 Redis: Fix sync erase, add btest for it 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
e6f1eea1b7 Remove default argument for callbacks, reorder function arguments 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
cc7b2dc890 Implement Storage::backend_opened and Storage::backend_lost events 2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
a99a13dc4c SQLite: expand expiration test 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
cca1d4f988 Redis: Fix thread-contention issues with Expire(), add more tests 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
b81e876ec8 Change how redis-server is run during btests, removing redis.conf 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
9ed3e33f97 Completely rework return values from storage operations 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
8ddda016ff Update some btests due to timing changes 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
e766af7322 Split sync/async handling into the BIF methods 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
c247de8ec3 Redis: Rework everything to only use async mode 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
a485b1d237 Make backend options a record, move actual options to be sub-records 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
28951dccf1 Split sync and async into separate script-land namespaces 2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
42ad5bbf7d Add btest that uses a Redis backend in a cluster 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
f1a7376e0a Return generic result for get operations that includes error messages 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
4695060d75 Allow opening and closing backends to be async 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
ea87c773cd Redis: Support non-native expiration when reading traces 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
08bebaa426 Redis: Add btests for the redis backend 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
6289eb8e15 SQLite: Fix some issues with expiration, including in the btest 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
6bc5f70236 SQLite: Add additional btests, which also cover general storage functionality
- New erase/overwrite tests
- Change existing sqlite-basic test to use async
- Test passing bad keys to validate backend type checking
- New test for compound keys and values
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
ec49f5d550 SQLite: Handle automated expiration 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
e95784db16 SQLite: Store/lookup prepared statements instead of recreating 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
9d1eef3fbc Add basic SQLite storage backend 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
7ad6a05f5b Add infrastructure for asynchronous storage operations 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
d07d27453a Add infrastructure for automated expiration of storage entries
This is used for backends that don't support expiration natively.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
8dee733a7d Change args to Storage::put to be a record
The number of args being passed to the put() methods was getting to be
fairly long, with more on the horizon. Changing to a record means simplifying
things a little bit.
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
69d940533d Pass key/value types for validation when opening backends 2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
2ea0f3e70a Lay out initial parts for the Storage framework
This includes a manager, component manager, BIF and script code, and
parts to support new storage backend plugins.
2025-03-18 09:32:34 -07:00
Johanna Amann
6023c8b906 SSH: make banner parsing more robust
This change revamps SSH banner parsing.  The previous behavior was both
a bit too strict in some regards, and too permissive in other.

Specifically, clients are now required to send a line starting with
"SSH-" as the first line.  This is in line with the RFC, as well with
observed behavior. This also prevents the creation of `ssh.log` for
non-SSH traffic on port 22.

For the server side, we now accept text before the SSH banner. This
previously led to a protocol violation but is allowed by the spec.

New tests are added to cover these cases.
2025-03-18 16:19:33 +00:00
Robin Sommer
3d25328204
Merge remote-tracking branch 'origin/topic/robin/gh-4250-vec-allocator'
* origin/topic/robin/gh-4250-vec-allocator:
  Spicy: Fix passing vectors to Zeek.
2025-03-18 09:08:50 +01:00
Arne Welzel
bb58148c64 Merge remote-tracking branch 'origin/topic/awelzel/4136-cluster-websocket-support'
* origin/topic/awelzel/4136-cluster-websocket-support:
  ci/opensuse-tumpleweed: Bust cache
  ci/macos/prepare: Install python@3 explicitly
  cluster/websocket: Implement WebSocket server
  cluster/websocket: Add IXWebsocket submodule
  ci/alpine: Install openssl package for testing
  ci: Install websockets from pip for all distros
  auxil/libunistd: Bump for ssize_t typedef
  auxil/broker: Bump to latest master version
  cluster/zeromq: Catch log_push.send() exception
  cluster/zeromq: Catch exceptions as const zmq::error_t&
  cluster/zeromq: No assert on inproc handling
  cluster/zeromq: Support configuring IO threads for proxy thread
  cluster/zeromq: Move variable lookups from DoInit() to DoInitPostScript()
  cluster/zeromq: Handle EINTR at shutdown
  cluster/zeromq: Queue one message at a time
  cluster/Backend: Queue a single message only
  cluster/zeromq: Adapt for OnLoopProcess changes
  cluster/ThreadedBackend: Switch to OnLoopProcess
  cluster/OnLoop: Introduce helper template class
  serializer/broker: Expose to_broker_event() and to_zeek_event()
2025-03-11 10:51:09 +01:00
Arne Welzel
e2b65acc6d btest/javascript: Add file_sniff() and file_state_remove() test
Using file_state_remove() or file_sniff() would previously crash with the
spicy-zip file analyzer. We don't have spicy-zip here, so it's not a
proper reproducer, but still makes sense to test these events.
2025-03-10 19:15:06 +01:00
Arne Welzel
6032741868 cluster/websocket: Implement WebSocket server 2025-03-10 17:07:30 +01:00
Evan Typanski
4237239325 Fix printing type constants like double
Fixes #4268
2025-03-07 11:54:04 -05:00
Tim Wojtulewicz
81f7e77801 Merge remote-tracking branch 'origin/topic/timw/4266-tunnel-ip-proto'
* origin/topic/timw/4266-tunnel-ip-proto:
  Set ip_proto when creating new tunnel encapsulations
2025-03-06 16:42:03 -07:00
Tim Wojtulewicz
479c4be08b Set ip_proto when creating new tunnel encapsulations 2025-03-06 09:25:03 -07:00
Johanna Amann
6fcd02b3a4 Merge remote-tracking branch 'origin/topic/johanna/gh-4251-discussion'
* origin/topic/johanna/gh-4251-discussion:
  Add two protocol mismatch testcases
  Remove violating t panalyzer from services field again
2025-03-06 13:12:51 +00:00
Arne Welzel
3b7bbb2651 ZAM: bif-tracking updates for blocking_lookup_hostname() 2025-03-05 19:23:08 +01:00