Arne Welzel
93813a5079
logging/ascii/json: Make TS_MILLIS signed, add TS_MILLIS_UNSIGNED
...
It seems TS_MILLIS is specifically for Elasticsearch and starting with
Elasticsearch 8.2 epoch_millis does (again?) support negative epoch_millis,
so make Zeek produce that by default.
If this breaks a given deployment, they can switch Zeek back to TS_MILLIS_UNSIGNED.
https://discuss.elastic.co/t/migration-from-es-6-8-to-7-17-issues-with-negative-date-epoch-timestamp/335259
https://github.com/elastic/elasticsearch/pull/80208
Thanks for @timo-mue for reporting!
Closes #4494
2025-05-30 17:23:29 +02:00
Tim Wojtulewicz
0fb4548ff0
Redis: return proper error if connection fails
2025-05-23 12:13:13 -07:00
Tim Wojtulewicz
25f144381c
SQLite: Fix typo in variable name causing pragmas not to retry on busy
2025-05-22 10:23:17 -07:00
Tim Wojtulewicz
25dd1a2702
Disable sqlite-cluster btest
...
This test is being flaky on some platforms and still having problems
with executing pragmas at startup. Disable it for now until it can be
fixed.
2025-05-21 15:42:29 -07:00
Tim Wojtulewicz
e91421a8de
Prefix sqlite-based btests with sqlite- to match redis tests
2025-05-21 09:38:27 -07:00
Tim Wojtulewicz
41bddae59f
Add sqlite cluster storage btest
2025-05-21 09:38:27 -07:00
Arne Welzel
00eabb6cbb
btest remaining: Use generic cluster-layout.zeek
2025-05-20 20:30:01 +02:00
Arne Welzel
9365f71965
btest/frameworks/logging: Use generic cluster-layout.zeek
2025-05-20 20:30:01 +02:00
Arne Welzel
d7b5955e5e
btest/frameworks/notice: Use generic cluster-layout.zeek
2025-05-20 20:30:01 +02:00
Arne Welzel
00a12a4cc5
btest/frameworks/intel: Use generic cluster-layout.zeek
2025-05-20 20:30:01 +02:00
Arne Welzel
4dec63936e
btest/frameworks/sumstats: Use generic cluster-layout.zeek
2025-05-20 20:30:01 +02:00
Arne Welzel
0a06a77c69
btest/frameworks/cluster: Use generic cluster-layout.zeek
2025-05-20 20:30:01 +02:00
Arne Welzel
e114b0e371
btest/frameworks/config: Use generic cluster-layout.zeek
2025-05-20 20:30:01 +02:00
Tim Wojtulewicz
6f8924596f
Merge remote-tracking branch 'origin/topic/johanna/fix-failed-service-logging'
...
* origin/topic/johanna/fix-failed-service-logging:
Fix policy/protocols/conn/failed-service-logging.zeek
2025-05-07 10:29:54 -07:00
Tim Wojtulewicz
58ee8d3c5c
Add Storage::is_connected BIF
2025-05-07 08:13:16 -07:00
Johanna Amann
f293d5a852
Fix policy/protocols/conn/failed-service-logging.zeek
...
In GH-4422 it was pointed out that the protocols/conn/failed-service-logging.zeek
policy script only works when
`DPD::track_removed_services_in_connection=T` is set.
This was caused by a logic error in the script. This commit fixes this
logic error and introduces an additional test that checks that
failed-service-logging works even when the option is not set to true.
2025-05-06 13:37:12 +01:00
Arne Welzel
0e327a0c12
testing/btest: Fix double commented @TEST- lines
...
sed -i 's/^# # @/# @/g'
2025-05-06 14:06:29 +02:00
Arne Welzel
85b8c8866b
testing/btest/*zeek: Comment all @TEST lines
2025-04-17 16:30:23 +02:00
Tim Wojtulewicz
cb1ef47a31
Add STORAGE_ prefixes for backends and serializers
2025-04-14 10:11:13 -07:00
Tim Wojtulewicz
88786a28a2
Add JSON storage serializer, use with existing backends/tests
2025-04-14 10:11:13 -07:00
Tim Wojtulewicz
32ae8f4eaa
Make storage events take a tag for the backend instead of a string
2025-03-27 16:12:24 -07:00
Arne Welzel
c3c6ee5a2b
telemetry: Run callbacks at collect time
...
Calling collect_metrics() from a script would not invoke metric
callbacks, resulting in most of the process metrics to be zero
when a Zeek process isn't scraped via Prometheus.
Fixes #4309
2025-03-26 12:07:27 +01:00
Tim Wojtulewicz
855c530b64
Redis: Handle other errors from requests, fix KEY_EXISTS for put operations
2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
3d7fcfb428
SQLite: handle existing keys when overwrite=F correctly
2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
ba9cf1e4db
Remove unnecessary type aliases from storage btests
2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
d5ebaf476d
Avoid thread-leak in scripts.base.frameworks.file-analysis.bifs.enable-disable btest
...
This btest uses the exit() BIF to shut down, which immediately calls
::exit() and kills Zeek without doing any shutdown. This will sometimes
leave the thread running the storage manager, which causes TSan to
complain about a thread leak. Switch to use the terminate() BIF instead
which cleanly shuts down all of Zeek.
2025-03-21 11:56:27 -07:00
Tim Wojtulewicz
d0741c8001
Allow sync methods to be called from when conditions, add related btest
2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
b067a6e588
Redis: Fix sync erase, add btest for it
2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
cc7b2dc890
Implement Storage::backend_opened and Storage::backend_lost events
2025-03-18 10:20:34 -07:00
Tim Wojtulewicz
a99a13dc4c
SQLite: expand expiration test
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
cca1d4f988
Redis: Fix thread-contention issues with Expire(), add more tests
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
b81e876ec8
Change how redis-server is run during btests, removing redis.conf
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
9ed3e33f97
Completely rework return values from storage operations
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
e766af7322
Split sync/async handling into the BIF methods
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
c247de8ec3
Redis: Rework everything to only use async mode
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
a485b1d237
Make backend options a record, move actual options to be sub-records
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
28951dccf1
Split sync and async into separate script-land namespaces
2025-03-18 10:20:33 -07:00
Tim Wojtulewicz
42ad5bbf7d
Add btest that uses a Redis backend in a cluster
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
f1a7376e0a
Return generic result for get operations that includes error messages
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
4695060d75
Allow opening and closing backends to be async
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
ea87c773cd
Redis: Support non-native expiration when reading traces
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
08bebaa426
Redis: Add btests for the redis backend
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
6289eb8e15
SQLite: Fix some issues with expiration, including in the btest
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
6bc5f70236
SQLite: Add additional btests, which also cover general storage functionality
...
- New erase/overwrite tests
- Change existing sqlite-basic test to use async
- Test passing bad keys to validate backend type checking
- New test for compound keys and values
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
ec49f5d550
SQLite: Handle automated expiration
2025-03-18 09:32:34 -07:00
Tim Wojtulewicz
9d1eef3fbc
Add basic SQLite storage backend
2025-03-18 09:32:34 -07:00
Johanna Amann
2daf692c95
Add two protocol mismatch testcases
...
These traces contain different protocols being used by originator/responder.
Traces from GH-4251
2025-03-04 15:38:20 +00:00
Johanna Amann
0fa1ecce8f
DPD: change policy script for service violation logging; add NEWS
...
This commit renames the `service_violation` column that can be added via
a policy script to `failed_service`. This expresses the intent of it
better - the column contains services that failed and were removed after
confirmation.
Furthermore, the script is fixed so it actually does this - before it
would sometimes add services to the list that were not actually removed.
In the course of this, the type of the column was changed from a vector
to an ordered set.
Due to the column rename, the policy script itself is also renamed.
Also adds a NEWS entry for the DPD changes.
2025-02-06 18:56:30 +00:00
Johanna Amann
e3493bc110
DPD changes - small script fixes and renames.
...
This addresses review feedback of GH-4200. No functional changes.
2025-02-05 13:55:43 +00:00
Johanna Amann
2f712c3c24
Allow to track service violations in conn.log.
...
This introduces ian options, DPD::track_removed_services_in_connection.
It adds failed services to the services column, prefixed with a
"-".
Alternatively, this commit also adds
policy/protocols/conn/failed-services.zeek, which provides the same
information in a new column in conn.log.
2025-01-30 16:59:44 +00:00
