Commit graph

19036 commits

Author SHA1 Message Date
Arne Welzel
41f04eda72 Merge remote-tracking branch 'origin/topic/awelzel/intel-indicator-hooks'
* origin/topic/awelzel/intel-indicator-hooks:
  intel/seen/manage-event-groups: Policy script for toggling intel event groups
  intel: Add indicator_inserted and indicator_removed hooks
2025-06-02 09:52:07 +02:00
Arne Welzel
0619fe2f4f intel/seen/manage-event-groups: Policy script for toggling intel event groups
Co-authored-by: Mohan Dhawan <mohan@corelight.com>
2025-06-02 09:51:14 +02:00
Arne Welzel
7eb849ddf4 intel: Add indicator_inserted and indicator_removed hooks
This change adds two new hooks to the Intel framework that can be used
to intercept added and removed indicators and their type.

These hooks are fairly low-level. One immediate use-case is to count the
number of indicators loaded per Intel::Type and enable and disable the
corresponding event groups of the intel/seen scripts.

I attempted to gauge the overhead and while it's definitely there, loading
a file with ~500k DOMAIN entries takes somewhere around ~0.5 seconds hooks
when populated via the min_data_store store mechanism. While that
doesn't sound great, it actually takes the manager on my system 2.5
seconds to serialize and Cluster::publish() the min_data_store alone
and its doing that serially for every active worker. Mostly to say that
the bigger overhead in that area on the manager doing redundant work
per worker.

Co-authored-by: Mohan Dhawan <mohan@corelight.com>
2025-06-02 09:50:48 +02:00
Benjamin Bannier
1760d99c49 Prefer std::move over copy 2025-06-02 08:45:32 +02:00
Vern Paxson
614eb8d343 minor BTest maintenance updates for -O gen-C++ 2025-05-31 12:52:44 -07:00
Vern Paxson
9117ccab12 fix for more robustly finding BTests to assess for -O gen-C++ 2025-05-31 12:50:14 -07:00
Vern Paxson
e165e64fa5 fix for -O gen-C++ dealing with type constants of unnamed compound types 2025-05-31 12:49:37 -07:00
zeek-bot
224519c11a Update doc submodule [nomail] [skip ci] 2025-05-31 00:26:58 +00:00
Tim Wojtulewicz
3282bbc429 Merge remote-tracking branch 'origin/topic/vern/ZAM-maint.May25'
* origin/topic/vern/ZAM-maint.May25:
  fix for crash when interpreting transformed ASTs that include multi-field record assignments/additions
  Remove unused ZAM compiler method
2025-05-30 13:07:01 -07:00
Tim Wojtulewicz
70bc0d9deb Merge remote-tracking branch 'origin/topic/timw/cleanup-cmake-summary-output'
* origin/topic/timw/cleanup-cmake-summary-output:
  Add utility methods to make CMake summary output nicer
2025-05-30 12:16:35 -07:00
Tim Wojtulewicz
e93242726b Add utility methods to make CMake summary output nicer 2025-05-30 11:57:43 -07:00
Tim Wojtulewicz
dc5dd8be45 Merge remote-tracking branch 'origin/topic/timw/new-ci-pr-labels'
* origin/topic/timw/new-ci-pr-labels:
  CI: Add PR label for skipping all CI jobs
  CI: Add PR label for running cluster tests
2025-05-30 10:29:37 -07:00
Tim Wojtulewicz
bc4cf14237 CI: Add PR label for skipping all CI jobs 2025-05-30 10:29:02 -07:00
Tim Wojtulewicz
e9544386fe CI: Add PR label for running cluster tests 2025-05-30 10:27:52 -07:00
Vern Paxson
dc68a62a1e fix for crash when interpreting transformed ASTs that include multi-field record assignments/additions 2025-05-30 09:44:26 -07:00
Vern Paxson
ba0b7492a7 Remove unused ZAM compiler method 2025-05-30 09:38:42 -07:00
Tim Wojtulewicz
9c290df47f Merge remote-tracking branch 'origin/topic/timw/ci-clang-tidy'
* origin/topic/timw/ci-clang-tidy:
  CI: Add new task to run clang-tidy as part of nightly builds
  CI: Update to clang 19 on ubuntu 24.04, add clang-tidy package
2025-05-30 08:39:36 -07:00
Tim Wojtulewicz
bf9813a7c6 CI: Add new task to run clang-tidy as part of nightly builds 2025-05-30 08:39:14 -07:00
Tim Wojtulewicz
dbd787a81f CI: Update to clang 19 on ubuntu 24.04, add clang-tidy package 2025-05-30 08:39:14 -07:00
Arne Welzel
f4cd92e24a Merge remote-tracking branch 'origin/topic/awelzel/4494-ts-millis-signed'
* origin/topic/awelzel/4494-ts-millis-signed:
  logging/ascii/json: Make TS_MILLIS signed, add TS_MILLIS_UNSIGNED
2025-05-30 17:24:17 +02:00
Arne Welzel
93813a5079 logging/ascii/json: Make TS_MILLIS signed, add TS_MILLIS_UNSIGNED
It seems TS_MILLIS is specifically for Elasticsearch and starting with
Elasticsearch 8.2 epoch_millis does (again?) support negative epoch_millis,
so make Zeek produce that by default.

If this breaks a given deployment, they can switch Zeek back to TS_MILLIS_UNSIGNED.

https://discuss.elastic.co/t/migration-from-es-6-8-to-7-17-issues-with-negative-date-epoch-timestamp/335259
https://github.com/elastic/elasticsearch/pull/80208

Thanks for @timo-mue for reporting!

Closes #4494
2025-05-30 17:23:29 +02:00
Tim Wojtulewicz
c387ec87be Merge remote-tracking branch 'origin/topic/timw/clang-tidy-performance-fixes'
* origin/topic/timw/clang-tidy-performance-fixes:
  Add move operations for LogWriteHeader
  Add missing setting of type in session::Key move operations
  Update .clang-tidy to have performance-* enabled with some exclusions
  Fix clang-tidy performance-inefficient-string-concatenation warnings
  Fix clang-tidy performance-unnecessary-copy-initialization warnings
  Fix clang-tidy performance-move-const-argument warnings (not move assignable/copyable)
  Fix clang-tidy performance-move-const-argument warnings (passing move to const argument)
  Fix clang-tidy performance-move-const-argument warnings (moving trivially copyable)
  Fix clang-tidy performance-move-const-argument warnings (moving const variables)
  Fix clang-tidy performance-inefficient-vector-operation warnings
  Fix clang-tidy performance-for-range-copy warnings
  Fix clang-tidy performance-faster-string-find warnings
  Fix clang-tidy performance-enum-size warnings
  Fix clang-tidy performance-avoid-endl warnings
2025-05-30 08:13:19 -07:00
Tim Wojtulewicz
6eb49a10cc Add move operations for LogWriteHeader 2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
df852255c6 Add missing setting of type in session::Key move operations 2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
db69773d23 Update .clang-tidy to have performance-* enabled with some exclusions 2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
c609d5c90a Fix clang-tidy performance-inefficient-string-concatenation warnings 2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
cb8c35748a Fix clang-tidy performance-unnecessary-copy-initialization warnings 2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
909413838c Fix clang-tidy performance-move-const-argument warnings (not move assignable/copyable) 2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
57c10a6ace Fix clang-tidy performance-move-const-argument warnings (passing move to const argument) 2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
ad4694f529 Fix clang-tidy performance-move-const-argument warnings (moving trivially copyable) 2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
144a3dee3a Fix clang-tidy performance-move-const-argument warnings (moving const variables) 2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
6196950567 Fix clang-tidy performance-inefficient-vector-operation warnings 2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
b8e28abb97 Fix clang-tidy performance-for-range-copy warnings 2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
178d7f4cd0 Fix clang-tidy performance-faster-string-find warnings 2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
f4c47d0357 Fix clang-tidy performance-enum-size warnings 2025-05-30 08:12:29 -07:00
Tim Wojtulewicz
11a1d8d506 Fix clang-tidy performance-avoid-endl warnings 2025-05-30 08:02:55 -07:00
Arne Welzel
10bfb34a8a Update doc submodule [nomail] [skip ci]
Fix botched last-minute cleanup.
2025-05-30 16:49:10 +02:00
Arne Welzel
1d241fabf4 Merge remote-tracking branch 'origin/topic/awelzel/zeekygen-field-directive'
* origin/topic/awelzel/zeekygen-field-directive:
  Bump doc submodule for ext/zeek.py modifications
  RecordType:DescribeReST: Render RecordType using zeek:field directive
2025-05-30 16:44:42 +02:00
Tim Wojtulewicz
08f335b5f6 Update broker submodule [nomail] 2025-05-30 07:30:41 -07:00
Arne Welzel
f16ebd34b3 Merge remote-tracking branch 'origin/topic/awelzel/4474-cluster-websocket-ipv6'
* origin/topic/awelzel/4474-cluster-websocket-ipv6:
  IXWebsocket: Bump to version with memset() sock addr fix
  cluster/websocket: Deprecate $listen_host, introduce $listen_addr
  cluster/websocket-ixwebsocket: Determine proper address_family
2025-05-30 11:47:55 +02:00
Arne Welzel
66fef5a9bd IXWebsocket: Bump to version with memset() sock addr fix 2025-05-30 11:02:41 +02:00
Arne Welzel
544d571089 cluster/websocket: Deprecate $listen_host, introduce $listen_addr
This only changes the script-layer API, but keeps the std::string host
in the C++ layer's ServerOptions. Mostly because the ixwebsocket library
takes host as std::string. Also, maybe at  some point we'd want to
support something scheme-based like unix:///var/run/zeek.sock and placing
that in a string could not be totally wrong.

Add tests for IPV6, too.
2025-05-30 11:02:41 +02:00
Arne Welzel
8b029d0050 cluster/websocket-ixwebsocket: Determine proper address_family
Closes #4474
2025-05-30 11:01:31 +02:00
Tim Wojtulewicz
372986f052 Update docs submodue [nomail] [skip ci] 2025-05-28 11:44:09 -07:00
Tim Wojtulewicz
2d7aad2c81 Merge remote-tracking branch 'origin/topic/timw/remove-with-bind'
* origin/topic/timw/remove-with-bind:
  Remove obsolete --with-bind configure flag
2025-05-28 08:29:50 -07:00
Arne Welzel
871e81dd6a Bump doc submodule for ext/zeek.py modifications 2025-05-28 16:18:44 +02:00
Arne Welzel
71fb301e3d RecordType:DescribeReST: Render RecordType using zeek:field directive
This is for zeek/zeek-docs#324.
2025-05-28 15:59:50 +02:00
Robin Sommer
e494fb5d19
Bump Spicy to pull in fix.
Includes a new regression test.

Closes #4501.
2025-05-28 15:50:47 +02:00
Tim Wojtulewicz
9db9861f75 Remove obsolete --with-bind configure flag 2025-05-27 17:55:49 -07:00
zeek-bot
9ffc87a90e Update doc submodule [nomail] [skip ci] 2025-05-28 00:38:27 +00:00