Commit graph

17477 commits

Author SHA1 Message Date
Johanna Amann
f36be3dafc Spicy-TLS: address review feedback re convert-functions
This switches convert-functions from being called in the evt file to
being called directly in the spicy file.

See GH-3765 for details.
2024-07-17 12:44:14 +01:00
Tim Wojtulewicz
9ba7c2ddaf Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy'
* origin/topic/bbannier/bump-spicy:
  Bump auxil/spicy to latest development snapshot
2024-07-16 10:16:15 -07:00
Theo Buehler
3ea67a3109 Use accessor to reach into X509_ALGOR
Despite already having an accessor, X509_ALGOR wasn't made opaque
during OpenSSL 1.1.0 development. It would be nice if this could be
fixed at some point, so avoid reaching into that struct by using the
accessor
2024-07-16 18:06:10 +02:00
Benjamin Bannier
41db05238b Bump auxil/spicy to latest development snapshot
This patch bump Spicy to the latest development snapshot. This
introduces a backwards-incompatible change in that it removes support
for a never officially supported syntax to specify unit fields (so I
would argue: not strictly a breaking change).
2024-07-16 08:17:31 +02:00
Tim Wojtulewicz
cb56837f07 Merge remote-tracking branch 'origin/topic/timw/fetchcontent_mkdir'
* origin/topic/timw/fetchcontent_mkdir:
  CI: Set FETCH_CONTENT_FULLY_DISCONNECTED flag for configure
  Update broker and cmake submodules [nomail]
2024-07-12 16:02:37 -07:00
Tim Wojtulewicz
4757536d7d CI: Set FETCH_CONTENT_FULLY_DISCONNECTED flag for configure 2024-07-12 15:14:31 -07:00
Tim Wojtulewicz
a46be1ea29 Update broker and cmake submodules [nomail] 2024-07-12 15:14:31 -07:00
Tim Wojtulewicz
f3bcf1a55d Fix a broken merge
I merged an old version of the branch on accident and then merged the right
one over top of it, but git ended up including both versions. This fixes
that mistake.
2024-07-12 10:00:13 -07:00
Tim Wojtulewicz
b935d2f59a Merge remote-tracking branch 'origin/topic/bbannier/lib-spicy-hooks'
* origin/topic/bbannier/lib-spicy-hooks:
  Do not emit hook files for builtin modules
2024-07-12 09:51:55 -07:00
Tim Wojtulewicz
7a38cee81f Merge remote-tracking branch 'origin/topic/bbannier/lib-spicy-hooks'
* origin/topic/bbannier/lib-spicy-hooks:
  Do not emit hook files for builtin modules
2024-07-12 09:48:12 -07:00
Tim Wojtulewicz
e4716b6c91 Merge remote-tracking branch 'origin/topic/timw/grealpath-make-dist-warning'
* origin/topic/timw/grealpath-make-dist-warning:
  Fix warning about grealpath when running 'make dist' on Linux
2024-07-12 09:46:51 -07:00
Benjamin Bannier
24d3454d61 Do not emit hook files for builtin modules
We would previously emit a C++ file with hooks for at least the builtin
`spicy` module even though that module like any other builtin module
never contains implementations of hooks for types in user code.

This patch prevents modules with skipped implementations (such as our
builtin modules) from being added to the compilation which prevents
generating their hook files.
2024-07-12 09:06:12 +02:00
Tim Wojtulewicz
5fd563da79 Fix warning about grealpath when running 'make dist' on Linux 2024-07-11 13:45:14 -07:00
Tim Wojtulewicz
822102382c Start of 7.1.0 development 2024-07-11 13:19:15 -07:00
Tim Wojtulewicz
f124b31904 Updating submodule(s) [nomail] 2024-07-11 12:21:02 -07:00
Christian Kreibich
c4bc9078ef Merge branch 'topic/christian/broker-prometheus-cpp'
* topic/christian/broker-prometheus-cpp:
  Update the scripts.base.frameworks.telemetry.internal-metrics test
  Revert "Temporarily disable the scripts/base/frameworks/telemetry/internal-metrics btest"
  Bump Broker to pull in new Prometheus support and pass in Zeek's registry
2024-07-11 12:15:16 -07:00
Christian Kreibich
77816f9a6b Update the scripts.base.frameworks.telemetry.internal-metrics test
This now uses different record fields, and for now we no longer have CAF
telemetry. We indicate we're running under test to get reliable ordering in the
baselined output.
2024-07-11 12:13:05 -07:00
Christian Kreibich
b387da3489 Revert "Temporarily disable the scripts/base/frameworks/telemetry/internal-metrics btest"
This reverts commit d6e97ab306.
Broker's telemetry is now available again.
2024-07-11 12:13:05 -07:00
Dominik Charousset
40e1e43c25 Bump Broker to pull in new Prometheus support and pass in Zeek's registry
Co-authored-by: Christian Kreibich <christian@corelight.com>
2024-07-11 12:13:05 -07:00
Johanna Amann
4acd0297b0 Merge remote-tracking branch 'origin/master' into topic/johanna/spicy-tls
* origin/master: (89 commits)
  Update doc submodule [nomail] [skip ci]
  Bump cmake submodule [nomail]
  testing/btest: Default to HILTI_JIT_PARALLELISM=1
  Revert "CI: Use ccache and a single CPU when building spicy analyzers for btests"
  Update doc submodule [nomail] [skip ci]
  CI: Use ccache and a single CPU when building spicy analyzers for btests
  Extend btest for logging of disabled analyzers
  Update zeek-aux submodule [nomail]
  Add logging of disabled analyzers to analyzer.log
  Bump auxil/spicy to latest development snapshot
  Management framework: bump cluster testsuite to pull in telemetry tests
  Management framework: bump zeek-client
  Management framework: augment deployed configs with instance IP addresses
  Management framework: add auto-enumeration of metrics ports
  Management framework: propagate metrics port from agent
  Management framework: add metrics port in management & Supervisor node records
  Harden the telemetry manager against unset Telemetry::metrics_address
  Comment-only tweaks for telemetry-related settings.
  Fix for --display-cmake in configure Moved build directory creation further down in the script so that --display-cmake has a chance to happen before build tree setup.
  Update submodules [nomail]
  ...
2024-07-11 14:02:28 +01:00
Benjamin Bannier
596f4114fc Do not emit hook files for builtin modules
We would previously emit a C++ file with hooks for at least the builtin
`spicy` module even though that module like any other builtin module
never contains implementations of hooks for types in user code.

This patch adds a blocklist of builtin modules which are skipped for
generating hook files.
2024-07-11 14:15:21 +02:00
zeek-bot
e99b94c18f Update doc submodule [nomail] [skip ci] 2024-07-11 00:10:52 +00:00
Tim Wojtulewicz
3f475ebf7d Bump cmake submodule [nomail] 2024-07-10 12:37:21 -07:00
Arne Welzel
a9c4daaa22 Merge remote-tracking branch 'jgras/topic/jgras/log-disable-analyzer'
* jgras/topic/jgras/log-disable-analyzer:
  Extend btest for logging of disabled analyzers
  Add logging of disabled analyzers to analyzer.log
2024-07-10 18:51:26 +02:00
Arne Welzel
1c8fedaf09 Merge remote-tracking branch 'origin/topic/awelzel/move-hilti-jit-parallelism-to-btest-cfg'
* origin/topic/awelzel/move-hilti-jit-parallelism-to-btest-cfg:
  testing/btest: Default to HILTI_JIT_PARALLELISM=1
  Revert "CI: Use ccache and a single CPU when building spicy analyzers for btests"
2024-07-10 11:06:46 +02:00
Arne Welzel
7b99fc01a9 testing/btest: Default to HILTI_JIT_PARALLELISM=1
This is a rework of b59bed9d06 moving
HILTI_JIT_PARALLELISM=1 into btest.cfg to make it default applicable to
btest -j users (and CI).

The background for this change is that spicyz may spawn up to nproc compiler
instances by default. Combined with btest -j, this may be nproc x nproc
instances worst case. Particularly with gcc, this easily overloads CI or
local systems, putting them into hard-to-recover-from thrashing/OOM states.

Exporting HILTI_JIT_PARALLELISM in the shell allows overriding.
2024-07-10 11:04:47 +02:00
Arne Welzel
cb500536ea Revert "CI: Use ccache and a single CPU when building spicy analyzers for btests"
This reverts commit b59bed9d06.
2024-07-10 10:02:53 +02:00
zeek-bot
b995924b21 Update doc submodule [nomail] [skip ci] 2024-07-10 00:21:07 +00:00
Tim Wojtulewicz
b309c423f8 Merge remote-tracking branch 'origin/topic/timw/ci-reduce-spicy-test-load'
* origin/topic/timw/ci-reduce-spicy-test-load:
  CI: Use ccache and a single CPU when building spicy analyzers for btests
2024-07-09 15:12:29 -07:00
Tim Wojtulewicz
b64f0e8fd8 Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy'
* origin/topic/bbannier/bump-spicy:
  Bump auxil/spicy to latest development snapshot
2024-07-09 14:53:49 -07:00
Tim Wojtulewicz
b59bed9d06 CI: Use ccache and a single CPU when building spicy analyzers for btests 2024-07-09 14:31:09 -07:00
Jan Grashoefer
c6c8d078c0 Extend btest for logging of disabled analyzers 2024-07-09 20:15:46 +02:00
Tim Wojtulewicz
c557b2156a Merge remote-tracking branch 'origin/topic/vern/script-opt-maint.Jul24'
* origin/topic/vern/script-opt-maint.Jul24:
  minor script optimization updates to reflect recent changes, Coverity findings
2024-07-09 10:15:26 -07:00
Tim Wojtulewicz
03e17a530a Update zeek-aux submodule [nomail] 2024-07-09 09:33:29 -07:00
Jan Grashoefer
0c06c604ab Add logging of disabled analyzers to analyzer.log 2024-07-09 18:22:43 +02:00
Benjamin Bannier
fcded0fc6a Bump auxil/spicy to latest development snapshot 2024-07-09 18:22:21 +02:00
Arne Welzel
e102715bc8 Merge remote-tracking branch 'cknill/topic/cknill/display_cmake_fix'
* cknill/topic/cknill/display_cmake_fix:
  Fix for --display-cmake in configure Moved build directory creation further down in the script so that --display-cmake has a chance to happen before build tree setup.
2024-07-09 17:10:28 +02:00
Christian Kreibich
6552c4e0fe Merge branch 'topic/christian/management-telemetry-additions'
* topic/christian/management-telemetry-additions:
  Management framework: bump cluster testsuite to pull in telemetry tests
  Management framework: bump zeek-client
  Management framework: augment deployed configs with instance IP addresses
  Management framework: add auto-enumeration of metrics ports
  Management framework: propagate metrics port from agent
  Management framework: add metrics port in management & Supervisor node records
  Harden the telemetry manager against unset Telemetry::metrics_address
  Comment-only tweaks for telemetry-related settings.
2024-07-09 00:08:28 -07:00
Christian Kreibich
cdd5062f45 Management framework: bump cluster testsuite to pull in telemetry tests 2024-07-08 23:05:24 -07:00
Christian Kreibich
7d802f27b7 Management framework: bump zeek-client 2024-07-08 23:05:24 -07:00
Christian Kreibich
8a4fb0ee19 Management framework: augment deployed configs with instance IP addresses
The controller learns IP addresses from agents that peer with it, but that
information has so far gotten lost when resulting configs get pushed out to the
agents. This makes these updates include that information.
2024-07-08 23:05:24 -07:00
Christian Kreibich
742f7fe340 Management framework: add auto-enumeration of metrics ports
This is quite redundant with the enumeration for Broker ports,
unfortunately. But the logic is subtly different: all nodes obtain a telemetry
port, while not all nodes require a Broker port, for example, and in the metrics
port assignment we also cross-check selected Broker ports. I found more unified
code actually harder to read in the end.

The logic for the two sets remains the same: from a start point, ports get
enumerated sequentially that aren't otherwise taken. These ports are assumed
available; there's nothing that checks their availability -- for now.

The default start port is 9000. I considered 9090, to align with the Prometheus
default, but counting upward from there is likely to hit trouble with the Broker
default ports (9999/9997), used by the Supervisor. Counting downward is a bit
unnatural, and shifting the Broker default ports brings subtle ordering issues.

This also changes the node ordering logic slightly since it seems more intuitive
to keep sequential ports on a given instance, instead of striping across them.
2024-07-08 23:05:24 -07:00
Christian Kreibich
fa6361af56 Management framework: propagate metrics port from agent
This propagates the metrics port from the node config passed through the
supervisor all the way into the script layer.
2024-07-08 23:05:24 -07:00
Christian Kreibich
563704a26e Management framework: add metrics port in management & Supervisor node records
This allows setting a metrics port for creation in new nodes.
2024-07-08 23:05:24 -07:00
Christian Kreibich
8eb74c04de Harden the telemetry manager against unset Telemetry::metrics_address
We populate that address from the ZEEK_DEFAULT_LISTEN_ADDRESS environment
variable, but weren't prepared for that not being set. We now fall back to
0.0.0.0. This may have the same IPv6 issues that we've encountered elsewhere
when doing so before (v6 interfaces need "::") -- but this is still more likely
to work than not having any string at all.
2024-07-08 23:05:24 -07:00
Christian Kreibich
3ecacf4f50 Comment-only tweaks for telemetry-related settings.
These weren't quite accurate any more.
2024-07-08 23:05:24 -07:00
cknill
8dc120df1d Fix for --display-cmake in configure
Moved build directory creation further down in the script so that --display-cmake
has a chance to happen before build tree setup.
2024-07-08 19:24:40 -06:00
Tim Wojtulewicz
4d31335391 Merge branch 'topic/timw/update-submodules'
* topic/timw/update-submodules:
  Update submodules [nomail]
2024-07-08 15:00:26 -07:00
Tim Wojtulewicz
74ab980992 Update submodules [nomail] 2024-07-08 13:03:30 -07:00
Vern Paxson
a53e28be87 minor script optimization updates to reflect recent changes, Coverity findings 2024-07-06 16:44:27 -07:00