Commit graph

16055 commits

Author SHA1 Message Date
Arne Welzel
48b27f30c9 Merge remote-tracking branch 'origin/topic/awelzel/plugin-hooks-bare-mode'
* origin/topic/awelzel/plugin-hooks-bare-mode:
  btest/plugins/hooks: Run in bare mode
2023-10-09 16:27:10 +02:00
Arne Welzel
2ed89ff84f Bump cmake submodule for 3.15 requirement 2023-10-09 16:23:12 +02:00
Arne Welzel
07ac6fa074 btest/plugins/hooks: Run in bare mode
Motivation is basically the same as in 88bb527026.
For plugin.hooks, one example is that adding a new option in the default script
changes the baseline due registration of change handlers. Also, the connection
record is printed in various places, resulting in churn when the default
scripts change.
2023-10-09 16:13:59 +02:00
Arne Welzel
4a8f868471 Merge remote-tracking branch 'origin/topic/awelzel/dump-events-in-bare-mode'
* origin/topic/awelzel/dump-events-in-bare-mode:
  btest/coverage: Add record-fields test
  btest/dump-events: Run in bare mode, load conn and smtp scripts only
2023-10-09 14:15:49 +02:00
Arne Welzel
998ab80385 btest/coverage: Add record-fields test
Justin pointed out that the misc/dump-events test shows added fields to
the connection record. Add a new test that prints the connection record
recursively in bare and default mode to cover that use-case
specifically.
2023-10-09 13:58:08 +02:00
Arne Welzel
88bb527026 btest/dump-events: Run in bare mode, load conn and smtp scripts only
The dump-events baseline changes are pure noise and have spurred confusion
for internal and external contributors. For example, adding new
analyzers have perturbed orderings of sets holding analyzer tags.

Running in non-bare mode, the baselines change almost whenever any of the
record types attached to connections change in the default scripts. This
causes continuous and seemingly little useful updates to the baselines.

This change switches the test to run in bare mode and explicitly loads
just base/protocols/conn and base/protocols/smtp. The primary intention
of the test should be testing the functionality of the misc/dump-events
script, not the raised events of all loaded default scripts (for that the
used PCAP is too narrow).

Protocol specific scripts that do want to leverage misc/dump-events for
baseline creation of their or their analyzer's events can add additional
specific tests with suitable PCAP files.
2023-10-09 12:20:10 +02:00
zeek-bot
c0e209f870 Update doc submodule [nomail] [skip ci] 2023-10-07 01:02:22 +00:00
Benjamin Bannier
8793cde485 Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy' 2023-10-06 16:09:05 +02:00
Benjamin Bannier
8d55a10ab0 Bump auxil/spicy to latest development snapshot 2023-10-06 12:40:45 +02:00
Christian Kreibich
26897bf906 Merge branch 'topic/christian/normalize-file-paths'
* topic/christian/normalize-file-paths:
  Btest for normalized reporter path reporting
  Normalize file paths in find_file_in_path()
2023-10-04 22:46:54 -07:00
Arne Welzel
0630bdbbd5 Merge remote-tracking branch 'origin/topic/awelzel/3314-lambda-redefinition-segfault'
* origin/topic/awelzel/3314-lambda-redefinition-segfault:
  Var/Func: Render function parameters using comma, not semicolon
  Var: Fix null-pointer deref on redefinition of lambdas
2023-10-04 20:16:29 +02:00
Arne Welzel
8109bbc52f Var/Func: Render function parameters using comma, not semicolon
A bit larger follow-up to what Tim pointed out: Function prototype descriptions
previously used semicolons to separate parameters.

Switch to use commas when a RecordType is used as function parameter.
Use existing "func_args" naming for consistency.
2023-10-04 17:42:30 +02:00
Arne Welzel
8ede22f6ec Var: Fix null-pointer deref on redefinition of lambdas
Closes #3314
2023-10-04 15:18:24 +02:00
Christian Kreibich
e84a7ab7f1 Btest for normalized reporter path reporting 2023-10-03 22:16:00 -07:00
Christian Kreibich
6108b18a3b Normalize file paths in find_file_in_path()
This renders script file names more nicely, for example when printed by the
reporter. It also avoids redundant prefixing of "./" to local scripts. I'm not
adding unit tests since normalize_path() already has them. A btest follows in
the next commit.
2023-10-03 22:14:40 -07:00
zeek-bot
e22bf8ebb6 Update doc submodule [nomail] [skip ci] 2023-10-03 01:28:03 +00:00
Tim Wojtulewicz
ff1d07b548 Merge remote-tracking branch 'origin/topic/awelzel/3330-get-current-packet-docs'
* origin/topic/awelzel/3330-get-current-packet-docs:
  get_current_packet: Add note/warning to docs
2023-10-02 11:51:07 -07:00
Tim Wojtulewicz
21024c22e0 Merge remote-tracking branch 'origin/topic/awelzel/3316-exit-earlier-on-errors'
* origin/topic/awelzel/3316-exit-earlier-on-errors:
  zeek-setup: Early exit when parsing failed
2023-10-02 11:44:32 -07:00
Tim Wojtulewicz
9dcf1b4cd1 Merge remote-tracking branch 'origin/topic/awelzel/3342-fix-broker-composite-key-tables-deletion'
* origin/topic/awelzel/3342-fix-broker-composite-key-tables-deletion:
  broker/Manager: Fix deletion of composite keys
  Add btest for brokerstore delete with complex index.
2023-10-02 11:43:19 -07:00
Tim Wojtulewicz
bf103738a1 Merge remote-tracking branch 'origin/topic/vern/CPP-Oct23-maint'
* origin/topic/vern/CPP-Oct23-maint:
  minor updates to -O gen-C++ maintenance notes and scripts
  ZAM fix for inlining bug when a local is assigned to a function parameter
  ZAM fixes for record creation and table indexing potentially having side-effects
  removed order-of-evaluation ambiguity from new BTest
  introduce internal "mask" expression node to avoid mixed-type overloading of "/"
  fix bug in failing to concretize empty vectors
  ZAM BTest update to match recent change to one of the tests
  fixed ZAM reduction bug in += / -= operators
2023-10-02 11:09:26 -07:00
Tim Wojtulewicz
3df30be833 Merge remote-tracking branch 'origin/topic/awelzel/generate-docs-increase-ccache-size'
* origin/topic/awelzel/generate-docs-increase-ccache-size:
  generate-docs: Bump ccache from 500M to 2000M
2023-10-02 10:13:13 -07:00
Tim Wojtulewicz
3bdc709829 Merge remote-tracking branch 'origin/topic/bbannier/hilti-cxx-compiler-launcher'
* origin/topic/bbannier/hilti-cxx-compiler-launcher:
  Enable ccache-backed Spicy JIT for users of developer scripts.
2023-10-02 10:10:36 -07:00
Tim Wojtulewicz
0424eefd96 Merge remote-tracking branch 'origin/topic/awelzel/3309-dns-mgr-revents-ouch'
* origin/topic/awelzel/3309-dns-mgr-revents-ouch:
  DNS_Mgr: Fix | vs & for pollfd.revents
2023-10-02 10:09:02 -07:00
Benjamin Bannier
eb18530d52 Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy-master' 2023-10-02 14:39:21 +02:00
Benjamin Bannier
e624c7500e Enable ccache-backed Spicy JIT for users of developer scripts.
For CI and more general BTest execution we already enabled ccache
previously, but developers using a Spicy build out of the build
directory still needed to manually set up caching.

With this patch we enable it for such cases as well via
`zeek-path-dev.*sh`.
2023-10-02 13:57:56 +02:00
Benjamin Bannier
13e3c47c47 Bump auxil/spicy to latest development snapshot 2023-10-02 10:57:48 +02:00
Arne Welzel
409f40270f Merge remote-tracking branch 'origin/topic/awelzel/ubuntu22-on-benchmark-nightly'
* origin/topic/awelzel/ubuntu22-on-benchmark-nightly:
  cirrus/ci: Run ubuntu22_task on benchmark-nightly, too
2023-09-28 20:29:56 +02:00
Arne Welzel
9badbc5232 broker/Manager: Fix deletion of composite keys
Follow-up for commit 44ae8f9e8f.

Closes #3342.
2023-09-28 18:33:12 +02:00
Jan Grashoefer
bba07cea8b Add btest for brokerstore delete with complex index.
Slightly adapted. I wasn't sure how much guarantee we have around the
change propagation when calling terminate() right after the delete.
Flipped it around a bit, having the worker outlive the manger instead.
2023-09-28 18:31:34 +02:00
Benjamin Bannier
2833eee3c8 Merge branch 'topic/bbannier/spicy-bitfield-visitation' 2023-09-28 10:01:11 +02:00
Vern Paxson
417a99ed0d minor updates to -O gen-C++ maintenance notes and scripts 2023-09-27 15:56:31 -07:00
Vern Paxson
cbc3d9c739 ZAM fix for inlining bug when a local is assigned to a function parameter 2023-09-27 14:03:45 -07:00
Vern Paxson
8f92e0d39b ZAM fixes for record creation and table indexing potentially having side-effects 2023-09-27 11:38:21 -07:00
Vern Paxson
3addda28d3 removed order-of-evaluation ambiguity from new BTest 2023-09-27 11:36:22 -07:00
Arne Welzel
d17e60e682 generate-docs: Bump ccache from 500M to 2000M
Since enabling Spicy, 500M (default) of ccache max-size size is not
sufficient for efficient cache usage of a Zeek build anymore, so we
end up thrashing the configured ccache. Build times have increased
from 5-10 minutes to 1+ hour after enabling Spicy in the docs build.
2023-09-27 18:20:36 +02:00
Arne Welzel
11f5bcbd27 cirrus/ci: Run ubuntu22_task on benchmark-nightly, too 2023-09-27 17:19:24 +02:00
Benjamin Bannier
517074bf41 Fix GCC warnings about unintended pointer comparisions
When inspecting Spicy values for marshalling to Zeek we should receive
field identifiers as owned `std::string` so it should be safe to
directly compare them against `const char[]`. It seems that e.g.,
gcc-9.4.0's analysis sees things differently and warns of a comparision
of `const char*` (which as written would compare pointer values; the
correct API for this would be `::strlen`).

This patch addresses that warning by visiting with type which supports
`==` comparison, in particular, take a `std::string_view` so we could
even visit `const char*` identifiers.
2023-09-27 17:12:24 +02:00
Arne Welzel
5c1df2b706 Merge remote-tracking branch 'origin/topic/awelzel/ubuntu22-spicy-task'
* origin/topic/awelzel/ubuntu22-spicy-task:
  cirrus/ci: Add ubuntu22_spicy_task and ubuntu22_spicy_head_task
  cirrus/ci: Add pre_build_script mechanism
2023-09-27 16:53:29 +02:00
Arne Welzel
04d0b6daa3 DNS_Mgr: Fix | vs & for pollfd.revents
Thank you Coverity.

Follow-up for c4c4a23bfd.
2023-09-27 13:48:23 +02:00
Arne Welzel
519372fa4a get_current_packet: Add note/warning to docs
Closes #3330
2023-09-27 12:09:17 +02:00
Arne Welzel
12e0dc110b zeek-setup: Early exit when parsing failed
When there are errors reported during yyparse(), Zeek still continued
running initialization functions like init_general_global_var(), init_net_var()
and run_bif_initializers(). These usually call abort() in unexpected
situations causing misleading and confusing errors. This patch prevents
this by exiting earlier.

Closes #3316
2023-09-27 09:54:57 +02:00
Vern Paxson
434a7e059d introduce internal "mask" expression node to avoid mixed-type overloading of "/" 2023-09-26 14:39:26 -07:00
Vern Paxson
b53a025b1e fix bug in failing to concretize empty vectors 2023-09-26 14:39:05 -07:00
Vern Paxson
59db8a11bd ZAM BTest update to match recent change to one of the tests 2023-09-26 13:29:07 -07:00
Vern Paxson
6096e03d90 fixed ZAM reduction bug in += / -= operators 2023-09-26 12:59:54 -07:00
Arne Welzel
80eaa4aa64 cirrus/ci: Add ubuntu22_spicy_task and ubuntu22_spicy_head_task
These tasks are meant to run nightly on the master branch. Currently,
the external dns, http and dhcp Spicy analyzers are installed via zkg
post building. The build artifact is uploaded to Cirrus and the benchmarker
API triggered.

For the spicy_head task, the auxil/spicy submodule is pulled to the latest
commit. This also provides a bit of a nightly integration test.
2023-09-26 19:45:07 +02:00
Arne Welzel
8a2c81aa0c cirrus/ci: Add pre_build_script mechanism
Allow tasks to run a bit of code just before ci/build.sh.
2023-09-26 19:37:32 +02:00
Arne Welzel
c6486e70ef Merge remote-tracking branch 'origin/topic/awelzel/3309-dns-mgr-fd-set-size-1024'
* origin/topic/awelzel/3309-dns-mgr-fd-set-size-1024:
  auxil/libunistd: Bump for poll() inclusion
  DNS_Mgr: Replace ares_fds() with ares_getsock()
2023-09-26 18:42:37 +02:00
Arne Welzel
55475af34c NEWS: Add entry for dot directory changes 2023-09-26 17:46:33 +02:00
Arne Welzel
a11961dffe Merge remote-tracking branch 'origin/topic/bbannier/btest-issue-98'
* origin/topic/bbannier/btest-issue-98:
  Use unique port for test `scripts.base.utils.active-http`
  Do not automatically load Spicy modules in dot paths.
  Exclude dot directories when searching ZEEK_PLUGIN_PATH
2023-09-26 16:24:15 +02:00