Commit graph

18601 commits

Author SHA1 Message Date
Tim Wojtulewicz
21024c22e0 Merge remote-tracking branch 'origin/topic/awelzel/3316-exit-earlier-on-errors'
* origin/topic/awelzel/3316-exit-earlier-on-errors:
  zeek-setup: Early exit when parsing failed
2023-10-02 11:44:32 -07:00
Tim Wojtulewicz
9dcf1b4cd1 Merge remote-tracking branch 'origin/topic/awelzel/3342-fix-broker-composite-key-tables-deletion'
* origin/topic/awelzel/3342-fix-broker-composite-key-tables-deletion:
  broker/Manager: Fix deletion of composite keys
  Add btest for brokerstore delete with complex index.
2023-10-02 11:43:19 -07:00
Tim Wojtulewicz
bf103738a1 Merge remote-tracking branch 'origin/topic/vern/CPP-Oct23-maint'
* origin/topic/vern/CPP-Oct23-maint:
  minor updates to -O gen-C++ maintenance notes and scripts
  ZAM fix for inlining bug when a local is assigned to a function parameter
  ZAM fixes for record creation and table indexing potentially having side-effects
  removed order-of-evaluation ambiguity from new BTest
  introduce internal "mask" expression node to avoid mixed-type overloading of "/"
  fix bug in failing to concretize empty vectors
  ZAM BTest update to match recent change to one of the tests
  fixed ZAM reduction bug in += / -= operators
2023-10-02 11:09:26 -07:00
Tim Wojtulewicz
3df30be833 Merge remote-tracking branch 'origin/topic/awelzel/generate-docs-increase-ccache-size'
* origin/topic/awelzel/generate-docs-increase-ccache-size:
  generate-docs: Bump ccache from 500M to 2000M
2023-10-02 10:13:13 -07:00
Tim Wojtulewicz
3bdc709829 Merge remote-tracking branch 'origin/topic/bbannier/hilti-cxx-compiler-launcher'
* origin/topic/bbannier/hilti-cxx-compiler-launcher:
  Enable ccache-backed Spicy JIT for users of developer scripts.
2023-10-02 10:10:36 -07:00
Tim Wojtulewicz
0424eefd96 Merge remote-tracking branch 'origin/topic/awelzel/3309-dns-mgr-revents-ouch'
* origin/topic/awelzel/3309-dns-mgr-revents-ouch:
  DNS_Mgr: Fix | vs & for pollfd.revents
2023-10-02 10:09:02 -07:00
Benjamin Bannier
eb18530d52 Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy-master' 2023-10-02 14:39:21 +02:00
Benjamin Bannier
e624c7500e Enable ccache-backed Spicy JIT for users of developer scripts.
For CI and more general BTest execution we already enabled ccache
previously, but developers using a Spicy build out of the build
directory still needed to manually set up caching.

With this patch we enable it for such cases as well via
`zeek-path-dev.*sh`.
2023-10-02 13:57:56 +02:00
Benjamin Bannier
13e3c47c47 Bump auxil/spicy to latest development snapshot 2023-10-02 10:57:48 +02:00
Arne Welzel
409f40270f Merge remote-tracking branch 'origin/topic/awelzel/ubuntu22-on-benchmark-nightly'
* origin/topic/awelzel/ubuntu22-on-benchmark-nightly:
  cirrus/ci: Run ubuntu22_task on benchmark-nightly, too
2023-09-28 20:29:56 +02:00
Arne Welzel
9badbc5232 broker/Manager: Fix deletion of composite keys
Follow-up for commit 44ae8f9e8f.

Closes #3342.
2023-09-28 18:33:12 +02:00
Jan Grashoefer
bba07cea8b Add btest for brokerstore delete with complex index.
Slightly adapted. I wasn't sure how much guarantee we have around the
change propagation when calling terminate() right after the delete.
Flipped it around a bit, having the worker outlive the manger instead.
2023-09-28 18:31:34 +02:00
Benjamin Bannier
2833eee3c8 Merge branch 'topic/bbannier/spicy-bitfield-visitation' 2023-09-28 10:01:11 +02:00
Vern Paxson
417a99ed0d minor updates to -O gen-C++ maintenance notes and scripts 2023-09-27 15:56:31 -07:00
Vern Paxson
cbc3d9c739 ZAM fix for inlining bug when a local is assigned to a function parameter 2023-09-27 14:03:45 -07:00
Vern Paxson
8f92e0d39b ZAM fixes for record creation and table indexing potentially having side-effects 2023-09-27 11:38:21 -07:00
Vern Paxson
3addda28d3 removed order-of-evaluation ambiguity from new BTest 2023-09-27 11:36:22 -07:00
Arne Welzel
d17e60e682 generate-docs: Bump ccache from 500M to 2000M
Since enabling Spicy, 500M (default) of ccache max-size size is not
sufficient for efficient cache usage of a Zeek build anymore, so we
end up thrashing the configured ccache. Build times have increased
from 5-10 minutes to 1+ hour after enabling Spicy in the docs build.
2023-09-27 18:20:36 +02:00
Arne Welzel
11f5bcbd27 cirrus/ci: Run ubuntu22_task on benchmark-nightly, too 2023-09-27 17:19:24 +02:00
Benjamin Bannier
517074bf41 Fix GCC warnings about unintended pointer comparisions
When inspecting Spicy values for marshalling to Zeek we should receive
field identifiers as owned `std::string` so it should be safe to
directly compare them against `const char[]`. It seems that e.g.,
gcc-9.4.0's analysis sees things differently and warns of a comparision
of `const char*` (which as written would compare pointer values; the
correct API for this would be `::strlen`).

This patch addresses that warning by visiting with type which supports
`==` comparison, in particular, take a `std::string_view` so we could
even visit `const char*` identifiers.
2023-09-27 17:12:24 +02:00
Arne Welzel
5c1df2b706 Merge remote-tracking branch 'origin/topic/awelzel/ubuntu22-spicy-task'
* origin/topic/awelzel/ubuntu22-spicy-task:
  cirrus/ci: Add ubuntu22_spicy_task and ubuntu22_spicy_head_task
  cirrus/ci: Add pre_build_script mechanism
2023-09-27 16:53:29 +02:00
Arne Welzel
04d0b6daa3 DNS_Mgr: Fix | vs & for pollfd.revents
Thank you Coverity.

Follow-up for c4c4a23bfd.
2023-09-27 13:48:23 +02:00
Arne Welzel
519372fa4a get_current_packet: Add note/warning to docs
Closes #3330
2023-09-27 12:09:17 +02:00
Arne Welzel
12e0dc110b zeek-setup: Early exit when parsing failed
When there are errors reported during yyparse(), Zeek still continued
running initialization functions like init_general_global_var(), init_net_var()
and run_bif_initializers(). These usually call abort() in unexpected
situations causing misleading and confusing errors. This patch prevents
this by exiting earlier.

Closes #3316
2023-09-27 09:54:57 +02:00
Vern Paxson
434a7e059d introduce internal "mask" expression node to avoid mixed-type overloading of "/" 2023-09-26 14:39:26 -07:00
Vern Paxson
b53a025b1e fix bug in failing to concretize empty vectors 2023-09-26 14:39:05 -07:00
Vern Paxson
59db8a11bd ZAM BTest update to match recent change to one of the tests 2023-09-26 13:29:07 -07:00
Vern Paxson
6096e03d90 fixed ZAM reduction bug in += / -= operators 2023-09-26 12:59:54 -07:00
Arne Welzel
80eaa4aa64 cirrus/ci: Add ubuntu22_spicy_task and ubuntu22_spicy_head_task
These tasks are meant to run nightly on the master branch. Currently,
the external dns, http and dhcp Spicy analyzers are installed via zkg
post building. The build artifact is uploaded to Cirrus and the benchmarker
API triggered.

For the spicy_head task, the auxil/spicy submodule is pulled to the latest
commit. This also provides a bit of a nightly integration test.
2023-09-26 19:45:07 +02:00
Arne Welzel
8a2c81aa0c cirrus/ci: Add pre_build_script mechanism
Allow tasks to run a bit of code just before ci/build.sh.
2023-09-26 19:37:32 +02:00
Arne Welzel
c6486e70ef Merge remote-tracking branch 'origin/topic/awelzel/3309-dns-mgr-fd-set-size-1024'
* origin/topic/awelzel/3309-dns-mgr-fd-set-size-1024:
  auxil/libunistd: Bump for poll() inclusion
  DNS_Mgr: Replace ares_fds() with ares_getsock()
2023-09-26 18:42:37 +02:00
Arne Welzel
55475af34c NEWS: Add entry for dot directory changes 2023-09-26 17:46:33 +02:00
Arne Welzel
a11961dffe Merge remote-tracking branch 'origin/topic/bbannier/btest-issue-98'
* origin/topic/bbannier/btest-issue-98:
  Use unique port for test `scripts.base.utils.active-http`
  Do not automatically load Spicy modules in dot paths.
  Exclude dot directories when searching ZEEK_PLUGIN_PATH
2023-09-26 16:24:15 +02:00
Benjamin Bannier
a4c4a23ba7 Use unique port for test scripts.base.utils.active-http 2023-09-26 12:49:59 +02:00
Benjamin Bannier
5f1b015500 Do not automatically load Spicy modules in dot paths. 2023-09-26 11:56:52 +02:00
Benjamin Bannier
0896f09081 Exclude dot directories when searching ZEEK_PLUGIN_PATH
Dot directories rarely contain anything we would want to load as a
dynamic plugin. Even worse, they likely contain files with externally
controlled lifetimes which might be removed while we are using them
(see e.g., zeek/btest#98).

With this patch we do not search _discovered_ dot directories anymore.
We continue to load from a user-specified `ZEEK_PLUGIN_PATH`, even if
its name starts with a dot.

Since this patch changes previous behavior it is a **BREAKING CHANGE**.
2023-09-26 11:56:47 +02:00
Arne Welzel
599037d215 Merge remote-tracking branch 'origin/topic/awelzel/embed-record-val'
* origin/topic/awelzel/embed-record-val:
  VectorVal: Embed vector_val
  RecordVal: Embed record_val
2023-09-26 11:07:33 +02:00
Arne Welzel
0668c15cf9 auxil/libunistd: Bump for poll() inclusion 2023-09-26 09:57:23 +02:00
Arne Welzel
c4c4a23bfd DNS_Mgr: Replace ares_fds() with ares_getsock()
On Slack, a user reported "fortify source" aborts within ares_fds() due
to the FDs used by c-ares exceeding 1024 and thereby larger than the
maximum fd value that a fd_set can hold.

Switch to ares_get_socks() and poll() to avoid this.

Closes #3309.
2023-09-26 09:57:20 +02:00
Benjamin Bannier
03597d210d Merge branch 'origin/topic/bbannier/spicy-reject_protocol-during-teardown' 2023-09-25 14:59:40 +02:00
Arne Welzel
9fedab9845 ci/benchmark: Also send CIRRUS_TASK_NAME to benchmarker API
We otherwise cannot distinguish between different benchmarking jobs when
we introduce them (think ubuntu22_jemalloc or ubuntu22_spicy).
2023-09-25 13:50:50 +02:00
Tim Wojtulewicz
5ff24dad23 Merge remote-tracking branch 'origin/topic/timw/disable-tsan-ci-task'
* origin/topic/timw/disable-tsan-ci-task:
  CI: Disable tsan task until it can be fixed
2023-09-22 14:38:38 -07:00
Tim Wojtulewicz
f7cfbfd51a CI: Disable tsan task until it can be fixed 2023-09-22 14:38:06 -07:00
Tim Wojtulewicz
3be86d4b0d Update af_packet plugin submodule [nomail] 2023-09-22 13:00:35 -07:00
Tim Wojtulewicz
e9922e960e Revert "Update af_packet plugin submodule [nomail]"
This reverts commit 32944ebc5d.
2023-09-22 12:59:14 -07:00
Arne Welzel
cbaf43e8ea VectorVal: Embed vector_val
Similar motivation as for RecordVal, save an extra malloc/free
and pointer indirection.

This breaks the `auto& RawVec()` API which previously returned
a reference to the std::vector*. It now returns a reference
to the vector instead. It's commented as intended for internal
and compiled code, so even though it's public API,

The previous `std::vector<std::optional<ZVal>>*&` return type was also very
likely not intended (all consumers just dereference it anyhow). I'm certain
this API was never meant to modify the actual pointer value.

I've switched to explicit typing, too.
2023-09-22 21:52:52 +02:00
Arne Welzel
f362935a66 RecordVal: Embed record_val
This should remove one malloc/free per created and destroyed record instance
and avoid one extra pointer indirection to access fields.
2023-09-22 19:43:07 +02:00
Tim Wojtulewicz
32944ebc5d Update af_packet plugin submodule [nomail] 2023-09-22 09:38:24 -07:00
Benjamin Bannier
d97a578e4f Do not require cookie for reject_protocol during teardown.
A user reported that they ran into a situation where a parse error
caused an assertion failure in `reject_protocol`. printf debugging points
to the `CookieSetter`s in the `try`/`catch` blocks during processing
already clearing the cookie RAII-style; since their `catch` blocks
already send an analyzer violation no further sending of one from
`reject_protocol` is required.

This patch replaces the assert in `reject_protocol` with a runtime check
and an early return if the cookie is already cleared.
2023-09-22 18:07:00 +02:00
Benjamin Bannier
0c1a8f8fc4 Bump spicy [nomail] 2023-09-22 15:12:17 +02:00