Commit graph

15559 commits

Author SHA1 Message Date
Arne Welzel
8a2c81aa0c cirrus/ci: Add pre_build_script mechanism
Allow tasks to run a bit of code just before ci/build.sh.
2023-09-26 19:37:32 +02:00
Arne Welzel
c6486e70ef Merge remote-tracking branch 'origin/topic/awelzel/3309-dns-mgr-fd-set-size-1024'
* origin/topic/awelzel/3309-dns-mgr-fd-set-size-1024:
  auxil/libunistd: Bump for poll() inclusion
  DNS_Mgr: Replace ares_fds() with ares_getsock()
2023-09-26 18:42:37 +02:00
Arne Welzel
55475af34c NEWS: Add entry for dot directory changes 2023-09-26 17:46:33 +02:00
Arne Welzel
a11961dffe Merge remote-tracking branch 'origin/topic/bbannier/btest-issue-98'
* origin/topic/bbannier/btest-issue-98:
  Use unique port for test `scripts.base.utils.active-http`
  Do not automatically load Spicy modules in dot paths.
  Exclude dot directories when searching ZEEK_PLUGIN_PATH
2023-09-26 16:24:15 +02:00
Benjamin Bannier
a4c4a23ba7 Use unique port for test scripts.base.utils.active-http 2023-09-26 12:49:59 +02:00
Benjamin Bannier
5f1b015500 Do not automatically load Spicy modules in dot paths. 2023-09-26 11:56:52 +02:00
Benjamin Bannier
0896f09081 Exclude dot directories when searching ZEEK_PLUGIN_PATH
Dot directories rarely contain anything we would want to load as a
dynamic plugin. Even worse, they likely contain files with externally
controlled lifetimes which might be removed while we are using them
(see e.g., zeek/btest#98).

With this patch we do not search _discovered_ dot directories anymore.
We continue to load from a user-specified `ZEEK_PLUGIN_PATH`, even if
its name starts with a dot.

Since this patch changes previous behavior it is a **BREAKING CHANGE**.
2023-09-26 11:56:47 +02:00
Arne Welzel
599037d215 Merge remote-tracking branch 'origin/topic/awelzel/embed-record-val'
* origin/topic/awelzel/embed-record-val:
  VectorVal: Embed vector_val
  RecordVal: Embed record_val
2023-09-26 11:07:33 +02:00
Arne Welzel
0668c15cf9 auxil/libunistd: Bump for poll() inclusion 2023-09-26 09:57:23 +02:00
Arne Welzel
c4c4a23bfd DNS_Mgr: Replace ares_fds() with ares_getsock()
On Slack, a user reported "fortify source" aborts within ares_fds() due
to the FDs used by c-ares exceeding 1024 and thereby larger than the
maximum fd value that a fd_set can hold.

Switch to ares_get_socks() and poll() to avoid this.

Closes #3309.
2023-09-26 09:57:20 +02:00
Benjamin Bannier
03597d210d Merge branch 'origin/topic/bbannier/spicy-reject_protocol-during-teardown' 2023-09-25 14:59:40 +02:00
Arne Welzel
9fedab9845 ci/benchmark: Also send CIRRUS_TASK_NAME to benchmarker API
We otherwise cannot distinguish between different benchmarking jobs when
we introduce them (think ubuntu22_jemalloc or ubuntu22_spicy).
2023-09-25 13:50:50 +02:00
Tim Wojtulewicz
5ff24dad23 Merge remote-tracking branch 'origin/topic/timw/disable-tsan-ci-task'
* origin/topic/timw/disable-tsan-ci-task:
  CI: Disable tsan task until it can be fixed
2023-09-22 14:38:38 -07:00
Tim Wojtulewicz
f7cfbfd51a CI: Disable tsan task until it can be fixed 2023-09-22 14:38:06 -07:00
Tim Wojtulewicz
3be86d4b0d Update af_packet plugin submodule [nomail] 2023-09-22 13:00:35 -07:00
Tim Wojtulewicz
e9922e960e Revert "Update af_packet plugin submodule [nomail]"
This reverts commit 32944ebc5d.
2023-09-22 12:59:14 -07:00
Arne Welzel
cbaf43e8ea VectorVal: Embed vector_val
Similar motivation as for RecordVal, save an extra malloc/free
and pointer indirection.

This breaks the `auto& RawVec()` API which previously returned
a reference to the std::vector*. It now returns a reference
to the vector instead. It's commented as intended for internal
and compiled code, so even though it's public API,

The previous `std::vector<std::optional<ZVal>>*&` return type was also very
likely not intended (all consumers just dereference it anyhow). I'm certain
this API was never meant to modify the actual pointer value.

I've switched to explicit typing, too.
2023-09-22 21:52:52 +02:00
Arne Welzel
f362935a66 RecordVal: Embed record_val
This should remove one malloc/free per created and destroyed record instance
and avoid one extra pointer indirection to access fields.
2023-09-22 19:43:07 +02:00
Tim Wojtulewicz
32944ebc5d Update af_packet plugin submodule [nomail] 2023-09-22 09:38:24 -07:00
Benjamin Bannier
d97a578e4f Do not require cookie for reject_protocol during teardown.
A user reported that they ran into a situation where a parse error
caused an assertion failure in `reject_protocol`. printf debugging points
to the `CookieSetter`s in the `try`/`catch` blocks during processing
already clearing the cookie RAII-style; since their `catch` blocks
already send an analyzer violation no further sending of one from
`reject_protocol` is required.

This patch replaces the assert in `reject_protocol` with a runtime check
and an early return if the cookie is already cleared.
2023-09-22 18:07:00 +02:00
Benjamin Bannier
0c1a8f8fc4 Bump spicy [nomail] 2023-09-22 15:12:17 +02:00
Benjamin Bannier
e80a75a058 Bump Spicy 2023-09-22 13:56:43 +02:00
Arne Welzel
193ad0d87b Merge remote-tracking branch 'origin/topic/bbannier/format-spicy_add_analyzer'
* origin/topic/bbannier/format-spicy_add_analyzer:
  Fix formatting of `spicy_add_analyzer`
2023-09-22 12:13:39 +02:00
Arne Welzel
5db8e4111e Update cmake submodule [nomail] 2023-09-22 12:12:41 +02:00
Benjamin Bannier
f1e58d0502 Fix formatting of spicy_add_analyzer 2023-09-22 10:59:14 +02:00
zeek-bot
5fe2efe319 Update doc submodule [nomail] [skip ci] 2023-09-22 01:16:10 +00:00
Robin Sommer
b5aecece68
Merge remote-tracking branch 'origin/topic/gh-196-spicy-docs'
* origin/topic/gh-196-spicy-docs:
  Add check to docs CI whether Spicy docs are up to date.
  Bump zeek-docs.
  Enable Spicy when generating docs.
  Zeekygen: Merge in Spicy analyzer documentation.
  Zeekygen: Refine location tracking for Spicy events.
  Zeekygen: Refactor code documenting analyzers.
  Add `%doc-*` tags to Syslog and Finger analyzers.
  Spicy: Support additional documentation tags inside EVT files.
  Fix lack of namespace qualification inside an include file.
  Exclude script from documentation that interferes with other locations.
2023-09-21 10:56:14 +02:00
Robin Sommer
08d8dd51ca
Add check to docs CI whether Spicy docs are up to date. 2023-09-21 10:54:03 +02:00
Robin Sommer
21d6cf8190
Bump zeek-docs. 2023-09-21 10:54:03 +02:00
Robin Sommer
1726081e80
Enable Spicy when generating docs. 2023-09-21 10:54:03 +02:00
Robin Sommer
a89694b32d
Zeekygen: Merge in Spicy analyzer documentation.
This now merges information on analyzers from both plugins and Spicy.

Closes #3306.
2023-09-21 10:54:03 +02:00
Robin Sommer
8a3d645caf
Zeekygen: Refine location tracking for Spicy events.
When Spicy defines an event through EVT, it would end up with
more-or-less random, stale Zeek-side location information. Now making
sure it's unset, plus then using any script-side prototype we may
encounter later to update it.

Also fixing a couple location comparisons that I believe were broken
because we do sometimes copy location objects and hence can't compare
by pointer.
2023-09-21 10:54:03 +02:00
Robin Sommer
d45d197c21
Zeekygen: Refactor code documenting analyzers.
No functional change, but we refactor the logic to

1. Untie the function signature from the fact that the information
   currently comes only out of plugins.
2. Avoid some redundancies

In the next commit, we'll use this to add information from Spicy EVT
files into the output.
2023-09-21 10:54:02 +02:00
Robin Sommer
330440bb70
Add %doc-* tags to Syslog and Finger analyzers.
Same ID/text as the legacy analyzers are using.
2023-09-21 10:54:02 +02:00
Robin Sommer
6f882af7cc
Spicy: Support additional documentation tags inside EVT files.
So far we had trouble documenting Spicy analyzers through Zeekygen
because they would show up as components belonging to the
`Zeek::Spicy` plugin; whereas traditional analyzers would be their own
plugins and hence documented individually on their own. This commit
teaches Zeekygen to track Spicy analyzers separately inside their own
`Info` instances. This information isn't further used in this commit
yet, but will be merged with the plugin output in a subsequent change
to get the expected joint output.

To pass additional information to Zeekygen, EVT files now also support
two new tags for Zeekygen purposes:

- `%doc-id = ID;` defines the global ID under which everything inside
  the EVT file will be documented by Zeekygen, conceptually comparable
  to plugin names (e.g., `Zeek::Syslog`).

- `%doc-description = "text" provides additional text to go into the
  documentation (comparable to plugin descriptions).

This information is carried through into the HLTO runtime
initialization code, from where it's registered with Zeekygen.

This commit also removes a couple of previous hacks of how Spicy
integrated with Zeekygen which (1) ended up generating broken doc output
for Spicy components, and (2) don't seem to be necessary anymore
anyways.
2023-09-21 10:54:02 +02:00
Robin Sommer
7544aedb6a
Fix lack of namespace qualification inside an include file. 2023-09-21 10:54:02 +02:00
Robin Sommer
94de54ee74
Exclude script from documentation that interferes with other locations. 2023-09-21 10:54:02 +02:00
Robin Sommer
a9de8eec21
Merge remote-tracking branch 'origin/topic/robin/gh-3298-bitfields'
* origin/topic/robin/gh-3298-bitfields:
  Spicy: Fix support for exporting bitfields.
  Bump Spicy.
2023-09-21 09:56:51 +02:00
Robin Sommer
f31aa0580a
Spicy: Fix support for exporting bitfields.
Anonymous bitfields now have their fields lifted into the surrounding
record.

Closes #3298.
2023-09-21 09:55:12 +02:00
Robin Sommer
7233498cdb
Bump Spicy. 2023-09-21 09:55:12 +02:00
Benjamin Bannier
0b940d8b3c Merge branch 'topic/bbannier/spicyz-export-with-debug-output' 2023-09-20 13:05:04 +02:00
Benjamin Bannier
df62ae1f7d Updating submodule(s) [nomail] 2023-09-20 12:47:43 +02:00
Arne Welzel
4a7f03ee0c Merge remote-tracking branch 'origin/topic/awelzel/ci-benchmark-more-parameters'
* origin/topic/awelzel/ci-benchmark-more-parameters:
  ci/benchmark: Send more Cirrus, GitHub and Zeek version information
2023-09-19 19:56:54 +02:00
Arne Welzel
7b28152a92 ci/benchmark: Send more Cirrus, GitHub and Zeek version information
I wonder if we hit a query string limitation and should instead switch
to post body here. It's the `curl -G -X POST` part making this a POST
with a long query string.
2023-09-19 19:10:09 +02:00
Tim Wojtulewicz
e6ffa41aad Merge remote-tracking branch 'origin/topic/timw/telemetry-includes'
* origin/topic/timw/telemetry-includes:
  Remove some unnecessary includes of telemetry headers
2023-09-18 10:46:44 -07:00
Benjamin Bannier
3e91fd3115 Correctly compute name for spicyz export with debug log.
In 36a6770e98 we changed the way Spicy
fields exported as
record fields are represented. This broke the `zeek` debug log which
hardcoded a different representation.

This patch brings the generation of the debug log in line with the
actual AST structure. We also enable debug logging in one test to
validate that it works, at least for the case we hit here.
2023-09-18 16:05:42 +02:00
Tim Wojtulewicz
10c4f99d50 Remove some unnecessary includes of telemetry headers 2023-09-15 13:08:06 -07:00
Arne Welzel
2c4f64bc15 Merge remote-tracking branch 'origin/topic/awelzel/zeekjs-0-9-5'
* origin/topic/awelzel/zeekjs-0-9-5:
  Update zeekjs submodule
2023-09-15 12:11:32 +02:00
Arne Welzel
4f9b974e04 Merge remote-tracking branch 'origin/topic/awelzel/cmake-spicy-python3-hint'
* origin/topic/awelzel/cmake-spicy-python3-hint:
  CMakeLists: Set Python3_EXECUTABLE hint for spicy
2023-09-15 11:58:23 +02:00
Arne Welzel
3b859f3def Merge remote-tracking branch 'origin/topic/christian/gh-3263-table-constructor-index'
* origin/topic/christian/gh-3263-table-constructor-index:
  Avoid assertion/tag error on invalid table constructor index expression
2023-09-15 11:57:22 +02:00