Commit graph

14420 commits

Author SHA1 Message Date
Arne Welzel
2fdc740bd0 Merge remote-tracking branch 'origin/topic/timw/fix-cirrus-macos-builds'
* origin/topic/timw/fix-cirrus-macos-builds:
  CI: Call 'brew install' for macOS to ensure we're using openssl@3 on all hosts

(cherry picked from commit 6047846353)
2023-02-13 14:32:32 -07:00
Arne Welzel
dc4b8b958c cirrus/container: Polish, fix and automated pushing of images
* Add ci/find-current-version.sh to figure out versions of lts and
  feature branch based on repository state.

* Do not use ZEEK_IMAGE_REPO for the local images that
  are expected to exist.

* Only use IMAGE_NAME/IMAGE_TAG instead of the ARCH and MANIFEST
  versions that contained the very same values

* Support and push additional tags

  In addition to the lts and feature tags, also push the base version
  tags. 5.0 or 5.1 such that someone pulling zeek/zeek:5.0 will get
  the most recent available 5.0 patch release.

(cherry picked from commit 357bdc71ad)
2023-02-13 13:46:02 -07:00
Christian Kreibich
bf8addc5b9 Update doc submodule [nomail] [skip ci] 2023-02-08 12:00:00 -08:00
Christian Kreibich
d8a28b8df2 Cover zeek-client updates in NEWS [skip ci]
(cherry picked from commit 6c0d44436c)
2023-02-08 11:25:37 -08:00
Tim Wojtulewicz
54b816f91e Updating CHANGES and VERSION. 2023-02-03 10:05:53 -07:00
Tim Wojtulewicz
e008ab1b71 Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy-plugin'
* origin/topic/bbannier/bump-spicy-plugin:
  Bump spicy-plugin to latest development snapshot.
2023-02-03 09:05:26 -07:00
Robin Sommer
e7e041be32
Merge remote-tracking branch 'origin/topic/robin/spicy-availability-check'
* origin/topic/robin/spicy-availability-check:
  Add BIF `have_spicy_analyzers()`.
2023-02-03 16:49:06 +01:00
Tim Wojtulewicz
96f82ea74d Update package-manager submodule [nomail] 2023-02-03 08:38:27 -07:00
Benjamin Bannier
811bdcdf53 Bump spicy-plugin to latest development snapshot. 2023-02-03 16:36:02 +01:00
Robin Sommer
bc252c63dc
Add BIF have_spicy_analyzers().
We previously used the Spicy plugin's `Spicy::available` to test for
Spicy support. However, having Spicy support does not necessarily mean that we
have built Zeek with its in-tree Spicy analyzers: the Spicy plugin
could have been pulled in from external. The new BIF now reliably
tells us whether the Spicy analyzers are available; its result
corresponds to what `zeek-config --have-spicy-analyzers` returns as
well.

We also move the two current checks over to use this BIF.

(Note: I refrained from renaming the CMake-side `USE_SPICY_ANALYERS`
to `HAVE_SPICY_ANALYZERS`. We should do this eventually for
consistency, but I didn't want to make more changes than necessary
right now.)
2023-02-03 13:47:26 +01:00
Arne Welzel
39f0b78043 Merge remote-tracking branch 'origin/topic/awelzel/no-binary-in-baselines'
* origin/topic/awelzel/no-binary-in-baselines:
  btest/decode_base_errors: Avoid binary output in baseline
2023-02-02 20:20:49 +01:00
Arne Welzel
4ac65ce8ca btest/decode_base_errors: Avoid binary output in baseline
Zeek prints the input string verbatim to the terminal and that's
upsetting btest just on Alpine.
2023-02-02 18:49:00 +01:00
Arne Welzel
622074108b Merge branch 'topic/awelzel/topic/no-container-images-on-branch-push-only'
* topic/awelzel/topic/no-container-images-on-branch-push-only:
  cirrus: No container images builds or cluster testing on branches
2023-02-02 18:36:16 +01:00
Arne Welzel
b6df7773a9 Merge remote-tracking branch 'origin/topic/awelzel/arm64-container-follow-ups-2'
* origin/topic/awelzel/arm64-container-follow-ups-2:
  Bump private testsuite for char handling fixes
  Base64: report byte as positive integer
  netbios_decode: use unsigned char for result
  analyzer/http: Do not assume char is signed
  cirrus: Run tests in a Debian 11 container, too
2023-02-02 18:35:02 +01:00
Arne Welzel
876bd3e974 cirrus: No container images builds or cluster testing on branches 2023-02-02 17:04:34 +01:00
Tim Wojtulewicz
916ffd25eb Updating submodule(s) [nomail] 2023-02-02 08:44:25 -07:00
Tim Wojtulewicz
335d376396 Updating submodule(s) [nomail] 2023-02-02 08:28:47 -07:00
Arne Welzel
bbe0a86d4a Bump private testsuite for char handling fixes 2023-02-02 16:24:41 +01:00
Arne Welzel
5dc54fb40e Base64: report byte as positive integer
A baseline difference between arm64 and x86 showed up. We would
print a wrong character as negative value on x86 due to chars
being signed by default. Force an unsigned interpretation which
is also more reasonable because we'd have never indexed the
base64 table with -112

    -XXXXXXXXXX.XXXXXX      XXXXXXXXXXX     131.243.99.154  3288 193.159.183.138 80      base64_illegal_encoding character -112 ignored by Base64 decoding       F       zeek    -
    +XXXXXXXXXX.XXXXXX      XXXXXXXXXXX     131.243.99.154  3288 193.159.183.138 80      base64_illegal_encoding character 144 ignored by Base64 decoding        F       zeek    -

Fixes more of #2742
2023-02-02 15:49:22 +01:00
Arne Welzel
c998cf697a netbios_decode: use unsigned char for result
The buf[i] < 3 condition in use previously allowed all chars (signed on x86)
through that had the 0x80 high-bit set after reconstructing from the two
bytes of the netbios name, resulting in escaped non-ascii content in
the logs.

Fixes more of #2742
2023-02-02 15:48:58 +01:00
Arne Welzel
71bcd15d2e analyzer/http: Do not assume char is signed
On aarch64, char is unsigned, so is_HTTP_token_char() allowed
non-ASCII stuff with the high-bit set.

Fixes part of #2742
2023-02-02 14:57:57 +01:00
Arne Welzel
e762efc9af cirrus: Run tests in a Debian 11 container, too 2023-02-02 14:57:57 +01:00
Arne Welzel
989f1df88c cirrus: There's no zeekurity repository on AWS ECR 2023-02-02 12:53:13 +01:00
Arne Welzel
c834ba96ca Merge remote-tracking branch 'origin/topic/awelzel/continue-to-push-container-images-to-zeekurity'
* origin/topic/awelzel/continue-to-push-container-images-to-zeekurity:
  cirrus: Also push images to the zeekurity account
2023-02-02 11:25:48 +01:00
Arne Welzel
846f1e4496 cirrus: Also push images to the zeekurity account
This was mistakenly removed during f9b0681c98.
2023-02-02 11:19:16 +01:00
Christian Kreibich
ca6374a648 Merge branch 'topic/christian/management-fix-node-state-baseline-race'
* topic/christian/management-fix-node-state-baseline-race:
  Bump cluster testsuite to pull in fix for test baseline race condition
2023-02-01 16:38:53 -08:00
Christian Kreibich
542a1d8f4e Bump cluster testsuite to pull in fix for test baseline race condition 2023-02-01 16:38:19 -08:00
Christian Kreibich
0630d9a10a Update btest submodule to v1.0 2023-02-01 16:05:54 -08:00
Christian Kreibich
83daa3fdfe Merge branch 'topic/christian/hook-test-fixes'
* topic/christian/hook-test-fixes:
  Tighten the scripts.base.frameworks.logging.hooks test
  Fix two btest-diff checks that couldn't fail
2023-02-01 16:04:47 -08:00
Christian Kreibich
f8dbf70e3b Tighten the scripts.base.frameworks.logging.hooks test
This avoids interference from other log streams in the policy hook test cases,
which could cause deviations in output vs baselines depending on build
configuration.
2023-02-01 15:12:20 -08:00
Christian Kreibich
b5c8421ac2 Fix two btest-diff checks that couldn't fail
:-)
2023-02-01 15:12:20 -08:00
Tim Wojtulewicz
66cf5b9126 Merge remote-tracking branch 'origin/topic/timw/update-broker-submodule-for-pybind-bump'
* origin/topic/timw/update-broker-submodule-for-pybind-bump:
  Update broker submodule [nomail]
2023-02-01 15:34:32 -07:00
Tim Wojtulewicz
3bd837e8f4 Update broker submodule [nomail] 2023-02-01 15:33:57 -07:00
Arne Welzel
3ec602a441 Merge remote-tracking branch 'origin/topic/awelzel/arm64-container-follow-ups'
* origin/topic/awelzel/arm64-container-follow-ups:
  docker: Extend tests to run a very basic Zeek script
  cirrus: No broker-tests and no cpp-tests for images
  docker: Update Makefile/README to align with the Cirrus CI setup
  cirrus: Add back testing of container images
2023-02-01 20:56:05 +01:00
Arne Welzel
13b0e7a2aa docker: Extend tests to run a very basic Zeek script
Mostly because a `-v` shortcuts quite, while a zeek -e 'print zeek_version()'
will also parse all base scripts, etc.
2023-02-01 20:54:09 +01:00
Arne Welzel
7efd9ad1f5 cirrus: No broker-tests and no cpp-tests for images 2023-02-01 20:54:09 +01:00
Arne Welzel
4523f678d2 docker: Update Makefile/README to align with the Cirrus CI setup 2023-02-01 20:54:05 +01:00
Arne Welzel
44c19086e8 cirrus: Add back testing of container images 2023-02-01 20:54:05 +01:00
Tim Wojtulewicz
0fd335f7f0 Merge remote-tracking branch 'security/topic/timw/131-smb-fscontrol-overflow'
* security/topic/timw/131-smb-fscontrol-overflow:
  Restore/rename field in SMB2::Fscontrol record type
2023-02-01 10:48:16 -07:00
Tim Wojtulewicz
f33c697d6c Merge remote-tracking branch 'security/topic/awelzel/127-http-timeout-again'
* security/topic/awelzel/127-http-timeout-again:
  testing/http: http-11-request-then-cruft
  testing/http: Add pcap extracted from m5-long external test-suite
  testing/external: m57-long baseline update
  analyzers/http: Update request_version on subsequent SetVersion() calls
2023-02-01 10:47:54 -07:00
Tim Wojtulewicz
9a0dc30e35 Merge remote-tracking branch 'security/topic/awelzel/125-ftp-timeout-three'
* security/topic/awelzel/125-ftp-timeout-three:
  testing/ftp: Add tests and pcaps with invalid reply lines
  ftp: Harden reply handing a bit and don't raise bad replies to script-land
  ftp: ignore invalid commands
2023-02-01 10:47:32 -07:00
Robin Sommer
a2dfd3922f
Merge remote-tracking branch 'origin/topic/robin/zeek-spicy-infra'
* origin/topic/robin/zeek-spicy-infra:
  Switch to shared Spicy libraries by default.
  Provide infrastructure to migrate legacy analyzers to Spicy.
  Extend Zeekygen manager with method to check if it's active.
2023-02-01 11:48:27 +01:00
Robin Sommer
2512fd1e68
Switch to shared Spicy libraries by default.
This should work now. It affects only the toolchain libraries
`libhilti`/`libspicy`. the runtime libraries `libhilti-rt` and
`libspicy-rt` are always built static (but they are small). Zeek
itself doesn't link against the toolchain anymore now anyways, but a
number of the Spicy tools do.

Note, we have an issue with Broker I believe: it looks like it always
overrides BUILD_SHARED_LIBS to `OFF`

Addresses #2675.
2023-02-01 11:33:48 +01:00
Robin Sommer
04a1ead978
Provide infrastructure to migrate legacy analyzers to Spicy.
As initial examples, this branch ports the Syslog and Finger analyzers
over. We leave the old analyzers in place for now and activate them
iff we compile without any Spicy.

Needs `zeek-spicy-infra` branches in `spicy/`, `spicy-plugin/`,
`CMake/`, and `zeek/zeek-testing-private`.

Note that the analyzer events remain associated with the Spicy plugin
for now: that's where they will show up with `-NN`, and also inside
the Zeekygen documentation.

We switch CMake over to linking the runtime library into the plugin,
vs. at the top-level through object libraries.
2023-02-01 11:33:48 +01:00
Robin Sommer
283bea346b
Extend Zeekygen manager with method to check if it's active. 2023-02-01 11:31:33 +01:00
Arne Welzel
cfd9979044 Merge remote-tracking branch 'origin/topic/awelzel/2674-arm64-containers-on-cirrus'
* origin/topic/awelzel/2674-arm64-containers-on-cirrus:
  cirrus/containers: Do not install btest pcaps
  cirrus: Replace curl with _cache
  cirrus: Add cluster testing
  cirrus: Polish container_image_manifest_docker_builder
  cirrus: Use ccache for faster container builds
  cirrus: Add docker_builder tasks to build and push images
  github: No more docker workflow
2023-02-01 09:02:33 +01:00
Arne Welzel
20cc5541e0 cirrus/containers: Do not install btest pcaps 2023-02-01 08:52:05 +01:00
Arne Welzel
3f7adbe6b8 cirrus: Replace curl with _cache 2023-02-01 08:52:05 +01:00
Arne Welzel
5a611d4bdb cirrus: Add cluster testing 2023-02-01 08:52:05 +01:00
Arne Welzel
b5dc4c6617 cirrus: Polish container_image_manifest_docker_builder 2023-02-01 08:52:05 +01:00