Commit graph

14988 commits

Author SHA1 Message Date
Robin Sommer
0040111955
Integrate the Spicy plugin into Zeek proper.
This reflects the `spicy-plugin` code as of `d8c296b81cc2a11`.

In addition to moving the code into Zeek's source tree, this comes
with a couple small functional changes:

- `spicyz` no longer tries to infer if it's running from the build
  directory. Instead `ZEEK_SPICY_LIBRARY` can be set to a custom
  location. `zeek-set-path.sh` does that now.

- ZEEK_CONFIG can be set to change what `spicyz -z` print out. This is
  primarily for backwards compatibility.

Some further notes on specifics:

- We raise the minimum Spicy version to 1.8 (i.e., current `main`
  branch).

- Renamed the `compiler/` subdirectory to `spicyz` to avoid
  include-path conflicts with the Spicy headers.

- In `cmake/`, the corresponding PR brings a new/extended version of
  `FindZeek`, which Spicy analyzer packages need. We also now install
  some of the files that the Spicy plugin used to bring for testing,
  so that existing packages keep working.

- For now, this all remains backwards compatible with the current
  `zkg` analyzer templates so that they work with both external and
  integrated Spicy support. Later, once we don't need to support any
  external Spicy plugin versions anymore, we can clean up the
  templates as well.

- All the plugin's tests have moved into the standard test suite. They
  are skipped if configure with `--disable-spicy`.

This holds off on adapting the new code further to Zeek's coding
conventions, so that it remains easier to maintain it in parallel to
the (now legacy) external plugin. We'll make a pass over the
formatting for (presumable) Zeek 6.1.
2023-05-16 10:17:45 +02:00
Robin Sommer
d8f7329227
Modernize plugin test.
Not using the `zeek/` include style could lead to path problems.
2023-05-15 16:08:47 +02:00
zeek-bot
fe8c2c69b5 Update doc submodule [nomail] [skip ci] 2023-05-13 00:41:08 +00:00
Arne Welzel
cfd239ad2c Merge remote-tracking branch 'origin/topic/awelzel/zeek-seed-env-2'
* origin/topic/awelzel/zeek-seed-env-2:
  Introduce ZEEK_SEED_VALUES environment variable
2023-05-12 23:45:47 +02:00
Tim Wojtulewicz
012e33426a Merge remote-tracking branch 'origin/topic/vern/event-tracing-update.May23B'
* origin/topic/vern/event-tracing-update.May23B:
  enhancements for event-tracing:   - reporting of potentially sensitive constants   - tracking of unsupported types enabling hand-editing to fix them   - fixed generation of "unspecified" aggregates   - fixed generation of IPv6 constants   - fixed generation when running without a packet source
2023-05-12 13:42:15 -07:00
Vern Paxson
1419803dbd enhancements for event-tracing:
- reporting of potentially sensitive constants
  - tracking of unsupported types enabling hand-editing to fix them
  - fixed generation of "unspecified" aggregates
  - fixed generation of IPv6 constants
  - fixed generation when running without a packet source
2023-05-12 13:17:18 -07:00
Tim Wojtulewicz
db00835797 Merge remote-tracking branch 'origin/topic/timw/update-broker'
* origin/topic/timw/update-broker:
  Update broker submodule [nomail]
2023-05-12 13:00:14 -07:00
Tim Wojtulewicz
eb5de051e3 Update broker submodule [nomail] 2023-05-12 12:44:31 -07:00
Tim Wojtulewicz
eba7f2c01e Update broker submodule [nomail] 2023-05-12 11:32:59 -07:00
Arne Welzel
b2d934226f Introduce ZEEK_SEED_VALUES environment variable
For "individually different but deterministic" runs specifying Zeek's
seed as an environment variable eases setups as one can avoid creating
extra seed files for each of the individual processes.

It is an error to specify the new ZEEK_SEED_VALUES variable together
with the existing ZEEK_SEED_FILE and -G. ZEEK_SEED takes precedence over
deterministic mode (-D) like ZEEK_SEED_FILE does today already.
2023-05-12 19:50:37 +02:00
Tim Wojtulewicz
b98ae9bdb2 Merge remote-tracking branch 'amazingpp/topic/fupeng/bad-pat-mem-leak'
* amazingpp/topic/fupeng/bad-pat-mem-leak:
  Fix memory leak caused by pattern compilation failure
2023-05-12 10:02:00 -07:00
zeek-bot
e56c7c9ac2 Update doc submodule [nomail] [skip ci] 2023-05-12 00:32:20 +00:00
Christian Kreibich
d305e09c9e Updating auxil/zeek-client submodule [nomail] 2023-05-11 13:26:31 -07:00
Johanna Amann
3d257e52e2 Merge branch 'topic/johanna/dtls13'
* topic/johanna/dtls13:
  DTLS1.3 - address review feedback
  DTLS 1.3: Update test baseline
  DTLS 1.3: finish implementation, add connection_id extension
  Add basic DTLSv1.3 support
  SSL: Fix logging of Hello Retry Requests
2023-05-11 15:04:26 +01:00
Arne Welzel
03be610684 remote_event_ts_compat: Fix NETWORK_TIMESTAMP to NetworkTimestamp 2023-05-11 14:58:37 +02:00
Arne Welzel
11776d60e0 Merge remote-tracking branch 'jgras/topic/jgras/event-ts'
* jgras/topic/jgras/event-ts:
  Add compatibility tests for timestamped events.
  Add timestamps to auto published broker events.
  Add timestamps to manually published broker events.
  Annotate scheduled events with intended timestamp.
  Add timestamp to events.

One timestamp to ts rename during the merge.
2023-05-11 14:02:08 +02:00
Jan Grashoefer
1780d4cc2f Add compatibility tests for timestamped events.
This adds compatibility tests for receiving non-timestamped events as
well as providing timestamps via broker websockets.

Co-authored-by: Arne Welzel <arne.welzel@corelight.com>
2023-05-11 12:54:11 +02:00
Jan Grashoefer
1e807a9f0a Add timestamps to auto published broker events. 2023-05-11 12:54:11 +02:00
Jan Grashoefer
c12640b048 Add timestamps to manually published broker events. 2023-05-11 12:53:52 +02:00
Jan Grashoefer
ae152f1777 Annotate scheduled events with intended timestamp. 2023-05-11 12:51:06 +02:00
Jan Grashoefer
fa84d2896a Add timestamp to events. 2023-05-11 12:51:06 +02:00
Johanna Amann
67f6abf908 DTLS1.3 - address review feedback
See GH-3036
2023-05-11 11:19:35 +01:00
zeek-bot
4d97316a6d Update doc submodule [nomail] [skip ci] 2023-05-11 00:44:56 +00:00
Arne Welzel
31d5c15573 Merge remote-tracking branch 'origin/topic/awelzel/community-id-logging-connection-state-remove'
* origin/topic/awelzel/community-id-logging-connection-state-remove:
  Bump external test suites
  policy/community-id: Do not use new_connection()
2023-05-11 00:01:58 +02:00
Arne Welzel
8c0e7cd8bc Merge remote-tracking branch 'origin/topic/neverlord/version-header'
* origin/topic/neverlord/version-header:
  testing/zeek-version-link: Assume nm is there
  Drop dependency for zeek_dynamic_plugin_base
  Fixup ifdef check in Plugin.h
  Update cmake_minimum_required() in test plugins
  testing: Add zeek-version-link tests
  Clean up ZEEK_CONFIG_SKIP_VERSION_H issues

cmake-format during merge
2023-05-10 23:33:11 +02:00
Arne Welzel
19a2ba6986 Bump external test suites 2023-05-10 19:24:01 +02:00
Arne Welzel
e82a7455d7 policy/community-id: Do not use new_connection()
Issue #3028 tracks how a flipped connections reset a connection's value
including any state set during new_connection(). For the time being,
update community-id functionality back to the original connection_state_remove()
approach to avoid missing community_ids on flipped connections.
2023-05-10 17:48:33 +02:00
Arne Welzel
ca75834c83 testing/zeek-version-link: Assume nm is there 2023-05-10 17:37:32 +02:00
Dominik Charousset
879f605407 Drop dependency for zeek_dynamic_plugin_base 2023-05-10 16:59:07 +02:00
Arne Welzel
8f45313672 Fixup ifdef check in Plugin.h 2023-05-10 15:53:13 +02:00
Arne Welzel
85934ec632 Update cmake_minimum_required() in test plugins 2023-05-10 15:53:13 +02:00
Arne Welzel
460dc16d32 testing: Add zeek-version-link tests 2023-05-10 15:53:10 +02:00
Johanna Amann
33ee2bffa9 DTLS 1.3: Update test baseline 2023-05-10 12:27:27 +00:00
Johanna Amann
527c0dc09f Merge remote-tracking branch 'origin/master' into topic/johanna/dtls13
* origin/master: (35 commits)
  Update doc submodule [nomail] [skip ci]
  Updating submodule(s) [nomail]
  zeek.bif: Add log2() and ceil()
  Use the same rules as cmake submodule to reformat Zeek
  Update cmake submodule after reformat
  Fixup Val.h/Val.cc: Actually move ValFromJSON into zeek::detail
  Implement from_json bif
  Revert "Skip version.h by default for Zeek sources"
  BTest baseline updates for -O gen-C++
  updates to C++ maintenance scripts to better handle uncompilable BTests
  added ZEEK_REPORT_UNCOMPILABLE environment variable for "-O report-uncompilable"
  Skip version.h by default for Zeek sources
  core.network_time.broker: Test reliability improvement
  cluster/supervisor: Multi-logger awareness
  Bump zeek-archiver submodule
  ci: Add public-ecr-vacuum.sh
  Update doc submodule [nomail] [skip ci]
  generate-docs: Only update submodule pointer during scheduled builds
  BTest baseline updates for ZAM
  NTP: Detect out-of-order packets
  ...
2023-05-10 13:02:08 +01:00
Johanna Amann
a8e84c6192 DTLS 1.3: finish implementation, add connection_id extension
This commit adds support for the connection_id extension, adds a trace
that uses DTLS 1.3 connection IDs, and adds parsing for the DTLS 1.3
unified header, in case connection IDs are not used.

In case connection IDs are used, parsing of the DTLS 1.3 unified header
is skipped. This is due to the fact, that the header then contains a
variable length element, with the length of the element not given in the
header. Instead, the length is given in the client/server hello message
of the opposite side of the connection (which we might have missed).

Furthermore, parsing is not of a high importance, since we are not
passing the connection ID, or any of the other parsed values of the
unified header into scriptland.
2023-05-10 11:17:24 +01:00
mAsk°
e08e4a5fee Fix memory leak caused by pattern compilation failure 2023-05-10 05:42:03 +00:00
zeek-bot
c543387ce0 Update doc submodule [nomail] [skip ci] 2023-05-10 00:37:23 +00:00
Tim Wojtulewicz
04d23a8b5b Merge remote-tracking branch 'origin/topic/awelzel/3013-btest-core-network-time-broker'
* origin/topic/awelzel/3013-btest-core-network-time-broker:
  core.network_time.broker: Test reliability improvement
2023-05-09 13:38:52 -07:00
Tim Wojtulewicz
9d46db484f Merge remote-tracking branch 'origin/topic/awelzel/3005-public-ecr-vacuum'
* origin/topic/awelzel/3005-public-ecr-vacuum:
  ci: Add public-ecr-vacuum.sh
2023-05-09 13:38:25 -07:00
Tim Wojtulewicz
51b7a81926 Merge remote-tracking branch 'origin/topic/timw/update-submodule-pointers'
* origin/topic/timw/update-submodule-pointers:
  Updating submodule(s) [nomail]
2023-05-09 12:54:36 -07:00
Tim Wojtulewicz
2a587ad9c9 Updating submodule(s) [nomail] 2023-05-09 12:52:28 -07:00
Arne Welzel
ef2b30ba34 Merge remote-tracking branch 'origin/topic/awelzel/2930-2931-add-log2-and-ceil'
* origin/topic/awelzel/2930-2931-add-log2-and-ceil:
  zeek.bif: Add log2() and ceil()
2023-05-09 19:11:35 +02:00
Arne Welzel
6cb391f9a2 zeek.bif: Add log2() and ceil()
Closes #2930, #2931.
2023-05-09 18:57:57 +02:00
Tim Wojtulewicz
c0d15d6d0e Merge remote-tracking branch 'origin/topic/timw/cmake-format'
* origin/topic/timw/cmake-format:
  Use the same rules as cmake submodule to reformat Zeek
  Update cmake submodule after reformat
2023-05-09 09:06:00 -07:00
Tim Wojtulewicz
84e3f414a7 Use the same rules as cmake submodule to reformat Zeek 2023-05-09 08:31:43 -07:00
Tim Wojtulewicz
fd72d81bad Update cmake submodule after reformat 2023-05-09 08:31:43 -07:00
Arne Welzel
9330a74fe1 Merge remote-tracking branch 'origin/topic/awelzel/zeek-archiver-multiple-loggers'
* origin/topic/awelzel/zeek-archiver-multiple-loggers:
  cluster/supervisor: Multi-logger awareness
  Bump zeek-archiver submodule
2023-05-09 15:20:53 +02:00
Arne Welzel
1facc34e09 Fixup Val.h/Val.cc: Actually move ValFromJSON into zeek::detail
Lost during merge..
2023-05-09 11:23:32 +02:00
Arne Welzel
264284150b Merge remote-tracking branch 'amazing-pp/topic/fupeng/from_json_bif'
* amazing-pp/topic/fupeng/from_json_bif:
  Implement from_json bif

Minor updates during merge: Moved ValFromJSON into zeek::detail for the
time being, removed gotos, normalized some error messages to lower case,
minimal test extension and added a raw reader input framework test reading
"json lines" as a demo, adding notes about the implicit type
conversions.
2023-05-09 10:36:58 +02:00
Dominik Charousset
986834dce7 Clean up ZEEK_CONFIG_SKIP_VERSION_H issues 2023-05-08 18:05:56 +02:00