Commit graph

16055 commits

Author SHA1 Message Date
Tim Wojtulewicz
4b69baa522 Add call stacks to script profiler output 2023-04-19 08:45:17 -07:00
Arne Welzel
ed169b537c file_analysis/Manager: Remove RunState.h include
The file_analysis/Manager.h header only needs run_state::terminating, so
declare just that similar to what Val.h does. This breaks ZBody compilation
due to OP_READING_LIVE_TRAFFIC_V and OP_READING_TRACES_V accessing
run-state::reading_live and run_state::reading_traces. Add the include to
ZBody.cc as a fix.

This may break external plugins, too, but then they didn't have the
right includes setup to begin with.
2023-04-19 15:53:33 +02:00
Arne Welzel
89c828ac14 Merge remote-tracking branch 'origin/topic/vern/record-optimizations.Apr23B'
* origin/topic/vern/record-optimizations.Apr23B:
  different fix for MSVC compiler issues
  more general approach for addressing MSVC compiler issues with IntrusivePtr
  restored RecordType::Create, now marked as deprecated tidying of namespaces and private class members simplification of flagging record field initializations that should be skipped address peculiar MSVC compilation complaint for IntrusivePtr's
  clarifications and tidying for record field initializations
  optimize record construction by deferring initializations of aggregates
  compile-scripts-to-C++ speedups by switching to raw record access
  logging speedup by switching to raw record access
  remove redundant record coercions

Removed the `#if 0` hunk during merging: Probably could have gone with a
doctest instead.
2023-04-19 11:59:56 +02:00
Arne Welzel
683cb80f61 Merge remote-tracking branch 'origin/topic/neverlord/gh-2953'
* origin/topic/neverlord/gh-2953:
  Improve CMake variables, update cmake submodule
  Fix builds with plugins that use zeek-config

Bumped cmake and used update-changes with explicit -p because
it wasn't detecting the right commit to start with. Suspect
something went off with the last bump.
2023-04-19 10:14:33 +02:00
Vern Paxson
11dc4c8ce9 different fix for MSVC compiler issues 2023-04-18 17:12:52 -07:00
Vern Paxson
1f61afa56b more general approach for addressing MSVC compiler issues with IntrusivePtr 2023-04-18 15:48:22 -07:00
Dominik Charousset
12ea6ed478 Improve CMake variables, update cmake submodule 2023-04-18 21:17:33 +02:00
Tim Wojtulewicz
f222c15374 Merge remote-tracking branch 'origin/topic/awelzel/2885-log-caching-cluster-flaky'
* origin/topic/awelzel/2885-log-caching-cluster-flaky:
  log-caching-cluster: Wait for X509::known_log_certs to populate
2023-04-18 11:11:53 -07:00
Vern Paxson
c19eba62d6 restored RecordType::Create, now marked as deprecated
tidying of namespaces and private class members
simplification of flagging record field initializations that should be skipped
address peculiar MSVC compilation complaint for IntrusivePtr's
2023-04-18 09:41:45 -07:00
Arne Welzel
4bcf5f4b76 NEWS: Add entry for ZeekControl and multi-loggers 2023-04-18 15:41:46 +02:00
Arne Welzel
1b1d4c47c7 Bump zeekctl to multi-logger version 2023-04-18 15:41:36 +02:00
Arne Welzel
a9a37c865e log-caching-cluster: Wait for X509::known_log_certs to populate
The known_log_certs table is populated asynchronously via broker after a
Broker::peer_added. It may take a variable amount of time depending on where
we run this test and it has been observed flaky specifically for the
arm_debian11 task. Instead of racing, give worker-2 3 seconds for receiving
the expected table content before continuing.

Fixes #2885
2023-04-18 15:19:12 +02:00
Arne Welzel
d89f16dfc9 logging: Support rotation_postprocessor_command_env
This new table provides a mechanism to add environment variables to the
postprocessor execution. Use case is from ZeekControl to inject a suffix
to be used when running with multiple logger.
2023-04-17 13:10:14 +00:00
Vern Paxson
ee358affda clarifications and tidying for record field initializations 2023-04-15 20:12:49 -07:00
Dominik Charousset
1d81fb4fb0 Fix builds with plugins that use zeek-config
When building plugins externally with `zeek-config` (as opposed to using
`ZEEK_DIST`), they point into the install prefix. There, we add a new
file `ZeekPluginBootstrap.cmake` that helps `ZeekPlugin.cmake` to find
everything else it needs from there.

Our template for plugins generates a `configure` script that sets
various variables with values from `zeek-config`. We only need
`BROKER_ROOT_DIR` with the new bootstrapping logic. Everything else, we
can get from the new bootstrapping file and from the CMake package file
for Zeek.
2023-04-15 00:08:42 +02:00
Arne Welzel
75245bd365 Merge remote-tracking branch 'origin/topic/awelzel/javascript'
* origin/topic/awelzel/javascript:
  ci/debian-11: Install libnode-dev, too
  CMakeLists: Convert string append to list append
  Add experimental JavaScript support when libnode is available
2023-04-14 13:02:57 +02:00
Arne Welzel
193352bb8c ci/debian-11: Install libnode-dev, too
Debian 11 doesn't have a new enough libnode version, so JavaScript
should not be attempted to be built.
2023-04-14 11:26:47 +02:00
Arne Welzel
832291fd92 CMakeLists: Convert string append to list append
Seems the builtin plugins started with string(APPEND ...) and that
was copied over. Make it list(APPEND ...) instead.
2023-04-14 11:26:47 +02:00
Arne Welzel
f0b9c59adb Add experimental JavaScript support when libnode is available
zeek.on('zeek_init', () => {
        console.log('Hello, Zeek!');
    });

For interaction with external systems and HTTP APIs, JavaScript and the
Node.js ecosystem beat Zeek script. Make it more easily accessible by
including ZeekJS with Zeek directly.

When a recent enough libnode version is found on the build system, ZeekJS is
added as a builtin plugin. This behavior can be disabled via
``--disable-javascript``. Linux distributions providing such a package are
Ubuntu (22.10) and Debian (testing/bookworm) as libnode-dev.
Fedora provides it as nodejs-devel.

This plugin takes over loading of .js or .cjs files. When no such files
are provided to Zeek, Node and the V8 engine are not initialized and
should not get into the way.

This should be considered experimental.
2023-04-14 11:26:41 +02:00
Arne Welzel
5db2e5fd8e Merge remote-tracking branch 'origin/topic/awelzel/revert-type-manager-intro'
* origin/topic/awelzel/revert-type-manager-intro:
  Revert "Type: Add TypeManager->TypeList() and use for ListVal()"
2023-04-14 10:33:01 +02:00
Arne Welzel
a0540f96a1 Revert "Type: Add TypeManager->TypeList() and use for ListVal()"
This reverts commit 24c606b4df.

This commit introduced a memory leak ListVal::Append() modifying
the cached TYPE_ANY type list.
2023-04-14 09:49:05 +02:00
zeek-bot
10f96aeba3 Update doc submodule [nomail] [skip ci] 2023-04-14 00:37:16 +00:00
Tim Wojtulewicz
13acd45383 Merge remote-tracking branch 'origin/topic/neverlord/cmake-redesign-phase1'
* origin/topic/neverlord/cmake-redesign-phase1:
  Add NEWS entry for the CMake changes
  Redesign subdir-libs and plugin scaffolding
2023-04-13 15:54:12 -07:00
Dominik Charousset
2704934267 Add NEWS entry for the CMake changes 2023-04-13 22:31:05 +02:00
Dominik Charousset
9aeed5284d Redesign subdir-libs and plugin scaffolding 2023-04-13 22:23:46 +02:00
Tim Wojtulewicz
ca43ada578 Merge remote-tracking branch 'origin/topic/timw/2920-initialized-container-options'
* origin/topic/timw/2920-initialized-container-options:
  GH-2920: Don't warn on uninitialized container options
2023-04-13 09:05:40 -07:00
Tim Wojtulewicz
fcf7af259c GH-2920: Don't warn on uninitialized container options 2023-04-13 09:05:17 -07:00
Tim Wojtulewicz
c5ce82143a Merge remote-tracking branch 'origin/topic/timw/2947-input-config-regex'
* origin/topic/timw/2947-input-config-regex:
  Reimplement fixes to handle commas at the end of config parser lines
  Revert "Convert config framework to use std::regex"
  Revert "GH-636: Fix regex to handle commas at the end of config parser lines"
2023-04-13 08:57:30 -07:00
Arne Welzel
b739bdbbfb Merge remote-tracking branch 'origin/topic/awelzel/rotation-format-func-segfault'
* origin/topic/awelzel/rotation-format-func-segfault:
  logging/Manager: Fix crash for rotation format function not returning
2023-04-13 09:24:47 +02:00
Arne Welzel
a5e7faf564 logging/Manager: Fix crash for rotation format function not returning
While working on a rotation format function, ran into Zeek crashing
when not returning a value from it, fix and recover the same way as
for scripting errors.
2023-04-13 09:23:51 +02:00
Tim Wojtulewicz
187773eaa3 Reimplement fixes to handle commas at the end of config parser lines 2023-04-12 16:26:01 -07:00
Tim Wojtulewicz
777dff18cd Revert "Convert config framework to use std::regex"
This reverts commit 65ee2287e9.
2023-04-12 16:05:46 -07:00
Tim Wojtulewicz
a811722d72 Revert "GH-636: Fix regex to handle commas at the end of config parser lines"
This reverts commit 05bb509789.
2023-04-12 16:05:39 -07:00
Tim Wojtulewicz
6ac72a31bf Merge remote-tracking branch 'origin/topic/arne.welzel/fix-poll-interval-is-live'
* origin/topic/arne.welzel/fix-poll-interval-is-live:
  iosource/Manager: Fix poll_interval updating using not-yet valid IsLive()
2023-04-12 13:23:46 -07:00
zeek-bot
384bc9eecb Update doc submodule [nomail] [skip ci] 2023-04-12 00:34:45 +00:00
Tim Wojtulewicz
149ad57742 Merge remote-tracking branch 'origin/topic/timw/silence-noisy-compiler-flag'
* origin/topic/timw/silence-noisy-compiler-flag:
  Only use -Wno-changes-meaning if it exists
2023-04-11 15:41:46 -07:00
Tim Wojtulewicz
d8c1a1babf Merge remote-tracking branch 'security/topic/awelzel/155-reassem-validate-seq-upper-overflow'
* security/topic/awelzel/155-reassem-validate-seq-upper-overflow:
  file_analysis/File: Report overflowing chunks as weird and discard/truncate
  Reassem: Reject blocks overflowing 64bit upper
  zeek-setup: Load scrips before running unit tests
2023-04-11 15:30:58 -07:00
Tim Wojtulewicz
f812ce53cf Merge remote-tracking branch 'security/topic/timw/154-rdp-timeout'
* security/topic/timw/154-rdp-timeout:
  RDP: Instantiate SSL analyzer instead of PIA
  RDP: add some enforcement to required values based on MS-RDPBCGR docs
2023-04-11 15:23:52 -07:00
Tim Wojtulewicz
0cc1d672b5 Merge remote-tracking branch 'security/topic/timw/150-dns-expiration'
* security/topic/timw/150-dns-expiration:
  Stop skipping expiration of empty DNS mappings
2023-04-11 15:16:47 -07:00
Tim Wojtulewicz
f701f1fc94 Merge remote-tracking branch 'security/topic/awelzel/152-smtp-validate-mail-transactions'
* security/topic/awelzel/152-smtp-validate-mail-transactions:
  smtp: Validate mail transaction and disable SMTP analyzer if excessive
  generic-analyzer-fuzzer: Detect disable_analyzer() from scripts
2023-04-11 15:16:25 -07:00
Tim Wojtulewicz
c670f3fdb2 Merge remote-tracking branch 'security/topic/awelzel/148-ftp-skip-get-pending-commands-multi-line-response'
* security/topic/awelzel/148-ftp-skip-get-pending-commands-multi-line-response:
  ftp/main: Special case for intermediate reply lines
  ftp/main: Skip get_pending_command() for intermediate reply lines
2023-04-11 14:50:55 -07:00
Tim Wojtulewicz
a7a895b178 Only use -Wno-changes-meaning if it exists 2023-04-11 14:35:56 -07:00
Tim Wojtulewicz
d1eb0f47fd Merge remote-tracking branch 'origin/topic/timw/update-broker-for-gcc-13'
* origin/topic/timw/update-broker-for-gcc-13:
  Add cstdint to WeirdState.h to fix compilation error on gcc13
  Ignore -Wchanges-meaning warning in Spicy
  Update broker submodule to pull in GCC 13 fix
  CI: Force rebuild of OpenSUSE Tumbleweed VM to pick up GCC 13
2023-04-11 12:05:48 -07:00
Tim Wojtulewicz
122b8d6052 Add cstdint to WeirdState.h to fix compilation error on gcc13 2023-04-11 11:55:34 -07:00
Tim Wojtulewicz
ca35c85983 Ignore -Wchanges-meaning warning in Spicy
This fixes a build failure with GCC 13. It's meant as a short-term fix to avoid
the failure in Zeek, but probably should be fixed more correctly upstream in
the Spicy repo.
2023-04-11 11:55:34 -07:00
Tim Wojtulewicz
a0dbfd2822 Update broker submodule to pull in GCC 13 fix 2023-04-11 11:55:34 -07:00
Tim Wojtulewicz
505ac36e43 CI: Force rebuild of OpenSUSE Tumbleweed VM to pick up GCC 13 2023-04-11 11:55:34 -07:00
Vern Paxson
0787c130d0 optimize record construction by deferring initializations of aggregates 2023-04-10 11:44:11 -07:00
Vern Paxson
2e2afa5e11 compile-scripts-to-C++ speedups by switching to raw record access 2023-04-10 11:43:40 -07:00
Vern Paxson
4600ca41f6 logging speedup by switching to raw record access 2023-04-10 11:43:19 -07:00