Commit graph

14939 commits

Author SHA1 Message Date
Arne Welzel
193352bb8c ci/debian-11: Install libnode-dev, too
Debian 11 doesn't have a new enough libnode version, so JavaScript
should not be attempted to be built.
2023-04-14 11:26:47 +02:00
Arne Welzel
832291fd92 CMakeLists: Convert string append to list append
Seems the builtin plugins started with string(APPEND ...) and that
was copied over. Make it list(APPEND ...) instead.
2023-04-14 11:26:47 +02:00
Arne Welzel
f0b9c59adb Add experimental JavaScript support when libnode is available
zeek.on('zeek_init', () => {
        console.log('Hello, Zeek!');
    });

For interaction with external systems and HTTP APIs, JavaScript and the
Node.js ecosystem beat Zeek script. Make it more easily accessible by
including ZeekJS with Zeek directly.

When a recent enough libnode version is found on the build system, ZeekJS is
added as a builtin plugin. This behavior can be disabled via
``--disable-javascript``. Linux distributions providing such a package are
Ubuntu (22.10) and Debian (testing/bookworm) as libnode-dev.
Fedora provides it as nodejs-devel.

This plugin takes over loading of .js or .cjs files. When no such files
are provided to Zeek, Node and the V8 engine are not initialized and
should not get into the way.

This should be considered experimental.
2023-04-14 11:26:41 +02:00
Arne Welzel
5db2e5fd8e Merge remote-tracking branch 'origin/topic/awelzel/revert-type-manager-intro'
* origin/topic/awelzel/revert-type-manager-intro:
  Revert "Type: Add TypeManager->TypeList() and use for ListVal()"
2023-04-14 10:33:01 +02:00
Arne Welzel
a0540f96a1 Revert "Type: Add TypeManager->TypeList() and use for ListVal()"
This reverts commit 24c606b4df.

This commit introduced a memory leak ListVal::Append() modifying
the cached TYPE_ANY type list.
2023-04-14 09:49:05 +02:00
zeek-bot
10f96aeba3 Update doc submodule [nomail] [skip ci] 2023-04-14 00:37:16 +00:00
Tim Wojtulewicz
13acd45383 Merge remote-tracking branch 'origin/topic/neverlord/cmake-redesign-phase1'
* origin/topic/neverlord/cmake-redesign-phase1:
  Add NEWS entry for the CMake changes
  Redesign subdir-libs and plugin scaffolding
2023-04-13 15:54:12 -07:00
Dominik Charousset
2704934267 Add NEWS entry for the CMake changes 2023-04-13 22:31:05 +02:00
Dominik Charousset
9aeed5284d Redesign subdir-libs and plugin scaffolding 2023-04-13 22:23:46 +02:00
Tim Wojtulewicz
ca43ada578 Merge remote-tracking branch 'origin/topic/timw/2920-initialized-container-options'
* origin/topic/timw/2920-initialized-container-options:
  GH-2920: Don't warn on uninitialized container options
2023-04-13 09:05:40 -07:00
Tim Wojtulewicz
fcf7af259c GH-2920: Don't warn on uninitialized container options 2023-04-13 09:05:17 -07:00
Tim Wojtulewicz
c5ce82143a Merge remote-tracking branch 'origin/topic/timw/2947-input-config-regex'
* origin/topic/timw/2947-input-config-regex:
  Reimplement fixes to handle commas at the end of config parser lines
  Revert "Convert config framework to use std::regex"
  Revert "GH-636: Fix regex to handle commas at the end of config parser lines"
2023-04-13 08:57:30 -07:00
Arne Welzel
b739bdbbfb Merge remote-tracking branch 'origin/topic/awelzel/rotation-format-func-segfault'
* origin/topic/awelzel/rotation-format-func-segfault:
  logging/Manager: Fix crash for rotation format function not returning
2023-04-13 09:24:47 +02:00
Arne Welzel
a5e7faf564 logging/Manager: Fix crash for rotation format function not returning
While working on a rotation format function, ran into Zeek crashing
when not returning a value from it, fix and recover the same way as
for scripting errors.
2023-04-13 09:23:51 +02:00
Tim Wojtulewicz
187773eaa3 Reimplement fixes to handle commas at the end of config parser lines 2023-04-12 16:26:01 -07:00
Tim Wojtulewicz
777dff18cd Revert "Convert config framework to use std::regex"
This reverts commit 65ee2287e9.
2023-04-12 16:05:46 -07:00
Tim Wojtulewicz
a811722d72 Revert "GH-636: Fix regex to handle commas at the end of config parser lines"
This reverts commit 05bb509789.
2023-04-12 16:05:39 -07:00
Tim Wojtulewicz
6ac72a31bf Merge remote-tracking branch 'origin/topic/arne.welzel/fix-poll-interval-is-live'
* origin/topic/arne.welzel/fix-poll-interval-is-live:
  iosource/Manager: Fix poll_interval updating using not-yet valid IsLive()
2023-04-12 13:23:46 -07:00
zeek-bot
384bc9eecb Update doc submodule [nomail] [skip ci] 2023-04-12 00:34:45 +00:00
Tim Wojtulewicz
149ad57742 Merge remote-tracking branch 'origin/topic/timw/silence-noisy-compiler-flag'
* origin/topic/timw/silence-noisy-compiler-flag:
  Only use -Wno-changes-meaning if it exists
2023-04-11 15:41:46 -07:00
Tim Wojtulewicz
d8c1a1babf Merge remote-tracking branch 'security/topic/awelzel/155-reassem-validate-seq-upper-overflow'
* security/topic/awelzel/155-reassem-validate-seq-upper-overflow:
  file_analysis/File: Report overflowing chunks as weird and discard/truncate
  Reassem: Reject blocks overflowing 64bit upper
  zeek-setup: Load scrips before running unit tests
2023-04-11 15:30:58 -07:00
Tim Wojtulewicz
f812ce53cf Merge remote-tracking branch 'security/topic/timw/154-rdp-timeout'
* security/topic/timw/154-rdp-timeout:
  RDP: Instantiate SSL analyzer instead of PIA
  RDP: add some enforcement to required values based on MS-RDPBCGR docs
2023-04-11 15:23:52 -07:00
Tim Wojtulewicz
0cc1d672b5 Merge remote-tracking branch 'security/topic/timw/150-dns-expiration'
* security/topic/timw/150-dns-expiration:
  Stop skipping expiration of empty DNS mappings
2023-04-11 15:16:47 -07:00
Tim Wojtulewicz
f701f1fc94 Merge remote-tracking branch 'security/topic/awelzel/152-smtp-validate-mail-transactions'
* security/topic/awelzel/152-smtp-validate-mail-transactions:
  smtp: Validate mail transaction and disable SMTP analyzer if excessive
  generic-analyzer-fuzzer: Detect disable_analyzer() from scripts
2023-04-11 15:16:25 -07:00
Tim Wojtulewicz
c670f3fdb2 Merge remote-tracking branch 'security/topic/awelzel/148-ftp-skip-get-pending-commands-multi-line-response'
* security/topic/awelzel/148-ftp-skip-get-pending-commands-multi-line-response:
  ftp/main: Special case for intermediate reply lines
  ftp/main: Skip get_pending_command() for intermediate reply lines
2023-04-11 14:50:55 -07:00
Tim Wojtulewicz
a7a895b178 Only use -Wno-changes-meaning if it exists 2023-04-11 14:35:56 -07:00
Tim Wojtulewicz
d1eb0f47fd Merge remote-tracking branch 'origin/topic/timw/update-broker-for-gcc-13'
* origin/topic/timw/update-broker-for-gcc-13:
  Add cstdint to WeirdState.h to fix compilation error on gcc13
  Ignore -Wchanges-meaning warning in Spicy
  Update broker submodule to pull in GCC 13 fix
  CI: Force rebuild of OpenSUSE Tumbleweed VM to pick up GCC 13
2023-04-11 12:05:48 -07:00
Tim Wojtulewicz
122b8d6052 Add cstdint to WeirdState.h to fix compilation error on gcc13 2023-04-11 11:55:34 -07:00
Tim Wojtulewicz
ca35c85983 Ignore -Wchanges-meaning warning in Spicy
This fixes a build failure with GCC 13. It's meant as a short-term fix to avoid
the failure in Zeek, but probably should be fixed more correctly upstream in
the Spicy repo.
2023-04-11 11:55:34 -07:00
Tim Wojtulewicz
a0dbfd2822 Update broker submodule to pull in GCC 13 fix 2023-04-11 11:55:34 -07:00
Tim Wojtulewicz
505ac36e43 CI: Force rebuild of OpenSUSE Tumbleweed VM to pick up GCC 13 2023-04-11 11:55:34 -07:00
Vern Paxson
0787c130d0 optimize record construction by deferring initializations of aggregates 2023-04-10 11:44:11 -07:00
Vern Paxson
2e2afa5e11 compile-scripts-to-C++ speedups by switching to raw record access 2023-04-10 11:43:40 -07:00
Vern Paxson
4600ca41f6 logging speedup by switching to raw record access 2023-04-10 11:43:19 -07:00
Vern Paxson
f866252e5e remove redundant record coercions 2023-04-10 11:42:48 -07:00
zeek-bot
aaefdcca55 Update doc submodule [nomail] [skip ci] 2023-04-09 00:19:52 +00:00
Tim Wojtulewicz
51ca741d92 Merge remote-tracking branch 'origin/topic/timw/continue-processing-math'
* origin/topic/timw/continue-processing-math:
  Fix a long-standing bug in the math around continue_processing
  Add is_processing_suspended BIF
2023-04-08 11:23:06 -07:00
zeek-bot
6d9a452136 Update doc submodule [nomail] [skip ci] 2023-04-07 00:20:15 +00:00
Tim Wojtulewicz
e3a25f1458 Merge remote-tracking branch 'origin/topic/timw/1280-packet-filtered-stats'
* origin/topic/timw/1280-packet-filtered-stats:
  Add NEWS entries for filtered packet statistics and telemetry
  Report packet statistics via the telemetry framework
  Add optional packet filtered statistics for packet sources
2023-04-06 13:46:42 -07:00
Tim Wojtulewicz
50c3d0f474 Add NEWS entries for filtered packet statistics and telemetry 2023-04-06 13:41:09 -07:00
Tim Wojtulewicz
67802e711a Report packet statistics via the telemetry framework 2023-04-06 13:41:09 -07:00
Tim Wojtulewicz
95b6069c6e Fix a long-standing bug in the math around continue_processing 2023-04-06 10:49:05 -07:00
Tim Wojtulewicz
adcaa459c9 Add is_processing_suspended BIF 2023-04-06 10:36:21 -07:00
Tim Wojtulewicz
ae3d6a4df0 Add optional packet filtered statistics for packet sources 2023-04-06 09:47:04 -07:00
Arne Welzel
8a23671b21 Merge remote-tracking branch 'origin/topic/awelzel/add-debian-bookworm'
* origin/topic/awelzel/add-debian-bookworm:
  ci: Add debian-12 / bookworm
2023-04-06 12:55:44 +02:00
Arne Welzel
c390c0203d iosource/Manager: Fix poll_interval updating using not-yet valid IsLive()
Testing io_poll_interval_live tweaks with @dopheide-esnet on a Myricom based
system to reduce CPU usage showed no visible effect.

It turns out, the pkt_src->IsLive() call used to update poll_interval is only
valid *after* calling ->Register() with the source. The conditional updating
of the poll_interval introduced in 4fa3e4b9b4
never worked out how it was intended to.

The fix ensures that

* we actually use a poll_interval of 10 in the live case
* changing io_poll_interval_live does have an effect

This is a bit of a major change due to lowering the default poll_interval
by a magnitude, but that seemed to have been the intention always. It's also
tunable via redef, so worst case it can be adapted via configuration.

As reference, with the default a Pcap::non_fd_timeout of 20usec *and* a
poll_interval of 100, theoretically we'd be trying to ask a non-selectable
packet source 500000 per second for a new packet. This is not a likely packet
rate that a single worker would currently observe or manage to process.
2023-04-06 11:44:50 +02:00
Arne Welzel
49e964d711 ci: Add debian-12 / bookworm
The next version of Debian (bookworm) had a hard-freeze on
2023-03-16. Seems reasonable to have it in CI now.
2023-04-05 12:57:22 +02:00
Arne Welzel
d5739982f8 Merge remote-tracking branch 'origin/topic/awelzel/reporter-generic-message-deprecation'
* origin/topic/awelzel/reporter-generic-message-deprecation:
  scan.l: Support @pragma, specifically push/pop ignore-deprecations
  Switch deprecations to reporter->Deprecation()
  Reporter: Add dedicated Deprecation() method
  Desc: Add obj_desc_short()
2023-04-05 10:39:42 +02:00
Arne Welzel
d9611132e8 scan.l: Support @pragma, specifically push/pop ignore-deprecations
This allows to ignore-deprecations as follows:

    @pragma push ignore-deprecations
    print r$deprecated;
    @pragma pop

A bit of infrastructure is put in place for the push and pop, but
nothing overly generalized for future pragmas.
2023-04-05 10:24:30 +02:00
Tim Wojtulewicz
cfd726afbd Merge remote-tracking branch 'origin/topic/timw/gen-zam-warnings'
* origin/topic/timw/gen-zam-warnings:
  Update gen-zam submodule for std::move warnings
2023-04-04 09:05:46 -07:00