Commit graph

16055 commits

Author SHA1 Message Date
Robin Sommer
8df0d02528 Bump documentation. 2022-07-11 12:52:12 +02:00
Robin Sommer
62e9bf5628 Merge remote-tracking branch 'origin/topic/awelzel/policy-pattern-literal-dots'
* origin/topic/awelzel/policy-pattern-literal-dots:
  policy: Use literal dots for patterns used against content-type and hostname
2022-07-11 12:51:08 +02:00
Arne Welzel
c4970e6d4a policy: Use literal dots for patterns used against content-type and hostname
The following two patterns were identified while reviewing patterns that
match on any characters. The intention likely was to match actual
literal dots.
2022-07-11 10:34:47 +02:00
Christian Kreibich
9607deeae0 Stop signal-masking upon running unit tests
It helps to be able to ctrl-c these.
2022-07-10 21:50:47 -07:00
Dominik Charousset
d3ad3c618a Avoid race conditions in broker.ssl-mismatch test 2022-07-10 15:47:43 +02:00
Christian Kreibich
9138d5c64e Pause signal-masking during script parsing
Script parsing includes reading script content from stdin, which in turn
includes interactive Zeek sessions. Keeping the signals masked there broke
ctrl-c behavior.
2022-07-09 11:19:41 -07:00
Christian Kreibich
5beb68194d Add btests to verify Zeek's handling of SIGTERM and reading stdin
The core.load-stdin test checks Zeek's ability to read scripts from stdin.
core.sigterm-regular and core.sigterm-stdin verify that SIGTERM shuts down a
Zeek process during normal operation and while reading script content from
stdin, respectively. For technical reasons we don't test with SIGINT, as ctrl-c
would trigger -- see comments for details.
2022-07-09 11:17:14 -07:00
mAsk°
ccc9d0a002
Merge branch 'zeek:master' into topic/amazingpp/table-values-and-keys 2022-07-09 21:07:49 +08:00
Benjamin Bannier
b13f8f4dd4 Fix default build with --disable-cpp-tests.
For a build configured with `--disable-cpp-tests` we globally set
`-DDOCTEST_CONFIG_DISABLE`. Spicy does not expect this define to be used
and its unit tests fail to build with it.

This patch adds a Zeek-side workaround for that. We now do not build the
Spicy unit test targets as part of `ALL` anymore.

Closes #2238.
2022-07-09 12:57:32 +02:00
Christian Kreibich
963b27f054 Add procps/procps-ng to several CI Docker images
It helps to have "ps" available for testing.
2022-07-09 00:52:44 -07:00
zeek-bot
d957f883df Update doc submodule [nomail] [skip ci] 2022-07-09 00:43:07 +00:00
Tim Wojtulewicz
c697c8e19f Merge remote-tracking branch 'origin/topic/timw/fix-broken-broker-convert-call'
* origin/topic/timw/fix-broken-broker-convert-call:
  Fix usage of broker::convert(double,timespan)
2022-07-08 10:13:12 -07:00
AmazingPP
ba552ceeaf Add table_keys function 2022-07-09 01:11:31 +08:00
Tim Wojtulewicz
a266d32dca Fix usage of broker::convert(double,timespan) 2022-07-08 09:57:12 -07:00
Tim Wojtulewicz
0728676fd6 Merge remote-tracking branch 'origin/topic/timw/avoid-extra-reallocs'
* origin/topic/timw/avoid-extra-reallocs:
  Avoid calling safe_realloc unnecessarily from ODesc::Grow()
2022-07-08 09:55:47 -07:00
Tim Wojtulewicz
c9127715c0 Avoid calling safe_realloc unnecessarily from ODesc::Grow() 2022-07-08 09:55:16 -07:00
AmazingPP
cb71b15eab Add table_values function 2022-07-09 00:19:06 +08:00
Robin Sommer
e8e6fa8fed Merge remote-tracking branch 'origin/topic/awelzel/913-plugin-debug-stream-validation'
One change: turns out that zeek-config isn't in PATH, so I prefixed it
with $BUILD.

* origin/topic/awelzel/913-plugin-debug-stream-validation:
  zeek-setup: Validate plugin debug streams during startup
2022-07-08 10:00:39 +02:00
Christian Kreibich
28b7ec2cc5 Bump broker and zeek-archiver submodules to pull in CI updates 2022-07-07 14:20:24 -07:00
Christian Kreibich
23cac286f3 Merge branch 'topic/christian/ci-update'
* topic/christian/ci-update:
  CI: add FreeBSD 13.1 and fix broken link
  CI: Add OpenSUSE Tumbleweed
  CI: drop Ubuntu 21.10, EOL 2022-07-14
  CI: drop Debian 9, EOL 2022-06-30
2022-07-07 14:14:15 -07:00
Vern Paxson
0706e8869d Fix for ZAM optimization of '?' conditional with vector-valued result 2022-07-07 13:08:28 -07:00
Vern Paxson
77ca9ae013 fix for script profiling of lambda functions 2022-07-07 11:01:54 -07:00
Arne Welzel
d703033ae6 zeek-setup: Validate plugin debug streams during startup
Providing an unknown plugin debug stream with -B was previously silently
accepted. This caused user confusing as the behavior is "no output, but
seems to work".

Check the enabled debug streams once all plugins have been loaded and
exit early on for invalid streams.

    $ ZEEK_PLUGIN_PATH=./build zeek -B plugin-zeek-myplugin -e 'print zeek_version();'
    error in <command line>, line 3: No plugin debug stream 'plugin-zeek-myplugin' found

Closes #913.
2022-07-07 17:56:19 +02:00
Robin Sommer
691b099de1 Merge remote-tracking branch 'origin/topic/awelzel/2120-logdir-leftover'
* origin/topic/awelzel/2120-logdir-leftover:
  sqlite default-logdir test: Remove ls ./logs baseline
  logging/sqlite: Recognize Log::default_logdir and place files there if set
  logging: Introduce Log::default_logdir deprecate LogAscii::logdir and per writer logdir
  logging/ascii: Fix .shadow paths when using LogAscii::logdir
2022-07-07 08:06:13 +02:00
Christian Kreibich
94ef182a97 CI: add FreeBSD 13.1 and fix broken link 2022-07-06 21:58:56 -07:00
Arne Welzel
a2bcb1bf28 sqlite default-logdir test: Remove ls ./logs baseline
Observed .sqlite-journal files and missing reporter.sqlite files
in CI runs. Subsequently reading the ./test.sqlite file is more
reliable and should be good enough.
2022-07-06 22:57:14 +02:00
Arne Welzel
93584c7c7f logging/sqlite: Recognize Log::default_logdir and place files there if set 2022-07-06 18:54:29 +02:00
Arne Welzel
aaa47a709c logging: Introduce Log::default_logdir deprecate LogAscii::logdir and per writer logdir
Also modify FormatRotationPath to keep rotated logs within
Log::default_logdir unless the rotation function explicitly
set dir, e.g. by when the user redef'ed default_rotation_interval.
2022-07-06 18:54:29 +02:00
Arne Welzel
513ea7e04f logging/ascii: Fix .shadow paths when using LogAscii::logdir
With the introduction of LogAscii::logdir, log filenames can now include
parent directories rather than being plain basenames. Enabling log rotation,
leftover log rotation and setting LogAscii::logdir broke due to not
handling this situation.

This change ensures that .shadow files are placed within the directory where
the respective .log file is created. Previously, the .shadow. (or .tmp.shadow.)
prefix was simply prepended, yielding non-sensical paths such as
.tmp.shadow.foo/bar/packet_filter.log for a logdir of foo/bar.

Additionally, respect LogAscii::logdir when searching for leftover log files
rather than defaulting to the current working directory.

The following quirk exist around LogAscii::logdir, but will be addressed
in a follow-up.

* By default, logs are currently rotated into the working directory of the
  process, rather than staying confined within LogAscii::logdir. One of
  the added tests shows this behavior.
2022-07-06 13:21:21 +02:00
zeek-bot
8ba44c656a Update doc submodule [nomail] [skip ci] 2022-07-06 00:48:20 +00:00
Christian Kreibich
39f98778e4 CI: Add OpenSUSE Tumbleweed 2022-07-05 16:21:47 -07:00
Tim Wojtulewicz
cf63b55a34 Merge remote-tracking branch 'origin/topic/timw/dict-templates-redux'
* origin/topic/timw/dict-templates-redux:
  Update NEWS with breakage/deprecation notes about Dictionary changes [nomail] [skip ci]
  Remove unnecssary includes of Dict.h
  Turn PDict into a type alias
  Convert Dictionary types to be templated classes
  Fix signedness comparison warning in ssl analyzer
2022-07-05 22:19:15 +00:00
Tim Wojtulewicz
6686a849d5 Update NEWS with breakage/deprecation notes about Dictionary changes [nomail] [skip ci] 2022-07-05 21:53:18 +00:00
Christian Kreibich
2c263aa6c7 CI: drop Ubuntu 21.10, EOL 2022-07-14 2022-07-05 13:53:51 -07:00
Christian Kreibich
a2b73f3586 CI: drop Debian 9, EOL 2022-06-30 2022-07-05 13:53:36 -07:00
Tim Wojtulewicz
9f97a91d39 Remove unnecssary includes of Dict.h 2022-07-05 13:33:34 -07:00
Tim Wojtulewicz
49a1dc8a40 Turn PDict into a type alias 2022-07-05 13:33:34 -07:00
Tim Wojtulewicz
47e7fe2cd1 Convert Dictionary types to be templated classes
This has the fortunate side-effect of also making it so we can store
the value objects as typed pointers, instead of void*.
2022-07-05 13:33:34 -07:00
Tim Wojtulewicz
15c0bd9b9d Fix signedness comparison warning in ssl analyzer 2022-07-05 13:33:34 -07:00
Christian Kreibich
83fecb6b8f Merge branch 'topic/christian/management-test-tweaks'
* topic/christian/management-test-tweaks:
  Management framework: make artifact filenames Github-compatible
  Management framework: mask testsuite result
2022-07-05 09:42:42 -07:00
Johanna Amann
6e1e6fefe5 Merge remote-tracking branch 'origin/topic/johanna/2198'
* origin/topic/johanna/2198:
  SSL/GH-2211: Address review feedback, remove USE_FLIPPED
  SSL Analyzer: track connection direction by messages
2022-07-05 15:09:34 +01:00
Johanna Amann
2315d0344a SSL/GH-2211: Address review feedback, remove USE_FLIPPED
It turns out that we can just do this by using an argument.
2022-07-05 13:23:18 +01:00
Robin Sommer
a1207d61a5 Merge remote-tracking branch 'origin/topic/awelzel/add-ghc-filesystem'
* origin/topic/awelzel/add-ghc-filesystem:
  Add gulrak/filesystem to auxil, expose via zeek::filesystem
2022-07-05 10:39:41 +02:00
Christian Kreibich
bc15d7bffb Management framework: make artifact filenames Github-compatible
The upload-artifact action refuses to process certain filenames:

  "Invalid characters include: Double quote ", Colon :, Less than <,
   Greater than >, Vertical bar |, Asterisk *, Question mark ?, Carriage
   return \r, Line feed \n"

On failure, this iterates over the btest artifacts in the testsuite's .tmp
directory and renames any of these to dots.
2022-07-04 11:12:14 -07:00
Christian Kreibich
85b0381f3e Management framework: mask testsuite result
There's currently a race condition in the agent-supervisor interaction that can
cause requests to the Supervisor to get lost. This temporarily masks the
testsuite result until I've had a chance to investigate.
2022-07-04 11:09:24 -07:00
Arne Welzel
b77f5fa14c Add gulrak/filesystem to auxil, expose via zeek::filesystem
This adds https://github.com/gulrak/filesystem as a submodule into auxil
as a compiler-independent std::filesystem replacement.

The ghc::filesystem namespace is exposed as zeek::filesystem in util.h.

In the build directory, we add 3rdparty/ghc as a symlink to auxil in
order to support building from the build tree.

    <build_dir>/src/3rdparty/ghc -> /path/to/zeek/src/auxil/filesystem/include/ghc

In the installation tree, the headers are installed into include/zeek/3rdparty:

    <install_dir>/include/zeek/3rdparty/ghc

Note, this differs from how we approached rapidjson which isn't included
using a zeek/3rdparty and instead requires an additional include path of
the following form for external plugins to find and use it.

    <install_dir>/include/zeek/3rdparty/rapidjson/include/

We diverge from this approach. Placing ghc directly into 3rdparty appears
nicer and avoids changing external components (DynamicPlugin.cmake / spicyc)
2022-07-04 13:32:18 +02:00
Johanna Amann
d506806a22 Merge branch 'update' of https://github.com/leres/zeek
* 'update' of https://github.com/leres/zeek:
  Return false on error from the other place we call fstat()
  Address concerns raised by @0xxon; avoid the new code path when reading from a pipe and return false if fstat() fails after sucessfully opening the file (unlikely).
  Tweak some new comments
  Remove child program check, it's probably wrong given the test failures it causes
  Conform to style police
  Only set mtime and ino in Raw::OpenInput() do this for MODE_STREAM and avoid breaking MODE_REREAD
  Implement tail -F semantics for input framework MODE_STREAM Open /dev/null if the file is missing during init and wait for file to be created Collect initial ino, dev, and mtime when first opening the file Detect if the file has been replaced and open the new version
  Only set mtime and ino in Raw::OpenInput() do this for MODE_STREAM and avoid breaking MODE_REREAD
  Implement tail -F semantics for input framework MODE_STREAM Open /dev/null if the file is missing during init and wait for file to be created Collect initial ino, dev, and mtime when first opening the file Detect if the file has been replaced and open the new version
2022-07-04 10:24:34 +01:00
Tim Wojtulewicz
035c543e9f Remove loops from Dict iterator invalidation unit test
This fixes Coverity finding 1490366
2022-07-01 14:35:54 -07:00
Craig Leres
9866bf6ec5
Merge branch 'zeek:master' into update 2022-07-01 14:23:25 -07:00
Tim Wojtulewicz
ba968434e4 Merge remote-tracking branch 'origin/topic/timw/fix-RANDMAX-check'
* origin/topic/timw/fix-RANDMAX-check:
  Remove unused util::detail::rand64bit method
2022-07-01 14:11:18 -07:00