Commit graph

15378 commits

Author SHA1 Message Date
Arne Welzel
aaa47a709c logging: Introduce Log::default_logdir deprecate LogAscii::logdir and per writer logdir
Also modify FormatRotationPath to keep rotated logs within
Log::default_logdir unless the rotation function explicitly
set dir, e.g. by when the user redef'ed default_rotation_interval.
2022-07-06 18:54:29 +02:00
Arne Welzel
513ea7e04f logging/ascii: Fix .shadow paths when using LogAscii::logdir
With the introduction of LogAscii::logdir, log filenames can now include
parent directories rather than being plain basenames. Enabling log rotation,
leftover log rotation and setting LogAscii::logdir broke due to not
handling this situation.

This change ensures that .shadow files are placed within the directory where
the respective .log file is created. Previously, the .shadow. (or .tmp.shadow.)
prefix was simply prepended, yielding non-sensical paths such as
.tmp.shadow.foo/bar/packet_filter.log for a logdir of foo/bar.

Additionally, respect LogAscii::logdir when searching for leftover log files
rather than defaulting to the current working directory.

The following quirk exist around LogAscii::logdir, but will be addressed
in a follow-up.

* By default, logs are currently rotated into the working directory of the
  process, rather than staying confined within LogAscii::logdir. One of
  the added tests shows this behavior.
2022-07-06 13:21:21 +02:00
zeek-bot
8ba44c656a Update doc submodule [nomail] [skip ci] 2022-07-06 00:48:20 +00:00
Christian Kreibich
39f98778e4 CI: Add OpenSUSE Tumbleweed 2022-07-05 16:21:47 -07:00
Tim Wojtulewicz
cf63b55a34 Merge remote-tracking branch 'origin/topic/timw/dict-templates-redux'
* origin/topic/timw/dict-templates-redux:
  Update NEWS with breakage/deprecation notes about Dictionary changes [nomail] [skip ci]
  Remove unnecssary includes of Dict.h
  Turn PDict into a type alias
  Convert Dictionary types to be templated classes
  Fix signedness comparison warning in ssl analyzer
2022-07-05 22:19:15 +00:00
Tim Wojtulewicz
6686a849d5 Update NEWS with breakage/deprecation notes about Dictionary changes [nomail] [skip ci] 2022-07-05 21:53:18 +00:00
Christian Kreibich
2c263aa6c7 CI: drop Ubuntu 21.10, EOL 2022-07-14 2022-07-05 13:53:51 -07:00
Christian Kreibich
a2b73f3586 CI: drop Debian 9, EOL 2022-06-30 2022-07-05 13:53:36 -07:00
Tim Wojtulewicz
9f97a91d39 Remove unnecssary includes of Dict.h 2022-07-05 13:33:34 -07:00
Tim Wojtulewicz
49a1dc8a40 Turn PDict into a type alias 2022-07-05 13:33:34 -07:00
Tim Wojtulewicz
47e7fe2cd1 Convert Dictionary types to be templated classes
This has the fortunate side-effect of also making it so we can store
the value objects as typed pointers, instead of void*.
2022-07-05 13:33:34 -07:00
Tim Wojtulewicz
15c0bd9b9d Fix signedness comparison warning in ssl analyzer 2022-07-05 13:33:34 -07:00
Christian Kreibich
83fecb6b8f Merge branch 'topic/christian/management-test-tweaks'
* topic/christian/management-test-tweaks:
  Management framework: make artifact filenames Github-compatible
  Management framework: mask testsuite result
2022-07-05 09:42:42 -07:00
Johanna Amann
6e1e6fefe5 Merge remote-tracking branch 'origin/topic/johanna/2198'
* origin/topic/johanna/2198:
  SSL/GH-2211: Address review feedback, remove USE_FLIPPED
  SSL Analyzer: track connection direction by messages
2022-07-05 15:09:34 +01:00
Johanna Amann
2315d0344a SSL/GH-2211: Address review feedback, remove USE_FLIPPED
It turns out that we can just do this by using an argument.
2022-07-05 13:23:18 +01:00
Robin Sommer
a1207d61a5 Merge remote-tracking branch 'origin/topic/awelzel/add-ghc-filesystem'
* origin/topic/awelzel/add-ghc-filesystem:
  Add gulrak/filesystem to auxil, expose via zeek::filesystem
2022-07-05 10:39:41 +02:00
Christian Kreibich
bc15d7bffb Management framework: make artifact filenames Github-compatible
The upload-artifact action refuses to process certain filenames:

  "Invalid characters include: Double quote ", Colon :, Less than <,
   Greater than >, Vertical bar |, Asterisk *, Question mark ?, Carriage
   return \r, Line feed \n"

On failure, this iterates over the btest artifacts in the testsuite's .tmp
directory and renames any of these to dots.
2022-07-04 11:12:14 -07:00
Christian Kreibich
85b0381f3e Management framework: mask testsuite result
There's currently a race condition in the agent-supervisor interaction that can
cause requests to the Supervisor to get lost. This temporarily masks the
testsuite result until I've had a chance to investigate.
2022-07-04 11:09:24 -07:00
Arne Welzel
b77f5fa14c Add gulrak/filesystem to auxil, expose via zeek::filesystem
This adds https://github.com/gulrak/filesystem as a submodule into auxil
as a compiler-independent std::filesystem replacement.

The ghc::filesystem namespace is exposed as zeek::filesystem in util.h.

In the build directory, we add 3rdparty/ghc as a symlink to auxil in
order to support building from the build tree.

    <build_dir>/src/3rdparty/ghc -> /path/to/zeek/src/auxil/filesystem/include/ghc

In the installation tree, the headers are installed into include/zeek/3rdparty:

    <install_dir>/include/zeek/3rdparty/ghc

Note, this differs from how we approached rapidjson which isn't included
using a zeek/3rdparty and instead requires an additional include path of
the following form for external plugins to find and use it.

    <install_dir>/include/zeek/3rdparty/rapidjson/include/

We diverge from this approach. Placing ghc directly into 3rdparty appears
nicer and avoids changing external components (DynamicPlugin.cmake / spicyc)
2022-07-04 13:32:18 +02:00
Johanna Amann
d506806a22 Merge branch 'update' of https://github.com/leres/zeek
* 'update' of https://github.com/leres/zeek:
  Return false on error from the other place we call fstat()
  Address concerns raised by @0xxon; avoid the new code path when reading from a pipe and return false if fstat() fails after sucessfully opening the file (unlikely).
  Tweak some new comments
  Remove child program check, it's probably wrong given the test failures it causes
  Conform to style police
  Only set mtime and ino in Raw::OpenInput() do this for MODE_STREAM and avoid breaking MODE_REREAD
  Implement tail -F semantics for input framework MODE_STREAM Open /dev/null if the file is missing during init and wait for file to be created Collect initial ino, dev, and mtime when first opening the file Detect if the file has been replaced and open the new version
  Only set mtime and ino in Raw::OpenInput() do this for MODE_STREAM and avoid breaking MODE_REREAD
  Implement tail -F semantics for input framework MODE_STREAM Open /dev/null if the file is missing during init and wait for file to be created Collect initial ino, dev, and mtime when first opening the file Detect if the file has been replaced and open the new version
2022-07-04 10:24:34 +01:00
Tim Wojtulewicz
035c543e9f Remove loops from Dict iterator invalidation unit test
This fixes Coverity finding 1490366
2022-07-01 14:35:54 -07:00
Craig Leres
9866bf6ec5
Merge branch 'zeek:master' into update 2022-07-01 14:23:25 -07:00
Tim Wojtulewicz
ba968434e4 Merge remote-tracking branch 'origin/topic/timw/fix-RANDMAX-check'
* origin/topic/timw/fix-RANDMAX-check:
  Remove unused util::detail::rand64bit method
2022-07-01 14:11:18 -07:00
Tim Wojtulewicz
ef659b8e82 Remove unused util::detail::rand64bit method 2022-07-01 14:10:33 -07:00
Craig Leres
6b52c5b2f9 Return false on error from the other place we call fstat() 2022-07-01 13:23:06 -07:00
Craig Leres
529a3d8e77
Merge branch 'zeek:master' into update 2022-07-01 12:33:08 -07:00
Christian Kreibich
d3b6f9f7a5 Update doc submodule [nomail] [skip ci] 2022-07-01 12:15:59 -07:00
Christian Kreibich
b96d8afa52 Merge branch 'topic/christian/management-default-instance'
* topic/christian/management-default-instance:
  Management framework: bump zeek-client and cluster testsuite
2022-07-01 11:11:46 -07:00
Craig Leres
c765dce5f6 Address concerns raised by @0xxon; avoid the new code path when
reading from a pipe and return false if fstat() fails after sucessfully
opening the file (unlikely).
2022-07-01 10:03:15 -07:00
Craig Leres
51773bb8b4
Merge branch 'zeek:master' into update 2022-07-01 08:33:07 -08:00
Tim Wojtulewicz
ee88aa3acb Merge remote-tracking branch 'origin/topic/timw/2229-weak-keys-typos'
* origin/topic/timw/2229-weak-keys-typos:
  GH-2229: Fix some typos in weak-keys.zeek
2022-07-01 09:23:49 -07:00
Craig Leres
1b8f5644ac
Merge branch 'zeek:master' into update 2022-07-01 08:20:51 -08:00
zeek-bot
3b70d3a10b Update doc submodule [nomail] [skip ci] 2022-07-01 00:36:52 +00:00
Tim Wojtulewicz
509718b51c GH-2229: Fix some typos in weak-keys.zeek 2022-06-30 15:12:10 -07:00
Tim Wojtulewicz
94c9747440 Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy'
* origin/topic/bbannier/bump-spicy:
  Bump Spicy and spicy-plugin to their latest release.
2022-06-30 13:38:49 -07:00
Tim Wojtulewicz
1c3c88fd2a Merge remote-tracking branch 'origin/topic/johanna/rand64bit'
* origin/topic/johanna/rand64bit:
  Optimize 64 bit random number generation
2022-06-30 13:37:49 -07:00
Tim Wojtulewicz
5ce7e2c12b Merge remote-tracking branch 'origin/topic/timw/remove-5.1-deprecations'
* origin/topic/timw/remove-5.1-deprecations:
  Remove other general deprecations
  Remove some deprecated ocsp/ssl base scripts
  Remove deprecated DESC_PORTABLE ODesc mode and ODesc::IsPortable()
  Remove deprecated UDP, ICMP, and TCP analyzers
  Remove deprecated protocol_confirmation/violation events and methods
  Remove deprecated IterCookie-based dictionary iteration
  Remove deprecated MemoryAllocation() methods and related code
  Remove deprecated --enable-mobile-ipv6 configure argument
  Add helpers for syncing commit files with external testsuites
  Fix typo in update-timing target for external testsuites
2022-06-30 12:28:01 -07:00
Tim Wojtulewicz
fb16ce3711 Remove other general deprecations 2022-06-30 19:17:13 +00:00
Tim Wojtulewicz
6130d32440 Remove some deprecated ocsp/ssl base scripts 2022-06-30 19:17:08 +00:00
Tim Wojtulewicz
765a8535e0 Remove deprecated DESC_PORTABLE ODesc mode and ODesc::IsPortable() 2022-06-30 19:11:59 +00:00
Tim Wojtulewicz
77aa80033b Remove deprecated UDP, ICMP, and TCP analyzers 2022-06-30 19:11:59 +00:00
Tim Wojtulewicz
644ed5d314 Remove deprecated protocol_confirmation/violation events and methods 2022-06-30 19:11:57 +00:00
Tim Wojtulewicz
392c2f3810 Remove deprecated IterCookie-based dictionary iteration 2022-06-30 19:06:01 +00:00
Tim Wojtulewicz
70e63d4749 Remove deprecated MemoryAllocation() methods and related code 2022-06-30 18:56:52 +00:00
Tim Wojtulewicz
d3169e48c0 Remove deprecated --enable-mobile-ipv6 configure argument 2022-06-30 17:42:18 +00:00
Christian Kreibich
86431f0a44 Management framework: bump zeek-client and cluster testsuite
No Zeek-side changes here, this just pulls in updates and tests for more
flexible cluster configs in the client.
2022-06-30 10:24:57 -07:00
Christian Kreibich
b3de9a0faf Merge branch 'topic/christian/external-testsuite-tweaks'
* topic/christian/external-testsuite-tweaks:
  Add helpers for syncing commit files with external testsuites
  Fix typo in update-timing target for external testsuites
2022-06-30 09:56:46 -07:00
Benjamin Bannier
e8bc953e8a Bump Spicy and spicy-plugin to their latest release. 2022-06-30 15:44:05 +02:00
Johanna Amann
31cf270565 Optimize 64 bit random number generation
rand64bit called random 4 times to generate one 64 bit number. There is
no reason to do this - random() is basically guaranteed to return a 32
bit number.

This also adds a static check to make sure that it does.
2022-06-29 14:37:39 +02:00
Christian Kreibich
bf9b1ebbbe Add helpers for syncing commit files with external testsuites
This provides "make sync-repos" to check out all locally available testsuites at
the commits indicated in their commit files, and "make sync-commits" to update
the commit files to the HEADs of the local testsuite repos.

Also adds the commit -> repo sync for the Makefile init target so initialization
always lands on the right version, and removes the corresponding explicit
checkout from the CI repo setup.
2022-06-28 13:20:14 -07:00