Commit graph

15378 commits

Author SHA1 Message Date
Arne Welzel
4a6ce8b2ad Remove suspend-processing test
This tested that timers continue working even if one calls
suspend_processing() in zeek -r mode. The new behavior is
that timers do not function in that scenario and the test
invalid.
2023-03-23 12:41:56 +01:00
Arne Welzel
aad66b3cff Add a set of suspend_processing tests
Placed into core/ but relies on a few not-so-core things.
2023-03-23 12:41:56 +01:00
Arne Welzel
9c5b439e8f btest: More verbose recursive-event output
I don't think something broken, but the one line output.
2023-03-23 12:41:56 +01:00
Arne Welzel
3269310e61 broker::Manager: No more network_time forwarding
network_time forwarding will happen in the main-loop before draining the
EventMgr so timers/events scheduled based on broker messages should
behave similarly. This also keeps network_time unaffected during
non pseudo-realtime trace processing.
2023-03-23 12:41:56 +01:00
Arne Welzel
c30b5cbc4d TimerMgr: No network_time updates in Process()
network_time forwarding will now happen centrally in the main loop.
The TimerMgr returns a valid timeout that can be waited for and will
trigger network_time advancement, so we don't need to do it.
2023-03-23 12:41:56 +01:00
Arne Welzel
2e8176305b Event: No more network_time updates
The whole docs read like this was only used to update the
network_time, so there may be a follow-up to ditch EventMgr
even being an IOSource (which could be argued it's not IO).
2023-03-23 12:41:56 +01:00
Arne Welzel
d4e31e7d2b RunState: Implement forward_network_time_if_applicable()
Add a central place where the decision when it's okay to update network time
to the current time (wallclock) is. It checks for pseudo_realtime and packet
source existence as well as packet source idleness.

A new const &redef allows to completely disable forwarding of network time.
2023-03-23 12:40:39 +01:00
Arne Welzel
eefa0150b9 PktSrc: Add HasBeenIdleFor() method
This method will be used by the main loop to determine if an interface
has become idle. Initially this will be used to determine when it is
acceptable to update network_time to the current time (wallclock).
2023-03-23 12:40:25 +01:00
Arne Welzel
a9f8627dd9 PktSrc: Move termination pseduo_realtime special case to RunState
This also removes setting pseduo_realtime to 0.0 in the main loop
when the packet source has been closed. I had tried to understand
the implications it actually seems, if we shutdown the iosource::Manager
anyway, it shouldn't and it's just confusing.
2023-03-23 12:33:32 +01:00
Tim Wojtulewicz
5882299730 Ensure master/release builds run for zeek-security 2023-03-22 15:49:37 -07:00
Christian Kreibich
a9cec2e4f4 Bump zeekctl and doc submodule for updates regarding Site::local_nets 2023-03-22 13:34:54 -07:00
zeek-bot
4f45d9e00a Update doc submodule [nomail] [skip ci] 2023-03-22 00:33:38 +00:00
Arne Welzel
93305ad9f4 Merge remote-tracking branch 'origin/topic/awelzel/make-some-deprecations-errors'
* origin/topic/awelzel/make-some-deprecations-errors:
  Expr: Factor out type tag extraction
  Var: Add version to deprecated initialization
  Stmt: Error on deprecated when/local usage
  Expr: Remove vector scalar operations
  parse.y: Make out-of-scope use errors
  scan.l: Remove unused deprecated_attr
2023-03-21 19:18:41 +01:00
Arne Welzel
63be88a1f8 Merge remote-tracking branch 'origin/topic/awelzel/supervisor-pcap-file'
* origin/topic/awelzel/supervisor-pcap-file:
  Add pcap_file option to supervised nodes.
2023-03-21 16:40:51 +01:00
Jan Grashoefer
1882307cf3 Add pcap_file option to supervised nodes.
This allows to start Supervised nodes with a pcap_file argument
rather than interface.

This is based on changes from @J-Gras.
2023-03-21 16:18:02 +01:00
Arne Welzel
859ecc7b8b Merge remote-tracking branch 'origin/topic/awelzel/main-loop-allow-multiple-zero-timeouts'
* origin/topic/awelzel/main-loop-allow-multiple-zero-timeouts:
  NEWS: main-loop changes around zero-timeout sources
  Add a new plugin test with verbose IO source output
  iosource: Make poll intervals configurable
  iomanager/Poll: Add zero-timeout timeout_src also when there's other events ready
  iomanager: Collect all sources with zero timeouts as ready
2023-03-21 10:11:56 +01:00
Arne Welzel
2b14503c3d NEWS: main-loop changes around zero-timeout sources 2023-03-21 09:25:15 +01:00
Arne Welzel
38c4611c7e Add a new plugin test with verbose IO source output
This is mostly for documentation/verification purposes of how the IO loop
currently does draining and when it picks up FD based (non packet) IO
source. For example, it shows that currently FD based sources are processed
fairly delayed and that we now also process two timeout sources that are ready.
2023-03-21 09:15:39 +01:00
Arne Welzel
46c432dc8b iosource: Make poll intervals configurable
This probably should not be changed by users, but it's useful for
testing and experimentation rather than needing to recompile.

Processing 100 packets without checking an FD based IO source can
actually mean that FD based sources are never checked during a read
of a very small pcap...
2023-03-21 09:15:33 +01:00
Arne Welzel
5f1a85803a iomanager/Poll: Add zero-timeout timeout_src also when there's other events ready
This would generally happen the next loop iteration around anyway, but
seems nice to ensure a zero timeout source will be processed at the same
time as sources with ready FDs.
2023-03-21 09:15:33 +01:00
Arne Welzel
f3fcaf776c iomanager: Collect all sources with zero timeouts as ready
Previously, if two iosources returned 0.0 as their timeout, only
one of them would be considered ready. An always ready source
therefore may starve other ready ones due to this and minimally
this behavior seems surprising.

Offline pcap sources are always ready and return 0.0 for
GetNextTimeout() (unless in pseudo-realtime), so we can
also remove the offline source special case.

One subtle side-effect of this change is that if an IO source
returns a 0.0 timeout *and* it's file descriptor is ready in
the same loop iteration, it may be processed twice.
2023-03-21 09:15:33 +01:00
Arne Welzel
cb5c43ecbd Expr: Factor out type tag extraction 2023-03-21 09:06:50 +01:00
Arne Welzel
c1f51cbf2a Var: Add version to deprecated initialization
This has been added for Zeek 5.0, so mark it for removal in Zeek 6.1.
2023-03-20 21:36:01 +01:00
Arne Welzel
bb80d80218 Stmt: Error on deprecated when/local usage
This has been around since Zeek v4.1, so it was warned about in Zeek 5.0
LTS and we could've removed it with 5.1.

Also removed merge_top_scope() from the zeek::detail namespace, it's
unused now.

Updated the when-aggregates test somehow. I'm not quite sure what had
been tested there :-/
2023-03-20 21:35:53 +01:00
Arne Welzel
5ef62b2de8 Expr: Remove vector scalar operations
These seem to have been deprecated since 2018, so do it now.
Unfortunately the message didn't contain a version when it'll
be removed, but it's been long enough.
2023-03-20 21:35:48 +01:00
Arne Welzel
12d5dca70f parse.y: Make out-of-scope use errors
This was marked to be done in Zeek 5.1, so do it now. The message
didn't include a version, unfortunately, but minimally there was
a comment when it should happen.
2023-03-20 21:35:48 +01:00
Arne Welzel
5b2c82c1e9 scan.l: Remove unused deprecated_attr 2023-03-20 21:35:48 +01:00
Tim Wojtulewicz
5cba1e0b09 Merge remote-tracking branch 'origin/topic/timw/update-zeekctl-test-baseline'
* origin/topic/timw/update-zeekctl-test-baseline:
  ci/centos-stream-8: Add which command
  Update zeekctl test baseline for command.peerstatus test
  Update zeekctl test baseline for changes to Site::local_nets
2023-03-20 10:05:12 -07:00
Arne Welzel
79ea84e5f4 ci/centos-stream-8: Add which command 2023-03-20 10:04:20 -07:00
Tim Wojtulewicz
1b6b6b3209 Update zeekctl test baseline for command.peerstatus test 2023-03-20 10:04:20 -07:00
Arne Welzel
1599235b58 cirrus: No container images on PRs
Skip building container images (and skip cluster testing)
unless running with fullci.

They don't provide a lot of additional testing coverage, so
should be fine to just run them after merges to master.
2023-03-20 12:27:01 +01:00
Tim Wojtulewicz
88c6d64cc5 Update zeekctl test baseline for changes to Site::local_nets 2023-03-18 20:23:00 -07:00
Tim Wojtulewicz
7c92a58e05 Merge remote-tracking branch 'origin/topic/neverlord/gh-2853'
* origin/topic/neverlord/gh-2853:
  Update Broker for fixing #2853
2023-03-18 17:10:46 -07:00
Christian Kreibich
ebbed2aa04 Merge branch 'topic/christian/fix-signedness-warning'
* topic/christian/fix-signedness-warning:
  Fix integer signedness comparison warning with GCC 12.2
2023-03-17 14:39:52 -07:00
Dominik Charousset
ffb31a64af Update Broker for fixing #2853 2023-03-17 19:52:18 +01:00
Christian Kreibich
2eabbd2a0d Fix integer signedness comparison warning with GCC 12.2
This fixes a "warning: comparison of integer expressions of different
signedness: ‘size_t’ {aka ‘long unsigned int’} and ‘int’".
2023-03-16 18:10:59 -07:00
zeek-bot
85acdea90f Update doc submodule [nomail] [skip ci] 2023-03-17 00:17:22 +00:00
Arne Welzel
61a5a16ed0 Merge remote-tracking branch 'origin/topic/awelzel/remove-dns-ad-cd-from-log-again'
* origin/topic/awelzel/remove-dns-ad-cd-from-log-again:
  dns: Remove AD and CD flags from log
2023-03-16 10:19:04 +01:00
Arne Welzel
cf2da5160b dns: Remove AD and CD flags from log
There was a misunderstanding whether to include them by default in
the dns.log, so remove them again.

There had also been a discussion and quirk that AD of a request would
always be overwritten by reply in the dns.log unless the reply is
missing. For now, let users extend dns.log themselves for what best
fits their requirements, rather than adding these flags by default.

Add a btest to print AD and CD flags for smoke testing still.
2023-03-16 10:09:27 +01:00
Christian Kreibich
e25d9bdee9 Merge branch 'topic/christian/gh-2761-default-localnets'
* topic/christian/gh-2761-default-localnets:
  Update NEWS to cover new Site::local_nets behavior.
  Bump external testsuites to reflect updated baselines
  Add btests for new Site::local_nets behavior
  Update plugins.hooks baseline to reflect added config framework activity
  Update scripts.policy.misc.dump-events baseline
  Update btest baselines to reflect the use of local address ranges.
  Tighten local-nets filtering in the logging framework's path-func-column-demote test
  Fix scripts.policy.frameworks.intel.removal test given address locality info
  Treat private address space as site-local by default
  Provide a mechanism to suppress logging of internal config framework activity
2023-03-15 20:05:40 -07:00
Christian Kreibich
7a21a90f19 Updating CHANGES and VERSION. 2023-03-15 20:00:10 -07:00
zeek-bot
edf726f84a Update doc submodule [nomail] [skip ci] 2023-03-16 00:41:33 +00:00
Christian Kreibich
c456f0206d Update NEWS to cover new Site::local_nets behavior. 2023-03-15 17:33:00 -07:00
Christian Kreibich
fa15d1b258 Bump external testsuites to reflect updated baselines 2023-03-15 17:32:56 -07:00
Christian Kreibich
b2319ba5d9 Add btests for new Site::local_nets behavior 2023-03-15 17:11:08 -07:00
Christian Kreibich
c3cde56a0a Update plugins.hooks baseline to reflect added config framework activity 2023-03-15 17:11:08 -07:00
Christian Kreibich
f73c4e91d1 Update scripts.policy.misc.dump-events baseline 2023-03-15 17:11:08 -07:00
Christian Kreibich
1843e2daae Update btest baselines to reflect the use of local address ranges. 2023-03-15 17:11:04 -07:00
Christian Kreibich
4281d704c1 Tighten local-nets filtering in the logging framework's path-func-column-demote test
With private addresses treated as local ones, this picked up some private-range
flows in the test pcap involved.
2023-03-15 17:01:01 -07:00
Christian Kreibich
d387da9f71 Fix scripts.policy.frameworks.intel.removal test given address locality info
This test used `Site::is_local_addr()` as part of a filtering criterion, perhaps
unintentionally. The fact that it applied to all tested addresses kept a Zeek
process from exiting, failing the test. It also doesn't need to prioritize its
zeek_init() handler.
2023-03-15 17:01:00 -07:00