Commit graph

14727 commits

Author SHA1 Message Date
Tim Wojtulewicz
cfd726afbd Merge remote-tracking branch 'origin/topic/timw/gen-zam-warnings'
* origin/topic/timw/gen-zam-warnings:
  Update gen-zam submodule for std::move warnings
2023-04-04 09:05:46 -07:00
Tim Wojtulewicz
0d4f842190 Update gen-zam submodule for std::move warnings 2023-04-04 09:02:05 -07:00
Arne Welzel
92f09f0db7 Merge remote-tracking branch 'origin/topic/vern/when-cleanup'
* origin/topic/vern/when-cleanup:
  test suite update for minor change in "when" error messages
  removed skeletal (non-functioning) "when" support from ZAM
  simplify WhenInfo and Trigger classes given removal of old capture semantics
  introduced notion of light-weight Frame clones
  changed function_ingredients struct to FunctionIngredients class with accessors

Renamed Frame::LightClone() to Frame::CloneForTrigger() during merge.
2023-04-04 10:02:01 +02:00
zeek-bot
5ab86131fa Update doc submodule [nomail] [skip ci] 2023-04-04 00:20:19 +00:00
Arne Welzel
1858c6b0c2 Merge remote-tracking branch 'origin/topic/vern/CPP-Apr23-maint'
* origin/topic/vern/CPP-Apr23-maint:
  addressed static analysis concern about possible null pointer
  tweaks for "-O C++" of BTest's with conditional code

Backed out changes from at-if-lambda during merge and instead skip
test when running with ZEEK_USE_CPP.
2023-04-03 16:42:18 +02:00
Arne Welzel
a12d1e1bc2 Merge remote-tracking branch 'origin/topic/vern/ZAM-Apr23-maint'
* origin/topic/vern/ZAM-Apr23-maint:
  minor ZAM BTest baseline updates
  fixed type mismatch for ssl_certificate_request event
  skip ZAM optimization of invalid scripts
  extended script validation to be call-able on a per-function basis
2023-04-03 13:33:09 +02:00
Arne Welzel
daa6fff154 Merge remote-tracking branch 'origin/topic/awelzel/pkt-src-timeout-setting'
* origin/topic/awelzel/pkt-src-timeout-setting:
  PktSrc: Introduce Pcap::non_fd_timeout
2023-04-03 13:24:30 +02:00
Vern Paxson
910b50ef0d test suite update for minor change in "when" error messages 2023-04-02 11:38:30 -07:00
Vern Paxson
61891e615a removed skeletal (non-functioning) "when" support from ZAM 2023-04-02 11:37:36 -07:00
Vern Paxson
84906171ba simplify WhenInfo and Trigger classes given removal of old capture semantics 2023-04-02 11:36:39 -07:00
Vern Paxson
4af6b52876 introduced notion of light-weight Frame clones 2023-04-02 11:35:15 -07:00
Vern Paxson
0c434ca4f8 changed function_ingredients struct to FunctionIngredients class with accessors 2023-04-02 11:34:16 -07:00
Vern Paxson
b5603fe5bf addressed static analysis concern about possible null pointer 2023-04-01 21:06:42 -07:00
Vern Paxson
2f9272f2d5 tweaks for "-O C++" of BTest's with conditional code 2023-04-01 20:26:55 -07:00
Vern Paxson
da643add2e minor ZAM BTest baseline updates 2023-04-01 12:57:21 -07:00
Vern Paxson
9560e38784 fixed type mismatch for ssl_certificate_request event 2023-04-01 12:56:54 -07:00
Vern Paxson
b16195d130 skip ZAM optimization of invalid scripts 2023-04-01 12:56:23 -07:00
Vern Paxson
3a99e60d1c extended script validation to be call-able on a per-function basis 2023-04-01 12:55:29 -07:00
Arne Welzel
f00d6198af PktSrc: Introduce Pcap::non_fd_timeout
Increasing this value 10x has lowered CPU usage on a Myricom based
deployment significantly with reportedly no adverse side-effects.

After reviewing the Zeek 3 IO loop, my hunch is that previously when
no packets were available, we'd sleep 20usec every loop iteration after
calling ->Process() on the packet source. With current master ->Process()
is called 10 times on a packet source before going to sleep just once
for 20 usec. Likely this explains the increased CPU usage reported.

It's probably too risky to increase the current value, so introduce
a const &redef value for advanced users to tweak it. A middle ground
might be to lower ``io_poll_interval_live`` to 5 and increase the new
``Pcap::non_fd_timeout`` setting to 100usec.

While this doesn't really fix #2296, we now have enough knobs for tweaking.

Closes #2296.
2023-03-31 18:48:08 +02:00
Arne Welzel
5718046b96 Merge branch 'topic/jgras/tunnel-fwd-fix' of https://github.com/J-Gras/zeek
* 'topic/jgras/tunnel-fwd-fix' of https://github.com/J-Gras/zeek:
  Add test cases for Geneve.
  Fix forwarding of tunnelled packets.
2023-03-31 10:39:42 +02:00
Arne Welzel
c1215ca47a Merge remote-tracking branch 'origin/topic/awelzel/val-mgr-list-val-type-list-caching'
* origin/topic/awelzel/val-mgr-list-val-type-list-caching:
  Type: Add TypeManager->TypeList() and use for ListVal()
2023-03-31 10:34:05 +02:00
Jan Grashoefer
65f0a09a83 Add test cases for Geneve. 2023-03-30 22:58:54 +02:00
Tim Wojtulewicz
bcb7bbd129 Merge remote-tracking branch 'origin/topic/timw/packet-analyzer-override-test'
* origin/topic/timw/packet-analyzer-override-test:
  Add reporter framework to core.packet-analyzer-override btest
2023-03-30 13:09:30 -07:00
Jan Grashoefer
073a8a6082 Fix forwarding of tunnelled packets.
This fixes a bug for AYIYA, Geneve and VXLAN forwarding encapsulated
content only if it's longer than their header. A new weird is introduced
to indicate empty tunnels.
2023-03-30 22:07:41 +02:00
Tim Wojtulewicz
d2eaced96a Add reporter framework to core.packet-analyzer-override btest 2023-03-30 12:28:39 -07:00
Arne Welzel
24c606b4df Type: Add TypeManager->TypeList() and use for ListVal()
It turns out that for every ListVal we construct, we also allocate
and construct a new TypeList instance, even though they are all the
same. Pre-create and cache the type instances in a new TypeManager.

The following script runs ~10% faster for me after this change.

    global tbl: table[string] of string;
    global i = 0;
    while ( ++i < 10000000 )
        tbl["a"] = "a";
2023-03-30 21:15:46 +02:00
Tim Wojtulewicz
69b6443ddb Merge remote-tracking branch 'jgras/topic/jgras/packet-fwd-fix'
* jgras/topic/jgras/packet-fwd-fix:
  Consider cap len when forwarding into packet analysis.
2023-03-30 09:52:33 -07:00
Tim Wojtulewicz
67335d92cd Merge remote-tracking branch 'jgras/topic/jgras/packet-analyzer-override-info'
* jgras/topic/jgras/packet-analyzer-override-info:
  Reintroduce info when overriding packet analyzer mappings.
2023-03-30 09:51:12 -07:00
Jan Grashoefer
bfa466f2eb Reintroduce info when overriding packet analyzer mappings.
In #2464 the warning when overriding a packet analyzer mapping was
removed. While a warning seems indeed excessive, some info would still
be nice to have.
2023-03-30 16:19:39 +02:00
Jan Grashoefer
fb2042ca76 Consider cap len when forwarding into packet analysis.
When forwarding into packet analysis from TCP or UDP, the protocol's
length fields were trusted. This might be dangerous in case of truncated
packets.
2023-03-30 15:47:01 +02:00
Arne Welzel
04a2ee7220 Merge remote-tracking branch 'origin/topic/timw/increase-ci-hardware-limits'
* origin/topic/timw/increase-ci-hardware-limits:
  Update Cirrus memory limits to avoid timeouts when building
2023-03-30 09:37:18 +02:00
Arne Welzel
81063d5561 Merge remote-tracking branch 'origin/topic/awelzel/get-child-analyzer'
* origin/topic/awelzel/get-child-analyzer:
  Analyzer: Add GetChildAnalyzer() and IsPreventedChildAnalyzer()
2023-03-30 09:36:22 +02:00
Tim Wojtulewicz
0a6f975a4b Update Cirrus memory limits to avoid timeouts when building 2023-03-29 17:17:47 -07:00
Tim Wojtulewicz
224e60a3f3 Merge remote-tracking branch 'origin/topic/bbannier/bump-spicy'
* origin/topic/bbannier/bump-spicy:
  Bump Spicy and spicy-plugin to their latest development snapshots.
2023-03-29 14:34:42 -07:00
Benjamin Bannier
53652a0673 Bump Spicy and spicy-plugin to their latest development snapshots. 2023-03-29 16:54:41 +02:00
Arne Welzel
b03e678438 Analyzer: Add GetChildAnalyzer() and IsPreventedChildAnalyzer()
GetChildAnalyzer() has the same semantics as HasChildAnalyzer(), but returns
the raw pointer to the child analyzer. Main issue is memory management: That
pointer is not guaranteed to stay valid. It might be disabled from script
land or otherwise removed from the analyzer tree and subsequent
deleted in one of the Forward* methods.

IsPreventedChildAnalyzer() provides minimal introspection for prevented
child analyzer tags and allows to remove some duplicated code.
2023-03-29 13:05:43 +02:00
Tim Wojtulewicz
136d54a68e Merge remote-tracking branch 'origin/topic/timw/std-move-warnings'
* origin/topic/timw/std-move-warnings:
  Fix a bunch of -Wunqualified-std-cast-call warnings from clang 15
2023-03-28 09:20:09 -07:00
Arne Welzel
2f5b9ce038 Merge remote-tracking branch 'origin/topic/awelzel/broker-no-network-time-init'
* origin/topic/awelzel/broker-no-network-time-init:
  btest/broker: Add test using Python bindings and zeek -r
  Broker: Remove network time initialization
2023-03-27 21:33:14 +02:00
Arne Welzel
cd0d55e3d6 btest/broker: Add test using Python bindings and zeek -r
This came up in community.zeek.org as a use-case and there
were a few gaps and holes, so add a test showing minimally
the current behavior.
2023-03-27 21:13:03 +02:00
Tim Wojtulewicz
7aa310ee50 Fix a bunch of -Wunqualified-std-cast-call warnings from clang 15 2023-03-24 13:58:01 -07:00
Arne Welzel
59b1e29e81 Broker: Remove network time initialization
Remove the special case of initializing network time if it hasn't
happened yet. The argument about broker.log containing 0.0 timestamps
is more a problem of the log, not something that would justify modifying
network time globally. For broker.log and possibly cluster.log, it might
be more reasonable to use current time, anyway.

I was a bit wary about tables backed by broker stores being populated
with network_time set to 0.0, but there seems to exist logic and assumptions
that this is okay: It should be the same as if one populates a table with
expirations set within zeek_init().

In fact, staring a bit more, *not setting* network time might be more correct
as workers that don't see packets would never set zeek_start_network_time
which is used within the expiration computation.
2023-03-24 15:13:29 +01:00
Arne Welzel
96bbb96b35 Merge remote-tracking branch 'origin/topic/awelzel/no-container-images-on-prs'
* origin/topic/awelzel/no-container-images-on-prs:
  cirrus: No container images on PRs
2023-03-24 09:12:26 +01:00
zeek-bot
0478c45f81 Update doc submodule [nomail] [skip ci] 2023-03-24 00:29:58 +00:00
Arne Welzel
71f487bd20 Merge remote-tracking branch 'origin/topic/timw/zeek-security-master-builds'
* origin/topic/timw/zeek-security-master-builds:
  Ensure master/release builds run for zeek-security
2023-03-23 13:40:37 +01:00
Arne Welzel
890d2b5e07 Merge remote-tracking branch 'origin/topic/awelzel/try-update-network-time'
* origin/topic/awelzel/try-update-network-time:
  NEWS: Some notes about timing related changes
  iosource: Remove non-existing ManagerBase friend
  broker::Manager: use_realtime_false when allow_network_time_forward=F
  A set of tests around set_network_time() and timer expiration
  Remove suspend-processing test
  Add a set of suspend_processing tests
  btest: More verbose recursive-event output
  broker::Manager: No more network_time forwarding
  TimerMgr: No network_time updates in Process()
  Event: No more network_time updates
  RunState: Implement forward_network_time_if_applicable()
  PktSrc: Add HasBeenIdleFor() method
  PktSrc: Move termination pseduo_realtime special case to RunState
2023-03-23 13:37:28 +01:00
Arne Welzel
fd15a33f8c NEWS: Some notes about timing related changes 2023-03-23 12:41:56 +01:00
Arne Welzel
833dd158de iosource: Remove non-existing ManagerBase friend 2023-03-23 12:41:56 +01:00
Arne Welzel
9b8ab7b210 broker::Manager: use_realtime_false when allow_network_time_forward=F
Run the broker in non-realtime mode when allow_network_time_forward=F.
This may need an extra option for really advanced use-cases, but for
now this seems reasonable.
2023-03-23 12:41:56 +01:00
Arne Welzel
8734f23724 A set of tests around set_network_time() and timer expiration
This tests that timer expiration happens after a call to set_network_time()
upon the next time around the loop. This should be fairly stable, but
suspect major changes in the main loop or around timer expiration may
subtly change behavior things.
2023-03-23 12:41:56 +01:00
Arne Welzel
4a6ce8b2ad Remove suspend-processing test
This tested that timers continue working even if one calls
suspend_processing() in zeek -r mode. The new behavior is
that timers do not function in that scenario and the test
invalid.
2023-03-23 12:41:56 +01:00