Increasing this value 10x has lowered CPU usage on a Myricom based
deployment significantly with reportedly no adverse side-effects.
After reviewing the Zeek 3 IO loop, my hunch is that previously when
no packets were available, we'd sleep 20usec every loop iteration after
calling ->Process() on the packet source. With current master ->Process()
is called 10 times on a packet source before going to sleep just once
for 20 usec. Likely this explains the increased CPU usage reported.
It's probably too risky to increase the current value, so introduce
a const &redef value for advanced users to tweak it. A middle ground
might be to lower ``io_poll_interval_live`` to 5 and increase the new
``Pcap::non_fd_timeout`` setting to 100usec.
While this doesn't really fix#2296, we now have enough knobs for tweaking.
Closes#2296.
This fixes a bug for AYIYA, Geneve and VXLAN forwarding encapsulated
content only if it's longer than their header. A new weird is introduced
to indicate empty tunnels.
It turns out that for every ListVal we construct, we also allocate
and construct a new TypeList instance, even though they are all the
same. Pre-create and cache the type instances in a new TypeManager.
The following script runs ~10% faster for me after this change.
global tbl: table[string] of string;
global i = 0;
while ( ++i < 10000000 )
tbl["a"] = "a";
In #2464 the warning when overriding a packet analyzer mapping was
removed. While a warning seems indeed excessive, some info would still
be nice to have.
GetChildAnalyzer() has the same semantics as HasChildAnalyzer(), but returns
the raw pointer to the child analyzer. Main issue is memory management: That
pointer is not guaranteed to stay valid. It might be disabled from script
land or otherwise removed from the analyzer tree and subsequent
deleted in one of the Forward* methods.
IsPreventedChildAnalyzer() provides minimal introspection for prevented
child analyzer tags and allows to remove some duplicated code.
* origin/topic/awelzel/broker-no-network-time-init:
btest/broker: Add test using Python bindings and zeek -r
Broker: Remove network time initialization
An invalid mail transaction is determined as
* RCPT TO command without a preceding MAIL FROM
* a DATA command without a preceding RCPT TO
and logged as a weird.
The testing pcap for invalid mail transactions was produced with a Python
script against a local exim4 configured to accept more errors and unknown
commands than 3 by default:
# exim4.conf.template
smtp_max_synprot_errors = 100
smtp_max_unknown_commands = 100
See also: https://www.rfc-editor.org/rfc/rfc5321#section-3.3
It is currently not possible to call a->Conn()->GetVal() or construct a
zeek/file_analysis/File object from within doctests, as these quickly
reference the unpopulated zeek::id namespace to construct Val objects
of various types, making it hard write basic tests without completely
re-organizing.
Move running of the unit tests after parsing the scripts, so it is possible
for some basic exercising of File objects within tests.
Remove the special case of initializing network time if it hasn't
happened yet. The argument about broker.log containing 0.0 timestamps
is more a problem of the log, not something that would justify modifying
network time globally. For broker.log and possibly cluster.log, it might
be more reasonable to use current time, anyway.
I was a bit wary about tables backed by broker stores being populated
with network_time set to 0.0, but there seems to exist logic and assumptions
that this is okay: It should be the same as if one populates a table with
expirations set within zeek_init().
In fact, staring a bit more, *not setting* network time might be more correct
as workers that don't see packets would never set zeek_start_network_time
which is used within the expiration computation.
Test if the analyzer is removed from the TCPSessionAdapter during
event processing. If we don't do this, we continue feeding the analyzer
even if scripts decided to disable the analyzer.
The analyzer instance isn't flagged as disabled itself, so we need
to look at the parent's children.
Intermediate lines of multiline replies usually do not contain valid status
codes (even if servers may opt to include them). Their content may be anything
and likely unrelated to the original command. There's little reason for us
trying to match them with a corresponding command.
OSS-Fuzz generated a large command reply with very many intermediate lines
which caused long processing times due to matching every line with all
currently pending commands.
This is a DoS vector against Zeek. The new ipv6-multiline-reply.trace and
ipv6-retr-samba.trace files have been extracted from the external ipv6.trace.
* origin/topic/awelzel/try-update-network-time:
NEWS: Some notes about timing related changes
iosource: Remove non-existing ManagerBase friend
broker::Manager: use_realtime_false when allow_network_time_forward=F
A set of tests around set_network_time() and timer expiration
Remove suspend-processing test
Add a set of suspend_processing tests
btest: More verbose recursive-event output
broker::Manager: No more network_time forwarding
TimerMgr: No network_time updates in Process()
Event: No more network_time updates
RunState: Implement forward_network_time_if_applicable()
PktSrc: Add HasBeenIdleFor() method
PktSrc: Move termination pseduo_realtime special case to RunState
Run the broker in non-realtime mode when allow_network_time_forward=F.
This may need an extra option for really advanced use-cases, but for
now this seems reasonable.
This tests that timer expiration happens after a call to set_network_time()
upon the next time around the loop. This should be fairly stable, but
suspect major changes in the main loop or around timer expiration may
subtly change behavior things.
