* topic/christian/telemetry-make-bifs-primary:
Telemetry framework: move BIFs to the primary-bif stage
Minor comment tweaks for init-frameworks-and-bifs.zeek
Adding a metric for the network time value itself should make it
possible to observe it stopping or growing slowly as compared to
realtime when Zeek isn't able to keep up.
Also, modify the telemetry/log.zeek test to include misc/stats and
log at a higher frequency with a more interesting pcap.
This stops invoking Telemetry::sync() via a scheduled event and instead
only invokes it on-demand. This makes metric collection network time
independent and lazier, too.
With Prometheus scrape requests being processed on Zeek's main thread
now, we can safely invoke the script layer Telemetry::sync() hook.
Closes#3947
This commit prevents most non-Modbus TCP traffic on port 502 to be
reported as Modbus in conn.log as well as in modbus.log.
To do so, we have introduced two &enforce checks in the Modbus
protocol definition that checks that some specific fields of the
(supposedly) Modbus header are compatible with values specified in
the specs.
To ensure non-regression, with this commit we also introduce a
new btest.
Closes#3962
* topic/vern/script-opt-headers-factoring:
factored CPP source's main header into collection of per-source-file headers
renamed script optimization Attrs.h header to prepare for factoring large Compile.h
factored ZAM source's main header into collection of per-source-file headers
This moves the Telemetry framework's BIF-defined functionalit from the
secondary-BIFs stage to the primary one. That is, this functionality is now
available from the end of init-bare.zeek, not only after the end of
init-frameworks-and-bifs.zeek.
This allows us to use script-layer telemetry in our Zeek's own code that get
pulled in during init-frameworks-and-bifs.
This change splits up the BIF features into functions, constants, and types,
because that's the granularity most workable in Func.cc and NetVar. It also now
defines the Telemetry::MetricsType enum once, not redundantly in BIFs and script
layer.
Due to subtle load ordering issues between the telemetry and cluster frameworks
this pushes the redef stage of Telemetry::metrics_port and address into
base/frameworks/telemetry/options.zeek, which is loaded sufficiently late in
init-frameworks-and-bifs.zeek to sidestep those issues. (When not doing this,
the effect is that the redef in telemetry/main.zeek doesn't yet find the
cluster-provided values, and Zeek does not end up listening on these ports.)
The need to add basic Zeek headers in script_opt/ZAM/ZBody.cc as a side-effect
of this is curious, but looks harmless.
Also includes baseline updates for the usual btests and adds a few doc strings.
This pins the generate-docs GH workflow to run on a fixed version of
ubuntu so we get defined semantics of `pip install` wrt system packages.
We were previously using the `ubuntu-latest` image and GH has been
switching this from ubuntu-22.04 to 24.04 and back again, also see
https://github.com/actions/runner-images/issues/10636.
This job uses `ubuntu-latest`, i.e., it will automatically use the
newest version of Ubuntu available. On current versions `pip install`
refuses to install packages into the system since this might break
system packages. Users are expected to explicitly opt into this
potential breakage by passing the `--break-system-packages` flag. This
is an acceptable workaround here since we do not distribute this setup,
and currently things appear to not break.
The GH action `pre-commit/action` installs `pre-commit` into the system
with `pip`.[^1] On newer distributions this is discouraged as it might break
system packages and would require a dedicated flag
`--break-system-packages` in order to pass.
Since the action does not work in such environments yet, instead pin the
distribution we use for the workflow to a fixed, older and working version.
[^1]: 576ff52938/action.yml (L11)
This isn't a straightforward fix, unfortunately. The existing GetLine()
implementation didn't deal well with input that's incrementally produced
where individually read chunks wouldn't end with the separator.
The prior implementation increased the buffer each time it failed to find
a separator in the current buffer, but then also ended up not searching the
full new buffer size for the terminator, doing that endlessly.
This change reworks the Raw reader to rely only on bufpos for reading
and searching purposes and skip reallocation if the buffer size if it
wasn't actually exhausted.
Closes#3957
Processing out-of-order commands or finishing commands based on invalid
server responses resulted in inconsistent analyzer state, potentially
triggering null pointer references for crafted traffic.
This commit reworks cf9fe91705 such that
too many pending commands are simply discarded, rather than any attempt
being made to process them. Further, invalid server responses do not
result in command completion anymore.
Test PCAP was crafted based on traffic produced by the OSS-Fuzz reproducer.
Closes#215
That test got flaky probably from #3949 on centosstream9 CI. You can
replicate that behavior by increasing the sleep time when waiting for
the file such that the test will attempt to read the missing file again.
Since the one second wait for file is glacially slow for this, speeding
it up should mean that the file gets created sooner and so the test
won't try to open the file again. But, it's always still technically
possible, since the test will wait for 10 seconds and the heartbeat
seems to be 1 second. At least if that happens, it's probably a bug or
massive slowdown of some kind.
It seems like other similar tests get by because they have more "stuff"
before they call `terminate()` most likely. But, to be safe, just
removing the "received termination signal" line seems like the best
approach.
