* origin/topic/timw/remove-abspath-cleanup:
diff-remove-abspath: Add separate handling of Windows paths
diff-remove-abspath: Remove capture of windows drive letters from POSIX regex
* origin/topic/vern/zam-asan-fixes:
ZAM fixes for assignments involving "any" record fields
fixes for (mostly ZAM) vector operation issues found by ASAN
Including a fix for mmdb/explicit-open.zeek to avoid using assert.
* origin/topic/timw/update-c-ares-to-latest-release:
DNS_Mgr: Remove processing of dns aliases in general
ci: Add dnsmasq to a few platforms for testing
DNS_Mgr: Fix aliases memory issues
btest: Add integration test for DNS_Mgr
DNS_Mgr: Remove usage of ares_getsock from Lookup
DNS_Mgr: Remove usage of ares_getsock from GetNextTimeout
DNS_Mgr: Switch to ares_set_servers_csv
DNS_Mgr: Use ares_dns_record methods for queries
Update vcpkg submodule to pick up c-ares v1.34.2
Update c-ares submodule to v1.34.2
This wasn't possible before #3028 was fixed, but now it's safe to set
the value in new_connection() and allow other users access to the
field much earlier. We do not have to deal with connection_flipped()
because the community-id hash is symmetric.
So far, when Zeek didn't see a connection's regular tear-down (e.g.,
because its state timed-out before we got to the end), we'd still
signal a regular end-of-data to Spicy parsers. As a result, they would
then typically raise a parse error because they were probably still
expecting data and would now declare it missing. That's not very
useful because semantically it's not really a protocol issue if the
data just doesn't make it over to us; it's a transport-layer issue
that Zeek already handles elsewhere. So we now switch to signaling
end-of-data to Spicy analyzers only if the connection indeed shuts
down regularly. This is also matches how BinPAC handles it.
This also comes with a test exercising various combinations of
end-of-data behavior so that we ensure consistent/desired behavior.
Closes#4007.
This admittedly is a quite esoteric combination of protocols. But - as
we do correctly support them, it seems nice to have a slightly more
complete testcase that covers this.
* origin/topic/awelzel/data-end-offset-v1:
signatures: Add data_end_offset to signature_match() and custom events
Add pattern_end_offset to signature_state
This change tracks the current offset (number of bytes fed into matchers)
on the top-level RuleEndpointState such that we can compute the relative ending
for matched texts individually.
Additionally, it adds the data_end_offset as a new optional parameter to
signature_match().
This commit fixes three issues with Zeek's Modbus message logging:
1 - Some exception responses (e.g., READ_COILS_EXCEPTION) are logged
twice: once without and once with the exception message.
2 - Some exception responses (e.g., PROGRAM_484_EXCEPTION) are not
logged.
3 - Some known but reserved function codes (e.g., PROGRAM_UNITY) are
logged as unk-xxx (e.g., unk-90), while it would be possible to
log their known name.
To address these inconsistencies, the modbus parser has been updated
to parse all exception responses (i.e., all responses where the MSB
of the function code is set) using the already defined Exception
message.
Also, the Modbus main.zeek script has been updated to consistently
demand logging exception responses to the specialized
modbus_exception event, rather than logging some exception responses
in the modbus_message event and others in the modbus_exception event.
Finally, the main.zeek script has been updated to make sure that
for every known function code, the corresponding exception code was
also present, and the enumeration of known function codes in
consts.zeek has been expanded.
Closes#3984
* origin/topic/vern/CPP-when-capture-naming:
fixed "-O gen-C++" naming of "when" captures to avoid ambiguities due to inlining
Simplified & made more robust maintenance helper script for "-O gen-C++" testing
"-a cpp" baseline updates to reflect recent BTest changes
remove instance of plus sign to account for real plus in sql
account for spaces encoding to plus signs in sqli regex detection
add test cases for sqli space to plus
account for spaces encoding to plus signs in sqli regex detection
forgot semicolon
account for spaces encoding to plus signs in sqli regex detection
* topic/christian/telemetry-make-bifs-primary:
Telemetry framework: move BIFs to the primary-bif stage
Minor comment tweaks for init-frameworks-and-bifs.zeek
Adding a metric for the network time value itself should make it
possible to observe it stopping or growing slowly as compared to
realtime when Zeek isn't able to keep up.
Also, modify the telemetry/log.zeek test to include misc/stats and
log at a higher frequency with a more interesting pcap.
This stops invoking Telemetry::sync() via a scheduled event and instead
only invokes it on-demand. This makes metric collection network time
independent and lazier, too.
With Prometheus scrape requests being processed on Zeek's main thread
now, we can safely invoke the script layer Telemetry::sync() hook.
Closes#3947
This commit prevents most non-Modbus TCP traffic on port 502 to be
reported as Modbus in conn.log as well as in modbus.log.
To do so, we have introduced two &enforce checks in the Modbus
protocol definition that checks that some specific fields of the
(supposedly) Modbus header are compatible with values specified in
the specs.
To ensure non-regression, with this commit we also introduce a
new btest.
Closes#3962
This moves the Telemetry framework's BIF-defined functionalit from the
secondary-BIFs stage to the primary one. That is, this functionality is now
available from the end of init-bare.zeek, not only after the end of
init-frameworks-and-bifs.zeek.
This allows us to use script-layer telemetry in our Zeek's own code that get
pulled in during init-frameworks-and-bifs.
This change splits up the BIF features into functions, constants, and types,
because that's the granularity most workable in Func.cc and NetVar. It also now
defines the Telemetry::MetricsType enum once, not redundantly in BIFs and script
layer.
Due to subtle load ordering issues between the telemetry and cluster frameworks
this pushes the redef stage of Telemetry::metrics_port and address into
base/frameworks/telemetry/options.zeek, which is loaded sufficiently late in
init-frameworks-and-bifs.zeek to sidestep those issues. (When not doing this,
the effect is that the redef in telemetry/main.zeek doesn't yet find the
cluster-provided values, and Zeek does not end up listening on these ports.)
The need to add basic Zeek headers in script_opt/ZAM/ZBody.cc as a side-effect
of this is curious, but looks harmless.
Also includes baseline updates for the usual btests and adds a few doc strings.
This isn't a straightforward fix, unfortunately. The existing GetLine()
implementation didn't deal well with input that's incrementally produced
where individually read chunks wouldn't end with the separator.
The prior implementation increased the buffer each time it failed to find
a separator in the current buffer, but then also ended up not searching the
full new buffer size for the terminator, doing that endlessly.
This change reworks the Raw reader to rely only on bufpos for reading
and searching purposes and skip reallocation if the buffer size if it
wasn't actually exhausted.
Closes#3957
Processing out-of-order commands or finishing commands based on invalid
server responses resulted in inconsistent analyzer state, potentially
triggering null pointer references for crafted traffic.
This commit reworks cf9fe91705 such that
too many pending commands are simply discarded, rather than any attempt
being made to process them. Further, invalid server responses do not
result in command completion anymore.
Test PCAP was crafted based on traffic produced by the OSS-Fuzz reproducer.
Closes#215
