Mostly, treat HTTP0.9 completely separate. Because we're doing raw
delivery of a body directly, fake enough (connection_close=1, and finish
headers manually) so that the MIME infrastructure thinks it is seeing a
body.
This deals better with the body due to accounting for the first line. Also
it avoids the content line analyzer to strip CRLF/LF and the analyzer
then adding CRLF unconditionally by fully bypassing the content line
analyzer.
Concretely, the vlan-mpls test case contains a HTTP response with LF only,
but the previous implementation would use CRLF, accounting for two many bytes.
Same for the http.no-version test which would previously report a body
length of 280 and now is at 323 (which agrees with wireshark).
Further, the mime_type detection for the http-09 test case works because
it's now seeing the full body.
Drawback: We don't extract headers when a server actually replies with
a HTTP/1.1 message, but grrr, something needs to give I guess.
It wasn't possible from script land to determine the total size
of the cache table held by the DNS_Mgr. Add the total and also
also the TEXT entries count.
This commit introduces parsing of the CertificateRequest message in the
TLS handshake. It introduces a new event ssl_certificate_request, as
well as a new function parse_distinguished_name, which can be used to
parse part of the ssl_certificate_request event parameters.
This commit also introduces a new policy script, which appends
information about the CAs a TLS server requests in the
CertificateRequest message, if it sends it.
This adds two example plugins within testing/builtin-plugins/Files:
* protocol-plugin copied over from testing/btest/plugins/protocol-plugin
* py-lib-plugin that embeds Python to have a dependency on an external
shared library which was already available in CI and fun to use, too.
Closes#2837
* origin/topic/vern/Mar23-script-opt-maint:
extend BTest "path" canonicalization to include compiled-to-C++ variable names
use a subclass of TableType for incremental construction of compiled scripts
script_opt/CPP: errors, recursive type fixes, fix embedded comments
Fix for EnumVal's returning their underlying value
better (than nothing) run-time errors for compiled scripts
fixes for dealing with recursive types in compiled scripts
fix for values in compiled scripts containing embedded comment markers
The ja3 package uses next instead of return and triggers the new
errors with Zeek 5.2. That seems somewhat bad.
In case we want to do a 5.2.1 that makes this a warning, this
would be change to do so.
This change exposes the signature tyope inside the signed portion of an
X.509 certificate. In the past, we only exposed the signature type that
is contained inside the signature, which is outside the signed portion
of the X.509 certificate.
In theory, both signature fields should have the same value; it is,
however, possible to encode differing values in both fields. The new
field is not logged by default.
This adds one metric per log stream and one metric per log writer (path based)
to track the number of writes on a stream level as well as on a writer level.
$ curl -sSf localhost:8181/metrics | grep Conn
zeek_log_writer_writes_total{endpoint="",filter-name="default",module="HTTP",path="http",stream="HTTP::LOG",writer="Log::WRITER_SQLITE"} 1 1677497572770
zeek_log_stream_writes_total{endpoint="",module="HTTP",stream="HTTP::LOG"} 1 1677497572770
The initial version of this change also included metrics around log
write vetoes, but given no log policies exist in the default configuration
and they are mostly interesting for a few streams/writers only, skip this
for now. These can always be added by the script writer, too.
The difference between the stream level writes and concrete writers can
be used to deduce the number of vetoes (or errors) as a starting point.
* origin/topic/awelzel/2572-event-handler-stats:
Bump external baselines
external/testing-setup: Less telemetry.log volume
EventHandler: Use telemetry framework for EventHandler.call_count
For traces spanning a long time period, don't produce immense
amounts of logs. Also filter down the event handler invocation
entries to connection related events in telemetry.log.
Put the IntCounter into a std::optional rather than initializing
it at EventHandler construction time as that will currently expose
a time series per event handler through the Prometheus API.
The user and password fields are replicated to each of the ftp.log
entries. Using a very large username (100s of KBs) allows to bloat
the log without actually sending much traffic. Further, limit the
arg and reply_msg columns to large, but not unbounded values.
It's happening regularly to me that I forget the type specifier when redef'ing
records or enums and usually it takes me a while to figure out what's going
on as the errors are not descriptive. Improve the error reporting and just
bail as there's no sensible way to continue.
Closes#2777
* ekoyle/add-protocol-pbb:
Update seemingly-unrelated btests
Use a default analyzer
Simplify PBB analyzer by using Ethernet analyzer
Add btest for PBB and update baselines
Use constexpr instead of #define
Cleanup and add customer MAC addresses
Add PBB (802.1ah) support
We previously would include any and all output from stderr during
compilation in the test baseline. Depending on the used compiler this
output may contain C++ compilation warnings which are uninteresting for
the behavior under test.
(cherry picked from commit 5221edf474)
* origin/topic/awelzel/propagate-on-change-through-copy:
TableVal: Propagate &on_change attribute through copy()
testing/btest: Add test showing &expire_func/&create_expire is copied
Copying an &ordered table or set would result in a copy that is not ordered.
This seems rather surprising behavior, so propagate the &ordered attribute.
Closes#2793
Mostly for consistency with &default, &expire_func and other attributes
being propagated through a copy(). Seems this was just missed during
the implementation and/or was never tested for.
After the first 4 bytes, this traffic actually just looks like Ethernet.
Rather than try to re-implement the ethernet analyzer, just check the
length, skip 4 bytes, and pass it on.
This adds a new utility called ci/collect-repo-info.py to produce a JSON
document that is then baked into the Zeek executable file. Further, when
creating a tarball via `make dist`, put a top-level repo-info.json file
in place that is picked when no .git directory exists.
Closes#1405
We previously used the Spicy plugin's `Spicy::available` to test for
Spicy support. However, having Spicy support does not necessarily mean that we
have built Zeek with its in-tree Spicy analyzers: the Spicy plugin
could have been pulled in from external. The new BIF now reliably
tells us whether the Spicy analyzers are available; its result
corresponds to what `zeek-config --have-spicy-analyzers` returns as
well.
We also move the two current checks over to use this BIF.
(Note: I refrained from renaming the CMake-side `USE_SPICY_ANALYERS`
to `HAVE_SPICY_ANALYZERS`. We should do this eventually for
consistency, but I didn't want to make more changes than necessary
right now.)
* origin/topic/awelzel/arm64-container-follow-ups-2:
Bump private testsuite for char handling fixes
Base64: report byte as positive integer
netbios_decode: use unsigned char for result
analyzer/http: Do not assume char is signed
cirrus: Run tests in a Debian 11 container, too
A baseline difference between arm64 and x86 showed up. We would
print a wrong character as negative value on x86 due to chars
being signed by default. Force an unsigned interpretation which
is also more reasonable because we'd have never indexed the
base64 table with -112
-XXXXXXXXXX.XXXXXX XXXXXXXXXXX 131.243.99.154 3288 193.159.183.138 80 base64_illegal_encoding character -112 ignored by Base64 decoding F zeek -
+XXXXXXXXXX.XXXXXX XXXXXXXXXXX 131.243.99.154 3288 193.159.183.138 80 base64_illegal_encoding character 144 ignored by Base64 decoding F zeek -
Fixes more of #2742
