Commit graph

4716 commits

Author SHA1 Message Date
Tim Wojtulewicz
ae3d6a4df0 Add optional packet filtered statistics for packet sources 2023-04-06 09:47:04 -07:00
Arne Welzel
d9611132e8 scan.l: Support @pragma, specifically push/pop ignore-deprecations
This allows to ignore-deprecations as follows:

    @pragma push ignore-deprecations
    print r$deprecated;
    @pragma pop

A bit of infrastructure is put in place for the push and pop, but
nothing overly generalized for future pragmas.
2023-04-05 10:24:30 +02:00
Arne Welzel
92f09f0db7 Merge remote-tracking branch 'origin/topic/vern/when-cleanup'
* origin/topic/vern/when-cleanup:
  test suite update for minor change in "when" error messages
  removed skeletal (non-functioning) "when" support from ZAM
  simplify WhenInfo and Trigger classes given removal of old capture semantics
  introduced notion of light-weight Frame clones
  changed function_ingredients struct to FunctionIngredients class with accessors

Renamed Frame::LightClone() to Frame::CloneForTrigger() during merge.
2023-04-04 10:02:01 +02:00
Arne Welzel
1858c6b0c2 Merge remote-tracking branch 'origin/topic/vern/CPP-Apr23-maint'
* origin/topic/vern/CPP-Apr23-maint:
  addressed static analysis concern about possible null pointer
  tweaks for "-O C++" of BTest's with conditional code

Backed out changes from at-if-lambda during merge and instead skip
test when running with ZEEK_USE_CPP.
2023-04-03 16:42:18 +02:00
Vern Paxson
910b50ef0d test suite update for minor change in "when" error messages 2023-04-02 11:38:30 -07:00
Vern Paxson
2f9272f2d5 tweaks for "-O C++" of BTest's with conditional code 2023-04-01 20:26:55 -07:00
Vern Paxson
da643add2e minor ZAM BTest baseline updates 2023-04-01 12:57:21 -07:00
Vern Paxson
9560e38784 fixed type mismatch for ssl_certificate_request event 2023-04-01 12:56:54 -07:00
Arne Welzel
5718046b96 Merge branch 'topic/jgras/tunnel-fwd-fix' of https://github.com/J-Gras/zeek
* 'topic/jgras/tunnel-fwd-fix' of https://github.com/J-Gras/zeek:
  Add test cases for Geneve.
  Fix forwarding of tunnelled packets.
2023-03-31 10:39:42 +02:00
Jan Grashoefer
65f0a09a83 Add test cases for Geneve. 2023-03-30 22:58:54 +02:00
Tim Wojtulewicz
d2eaced96a Add reporter framework to core.packet-analyzer-override btest 2023-03-30 12:28:39 -07:00
Jan Grashoefer
bfa466f2eb Reintroduce info when overriding packet analyzer mappings.
In #2464 the warning when overriding a packet analyzer mapping was
removed. While a warning seems indeed excessive, some info would still
be nice to have.
2023-03-30 16:19:39 +02:00
Arne Welzel
cd0d55e3d6 btest/broker: Add test using Python bindings and zeek -r
This came up in community.zeek.org as a use-case and there
were a few gaps and holes, so add a test showing minimally
the current behavior.
2023-03-27 21:13:03 +02:00
Arne Welzel
b8dc6ad120 smtp: Validate mail transaction and disable SMTP analyzer if excessive
An invalid mail transaction is determined as

* RCPT TO command without a preceding MAIL FROM
* a DATA command without a preceding RCPT TO

and logged as a weird.

The testing pcap for invalid mail transactions was produced with a Python
script against a local exim4 configured to accept more errors and unknown
commands than 3 by default:

    # exim4.conf.template
    smtp_max_synprot_errors = 100
    smtp_max_unknown_commands = 100

See also: https://www.rfc-editor.org/rfc/rfc5321#section-3.3
2023-03-27 18:41:47 +02:00
Tim Wojtulewicz
a9d3245e80 RDP: add some enforcement to required values based on MS-RDPBCGR docs 2023-03-24 10:33:21 -07:00
Arne Welzel
59b1e29e81 Broker: Remove network time initialization
Remove the special case of initializing network time if it hasn't
happened yet. The argument about broker.log containing 0.0 timestamps
is more a problem of the log, not something that would justify modifying
network time globally. For broker.log and possibly cluster.log, it might
be more reasonable to use current time, anyway.

I was a bit wary about tables backed by broker stores being populated
with network_time set to 0.0, but there seems to exist logic and assumptions
that this is okay: It should be the same as if one populates a table with
expirations set within zeek_init().

In fact, staring a bit more, *not setting* network time might be more correct
as workers that don't see packets would never set zeek_start_network_time
which is used within the expiration computation.
2023-03-24 15:13:29 +01:00
Arne Welzel
1b3e8a611e ftp/main: Skip get_pending_command() for intermediate reply lines
Intermediate lines of multiline replies usually do not contain valid status
codes (even if servers may opt to include them). Their content may be anything
and likely unrelated to the original command. There's little reason for us
trying to match them with a corresponding command.

OSS-Fuzz generated a large command reply with very many intermediate lines
which caused long processing times due to matching every line with all
currently pending commands.
This is a DoS vector against Zeek. The new ipv6-multiline-reply.trace and
ipv6-retr-samba.trace files have been extracted from the external ipv6.trace.
2023-03-23 13:50:36 +01:00
Arne Welzel
8734f23724 A set of tests around set_network_time() and timer expiration
This tests that timer expiration happens after a call to set_network_time()
upon the next time around the loop. This should be fairly stable, but
suspect major changes in the main loop or around timer expiration may
subtly change behavior things.
2023-03-23 12:41:56 +01:00
Arne Welzel
4a6ce8b2ad Remove suspend-processing test
This tested that timers continue working even if one calls
suspend_processing() in zeek -r mode. The new behavior is
that timers do not function in that scenario and the test
invalid.
2023-03-23 12:41:56 +01:00
Arne Welzel
aad66b3cff Add a set of suspend_processing tests
Placed into core/ but relies on a few not-so-core things.
2023-03-23 12:41:56 +01:00
Arne Welzel
9c5b439e8f btest: More verbose recursive-event output
I don't think something broken, but the one line output.
2023-03-23 12:41:56 +01:00
Arne Welzel
93305ad9f4 Merge remote-tracking branch 'origin/topic/awelzel/make-some-deprecations-errors'
* origin/topic/awelzel/make-some-deprecations-errors:
  Expr: Factor out type tag extraction
  Var: Add version to deprecated initialization
  Stmt: Error on deprecated when/local usage
  Expr: Remove vector scalar operations
  parse.y: Make out-of-scope use errors
  scan.l: Remove unused deprecated_attr
2023-03-21 19:18:41 +01:00
Jan Grashoefer
1882307cf3 Add pcap_file option to supervised nodes.
This allows to start Supervised nodes with a pcap_file argument
rather than interface.

This is based on changes from @J-Gras.
2023-03-21 16:18:02 +01:00
Arne Welzel
38c4611c7e Add a new plugin test with verbose IO source output
This is mostly for documentation/verification purposes of how the IO loop
currently does draining and when it picks up FD based (non packet) IO
source. For example, it shows that currently FD based sources are processed
fairly delayed and that we now also process two timeout sources that are ready.
2023-03-21 09:15:39 +01:00
Arne Welzel
5f1a85803a iomanager/Poll: Add zero-timeout timeout_src also when there's other events ready
This would generally happen the next loop iteration around anyway, but
seems nice to ensure a zero timeout source will be processed at the same
time as sources with ready FDs.
2023-03-21 09:15:33 +01:00
Arne Welzel
c1f51cbf2a Var: Add version to deprecated initialization
This has been added for Zeek 5.0, so mark it for removal in Zeek 6.1.
2023-03-20 21:36:01 +01:00
Arne Welzel
bb80d80218 Stmt: Error on deprecated when/local usage
This has been around since Zeek v4.1, so it was warned about in Zeek 5.0
LTS and we could've removed it with 5.1.

Also removed merge_top_scope() from the zeek::detail namespace, it's
unused now.

Updated the when-aggregates test somehow. I'm not quite sure what had
been tested there :-/
2023-03-20 21:35:53 +01:00
Arne Welzel
5ef62b2de8 Expr: Remove vector scalar operations
These seem to have been deprecated since 2018, so do it now.
Unfortunately the message didn't contain a version when it'll
be removed, but it's been long enough.
2023-03-20 21:35:48 +01:00
Arne Welzel
12d5dca70f parse.y: Make out-of-scope use errors
This was marked to be done in Zeek 5.1, so do it now. The message
didn't include a version, unfortunately, but minimally there was
a comment when it should happen.
2023-03-20 21:35:48 +01:00
Arne Welzel
cf2da5160b dns: Remove AD and CD flags from log
There was a misunderstanding whether to include them by default in
the dns.log, so remove them again.

There had also been a discussion and quirk that AD of a request would
always be overwritten by reply in the dns.log unless the reply is
missing. For now, let users extend dns.log themselves for what best
fits their requirements, rather than adding these flags by default.

Add a btest to print AD and CD flags for smoke testing still.
2023-03-16 10:09:27 +01:00
Christian Kreibich
e25d9bdee9 Merge branch 'topic/christian/gh-2761-default-localnets'
* topic/christian/gh-2761-default-localnets:
  Update NEWS to cover new Site::local_nets behavior.
  Bump external testsuites to reflect updated baselines
  Add btests for new Site::local_nets behavior
  Update plugins.hooks baseline to reflect added config framework activity
  Update scripts.policy.misc.dump-events baseline
  Update btest baselines to reflect the use of local address ranges.
  Tighten local-nets filtering in the logging framework's path-func-column-demote test
  Fix scripts.policy.frameworks.intel.removal test given address locality info
  Treat private address space as site-local by default
  Provide a mechanism to suppress logging of internal config framework activity
2023-03-15 20:05:40 -07:00
Christian Kreibich
fa15d1b258 Bump external testsuites to reflect updated baselines 2023-03-15 17:32:56 -07:00
Christian Kreibich
b2319ba5d9 Add btests for new Site::local_nets behavior 2023-03-15 17:11:08 -07:00
Christian Kreibich
c3cde56a0a Update plugins.hooks baseline to reflect added config framework activity 2023-03-15 17:11:08 -07:00
Christian Kreibich
f73c4e91d1 Update scripts.policy.misc.dump-events baseline 2023-03-15 17:11:08 -07:00
Christian Kreibich
1843e2daae Update btest baselines to reflect the use of local address ranges. 2023-03-15 17:11:04 -07:00
Christian Kreibich
4281d704c1 Tighten local-nets filtering in the logging framework's path-func-column-demote test
With private addresses treated as local ones, this picked up some private-range
flows in the test pcap involved.
2023-03-15 17:01:01 -07:00
Christian Kreibich
d387da9f71 Fix scripts.policy.frameworks.intel.removal test given address locality info
This test used `Site::is_local_addr()` as part of a filtering criterion, perhaps
unintentionally. The fact that it applied to all tested addresses kept a Zeek
process from exiting, failing the test. It also doesn't need to prioritize its
zeek_init() handler.
2023-03-15 17:01:00 -07:00
Johanna Amann
aa3053db00 Merge remote-tracking branch 'origin/topic/johanna/fix-scripts.base.protocols.ssl.x509-invalid-extension-on-openssl-3.1'
* origin/topic/johanna/fix-scripts.base.protocols.ssl.x509-invalid-extension-on-openssl-3.1:
  Fix the x509-invalid-extension test on OpenSSL 3.1
2023-03-15 14:11:10 +01:00
Johanna Amann
dcbc809189 Merge remote-tracking branch 'origin/topic/johanna/certificate-req'
* origin/topic/johanna/certificate-req:
  SSL/TLS CertificateRequest message: Address review feedback
  SSL/TLS: Parse CertificateRequest message
2023-03-15 10:03:25 +01:00
Johanna Amann
3524883008 Fix the x509-invalid-extension test on OpenSSL 3.1
OpenSSL 3.1 switched from outputting UNDEF to not giving a short name in
this case. Luckily this only requires a tiny test change.

We might consider pulling this into older versions, for ease of CI
testing.

Fixes GH-2869
2023-03-15 09:35:10 +01:00
Johanna Amann
b8d658ac77 SSL/TLS CertificateRequest message: Address review feedback
Minor stylistic changes; see https://github.com/zeek/zeek/pull/2855 for
details
2023-03-15 09:19:26 +01:00
Arne Welzel
33090d7a27 Merge branch 'dnssec-flag-parse' of github.com:micrictor/zeek-codespace
* 'dnssec-flag-parse' of github.com:micrictor/zeek-codespace:
  Update external testing commit hash for DNS flag changes
  Parse DNSSEC AD and CD bits

Updated dump-events baseline which seemed unrelated.
2023-03-14 10:35:50 +01:00
Michael
33a6dd131d Update external testing commit hash for DNS flag changes
Update external testing commit hash to match
https://github.com/zeek/zeek-testing/pull/11
2023-03-13 14:35:43 -07:00
Michael R. Torres
fe8390c646 Parse DNSSEC AD and CD bits
Parse authentic data (AD) and checking disabled (CD) bits according to
RFC 2535. Leaves the Z field as-is, in case users are already handling
this elsewhere and depend on the value being the integer for all 3 bits.

https://www.rfc-editor.org/rfc/rfc2535#section-6.1

Fixes #2672
2023-03-13 14:35:06 -07:00
Arne Welzel
c29b98b224 Merge remote-tracking branch 'origin/topic/awelzel/http-content-range-parsing-robustness'
* origin/topic/awelzel/http-content-range-parsing-robustness:
  HTTP: Make Content-Range parsing more robust
2023-03-13 18:41:16 +01:00
Arne Welzel
b21e6f72da HTTP: Make Content-Range parsing more robust
This was exposed by OSS-Fuzz after the HTTP/0.9 changes in zeek/zeek#2851:
We do not check the result of parsing the from and last bytes of a
Content-Range header and would reference uninitialized values on the stack
if these were not valid.

This doesn't seem as bad as it sounds outside of yielding non-sensible values:
If the result was negative, we weird/bailed. If the result was positive, we
already had to treat it with suspicion anyway and the SetPlainDelivery()
logic accounts for that.
2023-03-13 18:00:39 +01:00
Arne Welzel
fbf9d53c44 HTTP: Reset reply_message for HTTP/0.9
OSS-Fuzz tickled an assert when sending a HTTP response before a HTTP/0.9
request. Avoid this by resetting reply_message upon seeing a HTTP/0.9 request.

PCAP was generated artificially: Server sending a reply providing a
Content-Length. Because HTTP/0.9 processing would remove the ContentLine
support analyzer, more data was delivered to the HTTP_Message than
expected, triggering an assert.

This is a follow-up for zeek/zeek#2851.
2023-03-13 14:13:50 +01:00
Arne Welzel
dc068270ea Merge remote-tracking branch 'origin/topic/awelzel/2837-builtin-plugin-ci-take-two'
* origin/topic/awelzel/2837-builtin-plugin-ci-take-two:
  cirrus: Add smoke testing for builtin plugins
  ci/collect-repo-info: Make plugin VERSION reading more robust
  configure: Quote --include-plugins argument
  Fix --no-install-recommends typo
2023-03-13 09:41:12 +01:00
Tim Wojtulewicz
22b98e16aa Merge remote-tracking branch 'origin/topic/awelzel/more-dns-stats'
* origin/topic/awelzel/more-dns-stats:
  get_dns_stats: Expose total cache size and cached text entries
2023-03-12 13:09:53 -07:00