Commit graph

3359 commits

Author SHA1 Message Date
Christian Kreibich
f97a33e14d Fix type clash fatal error with log filters that use $ext_func and $include/$exclude
The logging manager's Manager::TraverseRecord(), called when adding a
log filter to a stream, skipped any fields intoduced by a filter's
$ext_func when such fields weren't mentioned in a $include restriction
or mentioned in an $exclude restriction. This was inconsistent with
Manager::RecordToFilterVals, used when actually writing log entries,
which does include those values.

The result was that the record indices descent in Manager::RecordToFilterVals
expects to find only record values, when in fact only the record
provided by ext_func is present. This leads to type mismatches and
hard Zeek exits like this one:

1300475173.475401 fatal error in zeek/share/zeek//base/init-bare.zeek, line 4810: Val::CONVERTER (string/record) (zeek)

The fix makes ext_func's field additions decisive, meaning the
filter's include/exclude lists don't apply to it. If a user really
wants to override this, they can reset the filter's ext_func back to
our no-op default.

The included btest produces the above error when the fix is not present.
2020-11-10 18:13:16 -08:00
Tim Wojtulewicz
45167953b3 Update external testing repo hashes for canonification updates 2020-11-10 09:02:30 -07:00
Tim Wojtulewicz
ad46a8b717 Merge remote-tracking branch 'origin/topic/timw/1221-unknown-protocols'
* origin/topic/timw/1221-unknown-protocols:
  GH-1221: Add unknown_protocols.log for logging packet analyzer lookup failures
  Remove default_analyzer for Ethernet packet analzyer
2020-11-10 08:42:26 -07:00
Tim Wojtulewicz
c3cf36e135 GH-1221: Add unknown_protocols.log for logging packet analyzer lookup failures 2020-11-09 20:37:26 -07:00
Jon Siwek
b7bd87d9cf GH-1252: rename files with colons for Windows compatibility
A short-term measure so that the Zeek source tree can simply exist on a
Windows filesystem.  For true support, the logic that decides/generates
the filename format will need to change.
2020-11-04 10:29:52 -08:00
Tim Wojtulewicz
cd06bf34c7 GH-1215: Remove dispatch_map from packet analysis, replace with BIF methods for registering dispatches 2020-11-02 19:03:25 +00:00
Tim Wojtulewicz
008cdef712 Merge remote-tracking branch 'origin/topic/jsiwek/better-zeekygen-redef-info'
* origin/topic/jsiwek/better-zeekygen-redef-info:
  Improve how Zeekygen generated record/enum redefinition docs
2020-10-26 11:04:17 -07:00
Tim Wojtulewicz
75caf60990 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1245-tld-validity'
* origin/topic/jsiwek/gh-1245-tld-validity:
  GH-1245: require TLD of hostname literals to start with a letter
2020-10-26 10:42:15 -07:00
Tim Wojtulewicz
5ee06d8535 Merge remote-tracking branch 'origin/topic/timw/1116-coveralls'
* origin/topic/timw/1116-coveralls:
  Add coveralls badge to README.md
  Remove the entire auxil directory from the results
  Prevent non-master builds, add cirrus information to upload
  Add support for sending data to coveralls with lcov_html.sh via Cirrus
2020-10-23 14:48:15 -07:00
Jon Siwek
fb85a19fa8 GH-1245: require TLD of hostname literals to start with a letter 2020-10-23 14:17:10 -07:00
Tim Wojtulewicz
56c1eea5d3 Remove the entire auxil directory from the results 2020-10-23 19:44:51 +00:00
Jon Siwek
73c1af838c GH-251 (revert): remove coercion-to-signed-integer for |x| expressions
For `|x|`, where `x` is an expression with an integral result, an
implicit coercion of that result into signed `int` type no longer takes
place.

This was actually the behavior before Zeek 3.0 as well, but the attempt
to prevent mistakes that easily result from integer literals in Zeek
being unsigned like `|5 - 9|` causing an overflow/wraparound and
yielding a very large number is not generally consistent since overflows
are still generally able to happen in other ways and also in other
contexts besides just absolute-values.  So the preference was to revert
to a behavior that favors consistency.  For reference, see
https://github.com/zeek/zeek/pull/251#issuecomment-713956976
2020-10-22 17:41:53 -07:00
Seth Hall
fe94249ad3 Fixing a test 2020-10-22 13:30:30 -04:00
Seth Hall
552a24e07c Add an option to ignore packets sourced from particular subnets.
It's implemented with a new set[subnet] option named ignore_checksums_nets.

If you populate this set with subnets, any packet with a src address within
that set of subnets will not have it's checksum validated.
2020-10-22 13:23:10 -04:00
Tim Wojtulewicz
2c9c55b058 Prevent non-master builds, add cirrus information to upload 2020-10-22 00:56:20 +00:00
Tim Wojtulewicz
e2053551cd Add support for sending data to coveralls with lcov_html.sh via Cirrus 2020-10-21 21:55:30 +00:00
Robin Sommer
df48b7ea57 Update BTest to 0.63.
New baselines will now be store canonified.

Also setting new MinVersion option in btest.cfg.
2020-10-20 09:01:11 +00:00
Jon Siwek
487d67ac50 Improve how Zeekygen generated record/enum redefinition docs
It now provides a summary of the new fields/enums added by any given
redefinition along with associated commentary.
2020-10-19 18:33:22 -07:00
Jon Siwek
5e164469f3 Merge remote-tracking branch 'origin/topic/vlad/gh-1225'
* origin/topic/vlad/gh-1225:
  Extract length as a uint8
  Change ICMP ND length to a uint16
2020-10-16 10:37:08 -07:00
Johanna Amann
65125121d8 Merge branch 'topic/johanna/GH-169'
* topic/johanna/GH-169:
  Make event ordering deterministic
  dump-events: try to make baseline work on all systems
  Introduce generate_all_events bif and add option to misc/dump-events

Fixes GH-169
2020-10-16 07:11:57 +00:00
Jon Siwek
2c7f68308a Update external test suite commit pointers 2020-10-15 16:06:44 -07:00
Vlad Grigorescu
b5d11d1ace Change ICMP ND length to a uint16 2020-10-15 16:56:05 -05:00
Tim Wojtulewicz
a99b540e46 Rework Sessions::Weird 2020-10-15 13:03:11 -07:00
Tim Wojtulewicz
a7d4364334 Review cleanup 2020-10-15 12:44:45 -07:00
Tim Wojtulewicz
89a9c96673 Update external testing repo hashes 2020-10-15 12:34:37 -07:00
Tim Wojtulewicz
1cf251d1ca Move IP and IP tunnel code from Sessions into packet analyzers 2020-10-15 12:18:30 -07:00
Johanna Amann
5f29f1dca1 Make event ordering deterministic
NetControl::init and filter_change_tracking could basically be raised in
random order.
2020-10-15 16:35:52 +00:00
Johanna Amann
e6805258dc dump-events: try to make baseline work on all systems 2020-10-15 15:35:51 +00:00
Johanna Amann
e95e9b8a5d Merge remote-tracking branch 'origin/master' into topic/johanna/GH-169 2020-10-15 15:31:23 +00:00
Jon Siwek
5f1ee35d31 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1211-improve-already-defined-error' into master
* origin/topic/jsiwek/gh-1211-improve-already-defined-error:
  GH-1211: Improve error message for already-defined functions
2020-10-14 10:48:05 -07:00
Jon Siwek
d62fb3ab9a Merge remote-tracking branch 'origin/topic/jsiwek/zeek-script-args'
* origin/topic/jsiwek/zeek-script-args:
  Improve zeek_script_args test case and documentation
  Apply suggestions from code review
  Add a test for script args.
  Fixed an option processing bug
  Make it possible to pass command line options through to scripts.
2020-10-13 13:34:18 -07:00
Jon Siwek
2d3b4dab74 Improve zeek_script_args test case and documentation 2020-10-13 12:40:53 -07:00
Jon Siwek
86e10bfb7e Merge remote-tracking branch 'origin/topic/vlad/caploss_no_traffic'
- Tweaked the Too_Little_Traffic notice message to avoid
  cluster-specific terminology.

* origin/topic/vlad/caploss_no_traffic:
  Fix scheduling due to network_time being 0 in zeek_init
  Add test for CaptureLoss::Too_Little_Traffic
  Add CaptureLoss::Too_Little_Traffic
  Add CaptureLoss::initial_watch_interval for a quick read on cluster health after startup.
  Documentation update, reference the threshold variable. [nomail] [skip ci]
  Whitespace fixes only [nomail] [skip ci]
2020-10-12 17:05:05 -07:00
Jon Siwek
8c85f2135e GH-1211: Improve error message for already-defined functions 2020-10-12 16:19:19 -07:00
Johanna Amann
f4d9cf5f89 Introduce generate_all_events bif and add option to misc/dump-events
generate_all_events causes all events to be raised internally; this
makes it possible for dump_events to really capture all events (and not
just those that were handled).

Addresses GH-169
2020-10-12 16:21:18 +02:00
Jon Siwek
5904d0708f GH-779: Add "udp-state" signature condition
It accepts "originator" or "responder" states as a way to enforce that
the signature only matches packets in the associated direction.
The "established" state is rejected as an error since it doesn't
have a useful meaning like it does for the "tcp-state" condition.
2020-10-09 13:43:17 -07:00
Seth Hall
ce590ae05f Add a test for script args. 2020-10-08 16:32:54 -04:00
Vlad Grigorescu
c58cf69890 Add test for CaptureLoss::Too_Little_Traffic 2020-10-08 11:34:04 -05:00
Robin Sommer
b0bf9f02c8 Merge remote-tracking branch 'origin/topic/christian/364-logfilter-hooks' into master
(Adding a NEWS entry.)

* origin/topic/christian/364-logfilter-hooks:
  Update testing/btest/scripts/base/frameworks/logging/hooks.zeek
  Btests for log filter policy hooks
  Btest baseline updates to reflect new logging policy hooks
  Migrate existing use of filter predicates to policy hooks
  Support for log filter policy hooks
2020-10-07 08:44:50 +00:00
Robin Sommer
d782c60f19 Unify type comparisions for records.
For records, same_type(r1, r2) would not check if the fields'
attributes match as well. That seems like an oversight, and some
callers of same_type() did indeed add that check on their end. This
commit moves the check into same_type() itself. That generally doesn't
seem make any differences except for a couple of places validating
code, which we update a bit. That in turn leans to slightly different
(better?) error messages for a couple of test cases.
2020-10-05 10:47:46 +00:00
Christian Kreibich
c662237499 Update testing/btest/scripts/base/frameworks/logging/hooks.zeek
Co-authored-by: Robin Sommer <robin@corelight.com>
2020-09-30 13:10:35 -07:00
Christian Kreibich
3d2c3284c7 Btests for log filter policy hooks 2020-09-30 13:10:35 -07:00
Christian Kreibich
2a49715fe6 Btest baseline updates to reflect new logging policy hooks 2020-09-30 13:10:07 -07:00
Jon Siwek
c8545c85d8 Improve broker/ssl_auth_failure.zeek test
There was a race that previously may cause it to not truly test
whether the connecting-process exited due to SSL authentication
failure or just because the listening-process wasn't available yet
(e.g. due to process scheduling variability).
2020-09-29 17:12:03 -07:00
Jon Siwek
b065b6ed61 Merge remote-tracking branch 'origin/topic/robin/filtered-trace'
* origin/topic/robin/filtered-trace:
  Tweak find-filtered-trace to not flag traces if they have non-TCP traffic.
2020-09-25 15:12:04 -07:00
Jon Siwek
c1492942bb Merge branch 'add-dce-rpc-payloads' of https://github.com/ynadji/zeek
- Changed the new stub events to correctly check for existence of
  their associated handler before generating an event

- Added a test case for the new stub event

* 'add-dce-rpc-payloads' of https://github.com/ynadji/zeek:
  Add stub payload to dce_rpc_request and dce_rpc_response
2020-09-25 14:39:30 -07:00
Robin Sommer
1f6fc4415f Tweak find-filtered-trace to not flag traces if they have non-TCP
traffic.

Closes #160.
2020-09-25 11:29:44 +00:00
Jon Siwek
8feca7291b Merge remote-tracking branch 'origin/topic/jsiwek/gh-822-ubsan-ci'
* origin/topic/jsiwek/gh-822-ubsan-ci:
  Fix negative-value-left-shift undefined behavior in patricia trie
  Improve negation of ConstExpr
  Avoid signed integer overflow when combining SMB header PID bits
  Avoid unary negation of INT64_MIN in modp_litoa10
  Avoid double-to-int conversion overflows in modp_dtoa functions
  Fix divide-by-zero in Entropy analyzer
  Fix divide-by-zero in stats/profiling memory usage calculation
  Fix uninitialized field in POP3 fuzzer
  Add framework for running UndefinedBehaviorSanitizer in CI
2020-09-24 08:17:58 -07:00
Jon Siwek
b0db3cc76f Update external test suite commit pointers 2020-09-23 21:26:46 -07:00
Tim Wojtulewicz
bfe31f1c58 Update hashes for external testing repos 2020-09-23 11:13:59 -07:00