Commit graph

3882 commits

Author SHA1 Message Date
Tim Wojtulewicz
01ec5ebdcd Reverts the regex change in dead3226a5.
The regex change broke some of the external tests. I added some more cases
to the regular email btest to hopefully cover all of the cases better.
2020-11-18 15:35:17 -07:00
Tim Wojtulewicz
a56fbe408f Merge remote-tracking branch 'christina23/master'
* christina23/master:
  Added unit tests for regex fix
  Improved regex for SMTP parsing
2020-11-17 15:01:21 -07:00
Jon Siwek
824ff81bf6 Merge remote-tracking branch 'origin/topic/johanna/1033'
- Fixed signed/unsigned comparison compiler warning
- Did other minor changes to address efficiency nitpicks

* origin/topic/johanna/1033:
  BrokerStore <-> Zeek Tables: support complex indices
2020-11-17 13:34:26 -08:00
Johanna Amann
44ae8f9e8f BrokerStore <-> Zeek Tables: support complex indices
This change adds support for complex indexes for sets and tables. With
this change, sets with indexes like:

set[string, count, count]

will function. Before this change, Zeek raised an error message in these
cases.

Addresses GH-1033
2020-11-17 14:48:10 +00:00
Jon Siwek
6ed4850a26 Merge remote-tracking branch 'origin/topic/jsiwek/gh-352-improve-sql-injection-regex' into master
* origin/topic/jsiwek/gh-352-improve-sql-injection-regex:
  GH-352: Improve HTTP::match_sql_injection_uri regex
2020-11-16 11:51:04 -08:00
Jon Siwek
02c0b33b54 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1264-ssh-host-key-fingerprints' into master
* origin/topic/jsiwek/gh-1264-ssh-host-key-fingerprints:
  Simply ssh/main.zeek by using "ssh_server_host_key" for fingerprinting
  Deprecate "ssh1_server_host_key" parameters *e* and *p*
  GH-1264: Implement "ssh_server_host_key" event
2020-11-16 11:22:37 -08:00
christina23
bb7af9a823 Added unit tests for regex fix 2020-11-16 11:43:35 -05:00
Johanna Amann
7362f30c3a Sumstats: allow users to manage epoch manually
This change allows users to specify an epoch length of 0, which means
that the user manually has to finish the epochs. A new next_epoch
function is introduced to allow users to manually end epochs.

Addresses GH-348
2020-11-16 15:48:46 +00:00
Jon Siwek
331b94db39 Simply ssh/main.zeek by using "ssh_server_host_key" for fingerprinting 2020-11-14 08:40:27 -08:00
Brevet Vivien
3769ed6c66 [SSH] Handle SSH version 1.99
SSH can set in its identification a version 1.99 (SSH-1.99-xxx).
That means the client/server is compatible with SSHv1 and SSHv2.
So the version choice depends of the both side.

1.99 : 1.99 => 2.0
1.99 : 1.x  => 1.x
1.99 : 2.0  => 2.O

(see "Compatibility With Old SSH Versions" in RFC 4253)
2020-11-14 15:33:34 +01:00
Jon Siwek
45449dad72 Deprecate "ssh1_server_host_key" parameters *e* and *p*
They are named such that *e* is actually the modulus, not the exponent.
The replacement parameters are named *exponent* and *modulus* for
clarity.
2020-11-13 22:58:56 -08:00
Jon Siwek
bd40a97a78 GH-1264: Implement "ssh_server_host_key" event
This event provides host key fingerprints for both SSH1 and SSH2.
2020-11-13 22:58:56 -08:00
Jon Siwek
ae923106f1 GH-352: Improve HTTP::match_sql_injection_uri regex
Changes \x00-\x37 ranges to \x00-\x1f with assumption that the former
was attempting to match ASCII control characters, but mistook an octal
range for hex.  This change reduces some false positives.
2020-11-12 16:19:35 -08:00
Jon Siwek
fc114069b0 Merge remote-tracking branch 'origin/topic/jsiwek/unknown-protocol-options'
* origin/topic/jsiwek/unknown-protocol-options:
  Move UnknownProtocol options to init-bare.zeek
  Coverity 1436183: Initialize packet_analysis::Manager fields
2020-11-12 14:35:01 -08:00
Jon Siwek
4618df762d Merge remote-tracking branch 'origin/topic/jsiwek/gh-1273-size-expr-any'
* origin/topic/jsiwek/gh-1273-size-expr-any:
  GH-1273: Change SizeExpr to yield "any" type when operating on "any"
2020-11-12 14:30:52 -08:00
Jon Siwek
aab99b743d Merge remote-tracking branch 'origin/topic/jsiwek/type-alias-introspection'
* origin/topic/jsiwek/type-alias-introspection:
  Add enum_names() BIF to return names of an enum type's values
  Add type_aliases() BIF for introspecting type-names of types/values
  Change Type::type_aliases map to store IntrusivePtr
  Fix lookup_ID() BIF to return enum values
2020-11-12 14:30:32 -08:00
Robin Sommer
78fc5615a5 Merge remote-tracking branch 'origin/topic/christian/fix-extfunc-includes'
* origin/topic/christian/fix-extfunc-includes:
  Fix type clash fatal error with log filters that use $ext_func and $include/$exclude
2020-11-12 10:33:48 +00:00
Jon Siwek
89af6f2004 Move UnknownProtocol options to init-bare.zeek
Otherwise the `unknown_protocol` event cannot be used independently
from `policy/mic/unknown-protocols.zeek`.
2020-11-11 12:58:38 -08:00
Tim Wojtulewicz
a6a4b976ec Support for additional DNS RR Type: LOC[29], SSHFP[44], NSEC3PARAM[51], custom BIND9 signaling[65534] 2020-11-11 13:35:51 -07:00
Christian Kreibich
f97a33e14d Fix type clash fatal error with log filters that use $ext_func and $include/$exclude
The logging manager's Manager::TraverseRecord(), called when adding a
log filter to a stream, skipped any fields intoduced by a filter's
$ext_func when such fields weren't mentioned in a $include restriction
or mentioned in an $exclude restriction. This was inconsistent with
Manager::RecordToFilterVals, used when actually writing log entries,
which does include those values.

The result was that the record indices descent in Manager::RecordToFilterVals
expects to find only record values, when in fact only the record
provided by ext_func is present. This leads to type mismatches and
hard Zeek exits like this one:

1300475173.475401 fatal error in zeek/share/zeek//base/init-bare.zeek, line 4810: Val::CONVERTER (string/record) (zeek)

The fix makes ext_func's field additions decisive, meaning the
filter's include/exclude lists don't apply to it. If a user really
wants to override this, they can reset the filter's ext_func back to
our no-op default.

The included btest produces the above error when the fix is not present.
2020-11-10 18:13:16 -08:00
Tim Wojtulewicz
45167953b3 Update external testing repo hashes for canonification updates 2020-11-10 09:02:30 -07:00
Tim Wojtulewicz
ad46a8b717 Merge remote-tracking branch 'origin/topic/timw/1221-unknown-protocols'
* origin/topic/timw/1221-unknown-protocols:
  GH-1221: Add unknown_protocols.log for logging packet analyzer lookup failures
  Remove default_analyzer for Ethernet packet analzyer
2020-11-10 08:42:26 -07:00
Tim Wojtulewicz
c3cf36e135 GH-1221: Add unknown_protocols.log for logging packet analyzer lookup failures 2020-11-09 20:37:26 -07:00
Jon Siwek
d4528162d1 GH-1273: Change SizeExpr to yield "any" type when operating on "any" 2020-11-09 19:03:42 -08:00
Jon Siwek
f147c47271 Add enum_names() BIF to return names of an enum type's values 2020-11-06 18:31:41 -08:00
Jon Siwek
77514f234f Add type_aliases() BIF for introspecting type-names of types/values 2020-11-06 17:18:44 -08:00
Jon Siwek
bfb7afc600 Fix lookup_ID() BIF to return enum values
Looking up an enum value from a string equal to its name previously
returned "<no ID value>".
2020-11-06 15:37:54 -08:00
Jon Siwek
b7bd87d9cf GH-1252: rename files with colons for Windows compatibility
A short-term measure so that the Zeek source tree can simply exist on a
Windows filesystem.  For true support, the logic that decides/generates
the filename format will need to change.
2020-11-04 10:29:52 -08:00
Tim Wojtulewicz
cd06bf34c7 GH-1215: Remove dispatch_map from packet analysis, replace with BIF methods for registering dispatches 2020-11-02 19:03:25 +00:00
Tim Wojtulewicz
008cdef712 Merge remote-tracking branch 'origin/topic/jsiwek/better-zeekygen-redef-info'
* origin/topic/jsiwek/better-zeekygen-redef-info:
  Improve how Zeekygen generated record/enum redefinition docs
2020-10-26 11:04:17 -07:00
Tim Wojtulewicz
75caf60990 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1245-tld-validity'
* origin/topic/jsiwek/gh-1245-tld-validity:
  GH-1245: require TLD of hostname literals to start with a letter
2020-10-26 10:42:15 -07:00
Tim Wojtulewicz
5ee06d8535 Merge remote-tracking branch 'origin/topic/timw/1116-coveralls'
* origin/topic/timw/1116-coveralls:
  Add coveralls badge to README.md
  Remove the entire auxil directory from the results
  Prevent non-master builds, add cirrus information to upload
  Add support for sending data to coveralls with lcov_html.sh via Cirrus
2020-10-23 14:48:15 -07:00
Jon Siwek
fb85a19fa8 GH-1245: require TLD of hostname literals to start with a letter 2020-10-23 14:17:10 -07:00
Tim Wojtulewicz
56c1eea5d3 Remove the entire auxil directory from the results 2020-10-23 19:44:51 +00:00
Jon Siwek
73c1af838c GH-251 (revert): remove coercion-to-signed-integer for |x| expressions
For `|x|`, where `x` is an expression with an integral result, an
implicit coercion of that result into signed `int` type no longer takes
place.

This was actually the behavior before Zeek 3.0 as well, but the attempt
to prevent mistakes that easily result from integer literals in Zeek
being unsigned like `|5 - 9|` causing an overflow/wraparound and
yielding a very large number is not generally consistent since overflows
are still generally able to happen in other ways and also in other
contexts besides just absolute-values.  So the preference was to revert
to a behavior that favors consistency.  For reference, see
https://github.com/zeek/zeek/pull/251#issuecomment-713956976
2020-10-22 17:41:53 -07:00
Seth Hall
fe94249ad3 Fixing a test 2020-10-22 13:30:30 -04:00
Seth Hall
552a24e07c Add an option to ignore packets sourced from particular subnets.
It's implemented with a new set[subnet] option named ignore_checksums_nets.

If you populate this set with subnets, any packet with a src address within
that set of subnets will not have it's checksum validated.
2020-10-22 13:23:10 -04:00
Tim Wojtulewicz
2c9c55b058 Prevent non-master builds, add cirrus information to upload 2020-10-22 00:56:20 +00:00
Tim Wojtulewicz
e2053551cd Add support for sending data to coveralls with lcov_html.sh via Cirrus 2020-10-21 21:55:30 +00:00
Robin Sommer
df48b7ea57 Update BTest to 0.63.
New baselines will now be store canonified.

Also setting new MinVersion option in btest.cfg.
2020-10-20 09:01:11 +00:00
Jon Siwek
487d67ac50 Improve how Zeekygen generated record/enum redefinition docs
It now provides a summary of the new fields/enums added by any given
redefinition along with associated commentary.
2020-10-19 18:33:22 -07:00
Jon Siwek
5e164469f3 Merge remote-tracking branch 'origin/topic/vlad/gh-1225'
* origin/topic/vlad/gh-1225:
  Extract length as a uint8
  Change ICMP ND length to a uint16
2020-10-16 10:37:08 -07:00
Johanna Amann
65125121d8 Merge branch 'topic/johanna/GH-169'
* topic/johanna/GH-169:
  Make event ordering deterministic
  dump-events: try to make baseline work on all systems
  Introduce generate_all_events bif and add option to misc/dump-events

Fixes GH-169
2020-10-16 07:11:57 +00:00
Jon Siwek
2c7f68308a Update external test suite commit pointers 2020-10-15 16:06:44 -07:00
Vlad Grigorescu
b5d11d1ace Change ICMP ND length to a uint16 2020-10-15 16:56:05 -05:00
Tim Wojtulewicz
a99b540e46 Rework Sessions::Weird 2020-10-15 13:03:11 -07:00
Tim Wojtulewicz
a7d4364334 Review cleanup 2020-10-15 12:44:45 -07:00
Tim Wojtulewicz
89a9c96673 Update external testing repo hashes 2020-10-15 12:34:37 -07:00
Tim Wojtulewicz
1cf251d1ca Move IP and IP tunnel code from Sessions into packet analyzers 2020-10-15 12:18:30 -07:00
Johanna Amann
5f29f1dca1 Make event ordering deterministic
NetControl::init and filter_change_tracking could basically be raised in
random order.
2020-10-15 16:35:52 +00:00