Commit graph

3411 commits

Author SHA1 Message Date
Jon Siwek
7393fc6d24 Update alternative plugins.hooks baselines 2020-12-13 13:24:47 -08:00
Jon Siwek
30df2c58c0 Merge branch 'master' into topic/vern/script-inlining 2020-12-13 10:42:26 -08:00
Jon Siwek
a35cd2a726 GH-1321: Prevent compounding of connection_status_update event timers
Particularly for ICMP connections, a new timer got added every time a
`connection` record was updated even if there was still a pending timer
for that connection.
2020-12-08 11:20:02 -08:00
Jon Siwek
0b8535b879 Merge remote-tracking branch 'origin/topic/vlad/gh-1286'
Merge adjustments:
- Rewrote the check for error response as a switch statement to
  fix compiler warning about signed/unsigned comparison and also
  to just simplify/clarify the logic.
- Changed the btest to use `zeek -b`.

* origin/topic/vlad/gh-1286:
  Add tests for new SMB3 multichannel support
  Fix SMB2 response status parsing. Fixes #1286
2020-12-07 16:08:04 -08:00
Vlad Grigorescu
f30ca69402 Add tests for new SMB3 multichannel support 2020-12-07 15:35:04 -06:00
Johanna Amann
8955e6c060 Merge branch 'master' of github.com:zeek/zeek 2020-12-07 18:23:18 +00:00
Christian Kreibich
b04082c140 Additional use of btest-diff --binary 2020-12-06 20:19:52 -08:00
Christian Kreibich
17cd740519 Update btest-diff calls on binary files to using "--binary"
This converts Johanna's TEST_DIFF_CANONIFIER removals via "env -u" to
using the new "btest-diff --binary", and updates the affected baselines.
2020-12-06 20:19:52 -08:00
Johanna Amann
5f6b95c027 Update external baseline commit hashes 2020-12-06 20:19:52 -08:00
Johanna Amann
442cb40db8 Fix binary baseline & line-end problem
By default all baslines are run through diff-remove-timestamp. On a BSD
sed implementation, this means that a newline is added to the end of the
file, if no newline was there originally. This behavior differs from GNU
sed, which does not add a newline.

In this commit we unify this behavior by always adding a newline, even
when using GNU sed. This commit also disables the canonifier for a bunch
of binary baselines, so we do not have to change them.
2020-12-06 20:19:52 -08:00
Johanna Amann
7040e30431 Fix diff-remove-abspath on OSX
\t does not work on OS-X and just matches the letter t. This commit
replaces it with a literal tab instead.
2020-12-06 20:19:52 -08:00
Christian Kreibich
c0bd02e18f Canonifier improvements for the scripts.base.frameworks.logging.ascii-double test
This now more surgically applies canonifiers so that the double-format
numerical output isn't itself canonified.
2020-12-06 20:19:52 -08:00
Christian Kreibich
0b674eb851 Baseline refresh to reflect btest 0.64 2020-12-06 20:19:49 -08:00
Christian Kreibich
3984d99a86 Make diff-remove-abspath canonifier match on non-whitespace paths only
This avoids swallowing multiple separate paths separated by unrelated
content into one substitution, like here:

orig_p=59856<...>/tcp] -> orig_p=59856/tcp, resp_h=192.150.187.43, resp_p=80/tcp]
2020-12-06 18:19:06 -08:00
Christian Kreibich
3ca804421e Harden diff-remove-timestamps canonifier
This pattern got mislead by matching suffixes of other numbers, and
noramlizing exact 0-timestamps isn't really required.

- Remove eplicit "0.000000" number pattern from timestamp normalization

- Require beginning of line or non-numeric character before the
  beginning of the number replacement
2020-12-06 18:19:06 -08:00
Jon Siwek
ff7d9e3d72 Merge branch 'master' into topic/vern/script-inlining 2020-12-04 16:18:14 -08:00
Johanna Amann
47ceac2491 Merge remote-tracking branch 'origin/topic/johanna/GH-348'
* origin/topic/johanna/GH-348:
  Sumstats: allow users to manage epoch manually
  Sumstats: epoch_finished was not called under certain circumstances
2020-12-04 18:49:53 +00:00
Jon Siwek
71f9340e75 Merge branch 'add-X-to-double' of https://github.com/ynadji/zeek
- Minor whitespace/grammar/doc tweaks during merge

* 'add-X-to-double' of https://github.com/ynadji/zeek:
  Add `count_to_double` and `int_to_double` bif functions
2020-12-02 12:54:08 -08:00
Jon Siwek
3605e04d83 Update minimum required CMake to 3.5
Also now uses CMake's ENABLE_EXPORTS target property for the zeek
executable to ensure symbols are visible to plugins.  Prior to CMake
3.4, the policy was to export symbols by default for certain platforms,
but later versions need either the explicit target property or policy.
2020-12-01 22:13:52 -08:00
Yacin Nadji
607af85ac1 Add count_to_double and int_to_double bif functions 2020-12-01 16:35:49 -05:00
Tim Wojtulewicz
49293c080d Merge remote-tracking branch 'origin/topic/timw/1184-additional-weird-info'
* origin/topic/timw/1184-additional-weird-info:
  GH-1184: Add 'source' field to weird log denoting where the weird was reported
2020-12-01 10:18:00 -07:00
Tim Wojtulewicz
e27008ef26 GH-1184: Add 'source' field to weird log denoting where the weird was reported 2020-12-01 09:34:37 -07:00
Robin Sommer
eccbbb4476 Merge remote-tracking branch 'origin/topic/robin/gh-1179-plugin-loading'
* origin/topic/robin/gh-1179-plugin-loading:
  Retry loading plugins on failure to resolve to dependencies.
  Fix use of deprecated functionality in test.
  When attempting to activate a plugin, load dynamic libraries first.
  Add test creating multiple plugins with load dependencies.
2020-12-01 14:51:49 +00:00
Jon Siwek
7d68f2ce9d Remove Python 2 compatibility logic in httpd test script 2020-11-25 14:57:09 -08:00
Jon Siwek
9717c623c2 Update Python invocations to explicit python3 2020-11-25 14:57:09 -08:00
Robin Sommer
b780bc146f Fix use of deprecated functionality in test. 2020-11-24 17:00:20 +00:00
Robin Sommer
3ebfcdf0ae Add test creating multiple plugins with load dependencies.
If we load plugins purely alphabetically, the 1st Zeek run in the test
will success while the 2nd will fail.
2020-11-24 16:59:11 +00:00
Vern Paxson
6894e4c5e1 updates from test suite 2020-11-19 17:13:36 -08:00
Vern Paxson
c42586af2c inlining of Zeek script functions 2020-11-19 16:16:59 -08:00
Tim Wojtulewicz
01ec5ebdcd Reverts the regex change in dead3226a5.
The regex change broke some of the external tests. I added some more cases
to the regular email btest to hopefully cover all of the cases better.
2020-11-18 15:35:17 -07:00
Tim Wojtulewicz
a56fbe408f Merge remote-tracking branch 'christina23/master'
* christina23/master:
  Added unit tests for regex fix
  Improved regex for SMTP parsing
2020-11-17 15:01:21 -07:00
Jon Siwek
824ff81bf6 Merge remote-tracking branch 'origin/topic/johanna/1033'
- Fixed signed/unsigned comparison compiler warning
- Did other minor changes to address efficiency nitpicks

* origin/topic/johanna/1033:
  BrokerStore <-> Zeek Tables: support complex indices
2020-11-17 13:34:26 -08:00
Johanna Amann
44ae8f9e8f BrokerStore <-> Zeek Tables: support complex indices
This change adds support for complex indexes for sets and tables. With
this change, sets with indexes like:

set[string, count, count]

will function. Before this change, Zeek raised an error message in these
cases.

Addresses GH-1033
2020-11-17 14:48:10 +00:00
Jon Siwek
6ed4850a26 Merge remote-tracking branch 'origin/topic/jsiwek/gh-352-improve-sql-injection-regex' into master
* origin/topic/jsiwek/gh-352-improve-sql-injection-regex:
  GH-352: Improve HTTP::match_sql_injection_uri regex
2020-11-16 11:51:04 -08:00
Jon Siwek
02c0b33b54 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1264-ssh-host-key-fingerprints' into master
* origin/topic/jsiwek/gh-1264-ssh-host-key-fingerprints:
  Simply ssh/main.zeek by using "ssh_server_host_key" for fingerprinting
  Deprecate "ssh1_server_host_key" parameters *e* and *p*
  GH-1264: Implement "ssh_server_host_key" event
2020-11-16 11:22:37 -08:00
christina23
bb7af9a823 Added unit tests for regex fix 2020-11-16 11:43:35 -05:00
Johanna Amann
7362f30c3a Sumstats: allow users to manage epoch manually
This change allows users to specify an epoch length of 0, which means
that the user manually has to finish the epochs. A new next_epoch
function is introduced to allow users to manually end epochs.

Addresses GH-348
2020-11-16 15:48:46 +00:00
Jon Siwek
331b94db39 Simply ssh/main.zeek by using "ssh_server_host_key" for fingerprinting 2020-11-14 08:40:27 -08:00
Brevet Vivien
3769ed6c66 [SSH] Handle SSH version 1.99
SSH can set in its identification a version 1.99 (SSH-1.99-xxx).
That means the client/server is compatible with SSHv1 and SSHv2.
So the version choice depends of the both side.

1.99 : 1.99 => 2.0
1.99 : 1.x  => 1.x
1.99 : 2.0  => 2.O

(see "Compatibility With Old SSH Versions" in RFC 4253)
2020-11-14 15:33:34 +01:00
Jon Siwek
45449dad72 Deprecate "ssh1_server_host_key" parameters *e* and *p*
They are named such that *e* is actually the modulus, not the exponent.
The replacement parameters are named *exponent* and *modulus* for
clarity.
2020-11-13 22:58:56 -08:00
Jon Siwek
bd40a97a78 GH-1264: Implement "ssh_server_host_key" event
This event provides host key fingerprints for both SSH1 and SSH2.
2020-11-13 22:58:56 -08:00
Jon Siwek
ae923106f1 GH-352: Improve HTTP::match_sql_injection_uri regex
Changes \x00-\x37 ranges to \x00-\x1f with assumption that the former
was attempting to match ASCII control characters, but mistook an octal
range for hex.  This change reduces some false positives.
2020-11-12 16:19:35 -08:00
Jon Siwek
fc114069b0 Merge remote-tracking branch 'origin/topic/jsiwek/unknown-protocol-options'
* origin/topic/jsiwek/unknown-protocol-options:
  Move UnknownProtocol options to init-bare.zeek
  Coverity 1436183: Initialize packet_analysis::Manager fields
2020-11-12 14:35:01 -08:00
Jon Siwek
4618df762d Merge remote-tracking branch 'origin/topic/jsiwek/gh-1273-size-expr-any'
* origin/topic/jsiwek/gh-1273-size-expr-any:
  GH-1273: Change SizeExpr to yield "any" type when operating on "any"
2020-11-12 14:30:52 -08:00
Jon Siwek
aab99b743d Merge remote-tracking branch 'origin/topic/jsiwek/type-alias-introspection'
* origin/topic/jsiwek/type-alias-introspection:
  Add enum_names() BIF to return names of an enum type's values
  Add type_aliases() BIF for introspecting type-names of types/values
  Change Type::type_aliases map to store IntrusivePtr
  Fix lookup_ID() BIF to return enum values
2020-11-12 14:30:32 -08:00
Robin Sommer
78fc5615a5 Merge remote-tracking branch 'origin/topic/christian/fix-extfunc-includes'
* origin/topic/christian/fix-extfunc-includes:
  Fix type clash fatal error with log filters that use $ext_func and $include/$exclude
2020-11-12 10:33:48 +00:00
Jon Siwek
89af6f2004 Move UnknownProtocol options to init-bare.zeek
Otherwise the `unknown_protocol` event cannot be used independently
from `policy/mic/unknown-protocols.zeek`.
2020-11-11 12:58:38 -08:00
Tim Wojtulewicz
a6a4b976ec Support for additional DNS RR Type: LOC[29], SSHFP[44], NSEC3PARAM[51], custom BIND9 signaling[65534] 2020-11-11 13:35:51 -07:00
Christian Kreibich
f97a33e14d Fix type clash fatal error with log filters that use $ext_func and $include/$exclude
The logging manager's Manager::TraverseRecord(), called when adding a
log filter to a stream, skipped any fields intoduced by a filter's
$ext_func when such fields weren't mentioned in a $include restriction
or mentioned in an $exclude restriction. This was inconsistent with
Manager::RecordToFilterVals, used when actually writing log entries,
which does include those values.

The result was that the record indices descent in Manager::RecordToFilterVals
expects to find only record values, when in fact only the record
provided by ext_func is present. This leads to type mismatches and
hard Zeek exits like this one:

1300475173.475401 fatal error in zeek/share/zeek//base/init-bare.zeek, line 4810: Val::CONVERTER (string/record) (zeek)

The fix makes ext_func's field additions decisive, meaning the
filter's include/exclude lists don't apply to it. If a user really
wants to override this, they can reset the filter's ext_func back to
our no-op default.

The included btest produces the above error when the fix is not present.
2020-11-10 18:13:16 -08:00
Tim Wojtulewicz
45167953b3 Update external testing repo hashes for canonification updates 2020-11-10 09:02:30 -07:00