Commit graph

13652 commits

Author SHA1 Message Date
Arne Welzel
8d19fa23ef Remove unified2 file analyzer 2022-08-19 14:05:00 +02:00
Christian Kreibich
b20f2b5b98 Merge branch 'topic/christian/news-log-preds-coverage'
* topic/christian/news-log-preds-coverage:
  Retroactively cover log filter predicate deprecation/removal in NEWS
2022-08-18 15:57:11 -07:00
Christian Kreibich
a2e05a17e2 Retroactively cover log filter predicate deprecation/removal in NEWS 2022-08-18 14:51:38 -07:00
zeek-bot
bd1190bfeb Update doc submodule [nomail] [skip ci] 2022-08-18 00:42:35 +00:00
Tim Wojtulewicz
20fe96c706 Merge remote-tracking branch 'origin/topic/timw/695-bytestring-to-count'
* origin/topic/timw/695-bytestring-to-count:
  Support other byte lengths in bytestring_to_count
2022-08-17 15:46:17 -07:00
Tim Wojtulewicz
90f0e7a6ea Support other byte lengths in bytestring_to_count 2022-08-17 15:45:30 -07:00
Christian Kreibich
73f20ba761 Merge branch 'topic/christian/help-tweak'
* topic/christian/help-tweak:
  Alignment tweak in help output
2022-08-17 13:31:23 -07:00
Tim Wojtulewicz
60d840c231 Merge remote-tracking branch 'origin/topic/timw/163-event-lookup'
* origin/topic/timw/163-event-lookup:
  GH-163: Use ID name (including module name) to create EventExpr when possible
2022-08-17 13:29:10 -07:00
Tim Wojtulewicz
687dd05ddb GH-163: Use ID name (including module name) to create EventExpr when possible
The change to the capture-loss test is actually a fix for a bug exposed by the
code change. Previously it wasn't firing the scheduled event because of a failed
name lookup. Now that the lookup has been fixed, the event happens twice.
2022-08-17 13:15:01 -07:00
Christian Kreibich
b0001613f5 Alignment tweak in help output 2022-08-17 12:35:43 -07:00
Tim Wojtulewicz
a0a2a18337 Merge remote-tracking branch 'origin/topic/timw/fuzzer-forward-stream'
* origin/topic/timw/fuzzer-forward-stream:
  Use ForwardStream in fuzzers instead of DeliverStream
2022-08-17 09:01:03 -07:00
Tim Wojtulewicz
90e721807a Use ForwardStream in fuzzers instead of DeliverStream 2022-08-17 08:56:37 -07:00
Robin Sommer
fad18cb352
Fix code formatting. 2022-08-17 10:38:33 +02:00
Robin Sommer
598cef21bd
Merge ssh://github.com/fatemabw/zeek
* ssh://github.com/fatemabw/zeek:
  Update options.zeek
  Create out-27
  Add files via upload
  Update src/packet_analysis/protocol/tcp/TCPSessionAdapter.cc
  Updating the weird names to use all lower case
  Fixing whitespaces..
  Fixing clang pre-commit error
  Add check for option 27
  Add the parsed fields for TCP option 27
  Add TCP options bad length check
2022-08-17 09:09:58 +02:00
Robin Sommer
39148ef706
Merge remote-tracking branch 'origin/topic/awelzel/files-log-unrolling'
I removed `deprecated-txhosts-rxhosts-connuids.zeek` from
`local.zeek`, seems preferable not to have a script-to-go-away in the
standard configuration for new users. Also tweaked `NEWS` just a tiny
bit.

* origin/topic/awelzel/files-log-unrolling:
  files.log: Unroll and introduce uid and id fields
2022-08-17 09:01:43 +02:00
Tim Wojtulewicz
963ba61b8e Merge remote-tracking branch 'origin/topic/timw/coverity-fixes'
* origin/topic/timw/coverity-fixes:
  Fix a swath of AUTO_CAUSES_COPY issues reported by Coverity
2022-08-16 09:17:22 -07:00
Arne Welzel
d2314d2666 files.log: Unroll and introduce uid and id fields
This is a script-only change that unrolls File::Info records into
multiple files.log entries if the same file was seen over different
connections by single worker. Consequently, the File::Info record
gets the commonly used uid and id fields added. These fields are
optional for File::Info - a file may be analyzed without relation
to a network connection (e.g by using Input::add_analysis()).

The existing tx_hosts, rx_hosts and conn_uids fields of Files::Info
are not meaningful after this change and removed by default. Therefore,
files.log will have them removed, too.

The tx_hosts, rx_hosts and conn_uids fields can be revived by using the
policy script frameworks/files/deprecated-txhosts-rxhosts-connuids.zeek
included in the distribution. However, with v6.1 this script will be
removed.
2022-08-16 17:22:20 +02:00
Tim Wojtulewicz
2abc82722f Fix a swath of AUTO_CAUSES_COPY issues reported by Coverity 2022-08-15 09:45:46 -07:00
Tim Wojtulewicz
f631551ffb Merge remote-tracking branch 'origin/topic/timw/update-private-tests'
* origin/topic/timw/update-private-tests:
  Update zeek-testing-private commit hash after email regex changes
2022-08-15 09:22:44 -07:00
Tim Wojtulewicz
b5b022449e Update zeek-testing-private commit hash after email regex changes
Related to https://github.com/zeek/zeek/pull/2221
2022-08-12 16:44:25 -07:00
Fatema BW
61244738c8
Update options.zeek 2022-08-11 18:29:12 -07:00
Fatema BW
e92c252b3d
Create out-27 2022-08-11 18:27:28 -07:00
Fatema BW
4609429aa4
Add files via upload 2022-08-11 18:23:02 -07:00
Fatema BW
d9632631ce
Merge branch 'zeek:master' into master 2022-08-11 18:20:31 -07:00
zeek-bot
148f5c1403 Update doc submodule [nomail] [skip ci] 2022-08-12 00:42:29 +00:00
Tim Wojtulewicz
7fe6290974 Merge remote-tracking branch 'micrictor/master'
* micrictor/master:
  Add a field to Modbus/TCP log to indicate the Modbus PDU type
  Add modbus transaction and unit ids to logs
  Enable modbus logging for requests
2022-08-11 11:57:10 -07:00
Tim Wojtulewicz
e618be094a Merge remote-tracking branch 'theavgjojo/master'
* theavgjojo/master:
  UPDATED: improving email address splitting for common comma case
2022-08-11 10:41:18 -07:00
Tim Wojtulewicz
957dc1d6dd Merge remote-tracking branch 'origin/topic/timw/2142-more-fuzzers'
* origin/topic/timw/2142-more-fuzzers:
  Fix a crash related to a broken IPv6 chain
  Add a couple of ICMP files to corpus for packet-fuzzer
  Trick event handlers into returning that they exist during fuzzing
  Add http, ftp, imap, and smtp fuzzers and corpora
  Add section to fuzzer README about generating corpus from pcaps
  Rename fuzzers/README to README.rst so github renders it
2022-08-11 10:30:42 -07:00
Tim Wojtulewicz
95029bd471 Fix a crash related to a broken IPv6 chain 2022-08-11 09:08:07 -07:00
Tim Wojtulewicz
8dc641f63f Add a couple of ICMP files to corpus for packet-fuzzer 2022-08-11 09:08:07 -07:00
Tim Wojtulewicz
7f47fa24fd Trick event handlers into returning that they exist during fuzzing 2022-08-11 09:08:07 -07:00
Tim Wojtulewicz
0b8615942d Add http, ftp, imap, and smtp fuzzers and corpora 2022-08-11 08:51:28 -07:00
Tim Wojtulewicz
d54103c6fa Add section to fuzzer README about generating corpus from pcaps 2022-08-11 08:51:28 -07:00
Tim Wojtulewicz
34fb3e7453 Rename fuzzers/README to README.rst so github renders it 2022-08-11 08:51:28 -07:00
Tim Wojtulewicz
313e303fda Merge remote-tracking branch 'AmazingPP/topic/amazingpp/table-values-and-keys'
* AmazingPP/topic/amazingpp/table-values-and-keys:
  Add more test cases
  Add table_keys function
  Add table_values function
2022-08-11 08:49:34 -07:00
Robin Sommer
6d27df548e
Merge remote-tracking branch 'origin/topic/awelzel/1678-disabling-analyzer-hook'
* origin/topic/awelzel/1678-disabling-analyzer-hook:
  Add NEWS entry and zeekygen-smithing for disabling_analyzer()
  Introduce global disabling_analyzer() hook to veto disable_analyzer()
  ssl: Only delete c$ssl$analyzer_id when disabling the analyzer was successful
2022-08-11 16:02:37 +02:00
Arne Welzel
6e54d6c095 Add NEWS entry and zeekygen-smithing for disabling_analyzer() 2022-08-11 13:32:23 +02:00
Robin Sommer
ee7288b1e6
Merge remote-tracking branch 'origin/topic/awelzel/record-field-redef'
* origin/topic/awelzel/record-field-redef:
  Support redef'ing the &log attribute of record fields
2022-08-11 11:59:26 +02:00
Robin Sommer
9de2eceb2a
Merge remote-tracking branch 'origin/topic/awelzel/2262-telemetry-ditch-singleton-metrics'
* origin/topic/awelzel/2262-telemetry-ditch-singleton-metrics:
  telemetry: Remove singleton BIFs and the C++ pieces
2022-08-11 11:54:52 +02:00
Robin Sommer
ceacabe3a2
Merge remote-tracking branch 'origin/topic/awelzel/use-family-num-buckets-upper-bound-at'
* origin/topic/awelzel/use-family-num-buckets-upper-bound-at:
  telemetry: Switch to histogram family bucket and bound accessors
2022-08-11 11:52:58 +02:00
Arne Welzel
abb7f9a509 Introduce global disabling_analyzer() hook to veto disable_analyzer()
This hook can be used to coordinate disabling an analyzer for a given
connection. The contract is simple: Any script can veto a disable_analyzer()
call by breaking from this hook. The decision is local to the script taking
into account any state attached to the connection object or script specific
state stored elsewhere.
A script breaking from the hook takes over the responsibility to call
disable_analyzer() at a later point when it finds the condition due to which
it vetoed fulfilled (which may be never).

Signature:

    disabling_analyzer: hook(c: connection, atype: AllAnalyzers::Tag, aid: count);

Example use-cases are keeping the SSL analyzer enabled for finger-printing
until a certain amount of bytes or packets have been transferred or
similarly the connection duration exceed a certain threshold.

Other example use-cases might be keeping analyzers for SSH, RDP or SSL
enabled for connections from specific subnets.

It's a bit quirky as it makes disable_analyzer() a maybe operation. While log
policy hooks and/or the notice hook have similar semantics, they are not as
stateful. It still seems like a quite powerful primitive.

The disable_analyzer() call in dpd/main.zeek may motivate the addition of a
force flag as a follow-up for situations where the caller "knows better" or
absolutely wants to override.

Closes #1678 #1593.
2022-08-11 09:40:36 +02:00
Arne Welzel
02985b9966 ssl: Only delete c$ssl$analyzer_id when disabling the analyzer was successful
The next patch will have a test script rely on c$ssl$analyzer_id staying
around when disable_analyzer() wasn't successful.

I was tempted to remove the `delete` completely as neither RDP nor SSH
have that and not sure why SSL is special here.
2022-08-11 09:40:34 +02:00
AmazingPP
206e6d28a4 Add more test cases 2022-08-11 13:35:27 +08:00
zeek-bot
a15cd2a7b7 Update doc submodule [nomail] [skip ci] 2022-08-11 00:41:03 +00:00
Christian Kreibich
3a0a702723 Merge branch 'topic/christian/management-bugfixes'
* topic/christian/management-bugfixes:
  Management framework: bump cluster testsuite
  Management framework: log node set in dispatch requests cleanly
  Management framework: log additional node events
  Management framework: upon deployment, make agent log multiple node results
  Management framework: fix early return condition for get-id-value
2022-08-10 14:18:38 -07:00
Christian Kreibich
20e08f0d38 Management framework: bump cluster testsuite 2022-08-10 11:24:10 -07:00
Arne Welzel
985bbe4e57 Support redef'ing the &log attribute of record fields
Add new syntax for adding and removing attributes from record fields:

    redef RecordType$field_name += { &log };
    redef RecordType$field_name -= { &log };

For now this only allowed for the &log attribute as the semantics are clear.
For &default and &optional the semantics aren't obvious and no use-cases have
been identified where those would make sense to change.

This enables a mechanism to add potentially interesting fields to the typical
Info records in base scripts, but letting users opt-into actually including
them into their log. At the same time, users that find specific fields in a
standard log uninteresting can opt-out without using `Log::Filter$exclude`
which can be difficult to use correctly. Patching or forking external packages
to remove columns from a log can also be avoided with this mechanism.

Closes #2000.
2022-08-10 17:27:05 +02:00
Christian Kreibich
fb733eb664 Management framework: log node set in dispatch requests cleanly
Converting to a (sorted) vector both renders the empty set cleanly (without
whitespace) and ensures consistent ordering.
2022-08-09 15:12:39 -07:00
Christian Kreibich
7d4dd22aba Management framework: log additional node events 2022-08-09 15:12:10 -07:00
Christian Kreibich
63291ba2df Management framework: upon deployment, make agent log multiple node results
This erroneously only logged the result of the last node iterated over.
2022-08-09 15:11:31 -07:00