Commit graph

15378 commits

Author SHA1 Message Date
Tim Wojtulewicz
8dc641f63f Add a couple of ICMP files to corpus for packet-fuzzer 2022-08-11 09:08:07 -07:00
Tim Wojtulewicz
7f47fa24fd Trick event handlers into returning that they exist during fuzzing 2022-08-11 09:08:07 -07:00
Tim Wojtulewicz
0b8615942d Add http, ftp, imap, and smtp fuzzers and corpora 2022-08-11 08:51:28 -07:00
Tim Wojtulewicz
d54103c6fa Add section to fuzzer README about generating corpus from pcaps 2022-08-11 08:51:28 -07:00
Tim Wojtulewicz
34fb3e7453 Rename fuzzers/README to README.rst so github renders it 2022-08-11 08:51:28 -07:00
Tim Wojtulewicz
313e303fda Merge remote-tracking branch 'AmazingPP/topic/amazingpp/table-values-and-keys'
* AmazingPP/topic/amazingpp/table-values-and-keys:
  Add more test cases
  Add table_keys function
  Add table_values function
2022-08-11 08:49:34 -07:00
Robin Sommer
6d27df548e
Merge remote-tracking branch 'origin/topic/awelzel/1678-disabling-analyzer-hook'
* origin/topic/awelzel/1678-disabling-analyzer-hook:
  Add NEWS entry and zeekygen-smithing for disabling_analyzer()
  Introduce global disabling_analyzer() hook to veto disable_analyzer()
  ssl: Only delete c$ssl$analyzer_id when disabling the analyzer was successful
2022-08-11 16:02:37 +02:00
Arne Welzel
6e54d6c095 Add NEWS entry and zeekygen-smithing for disabling_analyzer() 2022-08-11 13:32:23 +02:00
Robin Sommer
ee7288b1e6
Merge remote-tracking branch 'origin/topic/awelzel/record-field-redef'
* origin/topic/awelzel/record-field-redef:
  Support redef'ing the &log attribute of record fields
2022-08-11 11:59:26 +02:00
Robin Sommer
9de2eceb2a
Merge remote-tracking branch 'origin/topic/awelzel/2262-telemetry-ditch-singleton-metrics'
* origin/topic/awelzel/2262-telemetry-ditch-singleton-metrics:
  telemetry: Remove singleton BIFs and the C++ pieces
2022-08-11 11:54:52 +02:00
Robin Sommer
ceacabe3a2
Merge remote-tracking branch 'origin/topic/awelzel/use-family-num-buckets-upper-bound-at'
* origin/topic/awelzel/use-family-num-buckets-upper-bound-at:
  telemetry: Switch to histogram family bucket and bound accessors
2022-08-11 11:52:58 +02:00
Arne Welzel
abb7f9a509 Introduce global disabling_analyzer() hook to veto disable_analyzer()
This hook can be used to coordinate disabling an analyzer for a given
connection. The contract is simple: Any script can veto a disable_analyzer()
call by breaking from this hook. The decision is local to the script taking
into account any state attached to the connection object or script specific
state stored elsewhere.
A script breaking from the hook takes over the responsibility to call
disable_analyzer() at a later point when it finds the condition due to which
it vetoed fulfilled (which may be never).

Signature:

    disabling_analyzer: hook(c: connection, atype: AllAnalyzers::Tag, aid: count);

Example use-cases are keeping the SSL analyzer enabled for finger-printing
until a certain amount of bytes or packets have been transferred or
similarly the connection duration exceed a certain threshold.

Other example use-cases might be keeping analyzers for SSH, RDP or SSL
enabled for connections from specific subnets.

It's a bit quirky as it makes disable_analyzer() a maybe operation. While log
policy hooks and/or the notice hook have similar semantics, they are not as
stateful. It still seems like a quite powerful primitive.

The disable_analyzer() call in dpd/main.zeek may motivate the addition of a
force flag as a follow-up for situations where the caller "knows better" or
absolutely wants to override.

Closes #1678 #1593.
2022-08-11 09:40:36 +02:00
Arne Welzel
02985b9966 ssl: Only delete c$ssl$analyzer_id when disabling the analyzer was successful
The next patch will have a test script rely on c$ssl$analyzer_id staying
around when disable_analyzer() wasn't successful.

I was tempted to remove the `delete` completely as neither RDP nor SSH
have that and not sure why SSL is special here.
2022-08-11 09:40:34 +02:00
AmazingPP
206e6d28a4 Add more test cases 2022-08-11 13:35:27 +08:00
zeek-bot
a15cd2a7b7 Update doc submodule [nomail] [skip ci] 2022-08-11 00:41:03 +00:00
Christian Kreibich
3a0a702723 Merge branch 'topic/christian/management-bugfixes'
* topic/christian/management-bugfixes:
  Management framework: bump cluster testsuite
  Management framework: log node set in dispatch requests cleanly
  Management framework: log additional node events
  Management framework: upon deployment, make agent log multiple node results
  Management framework: fix early return condition for get-id-value
2022-08-10 14:18:38 -07:00
Christian Kreibich
20e08f0d38 Management framework: bump cluster testsuite 2022-08-10 11:24:10 -07:00
Arne Welzel
985bbe4e57 Support redef'ing the &log attribute of record fields
Add new syntax for adding and removing attributes from record fields:

    redef RecordType$field_name += { &log };
    redef RecordType$field_name -= { &log };

For now this only allowed for the &log attribute as the semantics are clear.
For &default and &optional the semantics aren't obvious and no use-cases have
been identified where those would make sense to change.

This enables a mechanism to add potentially interesting fields to the typical
Info records in base scripts, but letting users opt-into actually including
them into their log. At the same time, users that find specific fields in a
standard log uninteresting can opt-out without using `Log::Filter$exclude`
which can be difficult to use correctly. Patching or forking external packages
to remove columns from a log can also be avoided with this mechanism.

Closes #2000.
2022-08-10 17:27:05 +02:00
Christian Kreibich
fb733eb664 Management framework: log node set in dispatch requests cleanly
Converting to a (sorted) vector both renders the empty set cleanly (without
whitespace) and ensures consistent ordering.
2022-08-09 15:12:39 -07:00
Christian Kreibich
7d4dd22aba Management framework: log additional node events 2022-08-09 15:12:10 -07:00
Christian Kreibich
63291ba2df Management framework: upon deployment, make agent log multiple node results
This erroneously only logged the result of the last node iterated over.
2022-08-09 15:11:31 -07:00
Christian Kreibich
6c3e545306 Management framework: fix early return condition for get-id-value
This erroneously used connectedness of instances, not presence of a deployed
cluster. Without a deployment, there's no point in trying to retrieve global ID
values.
2022-08-09 14:07:16 -07:00
Tim Wojtulewicz
0d0a057d89 Merge remote-tracking branch 'origin/topic/timw/1443-global-type-clash'
* origin/topic/timw/1443-global-type-clash:
  Fix module-scoped type definitions that conflict with existing global ones
2022-08-09 09:50:19 -07:00
Arne Welzel
32ee6960f2 telemetry: Switch to histogram family bucket and bound accessors
Instead of determining the bucket bounds from a concrete metric instance,
broker now provides accessors via the concrete histogram family handles.
2022-08-09 11:00:07 +02:00
Tim Wojtulewicz
514df9f179 Fix module-scoped type definitions that conflict with existing global ones 2022-08-08 15:30:34 -07:00
Tim Wojtulewicz
cb3abccfb1 Merge remote-tracking branch 'ynadji/topic/yacin/2319-add-change-handler-to-site'
* ynadji/topic/yacin/2319-add-change-handler-to-site:
  update plugins.hooks baseline
  lower priority for change handlers
  split update_zones_regex into two functions
  GH-2319: Add change handlers to Site
2022-08-08 11:09:16 -07:00
Tim Wojtulewicz
3da1848279 Merge remote-tracking branch 'AmazingPP/topic/amazingpp/command-line-script-warning'
* AmazingPP/topic/amazingpp/command-line-script-warning:
  Squelch the zeekygen warnings for command line
2022-08-08 11:03:32 -07:00
Arne Welzel
c2ca92d772 Try adding Broker::metrics_import_topics, stuck 2022-08-08 17:20:13 +02:00
Yacin Nadji
84610ed832 update plugins.hooks baseline 2022-08-08 11:52:06 +02:00
Yacin Nadji
825fb1c24a lower priority for change handlers 2022-08-08 11:47:52 +02:00
Yacin Nadji
dc1102e9dd split update_zones_regex into two functions 2022-08-08 11:40:18 +02:00
AmazingPP
9869226e1e Squelch the zeekygen warnings for command line 2022-08-06 11:35:05 +08:00
zeek-bot
857b21ae73 Update doc submodule [nomail] [skip ci] 2022-08-06 00:46:41 +00:00
TheAvgJojo
1e37e91eda UPDATED: improving email address splitting for common comma case 2022-08-05 17:38:54 -04:00
Arne Welzel
acacc6b6c2 telemetry: Remove singleton BIFs and the C++ pieces
The low-level singleton Telemetry BIFs have been removed with the that there
haven't been any users. Singleton metrics can be instantiated by providing
an empty label vector instead and aren't in any way a special concept.

Closes #2262.
2022-08-05 19:33:49 +02:00
Tim Wojtulewicz
9524963da6 Merge remote-tracking branch 'anniebryan/notice-suppression-bug-fix'
* anniebryan/notice-suppression-bug-fix:
  Added test case with back-to-back notices
  Fix notice suppression atomicity bug
2022-08-05 16:21:29 +00:00
Tim Wojtulewicz
66c4e57442 Merge remote-tracking branch 'origin/topic/timw/2034-ifdef-namespace-name'
* origin/topic/timw/2034-ifdef-namespace-name:
  GH-2034: Store module names and use them in lookups for ifdef
2022-08-05 16:20:12 +00:00
Tim Wojtulewicz
1870d26684 GH-2034: Store module names and use them in lookups for ifdef 2022-08-05 15:36:21 +00:00
Robin Sommer
686e740bbe
Merge remote-tracking branch 'origin/topic/awelzel/telemetry-script-land-v0'
* origin/topic/awelzel/telemetry-script-land-v0:
  Introduce telemetry framework
  Bump broker submodule to master.
2022-08-05 17:14:23 +02:00
Yacin Nadji
4a04b563f5 GH-2319: Add change handlers to Site 2022-08-05 16:17:50 +02:00
Arne Welzel
3fe930dbf2 Introduce telemetry framework
Adds base/frameworks/telemetry with wrappers around telemetry.bif
and updates telemetry/Manager to support collecting metrics from
script land.

Add policy/frameworks/telemetry/log for logging of metrics data
into a new telemetry.log and telemetry_histogram.log and add into
local.zeek by default.
2022-08-05 11:43:44 +02:00
zeek-bot
901bc1e983 Update doc submodule [nomail] [skip ci] 2022-08-05 00:35:30 +00:00
Annie Bryan
d916469510 Added test case with back-to-back notices 2022-08-04 11:25:20 +02:00
Annie Bryan
e96d3743b4
Merge branch 'zeek:master' into notice-suppression-bug-fix 2022-08-04 10:03:51 +02:00
Arne Welzel
95fba8fd29 Bump broker submodule to master.
Includes changes to run a collector in the broker::telemetry layer,
as well as pybind upgrade, no-discard removals and formatting changes.
2022-08-04 09:16:10 +02:00
Tim Wojtulewicz
9b546cbb2b Merge remote-tracking branch 'origin/topic/awelzel/config-util-fmt-thread-fmt'
* origin/topic/awelzel/config-util-fmt-thread-fmt:
  input/config: Use thread-safe Fmt() rather than util::fmt() for regex formatting
2022-08-03 10:27:38 -07:00
Tim Wojtulewicz
930fd75ed8 Merge remote-tracking branch 'origin/topic/awelzel/2010-zeek-done-change-handlers'
* origin/topic/awelzel/2010-zeek-done-change-handlers:
  testing/missing-enum-value: redef exit_only_after_terminate=T
  option.bif: Short-circuit option changes when terminating
2022-08-03 10:27:06 -07:00
Tim Wojtulewicz
3ffffe33bc Merge remote-tracking branch 'origin/topic/vern/bit-shift-fixes'
* origin/topic/vern/bit-shift-fixes:
  btest portability fix address review comment about shifting corner-case
  canonicalize filenames for new vector deprecation btest
  updates for gen-C++ maintenance, including skipping some inappropriate tests
  fix for profiling "when" statements
  gen-C++ support for vector bit-shift operations
  corrected wording in some btest comments
  make gen-C++ maintenance scripts directly executable
  ZAM support for bit-shifting
  don't allow deprecated-style mixing of vectors and scaling for shifting leverage restrictions placed on shifting (RHS is always unsigned) split deprecated vector operations into separate test, with separate ZAM baseline
  ZAM fix for vector "in" operator
  ensure that language tests pay attention to .stderr
  fix vector tests, including checking for errors
2022-08-03 09:56:55 -07:00
Annie Bryan
82ee17be8d Fix notice suppression atomicity bug 2022-08-03 16:30:57 +02:00
Vern Paxson
c10885e5f9 btest portability fix
address review comment about shifting corner-case
2022-08-02 19:36:25 -07:00