Commit graph

4015 commits

Author SHA1 Message Date
Tim Wojtulewicz
8829490045 Merge remote-tracking branch 'origin/topic/awelzel/2075-if-conditional-parsing'
* origin/topic/awelzel/2075-if-conditional-parsing:
  parse.y/anonymous_function: Allow conditionals between begin_lambda and lambda_body
2022-08-23 20:28:17 -07:00
Christian Kreibich
c887bcb517 Merge remote-tracking branch 'origin/topic/awelzel/simple-scan-over-scan-zeek'
* origin/topic/awelzel/simple-scan-over-scan-zeek:
  Deprecate misc/scan.zeek
2022-08-23 10:19:47 -07:00
Christian Kreibich
4ed15857b8 Merge remote-tracking branch 'origin/topic/awelzel/vector-value-iteration'
* origin/topic/awelzel/vector-value-iteration:
  stmt: Support iterating over vector values
2022-08-23 09:40:12 -07:00
Arne Welzel
6dc585bd8c Deprecate misc/scan.zeek
Update bare-mode-errors test to ignore that specific message.
2022-08-23 09:10:53 +02:00
Arne Welzel
f334df3b79 stmt: Support iterating over vector values
I ran into wanting to iterate over just the values of a vector and wondering
whether that could just work.

This adds support for the following, where v will be value of vec[i].

    local vec = vector("zero", "one", "two");

    for ( i, v in vec )
        print i, v;
2022-08-22 16:57:37 +02:00
Christian Kreibich
f1a9108199 Remove long-unused BRO_DISABLE_BROXYGEN env var from btests 2022-08-19 15:31:55 -07:00
Christian Kreibich
0f8e675a49 Merge branch 'topic/awelzel/unified2-drop'
* topic/awelzel/unified2-drop:
  Remove unified2 file analyzer
2022-08-19 15:23:34 -07:00
Arne Welzel
8d19fa23ef Remove unified2 file analyzer 2022-08-19 14:05:00 +02:00
Arne Welzel
78beecf37d Remove barnyard2 integration scripts 2022-08-19 14:04:27 +02:00
Tim Wojtulewicz
90f0e7a6ea Support other byte lengths in bytestring_to_count 2022-08-17 15:45:30 -07:00
Tim Wojtulewicz
687dd05ddb GH-163: Use ID name (including module name) to create EventExpr when possible
The change to the capture-loss test is actually a fix for a bug exposed by the
code change. Previously it wasn't firing the scheduled event because of a failed
name lookup. Now that the lookup has been fixed, the event happens twice.
2022-08-17 13:15:01 -07:00
Arne Welzel
c6ca89a590 parse.y/anonymous_function: Allow conditionals between begin_lambda and lambda_body
This is to allow conditional statements following a lambda header
(begin_lambda) just as is done with func_hdr conditional_list func_body.
2022-08-17 13:40:25 +02:00
Robin Sommer
598cef21bd
Merge ssh://github.com/fatemabw/zeek
* ssh://github.com/fatemabw/zeek:
  Update options.zeek
  Create out-27
  Add files via upload
  Update src/packet_analysis/protocol/tcp/TCPSessionAdapter.cc
  Updating the weird names to use all lower case
  Fixing whitespaces..
  Fixing clang pre-commit error
  Add check for option 27
  Add the parsed fields for TCP option 27
  Add TCP options bad length check
2022-08-17 09:09:58 +02:00
Robin Sommer
39148ef706
Merge remote-tracking branch 'origin/topic/awelzel/files-log-unrolling'
I removed `deprecated-txhosts-rxhosts-connuids.zeek` from
`local.zeek`, seems preferable not to have a script-to-go-away in the
standard configuration for new users. Also tweaked `NEWS` just a tiny
bit.

* origin/topic/awelzel/files-log-unrolling:
  files.log: Unroll and introduce uid and id fields
2022-08-17 09:01:43 +02:00
Arne Welzel
d2314d2666 files.log: Unroll and introduce uid and id fields
This is a script-only change that unrolls File::Info records into
multiple files.log entries if the same file was seen over different
connections by single worker. Consequently, the File::Info record
gets the commonly used uid and id fields added. These fields are
optional for File::Info - a file may be analyzed without relation
to a network connection (e.g by using Input::add_analysis()).

The existing tx_hosts, rx_hosts and conn_uids fields of Files::Info
are not meaningful after this change and removed by default. Therefore,
files.log will have them removed, too.

The tx_hosts, rx_hosts and conn_uids fields can be revived by using the
policy script frameworks/files/deprecated-txhosts-rxhosts-connuids.zeek
included in the distribution. However, with v6.1 this script will be
removed.
2022-08-16 17:22:20 +02:00
Tim Wojtulewicz
b5b022449e Update zeek-testing-private commit hash after email regex changes
Related to https://github.com/zeek/zeek/pull/2221
2022-08-12 16:44:25 -07:00
Fatema BW
61244738c8
Update options.zeek 2022-08-11 18:29:12 -07:00
Fatema BW
e92c252b3d
Create out-27 2022-08-11 18:27:28 -07:00
Fatema BW
4609429aa4
Add files via upload 2022-08-11 18:23:02 -07:00
Fatema BW
d9632631ce
Merge branch 'zeek:master' into master 2022-08-11 18:20:31 -07:00
Tim Wojtulewicz
7fe6290974 Merge remote-tracking branch 'micrictor/master'
* micrictor/master:
  Add a field to Modbus/TCP log to indicate the Modbus PDU type
  Add modbus transaction and unit ids to logs
  Enable modbus logging for requests
2022-08-11 11:57:10 -07:00
Tim Wojtulewicz
e618be094a Merge remote-tracking branch 'theavgjojo/master'
* theavgjojo/master:
  UPDATED: improving email address splitting for common comma case
2022-08-11 10:41:18 -07:00
Tim Wojtulewicz
313e303fda Merge remote-tracking branch 'AmazingPP/topic/amazingpp/table-values-and-keys'
* AmazingPP/topic/amazingpp/table-values-and-keys:
  Add more test cases
  Add table_keys function
  Add table_values function
2022-08-11 08:49:34 -07:00
Robin Sommer
6d27df548e
Merge remote-tracking branch 'origin/topic/awelzel/1678-disabling-analyzer-hook'
* origin/topic/awelzel/1678-disabling-analyzer-hook:
  Add NEWS entry and zeekygen-smithing for disabling_analyzer()
  Introduce global disabling_analyzer() hook to veto disable_analyzer()
  ssl: Only delete c$ssl$analyzer_id when disabling the analyzer was successful
2022-08-11 16:02:37 +02:00
Robin Sommer
ee7288b1e6
Merge remote-tracking branch 'origin/topic/awelzel/record-field-redef'
* origin/topic/awelzel/record-field-redef:
  Support redef'ing the &log attribute of record fields
2022-08-11 11:59:26 +02:00
Robin Sommer
9de2eceb2a
Merge remote-tracking branch 'origin/topic/awelzel/2262-telemetry-ditch-singleton-metrics'
* origin/topic/awelzel/2262-telemetry-ditch-singleton-metrics:
  telemetry: Remove singleton BIFs and the C++ pieces
2022-08-11 11:54:52 +02:00
Arne Welzel
abb7f9a509 Introduce global disabling_analyzer() hook to veto disable_analyzer()
This hook can be used to coordinate disabling an analyzer for a given
connection. The contract is simple: Any script can veto a disable_analyzer()
call by breaking from this hook. The decision is local to the script taking
into account any state attached to the connection object or script specific
state stored elsewhere.
A script breaking from the hook takes over the responsibility to call
disable_analyzer() at a later point when it finds the condition due to which
it vetoed fulfilled (which may be never).

Signature:

    disabling_analyzer: hook(c: connection, atype: AllAnalyzers::Tag, aid: count);

Example use-cases are keeping the SSL analyzer enabled for finger-printing
until a certain amount of bytes or packets have been transferred or
similarly the connection duration exceed a certain threshold.

Other example use-cases might be keeping analyzers for SSH, RDP or SSL
enabled for connections from specific subnets.

It's a bit quirky as it makes disable_analyzer() a maybe operation. While log
policy hooks and/or the notice hook have similar semantics, they are not as
stateful. It still seems like a quite powerful primitive.

The disable_analyzer() call in dpd/main.zeek may motivate the addition of a
force flag as a follow-up for situations where the caller "knows better" or
absolutely wants to override.

Closes #1678 #1593.
2022-08-11 09:40:36 +02:00
AmazingPP
206e6d28a4 Add more test cases 2022-08-11 13:35:27 +08:00
Christian Kreibich
3a0a702723 Merge branch 'topic/christian/management-bugfixes'
* topic/christian/management-bugfixes:
  Management framework: bump cluster testsuite
  Management framework: log node set in dispatch requests cleanly
  Management framework: log additional node events
  Management framework: upon deployment, make agent log multiple node results
  Management framework: fix early return condition for get-id-value
2022-08-10 14:18:38 -07:00
Christian Kreibich
20e08f0d38 Management framework: bump cluster testsuite 2022-08-10 11:24:10 -07:00
Arne Welzel
985bbe4e57 Support redef'ing the &log attribute of record fields
Add new syntax for adding and removing attributes from record fields:

    redef RecordType$field_name += { &log };
    redef RecordType$field_name -= { &log };

For now this only allowed for the &log attribute as the semantics are clear.
For &default and &optional the semantics aren't obvious and no use-cases have
been identified where those would make sense to change.

This enables a mechanism to add potentially interesting fields to the typical
Info records in base scripts, but letting users opt-into actually including
them into their log. At the same time, users that find specific fields in a
standard log uninteresting can opt-out without using `Log::Filter$exclude`
which can be difficult to use correctly. Patching or forking external packages
to remove columns from a log can also be avoided with this mechanism.

Closes #2000.
2022-08-10 17:27:05 +02:00
Tim Wojtulewicz
514df9f179 Fix module-scoped type definitions that conflict with existing global ones 2022-08-08 15:30:34 -07:00
Tim Wojtulewicz
cb3abccfb1 Merge remote-tracking branch 'ynadji/topic/yacin/2319-add-change-handler-to-site'
* ynadji/topic/yacin/2319-add-change-handler-to-site:
  update plugins.hooks baseline
  lower priority for change handlers
  split update_zones_regex into two functions
  GH-2319: Add change handlers to Site
2022-08-08 11:09:16 -07:00
Tim Wojtulewicz
3da1848279 Merge remote-tracking branch 'AmazingPP/topic/amazingpp/command-line-script-warning'
* AmazingPP/topic/amazingpp/command-line-script-warning:
  Squelch the zeekygen warnings for command line
2022-08-08 11:03:32 -07:00
Yacin Nadji
84610ed832 update plugins.hooks baseline 2022-08-08 11:52:06 +02:00
AmazingPP
9869226e1e Squelch the zeekygen warnings for command line 2022-08-06 11:35:05 +08:00
TheAvgJojo
1e37e91eda UPDATED: improving email address splitting for common comma case 2022-08-05 17:38:54 -04:00
Arne Welzel
acacc6b6c2 telemetry: Remove singleton BIFs and the C++ pieces
The low-level singleton Telemetry BIFs have been removed with the that there
haven't been any users. Singleton metrics can be instantiated by providing
an empty label vector instead and aren't in any way a special concept.

Closes #2262.
2022-08-05 19:33:49 +02:00
Tim Wojtulewicz
9524963da6 Merge remote-tracking branch 'anniebryan/notice-suppression-bug-fix'
* anniebryan/notice-suppression-bug-fix:
  Added test case with back-to-back notices
  Fix notice suppression atomicity bug
2022-08-05 16:21:29 +00:00
Tim Wojtulewicz
1870d26684 GH-2034: Store module names and use them in lookups for ifdef 2022-08-05 15:36:21 +00:00
Robin Sommer
686e740bbe
Merge remote-tracking branch 'origin/topic/awelzel/telemetry-script-land-v0'
* origin/topic/awelzel/telemetry-script-land-v0:
  Introduce telemetry framework
  Bump broker submodule to master.
2022-08-05 17:14:23 +02:00
Yacin Nadji
4a04b563f5 GH-2319: Add change handlers to Site 2022-08-05 16:17:50 +02:00
Arne Welzel
3fe930dbf2 Introduce telemetry framework
Adds base/frameworks/telemetry with wrappers around telemetry.bif
and updates telemetry/Manager to support collecting metrics from
script land.

Add policy/frameworks/telemetry/log for logging of metrics data
into a new telemetry.log and telemetry_histogram.log and add into
local.zeek by default.
2022-08-05 11:43:44 +02:00
Annie Bryan
d916469510 Added test case with back-to-back notices 2022-08-04 11:25:20 +02:00
Tim Wojtulewicz
930fd75ed8 Merge remote-tracking branch 'origin/topic/awelzel/2010-zeek-done-change-handlers'
* origin/topic/awelzel/2010-zeek-done-change-handlers:
  testing/missing-enum-value: redef exit_only_after_terminate=T
  option.bif: Short-circuit option changes when terminating
2022-08-03 10:27:06 -07:00
Tim Wojtulewicz
3ffffe33bc Merge remote-tracking branch 'origin/topic/vern/bit-shift-fixes'
* origin/topic/vern/bit-shift-fixes:
  btest portability fix address review comment about shifting corner-case
  canonicalize filenames for new vector deprecation btest
  updates for gen-C++ maintenance, including skipping some inappropriate tests
  fix for profiling "when" statements
  gen-C++ support for vector bit-shift operations
  corrected wording in some btest comments
  make gen-C++ maintenance scripts directly executable
  ZAM support for bit-shifting
  don't allow deprecated-style mixing of vectors and scaling for shifting leverage restrictions placed on shifting (RHS is always unsigned) split deprecated vector operations into separate test, with separate ZAM baseline
  ZAM fix for vector "in" operator
  ensure that language tests pay attention to .stderr
  fix vector tests, including checking for errors
2022-08-03 09:56:55 -07:00
Vern Paxson
c10885e5f9 btest portability fix
address review comment about shifting corner-case
2022-08-02 19:36:25 -07:00
Tim Wojtulewicz
c82eb13cc8 Merge remote-tracking branch 'pbcullen/topic/pbcullen/prevent-large-dhcp-logs'
* pbcullen/topic/pbcullen/prevent-large-dhcp-logs:
  Update plugins.hooks baseline with new DHCP options
  Prevent large dhcp log entries
2022-08-02 12:38:03 -07:00
Vern Paxson
ea3d25b380 canonicalize filenames for new vector deprecation btest 2022-08-02 12:37:33 -07:00
Tim Wojtulewicz
18126c2d50 Add support for /s modifier to RE matcher and parser 2022-08-02 11:31:57 -07:00