Commit graph

3850 commits

Author SHA1 Message Date
Tim Wojtulewicz
73273fc87b Merge remote-tracking branch 'origin/topic/vern/ZAM-inlining'
* origin/topic/vern/ZAM-inlining:
  speed up ZAM compilation by capping function size when inlining
2022-05-19 11:45:50 -07:00
Christian Kreibich
0641d032c3 Zeekify the scripts.base.utils.paths test 2022-05-19 09:51:36 -07:00
Christian Kreibich
9d59a48ae2 Expand build_path() function to handle empty dir arguments gracefully
When passing an empty string as a directory, the function would produce
filenames starting with a slash even when the given file_name is not an absolute
path. Defaulting to the root directory is likely never intended and might
conveivably be dangerous. The middle "/" is now skipped also if dir is an empty
string.
2022-05-19 09:45:52 -07:00
Tim Wojtulewicz
aaaca5d5a6 Merge remote-tracking branch 'origin/topic/bbannier/spicy'
* origin/topic/bbannier/spicy:
  Add NEWS item for new requirements due to default-enabled Spicy support.
  Include headers in binary dir before other headers.
  Increase memory in CI.
  Add tests for bundled Spicy infrastructure.
  Include spicy in build.
  Add `bare_mode` bif.
  Change test so included plugins can load notice framework.
  Do not log function arguments in test.
2022-05-17 08:50:33 -07:00
Tim Wojtulewicz
3233400930 Merge remote-tracking branch 'origin/topic/timw/1857-input-log-messages'
* origin/topic/timw/1857-input-log-messages:
  Move some repetitive code into a separate method
  Store location during Ascii input parsing for use in error messages
  Include file information in input reader error messages
2022-05-16 17:00:34 -07:00
Tim Wojtulewicz
3d133ccb0a Merge remote-tracking branch 'origin/topic/vern/CPP-feature-complete'
* origin/topic/vern/CPP-feature-complete:
  NEWS items for script profiling, "footprint" BiFs, and Trigger method deprecations
  tweaks per reviewing feedback
  make deprecation advice a bit more helpful
  format shell scripts per style guide
  add deprecation for Trigger Cache/Lookup interfaces
  helper scripts for -O C++ maintenance
  "-O use-C++" now fails if no C++ bodies wound up being applicable
  update environemnt variable names to match command-line arguments
  final changes for -O C++ feature completeness, mainly "when" statement support
  Change "when" statements that don't require closures to use new implementation. Provide hooks for script optimization access to "when" statements. Regularize treatment of naming and timeouts for Triggers.
  switch cached Trigger values to be opaque-and-generic, rather than assuming CallExpr's
  tracking of when statements/expressions occur in a "when" context
  fix bug that sometimes associated the wrong location with global statements
  fix incomplete AST traversal of "when" statements
  add some missing "when" closures
  regularize test suite names to avoid custom/outdated suffices
2022-05-16 16:59:59 -07:00
Tim Wojtulewicz
ec50b66ff3 Store location during Ascii input parsing for use in error messages 2022-05-16 13:15:06 -07:00
Tim Wojtulewicz
098a5d3348 Include file information in input reader error messages 2022-05-16 13:15:06 -07:00
Tim Wojtulewicz
1d20e28e5c Merge remote-tracking branch 'origin/topic/timw/retry-update-traces'
* origin/topic/timw/retry-update-traces:
  Add --retry to curl invocations in update-traces
2022-05-16 09:09:47 -07:00
Tim Wojtulewicz
55297a2e73 Merge remote-tracking branch 'origin/topic/timw/2103-bifs-footprint-leak'
* origin/topic/timw/2103-bifs-footprint-leak:
  GH-2103: Disable leak detection for bifs.footprint in another way
2022-05-16 09:09:17 -07:00
Benjamin Bannier
e40aa0f6d2 Add tests for bundled Spicy infrastructure. 2022-05-16 09:07:11 +02:00
Benjamin Bannier
95aff9a1e3 Include spicy in build. 2022-05-16 09:07:11 +02:00
Benjamin Bannier
e83c4c834d Add bare_mode bif. 2022-05-16 09:07:11 +02:00
Benjamin Bannier
0f1dd8ef1b Change test so included plugins can load notice framework. 2022-05-16 09:07:11 +02:00
Benjamin Bannier
1e4f368c68 Do not log function arguments in test.
This was excercising hooks to validate that they were called. For that
it uses pretty verbose logging. Since version numbers are not stable and
hard to canonify this script already unloads the version module (there
is e.g., code in `base/misc/version` which splits the version strings
and works on single components which might appear like pretty ordinary
floating point numbers in the log).

This test however worked under the assumption that nothing in bare mode
loads `base/misc/version` which is hard to guarantee, especially
considering that plugins can be embedded and might load that module
themself.

With this patch we now make the logging slightly less verbose so that
function call arguments are not logged anymore.
2022-05-16 09:07:11 +02:00
Vern Paxson
7d00ce0082 speed up ZAM compilation by capping function size when inlining 2022-05-14 14:52:31 -07:00
Tim Wojtulewicz
bd1ea5edda Merge remote-tracking branch 'origin/topic/johanna/bloomfilter'
* origin/topic/johanna/bloomfilter:
  Bloom filter changes: address review feedback
  Bloom-filter: fix documentation syntax problem
  Bloomfilter related clang-format fixes
  Correcly typify bloom filters in all cases during merge
  Add intersect operation for bloom filters
  Add bloomfilter_decrement bif
2022-05-13 16:49:13 -07:00
Tim Wojtulewicz
56fe393691 GH-2103: Disable leak detection for bifs.footprint in another way 2022-05-13 16:26:41 -07:00
Tim Wojtulewicz
107f4fdc99 Add --retry to curl invocations in update-traces 2022-05-13 11:54:27 -07:00
Vern Paxson
8928aa57e7 add some missing "when" closures 2022-05-12 13:39:26 -07:00
Vern Paxson
025b9b1534 regularize test suite names to avoid custom/outdated suffices 2022-05-12 13:32:49 -07:00
Johanna Amann
5118e7f86b Merge remote-tracking branch 'origin/topic/johanna/cert-weak-key'
* origin/topic/johanna/cert-weak-key:
  Include certificate information in SSL::Weak_Key notice
2022-05-12 11:04:34 +01:00
Tim Wojtulewicz
8b0263cb39 Merge remote-tracking branch 'origin/topic/vern/script-profiling'
* origin/topic/vern/script-profiling:
  tidy up after generating profile
  test suite updates for refined script coverage, use of new BiF to speed startup
  fix for coverage reporting for functions that use "when" statements
  new global_options() BiF to speed up startup, plus a micro-preen
  hooks for new --profile-scripts option
  classes for managing script profiles
  address some holes in script coverage
  fix for script coverage missing on-exit activity
  memory management fixes for loggers
  make curr_CPU_time() broadly available rather than just isolated to ZAM
2022-05-11 12:56:41 -07:00
Johanna Amann
5f04f216bc Include certificate information in SSL::Weak_Key notice 2022-05-11 18:56:04 +01:00
Johanna Amann
0aafc8ae6c Merge remote-tracking branch 'origin/topic/vern/type-names'
* origin/topic/vern/type-names:
  regularize deprecation warning
  make DoDescribe protected for a tidier interface mark use of DESC_PORTABLE as deprecated
  when printing a description of a type, use its name if available
2022-05-09 11:15:36 +01:00
Christian Kreibich
8d101f21be Merge branch 'topic/christian/management-consistency-tweaks'
* topic/christian/management-consistency-tweaks:
  Management framework: consistency tweaks in zeek-client, plus test updates
2022-05-06 15:28:48 -07:00
Tim Wojtulewicz
cf51931615 Merge remote-tracking branch 'origin/topic/vern/footprint'
* origin/topic/vern/footprint:
  new environment variable to enable BTests to skip ASAN checks
  skip new BiF test for ASAN CI runs
  btest update to include recursive value that doesn't require a record
  to avoid recursion, track all aggregates, not just records isolate the internal methods
  simpler public calling interface for computing footprint
  use stack-based set to prevent infinite recursion rather than a static one
  change value_footprint() to val_footprint() to be more similar to val_size()
  make including count of container elements non-optional
  btest for mutually-recursive case
  fix for tracking footprints of mutually-recursive records
  added value_footprint() and global_container_footprints() BiFs
2022-05-06 11:32:49 -07:00
Christian Kreibich
8c76af6ab9 Management framework: consistency tweaks in zeek-client, plus test updates
A minor changeset that reflects zeek-client updates, with some effects on
its output and hence test cases.
2022-05-06 11:22:48 -07:00
Christian Kreibich
7198c847e8 Merge branch 'topic/christian/management-get-config'
* topic/christian/management-get-config:
  Management framework: add get_configuration_request/response transaction
2022-05-05 18:10:46 -07:00
Vern Paxson
a387157ead skip new BiF test for ASAN CI runs 2022-05-05 17:07:46 -07:00
Vern Paxson
d5f60cfaba btest update to include recursive value that doesn't require a record 2022-05-05 16:55:03 -07:00
Christian Kreibich
001de561fc Management framework: add get_configuration_request/response transaction
Includes submodule bumps for Broker (to pull in better handling of data
structures that are difficult to unserialize in Python), zeek-client (for the
get-config command), and a commit hash update for the external testsuite.
2022-05-05 16:09:21 -07:00
Tim Wojtulewicz
1d1cad95c9 Merge remote-tracking branch 'origin/topic/vern/zam-maint-01May22'
* origin/topic/vern/zam-maint-01May22:
  fix for coverage reporting for functions that use "when" statements
  ZAM maintenance for recent changes and some newly exercised corner cases
2022-05-04 11:58:02 -07:00
Vern Paxson
690a4590b7 test suite updates for refined script coverage, use of new BiF to speed startup 2022-05-03 12:36:50 -07:00
Johanna Amann
42bc6db359 Merge remote-tracking branch 'origin/master' into topic/johanna/bloomfilter 2022-05-03 16:43:36 +01:00
Vern Paxson
f7581437d9 fix for sets containing recursive records 2022-05-02 13:19:58 -07:00
Vern Paxson
7a9c6c476e ZAM maintenance for recent changes and some newly exercised corner cases 2022-05-01 14:33:58 -07:00
Vern Paxson
edf276520a make including count of container elements non-optional 2022-04-29 09:03:35 -07:00
Vern Paxson
b670046a69 btest for mutually-recursive case 2022-04-29 08:44:58 -07:00
Vern Paxson
cbb18207dc when printing a description of a type, use its name if available 2022-04-28 18:23:02 -07:00
Vern Paxson
16c37034de added value_footprint() and global_container_footprints() BiFs 2022-04-28 16:42:14 -07:00
Dominik Charousset
a8196cf4fb Fix formatting 2022-04-27 23:02:27 +02:00
Dominik Charousset
02608df99c Canonify intel.log in read-file-dist-cluster test 2022-04-27 23:02:27 +02:00
Dominik Charousset
4ea1a593a9 Port Zeek to latest Broker API 2022-04-27 23:02:27 +02:00
Tim Wojtulewicz
e8f833b8a6 Store all mappings in a single map instead of split by type
This opens up the possibility of storing other request types outside
of T_A, T_PTR and T_TXT without requiring redoing the caching. It
also fixes the caching code in DNS_Mapping, adding a version number
to the start of the cache file so the cache structure can be modified
and old caches invalidated more easily.
2022-04-22 13:10:37 -07:00
Christian Kreibich
2d73edb04c Merge branch 'topic/christian/management-print-cmd'
* topic/christian/management-print-cmd:
  Management framework: bump external testsuite
  Management framework: allow selecting cluster nodes in get_id_value
  Management framework: minor tweaks to logging component
  Management framework: bump zeek-client to pull in get-id-value command
  Avoid whitespace around function type strings in JSON rendering
  Management framework: improve handling of node run states
  Management framework: add get_id_value dispatch
  Management framework: allow dispatching "actions" on cluster nodes.
  Management framework: some renaming to avoid the term "data cluster"
  Management framework: allow agents to communicate with cluster nodes
2022-04-18 16:40:49 -07:00
Christian Kreibich
e2d0db73a5 Management framework: bump external testsuite 2022-04-18 13:11:03 -07:00
Christian Kreibich
337c7267e0 Management framework: allow agents to communicate with cluster nodes
This provides Broker-level plumbing that allows agents to reach out to their
managed Zeek nodes and collect responses.

As a first event, it establishes Management::Node::API::notify_agent_hello,
to notify the agent when the cluster node is ready to communicate.

Also a bit of comment rewording to replace use of "data cluster" with simply
"cluster", to avoid ambiguity with data nodes in SumStats, and expansion of
test-all-policy.zeek and related/dependent tests, since we're introducing new
scripts.
2022-04-15 18:51:54 -07:00
Christian Kreibich
2d93a19f45 Expand plugin.reporter-hook testcase to verify -NN output 2022-04-14 15:20:08 -07:00
Tim Wojtulewicz
a3b022ed98 Merge remote-tracking branch 'origin/topic/robin/loads-sigs-path'
* origin/topic/robin/loads-sigs-path:
  Re-instantiate providing location information to `LoadFile` hooks.
2022-04-14 10:13:35 -07:00