Commit graph

2372 commits

Author SHA1 Message Date
Tim Wojtulewicz
38702053bd Merge remote-tracking branch 'origin/topic/timw/c-ares-include-path'
* origin/topic/timw/c-ares-include-path:
  GH-2101: Update cmake submodule to fix c-ares include path problem
2022-06-01 09:20:33 -07:00
Christian Kreibich
65589c9bcb Merge branch 'topic/christian/management-verify-nodestarts'
* topic/christian/management-verify-nodestarts:
  Management framework: bump external cluster testsuite
  Management framework: bump zeek-client to pull in set-config rendering
  Management framework: enable stdout/stderr reporting
  Management framework: Supervisor extensions for stdout/stderr handling
  Management framework: disambiguate redef field names in agent and controller
  Management framework: move to ResultVec in agent's set_configuration response
  Management framework: tune request timeout granularity and interval
  Management framework: verify node starts when deploying a configuration
  Management framework: a bit of debug-level logging for troubleshooting
2022-05-31 12:59:18 -07:00
Christian Kreibich
93bed5a261 Merge branch 'topic/christian/node-status-notification'
* topic/christian/node-status-notification:
  Add Supervisor::node_status notification event
2022-05-31 12:53:18 -07:00
Tim Wojtulewicz
5a07aa9a0b Merge remote-tracking branch 'origin/topic/vern/find-unused'
* origin/topic/vern/find-unused:
  Update spicy-plugin with change that checks for zeek version
  deprecation messages for unused base script functions
  clearer messages for warning about unused functions
  Fixes from review, post-rebase
  code formatting and more btest updates
  baseline & btest updates
  annotate orphan base script components with &deprecated
  annotate base scripts with &is_used as needed
  --no-usage-warnings flag to suppress analysis
  support for associating &is_used attributes with functions
  classes for evaluating function/hook/event usage
  broader support for AST traversal, including Attr and Attributes objects
  include attributes in descriptions of sets and tables
  low-level tidying
2022-05-31 11:14:22 -07:00
Tim Wojtulewicz
bdfa7e70f5 Merge remote-tracking branch 'origin/topic/timw/fix-rst-warning'
* origin/topic/timw/fix-rst-warning:
  Update zeek-aux submodule to fix a compiler warning
2022-05-27 14:34:01 -07:00
Christian Kreibich
415bbe17d6 Merge branch 'topic/christian/management-cluster-dirs'
* topic/christian/management-cluster-dirs:
  Management framework: bump zeek-client to pull in instance serialization fixes
  Management framework: bump external cluster testsuite
  Management framework: update agent-checkin test to reflect recent changes
  Management framework: place each Zeek process in its own working dir
  Management framework: set defaults for log rotation and persistent state
  Management framework: add spool and state directory config settings
  Management framework: establish stdout/stderr files also for cluster nodes
  Management framework: default to having agents check in with the (local) controller
  Management framework: move role variable from logging into framework-wide config
  Management framework: distinguish supervisor/supervisee when loading agent/controller
  Management framework: simplify agent and controller stdout/stderr files
  Management framework: prefix the management logs with "management-"
  Management framework: comment and layouting tweaks, no functional change
  Management framework: rename env var that labels agents/controllers
  Management framework: increase robustness of agent/controller naming
2022-05-26 16:10:14 -07:00
Tim Wojtulewicz
fb01c8967a Merge remote-tracking branch 'origin/topic/timw/2054-null-string-join'
* origin/topic/timw/2054-null-string-join:
  GH-2054: Allow nulls as separators for join_string_vec
2022-05-26 08:23:53 -07:00
Tim Wojtulewicz
2f453cd0cd Merge remote-tracking branch 'origin/topic/timw/1995-pcap-filter-error-messages'
* origin/topic/timw/1995-pcap-filter-error-messages:
  Propagate BPF_Program error message to script land
  Allow pcap pktsrc to use other BPF_Program::Compile method
2022-05-25 12:09:52 -07:00
Tim Wojtulewicz
515e5bf8a3 Merge remote-tracking branch 'origin/topic/bbannier/spicy-bump'
* origin/topic/bbannier/spicy-bump:
  Bump Spicy and spicy-plugin.
  Clean up test `spicy.spicyz-jit`.
2022-05-25 09:40:26 -07:00
Tim Wojtulewicz
7c56b6923b Merge remote-tracking branch 'origin/topic/timw/2125-scriptprofile-warnings'
* origin/topic/timw/2125-scriptprofile-warnings:
  GH-2125: Fix compiler warnings in ScriptProfile
2022-05-25 09:38:30 -07:00
Tim Wojtulewicz
97e8f9d268 Merge remote-tracking branch 'origin/topic/timw/dict-debug-fixes'
* origin/topic/timw/dict-debug-fixes:
  Use ZEEK_DICT_DEBUG more in Dict.cc instead of DEBUG
2022-05-25 09:38:06 -07:00
Tim Wojtulewicz
0b1c8575cc Merge remote-tracking branch 'origin/topic/timw/avoid-needing-flex-arguments'
* origin/topic/timw/avoid-needing-flex-arguments:
  Move fuzzer setup in CMake below builtin plugin setup
  Remove flex/bison flags from CI build
  Update cmake submodule [nomail]
2022-05-25 09:37:31 -07:00
Christian Kreibich
d4ecfa0a67 Merge branch 'topic/christian/installation-dirs-in-scriptland'
* topic/christian/installation-dirs-in-scriptland:
  Add scripts.base.misc.installation btest
  Add base/misc/installation.zeek, with Zeek installation directories
  Ensure presence of Zeek-related directories in toplevel CMakeLists.txt
2022-05-24 12:12:05 -07:00
Tim Wojtulewicz
a61bc16b10 Merge remote-tracking branch 'origin/topic/timw/dockerfile-versions'
* origin/topic/timw/dockerfile-versions:
  Add DOCKERFILE_VERSION variable to all CI dockerfiles
2022-05-23 17:47:20 -07:00
Tim Wojtulewicz
73273fc87b Merge remote-tracking branch 'origin/topic/vern/ZAM-inlining'
* origin/topic/vern/ZAM-inlining:
  speed up ZAM compilation by capping function size when inlining
2022-05-19 11:45:50 -07:00
Christian Kreibich
7b43153785 Merge branch 'topic/christian/build-path-tweak'
* topic/christian/build-path-tweak:
  Zeekify the scripts.base.utils.paths test
  Expand build_path() function to handle empty dir arguments gracefully
2022-05-19 11:25:08 -07:00
Tim Wojtulewicz
48c978ae17 Merge remote-tracking branch 'origin/topic/timw/cirrus-ccache'
* origin/topic/timw/cirrus-ccache:
  Rework .cirrus.yml a bit to commonize some parts
  Enable 'greedy' mode for other CI builds
  Enable ccache for Cirrus builds
  Fix SSL tests on Centos Stream 9
2022-05-19 10:52:11 -07:00
Tim Wojtulewicz
ee85595e48 Merge remote-tracking branch 'origin/topic/timw/disable-spicy-for-coverity-builds'
* origin/topic/timw/disable-spicy-for-coverity-builds:
  Disable Spicy in Zeek Coverity builds
2022-05-19 09:13:25 -07:00
Tim Wojtulewicz
1223386fd3 Merge remote-tracking branch 'origin/topic/bbannier/spicy-helper-scripts'
* origin/topic/bbannier/spicy-helper-scripts:
  Always generate helper scripts for Spicy.
2022-05-18 11:12:08 -07:00
Tim Wojtulewicz
840a57b458 Merge remote-tracking branch 'origin/topic/johanna/openssl-3-fips'
* origin/topic/johanna/openssl-3-fips:
  Partially revert OpenSSL hashing changes
2022-05-18 11:07:40 -07:00
Tim Wojtulewicz
327eca32fd Merge remote-tracking branch 'origin/topic/timw/1672-table-expiration-deletion'
* origin/topic/timw/1672-table-expiration-deletion:
  GH-1672: Keep a reference to a table being expired, just in case
2022-05-18 11:07:13 -07:00
Tim Wojtulewicz
92a7773dd5 Merge remote-tracking branch 'origin/topic/timw/unneeded-fmt'
* origin/topic/timw/unneeded-fmt:
  Remove unnecessary uses of util::fmt()
2022-05-17 09:15:37 -07:00
Tim Wojtulewicz
aaaca5d5a6 Merge remote-tracking branch 'origin/topic/bbannier/spicy'
* origin/topic/bbannier/spicy:
  Add NEWS item for new requirements due to default-enabled Spicy support.
  Include headers in binary dir before other headers.
  Increase memory in CI.
  Add tests for bundled Spicy infrastructure.
  Include spicy in build.
  Add `bare_mode` bif.
  Change test so included plugins can load notice framework.
  Do not log function arguments in test.
2022-05-17 08:50:33 -07:00
Tim Wojtulewicz
3233400930 Merge remote-tracking branch 'origin/topic/timw/1857-input-log-messages'
* origin/topic/timw/1857-input-log-messages:
  Move some repetitive code into a separate method
  Store location during Ascii input parsing for use in error messages
  Include file information in input reader error messages
2022-05-16 17:00:34 -07:00
Tim Wojtulewicz
3d133ccb0a Merge remote-tracking branch 'origin/topic/vern/CPP-feature-complete'
* origin/topic/vern/CPP-feature-complete:
  NEWS items for script profiling, "footprint" BiFs, and Trigger method deprecations
  tweaks per reviewing feedback
  make deprecation advice a bit more helpful
  format shell scripts per style guide
  add deprecation for Trigger Cache/Lookup interfaces
  helper scripts for -O C++ maintenance
  "-O use-C++" now fails if no C++ bodies wound up being applicable
  update environemnt variable names to match command-line arguments
  final changes for -O C++ feature completeness, mainly "when" statement support
  Change "when" statements that don't require closures to use new implementation. Provide hooks for script optimization access to "when" statements. Regularize treatment of naming and timeouts for Triggers.
  switch cached Trigger values to be opaque-and-generic, rather than assuming CallExpr's
  tracking of when statements/expressions occur in a "when" context
  fix bug that sometimes associated the wrong location with global statements
  fix incomplete AST traversal of "when" statements
  add some missing "when" closures
  regularize test suite names to avoid custom/outdated suffices
2022-05-16 16:59:59 -07:00
Tim Wojtulewicz
8184073ef8 Merge remote-tracking branch 'origin/topic/timw/fix-freebsd-14-pip'
* origin/topic/timw/fix-freebsd-14-pip:
  Call pip using python -m flag during FreeBSD CI prep
2022-05-16 13:14:43 -07:00
Tim Wojtulewicz
5c34c777bd Merge remote-tracking branch 'origin/topic/timw/1797-string-allocations'
* origin/topic/timw/1797-string-allocations:
  Store hash analyzer kind StringVals for later reuse
  Add Component::GetCanonicalNameVal to avoid extra StringVal allocations
  Avoid type unnecessary type conversion in ComponentManager::GetComponentName
  Only set connection history if strings differ
2022-05-16 09:11:06 -07:00
Tim Wojtulewicz
6268baed35 Merge remote-tracking branch 'origin/topic/timw/script-profile-move-operator'
* origin/topic/timw/script-profile-move-operator:
  Explicitly define copy and move operators/constructors for ScriptProfileStats
2022-05-16 09:10:30 -07:00
Tim Wojtulewicz
1d20e28e5c Merge remote-tracking branch 'origin/topic/timw/retry-update-traces'
* origin/topic/timw/retry-update-traces:
  Add --retry to curl invocations in update-traces
2022-05-16 09:09:47 -07:00
Tim Wojtulewicz
55297a2e73 Merge remote-tracking branch 'origin/topic/timw/2103-bifs-footprint-leak'
* origin/topic/timw/2103-bifs-footprint-leak:
  GH-2103: Disable leak detection for bifs.footprint in another way
2022-05-16 09:09:17 -07:00
Tim Wojtulewicz
6cd3724747 Merge remote-tracking branch 'origin/topic/neverlord/gh-2084'
* origin/topic/neverlord/gh-2084:
  Fix formatting
  Fix UB during early shutdown on OpenSSL state
2022-05-13 16:51:05 -07:00
Tim Wojtulewicz
bd1ea5edda Merge remote-tracking branch 'origin/topic/johanna/bloomfilter'
* origin/topic/johanna/bloomfilter:
  Bloom filter changes: address review feedback
  Bloom-filter: fix documentation syntax problem
  Bloomfilter related clang-format fixes
  Correcly typify bloom filters in all cases during merge
  Add intersect operation for bloom filters
  Add bloomfilter_decrement bif
2022-05-13 16:49:13 -07:00
Johanna Amann
5118e7f86b Merge remote-tracking branch 'origin/topic/johanna/cert-weak-key'
* origin/topic/johanna/cert-weak-key:
  Include certificate information in SSL::Weak_Key notice
2022-05-12 11:04:34 +01:00
Tim Wojtulewicz
8b0263cb39 Merge remote-tracking branch 'origin/topic/vern/script-profiling'
* origin/topic/vern/script-profiling:
  tidy up after generating profile
  test suite updates for refined script coverage, use of new BiF to speed startup
  fix for coverage reporting for functions that use "when" statements
  new global_options() BiF to speed up startup, plus a micro-preen
  hooks for new --profile-scripts option
  classes for managing script profiles
  address some holes in script coverage
  fix for script coverage missing on-exit activity
  memory management fixes for loggers
  make curr_CPU_time() broadly available rather than just isolated to ZAM
2022-05-11 12:56:41 -07:00
Johanna Amann
0aafc8ae6c Merge remote-tracking branch 'origin/topic/vern/type-names'
* origin/topic/vern/type-names:
  regularize deprecation warning
  make DoDescribe protected for a tidier interface mark use of DESC_PORTABLE as deprecated
  when printing a description of a type, use its name if available
2022-05-09 11:15:36 +01:00
Christian Kreibich
8d101f21be Merge branch 'topic/christian/management-consistency-tweaks'
* topic/christian/management-consistency-tweaks:
  Management framework: consistency tweaks in zeek-client, plus test updates
2022-05-06 15:28:48 -07:00
Tim Wojtulewicz
c55c06a325 Merge remote-tracking branch 'origin/topic/timw/doctest-update'
* origin/topic/timw/doctest-update:
  Update doctest to v2.4.8
2022-05-06 12:47:58 -07:00
Tim Wojtulewicz
cf51931615 Merge remote-tracking branch 'origin/topic/vern/footprint'
* origin/topic/vern/footprint:
  new environment variable to enable BTests to skip ASAN checks
  skip new BiF test for ASAN CI runs
  btest update to include recursive value that doesn't require a record
  to avoid recursion, track all aggregates, not just records isolate the internal methods
  simpler public calling interface for computing footprint
  use stack-based set to prevent infinite recursion rather than a static one
  change value_footprint() to val_footprint() to be more similar to val_size()
  make including count of container elements non-optional
  btest for mutually-recursive case
  fix for tracking footprints of mutually-recursive records
  added value_footprint() and global_container_footprints() BiFs
2022-05-06 11:32:49 -07:00
Christian Kreibich
7198c847e8 Merge branch 'topic/christian/management-get-config'
* topic/christian/management-get-config:
  Management framework: add get_configuration_request/response transaction
2022-05-05 18:10:46 -07:00
Tim Wojtulewicz
1d1cad95c9 Merge remote-tracking branch 'origin/topic/vern/zam-maint-01May22'
* origin/topic/vern/zam-maint-01May22:
  fix for coverage reporting for functions that use "when" statements
  ZAM maintenance for recent changes and some newly exercised corner cases
2022-05-04 11:58:02 -07:00
Tim Wojtulewicz
00c85425ae Merge remote-tracking branch 'origin/topic/timw/coverity-fixes'
* origin/topic/timw/coverity-fixes:
  Fixes Coverity issues reported from DNS_Mgr merge
2022-05-04 09:24:07 -07:00
Tim Wojtulewicz
edd990359f Merge remote-tracking branch 'origin/topic/vern/set-of-rec-record'
* origin/topic/vern/set-of-rec-record:
  fix for sets containing recursive records
2022-05-03 14:41:05 -07:00
Johanna Amann
ded4c521c6 Merge remote-tracking branch 'origin/topic/johanna/update-ssl-consts-yet-again'
* origin/topic/johanna/update-ssl-consts-yet-again:
  Update X.509 and CT trust anchors
2022-05-03 17:10:12 +01:00
Tim Wojtulewicz
03bcc3523b Merge remote-tracking branch 'origin/topic/neverlord/alm'
* origin/topic/neverlord/alm:
  Fix build on CentOS 7
  Initialize OpenSSL on startup
  Increase memory for CI jobs
  Fix formatting
  Avoid double-initialization of OpenSSL
  Canonify intel.log in read-file-dist-cluster test
  Fix formatting
  Port Zeek to latest Broker API
2022-04-28 19:52:41 +00:00
Tim Wojtulewicz
013070d1a9 Merge remote-tracking branch 'origin/topic/johanna/openssl-initialisation'
* origin/topic/johanna/openssl-initialisation:
  Disable OpenSSL initialization starting with 1.1.0
2022-04-27 21:00:45 +00:00
Tim Wojtulewicz
5a3dee2562 Merge remote-tracking branch 'origin/topic/timw/async-dns-crash'
* origin/topic/timw/async-dns-crash:
  Wrap call to doctest's MESSAGE() method in Reporter in try/catch block
  Pre-initialize c-ares channel object. Fixes Coverity 1488318
  Ask c-ares for the next timeout instead of passing a fixed value
  Remove obsolete DNS_Mgr::asyncs_timeouts
2022-04-27 17:43:03 +00:00
Christian Kreibich
aced9c3e56 Merge branch 'topic/christian/management-consistency-fixes'
* topic/christian/management-consistency-fixes:
  Management framework: consistency fixes around event() vs Broker::publish()
2022-04-27 09:43:39 -07:00
Tim Wojtulewicz
4d24b9d9b8 Merge remote-tracking branch 'origin/topic/timw/cares-cmake-hates-me'
* origin/topic/timw/cares-cmake-hates-me:
  Rework FindCAres.cmake to not use ExternalProject, fixing OBS builds
2022-04-26 09:52:47 -07:00
Tim Wojtulewicz
66624d6840 Merge remote-tracking branch 'origin/topic/timw/1635-c-ares'
* origin/topic/timw/1635-c-ares:
  Add DNS fuzzing corpus from c-ares
  Update NEWS with information about DNS_Mgr changes
  Set larger UDP buffer to avoid TCP fallback if possible
  Add new features to IOSource::Manager, used by DNS_Mgr
  Add out_ptr, use for c-ares interface calls
  Use shared_ptrs for DNS_Mapping objects
  Similar map changes for async requests
  Store all mappings in a single map instead of split by type
  Don't split ipv4/ipv6 mappings into separate pointers
  Handle cancelled/destruction statuses in callback methods
  Pass dns_mgr down into the DNS_Request objects
  Move PTR request handling to ares_create_query()
  Add merging to DNS_Mgr::AddResult() to support both ipv4 and ipv6 responses simultaneously
  Rework DNS_Mgr API to be more consistent and to support more request types
  Replace nb_dns library with C-Ares
  Add unit testing for DNS_Mgr and related classes
  Update doc gen VM to ubuntu-latest, output cmake version during configure
  Use doctest macro to tie Reporter output to test cases
  Add const versions of dereference operators for DictEntry
  Add DNS fuzzer
2022-04-22 13:11:50 -07:00
Tim Wojtulewicz
91db2e984e Merge remote-tracking branch 'bbannier/topic/bbannier/generate_docs_action_forks'
* bbannier/topic/bbannier/generate_docs_action_forks:
  Fix generate-docs action for running on forks.
2022-04-22 07:01:07 -07:00