Commit graph

537 commits

Author SHA1 Message Date
Jon Siwek
fd7402e8d6 Merge remote-tracking branch 'origin/topic/timw/analyzer-iterators'
- During merge, simplified the API docs for Analyzer::DeleteChild()
  (found it unclear/unnecessary to refer to any `std` container types)

* origin/topic/timw/analyzer-iterators:
  Clean up iterator usage in Analyzer
2021-03-23 16:33:17 -07:00
Jon Siwek
80221cb958 Merge remote-tracking branch 'origin/topic/vern/use-defs'
- Added explicit test case of "unused assignement" warning and
  &is_used suppression during merge.

* origin/topic/vern/use-defs:
  activate &is_used
  removed unnecessary statement flagged by Coverity (thanks, Jon!)
  Adjust reference/move nitpicks in use-def/reduce code
  Adjust some whitespace in UseDefs.cc
  updates to "usage" test suite alternative now that more warnings are generated
  suppress usage warning in baseline script
  "xform" alternative baseline update, needed for recent change to master
  baseline update due to shift in number of lines in base intel framework script
  environment variable (which has precedence) not flags for baseline usage test
  splitting out "usage" test suite alternative into -u/-uu versions
  adding &is_used attribute for base scripts - not actually needed yet, but will be once optimization is added
  removing unused assignments from base scripts
  driver glue for invoking use-def construction
  classes for managing and propagating use-defs
  enhancements/changes to the Reduce class in preparation for use-defs
  some tidying with smart pointers
  flag/environment variable for dumping use-defs
  &is_used attribute to suppress set-but-not-used usage warnings
  whitespace micro-preening
2021-02-23 12:11:24 -08:00
Jon Siwek
03f74958f3 Merge remote-tracking branch 'origin/topic/vern/reaching-defs'
* origin/topic/vern/reaching-defs: (36 commits)
  added &is_assigned test case for variable rather than record field
  Speedup ReachingDefs logic by ~15%
  Simplify ReachingDefs::RDMap() accessor
  test for -uu correctly tracking $?, and not misled by conditional assignments
  &is_set => &is_assigned
  remove pending maybe-reconsider-this comment
  fixes for ?$ operator - always track it, and assume subrecords are initialized
  speedup (and more coherent memory management) for tracking RDs
  fixes for generating and evaluating RDs associate with ?$ expressions
  fix for failure to reduce InlineExpr's to CatchReturnStmt's
  inlining fix: propagate identifier attributes (such as &is_set)
  tidier memory management
  fix for an ancient bug - surprising that this hasn't caused problems previously
  Fix IntrusivePtr release leaks in reaching-def logic
  Change dynamic_cast in reaching-def logic to static_cast
  Adjust some reaching-def memory management
  Update a couple baselines for "xform" alternative
  Adjust various reaching-def move/reference semantics
  Change LambdaExpr::OuterIDs() accessor to return const-reference
  Simplify declaration of DefPointType enum
  ...
2021-02-05 11:02:01 -08:00
Jon Siwek
c27bf62217 Merge remote-tracking branch 'origin/topic/timw/1389-vntag'
Merge adjustment: changed test case to use `zeek -b`

* origin/topic/timw/1389-vntag:
  GH-1389: Skip VN-Tag headers
2021-02-03 11:22:13 -08:00
Jon Siwek
b8c563dbdd Deprecate zeekenv() and use getenv() directly 2021-01-29 16:55:44 -08:00
Jon Siwek
bd0c1bcde7 Merge remote-tracking branch 'origin/topic/timw/dict-iterators'
* origin/topic/timw/dict-iterators:
  Implement standard-library-compatible iterators for Dictionary
2021-01-14 19:31:17 -08:00
Tim Wojtulewicz
892124378c Implement standard-library-compatible iterators for Dictionary 2021-01-14 13:49:40 -07:00
Jon Siwek
c166288248 Add NEWS entries for lambda/capture-list functionality 2021-01-11 16:29:39 -08:00
Jon Siwek
321a027d07 Remove unusable/broken RocksDB code and options
The Broker RockSDB data store backend was previously unusable
and broken, so all code and options related to it are now removed.
2021-01-11 11:12:59 -08:00
Jon Siwek
604fbea00d Merge remote-tracking branch 'origin/topic/johanna/1323'
* origin/topic/johanna/1323:
  TLS 1.3 changes: Address review feedback
  Add one more TLS 1.3 testcase and update NEWS
  Fix TLS 1.3 session resumption detection.
  Introduce ssl_probable_encrypted_handshake_message event
  SSL Analyzer: ignore CCS for TLS 1.3
  TLS analyzer: change logic to track TLS 1.3 connection establishment
2020-12-22 11:00:59 -08:00
Johanna Amann
886d7178ef TLS 1.3 changes: Address review feedback
Only minor changes, new consts, and documentation updates.

Part of GH-1335. Addresses GH-1323.
2020-12-18 10:51:36 +00:00
Johanna Amann
22ed75c3ce Add one more TLS 1.3 testcase and update NEWS 2020-12-15 16:57:26 +00:00
Jon Siwek
a475cc7a29 Update NEWS with 4.1.0 placeholder sections 2020-12-14 21:44:30 -08:00
Christian Kreibich
d1d218b5cc Install zkg as part of the Zeek distribution.
- Add auxil/package-manager submodule as an optional build
  source. When the submodule is present, zkg gets installed into the
  Zeek installation's bin directory, its config file into etc/zkg, and
  its state into var/lib/zkg. Like zeekctl, zkg finds its own module
  independently of any PYTHONPATH. Installation via pip remains
  supported. You can skip zkg explicitly via --disable-zkg. See the
  NEWS update for details.

- Establish a "zeek/python" subdirectory under libdir as the common place
  for Python modules in the Zeek distribution. This now separates out
  the Broker Python bindings, ZeekControl, and zkg's Python module.

- Add configure flags to allow customizing this Python folder, in
  three ways: --python-dir, --python-prefix, and --python-home. These
  differ in the logic they automatically add to the path, and build on
  the logic already used in Broker.

- Include a (comented-out) @load for zkg's packages folder in
  local.zeek.

- Bump zeekctl to move to this new location.

- Bump doc to include installation instructions

- Update NEWS accordingly.
2020-12-11 18:43:47 -08:00
Jon Siwek
bb11ce47a0 Merge remote-tracking branch 'origin/topic/neverlord/caf-migration'
Merge adjustments:
- Fix the `--with-caf=` option from emitting messages about a variable
  having an unset value.
- Fix the `zeek-config --caf_root` option

* origin/topic/neverlord/caf-migration:
  Update submodule(s)
  Fix message ordering of Broker messages
  Fix UB in shutdown of Broker manager
  Preserve string output of Broker errors
  Migrate to CAF 0.18
2020-12-10 14:14:38 -08:00
Johanna Amann
47ceac2491 Merge remote-tracking branch 'origin/topic/johanna/GH-348'
* origin/topic/johanna/GH-348:
  Sumstats: allow users to manage epoch manually
  Sumstats: epoch_finished was not called under certain circumstances
2020-12-04 18:49:53 +00:00
Jon Siwek
04385ab8b2 Update NEWS 2020-12-03 16:33:07 -08:00
Jon Siwek
71f9340e75 Merge branch 'add-X-to-double' of https://github.com/ynadji/zeek
- Minor whitespace/grammar/doc tweaks during merge

* 'add-X-to-double' of https://github.com/ynadji/zeek:
  Add `count_to_double` and `int_to_double` bif functions
2020-12-02 12:54:08 -08:00
Jon Siwek
3605e04d83 Update minimum required CMake to 3.5
Also now uses CMake's ENABLE_EXPORTS target property for the zeek
executable to ensure symbols are visible to plugins.  Prior to CMake
3.4, the policy was to export symbols by default for certain platforms,
but later versions need either the explicit target property or policy.
2020-12-01 22:13:52 -08:00
Jon Siwek
33a55a04c3 Update documentation for Python >= 3.5 requirement 2020-11-25 14:57:09 -08:00
Jon Siwek
499cb11dc3 Merge remote-tracking branch 'origin/topic/christian/libdir-improvements'
* origin/topic/christian/libdir-improvements:
  Update CMakeLists.txt to make portability symlink absolute
  Improve support for custom libdir locations
2020-11-24 15:24:14 -08:00
Jon Siwek
02c0b33b54 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1264-ssh-host-key-fingerprints' into master
* origin/topic/jsiwek/gh-1264-ssh-host-key-fingerprints:
  Simply ssh/main.zeek by using "ssh_server_host_key" for fingerprinting
  Deprecate "ssh1_server_host_key" parameters *e* and *p*
  GH-1264: Implement "ssh_server_host_key" event
2020-11-16 11:22:37 -08:00
Johanna Amann
50a49eabb5 Merge branch 'topic/vbrevet/ssh_version' of https://github.com/vbrevet/zeek
* 'topic/vbrevet/ssh_version' of https://github.com/vbrevet/zeek:
  [SSH] Handle SSH version 1.99 SSH can set in its identification a version 1.99 (SSH-1.99-xxx). That means the client/server is compatible with SSHv1 and SSHv2. So the version choice depends of the both side.
2020-11-16 11:05:30 +00:00
Jon Siwek
45449dad72 Deprecate "ssh1_server_host_key" parameters *e* and *p*
They are named such that *e* is actually the modulus, not the exponent.
The replacement parameters are named *exponent* and *modulus* for
clarity.
2020-11-13 22:58:56 -08:00
Jon Siwek
aab99b743d Merge remote-tracking branch 'origin/topic/jsiwek/type-alias-introspection'
* origin/topic/jsiwek/type-alias-introspection:
  Add enum_names() BIF to return names of an enum type's values
  Add type_aliases() BIF for introspecting type-names of types/values
  Change Type::type_aliases map to store IntrusivePtr
  Fix lookup_ID() BIF to return enum values
2020-11-12 14:30:32 -08:00
Tim Wojtulewicz
a6a4b976ec Support for additional DNS RR Type: LOC[29], SSHFP[44], NSEC3PARAM[51], custom BIND9 signaling[65534] 2020-11-11 13:35:51 -07:00
Tim Wojtulewicz
c3cf36e135 GH-1221: Add unknown_protocols.log for logging packet analyzer lookup failures 2020-11-09 20:37:26 -07:00
Jon Siwek
1dda387ac9 Change Type::type_aliases map to store IntrusivePtr
And deprecate Type::GetAliases() and Type::AddAlias() since they
took raw pointers.  Now replaced with Type::Aliases() and
Type::RegisterAlias().
2020-11-06 17:18:20 -08:00
Jon Siwek
73c1af838c GH-251 (revert): remove coercion-to-signed-integer for |x| expressions
For `|x|`, where `x` is an expression with an integral result, an
implicit coercion of that result into signed `int` type no longer takes
place.

This was actually the behavior before Zeek 3.0 as well, but the attempt
to prevent mistakes that easily result from integer literals in Zeek
being unsigned like `|5 - 9|` causing an overflow/wraparound and
yielding a very large number is not generally consistent since overflows
are still generally able to happen in other ways and also in other
contexts besides just absolute-values.  So the preference was to revert
to a behavior that favors consistency.  For reference, see
https://github.com/zeek/zeek/pull/251#issuecomment-713956976
2020-10-22 17:41:53 -07:00
Johanna Amann
65125121d8 Merge branch 'topic/johanna/GH-169'
* topic/johanna/GH-169:
  Make event ordering deterministic
  dump-events: try to make baseline work on all systems
  Introduce generate_all_events bif and add option to misc/dump-events

Fixes GH-169
2020-10-16 07:11:57 +00:00
Jon Siwek
6cb0cb9746 Merge remote-tracking branch 'origin/topic/seth/make-cluster-port-optional'
* origin/topic/seth/make-cluster-port-optional:
  Apply suggestions from code review
  Update scripts/base/frameworks/cluster/main.zeek
  Make defining a port number for hosts in a cluster that only connect outbound optional
2020-10-13 15:35:43 -07:00
Jon Siwek
d62fb3ab9a Merge remote-tracking branch 'origin/topic/jsiwek/zeek-script-args'
* origin/topic/jsiwek/zeek-script-args:
  Improve zeek_script_args test case and documentation
  Apply suggestions from code review
  Add a test for script args.
  Fixed an option processing bug
  Make it possible to pass command line options through to scripts.
2020-10-13 13:34:18 -07:00
Jon Siwek
2d3b4dab74 Improve zeek_script_args test case and documentation 2020-10-13 12:40:53 -07:00
Jon Siwek
86e10bfb7e Merge remote-tracking branch 'origin/topic/vlad/caploss_no_traffic'
- Tweaked the Too_Little_Traffic notice message to avoid
  cluster-specific terminology.

* origin/topic/vlad/caploss_no_traffic:
  Fix scheduling due to network_time being 0 in zeek_init
  Add test for CaptureLoss::Too_Little_Traffic
  Add CaptureLoss::Too_Little_Traffic
  Add CaptureLoss::initial_watch_interval for a quick read on cluster health after startup.
  Documentation update, reference the threshold variable. [nomail] [skip ci]
  Whitespace fixes only [nomail] [skip ci]
2020-10-12 17:05:05 -07:00
Jon Siwek
b73cc816e9 Merge remote-tracking branch 'origin/topic/jsiwek/gh-779-udp-state-sig'
* origin/topic/jsiwek/gh-779-udp-state-sig:
  GH-779: Add "udp-state" signature condition
  Rename RuleConditionTCPState::TCPState enum values
  Rename signature parser tokens to not be TCP-specific
2020-10-12 11:16:54 -07:00
Robin Sommer
b0bf9f02c8 Merge remote-tracking branch 'origin/topic/christian/364-logfilter-hooks' into master
(Adding a NEWS entry.)

* origin/topic/christian/364-logfilter-hooks:
  Update testing/btest/scripts/base/frameworks/logging/hooks.zeek
  Btests for log filter policy hooks
  Btest baseline updates to reflect new logging policy hooks
  Migrate existing use of filter predicates to policy hooks
  Support for log filter policy hooks
2020-10-07 08:44:50 +00:00
Jon Siwek
c1492942bb Merge branch 'add-dce-rpc-payloads' of https://github.com/ynadji/zeek
- Changed the new stub events to correctly check for existence of
  their associated handler before generating an event

- Added a test case for the new stub event

* 'add-dce-rpc-payloads' of https://github.com/ynadji/zeek:
  Add stub payload to dce_rpc_request and dce_rpc_response
2020-09-25 14:39:30 -07:00
Tim Wojtulewicz
a9d5d083a2 Update NEWS 2020-09-23 11:31:42 -07:00
Jon Siwek
05cf511f18 GH-1119: add base/protcols/conn/removal-hooks.zeek
This adds two new functions: `Conn::register_removal_hook()` and
`Conn::unregister_removal_hook()` for registering a hook function to be
called back during `connection_state_remove`.  The benefit of using hook
callback approach is better scalability: the overhead of unrelated
protocols having to dispatch no-op `connection_state_remove` handlers is
avoided.
2020-09-11 12:12:10 -07:00
Tim Wojtulewicz
73080c1e60 Mark Continuation.h and PacketDumper.h as deprecated 2020-08-21 15:56:35 -07:00
Tim Wojtulewicz
476e98b413 Remove PRI_PTR_COMPAT macros 2020-08-20 16:11:46 -07:00
Tim Wojtulewicz
8411adf9e1 Merge remote-tracking branch 'FlyingWithJerome/master'
* FlyingWithJerome/master:
  remove variables in netvar, use tabs in DNS.h and polish comments in dns events.
  add new line at the end of the test baseline
  add data+=option_len to skip unknown bytes
  better explanation to server cookie
  better explanation to server cookie
  add edns-cookie testcase
  remove data+=option_len error in cookie and keepalive
  add EDNS cookie parsing
  lazy commit
  lazy commit
  lazy commit
  add testcases
  add units (100ms) to comments
  initial adding EDNS TCP keepalive
2020-08-20 10:41:51 -07:00
Jon Siwek
670bf02c95 Update NEWS about change in Dictionary implementation 2020-08-11 11:26:21 -07:00
Jon Siwek
170752fa99 Fix extract_first_email_addr() to really return the first email
The use of find_all() in extract_email_addrs_vec() extracted occurrences
to an intermediate set and thus lost any sense of ordering.

This changes extract_email_addrs_vec() to use find_all_ordered() and
return all occurrences of email addresses found in the argument,
included duplicates, with their order of occurrence preserved.
2020-08-11 11:26:21 -07:00
Johanna Amann
8e99d4b170 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1080-error-on-enum-redefinition-conflict'
* origin/topic/jsiwek/gh-1080-error-on-enum-redefinition-conflict:
  Fix incorrect conflict detection of namespaced-enum-names
  Improve error message for an enum name conflicting with non-enum ID
  GH-1080: Treat enum name re-use across different enum types as an error
  GH-1080: Rename conflicting NetControl::DROP enum definitions

Fixes GH-1080
2020-08-04 20:25:40 +00:00
Jon Siwek
613b27eec7 GH-1080: Rename conflicting NetControl::DROP enum definitions
``NetControl::DROP`` had 3 conflicting definitions that could potentially
be used incorrectly without any warnings or type-checking errors.
Such enum redefinition conflicts are now caught and treated as errors,
so the ``NetControl::DROP`` enums had to be renamed:

* The use as enum of type ``Log::ID`` is renamed to ``NetControl::DROP_LOG``

* The use as enum of type ``NetControl::CatchReleaseInfo`` is renamed to
  ``NetControl::DROP_REQUESTED``

* The use as enum of type ``NetControl::RuleType`` is unchanged and still
  named ``NetControl::DROP``
2020-07-31 16:11:35 -07:00
Johanna Amann
cd3400f957 Re-add TYPE_COUNTER without function and deprecation marker. 2020-07-31 14:53:49 -07:00
Jon Siwek
20c9e54f9b Reserve spot in NEWS for 4.0.0 2020-07-27 11:53:01 -07:00
Tim Wojtulewicz
ac57377a94 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1068-zeek-include-dir'
* origin/topic/jsiwek/gh-1068-zeek-include-dir:
  GH-1068: Add zeek symlink to allow "zeek/Foo.h" include style
2020-07-23 08:32:20 -07:00
Robin Sommer
c3f4971eb2 Merge remote-tracking branch 'origin/topic/johanna/table-changes'
* origin/topic/johanna/table-changes: (26 commits)
  TableSync: try to make test more robust & add debug output
  Increase timeouts to see if FreeBSD will be happy with this.
  Try to make FreeBSD test happy with larger timeout.
  TableSync: refactor common functionality into function
  TableSync: don't raise &on_change, smaller fixes
  TableSync: rename auto_store -> table_store
  SyncTables: address feedback part 1 - naming (broker and zeek)
  BrokerStore <-> Zeek Tables: cleanup and bug workaround
  Zeek Table<->Brokerstore: cleanup, documentation, small fixes
  BrokerStore<->Zeek table: adopt to recent Zeek API changes
  BrokerStore<->Zeek Tables Fix a few small test failures.
  BrokerStore<->Zeek tables: allow setting storage location & tests
  BrokerStore<->Zeek tables: &backend works for in-memory stores.
  BrokerStore<->Zeek table - introdude &backend attribute
  BrokerStore<->Zeek tables: test for clones synchronizing to a master
  BrokerStore<->Zeek tables: load persistent tables on startup.
  Brokerstore<->Tables: attribute conflicts
  Zeek/Brokerstore updates: expiration
  Zeek/Brokerstore updates: add test that includes updates from clones
  Zeek/Brokerstore updates: first working end-to-end test
  ...
2020-07-21 15:39:39 +00:00