Commit graph

2710 commits

Author SHA1 Message Date
Jon Siwek
7bf885b0b8 Merge remote-tracking branch 'origin/topic/vlad/expose_supervisor_rotation_func'
- Restored a deprecated version of 'supervisor_rotation_format_func'
  during merge.

* origin/topic/vlad/expose_supervisor_rotation_func:
  Rename supervisor_rotation_format_func to archiver_rotation_format_func, and expose it for non-supervised setups
2021-03-26 17:18:52 -07:00
Vlad Grigorescu
acfb21c5a6 Rename supervisor_rotation_format_func to archiver_rotation_format_func, and expose it for non-supervised setups
Closes #1463
2021-03-26 15:26:48 -05:00
Johanna Amann
c23e3ca105 Heartbleed: fix substraction order.
The larger number was substracted from the smaller one leading to an
integer overflow. However, no information was lost due to everything
also being present in the notice message.

Fixes GH-1454
2021-03-25 13:41:12 +00:00
Michael Dopheide
83d5b44462 Sometimes user_agent is missing 2021-03-24 15:48:09 -05:00
Tim Wojtulewicz
f45df63cd0 Merge remote-tracking branch 'origin/topic/vern/zval'
* origin/topic/vern/zval: (42 commits)
  whitespace tweaks
  resolved some TODO comments
  remove unnecessary casts, and change necessary ones to use static_cast<>
  explain cmp_func default
  change functions for ZVal type management to static members
  fix some unsigned/signed integer warnings
  address lint concern about uninitialized variable
  Remove use of obsolete forward-declaration macros
  fix #include's that lack zeek/ prefixes
  explicitly populate holes created in vectors
  fixes for now-incorrect assumption that GetField always returns an existing ValPtr
  memory management for assignment to vector elements
  memory management for assignment to record fields
  destructor cleanup from ZAM_vector/ZAM_record
  fix #include's that lack zeek/ prefixes
  overlooked another way in which vector holes can be created
  initialize vector holes to the correct corresponding type
  explicitly populate holes created in vectors
  fix other instances of GetField().get() assuming long-lived ValPtr's
  fix for now-incorrect assumption that GetField always returns an existing ValPtr
  ...
2021-03-23 20:44:19 -07:00
Tim Wojtulewicz
4f24c02cd5 Merge remote-tracking branch 'origin/topic/vern/cpp-prep-fixes'
* origin/topic/vern/cpp-prep-fixes:
  fix for associating current scope with the name of enums; name tidying
  avoid infinite recursion in same_type() if it is analyzing recursive types
  remove iffy reliance on type punning that relies on interpreter's behavior
  fixes for propagating optimization options, and pruning script function analysis
2021-03-18 12:53:14 -07:00
Vern Paxson
b473bc48e1 remove iffy reliance on type punning that relies on interpreter's behavior 2021-03-18 08:46:18 -07:00
Vern Paxson
5a8ba8c3e5 lint fixes: ensuring functions return values, robustness to nil Val's 2021-03-18 08:21:19 -07:00
Vern Paxson
e21c0f1115 correcting some simple typos 2021-03-18 08:19:46 -07:00
Seth Hall
db7aba0835 ts fields in SMB logs now default to network_time()
This avoids a problem identified by amanbansal2709 in pull
request #1288. I fixed it in a different way than that pull request
by making sure the ts field is always set so that this isssue doesn't
return in the future.
2021-03-10 13:52:44 +00:00
Jon Siwek
6af436aad3 GH-1426: Improve handling of Broker data store creation failures
Broker::create_master() and Broker::create_clone() now return
a valid value even when there's a failure to open the backend database
(e.g. SQLite filesystem error).  In that case, the returned value can
still be passed into other data store operations, but they'll fail
immediately with an error.  Broker::is_closed() can now also be used to
determine whether the data store creation calls failed.
2021-03-06 02:32:29 -08:00
Vern Paxson
b065582319 fix mis-typed record fields that happened to work in original RecordVal impl. 2021-02-24 16:33:17 -08:00
Jon Siwek
737d2c390b Support explicit disabling of file analyzers 2021-02-23 15:50:18 -08:00
Vern Paxson
f21f7b65d5 adding &is_used attribute for base scripts - not actually needed yet, but will be once optimization is added 2021-02-06 10:59:58 -08:00
Vern Paxson
4f2b138e72 removing unused assignments from base scripts 2021-02-06 10:59:37 -08:00
Jon Siwek
03f74958f3 Merge remote-tracking branch 'origin/topic/vern/reaching-defs'
* origin/topic/vern/reaching-defs: (36 commits)
  added &is_assigned test case for variable rather than record field
  Speedup ReachingDefs logic by ~15%
  Simplify ReachingDefs::RDMap() accessor
  test for -uu correctly tracking $?, and not misled by conditional assignments
  &is_set => &is_assigned
  remove pending maybe-reconsider-this comment
  fixes for ?$ operator - always track it, and assume subrecords are initialized
  speedup (and more coherent memory management) for tracking RDs
  fixes for generating and evaluating RDs associate with ?$ expressions
  fix for failure to reduce InlineExpr's to CatchReturnStmt's
  inlining fix: propagate identifier attributes (such as &is_set)
  tidier memory management
  fix for an ancient bug - surprising that this hasn't caused problems previously
  Fix IntrusivePtr release leaks in reaching-def logic
  Change dynamic_cast in reaching-def logic to static_cast
  Adjust some reaching-def memory management
  Update a couple baselines for "xform" alternative
  Adjust various reaching-def move/reference semantics
  Change LambdaExpr::OuterIDs() accessor to return const-reference
  Simplify declaration of DefPointType enum
  ...
2021-02-05 11:02:01 -08:00
Vern Paxson
c991c54690 &is_set => &is_assigned 2021-02-04 12:18:46 -08:00
Jon Siwek
cfe29c2488 Merge branch 'master' into topic/vern/reaching-defs 2021-02-01 17:27:37 -08:00
Tim Wojtulewicz
f53448ccc9 GH-1389: Skip VN-Tag headers 2021-02-01 14:34:56 -07:00
Jon Siwek
1ca85f0221 Merge remote-tracking branch 'origin/topic/timw/deprecation-cleanup'
Merge adjustments:
- Removed some stale str_split() references from docs
- Renumbered TypeTag enum comments
- Simplified test-case for @unload (don't need .bro files anymore)

* origin/topic/timw/deprecation-cleanup:
  Doc updates
  Fix language.init-in-anon-function btest due to changes to log filter predicates
  Remove deprecated log filter predicates for 4.1
  Remove Plugin::HookCallFunction and fix tests related to it
  Remove support for .bro script extension and BRO_ environment variables
  Remove deprecated ICMP events
  Remove some deprected methods/events from bif files
  Remove TYPE_COUNTER
  Remove all of the random single-file deprecations
  Remove all fully-deprecated files
  Update bifcl submodule to remove deprecations from generated code
2021-01-29 16:40:54 -08:00
Jon Siwek
e82824b638 Fix various broken links in script documentation 2021-01-28 17:46:58 -08:00
Christian Kreibich
4ce3bf3cd2 Remove deprecated log filter predicates for 4.1
Update the logging framework tests: since hooks operate
by name, they cannot be anonymous. I'm also dropping the &optional
attribute from the status field, since here know that the values are
actually defined, and access to an optional status field should
normally be guarded by the existence test operator.

Also includes baseline update for plugins.hooks, which picks up the
fact that the pred record field is now gone.
2021-01-27 10:52:40 -07:00
Vern Paxson
0d77b474e6 adding &is_set attributes to base scripts so -u output isn't cluttered 2021-01-23 10:55:27 -08:00
Jon Siwek
3449bc81af Merge remote-tracking branch 'origin/topic/johanna/update-tls-constants-which-changed-a-lot-since-I-last-looked-which-just-shows-i-should-really-look-at-them-more-regularly-but-luckily-they-are-not-really-used-for-anything-so-it-does-not-really-matter-that-much'
* origin/topic/johanna/update-tls-constants-which-changed-a-lot-since-I-last-looked-which-just-shows-i-should-really-look-at-them-more-regularly-but-luckily-they-are-not-really-used-for-anything-so-it-does-not-really-matter-that-much:
  Update SSL consts
2021-01-21 17:33:36 -08:00
Jon Oakley
25de6f28e5 GH-1352: Added flag to stop processing SMTP headers in attached
messages
2021-01-21 14:55:10 -05:00
Johanna Amann
079d4164c0 Merge remote-tracking branch 'origin/topic/christian/fix-config-segfault'
* origin/topic/christian/fix-config-segfault:
  Btest tweak for improved type rendering in config framework errors and set types
  More precise type information in a config framework error message
  Explicitly don't support sets with multiple index types in input/config frameworks
2021-01-21 11:30:36 +00:00
Johanna Amann
8ff05bae1b Update SSL consts
There have been a bunch of mostly TLS 1.3 related changes.
2021-01-20 14:14:47 +00:00
Christian Kreibich
421639e7a7 Explicitly don't support sets with multiple index types in input/config frameworks
The input framework's Manager::IsCompatibleType() already rejected
sets with multiple index types that aren't all the same (i.e. that are
not pure). Pure ones (e.g. "set[addr,addr]") slipped through and could
cause Zeek to segfault elsewhere in the config framework due to type
comparison subtleties. Note that the ASCII reader can't read such sets
anyway, so this method now rejects sets with any kind of index-type
tuple.

In the config framework, the script-level change handler has a risky
conversion from any to set[bool], which can trigger segfaults when the
underlying set's index is a type tuple. We now prevent this code path
by ensuring it only applies to sets with a single index type.
2021-01-11 13:35:46 -08:00
Jon Siwek
321a027d07 Remove unusable/broken RocksDB code and options
The Broker RockSDB data store backend was previously unusable
and broken, so all code and options related to it are now removed.
2021-01-11 11:12:59 -08:00
Jon Siwek
7cf08d4e58 Merge remote-tracking branch 'origin/topic/neverlord/1336'
* origin/topic/neverlord/1336:
  Fix subtle race on data store initialization
2020-12-23 10:36:09 -08:00
Dominik Charousset
8d726ed07a Fix subtle race on data store initialization 2020-12-22 21:15:17 +01:00
Johanna Amann
886d7178ef TLS 1.3 changes: Address review feedback
Only minor changes, new consts, and documentation updates.

Part of GH-1335. Addresses GH-1323.
2020-12-18 10:51:36 +00:00
Johanna Amann
3c95c9a956 Fix TLS 1.3 session resumption detection.
Now we detect TLS 1.3 session resumption by looking if both sides have
the PSK extension set, which is much more exact than the previous
approach.
2020-12-15 16:34:47 +00:00
Johanna Amann
bea3075c1f TLS analyzer: change logic to track TLS 1.3 connection establishment
This commit changes the logic that is used to tracks connection
establishment - and moves it from scriptland into the core.

TLS 1.3 connection establishment is much more finnicky for us than the
establishment of earlier versions - since we cannot rely on the CCS
message anymore (which is meaningless and not sent in a lot of cases).

With this commit, the ssl_encrypted_data message gets raised for
encrypted TLS 1.3 handshake messages - which is much more correct than
the behavior before that just interpreted them as plaintext messages.

I will refine this a bit more - at the moment the connection established
event happens a bit too early - earlier than TLS 1.3 connections
actually can be estasblished.

Part of GH-1323
2020-12-14 19:51:05 +00:00
Christian Kreibich
d1d218b5cc Install zkg as part of the Zeek distribution.
- Add auxil/package-manager submodule as an optional build
  source. When the submodule is present, zkg gets installed into the
  Zeek installation's bin directory, its config file into etc/zkg, and
  its state into var/lib/zkg. Like zeekctl, zkg finds its own module
  independently of any PYTHONPATH. Installation via pip remains
  supported. You can skip zkg explicitly via --disable-zkg. See the
  NEWS update for details.

- Establish a "zeek/python" subdirectory under libdir as the common place
  for Python modules in the Zeek distribution. This now separates out
  the Broker Python bindings, ZeekControl, and zkg's Python module.

- Add configure flags to allow customizing this Python folder, in
  three ways: --python-dir, --python-prefix, and --python-home. These
  differ in the logic they automatically add to the path, and build on
  the logic already used in Broker.

- Include a (comented-out) @load for zkg's packages folder in
  local.zeek.

- Bump zeekctl to move to this new location.

- Bump doc to include installation instructions

- Update NEWS accordingly.
2020-12-11 18:43:47 -08:00
Johanna Amann
14a448cddc Update Mozilla Root Store 2020-12-09 21:46:32 +00:00
Jon Siwek
1f352a1e4d Fix misassociation of Zeekygen-style comments within function scopes
All Zeekygen-style comments relate to entities at global scope, so those
found within functions are now ignored instead of misassociated.
2020-12-04 19:29:11 -08:00
Jon Siwek
6323b0a8c2 Remove an extraneous Zeekygen-style comment 2020-12-04 19:20:57 -08:00
Johanna Amann
47ceac2491 Merge remote-tracking branch 'origin/topic/johanna/GH-348'
* origin/topic/johanna/GH-348:
  Sumstats: allow users to manage epoch manually
  Sumstats: epoch_finished was not called under certain circumstances
2020-12-04 18:49:53 +00:00
Tim Wojtulewicz
e27008ef26 GH-1184: Add 'source' field to weird log denoting where the weird was reported 2020-12-01 09:34:37 -07:00
Tim Wojtulewicz
01ec5ebdcd Reverts the regex change in dead3226a5.
The regex change broke some of the external tests. I added some more cases
to the regular email btest to hopefully cover all of the cases better.
2020-11-18 15:35:17 -07:00
Tim Wojtulewicz
a56fbe408f Merge remote-tracking branch 'christina23/master'
* christina23/master:
  Added unit tests for regex fix
  Improved regex for SMTP parsing
2020-11-17 15:01:21 -07:00
Jon Siwek
6ed4850a26 Merge remote-tracking branch 'origin/topic/jsiwek/gh-352-improve-sql-injection-regex' into master
* origin/topic/jsiwek/gh-352-improve-sql-injection-regex:
  GH-352: Improve HTTP::match_sql_injection_uri regex
2020-11-16 11:51:04 -08:00
Jon Siwek
02c0b33b54 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1264-ssh-host-key-fingerprints' into master
* origin/topic/jsiwek/gh-1264-ssh-host-key-fingerprints:
  Simply ssh/main.zeek by using "ssh_server_host_key" for fingerprinting
  Deprecate "ssh1_server_host_key" parameters *e* and *p*
  GH-1264: Implement "ssh_server_host_key" event
2020-11-16 11:22:37 -08:00
christina23
dead3226a5 Improved regex for SMTP parsing 2020-11-16 11:43:35 -05:00
Johanna Amann
7362f30c3a Sumstats: allow users to manage epoch manually
This change allows users to specify an epoch length of 0, which means
that the user manually has to finish the epochs. A new next_epoch
function is introduced to allow users to manually end epochs.

Addresses GH-348
2020-11-16 15:48:46 +00:00
Johanna Amann
8ffbc69a4d Sumstats: epoch_finished was not called under certain circumstances
In non-clustered mode, epoch_finished was not called when there was no
data during the epoch.

This behavior does not fit the documentation, and also is different in
cluster-mode, where epoch_finished is, indeed, called after every epoch.

This small change fixes this behavior.
2020-11-16 15:36:50 +00:00
Jon Siwek
331b94db39 Simply ssh/main.zeek by using "ssh_server_host_key" for fingerprinting 2020-11-14 08:40:27 -08:00
Brevet Vivien
3769ed6c66 [SSH] Handle SSH version 1.99
SSH can set in its identification a version 1.99 (SSH-1.99-xxx).
That means the client/server is compatible with SSHv1 and SSHv2.
So the version choice depends of the both side.

1.99 : 1.99 => 2.0
1.99 : 1.x  => 1.x
1.99 : 2.0  => 2.O

(see "Compatibility With Old SSH Versions" in RFC 4253)
2020-11-14 15:33:34 +01:00
Jon Siwek
45449dad72 Deprecate "ssh1_server_host_key" parameters *e* and *p*
They are named such that *e* is actually the modulus, not the exponent.
The replacement parameters are named *exponent* and *modulus* for
clarity.
2020-11-13 22:58:56 -08:00