Commit graph

2292 commits

Author SHA1 Message Date
Jon Siwek
321a027d07 Remove unusable/broken RocksDB code and options
The Broker RockSDB data store backend was previously unusable
and broken, so all code and options related to it are now removed.
2021-01-11 11:12:59 -08:00
Jon Siwek
7cf08d4e58 Merge remote-tracking branch 'origin/topic/neverlord/1336'
* origin/topic/neverlord/1336:
  Fix subtle race on data store initialization
2020-12-23 10:36:09 -08:00
Dominik Charousset
8d726ed07a Fix subtle race on data store initialization 2020-12-22 21:15:17 +01:00
Johanna Amann
886d7178ef TLS 1.3 changes: Address review feedback
Only minor changes, new consts, and documentation updates.

Part of GH-1335. Addresses GH-1323.
2020-12-18 10:51:36 +00:00
Johanna Amann
3c95c9a956 Fix TLS 1.3 session resumption detection.
Now we detect TLS 1.3 session resumption by looking if both sides have
the PSK extension set, which is much more exact than the previous
approach.
2020-12-15 16:34:47 +00:00
Johanna Amann
bea3075c1f TLS analyzer: change logic to track TLS 1.3 connection establishment
This commit changes the logic that is used to tracks connection
establishment - and moves it from scriptland into the core.

TLS 1.3 connection establishment is much more finnicky for us than the
establishment of earlier versions - since we cannot rely on the CCS
message anymore (which is meaningless and not sent in a lot of cases).

With this commit, the ssl_encrypted_data message gets raised for
encrypted TLS 1.3 handshake messages - which is much more correct than
the behavior before that just interpreted them as plaintext messages.

I will refine this a bit more - at the moment the connection established
event happens a bit too early - earlier than TLS 1.3 connections
actually can be estasblished.

Part of GH-1323
2020-12-14 19:51:05 +00:00
Johanna Amann
14a448cddc Update Mozilla Root Store 2020-12-09 21:46:32 +00:00
Jon Siwek
6323b0a8c2 Remove an extraneous Zeekygen-style comment 2020-12-04 19:20:57 -08:00
Johanna Amann
47ceac2491 Merge remote-tracking branch 'origin/topic/johanna/GH-348'
* origin/topic/johanna/GH-348:
  Sumstats: allow users to manage epoch manually
  Sumstats: epoch_finished was not called under certain circumstances
2020-12-04 18:49:53 +00:00
Tim Wojtulewicz
e27008ef26 GH-1184: Add 'source' field to weird log denoting where the weird was reported 2020-12-01 09:34:37 -07:00
Tim Wojtulewicz
01ec5ebdcd Reverts the regex change in dead3226a5.
The regex change broke some of the external tests. I added some more cases
to the regular email btest to hopefully cover all of the cases better.
2020-11-18 15:35:17 -07:00
Tim Wojtulewicz
a56fbe408f Merge remote-tracking branch 'christina23/master'
* christina23/master:
  Added unit tests for regex fix
  Improved regex for SMTP parsing
2020-11-17 15:01:21 -07:00
Jon Siwek
02c0b33b54 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1264-ssh-host-key-fingerprints' into master
* origin/topic/jsiwek/gh-1264-ssh-host-key-fingerprints:
  Simply ssh/main.zeek by using "ssh_server_host_key" for fingerprinting
  Deprecate "ssh1_server_host_key" parameters *e* and *p*
  GH-1264: Implement "ssh_server_host_key" event
2020-11-16 11:22:37 -08:00
christina23
dead3226a5 Improved regex for SMTP parsing 2020-11-16 11:43:35 -05:00
Johanna Amann
7362f30c3a Sumstats: allow users to manage epoch manually
This change allows users to specify an epoch length of 0, which means
that the user manually has to finish the epochs. A new next_epoch
function is introduced to allow users to manually end epochs.

Addresses GH-348
2020-11-16 15:48:46 +00:00
Johanna Amann
8ffbc69a4d Sumstats: epoch_finished was not called under certain circumstances
In non-clustered mode, epoch_finished was not called when there was no
data during the epoch.

This behavior does not fit the documentation, and also is different in
cluster-mode, where epoch_finished is, indeed, called after every epoch.

This small change fixes this behavior.
2020-11-16 15:36:50 +00:00
Jon Siwek
331b94db39 Simply ssh/main.zeek by using "ssh_server_host_key" for fingerprinting 2020-11-14 08:40:27 -08:00
Brevet Vivien
3769ed6c66 [SSH] Handle SSH version 1.99
SSH can set in its identification a version 1.99 (SSH-1.99-xxx).
That means the client/server is compatible with SSHv1 and SSHv2.
So the version choice depends of the both side.

1.99 : 1.99 => 2.0
1.99 : 1.x  => 1.x
1.99 : 2.0  => 2.O

(see "Compatibility With Old SSH Versions" in RFC 4253)
2020-11-14 15:33:34 +01:00
Jon Siwek
45449dad72 Deprecate "ssh1_server_host_key" parameters *e* and *p*
They are named such that *e* is actually the modulus, not the exponent.
The replacement parameters are named *exponent* and *modulus* for
clarity.
2020-11-13 22:58:56 -08:00
Jon Siwek
bd40a97a78 GH-1264: Implement "ssh_server_host_key" event
This event provides host key fingerprints for both SSH1 and SSH2.
2020-11-13 22:58:56 -08:00
Jon Siwek
89af6f2004 Move UnknownProtocol options to init-bare.zeek
Otherwise the `unknown_protocol` event cannot be used independently
from `policy/mic/unknown-protocols.zeek`.
2020-11-11 12:58:38 -08:00
Tim Wojtulewicz
a6a4b976ec Support for additional DNS RR Type: LOC[29], SSHFP[44], NSEC3PARAM[51], custom BIND9 signaling[65534] 2020-11-11 13:35:51 -07:00
Tim Wojtulewicz
efe42bc67b Remove default_analyzer for Ethernet packet analzyer 2020-11-09 19:54:45 -07:00
Tim Wojtulewicz
cd06bf34c7 GH-1215: Remove dispatch_map from packet analysis, replace with BIF methods for registering dispatches 2020-11-02 19:03:25 +00:00
Duffy O'Craven
b32eba579d updated to mention notice_alarm.log by name. That has taken the place of the file formerly named alarm.log 2020-10-27 13:56:30 -07:00
Seth Hall
56e234a762 Changes in response to Johanna's code review. 2020-10-22 13:30:30 -04:00
Seth Hall
552a24e07c Add an option to ignore packets sourced from particular subnets.
It's implemented with a new set[subnet] option named ignore_checksums_nets.

If you populate this set with subnets, any packet with a src address within
that set of subnets will not have it's checksum validated.
2020-10-22 13:23:10 -04:00
Johanna Amann
65125121d8 Merge branch 'topic/johanna/GH-169'
* topic/johanna/GH-169:
  Make event ordering deterministic
  dump-events: try to make baseline work on all systems
  Introduce generate_all_events bif and add option to misc/dump-events

Fixes GH-169
2020-10-16 07:11:57 +00:00
Tim Wojtulewicz
1cf251d1ca Move IP and IP tunnel code from Sessions into packet analyzers 2020-10-15 12:18:30 -07:00
Johanna Amann
5f29f1dca1 Make event ordering deterministic
NetControl::init and filter_change_tracking could basically be raised in
random order.
2020-10-15 16:35:52 +00:00
Jon Siwek
6cb0cb9746 Merge remote-tracking branch 'origin/topic/seth/make-cluster-port-optional'
* origin/topic/seth/make-cluster-port-optional:
  Apply suggestions from code review
  Update scripts/base/frameworks/cluster/main.zeek
  Make defining a port number for hosts in a cluster that only connect outbound optional
2020-10-13 15:35:43 -07:00
Seth Hall
cd330c801d
Apply suggestions from code review
Co-authored-by: Jon Siwek <jsiwek@corelight.com>
2020-10-13 16:48:15 -04:00
Seth Hall
e78386d6e5
Update scripts/base/frameworks/cluster/main.zeek
Co-authored-by: Jon Siwek <jsiwek@corelight.com>
2020-10-13 16:46:26 -04:00
Jon Siwek
d62fb3ab9a Merge remote-tracking branch 'origin/topic/jsiwek/zeek-script-args'
* origin/topic/jsiwek/zeek-script-args:
  Improve zeek_script_args test case and documentation
  Apply suggestions from code review
  Add a test for script args.
  Fixed an option processing bug
  Make it possible to pass command line options through to scripts.
2020-10-13 13:34:18 -07:00
Jon Siwek
2d3b4dab74 Improve zeek_script_args test case and documentation 2020-10-13 12:40:53 -07:00
Jon Siwek
961532a8f7 Merge remote-tracking branch 'origin/topic/seth/pcap_findalldevs'
- Minor adjustments to whitespace/formatting

* origin/topic/seth/pcap_findalldevs:
  Finishing changes from code review.
  Update src/iosource/pcap/pcap.bif
  Update src/iosource/pcap/pcap.bif
  Update scripts/base/init-bare.zeek
  Update src/iosource/pcap/pcap.bif
  I accidentally missed a paren
  New bif to wrap pcap_findalldevs
2020-10-13 10:52:14 -07:00
Seth Hall
92eb7c10da Finishing changes from code review. 2020-10-13 08:35:45 -04:00
Seth Hall
dfa21d54c8
Update scripts/base/init-bare.zeek
Co-authored-by: Jon Siwek <jsiwek@corelight.com>
2020-10-13 08:12:20 -04:00
Seth Hall
7bcbc57401 New bif to wrap pcap_findalldevs 2020-10-12 12:47:23 -04:00
Seth Hall
cf8671d078 Make defining a port number for hosts in a cluster that only connect outbound optional 2020-10-12 10:46:28 -04:00
Seth Hall
2bdc56dfcd Make it possible to pass command line options through to scripts.
The feature is documented with the zeek_script_args variable in
init-bare.zeek.
2020-10-08 15:46:28 -04:00
Robin Sommer
b0bf9f02c8 Merge remote-tracking branch 'origin/topic/christian/364-logfilter-hooks' into master
(Adding a NEWS entry.)

* origin/topic/christian/364-logfilter-hooks:
  Update testing/btest/scripts/base/frameworks/logging/hooks.zeek
  Btests for log filter policy hooks
  Btest baseline updates to reflect new logging policy hooks
  Migrate existing use of filter predicates to policy hooks
  Support for log filter policy hooks
2020-10-07 08:44:50 +00:00
Jon Siwek
506d961b27 Merge branch 'logging-ascii-enable-shadow-logs' of https://github.com/awelzel/zeek into master
- Improved documentation/comment for the new option

* 'logging-ascii-enable-shadow-logs' of https://github.com/awelzel/zeek:
  logging/ascii: Support leftover log rotation in non-supervisor setups
2020-10-06 10:15:34 -07:00
Arne Welzel
1f5ab4878b logging/ascii: Support leftover log rotation in non-supervisor setups
We have a use case to rotate leftover log files in a non-supervisor
setup. There doesn't seem to be a strict requirement on supervisor
functionality. Allow enabling leftover log rotation through
LogAscii::enable_leftover_log_rotation and redef this for the
logger node in a supervisor setup individually.
2020-10-02 20:38:48 +02:00
Christian Kreibich
1bd658da8f Support for log filter policy hooks
This adds a "policy" hook into the logging framework's streams and
filters to replace the existing log filter predicates. The hook
signature is as follows:

    hook(rec: any, id: Log::ID, filter: Log::Filter);

The logging manager invokes hooks on each log record. Hooks can veto
log records via a break, and modify them if necessary. Log filters
inherit the stream-level hook, but can override or remove the hook as
needed.

The distribution's existing log streams now come with pre-defined
hooks that users can add handlers to. Their name is standardized as
"log_policy" by convention, with additional suffixes when a module
provides multiple streams. The following adds a handler to the Conn
module's default log policy hook:

    hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter)
            {
            if ( some_veto_reason(rec) )
                break;
            }

By default, this handler will get invoked for any log filter
associated with the Conn::LOG stream.

The existing predicates are deprecated for removal in 4.1 but continue
to work.
2020-09-30 12:32:45 -07:00
Robin Sommer
1f6fc4415f Tweak find-filtered-trace to not flag traces if they have non-TCP
traffic.

Closes #160.
2020-09-25 11:29:44 +00:00
Jon Siwek
cee10b5dc6 Fix a Sphinx warning about misformatted packet analyzer comment 2020-09-23 22:52:35 -07:00
Jan Grashoefer
7ede4f48bd Simplify packet analyzer config. 2020-09-23 11:13:29 -07:00
Jan Grashoefer
efa262a229 Make default packet analyzer definition explicit. 2020-09-23 11:13:29 -07:00
Jan Grashoefer
8f951574d7 Add explicit root analyzer for packet analysis. 2020-09-23 11:13:29 -07:00