Commit graph

2633 commits

Author SHA1 Message Date
Vlad Grigorescu
4d998742e2
Fix scheduling due to network_time being 0 in zeek_init
Co-authored-by: Jon Siwek <jsiwek@corelight.com>
2020-10-09 14:43:04 -05:00
Vlad Grigorescu
59620ed75d Add CaptureLoss::Too_Little_Traffic 2020-10-08 11:26:46 -05:00
Vlad Grigorescu
cb9d419fa4 Add CaptureLoss::initial_watch_interval for a quick read on cluster health after startup. 2020-10-08 09:41:29 -05:00
Vlad Grigorescu
bb3527c955 Documentation update, reference the threshold variable. [nomail] [skip ci] 2020-10-08 09:38:26 -05:00
Vlad Grigorescu
1b696490d0 Whitespace fixes only [nomail] [skip ci] 2020-10-08 09:36:18 -05:00
Robin Sommer
b0bf9f02c8 Merge remote-tracking branch 'origin/topic/christian/364-logfilter-hooks' into master
(Adding a NEWS entry.)

* origin/topic/christian/364-logfilter-hooks:
  Update testing/btest/scripts/base/frameworks/logging/hooks.zeek
  Btests for log filter policy hooks
  Btest baseline updates to reflect new logging policy hooks
  Migrate existing use of filter predicates to policy hooks
  Support for log filter policy hooks
2020-10-07 08:44:50 +00:00
Jon Siwek
506d961b27 Merge branch 'logging-ascii-enable-shadow-logs' of https://github.com/awelzel/zeek into master
- Improved documentation/comment for the new option

* 'logging-ascii-enable-shadow-logs' of https://github.com/awelzel/zeek:
  logging/ascii: Support leftover log rotation in non-supervisor setups
2020-10-06 10:15:34 -07:00
Arne Welzel
1f5ab4878b logging/ascii: Support leftover log rotation in non-supervisor setups
We have a use case to rotate leftover log files in a non-supervisor
setup. There doesn't seem to be a strict requirement on supervisor
functionality. Allow enabling leftover log rotation through
LogAscii::enable_leftover_log_rotation and redef this for the
logger node in a supervisor setup individually.
2020-10-02 20:38:48 +02:00
Christian Kreibich
7bd8351719 Migrate existing use of filter predicates to policy hooks 2020-09-30 12:45:20 -07:00
Christian Kreibich
1bd658da8f Support for log filter policy hooks
This adds a "policy" hook into the logging framework's streams and
filters to replace the existing log filter predicates. The hook
signature is as follows:

    hook(rec: any, id: Log::ID, filter: Log::Filter);

The logging manager invokes hooks on each log record. Hooks can veto
log records via a break, and modify them if necessary. Log filters
inherit the stream-level hook, but can override or remove the hook as
needed.

The distribution's existing log streams now come with pre-defined
hooks that users can add handlers to. Their name is standardized as
"log_policy" by convention, with additional suffixes when a module
provides multiple streams. The following adds a handler to the Conn
module's default log policy hook:

    hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter)
            {
            if ( some_veto_reason(rec) )
                break;
            }

By default, this handler will get invoked for any log filter
associated with the Conn::LOG stream.

The existing predicates are deprecated for removal in 4.1 but continue
to work.
2020-09-30 12:32:45 -07:00
Robin Sommer
1f6fc4415f Tweak find-filtered-trace to not flag traces if they have non-TCP
traffic.

Closes #160.
2020-09-25 11:29:44 +00:00
Jon Siwek
cee10b5dc6 Fix a Sphinx warning about misformatted packet analyzer comment 2020-09-23 22:52:35 -07:00
Jan Grashoefer
7ede4f48bd Simplify packet analyzer config. 2020-09-23 11:13:29 -07:00
Jan Grashoefer
efa262a229 Make default packet analyzer definition explicit. 2020-09-23 11:13:29 -07:00
Jan Grashoefer
8f951574d7 Add explicit root analyzer for packet analysis. 2020-09-23 11:13:29 -07:00
Jan Grashoefer
3f3f00030d Simplify MPLS analysis. 2020-09-23 11:13:29 -07:00
Jan Grashoefer
d5ca0f9da5 Rename DefaultAnalyzer to IP. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
0ec7516602 Small cleanup of packet analysis. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
0925b3bbec Remove encap_hdr_size (replaced by skip analyzer). 2020-09-23 11:13:28 -07:00
Jan Grashoefer
fc814bd7e2 Add SkipAnalyzer.
This is WIP: The test case would require a new pcap or the possibility
to overwrite analyzer mappings. The CustomEncapsulationSkip method and
the corresponding options need to be removed.
2020-09-23 11:13:28 -07:00
Jan Grashoefer
4aeab7402d Improve naming in packet analysis. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
6365fa6d80 Migrate all packet analyzers to new API. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
cbdaa53f85 Remove magic identifiers from Ethernet analyzer. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
462b1fe3a2 Bring back default packet analysis.
Default analyzers can be configured per packet analyzer by omitting the
identifier in the ConfigEntry.
2020-09-23 11:13:28 -07:00
Jan Grashoefer
9feda100b9 Move dispatching into packet analyzers.
WIP that updates only the Ethernet analyzer.
2020-09-23 11:13:28 -07:00
Jan Grashoefer
e53ec46c23 Renamed LL-Analyzers to Packet Analyzers. 2020-09-23 11:13:28 -07:00
Peter Oettig
b2e6c9ac9a Initial implementation of Lower-Level analyzers 2020-09-23 11:13:25 -07:00
Jon Siwek
f744d4c070 Improve fa_file/fa_metadata documentation
Related to https://github.com/zeek/zeek-docs/pull/38
2020-09-22 16:04:00 -07:00
Tim Wojtulewicz
6b93020c6f Merge remote-tracking branch 'origin/topic/jsiwek/gh-1164-fix-rst-conn-state'
* origin/topic/jsiwek/gh-1164-fix-rst-conn-state:
  GH-1164: Fix incorrect RSTOS0 conn_state determinations
2020-09-17 13:40:06 -07:00
Jon Siwek
a16bd47bf7 GH-1164: Fix incorrect RSTOS0 conn_state determinations
The RSTOS0 `conn_state` label is documented as "Originator sent a SYN
followed by a RST, never saw SYN-ACK from responder", but was previously
applied to cases where no originator SYN exists, like a single RST-only
packet.
2020-09-11 16:14:41 -07:00
Jon Siwek
05cf511f18 GH-1119: add base/protcols/conn/removal-hooks.zeek
This adds two new functions: `Conn::register_removal_hook()` and
`Conn::unregister_removal_hook()` for registering a hook function to be
called back during `connection_state_remove`.  The benefit of using hook
callback approach is better scalability: the overhead of unrelated
protocols having to dispatch no-op `connection_state_remove` handlers is
avoided.
2020-09-11 12:12:10 -07:00
Jon Siwek
49e2047da0 Avoid run-time cost of find-filtered-trace.zeek for live traffic 2020-09-10 12:53:47 -07:00
Jon Siwek
5f435c2644 Remove connection_successful and successful_connection_remove events
Related to https://github.com/zeek/zeek/issues/1119
2020-09-10 12:06:50 -07:00
Jon Siwek
260895fcb1 Merge remote-tracking branch 'origin/topic/robin/gh-343-http-sigs'
* origin/topic/robin/gh-343-http-sigs:
  Change HTTP's DPD signatures so that each side can trigger the analyzer on its own.
2020-09-08 17:24:29 -07:00
Jon Siwek
991bbc961d Merge remote-tracking branch 'origin/topic/robin/gh-623-sampling'
- Merge adjustments:
  - Minor whitespace/style tweaks
  - Fixed portability of the btest due to differences in `uniq -c`
    output format

* origin/topic/robin/gh-623-sampling:
  Extend weird sampling with option to track selected weirds globally.
2020-09-08 17:06:40 -07:00
Robin Sommer
0af57d12b2 Change HTTP's DPD signatures so that each side can trigger the analyzer on its own.
This is to avoid missing large sessions where a single side exceeds
the DPD buffer size. It comes with the trade-off that now the analyzer
can be triggered by anybody controlling one of the endpoints (instead
of both).

Test suite changes are minor, and nothing in "external".

Closes #343.
2020-09-08 07:33:36 +00:00
Robin Sommer
5fa94973da Extend weird sampling with option to track selected weirds globally.
The new set "sampling_global_list" lists weirds to rate-limite
globally instead of per connection/flow.

Closes #623.
2020-09-07 07:31:22 +00:00
Jon Siwek
2a8de33c63 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1122'
* origin/topic/jsiwek/gh-1122:
  GH-165: Fix global initializations that indirectly use builtin types
  Improve how primary/top-level BIFs get initialized
  GH-1122: Allow initializing globals with calls to subdir BIFs
  GH-1122: Improve error for global record initialization exceptions
2020-09-04 17:24:50 -07:00
Jon Siwek
c7408482b4 Change a script comment to use "code-block" directive 2020-09-03 15:03:12 -07:00
Jon Siwek
4a8640d702 Improve how primary/top-level BIFs get initialized 2020-08-27 13:54:43 -07:00
Jon Siwek
1bbae2368d GH-1122: Allow initializing globals with calls to subdir BIFs 2020-08-27 12:20:37 -07:00
Jon Siwek
99d9a3a48c Fix closing timestamp of rotated log files in supervised-cluster mode 2020-08-25 17:06:10 -07:00
FlyingWithJerome
fe6efc8524 better explanation to server cookie 2020-08-20 09:04:56 -04:00
FlyingWithJerome
d75a385f8f remove data+=option_len error in cookie and keepalive 2020-08-20 09:04:56 -04:00
FlyingWithJerome
74efbd0abc add EDNS cookie parsing 2020-08-20 09:04:56 -04:00
FlyingWithJerome
7dcf974daf add units (100ms) to comments 2020-08-20 09:04:56 -04:00
FlyingWithJerome
1d2b531062 initial adding EDNS TCP keepalive 2020-08-20 09:04:56 -04:00
Johanna Amann
b948180247 Fix minimize_info in ftp/main not returning a value.
Fixes GH-1120
2020-08-12 19:53:53 +00:00
Johanna Amann
db3d8faf6e Merge branch 'action-drop' of https://github.com/LBL-gov/zeek
* 'action-drop' of https://github.com/LBL-gov/zeek:
  Moved verb ACTION_DROP from policy/frameworks/netcontrol/catch-and-release.zeek to base/frameworks/notice/main.zeek.
2020-08-12 19:49:40 +00:00
Johanna Amann
640e9c50e6 Merge branch 'netcontrol-acld-verbs' of https://github.com/LBL-gov/zeek
* 'netcontrol-acld-verbs' of https://github.com/LBL-gov/zeek:
  Added new acld verbs of filter and nofilter.
2020-08-12 19:43:14 +00:00