Commit graph

2243 commits

Author SHA1 Message Date
Jan Grashoefer
8f951574d7 Add explicit root analyzer for packet analysis. 2020-09-23 11:13:29 -07:00
Jan Grashoefer
3f3f00030d Simplify MPLS analysis. 2020-09-23 11:13:29 -07:00
Jan Grashoefer
d5ca0f9da5 Rename DefaultAnalyzer to IP. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
0ec7516602 Small cleanup of packet analysis. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
0925b3bbec Remove encap_hdr_size (replaced by skip analyzer). 2020-09-23 11:13:28 -07:00
Jan Grashoefer
fc814bd7e2 Add SkipAnalyzer.
This is WIP: The test case would require a new pcap or the possibility
to overwrite analyzer mappings. The CustomEncapsulationSkip method and
the corresponding options need to be removed.
2020-09-23 11:13:28 -07:00
Jan Grashoefer
4aeab7402d Improve naming in packet analysis. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
6365fa6d80 Migrate all packet analyzers to new API. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
cbdaa53f85 Remove magic identifiers from Ethernet analyzer. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
462b1fe3a2 Bring back default packet analysis.
Default analyzers can be configured per packet analyzer by omitting the
identifier in the ConfigEntry.
2020-09-23 11:13:28 -07:00
Jan Grashoefer
9feda100b9 Move dispatching into packet analyzers.
WIP that updates only the Ethernet analyzer.
2020-09-23 11:13:28 -07:00
Jan Grashoefer
e53ec46c23 Renamed LL-Analyzers to Packet Analyzers. 2020-09-23 11:13:28 -07:00
Peter Oettig
b2e6c9ac9a Initial implementation of Lower-Level analyzers 2020-09-23 11:13:25 -07:00
Jon Siwek
f744d4c070 Improve fa_file/fa_metadata documentation
Related to https://github.com/zeek/zeek-docs/pull/38
2020-09-22 16:04:00 -07:00
Tim Wojtulewicz
6b93020c6f Merge remote-tracking branch 'origin/topic/jsiwek/gh-1164-fix-rst-conn-state'
* origin/topic/jsiwek/gh-1164-fix-rst-conn-state:
  GH-1164: Fix incorrect RSTOS0 conn_state determinations
2020-09-17 13:40:06 -07:00
Jon Siwek
a16bd47bf7 GH-1164: Fix incorrect RSTOS0 conn_state determinations
The RSTOS0 `conn_state` label is documented as "Originator sent a SYN
followed by a RST, never saw SYN-ACK from responder", but was previously
applied to cases where no originator SYN exists, like a single RST-only
packet.
2020-09-11 16:14:41 -07:00
Jon Siwek
05cf511f18 GH-1119: add base/protcols/conn/removal-hooks.zeek
This adds two new functions: `Conn::register_removal_hook()` and
`Conn::unregister_removal_hook()` for registering a hook function to be
called back during `connection_state_remove`.  The benefit of using hook
callback approach is better scalability: the overhead of unrelated
protocols having to dispatch no-op `connection_state_remove` handlers is
avoided.
2020-09-11 12:12:10 -07:00
Jon Siwek
49e2047da0 Avoid run-time cost of find-filtered-trace.zeek for live traffic 2020-09-10 12:53:47 -07:00
Jon Siwek
5f435c2644 Remove connection_successful and successful_connection_remove events
Related to https://github.com/zeek/zeek/issues/1119
2020-09-10 12:06:50 -07:00
Jon Siwek
260895fcb1 Merge remote-tracking branch 'origin/topic/robin/gh-343-http-sigs'
* origin/topic/robin/gh-343-http-sigs:
  Change HTTP's DPD signatures so that each side can trigger the analyzer on its own.
2020-09-08 17:24:29 -07:00
Jon Siwek
991bbc961d Merge remote-tracking branch 'origin/topic/robin/gh-623-sampling'
- Merge adjustments:
  - Minor whitespace/style tweaks
  - Fixed portability of the btest due to differences in `uniq -c`
    output format

* origin/topic/robin/gh-623-sampling:
  Extend weird sampling with option to track selected weirds globally.
2020-09-08 17:06:40 -07:00
Robin Sommer
0af57d12b2 Change HTTP's DPD signatures so that each side can trigger the analyzer on its own.
This is to avoid missing large sessions where a single side exceeds
the DPD buffer size. It comes with the trade-off that now the analyzer
can be triggered by anybody controlling one of the endpoints (instead
of both).

Test suite changes are minor, and nothing in "external".

Closes #343.
2020-09-08 07:33:36 +00:00
Robin Sommer
5fa94973da Extend weird sampling with option to track selected weirds globally.
The new set "sampling_global_list" lists weirds to rate-limite
globally instead of per connection/flow.

Closes #623.
2020-09-07 07:31:22 +00:00
Jon Siwek
2a8de33c63 Merge remote-tracking branch 'origin/topic/jsiwek/gh-1122'
* origin/topic/jsiwek/gh-1122:
  GH-165: Fix global initializations that indirectly use builtin types
  Improve how primary/top-level BIFs get initialized
  GH-1122: Allow initializing globals with calls to subdir BIFs
  GH-1122: Improve error for global record initialization exceptions
2020-09-04 17:24:50 -07:00
Jon Siwek
c7408482b4 Change a script comment to use "code-block" directive 2020-09-03 15:03:12 -07:00
Jon Siwek
4a8640d702 Improve how primary/top-level BIFs get initialized 2020-08-27 13:54:43 -07:00
Jon Siwek
1bbae2368d GH-1122: Allow initializing globals with calls to subdir BIFs 2020-08-27 12:20:37 -07:00
Jon Siwek
99d9a3a48c Fix closing timestamp of rotated log files in supervised-cluster mode 2020-08-25 17:06:10 -07:00
FlyingWithJerome
fe6efc8524 better explanation to server cookie 2020-08-20 09:04:56 -04:00
FlyingWithJerome
d75a385f8f remove data+=option_len error in cookie and keepalive 2020-08-20 09:04:56 -04:00
FlyingWithJerome
74efbd0abc add EDNS cookie parsing 2020-08-20 09:04:56 -04:00
FlyingWithJerome
7dcf974daf add units (100ms) to comments 2020-08-20 09:04:56 -04:00
FlyingWithJerome
1d2b531062 initial adding EDNS TCP keepalive 2020-08-20 09:04:56 -04:00
Johanna Amann
b948180247 Fix minimize_info in ftp/main not returning a value.
Fixes GH-1120
2020-08-12 19:53:53 +00:00
Johanna Amann
db3d8faf6e Merge branch 'action-drop' of https://github.com/LBL-gov/zeek
* 'action-drop' of https://github.com/LBL-gov/zeek:
  Moved verb ACTION_DROP from policy/frameworks/netcontrol/catch-and-release.zeek to base/frameworks/notice/main.zeek.
2020-08-12 19:49:40 +00:00
Johanna Amann
640e9c50e6 Merge branch 'netcontrol-acld-verbs' of https://github.com/LBL-gov/zeek
* 'netcontrol-acld-verbs' of https://github.com/LBL-gov/zeek:
  Added new acld verbs of filter and nofilter.
2020-08-12 19:43:14 +00:00
Aashish Sharma
496f6d4935 Moved verb ACTION_DROP from policy/frameworks/netcontrol/catch-and-release.zeek to base/frameworks/notice/main.zeek.
ACTION_DROP is not only part of catch-n-release subsystem.

Also, historically ACTION_DROP has been bundled with ACTION_LOG, ACTION_ALARM, ACTION_EMAIL... and its helpful that this verb remains in base/frameworks/notice/main.zeek
2020-08-12 10:13:27 -07:00
Aashish Sharma
2496088983 Added new acld verbs of filter and nofilter.
These are to support actions taken on corsa and keep them distinct from ACLD drops/restore or BGP nullzero/nonullzero.

This does not negatively impacts anything - only adds new verbs for handling zeek-netcontrol actions.
2020-08-12 09:24:28 -07:00
Jon Siwek
fef4fda2f7 Fix DNS script deleting a table element while iterating
Doesn't typically cause any problems since the loop breaks after
deleting, except there's now an assert in debug builds catching
potential problems like this.
2020-08-11 11:26:21 -07:00
Jon Siwek
170752fa99 Fix extract_first_email_addr() to really return the first email
The use of find_all() in extract_email_addrs_vec() extracted occurrences
to an intermediate set and thus lost any sense of ordering.

This changes extract_email_addrs_vec() to use find_all_ordered() and
return all occurrences of email addresses found in the argument,
included duplicates, with their order of occurrence preserved.
2020-08-11 11:26:21 -07:00
Tim Wojtulewicz
b8287a3375 Fix issue with sumstats script and fix baselines that were crashing previously 2020-08-09 21:13:12 -07:00
Jon Siwek
613b27eec7 GH-1080: Rename conflicting NetControl::DROP enum definitions
``NetControl::DROP`` had 3 conflicting definitions that could potentially
be used incorrectly without any warnings or type-checking errors.
Such enum redefinition conflicts are now caught and treated as errors,
so the ``NetControl::DROP`` enums had to be renamed:

* The use as enum of type ``Log::ID`` is renamed to ``NetControl::DROP_LOG``

* The use as enum of type ``NetControl::CatchReleaseInfo`` is renamed to
  ``NetControl::DROP_REQUESTED``

* The use as enum of type ``NetControl::RuleType`` is unchanged and still
  named ``NetControl::DROP``
2020-07-31 16:11:35 -07:00
Tim Wojtulewicz
3eab3e600e Merge remote-tracking branch 'ronwellman/parse_ecs'
* ronwellman/parse_ecs:
  Avoid typecast to int& in EDNS parsing.
  Validate option_len in EDNS packets.
  Adjust for zeek namespace.
  Implement EDNS Client Subnet Option
2020-07-24 09:26:06 -07:00
Robin Sommer
c3f4971eb2 Merge remote-tracking branch 'origin/topic/johanna/table-changes'
* origin/topic/johanna/table-changes: (26 commits)
  TableSync: try to make test more robust & add debug output
  Increase timeouts to see if FreeBSD will be happy with this.
  Try to make FreeBSD test happy with larger timeout.
  TableSync: refactor common functionality into function
  TableSync: don't raise &on_change, smaller fixes
  TableSync: rename auto_store -> table_store
  SyncTables: address feedback part 1 - naming (broker and zeek)
  BrokerStore <-> Zeek Tables: cleanup and bug workaround
  Zeek Table<->Brokerstore: cleanup, documentation, small fixes
  BrokerStore<->Zeek table: adopt to recent Zeek API changes
  BrokerStore<->Zeek Tables Fix a few small test failures.
  BrokerStore<->Zeek tables: allow setting storage location & tests
  BrokerStore<->Zeek tables: &backend works for in-memory stores.
  BrokerStore<->Zeek table - introdude &backend attribute
  BrokerStore<->Zeek tables: test for clones synchronizing to a master
  BrokerStore<->Zeek tables: load persistent tables on startup.
  Brokerstore<->Tables: attribute conflicts
  Zeek/Brokerstore updates: expiration
  Zeek/Brokerstore updates: add test that includes updates from clones
  Zeek/Brokerstore updates: first working end-to-end test
  ...
2020-07-21 15:39:39 +00:00
Johanna Amann
66f586c02c Update Mozilla CA & Google CT lists 2020-07-17 23:26:37 +00:00
Johanna Amann
930a5c8ebd TableSync: rename auto_store -> table_store 2020-07-17 11:40:59 -07:00
Johanna Amann
6d2aa84952 SyncTables: address feedback part 1 - naming (broker and zeek)
This commit fixes capitalization issues.
2020-07-17 10:56:28 -07:00
Jon Siwek
c84a51ac09 GH-837: emit Reporter errors for Broker errors
Instead of only writing them in broker.log, which may be easy to
overlook.
2020-07-16 18:07:00 -07:00
Johanna Amann
7c37226eaa Merge remote-tracking branch 'origin/master' into topic/johanna/table-changes 2020-07-13 17:11:55 -07:00
Johanna Amann
2b2a40f49c Zeek Table<->Brokerstore: cleanup, documentation, small fixes
This commit adds script/c++ documentation and fixes a few loose ends.
It also adds tests for corner cases and massively improves error
messages.

This also actually introduces type-compatibility checking and introduces
a new attribute that lets a user override this if they really know what
they are doing. I am not quite sure if we should really let that stay in
- but it can be very convenient to have this functionality.

One test is continuing to fail - the expiry test is very flaky. This is,
I think, caused by delays of the broker store forwarding. I am unsure if
we can actually do anything about that.
2020-07-10 16:58:34 -07:00