Commit graph

6667 commits

Author SHA1 Message Date
Robin Sommer
d782c60f19 Unify type comparisions for records.
For records, same_type(r1, r2) would not check if the fields'
attributes match as well. That seems like an oversight, and some
callers of same_type() did indeed add that check on their end. This
commit moves the check into same_type() itself. That generally doesn't
seem make any differences except for a couple of places validating
code, which we update a bit. That in turn leans to slightly different
(better?) error messages for a couple of test cases.
2020-10-05 10:47:46 +00:00
Arne Welzel
1f5ab4878b logging/ascii: Support leftover log rotation in non-supervisor setups
We have a use case to rotate leftover log files in a non-supervisor
setup. There doesn't seem to be a strict requirement on supervisor
functionality. Allow enabling leftover log rotation through
LogAscii::enable_leftover_log_rotation and redef this for the
logger node in a supervisor setup individually.
2020-10-02 20:38:48 +02:00
Jon Siwek
a87281a1d9 Merge remote-tracking branch 'origin/topic/jsiwek/improve-checksum'
* origin/topic/jsiwek/improve-checksum:
  Switch one's complement checksum implementation
2020-10-02 10:26:43 -07:00
Christian Kreibich
1bd658da8f Support for log filter policy hooks
This adds a "policy" hook into the logging framework's streams and
filters to replace the existing log filter predicates. The hook
signature is as follows:

    hook(rec: any, id: Log::ID, filter: Log::Filter);

The logging manager invokes hooks on each log record. Hooks can veto
log records via a break, and modify them if necessary. Log filters
inherit the stream-level hook, but can override or remove the hook as
needed.

The distribution's existing log streams now come with pre-defined
hooks that users can add handlers to. Their name is standardized as
"log_policy" by convention, with additional suffixes when a module
provides multiple streams. The following adds a handler to the Conn
module's default log policy hook:

    hook Conn::log_policy(rec: Conn::Info, id: Log::ID, filter: Log::Filter)
            {
            if ( some_veto_reason(rec) )
                break;
            }

By default, this handler will get invoked for any log filter
associated with the Conn::LOG stream.

The existing predicates are deprecated for removal in 4.1 but continue
to work.
2020-09-30 12:32:45 -07:00
Jon Siwek
98ae204fc0 Fix multipart MIME leak of sub-part found after closing-boundary
After detecting a closing-boundary for a given multipart MIME entity, it
enters into an "end of data" state, however any subsequent boundary
delimiter could still cause the allocation of a sub-entity object that
is never released due to cleanup logic being bypassed upon finding the
"end of data" state already reached.

This change prevents allocation/processing of sub-entities after the
"end of data" state is reached (e.g. from detecting a multipart
closing-boundary).  This new behavior still aligns with RFC 2046
expectations:

"There appears to be room for additional information prior to the first
boundary delimiter line and following the final boundary delimiter line.
These areas should generally be left blank, and implementations must
ignore anything that appears before the first boundary delimiter line or
after the last one."

Credit to OSS-Fuzz for discovery
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26027
(Link to details becomes public 30 days after patch release)
2020-09-29 15:48:01 -07:00
Jon Siwek
c1492942bb Merge branch 'add-dce-rpc-payloads' of https://github.com/ynadji/zeek
- Changed the new stub events to correctly check for existence of
  their associated handler before generating an event

- Added a test case for the new stub event

* 'add-dce-rpc-payloads' of https://github.com/ynadji/zeek:
  Add stub payload to dce_rpc_request and dce_rpc_response
2020-09-25 14:39:30 -07:00
Yacin Nadji
fd58c724a5 Add stub payload to dce_rpc_request and dce_rpc_response 2020-09-25 11:40:06 -04:00
Tim Wojtulewicz
ee5212c4a9 Fix namespace warning in fuzzer code 2020-09-24 20:44:30 -07:00
Tim Wojtulewicz
8d856df305 Merge remote-tracking branch 'origin/topic/jsiwek/packet-analyzer-docs'
* origin/topic/jsiwek/packet-analyzer-docs:
  Fix a Sphinx warning about misformatted packet analyzer comment
  Add Zeekygen documentation support for packet analyzers
2020-09-24 16:03:58 -07:00
Tim Wojtulewicz
23bbe0ac38 Move packet_mgr to the zeek namespace 2020-09-24 09:56:55 -07:00
Jon Siwek
d070709c57 Switch one's complement checksum implementation
Borrows the `in_cksum` code from tcpdump, which borrowed from FreeBSD.
It handles unaligned data better and also unrolls the inner loop to
process 16 two-byte values at a time versus 2 one-byte values at a time
in the previous version.  Generally measured as ~1.5x faster in a
release build.  The new API should generally be more amenable to any
future optimization explorations since all relevant data blocks are
available within a single call rather than spread across multiple.
2020-09-24 09:41:15 -07:00
Jon Siwek
8feca7291b Merge remote-tracking branch 'origin/topic/jsiwek/gh-822-ubsan-ci'
* origin/topic/jsiwek/gh-822-ubsan-ci:
  Fix negative-value-left-shift undefined behavior in patricia trie
  Improve negation of ConstExpr
  Avoid signed integer overflow when combining SMB header PID bits
  Avoid unary negation of INT64_MIN in modp_litoa10
  Avoid double-to-int conversion overflows in modp_dtoa functions
  Fix divide-by-zero in Entropy analyzer
  Fix divide-by-zero in stats/profiling memory usage calculation
  Fix uninitialized field in POP3 fuzzer
  Add framework for running UndefinedBehaviorSanitizer in CI
2020-09-24 08:17:58 -07:00
Jon Siwek
0d58e97349 Add Zeekygen documentation support for packet analyzers 2020-09-23 22:50:52 -07:00
Tim Wojtulewicz
c21504deed Fix build on FreeBSD, which was missing full definition of sockaddr for ARP 2020-09-23 11:14:01 -07:00
Jan Grashoefer
1658b8dc84 Fix handling of session analysis.
This reverts commit 0e51be18ac59d7a176f18780f1f8df85561449b9.
2020-09-23 11:13:29 -07:00
Tim Wojtulewicz
6a0bd63756 Fix Sessions::DoNextPacket to ignore ARP packets and not report weirds for them 2020-09-23 11:13:29 -07:00
Jan Grashoefer
8d834a1d89 Packet analysis cleanup. 2020-09-23 11:13:29 -07:00
Tim Wojtulewicz
62562504d5 Minor cleanup 2020-09-23 11:13:29 -07:00
Jan Grashoefer
7ede4f48bd Simplify packet analyzer config. 2020-09-23 11:13:29 -07:00
Jan Grashoefer
efa262a229 Make default packet analyzer definition explicit. 2020-09-23 11:13:29 -07:00
Jan Grashoefer
8f951574d7 Add explicit root analyzer for packet analysis. 2020-09-23 11:13:29 -07:00
Jan Grashoefer
3f3f00030d Simplify MPLS analysis. 2020-09-23 11:13:29 -07:00
Jan Grashoefer
38337d799b Improve packet analysis data flow. 2020-09-23 11:13:29 -07:00
Jan Grashoefer
90eb97876f Improve packet analyzer API. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
d5ca0f9da5 Rename DefaultAnalyzer to IP. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
24babf096e Move ARP analysis into packet analyzer. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
0ec7516602 Small cleanup of packet analysis. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
0925b3bbec Remove encap_hdr_size (replaced by skip analyzer). 2020-09-23 11:13:28 -07:00
Jan Grashoefer
54961b5ea2 Allow to overwrite packet analysis mappings. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
fc814bd7e2 Add SkipAnalyzer.
This is WIP: The test case would require a new pcap or the possibility
to overwrite analyzer mappings. The CustomEncapsulationSkip method and
the corresponding options need to be removed.
2020-09-23 11:13:28 -07:00
Jan Grashoefer
6f6e5b4df0 Suggested code improvements for packet analysis. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
4aeab7402d Improve naming in packet analysis. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
6365fa6d80 Migrate all packet analyzers to new API. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
cbdaa53f85 Remove magic identifiers from Ethernet analyzer. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
462b1fe3a2 Bring back default packet analysis.
Default analyzers can be configured per packet analyzer by omitting the
identifier in the ConfigEntry.
2020-09-23 11:13:28 -07:00
Jan Grashoefer
d4ff5a236c Further simplified the packet analysis API.
This is still WIP and includes the following changes:
* Dispatchers are now part of analyzers (moving dispatching logic from
  the manager to the analyzers)
* All available analyzers are instantiated on start up
* Removal of configuration class
2020-09-23 11:13:28 -07:00
Jan Grashoefer
9feda100b9 Move dispatching into packet analyzers.
WIP that updates only the Ethernet analyzer.
2020-09-23 11:13:28 -07:00
Jan Grashoefer
96d0e11bb8 Move cur_pos from packet into packet manager loop. 2020-09-23 11:13:28 -07:00
Tim Wojtulewicz
c2500d03d6 Remove packet_analysis/Defines.h
- Replace uses of identifier_t with uint32_t
- Replace repeated usage of tuple type for Analysis results with type alias
2020-09-23 11:13:28 -07:00
Tim Wojtulewicz
b46e600775 Move VectorDispatcher to be the only dispatcher 2020-09-23 11:13:28 -07:00
Tim Wojtulewicz
d22481aef3 Remove Manager::Reset() method 2020-09-23 11:13:28 -07:00
Tim Wojtulewicz
bd6d3e0112 Remove enabled state from Components, ability to enable/disable from Manager 2020-09-23 11:13:28 -07:00
Tim Wojtulewicz
f39d6bb4c4 Use shared_ptr instead of raw pointers in packet_analysis for analyzers and dispatchers 2020-09-23 11:13:28 -07:00
Tim Wojtulewicz
07b7a3be40 Whitespace fixes from review 2020-09-23 11:13:28 -07:00
Tim Wojtulewicz
27fea2b218 Reorganize some pointer handling 2020-09-23 11:13:28 -07:00
Tim Wojtulewicz
94ec639bfb Fix ordering of debug logger strings to match the enum 2020-09-23 11:13:28 -07:00
Tim Wojtulewicz
08ceea8de1 Fixes for various btest issues
- Fix handling of truncated ethernet headers, fix core.truncation test output
- Update commit hashes for external private test repo
2020-09-23 11:13:28 -07:00
Tim Wojtulewicz
1c3ded7dd5 Merge ProtocolAnalyzerSet into Manager, remove AnalyzerSet base class 2020-09-23 11:13:28 -07:00
Jan Grashoefer
1e0e8e35af Minor fixes for packet analyzer renaming. 2020-09-23 11:13:28 -07:00
Jan Grashoefer
e53ec46c23 Renamed LL-Analyzers to Packet Analyzers. 2020-09-23 11:13:28 -07:00