This could lead to a noticeable (single-percent) performance
improvement.
Most of the functionality for this is in the packet analyzers that now
cache ignore_chesksums_nets.
Based on a patch by Arne Welzel (Corelight).
* origin/topic/vern/ZAM-prep: (45 commits)
whoops overlooked the need to canonicalize filenames
another set of tweaks per review comments
addressed a number of code review comments
baseline updates for merge
support "any" coercions for "-O gen-C++"
better descriptions for named record constructors
test suite baseline updates for "-a opt" optimize-AST alternative
test suite baseline updates for "-a xform" alternative / AST transformation
error propagation fix for AST reduction
updates to "-a inline" test suite alternative baseline
updates for the main test suite baseline
updates to test suite tests for compatibility with upcoming ZAM functionality
"-O compile-all" option to specify compilation of inlined functions
compile inlined functions if they're also used indirectly
provide ZAM-generated code with low-level access to record fields
fix for cloning records with fields of type "any"
direct access for ZAM to VectorVal internal vector
ZVal constructors, accessors & methods in support of ZAM
switch ZVal representation of types from Type objects to TypeVal's
revised error-reporting interface for ZVal's, to accommodate ZAM inner loop
...
The change in 44f558df7b that made analyzer_list
a std::vector instead of a std::list doesn't take into account that in some
cases an analyzer may chain back into itself, such as with UDP-in-UDP tunnels.
In these cases, the second call to ForwardPacket may cause iterator
invalidation, leading to a crash.
Use of --parse-only would previously exit before --usage-issues had a
chance to analyze scripts and report any discovered issues and it can be
useful to combine both flags for checking scripts for mistakes without
actually executing any code.
This also improves the behavior of --parse-only in combination with
reporting problems in signature files (previously, it exited before
even reading them).
Merge adjustments:
- Removed some stale str_split() references from docs
- Renumbered TypeTag enum comments
- Simplified test-case for @unload (don't need .bro files anymore)
* origin/topic/timw/deprecation-cleanup:
Doc updates
Fix language.init-in-anon-function btest due to changes to log filter predicates
Remove deprecated log filter predicates for 4.1
Remove Plugin::HookCallFunction and fix tests related to it
Remove support for .bro script extension and BRO_ environment variables
Remove deprecated ICMP events
Remove some deprected methods/events from bif files
Remove TYPE_COUNTER
Remove all of the random single-file deprecations
Remove all fully-deprecated files
Update bifcl submodule to remove deprecations from generated code
Particularly for ICMP connections, a new timer got added every time a
`connection` record was updated even if there was still a pending timer
for that connection.
It's implemented with a new set[subnet] option named ignore_checksums_nets.
If you populate this set with subnets, any packet with a src address within
that set of subnets will not have it's checksum validated.
This is WIP: The test case would require a new pcap or the possibility
to overwrite analyzer mappings. The CustomEncapsulationSkip method and
the corresponding options need to be removed.
- Merge adjustments:
- Minor whitespace/style tweaks
- Fixed portability of the btest due to differences in `uniq -c`
output format
* origin/topic/robin/gh-623-sampling:
Extend weird sampling with option to track selected weirds globally.
- Use `-b` most everywhere, it will save time.
- Start some intel tests upon the input file being fully read instead of
at an arbitrary time.
- Improve termination condition for some sumstats/cluster tests.
- Filter uninteresting output from some supervisor tests.
- Test for `notice_policy.log` is no longer needed.
Previously, a single `icmp_conn` record was built per ICMP "connection"
and re-used for all events generated from it. This may have been a
historical attempt at performance optimization, but:
* By default, Zeek does not load any scripts that handle ICMP events.
* The one script Zeek ships with that does handle ICMP events,
"detect-traceroute", is already noted as being disabled due to
potential performance problems of doing that kind of analysis.
* Re-use of the original `icmp_conn` record tends to misreport
TTL and length values since they come from original packet instead
of the current one.
* Even if we chose to still re-use `icmp_conn` records and just fill
in a new TTL and length value each packet, a user script could have
stored a reference to the record and not be expecting those values
to be changed out from underneath them.
Now, a new `icmp_info` record is created/populated in all ICMP events
and should be used instead of `icmp_conn`. It also removes the
orig_h/resp_h fields as those are redundant with what's already
available in the connection record.
Duplicate script `@load` directives are now detected by comparing
against canonical paths formed by realpath(). This fixes the previous,
unexpected behavior of treating scripts that hardlink to same
inode as duplicates: such links will now be loaded as distinct scripts
since their canonical path differs.
* Variables of `string` type can now be set to an empty string
* Trying to set a variable with non-`string` type to an empty value
now emits an error instead of silently doing nothing
* Providing an invalid identifier now emits an "unknown identifier"
error instead of silently doing nothing
This introduces a new sampling state-map for expired connections to fix
segfaults that previously occured when passing in a `connection` record
to `Reporter::conn_weird()` for which the internal `Connection` object
had already been expired and deleted. This also introduces a new event
called `expired_conn_weird`, which is similar to `conn_weird`, except
the full `connection` record is no longer available, just the `conn_id`
and UID string.
- Updated the logic significantly: still filters out ICMP from being
considered an active service (like before) and adds a new
"Known::service_udp_requires_response" option (defaults to true) for
whether to require UDP server response before being considered an
active service.
* 'topic/dopheide/known-services' of https://github.com/dopheide-esnet/zeek:
Log services with unknown protocols
