New connections already do
conn_val->Assign(6, val_mgr->EmptyString());
This second assignment was effectively doing
conn_val->Assign(6, "")
for all new connections, causing a new empty ZeekString to be allocated.
On a pcap containing 100% syn packets this gives a noticeable perf improvement.
Benchmark #1: zeek.orig -r /data/pcaps/scan.pcap
Time (mean ± σ): 47.082 s ± 0.547 s [User: 57.555 s, System: 9.114 s]
Range (min … max): 46.516 s … 47.834 s 5 runs
Benchmark #2: zeek -r /data/pcaps/scan.pcap
Time (mean ± σ): 45.260 s ± 0.378 s [User: 55.438 s, System: 8.537 s]
Range (min … max): 44.783 s … 45.789 s 5 runs
Summary
'zeek -r /data/pcaps/scan.pcap' ran
1.04 ± 0.01 times faster than 'zeek.orig -r /data/pcaps/scan.pcap'
Using `brew update-reset` causes homebrew to reset to homebrew's HEAD commit, which may
be buggy and broken. It appears whatever Cirrus was doing previously on their Catalina
VM is no longer a problem, and so update-reset isn't required anymore. Switch to
`brew update` to make sure we still get newer versions of the packages, but is
actually a versioned release of homebrew.
* origin/topic/christian/rehash-comphash:
Add btests for new functionality
Remove unused HashKey constructor and reorder for consistency
Refactor CompHash class to use new HashKey buffering features
Add debug string and ODesc support to HashKey class
Refactor HashKey class to support read/write operations
Add unit tests for memory helpers
Add memory sizing/alignment helpers to util.cc/h
Ensure table/set HashKey buffer reservation and writes happen in same order
- Expand language.set to cover sets of sets
- Expand language.table to cover tables indexed with tables
- Add language.table-nested-set-ordering to capture the reproducer from GHI-1753
This preserves the previous hash key buffer layout (so the testsuite still
passes) and overall approach but gets rid of the codepath for writing singleton
serializations. This code path required a fourth switch block over all types
(besides reads, writes, and size computation) and was inconsistent with the one
for writing non-atomic types.
This allows tracing of hash key buffer reservations, reads, and writes via a new
debug stream, and supports printing a summary of a HashKey object via
Describe(). The latter comes in handy e.g. in TableVal::Describe() (where
including the hash key is now available but commented out).
This preserves the optimization of storing values directly in the key_u member
union when feasible, and using a variable size buffer otherwise. It also adds
bounds-checking for that buffer, moves size arguments to size_t, decouples
construction from hash computation, emulates the tagging feature found in
SerializationFormat to assist troubleshooting, and switches feasible
reinterpret_casts to static_casts.
This functionality previously lived in the CompHash class, with one difference:
this removes a discrepancy between the offset aligner and the memory pointer
aligner/padder. The size aligner used to align the provided offset and then add an
additional alignment size (for example, 1 aligned to 4 wouldn't yield 4 but 8).
Like the memory aligners it now only rounds up as needed.
Includes unit tests.
This takes the existing sorting for table index hashkeys we had in place during
hash key writes and applies it also during buffer size reservation. It changes
the approach slightly: the underlying map now points to the TableVal entry index
vals directly, rather than to the numerical index into an additional list that
gets built up to store those indexes. Doing so removes the need for that list.
Changes during merge:
- Add dedicated test (w/ trace "client_timestamp_enabled.pcapng" from Cloudshark)
- Change types from signed to unsigned.
- Add cast for bit-shifting operand.
- clang-format run
* origin/topic/timw/386-clang-format:
Minor fixes for build problems after reformatting
Reformat the world
Fixes for includes ahead of reformatting
GH-386: Add clang-format config and scripts
* origin/topic/vern/zam-coverity:
still more nit-squashing
avoiding using back() for an empty std::string
additional tweaks to address some compiler warnings
tweaks to address some compiler warnings
tweaks to address concerns flagged by Coverity
* origin/topic/seth/tsv-logs-utf8-by-default:
Fix mis-usage of string::append that leads to an overflow
Use json_escape_utf8 for all utf8 data in ODesc
Switch the TSV Zeek logs to be UTF8 by default.
* origin/topic/vern/ZAM-remainder: (37 commits)
fix race condition in btest output ordering
whoops, forgot to canonicalize filenames in new btest
extend btest to include a coercion overflow
fixed a typo in a comment
fixes for vector coercion overflows, typing, and holes
factoring out logic to check for overflows during coercions
test case for vector coercions, including holes
low-level cleanups found by code review
additional conversions of size() to empty() checks that were missed previously
indentation nit
flag loop that has slightly subtle logic
use ## to start major sections
a number of low-level tweaks from code review
use std::find_if rather than explicit loop
switch simple loops that don't need indices to being iterator-based
use container empty() rather than size() where appropriate
Baseline variants for "-a zam"
new "-a ZAM" testing baseline alternative
updates for usage issues: support for -uu, maybe/definitely distinctions
enable reducer to track folding to enable constant propagation
...
