* origin/topic/jsiwek/32bit-compat:
Improve formatting of doubles that are close to integers
Improve HTTP version number comparisons
Add a 32-bit task to Cirrus CI config
Replace va_list fmt() overload with vfmt()
Format tables indexed by patterns consistently across 32-bit/64-bit
Format interval values consistently across 32-bit/64-bit platforms
Minor formatting/style changes in merge.
* 'refactor_obj' of https://github.com/MaxKellermann/zeek:
Val: use C++ initializers
Val: add BroValUnion constructors
Val: reduce duplicate code by using delegating constructors
Val: remove unused default constructors and `friend` declarations
Val: remove the unnecessary BroValUnion typedef
Type: remove unnecessary enum typedefs
Type: use C++ initializers
Type: move code from BroType::BroType() to constexpr functions
Type: remove useless BroType destructor
Obj: disallow copying BroObj
Obj: use C++ initializers
Obj: make `no_location` constexpr
Minor whitespace fixes during merge.
* 'smart_ptr' of https://github.com/MaxKellermann/zeek:
OpaqueVal: remove misplaced `virtual` keywords
CompHash: use class IntrusivePtr for the `type` field
IntrusivePtr: replace the "add_ref" parameter with tag structs
IntrusivePtr: remove reset(), nobody uses it
IntrusivePtr: remove ordering operators
IntrusivePtr: rename detach() to release()
IntrusivePtr: move nullptr initializer to field declaration
Only 1% build time speedup, but still, it declutters the headers a bit.
Before this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
After this patch:
2537.19user 142.94system 2:26.90elapsed 1824%CPU (0avgtext+0avgdata 1434268maxresident)k
16240inputs+8887152outputs (1931major+48728888minor)pagefaults 0swaps
The Zeek code base has very inconsistent #includes. Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed. Another side effect was a lot of header
bloat which slows down the build.
First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.
After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations. In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.
This patch speeds up the build by 19%, because each compilation unit
gets smaller. Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):
Before this patch:
3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps
After this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
We now have an &broker_store attribute which automatically sends
inserts/deletes into a set/table to broker.
This might work - I actually did not test if the data ends up in the
broker store in the end. A limitation is that the table/set currently
only can have a one-element type since Broker doesn't support the list
type.
Tweaks:
- Small change to the logic for removing quotes around strings.
- Updated NEWS & COPYING.3rdparty
- Use of intrusive_ptr for stack-allocated StringVals
- Little bit of refactoring (I would love to merge the two BuildJSON() functions, too, but that's a larger task)
* origin/topic/timw/595-rapidjson:
Use the list of files from clang-tidy when searching for unit tests
Optimize json_escape_utf8 a bit by removing repeated calls to string methods
Expand unit test for json_escape_utf8 to include all of the strings from the ascii-json-utf8 btest
GHI-595: Convert from nlohmann/json to rapidjson for performance reasons
Convert type-checking macros to actual functions
* origin/topic/jsiwek/asan: (28 commits)
Increase timeout for a btest
Switch CI scripts to use ASAN/LSAN instead of gperftools
Remove redundant memory leak btests
Fix reference counting issues related to lambdas/closures
Disable LeakSanitizer for btests that have known leaks
Rewrite the btest for when-statement timeouts
Fix memory leak when a logging plugin hook prevents a write
Fix memory leaks in various input framework error-handling cases
Fix memory leak in Reporter::get_weird_sampling_whitelist() BIF
Fix reference counting of Log::Filter "config" field
Fix memory leak in system_env() BIF
Clean up triggers awaiting global state modification at shutdown
Fix memory leak in initializing log writers with no local backend
Fix packet filter memory leaks
Skip sending thread heartbeat if it alread asked to be finished
Fix memory leak of sqlite input reader prepared statement
Prevent duplicate "finish" threading messages
Fix memory leak when table-based input stream overwrites old entries
Fix scripting error in a DHCP btest
Fix memory leaks in Kerberos ticket decryption
...
For example, circular references between a lambda function the frame
it's stored within and/or its closure could cause memory leaks.
This also fixes other various reference-count ownership issues that
could lead to memory errors.
There may still be some potential/undiscovered issues because the "outer
ID" finding logic doesn't look quite right as the AST traversal descends
within nested lambdas and considers their locals as "outer", but
possibly the other logic for locating values in closures or cloning
closures just works around that behavior.
safe_snprintf and safe_vsnprintf just exist to ensure that the resulting strings are always null-terminated. The documentation for snprintf/vsnprintf states that the output of those methods are always null-terminated, thus making the safe versions obsolete.
ODesc allocated with malloc() and BroString deallocated with delete[],
but really the intermediate BroString wasn't even needed when copying
into std::string.
Manual memory management via Ref/Unref is verbose and prone to error. An
intrusive smart pointer automates the reference counting, makes code
more robust (in particular w.r.t. to exceptions) and reduces boilerplate
code. A big benefit of the intrusive smart pointers for Zeek is that
they can co-exist with the manual memory management. Rather than having
to port the entire code base at once, we can migrate components
one-by-one. In this first step, we add the new template
`IntrusivePtr<T>` and start using it in the Broker Manager. This makes
the previous `unref_guard` obsolete.
OS X complains about std::pair not having a constexpr constructor. Which
seems to be the case for C++11. Hence, let's make it static const for
the moment.
This commit should be reverted when we move to C++17.
* origin/topic/timw/595-json-perf:
Update COPYING.3rdparty
Use json::emplace to avoid some extra calls to operator[]
Use tessil/unordered-map instead of nlohmann/fifo-map to mitigate performance issues when logging JSON
No need to create a record introspection table each time when all
the required information can be obtained directly in the RecordVal
and RecordType objects. Besides the additional overhead, using such
a table will re-order the fields arbitrarily instead of using the
order in which they're defined.
