Logs that got sent sparsely or burstily would get buffered for long
periods of time since the logic to flush them only does so on the next
log write. In the worst case, a subsequent log write could never happen
and cause a log entry to be indefinitely buffered.
This fix introduces a recurring event/timer to simply flush all pending
logs at frequency of Broker::log_batch_interval.
This change standardizes threading formatter error handling and moves
the remaining error calls to be warnings instead.
This is in line with already existing code - in most cases warnings were
raised, only a few cases raised errors. These cases do not differ
significantly from other cases in which warnings are raised.
This also fixes GH-692, in which misformatted lines prevent future file
parsing.
This commit also moves the FailWarn method that is used by both the
config and the ascii reader up to the ReaderBackend. Furthermore it
makes the Warning method of ReaderBackend respect the warning
suppression that is introduced by the FailWarn method.
The Zeek code base has very inconsistent #includes. Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed. Another side effect was a lot of header
bloat which slows down the build.
First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.
After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations. In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.
This patch speeds up the build by 19%, because each compilation unit
gets smaller. Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):
Before this patch:
3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps
After this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
During merge I split the test for bro_init/bro_done/bro_script_loaded
event errors into individual tests since the other testing of the zeek
versions of those events seemed fine to otherwise keep.
* origin/topic/robin/631-deprecation-v2:
Update NEWS for naming changes.
Small cleanup and updating submodules.
Remove test for legacy plugin.
Remove legancy symlinks in aux/.
Add warnings when loading scripts ending in ".bro", or using legacy environment variables.
Fix missing rename.
No longer symlink local.zeek to local.bro.
Update notice user agent.
Remove old_comm_usage_is_ok.
Remove bro-config.h.in and bro-path-dev.in.
Change Bro wrapper script to now abort when old executable names are still used.
Remove APIs that were explicitly deprecated to be removed in 3.1.
* origin/topic/jsiwek/supervisor: (44 commits)
Add note that Supervisor script APIs are unstable until 4.0
Move command-line arg parsing functions to Options.{h,cc}
Add btests for supervisor stem/leaf process revival
Move supervisor control events into SupervisorControl namespace
Fix supervisor "destroy" call on nodes not currently alive
Move supervisor source files into supervisor/
Address supervisor code re-factoring feedback from Robin
Convert supervisor internals to rapidjson
Add Supervisor documentation
Add supervisor btests
Improve logging of supervised node errors
Fix supervised node inheritence of command-line script paths
Improve normalize_path() util function
Use a timer to check for death of supervised node's parent
Improve supervisor checks for parent process termination
Improve handling of premature supervisor stem exit
Improve supervisor signal handler safety
Remove unused supervisor config options
Cleanup minor Supervisor TODOs
Improve supervisor debug logging
...
I'd expect the portnum component to remain uninitialized in case the
given URI specifies no port number -- however, decompose_uri() raises an
error instead. Will address this in a subsequent commit.
Tweaks:
- Small change to the logic for removing quotes around strings.
- Updated NEWS & COPYING.3rdparty
- Use of intrusive_ptr for stack-allocated StringVals
- Little bit of refactoring (I would love to merge the two BuildJSON() functions, too, but that's a larger task)
* origin/topic/timw/595-rapidjson:
Use the list of files from clang-tidy when searching for unit tests
Optimize json_escape_utf8 a bit by removing repeated calls to string methods
Expand unit test for json_escape_utf8 to include all of the strings from the ascii-json-utf8 btest
GHI-595: Convert from nlohmann/json to rapidjson for performance reasons
Convert type-checking macros to actual functions
Fixes in merge:
- Memory leak in HashKey ctor
- Minor whitespace/style changes
* origin/topic/dev/patterns-in-sets:
Enable Patterns as Table index in non singleton cases
Added support to create a Hashkey for PatternVals using their Pattern Texts
* origin/topic/jsiwek/asan: (28 commits)
Increase timeout for a btest
Switch CI scripts to use ASAN/LSAN instead of gperftools
Remove redundant memory leak btests
Fix reference counting issues related to lambdas/closures
Disable LeakSanitizer for btests that have known leaks
Rewrite the btest for when-statement timeouts
Fix memory leak when a logging plugin hook prevents a write
Fix memory leaks in various input framework error-handling cases
Fix memory leak in Reporter::get_weird_sampling_whitelist() BIF
Fix reference counting of Log::Filter "config" field
Fix memory leak in system_env() BIF
Clean up triggers awaiting global state modification at shutdown
Fix memory leak in initializing log writers with no local backend
Fix packet filter memory leaks
Skip sending thread heartbeat if it alread asked to be finished
Fix memory leak of sqlite input reader prepared statement
Prevent duplicate "finish" threading messages
Fix memory leak when table-based input stream overwrites old entries
Fix scripting error in a DHCP btest
Fix memory leaks in Kerberos ticket decryption
...
Or otherwise convert into a regular btest if it didn't already seem to
be covered.
There's no need for a separate memory leak test group since compiling
with LeakSanitizer now covers leak checking for the full btest suite.
E.g. ones that throw interpreter exceptions, as those are currently
known to potentially cause leaks. Fixing the underlying leaks involves
the larger task of more IntrusivePtr usage.
Reference cycles may also cause leaks.
To avoid a memory leak in DNS lookups that's hard to work around and
does not otherwise effect typical operation when Zeek is allowed to
continue to run after zeek_init().
Adjustments during merge:
- kept the UNKNOWN Log::ID as placeholder value
- changed the coverage.find-bro-logs test to check for arbitrary $path
field values instead of just string literals
- don't force EnumVal to unsigned integer since the relevant union member
is the signed integer and added the relevant enum values/types to
.bif files for easier access
- compare FILE* versus file name to check for stdout equality (don't
think it matters much, just a bit more efficient)
- minor whitespace/style tweaks
* origin/topic/dev/print-to-log:
Added a non boolean configuration and other changes as suggested by Jon
Allow Print Statements to be redirected to a Log# This is a combination of 3 commits.
