Logs that got sent sparsely or burstily would get buffered for long
periods of time since the logic to flush them only does so on the next
log write. In the worst case, a subsequent log write could never happen
and cause a log entry to be indefinitely buffered.
This fix introduces a recurring event/timer to simply flush all pending
logs at frequency of Broker::log_batch_interval.
This change standardizes threading formatter error handling and moves
the remaining error calls to be warnings instead.
This is in line with already existing code - in most cases warnings were
raised, only a few cases raised errors. These cases do not differ
significantly from other cases in which warnings are raised.
This also fixes GH-692, in which misformatted lines prevent future file
parsing.
This commit also moves the FailWarn method that is used by both the
config and the ascii reader up to the ReaderBackend. Furthermore it
makes the Warning method of ReaderBackend respect the warning
suppression that is introduced by the FailWarn method.
Only 1% build time speedup, but still, it declutters the headers a bit.
Before this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
After this patch:
2537.19user 142.94system 2:26.90elapsed 1824%CPU (0avgtext+0avgdata 1434268maxresident)k
16240inputs+8887152outputs (1931major+48728888minor)pagefaults 0swaps
The Zeek code base has very inconsistent #includes. Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed. Another side effect was a lot of header
bloat which slows down the build.
First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.
After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations. In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.
This patch speeds up the build by 19%, because each compilation unit
gets smaller. Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):
Before this patch:
3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps
After this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
- Also removed the setting of BinPAC_ROOT_DIR in the configure
script's --with-binpac= option as that breaks the cross-compilation
use-case
* 'master' of https://github.com/ffontaine/zeek:
CMakeLists.txt: fix cross-compilation with binpac
* origin/topic/timw/ioloop-followup:
Increase timeout value for live interfaces without file descriptors
Use ranged-for loops in a few places in iosource Manager
Change order of includes in iosource Manager, fixes build on FreeBSD 11
When cross-compiling, BINPAC_EXE_PATH will be set by the user to the
host binpac binary which is fine however aux/binpac won't be built which
will raise a build failure as target binpac (headers, library) won't be
installed or built
Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
- Minor changes in merge: extended unit test, prefer emplace_back(),
remove unused "found" count in new function
* 'optimize_normalize_path' of https://github.com/MaxKellermann/zeek:
util: add a tokenize_string() overload which returns string_views
util: store std::string_view in "final_components" vector
util: use "auto" in normalize_path()
util: reserve space in normalize_path()
util: skip "." completely in normalize_path()
util: pass std::string_view to normalize_path()
util: pass std::string_view to tokenize_string()
util: don't modify the input string in tokenize_string()
- Minor code formatting change in merge
* 'misc_cleanup' of https://github.com/MaxKellermann/zeek:
Desc: move realloc() call out of the loop
SerializationFormat: move realloc() call out of the loop
PacketDumper: remove unused types
* origin/topic/timw/264-alternative:
Add better error messaging when RegisterFd/UnregisterFd fail
Update Supervisor code for the new IOSource API
Show percentage of packets dropped in the final process output
Update NEWS and docs submodule
Remove concept of multiple timer managers
Test changes caused by minor order-of-operation changes related to the new loop architecture
Have terminate_processing() raise SIGTERM instead of calling the signal handler directly
PktSrc iosource changes to match the new IOSource API
Broker manager changes to match the new IOSource API and loop architecture
DNS_Mgr changes to match the new IOSource API and loop architecture
Threading changes for the new loop architecture
Add Trigger manager for managing triggers created by things like 'when' statements
Make TimerMgr an IOSource
Modify IOSource Manager to implement new loop architecture
Minor amount of code cleanup in Pcap IO source
IOSource API changes for new loop architecture
Type and variable usage cleanup in Net.h
Remove #include of some iosource files from Net.h
Only allow a single trace file (-r) or interface (-i) option on the command-line
Remove CQ_TimerMgr in favor of PQ_TimerMgr
- All timers are now handled by a single global timer manager, which simplifies how they handled by the IOSource manager.
- This change flows down a number of changes to other parts of the code. The timer manager tag field is removed, which means that matching connections to a timer manager is also removed. This removes the ability to tag a connection as internal or external, since that's how the connections where differentiated. This in turn removes the `current_conns_extern` field from the `ConnStats` record type in the script layer.
- threading::Manager is no longer an IOSource.
- threading::MsgThread is now an IOSource. This allows threads themselves to signal when they have data to process instead of continually checking each of the threads on every loop pass.
- Make the thread heartbeat timer an actual timer and let it fire as necessary instead of checking to see if it should fire
- Adds new trigger namespace
- Adds trigger::Manager class as a new IOSource for keeping track of triggers and integrating them into the loop. Previously the loop relied on the event manager Drain() method to process all triggers on every loop, but now that the loop actively waits for events to occur, triggers would not fire when they needed to. Adding them as part of the loop ensures they're checked.
- This allows the loop to check what the next timeout is and use that as the basis for the timeout of the poll
- This commit also removes the TimerMgr::Tag type, since it causes a name clash with other code in IOSource
- Removes entire FindSoonest method that includes all of the calls to select() for checking for ready sources
- Removes FD_Set checking against IOSources
- Adds system for registering and unregistering file descriptors from IOSources. This allows individual sources to mark themselves as ready to be checked by the loop as they become available.
- Adds entirely new loop architecture based on checking the IOSources for when their next timeout is, and then waiting for either that timeout or when the next source is ready. This also implements the polling based on what the OS supports, instead of just calling select() on all platforms. Currently it supports kqueue, epoll, and plain poll.
- Adds system for pinging the loop to force it to wake up
- Removed GetFD and NextTimestamp methods
- Removed concept of idle sources
- Renamed Init to InitSource to avoid a name clash with the thread code
- Added GetNextTimeout method
This commit also removes the FD_Set file since it's no longer used
