It can be visible overhead to call write() on the underlying pipe of the
EventMgr's flare whenever the first event is enqueued during an IO loop
iteration. Particularly in scenarios where there's about 1 event per packet
for long lived connections and script-side event processing is fast.
Given the event manager is drained anyhow at the end of the main loop, this
shouldn't be needed. In fact, the EventMgr.Process() method is basically
a stub. The one reason it is needed is when more events are enqueued during
a drain. That, however, can be dealt with by implementing GetNextTimeout()
to return 0.0 when there's more events queued. This way the main-loop's poll
timeout is 0.0 and it'll continue immediately.
This also allows to removes some extra code and drop the recently introduced
InitPostFork() addition: Without a pipe, there's no need to recreate it.
This largely copies over Spicy's `.clang-format` configuration file. The
one place where we deviate is header include order since Zeek depends on
headers being included in a certain order.
Because EventMgr is defined globally as an object (rather than a global
pointer to an EventMgr object), its pipe is created even before main()
is entered. This further means that in the fork-based supervisor setup,
all Zeek processes created from the top-level supervisor process share
the same pipe object for the EventMgr. In turn, whenever any of the
processes enqueued an event, the flare was fired and ready for reading
on all other processes in the cluster, causing much contention and
unneeded overhead.
Closes#3190
* jgras/topic/jgras/event-ts:
Add compatibility tests for timestamped events.
Add timestamps to auto published broker events.
Add timestamps to manually published broker events.
Annotate scheduled events with intended timestamp.
Add timestamp to events.
One timestamp to ts rename during the merge.
* origin/topic/timw/nullptr:
The remaining nulls
plugin/probabilistic/zeekygen: Replace nulls with nullptr
file_analysis: Replace nulls with nullptr
analyzer: Replace nulls with nullptr
iosource/threading/input/logging: Replace nulls with nullptr
Only 1% build time speedup, but still, it declutters the headers a bit.
Before this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
After this patch:
2537.19user 142.94system 2:26.90elapsed 1824%CPU (0avgtext+0avgdata 1434268maxresident)k
16240inputs+8887152outputs (1931major+48728888minor)pagefaults 0swaps
The Zeek code base has very inconsistent #includes. Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed. Another side effect was a lot of header
bloat which slows down the build.
First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.
After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations. In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.
This patch speeds up the build by 19%, because each compilation unit
gets smaller. Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):
Before this patch:
3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps
After this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
Note - this compiles, but you cannot run Bro anymore - it crashes
immediately with a 0-pointer access. The reason behind it is that the
required clone functionality does not work anymore.
Added ConnectionEventFast() and QueueEventFast() methods to avoid
redundant event handler existence checks.
It's common practice for caller to already check for event handler
existence before doing all the work of constructing the arguments, so
it's desirable to not have to check for existence again.
E.g. going through ConnectionEvent() means 3 existence checks:
one you do yourself before calling it, one in ConnectionEvent(), and then
another in QueueEvent().
The existence check itself can be more than a few operations sometimes
as it needs to check a few flags that determine if it's enabled, has
a local body, or has any remote receivers in the old comm. system or
has been flagged as something to publish in the new comm. system.
Majority of PLists are now created as automatic/stack objects,
rather than on heap and initialized either with the known-capacity
reserved upfront or directly from an initializer_list (so there's no
wasted slack in the memory that gets allocated for lists containing
a fixed/known number of elements).
Added versions of the ConnectionEvent/QueueEvent methods that take
a val_list by value.
Added a move ctor/assign-operator to Plists to allow passing them
around without having to copy the underlying array of pointers.
This commit marks (hopefully) ever one-parameter constructor as explicit.
It also uses override in (hopefully) all circumstances where a virtual
method is overridden.
There are a very few other minor changes - most of them were necessary
to get everything to compile (like one additional constructor). In one
case I changed an implicit operation to an explicit string conversion -
I think the automatically chosen conversion was much more convoluted.
This took longer than I want to admit but not as long as I feared :)
- Removed the gap_report event. It wasn't used anymore
and functionally no more capable that scheduling events
and using the get_gap_summary bif.
- Added functionality to Dictionaries to count cumulative
numbers of inserts performed. This is further used to
measure the total number of connections of various types.
Previously only the number of active connections was
available.
- The Reassembler base class now tracks active reassembly
size for all subclasses (File/TCP/Frag & unknown).
- Improvements to the stats.log. Mostly, more information.
