The Zeek code base has very inconsistent #includes. Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed. Another side effect was a lot of header
bloat which slows down the build.
First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.
After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations. In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.
This patch speeds up the build by 19%, because each compilation unit
gets smaller. Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):
Before this patch:
3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps
After this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
* origin/topic/jsiwek/dns-mgr-fixes:
Fix timing out DNS lookups that were already resolved
Remove an unhelpful/optimistic DNS_Mgr optimization
Fix DNS_Mgr priority_queue usage
Remove dead code from DNS_Mgr
Improve DNS_Mgr I/O loop: prevent starvation due to busy Broker
Fix a ref counnting bug in DNS_Mgr
This could happen in the case of making repeated lookup requests
for the same thing within a short period of time: cleaning up an
old request that already got resolved would mistakenly see a new,
yet-to-be-resolved request with identical host/addr and mistakenly
assume it's in need of being timed out.
DNS_Mgr is always "idle", so Process() is always called when the
fd signals there's really something ready (except when flushing
at termination-time), so checking whether all pending request maps
are empty within Process() doesn't help much. If they are empty,
but there's somehow something to pull off the socket, the main loop
is just going to keep trying to call Process() until it gets read
(which would be bad if it's preventing another IOSource from getting
real work done).
This also installs symlinks from "zeek" and "bro-config" to a wrapper
script that prints a deprecation warning.
The btests pass, but this is still WIP. broctl renaming is still
missing.
#239
* origin/topic/jsiwek/plist-and-event-cleanup:
Add comments to QueueEvent() and ConnectionEvent()
Add methods to queue events without handler existence check
Cleanup/improve PList usage and Event API
The later simply doesn't work well in conjunction with hostname
literals. i.e. "google.com" (without quotes) needs to be resolved
to a set of addresses at parse-time, so if a user wishes to use a
custom resolver, we need that to be configured independently from
the order in which scripts get parsed. Configuring 'dns_resolver'
via scripting "redef" is clearly dependent on parse order.
Note 'dns_resolver' hasn't been in any release version yet, so
I'm removing it outright, no deprecation. The ZEEK_DNS_RESOLVER
environment variable now serves the original purpose.
Added ConnectionEventFast() and QueueEventFast() methods to avoid
redundant event handler existence checks.
It's common practice for caller to already check for event handler
existence before doing all the work of constructing the arguments, so
it's desirable to not have to check for existence again.
E.g. going through ConnectionEvent() means 3 existence checks:
one you do yourself before calling it, one in ConnectionEvent(), and then
another in QueueEvent().
The existence check itself can be more than a few operations sometimes
as it needs to check a few flags that determine if it's enabled, has
a local body, or has any remote receivers in the old comm. system or
has been flagged as something to publish in the new comm. system.
Majority of PLists are now created as automatic/stack objects,
rather than on heap and initialized either with the known-capacity
reserved upfront or directly from an initializer_list (so there's no
wasted slack in the memory that gets allocated for lists containing
a fixed/known number of elements).
Added versions of the ConnectionEvent/QueueEvent methods that take
a val_list by value.
Added a move ctor/assign-operator to Plists to allow passing them
around without having to copy the underlying array of pointers.
This commit marks (hopefully) ever one-parameter constructor as explicit.
It also uses override in (hopefully) all circumstances where a virtual
method is overridden.
There are a very few other minor changes - most of them were necessary
to get everything to compile (like one additional constructor). In one
case I changed an implicit operation to an explicit string conversion -
I think the automatically chosen conversion was much more convoluted.
This took longer than I want to admit but not as long as I feared :)
A bunch of infrastructure work to move IOSource, IOSourceRegistry (now
iosource::Manager) and PktSrc/PktDumper code into iosource/, and over
to a plugin structure.
Other IOSources aren't touched yet, they are still in src/*.
It compiles and does something with a small trace, but that's all I've
tested so far. There are quite certainly a number of problems left, as
well as various TODOs and cleanup; and nothing's cast in stone yet.
Will continue to work on this.
Replaced some with InternalWarning or InternalAnalyzerError, the later
being a new method which signals the analyzer to not process further
input. Some usages I just removed if they didn't make sense or clearly
couldn't happen. Also did some minor refactors of related code while
reviewing/exploring ways to get rid of InternalError usages.
Also, for TCP content file write failures there's a new event:
"contents_file_write_failure".
But not really since the global dns_mgr should be equal to "this" while
in all the member funcs. Still, better that they always refer to their
own instance instead of the global one.
A symptom of this is a "can't issue DNS request" warning in
reporter.log. (Doesn't look like this should be a typical thing
that happens in most environments).
* vlad/topic/vladg/dns_txt_queries:
Add detection rate threshold for MHR.
Make sure lookup_hostname_txt isn't checking lookup_hostname's cache for answers.
A couple of lookup_hostname_txt fixes.
Reverting the earlier function-join during merging.
* vlad/topic/vladg/dns_txt_queries:
DNS TXT support
I've tweaked it a little bit, still seems to work ...
I'd like to add a test for this but I'n not quite sure how to do that.
We'd have to hardcode a destination server that then'd be contacted
each time the test-suite runs.
