The Zeek code base has very inconsistent #includes. Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed. Another side effect was a lot of header
bloat which slows down the build.
First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.
After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations. In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.
This patch speeds up the build by 19%, because each compilation unit
gets smaller. Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):
Before this patch:
3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps
After this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
Note - this compiles, but you cannot run Bro anymore - it crashes
immediately with a 0-pointer access. The reason behind it is that the
required clone functionality does not work anymore.
Majority of PLists are now created as automatic/stack objects,
rather than on heap and initialized either with the known-capacity
reserved upfront or directly from an initializer_list (so there's no
wasted slack in the memory that gets allocated for lists containing
a fixed/known number of elements).
Added versions of the ConnectionEvent/QueueEvent methods that take
a val_list by value.
Added a move ctor/assign-operator to Plists to allow passing them
around without having to copy the underlying array of pointers.
Changes:
- Changing semantics of the new_event() meta event: it's raised
only for events that have a handler defined. There are too many
checks in Bro that prevent events wo/ handler from being even
prepared to raise to do that differently.
- Adding test case.
* topic/robin/event-dumper:
New script misc/dump-events.bro, along with core support, that dumps events Bro is raising in an easily readable form.
Prettyfing Describe() for record types.
in an easily readable form.
This is for debugging purposes, obviously.
Example, including only SMTP events:
> bro -r smtp.trace misc/dump-events.bro DumpEvents::include=/smtp/
[...]
1254722768.219663 smtp_reply
[0] c: connection = [id=[orig_h=10.10.1.4, orig_p=1470/tcp, resp_h=74.53.140.153, [...]
[1] is_orig: bool = F
[2] code: count = 220
[3] cmd: string = >
[4] msg: string = xc90.websitewelcome.com ESMTP Exim 4.69 #1 Mon, 05 Oct 2009 01:05:54 -0500
[5] cont_resp: bool = T
1254722768.219663 smtp_reply
[0] c: connection = [id=[orig_h=10.10.1.4, orig_p=1470/tcp, resp_h=74.53.140.153, [...]
[1] is_orig: bool = F
[2] code: count = 220
[3] cmd: string = >
[4] msg: string = We do not authorize the use of this system to transport unsolicited,
[5] cont_resp: bool = T
[...]
Currently, a lot of interpreter runtime errors, such as an access to
an unset optional record field, cause Bro to abort with an internal
error. This is an experimental branch that turns such errors into
non-fatal runtime errors by internally raising exceptions. These are
caught upstream and processing continues afterwards.
For now, not many errors actually raise exceptions (the example above
does though). We'll need to go through them eventually and adapt the
current Internal() calls (and potentially others). More generally, at
some point we should cleanup the interpreter error handling (unifying
errors reported at parse- and runtime; and switching to exceptions for
all Expr/Stmt/Vals). But that's a larger change and left for later.
The main question for now is if this code is already helpful enough to
go into 2.0. It will quite likely prevent a number of crashes due to
script errors.
- Fixing a crash with an invalid pointer.
- Fixing a namespacing problem with is_ftp_data_conn() and check_relay_3().
- Fixing the do-we-have-an-event-handler-defined check.
Standard test-suite passes.
Seth, I think you can give it a try now ...
