The Zeek code base has very inconsistent #includes. Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed. Another side effect was a lot of header
bloat which slows down the build.
First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.
After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations. In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.
This patch speeds up the build by 19%, because each compilation unit
gets smaller. Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):
Before this patch:
3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps
After this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
* "bro_is_terminating" is now "zeek_is_terminating"
* "bro_version" is now "zeek_version"
The old function names still exist for now, but are deprecated.
This also installs symlinks from "zeek" and "bro-config" to a wrapper
script that prints a deprecation warning.
The btests pass, but this is still WIP. broctl renaming is still
missing.
#239
The hook being added is:
bool HookReporter(const std::string& prefix, const EventHandlerPtr event,
const Connection* conn, const val_list* addl, bool location,
const Location* location1, const Location* location2,
bool time, const std::string& buffer) override;
This hook gives access to basically all information that is available in
the function in Reporter.cc that performs the logging. The hook is
called each time when anything passes through the reporter in the cases
in which an event usually would be called. This includes weirds. The
hook can return false to prevent the normal reporter events from being
raised.
This commit fixes and extends the behavior of HookLoadFile. Before this
change, HookLoadFile appended ".bro" to each path that was @loaded, even
if the path specified directory names. Furthermore it only gave the path
of the file as it was specified in the Bro script without revealing the
final path of the file that it was going to load.
This patch changes this behavior - in addition to giving the unmodified
path given in the @load command, the hook now returns the resolved path
of the file or directory it is going to load (if found). The hook is
furthermore raises for @load-sigs and @load-plugin; a enum specifies the
kind of load that is happening.
I added another small change - since we are inlining Configure(), we can
just set bro_plugin directly to BRO_PLUGIN_BRO_VERSION in
src/plugin/Plugin.h, instead of depending on the plugin to do it. This
also means we do not need to change init-plugin in bro-aux at this
moment.
BIT-1828 #closed
* origin/topic/robin/plugin-version-check:
Adding plugin API number into versioned function name, and removing old runtime API version check.
Extend plugin infrastructure to catch Bro version mismatches at link time.
time.
People keep running into the problem that they upgrade Bro but forget
to recompile their plugins--which can lead to crashes. While the
plugins' API version was supposed to catch this, it's not reliable as
that check may come too late. This change takes a different tack: We
compile a C function into the Bro binary that has Bro's version number
encoded into its name. A plugin can then reference that function. If
the Bro version changes, the function goes away and the plugin won't
load anymore.
I've integrated that function reference into the plugin skeleton code
so that new plugins get it automatically (unless explicitly removed).
I couldn't see a way to do it transparently for already existing
plugins unfortunately.
The version number used for the function name is slightly normalized
to skip any git revision postfixes (i.e., "2.5-xxx" is always treated
as "2.5-git") so that one doesn't need to recompile all plugins after
every master commit. That seems good enough, usually people run into
this when upgrading to a new release.
If one loads an old plugin into a new Bro, the error message looks
like this:
$ bro -NN Demo::Foo
fatal error in /home/robin/bro/master/scripts/base/init-bare.bro, line 1:
cannot load plugin library /home/robin/tmp/p/build//lib/Demo-Foo.linux-x86_64.so:
/home/robin/tmp/p/build//lib/Demo-Foo.linux-x86_64.so: undefined symbol: bro_version_2_5_git_debug
Not the prettiest, but better than a crash!
TODO: I'm still unsure if we should remove the plugin API version
altogetger now. This link-time check should catch everything the API
version does, except for master commits.
The two hooks being added are:
void HookLogInit(const std::string& writer, const std::string& instantiating_filter, bool local, bool remote, const logging::WriterBackend::WriterInfo& info, int num_fields, const threading::Field* const* fields);
which is called when a writer is being instantiated and contains
information about the fields being logged, as well as
bool HookLogWrite(const std::string& writer, const std::string& filter, const logging::WriterBackend::WriterInfo& info, int num_fields, const threading::Field* const* fields, threading::Value** vals);
which is called for each log line being written by each writer. It
contains all the data being written. The data can be changed in the
function call and lines can be prevented from being written.
This commit also fixes a few small problems with plugin hooks itself,
and extends the tests that were already there, besides introducing tests
for the added functionality.
The order in which the plugin initializers are executed is compiler
dependent. With this change, Tags will always be generated in
alphabetical ordering, not in compiler-dependent order.
Making two changes here:
- Renaming the hook to SetupAnalyzerTree.
- Reverting the reversal of the script load order. Instead, I'm
adding an additional script that Bro looks for to load first,
"__preload__.bro". Also extending the plugin docs to cover this.
- Increasing plugin API version, as I suppose adding a new virtual
function may invalidate binary compatibility.
* 'topic/jswaro/feature/HookAddToAnalyzer-tcprs-support' of https://github.com/jswaro/bro:
Add hook 'HookAddToAnalyzerTree' to support TCPRS plugin
This commit introduces a new hook, HookAddToAnalyzerTree, which
allows plugins to add a new analyzer to the analyzer tree during
analyzer tree creation. This hook is necessary to support the
TCPRS plugin.
Additionally, the order in which the scripts were loaded has been
changed to address a problem with undefined variable errors due
to load order issues.
Signed-off-by: James Swaro <james.swaro@gmail.com>
* origin/topic/gilbert/plugin-api-tweak:
Updating plugin.hooks baseline so that test succeeds
Revert spacing change that shouldn't have been included with the previous changeset ... should fix all of the plugin tests save hooks, which needs to be updated.
More small fixes
Small fixes
Incremental
Re-updating plugin.hooks test to include new argument output (after merge).
Fixing logic errors in HandlePluginResult
Updating tests and tweaking HookArgument to include Frame support.
Incremental commit: implementing a wrapper for the Val class.
Reverting change to const status of network_time. Also, see FIXME: in Func.cc / HandlePluginResult ...
Tweaks to result handling to make things a little more sane.
Plugin API: minor change (adding parent frame) to support calling methods from hook. Also declare network time update argument to be const because good practice.
BIT-1270 #merged
Conflicts:
testing/btest/Baseline/plugins.hooks/output
* Add frame support to HookArgument, since it's a new argument to HookCallFunction
* Fix test in api-version-mismatch to remove absolute paths from output
* Update test plugin to use new HookCallFunction interface
This is mainly an experiment to see if this makes sense. I'm not very
fond of the arguments being wrapped into a discriminating union, but I
like it better than other alternatives at least.
The new code is untested.
The Plugin.cc file is now just a standard class, with the interface
changed a bit to make it more easy to write. However, there're still
some conventions that one must follow to make everything work (like
using the right namespace structure).
This commit also includes the option to compile built-in plugins
dynamically instead of statically by adding
SET(BRO_PLUGIN_BUILD_DYNAMIC TRUE) to their CMake config. This hasn't
been tested much yet, and I'm still undecided if it's somethign we
would want to do by default--but we could now if wanted. :)
Also some minor other cleanup of plugin APIs and built infrastructure.
All tested on MacOS only currently.
Includes:
- Cleanup of the plugin API, in particular generally changing
const char* to std::string
- Renaming environment variable BRO_PLUGINS to BRO_PLUGIN_PATH,
defaulting to <prefix>/lib/bro/plugins
- Reworking how dynamic plugins are searched and activated. See
doc/devel/plugins.rst for details.
- New @load-plugin directive to explicitly activate a plugin
- Support for Darwin. (Linux untested right now)
- The init-plugin updates come with support for "make test", "make
sdist", and "make bdist" (see how-to).
- Test updates.
Notes: The new hook mechanism, which allows plugins to hook into Bro's
core a well-defined points, is still essentially untested.
I got rid of the earlier separate InterpreterPlugin class. Instead
Plugin now has a set of virtual methods HookSomething()... that
plugins can override. For efficiency purposes, they however need to
register first that they are interested in a hook, otherwise the
virtual method will never be called. The idea is to extend the set of
hooks over time as we figure out what's useful.
This is a checkpoint commit that's essentially untested and probably
broken. It compiles, though.
This is essentially the code from the dynamic-plugin branch except for
some pieces that I have split out into separate, earlier commits.
I'm going to updatre things in this branch going forward.
We now explicitly mark plugins as static vs dynamic (though we don't
have the latter yet) instead of piggy-backing that on the version.
Also, versions are now ignored for static plugins.
Adding one todo back in as that's something we indeed still need to do.
* origin/topic/jsiwek/plugins-cleanup:
Fix various documentation/typos; remove a few superfluous things.
