This unfortunately cuases a ton of flow-down changes because a lot of other
code was depending on that definition existing. This has a fairly large chance
to break builds of external plugins, considering how many internal ones it broke.
The Zeek code base has very inconsistent #includes. Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed. Another side effect was a lot of header
bloat which slows down the build.
First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.
After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations. In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.
This patch speeds up the build by 19%, because each compilation unit
gets smaller. Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):
Before this patch:
3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps
After this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
For backward compatibility when reading values, we first check
the ZEEK-prefixed value, and if not set, then check the corresponding
BRO-prefixed value.
When searching for script files, look for both the new and old file
extensions. If a file with ".zeek" can't be found, then search for
a file with ".bro" as a fallback.
The hook being added is:
bool HookReporter(const std::string& prefix, const EventHandlerPtr event,
const Connection* conn, const val_list* addl, bool location,
const Location* location1, const Location* location2,
bool time, const std::string& buffer) override;
This hook gives access to basically all information that is available in
the function in Reporter.cc that performs the logging. The hook is
called each time when anything passes through the reporter in the cases
in which an event usually would be called. This includes weirds. The
hook can return false to prevent the normal reporter events from being
raised.
This commit fixes and extends the behavior of HookLoadFile. Before this
change, HookLoadFile appended ".bro" to each path that was @loaded, even
if the path specified directory names. Furthermore it only gave the path
of the file as it was specified in the Bro script without revealing the
final path of the file that it was going to load.
This patch changes this behavior - in addition to giving the unmodified
path given in the @load command, the hook now returns the resolved path
of the file or directory it is going to load (if found). The hook is
furthermore raises for @load-sigs and @load-plugin; a enum specifies the
kind of load that is happening.
The two hooks being added are:
void HookLogInit(const std::string& writer, const std::string& instantiating_filter, bool local, bool remote, const logging::WriterBackend::WriterInfo& info, int num_fields, const threading::Field* const* fields);
which is called when a writer is being instantiated and contains
information about the fields being logged, as well as
bool HookLogWrite(const std::string& writer, const std::string& filter, const logging::WriterBackend::WriterInfo& info, int num_fields, const threading::Field* const* fields, threading::Value** vals);
which is called for each log line being written by each writer. It
contains all the data being written. The data can be changed in the
function call and lines can be prevented from being written.
This commit also fixes a few small problems with plugin hooks itself,
and extends the tests that were already there, besides introducing tests
for the added functionality.
Nice, thanks!
BIT-1671 #merged
* origin/topic/vladg/bit-1671:
Add descriptions for what the SMB1 events do, and references to the specification.
Add descriptions for what the SMB2 events do, and references to the specification.
Improve the SMB2 documentation.
Improve the SMB1 documentation a bit. Clean it up, make it more consistent, and add references to similar events.
Add DCE-RPC and NTLM abbreviation documentation for their events.
Remove reference to epm_map_response in RPC documentation. Addresses BIT-1671.
Rename events referenced in SMB1::Header documentation with new SMB analyzer. Addresses BIT-1671.
Because plugins register their script directories with the BROPATH,
Broxygen stripped them out from plugin script paths it was indexing.
That then led to multiple plugins ending up with the same script
paths, triggering warnings about duplicates.
I fixed this by checking if a script comes out of a plugin. If so, it
gets an artifcial index prefix "<plugin-name>:", followed by the
script's relative path inside the plugin's top-level directory. For
example, "/opt/bro/lib/bro/plugins/Bro_Netmap/scripts/init.bro" now
turns into "Bro::Netmap:scripts/init.bro" for Broxygen purposes
(whereas it used to be just "init.bro").
Addresses BIT-1663.
(Can't think of a good way to add a test for this unfortunately.)
The order in which the plugin initializers are executed is compiler
dependent. With this change, Tags will always be generated in
alphabetical ordering, not in compiler-dependent order.
Making two changes here:
- Renaming the hook to SetupAnalyzerTree.
- Reverting the reversal of the script load order. Instead, I'm
adding an additional script that Bro looks for to load first,
"__preload__.bro". Also extending the plugin docs to cover this.
- Increasing plugin API version, as I suppose adding a new virtual
function may invalidate binary compatibility.
* 'topic/jswaro/feature/HookAddToAnalyzer-tcprs-support' of https://github.com/jswaro/bro:
Add hook 'HookAddToAnalyzerTree' to support TCPRS plugin
This commit introduces a new hook, HookAddToAnalyzerTree, which
allows plugins to add a new analyzer to the analyzer tree during
analyzer tree creation. This hook is necessary to support the
TCPRS plugin.
Additionally, the order in which the scripts were loaded has been
changed to address a problem with undefined variable errors due
to load order issues.
Signed-off-by: James Swaro <james.swaro@gmail.com>
* origin/topic/gilbert/plugin-api-tweak:
Updating plugin.hooks baseline so that test succeeds
Revert spacing change that shouldn't have been included with the previous changeset ... should fix all of the plugin tests save hooks, which needs to be updated.
More small fixes
Small fixes
Incremental
Re-updating plugin.hooks test to include new argument output (after merge).
Fixing logic errors in HandlePluginResult
Updating tests and tweaking HookArgument to include Frame support.
Incremental commit: implementing a wrapper for the Val class.
Reverting change to const status of network_time. Also, see FIXME: in Func.cc / HandlePluginResult ...
Tweaks to result handling to make things a little more sane.
Plugin API: minor change (adding parent frame) to support calling methods from hook. Also declare network time update argument to be const because good practice.
BIT-1270 #merged
Conflicts:
testing/btest/Baseline/plugins.hooks/output
This is mainly an experiment to see if this makes sense. I'm not very
fond of the arguments being wrapped into a discriminating union, but I
like it better than other alternatives at least.
The new code is untested.
The Plugin.cc file is now just a standard class, with the interface
changed a bit to make it more easy to write. However, there're still
some conventions that one must follow to make everything work (like
using the right namespace structure).
This commit also includes the option to compile built-in plugins
dynamically instead of statically by adding
SET(BRO_PLUGIN_BUILD_DYNAMIC TRUE) to their CMake config. This hasn't
been tested much yet, and I'm still undecided if it's somethign we
would want to do by default--but we could now if wanted. :)
Also some minor other cleanup of plugin APIs and built infrastructure.
All tested on MacOS only currently.
Includes:
- Cleanup of the plugin API, in particular generally changing
const char* to std::string
- Renaming environment variable BRO_PLUGINS to BRO_PLUGIN_PATH,
defaulting to <prefix>/lib/bro/plugins
- Reworking how dynamic plugins are searched and activated. See
doc/devel/plugins.rst for details.
- New @load-plugin directive to explicitly activate a plugin
- Support for Darwin. (Linux untested right now)
- The init-plugin updates come with support for "make test", "make
sdist", and "make bdist" (see how-to).
- Test updates.
Notes: The new hook mechanism, which allows plugins to hook into Bro's
core a well-defined points, is still essentially untested.
