This is essentially the code from the dynamic-plugin branch except for
some pieces that I have split out into separate, earlier commits.
I'm going to updatre things in this branch going forward.
Thanks to git this merge was less troublesome that I was afraid it
would be. Not all tests pass yet though (and file hashes have changed
unfortunately).
Conflicts:
cmake
doc/scripts/DocSourcesList.cmake
scripts/base/init-bare.bro
scripts/base/protocols/ftp/main.bro
scripts/base/protocols/irc/dcc-send.bro
scripts/test-all-policy.bro
src/AnalyzerTags.h
src/CMakeLists.txt
src/analyzer/Analyzer.cc
src/analyzer/protocol/file/File.cc
src/analyzer/protocol/file/File.h
src/analyzer/protocol/http/HTTP.cc
src/analyzer/protocol/http/HTTP.h
src/analyzer/protocol/mime/MIME.cc
src/event.bif
src/main.cc
src/util-config.h.in
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/istate.events-ssl/receiver.http.log
testing/btest/Baseline/istate.events-ssl/sender.http.log
testing/btest/Baseline/istate.events/receiver.http.log
testing/btest/Baseline/istate.events/sender.http.log
And changed the endianness parameter of bytestring_to_count() BIF to
default to false (big endian), mostly just to prove that the BIF parser
doesn't choke on default parameters.
There's now a new directory "src/protocols/", and the plan is for each
protocol analyzer to eventually have its own subdirectory in there
that contains everything it defines (C++/pac/bif). The infrastructure
to make that happen is in place, and two analyzers have been
converted to the new model, HTTP and SSL; there's no further
HTTP/SSL-specific code anywhere else in the core anymore (I believe :-)
Further changes:
- -N lists available plugins, -NN lists more details on what these
plugins provide (analyzers, bif elements). (The latter does not
work for analyzers that haven't been converted yet).
- *.bif.bro files now go into scripts/base/bif/; and
scripts/base/bif/plugins/ for bif files provided by plugins.
- I've factored out the bifcl/binpac CMake magic from
src/CMakeLists.txt to cmake/{BifCl,Binpac}
- There's a new cmake/BroPlugin that contains magic to allow
plugins to have a simple CMakeLists.txt. The hope is that
eventually the same CMakeLists.txt can be used for compiling a
plugin either statically or dynamically.
- bifcl has a new option -c that changes the code it generates so
that it can be used with a plugin.
TODOs:
- "make install" is probably broken.
- Broxygen is probably broken for plugin-defined events.
- event groups are broken (do we want to keep them?)
This is a larger internal change that moves the analyzer
infrastructure to a more flexible model where the available analyzers
don't need to be hardcoded at compile time anymore. While currently
they actually still are, this will in the future enable external
analyzer plugins. For now, it does already add the capability to
dynamically enable/disable analyzers from script-land, replacing the
old Analyzer::Available() methods.
There are three major parts going into this:
- A new plugin infrastructure in src/plugin. This is independent
of analyzers and will eventually support plugins for other parts
of Bro as well (think: readers and writers). The goal is that
plugins can be alternatively compiled in statically or loadead
dynamically at runtime from a shared library. While the latter
isn't there yet, there'll be almost no code change for a plugin
to make it dynamic later (hopefully :)
- New analyzer infrastructure in src/analyzer. I've moved a number
of analyzer-related classes here, including Analyzer and DPM;
the latter now renamed to Analyzer::Manager. More will move here
later. Currently, there's only one plugin here, which provides
*all* existing analyzers. We can modularize this further in the
future (or not).
- A new script interface in base/framework/analyzer. I think that
this will eventually replace the dpm framework, but for now
that's still there as well, though some parts have moved over.
I've also remove the dpd_config table; ports are now configured via
the analyzer framework. For exmaple, for SSH:
const ports = { 22/tcp } &redef;
event bro_init() &priority=5
{
...
Analyzer::register_for_ports(Analyzer::ANALYZER_SSH, ports);
}
As you can see, the old ANALYZER_SSH constants have more into an enum
in the Analyzer namespace.
This is all hardly tested right now, and not everything works yet.
There's also a lot more cleanup to do (moving more classes around;
removing no longer used functionality; documenting script and C++
interfaces; regression tests). But it seems to generally work with a
small trace at least.
The debug stream "dpm" shows more about the loaded/enabled analyzers.
A new option -N lists loaded plugins and what they provide (including
those compiled in statically; i.e., right now it outputs all the
analyzers).
This is all not cast-in-stone yet, for some things we need to see if
they make sense this way. Feedback welcome.
* origin/topic/matthias/opaque:
Add new unit test for opaque serialization.
Migrate entropy testing to opaque.
C++ify RandTest.*
Fix a hard-to-spot bug.
Use more descriptive error message.
Fix the fix :-/.
Fix initialization of hash values.
Be clearer about delegation.
Implement serialization of opaque types.
Update hash BiF documentation.
Migrate free SHA* functions to SHA*Val::digest().
Add missing type name that caused failing tests.
Update base scripts and unit tests.
Simplify hash function BiFs.
Add support for opaque hash values.
Adapt BiF & Bro parser to handle opaque types.
More lexer/parser work.
Implement equivalence relation for opaque types.
Support basic serialization of opaque.
Add opaque type to lexer, parser, and BroType.
Closes#925
Conflicts:
aux/broccoli
The Logger class is now in charge of reporting all errors, warnings,
informational messages, weirds, and syslogs. All other components
route their messages through the global bro_logger singleton.
The Logger class comes with these reporting methods:
void Message(const char* fmt, ...);
void Warning(const char* fmt, ...);
void Error(const char* fmt, ...);
void FatalError(const char* fmt, ...); // Terminate Bro.
void Weird(const char* name);
[ .. some more Weird() variants ... ]
void Syslog(const char* fmt, ...);
void InternalWarning(const char* fmt, ...);
void InternalError(const char* fmt, ...); // Terminates Bro.
See Logger.h for more information on these.
Generally, the reporting now works as follows:
- All non-fatal message are reported in one of two ways:
(1) At startup (i.e., before we start processing packets),
they are logged to stderr.
(2) During processing, they turn into events:
event log_message%(msg: string, location: string%);
event log_warning%(msg: string, location: string%);
event log_error%(msg: string, location: string%);
The script level can then handle them as desired.
If we don't have an event handler, we fall back to
reporting on stderr.
- All fatal errors are logged to stderr and Bro terminates
immediately.
- Syslog(msg) directly syslogs, but doesn't do anything else.
The three main types of messages can also be generated on the
scripting layer via new Log::* bifs:
Log::error(msg: string);
Log::warning(msg: string);
Log::message(msg: string);
These pass through the bro_logger as well and thus are handled in the
same way. Their output includes location information.
More changes:
- Removed the alarm statement and the alarm_hook event.
- Adapted lots of locations to use the bro_logger, including some
of the messages that were previously either just written to
stdout, or even funneled through the alarm mechanism.
- No distinction anymore between Error() and RunTime(). There's
now only one class of errors; the line was quite blurred already
anyway.
- util.h: all the error()/warn()/message()/run_time()/pinpoint()
functions are gone. Use the bro_logger instead now.
- Script errors are formatted a bit differently due to the
changes. What I've seen so far looks ok to me, but let me know
if there's something odd.
Notes:
- The default handlers for the new log_* events are just dummy
implementations for now since we need to integrate all this into
the new scripts anyway.
- I'm not too happy with the names of the Logger class and its
instance bro_logger. We now have a LogMgr as well, which makes
this all a bit confusing. But I didn't have a good idea for
better names so I stuck with them for now.
Perhaps we should merge Logger and LogMgr?
- Summary comments (##!) can now be placed at the beginning of
BiF files (but still outside C segments). An issue was fixed where
these comments would mistakenly be transferred into the generated
.func_def file and cause a compile error. I completely removed writing
any opt_ws value into the .func_def file because it was currently not
writing anything besides whitespace.
- The generation of reST for the collecting of "groups" of policy
script documentation now happens at build time of `make doc` through the
use of a helper script rather than doing this at configure time so that
changes to summary text will always be reflected in the documentation.
Removing everything related to trace rewriting.
(I wasn't too careful in ensuring that I catch everything in the
scripts; Seth is working on those anyway.)
(Merging by cherry-picking the corresponding commit, as the branch was
accidentally made off of the logging stuff).
* origin/topic/gregor/bif-tuning:
Refactor: BifTypePtr --> BifType
Bif const: make sure const is indeed a constant.
Support any type in bif const declaration.
Tweak for bifcl
Fix to bifcl wrt namespaces.
Enable declaration of set, vector, and table types in bifs.
Moving type declarations into its own bif file
Support namespaces / modules in bif. Checkpoint.
Support namespaces / modules in bif. Checkpoint.
Remove leftovers from removing "declare enum" from bifcl
Use namespaces for NetVar type pointers.
Remove unused and unnecessary "declare enum" from bifcl
Bif: add record type declaration.
Minor tweaks for bif language.
enum type: don't allow mixing of explicit value and auto-increment.
Add support for enum with explicit enumerator values.
Closes#403.
Revamp of const delcaration in bifs:
* Can only declare are const in the bif, but we cannot assign a value
or attribute to it. One has to do this in a policy file (bro.init)
* Type specification in bif is now mandatory
* Support any type in bifs (previously only bools were supported).
This will also help with automatic documentation generation, since all
const are now defined in the policy layer and thus can be documented
from there. The bif just gives the C++ layer easy access.
(now actually commiting all the files)
This change is actually two-fold:
a) bif's now accept module XYZ; statements and module::ID for
function, const, event, enum, etc. declartation
b) Added C++-namespaces to variables, functions, etc. that are declared
in bif but accessed from C++
This required some (lightweight) re-factoring of the C++ codes.
Note, event's don't have their own C++ namespace yet, since this
would require a rather huge re-factoring.
Compiles and passes test suite.
New namespace feature not tested yet.
Documentation to follow.
Enums defined in bifs and records declared in bifs are now available
in the C++ layer in namespaces (before they were in the global namespace
with enum_* and rectype_* prefixes).
Namespaces are now BroTypePtr::Enum::<name-of-enum> and
BroTypePtr::Record::<name-of-record>
One can now declare (but not define) a record type in bif:
type <my_record_type_name> : record;
This adds the netvar glue so that the event engine knows about the type. One
still has to define the type in bro.init. Would be nice, if we could
just define the record type here and then copy to the .bif.bro file, but
type delcarations in bro can be quite powerful. Don't know whether it's
worth it extend the bif-language to be able to handle that all.... Or
we just support a simple form of record type definitions
The type has be called <my_record_type_name> in bro.init and it will
be availabe as a RecordType * rectype_<my_record_type_name> in the event
engine.
TODO: add other types (tables, sets)
* Adding support for enums with explicit enumerator values (see doc
below) to bifcl and policy layer.
* Bifcl: remove (partially written) output files on error and
do a nice exit(1) instead of harsh abort() on parse errors.
* CMakeText: if bifcl fails, remove output files (failsafe,
in case bifcl fails to clean up after itself).
Enum description
----------------
Enum's are supported in .bif and .bro scripts.
An enum in a bif will become available in the event engine and
the policy layer.
Enums are "C-style". The first element in an enum will have a
value of 0, the next value will be 1, etc.
It is possible to assign an enumerator value to an element. If
next element does not have an explicit value, its values will be
the value of the last element + 1
Example::
type foo: enum {
BAR_A, # value will be 0
BAR_B, # value will be 1
BAR_C = 10, # value will be 10
BAR_D, # value will be 11
};
Enumerator values can only by positive integer literals.
The literals can be specified in (0x....), but not in octal (bro policy
layer limitation). So, do not use 0123 as value in bifs!
Each enumerator value can only be used once per enum (C allows
to use the same value multiple times). This makes reverse mapping from
value to name (e.g., in %s format strings) unambigious. This is enforced
in by the policy script.
Enums can be redef'ed, i.e., extended. Enumerator values will continue
to increment. If there are multiple redefs in different policy scripts,
then name <-> value mappings will obviously depend on the order in
which scripts are loaded (which might not be obvious).
Example::
redef enum foo += {
BAR_E, # value will be 12
BAR_F = 5, # value will be 5
BAR_G, # value will be 6
};
