This is a larger internal change that moves the analyzer
infrastructure to a more flexible model where the available analyzers
don't need to be hardcoded at compile time anymore. While currently
they actually still are, this will in the future enable external
analyzer plugins. For now, it does already add the capability to
dynamically enable/disable analyzers from script-land, replacing the
old Analyzer::Available() methods.
There are three major parts going into this:
- A new plugin infrastructure in src/plugin. This is independent
of analyzers and will eventually support plugins for other parts
of Bro as well (think: readers and writers). The goal is that
plugins can be alternatively compiled in statically or loadead
dynamically at runtime from a shared library. While the latter
isn't there yet, there'll be almost no code change for a plugin
to make it dynamic later (hopefully :)
- New analyzer infrastructure in src/analyzer. I've moved a number
of analyzer-related classes here, including Analyzer and DPM;
the latter now renamed to Analyzer::Manager. More will move here
later. Currently, there's only one plugin here, which provides
*all* existing analyzers. We can modularize this further in the
future (or not).
- A new script interface in base/framework/analyzer. I think that
this will eventually replace the dpm framework, but for now
that's still there as well, though some parts have moved over.
I've also remove the dpd_config table; ports are now configured via
the analyzer framework. For exmaple, for SSH:
const ports = { 22/tcp } &redef;
event bro_init() &priority=5
{
...
Analyzer::register_for_ports(Analyzer::ANALYZER_SSH, ports);
}
As you can see, the old ANALYZER_SSH constants have more into an enum
in the Analyzer namespace.
This is all hardly tested right now, and not everything works yet.
There's also a lot more cleanup to do (moving more classes around;
removing no longer used functionality; documenting script and C++
interfaces; regression tests). But it seems to generally work with a
small trace at least.
The debug stream "dpm" shows more about the loaded/enabled analyzers.
A new option -N lists loaded plugins and what they provide (including
those compiled in statically; i.e., right now it outputs all the
analyzers).
This is all not cast-in-stone yet, for some things we need to see if
they make sense this way. Feedback welcome.
* origin/topic/matthias/opaque:
Add new unit test for opaque serialization.
Migrate entropy testing to opaque.
C++ify RandTest.*
Fix a hard-to-spot bug.
Use more descriptive error message.
Fix the fix :-/.
Fix initialization of hash values.
Be clearer about delegation.
Implement serialization of opaque types.
Update hash BiF documentation.
Migrate free SHA* functions to SHA*Val::digest().
Add missing type name that caused failing tests.
Update base scripts and unit tests.
Simplify hash function BiFs.
Add support for opaque hash values.
Adapt BiF & Bro parser to handle opaque types.
More lexer/parser work.
Implement equivalence relation for opaque types.
Support basic serialization of opaque.
Add opaque type to lexer, parser, and BroType.
Closes#925
Conflicts:
aux/broccoli
When using --enable-debug, values keep track of the last identifier
to which they were bound by storing a ref'd ID pointer. This could
lead to some circular dependencies in which an ID is never reclaimed
because the Val is bound to the ID and the ID is bound to the Val, with
both holding references to each other.
There might be more cases where this feature of --enable-debug caused
a leak, but it showed up in particular when running the
core.leaks.remote unit test due to the internal
SendID("peer_description") call during the handshake between remote
processes. Other tests showed the send_id() BIF leaked more generally.
Tracking the ID last bound to a Val through just the identifier string
instead of a ref'd ID pointer fixes the leak.
* origin/topic/icmp6:
Fixes for IPv6 truncation and ICMP/ICMP6 analysis.
Change ICMPv6 checksum calculation to use IP_Hdr wrapper.
Update IPv6 atomic fragment unit test to filter output of ICMPv6.
Add more data to icmp events
More code cleanup
Add more icmpv6 events, and general code cleanup
Fix compile failure after merge from master
Significant edit pass over ICMPv6 code.
Porting Matti's branch to git.
Closes#808.
As we can't use the IPAddr class (because it's not thread-safe), this
involved a bit manual address manipulation and also shuffling some
things around a bit.
Not fully working yet, the tests for remote logging still fail.
Seems to work -- all test pass.
But there are thread-safety issues at the moment, because the constructors of IPAddr and IPPrefix are not thread-safe, but needed by workers.
Conflicts:
src/logging/Manager.cc
pass yet.
Changes:
- Gave IPAddress/IPPrefix methods AsString() so that one doesn't need
to cast to get a string represenation.
- Val::AsAddr()/AsSubnet() return references rather than pointers. I
find that more intuitive.
- ODesc/Serializer/SerializationFormat get methods to support
IPAddress/IPPrefix directly.
- Reformatted the comments in IPAddr.h from /// to /** style.
- Given IPPrefix a Contains() method.
- A bit of cleanup.
Internally, all BROv6 preprocessor switches were removed and
addr/subnet representations wrapped in the new IPAddr/IPPrefix classes.
Some script-layer changes of note:
- dns_AAAA_reply event signature changed: the string representation
of an IPv6 addr is easily derived from the addr value, it doesn't
need to be another parameter. This event also now generated directly
by the DNS analyzer instead of being "faked" into a dns_A_reply event.
- removed addr_to_count BIF. It used to return the host-order
count representation of IPv4 addresses only. To make it more
generic, we might later add a BIF to return a vector of counts
in order to support IPv6.
- changed the result of enclosing addr variables in vertical pipes
(e.g. |my_addr|) to return the bit-width of the address type which
is 128 for IPv6 and 32 for IPv4. It used to function the same
way as addr_to_count mentioned above.
- remove bro_has_ipv6 BIF
Some of the changes only clean up at termination to make perftools
happt, but there were some "real" leaks as well.
This fixes all DNS leaks I could reproducem, including most likely
what's reported in #534. Closing #534.
I'm also adding a new btest subdir core/leaks with tests requiring
perftools support. These don't compare against base lines but abort
whenever perftools reports a leak (with stack information to track it
down). Right now, these are passing.
* origin/topic/jsiwek/path-func-record-demote:
Fix filter path_func to allow record argument as a subset of stream's columns.
Conflicts:
src/LogMgr.cc
Closes#600.
This required adding the ability for RecordVal::CoerceTo functions to
optionally allow orphaning fields. The default is to not allow it, but
now before writing to a log, the value of the stream's columns is coerced
down, if necessary, before passing it on to the filter's path_func.
Addresses #600.
This is obviously a change that break backwards-compatibility. I hope
I caught all cases where vectors are used ...
I've completely removed the VECTOR_MIN constant. Turns out that was
already not working: some code pieces were nevertheless hard-coding
the 1-based indexing ...
Changed BroType to track a char* instead of an ID* that represents
the declared type's identifier. It was also necessary to serialize
this information or else it can be lost (e.g. FieldDecl's in RecordType
always seem to get serialized at some point).
DescribeReST() functions added to many classes to get the output
closer to being reST compatible; still needs tweaking for Sphinx
(reST->HTML) compatibility.
- Moving all functions into the Log::* namespace, using the recent
bifcl updates. Moved logging-specific stuff to logging.bif.
- Log::create_stream() now takes a record Log::Stream as its second
argument, which specifies columns and (optionally) the event.
- All the internal BiFs are now called "Log::__<something>", with
script-level wrappers "Log::<something>". That first allows to add
additional code at the script-level, and second makes things better
comprehendible as now all relevant functionality is collected (and
later documetned) in policy/logging.bro.
- New function Log::flush(id), which does the obvious assuming the
writer supports it.
- add_default_filter() is now called implicitly with every
create_stream(). Seems that we usually want that functionality, and
when not, remove_default_filter() gets rid of it.
- The namespace of a stream's ID is now used as the default "path"
(e.g., if the namespace is SSH, the default log file is "ssh.log").
- Updated policy/test-logging.bro as well as the btest tests according
to these changes.
logging framework.
- To enable passing a type into a bif, there's now a new
BroType-derived class TypeType and a corresponding TYPE_TYPE tag.
With that, a Val can now have a type as its value.
This is experimental for now.
- RecordVal's get a new method CoerceTo() to coerce their value into a
another record type with the usual semantics. Most of the code in
there was previously in RecordContructorExpr::InitVal(), which is
now calling the new CoerceTo() method.
Val::Val had prototypes for int, long, int64, etc. But depending on the
architecture some of those might be the same (int64 and long) thus
yielding a compile error.
Fix: only use int32, int64, etc. for prototype. ints and longs can still
be passed, since they will match one of these fixed-width types
regardless of platform.
Also fix some more compiler warnings with format strings.
