* Add deprecation for MIME_Entity::ContentType(), use GetContentType()
* Add deprecation for MIME_Entity::ContentSubType(), use GetContentSubType()
* Add deprecation for MIME_Message::BuildHeaderVal(), use ToHeaderVal()
* Add deprecation for MIME_Message::BuildHeaderTable(), use ToHeaderTable()
* Add deprecation for mime::new_string_val(), use mime::to_stringval()
* Add deprecation for ARP_Analyzer::ConstructAddrVal(), use ToAddrVal()
* Add deprecation for ARP_Analyzer::EthAddrToStr(), use ToEthAddrStr()
* Change the Func::Call() replacement to be named Func::Invoke()
This also changes the argument type of Func::operator() to zeek::Args*
to allow plugins to be able to alter function arguments in place as
was previously documented.
* 'intrusive_ptr' of https://github.com/MaxKellermann/zeek: (32 commits)
Scope: store IntrusivePtr in `local`
Scope: pass IntrusivePtr to AddInit()
DNS_Mgr: use class IntrusivePtr
Scope: use class IntrusivePtr
Attr: use class IntrusivePtr
Expr: check_and_promote_expr() returns IntrusivePtr
Frame: use class IntrusivePtr
Val: RecordVal::LookupWithDefault() returns IntrusivePtr
Type: RecordType::FieldDefault() returns IntrusivePtr
Val: TableVal::Delete() returns IntrusivePtr
Type: base_type() returns IntrusivePtr
Type: init_type() returns IntrusivePtr
Type: merge_types() returns IntrusivePtr
Type: use class IntrusivePtr in VectorType
Type: use class IntrusivePtr in EnumType
Type: use class IntrusivePtr in FileType
Type: use class IntrusivePtr in TypeDecl
Type: make TypeDecl `final` and the dtor non-`virtual`
Type: use class IntrusivePtr in TypeType
Type: use class IntrusivePtr in FuncType
...
The Zeek code base has very inconsistent #includes. Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed. Another side effect was a lot of header
bloat which slows down the build.
First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.
After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations. In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.
This patch speeds up the build by 19%, because each compilation unit
gets smaller. Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):
Before this patch:
3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps
After this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
This also installs symlinks from "zeek" and "bro-config" to a wrapper
script that prints a deprecation warning.
The btests pass, but this is still WIP. broctl renaming is still
missing.
#239
Majority of PLists are now created as automatic/stack objects,
rather than on heap and initialized either with the known-capacity
reserved upfront or directly from an initializer_list (so there's no
wasted slack in the memory that gets allocated for lists containing
a fixed/known number of elements).
Added versions of the ConnectionEvent/QueueEvent methods that take
a val_list by value.
Added a move ctor/assign-operator to Plists to allow passing them
around without having to copy the underlying array of pointers.
Signatures using an eval-condition that had no return value caused a
segmentation fault. This fix just returns false in this case, as it is
done for an interpreter error.
- Introducing analyzer::<protocol> namespaces.
- Moving protocol-specific events out of events.bif into analyzer/protocol/<protocol>/events.bif
- Moving ARP over (even though it's not an actual analyzer).
- Moving NetFlow over (even though it's not an actual analyzer).
- Moving MIME over (even though it's not an actual analyzer).
This is a larger internal change that moves the analyzer
infrastructure to a more flexible model where the available analyzers
don't need to be hardcoded at compile time anymore. While currently
they actually still are, this will in the future enable external
analyzer plugins. For now, it does already add the capability to
dynamically enable/disable analyzers from script-land, replacing the
old Analyzer::Available() methods.
There are three major parts going into this:
- A new plugin infrastructure in src/plugin. This is independent
of analyzers and will eventually support plugins for other parts
of Bro as well (think: readers and writers). The goal is that
plugins can be alternatively compiled in statically or loadead
dynamically at runtime from a shared library. While the latter
isn't there yet, there'll be almost no code change for a plugin
to make it dynamic later (hopefully :)
- New analyzer infrastructure in src/analyzer. I've moved a number
of analyzer-related classes here, including Analyzer and DPM;
the latter now renamed to Analyzer::Manager. More will move here
later. Currently, there's only one plugin here, which provides
*all* existing analyzers. We can modularize this further in the
future (or not).
- A new script interface in base/framework/analyzer. I think that
this will eventually replace the dpm framework, but for now
that's still there as well, though some parts have moved over.
I've also remove the dpd_config table; ports are now configured via
the analyzer framework. For exmaple, for SSH:
const ports = { 22/tcp } &redef;
event bro_init() &priority=5
{
...
Analyzer::register_for_ports(Analyzer::ANALYZER_SSH, ports);
}
As you can see, the old ANALYZER_SSH constants have more into an enum
in the Analyzer namespace.
This is all hardly tested right now, and not everything works yet.
There's also a lot more cleanup to do (moving more classes around;
removing no longer used functionality; documenting script and C++
interfaces; regression tests). But it seems to generally work with a
small trace at least.
The debug stream "dpm" shows more about the loaded/enabled analyzers.
A new option -N lists loaded plugins and what they provide (including
those compiled in statically; i.e., right now it outputs all the
analyzers).
This is all not cast-in-stone yet, for some things we need to see if
they make sense this way. Feedback welcome.
* origin/fastpath:
Update documentation for builtin types
Adding an identifier to the SMTP blocklist notices for duplicate suppression.
Doc fixes for signature 'eval' conditions.
Remove orphaned unit tests.
Add type checking for signature 'eval' condition functions.
Otherwise functions could be called with a mismatching argument list
and cause a crash at run-time. The incorrect function type is now
reported at parse-time.
Currently, a lot of interpreter runtime errors, such as an access to
an unset optional record field, cause Bro to abort with an internal
error. This is an experimental branch that turns such errors into
non-fatal runtime errors by internally raising exceptions. These are
caught upstream and processing continues afterwards.
For now, not many errors actually raise exceptions (the example above
does though). We'll need to go through them eventually and adapt the
current Internal() calls (and potentially others). More generally, at
some point we should cleanup the interpreter error handling (unifying
errors reported at parse- and runtime; and switching to exceptions for
all Expr/Stmt/Vals). But that's a larger change and left for later.
The main question for now is if this code is already helpful enough to
go into 2.0. It will quite likely prevent a number of crashes due to
script errors.
The Logger class is now in charge of reporting all errors, warnings,
informational messages, weirds, and syslogs. All other components
route their messages through the global bro_logger singleton.
The Logger class comes with these reporting methods:
void Message(const char* fmt, ...);
void Warning(const char* fmt, ...);
void Error(const char* fmt, ...);
void FatalError(const char* fmt, ...); // Terminate Bro.
void Weird(const char* name);
[ .. some more Weird() variants ... ]
void Syslog(const char* fmt, ...);
void InternalWarning(const char* fmt, ...);
void InternalError(const char* fmt, ...); // Terminates Bro.
See Logger.h for more information on these.
Generally, the reporting now works as follows:
- All non-fatal message are reported in one of two ways:
(1) At startup (i.e., before we start processing packets),
they are logged to stderr.
(2) During processing, they turn into events:
event log_message%(msg: string, location: string%);
event log_warning%(msg: string, location: string%);
event log_error%(msg: string, location: string%);
The script level can then handle them as desired.
If we don't have an event handler, we fall back to
reporting on stderr.
- All fatal errors are logged to stderr and Bro terminates
immediately.
- Syslog(msg) directly syslogs, but doesn't do anything else.
The three main types of messages can also be generated on the
scripting layer via new Log::* bifs:
Log::error(msg: string);
Log::warning(msg: string);
Log::message(msg: string);
These pass through the bro_logger as well and thus are handled in the
same way. Their output includes location information.
More changes:
- Removed the alarm statement and the alarm_hook event.
- Adapted lots of locations to use the bro_logger, including some
of the messages that were previously either just written to
stdout, or even funneled through the alarm mechanism.
- No distinction anymore between Error() and RunTime(). There's
now only one class of errors; the line was quite blurred already
anyway.
- util.h: all the error()/warn()/message()/run_time()/pinpoint()
functions are gone. Use the bro_logger instead now.
- Script errors are formatted a bit differently due to the
changes. What I've seen so far looks ok to me, but let me know
if there's something odd.
Notes:
- The default handlers for the new log_* events are just dummy
implementations for now since we need to integrate all this into
the new scripts anyway.
- I'm not too happy with the names of the Logger class and its
instance bro_logger. We now have a LogMgr as well, which makes
this all a bit confusing. But I didn't have a good idea for
better names so I stuck with them for now.
Perhaps we should merge Logger and LogMgr?
