BIT-1098
* origin/topic/jsiwek/broxygen:
Fix Broxygen-related compile errors.
Add a Broxygen coverage test.
Internal Broxygen organization/documentation/polish.
Add unit tests for Broxygen config file targets.
Change Broxygen config file format.
Broxygen doc-related test updates. Fix two regressions.
A couple documentation fixes.
Integrate new Broxygen functionality into Sphinx.
Implement majority of Broxygen features delegated to Bro.
Broxygen can now read a config file specifying particular targets.
Remove unneeded Broxygen comments in scan.bro.
Replace safe_basename/safe_dirname w/ SafeBasename/SafeDirname.
Add BIF interface for retrieving comments/docs.
Quick optimization to Broxygen doc gathering.
Flesh out Broxygen doc-gathering skeleton.
Refactor search_for_file() util function.
Initial skeleton of new Broxygen infrastructure.
May fix a sporadic failure, in which case it was just not getting enough
time to run or lookup_hostname() taking longer than ~3 seconds. Else,
the new output should give more hints on what's going wrong. In any
case, termination conditions for the test are now explicit.
Slice ranges were not correctly determined for negative indices and also
off by one in general (included one more element at the end of the
substring than what actually matched the index range).
It's now equivalent to Python slice notation. Accessing a string at
a single index is also the same as Python except that an out-of-range
index returns an empty string instead of throwing an expection.
Broccoli clients can only recv events w/ vectors for now.
Also changed ordering of Bro type tag enum -- the addition of opaque
types changed the value of the vector type, making broccoli.h's
definition out of sync. Probably could have just changed broccoli's
definition, but seems more correct to go back to using the same value
for vectors as they were before opaques. It's also better in case
there's some other location I'm not aware of where the values are
replicated.
I got rid of the earlier separate InterpreterPlugin class. Instead
Plugin now has a set of virtual methods HookSomething()... that
plugins can override. For efficiency purposes, they however need to
register first that they are interested in a hook, otherwise the
virtual method will never be called. The idea is to extend the set of
hooks over time as we figure out what's useful.
This is a checkpoint commit that's essentially untested and probably
broken. It compiles, though.
This is essentially the code from the dynamic-plugin branch except for
some pieces that I have split out into separate, earlier commits.
I'm going to updatre things in this branch going forward.
- Enable manager to associate analyzers with a MIME type. With that,
one can now say enable all analyzers for, e.g., "image/gif". This is
exposed to script-land as
Files::add_analyzers_for_mime_type(f: fa_file, mtype: string)
For MIME types identified via libmagic, this happens automatically
(via the file_new() handler in files/main.bro).
- Extend the analyzer API to better match that of protocol analyzers:
- Adding unique analyzer IDs so that we can refer to instances
from script-land.
- Adding subtypes to Components so that a single analyzer
implementation can support different types of analyzers
internally.
- Add an analyzer method SetTag() that allows to set the tag after
construction.
- Adding Init() and Done() methods for consistency with what other
classes offer.
- Add debug logging to the file_analysis stream.
TODO: test cases missing for the new script-land functionality.
in an easily readable form.
This is for debugging purposes, obviously.
Example, including only SMTP events:
> bro -r smtp.trace misc/dump-events.bro DumpEvents::include=/smtp/
[...]
1254722768.219663 smtp_reply
[0] c: connection = [id=[orig_h=10.10.1.4, orig_p=1470/tcp, resp_h=74.53.140.153, [...]
[1] is_orig: bool = F
[2] code: count = 220
[3] cmd: string = >
[4] msg: string = xc90.websitewelcome.com ESMTP Exim 4.69 #1 Mon, 05 Oct 2009 01:05:54 -0500
[5] cont_resp: bool = T
1254722768.219663 smtp_reply
[0] c: connection = [id=[orig_h=10.10.1.4, orig_p=1470/tcp, resp_h=74.53.140.153, [...]
[1] is_orig: bool = F
[2] code: count = 220
[3] cmd: string = >
[4] msg: string = We do not authorize the use of this system to transport unsolicited,
[5] cont_resp: bool = T
[...]
Fixed typos in documentation of hexstr_to_bytestring.
Also added documentation that was missing for function parameters
and return values of other BIFs.
- Move notice index wrapper doc to doc/script-reference -- doc/scripts
no longer contains any static documentation because that location
will be managed by Bro to generate per-script docs.
- :doc: references for generated per-script docs now need the ".bro"
suffix. (IMO this is better since it directly mirrors the actual
script's file name and can't be confused w/ a package).
Add a "broxygen" domain Sphinx extension w/ directives to allow
on-the-fly documentation to be generated w/ Bro and included in files.
This means all autogenerated reST docs are now done by Bro. The odd
CMake/Python glue scipts which used to generate some portions are now
gone. Bro and the Sphinx extension handle checking for outdated docs
themselves.
Parallel builds of `make doc` target should now work (mostly because
I don't think there's any tasks that can be done in parallel anymore).
Overall, this seems to simplify things and make the Broxygen-generated
portions of the documentation visible/traceable from the main Sphinx
source tree. The one odd thing still is that per-script documentation
is rsync'd in to a shadow copy of the Sphinx source tree within the
build dir. This is less elegant than using the new broxygen extension
to make per-script docs, but rsync is faster and simpler. Simpler as in
less code because it seems like, in the best case, I'd need to write a
custom Sphinx Builder to be able to get that to even work.
Segmentation fault caused by accessing fields with pos which is
one-based for setting SQLite field values. Fix is to simply subtract one
from pos. Discovered when trying to store HTTP traffic to a SQLite
database with the following Bro script:
event bro_init() {
local filter: Log::Filter = [
$name = "sqlite",
$path = "http",
$config = table(["tablename"] = "http_logs"),
$writer = Log::WRITER_SQLITE
];
Log::add_filter(HTTP::LOG, filter);
}
* origin/fastpath:
Fix typos in sumstats doc and update doc test
Update docs and tests for a recent change to detect-MHR.bro
Update tests and baselines for sumstats docs
