Great work, and great documentation!
I'm getting one test failure with
scripts.base.frameworks.netcontrol.catch-and-release-cluster Going
ahead and commiting, Jenkins will show the details I assume.
BIT-1584 #merged
* origin/topic/johanna/netcontrol-improvements:
SMTP does not need to pull in the notice framework.
Write NetControl framework documentation.
Use NetControl for ACTION_DROP of notice framework.
NetControl: slightly update catch and release logging
NetControl: fix several small logging issues
NetControl: more catch and release logging and cluster fix
NetControl: rewrite catch and release and small fixes.
NetControl: find_rules_subnet works in cluster mode
NetControl: fix acld whitelist command
NetControl: add rule exists as state besides added and failure.
NetControl: Suppress duplicate "plugin activated" messages.
NetControl: make new broker plugin options accessible
NetControl: add predicates to broker plugin
This caused test baseline changes in one of the test: notice now ties in
netcontrol due to ACTION_DROP. Catch and release uses the new_connection
event, which was not before triggered and can cause uids to be generated
for connections that are not usually assigned uids in bare mode.
One tweak: I made ts optional and set it to network_time() if not given.
BIT-1578 #merged
* origin/topic/johanna/bit-1578:
Weird: fix potential small issue when ignoring duplicates
Rewrite weird logging.
BIT-1594 #merged
* origin/topic/johanna/rawleak:
Exec: fix reader cleanup when using read_files
Raw Writer: First step - make code more c++11-y, remove raw pointers.
- SMTP protocol headers now do some minimal parsing to clean up
email addresses.
- New function named split_mime_email_addresses to take MIME headers
and get addresses split apart but including the display name.
- Update tests.
Wen using read_files, the Exec framework called Input::remove on the
wrong input stream: it always got called on the input stream of the
execution, not on the input stream of the current file that was being
read.
This lead to threads never being closed and file handles being kept open
until Bro is closed. This means that before this patch, every time
ActiveHTTP is used, a thread stays around and several file handles are
used.
In all versions so far, the identifier string that was used for
comparisons might have been different from the identifier string that
was added (when certain notices are used).
This commit rewrites the way that weirds are logged and fixes a number
of issues on the way. Most prominently, flow weirds now actually log
information about the flow that they occur in (before this change, they
only logged the name of the weird, which is only marginally helpful).
Besides restructuring how weird logging works internally, weirds can now
also be generated by calling Weird::weird with the info record directly,
allowing more fine-granular passing of information. This is e.g. used
for DNS weirds, which do not have the connection record available any
more when they are generated (before data like the connection ID was
just not logged in these instances).
Addresses BIT-1578
We now extract email addresses in the fields that one would expect
to contain addresses. This makes further downstream processing of
these fields easier like log analysis or using these fields in the
Intel framework. The primary downside is that any other content
in these fields is no longer available such as full name and any
group information. I believe the simplification of the content in
these fields is worth the change.
Added "cc" to the script that feeds information from SMTP into the
Intel framework.
A new script for email handling utility functions has been created
as a side effect of these changes.
This changes the HTTP log format slightly but shouldn't mess
up anything that anyone was doing because the old "filename"
field was never actually filled out. Tests are updated as well.
