Great work, and great documentation!
I'm getting one test failure with
scripts.base.frameworks.netcontrol.catch-and-release-cluster Going
ahead and commiting, Jenkins will show the details I assume.
BIT-1584 #merged
* origin/topic/johanna/netcontrol-improvements:
SMTP does not need to pull in the notice framework.
Write NetControl framework documentation.
Use NetControl for ACTION_DROP of notice framework.
NetControl: slightly update catch and release logging
NetControl: fix several small logging issues
NetControl: more catch and release logging and cluster fix
NetControl: rewrite catch and release and small fixes.
NetControl: find_rules_subnet works in cluster mode
NetControl: fix acld whitelist command
NetControl: add rule exists as state besides added and failure.
NetControl: Suppress duplicate "plugin activated" messages.
NetControl: make new broker plugin options accessible
NetControl: add predicates to broker plugin
This caused test baseline changes in one of the test: notice now ties in
netcontrol due to ACTION_DROP. Catch and release uses the new_connection
event, which was not before triggered and can cause uids to be generated
for connections that are not usually assigned uids in bare mode.
One tweak: I made ts optional and set it to network_time() if not given.
BIT-1578 #merged
* origin/topic/johanna/bit-1578:
Weird: fix potential small issue when ignoring duplicates
Rewrite weird logging.
- SMTP protocol headers now do some minimal parsing to clean up
email addresses.
- New function named split_mime_email_addresses to take MIME headers
and get addresses split apart but including the display name.
- Update tests.
This commit rewrites the way that weirds are logged and fixes a number
of issues on the way. Most prominently, flow weirds now actually log
information about the flow that they occur in (before this change, they
only logged the name of the weird, which is only marginally helpful).
Besides restructuring how weird logging works internally, weirds can now
also be generated by calling Weird::weird with the info record directly,
allowing more fine-granular passing of information. This is e.g. used
for DNS weirds, which do not have the connection record available any
more when they are generated (before data like the connection ID was
just not logged in these instances).
Addresses BIT-1578
We now extract email addresses in the fields that one would expect
to contain addresses. This makes further downstream processing of
these fields easier like log analysis or using these fields in the
Intel framework. The primary downside is that any other content
in these fields is no longer available such as full name and any
group information. I believe the simplification of the content in
these fields is worth the change.
Added "cc" to the script that feeds information from SMTP into the
Intel framework.
A new script for email handling utility functions has been created
as a side effect of these changes.
This changes the HTTP log format slightly but shouldn't mess
up anything that anyone was doing because the old "filename"
field was never actually filled out. Tests are updated as well.
* origin/topic/dnthayer/doc-improvements:
Fix some scripting tutorial examples
Add some missing Bro script documentation
Fix more "make doc" warnings
Fix some "make doc" warnings
Improve documentation of the "for" statement
Add more documentation to sumstats framework scripts
Update traffic per core estimate in the cluster doc
Update documentation for DNS "Z" field
Improve documentation of Bro script statements
BIT-1520 #merged
* martin/topic/fox/rfb:
Fixed issue in state machine
Some styling tweaks
Implement protocol confirmation
Analyzer and bro script for RFB protocol (VNC)
* <seth> I also applied a bit of clean up to the base
script to make it match other scripts better and
updated tests.
Do not set the service field in the bro script but
use the protocol confirmation paradigm.
Protocol is considered confirmed if both a
succesful client and server banner have been
parsed.
This analyzer parses the Remote Frame Buffer
protocol, usually referred to as the 'VNC protocol'.
It supports several dialects (3.3, 3.7, 3.8) and
also handles the Apple Remote Desktop variant.
It will log such facts as client/server versions,
authentication method used, authentication result,
height, width and name of the shared screen.
It also includes two testcases.
Todo: Apple Remote Desktop seems to have some
bytes prepended to the screen name. This is
not interepreted correctly.
The server-reported file size was being collected poorly and if
a file name had a number in it, that was reported as the file
size instead of the actual size.
A new test is included to avoid reintroducing the problem.
* 'patch-4' of https://github.com/aeppert/bro:
(BIT-1545) Add "disable_analyzer_after_detection" en lieu of "skip_processing_after_detection"
I also removed the old disable_analyzer_after_detection option
completely - if someone wants that, they can just catch the event
themselves and call skip_further_processing.
I also adjusted the ssh test case to contain conn.log to prevent
re-addition of this problem in the future.
BIT-1545 #merged
The default of "skip_processing_after_detection" is confusing and causes conn.log to not be written as one would assume, plus the counters are not incremented and thus some kinds of potential detections are short-changed. I propose adding "disable_analyzer_after_detection" which would react, on the surface, the same way by disabling the SSH analyzer, but allowing conn.log to be written appropriately.
According to RFC 2535, RFC 3655, and RFC 4035, the Z field has been
partitioned into three 1-bit fields. Therefore, we cannot claim in
the documentation that it always has the value zero.
- Daniel Guerra pointed out a type issue for SIP
request and response code length fields which is now
corrected.
- Some redundant code was removed.
- if/else tree modified to use switch instead.
The alert in this case is caused by the server name in the SNI not being
recognized by the server, which triggers an alert. Since the server is
an apache, and this might happen reasonably often, the new signature
allows one TLS alert before the server hello is expected.
* 'topic/jgras/base64-logging' of https://github.com/J-Gras/bro:
Update calls of Base64 functions.
Refactoring of Base64 functions.
I've removed the additional bif for encoding with a connection, as I'm
not sure there's much of a use case for it; we can always add it back
later if it turns out there is. I've also renamed
decode_base64_intern() to decode_base64_conn() to be a bit more
explicit about the difference.
- Q (MULTI_FLAG_PKT) was not in the documentation for the history field.
- I (FIN_RST_PKT) was documented incorrectly. It was documented as a
SYN+RST, when it actually represents a FIN+RST.
The new documentation was derived from:
d3f513f/src/analyzer/protocol/tcp/TCP.cc (L493)
Addresses BIT-1466
Contributed by Martina Balint in https://github.com/bro/bro/pull/39.
(I didn't merge the github branch, as that has some more stuff in its
history. Instead I applied the single-line change directly.)
Base64 encoding-errors during authentication in POP3 analyzer,
authentication in FTP analyzer (using GSI) and basic
authentication on HTTP will be logged to Weird.
