One tweak: I made ts optional and set it to network_time() if not given.
BIT-1578 #merged
* origin/topic/johanna/bit-1578:
Weird: fix potential small issue when ignoring duplicates
Rewrite weird logging.
- SMTP protocol headers now do some minimal parsing to clean up
email addresses.
- New function named split_mime_email_addresses to take MIME headers
and get addresses split apart but including the display name.
- Update tests.
In all versions so far, the identifier string that was used for
comparisons might have been different from the identifier string that
was added (when certain notices are used).
This commit rewrites the way that weirds are logged and fixes a number
of issues on the way. Most prominently, flow weirds now actually log
information about the flow that they occur in (before this change, they
only logged the name of the weird, which is only marginally helpful).
Besides restructuring how weird logging works internally, weirds can now
also be generated by calling Weird::weird with the info record directly,
allowing more fine-granular passing of information. This is e.g. used
for DNS weirds, which do not have the connection record available any
more when they are generated (before data like the connection ID was
just not logged in these instances).
Addresses BIT-1578
We now extract email addresses in the fields that one would expect
to contain addresses. This makes further downstream processing of
these fields easier like log analysis or using these fields in the
Intel framework. The primary downside is that any other content
in these fields is no longer available such as full name and any
group information. I believe the simplification of the content in
these fields is worth the change.
Added "cc" to the script that feeds information from SMTP into the
Intel framework.
A new script for email handling utility functions has been created
as a side effect of these changes.
This changes the HTTP log format slightly but shouldn't mess
up anything that anyone was doing because the old "filename"
field was never actually filled out. Tests are updated as well.
The expiration attribute expression is now evaluated for every use. Thus
later adjustments of the value (e.g. by redefining a const) will now
take effect. Values less than 0 will disable expiration.
Added a new BIF haversine_distance that computes distance between two
geographic locations.
Added a new Bro script function haversine_distance_ip that does the same
but takes two IP addresses instead of latitude/longitude. This function
requires that Bro be built with libgeoip.
The link-layer addresses are now part of the connection endpoints
following the originator-responder-pattern. The addresses are printed
with leading zeros. Additionally link-layer addresses are also extracted
for 802.11 plus RadioTap.
The ascii reader now accepts \r\n newlines without complaining.
Furthermore, the reader was slightly rewritten in a more c++11-y way,
removing all raw pointers from the class.
Addresses BIT-1198
I was hoping to report this right at startup through a static check
but turns out we don't have the right machinery in place for that.
That would need to be done after the AST has been finalized, but our
AST traversal code can't iterate over types. So instead I've changed
this so that it's still being reported at runtime but at least
doesn't crash Bro anymore.
Closes BIT-1597.
* origin/topic/johanna/fix-analyzer-addition:
DTLS: Use magix constant from rfc5389 for stun detection.
DTLS: Fix binpac bug with DTLSv1.2 client hellos
Forgot to remove debug output.
DTLS: Fix interaction with STUN
Fix the way that child analyzers are added.
BIT-1604 #merged
Now the DTLS analyzer cleanly skips all STUN messages; no warnings
should be logged to dpd.log and parsing should work flawlessly with
intermixed STUN messages.
By default, OS X 10.11 does not include openssl headers.
Since building a Bro plugin #includes Bro headers, which #include openssl
headers, we need to tell cmake to find these so that the compiler
can use them.
Scheduling a local event variable resulted in a global lookup instead of
evaluating the local variable. To prevent misunderstandings, this will
trigger an error now.
When receiving a remote log via broker, there was a bug that would
prevent a log from being written if the log record contained a field
without the &log attribute that was followed by a field with the &log
attribute.
Updated a test case to catch this error.
When Bro was compiled with broker disabled, then some Bro scripts
were referencing functions and types that were not defined. Fixed
by adding @ifdefs to several scripts. Removed one @ifdef because
it was causing several unit tests to fail.
Also fixed the @TEST-REQUIRES check in tests that rely on broker so
that such tests are skipped when broker is disabled.
(Cleaned up some code a little bit.)
* origin/topic/seth/stats-improvement:
Fixing tests for stats improvements
Rename the reporting interval variable for stats.
Removing more broken functionality due to changed stats apis.
Removing some references to resource_usage()
Removing Broker stats, it was broken and incomplete.
Fixing default stats collection interval to every 5 minutes.
Add DNS stats to the stats.log
Small stats script tweaks and beginning broker stats.
Continued stats cleanup and extension.
More stats collection extensions.
More stats improvements
Slight change to Mach API for collecting memory usage.
Fixing some small mistakes.
Updating the cmake submodule for the stats updates.
Fix memory usage collection on Mac OS X.
Cleaned up stats collection.
BIT-1581 #merged
* origin/topic/dnthayer/doc-improvements:
Fix some scripting tutorial examples
Add some missing Bro script documentation
Fix more "make doc" warnings
Fix some "make doc" warnings
Improve documentation of the "for" statement
Add more documentation to sumstats framework scripts
Update traffic per core estimate in the cluster doc
Update documentation for DNS "Z" field
Improve documentation of Bro script statements
BIT-1520 #merged
* origin/topic/johanna/xmpp-starttls:
XMPP: Add StartTLS event and update tests
Add xmpp dpd sig and fix a few parsing problems for connections that do not upgrade to TLS.
Add simple XMPP StartTLS analyzer.
BIT-1579 #merged
Some of the examples in the scripting tutorial were regularly getting
out of sync with the base scripts (because the line numbering would need
to be updated). Fixed this maintenance burden by using small example
scripts instead of actual Bro scripts. These small example scripts
do not need to be kept in sync with the bro base scripts.
