One tweak: I made ts optional and set it to network_time() if not given.
BIT-1578 #merged
* origin/topic/johanna/bit-1578:
Weird: fix potential small issue when ignoring duplicates
Rewrite weird logging.
- SMTP protocol headers now do some minimal parsing to clean up
email addresses.
- New function named split_mime_email_addresses to take MIME headers
and get addresses split apart but including the display name.
- Update tests.
This commit rewrites the way that weirds are logged and fixes a number
of issues on the way. Most prominently, flow weirds now actually log
information about the flow that they occur in (before this change, they
only logged the name of the weird, which is only marginally helpful).
Besides restructuring how weird logging works internally, weirds can now
also be generated by calling Weird::weird with the info record directly,
allowing more fine-granular passing of information. This is e.g. used
for DNS weirds, which do not have the connection record available any
more when they are generated (before data like the connection ID was
just not logged in these instances).
Addresses BIT-1578
We now extract email addresses in the fields that one would expect
to contain addresses. This makes further downstream processing of
these fields easier like log analysis or using these fields in the
Intel framework. The primary downside is that any other content
in these fields is no longer available such as full name and any
group information. I believe the simplification of the content in
these fields is worth the change.
Added "cc" to the script that feeds information from SMTP into the
Intel framework.
A new script for email handling utility functions has been created
as a side effect of these changes.
This changes the HTTP log format slightly but shouldn't mess
up anything that anyone was doing because the old "filename"
field was never actually filled out. Tests are updated as well.
* origin/topic/dnthayer/doc-improvements:
Fix some scripting tutorial examples
Add some missing Bro script documentation
Fix more "make doc" warnings
Fix some "make doc" warnings
Improve documentation of the "for" statement
Add more documentation to sumstats framework scripts
Update traffic per core estimate in the cluster doc
Update documentation for DNS "Z" field
Improve documentation of Bro script statements
BIT-1520 #merged
- NTLM Authentication failures over SMB2 are now marked as such in
the ntlm.log.
- Slightly updated filtering mechanism for DCE/RPC operations.
- Uncommented the atsvc file so it compiles now.
If only one side of a connection was seen, the ntlm.log
would indicate that the authentication failed. This has been
modified so that the success is listed as null since it's not
known whether or not the authentication was successful.
It can be inferred from continued SMB analysis though because
activity will continue taking place. I changed it though
because the log shouldn't assume more than what it sees.
- Fix an issue with svcctl uuid -> operation mapping.
- Add a heuristic to fill out the endpoint name in
case the original dce/rpc binding wasn't seen.
- Improve naming and code structure in the dce/rpc scripts.
* martin/topic/fox/rfb:
Fixed issue in state machine
Some styling tweaks
Implement protocol confirmation
Analyzer and bro script for RFB protocol (VNC)
* <seth> I also applied a bit of clean up to the base
script to make it match other scripts better and
updated tests.
Do not set the service field in the bro script but
use the protocol confirmation paradigm.
Protocol is considered confirmed if both a
succesful client and server banner have been
parsed.
This analyzer parses the Remote Frame Buffer
protocol, usually referred to as the 'VNC protocol'.
It supports several dialects (3.3, 3.7, 3.8) and
also handles the Apple Remote Desktop variant.
It will log such facts as client/server versions,
authentication method used, authentication result,
height, width and name of the shared screen.
It also includes two testcases.
Todo: Apple Remote Desktop seems to have some
bytes prepended to the screen name. This is
not interepreted correctly.
- Previously there was an (incorrect) assumption that a TCP conneciton
would only ever have one DCE_RPC binding. That assumption was
incorrect and with named pipes over SMB there can be multiple
concurrent DCE_RPC bindings. This commit fixes that assumption by
dynamically creating a new DCE_RPC analyzer whenever a new, unknown
binding is created.
- There is a crash fix in how string handling in the bind_ack message
was done.
- Named pipe handling over SMB1 is still not working quite right
and problems will show up with multiplexed DCE_RPC bindings.
- Looser coupling between these analyzers.
- New ntlm.log (still pretty early)
- Improved string handling for NTLM (convert UTF16 to UTF8)
- SMB2 analyzer now supports GSSAPI.
- Improved abstraction of DCE_RPC operations (still not finished)
- Lots of whitespace cleanup.
- Binding context versions are provided correctly now.
- RPC stubs are no longer provided and instead only the length is given.
- Some parsing problems related to handling the optional
auth fields in RPC messages are fixed. This removes some
dpd log messages I was seeing.
There were some cases where the log would be missing a field
or data wouldn't get sent to file analysis. At least some of
this is fixed now and I get confused a bit less when I look
at the logs now.
Also, I made the default handling "FILE" so that things like
FILE_UNKNOWN wouldn't show up in the logs so regularly. It's
technically correct that way, but it doesn't look good and it's
correct as FILE often enough that it make sense to make it the
default I think.
- It works with DCE/RPC over SMB1+2 now.
- Using named pipes in 1+2 and the transaction cmd in SMB1.
- Base scripts based on work by Josh Liburdi.
- New dce_rpc.log. Feedback on how to make this log more compact
and useful would be appreciated.
