Broker had changed the semantics of remote logging: it sent over the
original Bro record containing the values to be logged, which on the
receiving side would then pass through the logging framework normally,
including triggering filters and events. The old communication system
however special-cases logs: it sends already processed log entries,
just as they go into the log files, and without any receiver-side
filtering etc. This more efficient as it short-cuts the processing
path, and also avoids the more expensive Val serialization. It also
lets the sender determine the specifics of what gets logged (and how).
This commit changes Broker over to now use the same semantics as the
old communication system.
TODOs:
- The new Broker code doesn't have consistent #ifdefs yet.
- Right now, when a new log receiver connects, all existing logs
are broadcasted out again to all current clients. That doesn't so
any harm, but is unncessary. Need to add a way to send the
existing logs to just the new client.
Re-enable logging, now in policy because it probably is interesting to
no-one. We also only log ocsp replies.
Fix all tests.
Fix an issue where ocsp replies were added to the x.509 certificate
list.
With this change, we also parse signed certificate timestamps from OCSP
replies. This introduces a common base class between the OCSP and X509
analyzer, which now share a bit of common code. The event for signed
certificate timestamps is raised by both and thus renamed do:
x509_ocsp_ext_signed_certificate_timestamp
This is a tiny bit evil because it uses parts of the SSL protocol
analyzer in the X.509 certificate parser. Which is the fault of the
protocol, which replicates the functionality.
This event is the replacement for ssl_application_data, which is removed
in the same commit. It is more generic, containing more information than
ssl_application_dataand is raised for all SSL/TLS messages that are
exchanged before encryption starts.
It is used by Bro internally to determine when a TLS1.3 session has been
completely established. Apart from that, it can be used to, e.g.,
determine the record layer TLS version.
This exposes the record layer version of the fragment in addition to the
content type and the length. The ordering of the arguments in the event
is the same as the ordering in the protocol message (first type, then
version, then length).
This also includes a slight change to the analyzer, no longer calling
the generate function if the event is not used.
This is a small caveat to this implementation. The ethernet
header that is carried over the tunnel is ignored. If a user
tries to do MAC address logging, it will only show the MAC
addresses for the outer tunnel and the inner MAC addresses
will be stripped and not available anywhere.
This change adds compression methods to the ssl_client_hello event. It
not being included was an oversight from a long time ago.
This change means that the signature of ssl_client_hello changes
slightly and scripts will have to be adjusted; since this is a commonly
used event, the impact of it might be higher than usually for event
changes.
If connection flipping occured in Sessions.cc code (invoked e.g. when
the original SYN is missing), layer 2 flipping was not performed. This
change switches to always use the connection flipping code in Conn.cc
which performs the switch correctly.
- New fields: extracted_cutoff and extracted_size.
These fields will be null if the file isn't extracted.
- Extended the extraction test to test the files log too.
* origin/topic/seth/smb-auth-fixes:
Disable SMB2 error data parsing.
Removed some files that aren't being used.
Fix SMB tree connect handling.
Fix a small issue where DCE_RPC commands were improperly being logged.
SMB fixes and cleanup.
Including a test for raw NTLM in SMB
Updates for SMB auth handling from Martin van Hensbergen.
BIT-1721 #merged
SMB error handling improved. The analyzer isn't destroyed when a problem
is encoutered anymore. The flowbuffer in the parser is now flushed and
the analyzer is set to resync against an SMB command. This was needed
because there is some state about open files that is kept within the
parser itself which was being destroyed and that was causing analysis
after content gaps or parse errors to be faulty. The new mechanism
doesn't detroy the parser so parsing after gaps is improved.
DCE_RPC handling in SMB is improved in the edge case where a drive
mapping isn't seen. There is a new const named SMB::pipe_filenames
which is used as a heuristic for identifying "files" opened on named
pipe shares. If the share mapping type isn't known and a filename
in this set is found, the share type will change to "PIPE" by
generating an event named "smb_pipe_connect_heuristic". Reads and
writes to that file will be sent to the DCE_RPC analyzer instead of
to the files framework.
The concept of "unknown" share types has been removed due to the new
heuristic detection of share types.
Some general clean up of how the SMB cmd log is written and when.
* origin/topic/vladg/bit-1641:
Logic fix for ssh/main.bro when the auth status is indeterminate, and fix a test. Addresses BIT-1641.
Clean up the logic for ssh_auth_failed. Addresses BIT-1641
Update baselines for adding a field to ssh.log as part of BIT-1641
Script-land changes for BIT-1641.
Change SSH.cc to use ssh_auth_attempted instead of ssh_auth_failed. Addresses BIT-1641.
Revert "Fixing duplicate SSH authentication failure events."
Create new SSH events ssh_auth_attempt and ssh_auth_result. Add auth_attempts to SSH::Info. Address BIT-1641.
I extended the tests a bit and did some small cleanups. I also moved the
SSH events back to the global namespace for backwards compatibility and
for consistency (the way it was at the moment, some of them were global
some SSH::).
Furthermore, I fixed the ssh_auth_result result event, it was only
raised in the success case. ssh_auth_result is now also checked in the
testcases. I also have a suspicion that the intel integration never
really worked before.
BIT-1641 #merged
Filter out the python 2 "L" suffix on long integers (not valid in Python 3),
and change python 3 "object at" to "instance at" when printing
class instances.
BIT-1727 #merged
* origin/topic/johanna/tls13:
Better way to deal with overloaded Assign constructors.
A few tabbing fixes in TLS 1.3 support
TLS 1.3 support.
- DCE_RPC fragmentation handling returns!
- Fixed some general parsing issues
- Fixed an issue with the DCE_RPC signature not working for IPv6 connections.
