- Formatters have been abstracted similarly to readers and writers now.
- The Ascii writer has a new option for writing out logs as JSON.
- The Ascii writer now has all options availble as per-filter
options as well as global.
* origin/topic/seth/dns-srv-fix:
No longer accidentally attempting to parse NBSTAT RRs as SRV RRs.
Fix DNS SRV responses and a small issue with NBNS queries and label length.
BIT-1147 #merged
It didn't swap address values right and also didn't consider that
analyzers might be scheduled for the new connection tuple.
Issues were reported by Kevin McMahon, thanks.
- Improve or just remove some file magic signatures ported from libmagic
that were too general and matched incorrectly too often.
- Fix MHR script's use of fa_file$mime_type before checking if it's
initialized. It may be uninitialized if no signatures match.
- The "fa_file" record now contains a "mime_types" field that contains
all magic signatures that matched the file content (where the
"mime_type" field is just a shortcut for the strongest match).
The NetBios name service RFC (1002) specified NBSTAT (NetBios Status)
resource records to have identifier 0x0021. The DNS SRV RFC specified
SRV records to have identifier 33. Unfortunately those are the
same number. :)
We now check the resp port to handle this situation better so that
we won't be attempting to parse NBSTAT records as SRV (which
causes several weird messages).
- DNS SRV responses never had the code written to actually
generate the dns_SRV_reply event. Adding this required
extending the event a bit to add extra information. SRV responses
now appear in the dns.log file correctly.
- Fixed an issue where some Microsoft NetBIOS Name Service lookups
would exceed the max label length for DNS and cause an incorrect
"DNS_label_too_long" weird.
Put some methods in file_analysis::Manager that can perform the
matching process and return MIME type results. Also helps to
centralize the management/re-use of a signature matcher object.
That means that, for example, connections that are terminated with an alert during the
handshake never appear in the ssl.log.
This patch changes this behavior - now all ssl connections that fire any event are logged.
The protocol confirmation of the ssl analyzer is moved to the client_hello instead to
the server hello. Furthermore, an additional field is added to ssl.log, which indicates
if a connection has been established or not (which probably indicates a handshake problem).
Notable changes:
- libmagic is no longer used at all. All MIME type detection is
done through new Bro signatures, and there's no longer a means to get
verbose file type descriptions (e.g. "PNG image data, 1435 x 170").
The majority of the default file magic signatures are derived
from the default magic database of libmagic ~5.17.
- File magic signatures consist of two new constructs in the
signature rule parsing grammar: "file-magic" gives a regular
expression to match against, and "file-mime" gives the MIME type
string of content that matches the magic and an optional strength
value for the match.
- Modified signature/rule syntax for identifiers: they can no longer
start with a '-', which made for ambiguous syntax when doing negative
strength values in "file-mime". Also brought syntax for Bro script
identifiers in line with reality (they can't start with numbers or
include '-' at all).
- A new Built-In Function, "file_magic", can be used to get all
file magic matches and their corresponding strength against a given
chunk of data
- The second parameter of the "identify_data" Built-In Function
can no longer be used to get verbose file type descriptions, though it
can still be used to get the strongest matching file magic signature.
- The "file_transferred" event's "descr" parameter no longer
contains verbose file type descriptions.
- The BROMAGIC environment variable no longer changes any behavior
in Bro as magic databases are no longer used/installed.
- Reverted back to minimum requirement of CMake 2.6.3 from 2.8.0
(it's back to being the same requirement as the Bro v2.2 release).
The bump was to accomodate building libmagic as an external project,
which is no longer needed.
Addresses BIT-1143.
SSL::Info now holds a reference to Files::Info instead of the
fa_files record.
Everything should work now, if everyone thinks that the interface is
ok I will update the test baselines in a bit.
addresses BIT-953, BIT-760
work fine now.
Todo:
* update all baselines
* fix the circular reference to the fa_file structure I introduced :)
Sadly this does not seem to be entirely straightforward.
addresses BIT-953, BIT-760
* topic/robin/http-connect:
HTTP fix for output handlers.
Expanding the HTTP methods used in the signature to detect HTTP traffic.
Updating submodule(s).
Fixing removal of support analyzers, plus some tweaking and cleanup of CONNECT code.
HTTP CONNECT proxy support.
BIT-1132 #merged
chain. This means that it is now possible to get information about the
root-certificates that were used to secure a connection.
Intermediate commit before changing the script interface again.
addresses BIT-953, BIT-760
CONNECT code.
Removal of support analyzers was broken. The code now actually doesn't
delete them immediately anymore but instead just flags them as
disabled. They'll be destroyed with the parent analyzer later.
Also includes a new leak tests exercising the CONNECT code.
Lines starting # with '#' will be ignored, and an empty message aborts
the commit. # On branch topic/robin/http-connect # Changes to be
committed: # modified: scripts/base/protocols/http/main.bro #
modified: scripts/base/protocols/ssl/consts.bro # modified:
src/analyzer/Analyzer.cc # modified: src/analyzer/Analyzer.h #
modified: src/analyzer/protocol/http/HTTP.cc # new file:
testing/btest/core/leaks/http-connect.bro # modified:
testing/btest/scripts/base/protocols/http/http-connect.bro # #
Untracked files: # .tags # changes.txt # conn.log # debug.log # diff #
mpls-in-vlan.patch # newfile.pcap # packet_filter.log # reporter.log #
src/PktSrc.cc.orig # weird.log #
This changes the internal type that is used to signal that a vector
is unspecified from any to void.
I tried to verify that the behavior of Bro is still the same. After
a lot of playing around, I think everything still should worl as before.
However, it might be good for someone to take a look at this.
addresses BIT-1144
This supports parsing of SNMPv1 (RFC 1157), SNMPv2 (RFC 1901/3416), and
SNMPv2 (RFC 3412). An event is raised for each SNMP PDU type, though
there's not currently any event handlers for them and not a default
snmp.log either. However, simple presence of SNMP is currently visible
now in conn.log service field and known_services.log.
* origin/topic/vladg/socks_fix:
Fix misidentification of SOCKS traffic. Traffic that had a certain bytestring would get incorrectly identified as SOCKS. This seemed to happen a lot with DCE/RPC traffic.
Tweaked the error message and added SetSkip() to the "reserved" case
as well.
BIT-1130 #merged
* origin/topic/jsiwek/dns-improvements:
Rewrite DNS state tracking which matches queries and replies.
Change dns.log to include only standard DNS queries.
Improve DNS analysis.
* origin/topic/jsiwek/http-file-id-caching:
Revert use of HTTP file ID caching for gaps range request content.
Extend file analysis API to allow file ID caching, adapt HTTP to use it.
BIT-1125 #merged
This allows an analyzer to either provide file IDs associated with some
file content or to cache a file ID that was already determined by
script-layer logic so that subsequent calls to the file analysis
interface can bypass costly detours through script-layer. This can
yield a decent performance improvement for analyzers that are able to
take advantage of it and deal with streaming content (like HTTP).
* origin/topic/jsiwek/tcp-improvements:
Fix file_over_new_connection event to trigger when entire file is missed.
Improve TCP connection size reporting for half-open connections.
Improve gap reporting in TCP connections that never see data.
Improve TCP FIN retransmission handling.
BIT-1119
