- Backed out eTag changes. The real world is more complicated
than just using eTags to identify the same file.
- A bit of code simplication in the http base scripts.
- Test updates (more existing small problems were identified!).
-
Removed "file_mime_type" and "file_mime_types" event, replacing them
with a new event called "file_metadata_inferred". It has a record
argument of type "inferred_file_metadata", which contains the mime type
information that the earlier events used to supply. The idea here is
that future extensions to the record with new metadata will be less
likely to break user code than the alternatives (adding new events or
new event parameters).
Addresses BIT-1368.
- Lots of cleanup and expansion of XML match types.
- Signatures for ATOM and RSS (text/atom, text/rss).
- Improved SOAP signature.
- Improved text/cross-domain-policy signature
- Improved and expanded javascript matching a bit.
- Removed a lot of potentially problematic signatures (performance)
- Split out more signatures from libmagic.sig
- Added a signature for matching JSON. Seems to work ok.
- Signature for MPEGv4 audio.
- Expanded java applet signature.
- Improved PNG matching.
- Improved MP3 matching.
* origin/topic/seth/rdp: (31 commits)
Improved transition into SSL/TLS from RDP.
Fixes tests in RDP branch.
add a special case to the X509 code that deals with RDP certificates.
A few more changes to handling encryption in RDP.
Adds some comments and fixes a broxygen warning.
Fixes another optional part of an RDP unit.
Support RDP negotiation requests optionally and support zero length cookies.
Changed UTF-16 to UTF-8 conversion to be more lenient.
Fixed an issue with parse failure on an optional field.
Removing a stray printf from RDP analyzer.
Another big RDP update.
New script to add a field to rdp.log when the connection is upgraded to SSL.
Huge updates to the RDP analyzer from Josh Liburdi.
FreeRDP test trace showing SSL encryption -- RDP analyzer does not currently handle this and SSL analyzer does not identify it either
Wireshark test trace for native encryption -- generates a binpac error
Delete RDP-004.pcap
Delete nla_win7_win2k8r2.pcap
Update dpd.sig
Fixed typo
Added check for connection existence
...
BIT-1340 #merged
- Some scripts used wrong SSH module/namespace scoping on events.
- Fix outdated notice documentation related to SSH password guessing.
- Add a unit test for SSH pasword guessing notice.
I replaced a few strcmps with either calls to std::str.compare
or with the == operator of BroString.
Also changed two of the input framework tests that did not pass
anymore after the merge. The new SSH analyzer no longer loads the
scripts that let network time run, hence those tests failed because
updates were not propagated from the threads (that took a while
to find.)
* origin/topic/vladg/ssh: (25 commits)
SSH: Register analyzer for 22/tcp.
SSH: Add 22/tcp to likely_server_ports
SSH: Ignore encrypted packets by default.
SSH: Fix some edge-cases which created BinPAC exceptions
SSH: Add memleak btest
SSH: Update baselines
SSH: Added some more events for SSH2
SSH: Intel framework integration (PUBKEY_HASH)
Update baselines for new SSH analyzer.
Update SSH policy scripts with new events.
SSH: Add documentation
Refactoring ssh-protocol.pac:
SSH: Use the compression_algorithms const in another place.
Some cleanup and refactoring on SSH main.bro.
SSH: A bit of code cleanup.
Move SSH constants to consts.pac
SSH: Cleanup code style.
SSH: Fix some memleaks.
Refactored the SSH analyzer. Added supported for algorithm detection and more key exchange message types.
Add host key support for SSH1.
Add support for SSH1
Move SSH analyzer to new plugin architecture.
...
Conflicts:
scripts/base/protocols/ssh/main.bro
testing/btest/Baseline/core.print-bpf-filters/output2
testing/btest/Baseline/plugins.hooks/output
BIT-1344: #merged
* origin/topic/johanna/dtls:
a few more small script-level fixes
update test baselines
add a simple leak test for dtls
add signature for dtls client hello
Make the plugin structure more... legal.
Only force logging of SSL if it actually was the SSL analyzer that failed.
DTLS working.
Implement correct parsing of TLS record fragmentation.
Make handshake analyzer flow-based. This means we can feed data to it in chunks, which makes dealing with fragmentation a little bit more convenient.
When setting the SSL analyzer to fail, also stop processing data that already has been delivered to the analyzer, not just future data.
First step for a DTLS analyzer.
BIT-1347 #merged
Conflicts:
scripts/base/protocols/ssl/main.bro
testing/btest/Baseline/plugins.hooks/output
I added the $path to the create_stream() calls inside doc/ as well.
* origin/topic/jsiwek/bit-1324:
Allow logging filters to inherit default path from stream.
BIT-1324: #merged
This allows the path for the default filter to be specified explicitly
when creating a stream and reduces the need to rely on the default path
function to magically supply the path.
The default path function is now only used if, when a filter is added to
a stream, it has neither a path nor a path function already.
Adapted the existing Log::create_stream calls to explicitly specify a
path value.
Addresses BIT-1324
- Plain text now identified with BOMs for UTF8,16,32
(even though 16 and 32 wouldn't get identified as plain text, oh-well)
- X.509 certificates are now populating files.log with
the mime type application/pkix-cert.
- File signatures are split apart into file types
to help group and organize signatures a bit better.
- Normalized some FILE_ANALYSIS debug messages.
- Improved Javascript detection.
- Improved HTML detection.
- Removed a bunch of bad signatures.
- Merged a bunch of signatures that ultimately detected
the same mime type.
- Added detection for MS LNK files.
- Added detection for cross-domain-policy XML files.
- Added detection for SOAP envelopes.
The only thing that is missing is a signature to detect the protocol (it
has no well-known port).
Reassembly is kind of fidgety - at the moment we only support
re-assembling one simultaneous message per direction (which looking at
our test-traffic might not be a problem). And I am not quite sure if I
got all cases correct...
But - it works :)
This commit mostly does a lot of refactoring of the current SSL
analyzer, which is split into several parts.
The handshake protocol is completely taken out of the SSL analyzer and
was refactored into its own analyzer (called tls-handshake-analyzer).
This will also (finally) make it possible to deal with TLS record
fragmentation.
Apart from that, the parts of the SSL analyzer that are common to DTLS
were split into their own pac files. Both the SSL analyzer and the (very
basic, mostly nonfunctional) DTLS analyzer use their own pac files and
those shared pac files.
All SSL tests still pass after refactoring so I hope I did not break
anything too badly.
At the moment, we have two different modules in one directory and I
guess the way I am doing this might be an abuse of the system. It seems
to work though...
- New fields for certificate type, number of certificates,
if certificates are permanent on the server, and the selected
security protocol.
- Fixed some issues with X.509 certificate handling over RDP
(the event handler wasn't sufficiently constrained).
- Better detection of and transition into encrypted mode. No more
binpac parse failures from the test traces anymore!
- Some event name clean up and new events.
- X.509 Certificate chains are now handled correctly (was only grabbing
a single certificate).
- More data pulled into scriptland.
- Logs expanded with client screen resolution and desired color depth.
- Values in UTF-16 on the wire are converted to UTF-8 before being
sent to scriptland.
- If the RDP turns into SSL records, we now pass data that appears
to be SSL to the PIA analyzer.
- If RDP uses native encryption with X.509 certs we pass those
certs to the files framework and the base scripts pass them forward
to the X.509 analyzer.
- Lots of cleanup and adjustment to fit the documented protocol
a bit better.
- Cleaned up the DPD signatures.
- Moved to flowunit instead of datagram.
- Added tests.
