* origin/topic/seth/more-file-type-ident-fixes:
File API updates complete.
Fixes for file type identification.
API changes to file analysis mime type detection.
Make HTTP 206 reassembly require ETags by default.
More file type identification improvements
Fix an issue with files having gaps before the bof_buffer is filled.
Fix an issue with packet loss in http file reporting.
Adding WOFF fonts to file type identification.
Extended JSON matching and added OCSP responses.
Another large signature update.
More signature updates.
Even more file type ident clean up.
Lots of fixes for file type identification.
BIT-1368 #merged
- Backed out eTag changes. The real world is more complicated
than just using eTags to identify the same file.
- A bit of code simplication in the http base scripts.
- Test updates (more existing small problems were identified!).
-
Removed "file_mime_type" and "file_mime_types" event, replacing them
with a new event called "file_metadata_inferred". It has a record
argument of type "inferred_file_metadata", which contains the mime type
information that the earlier events used to supply. The idea here is
that future extensions to the record with new metadata will be less
likely to break user code than the alternatives (adding new events or
new event parameters).
Addresses BIT-1368.
- Lots of cleanup and expansion of XML match types.
- Signatures for ATOM and RSS (text/atom, text/rss).
- Improved SOAP signature.
- Improved text/cross-domain-policy signature
- Improved and expanded javascript matching a bit.
- Removed a lot of potentially problematic signatures (performance)
- Split out more signatures from libmagic.sig
- Added a signature for matching JSON. Seems to work ok.
- Signature for MPEGv4 audio.
- Expanded java applet signature.
- Improved PNG matching.
- Improved MP3 matching.
I replaced a few strcmps with either calls to std::str.compare
or with the == operator of BroString.
Also changed two of the input framework tests that did not pass
anymore after the merge. The new SSH analyzer no longer loads the
scripts that let network time run, hence those tests failed because
updates were not propagated from the threads (that took a while
to find.)
* origin/topic/vladg/ssh: (25 commits)
SSH: Register analyzer for 22/tcp.
SSH: Add 22/tcp to likely_server_ports
SSH: Ignore encrypted packets by default.
SSH: Fix some edge-cases which created BinPAC exceptions
SSH: Add memleak btest
SSH: Update baselines
SSH: Added some more events for SSH2
SSH: Intel framework integration (PUBKEY_HASH)
Update baselines for new SSH analyzer.
Update SSH policy scripts with new events.
SSH: Add documentation
Refactoring ssh-protocol.pac:
SSH: Use the compression_algorithms const in another place.
Some cleanup and refactoring on SSH main.bro.
SSH: A bit of code cleanup.
Move SSH constants to consts.pac
SSH: Cleanup code style.
SSH: Fix some memleaks.
Refactored the SSH analyzer. Added supported for algorithm detection and more key exchange message types.
Add host key support for SSH1.
Add support for SSH1
Move SSH analyzer to new plugin architecture.
...
Conflicts:
scripts/base/protocols/ssh/main.bro
testing/btest/Baseline/core.print-bpf-filters/output2
testing/btest/Baseline/plugins.hooks/output
BIT-1344: #merged
This allows the path for the default filter to be specified explicitly
when creating a stream and reduces the need to rely on the default path
function to magically supply the path.
The default path function is now only used if, when a filter is added to
a stream, it has neither a path nor a path function already.
Adapted the existing Log::create_stream calls to explicitly specify a
path value.
Addresses BIT-1324
- Plain text now identified with BOMs for UTF8,16,32
(even though 16 and 32 wouldn't get identified as plain text, oh-well)
- X.509 certificates are now populating files.log with
the mime type application/pkix-cert.
- File signatures are split apart into file types
to help group and organize signatures a bit better.
- Normalized some FILE_ANALYSIS debug messages.
- Improved Javascript detection.
- Improved HTML detection.
- Removed a bunch of bad signatures.
- Merged a bunch of signatures that ultimately detected
the same mime type.
- Added detection for MS LNK files.
- Added detection for cross-domain-policy XML files.
- Added detection for SOAP envelopes.
Also changed asynchronous data store query code a bit; trying to make
memory management and handling of corner cases a bit clearer (former
maybe could still be better, but I need to lookup queries by memory
address to associate response cookies to them, and so wrapping pointers
kind of just gets in the way).
These functions are now deprecated in favor of alternative versions that
return a vector of strings rather than a table of strings.
Deprecated functions:
- split: use split_string instead.
- split1: use split_string1 instead.
- split_all: use split_string_all instead.
- split_n: use split_string_n instead.
- cat_string_array: see join_string_vec instead.
- cat_string_array_n: see join_string_vec instead.
- join_string_array: see join_string_vec instead.
- sort_string_array: use sort instead instead.
- find_ip_addresses: use extract_ip_addresses instead.
Changed functions:
- has_valid_octets: uses a string_vec parameter instead of string_array.
Addresses BIT-924, BIT-757.
- Re-arrange how some fa_file fields (e.g. source, connection info, mime
type) get updated/set for consistency.
- Add more robust mechanisms for flushing the reassembly buffer.
The goal being to report all gaps and deliveries to file analyzers
regardless of the state of the reassembly buffer at the time it has to
be flushed.
- Remove all of the x-c detections. Nearly all false
positives.
- Remove the back up TAR detections. Not very helpful.
- Remove one of the x-elc detections that was too loose
and caused many false positives.
- Change to the default BOF buffer size to 3000 (was 1024).
- Reorganized MS signatures into a separate file
- Improved lots of the signatures and added new ones.
- This moves the signatures out of the libmagic imported signatures
and into our own general.sig.
- Expand the detection to LZMA compressed flash files.
The intel framework can now indicate which node discovered a
hit on an intel item through the new "node" field in the
Intel::Seen data structure. On clusters, this field will
contain the name of the node where the hit was seen.
