* origin/topic/vladg/file-analysis-exe-analyzer: (31 commits)
Tweak the PE OS versions based on real-world traffic.
Update pe/main.bro to user register_for_mime_types, ensuring it will also work with the upcoming Files framework changes.
A bit of final core-level cleanup.
A bit of final script cleanup.
Update baselines.
Add a btest for the PE analyzer.
Add a PE memleak test, and fix a memleak.
Documentation and a bit of overall cleanup.
Add data about which tables are present.
Remove the .idata parsing, as it can be more complicated in some cases.
Fix a PE analyzer failure where the IAT isn't aligned with a section boundary.
PE: Rehash the log a bit.
Make base_of_data optional.
Fix support for PE32+ files.
PE Analyzer cleanup.
Checkpoint - Import Address Table being parsed.
Some changes to fix PE analyzer on master.
Parse PE section headers.
Updated PE analyzer to work with changes in master.
In progress checkpoint. Things are starting to work.
...
BIT-1369 #merged
* origin/topic/seth/more-file-type-ident-fixes:
File API updates complete.
Fixes for file type identification.
API changes to file analysis mime type detection.
Make HTTP 206 reassembly require ETags by default.
More file type identification improvements
Fix an issue with files having gaps before the bof_buffer is filled.
Fix an issue with packet loss in http file reporting.
Adding WOFF fonts to file type identification.
Extended JSON matching and added OCSP responses.
Another large signature update.
More signature updates.
Even more file type ident clean up.
Lots of fixes for file type identification.
BIT-1368 #merged
Removed "file_mime_type" and "file_mime_types" event, replacing them
with a new event called "file_metadata_inferred". It has a record
argument of type "inferred_file_metadata", which contains the mime type
information that the earlier events used to supply. The idea here is
that future extensions to the record with new metadata will be less
likely to break user code than the alternatives (adding new events or
new event parameters).
Addresses BIT-1368.
When files had gaps prior to the bof_buffer completely filling, the
file gap handling code was never sniffing and passing along as much
data as possible so file type identification wasn't working correctly.
- Plain text now identified with BOMs for UTF8,16,32
(even though 16 and 32 wouldn't get identified as plain text, oh-well)
- X.509 certificates are now populating files.log with
the mime type application/pkix-cert.
- File signatures are split apart into file types
to help group and organize signatures a bit better.
- Normalized some FILE_ANALYSIS debug messages.
- Improved Javascript detection.
- Improved HTML detection.
- Removed a bunch of bad signatures.
- Merged a bunch of signatures that ultimately detected
the same mime type.
- Added detection for MS LNK files.
- Added detection for cross-domain-policy XML files.
- Added detection for SOAP envelopes.
Basically, at least some rdp certificates specify a completely invalid
and nonsensical value for theyr key type. OpenSSL does not like this and
refuses to parse the key in this case. With this change, we detect this
case and special-case it, hinting to OpenSSL what kind of key we have.
This gives us additional information that we would not have otherwhise
in the log file (like key length and the exponent).
* origin/topic/johanna/x509-cn:
Use our new features to send the CN and SAN fields of certificates to the intel framework.
Do not log common name by default (it is most interesting for scripts) and add a test case.
extract most specific common name from certificates
BIT-1323 #merged
- Any files where the total size was below the size of the
default bof_buffer size couldn't have stream analyzers successfully
attached because the bof_buffer never reached the full size
and was never flushed. This branch explicitly marks the buf_buffer
as full and flushes it when the file is being removed.
- Re-arrange how some fa_file fields (e.g. source, connection info, mime
type) get updated/set for consistency.
- Add more robust mechanisms for flushing the reassembly buffer.
The goal being to report all gaps and deliveries to file analyzers
regardless of the state of the reassembly buffer at the time it has to
be flushed.
MIME entities buffered data and passed it along to protocol analyzers in
discrete amounts, but a gap is always passed along right away, so the
ordering of these "events" can cause incorrect file analysis. The
change here is to never leave any MIME data buffered -- it should now be
passed along line by line as it is seen, but may still temporarily make
use of a buffer allocated by the analyzer as it works on decoding
content.
* origin/topic/johanna/ticket-1212:
Fix ocsp reply validation - there were a few things that definitely were wrong.
fix null pointer dereference in ocsp verification code in case no certificate is sent as part as the ocsp reply.
Now the right signer certificate for the reply is looked up (and no longer assumed that it is the first one) and a few compares are fixed. Plus - there are more test cases that partially send certificates in the ocsp message and partially do not - and it seems to work fine in all cases.
Addresses BIT-1212
is sent as part as the ocsp reply.
Addresses BIT-1212
There is an additional issue here that prevents the correct verification of
proofs in quite a few cases; this will be addressed in a separate commit.
with a MIME type.
Whenever that MIME is detected, Bro will now automatically activate
the analyzer. The interface mimics how well-known ports are defined
for protocol analyzers.
This isn't actually used by any existing file analyzer (because we
don't have any yet that target a specific file format), but there's a
test making sure it works.
