This is a tiny bit evil because it uses parts of the SSL protocol
analyzer in the X.509 certificate parser. Which is the fault of the
protocol, which replicates the functionality.
This undoes the changes applied in merge 9db27a6d60
and goes back to the state in the branch as of the merge 5ab3b86.
Getting rid of the additional layer of removing analyzers and just
keeping them in the set introduced subtle differences in behavior since
a few calls were still passed along. Skipping all of these with SetSkip
introduced yet other subtle behavioral differences.
* origin/topic/robin/file-analysis-fixes:
Adding test with command line that used to trigger a crash.
Cleaning up a couple of comments.
Fix delay in disabling file analyzers.
Fix file analyzer memory management.
The merge changes around functionality a bit again - instead of having
a list of done analyzers, analyzers are simply set to skipping when they
are removed, and cleaned up later on destruction of the AnalyzerSet.
BIT-1782 #merged
When a file analyzer signaled being done with data delivery, the
analyzer would only be scheduled for removal at that poing, meaning it
could still receive more data until that action actually took effect.
Now we make sure to not send any more data to an analyzer.
File analyzers got deleted immediately once the queue with the
corresponding removal operation got drained. That however can happen
while the analyzer is still doing stuff: the queue is drained whenever
any the "special" file analysis events needing immediate attention has
been executed. This fix now only schedules the analyzer for deletion
at that time, but postpones the actual operation until file object
itself is being destroyed.
At one place in the code, we do not check the correct return code. This
makes it possible for a reply to get a response of "good", when the ocsp
reply is not actually signed by the responder in question.
This also instructs ocsp verication to skip certificate chain
validation, which we do ourselves earlier because the OCSP verify
function cannot do it correctly (no way to pass timestamp).
The "file_extraction_limit" event was passing a Files::AnalyzerArgs
record as an "any" type. This is not right at the least and may
have been causing a crash for a user at worst.
The order in which the plugin initializers are executed is compiler
dependent. With this change, Tags will always be generated in
alphabetical ordering, not in compiler-dependent order.
Compiling a plugin required having access to OpenSSL headers because
they were pulled in by Bro headers that the plugin had to include.
Removinng then OpenSSL dependency from those Bro headers.
I'm also reverting a4e5591e. This is a different fix for the same
problem, and reverting that commit gives us a test case. :-)
There was a bug in the new parsing code, introduced in
708ede22c6 which parses validity times
incorrectly if they are before the year 2000. What happens in this case
is that the 2-digit year will be interpreted to be in the 21st century
(1999 will be parsed as 2099, e.g.).
Broke out the stats collection into a bunch of new Bifs
in stats.bif. Scripts that use stats collection functions
have also been updated. More work to do.
- Removed the gap_report event. It wasn't used anymore
and functionally no more capable that scheduling events
and using the get_gap_summary bif.
- Added functionality to Dictionaries to count cumulative
numbers of inserts performed. This is further used to
measure the total number of connections of various types.
Previously only the number of active connections was
available.
- The Reassembler base class now tracks active reassembly
size for all subclasses (File/TCP/Frag & unknown).
- Improvements to the stats.log. Mostly, more information.
The generalizedtime support in for certificates now fits more
seamlessly to how the rest of the code was structured and does the
different processing for UTC and generalized times at the beginning,
when checking for them.
The test does not output the common name anymore, since the output
format might change accross openssl versions (inserted the serial
instead).
I also added a bit more error checking for the UTC time case.
These changes should be safe -- testing the failure cases proves a bit
difficult at the moment due to the fact that OpenSSL seems to fix the
values that are present in the original ASN.1 before passing them on to
us. It is thus not directly easily possible to trigger the error cases
from scriptland.
This also means that a lot of the new error cases we try to catch here
can probably never happen.
* origin/topic/vladg/file-analysis-exe-analyzer: (31 commits)
Tweak the PE OS versions based on real-world traffic.
Update pe/main.bro to user register_for_mime_types, ensuring it will also work with the upcoming Files framework changes.
A bit of final core-level cleanup.
A bit of final script cleanup.
Update baselines.
Add a btest for the PE analyzer.
Add a PE memleak test, and fix a memleak.
Documentation and a bit of overall cleanup.
Add data about which tables are present.
Remove the .idata parsing, as it can be more complicated in some cases.
Fix a PE analyzer failure where the IAT isn't aligned with a section boundary.
PE: Rehash the log a bit.
Make base_of_data optional.
Fix support for PE32+ files.
PE Analyzer cleanup.
Checkpoint - Import Address Table being parsed.
Some changes to fix PE analyzer on master.
Parse PE section headers.
Updated PE analyzer to work with changes in master.
In progress checkpoint. Things are starting to work.
...
BIT-1369 #merged
* origin/topic/seth/more-file-type-ident-fixes:
File API updates complete.
Fixes for file type identification.
API changes to file analysis mime type detection.
Make HTTP 206 reassembly require ETags by default.
More file type identification improvements
Fix an issue with files having gaps before the bof_buffer is filled.
Fix an issue with packet loss in http file reporting.
Adding WOFF fonts to file type identification.
Extended JSON matching and added OCSP responses.
Another large signature update.
More signature updates.
Even more file type ident clean up.
Lots of fixes for file type identification.
BIT-1368 #merged
