* origin/topic/vladg/file-analysis-exe-analyzer: (31 commits)
Tweak the PE OS versions based on real-world traffic.
Update pe/main.bro to user register_for_mime_types, ensuring it will also work with the upcoming Files framework changes.
A bit of final core-level cleanup.
A bit of final script cleanup.
Update baselines.
Add a btest for the PE analyzer.
Add a PE memleak test, and fix a memleak.
Documentation and a bit of overall cleanup.
Add data about which tables are present.
Remove the .idata parsing, as it can be more complicated in some cases.
Fix a PE analyzer failure where the IAT isn't aligned with a section boundary.
PE: Rehash the log a bit.
Make base_of_data optional.
Fix support for PE32+ files.
PE Analyzer cleanup.
Checkpoint - Import Address Table being parsed.
Some changes to fix PE analyzer on master.
Parse PE section headers.
Updated PE analyzer to work with changes in master.
In progress checkpoint. Things are starting to work.
...
BIT-1369 #merged
* origin/topic/seth/more-file-type-ident-fixes:
File API updates complete.
Fixes for file type identification.
API changes to file analysis mime type detection.
Make HTTP 206 reassembly require ETags by default.
More file type identification improvements
Fix an issue with files having gaps before the bof_buffer is filled.
Fix an issue with packet loss in http file reporting.
Adding WOFF fonts to file type identification.
Extended JSON matching and added OCSP responses.
Another large signature update.
More signature updates.
Even more file type ident clean up.
Lots of fixes for file type identification.
BIT-1368 #merged
- Backed out eTag changes. The real world is more complicated
than just using eTags to identify the same file.
- A bit of code simplication in the http base scripts.
- Test updates (more existing small problems were identified!).
-
Removed "file_mime_type" and "file_mime_types" event, replacing them
with a new event called "file_metadata_inferred". It has a record
argument of type "inferred_file_metadata", which contains the mime type
information that the earlier events used to supply. The idea here is
that future extensions to the record with new metadata will be less
likely to break user code than the alternatives (adding new events or
new event parameters).
Addresses BIT-1368.
* origin/topic/seth/rdp: (31 commits)
Improved transition into SSL/TLS from RDP.
Fixes tests in RDP branch.
add a special case to the X509 code that deals with RDP certificates.
A few more changes to handling encryption in RDP.
Adds some comments and fixes a broxygen warning.
Fixes another optional part of an RDP unit.
Support RDP negotiation requests optionally and support zero length cookies.
Changed UTF-16 to UTF-8 conversion to be more lenient.
Fixed an issue with parse failure on an optional field.
Removing a stray printf from RDP analyzer.
Another big RDP update.
New script to add a field to rdp.log when the connection is upgraded to SSL.
Huge updates to the RDP analyzer from Josh Liburdi.
FreeRDP test trace showing SSL encryption -- RDP analyzer does not currently handle this and SSL analyzer does not identify it either
Wireshark test trace for native encryption -- generates a binpac error
Delete RDP-004.pcap
Delete nla_win7_win2k8r2.pcap
Update dpd.sig
Fixed typo
Added check for connection existence
...
BIT-1340 #merged
I replaced a few strcmps with either calls to std::str.compare
or with the == operator of BroString.
Also changed two of the input framework tests that did not pass
anymore after the merge. The new SSH analyzer no longer loads the
scripts that let network time run, hence those tests failed because
updates were not propagated from the threads (that took a while
to find.)
* origin/topic/vladg/ssh: (25 commits)
SSH: Register analyzer for 22/tcp.
SSH: Add 22/tcp to likely_server_ports
SSH: Ignore encrypted packets by default.
SSH: Fix some edge-cases which created BinPAC exceptions
SSH: Add memleak btest
SSH: Update baselines
SSH: Added some more events for SSH2
SSH: Intel framework integration (PUBKEY_HASH)
Update baselines for new SSH analyzer.
Update SSH policy scripts with new events.
SSH: Add documentation
Refactoring ssh-protocol.pac:
SSH: Use the compression_algorithms const in another place.
Some cleanup and refactoring on SSH main.bro.
SSH: A bit of code cleanup.
Move SSH constants to consts.pac
SSH: Cleanup code style.
SSH: Fix some memleaks.
Refactored the SSH analyzer. Added supported for algorithm detection and more key exchange message types.
Add host key support for SSH1.
Add support for SSH1
Move SSH analyzer to new plugin architecture.
...
Conflicts:
scripts/base/protocols/ssh/main.bro
testing/btest/Baseline/core.print-bpf-filters/output2
testing/btest/Baseline/plugins.hooks/output
BIT-1344: #merged
* origin/topic/johanna/dtls:
a few more small script-level fixes
update test baselines
add a simple leak test for dtls
add signature for dtls client hello
Make the plugin structure more... legal.
Only force logging of SSL if it actually was the SSL analyzer that failed.
DTLS working.
Implement correct parsing of TLS record fragmentation.
Make handshake analyzer flow-based. This means we can feed data to it in chunks, which makes dealing with fragmentation a little bit more convenient.
When setting the SSL analyzer to fail, also stop processing data that already has been delivered to the analyzer, not just future data.
First step for a DTLS analyzer.
BIT-1347 #merged
Conflicts:
scripts/base/protocols/ssl/main.bro
testing/btest/Baseline/plugins.hooks/output
This allows the path for the default filter to be specified explicitly
when creating a stream and reduces the need to rely on the default path
function to magically supply the path.
The default path function is now only used if, when a filter is added to
a stream, it has neither a path nor a path function already.
Adapted the existing Log::create_stream calls to explicitly specify a
path value.
Addresses BIT-1324
* origin/topic/gilbert/plugin-api-tweak:
Updating plugin.hooks baseline so that test succeeds
Revert spacing change that shouldn't have been included with the previous changeset ... should fix all of the plugin tests save hooks, which needs to be updated.
More small fixes
Small fixes
Incremental
Re-updating plugin.hooks test to include new argument output (after merge).
Fixing logic errors in HandlePluginResult
Updating tests and tweaking HookArgument to include Frame support.
Incremental commit: implementing a wrapper for the Val class.
Reverting change to const status of network_time. Also, see FIXME: in Func.cc / HandlePluginResult ...
Tweaks to result handling to make things a little more sane.
Plugin API: minor change (adding parent frame) to support calling methods from hook. Also declare network time update argument to be const because good practice.
BIT-1270 #merged
Conflicts:
testing/btest/Baseline/plugins.hooks/output
These functions are now deprecated in favor of alternative versions that
return a vector of strings rather than a table of strings.
Deprecated functions:
- split: use split_string instead.
- split1: use split_string1 instead.
- split_all: use split_string_all instead.
- split_n: use split_string_n instead.
- cat_string_array: see join_string_vec instead.
- cat_string_array_n: see join_string_vec instead.
- join_string_array: see join_string_vec instead.
- sort_string_array: use sort instead instead.
- find_ip_addresses: use extract_ip_addresses instead.
Changed functions:
- has_valid_octets: uses a string_vec parameter instead of string_array.
Addresses BIT-924, BIT-757.
Retrieval of extended alert information from sid-msg.map, gen-msg.map,
and classification.config files uses Bro's input framework, but since
the unified2 file analyzer also relies on the input framework,
coordination is needed to start analysis only after extended info has
been read at least once.
- Re-arrange how some fa_file fields (e.g. source, connection info, mime
type) get updated/set for consistency.
- Add more robust mechanisms for flushing the reassembly buffer.
The goal being to report all gaps and deliveries to file analyzers
regardless of the state of the reassembly buffer at the time it has to
be flushed.
