* origin/topic/vladg/file-analysis-exe-analyzer: (31 commits)
Tweak the PE OS versions based on real-world traffic.
Update pe/main.bro to user register_for_mime_types, ensuring it will also work with the upcoming Files framework changes.
A bit of final core-level cleanup.
A bit of final script cleanup.
Update baselines.
Add a btest for the PE analyzer.
Add a PE memleak test, and fix a memleak.
Documentation and a bit of overall cleanup.
Add data about which tables are present.
Remove the .idata parsing, as it can be more complicated in some cases.
Fix a PE analyzer failure where the IAT isn't aligned with a section boundary.
PE: Rehash the log a bit.
Make base_of_data optional.
Fix support for PE32+ files.
PE Analyzer cleanup.
Checkpoint - Import Address Table being parsed.
Some changes to fix PE analyzer on master.
Parse PE section headers.
Updated PE analyzer to work with changes in master.
In progress checkpoint. Things are starting to work.
...
BIT-1369 #merged
* origin/topic/seth/more-file-type-ident-fixes:
File API updates complete.
Fixes for file type identification.
API changes to file analysis mime type detection.
Make HTTP 206 reassembly require ETags by default.
More file type identification improvements
Fix an issue with files having gaps before the bof_buffer is filled.
Fix an issue with packet loss in http file reporting.
Adding WOFF fonts to file type identification.
Extended JSON matching and added OCSP responses.
Another large signature update.
More signature updates.
Even more file type ident clean up.
Lots of fixes for file type identification.
BIT-1368 #merged
one can now add an option "offset" to the config map. Positive offsets
are interpreted to be from the beginning of the file, negative from the
end of the file (-1 is end of file).
Only works for raw reader in streaming or manual mode. Does not work
with executables.
Addresses BIT-985
This also patches a few tests to contain certificates that were removed.
Furthermore, we include the old CA file with the external tests and load
it automatically. Those traces are kind of old now, more and more of the
CAs in them are no longer valid and it does not really make sense to
update them on each change...
The HTTP analyzer was propogating Gaps to the files framework even
in the case of a packet drop occurring immediately after the headers
are completed in an HTTP response when the response content length
was declared to be zero (no file started, so no loss).
Includes passing test.
* origin/topic/seth/rdp: (31 commits)
Improved transition into SSL/TLS from RDP.
Fixes tests in RDP branch.
add a special case to the X509 code that deals with RDP certificates.
A few more changes to handling encryption in RDP.
Adds some comments and fixes a broxygen warning.
Fixes another optional part of an RDP unit.
Support RDP negotiation requests optionally and support zero length cookies.
Changed UTF-16 to UTF-8 conversion to be more lenient.
Fixed an issue with parse failure on an optional field.
Removing a stray printf from RDP analyzer.
Another big RDP update.
New script to add a field to rdp.log when the connection is upgraded to SSL.
Huge updates to the RDP analyzer from Josh Liburdi.
FreeRDP test trace showing SSL encryption -- RDP analyzer does not currently handle this and SSL analyzer does not identify it either
Wireshark test trace for native encryption -- generates a binpac error
Delete RDP-004.pcap
Delete nla_win7_win2k8r2.pcap
Update dpd.sig
Fixed typo
Added check for connection existence
...
BIT-1340 #merged
- Some scripts used wrong SSH module/namespace scoping on events.
- Fix outdated notice documentation related to SSH password guessing.
- Add a unit test for SSH pasword guessing notice.
I replaced a few strcmps with either calls to std::str.compare
or with the == operator of BroString.
Also changed two of the input framework tests that did not pass
anymore after the merge. The new SSH analyzer no longer loads the
scripts that let network time run, hence those tests failed because
updates were not propagated from the threads (that took a while
to find.)
* origin/topic/vladg/ssh: (25 commits)
SSH: Register analyzer for 22/tcp.
SSH: Add 22/tcp to likely_server_ports
SSH: Ignore encrypted packets by default.
SSH: Fix some edge-cases which created BinPAC exceptions
SSH: Add memleak btest
SSH: Update baselines
SSH: Added some more events for SSH2
SSH: Intel framework integration (PUBKEY_HASH)
Update baselines for new SSH analyzer.
Update SSH policy scripts with new events.
SSH: Add documentation
Refactoring ssh-protocol.pac:
SSH: Use the compression_algorithms const in another place.
Some cleanup and refactoring on SSH main.bro.
SSH: A bit of code cleanup.
Move SSH constants to consts.pac
SSH: Cleanup code style.
SSH: Fix some memleaks.
Refactored the SSH analyzer. Added supported for algorithm detection and more key exchange message types.
Add host key support for SSH1.
Add support for SSH1
Move SSH analyzer to new plugin architecture.
...
Conflicts:
scripts/base/protocols/ssh/main.bro
testing/btest/Baseline/core.print-bpf-filters/output2
testing/btest/Baseline/plugins.hooks/output
BIT-1344: #merged
* origin/topic/johanna/dtls:
a few more small script-level fixes
update test baselines
add a simple leak test for dtls
add signature for dtls client hello
Make the plugin structure more... legal.
Only force logging of SSL if it actually was the SSL analyzer that failed.
DTLS working.
Implement correct parsing of TLS record fragmentation.
Make handshake analyzer flow-based. This means we can feed data to it in chunks, which makes dealing with fragmentation a little bit more convenient.
When setting the SSL analyzer to fail, also stop processing data that already has been delivered to the analyzer, not just future data.
First step for a DTLS analyzer.
BIT-1347 #merged
Conflicts:
scripts/base/protocols/ssl/main.bro
testing/btest/Baseline/plugins.hooks/output
* origin/topic/johanna/bit-1199:
add a basic leak test for an unparseable enum
Change the way the input framework deals with values it cannot convert into BroVals (especially enums)
Make error message when encountering not existing enums better.
BIT-1199: #merged
into BroVals (especially enums)
Not we do not force an internal error anymore. Instead, we raise an
normal error and set an error flag that signals to the top-level
functions that the value could not be converted and should not be
propagated to the Bro core. This sadly makes the already messy code even
more messy - but since errors can happen in deeply nested data
structures, the alternative (catching the error at every possible
location and then trying to clean up there instead of recursively
deleting the data that cannot be used later) is much worse.
Addresses BIT-1199
* origin/topic/johanna/cert-validation:
and still use the hash for notice suppression.
add knob to revert to old validation behavior
Update certificate validation script - new version will cache valid intermediate chains that it encounters on the wire and use those to try to validate chains that might be missing intermediate certificates.
BIT-1332 #merged
The only thing that is missing is a signature to detect the protocol (it
has no well-known port).
Reassembly is kind of fidgety - at the moment we only support
re-assembling one simultaneous message per direction (which looking at
our test-traffic might not be a problem). And I am not quite sure if I
got all cases correct...
But - it works :)
intermediate chains that it encounters on the wire and use those to try
to validate chains that might be missing intermediate certificates.
This vastly improves the number of certificates that Bro can validate.
The only drawback is that now validation behavior is not entirely
predictable anymore - the certificate of a server can fail to validate
when Bro just started up (due to the intermediate missing), and succeed
later, when the intermediate can be found in the cache.
Has been tested on big-ish clusters and should not introduce any
performance problems.
- More data pulled into scriptland.
- Logs expanded with client screen resolution and desired color depth.
- Values in UTF-16 on the wire are converted to UTF-8 before being
sent to scriptland.
- If the RDP turns into SSL records, we now pass data that appears
to be SSL to the PIA analyzer.
- If RDP uses native encryption with X.509 certs we pass those
certs to the files framework and the base scripts pass them forward
to the X.509 analyzer.
- Lots of cleanup and adjustment to fit the documented protocol
a bit better.
- Cleaned up the DPD signatures.
- Moved to flowunit instead of datagram.
- Added tests.
ssl connections with old versions as well as unsafe cipher suites.
Also make the notice suppression handling of other ssl policy files
a tad more robust.
* origin/fastpath:
Crashing bug in WriterBackend when deserializing WriterInfo where config is present. Testcase crashes on unpatched versions of Bro.
Fix wrong value test in WriterBackend. Found by Aaron Eppert (aeppert@gmail.com)
is present. Testcase crashes on unpatched versions of Bro.
Found by Aaron Eppert <aeppert@gmail.com>.
This (probably) fixes the crash issue with sqlite a few people have
reported on the mailing list in the past.
- This addresses BIT-1011
- Add a new field to socks.log; "password".
- Two new events; socks_login_userpass and socks_login_reply.
- One new weird for unsupported authentication method.
- A new test for authenticated socks traffic.
- Credit to Nicolas Retrain for the initial patch. Thanks!
- Any files where the total size was below the size of the
default bof_buffer size couldn't have stream analyzers successfully
attached because the bof_buffer never reached the full size
and was never flushed. This branch explicitly marks the buf_buffer
as full and flushes it when the file is being removed.
These functions are now deprecated in favor of alternative versions that
return a vector of strings rather than a table of strings.
Deprecated functions:
- split: use split_string instead.
- split1: use split_string1 instead.
- split_all: use split_string_all instead.
- split_n: use split_string_n instead.
- cat_string_array: see join_string_vec instead.
- cat_string_array_n: see join_string_vec instead.
- join_string_array: see join_string_vec instead.
- sort_string_array: use sort instead instead.
- find_ip_addresses: use extract_ip_addresses instead.
Changed functions:
- has_valid_octets: uses a string_vec parameter instead of string_array.
Addresses BIT-924, BIT-757.
* origin/topic/robin/dnp3-merge-v4:
add test trace in which DNP3 packets are over UDP; update test scripts and baseline results
A bit more DNP3 tweaking.
remove redundnt codes; find a way to use the analyzer function, such as Weird; fix a small bug in ProcessData function in DNP3.cc; passed the test
Renameing the DNP3 TCP analyzer
quickly fix another bug; adding missing field of the declaration of dnp3_request_application_header and dnp3_response_application_header
Removing the debug printf in DNP3.cc
fixed the bug of deciding the size of object 1 varition 1 in DNP3
Fix some things in DNP3 UDP analyzer.
changed a bug, but still not working
modify DNP3.cc and DNP3.h to add DNP3_UDP_Analyzer; binpac unchanged
BIT-1231 #merged
