* origin/topic/dnthayer/test-improvements:
Fix the ip-broken-header.bro test on macOS
Improve reliability of the logging rotate.bro test
Improve reliability of missing-file-initially.bro test
BIT-1929 #merged
* origin/topic/seth/dhcp-update:
Rework to the DHCP analyzer.
First step of DHCP analyzer rearchitecture.
Add .btest scripts for dhck_ack and dhcp_discover messages verifying that new options are correctly reported in dhcp.log records.
Extend DHCP protocol analyzer with new options.
BIT-1924 #merged
Additional changes:
* Removed known-hosts.bro as the only thing populating its table was
the already-removed known-hosts-and-devices.bro. So a
known_devices.log will no longer be generated.
* In dhcp-options.pac, the process_relay_agent_inf_option had a memleak
and also process_auto_proxy_config_option looked like it accessed one
byte past the end of the available bytestring, so fixed those.
The way in which TLS 1.3 is negotiated was changed slightly in later
revisions of the standard. The final version is only sent in an
extension - while the version field in the server hello still shows TLS
1.2.
This patch makes ssl.log show the correct version again.
Closes BIT-1897.
* origin/topic/johanna/ssl_signature_details:
Make parsing of ServerKeyExchange work for D(TLS) < 1.2.
Add more details to ssl_server_signature.
* 'smb-transaction-messages' of https://github.com/jbencteux/bro:
add test for smb1_com_transaction_response event changes
add test for smb1_com_transaction2_secondary_request event changes
add test for smb1_com_transaction2_request event changes
add test for smb1_com_transaction_secondary_request event changes
add test for smb1_com_transaction_request event changes
fix setup field handling in smb1_com_transaction_request messages
fix smb1_com_transaction* messages
add smb1_transaction2_secondary_request event
add smb1_transaction_secondary_request event
add parameters and data to smb1_transaction_request/response messages
add SMB_Parameters.Words to smb1_transaction2_request event
* 'nfs-updates' of https://github.com/dtrejod/bro:
Format print nfs units tests to improve output readability. Add unit tests for new NFS events -- nfs_proc_symlink, nfs_proc_link, nfs_proc_sattr.
Bug fix: nfs3_writeargs didn't properly return filehandle.
Add nfs_proc_symlink, nfs_proc_link, nfs_proc_sattr.
* 'mount-protocol' of https://github.com/dtrejod/bro:
Add unit tests for new MOUNT events -- mount_proc_mnt, mount_proc_umnt, mount_proc_umnt_all, mount_proc_not_implemented.
Add mount_proc_null, mount_proc_mnt, mount_proc_umnt, mount_proc_umnt_all, mount_proc_not_implemented, mount_reply_status.
Now we only parse the SignatureAndHashalgorithm field in cases where it
is present. This change also takes care to respect SCTs, which do
include the SignatureAndHashalgorithm in their digitally-signed struct,
even when used in protocol versions that do not have the
SignatureAndHashalgorithm in the protocols digitally-signed struct.
I also added tests to make sure this does indeed work with TLS 1.1 - it
turns out that so far we did not have a single TLS 1.1 pcap.
In ContentLine_Analyzer, prevent excessively long lines being assembled.
The line length will default to just under 16MB, but can be overriden on
a per-analyzer basis. This is done for the finger,ident, and irc
analyzers.
IP packets that have a header length that is greater than the total
length of the packet cause a integer overflow, which cause range-checks
to fail, which causes OOB reads.
Furthermore Bro does not currently check the version field of IP packets
that are read from tunnels. I added this check - otherwhise Bro reports
bogus IP information in its error messages, just converting the data
from the place where the IP information is supposed to be to IPs.
This behavior brings us closer to what other software (e.g. Wireshark)
displays in these cases.
It turns out that Chrome supports an experimental mode to support TLS
1.3, which uses a non-standard way to negotiate TLS 1.3 with a server.
This non-standard way to negotiate TLS 1.3 breaks the current draft RFC
and re-uses an extension on the server-side with a different binary
formatting, causing us to throw a binpac exception.
This patch ignores the extension when sent by the server, continuing to
correctly parse the server_hello reply (as far as possible).
From what I can tell this seems to be google working around the fact
that MITM equipment cannot deal with TLS 1.3 server hellos; this change
makes the fact that TLS 1.3 is used completely opaque unless one looks
into a few extensions.
We currently log this as TLS 1.2.
The pcap file format has a global header and a header per packet. The
global header of the pcap in question had a snaplen of 1, but with
packet headers indicating the full number of bytes saved within the
file. It seems like the pcap file must of been artifically edited in
order for it to be this way.
When reporting the captured length of a packet, Apple's version of
libpcap now seems to report the full number of bytes saved within the
pcap's per-packet headers, but other versions seem to report the snaplen
from the global pcap header. This caused the core.truncation test to
behave differently on macOS from other platforms.
I've manually hexedit'd the pcap so that the snaplen is still 1, but
contains just a single packet with a pcap header indicating a length of
8, which is less than the size of the link layer header and so should
still test the original code path that the unit test intended to
exercise.
This adds a slight patch to the HTTP analyzer, which recognizez when a connection is
upgraded to a different protocol (using a 101 reply with a few specific headers being
set).
In this case, the analyzer stops further processing of the connection (which will
result in DPD errors) and raises a new event:
event http_connection_upgrade(c: connection, protocol: string);
Protocol contains the name of the protocol that is being upgraded to, as specified in
one of the header values.
Closes#1830.
* origin/topic/johanna/ocsp-sct-validate: (82 commits)
Tiny script changes for SSL.
Update CT Log list
SSL: Update OCSP/SCT scripts and documentation.
Revert "add parameter 'status_type' to event ssl_stapled_ocsp"
Revert "parse multiple OCSP stapling responses"
SCT: Fix script error when mime type of file unknown.
SCT: another memory leak in SCT parsing.
SCT validation: fix small memory leak (public keys were not freed)
Change end-of-connection handling for validation
OCSP/TLS/SCT: Fix a number of test failures.
SCT Validate: make caching a bit less aggressive.
SSL: Fix type of ssl validation result
TLS-SCT: compile on old versions of OpenSSL (1.0.1...)
SCT: Add caching support for validation
SCT: Add signed certificate timestamp validation script.
SCT: Allow verification of SCTs in Certs.
SCT: only compare correct OID/NID for Cert/OCSP.
SCT: add validation of proofs for extensions and OCSP.
SCT: pass timestamp as uint64 instead of time
Add CT log information to Bro
...
* 'nfs_changes' of https://github.com/jwallior/bro:
Add nfs unittest. Includes an example for the new nfs_proc_rename.
Added rename event to rpc/nfs protocol analyzer. This event identifies and reports information about nfs/rpc calls and replies of the type rename.
Expand parsing of RPC Call packets to add Uid, Gid, Stamp, MachineName and AuxGIDs
Fix NFS protocol parser.
The dpd signature missed a few cases that are used for TLS 1.3,
especially when draft versions (which are all that we are seeing at the
moment) are being negotiated.
This fix mostly allows draft versions in the server hello (identified by
7F[version]; since we do not know how many drafts there will be, we are
currently allowing a rather safe upper limit.
This also rewrites the certificate validation script (which we need for
this) slightly.
This could need a bit of caching, but should generally work very
reliably.
- This fixes BIT-1769 by logging all requests even in the absence of a
reply. The way that request and replying matching were being handled
was restructured to mostly ignore the transaction ids because they
aren't that helpful for network monitoring and it makes the script
structure more complicated.
- Add `framed_addr` field to the radius log to indicate if the radius
server is hinting at an address for the client.
- Add `ttl` field to indicate how quickly the radius server is replying
to the network access server.
- Fix a bunch of indentation inconsistencies.
This is a tiny bit evil because it uses parts of the SSL protocol
analyzer in the X.509 certificate parser. Which is the fault of the
protocol, which replicates the functionality.
