* origin/topic/seth/dhcp-update:
Rework to the DHCP analyzer.
First step of DHCP analyzer rearchitecture.
Add .btest scripts for dhck_ack and dhcp_discover messages verifying that new options are correctly reported in dhcp.log records.
Extend DHCP protocol analyzer with new options.
BIT-1924 #merged
Additional changes:
* Removed known-hosts.bro as the only thing populating its table was
the already-removed known-hosts-and-devices.bro. So a
known_devices.log will no longer be generated.
* In dhcp-options.pac, the process_relay_agent_inf_option had a memleak
and also process_auto_proxy_config_option looked like it accessed one
byte past the end of the available bytestring, so fixed those.
Removed duplicated entries (`000001a0-0000-0000-c000-000000000046`, with `i=3, i=4`) and related comments line
This change should make this file more readable and it will be easier to convert it to other formats (personally I found it out while I was trying to convert this to a JavaScript structure).
The way in which TLS 1.3 is negotiated was changed slightly in later
revisions of the standard. The final version is only sent in an
extension - while the version field in the server hello still shows TLS
1.2.
This patch makes ssl.log show the correct version again.
Highlights:
- Reduced all DHCP events into a single dhcp_message event. (removed legacy events since they weren't widely used anyway)
- Support many more DHCP options.
- DHCP log is completely reworked and now represents DHCP sessions
based on the transaction ID (and works on clusters).
- Removed the known-devices-and-hostnames script since it's generally
less relevant now with the updated log.
Old event prototypes have changed and the events are broken right
now and may be removed in favor of the new generic "dhcp_message"
event.
DHCP option parsing is abstracted from the main code base of the
protocol parser and are all now located in their own file.
Documentation, tests, and final code cleanup are still pending.
This makes conn.logs a bit prettier (and smaller) because all lines that
do not use a tunnel will now have a "-" instead of the "(empty)" for
tunnel_parents.
Add the folowing option types:
- 55 Parameters Request List;
- 58 Renewal time;
- 59 Rebinding time;
- 61 Client Identifier;
- 82 Relay Agent Information.
Extend the following events with new parameters, specifically:
- dhcp_discover exports client identifier and parameters request list;
- dhcp_request exports client_identifier and parameters request list;
- dhcp_ack exports rebinding time, renewal time and list of suboptions value of
dhcp relay agent information option;
- dhcp_inform exports parameters request list.
Add option type specific variables within the scope of DHCP module
(see src/analyzer/protocol/dhcp/types.bif).
Move protocol specific variables "dhcp_msg" and "dhcp_router_list" from scope Global to DHCP::
and adapt inet_net_var in src/NetVar.cc consequently.
Extend src/analyzer/protocols/dhcp/main.bro to handle the new events and to log
dhcp_ack, dhcp_request and dhcp_discover.
Modify scripts/policy/protocols/dhcp/known-devices-and-hostnames.bro to
include new events' variables.
These have been lingering for a while and they generally annoy
everyone because of the sheer volume. They also don't really add
any useful information for debugging and they were generated differently
than most other weirds anyway (which was a little weird...).
This introduces a new option, SOCKS::default_capture_password which can
be used to specify if Socks passwords are logged by default
Like fot FTP/HTTP, this option is set to false by default.
Addresses BIT-1791
This commit fixes a few small issues.
* server key exchange parameters are only parsed when a named curve is
given.
* I removed the ssl-verbose.bro and moved the functionality into the
testcase.
The information that we get with these events is likely irrelevant to
the majority of Bro users; I do not think that we have to ship a
script that uses them by default. A script like this would be
something to publish via the Bro package manager instead; this is the
approach that we have taken with a number of the recent SSL addition.
* I marked the ssl_server_curve event as deprecated. More information is
contained in the new ssl_ecdh_server_params event.
This is an events that is probably seldomly (or never) directly used
by anyone; I plan to completely remove it right after the 2.6 release.
The bytes_threshold_crossed event in the gridftp analyzer is not first
checking to see if the connection passed the initial criteria. This
causes the script to add the gridftp-data service to any connection that
crosses a threshold that is the same as or greater than the gridftp
size_threshold.
Closes#1830.
* origin/topic/johanna/ocsp-sct-validate: (82 commits)
Tiny script changes for SSL.
Update CT Log list
SSL: Update OCSP/SCT scripts and documentation.
Revert "add parameter 'status_type' to event ssl_stapled_ocsp"
Revert "parse multiple OCSP stapling responses"
SCT: Fix script error when mime type of file unknown.
SCT: another memory leak in SCT parsing.
SCT validation: fix small memory leak (public keys were not freed)
Change end-of-connection handling for validation
OCSP/TLS/SCT: Fix a number of test failures.
SCT Validate: make caching a bit less aggressive.
SSL: Fix type of ssl validation result
TLS-SCT: compile on old versions of OpenSSL (1.0.1...)
SCT: Add caching support for validation
SCT: Add signed certificate timestamp validation script.
SCT: Allow verification of SCTs in Certs.
SCT: only compare correct OID/NID for Cert/OCSP.
SCT: add validation of proofs for extensions and OCSP.
SCT: pass timestamp as uint64 instead of time
Add CT log information to Bro
...
Move from using CCS (before: established) to just doing certificate
validation at the end of the connection.
This is (again) more robust in the case of aborted connection. I am
moving this into a hook because of the complexity of the
end-of-connection handling for SSL.
This should probably be extended to not just handle SSL validation, but
all other logging constructs that are currently called in _established.
The dpd signature missed a few cases that are used for TLS 1.3,
especially when draft versions (which are all that we are seeing at the
moment) are being negotiated.
This fix mostly allows draft versions in the server hello (identified by
7F[version]; since we do not know how many drafts there will be, we are
currently allowing a rather safe upper limit.
This commit add the table SSL::ct_logs to Bro. This table is populated
with information about the currently active certificate transparency
logs (data from Google). The data can, e.g., be used to identify which
Logs are being used in SCTs.
- This fixes BIT-1769 by logging all requests even in the absence of a
reply. The way that request and replying matching were being handled
was restructured to mostly ignore the transaction ids because they
aren't that helpful for network monitoring and it makes the script
structure more complicated.
- Add `framed_addr` field to the radius log to indicate if the radius
server is hinting at an address for the client.
- Add `ttl` field to indicate how quickly the radius server is replying
to the network access server.
- Fix a bunch of indentation inconsistencies.
Re-enable logging, now in policy because it probably is interesting to
no-one. We also only log ocsp replies.
Fix all tests.
Fix an issue where ocsp replies were added to the x.509 certificate
list.
This makes it much easier for protocols where the mime type is known in
advance like, for example, TLS. We now do no longer have to perform deep
script-level magic.
Moved the definitions of DCE_RPC::BackingState and DCE_RPC::State types
into the export block. These types are used in the redef of the
"connection" record.
* origin/topic/seth/smb-auth-fixes:
Disable SMB2 error data parsing.
Removed some files that aren't being used.
Fix SMB tree connect handling.
Fix a small issue where DCE_RPC commands were improperly being logged.
SMB fixes and cleanup.
Including a test for raw NTLM in SMB
Updates for SMB auth handling from Martin van Hensbergen.
BIT-1721 #merged
SMB error handling improved. The analyzer isn't destroyed when a problem
is encoutered anymore. The flowbuffer in the parser is now flushed and
the analyzer is set to resync against an SMB command. This was needed
because there is some state about open files that is kept within the
parser itself which was being destroyed and that was causing analysis
after content gaps or parse errors to be faulty. The new mechanism
doesn't detroy the parser so parsing after gaps is improved.
DCE_RPC handling in SMB is improved in the edge case where a drive
mapping isn't seen. There is a new const named SMB::pipe_filenames
which is used as a heuristic for identifying "files" opened on named
pipe shares. If the share mapping type isn't known and a filename
in this set is found, the share type will change to "PIPE" by
generating an event named "smb_pipe_connect_heuristic". Reads and
writes to that file will be sent to the DCE_RPC analyzer instead of
to the files framework.
The concept of "unknown" share types has been removed due to the new
heuristic detection of share types.
Some general clean up of how the SMB cmd log is written and when.
- The logic for fragment handling has been rewritten and should be correct now.
- There are now tunables for fragment handling overflow situations.
- DCE_RPC::max_cmd_reassembly and DCE_RPC::max_frag_data
- They result in weirds and analyzer removal.
- Memory leak fixed by unique_ptr auto cleanup.
- DCE_RPC is now intolerate of content gaps and will stop
analyzing traffic if content gaps happen (like most other analyzers currently).
