While we support initializing records via coercion from an expression
list, e.g.,
local x: X = [$x1=1, $x2=2];
this can sometimes obscure the code to readers, e.g., when assigning to
value declared and typed elsewhere. The language runtime has a similar
overhead since instead of just constructing a known type it needs to
check at runtime that the coercion from the expression list is valid;
this can be slower than just writing the readible code in the first
place, see #4559.
With this patch we use explicit construction, e.g.,
local x = X($x1=1, $x2=2);
The seen/file-names script relies on f$info$filename to be populated.
For HTTP and other network protocols, however, this field is only
populated during file_over_new_connection() that's running after
file_new().
Use the file_new() event only for files without connections and
file_over_new_connection() implies that f$conns is populated, anyway.
Special case SMB to avoid finding files twice, because there's a
custom implementation in seen/smb-filenames.zeek.
Fixes#2647
This commit changes the SSL and X.509 logging formats to something that,
hopefully, slowly approaches what they will look like in the future.
X.509 log is not yet deduplicated; this will come in the future.
This commit introduces two new options, which determine if certificate
issuers and subjects are still logged in ssl.log. The default is to have
the host subject/issuer logged, but to remove client-certificate
information. Client-certificates are not a typically used feature
nowadays.
* 'master' of https://github.com/hosom/zeek:
Normalize the intel seen filename for smb.
load smb-filenames in scripts/policy/frameworks/intel/seen/__load__.bro
Add SMB::IN_FILE_NAME to Intel::Where enum
Support filenamess for SMB files
I added a test case
* origin/topic/vladg/bit-1641:
Logic fix for ssh/main.bro when the auth status is indeterminate, and fix a test. Addresses BIT-1641.
Clean up the logic for ssh_auth_failed. Addresses BIT-1641
Update baselines for adding a field to ssh.log as part of BIT-1641
Script-land changes for BIT-1641.
Change SSH.cc to use ssh_auth_attempted instead of ssh_auth_failed. Addresses BIT-1641.
Revert "Fixing duplicate SSH authentication failure events."
Create new SSH events ssh_auth_attempt and ssh_auth_result. Add auth_attempts to SSH::Info. Address BIT-1641.
I extended the tests a bit and did some small cleanups. I also moved the
SSH events back to the global namespace for backwards compatibility and
for consistency (the way it was at the moment, some of them were global
some SSH::).
Furthermore, I fixed the ssh_auth_result result event, it was only
raised in the success case. ssh_auth_result is now also checked in the
testcases. I also have a suspicion that the intel integration never
really worked before.
BIT-1641 #merged
- SMTP protocol headers now do some minimal parsing to clean up
email addresses.
- New function named split_mime_email_addresses to take MIME headers
and get addresses split apart but including the display name.
- Update tests.
We now extract email addresses in the fields that one would expect
to contain addresses. This makes further downstream processing of
these fields easier like log analysis or using these fields in the
Intel framework. The primary downside is that any other content
in these fields is no longer available such as full name and any
group information. I believe the simplification of the content in
these fields is worth the change.
Added "cc" to the script that feeds information from SMTP into the
Intel framework.
A new script for email handling utility functions has been created
as a side effect of these changes.
Intel notices are identified by a direction independent 3-tuple
(indicator, originator IP, responder IP). This allows notice
suppression. Additionally service and intel source are added to the
notice mail.
The extension mechanism is basically the one that Seth introduced with
his intel extensions. The main difference lies in using a hook instead
of an event. An example policy implements whitelisting.
This patch allows users to provide the fuid or the connection id
directly, in case they do not have access to either in the event that
they handle.
An example for this is the handling of certificates in SSL, where the
fa_file record cannot be retained because this would create a cyclic
data structure.
This patch also provides file IDs for hostname matches in certificates,
which was not possible with the previous API.
* origin/topic/johanna/x509-cn:
Use our new features to send the CN and SAN fields of certificates to the intel framework.
Do not log common name by default (it is most interesting for scripts) and add a test case.
extract most specific common name from certificates
BIT-1323 #merged
These functions are now deprecated in favor of alternative versions that
return a vector of strings rather than a table of strings.
Deprecated functions:
- split: use split_string instead.
- split1: use split_string1 instead.
- split_all: use split_string_all instead.
- split_n: use split_string_n instead.
- cat_string_array: see join_string_vec instead.
- cat_string_array_n: see join_string_vec instead.
- join_string_array: see join_string_vec instead.
- sort_string_array: use sort instead instead.
- find_ip_addresses: use extract_ip_addresses instead.
Changed functions:
- has_valid_octets: uses a string_vec parameter instead of string_array.
Addresses BIT-924, BIT-757.
IP addresses are often seen in the HTTP host field; this change checks if the value in the host field is a valid IP address and processes the Intel::seen event to check for an Intel::ADDR indicator.
* origin/topic/bernhard/file-analysis-x509:
Forgot the preamble for the new leak test
(hopefully) last change -> return real opaque vec instead of any_vec
Fix dump-events - it cannot be used with ssl anymore, because openssl does not give the same string results in all versions.
Finishing touches of the x509 file analyzer.
Revert change to only log certificates once per hour.
Change x509 log - now certificates are only logged once per hour.
Fix circular reference problem and a few other small things.
X509 file analyzer nearly done. Verification and most other policy scripts work fine now.
Add verify functionality, including the ability to get the validated chain. This means that it is now possible to get information about the root-certificates that were used to secure a connection.
Second try on the event interface.
Backport crash fix that made it into master with the x509_extension backport from here.
Make x509 certificates an opaque type
rip out x509 code from ssl analyzer. Note that since at the moment the file analyzer does not yet re-populate the info record that means quite a lot of information is simply not available.
parse out extension. One event for general extensions (just returns the openssl-parsed string-value), one event for basicconstraints (is a certificate a CA or not) and one event for subject-alternative-names (only DNS parts).
Very basic file-analyzer for x509 certificates. Mostly ripped from the ssl-analyzer and the topic/bernhard/x509 branch.
work fine now.
Todo:
* update all baselines
* fix the circular reference to the fa_file structure I introduced :)
Sadly this does not seem to be entirely straightforward.
addresses BIT-953, BIT-760
