* origin/topic/seth/dhcp-update:
Rework to the DHCP analyzer.
First step of DHCP analyzer rearchitecture.
Add .btest scripts for dhck_ack and dhcp_discover messages verifying that new options are correctly reported in dhcp.log records.
Extend DHCP protocol analyzer with new options.
BIT-1924 #merged
Additional changes:
* Removed known-hosts.bro as the only thing populating its table was
the already-removed known-hosts-and-devices.bro. So a
known_devices.log will no longer be generated.
* In dhcp-options.pac, the process_relay_agent_inf_option had a memleak
and also process_auto_proxy_config_option looked like it accessed one
byte past the end of the available bytestring, so fixed those.
Removed duplicated entries (`000001a0-0000-0000-c000-000000000046`, with `i=3, i=4`) and related comments line
This change should make this file more readable and it will be easier to convert it to other formats (personally I found it out while I was trying to convert this to a JavaScript structure).
The way in which TLS 1.3 is negotiated was changed slightly in later
revisions of the standard. The final version is only sent in an
extension - while the version field in the server hello still shows TLS
1.2.
This patch makes ssl.log show the correct version again.
Highlights:
- Reduced all DHCP events into a single dhcp_message event. (removed legacy events since they weren't widely used anyway)
- Support many more DHCP options.
- DHCP log is completely reworked and now represents DHCP sessions
based on the transaction ID (and works on clusters).
- Removed the known-devices-and-hostnames script since it's generally
less relevant now with the updated log.
Closes BIT-1900.
* origin/topic/johanna/config:
Use port_mgr->Get() in the input framework config changes.
Allow the empty field separator to be empty; use in config framework.
Fix small bug in config reader.
Fix segmentation fault when parsing sets containing invalid elements.
Add config framework.
* 'smb-transaction-messages' of https://github.com/jbencteux/bro:
add test for smb1_com_transaction_response event changes
add test for smb1_com_transaction2_secondary_request event changes
add test for smb1_com_transaction2_request event changes
add test for smb1_com_transaction_secondary_request event changes
add test for smb1_com_transaction_request event changes
fix setup field handling in smb1_com_transaction_request messages
fix smb1_com_transaction* messages
add smb1_transaction2_secondary_request event
add smb1_transaction_secondary_request event
add parameters and data to smb1_transaction_request/response messages
add SMB_Parameters.Words to smb1_transaction2_request event
* 'nfs-updates' of https://github.com/dtrejod/bro:
Format print nfs units tests to improve output readability. Add unit tests for new NFS events -- nfs_proc_symlink, nfs_proc_link, nfs_proc_sattr.
Bug fix: nfs3_writeargs didn't properly return filehandle.
Add nfs_proc_symlink, nfs_proc_link, nfs_proc_sattr.
* 'mount-protocol' of https://github.com/dtrejod/bro:
Add unit tests for new MOUNT events -- mount_proc_mnt, mount_proc_umnt, mount_proc_umnt_all, mount_proc_not_implemented.
Add mount_proc_null, mount_proc_mnt, mount_proc_umnt, mount_proc_umnt_all, mount_proc_not_implemented, mount_reply_status.
Old event prototypes have changed and the events are broken right
now and may be removed in favor of the new generic "dhcp_message"
event.
DHCP option parsing is abstracted from the main code base of the
protocol parser and are all now located in their own file.
Documentation, tests, and final code cleanup are still pending.
This small change allows the empty field separator to be empty. This
means that we can represent an empty list by a empty input string, which
was not possible before.
Before, an empty empty field separator meant that there is no empty
field - to get back to this behavior one now has to set the empty field
separator to a string that is guaranteed to not be part of the input
data. Note that we did not use "empty" empty field separators anywhere
and I am not aware of this being used by anyone - the new behavior seems
like it is much more useful in practice.
This also changes the config framework to interpret empty lists as...
empty, instead of interpreting them as lists that have one zero-length
element; this seems like the saner default.
* Add proper namespace scoping to a 'SumStats::process_epoch_result'
scheduled event.
* Fix iterator invalidation within 'SumStats::process_epoch_result'
* Give 'SumStats::process_epoch_result' a copy of the result table so
that the SumStats framework can clear the original and move on to the
next epoch immediately.
* The previous baseline of the basic sumstats unit test did look wrong
to me and probably was actually indicative of the iterator invalidation
problem.
Thanks to Jim Mellander for reporting the issues.
* origin/topic/feature/logging-filter-list:
Logging: implement get_filter_names and small fixes.
Removed some superfluous existence checks before deleting table indices.
BIT-1890 #merged
get_filter_names(id: ID) : set[string] returns the names of the current
list of filters for a specified log stream.
Furthermore this commit makes a number of logging functions more robust
by checking existence of values before trying to modify them. This
commit also really implements (and tests) the enable_stream function.
This makes conn.logs a bit prettier (and smaller) because all lines that
do not use a tunnel will now have a "-" instead of the "(empty)" for
tunnel_parents.
There are two new script level functions to query and lookup files
from the core by their IDs. These are adding feature parity for
similarly named functions for files. The function prototypes are
as follows:
Files::file_exists(fuid: string): bool
Files::lookup_File(fuid: string): fa_file
Add the folowing option types:
- 55 Parameters Request List;
- 58 Renewal time;
- 59 Rebinding time;
- 61 Client Identifier;
- 82 Relay Agent Information.
Extend the following events with new parameters, specifically:
- dhcp_discover exports client identifier and parameters request list;
- dhcp_request exports client_identifier and parameters request list;
- dhcp_ack exports rebinding time, renewal time and list of suboptions value of
dhcp relay agent information option;
- dhcp_inform exports parameters request list.
Add option type specific variables within the scope of DHCP module
(see src/analyzer/protocol/dhcp/types.bif).
Move protocol specific variables "dhcp_msg" and "dhcp_router_list" from scope Global to DHCP::
and adapt inet_net_var in src/NetVar.cc consequently.
Extend src/analyzer/protocols/dhcp/main.bro to handle the new events and to log
dhcp_ack, dhcp_request and dhcp_discover.
Modify scripts/policy/protocols/dhcp/known-devices-and-hostnames.bro to
include new events' variables.
These have been lingering for a while and they generally annoy
everyone because of the sheer volume. They also don't really add
any useful information for debugging and they were generated differently
than most other weirds anyway (which was a little weird...).
This introduces a new option, SOCKS::default_capture_password which can
be used to specify if Socks passwords are logged by default
Like fot FTP/HTTP, this option is set to false by default.
Addresses BIT-1791
The configuration framework consists of three mostly distinct parts:
* option variables
* the config reader
* the script level framework
I will describe the three elements in the following.
Internally, this commit also performs a range of changes to the Input
manager; it marks a lot of functions as const and introduces a new
ValueToVal method (which could in theory replace the already existing
one - it is a bit more powerful).
This also changes SerialTypes to have a subtype for Values, just as
Fields already have it; I think it was mostly an oversight that this was
not introduced from the beginning. This should not necessitate any code
changes for people already using SerialTypes.
option variable
===============
The option keyword allows variables to be specified as run-tine options.
Such variables cannot be changed using normal assignments. Instead, they
can be changed using Option::set. It is possible to "subscribe" to
options and be notified when an option value changes.
Change handlers can also change values before they are applied; this
gives them the opportunity to reject changes. Priorities can be
specified if there are several handlers for one option.
Example script:
option testbool: bool = T;
function option_changed(ID: string, new_value: bool): bool
{
print fmt("Value of %s changed from %s to %s", ID, testbool, new_value);
return new_value;
}
event bro_init()
{
print "Old value", testbool;
Option::set_change_handler("testbool", option_changed);
Option::set("testbool", F);
print "New value", testbool;
}
config reader
=============
The config reader provides a way to read configuration files back into
Bro. Most importantly it automatically converts values to the correct
types. This is important because it is at least inconvenient (and
sometimes near impossible) to perform the necessary type conversions in
Bro scripts themselves. This is especially true for sets/vectors.
Configuration generally look like this:
[option name][tab/spaces][new variable value]
so, for example:
testaddr 2607:f8b0:4005:801::200e
testinterval 60
testtime 1507321987
test_set a b c d erdbeerschnitzel
The reader uses the option name to look up the type that variable has in
the Bro core and automatically converts the value to the correct type.
Example script use:
type Idx: record {
option_name: string;
};
type Val: record {
option_val: string;
};
global currconfig: table[string] of string = table();
event InputConfig::new_value(name: string, source: string, id: string, value: any)
{
print id, value;
}
event bro_init()
{
Input::add_table([$reader=Input::READER_CONFIG, $source="../configfile", $name="configuration", $idx=Idx, $val=Val, $destination=currconfig, $want_record=F]);
}
Script-level config framework
=============================
The script-level framework ties these two features together and makes
them a bit more convenient to use. Configuration files can simply be
specified by placing them into Config::config_files. The framework also
creates a config.log that shows all value changes that took place.
Usage example:
redef Config::config_files += {configfile};
export {
option testbool : bool = F;
}
The file is now monitored for changes; when a change occurs the
respective option values are automatically updated and the value change
is written to config.log.
This commit fixes a few small issues.
* server key exchange parameters are only parsed when a named curve is
given.
* I removed the ssl-verbose.bro and moved the functionality into the
testcase.
The information that we get with these events is likely irrelevant to
the majority of Bro users; I do not think that we have to ship a
script that uses them by default. A script like this would be
something to publish via the Bro package manager instead; this is the
approach that we have taken with a number of the recent SSL addition.
* I marked the ssl_server_curve event as deprecated. More information is
contained in the new ssl_ecdh_server_params event.
This is an events that is probably seldomly (or never) directly used
by anyone; I plan to completely remove it right after the 2.6 release.
In ContentLine_Analyzer, prevent excessively long lines being assembled.
The line length will default to just under 16MB, but can be overriden on
a per-analyzer basis. This is done for the finger,ident, and irc
analyzers.
a broctl print triggers this error
Reporter::ERROR no such index (Cluster::nodes[Intel::p$descr])
/usr/local/bro/share/bro/base/frameworks/intel/./cluster.bro, line 39
when broctl connects p$descr is empty. It should probably be set to
'control' somewhere inside broctl, but that would only fix broctl, not
other clients.
diff --git a/aux/bro-aux b/aux/bro-aux
index 02f710a43..43f4b90bb 160000
--- a/aux/bro-aux
+++ b/aux/bro-aux
@@ -1 +1 @@
-Subproject commit 02f710a436dfe285bae0d48d7f7bc498783e11a8
+Subproject commit 43f4b90bbaf87dae1a1073e7bf13301e58866011
diff --git a/aux/broctl b/aux/broctl
index e960be2c1..d3e6cdfba 160000
--- a/aux/broctl
+++ b/aux/broctl
@@ -1 +1 @@
-Subproject commit e960be2c192a02f1244ebca3ec31ca57d64e23dc
+Subproject commit d3e6cdfba496879bd55542c668ea959f524bd723
diff --git a/aux/btest b/aux/btest
index 2810ccee2..e638fc65a 160000
--- a/aux/btest
+++ b/aux/btest
@@ -1 +1 @@
-Subproject commit 2810ccee25f6f20be5cd241155f12d02a79d592a
+Subproject commit e638fc65aa12bd136594451b8c185a7a01ef3e9a
diff --git a/scripts/base/frameworks/intel/cluster.bro b/scripts/base/frameworks/intel/cluster.bro
index 820a5497a..e75bdd057 100644
--- a/scripts/base/frameworks/intel/cluster.bro
+++ b/scripts/base/frameworks/intel/cluster.bro
@@ -32,7 +32,7 @@ event remote_connection_handshake_done(p: event_peer)
{
# When a worker connects, send it the complete minimal data store.
# It will be kept up to date after this by the cluster_new_item event.
- if ( Cluster::nodes[p$descr]$node_type == Cluster::WORKER )
+ if ( p$descr in Cluster::nodes && Cluster::nodes[p$descr]$node_type == Cluster::WORKER )
{
send_id(p, "Intel::min_data_store");
}
The bytes_threshold_crossed event in the gridftp analyzer is not first
checking to see if the connection passed the initial criteria. This
causes the script to add the gridftp-data service to any connection that
crosses a threshold that is the same as or greater than the gridftp
size_threshold.
When Bro writes a compressed log, it uses a file extension of ".gz".
However, upon log rotation the ascii writer script function
"default_rotation_postprocessor_func" was discarding the ".gz"
file extension. Fixed so that the correct file extension is
preserved after rotation.
Closes#1830.
* origin/topic/johanna/ocsp-sct-validate: (82 commits)
Tiny script changes for SSL.
Update CT Log list
SSL: Update OCSP/SCT scripts and documentation.
Revert "add parameter 'status_type' to event ssl_stapled_ocsp"
Revert "parse multiple OCSP stapling responses"
SCT: Fix script error when mime type of file unknown.
SCT: another memory leak in SCT parsing.
SCT validation: fix small memory leak (public keys were not freed)
Change end-of-connection handling for validation
OCSP/TLS/SCT: Fix a number of test failures.
SCT Validate: make caching a bit less aggressive.
SSL: Fix type of ssl validation result
TLS-SCT: compile on old versions of OpenSSL (1.0.1...)
SCT: Add caching support for validation
SCT: Add signed certificate timestamp validation script.
SCT: Allow verification of SCTs in Certs.
SCT: only compare correct OID/NID for Cert/OCSP.
SCT: add validation of proofs for extensions and OCSP.
SCT: pass timestamp as uint64 instead of time
Add CT log information to Bro
...
