'only_single_header_row' that turns the output into CSV format.
In that mode all meta data is skipped except for a single header line
with the fields names. Example:
local my_filter: Log::Filter = [$name = "my-filter", $writer = Log::WRITER_ASCII, $config = table(["only_single_header_row"] = "T")];
Contributed by Carsten Langer.
Instead of these events being generated for invalid byte count values
(they should always be even, not odd), a protocol_violation is raised.
modbus_read_holding_registers_response
modbus_read_input_registers_response
modbus_write_multiple_registers_request
modbus_read_write_multiple_registers_request
modbus_read_write_multiple_registers_response
modbus_read_fifo_queue_respons
For modbus message types that include variable amount of register values
(uint16[]), setting a &length attribute without an explicit array size
could trigger a parsing assertion since it allows for the "element" data
pointer to travel past the "end of data" (e.g. when &length is odd).
This is changed to now give both an array size and &length to earlier
terminate the parsing of elements before the assert is checked and
so a single out-of-bound check can be done for the entire array
(leaving off &length causes an out-of-bound check for each element).
Added another parameter to modbus events that carry register arrays to
the script-layer which indicates the associated byte count from the
message (allowing for invalid values to be detected):
modbus_read_holding_registers_response
modbus_read_input_registers_response
modbus_write_multiple_registers_request
modbus_read_write_multiple_registers_request
modbus_read_write_multiple_registers_response
modbus_read_fifo_queue_response
- Added a test for binpac exception handling -- the generated code
should use "binpac::Exception" and not "Exception" for exception
handling logic to avoid accidental overshadowing by
the analyzer-specific type "binpac::ModbusTCP::Exception", which
could lead to interesting asserts being triggered in binpac.
- Update baseline for the event coverage test -- seems that more
events get generated with working exception handling in the generated
binpac parser code.
- Coverage baseline was giving wrong number of events covered.
* topic/robin/intel-framework-merge: (22 commits)
Fixing tests after intel-framework merge.
Extracting URLs from message bodies over SMTP and sending them to Intel framework.
Small comment updates in the Intel framework CIF support.
Intelligence framework documentation first draft.
Only the manager tries to read files with the input framework now.
Initial support for Bro's Intel framework with the Collective Intelligence Framework.
Initial API for Intel framework is complete.
Fixed an issue with cluster data distribution.
Updating some intel framework test baselines.
Reworked cluster intelligence data distribution mechanism and fixed tests.
Lots more intelligence checking in SMTP traffic.
Added intelligence check for "Received" path checking and a bit of reshuffling.
Added sources to the intel log.
Fixing a problem with intel distribution on clusters.
Updated intel framework test to include matching.
Restructuring the scripts that feed data into the intel framework slightly.
One test for cluster transparency of the intel framework.
Fixed a cluster support bug.
Intelligence framework checkpoint
Major updates to fix the Intel framework API.
...
Closes#914.
* remotes/origin/topic/seth/modbus-merge:
Small modbus documentation update and tiny refactoring.
Final touches to modbus analyzer for now.
Major revisions to Modbus analyzer support (not quite done yet).
put some make-up on Modbus analyser
Modbus analyser, added support: FC=20,21
Modbus analyzer,added support: FC=1,2,15,24
Modbus analyzer, current support: FC=3,4,5,6,7,16,22,23
Closes#915.
* origin/topic/seth/intel-framework: (21 commits)
Extracting URLs from message bodies over SMTP and sending them to Intel framework.
Small comment updates in the Intel framework CIF support.
Intelligence framework documentation first draft.
Only the manager tries to read files with the input framework now.
Initial support for Bro's Intel framework with the Collective Intelligence Framework.
Initial API for Intel framework is complete.
Fixed an issue with cluster data distribution.
Updating some intel framework test baselines.
Reworked cluster intelligence data distribution mechanism and fixed tests.
Lots more intelligence checking in SMTP traffic.
Added intelligence check for "Received" path checking and a bit of reshuffling.
Added sources to the intel log.
Fixing a problem with intel distribution on clusters.
Updated intel framework test to include matching.
Restructuring the scripts that feed data into the intel framework slightly.
One test for cluster transparency of the intel framework.
Fixed a cluster support bug.
Intelligence framework checkpoint
Major updates to fix the Intel framework API.
Checkpoint commit. This is all a huge mess right now. :)
...
Closes#914.
* origin/topic/jsiwek/gridftp:
Add memory leak unit test for GridFTP.
Enable GridFTP detection by default. Track/log SSL client certs.
Add analyzer for GSI mechanism of GSSAPI FTP AUTH method.
Add an example of a GridFTP data channel detection script.
more cases.
It will now not only fire after table-reads have been completed,
but also after the last event of a whole-file-read (or whole-db-read, etc.).
The interface also has been extended a bit to allow readers to
directly fire the event should they so choose. This allows the
event to be fired in direct table-setting/event-sending modes,
which was previously not possible.
In the *service* field of connection records, GridFTP control channels
are labeled as "gridftp" and data channels as "gridftp-data".
Added *client_subject* and *client_issuer_subject* as &log'd fields to
SSL::Info record. Also added *client_cert* and *client_cert_chain*
fields to track client cert chain.
- Intel data distribution on clusters is now pushed in whole
by the manager when a worker connects. Additions after that point
are managed by the normal single-item distribution mechanism already
built into the intelligence framework.
- The manager maintains the complete "minimal" data store that the
workers use to do their matching so that full "minimal" data
distribution is very easy.
- Tests are cleaned up and work.
without a final \0 - which means that strings read by the input framework are
unusable by basically all internal functions (like to_count).
the basic test now also checks this.
Thanks at Sheharbano for noticing this.
It relies on the heuristics of GridFTP data channels commonly default to
SSL mutual authentication with a NULL bulk cipher and that they usually
transfer large datasets (default threshold of script is 1 GB). The
script also defaults to skip_further_processing() after detection to try
to save cycles analyzing the large, benign connection.
Also added a script in base/protocols/conn/polling that generalizes the
process of polling a connection for interesting features. The GridFTP
data channel detection script depends on it to monitor bytes
transferred.
Sometimes manager node was shutting everything down before others had
a chance to generate output. It now waits for all nodes to fully
connect with each other.
Not the greatest solution, but makes the 3 bro processes more likely to
run sequentially so that the controller2 process doesn't happen to be
scheduled before the controller process. In that case, the controllee
gets the shutdown request before the configuration update. FreeBSD
especially seemed to schedule them the unintended way frequently.
* origin/topic/dina/modbus:
put some make-up on Modbus analyser
Modbus analyser, added support: FC=20,21
Modbus analyzer,added support: FC=1,2,15,24
Modbus analyzer, current support: FC=3,4,5,6,7,16,22,23
I cleaned up the code a bit, mainly layout style.
I did not include the *.bro scripts for now, but a test script
../testing/btest/scripts/base/protocols/modbus/events.bro that prints
out the value for each event.
Merged the Modbus traces from the ics repository into a single trace
as input for the test. They currently trigger 20 of the 34 events.
Addresses #870.
* remotes/origin/topic/bernhard/input-warn-on-invalid-numbers:
...and another small change to error handling -> now errors in single lines do not kill processing, but simply ignore the line, log it, and continue.
Ok, this one was a little bit sneaky.
ok, this one might really be a bit too big for 2.1
* origin/fastpath:
Ok, this one is not really necessary for 2.1 and more of a nice-to-have
another small bug found while searching for something else...
Fix two little bugs:
sorry. the patch for the set_separator.
make set_separators different from , work for input framework.
Bug found bei Keith & Seth: input framework was not handling counts and ints out of 32-bit-range correctly.
Before this patch, empty values were not hashed at all. Which had the unfortunate side-effect
that e.g. the lines
TEST -
and
- TEST
have the same hash values. On re-reads that means that the change will
be ignored.
This is probably pretty academic, but this patch changes it and adds a testcase.
Output of the reread test changes due to re-ordering of the output (probably
due to the fact that the internal hash values are changed and thus transferred
in a different order)
Escaped ,'s in sets and vectors were unescaped before tokenization
Handling of zero-length-strings as last element in a set was broken (sets ending with a ,).
Hashing of lines just containing zero-length-strings was broken (now a \0 is appended to each
string before it is hashed - giving us a hash of something for a line just consisting of \0s.
This also allows to differentiate between vectors with varying numbers of zero-length-strings).
* origin/topic/bernhard/input-allow_invalid_types:
to be sure - add a small assertion
add an option to the input framework that allows the user to chose to not die upon encountering files/functions.
That's the last feature for 2.1!
If the metrics break interval happened to occur between first and second
worker starting up and getting connected to the cluster, the test
would fail because the second worker didn't get a chance to connect and
send data. The test now waits for the cluster setup to complete
before workers send metrics data.
