So much nicer!
Closes#954.
* origin/topic/seth/notice-framework-updates:
Update notice framework documentation to represent the new reality.
Complete removal of the old table based notice policy mechanism.
Updates for the notices framework.
- Moved the Notice::notice event and Notice::policy table to both be hooks.
- Renamed the old Notice::policy to Notice::policy_table and documented it as deprecated.
Added a generic gtpv1_message event generated for any GTP message type.
Added specific events for the create/update/delete PDP context
request/response messages.
Addresses #934.
* origin/topic/matthias/notary:
Small cosmetic changes.
Give log buffer the correct name.
Simplify delayed logging of SSL records.
Implement delay-token style SSL logging.
More style tweaks: replace spaces with tabs.
Factor notary code into separte file.
Adhere to Bro coding style guidelines.
Enhance ssl.log with information from notary.
Closes#928
* topic/robin/exit-after-terminate:
Updating submodule(s).
Fixing exit-after-terminate when used with bare mode.
New option exit_only_after_terminate to prevent Bro from exiting.
These cases should be avoidable by fixing scripts where they occur and
they can also help catch typos that would lead to unintentional runtime
behavior.
Adding this already revealed several scripts where a field in an inlined
record was never removed after a code refactor.
* origin/topic/bernhard/input-logging-commmon-functions:
add the last of Robins suggestions (separate info-struct for constructors).
port memory leak fix from master
harmonize function naming
move AsciiInputOutput over to threading
and thinking about it, ascii-io doesn't need the separator
change constructors
and factor stuff out the input framework too.
factor out ascii input/output.
std::string accessors to escape_sequence functionality
intermediate commit - it has been over a month since I touched this...
I cleaned up the AsciiInputOutput class somewhat, including renaming
it to AsciiFormatter, renaming some of its methods, and turning the
static methods into members for consistency.
Closes#929.
Moved this functionality to be internal instead of in the script-layer
event handlers. The issue with the later is that bad things can happen
between the time a reporter event handler is dispatched and the time it
is executed, and if bro crashes in that time, the message may never be
seen/logged.
Addressed #930 (and revisits #836).
This commit moves the notary script into the policy directory, along with some
architectural changes: the main SSL script now has functionality to add and
remove tokens for a given record. When adding a token, the script delays the
logging until the token has been removed or until the record exceeds a maximum
delay time.
As before, the base SSL script stores all records sequentially and buffers even
non-delayed records for the sake of having an ordered log file. If this turns
out to be not so important, we can easily revert to a simpler logic.
(This is still WiP, some debuggin statements still linger.)
There exists one complication: the new file notary.bro requires the definition
of the SSL::Info record, but as does main.bro. Because I did not really know
where to put the common code (it's not a constant, so ssl/const.bro does not
really fit), I put it into __load.bro__ so that it sticks out for now. If
anybody has an idea how to solve this elegantly, please let me know.
* origin/topic/matthias/opaque:
Add new unit test for opaque serialization.
Migrate entropy testing to opaque.
C++ify RandTest.*
Fix a hard-to-spot bug.
Use more descriptive error message.
Fix the fix :-/.
Fix initialization of hash values.
Be clearer about delegation.
Implement serialization of opaque types.
Update hash BiF documentation.
Migrate free SHA* functions to SHA*Val::digest().
Add missing type name that caused failing tests.
Update base scripts and unit tests.
Simplify hash function BiFs.
Add support for opaque hash values.
Adapt BiF & Bro parser to handle opaque types.
More lexer/parser work.
Implement equivalence relation for opaque types.
Support basic serialization of opaque.
Add opaque type to lexer, parser, and BroType.
Closes#925
Conflicts:
aux/broccoli
* origin/fastpath:
Fix memory leak in ascii input reader.
Improvements for the "bad checksums" detector to make it detect bad TCP checksums.
Improved file name extraction for SMTP when file name is included in Content-Type header.
Small tweak to "bad checksum" script to avoid potential division by
zeros.
Both local and global variables declared with "const" could be modified,
but now expressions that would modify them should generate an error
message at parse-time.
consisting of letters [A-Za-z].
I had some bogus HTTP sessions now with the test-suite that reported
data as HTTP because it started with "<!... ". Requiring letters seems
a reasonable constraint.
* vlad/topic/vladg/http-verbs:
A test for HTTP methods, including some horribly illegal requests.
Remove hardcoded HTTP verbs from the analyzer (#741)
I added a "bad_HTTP_request" weird for HTTP request lines that don't
have more than a single word.
Closes#741.
First step - factored out everything the logging classes
use ( so only output ).
Moved the script-level configuration to logging/main,
and made the individual writers just refer to it -
no idea if this is good design. It works. But I am happy
about opinions :)
Next step - add support for input...
'only_single_header_row' that turns the output into CSV format.
In that mode all meta data is skipped except for a single header line
with the fields names. Example:
local my_filter: Log::Filter = [$name = "my-filter", $writer = Log::WRITER_ASCII, $config = table(["only_single_header_row"] = "T")];
Contributed by Carsten Langer.
* topic/robin/intel-framework-merge: (22 commits)
Fixing tests after intel-framework merge.
Extracting URLs from message bodies over SMTP and sending them to Intel framework.
Small comment updates in the Intel framework CIF support.
Intelligence framework documentation first draft.
Only the manager tries to read files with the input framework now.
Initial support for Bro's Intel framework with the Collective Intelligence Framework.
Initial API for Intel framework is complete.
Fixed an issue with cluster data distribution.
Updating some intel framework test baselines.
Reworked cluster intelligence data distribution mechanism and fixed tests.
Lots more intelligence checking in SMTP traffic.
Added intelligence check for "Received" path checking and a bit of reshuffling.
Added sources to the intel log.
Fixing a problem with intel distribution on clusters.
Updated intel framework test to include matching.
Restructuring the scripts that feed data into the intel framework slightly.
One test for cluster transparency of the intel framework.
Fixed a cluster support bug.
Intelligence framework checkpoint
Major updates to fix the Intel framework API.
...
Closes#914.
* remotes/origin/topic/seth/modbus-merge:
Small modbus documentation update and tiny refactoring.
Final touches to modbus analyzer for now.
Major revisions to Modbus analyzer support (not quite done yet).
put some make-up on Modbus analyser
Modbus analyser, added support: FC=20,21
Modbus analyzer,added support: FC=1,2,15,24
Modbus analyzer, current support: FC=3,4,5,6,7,16,22,23
Closes#915.
* origin/topic/seth/intel-framework: (21 commits)
Extracting URLs from message bodies over SMTP and sending them to Intel framework.
Small comment updates in the Intel framework CIF support.
Intelligence framework documentation first draft.
Only the manager tries to read files with the input framework now.
Initial support for Bro's Intel framework with the Collective Intelligence Framework.
Initial API for Intel framework is complete.
Fixed an issue with cluster data distribution.
Updating some intel framework test baselines.
Reworked cluster intelligence data distribution mechanism and fixed tests.
Lots more intelligence checking in SMTP traffic.
Added intelligence check for "Received" path checking and a bit of reshuffling.
Added sources to the intel log.
Fixing a problem with intel distribution on clusters.
Updated intel framework test to include matching.
Restructuring the scripts that feed data into the intel framework slightly.
One test for cluster transparency of the intel framework.
Fixed a cluster support bug.
Intelligence framework checkpoint
Major updates to fix the Intel framework API.
Checkpoint commit. This is all a huge mess right now. :)
...
Closes#914.
- I added this to the base scripts because it's very minimal overhead
and we get questions about this all the time. Now that reporter
messages are printed to the console, people will see the message
clearly if they have bad checksums.
- There are still some broken events in the modbus analyzer because
I don't have traffic to test with (coil and record related events primarily).
- There are a few example scripts in policy/protocols/modbus
