With this commit, the data structure that is transfered for notice
suppression is much smaller than before, not including potentially
complex data structures like the fa_file record.
I replaced a few strcmps with either calls to std::str.compare
or with the == operator of BroString.
Also changed two of the input framework tests that did not pass
anymore after the merge. The new SSH analyzer no longer loads the
scripts that let network time run, hence those tests failed because
updates were not propagated from the threads (that took a while
to find.)
* origin/topic/vladg/ssh: (25 commits)
SSH: Register analyzer for 22/tcp.
SSH: Add 22/tcp to likely_server_ports
SSH: Ignore encrypted packets by default.
SSH: Fix some edge-cases which created BinPAC exceptions
SSH: Add memleak btest
SSH: Update baselines
SSH: Added some more events for SSH2
SSH: Intel framework integration (PUBKEY_HASH)
Update baselines for new SSH analyzer.
Update SSH policy scripts with new events.
SSH: Add documentation
Refactoring ssh-protocol.pac:
SSH: Use the compression_algorithms const in another place.
Some cleanup and refactoring on SSH main.bro.
SSH: A bit of code cleanup.
Move SSH constants to consts.pac
SSH: Cleanup code style.
SSH: Fix some memleaks.
Refactored the SSH analyzer. Added supported for algorithm detection and more key exchange message types.
Add host key support for SSH1.
Add support for SSH1
Move SSH analyzer to new plugin architecture.
...
Conflicts:
scripts/base/protocols/ssh/main.bro
testing/btest/Baseline/core.print-bpf-filters/output2
testing/btest/Baseline/plugins.hooks/output
BIT-1344: #merged
This allows the path for the default filter to be specified explicitly
when creating a stream and reduces the need to rely on the default path
function to magically supply the path.
The default path function is now only used if, when a filter is added to
a stream, it has neither a path nor a path function already.
Adapted the existing Log::create_stream calls to explicitly specify a
path value.
Addresses BIT-1324
The MHR script involves a "when" statement which can be expensive due to
the way it clones frames/vals. In this case, the fa_file record is
expensive to clone, but this change works around that by unrolling only
the necessary fields from it that are needed to populate a Notice::Info
record. A drawback to this is that the full fa_file or connection
records aren't available in the Notice::Info record when evaluating
Notice::policy hooks for MHR hit notices (though they can possibly be
recovered by using e.g. the lookup_connection() builtin_function).
* origin/topic/jazoff/notice_file_info:
Include file information in notices
BIT-1113 #merged
I'm wondering if there's a way to test creating these emails ... ?
Instead of storing the entire notice in Notice::suppressing,
just store the time the notice should be suppressed until.
This has the same functionality, except that end_suppression can no
longer be generated.
This cleans up most of the warnings from sphinx (broken :doc: links,
broxygen role misuses, etc.). The remaining ones should be harmless,
but not quick to silence.
I found that the README for each component was a copy from the actual
repo, so I turned those in to symlinks so they don't get out of date.
* origin/topic/seth/faf-updates: (27 commits)
Undoing the FTP tests I updated earlier.
Update the last two btest FAF tests.
File analysis fixes and test updates.
Fix a bug with getting analyzer tags.
A few test updates.
Some tests work now (at least they all don't fail anymore!)
Forgot a file.
Added protocol description functions that provide a super compressed log representation.
Fix a bug where orig file information in http wasn't working right.
Added mime types to http.log
Clean up queued but unused file_over_new_connections event args.
Add jar files to the default MHR lookups.
Adding CAB files for MHR checking.
Improve malware hash registry script.
Fix a small issue with finding smtp entities.
Added support for files to the notice framework.
Make the custom libmagic database a git submodule.
Add an is_orig parameter to file_over_new_connection event.
Make magic for emitting application/msword mime type less strict.
Disable more libmagic builtin checks that override the magic database.
...
Conflicts:
doc/scripts/DocSourcesList.cmake
scripts/base/init-bare.bro
scripts/test-all-policy.bro
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
So much nicer!
Closes#954.
* origin/topic/seth/notice-framework-updates:
Update notice framework documentation to represent the new reality.
Complete removal of the old table based notice policy mechanism.
Updates for the notices framework.
- Moved the Notice::notice event and Notice::policy table to both be hooks.
- Renamed the old Notice::policy to Notice::policy_table and documented it as deprecated.
* origin/topic/seth/notice-email-delay:
The hostname notice email extension works now.
Fixed more bugs with delayed emails.
Working around a problem with setting default container types.
Ugh, still major failure. I'm just cutting the timeout handling for now.
Fixed a small bug major problem with email delay timeout catching.
Initial fixes for the problem of async actions with notice email extensions.
Closes#727.
- $result is renamed to $action to reflect changes to the notice framework
since there is already another result-like field ($suppress_for) and
there may be more in the future.
- Slipped in a change to add connection information to notice emails too.
- Fixed the notice_policy log.
- Predicate functions are now logged.
- Predicate functions are now optional. If not given, it's assumed that
the result should always apply.
- Worker raised notices are printed a single time by the manager now.
- Cluster/notices integration cleaned up.
- New tests for cluster/notice integration.
- Notice suppression fixes and tests.
- The test needed to append the second notice the event queue so that
the notice would be suppressed correctly.
- Setting $suppress_for manually in apply_policy since it doesn't pick
up the &default when directly creating the record inside the NOTICE call.
- Duplicate notices are discovered with the new Notice::Info
field $identifier. It's a string that is left up to the
notice implementor to define which would indicate a
fundamentally duplicate notice. The field is optional and
if it's not included it's not possible for notice
suppression to take place.
- Duplicate notices are suppressed by default for the interval
defined by the Notice::default_suppression_interval variable
(1 hour by default).
- A new notice action was defined ACTION_NO_SUPPRESS to prevent
suppression for a specific notice instance. A convenience set
named not_suppressed_types was also created to not suppress
entire notice types.
- A new field was added to the PolicyItem type to modify the length
of time a notice should be suppressed if the predicate matches.
The field is named $suppress_for. This name makes the code more
readable like this: $suppress_for = 1day
- New events were created to give visibility into the notice
framework's suppression activity.
- event Notice::begin_suppression(n: Notice::Info)
- event Notice::suppressed(n: Notice::Info)
- event Notice::end_suppression(n: Notice::Info)
- The suppression.bro script doesn't have a baseline because
it is causing a segfault in Bro. This one test is the
reason that this is being integrated into a branch instead
of master.
Log rotation is now controlled directly through Filter records.
Also addressed a TODO in the default_path_func regarding the
LogMgr::AddFilter function generating internal filter path
suggestions/fallbacks. Now, if the user doesn't explicitly set a filter
path, the filter's path will be the result of the first call to
default_path_func (happens during the first write to the log). And in
that case the path suggestion argument to the path_func is an empty
string.
- Log path's are generated in the scripting land
now. The default Log stream ID to path string
mapping works like this:
- Notice::LOG -> "notice"
- Notice::POLICY_LOG -> "notice_policy"
- TestModule::LOG -> "test_module"
- Logging streams updated across all of the shipped
scripts to be more user friendly. Instead of
the logging stream ID HTTP::HTTP, we now have
HTTP::LOG, etc.
- The priorities on some bro_init handlers have
been adjusted to make the process of applying
filters or disabling streams easier for users.
