This is a fairly straightforward change. Previously, users had no
control over whether this script was loaded. By relocating it to
policy, users can now choose whether or not this is necessary
functionality without modifying core Bro scripts.
In ContentLine_Analyzer, prevent excessively long lines being assembled.
The line length will default to just under 16MB, but can be overriden on
a per-analyzer basis. This is done for the finger,ident, and irc
analyzers.
With this commit, the data structure that is transfered for notice
suppression is much smaller than before, not including potentially
complex data structures like the fa_file record.
In all versions so far, the identifier string that was used for
comparisons might have been different from the identifier string that
was added (when certain notices are used).
This commit rewrites the way that weirds are logged and fixes a number
of issues on the way. Most prominently, flow weirds now actually log
information about the flow that they occur in (before this change, they
only logged the name of the weird, which is only marginally helpful).
Besides restructuring how weird logging works internally, weirds can now
also be generated by calling Weird::weird with the info record directly,
allowing more fine-granular passing of information. This is e.g. used
for DNS weirds, which do not have the connection record available any
more when they are generated (before data like the connection ID was
just not logged in these instances).
Addresses BIT-1578
I replaced a few strcmps with either calls to std::str.compare
or with the == operator of BroString.
Also changed two of the input framework tests that did not pass
anymore after the merge. The new SSH analyzer no longer loads the
scripts that let network time run, hence those tests failed because
updates were not propagated from the threads (that took a while
to find.)
* origin/topic/vladg/ssh: (25 commits)
SSH: Register analyzer for 22/tcp.
SSH: Add 22/tcp to likely_server_ports
SSH: Ignore encrypted packets by default.
SSH: Fix some edge-cases which created BinPAC exceptions
SSH: Add memleak btest
SSH: Update baselines
SSH: Added some more events for SSH2
SSH: Intel framework integration (PUBKEY_HASH)
Update baselines for new SSH analyzer.
Update SSH policy scripts with new events.
SSH: Add documentation
Refactoring ssh-protocol.pac:
SSH: Use the compression_algorithms const in another place.
Some cleanup and refactoring on SSH main.bro.
SSH: A bit of code cleanup.
Move SSH constants to consts.pac
SSH: Cleanup code style.
SSH: Fix some memleaks.
Refactored the SSH analyzer. Added supported for algorithm detection and more key exchange message types.
Add host key support for SSH1.
Add support for SSH1
Move SSH analyzer to new plugin architecture.
...
Conflicts:
scripts/base/protocols/ssh/main.bro
testing/btest/Baseline/core.print-bpf-filters/output2
testing/btest/Baseline/plugins.hooks/output
BIT-1344: #merged
This allows the path for the default filter to be specified explicitly
when creating a stream and reduces the need to rely on the default path
function to magically supply the path.
The default path function is now only used if, when a filter is added to
a stream, it has neither a path nor a path function already.
Adapted the existing Log::create_stream calls to explicitly specify a
path value.
Addresses BIT-1324
At the moment, we have support for:
elliptic_curves: client supported elliptic curves
ec_point_formats: list of client supported EC point formats
application_layer_protocol_negotiation: list of supported application layer protocols (used for spdy/http2 negotiation)
server_name: server name sent by client. This was supported before, but... a bit brittle.
The MHR script involves a "when" statement which can be expensive due to
the way it clones frames/vals. In this case, the fa_file record is
expensive to clone, but this change works around that by unrolling only
the necessary fields from it that are needed to populate a Notice::Info
record. A drawback to this is that the full fa_file or connection
records aren't available in the Notice::Info record when evaluating
Notice::policy hooks for MHR hit notices (though they can possibly be
recovered by using e.g. the lookup_connection() builtin_function).
* origin/topic/jazoff/notice_file_info:
Include file information in notices
BIT-1113 #merged
I'm wondering if there's a way to test creating these emails ... ?
Instead of storing the entire notice in Notice::suppressing,
just store the time the notice should be suppressed until.
This has the same functionality, except that end_suppression can no
longer be generated.
This cleans up most of the warnings from sphinx (broken :doc: links,
broxygen role misuses, etc.). The remaining ones should be harmless,
but not quick to silence.
I found that the README for each component was a copy from the actual
repo, so I turned those in to symlinks so they don't get out of date.
* origin/topic/seth/faf-updates: (27 commits)
Undoing the FTP tests I updated earlier.
Update the last two btest FAF tests.
File analysis fixes and test updates.
Fix a bug with getting analyzer tags.
A few test updates.
Some tests work now (at least they all don't fail anymore!)
Forgot a file.
Added protocol description functions that provide a super compressed log representation.
Fix a bug where orig file information in http wasn't working right.
Added mime types to http.log
Clean up queued but unused file_over_new_connections event args.
Add jar files to the default MHR lookups.
Adding CAB files for MHR checking.
Improve malware hash registry script.
Fix a small issue with finding smtp entities.
Added support for files to the notice framework.
Make the custom libmagic database a git submodule.
Add an is_orig parameter to file_over_new_connection event.
Make magic for emitting application/msword mime type less strict.
Disable more libmagic builtin checks that override the magic database.
...
Conflicts:
doc/scripts/DocSourcesList.cmake
scripts/base/init-bare.bro
scripts/test-all-policy.bro
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
So much nicer!
Closes#954.
* origin/topic/seth/notice-framework-updates:
Update notice framework documentation to represent the new reality.
Complete removal of the old table based notice policy mechanism.
Updates for the notices framework.
- Moved the Notice::notice event and Notice::policy table to both be hooks.
- Renamed the old Notice::policy to Notice::policy_table and documented it as deprecated.
Both local and global variables declared with "const" could be modified,
but now expressions that would modify them should generate an error
message at parse-time.
This could, for example, result in duplicate emails being sent (one from
manager and one from worker) if Notice::emailed_types is redef'd in
local.bro (or any script that gets loaded on all cluster nodes).
The problem was that Notice::policy is used to populate the internal
Notice::ordered_policy vector in a priority 10 bro_init handler (in
scripts/base/frameworks/notice/main.bro) and then that is what is used
when applying policy to notices. In order for
scripts/base/frameworks/notice/cluster.bro to prevent Notice::policy
from being used on non-manager nodes, it needs to clear it in a
bro_init hander of higher priority than 10.
* origin/topic/robin/pp-alarms:
The silliest, tiniest little whitespace fixes.
Update missing in last commit to this branch.
Adding test for alarm mail.
Tuning the pretty-printed alarms output.
* origin/topic/seth/notice-email-delay:
The hostname notice email extension works now.
Fixed more bugs with delayed emails.
Working around a problem with setting default container types.
Ugh, still major failure. I'm just cutting the timeout handling for now.
Fixed a small bug major problem with email delay timeout catching.
Initial fixes for the problem of async actions with notice email extensions.
Closes#727.
