* origin/topic/gilbert/ascii-header:
Updated tests; removed net type from type conversion code.
Updated header format (see #558)
Header modification to LogWriterAscii to make it easier for scripts to understand bro log files.
Notes:
- I've refactored the code a bit, also adapting the style a bit.
Also edited the header format slightly.
- I'm skipping the testing/btest/profiles directory, which seems
unrelated.
- I'm also skipping the baseline updates as they weren't
up-to-date anymore. Will update them in a subsequent commit.
- Log path's are generated in the scripting land
now. The default Log stream ID to path string
mapping works like this:
- Notice::LOG -> "notice"
- Notice::POLICY_LOG -> "notice_policy"
- TestModule::LOG -> "test_module"
- Logging streams updated across all of the shipped
scripts to be more user friendly. Instead of
the logging stream ID HTTP::HTTP, we now have
HTTP::LOG, etc.
- The priorities on some bro_init handlers have
been adjusted to make the process of applying
filters or disabling streams easier for users.
This can be used from a Bro script to cause Bro to exit with non-zero
status and a message to stderr. Uses of the exit() BIF in scripts
for terminating Bro because of an error were replaced with this one,
which addresses #564.
- Since each host in a cluster has it's own view of the metrics
the only time the manager would get a chance for a global view
is the break_interval. This update improves that time. If a
worker crosses 10% of the full threshold, it will send it's
value to the manager which can then ask the rest of the cluster
for a global view. The manager then adds all of the values for
each workers metric indexes together and will do the notice
if it crosses the threshold so that it isn't dependent on
waiting for the break interval to hit. This functionality
works completely independently of the break_interval too. Logging
will happen as normal.
- Small update for SSH bruteforcer detection to match additions in
the metrics framework API.
- The hope is that this update is mostly invisible from anyone's
perspective. The only affect it should have on users is to better
the detection of metric values crossing thresholds on cluster
deployments.
- fixing some Metrics::add_data() call signatures
- slight refactors to cluster framework @if and adding a NONE NodeType for
so local_node_type() will return that instead of just emitting an error
when cluster mode isn't enabled
- `make restdoc` target now uses bro's bare-mode
- bro scripts generated from bifs now really only live in the build/src/base/
directory and changed the DocSourcesList.cmake to dynamically figure out
what bifs exist by looking in src/ instead of build/src/
- add some missing @load dependencies
- Metrics now work on cluster deployments with no caveats. It should be
completely transparent. Intermediate updates to speed some detection
will come later.
* origin/topic/jsiwek/autodoc-fixes:
Update doc sources and touch up a few script comments.
Fixes for script auto-documentation.
Conflicts:
scripts/base/frameworks/logging/main.bro
- New ACTION_ADD_GEODATA to add geodata to notices in an extension
field named remote_location.
- Loading extend-email/hostnames by default now that it only
does anything when the ACTION_EMAIL action is applied (finally).
The main script now uses the Reporter::error BIF instead of scheduling
a reporter_error event to report an invalid node name. The later only
works if the reporter framework is loaded, but the BIF will do the right
thing and use stderr if there's no event handler.
I also @if'd out most of the setup-connections script when the node is
invalid because that's what the cluster/__load__.bro would normally do.
- Metrics API is much more similar to the Logging framework's API now.
- Filters define all output and metrics collection now.
- Initial attempt at thresholding and generating notices.
- Fixing the parts of the `make restdoc` and `make doc` process that were
broken by the last Bro script re-organization
- Generated documentation for Bro scripts derived from BiFs now use the
original BiF source file as the "original source file" link
- Renaming of the internal POLICYDEST definition and other misc places that
refer to "policy" scripts; that terminology doesn't make total sense now
- Added a documentation blacklist reminder test that will fail if there's
scripts that are blacklisted from being documentated because they're still
in progress
- Some minor Bro script changes to fix small @load dependency errors
Addresses #543
* topic/robin/rotation-pp:
Adding a default_path_func that makes the default naming scheme script-level controlled.
Reworking logging's postprocessor logic.
Conflicts:
scripts/base/frameworks/logging/main.bro
testing/btest/policy/frameworks/logging/rotate-custom.bro
The communication subsystem is now disabled until a new BiF,
enable_communication(), is called. The base scripts do this
automatically when either a Communication::Node is defined, or Bro is
asked to listen for incoming connections.
- policy/ renamed to scripts/
- By default BROPATH now contains:
- scripts/
- scripts/policy
- scripts/site
- *Nearly* all tests pass.
- All of scripts/base/ is loaded by main.cc
- Can be disabled by setting $BRO_NO_BASE_SCRIPTS
- Scripts in scripts/base/ don't use relative path loading to ease use of BRO_NO_BASE_SCRIPTS (to copy and paste that script).
- The scripts in scripts/base/protocols/ only (or soon will only) do logging and state building.
- The scripts in scripts/base/frameworks/ add functionality without causing any additional overhead.
- All "detection" activity happens through scripts in scripts/policy/.
- Communications framework modified temporarily to need an environment variable to actually enable (ENABLE_COMMUNICATION=1)
- This is so the communications framework can be loaded as part
of the base without causing trouble when it's not needed.
- This will be removed once a resolution to ticket #540 is reached.
