- Metrics now work on cluster deployments with no caveats. It should be
completely transparent. Intermediate updates to speed some detection
will come later.
The function's code is rendered as ASCII and included as a string.
Closes#506.
Note that I'm not sure if the formatting is as desired: should the LFs
and tabs be rendered as \xXX or removed?.
- While updating, I did some further work on the branch.
- New function in the base/utils/files for extracting filenames
from content-dispositions.
- New script for entity excerpt extraction if you aren't interested
in full extraction. The data goes a log field too.
- Some renaming and reorganization of types.
- Updated tests to work with new code.
* origin/topic/jsiwek/smtp-refactor:
Make the doc.coverage test happy.
SMTP script refactor. (addresses #509)
Conflicts:
doc/scripts/DocSourcesList.cmake
policy/protocols/smtp/__load__.bro
policy/protocols/smtp/base/__load__.bro
* test-all.bro renamed to test-all-policy.bro because it lists
only the optional scripts now.
* A new test that checks that the default config loads everything
in base/*/
* A new test that runs bare mode but loads all optional policy
scripts (which fails horribly right now ...)
* A new loaded_scripts test for the bare mode.
* topic/robin/rotation-pp:
Adding a default_path_func that makes the default naming scheme script-level controlled.
Reworking logging's postprocessor logic.
Conflicts:
scripts/base/frameworks/logging/main.bro
testing/btest/policy/frameworks/logging/rotate-custom.bro
- bro.init was renamed to base/init-bare.bro and base/all.bro
was renamed to init-default.bro.
- To run in "bare mode" with only the init-bare.bro and no other
scripts from base/, use either -b or --bare-mode.
- The environment variable to run in "bare mode" has been removed.
- policy/ renamed to scripts/
- By default BROPATH now contains:
- scripts/
- scripts/policy
- scripts/site
- *Nearly* all tests pass.
- All of scripts/base/ is loaded by main.cc
- Can be disabled by setting $BRO_NO_BASE_SCRIPTS
- Scripts in scripts/base/ don't use relative path loading to ease use of BRO_NO_BASE_SCRIPTS (to copy and paste that script).
- The scripts in scripts/base/protocols/ only (or soon will only) do logging and state building.
- The scripts in scripts/base/frameworks/ add functionality without causing any additional overhead.
- All "detection" activity happens through scripts in scripts/policy/.
- Communications framework modified temporarily to need an environment variable to actually enable (ENABLE_COMMUNICATION=1)
- This is so the communications framework can be loaded as part
of the base without causing trouble when it's not needed.
- This will be removed once a resolution to ticket #540 is reached.
* origin/topic/jsiwek/unit-tests:
Fix utils/conn-ids test due to renamed conn-ids.bro
Moving the test for site.bro to live w/ other utils/ tests.
Fix test due to moving of site.bro
More policy/utils unit tests and documentation.
Updating documentation for some utils/ policy scripts
Add unit tests for utils/paths.bro with some changes
Adding unit tests for utils.
Adding test for utils/addrs.bro.
Add unit test for site.bro.
Conflicts:
policy/utils/site.bro
Closes#525.
The main change is that the postprocessor commands are no longer run
by the log writers themselves. Instead, the writers send back a
message to the log mgr once they have rotated. The manager then calls
a script level function to do somethign with the rotated file. By
default, it will be renamed to somethingn nice and then a
postprocessor shell command will be run on it if defined.
Pieces going into this:
- Terminology change: "postprocessor" now refers to a script
*function*. In addition, there are "postprocessor commands", which
are shell commands that may be triggered by the function to run on
a rotated file.
- The RotationInfo record now comes with all the information that
was previously provided internally to the C++ function running the
post-processor command.
- Changing the default time format to %Y-%m-%d-%H-%M-%S
- rotation_path_func is gone
- The default postprocessor function is defined individually by
each LogWriter in frameworks/logging/plugin/*
- The interface to postprocessor shell commands remains the same.
Needs a bit more testing ...
- message header state tracking is now done by handling mime_one_header
instead of parsing the data in the smtp_data event
- changed the logging point to be when an smtp_reply is seen in response
to the end of a DATA section
- the smtp package now uses it's own mime script and logging stream for
logging entities, extraction, etc.
- fixes for mime file extraction: now logs the extracted file name, and
the count of extracted files needed to be maintained in the State record
sed on some platforms like OS X (maybe FreeBSD in general) won't recognize
semi-colon delimited commands as multiple commands, instead use the -e
option multiple times to build the command list.
If a test doesn't rely on libmagic, mime type related columns of baselined
logs are filtered out.
If a test does rely on libmagic, it needs to use the TEST-REQUIRES btest
macro to check that the bro build supports it, and then mime type related
columns of logs can be normalized via a logging filter to reduce sensitivity
to varying version of libmagic.
- Fixing more vestiges from moving site.bro and removing functions.bro
- Updates comments on analysis-groups.bro
- Added the trim-trace-file script from broctl.
- rename extract_directory() to extract_path() (later seemed clearer)
and made it work with more than just path string in FTP response msgs
- rename build_full_path() and absolute_path()
- compress_path() should now work with relative paths also
- Control framework is for runtime control of Bro instances.
It was extracted from BroControl and made more generic.
- Tests for cluster frameworks and control framework.
- Small fix for btest.cfg
- Fixed a bug in the cluster framework that was causing things to break.
Also removing the -l command-line option as that can now be done at
the script-level.
A couple tests fail now that use -l. Leaving that until we have
script-level replacement.
This is for testing only. There are also two test: one that checks
that test-all.bro loads correctly, and one that ensures that test-all
is actually loading all scripts found in policy/*.
- core.load-unload: scripts that get loaded by default changed, so to
make the test insensitive to that in the future, I changed the test
to just check that the stdout is empty (the @unload'd script would have
had output there)
- policy.frameworks.logging.rotate-custom: I saw that the ordering of
the log postprocessor output caused a failure for me even though the
overall content was the same, so it now sorts that part before diff'ing
- core.print-bpf-filters-ipv[4|6]: packet-filter log file name changed
- policy.protocols.conn.known-services: logging file names changes and
local_nets is now in the Site module
