- While updating, I did some further work on the branch.
- New function in the base/utils/files for extracting filenames
from content-dispositions.
- New script for entity excerpt extraction if you aren't interested
in full extraction. The data goes a log field too.
- Some renaming and reorganization of types.
- Updated tests to work with new code.
* origin/topic/jsiwek/smtp-refactor:
Make the doc.coverage test happy.
SMTP script refactor. (addresses #509)
Conflicts:
doc/scripts/DocSourcesList.cmake
policy/protocols/smtp/__load__.bro
policy/protocols/smtp/base/__load__.bro
and runs; need to go back through and make sure this code is actually
doing what I want it to do.
Note: Added new function unique_id_from(pool: string, prefix: string)
that allows the user to explicitly specify a randomness pool to use when
generating unique IDs.
* test-all.bro renamed to test-all-policy.bro because it lists
only the optional scripts now.
* A new test that checks that the default config loads everything
in base/*/
* A new test that runs bare mode but loads all optional policy
scripts (which fails horribly right now ...)
* A new loaded_scripts test for the bare mode.
- Fixing the parts of the `make restdoc` and `make doc` process that were
broken by the last Bro script re-organization
- Generated documentation for Bro scripts derived from BiFs now use the
original BiF source file as the "original source file" link
- Renaming of the internal POLICYDEST definition and other misc places that
refer to "policy" scripts; that terminology doesn't make total sense now
- Added a documentation blacklist reminder test that will fail if there's
scripts that are blacklisted from being documentated because they're still
in progress
- Some minor Bro script changes to fix small @load dependency errors
Addresses #543
* topic/robin/rotation-pp:
Adding a default_path_func that makes the default naming scheme script-level controlled.
Reworking logging's postprocessor logic.
Conflicts:
scripts/base/frameworks/logging/main.bro
testing/btest/policy/frameworks/logging/rotate-custom.bro
- bro.init was renamed to base/init-bare.bro and base/all.bro
was renamed to init-default.bro.
- To run in "bare mode" with only the init-bare.bro and no other
scripts from base/, use either -b or --bare-mode.
- The environment variable to run in "bare mode" has been removed.
- policy/ renamed to scripts/
- By default BROPATH now contains:
- scripts/
- scripts/policy
- scripts/site
- *Nearly* all tests pass.
- All of scripts/base/ is loaded by main.cc
- Can be disabled by setting $BRO_NO_BASE_SCRIPTS
- Scripts in scripts/base/ don't use relative path loading to ease use of BRO_NO_BASE_SCRIPTS (to copy and paste that script).
- The scripts in scripts/base/protocols/ only (or soon will only) do logging and state building.
- The scripts in scripts/base/frameworks/ add functionality without causing any additional overhead.
- All "detection" activity happens through scripts in scripts/policy/.
- Communications framework modified temporarily to need an environment variable to actually enable (ENABLE_COMMUNICATION=1)
- This is so the communications framework can be loaded as part
of the base without causing trouble when it's not needed.
- This will be removed once a resolution to ticket #540 is reached.
When using a `print` statement to write to a file that has raw output
enabled, NUL characters in string are no longer interpreted into "\0",
no newline is appended afterwards, and each argument to `print` is
written to the file without any additional separation.
(Re)Assigning to identifiers with the &raw_output attribute should also
now correctly apply the attribute to the file value being assigned.
Note that the write_file BiF should already be capable of raw string
data to a file, expect it bypasses the print_hook event.
Addresses #474
When reading from trace files, 'dropped' and 'link' fields are now
just zeroed.
When reading from an interface, the values filled in by pcap_stats()
are now only used when that function indicates success.
Closes#500.
* origin/topic/jsiwek/unit-tests:
Fix utils/conn-ids test due to renamed conn-ids.bro
Moving the test for site.bro to live w/ other utils/ tests.
Fix test due to moving of site.bro
More policy/utils unit tests and documentation.
Updating documentation for some utils/ policy scripts
Add unit tests for utils/paths.bro with some changes
Adding unit tests for utils.
Adding test for utils/addrs.bro.
Add unit test for site.bro.
Conflicts:
policy/utils/site.bro
Closes#525.
The main change is that the postprocessor commands are no longer run
by the log writers themselves. Instead, the writers send back a
message to the log mgr once they have rotated. The manager then calls
a script level function to do somethign with the rotated file. By
default, it will be renamed to somethingn nice and then a
postprocessor shell command will be run on it if defined.
Pieces going into this:
- Terminology change: "postprocessor" now refers to a script
*function*. In addition, there are "postprocessor commands", which
are shell commands that may be triggered by the function to run on
a rotated file.
- The RotationInfo record now comes with all the information that
was previously provided internally to the C++ function running the
post-processor command.
- Changing the default time format to %Y-%m-%d-%H-%M-%S
- rotation_path_func is gone
- The default postprocessor function is defined individually by
each LogWriter in frameworks/logging/plugin/*
- The interface to postprocessor shell commands remains the same.
Needs a bit more testing ...
- message header state tracking is now done by handling mime_one_header
instead of parsing the data in the smtp_data event
- changed the logging point to be when an smtp_reply is seen in response
to the end of a DATA section
- the smtp package now uses it's own mime script and logging stream for
logging entities, extraction, etc.
- fixes for mime file extraction: now logs the extracted file name, and
the count of extracted files needed to be maintained in the State record
sed on some platforms like OS X (maybe FreeBSD in general) won't recognize
semi-colon delimited commands as multiple commands, instead use the -e
option multiple times to build the command list.
- The CMake targets for generating reST docs from policy scripts are now
automatically generated via the genDocSourcesList.sh script
- Fixed a lot of parsing errors in policy scripts that I saw along the way
If a test doesn't rely on libmagic, mime type related columns of baselined
logs are filtered out.
If a test does rely on libmagic, it needs to use the TEST-REQUIRES btest
macro to check that the bro build supports it, and then mime type related
columns of logs can be normalized via a logging filter to reduce sensitivity
to varying version of libmagic.
If a test doesn't rely on libmagic, mime type related columns of baselined
logs are filtered out.
If a test does rely on libmagic, it needs to use the TEST-REQUIRES btest
macro to check that the bro build supports it, and then mime type related
columns of logs can be normalized via a logging filter to reduce sensitivity
to varying version of libmagic.
* origin/topic/script-load-changes:
Fix reST file name associated w/ stdin when in doc mode (closes#497)
Update @prefixes test.
Rewrite a test using btest's TEST-START-FILE directive
Fix @unload'd files from generating bro_script_loaded event.
Renaming a test better.
Reimplementation of the @prefixes statement.
Fix accidental overwrite of BROPATH copy.
Make @load statements recognize relative paths.
* origin/topic/jsiwek/irc-orig:
Shorten what's displayed in the IRC's log mime_type column for DCC transfers
Add IRC unit tests.
Small tweak to IRC event handlder priorities
Fix IRC analyzer supplying wrong type to irc_dcc_message event.
Changes to IRC analyzer and events (addresses #469).
