When not reporting via events, the final contents of the message buffer
after formatting was being used as a format string to fprintf instead of
writing out the actual string.
The function's code is rendered as ASCII and included as a string.
Closes#506.
Note that I'm not sure if the formatting is as desired: should the LFs
and tabs be rendered as \xXX or removed?.
The format string given to the reporter warning call wasn't printing
the handler names. Also changed it so that each warning message has
the full context of the warning.
- While updating, I did some further work on the branch.
- New function in the base/utils/files for extracting filenames
from content-dispositions.
- New script for entity excerpt extraction if you aren't interested
in full extraction. The data goes a log field too.
- Some renaming and reorganization of types.
- Updated tests to work with new code.
* origin/topic/jsiwek/smtp-refactor:
Make the doc.coverage test happy.
SMTP script refactor. (addresses #509)
Conflicts:
doc/scripts/DocSourcesList.cmake
policy/protocols/smtp/__load__.bro
policy/protocols/smtp/base/__load__.bro
and runs; need to go back through and make sure this code is actually
doing what I want it to do.
Note: Added new function unique_id_from(pool: string, prefix: string)
that allows the user to explicitly specify a randomness pool to use when
generating unique IDs.
* test-all.bro renamed to test-all-policy.bro because it lists
only the optional scripts now.
* A new test that checks that the default config loads everything
in base/*/
* A new test that runs bare mode but loads all optional policy
scripts (which fails horribly right now ...)
* A new loaded_scripts test for the bare mode.
- Fixing the parts of the `make restdoc` and `make doc` process that were
broken by the last Bro script re-organization
- Generated documentation for Bro scripts derived from BiFs now use the
original BiF source file as the "original source file" link
- Renaming of the internal POLICYDEST definition and other misc places that
refer to "policy" scripts; that terminology doesn't make total sense now
- Added a documentation blacklist reminder test that will fail if there's
scripts that are blacklisted from being documentated because they're still
in progress
- Some minor Bro script changes to fix small @load dependency errors
Addresses #543
* topic/robin/rotation-pp:
Adding a default_path_func that makes the default naming scheme script-level controlled.
Reworking logging's postprocessor logic.
Conflicts:
scripts/base/frameworks/logging/main.bro
testing/btest/policy/frameworks/logging/rotate-custom.bro
- bro.init was renamed to base/init-bare.bro and base/all.bro
was renamed to init-default.bro.
- To run in "bare mode" with only the init-bare.bro and no other
scripts from base/, use either -b or --bare-mode.
- The environment variable to run in "bare mode" has been removed.
- policy/ renamed to scripts/
- By default BROPATH now contains:
- scripts/
- scripts/policy
- scripts/site
- *Nearly* all tests pass.
- All of scripts/base/ is loaded by main.cc
- Can be disabled by setting $BRO_NO_BASE_SCRIPTS
- Scripts in scripts/base/ don't use relative path loading to ease use of BRO_NO_BASE_SCRIPTS (to copy and paste that script).
- The scripts in scripts/base/protocols/ only (or soon will only) do logging and state building.
- The scripts in scripts/base/frameworks/ add functionality without causing any additional overhead.
- All "detection" activity happens through scripts in scripts/policy/.
- Communications framework modified temporarily to need an environment variable to actually enable (ENABLE_COMMUNICATION=1)
- This is so the communications framework can be loaded as part
of the base without causing trouble when it's not needed.
- This will be removed once a resolution to ticket #540 is reached.
When using a `print` statement to write to a file that has raw output
enabled, NUL characters in string are no longer interpreted into "\0",
no newline is appended afterwards, and each argument to `print` is
written to the file without any additional separation.
(Re)Assigning to identifiers with the &raw_output attribute should also
now correctly apply the attribute to the file value being assigned.
Note that the write_file BiF should already be capable of raw string
data to a file, expect it bypasses the print_hook event.
Addresses #474
When reading from trace files, 'dropped' and 'link' fields are now
just zeroed.
When reading from an interface, the values filled in by pcap_stats()
are now only used when that function indicates success.
Closes#500.
* origin/topic/jsiwek/unit-tests:
Fix utils/conn-ids test due to renamed conn-ids.bro
Moving the test for site.bro to live w/ other utils/ tests.
Fix test due to moving of site.bro
More policy/utils unit tests and documentation.
Updating documentation for some utils/ policy scripts
Add unit tests for utils/paths.bro with some changes
Adding unit tests for utils.
Adding test for utils/addrs.bro.
Add unit test for site.bro.
Conflicts:
policy/utils/site.bro
Closes#525.
The main change is that the postprocessor commands are no longer run
by the log writers themselves. Instead, the writers send back a
message to the log mgr once they have rotated. The manager then calls
a script level function to do somethign with the rotated file. By
default, it will be renamed to somethingn nice and then a
postprocessor shell command will be run on it if defined.
Pieces going into this:
- Terminology change: "postprocessor" now refers to a script
*function*. In addition, there are "postprocessor commands", which
are shell commands that may be triggered by the function to run on
a rotated file.
- The RotationInfo record now comes with all the information that
was previously provided internally to the C++ function running the
post-processor command.
- Changing the default time format to %Y-%m-%d-%H-%M-%S
- rotation_path_func is gone
- The default postprocessor function is defined individually by
each LogWriter in frameworks/logging/plugin/*
- The interface to postprocessor shell commands remains the same.
Needs a bit more testing ...
- message header state tracking is now done by handling mime_one_header
instead of parsing the data in the smtp_data event
- changed the logging point to be when an smtp_reply is seen in response
to the end of a DATA section
- the smtp package now uses it's own mime script and logging stream for
logging entities, extraction, etc.
- fixes for mime file extraction: now logs the extracted file name, and
the count of extracted files needed to be maintained in the State record
sed on some platforms like OS X (maybe FreeBSD in general) won't recognize
semi-colon delimited commands as multiple commands, instead use the -e
option multiple times to build the command list.
- The CMake targets for generating reST docs from policy scripts are now
automatically generated via the genDocSourcesList.sh script
- Fixed a lot of parsing errors in policy scripts that I saw along the way
If a test doesn't rely on libmagic, mime type related columns of baselined
logs are filtered out.
If a test does rely on libmagic, it needs to use the TEST-REQUIRES btest
macro to check that the bro build supports it, and then mime type related
columns of logs can be normalized via a logging filter to reduce sensitivity
to varying version of libmagic.
If a test doesn't rely on libmagic, mime type related columns of baselined
logs are filtered out.
If a test does rely on libmagic, it needs to use the TEST-REQUIRES btest
macro to check that the bro build supports it, and then mime type related
columns of logs can be normalized via a logging filter to reduce sensitivity
to varying version of libmagic.
* origin/topic/script-load-changes:
Fix reST file name associated w/ stdin when in doc mode (closes#497)
Update @prefixes test.
Rewrite a test using btest's TEST-START-FILE directive
Fix @unload'd files from generating bro_script_loaded event.
Renaming a test better.
Reimplementation of the @prefixes statement.
Fix accidental overwrite of BROPATH copy.
Make @load statements recognize relative paths.
