* origin/topic/jsiwek/exec-module:
Exec module changes/fixes.
Coverage test fixes and whitespace/doc tweaks.
Update to make Dir::monitor watch inodes instead of file names.
Updates to use new input framework mechanism to execute command line programs.
Added Exec, Dir, and ActiveHTTP modules.
BIT-1046 #merged.
Conflicts:
magic
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
* origin/topic/seth/faf-updates: (27 commits)
Undoing the FTP tests I updated earlier.
Update the last two btest FAF tests.
File analysis fixes and test updates.
Fix a bug with getting analyzer tags.
A few test updates.
Some tests work now (at least they all don't fail anymore!)
Forgot a file.
Added protocol description functions that provide a super compressed log representation.
Fix a bug where orig file information in http wasn't working right.
Added mime types to http.log
Clean up queued but unused file_over_new_connections event args.
Add jar files to the default MHR lookups.
Adding CAB files for MHR checking.
Improve malware hash registry script.
Fix a small issue with finding smtp entities.
Added support for files to the notice framework.
Make the custom libmagic database a git submodule.
Add an is_orig parameter to file_over_new_connection event.
Make magic for emitting application/msword mime type less strict.
Disable more libmagic builtin checks that override the magic database.
...
Conflicts:
doc/scripts/DocSourcesList.cmake
scripts/base/init-bare.bro
scripts/test-all-policy.bro
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
- Fix examples/references in the file analysis how-to/usage doc.
- Add Broxygen-generated docs for file analyzer plugins.
- Break FTP::Info type declaration out in to its own file to get
rid of some circular dependencies (between s/b/p/ftp/main and
s/b/p/ftp/utils).
- Several places were just using old variable names or not loading
scripts correctly after they'd been renamed/moved.
- Revert/adjust a change in how HTTP file handles are generated that
broke partial content responses.
- Turn some libmagic builtin checks back on; seems some are actually
useful (e.g. text detection seems to be a builtin). The rule going
forward probably will be only to turn off a builtin if we confirm it
causes issues.
- Removed some tests that are redundant or not necessary anymore because
the generic file analysis tests cover them.
- A couple FTP tests still fail that I think need an actual solution via
script changes.
in.
No more manual includes to pull them in.
(It doesn't quite work fully automatically yet for some bifs that need
script-level types defined, like the input and logging frameworks.
They still do a manual "@load foo.bif" in their main.bro to get the
order right. It's a bit tricky to fix that and would probably need
splitting main.bro into two parts; not sure that's worth it.)
- Give Dir::monitor() a param for the polling interval, so different
dirs can be monitored at different frequencies.
- Fix race in Exec::run() when reading extra output files produced by
a process -- it was possible for Exec::run() to return before all
extra output files had been fully read.
- Add test cases.
- Intel importing format has changed (refer to docs).
- All string matching is now case insensitive.
- SMTP intel script has been updated to extract email
addresses correctly.
- Small fix sneaking into the smtp base script to actually
extract individual email addresses in the To: field
correctly.
Closes#1021.
* origin/topic/bernhard/input-update:
this event handler fails the unused-event-handlers test because it is a bit of a special case.
...and fix the event ordering issue. Dispatch != QueueEvent
add Terminate to input framework to prevent potential shutdown race-conditions.
fix warning.
fix stderr test. ls behaves differently on errors on linux...
small fixes.
linux does not have strnstr
and close only fds that are currently open (the logging framework really did not like that :) )
A bunch of more changes for the raw reader
make reading from stdout and stderr simultaneously work.
allow sending data to stdin of child process
Streaming reads from external commands work without blocking anything.
replace popen with fork and exec.
change raw reader to use basic c io instead of fdstream encapsulation class.
- Thanks for help from Rafal Lesniak in nailing down the location
of the bug and supplying test traffic.
- Test traffic with a TLS 1.2 connection.
- Addresses ticket #1020
it is a bit of a special case.
It is only called via the SendEvent function from a reader. The reader
does (at least with the current interface) however not provide
the function pointer, but looks up the name of the event dynamically.
Hence, internal_handler is never called for the event.
Even if resolving the event in the reader, e.g. in an initialization
function, this would not solve the issue - the initialization function
is only called when the first Raw reader is initialized - and in the
base configuration the raw reader will never be used (hence, internal_handler
also won't be called).
Calling it once in the manager seems like a really dirty hack. So - now
it is the second exception in the testcase, unless anyone has a better
idea :)
Update baseline of stderr test to what it should be. There still is
a message ordering issue there (which is the last issue in the new
Raw reader I know of).
One message that sidesteps a bit of the usual processing does
not always arrive at the correct time (meaning it pops up from the
event queue too early). Even though it sidesteps a bit of the usual
processing that should not happen in my opinion (which clearly
does not matter). And I have not yet fully grasped how this can happen.
