Scripting errors/mistakes now consistently generate a runtime error
which have the behavior of unwinding the call stack all the way out of
the current event handler.
Before, such errors were not treated consistently and either aborted
the process entirely or emitted a message while continuing to execute
subsequent statements without well-defined behavior (possibly causing
a cascade of errors).
The previous behavior also would only unwind out of the current
function (if within a function body), not out the current event
handler, which is especially problematic for functions that return
a value: the caller is essentially left a mess with no way to deal
with it.
This also changes the behavior of the startup/initialization process
to abort if there's errors during bro_init() rather than continue one
to the main run loop. The `allow_init_errors` option may change this
new, default behavior.
These are probably some of the most desired options to be dynamically
changeable; since they only are accessed in script-land there should not
be any problems with them changing on the fly.
* origin/topic/seth/fix-raw-reader-subprocess-exit:
Fix an issue with raw reader culling streams for dead processes.
Updated the 'exec' utility to no longer remove input streams for
processes that are finished as the core C++ code will take care of that
(and trying to remove a stream multiple times emits a warning message).
* origin/topic/vern/vec-append:
d'oh, still have a (deprecated) string_array rather than string_vector
forgot to update test suite results for v += e
reap the fruits of v += e
test case for v += e
documentation of v += e
v += e implemented
Fixed a mistake in find_ip_addresses()
BIT-1594 #merged
* origin/topic/johanna/rawleak:
Exec: fix reader cleanup when using read_files
Raw Writer: First step - make code more c++11-y, remove raw pointers.
- SMTP protocol headers now do some minimal parsing to clean up
email addresses.
- New function named split_mime_email_addresses to take MIME headers
and get addresses split apart but including the display name.
- Update tests.
Wen using read_files, the Exec framework called Input::remove on the
wrong input stream: it always got called on the input stream of the
execution, not on the input stream of the current file that was being
read.
This lead to threads never being closed and file handles being kept open
until Bro is closed. This means that before this patch, every time
ActiveHTTP is used, a thread stays around and several file handles are
used.
We now extract email addresses in the fields that one would expect
to contain addresses. This makes further downstream processing of
these fields easier like log analysis or using these fields in the
Intel framework. The primary downside is that any other content
in these fields is no longer available such as full name and any
group information. I believe the simplification of the content in
these fields is worth the change.
Added "cc" to the script that feeds information from SMTP into the
Intel framework.
A new script for email handling utility functions has been created
as a side effect of these changes.
Added a new BIF haversine_distance that computes distance between two
geographic locations.
Added a new Bro script function haversine_distance_ip that does the same
but takes two IP addresses instead of latitude/longitude. This function
requires that Bro be built with libgeoip.
BIT-1550 #merged
* origin/topic/johanna/netcontrol: (72 commits)
Update baselines and news
Move prefixtable back to all IPv6 internal handling.
NetControl: Add functions to search for rules affecting IPs/subnets
Add check_subnet bif that allows exact membership test for subnet tables.
Rewrite internal handling of rules.
Add bif that allows searching for all matching subnets in table.
Add signaling of succesful initialization of plugins to NetControl.
Add rule hooks to the acld plugin.
Add new logfiles for shunting and drops to netcontrol
Extend NetControl logging and fix bugs.
Update OpenFlow API and events.
small acld plugin fix
Revert "introduce &weaken attribute"
Fix crash when printing type of recursive structures.
Testcase for crash when a record contains a function referencing a record.
Rename Pacf to NetControl
fix acld plugin to use address instead of subnet (and add functions for conversion)
implement quarantine
miscelaneous missing bits and pieces
Acld implementation for Pacf - Bro side.
...
The server-reported file size was being collected poorly and if
a file name had a number in it, that was reported as the file
size instead of the actual size.
A new test is included to avoid reintroducing the problem.
The API now does not follow the openflow specification quite as closely,
however I think it is much more usable. Furthermore, the Ryu plugin was
basically completely rewritten and is now more usable for general flow
manipulation.
This also adds a debug mode that just outputs the json fragments that
would be sent to ryu. At the moment, Ryu still assumes that every
request that it receives succeeds - it is not possible to get an error
message from the controller. Instead, one has to check if a flow was
added by doing a second REST request. Which seems unnecessary, and also
requires complete json parsing functionality. Hence we are not doing
that at the moment.
The alternative would be to use an external script for the actual
add-and-check-operation.
These functions are now deprecated in favor of alternative versions that
return a vector of strings rather than a table of strings.
Deprecated functions:
- split: use split_string instead.
- split1: use split_string1 instead.
- split_all: use split_string_all instead.
- split_n: use split_string_n instead.
- cat_string_array: see join_string_vec instead.
- cat_string_array_n: see join_string_vec instead.
- join_string_array: see join_string_vec instead.
- sort_string_array: use sort instead instead.
- find_ip_addresses: use extract_ip_addresses instead.
Changed functions:
- has_valid_octets: uses a string_vec parameter instead of string_array.
Addresses BIT-924, BIT-757.
- I've changed/extended the URI record fields a bit:
- path is always the full path including the full file name
- if there's no path, the field still still be set set "/".
- file_name is the full name including extenstion, and
file_base and file_ext split it out.
- Adding a test exercising a bunch of URLs.
* origin/topic/struck/BIT-1277:
[ADD] Added the feature to return 0 content to the python http test server and added functionality for post requests
[ADD] added baseline for the new active-http test and added a test to check for the content-length 0 fix.
[ADD] added baseline for the new exec test and added a test to check for the empty files fix.
[FIX] exec should write an empty string when file is empty instead of the filename
[FIX] Add files to result table even if the files are empty
BIT-1277 #merged
Some documentation of commented-out items was being output to HTML,
which was causing that documentation to be merged with the
documentation of the next item. Fixed by changing "##" to "#" so
that the doc comments are not included in the HTML.
This cleans up most of the warnings from sphinx (broken :doc: links,
broxygen role misuses, etc.). The remaining ones should be harmless,
but not quick to silence.
I found that the README for each component was a copy from the actual
repo, so I turned those in to symlinks so they don't get out of date.
- Do stream mode for commands done by exec module, it seems important
in some cases (e.g. ensure requested stdin is fully written).
- For cases where the raw input reader knows the child process has been
reaped, set the childpid member to a sentinel value to indicate such
so we don't later think we should kill it or wait on it anymore.
- More error checking on dup2/close calls. Set sentinel values when
closing ends of pipes to prevent double closing a fd.
- Signal flag not set when raw input reader's child exits as a result
of a signal. Left out a test for this -- might be portability issues
(e.g. Ubuntu seems to do things different regarding the exit code and
also is printing "Killed" to stderr where other platforms don't).
