* 'paraglob' of https://github.com/ZekeMedley/zeek:
Add leak test to paraglob.
Catch paraglob serialization errors in DoClone.
Update paraglob serialization.
Stop execution on paraglob error.
Update paraglob submodule
Change C++11 detection in paraglob.
Make paraglob serializable and copyable.
Initial paraglob integration.
I made a bunch of small changes:
* paraglob now deals better with \0 characters
* I rolled back the changes to Binary Serialization format,
* there were some small formatting issue
* the error output was slightly unsafe
* build_unique is now in util.h.
and perhaps a few more small things.
For backward compatibility when reading values, we first check
the ZEEK-prefixed value, and if not set, then check the corresponding
BRO-prefixed value.
This also installs symlinks from "zeek" and "bro-config" to a wrapper
script that prints a deprecation warning.
The btests pass, but this is still WIP. broctl renaming is still
missing.
#239
When searching for script files, look for both the new and old file
extensions. If a file with ".zeek" can't be found, then search for
a file with ".bro" as a fallback.
This commit marks (hopefully) ever one-parameter constructor as explicit.
It also uses override in (hopefully) all circumstances where a virtual
method is overridden.
There are a very few other minor changes - most of them were necessary
to get everything to compile (like one additional constructor). In one
case I changed an implicit operation to an explicit string conversion -
I think the automatically chosen conversion was much more convoluted.
This took longer than I want to admit but not as long as I feared :)
This change introduces error events for Table and Event readers. Users
can now specify an event that is called when an info, warning, or error
is emitted by their input reader. This can, e.g., be used to raise
notices in case errors occur when reading an important input stream.
Example:
event error_event(desc: Input::TableDescription, msg: string, level: Reporter::Level)
{
...
}
event bro_init()
{
Input::add_table([$source="a", $error_ev=error_event, ...]);
}
For the moment, this converts all errors in the Asciiformatter into
warnings (to show that they are non-fatal) - the Reader itself also has
to throw an Error to show that a fatal error occurred and processing
will be abort.
It might be nicer to change this and require readers to mark fatal
errors as such when throwing them.
Addresses BIT-1181
Addig a new random seed for external tests.
I added a wrapper around the siphash() function to make calling it a
little bit safer at least.
BIT-1612 #merged
* origin/topic/johanna/bit-1612:
HLL: Fix missing typecast in test case.
Remove the -K/-J options for setting keys.
Add test checking the quality of HLL by adding a lot of elements.
Fix serializing probabilistic hashers.
Baseline updates after hash function change.
Also switch BloomFilters from H3 to siphash.
Change Hashing from H3 to Siphash.
HLL: Remove unnecessary comparison.
Hyperloglog: change calculation of Rho
The options were never really used and do not seem especially useful;
initialization with a seed file still works.
This also fixes a bug with the initialization of the siphash key.
This commit mostly changes the hash function that is used for Internal
hashing of data < 36 bytes from H3 to Siphash. This change is motivated
by the fact that it turns out that H3 apparently does not deliver a very
good source of data uniqueness; running HLL with H3 as a hashing
function results in quite poor results (up to of 75% off in my tests).
In difference, running HLL with Siphash (or HMAC-MD5) changes this
factor to ~2%.
This also fixes a long-standing bug in Hash.h which truncated our hash
values to 32 bit on most machines.
Furthermore, it once again fixes a problem with the Rank function in
HLL.
More specifically, this removes the functions:
strcasecmp_n
strchr_n
strrchr_n
and replaces the calls with the respective C-library calls that should
be part of just about all operating systems by now.
Broke out the stats collection into a bunch of new Bifs
in stats.bif. Scripts that use stats collection functions
have also been updated. More work to do.
As of gperftools 2.0 (Feb. 2012), they've been renamed in to
gperftools/ instead of google/, and as of gperftools 2.2, including
the later emits deprecation warnings.
Notable changes:
- libmagic is no longer used at all. All MIME type detection is
done through new Bro signatures, and there's no longer a means to get
verbose file type descriptions (e.g. "PNG image data, 1435 x 170").
The majority of the default file magic signatures are derived
from the default magic database of libmagic ~5.17.
- File magic signatures consist of two new constructs in the
signature rule parsing grammar: "file-magic" gives a regular
expression to match against, and "file-mime" gives the MIME type
string of content that matches the magic and an optional strength
value for the match.
- Modified signature/rule syntax for identifiers: they can no longer
start with a '-', which made for ambiguous syntax when doing negative
strength values in "file-mime". Also brought syntax for Bro script
identifiers in line with reality (they can't start with numbers or
include '-' at all).
- A new Built-In Function, "file_magic", can be used to get all
file magic matches and their corresponding strength against a given
chunk of data
- The second parameter of the "identify_data" Built-In Function
can no longer be used to get verbose file type descriptions, though it
can still be used to get the strongest matching file magic signature.
- The "file_transferred" event's "descr" parameter no longer
contains verbose file type descriptions.
- The BROMAGIC environment variable no longer changes any behavior
in Bro as magic databases are no longer used/installed.
- Reverted back to minimum requirement of CMake 2.6.3 from 2.8.0
(it's back to being the same requirement as the Bro v2.2 release).
The bump was to accomodate building libmagic as an external project,
which is no longer needed.
Addresses BIT-1143.
Includes:
- Cleanup of the plugin API, in particular generally changing
const char* to std::string
- Renaming environment variable BRO_PLUGINS to BRO_PLUGIN_PATH,
defaulting to <prefix>/lib/bro/plugins
- Reworking how dynamic plugins are searched and activated. See
doc/devel/plugins.rst for details.
- New @load-plugin directive to explicitly activate a plugin
- Support for Darwin. (Linux untested right now)
- The init-plugin updates come with support for "make test", "make
sdist", and "make bdist" (see how-to).
- Test updates.
Notes: The new hook mechanism, which allows plugins to hook into Bro's
core a well-defined points, is still essentially untested.
- Move more functionality into base class.
- Remove cctors and assignment operators (weren't actually needed anymore)
- Switch from const char* to std::string.
BIT-1098
* origin/topic/jsiwek/broxygen:
Fix Broxygen-related compile errors.
Add a Broxygen coverage test.
Internal Broxygen organization/documentation/polish.
Add unit tests for Broxygen config file targets.
Change Broxygen config file format.
Broxygen doc-related test updates. Fix two regressions.
A couple documentation fixes.
Integrate new Broxygen functionality into Sphinx.
Implement majority of Broxygen features delegated to Bro.
Broxygen can now read a config file specifying particular targets.
Remove unneeded Broxygen comments in scan.bro.
Replace safe_basename/safe_dirname w/ SafeBasename/SafeDirname.
Add BIF interface for retrieving comments/docs.
Quick optimization to Broxygen doc gathering.
Flesh out Broxygen doc-gathering skeleton.
Refactor search_for_file() util function.
Initial skeleton of new Broxygen infrastructure.
I got rid of the earlier separate InterpreterPlugin class. Instead
Plugin now has a set of virtual methods HookSomething()... that
plugins can override. For efficiency purposes, they however need to
register first that they are interested in a hook, otherwise the
virtual method will never be called. The idea is to extend the set of
hooks over time as we figure out what's useful.
This is a checkpoint commit that's essentially untested and probably
broken. It compiles, though.
This is essentially the code from the dynamic-plugin branch except for
some pieces that I have split out into separate, earlier commits.
I'm going to updatre things in this branch going forward.
It was getting too bloated and allocated memory in ways that were
difficult to understand how to manage. Separated out primarily in to
new find_file() and open_file()/open_package() functions.
Also renamed other util functions for path-related things.
Doesn't generate any docs, but it's hooked in to all places needed to
gather the necessary stuff w/ significantly less coupling than before.
The gathering now always occurs unconditionally to make documentation
available at runtime and a command line switch (-X) only toggles whether
to output docs to disk (reST format).
Should also improve the treatment of type name aliasing which wasn't a
big problem in practice before, but I think it's more correct now:
there's now a distinct BroType for each alias, but extensible types
(record/enum) will automatically update the types for aliases on redef.
Other misc refactoring of note:
- Removed a redundant/unused way of declaring event types.
- Changed type serialization format/process to preserve type name
information and remove compatibility code (since broccoli will
have be updated anyway).
