It turns out that --with-openssl does not currently work because even
when the correct library is chosen, the include path ordering will
(typically) cause the system OpenSSL to be included.
Push OpenSSL header directory as the highest priority one. This probably
also makes sense since OpenSSL is/will be one of the most common
libraries that needs specific include directories.
This just provides a convient way of indicating that ccache should
be used as compiler-wrapper during builds. e.g. when I want dev/debug
builds that (re)compile quickly, I do:
./configure --build-type=debug --generator=Ninja --ccache
time.
People keep running into the problem that they upgrade Bro but forget
to recompile their plugins--which can lead to crashes. While the
plugins' API version was supposed to catch this, it's not reliable as
that check may come too late. This change takes a different tack: We
compile a C function into the Bro binary that has Bro's version number
encoded into its name. A plugin can then reference that function. If
the Bro version changes, the function goes away and the plugin won't
load anymore.
I've integrated that function reference into the plugin skeleton code
so that new plugins get it automatically (unless explicitly removed).
I couldn't see a way to do it transparently for already existing
plugins unfortunately.
The version number used for the function name is slightly normalized
to skip any git revision postfixes (i.e., "2.5-xxx" is always treated
as "2.5-git") so that one doesn't need to recompile all plugins after
every master commit. That seems good enough, usually people run into
this when upgrading to a new release.
If one loads an old plugin into a new Bro, the error message looks
like this:
$ bro -NN Demo::Foo
fatal error in /home/robin/bro/master/scripts/base/init-bare.bro, line 1:
cannot load plugin library /home/robin/tmp/p/build//lib/Demo-Foo.linux-x86_64.so:
/home/robin/tmp/p/build//lib/Demo-Foo.linux-x86_64.so: undefined symbol: bro_version_2_5_git_debug
Not the prettiest, but better than a crash!
TODO: I'm still unsure if we should remove the plugin API version
altogetger now. This link-time check should catch everything the API
version does, except for master commits.
I reworked this a bit:
- Moved the globals into a new Pcap::* namespace, and renamed them
slightly.
- Moved the definitions of the globals into pcap/const.bif.
- Also moved the existing 'snaplen' into Pcap::* and removed
SnapLen() from the PktSrc API (it's really a pcap thing).
- Likewise moved the existing functions precompile_pcap_filter,
install_pcap_filter, and pcap_error, into Pcap::*.
- Did some more refactoring for the pcap code.
* 'master' of https://github.com/knielander/bro:
Refactored patch (removed options, less ambiguous name)
Allow Bro to run in fanout mode.
Allow libpcap buffer size to be set manually.
Allow Bro to run in fanout mode.
Allowed libpcap buffer size to be set via configuration.
The new --enable-broker flag can be used to toggle the use of Broker,
which also implies building with -std=c++11, though nothing makes
use of these features at the moment.
* origin/topic/robin/reader-writer-plugins:
Adding plugin testing to Makefile's test-all.
Fixing tests.
Removing DataSeries and ElasticSearch from configure script.
Updating submodule.
Move DataSeries and ElasticSearch into plugins.
More polishing of some of the branche's changes.
More polishing.
Test case for a dynamic input reader.
Test case for a dynamic log writer.
Final fixes, preparing for merge.
Converting log writers and input readers to plugins.
BIT-1222 #merged
Notable changes:
- libmagic is no longer used at all. All MIME type detection is
done through new Bro signatures, and there's no longer a means to get
verbose file type descriptions (e.g. "PNG image data, 1435 x 170").
The majority of the default file magic signatures are derived
from the default magic database of libmagic ~5.17.
- File magic signatures consist of two new constructs in the
signature rule parsing grammar: "file-magic" gives a regular
expression to match against, and "file-mime" gives the MIME type
string of content that matches the magic and an optional strength
value for the match.
- Modified signature/rule syntax for identifiers: they can no longer
start with a '-', which made for ambiguous syntax when doing negative
strength values in "file-mime". Also brought syntax for Bro script
identifiers in line with reality (they can't start with numbers or
include '-' at all).
- A new Built-In Function, "file_magic", can be used to get all
file magic matches and their corresponding strength against a given
chunk of data
- The second parameter of the "identify_data" Built-In Function
can no longer be used to get verbose file type descriptions, though it
can still be used to get the strongest matching file magic signature.
- The "file_transferred" event's "descr" parameter no longer
contains verbose file type descriptions.
- The BROMAGIC environment variable no longer changes any behavior
in Bro as magic databases are no longer used/installed.
- Reverted back to minimum requirement of CMake 2.6.3 from 2.8.0
(it's back to being the same requirement as the Bro v2.2 release).
The bump was to accomodate building libmagic as an external project,
which is no longer needed.
Addresses BIT-1143.
The Plugin.cc file is now just a standard class, with the interface
changed a bit to make it more easy to write. However, there're still
some conventions that one must follow to make everything work (like
using the right namespace structure).
This commit also includes the option to compile built-in plugins
dynamically instead of statically by adding
SET(BRO_PLUGIN_BUILD_DYNAMIC TRUE) to their CMake config. This hasn't
been tested much yet, and I'm still undecided if it's somethign we
would want to do by default--but we could now if wanted. :)
Also some minor other cleanup of plugin APIs and built infrastructure.
All tested on MacOS only currently.
Includes:
- Cleanup of the plugin API, in particular generally changing
const char* to std::string
- Renaming environment variable BRO_PLUGINS to BRO_PLUGIN_PATH,
defaulting to <prefix>/lib/bro/plugins
- Reworking how dynamic plugins are searched and activated. See
doc/devel/plugins.rst for details.
- New @load-plugin directive to explicitly activate a plugin
- Support for Darwin. (Linux untested right now)
- The init-plugin updates come with support for "make test", "make
sdist", and "make bdist" (see how-to).
- Test updates.
Notes: The new hook mechanism, which allows plugins to hook into Bro's
core a well-defined points, is still essentially untested.
This is essentially the code from the dynamic-plugin branch except for
some pieces that I have split out into separate, earlier commits.
I'm going to updatre things in this branch going forward.
The magic files couldn't be in the root of that repo or else
libmagic would abort when it ran in to the .git* files and tried
to treat them like magic files, too.
- It's derived from the magic database of libmagic 5.14, but with most
everything not related to mime types removed.
- The custom database is always used by default for mime detection, but
the more verbose file type detection will fall back on the default
libmagic installation's database. The result is: mime type strings
are now guaranteed to be consistent across platforms, but the verbose
file type descriptions are not.
- The custom database gets installed in $prefix/share/bro/magic, and
should even be extensible if files with new patterns are added inside
the directory.
- The search path for the mime magic database can be controlled via
BROMAGIC environment variable.
- Remove mime_desc field from ftp.log.
- Stop using the mime/file type canonifier with unit tests.
- libmagic >= 5.04 is now a requirement.
