Added a new section that is intended as a how-to for configuring a Bro
cluster (this section does not discuss cluster architecture or theory)
that is aimed at beginners to Bro. Most of this content was moved here
from the BroControl doc (which is now intended as more of a reference guide
for more experienced users) and the load balancing FAQ on the website.
* origin/topic/jsiwek/file-signatures:
File type detection changes and fix https.log {orig,resp}_fuids fields.
Various minor changes related to file mime type detection.
Refactor common MIME magic matching code.
Replace libmagic w/ Bro signatures for file MIME type identification.
Conflicts:
scripts/base/init-default.bro
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
BIT-1143 #merged
Renamed the bro cluster doc to better indicate its purpose (it provides
a high-level overview rather than detailed configuration instructions).
Moved the location of the bro cluster doc in the index so that it makes
more sense (it is an introductory section, not a section about using bro).
Added links in the quick start guide and the bro cluster doc so that
readers can more easily locate more detailed information on configuring
a bro cluster.
Addresses BIT-1160
Notable changes:
- libmagic is no longer used at all. All MIME type detection is
done through new Bro signatures, and there's no longer a means to get
verbose file type descriptions (e.g. "PNG image data, 1435 x 170").
The majority of the default file magic signatures are derived
from the default magic database of libmagic ~5.17.
- File magic signatures consist of two new constructs in the
signature rule parsing grammar: "file-magic" gives a regular
expression to match against, and "file-mime" gives the MIME type
string of content that matches the magic and an optional strength
value for the match.
- Modified signature/rule syntax for identifiers: they can no longer
start with a '-', which made for ambiguous syntax when doing negative
strength values in "file-mime". Also brought syntax for Bro script
identifiers in line with reality (they can't start with numbers or
include '-' at all).
- A new Built-In Function, "file_magic", can be used to get all
file magic matches and their corresponding strength against a given
chunk of data
- The second parameter of the "identify_data" Built-In Function
can no longer be used to get verbose file type descriptions, though it
can still be used to get the strongest matching file magic signature.
- The "file_transferred" event's "descr" parameter no longer
contains verbose file type descriptions.
- The BROMAGIC environment variable no longer changes any behavior
in Bro as magic databases are no longer used/installed.
- Reverted back to minimum requirement of CMake 2.6.3 from 2.8.0
(it's back to being the same requirement as the Bro v2.2 release).
The bump was to accomodate building libmagic as an external project,
which is no longer needed.
Addresses BIT-1143.
* origin/topic/jsiwek/dns-improvements:
Rewrite DNS state tracking which matches queries and replies.
Change dns.log to include only standard DNS queries.
Improve DNS analysis.
* origin/topic/jdopheid/bro_documentation:
Updates to httpmonitor and mimestats documentation.
Updates to Logs and Cluster documentation
Updates to Bro IDS documentation
BIT-1127 #merged
* origin/topic/jdopheid/bro/edits_to_installation_and_getting_started:
Added some grammar and spelling corrections to Installation and Quick Start Guide.
Minor grammar edits to Installation and Quick Start pages
BIT-1123 #merged
The scope of dns.log is now only standard queries (OPCODE == 0). Other
kinds of queries (e.g. inverse query) were not handled correctly and
could interfere with the state tracking of the default DNS scripts.
The Plugin.cc file is now just a standard class, with the interface
changed a bit to make it more easy to write. However, there're still
some conventions that one must follow to make everything work (like
using the right namespace structure).
This commit also includes the option to compile built-in plugins
dynamically instead of statically by adding
SET(BRO_PLUGIN_BUILD_DYNAMIC TRUE) to their CMake config. This hasn't
been tested much yet, and I'm still undecided if it's somethign we
would want to do by default--but we could now if wanted. :)
Also some minor other cleanup of plugin APIs and built infrastructure.
All tested on MacOS only currently.
- Internals: move type alias table to private static BroType member.
- Sphinx extension: now uses absolute path to bro binary.
- reST ouput formatting: remove "param" from function desriptions
and change package overview docs so script link+summaries render
consistently.
BIT-1098
* origin/topic/jsiwek/broxygen:
Fix Broxygen-related compile errors.
Add a Broxygen coverage test.
Internal Broxygen organization/documentation/polish.
Add unit tests for Broxygen config file targets.
Change Broxygen config file format.
Broxygen doc-related test updates. Fix two regressions.
A couple documentation fixes.
Integrate new Broxygen functionality into Sphinx.
Implement majority of Broxygen features delegated to Bro.
Broxygen can now read a config file specifying particular targets.
Remove unneeded Broxygen comments in scan.bro.
Replace safe_basename/safe_dirname w/ SafeBasename/SafeDirname.
Add BIF interface for retrieving comments/docs.
Quick optimization to Broxygen doc gathering.
Flesh out Broxygen doc-gathering skeleton.
Refactor search_for_file() util function.
Initial skeleton of new Broxygen infrastructure.
