* origin/topic/jsiwek/file-signatures:
File type detection changes and fix https.log {orig,resp}_fuids fields.
Various minor changes related to file mime type detection.
Refactor common MIME magic matching code.
Replace libmagic w/ Bro signatures for file MIME type identification.
Conflicts:
scripts/base/init-default.bro
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
BIT-1143 #merged
Notable changes:
- libmagic is no longer used at all. All MIME type detection is
done through new Bro signatures, and there's no longer a means to get
verbose file type descriptions (e.g. "PNG image data, 1435 x 170").
The majority of the default file magic signatures are derived
from the default magic database of libmagic ~5.17.
- File magic signatures consist of two new constructs in the
signature rule parsing grammar: "file-magic" gives a regular
expression to match against, and "file-mime" gives the MIME type
string of content that matches the magic and an optional strength
value for the match.
- Modified signature/rule syntax for identifiers: they can no longer
start with a '-', which made for ambiguous syntax when doing negative
strength values in "file-mime". Also brought syntax for Bro script
identifiers in line with reality (they can't start with numbers or
include '-' at all).
- A new Built-In Function, "file_magic", can be used to get all
file magic matches and their corresponding strength against a given
chunk of data
- The second parameter of the "identify_data" Built-In Function
can no longer be used to get verbose file type descriptions, though it
can still be used to get the strongest matching file magic signature.
- The "file_transferred" event's "descr" parameter no longer
contains verbose file type descriptions.
- The BROMAGIC environment variable no longer changes any behavior
in Bro as magic databases are no longer used/installed.
- Reverted back to minimum requirement of CMake 2.6.3 from 2.8.0
(it's back to being the same requirement as the Bro v2.2 release).
The bump was to accomodate building libmagic as an external project,
which is no longer needed.
Addresses BIT-1143.
* origin/topic/jdopheid/bro/edits_to_installation_and_getting_started:
Added some grammar and spelling corrections to Installation and Quick Start Guide.
Minor grammar edits to Installation and Quick Start pages
BIT-1123 #merged
BSD and debian-based Linux do not include gawk by default. Noticed
that a test was failing on these platforms due to the use of a bro-cut
option that requires gawk.
Added a few missing packages to the install instructions.
Combined the two different sets of GeoIP install instructions into one
location and updated it.
Fixed a couple minor typos.
I'm merging in the documentation branch to avoid that it keeps getting
out of sync. We still need to work on this a bit more, I'll summarize
in a mail.
* origin/topic/documentation: (68 commits)
Going over initial parts of the scripting overview.
Adding lines in scripting/index.txt. No other change.
Updating submodule.
Pass over the Using Bro section.
Applying doc updates from BIT-1063. (Anthony Verez)
Breaking lines in using/*.
More doc reorg, and a light pass over the first 3 sections.
Starting to put a new structure in place.
Updating submodule(s).
Updating submodule(s).
Adding some temporary testing/demonstration to front page.
Switching btest to topic/robin/parts branch.
Basic cross-referencing UIDs between files, btests, and baselines.
A truly embarassing number of spelling corrections.
Using redirection with bro-cut. Include initial btests for this document.
Corrected a mis-typed RST include.
Initial start for "Using Bro".
Spelling corrections.
Include Notice Policy shortcuts in the Scripting User Manual.
Notice::policy hooks and tests.
...
Conflicts:
src/3rdparty
I made a light pass over the text. Switched the includes over to the
new btest-include and adapted the other TEXT-EXECs a bit.
Also includes more tweaking all over the Sphinx setup.
