At the moment, we have support for:
elliptic_curves: client supported elliptic curves
ec_point_formats: list of client supported EC point formats
application_layer_protocol_negotiation: list of supported application layer protocols (used for spdy/http2 negotiation)
server_name: server name sent by client. This was supported before, but... a bit brittle.
This prevents the worker nodes from crashing, when request_key is used
in cluster mode and called on the worker and the manager nodes (i.e. when
a non-cluster-aware script is used).
Addresses BIT-1177
* origin/topic/jsiwek/file-signatures:
File type detection changes and fix https.log {orig,resp}_fuids fields.
Various minor changes related to file mime type detection.
Refactor common MIME magic matching code.
Replace libmagic w/ Bro signatures for file MIME type identification.
Conflicts:
scripts/base/init-default.bro
testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log
testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log
BIT-1143 #merged
* origin/topic/jsiwek/improve-type-checks:
Add unit tests covering vector/set/table ctors/inits.
Fix parsing of "local" named table constructors.
Improve type checking of records, addresses BIT-1159.
BIT-1159 #merged
- Removed "binary" and "octet-stream" mime type detections. They don't
provide any more information than an uninitialized mime_type field
which implicitly means no magic signature matches and so the media
type is unknown to Bro.
- Slight change to "text/plain" signature. It's still not the most
accurate, which is reflected in its -20 strength value.
- The logic for adding file ids to {orig,resp}_fuids fields of
the http.log incorrectly depended on the state of
{orig,resp}_mime_types fields, so sometimes not all file ids
associated w/ the session were logged.
* origin/topic/seth/json-formatter:
Updating a couple of tests.
Expanded support for modifying the timestamp format in the JSON formatter.
Ascii input reader now supports all config options per-input stream.
Added an option to the JSON formatter to use ISO 8601 for timestamps.
Refactored formatters and updated the the writers a bit.
Includes some minor bugfixes and cleanup at various places, including
in old code.
The MHR script involves a "when" statement which can be expensive due to
the way it clones frames/vals. In this case, the fa_file record is
expensive to clone, but this change works around that by unrolling only
the necessary fields from it that are needed to populate a Notice::Info
record. A drawback to this is that the full fa_file or connection
records aren't available in the Notice::Info record when evaluating
Notice::policy hooks for MHR hit notices (though they can possibly be
recovered by using e.g. the lookup_connection() builtin_function).
- It's not *exactly* ISO 8601 which doesn't seem to support
subseconds, but subseconds are very important to us and
most things that support ISO8601 seem to also support subseconds
in the way I'm implemented it.
- Formatters have been abstracted similarly to readers and writers now.
- The Ascii writer has a new option for writing out logs as JSON.
- The Ascii writer now has all options availble as per-filter
options as well as global.
- Improve or just remove some file magic signatures ported from libmagic
that were too general and matched incorrectly too often.
- Fix MHR script's use of fa_file$mime_type before checking if it's
initialized. It may be uninitialized if no signatures match.
- The "fa_file" record now contains a "mime_types" field that contains
all magic signatures that matched the file content (where the
"mime_type" field is just a shortcut for the strongest match).
Notable changes:
- libmagic is no longer used at all. All MIME type detection is
done through new Bro signatures, and there's no longer a means to get
verbose file type descriptions (e.g. "PNG image data, 1435 x 170").
The majority of the default file magic signatures are derived
from the default magic database of libmagic ~5.17.
- File magic signatures consist of two new constructs in the
signature rule parsing grammar: "file-magic" gives a regular
expression to match against, and "file-mime" gives the MIME type
string of content that matches the magic and an optional strength
value for the match.
- Modified signature/rule syntax for identifiers: they can no longer
start with a '-', which made for ambiguous syntax when doing negative
strength values in "file-mime". Also brought syntax for Bro script
identifiers in line with reality (they can't start with numbers or
include '-' at all).
- A new Built-In Function, "file_magic", can be used to get all
file magic matches and their corresponding strength against a given
chunk of data
- The second parameter of the "identify_data" Built-In Function
can no longer be used to get verbose file type descriptions, though it
can still be used to get the strongest matching file magic signature.
- The "file_transferred" event's "descr" parameter no longer
contains verbose file type descriptions.
- The BROMAGIC environment variable no longer changes any behavior
in Bro as magic databases are no longer used/installed.
- Reverted back to minimum requirement of CMake 2.6.3 from 2.8.0
(it's back to being the same requirement as the Bro v2.2 release).
The bump was to accomodate building libmagic as an external project,
which is no longer needed.
Addresses BIT-1143.
* origin/topic/jazoff/notice_file_info:
Include file information in notices
BIT-1113 #merged
I'm wondering if there's a way to test creating these emails ... ?
- The reassembly behavior can be modified per-file by enabling or
disabling the reassembler and/or modifying the size of the reassembly
buffer.
- Changed the file extraction analyzer to use the stream to avoid
issues with the chunk based approach not immediately triggering
the file_new event due to mime-type detection delay. Early chunks
frequently ended up lost before.
- Generally things are working now and I'd consider this in testing.
Instead of storing the entire notice in Notice::suppressing,
just store the time the notice should be suppressed until.
This has the same functionality, except that end_suppression can no
longer be generated.
* origin/topic/seth/ie11-software-parsing:
Updated software framework to support parsing IE11 user-agent strings.
Fix the irc_reply event for several server message types.
BIT-1104 #merged
- Enable manager to associate analyzers with a MIME type. With that,
one can now say enable all analyzers for, e.g., "image/gif". This is
exposed to script-land as
Files::add_analyzers_for_mime_type(f: fa_file, mtype: string)
For MIME types identified via libmagic, this happens automatically
(via the file_new() handler in files/main.bro).
- Extend the analyzer API to better match that of protocol analyzers:
- Adding unique analyzer IDs so that we can refer to instances
from script-land.
- Adding subtypes to Components so that a single analyzer
implementation can support different types of analyzers
internally.
- Add an analyzer method SetTag() that allows to set the tag after
construction.
- Adding Init() and Done() methods for consistency with what other
classes offer.
- Add debug logging to the file_analysis stream.
TODO: test cases missing for the new script-land functionality.
Add a "broxygen" domain Sphinx extension w/ directives to allow
on-the-fly documentation to be generated w/ Bro and included in files.
This means all autogenerated reST docs are now done by Bro. The odd
CMake/Python glue scipts which used to generate some portions are now
gone. Bro and the Sphinx extension handle checking for outdated docs
themselves.
Parallel builds of `make doc` target should now work (mostly because
I don't think there's any tasks that can be done in parallel anymore).
Overall, this seems to simplify things and make the Broxygen-generated
portions of the documentation visible/traceable from the main Sphinx
source tree. The one odd thing still is that per-script documentation
is rsync'd in to a shadow copy of the Sphinx source tree within the
build dir. This is less elegant than using the new broxygen extension
to make per-script docs, but rsync is faster and simpler. Simpler as in
less code because it seems like, in the best case, I'd need to write a
custom Sphinx Builder to be able to get that to even work.
* origin/topic/dnthayer/doc-changes-for-2.2:
Add gawk to list of optional packages
Add more script package README files
Add NEWS about new features of broctl and upgrade info
Some documentation of commented-out items was being output to HTML,
which was causing that documentation to be merged with the
documentation of the next item. Fixed by changing "##" to "#" so
that the doc comments are not included in the HTML.
Snapshotting the work in this branch. I'll merge it again later as we
get closer to the release.
* origin/topic/dnthayer/doc-changes-for-2.2: (29 commits)
Add README files for base/protocols
Fix incorrect uses of reST directives
Fix typos and formatting in the BiFs docs
Fix typos and formatting in the base/utils docs
Fix typos and formatting in the other protocol docs
Fix typos and formatting in the ssl protocol docs
Fix typos and formatting in the http protocol docs
Fix typos and formatting in the ftp protocol docs
Fix typos and formatting in the dns protocol docs
Fix typos and formatting in the dhcp protocol docs
Adjust line numbers to match changes in conn/main.bro
Fix typos and formatting in the conn protocol docs
Update FreeBSD install instructions
Improvements to file analysis docs
Add README files for most Bro frameworks
Fix typos and formatting in various other framework docs
Fix typos and formatting in the software framework docs
Fix typos and formatting in the sumstats docs
Fix typos and formatting in the packet filter docs
Fix typos and formatting in the logging framework docs
...
Fixed reference to wrong field name.
Added documentation of a function arg.
Added a couple references to other parts of the documentation.
Explained how not specifying extraction filename results in automatic
filename generation.
Several other minor clarifications.
The text from these README files appears on the "Bro Script Packages"
page after building the documentation. The text for these was mostly just
copied from the existing docs.
