* origin/topic/robin/reporting:
Syslog BiF now goes through the reporter as well.
Avoiding infinite loops when an error message handlers triggers errors itself.
Renaming the Logger to Reporter.
Overhauling the internal reporting of messages to the user.
Updating a bunch of tests/baselines as well.
Conflicts:
aux/broccoli
policy.old/alarm.bro
policy/all.bro
policy/bro.init
policy/frameworks/notice/weird.bro
policy/notice.bro
src/SSL-binpac.cc
src/bro.bif
src/main.cc
* origin/topic/seth/net-stats-bif:
Removing a stray print statement.
Changed netstats (packet loss) handling to script-land.
Nice idea to pass the old data into a regular scheduled event!
Conflicts:
src/event.bif
The Logger class is now in charge of reporting all errors, warnings,
informational messages, weirds, and syslogs. All other components
route their messages through the global bro_logger singleton.
The Logger class comes with these reporting methods:
void Message(const char* fmt, ...);
void Warning(const char* fmt, ...);
void Error(const char* fmt, ...);
void FatalError(const char* fmt, ...); // Terminate Bro.
void Weird(const char* name);
[ .. some more Weird() variants ... ]
void Syslog(const char* fmt, ...);
void InternalWarning(const char* fmt, ...);
void InternalError(const char* fmt, ...); // Terminates Bro.
See Logger.h for more information on these.
Generally, the reporting now works as follows:
- All non-fatal message are reported in one of two ways:
(1) At startup (i.e., before we start processing packets),
they are logged to stderr.
(2) During processing, they turn into events:
event log_message%(msg: string, location: string%);
event log_warning%(msg: string, location: string%);
event log_error%(msg: string, location: string%);
The script level can then handle them as desired.
If we don't have an event handler, we fall back to
reporting on stderr.
- All fatal errors are logged to stderr and Bro terminates
immediately.
- Syslog(msg) directly syslogs, but doesn't do anything else.
The three main types of messages can also be generated on the
scripting layer via new Log::* bifs:
Log::error(msg: string);
Log::warning(msg: string);
Log::message(msg: string);
These pass through the bro_logger as well and thus are handled in the
same way. Their output includes location information.
More changes:
- Removed the alarm statement and the alarm_hook event.
- Adapted lots of locations to use the bro_logger, including some
of the messages that were previously either just written to
stdout, or even funneled through the alarm mechanism.
- No distinction anymore between Error() and RunTime(). There's
now only one class of errors; the line was quite blurred already
anyway.
- util.h: all the error()/warn()/message()/run_time()/pinpoint()
functions are gone. Use the bro_logger instead now.
- Script errors are formatted a bit differently due to the
changes. What I've seen so far looks ok to me, but let me know
if there's something odd.
Notes:
- The default handlers for the new log_* events are just dummy
implementations for now since we need to integrate all this into
the new scripts anyway.
- I'm not too happy with the names of the Logger class and its
instance bro_logger. We now have a LogMgr as well, which makes
this all a bit confusing. But I didn't have a good idea for
better names so I stuck with them for now.
Perhaps we should merge Logger and LogMgr?
- Removed the net_stats_update event.
- Created a net_stats function for building and retrieving the
current network statistics.
- Removed the internal timer for firing the net_stats_update event
along with the global heartbeat_interval variable.
- Updated the netstats script to use the new BiF.
- Updated the stats script to use the new BiF.
- Defaults for all built-in asset tracking changed to LOCAL_HOSTS
- Added a tuning script for changing asset tracking
to ALL_HOSTS in all of the core scripts that do
asset tracking.
- Default Notice::policy files notices instead of alarming on them.
- Moved KnownHosts::Info back to export section because
the log_known_hosts event can't be defined in the
export section without it.
- Moved the Malware Hash Registry detection out of
the core HTTP protocol scripts and added it to the
all.bro script.
- Split enum values into two separate enums.
- Renamed to fit the enum naming convention.
- New global variable named default_asset_tracking
that changes default behavior of any script that
tracks assets, usually by storing some amount
of information about the network in memory.
* origin/topic/seth/ssl-binpac:
Fixed bug due to vectors now initially indexed on 0.
Finished core support for new SSL analyzer.
SSL analyzer changes with accompanying BiF.
A table_s_of_s type to get around bifcl type limitation.
Regenerated the Mozilla CA bundle without the untrusted server authentication certs.
Complete rewrite to SSL analyzer.
Conflicts:
src/AnalyzerTags.h
src/CMakeLists.txt
Notes:
- Haven't looked at the script-level, postponed to
policy-scripts-new.
- I renamed X509Extension to X509_extension for consistency.
- The install_pcap_filter BiF no longer will prevent
a filter from being installed if a -f filter was
provided. This was already causing a problem where
command line supplied filters weren't working.
The pcap.bro script is now where the filter update
limitation is placed making it possible for users
to still update the filter at runtime if they really
want to.
If 'use_conn_size_analyzer' is true, the event engine tracks number of
packets and raw IP bytes per connection. If report_conn_size_analyzer
is true, these values are included as four new columns into conn.log
I changed conn.bro so that the value of report_conn_size_analyzer
follows that of use_conn_size_analyzer. For the new conn.log, we
probably want to get rid of report_conn_size_analyzer anyway.
filter. (Seth Hall and Robin Sommer)
- Merging in the patch from #264, which provides support for mixed
VLAN and MPLS traffic.
- Changing Bro's default filter from being built dynamically to being
a static "ip or not ip". To get the old behaviour back (i.e., the
dynamically built filter), redef "all_packets" to false.
- print-filter.bro now always prints the filter that Bro is actually
using, even if overriden from the command line.
* origin/topic/robin/conn-ids:
Moving uid from conn_id to connection, and making output determistic if a hash seed is given.
Extending conn_id with a globally unique identifiers.
Removing everything related to trace rewriting.
(I wasn't too careful in ensuring that I catch everything in the
scripts; Seth is working on those anyway.)
(Merging by cherry-picking the corresponding commit, as the branch was
accidentally made off of the logging stuff).
The &log keyword now operates as discussed:
- When associated with individual record fields, it defines them
as being logged.
- When associated with a complete record type, it defines all fields
to be logged.
- When associated with a record extension, it defines all added
fields to be logged.
Note that for nested record types, the inner fields must likewise
be declared with &log. Consequently, conn_id is now declared with
&log in bro.init.
Vectors are now allowed to be logged and will be recorded as an
ordered set of items.
* I haven't removed handwritten analyzer code yet although it isn't built anymore.
* The ssl.bro script is just an example and doesn't keep any state yet.
module LogAscii;
export {
# Output everything to stdout rather than into files. This is primarily
# for testing purposes.
const output_to_stdout = F &redef;
# The separator between fields.
const separator = "\t" &redef;
# True to include a header line with column names.
const include_header = T &redef;
}
* Removed terminate-connection.bro
as a dependency on notice-actions-filters.bro
* geo_location type now has full set of defaults.
* Many conn.bro logged variables are now optional
instead of having defaults.
* Many updates to the notice.bro script. Probably
many more to come. Mostly involved with
modernizing it by extracting functionality,
reducing chained dependencies, and modularization.
* Updated used Notice API in ssh.bro to the
modified notice.bro script.
This follows rather closely how rotation currently works in
rotate-logs.bro. logging.bro now defines:
# Default rotation interval; zero disables rotation.
const default_rotation_interval = 0secs &redef;
# Default naming suffix format.
const default_rotation_date_format = "%y-%m-%d_%H.%M.%S" &redef;
# Default postprocessor for writers outputting into files.
const default_rotation_postprocessor = "" &redef;
# Default function to construct the name of the rotated file.
# The default implementation includes
# default_rotation_date_format into the file name.
global default_rotation_path_func: function(info: RotationInfo) : string &redef;
Writer support for rotation is optional, usually it will only make
sense for file-based writers.
TODO: Currently, there's no way to customize rotation on a per file
basis, there are only the global defaults as described above.
Individual customization is coming next.
