This commit rewrites catch and release, fixing issues with it and making
it fully cluster capable. A dedicated netcontrol_catch_release.log is
also added.
This is not quite done yet; a few more log messages are missing. There
should hopefully not be many big issues left.
This introduces two new events, NetControl::rule_new and
NetControl::rule_destroyed, which are raised when rules are first added
and then deleted from the internal state tracking.
* origin/topic/johanna/fix-analyzer-addition:
DTLS: Use magix constant from rfc5389 for stun detection.
DTLS: Fix binpac bug with DTLSv1.2 client hellos
Forgot to remove debug output.
DTLS: Fix interaction with STUN
Fix the way that child analyzers are added.
BIT-1604 #merged
Now the DTLS analyzer cleanly skips all STUN messages; no warnings
should be logged to dpd.log and parsing should work flawlessly with
intermixed STUN messages.
- When a log record is being "unrolled" (sub-records flattened
out into a single record), it's now possible to choose the
character/string to separate the outer name from the inner
name. This can be used to work around the problems
with ElasticSearch 2.0 not supporting dots "." in field names.
This value can be provided per-filter as well as a global
default value.
- Log fields can be renamed by providing a table per-filter
(or a global default) to rename fields for any log writer.
The name translation is performed after unrolling so the
value in the field name table must match whatever is being
used to separate field names.
For example if the unrolling separator was set to "*":
redef Log::default_unrolling_sep = "*";
The field name map would need to reflect it:
redef Log::default_field_name_map = {
["id*orig_h"] = "src",
["id*orig_p"] = "src_port",
["id*resp_h"] = "dst",
["id*resp_p"] = "dst_port",
};
The extension mechanism is basically the one that Seth introduced with
his intel extensions. The main difference lies in using a hook instead
of an event. An example policy implements whitelisting.
When Bro was compiled with broker disabled, then some Bro scripts
were referencing functions and types that were not defined. Fixed
by adding @ifdefs to several scripts. Removed one @ifdef because
it was causing several unit tests to fail.
Also fixed the @TEST-REQUIRES check in tests that rely on broker so
that such tests are skipped when broker is disabled.
BIT-1449 #merged
* origin/topic/dnthayer/ticket1449:
Rename broker BIF wrapper functions in a few more places
Sync the core/leaks/broker/data.bro test with broker/data.bro
Add missing tests for broker data BIFs
Code cleanup for some broker tests
Add script wrapper functions for broker data BIFs
Add script wrapper functions for broker BIFs
BIT-1574 #merged
* origin/topic/johanna/imap-starttls:
IMAP: add c++11 header file that gcc complains about.
IMAP: documentation and test updates
also generate an event when starttls is encounterd for imap.
Add support of getting server capabilities to IMAP parser.
Basic IMAP StartTLS analyzer.
Also renamed the "print" function to "send_print" and the "event"
function to "send_event" because Bro shows a syntax error when a
Bro script function is named "event" or "print".
There was a bug in the new parsing code, introduced in
708ede22c6 which parses validity times
incorrectly if they are before the year 2000. What happens in this case
is that the 2-digit year will be interpreted to be in the 21st century
(1999 will be parsed as 2099, e.g.).
* origin/topic/dnthayer/broker-namespace:
Split the broker main.bro into two scripts
Rename the BrokerStore namespace to Broker
Rename the BrokerComm namespace to Broker
BIT-1563 #merged
This analyzer parses the Remote Frame Buffer
protocol, usually referred to as the 'VNC protocol'.
It supports several dialects (3.3, 3.7, 3.8) and
also handles the Apple Remote Desktop variant.
It will log such facts as client/server versions,
authentication method used, authentication result,
height, width and name of the shared screen.
It also includes two testcases.
Todo: Apple Remote Desktop seems to have some
bytes prepended to the screen name. This is
not interepreted correctly.
There were some cases where the log would be missing a field
or data wouldn't get sent to file analysis. At least some of
this is fixed now and I get confused a bit less when I look
at the logs now.
Also, I made the default handling "FILE" so that things like
FILE_UNKNOWN wouldn't show up in the logs so regularly. It's
technically correct that way, but it doesn't look good and it's
correct as FILE often enough that it make sense to make it the
default I think.
- It works with DCE/RPC over SMB1+2 now.
- Using named pipes in 1+2 and the transaction cmd in SMB1.
- Base scripts based on work by Josh Liburdi.
- New dce_rpc.log. Feedback on how to make this log more compact
and useful would be appreciated.
The intel-framework now supports the new indicator type Intel::SUBNET.
As subnets are matched against seen addresses, the field matched was
introduced to indicate which indicator types caused the hit. A testcase
for subents was added and the old ones have been updated accordingly.
By addind debug output to Intel::insert() the testcase reveals that
updating an intel item will cause its metadata to be inserted again,
without the old being deleted.
BIT-1550 #merged
* origin/topic/johanna/netcontrol: (72 commits)
Update baselines and news
Move prefixtable back to all IPv6 internal handling.
NetControl: Add functions to search for rules affecting IPs/subnets
Add check_subnet bif that allows exact membership test for subnet tables.
Rewrite internal handling of rules.
Add bif that allows searching for all matching subnets in table.
Add signaling of succesful initialization of plugins to NetControl.
Add rule hooks to the acld plugin.
Add new logfiles for shunting and drops to netcontrol
Extend NetControl logging and fix bugs.
Update OpenFlow API and events.
small acld plugin fix
Revert "introduce &weaken attribute"
Fix crash when printing type of recursive structures.
Testcase for crash when a record contains a function referencing a record.
Rename Pacf to NetControl
fix acld plugin to use address instead of subnet (and add functions for conversion)
implement quarantine
miscelaneous missing bits and pieces
Acld implementation for Pacf - Bro side.
...
The server-reported file size was being collected poorly and if
a file name had a number in it, that was reported as the file
size instead of the actual size.
A new test is included to avoid reintroducing the problem.
Adds the functions
NetControl::find_rules_addr and NetControl::fund_rules_subnet
which return a vector containing all rules affecting a certain IP or
subnet.
This has no user-facing changes. It makes the internal handling of rules
much easier (no crazy duplicate rules in case our rules are added to
several backends).
It also fixes several open ends and small bugs in the process.
This does not really have many user-facing changes. The one big change
is that users now should initialize plugins in the
NetControl::init()
event instead of bro_init.
Once all plugins finished initializing and the NetControl framework
starts operations, the NetControl::init_done() event is raised.
Rules that are sent to NetControl before the plugins have finished
initializing are ignored - this is important when several plugins that
require external connections have to be initialized at the beginning.
Without this delay, rules could end up at the wrong plugin.
The hook name is NetControl::acld_rule_policy and allows the
modification of acld rules before they are sent out to the network.
This allows, e.g. network policies to use nullzero instead of drop in
certain circumstances.
* 'patch-4' of https://github.com/aeppert/bro:
(BIT-1545) Add "disable_analyzer_after_detection" en lieu of "skip_processing_after_detection"
I also removed the old disable_analyzer_after_detection option
completely - if someone wants that, they can just catch the event
themselves and call skip_further_processing.
I also adjusted the ssh test case to contain conn.log to prevent
re-addition of this problem in the future.
BIT-1545 #merged
