Add rate-limiting sampling mechanism for weird events

The generation of weird events, by default, are now rate-limited
according to these tunable options:

  - Weird::sampling_whitelist
  - Weird::sampling_threshold
  - Weird::sampling_rate
  - Weird::sampling_duration

The new get_reporter_stats() BIF also allows one to query the
total number of weirds generated (pre-sampling) which the new
policy/misc/weird-stats.bro script uses periodically to populate
a weird_stats.log.

There's also new reporter BIFs to allow generating weirds from the
script-layer such that they go through the same, internal
rate-limiting/sampling mechanisms:

  - Reporter::conn_weird
  - Reporter::flow_weird
  - Reporter::net_weird

Some of the code was adapted from previous work by Johanna Amann.
This commit is contained in:
Jon Siwek 2018-07-26 19:57:36 -05:00
parent e60b0bfb25
commit 35827eeb31
25 changed files with 1037 additions and 10 deletions

View file

@ -5,6 +5,9 @@
#include <sys/types.h>
#include <unordered_map>
#include <string>
#include "Dict.h"
#include "Val.h"
#include "Timer.h"
@ -264,6 +267,9 @@ public:
uint32 GetOrigFlowLabel() { return orig_flow_label; }
uint32 GetRespFlowLabel() { return resp_flow_label; }
bool PermitWeird(const char* name, uint64 threshold, uint64 rate,
double duration);
protected:
Connection() { persistent = 0; }
@ -328,6 +334,14 @@ protected:
analyzer::pia::PIA* primary_PIA;
Bro::UID uid; // Globally unique connection ID.
struct WeirdState {
WeirdState() { count = 0; sampling_start_time = 0; }
uint64 count = 0;
double sampling_start_time = 0;
};
std::unordered_map<std::string, WeirdState> weird_state;
};
class ConnectionTimer : public Timer {