Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-07 00:58:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Add rate-limiting sampling mechanism for weird events

Browse source 
The generation of weird events, by default, are now rate-limited
according to these tunable options:

  - Weird::sampling_whitelist
  - Weird::sampling_threshold
  - Weird::sampling_rate
  - Weird::sampling_duration

The new get_reporter_stats() BIF also allows one to query the
total number of weirds generated (pre-sampling) which the new
policy/misc/weird-stats.bro script uses periodically to populate
a weird_stats.log.

There's also new reporter BIFs to allow generating weirds from the
script-layer such that they go through the same, internal
rate-limiting/sampling mechanisms:

  - Reporter::conn_weird
  - Reporter::flow_weird
  - Reporter::net_weird

Some of the code was adapted from previous work by Johanna Amann.
This commit is contained in:
Jon Siwek 2018-07-26 19:57:36 -05:00
parent e60b0bfb25
commit 35827eeb31
25 changed files with 1037 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

51
src/Reporter.h
View file

@ -7,6 +7,10 @@
#include <list>
#include <utility>
#include <string>
#include <map>
#include <unordered_set>
#include <unordered_map>
#include "util.h"
#include "EventHandler.h"
@ -36,6 +40,11 @@ protected:
class Reporter {
public:
using IPPair = std::pair<IPAddr, IPAddr>;
using WeirdCountMap = std::unordered_map<std::string, uint64>;
using WeirdFlowMap = std::map<IPPair, WeirdCountMap>;
using WeirdSet = std::unordered_set<std::string>;
Reporter();
~Reporter();
@ -76,7 +85,6 @@ public:
// that may lead to incorrectly processing a connnection.
void Weird(const char* name); // Raises net_weird().
void Weird(Connection* conn, const char* name, const char* addl = ""); // Raises conn_weird().
void Weird(Val* conn_val, const char* name, const char* addl = ""); // Raises conn_weird().
void Weird(const IPAddr& orig, const IPAddr& resp, const char* name); // Raises flow_weird().
// Syslog a message. This methods does nothing if we're running
@ -120,6 +128,30 @@ public:
// Signals that we're done processing an error handler event.
void EndErrorHandler() { --in_error_handler; }
/**
* Reset/cleanup state tracking for a "net" weird.
*/
void ResetNetWeird(const std::string& name);
/**
* Reset/cleanup state tracking for a "flow" weird.
*/
void ResetFlowWeird(const IPAddr& orig, const IPAddr& resp);
/**
* Return the total number of weirds generated (counts weirds before
* any rate-limiting occurs).
*/
uint64 GetWeirdCount() const
{ return weird_count; }
/**
* Return number of weirds generated per weird type/name (counts weirds
* before any rate-limiting occurs).
*/
const WeirdCountMap& GetWeirdsByType() const
{ return weird_count_by_type; }
private:
void DoLog(const char* prefix, EventHandlerPtr event, FILE* out,
Connection* conn, val_list* addl, bool location, bool time,
@ -129,6 +161,11 @@ private:
// contain format specifiers
void WeirdHelper(EventHandlerPtr event, Val* conn_val, const char* addl, const char* fmt_name, ...) __attribute__((format(printf, 5, 6)));;
void WeirdFlowHelper(const IPAddr& orig, const IPAddr& resp, const char* fmt_name, ...) __attribute__((format(printf, 4, 5)));;
void UpdateWeirdStats(const char* name);
inline bool WeirdOnSamplingWhiteList(const char* name)
{ return weird_sampling_whitelist.find(name) != weird_sampling_whitelist.end(); }
bool PermitNetWeird(const char* name);
bool PermitFlowWeird(const char* name, const IPAddr& o, const IPAddr& r);
int errors;
bool via_events;
@ -138,6 +175,18 @@ private:
bool errors_to_stderr;
std::list<std::pair<const Location*, const Location*> > locations;
uint64 weird_count;
WeirdCountMap weird_count_by_type;
WeirdCountMap net_weird_state;
WeirdFlowMap flow_weird_state;
WeirdSet weird_sampling_whitelist;
uint64 weird_sampling_threshold;
uint64 weird_sampling_rate;
double weird_sampling_duration;
};
extern Reporter* reporter;