Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-08 09:38:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

New Bro Manual Development Edition and basic.css to fix btest output overflow problem (Update 1).

Browse source
This commit is contained in:
Rafael Bonilla 2013-11-21 12:56:00 -06:00
parent 69ef268a44
commit a33d25b3bd
6 changed files with 647 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

9
doc/_static/basic.css vendored
View file

@ -439,8 +439,17 @@ td.linenos pre {
color: #aaa;
}
.highlight-guess {
overflow:auto;
}
.highlight-none {
overflow:auto;
}
table.highlighttable {
margin-left: 0.5em;
overflow:scroll;
}
table.highlighttable td {

155
doc/broids/index.rst Normal file
View file

@ -0,0 +1,155 @@
__ http://www.bro.org/sphinx-git/scripts/base/protocols/ftp/main.bro.html#id-FTP::parse_ftp_reply_code
__ http://www.bro.org/sphinx-git/frameworks/sumstats.html
__ http://www.bro.org/sphinx-git/frameworks/notice.html
__ http://www.bro.org/sphinx-git/_downloads/detect-bruteforcing.bro
__ http://www.bro.org/sphinx-git/scripts/policy/frameworks/files/detect-MHR.bro.html
.. _bro-ids:
=======
Bro IDS
=======
An Intrusion Detection System (IDS) allows you to detect suspicious activities happening on your network as a result of a past or active
attack. Because of its programming capabilities, Bro can easily be configured to behave like traditional IDSs and detect common attacks
with well known patterns, or you can create your own scripts to detect conditions specific to your particular case.
In the following sections, we present a few examples of common uses of Bro as an IDS.
------------------------------------------------
Detecting an FTP Bruteforce attack and notifying
------------------------------------------------
For the purpose of this exercise, we define FTP bruteforcing as too many rejected usernames and passwords occurring from a single address.
We start by defining a threshold for the number of attempts and a monitoring interval in minutes.
.. code:: bro
export {
## How many rejected usernames or passwords are required before being
## considered to be bruteforcing.
const bruteforce_threshold: double = 20 &redef;
## The time period in which the threshold needs to be crossed before
## being reset.
const bruteforce_measurement_interval = 15mins &redef;
}
Now, using the ftp_reply event, we check for error codes from the `500 series <http://en.wikipedia.org/wiki/List_of_FTP_server_return_codes>`_ for the "USER" and "PASS" commands, representing rejected usernames or passwords. For this, we can use the `FTP::parse_ftp_reply`__ function to break down the reply code and check if the first digit is a "5" or not. If true, we then use the `SumStats`__ framework to keep track of the number of failed attempts.
.. code:: bro
event ftp_reply(c: connection, code: count, msg: string, cont_resp: bool)
{
local cmd = c$ftp$cmdarg$cmd;
if ( cmd == "USER" || cmd == "PASS" )
{
if ( FTP::parse_ftp_reply_code(code)$x == 5 )
SumStats::observe("ftp.failed_auth", [$host=c$id$orig_h], [$str=cat(c$id$resp_h)]);
}
}
Next, we use the SumStats framework to automatically print a message on the console alerting of the attack when the number of failed attempts
exceeds the specified threshold during the measuring interval.
.. code:: bro
event bro_init()
{
local r1: SumStats::Reducer = [$stream="ftp.failed_auth", $apply=set(SumStats::UNIQUE), $unique_max=double_to_count(bruteforce_threshold+2)];
SumStats::create([$name="ftp-detect-bruteforcing",
$epoch=bruteforce_measurement_interval,
$reducers=set(r1),
$threshold_val(key: SumStats::Key, result: SumStats::Result) =
{
return result["ftp.failed_auth"]$num+0.0;
},
$threshold=bruteforce_threshold,
$threshold_crossed(key: SumStats::Key, result: SumStats::Result) =
{
local r = result["ftp.failed_auth"];
local dur = duration_to_mins_secs(r$end-r$begin);
local plural = r$unique>1 ? "s" : "";
local message = fmt("%s had %d failed logins on %d FTP server%s in %s", key$host, r$num, r$unique, plural, dur);
}]);
}
Printing a message on the console is a good start but it will be better if we raise an alarm instead using the `Notice`__ framework. For this, we need to define a new Notice type and trigger the alarm under the right
conditions. Below is the final code for our script.
.. code:: bro
##! FTP brute-forcing detector, triggering when too many rejected usernames or
##! failed passwords have occurred from a single address.
@load base/protocols/ftp
@load base/frameworks/sumstats
@load base/utils/time
module FTP;
export {
redef enum Notice::Type += {
## Indicates a host bruteforcing FTP logins by watching for too
## many rejected usernames or failed passwords.
Bruteforcing
};
## How many rejected usernames or passwords are required before being
## considered to be bruteforcing.
const bruteforce_threshold: double = 20 &redef;
## The time period in which the threshold needs to be crossed before
## being reset.
const bruteforce_measurement_interval = 15mins &redef;
}
event bro_init()
{
local r1: SumStats::Reducer = [$stream="ftp.failed_auth", $apply=set(SumStats::UNIQUE), $unique_max=double_to_count(bruteforce_threshold+2)];
SumStats::create([$name="ftp-detect-bruteforcing",
$epoch=bruteforce_measurement_interval,
$reducers=set(r1),
$threshold_val(key: SumStats::Key, result: SumStats::Result) =
{
return result["ftp.failed_auth"]$num+0.0;
},
$threshold=bruteforce_threshold,
$threshold_crossed(key: SumStats::Key, result: SumStats::Result) =
{
local r = result["ftp.failed_auth"];
local dur = duration_to_mins_secs(r$end-r$begin);
local plural = r$unique>1 ? "s" : "";
local message = fmt("%s had %d failed logins on %d FTP server%s in %s", key$host, r$num, r$unique, plural, dur);
NOTICE([$note=FTP::Bruteforcing,
$src=key$host,
$msg=message,
$identifier=cat(key$host)]);
}]);
}
event ftp_reply(c: connection, code: count, msg: string, cont_resp: bool)
{
local cmd = c$ftp$cmdarg$cmd;
if ( cmd == "USER" || cmd == "PASS" )
{
if ( FTP::parse_ftp_reply_code(code)$x == 5 )
SumStats::observe("ftp.failed_auth", [$host=c$id$orig_h], [$str=cat(c$id$resp_h)]);
}
}
As a final note, the `detect-bruteforcing.bro`__ script above is include with Bro out of the box, so you only need to load it at startup to instruct Bro to detect and notify of FTP bruteforce attacks.
-------------
Other Attacks
-------------
Detecting SQL Injection attacks
-------------------------------
Checking files against known malware hashes
-------------------------------------------
Files transmitted on your network could either be completely harmless or contain viruses and other threats. One possible action against
this threat is to compute the hashes of the files and compare them against a list of known malware hashes. Bro simplifies this task
by offering a `detect-MHR.bro`__ script that creates and compares
hashes against the `Malware Hash Registry <https://www.team-cymru.org/Services/MHR/>`_ maintained by Team Cymru. You only need to load this
script along with your other scripts at startup time.

256
doc/httpmonitor/index.rst Normal file
View file

@ -0,0 +1,256 @@
__ http://www.bro.org/sphinx-git/scripts/base/protocols/http/main.bro.html
__ http://www.bro.org/sphinx-git/frameworks/file-analysis.html
.. _http-monitor:
================================
Monitoring HTTP Traffic with Bro
================================
Bro can be used to log the entire HTTP traffic from your network to the http.log file.
This file can then be used for analysis and auditing purposes.
In the sections below we briefly explain the structure of the http.log file. Then, we
show you how to perform basic HTTP traffic monitoring and analysis tasks with Bro. Some
of these ideas and techniques can later be applied to monitor different protocols in a
similar way.
----------------------------
Introduction to the HTTP log
----------------------------
The http.log file contains a summary of all HTTP requests and responses sent over a Bro-monitored
network. Here are the first few columns of
``http.log``::
# ts uid orig_h orig_p resp_h resp_p
1311627961.8 HSH4uV8KVJg 192.168.1.100 52303 192.150.187.43 80
Every single line in this log starts with a timestamp, a unique connection identifier (UID), and a
connection 4-tuple (originator host/port and responder host/port). The UID can be used to
identify all logged activity (possibly across multiple log files) associated
with a given connection 4-tuple over its lifetime.
The remaining columns detail the activity that's occurring. For example, the columns on the line below
(shortened for brevity) show a request to the root of Bro website::
# method host uri referrer user_agent
GET bro.org / - <...>Chrome/12.0.742.122<...>
Network administrators and security engineers, for instance, can use the information in this log to understand
the HTTP activity on the network and troubleshoot network problems or search for anomalous activities. At this
point, we would like to stress out the fact that there is no just one right way to perform analysis; it will
depend on the expertise of the person doing the analysis and the specific details of the task to accomplish.
For more information about how to handle the HTTP protocol in Bro, including a complete list
of the fields available in http.log, go to Bro's HTTP reference `page`__.
------------------------
Detecting a Proxy Server
------------------------
A proxy server is a device on your network configured to request a service on behalf of a third system; one of the
most common examples is a Web proxy server. A client without Internet access connects to the proxy and requests
a Web page; the proxy then sends the request to the actual Web server, receives the response and passes it to the original
client.
Proxies were conceived to help manage a network and provide better encapsulation. By themselves, proxies are not a security
threat, but a misconfigured or unauthorized proxy can allow others, either inside or outside the network, to access any
Web site and even conduct malicious activities anonymously using the network resources.
What Proxy Server traffic looks like
-------------------------------------
In general, when a client starts talking with a proxy server, the traffic consists of two parts: (i) a GET request, and
(ii) an HTTP/ reply::
Request: GET http://www.bro.org/ HTTP/1.1
Reply: HTTP/1.0 200 OK
This will differ from traffic between a client and a normal Web server because GET requests should not include "http" on
the string. So we can use this to identify a proxy server.
We can write a basic script in Bro to handle the http_reply event and detect a reply for a ``GET http://`` request.
.. code:: bro
event http_reply(c: connection, version: string, code: count, reason: string)
{
if ( /^[hH][tT][tT][pP]:/ in c$http$uri && c$http$status_code == 200 )
{
print fmt("A local server is acting as an open proxy: ", c$id$resp_h);
}
}
Basically, the script is checking for a "200 OK" status code on a reply for a request that includes "http:". In reality, the HTTP
protocol defines several success status codes other than 200, so we will extend our basic script to also consider the additional codes.
.. code:: bro
export {
global success_status_codes: set[count] = {
200,
201,
202,
203,
204,
205,
206,
207,
208,
226,
304
};
}
event http_reply(c: connection, version: string, code: count, reason: string)
{
if ( /^[hH][tT][tT][pP]:/ in c$http$uri && c$http$status_code in success_status_codes )
{
print fmt("A local server is acting as an open proxy: ", c$id$resp_h);
}
}
Next, we will make sure that the responding proxy is part of our local network.
.. code:: bro
export {
global success_status_codes: set[count] = {
200,
201,
202,
203,
204,
205,
206,
207,
208,
226,
304
};
}
event http_reply(c: connection, version: string, code: count, reason: string)
{
if ( Site::is_local_addr(c$id$resp_h) && /^[hH][tT][tT][pP]:/ in c$http$uri && c$http$status_code in success_status_codes )
{
print fmt("A local server is acting as an open proxy: ", c$id$resp_h);
}
}
Finally, our goal should be to generate an alert when a proxy has been detected instead of printing a message on the console output.
For that, we will tag the traffic accordingly and define a new ``Open_Proxy`` ``Notice`` type to alert of all tagged communications. Once a
notification has been fired, we will further suppress it for one day. Below is the complete script.
.. code:: bro
@load base/frameworks/notice
module HTTP;
export {
redef enum HTTP::Tags += {
OPEN_PROXY_TAG
};
redef enum Notice::Type += {
Open_Proxy
};
global success_status_codes: set[count] = {
200,
201,
202,
203,
204,
205,
206,
207,
208,
226,
304
};
}
redef Notice::emailed_types += {
Open_Proxy,
};
function open_proxy_only(rec: HTTP::Info) : bool
{
# Only write out connections with the OPEN_PROXY_TAG.
return OPEN_PROXY_TAG in rec$tags;
}
event http_reply(c: connection, version: string, code: count, reason: string)
{
# make sure responding host is local
#if ( Site::is_local_addr(c$id$resp_h) && /^[hH][tT][tT][pP]:/ in c$http$uri && c$http$status_code in success_status_codes )
{
add c$http$tags[OPEN_PROXY_TAG];
local ident = cat(c$id$resp_h);
if ( c$http?$host ) #check if the optional host field exists in http
{
print fmt("Originator host: %s", c$id$orig_h);
NOTICE([$note=HTTP::Open_Proxy,
$msg=cat("A local server is acting as an open proxy: ", c$id$resp_h),
$conn=c, $identifier=cat(ident, c$id$resp_h),
$suppress_for=1day]);
}
}
}
event bro_init()
{
#Creating a new filter for all open proxy logs.
local filter: Log::Filter = [$name="open_proxy", $path="open_proxy", $pred=open_proxy_only];
Log::add_filter(HTTP::LOG, filter);
}
----------------
Inspecting Files
----------------
Files are often transmitted on regular HTTP conversations between a client and a server. Most of the time these files are harmless,
just images and some other multimedia content, but there are also types of files, specially executable files, that can damage
your system. We can instruct Bro to create a copy of all executable files that it sees for later analysis using the `File Analysis
Framework`__ (introduced with Bro 2.2) as shown in the following script.
.. code:: bro
global ext_map: table[string] of string = {
["application/x-dosexec"] = "exe",
} &default ="";
event file_new(f: fa_file)
{
local ext = "";
if ( f?$mime_type )
ext = ext_map[f$mime_type];
local fname = fmt("%s-%s.%s", f$source, f$id, ext);
Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=fname]);
}
Bro will extract all files from the traffic and write them on a new ``extract_files/`` subdirectory and change the file name with the right
suffix (extension) based on the content of the ext_map table. So, if you want to do the same for other extracted files besides executables
you just need to add those types to the ``ext_map`` table like this.
.. code:: bro
global ext_map: table[string] of string = {
["application/x-dosexec"] = "exe",
["text/plain"] = "txt",
["image/jpeg"] = "jpg",
["image/png"] = "png",
["text/html"] = "html",
} &default ="";
Bro will now write the appropriate suffix for text, JPEG, PNG, and HTML files stored in the ``extract_files/`` subdirectory.

38
doc/index.rst
View file

@ -1,9 +1,12 @@
.. Bro documentation master file
=================
Bro Documentation
=================
==========
Bro Manual
==========
Introduction Section
====================
.. toctree::
:maxdepth: 2
@ -11,13 +14,38 @@ Bro Documentation
intro/index.rst
install/index.rst
quickstart/index.rst
using/index.rst
..
Using Bro Section
=================
.. toctree::
:maxdepth: 2
logs/index.rst
httpmonitor/index.rst
broids/index.rst
mimestats/index.rst
cluster/index.rst
..
Reference Section
=================
.. toctree::
:maxdepth: 2
scripting/index.rst
frameworks/index.rst
cluster/index.rst
script-reference/index.rst
components/index.rst
..
* `Notice Index <bro-noticeindex.html>`_ (TODO: Move to reference
section, but can't figure out how to include it into toctree)
* :ref:`General Index <genindex>`
* :ref:`search`

49
doc/using/index.rst → doc/logs/index.rst
View file

@ -1,3 +1,13 @@
__ http://www.bro.org/sphinx-git/scripts/base/protocols/http/main.bro.html#type-HTTP::Info
__ http://www.bro.org/sphinx-git/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info
__ http://www.bro.org/sphinx-git/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info
__ http://www.bro.org/sphinx-git/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo
__ http://www.bro.org/sphinx-git/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info
__ http://www.bro.org/sphinx-git/scripts/base/protocols/dns/main.bro.html#type-DNS::Info
__ http://www.bro.org/sphinx-git/scripts/base/protocols/conn/main.bro.html#type-Conn::Info
__ http://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info
__ http://www.bro.org/sphinx-git/scripts/base/frameworks/files/main.bro.html#type-Files::Info
__ http://www.bro.org/sphinx-git/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info
.. _using-bro:
@ -251,3 +261,42 @@ stream and Bro is able to extract and track that information for you,
giving you an in-depth and structured view into HTTP traffic on your
network.
-----------------------
Common Log Files
-----------------------
As a monitoring tool, Bro records a detailed view of the traffic inspected and the events generated in
a series of relevant log files. These files can later be reviewed for monitoring, auditing and troubleshooting
purposes.
In this section we present a brief explanation of the most commonly used log files generated by Bro including links
to descriptions of some of the fields for each log type.
+-----------------+---------------------------------------+------------------------------+
| Log File | Description | Field Descriptions |
+=================+=======================================+==============================+
| http.log | Shows all HTTP requests and replies | :bro:type:`HTTP::Info` |
+-----------------+---------------------------------------+------------------------------+
| ftp.log | Records FTP activity | :bro:type:`FTP::Info` |
+-----------------+---------------------------------------+------------------------------+
| ssl.log | Records SSL sessions including | :bro:type:`SSL::Info` |
| | certificates used | |
+-----------------+---------------------------------------+------------------------------+
| known_certs.log | Includes SSL certificates used | :bro:type:`Known::CertsInfo` |
+-----------------+---------------------------------------+------------------------------+
| smtp.log | Summarizes SMTP traffic on a network | :bro:type:`SMTP::Info` |
+-----------------+---------------------------------------+------------------------------+
| dns.log | Shows all DNS activity on a network | :bro:type:`DNS::Info` |
+-----------------+---------------------------------------+------------------------------+
| conn.log | Records all connections seen by Bro | :bro:type:`Conn::Info` |
+-----------------+---------------------------------------+------------------------------+
| dpd.log | Shows network activity on | :bro:type:`DPD::Info` |
| | non-standard ports | |
+-----------------+---------------------------------------+------------------------------+
| files.log | Records information about all files | :bro:type:`Files::Info` |
| | transmitted over the network | |
+-----------------+---------------------------------------+------------------------------+
| weird.log | Records unexpected protocol-level | :bro:type:`Weird::Info` |
| | activity | |
+-----------------+---------------------------------------+------------------------------+

145
doc/mimestats/index.rst Normal file
View file

@ -0,0 +1,145 @@
__ http://www.bro.org/sphinx-git/frameworks/sumstats.html
.. _mime-stats:
====================
MIME Type Statistics
====================
Files are constantly transmitted over HTTP on regular networks. These files belong to a specific category (i.e., executable, text, image, etc.) identified
by a `Multipurpose Internet Mail Extension (MIME) <http://en.wikipedia.org/wiki/MIME>`_. Although MIME was originally developed to identify the type of
non-text attachments on email, it is also used by Web browser to identify the type of files transmitted and present them accordingly.
In this tutorial, we will show how to use the Sumstats Framework to collect some statistics information based on MIME types, specifically the total number of
occurrences, size in bytes, and number of unique hosts transmitting files over HTTP per each type. For instructions about extracting and creating a local copy
of these files, visit `this <../httpmonitor/index.html#inspecting-files>`_ tutorial instead.
------------------------------------------------
MIME Statistics with Sumstats
------------------------------------------------
When working with the `Sumstats`__ framework, you need to define three different pieces: (i) Observations, where
the event is observed and fed into the framework. (ii) Reducers, where observations are collected and measured. (iii) Sumstats, where the main functionality
is implemented.
So, we start by defining our observation along with a record to store all statistics values and an observation interval. We are conducting our observation on
the `HTTP::log_http` event and we are interested in the MIME type, size of the file ("response_body_len") and the originator host ("orig_h"). We use the MIME
type as our key and create observers for the other two values.
.. code:: bro
export {
redef enum Log::ID += { LOG };
type Info: record {
## Timestamp when the log line was finished and written.
ts: time &log;
## Time interval that the log line covers.
ts_delta: interval &log;
## The mime type
mtype: string &log;
## The number of unique local hosts that fetched this mime type
uniq_hosts: count &log;
## The number of hits to the mime type
hits: count &log;
## The total number of bytes received by this mime type
bytes: count &log;
};
## The frequency of logging the stats collected by this script.
const break_interval = 5mins &redef;
}
event HTTP::log_http(rec: HTTP::Info)
{
if(Site::is_local_addr(rec$id$orig_h) && rec?$resp_mime_types) {
local mime_type = rec$resp_mime_types[0];
SumStats::observe("mime.bytes", [$str=mime_type], [$num=rec$response_body_len]);
SumStats::observe("mime.hits", [$str=mime_type], [$str=cat(rec$id$orig_h)]);
}
}
Next, we create the reducers. The first one will accumulate file sizes and the second one will make sure we only store a host ID once. Below is the partial code.
.. code:: bro
local r1: SumStats::Reducer = [$stream="mime.bytes", $apply=set(SumStats::SUM)];
local r2: SumStats::Reducer = [$stream="mime.hits", $apply=set(SumStats::UNIQUE)];
In our final step, we create the SumStats where we check for the observation interval and once it expires, we populate the record (defined above) with all the
relevant data and write it to a log.
.. code:: bro
SumStats::create([$name="mime-metrics",
$epoch=break_interval,
$reducers=set(r1, r2),
$epoch_result(ts: time, key: SumStats::Key, result: SumStats::Result) =
{
local l: Info;
l$ts = network_time();
l$ts_delta = break_interval;
l$mtype = key$str;
l$bytes = double_to_count(floor(result["mime.bytes"]$sum));
l$hits = result["mime.hits"]$num;
l$uniq_hosts = result["mime.hits"]$unique;
Log::write(LOG, l);
}]);
Putting everything together we end up with the following final code for our script.
.. code:: bro
@load base/frameworks/sumstats
module MimeMetrics;
export {
redef enum Log::ID += { LOG };
type Info: record {
## Timestamp when the log line was finished and written.
ts: time &log;
## Time interval that the log line covers.
ts_delta: interval &log;
## The mime type
mtype: string &log;
## The number of unique local hosts that fetched this mime type
uniq_hosts: count &log;
## The number of hits to the mime type
hits: count &log;
## The total number of bytes received by this mime type
bytes: count &log;
};
## The frequency of logging the stats collected by this script.
const break_interval = 5mins &redef;
}
event bro_init() &priority=3
{
Log::create_stream(MimeMetrics::LOG, [$columns=Info]);
local r1: SumStats::Reducer = [$stream="mime.bytes", $apply=set(SumStats::SUM)];
local r2: SumStats::Reducer = [$stream="mime.hits", $apply=set(SumStats::UNIQUE)];
SumStats::create([$name="mime-metrics",
$epoch=break_interval,
$reducers=set(r1, r2),
$epoch_result(ts: time, key: SumStats::Key, result: SumStats::Result) =
{
local l: Info;
l$ts = network_time();
l$ts_delta = break_interval;
l$mtype = key$str;
l$bytes = double_to_count(floor(result["mime.bytes"]$sum));
l$hits = result["mime.hits"]$num;
l$uniq_hosts = result["mime.hits"]$unique;
Log::write(LOG, l);
}]);
}
event HTTP::log_http(rec: HTTP::Info)
{
if(Site::is_local_addr(rec$id$orig_h) && rec?$resp_mime_types) {
local mime_type = rec$resp_mime_types[0];
SumStats::observe("mime.bytes", [$str=mime_type], [$num=rec$response_body_len]);
SumStats::observe("mime.hits", [$str=mime_type], [$str=cat(rec$id$orig_h)]);
}
}