Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-17 05:58:20 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Using redirection with bro-cut. Include initial btests for this document.

Browse source
This commit is contained in:
Scott Runnels 2013-05-07 05:56:45 -04:00
parent 4a7de4a616
commit a93a6535e7
5 changed files with 117 additions and 4 deletions
Download patch file Download diff file Expand all files Collapse all files

51
doc/using/index.rst
View file

@ -3,7 +3,7 @@
Using Bro
=========
Once Bro has been deployed in an environment and monitoring live traffic, it will, in its default configuration, begin to produce human-readable ASCII logs. Each log file, produced by Bro's Logging Framework, is populated with organized, connection-oriented data. As the logfiles are simple ASCII data, working with the data contained in them can be done from a command line terminal once you have been familiarized with the types of data you can find in each log file.
Once Bro has been deployed in an environment and monitoring live traffic, it will, in its default configuration, begin to produce human-readable ASCII logs. Each log file, produced by Bro's Logging Framework, is populated with organized, connection-oriented data. As the logfiles are simple ASCII data, working with the data contained in them can be done from a command line terminal once you have been familiarized with the types of data that can be found in each log file.
----------------------
Structure of Log Files
@ -11,15 +11,58 @@ Structure of Log Files
The log files produced by Bro adhere to a structure as defined by the scripts that produced through which they were produced. However, as each log file has been produced using the Logging Framework, there are similiarites shared by each log file. Without breaking into the scripting aspect of Bro, a bird's eye view of how the log files are produced would progress as follows. The script's author defines the kinds of data, such as the originating IP address or the duration of a connection, which will be used as fields in the log file. The author then decides what behavior should generate a logfile entry, these behaviors can range from a connection having been completed or an HTTP GET method being issued by an orignator. Once these behaviors have been observed, the data is passed to the Logging Framework which, in turn, adds an entry to the appropriate log file. While the fields of the log entries can be modified by the user, the Logging Framework makes use of a header entry in each logfile to ensure that it remains self-describing. This header entry can be see by running the unix utility ``head`` and outputting the first eight lines of the file.
.. btest:: framework_logging_factorial_02
.. btest:: using_bro_cmd_line_01
@TEST-EXEC: btest-rst-cmd head -8 ${TESTBASE}/Baseline/core.pppoe/conn.log
The sample above shows the header for a ``conn.log`` file which gives a detailed account of each connection as seen by Bro. As you can see, header includes information such as what separators are being used for various types of data, what an empty field looks like and what an unset field looks like. In this example, the default TAB separator is being used as the delimiter between fiends (\x09 is the tab character in hex). It also lists the comma as the separator for set data, the string "(empty)" as the indicator for an empty field and the '-' character as the indicator for a field that hasn't been set. The timestampe for when the file was created is included under "#open". The header then goes on to detail the fields being listed in the file and the datatypes of those fields in #fields and #types respectively. These two entries are often the two most significant points of interest as they detail not only the field name but the data type used.
The sample above shows the header for a ``conn.log`` file which gives a detailed account of each connection as seen by Bro. As you can see, header includes information such as what separators are being used for various types of data, what an empty field looks like and what an unset field looks like. In this example, the default TAB separator is being used as the delimiter between fiends (\x09 is the tab character in hex). It also lists the comma as the separator for set data, the string "(empty)" as the indicator for an empty field and the '-' character as the indicator for a field that hasn't been set. The timestampe for when the file was created is included under "#open". The header then goes on to detail the fields being listed in the file and the datatypes of those fields in #fields and #types respectively. These two entries are often the two most significant points of interest as they detail not only the field name but the data type used. Navigating through the different log files produced by Bro, often requires the use of different elements of the unix tool chain such as ``sed``, ``awk``, or ``grep`` and having the field definitions readily available will save the user some mental legwork. The field names are also a key resource for using the ``bro-cut`` utility included with Bro.
-------------
Using bro-cut
-------------
With an understanding of the structure of logfiles, we can start looking at using tools such as ``bro-cut`` to retrieve information from log files.
The ``bro-cut`` utility can be used in place of other tools to build terminal commands that remain flexible and accurate independent of possible changes that can be made to the log file itself. It accomplishes this by parsing the header in each file and allowing the user to refer to the specific columnar data available. In contrast tools like ``awk`` require the user to refer to fields referenced by their position. For example, the two commands listed below produce the same output given a default configuration of Bro.
.. btest:: using_bro_bro_cut_01
@TEST-EXEC: btest-rst-cmd awk \'{print \$3, \$4, \$5, \$6, \$9}\' ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/conn.log
.. btest:: using_bro_bro_cut_02
@TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/conn.log | btest-rst-cmd bro-cut id.orig_h id.orig_p id.resp_h duration
While the output is similar, the advantages to using bro-cut over awk lay in that, while awk is flexible and powerful, ``bro-cut`` was specifically designed to work with log files. Firstly, the ``bro-cut`` output includes only the logfile entries, while the ``awk`` output includes the header parts of the logfile, which would require the user to use a secondary utility to suppress those lones. Secondly, since ``bro-cut`` uses the field descriptors to identify and extract data, it allows for flexibilty independent of the format and contents of the logfile. It's not uncommon for a Bro configuration to add extra fields to various logfiles as required by the environment. In this case, the fields in the ``awk`` command would have to be altered to compensate for the new position whereas the ``bro-cut`` output would not change.
As you may have noticed, the command for ``bro-cut`` uses the output redirection through the ``cat`` command and ``|`` operator. Whereas tools like ``awk`` allow you to indicate the log file as a command line option, bro-cut only takes input through redirection such as ``\`` and ``<``. There are a couple of ways to direct log file data into ``bro-cut``, each dependent upon the type of log file you're processing. A caveat of its use, however, is that the 8 lines of header data must be present. In its default setup, Bro will rotate log files on an hourly basis, moving the current log file into a directory with format ``YYYY-MM-DD`` and gzip compressing the file with a file format that includes the log file type and time range of the file. In the case of processing a compressed log file you simply adjust your command line tools to use the complementary z* versions of commands such as cat (``zcat``), ``grep`` (``zgrep``), and ``head`` (``zhead``).
.......................
Working with timestamps
.......................
The ``bro-cut`` accepts the flag ``-d`` to convert the epoch time values in the log files to human-readable format. The following command includes the human readable time stamp, the unique identifier and the HTTP host and HTTP uri as parsed from the ``http.log`` file.
.. btest:: using_bro_bro_cut_time_01
@TEST-EXEC: btest-rst-cmd bro-cut -d ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log
Often times logfiles from multiple sources are stored in UTC time to allow easy correlation. Converting the timestamp from a logfile to UTC can be accomplished with the ``-u`` command.
.. btest:: using_bro_bro_cut_time_02
@TEST-EXEC: btest-rst-cmd bro-cut -u ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log
The default time format when using the ``-d`` or ``-u`` is the ``strftime`` format string %Y-%m-%dT%H:%M:%S%z which results in a string with year, month, day of month, followed by hour, minutes, seconds and the timezone offset. The default ``strftime`` can be altered by using the ``-D`` and ``-U`` flags. For example, to format the timestamp in the US-typical "Middle Endian" you could use a format string of: %d-%m-%YT%H:%M:%S%z
.. btest:: using_bro_bro_cut_time_03
@TEST-EXEC: btest-rst-cmd bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log
----------------------
Working with Log Files
----------------------
As Bro runs, it deposits its logfiles in

0
testing/btest/Baseline/doc.manual.using_bro_sandbox_01/.stdout Normal file
View file

43
testing/btest/Baseline/doc.manual.using_bro_sandbox_01/conn.log Normal file
View file

@ -0,0 +1,43 @@
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path conn
#open 2013-05-05-20-51-24
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents
#types time string addr port addr port enum string interval count count string bool count string count count count count table[string]
1300475167.096535 UWkUyAuUGXf 141.142.220.202 5353 224.0.0.251 5353 udp dns - - - S0 - 0 D 1 73 0 0 (empty)
1300475167.097012 arKYeMETxOg fe80::217:f2ff:fed7:cf65 5353 ff02::fb 5353 udp - - - - S0 - 0 D 1 199 0 0 (empty)
1300475167.099816 k6kgXLOoSKl 141.142.220.50 5353 224.0.0.251 5353 udp - - - - S0 - 0 D 1 179 0 0 (empty)
1300475168.853899 TEfuqmmG4bh 141.142.220.118 43927 141.142.2.2 53 udp dns 0.000435 38 89 SF - 0 Dd 1 66 1 117 (empty)
1300475168.854378 FrJExwHcSal 141.142.220.118 37676 141.142.2.2 53 udp dns 0.000420 52 99 SF - 0 Dd 1 80 1 127 (empty)
1300475168.854837 5OKnoww6xl4 141.142.220.118 40526 141.142.2.2 53 udp dns 0.000392 38 183 SF - 0 Dd 1 66 1 211 (empty)
1300475168.857956 fRFu0wcOle6 141.142.220.118 32902 141.142.2.2 53 udp dns 0.000317 38 89 SF - 0 Dd 1 66 1 117 (empty)
1300475168.858306 qSsw6ESzHV4 141.142.220.118 59816 141.142.2.2 53 udp dns 0.000343 52 99 SF - 0 Dd 1 80 1 127 (empty)
1300475168.858713 iE6yhOq3SF 141.142.220.118 59714 141.142.2.2 53 udp dns 0.000375 38 183 SF - 0 Dd 1 66 1 211 (empty)
1300475168.891644 qCaWGmzFtM5 141.142.220.118 58206 141.142.2.2 53 udp dns 0.000339 38 89 SF - 0 Dd 1 66 1 117 (empty)
1300475168.892037 70MGiRM1Qf4 141.142.220.118 38911 141.142.2.2 53 udp dns 0.000335 52 99 SF - 0 Dd 1 80 1 127 (empty)
1300475168.892414 h5DsfNtYzi1 141.142.220.118 59746 141.142.2.2 53 udp dns 0.000421 38 183 SF - 0 Dd 1 66 1 211 (empty)
1300475168.893988 c4Zw9TmAE05 141.142.220.118 45000 141.142.2.2 53 udp dns 0.000384 38 89 SF - 0 Dd 1 66 1 117 (empty)
1300475168.894422 EAr0uf4mhq 141.142.220.118 48479 141.142.2.2 53 udp dns 0.000317 52 99 SF - 0 Dd 1 80 1 127 (empty)
1300475168.894787 GvmoxJFXdTa 141.142.220.118 48128 141.142.2.2 53 udp dns 0.000423 38 183 SF - 0 Dd 1 66 1 211 (empty)
1300475168.901749 slFea8xwSmb 141.142.220.118 56056 141.142.2.2 53 udp dns 0.000402 36 131 SF - 0 Dd 1 64 1 159 (empty)
1300475168.902195 UfGkYA2HI2g 141.142.220.118 55092 141.142.2.2 53 udp dns 0.000374 36 198 SF - 0 Dd 1 64 1 226 (empty)
1300475169.899438 BWaU4aSuwkc 141.142.220.44 5353 224.0.0.251 5353 udp dns - - - S0 - 0 D 1 85 0 0 (empty)
1300475170.862384 10XodEwRycf 141.142.220.226 137 141.142.220.255 137 udp dns 2.613017 350 0 S0 - 0 D 7 546 0 0 (empty)
1300475171.675372 zno26fFZkrh fe80::3074:17d5:2052:c324 65373 ff02::1:3 5355 udp dns 0.100096 66 0 S0 - 0 D 2 162 0 0 (empty)
1300475171.677081 v5rgkJBig5l 141.142.220.226 55131 224.0.0.252 5355 udp dns 0.100021 66 0 S0 - 0 D 2 122 0 0 (empty)
1300475173.116749 eWZCH7OONC1 fe80::3074:17d5:2052:c324 54213 ff02::1:3 5355 udp dns 0.099801 66 0 S0 - 0 D 2 162 0 0 (empty)
1300475173.117362 0Pwk3ntf8O3 141.142.220.226 55671 224.0.0.252 5355 udp dns 0.099849 66 0 S0 - 0 D 2 122 0 0 (empty)
1300475173.153679 0HKorjr8Zp7 141.142.220.238 56641 141.142.220.255 137 udp dns - - - S0 - 0 D 1 78 0 0 (empty)
1300475168.859163 GSxOnSLghOa 141.142.220.118 49998 208.80.152.3 80 tcp http 0.215893 1130 734 S1 - 0 ShADad 6 1450 4 950 (empty)
1300475168.652003 nQcgTWjvg4c 141.142.220.118 35634 208.80.152.2 80 tcp - 0.061329 463 350 OTH - 0 DdA 2 567 1 402 (empty)
1300475168.895267 0Q4FH8sESw5 141.142.220.118 50001 208.80.152.3 80 tcp http 0.227284 1178 734 S1 - 0 ShADad 6 1498 4 950 (empty)
1300475168.902635 i2rO3KD1Syg 141.142.220.118 35642 208.80.152.2 80 tcp http 0.120041 534 412 S1 - 0 ShADad 4 750 3 576 (empty)
1300475168.892936 Tw8jXtpTGu6 141.142.220.118 50000 208.80.152.3 80 tcp http 0.229603 1148 734 S1 - 0 ShADad 6 1468 4 950 (empty)
1300475168.855305 3PKsZ2Uye21 141.142.220.118 49996 208.80.152.3 80 tcp http 0.218501 1171 733 S1 - 0 ShADad 6 1491 4 949 (empty)
1300475168.892913 P654jzLoe3a 141.142.220.118 49999 208.80.152.3 80 tcp http 0.220961 1137 733 S1 - 0 ShADad 6 1457 4 949 (empty)
1300475169.780331 2cx26uAvUPl 141.142.220.235 6705 173.192.163.128 80 tcp - - - - OTH - 0 h 0 0 1 48 (empty)
1300475168.724007 j4u32Pc5bif 141.142.220.118 48649 208.80.152.118 80 tcp http 0.119905 525 232 S1 - 0 ShADad 4 741 3 396 (empty)
1300475168.855330 VW0XPVINV8a 141.142.220.118 49997 208.80.152.3 80 tcp http 0.219720 1125 734 S1 - 0 ShADad 6 1445 4 950 (empty)
#close 2013-05-05-20-51-24

23
testing/btest/Baseline/doc.manual.using_bro_sandbox_01/http.log Normal file
View file

@ -0,0 +1,23 @@
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path http
#open 2013-05-05-21-12-40
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p trans_depth method host uri referrer user_agent request_body_len response_body_len status_code status_msg info_code info_msg filename tags username password proxied mime_type md5 extraction_file
#types time string addr port addr port count string string string string string count count count string count string string table[enum] string string table[string] string string file
1300475168.784020 j4u32Pc5bif 141.142.220.118 48649 208.80.152.118 80 1 GET bits.wikimedia.org /skins-1.5/monobook/main.css http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - -
1300475168.916018 VW0XPVINV8a 141.142.220.118 49997 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/6/63/Wikipedia-logo.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - -
1300475168.916183 3PKsZ2Uye21 141.142.220.118 49996 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/thumb/b/bb/Wikipedia_wordmark.svg/174px-Wikipedia_wordmark.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - -
1300475168.918358 GSxOnSLghOa 141.142.220.118 49998 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/b/bd/Bookshelf-40x201_6.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - -
1300475168.952307 Tw8jXtpTGu6 141.142.220.118 50000 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/thumb/8/8a/Wikinews-logo.png/35px-Wikinews-logo.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - -
1300475168.952296 P654jzLoe3a 141.142.220.118 49999 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/4/4a/Wiktionary-logo-en-35px.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - -
1300475168.954820 0Q4FH8sESw5 141.142.220.118 50001 208.80.152.3 80 1 GET upload.wikimedia.org /wikipedia/commons/thumb/f/fa/Wikiquote-logo.svg/35px-Wikiquote-logo.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - -
1300475168.962687 i2rO3KD1Syg 141.142.220.118 35642 208.80.152.2 80 1 GET meta.wikimedia.org /images/wikimedia-button.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - -
1300475168.975934 VW0XPVINV8a 141.142.220.118 49997 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/f/fa/Wikibooks-logo.svg/35px-Wikibooks-logo.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - -
1300475168.976436 3PKsZ2Uye21 141.142.220.118 49996 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/d/df/Wikispecies-logo.svg/35px-Wikispecies-logo.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - -
1300475168.979264 GSxOnSLghOa 141.142.220.118 49998 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/4/4c/Wikisource-logo.svg/35px-Wikisource-logo.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - -
1300475169.014619 Tw8jXtpTGu6 141.142.220.118 50000 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/4/4a/Commons-logo.svg/35px-Commons-logo.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - -
1300475169.014593 P654jzLoe3a 141.142.220.118 49999 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/9/91/Wikiversity-logo.svg/35px-Wikiversity-logo.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - -
1300475169.014927 0Q4FH8sESw5 141.142.220.118 50001 208.80.152.3 80 2 GET upload.wikimedia.org /wikipedia/commons/thumb/7/75/Wikimedia_Community_Logo.svg/35px-Wikimedia_Community_Logo.svg.png http://www.wikipedia.org/ Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.15) Gecko/20110303 Ubuntu/10.04 (lucid) Firefox/3.6.15 0 0 304 Not Modified - - - (empty) - - - - - -
#close 2013-05-05-21-12-40

4
testing/btest/doc/manual/using_bro_sandbox_01 Normal file
View file

@ -0,0 +1,4 @@
# @TEST-EXEC: bro -r ${TRACES}/wikipedia.trace
# @TEST-EXEC: btest-diff conn.log
# @TEST-EXEC: btest-diff http.log