From 399899c49be4e8278885274e5e758080b8ef7a6f Mon Sep 17 00:00:00 2001 From: Robin Sommer Date: Thu, 22 Aug 2013 15:55:44 -0700 Subject: [PATCH] Pass over the Using Bro section. I edited the text little bit, reorganized the structure somewhat and extended some parts. I've also simplified the tests a bit, using some of the BTest tweaks commited in parallel. --- aux/btest | 2 +- doc/frameworks/logging.rst | 2 + doc/using/index.rst | 295 +++++++++++------- .../btest-doc.sphinx.using_bro#1 | 19 ++ .../btest-doc.sphinx.using_bro#2 | 15 + .../btest-doc.sphinx.using_bro#3 | 15 + .../btest-doc.sphinx.using_bro#4 | 10 + .../btest-doc.sphinx.using_bro#5 | 10 + .../btest-doc.sphinx.using_bro#6 | 10 + .../btest-doc.sphinx.using_bro#7 | 9 + .../btest-doc.sphinx.using_bro#8 | 6 + testing/btest/btest.cfg | 5 +- testing/btest/doc/sphinx/test.btest | 3 - testing/btest/doc/sphinx/using_bro.btest | 1 + testing/btest/doc/sphinx/using_bro.btest#2 | 1 + testing/btest/doc/sphinx/using_bro.btest#3 | 1 + testing/btest/doc/sphinx/using_bro.btest#4 | 1 + testing/btest/doc/sphinx/using_bro.btest#5 | 1 + testing/btest/doc/sphinx/using_bro.btest#6 | 1 + testing/btest/doc/sphinx/using_bro.btest#7 | 1 + testing/btest/doc/sphinx/using_bro.btest#8 | 1 + testing/scripts/diff-remove-timestamps | 2 +- testing/scripts/rst-filter | 5 + 23 files changed, 296 insertions(+), 120 deletions(-) create mode 100644 testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#1 create mode 100644 testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#2 create mode 100644 testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#3 create mode 100644 testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#4 create mode 100644 testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#5 create mode 100644 testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#6 create mode 100644 testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#7 create mode 100644 testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#8 delete mode 100644 testing/btest/doc/sphinx/test.btest create mode 100644 testing/btest/doc/sphinx/using_bro.btest create mode 100644 testing/btest/doc/sphinx/using_bro.btest#2 create mode 100644 testing/btest/doc/sphinx/using_bro.btest#3 create mode 100644 testing/btest/doc/sphinx/using_bro.btest#4 create mode 100644 testing/btest/doc/sphinx/using_bro.btest#5 create mode 100644 testing/btest/doc/sphinx/using_bro.btest#6 create mode 100644 testing/btest/doc/sphinx/using_bro.btest#7 create mode 100644 testing/btest/doc/sphinx/using_bro.btest#8 create mode 100755 testing/scripts/rst-filter diff --git a/aux/btest b/aux/btest index 69606f8f3c..063b055621 160000 --- a/aux/btest +++ b/aux/btest @@ -1 +1 @@ -Subproject commit 69606f8f3cc84d694ca1da14868a5fecd4abbc96 +Subproject commit 063b05562172c9cae160cf8df899afe366d1b108 diff --git a/doc/frameworks/logging.rst b/doc/frameworks/logging.rst index 06935647d3..9c8df62043 100644 --- a/doc/frameworks/logging.rst +++ b/doc/frameworks/logging.rst @@ -1,4 +1,6 @@ +.. _framework-logging: + ================= Logging Framework ================= diff --git a/doc/using/index.rst b/doc/using/index.rst index df2b48bf48..c9226fadd2 100644 --- a/doc/using/index.rst +++ b/doc/using/index.rst @@ -5,97 +5,155 @@ Using Bro ========= +.. contents:: + Once Bro has been deployed in an environment and monitoring live traffic, it will, in its default configuration, begin to produce -human-readable ASCII logs. Each log file, produced by Bro's Logging -Framework, is populated with organized, connection-oriented data. As -the log files are simple ASCII data, working with the data contained -in them can be done from a command line terminal once you have been -familiarized with the types of data that can be found in each log -file. +human-readable ASCII logs. Each log file, produced by Bro's +:ref:`framework-logging`, is populated with organized, mostly +connection-oriented data. As the standard log files are simple ASCII +data, working with the data contained in them can be done from a +command line terminal once you have been familiarized with the types +of data that can be found in each file. In the following, we work +through the logs general structure and then examine some standard ways +of working with them. ---------------------- -Structure of Log Files +Working with Log Files ---------------------- -The log files produced by Bro adhere to a structure as defined by the -scripts that produced through which they were produced. However, as -each log file has been produced using the Logging Framework, there are -similarities shared by each log file. Without breaking into the -scripting aspect of Bro, a bird's eye view of how the log files are -produced would progress as follows. The script's author defines the -kinds of data, such as the originating IP address or the duration of a -connection, which will be used as fields in the log file. The author -then decides what behavior should generate a log file entry, these -behaviors can range from a connection having been completed or an HTTP -GET method being issued by an originator. Once these behaviors have -been observed, the data is passed to the Logging Framework which, in -turn, adds an entry to the appropriate log file. While the fields of -the log entries can be modified by the user, the Logging Framework -makes use of a header entry in each log file to ensure that it remains -self-describing. This header entry can be see by running the unix -utility ``head`` and outputting the first eight lines of the file. +Generally, all of Bro's log files are produced by a corresponding +script that defines their individual structure. However, as each log +file flows through the Logging Framework, there share a set of +structural similarities. Without breaking into the scripting aspect of +Bro here, a bird's eye view of how the log files are produced would +progress as follows. The script's author defines the kinds of data, +such as the originating IP address or the duration of a connection, +which will make up the fields (i.e., columns) of the log file. The +author then decides what network activity should generate a single log +file entry (i.e., one line); that could, e.g., be a connection having +been completed or an HTTP ``GET`` method being issued by an +originator. When these behaviors are observed during operation, the +data is passed to the Logging Framework which, in turn, adds the entry +to the appropriate log file. -.. btest:: using_bro_cmd_line_01 +As the fields of the log entries can be further customized by the +user, the Logging Framework makes use of a header block to ensure that +it remains self-describing. This header entry can be see by running +the Unix utility ``head`` and outputting the first lines of the file: + +.. btest:: using_bro - @TEST-EXEC: btest-rst-cmd head -8 ${TESTBASE}/Baseline/core.pppoe/conn.log + @TEST-EXEC: btest-rst-cmd "bro -r $TRACES/wikipedia.trace && head -15 conn.log" -The sample above shows the header for a ``conn.log`` file which gives -a detailed account of each connection as seen by Bro. As you can see, -header includes information such as what separators are being used for +As you can see, the header consists of lines prefixed by ``#`` and +includes information such as what separators are being used for various types of data, what an empty field looks like and what an unset field looks like. In this example, the default TAB separator is -being used as the delimiter between fiends (\x09 is the tab character -in hex). It also lists the comma as the separator for set data, the -string "(empty)" as the indicator for an empty field and the '-' -character as the indicator for a field that hasn't been set. The -timestamp for when the file was created is included under "#open". -The header then goes on to detail the fields being listed in the file -and the data types of those fields in #fields and #types respectively. -These two entries are often the two most significant points of -interest as they detail not only the field name but the data type -used. Navigating through the different log files produced by Bro, -often requires the use of different elements of the unix tool chain -such as ``sed``, ``awk``, or ``grep`` and having the field definitions -readily available will save the user some mental leg work. The field -names are also a key resource for using the ``bro-cut`` utility -included with Bro. +being used as the delimiter between fields (``\x09`` is the tab +character in hex). It also lists the comma as the separator for set +data, the string ``(empty)`` as the indicator for an empty field and +the ``-`` character as the indicator for a field that hasn't been set. +The timestamp for when the file was created is included under +``#open``. The header then goes on to detail the fields being listed +in the file and the data types of those fields in ``#fields`` and +``#types``, respectively. These two entries are often the two most +significant points of interest as they detail not only the field names +but the data types used. When navigating through the different log +files with tools like ``sed``, ``awk``, or ``grep``, having the field +definitions readily available saves the user some mental leg work. The +field names are also a key resource for using the :ref:`bro-cut +` utility included with Bro, see below. -------------- -Using bro-cut -------------- +Next to the header follows the main content; in this example we see 7 +connections with their key properties, such as originator and +responder IP addresses (note how Bro transparely handles both IPv4 and +IPv6), transport-layer ports, application-layer services - the +``service`` field is filled ias Bro determines a specific protocol to +be in use, independent of the connection's ports - payload size, and +more. See :bro:type:`Conn::Info` for a description of all fields. + +In addition to ``conn.log``, Bro generates many further logs by +default, including: + +``dpd.log`` + A summary of protocols encountered on non-standard ports. + +``dns.log`` + All DNS activity. + +``ftp.log`` + A log of FTP session-level activity. + +``files.log`` + Summaries of files transfered over the network. This information + is aggregrated from different protocols, including HTTP, FTP, and + SMTP. + +``http.log`` + A summary of all HTTP requests with their replies. + +``known_certs.log`` + SSL certificates seen in use. + +``smtp.log`` + A summary of SMTP activity. + +``ssl.log`` + A record of SSL sessions, including certificates being used. + +``weird.log`` + A log of unexpected protocol-level activity. Whenever Bro's + protocol analysis encounters a situation it would not expect + (e.g., an RFC violation) is logs it in this file. Note that in + practice, real-world networks tend to exhibit a large number of + such "crud" that is usually not worth following up on. + +As you can see, some log files are specific to a particular protocol, +while others aggregate information across different types of activity. + +.. _bro-cut: + +Using ``bro-cut`` +----------------- The ``bro-cut`` utility can be used in place of other tools to build terminal commands that remain flexible and accurate independent of -possible changes that can be made to the log file itself. It -accomplishes this by parsing the header in each file and allowing the -user to refer to the specific columnar data available. In contrast -tools like ``awk`` require the user to refer to fields referenced by -their position. For example, the two commands listed below produce -the same output given a default configuration of Bro. +possible changes to log file itself. It accomplishes this by parsing +the header in each file and allowing the user to refer to the specific +columnar data available (in contrast to tools like ``awk`` that +require the user to refer to fields referenced by their position). +For example, the following command extracts just the given columns +from a ``conn.log``: -.. btest:: using_bro_bro_cut_01 +.. btest:: using_bro - @TEST-EXEC: btest-rst-cmd awk \'{print \$3, \$4, \$5, \$6, \$9}\' ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/conn.log + @TEST-EXEC: btest-rst-cmd -n 10 "cat conn.log | bro-cut id.orig_h id.orig_p id.resp_h duration" -.. btest:: using_bro_bro_cut_02 +The correspding ``awk`` command would look like this: - @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/conn.log | btest-rst-cmd -c "cat conn.log | bro-cut id.orig_h id.orig_p id.resp_h duration " bro-cut id.orig_h id.orig_p id.resp_h duration +.. btest:: using_bro + @TEST-EXEC: btest-rst-cmd -n 10 awk \'/^[^#]/ {print \$3, \$4, \$5, \$6, \$9}\' conn.log -While the output is similar, the advantages to using bro-cut over awk -lay in that, while awk is flexible and powerful, ``bro-cut`` was -specifically designed to work with log files. Firstly, the +While the output is similar, the advantages to using bro-cut over +``awk`` lay in that, while ``awk`` is flexible and powerful, ``bro-cut`` +was specifically designed to work with Bro's log files. Firstly, the ``bro-cut`` output includes only the log file entries, while the -``awk`` output includes the header parts of the log file, which would -require the user to use a secondary utility to suppress those lines. -Secondly, since ``bro-cut`` uses the field descriptors to identify and -extract data, it allows for flexibility independent of the format and -contents of the log file. It's not uncommon for a Bro configuration -to add extra fields to various log files as required by the -environment. In this case, the fields in the ``awk`` command would -have to be altered to compensate for the new position whereas the -``bro-cut`` output would not change. +``awk`` solution needs to skip the header manually. Secondly, since +``bro-cut`` uses the field descriptors to identify and extract data, +it allows for flexibility independent of the format and contents of +the log file. It's not uncommon for a Bro configuration to add extra +fields to various log files as required by the environment. In this +case, the fields in the ``awk`` command would have to be altered to +compensate for the new position whereas the ``bro-cut`` output would +not change. + +.. note:: + + The sequence of field names given to ``bro-cut`` determines the + output order, which means you can also use ``bro-cut`` to reorder + fields. That can be helpful when piping into, e.g., ``sort``. As you may have noticed, the command for ``bro-cut`` uses the output redirection through the ``cat`` command and ``|`` operator. Whereas @@ -104,51 +162,60 @@ line option, bro-cut only takes input through redirection such as ``|`` and ``<``. There are a couple of ways to direct log file data into ``bro-cut``, each dependent upon the type of log file you're processing. A caveat of its use, however, is that the 8 lines of -header data must be present. In its default setup, Bro will rotate -log files on an hourly basis, moving the current log file into a -directory with format ``YYYY-MM-DD`` and gzip compressing the file -with a file format that includes the log file type and time range of -the file. In the case of processing a compressed log file you simply -adjust your command line tools to use the complementary z* versions of -commands such as cat (``zcat``), ``grep`` (``zgrep``), and ``head`` -(``zhead``). +header data must be present. -....................... -Working with timestamps -....................... +.. note:: -The ``bro-cut`` accepts the flag ``-d`` to convert the epoch time -values in the log files to human-readable format. The following -command includes the human readable time stamp, the unique identifier -and the HTTP host and HTTP uri as parsed from the ``http.log`` file. + ``bro-cut`` provides an option ``-c`` to include a corresponding + format header into the output, which allows to chain multiple + ``bro-cut`` instances or perform further post-processing that + evaluates the header information. -.. btest:: using_bro_bro_cut_time_01 +In its default setup, Bro will rotate log files on an hourly basis, +moving the current log file into a directory with format +``YYYY-MM-DD`` and gzip compressing the file with a file format that +includes the log file type and time range of the file. In the case of +processing a compressed log file you simply adjust your command line +tools to use the complementary ``z*`` versions of commands such as cat +(``zcat``), ``grep`` (``zgrep``), and ``head`` (``zhead``). - @TEST-EXEC: btest-rst-cmd -c "bro-cut -d ts uid host uri < http.log" bro-cut -d ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log +Working with Timestamps +----------------------- + +``bro-cut`` accepts the flag ``-d`` to convert the epoch time values +in the log files to human-readable format. The following command +includes the human readable time stamp, the unique identifier and the +HTTP ``Host`` and HTTP ``URI`` as extracted from the ``http.log`` +file: + +.. btest:: using_bro + + @TEST-EXEC: btest-rst-cmd -n 5 "bro-cut -d ts uid host uri < http.log" Often times log files from multiple sources are stored in UTC time to allow easy correlation. Converting the timestamp from a log file to -UTC can be accomplished with the ``-u`` command. +UTC can be accomplished with the ``-u`` option: -.. btest:: using_bro_bro_cut_time_02 +.. btest:: using_bro - @TEST-EXEC: btest-rst-cmd -c "bro-cut -u ts uid host uri < http.log" bro-cut -u ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log + @TEST-EXEC: btest-rst-cmd -n 5 "bro-cut -u ts uid host uri < http.log" The default time format when using the ``-d`` or ``-u`` is the -``strftime`` format string %Y-%m-%dT%H:%M:%S%z which results in a +``strftime`` format string ``%Y-%m-%dT%H:%M:%S%z`` which results in a string with year, month, day of month, followed by hour, minutes, -seconds and the timezone offset. The default ``strftime`` can be -altered by using the ``-D`` and ``-U`` flags. For example, to format -the timestamp in the US-typical "Middle Endian" you could use a format -string of: %d-%m-%YT%H:%M:%S%z +seconds and the timezone offset. The default format can be altered by +using the ``-D`` and ``-U`` flags, using the standard ``strftime`` +syntax. For example, to format the timestamp in the US-typical "Middle +Endian" you could use a format string of: ``%d-%m-%YT%H:%M:%S%z`` -.. btest:: using_bro_bro_cut_time_03 +.. btest:: using_bro - @TEST-EXEC: btest-rst-cmd -c "bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < http.log" bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log + @TEST-EXEC: btest-rst-cmd -n 5 "bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < http.log" ----------------------- -Working with Log Files ----------------------- +See ``man strfime`` for more options for the format string. + +Using UIDs +---------- While Bro can do signature based analysis, its primary focus is on behavioral detection which alters the practice of log review from @@ -156,8 +223,8 @@ behavioral detection which alters the practice of log review from trip. A common progression of review includes correlating a session across multiple log files. As a connection is processed by Bro, a unique identifier is assigned to each session. This unique identifier -is almost always included in any log file entry specific to that -connection and can be used to cross-reference log files. +is generally included in any log file entry associated with that +connection and can be used to cross-reference different log files. A simple example would be to cross-reference a UID seen in a ``conn.log`` file. Here, we're looking for the connection with the @@ -165,19 +232,21 @@ largest number of bytes from the responder by redirecting the output for ``cat conn.log`` into bro-cut to extract the UID and the resp_bytes, then sorting that output by the resp_bytes field. -.. btest:: using_bro_practical_02 +.. btest:: using_bro - @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_02/conn.log | bro-cut uid resp_bytes | btest-rst-cmd -c "cat conn.log | bro-cut uid resp_bytes | btest-rst-cmd sort -nrk2" sort -nrk2 - -With the UID of the largest response, it can be crossreferenced with -the UIDs in the ``http.log`` file. + @TEST-EXEC: btest-rst-cmd "cat conn.log | bro-cut uid resp_bytes | sort -nrk2 | head -5" -.. btest:: using_bro_practical_03 +Taking the UID of the first of the top responses, we can now +crossreference that with the UIDs in the ``http.log`` file. - @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_02/http.log | bro-cut uid id.resp_h method status_code host uri | btest-rst-cmd -c "cat http.log | bro-cut uid id.resp_h method status_code host uri | grep j4u32Pc5bif" grep j4u32Pc5bif +.. btest:: using_bro + + @TEST-EXEC: btest-rst-cmd "cat http.log | bro-cut uid id.resp_h method status_code host uri | grep VW0XPVINV8a" + +As you can see there are two HTTP ``GET`` requests within the +session that Bro identified and logged. Given that HTTP is a stream +protocol, it can have multiple ``GET``/``POST``/etc requests in a +stream and Bro is able to extract and track that information for you, +giving you an in-depth and structured view into HTTP traffic on your +network. -As you can see there are multiple HTTP GET requests within the session -that Bro identified and logged. Given that HTTP is a stream protocol, -it can have multiple GET/POST/etc requests in a stream and Bro is able -to extract and track that information for you, giving you an in-depth -and structured view into HTTP traffic on your network. diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#1 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#1 new file mode 100644 index 0000000000..1512c0c8ef --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#1 @@ -0,0 +1,19 @@ +.. code-block:: none + + # bro -r wikipedia.trace && head -15 conn.log + #separator \x09 + #set_separator , + #empty_field (empty) + #unset_field - + #path conn + #open 2013-08-22-22-52-46 + #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents + #types time string addr port addr port enum string interval count count string bool count string count count count count table[string] + 1300475167.096535 UWkUyAuUGXf 141.142.220.202 5353 224.0.0.251 5353 udp dns - - - S0 - 0 D 1 73 0 0 (empty) + 1300475167.097012 arKYeMETxOg fe80::217:f2ff:fed7:cf65 5353 ff02::fb 5353 udp - - - - S0 - 0 D 1 199 0 0 (empty) + 1300475167.099816 k6kgXLOoSKl 141.142.220.50 5353 224.0.0.251 5353 udp - - - - S0 - 0 D 1 179 0 0 (empty) + 1300475168.853899 TEfuqmmG4bh 141.142.220.118 43927 141.142.2.2 53 udp dns 0.000435 38 89 SF - 0 Dd 1 66 1 117 (empty) + 1300475168.854378 FrJExwHcSal 141.142.220.118 37676 141.142.2.2 53 udp dns 0.000420 52 99 SF - 0 Dd 1 80 1 127 (empty) + 1300475168.854837 5OKnoww6xl4 141.142.220.118 40526 141.142.2.2 53 udp dns 0.000392 38 183 SF - 0 Dd 1 66 1 211 (empty) + 1300475168.857956 fRFu0wcOle6 141.142.220.118 32902 141.142.2.2 53 udp dns 0.000317 38 89 SF - 0 Dd 1 66 1 117 (empty) + diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#2 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#2 new file mode 100644 index 0000000000..13ddfbb931 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#2 @@ -0,0 +1,15 @@ +.. code-block:: none + + # cat conn.log | bro-cut id.orig_h id.orig_p id.resp_h duration + 141.142.220.202 5353 224.0.0.251 - + fe80::217:f2ff:fed7:cf65 5353 ff02::fb - + 141.142.220.50 5353 224.0.0.251 - + 141.142.220.118 43927 141.142.2.2 0.000435 + 141.142.220.118 37676 141.142.2.2 0.000420 + 141.142.220.118 40526 141.142.2.2 0.000392 + 141.142.220.118 32902 141.142.2.2 0.000317 + 141.142.220.118 59816 141.142.2.2 0.000343 + 141.142.220.118 59714 141.142.2.2 0.000375 + 141.142.220.118 58206 141.142.2.2 0.000339 + [...] + diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#3 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#3 new file mode 100644 index 0000000000..8a35312c44 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#3 @@ -0,0 +1,15 @@ +.. code-block:: none + + # awk '/^[^#]/ {print $3, $4, $5, $6, $9}' conn.log + 141.142.220.202 5353 224.0.0.251 5353 - + fe80::217:f2ff:fed7:cf65 5353 ff02::fb 5353 - + 141.142.220.50 5353 224.0.0.251 5353 - + 141.142.220.118 43927 141.142.2.2 53 0.000435 + 141.142.220.118 37676 141.142.2.2 53 0.000420 + 141.142.220.118 40526 141.142.2.2 53 0.000392 + 141.142.220.118 32902 141.142.2.2 53 0.000317 + 141.142.220.118 59816 141.142.2.2 53 0.000343 + 141.142.220.118 59714 141.142.2.2 53 0.000375 + 141.142.220.118 58206 141.142.2.2 53 0.000339 + [...] + diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#4 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#4 new file mode 100644 index 0000000000..f171bab392 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#4 @@ -0,0 +1,10 @@ +.. code-block:: none + + # bro-cut -d ts uid host uri < http.log + 2011-03-18T19:06:08+0000 j4u32Pc5bif bits.wikimedia.org /skins-1.5/monobook/main.css + 2011-03-18T19:06:08+0000 VW0XPVINV8a upload.wikimedia.org /wikipedia/commons/6/63/Wikipedia-logo.png + 2011-03-18T19:06:08+0000 3PKsZ2Uye21 upload.wikimedia.org /wikipedia/commons/thumb/b/bb/Wikipedia_wordmark.svg/174px-Wikipedia_wordmark.svg.png + 2011-03-18T19:06:08+0000 GSxOnSLghOa upload.wikimedia.org /wikipedia/commons/b/bd/Bookshelf-40x201_6.png + 2011-03-18T19:06:08+0000 Tw8jXtpTGu6 upload.wikimedia.org /wikipedia/commons/thumb/8/8a/Wikinews-logo.png/35px-Wikinews-logo.png + [...] + diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#5 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#5 new file mode 100644 index 0000000000..491930cbc1 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#5 @@ -0,0 +1,10 @@ +.. code-block:: none + + # bro-cut -u ts uid host uri < http.log + 2011-03-18T19:06:08+0000 j4u32Pc5bif bits.wikimedia.org /skins-1.5/monobook/main.css + 2011-03-18T19:06:08+0000 VW0XPVINV8a upload.wikimedia.org /wikipedia/commons/6/63/Wikipedia-logo.png + 2011-03-18T19:06:08+0000 3PKsZ2Uye21 upload.wikimedia.org /wikipedia/commons/thumb/b/bb/Wikipedia_wordmark.svg/174px-Wikipedia_wordmark.svg.png + 2011-03-18T19:06:08+0000 GSxOnSLghOa upload.wikimedia.org /wikipedia/commons/b/bd/Bookshelf-40x201_6.png + 2011-03-18T19:06:08+0000 Tw8jXtpTGu6 upload.wikimedia.org /wikipedia/commons/thumb/8/8a/Wikinews-logo.png/35px-Wikinews-logo.png + [...] + diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#6 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#6 new file mode 100644 index 0000000000..a2b2316257 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#6 @@ -0,0 +1,10 @@ +.. code-block:: none + + # bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < http.log + 18-03-2011T19:06:08+0000 j4u32Pc5bif bits.wikimedia.org /skins-1.5/monobook/main.css + 18-03-2011T19:06:08+0000 VW0XPVINV8a upload.wikimedia.org /wikipedia/commons/6/63/Wikipedia-logo.png + 18-03-2011T19:06:08+0000 3PKsZ2Uye21 upload.wikimedia.org /wikipedia/commons/thumb/b/bb/Wikipedia_wordmark.svg/174px-Wikipedia_wordmark.svg.png + 18-03-2011T19:06:08+0000 GSxOnSLghOa upload.wikimedia.org /wikipedia/commons/b/bd/Bookshelf-40x201_6.png + 18-03-2011T19:06:08+0000 Tw8jXtpTGu6 upload.wikimedia.org /wikipedia/commons/thumb/8/8a/Wikinews-logo.png/35px-Wikinews-logo.png + [...] + diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#7 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#7 new file mode 100644 index 0000000000..bdbf0e20a0 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#7 @@ -0,0 +1,9 @@ +.. code-block:: none + + # cat conn.log | bro-cut uid resp_bytes | sort -nrk2 | head -5 + VW0XPVINV8a 734 + Tw8jXtpTGu6 734 + GSxOnSLghOa 734 + 0Q4FH8sESw5 734 + P654jzLoe3a 733 + diff --git a/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#8 b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#8 new file mode 100644 index 0000000000..67d0417de8 --- /dev/null +++ b/testing/btest/Baseline/doc.sphinx.using_bro/btest-doc.sphinx.using_bro#8 @@ -0,0 +1,6 @@ +.. code-block:: none + + # cat http.log | bro-cut uid id.resp_h method status_code host uri | grep VW0XPVINV8a + VW0XPVINV8a 208.80.152.3 GET 304 upload.wikimedia.org /wikipedia/commons/6/63/Wikipedia-logo.png + VW0XPVINV8a 208.80.152.3 GET 304 upload.wikimedia.org /wikipedia/commons/thumb/f/fa/Wikibooks-logo.svg/35px-Wikibooks-logo.svg.png + diff --git a/testing/btest/btest.cfg b/testing/btest/btest.cfg index 3bfdea79d8..da0487f5ef 100644 --- a/testing/btest/btest.cfg +++ b/testing/btest/btest.cfg @@ -4,7 +4,7 @@ TmpDir = %(testbase)s/.tmp BaselineDir = %(testbase)s/Baseline IgnoreDirs = .svn CVS .tmp IgnoreFiles = *.tmp *.swp #* *.trace .DS_Store -Finalizer = btest-diff-rst +PartFinalizer = btest-diff-rst [environment] BROPATH=`bash -c %(testbase)s/../../build/bro-path-dev` @@ -18,6 +18,7 @@ TRACES=%(testbase)s/Traces SCRIPTS=%(testbase)s/../scripts DIST=%(testbase)s/../.. BUILD=%(testbase)s/../../build -TEST_DIFF_CANONIFIER=$SCRIPTS/diff-canonifier +TEST_DIFF_CANONIFIER=%(testbase)s/../scripts/diff-canonifier TMPDIR=%(testbase)s/.tmp BRO_PROFILER_FILE=%(testbase)s/.tmp/script-coverage.XXXXXX +BTEST_RST_FILTER=$SCRIPTS/rst-filter diff --git a/testing/btest/doc/sphinx/test.btest b/testing/btest/doc/sphinx/test.btest deleted file mode 100644 index f2f1bcff53..0000000000 --- a/testing/btest/doc/sphinx/test.btest +++ /dev/null @@ -1,3 +0,0 @@ -@TEST-COPY-FILE: ${TRACES}/wikipedia.trace -@TEST-EXEC: btest-rst-cmd bro -r wikipedia.trace -@TEST-EXEC: btest-rst-cmd "cat http.log | bro-cut ts id.orig_h | head -5" diff --git a/testing/btest/doc/sphinx/using_bro.btest b/testing/btest/doc/sphinx/using_bro.btest new file mode 100644 index 0000000000..3551104043 --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd "bro -r $TRACES/wikipedia.trace && head -15 conn.log" diff --git a/testing/btest/doc/sphinx/using_bro.btest#2 b/testing/btest/doc/sphinx/using_bro.btest#2 new file mode 100644 index 0000000000..afa29e6184 --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest#2 @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd -n 10 "cat conn.log | bro-cut id.orig_h id.orig_p id.resp_h duration" diff --git a/testing/btest/doc/sphinx/using_bro.btest#3 b/testing/btest/doc/sphinx/using_bro.btest#3 new file mode 100644 index 0000000000..a3ce44357e --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest#3 @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd -n 10 awk \'/^[^#]/ {print \$3, \$4, \$5, \$6, \$9}\' conn.log diff --git a/testing/btest/doc/sphinx/using_bro.btest#4 b/testing/btest/doc/sphinx/using_bro.btest#4 new file mode 100644 index 0000000000..ce10bba56a --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest#4 @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd -n 5 "bro-cut -d ts uid host uri < http.log" diff --git a/testing/btest/doc/sphinx/using_bro.btest#5 b/testing/btest/doc/sphinx/using_bro.btest#5 new file mode 100644 index 0000000000..786aebffab --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest#5 @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd -n 5 "bro-cut -u ts uid host uri < http.log" diff --git a/testing/btest/doc/sphinx/using_bro.btest#6 b/testing/btest/doc/sphinx/using_bro.btest#6 new file mode 100644 index 0000000000..cc52531b9c --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest#6 @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd -n 5 "bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < http.log" diff --git a/testing/btest/doc/sphinx/using_bro.btest#7 b/testing/btest/doc/sphinx/using_bro.btest#7 new file mode 100644 index 0000000000..4662f67c88 --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest#7 @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd "cat conn.log | bro-cut uid resp_bytes | sort -nrk2 | head -5" diff --git a/testing/btest/doc/sphinx/using_bro.btest#8 b/testing/btest/doc/sphinx/using_bro.btest#8 new file mode 100644 index 0000000000..dfb7b3a715 --- /dev/null +++ b/testing/btest/doc/sphinx/using_bro.btest#8 @@ -0,0 +1 @@ +@TEST-EXEC: btest-rst-cmd "cat http.log | bro-cut uid id.resp_h method status_code host uri | grep VW0XPVINV8a" diff --git a/testing/scripts/diff-remove-timestamps b/testing/scripts/diff-remove-timestamps index 138b901743..44422f6f55 100755 --- a/testing/scripts/diff-remove-timestamps +++ b/testing/scripts/diff-remove-timestamps @@ -11,4 +11,4 @@ fi # The first sed uses a "basic" regexp, the 2nd a "modern:. sed 's/[0-9]\{10\}\.[0-9]\{2,8\}/XXXXXXXXXX.XXXXXX/g' | \ -$sed 's/^#(open|close).(19|20)..-..-..-..-..-..$/#\1 XXXX-XX-XX-XX-XX-XX/g' +$sed 's/^ *#(open|close).(19|20)..-..-..-..-..-..$/#\1 XXXX-XX-XX-XX-XX-XX/g' diff --git a/testing/scripts/rst-filter b/testing/scripts/rst-filter new file mode 100755 index 0000000000..b06d5efd5e --- /dev/null +++ b/testing/scripts/rst-filter @@ -0,0 +1,5 @@ +# /usr/bin/env bash +# +# Filters the output of btest-rst-cmd. + +sed "s#${TRACES}/\{0,1\}##g"