diff --git a/doc/using/index.rst b/doc/using/index.rst index a6d42ed211..fc979dd30a 100644 --- a/doc/using/index.rst +++ b/doc/using/index.rst @@ -29,7 +29,7 @@ The ``bro-cut`` utility can be used in place of other tools to build terminal co .. btest:: using_bro_bro_cut_02 - @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/conn.log | btest-rst-cmd bro-cut id.orig_h id.orig_p id.resp_h duration + @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/conn.log | btest-rst-cmd -c "cat conn.log | bro-cut id.orig_h id.orig_p id.resp_h duration " bro-cut id.orig_h id.orig_p id.resp_h duration While the output is similar, the advantages to using bro-cut over awk lay in that, while awk is flexible and powerful, ``bro-cut`` was specifically designed to work with log files. Firstly, the ``bro-cut`` output includes only the log file entries, while the ``awk`` output includes the header parts of the log file, which would require the user to use a secondary utility to suppress those lines. Secondly, since ``bro-cut`` uses the field descriptors to identify and extract data, it allows for flexibility independent of the format and contents of the log file. It's not uncommon for a Bro configuration to add extra fields to various log files as required by the environment. In this case, the fields in the ``awk`` command would have to be altered to compensate for the new position whereas the ``bro-cut`` output would not change. @@ -44,25 +44,36 @@ The ``bro-cut`` accepts the flag ``-d`` to convert the epoch time values in the .. btest:: using_bro_bro_cut_time_01 - @TEST-EXEC: btest-rst-cmd bro-cut -d ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log + @TEST-EXEC: btest-rst-cmd -c "bro-cut -d ts uid host uri < http.log" bro-cut -d ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log Often times log files from multiple sources are stored in UTC time to allow easy correlation. Converting the timestamp from a log file to UTC can be accomplished with the ``-u`` command. .. btest:: using_bro_bro_cut_time_02 - @TEST-EXEC: btest-rst-cmd bro-cut -u ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log + @TEST-EXEC: btest-rst-cmd -c "bro-cut -u ts uid host uri < http.log" bro-cut -u ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log The default time format when using the ``-d`` or ``-u`` is the ``strftime`` format string %Y-%m-%dT%H:%M:%S%z which results in a string with year, month, day of month, followed by hour, minutes, seconds and the timezone offset. The default ``strftime`` can be altered by using the ``-D`` and ``-U`` flags. For example, to format the timestamp in the US-typical "Middle Endian" you could use a format string of: %d-%m-%YT%H:%M:%S%z .. btest:: using_bro_bro_cut_time_03 - @TEST-EXEC: btest-rst-cmd bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log + @TEST-EXEC: btest-rst-cmd -c "bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < http.log" bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log ---------------------- Working with Log Files ---------------------- -As Bro runs, it deposits its log files in +While Bro can do signature based analysis, its primary focus is on behavioral detection which alters the practice of log review from "reactionary review" to a process a little more akin to a hunting trip. A common progression of review includes correlating a session across multiple log files. As a connection is processed by Bro, a unique identifier is assigned to each session. This unique identifier is almost always included in any log file entry specific to that connection and can be used to cross-reference log files. +A simple example would be to cross-reference a UID seen in a ``conn.log`` file. Here, we're looking for the connection with the largest number of bytes from the responder by redirecting the output for ``cat conn.log`` into bro-cut to extract the UID and the resp_bytes, then sorting that output by the resp_bytes field. +.. btest:: using_bro_practical_02 + @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_02/conn.log | bro-cut uid resp_bytes | btest-rst-cmd -c "cat conn.log | bro-cut uid resp_bytes | btest-rst-cmd sort -nrk2" sort -nrk2 + +With the UID of the largest response, it can be crossreferenced with the UIDs in the ``http.log`` file. + +.. btest:: using_bro_practical_03 + + @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_02/http.log | bro-cut uid id.resp_h method status_code host uri | btest-rst-cmd -c "cat http.log | bro-cut uid id.resp_h method status_code host uri | grep j4u32Pc5bif" grep j4u32Pc5bif + +As you can see there are multiple HTTP GET requests within the session that Bro identified and logged. Given that HTTP is a stream protocol, it can have multiple GET/POST/etc requests in a stream and Bro is able to extract and track that information for you, giving you an in-depth and structured view into HTTP traffic on your network. diff --git a/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/conn.log b/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/conn.log new file mode 100644 index 0000000000..1227e60ad3 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/conn.log @@ -0,0 +1,15 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path conn +#open 2013-05-07-14-38-27 +#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents +#types time string addr port addr port enum string interval count count string bool count string count count count count table[string] +1320329757.771503 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 tcp http 15.161537 2899 1127 S2 - 0 ShADadF 20 3719 19 1891 (empty) +1320329757.771262 nQcgTWjvg4c 10.0.2.15 49285 192.150.187.43 80 tcp http 15.161772 889 377 S2 - 0 ShADadF 8 1229 8 701 (empty) +1320329757.761327 arKYeMETxOg 10.0.2.15 49283 192.150.187.43 80 tcp http 15.168898 459 189 S2 - 0 ShADadF 5 679 4 353 (empty) +1320329757.458867 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 tcp http 15.471378 1824 751 S2 - 0 ShADadF 12 2324 13 1275 (empty) +1320329757.761638 k6kgXLOoSKl 10.0.2.15 49284 192.150.187.43 80 tcp http 15.168613 898 376 S2 - 0 ShADadF 8 1238 8 700 (empty) +1320329757.771755 TEfuqmmG4bh 10.0.2.15 49287 192.150.187.43 80 tcp http 15.161267 900 376 S2 - 0 ShADadF 8 1240 8 700 (empty) +#close 2013-05-07-14-38-27 diff --git a/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/http.log b/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/http.log new file mode 100644 index 0000000000..031a9ce2ce --- /dev/null +++ b/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/http.log @@ -0,0 +1,26 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path http +#open 2013-05-07-14-38-27 +#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p trans_depth method host uri referrer user_agent request_body_len response_body_len status_code status_msg info_code info_msg filename tags username password proxied mime_type md5 extraction_file +#types time string addr port addr port count string string string string string count count count string count string string table[enum] string string table[string] string string file +1320329757.460004 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 1 GET bro-ids.org / - Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.772457 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 2 GET bro-ids.org /css/pygments.css http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.874406 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 3 GET bro-ids.org /js/jquery.zrssfeed.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.775110 k6kgXLOoSKl 10.0.2.15 49284 192.150.187.43 80 1 GET bro-ids.org /css/960.css http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.776072 TEfuqmmG4bh 10.0.2.15 49287 192.150.187.43 80 1 GET bro-ids.org /js/jquery.cycle.all.min.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.776421 nQcgTWjvg4c 10.0.2.15 49285 192.150.187.43 80 1 GET bro-ids.org /js/jquery.tweet.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.776240 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 1 GET bro-ids.org /js/jquery.fancybox-1.3.4.pack.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.775251 arKYeMETxOg 10.0.2.15 49283 192.150.187.43 80 1 GET bro-ids.org /css/bro-ids.css http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.975651 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 4 GET bro-ids.org /js/jquery.tableofcontents.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.979943 k6kgXLOoSKl 10.0.2.15 49284 192.150.187.43 80 2 GET bro-ids.org /js/superfish.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.985656 TEfuqmmG4bh 10.0.2.15 49287 192.150.187.43 80 2 GET bro-ids.org /js/hoverIntent.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.989904 nQcgTWjvg4c 10.0.2.15 49285 192.150.187.43 80 2 GET bro-ids.org /js/general.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.991315 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 2 GET bro-ids.org /js/jquery.collapse.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329758.172397 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 3 GET bro-ids.org /css/print.css http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329759.998388 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 4 GET bro-ids.org /documentation/index.html http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329760.146412 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 5 GET bro-ids.org /js/breadcrumbs.js http://bro-ids.org/documentation/index.html Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329762.971726 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 6 GET bro-ids.org /documentation/reporting-problems.html http://bro-ids.org/documentation/index.html Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +#close 2013-05-07-14-38-27 diff --git a/testing/btest/Traces/workshop_2011_browse.trace b/testing/btest/Traces/workshop_2011_browse.trace new file mode 100644 index 0000000000..74d2f9c21f Binary files /dev/null and b/testing/btest/Traces/workshop_2011_browse.trace differ diff --git a/testing/btest/doc/manual/using_bro_sandbox_02 b/testing/btest/doc/manual/using_bro_sandbox_02 new file mode 100644 index 0000000000..5a21b59800 --- /dev/null +++ b/testing/btest/doc/manual/using_bro_sandbox_02 @@ -0,0 +1,4 @@ +# @TEST-EXEC: bro -r ${TRACES}/workshop_2011_browse.trace +# @TEST-EXEC: btest-diff conn.log +# @TEST-EXEC: btest-diff http.log +