From 4aa51e760845a04097e9375b80ca97a464eb720a Mon Sep 17 00:00:00 2001 From: Scott Runnels Date: Tue, 7 May 2013 13:33:38 -0400 Subject: [PATCH] Basic cross-referencing UIDs between files, btests, and baselines. Also includes appropriate btest-rst-cmd directives with titles. --- doc/using/index.rst | 21 ++++++++++---- .../doc.manual.using_bro_sandbox_02/conn.log | 15 ++++++++++ .../doc.manual.using_bro_sandbox_02/http.log | 26 ++++++++++++++++++ .../btest/Traces/workshop_2011_browse.trace | Bin 0 -> 19909 bytes testing/btest/doc/manual/using_bro_sandbox_02 | 4 +++ 5 files changed, 61 insertions(+), 5 deletions(-) create mode 100644 testing/btest/Baseline/doc.manual.using_bro_sandbox_02/conn.log create mode 100644 testing/btest/Baseline/doc.manual.using_bro_sandbox_02/http.log create mode 100644 testing/btest/Traces/workshop_2011_browse.trace create mode 100644 testing/btest/doc/manual/using_bro_sandbox_02 diff --git a/doc/using/index.rst b/doc/using/index.rst index a6d42ed211..fc979dd30a 100644 --- a/doc/using/index.rst +++ b/doc/using/index.rst @@ -29,7 +29,7 @@ The ``bro-cut`` utility can be used in place of other tools to build terminal co .. btest:: using_bro_bro_cut_02 - @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/conn.log | btest-rst-cmd bro-cut id.orig_h id.orig_p id.resp_h duration + @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/conn.log | btest-rst-cmd -c "cat conn.log | bro-cut id.orig_h id.orig_p id.resp_h duration " bro-cut id.orig_h id.orig_p id.resp_h duration While the output is similar, the advantages to using bro-cut over awk lay in that, while awk is flexible and powerful, ``bro-cut`` was specifically designed to work with log files. Firstly, the ``bro-cut`` output includes only the log file entries, while the ``awk`` output includes the header parts of the log file, which would require the user to use a secondary utility to suppress those lines. Secondly, since ``bro-cut`` uses the field descriptors to identify and extract data, it allows for flexibility independent of the format and contents of the log file. It's not uncommon for a Bro configuration to add extra fields to various log files as required by the environment. In this case, the fields in the ``awk`` command would have to be altered to compensate for the new position whereas the ``bro-cut`` output would not change. @@ -44,25 +44,36 @@ The ``bro-cut`` accepts the flag ``-d`` to convert the epoch time values in the .. btest:: using_bro_bro_cut_time_01 - @TEST-EXEC: btest-rst-cmd bro-cut -d ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log + @TEST-EXEC: btest-rst-cmd -c "bro-cut -d ts uid host uri < http.log" bro-cut -d ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log Often times log files from multiple sources are stored in UTC time to allow easy correlation. Converting the timestamp from a log file to UTC can be accomplished with the ``-u`` command. .. btest:: using_bro_bro_cut_time_02 - @TEST-EXEC: btest-rst-cmd bro-cut -u ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log + @TEST-EXEC: btest-rst-cmd -c "bro-cut -u ts uid host uri < http.log" bro-cut -u ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log The default time format when using the ``-d`` or ``-u`` is the ``strftime`` format string %Y-%m-%dT%H:%M:%S%z which results in a string with year, month, day of month, followed by hour, minutes, seconds and the timezone offset. The default ``strftime`` can be altered by using the ``-D`` and ``-U`` flags. For example, to format the timestamp in the US-typical "Middle Endian" you could use a format string of: %d-%m-%YT%H:%M:%S%z .. btest:: using_bro_bro_cut_time_03 - @TEST-EXEC: btest-rst-cmd bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log + @TEST-EXEC: btest-rst-cmd -c "bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < http.log" bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log ---------------------- Working with Log Files ---------------------- -As Bro runs, it deposits its log files in +While Bro can do signature based analysis, its primary focus is on behavioral detection which alters the practice of log review from "reactionary review" to a process a little more akin to a hunting trip. A common progression of review includes correlating a session across multiple log files. As a connection is processed by Bro, a unique identifier is assigned to each session. This unique identifier is almost always included in any log file entry specific to that connection and can be used to cross-reference log files. +A simple example would be to cross-reference a UID seen in a ``conn.log`` file. Here, we're looking for the connection with the largest number of bytes from the responder by redirecting the output for ``cat conn.log`` into bro-cut to extract the UID and the resp_bytes, then sorting that output by the resp_bytes field. +.. btest:: using_bro_practical_02 + @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_02/conn.log | bro-cut uid resp_bytes | btest-rst-cmd -c "cat conn.log | bro-cut uid resp_bytes | btest-rst-cmd sort -nrk2" sort -nrk2 + +With the UID of the largest response, it can be crossreferenced with the UIDs in the ``http.log`` file. + +.. btest:: using_bro_practical_03 + + @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_02/http.log | bro-cut uid id.resp_h method status_code host uri | btest-rst-cmd -c "cat http.log | bro-cut uid id.resp_h method status_code host uri | grep j4u32Pc5bif" grep j4u32Pc5bif + +As you can see there are multiple HTTP GET requests within the session that Bro identified and logged. Given that HTTP is a stream protocol, it can have multiple GET/POST/etc requests in a stream and Bro is able to extract and track that information for you, giving you an in-depth and structured view into HTTP traffic on your network. diff --git a/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/conn.log b/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/conn.log new file mode 100644 index 0000000000..1227e60ad3 --- /dev/null +++ b/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/conn.log @@ -0,0 +1,15 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path conn +#open 2013-05-07-14-38-27 +#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents +#types time string addr port addr port enum string interval count count string bool count string count count count count table[string] +1320329757.771503 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 tcp http 15.161537 2899 1127 S2 - 0 ShADadF 20 3719 19 1891 (empty) +1320329757.771262 nQcgTWjvg4c 10.0.2.15 49285 192.150.187.43 80 tcp http 15.161772 889 377 S2 - 0 ShADadF 8 1229 8 701 (empty) +1320329757.761327 arKYeMETxOg 10.0.2.15 49283 192.150.187.43 80 tcp http 15.168898 459 189 S2 - 0 ShADadF 5 679 4 353 (empty) +1320329757.458867 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 tcp http 15.471378 1824 751 S2 - 0 ShADadF 12 2324 13 1275 (empty) +1320329757.761638 k6kgXLOoSKl 10.0.2.15 49284 192.150.187.43 80 tcp http 15.168613 898 376 S2 - 0 ShADadF 8 1238 8 700 (empty) +1320329757.771755 TEfuqmmG4bh 10.0.2.15 49287 192.150.187.43 80 tcp http 15.161267 900 376 S2 - 0 ShADadF 8 1240 8 700 (empty) +#close 2013-05-07-14-38-27 diff --git a/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/http.log b/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/http.log new file mode 100644 index 0000000000..031a9ce2ce --- /dev/null +++ b/testing/btest/Baseline/doc.manual.using_bro_sandbox_02/http.log @@ -0,0 +1,26 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path http +#open 2013-05-07-14-38-27 +#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p trans_depth method host uri referrer user_agent request_body_len response_body_len status_code status_msg info_code info_msg filename tags username password proxied mime_type md5 extraction_file +#types time string addr port addr port count string string string string string count count count string count string string table[enum] string string table[string] string string file +1320329757.460004 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 1 GET bro-ids.org / - Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.772457 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 2 GET bro-ids.org /css/pygments.css http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.874406 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 3 GET bro-ids.org /js/jquery.zrssfeed.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.775110 k6kgXLOoSKl 10.0.2.15 49284 192.150.187.43 80 1 GET bro-ids.org /css/960.css http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.776072 TEfuqmmG4bh 10.0.2.15 49287 192.150.187.43 80 1 GET bro-ids.org /js/jquery.cycle.all.min.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.776421 nQcgTWjvg4c 10.0.2.15 49285 192.150.187.43 80 1 GET bro-ids.org /js/jquery.tweet.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.776240 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 1 GET bro-ids.org /js/jquery.fancybox-1.3.4.pack.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.775251 arKYeMETxOg 10.0.2.15 49283 192.150.187.43 80 1 GET bro-ids.org /css/bro-ids.css http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.975651 UWkUyAuUGXf 10.0.2.15 49282 192.150.187.43 80 4 GET bro-ids.org /js/jquery.tableofcontents.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.979943 k6kgXLOoSKl 10.0.2.15 49284 192.150.187.43 80 2 GET bro-ids.org /js/superfish.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.985656 TEfuqmmG4bh 10.0.2.15 49287 192.150.187.43 80 2 GET bro-ids.org /js/hoverIntent.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.989904 nQcgTWjvg4c 10.0.2.15 49285 192.150.187.43 80 2 GET bro-ids.org /js/general.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329757.991315 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 2 GET bro-ids.org /js/jquery.collapse.js http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329758.172397 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 3 GET bro-ids.org /css/print.css http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329759.998388 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 4 GET bro-ids.org /documentation/index.html http://bro-ids.org/ Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329760.146412 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 5 GET bro-ids.org /js/breadcrumbs.js http://bro-ids.org/documentation/index.html Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +1320329762.971726 j4u32Pc5bif 10.0.2.15 49286 192.150.187.43 80 6 GET bro-ids.org /documentation/reporting-problems.html http://bro-ids.org/documentation/index.html Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2 0 0 304 Not Modified - - - (empty) - - - - - - +#close 2013-05-07-14-38-27 diff --git a/testing/btest/Traces/workshop_2011_browse.trace b/testing/btest/Traces/workshop_2011_browse.trace new file mode 100644 index 0000000000000000000000000000000000000000..74d2f9c21f60eec207455470b38c775487aa52e2 GIT binary patch literal 19909 zcmeHP3v^V~xjr+A$Q>FI=&eR8cnT;ZW*%oACvzrHpb#aEu~P|BD_SRK&LjiLOqiL( z5X@S)i1J!$Sv;;Sw^qGyZ?A~A*oQ$)Lm<-f5cMj^sG#Lh=*2cfN^2jG`~Ul7&g^-d zy_UvUa#v=py%w1TU%vf6zWwii@6B7gpMT28<#WkTUmwRA;D>QfKYx3SyOR4l{29j` z)!bOOaU?f+#rjDjxpI!1+I3_GH)BNaSQBR~+_~|kn|Chbd-fiLw(ueX?CkrGQ~E9 z{K=C9vXqDANFWrh_42i?p~jMc60G;fm6BLMuG9X`ZT@g=lfPE+@=Dk;Z*GYa&H$NJ z=a0q|U{x{q8y1hpEm?$;rg+feA%Cy5ME#}C0#ilMa(g7KSgQPSxz5W^aJvLosc3PQ z);KJ#8hgO0*yVualI;`7->m{_!H^QL%ngNQ`1RFwO(neD$=@Dn<{frH;01@*DR~`E z{?@AM0#ktrLv-5+Mu?iFpOO&0Eh9vY2D7-)SJc-hJ_6B5K8Ur{YT|m~CmdPfenR>= zVzob(6RWtd&+;>lONJHiw7YXNZ+hgA!?qPji>>4gQvCg56$VDzEl~&zTND z1FQlAFmm)=lA{yTvrF6vIojbXn%e%CYt$S)Y%}Sr#uVun#3i?8n6ZPn1k7muCC3$* z&?m^~7+^6Ny+;^L$ib+SFuGbr>G|O(hC)tvAWCe9hVmfiD_OoTf*epG6R)Q!JhD6i@xa0G4b-MvFGmj4m)4 zO?M)r*a3}^>XAmkBaFUYsWSRyH_Q4*`Vb{?RzrD+^Hq$n{3vPtZ(o~c{rm9yq+kF0 z6&7-vjL7bl8uG)OkFQ*rmqa!<>LYL6!$LMUA@b^G4S5CUbDux@r6h6(hzzw_--q8P zIfwR+Vy%rw5xM714VlydBzrr3#Z%iCezAXTlw&bl6 zLkTKT6xO>tQCNwg576Z_*g4DVh&>gwf`+D#`{nBM@k?(F}^wL*kju z29S{_k7LnS$df;qGc9~?oNZ*n|ar>%1Bxb$L>SAIjy|zD-GjB-)47;BzdM?MeU& zYqw%g%0Z1gAb-MA+3lv~VD4M2^OgfgD4HXK46Jy+#Hlcs5JF0s?MlNv+`}0(wOhZ;9A+G1sABYmssT`a=l{p*VkJcLSgH|AycG`=E!)p+GK6Y6YEY5(_oeZx5_k2!4 zNL+x521iy%oK6@pB#wMl3kkZ;f{^$(S5`>G?^l%ga1ab`5I8LkM~(8C1;N+^Q)4=b z#?Xa3Zf=b3^T&>&#w?1`8nc&;wUb~?r#1n&Eiv+ytKLAY_im)I-hG$JtBYWLmBPXe zg|T9U?K_C|#bY$qq3QxwLvj~lwJ)QvQdDqvM5yGdqFP0yy?tp>Aq0%oSLiq6dD_eXP( zUSwn)75ql(&Xt+{Sdf`>zjAqICL9!LCk1vr^J8D3^Mld)rkoE5gykNa!tl}J#IgyW{Xy^et&3$8iE2vD=MhXkmU5uk2^ zx$G?Fx-G4Vw(Hzz)++of#A zENFv+8f0|(JUABg0!Nl+oY+A25!cPii2?^EFF@Is>{-N`8kJ};6ssGq2nMkt5L_}(u`pQ>kYE~hWg$x?;Jyv0L@R}r+K)hZ z|7-0KPzkWVPNN93=_)XQeomTOFeg{9(6Whi0FNDoO&L)MOP`vPV@4LTDvSNZC5{X; zc93lynDL%}r&XdDuo#S%5JnH@U_?~Ha#3aEF%@#>K_PHtsl+LBE26CYbxsseiCadd zRH7~dmpdv*Ju$2kgQ^pyf<>~uo@0wXBR(g6~b&Wxb!bf5p=2dEQuaB`x4Fioah z&D4ok5v%=v3M@ev0%OnsJAfG9yM~2kBOU8AlY-a{EmtqeLEY?B6s-DaR7IJzS?UD!+!w z2<~D5BMZd{9os>Aalq){e40@eU@>e|OKkMG#wcYY!svZlU+*AI*Tpo+iUZV{Y8SF%bR@NvVR!2jn#z;R^tdQM4i zU_`brrXn(RngtQrJ!c3=jo{KdTUPt97r_Mji8S4fDG=MIr2x4)aG(qh^++PkhO1>L zHj~FOQrj&34;nHpwNY$3D@F*{;4(S}N@VqeSabP8?qf(u99c>EzioBM$azR(M5i_+ zum<^9DIQzE76ZjZy{D%qQh}Zx7sv17PVpWP=eMGVY@05YjSJCW$9I$Wfa49@Qbu^Oz2S-3g>f zm|}!o4s|C4j2?p<*Oz08q&q}s^GFtj#gcSP1HR72qOYyY%H8_q(MvFrR?jz7I z99gT#N$EY}ky?sTYDEG@dyWqUkLa!91tzpW$0e2>+d^nZ_eRZ*q&8c#PCUJp*zs}b z&Lc_#Ja_~kGO{YXb8QO|CG+A}KMI`MyYK9;@W^w9Xeb;XzP`NHZ>YLFGn78O*EK&*e*sKiU*_5tk@Q@jo}Q1~EuMbXg$M3y?@-f|H|Og+WQqh2 zBfuf!(~bD7i#P;~xEInhRV+>hEC!=`!f0I%Mj~P4i&@j`IQ$xd|UXsc&pU;egU&#fP>-@*UJb-V6_kM^kng&ZL}@-`}q^ zf*>jOPkMRTlYAzDVf2OR%pE113PE5SGXh)WD<4~cO7oW1rQiW;(h6pEb_ucC>8G%u zTFQ668(tnuLF z5UhA9!z1U2N3f#OYCOdHF<>zmMF^wsrHce%^buhstBgJ@GjgLqYH(yqjp1TjBceE8 z%!yK5G3N-h1d}%hYNCoiAV-@TYGVH%J5PFIdf7a>gqts=mQqE#oIaq26b9H!(^-Xz z7m5%F*1n8j{fMtzE1-P5ak#%uA%TDYfjmZ`;)UWEu*!2{LE!I)3yPn5w>^mRAyN$~ zUGt+U*TD1{xw@F->Z9pdwpI{NZ`c;shu@JL@*(bP-Wvdo>jLbZ^Ym_Ss`qlnqRjt`msT`(+J< z?)ZSRX~|st)dhP zEW0E|AWGt64TbIp0?KP$D#|P-N?S9cxKsag0_ld4s0sPTfRDbXqHG5g-HRG#?=B8d z+NV)bgNM+a@T77o=JCJ4^+WZ8HhzVTwVPmNJcQ`(g>#+{5z7kIYYJ;W8%uo%QAY8C z6Pk`zIHI@#u|l(HtS6>2=lBZY<)yITaAA8Zbk{c_Rvns$d~lA>u(8Mm@OJtFIP}H0 nEQP-CwTM*&eeo35PuN($BE0C=4zP=RE(96-EyOzLNn-sU+snk@ literal 0 HcmV?d00001 diff --git a/testing/btest/doc/manual/using_bro_sandbox_02 b/testing/btest/doc/manual/using_bro_sandbox_02 new file mode 100644 index 0000000000..5a21b59800 --- /dev/null +++ b/testing/btest/doc/manual/using_bro_sandbox_02 @@ -0,0 +1,4 @@ +# @TEST-EXEC: bro -r ${TRACES}/workshop_2011_browse.trace +# @TEST-EXEC: btest-diff conn.log +# @TEST-EXEC: btest-diff http.log +