Basic cross-referencing UIDs between files, btests, and baselines.

Also includes appropriate btest-rst-cmd directives with titles.

doc/using/index.rst

@@ -29,7 +29,7 @@ The ``bro-cut`` utility can be used in place of other tools to build terminal co
 
 .. btest:: using_bro_bro_cut_02
 
-   @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/conn.log | btest-rst-cmd bro-cut id.orig_h id.orig_p id.resp_h duration 
+   @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/conn.log | btest-rst-cmd -c "cat conn.log | bro-cut id.orig_h id.orig_p id.resp_h duration " bro-cut id.orig_h id.orig_p id.resp_h duration 
 
 
 While the output is similar, the advantages to using bro-cut over awk lay in that,  while awk is flexible and powerful, ``bro-cut`` was specifically designed to work with log files.  Firstly, the ``bro-cut`` output includes only the log file entries, while the ``awk`` output includes the header parts of the log file, which would require the user to use a secondary utility to suppress those lines.  Secondly, since ``bro-cut`` uses the field descriptors to identify and extract data, it allows for flexibility independent of the format and contents of the log file.  It's not uncommon for a Bro configuration to add extra fields to various log files as required by the environment.  In this case, the fields in the ``awk`` command would have to be altered to compensate for the new position whereas the ``bro-cut`` output would not change.
@@ -44,25 +44,36 @@ The ``bro-cut`` accepts the flag ``-d`` to convert the epoch time values in the
 
 .. btest:: using_bro_bro_cut_time_01
 
-   @TEST-EXEC: btest-rst-cmd bro-cut -d ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log
+   @TEST-EXEC: btest-rst-cmd -c "bro-cut -d ts uid host uri < http.log" bro-cut -d ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log
 
 Often times log files from multiple sources are stored in UTC time to allow easy correlation.  Converting the timestamp from a log file to UTC can be accomplished with the ``-u`` command.  
 
 .. btest:: using_bro_bro_cut_time_02
 
-   @TEST-EXEC: btest-rst-cmd bro-cut -u ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log
+   @TEST-EXEC: btest-rst-cmd -c "bro-cut -u ts uid host uri < http.log" bro-cut -u ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log
 
 The default time format when using the ``-d`` or ``-u`` is the ``strftime`` format string %Y-%m-%dT%H:%M:%S%z which results in a string with year, month, day of month, followed by hour, minutes, seconds and the timezone offset.  The default ``strftime`` can be altered by using the ``-D`` and ``-U`` flags. For example, to format the timestamp in the US-typical "Middle Endian" you could use a format string of: %d-%m-%YT%H:%M:%S%z
 
 .. btest:: using_bro_bro_cut_time_03
 
-   @TEST-EXEC: btest-rst-cmd bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log
+   @TEST-EXEC: btest-rst-cmd -c "bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < http.log" bro-cut -D %d-%m-%YT%H:%M:%S%z ts uid host uri < ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_01/http.log
 
 ----------------------
 Working with Log Files
 ----------------------
 
-As Bro runs, it deposits its log files in 
+While Bro can do signature based analysis, its primary focus is on behavioral detection which alters the practice of log review from "reactionary review" to a process a little more akin to a hunting trip.  A common progression of review includes correlating a session across multiple log files.  As a connection is processed by Bro, a unique identifier is assigned to each session.  This unique identifier is almost always included in any log file entry specific to that connection and can be used to cross-reference log files.  
 
+A simple example would be to cross-reference a UID seen in a ``conn.log`` file.  Here, we're looking for the connection with the largest number of bytes from the responder by redirecting the output for ``cat conn.log`` into bro-cut to extract the UID and the resp_bytes, then sorting that output by the resp_bytes field.
 
+.. btest:: using_bro_practical_02
 
+   @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_02/conn.log | bro-cut uid resp_bytes | btest-rst-cmd -c "cat conn.log | bro-cut uid resp_bytes | btest-rst-cmd sort -nrk2" sort -nrk2
+   
+With the UID of the largest response, it can be crossreferenced with the UIDs in the ``http.log`` file.
+
+.. btest:: using_bro_practical_03
+
+   @TEST-EXEC: cat ${TESTBASE}/Baseline/doc.manual.using_bro_sandbox_02/http.log | bro-cut uid id.resp_h method status_code host uri | btest-rst-cmd -c "cat http.log | bro-cut uid id.resp_h method status_code host uri | grep j4u32Pc5bif" grep j4u32Pc5bif
+
+As you can see there are multiple HTTP GET requests within the session that Bro identified and logged.  Given that HTTP is a stream protocol, it can have multiple GET/POST/etc requests in a stream and Bro is able to extract and track that information for you, giving you an in-depth and structured view into HTTP traffic on your network.

testing/btest/Baseline/doc.manual.using_bro_sandbox_02/conn.log

@@ -0,0 +1,15 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	conn
+#open	2013-05-07-14-38-27
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	proto	service	duration	orig_bytes	resp_bytes	conn_state	local_orig	missed_bytes	history	orig_pkts	orig_ip_bytes	resp_pkts	resp_ip_bytes	tunnel_parents
+#types	time	string	addr	port	addr	port	enum	string	interval	count	count	string	bool	count	string	count	count	count	count	table[string]
+1320329757.771503	j4u32Pc5bif	10.0.2.15	49286	192.150.187.43	80	tcp	http	15.161537	2899	1127	S2	-	0	ShADadF	20	3719	19	1891	(empty)
+1320329757.771262	nQcgTWjvg4c	10.0.2.15	49285	192.150.187.43	80	tcp	http	15.161772	889	377	S2	-	0	ShADadF	8	1229	8	701	(empty)
+1320329757.761327	arKYeMETxOg	10.0.2.15	49283	192.150.187.43	80	tcp	http	15.168898	459	189	S2	-	0	ShADadF	5	679	4	353	(empty)
+1320329757.458867	UWkUyAuUGXf	10.0.2.15	49282	192.150.187.43	80	tcp	http	15.471378	1824	751	S2	-	0	ShADadF	12	2324	13	1275	(empty)
+1320329757.761638	k6kgXLOoSKl	10.0.2.15	49284	192.150.187.43	80	tcp	http	15.168613	898	376	S2	-	0	ShADadF	8	1238	8	700	(empty)
+1320329757.771755	TEfuqmmG4bh	10.0.2.15	49287	192.150.187.43	80	tcp	http	15.161267	900	376	S2	-	0	ShADadF	8	1240	8	700	(empty)
+#close	2013-05-07-14-38-27

testing/btest/Baseline/doc.manual.using_bro_sandbox_02/http.log

@@ -0,0 +1,26 @@
+#separator \x09
+#set_separator	,
+#empty_field	(empty)
+#unset_field	-
+#path	http
+#open	2013-05-07-14-38-27
+#fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	trans_depth	method	host	uri	referrer	user_agent	request_body_len	response_body_len	status_code	status_msg	info_code	info_msg	filename	tags	username	password	proxied	mime_type	md5	extraction_file
+#types	time	string	addr	port	addr	port	count	string	string	string	string	string	count	count	count	string	count	string	string	table[enum]	string	string	table[string]	string	string	file
+1320329757.460004	UWkUyAuUGXf	10.0.2.15	49282	192.150.187.43	80	1	GET	bro-ids.org	/	-	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329757.772457	UWkUyAuUGXf	10.0.2.15	49282	192.150.187.43	80	2	GET	bro-ids.org	/css/pygments.css	http://bro-ids.org/	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329757.874406	UWkUyAuUGXf	10.0.2.15	49282	192.150.187.43	80	3	GET	bro-ids.org	/js/jquery.zrssfeed.js	http://bro-ids.org/	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329757.775110	k6kgXLOoSKl	10.0.2.15	49284	192.150.187.43	80	1	GET	bro-ids.org	/css/960.css	http://bro-ids.org/	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329757.776072	TEfuqmmG4bh	10.0.2.15	49287	192.150.187.43	80	1	GET	bro-ids.org	/js/jquery.cycle.all.min.js	http://bro-ids.org/	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329757.776421	nQcgTWjvg4c	10.0.2.15	49285	192.150.187.43	80	1	GET	bro-ids.org	/js/jquery.tweet.js	http://bro-ids.org/	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329757.776240	j4u32Pc5bif	10.0.2.15	49286	192.150.187.43	80	1	GET	bro-ids.org	/js/jquery.fancybox-1.3.4.pack.js	http://bro-ids.org/	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329757.775251	arKYeMETxOg	10.0.2.15	49283	192.150.187.43	80	1	GET	bro-ids.org	/css/bro-ids.css	http://bro-ids.org/	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329757.975651	UWkUyAuUGXf	10.0.2.15	49282	192.150.187.43	80	4	GET	bro-ids.org	/js/jquery.tableofcontents.js	http://bro-ids.org/	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329757.979943	k6kgXLOoSKl	10.0.2.15	49284	192.150.187.43	80	2	GET	bro-ids.org	/js/superfish.js	http://bro-ids.org/	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329757.985656	TEfuqmmG4bh	10.0.2.15	49287	192.150.187.43	80	2	GET	bro-ids.org	/js/hoverIntent.js	http://bro-ids.org/	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329757.989904	nQcgTWjvg4c	10.0.2.15	49285	192.150.187.43	80	2	GET	bro-ids.org	/js/general.js	http://bro-ids.org/	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329757.991315	j4u32Pc5bif	10.0.2.15	49286	192.150.187.43	80	2	GET	bro-ids.org	/js/jquery.collapse.js	http://bro-ids.org/	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329758.172397	j4u32Pc5bif	10.0.2.15	49286	192.150.187.43	80	3	GET	bro-ids.org	/css/print.css	http://bro-ids.org/	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329759.998388	j4u32Pc5bif	10.0.2.15	49286	192.150.187.43	80	4	GET	bro-ids.org	/documentation/index.html	http://bro-ids.org/	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329760.146412	j4u32Pc5bif	10.0.2.15	49286	192.150.187.43	80	5	GET	bro-ids.org	/js/breadcrumbs.js	http://bro-ids.org/documentation/index.html	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+1320329762.971726	j4u32Pc5bif	10.0.2.15	49286	192.150.187.43	80	6	GET	bro-ids.org	/documentation/reporting-problems.html	http://bro-ids.org/documentation/index.html	Mozilla/5.0 (X11; Linux i686) AppleWebKit/535.2 (KHTML, like Gecko) Chrome/15.0.874.106 Safari/535.2	0	0	304	Not Modified	-	-	-	(empty)	-	-	-	-	-	-
+#close	2013-05-07-14-38-27

testing/btest/doc/manual/using_bro_sandbox_02

@@ -0,0 +1,4 @@
+# @TEST-EXEC: bro -r ${TRACES}/workshop_2011_browse.trace
+# @TEST-EXEC: btest-diff conn.log
+# @TEST-EXEC: btest-diff http.log
+