Matthias Vallentin ff8184242a Enhance ssl.log with information from notary. ... This commit brings enhances each log line with the data from the notary when available. The added fields include: - notary.first_seen - notary.last_seen - notary.times_seen - notary.valid The semantics of these fields map 1-to-1 to the corresponding fields in DNS TXT lookups from the notary. The implementation of this feature required a bit plumbing: when Bro finishes the analysis, the log record is copied into table indexed by connection ID where it remains until either Bro terminates or the answer of the notary arrives. The script accummulates requests for a given digest into a "waitlist," to avoid multiple redundant lookups for high-profile websites who receive a large chunk of traffic. When a DNS reply arrives asynchronously, the when handler clears the waitlist and assigns the information to all records in the buffered. The script also adds Each log entry into a double-ended queue to make sure the records arrive on disk in the same way Bro sees them. Each reply also triggers a sweep through this deque which flushes the buffer up to the first outstanding reply. Here is an example from the public M57 trace from 2009: % bro-cut ts id.orig_h id.resp_h server_name notary.first_seen notary.last_seen notary.times_seen notary.valid < ssl.log 1258562650.121682 192.168.1.104 208.97.132.223 mail.m57.biz - - - - 1258535660.267128 192.168.1.104 65.55.184.16 - - - - - 1258561662.604948 192.168.1.105 66.235.128.158 - - - - - 1258561885.571010 192.168.1.105 65.55.184.155 www.update.microsoft.com - - - - 1258563578.455331 192.168.1.103 208.97.132.223 - - - - - 1258563716.527681 192.168.1.103 96.6.248.124 - - - - - 1258563884.667153 192.168.1.103 66.235.139.152 - - - - - 1258564818.755676 192.168.1.103 12.41.118.177 - - - - - 1258564821.637874 192.168.1.103 12.41.118.177 - - - - - 1258564821.637871 192.168.1.103 12.41.118.177 - - - - - 1258564821.637876 192.168.1.103 12.41.118.177 - - - - - 1258564821.638126 192.168.1.103 12.41.118.177 - - - - - 1258562467.525034 192.168.1.104 208.97.132.223 mail.m57.biz 15392 15695 301 F 1258563063.965975 192.168.1.104 63.245.209.105 aus2.mozilla.org - - - - 1258563064.091396 192.168.1.104 63.245.209.91 addons.mozilla.org - - - - 1258563329.202273 192.168.1.103 208.97.132.223 - 15392 15695 301 F 1258563712.945933 192.168.1.103 65.55.16.121 - - - - - 1258563714.044500 192.168.1.103 65.54.186.79 - - - - - 1258563716.146680 192.168.1.103 96.6.248.124 - - - - - 1258563737.432312 192.168.1.103 96.6.245.186 - - - - - 1258563716.526933 192.168.1.103 96.6.245.186 - - - - - 1258563716.527430 192.168.1.103 96.6.245.186 - - - - - 1258563716.527179 192.168.1.103 96.6.245.186 - - - - - 1258563716.527683 192.168.1.103 96.6.245.186 - - - - - 1258563716.527432 192.168.1.103 96.6.245.186 - - - - - 1258563751.178683 192.168.1.103 66.235.139.152 - - - - - 1258563751.171938 192.168.1.103 65.54.234.75 - - - - - 1258563751.182433 192.168.1.103 65.242.27.35 - - - - - 1258563883.414188 192.168.1.103 65.55.16.121 - - - - - 1258563884.702380 192.168.1.103 65.242.27.35 - - - - - 1258563885.678766 192.168.1.103 65.54.186.79 - - - - - 1258563886.124987 192.168.1.103 65.54.186.79 - - - - - 1258564027.877525 192.168.1.103 65.54.234.75 - - - - - 1258564688.206859 192.168.1.103 65.54.186.107 - - - - - 1258567162.001225 192.168.1.105 208.97.132.223 mail.m57.biz - - - - 1258568040.512840 192.168.1.103 208.97.132.223 - - - - - 1258564688.577376 192.168.1.103 207.46.120.170 - - - - - 1258564723.029005 192.168.1.103 65.54.186.107 - - - - - 1258564723.784032 192.168.1.103 65.55.194.249 - - - - - 1258564748.521756 192.168.1.103 65.54.186.107 - - - - - 1258564817.601152 192.168.1.103 12.41.118.177 - - - - - 1258565684.353653 192.168.1.105 208.97.132.223 mail.m57.biz 15392 15695 301 F 1258565710.188691 192.168.1.105 74.125.155.109 pop.gmail.com - - - - 1258566061.103696 192.168.1.103 208.97.132.223 - 15392 15695 301 F 1258566893.914987 192.168.1.102 208.97.132.223 - 15392 15695 301 F		2012-12-21 17:03:39 -08:00
..
conn	Merge remote-tracking branch 'origin/topic/jsiwek/gridftp'	2012-10-12 10:43:16 -07:00
dns	Fixed a DNS attribute issue (reported by Matt Thompson).	2012-11-26 15:58:25 -05:00
ftp	Change how "gridftp" gets added to service field of connection records.	2012-10-17 12:09:12 -05:00
http	Merge remote-tracking branch 'origin/topic/matthias/opaque'	2012-12-20 16:30:22 -08:00
irc	Fix some Info:Record field documentation.	2012-07-13 14:04:24 -04:00
modbus	Fixing tests after modbus merge.	2012-11-05 15:58:38 -08:00
smtp	Merge remote-tracking branch 'origin/topic/matthias/opaque'	2012-12-20 16:30:22 -08:00
socks	Fix some Info:Record field documentation.	2012-07-13 14:04:24 -04:00
ssh	Fix some Info:Record field documentation.	2012-07-13 14:04:24 -04:00
ssl	Enhance ssl.log with information from notary.	2012-12-21 17:03:39 -08:00
syslog	Fix some Info:Record field documentation.	2012-07-13 14:04:24 -04:00