Creating a branch release/1.5 with the current 1.5.3 release code.

This is so that people working from the current stable version can
still start using git.

CHANGES

@@ -2,30 +2,65 @@
 
 -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 
-1.5.2.7 Sun Sep 12 19:39:49 PDT 2010
+1.5.3 Thu Mar  3 08:55:11 PST 2011
 
-- Addressed a number of lint nits (Vern Paxson).
+- Removing aux/broctl/policy/cluster-addrs.hot.bro from the
+  distribution. The script is no longer needed and could in fact break
+  an installation because it redefines an old variable that has went
+  away. (Robin Sommer)
 
+- Smarter way to increase the communication module's pipe's socket
+  buffer size, resulting in a value closer to the allowed maximum. 
+  (Craig Leres)
 
-1.5.2.6 Sun Sep 12 17:00:13 PDT 2010
+- BroControl now also maintains links from the log archive to the
+  current set of logs when running in standalone mode. (Robin Sommer)
+
+- Bug fix for a file descriptor leak in the remote communication
+  module. (Scott Campbell)
+
+- Bug fix for BroControl to now activate trace-summary's sampling in
+  cluster mode, but not anymore in standalone mode. (Robin Sommer)
+
+- Broccoli updates:
+
+    * Accept empty strings ("") as values in the configuration file.
+      (Craig Leres)
+
+    * Support for specifying a separate host key for SSL-enabled
+      operation, with documentation update. (Craig Leres)
+
+1.5.2 Wed Jan 12 17:34:55 PST 2011
+
+- Portability fixes for --enable-int64 (Vern Paxson).
+
+- Bug fix for Active Mapping support (Kevin Lo).
+
+- Broccoli compiler warning fixes (Kevin Lo).
+
+- Bug fixes for --enable-int64 and for avoiding bogus statistics /
+  bad memory references when generating profiling information upon
+  exit (Vern Paxson).
+
+- Bug fixes for terminating connections (Tyler Schoenke and Vern Paxson).
+
+- Removed now-quite-stale SSHv1 overflow detection, as it's more prone
+  to false positives than useful detection (Vern Paxson).
 
 - The SWIG file now explicitly lists those pieces from broccoli.h which it
   wants to wrap, rather than just including all of broccoli.h (Robin Sommer).
-  This fixes the problem that the SWIG bindings depend on what configure
-  finds out about the availability of libpcap even though the corresponding
-  functions don't need to be wrapped anyway.
 
-- http-header.bro now includes a global include_header: set[string]
+- http-header.bro now includes a global "include_header: set[string]" If it
-  (Robin Sommer).  If it contains any strings, then only those headers
+  contains any strings, then only those headers will be processed.  If left
-  will be processed.  If left empty, then you continue to get the current
+  empty, then you continue to get the current behavior of processing all
-  behavior of processing all headers.
+  headers. (Robin Sommer).  
 
 - Several changes to drop.bro (Robin Sommer):
 
-	* If true, the new flag Drop::dont_drop_locals indicates that
+	* If True, the new flag Drop::dont_drop_locals indicates that
 	  local hosts should never be dropped.  On by default.
 
-	* If true, the new flag Drop::debugging activates extensive debugging
+	* If True, the new flag Drop::debugging activates extensive debugging
 	  output for the catch-and-release logic.  Off by default.
 
 	* The timeout for tracking dropping information is now 1 day
@@ -39,14 +74,14 @@
   Sommer).
 
 - The HTTP analyzer no longer attempts to track Server/User-Agent
-  versions, as these are hugely voluminous (Seth Hall).  Ideally this
+  versions, as these are hugely voluminous (Seth Hall).
-  would still be available as an option for someone who truly wants
-  the full set.
 
 - HTTP and SMTP no longer have extra-short inactivity timeouts, as
   these were too often leading to premature expiration of a connection
   (Robin Sommer).
 
+- Tracking of HTTP refer[r]er's by setting log_referrer. (Vern Paxson).
+
 - The "rst" tool (aux/rst/) now takes an optional "-I <text>" argument
   that instructs it to inject <text> as payload rather than sending a RST
   packet (Vern Paxson).  <text> must be NUL-terminated, and the NUL is not
@@ -55,39 +90,113 @@
 - Bug fix for crashes in the DNS analyzer when processing replies for
   which no request was seen (Robin Sommer).
 
+- Addressed a number of lint nits (Vern Paxson).
 
-1.5.2.5 Mon Jul 19 16:20:58 PDT 2010
+- Rotation post-processors are now passed an additional argument
+  indicating whether Bro is terminating (Robin Sommer).
 
-- Removed now-quite-stale SSHv1 overflow detection, as it's more prone
+- Bro now consistently generates a file_opened event for all fopen() calls.
-  to false positives than useful detection (Vern Paxson).
+  (Robin Sommer).
 
+- The "cf" utility now ignores a leading "t=" prefix, for compatibility
+  with Bro's "tagged" logging format (Robin Sommer).
 
-1.5.2.4 Fri Jun  4 16:02:11 PDT 2010
+- You can now redefine the email_notice_to function (Robin Sommer).
 
-- Bug fixes for terminating connections (Tyler Schoenke and Vern Paxson).
+- Fix for packet processing resumption when a remote Bro dies during
+  state synchronization (Robin Sommer).
 
+- OpenSSL/X509 portability fix, at long last (Gregor Maier & Christian
+  Kreibich).
 
-1.5.2.3 Wed Mar 24 18:23:57 PDT 2010
+- Fix for compatibility with newer versions of autoconf (Gregor Maier).
 
-- Bug fixes for --enable-int64 and for avoiding bogus statistics /
+- A larger BroControl update (Robin Sommer, if not marked otherwise):
-  bad memory references when generating profiling information upon
-  exit (Vern Paxson).
 
+  o Increasing default timeouts for scan detector significantly.
 
-1.5.2.2 Tue Jan 12 12:33:42 PST 2010
+  o Increasing the manager's max_remote_events_processed to
+    something large, as it would slow down the process too much
+    otherwise and there's no other work to be interleaved with it
+    anyway.
 
-- Broccoli compiler warning fixes (Kevin Lo).
+  o Adding debug output to cluster's part of catch-and-release
+    (extends the debugging already present in policy/debug.bro)
 
+  o Fixing typo in util.py. Closes #223.
 
-1.5.2.1 Sun Jan 10 16:59:01 PST 2010
+  o Added note to README pointing to HTML version.
 
-- Bug fix for Active Mapping support (Kevin Lo).
+  o Disabling print_hook for proxies' remote.log.
 
+  o broctl's capstats now reports a total as well, and stats.log
+    tracks these totals. Closes #160.
 
-1.5.2 Sat Dec 26 18:38:37 PST 2009
+  o Avoiding spurious "waiting for lock" messages in cron mode.
+    Closes #206.
 
-- Portability fixes for --enable-int64 (Vern Paxson).
+  o Bug fixes for installation on NFS.
 
+  o Bug fix for top command on FreeBSD 8.
+
+  o crash-diag now checks whether gdb is available.
+
+  o trace-summary reports the sample factor in use in its output,
+    and now also applies it to the top-local-networks output (not
+    doing the latter was a bug).
+
+  o Removed the default twice-a-day rotation for conn.log. The
+    default rotation for conn.log now is now once every 24h, just
+    like for all other logs with the exception of mail.log (which is
+    still rotated twice a day, and thus the alarms are still mailed
+    out twice a day).
+
+  o Fixed the problem of logs sometimes being filed into the wrong
+    directory (see the (now gone) FAQ entry in the README).
+
+  o One can now customize the archive naming scheme. See the
+    corresponding FAQ entry in the README.
+
+  o Cleaned up, and extended, collection of cluster statistics.
+
+    ${logdir}/stats now looks like this:
+
+      drwxr-xr-x   4 bro  wheel      59392 Apr  5 17:55 .
+      drwxr-xr-x  96 bro  wheel       2560 Apr  6 12:00 ..
+      -rw-r--r--   1 bro  wheel        576 Apr  6 16:40 meta.dat
+      drwxr-xr-x   2 bro  wheel       2048 Apr  6 16:40 profiling
+      -rw-r--r--   1 bro  wheel  771834825 Apr  6 16:40 stats.log
+      drwxr-xr-x   2 bro  wheel       2048 Apr  6 16:25 www
+
+    stats.log accumulates cluster statistics collected every time
+    "cron" is called.
+
+    - profiling/ keeps the nodes' prof.logs.
+
+    - www/ keeps a subset of stats.log in CSV format for easy plotting.
+
+    - meta.dat contains meta information about the current cluster
+    state (in particular which nodes we have, and when the last
+    stats update was done).
+
+    Note that there is no Web setup yet to actually visualize the data in
+	www/.
+
+  o BroControl now automatically maintains links inside today's log
+    archive directory pointing to the current live version of the
+    corresponding log file (if Bro is running). For example:
+
+        smtp.log.11:52:18-current -> /usr/local/cluster/spool/manager/smtp.log
+
+  o Alarms mailed out by BroControl now (1) have the notice msg in the
+    subject; and (2) come with the full mail.log entry in the body.
+
+  o Fixing broctl's top output. (Seth Hall).
+
+  o Fixing broctl's df output in certain situations.
+
+  o BroControl fix for dealing with large vsize values reported by
+    "top" (Craig Leres).
 
 1.5.1 Fri Dec 18 15:17:12 PST 2009
 

Checklist-for-Release

@@ -30,6 +30,10 @@
 	bro-1.X-current.tar.gz
 	bro-1.<n>-release.tar.gz
 
+- Create symlink for HTTP:
+
+	/ftp/BROIDS/bro-XXX.tar.gz -> /www/BROIDS/download/bro-XXX.tar.gz
+
 - Update crd:/www/BROIDS/download.html to reflect new version.  This page
   is generated from trunk/bro-web/download.xml.  Edit this file, and also
   update the (web page) version in build.xml, the copyright year in

README

@@ -1,4 +1,4 @@
-This is release 1.5 of Bro, a system for detecting network intruders in
+This is release 1.5.3 of Bro, a system for detecting network intruders in
 real-time using passive network monitoring.
 
 Please see the file INSTALL for installation instructions and some examples

VERSION

@@ -1 +1 @@
-1.5.2.7
+1.5.3

acinclude.m4

@@ -60,26 +60,24 @@ AC_DEFUN([AC_LBL_TYPE_SIGNAL],
     esac]])
 
 dnl
-dnl Determine which compiler we're using (cc or gcc)
+dnl Do whatever AC_LBL_C_INIT work is necessary before using AC_PROG_CC.
-dnl If using gcc, determine the version number
-dnl If using cc, require that it support ansi prototypes
-dnl If using gcc, use -O2 (otherwise use -O)
-dnl If using cc, explicitly specify /usr/local/include
 dnl
-dnl usage:
+dnl It appears that newer versions of autoconf (2.64 and later) will,
+dnl if you use AC_TRY_COMPILE in a macro, stick AC_PROG_CC at the
+dnl beginning of the macro, even if the macro itself calls AC_PROG_CC.
+dnl See the "Prerequisite Macros" and "Expanded Before Required" sections
+dnl in the Autoconf documentation.
 dnl
-dnl	AC_LBL_C_INIT(copt, incls)
+dnl This causes a steaming heap of fail in our case, as we were, in
+dnl AC_LBL_C_INIT, doing the tests we now do in AC_LBL_C_INIT_BEFORE_CC,
+dnl calling AC_PROG_CC, and then doing the tests we now do in
+dnl AC_LBL_C_INIT.  Now, we run AC_LBL_C_INIT_BEFORE_CC, AC_PROG_CC,
+dnl and AC_LBL_C_INIT at the top level.
 dnl
-dnl results:
+dnl Borrowed from libpcap-1.1.1 by Gregor 
-dnl
+AC_DEFUN([AC_LBL_C_INIT_BEFORE_CC],
-dnl	$1 (copt set)
-dnl	$2 (incls set)
-dnl	CC
-dnl	LDFLAGS
-dnl	LBL_CFLAGS
-dnl
-AC_DEFUN([AC_LBL_C_INIT],
     [AC_PREREQ(2.12)
+    AC_BEFORE([$0], [AC_LBL_C_INIT])
     AC_BEFORE([$0], [AC_PROG_CC])
     AC_BEFORE([$0], [AC_LBL_FIXINCLUDES])
     AC_BEFORE([$0], [AC_LBL_DEVEL])
@@ -108,7 +106,31 @@ AC_DEFUN([AC_LBL_C_INIT],
 	    CC=cc
 	    export CC
     fi
-    AC_PROG_CC
+])
+
+dnl
+dnl Determine which compiler we're using (cc or gcc)
+dnl If using gcc, determine the version number
+dnl If using cc, require that it support ansi prototypes
+dnl If using gcc, use -O2 (otherwise use -O)
+dnl If using cc, explicitly specify /usr/local/include
+dnl
+dnl usage:
+dnl
+dnl	AC_LBL_C_INIT(copt, incls)
+dnl
+dnl results:
+dnl
+dnl	$1 (copt set)
+dnl	$2 (incls set)
+dnl	CC
+dnl	LDFLAGS
+dnl	LBL_CFLAGS
+dnl
+AC_DEFUN([AC_LBL_C_INIT],
+    [AC_PREREQ(2.12)
+    AC_BEFORE([$0], [AC_LBL_FIXINCLUDES])
+    AC_BEFORE([$0], [AC_LBL_DEVEL])
     if test "$GCC" != yes ; then
 	    AC_MSG_CHECKING(that $CC handles ansi prototypes)
 	    AC_CACHE_VAL(ac_cv_lbl_cc_ansi_prototypes,

aux/broccoli/ChangeLog

@@ -1,9 +1,13 @@
 Broccoli Changelog
 ========================================================================
 
-Tue Jan 12 12:32:12 PST 2010             Christian <christian@whoop.org>
+Wed Mar  2 15:38:02 PST 2011             Christian <christian@whoop.org>
 
-- Build warning fixes (Kevin Lo).
+- Accept empty strings ("") as values in the configuration file
+  (Craig Leres).
+- Support for specifying a separate host key for SSL-enabled operation,
+  with documentation update (Craig Leres).
+- Version bump to 1.5.3.
 
 ------------------------------------------------------------------------
 

aux/broccoli/configure.in

@@ -8,7 +8,7 @@ AC_CANONICAL_HOST
 
 AC_CONFIG_AUX_DIR(.)
 AM_CONFIG_HEADER(config.h)
-AM_INIT_AUTOMAKE(broccoli, 1.5.0)
+AM_INIT_AUTOMAKE(broccoli, 1.5.3)
 
 dnl Commands for funkier shell output:
 BLD_ON=`./shtool echo -n -e %B`

aux/broccoli/docs/broccoli-manual.sgml

@@ -1,7 +1,7 @@
 <!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook V4.2//EN" [
 <!ENTITY bc "<function>broccoli</function>">
 <!ENTITY bcc "<filename>broccoli-config</filename>">
-<!ENTITY bc-latest-rel "1.5">
+<!ENTITY bc-latest-rel "1.5.3">
 <!ENTITY bc-header SYSTEM "sgml/broccoli.sgml">
 <!ENTITY bp "<function>broping</function>">
 ]>
@@ -19,7 +19,7 @@
     <abstract>
       <para>
 	This is documentation for release <emphasis>&bc-latest-rel;</emphasis>
-	of Broccoli, compatible with Bro IDS releases of <emphasis>1.4</emphasis>
+	of Broccoli, compatible with Bro IDS releases of <emphasis>1.5</emphasis>
 	or newer. Broccoli is free software under terms of the BSD license as given
 	in the <link linkend="license" endterm="license.title">License</link>
 	section. This documentation is always available on the web for download
@@ -1531,6 +1531,8 @@ Bar/SomeLongStr       "Hello World"
 	need to put the CA certificate and the peer certificate in the
 	<varname>/broccoli/ca_cert</varname> and 
 	<varname>/broccoli/host_cert</varname> keys, respectively, in the configuration file.
+	Optionally, you can store the private key in a separate file specified by
+	<varname>/broccoli/host_key</varname>. 
 	To quickly enable/disable a certificate configuration, the
 	<varname>/broccoli/use_ssl</varname> key can be used.
 	<caution>
@@ -1555,6 +1557,7 @@ Bar/SomeLongStr       "Hello World"
 /broccoli/use_ssl	   yes
 /broccoli/ca_cert          <path>/ca_cert.pem
 /broccoli/host_cert        <path>/bro_cert.pem
+/broccoli/host_key         <path>/bro_cert.key
 ]]>
       </programlisting>
       <para>

aux/broccoli/docs/html/a3621.html

@@ -1,210 +0,0 @@
-<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
-<HTML
-><HEAD
-><TITLE
->Appendix</TITLE
-><META
-NAME="GENERATOR"
-CONTENT="Modular DocBook HTML Stylesheet Version 1.7"><LINK
-REL="HOME"
-TITLE="Broccoli: The Bro Client Communications Library"
-HREF="index.html"><LINK
-REL="PREVIOUS"
-TITLE="broccoli"
-HREF="broccoli-broccoli.html"><LINK
-REL="STYLESHEET"
-TYPE="text/css"
-HREF="stylesheet.css"></HEAD
-><BODY
-CLASS="APPENDIX"
-BGCOLOR="#FFFFFF"
-TEXT="#000000"
-LINK="#0000FF"
-VLINK="#840084"
-ALINK="#0000FF"
-><DIV
-CLASS="NAVHEADER"
-><TABLE
-SUMMARY="Header navigation table"
-WIDTH="100%"
-BORDER="0"
-CELLPADDING="0"
-CELLSPACING="0"
-><TR
-><TH
-COLSPAN="3"
-ALIGN="center"
->Broccoli: The Bro Client Communications Library</TH
-></TR
-><TR
-><TD
-WIDTH="10%"
-ALIGN="left"
-VALIGN="bottom"
-><A
-HREF="broccoli-broccoli.html"
-ACCESSKEY="P"
->Prev</A
-></TD
-><TD
-WIDTH="80%"
-ALIGN="center"
-VALIGN="bottom"
-></TD
-><TD
-WIDTH="10%"
-ALIGN="right"
-VALIGN="bottom"
->&nbsp;</TD
-></TR
-></TABLE
-><HR
-ALIGN="LEFT"
-WIDTH="100%"></DIV
-><DIV
-CLASS="APPENDIX"
-><H1
-><A
-NAME="AEN3621"
-></A
->Appendix A. Appendix</H1
-><DIV
-CLASS="TOC"
-><DL
-><DT
-><B
->Table of Contents</B
-></DT
-><DT
->A.1. <A
-HREF="a3621.html#LICENSE"
->License</A
-></DT
-><DT
->A.2. <A
-HREF="a3621.html#ABOUT"
->About this document</A
-></DT
-></DL
-></DIV
-><BR
-CLEAR="all"><DIV
-CLASS="SECT1"
-><H1
-CLASS="SECT1"
-><A
-NAME="LICENSE"
->A.1. License</A
-></H1
-><P
->	Copyright (C) 2004-2008 Christian Kreibich and various contributors.
-      </P
-><P
->	
-	Permission is hereby granted, free of charge, to any person obtaining a copy
-	of this software and associated documentation files (the "Software"), to
-	deal in the Software without restriction, including without limitation the
-	rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
-	sell copies of the Software, and to permit persons to whom the Software is
-	furnished to do so, subject to the following conditions:
-      </P
-><P
->	The above copyright notice and this permission notice shall be included in
-	all copies of the Software and its documentation and acknowledgment shall be
-	given in the documentation and software packages that this Software was
-	used.
-      </P
-><P
->	
-	THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-	IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-	FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
-	THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
-	IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-	CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.	
-      </P
-></DIV
-><BR
-CLEAR="all"><DIV
-CLASS="SECT1"
-><H1
-CLASS="SECT1"
-><A
-NAME="ABOUT"
->A.2. About this document</A
-></H1
-><P
->	This documentation is maintained in SGML <A
-HREF="http://www.docbook.org"
-TARGET="_top"
->DocBook</A
->,
-	API documentation is extracted from the code using the
-	<A
-HREF="http://www.gtk.org/gtk-doc/"
-TARGET="_top"
-><B
-CLASS="COMMAND"
->gtk-doc</B
-></A
-> tools.
-      </P
-></DIV
-></DIV
-><DIV
-CLASS="NAVFOOTER"
-><HR
-ALIGN="LEFT"
-WIDTH="100%"><TABLE
-SUMMARY="Footer navigation table"
-WIDTH="100%"
-BORDER="0"
-CELLPADDING="0"
-CELLSPACING="0"
-><TR
-><TD
-WIDTH="33%"
-ALIGN="left"
-VALIGN="top"
-><A
-HREF="broccoli-broccoli.html"
-ACCESSKEY="P"
->Prev</A
-></TD
-><TD
-WIDTH="34%"
-ALIGN="center"
-VALIGN="top"
-><A
-HREF="index.html"
-ACCESSKEY="H"
->Home</A
-></TD
-><TD
-WIDTH="33%"
-ALIGN="right"
-VALIGN="top"
->&nbsp;</TD
-></TR
-><TR
-><TD
-WIDTH="33%"
-ALIGN="left"
-VALIGN="top"
->broccoli</TD
-><TD
-WIDTH="34%"
-ALIGN="center"
-VALIGN="top"
->&nbsp;</TD
-><TD
-WIDTH="33%"
-ALIGN="right"
-VALIGN="top"
->&nbsp;</TD
-></TR
-></TABLE
-></DIV
-></BODY
-></HTML
->

aux/broccoli/docs/html/a3638.html

@@ -0,0 +1,210 @@
+<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
+<HTML
+><HEAD
+><TITLE
+>Appendix</TITLE
+><META
+NAME="GENERATOR"
+CONTENT="Modular DocBook HTML Stylesheet Version 1.79"><LINK
+REL="HOME"
+TITLE="Broccoli: The Bro Client Communications Library"
+HREF="index.html"><LINK
+REL="PREVIOUS"
+TITLE="broccoli"
+HREF="broccoli-broccoli.html"><LINK
+REL="STYLESHEET"
+TYPE="text/css"
+HREF="stylesheet.css"></HEAD
+><BODY
+CLASS="APPENDIX"
+BGCOLOR="#FFFFFF"
+TEXT="#000000"
+LINK="#0000FF"
+VLINK="#840084"
+ALINK="#0000FF"
+><DIV
+CLASS="NAVHEADER"
+><TABLE
+SUMMARY="Header navigation table"
+WIDTH="100%"
+BORDER="0"
+CELLPADDING="0"
+CELLSPACING="0"
+><TR
+><TH
+COLSPAN="3"
+ALIGN="center"
+>Broccoli: The Bro Client Communications Library</TH
+></TR
+><TR
+><TD
+WIDTH="10%"
+ALIGN="left"
+VALIGN="bottom"
+><A
+HREF="broccoli-broccoli.html"
+ACCESSKEY="P"
+>Prev</A
+></TD
+><TD
+WIDTH="80%"
+ALIGN="center"
+VALIGN="bottom"
+></TD
+><TD
+WIDTH="10%"
+ALIGN="right"
+VALIGN="bottom"
+>&nbsp;</TD
+></TR
+></TABLE
+><HR
+ALIGN="LEFT"
+WIDTH="100%"></DIV
+><DIV
+CLASS="APPENDIX"
+><H1
+><A
+NAME="AEN3638"
+></A
+>Appendix A. Appendix</H1
+><DIV
+CLASS="TOC"
+><DL
+><DT
+><B
+>Table of Contents</B
+></DT
+><DT
+>A.1. <A
+HREF="a3638.html#LICENSE"
+>License</A
+></DT
+><DT
+>A.2. <A
+HREF="a3638.html#ABOUT"
+>About this document</A
+></DT
+></DL
+></DIV
+><BR
+CLEAR="all"><DIV
+CLASS="SECT1"
+><H1
+CLASS="SECT1"
+><A
+NAME="LICENSE"
+>A.1. License</A
+></H1
+><P
+>	Copyright (C) 2004-2008 Christian Kreibich and various contributors.
+      </P
+><P
+>	
+	Permission is hereby granted, free of charge, to any person obtaining a copy
+	of this software and associated documentation files (the "Software"), to
+	deal in the Software without restriction, including without limitation the
+	rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+	sell copies of the Software, and to permit persons to whom the Software is
+	furnished to do so, subject to the following conditions:
+      </P
+><P
+>	The above copyright notice and this permission notice shall be included in
+	all copies of the Software and its documentation and acknowledgment shall be
+	given in the documentation and software packages that this Software was
+	used.
+      </P
+><P
+>	
+	THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+	IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+	FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+	THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
+	IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
+	CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.	
+      </P
+></DIV
+><BR
+CLEAR="all"><DIV
+CLASS="SECT1"
+><H1
+CLASS="SECT1"
+><A
+NAME="ABOUT"
+>A.2. About this document</A
+></H1
+><P
+>	This documentation is maintained in SGML <A
+HREF="http://www.docbook.org"
+TARGET="_top"
+>DocBook</A
+>,
+	API documentation is extracted from the code using the
+	<A
+HREF="http://www.gtk.org/gtk-doc/"
+TARGET="_top"
+><B
+CLASS="COMMAND"
+>gtk-doc</B
+></A
+> tools.
+      </P
+></DIV
+></DIV
+><DIV
+CLASS="NAVFOOTER"
+><HR
+ALIGN="LEFT"
+WIDTH="100%"><TABLE
+SUMMARY="Footer navigation table"
+WIDTH="100%"
+BORDER="0"
+CELLPADDING="0"
+CELLSPACING="0"
+><TR
+><TD
+WIDTH="33%"
+ALIGN="left"
+VALIGN="top"
+><A
+HREF="broccoli-broccoli.html"
+ACCESSKEY="P"
+>Prev</A
+></TD
+><TD
+WIDTH="34%"
+ALIGN="center"
+VALIGN="top"
+><A
+HREF="index.html"
+ACCESSKEY="H"
+>Home</A
+></TD
+><TD
+WIDTH="33%"
+ALIGN="right"
+VALIGN="top"
+>&nbsp;</TD
+></TR
+><TR
+><TD
+WIDTH="33%"
+ALIGN="left"
+VALIGN="top"
+>broccoli</TD
+><TD
+WIDTH="34%"
+ALIGN="center"
+VALIGN="top"
+>&nbsp;</TD
+><TD
+WIDTH="33%"
+ALIGN="right"
+VALIGN="top"
+>&nbsp;</TD
+></TR
+></TABLE
+></DIV
+></BODY
+></HTML
+>

aux/broccoli/docs/html/c54.html

@@ -107,7 +107,7 @@ CLASS="COMMAND"
 >--enable-debug</B
 >: enables debugging output.
           Please refer to the <A
-HREF="c84.html#AEN818"
+HREF="c84.html#AEN819"
 >Broccoli debugging</A
 >
 	  section for details on configuring and using debugging output.

aux/broccoli/docs/html/c84.html

@@ -107,17 +107,17 @@ HREF="c84.html#AEN738"
 ></DT
 ><DT
 >3.5. <A
-HREF="c84.html#AEN784"
+HREF="c84.html#AEN785"
 >Configuring event reception in Bro policies</A
 ></DT
 ><DT
 >3.6. <A
-HREF="c84.html#AEN818"
+HREF="c84.html#AEN819"
 >Configuring debugging output</A
 ></DT
 ><DT
 >3.7. <A
-HREF="c84.html#AEN842"
+HREF="c84.html#AEN843"
 >Test programs</A
 ></DT
 ></DL
@@ -1169,7 +1169,7 @@ CLASS="FUNCTION"
 >Remote::destinations</CODE
 > configuration.
 	    See <A
-HREF="c84.html#AEN784"
+HREF="c84.html#AEN785"
 >below</A
 > for how to do this.
 	    Finally, in order to obtain the class of a connection as indicated by the remote side, use
@@ -1230,7 +1230,7 @@ CLASS="EMPHASIS"
 >	    You need to make sure that the remote Bro agent is interested in receiving
 	    the events you send. This interest is expressed in policy configuration.
 	    We'll explain this in more detail <A
-HREF="c84.html#AEN784"
+HREF="c84.html#AEN785"
 >below</A
 >
 	    and for now assume that our remote peer is configured to receive the
@@ -3351,6 +3351,11 @@ CLASS="VARNAME"
 CLASS="VARNAME"
 >/broccoli/host_cert</CODE
 > keys, respectively, in the configuration file.
+	Optionally, you can store the private key in a separate file specified by
+	<CODE
+CLASS="VARNAME"
+>/broccoli/host_key</CODE
+>. 
 	To quickly enable/disable a certificate configuration, the
 	<CODE
 CLASS="VARNAME"
@@ -3429,6 +3434,7 @@ CLASS="PROGRAMLISTING"
 >/broccoli/use_ssl	   yes
 /broccoli/ca_cert          &#60;path&#62;/ca_cert.pem
 /broccoli/host_cert        &#60;path&#62;/bro_cert.pem
+/broccoli/host_key         &#60;path&#62;/bro_cert.key
       </PRE
 ></TD
 ></TR
@@ -3533,7 +3539,7 @@ CLASS="SECT1"
 ><H1
 CLASS="SECT1"
 ><A
-NAME="AEN784"
+NAME="AEN785"
 >3.5. Configuring event reception in Bro policies</A
 ></H1
 ><P
@@ -3690,7 +3696,7 @@ CLASS="COMMAND"
 >broping</B
 > tool
 	      explained in the <A
-HREF="c84.html#AEN842"
+HREF="c84.html#AEN843"
 >section on testing</A
 > below.
 	      It will allow an agent on the local host to connect and send "ping" events.
@@ -3708,7 +3714,7 @@ CLASS="SECT1"
 ><H1
 CLASS="SECT1"
 ><A
-NAME="AEN818"
+NAME="AEN819"
 >3.6. Configuring debugging output</A
 ></H1
 ><P
@@ -3804,7 +3810,7 @@ CLASS="SECT1"
 ><H1
 CLASS="SECT1"
 ><A
-NAME="AEN842"
+NAME="AEN843"
 >3.7. Test programs</A
 ></H1
 ><P
@@ -3818,8 +3824,8 @@ CLASS="FUNCTION"
 >broping</CODE
 >
 	  <A
-NAME="AEN847"
+NAME="AEN848"
-HREF="#FTN.AEN847"
+HREF="#FTN.AEN848"
 ><SPAN
 CLASS="footnote"
 >[2]</SPAN
@@ -3958,8 +3964,8 @@ ALIGN="LEFT"
 VALIGN="TOP"
 WIDTH="5%"
 ><A
-NAME="FTN.AEN847"
+NAME="FTN.AEN848"
-HREF="c84.html#AEN847"
+HREF="c84.html#AEN848"
 ><SPAN
 CLASS="footnote"
 >[2]</SPAN

aux/broccoli/docs/html/index.html

@@ -49,7 +49,7 @@ NAME="AEN9"
 CLASS="emphasis"
 ><B
 CLASS="EMPHASIS"
->1.5</B
+>1.5.3</B
 ></SPAN
 >
 	of Broccoli, compatible with Bro IDS releases of <SPAN
@@ -61,7 +61,7 @@ CLASS="EMPHASIS"
 >
 	or newer. Broccoli is free software under terms of the BSD license as given
 	in the <A
-HREF="a3637.html#LICENSE"
+HREF="a3638.html#LICENSE"
 >License</A
 >
 	section. This documentation is always available on the web for download
@@ -225,17 +225,17 @@ HREF="c84.html#AEN738"
 ></DT
 ><DT
 >3.5. <A
-HREF="c84.html#AEN784"
+HREF="c84.html#AEN785"
 >Configuring event reception in Bro policies</A
 ></DT
 ><DT
 >3.6. <A
-HREF="c84.html#AEN818"
+HREF="c84.html#AEN819"
 >Configuring debugging output</A
 ></DT
 ><DT
 >3.7. <A
-HREF="c84.html#AEN842"
+HREF="c84.html#AEN843"
 >Test programs</A
 ></DT
 ></DL
@@ -256,19 +256,19 @@ HREF="broccoli-broccoli.html"
 ></DD
 ><DT
 >A. <A
-HREF="a3637.html"
+HREF="a3638.html"
 >Appendix</A
 ></DT
 ><DD
 ><DL
 ><DT
 >A.1. <A
-HREF="a3637.html#LICENSE"
+HREF="a3638.html#LICENSE"
 >License</A
 ></DT
 ><DT
 >A.2. <A
-HREF="a3637.html#ABOUT"
+HREF="a3638.html#ABOUT"
 >About this document</A
 ></DT
 ></DL

aux/broccoli/src/bro_lexer.l

@@ -27,11 +27,11 @@ no|false|off    	 { yylval.i = 0; return BROINT; }
 [ \t]+           	 ;
 [0-9]+                   { yylval.i = strtol(yytext, NULL, 10); return BROINT; }
 [0-9]+\.[0-9]+           { yylval.d = strtod(yytext, NULL); return BRODOUBLE; }
-[[:alnum:][:punct:]]+    { yylval.s = strdup(yytext); return BROWORD; }
 \".*\"                   { yylval.s = strdup(yytext+1);
                            yylval.s[strlen(yylval.s) - 1] = '\0';
                            return BROSTRING;
                          }
+[[:alnum:][:punct:]]+    { yylval.s = strdup(yytext); return BROWORD; }
 
 "#".*\n                  { bro_parse_lineno++; }
 "//".*\n                 { bro_parse_lineno++; }

aux/broccoli/src/bro_openssl.c

@@ -244,7 +244,7 @@ __bro_openssl_init(void)
 {
   static int deja_vu = FALSE;
   int use_ssl = FALSE;
-  const char *our_cert, *our_pass, *ca_cert;
+  const char *our_cert, *our_key, *our_pass, *ca_cert;
   
   D_ENTER;
 
@@ -284,7 +284,15 @@ __bro_openssl_init(void)
       D_RETURN_(TRUE);
     }
 
-  if (! (our_cert = __bro_conf_get_str("/broccoli/host_cert")))
+  our_cert = __bro_conf_get_str("/broccoli/host_cert");
+  our_key = __bro_conf_get_str("/broccoli/host_key");
+  if (our_key == NULL)
+    {
+    /* No private key configured; get it from the certificate file */
+    our_key = our_cert;
+    }
+
+  if (our_cert == NULL)
     {
       if (use_ssl)
 	{
@@ -298,6 +306,21 @@ __bro_openssl_init(void)
 	}
     }
 
+  if (our_key == NULL)
+    {
+      if (use_ssl)
+       {
+         D(("SSL requested but host key not given -- aborting.\n"));
+         D_RETURN_(FALSE);
+       }
+      else
+       {
+         D(("use_ssl not used and host key not given -- not using SSL.\n"));
+         D_RETURN_(TRUE);
+       }
+    }
+
+
   /* At this point we either haven't seen use_ssl but a host_cert, or
    * we have seen use_ssl and it is set to true. Either way, we attempt
    * to set up an SSL connection now and abort if this fails in any way.
@@ -326,9 +349,9 @@ __bro_openssl_init(void)
       SSL_CTX_set_default_passwd_cb_userdata(ctx, (void *) our_pass);
     }
   
-  if (SSL_CTX_use_PrivateKey_file(ctx, our_cert, SSL_FILETYPE_PEM) != 1)
+  if (SSL_CTX_use_PrivateKey_file(ctx, our_key, SSL_FILETYPE_PEM) != 1)
     {
-      D(("SSL used but error loading private key from '%s' -- aborting.\n", our_cert));
+      D(("SSL used but error loading private key from '%s' -- aborting.\n", our_key));
       goto error_return;
     }
   
@@ -356,6 +379,13 @@ __bro_openssl_init(void)
       goto error_return;
     }
   
+  /* Check the consistency of the certificate vs. the private key */
+  if (SSL_CTX_check_private_key(ctx) != 1)
+    {
+      D(("SSL used but private key does not match the certificate -- aborting\n"));
+      goto error_return;
+    }
+  
   /* Only use real ciphers.
    */
   if (! SSL_CTX_set_cipher_list(ctx, "HIGH"))

aux/broctl/BroControl/control.py

@@ -585,7 +585,7 @@ def getTopOutput(nodes):
             d = {}
             d["pid"] = int(p[0])
             d["proc"] = (p[0] == parents[node.tag] and "parent" or "child") 
-            d["vsize"] = int(p[1])
+            d["vsize"] = int(float(p[1]))
             d["rss"] = int(p[2])
             d["cpu"] = p[3]
             d["cmd"] = " ".join(p[4:])
@@ -761,6 +761,9 @@ def attachGdb(nodes):
 #
 # Tags are those as returned by capstats on the command-line
 #
+# There is one "pseudo-node" of the name "$total" with the sum of all
+# individual values. 
+#
 # We do all the stuff in parallel across all nodes which is why this looks 
 # a bit confusing ...
 
@@ -798,6 +801,8 @@ def getCapstatsOutput(nodes, interval):
 
     outputs = execute.runHelperParallel(cmds) 
 
+    totals = {}
+    
     for (node, success, output) in outputs:
 
         if not success:
@@ -810,13 +815,22 @@ def getCapstatsOutput(nodes, interval):
         try:
             for field in fields[1:]:
                 (key, val) = field.split("=")
-                vals[key] = float(val)
+                val = float(val)
+                vals[key] = val
+                
+                try:
+                    totals[key] += val
+                except KeyError:
+                    totals[key] = val
 
             results += [(node, None, vals)]
 
         except ValueError:
             results += [(node, "%s: unexpected capstats output: %s" % (node.tag, output[0]), {})]
             
+    # Add pseudo-node for totals
+    results += [(config.Node("$total"), None, totals)]
+
     return results
 
 # Get current statistics from cFlow. 
@@ -861,16 +875,9 @@ def calculateCFlowRate(start, stop, interval):
 def capstats(nodes, interval):
 
     def output(tag, data):
-        util.output("\n%-12s %-10s %-10s (%ds average)" % (tag, "kpps", "mbps", interval))
-        util.output("-" * 30)
         
-        for (port, error, vals) in data:
+        def outputOne(tag, vals):
-
+            util.output("%-12s " % tag, nl=False)
-            if error:
-                util.output(error)
-                continue
-
-            util.output("%-12s " % port, nl=False)
 
             if not error:
                 util.output("%-10s " % vals["kpps"], nl=False)
@@ -880,6 +887,27 @@ def capstats(nodes, interval):
             else:
                 util.output("<%s> " % error)
         
+        util.output("\n%-12s %-10s %-10s (%ds average)" % (tag, "kpps", "mbps", interval))
+        util.output("-" * 30)
+
+        totals = None
+        
+        for (port, error, vals) in data:
+
+            if error:
+                util.output(error)
+                continue
+
+            if str(port) != "$total":
+                outputOne(port, vals)
+            else:
+                totals = vals
+                
+        if totals:
+            util.output("")
+            outputOne("Total", totals)
+            util.output("")
+
     have_cflow = config.Config.cflowaddress and config.Config.cflowuser and config.Config.cflowpassword
     have_capstats = config.Config.capstats
 
@@ -960,6 +988,11 @@ def getDf(nodes):
 
         cmds = []
         for node in nodes:
+            
+            if dir == "logdir" and node.type != "manager":
+                # Don't need this on the workers/proxies.
+                continue
+            
             cmds += [(node, "df", [path])]
 
         results = execute.runHelperParallel(cmds)

aux/broctl/BroControl/cron.py

@@ -20,11 +20,12 @@ def doCron():
     if config.Config.cronenabled == "0":
         return
 
+    config.Config.config["cron"] = "1"  # Flag to indicate that we're running from cron.
+    
     if not util.lock():
         return
 
     util.bufferOutput()
-    config.Config.config["cron"] = "1"  # Flag to indicate that we're running from cron.
 
     # Check whether nodes are still running an restart if neccessary.
     for (node, isrunning) in control.isRunning(config.Config.nodes()):
@@ -55,10 +56,10 @@ def doCron():
     if output:
         util.sendMail("cron: " + output.split("\n")[0], output)
 
-    config.Config.config["cron"] = "0"
-
     util.unlock()
 
+    config.Config.config["cron"] = "0"
+    
 def logAction(node, action):
     t = time.time()
     out = open(config.Config.statslog, "a")
@@ -191,15 +192,6 @@ def _checkHosts():
         config.Config._setState(tag, alive)
 
 def _getProfLogs():        
-
-    dir = config.Config.statsdir
-    if not os.path.exists(dir):
-        os.mkdir(dir)
-
-    if not os.path.exists(dir) or not os.path.isdir(dir):
-        util.output("cannot create directory %s" % dir)
-        return
-
     cmds = []
 
     for node in config.Config.hosts():
@@ -211,14 +203,10 @@ def _getProfLogs():
             util.output("cannot get prof.log from %s" % node.tag)
 
 def _updateHTTPStats():
-
     # Get the prof.logs.
     _getProfLogs()
 
-    # Copy stats.dat.
+    # Create meta file. 
-    shutil.copy(config.Config.statslog, config.Config.statsdir)
-
-    # Creat meta file. 
     meta = open(os.path.join(config.Config.statsdir, "meta.dat"), "w")
     for node in config.Config.hosts():
         print >>meta, "node", node.tag, node.type, node.host
@@ -238,5 +226,12 @@ def _updateHTTPStats():
 
     meta.close()
 
+    # Run the update-stats script.
+    (success, output) = execute.runLocalCmd(os.path.join(config.Config.scriptsdir, "update-stats"))
+    
+    if not success:
+        util.output("error running update-stats\n\n")
+        util.output(output)
+    
 
 

aux/broctl/BroControl/execute.py

@@ -79,7 +79,7 @@ def mkdirs(dirs):
         else:
             cmds += [(node, [], [])]
                 # Need to be careful here as our helper scripts may not be installed yet. 
-            fullcmds += [("test -d %s || mkdir %s 2>/dev/null; echo $?; echo ~~~" % (dir, dir))]
+            fullcmds += [("test -d %s || mkdir -p %s 2>/dev/null; echo $?; echo ~~~" % (dir, dir))]
 
     for (node, success, output) in runHelperParallel(cmds, fullcmds=fullcmds):
         results += [(node, success)]
@@ -147,7 +147,14 @@ def install(host, src, dst):
             os.remove(dst)
 
         util.debug(1, "cp %s %s" % (src, dst))
-        shutil.copy2(src, dst)
+        
+        try:
+            shutil.copy2(src, dst)
+        except OSError:
+            # Python 2.6 has a bug where this may fail on NFS. So we just
+            # ignore errors.
+            pass
+            
         return True
     else:
         util.error("install() not yet supported for remote hosts")

aux/broctl/BroControl/install.py

@@ -72,6 +72,7 @@ Targets = [
     ("${distdir}/aux/broctl/bin/delete-log", "${scriptsdir}", True),
     ("${distdir}/aux/broctl/bin/expire-logs.in", "${scriptsdir}", True),
     ("${distdir}/aux/broctl/bin/post-terminate.in", "${scriptsdir}", True),
+    ("${distdir}/aux/broctl/bin/stat-ctime", "${scriptsdir}", True),
     ("${distdir}/aux/broctl/bin/crash-diag.in", "${scriptsdir}", True),
     ("${distdir}/aux/broctl/bin/send-mail.in", "${scriptsdir}", True),
     ("${distdir}/aux/broctl/bin/mail-alarm.in", "${scriptsdir}", True),
@@ -82,6 +83,12 @@ Targets = [
     ("${distdir}/aux/broctl/bin/cflow-stats.in", "${scriptsdir}", True),
     ("${distdir}/aux/broctl/bin/get-prof-log.in", "${scriptsdir}", True),
     ("${distdir}/aux/broctl/bin/mail-contents.in", "${scriptsdir}", True),
+    ("${distdir}/aux/broctl/bin/make-archive-name", "${scriptsdir}", True),
+    ("${distdir}/aux/broctl/bin/create-link-for-log.in", "${scriptsdir}", True),
+    ("${distdir}/aux/broctl/bin/remove-link-for-log.in", "${scriptsdir}", True),
+    ("${distdir}/aux/broctl/bin/update-stats.in", "${scriptsdir}", True),
+    ("${distdir}/aux/broctl/bin/stats-to-csv", "${scriptsdir}", True),
+    ("${distdir}/aux/broctl/bin/fmt-time", "${scriptsdir}", True),
     ("${distdir}/aux/broctl/bin/helpers/start.in", "${helperdir}", True),
     ("${distdir}/aux/broctl/bin/helpers/stop", "${helperdir}", True),
     ("${distdir}/aux/broctl/bin/helpers/check-pid", "${helperdir}", True),
@@ -318,7 +325,7 @@ def install(local_only, make_install):
         try:
             os.symlink(manager.cwd(), current)
         except (IOError, OSError), e:
-            util.warn("cannot link %s to %s: %s" % (manager.cwd(), current, e))
+            pass
 
     if local_only:
         return
@@ -362,17 +369,21 @@ def install(local_only, make_install):
         # already take care of that.
 
     else:
-        # NFS. We only need to take care of the spool/log directoryies.
+        # NFS. We only need to take care of the spool/log directories.
         paths = [config.Config.spooldir]
-        paths += [config.Config.logdir]
+        paths += [config.Config.tmpdir]
 
         dirs = []
         for dir in paths:
             dirs += [(n, dir) for n in nodes]
 
+        # We need this only on the manager.
+        dirs += [(manager, config.Config.logdir)]
+            
         for (node, success) in execute.mkdirs(dirs):
             if not success:
-                util.warn("cannot create directory on %s" % (dir, node.tag))
+                util.warn("cannot create (some of the) directories %s on %s" % (",".join(paths), node.tag))
+            
         util.output("done.")
 
 # Create Bro-side broctl configuration broctl-layout.bro.        

aux/broctl/BroControl/options.py

@@ -33,6 +33,8 @@ options = [
 
     Option("LogDir", "${BroBase}/logs", "string", Option.USER, False,
            "Directory for archived log files."),
+    Option("MakeArchiveName", "${BroBase}/share/broctl/scripts/make-archive-name", "string", Option.USER, False,
+           "Script to generate filenames for archived log files."),
            
     Option("SendMail", "1", "bool", Option.USER, False,
            "True if shell may send mails."),

aux/broctl/BroControl/util.py

@@ -177,7 +177,7 @@ def lock():
         else:
             do_output = 2
             
-        if do_ouput:
+        if do_output:
             output("waiting for lock ...", nl=False)
 
         count = 0

aux/broctl/README

@@ -2,7 +2,11 @@
 //
 // $Id: README 6948 2009-12-03 20:59:41Z robin $
 //
-// FIXME: This needs asciidoc 8.2.x plus some custom config files.
+// NOTE: This README contains only parts of the BroControl documentation.
+// Please see README.html for the complete document.
+
+// (To generate the HTML version, one needs asciidoc 8.2.x plus some custom
+// config files.)
 
 BroControl
 ===========
@@ -223,7 +227,7 @@ expects commands on its command-line (alternatively, +broctl+ can
 also be started with a single command directly on the shell's
 command line):
 
-  > cluster
+  > broctl
   Welcome to BroControl 0.2
 
   Type "help" for help.
@@ -425,17 +429,6 @@ Note for folks who have used the old "cluster shell": the
 development mode corresponds to the old default behaviour, which
 worked with any +make install-broctl+.
 
-After a Bro crash, the timestamps of the archived log files sometimes seem to be wrong???
-   When Bro crashes, broctl archives the log files produced so far
-   at the normal location. However, for some files it can't (easily)
-   determine the right timestamps to put into the filename. This
-   affects in particular those log files that are not rotated on
-   regular basis (e.g., +stdout.log+, +prof.log+); their filenames
-   will indicate as their start time the point when all the other
-   files were _rotated_ most recently. In addition, for all log
-   files, after a crash the start/end times indicated by the file
-   names might be off a few seconds. 
-
 [[devversion]]Anything special to consider when using development versions???
    If you are using a _development version_, _BroControl_ might
    require patching Bro itself to work correctly. A "development
@@ -448,3 +441,11 @@ After a Bro crash, the timestamps of the archived log files sometimes seem to be
    > cd /path/to/bro/source/distribution
    > patch -p0 <aux/broctl/patch-bro.diff
    > ./autogen.sh
+
+Can I change the naming scheme that BroControl uses for archived log files?  
+    Yes, set xref:opt_MakeArchiveName[+MakeArchiveName+]+ to a
+    script that outputs the desired destination file name for an
+    archived log file. The default script for that task is
+    +<BroBase>/share/broctl/scripts/make-archive-name+, which you
+    can use that as a template for creating your own version. See
+    the beginning of that script for instructions. 

aux/broctl/README.html

@@ -764,7 +764,7 @@ also be started with a single command directly on the shell's
 command line):</p>
 <div class="literalblock">
 <div class="content">
-<pre><tt>&gt; cluster
+<pre><tt>&gt; broctl
 Welcome to BroControl 0.2</tt></pre>
 </div></div>
 <div class="literalblock">
@@ -1454,6 +1454,14 @@ Destination address for broctl-generated non-alarm mails. Default is to use the
 </p>
 </dd>
 <dt>
+<a id="opt_MakeArchiveName"></a> <strong>MakeArchiveName</strong> (string, default "${BroBase}/share/broctl/scripts/make-archive-name")
+</dt>
+<dd>
+<p>
+Script to generate filenames for archived log files.
+</p>
+</dd>
+<dt>
 <a id="opt_MemLimit"></a> <strong>MemLimit</strong> (string, default "unlimited")
 </dt>
 <dd>
@@ -1941,22 +1949,6 @@ worked with any <tt>make install-broctl</tt>.</p>
 </li>
 <li>
 <p><em>
-After a Bro crash, the timestamps of the archived log files sometimes seem to be wrong?
-</em></p>
-<p>
-   When Bro crashes, broctl archives the log files produced so far
-   at the normal location. However, for some files it can't (easily)
-   determine the right timestamps to put into the filename. This
-   affects in particular those log files that are not rotated on
-   regular basis (e.g., <tt>stdout.log</tt>, <tt>prof.log</tt>); their filenames
-   will indicate as their start time the point when all the other
-   files were <em>rotated</em> most recently. In addition, for all log
-   files, after a crash the start/end times indicated by the file
-   names might be off a few seconds.
-</p>
-</li>
-<li>
-<p><em>
 <a id="devversion"></a>Anything special to consider when using development versions?
 </em></p>
 <p>
@@ -1976,10 +1968,17 @@ After a Bro crash, the timestamps of the archived log files sometimes seem to be
 </div></div>
 </li>
 </ol>
+<p>Can I change the naming scheme that BroControl uses for archived log files?
+    Yes, set <a href="#opt_MakeArchiveName"><tt>MakeArchiveName</tt></a><tt> to a
+    script that outputs the desired destination file name for an
+    archived log file. The default script for that task is
+    </tt>&lt;BroBase&gt;/share/broctl/scripts/make-archive-name+, which you
+    can use that as a template for creating your own version. See
+    the beginning of that script for instructions.</p>
 </div>
 <div id="footer">
 <div id="footer-text">
-Last modified at 2009-12-03 12:58:36 PDT - Robin Sommer
+Last modified at 2010-10-18 16:49:08 PDT - Robin Sommer
 </div>
 </div>
 </body>

aux/broctl/README.options

@@ -46,6 +46,8 @@ Reply-to address for broctl-generated mails.
 General Subject prefix for broctl-generated mails.
 [[opt_MailTo]] *MailTo* (string, default "<user>")::
 Destination address for broctl-generated non-alarm mails. Default is to use the same address as +MailTo+.
+[[opt_MakeArchiveName]] *MakeArchiveName* (string, default "$\{BroBase}/share/broctl/scripts/make-archive-name")::
+Script to generate filenames for archived log files.
 [[opt_MemLimit]] *MemLimit* (string, default "unlimited")::
 Maximum amount of memory for Bro processes to use (in KB, or the string 'unlimited').
 [[opt_MinDiskSpace]] *MinDiskSpace* (int, default 5)::

aux/broctl/aux/trace-summary/trace-summary

@@ -184,6 +184,9 @@ class Interval:
             s += fmt("Connections", self.pkts) + \
                  fmt("Payload", self.payload)
         
+        if Options.factor != 1:
+            s += "Sampling %.2f%% -" % ( 100.0 / Options.factor )
+                 
         if Options.verbose:    
             ports = topx(self.ports)
             srcs = topx(self.srcs)
@@ -848,13 +851,18 @@ print Total.format(conns=Options.conns, title="Total")
 
 locals = LocalNets.keys()    
 
+for net in locals:        
+    (txt, i) = LocalNets[net]
+    if i.updates:
+        i.applySampleFactor()
+
 if locals:
 
     type = "packets"
     if Options.conns:
         type = "connections"
 
-    locals.sort(lambda x,y: LocalNets[y][1].pkts - LocalNets[x][1].pkts)    
+    locals.sort(lambda x,y: int(LocalNets[y][1].pkts - LocalNets[x][1].pkts))    
 
     print "\n>== Top %d local networks by number of %s\n" % (Options.topx, type)
         
@@ -876,9 +884,6 @@ for net in locals:
     (txt, i) = LocalNets[net]
 
     if i.updates:
-#        i.start += TotalIntervals.start
-#        i.end += TotalIntervals.start
-        i.applySampleFactor()
         print i.format(conns=Options.conns, title=net + " " + txt)
 
 print "First: %16s (%.6f) Last: %s %.6f" % (isoTime(Total.start), Total.start, isoTime(Total.end), Total.end)

aux/broctl/bin/archive-log.in

@@ -4,9 +4,7 @@
 #
 # Bro postprocessor script to archive log files. 
 # 
-# archive-log <rotated-file-name> <base-name> <timestamp-when-opened> <timestamp-when-closed> [<tag>]
+# archive-log <rotated-file-name> <base-name> <timestamp-when-opened> <timestamp-when-closed> <terminating> [<tag>]
-
-base=${logdir}
 
 delete=1
 if [ "$1" == "-c" ]; then
@@ -14,31 +12,36 @@ if [ "$1" == "-c" ]; then
     shift
 fi
 
-# Record time of last rotation.
-date +%y-%m-%d_%H.%M.%S >.rotate # Bro default format when rotating files.
-
 # We do not keep the logs for workers/proxies.
 if [ -e .worker -o -e .proxy ]; then
     test $delete = 0 || rm -rf $1
     exit 0
 fi
 
-# Build archive name
+terminating=$5
-day=`echo $3 | sed 's/_.*$//'`
-from=`echo $3 | sed 's/^.*_//' | sed 's/\./:/g'`
-to=`echo $4 | sed 's/^.*._//' | sed 's/\./:/g'`
-century=`date +%Y | sed 's/..$//g'`
-day="$century$day"
 
-if [ ! -d "$base/$day" ]; then
+century=`date +%Y | sed 's/..$//g'`
-    mkdir "$base/$day" 2>/dev/null
+
+from=`echo $3 | sed 's/[_.]/-/g'`
+from="$century$from"
+
+to=`echo $4 | sed 's/[_.]/-/g'`
+to="$century$to"
+
+dest=`${makearchivename} $2 $from $to`
+
+echo $dest | grep -q '^/' 
+
+if [ $? != 0 ]; then
+    dest="${logdir}/$dest"
 fi
 
-#if [ $# == 5 ]; then
+dest_dir=`dirname $dest`
-#    dest="$base/$day/$5.$2.$from-$to.gz"
+
-#else
+mkdir -p $dest_dir # Makes sure all parent directories exist. 
-    dest="$base/$day/$2.$from-$to.gz"
+
-#fi
+# Record time of last rotation.
+date +%y-%m-%d_%H.%M.%S >.rotated.$2 # Bro default format when rotating files.
 
 # Run other postprocessors. 
 for pp in ${postprocdir}/*; do
@@ -46,9 +49,11 @@ for pp in ${postprocdir}/*; do
 done
 
 if [ -e $1 ]; then
-   nice gzip -9 <$1 >$dest 2>/dev/null 
+   nice gzip -9 <$1 >$dest.gz 2>/dev/null &
 fi   
 
+wait
+
 if [ "$?" == "0" ]; then
     if [ "$delete" == "1" ]; then 
         rm -rf $1
@@ -57,3 +62,8 @@ if [ "$?" == "0" ]; then
         find $1 -size +104857600c -delete
     fi
 fi
+
+if [ "$terminating" == "1" ]; then
+    ${scriptsdir}/remove-link-for-log $2
+fi
+

aux/broctl/bin/broctl.in

@@ -323,9 +323,9 @@ class BroCtlCmdLoop(cmd.Cmd):
         manually: all the maintainance tasks will then just be performed one
         more time."""
 
-        self.lock()
-
         if len(args) > 0:
+            self.lock()
+            
             if args == "enable":
                 config.Config._setState("cronenabled", "1")
                 util.output("cron enabled")
@@ -336,6 +336,7 @@ class BroCtlCmdLoop(cmd.Cmd):
                 util.output("cron " + (config.Config.cronenabled == "1"  and "enabled" or "disabled"))
             else:
                 util.output("wrong cron argument")
+                
             return
 
         cron.doCron()

aux/broctl/bin/check-config.in

@@ -25,6 +25,7 @@ shift
 export PATH=${bindir}:${scriptsdir}:$PATH
 
 echo $@ >.cmdline
+touch .checking
 
 if [ "${devmode}" == "0" ]; then
     ${bro} $@

aux/broctl/bin/crash-diag.in

@@ -44,12 +44,16 @@ echo
 
 core=`ls -t *core* 2>&1`
 
-for c in $core; do 
+if which gdb >/dev/null 2>&1; then
-   if [ -e $c ]; then
+    for c in $core; do 
-      echo $c
+        if [ -e $c ]; then
-      echo "bt" | gdb --batch -x /dev/stdin ${bro} $c
+            echo $c
-   fi
+            echo "bt" | gdb --batch -x /dev/stdin ${bro} $c
-done
+        fi
+    done
+else
+    echo "No gdb installed."
+fi
 
 ) >.crash-diag.log
 

aux/broctl/bin/create-link-for-log.in

@@ -0,0 +1,65 @@
+#! /usr/bin/env bash
+#
+# create-link-for <file-name>
+#
+# Creates a link from `pwd`/$1 into the current archive directory.
+
+if [ ! -e .manager -a ! -e .standalone ]; then
+   # We only create links on the manager/standalone.
+   exit 0
+fi   
+
+if [ -e .checking ]; then
+   # Just checking configuration, don't create links.
+   exit 0
+fi   
+
+if [ ! -f $1 ]; then
+   # Doesn't exist. 
+   exit 0
+fi   
+
+echo $1 | grep -q '^\.' 
+
+if [ $? == 0 ]; then
+   # Don't link internal files.
+   exit 0
+fi
+
+date=`date +%Y-%m-%d-%H-%M-%S`
+link=`${makearchivename} $1 $date`
+
+echo $link | grep -q '^/' 
+
+if [ $? != 0 ]; then
+    link="${logdir}/$link"
+fi
+
+dest_dir=`dirname $link`
+mkdir -p $dest_dir # Makes sure all parent directories exist. 
+
+if [ -e $link ]; then
+   if [ ! -L $link ]; then
+       # Exists, but isn't a link. Don't touch.
+       exit 0
+   fi
+   
+   # Link exists already for some reason, remove it. 
+   rm -f $link
+fi
+
+# Remove last link we did for this file.
+if [ -e .link.$1 ]; then
+    rm -f `cat .link.$1 | tail -1`
+fi    
+
+# Do the link.
+ln -s `pwd`/$1 $link
+
+# Record the link.
+echo $link >.link.$1
+
+
+
+
+

aux/broctl/bin/delete-log

@@ -4,7 +4,7 @@
 #
 # Bro postprocessor script to archive log files. 
 # 
-# archive-log <rotated-file-name> <base-name> <timestamp-when-opened> <timestamp-when-closed> [<tag>]
+# archive-log <rotated-file-name> <base-name> <timestamp-when-opened> <timestamp-when-closed> <terminating> [<tag>]
 
 rm -rf $1
 

aux/broctl/bin/fmt-time

@@ -0,0 +1,19 @@
+#! /usr/bin/env python
+#
+# Prints a Unix timestamp $1 in the format Bro uses for its rotation timestamps.
+
+fmt="%y-%m-%d_%H.%M.%S" # From rotate-logs.bro
+
+import sys
+import time
+
+if len(sys.argv) != 2:
+    print >>sys.stderr, "usage: fmt-time unix-timestamp"
+    sys.exit(1)
+
+t = float(sys.argv[1])
+    
+print time.strftime(fmt, time.localtime(int(t)))
+	
+	
+	

aux/broctl/bin/get-prof-log.in

@@ -8,7 +8,9 @@ tag=$1
 host=$2
 path=$3
 
-dstbase=${statsdir}/prof.$tag
+mkdir -p ${statsdir}/profiling
+
+dstbase=${statsdir}/profiling/prof.$tag
 tmp=$dstbase.$$.log.tmp
 
 # Ignore errors.

aux/broctl/bin/helpers/df.in

@@ -7,5 +7,5 @@
 # Returns:  <fs> <fs-size> <fs-used> <fs-avail>
 
 echo 0
-df -h $1 | awk '{print $1, $2, $3, $4}' | tail -1 | awk -f ${helperdir}/to-bytes.awk
+df -kP $1 | awk '{print $1, $2, $3, $4}' | tail -1 | awk -v def_factor=1024 -f ${helperdir}/to-bytes.awk
 echo ~~~

aux/broctl/bin/helpers/to-bytes.awk

@@ -1,6 +1,12 @@
 # $Id: to-bytes.awk 6811 2009-07-06 20:41:10Z robin $
 
 # Converts strings such as 12K, 42M, etc. into bytes.
+# If def_factor is set, it's applied to values without any unit.
+
+BEGIN {
+    if ( def_factor == 0 )
+        def_factor = 1;
+    }
 
 {
     for ( i = 1; i <= NF; i++) {
@@ -9,6 +15,7 @@
 	    else if ( match($i, "^(-?[0-9.]+)Mi?$") ){ $i = substr($i, RSTART, RLENGTH-1) * 1024 * 1024; }
 	    else if ( match($i, "^(-?[0-9.]+)Gi?$") ){ $i = substr($i, RSTART, RLENGTH-1) * 1024 * 1024 * 1024; }
 	    else if ( match($i, "^(-?[0-9.]+)Te?$") ){ $i = substr($i, RSTART, RLENGTH-1) * 1024 * 1024 * 1024 * 1024; }
+	    else if ( match($i, "^(-?[0-9.]+)$") )   { $i = substr($i, RSTART, RLENGTH) * def_factor; }
 	    printf("%s ", $i);
 	}
 

aux/broctl/bin/helpers/top.in

@@ -14,11 +14,11 @@ cmd_freebsd_nonsmp='top -u -b all | awk "/^ *[0-9]+ /{printf(\"%d %s %s %d %s\\n
 cmd_darwin='top -l 1 | awk "/^ *[0-9]+ /{printf(\"%d %dK %dK %d %s\\n\", \$1, \$11, \$10, \$3, \$2)}"'
 cmd_netbsd='top -b -u  | awk "/^ *[0-9]+ /{printf(\"%d %s %s %d %s\\n\", \$1, \$5, \$6, \$10, \$11)}"'
 
-cmd="$cmd_${os}"
+eval cmd="\$cmd_${os}"
 
 if [ "${os}" == "freebsd" ]; then
    # Top's output looks different on non-SMP FreeBSD machines.
-   top -u -b all | grep -q "STATE  C   TIME" || cmd="$cmd_freebsd_nonsmp"
+   top -u -b all | grep -q "STATE  *C  *TIME" || cmd="$cmd_freebsd_nonsmp"
 fi
 
 unset LINES

aux/broctl/bin/make-archive-name

@@ -0,0 +1,41 @@
+#! /usr/bin/env bash
+#
+# $Id: archive-log.in 6847 2009-07-30 16:54:58Z robin $
+#
+# Returns a path for archived log files. This script is called
+# once for each log file being archived. Usage is:
+#
+# make-archive-name <basename> <timestamp-when-opened> [<timestamp-when-closed>]
+#
+#   basename: The base file name of the log file being archived (e.g., conn.log).
+#   timestamp-when-opened: The timestamp when the log file being archived was created. 
+#   timestamp-when-closed: The timestamp when the log file being archived was finished. 
+#                          Optional. If not given, the name is used to create a link to 
+#                          the current live version of the file.
+#
+# Times are given in the form "year-month-day-hour-minute-second",
+# e.g., "2010-03-30-13-12-04"
+#
+# The script must return the path under which the file should be
+# archived. If it's a relative path, it will be interpreted as
+# relative to BroControl's standard log directory. 
+#
+# Note that even though the logs will later be compressed, this
+# script should return the filename without any .gz extension; that
+# extension will be appended later.
+
+name=$1
+opened=$2
+closed=$3
+
+day=`echo $opened  | awk -F - '{printf "%s-%s-%s", $1, $2, $3}'`
+from=`echo $opened | awk -F - '{printf "%s:%s:%s", $4, $5, $6}'`
+to=`echo $closed | awk -F - '{printf "%s:%s:%s", $4, $5, $6}'`
+
+if [ "$closed" != "" ]; then
+   echo $day/$name.$from-$to
+else
+   echo $day/$name.$from-current
+fi   
+
+

aux/broctl/bin/post-terminate.in

@@ -57,20 +57,33 @@ if [ ! -f .startup ]; then
    exit
 fi
 
+brostart=`cat .startup | tail -1`
 end=`date +%y-%m-%d_%H.%M.%S`
-start=`cat .startup | tail -1`
-if [ "$crash" = "1" -a -e .rotate ]; then
-   start=`cat .rotate | tail -1`
-fi   
 
+# Old. Remove later. 
+#
+#if [ "$crash" = "1" -a -e .rotate ]; then
+#   start=`cat .rotate | tail -1`
+#fi   
+
+# Likewise old.
 #if [ -e .peer_description ]; then
 #   tag=`cat .peer_description | tail -1`
 #fi   
 
 ( for i in *.log; do
     if [ -s $i ]; then
-       ${scriptsdir}/archive-log $archive_flags $i $i $start $end $tag >/dev/null &
+       if [ -e .rotated.$i ]; then
+          start=`cat .rotated.$i`
+       else
+          start=$brostart
+       fi
+       
+       ${scriptsdir}/archive-log $archive_flags "$i" "$i" "$start" "$end" 1 "$tag" >/dev/null &
     fi
+    
+    ${scriptsdir}/remove-link-for-log $i
+
   done && wait && if [ "$crash" = "0" ]; then rm -rf $tmp; fi ) &
 
 

aux/broctl/bin/postprocessors/mail-log.in

@@ -5,7 +5,7 @@
 # Formats Bro's mail.log, archives, encrypts and mails it (if requested).
 #
 # It's called as a Bro postprocessor so its arguments are:
-# 	   mail-log <logfile> <basename> <timestamp-when-opened> <timestamp-when-closed> [<tag>]
+# 	   mail-log <logfile> <basename> <timestamp-when-opened> <timestamp-when-closed> <terminating> [<tag>]
 
 if [ "$2" != "mail.log" ]; then
    exit 0
@@ -15,7 +15,8 @@ log=$1
 base=$2
 open=$3
 close=$4
-tag=$5
+terminating=$5
+tag=$6
 
 # Do nothing if log is empty
 if [ ! -s $log ]; then

aux/broctl/bin/postprocessors/summarize-connections.in

@@ -6,7 +6,7 @@
 #
 # Needs trace-summary script.
 #
-# summarize-conns <rotated-file-name> <base-name> <timestamp-when-opened> <timestamp-when-closed> [<tag>]
+# summarize-conns <rotated-file-name> <base-name> <timestamp-when-opened> <timestamp-when-closed> <terminating> [<tag>]
 
 if [ "$2" != "conn.log" ]; then
    exit 0
@@ -15,7 +15,7 @@ fi
 summary_options="-c -r"
 
 # If we're a cluster installation, we assume we have lots of traffic and activate sampling.
-if [ "${standalone}" != "0" ]; then
+if [ "${standalone}" = "0" ]; then
    summary_options="$summary_options -S 0.01" 
 fi    
 

aux/broctl/bin/remove-link-for-log.in

@@ -0,0 +1,8 @@
+#! /usr/bin/env bash
+#
+# remove-link-for-log <filename>
+
+if [ -e .link.$1 ]; then
+    rm -f `cat .link.$1 | tail -1`
+    rm -f .link.$1
+fi    

aux/broctl/bin/stat-ctime

@@ -0,0 +1,25 @@
+#! /usr/bin/env python
+#
+# Prints a given paths ctime in the format Bro uses for its rotation
+# timestamps.
+
+fmt="%y-%m-%d_%H.%M.%S" # From rotate-logs.bro
+
+import sys
+import os
+import stat
+import time
+
+if len(sys.argv) != 2:
+    print >>sys.stderr, "usage: stat-ctime <path>"
+
+try:	
+	ctime = os.stat(sys.argv[1])[stat.ST_CTIME]
+except OSError, e:
+	print e
+	sys.exit(1)
+	
+print time.strftime(fmt, time.localtime(ctime))
+	
+	
+	

aux/broctl/bin/stats-to-csv

@@ -0,0 +1,137 @@
+#! /usr/bin/env python
+#
+# stats-to-csv <stats.log> <meta.dat> <wwwdir>
+#
+# Reads information from stats directory and outputs csv files <wwwdir>/<node>.<type>.csv. 
+# If any of these files already exists, we append (without writing the header line again).
+
+import os.path
+import os
+import sys
+
+Workers = set()
+Proxies = set()
+
+def readNodes(meta):
+    for line in open(meta):
+        m = line.split()
+        
+        if m[0] == "node":
+            if m[2] == "worker":
+                Workers.add(m[1])
+                
+            if m[2] == "proxy":
+                Workers.add(m[1])
+
+def processNode(stats, wwwdir, node, iface):
+    
+    print node, "..."
+    
+    def openFile(tag, columns):
+        
+        name = os.path.join(wwwdir, "%s.%s.csv" % (node, tag))
+        
+        if os.path.exists(name):
+            return open(name, "a")
+        else:
+            f = open(name, "w")
+            print >>f, "time," + ",".join(columns)
+            
+        return f
+    
+    iface_mbps = openFile("mbps", ["MBits/sec"])
+    iface_pkts = openFile("pkts", ["TCP", "UDP", "ICMP", "Other"])
+    cpu = openFile("cpu", ["CPU"])
+    mem = openFile("mem", ["Memory"])
+    cflow = openFile("in", ["MBits/sec"])
+    
+    def printEntry(t, entry):
+
+        try:
+            val = int(entry["parent-cpu"]) + int(entry["child-cpu"])
+            print >>cpu, "%s,%s" % (t, val)
+        except KeyError:
+            pass
+            
+        try:
+            val = int(entry["parent-vsize"]) + int(entry["child-vsize"])
+            print >>mem, "%s,%s" % (t, val)
+        except KeyError:
+            pass
+            
+        if iface:
+            try:
+                print >>iface_mbps, "%s,%s" % (t, entry["interface-mbps"])
+            except KeyError:
+                    pass
+                
+            try:
+                tc = float(entry["interface-t"])
+                ud = float(entry["interface-u"])
+                ic = float(entry["interface-i"])
+                ot = float(entry["interface-o"])
+                print >>iface_pkts, "%s,%s,%s,%s,%s" % (t, tc, ud, ic, ot)
+                
+            except KeyError:
+                    pass
+                
+        if "in-mbps" in entry:
+            print >>cflow, "%s,%s" % (t, entry["in-mbps"])
+    
+    entry = {}
+    first = -1
+    
+    for line in open(stats):
+        m = line.split()
+        
+        if m[1] != node:
+            continue
+
+        t = m[0]
+        
+        if t != first and first >= 0:
+            printEntry(t, entry)
+            entry = {}
+            
+        first = t
+            
+        try:
+            entry["%s-%s" % (m[2], m[3])] = m[4]
+        except IndexError:
+            pass
+        
+    if first >= 0:
+        printEntry(t, entry)
+                
+    iface_mbps.close()
+    iface_pkts.close()
+    cpu.close()
+    mem.close()
+    cflow.close()
+
+if len(sys.argv) != 4:
+    print "usage: %s <stats.log> <meta.dat> <www-dir>" % sys.argv[0]
+    sys.exit(1)
+
+stats = sys.argv[1]
+meta = sys.argv[2]
+wwwdir = sys.argv[3]
+    
+try:
+    os.mkdir(wwwdir)
+except OSError:
+    pass
+
+readNodes(meta)
+
+for w in Workers:
+    processNode(stats, wwwdir, w, True)
+
+for p in Proxies:
+    processNode(stats, wwwdir, p, False)
+
+processNode(stats, wwwdir, "manager", False)
+processNode(stats, wwwdir, "cflow", False)
+    
+    
+    

aux/broctl/bin/update-stats.in

@@ -0,0 +1,15 @@
+#! /usr/bin/env bash
+#
+# $Id: archive-log.in 6847 2009-07-30 16:54:58Z robin $
+#
+# Saves the current stats.log from spool to ${statsdir}, and 
+# updates the WWW data. 
+
+dst=${statsdir}/`basename ${statslog}`
+cat ${statslog} >>$dst
+
+cp ${statsdir}/meta.dat ${statsdir}/www
+
+${scriptsdir}/stats-to-csv ${statslog} ${statsdir}/meta.dat ${statsdir}/www
+
+rm -f ${statslog}

aux/broctl/policy/broctl-check.bro

@@ -2,9 +2,5 @@
 #
 # Only loaded when checking configuration, not when running live.
 
-@load rotate-logs
+redef RotateLogs::rotate_on_shutdown = F;
-
-redef RotateLogs::rotate_on_shutdown=F;
-
-	
 		

aux/broctl/policy/broctl-live.bro

@@ -3,4 +3,3 @@
 # Only loaded when running live, not when just checking configuration.
 
 
-		

aux/broctl/policy/broctl.bro

@@ -20,6 +20,7 @@ redef MANAGER = MANAGER > 0 ? MANAGER : 1;
 @load cluster-by-addrs
 @load remote-update
 @load checkpoint 
+@load rotate-logs
 
 # FIXME: Load them here to work around a namespace bug.
 @load conn

aux/broctl/policy/cluster-manager.bro

@@ -8,7 +8,6 @@
 @load filter-duplicates
 @load notice
 @load remote
-@load rotate-logs
 @load mail-alarms
 	
 # Since we don't capture, don't bother with this.
@@ -32,6 +31,9 @@ redef interfaces = "";
 # Give us a name. 
 redef peer_description = BroCtl::manager$tag;
 
+# We're processing essentially *only* remote events.
+redef max_remote_events_processed = 10000;
+
 # Reraise remote notices locally.
 event notice_action(n: notice_info, action: NoticeAction)
 	{

aux/broctl/policy/cluster-manager.drop.bro

@@ -3,27 +3,8 @@
 # These will be generated by the workers.
 event Drop::address_seen_again(a: addr)
 	{
-	if ( ! use_catch_release )
+	debug_log(fmt("received seen_again for %s", a));
-		return;
+	
-	
-	if ( a !in drop_info )
-		# Never dropped.
-		return;
-	
-	local di = drop_info[a];
-	if ( is_dropped(a) )
-		# Still dropped.
-		return;
-	
-	NOTICE([$note=AddressSeenAgain, $src=a,
-			$msg=fmt("%s seen again after release", a)]);
-	}
-
-# $Id$
-
-# These will be generated by the workers.
-event Drop::address_seen_again(a: addr)
-	{
 	if ( ! use_catch_release )
 		return;
 	

aux/broctl/policy/cluster-manager.icmp.bro

@@ -4,9 +4,3 @@ redef FilterDuplicates::filters += {
     [ICMPAddressScan] = FilterDuplicates::match_src_num
 };
 	
-# $Id: cluster-manager.scan.bro 6740 2009-06-12 17:59:44Z robin $
-
-redef FilterDuplicates::filters += {
-    [ICMPAddressScan] = FilterDuplicates::match_src_num
-};
-	

aux/broctl/policy/cluster-manager.rotate-logs.bro

@@ -4,4 +4,10 @@ redef log_rotate_interval = 24hrs;
 redef log_rotate_base_time = "0:00";
 redef RotateLogs::default_postprocessor = "archive-log";
 
-redef conn_file &rotate_interval = 12hrs;
+event file_opened(f: file)
+	{
+	# Create a link from the archive directory to the newly created file.
+	if ( MANAGER == 1 && ! bro_is_terminating() )
+		system(fmt("create-link-for-log %s", get_file_name(f)));
+	}
+

aux/broctl/policy/cluster-proxy.bro

@@ -6,7 +6,6 @@
 
 @load broctl
 @load remote
-@load rotate-logs
 	
 # Since we don't capture, don't bother with this.
 @unload print-filter

aux/broctl/policy/cluster-proxy.remote.bro

@@ -1,5 +1,8 @@
 # $Id: cluster-proxy.remote.bro 6811 2009-07-06 20:41:10Z robin $
 
+# Do not copy the proxies's remote.log to the manager
+redef Remote::rm_log &disable_print_hook;
+
 event bro_init() 
 	{
 	# Set up worker connections.

aux/broctl/policy/cluster-worker.bro

@@ -7,7 +7,6 @@
 
 @load broctl
 @load remote
-@load rotate-logs
 
 @load trim-trace-file	
 

aux/broctl/policy/cluster-worker.drop.bro

@@ -7,18 +7,21 @@ global watch_addr_table: set[addr] &read_expire=7days &persistent;
 
 global address_seen_again: event(a: addr);
 
-event address_restored(a: addr)
+event Drop::address_restored(a: addr)
 	{
+	debug_log(fmt("received restored for %s", a));
 	add watch_addr_table[a];
 	}
 
-event address_dropped(a: addr)
+event Drop::address_dropped(a: addr)
 	{
+	debug_log(fmt("received dropped for %s", a));
 	delete watch_addr_table[a];
 	}
 
-event address_cleared(a: addr)
+event Drop::address_cleared(a: addr)
 	{
+	debug_log(fmt("received cleared for %s", a));
 	delete watch_addr_table[a];
 	}
 
@@ -28,6 +31,7 @@ event new_connection(c: connection)
 	local a = c$id$orig_h;
 	if ( a in watch_addr_table )
 		{
+		debug_log(fmt("sending seen_again for %s", a));
 		event Drop::address_seen_again(a);
 		delete watch_addr_table[a];
 		}

aux/broctl/policy/cluster.scan.bro

@@ -5,9 +5,9 @@ redef ignore_scanners_threshold = 500;
 
 redef pre_distinct_peers &read_expire = 12hrs;
 
-redef distinct_backscatter_peers &read_expire = 30mins;
+redef distinct_backscatter_peers &create_expire = 5hrs;
-redef distinct_peers &read_expire = 30mins;
+redef distinct_peers &create_expire = 5hrs;
-redef distinct_ports &read_expire = 30mins;
+redef distinct_ports &create_expire = 5hrs;
-redef distinct_low_ports &read_expire = 30mins;
+redef distinct_low_ports &create_expire = 5hrs;
-redef possible_scan_sources &read_expire = 30mins;
+redef possible_scan_sources &create_expire = 5hrs;
 

aux/broctl/policy/mail-alarms.bro

@@ -22,18 +22,38 @@ export 	{
 	global output = open_log_file( "mail" );
 	}
 
-function do_msg(line1: string, line2: string, line3: string, host: addr, name: string)
+function do_msg(n: notice_info, line1: string, line2: string, line3: string, host: addr, name: string, dest: string)
 	{
 	if ( host != 0.0.0.0 )
 		name = fmt("%s = %s", host, name);
 	
-	print output, cat(line1, name);
+	line1 = cat(line1, name);
-	print output, line2;
+	
-	if ( line3 != "" )
+	if ( dest == "" ) 
-		print output, line3;
+		{
+		# Append to mail.log.
+		print output, line1;
+		print output, line2;
+		if ( line3 != "" )
+			print output, line3;
+		}
+	
+	else 
+		{
+		line1 = str_shell_escape(line1);
+		line2 = str_shell_escape(line2);
+		line3 = str_shell_escape(line3);
+		
+		# Mail out an individual alarm. 
+		local mail_cmd =
+			fmt("( echo \"%s\"; echo \"%s\"; echo \"%s\" ) | %s -s \"[Bro Alarm] %s: %s\" %s",
+				line1, line2, line3, mail_script, n$note, str_shell_escape(n$msg), dest);
+		
+		system(mail_cmd);
+		}
 	}
 
-function message(msg: string, flag: bool, host: addr, n: notice_info)
+function message(msg: string, flag: bool, host: addr, n: notice_info, dest: string)
 	{
 	if ( length(include_only) > 0 && n$note !in include_only )
 		return;
@@ -52,30 +72,22 @@ function message(msg: string, flag: bool, host: addr, n: notice_info)
 
 	if ( host == 0.0.0.0 )
 		{
-		do_msg(line1, line2, line3, 0.0.0.0, "");
+		do_msg(n, line1, line2, line3, 0.0.0.0, "", dest);
 		return;
 		}
 	
 	when ( local name = lookup_addr(host) )
 		{
-		do_msg(line1, line2, line3, host, name);
+		do_msg(n, line1, line2, line3, host, name, dest);
 		}
 	timeout 5secs
 		{
-		do_msg(line1, line2, line3, host, "(dns timeout)");
+		do_msg(n, line1, line2, line3, host, "(dns timeout)", dest);
 		}
 	}
 
-event bro_init()
+function make_alarm(n: notice_info, dest: string)
-    {
-    set_buf( output, F );
-    }
-
-event notice_alarm(n: notice_info, action: NoticeAction) &priority = -10
 	{
-	if ( is_remote_event() )
-		return;
-
 	if ( n$note in ignore )
 		return;
 	
@@ -112,6 +124,33 @@ event notice_alarm(n: notice_info, action: NoticeAction) &priority = -10
 	if ( orig in flag_nets || resp in flag_nets )
 		flag = T;
 
-	message(msg, flag, host, n);
+	message(msg, flag, host, n, dest);
 	}
 
+
+event bro_init()
+    {
+    set_buf( output, F );
+    }
+
+event notice_alarm(n: notice_info, action: NoticeAction) &priority = -10
+	{
+	if ( is_remote_event() )
+		return;
+
+	make_alarm(n, "");
+	}
+
+function broctl_email_notice_to(n: notice_info, dest: string)
+	{
+	if ( reading_traces() || dest == "" )
+		return;
+
+	if ( dest == "" )
+		return;
+	
+	make_alarm(n, dest);
+	}
+
+# Make the alarm mails nicer. 
+redef email_notice_to = broctl_email_notice_to;

aux/broctl/policy/standalone.bro

@@ -10,7 +10,6 @@
 @load broctl
 @load notice
 @load remote
-@load rotate-logs
 @load mail-alarms
 	
 @load trim-trace-file	

aux/broctl/policy/standalone.rotate-logs.bro

@@ -1,9 +1,13 @@
 # $Id: standalone.rotate-logs.bro 6811 2009-07-06 20:41:10Z robin $
 
-@load mail-alarms
-
 redef log_rotate_interval = 24hrs;
 redef log_rotate_base_time = "0:00";
 redef RotateLogs::default_postprocessor = "archive-log";
 
-redef conn_file &rotate_interval = 12hrs;
+event file_opened(f: file)
+	{
+	# Create a link from the archive directory to the newly created file.
+	if ( ! bro_is_terminating() )
+		system(fmt("create-link-for-log %s", get_file_name(f)));
+	}
+

aux/cf/cf.1

@@ -40,7 +40,7 @@ with a formated time and date time and date. For example:
 .RS
 .na
 .nh
-\% echo '1074558944 default format' | cf
+% echo '1074558944 default format' | cf
 .br  
 Jan 19 16:35:44 default format
 .ad
@@ -66,6 +66,12 @@ and
 flags override the
 .B CFTIMEFMT
 environment variable.
+
+Note that filter skips over an instance of "t=" at the beginning of
+a line, to provide compatibility with Bro's
+.I
+tagged
+logging format.
 .SH OPTIONS
 .LP
 .TP

aux/cf/cf.c

@@ -143,6 +143,12 @@ doone(fin, fout)
 	while (fgets(buf, sizeof(buf), fin)) {
 		bp = buf;
 		dotbp = NULL;
+
+		if (*bp == 't' && *(bp+1) == '=') {
+			fputs("t=", fout);
+			bp += 2;
+		}
+
 		if (isdigit(*bp)) {
 			ts = atol(bp);
 			++bp;

configure.in

@@ -30,6 +30,8 @@ AC_CANONICAL_SYSTEM
 #AM_INIT_AUTOMAKE(bro, 0.1.0)
 AM_INIT_AUTOMAKE(bro,  esyscmd([tr -d '\n' < VERSION]))
 AM_CONFIG_HEADER(config.h)
+AC_LBL_C_INIT_BEFORE_CC(V_CCOPT, V_INCLS)
+AC_PROG_CC
 AC_LBL_C_INIT(V_CCOPT, V_INCLS)
 AM_PROG_LEX
 
@@ -136,6 +138,20 @@ AC_LBL_ENABLE_CHECK([activemapping binpac broccoli brov6 debug \
 	expire-dfa-states gtk-doc int64 openssl perftools perl \
 	select-loop shippedpcap broctl cluster nbdns])
 
+dnl ################################################
+dnl # Writing around broken autoconf 
+dnl ################################################
+dnl  It seems that AC_CHECK_HEADER defines a bash function called 
+dnl  ac_fn_c_check_header_compile in the output when it is first
+dnl  encountered. While in general a neat idea, this fails, if the
+dnl  first use of AC_CHECK_HEADER is in an if/else clause. In this
+dnl  case the function's scope is limited to the enclosing if/els
+dnl  block and later calls to the function fail (more or less silently)
+dnl  Solution: we just place a phony AC_CHECK_HEADER call here. 
+AC_CHECK_HEADER([stdio.h])
+AC_CHECK_HEADERS([stdio.h stdio.h])
+
+
 dnl ################################################
 dnl # OpenSSL
 dnl ################################################
@@ -168,9 +184,9 @@ if test "$use_openssl" = "yes"; then
    # (CHECK_HEADER doesn't work here)
    saved_cflags="${CFLAGS}"
    CFLAGS="${CFLAGS} -I${OPENSSL}/include"
-   AC_COMPILE_IFELSE([#include <openssl/ssl.h>],,
+   AC_COMPILE_IFELSE([AC_LANG_SOURCE([[#include <openssl/ssl.h>]])],,
       CFLAGS="${CFLAGS} -I/usr/kerberos/include"
-      AC_CHECK_HEADER(krb5.h, 
+      AC_CHECK_HEADER([krb5.h], 
          V_INCLS="${V_INCLS} -I/usr/kerberos/include"
          AC_DEFINE(NEED_KRB5_H,,[Include krb5.h]),
          use_openssl=no
@@ -188,7 +204,7 @@ if test "$use_openssl" = "yes"; then
    saved_libs="${LIBS}"
    LIBS="${LIBS} -lssl -lcrypto"
    AC_MSG_CHECKING([for OpenSSL >= 0.9.7])
-   AC_LINK_IFELSE(AC_LANG_PROGRAM([[#include <openssl/evp.h>]], [[OPENSSL_add_all_algorithms_conf();]]),
+   AC_LINK_IFELSE([AC_LANG_PROGRAM([[#include <openssl/evp.h>]], [[OPENSSL_add_all_algorithms_conf();]])],
       AC_MSG_RESULT(yes)
       use_openssl=yes,
       AC_MSG_RESULT(no)
@@ -212,9 +228,9 @@ if test "$use_openssl" = "yes"; then
   AC_MSG_CHECKING([whether d2i_X509() uses a const unsigned char**])
   AC_LANG_PUSH([C++])
   AC_COMPILE_IFELSE(
-    AC_LANG_PROGRAM([[#include <openssl/x509.h>]],
+    [AC_LANG_PROGRAM([[#include <openssl/x509.h>]],
                     [[const unsigned char** cpp = 0;
-		      X509** x = 0; d2i_X509(x, cpp, 0);]]),
+		      X509** x = 0; d2i_X509(x, cpp, 0);]])],
     AC_DEFINE(OPENSSL_D2I_X509_USES_CONST_CHAR,,[d2i_x509 uses const char**])
     AC_MSG_RESULT(yes),
     AC_MSG_RESULT(no))
@@ -288,7 +304,7 @@ freebsd*)
 
 darwin*)
 	AC_MSG_CHECKING([if we need to include arpa/nameser_compat.h])
-	AC_COMPILE_IFELSE(AC_LANG_PROGRAM([[#include <arpa/nameser.h>]], [[HEADER *hdr; int d = NS_IN6ADDRSZ;]]), bro_ns_header_defined=yes, bro_ns_header_defined=no)
+	AC_COMPILE_IFELSE([AC_LANG_PROGRAM([[#include <arpa/nameser.h>]], [[HEADER *hdr; int d = NS_IN6ADDRSZ;]])], bro_ns_header_defined=yes, bro_ns_header_defined=no)
 	# if the header is found, we don't need compatibility
 	if test "x$bro_ns_header_defined" = xyes; then
 	   AC_MSG_RESULT(no)
@@ -353,14 +369,14 @@ AC_LBL_CHECK_TYPE(u_int16_t, u_short)
 AC_LBL_CHECK_TYPE(u_int8_t, u_char)
 AC_HEADER_TIME
 
-AC_CHECK_HEADERS(memory.h netinet/in.h socket.h getopt.h)
+AC_CHECK_HEADERS([memory.h netinet/in.h socket.h getopt.h])
-AC_CHECK_HEADERS(net/ethernet.h netinet/ether.h netinet/if_ether.h sys/ethernet.h,,,
+AC_CHECK_HEADERS([net/ethernet.h netinet/ether.h netinet/if_ether.h sys/ethernet.h],,,
     [#include <sys/types.h>
     #include <sys/socket.h>
     #include <netinet/in.h>
     #include <net/if.h>])
 
-AC_CHECK_HEADERS(netinet/ip6.h,,,
+AC_CHECK_HEADERS([netinet/ip6.h],,,
     [#include <sys/types.h>
     #include <sys/socket.h>
     #include <netinet/in.h>
@@ -416,7 +432,7 @@ if test "$pcap_local" = "NO"; then
     dnl ################################################
     AC_MSG_CHECKING([for pcap_version in libpcap])
     AC_LINK_IFELSE(
-     AC_LANG_PROGRAM([extern char pcap_version[];], [puts(pcap_version);]),
+     [AC_LANG_PROGRAM([extern char pcap_version[];], [puts(pcap_version);])],
       AC_MSG_RESULT(yes)
       AC_DEFINE(PCAP_VERSION_STRING,,[Have a version string in libpcap]),
       AC_MSG_RESULT(no))
@@ -448,7 +464,7 @@ dnl #
 AC_MSG_CHECKING([if char_traits defines all methods])
 AC_LANG_PUSH([C++])
 AC_LINK_IFELSE(
-  AC_LANG_PROGRAM([[
+  [AC_LANG_PROGRAM([[
 #include <string>
 using namespace std;
 class Foo { };
@@ -456,7 +472,7 @@ class Foo { };
 char_traits<Foo*> foo;
 Foo f;
 Foo *fp;
-foo.assign(&fp, 10, &f);]]),
+foo.assign(&fp, 10, &f);]])],
   AC_MSG_RESULT([yes])
   basic_string_works=yes,
   AC_MSG_RESULT([no])
@@ -575,17 +591,17 @@ else
       bro_ns_initparse_works=no
       bro_res_mkquery_works=no
 
-      AC_LINK_IFELSE(AC_LANG_PROGRAM([[#include <arpa/nameser.h>]],
+      AC_LINK_IFELSE([AC_LANG_PROGRAM([[#include <arpa/nameser.h>]],
-				      [[ns_initparse(0,0,0);]]),
+				      [[ns_initparse(0,0,0);]])],
 		     bro_ns_initparse_works=yes)
 
-      AC_LINK_IFELSE(AC_LANG_PROGRAM([[
+      AC_LINK_IFELSE([AC_LANG_PROGRAM([[
 #include <sys/types.h>
 #include <sys/socket.h>
 #include <netinet/in.h>
 #include <arpa/nameser.h>
 #include <resolv.h>]],
-[[int (*p)() = res_mkquery]]), bro_res_mkquery_works=yes)
+[[int (*p)() = res_mkquery]])], bro_res_mkquery_works=yes)
 
       if test $bro_ns_initparse_works = yes && test $bro_res_mkquery_works = yes && test $nbdns = yes; then
          AC_MSG_RESULT(yes)

policy/http-reply.bro

@@ -54,7 +54,13 @@ function http_reply_done(c: connection, stat: http_message_stat)
 		--s$num_pending_requests;
 		++s$first_pending_request;
 
-		req = fmt("%s %s", r$method, r$URI);
+		if ( log_referrer )
+			req = fmt("%s %s [ref %s]", r$method, r$URI,
+					  req_msg$referrer == "" ?
+					  "<NONE>" : req_msg$referrer);
+		else
+			req = fmt("%s %s", r$method, r$URI);
+
 		log_it = r$log_it;
 		}
 
@@ -113,5 +119,7 @@ event http_header(c: connection, is_orig: bool, name: string, value: string)
 			else
 				msg$host = value;
 			}
+		else if ( is_orig && name == "REFERER" )
+			msg$referrer = value;
 		}
 	}

policy/http-request.bro

@@ -51,6 +51,9 @@ export {
 	&redef;
 
 	const sensitive_post_URIs = /wwwroot|WWWROOT/ &redef;
+
+	# Include the referrer header in the log.
+	const log_referrer = F &redef;
 }
 
 redef capture_filters +=  {

policy/http.bro

@@ -50,6 +50,7 @@ type http_message: record {
 	abstract: string;	# data abstract
 	skip_abstract: bool;	# to skip abstract for certain content types
 	host: string;		# host indicated in Host header
+	referrer: string;	# "Referer" [sic] field
 };
 
 type http_pending_request_stream: record {
@@ -105,7 +106,7 @@ function init_http_message(msg: http_message)
 	msg$header_slot = 0;
 	msg$abstract = "";
 	msg$skip_abstract = F;
-	msg$host = "";
+	msg$referrer = msg$host = "";
 	}
 
 function new_http_message(): http_message

policy/notice.bro

@@ -272,6 +272,8 @@ function build_notice_info_string_tagged(n: notice_info) : string
 	return cur_info;
 	}
 
+global email_notice_to: function(n: notice_info, dest: string) &redef;
+
 function email_notice_to(n: notice_info, dest: string)
 	{
 	if ( reading_traces() || dest == "" )

policy/rotate-logs.bro

@@ -56,10 +56,11 @@ function run_pp(info: rotate_info)
 	if ( pp != "" )
 		# The date format is hard-coded here to provide a standardized
 		# script interface.
-		system(fmt("%s %s %s %s %s %s",
+		system(fmt("%s %s %s %s %s %s %s",
 				pp, info$new_name, info$old_name,
 				strftime("%y-%m-%d_%H.%M.%S", info$open),
 				strftime("%y-%m-%d_%H.%M.%S", info$close),
+				bro_is_terminating() ? "1" : "0",
 				tag));
 	else
 		system(fmt("/bin/mv %s %s %s",

src/File.cc

@@ -217,11 +217,8 @@ bool BroFile::Open(FILE* file)
 		return false;
 		}
 
-	val_list* vl = new val_list;
+	RaiseOpenEvent();
-	Ref(this);
+
-	vl->append(new Val(this));
-	Event* event = new ::Event(::file_opened, vl);
-	mgr.Dispatch(event, true);
 	return true;
 	}
 
@@ -305,6 +302,7 @@ FILE* BroFile::BringIntoCache()
 		return f;
 		}
 
+	RaiseOpenEvent();
 	UpdateFileSize();
 
 	if ( fseek(f, position, SEEK_SET) < 0 )
@@ -809,6 +807,18 @@ int BroFile::Write(const char* data, int len)
 	return true;
 	}
 
+void BroFile::RaiseOpenEvent()
+	{
+	if ( ! ::file_opened )
+		return;
+
+	val_list* vl = new val_list;
+	Ref(this);
+	vl->append(new Val(this));
+	Event* event = new ::Event(::file_opened, vl);
+	mgr.Dispatch(event, true);
+	}
+
 void BroFile::UpdateFileSize()
 	{
 	struct stat s;

src/File.h

@@ -114,6 +114,9 @@ protected:
 	// Stats the file to get its current size.
 	void UpdateFileSize();
 
+	// Raises a file_opened event.
+	void RaiseOpenEvent();
+
 	// Initialize encryption with the given public key.
 	void InitEncrypt(const char* keyfile);
 	// Finalize encryption.

src/RemoteSerializer.cc

@@ -544,6 +544,36 @@ void RemoteSerializer::Init()
 	initialized = 1;
 	}
 
+void RemoteSerializer::SetSocketBufferSize(int fd, int opt, const char *what, int size, int verbose)
+	{
+	int defsize = 0;
+	socklen_t len = sizeof(defsize);
+
+	if ( getsockopt(fd, SOL_SOCKET, opt, (void *)&defsize, &len) < 0 )
+		{
+		if ( verbose )
+			Log(LogInfo, fmt("warning: cannot get socket buffer size (%s): %s", what, strerror(errno)));
+		return;
+		}
+
+	for ( int trysize = size; trysize > defsize; trysize -= 1024 )
+		{
+		if ( setsockopt(fd, SOL_SOCKET, opt, &trysize, sizeof(trysize)) >= 0 )
+			{
+			if ( verbose )
+			    {
+			    if ( trysize == size )
+				    Log(LogInfo, fmt("raised pipe's socket buffer size from %dK to %dK", defsize / 1024, trysize / 1024));
+			    else
+				    Log(LogInfo, fmt("raised pipe's socket buffer size from %dK to %dK (%dK was requested)", defsize / 1024, trysize / 1024, size / 1024));
+			    }
+			return;
+			}
+		}
+
+	Log(LogInfo, fmt("warning: cannot increase %s socket buffer size from %dK (%dK was requested)", what, defsize / 1024, size / 1024));
+	}
+
 void RemoteSerializer::Fork()
 	{
 	if ( child_pid )
@@ -562,25 +592,11 @@ void RemoteSerializer::Fork()
 		return;
 		}
 
-	int bufsize;
+	// Try to increase the size of the socket send and receive buffers.
-	socklen_t len = sizeof(bufsize);
+	SetSocketBufferSize(pipe[0], SO_SNDBUF, "SO_SNDBUF", SOCKBUF_SIZE, 1);
-
+	SetSocketBufferSize(pipe[0], SO_RCVBUF, "SO_RCVBUF", SOCKBUF_SIZE, 0);
-	if ( getsockopt(pipe[0], SOL_SOCKET, SO_SNDBUF, &bufsize, &len ) < 0 )
+	SetSocketBufferSize(pipe[1], SO_SNDBUF, "SO_SNDBUF", SOCKBUF_SIZE, 0);
-		Log(LogInfo, fmt("warning: cannot get socket buffer size: %s", strerror(errno)));
+	SetSocketBufferSize(pipe[1], SO_RCVBUF, "SO_RCVBUF", SOCKBUF_SIZE, 0);
-	else
-		Log(LogInfo, fmt("pipe's socket buffer size is %d, setting to %d", bufsize, SOCKBUF_SIZE));
-
-	bufsize = SOCKBUF_SIZE;
-
-	if ( setsockopt(pipe[0], SOL_SOCKET, SO_SNDBUF,
-			&bufsize, sizeof(bufsize) ) < 0 ||
-	     setsockopt(pipe[0], SOL_SOCKET, SO_RCVBUF,
-			&bufsize, sizeof(bufsize) ) < 0 ||
-	     setsockopt(pipe[1], SOL_SOCKET, SO_SNDBUF,
-			&bufsize, sizeof(bufsize) ) < 0 ||
-	     setsockopt(pipe[1], SOL_SOCKET, SO_RCVBUF,
-			&bufsize, sizeof(bufsize) ) < 0 )
-		Log(LogInfo, fmt("warning: cannot set socket buffer size to %dK: %s", bufsize / 1024, strerror(errno)));
 
 	child_pid = 0;
 
@@ -681,7 +697,7 @@ bool RemoteSerializer::CloseConnection(Peer* peer)
 	if ( peer->suspended_processing )
 		{
 		net_continue_processing();
-		current_peer->suspended_processing = false;
+		peer->suspended_processing = false;
 		}
 
 	if ( peer->state == Peer::CLOSING )
@@ -1614,6 +1630,12 @@ void RemoteSerializer::PeerDisconnected(Peer* peer)
 	{
 	assert(peer);
 
+	if ( peer->suspended_processing )
+		{
+		net_continue_processing();
+		peer->suspended_processing = false;
+		}
+
 	if ( peer->state == Peer::CLOSED || peer->state == Peer::INIT )
 		return;
 
@@ -1744,6 +1766,12 @@ void RemoteSerializer::UnregisterHandlers(Peer* peer)
 
 void RemoteSerializer::RemovePeer(Peer* peer)
 	{
+	if ( peer->suspended_processing )
+		{
+		net_continue_processing();
+		peer->suspended_processing = false;
+		}
+
 	peers.remove(peer);
 	UnregisterHandlers(peer);
 
@@ -2941,7 +2969,7 @@ void SocketComm::Run()
 		struct timeval small_timeout;
 		small_timeout.tv_sec = 0;
 		small_timeout.tv_usec =
-			io->CanWrite() || io->CanRead() ? 10 : 10000;
+			io->CanWrite() || io->CanRead() ? 1 : 10;
 
 		int a = select(max_fd + 1, &fd_read, &fd_write, &fd_except,
 				&small_timeout);
@@ -3575,6 +3603,7 @@ bool SocketComm::Listen(uint32 ip, uint16 port, bool expect_ssl)
 	if ( bind(*listen_fd, (sockaddr*) &server, sizeof(server)) < 0 )
 		{
 		Error(fmt("can't bind to port %d, %s", port, strerror(errno)));
+		close(*listen_fd);
 		*listen_fd = -1;
 
 		if ( errno == EADDRINUSE )

src/RemoteSerializer.h

@@ -297,6 +297,8 @@ protected:
 	bool SendToChild(char type, Peer* peer, int nargs, ...); // can send uints32 only
 	bool SendToChild(ChunkedIO::Chunk* c);
 
+	void SetSocketBufferSize(int fd, int opt, const char *what, int size, int verbose);
+
 private:
 	enum { TYPE, ARGS } msgstate;	// current state of reading comm.
 	Peer* current_peer;

src/X509.cc

@@ -192,7 +192,7 @@ int X509_Cert::verifyChain(Contents_SSL* e, const u_char* data, uint32 len)
 	// but in chain format).
 
 	// Init the stack.
-	STACK_OF(X509)* untrustedCerts = sk_new_null();
+	STACK_OF(X509)* untrustedCerts = sk_X509_new_null();
 	if ( ! untrustedCerts )
 		{
 		// Internal error allocating stack of untrusted certs.
@@ -233,7 +233,7 @@ int X509_Cert::verifyChain(Contents_SSL* e, const u_char* data, uint32 len)
 		else
 			// The remaining certificates (if any) are put into
 			// the list of untrusted certificates
-			sk_push(untrustedCerts, (char*) pTemp);
+			sk_X509_push(untrustedCerts, pTemp);
 
 		tempLength += certLength + 3;
 		}
@@ -259,7 +259,7 @@ int X509_Cert::verifyChain(Contents_SSL* e, const u_char* data, uint32 len)
 	// Free the stack, incuding. contents.
 
 	// FIXME: could this break Bro's memory tracking?
-	sk_pop_free(untrustedCerts, free);
+	sk_X509_pop_free(untrustedCerts, X509_free);
 
 	return ret;
 	}

src/ssl-analyzer.pac

@@ -382,7 +382,7 @@ refine analyzer SSLAnalyzer += {
 			STACK_OF(X509)* untrusted_certs = 0;
 			if ( certificates->size() > 1 )
 				{
-				untrusted_certs = sk_new_null();
+				untrusted_certs = sk_X509_new_null();
 				if ( ! untrusted_certs )
 					{
 					// X509_V_ERR_OUT_OF_MEM;
@@ -405,7 +405,7 @@ refine analyzer SSLAnalyzer += {
 						return false;
 						}
 
-					sk_push(untrusted_certs, (char*) pTemp);
+					sk_X509_push(untrusted_certs, pTemp);
 					}
 				}
 
@@ -417,7 +417,7 @@ refine analyzer SSLAnalyzer += {
 				certificate_error(csc.error);
 			X509_STORE_CTX_cleanup(&csc);
 
-			sk_pop_free(untrusted_certs, free_X509);
+			sk_X509_pop_free(untrusted_certs, X509_free);
 			}
 
 		X509_free(pCert);