Merge remote-tracking branch 'origin/topic/jsiwek/sftp-pp'

* origin/topic/jsiwek/sftp-pp:
  Add SFTP log postprocessor that transfers logs to remote hosts.
  Add FAQ entry about disabling NIC offloading features.

Closes #737.

CHANGES

@@ -1,4 +1,13 @@
 
+2.0-beta-181 | 2012-01-08 20:49:04 -0800
+
+  * Add SFTP log postprocessor that transfers logs to remote hosts.
+    Addresses #737. (Jon Siwek)
+
+  * Add FAQ entry about disabling NIC offloading features. (Jon Siwek)
+    
+  * Add a file NEWS with release notes. (Robin Sommer)
+
 2.0-beta-177 | 2012-01-05 15:01:07 -0800
 
   * Replace the --snaplen/-l command line option with a

VERSION

@@ -1 +1 @@
-2.0-beta-177
+2.0-beta-181

doc/faq.rst

@@ -28,6 +28,23 @@ Here are some pointers to more information:
   Lothar Braun et. al evaluates packet capture performance on
   commodity hardware
 
+Are there any gotchas regarding interface configuration for live capture?  Or why might I be seeing abnormally large packets much greater than interface MTU?
+-------------------------------------------------------------------------------------------------------------------------------------------------------------
+
+Some NICs offload the reassembly of traffic into "superpackets" so that
+fewer packets are then passed up the stack (e.g. "TCP segmentation
+offload", or "generic segmentation offload").  The result is that the
+capturing application will observe packets much larger than the MTU size
+of the interface they were captured from and may also interfere with the
+maximum packet capture length, ``snaplen``, so it's a good idea to disable
+an interface's offloading features.
+
+You can use the ``ethtool`` program on Linux to view and disable
+offloading features of an interface.  See this page for more explicit
+directions:
+
+http://securityonion.blogspot.com/2011/10/when-is-full-packet-capture-not-full.html
+
 What does an error message like ``internal error: NB-DNS error`` mean?
 ---------------------------------------------------------------------------------------------------------------------------------
 

scripts/base/frameworks/logging/postprocessors/__load__.bro

@@ -1 +1,2 @@
 @load ./scp
+@load ./sftp

scripts/base/frameworks/logging/postprocessors/sftp.bro

@@ -0,0 +1,65 @@
+##! This script defines a postprocessing function that can be applied
+##! to a logging filter in order to automatically SFTP
+##! a log stream (or a subset of it) to a remote host at configurable
+##! rotation time intervals.  Generally, to use this functionality
+##! you must handle the :bro:id:`bro_init` event and do the following
+##! in your handler:
+##!
+##! 1) Create a new :bro:type:`Log::Filter` record that defines a name/path,
+##!    rotation interval, and set the ``postprocessor`` to
+##!    :bro:id:`Log::sftp_postprocessor`.
+##! 2) Add the filter to a logging stream using :bro:id:`Log::add_filter`.
+##! 3) Add a table entry to :bro:id:`Log::sftp_destinations` for the filter's
+##!    writer/path pair which defines a set of :bro:type:`Log::SFTPDestination`
+##!    records.
+
+module Log;
+
+export {
+	## Securely transfers the rotated-log to all the remote hosts
+	## defined in :bro:id:`Log::sftp_destinations` and then deletes
+	## the local copy of the rotated-log.  It's not active when
+	## reading from trace files.
+	##
+	## info: A record holding meta-information about the log file to be
+	##       postprocessed.
+	##
+	## Returns: True if sftp system command was initiated or
+	##          if no destination was configured for the log as described
+	##          by *info*.
+	global sftp_postprocessor: function(info: Log::RotationInfo): bool;
+
+	## A container that describes the remote destination for the SFTP command,
+	## comprised of the username, host, and path at which to upload the file.
+	type SFTPDestination: record {
+		## The remote user to log in as.  A trust mechanism should be
+		## pre-established.
+		user: string;
+		## The remote host to which to transfer logs.
+		host: string;
+		## The path/directory on the remote host to send logs.
+		path: string;
+	};
+
+	## A table indexed by a particular log writer and filter path, that yields
+	## a set remote destinations.  The :bro:id:`Log::sftp_postprocessor`
+	## function queries this table upon log rotation and performs a secure
+	## transfer of the rotated-log to each destination in the set.  This
+	## table can be modified at run-time.
+	global sftp_destinations: table[Writer, string] of set[SFTPDestination];
+}
+
+function sftp_postprocessor(info: Log::RotationInfo): bool
+	{
+	if ( reading_traces() || [info$writer, info$path] !in sftp_destinations )
+		return T;
+
+	local command = "";
+	for ( d in sftp_destinations[info$writer, info$path] )
+		command += fmt("echo put %s %s | sftp -b - %s@%s;", info$fname, d$path,
+		               d$user, d$host);
+
+	command += fmt("/bin/rm %s", info$fname);
+	system(command);
+	return T;
+	}