Threaded logging framework.

This is based on Gilbert's code but I ended up refactoring it quite a
bit. That's why I didn't do a direct merge but started with a new
branch and copied things over to adapt. It looks quite a bit different
now as I tried to generalize things a bit more to also support the
Input Framework.

The larger changes code are:

    - Moved all logging code into subdirectory src/logging/. Code
      here is in namespace "logging".

    - Moved all threading code into subdirectory src/threading/. Code
      here is in namespace "threading".

    - Introduced a central thread manager that tracks threads and is
      in charge of termination and (eventually) statistics.

    - Refactored logging independent threading code into base classes
      BasicThread and MsgThread. The former encapsulates all the
      pthread code with simple start/stop methods and provides a
      single Run() method to override.

      The latter is derived from BasicThread and adds bi-directional
      message passing between main and child threads. The hope is that
      the Input Framework can reuse this part quite directly.

    - A log writer is now split into a general WriterFrontend
      (LogEmissary in Gilbert's code) and a type-specific
      WriterBackend. Specific writers are implemented by deriving from
      the latter. (The plugin interface is almost unchanged compared
      to the 2.0 version.).

      Frontend and backend communicate via MsgThread's message
      passing.

    - MsgThread (and thus WriterBackend) has a Heartbeat() method that
      a thread can override to execute code on a regular basis. It's
      triggered roughly once a second by the main thread.

    - Integration into "the rest of Bro". Threads can send messages to
      the reporter and do debugging output; they are hooked into the
      I/O loop for sending messages back; and there's a new debugging
      stream "threading" that logs, well, threading activity.

This all seems to work for the most part, but it's not done yet.

TODO list:

    - Not all tests pass yet. In particular, diffs for the external
      tests seem to indicate some memory problem (no crashes, just an
      occasional weird character).

    - Only tested in --enable-debug mode.

    - Only tested on Linux.

    - Needs leak check.

    - Each log write is currently a single inter-thread message. Bring
      Gilbert's bulk writes back.

    - Code needs further cleanup.

    - Document the class API.

    - Document the internal structure of the logging framework.

    - Check for robustness: live traffic, aborting, signals, etc.

    - Add thread statistics to profile.log (most of the code is there).

    - Customize the OS-visible thread names on platforms that support it.

src/logging/WriterBackend.cc

@@ -0,0 +1,161 @@
+// See the file "COPYING" in the main distribution directory for copyright.
+
+#include "util.h"
+
+#include "WriterBackend.h"
+#include "WriterFrontend.h"
+
+// Messages sent from backend to frontend (i.e., "OutputMessages").
+
+namespace logging  {
+
+class RotationFinishedMessage : public threading::OutputMessage<WriterFrontend>
+{
+public:
+        RotationFinishedMessage(WriterFrontend* writer, string new_name, string old_name,
+				double open, double close, bool terminating)
+		: threading::OutputMessage<WriterFrontend>("RotationFinished", writer),
+		new_name(new_name), old_name(old_name), open(open),
+		close(close), terminating(terminating)	{ }
+
+	virtual bool Process()
+		{
+		return log_mgr->FinishedRotation(Object(), new_name, old_name, open, close, terminating);
+		}
+
+private:
+        string new_name;
+        string old_name;
+        double open;
+        double close;
+        bool terminating;
+};
+
+class DisableMessage : public threading::OutputMessage<WriterFrontend>
+{
+public:
+        DisableMessage(WriterFrontend* writer)
+		: threading::OutputMessage<WriterFrontend>("Disable", writer)	{}
+
+	virtual bool Process()	{ Object()->SetDisable(); return true; }
+};
+
+}
+
+// Backend methods.
+
+using namespace logging;
+
+WriterBackend::WriterBackend(const string& name) : MsgThread(name)
+	{
+	path = "<not set>";
+	num_fields = 0;
+	fields = 0;
+	buffering = true;
+	}
+
+WriterBackend::~WriterBackend()
+	{
+	if ( fields )
+		{
+		for(int i = 0; i < num_fields; ++i)
+			delete fields[i];
+
+		delete [] fields;
+		}
+	}
+
+void WriterBackend::DeleteVals(Value** vals)
+	{
+	// Note this code is duplicated in Manager::DeleteVals().
+	for ( int i = 0; i < num_fields; i++ )
+		delete vals[i];
+
+	delete [] vals;
+	}
+
+bool WriterBackend::FinishedRotation(WriterFrontend* writer, string new_name, string old_name,
+				     double open, double close, bool terminating)
+	{
+	SendOut(new RotationFinishedMessage(writer, new_name, old_name, open, close, terminating));
+	return true;
+	}
+
+bool WriterBackend::Init(string arg_path, int arg_num_fields,
+		     const Field* const * arg_fields)
+	{
+	path = arg_path;
+	num_fields = arg_num_fields;
+	fields = arg_fields;
+
+	if ( ! DoInit(arg_path, arg_num_fields, arg_fields) )
+		return false;
+
+	return true;
+	}
+
+bool WriterBackend::Write(int arg_num_fields, Value** vals)
+	{
+	// Double-check that the arguments match. If we get this from remote,
+	// something might be mixed up.
+	if ( num_fields != arg_num_fields )
+		{
+
+#ifdef DEBUG
+		const char* msg = Fmt("Number of fields don't match in WriterBackend::Write() (%d vs. %d)",
+				      arg_num_fields, num_fields);
+		Debug(DBG_LOGGING, msg);
+#endif
+
+		DeleteVals(vals);
+		return false;
+		}
+
+	for ( int i = 0; i < num_fields; ++i )
+		{
+		if ( vals[i]->type != fields[i]->type )
+			{
+#ifdef DEBUG
+			const char* msg = Fmt("Field type doesn't match in WriterBackend::Write() (%d vs. %d)",
+					      vals[i]->type, fields[i]->type);
+			Debug(DBG_LOGGING, msg);
+#endif
+
+			DeleteVals(vals);
+			return false;
+			}
+		}
+
+	bool result = DoWrite(num_fields, fields, vals);
+
+	DeleteVals(vals);
+
+	return result;
+	}
+
+bool WriterBackend::SetBuf(bool enabled)
+	{
+	if ( enabled == buffering )
+		// No change.
+		return true;
+
+	buffering = enabled;
+
+	return DoSetBuf(enabled);
+	}
+
+bool WriterBackend::Rotate(WriterFrontend* writer, string rotated_path,
+			   double open, double close, bool terminating)
+	{
+	return DoRotate(writer, rotated_path, open, close, terminating);
+	}
+
+bool WriterBackend::Flush()
+	{
+	return DoFlush();
+	}
+
+bool WriterBackend::Finish()
+	{
+	return DoFinish();
+	}