Threaded logging framework.

This is based on Gilbert's code but I ended up refactoring it quite a
bit. That's why I didn't do a direct merge but started with a new
branch and copied things over to adapt. It looks quite a bit different
now as I tried to generalize things a bit more to also support the
Input Framework.

The larger changes code are:

    - Moved all logging code into subdirectory src/logging/. Code
      here is in namespace "logging".

    - Moved all threading code into subdirectory src/threading/. Code
      here is in namespace "threading".

    - Introduced a central thread manager that tracks threads and is
      in charge of termination and (eventually) statistics.

    - Refactored logging independent threading code into base classes
      BasicThread and MsgThread. The former encapsulates all the
      pthread code with simple start/stop methods and provides a
      single Run() method to override.

      The latter is derived from BasicThread and adds bi-directional
      message passing between main and child threads. The hope is that
      the Input Framework can reuse this part quite directly.

    - A log writer is now split into a general WriterFrontend
      (LogEmissary in Gilbert's code) and a type-specific
      WriterBackend. Specific writers are implemented by deriving from
      the latter. (The plugin interface is almost unchanged compared
      to the 2.0 version.).

      Frontend and backend communicate via MsgThread's message
      passing.

    - MsgThread (and thus WriterBackend) has a Heartbeat() method that
      a thread can override to execute code on a regular basis. It's
      triggered roughly once a second by the main thread.

    - Integration into "the rest of Bro". Threads can send messages to
      the reporter and do debugging output; they are hooked into the
      I/O loop for sending messages back; and there's a new debugging
      stream "threading" that logs, well, threading activity.

This all seems to work for the most part, but it's not done yet.

TODO list:

    - Not all tests pass yet. In particular, diffs for the external
      tests seem to indicate some memory problem (no crashes, just an
      occasional weird character).

    - Only tested in --enable-debug mode.

    - Only tested on Linux.

    - Needs leak check.

    - Each log write is currently a single inter-thread message. Bring
      Gilbert's bulk writes back.

    - Code needs further cleanup.

    - Document the class API.

    - Document the internal structure of the logging framework.

    - Check for robustness: live traffic, aborting, signals, etc.

    - Add thread statistics to profile.log (most of the code is there).

    - Customize the OS-visible thread names on platforms that support it.

src/logging/WriterFrontend.cc

@@ -0,0 +1,175 @@
+
+#include "WriterFrontend.h"
+#include "WriterBackend.h"
+
+namespace logging  {
+
+// Messages sent from frontend to backend (i.e., "InputMessages").
+
+class InitMessage : public threading::InputMessage<WriterBackend>
+{
+public:
+	InitMessage(WriterBackend* backend, const string path, const int num_fields, const Field* const *fields)
+		: threading::InputMessage<WriterBackend>("Init", backend),
+		path(path), num_fields(num_fields), fields(fields) { }
+
+	virtual bool Process() { return Object()->Init(path, num_fields, fields); }
+
+private:
+	const string path;
+	const int num_fields;
+	const Field * const* fields;
+};
+
+class RotateMessage : public threading::InputMessage<WriterBackend>
+{
+public:
+	RotateMessage(WriterBackend* backend, WriterFrontend* frontend, const string rotated_path, const double open,
+		      const double close, const bool terminating)
+		: threading::InputMessage<WriterBackend>("Rotate", backend),
+		frontend(frontend),
+		rotated_path(rotated_path), open(open),
+		close(close), terminating(terminating) { }
+
+	virtual bool Process() { return Object()->Rotate(frontend, rotated_path, open, close, terminating); }
+
+private:
+	WriterFrontend* frontend;
+	const string rotated_path;
+	const double open;
+	const double close;
+	const bool terminating;
+};
+
+class WriteMessage : public threading::InputMessage<WriterBackend>
+{
+public:
+	WriteMessage(WriterBackend* backend, const int num_fields, Value **vals)
+		: threading::InputMessage<WriterBackend>("Write", backend),
+		num_fields(num_fields), fields(fields), vals(vals)	{}
+
+	virtual bool Process() { return Object()->Write(num_fields, vals); }
+
+private:
+	int num_fields;
+	Field* const* fields;
+	Value **vals;
+};
+
+class SetBufMessage : public threading::InputMessage<WriterBackend>
+{
+public:
+	SetBufMessage(WriterBackend* backend, const bool enabled)
+		: threading::InputMessage<WriterBackend>("SetBuf", backend),
+		enabled(enabled) { }
+
+	virtual bool Process() { return Object()->SetBuf(enabled); }
+
+private:
+	const bool enabled;
+};
+
+class FlushMessage : public threading::InputMessage<WriterBackend>
+{
+public:
+	FlushMessage(WriterBackend* backend)
+		: threading::InputMessage<WriterBackend>("Flush", backend)	{}
+
+	virtual bool Process() { return Object()->Flush(); }
+};
+
+class FinishMessage : public threading::InputMessage<WriterBackend>
+{
+public:
+	FinishMessage(WriterBackend* backend)
+		: threading::InputMessage<WriterBackend>("Finish", backend)	{}
+
+	virtual bool Process() { return Object()->Finish(); }
+};
+
+}
+
+// Frontend methods.
+
+using namespace logging;
+
+WriterFrontend::WriterFrontend(bro_int_t type)
+	{
+	disabled = initialized = false;
+	backend = log_mgr->CreateBackend(type);
+
+	assert(backend);
+	backend->Start();
+	}
+
+WriterFrontend::~WriterFrontend()
+	{
+	}
+
+void WriterFrontend::Stop()
+	{
+	SetDisable();
+	backend->Stop();
+	}
+
+void WriterFrontend::Init(string arg_path, int arg_num_fields, const Field* const * arg_fields)
+	{
+	if ( disabled )
+		return;
+
+	if ( initialized )
+		reporter->InternalError("writer initialize twice");
+
+	path = arg_path;
+	num_fields = arg_num_fields;
+	fields = arg_fields;
+
+	initialized = true;
+	backend->SendIn(new InitMessage(backend, arg_path, arg_num_fields, arg_fields));
+	}
+
+void WriterFrontend::Write(int num_fields, Value** vals)
+	{
+	if ( disabled )
+		return;
+
+	backend->SendIn(new WriteMessage(backend, num_fields, vals));
+	}
+
+void WriterFrontend::SetBuf(bool enabled)
+	{
+	if ( disabled )
+		return;
+
+	backend->SendIn(new SetBufMessage(backend, enabled));
+	}
+
+void WriterFrontend::Flush()
+	{
+	if ( disabled )
+		return;
+
+	backend->SendIn(new FlushMessage(backend));
+	}
+
+void WriterFrontend::Rotate(string rotated_path, double open, double close, bool terminating)
+	{
+	if ( disabled )
+		return;
+
+	backend->SendIn(new RotateMessage(backend, this, rotated_path, open, close, terminating));
+	}
+
+void WriterFrontend::Finish()
+	{
+	if ( disabled )
+		return;
+
+	backend->SendIn(new FinishMessage(backend));
+	}
+
+
+
+
+
+