Threaded logging framework.

This is based on Gilbert's code but I ended up refactoring it quite a
bit. That's why I didn't do a direct merge but started with a new
branch and copied things over to adapt. It looks quite a bit different
now as I tried to generalize things a bit more to also support the
Input Framework.

The larger changes code are:

    - Moved all logging code into subdirectory src/logging/. Code
      here is in namespace "logging".

    - Moved all threading code into subdirectory src/threading/. Code
      here is in namespace "threading".

    - Introduced a central thread manager that tracks threads and is
      in charge of termination and (eventually) statistics.

    - Refactored logging independent threading code into base classes
      BasicThread and MsgThread. The former encapsulates all the
      pthread code with simple start/stop methods and provides a
      single Run() method to override.

      The latter is derived from BasicThread and adds bi-directional
      message passing between main and child threads. The hope is that
      the Input Framework can reuse this part quite directly.

    - A log writer is now split into a general WriterFrontend
      (LogEmissary in Gilbert's code) and a type-specific
      WriterBackend. Specific writers are implemented by deriving from
      the latter. (The plugin interface is almost unchanged compared
      to the 2.0 version.).

      Frontend and backend communicate via MsgThread's message
      passing.

    - MsgThread (and thus WriterBackend) has a Heartbeat() method that
      a thread can override to execute code on a regular basis. It's
      triggered roughly once a second by the main thread.

    - Integration into "the rest of Bro". Threads can send messages to
      the reporter and do debugging output; they are hooked into the
      I/O loop for sending messages back; and there's a new debugging
      stream "threading" that logs, well, threading activity.

This all seems to work for the most part, but it's not done yet.

TODO list:

    - Not all tests pass yet. In particular, diffs for the external
      tests seem to indicate some memory problem (no crashes, just an
      occasional weird character).

    - Only tested in --enable-debug mode.

    - Only tested on Linux.

    - Needs leak check.

    - Each log write is currently a single inter-thread message. Bring
      Gilbert's bulk writes back.

    - Code needs further cleanup.

    - Document the class API.

    - Document the internal structure of the logging framework.

    - Check for robustness: live traffic, aborting, signals, etc.

    - Add thread statistics to profile.log (most of the code is there).

    - Customize the OS-visible thread names on platforms that support it.

src/threading/BasicThread.cc

@@ -0,0 +1,129 @@
+
+#include <sys/signal.h>
+#include <signal.h>
+
+#include "BasicThread.h"
+#include "Manager.h"
+
+using namespace threading;
+
+BasicThread::BasicThread(const string& arg_name)
+	{
+	started = false;
+	terminating = false;
+	pthread = 0;
+
+	buf = 0;
+	buf_len = 1024;
+
+	char tmp[128];
+	snprintf(tmp, sizeof(tmp), "%s@%p", arg_name.c_str(), this);
+	name = string(tmp);
+
+	thread_mgr->AddThread(this);
+	}
+
+BasicThread::~BasicThread()
+	{
+	}
+
+const char* BasicThread::Fmt(const char* format, ...)
+	{
+	if ( ! buf )
+		buf = (char*) malloc(buf_len);
+
+	va_list al;
+	va_start(al, format);
+	int n = safe_vsnprintf(buf, buf_len, format, al);
+	va_end(al);
+
+	if ( (unsigned int) n >= buf_len )
+		{ // Not enough room, grow the buffer.
+		buf_len = n + 32;
+		buf = (char*) realloc(buf, buf_len);
+
+		// Is it portable to restart?
+		va_start(al, format);
+		n = safe_vsnprintf(buf, buf_len, format, al);
+		va_end(al);
+		}
+
+	return buf;
+	}
+
+void BasicThread::Start()
+	{
+	if ( sem_init(&terminate, 0, 0) != 0  )
+		reporter->FatalError("Cannot create terminate semaphore for thread %s", name.c_str());
+
+	if ( pthread_create(&pthread, 0, BasicThread::launcher, this) != 0 )
+		reporter->FatalError("Cannot create thread %s", name.c_str());
+
+	DBG_LOG(DBG_THREADING, "Started thread %s", name.c_str());
+
+	started = true;
+
+	OnStart();
+	}
+
+void BasicThread::Stop()
+	{
+	if ( ! started )
+		return;
+
+	if ( terminating )
+		return;
+
+	DBG_LOG(DBG_THREADING, "Signaling thread %s to terminate ...", name.c_str());
+
+	// Signal that it's ok for the thread to exit now.
+	if ( sem_post(&terminate) != 0 )
+		reporter->FatalError("Failure flagging terminate condition for thread %s", name.c_str());
+
+	terminating = true;
+
+	OnStop();
+	}
+
+void BasicThread::Join()
+	{
+	if ( ! started )
+		return;
+
+	if ( ! terminating )
+		Stop();
+
+	DBG_LOG(DBG_THREADING, "Joining thread %s ...", name.c_str());
+
+	if ( pthread_join(pthread, 0) != 0  )
+		reporter->FatalError("Failure joining thread %s", name.c_str());
+
+	sem_destroy(&terminate);
+
+	DBG_LOG(DBG_THREADING, "Done with thread %s", name.c_str());
+
+	pthread = 0;
+	}
+
+void* BasicThread::launcher(void *arg)
+	{
+	BasicThread* thread = (BasicThread *)arg;
+
+	// Block signals in thread. We handle signals only in the main
+	// process.
+	sigset_t mask_set;
+	sigfillset(&mask_set);
+	int res = pthread_sigmask(SIG_BLOCK, &mask_set, 0);
+	assert(res == 0);  //
+
+	// Run thread's main function.
+	thread->Run();
+
+	// Wait until somebody actually wants us to terminate.
+
+	if ( sem_wait(&thread->terminate) != 0 )
+		reporter->FatalError("Failure flagging terminate condition for thread %s", thread->Name().c_str());
+
+	return 0;
+	}
+