Threaded logging framework.

This is based on Gilbert's code but I ended up refactoring it quite a
bit. That's why I didn't do a direct merge but started with a new
branch and copied things over to adapt. It looks quite a bit different
now as I tried to generalize things a bit more to also support the
Input Framework.

The larger changes code are:

    - Moved all logging code into subdirectory src/logging/. Code
      here is in namespace "logging".

    - Moved all threading code into subdirectory src/threading/. Code
      here is in namespace "threading".

    - Introduced a central thread manager that tracks threads and is
      in charge of termination and (eventually) statistics.

    - Refactored logging independent threading code into base classes
      BasicThread and MsgThread. The former encapsulates all the
      pthread code with simple start/stop methods and provides a
      single Run() method to override.

      The latter is derived from BasicThread and adds bi-directional
      message passing between main and child threads. The hope is that
      the Input Framework can reuse this part quite directly.

    - A log writer is now split into a general WriterFrontend
      (LogEmissary in Gilbert's code) and a type-specific
      WriterBackend. Specific writers are implemented by deriving from
      the latter. (The plugin interface is almost unchanged compared
      to the 2.0 version.).

      Frontend and backend communicate via MsgThread's message
      passing.

    - MsgThread (and thus WriterBackend) has a Heartbeat() method that
      a thread can override to execute code on a regular basis. It's
      triggered roughly once a second by the main thread.

    - Integration into "the rest of Bro". Threads can send messages to
      the reporter and do debugging output; they are hooked into the
      I/O loop for sending messages back; and there's a new debugging
      stream "threading" that logs, well, threading activity.

This all seems to work for the most part, but it's not done yet.

TODO list:

    - Not all tests pass yet. In particular, diffs for the external
      tests seem to indicate some memory problem (no crashes, just an
      occasional weird character).

    - Only tested in --enable-debug mode.

    - Only tested on Linux.

    - Needs leak check.

    - Each log write is currently a single inter-thread message. Bring
      Gilbert's bulk writes back.

    - Code needs further cleanup.

    - Document the class API.

    - Document the internal structure of the logging framework.

    - Check for robustness: live traffic, aborting, signals, etc.

    - Add thread statistics to profile.log (most of the code is there).

    - Customize the OS-visible thread names on platforms that support it.

src/threading/MsgThread.h

@@ -0,0 +1,157 @@
+
+#ifndef THREADING_MSGTHREAD_H
+#define THREADING_MSGTHREAD_H
+
+#include <pthread.h>
+
+#include "DebugLogger.h"
+
+#include "BasicThread.h"
+#include "Queue.h"
+
+namespace threading {
+
+class BasicInputMessage;
+class BasicOutputMessage;
+class HeartbeatMessage;
+
+class MsgThread : public BasicThread
+{
+public:
+	MsgThread(const string& name);
+
+	void SendIn(BasicInputMessage* msg)	{ return SendIn(msg, false); }
+	void SendOut(BasicOutputMessage* msg)	{ return SendOut(msg, false); }
+
+	BasicOutputMessage* RetrieveOut();
+
+	// Report an informational message, nothing that needs specific
+	// attention.
+	void Info(const char* msg);
+
+	// Report a warning that may indicate a problem.
+	void Warning(const char* msg);
+
+	// Report a non-fatal error. Processing proceeds normally after the error
+	// has been reported.
+	void Error(const char* msg);
+
+	// Report a fatal error. Bro will terminate after the message has been
+	// reported.
+	void FatalError(const char* msg);
+
+	// Report a fatal error. Bro will terminate after the message has been
+	// reported and always generate a core dump.
+	void FatalErrorWithCore(const char* msg);
+
+	// Report about a potential internal problem. Bro will continue
+	// normally.
+	void InternalWarning(const char* msg);
+
+	// Report an internal program error. Bro will terminate with a core
+	// dump after the message has been reported.
+	void InternalError(const char* msg);
+
+#ifdef DEBUG
+	// Records a debug message for the given stream.
+	void Debug(DebugStream stream, const char* msg);
+#endif
+
+	void Heartbeat();
+
+	struct Stats
+		{
+		uint64_t sent_in;
+		uint64_t sent_out;
+		uint64_t pending_in;
+		uint64_t pending_out;
+		};
+
+	void GetStats(Stats* stats);
+
+protected:
+	friend class HeartbeatMessage;
+
+	virtual void Run();
+	virtual void OnStop();
+
+	virtual bool DoHeartbeat(double network_time, double current_time)	{ return true; }
+
+private:
+	friend class Manager;
+
+	BasicInputMessage* RetrieveIn();
+
+	void SendIn(BasicInputMessage* msg, bool force);
+	void SendOut(BasicOutputMessage* msg, bool force);
+
+	bool HasIn()	{ return queue_in.Ready(); }
+	bool HasOut()	{ return queue_out.Ready(); }
+
+	Queue_<BasicInputMessage *> queue_in;
+	Queue_<BasicOutputMessage *> queue_out;
+
+	uint64_t cnt_sent_in;
+	uint64_t cnt_sent_out;
+};
+
+class Message
+{
+public:
+	virtual ~Message();
+
+	const string& Name() const { return name; }
+
+	virtual bool Process() = 0; // Thread will be terminated if returngin false.
+
+protected:
+	Message(const string& arg_name)	{ name = arg_name; }
+
+private:
+	string name;
+};
+
+class BasicInputMessage : public Message
+{
+protected:
+	BasicInputMessage(const string& name) : Message(name)	{}
+};
+
+class BasicOutputMessage : public Message
+{
+protected:
+	BasicOutputMessage(const string& name) : Message(name)	{}
+};
+
+template<typename O>
+class InputMessage : public BasicInputMessage
+{
+public:
+	O* Object() const { return object; }
+
+protected:
+	InputMessage(const string& name, O* arg_object) : BasicInputMessage(name)
+		{ object = arg_object; }
+
+private:
+	O* object;
+};
+
+template<typename O>
+class OutputMessage : public BasicOutputMessage
+{
+public:
+	O* Object() const { return object; }
+
+protected:
+	OutputMessage(const string& name, O* arg_object) : BasicOutputMessage(name)
+		{ object = arg_object; }
+
+private:
+	O* object;
+};
+
+}
+
+
+#endif