Threaded logging framework.

This is based on Gilbert's code but I ended up refactoring it quite a
bit. That's why I didn't do a direct merge but started with a new
branch and copied things over to adapt. It looks quite a bit different
now as I tried to generalize things a bit more to also support the
Input Framework.

The larger changes code are:

    - Moved all logging code into subdirectory src/logging/. Code
      here is in namespace "logging".

    - Moved all threading code into subdirectory src/threading/. Code
      here is in namespace "threading".

    - Introduced a central thread manager that tracks threads and is
      in charge of termination and (eventually) statistics.

    - Refactored logging independent threading code into base classes
      BasicThread and MsgThread. The former encapsulates all the
      pthread code with simple start/stop methods and provides a
      single Run() method to override.

      The latter is derived from BasicThread and adds bi-directional
      message passing between main and child threads. The hope is that
      the Input Framework can reuse this part quite directly.

    - A log writer is now split into a general WriterFrontend
      (LogEmissary in Gilbert's code) and a type-specific
      WriterBackend. Specific writers are implemented by deriving from
      the latter. (The plugin interface is almost unchanged compared
      to the 2.0 version.).

      Frontend and backend communicate via MsgThread's message
      passing.

    - MsgThread (and thus WriterBackend) has a Heartbeat() method that
      a thread can override to execute code on a regular basis. It's
      triggered roughly once a second by the main thread.

    - Integration into "the rest of Bro". Threads can send messages to
      the reporter and do debugging output; they are hooked into the
      I/O loop for sending messages back; and there's a new debugging
      stream "threading" that logs, well, threading activity.

This all seems to work for the most part, but it's not done yet.

TODO list:

    - Not all tests pass yet. In particular, diffs for the external
      tests seem to indicate some memory problem (no crashes, just an
      occasional weird character).

    - Only tested in --enable-debug mode.

    - Only tested on Linux.

    - Needs leak check.

    - Each log write is currently a single inter-thread message. Bring
      Gilbert's bulk writes back.

    - Code needs further cleanup.

    - Document the class API.

    - Document the internal structure of the logging framework.

    - Check for robustness: live traffic, aborting, signals, etc.

    - Add thread statistics to profile.log (most of the code is there).

    - Customize the OS-visible thread names on platforms that support it.

src/logging/Manager.h

@@ -0,0 +1,153 @@
+// See the file "COPYING" in the main distribution directory for copyright.
+//
+// A class managing log writers and filters.
+
+#ifndef LOGGING_MANAGER_H
+#define LOGGING_MANAGER_H
+
+#include "../Val.h"
+#include "../EventHandler.h"
+#include "../RemoteSerializer.h"
+
+class SerializationFormat;
+class RemoteSerializer;
+class RotationTimer;
+
+namespace logging {
+
+// Description of a log field.
+struct Field {
+	string name;
+	TypeTag type;
+	// inner type of sets
+	TypeTag subtype;
+
+	Field() 	{ subtype = TYPE_VOID; }
+	Field(const Field& other)
+		: name(other.name), type(other.type), subtype(other.subtype) {  }
+
+	// (Un-)serialize.
+	bool Read(SerializationFormat* fmt);
+	bool Write(SerializationFormat* fmt) const;
+};
+
+// Values as logged by a writer.
+struct Value {
+	TypeTag type;
+	bool present; // False for unset fields.
+
+	// The following union is a subset of BroValUnion, including only the
+	// types we can log directly.
+	struct set_t { bro_int_t size; Value** vals; };
+	typedef set_t vec_t;
+
+	union _val {
+		bro_int_t int_val;
+		bro_uint_t uint_val;
+		uint32 addr_val[NUM_ADDR_WORDS];
+		subnet_type subnet_val;
+		double double_val;
+		string* string_val;
+		set_t set_val;
+		vec_t vector_val;
+	} val;
+
+	Value(TypeTag arg_type = TYPE_ERROR, bool arg_present = true)
+		: type(arg_type), present(arg_present)	{}
+	~Value();
+
+	// (Un-)serialize.
+	bool Read(SerializationFormat* fmt);
+	bool Write(SerializationFormat* fmt) const;
+
+	// Returns true if the type can be logged the framework. If
+	// `atomic_only` is true, will not permit composite types.
+	static bool IsCompatibleType(BroType* t, bool atomic_only=false);
+
+private:
+	Value(const Value& other)	{ }
+};
+
+class WriterBackend;
+class WriterFrontend;
+class RotationFinishedMessage;
+
+class Manager {
+public:
+	Manager();
+	~Manager();
+
+	// These correspond to the BiFs visible on the scripting layer. The
+	// actual BiFs just forward here.
+	bool CreateStream(EnumVal* id, RecordVal* stream);
+	bool EnableStream(EnumVal* id);
+	bool DisableStream(EnumVal* id);
+	bool AddFilter(EnumVal* id, RecordVal* filter);
+	bool RemoveFilter(EnumVal* id, StringVal* name);
+	bool RemoveFilter(EnumVal* id, string name);
+	bool Write(EnumVal* id, RecordVal* columns);
+	bool SetBuf(EnumVal* id, bool enabled);	// Adjusts all writers.
+	bool Flush(EnumVal* id);		// Flushes all writers..
+
+protected:
+	friend class WriterFrontend;
+	friend class RotationFinishedMessage;
+	friend class ::RemoteSerializer;
+	friend class ::RotationTimer;
+
+	// Instantiates a new WriterBackend of the given type (note that
+	// doing so creates a new thread!). 
+	WriterBackend* CreateBackend(bro_int_t type);
+
+	//// Function also used by the RemoteSerializer.
+
+	// Takes ownership of fields.
+	WriterFrontend* CreateWriter(EnumVal* id, EnumVal* writer, string path,
+				int num_fields, Field** fields);
+
+	// Takes ownership of values..
+	bool Write(EnumVal* id, EnumVal* writer, string path,
+		   int num_fields, Value** vals);
+
+	// Announces all instantiated writers to peer.
+	void SendAllWritersTo(RemoteSerializer::PeerID peer);
+
+	//// Functions safe to use by writers.
+
+	// Signals that a file has been rotated.
+	bool FinishedRotation(WriterFrontend* writer, string new_name, string old_name,
+			      double open, double close, bool terminating);
+
+	// Reports an error for the given writer.
+	void Error(WriterFrontend* writer, const char* msg);
+
+	// Deletes the values as passed into Write().
+	void DeleteVals(int num_fields, Value** vals);
+
+private:
+	struct Filter;
+	struct Stream;
+	struct WriterInfo;
+
+	bool TraverseRecord(Stream* stream, Filter* filter, RecordType* rt,
+			    TableVal* include, TableVal* exclude, string path, list<int> indices);
+
+	Value** RecordToFilterVals(Stream* stream, Filter* filter,
+				    RecordVal* columns);
+
+	Value* ValToLogVal(Val* val, BroType* ty = 0);
+	Stream* FindStream(EnumVal* id);
+	void RemoveDisabledWriters(Stream* stream);
+	void InstallRotationTimer(WriterInfo* winfo);
+	void Rotate(WriterInfo* info);
+	Filter* FindFilter(EnumVal* id, StringVal* filter);
+	WriterInfo* FindWriter(WriterFrontend* writer);
+
+	vector<Stream *> streams;	// Indexed by stream enum.
+};
+
+}
+
+extern logging::Manager* log_mgr;
+
+#endif

src/logging/WriterBackend.cc

@@ -0,0 +1,161 @@
+// See the file "COPYING" in the main distribution directory for copyright.
+
+#include "util.h"
+
+#include "WriterBackend.h"
+#include "WriterFrontend.h"
+
+// Messages sent from backend to frontend (i.e., "OutputMessages").
+
+namespace logging  {
+
+class RotationFinishedMessage : public threading::OutputMessage<WriterFrontend>
+{
+public:
+        RotationFinishedMessage(WriterFrontend* writer, string new_name, string old_name,
+				double open, double close, bool terminating)
+		: threading::OutputMessage<WriterFrontend>("RotationFinished", writer),
+		new_name(new_name), old_name(old_name), open(open),
+		close(close), terminating(terminating)	{ }
+
+	virtual bool Process()
+		{
+		return log_mgr->FinishedRotation(Object(), new_name, old_name, open, close, terminating);
+		}
+
+private:
+        string new_name;
+        string old_name;
+        double open;
+        double close;
+        bool terminating;
+};
+
+class DisableMessage : public threading::OutputMessage<WriterFrontend>
+{
+public:
+        DisableMessage(WriterFrontend* writer)
+		: threading::OutputMessage<WriterFrontend>("Disable", writer)	{}
+
+	virtual bool Process()	{ Object()->SetDisable(); return true; }
+};
+
+}
+
+// Backend methods.
+
+using namespace logging;
+
+WriterBackend::WriterBackend(const string& name) : MsgThread(name)
+	{
+	path = "<not set>";
+	num_fields = 0;
+	fields = 0;
+	buffering = true;
+	}
+
+WriterBackend::~WriterBackend()
+	{
+	if ( fields )
+		{
+		for(int i = 0; i < num_fields; ++i)
+			delete fields[i];
+
+		delete [] fields;
+		}
+	}
+
+void WriterBackend::DeleteVals(Value** vals)
+	{
+	// Note this code is duplicated in Manager::DeleteVals().
+	for ( int i = 0; i < num_fields; i++ )
+		delete vals[i];
+
+	delete [] vals;
+	}
+
+bool WriterBackend::FinishedRotation(WriterFrontend* writer, string new_name, string old_name,
+				     double open, double close, bool terminating)
+	{
+	SendOut(new RotationFinishedMessage(writer, new_name, old_name, open, close, terminating));
+	return true;
+	}
+
+bool WriterBackend::Init(string arg_path, int arg_num_fields,
+		     const Field* const * arg_fields)
+	{
+	path = arg_path;
+	num_fields = arg_num_fields;
+	fields = arg_fields;
+
+	if ( ! DoInit(arg_path, arg_num_fields, arg_fields) )
+		return false;
+
+	return true;
+	}
+
+bool WriterBackend::Write(int arg_num_fields, Value** vals)
+	{
+	// Double-check that the arguments match. If we get this from remote,
+	// something might be mixed up.
+	if ( num_fields != arg_num_fields )
+		{
+
+#ifdef DEBUG
+		const char* msg = Fmt("Number of fields don't match in WriterBackend::Write() (%d vs. %d)",
+				      arg_num_fields, num_fields);
+		Debug(DBG_LOGGING, msg);
+#endif
+
+		DeleteVals(vals);
+		return false;
+		}
+
+	for ( int i = 0; i < num_fields; ++i )
+		{
+		if ( vals[i]->type != fields[i]->type )
+			{
+#ifdef DEBUG
+			const char* msg = Fmt("Field type doesn't match in WriterBackend::Write() (%d vs. %d)",
+					      vals[i]->type, fields[i]->type);
+			Debug(DBG_LOGGING, msg);
+#endif
+
+			DeleteVals(vals);
+			return false;
+			}
+		}
+
+	bool result = DoWrite(num_fields, fields, vals);
+
+	DeleteVals(vals);
+
+	return result;
+	}
+
+bool WriterBackend::SetBuf(bool enabled)
+	{
+	if ( enabled == buffering )
+		// No change.
+		return true;
+
+	buffering = enabled;
+
+	return DoSetBuf(enabled);
+	}
+
+bool WriterBackend::Rotate(WriterFrontend* writer, string rotated_path,
+			   double open, double close, bool terminating)
+	{
+	return DoRotate(writer, rotated_path, open, close, terminating);
+	}
+
+bool WriterBackend::Flush()
+	{
+	return DoFlush();
+	}
+
+bool WriterBackend::Finish()
+	{
+	return DoFinish();
+	}

src/logging/WriterBackend.h

@@ -0,0 +1,189 @@
+// See the file "COPYING" in the main distribution directory for copyright.
+//
+// Bridge class between main process and writer threads.
+
+#ifndef LOGGING_WRITERBACKEND_H
+#define LOGGING_WRITERBACKEND_H
+
+#include "Manager.h"
+
+#include "threading/MsgThread.h"
+
+namespace logging  {
+
+// The backend runs in its own thread, separate from the main process.
+class WriterBackend : public threading::MsgThread
+{
+public:
+	WriterBackend(const string& name);
+	virtual ~WriterBackend();
+
+	// One-time initialization of the writer to define the logged fields.
+	// Interpretation of "path" is left to the writer, and will be
+	// corresponding the value configured on the script-level.
+	// 
+	// Returns false if an error occured, in which case the writer must
+	// not be used further.
+	//
+	// The new instance takes ownership of "fields", and will delete them
+	// when done.
+	bool Init(string path, int num_fields, const Field* const * fields);
+
+	// Writes one log entry. The method takes ownership of "vals" and
+	// will return immediately after queueing the write request, which is
+	// potentially before output has actually been written out.
+	//
+	// num_fields and the types of the Values must match what was passed
+	// to Init().
+	//
+	// Returns false if an error occured, in which case the writer must
+	// not be used any further.
+	bool Write(int num_fields, Value** vals);
+
+	// Sets the buffering status for the writer, if the writer supports
+	// that. (If not, it will be ignored).
+	bool SetBuf(bool enabled);
+
+	// Flushes any currently buffered output, if the writer supports 
+	// that. (If not, it will be ignored).
+	bool Flush();
+
+	// Triggers rotation, if the writer supports that. (If not, it will
+	// be ignored).
+	bool Rotate(WriterFrontend* writer, string rotated_path, double open, double close, bool terminating);
+
+	// Finishes writing to this logger regularly. Must not be called if
+	// an error has been indicated earlier. After calling this, no
+	// further writing must be performed.
+	bool Finish();
+
+	//// Thread-safe methods that may be called from the writer
+	//// implementation.
+
+	// The following methods return the information as passed to Init().
+	const string Path() const	{ return path; }
+	int NumFields() const	{ return num_fields; }
+	const Field* const * Fields() const	{ return fields; }
+
+	// Returns the current buffering state.
+	bool IsBuf()	{ return buffering; }
+
+	// Signals to the log manager that a file has been rotated.
+	//
+	// writer: The frontend writer that triggered the rotation. This must
+	// be the value passed into DoRotate().
+	//
+	// new_name: The filename of the rotated file. old_name: The filename
+	// of the origina file.
+	//
+	// open/close: The timestamps when the original file was opened and
+	// closed, respectively.
+	//
+	// terminating: True if rotation request occured due to the main Bro
+	// process shutting down.
+	bool FinishedRotation(WriterFrontend* writer, string new_name, string old_name,
+			      double open, double close, bool terminating);
+
+protected:
+	// Methods for writers to override. If any of these returs false, it
+	// will be assumed that a fatal error has occured that prevents the
+	// writer from further operation. It will then be disabled and
+	// deleted. When returning false, the writer should also report the
+	// error via Error(). Note that even if a writer does not support the
+	// functionality for one these methods (like rotation), it must still
+	// return true if that is not to be considered a fatal error.
+	//
+	// Called once for initialization of the writer.
+	virtual bool DoInit(string path, int num_fields,
+			    const Field* const * fields) = 0;
+
+	// Called once per log entry to record.
+	virtual bool DoWrite(int num_fields, const Field* const * fields,
+			     Value** vals) = 0;
+
+	// Called when the buffering status for this writer is changed. If
+	// buffering is disabled, the writer should attempt to write out
+	// information as quickly as possible even if doing so may have a
+	// performance impact. If enabled (which is the default), it may
+	// buffer data as helpful and write it out later in a way optimized
+	// for performance. The current buffering state can be queried via
+	// IsBuf().
+	//
+	// A writer may ignore buffering changes if it doesn't fit with its
+	// semantics (but must still return true in that case).
+	virtual bool DoSetBuf(bool enabled) = 0;
+
+	// Called to flush any currently buffered output.
+	//
+	// A writer may ignore flush requests if it doesn't fit with its
+	// semantics (but must still return true in that case).
+	virtual bool DoFlush() = 0;
+
+	// Called when a log output is to be rotated. Most directly this only
+	// applies to writers writing into files, which should then close the
+	// current file and open a new one.  However, a writer may also
+	// trigger other apppropiate actions if semantics are similar.
+	// 
+	// Once rotation has finished, the implementation should call
+	// RotationDone() to signal the log manager that potential
+	// postprocessors can now run.
+	//
+	// "writer" is the frontend writer that triggered the rotation. The
+	// *only* purpose of this value is to be passed into
+	// FinishedRotation() once done. You must not otherwise access the
+	// frontend, it's running in a different thread.
+	//
+	// "rotate_path" reflects the path to where the rotated output is to
+	// be moved, with specifics depending on the writer. It should
+	// generally be interpreted in a way consistent with that of "path"
+	// as passed into DoInit(). As an example, for file-based output, 
+	// "rotate_path" could be the original filename extended with a
+	// timestamp indicating the time of the rotation.
+	// 
+	// "open" and "close" are the network time's when the *current* file
+	// was opened and closed, respectively.
+	//
+	// "terminating" indicated whether the rotation request occurs due
+	// the main Bro prcoess terminating (and not because we've reach a
+	// regularly scheduled time for rotation).
+	//
+	// A writer may ignore rotation requests if it doesn't fit with its
+	// semantics (but must still return true in that case).
+	virtual bool DoRotate(WriterFrontend* writer, string rotated_path,
+			      double open, double close, bool terminating) = 0;
+
+	// Called once on termination. Not called when any of the other
+	// methods has previously signaled an error, i.e., executing this
+	// method signals a regular shutdown of the writer.
+	virtual bool DoFinish() = 0;
+
+	// Triggered by regular heartbeat messages from the main process.
+	virtual bool DoHeartbeat(double network_time, double current_time)	{ return true; };
+
+private:
+	friend class Manager;
+
+	// When an error occurs, we call this method to set a flag marking
+	// the writer as disabled. The Manager will check the flag later and
+	// remove the writer.
+	bool Disabled()	{ return disabled; }
+
+	// Deletes the values as passed into Write().
+	void DeleteVals(Value** vals);
+
+	string path;
+	int num_fields;
+	const Field* const * fields;
+	bool buffering;
+	bool disabled;
+
+	// For implementing Fmt().
+	char* buf;
+	unsigned int buf_len;
+};
+
+
+}
+
+#endif
+

src/logging/WriterFrontend.cc

@@ -0,0 +1,175 @@
+
+#include "WriterFrontend.h"
+#include "WriterBackend.h"
+
+namespace logging  {
+
+// Messages sent from frontend to backend (i.e., "InputMessages").
+
+class InitMessage : public threading::InputMessage<WriterBackend>
+{
+public:
+	InitMessage(WriterBackend* backend, const string path, const int num_fields, const Field* const *fields)
+		: threading::InputMessage<WriterBackend>("Init", backend),
+		path(path), num_fields(num_fields), fields(fields) { }
+
+	virtual bool Process() { return Object()->Init(path, num_fields, fields); }
+
+private:
+	const string path;
+	const int num_fields;
+	const Field * const* fields;
+};
+
+class RotateMessage : public threading::InputMessage<WriterBackend>
+{
+public:
+	RotateMessage(WriterBackend* backend, WriterFrontend* frontend, const string rotated_path, const double open,
+		      const double close, const bool terminating)
+		: threading::InputMessage<WriterBackend>("Rotate", backend),
+		frontend(frontend),
+		rotated_path(rotated_path), open(open),
+		close(close), terminating(terminating) { }
+
+	virtual bool Process() { return Object()->Rotate(frontend, rotated_path, open, close, terminating); }
+
+private:
+	WriterFrontend* frontend;
+	const string rotated_path;
+	const double open;
+	const double close;
+	const bool terminating;
+};
+
+class WriteMessage : public threading::InputMessage<WriterBackend>
+{
+public:
+	WriteMessage(WriterBackend* backend, const int num_fields, Value **vals)
+		: threading::InputMessage<WriterBackend>("Write", backend),
+		num_fields(num_fields), fields(fields), vals(vals)	{}
+
+	virtual bool Process() { return Object()->Write(num_fields, vals); }
+
+private:
+	int num_fields;
+	Field* const* fields;
+	Value **vals;
+};
+
+class SetBufMessage : public threading::InputMessage<WriterBackend>
+{
+public:
+	SetBufMessage(WriterBackend* backend, const bool enabled)
+		: threading::InputMessage<WriterBackend>("SetBuf", backend),
+		enabled(enabled) { }
+
+	virtual bool Process() { return Object()->SetBuf(enabled); }
+
+private:
+	const bool enabled;
+};
+
+class FlushMessage : public threading::InputMessage<WriterBackend>
+{
+public:
+	FlushMessage(WriterBackend* backend)
+		: threading::InputMessage<WriterBackend>("Flush", backend)	{}
+
+	virtual bool Process() { return Object()->Flush(); }
+};
+
+class FinishMessage : public threading::InputMessage<WriterBackend>
+{
+public:
+	FinishMessage(WriterBackend* backend)
+		: threading::InputMessage<WriterBackend>("Finish", backend)	{}
+
+	virtual bool Process() { return Object()->Finish(); }
+};
+
+}
+
+// Frontend methods.
+
+using namespace logging;
+
+WriterFrontend::WriterFrontend(bro_int_t type)
+	{
+	disabled = initialized = false;
+	backend = log_mgr->CreateBackend(type);
+
+	assert(backend);
+	backend->Start();
+	}
+
+WriterFrontend::~WriterFrontend()
+	{
+	}
+
+void WriterFrontend::Stop()
+	{
+	SetDisable();
+	backend->Stop();
+	}
+
+void WriterFrontend::Init(string arg_path, int arg_num_fields, const Field* const * arg_fields)
+	{
+	if ( disabled )
+		return;
+
+	if ( initialized )
+		reporter->InternalError("writer initialize twice");
+
+	path = arg_path;
+	num_fields = arg_num_fields;
+	fields = arg_fields;
+
+	initialized = true;
+	backend->SendIn(new InitMessage(backend, arg_path, arg_num_fields, arg_fields));
+	}
+
+void WriterFrontend::Write(int num_fields, Value** vals)
+	{
+	if ( disabled )
+		return;
+
+	backend->SendIn(new WriteMessage(backend, num_fields, vals));
+	}
+
+void WriterFrontend::SetBuf(bool enabled)
+	{
+	if ( disabled )
+		return;
+
+	backend->SendIn(new SetBufMessage(backend, enabled));
+	}
+
+void WriterFrontend::Flush()
+	{
+	if ( disabled )
+		return;
+
+	backend->SendIn(new FlushMessage(backend));
+	}
+
+void WriterFrontend::Rotate(string rotated_path, double open, double close, bool terminating)
+	{
+	if ( disabled )
+		return;
+
+	backend->SendIn(new RotateMessage(backend, this, rotated_path, open, close, terminating));
+	}
+
+void WriterFrontend::Finish()
+	{
+	if ( disabled )
+		return;
+
+	backend->SendIn(new FinishMessage(backend));
+	}
+
+
+
+
+
+

src/logging/WriterFrontend.h

@@ -0,0 +1,66 @@
+// See the file "COPYING" in the main distribution directory for copyright.
+//
+// Bridge class between main process and writer threads.
+
+#ifndef LOGGING_WRITERFRONTEND_H
+#define LOGGING_WRITERFRONTEND_H
+
+#include "Manager.h"
+
+#include "threading/MsgThread.h"
+
+namespace logging  {
+
+class WriterBackend;
+
+class WriterFrontend {
+public:
+	WriterFrontend(bro_int_t type);
+	virtual ~WriterFrontend();
+
+	// Disables the writers and stop the backend thread.
+	void Stop();
+
+	// Interface methods to interact with the writer from the main thread
+	// (and only from the main thread), typicalli from the log manager.
+	// All these methods forward (via inter-thread messaging) to the
+	// corresponding methods of an internally created WriterBackend. See
+	// there for documentation.
+	//
+	// If any of these operations fails, the writer will be automatically
+	// (but asynchronoulsy) disabled.
+
+	void Init(string path, int num_fields, const Field* const * fields);
+	void Write(int num_fields, Value** vals);
+	void SetBuf(bool enabled);
+	void Flush();
+	void Rotate(string rotated_path, double open, double close, bool terminating);
+	void Finish();
+
+	// Calling this disable the writer. All methods calls will be no-ops
+	// from now on. The Manager will eventually remove disabled writers.
+	void SetDisable()	{ disabled = true; }
+	bool Disabled()	{ return disabled; }
+
+	const string Path() const	{ return path; }
+	int NumFields() const	{ return num_fields; }
+	const Field* const * Fields() const	{ return fields; }
+
+protected:
+	friend class Manager;
+
+
+	WriterBackend* backend;
+	bool disabled;
+	bool initialized;
+
+	string path;
+	int num_fields;
+	const Field* const * fields;
+};
+
+}
+
+
+
+#endif

src/logging/writers/Ascii.cc

@@ -0,0 +1,353 @@
+// See the file "COPYING" in the main distribution directory for copyright.
+
+#include <string>
+#include <errno.h>
+
+#include "../../NetVar.h"
+
+#include "Ascii.h"
+
+using namespace logging;
+using namespace writer;
+
+Ascii::Ascii() : WriterBackend("Ascii")
+	{
+	file = 0;
+
+	output_to_stdout = BifConst::LogAscii::output_to_stdout;
+	include_header = BifConst::LogAscii::include_header;
+
+	separator_len = BifConst::LogAscii::separator->Len();
+	separator = new char[separator_len];
+	memcpy(separator, BifConst::LogAscii::separator->Bytes(),
+	       separator_len);
+
+	set_separator_len = BifConst::LogAscii::set_separator->Len();
+	set_separator = new char[set_separator_len];
+	memcpy(set_separator, BifConst::LogAscii::set_separator->Bytes(),
+	       set_separator_len);
+
+	empty_field_len = BifConst::LogAscii::empty_field->Len();
+	empty_field = new char[empty_field_len];
+	memcpy(empty_field, BifConst::LogAscii::empty_field->Bytes(),
+	       empty_field_len);
+
+	unset_field_len = BifConst::LogAscii::unset_field->Len();
+	unset_field = new char[unset_field_len];
+	memcpy(unset_field, BifConst::LogAscii::unset_field->Bytes(),
+	       unset_field_len);
+
+	header_prefix_len = BifConst::LogAscii::header_prefix->Len();
+	header_prefix = new char[header_prefix_len];
+	memcpy(header_prefix, BifConst::LogAscii::header_prefix->Bytes(),
+	       header_prefix_len);
+
+	desc.EnableEscaping();
+	desc.AddEscapeSequence(separator, separator_len);
+	}
+
+Ascii::~Ascii()
+	{
+	if ( file )
+		fclose(file);
+
+	delete [] separator;
+	delete [] set_separator;
+	delete [] empty_field;
+	delete [] unset_field;
+	delete [] header_prefix;
+	}
+
+bool Ascii::WriteHeaderField(const string& key, const string& val)
+	{
+	string str = string(header_prefix, header_prefix_len) +
+		key + string(separator, separator_len) + val + "\n";
+
+	return (fwrite(str.c_str(), str.length(), 1, file) == 1);
+	}
+
+bool Ascii::DoInit(string path, int num_fields,
+			    const Field* const * fields)
+	{
+	if ( output_to_stdout )
+		path = "/dev/stdout";
+
+	fname = IsSpecial(path) ? path : path + "." + LogExt();
+
+	if ( ! (file = fopen(fname.c_str(), "w")) )
+		{
+		Error(Fmt("cannot open %s: %s", fname.c_str(),
+			  strerror(errno)));
+
+		return false;
+		}
+
+	if ( include_header )
+		{
+		string str = string(header_prefix, header_prefix_len)
+			+ "separator " // Always use space as separator here.
+			+ get_escaped_string(string(separator, separator_len), false)
+			+ "\n";
+
+		if( fwrite(str.c_str(), str.length(), 1, file) != 1 )
+			goto write_error;
+
+		if ( ! (WriteHeaderField("set_separator", get_escaped_string(
+		            string(set_separator, set_separator_len), false)) &&
+		        WriteHeaderField("empty_field", get_escaped_string(
+		            string(empty_field, empty_field_len), false)) &&
+		        WriteHeaderField("unset_field", get_escaped_string(
+		            string(unset_field, unset_field_len), false)) &&
+		        WriteHeaderField("path", get_escaped_string(path, false))) )
+			goto write_error;
+
+		string names;
+		string types;
+
+		for ( int i = 0; i < num_fields; ++i )
+			{
+			if ( i > 0 )
+				{
+				names += string(separator, separator_len);
+				types += string(separator, separator_len);
+				}
+
+			const Field* field = fields[i];
+			names += field->name;
+			types += type_name(field->type);
+			if ( (field->type == TYPE_TABLE) || (field->type == TYPE_VECTOR) )
+				{
+					types += "[";
+					types += type_name(field->subtype);
+					types += "]";
+				}
+			}
+
+		if ( ! (WriteHeaderField("fields", names)
+			&& WriteHeaderField("types", types)) )
+			goto write_error;
+		}
+
+	return true;
+
+write_error:
+	Error(Fmt("error writing to %s: %s", fname.c_str(), strerror(errno)));
+	return false;
+	}
+
+bool Ascii::DoFlush()
+	{
+	fflush(file);
+	return true;
+	}
+
+bool Ascii::DoFinish()
+	{
+	return true;
+	}
+
+bool Ascii::DoWriteOne(ODesc* desc, Value* val, const Field* field)
+	{
+	if ( ! val->present )
+		{
+		desc->AddN(unset_field, unset_field_len);
+		return true;
+		}
+
+	switch ( val->type ) {
+
+	case TYPE_BOOL:
+		desc->Add(val->val.int_val ? "T" : "F");
+		break;
+
+	case TYPE_INT:
+		desc->Add(val->val.int_val);
+		break;
+
+	case TYPE_COUNT:
+	case TYPE_COUNTER:
+	case TYPE_PORT:
+		desc->Add(val->val.uint_val);
+		break;
+
+	case TYPE_SUBNET:
+		desc->Add(dotted_addr(val->val.subnet_val.net));
+		desc->Add("/");
+		desc->Add(val->val.subnet_val.width);
+		break;
+
+	case TYPE_ADDR:
+		desc->Add(dotted_addr(val->val.addr_val));
+		break;
+
+	case TYPE_TIME:
+	case TYPE_INTERVAL:
+		char buf[256];
+		modp_dtoa(val->val.double_val, buf, 6);
+		desc->Add(buf);
+		break;
+
+	case TYPE_DOUBLE:
+		desc->Add(val->val.double_val);
+	break;
+
+	case TYPE_ENUM:
+	case TYPE_STRING:
+	case TYPE_FILE:
+	case TYPE_FUNC:
+		{
+		int size = val->val.string_val->size();
+		const char* data = val->val.string_val->data();
+
+		if ( ! size )
+			{
+			desc->AddN(empty_field, empty_field_len);
+			break;
+			}
+
+		if ( size == unset_field_len && memcmp(data, unset_field, size) == 0 )
+			{
+			// The value we'd write out would match exactly the
+			// place-holder we use for unset optional fields. We
+			// escape the first character so that the output
+			// won't be ambigious.
+			static const char hex_chars[] = "0123456789abcdef";
+			char hex[6] = "\\x00";
+			hex[2] = hex_chars[((*data) & 0xf0) >> 4];
+			hex[3] = hex_chars[(*data) & 0x0f];
+			desc->AddRaw(hex, 4);
+
+			++data;
+			--size;
+			}
+
+		if ( size )
+			desc->AddN(data, size);
+
+		break;
+		}
+
+	case TYPE_TABLE:
+		{
+		if ( ! val->val.set_val.size )
+			{
+			desc->AddN(empty_field, empty_field_len);
+			break;
+			}
+
+		desc->AddEscapeSequence(set_separator, set_separator_len);
+		for ( int j = 0; j < val->val.set_val.size; j++ )
+			{
+			if ( j > 0 )
+				desc->AddRaw(set_separator, set_separator_len);
+
+			if ( ! DoWriteOne(desc, val->val.set_val.vals[j], field) )
+				{
+				desc->RemoveEscapeSequence(set_separator, set_separator_len);
+				return false;
+				}
+			}
+		desc->RemoveEscapeSequence(set_separator, set_separator_len);
+
+		break;
+		}
+
+	case TYPE_VECTOR:
+		{
+		if ( ! val->val.vector_val.size )
+			{
+			desc->AddN(empty_field, empty_field_len);
+			break;
+			}
+
+		desc->AddEscapeSequence(set_separator, set_separator_len);
+		for ( int j = 0; j < val->val.vector_val.size; j++ )
+			{
+			if ( j > 0 )
+				desc->AddRaw(set_separator, set_separator_len);
+
+			if ( ! DoWriteOne(desc, val->val.vector_val.vals[j], field) )
+				{
+				desc->RemoveEscapeSequence(set_separator, set_separator_len);
+				return false;
+				}
+			}
+		desc->RemoveEscapeSequence(set_separator, set_separator_len);
+
+		break;
+		}
+
+	default:
+		Error(Fmt("unsupported field format %d for %s", val->type,
+			  field->name.c_str()));
+		return false;
+	}
+
+	return true;
+	}
+
+bool Ascii::DoWrite(int num_fields, const Field* const * fields,
+			     Value** vals)
+	{
+	if ( ! file )
+		DoInit(Path(), NumFields(), Fields());
+
+	desc.Clear();
+
+	for ( int i = 0; i < num_fields; i++ )
+		{
+		if ( i > 0 )
+			desc.AddRaw(separator, separator_len);
+
+		if ( ! DoWriteOne(&desc, vals[i], fields[i]) )
+			return false;
+		}
+
+	desc.AddRaw("\n", 1);
+
+	if ( fwrite(desc.Bytes(), desc.Len(), 1, file) != 1 )
+		{
+		Error(Fmt("error writing to %s: %s", fname.c_str(), strerror(errno)));
+		return false;
+		}
+
+	if ( IsBuf() )
+		fflush(file);
+
+	return true;
+	}
+
+bool Ascii::DoRotate(WriterFrontend* writer, string rotated_path, double open,
+		     double close, bool terminating)
+	{
+	// Don't rotate special files or if there's not one currently open.
+	if ( ! file || IsSpecial(Path()) )
+		return true;
+
+	fclose(file);
+	file = 0;
+
+	string nname = rotated_path + "." + LogExt();
+	rename(fname.c_str(), nname.c_str());
+
+	if ( ! FinishedRotation(writer, nname, fname, open, close, terminating) )
+		{
+		Error(Fmt("error rotating %s to %s", fname.c_str(), nname.c_str()));
+		return false;
+		}
+
+	return true;
+	}
+
+bool Ascii::DoSetBuf(bool enabled)
+	{
+	// Nothing to do.
+	return true;
+	}
+
+string Ascii::LogExt()
+	{
+	const char* ext = getenv("BRO_LOG_SUFFIX");
+	if ( ! ext ) ext = "log";
+	return ext;
+	}

src/logging/writers/Ascii.h

@@ -0,0 +1,64 @@
+// See the file "COPYING" in the main distribution directory for copyright.
+//
+// Log writer for delimiter-separated ASCII logs.
+
+#ifndef LOGGING_WRITER_ASCII_H
+#define LOGGING_WRITER_ASCII_H
+
+#include "../WriterBackend.h"
+
+namespace logging { namespace writer {
+
+class Ascii : public WriterBackend {
+public:
+	Ascii();
+	~Ascii();
+
+	static WriterBackend* Instantiate()	{ return new Ascii; }
+	static string LogExt();
+
+protected:
+	virtual bool DoInit(string path, int num_fields,
+			    const Field* const * fields);
+	virtual bool DoWrite(int num_fields, const Field* const * fields,
+			     Value** vals);
+	virtual bool DoSetBuf(bool enabled);
+	virtual bool DoRotate(WriterFrontend* writer, string rotated_path,
+			      double open, double close, bool terminating);
+	virtual bool DoFlush();
+	virtual bool DoFinish();
+
+private:
+	bool IsSpecial(string path) 	{ return path.find("/dev/") == 0; }
+	bool DoWriteOne(ODesc* desc, Value* val, const Field* field);
+	bool WriteHeaderField(const string& key, const string& value);
+
+	FILE* file;
+	string fname;
+	ODesc desc;
+
+	// Options set from the script-level.
+	bool output_to_stdout;
+	bool include_header;
+
+	char* separator;
+	int separator_len;
+
+	char* set_separator;
+	int set_separator_len;
+
+	char* empty_field;
+	int empty_field_len;
+
+	char* unset_field;
+	int unset_field_len;
+
+	char* header_prefix;
+	int header_prefix_len;
+};
+
+}
+}
+
+
+#endif

src/logging/writers/None.cc

@@ -0,0 +1,19 @@
+
+#include "None.h"
+
+using namespace logging;
+using namespace writer;
+
+bool None::DoRotate(WriterFrontend* writer, string rotated_path,
+		    double open, double close, bool terminating)
+	{
+	if ( ! FinishedRotation(writer, string("/dev/null"), Path(), open, close, terminating))
+		{
+		Error(Fmt("error rotating %s", Path().c_str()));
+		return false;
+		}
+
+	return true;
+	}
+
+

src/logging/writers/None.h

@@ -0,0 +1,35 @@
+// See the file "COPYING" in the main distribution directory for copyright.
+//
+// Dummy log writer that just discards everything (but still pretends to rotate).
+
+#ifndef LOGGING_WRITER_NONE_H
+#define LOGGING_WRITER_NONE_H
+
+#include "../WriterBackend.h"
+
+namespace logging { namespace writer {
+
+class None : public WriterBackend {
+public:
+	None() : WriterBackend("None")	{}
+	~None()	{};
+
+	static WriterBackend* Instantiate()	{ return new None; }
+
+protected:
+	virtual bool DoInit(string path, int num_fields,
+			    const Field* const * fields)	{ return true; }
+
+	virtual bool DoWrite(int num_fields, const Field* const * fields,
+			     Value** vals)	{ return true; }
+	virtual bool DoSetBuf(bool enabled)	{ return true; }
+	virtual bool DoRotate(WriterFrontend* writer, string rotated_path,
+			      double open, double close, bool terminating);
+	virtual bool DoFlush()	{ return true; }
+	virtual bool DoFinish()	{ return true; }
+};
+
+}
+}
+
+#endif