Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-04 07:38:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Threaded logging framework.

Browse source 
This is based on Gilbert's code but I ended up refactoring it quite a
bit. That's why I didn't do a direct merge but started with a new
branch and copied things over to adapt. It looks quite a bit different
now as I tried to generalize things a bit more to also support the
Input Framework.

The larger changes code are:

    - Moved all logging code into subdirectory src/logging/. Code
      here is in namespace "logging".

    - Moved all threading code into subdirectory src/threading/. Code
      here is in namespace "threading".

    - Introduced a central thread manager that tracks threads and is
      in charge of termination and (eventually) statistics.

    - Refactored logging independent threading code into base classes
      BasicThread and MsgThread. The former encapsulates all the
      pthread code with simple start/stop methods and provides a
      single Run() method to override.

      The latter is derived from BasicThread and adds bi-directional
      message passing between main and child threads. The hope is that
      the Input Framework can reuse this part quite directly.

    - A log writer is now split into a general WriterFrontend
      (LogEmissary in Gilbert's code) and a type-specific
      WriterBackend. Specific writers are implemented by deriving from
      the latter. (The plugin interface is almost unchanged compared
      to the 2.0 version.).

      Frontend and backend communicate via MsgThread's message
      passing.

    - MsgThread (and thus WriterBackend) has a Heartbeat() method that
      a thread can override to execute code on a regular basis. It's
      triggered roughly once a second by the main thread.

    - Integration into "the rest of Bro". Threads can send messages to
      the reporter and do debugging output; they are hooked into the
      I/O loop for sending messages back; and there's a new debugging
      stream "threading" that logs, well, threading activity.

This all seems to work for the most part, but it's not done yet.

TODO list:

    - Not all tests pass yet. In particular, diffs for the external
      tests seem to indicate some memory problem (no crashes, just an
      occasional weird character).

    - Only tested in --enable-debug mode.

    - Only tested on Linux.

    - Needs leak check.

    - Each log write is currently a single inter-thread message. Bring
      Gilbert's bulk writes back.

    - Code needs further cleanup.

    - Document the class API.

    - Document the internal structure of the logging framework.

    - Check for robustness: live traffic, aborting, signals, etc.

    - Add thread statistics to profile.log (most of the code is there).

    - Customize the OS-visible thread names on platforms that support it.
This commit is contained in:
Robin Sommer 2012-01-26 17:47:36 -08:00
parent 60ae6f01d1
commit e4e770d475
28 changed files with 1745 additions and 503 deletions
Download patch file Download diff file Expand all files Collapse all files

22
src/RemoteSerializer.cc
View file

@ -183,8 +183,8 @@
#include "Sessions.h"
#include "File.h"
#include "Conn.h"
#include "LogMgr.h"
#include "Reporter.h"
#include "logging/Manager.h"
extern "C" {
#include "setsignal.h"
@ -2476,7 +2476,7 @@ bool RemoteSerializer::ProcessRemotePrint()
return true;
}
bool RemoteSerializer::SendLogCreateWriter(EnumVal* id, EnumVal* writer, string path, int num_fields, const LogField* const * fields)
bool RemoteSerializer::SendLogCreateWriter(EnumVal* id, EnumVal* writer, string path, int num_fields, const logging::Field* const * fields)
{
loop_over_list(peers, i)
{
@ -2486,7 +2486,7 @@ bool RemoteSerializer::SendLogCreateWriter(EnumVal* id, EnumVal* writer, string
return true;
}
bool RemoteSerializer::SendLogCreateWriter(PeerID peer_id, EnumVal* id, EnumVal* writer, string path, int num_fields, const LogField* const * fields)
bool RemoteSerializer::SendLogCreateWriter(PeerID peer_id, EnumVal* id, EnumVal* writer, string path, int num_fields, const logging::Field* const * fields)
{
SetErrorDescr("logging");
@ -2540,7 +2540,7 @@ error:
return false;
}
bool RemoteSerializer::SendLogWrite(EnumVal* id, EnumVal* writer, string path, int num_fields, const LogVal* const * vals)
bool RemoteSerializer::SendLogWrite(EnumVal* id, EnumVal* writer, string path, int num_fields, const logging::Value* const * vals)
{
loop_over_list(peers, i)
{
@ -2550,7 +2550,7 @@ bool RemoteSerializer::SendLogWrite(EnumVal* id, EnumVal* writer, string path, i
return true;
}
bool RemoteSerializer::SendLogWrite(Peer* peer, EnumVal* id, EnumVal* writer, string path, int num_fields, const LogVal* const * vals)
bool RemoteSerializer::SendLogWrite(Peer* peer, EnumVal* id, EnumVal* writer, string path, int num_fields, const logging::Value* const * vals)
{
if ( peer->phase != Peer::HANDSHAKE && peer->phase != Peer::RUNNING )
return false;
@ -2641,7 +2641,7 @@ bool RemoteSerializer::ProcessLogCreateWriter()
EnumVal* id_val = 0;
EnumVal* writer_val = 0;
LogField** fields = 0;
logging::Field** fields = 0;
BinarySerializationFormat fmt;
fmt.StartRead(current_args->data, current_args->len);
@ -2658,11 +2658,11 @@ bool RemoteSerializer::ProcessLogCreateWriter()
if ( ! success )
goto error;
fields = new LogField* [num_fields];
fields = new logging::Field* [num_fields];
for ( int i = 0; i < num_fields; i++ )
{
fields[i] = new LogField;
fields[i] = new logging::Field;
if ( ! fields[i]->Read(&fmt) )
goto error;
}
@ -2703,7 +2703,7 @@ bool RemoteSerializer::ProcessLogWrite()
// Unserialize one entry.
EnumVal* id_val = 0;
EnumVal* writer_val = 0;
LogVal** vals = 0;
logging::Value** vals = 0;
int id, writer;
string path;
@ -2717,11 +2717,11 @@ bool RemoteSerializer::ProcessLogWrite()
if ( ! success )
goto error;
vals = new LogVal* [num_fields];
vals = new logging::Value* [num_fields];
for ( int i = 0; i < num_fields; i++ )
{
vals[i] = new LogVal;
vals[i] = new logging::Value;
if ( ! vals[i]->Read(&fmt) )
goto error;
}