diff --git a/CHANGES b/CHANGES index b1c5f03a1f..9da6a5d5d5 100644 --- a/CHANGES +++ b/CHANGES @@ -1,4 +1,119 @@ +2.3-203 | 2014-09-29 20:06:54 -0700 + + * Fix to use length parameter in DNP3 time conversion correctly now. + (Robin Sommer) + +2.3-202 | 2014-09-29 17:05:18 -0700 + + * New SSL extension type from IANA and a few other SSL const + changes. (Johanna Amann) + + * Make unexpected pipe errors fatal as precaution. Addresses + BIT-1260. (Jon Siwek) + + * Adding a function for DNP3 to translate the timestamp format. (Hui + Lin) + +2.3-197 | 2014-09-29 10:42:01 -0500 + + * Fix possible seg fault in TCP reassembler. (Jon Siwek) + +2.3-196 | 2014-09-25 17:53:27 -0700 + + * Changing prefix for packet sources/dumper from ':' to '%'. + Addresses BIT-1249. (Robin Sommer) + + * Remove timeouts from remote communication loop. The select() now + blocks until there's work to do instead of relying on a small + timeout value which can cause unproductive use of cpu cycles. (Jon + Siwek) + + * Improve error message when failing to activate a plugin. Also fix + a unit test helper script that checks plugin availability. (Jon + Siwek) + +2.3-183 | 2014-09-24 10:08:04 -0500 + + * Add a "node" field to Intel::Seen struture and intel.log to + indicate which node discovered a hit on an intel item. (Seth Hall) + + * BIT-1261: Fixes to plugin quick start doc. (Jon Siwek) + +2.3-180 | 2014-09-22 12:52:41 -0500 + + * BIT-1259: Fix issue w/ duplicate TCP reassembly deliveries. + (Jon Siwek) + +2.3-178 | 2014-09-18 14:29:46 -0500 + + * BIT-1256: Fix file analysis events from coming after bro_done(). + (Jon Siwek) + +2.3-177 | 2014-09-17 09:41:27 -0500 + + * Documentation fixes. (Chris Mavrakis) + +2.3-174 | 2014-09-17 09:37:09 -0500 + + * Fixed some "make doc" warnings caused by reST formatting + (Daniel Thayer). + +2.3-172 | 2014-09-15 13:38:52 -0500 + + * Remove unneeded allocations for HTTP messages. (Jon Siwek) + +2.3-171 | 2014-09-15 11:14:57 -0500 + + * Fix a compile error on systems without pcap-int.h. (Jon Siwek) + +2.3-170 | 2014-09-12 19:28:01 -0700 + + * Fix incorrect data delivery skips after gap in HTTP Content-Range. + Addresses BIT-1247. (Jon Siwek) + + * Fix file analysis placement of data after gap in HTTP + Content-Range. Addresses BIT-1248. (Jon Siwek) + + * Fix issue w/ TCP reassembler not delivering some segments. + Addresses BIT-1246. (Jon Siwek) + + * Fix MIME entity file data/gap ordering and raise http_entity_data + in line with data arrival. Addresses BIT-1240. (Jon Siwek) + + * Implement file ID caching for MIME_Mail. (Jon Siwek) + + * Fix a compile error. (Jon Siwek) + +2.3-161 | 2014-09-09 12:35:38 -0500 + + * Bugfixes and test updates/additions. (Robin Sommer) + + * Interface tweaks and docs for PktSrc/PktDumper. (Robin Sommer) + + * Moving PCAP-related bifs to iosource/pcap.bif. (Robin Sommer) + + * Moving some of the BPF filtering code into base class. + This will allow packet sources that don't support BPF natively to + emulate the filtering via libpcap. (Robin Sommer) + + * Removing FlowSrc. (Robin Sommer) + + * Removing remaining pieces of the 2ndary path, and left-over + files of packet sorter. (Robin Sommer) + + * A bunch of infrastructure work to move IOSource, IOSourceRegistry + (now iosource::Manager) and PktSrc/PktDumper code into iosource/, + and over to a plugin structure. (Robin Sommer) + +2.3-137 | 2014-09-08 19:01:13 -0500 + + * Fix Broxygen's rendering of opaque types. (Jon Siwek) + +2.3-136 | 2014-09-07 20:50:46 -0700 + + * Change more http links to https. (Johanna Amann) + 2.3-134 | 2014-09-04 16:16:36 -0700 * Fixed a number of issues with OCSP reply validation. Addresses diff --git a/NEWS b/NEWS index f06115d4ea..6da13833c3 100644 --- a/NEWS +++ b/NEWS @@ -22,7 +22,7 @@ New Functionality plugin can furthermore hook into Bro's processing a number of places to add custom logic. - See http://www.bro.org/sphinx-git/devel/plugins.html for more + See https://www.bro.org/sphinx-git/devel/plugins.html for more information on writing plugins. Changed Functionality diff --git a/VERSION b/VERSION index 0154ba2563..3065f78463 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.3-134 +2.3-203 diff --git a/aux/binpac b/aux/binpac index 4e5969f5a4..3a4684801a 160000 --- a/aux/binpac +++ b/aux/binpac @@ -1 +1 @@ -Subproject commit 4e5969f5a40f5cc192a751375cb61131d32c0fc1 +Subproject commit 3a4684801aafa0558383199e9abd711650b53af9 diff --git a/aux/bro-aux b/aux/bro-aux index 181f084432..95afe42e74 160000 --- a/aux/bro-aux +++ b/aux/bro-aux @@ -1 +1 @@ -Subproject commit 181f084432e277f899140647d9b788059b3cccb1 +Subproject commit 95afe42e7474113a16cb2cb09ebdf8b552c59744 diff --git a/aux/broccoli b/aux/broccoli index 6be54279bb..33d0ed4a54 160000 --- a/aux/broccoli +++ b/aux/broccoli @@ -1 +1 @@ -Subproject commit 6be54279bb7ecb5e03d8bcdc7660d323dc4de1bc +Subproject commit 33d0ed4a54a6ecf08a0b5fe18831aa413b437066 diff --git a/aux/broctl b/aux/broctl index f0e0efda05..2f808bc854 160000 --- a/aux/broctl +++ b/aux/broctl @@ -1 +1 @@ -Subproject commit f0e0efda05e4b20924efc1b826ad5d85c8b65f83 +Subproject commit 2f808bc8541378b1a4953cca02c58c43945d154f diff --git a/aux/plugins b/aux/plugins index 6de518922e..ad600b5bdc 160000 --- a/aux/plugins +++ b/aux/plugins @@ -1 +1 @@ -Subproject commit 6de518922e5f89d52d831ea6fb6adb7fff94437e +Subproject commit ad600b5bdcd56a2723e323c0f2c8e1708956ca4f diff --git a/cmake b/cmake index aa15263ae3..03de0cc467 160000 --- a/cmake +++ b/cmake @@ -1 +1 @@ -Subproject commit aa15263ae39667e5e9bd73690b05aa4af9147ca3 +Subproject commit 03de0cc467d2334dcb851eddd843d59fef217909 diff --git a/doc/devel/plugins.rst b/doc/devel/plugins.rst index cc34c399d2..76f5c75a68 100644 --- a/doc/devel/plugins.rst +++ b/doc/devel/plugins.rst @@ -57,10 +57,10 @@ called ``Demo::Rot13``. The ``init-plugin`` script puts a number of files in place. The full layout is described later. For now, all we need is -``src/functions.bif``. It's initially empty, but we'll add our new bif +``src/rot13.bif``. It's initially empty, but we'll add our new bif there as follows:: - # cat scripts/functions.bif + # cat src/rot13.bif module CaesarCipher; function rot13%(s: string%) : string @@ -73,23 +73,25 @@ there as follows:: *p = (*p - b + 13) % 26 + b; } - return new StringVal(new BroString(1, rot13, strlen(rot13))); + BroString* bs = new BroString(1, reinterpret_cast(rot13), + strlen(rot13)); + return new StringVal(bs); %} The syntax of this file is just like any other ``*.bif`` file; we won't go into it here. Now we can already compile our plugin, we just need to tell the -Makefile put in place by ``init-plugin`` where the Bro source tree is -located (Bro needs to have been built there first):: +configure script put in place by ``init-plugin`` where the Bro source +tree is located (Bro needs to have been built there first):: - # make BRO=/path/to/bro/dist + # ./configure --bro-dist=/path/to/bro/dist && make [... cmake output ...] Now our ``rot13-plugin`` directory has everything that it needs for Bro to recognize it as a dynamic plugin. Once we point Bro to it, it will pull it in automatically, as we can check with the ``-N`` -option: +option:: # export BRO_PLUGIN_PATH=/path/to/rot13-plugin # bro -N @@ -100,7 +102,7 @@ option: That looks quite good, except for the dummy description that we should replace with something nicer so that users will know what our plugin is about. We do this by editing the ``config.description`` line in -``src/Plugin.cc``, like this: +``src/Plugin.cc``, like this:: [...] plugin::Configuration Configure() @@ -193,7 +195,7 @@ directory. A directory with the plugin's custom Bro scripts. When the plugin gets activated, this directory will be automatically added to ``BROPATH``, so that any scripts/modules inside can be - ``@load``ed. + "@load"ed. ``scripts``/__load__.bro A Bro script that will be loaded immediately when the plugin gets diff --git a/doc/install/install.rst b/doc/install/install.rst index 9a258773ce..0052acafb0 100644 --- a/doc/install/install.rst +++ b/doc/install/install.rst @@ -180,7 +180,7 @@ automatically. Finally, use ``make install-aux`` to install some of the other programs that are in the ``aux/bro-aux`` directory. OpenBSD users, please see our `FAQ -`_ if you are having +`_ if you are having problems installing Bro. Finally, if you want to build the Bro documentation (not required, because diff --git a/doc/quickstart/index.rst b/doc/quickstart/index.rst index 173373c769..bb642ee75a 100644 --- a/doc/quickstart/index.rst +++ b/doc/quickstart/index.rst @@ -1,5 +1,5 @@ -.. _FAQ: http://www.bro.org/documentation/faq.html +.. _FAQ: //www.bro.org/documentation/faq.html .. _quickstart: diff --git a/doc/scripting/index.rst b/doc/scripting/index.rst index efb9aced15..559e131b8f 100644 --- a/doc/scripting/index.rst +++ b/doc/scripting/index.rst @@ -730,7 +730,7 @@ Bro supports ``usec``, ``msec``, ``sec``, ``min``, ``hr``, or ``day`` which repr microseconds, milliseconds, seconds, minutes, hours, and days respectively. In fact, the interval data type allows for a surprising amount of variation in its definitions. There can be a space between -the numeric constant or they can crammed together like a temporal +the numeric constant or they can be crammed together like a temporal portmanteau. The time unit can be either singular or plural. All of this adds up to to the fact that both ``42hrs`` and ``42 hr`` are perfectly valid and logically equivalent in Bro. The point, however, @@ -819,7 +819,7 @@ with the ``typedef`` and ``struct`` keywords, Bro allows you to cobble together new data types to suit the needs of your situation. When combined with the ``type`` keyword, ``record`` can generate a -composite type. We have, in fact, already encountered a a complex +composite type. We have, in fact, already encountered a complex example of the ``record`` data type in the earlier sections, the :bro:type:`connection` record passed to many events. Another one, :bro:type:`Conn::Info`, which corresponds to the fields logged into @@ -1014,8 +1014,8 @@ remaining logs to factor.log. :lines: 38-62 :linenos: -To dynamically alter the file in which a stream writes its logs a -filter can specify function returns a string to be used as the +To dynamically alter the file in which a stream writes its logs, a +filter can specify a function that returns a string to be used as the filename for the current call to ``Log::write``. The definition for this function has to take as its parameters a ``Log::ID`` called id, a string called ``path`` and the appropriate record type for the logs called diff --git a/scripts/base/frameworks/files/main.bro b/scripts/base/frameworks/files/main.bro index 5f0e5a2e00..d680c467b6 100644 --- a/scripts/base/frameworks/files/main.bro +++ b/scripts/base/frameworks/files/main.bro @@ -264,10 +264,10 @@ export { ## Returns: The set of MIME types. global registered_mime_types: function(tag: Analyzer::Tag) : set[string]; - ## Returns a table of all MIME-type-to-analyzer mappings currently registered. + ## Returns a table of all MIME-type-to-analyzer mappings currently registered. ## - ## Returns: A table mapping each analyzer to the set of MIME types registered for - ## it. + ## Returns: A table mapping each analyzer to the set of MIME types + ## registered for it. global all_registered_mime_types: function() : table[Analyzer::Tag] of set[string]; ## Event that can be handled to access the Info record as it is sent on diff --git a/scripts/base/frameworks/intel/main.bro b/scripts/base/frameworks/intel/main.bro index fb3a9a3613..5b31dd964e 100644 --- a/scripts/base/frameworks/intel/main.bro +++ b/scripts/base/frameworks/intel/main.bro @@ -81,6 +81,9 @@ export { ## Where the data was discovered. where: Where &log; + ## The name of the node where the match was discovered. + node: string &optional &log; + ## If the data was discovered within a connection, the ## connection record should go here to give context to the data. conn: connection &optional; @@ -240,6 +243,11 @@ function Intel::seen(s: Seen) s$indicator_type = Intel::ADDR; } + if ( ! s?$node ) + { + s$node = peer_description; + } + if ( have_full_data ) { local items = get_items(s); diff --git a/scripts/base/init-bare.bro b/scripts/base/init-bare.bro index 1199bdd7bc..efce524fc5 100644 --- a/scripts/base/init-bare.bro +++ b/scripts/base/init-bare.bro @@ -2485,8 +2485,7 @@ type http_message_stat: record { header_length: count; }; -## Maximum number of HTTP entity data delivered to events. The amount of data -## can be limited for better performance, zero disables truncation. +## Maximum number of HTTP entity data delivered to events. ## ## .. bro:see:: http_entity_data skip_http_entity_data skip_http_data global http_entity_data_delivery_size = 1500 &redef; diff --git a/scripts/base/protocols/ssl/consts.bro b/scripts/base/protocols/ssl/consts.bro index a19aaecbe5..54952988f0 100644 --- a/scripts/base/protocols/ssl/consts.bro +++ b/scripts/base/protocols/ssl/consts.bro @@ -30,6 +30,7 @@ export { const HELLO_REQUEST = 0; const CLIENT_HELLO = 1; const SERVER_HELLO = 2; + const HELLO_VERIFY_REQUEST = 3; # RFC 6347 const SESSION_TICKET = 4; # RFC 5077 const CERTIFICATE = 11; const SERVER_KEY_EXCHANGE = 12; @@ -40,6 +41,7 @@ export { const FINISHED = 20; const CERTIFICATE_URL = 21; # RFC 3546 const CERTIFICATE_STATUS = 22; # RFC 3546 + const SUPPLEMENTAL_DATA = 23; # RFC 4680 ## Mapping between numeric codes and human readable strings for alert ## levels. @@ -112,7 +114,8 @@ export { [19] = "client_certificate_type", [20] = "server_certificate_type", [21] = "padding", # temporary till 2015-03-12 - [22] = "encrypt_then_mac", # temporary till 2015-06-05 + [22] = "encrypt_then_mac", + [23] = "extended_master_secret", # temporary till 2015-09-26 [35] = "SessionTicket TLS", [40] = "extended_random", [13172] = "next_protocol_negotiation", diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 04867b7189..9f94f8f1e3 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -157,6 +157,7 @@ add_subdirectory(analyzer) add_subdirectory(broxygen) add_subdirectory(file_analysis) add_subdirectory(input) +add_subdirectory(iosource) add_subdirectory(logging) add_subdirectory(probabilistic) @@ -254,7 +255,6 @@ set(bro_SRCS Anon.cc Attr.cc Base64.cc - BPF_Program.cc Brofiler.cc BroString.cc CCL.cc @@ -279,14 +279,13 @@ set(bro_SRCS EventRegistry.cc Expr.cc File.cc - FlowSrc.cc + Flare.cc Frag.cc Frame.cc Func.cc Hash.cc ID.cc IntSet.cc - IOSource.cc IP.cc IPAddr.cc List.cc @@ -299,7 +298,7 @@ set(bro_SRCS OSFinger.cc PacketFilter.cc PersistenceSerializer.cc - PktSrc.cc + Pipe.cc PolicyFile.cc PrefixTable.cc PriorityQueue.cc diff --git a/src/ChunkedIO.cc b/src/ChunkedIO.cc index 54e2e59575..722b209bcd 100644 --- a/src/ChunkedIO.cc +++ b/src/ChunkedIO.cc @@ -210,6 +210,7 @@ bool ChunkedIOFd::WriteChunk(Chunk* chunk, bool partial) else pending_head = pending_tail = q; + write_flare.Fire(); return Flush(); } @@ -232,6 +233,7 @@ bool ChunkedIOFd::PutIntoWriteBuffer(Chunk* chunk) write_len += len; delete chunk; + write_flare.Fire(); if ( network_time - last_flush > 0.005 ) FlushWriteBuffer(); @@ -269,6 +271,10 @@ bool ChunkedIOFd::FlushWriteBuffer() if ( unsigned(written) == len ) { write_pos = write_len = 0; + + if ( ! pending_head ) + write_flare.Extinguish(); + return true; } @@ -318,7 +324,12 @@ bool ChunkedIOFd::Flush() } } - return FlushWriteBuffer(); + bool rval = FlushWriteBuffer(); + + if ( ! pending_head && write_len == 0 ) + write_flare.Extinguish(); + + return rval; } uint32 ChunkedIOFd::ChunkAvailable() @@ -394,6 +405,9 @@ bool ChunkedIOFd::Read(Chunk** chunk, bool may_block) #ifdef DEBUG_COMMUNICATION AddToBuffer("", true); #endif + if ( ! ChunkAvailable() ) + read_flare.Extinguish(); + return false; } @@ -402,9 +416,15 @@ bool ChunkedIOFd::Read(Chunk** chunk, bool may_block) #ifdef DEBUG_COMMUNICATION AddToBuffer("", true); #endif + read_flare.Extinguish(); return true; } + if ( ChunkAvailable() ) + read_flare.Fire(); + else + read_flare.Extinguish(); + #ifdef DEBUG if ( *chunk ) DBG_LOG(DBG_CHUNKEDIO, "read of size %d %s[%s]", @@ -481,6 +501,9 @@ bool ChunkedIOFd::ReadChunk(Chunk** chunk, bool may_block) read_pos = 0; read_len = bytes_left; + if ( ! ChunkAvailable() ) + read_flare.Extinguish(); + // If allowed, wait a bit for something to read. if ( may_block ) { @@ -607,6 +630,14 @@ bool ChunkedIOFd::IsFillingUp() return stats.pending > MAX_BUFFERED_CHUNKS_SOFT; } +iosource::FD_Set ChunkedIOFd::ExtraReadFDs() const + { + iosource::FD_Set rval; + rval.Insert(write_flare.FD()); + rval.Insert(read_flare.FD()); + return rval; + } + void ChunkedIOFd::Clear() { while ( pending_head ) @@ -618,6 +649,9 @@ void ChunkedIOFd::Clear() } pending_head = pending_tail = 0; + + if ( write_len == 0 ) + write_flare.Extinguish(); } const char* ChunkedIOFd::Error() @@ -830,6 +864,7 @@ bool ChunkedIOSSL::Write(Chunk* chunk) else write_head = write_tail = q; + write_flare.Fire(); Flush(); return true; } @@ -935,6 +970,7 @@ bool ChunkedIOSSL::Flush() write_state = LEN; } + write_flare.Extinguish(); return true; } @@ -1104,6 +1140,13 @@ bool ChunkedIOSSL::IsFillingUp() return false; } +iosource::FD_Set ChunkedIOSSL::ExtraReadFDs() const + { + iosource::FD_Set rval; + rval.Insert(write_flare.FD()); + return rval; + } + void ChunkedIOSSL::Clear() { while ( write_head ) @@ -1114,6 +1157,7 @@ void ChunkedIOSSL::Clear() write_head = next; } write_head = write_tail = 0; + write_flare.Extinguish(); } const char* ChunkedIOSSL::Error() diff --git a/src/ChunkedIO.h b/src/ChunkedIO.h index a9865e4c05..b590453a72 100644 --- a/src/ChunkedIO.h +++ b/src/ChunkedIO.h @@ -6,7 +6,8 @@ #include "config.h" #include "List.h" #include "util.h" - +#include "Flare.h" +#include "iosource/FD_Set.h" #include #ifdef NEED_KRB5_H @@ -95,6 +96,11 @@ public: // Returns underlying fd if available, -1 otherwise. virtual int Fd() { return -1; } + // Returns supplementary file descriptors that become read-ready in order + // to signal that there is some work that can be performed. + virtual iosource::FD_Set ExtraReadFDs() const + { return iosource::FD_Set(); } + // Makes sure that no additional protocol data is written into // the output stream. If this is activated, the output cannot // be read again by any of these classes! @@ -177,6 +183,7 @@ public: virtual void Clear(); virtual bool Eof() { return eof; } virtual int Fd() { return fd; } + virtual iosource::FD_Set ExtraReadFDs() const; virtual void Stats(char* buffer, int length); private: @@ -240,6 +247,8 @@ private: ChunkQueue* pending_tail; pid_t pid; + bro::Flare write_flare; + bro::Flare read_flare; }; // Chunked I/O using an SSL connection. @@ -262,6 +271,7 @@ public: virtual void Clear(); virtual bool Eof() { return eof; } virtual int Fd() { return socket; } + virtual iosource::FD_Set ExtraReadFDs() const; virtual void Stats(char* buffer, int length); private: @@ -303,6 +313,8 @@ private: // One SSL for all connections. static SSL_CTX* ctx; + + bro::Flare write_flare; }; #include @@ -328,6 +340,8 @@ public: virtual bool Eof() { return io->Eof(); } virtual int Fd() { return io->Fd(); } + virtual iosource::FD_Set ExtraReadFDs() const + { return io->ExtraReadFDs(); } virtual void Stats(char* buffer, int length); void EnableCompression(int level) diff --git a/src/DNS_Mgr.cc b/src/DNS_Mgr.cc index 9188d61b96..2c049ba803 100644 --- a/src/DNS_Mgr.cc +++ b/src/DNS_Mgr.cc @@ -35,6 +35,7 @@ #include "Net.h" #include "Var.h" #include "Reporter.h" +#include "iosource/Manager.h" extern "C" { extern int select(int, fd_set *, fd_set *, fd_set *, struct timeval *); @@ -404,17 +405,17 @@ DNS_Mgr::~DNS_Mgr() delete [] dir; } -bool DNS_Mgr::Init() +void DNS_Mgr::InitPostScript() { if ( did_init ) - return true; + return; const char* cache_dir = dir ? dir : "."; if ( mode == DNS_PRIME && ! ensure_dir(cache_dir) ) { did_init = 0; - return false; + return; } cache_name = new char[strlen(cache_dir) + 64]; @@ -433,14 +434,12 @@ bool DNS_Mgr::Init() did_init = 1; - io_sources.Register(this, true); + iosource_mgr->Register(this, true); // We never set idle to false, having the main loop only calling us from // time to time. If we're issuing more DNS requests than we can handle // in this way, we are having problems anyway ... - idle = true; - - return true; + SetIdle(true); } static TableVal* fake_name_lookup_result(const char* name) @@ -1217,9 +1216,10 @@ void DNS_Mgr::IssueAsyncRequests() } } -void DNS_Mgr::GetFds(int* read, int* write, int* except) +void DNS_Mgr::GetFds(iosource::FD_Set* read, iosource::FD_Set* write, + iosource::FD_Set* except) { - *read = nb_dns_fd(nb_dns); + read->Insert(nb_dns_fd(nb_dns)); } double DNS_Mgr::NextTimestamp(double* network_time) diff --git a/src/DNS_Mgr.h b/src/DNS_Mgr.h index 7864505add..d8f420e6cc 100644 --- a/src/DNS_Mgr.h +++ b/src/DNS_Mgr.h @@ -12,7 +12,7 @@ #include "BroList.h" #include "Dict.h" #include "EventHandler.h" -#include "IOSource.h" +#include "iosource/IOSource.h" #include "IPAddr.h" class Val; @@ -40,12 +40,12 @@ enum DNS_MgrMode { // Number of seconds we'll wait for a reply. #define DNS_TIMEOUT 5 -class DNS_Mgr : public IOSource { +class DNS_Mgr : public iosource::IOSource { public: DNS_Mgr(DNS_MgrMode mode); virtual ~DNS_Mgr(); - bool Init(); + void InitPostScript(); void Flush(); // Looks up the address or addresses of the given host, and returns @@ -132,7 +132,8 @@ protected: void DoProcess(bool flush); // IOSource interface. - virtual void GetFds(int* read, int* write, int* except); + virtual void GetFds(iosource::FD_Set* read, iosource::FD_Set* write, + iosource::FD_Set* except); virtual double NextTimestamp(double* network_time); virtual void Process(); virtual const char* Tag() { return "DNS_Mgr"; } diff --git a/src/DebugLogger.cc b/src/DebugLogger.cc index cab2b0494f..6f025e3c2b 100644 --- a/src/DebugLogger.cc +++ b/src/DebugLogger.cc @@ -16,9 +16,10 @@ DebugLogger::Stream DebugLogger::streams[NUM_DBGS] = { { "compressor", 0, false }, {"string", 0, false }, { "notifiers", 0, false }, { "main-loop", 0, false }, { "dpd", 0, false }, { "tm", 0, false }, - { "logging", 0, false }, {"input", 0, false }, + { "logging", 0, false }, {"input", 0, false }, { "threading", 0, false }, { "file_analysis", 0, false }, - { "plugins", 0, false }, { "broxygen", 0, false } + { "plugins", 0, false }, { "broxygen", 0, false }, + { "pktio", 0, false} }; DebugLogger::DebugLogger(const char* filename) diff --git a/src/DebugLogger.h b/src/DebugLogger.h index 845cf7a1a4..9cd09dada1 100644 --- a/src/DebugLogger.h +++ b/src/DebugLogger.h @@ -31,6 +31,7 @@ enum DebugStream { DBG_FILE_ANALYSIS, // File analysis DBG_PLUGINS, // Plugin system DBG_BROXYGEN, // Broxygen + DBG_PKTIO, // Packet sources and dumpers. NUM_DBGS // Has to be last }; diff --git a/src/Flare.cc b/src/Flare.cc new file mode 100644 index 0000000000..dcb5fa2c1f --- /dev/null +++ b/src/Flare.cc @@ -0,0 +1,74 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#include "Flare.h" +#include "Reporter.h" +#include +#include +#include + +using namespace bro; + +Flare::Flare() + : pipe(FD_CLOEXEC, FD_CLOEXEC, O_NONBLOCK, O_NONBLOCK) + { + } + +static void bad_pipe_op(const char* which) + { + char buf[256]; + strerror_r(errno, buf, sizeof(buf)); + reporter->FatalErrorWithCore("unexpected pipe %s failure: %s", which, buf); + } + +void Flare::Fire() + { + char tmp; + + for ( ; ; ) + { + int n = write(pipe.WriteFD(), &tmp, 1); + + if ( n > 0 ) + // Success -- wrote a byte to pipe. + break; + + if ( n < 0 ) + { + if ( errno == EAGAIN ) + // Success: pipe is full and just need at least one byte in it. + break; + + if ( errno == EINTR ) + // Interrupted: try again. + continue; + + bad_pipe_op("write"); + } + + // No error, but didn't write a byte: try again. + } + } + +void Flare::Extinguish() + { + char tmp[256]; + + for ( ; ; ) + { + int n = read(pipe.ReadFD(), &tmp, sizeof(tmp)); + + if ( n >= 0 ) + // Pipe may not be empty yet: try again. + continue; + + if ( errno == EAGAIN ) + // Success: pipe is now empty. + break; + + if ( errno == EINTR ) + // Interrupted: try again. + continue; + + bad_pipe_op("read"); + } + } diff --git a/src/Flare.h b/src/Flare.h new file mode 100644 index 0000000000..fb6d7cae4c --- /dev/null +++ b/src/Flare.h @@ -0,0 +1,44 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#ifndef BRO_FLARE_H +#define BRO_FLARE_H + +#include "Pipe.h" + +namespace bro { + +class Flare { +public: + + /** + * Create a flare object that can be used to signal a "ready" status via + * a file descriptor that may be integrated with select(), poll(), etc. + * Not thread-safe, but that should only require Fire()/Extinguish() calls + * to be made mutually exclusive (across all copies of a Flare). + */ + Flare(); + + /** + * @return a file descriptor that will become ready if the flare has been + * Fire()'d and not yet Extinguished()'d. + */ + int FD() const + { return pipe.ReadFD(); } + + /** + * Put the object in the "ready" state. + */ + void Fire(); + + /** + * Take the object out of the "ready" state. + */ + void Extinguish(); + +private: + Pipe pipe; +}; + +} // namespace bro + +#endif // BRO_FLARE_H diff --git a/src/FlowSrc.cc b/src/FlowSrc.cc deleted file mode 100644 index 8eed94fcea..0000000000 --- a/src/FlowSrc.cc +++ /dev/null @@ -1,228 +0,0 @@ -// See the file "COPYING" in the main distribution directory for copyright. -// -// Written by Bernhard Ager, TU Berlin (2006/2007). - -#include -#include -#include -#include - -#include "FlowSrc.h" -#include "Net.h" -#include "analyzer/protocol/netflow/netflow_pac.h" -#include - -FlowSrc::FlowSrc() - { // TODO: v9. - selectable_fd = -1; - idle = false; - data = 0; - pdu_len = -1; - exporter_ip = 0; - current_timestamp = next_timestamp = 0.0; - netflow_analyzer = new binpac::NetFlow::NetFlow_Analyzer(); - } - -FlowSrc::~FlowSrc() - { - delete netflow_analyzer; - } - -void FlowSrc::GetFds(int* read, int* write, int* except) - { - if ( selectable_fd >= 0 ) - *read = selectable_fd; - } - -double FlowSrc::NextTimestamp(double* network_time) - { - if ( ! data && ! ExtractNextPDU() ) - return -1.0; - else - return next_timestamp; - } - -void FlowSrc::Process() - { - if ( ! data && ! ExtractNextPDU() ) - return; - - // This is normally done by calling net_packet_dispatch(), - // but as we don't have a packet to dispatch ... - net_update_time(next_timestamp); - expire_timers(); - - netflow_analyzer->downflow()->set_exporter_ip(exporter_ip); - - // We handle exceptions in NewData (might have changed w/ new binpac). - netflow_analyzer->NewData(0, data, data + pdu_len); - data = 0; - } - -void FlowSrc::Close() - { - safe_close(selectable_fd); - } - - -FlowSocketSrc::~FlowSocketSrc() - { - } - -int FlowSocketSrc::ExtractNextPDU() - { - sockaddr_in from; - socklen_t fromlen = sizeof(from); - pdu_len = recvfrom(selectable_fd, buffer, NF_MAX_PKT_SIZE, 0, - (struct sockaddr*) &from, &fromlen); - if ( pdu_len < 0 ) - { - reporter->Error("problem reading NetFlow data from socket"); - data = 0; - next_timestamp = -1.0; - closed = 1; - return 0; - } - - if ( fromlen != sizeof(from) ) - { - reporter->Error("malformed NetFlow PDU"); - return 0; - } - - data = buffer; - exporter_ip = from.sin_addr.s_addr; - next_timestamp = current_time(); - - if ( next_timestamp < current_timestamp ) - next_timestamp = current_timestamp; - else - current_timestamp = next_timestamp; - - return 1; - } - -FlowSocketSrc::FlowSocketSrc(const char* listen_parms) - { - int n = strlen(listen_parms) + 1; - - char laddr[n], port[n], ident[n]; - laddr[0] = port[0] = ident[0] = '\0'; - - int ret = sscanf(listen_parms, "%[^:]:%[^=]=%s", laddr, port, ident); - if ( ret < 2 ) - { - snprintf(errbuf, BRO_FLOW_ERRBUF_SIZE, - "parsing your listen-spec went nuts: laddr='%s', port='%s'\n", - laddr[0] ? laddr : "", port[0] ? port : ""); - closed = 1; - return; - } - - const char* id = (ret == 3) ? ident : listen_parms; - netflow_analyzer->downflow()->set_identifier(id); - - struct addrinfo aiprefs = { - 0, PF_INET, SOCK_DGRAM, IPPROTO_UDP, 0, NULL, NULL, NULL - }; - struct addrinfo* ainfo = 0; - if ( (ret = getaddrinfo(laddr, port, &aiprefs, &ainfo)) != 0 ) - { - snprintf(errbuf, BRO_FLOW_ERRBUF_SIZE, - "getaddrinfo(%s, %s, ...): %s", - laddr, port, gai_strerror(ret)); - closed = 1; - return; - } - - if ( (selectable_fd = socket (PF_INET, SOCK_DGRAM, 0)) < 0 ) - { - snprintf(errbuf, BRO_FLOW_ERRBUF_SIZE, - "socket: %s", strerror(errno)); - closed = 1; - goto cleanup; - } - - if ( bind (selectable_fd, ainfo->ai_addr, ainfo->ai_addrlen) < 0 ) - { - snprintf(errbuf, BRO_FLOW_ERRBUF_SIZE, - "bind: %s", strerror(errno)); - closed = 1; - goto cleanup; - } - -cleanup: - freeaddrinfo(ainfo); - } - - -FlowFileSrc::~FlowFileSrc() - { - delete [] readfile; - } - -int FlowFileSrc::ExtractNextPDU() - { - FlowFileSrcPDUHeader pdu_header; - - if ( read(selectable_fd, &pdu_header, sizeof(pdu_header)) < - int(sizeof(pdu_header)) ) - return Error(errno, "read header"); - - if ( pdu_header.pdu_length > NF_MAX_PKT_SIZE ) - { - reporter->Error("NetFlow packet too long"); - - // Safely skip over the too-long PDU. - if ( lseek(selectable_fd, pdu_header.pdu_length, SEEK_CUR) < 0 ) - return Error(errno, "lseek"); - return 0; - } - - if ( read(selectable_fd, buffer, pdu_header.pdu_length) < - pdu_header.pdu_length ) - return Error(errno, "read data"); - - if ( next_timestamp < pdu_header.network_time ) - { - next_timestamp = pdu_header.network_time; - current_timestamp = pdu_header.network_time; - } - else - current_timestamp = next_timestamp; - - data = buffer; - pdu_len = pdu_header.pdu_length; - exporter_ip = pdu_header.ipaddr; - - return 1; - } - -FlowFileSrc::FlowFileSrc(const char* readfile) - { - int n = strlen(readfile) + 1; - char ident[n]; - this->readfile = new char[n]; - - int ret = sscanf(readfile, "%[^=]=%s", this->readfile, ident); - const char* id = (ret == 2) ? ident : this->readfile; - netflow_analyzer->downflow()->set_identifier(id); - - selectable_fd = open(this->readfile, O_RDONLY); - if ( selectable_fd < 0 ) - { - closed = 1; - snprintf(errbuf, BRO_FLOW_ERRBUF_SIZE, - "open: %s", strerror(errno)); - } - } - -int FlowFileSrc::Error(int errlvl, const char* errmsg) - { - snprintf(errbuf, BRO_FLOW_ERRBUF_SIZE, - "%s: %s", errmsg, strerror(errlvl)); - data = 0; - next_timestamp = -1.0; - closed = 1; - return 0; - } diff --git a/src/FlowSrc.h b/src/FlowSrc.h deleted file mode 100644 index 03dda2761d..0000000000 --- a/src/FlowSrc.h +++ /dev/null @@ -1,84 +0,0 @@ -// See the file "COPYING" in the main distribution directory for copyright. -// -// Written by Bernhard Ager, TU Berlin (2006/2007). - -#ifndef flowsrc_h -#define flowsrc_h - -#include "IOSource.h" -#include "NetVar.h" -#include "binpac.h" - -#define BRO_FLOW_ERRBUF_SIZE 512 - -// TODO: 1500 is enough for v5 - how about the others? -// 65536 would be enough for any UDP packet. -#define NF_MAX_PKT_SIZE 8192 - -struct FlowFileSrcPDUHeader { - double network_time; - int pdu_length; - uint32 ipaddr; -}; - -// Avoid including netflow_pac.h by explicitly declaring the NetFlow_Analyzer. -namespace binpac { - namespace NetFlow { - class NetFlow_Analyzer; - } -} - -class FlowSrc : public IOSource { -public: - virtual ~FlowSrc(); - - // IOSource interface: - bool IsReady(); - void GetFds(int* read, int* write, int* except); - double NextTimestamp(double* network_time); - void Process(); - - const char* Tag() { return "FlowSrc"; } - const char* ErrorMsg() const { return errbuf; } - -protected: - FlowSrc(); - - virtual int ExtractNextPDU() = 0; - virtual void Close(); - - int selectable_fd; - - double current_timestamp; - double next_timestamp; - binpac::NetFlow::NetFlow_Analyzer* netflow_analyzer; - - u_char buffer[NF_MAX_PKT_SIZE]; - u_char* data; - int pdu_len; - uint32 exporter_ip; // in network byte order - - char errbuf[BRO_FLOW_ERRBUF_SIZE]; -}; - -class FlowSocketSrc : public FlowSrc { -public: - FlowSocketSrc(const char* listen_parms); - virtual ~FlowSocketSrc(); - - int ExtractNextPDU(); -}; - -class FlowFileSrc : public FlowSrc { -public: - FlowFileSrc(const char* readfile); - ~FlowFileSrc(); - - int ExtractNextPDU(); - -protected: - int Error(int errlvl, const char* errmsg); - char* readfile; -}; - -#endif diff --git a/src/IOSource.cc b/src/IOSource.cc deleted file mode 100644 index d47007caad..0000000000 --- a/src/IOSource.cc +++ /dev/null @@ -1,176 +0,0 @@ -#include -#include -#include -#include - -#include - -#include "util.h" -#include "IOSource.h" - -IOSourceRegistry io_sources; - -IOSourceRegistry::~IOSourceRegistry() - { - for ( SourceList::iterator i = sources.begin(); i != sources.end(); ++i ) - delete *i; - - sources.clear(); - } - -void IOSourceRegistry::RemoveAll() - { - // We're cheating a bit here ... - dont_counts = sources.size(); - } - -IOSource* IOSourceRegistry::FindSoonest(double* ts) - { - // Remove sources which have gone dry. For simplicity, we only - // remove at most one each time. - for ( SourceList::iterator i = sources.begin(); - i != sources.end(); ++i ) - if ( ! (*i)->src->IsOpen() ) - { - delete *i; - sources.erase(i); - break; - } - - // Ideally, we would always call select on the fds to see which - // are ready, and return the soonest. Unfortunately, that'd mean - // one select-call per packet, which we can't afford in high-volume - // environments. Thus, we call select only every SELECT_FREQUENCY - // call (or if all sources report that they are dry). - - ++call_count; - - IOSource* soonest_src = 0; - double soonest_ts = 1e20; - double soonest_local_network_time = 1e20; - bool all_idle = true; - - // Find soonest source of those which tell us they have something to - // process. - for ( SourceList::iterator i = sources.begin(); i != sources.end(); ++i ) - { - if ( ! (*i)->src->IsIdle() ) - { - all_idle = false; - double local_network_time = 0; - double ts = (*i)->src->NextTimestamp(&local_network_time); - if ( ts > 0 && ts < soonest_ts ) - { - soonest_ts = ts; - soonest_src = (*i)->src; - soonest_local_network_time = - local_network_time ? - local_network_time : ts; - } - } - } - - // If we found one and aren't going to select this time, - // return it. - int maxx = 0; - - if ( soonest_src && (call_count % SELECT_FREQUENCY) != 0 ) - goto finished; - - // Select on the join of all file descriptors. - fd_set fd_read, fd_write, fd_except; - - FD_ZERO(&fd_read); - FD_ZERO(&fd_write); - FD_ZERO(&fd_except); - - for ( SourceList::iterator i = sources.begin(); - i != sources.end(); ++i ) - { - Source* src = (*i); - - if ( ! src->src->IsIdle() ) - // No need to select on sources which we know to - // be ready. - continue; - - src->fd_read = src->fd_write = src->fd_except = 0; - src->src->GetFds(&src->fd_read, &src->fd_write, &src->fd_except); - - FD_SET(src->fd_read, &fd_read); - FD_SET(src->fd_write, &fd_write); - FD_SET(src->fd_except, &fd_except); - - maxx = max(src->fd_read, maxx); - maxx = max(src->fd_write, maxx); - maxx = max(src->fd_except, maxx); - } - - // We can't block indefinitely even when all sources are dry: - // we're doing some IOSource-independent stuff in the main loop, - // so we need to return from time to time. (Instead of no time-out - // at all, we use a very small one. This lets FreeBSD trigger a - // BPF buffer switch on the next read when the hold buffer is empty - // while the store buffer isn't filled yet. - - struct timeval timeout; - - if ( all_idle ) - { - // Interesting: when all sources are dry, simply sleeping a - // bit *without* watching for any fd becoming ready may - // decrease CPU load. I guess that's because it allows - // the kernel's packet buffers to fill. - Robin - timeout.tv_sec = 0; - timeout.tv_usec = 20; // SELECT_TIMEOUT; - select(0, 0, 0, 0, &timeout); - } - - if ( ! maxx ) - // No selectable fd at all. - goto finished; - - timeout.tv_sec = 0; - timeout.tv_usec = 0; - - if ( select(maxx + 1, &fd_read, &fd_write, &fd_except, &timeout) > 0 ) - { // Find soonest. - for ( SourceList::iterator i = sources.begin(); - i != sources.end(); ++i ) - { - Source* src = (*i); - - if ( ! src->src->IsIdle() ) - continue; - - if ( FD_ISSET(src->fd_read, &fd_read) || - FD_ISSET(src->fd_write, &fd_write) || - FD_ISSET(src->fd_except, &fd_except) ) - { - double local_network_time = 0; - double ts = src->src->NextTimestamp(&local_network_time); - if ( ts > 0.0 && ts < soonest_ts ) - { - soonest_ts = ts; - soonest_src = src->src; - soonest_local_network_time = - local_network_time ? - local_network_time : ts; - } - } - } - } - -finished: - *ts = soonest_local_network_time; - return soonest_src; - } - -void IOSourceRegistry::Register(IOSource* src, bool dont_count) - { - Source* s = new Source; - s->src = src; - if ( dont_count ) - ++dont_counts; - return sources.push_back(s); - } diff --git a/src/IOSource.h b/src/IOSource.h deleted file mode 100644 index db50bbd2a9..0000000000 --- a/src/IOSource.h +++ /dev/null @@ -1,103 +0,0 @@ -// Interface for classes providing/consuming data during Bro's main loop. - -#ifndef iosource_h -#define iosource_h - -#include -#include "Timer.h" - -using namespace std; - -class IOSource { -public: - IOSource() { idle = closed = false; } - virtual ~IOSource() {} - - // Returns true if source has nothing ready to process. - bool IsIdle() const { return idle; } - - // Returns true if more data is to be expected in the future. - // Otherwise, source may be removed. - bool IsOpen() const { return ! closed; } - - // Returns select'able fds (leaves args untouched if we don't have - // selectable fds). - virtual void GetFds(int* read, int* write, int* except) = 0; - - // The following two methods are only called when either IsIdle() - // returns false or select() on one of the fds indicates that there's - // data to process. - - // Returns timestamp (in global network time) associated with next - // data item. If the source wants the data item to be processed - // with a local network time, it sets the argument accordingly. - virtual double NextTimestamp(double* network_time) = 0; - - // Processes and consumes next data item. - virtual void Process() = 0; - - // Returns tag of timer manager associated with last processed - // data item, nil for global timer manager. - virtual TimerMgr::Tag* GetCurrentTag() { return 0; } - - // Returns a descriptual tag for debugging. - virtual const char* Tag() = 0; - -protected: - // Derived classed are to set this to true if they have gone dry - // temporarily. - bool idle; - - // Derived classed are to set this to true if they have gone dry - // permanently. - bool closed; -}; - -class IOSourceRegistry { -public: - IOSourceRegistry() { call_count = 0; dont_counts = 0; } - ~IOSourceRegistry(); - - // If dont_count is true, this source does not contribute to the - // number of IOSources returned by Size(). The effect is that - // if all sources but the non-counting ones have gone dry, - // processing will shut down. - void Register(IOSource* src, bool dont_count = false); - - // This may block for some time. - IOSource* FindSoonest(double* ts); - - int Size() const { return sources.size() - dont_counts; } - - // Terminate IOSource processing immediately by removing all - // sources (and therefore returning a Size() of zero). - void Terminate() { RemoveAll(); } - -protected: - // When looking for a source with something to process, - // every SELECT_FREQUENCY calls we will go ahead and - // block on a select(). - static const int SELECT_FREQUENCY = 25; - - // Microseconds to wait in an empty select if no source is ready. - static const int SELECT_TIMEOUT = 50; - - void RemoveAll(); - - unsigned int call_count; - int dont_counts; - - struct Source { - IOSource* src; - int fd_read; - int fd_write; - int fd_except; - }; - - typedef list SourceList; - SourceList sources; -}; - -extern IOSourceRegistry io_sources; - -#endif diff --git a/src/Net.cc b/src/Net.cc index 6a5c65c537..adac9c02fd 100644 --- a/src/Net.cc +++ b/src/Net.cc @@ -29,6 +29,9 @@ #include "Anon.h" #include "Serializer.h" #include "PacketDumper.h" +#include "iosource/Manager.h" +#include "iosource/PktSrc.h" +#include "iosource/PktDumper.h" #include "plugin/Manager.h" extern "C" { @@ -39,10 +42,7 @@ extern "C" { extern int select(int, fd_set *, fd_set *, fd_set *, struct timeval *); } -PList(PktSrc) pkt_srcs; - -// FIXME: We should really merge PktDumper and PacketDumper. -PktDumper* pkt_dumper = 0; +iosource::PktDumper* pkt_dumper = 0; int reading_live = 0; int reading_traces = 0; @@ -63,8 +63,8 @@ const u_char* current_pkt = 0; int current_dispatched = 0; int current_hdr_size = 0; double current_timestamp = 0.0; -PktSrc* current_pktsrc = 0; -IOSource* current_iosrc; +iosource::PktSrc* current_pktsrc = 0; +iosource::IOSource* current_iosrc = 0; std::list files_scanned; std::vector sig_files; @@ -113,17 +113,21 @@ RETSIGTYPE watchdog(int /* signo */) // saving the packet which caused the // watchdog to trigger may be helpful, // so we'll save that one nevertheless. - pkt_dumper = new PktDumper("watchdog-pkt.pcap"); - if ( pkt_dumper->IsError() ) + pkt_dumper = iosource_mgr->OpenPktDumper("watchdog-pkt.pcap", false); + if ( ! pkt_dumper || pkt_dumper->IsError() ) { - reporter->Error("watchdog: can't open watchdog-pkt.pcap for writing\n"); - delete pkt_dumper; + reporter->Error("watchdog: can't open watchdog-pkt.pcap for writing"); pkt_dumper = 0; } } if ( pkt_dumper ) - pkt_dumper->Dump(current_hdr, current_pkt); + { + iosource::PktDumper::Packet p; + p.hdr = current_hdr; + p.data = current_pkt; + pkt_dumper->Dump(&p); + } } net_get_final_stats(); @@ -149,118 +153,40 @@ void net_update_time(double new_network_time) } void net_init(name_list& interfaces, name_list& readfiles, - name_list& netflows, name_list& flowfiles, - const char* writefile, const char* filter, - const char* secondary_filter, int do_watchdog) + const char* writefile, int do_watchdog) { - if ( readfiles.length() > 0 || flowfiles.length() > 0 ) + if ( readfiles.length() > 0 ) { reading_live = pseudo_realtime > 0.0; reading_traces = 1; for ( int i = 0; i < readfiles.length(); ++i ) { - PktFileSrc* ps = new PktFileSrc(readfiles[i], filter); + iosource::PktSrc* ps = iosource_mgr->OpenPktSrc(readfiles[i], false); + assert(ps); if ( ! ps->IsOpen() ) - reporter->FatalError("%s: problem with trace file %s - %s\n", - prog, readfiles[i], ps->ErrorMsg()); - else - { - pkt_srcs.append(ps); - io_sources.Register(ps); - } - - if ( secondary_filter ) - { - // We use a second PktFileSrc for the - // secondary path. - PktFileSrc* ps = new PktFileSrc(readfiles[i], - secondary_filter, - TYPE_FILTER_SECONDARY); - - if ( ! ps->IsOpen() ) - reporter->FatalError("%s: problem with trace file %s - %s\n", - prog, readfiles[i], - ps->ErrorMsg()); - else - { - pkt_srcs.append(ps); - io_sources.Register(ps); - } - - ps->AddSecondaryTablePrograms(); - } - } - - for ( int i = 0; i < flowfiles.length(); ++i ) - { - FlowFileSrc* fs = new FlowFileSrc(flowfiles[i]); - - if ( ! fs->IsOpen() ) - reporter->FatalError("%s: problem with netflow file %s - %s\n", - prog, flowfiles[i], fs->ErrorMsg()); - else - { - io_sources.Register(fs); - } + reporter->FatalError("problem with trace file %s (%s)", + readfiles[i], + ps->ErrorMsg()); } } - else if ((interfaces.length() > 0 || netflows.length() > 0)) + else if ( interfaces.length() > 0 ) { reading_live = 1; reading_traces = 0; for ( int i = 0; i < interfaces.length(); ++i ) { - PktSrc* ps; - ps = new PktInterfaceSrc(interfaces[i], filter); + iosource::PktSrc* ps = iosource_mgr->OpenPktSrc(interfaces[i], true); + assert(ps); if ( ! ps->IsOpen() ) - reporter->FatalError("%s: problem with interface %s - %s\n", - prog, interfaces[i], ps->ErrorMsg()); - else - { - pkt_srcs.append(ps); - io_sources.Register(ps); - } - - if ( secondary_filter ) - { - PktSrc* ps; - ps = new PktInterfaceSrc(interfaces[i], - filter, TYPE_FILTER_SECONDARY); - - if ( ! ps->IsOpen() ) - reporter->Error("%s: problem with interface %s - %s\n", - prog, interfaces[i], - ps->ErrorMsg()); - else - { - pkt_srcs.append(ps); - io_sources.Register(ps); - } - - ps->AddSecondaryTablePrograms(); - } + reporter->FatalError("problem with interface %s (%s)", + interfaces[i], + ps->ErrorMsg()); } - - for ( int i = 0; i < netflows.length(); ++i ) - { - FlowSocketSrc* fs = new FlowSocketSrc(netflows[i]); - - if ( ! fs->IsOpen() ) - { - reporter->Error("%s: problem with netflow socket %s - %s\n", - prog, netflows[i], fs->ErrorMsg()); - delete fs; - } - - else - io_sources.Register(fs); - } - } else @@ -272,12 +198,12 @@ void net_init(name_list& interfaces, name_list& readfiles, if ( writefile ) { - // ### This will fail horribly if there are multiple - // interfaces with different-lengthed media. - pkt_dumper = new PktDumper(writefile); - if ( pkt_dumper->IsError() ) - reporter->FatalError("%s: can't open write file \"%s\" - %s\n", - prog, writefile, pkt_dumper->ErrorMsg()); + pkt_dumper = iosource_mgr->OpenPktDumper(writefile, false); + assert(pkt_dumper); + + if ( ! pkt_dumper->IsOpen() ) + reporter->FatalError("problem opening dump file %s (%s)", + writefile, pkt_dumper->ErrorMsg()); ID* id = global_scope()->Lookup("trace_output_file"); if ( ! id ) @@ -298,7 +224,7 @@ void net_init(name_list& interfaces, name_list& readfiles, } } -void expire_timers(PktSrc* src_ps) +void expire_timers(iosource::PktSrc* src_ps) { SegmentProfiler(segment_logger, "expiring-timers"); TimerMgr* tmgr = @@ -311,8 +237,8 @@ void expire_timers(PktSrc* src_ps) } void net_packet_dispatch(double t, const struct pcap_pkthdr* hdr, - const u_char* pkt, int hdr_size, - PktSrc* src_ps) + const u_char* pkt, int hdr_size, + iosource::PktSrc* src_ps) { if ( ! bro_start_network_time ) bro_start_network_time = t; @@ -368,11 +294,11 @@ void net_run() { set_processing_status("RUNNING", "net_run"); - while ( io_sources.Size() || + while ( iosource_mgr->Size() || (BifConst::exit_only_after_terminate && ! terminating) ) { double ts; - IOSource* src = io_sources.FindSoonest(&ts); + iosource::IOSource* src = iosource_mgr->FindSoonest(&ts); #ifdef DEBUG static int loop_counter = 0; @@ -470,16 +396,19 @@ void net_run() void net_get_final_stats() { - loop_over_list(pkt_srcs, i) + const iosource::Manager::PktSrcList& pkt_srcs(iosource_mgr->GetPktSrcs()); + + for ( iosource::Manager::PktSrcList::const_iterator i = pkt_srcs.begin(); + i != pkt_srcs.end(); i++ ) { - PktSrc* ps = pkt_srcs[i]; + iosource::PktSrc* ps = *i; if ( ps->IsLive() ) { - struct PktSrc::Stats s; + iosource::PktSrc::Stats s; ps->Statistics(&s); - reporter->Info("%d packets received on interface %s, %d dropped\n", - s.received, ps->Interface(), s.dropped); + reporter->Info("%d packets received on interface %s, %d dropped", + s.received, ps->Path().c_str(), s.dropped); } } } @@ -499,8 +428,6 @@ void net_finish(int drain_events) sessions->Done(); } - delete pkt_dumper; - #ifdef DEBUG extern int reassem_seen_bytes, reassem_copied_bytes; // DEBUG_MSG("Reassembly (TCP and IP/Frag): %d bytes seen, %d bytes copied\n", @@ -521,29 +448,6 @@ void net_delete() delete ip_anonymizer[i]; } -// net_packet_match -// -// Description: -// - Checks if a packet matches a filter. It just wraps up a call to -// [pcap.h's] bpf_filter(). -// -// Inputs: -// - fp: a BPF-compiled filter -// - pkt: a pointer to the packet -// - len: the original packet length -// - caplen: the captured packet length. This is pkt length -// -// Output: -// - return: 1 if the packet matches the filter, 0 otherwise - -int net_packet_match(BPF_Program* fp, const u_char* pkt, - u_int len, u_int caplen) - { - // NOTE: I don't like too much un-const'ing the pkt variable. - return bpf_filter(fp->GetProgram()->bf_insns, (u_char*) pkt, len, caplen); - } - - int _processing_suspended = 0; static double suspend_start = 0; @@ -561,8 +465,12 @@ void net_continue_processing() if ( _processing_suspended == 1 ) { reporter->Info("processing continued"); - loop_over_list(pkt_srcs, i) - pkt_srcs[i]->ContinueAfterSuspend(); + + const iosource::Manager::PktSrcList& pkt_srcs(iosource_mgr->GetPktSrcs()); + + for ( iosource::Manager::PktSrcList::const_iterator i = pkt_srcs.begin(); + i != pkt_srcs.end(); i++ ) + (*i)->ContinueAfterSuspend(); } --_processing_suspended; diff --git a/src/Net.h b/src/Net.h index 5fa7210efb..2e466f8c7f 100644 --- a/src/Net.h +++ b/src/Net.h @@ -5,17 +5,15 @@ #include "net_util.h" #include "util.h" -#include "BPF_Program.h" #include "List.h" -#include "PktSrc.h" -#include "FlowSrc.h" #include "Func.h" #include "RemoteSerializer.h" +#include "iosource/IOSource.h" +#include "iosource/PktSrc.h" +#include "iosource/PktDumper.h" extern void net_init(name_list& interfaces, name_list& readfiles, - name_list& netflows, name_list& flowfiles, - const char* writefile, const char* filter, - const char* secondary_filter, int do_watchdog); + const char* writefile, int do_watchdog); extern void net_run(); extern void net_get_final_stats(); extern void net_finish(int drain_events); @@ -23,10 +21,8 @@ extern void net_delete(); // Reclaim all memory, etc. extern void net_update_time(double new_network_time); extern void net_packet_dispatch(double t, const struct pcap_pkthdr* hdr, const u_char* pkt, int hdr_size, - PktSrc* src_ps); -extern int net_packet_match(BPF_Program* fp, const u_char* pkt, - u_int len, u_int caplen); -extern void expire_timers(PktSrc* src_ps = 0); + iosource::PktSrc* src_ps); +extern void expire_timers(iosource::PktSrc* src_ps = 0); extern void termination_signal(); // Functions to temporarily suspend processing of live input (network packets @@ -83,13 +79,10 @@ extern const u_char* current_pkt; extern int current_dispatched; extern int current_hdr_size; extern double current_timestamp; -extern PktSrc* current_pktsrc; -extern IOSource* current_iosrc; +extern iosource::PktSrc* current_pktsrc; +extern iosource::IOSource* current_iosrc; -declare(PList,PktSrc); -extern PList(PktSrc) pkt_srcs; - -extern PktDumper* pkt_dumper; // where to save packets +extern iosource::PktDumper* pkt_dumper; // where to save packets extern char* writefile; diff --git a/src/Pipe.cc b/src/Pipe.cc new file mode 100644 index 0000000000..3f60409fdb --- /dev/null +++ b/src/Pipe.cc @@ -0,0 +1,83 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#include "Pipe.h" +#include "Reporter.h" +#include +#include +#include +#include + +using namespace bro; + +static void pipe_fail(int eno) + { + char tmp[256]; + strerror_r(eno, tmp, sizeof(tmp)); + reporter->FatalError("Pipe failure: %s", tmp); + } + +static void set_flags(int fd, int flags) + { + if ( flags ) + fcntl(fd, F_SETFD, fcntl(fd, F_GETFD) | flags); + } + +static void set_status_flags(int fd, int flags) + { + if ( flags ) + fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | flags); + } + +static int dup_or_fail(int fd, int flags) + { + int rval = dup(fd); + + if ( rval < 0 ) + pipe_fail(errno); + + set_flags(fd, flags); + return rval; + } + +Pipe::Pipe(int flags0, int flags1, int status_flags0, int status_flags1) + { + // pipe2 can set flags atomically, but not yet available everywhere. + if ( ::pipe(fds) ) + pipe_fail(errno); + + flags[0] = flags0; + flags[1] = flags1; + + set_flags(fds[0], flags[0]); + set_flags(fds[1], flags[1]); + set_status_flags(fds[0], status_flags0); + set_status_flags(fds[1], status_flags1); + } + +Pipe::~Pipe() + { + close(fds[0]); + close(fds[1]); + } + +Pipe::Pipe(const Pipe& other) + { + fds[0] = dup_or_fail(other.fds[0], other.flags[0]); + fds[1] = dup_or_fail(other.fds[1], other.flags[1]); + flags[0] = other.flags[0]; + flags[1] = other.flags[1]; + } + +Pipe& Pipe::operator=(const Pipe& other) + { + if ( this == &other ) + return *this; + + close(fds[0]); + close(fds[1]); + fds[0] = dup_or_fail(other.fds[0], other.flags[0]); + fds[1] = dup_or_fail(other.fds[1], other.flags[1]); + flags[0] = other.flags[0]; + flags[1] = other.flags[1]; + return *this; + } diff --git a/src/Pipe.h b/src/Pipe.h new file mode 100644 index 0000000000..77b341117e --- /dev/null +++ b/src/Pipe.h @@ -0,0 +1,56 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#ifndef BRO_PIPE_H +#define BRO_PIPE_H + +namespace bro { + +class Pipe { +public: + + /** + * Create a pair of file descriptors via pipe(), or aborts if it cannot. + * @param flags0 file descriptor flags to set on read end of pipe. + * @param flags1 file descriptor flags to set on write end of pipe. + * @param status_flags0 descriptor status flags to set on read end of pipe. + * @param status_flags1 descriptor status flags to set on write end of pipe. + */ + Pipe(int flags0 = 0, int flags1 = 0, int status_flags0 = 0, + int status_flags1 = 0); + + /** + * Close the pair of file descriptors owned by the object. + */ + ~Pipe(); + + /** + * Make a copy of another Pipe object (file descriptors are dup'd). + */ + Pipe(const Pipe& other); + + /** + * Assign a Pipe object by closing file descriptors and duping those of + * the other. + */ + Pipe& operator=(const Pipe& other); + + /** + * @return the file descriptor associated with the read-end of the pipe. + */ + int ReadFD() const + { return fds[0]; } + + /** + * @return the file descriptor associated with the write-end of the pipe. + */ + int WriteFD() const + { return fds[1]; } + +private: + int fds[2]; + int flags[2]; +}; + +} // namespace bro + +#endif // BRO_PIPE_H diff --git a/src/PktSrc.cc b/src/PktSrc.cc deleted file mode 100644 index b5ac3a5d69..0000000000 --- a/src/PktSrc.cc +++ /dev/null @@ -1,804 +0,0 @@ -// See the file "COPYING" in the main distribution directory for copyright. - -#include -#include - -#include "config.h" - -#include "util.h" -#include "PktSrc.h" -#include "Hash.h" -#include "Net.h" -#include "Sessions.h" - - -// ### This needs auto-confing. -#ifdef HAVE_PCAP_INT_H -#include -#endif - -PktSrc::PktSrc() - { - interface = readfile = 0; - data = last_data = 0; - memset(&hdr, 0, sizeof(hdr)); - hdr_size = 0; - datalink = 0; - netmask = 0xffffff00; - pd = 0; - idle = false; - - next_sync_point = 0; - first_timestamp = current_timestamp = next_timestamp = 0.0; - first_wallclock = current_wallclock = 0; - - stats.received = stats.dropped = stats.link = 0; - } - -PktSrc::~PktSrc() - { - Close(); - - loop_over_list(program_list, i) - delete program_list[i]; - - BPF_Program* code; - IterCookie* cookie = filters.InitForIteration(); - while ( (code = filters.NextEntry(cookie)) ) - delete code; - - delete [] interface; - delete [] readfile; - } - -void PktSrc::GetFds(int* read, int* write, int* except) - { - if ( pseudo_realtime ) - { - // Select would give erroneous results. But we simulate it - // by setting idle accordingly. - idle = CheckPseudoTime() == 0; - return; - } - - if ( selectable_fd >= 0 ) - *read = selectable_fd; - } - -int PktSrc::ExtractNextPacket() - { - // Don't return any packets if processing is suspended (except for the - // very first packet which we need to set up times). - if ( net_is_processing_suspended() && first_timestamp ) - { - idle = true; - return 0; - } - - data = last_data = pcap_next(pd, &hdr); - - if ( data && (hdr.len == 0 || hdr.caplen == 0) ) - { - sessions->Weird("empty_pcap_header", &hdr, data); - return 0; - } - - if ( data ) - next_timestamp = hdr.ts.tv_sec + double(hdr.ts.tv_usec) / 1e6; - - if ( pseudo_realtime ) - current_wallclock = current_time(true); - - if ( ! first_timestamp ) - first_timestamp = next_timestamp; - - idle = (data == 0); - - if ( data ) - ++stats.received; - - // Source has gone dry. If it's a network interface, this just means - // it's timed out. If it's a file, though, then the file has been - // exhausted. - if ( ! data && ! IsLive() ) - { - closed = true; - - if ( pseudo_realtime && using_communication ) - { - if ( remote_trace_sync_interval ) - remote_serializer->SendFinalSyncPoint(); - else - remote_serializer->Terminate(); - } - } - - return data != 0; - } - -double PktSrc::NextTimestamp(double* local_network_time) - { - if ( ! data && ! ExtractNextPacket() ) - return -1.0; - - if ( pseudo_realtime ) - { - // Delay packet if necessary. - double packet_time = CheckPseudoTime(); - if ( packet_time ) - return packet_time; - - idle = true; - return -1.0; - } - - return next_timestamp; - } - -void PktSrc::ContinueAfterSuspend() - { - current_wallclock = current_time(true); - } - -double PktSrc::CurrentPacketWallClock() - { - // We stop time when we are suspended. - if ( net_is_processing_suspended() ) - current_wallclock = current_time(true); - - return current_wallclock; - } - -double PktSrc::CheckPseudoTime() - { - if ( ! data && ! ExtractNextPacket() ) - return 0; - - if ( ! current_timestamp ) - return bro_start_time; - - if ( remote_trace_sync_interval ) - { - if ( next_sync_point == 0 || next_timestamp >= next_sync_point ) - { - int n = remote_serializer->SendSyncPoint(); - next_sync_point = first_timestamp + - n * remote_trace_sync_interval; - remote_serializer->Log(RemoteSerializer::LogInfo, - fmt("stopping at packet %.6f, next sync-point at %.6f", - current_timestamp, next_sync_point)); - - return 0; - } - } - - double pseudo_time = next_timestamp - first_timestamp; - double ct = (current_time(true) - first_wallclock) * pseudo_realtime; - - return pseudo_time <= ct ? bro_start_time + pseudo_time : 0; - } - -void PktSrc::Process() - { - if ( ! data && ! ExtractNextPacket() ) - return; - - current_timestamp = next_timestamp; - - int pkt_hdr_size = hdr_size; - - // Unfortunately some packets on the link might have MPLS labels - // while others don't. That means we need to ask the link-layer if - // labels are in place. - bool have_mpls = false; - - int protocol = 0; - - switch ( datalink ) { - case DLT_NULL: - { - protocol = (data[3] << 24) + (data[2] << 16) + (data[1] << 8) + data[0]; - - // From the Wireshark Wiki: "AF_INET6, unfortunately, has - // different values in {NetBSD,OpenBSD,BSD/OS}, - // {FreeBSD,DragonFlyBSD}, and {Darwin/Mac OS X}, so an IPv6 - // packet might have a link-layer header with 24, 28, or 30 - // as the AF_ value." As we may be reading traces captured on - // platforms other than what we're running on, we accept them - // all here. - if ( protocol != AF_INET - && protocol != AF_INET6 - && protocol != 24 - && protocol != 28 - && protocol != 30 ) - { - sessions->Weird("non_ip_packet_in_null_transport", &hdr, data); - data = 0; - return; - } - - break; - } - - case DLT_EN10MB: - { - // Get protocol being carried from the ethernet frame. - protocol = (data[12] << 8) + data[13]; - - switch ( protocol ) - { - // MPLS carried over the ethernet frame. - case 0x8847: - // Remove the data link layer and denote a - // header size of zero before the IP header. - have_mpls = true; - data += get_link_header_size(datalink); - pkt_hdr_size = 0; - break; - - // VLAN carried over the ethernet frame. - case 0x8100: - data += get_link_header_size(datalink); - - // Check for MPLS in VLAN. - if ( ((data[2] << 8) + data[3]) == 0x8847 ) - have_mpls = true; - - data += 4; // Skip the vlan header - pkt_hdr_size = 0; - - // Check for 802.1ah (Q-in-Q) containing IP. - // Only do a second layer of vlan tag - // stripping because there is no - // specification that allows for deeper - // nesting. - if ( ((data[2] << 8) + data[3]) == 0x0800 ) - data += 4; - - break; - - // PPPoE carried over the ethernet frame. - case 0x8864: - data += get_link_header_size(datalink); - protocol = (data[6] << 8) + data[7]; - data += 8; // Skip the PPPoE session and PPP header - pkt_hdr_size = 0; - - if ( protocol != 0x0021 && protocol != 0x0057 ) - { - // Neither IPv4 nor IPv6. - sessions->Weird("non_ip_packet_in_pppoe_encapsulation", &hdr, data); - data = 0; - return; - } - break; - } - - break; - } - - case DLT_PPP_SERIAL: - { - // Get PPP protocol. - protocol = (data[2] << 8) + data[3]; - - if ( protocol == 0x0281 ) - { - // MPLS Unicast. Remove the data link layer and - // denote a header size of zero before the IP header. - have_mpls = true; - data += get_link_header_size(datalink); - pkt_hdr_size = 0; - } - - else if ( protocol != 0x0021 && protocol != 0x0057 ) - { - // Neither IPv4 nor IPv6. - sessions->Weird("non_ip_packet_in_ppp_encapsulation", &hdr, data); - data = 0; - return; - } - break; - } - } - - if ( have_mpls ) - { - // Skip the MPLS label stack. - bool end_of_stack = false; - - while ( ! end_of_stack ) - { - end_of_stack = *(data + 2) & 0x01; - data += 4; - } - } - - if ( pseudo_realtime ) - { - current_pseudo = CheckPseudoTime(); - net_packet_dispatch(current_pseudo, &hdr, data, pkt_hdr_size, this); - if ( ! first_wallclock ) - first_wallclock = current_time(true); - } - - else - net_packet_dispatch(current_timestamp, &hdr, data, pkt_hdr_size, this); - - data = 0; - } - -bool PktSrc::GetCurrentPacket(const struct pcap_pkthdr** arg_hdr, - const u_char** arg_pkt) - { - if ( ! last_data ) - return false; - - *arg_hdr = &hdr; - *arg_pkt = last_data; - return true; - } - -int PktSrc::PrecompileFilter(int index, const char* filter) - { - // Compile filter. - BPF_Program* code = new BPF_Program(); - - if ( ! code->Compile(pd, filter, netmask, errbuf, sizeof(errbuf)) ) - { - delete code; - return 0; - } - - // Store it in hash. - HashKey* hash = new HashKey(HashKey(bro_int_t(index))); - BPF_Program* oldcode = filters.Lookup(hash); - if ( oldcode ) - delete oldcode; - - filters.Insert(hash, code); - delete hash; - - return 1; - } - -int PktSrc::SetFilter(int index) - { - // We don't want load-level filters for the secondary path. - if ( filter_type == TYPE_FILTER_SECONDARY && index > 0 ) - return 1; - - HashKey* hash = new HashKey(HashKey(bro_int_t(index))); - BPF_Program* code = filters.Lookup(hash); - delete hash; - - if ( ! code ) - { - safe_snprintf(errbuf, sizeof(errbuf), - "No precompiled pcap filter for index %d", - index); - return 0; - } - - if ( pcap_setfilter(pd, code->GetProgram()) < 0 ) - { - safe_snprintf(errbuf, sizeof(errbuf), - "pcap_setfilter(%d): %s", - index, pcap_geterr(pd)); - return 0; - } - -#ifndef HAVE_LINUX - // Linux doesn't clear counters when resetting filter. - stats.received = stats.dropped = stats.link = 0; -#endif - - return 1; - } - -void PktSrc::SetHdrSize() - { - int dl = pcap_datalink(pd); - hdr_size = get_link_header_size(dl); - - if ( hdr_size < 0 ) - { - safe_snprintf(errbuf, sizeof(errbuf), - "unknown data link type 0x%x", dl); - Close(); - } - - datalink = dl; - } - -void PktSrc::Close() - { - if ( pd ) - { - pcap_close(pd); - pd = 0; - closed = true; - } - } - -void PktSrc::AddSecondaryTablePrograms() - { - BPF_Program* program; - - loop_over_list(secondary_path->EventTable(), i) - { - SecondaryEvent* se = secondary_path->EventTable()[i]; - program = new BPF_Program(); - - if ( ! program->Compile(snaplen, datalink, se->Filter(), - netmask, errbuf, sizeof(errbuf)) ) - { - delete program; - Close(); - return; - } - - SecondaryProgram* sp = new SecondaryProgram(program, se); - program_list.append(sp); - } - } - -void PktSrc::Statistics(Stats* s) - { - if ( reading_traces ) - s->received = s->dropped = s->link = 0; - - else - { - struct pcap_stat pstat; - if ( pcap_stats(pd, &pstat) < 0 ) - { - reporter->Error("problem getting packet filter statistics: %s", - ErrorMsg()); - s->received = s->dropped = s->link = 0; - } - - else - { - s->dropped = pstat.ps_drop; - s->link = pstat.ps_recv; - } - } - - s->received = stats.received; - - if ( pseudo_realtime ) - s->dropped = 0; - - stats.dropped = s->dropped; - } - -PktInterfaceSrc::PktInterfaceSrc(const char* arg_interface, const char* filter, - PktSrc_Filter_Type ft) -: PktSrc() - { - char tmp_errbuf[PCAP_ERRBUF_SIZE]; - filter_type = ft; - - // Determine interface if not specified. - if ( ! arg_interface && ! (arg_interface = pcap_lookupdev(tmp_errbuf)) ) - { - safe_snprintf(errbuf, sizeof(errbuf), - "pcap_lookupdev: %s", tmp_errbuf); - return; - } - - interface = copy_string(arg_interface); - - // Determine network and netmask. - uint32 net; - if ( pcap_lookupnet(interface, &net, &netmask, tmp_errbuf) < 0 ) - { - // ### The lookup can fail if no address is assigned to - // the interface; and libpcap doesn't have any useful notion - // of error codes, just error strings - how bogus - so we - // just kludge around the error :-(. - // sprintf(errbuf, "pcap_lookupnet %s", tmp_errbuf); - // return; - net = 0; - netmask = 0xffffff00; - } - - // We use the smallest time-out possible to return almost immediately if - // no packets are available. (We can't use set_nonblocking() as it's - // broken on FreeBSD: even when select() indicates that we can read - // something, we may get nothing if the store buffer hasn't filled up - // yet.) - pd = pcap_open_live(interface, snaplen, 1, 1, tmp_errbuf); - - if ( ! pd ) - { - safe_snprintf(errbuf, sizeof(errbuf), - "pcap_open_live: %s", tmp_errbuf); - closed = true; - return; - } - - // ### This needs autoconf'ing. -#ifdef HAVE_PCAP_INT_H - reporter->Info("pcap bufsize = %d\n", ((struct pcap *) pd)->bufsize); -#endif - -#ifdef HAVE_LINUX - if ( pcap_setnonblock(pd, 1, tmp_errbuf) < 0 ) - { - safe_snprintf(errbuf, sizeof(errbuf), - "pcap_setnonblock: %s", tmp_errbuf); - pcap_close(pd); - closed = true; - return; - } -#endif - selectable_fd = pcap_fileno(pd); - - if ( PrecompileFilter(0, filter) && SetFilter(0) ) - { - SetHdrSize(); - - if ( closed ) - // Couldn't get header size. - return; - - reporter->Info("listening on %s, capture length %d bytes\n", interface, snaplen); - } - else - closed = true; - } - - -PktFileSrc::PktFileSrc(const char* arg_readfile, const char* filter, - PktSrc_Filter_Type ft) -: PktSrc() - { - readfile = copy_string(arg_readfile); - - filter_type = ft; - - pd = pcap_open_offline((char*) readfile, errbuf); - - if ( pd && PrecompileFilter(0, filter) && SetFilter(0) ) - { - SetHdrSize(); - - if ( closed ) - // Unknown link layer type. - return; - - // We don't put file sources into non-blocking mode as - // otherwise we would not be able to identify the EOF. - - selectable_fd = fileno(pcap_file(pd)); - - if ( selectable_fd < 0 ) - reporter->InternalError("OS does not support selectable pcap fd"); - } - else - closed = true; - } - - -SecondaryPath::SecondaryPath() - { - filter = 0; - - // Glue together the secondary filter, if exists. - Val* secondary_fv = internal_val("secondary_filters"); - if ( secondary_fv->AsTableVal()->Size() == 0 ) - return; - - int did_first = 0; - const TableEntryValPDict* v = secondary_fv->AsTable(); - IterCookie* c = v->InitForIteration(); - TableEntryVal* tv; - HashKey* h; - - while ( (tv = v->NextEntry(h, c)) ) - { - // Get the index values. - ListVal* index = - secondary_fv->AsTableVal()->RecoverIndex(h); - - const char* str = - index->Index(0)->Ref()->AsString()->CheckString(); - - if ( ++did_first == 1 ) - { - filter = copy_string(str); - } - else - { - if ( strlen(filter) > 0 ) - { - char* tmp_f = new char[strlen(str) + strlen(filter) + 32]; - if ( strlen(str) == 0 ) - sprintf(tmp_f, "%s", filter); - else - sprintf(tmp_f, "(%s) or (%s)", filter, str); - delete [] filter; - filter = tmp_f; - } - } - - // Build secondary_path event table item and link it. - SecondaryEvent* se = - new SecondaryEvent(index->Index(0)->Ref()->AsString()->CheckString(), - tv->Value()->AsFunc() ); - - event_list.append(se); - - delete h; - Unref(index); - } - } - -SecondaryPath::~SecondaryPath() - { - loop_over_list(event_list, i) - delete event_list[i]; - - delete [] filter; - } - - -SecondaryProgram::~SecondaryProgram() - { - delete program; - } - -PktDumper::PktDumper(const char* arg_filename, bool arg_append) - { - filename[0] = '\0'; - is_error = false; - append = arg_append; - dumper = 0; - open_time = 0.0; - - // We need a pcap_t with a reasonable link-layer type. We try to get it - // from the packet sources. If not available, we fall back to Ethernet. - // FIXME: Perhaps we should make this configurable? - int linktype = -1; - - if ( pkt_srcs.length() ) - linktype = pkt_srcs[0]->LinkType(); - - if ( linktype < 0 ) - linktype = DLT_EN10MB; - - pd = pcap_open_dead(linktype, snaplen); - if ( ! pd ) - { - Error("error for pcap_open_dead"); - return; - } - - if ( arg_filename ) - Open(arg_filename); - } - -bool PktDumper::Open(const char* arg_filename) - { - if ( ! arg_filename && ! *filename ) - { - Error("no filename given"); - return false; - } - - if ( arg_filename ) - { - if ( dumper && streq(arg_filename, filename) ) - // Already open. - return true; - - safe_strncpy(filename, arg_filename, FNBUF_LEN); - } - - if ( dumper ) - Close(); - - struct stat s; - int exists = 0; - - if ( append ) - { - // See if output file already exists (and is non-empty). - exists = stat(filename, &s); ; - - if ( exists < 0 && errno != ENOENT ) - { - Error(fmt("can't stat file %s: %s", filename, strerror(errno))); - return false; - } - } - - if ( ! append || exists < 0 || s.st_size == 0 ) - { - // Open new file. - dumper = pcap_dump_open(pd, filename); - if ( ! dumper ) - { - Error(pcap_geterr(pd)); - return false; - } - } - - else - { - // Old file and we need to append, which, unfortunately, - // is not supported by libpcap. So, we have to hack a - // little bit, knowing that pcap_dumpter_t is, in fact, - // a FILE ... :-( - dumper = (pcap_dumper_t*) fopen(filename, "a"); - if ( ! dumper ) - { - Error(fmt("can't open dump %s: %s", filename, strerror(errno))); - return false; - } - } - - open_time = network_time; - is_error = false; - return true; - } - -bool PktDumper::Close() - { - if ( dumper ) - { - pcap_dump_close(dumper); - dumper = 0; - is_error = false; - } - - return true; - } - -bool PktDumper::Dump(const struct pcap_pkthdr* hdr, const u_char* pkt) - { - if ( ! dumper ) - return false; - - if ( ! open_time ) - open_time = network_time; - - pcap_dump((u_char*) dumper, hdr, pkt); - - return true; - } - -void PktDumper::Error(const char* errstr) - { - safe_strncpy(errbuf, errstr, sizeof(errbuf)); - is_error = true; - } - -int get_link_header_size(int dl) - { - switch ( dl ) { - case DLT_NULL: - return 4; - - case DLT_EN10MB: - return 14; - - case DLT_FDDI: - return 13 + 8; // fddi_header + LLC - -#ifdef DLT_LINUX_SLL - case DLT_LINUX_SLL: - return 16; -#endif - - case DLT_PPP_SERIAL: // PPP_SERIAL - return 4; - - case DLT_RAW: - return 0; - } - - return -1; - } diff --git a/src/PktSrc.h b/src/PktSrc.h deleted file mode 100644 index 70eef4dd00..0000000000 --- a/src/PktSrc.h +++ /dev/null @@ -1,258 +0,0 @@ -// See the file "COPYING" in the main distribution directory for copyright. - -#ifndef pktsrc_h -#define pktsrc_h - -#include "Dict.h" -#include "Expr.h" -#include "BPF_Program.h" -#include "IOSource.h" -#include "RemoteSerializer.h" - -#define BRO_PCAP_ERRBUF_SIZE PCAP_ERRBUF_SIZE + 256 - -extern "C" { -#include -} - -declare(PDict,BPF_Program); - -// Whether a PktSrc object is used by the normal filter structure or the -// secondary-path structure. -typedef enum { - TYPE_FILTER_NORMAL, // the normal filter - TYPE_FILTER_SECONDARY, // the secondary-path filter -} PktSrc_Filter_Type; - - -// {filter,event} tuples conforming the secondary path. -class SecondaryEvent { -public: - SecondaryEvent(const char* arg_filter, Func* arg_event) - { - filter = arg_filter; - event = arg_event; - } - - const char* Filter() { return filter; } - Func* Event() { return event; } - -private: - const char* filter; - Func* event; -}; - -declare(PList,SecondaryEvent); -typedef PList(SecondaryEvent) secondary_event_list; - - - -class SecondaryPath { -public: - SecondaryPath(); - ~SecondaryPath(); - - secondary_event_list& EventTable() { return event_list; } - const char* Filter() { return filter; } - -private: - secondary_event_list event_list; - // OR'ed union of all SecondaryEvent filters - char* filter; -}; - -// Main secondary-path object. -extern SecondaryPath* secondary_path; - - -// {program, {filter,event}} tuple table. -class SecondaryProgram { -public: - SecondaryProgram(BPF_Program* arg_program, SecondaryEvent* arg_event) - { - program = arg_program; - event = arg_event; - } - - ~SecondaryProgram(); - - BPF_Program* Program() { return program; } - SecondaryEvent* Event() { return event; } - -private: - // Associated program. - BPF_Program *program; - - // Event that is run in case the program is matched. - SecondaryEvent* event; -}; - -declare(PList,SecondaryProgram); -typedef PList(SecondaryProgram) secondary_program_list; - - - -class PktSrc : public IOSource { -public: - ~PktSrc(); - - // IOSource interface - bool IsReady(); - void GetFds(int* read, int* write, int* except); - double NextTimestamp(double* local_network_time); - void Process(); - const char* Tag() { return "PktSrc"; } - - const char* ErrorMsg() const { return errbuf; } - void ClearErrorMsg() { *errbuf ='\0'; } - - // Returns the packet last processed; false if there is no - // current packet available. - bool GetCurrentPacket(const pcap_pkthdr** hdr, const u_char** pkt); - - int HdrSize() const { return hdr_size; } - int DataLink() const { return datalink; } - - void ConsumePacket() { data = 0; } - - int IsLive() const { return interface != 0; } - - pcap_t* PcapHandle() const { return pd; } - int LinkType() const { return pcap_datalink(pd); } - - const char* ReadFile() const { return readfile; } - const char* Interface() const { return interface; } - PktSrc_Filter_Type FilterType() const { return filter_type; } - void AddSecondaryTablePrograms(); - const secondary_program_list& ProgramTable() const - { return program_list; } - - // Signal packet source that processing was suspended and is now going - // to be continued. - void ContinueAfterSuspend(); - - // Only valid in pseudo-realtime mode. - double CurrentPacketTimestamp() { return current_pseudo; } - double CurrentPacketWallClock(); - - struct Stats { - unsigned int received; // pkts received (w/o drops) - unsigned int dropped; // pkts dropped - unsigned int link; // total packets on link - // (not always not available) - }; - - virtual void Statistics(Stats* stats); - - // Precompiles a filter and associates the given index with it. - // Returns true on success, 0 if a problem occurred. - virtual int PrecompileFilter(int index, const char* filter); - - // Activates the filter with the given index. - // Returns true on success, 0 if a problem occurred. - virtual int SetFilter(int index); - -protected: - PktSrc(); - - static const int PCAP_TIMEOUT = 20; - - void SetHdrSize(); - - virtual void Close(); - - // Returns 1 on success, 0 on time-out/gone dry. - virtual int ExtractNextPacket(); - - // Checks if the current packet has a pseudo-time <= current_time. - // If yes, returns pseudo-time, otherwise 0. - double CheckPseudoTime(); - - double current_timestamp; - double next_timestamp; - - // Only set in pseudo-realtime mode. - double first_timestamp; - double first_wallclock; - double current_wallclock; - double current_pseudo; - - struct pcap_pkthdr hdr; - const u_char* data; // contents of current packet - const u_char* last_data; // same, but unaffected by consuming - int hdr_size; - int datalink; - double next_sync_point; // For trace synchronziation in pseudo-realtime - - char* interface; // nil if not reading from an interface - char* readfile; // nil if not reading from a file - - pcap_t* pd; - int selectable_fd; - uint32 netmask; - char errbuf[BRO_PCAP_ERRBUF_SIZE]; - - Stats stats; - - PDict(BPF_Program) filters; // precompiled filters - - PktSrc_Filter_Type filter_type; // normal path or secondary path - secondary_program_list program_list; -}; - -class PktInterfaceSrc : public PktSrc { -public: - PktInterfaceSrc(const char* interface, const char* filter, - PktSrc_Filter_Type ft=TYPE_FILTER_NORMAL); -}; - -class PktFileSrc : public PktSrc { -public: - PktFileSrc(const char* readfile, const char* filter, - PktSrc_Filter_Type ft=TYPE_FILTER_NORMAL); -}; - - -extern int get_link_header_size(int dl); - -class PktDumper { -public: - PktDumper(const char* file = 0, bool append = false); - ~PktDumper() { Close(); } - - bool Open(const char* file = 0); - bool Close(); - bool Dump(const struct pcap_pkthdr* hdr, const u_char* pkt); - - pcap_dumper_t* PcapDumper() { return dumper; } - - const char* FileName() const { return filename; } - bool IsError() const { return is_error; } - const char* ErrorMsg() const { return errbuf; } - - // This heuristic will horribly fail if we're using packets - // with different link layers. (If we can't derive a reasonable value - // from the packet sources, our fall-back is Ethernet.) - int HdrSize() const - { return get_link_header_size(pcap_datalink(pd)); } - - // Network time when dump file was opened. - double OpenTime() const { return open_time; } - -private: - void InitPd(); - void Error(const char* str); - - static const int FNBUF_LEN = 1024; - char filename[FNBUF_LEN]; - - bool append; - pcap_dumper_t* dumper; - pcap_t* pd; - double open_time; - - bool is_error; - char errbuf[BRO_PCAP_ERRBUF_SIZE]; -}; - -#endif diff --git a/src/RemoteSerializer.cc b/src/RemoteSerializer.cc index 965e360690..b475c4a8cc 100644 --- a/src/RemoteSerializer.cc +++ b/src/RemoteSerializer.cc @@ -188,10 +188,10 @@ #include "File.h" #include "Conn.h" #include "Reporter.h" -#include "threading/SerialTypes.h" -#include "logging/Manager.h" #include "IPAddr.h" #include "bro_inet_ntop.h" +#include "iosource/Manager.h" +#include "logging/Manager.h" #include "logging/logging.bif.h" extern "C" { @@ -285,10 +285,10 @@ struct ping_args { \ if ( ! c ) \ { \ - idle = io->IsIdle();\ + SetIdle(io->IsIdle());\ return true; \ } \ - idle = false; \ + SetIdle(false); \ } static const char* msgToStr(int msg) @@ -534,7 +534,6 @@ RemoteSerializer::RemoteSerializer() current_sync_point = 0; syncing_times = false; io = 0; - closed = false; terminating = false; in_sync = 0; last_flush = 0; @@ -559,7 +558,7 @@ RemoteSerializer::~RemoteSerializer() delete io; } -void RemoteSerializer::Init() +void RemoteSerializer::Enable() { if ( initialized ) return; @@ -572,7 +571,7 @@ void RemoteSerializer::Init() Fork(); - io_sources.Register(this); + iosource_mgr->Register(this); Log(LogInfo, fmt("communication started, parent pid is %d, child pid is %d", getpid(), child_pid)); initialized = 1; @@ -1276,7 +1275,7 @@ bool RemoteSerializer::Listen(const IPAddr& ip, uint16 port, bool expect_ssl, return false; listening = true; - closed = false; + SetClosed(false); return true; } @@ -1345,7 +1344,7 @@ bool RemoteSerializer::StopListening() return false; listening = false; - closed = ! IsActive(); + SetClosed(! IsActive()); return true; } @@ -1368,12 +1367,14 @@ void RemoteSerializer::Unregister(ID* id) } } -void RemoteSerializer::GetFds(int* read, int* write, int* except) +void RemoteSerializer::GetFds(iosource::FD_Set* read, iosource::FD_Set* write, + iosource::FD_Set* except) { - *read = io->Fd(); + read->Insert(io->Fd()); + read->Insert(io->ExtraReadFDs()); if ( io->CanWrite() ) - *write = io->Fd(); + write->Insert(io->Fd()); } double RemoteSerializer::NextTimestamp(double* local_network_time) @@ -1383,7 +1384,7 @@ double RemoteSerializer::NextTimestamp(double* local_network_time) if ( received_logs > 0 ) { // If we processed logs last time, assume there's more. - idle = false; + SetIdle(false); received_logs = 0; return timer_mgr->Time(); } @@ -1398,7 +1399,7 @@ double RemoteSerializer::NextTimestamp(double* local_network_time) pt = timer_mgr->Time(); if ( packets.length() ) - idle = false; + SetIdle(false); if ( et >= 0 && (et < pt || pt < 0) ) return et; @@ -1477,7 +1478,7 @@ void RemoteSerializer::Process() } if ( packets.length() ) - idle = false; + SetIdle(false); } void RemoteSerializer::Finish() @@ -1509,7 +1510,7 @@ bool RemoteSerializer::Poll(bool may_block) } io->Flush(); - idle = false; + SetIdle(false); switch ( msgstate ) { case TYPE: @@ -1691,7 +1692,7 @@ bool RemoteSerializer::DoMessage() case MSG_TERMINATE: assert(terminating); - io_sources.Terminate(); + iosource_mgr->Terminate(); return true; case MSG_REMOTE_PRINT: @@ -1879,7 +1880,7 @@ void RemoteSerializer::RemovePeer(Peer* peer) delete peer->cache_out; delete peer; - closed = ! IsActive(); + SetClosed(! IsActive()); if ( in_sync == peer ) in_sync = 0; @@ -2841,7 +2842,7 @@ void RemoteSerializer::GotEvent(const char* name, double time, BufferedEvent* e = new BufferedEvent; // Our time, not the time when the event was generated. - e->time = pkt_srcs.length() ? + e->time = iosource_mgr->GetPktSrcs().size() ? time_t(network_time) : time_t(timer_mgr->Time()); e->src = current_peer->id; @@ -3086,7 +3087,7 @@ RecordVal* RemoteSerializer::GetPeerVal(PeerID id) void RemoteSerializer::ChildDied() { Log(LogError, "child died"); - closed = true; + SetClosed(true); child_pid = 0; // Shut down the main process as well. @@ -3185,7 +3186,7 @@ void RemoteSerializer::FatalError(const char* msg) Log(LogError, msg); reporter->Error("%s", msg); - closed = true; + SetClosed(true); if ( kill(child_pid, SIGQUIT) < 0 ) reporter->Warning("warning: cannot kill child pid %d, %s", child_pid, strerror(errno)); @@ -3356,6 +3357,15 @@ SocketComm::~SocketComm() static unsigned int first_rtime = 0; +static void fd_vector_set(const std::vector& fds, fd_set* set, int* max) + { + for ( size_t i = 0; i < fds.size(); ++i ) + { + FD_SET(fds[i], set); + *max = ::max(fds[i], *max); + } + } + void SocketComm::Run() { first_rtime = (unsigned int) current_time(true); @@ -3377,10 +3387,9 @@ void SocketComm::Run() FD_ZERO(&fd_write); FD_ZERO(&fd_except); - int max_fd = 0; - + int max_fd = io->Fd(); FD_SET(io->Fd(), &fd_read); - max_fd = io->Fd(); + max_fd = std::max(max_fd, io->ExtraReadFDs().Set(&fd_read)); loop_over_list(peers, i) { @@ -3389,6 +3398,8 @@ void SocketComm::Run() FD_SET(peers[i]->io->Fd(), &fd_read); if ( peers[i]->io->Fd() > max_fd ) max_fd = peers[i]->io->Fd(); + max_fd = std::max(max_fd, + peers[i]->io->ExtraReadFDs().Set(&fd_read)); } else { @@ -3439,38 +3450,17 @@ void SocketComm::Run() if ( ! io->IsFillingUp() && shutting_conns_down ) shutting_conns_down = false; - // We cannot rely solely on select() as the there may - // be some data left in our input/output queues. So, we use - // a small timeout for select and check for data - // manually afterwards. - static long selects = 0; static long canwrites = 0; - static long timeouts = 0; ++selects; if ( io->CanWrite() ) ++canwrites; - // FIXME: Fine-tune this (timeouts, flush, etc.) - struct timeval small_timeout; - small_timeout.tv_sec = 0; - small_timeout.tv_usec = - io->CanWrite() || io->CanRead() ? 1 : 10; - -#if 0 - if ( ! io->CanWrite() ) - usleep(10); -#endif - - int a = select(max_fd + 1, &fd_read, &fd_write, &fd_except, - &small_timeout); - - if ( a == 0 ) - ++timeouts; + int a = select(max_fd + 1, &fd_read, &fd_write, &fd_except, 0); if ( selects % 100000 == 0 ) - Log(fmt("selects=%ld canwrites=%ld timeouts=%ld", selects, canwrites, timeouts)); + Log(fmt("selects=%ld canwrites=%ld", selects, canwrites)); if ( a < 0 ) // Ignore errors for now. diff --git a/src/RemoteSerializer.h b/src/RemoteSerializer.h index 9dbfbd9dae..2af7610a7c 100644 --- a/src/RemoteSerializer.h +++ b/src/RemoteSerializer.h @@ -6,7 +6,7 @@ #include "Dict.h" #include "List.h" #include "Serializer.h" -#include "IOSource.h" +#include "iosource/IOSource.h" #include "Stats.h" #include "File.h" #include "logging/WriterBackend.h" @@ -22,13 +22,13 @@ namespace threading { } // This class handles the communication done in Bro's main loop. -class RemoteSerializer : public Serializer, public IOSource { +class RemoteSerializer : public Serializer, public iosource::IOSource { public: RemoteSerializer(); virtual ~RemoteSerializer(); // Initialize the remote serializer (calling this will fork). - void Init(); + void Enable(); // FIXME: Use SourceID directly (or rename everything to Peer*). typedef SourceID PeerID; @@ -140,7 +140,8 @@ public: void Finish(); // Overidden from IOSource: - virtual void GetFds(int* read, int* write, int* except); + virtual void GetFds(iosource::FD_Set* read, iosource::FD_Set* write, + iosource::FD_Set* except); virtual double NextTimestamp(double* local_network_time); virtual void Process(); virtual TimerMgr::Tag* GetCurrentTag(); diff --git a/src/Serializer.cc b/src/Serializer.cc index 36b1c74000..7306b0ded0 100644 --- a/src/Serializer.cc +++ b/src/Serializer.cc @@ -19,6 +19,7 @@ #include "Conn.h" #include "Timer.h" #include "RemoteSerializer.h" +#include "iosource/Manager.h" Serializer::Serializer(SerializationFormat* arg_format) { @@ -1045,7 +1046,7 @@ EventPlayer::EventPlayer(const char* file) Error(fmt("event replayer: cannot open %s", file)); if ( ReadHeader() ) - io_sources.Register(this); + iosource_mgr->Register(this); } EventPlayer::~EventPlayer() @@ -1067,9 +1068,10 @@ void EventPlayer::GotFunctionCall(const char* name, double time, // We don't replay function calls. } -void EventPlayer::GetFds(int* read, int* write, int* except) +void EventPlayer::GetFds(iosource::FD_Set* read, iosource::FD_Set* write, + iosource::FD_Set* except) { - *read = fd; + read->Insert(fd); } double EventPlayer::NextTimestamp(double* local_network_time) @@ -1085,7 +1087,7 @@ double EventPlayer::NextTimestamp(double* local_network_time) { UnserialInfo info(this); Unserialize(&info); - closed = io->Eof(); + SetClosed(io->Eof()); } if ( ! ne_time ) @@ -1142,7 +1144,7 @@ bool Packet::Serialize(SerialInfo* info) const static BroFile* profiling_output = 0; #ifdef DEBUG -static PktDumper* dump = 0; +static iosource::PktDumper* dump = 0; #endif Packet* Packet::Unserialize(UnserialInfo* info) @@ -1188,7 +1190,7 @@ Packet* Packet::Unserialize(UnserialInfo* info) p->hdr = hdr; p->pkt = (u_char*) pkt; p->tag = tag; - p->hdr_size = get_link_header_size(p->link_type); + p->hdr_size = iosource::PktSrc::GetLinkHeaderSize(p->link_type); delete [] tag; @@ -1213,9 +1215,15 @@ Packet* Packet::Unserialize(UnserialInfo* info) if ( debug_logger.IsEnabled(DBG_TM) ) { if ( ! dump ) - dump = new PktDumper("tm.pcap"); + dump = iosource_mgr->OpenPktDumper("tm.pcap", true); - dump->Dump(p->hdr, p->pkt); + if ( dump ) + { + iosource::PktDumper::Packet dp; + dp.hdr = p->hdr; + dp.data = p->pkt; + dump->Dump(&dp); + } } #endif diff --git a/src/Serializer.h b/src/Serializer.h index 543797a7af..558dce2086 100644 --- a/src/Serializer.h +++ b/src/Serializer.h @@ -15,7 +15,7 @@ #include "SerialInfo.h" #include "IP.h" #include "Timer.h" -#include "IOSource.h" +#include "iosource/IOSource.h" #include "Reporter.h" class SerializationCache; @@ -350,12 +350,13 @@ public: }; // Plays a file of events back. -class EventPlayer : public FileSerializer, public IOSource { +class EventPlayer : public FileSerializer, public iosource::IOSource { public: EventPlayer(const char* file); virtual ~EventPlayer(); - virtual void GetFds(int* read, int* write, int* except); + virtual void GetFds(iosource::FD_Set* read, iosource::FD_Set* write, + iosource::FD_Set* except); virtual double NextTimestamp(double* local_network_time); virtual void Process(); virtual const char* Tag() { return "EventPlayer"; } diff --git a/src/Sessions.cc b/src/Sessions.cc index ec275a1689..43e55dd95a 100644 --- a/src/Sessions.cc +++ b/src/Sessions.cc @@ -167,7 +167,7 @@ void NetSessions::Done() void NetSessions::DispatchPacket(double t, const struct pcap_pkthdr* hdr, const u_char* pkt, int hdr_size, - PktSrc* src_ps) + iosource::PktSrc* src_ps) { const struct ip* ip_hdr = 0; const u_char* ip_data = 0; @@ -184,10 +184,7 @@ void NetSessions::DispatchPacket(double t, const struct pcap_pkthdr* hdr, // Blanket encapsulation hdr_size += encap_hdr_size; - if ( src_ps->FilterType() == TYPE_FILTER_NORMAL ) - NextPacket(t, hdr, pkt, hdr_size); - else - NextPacketSecondary(t, hdr, pkt, hdr_size, src_ps); + NextPacket(t, hdr, pkt, hdr_size); } void NetSessions::NextPacket(double t, const struct pcap_pkthdr* hdr, @@ -262,53 +259,6 @@ void NetSessions::NextPacket(double t, const struct pcap_pkthdr* hdr, DumpPacket(hdr, pkt); } -void NetSessions::NextPacketSecondary(double /* t */, const struct pcap_pkthdr* hdr, - const u_char* const pkt, int hdr_size, - const PktSrc* src_ps) - { - SegmentProfiler(segment_logger, "processing-secondary-packet"); - - ++num_packets_processed; - - uint32 caplen = hdr->caplen - hdr_size; - if ( caplen < sizeof(struct ip) ) - { - Weird("truncated_IP", hdr, pkt); - return; - } - - const struct ip* ip = (const struct ip*) (pkt + hdr_size); - if ( ip->ip_v == 4 ) - { - const secondary_program_list& spt = src_ps->ProgramTable(); - - loop_over_list(spt, i) - { - SecondaryProgram* sp = spt[i]; - if ( ! net_packet_match(sp->Program(), pkt, - hdr->len, hdr->caplen) ) - continue; - - val_list* args = new val_list; - StringVal* cmd_val = - new StringVal(sp->Event()->Filter()); - args->append(cmd_val); - IP_Hdr ip_hdr(ip, false); - args->append(ip_hdr.BuildPktHdrVal()); - // ### Need to queue event here. - try - { - sp->Event()->Event()->Call(args); - } - - catch ( InterpreterException& e ) - { /* Already reported. */ } - - delete args; - } - } - } - int NetSessions::CheckConnectionTag(Connection* conn) { if ( current_iosrc->GetCurrentTag() ) @@ -1440,14 +1390,24 @@ void NetSessions::DumpPacket(const struct pcap_pkthdr* hdr, return; if ( len == 0 ) - pkt_dumper->Dump(hdr, pkt); + { + iosource::PktDumper::Packet p; + p.hdr = hdr; + p.data = pkt; + pkt_dumper->Dump(&p); + } + else { struct pcap_pkthdr h = *hdr; h.caplen = len; if ( h.caplen > hdr->caplen ) reporter->InternalError("bad modified caplen"); - pkt_dumper->Dump(&h, pkt); + + iosource::PktDumper::Packet p; + p.hdr = &h; + p.data = pkt; + pkt_dumper->Dump(&p); } } diff --git a/src/Sessions.h b/src/Sessions.h index 06cdbca978..c46c092263 100644 --- a/src/Sessions.h +++ b/src/Sessions.h @@ -69,11 +69,11 @@ public: ~NetSessions(); // Main entry point for packet processing. Dispatches the packet - // either through NextPacket() or NextPacketSecondary(), optionally - // employing the packet sorter first. + // either through NextPacket(), optionally employing the packet + // sorter first. void DispatchPacket(double t, const struct pcap_pkthdr* hdr, const u_char* const pkt, int hdr_size, - PktSrc* src_ps); + iosource::PktSrc* src_ps); void Done(); // call to drain events before destructing @@ -221,10 +221,6 @@ protected: void NextPacket(double t, const struct pcap_pkthdr* hdr, const u_char* const pkt, int hdr_size); - void NextPacketSecondary(double t, const struct pcap_pkthdr* hdr, - const u_char* const pkt, int hdr_size, - const PktSrc* src_ps); - // Record the given packet (if a dumper is active). If len=0 // then the whole packet is recorded, otherwise just the first // len bytes. diff --git a/src/Type.cc b/src/Type.cc index f941041414..ead31f1b7d 100644 --- a/src/Type.cc +++ b/src/Type.cc @@ -1381,6 +1381,11 @@ void OpaqueType::Describe(ODesc* d) const d->Add(name.c_str()); } +void OpaqueType::DescribeReST(ODesc* d, bool roles_only) const + { + d->Add(fmt(":bro:type:`%s` of %s", type_name(Tag()), name.c_str())); + } + IMPLEMENT_SERIAL(OpaqueType, SER_OPAQUE_TYPE); bool OpaqueType::DoSerialize(SerialInfo* info) const diff --git a/src/Type.h b/src/Type.h index a4c9bda541..a9f1e42a6d 100644 --- a/src/Type.h +++ b/src/Type.h @@ -534,6 +534,7 @@ public: const string& Name() const { return name; } void Describe(ODesc* d) const; + void DescribeReST(ODesc* d, bool roles_only = false) const; protected: OpaqueType() { } diff --git a/src/analyzer/Analyzer.cc b/src/analyzer/Analyzer.cc index bd85f8263a..fb5602f96e 100644 --- a/src/analyzer/Analyzer.cc +++ b/src/analyzer/Analyzer.cc @@ -4,6 +4,7 @@ #include "Analyzer.h" #include "Manager.h" +#include "binpac.h" #include "analyzer/protocol/pia/PIA.h" #include "../Event.h" diff --git a/src/analyzer/protocol/dhcp/events.bif b/src/analyzer/protocol/dhcp/events.bif index 49a77d969e..bbd27c71f7 100644 --- a/src/analyzer/protocol/dhcp/events.bif +++ b/src/analyzer/protocol/dhcp/events.bif @@ -18,8 +18,8 @@ ## event dhcp_discover%(c: connection, msg: dhcp_msg, req_addr: addr, host_name: string%); -## Generated for DHCP messages of type *DHCPOFFER* (server to client in response to -## DHCPDISCOVER with offer of configuration parameters). +## Generated for DHCP messages of type *DHCPOFFER* (server to client in response +## to DHCPDISCOVER with offer of configuration parameters). ## ## c: The connection record describing the underlying UDP flow. ## @@ -34,7 +34,7 @@ event dhcp_discover%(c: connection, msg: dhcp_msg, req_addr: addr, host_name: st ## serv_addr: The server address specified by the message. ## ## host_name: Optional host name value. May differ from the host name requested -## from the client. +## from the client. ## ## .. bro:see:: dhcp_discover dhcp_request dhcp_decline dhcp_ack dhcp_nak ## dhcp_release dhcp_inform @@ -103,7 +103,7 @@ event dhcp_decline%(c: connection, msg: dhcp_msg, host_name: string%); ## serv_addr: The server address specified by the message. ## ## host_name: Optional host name value. May differ from the host name requested -## from the client. +## from the client. ## ## .. bro:see:: dhcp_discover dhcp_offer dhcp_request dhcp_decline dhcp_nak ## dhcp_release dhcp_inform diff --git a/src/analyzer/protocol/dnp3/dnp3-analyzer.pac b/src/analyzer/protocol/dnp3/dnp3-analyzer.pac index 2ae783c82e..2065237f45 100644 --- a/src/analyzer/protocol/dnp3/dnp3-analyzer.pac +++ b/src/analyzer/protocol/dnp3/dnp3-analyzer.pac @@ -4,6 +4,24 @@ connection DNP3_Conn(bro_analyzer: BroAnalyzer) { downflow = DNP3_Flow(false); }; +%header{ + uint64 bytestring_to_time(const_bytestring time48, size_t length); + %} + +%code{ + uint64 bytestring_to_time(const_bytestring time48, size_t length) + { + /* In DNP3, a timestamp is represented by 6 bytes since epoch + in milliseconds. The 6 bytes are stored in big endian format. */ + uint64 epochTime = 0; + + for ( unsigned int i = 0; i < length; i++ ) + epochTime = time48[length - i - 1] + epochTime * 256; + + return epochTime; + } + %} + flow DNP3_Flow(is_orig: bool) { flowunit = DNP3_PDU(is_orig) withcontext (connection, this); @@ -222,7 +240,7 @@ flow DNP3_Flow(is_orig: bool) { BifEvent::generate_dnp3_frozen_counter_32wFlagTime( connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), - is_orig(), flag, count_value, bytestring_to_val(time48)); + is_orig(), flag, count_value, bytestring_to_time(time48, sizeof(time48))); } return true; @@ -236,7 +254,7 @@ flow DNP3_Flow(is_orig: bool) { BifEvent::generate_dnp3_frozen_counter_16wFlagTime( connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), - is_orig(), flag, count_value, bytestring_to_val(time48)); + is_orig(), flag, count_value, bytestring_to_time(time48, sizeof(time48))); } return true; @@ -390,7 +408,7 @@ flow DNP3_Flow(is_orig: bool) { BifEvent::generate_dnp3_frozen_analog_input_32wTime( connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), - is_orig(), flag, frozen_value, bytestring_to_val(time48)); + is_orig(), flag, frozen_value, bytestring_to_time(time48, sizeof(time48))); } return true; @@ -404,7 +422,7 @@ flow DNP3_Flow(is_orig: bool) { BifEvent::generate_dnp3_frozen_analog_input_16wTime( connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), - is_orig(), flag, frozen_value, bytestring_to_val(time48)); + is_orig(), flag, frozen_value, bytestring_to_time(time48, sizeof(time48))); } return true; @@ -502,7 +520,7 @@ flow DNP3_Flow(is_orig: bool) { BifEvent::generate_dnp3_analog_input_event_32wTime( connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), - is_orig(), flag, value, bytestring_to_val(time48)); + is_orig(), flag, value, bytestring_to_time(time48, sizeof(time48))); } return true; @@ -516,7 +534,7 @@ flow DNP3_Flow(is_orig: bool) { BifEvent::generate_dnp3_analog_input_event_16wTime( connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), - is_orig(), flag, value, bytestring_to_val(time48)); + is_orig(), flag, value, bytestring_to_time(time48, sizeof(time48))); } return true; @@ -558,7 +576,7 @@ flow DNP3_Flow(is_orig: bool) { BifEvent::generate_dnp3_analog_input_event_SPwTime( connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), - is_orig(), flag, value, bytestring_to_val(time48)); + is_orig(), flag, value, bytestring_to_time(time48, sizeof(time48))); } return true; @@ -572,7 +590,7 @@ flow DNP3_Flow(is_orig: bool) { BifEvent::generate_dnp3_analog_input_event_DPwTime( connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), - is_orig(), flag, value_low, value_high, bytestring_to_val(time48)); + is_orig(), flag, value_low, value_high, bytestring_to_time(time48, sizeof(time48))); } return true; @@ -614,7 +632,7 @@ flow DNP3_Flow(is_orig: bool) { BifEvent::generate_dnp3_frozen_analog_input_event_32wTime( connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), - is_orig(), flag, frozen_value, bytestring_to_val(time48)); + is_orig(), flag, frozen_value, bytestring_to_time(time48, sizeof(time48))); } return true; @@ -628,7 +646,7 @@ flow DNP3_Flow(is_orig: bool) { BifEvent::generate_dnp3_frozen_analog_input_event_16wTime( connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), - is_orig(), flag, frozen_value, bytestring_to_val(time48)); + is_orig(), flag, frozen_value, bytestring_to_time(time48, sizeof(time48))); } return true; @@ -670,7 +688,7 @@ flow DNP3_Flow(is_orig: bool) { BifEvent::generate_dnp3_frozen_analog_input_event_SPwTime( connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), - is_orig(), flag, frozen_value, bytestring_to_val(time48)); + is_orig(), flag, frozen_value, bytestring_to_time(time48, sizeof(time48))); } return true; @@ -684,7 +702,7 @@ flow DNP3_Flow(is_orig: bool) { BifEvent::generate_dnp3_frozen_analog_input_event_DPwTime( connection()->bro_analyzer(), connection()->bro_analyzer()->Conn(), - is_orig(), flag, frozen_value_low, frozen_value_high, bytestring_to_val(time48)); + is_orig(), flag, frozen_value_low, frozen_value_high, bytestring_to_time(time48, sizeof(time48))); } return true; diff --git a/src/analyzer/protocol/dnp3/events.bif b/src/analyzer/protocol/dnp3/events.bif index 80f9504a9e..18fc42d6f7 100644 --- a/src/analyzer/protocol/dnp3/events.bif +++ b/src/analyzer/protocol/dnp3/events.bif @@ -130,11 +130,11 @@ event dnp3_frozen_counter_16wFlag%(c: connection, is_orig: bool, flag:count, cou ## Generated for DNP3 objects with the group number 21 and variation number 5 ## frozen counter 32 bit with flag and time -event dnp3_frozen_counter_32wFlagTime%(c: connection, is_orig: bool, flag:count, count_value: count, time48: string%); +event dnp3_frozen_counter_32wFlagTime%(c: connection, is_orig: bool, flag:count, count_value: count, time48: count%); ## Generated for DNP3 objects with the group number 21 and variation number 6 ## frozen counter 16 bit with flag and time -event dnp3_frozen_counter_16wFlagTime%(c: connection, is_orig: bool, flag:count, count_value: count, time48: string%); +event dnp3_frozen_counter_16wFlagTime%(c: connection, is_orig: bool, flag:count, count_value: count, time48: count%); ## Generated for DNP3 objects with the group number 21 and variation number 9 ## frozen counter 32 bit without flag @@ -178,11 +178,11 @@ event dnp3_frozen_analog_input_16wFlag%(c: connection, is_orig: bool, flag: coun ## Generated for DNP3 objects with the group number 31 and variation number 3 ## frozen analog input 32 bit with time-of-freeze -event dnp3_frozen_analog_input_32wTime%(c: connection, is_orig: bool, flag: count, frozen_value: count, time48: string%); +event dnp3_frozen_analog_input_32wTime%(c: connection, is_orig: bool, flag: count, frozen_value: count, time48: count%); ## Generated for DNP3 objects with the group number 31 and variation number 4 ## frozen analog input 16 bit with time-of-freeze -event dnp3_frozen_analog_input_16wTime%(c: connection, is_orig: bool, flag: count, frozen_value: count, time48: string%); +event dnp3_frozen_analog_input_16wTime%(c: connection, is_orig: bool, flag: count, frozen_value: count, time48: count%); ## Generated for DNP3 objects with the group number 31 and variation number 5 ## frozen analog input 32 bit without flag @@ -210,11 +210,11 @@ event dnp3_analog_input_event_16woTime%(c: connection, is_orig: bool, flag: coun ## Generated for DNP3 objects with the group number 32 and variation number 3 ## analog input event 32 bit with time -event dnp3_analog_input_event_32wTime%(c: connection, is_orig: bool, flag: count, value: count, time48: string%); +event dnp3_analog_input_event_32wTime%(c: connection, is_orig: bool, flag: count, value: count, time48: count%); ## Generated for DNP3 objects with the group number 32 and variation number 4 ## analog input event 16 bit with time -event dnp3_analog_input_event_16wTime%(c: connection, is_orig: bool, flag: count, value: count, time48: string%); +event dnp3_analog_input_event_16wTime%(c: connection, is_orig: bool, flag: count, value: count, time48: count%); ## Generated for DNP3 objects with the group number 32 and variation number 5 ## analog input event single-precision float point without time @@ -226,11 +226,11 @@ event dnp3_analog_input_event_DPwoTime%(c: connection, is_orig: bool, flag: coun ## Generated for DNP3 objects with the group number 32 and variation number 7 ## analog input event single-precision float point with time -event dnp3_analog_input_event_SPwTime%(c: connection, is_orig: bool, flag: count, value: count, time48: string%); +event dnp3_analog_input_event_SPwTime%(c: connection, is_orig: bool, flag: count, value: count, time48: count%); ## Generated for DNP3 objects with the group number 32 and variation number 8 ## analog input event double-precisiion float point with time -event dnp3_analog_input_event_DPwTime%(c: connection, is_orig: bool, flag: count, value_low: count, value_high: count, time48: string%); +event dnp3_analog_input_event_DPwTime%(c: connection, is_orig: bool, flag: count, value_low: count, value_high: count, time48: count%); ## Generated for DNP3 objects with the group number 33 and variation number 1 ## frozen analog input event 32 bit without time @@ -242,11 +242,11 @@ event dnp3_frozen_analog_input_event_16woTime%(c: connection, is_orig: bool, fla ## Generated for DNP3 objects with the group number 33 and variation number 3 ## frozen analog input event 32 bit with time -event dnp3_frozen_analog_input_event_32wTime%(c: connection, is_orig: bool, flag: count, frozen_value: count, time48: string%); +event dnp3_frozen_analog_input_event_32wTime%(c: connection, is_orig: bool, flag: count, frozen_value: count, time48: count%); ## Generated for DNP3 objects with the group number 33 and variation number 4 ## frozen analog input event 16 bit with time -event dnp3_frozen_analog_input_event_16wTime%(c: connection, is_orig: bool, flag: count, frozen_value: count, time48: string%); +event dnp3_frozen_analog_input_event_16wTime%(c: connection, is_orig: bool, flag: count, frozen_value: count, time48: count%); ## Generated for DNP3 objects with the group number 33 and variation number 5 ## frozen analog input event single-precision float point without time @@ -258,11 +258,11 @@ event dnp3_frozen_analog_input_event_DPwoTime%(c: connection, is_orig: bool, fla ## Generated for DNP3 objects with the group number 33 and variation number 7 ## frozen analog input event single-precision float point with time -event dnp3_frozen_analog_input_event_SPwTime%(c: connection, is_orig: bool, flag: count, frozen_value: count, time48: string%); +event dnp3_frozen_analog_input_event_SPwTime%(c: connection, is_orig: bool, flag: count, frozen_value: count, time48: count%); ## Generated for DNP3 objects with the group number 34 and variation number 8 ## frozen analog input event double-precision float point with time -event dnp3_frozen_analog_input_event_DPwTime%(c: connection, is_orig: bool, flag: count, frozen_value_low: count, frozen_value_high: count, time48: string%); +event dnp3_frozen_analog_input_event_DPwTime%(c: connection, is_orig: bool, flag: count, frozen_value_low: count, frozen_value_high: count, time48: count%); ## g70 event dnp3_file_transport%(c: connection, is_orig: bool, file_handle: count, block_num: count, file_data: string%); diff --git a/src/analyzer/protocol/finger/Plugin.cc b/src/analyzer/protocol/finger/Plugin.cc index 4fbfed8e12..7dbaaf702d 100644 --- a/src/analyzer/protocol/finger/Plugin.cc +++ b/src/analyzer/protocol/finger/Plugin.cc @@ -1,6 +1,5 @@ // See the file in the main distribution directory for copyright. - #include "plugin/Plugin.h" #include "Finger.h" diff --git a/src/analyzer/protocol/http/HTTP.cc b/src/analyzer/protocol/http/HTTP.cc index 02b6947b9f..4d00ca85aa 100644 --- a/src/analyzer/protocol/http/HTTP.cc +++ b/src/analyzer/protocol/http/HTTP.cc @@ -243,10 +243,13 @@ int HTTP_Entity::Undelivered(int64_t len) return 0; if ( is_partial_content ) + { precomputed_file_id = file_mgr->Gap(body_length, len, http_message->MyHTTP_Analyzer()->GetAnalyzerTag(), http_message->MyHTTP_Analyzer()->Conn(), http_message->IsOrig(), precomputed_file_id); + offset += len; + } else precomputed_file_id = file_mgr->Gap(body_length, len, http_message->MyHTTP_Analyzer()->GetAnalyzerTag(), @@ -542,12 +545,9 @@ HTTP_Message::HTTP_Message(HTTP_Analyzer* arg_analyzer, current_entity = 0; top_level = new HTTP_Entity(this, 0, expect_body); + entity_data_buffer = 0; BeginEntity(top_level); - buffer_offset = buffer_size = 0; - data_buffer = 0; - total_buffer_size = 0; - start_time = network_time; body_length = 0; content_gap_length = 0; @@ -557,6 +557,7 @@ HTTP_Message::HTTP_Message(HTTP_Analyzer* arg_analyzer, HTTP_Message::~HTTP_Message() { delete top_level; + delete [] entity_data_buffer; } Val* HTTP_Message::BuildMessageStat(const int interrupted, const char* msg) @@ -604,22 +605,14 @@ void HTTP_Message::Done(const int interrupted, const char* detail) } MyHTTP_Analyzer()->HTTP_MessageDone(is_orig, this); - - delete_strings(buffers); - - if ( data_buffer ) - { - delete data_buffer; - data_buffer = 0; - } } int HTTP_Message::Undelivered(int64_t len) { - if ( ! top_level ) - return 0; + HTTP_Entity* e = current_entity ? current_entity + : static_cast(top_level); - if ( ((HTTP_Entity*) top_level)->Undelivered(len) ) + if ( e && e->Undelivered(len) ) { content_gap_length += len; return 1; @@ -652,8 +645,6 @@ void HTTP_Message::EndEntity(mime::MIME_Entity* entity) body_length += ((HTTP_Entity*) entity)->BodyLength(); header_length += ((HTTP_Entity*) entity)->HeaderLength(); - DeliverEntityData(); - if ( http_end_entity ) { val_list* vl = new val_list(); @@ -720,31 +711,18 @@ void HTTP_Message::SubmitTrailingHeaders(mime::MIME_HeaderList& /* hlist */) void HTTP_Message::SubmitData(int len, const char* buf) { - if ( buf != (const char*) data_buffer->Bytes() + buffer_offset || - buffer_offset + len > buffer_size ) - { - reporter->AnalyzerError(MyHTTP_Analyzer(), - "HTTP message buffer misalignment"); - return; - } - - buffer_offset += len; - if ( buffer_offset >= buffer_size ) - { - buffers.push_back(data_buffer); - data_buffer = 0; - } + if ( http_entity_data ) + MyHTTP_Analyzer()->HTTP_EntityData(is_orig, + new BroString(reinterpret_cast(buf), len, 0)); } int HTTP_Message::RequestBuffer(int* plen, char** pbuf) { - if ( ! data_buffer ) - if ( ! InitBuffer(mime_segment_length) ) - return 0; - - *plen = data_buffer->Len() - buffer_offset; - *pbuf = (char*) data_buffer->Bytes() + buffer_offset; + if ( ! entity_data_buffer ) + entity_data_buffer = new char[http_entity_data_delivery_size]; + *plen = http_entity_data_delivery_size; + *pbuf = entity_data_buffer; return 1; } @@ -785,9 +763,6 @@ void HTTP_Message::SetPlainDelivery(int64_t length) if ( length > 0 && BifConst::skip_http_data ) content_line->SkipBytesAfterThisLine(length); - - if ( ! data_buffer ) - InitBuffer(length); } void HTTP_Message::SkipEntityData() @@ -796,87 +771,6 @@ void HTTP_Message::SkipEntityData() current_entity->SkipBody(); } -void HTTP_Message::DeliverEntityData() - { - if ( http_entity_data ) - { - const BroString* entity_data = 0; - - if ( data_buffer && buffer_offset > 0 ) - { - if ( buffer_offset < buffer_size ) - { - entity_data = new BroString(data_buffer->Bytes(), buffer_offset, 0); - delete data_buffer; - } - else - entity_data = data_buffer; - - data_buffer = 0; - - if ( buffers.empty() ) - MyHTTP_Analyzer()->HTTP_EntityData(is_orig, - entity_data); - else - buffers.push_back(entity_data); - - entity_data = 0; - } - - if ( ! buffers.empty() ) - { - if ( buffers.size() == 1 ) - { - entity_data = buffers[0]; - buffers.clear(); - } - else - { - entity_data = concatenate(buffers); - delete_strings(buffers); - } - - MyHTTP_Analyzer()->HTTP_EntityData(is_orig, entity_data); - } - } - else - { - delete_strings(buffers); - - if ( data_buffer ) - delete data_buffer; - - data_buffer = 0; - } - - total_buffer_size = 0; - } - -int HTTP_Message::InitBuffer(int64_t length) - { - if ( length <= 0 ) - return 0; - - if ( total_buffer_size >= http_entity_data_delivery_size ) - DeliverEntityData(); - - if ( total_buffer_size + length > http_entity_data_delivery_size ) - { - length = http_entity_data_delivery_size - total_buffer_size; - if ( length <= 0 ) - return 0; - } - - u_char* b = new u_char[length]; - data_buffer = new BroString(0, b, length); - - buffer_size = length; - total_buffer_size += length; - buffer_offset = 0; - - return 1; - } - void HTTP_Message::Weird(const char* msg) { analyzer->Weird(msg); @@ -1823,7 +1717,7 @@ void HTTP_Analyzer::ParseVersion(data_chunk_t ver, const IPAddr& host, } } -void HTTP_Analyzer::HTTP_EntityData(int is_orig, const BroString* entity_data) +void HTTP_Analyzer::HTTP_EntityData(int is_orig, BroString* entity_data) { if ( http_entity_data ) { @@ -1831,8 +1725,7 @@ void HTTP_Analyzer::HTTP_EntityData(int is_orig, const BroString* entity_data) vl->append(BuildConnVal()); vl->append(new Val(is_orig, TYPE_BOOL)); vl->append(new Val(entity_data->Len(), TYPE_COUNT)); - // FIXME: Make sure that removing the const here is indeed ok... - vl->append(new StringVal(const_cast(entity_data))); + vl->append(new StringVal(entity_data)); ConnectionEvent(http_entity_data, vl); } else diff --git a/src/analyzer/protocol/http/HTTP.h b/src/analyzer/protocol/http/HTTP.h index 5785d93198..876bcfe5aa 100644 --- a/src/analyzer/protocol/http/HTTP.h +++ b/src/analyzer/protocol/http/HTTP.h @@ -132,13 +132,7 @@ protected: tcp::ContentLine_Analyzer* content_line; bool is_orig; - vector buffers; - - // Controls the total buffer size within http_entity_data_delivery_size. - int total_buffer_size; - - int buffer_offset, buffer_size; - BroString* data_buffer; + char* entity_data_buffer; double start_time; @@ -151,9 +145,6 @@ protected: HTTP_Entity* current_entity; - int InitBuffer(int64_t length); - void DeliverEntityData(); - Val* BuildMessageStat(const int interrupted, const char* msg); }; @@ -165,7 +156,7 @@ public: void Undelivered(tcp::TCP_Endpoint* sender, uint64 seq, int len); void HTTP_Header(int is_orig, mime::MIME_Header* h); - void HTTP_EntityData(int is_orig, const BroString* entity_data); + void HTTP_EntityData(int is_orig, BroString* entity_data); void HTTP_MessageDone(int is_orig, HTTP_Message* message); void HTTP_Event(const char* category, const char* detail); void HTTP_Event(const char* category, StringVal *detail); diff --git a/src/analyzer/protocol/mime/MIME.cc b/src/analyzer/protocol/mime/MIME.cc index 6f992c9256..c3fde12a5d 100644 --- a/src/analyzer/protocol/mime/MIME.cc +++ b/src/analyzer/protocol/mime/MIME.cc @@ -142,8 +142,9 @@ int fputs(data_chunk_t b, FILE* fp) void MIME_Mail::Undelivered(int len) { // is_orig param not available, doesn't matter as long as it's consistent - file_mgr->Gap(cur_entity_len, len, analyzer->GetAnalyzerTag(), analyzer->Conn(), - false); + cur_entity_id = file_mgr->Gap(cur_entity_len, len, + analyzer->GetAnalyzerTag(), analyzer->Conn(), + false, cur_entity_id); } int strcasecmp_n(data_chunk_t s, const char* t) @@ -643,11 +644,7 @@ void MIME_Entity::EndOfData() if ( content_encoding == CONTENT_ENCODING_BASE64 ) FinishDecodeBase64(); - if ( data_buf_offset > 0 ) - { - SubmitData(data_buf_offset, data_buf_data); - data_buf_offset = -1; - } + FlushData(); } message->EndEntity (this); @@ -1001,6 +998,7 @@ void MIME_Entity::DecodeDataLine(int len, const char* data, int trailing_CRLF) DecodeBinary(len, data, trailing_CRLF); break; } + FlushData(); } void MIME_Entity::DecodeBinary(int len, const char* data, int trailing_CRLF) @@ -1179,6 +1177,15 @@ void MIME_Entity::DataOctets(int len, const char* data) } } +void MIME_Entity::FlushData() + { + if ( data_buf_offset > 0 ) + { + SubmitData(data_buf_offset, data_buf_data); + data_buf_offset = -1; + } + } + void MIME_Entity::SubmitHeader(MIME_Header* h) { message->SubmitHeader(h); @@ -1325,6 +1332,7 @@ MIME_Mail::~MIME_Mail() void MIME_Mail::BeginEntity(MIME_Entity* /* entity */) { cur_entity_len = 0; + cur_entity_id.clear(); if ( mime_begin_entity ) { @@ -1364,6 +1372,7 @@ void MIME_Mail::EndEntity(MIME_Entity* /* entity */) } file_mgr->EndOfFile(analyzer->GetAnalyzerTag(), analyzer->Conn()); + cur_entity_id.clear(); } void MIME_Mail::SubmitHeader(MIME_Header* h) @@ -1426,8 +1435,9 @@ void MIME_Mail::SubmitData(int len, const char* buf) } // is_orig param not available, doesn't matter as long as it's consistent - file_mgr->DataIn(reinterpret_cast(buf), len, - analyzer->GetAnalyzerTag(), analyzer->Conn(), false); + cur_entity_id = file_mgr->DataIn(reinterpret_cast(buf), len, + analyzer->GetAnalyzerTag(), analyzer->Conn(), false, + cur_entity_id); cur_entity_len += len; buffer_start = (buf + len) - (char*)data_buffer->Bytes(); diff --git a/src/analyzer/protocol/mime/MIME.h b/src/analyzer/protocol/mime/MIME.h index 2b2f88105d..3e25a85360 100644 --- a/src/analyzer/protocol/mime/MIME.h +++ b/src/analyzer/protocol/mime/MIME.h @@ -133,6 +133,7 @@ protected: int GetDataBuffer(); void DataOctet(char ch); void DataOctets(int len, const char* data); + void FlushData(); virtual void SubmitData(int len, const char* buf); virtual void SubmitHeader(MIME_Header* h); @@ -257,6 +258,7 @@ protected: BroString* data_buffer; uint64 cur_entity_len; + string cur_entity_id; }; diff --git a/src/analyzer/protocol/tcp/TCP_Reassembler.cc b/src/analyzer/protocol/tcp/TCP_Reassembler.cc index 053e8c8f60..0f7699011e 100644 --- a/src/analyzer/protocol/tcp/TCP_Reassembler.cc +++ b/src/analyzer/protocol/tcp/TCP_Reassembler.cc @@ -117,6 +117,45 @@ void TCP_Reassembler::SetContentsFile(BroFile* f) record_contents_file = f; } +static inline bool established(const TCP_Endpoint* a, const TCP_Endpoint* b) + { + return a->state == TCP_ENDPOINT_ESTABLISHED && + b->state == TCP_ENDPOINT_ESTABLISHED; + } + +static inline bool report_gap(const TCP_Endpoint* a, const TCP_Endpoint* b) + { + return content_gap && + ( BifConst::report_gaps_for_partial || established(a, b) ); + } + +void TCP_Reassembler::Gap(uint64 seq, uint64 len) + { + // Only report on content gaps for connections that + // are in a cleanly established state. In other + // states, these can arise falsely due to things + // like sequence number mismatches in RSTs, or + // unseen previous packets in partial connections. + // The one opportunity we lose here is on clean FIN + // handshakes, but Oh Well. + + if ( report_gap(endp, endp->peer) ) + { + val_list* vl = new val_list; + vl->append(dst_analyzer->BuildConnVal()); + vl->append(new Val(IsOrig(), TYPE_BOOL)); + vl->append(new Val(seq, TYPE_COUNT)); + vl->append(new Val(len, TYPE_COUNT)); + dst_analyzer->ConnectionEvent(content_gap, vl); + } + + if ( type == Direct ) + dst_analyzer->NextUndelivered(seq, len, IsOrig()); + else + dst_analyzer->ForwardUndelivered(seq, len, IsOrig()); + + had_gap = true; + } void TCP_Reassembler::Undelivered(uint64 up_to_seq) { @@ -189,48 +228,35 @@ void TCP_Reassembler::Undelivered(uint64 up_to_seq) if ( ! skip_deliveries ) { - // This can happen because we're processing a trace - // that's been filtered. For example, if it's just - // SYN/FIN data, then there can be data in the FIN - // packet, but it's undelievered because it's out of - // sequence. - - uint64 seq = last_reassem_seq; - uint64 len = up_to_seq - last_reassem_seq; - - // Only report on content gaps for connections that - // are in a cleanly established state. In other - // states, these can arise falsely due to things - // like sequence number mismatches in RSTs, or - // unseen previous packets in partial connections. - // The one opportunity we lose here is on clean FIN - // handshakes, but Oh Well. - - if ( content_gap && - (BifConst::report_gaps_for_partial || - (endpoint->state == TCP_ENDPOINT_ESTABLISHED && - peer->state == TCP_ENDPOINT_ESTABLISHED ) ) ) + // If we have blocks that begin below up_to_seq, deliver them. + DataBlock* b = blocks; + while ( b ) { - val_list* vl = new val_list; - vl->append(dst_analyzer->BuildConnVal()); - vl->append(new Val(IsOrig(), TYPE_BOOL)); - vl->append(new Val(seq, TYPE_COUNT)); - vl->append(new Val(len, TYPE_COUNT)); + if ( b->seq < last_reassem_seq ) + { + // Already delivered this block. + b = b->next; + continue; + } - dst_analyzer->ConnectionEvent(content_gap, vl); + if ( b->seq >= up_to_seq ) + // Block is beyond what we need to process at this point. + break; + + uint64 gap_at_seq = last_reassem_seq; + uint64 gap_len = b->seq - last_reassem_seq; + + Gap(gap_at_seq, gap_len); + last_reassem_seq += gap_len; + BlockInserted(b); + // Inserting a block may cause trimming of what's buffered, + // so have to assume 'b' is invalid, hence re-assign to start. + b = blocks; } - if ( type == Direct ) - dst_analyzer->NextUndelivered(last_reassem_seq, - len, IsOrig()); - else - { - dst_analyzer->ForwardUndelivered(last_reassem_seq, - len, IsOrig()); - } + if ( up_to_seq > last_reassem_seq ) + Gap(last_reassem_seq, up_to_seq - last_reassem_seq); } - - had_gap = true; } // We should record and match undelivered even if we are skipping @@ -243,7 +269,8 @@ void TCP_Reassembler::Undelivered(uint64 up_to_seq) MatchUndelivered(up_to_seq, false); // But we need to re-adjust last_reassem_seq in either case. - last_reassem_seq = up_to_seq; // we've done our best ... + if ( up_to_seq > last_reassem_seq ) + last_reassem_seq = up_to_seq; // we've done our best ... } void TCP_Reassembler::MatchUndelivered(uint64 up_to_seq, bool use_last_upper) diff --git a/src/analyzer/protocol/tcp/TCP_Reassembler.h b/src/analyzer/protocol/tcp/TCP_Reassembler.h index 3dfe75bf10..5d8badcef1 100644 --- a/src/analyzer/protocol/tcp/TCP_Reassembler.h +++ b/src/analyzer/protocol/tcp/TCP_Reassembler.h @@ -94,6 +94,7 @@ private: DECLARE_SERIAL(TCP_Reassembler); void Undelivered(uint64 up_to_seq); + void Gap(uint64 seq, uint64 len); void RecordToSeq(uint64 start_seq, uint64 stop_seq, BroFile* f); void RecordBlock(DataBlock* b, BroFile* f); diff --git a/src/bro.bif b/src/bro.bif index 1029896295..1757a9d12e 100644 --- a/src/bro.bif +++ b/src/bro.bif @@ -21,6 +21,7 @@ #include "IPAddr.h" #include "util.h" #include "file_analysis/Manager.h" +#include "iosource/Manager.h" using namespace std; @@ -33,7 +34,7 @@ TableType* var_sizes; // and hence it's declared in NetVar.{h,cc}. extern RecordType* gap_info; -static PktDumper* addl_pkt_dumper = 0; +static iosource::PktDumper* addl_pkt_dumper = 0; bro_int_t parse_int(const char*& fmt) { @@ -1675,11 +1676,14 @@ function net_stats%(%): NetStats unsigned int drop = 0; unsigned int link = 0; - loop_over_list(pkt_srcs, i) - { - PktSrc* ps = pkt_srcs[i]; + const iosource::Manager::PktSrcList& pkt_srcs(iosource_mgr->GetPktSrcs()); - struct PktSrc::Stats stat; + for ( iosource::Manager::PktSrcList::const_iterator i = pkt_srcs.begin(); + i != pkt_srcs.end(); i++ ) + { + iosource::PktSrc* ps = *i; + + struct iosource::PktSrc::Stats stat; ps->Statistics(&stat); recv += stat.received; drop += stat.dropped; @@ -3224,10 +3228,15 @@ function dump_current_packet%(file_name: string%) : bool return new Val(0, TYPE_BOOL); if ( ! addl_pkt_dumper ) - addl_pkt_dumper = new PktDumper(0, true); + addl_pkt_dumper = iosource_mgr->OpenPktDumper(file_name->CheckString(), true); - addl_pkt_dumper->Open(file_name->CheckString()); - addl_pkt_dumper->Dump(hdr, pkt); + if ( addl_pkt_dumper ) + { + iosource::PktDumper::Packet p; + p.hdr = hdr; + p.data = pkt; + addl_pkt_dumper->Dump(&p); + } return new Val(! addl_pkt_dumper->IsError(), TYPE_BOOL); %} @@ -3284,10 +3293,15 @@ function dump_packet%(pkt: pcap_packet, file_name: string%) : bool hdr.len = (*pkt_vl)[3]->AsCount(); if ( ! addl_pkt_dumper ) - addl_pkt_dumper = new PktDumper(0, true); + addl_pkt_dumper = iosource_mgr->OpenPktDumper(file_name->CheckString(), true); - addl_pkt_dumper->Open(file_name->CheckString()); - addl_pkt_dumper->Dump(&hdr, (*pkt_vl)[4]->AsString()->Bytes()); + if ( addl_pkt_dumper ) + { + iosource::PktDumper::Packet p; + p.hdr = &hdr; + p.data = (*pkt_vl)[4]->AsString()->Bytes(); + addl_pkt_dumper->Dump(&p); + } return new Val(addl_pkt_dumper->IsError(), TYPE_BOOL); %} @@ -4110,14 +4124,14 @@ function rotate_file_by_name%(f: string%): rotate_info bool is_addl_pkt_dumper = false; // Special case: one of current dump files. - if ( pkt_dumper && streq(pkt_dumper->FileName(), f->CheckString()) ) + if ( pkt_dumper && streq(pkt_dumper->Path().c_str(), f->CheckString()) ) { is_pkt_dumper = true; pkt_dumper->Close(); } if ( addl_pkt_dumper && - streq(addl_pkt_dumper->FileName(), f->CheckString()) ) + streq(addl_pkt_dumper->Path().c_str(), f->CheckString()) ) { is_addl_pkt_dumper = true; addl_pkt_dumper->Close(); @@ -4214,103 +4228,6 @@ function enable_raw_output%(f: file%): any # # =========================================================================== -## Precompiles a PCAP filter and binds it to a given identifier. -## -## id: The PCAP identifier to reference the filter *s* later on. -## -## s: The PCAP filter. See ``man tcpdump`` for valid expressions. -## -## Returns: True if *s* is valid and precompiles successfully. -## -## .. bro:see:: install_pcap_filter -## install_src_addr_filter -## install_src_net_filter -## uninstall_src_addr_filter -## uninstall_src_net_filter -## install_dst_addr_filter -## install_dst_net_filter -## uninstall_dst_addr_filter -## uninstall_dst_net_filter -## pcap_error -function precompile_pcap_filter%(id: PcapFilterID, s: string%): bool - %{ - bool success = true; - - loop_over_list(pkt_srcs, i) - { - pkt_srcs[i]->ClearErrorMsg(); - - if ( ! pkt_srcs[i]->PrecompileFilter(id->ForceAsInt(), - s->CheckString()) ) - { - reporter->Error("precompile_pcap_filter: %s", - pkt_srcs[i]->ErrorMsg()); - success = false; - } - } - - return new Val(success, TYPE_BOOL); - %} - -## Installs a PCAP filter that has been precompiled with -## :bro:id:`precompile_pcap_filter`. -## -## id: The PCAP filter id of a precompiled filter. -## -## Returns: True if the filter associated with *id* has been installed -## successfully. -## -## .. bro:see:: precompile_pcap_filter -## install_src_addr_filter -## install_src_net_filter -## uninstall_src_addr_filter -## uninstall_src_net_filter -## install_dst_addr_filter -## install_dst_net_filter -## uninstall_dst_addr_filter -## uninstall_dst_net_filter -## pcap_error -function install_pcap_filter%(id: PcapFilterID%): bool - %{ - bool success = true; - - loop_over_list(pkt_srcs, i) - { - pkt_srcs[i]->ClearErrorMsg(); - - if ( ! pkt_srcs[i]->SetFilter(id->ForceAsInt()) ) - success = false; - } - - return new Val(success, TYPE_BOOL); - %} - -## Returns a string representation of the last PCAP error. -## -## Returns: A descriptive error message of the PCAP function that failed. -## -## .. bro:see:: precompile_pcap_filter -## install_pcap_filter -## install_src_addr_filter -## install_src_net_filter -## uninstall_src_addr_filter -## uninstall_src_net_filter -## install_dst_addr_filter -## install_dst_net_filter -## uninstall_dst_addr_filter -## uninstall_dst_net_filter -function pcap_error%(%): string - %{ - loop_over_list(pkt_srcs, i) - { - const char* err = pkt_srcs[i]->ErrorMsg(); - if ( *err ) - return new StringVal(err); - } - - return new StringVal("no error"); - %} - ## Installs a filter to drop packets from a given IP source address with ## a certain probability if none of a given set of TCP flags are set. ## Note that for IPv6 packets with a Destination options header that has @@ -4542,7 +4459,7 @@ function enable_communication%(%): any return 0; using_communication = 1; - remote_serializer->Init(); + remote_serializer->Enable(); return 0; %} diff --git a/src/file_analysis/File.cc b/src/file_analysis/File.cc index 50d7d48336..4509fc7d42 100644 --- a/src/file_analysis/File.cc +++ b/src/file_analysis/File.cc @@ -437,7 +437,7 @@ void File::EndOfFile() void File::Gap(uint64 offset, uint64 len) { DBG_LOG(DBG_FILE_ANALYSIS, "[%s] Gap of size %" PRIu64 " at offset %" PRIu64, - id.c_str(), offset, len); + id.c_str(), len, offset); analyzers.DrainModifications(); diff --git a/src/BPF_Program.cc b/src/iosource/BPF_Program.cc similarity index 74% rename from src/BPF_Program.cc rename to src/iosource/BPF_Program.cc index 5260429eb0..70469c97e7 100644 --- a/src/BPF_Program.cc +++ b/src/iosource/BPF_Program.cc @@ -58,7 +58,14 @@ int pcap_compile_nopcap(int snaplen_arg, int linktype_arg, } #endif -BPF_Program::BPF_Program() : m_compiled(), m_program() +// Simple heuristic to identify filters that always match, so that we can +// skip the filtering in that case. "ip or not ip" is Bro's default filter. +static bool filter_matches_anything(const char *filter) + { + return (! filter) || strlen(filter) == 0 || strcmp(filter, "ip or not ip") == 0; + } + +BPF_Program::BPF_Program() : m_compiled(), m_matches_anything(false), m_program() { } @@ -86,12 +93,14 @@ bool BPF_Program::Compile(pcap_t* pcap, const char* filter, uint32 netmask, } m_compiled = true; + m_matches_anything = filter_matches_anything(filter); return true; } bool BPF_Program::Compile(int snaplen, int linktype, const char* filter, - uint32 netmask, char* errbuf, bool optimize) + uint32 netmask, char* errbuf, unsigned int errbuf_len, + bool optimize) { FreeCode(); @@ -99,15 +108,23 @@ bool BPF_Program::Compile(int snaplen, int linktype, const char* filter, char my_error[PCAP_ERRBUF_SIZE]; int err = pcap_compile_nopcap(snaplen, linktype, &m_program, - (char *) filter, optimize, netmask, error); + (char *) filter, optimize, netmask, my_error); if ( err < 0 && errbuf ) - safe_strncpy(errbuf, my_errbuf, PCAP_ERRBUF_SIZE); + safe_strncpy(errbuf, my_error, errbuf_len); + *errbuf = '\0'; #else int err = pcap_compile_nopcap(snaplen, linktype, &m_program, (char*) filter, optimize, netmask); + + if ( err < 0 && errbuf && errbuf_len ) + *errbuf = '\0'; #endif + if ( err == 0 ) + { m_compiled = true; + m_matches_anything = filter_matches_anything(filter); + } return err == 0; } diff --git a/src/BPF_Program.h b/src/iosource/BPF_Program.h similarity index 82% rename from src/BPF_Program.h rename to src/iosource/BPF_Program.h index 88ed669da2..88a4512d4e 100644 --- a/src/BPF_Program.h +++ b/src/iosource/BPF_Program.h @@ -30,12 +30,17 @@ public: // similarly to pcap_compile_nopcap(). Parameters are // similar. Returns true on success. bool Compile(int snaplen, int linktype, const char* filter, - uint32 netmask, char* errbuf = 0, bool optimize = true); + uint32 netmask, char* errbuf = 0, unsigned int errbuf_len = 0, + bool optimize = true); // Returns true if this program currently contains compiled // code, false otherwise. bool IsCompiled() { return m_compiled; } + // Returns true if this program matches any packets. This is not + // comprehensive, but can identify a few cases where it does. + bool MatchesAnything() { return m_matches_anything; } + // Accessor to the compiled program. Returns nil when // no program is currently compiled. bpf_program* GetProgram(); @@ -46,6 +51,7 @@ protected: // (I like to prefix member variables with m_, makes it clear // in the implementation whether it's a global or not. --ck) bool m_compiled; + bool m_matches_anything; struct bpf_program m_program; }; diff --git a/src/iosource/CMakeLists.txt b/src/iosource/CMakeLists.txt new file mode 100644 index 0000000000..a36667aee7 --- /dev/null +++ b/src/iosource/CMakeLists.txt @@ -0,0 +1,23 @@ + +include(BroSubdir) + +include_directories(BEFORE + ${CMAKE_CURRENT_SOURCE_DIR} + ${CMAKE_CURRENT_BINARY_DIR} +) + +add_subdirectory(pcap) + +set(iosource_SRCS + BPF_Program.cc + Component.cc + Manager.cc + PktDumper.cc + PktSrc.cc +) + +bif_target(pcap.bif) + +bro_add_subdir_library(iosource ${iosource_SRCS}) +add_dependencies(bro_iosource generate_outputs) + diff --git a/src/iosource/Component.cc b/src/iosource/Component.cc new file mode 100644 index 0000000000..1e4f029400 --- /dev/null +++ b/src/iosource/Component.cc @@ -0,0 +1,165 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#include "Component.h" + +#include "Desc.h" +#include "Reporter.h" + +using namespace iosource; + +Component::Component(const std::string& name) + : plugin::Component(plugin::component::IOSOURCE, name) + { + } + +Component::Component(plugin::component::Type type, const std::string& name) + : plugin::Component(type, name) + { + } + +Component::~Component() + { + } + +PktSrcComponent::PktSrcComponent(const std::string& arg_name, const std::string& arg_prefix, InputType arg_type, factory_callback arg_factory) + : iosource::Component(plugin::component::PKTSRC, arg_name) + { + tokenize_string(arg_prefix, "%", &prefixes); + type = arg_type; + factory = arg_factory; + } + +PktSrcComponent::~PktSrcComponent() + { + } + +const std::vector& PktSrcComponent::Prefixes() const + { + return prefixes; + } + +bool PktSrcComponent::HandlesPrefix(const string& prefix) const + { + for ( std::vector::const_iterator i = prefixes.begin(); + i != prefixes.end(); i++ ) + { + if ( *i == prefix ) + return true; + } + + return false; + } + +bool PktSrcComponent::DoesLive() const + { + return type == LIVE || type == BOTH; + } + +bool PktSrcComponent::DoesTrace() const + { + return type == TRACE || type == BOTH; + } + +PktSrcComponent::factory_callback PktSrcComponent::Factory() const + { + return factory; + } + +void PktSrcComponent::DoDescribe(ODesc* d) const + { + iosource::Component::DoDescribe(d); + + string prefs; + + for ( std::vector::const_iterator i = prefixes.begin(); + i != prefixes.end(); i++ ) + { + if ( prefs.size() ) + prefs += ", "; + + prefs += '"' + *i + '"'; + } + + d->Add("interface prefix"); + if ( prefixes.size() > 1 ) + d->Add("es"); + + d->Add(" "); + d->Add(prefs); + d->Add("; supports "); + + switch ( type ) { + case LIVE: + d->Add("live input"); + break; + + case TRACE: + d->Add("trace input"); + break; + + case BOTH: + d->Add("live and trace input"); + break; + + default: + reporter->InternalError("unknown PkrSrc type"); + } + + } + +PktDumperComponent::PktDumperComponent(const std::string& name, const std::string& arg_prefix, factory_callback arg_factory) + : plugin::Component(plugin::component::PKTDUMPER, name) + { + tokenize_string(arg_prefix, "%", &prefixes); + factory = arg_factory; + } + +PktDumperComponent::~PktDumperComponent() + { + } + +PktDumperComponent::factory_callback PktDumperComponent::Factory() const + { + return factory; + } + +const std::vector& PktDumperComponent::Prefixes() const + { + return prefixes; + } + +bool PktDumperComponent::HandlesPrefix(const string& prefix) const + { + for ( std::vector::const_iterator i = prefixes.begin(); + i != prefixes.end(); i++ ) + { + if ( *i == prefix ) + return true; + } + + return false; + } + +void PktDumperComponent::DoDescribe(ODesc* d) const + { + plugin::Component::DoDescribe(d); + + string prefs; + + for ( std::vector::const_iterator i = prefixes.begin(); + i != prefixes.end(); i++ ) + { + if ( prefs.size() ) + prefs += ", "; + + prefs += '"' + *i + '"'; + } + + d->Add("dumper prefix"); + + if ( prefixes.size() > 1 ) + d->Add("es"); + + d->Add(": "); + d->Add(prefs); + } diff --git a/src/iosource/Component.h b/src/iosource/Component.h new file mode 100644 index 0000000000..4a38a9cd22 --- /dev/null +++ b/src/iosource/Component.h @@ -0,0 +1,177 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#ifndef IOSOURCE_PLUGIN_COMPONENT_H +#define IOSOURCE_PLUGIN_COMPONENT_H + +#include +#include + +#include "plugin/Component.h" + +namespace iosource { + +class IOSource; +class PktSrc; +class PktDumper; + +/** + * Component description for plugins providing IOSources. + */ +class Component : public plugin::Component { +public: + typedef IOSource* (*factory_callback)(); + + /** + * Constructor. + * + * @param name A descriptive name for the component. This name must + * be unique across all components of this type. + */ + Component(const std::string& name); + + /** + * Copy constructor. + */ + Component(const Component& other); + + /** + * Destructor. + */ + ~Component(); + +protected: + /** + * Constructor to use by derived classes. + * + * @param type The type of the componnent. + * + * @param name A descriptive name for the component. This name must + * be unique across all components of this type. + */ + Component(plugin::component::Type type, const std::string& name); +}; + +/** + * Component description for plugins providing a PktSrc for packet input. + */ +class PktSrcComponent : public iosource::Component { +public: + /** + * Type of input a packet source supports. + */ + enum InputType { + LIVE, ///< Live input. + TRACE, ///< Offline input from trace file. + BOTH ///< Live input as well as offline. + }; + + typedef PktSrc* (*factory_callback)(const std::string& path, bool is_live); + + /** + * Constructor. + * + * @param name A descriptive name for the component. This name must + * be unique across all components of this type. + * + * @param prefixes The list of interface/file prefixes associated + * with this component. + * + * @param type Type of input the component supports. + * + * @param factor Factory function to instantiate component. + */ + PktSrcComponent(const std::string& name, const std::string& prefixes, InputType type, factory_callback factory); + + /** + * Destructor. + */ + virtual ~PktSrcComponent(); + + /** + * Returns the prefix(es) passed to the constructor. + */ + const std::vector& Prefixes() const; + + /** + * Returns true if the given prefix is among the one specified for the component. + */ + bool HandlesPrefix(const std::string& prefix) const; + + /** + * Returns true if packet source instantiated by the component handle + * live traffic. + */ + bool DoesLive() const; + + /** + * Returns true if packet source instantiated by the component handle + * offline traces. + */ + bool DoesTrace() const; + + /** + * Returns the source's factory function. + */ + factory_callback Factory() const; + + /** + * Generates a human-readable description of the component. This goes + * into the output of \c "bro -NN". + */ + virtual void DoDescribe(ODesc* d) const; + +private: + std::vector prefixes; + InputType type; + factory_callback factory; +}; + +/** + * Component description for plugins providing a PktDumper for packet output. + * + * PktDumpers aren't IOSurces but we locate them here to keep them along with + * the PktSrc. + */ +class PktDumperComponent : public plugin::Component { +public: + typedef PktDumper* (*factory_callback)(const std::string& path, bool append); + + /** + * XXX + */ + PktDumperComponent(const std::string& name, const std::string& prefixes, factory_callback factory); + + /** + * Destructor. + */ + ~PktDumperComponent(); + + /** + * Returns the prefix(es) passed to the constructor. + */ + const std::vector& Prefixes() const; + + /** + * Returns true if the given prefix is among the one specified for the component. + */ + bool HandlesPrefix(const std::string& prefix) const; + + /** + * Returns the source's factory function. + */ + factory_callback Factory() const; + + /** + * Generates a human-readable description of the component. This goes + * into the output of \c "bro -NN". + */ + virtual void DoDescribe(ODesc* d) const; + +private: + std::vector prefixes; + factory_callback factory; +}; + +} + +#endif diff --git a/src/iosource/FD_Set.h b/src/iosource/FD_Set.h new file mode 100644 index 0000000000..402d28b8cb --- /dev/null +++ b/src/iosource/FD_Set.h @@ -0,0 +1,87 @@ +#ifndef BRO_FD_SET_H +#define BRO_FD_SET_H + +#include +#include + +namespace iosource { + +/** + * A container holding a set of file descriptors. + */ +class FD_Set { +public: + + /** + * Constructor. The set is initially empty. + */ + FD_Set() : max(-1), fds() + { } + + /** + * Insert a file descriptor in to the set. + * @param fd the fd to insert in the set. + * @return false if fd was already in the set, else true. + */ + bool Insert(int fd) + { + if ( max < fd ) + max = fd; + + return fds.insert(fd).second; + } + + /** + * Inserts all the file descriptors from another set in to this one. + * @param other a file descriptor set to merge in to this one. + */ + void Insert(const FD_Set& other) + { + for ( std::set::const_iterator it = other.fds.begin(); + it != other.fds.end(); ++it ) + Insert(*it); + } + + /** + * Empties the set. + */ + void Clear() + { max = -1; fds.clear(); } + + /** + * Insert file descriptors in to a fd_set for use with select(). + * @return the greatest file descriptor inserted. + */ + int Set(fd_set* set) const + { + for ( std::set::const_iterator it = fds.begin(); it != fds.end(); + ++it ) + FD_SET(*it, set); + + return max; + } + + /** + * @return Whether a file descriptor belonging to this set is within the + * fd_set arugment. + */ + bool Ready(fd_set* set) const + { + for ( std::set::const_iterator it = fds.begin(); it != fds.end(); + ++it ) + { + if ( FD_ISSET(*it, set) ) + return true; + } + + return false; + } + +private: + int max; + std::set fds; +}; + +} // namespace bro + +#endif // BRO_FD_SET_H diff --git a/src/iosource/IOSource.h b/src/iosource/IOSource.h new file mode 100644 index 0000000000..df82012268 --- /dev/null +++ b/src/iosource/IOSource.h @@ -0,0 +1,140 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#ifndef IOSOURCE_IOSOURCE_H +#define IOSOURCE_IOSOURCE_H + +extern "C" { +#include +} + +#include +#include "FD_Set.h" +#include "Timer.h" + +namespace iosource { + +/** + * Interface class for components providing/consuming data inside Bro's main + * loop. + */ +class IOSource { +public: + /** + * Constructor. + */ + IOSource() { idle = false; closed = false; } + + /** + * Destructor. + */ + virtual ~IOSource() {} + + /** + * Returns true if source has nothing ready to process. + */ + bool IsIdle() const { return idle; } + + /** + * Returns true if more data is to be expected in the future. + * Otherwise, source may be removed. + */ + bool IsOpen() const { return ! closed; } + + /** + * Initializes the source. Can be overwritten by derived classes. + */ + virtual void Init() { } + + /** + * Finalizes the source when it's being closed. Can be overwritten by + * derived classes. + */ + virtual void Done() { } + + /** + * Returns select'able file descriptors for this source. Leaves the + * passed values untouched if not available. + * + * @param read Pointer to container where to insert a read descriptor. + * + * @param write Pointer to container where to insert a write descriptor. + * + * @param except Pointer to container where to insert a except descriptor. + */ + virtual void GetFds(FD_Set* read, FD_Set* write, FD_Set* except) = 0; + + /** + * Returns the timestamp (in \a global network time) associated with + * next data item from this source. If the source wants the data + * item to be processed with a local network time, it sets the + * argument accordingly. + * + * This method will be called only when either IsIdle() returns + * false, or select() on one of the fds returned by GetFDs() + * indicates that there's data to process. + * + * Must be overridden by derived classes. + * + * @param network_time A pointer to store the \a local network time + * associated with the next item (as opposed to global network time). + * + * @return The global network time of the next entry, or a value + * smaller than zero if none is available currently. + */ + virtual double NextTimestamp(double* network_time) = 0; + + /** + * Processes and consumes next data item. + * + * This method will be called only when either IsIdle() returns + * false, or select() on one of the fds returned by GetFDs() + * indicates that there's data to process. + * + * Must be overridden by derived classes. + */ + virtual void Process() = 0; + + /** + * Returns the tag of the timer manafger associated with the last + * procesees data item. + * + * Can be overridden by derived classes. + * + * @return The tag, or null for the global timer manager. + * + */ + virtual TimerMgr::Tag* GetCurrentTag() { return 0; } + + /** + * Returns a descriptual tag representing the source for debugging. + * + * Can be overridden by derived classes. + * + * @return The debugging name. + */ + virtual const char* Tag() = 0; + +protected: + /* + * Callback for derived classes to call when they have gone dry + * temporarily. + * + * @param is_idle True if the source is idle currently. + */ + void SetIdle(bool is_idle) { idle = is_idle; } + + /* + * Callback for derived class to call when they have shutdown. + * + * @param is_closed True if the source is now closed. + */ + void SetClosed(bool is_closed) { closed = is_closed; } + +private: + bool idle; + bool closed; +}; + +} + +#endif diff --git a/src/iosource/Manager.cc b/src/iosource/Manager.cc new file mode 100644 index 0000000000..26a48c2911 --- /dev/null +++ b/src/iosource/Manager.cc @@ -0,0 +1,311 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#include +#include +#include +#include + +#include + +#include "Manager.h" +#include "IOSource.h" +#include "PktSrc.h" +#include "PktDumper.h" +#include "plugin/Manager.h" + +#include "util.h" + +#define DEFAULT_PREFIX "pcap" + +using namespace iosource; + +Manager::~Manager() + { + for ( SourceList::iterator i = sources.begin(); i != sources.end(); ++i ) + { + (*i)->src->Done(); + delete *i; + } + + sources.clear(); + + for ( PktDumperList::iterator i = pkt_dumpers.begin(); i != pkt_dumpers.end(); ++i ) + { + (*i)->Done(); + delete *i; + } + + pkt_dumpers.clear(); + } + +void Manager::RemoveAll() + { + // We're cheating a bit here ... + dont_counts = sources.size(); + } + +IOSource* Manager::FindSoonest(double* ts) + { + // Remove sources which have gone dry. For simplicity, we only + // remove at most one each time. + for ( SourceList::iterator i = sources.begin(); + i != sources.end(); ++i ) + if ( ! (*i)->src->IsOpen() ) + { + (*i)->src->Done(); + delete *i; + sources.erase(i); + break; + } + + // Ideally, we would always call select on the fds to see which + // are ready, and return the soonest. Unfortunately, that'd mean + // one select-call per packet, which we can't afford in high-volume + // environments. Thus, we call select only every SELECT_FREQUENCY + // call (or if all sources report that they are dry). + + ++call_count; + + IOSource* soonest_src = 0; + double soonest_ts = 1e20; + double soonest_local_network_time = 1e20; + bool all_idle = true; + + // Find soonest source of those which tell us they have something to + // process. + for ( SourceList::iterator i = sources.begin(); i != sources.end(); ++i ) + { + if ( ! (*i)->src->IsIdle() ) + { + all_idle = false; + double local_network_time = 0; + double ts = (*i)->src->NextTimestamp(&local_network_time); + if ( ts > 0 && ts < soonest_ts ) + { + soonest_ts = ts; + soonest_src = (*i)->src; + soonest_local_network_time = + local_network_time ? + local_network_time : ts; + } + } + } + + // If we found one and aren't going to select this time, + // return it. + int maxx = 0; + + if ( soonest_src && (call_count % SELECT_FREQUENCY) != 0 ) + goto finished; + + // Select on the join of all file descriptors. + fd_set fd_read, fd_write, fd_except; + + FD_ZERO(&fd_read); + FD_ZERO(&fd_write); + FD_ZERO(&fd_except); + + for ( SourceList::iterator i = sources.begin(); + i != sources.end(); ++i ) + { + Source* src = (*i); + + if ( ! src->src->IsIdle() ) + // No need to select on sources which we know to + // be ready. + continue; + + src->Clear(); + src->src->GetFds(&src->fd_read, &src->fd_write, &src->fd_except); + src->SetFds(&fd_read, &fd_write, &fd_except, &maxx); + } + + // We can't block indefinitely even when all sources are dry: + // we're doing some IOSource-independent stuff in the main loop, + // so we need to return from time to time. (Instead of no time-out + // at all, we use a very small one. This lets FreeBSD trigger a + // BPF buffer switch on the next read when the hold buffer is empty + // while the store buffer isn't filled yet. + + struct timeval timeout; + + if ( all_idle ) + { + // Interesting: when all sources are dry, simply sleeping a + // bit *without* watching for any fd becoming ready may + // decrease CPU load. I guess that's because it allows + // the kernel's packet buffers to fill. - Robin + timeout.tv_sec = 0; + timeout.tv_usec = 20; // SELECT_TIMEOUT; + select(0, 0, 0, 0, &timeout); + } + + if ( ! maxx ) + // No selectable fd at all. + goto finished; + + timeout.tv_sec = 0; + timeout.tv_usec = 0; + + if ( select(maxx + 1, &fd_read, &fd_write, &fd_except, &timeout) > 0 ) + { // Find soonest. + for ( SourceList::iterator i = sources.begin(); + i != sources.end(); ++i ) + { + Source* src = (*i); + + if ( ! src->src->IsIdle() ) + continue; + + if ( src->Ready(&fd_read, &fd_write, &fd_except) ) + { + double local_network_time = 0; + double ts = src->src->NextTimestamp(&local_network_time); + if ( ts > 0.0 && ts < soonest_ts ) + { + soonest_ts = ts; + soonest_src = src->src; + soonest_local_network_time = + local_network_time ? + local_network_time : ts; + } + } + } + } + +finished: + *ts = soonest_local_network_time; + return soonest_src; + } + +void Manager::Register(IOSource* src, bool dont_count) + { + src->Init(); + Source* s = new Source; + s->src = src; + if ( dont_count ) + ++dont_counts; + + sources.push_back(s); + } + +void Manager::Register(PktSrc* src) + { + pkt_srcs.push_back(src); + Register(src, false); + } + +static std::pair split_prefix(std::string path) + { + // See if the path comes with a prefix telling us which type of + // PktSrc to use. If not, choose default. + std::string prefix; + + std::string::size_type i = path.find("%"); + if ( i != std::string::npos ) + { + prefix = path.substr(0, i); + path = path.substr(++i, std::string::npos); + } + + else + prefix= DEFAULT_PREFIX; + + return std::make_pair(prefix, path); + } + +PktSrc* Manager::OpenPktSrc(const std::string& path, bool is_live) + { + std::pair t = split_prefix(path); + std::string prefix = t.first; + std::string npath = t.second; + + // Find the component providing packet sources of the requested prefix. + + PktSrcComponent* component = 0; + + std::list all_components = plugin_mgr->Components(); + + for ( std::list::const_iterator i = all_components.begin(); + i != all_components.end(); i++ ) + { + PktSrcComponent* c = *i; + + if ( c->HandlesPrefix(prefix) && + (( is_live && c->DoesLive() ) || + (! is_live && c->DoesTrace())) ) + { + component = c; + break; + } + } + + + if ( ! component ) + reporter->FatalError("type of packet source '%s' not recognized, or mode not supported", prefix.c_str()); + + // Instantiate packet source. + + PktSrc* ps = (*component->Factory())(npath, is_live); + assert(ps); + + if ( ! ps->IsOpen() && ps->IsError() ) + // Set an error message if it didn't open successfully. + ps->Error("could not open"); + + DBG_LOG(DBG_PKTIO, "Created packet source of type %s for %s", component->Name().c_str(), npath.c_str()); + + Register(ps); + return ps; + } + + +PktDumper* Manager::OpenPktDumper(const string& path, bool append) + { + std::pair t = split_prefix(path); + std::string prefix = t.first; + std::string npath = t.second; + + // Find the component providing packet dumpers of the requested prefix. + + PktDumperComponent* component = 0; + + std::list all_components = plugin_mgr->Components(); + + for ( std::list::const_iterator i = all_components.begin(); + i != all_components.end(); i++ ) + { + if ( (*i)->HandlesPrefix(prefix) ) + { + component = (*i); + break; + } + } + + if ( ! component ) + reporter->FatalError("type of packet dumper '%s' not recognized", prefix.c_str()); + + // Instantiate packet dumper. + + PktDumper* pd = (*component->Factory())(npath, append); + assert(pd); + + if ( ! pd->IsOpen() && pd->IsError() ) + // Set an error message if it didn't open successfully. + pd->Error("could not open"); + + DBG_LOG(DBG_PKTIO, "Created packer dumper of type %s for %s", component->Name().c_str(), npath.c_str()); + + pd->Init(); + pkt_dumpers.push_back(pd); + + return pd; + } + +void Manager::Source::SetFds(fd_set* read, fd_set* write, fd_set* except, + int* maxx) const + { + *maxx = std::max(*maxx, fd_read.Set(read)); + *maxx = std::max(*maxx, fd_write.Set(write)); + *maxx = std::max(*maxx, fd_except.Set(except)); + } diff --git a/src/iosource/Manager.h b/src/iosource/Manager.h new file mode 100644 index 0000000000..fb4f6676b6 --- /dev/null +++ b/src/iosource/Manager.h @@ -0,0 +1,146 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#ifndef IOSOURCE_MANAGER_H +#define IOSOURCE_MANAGER_H + +#include +#include +#include "iosource/FD_Set.h" + +namespace iosource { + +class IOSource; +class PktSrc; +class PktDumper; + +/** + * Singleton class managing all IOSources. + */ +class Manager { +public: + /** + * Constructor. + */ + Manager() { call_count = 0; dont_counts = 0; } + + /** + * Destructor. + */ + ~Manager(); + + /** + * Registers an IOSource with the manager. + * + * @param src The source. The manager takes ownership. + * + * @param dont_count If true, this source does not contribute to the + * number of IOSources returned by Size(). The effect is that if all + * sources except for the non-counting ones have gone dry, processing + * will shut down. + */ + void Register(IOSource* src, bool dont_count = false); + + /** + * Returns the packet source with the soonest available input. This + * may block for a little while if all are dry. + * + * @param ts A pointer where to store the timestamp of the input that + * the soonest source has available next. + * + * @return The source, or null if no source has input. + */ + IOSource* FindSoonest(double* ts); + + /** + * Returns the number of registered and still active sources, + * excluding those that are registered as \a dont_cont. + */ + int Size() const { return sources.size() - dont_counts; } + + typedef std::list PktSrcList; + + /** + * Returns a list of all registered PktSrc instances. This is a + * subset of all registered IOSource instances. + */ + const PktSrcList& GetPktSrcs() const { return pkt_srcs; } + + /** + * Terminate all processing immediately by removing all sources (and + * therefore now returning a Size() of zero). + */ + void Terminate() { RemoveAll(); } + + /** + * Opens a new packet source. + * + * @param path The interface or file name, as one would give to Bro \c -i. + * + * @param is_live True if \a path represents a live interface, false + * for a file. + * + * @return The new packet source, or null if an error occured. + */ + PktSrc* OpenPktSrc(const std::string& path, bool is_live); + + /** + * Opens a new packet dumper. + * + * @param path The file name to dump into. + * + * @param append True to append if \a path already exists. + * + * @return The new packet dumper, or null if an error occured. + */ + PktDumper* OpenPktDumper(const std::string& path, bool append); + +private: + /** + * When looking for a source with something to process, every + * SELECT_FREQUENCY calls we will go ahead and block on a select(). + */ + static const int SELECT_FREQUENCY = 25; + + /** + * Microseconds to wait in an empty select if no source is ready. + */ + static const int SELECT_TIMEOUT = 50; + + void Register(PktSrc* src); + void RemoveAll(); + + unsigned int call_count; + int dont_counts; + + struct Source { + IOSource* src; + FD_Set fd_read; + FD_Set fd_write; + FD_Set fd_except; + + bool Ready(fd_set* read, fd_set* write, fd_set* except) const + { return fd_read.Ready(read) || fd_write.Ready(write) || + fd_except.Ready(except); } + + void SetFds(fd_set* read, fd_set* write, fd_set* except, + int* maxx) const; + + void Clear() + { fd_read.Clear(); fd_write.Clear(); fd_except.Clear(); } + }; + + typedef std::list SourceList; + SourceList sources; + + typedef std::list PktDumperList; + + PktSrcList pkt_srcs; + PktDumperList pkt_dumpers; +}; + +} + +extern iosource::Manager* iosource_mgr; + +#endif + diff --git a/src/iosource/PktDumper.cc b/src/iosource/PktDumper.cc new file mode 100644 index 0000000000..a4bc3a82f8 --- /dev/null +++ b/src/iosource/PktDumper.cc @@ -0,0 +1,84 @@ + +// See the file "COPYING" in the main distribution directory for copyright. + +#include +#include + +#include "config.h" + +#include "PktDumper.h" + +using namespace iosource; + +PktDumper::PktDumper() + { + is_open = false; + errmsg = ""; + } + +PktDumper::~PktDumper() + { + } + +void PktDumper::Init() + { + Open(); + } + +void PktDumper::Done() + { + Close(); + } + +const std::string& PktDumper::Path() const + { + return props.path; + } + +bool PktDumper::IsOpen() const + { + return is_open; + } + +double PktDumper::OpenTime() const + { + return is_open ? props.open_time : 0; + } + +bool PktDumper::IsError() const + { + return errmsg.size(); + } + +const char* PktDumper::ErrorMsg() const + { + return errmsg.size() ? errmsg.c_str() : 0; + } + +int PktDumper::HdrSize() const + { + return is_open ? props.hdr_size : -1; + } + +void PktDumper::Opened(const Properties& arg_props) + { + is_open = true; + props = arg_props; + DBG_LOG(DBG_PKTIO, "Opened dumper %s", props.path.c_str()); + } + +void PktDumper::Closed() + { + is_open = false; + DBG_LOG(DBG_PKTIO, "Closed dumper %s", props.path.c_str()); + props.path = ""; + } + +void PktDumper::Error(const std::string& msg) + { + errmsg = msg; + + DBG_LOG(DBG_PKTIO, "Error with dumper %s: %s", + IsOpen() ? props.path.c_str() : "", + msg.c_str()); + } diff --git a/src/iosource/PktDumper.h b/src/iosource/PktDumper.h new file mode 100644 index 0000000000..56555c247a --- /dev/null +++ b/src/iosource/PktDumper.h @@ -0,0 +1,163 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#ifndef IOSOURCE_PKTSRC_PKTDUMPER_H +#define IOSOURCE_PKTSRC_PKTDUMPER_H + +#include "IOSource.h" + +namespace iosource { + +/** + * Base class for packet dumpers. + */ +class PktDumper { +public: + /** + * Structure describing a packet. + */ + struct Packet { + /** + * The pcap header associated with the packet. + */ + const struct pcap_pkthdr* hdr; + + /** + * The full content of the packet. + */ + const unsigned char* data; + }; + + /** + * Constructor. + */ + PktDumper(); + + /** + * Destructor. + */ + virtual ~PktDumper(); + + /** + * Returns the path associated with the dumper. + */ + const std::string& Path() const; + + /** + * Returns true if the dumper is open for writing. + */ + bool IsOpen() const; + + /** + * Returns the time when the dumper was opened for writing. + */ + double OpenTime() const; + + /** + * Returns returns true if the dumper has encountered an error. + */ + bool IsError() const; + + /** + * Returns if the dumper has encountered an error, returns a + * corresponding error message. Returns an emoty string otherwise. + */ + const char* ErrorMsg() const; + + /** + * Returns the size of the link-layer headers with this dumper. + */ + int HdrSize() const; + + /** + * Writes a packet to the dumper. + * + * @param pkt The packet to record. + */ + bool Record(const Packet* pkt); + + // PktDumper interface for derived classes to implement. + + /** + * Called by the manager system to open the source. + * + * Derived classes must implement this method. If successful, the + * implementation must call \a Opened(); if not, it must call Error() + * with a corresponding message. + */ + virtual void Open() = 0; + + /** + * Called by the manager system to close the dumper. + * + * Derived classes must implement this method. If successful, the + * implementation must call \a Closed(); if not, it must call Error() + * with a corresponding message. + */ + virtual void Close() = 0; + + /** + * Called to write a packet to the dumper. + * + * Derived classes must implement this method. + * + * @param pkt The packet to record. + * + * @return True if succesful, false otherwise (in which case \a + * Error() must have been called.) + */ + virtual bool Dump(const Packet* pkt) = 0; + +protected: + friend class Manager; + + /** + * Structure to pass back information about the packet dumper to the + * base class. Derived class pass an instance of this to \a Opened(). + */ + struct Properties { + std::string path; + int hdr_size; + double open_time; + }; + + /** + * Called from the implementations of \a Open() to signal that the + * source has been successully opened. + * + * @param props A properties instance describing the now open source. + */ + void Opened(const Properties& props); + + /** + * Called from the implementations of \a Close() to signal that the + * source has been closed. + */ + void Closed(); + + /** + * Called from derived classes to signal an error. + * + * @param msg A corresponding error message. + */ + void Error(const std::string& msg); + + /** + * Called by the manager to initialize the dumper. + */ + void Init(); + + /** + * Called by the manager to shutdown the dumper. + */ + void Done(); + +private: + bool is_open; + Properties props; + + std::string errmsg; +}; + +} + +#endif diff --git a/src/iosource/PktSrc.cc b/src/iosource/PktSrc.cc new file mode 100644 index 0000000000..6ee95a48a1 --- /dev/null +++ b/src/iosource/PktSrc.cc @@ -0,0 +1,524 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#include +#include + +#include "config.h" + +#include "util.h" +#include "PktSrc.h" +#include "Hash.h" +#include "Net.h" +#include "Sessions.h" + +using namespace iosource; + +PktSrc::PktSrc() + { + have_packet = false; + errbuf = ""; + SetClosed(true); + + next_sync_point = 0; + first_timestamp = 0.0; + first_wallclock = current_wallclock = 0; + } + +PktSrc::~PktSrc() + { + BPF_Program* code; + IterCookie* cookie = filters.InitForIteration(); + while ( (code = filters.NextEntry(cookie)) ) + delete code; + } + +const std::string& PktSrc::Path() const + { + static std::string not_open("not open"); + return IsOpen() ? props.path : not_open; + } + +const char* PktSrc::ErrorMsg() const + { + return errbuf.size() ? errbuf.c_str() : 0; + } + +int PktSrc::LinkType() const + { + return IsOpen() ? props.link_type : -1; + } + +uint32 PktSrc::Netmask() const + { + return IsOpen() ? props.netmask : PCAP_NETMASK_UNKNOWN; + } + +bool PktSrc::IsError() const + { + return ErrorMsg(); + } + +int PktSrc::HdrSize() const + { + return IsOpen() ? props.hdr_size : -1; + } + +int PktSrc::SnapLen() const + { + return snaplen; // That's a global. Change? + } + +bool PktSrc::IsLive() const + { + return props.is_live; + } + +double PktSrc::CurrentPacketTimestamp() + { + return current_pseudo; + } + +double PktSrc::CurrentPacketWallClock() + { + // We stop time when we are suspended. + if ( net_is_processing_suspended() ) + current_wallclock = current_time(true); + + return current_wallclock; + } + +void PktSrc::Opened(const Properties& arg_props) + { + if ( arg_props.hdr_size < 0 ) + { + char buf[512]; + safe_snprintf(buf, sizeof(buf), + "unknown data link type 0x%x", props.link_type); + Error(buf); + Close(); + return; + } + + props = arg_props; + SetClosed(false); + + if ( ! PrecompileFilter(0, "") || ! SetFilter(0) ) + { + Close(); + return; + } + + if ( props.is_live ) + Info(fmt("listening on %s, capture length %d bytes\n", props.path.c_str(), SnapLen())); + + DBG_LOG(DBG_PKTIO, "Opened source %s", props.path.c_str()); + } + +void PktSrc::Closed() + { + SetClosed(true); + + DBG_LOG(DBG_PKTIO, "Closed source %s", props.path.c_str()); + } + +void PktSrc::Error(const std::string& msg) + { + // We don't report this immediately, Bro will ask us for the error + // once it notices we aren't open. + errbuf = msg; + DBG_LOG(DBG_PKTIO, "Error with source %s: %s", + IsOpen() ? props.path.c_str() : "", + msg.c_str()); + } + +void PktSrc::Info(const std::string& msg) + { + reporter->Info("%s", msg.c_str()); + } + +void PktSrc::Weird(const std::string& msg, const Packet* p) + { + sessions->Weird(msg.c_str(), p->hdr, p->data, 0); + } + +void PktSrc::InternalError(const std::string& msg) + { + reporter->InternalError("%s", msg.c_str()); + } + +void PktSrc::ContinueAfterSuspend() + { + current_wallclock = current_time(true); + } + +int PktSrc::GetLinkHeaderSize(int link_type) + { + switch ( link_type ) { + case DLT_NULL: + return 4; + + case DLT_EN10MB: + return 14; + + case DLT_FDDI: + return 13 + 8; // fddi_header + LLC + +#ifdef DLT_LINUX_SLL + case DLT_LINUX_SLL: + return 16; +#endif + + case DLT_PPP_SERIAL: // PPP_SERIAL + return 4; + + case DLT_RAW: + return 0; + } + + return -1; + } + +double PktSrc::CheckPseudoTime() + { + if ( ! IsOpen() ) + return 0; + + if ( ! ExtractNextPacketInternal() ) + return 0; + + if ( remote_trace_sync_interval ) + { + if ( next_sync_point == 0 || current_packet.ts >= next_sync_point ) + { + int n = remote_serializer->SendSyncPoint(); + next_sync_point = first_timestamp + + n * remote_trace_sync_interval; + remote_serializer->Log(RemoteSerializer::LogInfo, + fmt("stopping at packet %.6f, next sync-point at %.6f", + current_packet.ts, next_sync_point)); + + return 0; + } + } + + double pseudo_time = current_packet.ts - first_timestamp; + double ct = (current_time(true) - first_wallclock) * pseudo_realtime; + + return pseudo_time <= ct ? bro_start_time + pseudo_time : 0; + } + +void PktSrc::Init() + { + Open(); + } + +void PktSrc::Done() + { + if ( IsOpen() ) + Close(); + } + +void PktSrc::GetFds(iosource::FD_Set* read, iosource::FD_Set* write, + iosource::FD_Set* except) + { + if ( pseudo_realtime ) + { + // Select would give erroneous results. But we simulate it + // by setting idle accordingly. + SetIdle(CheckPseudoTime() == 0); + return; + } + + if ( IsOpen() && props.selectable_fd >= 0 ) + read->Insert(props.selectable_fd); + } + +double PktSrc::NextTimestamp(double* local_network_time) + { + if ( ! IsOpen() ) + return -1.0; + + if ( ! ExtractNextPacketInternal() ) + return -1.0; + + if ( pseudo_realtime ) + { + // Delay packet if necessary. + double packet_time = CheckPseudoTime(); + if ( packet_time ) + return packet_time; + + SetIdle(true); + return -1.0; + } + + return current_packet.ts; + } + +void PktSrc::Process() + { + if ( ! IsOpen() ) + return; + + if ( ! ExtractNextPacketInternal() ) + return; + + int pkt_hdr_size = props.hdr_size; + + // Unfortunately some packets on the link might have MPLS labels + // while others don't. That means we need to ask the link-layer if + // labels are in place. + bool have_mpls = false; + + int protocol = 0; + const u_char* data = current_packet.data; + + switch ( props.link_type ) { + case DLT_NULL: + { + protocol = (data[3] << 24) + (data[2] << 16) + (data[1] << 8) + data[0]; + + // From the Wireshark Wiki: "AF_INET6, unfortunately, has + // different values in {NetBSD,OpenBSD,BSD/OS}, + // {FreeBSD,DragonFlyBSD}, and {Darwin/Mac OS X}, so an IPv6 + // packet might have a link-layer header with 24, 28, or 30 + // as the AF_ value." As we may be reading traces captured on + // platforms other than what we're running on, we accept them + // all here. + if ( protocol != AF_INET + && protocol != AF_INET6 + && protocol != 24 + && protocol != 28 + && protocol != 30 ) + { + Weird("non_ip_packet_in_null_transport", ¤t_packet); + goto done; + } + + break; + } + + case DLT_EN10MB: + { + // Get protocol being carried from the ethernet frame. + protocol = (data[12] << 8) + data[13]; + + switch ( protocol ) + { + // MPLS carried over the ethernet frame. + case 0x8847: + // Remove the data link layer and denote a + // header size of zero before the IP header. + have_mpls = true; + data += GetLinkHeaderSize(props.link_type); + pkt_hdr_size = 0; + break; + + // VLAN carried over the ethernet frame. + case 0x8100: + data += GetLinkHeaderSize(props.link_type); + + // Check for MPLS in VLAN. + if ( ((data[2] << 8) + data[3]) == 0x8847 ) + have_mpls = true; + + data += 4; // Skip the vlan header + pkt_hdr_size = 0; + + // Check for 802.1ah (Q-in-Q) containing IP. + // Only do a second layer of vlan tag + // stripping because there is no + // specification that allows for deeper + // nesting. + if ( ((data[2] << 8) + data[3]) == 0x0800 ) + data += 4; + + break; + + // PPPoE carried over the ethernet frame. + case 0x8864: + data += GetLinkHeaderSize(props.link_type); + protocol = (data[6] << 8) + data[7]; + data += 8; // Skip the PPPoE session and PPP header + pkt_hdr_size = 0; + + if ( protocol != 0x0021 && protocol != 0x0057 ) + { + // Neither IPv4 nor IPv6. + Weird("non_ip_packet_in_pppoe_encapsulation", ¤t_packet); + goto done; + } + break; + } + + break; + } + + case DLT_PPP_SERIAL: + { + // Get PPP protocol. + protocol = (data[2] << 8) + data[3]; + + if ( protocol == 0x0281 ) + { + // MPLS Unicast. Remove the data link layer and + // denote a header size of zero before the IP header. + have_mpls = true; + data += GetLinkHeaderSize(props.link_type); + pkt_hdr_size = 0; + } + + else if ( protocol != 0x0021 && protocol != 0x0057 ) + { + // Neither IPv4 nor IPv6. + Weird("non_ip_packet_in_ppp_encapsulation", ¤t_packet); + goto done; + } + break; + } + } + + if ( have_mpls ) + { + // Skip the MPLS label stack. + bool end_of_stack = false; + + while ( ! end_of_stack ) + { + end_of_stack = *(data + 2) & 0x01; + data += 4; + } + } + + if ( pseudo_realtime ) + { + current_pseudo = CheckPseudoTime(); + net_packet_dispatch(current_pseudo, current_packet.hdr, data, pkt_hdr_size, this); + if ( ! first_wallclock ) + first_wallclock = current_time(true); + } + + else + net_packet_dispatch(current_packet.ts, current_packet.hdr, data, pkt_hdr_size, this); + +done: + have_packet = 0; + DoneWithPacket(); + } + +const char* PktSrc::Tag() + { + return "PktSrc"; + } + +bool PktSrc::ExtractNextPacketInternal() + { + if ( have_packet ) + return true; + + have_packet = false; + + // Don't return any packets if processing is suspended (except for the + // very first packet which we need to set up times). + if ( net_is_processing_suspended() && first_timestamp ) + { + SetIdle(true); + return 0; + } + + if ( pseudo_realtime ) + current_wallclock = current_time(true); + + if ( ExtractNextPacket(¤t_packet) ) + { + if ( ! first_timestamp ) + first_timestamp = current_packet.ts; + + have_packet = true; + return 1; + } + + if ( pseudo_realtime && using_communication && ! IsOpen() ) + { + // Source has gone dry, we're done. + if ( remote_trace_sync_interval ) + remote_serializer->SendFinalSyncPoint(); + else + remote_serializer->Terminate(); + } + + SetIdle(true); + return 0; + } + +bool PktSrc::PrecompileBPFFilter(int index, const std::string& filter) + { + if ( index < 0 ) + return false; + + char errbuf[PCAP_ERRBUF_SIZE]; + + // Compile filter. + BPF_Program* code = new BPF_Program(); + + if ( ! code->Compile(SnapLen(), LinkType(), filter.c_str(), Netmask(), errbuf, sizeof(errbuf)) ) + { + string msg = fmt("cannot compile BPF filter \"%s\"", filter.c_str()); + + if ( *errbuf ) + msg += ": " + string(errbuf); + + Error(msg); + + delete code; + return 0; + } + + // Store it in hash. + HashKey* hash = new HashKey(HashKey(bro_int_t(index))); + BPF_Program* oldcode = filters.Lookup(hash); + if ( oldcode ) + delete oldcode; + + filters.Insert(hash, code); + delete hash; + + return 1; + } + +BPF_Program* PktSrc::GetBPFFilter(int index) + { + if ( index < 0 ) + return 0; + + HashKey* hash = new HashKey(HashKey(bro_int_t(index))); + BPF_Program* code = filters.Lookup(hash); + delete hash; + return code; + } + +bool PktSrc::ApplyBPFFilter(int index, const struct pcap_pkthdr *hdr, const u_char *pkt) + { + BPF_Program* code = GetBPFFilter(index); + + if ( ! code ) + { + Error(fmt("BPF filter %d not compiled", index)); + Close(); + } + + if ( code->MatchesAnything() ) + return true; + + return pcap_offline_filter(code->GetProgram(), hdr, pkt); + } + +bool PktSrc::GetCurrentPacket(const pcap_pkthdr** hdr, const u_char** pkt) + { + if ( ! have_packet ) + return false; + + *hdr = current_packet.hdr; + *pkt = current_packet.data; + return true; + } diff --git a/src/iosource/PktSrc.h b/src/iosource/PktSrc.h new file mode 100644 index 0000000000..9c05115257 --- /dev/null +++ b/src/iosource/PktSrc.h @@ -0,0 +1,417 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#ifndef IOSOURCE_PKTSRC_PKTSRC_H +#define IOSOURCE_PKTSRC_PKTSRC_H + +#include "IOSource.h" +#include "BPF_Program.h" +#include "Dict.h" + +declare(PDict,BPF_Program); + +namespace iosource { + +/** + * Base class for packet sources. + */ +class PktSrc : public IOSource { +public: + /** + * Struct for returning statistics on a packet source. + */ + struct Stats { + /** + * Packets received by source after filtering (w/o drops). + */ + unsigned int received; + + /** + * Packets dropped by source. + */ + unsigned int dropped; // pkts dropped + + /** + * Total number of packets on link before filtering. + * Optional, can be left unset if not available. + */ + unsigned int link; + + Stats() { received = dropped = link = 0; } + }; + + /** + * Constructor. + */ + PktSrc(); + + /** + * Destructor. + */ + virtual ~PktSrc(); + + /** + * Returns the path associated with the source. This is the interface + * name for live source, and a filename for offline sources. + */ + const std::string& Path() const; + + /** + * Returns true if this is a live source. + */ + bool IsLive() const; + + /** + * Returns the link type of the source. + */ + int LinkType() const; + + /** + * Returns the netmask associated with the source, or \c + * PCAP_NETMASK_UNKNOWN if unknown. + */ + uint32 Netmask() const; + + /** + * Returns true if the source has flagged an error. + */ + bool IsError() const; + + /** + * If the source encountered an error, returns a corresponding error + * message. Returns an empty string otherwise. + */ + const char* ErrorMsg() const; + + /** + * Returns the size of the link-layer header for this source. + */ + int HdrSize() const; + + /** + * Returns the snap length for this source. + */ + int SnapLen() const; + + /** + * In pseudo-realtime mode, returns the logical timestamp of the + * current packet. Undefined if not running pseudo-realtime mode. + */ + double CurrentPacketTimestamp(); + + /** + * In pseudo-realtime mode, returns the wall clock time associated + * with current packet. Undefined if not running pseudo-realtime + * mode. + */ + double CurrentPacketWallClock(); + + /** + * Signals packet source that processing is going to be continued + * after previous suspension. + */ + void ContinueAfterSuspend(); + + /** + * Precompiles a BPF filter and associates the given index with it. + * The compiled filter will be then available via \a GetBPFFilter(). + * + * This is primarily a helper for packet source implementation that + * want to apply BPF filtering to their packets. + * + * @param index The index to associate with the filter. + * + * @param BPF filter The filter string to precompile. + * + * @return True on success, false if a problem occurred. + */ + bool PrecompileBPFFilter(int index, const std::string& filter); + + /** + * Returns the precompiled BPF filter associated with a given index, + * if any, as compiled by \a PrecompileBPFFilter(). + * + * This is primarily a helper for packet source implementation that + * want to apply BPF filtering to their packets. + * + * @return The BPF filter associated, or null if none has been + * (successfully) compiled. + */ + BPF_Program* GetBPFFilter(int index); + + /** + * Applies a precompiled BPF filter to a packet. This will close the + * source with an error message if no filter with that index has been + * compiled. + * + * This is primarily a helper for packet source implementation that + * want to apply BPF filtering to their packets. + * + * @param index The index of the filter to apply. + * + * @param hdr The header of the packet to filter. + * + * @param pkt The content of the packet to filter. + * + * @return True if it maches. */ + bool ApplyBPFFilter(int index, const struct pcap_pkthdr *hdr, const u_char *pkt); + + /** + * Returns the packet currently being processed, if available. + * + * @param hdr A pointer to pass the header of the current packet back. + * + * @param pkt A pointer to pass the content of the current packet + * back. + * + * @return True if the current packet is available, or false if not. + */ + bool GetCurrentPacket(const pcap_pkthdr** hdr, const u_char** pkt); + + // PacketSource interace for derived classes to override. + + /** + * Precompiles a filter and associates a given index with it. The + * filter syntax is defined by the packet source's implenentation. + * + * Derived classes must implement this to implement their filtering. + * If they want to use BPF but don't support it natively, they can + * call the corresponding helper method provided by \a PktSrc. + * + * @param index The index to associate with the filter + * + * @param filter The filter string to precompile. + * + * @return True on success, false if a problem occurred or filtering + * is not supported. + */ + virtual bool PrecompileFilter(int index, const std::string& filter) = 0; + + /** + * Activates a precompiled filter with the given index. + * + * Derived classes must implement this to implement their filtering. + * If they want to use BPF but don't support it natively, they can + * call the corresponding helper method provided by \a PktSrc. + * + * @param index The index of the filter to activate. + * + * @return True on success, false if a problem occurred or the + * filtering is not supported. + */ + virtual bool SetFilter(int index) = 0; + + /** + * Returns current statistics about the source. + * + * Derived classes must implement this method. + * + * @param stats A statistics structure that the method fill out. + */ + virtual void Statistics(Stats* stats) = 0; + + /** + * Helper method to return the header size for a given link tyoe. + * + * @param link_type The link tyoe. + * + * @return The header size in bytes. + */ + static int GetLinkHeaderSize(int link_type); + +protected: + friend class Manager; + + // Methods to use by derived classes. + + /** + * Structure to pass back information about the packet source to the + * base class. Derived class pass an instance of this to \a Opened(). + */ + struct Properties { + /** + * The path associated with the source. This is the interface + * name for live source, and a filename for offline sources. + */ + std::string path; + + /** + * A file descriptor suitable to use with \a select() for + * determining if there's input available from this source. + */ + int selectable_fd; + + /** + * The link type for packets from this source. + */ + int link_type; + + /** + * The size of the link-layer header for packets from this + * source. \a GetLinkHeaderSize() may be used to derive this + * value. + */ + int hdr_size; + + /** + * The netmask associated with the source, or \c + * PCAP_NETMASK_UNKNOWN if unknown. + */ + uint32 netmask; + + /** + * True if the source is reading live inout, false for + * working offline. + */ + bool is_live; + + Properties() + { + netmask = PCAP_NETMASK_UNKNOWN; + } + }; + + /** + * Structure describing a packet. + */ + struct Packet { + /** + * Time associated with the packet. + */ + double ts; + + /** + * The pcap header associated with the packet. + */ + const struct ::pcap_pkthdr* hdr; + + /** + * The full content of the packet. + */ + const u_char* data; + }; + + /** + * Called from the implementations of \a Open() to signal that the + * source has been successully opened. + * + * @param props A properties instance describing the now open source. + */ + void Opened(const Properties& props); + + /** + * Called from the implementations of \a Close() to signal that the + * source has been closed. + */ + void Closed(); + + /** + * Can be called from derived classes to send an informational + * message to the user. + * + * @param msg The message to pass on. + */ + void Info(const std::string& msg); + + /** + * Can be called from derived classes to flag send an error. + * + * @param msg The message going with the error. + */ + void Error(const std::string& msg); + + /** + * Can be called from derived classes to flah a "weird" situation. + * + * @param msg The message to pass on. + * + * @param pkt The packet associated with the weird, or null if none. + */ + void Weird(const std::string& msg, const Packet* pkt); + + /** + * Can be called from derived classes to flag an internal error, + * which will abort execution. + * + * @param msg The message to pass on. + */ + void InternalError(const std::string& msg); + + // PktSrc interface for derived classes to implement. + + /** + * Called by the manager system to open the source. + * + * Derived classes must implement this method. If successful, the + * implementation must call \a Opened(); if not, it must call Error() + * with a corresponding message. + */ + virtual void Open() = 0; + + /** + * Called by the manager system to close the source. + * + * Derived classes must implement this method. If successful, the + * implementation must call \a Closed(); if not, it must call Error() + * with a corresponding message. + */ + virtual void Close() = 0; + + /** + * Provides the next packet from the source. + * + * @param pkt The packet structure to fill in with the packet's + * information. The callee keep ownership of the data but must + * guaranetee that it stays available at least until \a + * DoneWithPacket() is called. It is guaranteed that no two calls to + * this method will hapen with \a DoneWithPacket() in between. + * + * @return True if a packet is available and *pkt* filled in. False + * if not packet is available or an error occured (which must be + * flageed via Error()). + */ + virtual bool ExtractNextPacket(Packet* pkt) = 0; + + /** + * Signals that the data of previously extracted packet will no + * longer be needed. + */ + virtual void DoneWithPacket() = 0; + +private: + // Checks if the current packet has a pseudo-time <= current_time. If + // yes, returns pseudo-time, otherwise 0. + double CheckPseudoTime(); + + // Internal helper for ExtractNextPacket(). + bool ExtractNextPacketInternal(); + + // IOSource interface implementation. + virtual void Init(); + virtual void Done(); + virtual void GetFds(iosource::FD_Set* read, iosource::FD_Set* write, + iosource::FD_Set* except); + virtual double NextTimestamp(double* local_network_time); + virtual void Process(); + virtual const char* Tag(); + + Properties props; + + bool have_packet; + Packet current_packet; + + // For BPF filtering support. + PDict(BPF_Program) filters; + + // Only set in pseudo-realtime mode. + double first_timestamp; + double first_wallclock; + double current_wallclock; + double current_pseudo; + double next_sync_point; // For trace synchronziation in pseudo-realtime + + std::string errbuf; +}; + +} + +#endif diff --git a/src/iosource/pcap.bif b/src/iosource/pcap.bif new file mode 100644 index 0000000000..ee4e1e6c06 --- /dev/null +++ b/src/iosource/pcap.bif @@ -0,0 +1,104 @@ + +## Precompiles a PCAP filter and binds it to a given identifier. +## +## id: The PCAP identifier to reference the filter *s* later on. +## +## s: The PCAP filter. See ``man tcpdump`` for valid expressions. +## +## Returns: True if *s* is valid and precompiles successfully. +## +## .. bro:see:: install_pcap_filter +## install_src_addr_filter +## install_src_net_filter +## uninstall_src_addr_filter +## uninstall_src_net_filter +## install_dst_addr_filter +## install_dst_net_filter +## uninstall_dst_addr_filter +## uninstall_dst_net_filter +## pcap_error +function precompile_pcap_filter%(id: PcapFilterID, s: string%): bool + %{ + bool success = true; + + const iosource::Manager::PktSrcList& pkt_srcs(iosource_mgr->GetPktSrcs()); + + for ( iosource::Manager::PktSrcList::const_iterator i = pkt_srcs.begin(); + i != pkt_srcs.end(); i++ ) + { + iosource::PktSrc* ps = *i; + + if ( ! ps->PrecompileFilter(id->ForceAsInt(), + s->CheckString()) ) + success = false; + } + + return new Val(success, TYPE_BOOL); + %} + +## Installs a PCAP filter that has been precompiled with +## :bro:id:`precompile_pcap_filter`. +## +## id: The PCAP filter id of a precompiled filter. +## +## Returns: True if the filter associated with *id* has been installed +## successfully. +## +## .. bro:see:: precompile_pcap_filter +## install_src_addr_filter +## install_src_net_filter +## uninstall_src_addr_filter +## uninstall_src_net_filter +## install_dst_addr_filter +## install_dst_net_filter +## uninstall_dst_addr_filter +## uninstall_dst_net_filter +## pcap_error +function install_pcap_filter%(id: PcapFilterID%): bool + %{ + bool success = true; + + const iosource::Manager::PktSrcList& pkt_srcs(iosource_mgr->GetPktSrcs()); + + for ( iosource::Manager::PktSrcList::const_iterator i = pkt_srcs.begin(); + i != pkt_srcs.end(); i++ ) + { + iosource::PktSrc* ps = *i; + + if ( ! ps->SetFilter(id->ForceAsInt()) ) + success = false; + } + + return new Val(success, TYPE_BOOL); + %} + +## Returns a string representation of the last PCAP error. +## +## Returns: A descriptive error message of the PCAP function that failed. +## +## .. bro:see:: precompile_pcap_filter +## install_pcap_filter +## install_src_addr_filter +## install_src_net_filter +## uninstall_src_addr_filter +## uninstall_src_net_filter +## install_dst_addr_filter +## install_dst_net_filter +## uninstall_dst_addr_filter +## uninstall_dst_net_filter +function pcap_error%(%): string + %{ + const iosource::Manager::PktSrcList& pkt_srcs(iosource_mgr->GetPktSrcs()); + + for ( iosource::Manager::PktSrcList::const_iterator i = pkt_srcs.begin(); + i != pkt_srcs.end(); i++ ) + { + iosource::PktSrc* ps = *i; + + const char* err = ps->ErrorMsg(); + if ( *err ) + return new StringVal(err); + } + + return new StringVal("no error"); + %} diff --git a/src/iosource/pcap/CMakeLists.txt b/src/iosource/pcap/CMakeLists.txt new file mode 100644 index 0000000000..1c57bb6ac9 --- /dev/null +++ b/src/iosource/pcap/CMakeLists.txt @@ -0,0 +1,8 @@ + +include(BroPlugin) + +include_directories(BEFORE ${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR}) + +bro_plugin_begin(Bro Pcap) +bro_plugin_cc(Source.cc Dumper.cc Plugin.cc) +bro_plugin_end() diff --git a/src/iosource/pcap/Dumper.cc b/src/iosource/pcap/Dumper.cc new file mode 100644 index 0000000000..5d0b5e599b --- /dev/null +++ b/src/iosource/pcap/Dumper.cc @@ -0,0 +1,112 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#include +#include + +#include "Dumper.h" +#include "../PktSrc.h" +#include "../../Net.h" + +using namespace iosource::pcap; + +PcapDumper::PcapDumper(const std::string& path, bool arg_append) + { + append = arg_append; + props.path = path; + dumper = 0; + pd = 0; + } + +PcapDumper::~PcapDumper() + { + } + +void PcapDumper::Open() + { + int linktype = -1; + + pd = pcap_open_dead(DLT_EN10MB, snaplen); + if ( ! pd ) + { + Error("error for pcap_open_dead"); + return; + } + + if ( props.path.empty() ) + { + Error("no filename given"); + return; + } + + struct stat s; + int exists = 0; + + if ( append ) + { + // See if output file already exists (and is non-empty). + exists = stat(props.path.c_str(), &s); ; + + if ( exists < 0 && errno != ENOENT ) + { + Error(fmt("can't stat file %s: %s", props.path.c_str(), strerror(errno))); + return; + } + } + + if ( ! append || exists < 0 || s.st_size == 0 ) + { + // Open new file. + dumper = pcap_dump_open(pd, props.path.c_str()); + if ( ! dumper ) + { + Error(pcap_geterr(pd)); + return; + } + } + + else + { + // Old file and we need to append, which, unfortunately, + // is not supported by libpcap. So, we have to hack a + // little bit, knowing that pcap_dumpter_t is, in fact, + // a FILE ... :-( + dumper = (pcap_dumper_t*) fopen(props.path.c_str(), "a"); + if ( ! dumper ) + { + Error(fmt("can't open dump %s: %s", props.path.c_str(), strerror(errno))); + return; + } + } + + props.open_time = network_time; + props.hdr_size = PktSrc::GetLinkHeaderSize(pcap_datalink(pd)); + Opened(props); + } + +void PcapDumper::Close() + { + if ( ! dumper ) + return; + + pcap_dump_close(dumper); + pcap_close(pd); + dumper = 0; + pd = 0; + + Closed(); + } + +bool PcapDumper::Dump(const Packet* pkt) + { + if ( ! dumper ) + return false; + + pcap_dump((u_char*) dumper, pkt->hdr, pkt->data); + + return true; + } + +iosource::PktDumper* PcapDumper::Instantiate(const std::string& path, bool append) + { + return new PcapDumper(path, append); + } diff --git a/src/iosource/pcap/Dumper.h b/src/iosource/pcap/Dumper.h new file mode 100644 index 0000000000..7950912d56 --- /dev/null +++ b/src/iosource/pcap/Dumper.h @@ -0,0 +1,41 @@ +// See the file in the main distribution directory for copyright. + +#ifndef IOSOURCE_PKTSRC_PCAP_DUMPER_H +#define IOSOURCE_PKTSRC_PCAP_DUMPER_H + +extern "C" { +#include +} + +#include "../PktDumper.h" + +namespace iosource { +namespace pcap { + +class PcapDumper : public PktDumper { +public: + PcapDumper(const std::string& path, bool append); + virtual ~PcapDumper(); + + static PktDumper* Instantiate(const std::string& path, bool appen); + +protected: + // PktDumper interface. + virtual void Open(); + virtual void Close(); + virtual bool Dump(const Packet* pkt); + +private: + Properties props; + + bool append; + pcap_dumper_t* dumper; + pcap_t* pd; +}; + +} +} + +#endif + + diff --git a/src/iosource/pcap/Plugin.cc b/src/iosource/pcap/Plugin.cc new file mode 100644 index 0000000000..f0490e6e3d --- /dev/null +++ b/src/iosource/pcap/Plugin.cc @@ -0,0 +1,27 @@ +// See the file in the main distribution directory for copyright. + +#include "plugin/Plugin.h" + +#include "Source.h" +#include "Dumper.h" + +namespace plugin { +namespace Bro_Pcap { + +class Plugin : public plugin::Plugin { +public: + plugin::Configuration Configure() + { + AddComponent(new ::iosource::PktSrcComponent("PcapReader", "pcap", ::iosource::PktSrcComponent::BOTH, ::iosource::pcap::PcapSource::Instantiate)); + AddComponent(new ::iosource::PktDumperComponent("PcapWriter", "pcap", ::iosource::pcap::PcapDumper::Instantiate)); + + plugin::Configuration config; + config.name = "Bro::Pcap"; + config.description = "Packet aquisition via libpcap"; + return config; + } +} plugin; + +} +} + diff --git a/src/iosource/pcap/Source.cc b/src/iosource/pcap/Source.cc new file mode 100644 index 0000000000..e96933aaa6 --- /dev/null +++ b/src/iosource/pcap/Source.cc @@ -0,0 +1,272 @@ +// See the file in the main distribution directory for copyright. + +#include + +#include "config.h" + +#include "Source.h" + +#ifdef HAVE_PCAP_INT_H +#include +#endif + +using namespace iosource::pcap; + +PcapSource::~PcapSource() + { + Close(); + } + +PcapSource::PcapSource(const std::string& path, bool is_live) + { + props.path = path; + props.is_live = is_live; + last_data = 0; + } + +void PcapSource::Open() + { + if ( props.is_live ) + OpenLive(); + else + OpenOffline(); + } + +void PcapSource::Close() + { + if ( ! pd ) + return; + + pcap_close(pd); + pd = 0; + last_data = 0; + + Closed(); + } + +void PcapSource::OpenLive() + { + char errbuf[PCAP_ERRBUF_SIZE]; + char tmp_errbuf[PCAP_ERRBUF_SIZE]; + + // Determine interface if not specified. + if ( props.path.empty() ) + props.path = pcap_lookupdev(tmp_errbuf); + + if ( props.path.empty() ) + { + safe_snprintf(errbuf, sizeof(errbuf), + "pcap_lookupdev: %s", tmp_errbuf); + Error(errbuf); + return; + } + + // Determine network and netmask. + uint32 net; + if ( pcap_lookupnet(props.path.c_str(), &net, &props.netmask, tmp_errbuf) < 0 ) + { + // ### The lookup can fail if no address is assigned to + // the interface; and libpcap doesn't have any useful notion + // of error codes, just error std::strings - how bogus - so we + // just kludge around the error :-(. + // sprintf(errbuf, "pcap_lookupnet %s", tmp_errbuf); + // return; + props.netmask = 0xffffff00; + } + + // We use the smallest time-out possible to return almost immediately if + // no packets are available. (We can't use set_nonblocking() as it's + // broken on FreeBSD: even when select() indicates that we can read + // something, we may get nothing if the store buffer hasn't filled up + // yet.) + pd = pcap_open_live(props.path.c_str(), SnapLen(), 1, 1, tmp_errbuf); + + if ( ! pd ) + { + Error(tmp_errbuf); + return; + } + + // ### This needs autoconf'ing. +#ifdef HAVE_PCAP_INT_H + Info(fmt("pcap bufsize = %d\n", ((struct pcap *) pd)->bufsize)); +#endif + +#ifdef HAVE_LINUX + if ( pcap_setnonblock(pd, 1, tmp_errbuf) < 0 ) + { + PcapError(); + return; + } +#endif + + props.selectable_fd = pcap_fileno(pd); + + SetHdrSize(); + + if ( ! pd ) + // Was closed, couldn't get header size. + return; + + props.is_live = true; + + Opened(props); + } + +void PcapSource::OpenOffline() + { + char errbuf[PCAP_ERRBUF_SIZE]; + + pd = pcap_open_offline(props.path.c_str(), errbuf); + + if ( ! pd ) + { + Error(errbuf); + return; + } + + SetHdrSize(); + + if ( ! pd ) + // Was closed, unknown link layer type. + return; + + props.selectable_fd = fileno(pcap_file(pd)); + + if ( props.selectable_fd < 0 ) + InternalError("OS does not support selectable pcap fd"); + + props.is_live = false; + Opened(props); + } + +bool PcapSource::ExtractNextPacket(Packet* pkt) + { + if ( ! pd ) + return false; + + const u_char* data = pcap_next(pd, ¤t_hdr); + + if ( ! data ) + { + // Source has gone dry. If it's a network interface, this just means + // it's timed out. If it's a file, though, then the file has been + // exhausted. + if ( ! props.is_live ) + Close(); + + return false; + } + + pkt->ts = current_hdr.ts.tv_sec + double(current_hdr.ts.tv_usec) / 1e6; + pkt->hdr = ¤t_hdr; + pkt->data = last_data = data; + + if ( current_hdr.len == 0 || current_hdr.caplen == 0 ) + { + Weird("empty_pcap_header", pkt); + return false; + } + + last_hdr = current_hdr; + last_data = data; + ++stats.received; + return true; + } + +void PcapSource::DoneWithPacket() + { + // Nothing to do. + } + +bool PcapSource::PrecompileFilter(int index, const std::string& filter) + { + return PktSrc::PrecompileBPFFilter(index, filter); + } + +bool PcapSource::SetFilter(int index) + { + if ( ! pd ) + return true; // Prevent error message + + char errbuf[PCAP_ERRBUF_SIZE]; + + BPF_Program* code = GetBPFFilter(index); + + if ( ! code ) + { + safe_snprintf(errbuf, sizeof(errbuf), + "No precompiled pcap filter for index %d", + index); + Error(errbuf); + return false; + } + + if ( pcap_setfilter(pd, code->GetProgram()) < 0 ) + { + PcapError(); + return false; + } + +#ifndef HAVE_LINUX + // Linux doesn't clear counters when resetting filter. + stats.received = stats.dropped = stats.link = 0; +#endif + + return true; + } + +void PcapSource::Statistics(Stats* s) + { + char errbuf[PCAP_ERRBUF_SIZE]; + + if ( ! (props.is_live && pd) ) + s->received = s->dropped = s->link = 0; + + else + { + struct pcap_stat pstat; + if ( pcap_stats(pd, &pstat) < 0 ) + { + PcapError(); + s->received = s->dropped = s->link = 0; + } + + else + { + s->dropped = pstat.ps_drop; + s->link = pstat.ps_recv; + } + } + + s->received = stats.received; + + if ( ! props.is_live ) + s->dropped = 0; + } + +void PcapSource::PcapError() + { + if ( pd ) + Error(fmt("pcap_error: %s", pcap_geterr(pd))); + else + Error("pcap_error: not open"); + + Close(); + } + +void PcapSource::SetHdrSize() + { + if ( ! pd ) + return; + + char errbuf[PCAP_ERRBUF_SIZE]; + + props.link_type = pcap_datalink(pd); + props.hdr_size = GetLinkHeaderSize(props.link_type); + } + +iosource::PktSrc* PcapSource::Instantiate(const std::string& path, bool is_live) + { + return new PcapSource(path, is_live); + } diff --git a/src/iosource/pcap/Source.h b/src/iosource/pcap/Source.h new file mode 100644 index 0000000000..f627e30afa --- /dev/null +++ b/src/iosource/pcap/Source.h @@ -0,0 +1,47 @@ +// See the file "COPYING" in the main distribution directory for copyright. + +#ifndef IOSOURCE_PKTSRC_PCAP_SOURCE_H +#define IOSOURCE_PKTSRC_PCAP_SOURCE_H + +#include "../PktSrc.h" + +namespace iosource { +namespace pcap { + +class PcapSource : public iosource::PktSrc { +public: + PcapSource(const std::string& path, bool is_live); + virtual ~PcapSource(); + + static PktSrc* Instantiate(const std::string& path, bool is_live); + +protected: + // PktSrc interface. + virtual void Open(); + virtual void Close(); + virtual bool ExtractNextPacket(Packet* pkt); + virtual void DoneWithPacket(); + virtual bool PrecompileFilter(int index, const std::string& filter); + virtual bool SetFilter(int index); + virtual void Statistics(Stats* stats); + +private: + void OpenLive(); + void OpenOffline(); + void PcapError(); + void SetHdrSize(); + + Properties props; + Stats stats; + + pcap_t *pd; + + struct pcap_pkthdr current_hdr; + struct pcap_pkthdr last_hdr; + const u_char* last_data; +}; + +} +} + +#endif diff --git a/src/main.cc b/src/main.cc index de11cc1535..63949c5093 100644 --- a/src/main.cc +++ b/src/main.cc @@ -57,6 +57,7 @@ extern "C" void OPENSSL_add_all_algorithms_conf(void); #include "plugin/Manager.h" #include "file_analysis/Manager.h" #include "broxygen/Manager.h" +#include "iosource/Manager.h" #include "binpac_bro.h" @@ -92,6 +93,7 @@ plugin::Manager* plugin_mgr = 0; analyzer::Manager* analyzer_mgr = 0; file_analysis::Manager* file_mgr = 0; broxygen::Manager* broxygen_mgr = 0; +iosource::Manager* iosource_mgr = 0; Stmt* stmts; EventHandlerPtr net_done = 0; RuleMatcher* rule_matcher = 0; @@ -108,7 +110,6 @@ int signal_val = 0; int optimize = 0; int do_notice_analysis = 0; int rule_bench = 0; -SecondaryPath* secondary_path = 0; extern char version[]; char* command_line_policy = 0; vector params; @@ -340,6 +341,10 @@ void terminate_bro() terminating = true; + // File analysis termination may produce events, so do it early on in + // the termination process. + file_mgr->Terminate(); + brofiler.WriteStats(); EventHandlerPtr bro_done = internal_handler("bro_done"); @@ -364,7 +369,6 @@ void terminate_bro() mgr.Drain(); - file_mgr->Terminate(); log_mgr->Terminate(); input_mgr->Terminate(); thread_mgr->Terminate(); @@ -375,20 +379,16 @@ void terminate_bro() delete broxygen_mgr; delete timer_mgr; - delete dns_mgr; delete persistence_serializer; - delete event_player; delete event_serializer; delete state_serializer; delete event_registry; - delete secondary_path; - delete remote_serializer; delete analyzer_mgr; delete file_mgr; delete log_mgr; delete plugin_mgr; - delete thread_mgr; delete reporter; + delete iosource_mgr; reporter = 0; } @@ -448,8 +448,6 @@ int main(int argc, char** argv) name_list interfaces; name_list read_files; - name_list netflows; - name_list flow_files; name_list rule_files; char* bst_file = 0; char* id_name = 0; @@ -551,7 +549,7 @@ int main(int argc, char** argv) opterr = 0; char opts[256]; - safe_strncpy(opts, "B:D:e:f:I:i:K:l:n:p:R:r:s:T:t:U:w:x:X:y:Y:z:CFGLNOPSWabdghvZQ", + safe_strncpy(opts, "B:D:e:f:I:i:K:l:n:p:R:r:s:T:t:U:w:x:X:z:CFGLNOPSWabdghvZQ", sizeof(opts)); #ifdef USE_PERFTOOLS_DEBUG @@ -611,10 +609,6 @@ int main(int argc, char** argv) writefile = optarg; break; - case 'y': - flow_files.append(optarg); - break; - case 'z': if ( streq(optarg, "notice") ) do_notice_analysis = 1; @@ -708,10 +702,6 @@ int main(int argc, char** argv) do_watchdog = 1; break; - case 'Y': - netflows.append(optarg); - break; - case 'h': usage(); break; @@ -799,8 +789,7 @@ int main(int argc, char** argv) // seed the PRNG. We should do this here (but at least Linux, FreeBSD // and Solaris provide /dev/urandom). - if ( (interfaces.length() > 0 || netflows.length() > 0) && - (read_files.length() > 0 || flow_files.length() > 0 )) + if ( interfaces.length() > 0 && read_files.length() > 0 ) usage(); #ifdef USE_IDMEF @@ -823,7 +812,7 @@ int main(int argc, char** argv) plugin_mgr->SearchDynamicPlugins(bro_plugin_path()); if ( optind == argc && - read_files.length() == 0 && flow_files.length() == 0 && + read_files.length() == 0 && interfaces.length() == 0 && ! (id_name || bst_file) && ! command_line_policy && ! print_plugins ) add_input_file("-"); @@ -850,6 +839,7 @@ int main(int argc, char** argv) // policy, but we can't parse policy without DNS resolution. dns_mgr->SetDir(".state"); + iosource_mgr = new iosource::Manager(); persistence_serializer = new PersistenceSerializer(); remote_serializer = new RemoteSerializer(); event_registry = new EventRegistry(); @@ -863,9 +853,17 @@ int main(int argc, char** argv) file_mgr->InitPreScript(); broxygen_mgr->InitPreScript(); + bool missing_plugin = false; + for ( set::const_iterator i = requested_plugins.begin(); i != requested_plugins.end(); i++ ) - plugin_mgr->ActivateDynamicPlugin(*i); + { + if ( ! plugin_mgr->ActivateDynamicPlugin(*i) ) + missing_plugin = true; + } + + if ( missing_plugin ) + reporter->FatalError("Failed to activate requested dynamic plugin(s)."); plugin_mgr->ActivateDynamicPlugins(! bare_mode); @@ -916,6 +914,7 @@ int main(int argc, char** argv) analyzer_mgr->InitPostScript(); file_mgr->InitPostScript(); + dns_mgr->InitPostScript(); if ( parse_only ) { @@ -981,8 +980,7 @@ int main(int argc, char** argv) // ### Add support for debug command file. dbg_init_debugger(0); - if ( (flow_files.length() == 0 || read_files.length() == 0) && - (netflows.length() == 0 || interfaces.length() == 0) ) + if ( read_files.length() == 0 && interfaces.length() == 0 ) { Val* interfaces_val = internal_val("interfaces"); if ( interfaces_val ) @@ -999,13 +997,8 @@ int main(int argc, char** argv) snaplen = internal_val("snaplen")->AsCount(); - // Initialize the secondary path, if it's needed. - secondary_path = new SecondaryPath(); - if ( dns_type != DNS_PRIME ) - net_init(interfaces, read_files, netflows, flow_files, - writefile, "", - secondary_path->Filter(), do_watchdog); + net_init(interfaces, read_files, writefile, do_watchdog); BroFile::SetDefaultRotation(log_rotate_interval, log_max_size); @@ -1164,9 +1157,9 @@ int main(int argc, char** argv) have_pending_timers = ! reading_traces && timer_mgr->Size() > 0; - io_sources.Register(thread_mgr, true); + iosource_mgr->Register(thread_mgr, true); - if ( io_sources.Size() > 0 || + if ( iosource_mgr->Size() > 0 || have_pending_timers || BifConst::exit_only_after_terminate ) { diff --git a/src/plugin/Component.cc b/src/plugin/Component.cc index 006806d1fb..4ace2f96af 100644 --- a/src/plugin/Component.cc +++ b/src/plugin/Component.cc @@ -50,6 +50,18 @@ void Component::Describe(ODesc* d) const d->Add("File Analyzer"); break; + case component::IOSOURCE: + d->Add("I/O Source"); + break; + + case component::PKTSRC: + d->Add("Packet Source"); + break; + + case component::PKTDUMPER: + d->Add("Packet Dumper"); + break; + default: reporter->InternalWarning("unknown component type in plugin::Component::Describe"); d->Add(""); diff --git a/src/plugin/Component.h b/src/plugin/Component.h index 71547393e8..de24a7dbde 100644 --- a/src/plugin/Component.h +++ b/src/plugin/Component.h @@ -12,20 +12,23 @@ namespace plugin { namespace component { /** - * Component types. + * Component types. */ enum Type { READER, /// An input reader (not currently used). WRITER, /// A logging writer (not currenly used). ANALYZER, /// A protocol analyzer. - FILE_ANALYZER /// A file analyzer. + FILE_ANALYZER, /// A file analyzer. + IOSOURCE, /// An I/O source, excluding packet sources. + PKTSRC, /// A packet source. + PKTDUMPER /// A packet dumper. }; } /** * Base class for plugin components. A component is a specific piece of * functionality that a plugin provides, such as a protocol analyzer or a log - * writer. + * writer. */ class Component { diff --git a/src/plugin/Plugin.h b/src/plugin/Plugin.h index 9119d338d5..1269cf9a43 100644 --- a/src/plugin/Plugin.h +++ b/src/plugin/Plugin.h @@ -9,6 +9,7 @@ #include "config.h" #include "analyzer/Component.h" #include "file_analysis/Component.h" +#include "iosource/Component.h" // We allow to override this externally for testing purposes. #ifndef BRO_PLUGIN_API_VERSION diff --git a/src/threading/Manager.cc b/src/threading/Manager.cc index 4491cd42b5..449f2a8ad1 100644 --- a/src/threading/Manager.cc +++ b/src/threading/Manager.cc @@ -11,7 +11,7 @@ Manager::Manager() did_process = true; next_beat = 0; terminating = false; - idle = true; + SetIdle(true); } Manager::~Manager() @@ -47,8 +47,8 @@ void Manager::Terminate() all_threads.clear(); msg_threads.clear(); - idle = true; - closed = true; + SetIdle(true); + SetClosed(true); terminating = false; } @@ -56,7 +56,7 @@ void Manager::AddThread(BasicThread* thread) { DBG_LOG(DBG_THREADING, "Adding thread %s ...", thread->Name()); all_threads.push_back(thread); - idle = false; + SetIdle(false); } void Manager::AddMsgThread(MsgThread* thread) @@ -65,7 +65,8 @@ void Manager::AddMsgThread(MsgThread* thread) msg_threads.push_back(thread); } -void Manager::GetFds(int* read, int* write, int* except) +void Manager::GetFds(iosource::FD_Set* read, iosource::FD_Set* write, + iosource::FD_Set* except) { } diff --git a/src/threading/Manager.h b/src/threading/Manager.h index e839749a91..70e592fa10 100644 --- a/src/threading/Manager.h +++ b/src/threading/Manager.h @@ -4,7 +4,7 @@ #include -#include "IOSource.h" +#include "iosource/IOSource.h" #include "BasicThread.h" #include "MsgThread.h" @@ -21,7 +21,7 @@ namespace threading { * their outgoing message queue on a regular basis and feeds data sent into * the rest of Bro. It also triggers the regular heartbeats. */ -class Manager : public IOSource +class Manager : public iosource::IOSource { public: /** @@ -103,7 +103,8 @@ protected: /** * Part of the IOSource interface. */ - virtual void GetFds(int* read, int* write, int* except); + virtual void GetFds(iosource::FD_Set* read, iosource::FD_Set* write, + iosource::FD_Set* except); /** * Part of the IOSource interface. diff --git a/src/util.cc b/src/util.cc index 15fe198f12..60a92af45f 100644 --- a/src/util.cc +++ b/src/util.cc @@ -43,6 +43,7 @@ #include "NetVar.h" #include "Net.h" #include "Reporter.h" +#include "iosource/Manager.h" /** * Return IP address without enclosing brackets and any leading 0x. @@ -1424,11 +1425,13 @@ double current_time(bool real) double t = double(tv.tv_sec) + double(tv.tv_usec) / 1e6; - if ( ! pseudo_realtime || real || pkt_srcs.length() == 0 ) + const iosource::Manager::PktSrcList& pkt_srcs(iosource_mgr->GetPktSrcs()); + + if ( ! pseudo_realtime || real || pkt_srcs.empty() ) return t; // This obviously only works for a single source ... - PktSrc* src = pkt_srcs[0]; + iosource::PktSrc* src = pkt_srcs.front(); if ( net_is_processing_suspended() ) return src->CurrentPacketTimestamp(); diff --git a/testing/btest/Baseline/core.pcap.dumper/output b/testing/btest/Baseline/core.pcap.dumper/output new file mode 100644 index 0000000000..1055e73ebe --- /dev/null +++ b/testing/btest/Baseline/core.pcap.dumper/output @@ -0,0 +1 @@ +00000010 ff ff 00 00 01 00 00 00 1d a2 b2 4e 73 00 07 00 | | 00000010 00 20 00 00 01 00 00 00 1d a2 b2 4e 73 00 07 00 | diff --git a/testing/btest/Baseline/core.pcap.dynamic-filter/conn.log b/testing/btest/Baseline/core.pcap.dynamic-filter/conn.log new file mode 100644 index 0000000000..f42999c4fa --- /dev/null +++ b/testing/btest/Baseline/core.pcap.dynamic-filter/conn.log @@ -0,0 +1,25 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path conn +#open 2014-08-24-15-51-55 +#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents +#types time string addr port addr port enum string interval count count string bool count string count count count count set[string] +1300475167.096535 CXWv6p3arKYeMETxOg 141.142.220.202 5353 224.0.0.251 5353 udp dns - - - S0 - 0 D 1 73 0 0 (empty) +1300475168.853899 CCvvfg3TEfuqmmG4bh 141.142.220.118 43927 141.142.2.2 53 udp dns 0.000435 38 89 SF - 0 Dd 1 66 1 117 (empty) +1300475168.854378 CsRx2w45OKnoww6xl4 141.142.220.118 37676 141.142.2.2 53 udp dns 0.000420 52 99 SF - 0 Dd 1 80 1 127 (empty) +1300475168.854837 CRJuHdVW0XPVINV8a 141.142.220.118 40526 141.142.2.2 53 udp dns 0.000392 38 183 SF - 0 Dd 1 66 1 211 (empty) +1300475168.857956 CPbrpk1qSsw6ESzHV4 141.142.220.118 32902 141.142.2.2 53 udp dns 0.000317 38 89 SF - 0 Dd 1 66 1 117 (empty) +1300475168.858306 C6pKV8GSxOnSLghOa 141.142.220.118 59816 141.142.2.2 53 udp dns 0.000343 52 99 SF - 0 Dd 1 80 1 127 (empty) +1300475168.858713 CIPOse170MGiRM1Qf4 141.142.220.118 59714 141.142.2.2 53 udp dns 0.000375 38 183 SF - 0 Dd 1 66 1 211 (empty) +1300475168.891644 C7XEbhP654jzLoe3a 141.142.220.118 58206 141.142.2.2 53 udp dns 0.000339 38 89 SF - 0 Dd 1 66 1 117 (empty) +1300475168.892037 CJ3xTn1c4Zw9TmAE05 141.142.220.118 38911 141.142.2.2 53 udp dns 0.000335 52 99 SF - 0 Dd 1 80 1 127 (empty) +1300475168.892414 CMXxB5GvmoxJFXdTa 141.142.220.118 59746 141.142.2.2 53 udp dns 0.000421 38 183 SF - 0 Dd 1 66 1 211 (empty) +1300475168.893988 Caby8b1slFea8xwSmb 141.142.220.118 45000 141.142.2.2 53 udp dns 0.000384 38 89 SF - 0 Dd 1 66 1 117 (empty) +1300475168.894422 Che1bq3i2rO3KD1Syg 141.142.220.118 48479 141.142.2.2 53 udp dns 0.000317 52 99 SF - 0 Dd 1 80 1 127 (empty) +1300475168.894787 C3SfNE4BWaU4aSuwkc 141.142.220.118 48128 141.142.2.2 53 udp dns 0.000423 38 183 SF - 0 Dd 1 66 1 211 (empty) +1300475168.901749 CEle3f3zno26fFZkrh 141.142.220.118 56056 141.142.2.2 53 udp dns 0.000402 36 131 SF - 0 Dd 1 64 1 159 (empty) +1300475168.902195 CwSkQu4eWZCH7OONC1 141.142.220.118 55092 141.142.2.2 53 udp dns 0.000374 36 198 SF - 0 Dd 1 64 1 226 (empty) +1300475168.652003 CjhGID4nQcgTWjvg4c 141.142.220.118 35634 208.80.152.2 80 tcp - - - - OTH - 0 D 1 515 0 0 (empty) +#close 2014-08-24-15-51-55 diff --git a/testing/btest/Baseline/core.pcap.dynamic-filter/output b/testing/btest/Baseline/core.pcap.dynamic-filter/output new file mode 100644 index 0000000000..e8be0b0195 --- /dev/null +++ b/testing/btest/Baseline/core.pcap.dynamic-filter/output @@ -0,0 +1,30 @@ +1, [orig_h=141.142.220.202, orig_p=5353/udp, resp_h=224.0.0.251, resp_p=5353/udp] +2, [orig_h=141.142.220.118, orig_p=35634/tcp, resp_h=208.80.152.2, resp_p=80/tcp] +3, [orig_h=141.142.220.118, orig_p=43927/udp, resp_h=141.142.2.2, resp_p=53/udp] +4, [orig_h=141.142.220.118, orig_p=43927/udp, resp_h=141.142.2.2, resp_p=53/udp] +5, [orig_h=141.142.220.118, orig_p=37676/udp, resp_h=141.142.2.2, resp_p=53/udp] +6, [orig_h=141.142.220.118, orig_p=37676/udp, resp_h=141.142.2.2, resp_p=53/udp] +7, [orig_h=141.142.220.118, orig_p=40526/udp, resp_h=141.142.2.2, resp_p=53/udp] +8, [orig_h=141.142.220.118, orig_p=40526/udp, resp_h=141.142.2.2, resp_p=53/udp] +9, [orig_h=141.142.220.118, orig_p=32902/udp, resp_h=141.142.2.2, resp_p=53/udp] +10, [orig_h=141.142.220.118, orig_p=32902/udp, resp_h=141.142.2.2, resp_p=53/udp] +11, [orig_h=141.142.220.118, orig_p=59816/udp, resp_h=141.142.2.2, resp_p=53/udp] +12, [orig_h=141.142.220.118, orig_p=59816/udp, resp_h=141.142.2.2, resp_p=53/udp] +13, [orig_h=141.142.220.118, orig_p=59714/udp, resp_h=141.142.2.2, resp_p=53/udp] +14, [orig_h=141.142.220.118, orig_p=59714/udp, resp_h=141.142.2.2, resp_p=53/udp] +15, [orig_h=141.142.220.118, orig_p=58206/udp, resp_h=141.142.2.2, resp_p=53/udp] +16, [orig_h=141.142.220.118, orig_p=58206/udp, resp_h=141.142.2.2, resp_p=53/udp] +17, [orig_h=141.142.220.118, orig_p=38911/udp, resp_h=141.142.2.2, resp_p=53/udp] +18, [orig_h=141.142.220.118, orig_p=38911/udp, resp_h=141.142.2.2, resp_p=53/udp] +19, [orig_h=141.142.220.118, orig_p=59746/udp, resp_h=141.142.2.2, resp_p=53/udp] +20, [orig_h=141.142.220.118, orig_p=59746/udp, resp_h=141.142.2.2, resp_p=53/udp] +21, [orig_h=141.142.220.118, orig_p=45000/udp, resp_h=141.142.2.2, resp_p=53/udp] +22, [orig_h=141.142.220.118, orig_p=45000/udp, resp_h=141.142.2.2, resp_p=53/udp] +23, [orig_h=141.142.220.118, orig_p=48479/udp, resp_h=141.142.2.2, resp_p=53/udp] +24, [orig_h=141.142.220.118, orig_p=48479/udp, resp_h=141.142.2.2, resp_p=53/udp] +25, [orig_h=141.142.220.118, orig_p=48128/udp, resp_h=141.142.2.2, resp_p=53/udp] +26, [orig_h=141.142.220.118, orig_p=48128/udp, resp_h=141.142.2.2, resp_p=53/udp] +27, [orig_h=141.142.220.118, orig_p=56056/udp, resp_h=141.142.2.2, resp_p=53/udp] +28, [orig_h=141.142.220.118, orig_p=56056/udp, resp_h=141.142.2.2, resp_p=53/udp] +29, [orig_h=141.142.220.118, orig_p=55092/udp, resp_h=141.142.2.2, resp_p=53/udp] +30, [orig_h=141.142.220.118, orig_p=55092/udp, resp_h=141.142.2.2, resp_p=53/udp] diff --git a/testing/btest/Baseline/core.pcap.filter-error/output b/testing/btest/Baseline/core.pcap.filter-error/output new file mode 100644 index 0000000000..82804bb483 --- /dev/null +++ b/testing/btest/Baseline/core.pcap.filter-error/output @@ -0,0 +1,3 @@ +fatal error in /home/robin/bro/master/scripts/base/frameworks/packet-filter/./main.bro, line 282: Bad pcap filter 'kaputt' +---- +error, cannot compile BPF filter "kaputt, too" diff --git a/testing/btest/Baseline/core.pcap.input-error/output2 b/testing/btest/Baseline/core.pcap.input-error/output2 new file mode 100644 index 0000000000..74666797b9 --- /dev/null +++ b/testing/btest/Baseline/core.pcap.input-error/output2 @@ -0,0 +1,2 @@ +fatal error: problem with interface NO_SUCH_INTERFACE +fatal error: problem with trace file NO_SUCH_TRACE (NO_SUCH_TRACE: No such file or directory) diff --git a/testing/btest/Baseline/core.pcap.pseudo-realtime/output b/testing/btest/Baseline/core.pcap.pseudo-realtime/output new file mode 100644 index 0000000000..d708959ce9 --- /dev/null +++ b/testing/btest/Baseline/core.pcap.pseudo-realtime/output @@ -0,0 +1 @@ +real time matches trace time diff --git a/testing/btest/Baseline/core.pcap.read-trace-with-filter/conn.log b/testing/btest/Baseline/core.pcap.read-trace-with-filter/conn.log new file mode 100644 index 0000000000..8522d69ae6 --- /dev/null +++ b/testing/btest/Baseline/core.pcap.read-trace-with-filter/conn.log @@ -0,0 +1,10 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path conn +#open 2014-08-23-18-29-48 +#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents +#types time string addr port addr port enum string interval count count string bool count string count count count count set[string] +1300475168.892936 CXWv6p3arKYeMETxOg 141.142.220.118 50000 208.80.152.3 80 tcp http 0.229603 1148 734 S1 - 0 ShADad 6 1468 4 950 (empty) +#close 2014-08-23-18-29-48 diff --git a/testing/btest/Baseline/core.pcap.read-trace-with-filter/packet_filter.log b/testing/btest/Baseline/core.pcap.read-trace-with-filter/packet_filter.log new file mode 100644 index 0000000000..75b09c608a --- /dev/null +++ b/testing/btest/Baseline/core.pcap.read-trace-with-filter/packet_filter.log @@ -0,0 +1,10 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path packet_filter +#open 2014-08-23-18-29-48 +#fields ts node filter init success +#types time string string bool bool +1408818588.510297 bro port 50000 T T +#close 2014-08-23-18-29-48 diff --git a/testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log b/testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log index bcd32fa94c..1a8685c86a 100644 --- a/testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log +++ b/testing/btest/Baseline/coverage.bare-load-baseline/canonified_loaded_scripts.log @@ -3,7 +3,7 @@ #empty_field (empty) #unset_field - #path loaded_scripts -#open 2014-08-14-04-31-10 +#open 2014-09-06-01-19-42 #fields name #types string scripts/base/init-bare.bro @@ -43,6 +43,7 @@ scripts/base/init-bare.bro scripts/base/frameworks/files/magic/__load__.bro build/scripts/base/bif/__load__.bro build/scripts/base/bif/broxygen.bif.bro + build/scripts/base/bif/pcap.bif.bro build/scripts/base/bif/bloom-filter.bif.bro build/scripts/base/bif/cardinality-counter.bif.bro build/scripts/base/bif/top-k.bif.bro @@ -113,4 +114,4 @@ scripts/base/init-bare.bro build/scripts/base/bif/plugins/Bro_SQLiteWriter.sqlite.bif.bro scripts/policy/misc/loaded-scripts.bro scripts/base/utils/paths.bro -#close 2014-08-14-04-31-10 +#close 2014-09-06-01-19-42 diff --git a/testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log b/testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log index b100d86ecb..ebcb980eec 100644 --- a/testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log +++ b/testing/btest/Baseline/coverage.default-load-baseline/canonified_loaded_scripts.log @@ -3,7 +3,7 @@ #empty_field (empty) #unset_field - #path loaded_scripts -#open 2014-08-14-04-31-11 +#open 2014-09-06-01-20-32 #fields name #types string scripts/base/init-bare.bro @@ -43,6 +43,7 @@ scripts/base/init-bare.bro scripts/base/frameworks/files/magic/__load__.bro build/scripts/base/bif/__load__.bro build/scripts/base/bif/broxygen.bif.bro + build/scripts/base/bif/pcap.bif.bro build/scripts/base/bif/bloom-filter.bif.bro build/scripts/base/bif/cardinality-counter.bif.bro build/scripts/base/bif/top-k.bif.bro @@ -242,4 +243,4 @@ scripts/base/init-default.bro scripts/base/misc/find-checksum-offloading.bro scripts/base/misc/find-filtered-trace.bro scripts/policy/misc/loaded-scripts.bro -#close 2014-08-14-04-31-11 +#close 2014-09-06-01-20-32 diff --git a/testing/btest/Baseline/plugins.bifs-and-scripts-install/output b/testing/btest/Baseline/plugins.bifs-and-scripts-install/output index a4187d0f7c..f03cfddc81 100644 --- a/testing/btest/Baseline/plugins.bifs-and-scripts-install/output +++ b/testing/btest/Baseline/plugins.bifs-and-scripts-install/output @@ -1,6 +1,6 @@ Demo::Foo - (dynamic, version 1.0) - [Event] plugin_event [Function] hello_plugin_world + [Event] plugin_event plugin: automatically loaded at startup calling bif, Hello from the plugin! diff --git a/testing/btest/Baseline/plugins.bifs-and-scripts/output b/testing/btest/Baseline/plugins.bifs-and-scripts/output index a082b3d690..47dd6ed430 100644 --- a/testing/btest/Baseline/plugins.bifs-and-scripts/output +++ b/testing/btest/Baseline/plugins.bifs-and-scripts/output @@ -1,6 +1,6 @@ Demo::Foo - (dynamic, version 1.0) - [Event] plugin_event [Function] hello_plugin_world + [Event] plugin_event === plugin: automatically loaded at startup diff --git a/testing/btest/Baseline/plugins.hooks/output b/testing/btest/Baseline/plugins.hooks/output index 7b0f9262ae..83341f3075 100644 --- a/testing/btest/Baseline/plugins.hooks/output +++ b/testing/btest/Baseline/plugins.hooks/output @@ -182,7 +182,7 @@ 0.000000 MetaHookPost CallFunction(Log::__create_stream, (Unified2::LOG, [columns=, ev=Unified2::log_unified2])) -> 0.000000 MetaHookPost CallFunction(Log::__create_stream, (Weird::LOG, [columns=, ev=Weird::log_weird])) -> 0.000000 MetaHookPost CallFunction(Log::__create_stream, (X509::LOG, [columns=, ev=X509::log_x509])) -> -0.000000 MetaHookPost CallFunction(Log::__write, (PacketFilter::LOG, [ts=1405981560.501473, node=bro, filter=ip or not ip, init=T, success=T])) -> +0.000000 MetaHookPost CallFunction(Log::__write, (PacketFilter::LOG, [ts=1409853900.737227, node=bro, filter=ip or not ip, init=T, success=T])) -> 0.000000 MetaHookPost CallFunction(Log::add_default_filter, (Cluster::LOG)) -> 0.000000 MetaHookPost CallFunction(Log::add_default_filter, (Communication::LOG)) -> 0.000000 MetaHookPost CallFunction(Log::add_default_filter, (Conn::LOG)) -> @@ -273,8 +273,8 @@ 0.000000 MetaHookPost CallFunction(Log::create_stream, (Unified2::LOG, [columns=, ev=Unified2::log_unified2])) -> 0.000000 MetaHookPost CallFunction(Log::create_stream, (Weird::LOG, [columns=, ev=Weird::log_weird])) -> 0.000000 MetaHookPost CallFunction(Log::create_stream, (X509::LOG, [columns=, ev=X509::log_x509])) -> -0.000000 MetaHookPost CallFunction(Log::default_path_func, (PacketFilter::LOG, , [ts=1405981560.501473, node=bro, filter=ip or not ip, init=T, success=T])) -> -0.000000 MetaHookPost CallFunction(Log::write, (PacketFilter::LOG, [ts=1405981560.501473, node=bro, filter=ip or not ip, init=T, success=T])) -> +0.000000 MetaHookPost CallFunction(Log::default_path_func, (PacketFilter::LOG, , [ts=1409853900.737227, node=bro, filter=ip or not ip, init=T, success=T])) -> +0.000000 MetaHookPost CallFunction(Log::write, (PacketFilter::LOG, [ts=1409853900.737227, node=bro, filter=ip or not ip, init=T, success=T])) -> 0.000000 MetaHookPost CallFunction(Notice::want_pp, ()) -> 0.000000 MetaHookPost CallFunction(PacketFilter::build, ()) -> 0.000000 MetaHookPost CallFunction(PacketFilter::combine_filters, (ip or not ip, and, )) -> @@ -316,7 +316,11 @@ 0.000000 MetaHookPost LoadFile(../main) -> -1 0.000000 MetaHookPost LoadFile(./Bro_ARP.events.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_AYIYA.events.bif.bro) -> -1 +0.000000 MetaHookPost LoadFile(./Bro_AsciiReader.ascii.bif.bro) -> -1 +0.000000 MetaHookPost LoadFile(./Bro_AsciiWriter.ascii.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_BackDoor.events.bif.bro) -> -1 +0.000000 MetaHookPost LoadFile(./Bro_BenchmarkReader.benchmark.bif.bro) -> -1 +0.000000 MetaHookPost LoadFile(./Bro_BinaryReader.binary.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_BitTorrent.events.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_ConnSize.events.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_DCE_RPC.events.bif.bro) -> -1 @@ -347,16 +351,20 @@ 0.000000 MetaHookPost LoadFile(./Bro_NetBIOS.events.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_NetBIOS.functions.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_NetFlow.events.bif.bro) -> -1 +0.000000 MetaHookPost LoadFile(./Bro_NoneWriter.none.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_PIA.events.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_POP3.events.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_RADIUS.events.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_RPC.events.bif.bro) -> -1 +0.000000 MetaHookPost LoadFile(./Bro_RawReader.raw.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_SMB.events.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_SMTP.events.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_SMTP.functions.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_SNMP.events.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_SNMP.types.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_SOCKS.events.bif.bro) -> -1 +0.000000 MetaHookPost LoadFile(./Bro_SQLiteReader.sqlite.bif.bro) -> -1 +0.000000 MetaHookPost LoadFile(./Bro_SQLiteWriter.sqlite.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_SSH.events.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_SSL.events.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./Bro_SteppingStone.events.bif.bro) -> -1 @@ -380,21 +388,20 @@ 0.000000 MetaHookPost LoadFile(./cardinality-counter.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./const.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./consts) -> -1 -0.000000 MetaHookPost LoadFile(./consts.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./consts.bro) -> -1 0.000000 MetaHookPost LoadFile(./contents) -> -1 0.000000 MetaHookPost LoadFile(./dcc-send) -> -1 0.000000 MetaHookPost LoadFile(./entities) -> -1 0.000000 MetaHookPost LoadFile(./event.bif.bro) -> -1 -0.000000 MetaHookPost LoadFile(./events.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./exec) -> -1 0.000000 MetaHookPost LoadFile(./file_analysis.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./files) -> -1 -0.000000 MetaHookPost LoadFile(./functions.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./gridftp) -> -1 0.000000 MetaHookPost LoadFile(./hll_unique) -> -1 +0.000000 MetaHookPost LoadFile(./hooks.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./inactivity) -> -1 0.000000 MetaHookPost LoadFile(./info) -> -1 +0.000000 MetaHookPost LoadFile(./init.bro) -> -1 0.000000 MetaHookPost LoadFile(./input) -> -1 0.000000 MetaHookPost LoadFile(./input.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./last) -> -1 @@ -408,6 +415,7 @@ 0.000000 MetaHookPost LoadFile(./netstats) -> -1 0.000000 MetaHookPost LoadFile(./non-cluster) -> -1 0.000000 MetaHookPost LoadFile(./patterns) -> -1 +0.000000 MetaHookPost LoadFile(./pcap.bif.bro) -> -1 0.000000 MetaHookPost LoadFile(./plugins) -> -1 0.000000 MetaHookPost LoadFile(./polling) -> -1 0.000000 MetaHookPost LoadFile(./postprocessors) -> -1 @@ -432,9 +440,7 @@ 0.000000 MetaHookPost LoadFile(.<...>/ascii) -> -1 0.000000 MetaHookPost LoadFile(.<...>/benchmark) -> -1 0.000000 MetaHookPost LoadFile(.<...>/binary) -> -1 -0.000000 MetaHookPost LoadFile(.<...>/dataseries) -> -1 0.000000 MetaHookPost LoadFile(.<...>/drop) -> -1 -0.000000 MetaHookPost LoadFile(.<...>/elasticsearch) -> -1 0.000000 MetaHookPost LoadFile(.<...>/email_admin) -> -1 0.000000 MetaHookPost LoadFile(.<...>/hostnames) -> -1 0.000000 MetaHookPost LoadFile(.<...>/none) -> -1 @@ -699,7 +705,7 @@ 0.000000 MetaHookPre CallFunction(Log::__create_stream, (Unified2::LOG, [columns=, ev=Unified2::log_unified2])) 0.000000 MetaHookPre CallFunction(Log::__create_stream, (Weird::LOG, [columns=, ev=Weird::log_weird])) 0.000000 MetaHookPre CallFunction(Log::__create_stream, (X509::LOG, [columns=, ev=X509::log_x509])) -0.000000 MetaHookPre CallFunction(Log::__write, (PacketFilter::LOG, [ts=1405981560.501473, node=bro, filter=ip or not ip, init=T, success=T])) +0.000000 MetaHookPre CallFunction(Log::__write, (PacketFilter::LOG, [ts=1409853900.737227, node=bro, filter=ip or not ip, init=T, success=T])) 0.000000 MetaHookPre CallFunction(Log::add_default_filter, (Cluster::LOG)) 0.000000 MetaHookPre CallFunction(Log::add_default_filter, (Communication::LOG)) 0.000000 MetaHookPre CallFunction(Log::add_default_filter, (Conn::LOG)) @@ -790,8 +796,8 @@ 0.000000 MetaHookPre CallFunction(Log::create_stream, (Unified2::LOG, [columns=, ev=Unified2::log_unified2])) 0.000000 MetaHookPre CallFunction(Log::create_stream, (Weird::LOG, [columns=, ev=Weird::log_weird])) 0.000000 MetaHookPre CallFunction(Log::create_stream, (X509::LOG, [columns=, ev=X509::log_x509])) -0.000000 MetaHookPre CallFunction(Log::default_path_func, (PacketFilter::LOG, , [ts=1405981560.501473, node=bro, filter=ip or not ip, init=T, success=T])) -0.000000 MetaHookPre CallFunction(Log::write, (PacketFilter::LOG, [ts=1405981560.501473, node=bro, filter=ip or not ip, init=T, success=T])) +0.000000 MetaHookPre CallFunction(Log::default_path_func, (PacketFilter::LOG, , [ts=1409853900.737227, node=bro, filter=ip or not ip, init=T, success=T])) +0.000000 MetaHookPre CallFunction(Log::write, (PacketFilter::LOG, [ts=1409853900.737227, node=bro, filter=ip or not ip, init=T, success=T])) 0.000000 MetaHookPre CallFunction(Notice::want_pp, ()) 0.000000 MetaHookPre CallFunction(PacketFilter::build, ()) 0.000000 MetaHookPre CallFunction(PacketFilter::combine_filters, (ip or not ip, and, )) @@ -833,7 +839,11 @@ 0.000000 MetaHookPre LoadFile(../main) 0.000000 MetaHookPre LoadFile(./Bro_ARP.events.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_AYIYA.events.bif.bro) +0.000000 MetaHookPre LoadFile(./Bro_AsciiReader.ascii.bif.bro) +0.000000 MetaHookPre LoadFile(./Bro_AsciiWriter.ascii.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_BackDoor.events.bif.bro) +0.000000 MetaHookPre LoadFile(./Bro_BenchmarkReader.benchmark.bif.bro) +0.000000 MetaHookPre LoadFile(./Bro_BinaryReader.binary.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_BitTorrent.events.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_ConnSize.events.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_DCE_RPC.events.bif.bro) @@ -864,16 +874,20 @@ 0.000000 MetaHookPre LoadFile(./Bro_NetBIOS.events.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_NetBIOS.functions.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_NetFlow.events.bif.bro) +0.000000 MetaHookPre LoadFile(./Bro_NoneWriter.none.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_PIA.events.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_POP3.events.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_RADIUS.events.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_RPC.events.bif.bro) +0.000000 MetaHookPre LoadFile(./Bro_RawReader.raw.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_SMB.events.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_SMTP.events.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_SMTP.functions.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_SNMP.events.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_SNMP.types.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_SOCKS.events.bif.bro) +0.000000 MetaHookPre LoadFile(./Bro_SQLiteReader.sqlite.bif.bro) +0.000000 MetaHookPre LoadFile(./Bro_SQLiteWriter.sqlite.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_SSH.events.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_SSL.events.bif.bro) 0.000000 MetaHookPre LoadFile(./Bro_SteppingStone.events.bif.bro) @@ -897,21 +911,20 @@ 0.000000 MetaHookPre LoadFile(./cardinality-counter.bif.bro) 0.000000 MetaHookPre LoadFile(./const.bif.bro) 0.000000 MetaHookPre LoadFile(./consts) -0.000000 MetaHookPre LoadFile(./consts.bif.bro) 0.000000 MetaHookPre LoadFile(./consts.bro) 0.000000 MetaHookPre LoadFile(./contents) 0.000000 MetaHookPre LoadFile(./dcc-send) 0.000000 MetaHookPre LoadFile(./entities) 0.000000 MetaHookPre LoadFile(./event.bif.bro) -0.000000 MetaHookPre LoadFile(./events.bif.bro) 0.000000 MetaHookPre LoadFile(./exec) 0.000000 MetaHookPre LoadFile(./file_analysis.bif.bro) 0.000000 MetaHookPre LoadFile(./files) -0.000000 MetaHookPre LoadFile(./functions.bif.bro) 0.000000 MetaHookPre LoadFile(./gridftp) 0.000000 MetaHookPre LoadFile(./hll_unique) +0.000000 MetaHookPre LoadFile(./hooks.bif.bro) 0.000000 MetaHookPre LoadFile(./inactivity) 0.000000 MetaHookPre LoadFile(./info) +0.000000 MetaHookPre LoadFile(./init.bro) 0.000000 MetaHookPre LoadFile(./input) 0.000000 MetaHookPre LoadFile(./input.bif.bro) 0.000000 MetaHookPre LoadFile(./last) @@ -925,6 +938,7 @@ 0.000000 MetaHookPre LoadFile(./netstats) 0.000000 MetaHookPre LoadFile(./non-cluster) 0.000000 MetaHookPre LoadFile(./patterns) +0.000000 MetaHookPre LoadFile(./pcap.bif.bro) 0.000000 MetaHookPre LoadFile(./plugins) 0.000000 MetaHookPre LoadFile(./polling) 0.000000 MetaHookPre LoadFile(./postprocessors) @@ -949,9 +963,7 @@ 0.000000 MetaHookPre LoadFile(.<...>/ascii) 0.000000 MetaHookPre LoadFile(.<...>/benchmark) 0.000000 MetaHookPre LoadFile(.<...>/binary) -0.000000 MetaHookPre LoadFile(.<...>/dataseries) 0.000000 MetaHookPre LoadFile(.<...>/drop) -0.000000 MetaHookPre LoadFile(.<...>/elasticsearch) 0.000000 MetaHookPre LoadFile(.<...>/email_admin) 0.000000 MetaHookPre LoadFile(.<...>/hostnames) 0.000000 MetaHookPre LoadFile(.<...>/none) @@ -1216,7 +1228,7 @@ 0.000000 | HookCallFunction Log::__create_stream(Unified2::LOG, [columns=, ev=Unified2::log_unified2]) 0.000000 | HookCallFunction Log::__create_stream(Weird::LOG, [columns=, ev=Weird::log_weird]) 0.000000 | HookCallFunction Log::__create_stream(X509::LOG, [columns=, ev=X509::log_x509]) -0.000000 | HookCallFunction Log::__write(PacketFilter::LOG, [ts=1405981560.501473, node=bro, filter=ip or not ip, init=T, success=T]) +0.000000 | HookCallFunction Log::__write(PacketFilter::LOG, [ts=1409853900.737227, node=bro, filter=ip or not ip, init=T, success=T]) 0.000000 | HookCallFunction Log::add_default_filter(Cluster::LOG) 0.000000 | HookCallFunction Log::add_default_filter(Communication::LOG) 0.000000 | HookCallFunction Log::add_default_filter(Conn::LOG) @@ -1307,8 +1319,8 @@ 0.000000 | HookCallFunction Log::create_stream(Unified2::LOG, [columns=, ev=Unified2::log_unified2]) 0.000000 | HookCallFunction Log::create_stream(Weird::LOG, [columns=, ev=Weird::log_weird]) 0.000000 | HookCallFunction Log::create_stream(X509::LOG, [columns=, ev=X509::log_x509]) -0.000000 | HookCallFunction Log::default_path_func(PacketFilter::LOG, , [ts=1405981560.501473, node=bro, filter=ip or not ip, init=T, success=T]) -0.000000 | HookCallFunction Log::write(PacketFilter::LOG, [ts=1405981560.501473, node=bro, filter=ip or not ip, init=T, success=T]) +0.000000 | HookCallFunction Log::default_path_func(PacketFilter::LOG, , [ts=1409853900.737227, node=bro, filter=ip or not ip, init=T, success=T]) +0.000000 | HookCallFunction Log::write(PacketFilter::LOG, [ts=1409853900.737227, node=bro, filter=ip or not ip, init=T, success=T]) 0.000000 | HookCallFunction Notice::want_pp() 0.000000 | HookCallFunction PacketFilter::build() 0.000000 | HookCallFunction PacketFilter::combine_filters(ip or not ip, and, ) diff --git a/testing/btest/Baseline/plugins.pktdumper/output b/testing/btest/Baseline/plugins.pktdumper/output new file mode 100644 index 0000000000..05ffec25cb --- /dev/null +++ b/testing/btest/Baseline/plugins.pktdumper/output @@ -0,0 +1,12 @@ +Demo::Foo - A Foo packet dumper (dynamic, version 1.0) + [Packet Dumper] FooPktDumper (dumper prefix: foo) + +=== +Dumping to XXX: 1373858797.646968 len 94 +Dumping to XXX: 1373858797.646998 len 94 +Dumping to XXX: 1373858797.647041 len 86 +Dumping to XXX: 1373858797.647147 len 98 +Dumping to XXX: 1373858797.647186 len 86 +Dumping to XXX: 1373858797.647250 len 86 +Dumping to XXX: 1373858797.647317 len 86 +Dumping to XXX: 1373858797.647350 len 86 diff --git a/testing/btest/Baseline/plugins.pktsrc/conn.log b/testing/btest/Baseline/plugins.pktsrc/conn.log new file mode 100644 index 0000000000..ab218f18fd --- /dev/null +++ b/testing/btest/Baseline/plugins.pktsrc/conn.log @@ -0,0 +1,10 @@ +#separator \x09 +#set_separator , +#empty_field (empty) +#unset_field - +#path conn +#open 2014-09-04-18-06-05 +#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p proto service duration orig_bytes resp_bytes conn_state local_orig missed_bytes history orig_pkts orig_ip_bytes resp_pkts resp_ip_bytes tunnel_parents +#types time string addr port addr port enum string interval count count string bool count string count count count count set[string] +1409193037.000000 CXWv6p3arKYeMETxOg 1.2.0.2 2527 1.2.0.3 6649 tcp - - - - S0 - 0 S 1 64 0 0 (empty) +#close 2014-09-04-18-06-05 diff --git a/testing/btest/Baseline/plugins.writer/output b/testing/btest/Baseline/plugins.writer/output index f737e892a0..49c130d6e2 100644 --- a/testing/btest/Baseline/plugins.writer/output +++ b/testing/btest/Baseline/plugins.writer/output @@ -2,21 +2,21 @@ Demo::Foo - A Foo test logging writer (dynamic, version 1.0) [Writer] Foo (Log::WRITER_FOO) === -[packet_filter] 1406831942.605829|bro|ip or not ip|T|T [conn] 1340213005.165293|CXWv6p3arKYeMETxOg|10.0.0.55|53994|60.190.189.214|8124|tcp|-|4.314406|0|0|S0|-|0|S|5|320|0|0| -[tunnel] 1340213015.276495|-|10.0.0.55|0|60.190.189.214|8124|Tunnel::SOCKS|Tunnel::DISCOVER -[socks] 1340213015.276495|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|5|-|succeeded|-|www.osnews.com|80|192.168.0.31|-|2688 -[http] 1340213019.013158|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|1|GET|www.osnews.com|/images/printer2.gif|http://www.osnews.com/|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20100101 Firefox/10.0.2|0|0|304|Not Modified|-|-|-||-|-|-|-|-|-|- -[http] 1340213019.013426|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|2|GET|www.osnews.com|/img2/shorturl.jpg|http://www.osnews.com/|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20100101 Firefox/10.0.2|0|0|304|Not Modified|-|-|-||-|-|-|-|-|-|- -[http] 1340213019.580162|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|3|GET|www.osnews.com|/images/icons/9.gif|http://www.osnews.com/|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20100101 Firefox/10.0.2|0|0|304|Not Modified|-|-|-||-|-|-|-|-|-|- -[http] 1340213020.155861|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|4|GET|www.osnews.com|/images/icons/26.gif|http://www.osnews.com/|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20100101 Firefox/10.0.2|0|1368|200|OK|-|-|-||-|-|-|-|-|FBtZ7y1ppK8iIeY622|image/gif -[files] 1340213020.732581|FBtZ7y1ppK8iIeY622|60.190.189.214|10.0.0.55|CjhGID4nQcgTWjvg4c|HTTP|0||image/gif|-|0.000000|-|F|1368|1368|0|0|F|-|-|-|-|- -[http] 1340213020.732963|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|5|GET|www.osnews.com|/images/icons/17.gif|http://www.osnews.com/|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20100101 Firefox/10.0.2|0|0|304|Not Modified|-|-|-||-|-|-|-|-|-|- -[http] 1340213021.300269|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|6|GET|www.osnews.com|/images/left.gif|http://www.osnews.com/|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20100101 Firefox/10.0.2|0|0|304|Not Modified|-|-|-||-|-|-|-|-|-|- [conn] 1340213010.582723|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|tcp|http,socks|13.839419|3860|2934|SF|-|0|ShADadfF|23|5080|20|3986| -[http] 1340213021.861584|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|7|GET|www.osnews.com|/images/icons/32.gif|http://www.osnews.com/|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20100101 Firefox/10.0.2|0|0|304|Not Modified|-|-|-||-|-|-|-|-|-|- [conn] 1340213048.780152|CCvvfg3TEfuqmmG4bh|10.0.0.55|53994|60.190.189.214|8124|tcp|-|-|-|-|SH|-|0|F|1|52|0|0| [conn] 1340213097.272764|CsRx2w45OKnoww6xl4|10.0.0.55|53994|60.190.189.214|8124|tcp|-|-|-|-|SH|-|0|F|1|52|0|0| [conn] 1340213162.160367|CRJuHdVW0XPVINV8a|10.0.0.55|53994|60.190.189.214|8124|tcp|-|-|-|-|SH|-|0|F|1|52|0|0| [conn] 1340213226.561757|CPbrpk1qSsw6ESzHV4|10.0.0.55|53994|60.190.189.214|8124|tcp|-|-|-|-|SH|-|0|F|1|52|0|0| [conn] 1340213290.981995|C6pKV8GSxOnSLghOa|10.0.0.55|53994|60.190.189.214|8124|tcp|-|-|-|-|SH|-|0|F|1|52|0|0| +[files] 1340213020.732581|FBtZ7y1ppK8iIeY622|60.190.189.214|10.0.0.55|CjhGID4nQcgTWjvg4c|HTTP|0||image/gif|-|0.000000|-|F|1368|1368|0|0|F|-|-|-|-|- +[http] 1340213019.013158|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|1|GET|www.osnews.com|/images/printer2.gif|http://www.osnews.com/|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20100101 Firefox/10.0.2|0|0|304|Not Modified|-|-|-||-|-|-|-|-|-|- +[http] 1340213019.013426|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|2|GET|www.osnews.com|/img2/shorturl.jpg|http://www.osnews.com/|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20100101 Firefox/10.0.2|0|0|304|Not Modified|-|-|-||-|-|-|-|-|-|- +[http] 1340213019.580162|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|3|GET|www.osnews.com|/images/icons/9.gif|http://www.osnews.com/|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20100101 Firefox/10.0.2|0|0|304|Not Modified|-|-|-||-|-|-|-|-|-|- +[http] 1340213020.155861|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|4|GET|www.osnews.com|/images/icons/26.gif|http://www.osnews.com/|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20100101 Firefox/10.0.2|0|1368|200|OK|-|-|-||-|-|-|-|-|FBtZ7y1ppK8iIeY622|image/gif +[http] 1340213020.732963|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|5|GET|www.osnews.com|/images/icons/17.gif|http://www.osnews.com/|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20100101 Firefox/10.0.2|0|0|304|Not Modified|-|-|-||-|-|-|-|-|-|- +[http] 1340213021.300269|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|6|GET|www.osnews.com|/images/left.gif|http://www.osnews.com/|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20100101 Firefox/10.0.2|0|0|304|Not Modified|-|-|-||-|-|-|-|-|-|- +[http] 1340213021.861584|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|7|GET|www.osnews.com|/images/icons/32.gif|http://www.osnews.com/|Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20100101 Firefox/10.0.2|0|0|304|Not Modified|-|-|-||-|-|-|-|-|-|- +[packet_filter] 1409859343.786281|bro|ip or not ip|T|T +[socks] 1340213015.276495|CjhGID4nQcgTWjvg4c|10.0.0.55|53994|60.190.189.214|8124|5|-|succeeded|-|www.osnews.com|80|192.168.0.31|-|2688 +[tunnel] 1340213015.276495|-|10.0.0.55|0|60.190.189.214|8124|Tunnel::SOCKS|Tunnel::DISCOVER diff --git a/testing/btest/Baseline/scripts.base.frameworks.file-analysis.actions.data_event/out b/testing/btest/Baseline/scripts.base.frameworks.file-analysis.actions.data_event/out index cbd60840bf..d1cc77944c 100644 --- a/testing/btest/Baseline/scripts.base.frameworks.file-analysis.actions.data_event/out +++ b/testing/btest/Baseline/scripts.base.frameworks.file-analysis.actions.data_event/out @@ -5,18 +5,14 @@ FILE_BOF_BUFFER MIME_TYPE text/plain FILE_OVER_NEW_CONNECTION -file_stream, file #0, 1500, ^J0.26 | 2012-08-24 15:10:04 -0700^J^J * Fixing update-changes, which could pick the wrong control file. (Robin Sommer)^J^J * Fixing GPG signing script. (Robin Sommer)^J^J0.25 | 2012-08-01 13:55:46 -0500^J^J * Fix configure script to exit with non-zero status on error (Jon Siwek)^J^J0.24 | 2012-07-05 12:50:43 -0700^J^J * Raise minimum required CMake version to 2.6.3 (Jon Siwek)^J^J * Adding script to delete old fully-merged branches. (Robin Sommer)^J^J0.23-2 | 2012-01-25 13:24:01 -0800^J^J * Fix a bro-cut error message. (Daniel Thayer)^J^J0.23 | 2012-01-11 12:16:11 -0800^J^J * Tweaks to release scripts, plus a new one for signing files.^J (Robin Sommer)^J^J0.22 | 2012-01-10 16:45:19 -0800^J^J * Tweaks for OpenBSD support. (Jon Siwek)^J^J * bro-cut extensions and fixes. (Robin Sommer)^J ^J - If no field names are given on the command line, we now pass through^J all fields. Adresses #657.^J^J - Removing some GNUism from awk script. Addresses #653.^J^J - Added option for time output in UTC. Addresses #668.^J^J - Added output field separator option -F. Addresses #649.^J^J - Fixing option -c: only some header lines were passed through^J rather than all. (Robin Sommer)^J^J * Fix parallel make portability. (Jon Siwek)^J^J0.21-9 | 2011-11-07 05:44:14 -0800^J^J * Fixing compiler warnings. Addresses #388. (Jon Siwek)^J^J0.21-2 | 2011-11-02 18:12:13 -0700^J^J * Fix for misnaming temp file in update-changes script. (Robin Sommer)^J^J0.21-1 | 2011-11-02 18:10:39 -0700^J^J * Little fix for make-relea -file_chunk, file #0, 1500, 0, ^J0.26 | 2012-08-24 15:10:04 -0700^J^J * Fixing update-changes, which could pick the wrong control file. (Robin Sommer)^J^J * Fixing GPG signing script. (Robin Sommer)^J^J0.25 | 2012-08-01 13:55:46 -0500^J^J * Fix configure script to exit with non-zero status on error (Jon Siwek)^J^J0.24 | 2012-07-05 12:50:43 -0700^J^J * Raise minimum required CMake version to 2.6.3 (Jon Siwek)^J^J * Adding script to delete old fully-merged branches. (Robin Sommer)^J^J0.23-2 | 2012-01-25 13:24:01 -0800^J^J * Fix a bro-cut error message. (Daniel Thayer)^J^J0.23 | 2012-01-11 12:16:11 -0800^J^J * Tweaks to release scripts, plus a new one for signing files.^J (Robin Sommer)^J^J0.22 | 2012-01-10 16:45:19 -0800^J^J * Tweaks for OpenBSD support. (Jon Siwek)^J^J * bro-cut extensions and fixes. (Robin Sommer)^J ^J - If no field names are given on the command line, we now pass through^J all fields. Adresses #657.^J^J - Removing some GNUism from awk script. Addresses #653.^J^J - Added option for time output in UTC. Addresses #668.^J^J - Added output field separator option -F. Addresses #649.^J^J - Fixing option -c: only some header lines were passed through^J rather than all. (Robin Sommer)^J^J * Fix parallel make portability. (Jon Siwek)^J^J0.21-9 | 2011-11-07 05:44:14 -0800^J^J * Fixing compiler warnings. Addresses #388. (Jon Siwek)^J^J0.21-2 | 2011-11-02 18:12:13 -0700^J^J * Fix for misnaming temp file in update-changes script. (Robin Sommer)^J^J0.21-1 | 2011-11-02 18:10:39 -0700^J^J * Little fix for make-relea -file_stream, file #0, 1024, se script, which could pick out the wrong^J tag. (Robin Sommer)^J^J0.21 | 2011-10-27 17:40:45 -0700^J^J * Fixing bro-cut's usage message and argument error handling. (Robin Sommer)^J^J * Bugfix in update-changes script. (Robin Sommer)^J^J * update-changes now ignores commits it did itself. (Robin Sommer)^J^J * Fix a bug in the update-changes script. (Robin Sommer)^J^J * bro-cut now always installs to $prefix/bin by `make install`. (Jon Siwek)^J^J * Options to adjust time format for bro-cut. (Robin Sommer)^J^J The default with -d is now ISO format. The new option "-D "^J specifies a custom strftime()-style format string. Alternatively,^J the environment variable BRO_CUT_TIMEFMT can set the format as^J well.^J^J * bro-cut now understands the field separator header. (Robin Sommer)^J^J * Renaming options -h/-H -> -c/-C, and doing some general cleanup.^J^J0.2 | 2011-10-25 19:53:57 -0700^J^J * Adding support for replacing version string in a setup.py. (Robin^J Sommer)^J^J * Change generated root cert DN indices f -file_chunk, file #0, 1024, 1500, se script, which could pick out the wrong^J tag. (Robin Sommer)^J^J0.21 | 2011-10-27 17:40:45 -0700^J^J * Fixing bro-cut's usage message and argument error handling. (Robin Sommer)^J^J * Bugfix in update-changes script. (Robin Sommer)^J^J * update-changes now ignores commits it did itself. (Robin Sommer)^J^J * Fix a bug in the update-changes script. (Robin Sommer)^J^J * bro-cut now always installs to $prefix/bin by `make install`. (Jon Siwek)^J^J * Options to adjust time format for bro-cut. (Robin Sommer)^J^J The default with -d is now ISO format. The new option "-D "^J specifies a custom strftime()-style format string. Alternatively,^J the environment variable BRO_CUT_TIMEFMT can set the format as^J well.^J^J * bro-cut now understands the field separator header. (Robin Sommer)^J^J * Renaming options -h/-H -> -c/-C, and doing some general cleanup.^J^J0.2 | 2011-10-25 19:53:57 -0700^J^J * Adding support for replacing version string in a setup.py. (Robin^J Sommer)^J^J * Change generated root cert DN indices f -file_stream, file #0, 476, ormat for RFC2253^J compliance. (Jon Siwek)^J^J * New tool devel-tools/check-release to run before making releases.^J (Robin Sommer)^J^J * devel-tools/update-changes gets a new option -a to amend to^J previous commit if possible. Default is now not to (used to be the^J opposite). (Robin Sommer)^J^J * Change Mozilla trust root generation to index certs by subject DN. (Jon Siwek)^J^J * Change distclean to only remove build dir. (Jon Siwek)^J^J * Make dist now cleans the -file_chunk, file #0, 476, 2524, ormat for RFC2253^J compliance. (Jon Siwek)^J^J * New tool devel-tools/check-release to run before making releases.^J (Robin Sommer)^J^J * devel-tools/update-changes gets a new option -a to amend to^J previous commit if possible. Default is now not to (used to be the^J opposite). (Robin Sommer)^J^J * Change Mozilla trust root generation to index certs by subject DN. (Jon Siwek)^J^J * Change distclean to only remove build dir. (Jon Siwek)^J^J * Make dist now cleans the -file_stream, file #0, 1024, copied source (Jon Siwek)^J^J * Small tweak to make-release for forced git-clean. (Jon Siwek)^J^J * Fix to not let updates scripts loose their executable permissions.^J (Robin Sommer)^J^J * devel-tools/update-changes now looks for a 'release' tag to^J idenfify the stable version, and 'beta' for the beta versions.^J (Robin Sommer).^J^J * Distribution cleanup. (Robin Sommer)^J^J * New script devel-tools/make-release to create source tar balls.^J (Robin Sommer)^J^J * Removing bdcat. With the new log format, this isn't very useful^J anymore. (Robin Sommer)^J^J * Adding script that shows all pending git fastpath commits. (Robin^J Sommer)^J^J * Script to measure CPU time by loading an increasing set of^J scripts. (Robin Sommer)^J^J * extract-conn script now deals wit *.gz files. (Robin Sommer)^J^J * Tiny update to output a valid CA list file for SSL cert^J validation. (Seth Hall)^J^J * Adding "install-aux" target. Addresses #622. (Jon Siwek)^J^J * Distribution cleanup. (Jon Siwek and Robin Sommer)^J^J * FindPCAP -file_chunk, file #0, 1024, 3000, copied source (Jon Siwek)^J^J * Small tweak to make-release for forced git-clean. (Jon Siwek)^J^J * Fix to not let updates scripts loose their executable permissions.^J (Robin Sommer)^J^J * devel-tools/update-changes now looks for a 'release' tag to^J idenfify the stable version, and 'beta' for the beta versions.^J (Robin Sommer).^J^J * Distribution cleanup. (Robin Sommer)^J^J * New script devel-tools/make-release to create source tar balls.^J (Robin Sommer)^J^J * Removing bdcat. With the new log format, this isn't very useful^J anymore. (Robin Sommer)^J^J * Adding script that shows all pending git fastpath commits. (Robin^J Sommer)^J^J * Script to measure CPU time by loading an increasing set of^J scripts. (Robin Sommer)^J^J * extract-conn script now deals wit *.gz files. (Robin Sommer)^J^J * Tiny update to output a valid CA list file for SSL cert^J validation. (Seth Hall)^J^J * Adding "install-aux" target. Addresses #622. (Jon Siwek)^J^J * Distribution cleanup. (Jon Siwek and Robin Sommer)^J^J * FindPCAP -file_stream, file #0, 476, now links against thread library when necessary (e.g.^J PF_RING's libpcap) (Jon Siwek)^J^J * Install binaries with an RPATH (Jon Siwek)^J^J * Workaround for FreeBSD CMake port missing debug flags (Jon Siwek)^J^J * Rewrite of the update-changes script. (Robin Sommer)^J^J0.1-1 | 2011-06-14 21:12:41 -0700^J^J * Add a script for generating Mozilla's CA list for the SSL analyzer.^J (Seth Hall)^J^J0.1 | 2011-04-01 16:28:22 -0700^J^J * Converting build process to CMake. (Jon Siwek)^J -file_chunk, file #0, 476, 4024, now links against thread library when necessary (e.g.^J PF_RING's libpcap) (Jon Siwek)^J^J * Install binaries with an RPATH (Jon Siwek)^J^J * Workaround for FreeBSD CMake port missing debug flags (Jon Siwek)^J^J * Rewrite of the update-changes script. (Robin Sommer)^J^J0.1-1 | 2011-06-14 21:12:41 -0700^J^J * Add a script for generating Mozilla's CA list for the SSL analyzer.^J (Seth Hall)^J^J0.1 | 2011-04-01 16:28:22 -0700^J^J * Converting build process to CMake. (Jon Siwek)^J -file_stream, file #0, 205, ^J * Removing cf/hf/ca-* from distribution. The README has a note where^J to find them now. (Robin Sommer)^J^J * General cleanup. (Robin Sommer)^J^J * Initial import of bro/aux from SVN r7088. (Jon Siwek)^J -file_chunk, file #0, 205, 4500, ^J * Removing cf/hf/ca-* from distribution. The README has a note where^J to find them now. (Robin Sommer)^J^J * General cleanup. (Robin Sommer)^J^J * Initial import of bro/aux from SVN r7088. (Jon Siwek)^J +file_stream, file #0, 1146, ^J0.26 | 2012-08-24 15:10:04 -0700^J^J * Fixing update-changes, which could pick the wrong control file. (Robin Sommer)^J^J * Fixing GPG signing script. (Robin Sommer)^J^J0.25 | 2012-08-01 13:55:46 -0500^J^J * Fix configure script to exit with non-zero status on error (Jon Siwek)^J^J0.24 | 2012-07-05 12:50:43 -0700^J^J * Raise minimum required CMake version to 2.6.3 (Jon Siwek)^J^J * Adding script to delete old fully-merged branches. (Robin Sommer)^J^J0.23-2 | 2012-01-25 13:24:01 -0800^J^J * Fix a bro-cut error message. (Daniel Thayer)^J^J0.23 | 2012-01-11 12:16:11 -0800^J^J * Tweaks to release scripts, plus a new one for signing files.^J (Robin Sommer)^J^J0.22 | 2012-01-10 16:45:19 -0800^J^J * Tweaks for OpenBSD support. (Jon Siwek)^J^J * bro-cut extensions and fixes. (Robin Sommer)^J ^J - If no field names are given on the command line, we now pass through^J all fields. Adresses #657.^J^J - Removing some GNUism from awk script. Addresses #653.^J^J - Added option for time output in UTC. Addresses #668.^J^J - Added output field separator option -F. Addresses #649.^J^J - Fixing option -c: only some header lines were passed through^J +file_chunk, file #0, 1146, 0, ^J0.26 | 2012-08-24 15:10:04 -0700^J^J * Fixing update-changes, which could pick the wrong control file. (Robin Sommer)^J^J * Fixing GPG signing script. (Robin Sommer)^J^J0.25 | 2012-08-01 13:55:46 -0500^J^J * Fix configure script to exit with non-zero status on error (Jon Siwek)^J^J0.24 | 2012-07-05 12:50:43 -0700^J^J * Raise minimum required CMake version to 2.6.3 (Jon Siwek)^J^J * Adding script to delete old fully-merged branches. (Robin Sommer)^J^J0.23-2 | 2012-01-25 13:24:01 -0800^J^J * Fix a bro-cut error message. (Daniel Thayer)^J^J0.23 | 2012-01-11 12:16:11 -0800^J^J * Tweaks to release scripts, plus a new one for signing files.^J (Robin Sommer)^J^J0.22 | 2012-01-10 16:45:19 -0800^J^J * Tweaks for OpenBSD support. (Jon Siwek)^J^J * bro-cut extensions and fixes. (Robin Sommer)^J ^J - If no field names are given on the command line, we now pass through^J all fields. Adresses #657.^J^J - Removing some GNUism from awk script. Addresses #653.^J^J - Added option for time output in UTC. Addresses #668.^J^J - Added output field separator option -F. Addresses #649.^J^J - Fixing option -c: only some header lines were passed through^J +file_stream, file #0, 1448, rather than all. (Robin Sommer)^J^J * Fix parallel make portability. (Jon Siwek)^J^J0.21-9 | 2011-11-07 05:44:14 -0800^J^J * Fixing compiler warnings. Addresses #388. (Jon Siwek)^J^J0.21-2 | 2011-11-02 18:12:13 -0700^J^J * Fix for misnaming temp file in update-changes script. (Robin Sommer)^J^J0.21-1 | 2011-11-02 18:10:39 -0700^J^J * Little fix for make-release script, which could pick out the wrong^J tag. (Robin Sommer)^J^J0.21 | 2011-10-27 17:40:45 -0700^J^J * Fixing bro-cut's usage message and argument error handling. (Robin Sommer)^J^J * Bugfix in update-changes script. (Robin Sommer)^J^J * update-changes now ignores commits it did itself. (Robin Sommer)^J^J * Fix a bug in the update-changes script. (Robin Sommer)^J^J * bro-cut now always installs to $prefix/bin by `make install`. (Jon Siwek)^J^J * Options to adjust time format for bro-cut. (Robin Sommer)^J^J The default with -d is now ISO format. The new option "-D "^J specifies a custom strftime()-style format string. Alternatively,^J the environment variable BRO_CUT_TIMEFMT can set the format as^J well.^J^J * bro-cut now understands the field separator header. (Robin Sommer)^J^J * Renaming options -h/-H -> -c/-C, and doing some general cleanup.^J^J0.2 | 2011-10-25 19:53:57 -0700^J^J * Adding support for replacing version string in a setup.py. (Robin^J Sommer)^J^J * Change generated root cert DN indices format for RFC2253^J compliance. (Jon Siwek)^J^J * New tool devel-tool +file_chunk, file #0, 1448, 1146, rather than all. (Robin Sommer)^J^J * Fix parallel make portability. (Jon Siwek)^J^J0.21-9 | 2011-11-07 05:44:14 -0800^J^J * Fixing compiler warnings. Addresses #388. (Jon Siwek)^J^J0.21-2 | 2011-11-02 18:12:13 -0700^J^J * Fix for misnaming temp file in update-changes script. (Robin Sommer)^J^J0.21-1 | 2011-11-02 18:10:39 -0700^J^J * Little fix for make-release script, which could pick out the wrong^J tag. (Robin Sommer)^J^J0.21 | 2011-10-27 17:40:45 -0700^J^J * Fixing bro-cut's usage message and argument error handling. (Robin Sommer)^J^J * Bugfix in update-changes script. (Robin Sommer)^J^J * update-changes now ignores commits it did itself. (Robin Sommer)^J^J * Fix a bug in the update-changes script. (Robin Sommer)^J^J * bro-cut now always installs to $prefix/bin by `make install`. (Jon Siwek)^J^J * Options to adjust time format for bro-cut. (Robin Sommer)^J^J The default with -d is now ISO format. The new option "-D "^J specifies a custom strftime()-style format string. Alternatively,^J the environment variable BRO_CUT_TIMEFMT can set the format as^J well.^J^J * bro-cut now understands the field separator header. (Robin Sommer)^J^J * Renaming options -h/-H -> -c/-C, and doing some general cleanup.^J^J0.2 | 2011-10-25 19:53:57 -0700^J^J * Adding support for replacing version string in a setup.py. (Robin^J Sommer)^J^J * Change generated root cert DN indices format for RFC2253^J compliance. (Jon Siwek)^J^J * New tool devel-tool +file_stream, file #0, 1448, s/check-release to run before making releases.^J (Robin Sommer)^J^J * devel-tools/update-changes gets a new option -a to amend to^J previous commit if possible. Default is now not to (used to be the^J opposite). (Robin Sommer)^J^J * Change Mozilla trust root generation to index certs by subject DN. (Jon Siwek)^J^J * Change distclean to only remove build dir. (Jon Siwek)^J^J * Make dist now cleans the copied source (Jon Siwek)^J^J * Small tweak to make-release for forced git-clean. (Jon Siwek)^J^J * Fix to not let updates scripts loose their executable permissions.^J (Robin Sommer)^J^J * devel-tools/update-changes now looks for a 'release' tag to^J idenfify the stable version, and 'beta' for the beta versions.^J (Robin Sommer).^J^J * Distribution cleanup. (Robin Sommer)^J^J * New script devel-tools/make-release to create source tar balls.^J (Robin Sommer)^J^J * Removing bdcat. With the new log format, this isn't very useful^J anymore. (Robin Sommer)^J^J * Adding script that shows all pending git fastpath commits. (Robin^J Sommer)^J^J * Script to measure CPU time by loading an increasing set of^J scripts. (Robin Sommer)^J^J * extract-conn script now deals wit *.gz files. (Robin Sommer)^J^J * Tiny update to output a valid CA list file for SSL cert^J validation. (Seth Hall)^J^J * Adding "install-aux" target. Addresses #622. (Jon Siwek)^J^J * Distribution cleanup. (Jon Siwek and Robin Sommer)^J^J * FindPCAP now links against +file_chunk, file #0, 1448, 2594, s/check-release to run before making releases.^J (Robin Sommer)^J^J * devel-tools/update-changes gets a new option -a to amend to^J previous commit if possible. Default is now not to (used to be the^J opposite). (Robin Sommer)^J^J * Change Mozilla trust root generation to index certs by subject DN. (Jon Siwek)^J^J * Change distclean to only remove build dir. (Jon Siwek)^J^J * Make dist now cleans the copied source (Jon Siwek)^J^J * Small tweak to make-release for forced git-clean. (Jon Siwek)^J^J * Fix to not let updates scripts loose their executable permissions.^J (Robin Sommer)^J^J * devel-tools/update-changes now looks for a 'release' tag to^J idenfify the stable version, and 'beta' for the beta versions.^J (Robin Sommer).^J^J * Distribution cleanup. (Robin Sommer)^J^J * New script devel-tools/make-release to create source tar balls.^J (Robin Sommer)^J^J * Removing bdcat. With the new log format, this isn't very useful^J anymore. (Robin Sommer)^J^J * Adding script that shows all pending git fastpath commits. (Robin^J Sommer)^J^J * Script to measure CPU time by loading an increasing set of^J scripts. (Robin Sommer)^J^J * extract-conn script now deals wit *.gz files. (Robin Sommer)^J^J * Tiny update to output a valid CA list file for SSL cert^J validation. (Seth Hall)^J^J * Adding "install-aux" target. Addresses #622. (Jon Siwek)^J^J * Distribution cleanup. (Jon Siwek and Robin Sommer)^J^J * FindPCAP now links against +file_stream, file #0, 663, thread library when necessary (e.g.^J PF_RING's libpcap) (Jon Siwek)^J^J * Install binaries with an RPATH (Jon Siwek)^J^J * Workaround for FreeBSD CMake port missing debug flags (Jon Siwek)^J^J * Rewrite of the update-changes script. (Robin Sommer)^J^J0.1-1 | 2011-06-14 21:12:41 -0700^J^J * Add a script for generating Mozilla's CA list for the SSL analyzer.^J (Seth Hall)^J^J0.1 | 2011-04-01 16:28:22 -0700^J^J * Converting build process to CMake. (Jon Siwek)^J^J * Removing cf/hf/ca-* from distribution. The README has a note where^J to find them now. (Robin Sommer)^J^J * General cleanup. (Robin Sommer)^J^J * Initial import of bro/aux from SVN r7088. (Jon Siwek)^J +file_chunk, file #0, 663, 4042, thread library when necessary (e.g.^J PF_RING's libpcap) (Jon Siwek)^J^J * Install binaries with an RPATH (Jon Siwek)^J^J * Workaround for FreeBSD CMake port missing debug flags (Jon Siwek)^J^J * Rewrite of the update-changes script. (Robin Sommer)^J^J0.1-1 | 2011-06-14 21:12:41 -0700^J^J * Add a script for generating Mozilla's CA list for the SSL analyzer.^J (Seth Hall)^J^J0.1 | 2011-04-01 16:28:22 -0700^J^J * Converting build process to CMake. (Jon Siwek)^J^J * Removing cf/hf/ca-* from distribution. The README has a note where^J to find them now. (Robin Sommer)^J^J * General cleanup. (Robin Sommer)^J^J * Initial import of bro/aux from SVN r7088. (Jon Siwek)^J FILE_STATE_REMOVE file #0, 4705, 0 [orig_h=141.142.228.5, orig_p=59856/tcp, resp_h=192.150.187.43, resp_p=80/tcp] diff --git a/testing/btest/Baseline/scripts.base.frameworks.file-analysis.bifs.register_mime_type/files.log b/testing/btest/Baseline/scripts.base.frameworks.file-analysis.bifs.register_mime_type/files.log index 44a90b9ee6..b836d14e47 100644 --- a/testing/btest/Baseline/scripts.base.frameworks.file-analysis.bifs.register_mime_type/files.log +++ b/testing/btest/Baseline/scripts.base.frameworks.file-analysis.bifs.register_mime_type/files.log @@ -3,8 +3,8 @@ #empty_field (empty) #unset_field - #path files -#open 2014-07-21-14-26-07 +#open 2014-09-08-21-50-32 #fields ts fuid tx_hosts rx_hosts conn_uids source depth analyzers mime_type filename duration local_orig is_orig seen_bytes total_bytes missing_bytes overflow_bytes timedout parent_fuid md5 sha1 sha256 extracted #types time string set[addr] set[addr] set[string] string count set[string] string string interval bool bool count count count count bool string string string string string -1362692527.009721 FakNcS1Jfe01uljb3 192.150.187.43 141.142.228.5 CXWv6p3arKYeMETxOg HTTP 0 MD5 text/plain - 0.000054 - F 4705 4705 0 0 F - 397168fd09991a0e712254df7bc639ac - - - -#close 2014-07-21-14-26-07 +1362692527.009512 FakNcS1Jfe01uljb3 192.150.187.43 141.142.228.5 CXWv6p3arKYeMETxOg HTTP 0 MD5 text/plain - 0.000263 - F 4705 4705 0 0 F - 397168fd09991a0e712254df7bc639ac - - - +#close 2014-09-08-21-50-32 diff --git a/testing/btest/Baseline/scripts.base.frameworks.file-analysis.logging/files.log b/testing/btest/Baseline/scripts.base.frameworks.file-analysis.logging/files.log index cc185a4f1b..daf862e3b9 100644 --- a/testing/btest/Baseline/scripts.base.frameworks.file-analysis.logging/files.log +++ b/testing/btest/Baseline/scripts.base.frameworks.file-analysis.logging/files.log @@ -3,8 +3,8 @@ #empty_field (empty) #unset_field - #path files -#open 2014-04-01-23-13-35 +#open 2014-09-08-21-55-01 #fields ts fuid tx_hosts rx_hosts conn_uids source depth analyzers mime_type filename duration local_orig is_orig seen_bytes total_bytes missing_bytes overflow_bytes timedout parent_fuid md5 sha1 sha256 extracted #types time string set[addr] set[addr] set[string] string count set[string] string string interval bool bool count count count count bool string string string string string -1362692527.009721 FakNcS1Jfe01uljb3 192.150.187.43 141.142.228.5 CXWv6p3arKYeMETxOg HTTP 0 SHA256,DATA_EVENT,MD5,EXTRACT,SHA1 text/plain - 0.000054 - F 4705 4705 0 0 F - 397168fd09991a0e712254df7bc639ac 1dd7ac0398df6cbc0696445a91ec681facf4dc47 4e7c7ef0984119447e743e3ec77e1de52713e345cde03fe7df753a35849bed18 FakNcS1Jfe01uljb3-file -#close 2014-04-01-23-13-35 +1362692527.009512 FakNcS1Jfe01uljb3 192.150.187.43 141.142.228.5 CXWv6p3arKYeMETxOg HTTP 0 SHA256,DATA_EVENT,MD5,EXTRACT,SHA1 text/plain - 0.000263 - F 4705 4705 0 0 F - 397168fd09991a0e712254df7bc639ac 1dd7ac0398df6cbc0696445a91ec681facf4dc47 4e7c7ef0984119447e743e3ec77e1de52713e345cde03fe7df753a35849bed18 FakNcS1Jfe01uljb3-file +#close 2014-09-08-21-55-01 diff --git a/testing/btest/Baseline/scripts.base.frameworks.intel.cluster-transparency/manager-1.intel.log b/testing/btest/Baseline/scripts.base.frameworks.intel.cluster-transparency/manager-1.intel.log index cd314ab408..ba19f4e8d7 100644 --- a/testing/btest/Baseline/scripts.base.frameworks.intel.cluster-transparency/manager-1.intel.log +++ b/testing/btest/Baseline/scripts.base.frameworks.intel.cluster-transparency/manager-1.intel.log @@ -3,8 +3,8 @@ #empty_field (empty) #unset_field - #path intel -#open 2014-04-01-23-13-48 -#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc seen.indicator seen.indicator_type seen.where sources -#types time string addr port addr port string string string string enum enum set[string] -1396394028.821227 - - - - - - - - 123.123.123.123 Intel::ADDR Intel::IN_ANYWHERE worker-1 -#close 2014-04-01-23-13-58 +#open 2014-09-23-16-13-39 +#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc seen.indicator seen.indicator_type seen.where seen.node sources +#types time string addr port addr port string string string string enum enum string set[string] +1411488819.555114 - - - - - - - - 123.123.123.123 Intel::ADDR Intel::IN_ANYWHERE worker-2 worker-1 +#close 2014-09-23-16-13-49 diff --git a/testing/btest/Baseline/scripts.base.frameworks.intel.input-and-match/broproc.intel.log b/testing/btest/Baseline/scripts.base.frameworks.intel.input-and-match/broproc.intel.log index c1c81a662b..33c97c0c1e 100644 --- a/testing/btest/Baseline/scripts.base.frameworks.intel.input-and-match/broproc.intel.log +++ b/testing/btest/Baseline/scripts.base.frameworks.intel.input-and-match/broproc.intel.log @@ -3,9 +3,9 @@ #empty_field (empty) #unset_field - #path intel -#open 2014-04-01-23-14-04 -#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc seen.indicator seen.indicator_type seen.where sources -#types time string addr port addr port string string string string enum enum set[string] -1396394044.377145 - - - - - - - - e@mail.com Intel::EMAIL SOMEWHERE source1 -1396394044.377145 - - - - - - - - 1.2.3.4 Intel::ADDR SOMEWHERE source1 -#close 2014-04-01-23-14-04 +#open 2014-09-23-16-14-49 +#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc seen.indicator seen.indicator_type seen.where seen.node sources +#types time string addr port addr port string string string string enum enum string set[string] +1411488889.571819 - - - - - - - - e@mail.com Intel::EMAIL SOMEWHERE bro source1 +1411488889.571819 - - - - - - - - 1.2.3.4 Intel::ADDR SOMEWHERE bro source1 +#close 2014-09-23-16-14-49 diff --git a/testing/btest/Baseline/scripts.base.frameworks.intel.read-file-dist-cluster/manager-1.intel.log b/testing/btest/Baseline/scripts.base.frameworks.intel.read-file-dist-cluster/manager-1.intel.log index f7d62eb737..d8e2d43674 100644 --- a/testing/btest/Baseline/scripts.base.frameworks.intel.read-file-dist-cluster/manager-1.intel.log +++ b/testing/btest/Baseline/scripts.base.frameworks.intel.read-file-dist-cluster/manager-1.intel.log @@ -3,11 +3,11 @@ #empty_field (empty) #unset_field - #path intel -#open 2014-04-01-23-14-12 -#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc seen.indicator seen.indicator_type seen.where sources -#types time string addr port addr port string string string string enum enum set[string] -1396394052.512481 - - - - - - - - 1.2.3.4 Intel::ADDR Intel::IN_A_TEST source1 -1396394052.512481 - - - - - - - - e@mail.com Intel::EMAIL Intel::IN_A_TEST source1 -1396394053.554897 - - - - - - - - 1.2.3.4 Intel::ADDR Intel::IN_A_TEST source1 -1396394053.554897 - - - - - - - - e@mail.com Intel::EMAIL Intel::IN_A_TEST source1 -#close 2014-04-01-23-14-21 +#open 2014-09-23-16-15-00 +#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p fuid file_mime_type file_desc seen.indicator seen.indicator_type seen.where seen.node sources +#types time string addr port addr port string string string string enum enum string set[string] +1411488900.900403 - - - - - - - - 1.2.3.4 Intel::ADDR Intel::IN_A_TEST worker-1 source1 +1411488900.900403 - - - - - - - - e@mail.com Intel::EMAIL Intel::IN_A_TEST worker-1 source1 +1411488901.923543 - - - - - - - - 1.2.3.4 Intel::ADDR Intel::IN_A_TEST worker-2 source1 +1411488901.923543 - - - - - - - - e@mail.com Intel::EMAIL Intel::IN_A_TEST worker-2 source1 +#close 2014-09-23-16-15-09 diff --git a/testing/btest/Baseline/scripts.base.protocols.http.content-range-gap/extract_files.thefile b/testing/btest/Baseline/scripts.base.protocols.http.content-range-gap/extract_files.thefile new file mode 100644 index 0000000000..fc6cc2271f Binary files /dev/null and b/testing/btest/Baseline/scripts.base.protocols.http.content-range-gap/extract_files.thefile differ diff --git a/testing/btest/Baseline/scripts.base.protocols.http.entity-gap/entity_data b/testing/btest/Baseline/scripts.base.protocols.http.entity-gap/entity_data new file mode 100644 index 0000000000..37d10fb294 --- /dev/null +++ b/testing/btest/Baseline/scripts.base.protocols.http.entity-gap/entity_data @@ -0,0 +1,4 @@ +^J0.26 | 2012-08-24 15:10:04 -0700^J^J * Fixing update-changes, which could pick the wrong control file. (Robin Sommer)^J^J * Fixing GPG signing script. (Robin Sommer)^J^J0.25 | 2012-08-01 13:55:46 -0500^J^J * Fix configure script to exit with non-zero status on error (Jon Siwek)^J^J0.24 | 2012-07-05 12:50:43 -0700^J^J * Raise minimum required CMake version to 2.6.3 (Jon Siwek)^J^J * Adding script to delete old fully-merged branches. (Robin Sommer)^J^J0.23-2 | 2012-01-25 13:24:01 -0800^J^J * Fix a bro-cut error message. (Daniel Thayer)^J^J0.23 | 2012-01-11 12:16:11 -0800^J^J * Tweaks to release scripts, plus a new one for signing files.^J (Robin Sommer)^J^J0.22 | 2012-01-10 16:45:19 -0800^J^J * Tweaks for OpenBSD support. (Jon Siwek)^J^J * bro-cut extensions and fixes. (Robin Sommer)^J ^J - If no field names are given on the command line, we now pass through^J all fields. Adresses #657.^J^J - Removing some GNUism from awk script. Addresses #653.^J^J - Added option for time output in UTC. Addresses #668.^J^J - Added output field separator option -F. Addresses #649.^J^J - Fixing option -c: only some header lines were passed through^J +<1448 byte gap> +s/check-release to run before making releases.^J (Robin Sommer)^J^J * devel-tools/update-changes gets a new option -a to amend to^J previous commit if possible. Default is now not to (used to be the^J opposite). (Robin Sommer)^J^J * Change Mozilla trust root generation to index certs by subject DN. (Jon Siwek)^J^J * Change distclean to only remove build dir. (Jon Siwek)^J^J * Make dist now cleans the copied source (Jon Siwek)^J^J * Small tweak to make-release for forced git-clean. (Jon Siwek)^J^J * Fix to not let updates scripts loose their executable permissions.^J (Robin Sommer)^J^J * devel-tools/update-changes now looks for a 'release' tag to^J idenfify the stable version, and 'beta' for the beta versions.^J (Robin Sommer).^J^J * Distribution cleanup. (Robin Sommer)^J^J * New script devel-tools/make-release to create source tar balls.^J (Robin Sommer)^J^J * Removing bdcat. With the new log format, this isn't very useful^J anymore. (Robin Sommer)^J^J * Adding script that shows all pending git fastpath commits. (Robin^J Sommer)^J^J * Script to measure CPU time by loading an increasing set of^J scripts. (Robin Sommer)^J^J * extract-conn script now deals wit *.gz files. (Robin Sommer)^J^J * Tiny update to output a valid CA list file for SSL cert^J validation. (Seth Hall)^J^J * Adding "install-aux" target. Addresses #622. (Jon Siwek)^J^J * Distribution cleanup. (Jon Siwek and Robin Sommer)^J^J * FindPCAP now links against + thread library when necessary (e.g.^J PF_RING's libpcap) (Jon Siwek)^J^J * Install binaries with an RPATH (Jon Siwek)^J^J * Workaround for FreeBSD CMake port missing debug flags (Jon Siwek)^J^J * Rewrite of the update-changes script. (Robin Sommer)^J^J0.1-1 | 2011-06-14 21:12:41 -0700^J^J * Add a script for generating Mozilla's CA list for the SSL analyzer.^J (Seth Hall)^J^J0.1 | 2011-04-01 16:28:22 -0700^J^J * Converting build process to CMake. (Jon Siwek)^J^J * Removing cf/hf/ca-* from distribution. The README has a note where^J to find them now. (Robin Sommer)^J^J * General cleanup. (Robin Sommer)^J^J * Initial import of bro/aux from SVN r7088. (Jon Siwek)^J diff --git a/testing/btest/Baseline/scripts.base.protocols.http.entity-gap/extract_files.file0 b/testing/btest/Baseline/scripts.base.protocols.http.entity-gap/extract_files.file0 new file mode 100644 index 0000000000..f032ec61d3 Binary files /dev/null and b/testing/btest/Baseline/scripts.base.protocols.http.entity-gap/extract_files.file0 differ diff --git a/testing/btest/Baseline/scripts.base.protocols.http.entity-gap2/entity_data b/testing/btest/Baseline/scripts.base.protocols.http.entity-gap2/entity_data new file mode 100644 index 0000000000..c6e5999e07 --- /dev/null +++ b/testing/btest/Baseline/scripts.base.protocols.http.entity-gap2/entity_data @@ -0,0 +1,4 @@ +^J0.26 | 2012-08-24 15:10:04 -0700^J^J * Fixing update-changes, which could pick the wrong control file. (Robin Sommer)^J^J * Fixing GPG signing script. (Robin Sommer)^J^J0.25 | 2012-08-01 13:55:46 -0500^J^J * Fix configure script to exit with non-zero status on error (Jon Siwek)^J^J0.24 | 2012-07-05 12:50:43 -0700^J^J * Raise minimum required CMake version to 2.6.3 (Jon Siwek)^J^J * Adding script to delete old fully-merged branches. (Robin Sommer)^J^J0.23-2 | 2012-01-25 13:24:01 -0800^J^J * Fix a bro-cut error message. (Daniel Thayer)^J^J0.23 | 2012-01-11 12:16:11 -0800^J^J * Tweaks to release scripts, plus a new one for signing files.^J (Robin Sommer)^J^J0.22 | 2012-01-10 16:45:19 -0800^J^J * Tweaks for OpenBSD support. (Jon Siwek)^J^J * bro-cut extensions and fixes. (Robin Sommer)^J ^J - If no field names are given on the command line, we now pass through^J all fields. Adresses #657.^J^J - Removing some GNUism from awk script. Addresses #653.^J^J - Added option for time output in UTC. Addresses #668.^J^J - Added output field separator option -F. Addresses #649.^J^J - Fixing option -c: only some header lines were passed through^J + rather than all. (Robin Sommer)^J^J * Fix parallel make portability. (Jon Siwek)^J^J0.21-9 | 2011-11-07 05:44:14 -0800^J^J * Fixing compiler warnings. Addresses #388. (Jon Siwek)^J^J0.21-2 | 2011-11-02 18:12:13 -0700^J^J * Fix for misnaming temp file in update-changes script. (Robin Sommer)^J^J0.21-1 | 2011-11-02 18:10:39 -0700^J^J * Little fix for make-release script, which could pick out the wrong^J tag. (Robin Sommer)^J^J0.21 | 2011-10-27 17:40:45 -0700^J^J * Fixing bro-cut's usage message and argument error handling. (Robin Sommer)^J^J * Bugfix in update-changes script. (Robin Sommer)^J^J * update-changes now ignores commits it did itself. (Robin Sommer)^J^J * Fix a bug in the update-changes script. (Robin Sommer)^J^J * bro-cut now always installs to $prefix/bin by `make install`. (Jon Siwek)^J^J * Options to adjust time format for bro-cut. (Robin Sommer)^J^J The default with -d is now ISO format. The new option "-D "^J specifies a custom strftime()-style format string. Alternatively,^J the environment variable BRO_CUT_TIMEFMT can set the format as^J well.^J^J * bro-cut now understands the field separator header. (Robin Sommer)^J^J * Renaming options -h/-H -> -c/-C, and doing some general cleanup.^J^J0.2 | 2011-10-25 19:53:57 -0700^J^J * Adding support for replacing version string in a setup.py. (Robin^J Sommer)^J^J * Change generated root cert DN indices format for RFC2253^J compliance. (Jon Siwek)^J^J * New tool devel-tool +<1448 byte gap> + thread library when necessary (e.g.^J PF_RING's libpcap) (Jon Siwek)^J^J * Install binaries with an RPATH (Jon Siwek)^J^J * Workaround for FreeBSD CMake port missing debug flags (Jon Siwek)^J^J * Rewrite of the update-changes script. (Robin Sommer)^J^J0.1-1 | 2011-06-14 21:12:41 -0700^J^J * Add a script for generating Mozilla's CA list for the SSL analyzer.^J (Seth Hall)^J^J0.1 | 2011-04-01 16:28:22 -0700^J^J * Converting build process to CMake. (Jon Siwek)^J^J * Removing cf/hf/ca-* from distribution. The README has a note where^J to find them now. (Robin Sommer)^J^J * General cleanup. (Robin Sommer)^J^J * Initial import of bro/aux from SVN r7088. (Jon Siwek)^J diff --git a/testing/btest/Baseline/scripts.base.protocols.http.entity-gap2/extract_files.file0 b/testing/btest/Baseline/scripts.base.protocols.http.entity-gap2/extract_files.file0 new file mode 100644 index 0000000000..8eb236cb55 Binary files /dev/null and b/testing/btest/Baseline/scripts.base.protocols.http.entity-gap2/extract_files.file0 differ diff --git a/testing/btest/Baseline/scripts.base.protocols.modbus.exception_handling/modbus.log b/testing/btest/Baseline/scripts.base.protocols.modbus.exception_handling/modbus.log index 407487756d..d51e336b0a 100644 --- a/testing/btest/Baseline/scripts.base.protocols.modbus.exception_handling/modbus.log +++ b/testing/btest/Baseline/scripts.base.protocols.modbus.exception_handling/modbus.log @@ -3,9 +3,13 @@ #empty_field (empty) #unset_field - #path modbus -#open 2013-08-26-19-04-11 +#open 2014-09-11-15-00-05 #fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p func exception #types time string addr port addr port string string 1153491909.414125 CXWv6p3arKYeMETxOg 192.168.66.235 2582 166.161.16.230 502 unknown-156 - +1153491911.997264 CXWv6p3arKYeMETxOg 192.168.66.235 2582 166.161.16.230 502 unknown-160 - 1153491913.013726 CXWv6p3arKYeMETxOg 192.168.66.235 2582 166.161.16.230 502 unknown-162 - -#close 2013-08-26-19-04-11 +1153491923.091742 CXWv6p3arKYeMETxOg 192.168.66.235 2582 166.161.16.230 502 unknown-175 - +1153491923.091742 CXWv6p3arKYeMETxOg 192.168.66.235 2582 166.161.16.230 502 unknown-179 - +1153491923.623460 CXWv6p3arKYeMETxOg 192.168.66.235 2582 166.161.16.230 502 unknown-165 - +#close 2014-09-11-15-00-05 diff --git a/testing/btest/Baseline/scripts.policy.misc.dump-events/all-events-no-args.log b/testing/btest/Baseline/scripts.policy.misc.dump-events/all-events-no-args.log index 6de44b1fbf..cad75c268d 100644 --- a/testing/btest/Baseline/scripts.policy.misc.dump-events/all-events-no-args.log +++ b/testing/btest/Baseline/scripts.policy.misc.dump-events/all-events-no-args.log @@ -64,10 +64,9 @@ 1254722770.692743 mime_begin_entity 1254722770.692743 mime_one_header 1254722770.692743 mime_one_header -1254722770.692786 get_file_handle +1254722770.692743 get_file_handle 1254722770.692786 file_new 1254722770.692786 file_over_new_connection -1254722770.692804 get_file_handle 1254722770.692804 mime_end_entity 1254722770.692804 get_file_handle 1254722770.692804 file_state_remove @@ -79,20 +78,10 @@ 1254722770.692804 mime_one_header 1254722770.692804 mime_one_header 1254722770.692804 mime_one_header -1254722770.692823 get_file_handle +1254722770.692804 get_file_handle 1254722770.692823 file_new 1254722770.692823 file_over_new_connection -1254722770.692823 get_file_handle 1254722770.695115 new_connection -1254722771.469814 get_file_handle -1254722771.494181 get_file_handle -1254722771.494181 get_file_handle -1254722771.494199 get_file_handle -1254722771.834628 get_file_handle -1254722771.834655 get_file_handle -1254722771.834655 get_file_handle -1254722771.858316 get_file_handle -1254722771.858334 get_file_handle 1254722771.858334 mime_end_entity 1254722771.858334 get_file_handle 1254722771.858334 file_state_remove diff --git a/testing/btest/Baseline/scripts.policy.misc.dump-events/all-events.log b/testing/btest/Baseline/scripts.policy.misc.dump-events/all-events.log index b8f576e497..157ca42d75 100644 --- a/testing/btest/Baseline/scripts.policy.misc.dump-events/all-events.log +++ b/testing/btest/Baseline/scripts.policy.misc.dump-events/all-events.log @@ -331,22 +331,17 @@ [0] c: connection = [id=[orig_h=10.10.1.4, orig_p=1470/tcp, resp_h=74.53.140.153, resp_p=25/tcp], orig=[size=1610, state=4, num_pkts=9, num_bytes_ip=518, flow_label=0], resp=[size=462, state=4, num_pkts=10, num_bytes_ip=870, flow_label=0], start_time=1254722767.529046, duration=3.163697, service={^J^ISMTP^J}, addl=, hot=0, history=ShAdDa, uid=CjhGID4nQcgTWjvg4c, tunnel=, dpd=, conn=, extract_orig=F, extract_resp=F, dhcp=, dnp3=, dns=, dns_state=, ftp=, ftp_data_reuse=F, ssl=, http=, http_state=, irc=, modbus=, radius=, snmp=, smtp=[ts=1254722768.219663, uid=CjhGID4nQcgTWjvg4c, id=[orig_h=10.10.1.4, orig_p=1470/tcp, resp_h=74.53.140.153, resp_p=25/tcp], trans_depth=1, helo=GP, mailfrom=, rcptto={^J^I^J}, date=Mon, 5 Oct 2009 11:36:07 +0530, from="Gurpartap Singh" , to={^J^I^J}, reply_to=, msg_id=<000301ca4581$ef9e57f0$cedb07d0$@in>, in_reply_to=, subject=SMTP, x_originating_ip=, first_received=, second_received=, last_reply=354 Enter message, ending with "." on a line by itself, path=[74.53.140.153, 10.10.1.4], user_agent=Microsoft Office Outlook 12.0, tls=F, process_received_from=T, has_client_activity=T, entity=[filename=], fuids=[Fel9gs4OtNEV6gUJZ5]], smtp_state=[helo=GP, messages_transferred=0, pending_messages=, mime_depth=4], socks=, ssh=, syslog=] [1] h: mime_header_rec = [name=CONTENT-TRANSFER-ENCODING, value=quoted-printable] -1254722770.692786 get_file_handle +1254722770.692743 get_file_handle [0] tag: enum = Analyzer::ANALYZER_SMTP - [1] c: connection = [id=[orig_h=10.10.1.4, orig_p=1470/tcp, resp_h=74.53.140.153, resp_p=25/tcp], orig=[size=3070, state=4, num_pkts=10, num_bytes_ip=2018, flow_label=0], resp=[size=462, state=4, num_pkts=10, num_bytes_ip=870, flow_label=0], start_time=1254722767.529046, duration=3.16374, service={^J^ISMTP^J}, addl=, hot=0, history=ShAdDa, uid=CjhGID4nQcgTWjvg4c, tunnel=, dpd=, conn=, extract_orig=F, extract_resp=F, dhcp=, dnp3=, dns=, dns_state=, ftp=, ftp_data_reuse=F, ssl=, http=, http_state=, irc=, modbus=, radius=, snmp=, smtp=[ts=1254722768.219663, uid=CjhGID4nQcgTWjvg4c, id=[orig_h=10.10.1.4, orig_p=1470/tcp, resp_h=74.53.140.153, resp_p=25/tcp], trans_depth=1, helo=GP, mailfrom=, rcptto={^J^I^J}, date=Mon, 5 Oct 2009 11:36:07 +0530, from="Gurpartap Singh" , to={^J^I^J}, reply_to=, msg_id=<000301ca4581$ef9e57f0$cedb07d0$@in>, in_reply_to=, subject=SMTP, x_originating_ip=, first_received=, second_received=, last_reply=354 Enter message, ending with "." on a line by itself, path=[74.53.140.153, 10.10.1.4], user_agent=Microsoft Office Outlook 12.0, tls=F, process_received_from=T, has_client_activity=T, entity=[filename=], fuids=[Fel9gs4OtNEV6gUJZ5]], smtp_state=[helo=GP, messages_transferred=0, pending_messages=, mime_depth=4], socks=, ssh=, syslog=] + [1] c: connection = [id=[orig_h=10.10.1.4, orig_p=1470/tcp, resp_h=74.53.140.153, resp_p=25/tcp], orig=[size=1610, state=4, num_pkts=9, num_bytes_ip=518, flow_label=0], resp=[size=462, state=4, num_pkts=10, num_bytes_ip=870, flow_label=0], start_time=1254722767.529046, duration=3.163697, service={^J^ISMTP^J}, addl=, hot=0, history=ShAdDa, uid=CjhGID4nQcgTWjvg4c, tunnel=, dpd=, conn=, extract_orig=F, extract_resp=F, dhcp=, dnp3=, dns=, dns_state=, ftp=, ftp_data_reuse=F, ssl=, http=, http_state=, irc=, modbus=, radius=, snmp=, smtp=[ts=1254722768.219663, uid=CjhGID4nQcgTWjvg4c, id=[orig_h=10.10.1.4, orig_p=1470/tcp, resp_h=74.53.140.153, resp_p=25/tcp], trans_depth=1, helo=GP, mailfrom=, rcptto={^J^I^J}, date=Mon, 5 Oct 2009 11:36:07 +0530, from="Gurpartap Singh" , to={^J^I^J}, reply_to=, msg_id=<000301ca4581$ef9e57f0$cedb07d0$@in>, in_reply_to=, subject=SMTP, x_originating_ip=, first_received=, second_received=, last_reply=354 Enter message, ending with "." on a line by itself, path=[74.53.140.153, 10.10.1.4], user_agent=Microsoft Office Outlook 12.0, tls=F, process_received_from=T, has_client_activity=T, entity=[filename=], fuids=[Fel9gs4OtNEV6gUJZ5]], smtp_state=[helo=GP, messages_transferred=0, pending_messages=, mime_depth=4], socks=, ssh=, syslog=] [2] is_orig: bool = F 1254722770.692786 file_new - [0] f: fa_file = [id=Ft4M3f2yMvLlmwtbq9, parent_id=, source=SMTP, is_orig=F, conns={^J^I[[orig_h=10.10.1.4, orig_p=1470/tcp, resp_h=74.53.140.153, resp_p=25/tcp]] = [id=[orig_h=10.10.1.4, orig_p=1470/tcp, resp_h=74.53.140.153, resp_p=25/tcp], orig=[size=3070, state=4, num_pkts=10, num_bytes_ip=2018, flow_label=0], resp=[size=462, state=4, num_pkts=10, num_bytes_ip=870, flow_label=0], start_time=1254722767.529046, duration=3.16374, service={^J^I^ISMTP^J^I}, addl=, hot=0, history=ShAdDa, uid=CjhGID4nQcgTWjvg4c, tunnel=, dpd=, conn=, extract_orig=F, extract_resp=F, dhcp=, dnp3=, dns=, dns_state=, ftp=, ftp_data_reuse=F, ssl=, http=, http_state=, irc=, modbus=, radius=, snmp=, smtp=[ts=1254722768.219663, uid=CjhGID4nQcgTWjvg4c, id=[orig_h=10.10.1.4, orig_p=1470/tcp, resp_h=74.53.140.153, resp_p=25/tcp], trans_depth=1, helo=GP, mailfrom=, rcptto={^J^I^I^J^I}, date=Mon, 5 Oct 2009 11:36:07 +0530, from="Gurpartap Singh" , to={^J^I^I^J^I}, reply_to=, msg_id=<000301ca4581$ef9e57f0$cedb07d0$@in>, in_reply_to=, subject=SMTP, x_originating_ip=, first_received=, second_received=, last_reply=354 Enter message, ending with "." on a line by itself, path=[74.53.140.153, 10.10.1.4], user_agent=Microsoft Office Outlook 12.0, tls=F, process_received_from=T, has_client_activity=T, entity=[filename=], fuids=[Fel9gs4OtNEV6gUJZ5]], smtp_state=[helo=GP, messages_transferred=0, pending_messages=, mime_depth=4], socks=, ssh=, syslog=]^J}, last_active=1254722770.692786, seen_bytes=0, total_bytes=, missing_bytes=0, overflow_bytes=0, timeout_interval=2.0 mins, bof_buffer_size=1024, bof_buffer=^M^J^M^J^M^J^M^J^M^J