Merge remote-tracking branch 'origin/master' into topic/robin/reader-writer-plugins

CHANGES

@@ -1,4 +1,83 @@
 
+2.3-83 | 2014-07-30 16:26:11 -0500
+
+  * Minor adjustments to plugin code/docs. (Jon Siwek)
+
+  * Dynamic plugin support. (Robin Sommer)
+
+    - An overview of main functionality is in doc/devel/plugins.rst.
+
+    - This removes the old Plugin macro magic, and hence touches all the
+      existing analyzers to move them to the new API.
+
+    - The plugin API changed to generally use std::strings instead of
+      const char*.
+
+    - There are a number of invocations of PLUGIN_HOOK_
+      {VOID,WITH_RESULT} across the code base, which allow plugins to
+      hook into the processing at those locations. These are macros to
+      make sure the overhead remains as low as possible when no plugin
+      actually defines a hook (i.e., the normal case). See
+      src/plugin/Manager.h for the macros' definition.
+
+    - There's one hook which could be potentially expensive: plugins can
+      be notified if a BroObj they are interested in gets destroyed. But
+      I didn't see a performance impact in my tests (with no such hook
+      defined), and the memory usage doesn't change due to field
+      alignment.
+
+    - Adds a few new accessor methods to various classes to allow
+      plugins to get to that information.
+
+    - network_time cannot be just assigned to anymore, there's now
+      function net_update_time() for that.
+
+    - Redos how builtin variables are initialized, so that it
+      works for plugins as well. No more init_net_var(), but instead
+      bifcl-generated code that registers them.
+
+    - same_type() gets an optional extra argument allowing record type
+      comparision to ignore if field names don't match.
+
+    - There are various changes for adjusting to the now dynamic
+      generation of analyzer instances.
+
+    - The file analysis API gets unified further with the protocol
+      analyzer API (assigning IDs to analyzers; adding Init()/Done()
+      methods; adding subtypes).
+
+    - Adding a new command line option -Q that prints some basic
+      execution time stats. Seems generally useful, and I'm planing
+      to provide a plugin hook for measuring custom stuff.
+
+    - I'm not yet happy with the current conventions for the C++
+      namespaces that plugins are in. I'm planing to clean that up later
+      though, as I have some more branches relying on the current scheme
+      and it will be easier to clean things up once everything is in.
+
+    - There's a new piece of functionality for the file analysis
+      framework: activate analyzers by MIME type. Pieces going in there:
+
+      - File::register_for_mime_type(tag: Analyzer::Tag, mt: string):
+        Associates a file analyzer with a MIME type.
+
+      - File::add_analyzers_for_mime_type(f: fa_file, mtype: string):
+        Activates all analyzers registered for a MIME type for the file.
+
+      - The default file_new() handler calls
+        File::add_analyzers_for_mime_type() with the file's MIME type.
+
+2.3-20 | 2014-07-22 17:41:02 -0700
+
+  * Updating submodule(s).
+
+2.3-19 | 2014-07-22 17:29:19 -0700
+
+  * Implement bytestring_to_coils() in Modbus analyzer so that coils
+    gets passed to the corresponding events. (Hui Lin)
+
+  * Add length field to ModbusHeaders. (Hui Lin)
+
 2.3-12 | 2014-07-10 19:17:37 -0500
 
   * Include yield of vectors in Broxygen's type descriptions.

VERSION

@@ -1 +1 @@
-2.3-12
+2.3-83

aux/binpac

@@ -1 +1 @@
-Subproject commit 896ddedde55c48ec2163577fc258b49c418abb3e
+Subproject commit 231a593eb851713cc8da1ece3ce90d9317fb46b7

aux/bro-aux

@@ -1 +1 @@
-Subproject commit 315d7a2ab10f3ff1025e813dca156e7eaff35cef
+Subproject commit 8bd5f849c8f0e1c1e8397c3ad678eb8d3532fb24

aux/broccoli

@@ -1 +1 @@
-Subproject commit 17ec437752837fb4214abfb0a2da49df74668d5d
+Subproject commit 1f1f2351635bf3f437f35ceb6ba898ccff28dfea

aux/broctl

@@ -1 +1 @@
-Subproject commit 6e01d6972f02d68ee82d05f392d1a00725595b7f
+Subproject commit 2ebe2e3c9298fd21b06ff93dab8c6980edef144b

aux/btest

@@ -1 +1 @@
-Subproject commit 26c3136d56493017bc33c5a2f22ae393d585c2d9
+Subproject commit 1efa4d10f943351efea96def68e598b053fd217a

cmake

@@ -1 +1 @@
-Subproject commit fd4076e487565062d236bbe3959a6452a6ca38c7
+Subproject commit fb67896d2c3fe61c6479e78c3e9b12de8637be87

doc/devel/plugins.rst

@@ -73,7 +73,7 @@ there as follows::
             *p  = (*p - b + 13) % 26 + b;
             }
 
-        return new StringVal(strlen(rot13), rot13);
+        return new StringVal(new BroString(1, rot13, strlen(rot13)));
         %}
 
 The syntax of this file is just like any other ``*.bif`` file; we
@@ -200,7 +200,7 @@ directory.
     activated. See below for more information on activating plugins.
 
 ``lib/bif/``
-    Directory with auto-generated Bro scripts that declare the plugins
+    Directory with auto-generated Bro scripts that declare the plugin's
     bif elements. The files here are produced by ``bifcl``.
 
 By convention, a plugin should put its custom scripts into sub folders
@@ -229,9 +229,9 @@ install``).
 ``make install`` copies over the ``lib`` and ``scripts`` directories,
 as well as the ``__bro_plugin__`` magic file and the ``README`` (which
 you should customize). One can add further CMake ``install`` rules to
-install additional files if neeed.
+install additional files if needed.
 
-``init-plugin`` will never override existing files, so it's safe to
+``init-plugin`` will never overwrite existing files, so it's safe to
 rerun in an existing plugin directory; it only put files in place that
 don't exist yet. That also provides a convenient way to revert a file
 back to what ``init-plugin`` created originally: just delete it and
@@ -420,7 +420,7 @@ At runtime, one then activates a plugin's debugging output with ``-B
 plugin-<name>``, where ``<name>`` is the name of the plugin as
 returned by its ``Configure()`` method, yet with the
 namespace-separator ``::`` replaced with a simple dash. Example: If
-the plugin is called ``Bro::Demo``, use ``-B plugin-Bro-Dome``. As
+the plugin is called ``Bro::Demo``, use ``-B plugin-Bro-Demo``. As
 usual, the debugging output will be recorded to ``debug.log`` if Bro's
 compiled in debug mode.
 

scripts/base/frameworks/files/main.bro

@@ -150,8 +150,8 @@ export {
 	##          for the file isn't currently active or the *args*
 	##          were invalid for the analyzer type.
 	global add_analyzer: function(f: fa_file,
-				      tag: Files::Tag,
+	                              tag: Files::Tag,
-				      args: AnalyzerArgs &default=AnalyzerArgs()): bool;
+	                              args: AnalyzerArgs &default=AnalyzerArgs()): bool;
 
 	## Adds all analyzers associated with a give MIME type to the analysis of
 	## a file.  Note that analyzers added via MIME types cannot take further

scripts/base/init-bare.bro

@@ -2738,6 +2738,7 @@ type ModbusRegisters: vector of count;
 type ModbusHeaders: record {
 	tid:           count;
 	pid:           count;
+	len:           count;
 	uid:           count;
 	function_code: count;
 };

src/3rdparty

@@ -1 +1 @@
-Subproject commit 42a4c9694a2b2677b050fbb7cbae26bc5ec4605a
+Subproject commit 7e15efe9d28d46bfa662fcdd1cbb15ce1db285c9

src/analyzer/protocol/modbus/modbus-analyzer.pac

@@ -17,6 +17,11 @@
 	VectorVal* bytestring_to_coils(bytestring coils, uint quantity)
 		{
 		VectorVal* modbus_coils = new VectorVal(BifType::Vector::ModbusCoils);
+		for ( uint i = 0; i < quantity; i++ )
+			{
+			char currentCoil = (coils[i/8] >> (i % 8)) % 2;
+			modbus_coils->Assign(i, new Val(currentCoil, TYPE_BOOL));
+			}
 
 		return modbus_coils;
 		}
@@ -26,8 +31,9 @@
 		RecordVal* modbus_header = new RecordVal(BifType::Record::ModbusHeaders);
 		modbus_header->Assign(0, new Val(header->tid(), TYPE_COUNT));
 		modbus_header->Assign(1, new Val(header->pid(), TYPE_COUNT));
-		modbus_header->Assign(2, new Val(header->uid(), TYPE_COUNT));
+		modbus_header->Assign(2, new Val(header->len(), TYPE_COUNT));
-		modbus_header->Assign(3, new Val(header->fc(), TYPE_COUNT));
+		modbus_header->Assign(3, new Val(header->uid(), TYPE_COUNT));
+		modbus_header->Assign(4, new Val(header->fc(), TYPE_COUNT));
 		return modbus_header;
 		}
 
@@ -614,6 +620,3 @@ refine flow ModbusTCP_Flow += {
 		return true;
 		%}
 };
-
-
-

src/file_analysis/Analyzer.h

@@ -31,13 +31,13 @@ public:
 	 * Initializes the analyzer before input processing starts.
 	 */
 	virtual void Init()
-		{ };
+		{ }
 
 	/**
 	 * Finishes the analyzer's operation after all input has been parsed.
 	 */
 	virtual void Done()
-		{ };
+		{ }
 
 	/**
 	 * Subclasses may override this metod to receive file data non-sequentially.

src/main.cc

@@ -933,9 +933,6 @@ int main(int argc, char** argv)
 		}
 
 	reporter->InitOptions();
-
-	init_general_global_var();
-
 	broxygen_mgr->GenerateDocs();
 
 	if ( user_pcap_filter )

src/plugin/Component.h

@@ -76,7 +76,7 @@ public:
 
 protected:
 	/**
-	 * Adds type specific information to the outout of Describe().
+	 * Adds type specific information to the output of Describe().
 	 *
 	 * The default version does nothing.
 	 *

src/plugin/ComponentManager.h

@@ -166,7 +166,7 @@ EnumType* ComponentManager<T, C>::GetTagEnumType() const
 template <class T, class C>
 const std::string& ComponentManager<T, C>::GetComponentName(T tag) const
 	{
-	static const std::string& error = "<error>";
+	static const std::string error = "<error>";
 
 	if ( ! tag )
 		return error;

src/plugin/Manager.cc

@@ -144,7 +144,7 @@ bool Manager::ActivateDynamicPluginInternal(const std::string& name, bool ok_if_
 
 		reporter->Error("plugin %s is not available", name.c_str());
 		return false;
-                }
+		}
 
 	if ( m->second == "" )
 		// Already activated.

src/plugin/Manager.h

@@ -73,7 +73,7 @@ public:
 	void SearchDynamicPlugins(const std::string& dir);
 
 	/**
-	 * Activates a plugin that SearchPlugins() has previously discovered.
+	 * Activates a plugin that SearchDynamicPlugins() has previously discovered.
 	 * Activating a plugin involves loading its dynamic module, making its
 	 * bifs available, and adding its script paths to BROPATH.
 	 *
@@ -86,8 +86,8 @@ public:
 	bool ActivateDynamicPlugin(const std::string& name);
 
 	/**
-	 * Activates plugins that SearchPlugins() has previously discovered. The
+	 * Activates plugins that SearchDynamicPlugins() has previously discovered.
-	 * effect is the same all calling \a ActivePlugin(name) for each plugin.
+	 * The effect is the same all calling \a ActivePlugin(name) for each plugin.
 	 *
 	 * @param all If true, activates all plugins that are found. If false,
 	 * activates only those that should always be activated unconditionally,
@@ -218,7 +218,7 @@ public:
 	// Hook entry functions.
 
 	/**
-	 * Hook that gives plugins a chance to take over loading an input input
+	 * Hook that gives plugins a chance to take over loading an input
 	 * file. This method must be called between InitPreScript() and
 	 * InitPostScript() for each input file Bro is about to load, either
 	 * given on the command line or via @load script directives. The hook can
@@ -288,7 +288,7 @@ public:
 	 * Internal method that registers a bif file's init function for a
 	 * plugin.
 	 *
-	 * @param plugin The plugin to reguster the function for.
+	 * @param plugin The plugin to register the function for.
 	 *
 	 * @param c The init function to register.
 	 */

src/plugin/Plugin.h

@@ -302,7 +302,7 @@ typedef std::list<HookArgument> HookArgumentList;
  *
  * A plugin needs to explicitly register all the functionality it provides.
  * For components, it needs to call AddComponent(); for BiFs AddBifItem();
- * and for hooks EnableHook() and then also implemennt the corresponding
+ * and for hooks EnableHook() and then also implement the corresponding
  * virtual methods.
  *
  */
@@ -346,7 +346,7 @@ public:
 
 	/**
 	 * For dynamic plugins, returns the base directory from which it was
-	 * loaded. For static plugins, returns null.
+	 * loaded. For static plugins, returns an empty string.
 	 **/
 	const std::string& PluginDirectory() const;
 
@@ -377,40 +377,6 @@ public:
 	 */
 	bif_item_list BifItems() const;
 
-	/**
-	 * A function called when the plugin is instantiated to query basic
-	 * configuration parameters.
-	 *
-	 * The plugin must override this method and return a suitably
-	 * initialized configuration object.
-	 *
-	 * @return A configuration describing the plugin.
-	 */
-	virtual Configuration Configure() = 0;
-
-	/**
-	 * First-stage initialization of the plugin called early during Bro's
-	 * startup, before scripts are parsed. This can be overridden by
-	 * derived classes; they must however call the parent's
-	 * implementation.
-	 */
-	virtual void InitPreScript();
-
-	/**
-	 * Second-stage initialization of the plugin called late during Bro's
-	 * startup, after scripts are parsed. This can be overridden by
-	 * derived classes; they must however call the parent's
-	 * implementation.
-	 */
-	virtual void InitPostScript();
-
-	/**
-	 * Finalizer method that derived classes can override for performing
-	 * custom tasks at shutdown.  This can be overridden by derived
-	 * classes; they must however call the parent's implementation.
-	 */
-	virtual void Done();
-
 	/**
 	 * Returns a textual description of the plugin.
 	 *
@@ -445,7 +411,7 @@ public:
 	 * will normally be a Bro script, but it passes through the plugin
 	 * system as well to load files with other extensions as supported by
 	 * any of the current plugins. In other words, calling this method is
-	 * similar to given a file on the command line. Note that the file
+	 * similar to giving a file on the command line. Note that the file
 	 * may be only queued for now, and actually loaded later.
 	 *
 	 * This method must not be called after InitPostScript().
@@ -461,6 +427,29 @@ public:
 protected:
 	friend class Manager;
 
+	/**
+	 * First-stage initialization of the plugin called early during Bro's
+	 * startup, before scripts are parsed. This can be overridden by
+	 * derived classes; they must however call the parent's
+	 * implementation.
+	 */
+	virtual void InitPreScript();
+
+	/**
+	 * Second-stage initialization of the plugin called late during Bro's
+	 * startup, after scripts are parsed. This can be overridden by
+	 * derived classes; they must however call the parent's
+	 * implementation.
+	 */
+	virtual void InitPostScript();
+
+	/**
+	 * Finalizer method that derived classes can override for performing
+	 * custom tasks at shutdown.  This can be overridden by derived
+	 * classes; they must however call the parent's implementation.
+	 */
+	virtual void Done();
+
 	/**
 	 * Registers and activates a component.
 	 *
@@ -471,7 +460,7 @@ protected:
 	/**
 	 * Enables a hook. The corresponding virtual method will now be
 	 * called as Bro's processing proceeds. Note that enabling hooks can
-	 * have performance impaxct as many trigger frequently inside Bro's
+	 * have performance impact as many trigger frequently inside Bro's
 	 * main processing path.
 	 *
 	 * Note that while hooks may be enabled/disabled dynamically at any
@@ -557,16 +546,16 @@ protected:
 	 * from executing it). In the latter case it must provide a matching
 	 * return value.
 	 *
-	 * The default implementation does never handle the call in any way.
+	 * The default implementation never handles the call in any way.
 	 *
 	 * @param func The function being called.
 	 *
 	 * @param args The function arguments. The method can modify the list
-	 * in place long as it ensures matching types and correct reference
+	 * in place as long as it ensures matching types and correct reference
 	 * counting.
 	 *
 	 * @return If the plugin handled the call, a Val with +1 reference
-	 * count containomg the result value to pass back to the interpreter
+	 * count containixnmg the result value to pass back to the interpreter
 	 * (for void functions and events any \a Val is fine; it will be
 	 * ignored; best to use a \c TYPE_ANY). If the plugin did not handle
 	 * the call, it must return null.
@@ -581,15 +570,14 @@ protected:
 	 * inspect the event, or take it over (i.e., prevent the interpreter
 	 * from queuing it itself).
 	 *
-	 * The default implementation does never handle the queuing in any
+	 * The default implementation never handles the queuing in any way.
-	 * way.
 	 *
-	 * @param event The even to be queued. The method can modify it in in
+	 * @param event The event to be queued. The method can modify it in
-	 * place long as it ensures matching types and correct reference
+	 * place as long as it ensures matching types and correct reference
 	 * counting.
 	 *
 	 * @return True if the plugin took charge of the event; in that case
-	 * it must have assumed ownership of the event and the intpreter will
+	 * it must have assumed ownership of the event and the interpreter will
 	 * not do anything further with it. False otherwise.
 	 */
 	virtual bool HookQueueEvent(Event* event);
@@ -609,7 +597,7 @@ protected:
 	virtual void HookUpdateNetworkTime(double network_time);
 
 	/**
-	 * Hook for destruction of objects registerd with
+	 * Hook for destruction of objects registered with
 	 * RequestBroObjDtor(). When Bro's reference counting triggers the
 	 * objects destructor to run, this method will be run. It may also
 	 * run for other objects that this plugin has not registered for.
@@ -652,6 +640,18 @@ protected:
 	virtual void MetaHookPost(HookType hook, const HookArgumentList& args, HookArgument result);
 
 private:
+
+	/**
+	 * A function called when the plugin is instantiated to query basic
+	 * configuration parameters.
+	 *
+	 * The plugin must override this method and return a suitably
+	 * initialized configuration object.
+	 *
+	 * @return A configuration describing the plugin.
+	 */
+	virtual Configuration Configure() = 0;
+
 	/**
 	 * Intializes the plugin's internal configuration. Called by the
 	 * manager before anything else.

src/scan.l

@@ -319,7 +319,7 @@ when	return TOK_WHEN;
 
 @load-plugin{WS}{ID} {
 	const char* plugin = skip_whitespace(yytext + 12);
-    plugin_mgr->ActivateDynamicPlugin(plugin);
+	plugin_mgr->ActivateDynamicPlugin(plugin);
 }
 
 @unload{WS}{FILE}	{
@@ -715,7 +715,7 @@ void add_input_file_at_front(const char* file)
 	if ( ! filename )
 		(void) load_files(file);
 	else
-        	input_files.insert(copy_string(file));
+		input_files.insert(copy_string(file));
 	}
 
 void add_to_name_list(char* s, char delim, name_list& nl)

src/util.h

@@ -178,7 +178,7 @@ bool is_file(const std::string& path);
 // Replaces all occurences of *o* in *s* with *n*.
 extern std::string strreplace(const std::string& s, const std::string& o, const std::string& n);
 
-// Remove all leading and trainling white space from string.
+// Remove all leading and trailing white space from string.
 extern std::string strstrip(std::string s);
 
 extern uint8 shared_hmac_md5_key[16];

testing/btest/Baseline/bifs.count_to_addr/out

@@ -0,0 +1,4 @@
+0.0.0.1
+48.21.133.122
+255.255.255.255
+0.0.0.0

testing/btest/Baseline/plugins.bifs-and-scripts-install/output

@@ -0,0 +1,7 @@
+Demo::Foo - <Insert description> (dynamic, version 1.0)
+    [Event] plugin_event
+    [Function] hello_plugin_world
+
+plugin: automatically loaded at startup
+calling bif, Hello from the plugin!
+plugin: manually loaded

testing/btest/Baseline/plugins.bifs-and-scripts/output

@@ -1,4 +1,4 @@
-Demo::Foo - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1.0)
+Demo::Foo - <Insert description> (dynamic, version 1.0)
     [Event] plugin_event
     [Function] hello_plugin_world
 

testing/btest/Baseline/scripts.base.protocols.modbus.coil_parsing_big/coverage

@@ -0,0 +1 @@
+5 of 28 events triggered by trace

testing/btest/Baseline/scripts.base.protocols.modbus.coil_parsing_small/coverage

@@ -0,0 +1 @@
+5 of 28 events triggered by trace

testing/btest/Baseline/scripts.base.protocols.modbus.coil_parsing_small/output

@@ -0,0 +1,64 @@
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55480/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55480/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55481/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55481/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55482/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55482/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55483/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55483/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55484/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55484/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55485/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55485/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55486/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55486/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55487/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55487/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55488/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55488/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55489/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55489/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55490/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55490/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55491/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55491/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55493/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55493/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55494/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55494/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55495/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55495/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], T
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55496/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], F
+1 modbus_message, [orig_h=118.189.96.132, orig_p=55496/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], T
+1 modbus_read_coils_request, [orig_h=118.189.96.132, orig_p=55481/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], 0, 3
+1 modbus_read_coils_request, [orig_h=118.189.96.132, orig_p=55483/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], 0, 3
+1 modbus_read_coils_request, [orig_h=118.189.96.132, orig_p=55485/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], 0, 3
+1 modbus_read_coils_request, [orig_h=118.189.96.132, orig_p=55487/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], 0, 3
+1 modbus_read_coils_request, [orig_h=118.189.96.132, orig_p=55489/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], 0, 3
+1 modbus_read_coils_request, [orig_h=118.189.96.132, orig_p=55491/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], 0, 3
+1 modbus_read_coils_request, [orig_h=118.189.96.132, orig_p=55494/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], 0, 3
+1 modbus_read_coils_request, [orig_h=118.189.96.132, orig_p=55496/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=1], 0, 3
+1 modbus_read_coils_response, [orig_h=118.189.96.132, orig_p=55481/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], [F, F, F, F, F, F, F, F]
+1 modbus_read_coils_response, [orig_h=118.189.96.132, orig_p=55483/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], [F, F, T, F, F, F, F, F]
+1 modbus_read_coils_response, [orig_h=118.189.96.132, orig_p=55485/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], [F, T, F, F, F, F, F, F]
+1 modbus_read_coils_response, [orig_h=118.189.96.132, orig_p=55487/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], [F, T, T, F, F, F, F, F]
+1 modbus_read_coils_response, [orig_h=118.189.96.132, orig_p=55489/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], [T, F, F, F, F, F, F, F]
+1 modbus_read_coils_response, [orig_h=118.189.96.132, orig_p=55491/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], [T, F, T, F, F, F, F, F]
+1 modbus_read_coils_response, [orig_h=118.189.96.132, orig_p=55494/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], [T, T, F, F, F, F, F, F]
+1 modbus_read_coils_response, [orig_h=118.189.96.132, orig_p=55496/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=4, uid=1, function_code=1], [T, T, T, F, F, F, F, F]
+1 modbus_write_multiple_coils_request, [orig_h=118.189.96.132, orig_p=55480/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], 0, [F, F, F]
+1 modbus_write_multiple_coils_request, [orig_h=118.189.96.132, orig_p=55482/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], 0, [F, F, T]
+1 modbus_write_multiple_coils_request, [orig_h=118.189.96.132, orig_p=55484/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], 0, [F, T, F]
+1 modbus_write_multiple_coils_request, [orig_h=118.189.96.132, orig_p=55486/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], 0, [F, T, T]
+1 modbus_write_multiple_coils_request, [orig_h=118.189.96.132, orig_p=55488/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], 0, [T, F, F]
+1 modbus_write_multiple_coils_request, [orig_h=118.189.96.132, orig_p=55490/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], 0, [T, F, T]
+1 modbus_write_multiple_coils_request, [orig_h=118.189.96.132, orig_p=55493/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], 0, [T, T, F]
+1 modbus_write_multiple_coils_request, [orig_h=118.189.96.132, orig_p=55495/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=8, uid=1, function_code=15], 0, [T, T, T]
+1 modbus_write_multiple_coils_response, [orig_h=118.189.96.132, orig_p=55480/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], 0, 3
+1 modbus_write_multiple_coils_response, [orig_h=118.189.96.132, orig_p=55482/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], 0, 3
+1 modbus_write_multiple_coils_response, [orig_h=118.189.96.132, orig_p=55484/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], 0, 3
+1 modbus_write_multiple_coils_response, [orig_h=118.189.96.132, orig_p=55486/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], 0, 3
+1 modbus_write_multiple_coils_response, [orig_h=118.189.96.132, orig_p=55488/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], 0, 3
+1 modbus_write_multiple_coils_response, [orig_h=118.189.96.132, orig_p=55490/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], 0, 3
+1 modbus_write_multiple_coils_response, [orig_h=118.189.96.132, orig_p=55493/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], 0, 3
+1 modbus_write_multiple_coils_response, [orig_h=118.189.96.132, orig_p=55495/tcp, resp_h=118.189.96.132, resp_p=502/tcp], [tid=1, pid=0, len=6, uid=1, function_code=15], 0, 3

testing/btest/Baseline/scripts.base.protocols.modbus.register_parsing/modbus.log

@@ -3,10 +3,10 @@
 #empty_field	(empty)
 #unset_field	-
 #path	modbus
-#open	2013-08-26-19-04-19
+#open	2014-06-18-02-19-52
 #fields	ts	uid	id.orig_h	id.orig_p	id.resp_h	id.resp_p	func	exception
 #types	time	string	addr	port	addr	port	string	string
 1342774775.305761	CXWv6p3arKYeMETxOg	10.1.1.234	51411	10.10.5.104	502	READ_INPUT_REGISTERS	-
 1342775209.493066	CjhGID4nQcgTWjvg4c	10.1.1.234	51411	10.10.5.104	502	READ_INPUT_REGISTERS	-
 1342776371.617757	CsRx2w45OKnoww6xl4	10.1.1.234	51411	10.10.5.104	502	READ_INPUT_REGISTERS	-
-#close	2013-08-26-19-04-19
+#close	2014-06-18-02-19-52

testing/btest/Baseline/scripts.base.protocols.modbus.register_parsing/output

@@ -1,4 +1,4 @@
-modbus_read_input_registers_request, [orig_h=10.1.1.234, orig_p=51411/tcp, resp_h=10.10.5.104, resp_p=502/tcp], [tid=1119, pid=0, uid=255, function_code=4], 900, 147
+modbus_read_input_registers_request, [orig_h=10.1.1.234, orig_p=51411/tcp, resp_h=10.10.5.104, resp_p=502/tcp], [tid=1119, pid=0, len=6, uid=255, function_code=4], 900, 147
-modbus_read_input_registers_response, [orig_h=10.1.1.234, orig_p=51411/tcp, resp_h=10.10.5.104, resp_p=502/tcp], [tid=2606, pid=0, uid=255, function_code=4], [0, 0, 0, 0, 0, 0, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690], 100
+modbus_read_input_registers_response, [orig_h=10.1.1.234, orig_p=51411/tcp, resp_h=10.10.5.104, resp_p=502/tcp], [tid=2606, pid=0, len=203, uid=255, function_code=4], [0, 0, 0, 0, 0, 0, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690], 100
-modbus_read_input_registers_request, [orig_h=10.1.1.234, orig_p=51411/tcp, resp_h=10.10.5.104, resp_p=502/tcp], [tid=12993, pid=0, uid=255, function_code=4], 400, 100
+modbus_read_input_registers_request, [orig_h=10.1.1.234, orig_p=51411/tcp, resp_h=10.10.5.104, resp_p=502/tcp], [tid=12993, pid=0, len=6, uid=255, function_code=4], 400, 100
-modbus_read_input_registers_response, [orig_h=10.1.1.234, orig_p=51411/tcp, resp_h=10.10.5.104, resp_p=502/tcp], [tid=17667, pid=0, uid=255, function_code=4], [49, 18012, 51, 42, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 49, 50, 51, 54324, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 69, 63, 64, 65, 66, 67, 68, 49, 189, 51, 52, 53, 54, 4151, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 136, 49, 50, 51, 212, 53, 54, 170, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690], 100
+modbus_read_input_registers_response, [orig_h=10.1.1.234, orig_p=51411/tcp, resp_h=10.10.5.104, resp_p=502/tcp], [tid=17667, pid=0, len=203, uid=255, function_code=4], [49, 18012, 51, 42, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 49, 50, 51, 54324, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 69, 63, 64, 65, 66, 67, 68, 49, 189, 51, 52, 53, 54, 4151, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 136, 49, 50, 51, 212, 53, 54, 170, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690, 43690], 100

testing/btest/plugins/bifs-and-scripts-install.sh

@@ -0,0 +1,47 @@
+# @TEST-EXEC: ${DIST}/aux/bro-aux/plugin-support/init-plugin Demo Foo
+# @TEST-EXEC: bash %INPUT
+# @TEST-EXEC: BRO_PLUGIN_INSTALL=`pwd`/test-install make BRO=${DIST}
+# @TEST-EXEC: make install
+# @TEST-EXEC: BRO_PLUGIN_PATH=`pwd`/test-install bro -NN Demo::Foo >>output
+# @TEST-EXEC: BRO_PLUGIN_PATH=`pwd` bro demo/foo -r $TRACES/empty.trace >>output
+# @TEST-EXEC: TEST_DIFF_CANONIFIER= btest-diff output
+
+cat >scripts/__load__.bro <<EOF
+@load ./demo/foo/base/at-startup.bro
+EOF
+
+cat >scripts/demo/foo/__load__.bro <<EOF
+@load ./manually.bro
+EOF
+
+cat >scripts/demo/foo/manually.bro <<EOF
+event bro_init() &priority=-10
+        {
+        print "plugin: manually loaded";
+        }
+EOF
+
+mkdir -p scripts/demo/foo/base/
+
+cat >scripts/demo/foo/base/at-startup.bro <<EOF
+event bro_init() &priority=10
+        {
+        print "plugin: automatically loaded at startup";
+        print "calling bif", hello_plugin_world();
+        }
+EOF
+
+cat >src/functions.bif <<EOF
+function hello_plugin_world%(%): string
+        %{
+        return new StringVal("Hello from the plugin!");
+        %}
+EOF
+
+cat >activate.bro <<EOF
+@load-plugin Demo::Foo
+EOF
+
+cat >src/events.bif <<EOF
+event plugin_event%(foo: count%);
+EOF

testing/btest/scripts/base/protocols/modbus/coil_parsing_big.bro

@@ -0,0 +1,47 @@
+#
+# @TEST-EXEC: bro -C -r $TRACES/modbus/modbusBig.pcap %INPUT | sort | uniq -c | sed 's/^ *//g' >output
+# @TEST-EXEC: btest-diff output
+# @TEST-EXEC: cat output | awk '{print $2}' | grep "^modbus_" | sort | uniq | wc -l >covered
+# @TEST-EXEC: cat ${DIST}/src/analyzer/protocol/modbus/events.bif  | grep "^event modbus_" | wc -l >total
+# @TEST-EXEC: echo `cat covered` of `cat total` events triggered by trace >coverage
+# @TEST-EXEC: btest-diff coverage
+
+event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool)
+{
+    print "modbus_message", c$id, headers, is_orig;
+}
+
+event modbus_exception(c: connection, headers: ModbusHeaders, code: count)
+{
+    print "modbus_exception", c$id, headers, code;
+}
+
+event modbus_read_coils_request(c: connection, headers: ModbusHeaders, start_address: count, quantity: count)
+{
+    print "modbus_read_coils_request", c$id, headers, start_address, quantity;
+}
+
+event modbus_read_coils_response(c: connection, headers: ModbusHeaders, coils: ModbusCoils)
+{
+    print "modbus_read_coils_response", c$id, headers, coils;
+}
+event modbus_write_single_coil_request(c: connection, headers: ModbusHeaders, address: count, value: bool)
+{
+    print "modbus_write_single_coil_request", c$id, headers, address, value;
+}
+
+event modbus_write_single_coil_response(c: connection, headers: ModbusHeaders, address: count, value: bool)
+{
+    print "modbus_write_single_coil_response", c$id, headers, address, value;
+}
+
+event modbus_write_multiple_coils_request(c: connection, headers: ModbusHeaders, start_address: count, coils: ModbusCoils)
+{
+    print "modbus_write_multiple_coils_request", c$id, headers, start_address, coils;
+}
+
+event modbus_write_multiple_coils_response(c: connection, headers: ModbusHeaders, start_address: count, quantity: count)
+{
+    print "modbus_write_multiple_coils_response", c$id, headers, start_address, quantity;
+}
+

testing/btest/scripts/base/protocols/modbus/coil_parsing_small.bro

@@ -0,0 +1,47 @@
+#
+# @TEST-EXEC: bro -C -r $TRACES/modbus/modbusSmall.pcap %INPUT | sort | uniq -c | sed 's/^ *//g' >output
+# @TEST-EXEC: btest-diff output
+# @TEST-EXEC: cat output | awk '{print $2}' | grep "^modbus_" | sort | uniq | wc -l >covered
+# @TEST-EXEC: cat ${DIST}/src/analyzer/protocol/modbus/events.bif  | grep "^event modbus_" | wc -l >total
+# @TEST-EXEC: echo `cat covered` of `cat total` events triggered by trace >coverage
+# @TEST-EXEC: btest-diff coverage
+
+event modbus_message(c: connection, headers: ModbusHeaders, is_orig: bool)
+{
+    print "modbus_message", c$id, headers, is_orig;
+}
+
+event modbus_exception(c: connection, headers: ModbusHeaders, code: count)
+{
+    print "modbus_exception", c$id, headers, code;
+}
+
+event modbus_read_coils_request(c: connection, headers: ModbusHeaders, start_address: count, quantity: count)
+{
+    print "modbus_read_coils_request", c$id, headers, start_address, quantity;
+}
+
+event modbus_read_coils_response(c: connection, headers: ModbusHeaders, coils: ModbusCoils)
+{
+    print "modbus_read_coils_response", c$id, headers, coils;
+}
+event modbus_write_single_coil_request(c: connection, headers: ModbusHeaders, address: count, value: bool)
+{
+    print "modbus_write_single_coil_request", c$id, headers, address, value;
+}
+
+event modbus_write_single_coil_response(c: connection, headers: ModbusHeaders, address: count, value: bool)
+{
+    print "modbus_write_single_coil_response", c$id, headers, address, value;
+}
+
+event modbus_write_multiple_coils_request(c: connection, headers: ModbusHeaders, start_address: count, coils: ModbusCoils)
+{
+    print "modbus_write_multiple_coils_request", c$id, headers, start_address, coils;
+}
+
+event modbus_write_multiple_coils_response(c: connection, headers: ModbusHeaders, start_address: count, quantity: count)
+{
+    print "modbus_write_multiple_coils_response", c$id, headers, start_address, quantity;
+}
+