Merge remote-tracking branch 'origin/master' into topic/robin/reader-writer-plugins

CHANGES

@@ -1,71 +1,69 @@
 
+2.3-86 | 2014-07-31 14:19:58 -0700
+
+  * Fix for redefining built-in constants. (Robin Sommer)
+    
+  * Adding missing check that a plugin's API version matches what Bro
+    defines. (Robin Sommer)
+
+  * Adding NEWS entry for plugins. (Robin Sommer)
+
 2.3-83 | 2014-07-30 16:26:11 -0500
 
   * Minor adjustments to plugin code/docs. (Jon Siwek)
 
-  * Dynamic plugin support. (Robin Sommer)
+  * Dynamic plugin support. (Rpbin Sommer)
 
-    - An overview of main functionality is in doc/devel/plugins.rst.
+    Bro now supports extending core functionality, like protocol and
+    file analysis, dynamically with external plugins in the form of
+    shared libraries. See doc/devel/plugins.rst for an overview of the
+    main functionality. Changes coming with this:
 
-    - This removes the old Plugin macro magic, and hence touches all the
+        - Replacing the old Plugin macro magic with a new API.
-      existing analyzers to move them to the new API.
           
-    - The plugin API changed to generally use std::strings instead of
+        - The plugin API changed to generally use std::strings instead
-      const char*.
+          of const char*.
 
-    - There are a number of invocations of PLUGIN_HOOK_
+        - There are a number of invocations of PLUGIN_HOOK_
-      {VOID,WITH_RESULT} across the code base, which allow plugins to
+          {VOID,WITH_RESULT} across the code base, which allow plugins
-      hook into the processing at those locations. These are macros to
+          to hook into the processing at those locations.
-      make sure the overhead remains as low as possible when no plugin
-      actually defines a hook (i.e., the normal case). See
-      src/plugin/Manager.h for the macros' definition.
 
-    - There's one hook which could be potentially expensive: plugins can
+        - A few new accessor methods to various classes to allow
-      be notified if a BroObj they are interested in gets destroyed. But
+          plugins to get to that information.
-      I didn't see a performance impact in my tests (with no such hook
-      defined), and the memory usage doesn't change due to field
-      alignment.
 
-    - Adds a few new accessor methods to various classes to allow
+        - network_time cannot be just assigned to anymore, there's now
-      plugins to get to that information.
+          function net_update_time() for that.
 
-    - network_time cannot be just assigned to anymore, there's now
+        - Redoing how builtin variables are initialized, so that it
-      function net_update_time() for that.
+          works for plugins as well. No more init_net_var(), but
+          instead bifcl-generated code that registers them.
 
-    - Redos how builtin variables are initialized, so that it
+        - Various changes for adjusting to the now dynamic generation
-      works for plugins as well. No more init_net_var(), but instead
+          of analyzer instances.
-      bifcl-generated code that registers them.
 
     - same_type() gets an optional extra argument allowing record type
-      comparision to ignore if field names don't match.
+      comparision to ignore if field names don't match. (Robin Sommer)
 
-    - There are various changes for adjusting to the now dynamic
+    - Further unify file analysis API with the protocol analyzer API
-      generation of analyzer instances.
+      (assigning IDs to analyzers; adding Init()/Done() methods;
+      adding subtypes). (Robin Sommer)
 
-    - The file analysis API gets unified further with the protocol
+    - A new command line option -Q that prints some basic execution
-      analyzer API (assigning IDs to analyzers; adding Init()/Done()
+      time stats. (Robin Sommer)
-      methods; adding subtypes).
 
-    - Adding a new command line option -Q that prints some basic
+    - Add support to the file analysis for activating analyzers by
-      execution time stats. Seems generally useful, and I'm planing
+      MIME type. (Robin Sommer)
-      to provide a plugin hook for measuring custom stuff.
 
-    - I'm not yet happy with the current conventions for the C++
+          - File::register_for_mime_type(tag: Analyzer::Tag, mt:
-      namespaces that plugins are in. I'm planing to clean that up later
+            string): Associates a file analyzer with a MIME type.
-      though, as I have some more branches relying on the current scheme
-      and it will be easier to clean things up once everything is in.
 
-    - There's a new piece of functionality for the file analysis
+          - File::add_analyzers_for_mime_type(f: fa_file, mtype:
-      framework: activate analyzers by MIME type. Pieces going in there:
+            string): Activates all analyzers registered for a MIME
+            type for the file.
 
-      - File::register_for_mime_type(tag: Analyzer::Tag, mt: string):
+          - The default file_new() handler calls
-        Associates a file analyzer with a MIME type.
+            File::add_analyzers_for_mime_type() with the file's MIME
-
+            type.
-      - File::add_analyzers_for_mime_type(f: fa_file, mtype: string):
-        Activates all analyzers registered for a MIME type for the file.
-
-      - The default file_new() handler calls
-        File::add_analyzers_for_mime_type() with the file's MIME type.
 
 2.3-20 | 2014-07-22 17:41:02 -0700
 

NEWS

@@ -4,6 +4,31 @@ release. For an exhaustive list of changes, see the ``CHANGES`` file
 (note that submodules, such as BroControl and Broccoli, come with
 their own ``CHANGES``.)
 
+Bro 2.4 (in progress)
+=====================
+
+Dependencies
+------------
+
+New Functionality
+-----------------
+
+- Bro now has support for external plugins that can extend its core
+  functionality, like protocol/file analysis, via shared libraries.
+  Plugins can be developed and distributed externally, and will be
+  pulled in dynamically at startup. Currently, a plugin can provide
+  custom protocol analyzers, file analyzers, log writers[TODO], input
+  readers[TODO], packet sources[TODO], and new built-in functions. A
+  plugin can furthermore hook into Bro's processing a number of places
+  to add custom logic.
+
+  See http://www.bro.org/sphinx-git/devel/plugins.html for more
+  information on writing plugins.
+
+Changed Functionality
+---------------------
+
+
 Bro 2.3
 =======
 

VERSION

@@ -1 +1 @@
-2.3-83
+2.3-86

src/Func.cc

@@ -628,11 +628,14 @@ void init_builtin_funcs()
 #include "reporter.bif.func_init"
 #include "strings.bif.func_init"
 
-#include "__all__.bif.init.cc" // Autogenerated for compiling in the bif_target() code.
-
 	did_builtin_init = true;
 	}
 
+void init_builtin_funcs_subdirs()
+{
+	#include "__all__.bif.init.cc" // Autogenerated for compiling in the bif_target() code.
+}
+
 bool check_built_in_call(BuiltinFunc* f, CallExpr* call)
 	{
 	if ( f->TheFunc() != BifFunc::bro_fmt )

src/Func.h

@@ -134,6 +134,7 @@ protected:
 
 extern void builtin_error(const char* msg, BroObj* arg = 0);
 extern void init_builtin_funcs();
+extern void init_builtin_funcs_subdirs();
 
 extern bool check_built_in_call(BuiltinFunc* f, CallExpr* call);
 

src/Tag.cc

@@ -55,6 +55,7 @@ Tag& Tag::operator=(const Tag& other)
 		{
 		type = other.type;
 		subtype = other.subtype;
+		Unref(val);
 		val = other.val;
 
 		if ( val )

src/main.cc

@@ -898,6 +898,7 @@ int main(int argc, char** argv)
 
 	init_general_global_var();
 	init_net_var();
+	init_builtin_funcs_subdirs();
 
 	plugin_mgr->InitBifs();
 

src/plugin/Manager.cc

@@ -219,6 +219,10 @@ bool Manager::ActivateDynamicPluginInternal(const std::string& name, bool ok_if_
 			current_plugin->SetDynamic(true);
 			current_plugin->DoConfigure();
 
+			if ( current_plugin->APIVersion() != BRO_PLUGIN_API_VERSION )
+				reporter->FatalError("plugin's API version does not match Bro (expected %d, got %d in %s)",
+						     BRO_PLUGIN_API_VERSION, current_plugin->APIVersion(), path);
+
 			// We execute the pre-script initialization here; this in
 			// fact could be *during* script initialization if we got
 			// triggered via @load-plugin.

src/plugin/Plugin.cc

@@ -33,13 +33,6 @@ const char* plugin::hook_name(HookType h)
 	return hook_names[int(h)];
 	}
 
-Configuration::Configuration()
-	{
-	name = "";
-	description = "";
-	api_version = BRO_PLUGIN_API_VERSION;
-	}
-
 BifItem::BifItem(const std::string& arg_id, Type arg_type)
 	{
 	id = arg_id;

src/plugin/Plugin.h

@@ -10,7 +10,10 @@
 #include "analyzer/Component.h"
 #include "file_analysis/Component.h"
 
-static const int BRO_PLUGIN_API_VERSION = 2;
+// We allow to override this externally for testing purposes.
+#ifndef BRO_PLUGIN_API_VERSION
+#define BRO_PLUGIN_API_VERSION 2
+#endif
 
 class ODesc;
 class Func;
@@ -75,13 +78,23 @@ public:
 	std::string description;	//< A short textual description of the plugin. Mandatory.
 	VersionNumber version;		//< THe plugin's version. Optional.
 
-	Configuration();
+	// We force this to inline so that the API version gets hardcoded
+	// into the external plugin. (Technically, it's not a "force", just a
+	// strong hint.). The attribute seems generally available.
+	inline Configuration() __attribute__((always_inline));
 
 private:
 	friend class Plugin;
 	int api_version;	// Current BRO_PLUGIN_API_VERSION. Automatically set.
 };
 
+inline Configuration::Configuration()
+		{
+		name = "";
+		description = "";
+		api_version = BRO_PLUGIN_API_VERSION;
+		}
+
 /**
  * A class describing an item defined in \c *.bif file.
  */

testing/btest/Baseline/plugins.api-version-mismatch/output

@@ -0,0 +1 @@
+fatal error in /home/robin/bro/master/scripts/base/init-bare.bro, line 1: plugin's API version does not match Bro (expected 2, got 42 in /home/robin/bro/master/testing/btest/.tmp/plugins.api-version-mismatch//lib/Demo-Foo.linux-x86_64.so)

testing/btest/plugins/api-version-mismatch.sh

@@ -0,0 +1,7 @@
+# @TEST-EXEC: ${DIST}/aux/bro-aux/plugin-support/init-plugin Demo Foo
+# @TEST-EXEC: bash %INPUT
+# @TEST-EXEC: make BRO=${DIST}
+# @TEST-EXEC-FAIL: BRO_PLUGIN_PATH=`pwd` bro -NN Demo::Foo >>output 2>&1
+# @TEST-EXEC: btest-diff output
+
+( echo '#define BRO_PLUGIN_API_VERSION 42'; cat src/Plugin.cc; ) >src/Plugin.cc.tmp && mv src/Plugin.cc.tmp src/Plugin.cc