From 7025d511e47a49defb44e234f2de2eaf79047112 Mon Sep 17 00:00:00 2001 From: Daniel Thayer Date: Wed, 13 Nov 2013 16:45:43 -0600 Subject: [PATCH 01/56] Update the documentation of types and attributes Documented the new substring extraction functionality. Clarified the description of "&priority" and "void". Also fixed various typos. --- doc/scripts/builtins.rst | 39 +++++++++++++++++++++++++++------------ 1 file changed, 27 insertions(+), 12 deletions(-) diff --git a/doc/scripts/builtins.rst b/doc/scripts/builtins.rst index aa1a097683..85e9cd14c8 100644 --- a/doc/scripts/builtins.rst +++ b/doc/scripts/builtins.rst @@ -23,7 +23,8 @@ The Bro scripting language supports the following built-in types. .. bro:type:: void - An internal Bro type representing the absence of a return type for a + An internal Bro type (i.e., "void" is not a reserved keyword in the Bro + scripting language) representing the absence of a return type for a function. .. bro:type:: bool @@ -132,10 +133,23 @@ The Bro scripting language supports the following built-in types. Strings support concatenation (``+``), and assignment (``=``, ``+=``). Strings also support the comparison operators (``==``, ``!=``, ``<``, - ``<=``, ``>``, ``>=``). Substring searching can be performed using - the "in" or "!in" operators (e.g., "bar" in "foobar" yields true). - The number of characters in a string can be found by enclosing the - string within pipe characters (e.g., ``|"abc"|`` is 3). + ``<=``, ``>``, ``>=``). The number of characters in a string can be + found by enclosing the string within pipe characters (e.g., ``|"abc"|`` + is 3). + + The subscript operator can extract an individual character or a substring + of a string (string indexing is zero-based, but an index of + -1 refers to the last character in the string, and -2 refers to the + second-to-last character, etc.). When extracting a substring, the + starting and ending index values are separated by a colon. For example:: + + local orig = "0123456789"; + local third_char = orig[2]; + local last_char = orig[-1]; + local first_three_chars = orig[0:2]; + + Substring searching can be performed using the "in" or "!in" + operators (e.g., "bar" in "foobar" yields true). Note that Bro represents strings internally as a count and vector of bytes rather than a NUL-terminated byte string (although string @@ -767,7 +781,7 @@ The Bro scripting language supports the following built-in types. .. bro:type:: hook A hook is another flavor of function that shares characteristics of - both a :bro:type:`function` and a :bro:type:`event`. They are like + both a :bro:type:`function` and an :bro:type:`event`. They are like events in that many handler bodies can be defined for the same hook identifier and the order of execution can be enforced with :bro:attr:`&priority`. They are more like functions in the way they @@ -856,14 +870,14 @@ scripting language supports the following built-in attributes. .. bro:attr:: &optional Allows a record field to be missing. For example the type ``record { - a: int, b: port &optional }`` could be instantiated both as + a: addr; b: port &optional; }`` could be instantiated both as singleton ``[$a=127.0.0.1]`` or pair ``[$a=127.0.0.1, $b=80/tcp]``. .. bro:attr:: &default Uses a default value for a record field, a function/hook/event parameter, or container elements. For example, ``table[int] of - string &default="foo" }`` would create a table that returns the + string &default="foo"`` would create a table that returns the :bro:type:`string` ``"foo"`` for any non-existing index. .. bro:attr:: &redef @@ -901,7 +915,7 @@ scripting language supports the following built-in attributes. Called right before a container element expires. The function's first parameter is of the same type of the container and the second parameter the same type of the container's index. The return - value is a :bro:type:`interval` indicating the amount of additional + value is an :bro:type:`interval` indicating the amount of additional time to wait before expiring the container element at the given index (which will trigger another execution of this function). @@ -925,7 +939,7 @@ scripting language supports the following built-in attributes. .. bro:attr:: &persistent - Makes a variable persistent, i.e., its value is writen to disk (per + Makes a variable persistent, i.e., its value is written to disk (per default at shutdown time). .. bro:attr:: &synchronized @@ -957,8 +971,9 @@ scripting language supports the following built-in attributes. .. bro:attr:: &priority - Specifies the execution priority of an event handler. Higher values - are executed before lower ones. The default value is 0. + Specifies the execution priority (as a signed integer) of a hook or + event handler. Higher values are executed before lower ones. The + default value is 0. .. bro:attr:: &group From a33d25b3bd2ce7c00b398fb6322e2c7c6abd2fc9 Mon Sep 17 00:00:00 2001 From: Rafael Bonilla Date: Thu, 21 Nov 2013 12:56:00 -0600 Subject: [PATCH 02/56] New Bro Manual Development Edition and basic.css to fix btest output overflow problem (Update 1). --- doc/_static/basic.css | 9 ++ doc/broids/index.rst | 155 ++++++++++++++++++++ doc/httpmonitor/index.rst | 256 ++++++++++++++++++++++++++++++++++ doc/index.rst | 38 ++++- doc/{using => logs}/index.rst | 49 +++++++ doc/mimestats/index.rst | 145 +++++++++++++++++++ 6 files changed, 647 insertions(+), 5 deletions(-) create mode 100644 doc/broids/index.rst create mode 100644 doc/httpmonitor/index.rst rename doc/{using => logs}/index.rst (73%) create mode 100644 doc/mimestats/index.rst diff --git a/doc/_static/basic.css b/doc/_static/basic.css index 1332c7b048..26e3450b65 100644 --- a/doc/_static/basic.css +++ b/doc/_static/basic.css @@ -439,8 +439,17 @@ td.linenos pre { color: #aaa; } +.highlight-guess { + overflow:auto; +} + +.highlight-none { + overflow:auto; +} + table.highlighttable { margin-left: 0.5em; + overflow:scroll; } table.highlighttable td { diff --git a/doc/broids/index.rst b/doc/broids/index.rst new file mode 100644 index 0000000000..dd0a0e8b22 --- /dev/null +++ b/doc/broids/index.rst @@ -0,0 +1,155 @@ +__ http://www.bro.org/sphinx-git/scripts/base/protocols/ftp/main.bro.html#id-FTP::parse_ftp_reply_code +__ http://www.bro.org/sphinx-git/frameworks/sumstats.html +__ http://www.bro.org/sphinx-git/frameworks/notice.html +__ http://www.bro.org/sphinx-git/_downloads/detect-bruteforcing.bro +__ http://www.bro.org/sphinx-git/scripts/policy/frameworks/files/detect-MHR.bro.html + +.. _bro-ids: + +======= +Bro IDS +======= + +An Intrusion Detection System (IDS) allows you to detect suspicious activities happening on your network as a result of a past or active +attack. Because of its programming capabilities, Bro can easily be configured to behave like traditional IDSs and detect common attacks +with well known patterns, or you can create your own scripts to detect conditions specific to your particular case. + +In the following sections, we present a few examples of common uses of Bro as an IDS. + +------------------------------------------------ +Detecting an FTP Bruteforce attack and notifying +------------------------------------------------ +For the purpose of this exercise, we define FTP bruteforcing as too many rejected usernames and passwords occurring from a single address. +We start by defining a threshold for the number of attempts and a monitoring interval in minutes. + + .. code:: bro + + export { + ## How many rejected usernames or passwords are required before being + ## considered to be bruteforcing. + const bruteforce_threshold: double = 20 &redef; + + ## The time period in which the threshold needs to be crossed before + ## being reset. + const bruteforce_measurement_interval = 15mins &redef; + } + +Now, using the ftp_reply event, we check for error codes from the `500 series `_ for the "USER" and "PASS" commands, representing rejected usernames or passwords. For this, we can use the `FTP::parse_ftp_reply`__ function to break down the reply code and check if the first digit is a "5" or not. If true, we then use the `SumStats`__ framework to keep track of the number of failed attempts. + + .. code:: bro + + event ftp_reply(c: connection, code: count, msg: string, cont_resp: bool) + { + local cmd = c$ftp$cmdarg$cmd; + if ( cmd == "USER" || cmd == "PASS" ) + { + if ( FTP::parse_ftp_reply_code(code)$x == 5 ) + SumStats::observe("ftp.failed_auth", [$host=c$id$orig_h], [$str=cat(c$id$resp_h)]); + } + } + +Next, we use the SumStats framework to automatically print a message on the console alerting of the attack when the number of failed attempts +exceeds the specified threshold during the measuring interval. + + .. code:: bro + + event bro_init() + { + local r1: SumStats::Reducer = [$stream="ftp.failed_auth", $apply=set(SumStats::UNIQUE), $unique_max=double_to_count(bruteforce_threshold+2)]; + SumStats::create([$name="ftp-detect-bruteforcing", + $epoch=bruteforce_measurement_interval, + $reducers=set(r1), + $threshold_val(key: SumStats::Key, result: SumStats::Result) = + { + return result["ftp.failed_auth"]$num+0.0; + }, + $threshold=bruteforce_threshold, + $threshold_crossed(key: SumStats::Key, result: SumStats::Result) = + { + local r = result["ftp.failed_auth"]; + local dur = duration_to_mins_secs(r$end-r$begin); + local plural = r$unique>1 ? "s" : ""; + local message = fmt("%s had %d failed logins on %d FTP server%s in %s", key$host, r$num, r$unique, plural, dur); + }]); + } + +Printing a message on the console is a good start but it will be better if we raise an alarm instead using the `Notice`__ framework. For this, we need to define a new Notice type and trigger the alarm under the right +conditions. Below is the final code for our script. + + .. code:: bro + + ##! FTP brute-forcing detector, triggering when too many rejected usernames or + ##! failed passwords have occurred from a single address. + + @load base/protocols/ftp + @load base/frameworks/sumstats + + @load base/utils/time + + module FTP; + + export { + redef enum Notice::Type += { + ## Indicates a host bruteforcing FTP logins by watching for too + ## many rejected usernames or failed passwords. + Bruteforcing + }; + + ## How many rejected usernames or passwords are required before being + ## considered to be bruteforcing. + const bruteforce_threshold: double = 20 &redef; + + ## The time period in which the threshold needs to be crossed before + ## being reset. + const bruteforce_measurement_interval = 15mins &redef; + } + + + event bro_init() + { + local r1: SumStats::Reducer = [$stream="ftp.failed_auth", $apply=set(SumStats::UNIQUE), $unique_max=double_to_count(bruteforce_threshold+2)]; + SumStats::create([$name="ftp-detect-bruteforcing", + $epoch=bruteforce_measurement_interval, + $reducers=set(r1), + $threshold_val(key: SumStats::Key, result: SumStats::Result) = + { + return result["ftp.failed_auth"]$num+0.0; + }, + $threshold=bruteforce_threshold, + $threshold_crossed(key: SumStats::Key, result: SumStats::Result) = + { + local r = result["ftp.failed_auth"]; + local dur = duration_to_mins_secs(r$end-r$begin); + local plural = r$unique>1 ? "s" : ""; + local message = fmt("%s had %d failed logins on %d FTP server%s in %s", key$host, r$num, r$unique, plural, dur); + NOTICE([$note=FTP::Bruteforcing, + $src=key$host, + $msg=message, + $identifier=cat(key$host)]); + }]); + } + + event ftp_reply(c: connection, code: count, msg: string, cont_resp: bool) + { + local cmd = c$ftp$cmdarg$cmd; + if ( cmd == "USER" || cmd == "PASS" ) + { + if ( FTP::parse_ftp_reply_code(code)$x == 5 ) + SumStats::observe("ftp.failed_auth", [$host=c$id$orig_h], [$str=cat(c$id$resp_h)]); + } + } + +As a final note, the `detect-bruteforcing.bro`__ script above is include with Bro out of the box, so you only need to load it at startup to instruct Bro to detect and notify of FTP bruteforce attacks. + +------------- +Other Attacks +------------- +Detecting SQL Injection attacks +------------------------------- +Checking files against known malware hashes +------------------------------------------- +Files transmitted on your network could either be completely harmless or contain viruses and other threats. One possible action against +this threat is to compute the hashes of the files and compare them against a list of known malware hashes. Bro simplifies this task +by offering a `detect-MHR.bro`__ script that creates and compares +hashes against the `Malware Hash Registry `_ maintained by Team Cymru. You only need to load this +script along with your other scripts at startup time. diff --git a/doc/httpmonitor/index.rst b/doc/httpmonitor/index.rst new file mode 100644 index 0000000000..41f9dd955c --- /dev/null +++ b/doc/httpmonitor/index.rst @@ -0,0 +1,256 @@ +__ http://www.bro.org/sphinx-git/scripts/base/protocols/http/main.bro.html +__ http://www.bro.org/sphinx-git/frameworks/file-analysis.html + +.. _http-monitor: + +================================ +Monitoring HTTP Traffic with Bro +================================ + +Bro can be used to log the entire HTTP traffic from your network to the http.log file. +This file can then be used for analysis and auditing purposes. + +In the sections below we briefly explain the structure of the http.log file. Then, we +show you how to perform basic HTTP traffic monitoring and analysis tasks with Bro. Some +of these ideas and techniques can later be applied to monitor different protocols in a +similar way. + +---------------------------- +Introduction to the HTTP log +---------------------------- + +The http.log file contains a summary of all HTTP requests and responses sent over a Bro-monitored +network. Here are the first few columns of +``http.log``:: + + # ts uid orig_h orig_p resp_h resp_p + 1311627961.8 HSH4uV8KVJg 192.168.1.100 52303 192.150.187.43 80 + +Every single line in this log starts with a timestamp, a unique connection identifier (UID), and a +connection 4-tuple (originator host/port and responder host/port). The UID can be used to +identify all logged activity (possibly across multiple log files) associated +with a given connection 4-tuple over its lifetime. + +The remaining columns detail the activity that's occurring. For example, the columns on the line below +(shortened for brevity) show a request to the root of Bro website:: + + # method host uri referrer user_agent + GET bro.org / - <...>Chrome/12.0.742.122<...> + +Network administrators and security engineers, for instance, can use the information in this log to understand +the HTTP activity on the network and troubleshoot network problems or search for anomalous activities. At this +point, we would like to stress out the fact that there is no just one right way to perform analysis; it will +depend on the expertise of the person doing the analysis and the specific details of the task to accomplish. + +For more information about how to handle the HTTP protocol in Bro, including a complete list +of the fields available in http.log, go to Bro's HTTP reference `page`__. + +------------------------ +Detecting a Proxy Server +------------------------ + +A proxy server is a device on your network configured to request a service on behalf of a third system; one of the +most common examples is a Web proxy server. A client without Internet access connects to the proxy and requests +a Web page; the proxy then sends the request to the actual Web server, receives the response and passes it to the original +client. + +Proxies were conceived to help manage a network and provide better encapsulation. By themselves, proxies are not a security +threat, but a misconfigured or unauthorized proxy can allow others, either inside or outside the network, to access any +Web site and even conduct malicious activities anonymously using the network resources. + +What Proxy Server traffic looks like +------------------------------------- + +In general, when a client starts talking with a proxy server, the traffic consists of two parts: (i) a GET request, and +(ii) an HTTP/ reply:: + + Request: GET http://www.bro.org/ HTTP/1.1 + Reply: HTTP/1.0 200 OK + +This will differ from traffic between a client and a normal Web server because GET requests should not include "http" on +the string. So we can use this to identify a proxy server. + +We can write a basic script in Bro to handle the http_reply event and detect a reply for a ``GET http://`` request. + + .. code:: bro + + event http_reply(c: connection, version: string, code: count, reason: string) + { + if ( /^[hH][tT][tT][pP]:/ in c$http$uri && c$http$status_code == 200 ) + { + print fmt("A local server is acting as an open proxy: ", c$id$resp_h); + } + } + +Basically, the script is checking for a "200 OK" status code on a reply for a request that includes "http:". In reality, the HTTP +protocol defines several success status codes other than 200, so we will extend our basic script to also consider the additional codes. + + .. code:: bro + + export { + + global success_status_codes: set[count] = { + 200, + 201, + 202, + 203, + 204, + 205, + 206, + 207, + 208, + 226, + 304 + }; + + } + + event http_reply(c: connection, version: string, code: count, reason: string) + { + if ( /^[hH][tT][tT][pP]:/ in c$http$uri && c$http$status_code in success_status_codes ) + { + print fmt("A local server is acting as an open proxy: ", c$id$resp_h); + } + } + +Next, we will make sure that the responding proxy is part of our local network. + + .. code:: bro + + export { + + global success_status_codes: set[count] = { + 200, + 201, + 202, + 203, + 204, + 205, + 206, + 207, + 208, + 226, + 304 + }; + + } + + event http_reply(c: connection, version: string, code: count, reason: string) + { + if ( Site::is_local_addr(c$id$resp_h) && /^[hH][tT][tT][pP]:/ in c$http$uri && c$http$status_code in success_status_codes ) + { + print fmt("A local server is acting as an open proxy: ", c$id$resp_h); + } + } + +Finally, our goal should be to generate an alert when a proxy has been detected instead of printing a message on the console output. +For that, we will tag the traffic accordingly and define a new ``Open_Proxy`` ``Notice`` type to alert of all tagged communications. Once a +notification has been fired, we will further suppress it for one day. Below is the complete script. + + .. code:: bro + + @load base/frameworks/notice + + module HTTP; + + export { + + redef enum HTTP::Tags += { + OPEN_PROXY_TAG + }; + redef enum Notice::Type += { + Open_Proxy + }; + + global success_status_codes: set[count] = { + 200, + 201, + 202, + 203, + 204, + 205, + 206, + 207, + 208, + 226, + 304 + }; + + } + + redef Notice::emailed_types += { + Open_Proxy, + }; + + function open_proxy_only(rec: HTTP::Info) : bool + { + # Only write out connections with the OPEN_PROXY_TAG. + return OPEN_PROXY_TAG in rec$tags; + } + + event http_reply(c: connection, version: string, code: count, reason: string) + { + # make sure responding host is local + #if ( Site::is_local_addr(c$id$resp_h) && /^[hH][tT][tT][pP]:/ in c$http$uri && c$http$status_code in success_status_codes ) + { + add c$http$tags[OPEN_PROXY_TAG]; + local ident = cat(c$id$resp_h); + if ( c$http?$host ) #check if the optional host field exists in http + { + print fmt("Originator host: %s", c$id$orig_h); + NOTICE([$note=HTTP::Open_Proxy, + $msg=cat("A local server is acting as an open proxy: ", c$id$resp_h), + $conn=c, $identifier=cat(ident, c$id$resp_h), + $suppress_for=1day]); + } + } + } + + event bro_init() + { + #Creating a new filter for all open proxy logs. + local filter: Log::Filter = [$name="open_proxy", $path="open_proxy", $pred=open_proxy_only]; + Log::add_filter(HTTP::LOG, filter); + } + +---------------- +Inspecting Files +---------------- + +Files are often transmitted on regular HTTP conversations between a client and a server. Most of the time these files are harmless, +just images and some other multimedia content, but there are also types of files, specially executable files, that can damage +your system. We can instruct Bro to create a copy of all executable files that it sees for later analysis using the `File Analysis +Framework`__ (introduced with Bro 2.2) as shown in the following script. + + .. code:: bro + + global ext_map: table[string] of string = { + ["application/x-dosexec"] = "exe", + } &default =""; + + event file_new(f: fa_file) + { + local ext = ""; + + if ( f?$mime_type ) + ext = ext_map[f$mime_type]; + + local fname = fmt("%s-%s.%s", f$source, f$id, ext); + Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=fname]); + } + +Bro will extract all files from the traffic and write them on a new ``extract_files/`` subdirectory and change the file name with the right +suffix (extension) based on the content of the ext_map table. So, if you want to do the same for other extracted files besides executables +you just need to add those types to the ``ext_map`` table like this. + + .. code:: bro + + global ext_map: table[string] of string = { + ["application/x-dosexec"] = "exe", + ["text/plain"] = "txt", + ["image/jpeg"] = "jpg", + ["image/png"] = "png", + ["text/html"] = "html", + } &default =""; + +Bro will now write the appropriate suffix for text, JPEG, PNG, and HTML files stored in the ``extract_files/`` subdirectory. diff --git a/doc/index.rst b/doc/index.rst index 34096694b3..98006034c2 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -1,9 +1,12 @@ .. Bro documentation master file -================= -Bro Documentation -================= +========== +Bro Manual +========== + +Introduction Section +==================== .. toctree:: :maxdepth: 2 @@ -11,13 +14,38 @@ Bro Documentation intro/index.rst install/index.rst quickstart/index.rst - using/index.rst + +.. + +Using Bro Section +================= + +.. toctree:: + :maxdepth: 2 + + logs/index.rst + httpmonitor/index.rst + broids/index.rst + mimestats/index.rst + cluster/index.rst + +.. + +Reference Section +================= + +.. toctree:: + :maxdepth: 2 + scripting/index.rst frameworks/index.rst - cluster/index.rst script-reference/index.rst components/index.rst +.. + +* `Notice Index `_ (TODO: Move to reference + section, but can't figure out how to include it into toctree) * :ref:`General Index ` * :ref:`search` diff --git a/doc/using/index.rst b/doc/logs/index.rst similarity index 73% rename from doc/using/index.rst rename to doc/logs/index.rst index 1ad05d74f8..a6844d6e03 100644 --- a/doc/using/index.rst +++ b/doc/logs/index.rst @@ -1,3 +1,13 @@ +__ http://www.bro.org/sphinx-git/scripts/base/protocols/http/main.bro.html#type-HTTP::Info +__ http://www.bro.org/sphinx-git/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info +__ http://www.bro.org/sphinx-git/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info +__ http://www.bro.org/sphinx-git/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo +__ http://www.bro.org/sphinx-git/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info +__ http://www.bro.org/sphinx-git/scripts/base/protocols/dns/main.bro.html#type-DNS::Info +__ http://www.bro.org/sphinx-git/scripts/base/protocols/conn/main.bro.html#type-Conn::Info +__ http://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info +__ http://www.bro.org/sphinx-git/scripts/base/frameworks/files/main.bro.html#type-Files::Info +__ http://www.bro.org/sphinx-git/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info .. _using-bro: @@ -251,3 +261,42 @@ stream and Bro is able to extract and track that information for you, giving you an in-depth and structured view into HTTP traffic on your network. +----------------------- +Common Log Files +----------------------- +As a monitoring tool, Bro records a detailed view of the traffic inspected and the events generated in +a series of relevant log files. These files can later be reviewed for monitoring, auditing and troubleshooting +purposes. + +In this section we present a brief explanation of the most commonly used log files generated by Bro including links +to descriptions of some of the fields for each log type. + ++-----------------+---------------------------------------+------------------------------+ +| Log File | Description | Field Descriptions | ++=================+=======================================+==============================+ +| http.log | Shows all HTTP requests and replies | :bro:type:`HTTP::Info` | ++-----------------+---------------------------------------+------------------------------+ +| ftp.log | Records FTP activity | :bro:type:`FTP::Info` | ++-----------------+---------------------------------------+------------------------------+ +| ssl.log | Records SSL sessions including | :bro:type:`SSL::Info` | +| | certificates used | | ++-----------------+---------------------------------------+------------------------------+ +| known_certs.log | Includes SSL certificates used | :bro:type:`Known::CertsInfo` | ++-----------------+---------------------------------------+------------------------------+ +| smtp.log | Summarizes SMTP traffic on a network | :bro:type:`SMTP::Info` | ++-----------------+---------------------------------------+------------------------------+ +| dns.log | Shows all DNS activity on a network | :bro:type:`DNS::Info` | ++-----------------+---------------------------------------+------------------------------+ +| conn.log | Records all connections seen by Bro | :bro:type:`Conn::Info` | ++-----------------+---------------------------------------+------------------------------+ +| dpd.log | Shows network activity on | :bro:type:`DPD::Info` | +| | non-standard ports | | ++-----------------+---------------------------------------+------------------------------+ +| files.log | Records information about all files | :bro:type:`Files::Info` | +| | transmitted over the network | | ++-----------------+---------------------------------------+------------------------------+ +| weird.log | Records unexpected protocol-level | :bro:type:`Weird::Info` | +| | activity | | ++-----------------+---------------------------------------+------------------------------+ + + diff --git a/doc/mimestats/index.rst b/doc/mimestats/index.rst new file mode 100644 index 0000000000..903131acd5 --- /dev/null +++ b/doc/mimestats/index.rst @@ -0,0 +1,145 @@ +__ http://www.bro.org/sphinx-git/frameworks/sumstats.html + +.. _mime-stats: + +==================== +MIME Type Statistics +==================== + +Files are constantly transmitted over HTTP on regular networks. These files belong to a specific category (i.e., executable, text, image, etc.) identified +by a `Multipurpose Internet Mail Extension (MIME) `_. Although MIME was originally developed to identify the type of +non-text attachments on email, it is also used by Web browser to identify the type of files transmitted and present them accordingly. + +In this tutorial, we will show how to use the Sumstats Framework to collect some statistics information based on MIME types, specifically the total number of +occurrences, size in bytes, and number of unique hosts transmitting files over HTTP per each type. For instructions about extracting and creating a local copy +of these files, visit `this <../httpmonitor/index.html#inspecting-files>`_ tutorial instead. + +------------------------------------------------ +MIME Statistics with Sumstats +------------------------------------------------ +When working with the `Sumstats`__ framework, you need to define three different pieces: (i) Observations, where +the event is observed and fed into the framework. (ii) Reducers, where observations are collected and measured. (iii) Sumstats, where the main functionality +is implemented. + +So, we start by defining our observation along with a record to store all statistics values and an observation interval. We are conducting our observation on +the `HTTP::log_http` event and we are interested in the MIME type, size of the file ("response_body_len") and the originator host ("orig_h"). We use the MIME +type as our key and create observers for the other two values. + + .. code:: bro + + export { + redef enum Log::ID += { LOG }; + type Info: record { + ## Timestamp when the log line was finished and written. + ts: time &log; + ## Time interval that the log line covers. + ts_delta: interval &log; + ## The mime type + mtype: string &log; + ## The number of unique local hosts that fetched this mime type + uniq_hosts: count &log; + ## The number of hits to the mime type + hits: count &log; + ## The total number of bytes received by this mime type + bytes: count &log; + }; + + ## The frequency of logging the stats collected by this script. + const break_interval = 5mins &redef; + } + + event HTTP::log_http(rec: HTTP::Info) + { + if(Site::is_local_addr(rec$id$orig_h) && rec?$resp_mime_types) { + local mime_type = rec$resp_mime_types[0]; + SumStats::observe("mime.bytes", [$str=mime_type], [$num=rec$response_body_len]); + SumStats::observe("mime.hits", [$str=mime_type], [$str=cat(rec$id$orig_h)]); + } + } + +Next, we create the reducers. The first one will accumulate file sizes and the second one will make sure we only store a host ID once. Below is the partial code. + + .. code:: bro + + local r1: SumStats::Reducer = [$stream="mime.bytes", $apply=set(SumStats::SUM)]; + local r2: SumStats::Reducer = [$stream="mime.hits", $apply=set(SumStats::UNIQUE)]; + +In our final step, we create the SumStats where we check for the observation interval and once it expires, we populate the record (defined above) with all the +relevant data and write it to a log. + + .. code:: bro + + SumStats::create([$name="mime-metrics", + $epoch=break_interval, + $reducers=set(r1, r2), + $epoch_result(ts: time, key: SumStats::Key, result: SumStats::Result) = + { + local l: Info; + l$ts = network_time(); + l$ts_delta = break_interval; + l$mtype = key$str; + l$bytes = double_to_count(floor(result["mime.bytes"]$sum)); + l$hits = result["mime.hits"]$num; + l$uniq_hosts = result["mime.hits"]$unique; + Log::write(LOG, l); + }]); + +Putting everything together we end up with the following final code for our script. + + .. code:: bro + + @load base/frameworks/sumstats + + module MimeMetrics; + + export { + redef enum Log::ID += { LOG }; + type Info: record { + ## Timestamp when the log line was finished and written. + ts: time &log; + ## Time interval that the log line covers. + ts_delta: interval &log; + ## The mime type + mtype: string &log; + ## The number of unique local hosts that fetched this mime type + uniq_hosts: count &log; + ## The number of hits to the mime type + hits: count &log; + ## The total number of bytes received by this mime type + bytes: count &log; + }; + + ## The frequency of logging the stats collected by this script. + const break_interval = 5mins &redef; + } + + event bro_init() &priority=3 + { + Log::create_stream(MimeMetrics::LOG, [$columns=Info]); + local r1: SumStats::Reducer = [$stream="mime.bytes", $apply=set(SumStats::SUM)]; + local r2: SumStats::Reducer = [$stream="mime.hits", $apply=set(SumStats::UNIQUE)]; + SumStats::create([$name="mime-metrics", + $epoch=break_interval, + $reducers=set(r1, r2), + $epoch_result(ts: time, key: SumStats::Key, result: SumStats::Result) = + { + local l: Info; + l$ts = network_time(); + l$ts_delta = break_interval; + l$mtype = key$str; + l$bytes = double_to_count(floor(result["mime.bytes"]$sum)); + l$hits = result["mime.hits"]$num; + l$uniq_hosts = result["mime.hits"]$unique; + Log::write(LOG, l); + }]); + } + + event HTTP::log_http(rec: HTTP::Info) + { + if(Site::is_local_addr(rec$id$orig_h) && rec?$resp_mime_types) { + local mime_type = rec$resp_mime_types[0]; + SumStats::observe("mime.bytes", [$str=mime_type], [$num=rec$response_body_len]); + SumStats::observe("mime.hits", [$str=mime_type], [$str=cat(rec$id$orig_h)]); + } + } + From 6f06705c23cb52aa059ae98d770ecc1cb29a80a2 Mon Sep 17 00:00:00 2001 From: Daniel Thayer Date: Fri, 22 Nov 2013 14:49:16 -0600 Subject: [PATCH 03/56] Fix typos in BIF documentation Fixed typos in documentation of hexstr_to_bytestring. Also added documentation that was missing for function parameters and return values of other BIFs. --- src/bro.bif | 20 +++++++++++--------- src/probabilistic/top-k.bif | 4 ++++ 2 files changed, 15 insertions(+), 9 deletions(-) diff --git a/src/bro.bif b/src/bro.bif index 24dff3c77c..d789ef9f4e 100644 --- a/src/bro.bif +++ b/src/bro.bif @@ -973,6 +973,8 @@ function entropy_test_finish%(handle: opaque of entropy%): entropy_test_result ## ## prefix: A custom string prepended to the result. ## +## Returns: A string identifier that is unique. +## ## .. bro:see:: unique_id_from function unique_id%(prefix: string%) : string %{ @@ -987,6 +989,8 @@ function unique_id%(prefix: string%) : string ## ## prefix: A custom string prepended to the result. ## +## Returns: A string identifier that is unique. +## ## .. bro:see:: unique_id function unique_id_from%(pool: int, prefix: string%) : string %{ @@ -1857,8 +1861,8 @@ function global_ids%(%): id_table ## ## id: The global identifier. ## -## Returns the value of *id*. If *id* does not describe a valid identifier, the -## function returns the string ``""`` or ``""``. +## Returns: The value of *id*. If *id* does not describe a valid identifier, +## the string ``""`` or ``""`` is returned. function lookup_ID%(id: string%) : any %{ ID* i = global_scope()->Lookup(id->CheckString()); @@ -2632,15 +2636,15 @@ function bytestring_to_hexstr%(bytestring: string%): string return new StringVal(hexstr); %} -## Converts a hex-string into its binary representation. +## Converts a hex-string into its binary representation. ## For example, ``"3034"`` would be converted to ``"04"``. ## -## The input string is assumed to only contain 0-9 and a-f, -## otherwhise behavior is undefines. +## The input string is assumed to contain an even number of hexadecimal digits +## (0-9, a-f, or A-F), otherwise behavior is undefined. ## -## hexstring: The hexadecimal string representation. +## hexstr: The hexadecimal string representation. ## -## Returns: The binary representation of *hexstring*. +## Returns: The binary representation of *hexstr*. ## ## .. bro:see:: hexdump bytestring_to_hexstr function hexstr_to_bytestring%(hexstr: string%): string @@ -4964,8 +4968,6 @@ function preserve_prefix%(a: addr, width: count%): any ## ## a: The subnet to preserve. ## -## width: The number of bits from the top that should remain intact. -## ## .. bro:see:: preserve_prefix anonymize_addr ## ## .. todo:: Currently dysfunctional. diff --git a/src/probabilistic/top-k.bif b/src/probabilistic/top-k.bif index e3dc891552..5362750467 100644 --- a/src/probabilistic/top-k.bif +++ b/src/probabilistic/top-k.bif @@ -138,6 +138,10 @@ function topk_sum%(handle: opaque of topk%): count ## Merge the second top-k data structure into the first. ## +## handle1: the first TopK handle. +## +## handle2: the second TopK handle. +## ## .. note:: This does not remove any elements, the resulting data structure ## can be bigger than the maximum size given on initialization. ## From 5b6468a302e275912418ac22da1a7d5cf9904eeb Mon Sep 17 00:00:00 2001 From: Daniel Thayer Date: Fri, 22 Nov 2013 16:36:08 -0600 Subject: [PATCH 04/56] Add documentation for event parameters Added documentation that was missing for some event parameters, and fixed documented name of event parameters. --- src/analyzer/protocol/dnp3/events.bif | 2 ++ src/analyzer/protocol/modbus/events.bif | 2 +- src/analyzer/protocol/tcp/events.bif | 4 ++-- src/event.bif | 6 +++++- src/file_analysis/analyzer/unified2/events.bif | 10 ++++++++++ 5 files changed, 20 insertions(+), 4 deletions(-) diff --git a/src/analyzer/protocol/dnp3/events.bif b/src/analyzer/protocol/dnp3/events.bif index a41f70897b..80f9504a9e 100644 --- a/src/analyzer/protocol/dnp3/events.bif +++ b/src/analyzer/protocol/dnp3/events.bif @@ -32,6 +32,8 @@ event dnp3_application_response_header%(c: connection, is_orig: bool, fc: count, ## ## qua_field: qualifier field. ## +## number: TODO. +## ## rf_low: the structure of the range field depends on the qualified field. ## In some cases, the range field contains only one logic part, e.g., ## number of objects, so only *rf_low* contains useful values. diff --git a/src/analyzer/protocol/modbus/events.bif b/src/analyzer/protocol/modbus/events.bif index dbbd7b78bb..537820f37d 100644 --- a/src/analyzer/protocol/modbus/events.bif +++ b/src/analyzer/protocol/modbus/events.bif @@ -149,7 +149,7 @@ event modbus_write_single_register_response%(c: connection, headers: ModbusHeade ## ## start_address: The memory address of the first coil to be written. ## -## value: The values to be written to the coils. +## coils: The values to be written to the coils. event modbus_write_multiple_coils_request%(c: connection, headers: ModbusHeaders, start_address: count, coils: ModbusCoils%); ## Generated for a Modbus write multiple coils response. diff --git a/src/analyzer/protocol/tcp/events.bif b/src/analyzer/protocol/tcp/events.bif index 216691dea1..cac18bfa3e 100644 --- a/src/analyzer/protocol/tcp/events.bif +++ b/src/analyzer/protocol/tcp/events.bif @@ -108,6 +108,8 @@ event connection_half_finished%(c: connection%); ## originator attempted to setup a TCP connection but the responder replied ## with a RST packet denying it. ## +## c: The connection. +## ## .. bro:see:: connection_EOF connection_SYN_packet connection_attempt ## connection_established connection_external connection_finished ## connection_first_ACK connection_half_finished connection_partial_close @@ -115,8 +117,6 @@ event connection_half_finished%(c: connection%); ## connection_status_update connection_timeout scheduled_analyzer_applied ## new_connection new_connection_contents partial_connection ## -## c: The connection. -## ## .. note:: ## ## If the responder does not respond at all, :bro:id:`connection_attempt` is diff --git a/src/event.bif b/src/event.bif index ddadb47f8a..bb0e98c7d6 100644 --- a/src/event.bif +++ b/src/event.bif @@ -591,7 +591,11 @@ event software_unparsed_version_found%(c: connection, host: addr, str: string%); ## and it raises this event for each system identified. The p0f fingerprints are ## defined by :bro:id:`passive_fingerprint_file`. ## -## TODO. +## c: The connection. +## +## host: The host running the reported OS. +## +## OS: The OS version string. ## ## .. bro:see:: passive_fingerprint_file software_parse_error ## software_version_found software_unparsed_version_found diff --git a/src/file_analysis/analyzer/unified2/events.bif b/src/file_analysis/analyzer/unified2/events.bif index c5f3dda6a4..a9134e5285 100644 --- a/src/file_analysis/analyzer/unified2/events.bif +++ b/src/file_analysis/analyzer/unified2/events.bif @@ -1,7 +1,17 @@ ## Abstract all of the various Unified2 event formats into ## a single event. +## +## f: The file. +## +## ev: TODO. +## event unified2_event%(f: fa_file, ev: Unified2::IDSEvent%); ## The Unified2 packet format event. +## +## f: The file. +## +## pkt: TODO. +## event unified2_packet%(f: fa_file, pkt: Unified2::Packet%); From b6c1b35bb80cb09fe0d0fefbb626358a30177ae3 Mon Sep 17 00:00:00 2001 From: Robin Sommer Date: Tue, 26 Nov 2013 10:35:04 -0800 Subject: [PATCH 05/56] Updating submodule(s). [nomail] --- aux/binpac | 2 +- aux/bro-aux | 2 +- aux/broccoli | 2 +- aux/broctl | 2 +- aux/btest | 2 +- cmake | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/aux/binpac b/aux/binpac index 54b321009b..896ddedde5 160000 --- a/aux/binpac +++ b/aux/binpac @@ -1 +1 @@ -Subproject commit 54b321009b750268526419bdbd841f421c839313 +Subproject commit 896ddedde55c48ec2163577fc258b49c418abb3e diff --git a/aux/bro-aux b/aux/bro-aux index ebf9c0d88a..6c5db7af6e 160000 --- a/aux/bro-aux +++ b/aux/bro-aux @@ -1 +1 @@ -Subproject commit ebf9c0d88ae8230845b91f15755156f93ff21aa8 +Subproject commit 6c5db7af6e93406be498b24f8a2d67ca281c1a03 diff --git a/aux/broccoli b/aux/broccoli index 17ec437752..889f9c6594 160000 --- a/aux/broccoli +++ b/aux/broccoli @@ -1 +1 @@ -Subproject commit 17ec437752837fb4214abfb0a2da49df74668d5d +Subproject commit 889f9c65944ceac20ad9230efc39d33e6e1221c3 diff --git a/aux/broctl b/aux/broctl index 6e01d6972f..0cd102805e 160000 --- a/aux/broctl +++ b/aux/broctl @@ -1 +1 @@ -Subproject commit 6e01d6972f02d68ee82d05f392d1a00725595b7f +Subproject commit 0cd102805e73343cab3f9fd4a76552e13940dad9 diff --git a/aux/btest b/aux/btest index 26c3136d56..ce366206e3 160000 --- a/aux/btest +++ b/aux/btest @@ -1 +1 @@ -Subproject commit 26c3136d56493017bc33c5a2f22ae393d585c2d9 +Subproject commit ce366206e3407e534a786ad572c342e9f9fef26b diff --git a/cmake b/cmake index e7a46cb82e..0187b33a29 160000 --- a/cmake +++ b/cmake @@ -1 +1 @@ -Subproject commit e7a46cb82ee10aa522c4d88115baf10181277d20 +Subproject commit 0187b33a29d5ec824f940feff60dc5d8c2fe314f From f0fe27002903ea7c9a17141e27b059bcc87b255d Mon Sep 17 00:00:00 2001 From: Robin Sommer Date: Tue, 26 Nov 2013 10:57:02 -0800 Subject: [PATCH 06/56] Minor interface changes to provide more accessor methods for class information. In particular, adding a few const versions of methods. --- src/Event.h | 2 ++ src/ID.h | 2 ++ src/Type.cc | 28 ++++++++++++++++++++++++---- src/Type.h | 13 +++++++++++-- 4 files changed, 39 insertions(+), 6 deletions(-) diff --git a/src/Event.h b/src/Event.h index 9d0a707cda..6f9c9d10c3 100644 --- a/src/Event.h +++ b/src/Event.h @@ -24,6 +24,8 @@ public: SourceID Source() const { return src; } analyzer::ID Analyzer() const { return aid; } TimerMgr* Mgr() const { return mgr; } + EventHandlerPtr Handler() const { return handler; } + val_list* Args() const { return args; } void Describe(ODesc* d) const; diff --git a/src/ID.h b/src/ID.h index 57e1222511..5b71830640 100644 --- a/src/ID.h +++ b/src/ID.h @@ -32,9 +32,11 @@ public: void SetType(BroType* t) { Unref(type); type = t; } BroType* Type() { return type; } + const BroType* Type() const { return type; } void MakeType() { is_type = 1; } BroType* AsType() { return is_type ? Type() : 0; } + const BroType* AsType() const { return is_type ? Type() : 0; } // If weak_ref is false, the Val is assumed to be already ref'ed // and will be deref'ed when the ID is deleted. diff --git a/src/Type.cc b/src/Type.cc index a6d8b90c6c..6b4b2eb970 100644 --- a/src/Type.cc +++ b/src/Type.cc @@ -430,6 +430,11 @@ BroType* IndexType::YieldType() return yield_type; } +const BroType* IndexType::YieldType() const + { + return yield_type; + } + void IndexType::Describe(ODesc* d) const { BroType::Describe(d); @@ -723,6 +728,11 @@ BroType* FuncType::YieldType() return yield; } +const BroType* FuncType::YieldType() const + { + return yield; + } + int FuncType::MatchesIndex(ListExpr*& index) const { return check_and_promote_args(index, args) ? @@ -1444,9 +1454,9 @@ void CommentedEnumType::AddNameInternal(const string& module_name, const char* n names[copy_string(fullname.c_str())] = val; } -bro_int_t EnumType::Lookup(const string& module_name, const char* name) +bro_int_t EnumType::Lookup(const string& module_name, const char* name) const { - NameMap::iterator pos = + NameMap::const_iterator pos = names.find(make_full_var_name(module_name.c_str(), name).c_str()); if ( pos == names.end() ) @@ -1455,9 +1465,9 @@ bro_int_t EnumType::Lookup(const string& module_name, const char* name) return pos->second; } -const char* EnumType::Lookup(bro_int_t value) +const char* EnumType::Lookup(bro_int_t value) const { - for ( NameMap::iterator iter = names.begin(); + for ( NameMap::const_iterator iter = names.begin(); iter != names.end(); ++iter ) if ( iter->second == value ) return iter->first; @@ -1465,6 +1475,16 @@ const char* EnumType::Lookup(bro_int_t value) return 0; } +EnumType::enum_name_list EnumType::Names() const + { + enum_name_list n; + for ( NameMap::const_iterator iter = names.begin(); + iter != names.end(); ++iter ) + n.push_back(std::make_pair(iter->first, iter->second)); + + return n; + } + void EnumType::DescribeReST(ODesc* d) const { d->Add(":bro:type:`"); diff --git a/src/Type.h b/src/Type.h index a6163d5152..6c162bdabb 100644 --- a/src/Type.h +++ b/src/Type.h @@ -304,6 +304,7 @@ public: TypeList* Indices() const { return indices; } const type_list* IndexTypes() const { return indices->Types(); } BroType* YieldType(); + const BroType* YieldType() const; void Describe(ODesc* d) const; void DescribeReST(ODesc* d) const; @@ -366,6 +367,7 @@ public: RecordType* Args() const { return args; } BroType* YieldType(); + const BroType* YieldType() const; void SetYieldType(BroType* arg_yield) { yield = arg_yield; } function_flavor Flavor() const { return flavor; } string FlavorString() const; @@ -522,6 +524,8 @@ protected: class EnumType : public BroType { public: + typedef std::list > enum_name_list; + EnumType(const string& arg_name); EnumType(EnumType* e); ~EnumType(); @@ -536,11 +540,15 @@ public: void AddName(const string& module_name, const char* name, bro_int_t val, bool is_export); // -1 indicates not found. - bro_int_t Lookup(const string& module_name, const char* name); - const char* Lookup(bro_int_t value); // Returns 0 if not found + bro_int_t Lookup(const string& module_name, const char* name) const; + const char* Lookup(bro_int_t value) const; // Returns 0 if not found string Name() const { return name; } + // Returns the list of defined names with their values. The names + // will be fully qualified with their module name. + enum_name_list Names() const; + void DescribeReST(ODesc* d) const; protected: @@ -592,6 +600,7 @@ public: VectorType(BroType* t); virtual ~VectorType(); BroType* YieldType() { return yield_type; } + const BroType* YieldType() const { return yield_type; } int MatchesIndex(ListExpr*& index) const; From d34f23c8d4153417d107db82167095f63dec9457 Mon Sep 17 00:00:00 2001 From: Robin Sommer Date: Tue, 26 Nov 2013 11:16:58 -0800 Subject: [PATCH 07/56] A set of file analysis extensions. - Enable manager to associate analyzers with a MIME type. With that, one can now say enable all analyzers for, e.g., "image/gif". This is exposed to script-land as Files::add_analyzers_for_mime_type(f: fa_file, mtype: string) For MIME types identified via libmagic, this happens automatically (via the file_new() handler in files/main.bro). - Extend the analyzer API to better match that of protocol analyzers: - Adding unique analyzer IDs so that we can refer to instances from script-land. - Adding subtypes to Components so that a single analyzer implementation can support different types of analyzers internally. - Add an analyzer method SetTag() that allows to set the tag after construction. - Adding Init() and Done() methods for consistency with what other classes offer. - Add debug logging to the file_analysis stream. TODO: test cases missing for the new script-land functionality. --- scripts/base/frameworks/files/main.bro | 39 ++++++--- scripts/base/init-bare.bro | 7 ++ src/file_analysis/Analyzer.cc | 8 ++ src/file_analysis/Analyzer.h | 54 ++++++++++++- src/file_analysis/AnalyzerSet.cc | 10 ++- src/file_analysis/Component.cc | 8 +- src/file_analysis/Component.h | 8 +- src/file_analysis/File.cc | 42 +++++++--- src/file_analysis/File.h | 2 +- src/file_analysis/Manager.cc | 105 +++++++++++++++++++++++-- src/file_analysis/Manager.h | 74 +++++++++++++++++ src/file_analysis/file_analysis.bif | 10 +++ 12 files changed, 337 insertions(+), 30 deletions(-) diff --git a/scripts/base/frameworks/files/main.bro b/scripts/base/frameworks/files/main.bro index cf2c11be45..7fecec3034 100644 --- a/scripts/base/frameworks/files/main.bro +++ b/scripts/base/frameworks/files/main.bro @@ -56,7 +56,7 @@ export { ## local file path which was read, or some other input source. source: string &log &optional; - ## A value to represent the depth of this file in relation + ## A value to represent the depth of this file in relation ## to its source. In SMTP, it is the depth of the MIME ## attachment on the message. In HTTP, it is the depth of the ## request within the TCP connection. @@ -72,7 +72,7 @@ export { mime_type: string &log &optional; ## A filename for the file if one is available from the source - ## for the file. These will frequently come from + ## for the file. These will frequently come from ## "Content-Disposition" headers in network protocols. filename: string &log &optional; @@ -148,9 +148,18 @@ export { ## Returns: true if the analyzer will be added, or false if analysis ## for the file isn't currently active or the *args* ## were invalid for the analyzer type. - global add_analyzer: function(f: fa_file, - tag: Files::Tag, - args: AnalyzerArgs &default=AnalyzerArgs()): bool; + global add_analyzer: function(f: fa_file, + tag: Files::Tag, + args: AnalyzerArgs &default=AnalyzerArgs()): bool; + + ## Adds all analyzers associated with a give MIME type to the analysis of + ## a file. Note that analyzers added via MIME types cannot take further + ## arguments. + ## + ## f: the file. + ## + ## mtype: the MIME type; it will be compared case-insensitive. + global add_analyzers_for_mime_type: function(f: fa_file, mtype: string); ## Removes an analyzer from the analysis of a given file. ## @@ -195,7 +204,7 @@ export { ## A callback to generate a file handle on demand when ## one is needed by the core. get_file_handle: function(c: connection, is_orig: bool): string; - + ## A callback to "describe" a file. In the case of an HTTP ## transfer the most obvious description would be the URL. ## It's like an extremely compressed version of the normal log. @@ -206,7 +215,7 @@ export { ## Register callbacks for protocols that work with the Files framework. ## The callbacks must uniquely identify a file and each protocol can ## only have a single callback registered for it. - ## + ## ## tag: Tag for the protocol analyzer having a callback being registered. ## ## reg: A :bro:see:`Files::ProtoRegistration` record. @@ -258,13 +267,13 @@ function set_info(f: fa_file) f$info$source = f$source; f$info$duration = f$last_active - f$info$ts; f$info$seen_bytes = f$seen_bytes; - if ( f?$total_bytes ) + if ( f?$total_bytes ) f$info$total_bytes = f$total_bytes; f$info$missing_bytes = f$missing_bytes; f$info$overflow_bytes = f$overflow_bytes; if ( f?$is_orig ) f$info$is_orig = f$is_orig; - if ( f?$mime_type ) + if ( f?$mime_type ) f$info$mime_type = f$mime_type; } @@ -288,6 +297,15 @@ function add_analyzer(f: fa_file, tag: Files::Tag, args: AnalyzerArgs): bool return T; } +function add_analyzers_for_mime_type(f: fa_file, mtype: string) + { + local dummy_args: AnalyzerArgs; + local analyzers = __add_analyzers_for_mime_type(f$id, mtype, dummy_args); + + for ( tag in analyzers ) + add f$info$analyzers[Files::analyzer_name(tag)]; + } + function register_analyzer_add_callback(tag: Files::Tag, callback: function(f: fa_file, args: AnalyzerArgs)) { analyzer_add_callbacks[tag] = callback; @@ -311,6 +329,9 @@ function analyzer_name(tag: Files::Tag): string event file_new(f: fa_file) &priority=10 { set_info(f); + + if ( f?$mime_type ) + add_analyzers_for_mime_type(f, f$mime_type); } event file_over_new_connection(f: fa_file, c: connection, is_orig: bool) &priority=10 diff --git a/scripts/base/init-bare.bro b/scripts/base/init-bare.bro index de26e6a41d..928697edb0 100644 --- a/scripts/base/init-bare.bro +++ b/scripts/base/init-bare.bro @@ -60,6 +60,13 @@ type addr_vec: vector of addr; ## directly and then remove this alias. type table_string_of_string: table[string] of string; +## A set of file analyzer tags. +## +## .. todo:: We need this type definition only for declaring builtin functions +## via ``bifcl``. We should extend ``bifcl`` to understand composite types +## directly and then remove this alias. +type files_tag_set: set[Files::Tag]; + ## A connection's transport-layer protocol. Note that Bro uses the term ## "connection" broadly, using flow semantics for ICMP and UDP. type transport_proto: enum { diff --git a/src/file_analysis/Analyzer.cc b/src/file_analysis/Analyzer.cc index e0b5011aa8..2d45c6579a 100644 --- a/src/file_analysis/Analyzer.cc +++ b/src/file_analysis/Analyzer.cc @@ -3,9 +3,17 @@ #include "Analyzer.h" #include "Manager.h" +file_analysis::ID file_analysis::Analyzer::id_counter = 0; + file_analysis::Analyzer::~Analyzer() { DBG_LOG(DBG_FILE_ANALYSIS, "Destroy file analyzer %s", file_mgr->GetComponentName(tag)); Unref(args); } + +void file_analysis::Analyzer::SetAnalyzerTag(const file_analysis::Tag& arg_tag) + { + assert(! tag || tag == arg_tag); + tag = arg_tag; + } diff --git a/src/file_analysis/Analyzer.h b/src/file_analysis/Analyzer.h index e20e2802cf..ab03b1f9ba 100644 --- a/src/file_analysis/Analyzer.h +++ b/src/file_analysis/Analyzer.h @@ -13,6 +13,8 @@ namespace file_analysis { class File; +typedef uint32 ID; + /** * Base class for analyzers that can be attached to file_analysis::File objects. */ @@ -25,6 +27,18 @@ public: */ virtual ~Analyzer(); + /** + * Initializes the analyzer before input processing starts. + */ + virtual void Init() + { }; + + /** + * Finishes the analyzer's operation after all input has been parsed. + */ + virtual void Done() + { }; + /** * Subclasses may override this metod to receive file data non-sequentially. * @param data points to start of a chunk of file data. @@ -72,6 +86,13 @@ public: */ file_analysis::Tag Tag() const { return tag; } + /** + * Returns the analyzer instance's internal ID. These IDs are unique + * across all analyzers instantiated and can thus be used to + * indentify a specific instance. + */ + ID GetID() const { return id; } + /** * @return the AnalyzerArgs associated with the analyzer. */ @@ -82,10 +103,19 @@ public: */ File* GetFile() const { return file; } + /** + * Sets the tag associated with the analyzer's type. Note that this + * can be called only right after construction, if the constructor + * did not receive a name or tag. The method cannot be used to change + * an existing tag. + */ + void SetAnalyzerTag(const file_analysis::Tag& tag); + protected: /** * Constructor. Only derived classes are meant to be instantiated. + * @param arg_tag the tag definining the analyzer's type. * @param arg_args an \c AnalyzerArgs (script-layer type) value specifiying * tunable options, if any, related to a particular analyzer type. * @param arg_file the file to which the the analyzer is being attached. @@ -94,13 +124,35 @@ protected: : tag(arg_tag), args(arg_args->Ref()->AsRecordVal()), file(arg_file) - {} + { + id = ++id_counter; + } + + /** + * Constructor. Only derived classes are meant to be instantiated. + * As this version of the constructor does not receive a name or tag, + * SetAnalyzerTag() must be called before the instance can be used. + * + * @param arg_args an \c AnalyzerArgs (script-layer type) value specifiying + * tunable options, if any, related to a particular analyzer type. + * @param arg_file the file to which the the analyzer is being attached. + */ + Analyzer(RecordVal* arg_args, File* arg_file) + : tag(), + args(arg_args->Ref()->AsRecordVal()), + file(arg_file) + { + id = ++id_counter; + } private: + ID id; /**< Unique instance ID. */ file_analysis::Tag tag; /**< The particular type of the analyzer instance. */ RecordVal* args; /**< \c AnalyzerArgs val gives tunable analyzer params. */ File* file; /**< The file to which the analyzer is attached. */ + + static ID id_counter; }; } // namespace file_analysis diff --git a/src/file_analysis/AnalyzerSet.cc b/src/file_analysis/AnalyzerSet.cc index 6fc3d2dfd0..aee5e0d62e 100644 --- a/src/file_analysis/AnalyzerSet.cc +++ b/src/file_analysis/AnalyzerSet.cc @@ -9,7 +9,10 @@ using namespace file_analysis; static void analyzer_del_func(void* v) { - delete (file_analysis::Analyzer*) v; + file_analysis::Analyzer* a = (file_analysis::Analyzer*)v; + + a->Done(); + delete a; } AnalyzerSet::AnalyzerSet(File* arg_file) : file(arg_file) @@ -98,6 +101,7 @@ bool AnalyzerSet::AddMod::Perform(AnalyzerSet* set) } set->Insert(a, key); + return true; } @@ -124,7 +128,9 @@ bool AnalyzerSet::Remove(file_analysis::Tag tag, HashKey* key) file_mgr->GetComponentName(tag), file->GetID().c_str()); + a->Done(); delete a; + return true; } @@ -176,6 +182,8 @@ void AnalyzerSet::Insert(file_analysis::Analyzer* a, HashKey* key) file_mgr->GetComponentName(a->Tag()), file->GetID().c_str()); analyzer_map.Insert(key, a); delete key; + + a->Init(); } void AnalyzerSet::DrainModifications() diff --git a/src/file_analysis/Component.cc b/src/file_analysis/Component.cc index 9c47f2c75e..87789eb113 100644 --- a/src/file_analysis/Component.cc +++ b/src/file_analysis/Component.cc @@ -8,13 +8,15 @@ using namespace file_analysis; -Component::Component(const char* arg_name, factory_callback arg_factory) +Component::Component(const char* arg_name, factory_callback arg_factory, Tag::subtype_t subtype) : plugin::Component(plugin::component::FILE_ANALYZER), - plugin::TaggedComponent() + plugin::TaggedComponent(subtype) { name = copy_string(arg_name); canon_name = canonify_name(arg_name); factory = arg_factory; + + file_mgr->RegisterComponent(this, "ANALYZER_"); } Component::Component(const Component& other) @@ -24,6 +26,8 @@ Component::Component(const Component& other) name = copy_string(other.name); canon_name = copy_string(other.canon_name); factory = other.factory; + + // TODO: Do we need the RegisterComponent() call here? } Component::~Component() diff --git a/src/file_analysis/Component.h b/src/file_analysis/Component.h index 4cf2dced60..6e2809349d 100644 --- a/src/file_analysis/Component.h +++ b/src/file_analysis/Component.h @@ -40,8 +40,14 @@ public: * from file_analysis::Analyzer. This is typically a static \c * Instatiate() method inside the class that just allocates and * returns a new instance. + * + * @param subtype A subtype associated with this component that + * further distinguishes it. The subtype will be integrated into the + * analyzer::Tag that the manager associates with this analyzer, and + * analyzer instances can accordingly access it via analyzer::Tag(). + * If not used, leave at zero. */ - Component(const char* name, factory_callback factory); + Component(const char* name, factory_callback factory, Tag::subtype_t subtype = 0); /** * Copy constructor. diff --git a/src/file_analysis/File.cc b/src/file_analysis/File.cc index 55b28763c8..980a8bfbce 100644 --- a/src/file_analysis/File.cc +++ b/src/file_analysis/File.cc @@ -1,6 +1,7 @@ // See the file "COPYING" in the main distribution directory for copyright. #include +#include #include "File.h" #include "FileTimer.h" @@ -82,7 +83,7 @@ File::File(const string& file_id, Connection* conn, analyzer::Tag tag, { StaticInit(); - DBG_LOG(DBG_FILE_ANALYSIS, "Creating new File object %s", file_id.c_str()); + DBG_LOG(DBG_FILE_ANALYSIS, "[%s] Creating new File object", file_id.c_str()); val = new RecordVal(fa_file_type); val->Assign(id_idx, new StringVal(file_id.c_str())); @@ -100,7 +101,7 @@ File::File(const string& file_id, Connection* conn, analyzer::Tag tag, File::~File() { - DBG_LOG(DBG_FILE_ANALYSIS, "Destroying File object %s", id.c_str()); + DBG_LOG(DBG_FILE_ANALYSIS, "[%s] Destroying File object", id.c_str()); Unref(val); // Queue may not be empty in the case where only content gaps were seen. @@ -229,6 +230,7 @@ void File::IncrementByteCount(uint64 size, int field_idx) void File::SetTotalBytes(uint64 size) { + DBG_LOG(DBG_FILE_ANALYSIS, "[%s] Total bytes %" PRIu64, id.c_str(), size); val->Assign(total_bytes_idx, new Val(size, TYPE_COUNT)); } @@ -251,11 +253,17 @@ void File::ScheduleInactivityTimer() const bool File::AddAnalyzer(file_analysis::Tag tag, RecordVal* args) { + DBG_LOG(DBG_FILE_ANALYSIS, "[%s] Queuing addition of %s analyzer", + id.c_str(), file_mgr->GetComponentName(tag)); + return done ? false : analyzers.QueueAdd(tag, args); } bool File::RemoveAnalyzer(file_analysis::Tag tag, RecordVal* args) { + DBG_LOG(DBG_FILE_ANALYSIS, "[%s] Queuing remove of %s analyzer", + id.c_str(), file_mgr->GetComponentName(tag)); + return done ? false : analyzers.QueueRemove(tag, args); } @@ -284,16 +292,18 @@ bool File::DetectMIME(const u_char* data, uint64 len) if ( mime ) { + // strip off charset const char* mime_end = strchr(mime, ';'); - if ( mime_end ) - // strip off charset - val->Assign(mime_type_idx, new StringVal(mime_end - mime, mime)); - else - val->Assign(mime_type_idx, new StringVal(mime)); + StringVal* mime_val = mime_end ? + new StringVal(mime_end - mime, mime) : + new StringVal(mime); + + val->Assign(mime_type_idx, mime_val); + DBG_LOG(DBG_FILE_ANALYSIS, "[%s] Detected MIME type %s", id.c_str(), mime_val->CheckString()); } - return mime; + return true; } void File::ReplayBOF() @@ -314,7 +324,6 @@ void File::ReplayBOF() val->Assign(bof_buffer_idx, new StringVal(bs)); DetectMIME(bs->Bytes(), bs->Len()); - FileEvent(file_new); for ( size_t i = 0; i < bof_buffer.chunks.size(); ++i ) @@ -333,6 +342,11 @@ void File::DataIn(const u_char* data, uint64 len, uint64 offset) first_chunk = false; } + DBG_LOG(DBG_FILE_ANALYSIS, "[%s] %" PRIu64 " bytes in at offset" PRIu64 "; %s [%s]", + id.c_str(), len, offset, + IsComplete() ? "complete" : "incomplete", + fmt_bytes((const char*) data, min((uint64)40, len)), len > 40 ? "..." : ""); + file_analysis::Analyzer* a = 0; IterCookie* c = analyzers.InitForIteration(); @@ -367,6 +381,11 @@ void File::DataIn(const u_char* data, uint64 len) missed_bof = false; } + DBG_LOG(DBG_FILE_ANALYSIS, "[%s] %" PRIu64 " bytes in; %s [%s]", + id.c_str(), len, + IsComplete() ? "complete" : "incomplete", + fmt_bytes((const char*) data, min((uint64)40, len)), len > 40 ? "..." : ""); + file_analysis::Analyzer* a = 0; IterCookie* c = analyzers.InitForIteration(); @@ -391,6 +410,8 @@ void File::DataIn(const u_char* data, uint64 len) void File::EndOfFile() { + DBG_LOG(DBG_FILE_ANALYSIS, "[%s] End of file", id.c_str()); + if ( done ) return; @@ -417,6 +438,9 @@ void File::EndOfFile() void File::Gap(uint64 offset, uint64 len) { + DBG_LOG(DBG_FILE_ANALYSIS, "[%s] Gap of size %" PRIu64 " at offset %" PRIu64, + id.c_str(), offset, len); + analyzers.DrainModifications(); // If we were buffering the beginning of the file, a gap means we've got diff --git a/src/file_analysis/File.h b/src/file_analysis/File.h index 6354c1c7e9..b225d77856 100644 --- a/src/file_analysis/File.h +++ b/src/file_analysis/File.h @@ -229,7 +229,7 @@ protected: * field in #val. * @param data pointer to a chunk of file data. * @param len number of bytes in the data chunk. - * @return whether mime type was available. + * @return true if mime type available. */ bool DetectMIME(const u_char* data, uint64 len); diff --git a/src/file_analysis/Manager.cc b/src/file_analysis/Manager.cc index 0337dbb098..a635a5dc14 100644 --- a/src/file_analysis/Manager.cc +++ b/src/file_analysis/Manager.cc @@ -12,10 +12,12 @@ #include "UID.h" #include "plugin/Manager.h" +#include "analyzer/Manager.h" using namespace file_analysis; TableVal* Manager::disabled = 0; +TableType* Manager::tag_set_type = 0; string Manager::salt; Manager::Manager() @@ -27,15 +29,13 @@ Manager::Manager() Manager::~Manager() { Terminate(); + + for ( MIMEMap::iterator i = mime_types.begin(); i != mime_types.end(); i++ ) + delete i->second; } void Manager::InitPreScript() { - std::list analyzers = plugin_mgr->Components(); - - for ( std::list::const_iterator i = analyzers.begin(); - i != analyzers.end(); ++i ) - RegisterComponent(*i, "ANALYZER_"); } void Manager::InitPostScript() @@ -72,6 +72,7 @@ void Manager::SetHandle(const string& handle) if ( handle.empty() ) return; + DBG_LOG(DBG_FILE_ANALYSIS, "Set current handle to %s", handle.c_str()); current_file_id = HashHandle(handle); } @@ -205,6 +206,28 @@ bool Manager::AddAnalyzer(const string& file_id, file_analysis::Tag tag, return file->AddAnalyzer(tag, args); } +TableVal* Manager::AddAnalyzersForMIMEType(const string& file_id, const string& mtype, + RecordVal* args) + { + if ( ! tag_set_type ) + tag_set_type = internal_type("files_tag_set")->AsTableType(); + + TableVal* sval = new TableVal(tag_set_type); + TagSet* l = LookupMIMEType(mtype, false); + + if ( ! l ) + return sval; + + for ( TagSet::const_iterator i = l->begin(); i != l->end(); i++ ) + { + file_analysis::Tag tag = *i; + if ( AddAnalyzer(file_id, tag, args) ) + sval->Assign(tag.AsEnumVal(), 0); + } + + return sval; + } + bool Manager::RemoveAnalyzer(const string& file_id, file_analysis::Tag tag, RecordVal* args) const { @@ -327,6 +350,9 @@ void Manager::GetFileHandle(analyzer::Tag tag, Connection* c, bool is_orig) if ( ! get_file_handle ) return; + DBG_LOG(DBG_FILE_ANALYSIS, "Raise get_file_handle() for protocol analyzer %s", + analyzer_mgr->GetComponentName(tag)); + EnumVal* tagval = tag.AsEnumVal(); Ref(tagval); @@ -376,5 +402,72 @@ Analyzer* Manager::InstantiateAnalyzer(Tag tag, RecordVal* args, File* f) const return 0; } - return c->Factory()(args, f); + DBG_LOG(DBG_FILE_ANALYSIS, "Instantiate analyzer %s for file %s", + GetComponentName(tag), f->id.c_str()); + + Analyzer* a = c->Factory()(args, f); + + if ( ! a ) + reporter->InternalError("file analyzer instantiation failed"); + + a->SetAnalyzerTag(tag); + + return a; + } + +Manager::TagSet* Manager::LookupMIMEType(const string& mtype, bool add_if_not_found) + { + MIMEMap::const_iterator i = mime_types.find(to_upper(mtype)); + + if ( i != mime_types.end() ) + return i->second; + + if ( ! add_if_not_found ) + return 0; + + TagSet* l = new TagSet; + mime_types.insert(std::make_pair(to_upper(mtype), l)); + return l; + } + +bool Manager::RegisterAnalyzerForMIMEType(EnumVal* tag, StringVal* mtype) + { + Component* p = Lookup(tag); + + if ( ! p ) + return false; + + return RegisterAnalyzerForMIMEType(p->Tag(), mtype->CheckString()); + } + +bool Manager::RegisterAnalyzerForMIMEType(Tag tag, const string& mtype) + { + TagSet* l = LookupMIMEType(mtype, true); + + DBG_LOG(DBG_FILE_ANALYSIS, "Register analyzer %s for MIME type %s", + GetComponentName(tag), mtype.c_str()); + + l->insert(tag); + return true; + } + +bool Manager::UnregisterAnalyzerForMIMEType(EnumVal* tag, StringVal* mtype) + { + Component* p = Lookup(tag); + + if ( ! p ) + return false; + + return UnregisterAnalyzerForMIMEType(p->Tag(), mtype->CheckString()); + } + +bool Manager::UnregisterAnalyzerForMIMEType(Tag tag, const string& mtype) + { + TagSet* l = LookupMIMEType(mtype, true); + + DBG_LOG(DBG_FILE_ANALYSIS, "Unregister analyzer %s for MIME type %s", + GetComponentName(tag), mtype.c_str()); + + l->erase(tag); + return true; } diff --git a/src/file_analysis/Manager.h b/src/file_analysis/Manager.h index cf73c6b52d..c6f86a0480 100644 --- a/src/file_analysis/Manager.h +++ b/src/file_analysis/Manager.h @@ -198,6 +198,18 @@ public: bool AddAnalyzer(const string& file_id, file_analysis::Tag tag, RecordVal* args) const; + /** + * Queue attachment of an all analyzers associated with a given MIME + * type to the file identifier. + * + * @param file_id the file identifier/hash. + * @param mtype the MIME type; comparisions will be performanced case-insensitive. + * @param args a \c AnalyzerArgs value which describes a file analyzer. + * @return A ref'ed \c set[Tag] with all added analyzers. + */ + TableVal* AddAnalyzersForMIMEType(const string& file_id, const string& mtype, + RecordVal* args); + /** * Queue removal of an analyzer for a given file identifier. * @param file_id the file identifier/hash. @@ -224,6 +236,62 @@ public: */ Analyzer* InstantiateAnalyzer(Tag tag, RecordVal* args, File* f) const; + /** + * Registers a MIME type for an analyzer. Once registered, files of + * that MIME type will automatically get a corresponding analyzer + * assigned. + * + * @param tag The analyzer's tag as an enum of script type \c + * Files::Tag. + * + * @param mtype The MIME type. It will be matched case-insenistive. + * + * @return True if successful. + */ + bool RegisterAnalyzerForMIMEType(EnumVal* tag, StringVal* mtype); + + /** + * Registers a MIME type for an analyzer. Once registered, files of + * that MIME type will automatically get a corresponding analyzer + * assigned. + * + * @param tag The analyzer's tag as an enum of script type \c + * Files::Tag. + * + * @param mtype The MIME type. It will be matched case-insenistive. + * + * @return True if successful. + */ + bool RegisterAnalyzerForMIMEType(Tag tag, const string& mtype); + + /** + * Unregisters a MIME type for an analyzer. + * + * @param tag The analyzer's tag as an enum of script type \c + * Files::Tag. + * + * @param mtype The MIME type. It will be matched case-insenistive. + * + * @return True if successful (incl. when the type wasn't actually + * registered for the analyzer). + * + */ + bool UnregisterAnalyzerForMIMEType(EnumVal* tag, StringVal* mtype); + + /** + * Unregisters a MIME type for an analyzer. + * + * @param tag The analyzer's tag as an enum of script type \c + * Files::Tag. + * + * @param mtype The MIME type. It will be matched case-insenistive. + * + * @return True if successful (incl. when the type wasn't actually + * registered for the analyzer). + * + */ + bool UnregisterAnalyzerForMIMEType(Tag tag, const string& mtype); + protected: friend class FileTimer; @@ -297,12 +365,18 @@ protected: static bool IsDisabled(analyzer::Tag tag); private: + typedef set TagSet; + typedef map MIMEMap; + + TagSet* LookupMIMEType(const string& mtype, bool add_if_not_found); IDMap id_map; /**< Map file ID to file_analysis::File records. */ IDSet ignored; /**< Ignored files. Will be finally removed on EOF. */ string current_file_id; /**< Hash of what get_file_handle event sets. */ + MIMEMap mime_types;/**< Mapping of MIME types to analyzers. */ static TableVal* disabled; /**< Table of disabled analyzers. */ + static TableType* tag_set_type; /**< Type for set[tag]. */ static string salt; /**< A salt added to file handles before hashing. */ }; diff --git a/src/file_analysis/file_analysis.bif b/src/file_analysis/file_analysis.bif index 0e904f298f..8205b870c1 100644 --- a/src/file_analysis/file_analysis.bif +++ b/src/file_analysis/file_analysis.bif @@ -26,6 +26,16 @@ function Files::__add_analyzer%(file_id: string, tag: Files::Tag, args: any%): b return new Val(result, TYPE_BOOL); %} +## :bro:see:`Files::add_analyzers_for_mime_type`. +function Files::__add_analyzers_for_mime_type%(file_id: string, mtype: string, args: any%): files_tag_set + %{ + using BifType::Record::Files::AnalyzerArgs; + RecordVal* rv = args->AsRecordVal()->CoerceTo(AnalyzerArgs); + Val* analyzers = file_mgr->AddAnalyzersForMIMEType(file_id->CheckString(), mtype->CheckString(), rv); + Unref(rv); + return analyzers; + %} + ## :bro:see:`Files::remove_analyzer`. function Files::__remove_analyzer%(file_id: string, tag: Files::Tag, args: any%): bool %{ From d727af097b2d78a77c85d98903278a1c1dbdc4ab Mon Sep 17 00:00:00 2001 From: Robin Sommer Date: Tue, 26 Nov 2013 11:22:23 -0800 Subject: [PATCH 08/56] Minor API changes to analyzers. --- src/analyzer/Analyzer.cc | 4 ++-- src/analyzer/Analyzer.h | 9 ++++++--- src/analyzer/Component.cc | 4 ++++ src/analyzer/Manager.cc | 10 +++++----- src/analyzer/Manager.h | 8 ++++++++ src/analyzer/protocol/tcp/TCP.h | 4 ++-- 6 files changed, 27 insertions(+), 12 deletions(-) diff --git a/src/analyzer/Analyzer.cc b/src/analyzer/Analyzer.cc index 03734f1a22..2927e3ad97 100644 --- a/src/analyzer/Analyzer.cc +++ b/src/analyzer/Analyzer.cc @@ -644,12 +644,12 @@ void Analyzer::FlipRoles() resp_supporters = tmp; } -void Analyzer::ProtocolConfirmation() +void Analyzer::ProtocolConfirmation(Tag arg_tag) { if ( protocol_confirmed ) return; - EnumVal* tval = tag.AsEnumVal(); + EnumVal* tval = arg_tag ? arg_tag.AsEnumVal() : tag.AsEnumVal(); Ref(tval); val_list* vl = new val_list; diff --git a/src/analyzer/Analyzer.h b/src/analyzer/Analyzer.h index f7ca07ca51..eec0303475 100644 --- a/src/analyzer/Analyzer.h +++ b/src/analyzer/Analyzer.h @@ -97,8 +97,8 @@ public: /** * Constructor. As this version of the constructor does not receive a - * name or tag, setTag() must be called before the instance can be - * used. + * name or tag, SetAnalyzerTag() must be called before the instance + * can be used. * * @param conn The connection the analyzer is associated with. */ @@ -471,8 +471,11 @@ public: * may turn into \c protocol_confirmed event at the script-layer (but * only once per analyzer for each connection, even if the method is * called multiple times). + * + * If tag is given, it overrides the analyzer tag passed to the + * scripting layer; the default is the one of the analyzer itself. */ - virtual void ProtocolConfirmation(); + virtual void ProtocolConfirmation(Tag tag = Tag()); /** * Signals Bro's protocol detection that the analyzer has found a diff --git a/src/analyzer/Component.cc b/src/analyzer/Component.cc index 66ab2213bb..255e03435c 100644 --- a/src/analyzer/Component.cc +++ b/src/analyzer/Component.cc @@ -17,6 +17,8 @@ Component::Component(const char* arg_name, factory_callback arg_factory, Tag::su factory = arg_factory; enabled = arg_enabled; partial = arg_partial; + + analyzer_mgr->RegisterComponent(this, "ANALYZER_"); } Component::Component(const Component& other) @@ -28,6 +30,8 @@ Component::Component(const Component& other) factory = other.factory; enabled = other.enabled; partial = other.partial; + + // TODO: Do we need the RegisterComponent() call here? } Component::~Component() diff --git a/src/analyzer/Manager.cc b/src/analyzer/Manager.cc index 6268fde177..748af34dfb 100644 --- a/src/analyzer/Manager.cc +++ b/src/analyzer/Manager.cc @@ -86,11 +86,6 @@ Manager::~Manager() void Manager::InitPreScript() { - std::list analyzers = plugin_mgr->Components(); - - for ( std::list::const_iterator i = analyzers.begin(); i != analyzers.end(); i++ ) - RegisterComponent(*i, "ANALYZER_"); - // Cache these tags. analyzer_backdoor = GetComponentTag("BACKDOOR"); analyzer_connsize = GetComponentTag("CONNSIZE"); @@ -202,6 +197,11 @@ void Manager::DisableAllAnalyzers() (*i)->SetEnabled(false); } +analyzer::Tag Manager::GetAnalyzerTag(const char* name) + { + return GetComponentTag(name); + } + bool Manager::IsEnabled(Tag tag) { if ( ! tag ) diff --git a/src/analyzer/Manager.h b/src/analyzer/Manager.h index 77b40a1d68..c842794d6c 100644 --- a/src/analyzer/Manager.h +++ b/src/analyzer/Manager.h @@ -133,6 +133,14 @@ public: */ void DisableAllAnalyzers(); + /** + * Returns the tag associated with an analyer name, or the tag + * associated with an error if no such analyzer exists. + * + * @param name The canonical analyzer name to check. + */ + Tag GetAnalyzerTag(const char* name); + /** * Returns true if an analyzer is enabled. * diff --git a/src/analyzer/protocol/tcp/TCP.h b/src/analyzer/protocol/tcp/TCP.h index b2649b4ab8..e01ed3a255 100644 --- a/src/analyzer/protocol/tcp/TCP.h +++ b/src/analyzer/protocol/tcp/TCP.h @@ -294,8 +294,8 @@ public: // TCP_ENDPOINT_RESET. If gen_event is true and the connection // is now fully closed, a connection_finished event will be // generated; otherwise not. - virtual void ConnectionClosed(TCP_Endpoint* endpoint, - TCP_Endpoint* peer, int gen_event); + virtual void ConnectionClosed(analyzer::tcp::TCP_Endpoint* endpoint, + analyzer::tcp::TCP_Endpoint* peer, int gen_event); virtual void ConnectionFinished(int half_finished); virtual void ConnectionReset(); From 7412470d6616a9f8c6a9a8174af244b570ebdcee Mon Sep 17 00:00:00 2001 From: Robin Sommer Date: Sat, 28 Sep 2013 16:06:40 -0700 Subject: [PATCH 09/56] Prettyfing Describe() for record types. If a record type has a name and ODesc is set to short, we now print the name instead of the full field list. --- src/Type.cc | 14 ++++++++++---- src/Var.cc | 2 +- 2 files changed, 11 insertions(+), 5 deletions(-) diff --git a/src/Type.cc b/src/Type.cc index 6b4b2eb970..52645533c3 100644 --- a/src/Type.cc +++ b/src/Type.cc @@ -1048,10 +1048,16 @@ void RecordType::Describe(ODesc* d) const { if ( d->IsReadable() ) { - d->AddSP("record {"); - DescribeFields(d); - d->SP(); - d->Add("}"); + if ( d->IsShort() && GetTypeID() ) + d->Add(GetTypeID()); + + else + { + d->AddSP("record {"); + DescribeFields(d); + d->SP(); + d->Add("}"); + } } else diff --git a/src/Var.cc b/src/Var.cc index d384fedc74..08129bdc3e 100644 --- a/src/Var.cc +++ b/src/Var.cc @@ -314,9 +314,9 @@ void add_type(ID* id, BroType* t, attr_list* attr, int /* is_event */) delete [] data; } - tnew->SetTypeID(copy_string(id->Name())); } + tnew->SetTypeID(copy_string(id->Name())); id->SetType(tnew); id->MakeType(); From 555df1e7ea3b1baa629cb5059f5acf55be8d9114 Mon Sep 17 00:00:00 2001 From: Robin Sommer Date: Tue, 26 Nov 2013 11:23:25 -0800 Subject: [PATCH 10/56] Checkpointing the dynamic plugin code. This is essentially the code from the dynamic-plugin branch except for some pieces that I have split out into separate, earlier commits. I'm going to updatre things in this branch going forward. --- CMakeLists.txt | 10 + cmake | 2 +- config.h.in | 11 + src/CMakeLists.txt | 15 +- src/DebugLogger.cc | 25 +- src/DebugLogger.h | 12 +- src/Event.cc | 6 + src/FlowSrc.cc | 2 +- src/Func.cc | 47 ++- src/Net.cc | 15 +- src/Net.h | 1 + src/RemoteSerializer.cc | 2 +- src/builtin-func.l | 6 +- src/builtin-func.y | 14 +- src/main.cc | 72 ++++- src/plugin/ComponentManager.h | 2 - src/plugin/Macros.h | 17 +- src/plugin/Manager.cc | 295 +++++++++++++++++- src/plugin/Manager.h | 122 +++++++- src/plugin/Plugin.cc | 135 +++++++- src/plugin/Plugin.h | 228 +++++++++++++- src/scan.l | 20 ++ src/util-config.h.in | 1 + src/util.cc | 87 ++++-- src/util.h | 15 +- .../Baseline/core.plugins.analyzer/output | 6 + .../Baseline/core.plugins.test-plugin/output | 11 + testing/btest/Traces/port4242.trace | Bin 0 -> 868 bytes .../plugins/analyzer-plugin/.btest-ignore | 0 .../plugins/analyzer-plugin/CMakeLists.txt | 20 ++ .../plugins/analyzer-plugin/__bro_plugin__ | 2 + .../scripts/Demo/Foo/base/main.bro | 7 + .../analyzer-plugin/scripts/__load__.bro | 1 + .../core/plugins/analyzer-plugin/src/Foo.cc | 59 ++++ .../core/plugins/analyzer-plugin/src/Foo.h | 31 ++ .../plugins/analyzer-plugin/src/Plugin.cc | 12 + .../plugins/analyzer-plugin/src/events.bif | 2 + .../analyzer-plugin/src/foo-analyzer.pac | 15 + .../analyzer-plugin/src/foo-protocol.pac | 4 + .../core/plugins/analyzer-plugin/src/foo.pac | 26 ++ .../plugins/analyzer-plugin/src/functions.bif | 0 testing/btest/core/plugins/analyzer.bro | 13 + testing/btest/core/plugins/test-plugin.sh | 45 +++ 43 files changed, 1306 insertions(+), 110 deletions(-) create mode 100644 testing/btest/Baseline/core.plugins.analyzer/output create mode 100644 testing/btest/Baseline/core.plugins.test-plugin/output create mode 100644 testing/btest/Traces/port4242.trace create mode 100644 testing/btest/core/plugins/analyzer-plugin/.btest-ignore create mode 100644 testing/btest/core/plugins/analyzer-plugin/CMakeLists.txt create mode 100644 testing/btest/core/plugins/analyzer-plugin/__bro_plugin__ create mode 100644 testing/btest/core/plugins/analyzer-plugin/scripts/Demo/Foo/base/main.bro create mode 100644 testing/btest/core/plugins/analyzer-plugin/scripts/__load__.bro create mode 100644 testing/btest/core/plugins/analyzer-plugin/src/Foo.cc create mode 100644 testing/btest/core/plugins/analyzer-plugin/src/Foo.h create mode 100644 testing/btest/core/plugins/analyzer-plugin/src/Plugin.cc create mode 100644 testing/btest/core/plugins/analyzer-plugin/src/events.bif create mode 100644 testing/btest/core/plugins/analyzer-plugin/src/foo-analyzer.pac create mode 100644 testing/btest/core/plugins/analyzer-plugin/src/foo-protocol.pac create mode 100644 testing/btest/core/plugins/analyzer-plugin/src/foo.pac create mode 100644 testing/btest/core/plugins/analyzer-plugin/src/functions.bif create mode 100644 testing/btest/core/plugins/analyzer.bro create mode 100644 testing/btest/core/plugins/test-plugin.sh diff --git a/CMakeLists.txt b/CMakeLists.txt index 1c427188ca..0738eb387d 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -1,5 +1,10 @@ project(Bro C CXX) + +# When changing the minimum version here, also adapt +# cmake/BroPluginDynamic and +# aux/bro-aux/plugin-support/skeleton/CMakeLists.txt cmake_minimum_required(VERSION 2.6.3 FATAL_ERROR) + include(cmake/CommonCMakeConfig.cmake) ######################################################################## @@ -17,6 +22,7 @@ set(BRO_SCRIPT_SOURCE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/scripts) get_filename_component(BRO_SCRIPT_INSTALL_PATH ${BRO_SCRIPT_INSTALL_PATH} ABSOLUTE) +set(BRO_PLUGIN_INSTALL_PATH ${BRO_ROOT_DIR}/share/bro/plugins) set(BRO_MAGIC_INSTALL_PATH ${BRO_ROOT_DIR}/share/bro/magic) set(BRO_MAGIC_SOURCE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/magic/database) @@ -183,6 +189,10 @@ include(MiscTests) include(PCAPTests) include(OpenSSLTests) include(CheckNameserCompat) +include(GetArchitecture) + +# Tell the plugin code that we're building as part of the main tree. +set(BRO_PLUGIN_INTERNAL_BUILD true CACHE INTERNAL "" FORCE) configure_file(${CMAKE_CURRENT_SOURCE_DIR}/config.h.in ${CMAKE_CURRENT_BINARY_DIR}/config.h) diff --git a/cmake b/cmake index 0187b33a29..fc53d57770 160000 --- a/cmake +++ b/cmake @@ -1 +1 @@ -Subproject commit 0187b33a29d5ec824f940feff60dc5d8c2fe314f +Subproject commit fc53d57770c86fbf9bd2d9694f06a1d539ebe35f diff --git a/config.h.in b/config.h.in index 2d065f755e..61124d2f67 100644 --- a/config.h.in +++ b/config.h.in @@ -209,3 +209,14 @@ /* Common IPv6 extension structure */ #cmakedefine HAVE_IP6_EXT + +/* String with host architecture (e.g., "linux-x86_64") */ +#define HOST_ARCHITECTURE "@HOST_ARCHITECTURE@" + +/* String with extension of dynamic libraries (e.g., ".so") */ +#define SHARED_LIBRARY_SUFFIX "@CMAKE_SHARED_LIBRARY_SUFFIX@" + +/* True if we're building outside of the main Bro source code tree. */ +#ifndef BRO_PLUGIN_INTERNAL_BUILD +#define BRO_PLUGIN_INTERNAL_BUILD @BRO_PLUGIN_INTERNAL_BUILD@ +#endif diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index c881cc4df1..34af4eb974 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -414,6 +414,17 @@ add_dependencies(bro bif_loader_plugins) # Install *.bif.bro. install(DIRECTORY ${CMAKE_BINARY_DIR}/scripts/base/bif DESTINATION ${BRO_SCRIPT_INSTALL_PATH}/base) -# Make clean removes the bif directory. -set_directory_properties(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES ${CMAKE_BINARY_DIR}/scripts/base/bif) +# Install the plugin directory with a short README. +set(plugin_binary_dir "${CMAKE_BINARY_DIR}/plugins") + +add_custom_command(OUTPUT ${plugin_binary_dir}/README + COMMAND mkdir ARGS -p ${plugin_binary_dir} + COMMAND echo ARGS "This directory holds dynamic Bro plugins." >${plugin_binary_dir}/README) + +add_custom_target(plugin-dir DEPENDS ${plugin_binary_dir}/README) +add_dependencies(bro plugin-dir) + +# Make clean removes the bif and plugin directories. +set_directory_properties(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES ${CMAKE_BINARY_DIR}/scripts/base/bif) +set_directory_properties(PROPERTIES ADDITIONAL_MAKE_CLEAN_FILES ${plugin_binary_dir}) diff --git a/src/DebugLogger.cc b/src/DebugLogger.cc index 95049ef70b..267383503d 100644 --- a/src/DebugLogger.cc +++ b/src/DebugLogger.cc @@ -5,6 +5,7 @@ #include "DebugLogger.h" #include "Net.h" +#include "plugin/Plugin.h" DebugLogger debug_logger("debug"); @@ -73,10 +74,12 @@ void DebugLogger::EnableStreams(const char* s) { if ( strcasecmp("verbose", tok) == 0 ) verbose = true; - else + else if ( strncmp(tok, "plugin-", 7) != 0 ) reporter->FatalError("unknown debug stream %s\n", tok); } + enabled_streams.insert(tok); + tok = strtok(0, ","); } @@ -105,4 +108,24 @@ void DebugLogger::Log(DebugStream stream, const char* fmt, ...) fflush(file); } +void DebugLogger::Log(const plugin::Plugin& plugin, const char* fmt, ...) + { + string tok = string("plugin-") + plugin.Name(); + tok = strreplace(tok, "::", "-"); + + if ( enabled_streams.find(tok) == enabled_streams.end() ) + return; + + fprintf(file, "%17.06f/%17.06f [plugin %s] ", + network_time, current_time(true), plugin.Name()); + + va_list ap; + va_start(ap, fmt); + vfprintf(file, fmt, ap); + va_end(ap); + + fputc('\n', file); + fflush(file); + } + #endif diff --git a/src/DebugLogger.h b/src/DebugLogger.h index c5744642f5..a5d3c2878c 100644 --- a/src/DebugLogger.h +++ b/src/DebugLogger.h @@ -7,6 +7,8 @@ #ifdef DEBUG #include +#include +#include // To add a new debugging stream, add a constant here as well as // an entry to DebugLogger::streams in DebugLogger.cc. @@ -27,7 +29,7 @@ enum DebugStream { DBG_INPUT, // Input streams DBG_THREADING, // Threading system DBG_FILE_ANALYSIS, // File analysis - DBG_PLUGINS, + DBG_PLUGINS, // Plugin support NUM_DBGS // Has to be last }; @@ -39,6 +41,10 @@ enum DebugStream { #define DBG_PUSH(stream) debug_logger.PushIndent(stream) #define DBG_POP(stream) debug_logger.PopIndent(stream) +#define PLUGIN_DBG_LOG(plugin, args...) debug_logger.Log(plugin, args) + +namespace plugin { class Plugin; } + class DebugLogger { public: // Output goes to stderr per default. @@ -46,6 +52,7 @@ public: ~DebugLogger(); void Log(DebugStream stream, const char* fmt, ...); + void Log(const plugin::Plugin& plugin, const char* fmt, ...); void PushIndent(DebugStream stream) { ++streams[int(stream)].indent; } @@ -76,6 +83,8 @@ private: bool enabled; }; + std::set enabled_streams; + static Stream streams[NUM_DBGS]; }; @@ -86,6 +95,7 @@ extern DebugLogger debug_logger; #define DBG_LOG_VERBOSE(args...) #define DBG_PUSH(stream) #define DBG_POP(stream) +#define PLUGIN_DBG_LOG(plugin, args...) #endif #endif diff --git a/src/Event.cc b/src/Event.cc index 0fb76d10bc..7774042ac4 100644 --- a/src/Event.cc +++ b/src/Event.cc @@ -6,6 +6,7 @@ #include "Func.h" #include "NetVar.h" #include "Trigger.h" +#include "plugin/Manager.h" EventMgr mgr; @@ -77,6 +78,9 @@ EventMgr::~EventMgr() void EventMgr::QueueEvent(Event* event) { + if ( plugin_mgr->QueueEvent(event) ) + return; + if ( ! head ) head = tail = event; else @@ -115,6 +119,8 @@ void EventMgr::Drain() SegmentProfiler(segment_logger, "draining-events"); + plugin_mgr->DrainEvents(); + draining = true; while ( head ) Dispatch(); diff --git a/src/FlowSrc.cc b/src/FlowSrc.cc index 32aa4c4e3a..563153b634 100644 --- a/src/FlowSrc.cc +++ b/src/FlowSrc.cc @@ -49,7 +49,7 @@ void FlowSrc::Process() // This is normally done by calling net_packet_dispatch(), // but as we don't have a packet to dispatch ... - network_time = next_timestamp; + net_update_time(next_timestamp); expire_timers(); netflow_analyzer->downflow()->set_exporter_ip(exporter_ip); diff --git a/src/Func.cc b/src/Func.cc index 11749a8a9c..d8e0ab2c96 100644 --- a/src/Func.cc +++ b/src/Func.cc @@ -46,6 +46,7 @@ #include "Event.h" #include "Traverse.h" #include "Reporter.h" +#include "plugin/Manager.h" extern RETSIGTYPE sig_handler(int signo); @@ -226,7 +227,7 @@ TraversalCode Func::Traverse(TraversalCallback* cb) const HANDLE_TC_STMT_PRE(tc); // FIXME: Traverse arguments to builtin functions, too. - if ( kind == BRO_FUNC ) + if ( kind == BRO_FUNC && scope ) { tc = scope->Traverse(cb); HANDLE_TC_STMT_PRE(tc); @@ -281,6 +282,50 @@ Val* BroFunc::Call(val_list* args, Frame* parent) const #ifdef PROFILE_BRO_FUNCTIONS DEBUG_MSG("Function: %s\n", id->Name()); #endif + + Val* plugin_result = plugin_mgr->CallFunction(this, args); + + if ( plugin_result ) + { + // TODO: We should factor this out into its own method. + switch ( Flavor() ) { + case FUNC_FLAVOR_EVENT: + Unref(plugin_result); + plugin_result = 0; + break; + + case FUNC_FLAVOR_HOOK: + if ( plugin_result->Type()->Tag() != TYPE_BOOL ) + reporter->InternalError("plugin returned non-bool for hook"); + + break; + + case FUNC_FLAVOR_FUNCTION: + { + BroType* yt = FType()->YieldType(); + + if ( (! yt) || yt->Tag() == TYPE_VOID ) + { + Unref(plugin_result); + plugin_result = 0; + } + + else + { + if ( plugin_result->Type()->Tag() != yt->Tag() ) + reporter->InternalError("plugin returned wrong type for function call"); + } + + break; + } + } + + loop_over_list(*args, i) + Unref((*args)[i]); + + return plugin_result; + } + if ( bodies.empty() ) { // Can only happen for events and hooks. diff --git a/src/Net.cc b/src/Net.cc index ac4dacf9b8..6cc148770e 100644 --- a/src/Net.cc +++ b/src/Net.cc @@ -30,6 +30,7 @@ #include "PacketSort.h" #include "Serializer.h" #include "PacketDumper.h" +#include "plugin/Manager.h" extern "C" { #include "setsignal.h" @@ -144,13 +145,17 @@ RETSIGTYPE watchdog(int /* signo */) return RETSIGVAL; } +void net_update_time(double new_network_time) + { + network_time = new_network_time; + plugin_mgr->UpdateNetworkTime(network_time); + } + void net_init(name_list& interfaces, name_list& readfiles, name_list& netflows, name_list& flowfiles, const char* writefile, const char* filter, const char* secondary_filter, int do_watchdog) { - init_net_var(); - if ( readfiles.length() > 0 || flowfiles.length() > 0 ) { reading_live = pseudo_realtime > 0.0; @@ -323,7 +328,7 @@ void net_packet_dispatch(double t, const struct pcap_pkthdr* hdr, : timer_mgr; // network_time never goes back. - network_time = tmgr->Time() < t ? t : tmgr->Time(); + net_update_time(tmgr->Time() < t ? t : tmgr->Time()); current_pktsrc = src_ps; current_iosrc = src_ps; @@ -456,7 +461,7 @@ void net_run() { // Take advantage of the lull to get up to // date on timers and events. - network_time = ct; + net_update_time(ct); expire_timers(); usleep(1); // Just yield. } @@ -478,7 +483,7 @@ void net_run() // date on timers and events. Because we only // have timers as sources, going to sleep here // doesn't risk blocking on other inputs. - network_time = current_time(); + net_update_time(current_time()); expire_timers(); // Avoid busy-waiting - pause for 100 ms. diff --git a/src/Net.h b/src/Net.h index 5b959d1688..765e9f709d 100644 --- a/src/Net.h +++ b/src/Net.h @@ -19,6 +19,7 @@ extern void net_run(); extern void net_get_final_stats(); extern void net_finish(int drain_events); extern void net_delete(); // Reclaim all memory, etc. +extern void net_update_time(double new_network_time); extern void net_packet_arrival(double t, const struct pcap_pkthdr* hdr, const u_char* pkt, int hdr_size, PktSrc* src_ps); diff --git a/src/RemoteSerializer.cc b/src/RemoteSerializer.cc index c8cf03667b..e60ad34b46 100644 --- a/src/RemoteSerializer.cc +++ b/src/RemoteSerializer.cc @@ -1455,7 +1455,7 @@ void RemoteSerializer::Process() // FIXME: The following chunk of code is copied from // net_packet_dispatch(). We should change that function // to accept an IOSource instead of the PktSrc. - network_time = p->time; + net_update_time(p->time); SegmentProfiler(segment_logger, "expiring-timers"); TimerMgr* tmgr = sessions->LookupTimerMgr(GetCurrentTag()); diff --git a/src/builtin-func.l b/src/builtin-func.l index 3e5f7bce0a..63df96eb80 100644 --- a/src/builtin-func.l +++ b/src/builtin-func.l @@ -247,14 +247,13 @@ void init_alternative_mode() fprintf(fp_func_init, "\n"); fprintf(fp_func_init, "#include \n"); fprintf(fp_func_init, "#include \n"); + fprintf(fp_func_init, "#include \"plugin/Plugin.h\"\n"); fprintf(fp_func_init, "#include \"%s.h\"\n", input_filename); fprintf(fp_func_init, "\n"); fprintf(fp_func_init, "namespace plugin { namespace %s {\n", plugin); fprintf(fp_func_init, "\n"); - fprintf(fp_func_init, "std::list > __bif_%s_init()\n", name); + fprintf(fp_func_init, "void __bif_%s_init(plugin::Plugin* plugin)\n", name); fprintf(fp_func_init, "\t{\n"); - fprintf(fp_func_init, "\tstd::list > bifs;\n"); - fprintf(fp_func_init, "\n"); } } @@ -266,7 +265,6 @@ void finish_alternative_mode() if ( plugin ) { fprintf(fp_func_init, "\n"); - fprintf(fp_func_init, "\treturn bifs;\n"); fprintf(fp_func_init, "\t}\n"); fprintf(fp_func_init, "} }\n"); fprintf(fp_func_init, "\n"); diff --git a/src/builtin-func.y b/src/builtin-func.y index 662256b8d6..1b22436fff 100644 --- a/src/builtin-func.y +++ b/src/builtin-func.y @@ -267,12 +267,12 @@ void print_event_c_body(FILE *fp) //fprintf(fp, "%s // end namespace\n", decl.generate_c_namespace_end.c_str()); } -void record_bif_item(const char* id, int type) +void record_bif_item(const char* id, const char* type) { if ( ! plugin ) return; - fprintf(fp_func_init, "\tbifs.push_back(std::make_pair(\"%s\", %d));\n", id, type); + fprintf(fp_func_init, "\tplugin->AddBifItem(\"%s\", plugin::BifItem::%s);\n", id, type); } %} @@ -358,7 +358,7 @@ type_def: TOK_TYPE opt_ws TOK_ID opt_ws ':' opt_ws type_def_types opt_ws ';' decl.c_fullname.c_str(), decl.bro_fullname.c_str(), type_name.c_str()); - record_bif_item(decl.bro_fullname.c_str(), 5); + record_bif_item(decl.bro_fullname.c_str(), "TYPE"); } ; @@ -401,7 +401,7 @@ enum_def: enum_def_1 enum_list TOK_RPB "\t%s = internal_type(\"%s\")->AsEnumType();\n", decl.c_fullname.c_str(), decl.bro_fullname.c_str()); - record_bif_item(decl.bro_fullname.c_str(), 5); + record_bif_item(decl.bro_fullname.c_str(), "TYPE"); } ; @@ -457,7 +457,7 @@ const_def: TOK_CONST opt_ws TOK_ID opt_ws ':' opt_ws TOK_ID opt_ws ';' decl.c_fullname.c_str(), decl.bro_fullname.c_str(), accessor); - record_bif_item(decl.bro_fullname.c_str(), 3); + record_bif_item(decl.bro_fullname.c_str(), "CONSTANT"); } attr_list: @@ -545,7 +545,7 @@ head_1: TOK_ID opt_ws arg_begin "Val* %s(Frame* frame, val_list* %s)", decl.c_fullname.c_str(), arg_list_name); - record_bif_item(decl.bro_fullname.c_str(), 1); + record_bif_item(decl.bro_fullname.c_str(), "FUNCTION"); } else if ( definition_type == EVENT_DEF ) { @@ -562,7 +562,7 @@ head_1: TOK_ID opt_ws arg_begin "\t%s = internal_handler(\"%s\");\n", decl.c_fullname.c_str(), decl.bro_fullname.c_str()); - record_bif_item(decl.bro_fullname.c_str(), 2); + record_bif_item(decl.bro_fullname.c_str(), "EVENT"); // C++ prototypes of bro_event_* functions will // be generated later. diff --git a/src/main.cc b/src/main.cc index 0f60a4c70f..bfbcf77a6d 100644 --- a/src/main.cc +++ b/src/main.cc @@ -198,6 +198,7 @@ void usage() fprintf(stderr, " -N|--print-plugins | print available plugins and exit (-NN for verbose)\n"); fprintf(stderr, " -O|--optimize | optimize policy script\n"); fprintf(stderr, " -P|--prime-dns | prime DNS\n"); + fprintf(stderr, " -Q|--time | print execution time summary to stderr\n"); fprintf(stderr, " -R|--replay | replay events\n"); fprintf(stderr, " -S|--debug-rules | enable rule debugging\n"); fprintf(stderr, " -T|--re-level | set 'RE_level' for rules\n"); @@ -220,8 +221,10 @@ void usage() fprintf(stderr, " -n|--idmef-dtd | specify path to IDMEF DTD file\n"); #endif - fprintf(stderr, " $BROPATH | file search path (%s)\n", bro_path()); + fprintf(stderr, " $BROPATH | file search path (%s)\n", bro_path().c_str()); fprintf(stderr, " $BROMAGIC | libmagic mime magic database search path (%s)\n", bro_magic_path()); + fprintf(stderr, " $BRO_PLUGINS | plugin search path (%s)\n", bro_plugin_path()); + fprintf(stderr, " $BRO_PREFIXES | prefix list (%s)\n", bro_prefixes()); fprintf(stderr, " $BRO_PREFIXES | prefix list (%s)\n", bro_prefixes().c_str()); fprintf(stderr, " $BRO_DNS_FAKE | disable DNS lookups (%s)\n", bro_dns_fake()); fprintf(stderr, " $BRO_SEED_FILE | file to load seeds from (not set)\n"); @@ -435,6 +438,8 @@ int main(int argc, char** argv) { std::set_new_handler(bro_new_handler); + double time_start = current_time(true); + brofiler.ReadStats(); bro_argc = argc; @@ -464,6 +469,7 @@ int main(int argc, char** argv) int rule_debug = 0; int RE_level = 4; int print_plugins = 0; + int time_bro = 0; static struct option long_opts[] = { {"bare-mode", no_argument, 0, 'b'}, @@ -545,7 +551,7 @@ int main(int argc, char** argv) opterr = 0; char opts[256]; - safe_strncpy(opts, "B:D:e:f:I:i:K:l:n:p:R:r:s:T:t:U:w:x:X:y:Y:z:CFGLNOPSWbdghvZ", + safe_strncpy(opts, "B:D:e:f:I:i:K:l:n:p:R:r:s:T:t:U:w:x:X:y:Y:z:CFGLNOPSWbdghvZQ", sizeof(opts)); #ifdef USE_PERFTOOLS_DEBUG @@ -674,6 +680,10 @@ int main(int argc, char** argv) dns_type = DNS_PRIME; break; + case 'Q': + time_bro = 1; + break; + case 'R': events_file = optarg; break; @@ -760,6 +770,7 @@ int main(int argc, char** argv) reporter = new Reporter(); thread_mgr = new threading::Manager(); + plugin_mgr = new plugin::Manager(); #ifdef DEBUG if ( debug_streams ) @@ -810,6 +821,8 @@ int main(int argc, char** argv) if ( ! bare_mode ) add_input_file("base/init-default.bro"); + plugin_mgr->LoadPluginsFrom(bro_plugin_path()); + if ( optind == argc && read_files.length() == 0 && flow_files.length() == 0 && interfaces.length() == 0 && @@ -842,7 +855,6 @@ int main(int argc, char** argv) analyzer_mgr = new analyzer::Manager(); log_mgr = new logging::Manager(); input_mgr = new input::Manager(); - plugin_mgr = new plugin::Manager(); file_mgr = new file_analysis::Manager(); plugin_mgr->InitPreScript(); @@ -877,9 +889,13 @@ int main(int argc, char** argv) yyparse(); - plugin_mgr->InitPostScript(); - analyzer_mgr->InitPostScript(); - file_mgr->InitPostScript(); + init_general_global_var(); + init_net_var(); + + plugin_mgr->InitBifs(); + + if ( reporter->Errors() > 0 ) + exit(1); if ( print_plugins ) { @@ -887,6 +903,10 @@ int main(int argc, char** argv) exit(1); } + plugin_mgr->InitPostScript(); + analyzer_mgr->InitPostScript(); + file_mgr->InitPostScript(); + #ifdef USE_PERFTOOLS_DEBUG } #endif @@ -916,8 +936,6 @@ int main(int argc, char** argv) reporter->InitOptions(); - init_general_global_var(); - if ( user_pcap_filter ) { ID* id = global_scope()->Lookup("cmd_line_bpf_filter"); @@ -1079,7 +1097,7 @@ int main(int argc, char** argv) if ( ! reading_live && ! reading_traces ) // Set up network_time to track real-time, since // we don't have any other source for it. - network_time = current_time(); + net_update_time(current_time()); EventHandlerPtr bro_init = internal_handler("bro_init"); if ( bro_init ) //### this should be a function @@ -1165,7 +1183,43 @@ int main(int argc, char** argv) #endif + double time_net_start = current_time(true);; + + unsigned int mem_net_start_total; + unsigned int mem_net_start_malloced; + + if ( time_bro ) + { + get_memory_usage(&mem_net_start_total, &mem_net_start_malloced); + + fprintf(stderr, "# initialization %.6f\n", time_net_start - time_start); + + fprintf(stderr, "# initialization %uM/%uM\n", + mem_net_start_total / 1024 / 1024, + mem_net_start_malloced / 1024 / 1024); + } + net_run(); + + double time_net_done = current_time(true);; + + unsigned int mem_net_done_total; + unsigned int mem_net_done_malloced; + + if ( time_bro ) + { + get_memory_usage(&mem_net_done_total, &mem_net_done_malloced); + + fprintf(stderr, "# total time %.6f, processing %.6f\n", + time_net_done - time_start, time_net_done - time_net_start); + + fprintf(stderr, "# total mem %uM/%uM, processing %uM/%uM\n", + mem_net_done_total / 1024 / 1024, + mem_net_done_malloced / 1024 / 1024, + (mem_net_done_total - mem_net_start_total) / 1024 / 1024, + (mem_net_done_malloced - mem_net_start_malloced) / 1024 / 1024); + } + done_with_network(); net_delete(); diff --git a/src/plugin/ComponentManager.h b/src/plugin/ComponentManager.h index ccc076db28..71994d82d1 100644 --- a/src/plugin/ComponentManager.h +++ b/src/plugin/ComponentManager.h @@ -83,8 +83,6 @@ public: */ T GetComponentTag(Val* v) const; -protected: - /** * Add a component the internal maps used to keep track of it and create * a script-layer ID for the component's enum value. diff --git a/src/plugin/Macros.h b/src/plugin/Macros.h index 9362642e91..3248031e98 100644 --- a/src/plugin/Macros.h +++ b/src/plugin/Macros.h @@ -28,18 +28,15 @@ * must be unique across all loaded plugins. */ #define BRO_PLUGIN_BEGIN(_ns, _name) \ - namespace plugin { namespace _ns ## _ ## _name {\ + namespace plugin { namespace _ns ## _ ## _name { \ class Plugin : public plugin::Plugin { \ protected: \ - void InitPreScript() \ + void InitPreScript() \ { \ SetName(#_ns "::" #_name); \ - SetVersion(-1);\ - SetAPIVersion(BRO_PLUGIN_API_VERSION);\ - SetDynamicPlugin(false); -// TODO: The SetDynamicPlugin() call is currently hardcoded to false. Change -// once we have dynamic plugins as well. - + SetVersion(-1); \ + SetAPIVersion(BRO_PLUGIN_API_VERSION); \ + SetDynamicPlugin(! BRO_PLUGIN_INTERNAL_BUILD);\ /** * Ends the definition of a plugin. @@ -76,8 +73,8 @@ * interpreter. */ #define BRO_PLUGIN_BIF_FILE(file) \ - extern std::list > __bif_##file##_init(); \ - AddBifInitFunction(&__bif_##file##_init); + extern void __bif_##file##_init(plugin::Plugin*); \ + __AddBifInitFunction(&__bif_##file##_init); /** * Defines a component implementing a protocol analyzer. diff --git a/src/plugin/Manager.cc b/src/plugin/Manager.cc index 67f4dea2bd..ddd953ab22 100644 --- a/src/plugin/Manager.cc +++ b/src/plugin/Manager.cc @@ -1,11 +1,23 @@ // See the file "COPYING" in the main distribution directory for copyright. +#include +#include +#include +#include +#include +#include + #include "Manager.h" #include "../Reporter.h" +#include "../Func.h" +#include "../Event.h" using namespace plugin; +string Manager::current_dir; +string Manager::current_sopath; + Manager::Manager() { init = false; @@ -16,18 +28,162 @@ Manager::~Manager() assert(! init); } -bool Manager::LoadPlugin(const std::string& path) +void Manager::LoadPluginsFrom(const string& dir) { assert(! init); - reporter->InternalError("plugin::Manager::LoadPlugin not yet implemented"); - return false; + + if ( dir.empty() ) + return; + + if ( dir.find(":") != string::npos ) + { + // Split at ":". + std::stringstream s(dir); + std::string d; + + while ( std::getline(s, d, ':') ) + LoadPluginsFrom(d); + + return; + } + + if ( ! is_dir(dir) ) + { + DBG_LOG(DBG_PLUGINS, "Not a valid plugin directory: %s", dir.c_str()); + return; + } + + int rc = LoadPlugin(dir); + + if ( rc >= 0 ) + return; + + DBG_LOG(DBG_PLUGINS, "Searching directory %s recursively for plugins", dir.c_str()); + + DIR* d = opendir(dir.c_str()); + + if ( ! d ) + { + DBG_LOG(DBG_PLUGINS, "Cannot open directory %s", dir.c_str()); + return; + } + + bool found = false; + + struct dirent *dp; + + while ( (dp = readdir(d)) ) + { + struct stat st; + + if ( strcmp(dp->d_name, "..") == 0 + || strcmp(dp->d_name, ".") == 0 ) + continue; + + string path = dir + "/" + dp->d_name; + + if( stat(path.c_str(), &st) < 0 ) + { + DBG_LOG(DBG_PLUGINS, "Cannot stat %s: %s", path.c_str(), strerror(errno)); + continue; + } + + if ( st.st_mode & S_IFDIR ) + LoadPluginsFrom(path); + } } -bool Manager::LoadPluginsFrom(const std::string& dir) +int Manager::LoadPlugin(const std::string& dir) { assert(! init); - reporter->InternalError("plugin::Manager::LoadPluginsFrom not yet implemented"); - return false; + + // Check if it's a plugin dirctory. + if ( ! is_file(dir + "/__bro_plugin__") ) + return -1; + + DBG_LOG(DBG_PLUGINS, "Loading plugin from %s", dir.c_str()); + + // Add the "scripts" and "bif" directories to BROPATH. + string scripts = dir + "/scripts"; + + if ( is_dir(scripts) ) + { + DBG_LOG(DBG_PLUGINS, " Adding %s to BROPATH", scripts.c_str()); + add_to_bro_path(scripts); + } + + string bif = dir + "/bif"; + + if ( is_dir(bif) ) + { + DBG_LOG(DBG_PLUGINS, " Adding %s to BROPATH", bif.c_str()); + add_to_bro_path(bif); + } + + // Load dylib/scripts/__load__.bro automatically. + string dyinit = dir + "/dylib/scripts/__load__.bro"; + + if ( is_file(dyinit) ) + { + DBG_LOG(DBG_PLUGINS, " Adding %s for loading", dyinit.c_str()); + add_input_file(dyinit.c_str()); + } + + // Load scripts/__load__.bro automatically. + string init = scripts + "/__load__.bro"; + + if ( is_file(init) ) + { + DBG_LOG(DBG_PLUGINS, " Adding %s for loading", init.c_str()); + add_input_file(init.c_str()); + } + + // Load bif/__load__.bro automatically. + init = bif + "/__load__.bro"; + + if ( is_file(init) ) + { + DBG_LOG(DBG_PLUGINS, " Adding %s for loading", init.c_str()); + add_input_file(init.c_str()); + } + + // Load shared libraries. + + string dypattern = dir + "/dylib/*." + HOST_ARCHITECTURE + SHARED_LIBRARY_SUFFIX; + + DBG_LOG(DBG_PLUGINS, " Searching for shared libraries %s", dypattern.c_str()); + + glob_t gl; + + if ( glob(dypattern.c_str(), 0, 0, &gl) == 0 ) + { + for ( size_t i = 0; i < gl.gl_pathc; i++ ) + { + const char* path = gl.gl_pathv[i]; + + current_dir = dir; + current_sopath = path; + void* hdl = dlopen(path, RTLD_LAZY | RTLD_GLOBAL); + current_dir.clear(); + current_sopath.clear(); + + if ( ! hdl ) + { + const char* err = dlerror(); + reporter->FatalError("cannot load plugin library %s: %s", path, err ? err : ""); + } + + DBG_LOG(DBG_PLUGINS, " Loaded %s", path); + } + } + + else + { + DBG_LOG(DBG_PLUGINS, " No shared library found"); + return 1; + } + + return 1; } static bool plugin_cmp(const Plugin* a, const Plugin* b) @@ -39,22 +195,61 @@ bool Manager::RegisterPlugin(Plugin *plugin) { Manager::PluginsInternal()->push_back(plugin); + if ( current_dir.size() && current_sopath.size() ) + plugin->SetPluginLocation(current_dir.c_str(), current_sopath.c_str()); + // Sort plugins by name to make sure we have a deterministic order. PluginsInternal()->sort(plugin_cmp); return true; } +static bool interpreter_plugin_cmp(const InterpreterPlugin* a, const InterpreterPlugin* b) + { + if ( a->Priority() == b->Priority() ) + return a->Name() < b->Name(); + + // Reverse sort. + return a->Priority() > b->Priority(); + } + void Manager::InitPreScript() { assert(! init); for ( plugin_list::iterator i = Manager::PluginsInternal()->begin(); i != Manager::PluginsInternal()->end(); i++ ) - (*i)->InitPreScript(); + { + Plugin* plugin = *i; + + if ( plugin->PluginType() == Plugin::INTERPRETER ) + interpreter_plugins.push_back(dynamic_cast(plugin)); + + plugin->InitPreScript(); + + // Track the file extensions the plugin can handle. + std::stringstream ext(plugin->FileExtensions()); + + // Split at ":". + std::string e; + + while ( std::getline(ext, e, ':') ) + { + DBG_LOG(DBG_PLUGINS, "Plugin %s handles *.%s", plugin->Name(), e.c_str()); + extensions.insert(std::make_pair(e, plugin)); + } + } + + interpreter_plugins.sort(interpreter_plugin_cmp); init = true; } +void Manager::InitBifs() + { + for ( plugin_list::iterator i = Manager::PluginsInternal()->begin(); i != Manager::PluginsInternal()->end(); i++ ) + (*i)->InitBifs(); + } + void Manager::InitPostScript() { assert(init); @@ -78,6 +273,28 @@ void Manager::FinishPlugins() init = false; } +int Manager::TryLoadFile(const char* file) + { + assert(file); + const char* ext = strrchr(file, '.'); + + if ( ! ext ) + return -1; + + extension_map::iterator i = extensions.find(++ext); + if ( i == extensions.end() ) + return -1; + + Plugin* plugin = i->second; + + DBG_LOG(DBG_PLUGINS, "Loading %s with %s", file, plugin->Name()); + + if ( i->second->LoadFile(file) ) + return 1; + + return 0; + } + Manager::plugin_list Manager::Plugins() const { return *Manager::PluginsInternal(); @@ -92,3 +309,67 @@ Manager::plugin_list* Manager::PluginsInternal() return plugins; } + +Val* Manager::CallFunction(const Func* func, val_list* args) const + { + Val* result = 0; + + for ( interpreter_plugin_list::const_iterator i = interpreter_plugins.begin(); + i != interpreter_plugins.end() && ! result; i++ ) + { + result = (*i)->CallFunction(func, args); + + if ( result ) + { + DBG_LOG(DBG_PLUGINS, "Plugin %s replaced call to %s", (*i)->Name(), func->Name()); + return result; + } + } + + return 0; + } + +bool Manager::QueueEvent(Event* event) const + { + for ( interpreter_plugin_list::const_iterator i = interpreter_plugins.begin(); + i != interpreter_plugins.end(); i++ ) + { + if ( (*i)->QueueEvent(event) ) + { + DBG_LOG(DBG_PLUGINS, "Plugin %s handled queueing of event %s", (*i)->Name(), event->Handler()->Name()); + return true; + } + } + + return false; + } + + +void Manager::UpdateNetworkTime(double network_time) const + { + for ( interpreter_plugin_list::const_iterator i = interpreter_plugins.begin(); + i != interpreter_plugins.end(); i++ ) + (*i)->UpdateNetworkTime(network_time); + } + +void Manager::DrainEvents() const + { + for ( interpreter_plugin_list::const_iterator i = interpreter_plugins.begin(); + i != interpreter_plugins.end(); i++ ) + (*i)->DrainEvents(); + } + +void Manager::DisableInterpreterPlugin(const InterpreterPlugin* plugin) + { + for ( interpreter_plugin_list::iterator i = interpreter_plugins.begin(); + i != interpreter_plugins.end(); i++ ) + { + if ( *i == plugin ) + { + interpreter_plugins.erase(i); + return; + } + } + } + + diff --git a/src/plugin/Manager.h b/src/plugin/Manager.h index 2bbcaeb0f1..85b60fb763 100644 --- a/src/plugin/Manager.h +++ b/src/plugin/Manager.h @@ -3,6 +3,8 @@ #ifndef PLUGIN_MANAGER_H #define PLUGIN_MANAGER_H +#include + #include "Plugin.h" #include "Component.h" @@ -17,6 +19,7 @@ class Manager { public: typedef std::list plugin_list; + typedef std::list interpreter_plugin_list; typedef Plugin::component_list component_list; /** @@ -30,24 +33,17 @@ public: ~Manager(); /** - * Loads a plugin dynamically from a file. This must be called only - * before InitPluginsPreScript() + * Loads all plugins dynamically from a set of directories. Multiple + * directories are split by ':'. If a directory does not contain a + * plugin itself, the method searches for plugins recursively. For + * plugins found, the method loads the plugin's shared library and + * makes its scripts available to the interpreter. * - * This is not currently implemented. - * - * @param file The path to the plugin to load. - */ - bool LoadPlugin(const std::string& file); - - /** - * Loads plugins dynamically found in a directory. This must be - * called only before InitPluginsPreScript(). - * - * This is not currently implemented. + * This must be called only before InitPluginsPreScript(). * * @param dir The directory to search for plugins. */ - bool LoadPluginsFrom(const std::string& dir); + void LoadPluginsFrom(const std::string& dir); /** * First-stage initializion of the manager. This is called early on @@ -57,7 +53,13 @@ public: void InitPreScript(); /** - * Second-stage initialization of the manager. This is called late + * Second-stage initialization of the manager. This is called in + * between pre- and post-script to make BiFs available. + */ + void InitBifs(); + + /** + * Third-stage initialization of the manager. This is called late * during Bro's initialization after any scripts are processed, and * forwards to the corresponding Plugin methods. */ @@ -69,6 +71,22 @@ public: */ void FinishPlugins(); + /** + * This tries to load the given file by searching for a plugin that + * support that extension. If a correspondign plugin is found, it's + * asked to loead the file. If that fails, the method reports an + * error message. + * + * This method must be called only between InitPreScript() and + * InitPostScript(). + * + * @return 1 if the file was sucessfully loaded by a plugin; 0 if a + * plugin was found that supports the file's extension, yet it + * encountered a problem loading the file; and -1 if we don't have a + * plugin that supports this extension. + */ + int TryLoadFile(const char* file); + /** * Returns a list of all available plugins. This includes all that * are compiled in statically, as well as those loaded dynamically so @@ -83,6 +101,52 @@ public: */ template std::list Components() const; + /** + * Filters a function/event/hook call through all interpreter plugins. + * + * @param func The function to be called. + * + * @param args The function call's arguments; they may be modified. + * + * @return If a plugin handled the call, a +1 Val with the result + * value to pass back to the interpreter (for void functions and + * events, it may be any Val and must be ignored). If no plugin + * handled the call, the method returns null. + */ + Val* CallFunction(const Func* func, val_list* args) const; + + /** + * Filter the queuing of an event through all interpreter plugins. + * + * @param event The event to be queued; it may be modified. + * + * @return Returns true if a plugin handled the queuing; in that case the + * plugin will have taken ownership. + * + */ + bool QueueEvent(Event* event) const; + + /** + * Informs all interpreter plugins about an update in network time. + * + * @param networkt_time The new network time. + */ + void UpdateNetworkTime(double network_time) const; + + /** + * Informs all interpreter plugins that the event queue has been drained. + */ + void DrainEvents() const; + + /** + * Disables an interpreter plugin's hooking of the script interpreter. + * The remaining functionality of the Plugin base class remains + * available. + * + * @param plugin The plugin to disable. + */ + void DisableInterpreterPlugin(const InterpreterPlugin* plugin); + /** * Internal method that registers a freshly instantiated plugin with * the manager. @@ -91,12 +155,38 @@ public: * ownership, yet assumes the pointer will stay valid at least until * the Manager is destroyed. */ - static bool RegisterPlugin(Plugin *plugin); + static bool RegisterPlugin(Plugin* plugin); + +protected: + /** + * Loads a plugin dynamically from a given directory. It loads the + * plugin's shared library, and makes its scripts available to the + * interpreter. Different from LoadPluginsFrom() this method does not + * further descend the directory tree recursively to search for + * plugins. + * + * This must be called only before InitPluginsPreScript() + * + * @param file The path to the plugin to load. + * + * @return 0 if there's a plugin in this directory, but there was a + * problem loading it; -1 if there's no plugin at all in this + * directory; 1 if there's a plugin in this directory and we loaded + * it successfully. + */ + int LoadPlugin(const std::string& dir); private: static plugin_list* PluginsInternal(); bool init; + typedef std::map extension_map; + extension_map extensions; + + interpreter_plugin_list interpreter_plugins; + + static string current_dir; + static string current_sopath; }; template diff --git a/src/plugin/Plugin.cc b/src/plugin/Plugin.cc index eaac8a3b25..ff3cf8f414 100644 --- a/src/plugin/Plugin.cc +++ b/src/plugin/Plugin.cc @@ -10,9 +10,9 @@ using namespace plugin; -BifItem::BifItem(const std::string& arg_id, Type arg_type) +BifItem::BifItem(const char* arg_id, Type arg_type) { - id = copy_string(arg_id.c_str()); + id = copy_string(arg_id); type = arg_type; } @@ -47,6 +47,9 @@ Plugin::Plugin() version = -9999; api_version = -9999; dynamic = false; + base_dir = 0; + sopath = 0; + extensions = 0; Manager::RegisterPlugin(this); } @@ -57,6 +60,14 @@ Plugin::~Plugin() delete [] name; delete [] description; + delete [] base_dir; + delete [] sopath; + delete [] extensions; + } + +Plugin::Type Plugin::PluginType() const + { + return STANDARD; } const char* Plugin::Name() const @@ -66,6 +77,7 @@ const char* Plugin::Name() const void Plugin::SetName(const char* arg_name) { + delete [] name; name = copy_string(arg_name); } @@ -76,6 +88,7 @@ const char* Plugin::Description() const void Plugin::SetDescription(const char* arg_description) { + delete [] description; description = copy_string(arg_description); } @@ -99,6 +112,16 @@ bool Plugin::DynamicPlugin() const return dynamic; } +const char* Plugin::PluginDirectory() const + { + return base_dir; + } + +const char* Plugin::PluginPath() const + { + return sopath; + } + void Plugin::SetAPIVersion(int arg_version) { api_version = arg_version; @@ -109,22 +132,26 @@ void Plugin::SetDynamicPlugin(bool arg_dynamic) dynamic = arg_dynamic; } +void Plugin::SetPluginLocation(const char* arg_dir, const char* arg_sopath) + { + delete [] base_dir; + delete [] sopath; + base_dir = copy_string(arg_dir); + sopath = copy_string(arg_sopath); + } + void Plugin::InitPreScript() { } void Plugin::InitPostScript() { - for ( bif_init_func_list::const_iterator f = bif_inits.begin(); f != bif_inits.end(); f++ ) - { - bif_init_func_result items = (**f)(); + } - for ( bif_init_func_result::const_iterator i = items.begin(); i != items.end(); i++ ) - { - BifItem bi((*i).first, (BifItem::Type)(*i).second); - bif_items.push_back(bi); - } - } +void Plugin::InitBifs() + { + for ( bif_init_func_list::const_iterator f = bif_inits.begin(); f != bif_inits.end(); f++ ) + (**f)(this); } Plugin::bif_item_list Plugin::BifItems() const @@ -143,6 +170,22 @@ Plugin::bif_item_list Plugin::CustomBifItems() const return bif_item_list(); } +const char* Plugin::FileExtensions() const + { + return extensions ? extensions : ""; + } + +void Plugin::SetFileExtensions(const char* ext) + { + extensions = copy_string(ext); + } + +bool Plugin::LoadFile(const char* file) + { + reporter->InternalError("Plugin::LoadFile not overriden for %s", file); + return false; + } + void Plugin::Done() { for ( component_list::const_iterator i = components.begin(); i != components.end(); i++ ) @@ -170,11 +213,23 @@ void Plugin::AddComponent(Component* c) components.sort(component_cmp); } -void Plugin::AddBifInitFunction(bif_init_func c) +bool Plugin::LoadBroFile(const char* file) + { + add_input_file(file); + return true; + } + +void Plugin::__AddBifInitFunction(bif_init_func c) { bif_inits.push_back(c); } +void Plugin::AddBifItem(const char* name, BifItem::Type type) + { + BifItem bi(name, (BifItem::Type)type); + bif_items.push_back(bi); + } + void Plugin::Describe(ODesc* d) const { d->Add("Plugin: "); @@ -190,7 +245,7 @@ void Plugin::Describe(ODesc* d) const { if ( version > 0 ) { - d->Add(" (version "); + d->Add(" (dynamic, version "); d->Add(version); d->Add(")"); } @@ -201,6 +256,18 @@ void Plugin::Describe(ODesc* d) const else d->Add(" (built-in)"); + switch ( PluginType() ) { + case STANDARD: + break; + + case INTERPRETER: + d->Add( " (interpreter plugin)"); + break; + + default: + reporter->InternalError("unknown plugin type in Plugin::Describe"); + } + d->Add("\n"); if ( d->IsShort() ) @@ -252,4 +319,46 @@ void Plugin::Describe(ODesc* d) const } } +InterpreterPlugin::InterpreterPlugin(int arg_priority) + { + priority = arg_priority; + } + +InterpreterPlugin::~InterpreterPlugin() + { + } + +int InterpreterPlugin::Priority() const + { + return priority; + } + +Plugin::Type InterpreterPlugin::PluginType() const + { + return INTERPRETER; + } + +Val* InterpreterPlugin::CallFunction(const Func* func, val_list* args) + { + return 0; + } + +bool InterpreterPlugin::QueueEvent(Event* event) + { + return false; + } + +void InterpreterPlugin::UpdateNetworkTime(double network_time) + { + } + +void InterpreterPlugin::DrainEvents() + { + } + +void InterpreterPlugin::DisableInterpreterPlugin() const + { + plugin_mgr->DisableInterpreterPlugin(this); + } + diff --git a/src/plugin/Plugin.h b/src/plugin/Plugin.h index 4abd260550..306c97e59e 100644 --- a/src/plugin/Plugin.h +++ b/src/plugin/Plugin.h @@ -9,6 +9,8 @@ #include "Macros.h" class ODesc; +class Func; +class Event; namespace plugin { @@ -22,8 +24,6 @@ class BifItem { public: /** * Type of the item. - * - * The values here must match the integers that \c bifcl generated. */ enum Type { FUNCTION = 1, EVENT = 2, CONSTANT = 3, GLOBAL = 4, TYPE = 5 }; @@ -35,7 +35,7 @@ public: * * @param type The type of the item. */ - BifItem(const std::string& id, Type type); + BifItem(const char* id, Type type); /** * Copy constructor. @@ -88,6 +88,26 @@ class Plugin { public: typedef std::list component_list; typedef std::list bif_item_list; + typedef std::list > bif_init_func_result; + typedef void (*bif_init_func)(Plugin *); + + /** + * Type of a plugin. Plugin types are set implicitly by deriving from + * the corresponding base class. */ + enum Type { + /** + * A standard plugin. This is the type for all plugins + * derived directly from \a Plugin. */ + STANDARD, + + /** + * An interpreter plugin. These plugins get hooked into the + * script interpreter and can modify, or even replace, its + * execution of Bro script code. To create an interpreter + * plugin, derive from \aInterpreterPlugin. + */ + INTERPRETER + }; /** * Constructor. @@ -99,6 +119,11 @@ public: */ virtual ~Plugin(); + /** + * Returns the type of the plugin. + */ + virtual Type PluginType() const; + /** * Returns the name of the plugin. */ @@ -121,6 +146,23 @@ public: */ bool DynamicPlugin() const; + /** + * Returns a colon-separated list of file extensions the plugin handles. + */ + const char* FileExtensions() const; + + /** + * For dynamic plugins, returns the base directory from which it was + * loaded. For static plugins, returns null. + **/ + const char* PluginDirectory() const; + + /** + * For dynamic plugins, returns the full path to the shared library + * from which it was loaded. For static plugins, returns null. + **/ + const char* PluginPath() const; + /** * Returns the internal API version that this plugin relies on. Only * plugins that match Bro's current API version may be used. For @@ -173,9 +215,38 @@ public: */ void Describe(ODesc* d) const; + /** + * Registering an individual BiF that the plugin defines. The + * information is for informational purpuses only and will show up in + * the result of BifItems() as well as in the Describe() output. + * Another way to add this information is via overriding + * CustomBifItems(). + * + * @param name The name of the BiF item. + * + * @param type The item's type. + */ + void AddBifItem(const char* name, BifItem::Type type); + + /** + * Adds a file to the list of files Bro loads at startup. This will + * normally be a Bro script, but it passes through the plugin system + * as well to load files with other extensions as supported by any of + * the current plugins. In other words, calling this method is + * similar to given a file on the command line. Note that the file + * may be only queued for now, and actually loaded later. + * + * This method must not be called after InitPostScript(). + * + * @param file The file to load. It will be searched along the standard paths. + * + * @return True if successful (which however may only mean + * "successfully queued"). + */ + bool LoadBroFile(const char* file); + protected: - typedef std::list > bif_init_func_result; - typedef bif_init_func_result (*bif_init_func)(); + friend class Manager; /** * Sets the plugins name. @@ -212,6 +283,28 @@ protected: */ void SetDynamicPlugin(bool dynamic); + /** + * Reports the extensions of input files the plugin handles. If Bro + * wants to load a file with one of these extensions it will pass + * them to LoadFile() and then then ignore otherwise. + * + * ext: A list of colon-separated file extensions the plugin handles. + */ + void SetFileExtensions(const char* ext); + + /** + * Sets the base directory and shared library path from which the + * plugin was loaded. This should be called only from the manager for + * dynamic plugins. + * + * @param dir The plugin directory. The functions makes an internal + * copy of string. + * + * @param sopath The full path the shared library loaded. The + * functions makes an internal copy of string. + */ + void SetPluginLocation(const char* dir, const char* sopath); + /** * Takes ownership. */ @@ -227,17 +320,41 @@ protected: */ virtual bif_item_list CustomBifItems() const; + /** + * Virtual method that can be overriden by derived class to load + * files with extensions reported via SetFileExtension(). + * + * This method will be called between InitPreScript() and + * InitPostScript(), but with no further order or timing guaranteed. + * It will be called once for each file encountered with of the + * specificed extensions (i.e., duplicates are filtered out + * automatically). + * + * @return True if the file was loaded successfuly, false if not. Bro + * will abort in the latter case. + */ + virtual bool LoadFile(const char* file); + + /** + * Initializes the BiF items added with AddBifItem(). Internal method + * that will be called by the manager at the right time. + */ + void InitBifs(); + /** * Internal function adding an entry point for registering * auto-generated BiFs. */ - void AddBifInitFunction(bif_init_func c); + void __AddBifInitFunction(bif_init_func c); private: typedef std::list bif_init_func_list; const char* name; const char* description; + const char* base_dir; + const char* sopath; + const char* extensions; int version; int api_version; bool dynamic; @@ -247,6 +364,105 @@ private: bif_init_func_list bif_inits; }; +/** + * Class for hooking into script execution. An interpreter plugin can do + * everything a normal plugin can, yet will also be interfaced to the script + * interpreter for learning about, modidying, or potentially replacing + * standard functionality. + */ +class InterpreterPlugin : public Plugin { +public: + /** + * Constructor. + * + * @param priority Imposes an order on InterpreterPlugins in which + * they'll be chained. The higher the prioritu, the earlier a plugin + * is called when the interpreter goes through the chain. */ + InterpreterPlugin(int priority); + + /** + * Destructor. + */ + virtual ~InterpreterPlugin(); + + /** + * Returns the plugins priority. + */ + int Priority() const; + + /** + * Callback for executing a function/event/hook. Whenever the script + * interpreter is about to execution a function, it first gives all + * InterpreterPlugins a chance to handle the call (in the order of their + * priorities). A plugin can either just inspect the call, or replace it + * (i.e., prevent the interpreter from executing it). In the latter case + * it must provide a matching return value. + * + * The default implementation does never handle the call in any way. + * + * @param func The function being called. + * + * @param args The function arguments. The method can modify the list + * in place long as it ensures matching types and correct reference + * counting. + * + * @return If the plugin handled the call, a +1 Val with the result + * value to pass back to the interpreter (for void functions and + * events any \a Val is fine; it will be ignored; best to use a \c + * TYPE_ANY). If the plugin did not handle the call, it must return + * null. + */ + virtual Val* CallFunction(const Func* func, val_list* args); + + /** + * Callback for raising an event. Whenever the script interpreter is + * about to queue an event for later execution, it first gives all + * InterpreterPlugins a chance to handle the queuing otherwise (in the + * order of their priorities). A plugin can either just inspect the + * event, or take it over (i.e., prevent the interpreter from queuing it + * it). + * + * The default implementation does never handle the queuing in any way. + * + * @param event The even to be queued. The method can modify it in in + * place long as it ensures matching types and correct reference + * counting. + * + * @return True if the plugin took charge of the event; in that case it + * must have assumed ownership of the event and the intpreter will not do + * anything further with it. False otherwise. + * + */ + virtual bool QueueEvent(Event* event); + + /** + * Callback for updates in network time. This method will be called + * whenever network time is advanced. + * + * @param networkt_time The new network time. + */ + virtual void UpdateNetworkTime(double network_time); + + /** + * Callback for event queue draining. This method will be called + * whenever the event manager has drained it queue. + */ + virtual void DrainEvents(); + + /** + * Disables interpreter hooking. The functionality of the Plugin base + * class remains available. + */ + void DisableInterpreterPlugin() const; + + // Overridden from base class. + virtual Type PluginType() const; + +private: + int priority; + +}; + } #endif diff --git a/src/scan.l b/src/scan.l index 636ec5b251..4c67a2f75e 100644 --- a/src/scan.l +++ b/src/scan.l @@ -30,6 +30,8 @@ #include "analyzer/Analyzer.h" +#include "plugin/Manager.h" + extern YYLTYPE yylloc; // holds start line and column of token extern int print_loaded_scripts; extern int generate_documentation; @@ -579,6 +581,24 @@ YYLTYPE GetCurrentLocation() static int load_files(const char* orig_file) { + int rc = plugin_mgr->TryLoadFile(orig_file); + + if ( rc == 1 ) + return 0; // A plugin took care of it, just skip. + + if ( rc == 0 ) + { + if ( ! reporter->Errors() ) + // This is just in case the plugin failed to report + // the error itself, in which case we want to at + // least tell the user that something went wrong. + reporter->Error("Plugin reported error loading %s", orig_file); + + exit(1); + } + + assert(rc == -1); // No plugin in charge of this file. + // Whether we pushed on a FileInfo that will restore the // current module after the final file has been scanned. bool did_module_restore = false; diff --git a/src/util-config.h.in b/src/util-config.h.in index ff5e28537a..90919a63f4 100644 --- a/src/util-config.h.in +++ b/src/util-config.h.in @@ -3,3 +3,4 @@ #define BRO_BUILD_SOURCE_PATH "@CMAKE_BINARY_DIR@/src" #define BRO_BUILD_SCRIPTS_PATH "@CMAKE_BINARY_DIR@/scripts" #define BRO_MAGIC_INSTALL_PATH "@BRO_MAGIC_INSTALL_PATH@" +#define BRO_PLUGIN_INSTALL_PATH "@BRO_PLUGIN_INSTALL_PATH@" diff --git a/src/util.cc b/src/util.cc index dd232a83fa..3b917495bc 100644 --- a/src/util.cc +++ b/src/util.cc @@ -619,13 +619,13 @@ bool ensure_dir(const char *dirname) return true; } -bool is_dir(const char* path) +bool is_dir(const std::string& path) { struct stat st; - if ( stat(path, &st) < 0 ) + if ( stat(path.c_str(), &st) < 0 ) { if ( errno != ENOENT ) - reporter->Warning("can't stat %s: %s", path, strerror(errno)); + reporter->Warning("can't stat %s: %s", path.c_str(), strerror(errno)); return false; } @@ -633,6 +633,37 @@ bool is_dir(const char* path) return S_ISDIR(st.st_mode); } +bool is_file(const std::string& path) + { + struct stat st; + if ( stat(path.c_str(), &st) < 0 ) + { + if ( errno != ENOENT ) + reporter->Warning("can't stat %s: %s", path.c_str(), strerror(errno)); + + return false; + } + + return S_ISREG(st.st_mode); + } + +string strreplace(const string& s, const string& o, const string& n) + { + string r = s; + + while ( true ) + { + size_t i = r.find(o); + + if ( i == std::string::npos ) + break; + + r.replace(i, o.size(), n); + } + + return r; +} + int hmac_key_set = 0; uint8 shared_hmac_md5_key[16]; @@ -872,16 +903,31 @@ int int_list_cmp(const void* v1, const void* v2) return 1; } -const char* bro_path() +static string bro_path_value; + +const std::string& bro_path() { - const char* path = getenv("BROPATH"); - if ( ! path ) - path = ".:" + if ( bro_path_value.empty() ) + { + const char* path = getenv("BROPATH"); + if ( ! path ) + path = ".:" BRO_SCRIPT_INSTALL_PATH ":" BRO_SCRIPT_INSTALL_PATH "/policy" ":" BRO_SCRIPT_INSTALL_PATH "/site"; - return path; + bro_path_value = path; + } + + return bro_path_value; + } + +extern void add_to_bro_path(const string& dir) + { + // Make sure path is initialized. + bro_path(); + + bro_path_value += string(":") + dir; } const char* bro_magic_path() @@ -894,6 +940,16 @@ const char* bro_magic_path() return path; } +const char* bro_plugin_path() + { + const char* path = getenv("BRO_PLUGINS"); + + if ( ! path ) + path = BRO_PLUGIN_INSTALL_PATH; + + return path; + } + string bro_prefixes() { string rval; @@ -1089,9 +1145,9 @@ FILE* search_for_file(const char* filename, const char* ext, // @loads can be referenced relatively. if ( current_scanned_file_path != "" && filename[0] == '.' ) safe_snprintf(path, sizeof(path), "%s:%s", - current_scanned_file_path.c_str(), bro_path()); + current_scanned_file_path.c_str(), bro_path().c_str()); else - safe_strncpy(path, bro_path(), sizeof(path)); + safe_strncpy(path, bro_path().c_str(), sizeof(path)); char* dir_beginning = path; char* dir_ending = path; @@ -1505,25 +1561,16 @@ void get_memory_usage(unsigned int* total, unsigned int* malloced) if ( malloced ) *malloced = mi.uordblks; - ret_total = mi.arena; +#endif - if ( total ) - *total = ret_total; -#else struct rusage r; getrusage(RUSAGE_SELF, &r); - if ( malloced ) - *malloced = 0; - // At least on FreeBSD it's in KB. ret_total = r.ru_maxrss * 1024; if ( total ) *total = ret_total; -#endif - - // return ret_total; } #ifdef malloc diff --git a/src/util.h b/src/util.h index fcdfd6d499..a7d493485d 100644 --- a/src/util.h +++ b/src/util.h @@ -147,7 +147,13 @@ extern const char* fmt_access_time(double time); extern bool ensure_dir(const char *dirname); // Returns true if path exists and is a directory. -bool is_dir(const char* path); +bool is_dir(const std::string& path); + +// Returns true if path exists and is a file. +bool is_file(const std::string& path); + +// Replaces all occurences of *o* in *s* with *n*. +extern std::string strreplace(const std::string& s, const std::string& o, const std::string& n); extern uint8 shared_hmac_md5_key[16]; @@ -202,9 +208,12 @@ static const SourceID SOURCE_LOCAL = 0; extern void pinpoint(); extern int int_list_cmp(const void* v1, const void* v2); -extern const char* bro_path(); +extern const std::string& bro_path(); +extern void add_to_bro_path(const std::string& dir); + extern const char* bro_magic_path(); -extern std::string bro_prefixes(); +extern const char* bro_plugin_path(); +extern const char* bro_prefixes(); std::string dot_canon(std::string path, std::string file, std::string prefix = ""); const char* normalize_path(const char* path); void get_script_subpath(const std::string& full_filename, const char** subpath); diff --git a/testing/btest/Baseline/core.plugins.analyzer/output b/testing/btest/Baseline/core.plugins.analyzer/output new file mode 100644 index 0000000000..deb8a7655d --- /dev/null +++ b/testing/btest/Baseline/core.plugins.analyzer/output @@ -0,0 +1,6 @@ +Plugin: Demo::Foo - A Foo test analyzer (dynamic, version 1) + [Analyzer] Foo (ANALYZER_FOO, enabled) + [Event] foo_message + +=== +foo_message, [orig_h=::1, orig_p=37927/tcp, resp_h=::1, resp_p=4242/tcp], Hello, Foo!^J diff --git a/testing/btest/Baseline/core.plugins.test-plugin/output b/testing/btest/Baseline/core.plugins.test-plugin/output new file mode 100644 index 0000000000..e0663ab42e --- /dev/null +++ b/testing/btest/Baseline/core.plugins.test-plugin/output @@ -0,0 +1,11 @@ +Plugin: Demo::Foo - (dynamic, version 1) + [Event] plugin_event + [Function] hello_plugin_world + +=== +plugin: automatically loaded at startup +calling bif, Hello from the plugin! +=== +plugin: automatically loaded at startup +calling bif, Hello from the plugin! +plugin: manually loaded diff --git a/testing/btest/Traces/port4242.trace b/testing/btest/Traces/port4242.trace new file mode 100644 index 0000000000000000000000000000000000000000..e999b43b92fd875862e79db6ddca132548fe2413 GIT binary patch literal 868 zcmca|c+)~A1{MYw`2U}Qff2}goBcS@;yx!s9FPsd$e`_R0*I@@<^ZCQ07xS;V4R{Z zFsV{7yBj39fN9k#MxY8Nmj6dsm^ipj@AK6Ki8C@Yvx964qu4fqNmJCnJ?4x9+Ss!| zXw|Cc2T^SU*$A_(5olW&&~^|;22j5#5bd`fppEej0*qlhKLh>32=N!lCXkukqcy_&;v^S1r5b`U?|Q2`|CNXjUZcs wD7FQfGG>57PY@V-&oFJ7O0g|Sewo<-vgIkJE!!xz1?rcXK))n_Lk!Ip07J)|iU0rr literal 0 HcmV?d00001 diff --git a/testing/btest/core/plugins/analyzer-plugin/.btest-ignore b/testing/btest/core/plugins/analyzer-plugin/.btest-ignore new file mode 100644 index 0000000000..e69de29bb2 diff --git a/testing/btest/core/plugins/analyzer-plugin/CMakeLists.txt b/testing/btest/core/plugins/analyzer-plugin/CMakeLists.txt new file mode 100644 index 0000000000..a032edbd89 --- /dev/null +++ b/testing/btest/core/plugins/analyzer-plugin/CMakeLists.txt @@ -0,0 +1,20 @@ + +project(Bro-Plugin-Demo-Foo) + +cmake_minimum_required(VERSION 2.6.3) + +if ( NOT BRO_DIST ) + message(FATAL_ERROR "BRO_DIST not set") +endif () + +set(CMAKE_MODULE_PATH ${BRO_DIST}/cmake) + +include(BroPlugin) + +bro_plugin_begin(Demo Foo) +bro_plugin_cc(src/Plugin.cc) +bro_plugin_cc(src/Foo.cc) +bro_plugin_bif(src/events.bif) +bro_plugin_bif(src/functions.bif) +bro_plugin_pac(src/foo.pac src/foo-protocol.pac src/foo-analyzer.pac) +bro_plugin_end() diff --git a/testing/btest/core/plugins/analyzer-plugin/__bro_plugin__ b/testing/btest/core/plugins/analyzer-plugin/__bro_plugin__ new file mode 100644 index 0000000000..c10d6ee17c --- /dev/null +++ b/testing/btest/core/plugins/analyzer-plugin/__bro_plugin__ @@ -0,0 +1,2 @@ +A place-holder file that marks a directory as holding a Bro plugin. +Content is ignored. diff --git a/testing/btest/core/plugins/analyzer-plugin/scripts/Demo/Foo/base/main.bro b/testing/btest/core/plugins/analyzer-plugin/scripts/Demo/Foo/base/main.bro new file mode 100644 index 0000000000..2e2d174b47 --- /dev/null +++ b/testing/btest/core/plugins/analyzer-plugin/scripts/Demo/Foo/base/main.bro @@ -0,0 +1,7 @@ + +const ports = { 4242/tcp }; + +event bro_init() &priority=5 + { + Analyzer::register_for_ports(Analyzer::ANALYZER_FOO, ports); + } diff --git a/testing/btest/core/plugins/analyzer-plugin/scripts/__load__.bro b/testing/btest/core/plugins/analyzer-plugin/scripts/__load__.bro new file mode 100644 index 0000000000..330718c604 --- /dev/null +++ b/testing/btest/core/plugins/analyzer-plugin/scripts/__load__.bro @@ -0,0 +1 @@ +@load Demo/Foo/base/main diff --git a/testing/btest/core/plugins/analyzer-plugin/src/Foo.cc b/testing/btest/core/plugins/analyzer-plugin/src/Foo.cc new file mode 100644 index 0000000000..b23c4b7817 --- /dev/null +++ b/testing/btest/core/plugins/analyzer-plugin/src/Foo.cc @@ -0,0 +1,59 @@ + +#include "Foo.h" +#include "foo_pac.h" +#include "events.bif.h" + +#include + +using namespace analyzer::Foo; + +Foo_Analyzer::Foo_Analyzer(Connection* conn) +: tcp::TCP_ApplicationAnalyzer("Foo", conn) + { + interp = new binpac::Foo::Foo_Conn(this); + } + +Foo_Analyzer::~Foo_Analyzer() + { + delete interp; + } + +void Foo_Analyzer::Done() + { + tcp::TCP_ApplicationAnalyzer::Done(); + + interp->FlowEOF(true); + interp->FlowEOF(false); + } + +void Foo_Analyzer::EndpointEOF(bool is_orig) + { + tcp::TCP_ApplicationAnalyzer::EndpointEOF(is_orig); + interp->FlowEOF(is_orig); + } + +void Foo_Analyzer::DeliverStream(int len, const u_char* data, bool orig) + { + tcp::TCP_ApplicationAnalyzer::DeliverStream(len, data, orig); + + assert(TCP()); + + if ( TCP()->IsPartial() ) + // punt on partial. + return; + + try + { + interp->NewData(orig, data, data + len); + } + catch ( const binpac::Exception& e ) + { + ProtocolViolation(fmt("Binpac exception: %s", e.c_msg())); + } + } + +void Foo_Analyzer::Undelivered(int seq, int len, bool orig) + { + tcp::TCP_ApplicationAnalyzer::Undelivered(seq, len, orig); + interp->NewGap(orig, len); + } diff --git a/testing/btest/core/plugins/analyzer-plugin/src/Foo.h b/testing/btest/core/plugins/analyzer-plugin/src/Foo.h new file mode 100644 index 0000000000..dfc50a9d76 --- /dev/null +++ b/testing/btest/core/plugins/analyzer-plugin/src/Foo.h @@ -0,0 +1,31 @@ + +#ifndef BRO_PLUGIN_DEMO_FOO_H +#define BRO_PLUGIN_DEMO_FOO_H + +#include "analyzer/protocol/tcp/TCP.h" +#include "analyzer/protocol/pia/PIA.h" + +namespace binpac { namespace Foo { class Foo_Conn; } } + +namespace analyzer { namespace Foo { + +class Foo_Analyzer : public tcp::TCP_ApplicationAnalyzer { +public: + Foo_Analyzer(Connection* conn); + ~Foo_Analyzer(); + + virtual void Done(); + virtual void DeliverStream(int len, const u_char* data, bool orig); + virtual void Undelivered(int seq, int len, bool orig); + virtual void EndpointEOF(bool is_orig); + + static analyzer::Analyzer* InstantiateAnalyzer(Connection* conn) + { return new Foo_Analyzer(conn); } + +protected: + binpac::Foo::Foo_Conn* interp; +}; + +} } // namespace analyzer::* + +#endif diff --git a/testing/btest/core/plugins/analyzer-plugin/src/Plugin.cc b/testing/btest/core/plugins/analyzer-plugin/src/Plugin.cc new file mode 100644 index 0000000000..b37a60d148 --- /dev/null +++ b/testing/btest/core/plugins/analyzer-plugin/src/Plugin.cc @@ -0,0 +1,12 @@ + +#include + +#include "Foo.h" + +BRO_PLUGIN_BEGIN(Demo, Foo) + BRO_PLUGIN_VERSION(1); + BRO_PLUGIN_DESCRIPTION("A Foo test analyzer"); + BRO_PLUGIN_ANALYZER("Foo", Foo::Foo_Analyzer); + BRO_PLUGIN_BIF_FILE(events); + BRO_PLUGIN_BIF_FILE(functions); +BRO_PLUGIN_END diff --git a/testing/btest/core/plugins/analyzer-plugin/src/events.bif b/testing/btest/core/plugins/analyzer-plugin/src/events.bif new file mode 100644 index 0000000000..4603fe4cf6 --- /dev/null +++ b/testing/btest/core/plugins/analyzer-plugin/src/events.bif @@ -0,0 +1,2 @@ + +event foo_message%(c: connection, data: string%); diff --git a/testing/btest/core/plugins/analyzer-plugin/src/foo-analyzer.pac b/testing/btest/core/plugins/analyzer-plugin/src/foo-analyzer.pac new file mode 100644 index 0000000000..a210a8430c --- /dev/null +++ b/testing/btest/core/plugins/analyzer-plugin/src/foo-analyzer.pac @@ -0,0 +1,15 @@ + +refine connection Foo_Conn += { + + function Foo_data(msg: Foo_Message): bool + %{ + StringVal* data = new StringVal(${msg.data}.length(), (const char*) ${msg.data}.data()); + BifEvent::generate_foo_message(bro_analyzer(), bro_analyzer()->Conn(), data); + return true; + %} + +}; + +refine typeattr Foo_Message += &let { + proc: bool = $context.connection.Foo_data(this); +}; diff --git a/testing/btest/core/plugins/analyzer-plugin/src/foo-protocol.pac b/testing/btest/core/plugins/analyzer-plugin/src/foo-protocol.pac new file mode 100644 index 0000000000..892513c4f0 --- /dev/null +++ b/testing/btest/core/plugins/analyzer-plugin/src/foo-protocol.pac @@ -0,0 +1,4 @@ + +type Foo_Message(is_orig: bool) = record { + data: bytestring &restofdata; +}; diff --git a/testing/btest/core/plugins/analyzer-plugin/src/foo.pac b/testing/btest/core/plugins/analyzer-plugin/src/foo.pac new file mode 100644 index 0000000000..826bcc624e --- /dev/null +++ b/testing/btest/core/plugins/analyzer-plugin/src/foo.pac @@ -0,0 +1,26 @@ +%include binpac.pac +%include bro.pac + +%extern{ +#include "Foo.h" + +#include "events.bif.h" +%} + +analyzer Foo withcontext { + connection: Foo_Conn; + flow: Foo_Flow; +}; + +connection Foo_Conn(bro_analyzer: BroAnalyzer) { + upflow = Foo_Flow(true); + downflow = Foo_Flow(false); +}; + +%include foo-protocol.pac + +flow Foo_Flow(is_orig: bool) { + datagram = Foo_Message(is_orig) withcontext(connection, this); +}; + +%include foo-analyzer.pac diff --git a/testing/btest/core/plugins/analyzer-plugin/src/functions.bif b/testing/btest/core/plugins/analyzer-plugin/src/functions.bif new file mode 100644 index 0000000000..e69de29bb2 diff --git a/testing/btest/core/plugins/analyzer.bro b/testing/btest/core/plugins/analyzer.bro new file mode 100644 index 0000000000..2a72797564 --- /dev/null +++ b/testing/btest/core/plugins/analyzer.bro @@ -0,0 +1,13 @@ +# @TEST-EXEC: ${DIST}/aux/bro-aux/plugin-support/init-plugin Demo Foo +# @TEST-EXEC: cp -r %DIR/analyzer-plugin/* . +# @TEST-EXEC: make BRO=${DIST} +# @TEST-EXEC: BROPLUGINS=`pwd` bro -NN | awk '/^Plugin:.*Demo/ {p=1; print; next} /^Plugin:/{p=0} p==1{print}' >>output +# @TEST-EXEC: echo === >>output +# @TEST-EXEC: BROPLUGINS=`pwd` bro -r $TRACES/port4242.trace %INPUT >>output +# @TEST-EXEC: btest-diff output + +event foo_message(c: connection, data: string) + { + print "foo_message", c$id, data; + } + diff --git a/testing/btest/core/plugins/test-plugin.sh b/testing/btest/core/plugins/test-plugin.sh new file mode 100644 index 0000000000..87b09b2e4d --- /dev/null +++ b/testing/btest/core/plugins/test-plugin.sh @@ -0,0 +1,45 @@ +# @TEST-EXEC: ${DIST}/aux/bro-aux/plugin-support/init-plugin Demo Foo +# @TEST-EXEC: bash %INPUT +# @TEST-EXEC: make BRO=${DIST} +# @TEST-EXEC: BROPLUGINS=`pwd` bro -NN | awk '/^Plugin:.*Demo/ {p=1; print; next} /^Plugin:/{p=0} p==1{print}' >>output +# @TEST-EXEC: echo === >>output +# @TEST-EXEC: BROPLUGINS=`pwd` bro -r $TRACES/empty.trace >>output +# @TEST-EXEC: echo === >>output +# @TEST-EXEC: BROPLUGINS=`pwd` bro demo/foo -r $TRACES/empty.trace >>output +# @TEST-EXEC: btest-diff output + +cat >scripts/__load__.bro <scripts/demo/foo/__load__.bro <scripts/demo/foo/manually.bro <scripts/demo/foo/base/at-startup.bro <src/functions.bif <src/events.bif < Date: Tue, 26 Nov 2013 13:55:58 -0800 Subject: [PATCH 11/56] Restructuring the plugin API to accomodate hooks. I got rid of the earlier separate InterpreterPlugin class. Instead Plugin now has a set of virtual methods HookSomething()... that plugins can override. For efficiency purposes, they however need to register first that they are interested in a hook, otherwise the virtual method will never be called. The idea is to extend the set of hooks over time as we figure out what's useful. This is a checkpoint commit that's essentially untested and probably broken. It compiles, though. --- src/Event.cc | 5 +- src/Func.cc | 2 +- src/Net.cc | 2 +- src/main.cc | 1 - src/plugin/Macros.h | 8 + src/plugin/Manager.cc | 200 +++++++++++-------- src/plugin/Manager.h | 137 ++++++++----- src/plugin/Plugin.cc | 166 ++++++++-------- src/plugin/Plugin.h | 449 ++++++++++++++++++++++-------------------- src/scan.l | 2 +- src/util.h | 2 +- 11 files changed, 529 insertions(+), 445 deletions(-) diff --git a/src/Event.cc b/src/Event.cc index 7774042ac4..6b345a9c5b 100644 --- a/src/Event.cc +++ b/src/Event.cc @@ -78,8 +78,7 @@ EventMgr::~EventMgr() void EventMgr::QueueEvent(Event* event) { - if ( plugin_mgr->QueueEvent(event) ) - return; + PLUGIN_HOOK_VOID(HOOK_QUEUE_EVENT, HookQueueEvent(event)); if ( ! head ) head = tail = event; @@ -119,7 +118,7 @@ void EventMgr::Drain() SegmentProfiler(segment_logger, "draining-events"); - plugin_mgr->DrainEvents(); + PLUGIN_HOOK_VOID(HOOK_DRAIN_EVENTS, HookDrainEvents()); draining = true; while ( head ) diff --git a/src/Func.cc b/src/Func.cc index d8e0ab2c96..f08cf095f0 100644 --- a/src/Func.cc +++ b/src/Func.cc @@ -283,7 +283,7 @@ Val* BroFunc::Call(val_list* args, Frame* parent) const DEBUG_MSG("Function: %s\n", id->Name()); #endif - Val* plugin_result = plugin_mgr->CallFunction(this, args); + Val* plugin_result = PLUGIN_HOOK_WITH_RESULT(HOOK_CALL_FUNCTION, HookCallFunction(this, args), 0); if ( plugin_result ) { diff --git a/src/Net.cc b/src/Net.cc index 6cc148770e..19663b18ed 100644 --- a/src/Net.cc +++ b/src/Net.cc @@ -148,7 +148,7 @@ RETSIGTYPE watchdog(int /* signo */) void net_update_time(double new_network_time) { network_time = new_network_time; - plugin_mgr->UpdateNetworkTime(network_time); + PLUGIN_HOOK_VOID(HOOK_UPDATE_NETWORK_TIME, HookUpdateNetworkTime(new_network_time)); } void net_init(name_list& interfaces, name_list& readfiles, diff --git a/src/main.cc b/src/main.cc index bfbcf77a6d..c3c32a4b14 100644 --- a/src/main.cc +++ b/src/main.cc @@ -224,7 +224,6 @@ void usage() fprintf(stderr, " $BROPATH | file search path (%s)\n", bro_path().c_str()); fprintf(stderr, " $BROMAGIC | libmagic mime magic database search path (%s)\n", bro_magic_path()); fprintf(stderr, " $BRO_PLUGINS | plugin search path (%s)\n", bro_plugin_path()); - fprintf(stderr, " $BRO_PREFIXES | prefix list (%s)\n", bro_prefixes()); fprintf(stderr, " $BRO_PREFIXES | prefix list (%s)\n", bro_prefixes().c_str()); fprintf(stderr, " $BRO_DNS_FAKE | disable DNS lookups (%s)\n", bro_dns_fake()); fprintf(stderr, " $BRO_SEED_FILE | file to load seeds from (not set)\n"); diff --git a/src/plugin/Macros.h b/src/plugin/Macros.h index 3248031e98..3657fb1e40 100644 --- a/src/plugin/Macros.h +++ b/src/plugin/Macros.h @@ -64,6 +64,14 @@ */ #define BRO_PLUGIN_VERSION(v) SetVersion(v) +/** + * Enables a hook for the plugin. Once enabled, Bro will call the + * corresponding virtual function. + * + * @param h The \a HookType to able. + */ +#define BRO_PLUGIN_ENABLE_HOOK(h) EnableHook(h) + /** * Adds script-level items defined in a \c *.bif file to what the plugin * provides. diff --git a/src/plugin/Manager.cc b/src/plugin/Manager.cc index ddd953ab22..be1d7827a5 100644 --- a/src/plugin/Manager.cc +++ b/src/plugin/Manager.cc @@ -21,11 +21,20 @@ string Manager::current_sopath; Manager::Manager() { init = false; + hooks = new hook_list*[NUM_HOOKS]; + + for ( int i = 0; i < NUM_HOOKS; i++ ) + hooks[i] = 0; } Manager::~Manager() { assert(! init); + + for ( int i = 0; i < NUM_HOOKS; i++ ) + delete hooks[i]; + + delete [] hooks; } void Manager::LoadPluginsFrom(const string& dir) @@ -204,15 +213,6 @@ bool Manager::RegisterPlugin(Plugin *plugin) return true; } -static bool interpreter_plugin_cmp(const InterpreterPlugin* a, const InterpreterPlugin* b) - { - if ( a->Priority() == b->Priority() ) - return a->Name() < b->Name(); - - // Reverse sort. - return a->Priority() > b->Priority(); - } - void Manager::InitPreScript() { assert(! init); @@ -221,26 +221,9 @@ void Manager::InitPreScript() { Plugin* plugin = *i; - if ( plugin->PluginType() == Plugin::INTERPRETER ) - interpreter_plugins.push_back(dynamic_cast(plugin)); - plugin->InitPreScript(); - - // Track the file extensions the plugin can handle. - std::stringstream ext(plugin->FileExtensions()); - - // Split at ":". - std::string e; - - while ( std::getline(ext, e, ':') ) - { - DBG_LOG(DBG_PLUGINS, "Plugin %s handles *.%s", plugin->Name(), e.c_str()); - extensions.insert(std::make_pair(e, plugin)); - } } - interpreter_plugins.sort(interpreter_plugin_cmp); - init = true; } @@ -273,28 +256,6 @@ void Manager::FinishPlugins() init = false; } -int Manager::TryLoadFile(const char* file) - { - assert(file); - const char* ext = strrchr(file, '.'); - - if ( ! ext ) - return -1; - - extension_map::iterator i = extensions.find(++ext); - if ( i == extensions.end() ) - return -1; - - Plugin* plugin = i->second; - - DBG_LOG(DBG_PLUGINS, "Loading %s with %s", file, plugin->Name()); - - if ( i->second->LoadFile(file) ) - return 1; - - return 0; - } - Manager::plugin_list Manager::Plugins() const { return *Manager::PluginsInternal(); @@ -310,66 +271,135 @@ Manager::plugin_list* Manager::PluginsInternal() return plugins; } -Val* Manager::CallFunction(const Func* func, val_list* args) const +static bool hook_cmp(std::pair a, std::pair b) { - Val* result = 0; + if ( a.first == b.first ) + return a.second->Name() < a.second->Name(); - for ( interpreter_plugin_list::const_iterator i = interpreter_plugins.begin(); - i != interpreter_plugins.end() && ! result; i++ ) + // Reverse sort. + return a.first > b.first; + } + +std::list > Manager::HooksEnabledForPlugin(const Plugin* plugin) const + { + std::list > enabled; + + for ( int i = 0; i < NUM_HOOKS; i++ ) { - result = (*i)->CallFunction(func, args); + hook_list* l = hooks[i]; - if ( result ) + if ( ! l ) + continue; + + for ( hook_list::iterator j = l->begin(); j != l->end(); j++ ) { - DBG_LOG(DBG_PLUGINS, "Plugin %s replaced call to %s", (*i)->Name(), func->Name()); - return result; + if ( (*j).second == plugin ) + enabled.push_back(std::make_pair((HookType)i, (*j).first)); } } + return enabled; + } + +void Manager::EnableHook(HookType hook, Plugin* plugin, int prio) + { + if ( ! hooks[hook] ) + hooks[hook] = new hook_list; + + hooks[hook]->push_back(std::make_pair(prio, plugin)); + hooks[hook]->sort(hook_cmp); + } + +void Manager::DisableHook(HookType hook, Plugin* plugin) + { + hook_list* l = hooks[hook]; + + if ( ! l ) + return; + + for ( hook_list::iterator i = l->begin(); i != l->end(); i++ ) + { + if ( (*i).second == plugin ) + { + l->erase(i); + break; + } + } + + if ( l->empty() ) + { + delete l; + hooks[hook] = 0; + } + } + +int Manager::HookLoadFile(const char* file) + { + hook_list* l = hooks[HOOK_LOAD_FILE]; + + for ( hook_list::iterator i = l->begin(); l && i != l->end(); i++ ) + { + Plugin* p = (*i).second; + + int rc = p->HookLoadFile(file); + + if ( rc >= 0 ) + return rc; + } + + return -1; + } + +Val* Manager::HookCallFunction(const Func* func, val_list* args) const + { + hook_list* l = hooks[HOOK_CALL_FUNCTION]; + + for ( hook_list::iterator i = l->begin(); l && i != l->end(); i++ ) + { + Plugin* p = (*i).second; + + Val* v = p->HookCallFunction(func, args); + + if ( v ) + return v; + } + return 0; } -bool Manager::QueueEvent(Event* event) const +bool Manager::HookQueueEvent(Event* event) const { - for ( interpreter_plugin_list::const_iterator i = interpreter_plugins.begin(); - i != interpreter_plugins.end(); i++ ) + hook_list* l = hooks[HOOK_QUEUE_EVENT]; + + for ( hook_list::iterator i = l->begin(); l && i != l->end(); i++ ) { - if ( (*i)->QueueEvent(event) ) - { - DBG_LOG(DBG_PLUGINS, "Plugin %s handled queueing of event %s", (*i)->Name(), event->Handler()->Name()); + Plugin* p = (*i).second; + + if ( p->HookQueueEvent(event) ) return true; - } } return false; } - -void Manager::UpdateNetworkTime(double network_time) const +void Manager::HookDrainEvents() const { - for ( interpreter_plugin_list::const_iterator i = interpreter_plugins.begin(); - i != interpreter_plugins.end(); i++ ) - (*i)->UpdateNetworkTime(network_time); - } + hook_list* l = hooks[HOOK_DRAIN_EVENTS]; -void Manager::DrainEvents() const - { - for ( interpreter_plugin_list::const_iterator i = interpreter_plugins.begin(); - i != interpreter_plugins.end(); i++ ) - (*i)->DrainEvents(); - } - -void Manager::DisableInterpreterPlugin(const InterpreterPlugin* plugin) - { - for ( interpreter_plugin_list::iterator i = interpreter_plugins.begin(); - i != interpreter_plugins.end(); i++ ) + for ( hook_list::iterator i = l->begin(); l && i != l->end(); i++ ) { - if ( *i == plugin ) - { - interpreter_plugins.erase(i); - return; - } + Plugin* p = (*i).second; + p->HookDrainEvents(); } } +void Manager::HookUpdateNetworkTime(double network_time) const + { + hook_list* l = hooks[HOOK_UPDATE_NETWORK_TIME]; + for ( hook_list::iterator i = l->begin(); l && i != l->end(); i++ ) + { + Plugin* p = (*i).second; + p->HookUpdateNetworkTime(network_time); + } + } diff --git a/src/plugin/Manager.h b/src/plugin/Manager.h index 85b60fb763..0c52679a64 100644 --- a/src/plugin/Manager.h +++ b/src/plugin/Manager.h @@ -12,6 +12,14 @@ namespace plugin { +// Macros that trigger a plugin hook. We put this into macros to short-cut +// the code for the most common case that no plugin defines the hook. +#define PLUGIN_HOOK_WITH_RESULT(hook, method_call, default_result) \ + (plugin_mgr->HavePluginForHook(plugin::hook) ? plugin_mgr->method_call : (default_result)) + +#define PLUGIN_HOOK_VOID(hook, method_call) \ + if ( plugin_mgr->HavePluginForHook(plugin::hook) ) plugin_mgr->method_call; + /** * A singleton object managing all plugins. */ @@ -19,7 +27,6 @@ class Manager { public: typedef std::list plugin_list; - typedef std::list interpreter_plugin_list; typedef Plugin::component_list component_list; /** @@ -30,7 +37,7 @@ public: /** * Destructor. */ - ~Manager(); + virtual ~Manager(); /** * Loads all plugins dynamically from a set of directories. Multiple @@ -71,22 +78,6 @@ public: */ void FinishPlugins(); - /** - * This tries to load the given file by searching for a plugin that - * support that extension. If a correspondign plugin is found, it's - * asked to loead the file. If that fails, the method reports an - * error message. - * - * This method must be called only between InitPreScript() and - * InitPostScript(). - * - * @return 1 if the file was sucessfully loaded by a plugin; 0 if a - * plugin was found that supports the file's extension, yet it - * encountered a problem loading the file; and -1 if we don't have a - * plugin that supports this extension. - */ - int TryLoadFile(const char* file); - /** * Returns a list of all available plugins. This includes all that * are compiled in statically, as well as those loaded dynamically so @@ -102,7 +93,65 @@ public: template std::list Components() const; /** - * Filters a function/event/hook call through all interpreter plugins. + * Returns true if there's at least one plugin interested in a given + * hook. + * + * @param The hook to check. + * + * @return True if there's a plugin for that hook. + */ + bool HavePluginForHook(HookType hook) const + { + // Inline to make avoid the function call. + return hooks[hook] != 0; + } + + /** + * Returns all the hooks, with their priorities, that are currently + * enabled for a given plugin. + * + * @param plugin The plugin to return the hooks for. + */ + std::list > HooksEnabledForPlugin(const Plugin* plugin) const; + + /** + * Enables a hook for a given plugin. + * + * hook: The hook to enable. + * + * plugin: The plugin defining the hook. + * + * prio: The priority to associate with the plugin for this hook. + */ + void EnableHook(HookType hook, Plugin* plugin, int prio); + + /** + * Disables a hook for a given plugin. + * + * hook: The hook to enable. + * + * plugin: The plugin that used to define the hook. + */ + void DisableHook(HookType hook, Plugin* plugin); + + // Hook entry functions. + + /** + * Hook that gives plugins a chance to take over loading an input + * input file. This method must be called between InitPreScript() and + * InitPostScript() for each input file Bro is about to load, either + * given on the command line or via @load script directives. The hook + * can take over the file, in which case Bro must not further process + * it otherwise. + * + * @return 1 if a plugin took over the file and loaded it + * successfully; 0 if a plugin took over the file but had trouble + * loading it; and -1 if no plugin was interested in the file at all. + */ + virtual int HookLoadFile(const char* file); + + /** + * Hook that filters calls to a script function/event/hook. * * @param func The function to be called. * @@ -113,39 +162,29 @@ public: * events, it may be any Val and must be ignored). If no plugin * handled the call, the method returns null. */ - Val* CallFunction(const Func* func, val_list* args) const; - - /** - * Filter the queuing of an event through all interpreter plugins. - * - * @param event The event to be queued; it may be modified. - * - * @return Returns true if a plugin handled the queuing; in that case the - * plugin will have taken ownership. - * - */ - bool QueueEvent(Event* event) const; + Val* HookCallFunction(const Func* func, val_list* args) const; /** - * Informs all interpreter plugins about an update in network time. + * Hook that filters the queuing of an event. * - * @param networkt_time The new network time. + * @param event The event to be queued; it may be modified. + * + * @return Returns true if a plugin handled the queuing; in that case + * the plugin will have taken ownership. */ - void UpdateNetworkTime(double network_time) const; + bool HookQueueEvent(Event* event) const; /** - * Informs all interpreter plugins that the event queue has been drained. + * Hook that informs plugins about an update in network time. + * + * @param network_time The new network time. */ - void DrainEvents() const; + void HookUpdateNetworkTime(double network_time) const; - /** - * Disables an interpreter plugin's hooking of the script interpreter. - * The remaining functionality of the Plugin base class remains - * available. - * - * @param plugin The plugin to disable. - */ - void DisableInterpreterPlugin(const InterpreterPlugin* plugin); + /** + * Hooks that informs plugins that the event queue is being drained. + */ + void HookDrainEvents() const; /** * Internal method that registers a freshly instantiated plugin with @@ -177,13 +216,17 @@ protected: int LoadPlugin(const std::string& dir); private: + // A hook list keeps pairs of plugin and priority interested in a + // given hook. + typedef std::list > hook_list; + static plugin_list* PluginsInternal(); bool init; - typedef std::map extension_map; - extension_map extensions; - interpreter_plugin_list interpreter_plugins; + // An array indexed by HookType. An entry is null if there's no hook + // of that type enabled. + hook_list** hooks; static string current_dir; static string current_sopath; diff --git a/src/plugin/Plugin.cc b/src/plugin/Plugin.cc index ff3cf8f414..20d436907b 100644 --- a/src/plugin/Plugin.cc +++ b/src/plugin/Plugin.cc @@ -10,6 +10,22 @@ using namespace plugin; +const char* hook_name(HookType h) + { + static const char* hook_names[int(NUM_HOOKS) + 1] = { + // Order must match that of HookType. + "LoadFile", + "CallFunction", + "QueueEvent", + "DrainEvents", + "UpdateNetworkTime", + // End marker. + "", + }; + + return hook_names[int(h)]; + } + BifItem::BifItem(const char* arg_id, Type arg_type) { id = copy_string(arg_id); @@ -49,7 +65,6 @@ Plugin::Plugin() dynamic = false; base_dir = 0; sopath = 0; - extensions = 0; Manager::RegisterPlugin(this); } @@ -62,12 +77,6 @@ Plugin::~Plugin() delete [] description; delete [] base_dir; delete [] sopath; - delete [] extensions; - } - -Plugin::Type Plugin::PluginType() const - { - return STANDARD; } const char* Plugin::Name() const @@ -170,22 +179,6 @@ Plugin::bif_item_list Plugin::CustomBifItems() const return bif_item_list(); } -const char* Plugin::FileExtensions() const - { - return extensions ? extensions : ""; - } - -void Plugin::SetFileExtensions(const char* ext) - { - extensions = copy_string(ext); - } - -bool Plugin::LoadFile(const char* file) - { - reporter->InternalError("Plugin::LoadFile not overriden for %s", file); - return false; - } - void Plugin::Done() { for ( component_list::const_iterator i = components.begin(); i != components.end(); i++ ) @@ -204,18 +197,9 @@ static bool component_cmp(const Component* a, const Component* b) return a->Name() < b->Name(); } -void Plugin::AddComponent(Component* c) - { - components.push_back(c); - - // Sort components by name to make sure we have a deterministic - // order. - components.sort(component_cmp); - } - bool Plugin::LoadBroFile(const char* file) { - add_input_file(file); + ::add_input_file(file); return true; } @@ -230,6 +214,53 @@ void Plugin::AddBifItem(const char* name, BifItem::Type type) bif_items.push_back(bi); } +void Plugin::AddComponent(Component* c) + { + components.push_back(c); + + // Sort components by name to make sure we have a deterministic + // order. + components.sort(component_cmp); + } + +Plugin::hook_list Plugin::EnabledHooks() const + { + return plugin_mgr->HooksEnabledForPlugin(this); + } + +void Plugin::EnableHook(HookType hook, int priority) + { + plugin_mgr->EnableHook(hook, this, priority); + } + +void Plugin::DisableHook(HookType hook) + { + plugin_mgr->DisableHook(hook, this); + } + +int Plugin::HookLoadFile(const char* file) + { + return -1; + } + +Val* Plugin::HookCallFunction(const Func* func, val_list* args) + { + return 0; + } + +bool Plugin::HookQueueEvent(Event* event) + { + return false; + } + +void Plugin::HookDrainEvents() + { + } + +void Plugin::HookUpdateNetworkTime(double network_time) + { + } + void Plugin::Describe(ODesc* d) const { d->Add("Plugin: "); @@ -256,18 +287,6 @@ void Plugin::Describe(ODesc* d) const else d->Add(" (built-in)"); - switch ( PluginType() ) { - case STANDARD: - break; - - case INTERPRETER: - d->Add( " (interpreter plugin)"); - break; - - default: - reporter->InternalError("unknown plugin type in Plugin::Describe"); - } - d->Add("\n"); if ( d->IsShort() ) @@ -317,48 +336,19 @@ void Plugin::Describe(ODesc* d) const d->Add((*i).GetID()); d->Add("\n"); } + + hook_list hooks = EnabledHooks(); + + for ( hook_list::iterator i = hooks.begin(); i != hooks.end(); i++ ) + { + HookType hook = (*i).first; + int prio = (*i).second; + + d->Add(" Implements "); + d->Add(hook_name(hook)); + d->Add(" (priority "); + d->Add(prio); + d->Add("]\n"); + } } -InterpreterPlugin::InterpreterPlugin(int arg_priority) - { - priority = arg_priority; - } - -InterpreterPlugin::~InterpreterPlugin() - { - } - -int InterpreterPlugin::Priority() const - { - return priority; - } - -Plugin::Type InterpreterPlugin::PluginType() const - { - return INTERPRETER; - } - -Val* InterpreterPlugin::CallFunction(const Func* func, val_list* args) - { - return 0; - } - -bool InterpreterPlugin::QueueEvent(Event* event) - { - return false; - } - -void InterpreterPlugin::UpdateNetworkTime(double network_time) - { - } - -void InterpreterPlugin::DrainEvents() - { - } - -void InterpreterPlugin::DisableInterpreterPlugin() const - { - plugin_mgr->DisableInterpreterPlugin(this); - } - - diff --git a/src/plugin/Plugin.h b/src/plugin/Plugin.h index 306c97e59e..915563e98c 100644 --- a/src/plugin/Plugin.h +++ b/src/plugin/Plugin.h @@ -17,6 +17,21 @@ namespace plugin { class Manager; class Component; +/** + * Hook types that a plugin may define. Each label maps to the corresponding + * virtual method in \a Plugin. + */ +enum HookType { + // Note: when changing this table, update hook_name() in Plugin.cc. + HOOK_LOAD_FILE, + HOOK_CALL_FUNCTION, + HOOK_QUEUE_EVENT, + HOOK_DRAIN_EVENTS, + HOOK_UPDATE_NETWORK_TIME, + // End marker. + NUM_HOOKS, +}; + /** * A class describing an item defined in \c *.bif file. */ @@ -70,45 +85,37 @@ private: /** * Base class for all plugins. * - * Plugins encapsulate functionality that extends one of Bro's major + * Plugins encapsulate functionality that extends one or more of Bro's major * subsystems, such as analysis of a specific protocol, or logging output in - * a particular format. A plugin is a logical container that can provide one - * or more \a components implementing functionality. For example, a RPC - * plugin could provide analyzer for set of related protocols (RPC, NFS, - * etc.), each of which would be a separate component. Likewise, a SQLite - * plugin could provide both a writer and reader component. In addition to - * components, a plugin can also provide of script-level elements defined in - * *.bif files. + * a particular format. A plugin acts a logical container that can provide a + * set of different functionality. Specifically, it may: + * + * - Provide one or more \a components implementing functionality. For + * example, a RPC plugin could provide analyzer for set of related + * protocols (RPC, NFS, etc.), each of which would be a separate component. + * Likewise, a SQLite plugin could provide both a writer and reader + * component. In addition to components, a plugin can also provide of + * script-level elements defined in *.bif files. + * + * - Provide BiF elements (functions, events, types, globals). + * + * - Provide hooks (aka callbacks) into Bro's core processing to inject + * and/or alter functionality. + * + * Note that a plugin needs to explicitly register all the functionality it + * provides. For components, it needs to call AddComponent(); for BiFs + * AddBifItem(); and for hooks EnableHook() and then also implemennt the + * corresponding virtual method). * - * Currently, all plugins are compiled statically into the final Bro binary. - * Later, we will extend the infrastructure to also support plugins loaded - * dynamically as shared libraries. */ class Plugin { public: typedef std::list component_list; typedef std::list bif_item_list; + typedef std::list > hook_list; typedef std::list > bif_init_func_result; typedef void (*bif_init_func)(Plugin *); - /** - * Type of a plugin. Plugin types are set implicitly by deriving from - * the corresponding base class. */ - enum Type { - /** - * A standard plugin. This is the type for all plugins - * derived directly from \a Plugin. */ - STANDARD, - - /** - * An interpreter plugin. These plugins get hooked into the - * script interpreter and can modify, or even replace, its - * execution of Bro script code. To create an interpreter - * plugin, derive from \aInterpreterPlugin. - */ - INTERPRETER - }; - /** * Constructor. */ @@ -119,11 +126,6 @@ public: */ virtual ~Plugin(); - /** - * Returns the type of the plugin. - */ - virtual Type PluginType() const; - /** * Returns the name of the plugin. */ @@ -146,11 +148,6 @@ public: */ bool DynamicPlugin() const; - /** - * Returns a colon-separated list of file extensions the plugin handles. - */ - const char* FileExtensions() const; - /** * For dynamic plugins, returns the base directory from which it was * loaded. For static plugins, returns null. @@ -201,8 +198,8 @@ public: /** * Finalizer method that derived classes can override for performing - * custom tasks at shutdown. Implementation must call the parent's - * version. + * custom tasks at shutdown. This can be overridden by derived + * classes; they must however call the parent's implementation. */ virtual void Done(); @@ -216,12 +213,14 @@ public: void Describe(ODesc* d) const; /** - * Registering an individual BiF that the plugin defines. The + * Registers an individual BiF that the plugin defines. The * information is for informational purpuses only and will show up in * the result of BifItems() as well as in the Describe() output. * Another way to add this information is via overriding * CustomBifItems(). * + * \todo Do we need both this an CustomBifItems()? + * * @param name The name of the BiF item. * * @param type The item's type. @@ -229,10 +228,10 @@ public: void AddBifItem(const char* name, BifItem::Type type); /** - * Adds a file to the list of files Bro loads at startup. This will - * normally be a Bro script, but it passes through the plugin system - * as well to load files with other extensions as supported by any of - * the current plugins. In other words, calling this method is + * Adds a file to the list of files that Bro loads at startup. This + * will normally be a Bro script, but it passes through the plugin + * system as well to load files with other extensions as supported by + * any of the current plugins. In other words, calling this method is * similar to given a file on the command line. Note that the file * may be only queued for now, and actually loaded later. * @@ -245,161 +244,99 @@ public: */ bool LoadBroFile(const char* file); -protected: - friend class Manager; - - /** - * Sets the plugins name. - * - * @param name The name. Makes a copy internally. - */ - void SetName(const char* name); - - /** - * Sets the plugin's textual description. - * - * @param name The description. Makes a copy internally. - */ - void SetDescription(const char* descr); - - /** - * Sets the plugin's version. - * - * @param version The version. - */ - void SetVersion(int version); - - /** - * Sets the API version the plugin requires. - * BRO_PLUGIN_VERSION_BUILTIN indicates that it's a plugin linked in - * statically. - */ - void SetAPIVersion(int version); - - /** - * Marks the plugin as statically or dynamically linked. - * - * @param dynamic True if this is a dynamically linked plugin. - */ - void SetDynamicPlugin(bool dynamic); - - /** - * Reports the extensions of input files the plugin handles. If Bro - * wants to load a file with one of these extensions it will pass - * them to LoadFile() and then then ignore otherwise. - * - * ext: A list of colon-separated file extensions the plugin handles. - */ - void SetFileExtensions(const char* ext); - - /** - * Sets the base directory and shared library path from which the - * plugin was loaded. This should be called only from the manager for - * dynamic plugins. - * - * @param dir The plugin directory. The functions makes an internal - * copy of string. - * - * @param sopath The full path the shared library loaded. The - * functions makes an internal copy of string. - */ - void SetPluginLocation(const char* dir, const char* sopath); - - /** - * Takes ownership. - */ - void AddComponent(Component* c); - - /** - * Virtual method that can be overriden by derived class to provide - * information about further script-level elements that the plugins - * provides on its own, i.e., outside of the standard mechanism - * processing *.bif files automatically. The returned information is - * for informational purpuses only and will show up in the result of - * BifItems() as well as in the Describe() output. - */ - virtual bif_item_list CustomBifItems() const; - - /** - * Virtual method that can be overriden by derived class to load - * files with extensions reported via SetFileExtension(). - * - * This method will be called between InitPreScript() and - * InitPostScript(), but with no further order or timing guaranteed. - * It will be called once for each file encountered with of the - * specificed extensions (i.e., duplicates are filtered out - * automatically). - * - * @return True if the file was loaded successfuly, false if not. Bro - * will abort in the latter case. - */ - virtual bool LoadFile(const char* file); - - /** - * Initializes the BiF items added with AddBifItem(). Internal method - * that will be called by the manager at the right time. - */ - void InitBifs(); - /** * Internal function adding an entry point for registering * auto-generated BiFs. */ void __AddBifInitFunction(bif_init_func c); -private: - typedef std::list bif_init_func_list; +protected: + friend class Manager; - const char* name; - const char* description; - const char* base_dir; - const char* sopath; - const char* extensions; - int version; - int api_version; - bool dynamic; - - component_list components; - bif_item_list bif_items; - bif_init_func_list bif_inits; -}; - -/** - * Class for hooking into script execution. An interpreter plugin can do - * everything a normal plugin can, yet will also be interfaced to the script - * interpreter for learning about, modidying, or potentially replacing - * standard functionality. - */ -class InterpreterPlugin : public Plugin { -public: /** - * Constructor. + * Registers and activates a component. * - * @param priority Imposes an order on InterpreterPlugins in which - * they'll be chained. The higher the prioritu, the earlier a plugin - * is called when the interpreter goes through the chain. */ - InterpreterPlugin(int priority); - - /** - * Destructor. + * @param c The component. The method takes ownership. */ - virtual ~InterpreterPlugin(); + void AddComponent(Component* c); /** - * Returns the plugins priority. + * Enables a hook. The corresponding virtual method will now be + * called as Bro's processing proceeds. Note that enabling hooks can + * have performance impaxct as many trigger frequently inside Bro's + * main processing path. + * + * Note that hooks may be enabled/disabled dynamically at any time, + * the output of Bro's \c -NN option will only reflect that state at + * startup time; hence usually one should call this for a plugin's + * hooks in either the plugin's ctor or in InitPreScript(). For + * consistency with other parts of the API, there's a macro + * PLUGIN_ENABLE_HOOK for use inside the ctor. + * + * @param hook The hook to enable. + * + * @param priority If multiple plugins enable the same hook, their + * priorities determine the order in which they'll be executed, from + * highest to lowest. If two plugins specify the same priority, order + * is undefined. */ - int Priority() const; + void EnableHook(HookType hook, int priority = 0); /** - * Callback for executing a function/event/hook. Whenever the script - * interpreter is about to execution a function, it first gives all - * InterpreterPlugins a chance to handle the call (in the order of their - * priorities). A plugin can either just inspect the call, or replace it - * (i.e., prevent the interpreter from executing it). In the latter case - * it must provide a matching return value. + * Disables a hook. Bro will no longer call the corresponding virtual + * method. + * + * @param hook The hook to disable. + */ + void DisableHook(HookType hook); + + /** + * Returns a list of hooks that are currently enabled for the plugin, + * along with their priorities. + */ + hook_list EnabledHooks() const; + + /** + * Virtual method that can be overriden by derived class to provide + * information about further script-level elements that the plugin + * provides on its own, i.e., outside of the standard mechanism + * processing *.bif files automatically. The returned information is + * for informational purposes only and will show up in the result of + * BifItems() as well as in the Describe() output. + * + * \todo Do we need both this an AddBifItem()? + */ + virtual bif_item_list CustomBifItems() const; + + // Hook functions. + + /** + * Hook into loading input files. This method will be called between + * InitPreScript() and InitPostScript(), but with no further order or + * timing guaranteed. It will be called once for each input file Bro + * is about to load, either given on the command line or via @load + * script directives. The hook can take over the file, in which case + * Bro not further process it otherwise. + * + * @return 1 if the plugin took over the file and loaded it + * successfully; 0 if the plugin took over the file but had trouble + * loading it (Bro will abort in this case, the plugin should have + * printed an error message); and -1 if the plugin wasn't interested + * in the file at all. + */ + virtual int HookLoadFile(const char* file); + + /** + * Hook into executing a script-level function/event/hook. Whenever + * the script interpreter is about to execution a function, it first + * gives all plugins with this hook enabled a chance to handle the + * call (in the order of their priorities). A plugin can either just + * inspect the call, or replace it (i.e., prevent the interpreter + * from executing it). In the latter case it must provide a matching + * return value. * * The default implementation does never handle the call in any way. - * + * * @param func The function being called. * * @param args The function arguments. The method can modify the list @@ -412,55 +349,133 @@ public: * TYPE_ANY). If the plugin did not handle the call, it must return * null. */ - virtual Val* CallFunction(const Func* func, val_list* args); - - /** - * Callback for raising an event. Whenever the script interpreter is - * about to queue an event for later execution, it first gives all - * InterpreterPlugins a chance to handle the queuing otherwise (in the - * order of their priorities). A plugin can either just inspect the - * event, or take it over (i.e., prevent the interpreter from queuing it - * it). - * - * The default implementation does never handle the queuing in any way. - * - * @param event The even to be queued. The method can modify it in in - * place long as it ensures matching types and correct reference - * counting. - * - * @return True if the plugin took charge of the event; in that case it - * must have assumed ownership of the event and the intpreter will not do - * anything further with it. False otherwise. - * - */ - virtual bool QueueEvent(Event* event); + virtual Val* HookCallFunction(const Func* func, val_list* args); /** - * Callback for updates in network time. This method will be called + * Hook into raising events. Whenever the script interpreter is about + * to queue an event for later execution, it first gives all plugins + * with this hook enabled a chance to handle the queuing otherwise + * (in the order of their priorities). A plugin can either just + * inspect the event, or take it over (i.e., prevent the interpreter + * from queuing it it). + * + * The default implementation does never handle the queuing in any + * way. + * + * @param event The even to be queued. The method can modify it in in + * place long as it ensures matching types and correct reference + * counting. + * + * @return True if the plugin took charge of the event; in that case + * it must have assumed ownership of the event and the intpreter will + * not do anything further with it. False otherwise. + */ + virtual bool HookQueueEvent(Event* event); + + /** + * Hook intp event queue draining. This method will be called + * whenever the event manager is draining its queue. + */ + virtual void HookDrainEvents(); + + /** + * Hook for updates to network time. This method will be called * whenever network time is advanced. * * @param networkt_time The new network time. */ - virtual void UpdateNetworkTime(double network_time); + virtual void HookUpdateNetworkTime(double network_time); + + // Methods that are used internally primarily. /** - * Callback for event queue draining. This method will be called - * whenever the event manager has drained it queue. + * Sets the plugins name. + * + * This is used primarily internally; plugin code should pass the + * name via the BRO_PLUGIN_BEGIN macro instead. + * + * @param name The name. Makes a copy internally. */ - virtual void DrainEvents(); + void SetName(const char* name); /** - * Disables interpreter hooking. The functionality of the Plugin base - * class remains available. + * Sets the plugin's textual description. + * + * This is used primarily internally; plugin code should pass the + * name via the BRO_PLUGIN_DESCRIPTION macro instead. + * + * @param name The description. Makes a copy internally. */ - void DisableInterpreterPlugin() const; + void SetDescription(const char* descr); - // Overridden from base class. - virtual Type PluginType() const; + /** + * Sets the plugin's version. + * + * This is used primarily internally; plugin code should pass the + * name via the BRO_PLUGIN_VERSION macro instead. + * + * @param version The version. + */ + void SetVersion(int version); + + /** + * Sets the API version the plugin requires. + * BRO_PLUGIN_VERSION_BUILTIN indicates that it's a plugin linked in + * statically. + * + * This is used primarily internally; plugins automatically set + * either API version of the Bro they are compiled dynamically for, + * or BRO_PLUGIN_VERSION_BUILTIN if they are linked in statically. + * + * @param version The version. + */ + void SetAPIVersion(int version); + + /** + * Marks the plugin as statically or dynamically linked. + * + * This is used primarily internally; plugins automatically set this + * based on which way they are compiled. + * + * @param dynamic True if this is a dynamically linked plugin. + */ + void SetDynamicPlugin(bool dynamic); + + /** + * Sets the base directory and shared library path from which the + * plugin was loaded. + * + * This is used primarily internally; plugins will have there + * location set automatically. + * + * @param dir The plugin directory. The functions makes an internal + * copy of string. + * + * @param sopath The full path the shared library loaded. The + * functions makes an internal copy of string. + */ + void SetPluginLocation(const char* dir, const char* sopath); private: - int priority; + /** + * Initializes the BiF items added with AddBifItem(). Internal method + * that will be called by the manager at the right time. + */ + void InitBifs(); + typedef std::list bif_init_func_list; + + const char* name; + const char* description; + const char* base_dir; + const char* sopath; + int version; + int api_version; + bool dynamic; + + component_list components; + bif_item_list bif_items; + bif_init_func_list bif_inits; }; } diff --git a/src/scan.l b/src/scan.l index 4c67a2f75e..599a660fee 100644 --- a/src/scan.l +++ b/src/scan.l @@ -581,7 +581,7 @@ YYLTYPE GetCurrentLocation() static int load_files(const char* orig_file) { - int rc = plugin_mgr->TryLoadFile(orig_file); + int rc = PLUGIN_HOOK_WITH_RESULT(HOOK_LOAD_FILE, HookLoadFile(orig_file), -1); if ( rc == 1 ) return 0; // A plugin took care of it, just skip. diff --git a/src/util.h b/src/util.h index a7d493485d..8526ded03f 100644 --- a/src/util.h +++ b/src/util.h @@ -213,7 +213,7 @@ extern void add_to_bro_path(const std::string& dir); extern const char* bro_magic_path(); extern const char* bro_plugin_path(); -extern const char* bro_prefixes(); +extern std::string bro_prefixes(); std::string dot_canon(std::string path, std::string file, std::string prefix = ""); const char* normalize_path(const char* path); void get_script_subpath(const std::string& full_filename, const char** subpath); From cbe48258f66cefaba1f9f9100b0a5cf6f5c49d1f Mon Sep 17 00:00:00 2001 From: Justin Azoff Date: Tue, 10 Dec 2013 11:27:19 -0500 Subject: [PATCH 12/56] fix the caching of recently validated certs The recently_validated_certs table was being checked for entries, but missing hashes were not being added to it after validation. --- scripts/policy/protocols/ssl/validate-certs.bro | 1 + 1 file changed, 1 insertion(+) diff --git a/scripts/policy/protocols/ssl/validate-certs.bro b/scripts/policy/protocols/ssl/validate-certs.bro index b34ec5a09a..886c28b6ac 100644 --- a/scripts/policy/protocols/ssl/validate-certs.bro +++ b/scripts/policy/protocols/ssl/validate-certs.bro @@ -40,6 +40,7 @@ event ssl_established(c: connection) &priority=3 { local result = x509_verify(c$ssl$cert, c$ssl$cert_chain, root_certs); c$ssl$validation_status = x509_err2str(result); + recently_validated_certs[c$ssl$cert_hash] = c$ssl$validation_status; } if ( c$ssl$validation_status != "ok" ) From 63c36d58f3c622238a84d0d32b854d33ed5ccc9b Mon Sep 17 00:00:00 2001 From: Jon Siwek Date: Wed, 11 Dec 2013 14:23:19 -0600 Subject: [PATCH 13/56] Another attempt to improve core.when-interpreter-exceptions unit test. lookup_hostname("localhost") occassionally timed out (after allowed 10 secs) when running test suite on some systems. Not sure why, but changed to use the Exec module for when block conditions instead as the scope of the test doesn't depend on a particular type of condition, it just needs something that will work reliably/quickly. --- .../bro.output | 20 +++---- .../core/when-interpreter-exceptions.bro | 54 +++++++++++-------- 2 files changed, 41 insertions(+), 33 deletions(-) diff --git a/testing/btest/Baseline/core.when-interpreter-exceptions/bro.output b/testing/btest/Baseline/core.when-interpreter-exceptions/bro.output index 154734f6d3..6d7ae52baf 100644 --- a/testing/btest/Baseline/core.when-interpreter-exceptions/bro.output +++ b/testing/btest/Baseline/core.when-interpreter-exceptions/bro.output @@ -1,13 +1,13 @@ -1386110869.157209 expression error in /Users/jon/Projects/bro/bro/testing/btest/.tmp/core.when-interpreter-exceptions/when-interpreter-exceptions.bro, line 96: field value missing [p$ip] -1386110869.157209 expression error in /Users/jon/Projects/bro/bro/testing/btest/.tmp/core.when-interpreter-exceptions/when-interpreter-exceptions.bro, line 63: field value missing [p$ip] -1386110869.157209 expression error in /Users/jon/Projects/bro/bro/testing/btest/.tmp/core.when-interpreter-exceptions/when-interpreter-exceptions.bro, line 79: field value missing [p$ip] -1386110869.157209 expression error in /Users/jon/Projects/bro/bro/testing/btest/.tmp/core.when-interpreter-exceptions/when-interpreter-exceptions.bro, line 36: field value missing [p$ip] -1386110869.157209 received termination signal +expression error in /Users/jon/Projects/bro/bro/testing/btest/.tmp/core.when-interpreter-exceptions/when-interpreter-exceptions.bro, line 48: field value missing [myrecord$notset] +expression error in /Users/jon/Projects/bro/bro/testing/btest/.tmp/core.when-interpreter-exceptions/when-interpreter-exceptions.bro, line 92: field value missing [myrecord$notset] +expression error in /Users/jon/Projects/bro/bro/testing/btest/.tmp/core.when-interpreter-exceptions/when-interpreter-exceptions.bro, line 73: field value missing [myrecord$notset] +expression error in /Users/jon/Projects/bro/bro/testing/btest/.tmp/core.when-interpreter-exceptions/when-interpreter-exceptions.bro, line 104: field value missing [myrecord$notset] +received termination signal +[f(F)] +f() done, no exception, T +[f(T)] +[bro_init()] +timeout g(), T timeout timeout g(), F -timeout g(), T g() done, no exception, T -localhost resolved -localhost resolved from f(), T -localhost resolved from f(), F -f() done, no exception, T diff --git a/testing/btest/core/when-interpreter-exceptions.bro b/testing/btest/core/when-interpreter-exceptions.bro index 5918f80ab5..151d8d2f57 100644 --- a/testing/btest/core/when-interpreter-exceptions.bro +++ b/testing/btest/core/when-interpreter-exceptions.bro @@ -1,10 +1,19 @@ -# @TEST-EXEC: btest-bg-run bro "bro -b --pseudo-realtime -r $TRACES/rotation.trace %INPUT >output 2>&1" +# @TEST-EXEC: btest-bg-run bro "bro -b %INPUT >output 2>&1" # @TEST-EXEC: btest-bg-wait 15 # @TEST-EXEC: TEST_DIFF_CANONIFIER="$SCRIPTS/diff-remove-abspath | $SCRIPTS/diff-remove-timestamps | $SCRIPTS/diff-sort" btest-diff bro/output # interpreter exceptions in "when" blocks shouldn't cause termination -global p: pkt_hdr; +@load base/utils/exec +@load base/frameworks/communication # let network-time run. otherwise there are no heartbeats... +redef exit_only_after_terminate = T; + +type MyRecord: record { + a: bool &default=T; + notset: bool &optional; +}; + +global myrecord: MyRecord; global c = 0; @@ -26,22 +35,21 @@ event termination_check() function f(do_exception: bool): bool { - return when ( local addrs = lookup_hostname("localhost") ) + local cmd = Exec::Command($cmd=fmt("echo 'f(%s)'", + do_exception)); + + return when ( local result = Exec::run(cmd) ) { - print "localhost resolved from f()", do_exception; + print result$stdout; if ( do_exception ) { event termination_check(); - print p$ip; + print myrecord$notset; } return T; } - timeout 10 sec - { - print "lookup_hostname in f() timed out unexpectedly"; - } check_term_condition(); return F; @@ -49,9 +57,11 @@ function f(do_exception: bool): bool function g(do_exception: bool): bool { - return when ( local addrs = lookup_hostname("localhost") ) + local stall = Exec::Command($cmd="sleep 30"); + + return when ( local result = Exec::run(stall) ) { - print "shouldn't get here, g()", do_exception; + print "shouldn't get here, g()", do_exception, result; } timeout 0 sec { @@ -60,7 +70,7 @@ function g(do_exception: bool): bool if ( do_exception ) { event termination_check(); - print p$ip; + print myrecord$notset; } return T; @@ -72,28 +82,26 @@ function g(do_exception: bool): bool event bro_init() { - when ( local addrs = lookup_hostname("localhost") ) + local cmd = Exec::Command($cmd="echo 'bro_init()'"); + local stall = Exec::Command($cmd="sleep 30"); + + when ( local result = Exec::run(cmd) ) { - print "localhost resolved"; + print result$stdout; event termination_check(); - print p$ip; - } - timeout 10 sec - { - print "lookup_hostname timed out unexpectedly"; - check_term_condition(); + print myrecord$notset; } - when ( local addrs2 = lookup_hostname("localhost") ) + when ( local result2 = Exec::run(stall) ) { - print "shouldn't get here"; + print "shouldn't get here", result2; check_term_condition(); } timeout 0 sec { print "timeout"; event termination_check(); - print p$ip; + print myrecord$notset; } when ( local b = f(T) ) From 8ea56ae567c3bd7c1d7f6349c0507ef6afef3d4d Mon Sep 17 00:00:00 2001 From: Jon Siwek Date: Thu, 12 Dec 2013 13:26:19 -0600 Subject: [PATCH 14/56] Improve warnings emitted from raw/execute input reader. Some return values of the setpgid() call in that parent proc are ok (or contradict reality). --- src/input/readers/Raw.cc | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/src/input/readers/Raw.cc b/src/input/readers/Raw.cc index d795430ba3..2bbdb4d0e0 100644 --- a/src/input/readers/Raw.cc +++ b/src/input/readers/Raw.cc @@ -204,9 +204,20 @@ bool Raw::Execute() // Parent also sets child process group immediately to avoid a race. if ( setpgid(childpid, childpid) == -1 ) { - char buf[256]; - strerror_r(errno, buf, sizeof(buf)); - Warning(Fmt("Could not set child process group: %s", buf)); + if ( errno == EACCES ) + // Child already did exec. That's fine since then it must have + // already done the setpgid() itself. + ; + else if ( errno == ESRCH && kill(childpid, 0) == 0 ) + // Sometimes (e.g. FreeBSD) this error is reported even though + // child exists, so do extra sanity check of whether it exists. + ; + else + { + char buf[256]; + strerror_r(errno, buf, sizeof(buf)); + Warning(Fmt("Could not set child process group: %s", buf)); + } } if ( ! UnlockForkMutex() ) From e9413c936134c90d15c27639e8da124bb7aae181 Mon Sep 17 00:00:00 2001 From: Robin Sommer Date: Thu, 12 Dec 2013 16:50:56 -0800 Subject: [PATCH 15/56] New default for plugin installation path. --- CMakeLists.txt | 2 +- aux/bro-aux | 2 +- cmake | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 0738eb387d..c006237850 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -22,7 +22,7 @@ set(BRO_SCRIPT_SOURCE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/scripts) get_filename_component(BRO_SCRIPT_INSTALL_PATH ${BRO_SCRIPT_INSTALL_PATH} ABSOLUTE) -set(BRO_PLUGIN_INSTALL_PATH ${BRO_ROOT_DIR}/share/bro/plugins) +set(BRO_PLUGIN_INSTALL_PATH ${BRO_ROOT_DIR}/lib/bro/plugins) set(BRO_MAGIC_INSTALL_PATH ${BRO_ROOT_DIR}/share/bro/magic) set(BRO_MAGIC_SOURCE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/magic/database) diff --git a/aux/bro-aux b/aux/bro-aux index 6c5db7af6e..715937ad72 160000 --- a/aux/bro-aux +++ b/aux/bro-aux @@ -1 +1 @@ -Subproject commit 6c5db7af6e93406be498b24f8a2d67ca281c1a03 +Subproject commit 715937ad72d9158d8c3142f81ee7005601d75db8 diff --git a/cmake b/cmake index fc53d57770..6eb4cbfbc7 160000 --- a/cmake +++ b/cmake @@ -1 +1 @@ -Subproject commit fc53d57770c86fbf9bd2d9694f06a1d539ebe35f +Subproject commit 6eb4cbfbc75f0bd13e247150b19932905a85e10c From 987452befff1e5bd99689eee2f97c18e43ef1cfd Mon Sep 17 00:00:00 2001 From: Robin Sommer Date: Thu, 12 Dec 2013 17:39:03 -0800 Subject: [PATCH 16/56] Cleanup of plugin component API. - Move more functionality into base class. - Remove cctors and assignment operators (weren't actually needed anymore) - Switch from const char* to std::string. --- src/BroDoc.cc | 4 +-- src/RuleAction.cc | 6 ++--- src/analyzer/Analyzer.cc | 4 +-- src/analyzer/Component.cc | 44 +++----------------------------- src/analyzer/Component.h | 34 +++++++----------------- src/analyzer/Manager.cc | 24 +++++++++-------- src/file_analysis/Analyzer.cc | 2 +- src/file_analysis/AnalyzerSet.cc | 12 ++++----- src/file_analysis/Component.cc | 41 +++-------------------------- src/file_analysis/Component.h | 34 +++++++----------------- src/file_analysis/File.cc | 4 +-- src/file_analysis/Manager.cc | 10 ++++---- src/plugin/Component.cc | 12 ++++++++- src/plugin/Component.h | 34 +++++++++++++++++++----- src/plugin/ComponentManager.h | 22 ++++++++-------- src/plugin/TaggedComponent.h | 32 ----------------------- src/util.cc | 9 +++---- src/util.h | 2 +- 18 files changed, 114 insertions(+), 216 deletions(-) diff --git a/src/BroDoc.cc b/src/BroDoc.cc index 93d8a34848..0a1006bd07 100644 --- a/src/BroDoc.cc +++ b/src/BroDoc.cc @@ -473,7 +473,7 @@ static void WritePluginSectionHeading(FILE* f, const plugin::Plugin* p) static void WriteAnalyzerComponent(FILE* f, const analyzer::Component* c) { EnumType* atag = analyzer_mgr->GetTagEnumType(); - string tag = fmt("ANALYZER_%s", c->CanonicalName()); + string tag = fmt("ANALYZER_%s", c->CanonicalName().c_str()); if ( atag->Lookup("Analyzer", tag.c_str()) < 0 ) reporter->InternalError("missing analyzer tag for %s", tag.c_str()); @@ -484,7 +484,7 @@ static void WriteAnalyzerComponent(FILE* f, const analyzer::Component* c) static void WriteAnalyzerComponent(FILE* f, const file_analysis::Component* c) { EnumType* atag = file_mgr->GetTagEnumType(); - string tag = fmt("ANALYZER_%s", c->CanonicalName()); + string tag = fmt("ANALYZER_%s", c->CanonicalName().c_str()); if ( atag->Lookup("Files", tag.c_str()) < 0 ) reporter->InternalError("missing analyzer tag for %s", tag.c_str()); diff --git a/src/RuleAction.cc b/src/RuleAction.cc index ec57c96bd2..8c6948ad10 100644 --- a/src/RuleAction.cc +++ b/src/RuleAction.cc @@ -60,11 +60,11 @@ RuleActionAnalyzer::RuleActionAnalyzer(const char* arg_analyzer) void RuleActionAnalyzer::PrintDebug() { if ( ! child_analyzer ) - fprintf(stderr, "|%s|\n", analyzer_mgr->GetComponentName(analyzer)); + fprintf(stderr, "|%s|\n", analyzer_mgr->GetComponentName(analyzer).c_str()); else fprintf(stderr, "|%s:%s|\n", - analyzer_mgr->GetComponentName(analyzer), - analyzer_mgr->GetComponentName(child_analyzer)); + analyzer_mgr->GetComponentName(analyzer).c_str(), + analyzer_mgr->GetComponentName(child_analyzer).c_str()); } diff --git a/src/analyzer/Analyzer.cc b/src/analyzer/Analyzer.cc index 2927e3ad97..1e4f0e20f4 100644 --- a/src/analyzer/Analyzer.cc +++ b/src/analyzer/Analyzer.cc @@ -75,7 +75,7 @@ analyzer::ID Analyzer::id_counter = 0; const char* Analyzer::GetAnalyzerName() const { assert(tag); - return analyzer_mgr->GetComponentName(tag); + return analyzer_mgr->GetComponentName(tag).c_str(); } void Analyzer::SetAnalyzerTag(const Tag& arg_tag) @@ -87,7 +87,7 @@ void Analyzer::SetAnalyzerTag(const Tag& arg_tag) bool Analyzer::IsAnalyzer(const char* name) { assert(tag); - return strcmp(analyzer_mgr->GetComponentName(tag), name) == 0; + return strcmp(analyzer_mgr->GetComponentName(tag).c_str(), name) == 0; } // Used in debugging output. diff --git a/src/analyzer/Component.cc b/src/analyzer/Component.cc index 255e03435c..91528b29d8 100644 --- a/src/analyzer/Component.cc +++ b/src/analyzer/Component.cc @@ -8,12 +8,11 @@ using namespace analyzer; -Component::Component(const char* arg_name, factory_callback arg_factory, Tag::subtype_t arg_subtype, bool arg_enabled, bool arg_partial) - : plugin::Component(plugin::component::ANALYZER), +Component::Component(const std::string& name, factory_callback arg_factory, Tag::subtype_t arg_subtype, bool arg_enabled, bool arg_partial) + : plugin::Component(plugin::component::ANALYZER, name), plugin::TaggedComponent(arg_subtype) { - name = copy_string(arg_name); - canon_name = canonify_name(arg_name); + canon_name = canonify_name(name); factory = arg_factory; enabled = arg_enabled; partial = arg_partial; @@ -21,31 +20,12 @@ Component::Component(const char* arg_name, factory_callback arg_factory, Tag::su analyzer_mgr->RegisterComponent(this, "ANALYZER_"); } -Component::Component(const Component& other) - : plugin::Component(Type()), - plugin::TaggedComponent(other) - { - name = copy_string(other.name); - canon_name = copy_string(other.canon_name); - factory = other.factory; - enabled = other.enabled; - partial = other.partial; - - // TODO: Do we need the RegisterComponent() call here? - } - Component::~Component() { - delete [] name; - delete [] canon_name; } -void Component::Describe(ODesc* d) const +void Component::DoDescribe(ODesc* d) const { - plugin::Component::Describe(d); - d->Add(name); - d->Add(" ("); - if ( factory ) { d->Add("ANALYZER_"); @@ -54,20 +34,4 @@ void Component::Describe(ODesc* d) const } d->Add(enabled ? "enabled" : "disabled"); - d->Add(")"); - } - -Component& Component::operator=(const Component& other) - { - plugin::TaggedComponent::operator=(other); - - if ( &other != this ) - { - name = copy_string(other.name); - factory = other.factory; - enabled = other.enabled; - partial = other.partial; - } - - return *this; } diff --git a/src/analyzer/Component.h b/src/analyzer/Component.h index 9bc8b357d7..d8a62bb38c 100644 --- a/src/analyzer/Component.h +++ b/src/analyzer/Component.h @@ -1,7 +1,7 @@ // See the file "COPYING" in the main distribution directory for copyright. -#ifndef ANALYZER_PLUGIN_COMPONENT_H -#define ANALYZER_PLUGIN_COMPONENT_H +#ifndef ANALYZER_COMPONENT_H +#define ANALYZER_COMPONENT_H #include "Tag.h" #include "plugin/Component.h" @@ -56,33 +56,20 @@ public: * connections has generally not seen much testing yet as virtually * no existing analyzer supports it. */ - Component(const char* name, factory_callback factory, Tag::subtype_t subtype = 0, bool enabled = true, bool partial = false); - - /** - * Copy constructor. - */ - Component(const Component& other); + Component(const std::string& name, factory_callback factory, Tag::subtype_t subtype = 0, bool enabled = true, bool partial = false); /** * Destructor. */ ~Component(); - /** - * Returns the name of the analyzer. This name is unique across all - * analyzers and used to identify it. The returned name is derived - * from what's passed to the constructor but upper-cased and - * canonified to allow being part of a script-level ID. - */ - virtual const char* Name() const { return name; } - /** * Returns a canonocalized version of the analyzer's name. The * returned name is derived from what's passed to the constructor but * upper-cased and transformed to allow being part of a script-level * ID. */ - const char* CanonicalName() const { return canon_name; } + const std::string& CanonicalName() const { return canon_name; } /** * Returns the analyzer's factory function. @@ -110,17 +97,14 @@ public: */ void SetEnabled(bool arg_enabled) { enabled = arg_enabled; } +protected: /** - * Generates a human-readable description of the component's main - * parameters. This goes into the output of \c "bro -NN". - */ - virtual void Describe(ODesc* d) const; - - Component& operator=(const Component& other); + * Overriden from plugin::Component. + */ + virtual void DoDescribe(ODesc* d) const; private: - const char* name; // The analyzer's name. - const char* canon_name; // The analyzer's canonical name. + std::string canon_name; // The analyzer's canonical name. factory_callback factory; // The analyzer's factory callback. bool partial; // True if the analyzer supports partial connections. bool enabled; // True if the analyzer is enabled. diff --git a/src/analyzer/Manager.cc b/src/analyzer/Manager.cc index 748af34dfb..e2cd2a7e05 100644 --- a/src/analyzer/Manager.cc +++ b/src/analyzer/Manager.cc @@ -104,7 +104,8 @@ void Manager::DumpDebug() DBG_LOG(DBG_ANALYZER, "Available analyzers after bro_init():"); list all_analyzers = GetComponents(); for ( list::const_iterator i = all_analyzers.begin(); i != all_analyzers.end(); ++i ) - DBG_LOG(DBG_ANALYZER, " %s (%s)", (*i)->Name(), IsEnabled((*i)->Tag()) ? "enabled" : "disabled"); + DBG_LOG(DBG_ANALYZER, " %s (%s)", (*i)->Name().c_str(), + IsEnabled((*i)->Tag()) ? "enabled" : "disabled"); DBG_LOG(DBG_ANALYZER, ""); DBG_LOG(DBG_ANALYZER, "Analyzers by port:"); @@ -143,7 +144,7 @@ bool Manager::EnableAnalyzer(Tag tag) if ( ! p ) return false; - DBG_LOG(DBG_ANALYZER, "Enabling analyzer %s", p->Name()); + DBG_LOG(DBG_ANALYZER, "Enabling analyzer %s", p->Name().c_str()); p->SetEnabled(true); return true; @@ -156,7 +157,7 @@ bool Manager::EnableAnalyzer(EnumVal* val) if ( ! p ) return false; - DBG_LOG(DBG_ANALYZER, "Enabling analyzer %s", p->Name()); + DBG_LOG(DBG_ANALYZER, "Enabling analyzer %s", p->Name().c_str()); p->SetEnabled(true); return true; @@ -169,7 +170,7 @@ bool Manager::DisableAnalyzer(Tag tag) if ( ! p ) return false; - DBG_LOG(DBG_ANALYZER, "Disabling analyzer %s", p->Name()); + DBG_LOG(DBG_ANALYZER, "Disabling analyzer %s", p->Name().c_str()); p->SetEnabled(false); return true; @@ -182,7 +183,7 @@ bool Manager::DisableAnalyzer(EnumVal* val) if ( ! p ) return false; - DBG_LOG(DBG_ANALYZER, "Disabling analyzer %s", p->Name()); + DBG_LOG(DBG_ANALYZER, "Disabling analyzer %s", p->Name().c_str()); p->SetEnabled(false); return true; @@ -254,7 +255,7 @@ bool Manager::RegisterAnalyzerForPort(Tag tag, TransportProto proto, uint32 port return false; #ifdef DEBUG - const char* name = GetComponentName(tag); + const char* name = GetComponentName(tag).c_str(); DBG_LOG(DBG_ANALYZER, "Registering analyzer %s for port %" PRIu32 "/%d", name, port, proto); #endif @@ -270,7 +271,7 @@ bool Manager::UnregisterAnalyzerForPort(Tag tag, TransportProto proto, uint32 po return true; // still a "successful" unregistration #ifdef DEBUG - const char* name = GetComponentName(tag); + const char* name = GetComponentName(tag).c_str(); DBG_LOG(DBG_ANALYZER, "Unregistering analyzer %s for port %" PRIu32 "/%d", name, port, proto); #endif @@ -293,7 +294,8 @@ Analyzer* Manager::InstantiateAnalyzer(Tag tag, Connection* conn) if ( ! c->Factory() ) { - reporter->InternalWarning("analyzer %s cannot be instantiated dynamically", GetComponentName(tag)); + reporter->InternalWarning("analyzer %s cannot be instantiated dynamically", + GetComponentName(tag).c_str()); return 0; } @@ -403,7 +405,7 @@ bool Manager::BuildInitialAnalyzerTree(Connection* conn) root->AddChildAnalyzer(analyzer, false); DBG_ANALYZER_ARGS(conn, "activated %s analyzer as scheduled", - analyzer_mgr->GetComponentName(*i)); + analyzer_mgr->GetComponentName(*i).c_str()); } } @@ -429,7 +431,7 @@ bool Manager::BuildInitialAnalyzerTree(Connection* conn) root->AddChildAnalyzer(analyzer, false); DBG_ANALYZER_ARGS(conn, "activated %s analyzer due to port %d", - analyzer_mgr->GetComponentName(*j), resp_port); + analyzer_mgr->GetComponentName(*j).c_str(), resp_port); } } } @@ -555,7 +557,7 @@ void Manager::ExpireScheduledAnalyzers() conns.erase(i); DBG_LOG(DBG_ANALYZER, "Expiring expected analyzer %s for connection %s", - analyzer_mgr->GetComponentName(a->analyzer), + analyzer_mgr->GetComponentName(a->analyzer).c_str(), fmt_conn_id(a->conn.orig, 0, a->conn.resp, a->conn.resp_p)); delete a; diff --git a/src/file_analysis/Analyzer.cc b/src/file_analysis/Analyzer.cc index 2d45c6579a..c04c02f818 100644 --- a/src/file_analysis/Analyzer.cc +++ b/src/file_analysis/Analyzer.cc @@ -8,7 +8,7 @@ file_analysis::ID file_analysis::Analyzer::id_counter = 0; file_analysis::Analyzer::~Analyzer() { DBG_LOG(DBG_FILE_ANALYSIS, "Destroy file analyzer %s", - file_mgr->GetComponentName(tag)); + file_mgr->GetComponentName(tag).c_str()); Unref(args); } diff --git a/src/file_analysis/AnalyzerSet.cc b/src/file_analysis/AnalyzerSet.cc index aee5e0d62e..b3f11b6816 100644 --- a/src/file_analysis/AnalyzerSet.cc +++ b/src/file_analysis/AnalyzerSet.cc @@ -53,7 +53,7 @@ bool AnalyzerSet::Add(file_analysis::Tag tag, RecordVal* args) if ( analyzer_map.Lookup(key) ) { DBG_LOG(DBG_FILE_ANALYSIS, "Instantiate analyzer %s skipped for file id" - " %s: already exists", file_mgr->GetComponentName(tag), + " %s: already exists", file_mgr->GetComponentName(tag).c_str(), file->GetID().c_str()); delete key; return true; @@ -93,7 +93,7 @@ bool AnalyzerSet::AddMod::Perform(AnalyzerSet* set) if ( set->analyzer_map.Lookup(key) ) { DBG_LOG(DBG_FILE_ANALYSIS, "Add analyzer %s skipped for file id" - " %s: already exists", file_mgr->GetComponentName(a->Tag()), + " %s: already exists", file_mgr->GetComponentName(a->Tag()).c_str(), a->GetFile()->GetID().c_str()); Abort(); @@ -120,12 +120,12 @@ bool AnalyzerSet::Remove(file_analysis::Tag tag, HashKey* key) if ( ! a ) { DBG_LOG(DBG_FILE_ANALYSIS, "Skip remove analyzer %s for file id %s", - file_mgr->GetComponentName(tag), file->GetID().c_str()); + file_mgr->GetComponentName(tag).c_str(), file->GetID().c_str()); return false; } DBG_LOG(DBG_FILE_ANALYSIS, "Remove analyzer %s for file id %s", - file_mgr->GetComponentName(tag), + file_mgr->GetComponentName(tag).c_str(), file->GetID().c_str()); a->Done(); @@ -169,7 +169,7 @@ file_analysis::Analyzer* AnalyzerSet::InstantiateAnalyzer(Tag tag, if ( ! a ) { reporter->Error("Failed file analyzer %s instantiation for file id %s", - file_mgr->GetComponentName(tag), file->GetID().c_str()); + file_mgr->GetComponentName(tag).c_str(), file->GetID().c_str()); return 0; } @@ -179,7 +179,7 @@ file_analysis::Analyzer* AnalyzerSet::InstantiateAnalyzer(Tag tag, void AnalyzerSet::Insert(file_analysis::Analyzer* a, HashKey* key) { DBG_LOG(DBG_FILE_ANALYSIS, "Add analyzer %s for file id %s", - file_mgr->GetComponentName(a->Tag()), file->GetID().c_str()); + file_mgr->GetComponentName(a->Tag()).c_str(), file->GetID().c_str()); analyzer_map.Insert(key, a); delete key; diff --git a/src/file_analysis/Component.cc b/src/file_analysis/Component.cc index 87789eb113..dc30b95986 100644 --- a/src/file_analysis/Component.cc +++ b/src/file_analysis/Component.cc @@ -8,58 +8,25 @@ using namespace file_analysis; -Component::Component(const char* arg_name, factory_callback arg_factory, Tag::subtype_t subtype) - : plugin::Component(plugin::component::FILE_ANALYZER), +Component::Component(const std::string& name, factory_callback arg_factory, Tag::subtype_t subtype) + : plugin::Component(plugin::component::FILE_ANALYZER, name), plugin::TaggedComponent(subtype) { - name = copy_string(arg_name); - canon_name = canonify_name(arg_name); + canon_name = canonify_name(name); factory = arg_factory; file_mgr->RegisterComponent(this, "ANALYZER_"); } -Component::Component(const Component& other) - : plugin::Component(Type()), - plugin::TaggedComponent(other) - { - name = copy_string(other.name); - canon_name = copy_string(other.canon_name); - factory = other.factory; - - // TODO: Do we need the RegisterComponent() call here? - } - Component::~Component() { - delete [] name; - delete [] canon_name; } -void Component::Describe(ODesc* d) const +void Component::DoDescribe(ODesc* d) const { - plugin::Component::Describe(d); - d->Add(name); - d->Add(" ("); - if ( factory ) { d->Add("ANALYZER_"); d->Add(canon_name); } - - d->Add(")"); - } - -Component& Component::operator=(const Component& other) - { - plugin::TaggedComponent::operator=(other); - - if ( &other != this ) - { - name = copy_string(other.name); - factory = other.factory; - } - - return *this; } diff --git a/src/file_analysis/Component.h b/src/file_analysis/Component.h index 6e2809349d..8f3b628a32 100644 --- a/src/file_analysis/Component.h +++ b/src/file_analysis/Component.h @@ -1,7 +1,7 @@ // See the file "COPYING" in the main distribution directory for copyright. -#ifndef FILE_ANALYZER_PLUGIN_COMPONENT_H -#define FILE_ANALYZER_PLUGIN_COMPONENT_H +#ifndef FILE_ANALYZER_COMPONENT_H +#define FILE_ANALYZER_COMPONENT_H #include "Tag.h" #include "plugin/Component.h" @@ -47,50 +47,34 @@ public: * analyzer instances can accordingly access it via analyzer::Tag(). * If not used, leave at zero. */ - Component(const char* name, factory_callback factory, Tag::subtype_t subtype = 0); - - /** - * Copy constructor. - */ - Component(const Component& other); + Component(const std::string& name, factory_callback factory, Tag::subtype_t subtype = 0); /** * Destructor. */ ~Component(); - /** - * Returns the name of the analyzer. This name is unique across all - * analyzers and used to identify it. The returned name is derived - * from what's passed to the constructor but upper-cased and - * canonified to allow being part of a script-level ID. - */ - virtual const char* Name() const { return name; } - /** * Returns a canonocalized version of the analyzer's name. The * returned name is derived from what's passed to the constructor but * upper-cased and transformed to allow being part of a script-level * ID. */ - const char* CanonicalName() const { return canon_name; } + const std::string& CanonicalName() const { return canon_name; } /** * Returns the analyzer's factory function. */ factory_callback Factory() const { return factory; } +protected: /** - * Generates a human-readable description of the component's main - * parameters. This goes into the output of \c "bro -NN". - */ - virtual void Describe(ODesc* d) const; - - Component& operator=(const Component& other); + * Overriden from plugin::Component. + */ + virtual void DoDescribe(ODesc* d) const; private: - const char* name; // The analyzer's name. - const char* canon_name; // The analyzer's canonical name. + std::string canon_name; // The analyzer's canonical name. factory_callback factory; // The analyzer's factory callback. }; diff --git a/src/file_analysis/File.cc b/src/file_analysis/File.cc index 980a8bfbce..6c15f47094 100644 --- a/src/file_analysis/File.cc +++ b/src/file_analysis/File.cc @@ -254,7 +254,7 @@ void File::ScheduleInactivityTimer() const bool File::AddAnalyzer(file_analysis::Tag tag, RecordVal* args) { DBG_LOG(DBG_FILE_ANALYSIS, "[%s] Queuing addition of %s analyzer", - id.c_str(), file_mgr->GetComponentName(tag)); + id.c_str(), file_mgr->GetComponentName(tag).c_str()); return done ? false : analyzers.QueueAdd(tag, args); } @@ -262,7 +262,7 @@ bool File::AddAnalyzer(file_analysis::Tag tag, RecordVal* args) bool File::RemoveAnalyzer(file_analysis::Tag tag, RecordVal* args) { DBG_LOG(DBG_FILE_ANALYSIS, "[%s] Queuing remove of %s analyzer", - id.c_str(), file_mgr->GetComponentName(tag)); + id.c_str(), file_mgr->GetComponentName(tag).c_str()); return done ? false : analyzers.QueueRemove(tag, args); } diff --git a/src/file_analysis/Manager.cc b/src/file_analysis/Manager.cc index a635a5dc14..de930e7589 100644 --- a/src/file_analysis/Manager.cc +++ b/src/file_analysis/Manager.cc @@ -351,7 +351,7 @@ void Manager::GetFileHandle(analyzer::Tag tag, Connection* c, bool is_orig) return; DBG_LOG(DBG_FILE_ANALYSIS, "Raise get_file_handle() for protocol analyzer %s", - analyzer_mgr->GetComponentName(tag)); + analyzer_mgr->GetComponentName(tag).c_str()); EnumVal* tagval = tag.AsEnumVal(); Ref(tagval); @@ -398,12 +398,12 @@ Analyzer* Manager::InstantiateAnalyzer(Tag tag, RecordVal* args, File* f) const if ( ! c->Factory() ) { reporter->InternalWarning("file analyzer %s cannot be instantiated " - "dynamically", c->CanonicalName()); + "dynamically", c->CanonicalName().c_str()); return 0; } DBG_LOG(DBG_FILE_ANALYSIS, "Instantiate analyzer %s for file %s", - GetComponentName(tag), f->id.c_str()); + GetComponentName(tag).c_str(), f->id.c_str()); Analyzer* a = c->Factory()(args, f); @@ -445,7 +445,7 @@ bool Manager::RegisterAnalyzerForMIMEType(Tag tag, const string& mtype) TagSet* l = LookupMIMEType(mtype, true); DBG_LOG(DBG_FILE_ANALYSIS, "Register analyzer %s for MIME type %s", - GetComponentName(tag), mtype.c_str()); + GetComponentName(tag).c_str(), mtype.c_str()); l->insert(tag); return true; @@ -466,7 +466,7 @@ bool Manager::UnregisterAnalyzerForMIMEType(Tag tag, const string& mtype) TagSet* l = LookupMIMEType(mtype, true); DBG_LOG(DBG_FILE_ANALYSIS, "Unregister analyzer %s for MIME type %s", - GetComponentName(tag), mtype.c_str()); + GetComponentName(tag).c_str(), mtype.c_str()); l->erase(tag); return true; diff --git a/src/plugin/Component.cc b/src/plugin/Component.cc index ee18d55cdf..a3d2eef194 100644 --- a/src/plugin/Component.cc +++ b/src/plugin/Component.cc @@ -7,15 +7,21 @@ using namespace plugin; -Component::Component(component::Type arg_type) +Component::Component(component::Type arg_type, const std::string& arg_name) { type = arg_type; + name = arg_name; } Component::~Component() { } +const std::string& Component::Name() const + { + return name; + } + component::Type Component::Type() const { return type; @@ -51,4 +57,8 @@ void Component::Describe(ODesc* d) const d->Add("]"); d->Add(" "); + d->Add(name); + d->Add(" ("); + DoDescribe(d); + d->Add(")"); } diff --git a/src/plugin/Component.h b/src/plugin/Component.h index ad02dc7e4b..0b6b5e9c49 100644 --- a/src/plugin/Component.h +++ b/src/plugin/Component.h @@ -3,6 +3,8 @@ #ifndef PLUGIN_COMPONENT_H #define PLUGIN_COMPONENT_H +#include + class ODesc; namespace plugin { @@ -32,8 +34,11 @@ public: * Constructor. * * @param type The type of the compoment. + * + * @param name A descriptive name for the component. This name must + * be unique across all components of the same type. */ - Component(component::Type type); + Component(component::Type type, const std::string& name); /** * Destructor. @@ -46,22 +51,37 @@ public: component::Type Type() const; /** - * Returns a descriptive name for the analyzer. This name must be - * unique across all components of the same type. + * Returns the compoment's name. */ - virtual const char* Name() const = 0; + const std::string& Name() const; /** - * Returns a textual representation of the component. The default - * version just output the type. Derived version should call the - * parent's implementation and that add further information. + * Returns a textual representation of the component. This goes into + * the output of "bro -NN". + * + * By default version, this just outputs the type and the name. + * Derived versions should override DoDescribe() to add type specific + * details. * * @param d The description object to use. */ virtual void Describe(ODesc* d) const; +protected: + /** + * Adds type specific information to the outout of Describe(). + * + * @param d The description object to use. + */ + virtual void DoDescribe(ODesc* d) const { } + private: + // Disable. + Component(const Component& other); + Component operator=(const Component& other); + component::Type type; + std::string name; }; } diff --git a/src/plugin/ComponentManager.h b/src/plugin/ComponentManager.h index 71994d82d1..b3ad944bec 100644 --- a/src/plugin/ComponentManager.h +++ b/src/plugin/ComponentManager.h @@ -37,7 +37,7 @@ public: /** * @return The script-layer module in which the component's "Tag" ID lives. */ - const char* GetModule() const; + const std::string& GetModule() const; /** * @return A list of all registered components. @@ -55,7 +55,7 @@ public: * @param tag A component's tag. * @return The canonical component name. */ - const char* GetComponentName(T tag) const; + const std::string& GetComponentName(T tag) const; /** * Get a component name from it's enum value. @@ -63,7 +63,7 @@ public: * @param val A component's enum value. * @return The canonical component name. */ - const char* GetComponentName(Val* val) const; + const std::string& GetComponentName(Val* val) const; /** * Get a component tag from its name. @@ -134,7 +134,7 @@ ComponentManager::ComponentManager(const string& arg_module) } template -const char* ComponentManager::GetModule() const +const std::string& ComponentManager::GetModule() const { return module.c_str(); } @@ -158,9 +158,9 @@ EnumType* ComponentManager::GetTagEnumType() const } template -const char* ComponentManager::GetComponentName(T tag) const +const std::string& ComponentManager::GetComponentName(T tag) const { - static const char* error = ""; + static const std::string& error = ""; if ( ! tag ) return error; @@ -176,7 +176,7 @@ const char* ComponentManager::GetComponentName(T tag) const } template -const char* ComponentManager::GetComponentName(Val* val) const +const std::string& ComponentManager::GetComponentName(Val* val) const { return GetComponentName(T(val->AsEnumVal())); } @@ -222,14 +222,14 @@ template void ComponentManager::RegisterComponent(C* component, const string& prefix) { - const char* cname = component->CanonicalName(); + std::string cname = component->CanonicalName(); if ( Lookup(cname) ) reporter->FatalError("Component '%s::%s' defined more than once", - module.c_str(), cname); + module.c_str(), cname.c_str()); DBG_LOG(DBG_PLUGINS, "Registering component %s (tag %s)", - component->Name(), component->Tag().AsString().c_str()); + component->Name().c_str(), component->Tag().AsString().c_str()); components_by_name.insert(std::make_pair(cname, component)); components_by_tag.insert(std::make_pair(component->Tag(), component)); @@ -237,7 +237,7 @@ void ComponentManager::RegisterComponent(C* component, component->Tag().AsEnumVal()->InternalInt(), component)); // Install an identfier for enum value - string id = fmt("%s%s", prefix.c_str(), cname); + string id = fmt("%s%s", prefix.c_str(), cname.c_str()); tag_enum_type->AddName(module, id.c_str(), component->Tag().AsEnumVal()->InternalInt(), true); } diff --git a/src/plugin/TaggedComponent.h b/src/plugin/TaggedComponent.h index 99eab9f230..7e3ee24bdf 100644 --- a/src/plugin/TaggedComponent.h +++ b/src/plugin/TaggedComponent.h @@ -23,28 +23,12 @@ public: */ TaggedComponent(typename T::subtype_t subtype = 0); - /** - * Copy constructor. - * - * @param other Another component from which to copy its tag value. - */ - TaggedComponent(const TaggedComponent& other); - - /** - * Assignment operator. - * - * @param other A component to assign. - * @return The assigned object. - */ - TaggedComponent& operator=(const TaggedComponent& other); - /** * @return The component's tag. */ T Tag() const; private: - T tag; /**< The automatically assigned analyzer tag. */ static typename T::type_t type_counter; /**< Used to generate globally unique tags. */ @@ -56,22 +40,6 @@ TaggedComponent::TaggedComponent(typename T::subtype_t subtype) tag = T(++type_counter, subtype); } -template -TaggedComponent::TaggedComponent(const TaggedComponent& other) - { - tag = other.tag; - } - -template -TaggedComponent& -TaggedComponent::operator =(const TaggedComponent& other) - { - if ( &other != this ) - tag = other.tag; - - return *this; - } - template T TaggedComponent::Tag() const { diff --git a/src/util.cc b/src/util.cc index 3b917495bc..e41925a414 100644 --- a/src/util.cc +++ b/src/util.cc @@ -1701,17 +1701,16 @@ const char* bro_magic_buffer(magic_t cookie, const void* buffer, size_t length) return rval; } -const char* canonify_name(const char* name) +std::string canonify_name(const std::string& name) { - unsigned int len = strlen(name); - char* nname = new char[len + 1]; + unsigned int len = name.size(); + std::string nname; for ( unsigned int i = 0; i < len; i++ ) { char c = isalnum(name[i]) ? name[i] : '_'; - nname[i] = toupper(c); + nname += toupper(c); } - nname[len] = '\0'; return nname; } diff --git a/src/util.h b/src/util.h index 8526ded03f..e8bce8221d 100644 --- a/src/util.h +++ b/src/util.h @@ -406,6 +406,6 @@ const char* bro_magic_buffer(magic_t cookie, const void* buffer, size_t length); * @param name The string to canonicalize. * @return The canonicalized version of \a name which caller may later delete[]. */ -const char* canonify_name(const char* name); +std::string canonify_name(const std::string& name); #endif From a80dd102153be5de92778226e8f80a466981b7c2 Mon Sep 17 00:00:00 2001 From: Robin Sommer Date: Mon, 16 Dec 2013 10:08:38 -0800 Subject: [PATCH 17/56] Updates of the dynamic plugin code. Includes: - Cleanup of the plugin API, in particular generally changing const char* to std::string - Renaming environment variable BRO_PLUGINS to BRO_PLUGIN_PATH, defaulting to /lib/bro/plugins - Reworking how dynamic plugins are searched and activated. See doc/devel/plugins.rst for details. - New @load-plugin directive to explicitly activate a plugin - Support for Darwin. (Linux untested right now) - The init-plugin updates come with support for "make test", "make sdist", and "make bdist" (see how-to). - Test updates. Notes: The new hook mechanism, which allows plugins to hook into Bro's core a well-defined points, is still essentially untested. --- CMakeLists.txt | 3 +- aux/bro-aux | 2 +- cmake | 2 +- config.h.in | 2 +- src/BroDoc.cc | 4 +- src/CMakeLists.txt | 3 + src/DebugLogger.cc | 2 +- src/input.h | 1 + src/main.cc | 10 +- src/plugin/Manager.cc | 156 ++++++++++++++++------ src/plugin/Manager.h | 83 +++++++----- src/plugin/Plugin.cc | 58 ++++---- src/plugin/Plugin.h | 36 ++--- src/scan.l | 16 +++ src/util.cc | 9 +- src/util.h | 3 + testing/btest/core/plugins/analyzer.bro | 4 +- testing/btest/core/plugins/test-plugin.sh | 6 +- 18 files changed, 257 insertions(+), 143 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index c006237850..c71762e112 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -22,10 +22,11 @@ set(BRO_SCRIPT_SOURCE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/scripts) get_filename_component(BRO_SCRIPT_INSTALL_PATH ${BRO_SCRIPT_INSTALL_PATH} ABSOLUTE) -set(BRO_PLUGIN_INSTALL_PATH ${BRO_ROOT_DIR}/lib/bro/plugins) +set(BRO_PLUGIN_INSTALL_PATH ${BRO_ROOT_DIR}/lib/bro/plugins CACHE STRING "Installation path for plugins" FORCE) set(BRO_MAGIC_INSTALL_PATH ${BRO_ROOT_DIR}/share/bro/magic) set(BRO_MAGIC_SOURCE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/magic/database) + configure_file(bro-path-dev.in ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev) file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev.sh "export BROPATH=`${CMAKE_CURRENT_BINARY_DIR}/bro-path-dev`\n" diff --git a/aux/bro-aux b/aux/bro-aux index 715937ad72..f89d870437 160000 --- a/aux/bro-aux +++ b/aux/bro-aux @@ -1 +1 @@ -Subproject commit 715937ad72d9158d8c3142f81ee7005601d75db8 +Subproject commit f89d870437dbbbb107fd29f1085aef070e3bf373 diff --git a/cmake b/cmake index 6eb4cbfbc7..763b6d2574 160000 --- a/cmake +++ b/cmake @@ -1 +1 @@ -Subproject commit 6eb4cbfbc75f0bd13e247150b19932905a85e10c +Subproject commit 763b6d2574294e62f116e167c564a6292206a655 diff --git a/config.h.in b/config.h.in index 61124d2f67..d3889a2d90 100644 --- a/config.h.in +++ b/config.h.in @@ -214,7 +214,7 @@ #define HOST_ARCHITECTURE "@HOST_ARCHITECTURE@" /* String with extension of dynamic libraries (e.g., ".so") */ -#define SHARED_LIBRARY_SUFFIX "@CMAKE_SHARED_LIBRARY_SUFFIX@" +#define DYNAMIC_PLUGIN_SUFFIX "@CMAKE_SHARED_MODULE_SUFFIX@" /* True if we're building outside of the main Bro source code tree. */ #ifndef BRO_PLUGIN_INTERNAL_BUILD diff --git a/src/BroDoc.cc b/src/BroDoc.cc index 0a1006bd07..03c29d2201 100644 --- a/src/BroDoc.cc +++ b/src/BroDoc.cc @@ -467,7 +467,7 @@ static void WritePluginSectionHeading(FILE* f, const plugin::Plugin* p) fprintf(f, "-"); fprintf(f, "\n\n"); - fprintf(f, "%s\n\n", p->Description()); + fprintf(f, "%s\n\n", p->Description().c_str()); } static void WriteAnalyzerComponent(FILE* f, const analyzer::Component* c) @@ -568,7 +568,7 @@ static void WritePluginBifItems(FILE* f, const plugin::Plugin* p, if ( o ) o->WriteReST(f); else - reporter->Warning("No docs for ID: %s\n", it->GetID()); + reporter->Warning("No docs for ID: %s\n", it->GetID().c_str()); } } diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt index 34af4eb974..8a0b9836c7 100644 --- a/src/CMakeLists.txt +++ b/src/CMakeLists.txt @@ -389,6 +389,9 @@ install(TARGETS bro DESTINATION bin) set(BRO_EXE bro CACHE STRING "Bro executable binary" FORCE) +set(BRO_EXE_PATH ${CMAKE_CURRENT_BINARY_DIR}/bro + CACHE STRING "Path to Bro executable binary" FORCE) + # Target to create all the autogenerated files. add_custom_target(generate_outputs_stage1) add_dependencies(generate_outputs_stage1 ${bro_ALL_GENERATED_OUTPUTS}) diff --git a/src/DebugLogger.cc b/src/DebugLogger.cc index 267383503d..88635a6fee 100644 --- a/src/DebugLogger.cc +++ b/src/DebugLogger.cc @@ -117,7 +117,7 @@ void DebugLogger::Log(const plugin::Plugin& plugin, const char* fmt, ...) return; fprintf(file, "%17.06f/%17.06f [plugin %s] ", - network_time, current_time(true), plugin.Name()); + network_time, current_time(true), plugin.Name().c_str()); va_list ap; va_start(ap, fmt); diff --git a/src/input.h b/src/input.h index 8fcceb256b..d07a82f2ee 100644 --- a/src/input.h +++ b/src/input.h @@ -15,6 +15,7 @@ extern int brolex(); extern char last_tok[128]; extern void add_input_file(const char* file); +extern void add_input_file_at_front(const char* file); // Adds the substrings (using the given delimiter) in a string to the // given namelist. diff --git a/src/main.cc b/src/main.cc index c3c32a4b14..abafff5a85 100644 --- a/src/main.cc +++ b/src/main.cc @@ -223,7 +223,7 @@ void usage() fprintf(stderr, " $BROPATH | file search path (%s)\n", bro_path().c_str()); fprintf(stderr, " $BROMAGIC | libmagic mime magic database search path (%s)\n", bro_magic_path()); - fprintf(stderr, " $BRO_PLUGINS | plugin search path (%s)\n", bro_plugin_path()); + fprintf(stderr, " $BRO_PLUGIN_PATH | plugin search path (%s)\n", bro_plugin_path()); fprintf(stderr, " $BRO_PREFIXES | prefix list (%s)\n", bro_prefixes().c_str()); fprintf(stderr, " $BRO_DNS_FAKE | disable DNS lookups (%s)\n", bro_dns_fake()); fprintf(stderr, " $BRO_SEED_FILE | file to load seeds from (not set)\n"); @@ -820,7 +820,7 @@ int main(int argc, char** argv) if ( ! bare_mode ) add_input_file("base/init-default.bro"); - plugin_mgr->LoadPluginsFrom(bro_plugin_path()); + plugin_mgr->SearchDynamicPlugins(bro_plugin_path()); if ( optind == argc && read_files.length() == 0 && flow_files.length() == 0 && @@ -860,6 +860,9 @@ int main(int argc, char** argv) analyzer_mgr->InitPreScript(); file_mgr->InitPreScript(); + if ( ! bare_mode ) + plugin_mgr->ActivateAllDynamicPlugins(); + if ( events_file ) event_player = new EventPlayer(events_file); @@ -896,13 +899,14 @@ int main(int argc, char** argv) if ( reporter->Errors() > 0 ) exit(1); + plugin_mgr->InitPostScript(); + if ( print_plugins ) { show_plugins(print_plugins); exit(1); } - plugin_mgr->InitPostScript(); analyzer_mgr->InitPostScript(); file_mgr->InitPostScript(); diff --git a/src/plugin/Manager.cc b/src/plugin/Manager.cc index be1d7827a5..76e154067a 100644 --- a/src/plugin/Manager.cc +++ b/src/plugin/Manager.cc @@ -1,6 +1,7 @@ // See the file "COPYING" in the main distribution directory for copyright. #include +#include #include #include #include @@ -15,6 +16,7 @@ using namespace plugin; +Plugin* Manager::current_plugin = 0; string Manager::current_dir; string Manager::current_sopath; @@ -37,7 +39,7 @@ Manager::~Manager() delete [] hooks; } -void Manager::LoadPluginsFrom(const string& dir) +void Manager::SearchDynamicPlugins(const std::string& dir) { assert(! init); @@ -51,7 +53,7 @@ void Manager::LoadPluginsFrom(const string& dir) std::string d; while ( std::getline(s, d, ':') ) - LoadPluginsFrom(d); + SearchDynamicPlugins(d); return; } @@ -62,12 +64,34 @@ void Manager::LoadPluginsFrom(const string& dir) return; } - int rc = LoadPlugin(dir); + // Check if it's a plugin dirctory. - if ( rc >= 0 ) + const std::string magic = dir + "/__bro_plugin__"; + + if ( is_file(magic) ) + { + // It's a plugin, get it's name. + std::ifstream in(magic); + + if ( in.fail() ) + reporter->FatalError("cannot open plugin magic file %s", magic.c_str()); + + std::string name; + std::getline(in, name); + strstrip(name); + + if ( name.empty() ) + reporter->FatalError("empty plugin magic file %s", magic.c_str()); + + // Record it, so that we can later activate it. + + dynamic_plugins.insert(std::make_pair(name, dir)); + + DBG_LOG(DBG_PLUGINS, "Found plugin %s in %s", name.c_str(), dir.c_str()); return; + } - DBG_LOG(DBG_PLUGINS, "Searching directory %s recursively for plugins", dir.c_str()); + // No plugin here, traverse subirectories. DIR* d = opendir(dir.c_str()); @@ -98,22 +122,33 @@ void Manager::LoadPluginsFrom(const string& dir) } if ( st.st_mode & S_IFDIR ) - LoadPluginsFrom(path); + SearchDynamicPlugins(path); } } -int Manager::LoadPlugin(const std::string& dir) +bool Manager::ActivateDynamicPluginInternal(const std::string& name) { - assert(! init); + dynamic_plugin_map::iterator m = dynamic_plugins.find(name); - // Check if it's a plugin dirctory. - if ( ! is_file(dir + "/__bro_plugin__") ) - return -1; + if ( m == dynamic_plugins.end() ) + { + reporter->Error("plugin %s is not available", name.c_str()); + return false; + } - DBG_LOG(DBG_PLUGINS, "Loading plugin from %s", dir.c_str()); + std::string dir = m->second + "/"; + + if ( dir.empty() ) + { + // That's our marker that we have already activated this + // plugin. Silently ignore the new request. + return true; + } + + DBG_LOG(DBG_PLUGINS, "Activating plugin %s", name.c_str()); // Add the "scripts" and "bif" directories to BROPATH. - string scripts = dir + "/scripts"; + std::string scripts = dir + "scripts"; if ( is_dir(scripts) ) { @@ -121,44 +156,27 @@ int Manager::LoadPlugin(const std::string& dir) add_to_bro_path(scripts); } - string bif = dir + "/bif"; + // Load {bif,scripts}/__load__.bro automatically. - if ( is_dir(bif) ) - { - DBG_LOG(DBG_PLUGINS, " Adding %s to BROPATH", bif.c_str()); - add_to_bro_path(bif); - } - - // Load dylib/scripts/__load__.bro automatically. - string dyinit = dir + "/dylib/scripts/__load__.bro"; - - if ( is_file(dyinit) ) - { - DBG_LOG(DBG_PLUGINS, " Adding %s for loading", dyinit.c_str()); - add_input_file(dyinit.c_str()); - } - - // Load scripts/__load__.bro automatically. - string init = scripts + "/__load__.bro"; + string init = dir + "lib/bif/__load__.bro"; if ( is_file(init) ) { - DBG_LOG(DBG_PLUGINS, " Adding %s for loading", init.c_str()); - add_input_file(init.c_str()); + DBG_LOG(DBG_PLUGINS, " Loading %s", init.c_str()); + scripts_to_load.push_back(init); } - // Load bif/__load__.bro automatically. - init = bif + "/__load__.bro"; + init = dir + "scripts/__load__.bro"; if ( is_file(init) ) { - DBG_LOG(DBG_PLUGINS, " Adding %s for loading", init.c_str()); - add_input_file(init.c_str()); + DBG_LOG(DBG_PLUGINS, " Loading %s", init.c_str()); + scripts_to_load.push_back(init); } // Load shared libraries. - string dypattern = dir + "/dylib/*." + HOST_ARCHITECTURE + SHARED_LIBRARY_SUFFIX; + string dypattern = dir + "/lib/*." + HOST_ARCHITECTURE + DYNAMIC_PLUGIN_SUFFIX; DBG_LOG(DBG_PLUGINS, " Searching for shared libraries %s", dypattern.c_str()); @@ -170,11 +188,10 @@ int Manager::LoadPlugin(const std::string& dir) { const char* path = gl.gl_pathv[i]; + current_plugin = 0; current_dir = dir; current_sopath = path; void* hdl = dlopen(path, RTLD_LAZY | RTLD_GLOBAL); - current_dir.clear(); - current_sopath.clear(); if ( ! hdl ) { @@ -182,6 +199,24 @@ int Manager::LoadPlugin(const std::string& dir) reporter->FatalError("cannot load plugin library %s: %s", path, err ? err : ""); } + if ( ! current_plugin ) + reporter->FatalError("load plugin library %s did not instantiate a plugin", path); + + // We execute the pre-script initialization here; this in + // fact could be *during* script initialization if we got + // triggered via @load-plugin. + current_plugin->InitPreScript(); + + // Make sure the name the plugin reports is consistent with + // what we expect from its magic file. + if ( string(current_plugin->Name()) != name ) + reporter->FatalError("inconsistent plugin name: %s vs %s", + current_plugin->Name().c_str(), name.c_str()); + + current_dir.clear(); + current_sopath.clear(); + current_plugin = 0; + DBG_LOG(DBG_PLUGINS, " Loaded %s", path); } } @@ -189,10 +224,42 @@ int Manager::LoadPlugin(const std::string& dir) else { DBG_LOG(DBG_PLUGINS, " No shared library found"); - return 1; } - return 1; + // Mark this plugin as activated by clearing the path. + m->second.clear(); + + return true; + } + +bool Manager::ActivateDynamicPlugin(const std::string& name) + { + if ( ! ActivateDynamicPluginInternal(name) ) + return false; + + UpdateInputFiles(); + return true; + } + +bool Manager::ActivateAllDynamicPlugins() + { + for ( dynamic_plugin_map::const_iterator i = dynamic_plugins.begin(); + i != dynamic_plugins.end(); i++ ) + { + if ( ! ActivateDynamicPluginInternal(i->first) ) + return false; + } + + UpdateInputFiles(); + + return true; + } + +void Manager::UpdateInputFiles() + { + for ( file_list::const_reverse_iterator i = scripts_to_load.rbegin(); + i != scripts_to_load.rend(); i++ ) + add_input_file_at_front((*i).c_str()); } static bool plugin_cmp(const Plugin* a, const Plugin* b) @@ -205,11 +272,13 @@ bool Manager::RegisterPlugin(Plugin *plugin) Manager::PluginsInternal()->push_back(plugin); if ( current_dir.size() && current_sopath.size() ) + // A dynamic plugin, record its location. plugin->SetPluginLocation(current_dir.c_str(), current_sopath.c_str()); // Sort plugins by name to make sure we have a deterministic order. PluginsInternal()->sort(plugin_cmp); + current_plugin = plugin; return true; } @@ -220,7 +289,6 @@ void Manager::InitPreScript() for ( plugin_list::iterator i = Manager::PluginsInternal()->begin(); i != Manager::PluginsInternal()->end(); i++ ) { Plugin* plugin = *i; - plugin->InitPreScript(); } @@ -333,7 +401,7 @@ void Manager::DisableHook(HookType hook, Plugin* plugin) } } -int Manager::HookLoadFile(const char* file) +int Manager::HookLoadFile(const string& file) { hook_list* l = hooks[HOOK_LOAD_FILE]; diff --git a/src/plugin/Manager.h b/src/plugin/Manager.h index 0c52679a64..5c5c7430be 100644 --- a/src/plugin/Manager.h +++ b/src/plugin/Manager.h @@ -40,17 +40,41 @@ public: virtual ~Manager(); /** - * Loads all plugins dynamically from a set of directories. Multiple - * directories are split by ':'. If a directory does not contain a - * plugin itself, the method searches for plugins recursively. For - * plugins found, the method loads the plugin's shared library and - * makes its scripts available to the interpreter. + * Searches a set of directories for plugins. If a specificed + * directory does not contain a plugin itself, the method searches + * for plugins recursively. For plugins found, the method makes them + * available for later activation via ActivatePlugin(). * * This must be called only before InitPluginsPreScript(). * - * @param dir The directory to search for plugins. + * @param dir The directory to search for plugins. Multiple + * directories are split by ':'. */ - void LoadPluginsFrom(const std::string& dir); + void SearchDynamicPlugins(const std::string& dir); + + /** + * Activates a plugin that SearchPlugins() has previously discovered. + * Activing a plugin involved loading its dynamic module, making its + * bifs available, and adding its script paths to BROPATH. + * + * @param name The name of the plugin, as determined previously by + * SearchPlugin(). + * + * @return True if the plugin has been loaded successfully. + * + */ + bool ActivateDynamicPlugin(const std::string& name); + + /** + * Activates all plugins that SearchPlugins() has previously + * discovered. The effect is the same all calling \a + * ActivePlugin(name) for every plugin. + * + * @return True if all plugins have been loaded successfully. If one + * fail to load, the method stops there without loading any furthers + * and returns false. + */ + bool ActivateAllDynamicPlugins(); /** * First-stage initializion of the manager. This is called early on @@ -148,7 +172,7 @@ public: * successfully; 0 if a plugin took over the file but had trouble * loading it; and -1 if no plugin was interested in the file at all. */ - virtual int HookLoadFile(const char* file); + virtual int HookLoadFile(const string& file); /** * Hook that filters calls to a script function/event/hook. @@ -196,40 +220,37 @@ public: */ static bool RegisterPlugin(Plugin* plugin); -protected: - /** - * Loads a plugin dynamically from a given directory. It loads the - * plugin's shared library, and makes its scripts available to the - * interpreter. Different from LoadPluginsFrom() this method does not - * further descend the directory tree recursively to search for - * plugins. - * - * This must be called only before InitPluginsPreScript() - * - * @param file The path to the plugin to load. - * - * @return 0 if there's a plugin in this directory, but there was a - * problem loading it; -1 if there's no plugin at all in this - * directory; 1 if there's a plugin in this directory and we loaded - * it successfully. - */ - int LoadPlugin(const std::string& dir); - private: + bool ActivateDynamicPluginInternal(const std::string& name); + void UpdateInputFiles(); + + // All found dynamic plugins, mapping their names to base directory. + typedef std::map dynamic_plugin_map; + dynamic_plugin_map dynamic_plugins; + + // We buffer scripts to load temporarliy to get them to load in the + // right order. + typedef std::list file_list; + file_list scripts_to_load; + + bool init; + // A hook list keeps pairs of plugin and priority interested in a // given hook. typedef std::list > hook_list; - static plugin_list* PluginsInternal(); - - bool init; - // An array indexed by HookType. An entry is null if there's no hook // of that type enabled. hook_list** hooks; + static Plugin* current_plugin; static string current_dir; static string current_sopath; + + // Returns a modifiable list of all plugins, both static and dynamic. + // This is a static method so that plugins can register themselves + // even before the manager exists. + static plugin_list* PluginsInternal(); }; template diff --git a/src/plugin/Plugin.cc b/src/plugin/Plugin.cc index 20d436907b..000bb2af52 100644 --- a/src/plugin/Plugin.cc +++ b/src/plugin/Plugin.cc @@ -26,15 +26,15 @@ const char* hook_name(HookType h) return hook_names[int(h)]; } -BifItem::BifItem(const char* arg_id, Type arg_type) +BifItem::BifItem(const std::string& arg_id, Type arg_type) { - id = copy_string(arg_id); + id = arg_id; type = arg_type; } BifItem::BifItem(const BifItem& other) { - id = copy_string(other.id); + id = other.id; type = other.type; } @@ -42,7 +42,7 @@ BifItem& BifItem::operator=(const BifItem& other) { if ( this != &other ) { - id = copy_string(other.id); + id = other.id; type = other.type; } @@ -51,20 +51,19 @@ BifItem& BifItem::operator=(const BifItem& other) BifItem::~BifItem() { - delete [] id; } Plugin::Plugin() { - name = copy_string(""); - description = copy_string(""); + name = ""; + description = ""; // These will be reset by the BRO_PLUGIN_* macros. version = -9999; api_version = -9999; dynamic = false; - base_dir = 0; - sopath = 0; + base_dir = ""; + sopath = ""; Manager::RegisterPlugin(this); } @@ -72,33 +71,26 @@ Plugin::Plugin() Plugin::~Plugin() { Done(); - - delete [] name; - delete [] description; - delete [] base_dir; - delete [] sopath; } -const char* Plugin::Name() const +const std::string& Plugin::Name() const { return name; } -void Plugin::SetName(const char* arg_name) +void Plugin::SetName(const std::string& arg_name) { - delete [] name; - name = copy_string(arg_name); + name = arg_name; } -const char* Plugin::Description() const +const std::string& Plugin::Description() const { return description; } -void Plugin::SetDescription(const char* arg_description) +void Plugin::SetDescription(const std::string& arg_description) { - delete [] description; - description = copy_string(arg_description); + description = arg_description; } int Plugin::Version() const @@ -121,12 +113,12 @@ bool Plugin::DynamicPlugin() const return dynamic; } -const char* Plugin::PluginDirectory() const +const std::string& Plugin::PluginDirectory() const { return base_dir; } -const char* Plugin::PluginPath() const +const std::string& Plugin::PluginPath() const { return sopath; } @@ -141,12 +133,10 @@ void Plugin::SetDynamicPlugin(bool arg_dynamic) dynamic = arg_dynamic; } -void Plugin::SetPluginLocation(const char* arg_dir, const char* arg_sopath) +void Plugin::SetPluginLocation(const std::string& arg_dir, const std::string& arg_sopath) { - delete [] base_dir; - delete [] sopath; - base_dir = copy_string(arg_dir); - sopath = copy_string(arg_sopath); + base_dir = arg_dir; + sopath = arg_sopath; } void Plugin::InitPreScript() @@ -197,9 +187,9 @@ static bool component_cmp(const Component* a, const Component* b) return a->Name() < b->Name(); } -bool Plugin::LoadBroFile(const char* file) +bool Plugin::LoadBroFile(const std::string& file) { - ::add_input_file(file); + ::add_input_file(file.c_str()); return true; } @@ -208,7 +198,7 @@ void Plugin::__AddBifInitFunction(bif_init_func c) bif_inits.push_back(c); } -void Plugin::AddBifItem(const char* name, BifItem::Type type) +void Plugin::AddBifItem(const std::string& name, BifItem::Type type) { BifItem bi(name, (BifItem::Type)type); bif_items.push_back(bi); @@ -238,7 +228,7 @@ void Plugin::DisableHook(HookType hook) plugin_mgr->DisableHook(hook, this); } -int Plugin::HookLoadFile(const char* file) +int Plugin::HookLoadFile(const std::string& file) { return -1; } @@ -266,7 +256,7 @@ void Plugin::Describe(ODesc* d) const d->Add("Plugin: "); d->Add(name); - if ( description && *description ) + if ( description.size() ) { d->Add(" - "); d->Add(description); diff --git a/src/plugin/Plugin.h b/src/plugin/Plugin.h index 915563e98c..7da5a74c6f 100644 --- a/src/plugin/Plugin.h +++ b/src/plugin/Plugin.h @@ -50,7 +50,7 @@ public: * * @param type The type of the item. */ - BifItem(const char* id, Type type); + BifItem(const std::string& id, Type type); /** * Copy constructor. @@ -70,7 +70,7 @@ public: /** * Returns the script-level ID as passed into the constructor. */ - const char* GetID() const { return id; } + const std::string& GetID() const { return id; } /** * Returns the type as passed into the constructor. @@ -78,7 +78,7 @@ public: Type GetType() const { return type; } private: - const char* id; + std::string id; Type type; }; @@ -113,7 +113,7 @@ public: typedef std::list component_list; typedef std::list bif_item_list; typedef std::list > hook_list; - typedef std::list > bif_init_func_result; + typedef std::list > bif_init_func_result; typedef void (*bif_init_func)(Plugin *); /** @@ -129,12 +129,12 @@ public: /** * Returns the name of the plugin. */ - const char* Name() const; + const std::string& Name() const; /** * Returns a short textual description of the plugin, if provided. */ - const char* Description() const; + const std::string& Description() const; /** * Returns the version of the plugin. Version are only meaningful for @@ -152,13 +152,13 @@ public: * For dynamic plugins, returns the base directory from which it was * loaded. For static plugins, returns null. **/ - const char* PluginDirectory() const; + const std::string& PluginDirectory() const; /** * For dynamic plugins, returns the full path to the shared library * from which it was loaded. For static plugins, returns null. **/ - const char* PluginPath() const; + const std::string& PluginPath() const; /** * Returns the internal API version that this plugin relies on. Only @@ -225,7 +225,7 @@ public: * * @param type The item's type. */ - void AddBifItem(const char* name, BifItem::Type type); + void AddBifItem(const std::string& name, BifItem::Type type); /** * Adds a file to the list of files that Bro loads at startup. This @@ -242,7 +242,7 @@ public: * @return True if successful (which however may only mean * "successfully queued"). */ - bool LoadBroFile(const char* file); + bool LoadBroFile(const std::string& file); /** * Internal function adding an entry point for registering @@ -324,7 +324,7 @@ protected: * printed an error message); and -1 if the plugin wasn't interested * in the file at all. */ - virtual int HookLoadFile(const char* file); + virtual int HookLoadFile(const std::string& file); /** * Hook into executing a script-level function/event/hook. Whenever @@ -396,7 +396,7 @@ protected: * * @param name The name. Makes a copy internally. */ - void SetName(const char* name); + void SetName(const std::string& name); /** * Sets the plugin's textual description. @@ -406,7 +406,7 @@ protected: * * @param name The description. Makes a copy internally. */ - void SetDescription(const char* descr); + void SetDescription(const std::string& descr); /** * Sets the plugin's version. @@ -454,7 +454,7 @@ protected: * @param sopath The full path the shared library loaded. The * functions makes an internal copy of string. */ - void SetPluginLocation(const char* dir, const char* sopath); + void SetPluginLocation(const std::string& dir, const std::string& sopath); private: /** @@ -465,10 +465,10 @@ private: typedef std::list bif_init_func_list; - const char* name; - const char* description; - const char* base_dir; - const char* sopath; + std::string name; + std::string description; + std::string base_dir; + std::string sopath; int version; int api_version; bool dynamic; diff --git a/src/scan.l b/src/scan.l index 599a660fee..d5eedc83c3 100644 --- a/src/scan.l +++ b/src/scan.l @@ -407,6 +407,11 @@ when return TOK_WHEN; new_sig_file); } +@load-plugin{WS}{ID} { + const char* plugin = skip_whitespace(yytext + 12); + plugin_mgr->ActivateDynamicPlugin(plugin); +} + @unload{WS}{FILE} { // Skip "@unload". const char* new_file = skip_whitespace(yytext + 7); @@ -829,6 +834,17 @@ void add_input_file(const char* file) input_files.append(copy_string(file)); } +void add_input_file_at_front(const char* file) + { + if ( ! file ) + reporter->InternalError("empty filename"); + + if ( ! filename ) + (void) load_files(file); + else + input_files.insert(copy_string(file)); + } + void add_to_name_list(char* s, char delim, name_list& nl) { while ( s ) diff --git a/src/util.cc b/src/util.cc index e41925a414..917ddf9388 100644 --- a/src/util.cc +++ b/src/util.cc @@ -664,6 +664,13 @@ string strreplace(const string& s, const string& o, const string& n) return r; } +std::string strstrip(std::string s) + { + s.erase(s.begin(), std::find_if(s.begin(), s.end(), std::not1(std::ptr_fun(std::isspace)))); + s.erase(std::find_if(s.rbegin(), s.rend(), std::not1(std::ptr_fun(std::isspace))).base(), s.end()); + return s; + } + int hmac_key_set = 0; uint8 shared_hmac_md5_key[16]; @@ -942,7 +949,7 @@ const char* bro_magic_path() const char* bro_plugin_path() { - const char* path = getenv("BRO_PLUGINS"); + const char* path = getenv("BRO_PLUGIN_PATH"); if ( ! path ) path = BRO_PLUGIN_INSTALL_PATH; diff --git a/src/util.h b/src/util.h index e8bce8221d..feae01c0fb 100644 --- a/src/util.h +++ b/src/util.h @@ -155,6 +155,9 @@ bool is_file(const std::string& path); // Replaces all occurences of *o* in *s* with *n*. extern std::string strreplace(const std::string& s, const std::string& o, const std::string& n); +// Remove all leading and trainling white space from string. +extern std::string strstrip(std::string s); + extern uint8 shared_hmac_md5_key[16]; extern int hmac_key_set; diff --git a/testing/btest/core/plugins/analyzer.bro b/testing/btest/core/plugins/analyzer.bro index 2a72797564..c9831ad269 100644 --- a/testing/btest/core/plugins/analyzer.bro +++ b/testing/btest/core/plugins/analyzer.bro @@ -1,9 +1,9 @@ # @TEST-EXEC: ${DIST}/aux/bro-aux/plugin-support/init-plugin Demo Foo # @TEST-EXEC: cp -r %DIR/analyzer-plugin/* . # @TEST-EXEC: make BRO=${DIST} -# @TEST-EXEC: BROPLUGINS=`pwd` bro -NN | awk '/^Plugin:.*Demo/ {p=1; print; next} /^Plugin:/{p=0} p==1{print}' >>output +# @TEST-EXEC: BRO_PLUGIN_PATH=`pwd` bro -NN | awk '/^Plugin:.*Demo/ {p=1; print; next} /^Plugin:/{p=0} p==1{print}' >>output # @TEST-EXEC: echo === >>output -# @TEST-EXEC: BROPLUGINS=`pwd` bro -r $TRACES/port4242.trace %INPUT >>output +# @TEST-EXEC: BRO_PLUGIN_PATH=`pwd` bro -r $TRACES/port4242.trace %INPUT >>output # @TEST-EXEC: btest-diff output event foo_message(c: connection, data: string) diff --git a/testing/btest/core/plugins/test-plugin.sh b/testing/btest/core/plugins/test-plugin.sh index 87b09b2e4d..3b67b9454f 100644 --- a/testing/btest/core/plugins/test-plugin.sh +++ b/testing/btest/core/plugins/test-plugin.sh @@ -1,11 +1,11 @@ # @TEST-EXEC: ${DIST}/aux/bro-aux/plugin-support/init-plugin Demo Foo # @TEST-EXEC: bash %INPUT # @TEST-EXEC: make BRO=${DIST} -# @TEST-EXEC: BROPLUGINS=`pwd` bro -NN | awk '/^Plugin:.*Demo/ {p=1; print; next} /^Plugin:/{p=0} p==1{print}' >>output +# @TEST-EXEC: BRO_PLUGIN_PATH=`pwd` bro -NN | awk '/^Plugin:.*Demo/ {p=1; print; next} /^Plugin:/{p=0} p==1{print}' >>output # @TEST-EXEC: echo === >>output -# @TEST-EXEC: BROPLUGINS=`pwd` bro -r $TRACES/empty.trace >>output +# @TEST-EXEC: BRO_PLUGIN_PATH=`pwd` bro -r $TRACES/empty.trace >>output # @TEST-EXEC: echo === >>output -# @TEST-EXEC: BROPLUGINS=`pwd` bro demo/foo -r $TRACES/empty.trace >>output +# @TEST-EXEC: BRO_PLUGIN_PATH=`pwd` bro demo/foo -r $TRACES/empty.trace >>output # @TEST-EXEC: btest-diff output cat >scripts/__load__.bro < Date: Mon, 16 Dec 2013 10:09:16 -0800 Subject: [PATCH 18/56] Start of a plugin writing how-to. See doc/devel/plugins.rst. It includes a simple example and background on how things work. --- doc/devel/plugins.rst | 339 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 339 insertions(+) create mode 100644 doc/devel/plugins.rst diff --git a/doc/devel/plugins.rst b/doc/devel/plugins.rst new file mode 100644 index 0000000000..f00bbb4d2c --- /dev/null +++ b/doc/devel/plugins.rst @@ -0,0 +1,339 @@ + +=================== +Writing Bro Plugins +=================== + +Bro is internally moving to a plugin structure that enables extending +the system dynamically, without modifying the core code base. That way +custom code remains self-contained and can be maintained, compiled, +and installed independently. Currently, plugins can add the following +functionality to Bro: + + - Bro scripts. + + - Builtin functions/events/types for the scripting language. + + - Protocol and file analyzers. + + - Packet sources and packet dumpers. TODO: Not yet. + + - Logging framework backends. TODO: Not yet. + + - Input framework readers. TODO: Not yet. + +A plugin's functionality is available to the user just as if Bro had +the corresponding code built-in. Indeed, internally many of Bro's +pieces are structured as plugins as well, they are just statically +compiled into the binary rather than loaded dynamically at runtime, as +external plugins are. + +Quick Start +=========== + +Writing a basic plugin is quite straight-forward as long as one +follows a few conventions. In the following we walk through adding a +new built-in function (bif) to Bro; we'll add `a `rot13(s: string) : +string``, a function that rotates every character in a string by 13 +places. + +A plugin comes in the form of a directory following a certain +structure. To get started, Bro's distribution provides a helper script +``aux/bro-aux/plugin-support/init-plugin`` that creates a skeleton +plugin that can then be customized. Let's use that:: + + # mkdir rot13-plugin + # cd rot13-plugin + # init-plugin Demo Rot13 + +As you can see the script takes two arguments. The first is a +namespace the plugin will live in, and the second a descriptive name +for the plugin itself. Bro uses the combination of the two to identify +a plugin. The namespace serves to avoid naming conflicts between +plugins written by independent developers; pick, e.g., the name of +your organisation (and note that the namespace ``Bro`` is reserved for +functionality distributed by the Bro Project). In our example, the +plugin will be called ``Demo::Rot13``. + +The ``init-plugin`` script puts a number of files in place. The full +layout is described later. For now, all we need is +``src/functions.bif``. It's initially empty, but we'll add our new bif +there as follows:: + + # cat scripts/functions.bif + module CaesarCipher; + + function camel_case%(s: string%) : string + %{ + char* rot13 = copy_string(s->CheckString()); + + for ( char* p = rot13; *p; p++ ) + { + char b = islower(*p) ? 'a' : 'A'; + *p = (*p - b + 13) % 26 + b; + } + + return new StringVal(strlen(rot13), rot13); + %} + +The syntax of this file is just like any other ``*.bif`` file; we +won't go into it here. + +Now we can already compile our plugin, we just need to tell the +Makefile put in place by ``init-plugin`` where the Bro source tree is +located (Bro needs to have been built there first):: + + # make BRO=/path/to/bro/dist + [... cmake output ...] + +Now our ``rot13-plugin`` directory has everything that it needs +for Bro to recognize it as a dynamic plugin. Once we point Bro to it, +it will pull it in automatically, as we can check with the ``-N`` +option: + + # export BRO_PLUGIN_PATH=/path/to/rot13-plugin + # bro -N + [...] + Plugin: Demo::Rot13 - (dynamic, version 1) + [...] + +That looks quite good, except for the dummy description that we should +replace with something nicer so that users will know what our plugin +is about. We do this by editing the ``BRO_PLUGIN_DESCRIPTION`` line +in ``src/Plugin.cc``, like this: + + # cat src/Plugin.cc + + #include + + BRO_PLUGIN_BEGIN(Demo, Rot13) + BRO_PLUGIN_VERSION(1); + BRO_PLUGIN_DESCRIPTION("Caesar cipher rotating a string's characters by 13 places."); + BRO_PLUGIN_BIF_FILE(consts); + BRO_PLUGIN_BIF_FILE(events); + BRO_PLUGIN_BIF_FILE(functions); + BRO_PLUGIN_END + + # make + [...] + # bro -N | grep Rot13 + Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) + +Better. Bro can also show us what exactly the plugin provides with the +more verbose option ``-NN``:: + + # bro -NN + [...] + Plugin: Demo::Rot13 - Caesar cipher rotating a string's characters by 13 places. (dynamic, version 1) + [Function] CaesarCipher::rot13 + [...] + +There's our function. Now let's use it:: + + # bro -e 'print CaesarCipher::rot13("Hello")' + Uryyb + +It works. We next install the plugin along with Bro itself, so that it +will find it directly without needing the ``BRO_PLUGIN_PATH`` +environment variable. If we first unset the variable, the function +will no longer be available:: + + # unset BRO_PLUGIN_PATH + # bro -e 'print CaesarCipher::rot13("Hello")' + error in , line 1: unknown identifier CaesarCipher::rot13, at or near "CaesarCipher::rot13" + +Once we install it, it works again:: + + # make install + # bro -e 'print CaesarCipher::rot13("Hello")' + Uryyb + +The installed version went into +``/lib/bro/plugins/Demo_Rot13``. + +We can distribute the plugin in either source or binary form by using +the Makefile's ``sdist`` and ``bdist`` target, respectively. Both +create corrsponding tarballs:: + + # make sdist + [...] + Source distribution in build/sdist/Demo_Rot13.tar.gz + + # make bdist + [...] + Binary distribution in build/Demo_Rot13-darwin-x86_64.tar.gz + +The source archive will contain everything in the plugin directory +except any generated files. The binary archive will contain anything +needed to install and run the plugin, i.e., just what ``make install`` +puts into place as well. As the binary distribution is +platform-dependent, its name includes the OS and architecture the +plugin was built on. + +Plugin Directory Layout +======================= + +A plugin's directory needs to follow a set of conventions so that Bro +(1) recognizes it as a plugin, and (2) knows what to load. While +``init-plugin`` takes care of most of this, the following is the full +story. We'll use ```` to represent a plugin's top-level +directory. + +``/__bro_plugin__`` + A file that marks a directory as containing a Bro plugin. The file + must exist, and its content must consist of a single line with the + qualified name of the plugin (e.g., "Demo::Rot13"). + +``/lib/--.so`` + The shared library containing the plugin's compiled code. Bro will + load this in dynamically at run-time if OS and architecture match + the current platform. + +``lib/bif/`` + Directory with auto-generated Bro scripts that declare the plugins + bif elements. The files here are produced by ``bifcl``. + +``scripts/`` + A directory with the plugin's custom Bro scripts. When the plugin + gets activated, this directory will be automatically added to + ``BROPATH``, so that any scripts/modules inside can be + ``@load``ed. + +``scripts``/__load__.bro + A Bro script that will be loaded immediately when the plugin gets + activated. See below for more information on activating plugins. + +By convention, a plugin should put its custom scripts into sub folders +of ``scripts/``, i.e., ``scripts//