Merge remote-tracking branch 'origin/fastpath'

* origin/fastpath:
  Fix the find-bro-logs.test
  Fix typos/formatting in NEWS

CHANGES

@@ -1,4 +1,14 @@
 
+2.5-957 | 2018-09-04 09:28:47 -0500
+
+  * Fix/improve the find-bro-logs.test (Daniel Thayer)
+
+  * Fix typos/formatting in NEWS (Daniel Thayer)
+
+  * Clarify 'old_comm_usage_is_ok' error message (Jon Siwek, Corelight)
+
+  * Update Mozilla CA list to NSS 3.39 (Johanna Amann, Corelight)
+
 2.5-952 | 2018-08-31 17:30:21 -0500
 
   * Update NEWS (finalizations/formatting) (Jon Siwek, Corelight)

NEWS

@@ -16,7 +16,7 @@ New Functionality
   redone. There's a new script-level "broker" framework that
   supersedes the old "communication" framework, which is now
   deprecated.  All scripts that ship with Bro have been ported to use
-  to Broker.  BroControl has likewise been ported to use Broker.
+  Broker.  BroControl has likewise been ported to use Broker.
 
   For more about the new Broker framework, see
   https://www.bro.org/sphinx-git/frameworks/broker.html.  There's also
@@ -223,7 +223,7 @@ New Functionality
     - x509_ocsp_ext_signed_certificate_timestamp
     - ssl_extension_signed_certificate_timestamp
 
-  - Functions
+  - Functions:
 
     - sct_verify
     - x509_subject_name_hash
@@ -271,7 +271,7 @@ New Functionality
   - Files::file_exists
   - Files::lookup_File
 
-- New functions in the logging API
+- New functions in the logging API:
 
   - Log::get_filter_names
   - Log::enable_stream
@@ -284,7 +284,7 @@ New Functionality
   - This is not enabled by default (no ports are registered and no
     DPD signatures exist, so no connections will end up attaching the
     new Mount analyzer).  If it were to be activated by users, the
-    following events are available
+    following events are available:
 
     - mount_proc_null
     - mount_proc_mnt
@@ -293,7 +293,7 @@ New Functionality
     - mount_proc_not_implemented
     - mount_reply_status
 
-- Added new NFS events
+- Added new NFS events:
 
   - nfs_proc_symlink
   - nfs_proc_link
@@ -301,12 +301,12 @@ New Functionality
 
 - The SMB scripts in ``policy/protocols/smb`` are now moved into
   ``base/protocols/smb`` and loaded/enabled by default.  If you previously
-  loaded these scripts from their ``policy/ location`` (in local.bro or
+  loaded these scripts from their ``policy/`` location (in local.bro or
   other custom scripts) you may now remove/change those although they
   should still work since ``policy/protocols/smb`` is simply a placeholder
   script that redirects to the new ``base/`` location.
 
-- Added new SMB events
+- Added new SMB events:
 
   - smb1_transaction_secondary_request
   - smb1_transaction2_secondary_request
@@ -442,7 +442,7 @@ Changed Functionality
   "application/x-x509-ca-cert" for CA certificates.
 
 - The "ssl_server_curve" event is considered deprecated and will be removed
-  in in the future.  See the new "ssl_ecdh_server_params" event for a
+  in the future.  See the new "ssl_ecdh_server_params" event for a
   replacement.
 
 - The Socks analyzer no longer logs passwords by default. This
@@ -541,11 +541,11 @@ Removed Functionality
 
 - The "ocsp_request" event no longer has "requestorName" parameter.
 
+- The node-specific ``site/local-*.bro`` scripts have been removed.
+
 - BroControl: The "IPv6Comm" and "ZoneID" options are no longer
   available (though Broker should be able to handle IPv6 automatically).
 
-- The node-specific ``site/local-*.bro`` scripts have been removed.
-
 Deprecated Functionality
 ------------------------
 

VERSION

@@ -1 +1 @@
-2.5-952
+2.5-957

doc/script-reference/log-files.rst

@@ -152,7 +152,7 @@ Miscellaneous
 +----------------------------+---------------------------------------+---------------------------------+
 | weird.log                  | Unexpected network-level activity     | :bro:type:`Weird::Info`         |
 +----------------------------+---------------------------------------+---------------------------------+
-| weird-stats.log            | Statistics about unexpected activity  | :bro:type:`WeirdStats::Info`    |
+| weird_stats.log            | Statistics about unexpected activity  | :bro:type:`WeirdStats::Info`    |
 +----------------------------+---------------------------------------+---------------------------------+
 
 Bro Diagnostics

testing/btest/Baseline/coverage.find-bro-logs/out

@@ -4,7 +4,7 @@ capture_loss
 cluster
 config
 conn
-dce__r_pc
+dce_rpc
 dhcp
 dnp3
 dns
@@ -14,16 +14,16 @@ ftp
 http
 intel
 irc
+kerberos
 known_certs
 known_hosts
 known_modbus
 known_services
-krb
 loaded_scripts
 modbus
 modbus_register_change
 mysql
-net_control
+netcontrol
 netcontrol_catch_release
 netcontrol_drop
 netcontrol_shunt
@@ -31,7 +31,7 @@ notice
 notice_alarm
 ntlm
 ocsp
-open_flow
+openflow
 packet_filter
 pe
 radius

testing/btest/coverage/find-bro-logs.test

@@ -1,8 +1,8 @@
-# This test is intended to help keep Bro's reference documentation up-to-date.
-# If this test fails, then it indicates that the set of all the log filenames
-# that Bro could potentially create (with the scripts included with Bro) has
-# changed.  In that case, the reference documentation listing all Bro log files
-# should be checked and updated if necessary.
+# This test is intended to help keep the "Log Files" documentation page
+# up-to-date.  The output of this test should match all the log filenames
+# listed on the "Log Files" page in the documentation.
+#
+# If this test fails, then the "Log Files" documentation page should be updated.
 
 # @TEST-EXEC: bash %INPUT
 # @TEST-EXEC: btest-diff out
@@ -14,42 +14,56 @@ if [ ! -d "${BROSCRIPTS}" ]; then
     exit 1
 fi
 
-# For a given Bro script, look for a call to "create_stream".  If found,
-# extract the log ID (adding the module name if necessary), and print the
-# log ID and script filename.
-cat << '_EOF_' > find_logid.awk
-/module[ ]+[A-Za-z0-9_]/ {
-    mod = $2
-    if ( substr(mod, length(mod), 1) == ";" ) {
-        mod = substr(mod, 1, length(mod)-1)
-    }
-}
+python find_logs.py "${BROSCRIPTS}" | sort > out
 
-/Log::create_stream/ {
-    if ( substr($1, 1, 1) != "#" ) {
-        x = index($1, "(")
-        logid = substr($1, x+1, length($1)-x-1)
-        if ( logid == "LOG" ) {
-            printf "%s::", mod
-        }
-        printf "%s", logid
-        printf " %s\n", FILENAME
-    }
-}
-_EOF_
+@TEST-START-FILE find_logs.py
+import os, sys
 
-find -L ${BROSCRIPTS} -type f -exec awk -f find_logid.awk {} \; > out.logid
+scriptdir = sys.argv[1]
 
-if [ ! -s out.logid ]; then
-    echo "Did not find Bro scripts in directory: ${BROSCRIPTS}" 1>&2
-    exit 1
-fi
+# Return a list of all bro script files.
+def find_scripts():
+    scripts = []
 
-# For each log ID, have Bro convert it to the corresponding log filename
-# using the default mechanism for generating a log filename (we must load
-# all Bro scripts so that all log IDs are defined).
-awk '{print $1}' out.logid | while read logid; do
-    bro ${BROSCRIPTS}/test-all-policy.bro -e "print Log::default_path_func(${logid}, \"\", 0);" >> out.tmp
-done
+    for r, d, f in os.walk(scriptdir):
+        for fname in f:
+            if fname.endswith(".bro"):
+                scripts.append(os.path.join(r, fname))
 
-grep -v WARNING out.tmp | sort -u > out
+    return scripts
+
+# For a given script file, return a list of all "Log::create_stream" lines.
+def find_log(fname):
+    f = open(fname, "r")
+
+    lines = []
+    get_semicolon = False
+
+    for line in f:
+        line = line.strip()
+        if not line:
+            continue
+        if line.startswith("#"):
+            continue
+
+        if get_semicolon:
+            lines[-1] += line
+            if line.endswith(";"):
+                get_semicolon = False
+        elif line.startswith("Log::create_stream"):
+            lines.append(line)
+            if not line.endswith(";"):
+                get_semicolon = True
+
+    f.close()
+
+    return lines
+
+for fname in find_scripts():
+    lines = find_log(fname)
+    for line in lines:
+        # Print the value of the "$path" field.
+        idx = line.find("$path")
+        if idx > 0:
+            print("%s" % line[idx:].split('"')[1])
+@TEST-END-FILE