Merge remote branch 'remotes/origin/topic/jsiwek/doc-framework'

* remotes/origin/topic/jsiwek/doc-framework:
  Adding example documentation for a script's use of logging features.
  Adding &log attribute to static attr_names array.
  Small typo fix.
  Bro doc mode now tracks record redefs that extend its field list.
  BroBifDoc was unneeded; now dead code, so removed.
  Bro doc mode now only does a "shallow" copy of declared record types
  Bro's doc mode now terminates after processing bro_init but before net_run
  Fixes related to `make doc` handling of script summary text (##! comments)
  Overhaul of "doc" build target for generating policy script documentation.
  Add parser error hint when in doc mode about checking ## comment syntax.
  Move stuff related to policy script documentation from doc/ to doc/scripts/
  Fixing example.bro's auto-reST generation baseline test.

doc/CMakeLists.txt

@@ -1,42 +1 @@
-set(POLICY_SRC_DIR ${PROJECT_SOURCE_DIR}/policy)
+add_subdirectory(scripts)
-set(DOC_OUTPUT_DIR ${CMAKE_CURRENT_BINARY_DIR}/out)
-set(DOC_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/source)
-set(DOC_SOURCE_WORKDIR ${CMAKE_CURRENT_BINARY_DIR}/source)
-
-file(GLOB_RECURSE DOC_SOURCES FOLLOW_SYMLINKS "*")
-
-configure_file(${CMAKE_CURRENT_SOURCE_DIR}/conf.py.in
-               ${CMAKE_CURRENT_BINARY_DIR}/conf.py
-               @ONLY)
-configure_file(${CMAKE_CURRENT_SOURCE_DIR}/scripts/generate_reST_docs.py.in
-               ${CMAKE_CURRENT_BINARY_DIR}/generate_reST_docs.py
-               @ONLY)
-configure_file(${CMAKE_CURRENT_SOURCE_DIR}/scripts/BroToReST.py.in
-               ${CMAKE_CURRENT_BINARY_DIR}/BroToReST.py
-               @ONLY)
-
-add_custom_target(doc
-                  COMMAND "${CMAKE_COMMAND}" -E copy_directory
-                          ${DOC_SOURCE_DIR}
-                          ${DOC_SOURCE_WORKDIR}
-                  COMMAND python generate_reST_docs.py
-                  COMMAND sphinx-build
-                          -b html
-                          -c ${CMAKE_CURRENT_BINARY_DIR}
-                          -d ${DOC_OUTPUT_DIR}/doctrees
-                          ${DOC_SOURCE_WORKDIR}
-                          ${DOC_OUTPUT_DIR}/html
-                  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
-                  COMMENT "[Sphinx] Generating Script Documentation"
-                  VERBATIM
-                  # SOURCES just adds stuff to IDE projects as a convienience
-                  SOURCES ${DOC_SOURCES})
-
-add_dependencies(doc bro doc-clean)
-
-add_custom_target(doc-clean
-                  COMMAND "${CMAKE_COMMAND}" -E remove_directory
-                          ${CMAKE_CURRENT_BINARY_DIR}/source
-                  COMMAND "${CMAKE_COMMAND}" -E remove_directory
-                          ${DOC_OUTPUT_DIR}
-                  VERBATIM)

doc/README

@@ -1,44 +1 @@
-This directory contains scripts and templates that can be used to automate
+TODO
-the generation of Bro script documentation.  Two build targets are defined
-by CMake:
-
-``make doc``
-
-    This target depends on a Python interpreter (>=2.5) and
-    `Sphinx <http://sphinx.pocoo.org/>`_ being installed.  Sphinx can be
-    installed like::
-
-        > sudo easy_install sphinx
-
-    This target will also first build the bro binary if it is not already
-    since the generation of reStructuredText (reST) documentation from
-    Bro scripts is integrated within the parsing process.
-
-    After completion, HTML documentation can be located inside the CMake
-    ``build/`` directory as ``build/doc/out/html``.  The generated reST
-    documentation will be located in ``build/doc/source/policy``.
-
-``make doc-clean``
-
-    This target removes Sphinx inputs and outputs from the CMake ``build/`` dir.
-
-To schedule a script to be documented, edit ``scripts/generate_reST_docs.py.in``
-and try adding the name of the script along with an optional script group to
-the ``docs`` dictionary.  That python script also shows other, more specialized
-methods for generating documentation for some types of corner-cases.
-
-When adding a new logical grouping for generated scripts, create a new
-reST document in ``source/policy/<group_name>.rst`` and add some default
-documentation.  References to (and summaries of) documents associated with
-the group get appended to this file during the ``make doc`` process.
-
-The Sphinx source tree template in ``source/`` can be modified to add more
-common/general documentation, style sheets, JavaScript, etc.  The Sphinx
-config file is produced from ``conf.py.in``, so that can be edited to change
-various Sphinx options, like setting the default HTML rendering theme.
-There is also a custom Sphinx domain implemented in ``source/ext/bro.py``
-which adds some reST directives and roles that aid in generating useful
-index entries and cross-references.
-
-See ``example.bro`` for an example of how to document a Bro script such that
-``make doc`` will be able to produce reST/HTML documentation for it.

doc/scripts/BroToReST.py.in

@@ -1,152 +0,0 @@
-#! /usr/bin/env python
-
-import os
-import subprocess
-import shutil
-import glob
-import string
-import sys
-
-BRO = "@CMAKE_BINARY_DIR@/src/bro"
-BROPATHDEV = "`@CMAKE_BINARY_DIR@/bro-path-dev`"
-BRO_ARGS = "--doc-scripts"
-DOC_DST_DIR = "@DOC_SOURCE_WORKDIR@/policy"
-BROPATH = subprocess.Popen("@CMAKE_BINARY_DIR@/bro-path-dev", shell=True,
-        stdout=subprocess.PIPE, stderr=subprocess.STDOUT).stdout.readline()
-
-class BroToReST:
-    """A class to encapsulate the the generation of reST documentation from
-    a given Bro script.
-    """
-
-    bro_src_file = None
-    doc_src_file = None
-    load_via_stdin = False
-    group = None
-
-    def __init__(self, src_file, load_method=False, search_dir=None, group=None):
-        """
-        :param src_file: the file name of a Bro script (not a path)
-        :param load_method: T if script must be loaded by Bro via a stdin
-            redirection of "@load <script>", F if script can be loaded as
-            a command line argument to Bro
-        :param search_dir: a list of directories in which to search for
-            src_file.  If None, the default BROPATH is used.
-        :param group: a string representing a logical group that the script's
-            documentation should belong to.  A corresponding <group>.rst
-            document must be pre-existing in the policy/ dir of the source tree
-            used by Sphinx.
-        """
-
-        self.bro_src_file = FindBroScript(src_file, search_dir)
-        self.load_via_stdin = load_method
-        self.group = group
-
-        # formulate doc_src_file from src_file
-        filename = os.path.basename(src_file)
-        basename, ext = os.path.splitext(filename)
-        if ext == ".bro":
-            self.doc_src_file = basename + ".rst"
-        else:
-            self.doc_src_file = filename + ".rst"
-
-    def __str__(self):
-        return "bro_src_file: " + self.bro_src_file \
-           + "\ndoc_src_file: " + self.doc_src_file \
-           + "\ndoc_dst_file: " + os.path.join(DOC_DST_DIR, self.doc_src_file) \
-           + "\nstdin_load: %s" % self.load_via_stdin \
-           + "\ngroup: %s" % self.group
-
-    def GenDoc(self):
-        """Generates the reST documentation for a Bro script and copies
-        both the documentation and original script into Sphinx source tree.
-        If the documentation belongs to a group, the necessary modifications
-        to add it to the group's documentation are done.  Afterwards, any
-        files with a ".rst" suffix are removed for the working directory.
-        """
-
-        bro_src_basename = os.path.basename(self.bro_src_file)
-
-        if self.load_via_stdin:
-            cmd = "echo '@load %s' | %s %s" % (bro_src_basename, BRO, BRO_ARGS)
-        else:
-            cmd = "%s %s %s" % (BRO, BRO_ARGS, self.bro_src_file)
-
-        p = subprocess.Popen(cmd, shell=True, env={"BROPATH": BROPATH})
-
-        if p.wait() == 0:
-            shutil.copy(self.doc_src_file, DOC_DST_DIR)
-            shutil.copy(self.bro_src_file, DOC_DST_DIR)
-            AppendToDocGroup(self.group, self.bro_src_file, self.doc_src_file)
-
-        for leftover in glob.glob("*.rst"):
-            os.remove(leftover)
-
-def GenDocs(doc_dict, load_method=False):
-    """Generates reST documentation for all scripts in the given dictionary.
-
-    :param doc_dict: a dictionary whose keys are file names of Bro scripts
-        (not paths), and whose value is the logical documentation group
-        it belongs to
-    :param load_method: T if script must be loaded by Bro via a stdin
-            redirection of "@load <script>", F if script can be loaded as
-            a command line argument to Bro
-    """
-
-    for k, v in doc_dict.iteritems():
-        doc = BroToReST(k, load_method, group=v)
-        print "Generating reST document for " + k
-        doc.GenDoc()
-
-def FindBroScript(src_file, search_dir=None):
-    """Search a set of paths for a given Bro script and return the absolute
-    path to it.
-
-    :param src_file: the file name of a Bro script (not a path)
-    :param search_dir: a list of directories in which to search for
-        src_file.  If None, the default BROPATH is used.
-    """
-    if search_dir is None:
-        search_dir = string.split(BROPATH, ":")
-    for path in search_dir:
-        abs_path = os.path.join(path, src_file)
-        if os.path.exists(abs_path):
-            return abs_path
-    print >> sys.stderr, "Couldn't find '%s'" % src_file
-    return None
-
-def AppendToDocGroup(group, src_file, doc_file):
-    """Adds a reference to the given documentation for a Bro script
-    to the documentation file for it's associated group.  Also, associated
-    summary text (comments marked up like "##!" in the original Bro script
-    source) are added.
-
-    :param group: a string representing a logical group that the script's
-        documentation should belong to.  A corresponding <group>.rst
-        document must be pre-existing in the policy/ dir of the source tree
-        used by Sphinx.
-    :param src_file: a path to the original Bro script source file
-    :param doc_file: the file name of a script's generated reST document
-    """
-    if group is None:
-        return
-
-    group_file = os.path.join(DOC_DST_DIR, group + ".rst")
-    if not os.path.exists(group_file):
-        print >> sys.stderr, "Group file doesn't exist: " + group_file
-        return
-
-    summary_comments = []
-
-    with open(src_file, 'r') as f:
-        for line in f:
-            sum_pos = string.find(line, "##!")
-            if sum_pos != -1:
-                summary_comments.append(line[(sum_pos+3):])
-
-    doc_name, ext = os.path.splitext(doc_file)
-
-    with open(group_file, 'a') as f:
-        f.write("\n:doc:`%s`\n" % doc_name)
-        for line in summary_comments:
-            f.write(line)

doc/scripts/CMakeLists.txt

@@ -0,0 +1,263 @@
+set(POLICY_SRC_DIR ${PROJECT_SOURCE_DIR}/policy)
+set(BIF_SRC_DIR ${PROJECT_SOURCE_DIR}/src)
+set(RST_OUTPUT_DIR ${CMAKE_CURRENT_BINARY_DIR}/rest_output)
+set(DOC_OUTPUT_DIR ${CMAKE_CURRENT_BINARY_DIR}/out)
+set(DOC_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/source)
+set(DOC_SOURCE_WORKDIR ${CMAKE_CURRENT_BINARY_DIR}/source)
+
+file(GLOB_RECURSE DOC_SOURCES FOLLOW_SYMLINKS "*")
+
+# configure the Sphinx config file (expand variables CMake might know about)
+configure_file(${CMAKE_CURRENT_SOURCE_DIR}/conf.py.in
+               ${CMAKE_CURRENT_BINARY_DIR}/conf.py
+               @ONLY)
+
+# find out what BROPATH to use when executing bro
+execute_process(COMMAND ${CMAKE_BINARY_DIR}/bro-path-dev
+                OUTPUT_VARIABLE BROPATH
+                RESULT_VARIABLE retval
+                OUTPUT_STRIP_TRAILING_WHITESPACE)
+if (NOT ${retval} EQUAL 0)
+    message(FATAL_ERROR "Problem setting BROPATH")
+endif ()
+
+# This macro is used to add a new makefile target for reST policy script
+# documentation that can be generated using Bro itself to parse policy scripts.
+# It's called like:
+#
+#     rest_target(srcDir broInput [group])
+#
+# srcDir: the directory which contains broInput
+# broInput: the file name of a bro policy script
+# group: optional name of group that the script documentation will belong to
+#
+# In addition to adding the makefile target, several CMake variables are set:
+#
+# MASTER_POLICY_INDEX_TEXT: a running list of policy scripts docs that have
+#   been generated so far, formatted such that it can be appended to a file
+#   that ends in a Sphinx toctree directive
+# ALL_REST_OUTPUTS: a running list (the CMake list type) of all reST docs
+#   that are to be generated
+# MASTER_GROUP_LIST: a running list (the CMake list type) of all script groups
+# ${group}_files: a running list of files belonging to a given group, from
+#   which summary text can be extracted at build time
+# ${group}_doc_names: a running list of reST style document names that can be
+#   given to a :doc: role, shared indices with ${group}_files
+#
+macro(REST_TARGET srcDir broInput)
+    get_filename_component(basename ${broInput} NAME_WE)
+    get_filename_component(extension ${broInput} EXT)
+    get_filename_component(relDstDir ${broInput} PATH)
+
+    set(sumTextSrc ${srcDir}/${broInput})
+    if (${extension} STREQUAL ".bif.bro")
+        set(basename "${basename}.bif")
+        # the summary text is taken at configure time, but .bif.bro files
+        # may not have been generated yet, so read .bif file instead
+        set(sumTextSrc ${BIF_SRC_DIR}/${basename})
+    elseif (${extension} STREQUAL ".init")
+        set(basename "${basename}.init")
+    endif ()
+
+    set (restFile "${basename}.rst")
+
+    if (NOT relDstDir)
+        set(docName "${basename}")
+        set(dstDir "${RST_OUTPUT_DIR}")
+    else ()
+        set(docName "${relDstDir}/${basename}")
+        set(dstDir "${RST_OUTPUT_DIR}/${relDstDir}")
+    endif ()
+
+    set(restOutput "${dstDir}/${restFile}")
+
+    set(indexEntry "   ${docName} <${docName}>")
+    set(MASTER_POLICY_INDEX_TEXT "${MASTER_POLICY_INDEX_TEXT}\n${indexEntry}")
+    list(APPEND ALL_REST_OUTPUTS ${restOutput})
+
+    if (NOT "${ARGN}" STREQUAL "")
+        set(group ${ARGN})
+        # add group to master group list if not already in it
+        list(FIND MASTER_GROUP_LIST ${group} _found)
+        if (_found EQUAL -1)
+            list(APPEND MASTER_GROUP_LIST ${group})
+            if (MASTER_GROUP_LIST_TEXT)
+               set(MASTER_GROUP_LIST_TEXT "${MASTER_GROUP_LIST_TEXT}\n${group}")
+            else ()
+               set(MASTER_GROUP_LIST_TEXT "${group}")
+            endif ()
+        endif ()
+
+        list(APPEND ${group}_files ${sumTextSrc})
+        list(APPEND ${group}_doc_names ${docName})
+    else ()
+        set(group "")
+    endif ()
+
+    if (${group} STREQUAL "default" OR ${group} STREQUAL "bifs")
+        set(BRO_ARGS --doc-scripts --exec '')
+    else ()
+        set(BRO_ARGS --doc-scripts ${srcDir}/${broInput})
+    endif ()
+
+    add_custom_command(OUTPUT ${restOutput}
+        # delete any leftover state from previous bro runs
+        COMMAND "${CMAKE_COMMAND}"
+        ARGS -E remove_directory .state
+        # generate the reST documentation using bro
+        COMMAND BROPATH=${BROPATH} ${CMAKE_BINARY_DIR}/src/bro
+        ARGS ${BRO_ARGS} || (rm -rf .state *.log *.rst && exit 1)
+        # move generated doc into a new directory tree that
+        # defines the final structure of documents
+        COMMAND "${CMAKE_COMMAND}"
+        ARGS -E make_directory ${dstDir}
+        COMMAND "${CMAKE_COMMAND}"
+        ARGS -E copy ${restFile} ${restOutput}
+        # copy the bro policy script, too
+        COMMAND "${CMAKE_COMMAND}"
+        ARGS -E copy ${srcDir}/${broInput} ${dstDir}
+        # clean up the build directory
+        COMMAND rm
+        ARGS -rf .state *.log *.rst
+        DEPENDS bro
+        DEPENDS ${srcDir}/${broInput}
+        COMMENT "[Bro] Generating reST docs for ${broInput}"
+    )
+
+endmacro(REST_TARGET)
+
+# Schedule Bro scripts for which to generate documentation.
+# Note: the script may be located in a subdirectory off of one of the main
+# directories in BROPATH.  In that case, just list the script as 'foo/bar.bro'
+rest_target(${POLICY_SRC_DIR} alarm.bro                         user)
+rest_target(${POLICY_SRC_DIR} arp.bro                           user)
+rest_target(${POLICY_SRC_DIR} conn.bro                          user)
+rest_target(${POLICY_SRC_DIR} dhcp.bro                          user)
+rest_target(${POLICY_SRC_DIR} dns.bro                           user)
+rest_target(${POLICY_SRC_DIR} ftp.bro                           user)
+rest_target(${POLICY_SRC_DIR} http.bro                          user)
+rest_target(${POLICY_SRC_DIR} http-reply.bro                    user)
+rest_target(${POLICY_SRC_DIR} http-request.bro                  user)
+rest_target(${POLICY_SRC_DIR} irc.bro                           user)
+rest_target(${POLICY_SRC_DIR} smtp.bro                          user)
+rest_target(${POLICY_SRC_DIR} ssl.bro                           user)
+rest_target(${POLICY_SRC_DIR} ssl-ciphers.bro                   user)
+rest_target(${POLICY_SRC_DIR} ssl-errors.bro                    user)
+rest_target(${POLICY_SRC_DIR} synflood.bro                      user)
+rest_target(${POLICY_SRC_DIR} tcp.bro                           user)
+rest_target(${POLICY_SRC_DIR} udp.bro                           user)
+rest_target(${POLICY_SRC_DIR} weird.bro                         user)
+rest_target(${CMAKE_CURRENT_SOURCE_DIR} example.bro             internal)
+
+# Finding out what scripts bro will generate documentation for by default
+# can be done like: `bro --doc-scripts --exec ""`
+rest_target(${POLICY_SRC_DIR} bro.init                          default)
+rest_target(${POLICY_SRC_DIR} logging-ascii.bro                 default)
+rest_target(${POLICY_SRC_DIR} logging.bro                       default)
+rest_target(${POLICY_SRC_DIR} pcap.bro                          default)
+rest_target(${POLICY_SRC_DIR} server-ports.bro                  default)
+rest_target(${CMAKE_BINARY_DIR}/src bro.bif.bro                 bifs)
+rest_target(${CMAKE_BINARY_DIR}/src const.bif.bro               bifs)
+rest_target(${CMAKE_BINARY_DIR}/src event.bif.bro               bifs)
+rest_target(${CMAKE_BINARY_DIR}/src logging.bif.bro             bifs)
+rest_target(${CMAKE_BINARY_DIR}/src strings.bif.bro             bifs)
+rest_target(${CMAKE_BINARY_DIR}/src types.bif.bro               bifs)
+
+# create temporary list of all docs to include in the master policy/index file
+file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/tmp_policy_index
+    "${MASTER_POLICY_INDEX_TEXT}")
+
+# create temporary file containing list of all groups
+file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/group_list
+    "${MASTER_GROUP_LIST_TEXT}")
+
+# create temporary files containing list of each source file in a given group
+foreach (group ${MASTER_GROUP_LIST})
+    if (EXISTS ${CMAKE_CURRENT_BINARY_DIR}/${group}_files)
+        file(REMOVE ${CMAKE_CURRENT_BINARY_DIR}/${group}_files)
+    endif ()
+    if (EXISTS ${CMAKE_CURRENT_BINARY_DIR}/${group}_doc_names)
+        file(REMOVE ${CMAKE_CURRENT_BINARY_DIR}/${group}_doc_names)
+    endif ()
+    foreach (src ${${group}_files})
+        file(APPEND ${CMAKE_CURRENT_BINARY_DIR}/${group}_files "${src}\n")
+    endforeach ()
+    foreach (dname ${${group}_doc_names})
+        file(APPEND ${CMAKE_CURRENT_BINARY_DIR}/${group}_doc_names "${dname}\n")
+    endforeach ()
+endforeach ()
+
+# remove previously generated docs no longer scheduled for generation
+if (EXISTS ${RST_OUTPUT_DIR})
+    file(GLOB_RECURSE EXISTING_REST_DOCS "${RST_OUTPUT_DIR}/*.rst")
+    foreach (_doc ${EXISTING_REST_DOCS})
+        list(FIND ALL_REST_OUTPUTS ${_doc} _found)
+        if (_found EQUAL -1)
+            file(REMOVE ${_doc})
+            message(STATUS "Removing stale reST doc: ${_doc}")
+        endif ()
+    endforeach ()
+endif ()
+
+# The "restdoc" target uses Bro to parse policy scripts in order to
+# generate reST documentation from them.
+add_custom_target(restdoc
+                  # create symlink to the reST output directory for convenience
+                  COMMAND "${CMAKE_COMMAND}" -E create_symlink
+                          ${RST_OUTPUT_DIR}
+                          ${CMAKE_BINARY_DIR}/reST
+                  DEPENDS ${ALL_REST_OUTPUTS})
+
+# The "restclean" target removes all generated reST documentation from the
+# build directory.
+add_custom_target(restclean
+                  COMMAND "${CMAKE_COMMAND}" -E remove_directory
+                          ${RST_OUTPUT_DIR}
+                  VERBATIM)
+
+# The "doc" target generates reST documentation for any outdated bro scripts
+# and then uses Sphinx to generate HTML documentation from the reST
+add_custom_target(doc
+                  # copy the template documentation to the build directory
+                  # to give as input for sphinx
+                  COMMAND "${CMAKE_COMMAND}" -E copy_directory
+                          ${DOC_SOURCE_DIR}
+                          ${DOC_SOURCE_WORKDIR}
+                  # copy generated policy script documentation into the
+                  # working copy of the template documentation
+                  COMMAND "${CMAKE_COMMAND}" -E copy_directory
+                          ${RST_OUTPUT_DIR}
+                          ${DOC_SOURCE_WORKDIR}/policy
+                  # append to the master index of all policy scripts
+                  COMMAND cat ${CMAKE_CURRENT_BINARY_DIR}/tmp_policy_index >>
+                          ${DOC_SOURCE_WORKDIR}/policy/index.rst
+                  # construct a reST file for each group
+                  COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/group_index_generator.py
+                          ${CMAKE_CURRENT_BINARY_DIR}/group_list
+                          ${CMAKE_CURRENT_BINARY_DIR}
+                          ${DOC_SOURCE_WORKDIR}
+                  # tell sphinx to generate html
+                  COMMAND sphinx-build
+                          -b html
+                          -c ${CMAKE_CURRENT_BINARY_DIR}
+                          -d ${DOC_OUTPUT_DIR}/doctrees
+                          ${DOC_SOURCE_WORKDIR}
+                          ${DOC_OUTPUT_DIR}/html
+                  # create symlink to the html output directory for convenience
+                  COMMAND "${CMAKE_COMMAND}" -E create_symlink
+                          ${DOC_OUTPUT_DIR}/html
+                          ${CMAKE_BINARY_DIR}/html
+                  WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
+                  COMMENT "[Sphinx] Generating HTML policy script docs"
+                  # SOURCES just adds stuff to IDE projects as a convienience
+                  SOURCES ${DOC_SOURCES}
+                  DEPENDS restdoc docclean)
+
+# The "docclean" target removes just the Sphinx input/output directories
+# from the build directory.
+add_custom_target(docclean
+                  COMMAND "${CMAKE_COMMAND}" -E remove_directory
+                          ${DOC_SOURCE_WORKDIR}
+                  COMMAND "${CMAKE_COMMAND}" -E remove_directory
+                          ${DOC_OUTPUT_DIR}
+                  VERBATIM)

doc/scripts/README

@@ -0,0 +1,61 @@
+This directory contains scripts and templates that can be used to automate
+the generation of Bro script documentation.  Several build targets are defined
+by CMake:
+
+``restdoc``
+
+    This target uses Bro to parse policy scripts in order to generate
+    reStructuredText (reST) documentation from them.  The list of scripts
+    for which to generate reST documentation is defined in the
+    ``CMakeLists.txt`` file in this directory.  Script documentation is
+    rebuild automatically if the policy script from which it is derived
+    or the Bro binary becomes out of date
+
+    The resulting output from this target can be found in the CMake
+    ``build/`` directory inside ``reST`` (a symlink to
+    ``doc/scripts/rest_output``).
+
+``doc``
+
+    This target depends on a Python interpreter (>=2.5) and
+    `Sphinx <http://sphinx.pocoo.org/>`_ being installed.  Sphinx can be
+    installed like::
+
+        > sudo easy_install sphinx
+
+    This target will first build ``restdoc`` target and then copy the
+    resulting reST files as an input directory to Sphinx.
+
+    After completion, HTML documentation can be located in the CMake
+    ``build/`` directory inside ``html`` (a symlink to
+    ``doc/scripts/out/html``)
+
+``restclean``
+
+    This target removes any reST documentation that has been generated so far.
+
+``docclean``
+
+    This target removes Sphinx inputs and outputs from the CMake ``build/`` dir.
+
+To schedule a script to be documented, edit ``CMakeLists.txt`` inside this
+directory add a call to the ``rest_target()`` macro.  Calling that macro
+with a group name for the script is optional, but if not given, the only
+link to the script will be in the master TOC tree for all policy scripts.
+
+When adding a new logical grouping for generated scripts, create a new
+reST document in ``source/<group_name>.rst`` and add some default
+documentation for the group.  References to (and summaries of) documents
+associated with the group get appended to this file during the
+``make doc`` process.
+
+The Sphinx source tree template in ``source/`` can be modified to add more
+common/general documentation, style sheets, JavaScript, etc.  The Sphinx
+config file is produced from ``conf.py.in``, so that can be edited to change
+various Sphinx options, like setting the default HTML rendering theme.
+There is also a custom Sphinx domain implemented in ``source/ext/bro.py``
+which adds some reST directives and roles that aid in generating useful
+index entries and cross-references.
+
+See ``example.bro`` for an example of how to document a Bro script such that
+``make doc`` will be able to produce reST/HTML documentation for it.

doc/scripts/example.bro

@@ -6,6 +6,18 @@
 ##!
 ##! .. tip:: You can embed directives and roles within ``##``-stylized comments.
 ##!
+##! A script's logging information has to be documented manually as minimally
+##! shown below.  Note that references may not always be possible (e.g.
+##! anonymous filter functions) and a script may not need to document
+##! each of "columns", "event", "filter" depending on exactly what it's doing.
+##!
+##! **Logging Stream ID:** :bro:enum:`Example::EXAMPLE`
+##!     :Columns:    :bro:type:`Example::Info`
+##!     :Event:      :bro:id:`Example::log_example`
+##!     :Filter:     ``example-filter``
+##!         uses :bro:id:`Example::filter_func` to determine whether to
+##!         exclude the ``ts`` field
+##!
 ##! :Author: Jon Siwek <jsiwek@ncsa.illinois.edu>
 
 # Comments that use a single pound sign (#) are not significant to
@@ -64,6 +76,12 @@ redef enum Notice += {
     Notice_Four,
 };
 
+# Redef'ing the ID enumeration for logging streams is automatically tracked.
+# Comments of the "##" form can be use to further document it, but it's
+# better to do all documentation related to logging in the summary section
+# as is shown above.
+redef enum Log::ID += { EXAMPLE };
+
 # Anything declared in the export section will show up in the rendered
 # documentation's "public interface" section
 
@@ -107,6 +125,11 @@ export {
         field2: bool; ##< toggles something
     };
 
+    ## document the record extension redef here
+    redef record SimpleRecord += {
+        ## document the extending field here
+        field_ext: string &optional; ##< (or here)
+    };
 
     ## general documentation for a type "ComplexRecord" goes here
     type ComplexRecord: record {
@@ -116,6 +139,13 @@ export {
         msg: string &default="blah"; ##< attributes are self-documenting
     } &redef;
 
+    ## An example record to be used with a logging stream.
+    type Info: record {
+        ts:       time       &log;
+        uid:      string     &log;
+        status:   count      &log &optional;
+    };
+
     ############## options ################
     # right now, I'm just defining an option as
     # any const with &redef (something that can
@@ -159,6 +189,15 @@ export {
     ## Give more details about "an_event" here.
     ## name: describe the argument here
     global an_event: event(name: string);
+
+    ## This is a declaration of an example event that can be used in
+    ## logging streams and is raised once for each log entry.
+    global log_example: event(rec: Info);
+}
+
+function filter_func(rec: Info): bool
+    {
+    return T;
     }
 
 # this function is documented in the "private interface" section
@@ -176,3 +215,14 @@ type PrivateRecord: record {
     field1: bool;
     field2: count;
 };
+
+event bro_init()
+    {
+    Log::create_stream(EXAMPLE, [$columns=Info, $ev=log_example]);
+    Log::add_filter(EXAMPLE, [
+        $name="example-filter",
+        $path="example-filter",
+        $pred=filter_func,
+        $exclude=set("ts")
+        ]);
+    }

doc/scripts/generate_reST_docs.py.in

@@ -1,71 +0,0 @@
-#! /usr/bin/env python
-
-import os
-import subprocess
-import shutil
-import glob
-import string
-import sys
-from BroToReST import *
-
-# TODO: generate docs for more scripts
-# TODO: the groups are just made up to test the functionality, fix them
-
-# Scripts that can be loaded by bro via command line argument:
-docs = {
-    "alarm.bro":        "internal",
-    "arp.bro":          "user",
-    "conn.bro":         "internal",
-    "dhcp.bro":         "user",
-    "dns.bro":          "user",
-    "ftp.bro":          "user",
-    "http.bro":         "user",
-    "http-reply.bro":   None,
-    "http-request.bro": None,
-    "irc.bro":          "user",
-    "smtp.bro":         "user",
-    "ssl.bro":          "user",
-    "ssl-ciphers.bro":  None,
-    "ssl-errors.bro":   None,
-    "synflood.bro":     "user",
-    "tcp.bro":          "user",
-    "udp.bro":          "user",
-    "weird.bro":        "internal",
-}
-
-# Scripts that can't be loaded by bro via command line argument (possible
-# due to dependency issues), but can be loaded via an @load on stdin:
-stdin_docs = {
-    "notice.bro": "internal",
-}
-
-GenDocs(docs)
-GenDocs(stdin_docs, True)
-
-# The example documentation script doesn't live on the BROPATH, so
-# explicitly generate the docs for it like this:
-BroToReST("example.bro", False, ["@PROJECT_SOURCE_DIR@/doc"], group="internal").GenDoc()
-
-# Generate documentation for stuff that's always loaded into bro by default:
-cmd = "echo '' | %s %s" % (BRO, BRO_ARGS)
-p = subprocess.Popen(cmd, shell=True, env={"BROPATH": BROPATH})
-if p.wait() == 0:
-    for doc in glob.glob("*.rst"):
-        if doc == "<stdin>.rst":
-            os.remove(doc)
-            continue
-
-        basename, ext = os.path.splitext(doc)
-        basename2, ext = os.path.splitext(basename)
-        if ext == ".init":
-            src_file = basename
-        else:
-            src_file = basename + ".bro"
-        src_file = FindBroScript(src_file)
-        shutil.copy(src_file, DOC_DST_DIR)
-        shutil.copy(doc, DOC_DST_DIR)
-        if ext == ".bif":
-            AppendToDocGroup("bifs", src_file, doc)
-        else:
-            AppendToDocGroup("default", src_file, doc)
-        os.remove(doc)

doc/scripts/group_index_generator.py

@@ -0,0 +1,52 @@
+#! /usr/bin/env python
+
+# This script automatically generates a reST documents that lists
+# a collection of Bro policy scripts that are "grouped" together.
+# The summary text (##! comments) of the policy script is embedded
+# in the list.
+#
+# 1st argument is the file containing list of groups
+# 2nd argument is the directory containing ${group}_files lists of
+#   scripts that belong to the group and ${group}_doc_names lists of
+#   document names that can be supplied to a reST :doc: role
+# 3rd argument is a directory in which write a ${group}.rst file (will
+#   append to existing file) that contains reST style references to
+#   script docs along with summary text contained in original script
+
+import sys
+import os
+import string
+
+group_list = sys.argv[1]
+file_manifest_dir = sys.argv[2]
+output_dir = sys.argv[3]
+
+with open(group_list, 'r') as f_group_list:
+    for group in f_group_list.read().splitlines():
+        #print group
+        file_manifest = os.path.join(file_manifest_dir, group + "_files")
+        doc_manifest = os.path.join(file_manifest_dir, group + "_doc_names")
+        src_files = []
+        doc_names = []
+
+        with open(file_manifest, 'r') as f_file_manifest:
+            src_files = f_file_manifest.read().splitlines()
+
+        with open(doc_manifest, 'r') as f_doc_manifest:
+            doc_names = f_doc_manifest.read().splitlines()
+
+        for i in range(len(src_files)):
+            src_file = src_files[i]
+            #print "\t" + src_file
+            summary_comments = []
+            with open(src_file, 'r') as f_src_file:
+                for line in f_src_file:
+                    sum_pos = string.find(line, "##!")
+                    if sum_pos != -1:
+                        summary_comments.append(line[(sum_pos+3):])
+            #print summary_comments
+            group_file = os.path.join(output_dir, group + ".rst")
+            with open(group_file, 'a') as f_group_file:
+                f_group_file.write("\n:doc:`/policy/%s`\n" % doc_names[i])
+                for line in summary_comments:
+                    f_group_file.write("   " + line)

doc/scripts/source/builtins.rst

@@ -118,4 +118,6 @@ The Bro scripting language supports the following built-in attributes.
 
 .. bro:attr:: &group
 
+.. bro:attr:: &log
+
 .. bro:attr:: (&tracked)

doc/scripts/source/index.rst

@@ -11,11 +11,11 @@ Contents:
 
    common
    builtins
-   policy/default
-   policy/user
-   policy/bifs
-   policy/internal
    policy/index
+   default
+   bifs
+   user
+   internal
 
 Indices and tables
 ==================

doc/scripts/source/policy/index.rst

@@ -1,10 +1,6 @@
 Index of All Policy Script Documentation
 ========================================
 
-Contents:
-
 .. toctree::
    :maxdepth: 1
-   :glob:
 
-   *

src/Attr.cc

@@ -19,7 +19,7 @@ const char* attr_name(attr_tag t)
 		"&persistent", "&synchronized", "&postprocessor",
 		"&encrypt", "&match", "&disable_print_hook",
 		"&raw_output", "&mergeable", "&priority",
-		"&group", "(&tracked)",
+		"&group", "&log", "(&tracked)",
 	};
 
 	return attr_names[int(t)];

src/BroBifDoc.cc

@@ -1,16 +0,0 @@
-#include <cstdio>
-#include <string>
-#include <list>
-
-#include "BroDoc.h"
-#include "BroBifDoc.h"
-
-BroBifDoc::BroBifDoc(const std::string& sourcename) : BroDoc(sourcename)
-	{
-	}
-
-// TODO: This needs to do something different than parent class's version.
-void BroBifDoc::WriteDocFile() const
-	{
-	BroDoc::WriteDocFile();
-	}

src/BroBifDoc.h

@@ -1,18 +0,0 @@
-#ifndef brobifdoc_h
-#define brobifdoc_h
-
-#include <cstdio>
-#include <string>
-#include <list>
-
-#include "BroDoc.h"
-
-class BroBifDoc : public BroDoc {
-public:
-	BroBifDoc(const std::string& sourcename);
-	virtual ~BroBifDoc()	{ }
-
-	void WriteDocFile() const;
-};
-
-#endif

src/BroDoc.cc

@@ -58,15 +58,8 @@ void BroDoc::AddImport(const std::string& s)
 
 	if ( ext_pos == std::string::npos )
 		imports.push_back(s);
-
 	else
-		{
-		if ( s.substr(ext_pos + 1) == "bro" )
 		imports.push_back(s.substr(0, ext_pos));
-		else
-			fprintf(stderr, "Warning: skipped documenting @load of file "
-			                "without .bro extension: %s\n", s.c_str());
-		}
 	}
 
 void BroDoc::SetPacketFilter(const std::string& s)
@@ -116,7 +109,15 @@ void BroDoc::WriteDocFile() const
 	if ( ! imports.empty() )
 		{
 		WriteToDoc(":Imports: ");
-		WriteStringList(":doc:`%s`, ", ":doc:`%s`\n", imports);
+		std::list<std::string>::const_iterator it;
+		for ( it = imports.begin(); it != imports.end(); ++it )
+			{
+			if ( it != imports.begin() )
+				WriteToDoc(", ");
+
+			WriteToDoc(":doc:`%s </policy/%s>`", it->c_str(), it->c_str());
+			}
+		WriteToDoc("\n");
 		}
 
 	WriteToDoc("\n");

src/CMakeLists.txt

@@ -286,7 +286,6 @@ set(bro_SRCS
     BitTorrent.cc
     BitTorrentTracker.cc
     BPF_Program.cc
-    BroBifDoc.cc
     BroDoc.cc
     BroDocObj.cc
     BroString.cc

src/Type.h

@@ -456,6 +456,8 @@ public:
 	// Given an offset, returns the field's name.
 	const char* FieldName(int field) const;
 
+	type_decl_list* Types() { return types; }
+
 	// Given an offset, returns the field's TypeDecl.
 	const TypeDecl* FieldDecl(int field) const;
 	TypeDecl* FieldDecl(int field);

src/Var.cc

@@ -241,6 +241,15 @@ void add_type(ID* id, BroType* t, attr_list* attr, int /* is_event */)
 	// is to add an ID* to class ID that tracks aliases and set it here if
 	// t->GetTypeID() is true.
 	if ( generate_documentation )
+		{
+		if ( t->Tag() == TYPE_RECORD )
+			{
+			// Only "shallow" copy record types because we want to be able
+			// to see additions to the original type's list of fields
+			tnew = new RecordType(t->AsRecordType()->Types());
+			}
+
+		else
 			{
 			SerializationFormat* form = new BinarySerializationFormat();
 			form->StartWrite();
@@ -258,6 +267,7 @@ void add_type(ID* id, BroType* t, attr_list* attr, int /* is_event */)
 			tnew = t->Unserialize(&uinfo);
 
 			delete [] data;
+			}
 
 		tnew->SetTypeID(copy_string(id->Name()));
 		}

src/builtin-func.y

@@ -319,8 +319,8 @@ definitions:	definitions definition opt_ws
 			fprintf(fp_netvar_h, "// %s\n\n", auto_gen_comment);
 			fprintf(fp_netvar_init, "// %s\n\n", auto_gen_comment);
 
+			fprintf(fp_bro_init, "%s", $1);
 			fprintf(fp_bro_init, "export {\n");
-			fprintf(fp_func_def, "%s", $1);
 			}
 	;
 

src/main.cc

@@ -9,6 +9,7 @@
 #include <unistd.h>
 #include <signal.h>
 #include <string.h>
+#include <list>
 #ifdef HAVE_GETOPT_H
 #include <getopt.h>
 #endif
@@ -47,6 +48,7 @@ extern "C" void OPENSSL_add_all_algorithms_conf(void);
 #include "Stats.h"
 #include "ConnCompressor.h"
 #include "DPM.h"
+#include "BroDoc.h"
 
 #include "binpac_bro.h"
 
@@ -103,6 +105,8 @@ char* proc_status_file = 0;
 
 int FLAGS_use_binpac = false;
 
+extern std::list<BroDoc*> docs_generated;
+
 // Keep copy of command line
 int bro_argc;
 char** bro_argv;
@@ -971,6 +975,20 @@ int main(int argc, char** argv)
 
 	mgr.Drain();
 
+	if ( generate_documentation )
+		{
+		std::list<BroDoc*>::iterator it;
+
+		for ( it = docs_generated.begin(); it != docs_generated.end(); ++it )
+			(*it)->WriteDocFile();
+
+		for ( it = docs_generated.begin(); it != docs_generated.end(); ++it )
+			delete *it;
+
+		terminate_bro();
+		return 0;
+		}
+
 	have_pending_timers = ! reading_traces && timer_mgr->Size() > 0;
 
 	if ( io_sources.Size() > 0 || have_pending_timers )

src/parse.y

@@ -1067,8 +1067,10 @@ decl:
 			}
 
 	|	TOK_REDEF TOK_RECORD global_id TOK_ADD_TO
-			'{' type_decl_list '}' opt_attr ';'
+			'{' { do_doc_token_start(); } type_decl_list '}' opt_attr ';'
 			{
+			do_doc_token_stop();
+
 			if ( ! $3->Type() )
 				$3->Error("unknown identifier");
 			else
@@ -1078,9 +1080,27 @@ decl:
 					$3->Error("not a record type");
 				else
 					{
-					const char* error = add_to->AddFields($6, $8);
+					const char* error = add_to->AddFields($7, $9);
 					if ( error )
 						$3->Error(error);
+					else if ( generate_documentation )
+						{
+						if ( fake_type_decl_list )
+							{
+							BroType* fake_record =
+								new RecordType(fake_type_decl_list);
+							ID* fake = create_dummy_id($3, fake_record);
+							fake_type_decl_list = 0;
+							current_reST_doc->AddRedef(
+								new BroDocObj(fake, reST_doc_comments, true));
+							}
+						else
+							{
+							fprintf(stderr, "Warning: doc mode did not process "
+								"record extension for '%s', CommentedTypeDecl"
+								"list unavailable.\n", $3->Name());
+							}
+						}
 					}
 				}
 			}
@@ -1622,7 +1642,7 @@ opt_doc_list:
 
 int yyerror(const char msg[])
 	{
-	char* msgbuf = new char[strlen(msg) + strlen(last_tok) + 64];
+	char* msgbuf = new char[strlen(msg) + strlen(last_tok) + 128];
 
 	if ( last_tok[0] == '\n' )
 		sprintf(msgbuf, "%s, on previous line", msg);
@@ -1631,6 +1651,10 @@ int yyerror(const char msg[])
 	else
 		sprintf(msgbuf, "%s, at or near \"%s\"", msg, last_tok);
 
+	if ( generate_documentation )
+		strcat(msgbuf, "\nDocumentation mode is enabled: "
+		       "remember to check syntax of ## style comments\n");
+
 	error(msgbuf);
 
 	return 0;

src/scan.l

@@ -16,7 +16,6 @@
 #include "PolicyFile.h"
 #include "broparse.h"
 #include "BroDoc.h"
-#include "BroBifDoc.h"
 #include "Analyzer.h"
 #include "AnalyzerTags.h"
 
@@ -58,7 +57,7 @@ char last_tok[128];
 static PList(char) files_scanned;
 
 // reST documents that we've created (or have at least opened so far).
-static std::list<BroDoc*> docs_generated;
+std::list<BroDoc*> docs_generated;
 
 // reST comments (those starting with ##) seen so far.
 std::list<std::string>* reST_doc_comments = 0;
@@ -611,17 +610,8 @@ static int load_files_with_prefix(const char* orig_file)
 
 			if ( generate_documentation )
 				{
-				const char* bifExtStart = strstr(full_filename, ".bif.bro");
+				current_reST_doc = new BroDoc(full_filename);
-				BroDoc* reST_doc;
+				docs_generated.push_back(current_reST_doc);
-
-				if ( bifExtStart )
-					reST_doc = new BroBifDoc(full_filename);
-				else
-					reST_doc = new BroDoc(full_filename);
-
-				docs_generated.push_back(reST_doc);
-
-				current_reST_doc = reST_doc;
 				}
 			}
 
@@ -877,18 +867,7 @@ int yywrap()
 		}
 
 	if ( generate_documentation )
-		{
-		std::list<BroDoc*>::iterator it;
-
-		for ( it = docs_generated.begin(); it != docs_generated.end(); ++it )
-			{
-			(*it)->WriteDocFile();
-			delete *it;
-			}
-
-		docs_generated.clear();
 		clear_reST_doc_comments();
-		}
 
 	// Otherwise, we are done.
 	return 1;

testing/btest/Baseline/doc.autogen-reST-example/example.rst

@@ -13,11 +13,23 @@ these comments are transferred directly into the auto-generated
 `reStructuredText <http://docutils.sourceforge.net/rst.html>`_
 (reST) document's summary section.
 
-.. tip:: You can embed directives and roles within ``##``-stylized comments
+.. tip:: You can embed directives and roles within ``##``-stylized comments.
+
+A script's logging information has to be documented manually as minimally
+shown below.  Note that references may not always be possible (e.g.
+anonymous filter functions) and a script may not need to document
+each of "columns", "event", "filter" depending on exactly what it's doing.
+
+**Logging Stream ID:** :bro:enum:`Example::EXAMPLE`
+    :Columns:    :bro:type:`Example::Info`
+    :Event:      :bro:id:`Example::log_example`
+    :Filter:     ``example-filter``
+        uses :bro:id:`Example::filter_func` to determine whether to
+        exclude the ``ts`` field
 
 :Author: Jon Siwek <jsiwek@ncsa.illinois.edu>
 
-:Imports: :doc:`notice`
+:Imports: :doc:`notice </policy/notice>`
 
 Summary
 ~~~~~~~
@@ -49,13 +61,20 @@ Types
                                                        goes here.
 
 :bro:type:`Example::ComplexRecord`: :bro:type:`record` general documentation for a type "ComplexRecord" goes here
+
+:bro:type:`Example::Info`: :bro:type:`record`          An example record to be used with a logging stream.
 ====================================================== ==========================================================
 
 Events
 ######
-============================================== ==========================
+================================================= =============================================================
 :bro:id:`Example::an_event`: :bro:type:`event`    Summarize "an_event" here.
-============================================== ==========================
+
+:bro:id:`Example::log_example`: :bro:type:`event` This is a declaration of an example event that can be used in
+                                                  logging streams and is raised once for each log entry.
+
+:bro:id:`bro_init`: :bro:type:`event`
+================================================= =============================================================
 
 Functions
 #########
@@ -65,9 +84,13 @@ Functions
 
 Redefinitions
 #############
-================================================= ====================================
+===================================================== ========================================
+:bro:type:`Log::ID`: :bro:type:`enum`
+
 :bro:type:`Example::SimpleEnum`: :bro:type:`enum`     document the "SimpleEnum" redef here
-================================================= ====================================
+
+:bro:type:`Example::SimpleRecord`: :bro:type:`record` document the record extension redef here
+===================================================== ========================================
 
 Namespaces
 ~~~~~~~~~~
@@ -180,6 +203,18 @@ Types
 
    general documentation for a type "ComplexRecord" goes here
 
+.. bro:type:: Example::Info
+
+   :Type: :bro:type:`record`
+
+      ts: :bro:type:`time` :bro:attr:`&log`
+
+      uid: :bro:type:`string` :bro:attr:`&log`
+
+      status: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&optional`
+
+   An example record to be used with a logging stream.
+
 Events
 ~~~~~~
 .. bro:id:: Example::an_event
@@ -191,6 +226,17 @@ Events
    
    :param name: describe the argument here
 
+.. bro:id:: Example::log_example
+
+   :Type: :bro:type:`event` (rec: :bro:type:`Example::Info`)
+
+   This is a declaration of an example event that can be used in
+   logging streams and is raised once for each log entry.
+
+.. bro:id:: bro_init
+
+   :Type: :bro:type:`event` ()
+
 Functions
 ~~~~~~~~~
 .. bro:id:: Example::a_function
@@ -213,6 +259,12 @@ Functions
 
 Redefinitions
 ~~~~~~~~~~~~~
+:bro:type:`Log::ID`
+
+   :Type: :bro:type:`enum`
+
+      .. bro:enum:: Example::EXAMPLE Log::ID
+
 :bro:type:`Example::SimpleEnum`
 
    :Type: :bro:type:`enum`
@@ -227,6 +279,16 @@ Redefinitions
 
    document the "SimpleEnum" redef here
 
+.. bro:type:: Example::SimpleRecord
+
+   :Type: :bro:type:`record`
+
+      field_ext: :bro:type:`string` :bro:attr:`&optional`
+         document the extending field here
+         (or here)
+
+   document the record extension redef here
+
 Port Analysis
 -------------
 :ref:`More Information <common_port_analysis_doc>`
@@ -234,8 +296,8 @@ Port Analysis
 SSL::
 
     [ports={
-        563/tcp,
+        443/tcp,
-        443/tcp
+        562/tcp
     }]
 
 Packet Filter
@@ -244,8 +306,8 @@ Packet Filter
 
 Filters added::
 
-    [nntps] = tcp port 563,
+    [ssl] = tcp port 443,
-    [ssl] = tcp port 443
+    [nntps] = tcp port 562
 
 Private Interface
 -----------------
@@ -260,8 +322,8 @@ State Variables
    ::
 
       {
-         563/tcp,
+         443/tcp,
-         443/tcp
+         562/tcp
       }
 
 Types
@@ -276,6 +338,10 @@ Types
 
 Functions
 ~~~~~~~~~
+.. bro:id:: Example::filter_func
+
+   :Type: :bro:type:`function` (rec: :bro:type:`Example::Info`) : :bro:type:`bool`
+
 .. bro:id:: Example::function_without_proto
 
    :Type: :bro:type:`function` (tag: :bro:type:`string`) : :bro:type:`string`

testing/btest/Baseline/doc.autogen-reST-record-add/autogen-reST-record-add.rst

@@ -0,0 +1,99 @@
+.. Automatically generated.  Do not edit.
+
+autogen-reST-record-add.bro
+===========================
+
+:download:`Original Source File <autogen-reST-record-add.bro>`
+
+Overview
+--------
+
+
+Summary
+~~~~~~~
+State Variables
+###############
+===================================== =
+:bro:id:`a`: :bro:type:`my_record`
+
+:bro:id:`b`: :bro:type:`super_record`
+===================================== =
+
+Types
+#####
+============================================ =
+:bro:type:`my_record`: :bro:type:`record`
+
+:bro:type:`super_record`: :bro:type:`record`
+============================================ =
+
+Functions
+#########
+===================================== =
+:bro:id:`test_func`: :bro:type:`func`
+===================================== =
+
+Redefinitions
+#############
+========================================= =
+:bro:type:`my_record`: :bro:type:`record`
+========================================= =
+
+Public Interface
+----------------
+State Variables
+~~~~~~~~~~~~~~~
+.. bro:id:: a
+
+   :Type: :bro:type:`my_record`
+   :Default:
+
+   ::
+
+      {
+         field1=<uninitialized>
+         field2=<uninitialized>
+         field3=<uninitialized>
+      }
+
+.. bro:id:: b
+
+   :Type: :bro:type:`super_record`
+   :Default:
+
+   ::
+
+      {
+         rec=[field1=<uninitialized>, field2=<uninitialized>, field3=<uninitialized>]
+      }
+
+Types
+~~~~~
+.. bro:type:: my_record
+
+   :Type: :bro:type:`record`
+
+      field1: :bro:type:`bool`
+
+      field2: :bro:type:`string`
+
+.. bro:type:: super_record
+
+   :Type: :bro:type:`record`
+
+      rec: :bro:type:`my_record`
+
+Functions
+~~~~~~~~~
+.. bro:id:: test_func
+
+   :Type: :bro:type:`function` () : :bro:type:`void`
+
+Redefinitions
+~~~~~~~~~~~~~
+.. bro:type:: my_record
+
+   :Type: :bro:type:`record`
+
+      field3: :bro:type:`count` :bro:attr:`&optional`
+

testing/btest/Baseline/doc.autogen-reST-type-aliases/autogen-reST-type-aliases.rst

@@ -24,7 +24,7 @@ Types
 ============================================ ==========================================================================
 :bro:type:`TypeAlias`: :bro:type:`bool`      This is just an alias for a builtin type ``bool``.
 
-:bro:type:`OtherTypeAlias`: :bro:type:`bool` We decided that creating alias "chains" might now be so useful to document
+:bro:type:`OtherTypeAlias`: :bro:type:`bool` We decided that creating alias "chains" might not be so useful to document
                                              so this type just creates a cross reference to ``bool``.
 ============================================ ==========================================================================
 
@@ -56,6 +56,6 @@ Types
 
    :Type: :bro:type:`bool`
 
-   We decided that creating alias "chains" might now be so useful to document
+   We decided that creating alias "chains" might not be so useful to document
    so this type just creates a cross reference to ``bool``.
 

testing/btest/Scripts/doc/example-diff-canonifier.py

@@ -0,0 +1,15 @@
+#!/usr/bin/python
+
+import sys
+import re
+
+# MutableVal derivatives (e.g. sets/tables) don't always generate the same
+# ordering in the reST documentation, so just don't bother diffing
+# the places where example.bro uses them.
+
+RE1 = "\d*/tcp"
+RE2 = "tcp port \d*"
+
+for line in sys.stdin.readlines():
+    if re.search(RE1, line) is None and re.search(RE2, line) is None:
+        print line

testing/btest/btest.cfg

@@ -13,5 +13,6 @@ LOCALE=C
 PATH=%(testbase)s/../../build/src:%(testbase)s/../../aux/btest:%(default_path)s
 TEST_DIFF_CANONIFIER=%(testbase)s/Scripts/diff-canonifier
 TRACES=%(testbase)s/Traces
+SCRIPTS=%(testbase)s/Scripts
 DIST=%(testbase)s/../..
 BUILD=%(testbase)s/../../build

testing/btest/doc/autogen-reST-all

@@ -0,0 +1 @@
+@TEST-EXEC: cd $BUILD && make restdoc

testing/btest/doc/autogen-reST-example

@@ -1,2 +1,2 @@
-@TEST-EXEC: bro --doc-scripts $DIST/doc/example.bro
+@TEST-EXEC: bro --doc-scripts $DIST/doc/scripts/example.bro
 @TEST-EXEC: btest-diff example.rst

testing/btest/doc/autogen-reST-record-add.bro

@@ -0,0 +1,32 @@
+# @TEST-EXEC: bro --doc-scripts %INPUT
+# @TEST-EXEC: btest-diff autogen-reST-record-add.rst
+
+# When in doc mode, bro will clone declared types (see add_type() in Var.cc)
+# in order to keep track of the identifier name associated with the new type.
+# This test makes sure that the cloning is done in a way that's compatible
+# with adding fields to a record type -- we want to be sure that cloning
+# a record that contains other record fields will correctly see field
+# additions to those contained-records.
+
+type my_record: record {
+    field1: bool;
+    field2: string;
+};
+
+type super_record: record {
+    rec: my_record;
+};
+
+redef record my_record += {
+    field3: count &optional;
+};
+
+global a: my_record;
+
+global b: super_record;
+
+function test_func()
+{
+    a?$field3;
+    b$rec?$field3;
+}

testing/btest/doc/autogen-reST-type-aliases.bro

@@ -4,7 +4,7 @@
 ## This is just an alias for a builtin type ``bool``.
 type TypeAlias: bool;
 
-## We decided that creating alias "chains" might now be so useful to document
+## We decided that creating alias "chains" might not be so useful to document
 ## so this type just creates a cross reference to ``bool``.
 type OtherTypeAlias: TypeAlias;