Merge remote branch 'remotes/origin/topic/jsiwek/doc-framework'

* remotes/origin/topic/jsiwek/doc-framework:
  Adding example documentation for a script's use of logging features.
  Adding &log attribute to static attr_names array.
  Small typo fix.
  Bro doc mode now tracks record redefs that extend its field list.
  BroBifDoc was unneeded; now dead code, so removed.
  Bro doc mode now only does a "shallow" copy of declared record types
  Bro's doc mode now terminates after processing bro_init but before net_run
  Fixes related to `make doc` handling of script summary text (##! comments)
  Overhaul of "doc" build target for generating policy script documentation.
  Add parser error hint when in doc mode about checking ## comment syntax.
  Move stuff related to policy script documentation from doc/ to doc/scripts/
  Fixing example.bro's auto-reST generation baseline test.
This commit is contained in:
Robin Sommer 2011-05-09 19:02:39 -07:00
commit 5cd6394916
40 changed files with 752 additions and 422 deletions

View file

@ -1,42 +1 @@
set(POLICY_SRC_DIR ${PROJECT_SOURCE_DIR}/policy)
set(DOC_OUTPUT_DIR ${CMAKE_CURRENT_BINARY_DIR}/out)
set(DOC_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/source)
set(DOC_SOURCE_WORKDIR ${CMAKE_CURRENT_BINARY_DIR}/source)
file(GLOB_RECURSE DOC_SOURCES FOLLOW_SYMLINKS "*")
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/conf.py.in
${CMAKE_CURRENT_BINARY_DIR}/conf.py
@ONLY)
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/scripts/generate_reST_docs.py.in
${CMAKE_CURRENT_BINARY_DIR}/generate_reST_docs.py
@ONLY)
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/scripts/BroToReST.py.in
${CMAKE_CURRENT_BINARY_DIR}/BroToReST.py
@ONLY)
add_custom_target(doc
COMMAND "${CMAKE_COMMAND}" -E copy_directory
${DOC_SOURCE_DIR}
${DOC_SOURCE_WORKDIR}
COMMAND python generate_reST_docs.py
COMMAND sphinx-build
-b html
-c ${CMAKE_CURRENT_BINARY_DIR}
-d ${DOC_OUTPUT_DIR}/doctrees
${DOC_SOURCE_WORKDIR}
${DOC_OUTPUT_DIR}/html
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
COMMENT "[Sphinx] Generating Script Documentation"
VERBATIM
# SOURCES just adds stuff to IDE projects as a convienience
SOURCES ${DOC_SOURCES})
add_dependencies(doc bro doc-clean)
add_custom_target(doc-clean
COMMAND "${CMAKE_COMMAND}" -E remove_directory
${CMAKE_CURRENT_BINARY_DIR}/source
COMMAND "${CMAKE_COMMAND}" -E remove_directory
${DOC_OUTPUT_DIR}
VERBATIM)
add_subdirectory(scripts)

View file

@ -1,44 +1 @@
This directory contains scripts and templates that can be used to automate
the generation of Bro script documentation. Two build targets are defined
by CMake:
``make doc``
This target depends on a Python interpreter (>=2.5) and
`Sphinx <http://sphinx.pocoo.org/>`_ being installed. Sphinx can be
installed like::
> sudo easy_install sphinx
This target will also first build the bro binary if it is not already
since the generation of reStructuredText (reST) documentation from
Bro scripts is integrated within the parsing process.
After completion, HTML documentation can be located inside the CMake
``build/`` directory as ``build/doc/out/html``. The generated reST
documentation will be located in ``build/doc/source/policy``.
``make doc-clean``
This target removes Sphinx inputs and outputs from the CMake ``build/`` dir.
To schedule a script to be documented, edit ``scripts/generate_reST_docs.py.in``
and try adding the name of the script along with an optional script group to
the ``docs`` dictionary. That python script also shows other, more specialized
methods for generating documentation for some types of corner-cases.
When adding a new logical grouping for generated scripts, create a new
reST document in ``source/policy/<group_name>.rst`` and add some default
documentation. References to (and summaries of) documents associated with
the group get appended to this file during the ``make doc`` process.
The Sphinx source tree template in ``source/`` can be modified to add more
common/general documentation, style sheets, JavaScript, etc. The Sphinx
config file is produced from ``conf.py.in``, so that can be edited to change
various Sphinx options, like setting the default HTML rendering theme.
There is also a custom Sphinx domain implemented in ``source/ext/bro.py``
which adds some reST directives and roles that aid in generating useful
index entries and cross-references.
See ``example.bro`` for an example of how to document a Bro script such that
``make doc`` will be able to produce reST/HTML documentation for it.
TODO

View file

@ -1,152 +0,0 @@
#! /usr/bin/env python
import os
import subprocess
import shutil
import glob
import string
import sys
BRO = "@CMAKE_BINARY_DIR@/src/bro"
BROPATHDEV = "`@CMAKE_BINARY_DIR@/bro-path-dev`"
BRO_ARGS = "--doc-scripts"
DOC_DST_DIR = "@DOC_SOURCE_WORKDIR@/policy"
BROPATH = subprocess.Popen("@CMAKE_BINARY_DIR@/bro-path-dev", shell=True,
stdout=subprocess.PIPE, stderr=subprocess.STDOUT).stdout.readline()
class BroToReST:
"""A class to encapsulate the the generation of reST documentation from
a given Bro script.
"""
bro_src_file = None
doc_src_file = None
load_via_stdin = False
group = None
def __init__(self, src_file, load_method=False, search_dir=None, group=None):
"""
:param src_file: the file name of a Bro script (not a path)
:param load_method: T if script must be loaded by Bro via a stdin
redirection of "@load <script>", F if script can be loaded as
a command line argument to Bro
:param search_dir: a list of directories in which to search for
src_file. If None, the default BROPATH is used.
:param group: a string representing a logical group that the script's
documentation should belong to. A corresponding <group>.rst
document must be pre-existing in the policy/ dir of the source tree
used by Sphinx.
"""
self.bro_src_file = FindBroScript(src_file, search_dir)
self.load_via_stdin = load_method
self.group = group
# formulate doc_src_file from src_file
filename = os.path.basename(src_file)
basename, ext = os.path.splitext(filename)
if ext == ".bro":
self.doc_src_file = basename + ".rst"
else:
self.doc_src_file = filename + ".rst"
def __str__(self):
return "bro_src_file: " + self.bro_src_file \
+ "\ndoc_src_file: " + self.doc_src_file \
+ "\ndoc_dst_file: " + os.path.join(DOC_DST_DIR, self.doc_src_file) \
+ "\nstdin_load: %s" % self.load_via_stdin \
+ "\ngroup: %s" % self.group
def GenDoc(self):
"""Generates the reST documentation for a Bro script and copies
both the documentation and original script into Sphinx source tree.
If the documentation belongs to a group, the necessary modifications
to add it to the group's documentation are done. Afterwards, any
files with a ".rst" suffix are removed for the working directory.
"""
bro_src_basename = os.path.basename(self.bro_src_file)
if self.load_via_stdin:
cmd = "echo '@load %s' | %s %s" % (bro_src_basename, BRO, BRO_ARGS)
else:
cmd = "%s %s %s" % (BRO, BRO_ARGS, self.bro_src_file)
p = subprocess.Popen(cmd, shell=True, env={"BROPATH": BROPATH})
if p.wait() == 0:
shutil.copy(self.doc_src_file, DOC_DST_DIR)
shutil.copy(self.bro_src_file, DOC_DST_DIR)
AppendToDocGroup(self.group, self.bro_src_file, self.doc_src_file)
for leftover in glob.glob("*.rst"):
os.remove(leftover)
def GenDocs(doc_dict, load_method=False):
"""Generates reST documentation for all scripts in the given dictionary.
:param doc_dict: a dictionary whose keys are file names of Bro scripts
(not paths), and whose value is the logical documentation group
it belongs to
:param load_method: T if script must be loaded by Bro via a stdin
redirection of "@load <script>", F if script can be loaded as
a command line argument to Bro
"""
for k, v in doc_dict.iteritems():
doc = BroToReST(k, load_method, group=v)
print "Generating reST document for " + k
doc.GenDoc()
def FindBroScript(src_file, search_dir=None):
"""Search a set of paths for a given Bro script and return the absolute
path to it.
:param src_file: the file name of a Bro script (not a path)
:param search_dir: a list of directories in which to search for
src_file. If None, the default BROPATH is used.
"""
if search_dir is None:
search_dir = string.split(BROPATH, ":")
for path in search_dir:
abs_path = os.path.join(path, src_file)
if os.path.exists(abs_path):
return abs_path
print >> sys.stderr, "Couldn't find '%s'" % src_file
return None
def AppendToDocGroup(group, src_file, doc_file):
"""Adds a reference to the given documentation for a Bro script
to the documentation file for it's associated group. Also, associated
summary text (comments marked up like "##!" in the original Bro script
source) are added.
:param group: a string representing a logical group that the script's
documentation should belong to. A corresponding <group>.rst
document must be pre-existing in the policy/ dir of the source tree
used by Sphinx.
:param src_file: a path to the original Bro script source file
:param doc_file: the file name of a script's generated reST document
"""
if group is None:
return
group_file = os.path.join(DOC_DST_DIR, group + ".rst")
if not os.path.exists(group_file):
print >> sys.stderr, "Group file doesn't exist: " + group_file
return
summary_comments = []
with open(src_file, 'r') as f:
for line in f:
sum_pos = string.find(line, "##!")
if sum_pos != -1:
summary_comments.append(line[(sum_pos+3):])
doc_name, ext = os.path.splitext(doc_file)
with open(group_file, 'a') as f:
f.write("\n:doc:`%s`\n" % doc_name)
for line in summary_comments:
f.write(line)

263
doc/scripts/CMakeLists.txt Normal file
View file

@ -0,0 +1,263 @@
set(POLICY_SRC_DIR ${PROJECT_SOURCE_DIR}/policy)
set(BIF_SRC_DIR ${PROJECT_SOURCE_DIR}/src)
set(RST_OUTPUT_DIR ${CMAKE_CURRENT_BINARY_DIR}/rest_output)
set(DOC_OUTPUT_DIR ${CMAKE_CURRENT_BINARY_DIR}/out)
set(DOC_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/source)
set(DOC_SOURCE_WORKDIR ${CMAKE_CURRENT_BINARY_DIR}/source)
file(GLOB_RECURSE DOC_SOURCES FOLLOW_SYMLINKS "*")
# configure the Sphinx config file (expand variables CMake might know about)
configure_file(${CMAKE_CURRENT_SOURCE_DIR}/conf.py.in
${CMAKE_CURRENT_BINARY_DIR}/conf.py
@ONLY)
# find out what BROPATH to use when executing bro
execute_process(COMMAND ${CMAKE_BINARY_DIR}/bro-path-dev
OUTPUT_VARIABLE BROPATH
RESULT_VARIABLE retval
OUTPUT_STRIP_TRAILING_WHITESPACE)
if (NOT ${retval} EQUAL 0)
message(FATAL_ERROR "Problem setting BROPATH")
endif ()
# This macro is used to add a new makefile target for reST policy script
# documentation that can be generated using Bro itself to parse policy scripts.
# It's called like:
#
# rest_target(srcDir broInput [group])
#
# srcDir: the directory which contains broInput
# broInput: the file name of a bro policy script
# group: optional name of group that the script documentation will belong to
#
# In addition to adding the makefile target, several CMake variables are set:
#
# MASTER_POLICY_INDEX_TEXT: a running list of policy scripts docs that have
# been generated so far, formatted such that it can be appended to a file
# that ends in a Sphinx toctree directive
# ALL_REST_OUTPUTS: a running list (the CMake list type) of all reST docs
# that are to be generated
# MASTER_GROUP_LIST: a running list (the CMake list type) of all script groups
# ${group}_files: a running list of files belonging to a given group, from
# which summary text can be extracted at build time
# ${group}_doc_names: a running list of reST style document names that can be
# given to a :doc: role, shared indices with ${group}_files
#
macro(REST_TARGET srcDir broInput)
get_filename_component(basename ${broInput} NAME_WE)
get_filename_component(extension ${broInput} EXT)
get_filename_component(relDstDir ${broInput} PATH)
set(sumTextSrc ${srcDir}/${broInput})
if (${extension} STREQUAL ".bif.bro")
set(basename "${basename}.bif")
# the summary text is taken at configure time, but .bif.bro files
# may not have been generated yet, so read .bif file instead
set(sumTextSrc ${BIF_SRC_DIR}/${basename})
elseif (${extension} STREQUAL ".init")
set(basename "${basename}.init")
endif ()
set (restFile "${basename}.rst")
if (NOT relDstDir)
set(docName "${basename}")
set(dstDir "${RST_OUTPUT_DIR}")
else ()
set(docName "${relDstDir}/${basename}")
set(dstDir "${RST_OUTPUT_DIR}/${relDstDir}")
endif ()
set(restOutput "${dstDir}/${restFile}")
set(indexEntry " ${docName} <${docName}>")
set(MASTER_POLICY_INDEX_TEXT "${MASTER_POLICY_INDEX_TEXT}\n${indexEntry}")
list(APPEND ALL_REST_OUTPUTS ${restOutput})
if (NOT "${ARGN}" STREQUAL "")
set(group ${ARGN})
# add group to master group list if not already in it
list(FIND MASTER_GROUP_LIST ${group} _found)
if (_found EQUAL -1)
list(APPEND MASTER_GROUP_LIST ${group})
if (MASTER_GROUP_LIST_TEXT)
set(MASTER_GROUP_LIST_TEXT "${MASTER_GROUP_LIST_TEXT}\n${group}")
else ()
set(MASTER_GROUP_LIST_TEXT "${group}")
endif ()
endif ()
list(APPEND ${group}_files ${sumTextSrc})
list(APPEND ${group}_doc_names ${docName})
else ()
set(group "")
endif ()
if (${group} STREQUAL "default" OR ${group} STREQUAL "bifs")
set(BRO_ARGS --doc-scripts --exec '')
else ()
set(BRO_ARGS --doc-scripts ${srcDir}/${broInput})
endif ()
add_custom_command(OUTPUT ${restOutput}
# delete any leftover state from previous bro runs
COMMAND "${CMAKE_COMMAND}"
ARGS -E remove_directory .state
# generate the reST documentation using bro
COMMAND BROPATH=${BROPATH} ${CMAKE_BINARY_DIR}/src/bro
ARGS ${BRO_ARGS} || (rm -rf .state *.log *.rst && exit 1)
# move generated doc into a new directory tree that
# defines the final structure of documents
COMMAND "${CMAKE_COMMAND}"
ARGS -E make_directory ${dstDir}
COMMAND "${CMAKE_COMMAND}"
ARGS -E copy ${restFile} ${restOutput}
# copy the bro policy script, too
COMMAND "${CMAKE_COMMAND}"
ARGS -E copy ${srcDir}/${broInput} ${dstDir}
# clean up the build directory
COMMAND rm
ARGS -rf .state *.log *.rst
DEPENDS bro
DEPENDS ${srcDir}/${broInput}
COMMENT "[Bro] Generating reST docs for ${broInput}"
)
endmacro(REST_TARGET)
# Schedule Bro scripts for which to generate documentation.
# Note: the script may be located in a subdirectory off of one of the main
# directories in BROPATH. In that case, just list the script as 'foo/bar.bro'
rest_target(${POLICY_SRC_DIR} alarm.bro user)
rest_target(${POLICY_SRC_DIR} arp.bro user)
rest_target(${POLICY_SRC_DIR} conn.bro user)
rest_target(${POLICY_SRC_DIR} dhcp.bro user)
rest_target(${POLICY_SRC_DIR} dns.bro user)
rest_target(${POLICY_SRC_DIR} ftp.bro user)
rest_target(${POLICY_SRC_DIR} http.bro user)
rest_target(${POLICY_SRC_DIR} http-reply.bro user)
rest_target(${POLICY_SRC_DIR} http-request.bro user)
rest_target(${POLICY_SRC_DIR} irc.bro user)
rest_target(${POLICY_SRC_DIR} smtp.bro user)
rest_target(${POLICY_SRC_DIR} ssl.bro user)
rest_target(${POLICY_SRC_DIR} ssl-ciphers.bro user)
rest_target(${POLICY_SRC_DIR} ssl-errors.bro user)
rest_target(${POLICY_SRC_DIR} synflood.bro user)
rest_target(${POLICY_SRC_DIR} tcp.bro user)
rest_target(${POLICY_SRC_DIR} udp.bro user)
rest_target(${POLICY_SRC_DIR} weird.bro user)
rest_target(${CMAKE_CURRENT_SOURCE_DIR} example.bro internal)
# Finding out what scripts bro will generate documentation for by default
# can be done like: `bro --doc-scripts --exec ""`
rest_target(${POLICY_SRC_DIR} bro.init default)
rest_target(${POLICY_SRC_DIR} logging-ascii.bro default)
rest_target(${POLICY_SRC_DIR} logging.bro default)
rest_target(${POLICY_SRC_DIR} pcap.bro default)
rest_target(${POLICY_SRC_DIR} server-ports.bro default)
rest_target(${CMAKE_BINARY_DIR}/src bro.bif.bro bifs)
rest_target(${CMAKE_BINARY_DIR}/src const.bif.bro bifs)
rest_target(${CMAKE_BINARY_DIR}/src event.bif.bro bifs)
rest_target(${CMAKE_BINARY_DIR}/src logging.bif.bro bifs)
rest_target(${CMAKE_BINARY_DIR}/src strings.bif.bro bifs)
rest_target(${CMAKE_BINARY_DIR}/src types.bif.bro bifs)
# create temporary list of all docs to include in the master policy/index file
file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/tmp_policy_index
"${MASTER_POLICY_INDEX_TEXT}")
# create temporary file containing list of all groups
file(WRITE ${CMAKE_CURRENT_BINARY_DIR}/group_list
"${MASTER_GROUP_LIST_TEXT}")
# create temporary files containing list of each source file in a given group
foreach (group ${MASTER_GROUP_LIST})
if (EXISTS ${CMAKE_CURRENT_BINARY_DIR}/${group}_files)
file(REMOVE ${CMAKE_CURRENT_BINARY_DIR}/${group}_files)
endif ()
if (EXISTS ${CMAKE_CURRENT_BINARY_DIR}/${group}_doc_names)
file(REMOVE ${CMAKE_CURRENT_BINARY_DIR}/${group}_doc_names)
endif ()
foreach (src ${${group}_files})
file(APPEND ${CMAKE_CURRENT_BINARY_DIR}/${group}_files "${src}\n")
endforeach ()
foreach (dname ${${group}_doc_names})
file(APPEND ${CMAKE_CURRENT_BINARY_DIR}/${group}_doc_names "${dname}\n")
endforeach ()
endforeach ()
# remove previously generated docs no longer scheduled for generation
if (EXISTS ${RST_OUTPUT_DIR})
file(GLOB_RECURSE EXISTING_REST_DOCS "${RST_OUTPUT_DIR}/*.rst")
foreach (_doc ${EXISTING_REST_DOCS})
list(FIND ALL_REST_OUTPUTS ${_doc} _found)
if (_found EQUAL -1)
file(REMOVE ${_doc})
message(STATUS "Removing stale reST doc: ${_doc}")
endif ()
endforeach ()
endif ()
# The "restdoc" target uses Bro to parse policy scripts in order to
# generate reST documentation from them.
add_custom_target(restdoc
# create symlink to the reST output directory for convenience
COMMAND "${CMAKE_COMMAND}" -E create_symlink
${RST_OUTPUT_DIR}
${CMAKE_BINARY_DIR}/reST
DEPENDS ${ALL_REST_OUTPUTS})
# The "restclean" target removes all generated reST documentation from the
# build directory.
add_custom_target(restclean
COMMAND "${CMAKE_COMMAND}" -E remove_directory
${RST_OUTPUT_DIR}
VERBATIM)
# The "doc" target generates reST documentation for any outdated bro scripts
# and then uses Sphinx to generate HTML documentation from the reST
add_custom_target(doc
# copy the template documentation to the build directory
# to give as input for sphinx
COMMAND "${CMAKE_COMMAND}" -E copy_directory
${DOC_SOURCE_DIR}
${DOC_SOURCE_WORKDIR}
# copy generated policy script documentation into the
# working copy of the template documentation
COMMAND "${CMAKE_COMMAND}" -E copy_directory
${RST_OUTPUT_DIR}
${DOC_SOURCE_WORKDIR}/policy
# append to the master index of all policy scripts
COMMAND cat ${CMAKE_CURRENT_BINARY_DIR}/tmp_policy_index >>
${DOC_SOURCE_WORKDIR}/policy/index.rst
# construct a reST file for each group
COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/group_index_generator.py
${CMAKE_CURRENT_BINARY_DIR}/group_list
${CMAKE_CURRENT_BINARY_DIR}
${DOC_SOURCE_WORKDIR}
# tell sphinx to generate html
COMMAND sphinx-build
-b html
-c ${CMAKE_CURRENT_BINARY_DIR}
-d ${DOC_OUTPUT_DIR}/doctrees
${DOC_SOURCE_WORKDIR}
${DOC_OUTPUT_DIR}/html
# create symlink to the html output directory for convenience
COMMAND "${CMAKE_COMMAND}" -E create_symlink
${DOC_OUTPUT_DIR}/html
${CMAKE_BINARY_DIR}/html
WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}
COMMENT "[Sphinx] Generating HTML policy script docs"
# SOURCES just adds stuff to IDE projects as a convienience
SOURCES ${DOC_SOURCES}
DEPENDS restdoc docclean)
# The "docclean" target removes just the Sphinx input/output directories
# from the build directory.
add_custom_target(docclean
COMMAND "${CMAKE_COMMAND}" -E remove_directory
${DOC_SOURCE_WORKDIR}
COMMAND "${CMAKE_COMMAND}" -E remove_directory
${DOC_OUTPUT_DIR}
VERBATIM)

61
doc/scripts/README Normal file
View file

@ -0,0 +1,61 @@
This directory contains scripts and templates that can be used to automate
the generation of Bro script documentation. Several build targets are defined
by CMake:
``restdoc``
This target uses Bro to parse policy scripts in order to generate
reStructuredText (reST) documentation from them. The list of scripts
for which to generate reST documentation is defined in the
``CMakeLists.txt`` file in this directory. Script documentation is
rebuild automatically if the policy script from which it is derived
or the Bro binary becomes out of date
The resulting output from this target can be found in the CMake
``build/`` directory inside ``reST`` (a symlink to
``doc/scripts/rest_output``).
``doc``
This target depends on a Python interpreter (>=2.5) and
`Sphinx <http://sphinx.pocoo.org/>`_ being installed. Sphinx can be
installed like::
> sudo easy_install sphinx
This target will first build ``restdoc`` target and then copy the
resulting reST files as an input directory to Sphinx.
After completion, HTML documentation can be located in the CMake
``build/`` directory inside ``html`` (a symlink to
``doc/scripts/out/html``)
``restclean``
This target removes any reST documentation that has been generated so far.
``docclean``
This target removes Sphinx inputs and outputs from the CMake ``build/`` dir.
To schedule a script to be documented, edit ``CMakeLists.txt`` inside this
directory add a call to the ``rest_target()`` macro. Calling that macro
with a group name for the script is optional, but if not given, the only
link to the script will be in the master TOC tree for all policy scripts.
When adding a new logical grouping for generated scripts, create a new
reST document in ``source/<group_name>.rst`` and add some default
documentation for the group. References to (and summaries of) documents
associated with the group get appended to this file during the
``make doc`` process.
The Sphinx source tree template in ``source/`` can be modified to add more
common/general documentation, style sheets, JavaScript, etc. The Sphinx
config file is produced from ``conf.py.in``, so that can be edited to change
various Sphinx options, like setting the default HTML rendering theme.
There is also a custom Sphinx domain implemented in ``source/ext/bro.py``
which adds some reST directives and roles that aid in generating useful
index entries and cross-references.
See ``example.bro`` for an example of how to document a Bro script such that
``make doc`` will be able to produce reST/HTML documentation for it.

View file

@ -6,6 +6,18 @@
##!
##! .. tip:: You can embed directives and roles within ``##``-stylized comments.
##!
##! A script's logging information has to be documented manually as minimally
##! shown below. Note that references may not always be possible (e.g.
##! anonymous filter functions) and a script may not need to document
##! each of "columns", "event", "filter" depending on exactly what it's doing.
##!
##! **Logging Stream ID:** :bro:enum:`Example::EXAMPLE`
##! :Columns: :bro:type:`Example::Info`
##! :Event: :bro:id:`Example::log_example`
##! :Filter: ``example-filter``
##! uses :bro:id:`Example::filter_func` to determine whether to
##! exclude the ``ts`` field
##!
##! :Author: Jon Siwek <jsiwek@ncsa.illinois.edu>
# Comments that use a single pound sign (#) are not significant to
@ -64,6 +76,12 @@ redef enum Notice += {
Notice_Four,
};
# Redef'ing the ID enumeration for logging streams is automatically tracked.
# Comments of the "##" form can be use to further document it, but it's
# better to do all documentation related to logging in the summary section
# as is shown above.
redef enum Log::ID += { EXAMPLE };
# Anything declared in the export section will show up in the rendered
# documentation's "public interface" section
@ -107,6 +125,11 @@ export {
field2: bool; ##< toggles something
};
## document the record extension redef here
redef record SimpleRecord += {
## document the extending field here
field_ext: string &optional; ##< (or here)
};
## general documentation for a type "ComplexRecord" goes here
type ComplexRecord: record {
@ -116,6 +139,13 @@ export {
msg: string &default="blah"; ##< attributes are self-documenting
} &redef;
## An example record to be used with a logging stream.
type Info: record {
ts: time &log;
uid: string &log;
status: count &log &optional;
};
############## options ################
# right now, I'm just defining an option as
# any const with &redef (something that can
@ -159,6 +189,15 @@ export {
## Give more details about "an_event" here.
## name: describe the argument here
global an_event: event(name: string);
## This is a declaration of an example event that can be used in
## logging streams and is raised once for each log entry.
global log_example: event(rec: Info);
}
function filter_func(rec: Info): bool
{
return T;
}
# this function is documented in the "private interface" section
@ -176,3 +215,14 @@ type PrivateRecord: record {
field1: bool;
field2: count;
};
event bro_init()
{
Log::create_stream(EXAMPLE, [$columns=Info, $ev=log_example]);
Log::add_filter(EXAMPLE, [
$name="example-filter",
$path="example-filter",
$pred=filter_func,
$exclude=set("ts")
]);
}

View file

@ -1,71 +0,0 @@
#! /usr/bin/env python
import os
import subprocess
import shutil
import glob
import string
import sys
from BroToReST import *
# TODO: generate docs for more scripts
# TODO: the groups are just made up to test the functionality, fix them
# Scripts that can be loaded by bro via command line argument:
docs = {
"alarm.bro": "internal",
"arp.bro": "user",
"conn.bro": "internal",
"dhcp.bro": "user",
"dns.bro": "user",
"ftp.bro": "user",
"http.bro": "user",
"http-reply.bro": None,
"http-request.bro": None,
"irc.bro": "user",
"smtp.bro": "user",
"ssl.bro": "user",
"ssl-ciphers.bro": None,
"ssl-errors.bro": None,
"synflood.bro": "user",
"tcp.bro": "user",
"udp.bro": "user",
"weird.bro": "internal",
}
# Scripts that can't be loaded by bro via command line argument (possible
# due to dependency issues), but can be loaded via an @load on stdin:
stdin_docs = {
"notice.bro": "internal",
}
GenDocs(docs)
GenDocs(stdin_docs, True)
# The example documentation script doesn't live on the BROPATH, so
# explicitly generate the docs for it like this:
BroToReST("example.bro", False, ["@PROJECT_SOURCE_DIR@/doc"], group="internal").GenDoc()
# Generate documentation for stuff that's always loaded into bro by default:
cmd = "echo '' | %s %s" % (BRO, BRO_ARGS)
p = subprocess.Popen(cmd, shell=True, env={"BROPATH": BROPATH})
if p.wait() == 0:
for doc in glob.glob("*.rst"):
if doc == "<stdin>.rst":
os.remove(doc)
continue
basename, ext = os.path.splitext(doc)
basename2, ext = os.path.splitext(basename)
if ext == ".init":
src_file = basename
else:
src_file = basename + ".bro"
src_file = FindBroScript(src_file)
shutil.copy(src_file, DOC_DST_DIR)
shutil.copy(doc, DOC_DST_DIR)
if ext == ".bif":
AppendToDocGroup("bifs", src_file, doc)
else:
AppendToDocGroup("default", src_file, doc)
os.remove(doc)

View file

@ -0,0 +1,52 @@
#! /usr/bin/env python
# This script automatically generates a reST documents that lists
# a collection of Bro policy scripts that are "grouped" together.
# The summary text (##! comments) of the policy script is embedded
# in the list.
#
# 1st argument is the file containing list of groups
# 2nd argument is the directory containing ${group}_files lists of
# scripts that belong to the group and ${group}_doc_names lists of
# document names that can be supplied to a reST :doc: role
# 3rd argument is a directory in which write a ${group}.rst file (will
# append to existing file) that contains reST style references to
# script docs along with summary text contained in original script
import sys
import os
import string
group_list = sys.argv[1]
file_manifest_dir = sys.argv[2]
output_dir = sys.argv[3]
with open(group_list, 'r') as f_group_list:
for group in f_group_list.read().splitlines():
#print group
file_manifest = os.path.join(file_manifest_dir, group + "_files")
doc_manifest = os.path.join(file_manifest_dir, group + "_doc_names")
src_files = []
doc_names = []
with open(file_manifest, 'r') as f_file_manifest:
src_files = f_file_manifest.read().splitlines()
with open(doc_manifest, 'r') as f_doc_manifest:
doc_names = f_doc_manifest.read().splitlines()
for i in range(len(src_files)):
src_file = src_files[i]
#print "\t" + src_file
summary_comments = []
with open(src_file, 'r') as f_src_file:
for line in f_src_file:
sum_pos = string.find(line, "##!")
if sum_pos != -1:
summary_comments.append(line[(sum_pos+3):])
#print summary_comments
group_file = os.path.join(output_dir, group + ".rst")
with open(group_file, 'a') as f_group_file:
f_group_file.write("\n:doc:`/policy/%s`\n" % doc_names[i])
for line in summary_comments:
f_group_file.write(" " + line)

View file

@ -118,4 +118,6 @@ The Bro scripting language supports the following built-in attributes.
.. bro:attr:: &group
.. bro:attr:: &log
.. bro:attr:: (&tracked)

View file

@ -11,11 +11,11 @@ Contents:
common
builtins
policy/default
policy/user
policy/bifs
policy/internal
policy/index
default
bifs
user
internal
Indices and tables
==================

View file

@ -1,10 +1,6 @@
Index of All Policy Script Documentation
========================================
Contents:
.. toctree::
:maxdepth: 1
:glob:
*

View file

@ -19,7 +19,7 @@ const char* attr_name(attr_tag t)
"&persistent", "&synchronized", "&postprocessor",
"&encrypt", "&match", "&disable_print_hook",
"&raw_output", "&mergeable", "&priority",
"&group", "(&tracked)",
"&group", "&log", "(&tracked)",
};
return attr_names[int(t)];

View file

@ -1,16 +0,0 @@
#include <cstdio>
#include <string>
#include <list>
#include "BroDoc.h"
#include "BroBifDoc.h"
BroBifDoc::BroBifDoc(const std::string& sourcename) : BroDoc(sourcename)
{
}
// TODO: This needs to do something different than parent class's version.
void BroBifDoc::WriteDocFile() const
{
BroDoc::WriteDocFile();
}

View file

@ -1,18 +0,0 @@
#ifndef brobifdoc_h
#define brobifdoc_h
#include <cstdio>
#include <string>
#include <list>
#include "BroDoc.h"
class BroBifDoc : public BroDoc {
public:
BroBifDoc(const std::string& sourcename);
virtual ~BroBifDoc() { }
void WriteDocFile() const;
};
#endif

View file

@ -58,15 +58,8 @@ void BroDoc::AddImport(const std::string& s)
if ( ext_pos == std::string::npos )
imports.push_back(s);
else
{
if ( s.substr(ext_pos + 1) == "bro" )
imports.push_back(s.substr(0, ext_pos));
else
fprintf(stderr, "Warning: skipped documenting @load of file "
"without .bro extension: %s\n", s.c_str());
}
}
void BroDoc::SetPacketFilter(const std::string& s)
@ -116,7 +109,15 @@ void BroDoc::WriteDocFile() const
if ( ! imports.empty() )
{
WriteToDoc(":Imports: ");
WriteStringList(":doc:`%s`, ", ":doc:`%s`\n", imports);
std::list<std::string>::const_iterator it;
for ( it = imports.begin(); it != imports.end(); ++it )
{
if ( it != imports.begin() )
WriteToDoc(", ");
WriteToDoc(":doc:`%s </policy/%s>`", it->c_str(), it->c_str());
}
WriteToDoc("\n");
}
WriteToDoc("\n");

View file

@ -286,7 +286,6 @@ set(bro_SRCS
BitTorrent.cc
BitTorrentTracker.cc
BPF_Program.cc
BroBifDoc.cc
BroDoc.cc
BroDocObj.cc
BroString.cc

View file

@ -456,6 +456,8 @@ public:
// Given an offset, returns the field's name.
const char* FieldName(int field) const;
type_decl_list* Types() { return types; }
// Given an offset, returns the field's TypeDecl.
const TypeDecl* FieldDecl(int field) const;
TypeDecl* FieldDecl(int field);

View file

@ -241,6 +241,15 @@ void add_type(ID* id, BroType* t, attr_list* attr, int /* is_event */)
// is to add an ID* to class ID that tracks aliases and set it here if
// t->GetTypeID() is true.
if ( generate_documentation )
{
if ( t->Tag() == TYPE_RECORD )
{
// Only "shallow" copy record types because we want to be able
// to see additions to the original type's list of fields
tnew = new RecordType(t->AsRecordType()->Types());
}
else
{
SerializationFormat* form = new BinarySerializationFormat();
form->StartWrite();
@ -258,6 +267,7 @@ void add_type(ID* id, BroType* t, attr_list* attr, int /* is_event */)
tnew = t->Unserialize(&uinfo);
delete [] data;
}
tnew->SetTypeID(copy_string(id->Name()));
}

View file

@ -319,8 +319,8 @@ definitions: definitions definition opt_ws
fprintf(fp_netvar_h, "// %s\n\n", auto_gen_comment);
fprintf(fp_netvar_init, "// %s\n\n", auto_gen_comment);
fprintf(fp_bro_init, "%s", $1);
fprintf(fp_bro_init, "export {\n");
fprintf(fp_func_def, "%s", $1);
}
;

View file

@ -9,6 +9,7 @@
#include <unistd.h>
#include <signal.h>
#include <string.h>
#include <list>
#ifdef HAVE_GETOPT_H
#include <getopt.h>
#endif
@ -47,6 +48,7 @@ extern "C" void OPENSSL_add_all_algorithms_conf(void);
#include "Stats.h"
#include "ConnCompressor.h"
#include "DPM.h"
#include "BroDoc.h"
#include "binpac_bro.h"
@ -103,6 +105,8 @@ char* proc_status_file = 0;
int FLAGS_use_binpac = false;
extern std::list<BroDoc*> docs_generated;
// Keep copy of command line
int bro_argc;
char** bro_argv;
@ -971,6 +975,20 @@ int main(int argc, char** argv)
mgr.Drain();
if ( generate_documentation )
{
std::list<BroDoc*>::iterator it;
for ( it = docs_generated.begin(); it != docs_generated.end(); ++it )
(*it)->WriteDocFile();
for ( it = docs_generated.begin(); it != docs_generated.end(); ++it )
delete *it;
terminate_bro();
return 0;
}
have_pending_timers = ! reading_traces && timer_mgr->Size() > 0;
if ( io_sources.Size() > 0 || have_pending_timers )

View file

@ -1067,8 +1067,10 @@ decl:
}
| TOK_REDEF TOK_RECORD global_id TOK_ADD_TO
'{' type_decl_list '}' opt_attr ';'
'{' { do_doc_token_start(); } type_decl_list '}' opt_attr ';'
{
do_doc_token_stop();
if ( ! $3->Type() )
$3->Error("unknown identifier");
else
@ -1078,9 +1080,27 @@ decl:
$3->Error("not a record type");
else
{
const char* error = add_to->AddFields($6, $8);
const char* error = add_to->AddFields($7, $9);
if ( error )
$3->Error(error);
else if ( generate_documentation )
{
if ( fake_type_decl_list )
{
BroType* fake_record =
new RecordType(fake_type_decl_list);
ID* fake = create_dummy_id($3, fake_record);
fake_type_decl_list = 0;
current_reST_doc->AddRedef(
new BroDocObj(fake, reST_doc_comments, true));
}
else
{
fprintf(stderr, "Warning: doc mode did not process "
"record extension for '%s', CommentedTypeDecl"
"list unavailable.\n", $3->Name());
}
}
}
}
}
@ -1622,7 +1642,7 @@ opt_doc_list:
int yyerror(const char msg[])
{
char* msgbuf = new char[strlen(msg) + strlen(last_tok) + 64];
char* msgbuf = new char[strlen(msg) + strlen(last_tok) + 128];
if ( last_tok[0] == '\n' )
sprintf(msgbuf, "%s, on previous line", msg);
@ -1631,6 +1651,10 @@ int yyerror(const char msg[])
else
sprintf(msgbuf, "%s, at or near \"%s\"", msg, last_tok);
if ( generate_documentation )
strcat(msgbuf, "\nDocumentation mode is enabled: "
"remember to check syntax of ## style comments\n");
error(msgbuf);
return 0;

View file

@ -16,7 +16,6 @@
#include "PolicyFile.h"
#include "broparse.h"
#include "BroDoc.h"
#include "BroBifDoc.h"
#include "Analyzer.h"
#include "AnalyzerTags.h"
@ -58,7 +57,7 @@ char last_tok[128];
static PList(char) files_scanned;
// reST documents that we've created (or have at least opened so far).
static std::list<BroDoc*> docs_generated;
std::list<BroDoc*> docs_generated;
// reST comments (those starting with ##) seen so far.
std::list<std::string>* reST_doc_comments = 0;
@ -611,17 +610,8 @@ static int load_files_with_prefix(const char* orig_file)
if ( generate_documentation )
{
const char* bifExtStart = strstr(full_filename, ".bif.bro");
BroDoc* reST_doc;
if ( bifExtStart )
reST_doc = new BroBifDoc(full_filename);
else
reST_doc = new BroDoc(full_filename);
docs_generated.push_back(reST_doc);
current_reST_doc = reST_doc;
current_reST_doc = new BroDoc(full_filename);
docs_generated.push_back(current_reST_doc);
}
}
@ -877,18 +867,7 @@ int yywrap()
}
if ( generate_documentation )
{
std::list<BroDoc*>::iterator it;
for ( it = docs_generated.begin(); it != docs_generated.end(); ++it )
{
(*it)->WriteDocFile();
delete *it;
}
docs_generated.clear();
clear_reST_doc_comments();
}
// Otherwise, we are done.
return 1;

View file

@ -13,11 +13,23 @@ these comments are transferred directly into the auto-generated
`reStructuredText <http://docutils.sourceforge.net/rst.html>`_
(reST) document's summary section.
.. tip:: You can embed directives and roles within ``##``-stylized comments
.. tip:: You can embed directives and roles within ``##``-stylized comments.
A script's logging information has to be documented manually as minimally
shown below. Note that references may not always be possible (e.g.
anonymous filter functions) and a script may not need to document
each of "columns", "event", "filter" depending on exactly what it's doing.
**Logging Stream ID:** :bro:enum:`Example::EXAMPLE`
:Columns: :bro:type:`Example::Info`
:Event: :bro:id:`Example::log_example`
:Filter: ``example-filter``
uses :bro:id:`Example::filter_func` to determine whether to
exclude the ``ts`` field
:Author: Jon Siwek <jsiwek@ncsa.illinois.edu>
:Imports: :doc:`notice`
:Imports: :doc:`notice </policy/notice>`
Summary
~~~~~~~
@ -49,13 +61,20 @@ Types
goes here.
:bro:type:`Example::ComplexRecord`: :bro:type:`record` general documentation for a type "ComplexRecord" goes here
:bro:type:`Example::Info`: :bro:type:`record` An example record to be used with a logging stream.
====================================================== ==========================================================
Events
######
============================================== ==========================
================================================= =============================================================
:bro:id:`Example::an_event`: :bro:type:`event` Summarize "an_event" here.
============================================== ==========================
:bro:id:`Example::log_example`: :bro:type:`event` This is a declaration of an example event that can be used in
logging streams and is raised once for each log entry.
:bro:id:`bro_init`: :bro:type:`event`
================================================= =============================================================
Functions
#########
@ -65,9 +84,13 @@ Functions
Redefinitions
#############
================================================= ====================================
===================================================== ========================================
:bro:type:`Log::ID`: :bro:type:`enum`
:bro:type:`Example::SimpleEnum`: :bro:type:`enum` document the "SimpleEnum" redef here
================================================= ====================================
:bro:type:`Example::SimpleRecord`: :bro:type:`record` document the record extension redef here
===================================================== ========================================
Namespaces
~~~~~~~~~~
@ -180,6 +203,18 @@ Types
general documentation for a type "ComplexRecord" goes here
.. bro:type:: Example::Info
:Type: :bro:type:`record`
ts: :bro:type:`time` :bro:attr:`&log`
uid: :bro:type:`string` :bro:attr:`&log`
status: :bro:type:`count` :bro:attr:`&log` :bro:attr:`&optional`
An example record to be used with a logging stream.
Events
~~~~~~
.. bro:id:: Example::an_event
@ -191,6 +226,17 @@ Events
:param name: describe the argument here
.. bro:id:: Example::log_example
:Type: :bro:type:`event` (rec: :bro:type:`Example::Info`)
This is a declaration of an example event that can be used in
logging streams and is raised once for each log entry.
.. bro:id:: bro_init
:Type: :bro:type:`event` ()
Functions
~~~~~~~~~
.. bro:id:: Example::a_function
@ -213,6 +259,12 @@ Functions
Redefinitions
~~~~~~~~~~~~~
:bro:type:`Log::ID`
:Type: :bro:type:`enum`
.. bro:enum:: Example::EXAMPLE Log::ID
:bro:type:`Example::SimpleEnum`
:Type: :bro:type:`enum`
@ -227,6 +279,16 @@ Redefinitions
document the "SimpleEnum" redef here
.. bro:type:: Example::SimpleRecord
:Type: :bro:type:`record`
field_ext: :bro:type:`string` :bro:attr:`&optional`
document the extending field here
(or here)
document the record extension redef here
Port Analysis
-------------
:ref:`More Information <common_port_analysis_doc>`
@ -234,8 +296,8 @@ Port Analysis
SSL::
[ports={
563/tcp,
443/tcp
443/tcp,
562/tcp
}]
Packet Filter
@ -244,8 +306,8 @@ Packet Filter
Filters added::
[nntps] = tcp port 563,
[ssl] = tcp port 443
[ssl] = tcp port 443,
[nntps] = tcp port 562
Private Interface
-----------------
@ -260,8 +322,8 @@ State Variables
::
{
563/tcp,
443/tcp
443/tcp,
562/tcp
}
Types
@ -276,6 +338,10 @@ Types
Functions
~~~~~~~~~
.. bro:id:: Example::filter_func
:Type: :bro:type:`function` (rec: :bro:type:`Example::Info`) : :bro:type:`bool`
.. bro:id:: Example::function_without_proto
:Type: :bro:type:`function` (tag: :bro:type:`string`) : :bro:type:`string`

View file

@ -0,0 +1,99 @@
.. Automatically generated. Do not edit.
autogen-reST-record-add.bro
===========================
:download:`Original Source File <autogen-reST-record-add.bro>`
Overview
--------
Summary
~~~~~~~
State Variables
###############
===================================== =
:bro:id:`a`: :bro:type:`my_record`
:bro:id:`b`: :bro:type:`super_record`
===================================== =
Types
#####
============================================ =
:bro:type:`my_record`: :bro:type:`record`
:bro:type:`super_record`: :bro:type:`record`
============================================ =
Functions
#########
===================================== =
:bro:id:`test_func`: :bro:type:`func`
===================================== =
Redefinitions
#############
========================================= =
:bro:type:`my_record`: :bro:type:`record`
========================================= =
Public Interface
----------------
State Variables
~~~~~~~~~~~~~~~
.. bro:id:: a
:Type: :bro:type:`my_record`
:Default:
::
{
field1=<uninitialized>
field2=<uninitialized>
field3=<uninitialized>
}
.. bro:id:: b
:Type: :bro:type:`super_record`
:Default:
::
{
rec=[field1=<uninitialized>, field2=<uninitialized>, field3=<uninitialized>]
}
Types
~~~~~
.. bro:type:: my_record
:Type: :bro:type:`record`
field1: :bro:type:`bool`
field2: :bro:type:`string`
.. bro:type:: super_record
:Type: :bro:type:`record`
rec: :bro:type:`my_record`
Functions
~~~~~~~~~
.. bro:id:: test_func
:Type: :bro:type:`function` () : :bro:type:`void`
Redefinitions
~~~~~~~~~~~~~
.. bro:type:: my_record
:Type: :bro:type:`record`
field3: :bro:type:`count` :bro:attr:`&optional`

View file

@ -24,7 +24,7 @@ Types
============================================ ==========================================================================
:bro:type:`TypeAlias`: :bro:type:`bool` This is just an alias for a builtin type ``bool``.
:bro:type:`OtherTypeAlias`: :bro:type:`bool` We decided that creating alias "chains" might now be so useful to document
:bro:type:`OtherTypeAlias`: :bro:type:`bool` We decided that creating alias "chains" might not be so useful to document
so this type just creates a cross reference to ``bool``.
============================================ ==========================================================================
@ -56,6 +56,6 @@ Types
:Type: :bro:type:`bool`
We decided that creating alias "chains" might now be so useful to document
We decided that creating alias "chains" might not be so useful to document
so this type just creates a cross reference to ``bool``.

View file

@ -0,0 +1,15 @@
#!/usr/bin/python
import sys
import re
# MutableVal derivatives (e.g. sets/tables) don't always generate the same
# ordering in the reST documentation, so just don't bother diffing
# the places where example.bro uses them.
RE1 = "\d*/tcp"
RE2 = "tcp port \d*"
for line in sys.stdin.readlines():
if re.search(RE1, line) is None and re.search(RE2, line) is None:
print line

View file

@ -13,5 +13,6 @@ LOCALE=C
PATH=%(testbase)s/../../build/src:%(testbase)s/../../aux/btest:%(default_path)s
TEST_DIFF_CANONIFIER=%(testbase)s/Scripts/diff-canonifier
TRACES=%(testbase)s/Traces
SCRIPTS=%(testbase)s/Scripts
DIST=%(testbase)s/../..
BUILD=%(testbase)s/../../build

View file

@ -0,0 +1 @@
@TEST-EXEC: cd $BUILD && make restdoc

View file

@ -1,2 +1,2 @@
@TEST-EXEC: bro --doc-scripts $DIST/doc/example.bro
@TEST-EXEC: bro --doc-scripts $DIST/doc/scripts/example.bro
@TEST-EXEC: btest-diff example.rst

View file

@ -0,0 +1,32 @@
# @TEST-EXEC: bro --doc-scripts %INPUT
# @TEST-EXEC: btest-diff autogen-reST-record-add.rst
# When in doc mode, bro will clone declared types (see add_type() in Var.cc)
# in order to keep track of the identifier name associated with the new type.
# This test makes sure that the cloning is done in a way that's compatible
# with adding fields to a record type -- we want to be sure that cloning
# a record that contains other record fields will correctly see field
# additions to those contained-records.
type my_record: record {
field1: bool;
field2: string;
};
type super_record: record {
rec: my_record;
};
redef record my_record += {
field3: count &optional;
};
global a: my_record;
global b: super_record;
function test_func()
{
a?$field3;
b$rec?$field3;
}

View file

@ -4,7 +4,7 @@
## This is just an alias for a builtin type ``bool``.
type TypeAlias: bool;
## We decided that creating alias "chains" might now be so useful to document
## We decided that creating alias "chains" might not be so useful to document
## so this type just creates a cross reference to ``bool``.
type OtherTypeAlias: TypeAlias;