Zeek scripts located on separate filesystems, but sharing the same inode
number leads to scripts not being loaded. The reason is that a `ScannedFile`
is only identified by `st_ino` which is not enough to uniquely identify a
file in a system.
This problem may be hit when `ZEEKPATH` points to separate filesystems and
two script files happen have the same `st_ino` value - definitely not very
likely, but possibly very confusing when it happens.
The following test case creates two zeek scripts on separate filesystems.
As the filesystems are freshly created and of the same type, the files will
(tested a few times with xfs/ext4) have the same `st_ino` values.
#!/bin/bash
ZEEKDIR=${ZEEKDIR:-/home/awelzel/projects/zeek}
export ZEEKPATH=.:${ZEEKDIR}/build/scripts:${ZEEKDIR}/scripts
cat << EOF > hello.zeek
event zeek_init() {
print("Hello, once or twice?");
}
EOF
for i in 1 2 ; do
dd if=/dev/urandom of=img${i} count=16 bs=1M 2>/dev/null
sudo mkfs.xfs -q ./img${i}
mkdir -p mount${i}
sudo mount ./img${i} ./mount${i}
sudo cp hello.zeek ./mount${i}/hello.zeek
done
ls ./mount*/*zeek
stat -c "%n: device=%d inode=%i" ./mount*/hello.zeek
${ZEEKDIR}/build/src/zeek -b ./mount1/hello.zeek ./mount2/hello.zeek
# Cleanup
for i in 1 2 ; do
sudo umount ./mount${i}
rm -rfv ./img${i} ./mount${i}
rm -rfv hello.zeek
done
Before this patch, `Hello, once or twice?` is printed only once,
afterwards twice:
$ sh testcase.sh
[sudo] password for awelzel:
./mount1/hello.zeek ./mount2/hello.zeek
./mount1/hello.zeek: device=1794 inode=6915
./mount2/hello.zeek: device=1795 inode=6915
Hello, once or twice?
Hello, once or twice?
The Zeek code base has very inconsistent #includes. Many sources
included a few headers, and those headers included other headers, and
in the end, nearly everything is included everywhere, so missing
#includes were never noticed. Another side effect was a lot of header
bloat which slows down the build.
First step to fix it: in each source file, its own header should be
included first to verify that each header's includes are correct, and
none is missing.
After adding the missing #includes, I replaced lots of #includes
inside headers with class forward declarations. In most headers,
object pointers are never referenced, so declaring the function
prototypes with forward-declared classes is just fine.
This patch speeds up the build by 19%, because each compilation unit
gets smaller. Here are the "time" numbers for a fresh build (with a
warm page cache but without ccache):
Before this patch:
3144.94user 161.63system 3:02.87elapsed 1808%CPU (0avgtext+0avgdata 2168608maxresident)k
760inputs+12008400outputs (1511major+57747204minor)pagefaults 0swaps
After this patch:
2565.17user 141.83system 2:25.46elapsed 1860%CPU (0avgtext+0avgdata 1489076maxresident)k
72576inputs+9130920outputs (1667major+49400430minor)pagefaults 0swaps
More aspects of the cluster configuration to get fleshed out later,
but a basic cluster like one would use for a live deployment
can now be instantiated and run under supervision. The new
clusterized-pcap-processing supervisor mode is also not done yet.
The full process hierarchy isn't set up yet, but these changes
help prepare by doing two things:
- Add a -j option to enable supervisor-mode. Currently, just a single
"stem" process gets forked early on to be used as the basis for
further forking into real cluster nodes.
- Separates the parsing of command-line options from their consumption.
i.e. need to parse whether we're in -j supervisor-mode before
modifying any global state since that would taint the "stem" process.
The new intermediate structure containing the parsed options may
also serve as a way to pass configuration info from "stem" to its
descendent cluster node processes.
This commit removed functions/events that have been deprecated in Bro
2.6. It also removes the detection code that checks if the old
communication framework is used (since all the functions that are
checked were removed).
Addresses parts of GH-243
I reworked this a bit:
- Moved the globals into a new Pcap::* namespace, and renamed them
slightly.
- Moved the definitions of the globals into pcap/const.bif.
- Also moved the existing 'snaplen' into Pcap::* and removed
SnapLen() from the PktSrc API (it's really a pcap thing).
- Likewise moved the existing functions precompile_pcap_filter,
install_pcap_filter, and pcap_error, into Pcap::*.
- Did some more refactoring for the pcap code.
* 'master' of https://github.com/knielander/bro:
Refactored patch (removed options, less ambiguous name)
Allow Bro to run in fanout mode.
Allow libpcap buffer size to be set manually.
Allow Bro to run in fanout mode.
Allowed libpcap buffer size to be set via configuration.
We could bring this back, now derived from PktSrc (though strickly
speaking it's of course not *packets). But not sure if we want that,
as the input framework seems the better place to host it. Then it
would turns into a reader.
A bunch of infrastructure work to move IOSource, IOSourceRegistry (now
iosource::Manager) and PktSrc/PktDumper code into iosource/, and over
to a plugin structure.
Other IOSources aren't touched yet, they are still in src/*.
It compiles and does something with a small trace, but that's all I've
tested so far. There are quite certainly a number of problems left, as
well as various TODOs and cleanup; and nothing's cast in stone yet.
Will continue to work on this.
This is essentially the code from the dynamic-plugin branch except for
some pieces that I have split out into separate, earlier commits.
I'm going to updatre things in this branch going forward.
It was getting too bloated and allocated memory in ways that were
difficult to understand how to manage. Separated out primarily in to
new find_file() and open_file()/open_package() functions.
Also renamed other util functions for path-related things.
Any added prefixes are now used *after* all input files have been
parsed to look for a prefixed, flattened version of the input file
somewhere in BROPATH and, if found, load it.
For example, if "lcl" is in @prefixes, and site.bro is loaded, then
a file named "lcl.site.bro" that's in BROPATH would end up being
automatically loaded as well. Packages work similarly, e.g. loading
"protocols/http" means a file named "lcl.protocols.http.bro" in BROPATH
gets loaded automatically.
Also removing the -l command-line option as that can now be done at
the script-level.
A couple tests fail now that use -l. Leaving that until we have
script-level replacement.
filter. (Seth Hall and Robin Sommer)
- Merging in the patch from #264, which provides support for mixed
VLAN and MPLS traffic.
- Changing Bro's default filter from being built dynamically to being
a static "ip or not ip". To get the old behaviour back (i.e., the
dynamically built filter), redef "all_packets" to false.
- print-filter.bro now always prints the filter that Bro is actually
using, even if overriden from the command line.
Removing everything related to trace rewriting.
(I wasn't too careful in ensuring that I catch everything in the
scripts; Seth is working on those anyway.)
(Merging by cherry-picking the corresponding commit, as the branch was
accidentally made off of the logging stuff).
