* origin/topic/jsiwek/improve_comm_loop:
Add a simple FD_Set wrapper/helper class.
Fix Pipe copy/assignment to make a copy of flags.
Fix possible abort on writing to a full pipe.
Remove timeouts from remote communication loop.
SerializationFormat::EndWrite now transfers ownership of the buffer
to the caller instead of doing a memcpy.
ChunkedIO::Chunk is no longer a POD type, hopefully the ctor/dtor
make it easier to manage its associated memory. It also now
tracks how to deallocate its buffer (i.e. delete vs. free).
A bunch of infrastructure work to move IOSource, IOSourceRegistry (now
iosource::Manager) and PktSrc/PktDumper code into iosource/, and over
to a plugin structure.
Other IOSources aren't touched yet, they are still in src/*.
It compiles and does something with a small trace, but that's all I've
tested so far. There are quite certainly a number of problems left, as
well as various TODOs and cleanup; and nothing's cast in stone yet.
Will continue to work on this.
This is essentially the code from the dynamic-plugin branch except for
some pieces that I have split out into separate, earlier commits.
I'm going to updatre things in this branch going forward.
Replaced some with InternalWarning or InternalAnalyzerError, the later
being a new method which signals the analyzer to not process further
input. Some usages I just removed if they didn't make sense or clearly
couldn't happen. Also did some minor refactors of related code while
reviewing/exploring ways to get rid of InternalError usages.
Also, for TCP content file write failures there's a new event:
"contents_file_write_failure".
Synchronization of state between connecting peers now skips over
identifiers that aren't initialized with a value yet. If they're
assigned a value later, that will be synchronized like usual.
When using --enable-debug, values keep track of the last identifier
to which they were bound by storing a ref'd ID pointer. This could
lead to some circular dependencies in which an ID is never reclaimed
because the Val is bound to the ID and the ID is bound to the Val, with
both holding references to each other.
There might be more cases where this feature of --enable-debug caused
a leak, but it showed up in particular when running the
core.leaks.remote unit test due to the internal
SendID("peer_description") call during the handshake between remote
processes. Other tests showed the send_id() BIF leaked more generally.
Tracking the ID last bound to a Val through just the identifier string
instead of a ref'd ID pointer fixes the leak.
If a log filter attempts to write to a path for which a writer is
already instantiated due to remote logging, it will re-use the writer
as long as the fields of the filter and writer are compatible, else
the filter path will be auto-adjusted to not conflict with existing
writer's. Conflicts between two local filters are still always
auto-adjusted even if field types agree (since they could still
be semantically different).
Addresses #842.
frameworks.
There were a number of cases that weren't thread-safe. In particular,
we don't use std::string anymore for anything that's passed between
threads (but instead plain old const char*, with manual memmory
managmenet).
This is still a check-point commit, I'll do more testing.
* robin/topic/writer-info:
Extending the log writer DoInit() API.
Reworking log writer API to make it easier to pass additional information to a writer's initialization method.
Conflicts:
src/logging/WriterBackend.cc
src/logging/WriterBackend.h
src/logging/WriterFrontend.cc
* origin/fastpath:
Fix inconsistencies in random number generation.
Updating input framework unit tests.
Add front-end name to InitMessage from WriterFrontend to Backend.
Small tweak to make test complete quicker.
Drain events before terminating log/thread managers.
Fix strict-aliasing warning in RemoteSerializer.cc (fixes#834).
Fix typos in event documentation
Fix typos in NEWS for Bro 2.1 beta
* origin/topic/jsiwek/ipv6-comm:
Enable Bro to communicate with peers over non-global IPv6 addresses.
Add unit tests for Broccoli SSL and Broccoli IPv6 connectivity.
Remove AI_ADDRCONFIG getaddrinfo hints flag for listening sockets.
Undo communication protocol version bump.
Add support to Bro for connecting with peers over IPv6.
Closes#820.
Conflicts:
src/bro.bif
This usually requires specifying an additional zone identifier
(see RFC 4007). The connect() and listen() BIFs have been
changed to accept this zone identifier as an argument.
Because, according to RFC 3493, that will cause getaddrinfo to
overlook the ::1 loopback if there's not some other interface with
a global IPv6 address. The rationale being that the flag helps
prevent unnecessary AAAA lookups, but since I set AI_NUMERICHOST,
lookups aren't going to happen anyway.
Also update the IPv6 Bro communication test to get it to work more
reliably.
Looks like it wasn't necessary because no message between remote peers
needed to be changed to support IPv6, just messages between Bro
parent and child processes were changed.
- Communication::listen_ipv6 needs to be redef'd to true in order
for IPv6 listening sockets to be opened.
- Added Communication::listen_retry option as an interval at which
to retry binding to socket addresses that were already in use.
- Added some explicit baselines to check in the istate.events
and istate.events-ssl tests -- the SSL test was incorrectly
passing because it compared two empty files. (The files being
empty because "http/base" was given as an argument to Bro which
it couldn't handle because that script doesn't exist anymore).
Also renaming --enable-perftools to --enable-perftool-debug to
indicate that the switch is only relevant for debugging the heap. It's
not needed to pick up tcmalloc for better performance.
--with-perftools can still (and always) be used to give a hint where
to find the libraries.
With the threading, using tcmalloc improves memory usage on FreeBSD
significantly when running on a trace. If it fixes the live problems,
remains to be seen ...
