* topic/robin/log-threads: (42 commits)
Two more tweaks to reliably terminate when reading from trace.
This could be fixing the memory problems finally.
Fix compile errors due to now-explicit IPAddr ctors and global IPFamily enum.
Switching log buffer size back to normal
Teaching cmake to always link in tcmalloc if it finds it.
Extending queue statistics.
Small fixes and tweaks.
Don't assert during shutdown.
Reverting accidental commit.
Finetuning communication CPU usage.
Adding new leak tests involving remote logging.
Removing some no longer needed checks.
Fixing problem logging remotely when local logging was turned off.
Preventing busy looping when no threads have been spawned.
Prevent manager from busy looping.
Adding missing includes needed on FreeBSD.
Updating submodule(s).
Updating submodule(s).
A number of bugfixes for the recent threading updates.
Making exchange of addresses between threads thread-safe.
...
The FragReassembler expire_timer was left uninitialized until after
the first fragment is added, but since the atomic fragment is also
the last, the reassembler thought expire_timer needed to be deleted.
This fix just initializes expire_timer before the first fragment is
added.
I copied the code over manually, no merging, because (1) it needed to
be adapted to the new threading API, and (2) there's more stuff in the
branch that I haven't ported yet.
The DS output generally seems to work, but it has seen no further
testing yet.
Not unit tests yet either.
There is still a tiny where I have not yet found where the delete could be missing.
For big table imports the memory footprint is significant nevertheless -- with tables of > 200000 entries, memory consumption can apparently reach in excess of 1.5Gb - and on a first glance this seems legitimate.
(The reason for this is probably that we use several hash tables to keep the performance impact small).
Also renaming --enable-perftools to --enable-perftool-debug to
indicate that the switch is only relevant for debugging the heap. It's
not needed to pick up tcmalloc for better performance.
--with-perftools can still (and always) be used to give a hint where
to find the libraries.
With the threading, using tcmalloc improves memory usage on FreeBSD
significantly when running on a trace. If it fixes the live problems,
remains to be seen ...
- For RH0 headers with non-zero segments left, a "routing0_segleft"
flow_weird event is raised (with a destination indicating the last
address in the routing header), and an "rh0_segleft" event can also
be handled if the other contents of the packet header are of interest.
No further analysis is done as the complexity required to correctly
identify destination endpoints of connections doesn't seem worth it
as RH0 has been deprecated by RFC 5095.
- For RH0 headers without any segments left, a "routing0_header"
flow_weird event is raised, but further analysis still occurs
as normal.
- New script measures a couple of aspects of SMTP traffic.
- Existing metrics scripts had a small amount of work done
to make them work with changes to metrics framework.
