Commit graph

946 commits

Author SHA1 Message Date
Robin Sommer
7131feefbc Working on DataSeries support.
- The option to use integers insteads of double was ignored.

   - Renaming script-level options to remove the ds_ prefix.

   - Log rotation didn't work.

   - A set of simple unit tests.
2012-04-09 17:46:56 -07:00
Robin Sommer
952b6b293a Merging in DataSeries support from topic/gilbert/logging.
I copied the code over manually, no merging, because (1) it needed to
be adapted to the new threading API, and (2) there's more stuff in the
branch that I haven't ported yet.

The DS output generally seems to work, but it has seen no further
testing yet.

Not unit tests yet either.
2012-04-03 22:14:56 -07:00
Bernhard Amann
fd70560017 Merge remote-tracking branch 'origin/topic/robin/log-threads' into topic/bernhard/input-threads 2012-03-30 11:00:51 -07:00
Bernhard Amann
1170a87769 make benchmark reader hartbeat inverval aware
fix small memleak on tablereader destruction
make timespread better configureable
2012-03-30 09:19:01 -07:00
Bernhard Amann
b47620e501 add a couple more configuration options 2012-03-30 09:18:44 -07:00
Bernhard Amann
ed5374b6d7 and another option for the benchmark reader (constant addition of lines) 2012-03-30 09:17:35 -07:00
Bernhard Amann
579a10d060 make benchmark reader more configureable 2012-03-30 09:17:25 -07:00
Bernhard Amann
355b85fcd7 most of the stuff we should need for benchmarking. next: search memory leaks, after 1.5million simulated
inputs we are leaking about 1Gb of ram...
2012-03-30 09:08:08 -07:00
Seth Hall
47f58e6340 Cluster support for the metrics framework returns and all tests work again. 2012-03-28 14:00:59 -04:00
Seth Hall
6600e62ea3 Ported scripts using metrics framework and added a new smtp script.
- New script measures a couple of aspects of SMTP traffic.

- Existing metrics scripts had a small amount of work done
  to make them work with changes to metrics framework.
2012-03-28 11:39:27 -04:00
Robin Sommer
d7c9471818 Extending queue statistics. 2012-03-23 15:57:25 -07:00
Seth Hall
5904043e85 Modifications to the metrics framework API and new features.
- Metrics:ID enum has been replaced with strings.

- Uniqueness can now be measured with the Metrics::add_unique function.

- Filters can change the index value with the $normalize_func field.
2012-03-22 16:46:37 -04:00
Bernhard Amann
3286d013c9 forgot two files. 2012-03-16 23:45:10 -07:00
Bernhard Amann
57ffe1be77 completely change interface again.
compiles, not really tested.

basic test works 70% of the time, coredumps in the other 30 - but was not easy to debug on a first glance (most interestingly the crash happens in the logging framework - I wonder how that works).
Other tests are not adjusted to the new interface yet.
2012-03-15 18:41:51 -07:00
Bernhard Amann
b4e6971aab Add regular debugging output for interesting operations (stream/filter operations) to input framework (this was way overdue) 2012-03-14 14:45:53 -07:00
Bernhard Amann
92555badd4 cleanup, more sanity tests, a little bit more documentation 2012-03-11 20:43:26 -07:00
Robin Sommer
8eaf40ec18 Reverting accidental commit.
Thanks, Seth!
2012-03-08 20:24:12 -08:00
Robin Sommer
51009b73bc Finetuning communication CPU usage. 2012-03-08 18:13:17 -08:00
Bernhard Amann
417542f283 Merge branch 'topic/bernhard/log-threads' into topic/bernhard/input-threads
Seems to work -- all test pass.
But there are thread-safety issues at the moment, because the constructors of IPAddr and IPPrefix are not thread-safe, but needed by workers.

Conflicts:
	src/logging/Manager.cc
2012-02-27 22:59:08 -08:00
Bernhard Amann
7e5f733826 raw input reader for seth, which can simply read a file into string-events given a line separator. 2012-02-22 09:44:45 -08:00
Bernhard Amann
91943c2655 * rework script interface, add autostart stream flag that starts up a stream automatically when first filter has been added ( probably the most common use case )
* change internal reader interface again
* remove some quite embarassing bugs that must have been in the interface for rather long
* add different read methods to script & internal interface (like normal, streaming, etc). Not implemented in ascii reader yet.
2012-02-16 15:03:20 -08:00
Bernhard Amann
84883348ec interface documentation.
to a big part stolen from the logging framework
2012-02-16 11:27:10 -08:00
Seth Hall
430cd9b146 Initial rework of packet filter framework.
- Large rework on packet filter framework to make many things easier.
   - Removed the PacketFilter::all_packets variable because it was confusing.
   - New variable (PacketFilter::enable_auto_protocol_capture_filters) to re-enable the old filtering model of only sniffing ports for analyzed protocols.
   - In progress plugin model for adding filtering mechanisms.
   - New default single item for capture_filters = { ["default"] = PacketFilter::default_capture_filter };
   - Mechanism and helper functions to "shunt" traffic with filters.

- Created the Protocols framework to assist with reworking how base protocol scripts are registered with DPD and other things.
   - Protocols framework creates BPF filters for registered analyzers. (if using PacketFilter framework in that mode).
2012-02-16 11:14:57 -05:00
Robin Sommer
7458ebf385 Checkpoint after pass. 2012-02-15 13:07:08 -08:00
Bernhard Amann
88233efb2c It works. Even including all unit tests.
But: there are still a few places where I am sure that there are race conditions & memory leaks & I do not really like the current interface & I have to add a few more messages between the front and backend.

But - it works :)
2012-02-13 22:29:55 -08:00
Bernhard Amann
4e868d282d Merge branch 'topic/bernhard/log-threads' into topic/bernhard/input-threads 2012-02-13 02:37:02 -08:00
Jon Siwek
2e2f8f5d06 Merge branch 'master' into topic/v6-addr
Conflicts:
	src/bro.bif
2012-02-06 11:06:55 -06:00
Jon Siwek
b3f1f45082 Remove --enable-brov6 flag, IPv6 now supported by default.
Internally, all BROv6 preprocessor switches were removed and
addr/subnet representations wrapped in the new IPAddr/IPPrefix classes.

Some script-layer changes of note:

- dns_AAAA_reply event signature changed: the string representation
  of an IPv6 addr is easily derived from the addr value, it doesn't
  need to be another parameter.  This event also now generated directly
  by the DNS analyzer instead of being "faked" into a dns_A_reply event.

- removed addr_to_count BIF.  It used to return the host-order
  count representation of IPv4 addresses only.  To make it more
  generic, we might later add a BIF to return a vector of counts
  in order to support IPv6.

- changed the result of enclosing addr variables in vertical pipes
  (e.g. |my_addr|) to return the bit-width of the address type which
  is 128 for IPv6 and 32 for IPv4.  It used to function the same
  way as addr_to_count mentioned above.

- remove bro_has_ipv6 BIF
2012-02-03 16:46:58 -06:00
Seth Hall
600d015dab One more very minor change I forgot to commit. 2012-02-03 16:27:51 -05:00
Seth Hall
2cd88ee4f6 Merge remote-tracking branch 'origin/topic/bernhard/software'
* origin/topic/bernhard/software:
  change software framework interface again. At the moment everything should worl.
  start reworking interface of software framework. working apart from detect-webapps.bro, which direcly manipulates a no longer available interface...
  after talking to seth - change host_a field in record back to host.
  forgotten policy files.
  Software framework stores ports for server software.
2012-02-03 16:17:04 -05:00
Bernhard Amann
eacdffff90 Merge remote-tracking branch 'origin/master' into topic/bernhard/software
Conflicts:
	scripts/base/frameworks/software/main.bro
	scripts/policy/protocols/ftp/software.bro
2012-01-20 12:51:58 -08:00
Bernhard Amann
d476e9f3f4 Merge remote-tracking branch 'origin/master' into topic/bernhard/input 2012-01-12 11:07:12 -08:00
Robin Sommer
3d2dc5f5fc Merge remote-tracking branch 'origin/topic/script-reference'
* origin/topic/script-reference: (50 commits)
  A few updates for the FAQ.
  Fixing some doc warnings.
  Forgot to add protocol identifier support for TLS 1.2
  Finished SSL & syslog autodocs.
  Adding the draft SSL extension type next_protocol_negotiation.
  Fix some documentation errors.
  Tweaks.
  A set of script-reference polishing.
  fixed a couple typos in comments
  Add summary documentation to bif files.
  Add ssl and syslog script documentation
  Add Conn and DNS protocol script documentation. (fixes #731)
  Small updates to the default local.bro.
  Documentation updates for HTTP & IRC scripts.
  SSH&FTP Documentation updates.
  Fixing a warning from the documentation generation.
  This completes framework documentation package 4.
  Minor notice documentation tweaks.
  Fix some malformed Broxygen xref roles.
  Minor doc tweaks to init-bare.bro.
  ...

Conflicts:
	aux/broccoli
	aux/broctl
	src/bro.bif
	src/strings.bif

Includes:

    - Updated baselines for autodoc tests.
    - Now excluding stats.bro from external texts, it's not stable.
2012-01-10 14:00:44 -08:00
Bernhard Amann
3138f015ce Merge remote-tracking branch 'origin/master' into topic/bernhard/input 2012-01-10 11:33:26 -08:00
Jon Siwek
f921a4d5db Change SFTP/SCP log rotators to use 4-digit year in filenames (fixes #745). 2012-01-10 09:38:17 -06:00
Seth Hall
4de670a10e Fixing some doc warnings. 2012-01-10 01:30:55 -05:00
Robin Sommer
e5a42e8a85 Merge branch 'topic/script-reference' of ssh://git.bro-ids.org/bro into topic/script-reference
Conflicts:
	scripts/base/frameworks/notice/actions/pp-alarms.bro
	scripts/base/frameworks/notice/main.bro
	src/bro.bif
	src/const.bif
	src/event.bif
	src/strings.bif
	src/types.bif
2012-01-09 18:07:43 -08:00
Jon Siwek
69a0206a82 Merge branch 'master' into topic/script-reference
Conflicts:
	scripts/base/frameworks/notice/actions/pp-alarms.bro
	scripts/base/frameworks/notice/main.bro
	scripts/base/init-bare.bro
	src/event.bif
2012-01-09 15:49:14 -06:00
Seth Hall
1afe8b011c Fixing a warning from the documentation generation. 2012-01-06 16:50:20 -05:00
Seth Hall
f603d0121b This completes framework documentation package 4.
- Closes ticket #709
2012-01-06 16:36:22 -05:00
Jon Siwek
e7cf347288 Add SFTP log postprocessor that transfers logs to remote hosts.
Addresses #737
2012-01-06 14:58:17 -06:00
Robin Sommer
7646ef1aed Merge remote-tracking branch 'origin/master' into topic/script-reference
Conflicts:
	scripts/base/frameworks/notice/actions/pp-alarms.bro
	scripts/base/frameworks/notice/main.bro
	scripts/base/init-bare.bro
	src/event.bif
2012-01-06 12:11:49 -08:00
Bernhard Amann
5bef49d625 Merge remote-tracking branch 'origin/master' into topic/bernhard/input
Conflicts:
	src/parse.y
2012-01-05 01:11:13 -08:00
Seth Hall
f8ec98625d Merge remote-tracking branch 'origin/topic/robin/pp-alarms'
* origin/topic/robin/pp-alarms:
  The silliest, tiniest little whitespace fixes.
  Update missing in last commit to this branch.
  Adding test for alarm mail.
  Tuning the pretty-printed alarms output.
2012-01-04 13:41:28 -05:00
Seth Hall
adfbed8e56 The silliest, tiniest little whitespace fixes. 2012-01-04 13:37:07 -05:00
Robin Sommer
5e9153d7d6 Merge remote-tracking branch 'origin/topic/bernhard/notice-proto'
* origin/topic/bernhard/notice-proto:
  log protocol in notices.

Conflicts:
	scripts/base/frameworks/notice/main.bro

Closes #718.
2012-01-03 14:52:07 -08:00
Jon Siwek
275420dd29 Minor notice documentation tweaks. 2011-12-19 16:28:30 -06:00
Jon Siwek
a4117016e9 Merge branch 'master' into topic/script-reference
Conflicts:
	aux/broccoli
	aux/broctl
	scripts/base/frameworks/notice/main.bro
	src/event.bif
2011-12-19 16:17:58 -06:00
Bernhard Amann
a14ec02d3b change empty field defenition like in logging framework 2011-12-19 12:43:25 -08:00
Bernhard Amann
59967d40ac Merge remote-tracking branch 'origin/master' into topic/bernhard/input
Conflicts:
	src/LogMgr.cc
	src/LogMgr.h
2011-12-19 12:36:53 -08:00