Commit graph

6667 commits

Author SHA1 Message Date
Robin Sommer
2be985433c Test-suite passes.
All tests pass with one exception: some Broxygen tests are broken
because dpd_config doesn't exist anymore. Need to update the mechanism
for auto-documenting well-known ports.
2013-03-26 15:40:23 -07:00
Jon Siwek
621fe51c82 FileAnalysis: fix access of invalidated std::map iterator. 2013-03-26 15:52:12 -05:00
Jon Siwek
497496ec83 FileAnalysis: replace script-layer SMTP file analysis.
Notable differences:

- Removed SMTP::MD5 notice.

- Removed ability to specify mime entity excerpt length per mime-type.
2013-03-26 15:48:52 -05:00
Robin Sommer
eef4858692 Fixes for non-OSX. 2013-03-26 13:08:03 -07:00
Robin Sommer
af1809aaa3 First prototype of new analyzer framework.
This is a larger internal change that moves the analyzer
infrastructure to a more flexible model where the available analyzers
don't need to be hardcoded at compile time anymore. While currently
they actually still are, this will in the future enable external
analyzer plugins. For now, it does already add the capability to
dynamically enable/disable analyzers from script-land, replacing the
old Analyzer::Available() methods.

There are three major parts going into this:

    - A new plugin infrastructure in src/plugin. This is independent
      of analyzers and will eventually support plugins for other parts
      of Bro as well (think: readers and writers). The goal is that
      plugins can be alternatively compiled in statically or loadead
      dynamically at runtime from a shared library. While the latter
      isn't there yet, there'll be almost no code change for a plugin
      to make it dynamic later (hopefully :)

    - New analyzer infrastructure in src/analyzer. I've moved a number
      of analyzer-related classes here, including Analyzer and DPM;
      the latter now renamed to Analyzer::Manager. More will move here
      later. Currently, there's only one plugin here, which provides
      *all* existing analyzers. We can modularize this further in the
      future (or not).

    - A new script interface in base/framework/analyzer. I think that
      this will eventually replace the dpm framework, but for now
      that's still there as well, though some parts have moved over.

I've also remove the dpd_config table; ports are now configured via
the analyzer framework. For exmaple, for SSH:

    const ports = { 22/tcp } &redef;

    event bro_init() &priority=5
        {
        ...
        Analyzer::register_for_ports(Analyzer::ANALYZER_SSH, ports);
        }

As you can see, the old ANALYZER_SSH constants have more into an enum
in the Analyzer namespace.

This is all hardly tested right now, and not everything works yet.
There's also a lot more cleanup to do (moving more classes around;
removing no longer used functionality; documenting script and C++
interfaces; regression tests). But it seems to generally work with a
small trace at least.

The debug stream "dpm" shows more about the loaded/enabled analyzers.

A new option -N lists loaded plugins and what they provide (including
those compiled in statically; i.e., right now it outputs all the
analyzers).

This is all not cast-in-stone yet, for some things we need to see if
they make sense this way. Feedback welcome.
2013-03-26 11:05:38 -07:00
Jon Siwek
84a0c2fdac FileAnalysis: file handles now set from events.
Versus from synchronous function calls, which doesn't work well because
the function call can see a script-layer state that doesn't reflect
the state as it will be in terms of the event/network stream.
2013-03-25 15:37:58 -05:00
Bernhard Amann
b05eef6541 Merge remote-tracking branch 'origin/master' into topic/bernhard/hyperloglog
Conflicts:
	src/bro.bif
2013-03-25 08:39:52 -07:00
Yun Zheng Hu
9a88dc500a Added reverse() function to strings.bif.
Closes #969.
2013-03-23 08:39:04 -07:00
Jon Siwek
71f0e2d276 FileAnalysis: replace script-layer http file analysis.
Other misc:

- Remove HTTP::MD5 notice.

- Add "last_active" field to FileAnalysis::Info record.

- Replace "conn_uids", "conn_ids" fields in FileAnalysis::Info record
  with just a "conns" fields containing full connection records.

- The http-methods unit test is failing now, but I think it will be
  fixed once I change the file handle callback mechanism to use events
  instead.
2013-03-22 16:14:06 -05:00
Robin Sommer
4255205026 Merge remote-tracking branch 'origin/topic/jsiwek/table-init-record-idx'
Closes #660.

* origin/topic/jsiwek/table-init-record-idx:
  Fix record constructors in table initializer indices.  Addresses #660.
2013-03-22 12:11:29 -07:00
Robin Sommer
8dca2dd971 Merge remote-tracking branch 'origin/topic/seth/q-in-q'
Closes #641.

* origin/topic/seth/q-in-q:
  Add support for 802.1ah (Q-in-Q).
2013-03-22 12:02:21 -07:00
Seth Hall
4e99d3a606 Add support for 802.1ah (Q-in-Q). 2013-03-22 12:38:43 -04:00
Jon Siwek
1ef7465e30 FileAnalysis: add more unit tests. 2013-03-20 12:47:45 -05:00
Jon Siwek
661677d452 FileAnalysis: separating IRC/FTP data analyzers.
It simplifies the file handle string callbacks.
2013-03-20 11:12:06 -05:00
Jon Siwek
59ed5c75f1 FileAnalysis: add unit tests covering current protocol integration.
And had to make various fixes/refinements after scrutinizing results.
2013-03-19 15:50:05 -05:00
Bernhard Amann
fed914252c and close only fds that are currently open (the logging framework really
did not like that :) )
2013-03-18 22:11:13 -07:00
Bernhard Amann
8875953751 A bunch of more changes for the raw reader
* send end_of_data event for all kind of streams
* send process_finished event containing exit code of child process for executed programs
* move raw-tests to separate directory
* expose name of input stream to readers
* better handling of some error cases in raw reader
* new force_kill option for raw reader which SIGKILLs progesses on exit

The ordering of events how they arrive in the main loop is a bit peculiar at the moment.
The process_finished event arrives in scriptland before all of the other events, even though
it should be sent last. I have not yet fully figured that out.
2013-03-18 21:49:16 -07:00
Bernhard Amann
f1c91f02ce Merge remote-tracking branch 'origin/master' into topic/bernhard/input-update 2013-03-18 19:16:47 -07:00
Jon Siwek
294570ec2e Merge branch 'master' into topic/jsiwek/file-analysis 2013-03-18 11:48:05 -05:00
Jon Siwek
550c3c477d FileAnalysis: integrate w/ SMTP analyzer.
More generally w/ MIME_Mail messages, which POP3 analyzer also uses.
2013-03-18 11:30:59 -05:00
Bernhard Amann
4aab9921be fix gcc compile warning in Benchmark reader 2013-03-18 09:04:18 -07:00
Bernhard Amann
873d0549bf fix gcc compile warning in base64 encoder 2013-03-18 09:01:03 -07:00
Robin Sommer
c39223e226 Fixing potential leak in DNS error case. From Vlad.
Closes #967.
2013-03-17 17:37:06 -07:00
Robin Sommer
6865f0438a Renaming Base64Decoder to Base64Converter.
It nows encodes and decodes.
2013-03-17 13:16:46 -07:00
Robin Sommer
d58a02aa01 Merge remote-tracking branch 'origin/topic/bernhard/base64'
* origin/topic/bernhard/base64:
  and re-enable caching of extracted certs
  and add bae64 bif tests.
  re-unify classes
  and modernize script.
  add base64-encode functionality and bif.

Closes #965.
2013-03-17 13:00:52 -07:00
Bernhard Amann
6fef99ee03 make reading from stdout and stderr simultaneously work.
Needs a few test-cases - but seems ok...
2013-03-16 09:54:48 -07:00
Robin Sommer
d11bd56b5d Changing semantics of thread stop methods.
PrepareStop() is now SignalStop() and just signals a thread that it
should terminate. After that's called, WaitForStop() (formerly Stop())
wait for it to actually finish processing.

When stopping writers during operation, we now no longer wait for them
to finish.
2013-03-15 17:57:58 -07:00
Bernhard Amann
3aeec7ec14 allow sending data to stdin of child process 2013-03-15 15:47:20 -07:00
Bernhard Amann
fc42c71dfa Streaming reads from external commands work without blocking anything. 2013-03-15 13:58:41 -07:00
Bernhard Amann
f2d67b5829 replace popen with fork and exec.
Note for future: eof only gets sent when the parent closes its in-pipe
after forking.
2013-03-15 13:32:28 -07:00
Bernhard Amann
5dd07e023d change raw reader to use basic c io instead of fdstream encapsulation class.
preparation for future changes.
2013-03-14 16:14:13 -07:00
Jon Siwek
31590fbb9b FileAnalysis: improve HTTP integration.
Seems to be working with various examples I tested, now including
multipart/byteranges.
2013-03-14 17:05:05 -05:00
Robin Sommer
38e1dc9ca4 Support for cleaning up threads that have terminated.
Once a BasicThread leaves its run() method, a thread is now marked for
cleaning up, and the ThreadMgr will soon join it to release the OS
resources.

Also, adding a function Log::remove_stream() that remove a logging
stream, stopping all writer threads that are associated with it.

Note, however, that removing a *filter* from a stream still doesn't
clean up any threads. The problem is that because of the output paths
potentially being created dynamically it's unclear if the writer
thread will still be needed in the future. We could add clean writers
up with timeouts, but that doesn't sound great either. So for now, the
only way to sure clean up logging threads is to remove the entire
stream.

Also note that cleanup doesn't work with input threads yet, which
don't seem to terminate (at least in the case I tried).
2013-03-14 14:59:05 -07:00
Jon Siwek
e0f3713912 FileAnalysis: change file handle -> file id mapping process.
They're now actually directly related via a hash function that will
produce the same results among different instances in a cluster.
2013-03-14 14:08:26 -05:00
Jon Siwek
637fe69cf9 FileAnalysis: buffer input that can't get unique file handle immediately
A retry happens on every new input and also periodically based on a
timer.  If a file handle is returned at those times, the input is
forwarded for analysis, else it keeps retrying until a timeout
threshold.
2013-03-14 10:57:16 -05:00
Seth Hall
b1f1b64dde Checkpoint 2013-03-14 11:19:39 -04:00
Jon Siwek
878dfff2f2 FileAnalysis: decentralize unique file handle generator callbacks.
The framework now cycles through callbacks based on a table indexed
by analyzer tags, or the special case of service strings if a given
analyzer is overloaded for multiple protocols (FTP/IRC data).  This
lets each protocol script bundle implement the callback locally and
reduces the FAF's external dependencies.
2013-03-13 10:48:26 -05:00
Jon Siwek
bb3228e8f6 FileAnalysis: small tweak to file handle generation.
For files that go over a single connection, add connection start time
to handle, so the file id will always differ even if the same connection
parameters are later used to transfer a file (same one or different).
2013-03-12 16:06:06 -05:00
Jon Siwek
3dd513e26e FileAnalysis: move unique file handle string generation to script-layer
And add minimal integration with HTTP analyzer.
2013-03-12 13:44:31 -05:00
Bernhard Amann
a5161783ef and add bae64 bif tests. 2013-03-12 09:33:49 -07:00
Bernhard Amann
a1896fde90 Merge remote-tracking branch 'origin/master' into topic/bernhard/base64 2013-03-12 09:28:07 -07:00
Bernhard Amann
2b28c3a578 re-unify classes 2013-03-12 09:27:59 -07:00
Bernhard Amann
fdc8de7596 add sqlite tests and fix small vector/set escaping bugs 2013-03-11 14:22:35 -07:00
Bernhard Amann
a251a1c39a fix small bug with vectors and sets.
On a first glance - this kind of seems to work. On mac-os you need
a newer than the system-installed sqlite - the hanging problem only
occurs with that one...
2013-03-11 13:10:56 -07:00
Bernhard Amann
5d12765886 make work with newer AsciiFormatter. 2013-03-11 12:01:49 -07:00
Bernhard Amann
8cb91de93a Merge remote-tracking branch 'origin/master' into topic/bernhard/sqlite
Conflicts:
	src/threading/AsciiFormatter.cc
2013-03-11 11:47:10 -07:00
Jon Siwek
1f6cac9b6d Merge branch 'master' into topic/jsiwek/file-analysis 2013-03-11 13:20:45 -05:00
Jon Siwek
90ca2b87c4 Add check for truncated link frames. Addresses #962.
Patch provided by jbaines, modified with a more descriptive Weird name.
2013-03-11 11:58:54 -05:00
Jon Siwek
8d5434ef2d Fix large memory allocation in IP fragment reassembly. Addresses #961.
Patch by jbaines modified slightly to return earlier so that the
problem packet can't cause any state change in the FragReassembler.
2013-03-11 10:54:51 -05:00
Robin Sommer
f830ed3edf s/bro-ids.org/bro.org/g 2013-03-07 19:33:04 -08:00