Commit graph

9059 commits

Author SHA1 Message Date
Jon Siwek
09438d2d06 Merge remote-tracking branch 'origin/topic/johanna/conn-tunnels-empty'
* origin/topic/johanna/conn-tunnels-empty:
  Make tunnel_parents in conn.log optional.

BIT-1889 #merged
2018-01-16 14:56:15 -06:00
Jon Siwek
4bd066ef4f Updating submodule(s).
[nomail]
2018-01-16 13:01:19 -06:00
Johanna Amann
6f9524e082 Make tunnel_parents in conn.log optional.
This makes conn.logs a bit prettier (and smaller) because all lines that
do not use a tunnel will now have a "-" instead of the "(empty)" for
tunnel_parents.
2018-01-12 13:46:00 -08:00
Johanna Amann
bdb7f62335 Correct include-path in x509Common.h
The file used a relative path which leads to problems if it is included
by an external Plugin. This commit changes this to an absolute path,
like everywhere else.
2018-01-12 10:58:20 -08:00
Johanna Amann
cd345caadb Merge remote-tracking branch 'origin/master' into topic/johanna/websockets 2018-01-12 10:27:23 -08:00
Johanna Amann
cc738544eb Merge remote-tracking branch 'remotes/origin/topic/jsiwek/prealloc-ports'
* remotes/origin/topic/jsiwek/prealloc-ports:
  Clean up PortManager class, mark PortVal ctors deprecated.
  Add BRO_DEPRECATED macro.
  Preallocate all possible PortVals.

BIT-1881 #merged
2018-01-12 10:06:30 -08:00
Johanna Amann
762e3c9f12 Merge remote-tracking branch 'origin/topic/seth/add-file-lookup-functions'
* origin/topic/seth/add-file-lookup-functions:
  Functions for retrieving files by their id.

 BIT-1887 #merged
2018-01-12 09:46:24 -08:00
Jeffrey Bencteux
6d497ea8b0 add test for smb1_com_transaction_request event changes 2018-01-12 13:00:05 +01:00
Jeffrey Bencteux
00be145b1b fix setup field handling in smb1_com_transaction_request messages
This field is an array of 16 bit words and was parsed as an array of
32 bit words. Moreover, one can not assume the format is going to be a
16 bits opcode followed by a 16 bit file ID, the content of the setup
field is different according to its first 16 bits word that defines
the subcommand code. See MS-CIFS section 2.2.4.33.1 :

Setup (variable): An array of two-byte words that provides transaction
context to the server. The size and content of the array are specific
to individual subcommands.
2018-01-12 13:00:01 +01:00
Jeffrey Bencteux
f7a8726ffc fix smb1_com_transaction* messages 2018-01-12 12:59:41 +01:00
Devin Trejo
e529268b0a Format print nfs units tests to improve output readability. Add unit
tests for new NFS events -- nfs_proc_symlink, nfs_proc_link,
nfs_proc_sattr.
2018-01-11 17:02:47 -05:00
Devin Trejo
f6cf4a41ff Add unit tests for new MOUNT events -- mount_proc_mnt, mount_proc_umnt,
mount_proc_umnt_all, mount_proc_not_implemented.
2018-01-11 17:00:15 -05:00
Julien Wallior
f165ff943e Expand smb2 unit test. 2018-01-11 15:33:31 -05:00
Julien Wallior
a76e50d2e1 Change smb2_create_response event arguments to single response struct.
Added disposition and create_action fields.
2018-01-11 15:33:21 -05:00
Julien Wallior
f041c97cdc Change smb2_create_request event arguments to single request struct.
Added disposition and create_option fields.
2018-01-11 15:32:55 -05:00
Jon Siwek
89ef5a7b7d Merge remote-tracking branch 'origin/fastpath'
* origin/fastpath:
  Fix a test that fails in some environments
2018-01-11 11:48:00 -06:00
Devin Trejo
99e411f621 Add mount_proc_null, mount_proc_mnt, mount_proc_umnt,
mount_proc_umnt_all, mount_proc_not_implemented, mount_reply_status.
2018-01-11 11:57:17 -05:00
Daniel Thayer
f7c115a47a Fix a test that fails in some environments
The "coverage/init-default.test" will always fail if there is a
path component named "build" anywhere before the bro install
directory (for example, if the tests are run from home dir of a user
named "build").  Fixed this by making a regex more specific so that
it matches the correct lines in loaded_scripts.log.
2018-01-10 13:23:30 -06:00
Devin Trejo
1981f2406a Bug fix: nfs3_writeargs didn't properly return filehandle. 2018-01-10 12:16:45 -05:00
Devin Trejo
b93691b15c Add nfs_proc_symlink, nfs_proc_link, nfs_proc_sattr. 2018-01-10 12:06:10 -05:00
Seth Hall
7cb6cf24a6 Functions for retrieving files by their id.
There are two new script level functions to query and lookup files
from the core by their IDs.  These are adding feature parity for
similarly named functions for files.  The function prototypes are
as follows:

  Files::file_exists(fuid: string): bool
  Files::lookup_File(fuid: string): fa_file
2018-01-09 12:16:17 -05:00
Valerio G
928e33a7b8 Add .btest scripts for dhck_ack and dhcp_discover messages verifying that new
options are correctly reported in dhcp.log records.
2018-01-08 21:55:24 +01:00
Valerio G
18499fd7d9 Extend DHCP protocol analyzer with new options.
Add the folowing option types:
- 55 Parameters Request List;
- 58 Renewal time;
- 59 Rebinding time;
- 61 Client Identifier;
- 82 Relay Agent Information.

Extend the following events with new parameters, specifically:
- dhcp_discover exports client identifier and parameters request list;
- dhcp_request exports client_identifier and parameters request list;
- dhcp_ack exports rebinding time, renewal time and list of suboptions value of
  dhcp relay agent information option;
- dhcp_inform exports parameters request list.

Add option type specific variables within the scope of DHCP module
(see src/analyzer/protocol/dhcp/types.bif).

Move protocol specific variables "dhcp_msg" and "dhcp_router_list" from scope Global to DHCP::
and adapt inet_net_var in src/NetVar.cc consequently.

Extend src/analyzer/protocols/dhcp/main.bro to handle the new events and to log
dhcp_ack, dhcp_request and dhcp_discover.

Modify scripts/policy/protocols/dhcp/known-devices-and-hostnames.bro to
include new events' variables.
2018-01-08 21:53:58 +01:00
Johanna Amann
8b28b73124 Add CVE ID for BIT-1856. 2018-01-02 09:12:09 +01:00
Andrew Woodford
f07fdc255f SSH protocol now assesses the packet length at an earlier stage within binpac. Stops SSH analyzer constantly raising binpac exceptions. Seems to be because a packet continues to go through binpac when empty and only calls the next packet when asked for more data and not on operations. 2017-12-21 16:34:26 +00:00
Jon Siwek
1c25df6f26 Merge remote-tracking branch 'origin/topic/seth/remove-dns-weird'
* origin/topic/seth/remove-dns-weird:
  Updated news.
  Remove some DNS weirds that caused volume and didn't help anyone.
2017-12-15 15:11:29 -06:00
Jon Siwek
4965a7e156 Fix typo in analyzer::Manager API docs 2017-12-13 14:24:54 -06:00
Seth Hall
f236dcdaa3 Updated news. 2017-12-13 15:09:52 -05:00
Seth Hall
ac955519b2 Remove some DNS weirds that caused volume and didn't help anyone.
These have been lingering for a while and they generally annoy
everyone because of the sheer volume.  They also don't really add
any useful information for debugging and they were generated differently
than most other weirds anyway (which was a little weird...).
2017-12-13 15:00:15 -05:00
Jon Siwek
d6d7f33f5c Clean up PortManager class, mark PortVal ctors deprecated.
Moved PortVal ctors back to public API, but deprecated, just in
case it helps give any external code a chance to adapt.
2017-12-12 11:35:55 -06:00
Jon Siwek
054c4a67c4 Add BRO_DEPRECATED macro. 2017-12-12 11:34:49 -06:00
Jon Siwek
1e4964de77 Preallocate all possible PortVals.
The performance benefit is small (maybe ~1% at most), however, it's a
trivial change without downsides.
2017-12-11 15:29:28 -06:00
Jon Siwek
895e7b06b1 Merge branch 'topic/jazoff/speedup-for'
* topic/jazoff/speedup-for:
  problem: for loops over empty tables are slow.

BIT-1878 #merged
2017-12-08 13:12:09 -06:00
Johanna Amann
03f98c7022 Fix recently introduced double free in OpenSSL code. 2017-12-07 14:47:56 -08:00
Justin Azoff
b730874ea1 problem: for loops over empty tables are slow.
This change doubles the performance of for loops over empty tables.

A bro binary that prints out this size shows for
testing/external/bro-testing/2009-M57-day11-18.trace, for loops are run
over tables of size:

11477 for size 0
8371 for size 1
1227 for size 3
239 for size 2
141 for size 6
57 for size 5
10 for size 4
5 for size 7
2 for size 13
2 for size 8
2 for size 11
1 for size 9

~53% of the for loops were across an empty table.  These loops come from
things like the for loop in the http script over c$http_state$pending

This change prevents the creation of an iteration cookie entirely if the
table is empty.

Using this test script:

    const scan_ports: table[port] of count = { };

    local x = 0;
    while ( x < 20000000 ) {
        for(p in scan_ports) {
        }
        ++x;
    }

$ time bro.orig -b ___bench.bro

real    0m10.732s
user    0m10.415s
sys     0m0.113s

$ time bro.nocookie -b ___bench.bro

real    0m4.694s
user    0m4.464s
sys     0m0.086s
2017-12-07 17:15:14 -05:00
Johanna Amann
6a93abea32 Adjust coding style & fix test failures.
I am still not 100% convinced that there is no memory leak hidden
somwehere...

This also makes everything compile with OpenSSL 1.0.2 for me.
2017-12-07 14:02:27 -08:00
Jeffrey Bencteux
bbe89a79a4 add smb1_transaction2_secondary_request event
parse and expose SMB_COM_TRANSACTION2_SECONDARY (0x33) message to
script level. See MS-CIFS section 2.2.4.47.1.
2017-12-07 10:01:25 +01:00
Jeffrey Bencteux
046c7bc481 add smb1_transaction_secondary_request event
expose SMB_COM_TRANSACTION_SECONDARY (0x26) message to script
language. See MS-CIFS section 2.2.4.34.1.
2017-12-07 10:01:20 +01:00
Jeffrey Bencteux
bd72710e3b add parameters and data to smb1_transaction_request/response messages
expose SMB_Data.Trans_Parameters and SMB_Data.Trans_Data fields of
SMB_COM_TRANSACTION (0x25) message type. See MS-CIFS section
2.2.4.33.1.

These fields are exposed to the script level as Bro strings. Note that
this commit also expose a new event smb1_transaction_response.
2017-12-07 10:01:13 +01:00
Jeffrey Bencteux
f2c3a9495d add SMB_Parameters.Words to smb1_transaction2_request event
expose the fields contained in SMB_Parameters.Words of the
SMB_COM_TRANSACTION2 (0x32) message to the script language. See
MS-CIFS section 2.2.46.1.
2017-12-07 10:00:55 +01:00
Hilko Bengen
ea4cf7dbe9 Adapt most of the X509 support to OpenSSL 1.1 2017-12-06 00:07:49 +01:00
Johanna Amann
bfe94641cf Fix another gcc7 warning.
Warning is:

/home/johanna/bro/master/src/Type.cc: In member function 'virtual bool IndexType::DoUnserialize(UnserialInfo*)':
/home/johanna/bro/master/src/Type.cc:548:60: warning: enum constant in boolean context [-Wint-in-bool-context]
  indices = (TypeList*) BroType::Unserialize(info, TYPE_LIST);
                                                            ^
/home/johanna/bro/master/src/Type.cc: In member function 'virtual bool FuncType::DoUnserialize(UnserialInfo*)':
/home/johanna/bro/master/src/Type.cc:868:61: warning: enum constant in boolean context [-Wint-in-bool-context]
  args = (RecordType*) BroType::Unserialize(info, TYPE_RECORD);
                                                             ^
/home/johanna/bro/master/src/Type.cc:872:62: warning: enum constant in boolean context [-Wint-in-bool-context]
  arg_types = (TypeList*) BroType::Unserialize(info, TYPE_LIST);

This one is a really nice catch in my opinion. GCC is completely correct
- the 2nd argument to Unserialize is a bool. This means that all these
calls always evaluate to Unserialize(info, true). Which is equivalent
with the default, so I just removed the type from the call.

This was probably caused by someone thinking of BroVal::Unserialize,
which needs the type as the 2nd argument.
2017-12-05 11:30:06 -08:00
Johanna Amann
7c03f4dec0 Fix compile warnings raised by gcc7.
/home/johanna/bro/master/src/Sessions.cc: In member function 'void NetSessions::DoNextPacket(double, const Packet*, const IP_Hdr*, const EncapsulationStack*)':
/home/johanna/bro/master/src/Sessions.cc:343:18: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
  if ( ip_hdr_len > len )
       ~~~~~~~~~~~^~~~~
/home/johanna/bro/master/src/Sessions.cc:349:18: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
  if ( ip_hdr_len > caplen )
       ~~~~~~~~~~~^~~~~~~~
/home/johanna/bro/master/src/Sessions.cc:399:20: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
    if ( ip_hdr_len > len )
         ~~~~~~~~~~~^~~~~

In file included from /usr/include/machine/endian.h:6:0,
                 from /usr/include/sys/types.h:44,
                 from /usr/include/unistd.h:37,
                 from /home/johanna/bro/master/src/Anon.cc:2:
/home/johanna/bro/master/src/Anon.cc: In member function 'virtual ipaddr32_t AnonymizeIPAddr_Seq::anonymize(ipaddr32_t)':
/home/johanna/bro/master/src/Anon.cc:85:18: warning: operation on '((AnonymizeIPAddr_Seq*)this)->AnonymizeIPAddr_Seq::seq' may be undefined [-Wsequence-point]
  return htonl(seq++);
                  ^
2017-12-05 10:27:23 -08:00
Jon Siwek
8827ece34a Updating submodule(s).
[nomail]
2017-12-05 11:19:57 -06:00
Jon Siwek
ecfacbe96e Fix documentation for ReassemblerStats. 2017-12-05 11:01:09 -06:00
Jon Siwek
9d2c41a4ff Merge remote-tracking branch 'origin/topic/johanna/socks-password'
* origin/topic/johanna/socks-password:
  Do not log SOCKS passwords by default.

BIT-1791 #merged
2017-12-02 09:53:56 -06:00
Johanna Amann
8ba5c03538 Do not log SOCKS passwords by default.
This introduces a new option, SOCKS::default_capture_password which can
be used to specify if Socks passwords are logged by default

Like fot FTP/HTTP, this option is set to false by default.

Addresses BIT-1791
2017-12-01 14:36:57 -08:00
Johanna Amann
94f55532f2 Make parsing of ServerKeyExchange work for D(TLS) < 1.2.
Now we only parse the SignatureAndHashalgorithm field in cases where it
is present. This change also takes care to respect SCTs, which do
include the SignatureAndHashalgorithm in their digitally-signed struct,
even when used in protocol versions that do not have the
SignatureAndHashalgorithm in the protocols digitally-signed struct.

I also added tests to make sure this does indeed work with TLS 1.1 - it
turns out that so far we did not have a single TLS 1.1 pcap.
2017-11-30 12:20:45 -08:00
Johanna Amann
fdf8717588 Merge remote-tracking branch 'origin/master' into topic/johanna/ssl_signature_details 2017-11-30 08:46:54 -08:00
Johanna Amann
1f6954ca3e Add missing ; in SSL binpac parser.
Binpac for some reason also works without it.

Found by Luke Valenta.
2017-11-30 08:42:55 -08:00