Mirror/zeek
1
0
Fork 0
mirror of https://github.com/zeek/zeek.git synced 2025-10-06 16:48:19 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 2

Merge branch 'master' into topic/jsiwek/supervisor

Browse source
This commit is contained in:
Jon Siwek 2020-01-21 12:17:56 -08:00
commit 9c0d252c2b
32 changed files with 479 additions and 318 deletions
Download patch file Download diff file Expand all files Collapse all files

30
CHANGES
View file

@ -1,4 +1,34 @@
3.1.0-dev.389 | 2020-01-18 10:49:15 +0000
* GHI-595: Convert from nlohmann/json to RapidJSON for performance
reasons. (Tim Wojtulewicz, Corelight)
* Optimize json_escape_utf8() and expand its unit tests. (Tim
Wojtulewicz, Corelight)
* Convert type-checking macros to actual functions. (Tim
Wojtulewicz, Corelight)
* Use the list of files from clang-tidy when searching for unit
tests. (Tim Wojtulewicz, Corelight)
3.1.0-dev.383 | 2020-01-17 11:51:01 +0000
* Various code modernization cleanup. (Tim Wojtulewicz, Corelight)
3.1.0-dev.378 | 2020-01-16 13:18:13 +0000
* Handle invalid Base64 encodings in FTP ADAT analyzer (Jon Siwek,
Corelight)
3.1.0-dev.376 | 2020-01-14 09:45:45 -0800
* Fix warning when reading files from non-network sources (Seth Hall, Corelight)
If files are being read from non-network sources, there was a warning in
the SSL base scripts about missing the f$conns field.
3.1.0-dev.372 | 2020-01-13 12:10:42 +0000 3.1.0-dev.372 | 2020-01-13 12:10:42 +0000
* Fix method returning a reference to a temporary. Found by * Fix method returning a reference to a temporary. Found by

74
COPYING.3rdparty
View file

@ -21,31 +21,67 @@ a legal notice, here is a blessing:
============================================================================== ==============================================================================
%%% JSON for Modern C++ %%% RapidJSON - A fast JSON parser/generator for C++ with both SAX/DOM style API
============================================================================== ==============================================================================
Licensed under the MIT License <http://opensource.org/licenses/MIT>. Tencent is pleased to support the open source community by making RapidJSON available.
SPDX-License-Identifier: MIT
Copyright (c) 2013-2019 Niels Lohmann <http://nlohmann.me>.
Permission is hereby granted, free of charge, to any person obtaining a copy Copyright (C) 2015 THL A29 Limited, a Tencent company, and Milo Yip. All rights reserved.
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all If you have downloaded a copy of the RapidJSON binary from Tencent, please note that the RapidJSON binary is licensed under the MIT License.
copies or substantial portions of the Software. If you have downloaded a copy of the RapidJSON source code from Tencent, please note that RapidJSON source code is licensed under the MIT License, except for the third-party components listed below which are subject to different license terms. Your integration of RapidJSON into your own projects may require compliance with the MIT License, as well as the other licenses applicable to the third-party components included within RapidJSON. To avoid the problematic JSON license in your own projects, it's sufficient to exclude the bin/jsonchecker/ directory, as it's the only code under the JSON license.
A copy of the MIT License is included in this file.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR Other dependencies and licenses:
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE Open Source Software Licensed Under the BSD License:
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER --------------------------------------------------------------------
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE The msinttypes r29
SOFTWARE. Copyright (c) 2006-2013 Alexander Chemeris
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
* Neither the name of copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS AND CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Open Source Software Licensed Under the JSON License:
--------------------------------------------------------------------
json.org
Copyright (c) 2002 JSON.org
All Rights Reserved.
JSON_checker
Copyright (c) 2002 JSON.org
All Rights Reserved.
Terms of the JSON License:
---------------------------------------------------
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
The Software shall be used for Good, not Evil.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
Terms of the MIT License:
--------------------------------------------------------------------
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
============================================================================== ==============================================================================

7
NEWS
View file

@ -44,7 +44,7 @@ Changed Functionality
- The tcp_option event is now correctly raised. - The tcp_option event is now correctly raised.
'- The base scripts shipped with Zeek now use the new - The base scripts shipped with Zeek now use the new
``successful_connection_remove`` event instead of ``successful_connection_remove`` event instead of
``connection_state_remove`` where possible (when the logic doesn't ``connection_state_remove`` where possible (when the logic doesn't
pertain to unestablished TCP connections). There's a performance pertain to unestablished TCP connections). There's a performance
@ -62,6 +62,11 @@ Changed Functionality
``connection`` record field named "successful" to help indicate this ``connection`` record field named "successful" to help indicate this
new property of connections. new property of connections.
- The JSON output formatters now use the RapidJSON library. This
improves their performance considerably over the library that was
previously used. Output from the formatters remains nearly
identical.
Removed Functionality Removed Functionality
--------------------- ---------------------

2
VERSION
View file

@ -1 +1 @@
3.1.0-dev.372 3.1.0-dev.389

2
aux/btest

@ -1 +1 @@
Subproject commit 37f8619c8ac89e7e9bcda6c167e8972a05d8e286 Subproject commit f7bd3a78092a0807d69e9cc54ac41fbe1908cc21

2
scripts/base/protocols/ssl/files.zeek
View file

@ -92,7 +92,7 @@ event zeek_init() &priority=5
event file_sniff(f: fa_file, meta: fa_metadata) &priority=5 event file_sniff(f: fa_file, meta: fa_metadata) &priority=5
{ {
if ( |f$conns| != 1 ) if ( ! f?$conns || |f$conns| != 1 )
return; return;
if ( ! f?$info || ! f$info?$mime_type ) if ( ! f?$info || ! f$info?$mime_type )

2
src/3rdparty

@ -1 +1 @@
Subproject commit 2b3206b7add3472ea0736f2841473e11d506a85e Subproject commit fae32236391d9117bf996e75d56ebd01ef076bc2

58
src/BroString.cc
View file

@ -25,52 +25,40 @@ const int BroString::BRO_STRING_LITERAL;
// arg_final_NUL == 1; when str is a sequence of n bytes, make // arg_final_NUL == 1; when str is a sequence of n bytes, make
// arg_final_NUL == 0. // arg_final_NUL == 0.
BroString::BroString(int arg_final_NUL, byte_vec str, int arg_n) BroString::BroString(bool arg_final_NUL, byte_vec str, int arg_n)
{ {
b = str; b = str;
n = arg_n; n = arg_n;
final_NUL = arg_final_NUL; final_NUL = arg_final_NUL;
use_free_to_delete = 0; use_free_to_delete = false;
} }
BroString::BroString(const u_char* str, int arg_n, int add_NUL) BroString::BroString(const u_char* str, int arg_n, bool add_NUL) : BroString()
{ {
b = 0;
n = 0;
use_free_to_delete = 0;
Set(str, arg_n, add_NUL); Set(str, arg_n, add_NUL);
} }
BroString::BroString(const char* str) BroString::BroString(const char* str) : BroString()
{ {
b = 0;
n = 0;
use_free_to_delete = 0;
Set(str); Set(str);
} }
BroString::BroString(const string &str) BroString::BroString(const string &str) : BroString()
{ {
b = 0;
n = 0;
use_free_to_delete = 0;
Set(str); Set(str);
} }
BroString::BroString(const BroString& bs) BroString::BroString(const BroString& bs) : BroString()
{ {
b = 0;
n = 0;
use_free_to_delete = 0;
*this = bs; *this = bs;
} }
BroString::BroString() BroString::BroString()
{ {
b = 0; b = nullptr;
n = 0; n = 0;
final_NUL = 0; final_NUL = false;
use_free_to_delete = 0; use_free_to_delete = false;
} }
void BroString::Reset() void BroString::Reset()
@ -80,10 +68,10 @@ void BroString::Reset()
else else
delete [] b; delete [] b;
b = 0; b = nullptr;
n = 0; n = 0;
final_NUL = 0; final_NUL = false;
use_free_to_delete = 0; use_free_to_delete = false;
} }
const BroString& BroString::operator=(const BroString &bs) const BroString& BroString::operator=(const BroString &bs)
@ -95,8 +83,8 @@ const BroString& BroString::operator=(const BroString &bs)
memcpy(b, bs.b, n); memcpy(b, bs.b, n);
b[n] = '\0'; b[n] = '\0';
final_NUL = 1; final_NUL = true;
use_free_to_delete = 0; use_free_to_delete = false;
return *this; return *this;
} }
@ -122,7 +110,7 @@ void BroString::Adopt(byte_vec bytes, int len)
n = len - final_NUL; n = len - final_NUL;
} }
void BroString::Set(const u_char* str, int len, int add_NUL) void BroString::Set(const u_char* str, int len, bool add_NUL)
{ {
Reset(); Reset();
@ -134,7 +122,7 @@ void BroString::Set(const u_char* str, int len, int add_NUL)
if ( add_NUL ) if ( add_NUL )
b[n] = 0; b[n] = 0;
use_free_to_delete = 0; use_free_to_delete = false;
} }
void BroString::Set(const char* str) void BroString::Set(const char* str)
@ -144,8 +132,8 @@ void BroString::Set(const char* str)
n = strlen(str); n = strlen(str);
b = new u_char[n+1]; b = new u_char[n+1];
memcpy(b, str, n+1); memcpy(b, str, n+1);
final_NUL = 1; final_NUL = true;
use_free_to_delete = 0; use_free_to_delete = false;
} }
void BroString::Set(const string& str) void BroString::Set(const string& str)
@ -155,8 +143,8 @@ void BroString::Set(const string& str)
n = str.size(); n = str.size();
b = new u_char[n+1]; b = new u_char[n+1];
memcpy(b, str.c_str(), n+1); memcpy(b, str.c_str(), n+1);
final_NUL = 1; final_NUL = true;
use_free_to_delete = 0; use_free_to_delete = false;
} }
void BroString::Set(const BroString& str) void BroString::Set(const BroString& str)
@ -307,7 +295,7 @@ BroString::Vec* BroString::Split(const BroString::IdxVec& indices) const
{ {
unsigned int i; unsigned int i;
if ( indices.size() == 0 ) if ( indices.empty() )
return 0; return 0;
// Copy input, ensuring space for "0": // Copy input, ensuring space for "0":
@ -453,7 +441,7 @@ BroString* concatenate(std::vector<data_chunk_t>& v)
*b = '\0'; *b = '\0';
return new BroString(1, (byte_vec) data, len); return new BroString(true, (byte_vec) data, len);
} }
BroString* concatenate(BroString::CVec& v) BroString* concatenate(BroString::CVec& v)
@ -474,7 +462,7 @@ BroString* concatenate(BroString::CVec& v)
} }
*b = '\0'; *b = '\0';
return new BroString(1, (byte_vec) data, len); return new BroString(true, (byte_vec) data, len);
} }
BroString* concatenate(BroString::Vec& v) BroString* concatenate(BroString::Vec& v)

10
src/BroString.h
View file

@ -33,13 +33,13 @@ public:
typedef IdxVec::const_iterator IdxVecCIt; typedef IdxVec::const_iterator IdxVecCIt;
// Constructors creating internal copies of the data passed in. // Constructors creating internal copies of the data passed in.
BroString(const u_char* str, int arg_n, int add_NUL); BroString(const u_char* str, int arg_n, bool add_NUL);
explicit BroString(const char* str); explicit BroString(const char* str);
explicit BroString(const std::string& str); explicit BroString(const std::string& str);
BroString(const BroString& bs); BroString(const BroString& bs);
// Constructor that takes owernship of the vector passed in. // Constructor that takes owernship of the vector passed in.
BroString(int arg_final_NUL, byte_vec str, int arg_n); BroString(bool arg_final_NUL, byte_vec str, int arg_n);
BroString(); BroString();
~BroString() { Reset(); } ~BroString() { Reset(); }
@ -61,7 +61,7 @@ public:
// current contents, if any, and then set the string's // current contents, if any, and then set the string's
// contents to a copy of the string given by the arguments. // contents to a copy of the string given by the arguments.
// //
void Set(const u_char* str, int len, int add_NUL=1); void Set(const u_char* str, int len, bool add_NUL=true);
void Set(const char* str); void Set(const char* str);
void Set(const std::string& str); void Set(const std::string& str);
void Set(const BroString &str); void Set(const BroString &str);
@ -146,8 +146,8 @@ protected:
byte_vec b; byte_vec b;
int n; int n;
unsigned int final_NUL:1; // whether we have added a final NUL bool final_NUL; // whether we have added a final NUL
unsigned int use_free_to_delete:1; // free() vs. operator delete bool use_free_to_delete; // free() vs. operator delete
}; };
// A comparison class that sorts pointers to BroString's according to // A comparison class that sorts pointers to BroString's according to

10
src/CMakeLists.txt
View file

@ -416,15 +416,14 @@ install(DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/
) )
install(FILES install(FILES
${CMAKE_CURRENT_SOURCE_DIR}/3rdparty/json.hpp
${CMAKE_CURRENT_SOURCE_DIR}/3rdparty/sqlite3.h ${CMAKE_CURRENT_SOURCE_DIR}/3rdparty/sqlite3.h
DESTINATION include/zeek/3rdparty DESTINATION include/zeek/3rdparty
) )
install(FILES install(FILES
${CMAKE_CURRENT_SOURCE_DIR}/3rdparty/tsl-ordered-map/ordered_map.h ${CMAKE_CURRENT_SOURCE_DIR}/3rdparty/rapidjson/include/rapidjson/document.h
${CMAKE_CURRENT_SOURCE_DIR}/3rdparty/tsl-ordered-map/ordered_hash.h ${CMAKE_CURRENT_SOURCE_DIR}/3rdparty/rapidjson/include/rapidjson/writer.h
DESTINATION include/zeek/3rdparty/tsl-ordered-map DESTINATION include/zeek/3rdparty/rapidjson/include/rapidjson
) )
######################################################################## ########################################################################
@ -442,9 +441,8 @@ create_clang_tidy_target()
# Scan all .cc files for TEST_CASE macros and generate CTest targets. # Scan all .cc files for TEST_CASE macros and generate CTest targets.
if (ENABLE_ZEEK_UNIT_TESTS) if (ENABLE_ZEEK_UNIT_TESTS)
file(GLOB_RECURSE all_cc_files "*.cc")
set(test_cases "") set(test_cases "")
foreach (cc_file ${all_cc_files}) foreach (cc_file ${TIDY_SRCS})
file (STRINGS ${cc_file} test_case_lines REGEX "TEST_CASE") file (STRINGS ${cc_file} test_case_lines REGEX "TEST_CASE")
foreach (line ${test_case_lines}) foreach (line ${test_case_lines})
string(REGEX REPLACE "TEST_CASE\\(\"(.+)\"\\)" "\\1" test_case "${line}") string(REGEX REPLACE "TEST_CASE\\(\"(.+)\"\\)" "\\1" test_case "${line}")

18
src/Conn.cc
View file

@ -18,7 +18,7 @@
#include "analyzer/Manager.h" #include "analyzer/Manager.h"
void ConnectionTimer::Init(Connection* arg_conn, timer_func arg_timer, void ConnectionTimer::Init(Connection* arg_conn, timer_func arg_timer,
int arg_do_expire) bool arg_do_expire)
{ {
conn = arg_conn; conn = arg_conn;
timer = arg_timer; timer = arg_timer;
@ -87,8 +87,8 @@ Connection::Connection(NetSessions* s, const ConnIDKey& k, double t, const ConnI
vlan = pkt->vlan; vlan = pkt->vlan;
inner_vlan = pkt->inner_vlan; inner_vlan = pkt->inner_vlan;
conn_val = 0; conn_val = nullptr;
login_conn = 0; login_conn = nullptr;
is_active = 1; is_active = 1;
skip = 0; skip = 0;
@ -108,8 +108,8 @@ Connection::Connection(NetSessions* s, const ConnIDKey& k, double t, const ConnI
hist_seen = 0; hist_seen = 0;
history = ""; history = "";
root_analyzer = 0; root_analyzer = nullptr;
primary_PIA = 0; primary_PIA = nullptr;
++current_connections; ++current_connections;
++total_connections; ++total_connections;
@ -172,7 +172,7 @@ void Connection::CheckEncapsulation(const EncapsulationStack* arg_encap)
EncapsulationStack empty; EncapsulationStack empty;
Event(tunnel_changed, 0, empty.GetVectorVal()); Event(tunnel_changed, 0, empty.GetVectorVal());
delete encapsulation; delete encapsulation;
encapsulation = 0; encapsulation = nullptr;
} }
else if ( arg_encap ) else if ( arg_encap )
@ -222,7 +222,7 @@ void Connection::NextPacket(double t, int is_orig,
last_time = t; last_time = t;
current_timestamp = 0; current_timestamp = 0;
current_pkt = 0; current_pkt = nullptr;
} }
void Connection::SetLifetime(double lifetime) void Connection::SetLifetime(double lifetime)
@ -533,7 +533,7 @@ void Connection::Weird(const char* name, const char* addl)
reporter->Weird(this, name, addl ? addl : ""); reporter->Weird(this, name, addl ? addl : "");
} }
void Connection::AddTimer(timer_func timer, double t, int do_expire, void Connection::AddTimer(timer_func timer, double t, bool do_expire,
TimerType type) TimerType type)
{ {
if ( timers_canceled ) if ( timers_canceled )
@ -609,7 +609,7 @@ void Connection::FlipRoles()
orig_flow_label = tmp_flow; orig_flow_label = tmp_flow;
Unref(conn_val); Unref(conn_val);
conn_val = 0; conn_val = nullptr;
if ( root_analyzer ) if ( root_analyzer )
root_analyzer->FlipRoles(); root_analyzer->FlipRoles();

8
src/Conn.h
View file

@ -303,7 +303,7 @@ protected:
// Add the given timer to expire at time t. If do_expire // Add the given timer to expire at time t. If do_expire
// is true, then the timer is also evaluated when Bro terminates, // is true, then the timer is also evaluated when Bro terminates,
// otherwise not. // otherwise not.
void AddTimer(timer_func timer, double t, int do_expire, void AddTimer(timer_func timer, double t, bool do_expire,
TimerType type); TimerType type);
void RemoveTimer(Timer* t); void RemoveTimer(Timer* t);
@ -367,7 +367,7 @@ protected:
class ConnectionTimer : public Timer { class ConnectionTimer : public Timer {
public: public:
ConnectionTimer(Connection* arg_conn, timer_func arg_timer, ConnectionTimer(Connection* arg_conn, timer_func arg_timer,
double arg_t, int arg_do_expire, TimerType arg_type) double arg_t, bool arg_do_expire, TimerType arg_type)
: Timer(arg_t, arg_type) : Timer(arg_t, arg_type)
{ Init(arg_conn, arg_timer, arg_do_expire); } { Init(arg_conn, arg_timer, arg_do_expire); }
~ConnectionTimer() override; ~ConnectionTimer() override;
@ -377,11 +377,11 @@ public:
protected: protected:
ConnectionTimer() {} ConnectionTimer() {}
void Init(Connection* conn, timer_func timer, int do_expire); void Init(Connection* conn, timer_func timer, bool do_expire);
Connection* conn; Connection* conn;
timer_func timer; timer_func timer;
int do_expire; bool do_expire;
}; };
#define ADD_TIMER(timer, t, do_expire, type) \ #define ADD_TIMER(timer, t, do_expire, type) \

8
src/Timer.h
View file

@ -12,7 +12,7 @@ extern "C" {
} }
// If you add a timer here, adjust TimerNames in Timer.cc. // If you add a timer here, adjust TimerNames in Timer.cc.
enum TimerType { enum TimerType : uint8_t {
TIMER_BACKDOOR, TIMER_BACKDOOR,
TIMER_BREAKPOINT, TIMER_BREAKPOINT,
TIMER_CONN_DELETE, TIMER_CONN_DELETE,
@ -52,8 +52,7 @@ class ODesc;
class Timer : public PQ_Element { class Timer : public PQ_Element {
public: public:
Timer(double t, TimerType arg_type) : PQ_Element(t) Timer(double t, TimerType arg_type) : PQ_Element(t), type(arg_type) { }
{ type = (char) arg_type; }
~Timer() override { } ~Timer() override { }
TimerType Type() const { return (TimerType) type; } TimerType Type() const { return (TimerType) type; }
@ -67,8 +66,7 @@ public:
protected: protected:
Timer() {} Timer() {}
TimerType type;
unsigned int type:8;
}; };
class TimerMgr { class TimerMgr {

64
src/Type.h
View file

@ -698,53 +698,53 @@ extern BroType* init_type(Expr* init);
// Returns true if argument is an atomic type. // Returns true if argument is an atomic type.
bool is_atomic_type(const BroType* t); bool is_atomic_type(const BroType* t);
// True if the given type tag corresponds to an integral type.
#define IsIntegral(t) (t == TYPE_INT || t == TYPE_COUNT || t == TYPE_COUNTER)
// True if the given type tag corresponds to an arithmetic type.
#define IsArithmetic(t) (IsIntegral(t) || t == TYPE_DOUBLE)
// True if the given type tag corresponds to a boolean type.
#define IsBool(t) (t == TYPE_BOOL)
// True if the given type tag corresponds to an interval type.
#define IsInterval(t) (t == TYPE_INTERVAL)
// True if the given type tag corresponds to a record type.
#define IsRecord(t) (t == TYPE_RECORD || t == TYPE_UNION)
// True if the given type tag corresponds to a function type.
#define IsFunc(t) (t == TYPE_FUNC)
// True if the given type type is a vector.
#define IsVector(t) (t == TYPE_VECTOR)
// True if the given type type is a string.
#define IsString(t) (t == TYPE_STRING)
// True if the given type tag corresponds to type that can be assigned to. // True if the given type tag corresponds to type that can be assigned to.
extern int is_assignable(BroType* t); extern int is_assignable(BroType* t);
// True if the given type tag corresponds to an integral type.
inline bool IsIntegral(TypeTag t) { return (t == TYPE_INT || t == TYPE_COUNT || t == TYPE_COUNTER); }
// True if the given type tag corresponds to an arithmetic type.
inline bool IsArithmetic(TypeTag t) { return (IsIntegral(t) || t == TYPE_DOUBLE); }
// True if the given type tag corresponds to a boolean type.
inline bool IsBool(TypeTag t) { return (t == TYPE_BOOL); }
// True if the given type tag corresponds to an interval type.
inline bool IsInterval(TypeTag t) { return (t == TYPE_INTERVAL); }
// True if the given type tag corresponds to a record type.
inline bool IsRecord(TypeTag t) { return (t == TYPE_RECORD || t == TYPE_UNION); }
// True if the given type tag corresponds to a function type.
inline bool IsFunc(TypeTag t) { return (t == TYPE_FUNC); }
// True if the given type type is a vector.
inline bool IsVector(TypeTag t) { return (t == TYPE_VECTOR); }
// True if the given type type is a string.
inline bool IsString(TypeTag t) { return (t == TYPE_STRING); }
// True if the given type tag corresponds to the error type. // True if the given type tag corresponds to the error type.
#define IsErrorType(t) (t == TYPE_ERROR) inline bool IsErrorType(TypeTag t) { return (t == TYPE_ERROR); }
// True if both tags are integral types. // True if both tags are integral types.
#define BothIntegral(t1, t2) (IsIntegral(t1) && IsIntegral(t2)) inline bool BothIntegral(TypeTag t1, TypeTag t2) { return (IsIntegral(t1) && IsIntegral(t2)); }
// True if both tags are arithmetic types. // True if both tags are arithmetic types.
#define BothArithmetic(t1, t2) (IsArithmetic(t1) && IsArithmetic(t2)) inline bool BothArithmetic(TypeTag t1, TypeTag t2) { return (IsArithmetic(t1) && IsArithmetic(t2)); }
// True if either tags is an arithmetic type. // True if either tags is an arithmetic type.
#define EitherArithmetic(t1, t2) (IsArithmetic(t1) || IsArithmetic(t2)) inline bool EitherArithmetic(TypeTag t1, TypeTag t2) { return (IsArithmetic(t1) || IsArithmetic(t2)); }
// True if both tags are boolean types. // True if both tags are boolean types.
#define BothBool(t1, t2) (IsBool(t1) && IsBool(t2)) inline bool BothBool(TypeTag t1, TypeTag t2) { return (IsBool(t1) && IsBool(t2)); }
// True if both tags are interval types. // True if both tags are interval types.
#define BothInterval(t1, t2) (IsInterval(t1) && IsInterval(t2)) inline bool BothInterval(TypeTag t1, TypeTag t2) { return (IsInterval(t1) && IsInterval(t2)); }
// True if both tags are string types. // True if both tags are string types.
#define BothString(t1, t2) (IsString(t1) && IsString(t2)) inline bool BothString(TypeTag t1, TypeTag t2) { return (IsString(t1) && IsString(t2)); }
// True if either tag is the error type. // True if either tag is the error type.
#define EitherError(t1, t2) (IsErrorType(t1) || IsErrorType(t2)) inline bool EitherError(TypeTag t1, TypeTag t2) { return (IsErrorType(t1) || IsErrorType(t2)); }

156
src/Val.cc
View file

@ -27,20 +27,7 @@
#include "broker/Data.h" #include "broker/Data.h"
#include "3rdparty/json.hpp" #include "threading/formatters/JSON.h"
#include "3rdparty/tsl-ordered-map/ordered_map.h"
// Define a class for use with the json library that orders the keys in the same order that
// they were inserted. By default, the json library orders them alphabetically and we don't
// want it like that.
template<class Key, class T, class Ignore, class Allocator,
class Hash = std::hash<Key>, class KeyEqual = std::equal_to<Key>,
class AllocatorPair = typename std::allocator_traits<Allocator>::template rebind_alloc<std::pair<Key, T>>,
class ValueTypeContainer = std::vector<std::pair<Key, T>, AllocatorPair>>
using ordered_map = tsl::ordered_map<Key, T, Hash, KeyEqual, AllocatorPair, ValueTypeContainer>;
using ZeekJson = nlohmann::basic_json<ordered_map>;
Val::Val(Func* f) Val::Val(Func* f)
{ {
@ -433,46 +420,56 @@ TableVal* Val::GetRecordFields()
return rt->GetRecordFieldsVal(rv); return rt->GetRecordFieldsVal(rv);
} }
// This is a static method in this file to avoid including json.hpp in Val.h since it's huge. // This is a static method in this file to avoid including rapidjson's headers in Val.h because they're huge.
static ZeekJson BuildJSON(Val* val, bool only_loggable=false, RE_Matcher* re=nullptr) static void BuildJSON(threading::formatter::JSON::NullDoubleWriter& writer, Val* val, bool only_loggable=false, RE_Matcher* re=nullptr, const string& key="")
{ {
// If the value wasn't set, return a nullptr. This will get turned into a 'null' in the json output. if ( !key.empty() )
if ( ! val ) writer.Key(key);
return nullptr;
ZeekJson j; // If the value wasn't set, write a null into the stream and return.
if ( ! val )
{
writer.Null();
return;
}
rapidjson::Value j;
BroType* type = val->Type(); BroType* type = val->Type();
switch ( type->Tag() ) switch ( type->Tag() )
{ {
case TYPE_BOOL: case TYPE_BOOL:
j = val->AsBool(); writer.Bool(val->AsBool());
break; break;
case TYPE_INT: case TYPE_INT:
j = val->AsInt(); writer.Int64(val->AsInt());
break; break;
case TYPE_COUNT: case TYPE_COUNT:
j = val->AsCount(); writer.Uint64(val->AsCount());
break; break;
case TYPE_COUNTER: case TYPE_COUNTER:
j = val->AsCounter(); writer.Uint64(val->AsCounter());
break; break;
case TYPE_TIME: case TYPE_TIME:
j = val->AsTime(); writer.Double(val->AsTime());
break; break;
case TYPE_DOUBLE: case TYPE_DOUBLE:
j = val->AsDouble(); writer.Double(val->AsDouble());
break; break;
case TYPE_PORT: case TYPE_PORT:
{ {
auto* pval = val->AsPortVal(); auto* pval = val->AsPortVal();
j.emplace("port", pval->Port()); writer.StartObject();
j.emplace("proto", pval->Protocol()); writer.Key("port");
writer.Int64(pval->Port());
writer.Key("proto");
writer.String(pval->Protocol());
writer.EndObject();
break; break;
} }
@ -484,7 +481,7 @@ static ZeekJson BuildJSON(Val* val, bool only_loggable=false, RE_Matcher* re=nul
ODesc d; ODesc d;
d.SetStyle(RAW_STYLE); d.SetStyle(RAW_STYLE);
val->Describe(&d); val->Describe(&d);
j = string(reinterpret_cast<const char*>(d.Bytes()), d.Len()); writer.String(reinterpret_cast<const char*>(d.Bytes()), d.Len());
break; break;
} }
@ -496,7 +493,7 @@ static ZeekJson BuildJSON(Val* val, bool only_loggable=false, RE_Matcher* re=nul
ODesc d; ODesc d;
d.SetStyle(RAW_STYLE); d.SetStyle(RAW_STYLE);
val->Describe(&d); val->Describe(&d);
j = json_escape_utf8(string(reinterpret_cast<const char*>(d.Bytes()), d.Len())); writer.String(json_escape_utf8(string(reinterpret_cast<const char*>(d.Bytes()), d.Len())));
break; break;
} }
@ -506,9 +503,9 @@ static ZeekJson BuildJSON(Val* val, bool only_loggable=false, RE_Matcher* re=nul
auto* tval = val->AsTableVal(); auto* tval = val->AsTableVal();
if ( tval->Type()->IsSet() ) if ( tval->Type()->IsSet() )
j = ZeekJson::array(); writer.StartArray();
else else
j = ZeekJson::object(); writer.StartObject();
HashKey* k; HashKey* k;
TableEntryVal* entry; TableEntryVal* entry;
@ -524,102 +521,125 @@ static ZeekJson BuildJSON(Val* val, bool only_loggable=false, RE_Matcher* re=nul
else else
entry_key = lv->Ref(); entry_key = lv->Ref();
ZeekJson key_json = BuildJSON(entry_key, only_loggable, re);
if ( tval->Type()->IsSet() ) if ( tval->Type()->IsSet() )
j.emplace_back(std::move(key_json)); BuildJSON(writer, entry_key, only_loggable, re);
else else
{ {
Val* entry_value = entry->Value(); rapidjson::StringBuffer buffer;
threading::formatter::JSON::NullDoubleWriter key_writer(buffer);
BuildJSON(key_writer, entry_key, only_loggable, re);
string key_str = buffer.GetString();
string key_string; if ( key_str.length() >= 2 &&
if ( key_json.is_string() ) key_str[0] == '"' &&
key_string = key_json; key_str[key_str.length() - 1] == '"' )
else // Strip quotes.
key_string = key_json.dump(); key_str = key_str.substr(1, key_str.length() - 2);
j.emplace(key_string, BuildJSON(entry_value, only_loggable, re)); BuildJSON(writer, entry->Value(), only_loggable, re, key_str);
} }
Unref(entry_key); Unref(entry_key);
Unref(lv); Unref(lv);
} }
if ( tval->Type()->IsSet() )
writer.EndArray();
else
writer.EndObject();
break; break;
} }
case TYPE_RECORD: case TYPE_RECORD:
{ {
j = ZeekJson::object(); writer.StartObject();
auto* rval = val->AsRecordVal(); auto* rval = val->AsRecordVal();
auto rt = rval->Type()->AsRecordType(); auto rt = rval->Type()->AsRecordType();
for ( auto i = 0; i < rt->NumFields(); ++i ) for ( auto i = 0; i < rt->NumFields(); ++i )
{ {
auto field_name = rt->FieldName(i);
std::string key_string;
if ( re && re->MatchAnywhere(field_name) != 0 )
{
StringVal blank("");
StringVal fn_val(field_name);
auto key_val = fn_val.Substitute(re, &blank, 0)->AsStringVal();
key_string = key_val->ToStdString();
Unref(key_val);
}
else
key_string = field_name;
Val* value = rval->LookupWithDefault(i); Val* value = rval->LookupWithDefault(i);
if ( value && ( ! only_loggable || rt->FieldHasAttr(i, ATTR_LOG) ) ) if ( value && ( ! only_loggable || rt->FieldHasAttr(i, ATTR_LOG) ) )
j.emplace(key_string, BuildJSON(value, only_loggable, re)); {
string key_str;
auto field_name = rt->FieldName(i);
if ( re && re->MatchAnywhere(field_name) != 0 )
{
auto blank = make_intrusive<StringVal>("");
auto fn_val = make_intrusive<StringVal>(field_name);
auto key_val = fn_val->Substitute(re, blank.get(), 0)->AsStringVal();
key_str = key_val->ToStdString();
Unref(key_val);
}
else
key_str = field_name;
BuildJSON(writer, value, only_loggable, re, key_str);
}
Unref(value); Unref(value);
} }
writer.EndObject();
break; break;
} }
case TYPE_LIST: case TYPE_LIST:
{ {
j = ZeekJson::array(); writer.StartArray();
auto* lval = val->AsListVal(); auto* lval = val->AsListVal();
size_t size = lval->Length(); size_t size = lval->Length();
for (size_t i = 0; i < size; i++) for (size_t i = 0; i < size; i++)
j.push_back(BuildJSON(lval->Index(i), only_loggable, re)); BuildJSON(writer, lval->Index(i), only_loggable, re);
writer.EndArray();
break; break;
} }
case TYPE_VECTOR: case TYPE_VECTOR:
{ {
j = ZeekJson::array(); writer.StartArray();
auto* vval = val->AsVectorVal(); auto* vval = val->AsVectorVal();
size_t size = vval->SizeVal()->AsCount(); size_t size = vval->SizeVal()->AsCount();
for (size_t i = 0; i < size; i++) for (size_t i = 0; i < size; i++)
j.push_back(BuildJSON(vval->Lookup(i), only_loggable, re)); BuildJSON(writer, vval->Lookup(i), only_loggable, re);
writer.EndArray();
break; break;
} }
case TYPE_OPAQUE: case TYPE_OPAQUE:
{ {
writer.StartObject();
writer.Key("opaque_type");
auto* oval = val->AsOpaqueVal(); auto* oval = val->AsOpaqueVal();
j = { { "opaque_type", OpaqueMgr::mgr()->TypeID(oval) } }; writer.String(OpaqueMgr::mgr()->TypeID(oval));
writer.EndObject();
break; break;
} }
default: break; default:
writer.Null();
break;
} }
return j;
} }
StringVal* Val::ToJSON(bool only_loggable, RE_Matcher* re) StringVal* Val::ToJSON(bool only_loggable, RE_Matcher* re)
{ {
ZeekJson j = BuildJSON(this, only_loggable, re); rapidjson::StringBuffer buffer;
return new StringVal(j.dump()); threading::formatter::JSON::NullDoubleWriter writer(buffer);
BuildJSON(writer, this, only_loggable, re, "");
return new StringVal(buffer.GetString());
} }
IntervalVal::IntervalVal(double quantity, double units) : IntervalVal::IntervalVal(double quantity, double units) :

12
src/analyzer/protocol/ftp/FTP.cc
View file

@ -224,8 +224,16 @@ void FTP_ADAT_Analyzer::DeliverStream(int len, const u_char* data, bool orig)
// framing is supposed to be required for the initial context // framing is supposed to be required for the initial context
// token, but GSI doesn't do that and starts right in on a // token, but GSI doesn't do that and starts right in on a
// TLS/SSL handshake, so look for that to identify it. // TLS/SSL handshake, so look for that to identify it.
const u_char* msg = decoded_adat->Bytes(); const u_char* msg = nullptr;
int msg_len = decoded_adat->Len(); int msg_len = 0;
if ( decoded_adat )
{
msg = decoded_adat->Bytes();
msg_len = decoded_adat->Len();
}
else
Weird("ftp_adat_bad_first_token_encoding");
// Just check that it looks like a viable TLS/SSL handshake // Just check that it looks like a viable TLS/SSL handshake
// record from the first byte (content type of 0x16) and // record from the first byte (content type of 0x16) and

6
src/analyzer/protocol/ident/Ident.cc
View file

@ -15,7 +15,7 @@ using namespace analyzer::ident;
Ident_Analyzer::Ident_Analyzer(Connection* conn) Ident_Analyzer::Ident_Analyzer(Connection* conn)
: tcp::TCP_ApplicationAnalyzer("IDENT", conn) : tcp::TCP_ApplicationAnalyzer("IDENT", conn)
{ {
did_bad_reply = did_deliver = 0; did_bad_reply = did_deliver = false;
orig_ident = new tcp::ContentLine_Analyzer(conn, true, 1000); orig_ident = new tcp::ContentLine_Analyzer(conn, true, 1000);
resp_ident = new tcp::ContentLine_Analyzer(conn, false, 1000); resp_ident = new tcp::ContentLine_Analyzer(conn, false, 1000);
@ -89,7 +89,7 @@ void Ident_Analyzer::DeliverStream(int length, const u_char* data, bool is_orig)
val_mgr->GetPort(remote_port, TRANSPORT_TCP), val_mgr->GetPort(remote_port, TRANSPORT_TCP),
}); });
did_deliver = 1; did_deliver = true;
} }
else else
@ -251,6 +251,6 @@ void Ident_Analyzer::BadReply(int length, const char* line)
{ {
BroString s((const u_char*)line, length, true); BroString s((const u_char*)line, length, true);
Weird("bad_ident_reply", s.CheckString()); Weird("bad_ident_reply", s.CheckString());
did_bad_reply = 1; did_bad_reply = true;
} }
} }

6
src/analyzer/protocol/ident/Ident.h
View file

@ -29,8 +29,8 @@ protected:
tcp::ContentLine_Analyzer* orig_ident; tcp::ContentLine_Analyzer* orig_ident;
tcp::ContentLine_Analyzer* resp_ident; tcp::ContentLine_Analyzer* resp_ident;
unsigned int did_deliver:1; bool did_deliver;
unsigned int did_bad_reply:1; bool did_bad_reply;
}; };
} } // namespace analyzer::* } } // namespace analyzer::*

20
src/analyzer/protocol/netbios/NetbiosSSN.h
View file

@ -32,9 +32,9 @@ typedef enum {
struct NetbiosSSN_RawMsgHdr { struct NetbiosSSN_RawMsgHdr {
NetbiosSSN_RawMsgHdr(const u_char*& data, int& len); NetbiosSSN_RawMsgHdr(const u_char*& data, int& len);
unsigned int type:8; uint8_t type;
unsigned int flags:8; uint8_t flags;
unsigned int length:16; uint16_t length;
}; };
// 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 // 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
@ -51,13 +51,13 @@ struct NetbiosSSN_RawMsgHdr {
struct NetbiosDGM_RawMsgHdr { struct NetbiosDGM_RawMsgHdr {
NetbiosDGM_RawMsgHdr(const u_char*& data, int& len); NetbiosDGM_RawMsgHdr(const u_char*& data, int& len);
unsigned int type:8; uint8_t type;
unsigned int flags:8; uint8_t flags;
unsigned int id:16; uint16_t id;
unsigned int srcip:32; uint32_t srcip;
unsigned int srcport:16; uint16_t srcport;
unsigned int length:16; uint16_t length;
unsigned int offset:16; uint16_t offset;
}; };

15
src/analyzer/protocol/tcp/ContentLine.cc
View file

@ -21,17 +21,17 @@ ContentLine_Analyzer::ContentLine_Analyzer(const char* name, Connection* conn, b
void ContentLine_Analyzer::InitState() void ContentLine_Analyzer::InitState()
{ {
flag_NULs = 0; flag_NULs = false;
CR_LF_as_EOL = (CR_as_EOL | LF_as_EOL); CR_LF_as_EOL = (CR_as_EOL | LF_as_EOL);
skip_deliveries = 0; skip_deliveries = false;
skip_partial = 0; skip_partial = false;
buf = 0; buf = 0;
seq_delivered_in_lines = 0; seq_delivered_in_lines = 0;
skip_pending = 0; skip_pending = 0;
seq = 0; seq = 0;
seq_to_skip = 0; seq_to_skip = 0;
plain_delivery_length = 0; plain_delivery_length = 0;
is_plain = 0; is_plain = false;
suppress_weirds = false; suppress_weirds = false;
InitBuffer(0); InitBuffer(0);
@ -70,7 +70,7 @@ ContentLine_Analyzer::~ContentLine_Analyzer()
delete [] buf; delete [] buf;
} }
int ContentLine_Analyzer::HasPartialLine() const bool ContentLine_Analyzer::HasPartialLine() const
{ {
return buf && offset > 0; return buf && offset > 0;
} }
@ -150,11 +150,11 @@ void ContentLine_Analyzer::DoDeliver(int len, const u_char* data)
last_char = 0; // clear last_char last_char = 0; // clear last_char
plain_delivery_length -= deliver_plain; plain_delivery_length -= deliver_plain;
is_plain = 1; is_plain = true;
ForwardStream(deliver_plain, data, IsOrig()); ForwardStream(deliver_plain, data, IsOrig());
is_plain = 0; is_plain = false;
data += deliver_plain; data += deliver_plain;
len -= deliver_plain; len -= deliver_plain;
@ -339,4 +339,3 @@ void ContentLine_Analyzer::SkipBytes(int64_t length)
skip_pending = 0; skip_pending = 0;
seq_to_skip = SeqDelivered() + length; seq_to_skip = SeqDelivered() + length;
} }

16
src/analyzer/protocol/tcp/ContentLine.h
View file

@ -35,12 +35,12 @@ public:
int CRLFAsEOL() int CRLFAsEOL()
{ return CR_LF_as_EOL ; } { return CR_LF_as_EOL ; }
int HasPartialLine() const; bool HasPartialLine() const;
bool SkipDeliveries() const bool SkipDeliveries() const
{ return skip_deliveries; } { return skip_deliveries; }
void SetSkipDeliveries(int should_skip) void SetSkipDeliveries(bool should_skip)
{ skip_deliveries = should_skip; } { skip_deliveries = should_skip; }
// We actually have two delivery modes: line delivery and plain // We actually have two delivery modes: line delivery and plain
@ -97,21 +97,21 @@ protected:
// Remaining bytes to deliver plain. // Remaining bytes to deliver plain.
int64_t plain_delivery_length; int64_t plain_delivery_length;
int is_plain; bool is_plain;
// Don't deliver further data. // Don't deliver further data.
int skip_deliveries; bool skip_deliveries;
bool suppress_weirds; bool suppress_weirds;
// If true, flag (first) line with embedded NUL. // If true, flag (first) line with embedded NUL.
unsigned int flag_NULs:1; bool flag_NULs;
// Whether single CR / LF are considered as EOL. // Whether single CR / LF are considered as EOL.
unsigned int CR_LF_as_EOL:2; uint8_t CR_LF_as_EOL:2;
// Whether to skip partial conns. // Whether to skip partial conns.
unsigned int skip_partial:1; bool skip_partial;
}; };
} } // namespace analyzer::* } } // namespace analyzer::*

16
src/analyzer/protocol/tcp/TCP_Reassembler.cc
View file

@ -28,9 +28,9 @@ TCP_Reassembler::TCP_Reassembler(analyzer::Analyzer* arg_dst_analyzer,
endp = arg_endp; endp = arg_endp;
had_gap = false; had_gap = false;
record_contents_file = 0; record_contents_file = 0;
deliver_tcp_contents = 0; deliver_tcp_contents = false;
skip_deliveries = 0; skip_deliveries = false;
did_EOF = 0; did_EOF = false;
seq_to_skip = 0; seq_to_skip = 0;
in_delivery = false; in_delivery = false;
@ -49,7 +49,7 @@ TCP_Reassembler::TCP_Reassembler(analyzer::Analyzer* arg_dst_analyzer,
if ( (IsOrig() && tcp_content_deliver_all_orig) || if ( (IsOrig() && tcp_content_deliver_all_orig) ||
(! IsOrig() && tcp_content_deliver_all_resp) || (! IsOrig() && tcp_content_deliver_all_resp) ||
(result && result->AsBool()) ) (result && result->AsBool()) )
deliver_tcp_contents = 1; deliver_tcp_contents = true;
Unref(dst_port_val); Unref(dst_port_val);
} }
@ -221,7 +221,7 @@ void TCP_Reassembler::Undelivered(uint64_t up_to_seq)
// the SYN packet carries data. // the SYN packet carries data.
// //
// Skip the undelivered part without reporting to the endpoint. // Skip the undelivered part without reporting to the endpoint.
skip_deliveries = 1; skip_deliveries = true;
} }
else else
{ {
@ -517,7 +517,7 @@ int TCP_Reassembler::DataSent(double t, uint64_t seq, int len,
{ {
tcp_analyzer->Weird("above_hole_data_without_any_acks"); tcp_analyzer->Weird("above_hole_data_without_any_acks");
ClearBlocks(); ClearBlocks();
skip_deliveries = 1; skip_deliveries = true;
} }
if ( tcp_excessive_data_without_further_acks && if ( tcp_excessive_data_without_further_acks &&
@ -525,7 +525,7 @@ int TCP_Reassembler::DataSent(double t, uint64_t seq, int len,
{ {
tcp_analyzer->Weird("excessive_data_without_further_acks"); tcp_analyzer->Weird("excessive_data_without_further_acks");
ClearBlocks(); ClearBlocks();
skip_deliveries = 1; skip_deliveries = true;
} }
return 1; return 1;
@ -592,7 +592,7 @@ void TCP_Reassembler::CheckEOF()
network_time, endp->IsOrig()); network_time, endp->IsOrig());
} }
did_EOF = 1; did_EOF = true;
tcp_analyzer->EndpointEOF(this); tcp_analyzer->EndpointEOF(this);
} }
} }

10
src/analyzer/protocol/tcp/TCP_Reassembler.h
View file

@ -96,10 +96,10 @@ private:
TCP_Endpoint* endp; TCP_Endpoint* endp;
unsigned int deliver_tcp_contents:1; bool deliver_tcp_contents;
unsigned int had_gap:1; bool had_gap;
unsigned int did_EOF:1; bool did_EOF;
unsigned int skip_deliveries:1; bool skip_deliveries;
uint64_t seq_to_skip; uint64_t seq_to_skip;
@ -114,4 +114,4 @@ private:
Type type; Type type;
}; };
} } // namespace analyzer::* } } // namespace analyzer::*

91
src/threading/formatters/JSON.cc
View file

@ -12,9 +12,18 @@
#include <stdint.h> #include <stdint.h>
#include "JSON.h" #include "JSON.h"
#include "3rdparty/rapidjson/include/rapidjson/internal/ieee754.h"
using namespace threading::formatter; using namespace threading::formatter;
bool JSON::NullDoubleWriter::Double(double d)
{
if ( rapidjson::internal::Double(d).IsNanOrInf() )
return rapidjson::Writer<rapidjson::StringBuffer>::Null();
return rapidjson::Writer<rapidjson::StringBuffer>::Double(d);
}
JSON::JSON(MsgThread* t, TimeFormat tf) : Formatter(t), surrounding_braces(true) JSON::JSON(MsgThread* t, TimeFormat tf) : Formatter(t), surrounding_braces(true)
{ {
timestamps = tf; timestamps = tf;
@ -27,21 +36,19 @@ JSON::~JSON()
bool JSON::Describe(ODesc* desc, int num_fields, const Field* const * fields, bool JSON::Describe(ODesc* desc, int num_fields, const Field* const * fields,
Value** vals) const Value** vals) const
{ {
ZeekJson j = ZeekJson::object(); rapidjson::StringBuffer buffer;
NullDoubleWriter writer(buffer);
writer.StartObject();
for ( int i = 0; i < num_fields; i++ ) for ( int i = 0; i < num_fields; i++ )
{ {
if ( vals[i]->present ) if ( vals[i]->present )
{ BuildJSON(writer, vals[i], fields[i]->name);
ZeekJson new_entry = BuildJSON(vals[i]);
if ( new_entry.is_null() )
return false;
j.emplace(fields[i]->name, new_entry);
}
} }
desc->Add(j.dump()); writer.EndObject();
desc->Add(buffer.GetString());
return true; return true;
} }
@ -54,14 +61,18 @@ bool JSON::Describe(ODesc* desc, Value* val, const string& name) const
return false; return false;
} }
if ( ! val->present ) if ( ! val->present || name.empty() )
return true; return true;
ZeekJson j = BuildJSON(val, name); rapidjson::Document doc;
if ( j.is_null() ) rapidjson::StringBuffer buffer;
return false; NullDoubleWriter writer(buffer);
desc->Add(j.dump()); writer.StartObject();
BuildJSON(writer, val, name);
writer.EndObject();
desc->Add(buffer.GetString());
return true; return true;
} }
@ -71,43 +82,47 @@ threading::Value* JSON::ParseValue(const string& s, const string& name, TypeTag
return nullptr; return nullptr;
} }
ZeekJson JSON::BuildJSON(Value* val, const string& name) const void JSON::BuildJSON(NullDoubleWriter& writer, Value* val, const string& name) const
{ {
// If the value wasn't set, return a nullptr. This will get turned into a 'null' in the json output.
if ( ! val->present ) if ( ! val->present )
return nullptr; {
writer.Null();
return;
}
if ( ! name.empty() )
writer.Key(name);
ZeekJson j;
switch ( val->type ) switch ( val->type )
{ {
case TYPE_BOOL: case TYPE_BOOL:
j = val->val.int_val != 0; writer.Bool(val->val.int_val != 0);
break; break;
case TYPE_INT: case TYPE_INT:
j = val->val.int_val; writer.Int64(val->val.int_val);
break; break;
case TYPE_COUNT: case TYPE_COUNT:
case TYPE_COUNTER: case TYPE_COUNTER:
j = val->val.uint_val; writer.Uint64(val->val.uint_val);
break; break;
case TYPE_PORT: case TYPE_PORT:
j = val->val.port_val.port; writer.Uint64(val->val.port_val.port);
break; break;
case TYPE_SUBNET: case TYPE_SUBNET:
j = Formatter::Render(val->val.subnet_val); writer.String(Formatter::Render(val->val.subnet_val));
break; break;
case TYPE_ADDR: case TYPE_ADDR:
j = Formatter::Render(val->val.addr_val); writer.String(Formatter::Render(val->val.addr_val));
break; break;
case TYPE_DOUBLE: case TYPE_DOUBLE:
case TYPE_INTERVAL: case TYPE_INTERVAL:
j = val->val.double_val; writer.Double(val->val.double_val);
break; break;
case TYPE_TIME: case TYPE_TIME:
@ -125,7 +140,7 @@ ZeekJson JSON::BuildJSON(Value* val, const string& name) const
GetThread()->Error(GetThread()->Fmt("json formatter: failure getting time: (%lf)", val->val.double_val)); GetThread()->Error(GetThread()->Fmt("json formatter: failure getting time: (%lf)", val->val.double_val));
// This was a failure, doesn't really matter what gets put here // This was a failure, doesn't really matter what gets put here
// but it should probably stand out... // but it should probably stand out...
j = "2000-01-01T00:00:00.000000"; writer.String("2000-01-01T00:00:00.000000");
} }
else else
{ {
@ -136,17 +151,17 @@ ZeekJson JSON::BuildJSON(Value* val, const string& name) const
frac += 1; frac += 1;
snprintf(buffer2, sizeof(buffer2), "%s.%06.0fZ", buffer, fabs(frac) * 1000000); snprintf(buffer2, sizeof(buffer2), "%s.%06.0fZ", buffer, fabs(frac) * 1000000);
j = buffer2; writer.String(buffer2, strlen(buffer2));
} }
} }
else if ( timestamps == TS_EPOCH ) else if ( timestamps == TS_EPOCH )
j = val->val.double_val; writer.Double(val->val.double_val);
else if ( timestamps == TS_MILLIS ) else if ( timestamps == TS_MILLIS )
{ {
// ElasticSearch uses milliseconds for timestamps // ElasticSearch uses milliseconds for timestamps
j = (uint64_t) (val->val.double_val * 1000); writer.Uint64((uint64_t) (val->val.double_val * 1000));
} }
break; break;
@ -157,36 +172,34 @@ ZeekJson JSON::BuildJSON(Value* val, const string& name) const
case TYPE_FILE: case TYPE_FILE:
case TYPE_FUNC: case TYPE_FUNC:
{ {
j = json_escape_utf8(string(val->val.string_val.data, val->val.string_val.length)); writer.String(json_escape_utf8(string(val->val.string_val.data, val->val.string_val.length)));
break; break;
} }
case TYPE_TABLE: case TYPE_TABLE:
{ {
j = ZeekJson::array(); writer.StartArray();
for ( int idx = 0; idx < val->val.set_val.size; idx++ ) for ( int idx = 0; idx < val->val.set_val.size; idx++ )
j.push_back(BuildJSON(val->val.set_val.vals[idx])); BuildJSON(writer, val->val.set_val.vals[idx]);
writer.EndArray();
break; break;
} }
case TYPE_VECTOR: case TYPE_VECTOR:
{ {
j = ZeekJson::array(); writer.StartArray();
for ( int idx = 0; idx < val->val.vector_val.size; idx++ ) for ( int idx = 0; idx < val->val.vector_val.size; idx++ )
j.push_back(BuildJSON(val->val.vector_val.vals[idx])); BuildJSON(writer, val->val.vector_val.vals[idx]);
writer.EndArray();
break; break;
} }
default: default:
reporter->Warning("Unhandled type in JSON::BuildJSON");
break; break;
} }
if ( ! name.empty() && ! j.is_null() )
return { { name, j } };
return j;
} }

27
src/threading/formatters/JSON.h
View file

@ -2,24 +2,14 @@
#pragma once #pragma once
#include "../Formatter.h" #define RAPIDJSON_HAS_STDSTRING 1
#include "3rdparty/json.hpp" #include "3rdparty/rapidjson/include/rapidjson/document.h"
#include "3rdparty/tsl-ordered-map/ordered_map.h" #include "3rdparty/rapidjson/include/rapidjson/writer.h"
#include "../Formatter.h"
namespace threading { namespace formatter { namespace threading { namespace formatter {
// Define a class for use with the json library that orders the keys in the same order that
// they were inserted. By default, the json library orders them alphabetically and we don't
// want it like that.
template<class Key, class T, class Ignore, class Allocator,
class Hash = std::hash<Key>, class KeyEqual = std::equal_to<Key>,
class AllocatorPair = typename std::allocator_traits<Allocator>::template rebind_alloc<std::pair<Key, T>>,
class ValueTypeContainer = std::vector<std::pair<Key, T>, AllocatorPair>>
using ordered_map = tsl::ordered_map<Key, T, Hash, KeyEqual, AllocatorPair, ValueTypeContainer>;
using ZeekJson = nlohmann::basic_json<ordered_map>;
/** /**
* A thread-safe class for converting values into a JSON representation * A thread-safe class for converting values into a JSON representation
* and vice versa. * and vice versa.
@ -40,9 +30,14 @@ public:
threading::Value** vals) const override; threading::Value** vals) const override;
threading::Value* ParseValue(const string& s, const string& name, TypeTag type, TypeTag subtype = TYPE_ERROR) const override; threading::Value* ParseValue(const string& s, const string& name, TypeTag type, TypeTag subtype = TYPE_ERROR) const override;
private: class NullDoubleWriter : public rapidjson::Writer<rapidjson::StringBuffer> {
public:
NullDoubleWriter(rapidjson::StringBuffer& stream) : rapidjson::Writer<rapidjson::StringBuffer>(stream) {}
bool Double(double d);
};
ZeekJson BuildJSON(Value* val, const string& name = "") const; private:
void BuildJSON(NullDoubleWriter& writer, Value* val, const string& name = "") const;
TimeFormat timestamps; TimeFormat timestamps;
bool surrounding_braces; bool surrounding_braces;

100
src/util.cc
View file

@ -2241,54 +2241,112 @@ TEST_CASE("util json_escape_utf8")
CHECK(json_escape_utf8("string") == "string"); CHECK(json_escape_utf8("string") == "string");
CHECK(json_escape_utf8("string\n") == "string\n"); CHECK(json_escape_utf8("string\n") == "string\n");
CHECK(json_escape_utf8("string\x82") == "string\\x82"); CHECK(json_escape_utf8("string\x82") == "string\\x82");
CHECK(json_escape_utf8("\x07\xd4\xb7o") == "\\x07Էo");
// These strings are duplicated from the scripts.base.frameworks.logging.ascii-json-utf8 btest
// Valid ASCII and valid ASCII control characters
CHECK(json_escape_utf8("a") == "a");
CHECK(json_escape_utf8("\b\f\n\r\t\x00\x15") == "\b\f\n\r\t\x00\x15");
// Table 3-7 in https://www.unicode.org/versions/Unicode12.0.0/ch03.pdf describes what is
// valid and invalid for the tests below
// Valid 2 Octet Sequence
CHECK(json_escape_utf8("\xc3\xb1") == "\xc3\xb1");
// Invalid 2 Octet Sequence
CHECK(json_escape_utf8("\xc3\x28") == "\\xc3(");
CHECK(json_escape_utf8("\xc0\x81") == "\\xc0\\x81");
CHECK(json_escape_utf8("\xc1\x81") == "\\xc1\\x81");
CHECK(json_escape_utf8("\xc2\xcf") == "\\xc2\\xcf");
// Invalid Sequence Identifier
CHECK(json_escape_utf8("\xa0\xa1") == "\\xa0\\xa1");
// Valid 3 Octet Sequence
CHECK(json_escape_utf8("\xe2\x82\xa1") == "\xe2\x82\xa1");
CHECK(json_escape_utf8("\xe0\xa3\xa1") == "\xe0\xa3\xa1");
// Invalid 3 Octet Sequence (in 2nd Octet)
CHECK(json_escape_utf8("\xe0\x80\xa1") == "\\xe0\\x80\\xa1");
CHECK(json_escape_utf8("\xe2\x28\xa1") == "\\xe2(\\xa1");
CHECK(json_escape_utf8("\xed\xa0\xa1") == "\\xed\\xa0\\xa1");
// Invalid 3 Octet Sequence (in 3rd Octet)
CHECK(json_escape_utf8("\xe2\x82\x28") == "\\xe2\\x82(");
// Valid 4 Octet Sequence
CHECK(json_escape_utf8("\xf0\x90\x8c\xbc") == "\xf0\x90\x8c\xbc");
CHECK(json_escape_utf8("\xf1\x80\x8c\xbc") == "\xf1\x80\x8c\xbc");
CHECK(json_escape_utf8("\xf4\x80\x8c\xbc") == "\xf4\x80\x8c\xbc");
// Invalid 4 Octet Sequence (in 2nd Octet)
CHECK(json_escape_utf8("\xf0\x80\x8c\xbc") == "\\xf0\\x80\\x8c\\xbc");
CHECK(json_escape_utf8("\xf2\x28\x8c\xbc") == "\\xf2(\\x8c\\xbc");
CHECK(json_escape_utf8("\xf4\x90\x8c\xbc") == "\\xf4\\x90\\x8c\\xbc");
// Invalid 4 Octet Sequence (in 3rd Octet)
CHECK(json_escape_utf8("\xf0\x90\x28\xbc") == "\\xf0\\x90(\\xbc");
// Invalid 4 Octet Sequence (in 4th Octet)
CHECK(json_escape_utf8("\xf0\x28\x8c\x28") == "\\xf0(\\x8c(");
// Invalid 4 Octet Sequence (too short)
CHECK(json_escape_utf8("\xf4\x80\x8c") == "\\xf4\\x80\\x8c");
CHECK(json_escape_utf8("\xf0") == "\\xf0");
} }
string json_escape_utf8(const string& val) string json_escape_utf8(const string& val)
{ {
string result;
result.reserve(val.length());
auto val_data = reinterpret_cast<const unsigned char*>(val.c_str()); auto val_data = reinterpret_cast<const unsigned char*>(val.c_str());
auto val_size = val.length();
// Reserve at least the size of the existing string to avoid resizing the string in the best-case
// scenario where we don't have any multi-byte characters.
string result;
result.reserve(val_size);
size_t idx; size_t idx;
for ( idx = 0; idx < val.length(); ) for ( idx = 0; idx < val_size; )
{ {
// Normal ASCII characters plus a few of the control characters can be inserted directly. The rest of const char ch = val[idx];
// the control characters should be escaped as regular bytes.
if ( ( val[idx] >= 32 && val[idx] <= 127 ) || // Normal ASCII characters plus a few of the control characters can be inserted directly. The
val[idx] == '\b' || val[idx] == '\f' || val[idx] == '\n' || val[idx] == '\r' || val[idx] == '\t' ) // rest of the control characters should be escaped as regular bytes.
if ( ( ch >= 32 && ch <= 127 ) ||
ch == '\b' || ch == '\f' || ch == '\n' || ch == '\r' || ch == '\t' )
{ {
result.push_back(val[idx]); result.push_back(ch);
++idx; ++idx;
continue; continue;
} }
else if ( val[idx] >= 0 && val[idx] < 32 ) else if ( ch >= 0 && ch < 32 )
{ {
result.append(json_escape_byte(val[idx])); result.append(json_escape_byte(ch));
++idx; ++idx;
continue; continue;
} }
// Find out how long the next character should be. // Find out how long the next character should be.
unsigned int char_size = getNumBytesForUTF8(val[idx]); unsigned int char_size = getNumBytesForUTF8(ch);
// If it says that it's a single character or it's not an invalid string UTF8 sequence, insert the one // If it says that it's a single character or it's not an valid string UTF8 sequence, insert
// escaped byte into the string, step forward one, and go to the next character. // the one escaped byte into the string, step forward one, and go to the next character.
if ( char_size == 0 || idx+char_size > val.length() || isLegalUTF8Sequence(val_data+idx, val_data+idx+char_size) == 0 ) if ( char_size == 0 || idx+char_size > val_size || isLegalUTF8Sequence(val_data+idx, val_data+idx+char_size) == 0 )
{ {
result.append(json_escape_byte(val[idx])); result.append(json_escape_byte(ch));
++idx; ++idx;
continue; continue;
} }
for ( size_t step = 0; step < char_size; step++, idx++ ) result.append(val, idx, char_size);
result.push_back(val[idx]); idx += char_size;
} }
// Insert any of the remaining bytes into the string as escaped bytes // Insert any of the remaining bytes into the string as escaped bytes
if ( idx != val.length() ) for ( ; idx < val_size; ++idx )
for ( ; idx < val.length(); ++idx ) result.append(json_escape_byte(val[idx]));
result.append(json_escape_byte(val[idx]));
return result; return result;
} }

2
src/util.h
View file

@ -127,7 +127,7 @@ std::string extract_ip_and_len(const std::string& i, int* len);
inline void bytetohex(unsigned char byte, char* hex_out) inline void bytetohex(unsigned char byte, char* hex_out)
{ {
static const char hex_chars[] = "0123456789abcdef"; static constexpr char hex_chars[] = "0123456789abcdef";
hex_out[0] = hex_chars[(byte & 0xf0) >> 4]; hex_out[0] = hex_chars[(byte & 0xf0) >> 4];
hex_out[1] = hex_chars[byte & 0x0f]; hex_out[1] = hex_chars[byte & 0x0f];
} }

10
testing/btest/Baseline/scripts.base.frameworks.logging.ascii-double/json.log
View file

@ -9,11 +9,11 @@
{"d":0.1234} {"d":0.1234}
{"d":50000.0} {"d":50000.0}
{"d":-50000.0} {"d":-50000.0}
{"d":3.14e+15} {"d":3140000000000000.0}
{"d":-3.14e+15} {"d":-3140000000000000.0}
{"d":1.79e+308} {"d":1.79e308}
{"d":-1.79e+308} {"d":-1.79e308}
{"d":1.23456789e-05} {"d":0.0000123456789}
{"d":2.23e-308} {"d":2.23e-308}
{"d":-2.23e-308} {"d":-2.23e-308}
{"d":null} {"d":null}

11
testing/btest/Baseline/scripts.base.protocols.ftp.bad-adat-encoding/weird.log Normal file
View file

@ -0,0 +1,11 @@
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path weird
#open 2020-01-15-20-41-16
#fields ts uid id.orig_h id.orig_p id.resp_h id.resp_p name addl notice peer
#types time string addr port addr port string string bool string
1348168976.514202 CHhAvVGS1DHFjwGM9 192.168.57.103 60108 192.168.57.101 2811 base64_illegal_encoding character 32 ignored by Base64 decoding F zeek
1348168976.514202 CHhAvVGS1DHFjwGM9 192.168.57.103 60108 192.168.57.101 2811 ftp_adat_bad_first_token_encoding - F zeek
#close 2020-01-15-20-41-16

2
testing/btest/Baseline/scripts.base.utils.json/output
View file

@ -2,7 +2,7 @@ true
123 123
-999 -999
3.14 3.14
-1.23456789e+308 -1.23456789e308
9e-308 9e-308
1480788576.868945 1480788576.868945
"-12.0 hrs" "-12.0 hrs"

BIN
testing/btest/Traces/globus-url-copy-bad-encoding.trace Normal file
View file

Binary file not shown.

2
testing/btest/scripts/base/protocols/ftp/bad-adat-encoding.zeek Normal file
View file

@ -0,0 +1,2 @@
# @TEST-EXEC: zeek -C -r $TRACES/globus-url-copy-bad-encoding.trace %INPUT
# @TEST-EXEC: btest-diff weird.log