diff --git a/CHANGES b/CHANGES index 27b3020272..cc361fa4c0 100644 --- a/CHANGES +++ b/CHANGES @@ -1,4 +1,73 @@ +2.3-793 | 2015-04-20 20:51:00 -0700 + + * Add decoding of PROXY-AUTHORIZATION header to HTTP analyze, + treating it the same as AUTHORIZATION. (Josh Liburdi) + + * Remove deprecated fields "hot" and "addl" from the connection + record. Remove the functions append_addl() and + append_addl_marker(). (Robin Sommer) + + * Removing the NetFlow analyzer, which hasn't been used anymore + since then corresponding command-line option went away. (Robin + Sommer) + +2.3-787 | 2015-04-20 19:15:23 -0700 + + * A file analyzer for Portable Executables. (Vlad Grigorescu/Seth + Hall). + + Activity gets logged into pe.log. It generates the following + events: + + event pe_dos_header(f: fa_file, h: PE::DOSHeader); + event pe_dos_code(f: fa_file, code: string); + event pe_file_header(f: fa_file, h: PE::FileHeader); + event pe_optional_header(f: fa_file, h: PE::OptionalHeader); + event pe_section_header(f: fa_file, h: PE::SectionHeader); + +2.3-741 | 2015-04-20 13:12:39 -0700 + + * API changes to file analysis mime type detection. Removed + "file_mime_type" and "file_mime_types" event, replacing them with + a new event called "file_metadata_inferred". Addresses BIT-1368. + (Jon Siwek) + + * A large series of improvements for file type identification. This + inludes a many signature updates (new types, cleanup, performance + improvments) and splitting out signatures into subfiles. (Seth + Hall) + + * Fix an issue with files having gaps before the bof_buffer is + filled, which could lead to file type identification not working + correctly. (Seth Hall) + + * Fix an issue with packet loss in HTTP file reporting for file type + identification wasn't working correctly zero-length bodies. (Seth + Hall) + + * X.509 certificates are now populating files.log with the mime type + application/pkix-cert. (Seth Hall) + + * Normalized some FILE_ANALYSIS debug messages. (Seth Hall) + +2.3-725 | 2015-04-20 12:54:54 -0700 + + * Updating submodule(s). + +2.3-724 | 2015-04-20 14:11:02 -0500 + + * Fix uninitialized field in raw input reader. (Jon Siwek) + +2.3-722 | 2015-04-20 12:59:03 -0500 + + * Remove unneeded documentation cross-referencing. (Jon Siwek) + +2.3-721 | 2015-04-20 12:47:05 -0500 + + * BIT-1380: Improve Broxygen output of &default expressions. + (Jon Siwek) + 2.3-720 | 2015-04-17 14:18:26 -0700 * Updating NEWS. diff --git a/NEWS b/NEWS index 72d880cbaf..c440eb4df9 100644 --- a/NEWS +++ b/NEWS @@ -28,9 +28,12 @@ New Functionality - Bro now has support for the Kerberos KRB5 protocol over TCP and UDP. Activity gets logged into krb.log. -- Bro now parses DTLS traffic. +- Bro now parses DTLS traffic. Activity gets logged into ssl.log. -- Bro now has an RDP analyzer. +- Bro now has an RDP analyzer. Activity gets logged into rdp.log. + +- Bro now has a file analyzer for Portable Executables. Activity gets + logged into pe.log. - Bro now features a completely rewritten, enhanced SSH analyzer, with a set of addedd events being generated. A lot more information about @@ -38,7 +41,10 @@ New Functionality failed or succeeded in most circumstances. - Bro's file analysis now supports reassembly of files that are not - transferred/seen sequentially. + transferred/seen sequentially. The default file reassembly buffer + size is set with the ``Files::reassembly_buffer_size`` variable. + +- Bro's file type identification has been greatly improved. - Bro's scripting language now has a ``while`` statement:: @@ -117,14 +123,17 @@ Changed Functionality - File analysis * Removed ``fa_file`` record's ``mime_type`` and ``mime_types`` - fields. The events ``file_mime_type`` and ``file_mime_types`` - have been added which contain the same information. The - ``mime_type`` field of ``Files::Info`` also still has this info. + fields. The event ``file_sniff`` has been added which provides + the same information. The ``mime_type`` field of ``Files::Info`` + also still has this info. * The earliest point that new mime type information is available is - in the ``file_mime_type`` event which comes after the ``file_new`` - and ``file_over_new_connection`` events. Scripts which inspected - mime type info within those events will need to be adapted. + in the ``file_sniff`` event which comes after the ``file_new`` and + ``file_over_new_connection`` events. Scripts which inspected mime + type info within those events will need to be adapted. (Note: for + users that worked w/ versions of Bro from git, for a while there was + also an event called ``file_mime_type`` which is now replaced with + the ``file_sniff`` event). * Removed ``Files::add_analyzers_for_mime_type`` function. @@ -179,6 +188,13 @@ Changed Functionality - BroControl now sends all normal command output (i.e., not error messages) to stdout. Error messages are still sent to stderr, however. +- The capability of processing NetFlow input has been removed for the + time being. + +- The deprecated fields "hot" and "addl" have been removed from the + connection record. Likewise, the functions append_addl() and + append_addl_marker() have been removed. + Deprecated Functionality ------------------------ diff --git a/VERSION b/VERSION index 4953236124..ed37397383 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -2.3-720 +2.3-793 diff --git a/aux/broctl b/aux/broctl index e864a0949e..d52d184bc9 160000 --- a/aux/broctl +++ b/aux/broctl @@ -1 +1 @@ -Subproject commit e864a0949e52a797f4000194b5c2980cf3618deb +Subproject commit d52d184bc9aa976ee465914e95ff5c0274a18216 diff --git a/aux/broker b/aux/broker index 0c25c1daa7..a9d74d9133 160000 --- a/aux/broker +++ b/aux/broker @@ -1 +1 @@ -Subproject commit 0c25c1daa7dcf885dd16cc1b725295dc36decafe +Subproject commit a9d74d91333b403be8d8c01f5aadb03a84968e9c diff --git a/doc/frameworks/file_analysis_02.bro b/doc/frameworks/file_analysis_02.bro index 141b11fca6..fd4f0e775e 100644 --- a/doc/frameworks/file_analysis_02.bro +++ b/doc/frameworks/file_analysis_02.bro @@ -1,7 +1,8 @@ -event file_mime_type(f: fa_file, mime_type: string) +event file_sniff(f: fa_file, meta: fa_metadata) { + if ( ! meta?$mime_type ) return; print "new file", f$id; - if ( mime_type == "text/plain" ) + if ( meta$mime_type == "text/plain" ) Files::add_analyzer(f, Files::ANALYZER_MD5); } diff --git a/doc/httpmonitor/file_extraction.bro b/doc/httpmonitor/file_extraction.bro index 3860cb361e..c387156b62 100644 --- a/doc/httpmonitor/file_extraction.bro +++ b/doc/httpmonitor/file_extraction.bro @@ -7,15 +7,18 @@ global mime_to_ext: table[string] of string = { ["text/html"] = "html", }; -event file_mime_type(f: fa_file, mime_type: string) +event file_sniff(f: fa_file, meta: fa_metadata) { if ( f$source != "HTTP" ) return; - if ( mime_type !in mime_to_ext ) + if ( ! meta?$mime_type ) return; - local fname = fmt("%s-%s.%s", f$source, f$id, mime_to_ext[mime_type]); + if ( meta$mime_type !in mime_to_ext ) + return; + + local fname = fmt("%s-%s.%s", f$source, f$id, mime_to_ext[meta$mime_type]); print fmt("Extracting file %s", fname); Files::add_analyzer(f, Files::ANALYZER_EXTRACT, [$extract_filename=fname]); - } \ No newline at end of file + } diff --git a/doc/scripting/index.rst b/doc/scripting/index.rst index 11c2119df9..2b5cfbb49c 100644 --- a/doc/scripting/index.rst +++ b/doc/scripting/index.rst @@ -363,7 +363,7 @@ decrypted from HTTP streams is stored in excerpt from :doc:`/scripts/base/protocols/http/main.bro` below. .. btest-include:: ${BRO_SRC_ROOT}/scripts/base/protocols/http/main.bro - :lines: 9-11,20-22,121 + :lines: 9-11,20-22,125 Because the constant was declared with the ``&redef`` attribute, if we needed to turn this option on globally, we could do so by adding the diff --git a/scripts/base/files/pe/__load__.bro b/scripts/base/files/pe/__load__.bro new file mode 100644 index 0000000000..0098b81a7a --- /dev/null +++ b/scripts/base/files/pe/__load__.bro @@ -0,0 +1,2 @@ +@load ./consts +@load ./main \ No newline at end of file diff --git a/scripts/base/files/pe/consts.bro b/scripts/base/files/pe/consts.bro new file mode 100644 index 0000000000..35ad9c3c61 --- /dev/null +++ b/scripts/base/files/pe/consts.bro @@ -0,0 +1,184 @@ + +module PE; + +export { + const machine_types: table[count] of string = { + [0x00] = "UNKNOWN", + [0x1d3] = "AM33", + [0x8664] = "AMD64", + [0x1c0] = "ARM", + [0x1c4] = "ARMNT", + [0xaa64] = "ARM64", + [0xebc] = "EBC", + [0x14c] = "I386", + [0x200] = "IA64", + [0x9041] = "M32R", + [0x266] = "MIPS16", + [0x366] = "MIPSFPU", + [0x466] = "MIPSFPU16", + [0x1f0] = "POWERPC", + [0x1f1] = "POWERPCFP", + [0x166] = "R4000", + [0x1a2] = "SH3", + [0x1a3] = "SH3DSP", + [0x1a6] = "SH4", + [0x1a8] = "SH5", + [0x1c2] = "THUMB", + [0x169] = "WCEMIPSV2" + } &default=function(i: count):string { return fmt("unknown-%d", i); }; + + const file_characteristics: table[count] of string = { + [0x1] = "RELOCS_STRIPPED", + [0x2] = "EXECUTABLE_IMAGE", + [0x4] = "LINE_NUMS_STRIPPED", + [0x8] = "LOCAL_SYMS_STRIPPED", + [0x10] = "AGGRESSIVE_WS_TRIM", + [0x20] = "LARGE_ADDRESS_AWARE", + [0x80] = "BYTES_REVERSED_LO", + [0x100] = "32BIT_MACHINE", + [0x200] = "DEBUG_STRIPPED", + [0x400] = "REMOVABLE_RUN_FROM_SWAP", + [0x800] = "NET_RUN_FROM_SWAP", + [0x1000] = "SYSTEM", + [0x2000] = "DLL", + [0x4000] = "UP_SYSTEM_ONLY", + [0x8000] = "BYTES_REVERSED_HI" + } &default=function(i: count):string { return fmt("unknown-%d", i); }; + + const dll_characteristics: table[count] of string = { + [0x40] = "DYNAMIC_BASE", + [0x80] = "FORCE_INTEGRITY", + [0x100] = "NX_COMPAT", + [0x200] = "NO_ISOLATION", + [0x400] = "NO_SEH", + [0x800] = "NO_BIND", + [0x2000] = "WDM_DRIVER", + [0x8000] = "TERMINAL_SERVER_AWARE" + } &default=function(i: count):string { return fmt("unknown-%d", i); }; + + const windows_subsystems: table[count] of string = { + [0] = "UNKNOWN", + [1] = "NATIVE", + [2] = "WINDOWS_GUI", + [3] = "WINDOWS_CUI", + [7] = "POSIX_CUI", + [9] = "WINDOWS_CE_GUI", + [10] = "EFI_APPLICATION", + [11] = "EFI_BOOT_SERVICE_DRIVER", + [12] = "EFI_RUNTIME_
DRIVER", + [13] = "EFI_ROM", + [14] = "XBOX" + } &default=function(i: count):string { return fmt("unknown-%d", i); }; + + const directories: table[count] of string = { + [0] = "Export Table", + [1] = "Import Table", + [2] = "Resource Table", + [3] = "Exception Table", + [4] = "Certificate Table", + [5] = "Base Relocation Table", + [6] = "Debug", + [7] = "Architecture", + [8] = "Global Ptr", + [9] = "TLS Table", + [10] = "Load Config Table", + [11] = "Bound Import", + [12] = "IAT", + [13] = "Delay Import Descriptor", + [14] = "CLR Runtime Header", + [15] = "Reserved" + } &default=function(i: count):string { return fmt("unknown-%d", i); }; + + const section_characteristics: table[count] of string = { + [0x8] = "TYPE_NO_PAD", + [0x20] = "CNT_CODE", + [0x40] = "CNT_INITIALIZED_DATA", + [0x80] = "CNT_UNINITIALIZED_DATA", + [0x100] = "LNK_OTHER", + [0x200] = "LNK_INFO", + [0x800] = "LNK_REMOVE", + [0x1000] = "LNK_COMDAT", + [0x8000] = "GPREL", + [0x20000] = "MEM_16BIT", + [0x40000] = "MEM_LOCKED", + [0x80000] = "MEM_PRELOAD", + [0x100000] = "ALIGN_1BYTES", + [0x200000] = "ALIGN_2BYTES", + [0x300000] = "ALIGN_4BYTES", + [0x400000] = "ALIGN_8BYTES", + [0x500000] = "ALIGN_16BYTES", + [0x600000] = "ALIGN_32BYTES", + [0x700000] = "ALIGN_64BYTES", + [0x800000] = "ALIGN_128BYTES", + [0x900000] = "ALIGN_256BYTES", + [0xa00000] = "ALIGN_512BYTES", + [0xb00000] = "ALIGN_1024BYTES", + [0xc00000] = "ALIGN_2048BYTES", + [0xd00000] = "ALIGN_4096BYTES", + [0xe00000] = "ALIGN_8192BYTES", + [0x1000000] = "LNK_NRELOC_OVFL", + [0x2000000] = "MEM_DISCARDABLE", + [0x4000000] = "MEM_NOT_CACHED", + [0x8000000] = "MEM_NOT_PAGED", + [0x10000000] = "MEM_SHARED", + [0x20000000] = "MEM_EXECUTE", + [0x40000000] = "MEM_READ", + [0x80000000] = "MEM_WRITE" + } &default=function(i: count):string { return fmt("unknown-%d", i); }; + + const os_versions: table[count, count] of string = { + [10,0] = "Windows 10", + [6,4] = "Windows 10 Technical Preview", + [6,3] = "Windows 8.1 or Server 2012 R2", + [6,2] = "Windows 8 or Server 2012", + [6,1] = "Windows 7 or Server 2008 R2", + [6,0] = "Windows Vista or Server 2008", + [5,2] = "Windows XP x64 or Server 2003", + [5,1] = "Windows XP", + [5,0] = "Windows 2000", + [4,90] = "Windows Me", + [4,10] = "Windows 98", + [4,0] = "Windows 95 or NT 4.0", + [3,51] = "Windows NT 3.51", + [3,50] = "Windows NT 3.5", + [3,2] = "Windows 3.2", + [3,11] = "Windows for Workgroups 3.11", + [3,10] = "Windows 3.1 or NT 3.1", + [3,0] = "Windows 3.0", + [2,11] = "Windows 2.11", + [2,10] = "Windows 2.10", + [2,0] = "Windows 2.0", + [1,4] = "Windows 1.04", + [1,3] = "Windows 1.03", + [1,1] = "Windows 1.01", + [1,0] = "Windows 1.0", + } &default=function(i: count, j: count):string { return fmt("unknown-%d.%d", i, j); }; + + const section_descs: table[string] of string = { + [".bss"] = "Uninitialized data", + [".cormeta"] = "CLR metadata that indicates that the object file contains managed code", + [".data"] = "Initialized data", + [".debug$F"] = "Generated FPO debug information", + [".debug$P"] = "Precompiled debug types", + [".debug$S"] = "Debug symbols", + [".debug$T"] = "Debug types", + [".drective"] = "Linker options", + [".edata"] = "Export tables", + [".idata"] = "Import tables", + [".idlsym"] = "Includes registered SEH to support IDL attributes", + [".pdata"] = "Exception information", + [".rdata"] = "Read-only initialized data", + [".reloc"] = "Image relocations", + [".rsrc"] = "Resource directory", + [".sbss"] = "GP-relative uninitialized data", + [".sdata"] = "GP-relative initialized data", + [".srdata"] = "GP-relative read-only data", + [".sxdata"] = "Registered exception handler data", + [".text"] = "Executable code", + [".tls"] = "Thread-local storage", + [".tls$"] = "Thread-local storage", + [".vsdata"] = "GP-relative initialized data", + [".xdata"] = "Exception information", + } &default=function(i: string):string { return fmt("unknown-%s", i); }; + +} diff --git a/scripts/base/files/pe/main.bro b/scripts/base/files/pe/main.bro new file mode 100644 index 0000000000..b2723e4138 --- /dev/null +++ b/scripts/base/files/pe/main.bro @@ -0,0 +1,137 @@ +module PE; + +@load ./consts.bro + +export { + redef enum Log::ID += { LOG }; + + type Info: record { + ## Current timestamp. + ts: time &log; + ## File id of this portable executable file. + id: string &log; + ## The target machine that the file was compiled for. + machine: string &log &optional; + ## The time that the file was created at. + compile_ts: time &log &optional; + ## The required operating system. + os: string &log &optional; + ## The subsystem that is required to run this file. + subsystem: string &log &optional; + ## Is the file an executable, or just an object file? + is_exe: bool &log &default=T; + ## Is the file a 64-bit executable? + is_64bit: bool &log &default=T; + ## Does the file support Address Space Layout Randomization? + uses_aslr: bool &log &default=F; + ## Does the file support Data Execution Prevention? + uses_dep: bool &log &default=F; + ## Does the file enforce code integrity checks? + uses_code_integrity: bool &log &default=F; + ## Does the file use structured exception handing? + uses_seh: bool &log &default=T; + ## Does the file have an import table? + has_import_table: bool &log &optional; + ## Does the file have an export table? + has_export_table: bool &log &optional; + ## Does the file have an attribute certificate table? + has_cert_table: bool &log &optional; + ## Does the file have a debug table? + has_debug_data: bool &log &optional; + ## The names of the sections, in order. + section_names: vector of string &log &optional; + }; + + ## Event for accessing logged records. + global log_pe: event(rec: Info); + + ## A hook that gets called when we first see a PE file. + global set_file: hook(f: fa_file); +} + +redef record fa_file += { + pe: Info &optional; +}; + +const pe_mime_types = { "application/x-dosexec" }; + +event bro_init() &priority=5 + { + Files::register_for_mime_types(Files::ANALYZER_PE, pe_mime_types); + Log::create_stream(LOG, [$columns=Info, $ev=log_pe, $path="pe"]); + } + +hook set_file(f: fa_file) &priority=5 + { + if ( ! f?$pe ) + f$pe = [$ts=network_time(), $id=f$id]; + } + +event pe_dos_header(f: fa_file, h: PE::DOSHeader) &priority=5 + { + hook set_file(f); + } + +event pe_file_header(f: fa_file, h: PE::FileHeader) &priority=5 + { + hook set_file(f); + + f$pe$machine = machine_types[h$machine]; + f$pe$compile_ts = h$ts; + f$pe$is_exe = ( h$optional_header_size > 0 ); + + for ( c in h$characteristics ) + { + if ( file_characteristics[c] == "32BIT_MACHINE" ) + f$pe$is_64bit = F; + } + } + +event pe_optional_header(f: fa_file, h: PE::OptionalHeader) &priority=5 + { + hook set_file(f); + + # Only EXEs have optional headers + if ( ! f$pe$is_exe ) + return; + + f$pe$os = os_versions[h$os_version_major, h$os_version_minor]; + f$pe$subsystem = windows_subsystems[h$subsystem]; + + for ( c in h$dll_characteristics ) + { + if ( dll_characteristics[c] == "DYNAMIC_BASE" ) + f$pe$uses_aslr = T; + if ( dll_characteristics[c] == "FORCE_INTEGRITY" ) + f$pe$uses_code_integrity = T; + if ( dll_characteristics[c] == "NX_COMPAT" ) + f$pe$uses_dep = T; + if ( dll_characteristics[c] == "NO_SEH" ) + f$pe$uses_seh = F; + } + + f$pe$has_export_table = (|h$table_sizes| > 0 && h$table_sizes[0] > 0); + f$pe$has_import_table = (|h$table_sizes| > 1 && h$table_sizes[1] > 0); + f$pe$has_cert_table = (|h$table_sizes| > 4 && h$table_sizes[4] > 0); + f$pe$has_debug_data = (|h$table_sizes| > 6 && h$table_sizes[6] > 0); + } + +event pe_section_header(f: fa_file, h: PE::SectionHeader) &priority=5 + { + hook set_file(f); + + # Only EXEs have section headers + if ( ! f$pe$is_exe ) + return; + + if ( ! f$pe?$section_names ) + f$pe$section_names = vector(); + f$pe$section_names[|f$pe$section_names|] = h$name; + } + +event file_state_remove(f: fa_file) &priority=-5 + { + if ( f?$pe && f$pe?$machine ) + Log::write(LOG, f$pe); + } + diff --git a/scripts/base/files/x509/main.bro b/scripts/base/files/x509/main.bro index a810132f8d..c097b84560 100644 --- a/scripts/base/files/x509/main.bro +++ b/scripts/base/files/x509/main.bro @@ -47,6 +47,9 @@ redef record Files::Info += { event x509_certificate(f: fa_file, cert_ref: opaque of x509, cert: X509::Certificate) &priority=5 { + if ( ! f$info?$mime_type ) + f$info$mime_type = "application/pkix-cert"; + f$info$x509 = [$ts=f$info$ts, $id=f$id, $certificate=cert, $handle=cert_ref]; } diff --git a/scripts/base/frameworks/files/magic/__load__.bro b/scripts/base/frameworks/files/magic/__load__.bro index c6ee799a53..34115f0a55 100644 --- a/scripts/base/frameworks/files/magic/__load__.bro +++ b/scripts/base/frameworks/files/magic/__load__.bro @@ -1,3 +1,9 @@ +@load-sigs ./archive +@load-sigs ./audio +@load-sigs ./font @load-sigs ./general +@load-sigs ./image @load-sigs ./msoffice -@load-sigs ./libmagic +@load-sigs ./video + +@load-sigs ./libmagic \ No newline at end of file diff --git a/scripts/base/frameworks/files/magic/archive.sig b/scripts/base/frameworks/files/magic/archive.sig new file mode 100644 index 0000000000..9b95f33b25 --- /dev/null +++ b/scripts/base/frameworks/files/magic/archive.sig @@ -0,0 +1,176 @@ + +signature file-tar { + file-magic /^[[:print:]\x00]{100}([[:digit:]\x20]{7}\x00){3}([[:digit:]\x20]{11}\x00){2}([[:digit:]\x00\x20]{7}[\x20\x00])[0-7\x00]/ + file-mime "application/x-tar", 100 +} + +# This is low priority so that files using zip as a +# container will be identified correctly. +signature file-zip { + file-mime "application/zip", 10 + file-magic /^PK\x03\x04.{2}/ +} + +# Multivolume Zip archive +signature file-multi-zip { + file-mime "application/zip", 10 + file-magic /^PK\x07\x08PK\x03\x04/ +} + +# RAR +signature file-rar { + file-mime "application/x-rar", 70 + file-magic /^Rar!/ +} + +# GZIP +signature file-gzip { + file-mime "application/x-gzip", 100 + file-magic /\x1f\x8b/ +} + +# Microsoft Cabinet +signature file-ms-cab { + file-mime "application/vnd.ms-cab-compressed", 110 + file-magic /^MSCF\x00\x00\x00\x00/ +} + +# Mac OS X DMG files +signature file-dmg { + file-magic /^(\x78\x01\x73\x0D\x62\x62\x60|\x78\xDA\x63\x60\x18\x05|\x78\x01\x63\x60\x18\x05|\x78\xDA\x73\x0D|\x78[\x01\xDA]\xED[\xD0-\xD9])/ + file-mime "application/x-dmg", 100 +} + +# XAR (eXtensible ARchive) format. +# Mac OS X uses this for the .pkg format. +signature file-xar { + file-magic /^xar\!/ + file-mime "application/x-xar", 100 +} + +# RPM +signature file-magic-auto352 { + file-mime "application/x-rpm", 70 + file-magic /^(drpm|\xed\xab\xee\xdb)/ +} + +# StuffIt +signature file-stuffit { + file-mime "application/x-stuffit", 70 + file-magic /^(SIT\x21|StuffIt)/ +} + +# Archived data +signature file-x-archive { + file-mime "application/x-archive", 70 + file-magic /^!?/ +} + +# ARC archive data +signature file-arc { + file-mime "application/x-arc", 70 + file-magic /^[\x00-\x7f]{2}[\x02-\x0a\x14\x48]\x1a/ +} + +# EET archive +signature file-eet { + file-mime "application/x-eet", 70 + file-magic /^\x1e\xe7\xff\x00/ +} + +# Zoo archive +signature file-zoo { + file-mime "application/x-zoo", 70 + file-magic /^.{20}\xdc\xa7\xc4\xfd/ +} + +# LZ4 compressed data (legacy format) +signature file-lz4-legacy { + file-mime "application/x-lz4", 70 + file-magic /(\x02\x21\x4c\x18)/ +} + +# LZ4 compressed data +signature file-lz4 { + file-mime "application/x-lz4", 70 + file-magic /^\x04\x22\x4d\x18/ +} + +# LRZIP compressed data +signature file-lrzip { + file-mime "application/x-lrzip", 1 + file-magic /^LRZI/ +} + +# LZIP compressed data +signature file-lzip { + file-mime "application/x-lzip", 70 + file-magic /^LZIP/ +} + +# Self-extracting PKZIP archive +signature file-magic-auto434 { + file-mime "application/zip", 340 + file-magic /^MZ.{28}(Copyright 1989\x2d1990 PKWARE Inc|PKLITE Copr)\x2e/ +} + +# LHA archive (LZH) +signature file-lzh { + file-mime "application/x-lzh", 80 + file-magic /^.{2}-(lh[ abcdex0-9]|lz[s2-8]|lz[s2-8]|pm[s012]|pc1)-/ +} + +# WARC Archive +signature file-warc { + file-mime "application/warc", 50 + file-magic /^WARC\x2f/ +} + +# 7-zip archive data +signature file-7zip { + file-mime "application/x-7z-compressed", 50 + file-magic /^7z\xbc\xaf\x27\x1c/ +} + +# XZ compressed data +signature file-xz { + file-mime "application/x-xz", 90 + file-magic /^\xfd7zXZ\x00/ +} + +# LHa self-extracting archive +signature file-magic-auto436 { + file-mime "application/x-lha", 120 + file-magic /^MZ.{34}LH[aA]\x27s SFX/ +} + +# ARJ archive data +signature file-arj { + file-mime "application/x-arj", 50 + file-magic /^\x60\xea/ +} + +# Byte-swapped cpio archive +signature file-bs-cpio { + file-mime "application/x-cpio", 50 + file-magic /(\x71\xc7|\xc7\x71)/ +} + +# CPIO archive +signature file-cpio { + file-mime "application/x-cpio", 50 + file-magic /^(\xc7\x71|\x71\xc7)/ +} + +# Compress'd data +signature file-compress { + file-mime "application/x-compress", 50 + file-magic /^\x1f\x9d/ +} + +# LZMA compressed data +signature file-lzma { + file-mime "application/x-lzma", 71 + file-magic /^\x5d\x00\x00/ +} + diff --git a/scripts/base/frameworks/files/magic/audio.sig b/scripts/base/frameworks/files/magic/audio.sig new file mode 100644 index 0000000000..efba99ed0d --- /dev/null +++ b/scripts/base/frameworks/files/magic/audio.sig @@ -0,0 +1,13 @@ + +# MPEG v3 audio +signature file-mpeg-audio { + file-mime "audio/mpeg", 20 + file-magic /^\xff[\xe2\xe3\xf2\xf3\xf6\xf7\xfa\xfb\xfc\xfd]/ +} + +# MPEG v4 audio +signature file-m4a { + file-mime "audio/m4a", 70 + file-magic /^....ftyp(m4a)/ +} + diff --git a/scripts/base/frameworks/files/magic/font.sig b/scripts/base/frameworks/files/magic/font.sig new file mode 100644 index 0000000000..8f2857f6e3 --- /dev/null +++ b/scripts/base/frameworks/files/magic/font.sig @@ -0,0 +1,41 @@ + +# Web Open Font Format +signature file-woff { + file-magic /^wOFF/ + file-mime "application/font-woff", 70 +} + +# TrueType font +signature file-ttf { + file-mime "application/x-font-ttf", 80 + file-magic /^\x00\x01\x00\x00\x00/ +} + +signature file-embedded-opentype { + file-mime "application/vnd.ms-fontobject", 50 + file-magic /^.{34}LP/ +} + +# X11 SNF font +signature file-snf { + file-mime "application/x-font-sfn", 70 + file-magic /^(\x04\x00\x00\x00|\x00\x00\x00\x04).{100}(\x04\x00\x00\x00|\x00\x00\x00\x04)/ +} + +# OpenType font +signature file-opentype { + file-mime "application/vnd.ms-opentype", 70 + file-magic /^OTTO/ +} + +# FrameMaker Font file +signature file-maker-screen-font { + file-mime "application/x-mif", 190 + file-magic /^\x3cMakerScreenFont/ +} + +# >0 string,=SplineFontDB: (len=13), ["Spline Font Database "], swap_endian=0 +signature file-spline-font-db { + file-mime "application/vnd.font-fontforge-sfd", 160 + file-magic /^SplineFontDB\x3a/ +} diff --git a/scripts/base/frameworks/files/magic/general.sig b/scripts/base/frameworks/files/magic/general.sig index 500c4f7be0..eb38d39c8c 100644 --- a/scripts/base/frameworks/files/magic/general.sig +++ b/scripts/base/frameworks/files/magic/general.sig @@ -1,18 +1,87 @@ # General purpose file magic signatures. +# Plaintext +# (Including BOMs for UTF-8, 16, and 32) signature file-plaintext { - file-magic /^([[:print:][:space:]]{10})/ - file-mime "text/plain", -20 + file-mime "text/plain", -20 + file-magic /^(\xef\xbb\xbf|(\x00\x00)?\xfe\xff|\xff\xfe(\x00\x00)?)?[[:space:]\x20-\x7E]{10}/ } -signature file-tar { - file-magic /^[[:print:]\x00]{100}([[:digit:]\x20]{7}\x00){3}([[:digit:]\x20]{11}\x00){2}([[:digit:]\x00\x20]{7}[\x20\x00])[0-7\x00]/ - file-mime "application/x-tar", 100 +signature file-json { + file-mime "text/json", 1 + file-magic /^(\xef\xbb\xbf)?[\x0d\x0a[:blank:]]*\{[\x0d\x0a[:blank:]]*(["][^"]{1,}["]|[a-zA-Z][a-zA-Z0-9\\_]*)[\x0d\x0a[:blank:]]*:[\x0d\x0a[:blank:]]*(["]|\[|\{|[0-9]|true|false)/ } -signature file-zip { - file-mime "application/zip", 10 - file-magic /^PK\x03\x04.{2}/ +signature file-json2 { + file-mime "text/json", 1 + file-magic /^(\xef\xbb\xbf)?[\x0d\x0a[:blank:]]*\[[\x0d\x0a[:blank:]]*(((["][^"]{1,}["]|[0-9]{1,}(\.[0-9]{1,})?|true|false)[\x0d\x0a[:blank:]]*,)|\{|\[)[\x0d\x0a[:blank:]]*/ +} + +# Match empty JSON documents. +signature file-json3 { + file-mime "text/json", 0 + file-magic /^(\xef\xbb\xbf)?[\x0d\x0a[:blank:]]*(\[\]|\{\})[\x0d\x0a[:blank:]]*$/ +} + +signature file-xml { + file-mime "application/xml", 10 + file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*<\?xml / +} + +signature file-xhtml { + file-mime "text/html", 100 + file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*<(![dD][oO][cC][tT][yY][pP][eE] {1,}[hH][tT][mM][lL]|[hH][tT][mM][lL]|[mM][eE][tT][aA] {1,}[hH][tT][tT][pP]-[eE][qQ][uU][iI][vV])/ +} + +signature file-html { + file-mime "text/html", 49 + file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*<([hH][eE][aA][dD]|[hH][tT][mM][lL]|[tT][iI][tT][lL][eE]|[bB][oO][dD][yY])/ +} + +signature file-rss { + file-mime "text/rss", 90 + file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*<[rR][sS][sS]/ +} + +signature file-atom { + file-mime "text/atom", 100 + file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*<([rR][sS][sS][^>]*xmlns:atom|[fF][eE][eE][dD][^>]*xmlns=["']?http:\/\/www.w3.org\/2005\/Atom["']?)/ +} + +signature file-soap { + file-mime "application/soap+xml", 49 + file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*<[sS][oO][aA][pP](-[eE][nN][vV])?:[eE][nN][vV][eE][lL][oO][pP][eE]/ +} + +signature file-cross-domain-policy { + file-mime "text/x-cross-domain-policy", 49 + file-magic /^([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*)?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*<[cC][rR][oO][sS][sS]-[dD][oO][mM][aA][iI][nN]-[pP][oO][lL][iI][cC][yY]/ +} + +signature file-xmlrpc { + file-mime "application/xml-rpc", 49 + file-magic /^(\xef\xbb\xbf)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*(<\?xml .*\?>)?([\x0d\x0a[:blank:]]*()?[\x0d\x0a[:blank:]]*)*<[mM][eE][tT][hH][oO][dD][rR][eE][sS][pP][oO][nN][sS][eE]>/ +} + +signature file-coldfusion { + file-mime "magnus-internal/cold-fusion", 20 + file-magic /^([\x0d\x0a[:blank:]]*()?)*<(CFPARAM|CFSET|CFIF)/ +} + +# Microsoft LNK files +signature file-lnk { + file-mime "application/x-ms-shortcut", 49 + file-magic /^\x4C\x00\x00\x00\x01\x14\x02\x00\x00\x00\x00\x00\xC0\x00\x00\x00\x00\x10\x00\x00\x00\x46/ } signature file-jar { @@ -21,8 +90,20 @@ signature file-jar { } signature file-java-applet { - file-magic /^\xca\xfe\xba\xbe...[\x2e-\x34]/ file-mime "application/x-java-applet", 71 + file-magic /^\xca\xfe\xba\xbe...[\x2d-\x34]/ +} + +# OCSP requests over HTTP. +signature file-ocsp-request { + file-magic /^.{11,19}\x06\x05\x2b\x0e\x03\x02\x1a/ + file-mime "application/ocsp-request", 71 +} + +# OCSP responses over HTTP. +signature file-ocsp-response { + file-magic /^.{11,19}\x06\x09\x2B\x06\x01\x05\x05\x07\x30\x01\x01/ + file-mime "application/ocsp-response", 71 } # Shockwave flash @@ -37,12 +118,6 @@ signature file-tnef { file-mime "application/vnd.ms-tnef", 100 } -# Mac OS X DMG files -signature file-dmg { - file-magic /^(\x78\x01\x73\x0D\x62\x62\x60|\x78\xDA\x63\x60\x18\x05|\x78\x01\x63\x60\x18\x05|\x78\xDA\x73\x0D|\x78[\x01\xDA]\xED[\xD0-\xD9])/ - file-mime "application/x-dmg", 100 -} - # Mac OS X Mach-O executable signature file-mach-o { file-magic /^[\xce\xcf]\xfa\xed\xfe/ @@ -55,13 +130,6 @@ signature file-mach-o-universal { file-mime "application/x-mach-o-executable", 100 } -# XAR (eXtensible ARchive) format. -# Mac OS X uses this for the .pkg format. -signature file-xar { - file-magic /^xar\!/ - file-mime "application/x-xar", 100 -} - signature file-pkcs7 { file-magic /^MIME-Version:.*protocol=\"application\/pkcs7-signature\"/ file-mime "application/pkcs7-signature", 100 @@ -79,16 +147,6 @@ signature file-jnlp { file-mime "application/x-java-jnlp-file", 100 } -signature file-ico { - file-magic /^\x00\x00\x01\x00/ - file-mime "image/x-icon", 70 -} - -signature file-cur { - file-magic /^\x00\x00\x02\x00/ - file-mime "image/x-cursor", 70 -} - signature file-pcap { file-magic /^(\xa1\xb2\xc3\xd4|\xd4\xc3\xb2\xa1)/ file-mime "application/vnd.tcpdump.pcap", 70 @@ -119,7 +177,58 @@ signature file-python { file-mime "text/x-python", 60 } +signature file-awk { + file-mime "text/x-awk", 60 + file-magic /^\x23\x21[^\n]{1,15}bin\/(env[[:space:]]+)?(g|n)?awk/ +} + +signature file-tcl { + file-mime "text/x-tcl", 60 + file-magic /^\x23\x21[^\n]{1,15}bin\/(env[[:space:]]+)?(wish|tcl)/ +} + +signature file-lua { + file-mime "text/x-lua", 49 + file-magic /^\x23\x21[^\n]{1,15}bin\/(env[[:space:]]+)?lua/ +} + +signature file-javascript { + file-mime "application/javascript", 60 + file-magic /^\x23\x21[^\n]{1,15}bin\/(env[[:space:]]+)?node(js)?/ +} + +signature file-javascript2 { + file-mime "application/javascript", 60 + file-magic /^[\x0d\x0a[:blank:]]*<[sS][cC][rR][iI][pP][tT][[:blank:]]+([tT][yY][pP][eE]|[lL][aA][nN][gG][uU][aA][gG][eE])=['"]?([tT][eE][xX][tT]\/)?[jJ][aA][vV][aA][sS][cC][rR][iI][pP][tT]/ +} + +signature file-javascript3 { + file-mime "application/javascript", 60 + # This seems to be a somewhat common idiom in javascript. + file-magic /^[\x0d\x0a[:blank:]]*for \(;;\);/ +} + +signature file-javascript4 { + file-mime "application/javascript", 60 + file-magic /^[\x0d\x0a[:blank:]]*document\.write(ln)?[:blank:]?\(/ +} + +signature file-javascript5 { + file-mime "application/javascript", 60 + file-magic /^\(function\(\)[[:blank:]\n]*\{/ +} + +signature file-javascript6 { + file-mime "application/javascript", 60 + file-magic /^[\x0d\x0a[:blank:]]*