ReversingLabs | Blog

File analysis and unpacking in the age of 40M new samples per year

With daily unique malware counts exceeding 100,000 pressure is exerted at sample analysis and automated unpacking systems. Known 400+ packer families and custom packers can be mixed together in layers and in parallel. Today's system has to be able to handle all known format schemas statically and dynamically while anticipating increases in complexity.

We will discuss the creation of a complex file identity model which layers out the entire binary object. This then enables utilization of a correct unpacking and analysis model for each of the identified segments. Object segmenting is done to cover all aspects of the binary object including the multiple packing layers, resources, sections and overlay. Identification methods will cover traditional file identification with special attention to methods used to fool detection tools as well as generic detection methods. We will describe creation and performance of a complex system handling identification and unpacking of large quantities of files, and contrast it against methods in use today. Static, dynamic and generic file unpacking models will be described showing their benefits and flaws in all viable black and white listing scenarios. Utilization of those binary content processors for each identified segment will be queried for performance and scalability.

See you in Helsinki...

As you remember few weeks ago ReversingLabs presented its NyxEngine to the World on BlackHat Europe security conference. Today the conference has published the presentation videos which can be found here, and here is a direct link to our talk video recording. Enjoy...

Two weeks ago we introduced our NyxEngine to the World and we got nothing but positive comments and responses about it. That is why for today's blog we have decided make it do something its not primarily designed to do. With that in mind we decided to create a simple program based on the NyxEngine which does archive conversion from one file type to another. For the purpose of this blog we designed the program called gzip2zip which as its name implies converts GZIP archives to ZIP ones without any sort of decompression/compression procedure involved. And this is possible only do to the fact that both ZIP and GZIP use DEFLATE compression algorithm which is why no data manipulation other than moving is necessary. In order to do a quick conversion we need to perform the following steps:

• Read GZIP data (file name, packed content & size and unpacked content size & CRC)
• Recreate ZIP header data in memory (recreate local and central directories)
• Write data to disk

This is quite a short and simple checklist which is why creation of such program is a relatively simple task. Reverse process is also possible and just as easy to create but since the ZIP file format is more popular we decided to stop at one way conversion. Until next week....

NyxEngine ReversingLabs Corporation

GZip2Zip (package contains the tool with source and the samples used

We had a great time during this year's BlackHat Europe Conference last week.  Now it is the time to sort out our impressions.  First of all, thanks to all that have made it to our talk and have been asking us in hallways about the new engine that we were working on. In a packed full room we have discussed archive steganography and the impacts such and other malformed files have on security products. These two aspects of file tampering overlap and we have shown how steganography implementations can break archive processors thus causing vulnerabilities in file processing. However it is the vulnerability aspect of our presentation that got the most press.  It has been covered here by CNET and here by PCWorld, and it has also been blogged about by ESET here and by one of the conference attendees here.  In addition to all the media and web mentions we have published presentation, white-paper and NyxEngine all which can be found here.  But to give you the whole picture about the presentation we will talk about it and our findings in short.

Our research has been focused around the impact that file malformations have on archive processors. We wanted to see if data could be hidden inside the archive in such a fashion that data itself is invisible to the user no matter which archive processing program is used. Starting from the most basic of methods of string obfuscation to most complex file malformation our research lead us to conclusion that there are multiple ways of achieving our goal. Steganography was not only possible,  it was present all along in the "wild". We have found two existing solutions that successfully implemented file hiding in ZIP archives. To gauge how prevalent is this in the wild we have turned to AccessData, the pioneer in digital investigation software, and its COO Brian Karney.  His answer was that to the best of his knowledge no one is really looking for this kind of steganographic hidden content and is thus not finding any evidence. This answer didn't surprise us since ti is quite a novel technique. Historically, the most common use of steganography was hiding messages in multimedia files.  However amount of data which can be hidden in such fashion is commonly limited by the size and type of file in which the data was hidden. That's not a limitation when hiding data in archives. In such case there are no limitations to the size and type of steganographic content.

Steganography in archive formats, which in itself interesting, had some serious implications. During the course of our research we found that modifications that we do to the archive in order to hide data is interfering with some security applications.  It made them skip or totally stop scanning the archive content.  Each security product was differently impacted.

If we observe this kind of detection evasion from the standpoint of gateway scanner, the impact on scanning your email and incoming traffic would be high as it interferes with the basic software functionality.  But if we observe this from a desktop or an endpoint scanner perspective, the impact is low since the potentially malicious payload is detected right after extraction.  One would argue that even a desktop solution must be able to scan the packed content it supports.  But not everyone agrees on this point, as protection ability of an end point scanning product would not be lessened.

Regardless of how you look at it, there is a very low threat to protected endpoints. But the general rule for security software is that it does not want to have any potentially harmful files resident anywhere on the protected endpoints.  Such detection evasions must be handled.

During our presentation we have shown the possibility of steganography data hiding and the interference that it can have with anti-malware software.  All vulnerabilities that we have found were disclosed to affected vendors in cooperation with CERT-FI and all issues were patched before the public disclosure. Thanks to CERT-FI for their hard work and support.  Further research in this area by ReversingLabs will  have the same disclosure dynamic. ReversingLabs is proud to contribute to the overall endpoint security and we will continue to further our research in the same direction.

Steganography is the art and science of writing hidden messages in such a way that no one, apart from the sender and intended recipient, suspects the existence of the message, a form of security through obscurity. When it comes to digital steganography no stone should be left unturned in the search for viable hidden data. Although digital steganography is commonly used to hide data inside multimedia files, a similar approach can be used to hide data in archives as well. Steganography imposes the following data hiding rule: Data must be hidden in such a fashion that the user has no clue about the hidden message or file's existence. This can be achieved by either hiding existing packed content from all programs designed to unpack the selected file format, or adding new data to existing compressed files, so that the file's usability is unchanged. To discover this hidden information we must go into deep analysis of systems that have developed their own archive processors and see the implications of format specifications being interpreted differently across such solutions.

We have designed NyxEngine to ensure that no byte is left unchecked in the search for interesting archive data. Furthermore Nyx performs detailed data inspection by which it identifies possible vulnerabilities and corruptions in the binary content of archives. By integrating the NyxEngine as the top layer in archive processing, we can successfully detect and prevent all known and future vulnerability attack vectors against archive processors, thus effectively eliminating the possibility of archive bombs and other exploits. In addition to shielding against exploits, Nyx also searches for viable hidden data that was intentionally cloaked from sight using steganographic principles. And since the engine does detailed data inspection, it can correct vulnerabilities and recover files, making it a perfect archive preprocessor.

Nyx engine’s exploit shield functionality checks the following archive areas: stored file name length and content, compression ratio, extract algorithm requirements, checksum tampering, multi-disk tampering, file entry duplication and other miscellaneous header data checks. Serving as a common denominator among all known archive processing solutions, Nyx classifies each instance of tampering in a functional group as vulnerabilities that affects that group.

By performing detailed checks and on-the-fly corrections, the maximum possible archive data is recovered and identified. This is the best way to find files that are present in the archive, but unreported in the archive header and to extract every possible bit from the archive. This method this works not only with unreported files, but with any kind of binary data present in the archive which isn’t assigned to any of the file content.

The detailed file analysis provided by Nyx makes it possible to recover the maximum amount of damaged, corrupt and invalid data.

TitanEngine is primarily envisioned as a portable executable file format unpacker and handling framework. However due to its static unpacking functions it can be used to unpack other file format types such as installers and archives. That is why today we are showing the utilization of the new static unpacking functions that will be available with the next update. We are discussing the upcoming features which is something we generally like to avoid but it is for a good reason. It is only because of the unveiling of the new SDK we have secretly been working on during these last few months that we are even touching the archive unpacking subject. What is it and what does it do will be talked about on BlackHat Europe next week. Until then we will tickle your imagination with an unusual blog about unpacking archives with TitanEngine.

Format we have selected is a simple a Debian archive file format called Deb. Debian packages (DEB files) are standard Unix ar archives that include two gzipped, bzipped or lzmaed tar archives: one that holds the control information and another that contains the data. These two files present in the archive are not compressed but instead they are just stored inside the binary package. Each stored item has its own header, which is defined like this:

typedef struct DEB_HEADER{
    char FileName[16];
    char FileTime[12];
    char Reserved0[6];
    char Reserved1[6];
    char Mode[8];
    char ItemSize[10];
    char TerminateQuote;
    char TerminateNewLine;
}DEB_HEADER, *PDEB_HEADER;

Preceding the first header which is used to describe the archive is the magic string "!<arch>\n" which is used to identify the binary package type. Therefore unpacking the DEB archive format is essentially reading the archive header and copying the binary content that follows it to the selected folder. Header for each binary content contains the file name and time information which can be used during the unpacking process to restore the packed item to its pre-packing state. Because this file format doesn't employ any compression by itself unpacking the DEB format only refers to extraction of the stored binary content. That content is additionally packed but with a different file format which commonly uses compression to reduce the size of the packed file on disk.

This is just one of the many uses for TitanEngine outside the area of unpacking and processing portable executable file format. As we have seen unpacking archives with TitanEngine is quite possible as long as there is no compression or content decompression is supported by the engine. Keep an eye out for our blog next week when we unveil our super secret project. Until then...

TitanEngine ReversingLabs Corporation

DEB unpacker (package contains source code, binary unpacker and a sample archive)