As you remember few weeks ago ReversingLabs presented its NyxEngine to the World on BlackHat Europe security conference. Today the conference has published the presentation videos which can be found here, and here is a direct link to our talk video recording. Enjoy...

Two weeks ago we introduced our NyxEngine to the World and we got nothing but positive comments and responses about it. That is why for today's blog we have decided make it do something its not primarily designed to do. With that in mind we decided to create a simple program based on the NyxEngine which does archive conversion from one file type to another. For the purpose of this blog we designed the program called gzip2zip which as its name implies converts GZIP archives to ZIP ones without any sort of decompression/compression procedure involved. And this is possible only do to the fact that both ZIP and GZIP use DEFLATE compression algorithm which is why no data manipulation other than moving is necessary. In order to do a quick conversion we need to perform the following steps:

• Read GZIP data (file name, packed content & size and unpacked content size & CRC)
• Recreate ZIP header data in memory (recreate local and central directories)
• Write data to disk

This is quite a short and simple checklist which is why creation of such program is a relatively simple task. Reverse process is also possible and just as easy to create but since the ZIP file format is more popular we decided to stop at one way conversion. Until next week....

NyxEngine ReversingLabs Corporation

GZip2Zip (package contains the tool with source and the samples used

We had a great time during this year's BlackHat Europe Conference last week.  Now it is the time to sort out our impressions.  First of all, thanks to all that have made it to our talk and have been asking us in hallways about the new engine that we were working on. In a packed full room we have discussed archive steganography and the impacts such and other malformed files have on security products. These two aspects of file tampering overlap and we have shown how steganography implementations can break archive processors thus causing vulnerabilities in file processing. However it is the vulnerability aspect of our presentation that got the most press.  It has been covered here by CNET and here by PCWorld, and it has also been blogged about by ESET here and by one of the conference attendees here.  In addition to all the media and web mentions we have published presentation, white-paper and NyxEngine all which can be found here.  But to give you the whole picture about the presentation we will talk about it and our findings in short.

Our research has been focused around the impact that file malformations have on archive processors. We wanted to see if data could be hidden inside the archive in such a fashion that data itself is invisible to the user no matter which archive processing program is used. Starting from the most basic of methods of string obfuscation to most complex file malformation our research lead us to conclusion that there are multiple ways of achieving our goal. Steganography was not only possible,  it was present all along in the "wild". We have found two existing solutions that successfully implemented file hiding in ZIP archives. To gauge how prevalent is this in the wild we have turned to AccessData, the pioneer in digital investigation software, and its COO Brian Karney.  His answer was that to the best of his knowledge no one is really looking for this kind of steganographic hidden content and is thus not finding any evidence. This answer didn't surprise us since ti is quite a novel technique. Historically, the most common use of steganography was hiding messages in multimedia files.  However amount of data which can be hidden in such fashion is commonly limited by the size and type of file in which the data was hidden. That's not a limitation when hiding data in archives. In such case there are no limitations to the size and type of steganographic content.

Steganography in archive formats, which in itself interesting, had some serious implications. During the course of our research we found that modifications that we do to the archive in order to hide data is interfering with some security applications.  It made them skip or totally stop scanning the archive content.  Each security product was differently impacted.

If we observe this kind of detection evasion from the standpoint of gateway scanner, the impact on scanning your email and incoming traffic would be high as it interferes with the basic software functionality.  But if we observe this from a desktop or an endpoint scanner perspective, the impact is low since the potentially malicious payload is detected right after extraction.  One would argue that even a desktop solution must be able to scan the packed content it supports.  But not everyone agrees on this point, as protection ability of an end point scanning product would not be lessened.

Regardless of how you look at it, there is a very low threat to protected endpoints. But the general rule for security software is that it does not want to have any potentially harmful files resident anywhere on the protected endpoints.  Such detection evasions must be handled.

During our presentation we have shown the possibility of steganography data hiding and the interference that it can have with anti-malware software.  All vulnerabilities that we have found were disclosed to affected vendors in cooperation with CERT-FI and all issues were patched before the public disclosure. Thanks to CERT-FI for their hard work and support.  Further research in this area by ReversingLabs will  have the same disclosure dynamic. ReversingLabs is proud to contribute to the overall endpoint security and we will continue to further our research in the same direction.

Steganography is the art and science of writing hidden messages in such a way that no one, apart from the sender and intended recipient, suspects the existence of the message, a form of security through obscurity. When it comes to digital steganography no stone should be left unturned in the search for viable hidden data. Although digital steganography is commonly used to hide data inside multimedia files, a similar approach can be used to hide data in archives as well. Steganography imposes the following data hiding rule: Data must be hidden in such a fashion that the user has no clue about the hidden message or file's existence. This can be achieved by either hiding existing packed content from all programs designed to unpack the selected file format, or adding new data to existing compressed files, so that the file's usability is unchanged. To discover this hidden information we must go into deep analysis of systems that have developed their own archive processors and see the implications of format specifications being interpreted differently across such solutions.

We have designed NyxEngine to ensure that no byte is left unchecked in the search for interesting archive data. Furthermore Nyx performs detailed data inspection by which it identifies possible vulnerabilities and corruptions in the binary content of archives. By integrating the NyxEngine as the top layer in archive processing, we can successfully detect and prevent all known and future vulnerability attack vectors against archive processors, thus effectively eliminating the possibility of archive bombs and other exploits. In addition to shielding against exploits, Nyx also searches for viable hidden data that was intentionally cloaked from sight using steganographic principles. And since the engine does detailed data inspection, it can correct vulnerabilities and recover files, making it a perfect archive preprocessor.

Nyx engine’s exploit shield functionality checks the following archive areas: stored file name length and content, compression ratio, extract algorithm requirements, checksum tampering, multi-disk tampering, file entry duplication and other miscellaneous header data checks. Serving as a common denominator among all known archive processing solutions, Nyx classifies each instance of tampering in a functional group as vulnerabilities that affects that group.

By performing detailed checks and on-the-fly corrections, the maximum possible archive data is recovered and identified. This is the best way to find files that are present in the archive, but unreported in the archive header and to extract every possible bit from the archive. This method this works not only with unreported files, but with any kind of binary data present in the archive which isn’t assigned to any of the file content.

The detailed file analysis provided by Nyx makes it possible to recover the maximum amount of damaged, corrupt and invalid data.

Introducing NyxEngine

Recovering steganography content with NyxEngine