In this talk we will disclose our recent findings about never before seen PE or Portable executable format malformations. These findings have serious consequences on security and reverse engineering tools and lead to multiple exploit vectors.

PE is the main executable image file format on Windows operating system since its introduction in Windows NT 18 years ago. PE file format itself can be found on numerous Windows-based devices including PCs, mobile and gaming devices, BIOS environments and others. Its proper understanding is the key for securing these platforms. The talk will focus on all aspects of PE file format parsing that leads to undesired behavior or prevents security and reverse engineering tools from inspecting malformated files due to incorrect parsing. Special attention will be given to differences between PECOFF documentation and the actual implementation done by the operating system loader. With respect to these differences we will demonstrate existence of files that can't possibly be considered valid from a documentation standpoint but which are still correctly processed and loaded by the operating system. These differences and numerous design logic flaws can lead to PE processing errors that have serious and hardly detectable security implications. Effects of these PE file format malformations will be compared against several reverse engineering tools, security applications and unpacking systems. Special attention will be given to following PE file format aspects and their malformation consequences:

• General PE header layout in respect to data positioning and consequences of different memory model implementations as specified by PECOFF documentation. Use of multiple PE headers in a single file along with self-destructing headers.
• Alignment fields with their impact on disk and memory layout with the section layout issues that can occur due to disk or memory data overlapping or splicing. In addition to this, section table content will be inspected for issues of data hiding and its limits will be tested for upper and lower content boundaries. We will demonstrate how such issues affect existing static and dynamic PE unpacking systems.
• Data tables, including imports and exports, will be discussed in detail to show how their malformated content can break analysis tools but is still considered valid from the operating system loader standpoint. We will demonstrate existence of files that can miss use existing PE features in order to cloak important file information and omit reverse engineering process. Furthermore based upon these methods a unique undetectable method of API hooking that requires no code for hooks insertion will be presented.
• PE file format will be inspected for integer overflows and we will show how their presence can lead to arbitrary code execution in otherwise safe analysis environments. We will show how PE fields themselves could be used to deliver code payload resulting in a completely new field of programming; via the file format itself.
• In addition to single field and table malformations more complex ones involving multiple fields and tables will also be discussed. As a demonstration of such use case scenario a unique malformation requiring multiple fields working together to establish custom file encryption will be presented. This simple, yet effective, encryption that is reversed during runtime by the operating system loader itself requires no code in the malformated binary itself to be executed. Its effectiveness is in a unique approach to encryption trough file format features themselves in order to prevent static and dynamic file analysis tools from processing such files.

This talk will be a Black Hat exclusive; Whitepaper accompanying the presentation materials will contain detailed description of all malformations discussed during the talk. This whitepaper aims to be a mandatory reading material for security analysts. It will continue to be maintained as new information on PE format malformations are discovered.

More information here.

BlackHat, one of the world's biggest security conferences, was held in Las Vegas two weeks ago. Among the BlackHat conferences this year, Las Vegas was by far the biggest event  - bringing thousands of security researchers to the heart of the Sin City. Bigger then ever before, BlackHat featured eleven tracks with an impressive number of high quality talks and trainings. We were there, and we were more than proud to present our newest file analysis tool, TitanMist, to the World.

In a fairly full room, for a reversing track that is, we presented the TitanMist project we have been working on for the past couple of months. Best described as  an automated PE32 file format identification and unpacking tool, TitanMist aims to improve collaboration among reverse engineers across the globe. That is an ambitious goal, but we have high hopes for this project and believe that we can grow it into something of great value in the coming months. Our detailed project roadmap will be unveiled on our blog next week.

The TitanMist presentation was accompanied by the Arsenal presentation that featured all of our tools: TitanEngine, NyxEngine and TitanMist. This kind of tool demo was introduced at this year's BlackHat USA and we must admit that we like the idea of tool presentations, during which authors receive direct feedback from the community. This feedback enables us to add new features and improve our tools even further. We got  many great ideas from talking to attendees with an interest in our tools and what we do. Ones attendees with particularly intriguing questions or ideas questions were rewarded with one of our TitanEngine T-Shirts. But t-shirt winner or not, we thank you all for your continuing support of ReversingLabs and the TitanEngine project!

In the picture on the left you can see our Arsenal booth and one of our senior software engineers, Mario Suvajac. Mario is one of the guys behind the TitanMist project, in charge of the byte pattern matching and overall tool design.

That is it for this report, until next week...

Looks cool? Want one? All you have to do is solve this challenge and tell us what is the password we seek. Sounds easy? Its not... Mail us with your solution at: blog(at)reversinglabs(dot)com; Challenge is now closed! Thanks to everyone who participated. Click read more for the solution...

We didn't even dream about getting so many people to participate in our little challenge. The sheer number of emails simply flooded our mailbox with possible solutions and compliments about  our challenge! One of those compliments expresses just what we want to do in the not-so-distant future, we quote: "Fun challenge, do more of these!" We definitely will!

Now to the solution, and discussion of the parts that proved troublesome for many...

We start by downloading the file and doing our initial analysis. Since the file is a ZIP archive we open it with any program that works with this archive format to find a folder named "r" with the file named "r.zip" in it. This part of the challenge is just a decoy to keep you busy and distracted from the real content which is appended to the archive as an archive comment. That real content is another archive in 7Zip file format, which once extracted produces a single CAB file, and that is where the things get interesting. The CAB file has a single PNG file in it, but to solve this challenge we must observe the image and the archive as two separate objects.

First the image part of the puzzle. The image, once opened, shows a normal picture with the logo of our company. However the picture itself has embedded steganography data. Since we didn't want to do any hard stego which can be solved by inspecting image pixels we embedded our hidden information between valid records inside the PNG file header. Something very similar to what we demonstrated on BlackHat Barcelona earlier this year. With the obvious difference that the file format is an image not an archive. Nonetheless the principle is the same. So, what's hidden? If you open the image file with any hex editor you will see a string "pSWD" near the start of the file. That string is followed by a 16 number sequence: 538B327278BBAB654747288999FBCDA1 which isn't the password we need. Nope, its not - even though many of you thought that that was the end solution. Why isn't it?

Because of the fact that the CAB file that compressed that PNG image holds the last piece of the puzzle. If we scan that CAB file with our NyxEngine we get the following output:

Steganography ID: 0x00000b
Possible steganography due to suspicious CAB extra data present between entries!
Data start: 0x5a; Data size: 0x0000f6

And in that data there is the following text block:

UmFyIRoHAM6Zc4AADQAAAAAAAAA8MSAOyRZcWCVhcEFcUfp
P4JdbtU2derwgjSYp+BpxVYkWJPDtQ/TITifo4qO7qyYz+yLpd9+6
nkwwxmomWHbHK0Bt6UPHOwL/pEKm6IGXo/5dioeP66Fq5brTldgi
Z7do5bbFjykQIsx6PMCBre4iUJ7jcwrwD2MDs69XwuuHL+fMKy9hD
UJQPDEgDskWXFjp6jPWFXoWVSNb4H1zjQpW

Which is, in fact, a base64 encoded password protected RAR file. But, what's the password? The password is the PNG image number sequence converted to lower case text. So, its: 538b327278bbab654747288999fbcda1 which isn't an MD5 and needs not to be bruteforced. Once its entered and the RAR file is decrypted we can see the file named "file" containing the following text: "Password is: 9ec4c12949a4f31474f299058ce2b22a". And that's it, the challenge is successfully completed at that point. No more stego or hidden files.

There are six accepted solutions to this challenge, but the one that simply astonished us is the following python script which solves our challenge:

#! /usr/bin/env python
URL="http://blog.reversinglabs.com/wp-content/uploads/2010/07/r.zip"

import os
import urllib2
import struct

os.chdir("/tmp")
rzip=urllib2.urlopen("http://blog.reversinglabs.com/wp-content/uploads/2010/07/r.zip").read()
r7z = rzip[rzip.find("7z"):]
open("r.7z","w").write(r7z)
os.system("7z e r.7z")
cab = open("puzzle.cab").read()
os.system("cabextract puzzle.cab")
open("r.rar","w").write(cab[0x5a:0x5a+250].decode("base64"))
png = open("ReversingLabs.png").read()
ppos = png.find("pSWD")
sz, = struct.unpack(">I", png[ppos-4:ppos])
pwd = png[ppos+4:ppos+4+sz]
os.system("unrar e -P%s r.rar" % pwd.encode("hex"))
print open("file").read()

Thanks to everyone who participated in our little competition. Winners, your T-shirts are in the mail. Until our next challenge...

NyxEngine ReversingLabs Corporation

Our challenge got beaten by our own NyxEngine! Oh, Nyx...

We had a great time during this year's REcon Conference last week.  Now it is the time to sort out our impressions. First of all, thanks to all that attended our TitanEngine training and during the course of 3 days learned how to make unpackers with our engine. We covered coding of both static and dynamic unpackers and showed how to deal with the complex protection options that reverse engineers come across on a daily basis. In addition to training attendees, we also want to thank everyone who grabbed one of our TitanEngine T-shirts to show support for the project. You want one too? Click here...

We can, without any false flattering, call REcon our favorite small conference and promise to be back next year too! But that doesn't mean that everything went smoothly, as there were some problems with the air conditioning that flooded the conference twitter feed with AC related rants. The heat was so bad that the conference opening talk dedicated a good amount of time to it. Being slightly older than the average REcon attendee, Richard Thieme, made a parallel between Woodstock and the problems we had. He argued that Woodstock wasn't that great either but that over time it became a myth due to people, rain and mud and that the same can be said about the heat in Montreal which will probably make us say on some later REcon "remember the one when the AC was broken? That one was great!" Because indeed it was, and as the AC problems went away, everyone's will to commune ignited. And the people who  attend the conference on a regular basis are probably the best thing about the conference. Don't get us wrong, the trainings were great, the talks were awesome but it was the people who impressed us the most. And it is these great people that we will meet again in two weeks at BlackHat US. Until then...

With the increase in packed and protected malicious payloads, collaboration and quick response among researchers has become critical. As new sample numbers are quickly approaching 40M samples per year, the solution to this problem has to come from reverse-engineers themselves, integrating their prior and current work. Huge databases of format identification data and unpacking scripts can be reused to maximize automation. Yet, where do we find a definite collection of functional tools, identification signatures and unpacking tools? And how do we integrate them in a meaningful and accurate way?

Come to this talk to hear how we plan to raise reversing collaboration to a whole new level with TitanMist. We will address today's and future challenges, source code, packaging and distribution, and define your role in making TitanMist the most powerful community tool for years to come.

This talk will be a BlackHat exclusive; a launch and demonstration of TitanMist, a new open-source project based on TitanEngine. All components will be available for distribution with the conference materials.

See you in Vegas...

Recording of a panel discussion at Source Boston 2010
Mario Vuksan - President, ReversingLabs Corporation

We had a great time during this year's CARO Workshop Conference held in Helsinki last week.  Now it is the time to sort out our impressions.  First of all, thanks to all that have made it to our talk and asked us many intriguing questions. Slides for our talk are available here. The picture you see above is from the brilliant keynote held by Dr. Alan Solomon. We absolutely enjoyed the keynote and Dr. Solomon's remark regarding the perfect antivirus represented by his three batch files.

Our talk was focused on improving the file analysis metrics and on unpacking technology performance testing,  comparing different solutions. During the talk we have presented a new idea for unpacking optimization.  We proposed unpacking through "binary layering" which enables the reuse of unpacking technology as much as possible. Put simply binary layering enables scanning various parts of the binary object and attributing them to known packing formats. Since multiple segments of the same file can have different formats attached to them we recognize that files commonly don't have simple identities but instead their complex layout is viewed as file's complex identity. These complex identities give much more detailed picture about the file itself and enable easy file categorization and further analysis.

We also talked about optimization that can improve file analysis system's metrics.   In this regard, we have shown that binary layering can improve the unpacking speed when identified segments are processed in parallel. Most objects can take advantage of this kind of optimization, but with some exceptions.  Specifically, this applies to cases where binary object requires other objects to be present in predefined way prevents unpacking one file at the time.   Similarly, it also applies to cases where there are multiple one way unpacking layers with output of the previous layer serving as input for the next one.

To test our hypothesis we did a comparative test using our lab tools and Kaspersky Anti-virus, which incorporates both file unpacking and malicious payload detection.   For the test to be relevant enough and to avoid inclusion of  malware scanning into unpacking metrics we have performed the following:

• Inspecting metrics for our internal lab unpacking tools
• Inspecting metrics for KAV on the predefined set of packed files
• Inspecting metrics for KAV on the set of unpacked files produced by our internal lab tools

It is necessary to perform these three steps together in order to obtain relevant results.  Third step excludes unpacking from scanning results and therefore gets a relatively good comparison for unpacking metrics.  For the purpose of our presentation we performed two distinct tests, one on packed portable executable files and one on installer packages.  The first test has employed one way unpacking while the second test has used non-parallel "binary layering" to detect and unpack files. Here are the results for the first test:

This first test was performed on 1627 portable executable files packed with 140 different packer families. It  demonstrated that our internal tool (referred here as the "BlackBox") has successfully unpacked 95% of the files in 530 seconds. Remaining 74 files we declared as invalid either for static or dynamic analysis, indicating that file recovery can not be applied to salvage corrupt data. This means that reported 1568 objects is the number of output files that were processed by this unpacking library. KAV processed the same amount of files in 534 seconds reporting 4533 objects and 249 events. To clarify, KAV counts all files it finds inside the packed content (every packing level is counted) and then reports the actual number of files detected by its signatures. Number of events refers to all additional operations KAV performs on scanned files such as malware detection, quarantine or deletion action. Finally, in the last step KAV scanned 1568 unpacked files that were produced by BlackBox. Third step eliminates the need for unpacking since all files are already unpacked. This part completed in 300 seconds and KAV reported2042 objects and 35 events. To take into the account the unpacking that was initially performed with BlackBox we have added its execution time to the scan time. Results: KAV performs its scan faster with fewer objects that need scanning. Additionally, there are less events indicating false positive detection on the packer formats themselves .  Granted a small amount of packers used in our test base should be blacklisted as their main use historically has been to hide malicious payload.

Its important to note that the unpacking methods used by BlackBox and KAV are completely different. While KAV mostly uses static unpacking to decompress data to memory, our BlackBox uses both dynamic and static unpacking  methods to decompress data to disk with multiple drive accesses. It is slowed down even further when unpacking dynamic link libraries due to snapshot comparison to repair relocation table. Optimizations can be performed to improve these unpacking results, but none were used. Hence we feel confidant that if all of these unpackers were done using TitanEngine, a significant unpacking speed increase would be gained.

Now, lets move to our second, more interesting test.  Here are the results:

Our second test was performed on 20 selected non-malicious installer packages. We used another internally developed tool, here referred to as "Core", to produce 4275 files in 95 seconds.  In comparison, KAV scanned these same input packages in 300 seconds, reporting 9174 found files. In our last step, we have performed the scan on unpacked files produced by Core.  In that case KAV reported 12175 files with the unpacking finishing in 360 seconds (this is with the added time for file unpacking done by Core). Number of events reported is two and they refer to scan start and scan finish. No malicious objects were detected. Results: This test shows that when performing unpacking on files  that have been already unpacked by Core, KAV is able to scan 3000 more files in the time that is very close to the time needed to scan the packed content.  Further optimizations could certainly apply that would reduce this number even further.

In conclusion, our initial "binary layering" experiment has performed great in comparison to existing solutions., while our first test has demonstrated the value of diligent support for various packing formats.  As these were only lab experiments, much space is left for further optimization and implementation improvements. Until next week...

We have already talked about fixing the damaged, broken or missing files in several occasions. Based on what we know we created the Nexus TitanEngine plugin to deal with cases of missing dependencies and damaged files. Implementing the basic TitanEngine features to correct file abnormalities does however change the file checksum since modifications  needed to correct detected problems modify file and memory content. And that doesn't go well with software protections that check the file integrity during execution. One of those software protectors is tELock, and that is the starting point for today's blog. That and a question "How can we work around checksums when file repairing is necessary?".

Luckily for us most software protections only check the file integrity on disk while the memory integrity checks are only limited to protected data and the protection itself. Therefore we only need to worry about the integrity of the file on disk. To be able to fool any software protection integrity check in a generic way we need to know how these checks are performed. Usually is as simple as opening a file, reading its content in a buffer, hashing it with a custom hashing algorithm and checking if the hash is different then the one stored during file protection. So the logical place to catch the integrity checks is by hooking functions used open the file. Most commonly that involves hooking CreateFile API since all protections use it to gain access to protected file.

Hooking an API in a remote process is easy but not very practical since it involves injecting a DLL into the unpacking process and that isn't something we want to do. Other option is to set a breakpoint at the selected API and filter the information returned to the protection. In order to fool the checksum checks we do the following:

• Detect if the file is broken (Nexus already did this)
• Correct the damaged file and produce a backup file (Nexus already did this)
• Catch all calls to CreateFileW API to determine when the integrity check is performed
• Open a handle to backup file (which is valid for execution since its checksum is unaltered)
• Pass the open handle back to protector so that backup file is hashed and its checksum is confirmed

Since we only place a breakpoint on CreateFileW API we need to filter the information somehow to make the program open the backup file which is unaltered and therefore has the correct checksum. We can alter the parameter string and possibly corrupt the memory or we can pass the correct handle back to the protection. To do that we open a handle to backup file inside the context of the debugger and duplicate it inside the context of the unpacking process. That new handle is then used by the software protection to read the data from the backup file which successfully fools any integrity check regardless of the checksum algorithm used. We do this handle switch only if the file which the protected file is trying to open is the file we are currently unpacking. Since this method is generic we can use it for any software protection, not just tELock.

To test out theory we intentionally damage the sample file by modifying a single non relevant byte. This damaged file is now named damaged.exe and the backup file which is the original one is named damaged.exe.bak. If we try to unpack damaged.exe file the unpacker will unpack the file correctly regardless of the damage done to the file. This process effectively simulates the scenario in which the Nexus plugin automatically corrects the damaged file. Until next week...

TitanEngine ReversingLabs Corporation

NexusCheckSum (package contains the plugin with source and the samples used)

With daily unique malware counts exceeding 100,000 pressure is exerted at sample analysis and automated unpacking systems. Known 400+ packer families and custom packers can be mixed together in layers and in parallel. Today's system has to be able to handle all known format schemas statically and dynamically while anticipating increases in complexity.

We will discuss the creation of a complex file identity model which layers out the entire binary object. This then enables utilization of a correct unpacking and analysis model for each of the identified segments. Object segmenting is done to cover all aspects of the binary object including the multiple packing layers, resources, sections and overlay. Identification methods will cover traditional file identification with special attention to methods used to fool detection tools as well as generic detection methods. We will describe creation and performance of a complex system handling identification and unpacking of large quantities of files, and contrast it against methods in use today. Static, dynamic and generic file unpacking models will be described showing their benefits and flaws in all viable black and white listing scenarios. Utilization of those binary content processors for each identified segment will be queried for performance and scalability.

See you in Helsinki...