Security is notoriously disunited. Every year multiple tools and projects are released and never maintained. TitanMist is its inverse opposite. Built on top of TitanEngine, it provides automation and manages all known and good PEID signatures, unpacking scripts and other tools in one unified tool. TitanMist is the nicely packaged and open source catch all tool that will become your first line of defense. The project also goes beyond pure tool development. It builds a forum to share information and reverse engineering experience built around the biggest online and collaborative knowledge base about software packers.

With the increase in packed and protected malicious payloads, collaboration and quick response between researchers has become critical. As new sample numbers are quickly closing to 40M samples per year, solution to this problem has to come from reverse engineers themselves, integrating the work that they have done in the past and they continue to do. Huge databases of format identification data and unpacking scripts can be reused in a way to maximize automation. Yet, where do we find a definite collection of functional tools, identification signatures and unpacking tools? And how do we integrate them in a meaningful and accurate way?

TitanMist approaches these problems in a manner recognizable to every reverse engineer. It aims to mimic, but automate, the reversing process enabling everyone to easily create unpackers and integrate them in an extensible system. This builds a powerful and fast growing community analysis tool. Overcoming the most basic problems of reverse engineering problems was the top priority for the TitanMist project. Hoping to bridge the programming knowledge barrier which troubles many reverse engineers TitanMist introduces a variety of programming languages in which unpackers can be written in.

TitanMist goes beyond languages that compile to native code relying heavily on popular and easy to learn script languages. Backed up by LUA and Python this project makes coding unpackers a much simpler task. However the challenge of making TitanMist as easy to adopt and extend as possible meant that the project has to go further than extending support for more programming languages. Knowing that most of reverse engineers are familiar with debugger level script language OllyScript we added the support for it as well. Combined with the full TitanEngine functionality these scripts become powerful automated unpackers which combined with the layer of file format identification create a unique database of file analysis tools.

Download

• TitanMist / TitanMist Source
• BlackHat Vegas 2010 whitepaper
• BlackHat Vegas 2010 presentation

Looks cool? Want one? All you have to do is solve this challenge and tell us what is the password we seek. Sounds easy? Its not... Mail us with your solution at: blog(at)reversinglabs(dot)com; Challenge is now closed! Thanks to everyone who participated. Click read more for the solution...

We didn't even dream about getting so many people to participate in our little challenge. The sheer number of emails simply flooded our mailbox with possible solutions and compliments about  our challenge! One of those compliments expresses just what we want to do in the not-so-distant future, we quote: "Fun challenge, do more of these!" We definitely will!

Now to the solution, and discussion of the parts that proved troublesome for many...

We start by downloading the file and doing our initial analysis. Since the file is a ZIP archive we open it with any program that works with this archive format to find a folder named "r" with the file named "r.zip" in it. This part of the challenge is just a decoy to keep you busy and distracted from the real content which is appended to the archive as an archive comment. That real content is another archive in 7Zip file format, which once extracted produces a single CAB file, and that is where the things get interesting. The CAB file has a single PNG file in it, but to solve this challenge we must observe the image and the archive as two separate objects.

First the image part of the puzzle. The image, once opened, shows a normal picture with the logo of our company. However the picture itself has embedded steganography data. Since we didn't want to do any hard stego which can be solved by inspecting image pixels we embedded our hidden information between valid records inside the PNG file header. Something very similar to what we demonstrated on BlackHat Barcelona earlier this year. With the obvious difference that the file format is an image not an archive. Nonetheless the principle is the same. So, what's hidden? If you open the image file with any hex editor you will see a string "pSWD" near the start of the file. That string is followed by a 16 number sequence: 538B327278BBAB654747288999FBCDA1 which isn't the password we need. Nope, its not - even though many of you thought that that was the end solution. Why isn't it?

Because of the fact that the CAB file that compressed that PNG image holds the last piece of the puzzle. If we scan that CAB file with our NyxEngine we get the following output:

Steganography ID: 0x00000b
Possible steganography due to suspicious CAB extra data present between entries!
Data start: 0x5a; Data size: 0x0000f6

And in that data there is the following text block:

UmFyIRoHAM6Zc4AADQAAAAAAAAA8MSAOyRZcWCVhcEFcUfp
P4JdbtU2derwgjSYp+BpxVYkWJPDtQ/TITifo4qO7qyYz+yLpd9+6
nkwwxmomWHbHK0Bt6UPHOwL/pEKm6IGXo/5dioeP66Fq5brTldgi
Z7do5bbFjykQIsx6PMCBre4iUJ7jcwrwD2MDs69XwuuHL+fMKy9hD
UJQPDEgDskWXFjp6jPWFXoWVSNb4H1zjQpW

Which is, in fact, a base64 encoded password protected RAR file. But, what's the password? The password is the PNG image number sequence converted to lower case text. So, its: 538b327278bbab654747288999fbcda1 which isn't an MD5 and needs not to be bruteforced. Once its entered and the RAR file is decrypted we can see the file named "file" containing the following text: "Password is: 9ec4c12949a4f31474f299058ce2b22a". And that's it, the challenge is successfully completed at that point. No more stego or hidden files.

There are six accepted solutions to this challenge, but the one that simply astonished us is the following python script which solves our challenge:

#! /usr/bin/env python
URL="http://blog.reversinglabs.com/wp-content/uploads/2010/07/r.zip"

import os
import urllib2
import struct

os.chdir("/tmp")
rzip=urllib2.urlopen("http://blog.reversinglabs.com/wp-content/uploads/2010/07/r.zip").read()
r7z = rzip[rzip.find("7z"):]
open("r.7z","w").write(r7z)
os.system("7z e r.7z")
cab = open("puzzle.cab").read()
os.system("cabextract puzzle.cab")
open("r.rar","w").write(cab[0x5a:0x5a+250].decode("base64"))
png = open("ReversingLabs.png").read()
ppos = png.find("pSWD")
sz, = struct.unpack(">I", png[ppos-4:ppos])
pwd = png[ppos+4:ppos+4+sz]
os.system("unrar e -P%s r.rar" % pwd.encode("hex"))
print open("file").read()

Thanks to everyone who participated in our little competition. Winners, your T-shirts are in the mail. Until our next challenge...

NyxEngine ReversingLabs Corporation

Our challenge got beaten by our own NyxEngine! Oh, Nyx...

We had a great time during this year's REcon Conference last week.  Now it is the time to sort out our impressions. First of all, thanks to all that attended our TitanEngine training and during the course of 3 days learned how to make unpackers with our engine. We covered coding of both static and dynamic unpackers and showed how to deal with the complex protection options that reverse engineers come across on a daily basis. In addition to training attendees, we also want to thank everyone who grabbed one of our TitanEngine T-shirts to show support for the project. You want one too? Click here...

We can, without any false flattering, call REcon our favorite small conference and promise to be back next year too! But that doesn't mean that everything went smoothly, as there were some problems with the air conditioning that flooded the conference twitter feed with AC related rants. The heat was so bad that the conference opening talk dedicated a good amount of time to it. Being slightly older than the average REcon attendee, Richard Thieme, made a parallel between Woodstock and the problems we had. He argued that Woodstock wasn't that great either but that over time it became a myth due to people, rain and mud and that the same can be said about the heat in Montreal which will probably make us say on some later REcon "remember the one when the AC was broken? That one was great!" Because indeed it was, and as the AC problems went away, everyone's will to commune ignited. And the people who  attend the conference on a regular basis are probably the best thing about the conference. Don't get us wrong, the trainings were great, the talks were awesome but it was the people who impressed us the most. And it is these great people that we will meet again in two weeks at BlackHat US. Until then...

When talking about new concepts, its always best to demonstrate them on something everyone is familiar with. In our case that's of-course UPX with which we are fairly familiar. It almost feels like we write one UPX unpacker each week, doesn't it?

Today we are presenting an optimization concept that enables us to unpack everything in a single go. Now, when talking about file unpacking we always unpack everything in one go, but we never unpack both the main executable module and all of its packed dependencies in a single run. Normally, you wold do this by batching through individual files.  But from a speed perspective, the best optimization imaginable comes from unpacking the main module and all of its dependencies at once. Since TitanEngine wasn't really designed to do that out-of-the-box, it needs just a little bit of help to pull it off.

The problem is the existence of multiple relocation tables, and more importantly multiple import tables. Since TitanEngine was designed to unpack files one at the time, we must do some additional coding around these boundaries to achieve our goal. Compared to a traditional TitanEngine dynamic unpacker, the only difference is the need to collect import table data for modules in one place, and use that data for any module that has reached its entry point jump. The UPX is a special case because it always imports packed file dependencies through the import table. This is, of course, a static way of importing libraries but our approach must be flexible enough to cover both dynamic and static importing.

To achieve our goal we have to scan the main module and all loaded libraries and try to find  the appropriate patterns. Once the patterns are found, we set breakpoints and store info about them so we know which module triggered which callback event. Normally we have three callbacks for UPX unpackers (LoadLibrary, GetProcAddress and EP jump) but since we are doing transverse unpacking we need one more: the load library event custom handler, which determines whether the loaded dependencies are packed with UPX by trying to find the neccessary breakpoint patterns. Even though it is impossible to have more than one module loading at a time, we still need to store the import data because the import tables for the main executable and dependencies might overlap if the modules are loaded dynamically. Once stored, the import info for each module is retrieved when it hits its entry point callback. Relocations aren't really a problem since there is just one module loading at a time, so we can use our "snapshot and compare" model, provided that modules load on non-default image bases. This can be done in numerous ways - one of the easiest is to compile the sample files so that they do that by default (which is considered cheating in the unpacking game), alternatively, we can pre-allocate the memory so that the modules have no choice but to pick another base address. For the purpose of this blog we cheated, in a real world application of this approach you mustn't.

In the real world you will hardly ever see this kind of case but if you do, you now know how to get everything in one go. Until next week...

TitanEngine ReversingLabs Corporation

RL!deUPX (package contains the unpacker with source and the samples used)

With the increase in packed and protected malicious payloads, collaboration and quick response among researchers has become critical. As new sample numbers are quickly approaching 40M samples per year, the solution to this problem has to come from reverse-engineers themselves, integrating their prior and current work. Huge databases of format identification data and unpacking scripts can be reused to maximize automation. Yet, where do we find a definite collection of functional tools, identification signatures and unpacking tools? And how do we integrate them in a meaningful and accurate way?

Come to this talk to hear how we plan to raise reversing collaboration to a whole new level with TitanMist. We will address today's and future challenges, source code, packaging and distribution, and define your role in making TitanMist the most powerful community tool for years to come.

This talk will be a BlackHat exclusive; a launch and demonstration of TitanMist, a new open-source project based on TitanEngine. All components will be available for distribution with the conference materials.

See you in Vegas...

If you remember not so long ago we wrote about minor inconveniences we encountered while working with OllyDBG. Today we return to that subject with challenges we face when reversing modern software protectors. One such protection is SafeEngine or NoobyProtect, which uses a simple portable executable malformation in order to crash OllyDBG. Even though this crash is limited to old OllyDBG versions, it is interesting for us nevertheless. The crash itself wouldn't be interesting if, in fact, the application didn't then operate normally. But it does, which makes it even more sinister. This leads us to conclude that this portable executable malformation is either ignored by the system or irrelevant to the system loader ... but not to our debugger. So, where's the trick? What does it apply to?

If you look at the import table you will see that the first entry is rather unusual. Its name in LordPE is "kernel32...". Which could be fine since the trailing ".dll" is optional. If only it wasn't for those three little dots. Those dots represent multiple new lines after the name. In fact if we open the file with a hex editor we see that there are exactly 0x100 bytes following the first import library name. And in addition, there are 0x3000 bytes that follow the name of the first imported function in the first library trunk. These extra bytes seem to be too large for Olly to handle, overwriting some memory and ultimately making it crash. Since EIP is rewritten to 0x0d0a0d0a this could be an exploitable attack vector. Now we knew about this kind of attack for a while, having first seen it as an attack via too long names in the export table. But this is a bigger issue, since the import table is affected.

If we investigate further we find an even more interesting issue that arises because of the way Windows processes the import table. If the import address table pointer inside the image import descriptor isn't nil and if it points to a sequence of eight zero bytes, the import entry will be ignored completely. Also, if the import lookup table pointer is valid and it points to at least one valid import address pointer, tools such as LordPE will read that data and display a list of functions that never get imported. This means that Windows will ignore the image import descriptor entry completely, which makes the dynamic link library name irrelevant as it will never reach the loading stage. This enables anyone to craft a file that only looks like its importing a specified library which may or may not be present on the system. And that can create confusion about which library is actually needed and which isn't.

This isn't something that NoobyProtect does - but it could if it wanted to. Working around our crash problem is as easy as updating to OllyDBG 2.0 or patching the broken import table. Until next week...

sampleFile
(package contains a harmless sample file that imports a non existent DLL)

We can, with a little help from TitanEngine's hooking library. The idea is to use our unpacker as a library which will be injected into the packed file during its execution. Such a library would place hooks inside the packer code, redirecting the control flow to our unpacker wherever data collection or execution handling is needed. Those places are usually spots where the packer processes the import table or relocations, jumps to the original entry point, or just switches execution from one layer to another.

What are the benefits of such an approach? Even though its slightly harder to create and test such unpackers, the most notable benefit of unpacking by hooking is total immunity to various anti-debugging tricks used to detect the unpacking process. The only detection applicable to this unpacking scenario is anti-hooking and memory checksumming. The first is hardly ever used in modern protections due to the large number of false positives it gives, which are triggered by the operating system itself, security software and various window skinning applications. The second one is rarely present, and when it is it only covers specific memory regions that correspond to a single protection layer. In conclusion this method of implementing the unpacking process should result in fewer things to worry about.

Implementing this kind of hooking requires building custom functions to process the hook events. This is necessary to maintain the packed program work flow, and is exactly why we preserve the register state with PUSHAD, and if there is a jump affected by our hook, even EFLAGS with PUSHFD. These ASM instructions are embedded in our C code and with the help of naked pre-processor instruction they become the prologue and epilogue of the function. To apply the hooks we use the DLL_PROCESS_ATTACH event. For example if we were to hook the UPX code which loads libraries the hook code flow would look like this:

Since our hooks are 5 bytes we need to "borrow" as many instructions as we need to insert the hook. In this case we are "borrowing" three instructions. These instructions will be executed right after our inserted function is called. This is done to preserve the packer work flow. As you can see from this diagram we are using hooks instead of breakpoints. Therefore these hooks will be placed on at least three places: when UPX calls LoadLibraryA, GetProcAddress and finally once it jumps to the entry point. The most basic sample UPX unpacker is limited to working on executables that don't import functions by ordinals and use the old jump to entry point method. It's quite limited, but it's enough for a proof-of-concept of our technique.

Debugging this kind of unpacker can be rather tricky. This video shows a quick and easy way to do it:

Since we are creating a hook library unpacker, we also need a loader which will execute the unpacking target and inject the unpacker library in it. This can be done in number of ways but we decided to do it via the debug - detach method. Once both the unpacker hook library and the loader are made, our unpacker is complete. We hope you got the idea on how to use this technique to build your own hooking unpackers from our short blog. Until next week...

TitanEngine ReversingLabs Corporation

upxHooks (package contains the unpacker with source and the samples used)

Recording of a panel discussion at Source Boston 2010
Mario Vuksan - President, ReversingLabs Corporation

We had a great time during this year's CARO Workshop Conference held in Helsinki last week.  Now it is the time to sort out our impressions.  First of all, thanks to all that have made it to our talk and asked us many intriguing questions. Slides for our talk are available here. The picture you see above is from the brilliant keynote held by Dr. Alan Solomon. We absolutely enjoyed the keynote and Dr. Solomon's remark regarding the perfect antivirus represented by his three batch files.

Our talk was focused on improving the file analysis metrics and on unpacking technology performance testing,  comparing different solutions. During the talk we have presented a new idea for unpacking optimization.  We proposed unpacking through "binary layering" which enables the reuse of unpacking technology as much as possible. Put simply binary layering enables scanning various parts of the binary object and attributing them to known packing formats. Since multiple segments of the same file can have different formats attached to them we recognize that files commonly don't have simple identities but instead their complex layout is viewed as file's complex identity. These complex identities give much more detailed picture about the file itself and enable easy file categorization and further analysis.

We also talked about optimization that can improve file analysis system's metrics.   In this regard, we have shown that binary layering can improve the unpacking speed when identified segments are processed in parallel. Most objects can take advantage of this kind of optimization, but with some exceptions.  Specifically, this applies to cases where binary object requires other objects to be present in predefined way prevents unpacking one file at the time.   Similarly, it also applies to cases where there are multiple one way unpacking layers with output of the previous layer serving as input for the next one.

To test our hypothesis we did a comparative test using our lab tools and Kaspersky Anti-virus, which incorporates both file unpacking and malicious payload detection.   For the test to be relevant enough and to avoid inclusion of  malware scanning into unpacking metrics we have performed the following:

• Inspecting metrics for our internal lab unpacking tools
• Inspecting metrics for KAV on the predefined set of packed files
• Inspecting metrics for KAV on the set of unpacked files produced by our internal lab tools

It is necessary to perform these three steps together in order to obtain relevant results.  Third step excludes unpacking from scanning results and therefore gets a relatively good comparison for unpacking metrics.  For the purpose of our presentation we performed two distinct tests, one on packed portable executable files and one on installer packages.  The first test has employed one way unpacking while the second test has used non-parallel "binary layering" to detect and unpack files. Here are the results for the first test:

This first test was performed on 1627 portable executable files packed with 140 different packer families. It  demonstrated that our internal tool (referred here as the "BlackBox") has successfully unpacked 95% of the files in 530 seconds. Remaining 74 files we declared as invalid either for static or dynamic analysis, indicating that file recovery can not be applied to salvage corrupt data. This means that reported 1568 objects is the number of output files that were processed by this unpacking library. KAV processed the same amount of files in 534 seconds reporting 4533 objects and 249 events. To clarify, KAV counts all files it finds inside the packed content (every packing level is counted) and then reports the actual number of files detected by its signatures. Number of events refers to all additional operations KAV performs on scanned files such as malware detection, quarantine or deletion action. Finally, in the last step KAV scanned 1568 unpacked files that were produced by BlackBox. Third step eliminates the need for unpacking since all files are already unpacked. This part completed in 300 seconds and KAV reported2042 objects and 35 events. To take into the account the unpacking that was initially performed with BlackBox we have added its execution time to the scan time. Results: KAV performs its scan faster with fewer objects that need scanning. Additionally, there are less events indicating false positive detection on the packer formats themselves .  Granted a small amount of packers used in our test base should be blacklisted as their main use historically has been to hide malicious payload.

Its important to note that the unpacking methods used by BlackBox and KAV are completely different. While KAV mostly uses static unpacking to decompress data to memory, our BlackBox uses both dynamic and static unpacking  methods to decompress data to disk with multiple drive accesses. It is slowed down even further when unpacking dynamic link libraries due to snapshot comparison to repair relocation table. Optimizations can be performed to improve these unpacking results, but none were used. Hence we feel confidant that if all of these unpackers were done using TitanEngine, a significant unpacking speed increase would be gained.

Now, lets move to our second, more interesting test.  Here are the results:

Our second test was performed on 20 selected non-malicious installer packages. We used another internally developed tool, here referred to as "Core", to produce 4275 files in 95 seconds.  In comparison, KAV scanned these same input packages in 300 seconds, reporting 9174 found files. In our last step, we have performed the scan on unpacked files produced by Core.  In that case KAV reported 12175 files with the unpacking finishing in 360 seconds (this is with the added time for file unpacking done by Core). Number of events reported is two and they refer to scan start and scan finish. No malicious objects were detected. Results: This test shows that when performing unpacking on files  that have been already unpacked by Core, KAV is able to scan 3000 more files in the time that is very close to the time needed to scan the packed content.  Further optimizations could certainly apply that would reduce this number even further.

In conclusion, our initial "binary layering" experiment has performed great in comparison to existing solutions., while our first test has demonstrated the value of diligent support for various packing formats.  As these were only lab experiments, much space is left for further optimization and implementation improvements. Until next week...

%~~~~~~~~~~~~~~~~~~~~~~~~%
|H4sIAAAAAAACA3P3dLOwTOxh|
|YGF4zsBg7tHJMApGwYgE////|
|V/zJwsjF8I9BB8QH5QkGjhYG|
|xj/MD'              gULH|
|JrY'                BbVi|
|Tlx|   Y4NgmoOxWoxH4yL5d|
|VDR|   oTseHh8f6WK359lQU|
|qJy\              \YJOGt|
|xhN5I\              \dlr|
|qoJvnIznRDXvHjPWZ   |SY7|
|Lz31nKtYPklkV0F6w   |AKr|
|1E17                ,Vk5|
|afng              ,hp63R|
|VsvNzy8u9qpU670lon11hvnS|
|KNWuSS+vrvNf3HV05beU0NXB|
|p71kJQQYrAFt8kQCpwMAAA==|
%~~~~~~~~~~~~~~~~~~~~~~~~%
  D  E  C  O  D  E  M  E

We are pretty sure that "S" stands for Sophos not Superman. Now, first thing that comes to mind when you look at the "picture" is that the data around the "S" is important. And if we look at the last two letters we see the base64 trademark signature. Which means that all that data is an encoded message or a file. To be able to decode it we must strip that "S" to form a proper base64 data chain. Once done that data looks like this:

H4sIAAAAAAACA3P3dLOwTOxhYGF4zsBg7tHJMApGwYgE////V/zJwsj
F8I9BB8QH5QkGjhYGxj/MDgULHJrYBbViTlxY4NgmoOxWoxH4yL5dVD
RoTseHh8f6WK359lQUqJyYJOGtxhN5IdlrqoJvnIznRDXvHjPWZSY7L
z31nKtYPklkV0F6wAKr1E17Vk5afnghp63RVsvNzy8u9qpU670lon11
hvnSKNWuSS+vrvNf3HV05beU0NXBp71kJQQYrAFt8kQCpwMAAA==

That data must be reverted to either text or binary to continue. To do that we use an online base64 decoder which at first returns a very strange string. But if we decode the data to a binary file and open that with the hex editor we see the well known 0x1F 0x8B signature which indicates that the decoded data is in fact a GZIP file. Now GZip file as we know may or may not store a file name and that is why when we decompress the packed data we do another hex data inspection just to find that the decompressed file is a GIF file. So its an image showing us this: Not quite readable but once it is zoomed and lower cased it points to: http://www.sophos.com/anz/sofarsogood.html which holds the last piece of the puzzle.

Sadly last piece of the puzzle has nothing to do with file analysis whatsoever. Its a crypto challenge requiring you to play with letter substitution crypto algorithms. And this isn't something we are really interested in. You are however more than welcome to fiddle with it if you like. For some help on solving it check this out. Until next week...