ReversingLabs | Blog

We are going to start today's blog with a short apology about the TitanEngine 2.0.3 availability during last week. Issue was that during certain amount of time during last week the old TitanEngine 2.0.2 was distributed instead of the fresh new version. This happened mainly because we were moving our hosting to a new server and mixed-up the TitanEngine packages.  We apologize for any inconvenience this might have caused and urge the users to update to current engine version. With that out of the way we can focus on the task at hand.

We have already talked about fixing the damaged, broken or missing files in several occasions. Based on what we know we created the Nexus TitanEngine plugin to deal with cases of missing dependencies and damaged files. Implementing the basic TitanEngine features to correct file abnormalities does however change the file checksum since modifications  needed to correct detected problems modify file and memory content. And that doesn't go well with software protections that check the file integrity during execution. One of those software protectors is tELock, and that is the starting point for today's blog. That and a question "How can we work around checksums when file repairing is necessary?".

Luckily for us most software protections only check the file integrity on disk while the memory integrity checks are only limited to protected data and the protection itself. Therefore we only need to worry about the integrity of the file on disk. To be able to fool any software protection integrity check in a generic way we need to know how these checks are performed. Usually is as simple as opening a file, reading its content in a buffer, hashing it with a custom hashing algorithm and checking if the hash is different then the one stored during file protection. So the logical place to catch the integrity checks is by hooking functions used open the file. Most commonly that involves hooking CreateFile API since all protections use it to gain access to protected file.

Hooking an API in a remote process is easy but not very practical since it involves injecting a DLL into the unpacking process and that isn't something we want to do. Other option is to set a breakpoint at the selected API and filter the information returned to the protection. In order to fool the checksum checks we do the following:

• Detect if the file is broken (Nexus already did this)
• Correct the damaged file and produce a backup file (Nexus already did this)
• Catch all calls to CreateFileW API to determine when the integrity check is performed
• Open a handle to backup file (which is valid for execution since its checksum is unaltered)
• Pass the open handle back to protector so that backup file is hashed and its checksum is confirmed

Since we only place a breakpoint on CreateFileW API we need to filter the information somehow to make the program open the backup file which is unaltered and therefore has the correct checksum. We can alter the parameter string and possibly corrupt the memory or we can pass the correct handle back to the protection. To do that we open a handle to backup file inside the context of the debugger and duplicate it inside the context of the unpacking process. That new handle is then used by the software protection to read the data from the backup file which successfully fools any integrity check regardless of the checksum algorithm used. We do this handle switch only if the file which the protected file is trying to open is the file we are currently unpacking. Since this method is generic we can use it for any software protection, not just tELock.

To test out theory we intentionally damage the sample file by modifying a single non relevant byte. This damaged file is now named damaged.exe and the backup file which is the original one is named damaged.exe.bak. If we try to unpack damaged.exe file the unpacker will unpack the file correctly regardless of the damage done to the file. This process effectively simulates the scenario in which the Nexus plugin automatically corrects the damaged file. Until next week...

TitanEngine ReversingLabs Corporation

NexusCheckSum (package contains the plugin with source and the samples used)

Its been a really long time since we made an unpacker for... well anything. Sure we did a format converter and some archive format unpacker but our last PE unpacker was (checks the blog) in February. So, lets get back to the basics and create a dynamic unpacker for PackMan. We already have an unpacker for PackMan? Its in the TitanEngine package already, you say? Well we do, but what's stopping us from having a little fun with unpacker optimizations?

There are a lot of optimizations one can do with the TitanEngine to make it work even faster then lightning. During the related unpacker execution timing research for our upcoming CARO Workshop talk we measured the impact that certain operations inside the engine itself have on the total unpacking time. We realized that there is significant space for performance improvement in certain unpacking areas which is especially important when we are processing large file volumes. Now, when unpacking files with unpackers built around the TitanEngine you get unpacker execution times quite similar to the sample execution time, except for cases where dynamic link library unpacking requires snapshots to correct the relocation table. in those cases we see a significant unpacking execution time increase. To counter this we can either do memory snapshots to memory or optimize relocation processing and avoid using snapshots at all.

Generally when talking about fixing relocation table we refer to the easy snap-and-compare method. However there is another way of making the unpacked dynamic link library valid for loading on non default base. We can use RelocaterGrabRelocationTableEx function for cases when the packer uses non modified relocation table, defined as it is in the PECOFF document. Relocation data is still compressed and can only be accessed just before the file is relocated, which is why we need a function to inspect the memory and determine the relocation table size. And that is exactly what RelocaterGrabRelocationTableEx does. It determines the size of the relocation table at the provided address and copies it to the engine for later exporting. If we look at the following PackMan code snippet which does the image relocation:

  OR ECX,ECX
  JE L018
  MOV EDI,DWORD PTR DS:[EBX+24]
  JMP L013
L004:
  XOR EAX,EAX
  LODS WORD PTR DS:[ESI]
  OR EAX,EAX
  JE L011
  AND AH,0F
  ADD EAX,DWORD PTR DS:[EBX]
  ADD DWORD PTR DS:[EDX+EAX],ECX
L011:
  CMP ESI,EDI
  JNZ L004
L013:
  MOV EDX,DWORD PTR DS:[EDI]
  LEA ESI,DWORD PTR DS:[EDI+8]
  ADD EDI,DWORD PTR DS:[EDI+4]
  TEST EDX,EDX
  JNZ L011
L018:
  POPAD
 

We can see that the relocation table is stored at EBX+0x24 address. Therefore by reading that memory pointer before the actual relocation occurs we have all the parameters we need to fix the relocation table. Passing that parameter to the RelocaterGrabRelocationTableEx will result in the engine reading the relocation table and estimating its size. Therefore we can just use the pointer we read at the EBX+0x24 address and the return from RelocaterEstimatedSize to correct the PE header for the unpacked file. However RelocaterEstimatedSize doesn't return the accurate size due to the system design. It must be reduced by 8 to be correct for all cases.

Since we are only updating the PE header data we can free the relocation table stored inside the engine with RelocaterCleanup. Once we dump the process relocation table fixing is as easy as updating the PE header fields. By doing the relocation table fixing this way we optimize the speed of execution by a significant percent. No actual data needs to be written to the file on the disk since it is already there and in the correct format. Furthermore you can start the debugging without the previously necessary DLL loading on the address other then default. If you choose to use that optimization as well packer execution time will be shorter since the file might not be relocated at all thus saving CPU cycles. Until next week...

TitanEngine ReversingLabs Corporation

RL!dePackMan (package contains the unpacker with source and the samples used)

After few months of intense work and code polishing we are proud to present the next major update for the TitanEngine project. Latest update we labeled as TitanEngine 2.0.3. Even though the version incrementation is small the number of changes and the pure size of the code is vast. That is why we dedicate today's blog for listing all additions and changes done to the engine. So, what is new?

This update can be declared as a script update as we were driven by the idea that the TitanEngine should be as easy to use as and as widespread as possible. That is why we have extended the programming language support to LUA and Python, two very popular script languages. Traditionally these script languages execute in a console window with little to no graphical user interface which is why for this update we have integrated easy to use unpacker interface. This interface can be customized to fit your unpacker project and still enables you to use the TitanEngine with any supported script language without the need to make your unpacker a console project. But we didn't stop there, we realized that with more than 400 functions TitanEngine can be overwhelming at first glance. That is why we tried to simplify writing the most basic dynamic unpackers, lowering the knowledge barrier to learning only 5 engine's functions.

In addition to this we introduced a way to set any breakpoint type, be it INT3 single or double byte or UD2 breakpoints. This option was present as a global setting in the engine making it possible to set only one breakpoint type at the time. Now you can set different breakpoints for any byte pattern you choose within the same existing breakpoint manipulation functions.

Release without bug fixes is unimaginable. In this release we fixed all bugs that we are aware of. Thank you for all your reports, you keep TitanEngine with as bugs free as possible.

*Note: If you downloaded the old TitanEngine 2.0.2 instead of the new one please re-download the package as there was a small mix-up with the files while moving hosting servers. We apologize for the inconvenience.

TitanEngine 2.0.3 in numbers

• 405 functions
• 28,000+ lines of code
• 44 usage samples
• 6 supported programming languages
• 390 pages of documentation
• 1 download waiting to happen...

File analysis and unpacking in the age of 40M new samples per year

With daily unique malware counts exceeding 100,000 pressure is exerted at sample analysis and automated unpacking systems. Known 400+ packer families and custom packers can be mixed together in layers and in parallel. Today's system has to be able to handle all known format schemas statically and dynamically while anticipating increases in complexity.

We will discuss the creation of a complex file identity model which layers out the entire binary object. This then enables utilization of a correct unpacking and analysis model for each of the identified segments. Object segmenting is done to cover all aspects of the binary object including the multiple packing layers, resources, sections and overlay. Identification methods will cover traditional file identification with special attention to methods used to fool detection tools as well as generic detection methods. We will describe creation and performance of a complex system handling identification and unpacking of large quantities of files, and contrast it against methods in use today. Static, dynamic and generic file unpacking models will be described showing their benefits and flaws in all viable black and white listing scenarios. Utilization of those binary content processors for each identified segment will be queried for performance and scalability.

See you in Helsinki...

As you remember few weeks ago ReversingLabs presented its NyxEngine to the World on BlackHat Europe security conference. Today the conference has published the presentation videos which can be found here, and here is a direct link to our talk video recording. Enjoy...

Two weeks ago we introduced our NyxEngine to the World and we got nothing but positive comments and responses about it. That is why for today's blog we have decided make it do something its not primarily designed to do. With that in mind we decided to create a simple program based on the NyxEngine which does archive conversion from one file type to another. For the purpose of this blog we designed the program called gzip2zip which as its name implies converts GZIP archives to ZIP ones without any sort of decompression/compression procedure involved. And this is possible only do to the fact that both ZIP and GZIP use DEFLATE compression algorithm which is why no data manipulation other than moving is necessary. In order to do a quick conversion we need to perform the following steps:

• Read GZIP data (file name, packed content & size and unpacked content size & CRC)
• Recreate ZIP header data in memory (recreate local and central directories)
• Write data to disk

This is quite a short and simple checklist which is why creation of such program is a relatively simple task. Reverse process is also possible and just as easy to create but since the ZIP file format is more popular we decided to stop at one way conversion. Until next week....

NyxEngine ReversingLabs Corporation

GZip2Zip (package contains the tool with source and the samples used

We had a great time during this year's BlackHat Europe Conference last week.  Now it is the time to sort out our impressions.  First of all, thanks to all that have made it to our talk and have been asking us in hallways about the new engine that we were working on. In a packed full room we have discussed archive steganography and the impacts such and other malformed files have on security products. These two aspects of file tampering overlap and we have shown how steganography implementations can break archive processors thus causing vulnerabilities in file processing. However it is the vulnerability aspect of our presentation that got the most press.  It has been covered here by CNET and here by PCWorld, and it has also been blogged about by ESET here and by one of the conference attendees here.  In addition to all the media and web mentions we have published presentation, white-paper and NyxEngine all which can be found here.  But to give you the whole picture about the presentation we will talk about it and our findings in short.

Our research has been focused around the impact that file malformations have on archive processors. We wanted to see if data could be hidden inside the archive in such a fashion that data itself is invisible to the user no matter which archive processing program is used. Starting from the most basic of methods of string obfuscation to most complex file malformation our research lead us to conclusion that there are multiple ways of achieving our goal. Steganography was not only possible,  it was present all along in the "wild". We have found two existing solutions that successfully implemented file hiding in ZIP archives. To gauge how prevalent is this in the wild we have turned to AccessData, the pioneer in digital investigation software, and its COO Brian Karney.  His answer was that to the best of his knowledge no one is really looking for this kind of steganographic hidden content and is thus not finding any evidence. This answer didn't surprise us since ti is quite a novel technique. Historically, the most common use of steganography was hiding messages in multimedia files.  However amount of data which can be hidden in such fashion is commonly limited by the size and type of file in which the data was hidden. That's not a limitation when hiding data in archives. In such case there are no limitations to the size and type of steganographic content.

Steganography in archive formats, which in itself interesting, had some serious implications. During the course of our research we found that modifications that we do to the archive in order to hide data is interfering with some security applications.  It made them skip or totally stop scanning the archive content.  Each security product was differently impacted.

If we observe this kind of detection evasion from the standpoint of gateway scanner, the impact on scanning your email and incoming traffic would be high as it interferes with the basic software functionality.  But if we observe this from a desktop or an endpoint scanner perspective, the impact is low since the potentially malicious payload is detected right after extraction.  One would argue that even a desktop solution must be able to scan the packed content it supports.  But not everyone agrees on this point, as protection ability of an end point scanning product would not be lessened.

Regardless of how you look at it, there is a very low threat to protected endpoints. But the general rule for security software is that it does not want to have any potentially harmful files resident anywhere on the protected endpoints.  Such detection evasions must be handled.

During our presentation we have shown the possibility of steganography data hiding and the interference that it can have with anti-malware software.  All vulnerabilities that we have found were disclosed to affected vendors in cooperation with CERT-FI and all issues were patched before the public disclosure. Thanks to CERT-FI for their hard work and support.  Further research in this area by ReversingLabs will  have the same disclosure dynamic. ReversingLabs is proud to contribute to the overall endpoint security and we will continue to further our research in the same direction.

Steganography is the art and science of writing hidden messages in such a way that no one, apart from the sender and intended recipient, suspects the existence of the message, a form of security through obscurity. When it comes to digital steganography no stone should be left unturned in the search for viable hidden data. Although digital steganography is commonly used to hide data inside multimedia files, a similar approach can be used to hide data in archives as well. Steganography imposes the following data hiding rule: Data must be hidden in such a fashion that the user has no clue about the hidden message or file's existence. This can be achieved by either hiding existing packed content from all programs designed to unpack the selected file format, or adding new data to existing compressed files, so that the file's usability is unchanged. To discover this hidden information we must go into deep analysis of systems that have developed their own archive processors and see the implications of format specifications being interpreted differently across such solutions.

We have designed NyxEngine to ensure that no byte is left unchecked in the search for interesting archive data. Furthermore Nyx performs detailed data inspection by which it identifies possible vulnerabilities and corruptions in the binary content of archives. By integrating the NyxEngine as the top layer in archive processing, we can successfully detect and prevent all known and future vulnerability attack vectors against archive processors, thus effectively eliminating the possibility of archive bombs and other exploits. In addition to shielding against exploits, Nyx also searches for viable hidden data that was intentionally cloaked from sight using steganographic principles. And since the engine does detailed data inspection, it can correct vulnerabilities and recover files, making it a perfect archive preprocessor.

Nyx engine’s exploit shield functionality checks the following archive areas: stored file name length and content, compression ratio, extract algorithm requirements, checksum tampering, multi-disk tampering, file entry duplication and other miscellaneous header data checks. Serving as a common denominator among all known archive processing solutions, Nyx classifies each instance of tampering in a functional group as vulnerabilities that affects that group.

By performing detailed checks and on-the-fly corrections, the maximum possible archive data is recovered and identified. This is the best way to find files that are present in the archive, but unreported in the archive header and to extract every possible bit from the archive. This method this works not only with unreported files, but with any kind of binary data present in the archive which isn’t assigned to any of the file content.

The detailed file analysis provided by Nyx makes it possible to recover the maximum amount of damaged, corrupt and invalid data.

TitanEngine is primarily envisioned as a portable executable file format unpacker and handling framework. However due to its static unpacking functions it can be used to unpack other file format types such as installers and archives. That is why today we are showing the utilization of the new static unpacking functions that will be available with the next update. We are discussing the upcoming features which is something we generally like to avoid but it is for a good reason. It is only because of the unveiling of the new SDK we have secretly been working on during these last few months that we are even touching the archive unpacking subject. What is it and what does it do will be talked about on BlackHat Europe next week. Until then we will tickle your imagination with an unusual blog about unpacking archives with TitanEngine.

Format we have selected is a simple a Debian archive file format called Deb. Debian packages (DEB files) are standard Unix ar archives that include two gzipped, bzipped or lzmaed tar archives: one that holds the control information and another that contains the data. These two files present in the archive are not compressed but instead they are just stored inside the binary package. Each stored item has its own header, which is defined like this:

typedef struct DEB_HEADER{
    char FileName[16];
    char FileTime[12];
    char Reserved0[6];
    char Reserved1[6];
    char Mode[8];
    char ItemSize[10];
    char TerminateQuote;
    char TerminateNewLine;
}DEB_HEADER, *PDEB_HEADER;

Preceding the first header which is used to describe the archive is the magic string "!<arch>\n" which is used to identify the binary package type. Therefore unpacking the DEB archive format is essentially reading the archive header and copying the binary content that follows it to the selected folder. Header for each binary content contains the file name and time information which can be used during the unpacking process to restore the packed item to its pre-packing state. Because this file format doesn't employ any compression by itself unpacking the DEB format only refers to extraction of the stored binary content. That content is additionally packed but with a different file format which commonly uses compression to reduce the size of the packed file on disk.

This is just one of the many uses for TitanEngine outside the area of unpacking and processing portable executable file format. As we have seen unpacking archives with TitanEngine is quite possible as long as there is no compression or content decompression is supported by the engine. Keep an eye out for our blog next week when we unveil our super secret project. Until then...

TitanEngine ReversingLabs Corporation

DEB unpacker (package contains source code, binary unpacker and a sample archive)

In the last couple of years we have seen a drastic increase in numbers of malicious sample we see a day. These numbers are quickly closing to 40M samples a year mark that we expect to see hit this year. That is why the sheer volume of data we are bombarded with each day raises an important question, where is the relevant data in this sea of information?  And even is all data we have relevant?

Prioritization is the main way of extracting relevant data with the techniques and methods used to highlight interesting information varying from one antivirus company to another. However we can think differently in order to sort this information. We can think in reverse asking ourselves which of this data isn’t interesting. With that question in mind we developed a system to exclude damaged, invalid and broken files from our sample bases. In depth file analysis tell us exactly which files have zero chance of execution on any system flagging them as crapware. But is everything broken to that extent?

If you remember recently we gave you a good idea what to do with broken files and how to implement TitanEngine statical validity analysis to identify and fix broken files. For this purpose we will update the TitanEngine Nexus plugin to automatically identify and fix broken files. This will extend this plugin functionality from creating missing dynamic link library dependencies to fixing every aspect of the broken inputted file. And since the plugin will work automatically it needs to be compatible with all existing unpackers. To achieve this we must recognize the basic dynamic unpacker model which looks like this:

As we can see from this flow chart all dynamic unpackers share a certain logic model. Perfect place for Nexus to handle the inputted file is at the start of unpacking process which is achieved by hooking TitanEngine's function IsPE32FileValidExW. This function is called before the unpacking process starts by all unpackers and if it estimates the file as invalid or broken unpacking is aborted. So what does our hook need to do? List of steps to do would be:

• Perform statical validity analysis by calling IsPE32FileValid
• Determine if file is valid or not and if it isn't do the following
  • Create a backup for inputted file
  • Perform statical file fixing by calling FixBrokenPE32FileEx
  • Validate the file fixing success
  • Return TRUE

But this is just the first step because in order to fix the file the TitanEngine can temporarily disable certain fields by removing them from PE header. To revert these changes we must add another hook to revert these changes. Since we are improving Nexus to automatically correct broken files for dynamic unpackers the function to hook is easily recognized as DumpProcessW. This function is called at the start of the unpacking process finalization, just before the necessary data is exported to file on the disk. That makes this function a perfect place to revert the changes to temporarily disabled PE fields. To do this we just need to call FixBrokenPE32FileEx again with the saved FILE_FIX_INFO structure.

By implementing these changes to TitanEngine's Nexus plugin we convert it to all purpose dynamic unpacker helper module because with its help we can unpack broken files and files that are missing their dependencies. And all this done with no modification to the source code of any unpacker we made in the past. As a demonstration of the plugin capabilities we have attached it and a broken UPX sample file with this blog. Until next week...

TitanEngine ReversingLabs Corporation

Nexus plugin (package contains Nexus plugin, UPX unpacker and a broken sample file)