The entry point of the packed file gives us a vague clue on how interesting this analysis will be. It looks like this:

  PUSHAD
  CALL L002
L002:
  POP EBP
  SUB EBP,00401006
  CALL 00407036
  JMP L007
  DB E9
L007:
  MOV EAX,DWORD PTR SS:[ESP+C]
  JMP SHORT 0040701E

The usual method of getting the code offset delta via CALL/POP, followed by simple obfuscations and the first packer controlled exception. Luckily for us the whole first protection layer is unprotected, so basic stub functions are accessible from start. They include antidebugging functions and more importantly, the decompression of the main protection layer and the transfer of control to it. This first layer's primary point of interest is the layer control transfer. To locate that part of the code we scroll down a bit and spot the aplib decompression code. That recognizable code:

  PUSHAD
  MOV ESI,DWORD PTR SS:[ESP+24]
  MOV EDI,DWORD PTR SS:[ESP+28]
  CLD
  MOV DL,80
  XOR EBX,EBX
  MOVS BYTE PTR ES:[EDI],BYTE PTR DS:[ESI]
...
  SUB EDI,DWORD PTR SS:[ESP+28]
  MOV DWORD PTR SS:[ESP+1C],EDI
  POPAD
  RET

It is located at offset +0x108 relative to the entry point. Since it is common for protectors to compress and/or encrypt their layers, setting a hardware breakpoint here seems like a good place to start. After a few exceptions we finally hit it, and we can collect the needed data from this function's input parameters. If we take a look at the stack we can see these numbers:

0012FF50   00407AB9  RETURN to 00407AB9 from 00407108 ;Called from
0012FF54   00408531  00408531 ;Packed layer content
0012FF58   00330000           ;Allocated layer buffe

If we look at the address from which the layer decompression was called, we see:

  CALL 00407108 ;aPLib decompression
  ADD ESP,8
  PUSHAD ;AntiTracing via RDTSC
 ...
  ADD ESP,4
  RDTSC
  MOV EBX,EAX
...
  ADD ESP,4
  MOV ECX,EDX
...
  ADD ESP,4
  RDTSC
  SUB EAX,EBX
...
  ADD ESP,4
  SBB EDX,ECX
  RDTSC
  ADD EAX,EBX
...
  ADD ESP,4
  ADC EDX,ECX
  RDTSC
  SUB EAX,EBX
...
  ADD ESP,4
...
  ADD ESP,4
  SBB EDX,ECX
...
  ADD ESP,4
  TEST EDX,EDX
  JNZ SHORT 00407B36 ;Executed if code is traced
  POPAD

This antitrace code is repeated throughout the code many times, and it is also present in all API redirections. Following this is a single exception that is executed just before control  is transferred to second protection layer. Instruction JMP EDI, which is located just above the custom memory checksum algorithm, transfers the control to the next layer. That next layer starts like this:

  JMP L003
L001:
  JMP L004
  ???
L003:
  JMP L001
L004:
  CALL L036
  TEST AL,83
  LES EAX,FWORD PTR DS:[EAX+EDX*4]
  NOP
  NOP
  NOP
  NOP
 

This protection layer holds all information necessary for unpacker coding in this layer. Imports are handled at these locations:

+0x9A5:  PUSH EBX
  LEA EDX,DWORD PTR SS:[EBP+401108]
  CALL NEAR EDX                       ;decompress IAT data
  ADD ESP,8
  MOV EDI,DWORD PTR SS:[EBP+4023A9]
+0x9B7:  CMP BYTE PTR DS:[EDI],0C3
  JNZ 00330A8C
  ADD EDI,2
  PUSH EDI
  CALL NEAR DWORD PTR SS:[EBP+4024C8] ;GetModuleHandleA
  TEST EAX,EAX
...
+0xA95:  INC EDI
  PUSH EDI
  PUSH DWORD PTR SS:[EBP+40247C]
  CALL NEAR DWORD PTR SS:[EBP+4024C4] ;GetProcAddress
  XOR EBX,EBX
  PUSHAD
 

And this is where we have a choice to either place breakpoints at these locations or analyze a memory buffer holding IAT data. That buffer is decompressed just before it is processed. If we examine it, we see the following pattern:

typedef struct ALEX_IAT_DLL{
       BYTE DLLSignature; //0xC3
       BYTE DLLNameLength;
       // DLLName[DLLNameLength] followed by 0x00
}ALEX_IAT_DLL, *PALEX_IAT_DLL;
 
typedef struct ALEX_IAT_APIENTRY{
       BYTE APINameLength;
       // APIName[APINameLength] followed by 0x00
       BYTE RedirectionNumber; //Number of redirections
       DWORD RedirectionAddress[RedirectionNumber];
}ALEX_IAT_APIENTRY, *PALEX_IAT_APIENTRY;

Either choice is a good one. However AlexProtector uses a protection technique called "import elimination" which means that parts of the code section which are used as call gates to Windows API are damaged and must be fixed. This data is stored in the import structure we just analyzed. However there is a problem with this since we don't know the original location of the import table and we must reserve some space for it. The solution to this problem is to reserve space in the new section but a detailed description on how to do this will be done next week in part two, when we create the unpacker for AlexProtector.

Solving the problem of imports leaves us with just one more problem, the entry point. After some quick tracing we find this code part:

+0x107B:  MOV ESI,DWORD PTR SS:[EBP+402385] ;SS:[00408385]=004012E6
  ADD EAX,EBX
  MOV BYTE PTR DS:[EAX],0E9 ;Write jump to EP
  INC EAX
  MOV ECX,ESI
...
+0x11E5:  MOV DWORD PTR SS:[ESP+1C],EAX
  POPAD
  JMP NEAR EAX              ;Jump to stolen EP code
...
  MOV EDI,DFB4AF72
  LEA EDI,DWORD PTR DS:[1A58BA5F]
  DEC EDI
  SHRD EDI,ESI,0F2
  BSR EDI,ESI
  TEST EDI,2730DC5C
  TEST EDI,874CDC1C
  AND EDI,9378C643
  JMP 004012E6              ;True jump to EP
 

The first part of this code at +0x107B writes the jump to entry point, more specifically the first instruction after a few instructions that have been stolen. Since those stolen instructions must be executed, the protector creates yet another layer which is executed just before the jump to entry point. That layer contains the stolen instructions, and a jump at the end of that code leads to the protected file code section. Stolen instructions are mixed with the junk instructions so its easiest to dump this layer to new sections and fix the jump to entry point. This was the last piece of information needed to create an unpacker for this protection, which is the topic for next week when we continue working on AlexProtector.

If you only want to unpack AlexProtect protected files you can see how it is done in our video tutorial which shows ImportStudio usage:

Writing an unpacker for AlexProtector should be a nice exercise for any reverser. However we will do that next week, until then...

TitanEngine ReversingLabs Corporation

Samples and Protector (package contains protector binary and samples used)