Mark Vincent Yason: The Art of Unpacking

Black Hat Briefings, USA 2007 [Audio] Presentations from the security conference.

Unpacking is an art - it is a mental challenge and is one of the most exciting mind games in the reverse engineering field. In some cases, the reverser needs to know the internals of the operating system in order to identify or solve very difficult anti-reversing tricks employed by packers/protectors, patience and cleverness are also major factors in a successful unpack. This challenge involves researchers creating the packers and on the other side, the researchers that are determined to bypass these protections.
The main purpose of this paper is to present anti-reversing techniques employed by executable packers/protectors and also discusses techniques and publicly available tools that can be used to bypass or disable this protections. This information will allow researchers, especially, malcode analysts to identify these techniques when utilized by packed malicious code, and then be able decide the next move when these anti-reversing techniques impedes successful analysis. As a secondary purpose, the information presented can also be used by researchers that are planning to add some level of protection in their software by slowing down reversers from analyzing their protected code, but of course, nothing will stop an skilled, informed, and determined reverser.