Skip to main content

Lund University Publications

LUND UNIVERSITY LIBRARIES

A new instruction overlapping technique for improved anti-disassembly and obfuscation of x86 binaries

Jämthagen, Christopher LU ; Lantz, Patrik LU and Hell, Martin LU (2013) Workshop on Anti-malware Testing Research (WATeR) p.25-33
Abstract
The problem of correctly recovering assembly instructions from a binary has received much attention and both malware and license validation code often relies on various anti-disassembly techniques in order to complicate analysis. One well-known anti-disassembly technique is to use overlapping code such that the disassembler starts decoding from an incorrect byte, but still recovers valid code. The actual code which is supposed to be executed is instead hidden inside a decoy instruction, and is overlapped with the disassembled code.



We propose and investigate a new novel anti-disassembly method that allows for exceptional flexibility in the hidden instructions, while at the same time providing a disassembled main path... (More)
The problem of correctly recovering assembly instructions from a binary has received much attention and both malware and license validation code often relies on various anti-disassembly techniques in order to complicate analysis. One well-known anti-disassembly technique is to use overlapping code such that the disassembler starts decoding from an incorrect byte, but still recovers valid code. The actual code which is supposed to be executed is instead hidden inside a decoy instruction, and is overlapped with the disassembled code.



We propose and investigate a new novel anti-disassembly method that allows for exceptional flexibility in the hidden instructions, while at the same time providing a disassembled main path that is executable. This allows the approach to be very efficient against static linear sweep disassembly, but also to be more difficult to detect using dynamic analysis methods. The idea is to utilize highly redundant instructions, e.g., multibyte no-operation instructions, and embed the hidden code in the

configurable portions of those instructions. By carefully selecting wrapping instructions, providing overlaps, the hidden execution path can be crafted with great flexibility. We also provide a detection-algorithm, together with testing results, for testing software such that the hidden execution path can be identified. (Less)
Please use this url to cite or link to this publication:
author
; and
organization
publishing date
type
Chapter in Book/Report/Conference proceeding
publication status
published
subject
keywords
overlapping instructions anti-disassembly hidden execution path obfuscation malware x86
host publication
Workshop on Anti-malware Testing Research (WATeR), Montreal, QC, Canada
pages
25 - 33
publisher
IEEE - Institute of Electrical and Electronics Engineers Inc.
conference name
Workshop on Anti-malware Testing Research (WATeR)
conference location
Montreal, Canada
conference dates
2013-10-30
external identifiers
  • wos:000332987700004
  • scopus:84893681398
ISBN
978-1-4799-2476-9
DOI
10.1109/WATeR.2013.6707878
language
English
LU publication?
yes
id
12be0b5a-9e2f-431c-a4be-74494faf6064 (old id 4058297)
date added to LUP
2016-04-04 10:51:38
date last changed
2022-03-31 17:32:34
@inproceedings{12be0b5a-9e2f-431c-a4be-74494faf6064,
  abstract     = {{The problem of correctly recovering assembly instructions from a binary has received much attention and both malware and license validation code often relies on various anti-disassembly techniques in order to complicate analysis. One well-known anti-disassembly technique is to use overlapping code such that the disassembler starts decoding from an incorrect byte, but still recovers valid code. The actual code which is supposed to be executed is instead hidden inside a decoy instruction, and is overlapped with the disassembled code.<br/><br>
<br/><br>
We propose and investigate a new novel anti-disassembly method that allows for exceptional flexibility in the hidden instructions, while at the same time providing a disassembled main path that is executable. This allows the approach to be very efficient against static linear sweep disassembly, but also to be more difficult to detect using dynamic analysis methods. The idea is to utilize highly redundant instructions, e.g., multibyte no-operation instructions, and embed the hidden code in the<br/><br>
configurable portions of those instructions. By carefully selecting wrapping instructions, providing overlaps, the hidden execution path can be crafted with great flexibility. We also provide a detection-algorithm, together with testing results, for testing software such that the hidden execution path can be identified.}},
  author       = {{Jämthagen, Christopher and Lantz, Patrik and Hell, Martin}},
  booktitle    = {{Workshop on Anti-malware Testing Research (WATeR), Montreal, QC, Canada}},
  isbn         = {{978-1-4799-2476-9}},
  keywords     = {{overlapping instructions anti-disassembly hidden execution path obfuscation malware x86}},
  language     = {{eng}},
  pages        = {{25--33}},
  publisher    = {{IEEE - Institute of Electrical and Electronics Engineers Inc.}},
  title        = {{A new instruction overlapping technique for improved anti-disassembly and obfuscation of x86 binaries}},
  url          = {{https://lup.lub.lu.se/search/files/78489284/nop_obfs.pdf}},
  doi          = {{10.1109/WATeR.2013.6707878}},
  year         = {{2013}},
}