Reading Group, July 21 2009Based on slides retrieved from http://arch.ece.gatech.edu/present/micro41.pdf

SHARK: Architectural Support for Autonomic Protection Against Stealth by

Rootkit Exploits

Vikas R. Vasisht, Hsien-Hsin S. LeeSchool of Electrical and Computer Engineering

Georgia Tech

Computer Architecture Lab at

2

Rootkits• Programs used to conceal malware presence– E.g. hide keylogger or network activity

• Hard to detect– Anti-rootkit tool executes inside compromised system that

tries to hide the intrusion

3

Simple Rootkit

API Function

User Program

Library

System Administrator (e.g., “ps”, “top”)

Choose Interrupt Handler from IDT

Choose Syscall from SSDT

Syscall Function

ReturnImport Address Table

Interrupt Descriptor Table System Service Descriptor Table

Use

r Spa

ceKe

rnel

Spa

ce

Malware removed from list of

running processes

4

Existing Anti-Rootkit Tools• Signature-Based

Only works with known rootkits

• Behavior-Based– Calculate time and instructions spent on system queries

• Unusually high count indicates the O/S is manipulating response

Too many false positives

• Cross-level system view– Query system in high and low level

• E.g., compare ps response against scheduler lists

The compromised O/S can manipulate all levels

5

Sophisticated (Anti-)Rootkits• Integrity-based detection– Compare snapshot of

memory against trusted baselineCan be fooled

• Shadow-Walker– Mark all pages as unavailable– On TLB-i (execute) page fault:

present malicious code– On TLB-d (snapshot) page

fault: return ‘correct’ page

App Page Table

TLB-i View

TLB-d View

AUDIT OK!

6

Even More Sophisticated Rootkits

M2M1Mal O/S

App2App1Host O/SHardware

App2App1Host O/S

Virtual Machine MonitorHardware

• SubVirt– Install virtual machine

monitor– Downgrade host O/S to

guest O/S– Run Malicious O/S on the

side

• Blue Pill– Install tiny malicious

hypervisor on-the-fly

7

Outline• Motivation– Rootkits

• SHARK– High-level View– Page Table Encryption, Update, Decryption– Address Space Protection– Stealth Checker

• Evaluation• Security Vulnerabilities• Conclusion

8

Problem• Nearly impossible to detect all rootkits– Especially the virtualization-based ones: guest O/S is clean– All tools depend on the malicious O/S

• Proposed solution– Have the processor securely report every application that

enjoys hardware resources– Malicious apps running in different VMM will be exposed

9

SHARK: Bird’s Eye View• Hardware should record running processes– Must force O/S to give real PID to CPU

• Trusted third party audits system– Compare H/W and O/S reports– Discrepancy raises alarm

• O/S cannot hide a malicious app

• Tamper-evident address space– O/S blocked from manipulating normal apps

• Malware hijacking an app will leave a trail

• Idea: Page Table Encryption

10

Page Table Encryption: Key• App’s page table is encrypted using PID– Decrypted on-the-fly during hardware walk using same PID

• O/S assigned PIDs are vulnerable– Frequent reuse problematic for security

• SHARK provides hardware PID (HPID)– 64-bit counter: won’t overflow, nonce

• Only care for memory-resident rootkits, reset will eliminate them

• During app creation– O/S request PID and provides an initialized page table– H/W encrypts page table and returns HPID++

11

Page Table Encryption (cont)• Encrypt last level page table entries (counter mode)• Encrypt first & last level valid bits (counter mode)– Force same PID to be used during walk every time

• All updates through special H/W instruction– MODPT (modify page table)– Must provide correct PID

• Otherwise meaningless 1st level decryption

12

VTranslation Array

Valid Bit Array Encryption

V1

Translation Array

0…11…000

…

11…0Page Directory1 Page Frame

1st 128-bit Valid Bit BlockCounter = HPID||HPID

2nd 128-bit Valid Bit BlockCounter = (HPID||HPID)+1

N 128-bit Valid Bit BlockCounter = (HPID||HPID)+N-1

Counter Base (128 bit) HPID||HPID

Pipelined Counter Mode

Encryption Engine

AES-128

Hardware Secret Key: Burnt-in, unexposable

Offset

Actual Frame stored in memory

13

Page Table: Initial Encryption

VTranslation Array

…1001…

Last LevelPage Directory1 Page Frame

ADDRNULLNULLADDR

Counter (128 bit) HPID||HPID

Pipelined Counter Mode

Encryption Engine

AES-128

Hardware Secret Key: Burnt-in, unexposable

4 entries concatenated

128-bit

VTranslation Array

ADDR

Actual Frame stored in memory

14

Page Table: Entry Update

If other valid entries, decrypt and reencrypt, otherwise just encrypt

Counter (128 bit) (HPID||HPID)+k

V

Page Table Decryption – TLB update

15

V PDE

Counter Mode

DecryptionAES-128

Page DirectoryHW Key

…1001…

Normal Page Table Walk

V PDE

ADDR

Last Level

Counter (128 bit) HPID||HPID

Counter Mode

DecryptionAES-128

HW Key

V…0001…

PDE

ADDR4 entries

encrypted 128-bit

Counter (128 bit) HPID||HPID

Counter Mode

DecryptionAES-128

HW Key

4 entries decrypted

128-bit

TLB update

Address Space Protection• O/S should not tamper with address space of apps– Protect applications from swap-hijacking– Expose malware during auditing

• Shouldn’t mark as unavailable and return convenient page

• H/W signs pages and checks signatures

16

Tamper-evident swappingCaution: No check during first page mapping

Mem Page

4KB

32B

Disk

Counter (128 bit) HPID||HPID

Counter Mode

EncryptionAES-128

HW Key

SHA-256 checksum

Mem Page

4KB

32B

Disk

SHA-256 checksum

Counter Mode

EncryptionAES-128

=?

Valid/Invalid update

17

Stealth Checking• Firmware called on context switch– Record incoming PID of new process– Every 10 switches, timestamp, encrypt and call O/S to ship

Tamper-evident timestamped packet. Lost packets or O/S tampering trigger alarm.

OSContext Switch

HardwareHPID

PID1PID2PID3

Master PID List

Other SHARK H/W

P I D s …

Alarm

18

Outline• Motivation– Rootkits

• SHARK– High-level View– Page Table Encryption, Update, Decryption– Address Space Protection– Stealth Checker

• Evaluation• Security Vulnerabilities• Conclusion

19

Evaluation• Emulation using Bochs

– All installed rootkits detected

• Timing results using Simics– Close match to Core 2 processors– In-order simulation– First 2 billion instructions of SPEC apps

• Most page faults (and overhead) during initialization

20Expected: More TLB entries, less overhead

Evaluation (cont)

21

Outline• Motivation– Rootkits

• SHARK– High-level View– Page Table Encryption, Update, Decryption– Address Space Protection– Stealth Checker

• Evaluation• Security Vulnerabilities• Conclusion

22

Security Hole #1• 1st page map cannot be checked– Possible exploit by cleaner-app?

Mali-cious App’s Page Table

Name: xcalc

Audit app launched:Pagefault

Write Access

Note: cleaner-app will leave a PID trace in the stealth checker

All pages marked as

invalid, app works

through TLBs

Present malicious signature & restore

mapping, but freeze auditing

23

Security Hole #2• All page tables but the last (completely) and first

(valid bits only) are plaintext– O/S can manipulate internal mappings

VALID

VALID

VALID

VALID

View during executionView during auditing

Level 1 Level 2 Level 3 Data Pages

24

Security Hole #3• Security 101– Encryption: to keep something secret, private– Signing: to make something unmodifiable, tamper-evident

and bind message pieces together

• BAD idea to use one for the other– Creates illusion of protection

• Why encrypt the page table?– O/S already knows it, so why keep it a secret?– Signing is necessary, encryption is useless– Paper based on assumption that any tampering will result

in meaningless page table

25

Security Hole #3: Exploit

CounterH/W Key

128-bit plaintext

AES 128

⊕128-bit

Ciphertext

CounterH/W Key

128-bit plaintext

AES 128

⊕ 128-bit Ciphertext

Counter Mode Encryption Counter Mode Decryption

D(E(A)) = E(A) Cloud⊕ = A ⊕ Cloud

⊕ Cloud = A

O/S can overwrite E(A) with: (E(A), A, B known to O/S)E(A) A B = ⊕ ⊕ A Cloud ⊕ ⊕ A B = B Cloud = E(B)⊕ ⊕H/W will normally decrypt to B

E(A) = A Cloud⊕

26

Security Hole #3 (cont)• What they intended to say– The entry for virtual address V, for process PID, at level lvl,

is physical P (next level pointer or data location)

• How it should have been written– P, σΚ(P, V, PID, lvl) (σ = signature)– Bind each page table entry with where it should be located

• What was actually written– Some entries are secret; I won’t tell you

• Though you can ask me• …And you already know

27

Security Hole #4• The virtual memory system is used as a fence– (theoretically) tamper-evident page table

• O/S can simply bypass the VMM– Issue writes to physical address directly– O/S can purge malicious pages prior to auditing

• Similar to cleaner-app, but without any trace

28

Security Vulnerability #1• Stealth checker employs upgradeable firmware• Rootkit that is stored in disk can upgrade firmware– System reboot will not clean system– (Beyond the scope of this paper)

• Compromised O/S may compromise firmware– Enjoy the CPU spying on you– Impossible to detect

29

Security Vulnerability #2• BAD idea to reuse counter of counter-mode

encryption

VTranslation Array

…1001…

Last LevelPage Directory1 Page Frame

ADDRNULLNULLADDR

Counter (128 bit) HPID||HPID

Pipelined Counter Mode

Encryption Engine

AES-128

Hardware Secret Key: Burnt-in, unexposable

4 entries concatenated

128-bit

VTranslation Array

ADDR

Actual Frame stored in memory

Same for every last-level

encryption

30

Conclusion• Rootkits are a growing concern• Hardware monitoring is a good idea for protection– Malware cannot cheat hardware that easily– Good idea for rootkits also: BIOS rootkit in the works

• Paper’s implementation does more harm than good– Broken Algorithm: signing of each page table level is needed

• But O/S can bypass virtual memory manager• Cleaner-app is also a problem

• LBA should target O/S protection

31

Questions?

SYSTEM PROTECTED