Title:
Improving operating systems security: two case studies

dc.contributor.advisor Pu, Calton
dc.contributor.author Wei, Jinpeng en_US
dc.contributor.committeeMember Ahamad, Mustaque
dc.contributor.committeeMember Blough, Douglas
dc.contributor.committeeMember Giffin, Jonathon
dc.contributor.committeeMember Li, Kang
dc.contributor.department Computing en_US
dc.date.accessioned 2010-01-29T20:00:22Z
dc.date.available 2010-01-29T20:00:22Z
dc.date.issued 2009-08-14 en_US
dc.description.abstract Malicious attacks on computer systems attempt to obtain and maintain illicit control over the victim system. To obtain unauthorized access, they often exploit vulnerabilities in the victim system, and to maintain illicit control, they apply various hiding techniques to remain stealthy. In this dissertation, we discuss and present solutions for two classes of security problems: TOCTTOU (time-of-check-to-time-of-use) and K-Queue. TOCTTOU is a vulnerability that can be exploited to obtain unauthorized root access, and K-Queue is a hiding technique that can be used to maintain stealthy control of the victim kernel. The first security problem is TOCTTOU, a race condition in Unix-style file systems in which an attacker exploits a small timing gap between a file system call that checks a condition and a use kernel call that depends on the condition. Our contributions on TOCTTOU include: (1) A model that enumerates the complete set of potential TOCTTOU vulnerabilities; (2) A set of tools that detect TOCTTOU vulnerabilities in Linux applications such as vi, gedit, and rpm; (3) A theoretical as well as an experimental evaluation of security risks that shows that TOCTTOU vulnerabilities can no longer be considered "low risk" given the wide-scale deployment of multiprocessors; (4) An event-driven protection mechanism and its implementation that defend Linux applications against TOCTTOU attacks at low performance overhead. The second security problem addressed in this dissertation is kernel queue or K-Queue, which can be used by the attacker to achieve continual malicious function execution without persistently changing either kernel code or data, which prevents state-of-the-art kernel integrity monitors such as CFI and SBCFI from detecting them. Based on our successful defense against a concrete instance of K-Queue-driven attacks that use the soft timer mechanism, we design and implement a solution to the general class of K-Queue-driven attacks, including (1) a unified static analysis framework and toolset that can generate specifications of legitimate K-Queue requests and the checker code in an automated way; (2) a runtime reference monitor that validates K-Queue invariants and guards such invariants against tampering; and (3) a comprehensive experimental evaluation of our static analysis framework and K-Queue Checkers. en_US
dc.description.degree Ph.D. en_US
dc.identifier.uri http://hdl.handle.net/1853/31849
dc.publisher Georgia Institute of Technology en_US
dc.subject Control flow integrity en_US
dc.subject Security and protection en_US
dc.subject Reliability en_US
dc.subject File systems management en_US
dc.subject.lcsh Operating systems (Computers) Security measures
dc.title Improving operating systems security: two case studies en_US
dc.type Text
dc.type.genre Dissertation
dspace.entity.type Publication
local.contributor.advisor Pu, Calton
local.contributor.corporatename College of Computing
local.relation.ispartofseries Doctor of Philosophy with a Major in Algorithms, Combinatorics, and Optimization
relation.isAdvisorOfPublication fc48a3de-da43-4d32-af59-414047eb7cd7
relation.isOrgUnitOfPublication c8892b3c-8db6-4b7b-a33a-1b67f7db2021
relation.isSeriesOfPublication 186126ed-fc79-4186-8523-2cb526aa622e
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
wei_jinpeng_200912_phd.pdf
Size:
783.39 KB
Format:
Adobe Portable Document Format
Description: