Addressing Design Trade-Offs for Practical Security Tools: Contributions to SGX I/O Performance and Directed Fuzzing Path Exploration

Security research often involves trade-offs that hinder the practical deployment of its solutions. In trusted execution environments such as Intel Software Guard Extensions (SGX), one such trade-off is the high cost of IO operations, which users must accept to gain the security guarantees of enclave execution. In a different context, directed fuzzing faces its own trade-off: state-of-the-art directed fuzzers often prioritize quickly reaching target locations, at the expense of path diversity, which limits the depth and coverage of security testing. This thesis addresses these trade-offs through focused solutions, each targeting a specific limitation in the broader areas of trusted execution environments and directed fuzzing. In the context of trusted execution environments, this thesis addresses the long-standing IO performance limitations of Intel SGX. Existing kernel-bypass solutions allow fast IO within enclaves but often expand the Trusted Computing Base (TCB), increase the attack surface, and complicate deployment. This work presents RAKIS, a system that enables enclave programs to securely access fast IO Linux kernel primitives without requiring any changes to user applications. RAKIS follows a security-by-design approach, maintaining a minimal and rigorously tested TCB, while achieving substantial performance improvements—demonstrating up to 4.6× higher network throughput and a 2.8× average speedup across real-world workloads compared to state-of-the-art SGX Library Operating System (LibOS) solutions. Complementing this systems-focused contribution, the second part of this thesis addresses a core limitation in directed fuzzing: its tendency to favor quick target hits at the expense of path diversity. Existing directed graybox fuzzers (DGFs) typically prioritize inputs that follow the shortest path to a target location, but this often results in inefficient exploration and missed coverage due to complex control-flow structures. This work introduces BULLSEYE, a DGF that leverages closeness centrality—a graph-theoretic metric capturing structural reachability—to guide exploration along more diverse paths. BULLSEYE also incorporates a novel program discovery mechanism to monitor fuzzing progress and dynamically adjust exploration intensity. Evaluated on 30 real-world targets, BULLSEYE achieves up to 31% higher directed coverage and generates 3x more unique paths to the target compared to prior DGFs, leading to improved bug reproduction and patch testing capabilities. Together, these contributions show that addressing practical trade-offs—such as performance limitations in trusted execution environments and reduced path diversity in directed fuzzing—can help improve the usability and reliability of security tools. By focusing on specific, real-world challenges and proposing targeted solutions, this thesis contributes to ongoing efforts in making security mechanisms and analysis tools more practical and effective in modern computing environments.

Date

2025-07-30

Resource Type

Text

Resource Subtype

Dissertation

Full item page