(Georgia Institute of Technology, 2013-08)
Lin, Zhiyuan; Chau, Duen Horng
Large graphs with billions of nodes and edges are increasingly common, calling for new kinds of scalable computation frameworks. Although popular, distributed approaches can be expensive to build, or require many resources to manage or tune. State-of-the-art approaches such as GraphChi and TurboGraph recently have demonstrated that a single machine can efficiently perform advanced computation on billion-node graphs. Although fast, they both use sophisticated data structures, memory management, and optimization techniques. We propose a minimalist approach that forgoes such complexities, by leveraging the memory mapping capability found on operating systems. Our experiments on large datasets, such as a 1.5 billion edge Twitter graph, show that our streamlined approach achieves up to 26 times faster than GraphChi, and comparable to TurboGraph. We con- tribute our crucial insight that by leveraging memory mapping, a fundamental operating system capability, we can outperform the latest graph computation techniques.
(Georgia Institute of Technology, 2013)
Sabrin, Kaeser Md.; Lin, Zhiyuan; Chau, Duen Horng; Lee, Ho; Kang, U.
Large graphs with billions of nodes and edges are increasingly common, calling for new kinds of scalable computation frameworks.
State-of-the-art approaches such as GraphChi and TurboGraph recently demonstrated that a single PC can efficiently perform advanced computation on billion-node graphs. Although fast, they use
sophisticated data structures, explicit memory management, and optimization techniques to achieve high speed and scalability. We propose a
minimalist
approach that forgoes such complexities,
by leveraging the fundamental
memory mapping
(MMap) capability
found on operating systems. We present multiple, major findings; we
contribute: (1) our crucial insight that MMap can be a viable technique for creating fast, scalable graph algorithms that surpass some
of the best techniques; (2) a
counterintuitive
result that
we can do less
and gain more
; MMap enables us to use a much simpler data structure
(edge list) and algorithm design, and to defer memory management
to the OS, while offering significantly faster or comparable performance as highly-optimized methods (e.g., 10
X as fast as GraphChi
PageRank on
1.47
billion edge Twitter graph); (3) we performed extensive experiments on real and synthetic graphs, including the
6.6 billion edge YahooWeb graph, and show that MMap’s benefits sustain in most conditions. We hope this work will inspire others to explore how memory mapping may help improve other methods or
algorithms to further increase their speed and scalability.