During the past decade the hardware landscape has changed at a dramatic pace. Modern I/O devices are not only capable of delivering very high performance, but also present a wide array of features such as remote memory access, OS-bypassing, offloaded protocol processing, zero-copy data movements, atomic operations, scatter-gather I/O etc.
However, single CPU speed improvements have hit a plateau. As a result, the traditional model of “a fast CPU connected to slow and unintelligent I/O devices” is no longer realistic.
Yet, modern distributed data stores, whose performance is depends considerably upon the performance of networking and storage devices, have largely ignored these features and are still built upon a legacy view of I/O devices from the 1980s. In this project, we explore the ways to integrate modern high-performance I/O devices into a distributed storage paradigm, and evaluate the related design tradeoffs and performance.
The integration of modern I/O devices in a distributed environment presents many interesting challenges and opportunities. First and foremost is the question regarding the suitable interface and abstraction for high-performance I/O. The abstraction must hide a nontrivial state setup and the performance idiosyncrasies of devices.
Second, how can we design data-processing applications to leverage a new I/O interface? Applications should refrain from any slow operation such as memory allocation or connection establishment etc. in fast I/O accesses. Last but not the least, most modern I/O devices perform state-full I/O processing.
Hence issues such as how to handle the complexity of distributed device management, setup, scaling and resource allocation remain open and challenging issues. Integration also presents an opportunity to rethink the traditional division of labor among the application, the operating system, and I/O devices.
RStore is an ongoing exploratory project designed from scratch to leverage the capabilities of modern I/O devices in the next generation of distributed, data store architecture.
We are experimenting with high-performance networks (iWARP and Infiniband) with non-volatile memories (flash or PCM), and use DRAM storage as a representative of NVM performance characteristics.
RStore's I/O stack is built upon the principles of high-performance networking, and separates the data path from the control path. The control path sets up states in a distributed environment — an action that involves allocating storage capacity, preparing I/O devices, opening connections and notification channels etc. The data path leverages Remote Direct Memory Access (RDMA) operations for remote data accesses.
This separation is achieved by using its intuitive memory-mapped I/O API (mmap and friends), to designate which applications can reserve, allocate, and map storage capacity in a distributed environment for data storage and access.
In order to harness the network and storage parallelism, RStore also stripes data across multiple devices and servers. RStore achieves high aggregate bandwidth and close-to-hardware latency (within 98% of the theoretical maximum) to the network limits.
We have developed a low-latency, distributed graph processing framework and a distributed Key-Value sorter using RStore’s memory-like API.
References
- "RStore: A Direct-Access DRAM-based Data Store,"
Animesh Trivedi, Patrick Stuedi, Bernard Metzler, Clemens Lutz, Martin Schmatz, Thomas R. Gross,
to appear in the 35th IEEE International Conference on Distributed Computing Systems (ICDCS'15), Columbus, Ohio, USA, June 29th - July 2nd, 2015.