About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Publication
CloudCom 2010
Conference paper
Correlation based file prefetching approach for Hadoop
Abstract
Hadoop Distributed File System (HDFS) has been widely adopted to support Internet applications because of its reliable, scalable and low-cost storage capability. BlueSky, one of the most popular e-Learning resource sharing systems in China, is utilizing HDFS to store massive courseware. However, due to the inefficient access mechanism of HDFS, access latency of reading files from HDFS significantly impacts the performance of processing user requests. This paper introduces a two-level correlation based file prefetching approach, taking the characteristics of HDFS into consideration, to improve performance by reducing access latency. Four placement patterns to store prefetched data are presented, with policies to achieve trade-off between performance and efficiency of HDFS prefetching. Moreover, a dynamic replica selection algorithm is investigated to improve the efficiency of HDFS prefetching. The proposed prefetching approach has been implemented in BlueSky, and experimental results prove that correlation based file prefetching can significantly reduce access latency therefore improve performance of Hadoop-based Internet applications. © 2010 IEEE.