Remote Data Access in Scientific Computing

Abstract

Remote data access is becoming increasingly common in scientific computing. The trends toward collaborative research, big-data, and cloud computing have increased the need to transfer large-scale datasets between geographically separated systems. The net result is that scientific computing is increasingly reliant on data transfers over wide-area networks (WANs), which are often high-cost and low-bandwidth. Furthermore, cluster computing has become the dominant platform for scientific computing, and remote data access is even more complicated when the systems are clusters of computers.This dissertation presents two file systems, Syndicate and Stargate, that provide remote data access over a WAN for scientific computing. Syndicate uses on-demand block-based data transfer augmented with prefetching and multi-tier caching to address the challenges of large-scale data transfer over a low-bandwidth WAN. However, Syndicate is not designed to run on a cluster, therefore it has limitations in network bandwidth use and caching. Stargate, its successor, implements a content-addressable protocol and intra-cluster data caching to avoid redundant data transfers. Stargate performs cluster-to-cluster concurrent data transfers that make more efficient use of network bandwidth. In addition, Stargate uses a novel approach that co-locates computations and transfers to achieve efficient data access in cluster computing, addressing the limitations in Syndicate. As an example, Syndicate was 62% slower than local HDFS with warm caches. In comparison, Stargate’s performance was comparable to local HDFS. Stargate’s performance on benchmarks with heavy workloads was 7% faster than WebHDFS and only 8% slower than local HDFS. Stargate outperformed staging using DistCP. In addition, Stargate’s caches effectively trade high-cost WAN traffic for low-cost LAN traffic. This dissertation explores the viability of on-demand data transfer in scientific computing on clusters. Stargate’s performance and reduction in WAN traffic proves that on-demand data transfer approach can be an acceptable solution for remote data access in scientific computing.