Fundamental Information-Theoretic Limits of Distributed Information Retrieval and Processing

Abstract

We consider an information theoretic study of the challenges facing distributed information processing and retrieval systems, in which the data is partitioned, stored, and processed across distributed machines/workers. In our research, we propose practical solutions leveraging tools from information and coding theory. In large-scale systems, there are several challenges in moving towards distributed algorithms compared to other centralized approaches. These challenges include communication bottlenecks due to data movement across machines, latency due to heterogeneity among working nodes, and privacy concerns from untrustworthy service providers. In this dissertation, we address the above challenges by a) devising novel coding schemes and optimal storage design to minimize the communication overhead needed to shuffle the data among distributed nodes; b) minimizing the impact of heterogeneity via novel iterative work exchange and balancing among nodes, and c) characterizing the fundamental limits of privately retrieving information from distributed databases under the following practical constraints: limited storage at databases, tolerable privacy leakage, and privacy for hidden latent variables.Data shuffling between a distributed cluster of nodes is one of the critical steps in implementing large-scale learning algorithms. Randomly shuffling the data-set among a cluster of workers allows different nodes to obtain fresh data assignments at each learning epoch. This process has been shown to provide statistical improvements in the learning process (via testing and training error). However, the statistical benefits of distributed data shuffling come at the cost of extra communication overhead from the master node to worker nodes, and can act as one of the major bottlenecks in the overall time for computation. Another major bottleneck that adversely impacts the time efficiency is the computational heterogeneity of distributed nodes, often limiting the task completion time due to the slowest worker. In our first contribution, we propose new approaches to increase the time efficiency in distributed computing systems. First, we study how to use codes and exploit excess storage at workers in a principled manner in order to reduce communication overhead for distributed data shuffling. Then, we present our approach of work exchange to combat the latency problem, in which faster workers can be reassigned additional leftover computations that were originally assigned to slower workers. In distributed retrieval systems, assuring privacy while retrieving information from public databases has become a crucial need for users. Private information retrieval (PIR) allows a user to retrieve a desired message from a set of databases without revealing the identity of the desired message. The replicated database scenario, where N databases store each of the K messages was considered by Sun and Jafar, and the optimal download cost was characterized. In our second objective, we consider a practical scenario where the databases are not replicated and have storage limitations. In particular, we study the problem of PIR from uncoded storage constrained databases, where each database has a limited storage capacity and is only allowed to store uncoded data. The novel aspect of this work is to characterize the optimum download cost of PIR from uncoded storage constrained databases for any storage value. In our third and final objective, we consider other practical scenarios for the PIR model where perfect privacy is not necessarily required. These scenarios arise in applications where some leakage in the privacy is tolerable with the goal of enhancing the PIR capacity. We focus on two models for privacy leakage. In the first model, we assume asymmetric bounded leakage for both user privacy, i.e., message identity and database privacy, i.e, information user obtains about unwanted messages. We refer to this model as Asymmetric Leaky PIR (AL-SPIR). We study the three-way tradeoff between user privacy, database privacy, and communication efficiency of PIR. In the second model, we propose a novel relaxed privacy definition for PIR. Instead of hiding the message index queried by the user, we focus on providing information-theoretic privacy for latent traits. We model the user profile with a latent variable model captured by a latent random variable S. Using this new privacy notion, also refereed to as Latent Variable PIR (LV-PIR), we show how the PIR download cost from a single database can be reduced.