Signal Based Reinforcement Learning: Studying the Behavior of Adaptive Agents on Signal Flows

Abstract

Today's world is increasingly populated by non-human intelligences. Embedded in our phones, homes, cities and more, these artificial intelligences have grown to play a significant role in our societies. When certain types of AI agents interact, their collective behavior resembles that of complex systems, a category of system better known for describing emergent behavior in ant colonies or flocks of birds. To better prepare for a world driven by complex systems of AI agents, it is important to improve both our scientific understanding and engineering capability in the area. This dissertation responds to that need through the development and use of the Signal Based Reinforcement Learning (SBRL) approach, a new method of analyzing and constructing learning systems which introduces complex systems concepts to the fields of reinforcement learning (RL) and Evolutionary Computation (EC). The approach focuses on the flows of signals in systems and on the way that the adaptive agents in those systems adapt to those flows. By focusing on the key concepts of signal flow and adaptive agents, the SBRL approach provides an alternate lens with which to view existing methods within RL and EC. After establishing the context and intuition behind the SBRL approach in Chapters 1 and 2, the remainder of the dissertation further develops the concept though a set of related applications. Chapters 3 and 4 illustrate the utility of the SBRL approach for algorithm development. Chapter 3 harnesses the signal and agent abstraction of SBRL to develop a hybrid algorithm which integrates the genetic algorithm and concepts from reinforcement learning. The chapter illustrates how, by viewing an algorithm as a system of agents exchanging signals, and then modifying the flow of those signals, performance can be improved. Chapter 4 reformulates the concept of the policy from RL as an adaptive agent, and uses this abstracted view to develop the stochastic policy mixing (SPM) methodology. The chapter shows how the behavior of multiple semi-independent agents can be coordinated to jointly affect the decision process in a reinforcement learning problem. Also included in the chapter are examples of the use of the SPM methodology to integrate a priori knowledge and other types of problem solvers into a policy mix. Finally, Chapter 5 illustrates how the SBRL approach may be embodied in software. The chapter describes the design of a new tool, CASA, which supports the exploration of SBRL system designs and the impact of adjustments to signal flows and multi-agent interactions. Each chapter contributes related but distinct new knowledge. Chapter 3 contributes an improvement to the standard GA for sparse reward environments as well as an improved understanding of the interaction between mutation rate and the likelihood of epsilon greedy devation from the policy. The chapter's introductory material also highlights a potentially fruitful connection between the study of the genotype-phenotype translation process in evolutionary computation and the multi-agent implementation of a hybrid GA/RL algorithm. Chapter 4 contributes a novel structure (SPM) for the combination of multiple policies and agents for the solution of a reinforcement learning problem. This addresses the problem of a lack of such a structure in the literature, which prevents the non-expert implementation of multi-policy methods and prevents the effective comparison of the same. The experiments in the chapter illustrate the ease with which a priori knowledge and external algorithms can be introduced within the structure by testing new policy agent combinations on a standard RL benchmark. The chapter also contributes a discussion of the potential pitfalls of combining policy agents in this way. Finally, Chapter 5 contributes a novel software, intended to support the development and use of SBRL type systems by non-experts and experts alike. The contribution of this software is primarily practical, as it provides a visual, Python-based, distributed design tool with support for the adaptive publish-subscribe-like signal routing necessary for SBRL work. It also contributes new knowledge of how the concepts of SBRL and complex adaptive systems can be embodied in modern software, especially in regards to signal routing. The contributions of this work matter because they begin to address the scientific challenge of better understanding and harnessing the emergent behavior which can arise from the interaction of multiple AI agents. While existing contemporary work in AI primarily focuses on large black box approaches, these methods ignore or minimize the possibility of complex interactions among components, which is highly likely in future technological environments in which multiple AIs are present. To address this gap in understanding, the chapters of this dissertation conduct work which, while related to contemporary methods, deviates in structure and, importantly, in the introduction of interactions which may lead to emergent behavior. The chapters are connected in their purpose as explorations of the effects of those interactions, a focus on the methodology needed to effectively study multi-agent emergent behavior within AI, and their contribution of extensible structures to support further exploration of SBRL concepts. By conducting this work, the chapters of this dissertation contribute to a new foundation for the scientific study of the rapidly growing AI ecosystems in our technological world.