Computer-Assisted Proofs of Chaos in the Long Short-Term Memory Model

Abstract

Long short-term memory (LSTM) is a type of recurrent neural network (RNN) that is usedin sequence modeling which aims to mitigate the vanishing gradient problem. When the input is a constant zero sequence, the LSTM model can be viewed as a discrete dynamical system which can exhibit chaotic properties. In this article, we take a closer look at these chaotic properties which include period doubling cascades and positive Lyapunov exponents. We also present computer-assisted proofs of chaos in this model. For the one-dimensional LSTM family we start be proving the existence of a topological horseshoe with positive topological entropy using methods developed by Zgliczynski (1997) for multiple parameters. Then, using this horseshoe, we prove the existence of an absolutely continuous invariant measure with positive Lyapunonv exponent for a positive measure set of parameters. Finally, we conjecture, for the two-dimensional family, the existence of an SRB measure for a positive measure set of parameters.