On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools

Vogt, Ryan and Puelma Touzel, Maximilian and Shlizerman, Eli and Lajoie, Guillaume (2022) On Lyapunov Exponents for RNNs: Understanding Information Propagation Using Dynamical Systems Tools. Frontiers in Applied Mathematics and Statistics, 8. ISSN 2297-4687

[thumbnail of pubmed-zip/versions/1/package-entries/fams-08-818799/fams-08-818799.pdf] Text
pubmed-zip/versions/1/package-entries/fams-08-818799/fams-08-818799.pdf - Published Version

Download (1MB)


Recurrent neural networks (RNNs) have been successfully applied to a variety of problems involving sequential data, but their optimization is sensitive to parameter initialization, architecture, and optimizer hyperparameters. Considering RNNs as dynamical systems, a natural way to capture stability, i.e., the growth and decay over long iterates, are the Lyapunov Exponents (LEs), which form the Lyapunov spectrum. The LEs have a bearing on stability of RNN training dynamics since forward propagation of information is related to the backward propagation of error gradients. LEs measure the asymptotic rates of expansion and contraction of non-linear system trajectories, and generalize stability analysis to the time-varying attractors structuring the non-autonomous dynamics of data-driven RNNs. As a tool to understand and exploit stability of training dynamics, the Lyapunov spectrum fills an existing gap between prescriptive mathematical approaches of limited scope and computationally-expensive empirical approaches. To leverage this tool, we implement an efficient way to compute LEs for RNNs during training, discuss the aspects specific to standard RNN architectures driven by typical sequential datasets, and show that the Lyapunov spectrum can serve as a robust readout of training stability across hyperparameters. With this exposition-oriented contribution, we hope to draw attention to this under-studied, but theoretically grounded tool for understanding training stability in RNNs.

Item Type: Article
Subjects: Eurolib Press > Mathematical Science
Depositing User: Managing Editor
Date Deposited: 20 Jan 2023 06:24
Last Modified: 18 May 2024 07:03
URI: http://info.submit4journal.com/id/eprint/926

Actions (login required)

View Item
View Item