Optimizing the Reservoir of a Recurrent Neural Network Using Gradient-Free Methods

Abstract

Reservoir computing is a framework for training recurrent neural networks in which the recurrent weight matrix, called the reservoir, is typically randomly generated, and only the output weights are trained. This approach has proven effective in applications like speech recognition and chaotic time series prediction, utilizing the reservoir's ability to capture temporal dependencies and its high-dimensional state space. Despite the success of this paradigm, we postulate that a randomly generated reservoir that has weights scaled as 1/N, where N is the number of neurons, may not be the most optimal for training. This study investigates alternative weight initialization techniques using a Gaussian mixture model and a piecewise linear spline distribution to generate an optimal reservoir. We aim to determine the network weight scaling coefficient through these methods to compare it to the conventional 1/N scaling. Our approach involves simulating networks with an increasing number of neurons using the proposed initialization techniques to fit a network weight scaling coefficient. The parameters of the proposed weight initialization techniques, such as the means and standard deviations of the Gaussian mixture model and the shape and scaling of the piecewise linear distribution, are found through gradient-free optimization methods, including evolutionary algorithms and particle swarm optimization. We show that these weight initialization methods effectively solve oscillator and input-driven tasks. Further, we observe that network size does not significantly impact performance, and the optimal scaling is task-dependent and deviates significantly from the conventional 1/N scaling law.