Optimization of randomized neural networks for transfer operator approximation

Researchers have proposed a new algorithm that optimizes the activation function of a randomized neural network architecture called RaNNDy, keeping its randomly initialized weights and biases fixed, to improve the approximation of mathematical operators for complex systems. [1][2] The architecture, known as RaNNDy, is designed for the data-driven approximation of transfer operators associated with complex dynamical systems. In a randomized neural network, the weights and biases of the hidden layers are randomly initialized and then kept static; only the output layer undergoes training. This approach offers a closed-form solution for the output layer and significantly lower training costs compared to fully optimized networks. [1][2] A neural network's artificial neurons compute an output via a non-linear function of their total inputs, a mechanism called the activation function. [3] The performance of RaNNDy is constrained by its initial, random parameter selection, which defines the basis functions for the operator approximation. Because these basis functions are determined by the activation function, its selection is critical. [1][2] The new method directly optimizes this activation function to provide a more suitable dictionary for the approximation task. [1][2] The researchers illustrated the efficacy of their approach using various benchmark problems, including stochastic differential equations and random walks on graphons. [1][2] Randomized neural networks differ from architectures like convolutional neural networks (CNNs), which learn features by optimizing filters, or kernels, through automated learning and are a standard for image processing tasks. [4] The work also contrasts with other machine learning paradigms, such as proximal policy optimization, a reinforcement learning algorithm used for training intelligent agents. [5]