Sciweavers

SOFSEM
2004
Springer

Approaches Based on Markovian Architectural Bias in Recurrent Neural Networks

13 years 9 months ago
Approaches Based on Markovian Architectural Bias in Recurrent Neural Networks
Recent studies show that state-space dynamics of randomly initialized recurrent neural network (RNN) has interesting and potentially useful properties even without training. More precisely, when initializing RNN with small weights, recurrent unit activities reflect history of inputs presented to the network according to the Markovian scheme. This property of RNN is called Markovian architectural bias. Our work focuses on various techniques that make use of architectural bias. The first technique is based on the substitution of RNN output layer with prediction model, resulting in capabilities to exploit interesting state representation. The second approach, known as echo state networks (ESNs), is based on large untrained randomly interconnected hidden layer, which serves as reservoir of interesting behavior. We have investigated both approaches and their combination and performed simulations to demonstrate their usefulness.
Matej Makula, Michal Cernanský, Lubica Benu
Added 02 Jul 2010
Updated 02 Jul 2010
Type Conference
Year 2004
Where SOFSEM
Authors Matej Makula, Michal Cernanský, Lubica Benusková
Comments (0)