The Power of Amnesia

10 years 4 months ago
The Power of Amnesia
We propose a learning algorithm for a variable memory length Markov process. Human communication, whether given as text, handwriting, or speech, has multi characteristic time scales. On short scales it is characterized mostly by the dynamics that generate the process, whereas on large scales, more syntactic and semantic information is carried. For that reason the conventionally used xed memory Markov models cannot capture e ectively the complexity of such structures. On the other hand using long memory models uniformly is not practical even for as short memory as four. The algorithm we propose is based on minimizing the statistical prediction error by extending the memory, or state length, adaptively, until the total prediction error is su ciently small. We demonstrate the algorithmby learning the structure of natural English text and applying the learned model to the correction of corrupted text. Using less than 3000 states the model's performance is far superior to that of xed ...
Dana Ron, Yoram Singer, Naftali Tishby
Added 02 Nov 2010
Updated 02 Nov 2010
Type Conference
Year 1993
Where NIPS
Authors Dana Ron, Yoram Singer, Naftali Tishby
Comments (0)