Beyond second-order statistics for learning: A pairwise interaction model for entropy estimation

Second order statistics have formed the basis of learning and adaptation due to its appeal and analytical simplicity. On the other hand, in many realistic engineering problems requiring adaptive solutions, it is not sufficient to consider only the second order statistics of the underlying distributions. Entropy, being the average information content of a distribution, is a better-suited criterion for adaptation purposes, since it allows the designer to manipulate the information content of the signals rather than merely their power. This paper introduces a nonparametric estimator of Renyi's entropy, which can be utilized in any adaptation scenario where entropy plays a role. This nonparametric estimator leads to an interesting analogy between learning and interacting particles in a potential field. It turns out that learning by second order statistics is a special case of this interaction model for learning. We investigate the mathematical properties of this nonparametric entropy ...

• Type: Journal
• Year: 2002
• Where: NC
• Authors: Deniz Erdogmus, José Carlos Príncipe, Kenneth E. Hild II
• Keywords: Entropy, NC 2002, Neural Networks, Nonparametric Estimator, Order Statistics