Sciweavers

Share
COLT
2004
Springer

Performance Guarantees for Regularized Maximum Entropy Density Estimation

11 years 7 months ago
Performance Guarantees for Regularized Maximum Entropy Density Estimation
Abstract. We consider the problem of estimating an unknown probability distribution from samples using the principle of maximum entropy (maxent). To alleviate overfitting with a very large number of features, we propose applying the maxent principle with relaxed constraints on the expectations of the features. By convex duality, this turns out to be equivalent to finding the Gibbs distribution minimizing a regularized version of the empirical log loss. We prove nonasymptotic bounds showing that, with respect to the true underlying distribution, this relaxed version of maxent produces density estimates that are almost as good as the best possible. These bounds are in terms of the deviation of the feature empirical averages relative to their true expectations, a number that can be bounded using standard uniform-convergence techniques. In particular, this leads to bounds that drop quickly with the number of samples, and that depend very moderately on the number or complexity of the feat...
Miroslav Dudík, Steven J. Phillips, Robert
Added 01 Jul 2010
Updated 01 Jul 2010
Type Conference
Year 2004
Where COLT
Authors Miroslav Dudík, Steven J. Phillips, Robert E. Schapire
Comments (0)
books