Estimation of non-stationary Markov Chain transition models

— Many decision systems rely on a precisely known Markov Chain model to guarantee optimal performance, and this paper considers the online estimation of unknown, nonstationary Markov Chain transition models with perfect state observation. In using a prior Dirichlet distribution on the uncertain rows, we derive a mean-variance equivalent of the Maximum A Posteriori (MAP) estimator. This recursive meanvariance estimator extends previous methods that recompute the moments at each time step using observed transition counts. It is shown that this mean-variance estimator responds slowly to changes in transition models (especially switching models) and a modification that uses ideas of pseudonoise addition from classical filtering is used to speed up the response of the estimator. This new, discounted mean-variance estimator has the intuitive interpretation of fading previous observations and provides a link to fading techniques used in Hidden Markov Model estimation. Our new estimation t...

• Type: Conference
• Year: 2008
• Where: CDC
• Authors: Luca F. Bertuccelli, Jonathan P. How
• Keywords: CDC 2008, Control Systems, Estimator, Mean-variance Estimator, Transition Models