One-Counter Markov Decision Processes

We study the computational complexity of some central analysis problems for One-Counter Markov Decision Processes (OC-MDPs), a class of finitely-presented, countable-state MDPs. OC-MDPs extend finite-state MDPs with an unbounded counter. The counter can be incremented, decremented, or not changed during each state transition, and transitions may be enabled or not depending on both the current state and on whether the counter value is 0 or not. Some states are "random", from where the next transition is chosen according to a given probability distribution, while other states are "controlled", from where the next transition is chosen by the controller. Different objectives for the controller give rise to different computational problems, aimed at computing optimal achievable objective values and optimal strategies. OC-MDPs are in fact equivalent to a controlled extension of (discrete-time) Quasi-Birth-Death processes (QBDs), a purely stochastic model heavily studied ...

• Type: Conference
• Year: 2010
• Where: SODA
• Authors: Tomas Brazdil, Vaclav Brozek, Kousha Etessami, Antonin Kucera, Dominik Wojtczak
• Keywords: Classic Stochastic Model, Counter Value, Discrete Algorithms, Lim Sup Value, SODA 2010