You are here

Adiabatic Markov Decision Process with application to queuing systems

TitleAdiabatic Markov Decision Process with application to queuing systems
Publication TypeConference Paper
Year of Publication2013
AuthorsDuong, T., D. Nguyen-Huu, and T. Nguyen
Conference Name47th Annual Conference on Information Sciences and Systems (CISS)
Pagination1 - 6
Date Published03/2013
Conference LocationBaltimore, MD, USA
ISBN Number978-1-4673-5238-3
Keywordsadiabatic, Markov decision process, value Iteration

Markov Decision Process (MDP) is a well-known framework for devising the optimal decision making strategies under uncertainty. Typically, the decision maker assumes a stationary environment which is characterized by a time-invariant transition probability matrix. However, in many real-world scenarios, this assumption is not justified, thus the optimal strategy might not provide the expected performance. In this paper, we study the performance of the classic Value Iteration (VI) algorithm for solving an MDP problem under non-stationary environments. Specifically, the non-stationary environment is modeled as a sequence of time-variant transition probability matrices governed by an adiabatic evolution inspired from quantum mechanics. We characterize the performance of the VI algorithm subject to the rate of change of the underlying environment. The performance is measured in terms of the convergence rate to the optimal average reward. We show two examples of queuing systems that make use of our analysis framework.