Markov Chains: A Comprehensive Guide for Computer Science, Summaries of Mathematics

Download Markov Chains: A Comprehensive Guide for Computer Science and more Summaries Mathematics in PDF only on Docsity! Markov Chains Carey Williamson Department of Computer Science University of Calgary  Plan: —Introduce basics of Markov models —Define terminology for Markov chains —Discuss properties of Markov chains —Show examples of Markov chain analysis  On-Off traffic model  Markov-Modulated Poisson Process  Erlang B blocking formula  TCP congestion window evolution Outline Some Terminology (2 of 3)  The time spent in a given state on a given visit is called the sojourn time  Sojourn times are exponentially distributed and independent  Each state i has a parameter q_i that characterizes its sojourn behaviour Some Terminology (3 of 3)  The probability of changing from state i to state j is denoted by p_ij  This is called the transition probability (sometimes called transition rate)  Often expressed in matrix format  Important parameters that characterize the system behaviour Desirable Properties of Markov Chains  Irreducibility: every state is reachable from every other state (i.e., there are no useless, redundant, or dead-end states)  Ergodicity: a Markov chain is ergodic if it is irreducible, aperiodic, and positive recurrent (i.e., can eventually return to a given state within finite time, and there are different path lengths for doing so)  Stationarity: stable behaviour over time