EECS495 Stochastic Models for Web 2.0

Course Information Sheet

Instructor: Dr. Vijay G. Subramanian

Contact Information:

Time and Place: 3.30-4.50 M Tech 164, 12.30-1.50 W Searle 2407

Prerequisites

• Good understanding of basic probability (e.g. ECE302)
  
• Some familiarity with Markov chains (e.g. EECS454/EECS422/EECS495 Randomized Algorithms/IEMS460)
  

Reference Texts:

• S. P. Meyn and R. L. Tweedie, Markov chains and stochastic stability, Second edition, Cambridge University Press, Cambridge, 2009.
• J. R. Norris, Markov chains, Cambridge Series in Statistical and Probabilistic Mathematics, 2, Cambridge University Press, Cambridge, 1998.
• N. Cesa-Bianchi and G. Lugosi, Prediction, learning, and games, Cambridge University Press, Cambridge, 2006.
• P. R. Kumar and P. Varaiya, Stochastic systems: Estimation, identification and adaptive control, Prentice Hall, Englewood Cliffs, N. J., 1986.
• M. Mitzenmacher and E. Upfal, Probability and computing: Randomized algorithms and probabilistic analysis, Cambridge University Press, Cambridge, 2005.
• Relevant papers.

Course Overview:
Starting with Napster and now with applications like Bit-Torrent, peer-to-peer networking has come to be one of the mainstays of networking.
Similarly, applications to online advertising have lead to a revival of the classical multi-armed bandit problem.
The purpose of this course is to discuss analytical models and theoretical foundations underlying these two application topics.

Course Handouts: Papers to be covered in course will be distributed in class or put up on this website (subject to copyright issues).

Problem Sets: Problems sets will be assigned on a quasi-weekly schedule.

Exams: There will be no exam for this course. However, there will be a final project.

Final Project: A portion of your grade will be based on a final project. The project can be on any topic related to the material covered in this
course and is expected to contain an analytical component. Possibilities include extending a result discussed in class, a comparison of several
different approaches for a problem in the literature, or performing a simulation study. Your write-up will be graded based on both technical content
and clarity of presentation. Be sure to include a bibliography and adequately cite any references used.

Course Grade: Your final grade will be determined by a mixture of class participation, problems sets and final project. The weightings used will be the following.

• Problem sets 30%
• Class participation 10%
• Final project 60%

Syllabus:

1. Markov chains and processes:
   • A review and state classification
   • Positive recurrence conditions: Foster-Lyapunov theorem
   • Transience conditions
   • Asymptotic analysis of Markov processes (Kurtz's theorem)
2. Controlled Markov chains and Markov decision processes
   • Discounted cost problems
   • Average cost problems
3. Martingales
   • Stopping times
   • Convergence theorems and Hoeffding-Azuma inequality
4. Multi-armed bandit problems
   • Gittins index
   • No-regret policies
   • Applications: online advertising, spectrum allocation
5. Peer-to-peer networking
   • Mathematical models
   • Stability analysis

Project Topics:

1. Project Topics - list of topics and important references.
2. Report due date June 6th 2011.

Rough Notes:

1. Basics - introductory material.
2. Martingales - Definition, Optional sampling theorem, Martingale convergence theorem, Submartingale inequality, Azuma-Hoeffding inequality.
3. Discrete-time Markov chains - Definition, Hitting times, Strong Markov property, Communicating classes, Recurrence/Transience, Equilibrium distributions, Convergence to equilibrium, Reversibility, Ergodic theorem.
4. Continuous-time Markov chains I - Jump times, holding times and jump chain description, Poisson process, characterizations of CTMCs, Strong Markov property, Backward and forward equations.
5. Continuous-time Markov chains II - Communicating classes, Recurrence, Transience, Invariant distribution, Convergence to equilibrium, Reversibility, Ergodic theorem.
6. Foster-Lyapunov I - Levy martingale for DTMCs, Recurrence and transience criteria via martingales and test functions.
7. Foster-Lyapunov II - Foster-Lyapunov criterion and generalizations for DTMCs.
8. Foster-Lyapunov III - Foster-Lyapunov for CTMCs and Kurtz's theorem for density dependent CTMCs.
9. Controlled Markov Chains - Fully observed Markov decision processes in discrete-time, dynamic programming, Bellman equation, total cost, discounted cost and long-run average cost.
10. Bandit Problems - Multi-armed bandit problems, Gittins index, no regret formulation, non-stochastic/adversial bandits.
11. Analytical Models for P2P Networks - CTMC model for P2P, Mean-field limit, Stability analysis.