EECS495: Stochastic Models for Web 2.0
EECS495 Stochastic Models for Web 2.0
Course Information Sheet
Instructor: Dr. Vijay G. Subramanian
Office: L313 Technological Institute
Tel: 467-5168, E-mail: vjsubram at eecs.northwestern.edu OR v-subramanian at northwestern.edu
Office Hours: Tuesdays 11-noon or by appointment
Time and Place:
3.30-4.50 M Tech 164, 12.30-1.50 W Searle 2407
Good understanding of basic probability (e.g. ECE302)
Some familiarity with Markov chains (e.g. EECS454/EECS422/EECS495 Randomized Algorithms/IEMS460)
S. P. Meyn and R. L. Tweedie, Markov chains and stochastic stability,
Second edition, Cambridge University Press, Cambridge, 2009.
J. R. Norris, Markov chains, Cambridge Series in Statistical and Probabilistic Mathematics, 2, Cambridge University Press, Cambridge, 1998.
N. Cesa-Bianchi and G. Lugosi, Prediction, learning, and games, Cambridge University Press, Cambridge, 2006.
P. R. Kumar and P. Varaiya, Stochastic systems: Estimation, identification and adaptive control, Prentice Hall, Englewood Cliffs, N. J., 1986.
M. Mitzenmacher and E. Upfal, Probability and computing: Randomized algorithms and probabilistic analysis, Cambridge University Press, Cambridge, 2005.
Starting with Napster and now with applications like Bit-Torrent, peer-to-peer networking has come to be one of the mainstays of networking.
Similarly, applications to online advertising have lead to a revival of the classical multi-armed bandit problem.
The purpose of this course is to discuss analytical models and theoretical foundations underlying these two application topics.
Papers to be covered in course will be distributed in class or put up on this website (subject to copyright issues).
Problems sets will be assigned on a quasi-weekly schedule.
There will be no exam for this course. However, there will be a final project.
A portion of your grade will be based on a final project. The project can be on any topic related
to the material covered in this
course and is expected to contain an analytical component. Possibilities
include extending a result discussed in class, a comparison of several
different approaches for a
problem in the literature, or performing a simulation study. Your
write-up will be graded based on both technical content
and clarity of presentation. Be sure to include a
bibliography and adequately cite any references used.
Your final grade will be determined by a mixture of class participation, problems sets and final project. The weightings used will be the following.
- Problem sets 30%
- Class participation 10%
- Final project 60%
- Markov chains and processes:
- A review and state classification
- Positive recurrence conditions: Foster-Lyapunov theorem
- Transience conditions
- Asymptotic analysis of Markov processes (Kurtz's theorem)
- Controlled Markov chains and Markov decision processes
- Discounted cost problems
- Average cost problems
- Stopping times
- Convergence theorems and Hoeffding-Azuma inequality
- Multi-armed bandit problems
- Gittins index
- No-regret policies
- Applications: online advertising, spectrum allocation
- Peer-to-peer networking
- Mathematical models
- Stability analysis
Project Topics - list of topics
and important references.
Report due date June 6th 2011.
Complete course notes - All parts above have been collected together in single document.
Basics - introductory material.
Martingales - Definition, Optional sampling theorem, Martingale convergence theorem, Submartingale inequality, Azuma-Hoeffding inequality.
Discrete-time Markov chains - Definition, Hitting times, Strong Markov property, Communicating classes, Recurrence/Transience, Equilibrium distributions, Convergence to equilibrium, Reversibility, Ergodic theorem.
Continuous-time Markov chains I - Jump times, holding times and jump chain description, Poisson process, characterizations of CTMCs, Strong Markov property, Backward and forward equations.
Continuous-time Markov chains II - Communicating classes, Recurrence, Transience, Invariant distribution, Convergence to equilibrium, Reversibility, Ergodic theorem.
Foster-Lyapunov I - Levy martingale for DTMCs, Recurrence and transience criteria via martingales and test functions.
Foster-Lyapunov II - Foster-Lyapunov criterion and generalizations for DTMCs.
Foster-Lyapunov III - Foster-Lyapunov for CTMCs and Kurtz's theorem for density dependent CTMCs.
Controlled Markov Chains - Fully observed Markov decision processes in discrete-time, dynamic programming, Bellman equation, total cost, discounted cost and long-run average cost.
Bandit Problems - Multi-armed bandit problems, Gittins index, no regret formulation, non-stochastic/adversial bandits.
Analytical Models for P2P Networks - CTMC model for P2P, Mean-field limit, Stability analysis.
See complete notes above.
Homework1 - due Wed April 6th 2011.
Homework2 - due Mon April 25th 2011.
Homework3 - due Wed May 4th 2011.
Homework4 - due Mon May 16th 2011.
Homework5 - due Wed May 25th 2011.
Homework6 - due Wed June 8th 2011.
Probability with engineering applications - Course notes by Prof. Bruce E. Hajek, Univ. of Illinois at Urbana-Champaign.
An exploration of random processes for engineers - Course notes by Prof. Bruce E. Hajek, Univ. of Illinois at Urbana-Champaign.
Communication network analysis - Course notes by Prof. Bruce E. Hajek, Univ. of Illinois at Urbana-Champaign.
Probability Notes - Prof. Chris King, Northeastern University.