EECS495: Stochastic Models for Web 2.0
EECS495 Stochastic Models for Web 2.0
Course Information Sheet
Instructor: Dr. Vijay G. Subramanian
Contact Information:
Office: L313 Technological Institute
Tel: 467-5168, E-mail: vjsubram at eecs.northwestern.edu OR v-subramanian at northwestern.edu
Office Hours: Tuesdays 11-noon or by appointment
Time and Place:
3.30-4.50 M Tech 164, 12.30-1.50 W Searle 2407
Prerequisites
-
Good understanding of basic probability (e.g. ECE302)
-
Some familiarity with Markov chains (e.g. EECS454/EECS422/EECS495 Randomized Algorithms/IEMS460)
Reference Texts:
-
S. P. Meyn and R. L. Tweedie, Markov chains and stochastic stability,
Second edition, Cambridge University Press, Cambridge, 2009.
-
J. R. Norris, Markov chains, Cambridge Series in Statistical and Probabilistic Mathematics, 2, Cambridge University Press, Cambridge, 1998.
-
N. Cesa-Bianchi and G. Lugosi, Prediction, learning, and games, Cambridge University Press, Cambridge, 2006.
-
P. R. Kumar and P. Varaiya, Stochastic systems: Estimation, identification and adaptive control, Prentice Hall, Englewood Cliffs, N. J., 1986.
-
M. Mitzenmacher and E. Upfal, Probability and computing: Randomized algorithms and probabilistic analysis, Cambridge University Press, Cambridge, 2005.
-
Relevant papers.
Course Overview:
Starting with Napster and now with applications like Bit-Torrent, peer-to-peer networking has come to be one of the mainstays of networking.
Similarly, applications to online advertising have lead to a revival of the classical multi-armed bandit problem.
The purpose of this course is to discuss analytical models and theoretical foundations underlying these two application topics.
Course Handouts:
Papers to be covered in course will be distributed in class or put up on this website (subject to copyright issues).
Problem Sets:
Problems sets will be assigned on a quasi-weekly schedule.
Exams:
There will be no exam for this course. However, there will be a final project.
Final Project:
A portion of your grade will be based on a final project. The project can be on any topic related
to the material covered in this
course and is expected to contain an analytical component. Possibilities
include extending a result discussed in class, a comparison of several
different approaches for a
problem in the literature, or performing a simulation study. Your
write-up will be graded based on both technical content
and clarity of presentation. Be sure to include a
bibliography and adequately cite any references used.
Course Grade:
Your final grade will be determined by a mixture of class participation, problems sets and final project. The weightings used will be the following.
- Problem sets 30%
- Class participation 10%
- Final project 60%
Syllabus:
- Markov chains and processes:
- A review and state classification
- Positive recurrence conditions: Foster-Lyapunov theorem
- Transience conditions
- Asymptotic analysis of Markov processes (Kurtz's theorem)
- Controlled Markov chains and Markov decision processes
- Discounted cost problems
- Average cost problems
- Martingales
- Stopping times
- Convergence theorems and Hoeffding-Azuma inequality
- Multi-armed bandit problems
- Gittins index
- No-regret policies
- Applications: online advertising, spectrum allocation
- Peer-to-peer networking
- Mathematical models
- Stability analysis
Project Topics:
-
Project Topics - list of topics
and important references.
-
Report due date June 6th 2011.
Rough Notes:
-
Basics - introductory material.
-
Martingales - Definition, Optional sampling theorem, Martingale convergence theorem, Submartingale inequality, Azuma-Hoeffding inequality.
-
Discrete-time Markov chains - Definition, Hitting times, Strong Markov property, Communicating classes, Recurrence/Transience, Equilibrium distributions, Convergence to equilibrium, Reversibility, Ergodic theorem.
-
Continuous-time Markov chains I - Jump times, holding times and jump chain description, Poisson process, characterizations of CTMCs, Strong Markov property, Backward and forward equations.
-
Continuous-time Markov chains II - Communicating classes, Recurrence, Transience, Invariant distribution, Convergence to equilibrium, Reversibility, Ergodic theorem.
-
Foster-Lyapunov I - Levy martingale for DTMCs, Recurrence and transience criteria via martingales and test functions.
-
Foster-Lyapunov II - Foster-Lyapunov criterion and generalizations for DTMCs.
-
Foster-Lyapunov III - Foster-Lyapunov for CTMCs and Kurtz's theorem for density dependent CTMCs.
-
Controlled Markov Chains - Fully observed Markov decision processes in discrete-time, dynamic programming, Bellman equation, total cost, discounted cost and long-run average cost.
-
Bandit Problems - Multi-armed bandit problems, Gittins index, no regret formulation, non-stochastic/adversial bandits.
-
Analytical Models for P2P Networks - CTMC model for P2P, Mean-field limit, Stability analysis.
Complete course notes - All parts above have been collected together in single document.
Homework Assignments:
-
Homework1 - due Wed April 6th 2011.
-
Homework2 - due Mon April 25th 2011.
-
Homework3 - due Wed May 4th 2011.
-
Homework4 - due Mon May 16th 2011.
-
Homework5 - due Wed May 25th 2011.
-
Homework6 - due Wed June 8th 2011.
See complete notes above.
Links:
-
Probability with engineering applications - Course notes by Prof. Bruce E. Hajek, Univ. of Illinois at Urbana-Champaign.
-
An exploration of random processes for engineers - Course notes by Prof. Bruce E. Hajek, Univ. of Illinois at Urbana-Champaign.
-
Communication network analysis - Course notes by Prof. Bruce E. Hajek, Univ. of Illinois at Urbana-Champaign.
-
Probability Notes - Prof. Chris King, Northeastern University.