EECS495: Stochastic Models for Web 2.0
EECS495 Stochastic Models for Web 2.0
Course Information Sheet
Instructor: Dr. Vijay G. Subramanian
Contact Information:
Office: L313 Technological Institute
Tel: 4675168, Email: vjsubram at eecs.northwestern.edu OR vsubramanian at northwestern.edu
Office Hours: Tuesdays 11noon or by appointment
Time and Place:
3.304.50 M Tech 164, 12.301.50 W Searle 2407
Prerequisites

Good understanding of basic probability (e.g. ECE302)

Some familiarity with Markov chains (e.g. EECS454/EECS422/EECS495 Randomized Algorithms/IEMS460)
Reference Texts:

S. P. Meyn and R. L. Tweedie, Markov chains and stochastic stability,
Second edition, Cambridge University Press, Cambridge, 2009.

J. R. Norris, Markov chains, Cambridge Series in Statistical and Probabilistic Mathematics, 2, Cambridge University Press, Cambridge, 1998.

N. CesaBianchi and G. Lugosi, Prediction, learning, and games, Cambridge University Press, Cambridge, 2006.

P. R. Kumar and P. Varaiya, Stochastic systems: Estimation, identification and adaptive control, Prentice Hall, Englewood Cliffs, N. J., 1986.

M. Mitzenmacher and E. Upfal, Probability and computing: Randomized algorithms and probabilistic analysis, Cambridge University Press, Cambridge, 2005.

Relevant papers.
Course Overview:
Starting with Napster and now with applications like BitTorrent, peertopeer networking has come to be one of the mainstays of networking.
Similarly, applications to online advertising have lead to a revival of the classical multiarmed bandit problem.
The purpose of this course is to discuss analytical models and theoretical foundations underlying these two application topics.
Course Handouts:
Papers to be covered in course will be distributed in class or put up on this website (subject to copyright issues).
Problem Sets:
Problems sets will be assigned on a quasiweekly schedule.
Exams:
There will be no exam for this course. However, there will be a final project.
Final Project:
A portion of your grade will be based on a final project. The project can be on any topic related
to the material covered in this
course and is expected to contain an analytical component. Possibilities
include extending a result discussed in class, a comparison of several
different approaches for a
problem in the literature, or performing a simulation study. Your
writeup will be graded based on both technical content
and clarity of presentation. Be sure to include a
bibliography and adequately cite any references used.
Course Grade:
Your final grade will be determined by a mixture of class participation, problems sets and final project. The weightings used will be the following.
 Problem sets 30%
 Class participation 10%
 Final project 60%
Syllabus:
 Markov chains and processes:
 A review and state classification
 Positive recurrence conditions: FosterLyapunov theorem
 Transience conditions
 Asymptotic analysis of Markov processes (Kurtz's theorem)
 Controlled Markov chains and Markov decision processes
 Discounted cost problems
 Average cost problems
 Martingales
 Stopping times
 Convergence theorems and HoeffdingAzuma inequality
 Multiarmed bandit problems
 Gittins index
 Noregret policies
 Applications: online advertising, spectrum allocation
 Peertopeer networking
 Mathematical models
 Stability analysis
Project Topics:

Project Topics  list of topics
and important references.

Report due date June 6th 2011.
Rough Notes:

Basics  introductory material.

Martingales  Definition, Optional sampling theorem, Martingale convergence theorem, Submartingale inequality, AzumaHoeffding inequality.

Discretetime Markov chains  Definition, Hitting times, Strong Markov property, Communicating classes, Recurrence/Transience, Equilibrium distributions, Convergence to equilibrium, Reversibility, Ergodic theorem.

Continuoustime Markov chains I  Jump times, holding times and jump chain description, Poisson process, characterizations of CTMCs, Strong Markov property, Backward and forward equations.

Continuoustime Markov chains II  Communicating classes, Recurrence, Transience, Invariant distribution, Convergence to equilibrium, Reversibility, Ergodic theorem.

FosterLyapunov I  Levy martingale for DTMCs, Recurrence and transience criteria via martingales and test functions.

FosterLyapunov II  FosterLyapunov criterion and generalizations for DTMCs.

FosterLyapunov III  FosterLyapunov for CTMCs and Kurtz's theorem for density dependent CTMCs.

Controlled Markov Chains  Fully observed Markov decision processes in discretetime, dynamic programming, Bellman equation, total cost, discounted cost and longrun average cost.

Bandit Problems  Multiarmed bandit problems, Gittins index, no regret formulation, nonstochastic/adversial bandits.

Analytical Models for P2P Networks  CTMC model for P2P, Meanfield limit, Stability analysis.
Complete course notes  All parts above have been collected together in single document.
Homework Assignments:

Homework1  due Wed April 6th 2011.

Homework2  due Mon April 25th 2011.

Homework3  due Wed May 4th 2011.

Homework4  due Mon May 16th 2011.

Homework5  due Wed May 25th 2011.

Homework6  due Wed June 8th 2011.
See complete notes above.
Links:

Probability with engineering applications  Course notes by Prof. Bruce E. Hajek, Univ. of Illinois at UrbanaChampaign.

An exploration of random processes for engineers  Course notes by Prof. Bruce E. Hajek, Univ. of Illinois at UrbanaChampaign.

Communication network analysis  Course notes by Prof. Bruce E. Hajek, Univ. of Illinois at UrbanaChampaign.

Probability Notes  Prof. Chris King, Northeastern University.