EECS495: Stochastic Models for Web 2.0

EECS495 Stochastic Models for Web 2.0

Course Information Sheet

Instructor: Dr. Vijay G. Subramanian

Contact Information:

Office: L313 Technological Institute
Tel: 467-5168, E-mail: vjsubram at OR v-subramanian at
Office Hours: Tuesdays 11-noon or by appointment

Time and Place: 3.30-4.50 M Tech 164, 12.30-1.50 W Searle 2407


Reference Texts:

Course Overview:
Starting with Napster and now with applications like Bit-Torrent, peer-to-peer networking has come to be one of the mainstays of networking.
Similarly, applications to online advertising have lead to a revival of the classical multi-armed bandit problem.
The purpose of this course is to discuss analytical models and theoretical foundations underlying these two application topics.

Course Handouts: Papers to be covered in course will be distributed in class or put up on this website (subject to copyright issues).

Problem Sets: Problems sets will be assigned on a quasi-weekly schedule.

Exams: There will be no exam for this course. However, there will be a final project.

Final Project: A portion of your grade will be based on a final project. The project can be on any topic related to the material covered in this
course and is expected to contain an analytical component. Possibilities include extending a result discussed in class, a comparison of several
different approaches for a problem in the literature, or performing a simulation study. Your write-up will be graded based on both technical content
and clarity of presentation. Be sure to include a bibliography and adequately cite any references used.

Course Grade: Your final grade will be determined by a mixture of class participation, problems sets and final project. The weightings used will be the following.


  1. Markov chains and processes:
  2. Controlled Markov chains and Markov decision processes
  3. Martingales
  4. Multi-armed bandit problems
  5. Peer-to-peer networking

Project Topics:

  1. Project Topics - list of topics and important references.
  2. Report due date June 6th 2011.

Rough Notes:

  1. Basics - introductory material.
  2. Martingales - Definition, Optional sampling theorem, Martingale convergence theorem, Submartingale inequality, Azuma-Hoeffding inequality.
  3. Discrete-time Markov chains - Definition, Hitting times, Strong Markov property, Communicating classes, Recurrence/Transience, Equilibrium distributions, Convergence to equilibrium, Reversibility, Ergodic theorem.
  4. Continuous-time Markov chains I - Jump times, holding times and jump chain description, Poisson process, characterizations of CTMCs, Strong Markov property, Backward and forward equations.
  5. Continuous-time Markov chains II - Communicating classes, Recurrence, Transience, Invariant distribution, Convergence to equilibrium, Reversibility, Ergodic theorem.
  6. Foster-Lyapunov I - Levy martingale for DTMCs, Recurrence and transience criteria via martingales and test functions.
  7. Foster-Lyapunov II - Foster-Lyapunov criterion and generalizations for DTMCs.
  8. Foster-Lyapunov III - Foster-Lyapunov for CTMCs and Kurtz's theorem for density dependent CTMCs.
  9. Controlled Markov Chains - Fully observed Markov decision processes in discrete-time, dynamic programming, Bellman equation, total cost, discounted cost and long-run average cost.
  10. Bandit Problems - Multi-armed bandit problems, Gittins index, no regret formulation, non-stochastic/adversial bandits.
  11. Analytical Models for P2P Networks - CTMC model for P2P, Mean-field limit, Stability analysis.
  • Complete course notes - All parts above have been collected together in single document.

    Homework Assignments:

    1. Homework1 - due Wed April 6th 2011.
    2. Homework2 - due Mon April 25th 2011.
    3. Homework3 - due Wed May 4th 2011.
    4. Homework4 - due Mon May 16th 2011.
    5. Homework5 - due Wed May 25th 2011.
    6. Homework6 - due Wed June 8th 2011.
  • See complete notes above.


    1. Probability with engineering applications - Course notes by Prof. Bruce E. Hajek, Univ. of Illinois at Urbana-Champaign.
    2. An exploration of random processes for engineers - Course notes by Prof. Bruce E. Hajek, Univ. of Illinois at Urbana-Champaign.
    3. Communication network analysis - Course notes by Prof. Bruce E. Hajek, Univ. of Illinois at Urbana-Champaign.
    4. Probability Notes - Prof. Chris King, Northeastern University.