Syllabus Math 606, Spring 2025

Stochastic Processes

Author
Affiliation

Luc Rey-Bellet

University of Massachusetts Amherst

Instructor

Luc Rey-Bellet
LGRT 1423K

Office hours:

  • W 11:00 AM–12:00 PM in LGRT 1423 K or on ZOOM
  • Friday 1:00 PM–2:00 in LGRT 1423 K or on ZOOM
  • By appointment is always possible, we can meet on zoom or in person, and/or ask your questions be email.

Class Meeting

Tu-Th, 11:30AM–12:45PM in LGRT 171

Class slides

These are the class slides from last year. They will be updated as the semester progress and the latest version will be posted here.

Part 1: Markov chains

Part 2: Poisson processes and continuous time Markov chains

Part 3: Martingales

Class homepage

On Canvas at https://umamherst.instructure.com/courses/9591

Syllabus

This is the second part of a 2-semester graduate sequence Math605-Math606 which leads to the Stochastics qualifying exam. Prerequisites are

  1. a solid working knowledge of probablilty (e.g. Math 605, STAT 607 or equivalent).
  2. a solid working knowedge of analysis (at least Math 523 or equivalent) and some basic linear algebra.
  3. some mathematical maturity

In Math 606 we will cover various aspects of stochastic processes. One of the main goal in the class is to develop a “probabilist intuition and way of thinking”. We will present some proofs and we will skip some others in order to provide a reasonably broad range of topics, concepts and techniques. We emphasize examples both in discrete and continuous time from a wide range of disciplines and with an eye on numerical implementation. Among the topics treated in the class are

  1. Discrete time Markov chains on discrete spaces. Definition and basic properties, classification of states (positive recurrence, recurrence and transience), stationary distribution and limit theorems, analysis of transient behavior, optimal stopping, Monte-Carlo Markov chains.
  2. Continuous-Time Markov chains. Poisson Process, Birth and Death Process, and Queueing models. Renewal processes and semi-Markov processes.
  3. Martingales and applications
  4. Brownian motion and applications.

Learning Objectives

The first basic objective is to introduce the basic stochastic processes, learn how to analyze them and apply them, in particular via Monte-Carlo methods. Our goal is to find a good balance between theory, modeling, and implementation and to develop probabilistic intuition.

Grade/assignment

  • There will be a weekly homework submitted via Gradescope https://www.gradescope.com/courses/966887. The homework problem canbe found at the end of each section in the slides. The homeworks will be not graded in detail but I will post solutions for each homework.

  • There will be a few quizzes (in class) to check your knowledge during the semester. The problems will of the same type as the homework problems.

  • There is a comprehensive final exam

  • To extend the reach of the class I expect a final project from you. I am somewhat flexible about topics but I will provide a list to choose from. Talk to me about your ideas. You should select a topic by the end of February. I will expect you to create a poster for a group presentation at the end of the class.

    Here is a list of topics

    • Mixing time of Markov chains: several projects can be carved out starting from the book Markov chain and mixing times by David A. Levin, Yuval Peres, and Elizabeth L. Wilmer.

    • Hidden Markov models

    • Continous-time Markov chains for chemical reactions networks. Widely used in biological and biochemical applications.

    • Queueing models.

    • Markov chains and chaotic dynamical systems. Deterministic chaotic systems can often be described using Markov chains. Learn how to do this.

    • Lyapunov function techniques to analyze Markov chain. Useful for Markov chains with continuous and non-compact state spaces. Good topics for an anal;ysis minded student

    • Quantum Markov chains. Understand the basic of quantum probability and quantum Markov chains.

    • More forthcoming….

  • Grading scheme: Handing you homework consistently 30%, quizz 20%, project 25%, final 25%

Textbooks

The following text book

Probability Theory and Stochastic Processes by Pierre Bremaud. Universitext. Springer 2020. ISBN: 978-3030401825

covers all the material in 605-606 (and some more). My class is designed in a similar spirit and is at the same “intermediate” level. I think it is a really good book to learn the basic ides and concepts, and you can move on to more comprehensive textbooks next.

I have been inspired by many textbooks over the years when preparing my class, see the list below. I strongly suggest you pick a textbook and spend time regularly reading from it, in parallel to the class. The book by Lawler is a marvelous short introduction which covers most topic in the class and is highly recommended as a first read. The books by Resnick and Bremaud are a bit more advanced and are both excellent as well. The topics of simulation and Markov chain is well covered in the books by Ross, Madras, and Rubinstein and Kroese. The book by David A. Levin, Yuval Peres, and Elizabeth L. Wilmer is a great book to learn modern Markov chain techniques.

  • Markov Chains Gibbs Fields, Monte Carlo Simulation, and Queues, 2nd edition (2007) by Pierre Bremaud, Springer.

  • Introduction to Stochastic Processes, 2nd edition (2007) by Gregory F. Lawler, Chapman&Hall.

  • Adventures in Stochastic processes, by Sidney I. Resnick, Birkhauser.

  • Essentials of Stochastic Processes, by Rick Durrett, Springer.

  • Introduction to Probability Models, , by Sheldon M. Ross, Academic Press

  • Lectures on Monte-Carlo Methods, by Neal N. Madras, American Mathematical Society

  • Simulation and the Monte Carlo Method by Reuven Y. Rubinstein and Dirk P. Kroese , Wiley

  • Markov chain and mixing times, by David A. Levin, Yuval Peres, and Elizabeth L. Wilmer, American Mathematical Society

  • Introduction to Stochastic Processes, by Paul G. Hoel, Sidney C. Port and Charles J. Stone, Waveland Press.

  • Stochastic Processes, by Sheldon M. Ross, Wiley.

  • A first course in Stochastic Processes, by Samuel Karlin and Howard M. Taylor, Academic Press.