1.2 Monotone Class Theorem

The next theorem is a version of many theorems of the same type in probability and measure theory.

1.3 Uniqueness of Measures

It is usually impossible to compute P(A) for all sets. An important appliction of the the monotone class theorem is that knowing the values of P on p-system generating \mathcal{A} determines P uniquely.

2.1 Motivation

• Given a probability space (\Omega, \mathcal{A}, P) we think of A \in \mathcal{A} as an event and P(A) is the probability to the event A occurs. Think of this an “observation”: how likely is it that the A occurs.

• A random variable is a more general kind of observation. Think for example that you are performing some measurement: to an outcome \omega \in \Omega you associate e.g. number X(\omega) \in \mathbb{R}. It could also be a vector or even some more general object (e.g. a probability measure!)

• Consider another state space (F,\mathcal{F}) (often we will take (\mathbb{R}, \mathcal{B}) where \mathcal{B} is the Borel \sigma-algebra) and a map X: \Omega \to F We will want to compute P(\{\omega, X(\omega) \in A\})= P(X \in A) = P(X^{-1}(A)) for some A \in \mathcal{F}.

• The notation X^{-1}(A) =\{ \omega\,:\, X(\omega)\in A\} is for the inverse image and for this to make sense we will need X^{-1}(A) \in \Omega.
  All of this motivates the following definitions.

2.2 Measurable functions and random variables

Given a function f:E \to F and B \subset F we write f^{-1}(B)= \left\{ x \in E \,;\, f(x) \in B \right\} for the inverse image. The following properties are easy to verify

• f^{-1}(\emptyset)=\emptyset

• f^{-1}(A\setminus B) = f^{-1}(A) \setminus f^{-1}(B)

• f^{-1}(\bigcup_{i} A_i)=\bigcup_{i} f^{-1}\left(A_i\right)

• f^{-1}(\bigcap_{i} A_i ) = \bigcap_{i} f^{-1}\left(A_i\right)

2.3 Operations on measurable functions

Composition of functions f: E \to F, \quad \quad g: F \to G, \quad \quad g\circ f: E \to G

Like continuity is preserved by composition so is measurability.

Given a function f:E \to \mathbb{R} we define positive/negative parts
f_+ = f \vee 0, \quad f_- = -( f \wedge 0) \quad \implies \quad f=f_+-f_- ,\quad |f|=f_++f_-

2.4 Simple functions

Remarks

• The A_i are not necessarily disjoint.

• A function is simple if and only if it takes finitely many different values (at most 2^N values including 0)

• The decomposition is not unique.

2.5 Supremum, infimum, limits

As we see next measurability is preserved by basic operations, in particular taking limits.

Refresher on \limsup and \liminf of sequences: Recall the definitions of \liminf and \limsup for sequences of real numbers (they always exists if we allow the values \pm \infty.) \liminf_n a_n = \sup_{n} \inf_{m \ge n} a_m = \lim_{n} \inf_{m \ge n} a_m = \textrm{ smallest accumulation point of } \{a_n\} \limsup_n a_n = \inf_{n} \sup_{m \ge n} a_m = \lim_{n} \sup_{m \ge n} a_m = \textrm{ largest accumulation point of } \{a_n\}

\lim_n a_n \textrm{ exists } \iff \liminf_n a_n = \limsup_n a_n

We have then

2.6 Approximation by simple functions

The following theorem is very important, because it reduces many a computation about measurable function to a computation about a simple function and then taking a limit. In that context one also uses all the time that any measurable f is the difference of two non-negative measurable functions.

2.7 Extended real-valued function

• Write \overline{\mathbb{R}} = \mathbb{R} \cup \{-\infty, \infty\}.

• Often it is useful to consider function which are allowed to take values \pm \infty.

• The Borel \sigma-algebra on \overline{\mathbb{R}} consists of all sets of the form A, A \cup \{-\infty\}, A \cup \{\infty\}, A \cup \{-\infty, \infty\}.

• This Borel \sigma-algebra is generated by the intervals of the form \{[-\infty, r]\}.

• All properties of measurable functions on f: E \to \mathbb{R} extend to functions f: E \to \overline{\mathbb{R}}: approximation by simple functions, supremeum, infimum, etc…

• We will use all this whenever we need it.

2.8 Homework problems

3.3 Continuous random variables

Another way to define a CDF is to use a PDF (=probability density function).

For now think of the integral as a Riemann integral (e.g. f is piecewise continuous). In particular by the fundamental theorem of Calculus we have F'(t) = f(t)

We will revisit this later when equipped with better integration tools. Many of the classical distributions in probability are given by densities. Here are some examples which will come back.

3.4 Random variables with mixed distribution.

It easy to build a random variable whose distribution is neither discrete nor continous.

Example: Flip a fair coin. If the coins lands on tail you win a prize X uniformly distributed on [0,1] and if the coins lands on tail you loose. Then X has an atom at 0 and F(x)= \left\{ \begin{array}{cl} 0 & x < 0 \\ \frac{1}{2} + \frac{1}{2} x & 0 \le x < 1 \\ 1 & x \ge 1 \end{array} \right.

More generally we can use the concept of mixture

In the previous example we had a (1/2,1/2) mixture of X_1=0 (a discrete RV) and X_2 a uniform RV on [0,1] (a continuous RV).

3.5 Devil’s staircase

We construct here a CDF with remarkable properties

• F(t) has no discontinuities (no atoms)

• F(t) does not have a density, that is F(t) cannot be written as F(t)=\int_0^x f(t) dt.

The construction is based on the Cantor set and F is defined iteratively.

• Set F_0(t)=t

• Define the function F_1 to be equal to \frac{1}{2} on [1/3, 2/3] continuous and linear [0,1] with F(0)=0 and F(1)=1. Then we have |F_1(t)-F_0(t)|< \frac{1}{2}.

• In the second step, let F_2 to be equal to \frac{1}{4} on [1/9, 2/9], unchanged on [1/3, 2/3], \frac{3}{4} on [1/9, 2/9], continuous and piecewise linear [0,1] with F(0)=0 and F(1)=1. We have |F_2(t)-F_1(t)|< \frac{1}{4}.

• Repeat the procedure now on the interval [1/27, 2/27], [7/27, 8/27], [19/27, 20/27], [25/27, 26/27]….

• It is not diificult to see, by induction, that |F_{n}(t)-F_{n-1}(t)| \le \frac{1}{2^{n}} and thus the sequence F_n converges uniformly to a continuous function F(t) which is increasing on [0,1]

3.6 Quantile functions

Intuitively a p-quantile for a RV X, where p \in (0,1), is a value t \in \mathbb{R} where the probability that F_X(t)=P(X\le t) reaches (or crosses over) p. For p=\frac{1}{2} it is usually referred to as the median. More formally

3.7 Application: Construction of probability measures on \mathbb{R}

The most important property of quantile is the following property which shows that Q is a form of functional inverse for F.

3.8 Simulation

In computers are built-in random number generators which generate a uniform RV on [0,1], that a RV whose distribution is P_0.

Example:

• If X has an exponential distribution, then F(t)= \int_0^t \lambda e^{-\lambda s} ds = 1-e^{-\lambda t} and Q(p)= -\frac{1}{\lambda} \ln(1 -p)

• If X is uniform on \{1,2,\cdots,n\} then the quantile function is the function Q(p)=\lceil np\rceil. Recall \lceil x\rceil is the smallest integer equal or greater than x.

• If X is a normal RV then the CDF is F(t)=\int_{-\infty}^t \frac{e^{-x^2/2}}{\sqrt{2\pi}} dx. The quantile Q=F^{-1} has no closed form, but there exists excellent numerical routine to compute it. This can be used to generate normal random variables.

The inverse methods has its limitation and we will learn other simulation methods later on.

4.2 Monotone Convergence

We extend monotonicity to general non-negative RVs.

The next theorem (Monotone convergence Theorem) is very useful in itself and, in addition, the other convergence theorems for expectations derive from it.

4.3 Further properties of the expectation

Remark: The monotone convergence theorem shows that if X_n is any sequence of simple function increasing to X then E[X]=\lim_n E[X_n].

We will extend the linearity of expectation to general function later after we have developed more theory.

4.4 Negligible sets and completion of a measure space

Let us discuss here a bit carefully sets of probability 0.

It is a fine point of measure theory that negligible set need not be measurable. This is true for example for the Borel \sigma-algebra and Lebesgue measure (see your Math 623 class for more details) and this related to the existence of non- Borel measurable sets.

There is a standard procedure, which is called the completion of a probability space to deal with such issue. The idea is to extends the \sigma-algebra and the probability measure P in such a way all negligible sets are measurable and without changing the probability assigned to sets of positive probability.

4.5 Almost sure properties

Generally speaking, almost sure properties are property which are true except possibly on a set of measure 0 (or on a neglgible set).
For example

• We say that two RVs X and Y are equal almost surely if P(X=Y)= P(\{\omega\,:\, X(\omega)=Y(\omega)\})=1 that is X and Y differ on a negligible set. We write X=Y a.s.

• If X=Y almost surely then E[X]=E[Y]. Indeed then the simple approximations satisfies X_n=Y_n almost surely. If two simple random variables are equal almost surely then their expectations are equal (use their canonical form to see this).

• We say, for example, that X \ge Y a.s if P(\{\omega\,:\, X(\omega)\ge Y(\omega)\}=1.

• We say X_n converges to X almost surely if there exists a set of measure 0, N, such that for all \omega \in \Omega \setminus N we have \lim_{n}X_n(\omega)=X(\omega).

4.6 Fatou’s Lemma

Our first convergence theorem was the monotone convergence theorem Theorem 4.2. Our second convergence theorem still deals with non-negative function random variables and is called the Fatou’s lemma.

4.9 Examples

Example: gamma random variable. The gamma random variable X has density f(x)= \left\{ \begin{array}{cl} \frac{\beta^\alpha}{\Gamma(\alpha)} x^{\alpha -1} e^{- \beta x} & x \ge 0 \\ 0 & x<0 \end{array} \right. and the Gamma function \Gamma(\alpha) given by \Gamma(\alpha)= \int_0^\infty x^{\alpha -1} e^{-x} dx \,.

Let us compute E[X^\delta] for some \delta >0. Using the expectation rule we find \begin{aligned} E[X^\delta]& = \int_0^\infty x^\delta \frac{\beta^\alpha}{\Gamma(\alpha)} x^{\alpha -1} e^{- \beta x} dx = \frac{\Gamma(\alpha + \delta)}{\Gamma(\alpha) \beta^\delta} \underbrace{\int_0^\infty \frac{\beta^{\alpha + \delta}}{\Gamma(\alpha+ \delta)} x^{\alpha + \delta -1} e^{- \beta x} dx}_{=1} \end{aligned} If \delta=n is an integer then we can use that \Gamma(\alpha+1)=\alpha \Gamma(\alpha) and so E[X^n]= \frac{\alpha (\alpha +1) \cdots \alpha + (n-1)}{\beta^\alpha}

4.10 Location and scale

We investigate how the pdf of a random variable transform under a linear transformation X \mapsto Y= aX+b.

Remark Alternatively you can prove this using the CDF, for example for a>0 F_Y(t)=P (Y \le t) = P(aX+b \le t) = P\left(X\le \frac{y-b}{a}\right)= F_X\left(\frac{y-b}{a}\right) and then differentiate. Do the case a<0.

5.1 Jensen inequality

An equivalent description of a convex fuction (“obvious” from a picture, with a proof in the homework) is that at any point x_0 we can find a supporting hyperplane: that is there exists a plane l(x) in \mathbb{R}^n \times \mathbb{R} which is tangent to the graph of \phi at x_0 (and thus l(x)=\phi(x_0) + c \cdot (x-x_0)) and such that the graph of \phi lies above l for all x, i.e. we have \color{blue} \phi(x) \ge \phi(x_0) + c\cdot(x-x_0) for all x \in \mathbb{R}^d.

If \phi is differentiable at x_0 the plane is given by the tangent plane to the graph at x_0, we have \color{blue} \phi(x) \ge \phi(x_0) + \nabla \phi(x_0)\cdot(x-x_0)

If f\phi is twice continuously differentiable then \phi is convex if and only if the matrix of second derivative D^2\phi(x) is positive definite for all x.

5.2 L^p-norm of random variables

Suppose (\Omega,\mathcal{A},P) is a probability space and X is a real-valued random variable.

Remarks

• It is easy to check that \|X\|_p=0 \implies X=0 \text{ almost surely}, \|cX\|_p = c \|X\|_p.

• \|X\|_p <\infty means that |X|^p is integrable (if 1 \le p < \infty) and that X is almost surely bounded (if p=\infty). Often \|X\|_\infty is called the essential supremum of X.

5.4 Markov, Chebyshev, and Chernov

Another very important inequality is the so-called Markov equality. Very simple and very useful.

5.5 Completeness of L^p spaces

The L^p spaces are normed vector spaces. It is a nice application of the Borel-Cantelli Lemma and Markov inequality that these spaces are complete.

5.6 Homework problems

6.1 L^2 spaces and duality

Among the L^p spaces, the space L^2 plays a special role because, on top of beinng a complete normed vector space, it is also a Hilbert space that is a complete inner product vector space space for the inner product \langle X \,,\, Y \rangle =E[XY] and the L^2 norm derives from the inner product. \| X\|_2^2 = \langle X\,,\, X\rangle

Hilbert space have all kind of good properties of which we are going to need one in this class.

It is not too difficult to show that B' is itself a normed vector space with \|l\| = \sup_{x \not =0}\frac{\|l(x)\|}{\|x\|}.

6.2 Absolute continuity and Radon-Nikodym Theorem

As we have seen in Exercise 4.3 we can build new probability mesures using densities. If P is a probabilty measure and Y \ge 0 is a random variable with E[Y]=\int Y dP =1 then we can build a new probabilty measure Q by setting Q(A) = E[1_A Y] = \int 1_A Y dP and we have then \int X dQ = \int X Y dP. The integral notation makes clear what probability measure is used. Other conventions are to use E_Q[X] =E_P[XY] in which the subscript indicates which probability measure we are using to define the expectation.

The notion makes sense at the forml level since \int X dQ = \int X \underbrace{\frac{dQ}{dP}}_{=Y} dP