23.1 Random Vectors

Vectors and matrices give us a compact way of referring to random sequences like $X_1, X_2, \ldots, X_n$. The algebra of vectors and matrices gives us powerful tools for studying linear combinations of random variables.

In this section we will develop matrix notation for random sequences and then express familiar consequences of linearity of expectation and bilinearity of covariance in matrix notation. The probability theory in this section is not new – it consists of expectation and covariance facts that you have known for some time. But the representation is new and leads us to new insights.

A vector valued random variable, or more simply, a random vector, is a list of random variables defined on the same space. We will think of it as an $n \times 1$ column vector.

For ease of display, we will sometimes write $\mathbf{X} = [X_1 X_2 \ldots X_n]^T$ where $\mathbf{M}^T$ is notation for the transpose of the matrix $\mathbf{M}$.

The mean vector of $\mathbf{X}$ is $\boldsymbol{\mu} = [\mu_1 ~ \mu_2 ~ \ldots ~ \mu_n]^T$ where $\mu_i = E(X_i)$.

The covariance matrix of $\mathbf{X}$ is the $n \times n$ matrix $\boldsymbol{\Sigma}$ whose $(i, j)$ element is $Cov(X_i, X_j)$.

The $i$th diagonal element of $\boldsymbol{\Sigma}$ is the variance of $X_i$. The matrix is symmetric because of the symmetry of covariance.

23.1.1Linear Transformation: Mean Vector¶

Let $\mathbf{A}$ be an $m \times n$ numerical matrix and $\mathbf{b}$ an $m \times 1$ numerical vector. Consider the $m \times 1$ random vector $\mathbf{Y} = \mathbf{AX} + \mathbf{b}$.

We will call this a “linear transformation” of $\mathbf{X}$ though in fact it is an affine transformation, that is, a composition of the linear transformation $\mathbf{AX}$ and the translation by $\mathbf{b}$.

This representation gives us a compact way to describe multiple linear combinations of $\mathbf{X}$ simultaneously. For example, if $\mathbf{b} = [0 ~~ 0 ~~ 0]^T$ and

then

In general, if $\mathbf{Y} = \mathbf{AX} + \mathbf{b}$ then the $i$th element of $\mathbf{Y}$ is

where $\mathbf{A}_{i*}$ denotes the $i$th row of $\mathbf{A}$ and $\mathbf{b}(i)$ denotes the $i$th element of $\mathbf{b}$. Written longhand,

where $a_{ij}$ is the $(i, j)$ entry of $\mathbf{A}$ and $b_i = \mathbf{b}(i)$.

Thus $Y_i$ is a linear combination of the elements of $\mathbf{X}$. Therefore by linearity of expectation,

Let $\boldsymbol{\mu}_\mathbf{Y}$ be the mean vector of $\mathbf{Y}$. Then by the calculation above,

23.1.2Linear Transformation: Covariance Matrix¶

$Cov(Y_i, Y_j)$ can be calculated using bilinearity of covariance.

This is the $(i, j)$ element of $\mathbf{A}\boldsymbol{\Sigma}\mathbf{A}^T$. So if $\boldsymbol{\Sigma}_\mathbf{Y}$ denotes the covariance matrix of $\mathbf{Y}$, then

Let us see what this formula implies for the variance of a single component of $\mathbf{Y}$.

Any component of $\mathbf{Y}$ is a linear combination of the elements of $\mathbf{X}$ and hence can be written as $\mathbf{aX}+b$ for some $1 \times n$ vector $\mathbf{a}$ and some real number $b$.

The variance of this component of $\mathbf{Y}$ is a diagonal element of $\boldsymbol{\Sigma}_\mathbf{Y}$. By our calculation above, the diagonal element is equal to $\mathbf{a}\boldsymbol{\Sigma}\mathbf{a}^T$.

23.1.3Constraints on $\boldsymbol{\Sigma}$¶

We know that $\boldsymbol{\Sigma}$ has to be symmetric. Also, no matter what $\mathbf{A}$ is, the elements on the main diagonal of $\boldsymbol{\Sigma}_\mathbf{Y}$ must all be non-negative as they are the variances of the elements of $\mathbf{Y}$.

By the observation above, this implies

That is, $\boldsymbol{\Sigma}$ must be positive semidefinite.

Usually, we will be working with covariance matrices that are positive definite, defined by

The reason is that if $\mathbf{a} \boldsymbol{\Sigma} \mathbf{a}^T = 0$ for some $\mathbf{a}$, then the linear transformation $\mathbf{aX}$ has variance 0 and hence is a constant. Hence you can write some of the elements of $\mathbf{X}$ as linear combinations of the others and just study a reduced set of elements.