25.1 Bilinearity in Matrix Notation
As a preliminary to regression, we will express bilinearity in a compact form using matrix notation. The results of this section are not new. They are simply restatements of familiar results about variances and covariances, using new notation and matrix representations.
Let X \mathbf{X} X p × 1 p \times 1 p × 1 m × p m \times p m × p A \mathbf{A} A m × 1 m \times 1 m × 1 b \mathbf{b} b
V a r ( A X + b ) = A Σ X A T Var(\mathbf{AX} + \mathbf{b}) ~ = ~ \mathbf{A}\boldsymbol{\Sigma}_\mathbf{X} \mathbf{A}^T Va r ( AX + b ) = A Σ X A T The results below are special cases of this.
25.1.1 Linear Combinations ¶ To define two generic linear combinations of elements of X \mathbf{X} X
A = [ a 1 a 2 ⋯ a p c 1 c 2 ⋯ c p ] = [ a T c T ] and b = [ b d ] \mathbf{A} ~ = ~
\begin{bmatrix}
a_1 & a_2 & \cdots & a_p \\
c_1 & c_2 & \cdots & c_p
\end{bmatrix}
~ = ~
\begin{bmatrix}
\mathbf{a}^T \\
\mathbf{c}^T
\end{bmatrix}
~~~~~~ \text{and} ~~~~~~
\mathbf{b} ~ = ~
\begin{bmatrix}
b \\
d
\end{bmatrix} A = [ a 1 c 1 a 2 c 2 ⋯ ⋯ a p c p ] = [ a T c T ] and b = [ b d ] Then
A X + b = [ a 1 X 1 + a 2 X 2 + ⋯ + a p X p + b c 1 X 1 + c 2 X 2 + ⋯ + c p X p + d ] = [ a T X + b c T X + d ] \mathbf{AX} + \mathbf{b} ~ = ~
\begin{bmatrix}
a_1X_1 + a_2X_2 + \cdots + a_pX_p + b \\
c_1X_1 + c_2X_2 + \cdots + c_pX_p + d
\end{bmatrix}
~ = ~
\begin{bmatrix}
\mathbf{a}^T\mathbf{X} + b \\
\mathbf{c}^T\mathbf{X} + d
\end{bmatrix} AX + b = [ a 1 X 1 + a 2 X 2 + ⋯ + a p X p + b c 1 X 1 + c 2 X 2 + ⋯ + c p X p + d ] = [ a T X + b c T X + d ] 25.1.2 Covariance of Two Linear Combinations ¶ The covariance of the two linear combinations is the ( 1 , 2 ) (1, 2) ( 1 , 2 ) A X + b \mathbf{AX} + \mathbf{b} AX + b ( 1 , 2 ) (1, 2) ( 1 , 2 ) A Σ X A T \mathbf{A}\boldsymbol{\Sigma}_\mathbf{X}\mathbf{A}^T A Σ X A T
C o v ( a T X + b , c T X + d ) = a T Σ X c Cov(\mathbf{a}^T\mathbf{X} + b, \mathbf{c}^T\mathbf{X} + d)
~ = ~ \mathbf{a}^T \boldsymbol{\Sigma}_\mathbf{X} \mathbf{c} C o v ( a T X + b , c T X + d ) = a T Σ X c 25.1.3 Variance of a Linear Combination ¶ The variance of the first linear combination is the ( 1 , 1 ) (1, 1) ( 1 , 1 ) A Σ X A T \mathbf{A}\boldsymbol{\Sigma}_\mathbf{X}\mathbf{A}^T A Σ X A T
V a r ( a T X + b ) = a T Σ X a Var(\mathbf{a}^T\mathbf{X} + b) ~ = ~
\mathbf{a}^T \boldsymbol{\Sigma}_\mathbf{X} \mathbf{a} Va r ( a T X + b ) = a T Σ X a 25.1.4 Covariance Vector ¶ To predict Y Y Y X \mathbf{X} X Y Y Y X \mathbf{X} X
σ X i , Y = C o v ( X i , Y ) \sigma_{X_i, Y} ~ = ~ Cov(X_i, Y) σ X i , Y = C o v ( X i , Y ) and define the covariance vector of X \mathbf{X} X Y Y Y
Σ X , Y = [ σ X 1 , Y σ X 2 , Y ⋮ σ X p , Y ] \boldsymbol{\Sigma}_{\mathbf{X}, Y} ~ = ~
\begin{bmatrix}
\sigma_{X_1, Y} \\
\sigma_{X_2, Y} \\
\vdots \\
\sigma_{X_p, Y}
\end{bmatrix} Σ X , Y = ⎣ ⎡ σ X 1 , Y σ X 2 , Y ⋮ σ X p , Y ⎦ ⎤ It will be convenient to also have a notation for the transpose of the covariance vector:
Σ Y , X = Σ X , Y T = [ σ X 1 , Y σ X 2 , Y … σ X p , Y ] \boldsymbol{\Sigma}_{Y, \mathbf{X}} ~ = ~ \boldsymbol{\Sigma}_{\mathbf{X}, Y}^T ~ = ~
[\sigma_{X_1, Y} ~ \sigma_{X_2, Y} ~ \ldots ~ \sigma_{X_p, Y}] Σ Y , X = Σ X , Y T = [ σ X 1 , Y σ X 2 , Y … σ X p , Y ] By the linearity of covariance,
C o v ( a T X , Y ) = a T Σ X , Y Cov(\mathbf{a}^T\mathbf{X}, Y) ~ = ~ \mathbf{a}^T \boldsymbol{\Sigma}_{\mathbf{X}, Y} C o v ( a T X , Y ) = a T Σ X , Y