Bilinearity in Matrix Notation
As a preliminary to regression, we will express bilinearity in a compact form using matrix notation. The results of this section are not new. They are simply restatements of familiar results about variances and covariances, using new notation and matrix representations.
Let $\mathbf{X}$ be a $p \times 1$ vector of predictor variables. We know that for an $m \times p$ matrix $\mathbf{A}$ and an $m \times 1$ vector $\mathbf{b}$,
The results below are special cases of this.
Linear Combinations
To define two generic linear combinations of elements of $\mathbf{X}$, let
Then
Covariance of Two Linear Combinations
The covariance of the two linear combinations is the $(1, 2)$ element of the covariance matrix of $\mathbf{AX} + \mathbf{b}$, which is the $(1, 2)$ element of $\mathbf{A}\boldsymbol{\Sigma}_\mathbf{X}\mathbf{A}^T$.
Variance of a Linear Combination
The variance of the first linear combination is the $(1, 1)$ element of $\mathbf{A}\boldsymbol{\Sigma}_\mathbf{X}\mathbf{A}^T$.
Covariance Vector
To predict $Y$ based on $\mathbf{X}$ we will need to work with the covariance of $Y$ and each of the elements of $\mathbf{X}$. Let
and define the covariance vector of $\mathbf{X}$ and $Y$ to be
It will be convenient to also have a notation for the transpose of the covariance vector:
By the linearity of covariance,