25.1. Bilinearity in Matrix Notation

As a preliminary to regression, we will express bilinearity in a compact form using matrix notation. The results of this section are not new. They are simply restatements of familiar results about variances and covariances, using new notation and matrix representations.

Let \(\mathbf{X}\) be a \(p \times 1\) vector of predictor variables. We know that for an \(m \times p\) matrix \(\mathbf{A}\) and an \(m \times 1\) vector \(\mathbf{b}\),

\[ Var(\mathbf{AX} + \mathbf{b}) ~ = ~ \mathbf{A}\boldsymbol{\Sigma}_\mathbf{X} \mathbf{A}^T \]

The results below are special cases of this.

25.1.1. Linear Combinations

To define two generic linear combinations of elements of \(\mathbf{X}\), let

\[\begin{split} \mathbf{A} ~ = ~ \begin{bmatrix} a_1 & a_2 & \cdots & a_p \\ c_1 & c_2 & \cdots & c_p \end{bmatrix} ~ = ~ \begin{bmatrix} \mathbf{a}^T \\ \mathbf{c}^T \end{bmatrix} ~~~~~~ \text{and} ~~~~~~ \mathbf{b} ~ = ~ \begin{bmatrix} b \\ d \end{bmatrix} \end{split}\]

Then

\[\begin{split} \mathbf{AX} + \mathbf{b} ~ = ~ \begin{bmatrix} a_1X_1 + a_2X_2 + \cdots + a_pX_p + b \\ c_1X_1 + c_2X_2 + \cdots + c_pX_p + d \end{bmatrix} ~ = ~ \begin{bmatrix} \mathbf{a}^T\mathbf{X} + b \\ \mathbf{c}^T\mathbf{X} + d \end{bmatrix} \end{split}\]

25.1.2. Covariance of Two Linear Combinations

The covariance of the two linear combinations is the \((1, 2)\) element of the covariance matrix of \(\mathbf{AX} + \mathbf{b}\), which is the \((1, 2)\) element of \(\mathbf{A}\boldsymbol{\Sigma}_\mathbf{X}\mathbf{A}^T\).

\[ Cov(\mathbf{a}^T\mathbf{X} + b, \mathbf{c}^T\mathbf{X} + d) ~ = ~ \mathbf{a}^T \boldsymbol{\Sigma}_\mathbf{X} \mathbf{c} \]

25.1.3. Variance of a Linear Combination

The variance of the first linear combination is the \((1, 1)\) element of \(\mathbf{A}\boldsymbol{\Sigma}_\mathbf{X}\mathbf{A}^T\).

\[ Var(\mathbf{a}^T\mathbf{X} + b) ~ = ~ \mathbf{a}^T \boldsymbol{\Sigma}_\mathbf{X} \mathbf{a} \]

25.1.4. Covariance Vector

To predict \(Y\) based on \(\mathbf{X}\) we will need to work with the covariance of \(Y\) and each of the elements of \(\mathbf{X}\). Let

\[ \sigma_{X_i, Y} ~ = ~ Cov(X_i, Y) \]

and define the covariance vector of \(\mathbf{X}\) and \(Y\) to be

\[\begin{split} \boldsymbol{\Sigma}_{\mathbf{X}, Y} ~ = ~ \begin{bmatrix} \sigma_{X_1, Y} \\ \sigma_{X_2, Y} \\ \vdots \\ \sigma_{X_p, Y} \end{bmatrix} \end{split}\]

It will be convenient to also have a notation for the transpose of the covariance vector:

\[ \boldsymbol{\Sigma}_{Y, \mathbf{X}} ~ = ~ \boldsymbol{\Sigma}_{\mathbf{X}, Y}^T ~ = ~ [\sigma_{X_1, Y} ~ \sigma_{X_2, Y} ~ \ldots ~ \sigma_{X_p, Y}] \]

By the linearity of covariance,

\[ Cov(\mathbf{a}^T\mathbf{X}, Y) ~ = ~ \mathbf{a}^T \boldsymbol{\Sigma}_{\mathbf{X}, Y} \]