Independence and Angles¶

In this section we take a closer look at how correlation can be interpreted geometrically in terms of angles.

We have defined $X$ and $Y$ to have the standard bivariate normal density with correlation $\rho$ if

$$ Y ~ = ~ \rho X + \sqrt{1 - \rho^2} Z $$

where $X$ and $Z$ are i.i.d. standard normal. We showed that $Y$ is also standard normal, and that the conditional density of $Y$ given $X = x$ is normal $(\rho x, 1 - \rho^2)$.

Note that we are assuming that $\rho$ is neither $1$ nor $-1$, so that the conditional density is not degenerate.

Bivariate Normal Joint Density¶

The joint density of $Y$ is the product of the marginal density of $X$ and the conditional density of $Y$ given $X$. That is,

\begin{align*} f(x, y) ~ &= ~ f_X(x)f_{Y \mid X=x} (y) \\ \\ &= ~ \frac{1}{\sqrt{2\pi}} \exp \Big{(}-\frac{1}{2} x^2\Big{)} \frac{1}{\sqrt{2\pi}\sqrt{1-\rho^2}} \exp \Big{(} -\frac{1}{2(1-\rho^2)} (y - \rho x)^2 \Big{)} \\ \\ &= ~ \frac{1}{2\pi \sqrt{1 - \rho^2}} \exp \Big{(} -\frac{1}{2(1-\rho^2)} (x^2 - 2\rho xy + y^2) \Big{)} \end{align*}

by algebraic simplification. There is no need to learn this formula for applications; just remember the marginal and conditional densities used to derive it.

Here we will use the joint density to observe some important properties of bivariate normal variables.

The graph below shows the joint density surface when $\rho = 0.8$. You can see that it has elliptical contours, consistent with the elliptical scatter plots we got in the previous sections by generating points $(X, Y)$.

Run the cell a few times with different values of $\rho$ in $(-1, 1)$ and notice how the surface changes. As the magnitude of $\rho$ gets smaller, the surface looks more like a bell with circular contours.

Uncorrelated Bivariate Normal Variables are Independent¶

You know that in general, uncorrelated random variables need not be independent. But uncorrelated bivariate normal variables are independent, as you can see in several ways.

Suppose $\rho = 0$. Then $X$ and $Y$ are independent, for any of the reasons listed below.

• By the definition of $Y$, we get $Y = Z$ which is independent of $X$.
• For every $x$, the conditional distribution of $Y$ given $X = x$ is normal $(0, 1)$, which doesn't involve $x$.
• The joint density of $X$ and $Y$ is $$ f(x, y) ~ = ~ \frac{1}{2\pi} \exp \big{(} -\frac{1}{2} (x^2 + y^2) \big{)} ~ = ~ \frac{1}{\sqrt{2\pi}} \exp \big{(} -\frac{1}{2} x^2 \big{)} \cdot \frac{1}{\sqrt{2\pi}} \exp \big{(} -\frac{1}{2} y^2 \big{)} ~ = ~ f_X(x)f_Y(y) $$

Establishing the independence of two random variables is not necessarily a simple task, but in the case of two bivariate normal random variables it is very simple indeed: just find the covariance between the two variables. If the covariance is zero, then the random variables are independent. If the covariance is not zero, then the random variables are dependent.

Correlation as a Cosine¶

Everything we have said above follows from the definition

$$ Y ~ = ~ \rho X + \sqrt{1 - \rho^2} Z $$

where $X$ and $Z$ are i.i.d. standard normal.

Let's see where this comes from. A good place to start is the joint density of $X$ and $Z$, which has circular symmetry.

The $X$ and $Z$ axes are orthogonal. Let's see what happens if we twist them.

Take any positive angle $\theta$ and draw a new axis at angle $\theta$ to the original $X$ axis. Every point $(X, Z)$ has a projection onto this axis. The figure below shows the projection of the point $(X, Z) = (1, 2)$ onto the gold axis which is at an angle of $\theta$ degress to the $X$ axis. The blue segment is the value of $X$. You get that by dropping the perpendicular from $(1, 2)$ to the horizontal axis. That's called projecting $(1, 2)$ onto the horizontal axis. The red segment is the projection of $(1, 2)$ onto the gold axes, obtained by dropping the perpendicular from $(1, 2)$ to the gold axis.

Vary the values of $\theta$ in the cell below to see how the projection changes as the gold axis rotates.

Let $Y$ be the length of the red segment, and remember that $X$ is the length of the blue segment. When $\theta$ is very small, $Y$ is almost equal to $X$. When $\theta$ approaches 90 degrees, $Y$ is almost equal to $Z$.

A little trigonometry shows that $Y ~ = ~ X \cos(\theta) + Z\sin(\theta)$.

Thus $$ Y ~ = ~ X\cos(\theta) + Z\sin(\theta) ~ = ~ \rho X + \sqrt{1 - \rho^2}Z $$ where $\rho = \cos(\theta)$.

The graphs below illustrate the transformation for $\theta = 30$ degrees.

The bivariate normal distribution is the joint distribution of the blue and red lengths $X$ and $Y$ when the original point $(X, Z)$ has i.i.d. standard normal coordinates. This transforms the circular contours of the joint density surface of $(X, Z)$ into the elliptical contours of the joint density surface of $(X, Y)$.

Small $\theta$¶

When $\theta$ is very small, there is hardly any change in the position of the axis, and so $X$ and $Y$ are almost equal.

The bivariate normal density of $X$ and $Y$, therefore, is essentially confined to the $X = Y$ line. The correlation $\cos(\theta)$ is large because $\theta$ is small; it is more than 0.999.

You can see the plotting function having trouble rendering this joint density surface.

Orthogonality and Independence¶

When $\theta$ is 90 degrees, the gold axis is orthogonal to the $X$ axis and $Y$ is equal to $Z$ which is independent of $X$.

When $\theta = 90$ degrees, $\cos(\theta) = 0$. The joint density surface of $(X, Y)$ is the same as that of $(X, Z)$ and has circular symmetry.