13.3.1. The Variance of a Sum#

Let $X_1,X_2,\dots X_n$ be random variables with sum

The variance of the sum is

We say that the variance of the sum is the sum of all the variances and all the covariances.

• The first sum has $n$ terms.

• The second sum has $n(n-1)$ terms.

Since $Cov(X_i,X_j)=Cov(X_j,X_i)$, the second sum can be written as $2{\sum \sum }_{1\le i<j\le n}Cov(X_i,X_j)$. But we will use the form given above.

13.3.2. Sum of Independent Random Variables#

If $X_1,X_2\dots ,X_n$ are independent, then all the covariance terms in the formula above are 0.

Therefore if $X_1,X_2,\dots ,X_n$ are independent, then

Thus for independent random variables $X_1,X_2,\dots ,X_n$, both the expectation and the variance add up nicely:

When the random variables are i.i.d., this simplifies even further.

13.3.3. Sum of an IID Sample#

Let $X_1,X_2,\dots ,X_n$ be i.i.d., each with mean $\mu$ and $SD$ $\sigma$. You can think of $X_1,X_2,\dots ,X_n$ as draws at random with replacement from a population, or the results of independent replications of the same experiment.

Let $S_n$ be the sample sum, as above. Then

This implies that as the sample size $n$ increases, the distribution of the sum $S_n$ shifts to the right and is more spread out. The expectation goes up linearly in $n$, but the SD goes up more slowly.

Here is an important application of the formula for the variance of an i.i.d. sample sum.

13.3.4. Variance of the Binomial#

Let $X$ have the binomial $(n,p)$ distribution. We know that

where $I_1,I_2,\dots ,I_n$ are i.i.d. indicators, each taking the value 1 with probability $p$. Each of these indicators has expectation $p$ and variance $pq=p(1-p)$. Therefore

For example, if $X$ is the number of heads in 100 tosses of a coin, then

Here is the distribution of $X$. You can see that there is almost no probability outside the range $E(X)\pm 3SD(X)$.

k = np.arange(25, 75, 1)
binom_probs = stats.binom.pmf(k, 100, 0.5)
binom_dist = Table().values(k).probabilities(binom_probs)
Plot(binom_dist, show_ev=True, show_sd=True)

13.3.5. Variance of the Poisson, Revisited#

We showed earlier that if $X$ has the Poisson $(\mu )$ distribution then $E(X)=\mu$, $Var(X)=\mu$, and $SD(X)=\sqrt{\mu }$. Now we have a way to understand the formula for the SD.

One way in which a Poisson $(\mu )$ distribution can arise is as an approximation to a binomial $(n,p)$ distribution where $n$ is large, $p$ is small, and $np=\mu$. The expectation of the binomial becomes the parameter of the approximating Poisson distribution, which is also the expectation of the Poisson.

Now let’s compare the standard deviations. The standard deviation of the binomial is

But $np=\mu$ in this setting, so the SD of the binomial is approximately $\sqrt{\mu }$. That’s the SD of its approximating Poisson $(\mu )$ distribution.