{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": true,
"tags": [
"remove_cell"
]
},
"outputs": [],
"source": [
"# HIDDEN\n",
"from datascience import *\n",
"from prob140 import *\n",
"import numpy as np\n",
"import matplotlib.pyplot as plt\n",
"plt.style.use('fivethirtyeight')\n",
"%matplotlib inline\n",
"from scipy import stats"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## The Beta-Binomial Distribution ##"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As in the previous section, let $X$ have the beta $(r, s)$ prior, and given $X = p$ let the $S_n$ be the number of heads in the first $n$ tosses of a $p$-coin.\n",
"\n",
"All the calculations we carried out in the previous section were under the condition that $S_n = k$, but we never needed to find the probability of this event. It was part of the constant that made the posterior density of $X$ integrate to 1. \n",
"\n",
"We can now find $P(S_n = k)$ by writing the posterior density in two ways:\n",
"\n",
"- By recalling that it is the beta $(r+k, s+n-k)$ density:\n",
"\n",
"$$\n",
"f_{X \\vert S_n=k} (p) ~ = ~ C(r+k, s+n-k)p^{r+k-1}(1-p)^{s+n-k-1}, ~~~~ 0 < p < 1\n",
"$$\n",
"\n",
"- By using Bayes' Rule:\n",
"\n",
"$$\n",
"f_{X \\vert S_n=k} (p) ~ = ~ \\frac{C(r, s) p^{r-1}(1-p)^{s-1} \\cdot \\binom{n}{k} p^k (1-p)^{n-k}}{P(S_n = k)}, ~~~~ 0 < p < 1\n",
"$$\n",
"\n",
"Now equate constants:\n",
"\n",
"$$\n",
"\\frac{C(r, s) \\binom{n}{k}}{P(S_n = k)} ~ = ~ C(r+k, s+n-k)\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Beta-Binomial Probabilities ###\n",
"So for $k$ in the range 0 through $n$,\n",
"\n",
"$$\n",
"P(S_n = k) ~ = ~ \\binom{n}{k} \\frac{C(r, s)}{C(r+k, s+n-k)}\n",
"$$\n",
"\n",
"where $C(r,s)$ is the constant in the beta $(r, s)$ density, given by\n",
"\n",
"$$\n",
"C(r, s) ~ = ~ \\frac{\\Gamma(r+s)}{\\Gamma(r)\\Gamma(s)}\n",
"$$\n",
"\n",
"\n",
"This discrete distribution is called the *beta-binomial* distribution with parameters $r$, $s$, and $n$. It is the distribution of the number of heads in $n$ tosses of a coin that lands heads with a probability picked according to the beta $(r, s)$ distribution.\n",
"\n",
"One $(r, s)$ pair is particularly interesting: $r = s = 1$. That's the case when $X$ has the uniform prior. The distribution of $S_n$ reduces to\n",
"\n",
"$$\n",
"P(S_n = k ) ~ = ~ \\frac{n!}{k!(n-k)!} \\cdot \\frac{1!}{0!0!} \\cdot \\frac{k!(n-k)!}{(n+1)!} ~ = ~ \\frac{1}{n+1}\n",
"$$\n",
"\n",
"There's no $k$ in the answer! The conclusion is that if you choose $p$ uniformly between 0 and 1 and toss a $p$-coin $n$ times, *the distribution of the number of heads is uniform* on $\\{ 0, 1, 2, \\ldots, n\\}$.\n",
"\n",
"If you choose $p$ uniformly between 0 and 1, then for the conditional distribution of $S_n$ given that $p$ was the selected value is binomial $(n, p)$. But the unconditional distribution of $S_n$ is uniform."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Checking by Integration ###\n",
"If you prefer, you can find the distribution of $S_n$ directly, by conditioning on $X$.\n",
"$$\n",
"\\begin{align*}\n",
"P(S_n = k) ~ &= \\int_0^1 P(S_n = k \\mid X = p)f_X(p)dp \\\\ \\\\\n",
"&= ~ \\int_0^1 \\binom{n}{k} p^k(1-p)^{n-k}C(r, s)p^{r-1}(1-p)^{s-1}dp \\\\ \\\\\n",
"&= ~ \\binom{n}{k} C(r, s) \\int_0^1 p^{r+k-1}(1-p)^{s+n-k-1} dp \\\\ \\\\\n",
"&= ~ \\binom{n}{k} C(r, s) \\frac{1}{C(r+k, s+n-k)}\n",
"\\end{align*}\n",
"$$"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Expectation ###\n",
"Given $X = p$, the conditional distribution of $S_n$ is binomial $(n, p)$. Therefore \n",
"\n",
"$$\n",
"E(S_n \\mid X = p) ~ = ~ np\n",
"$$\n",
"\n",
"or, equivalently,\n",
"\n",
"$$\n",
"E(S_n \\mid X) ~ = ~ nX\n",
"$$\n",
"\n",
"By iteration,\n",
"\n",
"$$\n",
"E(S_n) ~ = ~ E(nX) ~ = ~ nE(X) ~ = ~ n\\frac{r}{r+s}\n",
"$$\n",
"\n",
"The expected proportion of heads in $n$ tosses is\n",
"\n",
"$$\n",
"E\\big{(} \\frac{S_n}{n} \\big{)} ~ = ~ \\frac{r}{r+s}\n",
"$$\n",
"\n",
"which is the expectation of the prior distribution of $X$. \n",
"\n",
"In the next section we will examine the long run behavior of this random proportion."
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true
},
"source": [
"### Endnote ###\n",
"The unconditional probability $P(S_n = k)$ appeared in the denominator of our calculation of the posterior density of $X$ given $S_n$. Because of the simplifications that result from using conjugate priors, we were able to calculate the denominator in a couple of different ways. But often the calculation can be intractable, especially in high dimensional settings. Methods of dealing with this problem are covered in more advanced courses."
]
}
],
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.4"
}
},
"nbformat": 4,
"nbformat_minor": 1
}