Gaussian Probability Distribution

Consider a very large number of observations, $N\gg 1$ , made on a system with two possible outcomes. Suppose that the probability of outcome 1 is sufficiently large that the average number of occurrences after $N$ observations is much greater than unity: that is,

$$\overline{n_1} = N p \gg 1.$$ (2.54)

In this limit, the standard deviation of $n_1$ is also much greater than unity,

$${\mit\Delta}^\ast n_1 = \sqrt{N p q}\gg 1,$$ (2.55)

implying that there are very many probable values of $n_1$ scattered about the mean value, $\overline{n_1}$ . This suggests that the probability of obtaining $n_1$ occurrences of outcome 1 does not change significantly in going from one possible value of $n_1$ to an adjacent value. In other words,

$$\frac{\vert P_N(n_1+1)-P_N(n_1)\vert}{P_N(n_1)} \ll 1.$$ (2.56)

In this situation, it is useful to regard the probability as a smooth function of $n_1$ . Let $n$ be a continuous variable that is interpreted as the number of occurrences of outcome 1 (after $N$ observations) whenever it takes on a positive integer value. The probability that $n$ lies between $n$ and $n+dn$ is defined

$$P(n, n+dn) = {\cal P}(n) dn,$$ (2.57)

where ${\cal P}(n)$ is called the probability density, and is independent of $dn$ . The probability can be written in this form because $P(n, n+dn)$ can always be expanded as a Taylor series in $dn$ , and must go to zero as $dn\rightarrow 0$ . We can write

$$\int_{n_1-1/2}^{n_1+1/2} {\cal P}(n) dn = P_N(n_1),$$ (2.58)

which is equivalent to smearing out the discrete probability $P_N(n_1)$ over the range $n_1\pm 1/2$ . Given Equations (2.38) and (2.56), the previous relation can be approximated as

$${\cal P}(n) \simeq P_N(n) = \frac{N!}{n! (N-n)!} p^{ n} q^{ N-n}.$$ (2.59)

For large $N$ , the relative width of the probability distribution function is small: that is,

$$\frac{{\mit\Delta}^\ast n_1}{\overline{n_1}} = \sqrt{\frac{q}{p}}\frac{1}{\sqrt{N}} \ll 1.$$ (2.60)

This suggests that ${\cal P}(n)$ is strongly peaked around the mean value, $\overline{n}=\overline{n_1}$ . Suppose that $\ln{\cal P}(n)$ attains its maximum value at $n=\tilde{n}$ (where we expect $\tilde{n}\sim \overline{n_1}$ ). Let us Taylor expand $\ln{\cal P}(n)$ around $n=\tilde{n}$ . Note that we are expanding the slowly-varying function $\ln{\cal P}(n)$ , rather than the rapidly-varying function ${\cal P}(n)$ , because the Taylor expansion of ${\cal P}(n)$ does not converge sufficiently rapidly in the vicinity of $n=\tilde{n}$ to be useful. We can write

$$\ln {\cal P}(\tilde{n}+\eta) \simeq \ln{\cal P}(\tilde{n}) +\eta B_1+\frac{\eta^{ 2}}{2} B_2+\cdots ,$$ (2.61)

where

$$B_k = \left.\frac{d^{ k} \ln {\cal P}}{d n^{ k}}\right\vert _{n=\tilde{n}}.$$ (2.62)

By definition,

$$B_1 = 0,$$ (2.63)

$$B_2 <0,$$ (2.64)

if $n=\tilde{n}$ corresponds to the maximum value of $\ln{\cal P}(n)$ .

It follows from Equation (2.59) that

$$\ln {\cal P} = \ln N! - \ln n!- \ln (N-n)! +n\ln p +(N-n)\ln q.$$ (2.65)

If $n$ is a large integer, such that $n\gg 1$ , then $\ln n!$ is almost a continuous function of $n$ , because $\ln n!$ changes by only a relatively small amount when $n$ is incremented by unity. Hence,

$$\frac{d\ln n!}{dn} \simeq \frac{\ln (n+1)!-\ln n!}{1} = \ln \left[\frac{(n+1)!}{n!}\right] = \ln (n+1),$$ (2.66)

giving

$$\frac{d\ln n!}{d n} \simeq \ln n,$$ (2.67)

for $n\gg 1$ . The integral of this relation

$$\ln n! \simeq n \ln n - n +{\cal O}(1),$$ (2.68)

valid for $n\gg 1$ , is called Stirling's approximation, after the Scottish mathematician James Stirling, who first obtained it in 1730.

According to Equations (2.62), (2.65), and (2.67),

$$B_1 = -\ln\tilde{n} +\ln (N-\tilde{n})+\ln p - \ln q.$$ (2.69)

Hence, if $B_1=0$ then

$$(N-\tilde{n}) p = \tilde{n} q,$$ (2.70)

giving

$$\tilde{n} = N p = \overline{n_1},$$ (2.71)

because $p+q=1$ . [See Equations (2.11) and (2.43).] Thus, the maximum of $\ln{\cal P}(n)$ occurs exactly at the mean value of $n$ , which equals $\overline{n_1}$ .

Further differentiation of Equation (2.69) yields [see Equation (2.62)]

$$B_2 = -\frac{1}{\tilde{n}}-\frac{1}{N-\tilde{n}} = -\frac{1}{N p}-\frac{1}{N (1-p)}= - \frac{1}{N p q},$$ (2.72)

because $p+q=1$ . Note that $B_2<0$ , as required. According to Equation (2.55), the previous relation can also be written

$$B_2 = -\frac{1}{({\mit\Delta}^\ast n_1)^{ 2}}.$$ (2.73)

It follows, from the previous analysis, that the Taylor expansion of $\ln {\cal P}$ can be written

$$\ln{\cal P}(\overline{n_1}+\eta) \simeq \ln{\cal P}(\overline{n_1}) - \frac{\eta^{ 2}}{2 ({\mit\Delta}^\ast n_1)^{ 2}} +\cdots.$$ (2.74)

Taking the exponential of both sides, we obtain

$${\cal P}(n)\simeq {\cal P}(\overline{n_1})\exp\left[- \frac{(n-\overline{n_1})^{ 2}}{2 ({\mit\Delta}^\ast n_1)^{ 2}}\right].$$ (2.75)

The constant ${\cal P}(\overline{n_1})$ is most conveniently fixed by making use of the normalization condition

$$\sum_{n_1=0,N} P_N(n_1)=1,$$ (2.76)

which becomes

$$\int_0^N {\cal P}(n) dn \simeq 1$$ (2.77)

for a continuous distribution function. Because we only expect ${\cal P}(n)$ to be significant when $n$ lies in the relatively narrow range $\overline{n_1}\pm {\mit\Delta}^\ast n_1$ , the limits of integration in the previous expression can be replaced by $\pm \infty$ with negligible error. Thus,

$${\cal P}(\overline{n_1})\int_{-\infty}^{\infty}\!\exp\! \left[-\f... ...} {\mit\Delta}^\ast n_1\int_{-\infty}^{\infty} {\rm e}^{-x^{ 2}} dx\simeq 1.$$ (2.78)

As is well known,

$$\int_{-\infty}^{\infty} {\rm e}^{ -x^{ 2}} dx = \sqrt{\pi}.$$ (2.79)

(See Exercise 1.) It follows from the normalization condition (2.78) that

$${\cal P}(\overline{n_1})\simeq \frac{1}{ \sqrt{2\pi} {\mit\Delta}^\ast n_1}.$$ (2.80)

Finally, we obtain

$${\cal P}(n) \simeq \frac{1}{\sqrt{2\pi} {\mit\Delta}^\ast n_1}\... ...left[-\frac{(n-\overline{n_1})^{ 2}}{2 ({\mit\Delta}^\ast n_1)^{ 2}}\right].$$ (2.81)

This is the famous Gaussian probability distribution, named after the German mathematician Carl Friedrich Gauss, who discovered it while investigating the distribution of errors in measurements. The Gaussian distribution is only valid in the limits $N\gg 1$ and $\overline{n_1}\gg 1$ .

Suppose we were to plot the probability $P_N(n_1)$ against the integer variable $n_1$ , and then fit a continuous curve through the discrete points thus obtained. This curve would be equivalent to the continuous probability density curve ${\cal P}(n)$ , where $n$ is the continuous version of $n_1$ . According to Equation (2.81), the probability density attains its maximum value when $n$ equals the mean of $n_1$ , and is also symmetric about this point. In fact, when plotted with the appropriate ratio of vertical to horizontal scalings, the Gaussian probability density curve looks rather like the outline of a bell centered on $n= \overline{n_1}$ . Hence, this curve is sometimes called a bell curve. At one standard deviation away from the mean value--that is $n=\overline{n_1}\pm {\mit\Delta}^\ast n_1$ --the probability density is about 61% of its peak value. At two standard deviations away from the mean value, the probability density is about 13.5% of its peak value. Finally, at three standard deviations away from the mean value, the probability density is only about 1% of its peak value. We conclude that there is very little chance that $n_1$ lies more than about three standard deviations away from its mean value. In other words, $n_1$ is almost certain to lie in the relatively narrow range $\overline{n_1}\pm 3 {\mit\Delta}^\ast n_1$ .

In the previous analysis, we went from a discrete probability function, $P_N(n_1)$ , to a continuous probability density, ${\cal P}(n)$ . The normalization condition becomes

$$1= \sum_{n_1=0,N} P_N(n_1) \simeq \int_{-\infty}^{\infty}{\cal P}(n) dn$$ (2.82)

under this transformation. Likewise, the evaluations of the mean and variance of the distribution are written

$$\overline{n_1} = \sum_{n_1=0,N} P_N(n_1) n_1 \simeq \int_{-\infty}^{\infty} {\cal P}(n) n dn,$$ (2.83)

and

$$\overline{({\mit\Delta} n_1)^{ 2}}\equiv ({\mit\Delta}^\ast n_1)... ...^{ 2}\simeq \int_{-\infty}^{\infty}{\cal P}(n) (n-\overline{n_1})^{ 2} dn,$$ (2.84)

respectively. These results follow as simple generalizations of previously established results for the discrete function $P_N(n_1)$ . The limits of integration in the previous expressions can be approximated as $\pm \infty$ because ${\cal P}(n)$ is only non-negligible in a relatively narrow range of $n$ . Finally, it is easily demonstrated that Equations (2.82)-(2.84) are indeed true by substituting in the Gaussian probability density, Equation (2.81), and then performing a few elementary integrals. (See Exercise 3.)