Gaussian Probability Distribution

Consider a very large number of observations, $N\gg 1$, made on a system with two possible outcomes. (See Sections 5.1.2 and 5.1.4.) Suppose that the probability of outcome $P$ is sufficiently large that the average number of occurrences after $N$ observations is much greater than unity; that is,

$$\langle n\rangle = N\,p \gg 1.$$ (5.61)

In this limit, the standard deviation of $n$ is also much greater than unity,

$$\sigma_{n} = \sqrt{N\,p\,q}\gg 1,$$ (5.62)

implying that there are very many probable values of $n$ scattered about the mean value, $\langle n\rangle$. This suggests that the probability of obtaining $n$ occurrences of outcome $P$ does not change significantly in going from one possible value of $n$ to an adjacent value. In other words,

$$\frac{\vert P_N(n+1)-P_N(n)\vert}{P_N(n)} \ll 1.$$ (5.63)

In this situation, it is useful to regard the probability as a smooth function of $n$. Let $n$ now be a continuous variable that is interpreted as the number of occurrences of outcome $P$ (after $N$ observations) whenever it takes on a positive integer value. The probability that $n$ lies between $n$ and $n+dn$ is defined

$$P(n,n+dn)= P(n)\, dn,$$ (5.64)

where $P(n)$ is a probability density (see Section 5.1.6), and is independent of $dn$. The probability can be written in this form because $P(n, n+dn)$ can always be expanded as a Taylor series in $dn$, and must go to zero as $dn\rightarrow 0$. We can write

$$\int_{n-1/2}^{n+1/2} P(n)\, dn = P_N(n),$$ (5.65)

which is equivalent to smearing out the discrete probability $P_N(n)$ over the range $n\pm 1/2$. Given Equations (5.27) and (5.63), the previous relation can be approximated as

$$P(n) \simeq P_N(n) = \frac{N!}{n!\,(N-n)!}\,p^{n}\,q^{N-n}.$$ (5.66)

For large $N$, the relative width of the probability distribution function is small; that is,

$$\frac{\sigma_n}{\langle n\rangle } = \sqrt{\frac{q}{p}}\frac{1}{\sqrt{N}} \ll 1.$$ (5.67)

This suggests that $P(n)$ is strongly peaked around the mean value, $\langle n\rangle$. Suppose that $\ln P(n)$ attains its maximum value at $n=\tilde{n}$ (where we expect $\tilde{n}\sim \langle n\rangle$). Let us Taylor expand $\ln P(n)$ around $n=\tilde{n}$. Note that we are expanding the slowly-varying function $\ln P(n)$, rather than the rapidly-varying function $P(n)$, because the Taylor expansion of $P(n)$ does not converge sufficiently rapidly in the vicinity of $n=\tilde{n}$ to be useful. We can write

$$\ln P(\tilde{n}+\eta) \simeq \ln P(\tilde{n}) +\eta\, B_1+\frac{\eta^{2}}{2}\,B_2+\cdots ,$$ (5.68)

where

$$B_k = \left.\frac{d^{k} \ln P}{d n^{k}}\right\vert _{n=\tilde{n}}.$$ (5.69)

By definition,

$$B_1 =0,$$ (5.70)

$$B_2 <0,$$ (5.71)

if $n=\tilde{n}$ corresponds to the maximum value of $\ln P(n)$.

It follows from Equation (5.66) that

$$\ln P = \ln N! - \ln n!- \ln \,(N-n)! +n\ln p +(N-n)\ln q.$$ (5.72)

If $n$ is a large integer, such that $n\gg 1$, then $\ln n!$ is almost a continuous function of $n$, because $\ln n!$ changes by only a relatively small amount when $n$ is incremented by unity. Hence,

$$\frac{d\ln n!}{dn} \simeq \frac{\ln\,(n+1)!-\ln n!}{1} = \ln \left[\frac{(n+1)!}{n!}\right] = \ln\,(n+1),$$ (5.73)

giving

$$\frac{d\ln n!}{d n} \simeq \ln n,$$ (5.74)

for $n\gg 1$. The integral of this relation

$$\ln n! \simeq n\,\ln n - n +{\cal O}(1),$$ (5.75)

valid for $n\gg 1$, is called Stirling's approximation, after James Stirling, who first obtained it in 1730.

According to Equations (5.69), (5.72), and (5.74),

$$B_1 = -\ln\tilde{n} +\ln\,(N-\tilde{n})+\ln p - \ln q.$$ (5.76)

Hence, if $B_1=0$ then

$$(N-\tilde{n})\, p = \tilde{n}\,q,$$ (5.77)

giving

$$\tilde{n} = N\,p = \langle n\rangle,$$ (5.78)

because $p+q=1$. [See Equations (5.9) and (5.32).] Thus, the maximum of $\ln P(n)$ occurs exactly at the mean value of $n$.

Further differentiation of Equation (5.76) yields [see Equation (5.69)]

$$B_2 = -\frac{1}{\tilde{n}}-\frac{1}{N-\tilde{n}} = -\frac{1}{N\,p}-\frac{1}{N\,(1-p)}= - \frac{1}{N\,p\,q},$$ (5.79)

because $p+q=1$. Note that $B_2<0$, as required. According to Equation (5.62), the previous relation can also be written

$$B_2 = -\frac{1}{\sigma_{n}^{\,2}}.$$ (5.80)

It follows, from the previous analysis, that the Taylor expansion of $\ln P(n)$ can be written

$$\ln P(\langle n\rangle+\eta) \simeq \ln P (\langle n\rangle) - \frac{\eta^{2}}{2\,\sigma_n^{\,2}} +\cdots.$$ (5.81)

Taking the exponential of both sides, we obtain

$$P(n)\simeq P(\langle n\rangle)\exp\left[- \frac{(n-\langle n\rangle)^{2}}{2\,\sigma_n^{\,2}}\right].$$ (5.82)

The constant $P(\langle n\rangle)$ is most conveniently fixed by making use of the normalization condition,

$$\int_0^N P(n)\,dn \simeq 1,$$ (5.83)

for a continuous distribution function. [See Equation (5.54). Note that $n$ cannot take a negative value.] Because we only expect $P(n)$ to be significant when $n$ lies in the relatively narrow range $\langle n\rangle \pm \sigma_n^{\,2}$, the limits of integration in the previous expression can be replaced by $\pm \infty$ with negligible error. Thus,

$$P(\langle n\rangle) \int_{-\infty}^{\infty}\!\exp\! \left[-\frac{... ...angle ) \sqrt{2}\,\sigma_n\int_{-\infty}^{\infty} {\rm e}^{-x^{2}}\,dx\simeq 1.$$ (5.84)

As is well known,

$$\int_{-\infty}^{\infty} {\rm e}^{\,-x^{2}}\,dx = \sqrt{\pi}.$$ (5.85)

It follows from the normalization condition (5.84) that

$$P(\langle n\rangle)\simeq \frac{1}{ \sqrt{2\pi} \,\sigma_n}.$$ (5.86)

Finally, we obtain

$$P(n) \simeq \frac{1}{\sqrt{2\pi} \,\sigma_n}\, \exp\left[-\frac{(n-\langle n\rangle)^{2}}{2\,\sigma_n^{\,2}}\right].$$ (5.87)

This is probability distribution is known as Gaussian probability distribution, after the Carl F. Gauss, who discovered in 1809 it while investigating the distribution of errors in measurements. The Gaussian distribution is only valid in the limits $N\gg 1$ and $\langle n\rangle \gg 1$. According to this distribution, at one standard deviation away from the mean value—that is $n=\langle n\rangle\pm \sigma_n$—the probability density is about 61% of its peak value. At two standard deviations away from the mean value, the probability density is about 13.5% of its peak value. Finally, at three standard deviations away from the mean value, the probability density is only about 1% of its peak value. We conclude that there is very little chance that $n$ lies more than about three standard deviations away from its mean value. In other words, $n$ is almost certain to lie in the relatively narrow range $\langle n\rangle \pm 3\,\sigma_n$.

Consider the drunken walk discussed at the end of Section 5.1.2. Suppose that the drunken man is equally likely to take a step to the right as to take a step to the left. In other words, $p=q=1/2$. Thus, according to Equations (5.32) and (5.41),

$$\langle n\rangle = \frac{N}{2},$$ (5.88)

$$\sigma_n = \frac{\sqrt{N}}{2}.$$ (5.89)

Equations (5.18) and (5.19) state that the probability of the drunken man taking $m$ net steps to the right after $N$ total steps is

$$P_N'(m) = P_N(n),$$ (5.90)

where

$$n = \frac{N+m}{2}.$$ (5.91)

In the limit of very many steps, we can treat $m$ and $n$ as continuous variables. Let $P'(m)\,dm$ be the probability that $m$ lies between $m$ and $m+dm$. Likewise, let $P(n)\,dn$ be the probability that $n$ lies between $n$ and $n+dn$. It follows that

$$P'(m)\,dm = P(n)\,dn,$$ (5.92)

where $m$ and $n$ satisfy Equation (5.91). Hence,

$$P'(m) = \frac{1}{2}\,P\left(\frac{N+m}{2}\right) = \frac{1}{\sqrt{2\pi\,N}}\,\exp\left(-\frac{m^2}{2\,N}\right),$$ (5.93)

where use has been made of Equations (5.87), (5.88), (5.89), and (5.91). Suppose that each step is of length $l$, and that the man takes $f$ steps per second. It follows that the man's displacement from his starting point is $x=m\,l$. Moreover, $N=f\,t$. Let $P(x,t)\,dx$ be the probability that the man's displacement from his starting point after $t$ seconds lies between $x$ and $x+dx$. We have $P(x,t)\,dx = P'(m)\,dm$, which implies that $P(x)=P'(m)/l$. Hence, we obtain

$$P(x) = \frac{1}{\sqrt{4\pi\,D\,t}}\,\exp\left(-\frac{x^2}{4\,D\,t}\right),$$ (5.94)

where

$$D =\frac{1}{2}\, f\,l^{\,2}$$ (5.95)

is the diffusivity. It is easily demonstrated that

$$\left\langle x^2\right\rangle= 2\,D\,t.$$ (5.96)

Thus, it is evident from the analysis of Section 5.1.5 that the probability density distribution (5.94) corresponds to that of a random walk in one dimension. Equation (5.94) can also be thought of as describing the diffusion of probability density along the $x$-axis. (See Section 5.3.9.)