Entropy

Consider an isolated system whose energy is known to lie in a narrow range. Let ${\mit\Omega}$ be the number of accessible microstates. According to the principle of equal a priori probabilities, the system is equally likely to be found in any one of these states when it is in thermal equilibrium. The accessible states are just that set of microstates which are consistent with the macroscopic constraints imposed on the system. These constraints can usually be quantified by specifying the values of some parameters $y_1, \cdots, y_n$ which characterize the macrostate. Note that these parameters are not necessarily external: e.g., we could specify either the volume (an external parameter) or the mean pressure (the mean force conjugate to the volume). The number of accessible states is clearly a function of the chosen parameters, so we can write ${\mit\Omega}\equiv {\mit\Omega}(y_1,\cdots, y_n)$ for the number of microstates consistent with a macrostate in which the general parameter $y_\alpha$ lies in the range $y_\alpha$ to $y_\alpha+d y_\alpha$.

Suppose that we start from a system in thermal equilibrium. According to statistical mechanics, each of the ${\mit\Omega}_i$, say, accessible states are equally likely. Let us now remove, or relax, some of the constraints imposed on the system. Clearly, all of the microstates formally accessible to the system are still accessible, but many additional states will, in general, become accessible. Thus, removing or relaxing constraints can only have the effect of increasing, or possibly leaving unchanged, the number of microstates accessible to the system. If the final number of accessible states is ${\mit\Omega}_f$, then we can write

$${\mit\Omega}_f \geq {\mit\Omega}_i.$$ (211)

Immediately after the constraints are relaxed, the systems in the ensemble are not in any of the microstates from which they were previously excluded. So the systems only occupy a fraction

$$P_i =\frac{{\mit\Omega}_i}{{\mit\Omega}_f}$$ (212)

of the ${\mit\Omega}_f$ states now accessible to them. This is clearly not a equilibrium situation. Indeed, if ${\mit\Omega}_f\gg {\mit\Omega}_i$ then the configuration in which the systems are only distributed over the original ${\mit\Omega}_i$ states is an extremely unlikely one. In fact, its probability of occurrence is given by Eq. (212). According to the $H$ theorem (see Sect. 3.4), the ensemble will evolve in time until a more probable final state is reached in which the systems are evenly distributed over the ${\mit\Omega}_f$ available states.

As a simple example, consider a system consisting of a box divided into two regions of equal volume. Suppose that, initially, one region is filled with gas and the other is empty. The constraint imposed on the system is, thus, that the coordinates of all of the molecules must lie within the filled region. In other words, the volume accessible to the system is $V=V_i$, where $V_i$ is half the volume of the box. The constraints imposed on the system can be relaxed by removing the partition and allowing gas to flow into both regions. The volume accessible to the gas is now $V=V_f=2 \,V_i$. Immediately after the partition is removed, the system is in an extremely improbable state. We know, from Sect. 3.8, that at constant energy the variation of the number of accessible states of an ideal gas with the volume is

$${\mit\Omega} \propto V^N,$$ (213)

where $N$ is the number of particles. Thus, the probability of observing the state immediately after the partition is removed in an ensemble of equilibrium systems with volume $V=V_f$ is

$$P_i = \frac{{\mit\Omega}_i}{{\mit\Omega}_f} = \left(\frac{V_i}{V_f}\right)^N =\left(\frac{1}{2} \right)^N.$$ (214)

If the box contains of order 1 mole of molecules then $N\sim 10^{24}$ and this probability is fantastically small:

$$P_i \sim \exp\left(-10^{24}\right).$$ (215)

Clearly, the system will evolve towards a more probable state.

This discussion can also be phrased in terms of the parameters $y_1, \cdots, y_n$ of the system. Suppose that a constraint is removed. For instance, one of the parameters, $y$, say, which originally had the value $y=y_i$, is now allowed to vary. According to statistical mechanics, all states accessible to the system are equally likely. So, the probability $P(y)$ of finding the system in equilibrium with the parameter in the range $y$ to $y+\delta y$ is just proportional to the number of microstates in this interval: i.e.,

$$P(y) \propto {\mit\Omega}(y).$$ (216)

Usually, ${\mit\Omega}(y)$ has a very pronounced maximum at some particular value $\tilde{y}$ (see Sect. 5.2). This means that practically all systems in the final equilibrium ensemble have values of $y$ close to $\tilde{y}$. Thus, if $y_i\neq \tilde{y}$ initially then the parameter $y$ will change until it attains a final value close to $\tilde{y}$, where ${\mit\Omega}$ is maximum. This discussion can be summed up in a single phrase:

If some of the constraints of an isolated system are removed then the parameters of the system tend to readjust themselves in such a way that

$${\mit\Omega}(y_1,\cdots, y_n)\rightarrow {\rm maximum}.$$ (217)

Suppose that the final equilibrium state has been reached, so that the systems in the ensemble are uniformly distributed over the ${\mit\Omega}_f$ accessible final states. If the original constraints are reimposed then the systems in the ensemble still occupy these ${\mit\Omega}_f$ states with equal probability. Thus, if ${\mit\Omega}_f > {\mit\Omega}_i$, simply restoring the constraints does not restore the initial situation. Once the systems are randomly distributed over the ${\mit\Omega}_f$ states they cannot be expected to spontaneously move out of some of these states and occupy a more restricted class of states merely in response to the reimposition of a constraint. The initial condition can also not be restored by removing further constraints. This could only lead to even more states becoming accessible to the system.

Suppose that some process occurs in which an isolated system goes from some initial configuration to some final configuration. If the final configuration is such that the imposition or removal of constraints cannot by itself restore the initial condition then the process is deemed irreversible. On the other hand, if it is such that the imposition or removal of constraints can restore the initial condition then the process is deemed reversible. From what we have already said, an irreversible process is clearly one in which the removal of constraints leads to a situation where ${\mit\Omega}_f > {\mit\Omega}_i$. A reversible process corresponds to the special case where the removal of constraints does not change the number of accessible states, so that ${\mit\Omega}_f = {\mit\Omega}_i$. In this situation, the systems remain distributed with equal probability over these states irrespective of whether the constraints are imposed or not.

Our microscopic definition of irreversibility is in accordance with the macroscopic definition discussed in Sect. 3.6. Recall that on a macroscopic level an irreversible process is one which ``looks unphysical'' when viewed in reverse. On a microscopic level it is clearly plausible that a system should spontaneously evolve from an improbable to a probable configuration in response to the relaxation of some constraint. However, it is quite clearly implausible that a system should ever spontaneously evolve from a probable to an improbable configuration. Let us consider our example again. If a gas is initially restricted to one half of a box, via a partition, then the flow of gas from one side of the box to the other when the partition is removed is an irreversible process. This process is irreversible on a microscopic level because the initial configuration cannot be recovered by simply replacing the partition. It is irreversible on a macroscopic level because it is obviously unphysical for the molecules of a gas to spontaneously distribute themselves in such a manner that they only occupy half of the available volume.

It is actually possible to quantify irreversibility. In other words, in addition to stating that a given process is irreversible, we can also give some indication of how irreversible it is. The parameter which measures irreversibility is just the number of accessible states ${\mit\Omega}$. Thus, if ${\mit\Omega}$ for an isolated system spontaneously increases then the process is irreversible, the degree of irreversibility being proportional to the amount of the increase. If ${\mit\Omega}$ stays the same then the process is reversible. Of course, it is unphysical for ${\mit\Omega}$ to ever spontaneously decrease. In symbols, we can write

$${\mit\Omega}_f -{\mit\Omega}_i \equiv {\mit\Delta} {\mit\Omega}\geq 0,$$ (218)

for any physical process operating on an isolated system. In practice, ${\mit\Omega}$ itself is a rather unwieldy parameter with which to measure irreversibility. For instance, in the previous example, where an ideal gas doubles in volume (at constant energy) due to the removal of a partition, the fractional increase in ${\mit\Omega}$ is

$$\frac{{\mit\Omega}_f}{{\mit\Omega}_i} \simeq 10^{\,2\,\nu\times 10^{23}},$$ (219)

where $\nu$ is the number of moles. This is an extremely large number! It is far more convenient to measure irreversibility in terms of $\ln{\mit\Omega}$. If Eq. (218) is true then it is certainly also true that

$$\ln {\mit\Omega}_f -\ln {\mit\Omega}_i \equiv {\mit\Delta} \ln{\mit\Omega}\geq 0$$ (220)

for any physical process operating on an isolated system. The increase in $\ln{\mit\Omega}$ when an ideal gas doubles in volume (at constant energy) is

$$\ln {\mit\Omega}_f - \ln {\mit\Omega}_i = \nu\, N_A\,\ln 2,$$ (221)

where $N_A=6\times 10^{23}$. This is a far more manageable number! Since we usually deal with particles by the mole in laboratory physics, it makes sense to pre-multiply our measure of irreversibility by a number of order $1/N_A$. For historical reasons, the number which is generally used for this purpose is the Boltzmann constant $k$, which can be written

$$k = \frac{R}{N_A} ~~~{\rm joules/kelvin},$$ (222)

where

$$R= 8.3143 ~~~{\rm joules/kelvin/mole}$$ (223)

is the ideal gas constant which appears in the well-known equation of state for an ideal gas, $P\,V=\nu \,R\,T$. Thus, the final form for our measure of irreversibility is

$$S = k \,\ln{\mit\Omega}.$$ (224)

This quantity is termed ``entropy'', and is measured in joules per degree kelvin. The increase in entropy when an ideal gas doubles in volume (at constant energy) is

$$S_f - S_i = \nu \,R\, \ln 2,$$ (225)

which is order unity for laboratory scale systems (i.e., those containing about one mole of particles). The essential irreversibility of macroscopic phenomena can be summed up as follows:

$$S_f - S_i \equiv {\mit\Delta} S \geq 0,$$ (226)

for a process acting on an isolated system [this is equivalent to Eqs. (217), (218), and (220)]. Thus, the entropy of an isolated system tends to increase with time and can never decrease. This proposition is known as the second law of thermodynamics.

One way of thinking of the number of accessible states ${\mit\Omega}$ is that it is a measure of the disorder associated with a macrostate. For a system exhibiting a high degree of order we would expect a strong correlation between the motions of the individual particles. For instance, in a fluid there might be a strong tendency for the particles to move in one particular direction, giving rise to an ordered flow of the system in that direction. On the other hand, for a system exhibiting a low degree of order we expect far less correlation between the motions of individual particles. It follows that, all other things being equal, an ordered system is more constrained than a disordered system, since the former is excluded from microstates in which there is not a strong correlation between individual particle motions, whereas the latter is not. Another way of saying this is that an ordered system has less accessible microstates than a corresponding disordered system. Thus, entropy is effectively a measure of the disorder in a system (the disorder increases with $S$). With this interpretation, the second law of thermodynamics reduces to the statement that isolated systems tend to become more disordered with time, and can never become more ordered.

Note that the second law of thermodynamics only applies to isolated systems. The entropy of a non-isolated system can decrease. For instance, if a gas expands (at constant energy) to twice its initial volume after the removal of a partition, we can subsequently recompress the gas to its original volume. The energy of the gas will increase because of the work done on it during compression, but if we absorb some heat from the gas then we can restore it to its initial state. Clearly, in restoring the gas to its original state, we have restored its original entropy. This appears to violate the second law of thermodynamics because the entropy should have increased in what is obviously an irreversible process (just try to make a gas spontaneously occupy half of its original volume!). However, if we consider a new system consisting of the gas plus the compression and heat absorption machinery, then it is still true that the entropy of this system (which is assumed to be isolated) must increase in time. Thus, the entropy of the gas is only kept the same at the expense of increasing the entropy of the rest of the system, and the total entropy is increased. If we consider the system of everything in the Universe, which is certainly an isolated system since there is nothing outside it with which it could interact, then the second law of thermodynamics becomes:

The disorder of the Universe tends to increase with time and can never decrease.

An irreversible process is clearly one which increases the disorder of the Universe, whereas a reversible process neither increases nor decreases disorder. This definition is in accordance with our previous definition of an irreversible process as one which ``does not look right'' when viewed backwards. One easy way of viewing macroscopic events in reverse is to film them, and then play the film backwards through a projector. There is a famous passage in the novel ``Slaughterhouse 5,'' by Kurt Vonnegut, in which the hero, Billy Pilgrim, views a propaganda film of an American World War II bombing raid on a German city in reverse. This is what the film appeared to show:

``American planes, full of holes and wounded men and corpses took off backwards from an airfield in England. Over France, a few German fighter planes flew at them backwards, sucked bullets and shell fragments from some of the planes and crewmen. They did the same for wrecked American bombers on the ground, and those planes flew up backwards and joined the formation.

The formation flew backwards over a German city that was in flames. The bombers opened their bomb bay doors, exerted a miraculous magnetism which shrunk the fires, gathered them into cylindrical steel containers, and lifted the containers into the bellies of the planes. The containers were stored neatly in racks. The Germans had miraculous devices of their own, which were long steel tubes. They used them to suck more fragments from the crewmen and planes. But there were still a few wounded Americans, though, and some of the bombers were in bad repair. Over France, though, German fighters came up again, made everything and everybody as good as new.''

Vonnegut's point, I suppose, is that the morality of actions is inverted when you view them in reverse.

What is there about this passage which strikes us as surreal and fantastic? What is there that immediately tells us that the events shown in the film could never happen in reality? It is not so much that the planes appear to fly backwards and the bombs appear to fall upwards. After all, given a little ingenuity and a sufficiently good pilot, it is probably possible to fly a plane backwards. Likewise, if we were to throw a bomb up in the air with just the right velocity we could, in principle, fix it so that the velocity of the bomb matched that of a passing bomber when their paths intersected. Certainly, if you had never seen a plane before it would not be obvious which way around it was supposed to fly. However, certain events are depicted in the film, ``miraculous'' events in Vonnegut's words, which would immediately strike us as the wrong way around even if we had never seen them before. For instance, the film might show thousands of white hot bits of shrapnel approach each other from all directions at great velocity, compressing an explosive gas in the process, which slows them down such that when they meet they fit together exactly to form a metal cylinder enclosing the gases and moving upwards at great velocity. What strikes us as completely implausible about this event is the spontaneous transition from the disordered motion of the gases and metal fragments to the ordered upward motion of the bomb.