See also: Course-wide Conventions & Notation
Run the code: Notebook for Section 4.3, in which you can change and rerun the Python cells of this section in your browser.
Overview and Learning Objectives¶
Section 4.1 and Section 4.2 developed entropy entirely from macroscopic reasoning: a state function defined by , used to derive engine efficiency limits and the direction of spontaneous heat flow. This section shifts to the microscopic side, connecting entropy to the statistical-mechanical framework from Module 2. The payoff is substantial: entropy becomes a measure of how many ways a system can realize a macrostate, and its connection to the partition function provides a direct bridge between microscopic energy levels and macroscopic thermodynamic potentials.
We introduce three formulas for entropy — Boltzmann’s (isolated systems), the more general Gibbs form (any ensemble), and the canonical identity (the Gibbs form evaluated for the canonical ensemble) — and show that they are mutually consistent. The section culminates in the identification of the Helmholtz free energy , which was foreshadowed in Section 2.3 and Section 3.3.
Learning objectives:
State the second-law condition for isolated systems and interpret equilibrium as .
Use Boltzmann’s formula and explain why the logarithm ensures extensivity.
Compute entropy from a probability distribution via .
Derive the canonical identity .
Identify the Helmholtz free energy as a consequence of the canonical entropy formula.
Core Ideas and Derivations¶
Entropy Increases Until the System Reaches Equilibrium¶
The second law for an isolated system (Section 4.2) requires
with equality only at equilibrium. The qualitative picture, which Figure 4.3.1 sketches, is:
Out of equilibrium: entropy increases spontaneously ().
At equilibrium: entropy reaches its maximum value subject to the constraints and stops changing ().
Source
import sys
import matplotlib.pyplot as plt
import numpy as np
sys.path.append("..") # the book folder, which holds figstyle
import figstyle as fs
S_i, S_f = 1.0, 2.0 # schematic values
def draw_approach(c):
fig, ax = plt.subplots(figsize=(5.2, 3.1), constrained_layout=True)
t = np.linspace(0, 10, 400)
S = S_f - (S_f - S_i) * np.exp(-t / 1.6) # a smooth approach to the maximum
ax.axhline(S_f, color=c.gray, lw=1.2, ls=(0, (4, 3)))
ax.plot(t, S, color=c.blue, lw=fs.LINE, solid_capstyle="round")
ax.text(2.0, 1.52, "spontaneous change\n$dS > 0$", ha="left", va="center")
ax.text(8.0, 1.9, "equilibrium\n$dS = 0$", ha="center", va="top")
ax.set(xlim=(0, 10.3), ylim=(0.9, 2.12), xlabel="Time", ylabel="Entropy $S$")
ax.set_xticks([])
ax.set_yticks([S_i, S_f], ["$S_i$", "$S_f$"])
return fig
fs.show(draw_approach, alt="Schematic plot of entropy against time for an isolated system. The entropy starts at its "
"initial value, rises quickly at first and then more slowly, and levels off at its final, "
"maximum value, which is marked by a dashed line. The rising part is labeled spontaneous "
"change, dS greater than zero, and the flat part is labeled equilibrium, dS equal to zero.")Figure 4.3.1:Entropy of an isolated system during a spontaneous process (schematic). The entropy rises from to its maximum value and then stays constant at equilibrium.
A concrete example from Section 4.2 illustrates this. Two subsystems at different temperatures () in an insulated box exchange heat, and we showed that
as long as . Energy flows from to , raising and lowering , until . At that point the factor vanishes, , and the system has reached thermal equilibrium at the entropy maximum.
The Clausius Inequality¶
The entropy definition applies specifically to reversible heat transfer. A more general statement, valid for both reversible and irreversible processes, is the Clausius inequality:
Equality holds for a reversible process: .
Strict inequality holds for an irreversible process: .
Here is the temperature at which the heat is transferred to the system. For heat exchanged with a reservoir, it is the temperature of the reservoir.
The physical interpretation is that irreversibility produces additional entropy beyond the entropy “carried in” by heat flow. For an isolated system (), the Clausius inequality reduces to , recovering the second law.
The Clausius inequality also shows why the Carnot engine (Section 4.2) sets the maximum efficiency. Consider any engine, reversible or not, that in each cycle absorbs heat from the hot reservoir and releases heat to the cold reservoir (). For the Carnot engine these are and . The working substance returns to its initial state, so its entropy change over the cycle is zero, and the Clausius inequality, summed over the cycle, gives
Because , this rearranges to . The efficiency, defined as in Section 4.2, is therefore
Equality holds only for a reversible engine. An irreversible engine releases more heat to the cold reservoir for the same heat absorbed, so it delivers less work.
Entropy and the Number of Microstates: Boltzmann’s Formula¶
We now connect entropy to the microscopic picture. For an isolated system (microcanonical ensemble), the fundamental postulate of statistical mechanics (Section 2.1) assigns equal probability to each of the accessible microstates. The entropy of such a system is
Here is the number of microstates compatible with the macroscopic constraints (fixed , , ). You can think of as the degeneracy of the macrostate: fix the total energy and other extensive quantities, then count how many distinct microscopic configurations share those values.
As an example, place distinguishable particles in a box and record only which half of the box each particle is in. A microstate is one assignment of the particles to the two halves, and there are of them. The macrostate with particles in the left half contains
microstates. Figure 4.3.2 shows all microstates for and the shape of for larger .
Source
import sys
from itertools import product
from math import comb, lgamma
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.patches import Circle, Rectangle
sys.path.append("..") # the book folder, which holds figstyle
import figstyle as fs
N_SMALL = 4 # number of particles in panel (a)
SLOT = [(0.14, 0.34), (0.36, 0.34), (0.14, 0.12), (0.36, 0.12)] # place of each particle inside a half
BOX_W, BOX_H, STEP_X, STEP_Y = 1.0, 0.46, 1.16, 0.58 # size of one small box and spacing of the boxes
N_LARGE = [10, 100, 1000] # numbers of particles in panel (b)
def draw_microstate(ax, c, x, y, sides):
"""One microstate: a box with two halves; ``sides[k]`` is 0 if particle k is in the left half, 1 if in the right."""
colors = [c.blue, c.vermilion, c.green, c.purple]
ax.add_patch(Rectangle((x, y), BOX_W, BOX_H, fc=c.fill, ec=c.ink, lw=0.9))
ax.plot([x + BOX_W / 2] * 2, [y, y + BOX_H], color=c.gray, lw=0.9, ls=(0, (2, 2)))
for k, side in enumerate(sides):
ax.add_patch(Circle((x + side * BOX_W / 2 + SLOT[k][0], y + SLOT[k][1]), 0.075, fc=colors[k], ec="none"))
def relative_count(n_total):
"""Fraction in the left half and Omega / Omega_max for every n_L from 0 to n_total."""
n_left = np.arange(n_total + 1)
log_omega = np.array([lgamma(n_total + 1) - lgamma(n + 1) - lgamma(n_total - n + 1) for n in n_left])
return n_left / n_total, np.exp(log_omega - log_omega.max())
def draw_two_halves(c):
fig, (ax_a, ax_b) = plt.subplots(1, 2, figsize=(fs.WIDTH, 3.5), constrained_layout=True,
gridspec_kw={"width_ratios": [1.45, 1]})
# (a) All microstates for four particles, one column for each value of n_L
states = list(product([0, 1], repeat=N_SMALL))
for n_left in range(N_SMALL + 1):
column = [sides for sides in states if sides.count(0) == n_left]
x = (N_SMALL - n_left) * STEP_X # the column with all particles on the left stands first
for row, sides in enumerate(column):
draw_microstate(ax_a, c, x, row * STEP_Y, sides)
ax_a.text(x + BOX_W / 2, -0.16, f"$n_L = {n_left}$", ha="center", va="top")
ax_a.text(x + BOX_W / 2, -0.52, f"$\\Omega = {comb(N_SMALL, n_left)}$", ha="center", va="top")
ax_a.set(xlim=(-0.08, 4 * STEP_X + BOX_W + 0.08), ylim=(-0.95, 6 * STEP_Y))
ax_a.set_aspect("equal")
ax_a.axis("off")
fs.panel(ax_a, "a", "Four particles")
# (b) The count of microstates narrows around the even split as N grows
for n_total, shade in zip(N_LARGE, [0.55, 0.25, 0.0]):
fraction, omega = relative_count(n_total)
ax_b.plot(fraction, omega, color=c.tint(c.blue, shade), lw=fs.LINE, marker="o" if n_total == 10 else None,
ms=4.5, label=f"$N = {n_total}$")
ax_b.legend(loc="upper right", frameon=False, handlelength=1.3, borderaxespad=0.2)
ax_b.set(xlim=(0, 1), ylim=(0, 1.08), xlabel="Fraction in the left half, $n_L/N$",
ylabel="$\\Omega(n_L)\\,/\\,\\Omega_{\\max}$")
ax_b.set_xticks([0, 0.5, 1], ["0", "0.5", "1"])
ax_b.set_yticks([0, 0.5, 1], ["0", "0.5", "1"])
fs.panel(ax_b, "b", "More particles")
return fig
fs.show(draw_two_halves, alt="Panel a shows sixteen small boxes, each with two halves and four colored particles, in five "
"columns. The columns hold one box with four particles on the left, four boxes with three, "
"six boxes with two, four boxes with one, and one box with none. Panel b plots the number "
"of microstates relative to its maximum against the fraction of particles in the left "
"half for 10, 100, and 1000 particles. Each curve peaks at one half, and the peak becomes "
"much narrower as the number of particles grows.")Figure 4.3.2:Microstates of distinguishable particles in the two halves of a box. (a) The microstates for , sorted by the number of particles in the left half. Each color marks one particle. (b) The number of microstates , divided by its largest value , against the fraction of the particles in the left half for three values of . As grows, almost all microstates lie close to the even split.
In Figure 4.3.3, the particles of an ideal gas start in the left half of the box, as in the free expansion of Example 2 in Section 4.1. Open the partition, and follow and as the gas spreads.
Figure 4.3.3:Expansion of particles into a vacuum. Panel (a) shows the particles, which do not interact. Panel (b) shows the fraction of the particles in the left half and the entropy against time. The dashed lines mark the even split and the largest possible value of .
Two things to try:
Run the expansion with and then with . After the gas has spread, compare how far and move away from the dashed lines.
For , compare the largest value of with . Example 2 of Section 4.1 gives for a gas that doubles its volume, and .
For a few particles, falls below its largest value from time to time. For the 1023 particles of a macroscopic gas, such fluctuations are far too small to observe, which is why the Second Law can be stated as .
Why the Logarithm?¶
A key property of thermodynamic entropy is that it is extensive: for two independent subsystems and ,
But the number of microstates is multiplicative for independent systems:
The logarithm converts the product into a sum:
Without the logarithm, entropy would not be additive for independent subsystems.
Entropy in Terms of Probabilities: The Gibbs Entropy¶
The Gibbs (or Gibbs–Shannon) formula generalizes Boltzmann’s result from “counting equally likely microstates” to working with an arbitrary probability distribution over microstates :
A microstate with contributes nothing to the sum, because as . For two independent subsystems and , the probability that is in microstate while is in microstate is the product .
Figure 4.3.4 compares three distributions over the same six microstates.
Source
import sys
import numpy as np
import matplotlib.pyplot as plt
sys.path.append("..") # the book folder, which holds figstyle
import figstyle as fs
M_STATES = 6
RATIOS = [0.25, 0.65, 1.0] # each distribution is p_i proportional to ratio**(i - 1)
TITLES = ["Concentrated", "Spread out", "Uniform"]
def distribution(ratio):
weights = ratio ** np.arange(M_STATES)
return weights / weights.sum()
def gibbs_entropy(p):
"""Gibbs entropy in units of k_B."""
return -np.sum(p * np.log(p))
def draw_distributions(c):
fig, axes = plt.subplots(1, 3, figsize=(fs.WIDTH, 2.7), constrained_layout=True, sharey=True)
states = np.arange(1, M_STATES + 1)
for ax, letter, title, ratio in zip(axes, "abc", TITLES, RATIOS):
p = distribution(ratio)
ax.bar(states, p, width=0.7, color=c.blue)
ax.text(M_STATES + 0.45, 0.76, f"$S = {gibbs_entropy(p):.2f}\\,k_{{\\mathrm{{B}}}}$", ha="right", va="top")
ax.set(xlim=(0.35, M_STATES + 0.65), ylim=(0, 0.8), xlabel="Microstate $i$")
ax.set_xticks(states)
ax.set_yticks([0, 0.2, 0.4, 0.6, 0.8])
fs.panel(ax, letter, title)
axes[0].set_ylabel("Probability $p_i$")
return fig
fs.show(draw_distributions, alt="Three bar charts of probability against microstate for six microstates. In panel a, the "
"first microstate has a probability of 0.75 and the others fall off quickly, and the "
"entropy is 0.75 k B. In panel b, the probabilities fall off slowly from 0.38, and the "
"entropy is 1.56 k B. In panel c, all six probabilities are equal, and the entropy is "
"1.79 k B.")Figure 4.3.4:Three probability distributions over the same microstates, with the Gibbs entropy of each. The entropy is smallest for the distribution in (a), which is concentrated on few microstates, and largest for the uniform distribution in (c), for which .
This formula has two important features:
It reduces to Boltzmann’s formula in the microcanonical case. If all accessible microstates are equally likely (), then
It applies when microstates are not equally likely — for instance, when the system is in contact with a heat bath (canonical ensemble). In this case, , and different microstates have different probabilities.
Canonical Ensemble: Entropy in Terms of the Partition Function¶
We now evaluate the Gibbs entropy for the canonical ensemble. Recall from Section 2.2 that a closed system (fixed , ) in thermal contact with a reservoir at temperature has microstate probabilities
where the canonical partition function is
Derivation¶
Starting from the Gibbs entropy,
insert :
Using and the definition of internal energy from Section 2.3,
we obtain
Substituting :
This is a central result: it expresses the macroscopic state function directly in terms of the partition function and the internal energy , both of which we know how to compute from microscopic energy levels.
Connection to the Helmholtz Free Energy¶
Rearranging the boxed result gives
The left-hand side is the Helmholtz free energy, — a state function whose natural variables are and . We therefore have
This result was foreshadowed in Section 2.3 (where it was stated without derivation) and in Section 3.3 (where we noted that would be “convenient at constant and ”). It is arguably the single most important equation in canonical statistical mechanics: if you can compute , you can compute , and from you can derive all other thermodynamic quantities — , , , , and chemical potentials — by taking appropriate derivatives. Section 5.1 develops this machinery.
Microscopic Interpretation of Heat (Connection to Section 3.2)¶
The results above connect naturally to the microscopic interpretation of the First Law developed in Section 3.2. Differentiating gives
For a reversible process in a closed system with only work, comparing with , we identified (Section 3.2):
Heat: — energy exchange via redistributing probabilities among fixed energy levels.
Work: — energy exchange via shifting the energy levels themselves (e.g., changing volume changes the particle-in-a-box levels).
The Gibbs entropy formula leads to the same identification. Its differential is
because . Along a reversible path the system stays in equilibrium, so the probabilities remain canonical, with . Substituting, and using once more,
This is the definition of entropy from Section 4.1, now obtained from the microscopic side. In a reversible process the entropy changes only through the heat term. Reversible work alone () leaves the entropy unchanged, in agreement with the macroscopic result that on a reversible adiabat.
The restriction to reversible processes matters. In the free expansion of Example 2 in Section 4.1, no heat flows and no work is done, yet the distribution over microstates changes and the entropy of the gas increases.
Worked Examples¶
Example 1: Entropy of a Two-State Distribution¶
Problem. A system has two microstates with probabilities and . Compute the Gibbs entropy and find where it is maximized.
Solution. The Gibbs entropy is
The derivative vanishes at , where
To confirm this is a maximum, note that is concave (its second derivative for ) and is zero at the endpoints and .
Result. The maximum entropy occurs at , where the distribution is most spread out. This is consistent with Boltzmann’s formula: two equally likely microstates give and .
Example 2: Entropy of the Two-State System from Section 2.2¶
Problem. In Section 2.2, we analyzed a two-state system with energies and and found and at . Compute the entropy two ways: (a) from the Gibbs formula, and (b) from the canonical identity .
Solution.
(a) Gibbs formula.
Evaluating: and , so
(b) Canonical identity.
From Section 2.2: , and .
Result. Both methods give the same answer, as they must — the canonical identity is derived from the Gibbs formula. Notice that the entropy () is less than the maximum () because the distribution is not perfectly uniform: the ground state is slightly more populated than the excited state at 300 K. At higher temperatures, the two probabilities would become more equal, and would approach from below.
Concept Checks¶
How does “entropy increases until equilibrium” translate into a statement about probability distributions over microstates?
Why does require the microcanonical assumption ? What goes wrong if you try to apply it to a canonical ensemble?
In the Gibbs entropy formula, why does when for one state and for all others?
How can entropy be extensive (additive for independent subsystems) while probability distributions are normalized to 1?
Starting from , how would you obtain and as functions of ? (Hint: think about which partial derivatives of give and .)
Key Takeaways¶
Entropy quantifies how many microstates — or how much probability weight — a macrostate contains.
Boltzmann’s applies to isolated systems (microcanonical ensemble, equal probabilities). The logarithm ensures extensivity.
The Gibbs formula generalizes Boltzmann’s result to any probability distribution and reduces to it in the microcanonical limit.
In the canonical ensemble, ties entropy directly to the partition function.
Rearranging gives the Helmholtz free energy — the master potential from which all canonical thermodynamic quantities can be derived.
For a reversible process, the Gibbs formula gives , the definition of Section 4.1: the entropy changes through the heat term of the microscopic First Law (Section 3.2).