On the stubborn beauty of chance

I have been teaching introductory probability for the better part of a decade, and I still think it is the strangest subject on the undergraduate menu. It asks you to reason precisely about things that have not happened and may never happen. This post is the long version of a lecture I keep failing to fit into fifty minutes.

Most people meet probability as a bag of recipes: multiply along independent branches, add across mutually exclusive ones, divide by the total number of equally likely outcomes. The recipes work, which is exactly the problem — they work well enough that it is easy to never ask why. My aim here is to take the slow road, starting from almost nothing and arriving, eventually, at the two theorems that justify the whole enterprise. Pour something warm. This is going to take a while.

1. What are we even measuring?

Before we can assign a number to an uncertain event, we have to say what the events are. The modern answer, which took mathematicians an embarrassingly long time to settle on, is disarmingly concrete. We begin with a set — call it the sample space, written Ω — whose elements are the possible outcomes of our experiment, listed at whatever level of detail we care about.

Roll a single die and Ω = {1, 2, 3, 4, 5, 6}. Flip a coin twice and the natural sample space is {HH, HT, TH, TT}. Measure how long until a radioactive atom decays and Ω becomes the whole positive real line, which is where the trouble starts, but we will get there. The point of insisting on Ω first is that it forces honesty. Many apparent paradoxes in probability are just two people quietly using different sample spaces and never noticing.

An event is then simply a subset of Ω. "The die shows an even number" is the subset {2, 4, 6}. "At least one head in two flips" is {HH, HT, TH}. This is the first genuinely useful idea: by turning events into sets, we get to borrow the entire machinery of set theory. "A and B" becomes intersection, "A or B" becomes union, "not A" becomes complement. The logic of uncertainty inherits the grammar of sets for free.

The whole subject rests on a small act of translation: outcomes become points, events become sets, and the vague verb "probably" becomes a function that assigns numbers to those sets.

2. The three rules that start everything

In 1933 Andrey Kolmogorov wrote down the axioms that every modern treatment still uses. They are shockingly few. A probability is a function P that takes an event and returns a number, subject to three demands:

1. Nothing is negative. For every event A, we have P(A) ≥ 0. Probabilities are not allowed to be in debt.
2. Something happens. P(Ω) = 1. The total probability of the whole sample space is one; the experiment is certain to produce some outcome.
3. Separate things add up. If events A₁, A₂, … never overlap, then the probability of their union is the sum of their probabilities.

That is the entire foundation. Everything else — every formula you half-remember from school — is a theorem squeezed out of these three lines. For instance, the probability of "not A" must be 1 − P(A), because A and its complement are separate and together fill Ω. The empty event has probability zero. And for any two events, overlapping or not,

You subtract the overlap because adding the two pieces counts it twice. This is the inclusion–exclusion principle in its smallest form, and it generalises to any number of sets with alternating signs, a fact that is either elegant or a bookkeeping nightmare depending on your mood.

A note on the word "equally likely"

School probability leans hard on counting: probability equals favourable outcomes over total outcomes. That shortcut is a special case, valid only when every outcome in Ω genuinely deserves the same weight. It is a modelling assumption, not a law of nature, and it fails quietly all the time. The classic trap: "I either win the lottery or I don't, so my chance is one half." Two outcomes, yes — equally likely, emphatically no.

3. Conditional probability, or the art of updating

Here is where probability stops being combinatorics and starts being a theory of knowledge. Suppose you learn that event B has occurred. How should that reshape your belief about A? The definition is almost forced on us:

Read it as: restrict attention to the world in which B happened, and ask what fraction of that smaller world also has A. We are renormalising. B becomes the new sample space, and its probability — formerly less than one — is scaled back up to one so the bookkeeping still balances.

Rearranging gives the multiplication rule, P(A ∩ B) = P(A | B) · P(B), which is how you walk along a branch of a probability tree. And two events are called independent precisely when conditioning on one tells you nothing about the other: P(A | B) = P(A), equivalently P(A ∩ B) = P(A) · P(B). Independence is not the same as being disjoint — in fact disjoint events with positive probability are maximally dependent, since one happening rules the other out entirely. Students confuse these two for years. I did.

Bayes' theorem, finally

Flip the conditional around and you get the single most consequential formula in applied mathematics:

It looks like a triviality — it is a triviality, a one-line consequence of the definition — and yet it reorganises how you think. It says the probability you care about, P(A | B), is built from the probability you can usually measure, P(B | A), adjusted by how plausible A was to begin with. That last factor, the prior, is the part people forget, and forgetting it produces a whole genre of mistakes.

The medical-test example is tired but it earns its keep. A disease affects one person in a thousand. A test catches it 99% of the time and gives a false positive only 5% of the time. You test positive. Most people's gut says you are almost certainly sick. Run the numbers: out of 100,000 people, 100 are ill and about 99 of them test positive; of the 99,900 healthy people, roughly 4,995 also test positive. So among everyone who tests positive, only 99 out of about 5,094 are actually ill — under 2%. The test is good; the disease is just rare, and rarity dominates. The prior did all the work, and the gut ignored it.

4. Random variables: putting numbers on outcomes

It is awkward to keep saying "the outcome where the die shows four." We would rather do arithmetic. A random variable is the bridge: it is a function that assigns a real number to each outcome in Ω. Let X be the number of heads in three coin flips, and X turns eight unwieldy outcomes into the tidy values 0, 1, 2, 3, each with its own probability.

Once outcomes carry numbers, we can ask for a single number that summarises the whole distribution. The expectation — the mean, the long-run average — weights each value by how probable it is:

Expectation is linear, which is one of those facts that sounds modest and turns out to be a superpower. E[X + Y] = E[X] + E[Y] always — whether or not X and Y are independent. This is why you can compute the expected number of matches, collisions, or fixed points in fiendishly tangled problems by breaking them into indicator variables and adding, never once worrying about how they interact.

The variance measures spread: how far, on average, the variable lands from its own mean. Written Var(X) = E[(X − E[X])²], it is the expected squared deviation. Two distributions can share a mean and feel completely different — one tightly clustered, one wildly dispersed — and variance is the first number that tells them apart. Its square root, the standard deviation, lives in the same units as X, which is why it is the one people actually quote.

5. Why averages behave: the law of large numbers

Now we reach the results that make probability trustworthy rather than merely consistent. The law of large numbers is the theorem your intuition has been assuming all along without permission. It says: take independent repetitions of the same experiment, average the outcomes, and as the number of trials grows the average converges to the true expectation.

Flip a fair coin ten times and you might see seven heads; that 70% is unremarkable. Flip it ten thousand times and a result as lopsided as 70% becomes astronomically unlikely — the proportion is pinned ever more tightly to one half. The randomness in any single flip never goes away. What happens is that the fluctuations grow more slowly than the number of trials, so dividing by n crushes them toward zero.

This is the theorem casinos are built on. No individual bet is predictable; the aggregate of millions is predictable to the decimal. It is also the theorem most often misquoted as the "law of averages" — the false belief that a run of tails makes heads "due." The coin has no memory. The law promises convergence in the long run; it promises nothing about correcting the short run, and the gambler who waits for balance to be restored is waiting for a favour the mathematics never offered.

The law of large numbers tells you the average will settle down. It is studiously silent on when, and that silence has emptied a great many wallets.

6. The shape everything hides inside: the central limit theorem

If the law of large numbers is the reassuring theorem, the central limit theorem is the uncanny one. It says that when you add up many small, independent random effects, their sum looks like a bell curve — the normal distribution — almost regardless of how the individual pieces are distributed.

Sit with how strange that is. The ingredients can be lopsided, discrete, bizarre. Sum enough of them, rescale, and the lumps smooth into the same symmetric curve every time. This is why the normal distribution shows up in heights and measurement errors and test scores and noise: each of those is the accumulation of countless little independent pushes, and the theorem guarantees the pushes conspire into a bell. Nature did not decide to love the normal distribution. Addition did.

More precisely: take independent, identically distributed variables with finite mean μ and finite variance σ². Their standardised sum converges in distribution to a standard normal, the familiar curve with density proportional to e^−z2/2. The finite-variance condition matters more than textbooks admit — distributions with heavy enough tails break the theorem, and much of the drama of modern risk modelling is really an argument about whether a given quantity has a finite variance or only pretends to.

7. A detour that still bothers me

I promised earlier we would return to the continuous case. When Ω is the whole real line, every single outcome has probability zero — the chance of a dart landing on exactly the point π is nil — yet the dart lands somewhere. We are forced to assign probability to intervals rather than points, via a density we integrate. Fine. But then a genuinely subtle question surfaces: can we assign a sensible probability to every subset of the line?

The disquieting answer is no. There exist sets so pathological that no consistent notion of length, hence no probability, can be attached to them without contradiction. This is why measure theory exists, and why the fully rigorous version of this whole post restricts events to a special collection of "measurable" sets rather than all subsets of Ω. For every applied purpose you will ever meet, the measurable sets are all of them. But the fact that mathematics had to carve out an exception at all is, to me, the most honest thing about the subject. It admits where its own tools run out.

8. Where to go next

If this was your first slow pass, the natural next steps are the named distributions — binomial, Poisson, exponential, normal — each of which is just a recurring pattern of chance frozen into a formula. After that, Markov chains, where the future depends on the present but mercifully not the whole past, and where a surprising amount of the modern world (search rankings, queueing systems, the physics of diffusion) turns out to live.

But do not rush. The recipes will always be waiting. The ideas underneath — that uncertainty has a grammar, that belief can be updated by a rule, that chaos averages into order — are worth sitting with. I have been sitting with them for ten years and they have not gotten less surprising. If anything, the opposite.