Center, position, and spread: the handful of numbers that summarize a pile of data — and how to compute them without fooling yourself.
Before any model, test, or chart, someone asks "so what does the data look like?" The answer is a few numbers: where the middle is, how far things spread, and whether anything is wildly out of line. Every later module is built from them — a confidence interval is a mean plus or minus a standard deviation in disguise, and a regression line is fitted by minimizing squared deviations, the same quantity behind the variance.
Descriptive statistics are also where most real-world misleading happens. "The average salary here is $175k" can be literally true and completely wrong about what a typical employee earns.
Center. The mean is the balance point: every value pulls on it in proportion to its size, so one extreme value can drag it a long way. The median is the middle of the sorted data; it only cares about order, so it barely notices outliers. The mode is the most common value — the only center that works for categories.
Position. A quantile answers "what value is a fraction q of the way up the sorted data?" The median is the 0.5-quantile; the quartiles Q1 and Q3 are the 0.25 and 0.75. The five-number summary (min, Q1, median, Q3, max) is what a box plot draws.
Spread. The interquartile range (IQR = Q3 − Q1) is the width of the middle half — robust, like the median. The variance is the average squared distance from the mean, and the standard deviation is its square root, back in the data's own units. A z-score re-expresses a value as "standard deviations from the mean", which lets you compare things measured on different scales.
Sample versus population. If your data is the whole population, divide the squared deviations by n. If it's a sample and you're estimating the population's variance, divide by n − 1. The deviations are measured from the sample's own mean, which sits closer to the sample than the true mean does, so they come out too small on average; n − 1 corrects that (Bessel's correction).
| Quantity | Formula | In words |
|---|---|---|
| Mean | x̄ = Σx / n | the balance point |
| Median | middle of the sorted values | half above, half below |
| Quantile | h = (n − 1)·q, then interpolate between s[⌊h⌋] and s[⌊h⌋+1] | a fraction q of the way up |
| IQR | Q3 − Q1 | width of the middle half |
| Population variance | σ² = Σ(x − μ)² / N | you have every value |
| Sample variance | s² = Σ(x − x̄)² / (n − 1) | you have a sample |
| Standard deviation | s = √s² | typical distance from the mean |
| z-score | z = (x − x̄) / s | standard deviations from the mean |
| Tukey's fences | Q1 − 1.5·IQR and Q3 + 1.5·IQR | outside them: flag as an outlier |
slices.Sort sorts in place. A Median that sorts its argument reorders the caller's data — a nasty side effect. Sort a copy: s := slices.Clone(xs).sum / len(xs) doesn't compile when sum is a float64; write sum / float64(len(xs)). With integer data, 7 / 2 is 3.math.NaN() creates one; test with math.IsNaN(x), never x == math.NaN() — NaN isn't equal to anything, itself included.map[float64]int), with two catches: 0.0 and -0.0 are the same key, and every NaN is a different key.package main import ( "fmt" "slices" ) func mean(xs []float64) float64 { sum := 0.0 for _, x := range xs { sum += x } return sum / float64(len(xs)) } func median(xs []float64) float64 { s := slices.Clone(xs) // sort a copy, not the caller's data slices.Sort(s) n := len(s) if n%2 == 1 { return s[n/2] } return (s[n/2-1] + s[n/2]) / 2 } func main() { // Salaries at a small firm, in thousands of dollars (made-up numbers). staff := []float64{48, 52, 55, 58, 61, 63, 67, 72, 80} fmt.Printf("staff only: mean %6.1f median %5.1f\n", mean(staff), median(staff)) withCEO := append(slices.Clone(staff), 1200) // the CEO joins the spreadsheet fmt.Printf("plus the CEO: mean %6.1f median %5.1f\n", mean(withCEO), median(withCEO)) }
It prints:
The mean nearly triples while the median moves by one. Neither number is wrong — they answer different questions. The mean is what each person would get if the payroll were split evenly; the median is what a typical person actually earns. For skewed data like incomes, house prices, and response times, report the median (or both).
VAR.P/STDEV.P divide by n; VAR.S/STDEV.S by n − 1. NumPy's np.var defaults to n.Open exercises.go. Part A computes the center, Part B position (quantiles, the five-number summary, outliers), and Part C spread, ending with a streaming Running accumulator that never stores the data.
go test ./01-descriptive-statistics/ # all tests in the module go test -run TestQuantile -v ./01-descriptive-statistics/ # one test, verbose
Write Quantile early — Median, FiveNumber, and Outliers can all be built on it. The TestRunning case with values near one billion is the one the shortcut formula fails.
Everything below is also in the course download, in 01-descriptive-statistics/. Work in exercises.go; the tests decide when you’re done.
// Package describe is Module 1: descriptive statistics. // // Work in exercises.go. Run `go test ./01-descriptive-statistics/` to check // your work, or add `-tags solutions` to run the same tests against the // reference solutions in solution.go. // // A convention for the whole course: a statistic that isn't defined for the // input — the mean of nothing, the sample variance of one value — returns // math.NaN() rather than a made-up number or a panic. package describe // Summary is Tukey's five-number summary: the five numbers a box plot draws. type Summary struct { Min, Q1, Median, Q3, Max float64 } // IQR returns the interquartile range, Q3 − Q1: the width of the middle half // of the data. func (s Summary) IQR() float64 { return s.Q3 - s.Q1 } // Running computes the mean and variance of a stream in one pass without // storing the values (Welford's algorithm). The zero value is ready to use. type Running struct { n int mean float64 // mean of the values added so far m2 float64 // sum of squared deviations from the current mean } // N returns how many values have been added. func (r *Running) N() int { return r.n }
//go:build !solutions package describe // ============================================================ // Part A — the center // ============================================================ // Mean returns the arithmetic mean, Σx / n, or NaN for an empty slice. // [2 4 4 4 5 5 7 9] -> 5. Target: O(n). func Mean(xs []float64) float64 { // TODO return 0 } // Median returns the middle value of the sorted data — the average of the // two middle values when n is even — or NaN for an empty slice. // Don't modify xs: sort a copy (slices.Clone). [4 1 3 2] -> 2.5. // Target: O(n log n). func Median(xs []float64) float64 { // TODO return 0 } // Mode returns every value that occurs most often, in ascending order // (data can have several modes). [1 2 2 3 3] -> [2 3]. Empty input -> nil. // Pattern: count with a map. Target: O(n) plus sorting the modes. func Mode(xs []float64) []float64 { // TODO return nil } // ============================================================ // Part B — position // ============================================================ // Quantile returns the q-quantile (0 <= q <= 1) by linear interpolation // between order statistics: sort a copy, let h = (n−1)·q, and go fraction // h−⌊h⌋ of the way from s[⌊h⌋] to s[⌊h⌋+1]. This is the default in NumPy, // R, and Excel's PERCENTILE.INC. NaN for empty input or q outside [0, 1]. // [1 2 3 4], q=0.25 -> 1.75. Target: O(n log n). func Quantile(xs []float64, q float64) float64 { // TODO return 0 } // FiveNumber returns the minimum, Q1 (q=0.25), median, Q3 (q=0.75), and // maximum, all computed with Quantile. Empty input -> every field NaN. func FiveNumber(xs []float64) Summary { // TODO return Summary{} } // Outliers returns the values outside Tukey's fences, [Q1 − 1.5·IQR, // Q3 + 1.5·IQR], in their original order. Values exactly on a fence are // not outliers. [10 12 11 13 12 95 11 -40 12] -> [95 -40]. func Outliers(xs []float64) []float64 { // TODO return nil } // ============================================================ // Part C — spread // ============================================================ // PopVariance returns the population variance, Σ(x − x̄)² / n: use it when // xs is the whole population. NaN for empty input. Two passes: the mean // first, then the squared deviations. [2 4 4 4 5 5 7 9] -> 4. func PopVariance(xs []float64) float64 { // TODO return 0 } // Variance returns the sample variance, Σ(x − x̄)² / (n − 1): use it when xs // is a sample and you're estimating the population's variance. NaN when // n < 2. [2 4 4 4 5 5 7 9] -> 32/7 ≈ 4.571. func Variance(xs []float64) float64 { // TODO return 0 } // StdDev returns the sample standard deviation, √Variance. NaN when n < 2. func StdDev(xs []float64) float64 { // TODO return 0 } // ZScores returns (x − x̄) / s for every value, using the sample standard // deviation s: how many standard deviations each value sits from the mean. // Returns nil when n < 2 or every value is the same (s = 0). func ZScores(xs []float64) []float64 { // TODO return nil } // Add folds x into the running mean and variance (Welford's algorithm): // // n++; d := x − mean; mean += d/n; m2 += d·(x − mean) // // Note the second factor uses the UPDATED mean. Target: O(1) per value. func (r *Running) Add(x float64) { // TODO } // Mean returns the mean of the values added so far, or NaN if none. func (r *Running) Mean() float64 { // TODO return 0 } // Variance returns the sample variance of the values so far, m2 / (n − 1), // or NaN when fewer than two values have been added. func (r *Running) Variance() float64 { // TODO return 0 }
package describe import ( "math" "slices" "testing" ) // near reports whether a and b agree to about 9 significant digits. func near(a, b float64) bool { return math.Abs(a-b) <= 1e-9*math.Max(1, math.Abs(b)) } var eight = []float64{2, 4, 4, 4, 5, 5, 7, 9} func TestMean(t *testing.T) { for _, tc := range []struct { in []float64 want float64 }{ {eight, 5}, {[]float64{-1.5, 2.5}, 0.5}, {[]float64{42}, 42}, } { if got := Mean(tc.in); !near(got, tc.want) { t.Errorf("Mean(%v) = %v, want %v", tc.in, got, tc.want) } } if got := Mean(nil); !math.IsNaN(got) { t.Errorf("Mean(nil) = %v, want NaN", got) } } func TestMedian(t *testing.T) { odd := []float64{3, 1, 2} even := []float64{4, 1, 3, 2} if got := Median(odd); got != 2 { t.Errorf("Median(%v) = %v, want 2", odd, got) } if got := Median(even); got != 2.5 { t.Errorf("Median(%v) = %v, want 2.5", even, got) } if !slices.Equal(even, []float64{4, 1, 3, 2}) { t.Errorf("Median modified its input: %v", even) } if got := Median(nil); !math.IsNaN(got) { t.Errorf("Median(nil) = %v, want NaN", got) } } func TestMode(t *testing.T) { for _, tc := range []struct{ in, want []float64 }{ {[]float64{1, 2, 2, 3, 3}, []float64{2, 3}}, {[]float64{7, 7, 1}, []float64{7}}, {[]float64{5}, []float64{5}}, {[]float64{3, 1, 2}, []float64{1, 2, 3}}, } { if got := Mode(tc.in); !slices.Equal(got, tc.want) { t.Errorf("Mode(%v) = %v, want %v", tc.in, got, tc.want) } } if got := Mode(nil); len(got) != 0 { t.Errorf("Mode(nil) = %v, want empty", got) } } func TestQuantile(t *testing.T) { data := []float64{7, 1, 3, 9, 5, 11, 13, 2} for _, tc := range []struct{ q, want float64 }{ // from numpy.quantile {0, 1}, {0.1, 1.7}, {0.25, 2.75}, {0.5, 6}, {0.75, 9.5}, {0.9, 11.6}, {1, 13}, } { if got := Quantile(data, tc.q); !near(got, tc.want) { t.Errorf("Quantile(%v, %v) = %v, want %v", data, tc.q, got, tc.want) } } if !slices.Equal(data, []float64{7, 1, 3, 9, 5, 11, 13, 2}) { t.Errorf("Quantile modified its input: %v", data) } for _, q := range []float64{-0.1, 1.5} { if got := Quantile(data, q); !math.IsNaN(got) { t.Errorf("Quantile(data, %v) = %v, want NaN", q, got) } } if got := Quantile(nil, 0.5); !math.IsNaN(got) { t.Errorf("Quantile(nil, 0.5) = %v, want NaN", got) } } func TestFiveNumber(t *testing.T) { got := FiveNumber([]float64{7, 1, 3, 9, 5, 11, 13, 2}) want := Summary{Min: 1, Q1: 2.75, Median: 6, Q3: 9.5, Max: 13} if !near(got.Min, want.Min) || !near(got.Q1, want.Q1) || !near(got.Median, want.Median) || !near(got.Q3, want.Q3) || !near(got.Max, want.Max) { t.Errorf("FiveNumber = %+v, want %+v", got, want) } if got.IQR() != 6.75 { t.Errorf("IQR = %v, want 6.75", got.IQR()) } } func TestOutliers(t *testing.T) { data := []float64{10, 12, 11, 13, 12, 95, 11, -40, 12} // Q1 11, Q3 12: fences 9.5 and 13.5 if got, want := Outliers(data), []float64{95, -40}; !slices.Equal(got, want) { t.Errorf("Outliers(%v) = %v, want %v", data, got, want) } onFence := []float64{-15, 0, 0, 0, 10, 10, 10, 25} // Q1 0, Q3 10: fences exactly -15 and 25 if got := Outliers(onFence); len(got) != 0 { t.Errorf("Outliers(%v) = %v, want none (values on a fence are not outliers)", onFence, got) } } func TestVariance(t *testing.T) { if got := PopVariance(eight); !near(got, 4) { t.Errorf("PopVariance = %v, want 4", got) } if got := Variance(eight); !near(got, 32.0/7) { t.Errorf("Variance = %v, want 32/7 = %v", got, 32.0/7) } if got := StdDev(eight); !near(got, 2.138089935299395) { t.Errorf("StdDev = %v, want 2.138089935299395", got) } if got := Variance([]float64{3}); !math.IsNaN(got) { t.Errorf("Variance of one value = %v, want NaN", got) } if got := PopVariance(nil); !math.IsNaN(got) { t.Errorf("PopVariance(nil) = %v, want NaN", got) } } func TestZScores(t *testing.T) { want := []float64{-1.403121520040228, -0.46770717334674267, -0.46770717334674267, -0.46770717334674267, 0, 0, 0.9354143466934853, 1.8708286933869707} got := ZScores(eight) if len(got) != len(want) { t.Fatalf("ZScores returned %d values, want %d", len(got), len(want)) } for i := range want { if !near(got[i], want[i]) { t.Errorf("ZScores[%d] = %v, want %v", i, got[i], want[i]) } } if got := ZScores([]float64{4, 4, 4}); got != nil { t.Errorf("ZScores of identical values = %v, want nil", got) } } func TestRunning(t *testing.T) { data := []float64{3.1, -2.4, 8.8, 0.5, 7.25, 4.0, -1.1} var r Running if !math.IsNaN(r.Mean()) || !math.IsNaN(r.Variance()) { t.Error("an empty Running should report NaN mean and variance") } for _, x := range data { r.Add(x) } if r.N() != 7 || !near(r.Mean(), 2.8785714285714286) || !near(r.Variance(), 17.471547619047623) { t.Errorf("Running: n=%d mean=%v var=%v, want 7, 2.8785714285714286, 17.471547619047623", r.N(), r.Mean(), r.Variance()) } // Big values, small spread: the shortcut (Σx² − (Σx)²/n)/(n−1) returns 0 // here, because Σx² ≈ 4e18 has no digits left for the spread. Welford // keeps them. var big Running for _, x := range []float64{1e9 + 4, 1e9 + 7, 1e9 + 13, 1e9 + 16} { big.Add(x) } if got := big.Variance(); math.Abs(got-30) > 1e-6 { t.Errorf("Running variance of 1e9+{4,7,13,16} = %v, want 30", got) } }
Try not to open solution.go until your tests pass, or until you’ve been stuck for a solid half hour. Then compare: different-but-correct is fine; notice where the reference is simpler. go test -tags solutions ./01-descriptive-statistics/ runs the same tests against it.
solutions · 149 lines›//go:build solutions package describe import ( "math" "slices" ) // ============================================================ // Part A — the center // ============================================================ func Mean(xs []float64) float64 { if len(xs) == 0 { return math.NaN() } sum := 0.0 for _, x := range xs { sum += x } return sum / float64(len(xs)) } func Median(xs []float64) float64 { return Quantile(xs, 0.5) } func Mode(xs []float64) []float64 { counts := map[float64]int{} best := 0 for _, x := range xs { counts[x]++ best = max(best, counts[x]) } var out []float64 for x, c := range counts { if c == best { out = append(out, x) } } slices.Sort(out) return out } // ============================================================ // Part B — position // ============================================================ func Quantile(xs []float64, q float64) float64 { if len(xs) == 0 || q < 0 || q > 1 { return math.NaN() } s := slices.Clone(xs) slices.Sort(s) h := float64(len(s)-1) * q lo := int(math.Floor(h)) if lo >= len(s)-1 { return s[len(s)-1] } return s[lo] + (h-float64(lo))*(s[lo+1]-s[lo]) } func FiveNumber(xs []float64) Summary { return Summary{ Min: Quantile(xs, 0), Q1: Quantile(xs, 0.25), Median: Quantile(xs, 0.5), Q3: Quantile(xs, 0.75), Max: Quantile(xs, 1), } } func Outliers(xs []float64) []float64 { s := FiveNumber(xs) lo, hi := s.Q1-1.5*s.IQR(), s.Q3+1.5*s.IQR() var out []float64 for _, x := range xs { if x < lo || x > hi { out = append(out, x) } } return out } // ============================================================ // Part C — spread // ============================================================ func PopVariance(xs []float64) float64 { if len(xs) == 0 { return math.NaN() } return sumSq(xs) / float64(len(xs)) } func Variance(xs []float64) float64 { if len(xs) < 2 { return math.NaN() } return sumSq(xs) / float64(len(xs)-1) } // sumSq is Σ(x − x̄)². func sumSq(xs []float64) float64 { m := Mean(xs) ss := 0.0 for _, x := range xs { ss += (x - m) * (x - m) } return ss } func StdDev(xs []float64) float64 { return math.Sqrt(Variance(xs)) } func ZScores(xs []float64) []float64 { m, s := Mean(xs), StdDev(xs) if len(xs) < 2 || s == 0 { return nil } out := make([]float64, len(xs)) for i, x := range xs { out[i] = (x - m) / s } return out } func (r *Running) Add(x float64) { r.n++ d := x - r.mean r.mean += d / float64(r.n) r.m2 += d * (x - r.mean) } func (r *Running) Mean() float64 { if r.n == 0 { return math.NaN() } return r.mean } func (r *Running) Variance() float64 { if r.n < 2 { return math.NaN() } return r.m2 / float64(r.n-1) }
TrimmedMean(xs, 0.1): drop the lowest and highest 10% and average the rest — a compromise between mean and median.Four questions. Answer each one in your head — or out loud — before you open the answer.
Household incomes are right-skewed (a long tail of high earners). Which is usually larger?
B — The few very large values pull the mean toward the tail; the median only cares about the middle position.
Why does the sample variance divide by n − 1 instead of n?
B — x̄ is fitted to the sample, so Σ(x − x̄)² is never bigger than Σ(x − μ)². Dividing by n − 1 makes s² an unbiased estimate of σ².
A student scores 82 on a test with mean 70 and standard deviation 8. What is the z-score?
C — (82 − 70) / 8 = 1.5: one and a half standard deviations above the mean.
For the values 1e9+4, 1e9+7, 1e9+13, 1e9+16, the shortcut formula (Σx² − (Σx)²/n)/(n − 1) returns 0, but the true sample variance is 30. Why?
B — It's catastrophic cancellation. The formula is correct on paper; the arithmetic isn't. Subtract the mean first (two passes) or use Welford's update.
Commit your work before you move on to module 2, so git log shows your own progress through the course.