Data Analysis: Finding Patterns in the Numbers

Data analysis uses summary statistics to describe and compare datasets. The mean (average), median (middle value), and mode (most frequent value) each describe the centre of a distribution in different ways — and they can give very different answers depending on the shape of the data. The range, interquartile range, and standard deviation describe the spread. Outliers — values far from the rest — can distort means dramatically, which is why median household income is usually a better summary than mean household income.

From Gauss to the data revolution

Carl Friedrich Gauss formalised the method of least squares in 1809 while working on the orbit of the newly discovered asteroid Ceres — fitting a curve to noisy observations. This work laid the foundation for statistical data analysis. Francis Galton, studying heredity in the 1880s, discovered 'regression to the mean' — the tendency of extreme measurements to be followed by less extreme ones — and invented correlation as a way to measure the relationship between two variables. Modern statistics emerged from this tradition, and today 'data scientist' is one of the most sought-after job titles in the economy, largely because the tools of data analysis are the same as they've always been, just applied to vastly larger datasets.

Data analysis in every field

Medical researchers use data analysis to determine whether a treatment works in a clinical trial — comparing outcomes between treatment and control groups, and calculating whether any difference is statistically significant. Sports teams now employ analysts to identify undervalued players and optimal tactics (the 'Moneyball' phenomenon). Businesses analyse sales data to understand which products and customers are most profitable. Governments use census data analysis to allocate funding to schools and hospitals. Polling organisations analyse survey responses to predict election results. Wherever decisions need to be made from messy real-world data, data analysis is the process that brings clarity.