Prediction Revisited

A thought-provoking and startlingly insightful reworking of the science of prediction

In Prediction Revisited: The Importance of Observation, a team of renowned experts in the field of data-driven investing delivers a ground-breaking reassessment of the delicate science of prediction for anyone who relies on data to contemplate the future. The book reveals why standard approaches to prediction based on classical statistics fail to address the complexities of social dynamics, and it provides an alternative method based on the intuitive notion of relevance.

The authors describe, both conceptually and with mathematical precision, how relevance plays a central role in forming predictions from observed experience. Moreover, they propose a new and more nuanced measure of a prediction’s reliability. Prediction Revisited also offers:

• Clarifications of commonly accepted but less commonly understood notions of statistics
• Insight into the efficacy of traditional prediction models in a variety of fields
• Colorful biographical sketches of some of the key prediction scientists throughout history
• Mutually supporting conceptual and mathematical descriptions of the key insights and methods discussed within

With its strikingly fresh perspective grounded in scientific rigor, Prediction Revisited is sure to earn its place as an indispensable resource for data scientists, researchers, investors, and anyone else who aspires to predict the future from the data-driven lessons of the past.

Timeline of Innovations ix

Essential Concepts xi

Preface xv

1 Introduction 1

Relevance 2

Informativeness 3

Similarity 4

Roadmap 4

2 Observing Information 7

Observing Information Conceptually 7

Central Tendency 8

Spread 9

Information Theory 10

The Strong Pull of Normality 14

A Constant of Convenience 17

Key Takeaways 18

Observing Information Mathematically 20

Average 20

Spread 21

Information Distance 24

Observing Information Applied 26

Appendix 2.1: On the Inflection Point of the Normal Distribution 32

References 39

3 Co-occurrence 41

Co-occurrence Conceptually 41

Correlation as an Information-Weighted Average of Co-occurrence 46

Pairs of Pairs 49

Across Many Attributes 50

Key Takeaways 52

Co-occurrence Mathematically 54

The Covariance Matrix 58

Co-occurrence Applied 59

References 66

4 Relevance 67

Relevance Conceptually 67

Informativeness 68

Similarity 72

Relevance and Prediction 73

How Much Have You Regressed? 74

Partial Sample Regression 76

Asymmetry 80

Sensitivity 86

Memory and Bias 87

Key Takeaways 88

Relevance Mathematically 90

Prediction 95

Equivalence to Linear Regression 97

Partial Sample Regression 100

Asymmetry 102

Relevance Applied 107

Appendix 4.1: Predicting Binary Outcomes 114

Predicting Binary Outcomes Conceptually 114

Predicting Binary Outcomes Mathematically 116

References 121

5 Fit 123

Fit Conceptually 123

Failing Gracefully 125

Why Fit Varies 126

Avoiding Bias 129

Precision 130

Focus 133

Key Takeaways 134

Fit Mathematically 136

Components of Fit 138

Precision 139

Fit Applied 143

6 Reliability 149

Reliability Conceptually 149

Key Takeaways 153

Reliability Mathematically 155

Reliability Applied 163

References 168

7 Toward Complexity 169

Toward Complexity Conceptually 169

Learning by Example 170

Expanding on Relevance 171

Key Takeaways 175

Toward Complexity Mathematically 177

Complexity Applied 183

References 183

8 Foundations of Relevance 185

Observations and Relevance: A Brief Review of the Main Insights 186

Spread 187

Co-occurrence 187

Relevance 188

Asymmetry 188

Fit and Reliability 189

Partial Sample Regression and Machine Learning Algorithms 189

Abraham de Moivre (1667–1754) 190

Pierre-Simon Laplace (1749–1827) 192

Carl Friedrich Gauss (1777–1853) 193

Francis Galton (1822–1911) 195

Karl Pearson (1857–1936) 197

Ronald Fisher (1890–1962) 199

Prasanta Chandra Mahalanobis (1893–1972) 200

Claude Shannon (1916–2001) 202

References 206

Concluding Thoughts 209

Perspective 209

Insights 210

Prescriptions 210

Index 211

MEGAN CZASONIS is Managing Director and Head of Portfolio Management Research at State Street Associates.

MARK KRITZMAN is a Founding Partner and CEO of Windham Capital Management. He is also a Founding Partner of State Street Associates and teaches a graduate course at the Massachusetts Institute of Technology.

DAVID TURKINGTON is Senior Managing Director and Head of State Street Associates.

A thought-provoking and startlingly insightful reimagination of the science of prediction

In Prediction Revisited: The Importance of Observation, a team of renowned finance and risk experts at the top of their game describes a ground-breaking realignment of the connection between past experiences and future outcomes. The book reveals why standard approaches to prediction based on classical statistics fail to address the complexities of social dynamics, and it maps out an elegant prediction system based on a novel measure of statistical relevance.

Drawing upon information theory and an obscure yet profound mathematical equivalence, the authors describe, both conceptually and with mathematical precision, how relevance plays a central role in forming predictions. Additionally, they introduce a new and more nuanced measure of a prediction’s reliability, enabling researchers to fine tune their responses to specific predictions.

Prediction Revisited also:

• Illuminates many commonly accepted but less commonly understood notions of statistics
• Reveals several valuable yet previously unrecognized mathematical equivalences
• Includes colorful biographical sketches of some of the key scientists whose contributions paved the path to relevance-based prediction
• Enables access to the mathematically minded reader as well as those who prefer an intuitive and conceptual discussion of the book’s key ideas

With its strikingly fresh perspective grounded in scientific rigor, Prediction Revisited is a must-read for anyone who aspires to reach a new level of understanding and mastery of data-driven prediction.