Downloaded 23 times
![Predict ≠ Explain
+ ?
“we tried to benefit from an
extensive set of attributes
describing each of the movies in
the dataset. Those attributes
certainly carry a significant signal
and can explain some of the user
behavior. However… they could
not help at all for improving the
[predictive] accuracy.”
Bell et al., 2008](https://image.slidesharecdn.com/explainpredicttesthailandjan2018-180110152601/75/Statistical-Modeling-in-3D-Explaining-Predicting-Describing-27-2048.jpg)
![Predict ≠ Describe
Election Polls
“There is a subtle, but important, difference between
reflecting current public sentiment and predicting the
results of an election. Surveys have focused largely on
the former… [as opposed to] survey based prediction
models [that are] focused entirely on analysis and
projection”
Kenett, Pfefferman & Steinberg (2017) “Election Polls – A Survey, A Critique,
and Proposals”, Annual Rev of Stat & its Applications](https://image.slidesharecdn.com/explainpredicttesthailandjan2018-180110152601/75/Statistical-Modeling-in-3D-Explaining-Predicting-Describing-28-2048.jpg)
Uploaded byGalit Shmueli
Statistical Modeling in 3D: Explaining, Predicting, Describing
The document discusses the distinctions between explanatory, predictive, and descriptive statistical modeling, emphasizing their different purposes and methodologies. It addresses misconceptions in academia and industry about the interchangeability of these modeling types and highlights the importance of understanding their unique roles in research and practice. Additionally, it touches on the impact of predictive analytics in scientific research, suggesting that predictive power from explanatory models can provide value beyond mere utility.
More Related Content
Similar to Statistical Modeling in 3D: Explaining, Predicting, Describing
Statistical Modeling in 3D: Explaining, Predicting, Describing
- 1. Statistical Modeling in3D: Describing Explaining Predicting 11th Intl Conference of the Thailand Econometric Society, 1/2018 Galit Shmueli 徐茉 莉 Institute of Service Science
- 2. 1997-2000 (PhD, Statistics) IsraelInstitute of Technology Faculty of IE & M 2000-2002 Carnegie Mellon Univ. Department of Statistics 2002-2012 Univ. of Maryland Smith School of Business 2011-2014 Indian School of Business Hyderabad, India 2014-… National Tsing Hua Univ. Institute of Service Science My Academic Path My Research ‘Entrepreneurial’ statistical & data mining modeling Interdisciplinary Statistical Strategy • To Explain or To Predict? • Information Quality • Data Mining for Causality • Predicting with Causal Models
- 3. Road Map 1. Definitions 2.Monopolies & confusion in academia & industry 3. Explanatory, predictive, descriptive modeling & evaluation are different Why? Different modeling paths Explanatory power vs. predictive power 4. Where next?
- 4. Definitions: Explain Explanatory modeling theory-based,statistical testing of causal hypotheses Explanatory power strength of relationship in statistical model
- 5. Definitions: Predict Predictive modeling empiricalmethod for predicting new observations Predictive power ability to accurately predict new observations
- 6. Definitions: Describe Descriptive modeling statisticalmodel for approximating a distribution or relationship Descriptive power goodness of fit, generalizable to population
- 7.
- 9. Social sciences &management research Domination of ”Explain” Purpose: test causal theory (“explain”) Association-based statistical models Prediction & description nearly absent
- 10. Start with acausal theory Generate causal hypotheses on constructs Operationalize constructs → measurable variables Fit statistical model Classic journal paper Statistical inference → causal conclusions
- 11. Misconception #1: The samemodel is best for explaining, describing, predicting Social Sci & Mgmt: Build explanatory model and use it to ”predict” “A good explanatory model will also predict well” “You must understand the underlying causes in order to predict” “To examine the predictive power of the proposed model, we compare it to four models in terms of R2 adjusted”
- 12. Misconception #1: The samemodel is best for explaining, describing, predicting CS/eng/stat: Build a predictive model and use it to ”explain” in cs / stat / engineering / industry 2014 6th International Conference on Mobile Computing, Applications and Services (Agent-based modeling using census data) “our model is able to provide both predictions of how the population may vote and why they are voting this way”… 2009 IEEE International Conference on Systems, Man and Cybernetics
- 13. Misconception #2: explain >predict or predict > explain Emanuel Parzen, Comment on “Statistical Modeling: The Two Cultures” Statistical Science 2001 “Correlation supersedes causation, and science can advance even without coherent models, unified theories, or really any mechanistic explanation at all” *Chris Anderson is the editor in chief of Wired
- 15. Philosophy of Science “Explanationand prediction have the same logical structure” Hempel & Oppenheim, 1948 “It becomes pertinent to investigate the possibilities of predictive procedures autonomous of those used for explanation” Helmer & Rescher, 1959 “Theories of social and human behavior address themselves to two distinct goals of science: (1) prediction and (2) understanding” Dubin, Theory Building, 1969
- 16. Why statistical explanatory modeling predictivemodeling descriptive modeling are different
- 17. Explanatory Model: test/quantify causaleffect between constructs for “average” unit in population Descriptive Model: test/quantify distribution or correlation structure for measured “average” unit in population Predictive Model: predict values for new/future individual units Different Scientific Goals Different generalization
- 18. Theory vs. itsmanifestation ?
- 19. Notation Theoretical constructs: X,Y Causal theoretical model: Y=F(X) Measurable variables: X, Y Statistical model: E(y)=f(X) Breiman, “Stat Modeling: The Two Cultures”, Stat Science, 2001
- 20. Five aspects toconsider Theory – Causation – Retrospective – Bias – Average unit – Data Association Prospective Variance Individual unit
- 21. “The goal offinding models that are predictively accurate differs from the goal of finding models that are true.”
- 22. But there’s morethan bias-variance
- 23. Example: Regression Modelfor Explanation yi|xi = b0 + b1xi +b2 xcontrols + ei parameter of interest (inference) Chosen to avoid Omitted Var Bias (better to over-specify) Measures of X, Y constructs Underlying model: X Y Danger: endogeneity
- 24. yi|xi = b0+ b1 x1i +…+bp xpi + ei parameters of interest (inference) Chosen b/c related to Y Danger: multicollinearity All variables treated/interpreted as observable Remain in model only if statistically significant Residual analysis for GoF & test assumptions Example: Regression Model for Description
- 25. yi|xi = b0+ b1 x1i +…+bp xpi + ei Quantity of interest for new i’s (prediction) Chosen b/c possibly correlated with Y Danger: over-fitting All variables treated as observable, available at time of prediction Retain only if improve out- of-sample prediction Evaluate overfitting (train vs holdout) Example: Regression Model for Prediction
- 26.
- 27. Predict ≠ Explain +? “we tried to benefit from an extensive set of attributes describing each of the movies in the dataset. Those attributes certainly carry a significant signal and can explain some of the user behavior. However… they could not help at all for improving the [predictive] accuracy.” Bell et al., 2008
- 28. Predict ≠ Describe ElectionPolls “There is a subtle, but important, difference between reflecting current public sentiment and predicting the results of an election. Surveys have focused largely on the former… [as opposed to] survey based prediction models [that are] focused entirely on analysis and projection” Kenett, Pfefferman & Steinberg (2017) “Election Polls – A Survey, A Critique, and Proposals”, Annual Rev of Stat & its Applications
- 29. Goal Definition Design & Collection Data Preparation EDA Variables? Methods? Evaluation, Validation &Model Selection Model Use & Reporting
- 30. Observational or experiment? Primaryor secondary data? Instrument (reliability+validity vs. measurement accuracy) How much data? How to sample? Study design & data collection predict: increase group size explain/describe: increase #groups Multilevel (nested) data School Class Student
- 31. Data preprocessing Reduced-Feature Models Saar-Tsechansky& Provost, JMLR 2007
- 32. Data exploration, viz,reduction PCA Factor Analysis (interpretable) Dimension Reduction (fast, small)
- 33. Which variables? multicollinearity causation associations endogeneity ex-post availability identifiability A,B, A*B leading, coincident, lagging indicators
- 34. ensembles long/short regression omitted variablesbias shrinkage models variance bias Methods / Models blackbox / interpretable mapping to theory
- 35. Evaluation, Validation &Model Selection training datastatistical model holdout data Predictive power Over-fitting analysis theoretical model statistical model Data Validation Model fit ≠ Explanatory power
- 36. Point #2 Cannot inferone from the others explanatory power predictive power descriptive power
- 37. out-of-sample Performance Metrics type I,II errors goodness-of-fit p-values overall,specific over-fitting costs prediction accuracy interpretation training vs holdout R2
- 38.
- 39.
- 41. Currently in Academia (socialsciences, management) • Theory-based explanatory modeling • Prediction underappreciated • Distinction blurred • Unfamiliar with predictive modeling – getting better How/why use prediction (predictive models + evaluation) for scientific research beyond project-specific solution/utility/profit?
- 42. The predictive powerof an explanatory/descriptive model has important scientific value relevance, reality check, predictability
- 43. Generate new theory Developmeasures Compare theories Improve theory Assess relevance Evaluate predictability Prediction for Scientific Research Shmueli & Koppius, “Predictive Analytics in Information Systems Research” MIS Quarterly, 2011
- 44. Currently in Industry (andmachine learning) • Data-driven predictive modeling • Prediction over-appreciated • Distinction blurred • A-B testing • Unfamiliar with theory-based explanatory modeling Will the customer pay? What causes non-payment?
- 45. Implications: Short-term solutions Shallow/no understanding Ethical,social, human pitfalls Shmueli (2017) “Research Dilemmas With Behavioral Big Data”, Big Data, vol 5(2), pp. 98-119 How to do theory-based explanatory modeling with Behavioral Big Data?
- 46. Explain + Predict+ Describe
- 47. Can models (infact "designs") like RDD or RKD, which are "designed" to EXPLAIN causal effects, be used as PREDICTIVE models? RDD = Regression discontinuity design RKD = Regression kink design Prof. Hung Nguyen asked me (11/2017 email): 1. Can they generate predictions? 2. How can we use those predictions? 3. Can we modify them to predict better? Galit Shmueli 徐茉莉 Institute of Service Science