Statistical Modeling: Understanding Data Analysis and Model Selection - Prof. Brian R. Mit, Study notes of Earth Sciences

Download Statistical Modeling: Understanding Data Analysis and Model Selection - Prof. Brian R. Mit and more Study notes Earth Sciences in PDF only on Docsity! Natural Resources Data Analysis – Lecture Notes Brian R. Mitchell I. Week 1: A. Welcome and Introductions B. Review Syllabus 1. Review some specifics: 2. Written assignments are aimed towards developing a professional manuscript (thesis chapter or journal publication) describing your results. 3. Check the web site weekly for changes (and new readings if any are listed as “TBA”). I will post any changes by the end of the day on Monday for the following week’s class. 4. Let me know if there are topics you wanted to see that aren’t on the schedule; I will try to add topics in if there is interest. 5. First few weeks will focus on developing model sets, then quickly proceed into nuts and bolts of data analysis. C. Data sets 1. Each participant should give a brief summary of their research question and data set, and describe the statistical approach they want to use and why. D. Approaches to data analysis 1. I think it is worth taking some time to discuss the various approaches to testing statistical hypotheses. Let’s start at a really basic level and work up. 2. What is a hypothesis (in the most general sense)? 3. A hypothesis is a tentative explanation for an observation, phenomenon, or problem. 4. What is a scientific hypothesis? 5. A scientific hypothesis is a word model that tries to explain or make a prediction based on our current understanding of a problem (a scientific model). 6. What is a statistical hypothesis? 7. It is a statement about the attributes of a statistical model whose validity can be assessed by seeing how well the model matches data. Note that a statistical model is explicit, quantitative, and includes a description of uncertainty (error). 8. There should be a one-to-one correspondence between a scientific model and a statistical model. If there is a one-to-one correspondence, then learning about the adequacy of a statistical hypothesis can teach us about the adequacy of a scientific model. 9. Example: scientific model: in a sexually reproducing population with random mating and equal parental investment, natural selection should favor equal proportions of male and female offspring. Scientific hypothesis: there are equal proportions of male and female offspring. Statistical model: the number of males (or females) in a litter of a given size is binomially distributed. Statistical hypothesis: θ, the probability of a male offspring, = 0.5. 10. The next step is to collect data using an appropriately designed and thought-out strategy. 11. Then we need to evaluate whether the data are consistent with our statistical hypothesis. 12. What are the potential approaches we could use to evaluate a statistical hypothesis? 13. The general approaches are Frequentist, Bayesian, and Likelihood. Let’s explore each in turn. 14. Frequentist a) There are a couple of related approaches to classical hypothesis testing; they are all considered “frequentist” approaches. Why? (Hint: has a lot to do with the convoluted way you were taught to talk about p-values in multivariate stats) b) These approaches always consider the frequency with which your data or more extreme data would be collected, IF you conducted many replicate experiments. c) Fisherian hypothesis testing (1956) (1) Construct a statistical null hypothesis (e.g. θ = 0.5). (2) Choose an appropriate distribution (e.g. binomial distribution) or test statistic (e.g. t statistic). (3) Collect the data with random samples. (4) Determine the p value (probability of obtaining the value or one more extreme), assuming the null hypothesis is true. (5) Reject the null hypothesis if p is small (and always report the p value as a “strength of evidence” measure). (6) Fisher recommended a significance level of 0.05, but later argued that the significance level should depend on the circumstances. d) Neyman-Pearson hypothesis testing (1933) (1) Similar to Fisher’s approach, but: (2) Set significance level in advance, and interpret it as the proportion of times the null would be improperly rejected given many replicates and a true null hypothesis. (3) Explicitly incorporate an alternative hypothesis; this must be true if the null is false. (4) p value is not a strength of evidence; its only use is in deciding to accept or reject the null hypothesis. (5) Focus on Type I and Type II errors, as well as power of tests. e) A hybrid approach is common today (1) This is essentially the Neyman-Pearson approach, but with the view that p values are a strength of evidence (e.g. significant, very significant (0.01), and highly significant (0.001). f) Critique of the frequentist approach (1) What are the problems with the frequentist approach? (3) The likelihood function of a parameter or variable (e.g. the proportion of heads in a coin toss) can be thought of as a graph of the relative chance of observing a given value (on the y axis) against all possible values of the parameter (on the x axis). (4) The Law of Likelihood say that if hypothesis A implies that the probability of a random variable X taking value x = ρa(x), while hypothesis B implies that the probability of a random variable X taking value x = ρb(x), then the observation that X = x is evidence supporting A over B if ρa(x) > ρb(x), and the likelihood ratio ρa(x)/ρb(x) measures the strength of that evidence. (5) As a rule of thumb, likelihood ratios below 8 are considered weak evidence, between 8 and 32 is moderate evidence, and above 32 is strong evidence (Royall 2004). b) Example (1) We are given a coin that we suspect is biased towards excess heads. (2) We toss the coin n = 20 times, and get x = 12 heads. (3) Hypothesis A is that the coin is unbiased (π = 0.5), and Hypothesis B is that heads will occur 60% (π = 0.6) of the time. We choose this value for hypothesis B knowing it will yield the maximum likelihood ratio based on the data collected. (4) We use the binomial distribution: xnx xnx nxP −− − = )1( )!(! !)( ππ (5) ρa(x) = 0.1201, and ρb(x) = 0.1797, so the likelihood ratio is 1.50 (6) How strong is the evidence that this coin is biased? (7) What if n = 150? (8) ρa(x) = 0.00324, and ρb(x) = 0.06637, so the likelihood ratio is 20.50 (9) How strong is the evidence that this coin is biased now? c) Misleading evidence (1) Note that regardless of how strong the evidence is, it can still be misleading. In other words, there is still a chance that the coin in the example was not really biased, even though there is strong evidence that it is. The interpretation (that the coin was biased) is still correct; it was the evidence that was misleading. (2) The maximum probability of misleading evidence cannot exceed 0.021 when the likelihood ratio exceeds 8, and cannot exceed 0.004 when the likelihood ratio exceeds 32 (Royall 2004, citing Royall 1997). d) Likelihood-based hypothesis testing (1) Experimental design should consider the minimum sample size at which the probability of generating weak evidence for distinguishing between the hypotheses is low. Note that if the probability of generating weak evidence is low, the probability of misleading evidence will be even lower. (2) Observed data is assumed to fit some underlying probability model (as in frequentist methods) (3) The likelihood ratio provides an explicit and objective measure of the strength of the statistical evidence. (4) There is no dependence on a particular stopping rule; there is no reason not to collect additional data if the likelihood ratio indicates weak data; researchers are encouraged to examine the likelihood functions of their data and adjust the sample size accordingly. This is of course absolutely forbidden in the frequentist approach. e) Problems with likelihood analysis (1) Arbitrary levels of importance. (2) Does not incorporate prior information. (3) Any other thoughts? 17. Relationships between approaches a) Likelihood has some aspects of the frequentist approach, because likelihood ratios follow a χ2 distribution (actually 2*ln(LR) is the correct statistic) and can be tested using frequentist methods b) Likelihood also has aspects of the Bayesian approach. With a uniform prior, the Bayesian posterior probability distribution has an identical shape to the likelihood function. 18. What approach should be used, and when? a) When should the classical approach be used? (1) Strict experiments with control and treatment b) When should a Bayesian approach be used? (1) Any situation where you want to incorporate prior knowledge (2) These situations can include model selection and determination of effect sizes (3) Particularly well suited to learning algorithms (e.g. neural networks) c) When should a likelihood approach be used? (1) Natural experiments (2) Observational experiments (3) Determination of effect sizes (4) Model selection E. Approaches to model selection and averaging 1. In this class we are concerned with model selection. The model sets can range from alternative hypotheses about historical events, to hypotheses about processes that generated a data set, to hypotheses about which parameters are most important in a data set. 2. We might be interested in predicting future data, understanding the existing data, or estimating effect sizes. 3. Frequentist methods a) How would a frequentist go about selecting the best model? b) In general, the frequentist approach is not a good strategy for model selection, primarily because: 1) hypothesis tests between models are not independent, and 2) there are serious problems with the probability of Type I error due to multiple testing of the same data. c) Stepwise (1) Model set would be a nested set ranging from an intercept-only model to the most general model (i.e. including all parameters plus important interactions). (2) Stepwise parameter selection is used to compare two models at a time. Can be forwards stepwise (begin with no parameters) or backwards stepwise (begin with full model). At each step, evaluate whether adding the most important excluded parameter increases model fit, and evaluate whether removing the least important included parameter decreases model fit. Continue until you get to the point where adding any one of the remaining excluded parameters does not improve the fit, and removing any one of the included parameters decreases the fit. (3) Selection is generally based on the likelihood ratio F-test. d) All subsets (1) Same model set as above (2) Calculate some statistic (e.g. adjusted R2, AIC, BIC) for every model, and pick the model with the best value. (3) NOTE: using AIC and BIC in this situation is essentially a likelihood approach to model selection; it is valid but considered weak (exploratory). Adjusted R2 functions poorly as a model selection criteria. 4. Bayesian approach a) How would a Bayesian go about selecting the best model? b) Develop a model set; would probably not include all possible models. c) Explicitly consider the prior information about which model(s) are most likely. d) Calculate the Bayes factor for each model (or its approximations, BIC or AIC, depending on your model priors), and select the best model or use model averaging. 5. Likelihood approach a) How would you go about model selection using a likelihood approach? b) Develop a model set; would probably not include all possible models. c) For each model, calculate the likelihood of the data, given the model. d) Calculate AIC or some other information criterion, and select the best model (if evidence overwhelmingly supports it) or use model averaging (if multiple models are supported by the evidence). e) This is the Information-Theoretic approach to model selection and averaging, and is based on the concept of Kullback-Leibler distance F. Discuss specifics from the readings 1. Let’s take the rest of the class period to discuss the readings for this week, which provided background information for the information-theoretic approach. We’ll start with Johnson and Omland before going into B&A, which is more technical. 2. Johnson and Omland: a) In my view, the main points are: