Understanding Null Hypothesis Significance Testing: Definition, Example, Misconceptions, Study notes of Statistics

Download Understanding Null Hypothesis Significance Testing: Definition, Example, Misconceptions and more Study notes Statistics in PDF only on Docsity! Page 6.1 (hyp-test.docx, 5/8/2016) 6: Introduction to Null Hypothesis Significance Testing Acronyms and symbols P P value p binomial parameter “probability of success” n sample size H0 the null hypothesis Ha the alternative hypothesis P value Statistical inference is the act of generalizing from sample (the data) to a larger phenomenon (the population) with calculated degree of certainty. The prior chapter introduced the most important form of inference: estimation. This chapter introduces the second form of inference: null hypothesis significance tests (NHST), or “hypothesis testing” for short. The main statistical end product of NHST is the P value, which is the most commonly encountered inferential statistic and most frequently misunderstood, misinterpreted, and misconstrued statistics in the biomedical and public health literature.1 Most teachers of statistics do not fully understand P values. Not even specialist scientists can easily explain them. Since the process of NHST revolves around the P value, let us start with its definition, which is easiest to remember with this notation: P value ≡ Pr(data or data more extreme | H0 true) where Pr ≡ probability | ≡ “given” or “conditional upon” H0 ≡ the null hypothesis Thus, the P value answers the question “If the null hypothesis were true, what is the probability of observing the current data or data that is more extreme?” Note that the P value is NOT the probability that the hypothesis (or any other hypothesis) is right or wrong. In fact, it assumes the null hypothesis is right! In light of these facts, there are actually two classical schools of thought on how best to use the P value: the Fisher and Neymann-Pearson schools.2 There is also a Bayesian way to interpret the P value, but that presents a whole other set of dilemmas. As a starting point, we will consider the P value as a calculated index which, as it gets smaller-and- smaller, provides stronger-and-stronger evidence against the null hypothesis. 1 There is a large body of literature about the misinterpretation of P values. My favorite is: Cohen J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1002. 2 For an introduction to the distinct between these interpretations, see this video. You can get to the video by Googling “Gerstman p value Youtube.” Page 6.2 (hyp-test.docx, 5/8/2016) Example of a NHST The first step of NHST is to convert the research question into null and alterative hypotheses. Thus, the research question must be concisely articulated before starting this process. • The null hypothesis (H0) is a statement of “no difference,” “no association,” or “no treatment effect.” • The alternative hypothesis, Ha is a statement of “difference,” “association,” or “treatment effect.” H0 is assumed to be true until proven otherwise. However, Ha is the hypothesis the researcher hopes to bolster. Take as an example a treatment that is said to be 25% effective. A researcher claims she has a new treatment with improved efficacy. It is essential that we articulate the research question into “plain English.” In null (no difference) form, the new treatment is NO more effective than the existing treatment (25% effective). In alternative (“difference”) form, the new treatment is more effective than the existing treatment. We can see that this question is about a proportion (25% vs. not 25%). Thus, the parameter to be inferred is similar to binomial proportion p. Under the null hypothesis, H0: p = .25. This is the most important part of setting up the NHST. The researcher tests the new treatment in 3 patients. This experiment lends itself to the binomial distribution since it is based on series of trials which can each outcome can be characterized as a success or failure. Since the null hypothesis of “no difference" is assumed to be true until proven otherwise, the number of successes in the experiment should follow a binomial pmf with n = 3 and p = 0.25. This exact pmf was introduced in Chapter 4 and is also shown here: X Number of successes Pr(X = x) Probability Pr(X ≤ x) Cumulative Probability 0 0.4219 0.4219 1 0.4219 0.8438 2 0.1406 0.9844 3 0.0156 1.0000 We consider two possible outcomes of the experiment. If all 3 patients in this experiment responded to the new treatment, P value = (data or data more extreme| H0 true) = Pr(X=3) = 0.0156. This observation would be rare if the true success rate was only 25%. Thus, the evidence against H0 would be deemed to be significant. If 2 of the 3 patients in the experiment responded, P value = (data or data more extreme| H0 true) = Pr(X = 2) or Pr(X=3) = 0.1406 + 0.0156 = 0.1562. In this “2 out of 3” case, the P value is 0.1562 indicates that this would not be unusual if the probability of success was actually 0.25. Thus, the evidence against H0 is deemed non-significant. In neither case can we say that the evidence of efficacy is conclusive. However, the “3 out of 3” (P = .0156) provides some evidence of a “real difference” and is worthwhile of follow-up, where the “2 out of 3” (P = .1562) evidence is weak.