Download Understanding Hypothesis Testing & Statistical Significance in Psych Stats - Prof. Michael and more Study notes Psychology in PDF only on Docsity! 1 Hypothesis Testing 2 Advanced Psychological Statistics I Psychology 502 October 18, 2007 2 Overview ! Issues in hypothesis testing " Directional testing " Failing to reject " Statistical significance " Which hypothesis? ! Q&A on the midterm 3 Issues ! Not everything in the world of inferential statistics is cut- and-dried " This is not calculus or trig ! Many issues in the use and interpretation of inferential statistics that cannot be settled by mathematics " No formulae to grind ! No right or wrong answers " But there are poorly-considered ones ! You have to actually think about them " Thinking is widely regarded as error-prone and therefore dangerous " I!m willing to take that chance 4 Directional Hypotheses ! Two types of hypotheses/tests: " Directional " Non-directional ! Which one should we use? Why? " Directional test has better power " But only when true difference is in hypothesized direction! ! Statistics packages generally report two-tailed tests ! Discussion: " Howell seems to fall on the side of the two-tailed " Do you agree? Why or why not? 9 Statistical Significance ! When a value for the test statistic is found in the region of rejection, the test is said to be “statistically significant” at the alpha level " For example, “the difference is significant at the 0.05 level” " The adjective “statistical” is often dropped ! Dictionary definitions of “significant”: " Important in effect or meaning " Deserving to be considered; momentous ! Do those have anything to do with statistical significance? " No, not necessarily 10 “Significance” Example ! Someone claims the income of a particular group is, on average, $10 million " Null hypothesis: µ = 10,000,000 ! Alpha level 0.05, two tailed (reject if |z| > 1.96) ! The test " Collect a sample of 20,000 " Discovered mean is $10,000,100 " Standard deviation is $7,000 # S.E.M. = 7000/sqrt(20,000) = 49.49 " z = 100/49.49 = 2.02 " 2.02 > 1.96, reject null ! Is this difference “significant”? 11 Choice of Terminology ! “Statistical significance” is a technical term " Should not be confused with “importance” or “validity” " Quote from Hayes: “Highly analyzed and significant results are often confused with good results.” ! What does a p-value really represent? " Probability that result obtained would have occurred by chance if null is true " It!s a statement about reliability " In significance testing, a p of 0.0001 is treated the same as a p of 0.049 # Is this reasonable? ! Suggestion: Use term “statistically reliable” or “reliable” rather than “significant” 12 “Marginal” Significance ! Sometimes you!ll see a paper refer to a p-value between .05 and .10 as “marginal significance” ! What is that supposed to mean? ! This is technically incorrect " According to some, being “marginally significant” is like being “marginally pregnant” " Do you ever read any references to p-values only slightly better than .05 as “marginally insignificant”? ! Cannot have it both ways " If you!re hypothesis testing, it!s either significant or it isn!t " If you!re not, then you shouldn!t be using the term “significant” anyway 13 Testing Hypotheses ! Definition of “hypothesis”: " My dictionary: A tentative explanation that accounts for a set of facts and can be tested by further investigation " Judd, et al.: Relations between constructs, and relations between constructs and observable indicators " Hertzog: An emprically-testable assertion of truth ! Scientific method " The systematic empirical investigation of hypotheses " Test the empirically observable consequences of the hypotheses 14 Levels of Hypotheses
