Inference of Means in a One Way ANOVA - Notes | STA 6166, Exams of Data Analysis & Statistical Methods

Download Inference of Means in a One Way ANOVA - Notes | STA 6166 and more Exams Data Analysis & Statistical Methods in PDF only on Docsity! Topic 22 - ANOVA (II) 22-1 Topic 22 – Inference of Means in a One-Way ANOVA When Variance is Constant for All Treatments Once we have rejected the null hypothesis that all means are equal, and we have checked the assumptions of the testing procedure, we usually wish to do some specific tests that can elucidate the relationships among the means. Note how this differs from regression. There, we assumed the relationship among the means was linear and so the only test of interest was of the slope. Here we made no such assumption mainly because X is categorical and so regression would make no sense. So, we want to do specific comparisons between the levels of X. These are variously called multiple comparisons tests, contrasts, or tests of linear combinations of means. A priori Hypotheses: hypotheses about population means that are decided during the planning of the experiment and prior to any data analysis. They are the reason for performing the experiment! A posteriori Hypotheses: hypotheses generated as a result of looking at the data after the experiment has been performed. Also called data snooping or data dredging. This is almost ALWAYS inappropriate and to be avoided. The only valid Topic 22 - ANOVA (II) 22-2 reason for doing so is as an exploratory analysis that will guide future experimentation. Example (a posteriori testing): suppose a 1-way ANOVA is performed and the results are obtained. The analyst looks over the results and decides to test 2 means because they appear to be very different (e.g. the smallest and largest ones). Now, the effect could be due to a real difference in population means or to random occurrence due to sampling that makes them appear different. Investigating only comparisons for which the effect appears large leads to a true confidence level for a conclusion that is lower than the stated confidence level. In other words you are more likely to reject H0: not different. It can be shown that the actual confidence is 60% (!!!!) when 6 levels are used in an experiment and the statistical analysis always includes testing the difference between the largest and smallest means using a stated 95% confidence. Note also that that treatments compared each time need not be the same ones since the largest and smallest means could be for different treatments. There are times when it is possible to do a posteriori testing – BUT the statistical method needs to be modified appropriately to account for the data snooping (see later). Topic 22 - ANOVA (II) 22-5 Again, hypothesis testing is done using the t-test for two independent samples that we reviewed earlier this semester. EXAMPLE: Rehabilitation Therapy. A researcher is interested in the relationship between physical fitness in persons prior to knee surgery and the time required in physical therapy after surgery to obtain successful rehabilitation. 24 male subjects with a similar type of knee surgery during the past year were randomly selected from the patient records at the rehabilitation center. The number of days required for successful rehabilitation and prior physical fitness status were recorded for each patient. The patients were categorized into one of three levels of prior fitness. The hypotheses of interest are: 1) the mean time to recovery will differ among the three groups; 2) the above average fitness group will have a shorter recovery period than either the below average or average group and 3) the average group will have a shorter recovery than the below average group. In other words: 1) H0: belowaverageabove   HA: at least one mean differs 2) H0: averageabove   HA: averageabove   Topic 22 - ANOVA (II) 22-6 H0: belowabove   HA: belowabove   3) H0: belowaverage   HA: belowaverage   The SAS code and output for analyzing the dataset are: data fitness; input prior_fit $ recovery; datalines; below 29 below 42 below 38 below 40 below 43 below 40 below 30 below 42 average 30 average 35 average 39 average 28 average 31 average 31 average 29 average 35 above 26 above 32 above 21 above 20 above 23 above 22 above 25 above 23 ; Topic 22 - ANOVA (II) 22-7 proc boxplot; plot recovery*prior_fit; quit; proc glm data=fitness; class prior_fit; model recovery = prior_fit; lsmeans prior_fit / pdiff; * the pdiff option does pair-wise tests of means; quit; The output from Proc Boxplot: bel ow aver age above 20 25 30 35 40 45 r e c o v e r y pr i or _f i t The ANOVA table is: Topic 22 - ANOVA (II) 22-10 Least Squares Means for effect prior_fit Pr > |t| for H0: LSMean(i)=LSMean(j) i/j 1 2 3 1 0.0012 <.0001 2 0.0012 0.0163 3 <.0001 0.0163 NOTE: To ensure overall protection level, only probabilities associated with pre-planned comparisons should be used. These are the p-values for testing the individual hypotheses H0: μi = μj against HA: μi ≠ μj. Before we check these tests we need to talk about performing multiple tests using the same data. 3) Multiple Comparisons The procedures we’ve just seen have 2 IMPORTANT limitations: a) The confidence (1 – ) applies ONLY to a particular estimate (or test) not to the series of estimates (or tests) b) The confidence (1 – ) is appropriate ONLY if the estimate (or test) was not suggested by the data Topic 22 - ANOVA (II) 22-11 We typically perform multiple tests in order to piece the results together to draw a more complete conclusion. This is sometimes referred to as a “family of statements (or tests)” and it is important to provide some assurance that all of the statements in the family are correct. We call this assurance the family-wise or experiment-wise confidence. The experiment-wise error rate is denoted αE. If we are interested in only specific hypotheses and they are not used in combination in order to draw an overall conclusion, then we refer to that as the individual or comparison-wise confidence. The individual error rate αI refers to that specific estimate or test ONLY. There is a problem with multiple t-tests involving parts of the same dataset if we ignore the distinction between family-wise and individual tests significance levels. Suppose there are ten means and each t-test for comparing two means is performed at the 0.05 level. There are 10(10-1)/2=45 pairs of means to compare, each with a 0.05 probability of a type I error (a false rejection of the null hypothesis). Topic 22 - ANOVA (II) 22-12 The chance of making at least one type I error in the set of 45 tests is much higher than 0.05. It is difficult to calculate the exact probability, but you can derive a pessimistic approximation by assuming that the comparisons are independent, giving an upper bound to the probability of making at least one type I error. 90055.0 )05.1(1 )1(1 ) tests45in error I Type oneleast at Pr( 45     m I E   EXAMPLE Rehabilitation Time. In the study of recovery time, we had 3 means and 3 pairwise tests (1-2, 1-3, 2-3). So our chance of at least one of those tests having a type I error was 142625.0)05.01(1)error I Type oneleast at Pr( 3  So, if we want to draw the conclusion that belowaverageabove   then our real type I error for this experiment-wsie conclusion is between 0.05 and 0.143. As can be seen, as the number of tests increases, the chance of making at least one Type I error approaches 1. Topic 22 - ANOVA (II) 22-15 error rate at 05.0E . Then we should run each individual test at the comparison-wise error rate that satisfies I305.0  , i.e. we should use 0167.03/05.0 I for each of the three comparisons. EXAMPLE: Suppose there is a study of 4 types of devices for determining pH in soil samples. Are there differences among the devices in their readings? 6 soil samples of known pH are used with each device. Recorded is the difference between the known pH and the device measurement. Oneway Analysis of pH By Device p H -0.5 -0.25 0 0.25 0.5 0.75 A B C D Device Each Pair Student's t 0.05 All Pairs Tukey-Kramer 0.05 Topic 22 - ANOVA (II) 22-16 Analysis of Variance Table Source DF Sum of Squares Mean Square F Ratio Prob > F Device 3 0.5837482 0.194583 4.8470 0.0108 Error 20 0.8028957 0.040145 C. Total 23 1.3866438 Means for Oneway ANOVA Level n Mean Std Error Lower 95% Upper 95% A 6 -0.16050 0.08180 -0.3311 0.01013 B 6 0.09467 0.08180 -0.0760 0.26529 C 6 0.12267 0.08180 -0.0480 0.29329 D 6 0.27350 0.08180 0.1029 0.44413 Std Error uses a pooled estimate of error variance To do the Bonferroni method: We do comparisons for each pair using Student's t test but we adjust the individual error rate in order to control the family-wise error rate at 05.0E . So, with m = 6 and 05.0E , 0083.0/  mEI  . Topic 22 - ANOVA (II) 22-17 Using SAS I obtained the following table: Differences of Least Squares Means Std Effect Dev Dev Estimate Error DF t Value Pr>|t| Device A B -0.2552 0.1157 20 -2.21 0.0393 Device A C -0.2832 0.1157 20 -2.45 0.0237 Device A D -0.4340 0.1157 20 -3.75 0.0013 Device B C -0.0280 0.1157 20 -0.24 0.8112 Device B D -0.1788 0.1157 20 -1.55 0.1378 Device C D -0.1508 0.1157 20 -1.30 0.2071 Use Bonferroni’s method to compare the calculated p-values to the Bonferroni αI = 0.0083. We can see that only one pair of devices have population means that are statistically significantly different, μA vs. μD. Protected Fisher’s Least Significant Difference (LSD) Method Basically, this is the Student’s t-test approach for comparing two means where we control ONLY the individual pair-wise error rate and there is no control of the experiment-wise error rate. It is known as the protected Fisher's LSD if one performs the testing of pairs of means only if the F-test rejects the null hypothesis. The statement that rejection of the overall F-test provides protection against inflated type I experiment-wise errors Topic 22 - ANOVA (II) 22-20 Add this line of SAS code to Proc GLM: lsmeans year / pdiff adjust = Tukey; The output is: The GLM Procedure Least Squares Means Adjustment for Multiple Comparisons: Tukey breadth LSMEAN year LSMEAN Number 150AD 138.111111 1 1850BC 134.444444 2 4000BC 132.666667 3 Least Squares Means for effect year Pr > |t| for H0: LSMean(i)=LSMean(j) Dependent Variable: breadth i/j 1 2 3 1 0.1664 0.0265 2 0.1664 0.6386 3 0.0265 0.6386 Note the difference in p-values from the last output shown earlier (page 10). We draw very different conclusions using Tukey’s than the protected Fisher’s LSD approach. Topic 22 - ANOVA (II) 22-21 SOME EXTRA STUFF IF YOU ARE INTERESTED 4) Estimation Of A Linear Combination Of Means Pair-wise comparisons are actually special cases of the more general comparison known as a linear combination of means:    t i iicL 1  where, when 0 1   t i ic , L is called a contrast. Now, the unbiased estimator of L is    t i ii ycL 1 ˆ with standard error    t i i i n c MSELSE 1 2 )ˆ( if the variances are homogeneous. Under our assumptions of normality and random sampling, a (1– )100% Confidence Interval of the linear combination of the population means is Topic 22 - ANOVA (II) 22-22 )ˆ(ˆ , LSEtL df where dft , is the critical value for a t-distribution on df degrees of freedom. If the variances are equal df = nT – t. Example: Plant heights experiment. (Set AH = 1, AL = 2, BH = 3, BL = 4, and D = 5). Suppose we have 5 levels (AH, AL, BH, BL, D) and wish to estimate the average height for safelight A and the average height for safelight B: mean for safelight A: 2 21   A mean for safelight B: 2 43   B We may want to test the following hypothesis: HA: 22 4321    Here 4321 4321 5.05.05.05.0 22       L