Within-Subjects vs. Between-Subjects Designs: Statistical Efficiency and Context Effects, Lecture notes of Design

Download Within-Subjects vs. Between-Subjects Designs: Statistical Efficiency and Context Effects and more Lecture notes Design in PDF only on Docsity! Plychobgical Bulletin 1976. Vol. 83, No. 2, 314-320 Within-Subjects Designs: To Use or Not To Use? Anthony G. Greenwald Ohw State Unive~sity This article considers the several factors pertinent to deciding whether a within- or between-subjects design should be employed for a research appli- cation. A general principle favoring within-subjects designs is the statistical efficiency afforded by removing subject variance from error terms used to test treatment effects. Within-subjects designs, however, are often faulted for being subject to context effects of practice, sensitization, and carry-over that may limit interpretation of results. At the same time, between-subjects designs are not devoid of context effects, but rather have #the context that a single treat- ment affords itself. Since ecological validity of results depends on the corre- spondence of the research context to the generalization context, within-subjects designs may he preferred when the generalization context includes the equiv- alent of several concurrent treatments. The discussion focuses additionally on (a) procedures 'to minimize practice, sensitization and carry-over effects in within-subjects designs when they a;e these effects to advantage in research. not desired, and (b) means of using Frequently an investigator faces the choice of whether to examine the effects of two or more experimental treatments by exposing each subject to (a) only a single treatment (between-subjects design) or (b) several or all of the treatments (within-subjects or re- peated-measures design). Grice ( 19.66) has pointed out that the pattern of treatment ef- fects obtained may vary considerably between the two types of designs. However, only rarely does an investigator make a choice of type of design after consideration of the appropri- ateness of each type to the problem being investigated. I attempt to assemble here sev- eral considerations that may often be ap- propriate to the decision between a within- or between-subjects design. Nthough they are mentioned briefly, sta- tistical considerations relating to choice of design are not of primary interest here. These statistical matters are well handled in stan- dard statistical texts, as referenced below. My aim, rather, is to detail the psychological considerations that are critical to the choice Preparation of this report was facilitated by sup- port to the author from National Science Founda- tion Grant GS-42981 and National Institute of Men- tal Health Grant MH-20527. Requests for reprints should be sent to Anthony G. Greenwald, Department of Psychology, Ohio State University, 404C West 17th Avenue, Columbus, Ohio 43210. of design. Some of these points are also cov- ered in statistical texts, particularly insofar as they may affect the choice of statistical procedures. I have added only a few novel points to these earlier treatments and have aimed more a t (a) putting the several points together in a single place and (b) observing that the prevailing cautions against the use of within-subjects designs need to be mod- erated without, however, being abandoned. Poulton (1973, 1974; see also Rothstein, 19 74) has recently issued a general warning against within-subjects designs, pointing out that the context provided by exposure to other treatments ("range effect") may often alter the effect of a given treatment. This point is certainly valid and is acknowledged here by considering (a ) how procedures may serve to minimize or maximize such context effects and (b) when it may or may not be appropriate to allow the occurrence of con- text effects. The context effects that mav be generated by a within-subjects design are discussed under three headings: Practice, Sensitization, and Carry-Over. Context Effects in the Between-Subjects Design Poulton (1973) concluded that since con- text or range effects are to be expected in within-subjects designs, these designs should ordinarily be avoided or, if used, bolstered by 315 ANTHONY G. GREENWALD between-subjects design results. Implicit in this conclusion is the principle that the be- tween-subjects design provides a standard of validity against which results of a within- subjects design must be evaluated. This may be questioned on three grounds. First, as Poulton (1973) noted, "The influence of range of stimuli cannot always be prevented by restricting each man to a single stimulus" (p. 115). This may be because extralabora- tory experience leaves some residue of con- text. Second, even if the extralaboratory con- text can be safely ignored, the presentation of a single treatment to each subject does not really achieve the absence of context, but rather the presence of the context provided by the single treatment. An example makes this clearer. Example 1: Researcher 1 uses two designs to study the effect of foreperiod duration on simple reaction time. In a between-subjects design, each subject is assigned to a single foreperiod treatment: 0, 200, 500, or 1,000 msec. In a parallel within-subjects design, each subject receives a series of trials in which the four treatments are randomly sequenced. I t is known that Researcher 1's results will be different for the two types of design. The within-subjects design may produce either an increasing or a decreasing function relating reaction time to foreperiod duration (see Poulton, 1973, Table 1 ) . Which function will be obtained depends on whether the pro- cedures are arranged to produce increasing or decreasing expectation of the response signal as the foreperiod grows. Thus, it may be said that the within-subjects design introduces an expectancy or readiness process that is af- fected by the context of other treatments ( foreperiods) . I s this expectancy process absent from the between-subjects design? No-rather, readi- ness occurs and is focused at the end of the (single) expected foreperiod. Thus, the single treatment in the between-subjects design pro- vides a very real context that influences per- formance. This context effect could be avoided by presenting each subject with only a single trial a t the selected foreperiod duration, but this would be an impractical way of collecting data on the problem. Further, the researcher may well wish to ignore the first (or first sev- eral) trials, since these involve warm-up pro- cesses (effects due to lack of context!) that are not of interest. These considerations raise the third basis for questioning the notion that between-sub- jects designs provide a standard of validity against which to evaluate within-subjects de- signs. In fact, the ecological or external valid- ity (Campbell & Stanley, 1966) of a piece of research depends on the extent to which the research context approximates the context existing in the domain to which the researcher wishes to generalize the results. This point is considered further in the concluding section of this article. Statistical Considerations No attempt is made here to detail the tech- nical problems involved in statistics used to analyze the within- or between-subjects de- signs. However, a few ge6eral principles of a statistical nature must be considered as background. A more complete discussion of these points may be found in standard sources such as Myers (1972, especially chapter 7) and Winer (1971, especially chapter 4). Power. When each subject provides data for two or more treatments, the subject may be said to serve "as his own control" in com- parisons among treatment effects (i.e., treat- ment differences are not confounded with sub- ject differences). To the extent that the sub- jects classification in the ensuing analysis of variance constitutes a substantial source of variance, this feature of the within-subjects design results in substantially more sensitiv- ity to treatment effects (power) than would characterize a between-subjects design em- ploying the same number of observations. Since a k-treatment between-subjects design would employ k times the number of subjects used in a within-subjects design with the same number of observations, it is apparent that a within-subjects design might often reach a desired level of power while using fewer than 1/k times the number of subjects in an equally powerful between-subjects de- sign. The within-subjects design can there- fore represent an immense experimental econ- omy, particularly when per-subject costs are WITHIN-SUB JECTS DESIGNS 318 Example 4: Researcher 4 is interested in the effects of Drugs A, B, and C on performance on a simple reaction time task. In order to employ a within-subjects design, Researcher 4 gives each subject four performance tests separated by 20 minutes, each test being pre- ceded by the administration of a standard dosage injection of one of the three drugs or a placebo control and with the sequence of treatments being counterbalanced across sub- jects. In general, a carry-over effect occurs when the effect of one treatment persists in some fashion a t the time of measurement of the effect of another. In Example 4, there are two types of potential carry-over. One is due to practice a t the performance task and has been discussed separately above. The second is that traces of prior drug treatments may be present a t the time of testing the effects of a later treatment. Counterbalancing provides an only partially adequate solution to this prob- lem, since the interference effects may not be bidirectional and, further, they may obscure the treatment effects of the drugs taken in- dividually .= The chief means of reducing carry-over ef- fects is to separate the treatments in time. This would likely be an effective means of applying a within-subjects design to the prob- lem given in Example 4, assuming that prac- tice effects are not also involved. In general, the strategy of separating treatments in time will be effective in reducing intertreatment carry-over only to the extent that the effects of any treatment are not permanent. In addition to the study of learning, there are several other major areas of study in which the target of study is some process that can be interpreted as an intertreatment carry- over in the framework of a within-subjects design. Perceptual assimilation and contrast, incentive contrast, violation of expectation, transfer of training, primacy-recency in per- suasion, resistance to extinction, and various types of adaptation are some of these. The fact that intertreatment carry-overs are likely to be a major source of serendipitous findings should not be overlooked as one of the virtues of employing within-subjects designs in which treatments that would otherwise not be ex- amined in near temporal proximity are juxta- Procedures that permit the occurrence of carry-over effects present special problems for statistical analysis. Cochran and Cox (1957, pp. 133-142) discussed a variety of means of estimating separately the direct and carry- over effects of experimental treatments. Summary. When treatments have persistent effects, a within-subjects design may be un- satisfactorv because the effect of one treat- ment may still be in force a t the time of measuring another's effect. However, the within-subjects design may be salvaged in this case by increasing the separation of the treat- ments in time. Effects dependent on carry- over or, more generally, upon the sequence in which treatments are administered and their temporal proximity are frequently of psy- chological interest in and of themselves. Several of the concerns already treated are appropriate to evaluating the internal valid- ity of an experiment-that is, Does the with- in-subjects design permit the experimenter to test the hypothesis of interest, or will con- sequences of using the design in some way contaminate (by practice, sensitization, or carry-over) the hypothesis test? Now we take up a matter that may be at odds with some of these considerations and ask how the choice of design affects the external (or eco- logical) validity of the experiment (i.e., the ability of the researcher to account for the effects of treatment variations as they may occur in interesting nonresearch settings). (See Campbell and Stanley, 1966, for an ex- position of internal and external validity.) Example 5: Researcher 5 is interested in the effects of source credibility on persuasion, and is considering two possible designs. In one, a between-subjects design, communications on 1 This inadequacy of counterbalancing involves the same considerations mentioned in discussing the pos- sible inadequacy of counterbalancing in removing practice effects. Practice is certainly an instance of the general class of carry-over effects, but has been discussed separately because of the special status of learning effects in psychological research. 319 ANTHONY G. GREENWALD two topics are attributed, for some subjects, to a trustworthy and expert source whereas, for other subjects, the same communications are attributed to an untrustworthy and inex- pert source. I n an alternate within-subjects design, each subject is exposed to the same two communications, but one is attributed to the high-credible source, the other to the low- credible source, with source-communication assignments being counterbalanced across sub- jects. Which design is preferable? Persons familiar with persuasion research will be aware that the between-subjects de- sign is most often chosen for the examination of source credibility effects (but not always -see Osgood & Tannenbaum, 1955). But this is perhaps the less justifiable choice if the researcher's primary interest is in predicting or characterizing source effects in the non- laboratory environment. Consider that people tend to be exposed to persuasive communica- tions in clusters in many mass communica- tion settings, these communications frequently being identified with different sources (e.g., columns in a newspaper editorial section, poli: tical or product advertisements in magazines, or on radio or television). Therefore, the within-subjects design for studying the con- sequences of communicator credibility may have greater external validity than does the between-subjects d e ~ i g n . ~ Similar considerations would lead to a pref- erence for the between-subjects design for other problems. For example, a researcher may be interested in studying the effect of reinforcement-based versus psychoanalytically based therapies for phobia symptoms. In such a situation exposure of the same subjects to several different treatments would create a situation rather lacking in external validity. Considerations of external validity should not necessarily be uppermost in the research- er's mind. The between-subjects design may be preferred even in some situations for which the within-subjects design would have greater external validity, because the between-sub- jects design may allow cleaner tests of the- oretical hypotheses. Example 6 presents such a case, in which internal validity is of more concern to the investigator than is external validity. Example 6: Researcher .6 is interested in the effects of witnessing televised violence on sub- sequent aggressive behavior of children. A within-subjects design would involve exposure of each subject to several different program sequences of varying degrees of violence, each followed by the provision of some opportunity to act aggressively in a play situation with other children. Should ,this design be em- ployed? In this case, the within-subjects design might not be preferable because the carry- overs among treatments (subjects still being under the influence of Program A a t the time of Test B) might weaken the researcher's hypothesis test. Accordingly, the between-sub- jects design might be chosen even though the within-subjects design clearly has greater ex- ternal validity in its correspondence to the mixture of types of programs the child would normally see on television. In many cases, a greater stress on internal validity than on external validity would lead to a choice of the within-subjects design. This might be particularly true in cases of basic research for which there is no readily ap- parent nonlaboratory setting for which the research is an analog. For example, a neuro- psychologist studying functions of single cells in the central nervous system should almost certainly examine the consequences of the range of treatments in which he or she is in- terested on each of the research subjects. Summary. Considerations of external or ecological validity may sometimes be a t odds with considerations related to practice, sensi- tization, and carry-over effects. Thus, the 2 Both of the designs mentioned in this example are within-subjects or repeated-measurement designs in that the effects of two communications are studied on each subject. However, the treatment variation of credibility is a between-subjects variation in the first design and a within-subjects variation in the second. 3 Poulton's (1973) concerns about range effects are quite relevant here. The investigator who is in- terested in generalizing to nonlaboratory settings should be concerned to see that the range and dis- tribution of treatment variations in the experiment correspond to their range and distribution in the ap- propriate nonlahoratory setting. Otherwise, the ex- perimental treatment effects may misrepresent the ef- fects of their nonlaboratory analogs. WITHIN-SUB JECTS DESIGNS 320 within-subjects design may often have greater jects design may have the desirable conse- external validity because i t contains these quences of permitting the study of some in- confounds, but these may also interfere with teresting aspect of the context effect itself or the researcher's ability to isolate the treat- of increasing the external (ecological) valid- ment effects. ity of the research. A general force operating in the direction of selecting a within-subjects design is the statistical efficiency afforded by the removal of subject variance from error terms used to test treatment effects. However, context ef- fects may often interfere with hypothesis tests and, therefore, should take precedence over considerations of statistical efficiency when choosing a design. Context effects may occur in either a between- or a within-subjects de- sign, but the range of possible effects is much greater in the latter type of design and, cor- respondingly, the experimenter has greater potential control over them by selecting ranges of treatments to administer. In many situations a within-subjects design can be made more acceptable by appropriate counter- balancing of treatment sequences (to control practice effects), by camouflaging treatments (to reduce sensitization to the treatment di- mensions), or by separating treatments in time (to reduce carry-over effects). In still other circumstances, the deliberate introduc- tion of these context effects in a within-sub- REFERENCES Campbell, D. T., & Stanley, J. C. Experimental and quasi-experimental designs for research. Chicago: Rand McNally, 1966. Cochran, W. G., & Cox, G. Experimental designs. New York: Wiley, 1957. Grice, G. R. Dependence of empirical laws upon the source of experimental variation. Psychological Bulletin, 1966, 66, 488-498. Myers, J. L. Fundamentals o f experimental design (2nd ed.). Boston: Allyn & Bacon, 1972. Osgood, C. E., & Tannenbaum, P. H. The principle of congruity in the prediction of attitude change. Psychological Review, 1955,62, 42-55. Poor, D. D. S. Analysis of variance for repeated measures designs: Two approaches. Psychological Bulletin, 1973, 80, 204-209. Poulton, E. C. Unwanted range effects from using within-subjects experimental designs. Pyschological Bulletin, 1973, 80, 113-121. Poulton, E. C. Range effects are characteristic of a person serving in a within-subjects experimental design-A reply to Rothstein. Psychological Bul- letin, 1974, 81, 201-203. Rothstein, L. D. Reply to Poulton. Psyckological Bulletin, 1974, 81, 199-201. Winer, B. J. Statistical principles in experimental de- sign (2nd ed.). New York: McGraw-Hill, 1971. (Received December 6, 1974)