Computing in Higher Education - Design Research: A socially responsible approach | EDIT 9990, Lecture notes of Information Technology

Download Computing in Higher Education - Design Research: A socially responsible approach | EDIT 9990 and more Lecture notes Information Technology in PDF only on Docsity! DESIGN RESEARCH 96 Journal of Computing in Higher Education Spring 2005, Vol. 16(2), 97-116. Design Research: A Socially Responsible Approach to Instructional Technology Research in Higher Education Thomas C. Reeves The University of Georgia Jan Herrington University of Wollongong Ron Oliver Edith Cowan University ABSTRACT DESIGN RESEARCH has grown in importance since it wasfirst conceptualized in the early 90s, but it has not beenadopted for research in instructional technology in higher education to any great extent. Many researchers continue to conduct studies that principally seek to determine the effectiveness of the de- livery medium, rather than the instructional strategies and tasks. This article explores the various incentives for conducting research on the impact of computing and other technologies in higher education, examines the social relevance of that research, and recommends design research as a particularly appropriate approach to socially responsible inquiry. A description of the characteristics of design research is given, together with an argument for the more widespread adoption of this approach to enhance the quality and usefulness of research in com- puters and other technologies in education. (Keywords: design research, design experiments, development research, research methods, instruc- tional technology research, socially responsible research) Reeves, Herrington, and Oliver 97 INTRODUCTION ADMITTEDLY, the New Yorker magazine is an unexpectedsource of information about research focused on applicationsof instructional technology in higher education. However, the February 9, 2004, issue of this normally urbane magazine included a short piece titled “Chew On” by Ben McGrath. The story describes an educational computing research study undertaken by Dr. Kenneth L. Allen, an Assistant Professor of Dentistry at New York University. The purposes of the research, as also reported by Dr. Allen and his colleagues at the annual conference of the International Association of Dental Research (Allen, Galvis, & Katz, 2004), were to “(1) to compare two methods of teaching dental anatomy: ‘CD + lab’ versus ‘standard lecture + lab’; and, (2) to determine whether actively chew- ing gum during lecture, lab and studying would have an effect on learning.” According to the New Yorker article, Allen and his colleagues originally intended only to compare the effectiveness of a commercial anatomy CD-ROM and a standard anatomy lecture, but lacking fund- ing, incorporated chewing gum into the study at the behest of the Wrigley’s company which was interested in the effects of chewing its products on learning. No one familiar with the frustrating history of instructional technology’s impact on learning (Clark, 2001; Cuban, 2001) will be surprised that there were no statistically significant differences found between the test scores of the students using the dental anatomy CD-ROM versus those who attended a dental anatomy lecture. Although the chewing gum results also failed to reach sta- tistical significance, the authors concluded that the finding that “the chewing gum group (n=29) had an average of 83.6 [on a 25 question objective exam] vs. 78.8 for the no chewing gum group (n=27)” appeared to be “educationally significant.” The New Yorker writer poked fun at Dr. Allen, suggesting that he might want to extend his research to investigations of the impact on learning of chewing tobacco or biting fingernails, but there is little doubt that Dr. Allen, like numerous other faculty members from DESIGN RESEARCH 100 THE MOTIVES FOR CONDUCTING RESEARCH in highereducation are related to the concept of social responsibility.The social relevance of research is an issue that is obviously subject to much debate. One’s age, race, gender, socioeconomic sta- tus, education, religion, and political allegiance influence one’s inter- pretation of the social relevance of any given research study. For example, the 2004 presidential election in the USA was at least partially contested on the basis of differing perspectives on the value and morality of stem cell research. The following principles represent a core set of values to guide scientific research of any kind (derived from Casti, 1989): • Science is an ideology that consists of a cognitive structure concerning the nature of reality together with processes of inquiry, verification, and peer review. • Views of reality differ according to one’s philosophy of science, for example, realism maintains that an objective reality actually exists, instrumentalism asserts that reality is the readings noted on measuring instruments, and relativism claims that reality is what the community says it is. • Scientific research is a social activity that has certain stan- dards and norms, for example, it should not intentionally harm humans and it must be able to be replicated by other researchers. Socially responsible research in education in general, and instruc- tional technology in particular, must adhere to the basic principles listed above while at the same time addressing problems that detract from the quality of life for individuals and groups in society, espe- cially those problems related to learning and human development. Viewed simplistically, instructional technology research might appear unquestionably “socially responsible.” After all, at some level, all instructional technology research can be said to focus on questions of SOCIAL RESPONSIBILITY AND INSTRUCTIONAL TECHNOLOGY RESEARCH Reeves, Herrington, and Oliver 101 how people learn and perform, especially with respect to how learn- ing and performance are influenced, supported, or perhaps even caused by technology. As long as research is focused on learning and per- formance problems, and adheres to the principles listed above, it would seem to be socially responsible. Some authorities argue that concern for the social responsibility of instructional technology research is ludicrous. These people main- tain that the goal of research is knowledge in and of itself, and that whether research is socially responsible is a question that lies outside the bounds of science (Carroll, 1973). Researchers in the “hard” sciences like biology and chemistry appear to rarely concern them- selves with the relevance question. However, this debate has raged for decades among educational researchers. For example, as reported by Farley (1982), Nate Gage, a past president of the American Edu- cational Research Association (AERA), was long a staunch defender of the notion that the goal of basic research in education is simply “more valid and more positive conclusions” (p. 12). Farley reported that another past president of AERA, Robert Ebel, proclaimed: . . . the value of basic research in education is severely lim- ited, and here is the reason. The process of education is not a natural phenomenon of the kind that has sometimes rewarded scientific investigation. It is not one of the givens in our universe. It is man-made, designed to serve our needs. It is not governed by any natural laws. It is not in need of re- search to find out how it works. It is in need of creative invention to make it work better. (p. 18) Ebel’s statement speaks directly to the issue of socially respon- sible research in instructional technology. There is little social rel- evance in research studies that are largely focused on understanding “how” instructional technology works without substantial concern for how this understanding might make education better. On the other hand, there is considerable social relevance in instructional technol- ogy research studies that are largely focused on making education better (and which in the process may also help us understand more about how instructional technology works). As Desforges (2000) wrote: “The status of research deemed educational would have to be judged, DESIGN RESEARCH 102 first in terms of its disciplined quality and secondly in terms of its impact. Poor discipline is no discipline. And excellent research with- out impact is not educational” (p. 2). Most of the published research in instructional technology in higher education has been grounded in a “realist” philosophy of science, that is, conducted under the assumption that education is part of an ob- jective reality governed by natural laws and, therefore, can be studied in a manner similar to other natural sciences like chemistry and biology (Reeves, 1993). If this assumption about the nature of educational phenomena is erroneous, then the wrong questions are being asked by much of our research. And even if there are underlying “laws” that influence learning, the complexity inherent in these laws may defy our ability to perceive, much less control, them. Three decades ago, the noted educational research authority, Lee Cronbach, (1975) pointed out that quantitative comparative research of the kind most often conducted in instructional technology may be doomed to fail- ure, because we simply cannot pile up generalizations fast enough to adapt our instructional treatments to the myriad of variables inherent in any given instance of instruction. DESIGN RESEARCH FOR HIGHER EDUCATION FACULTY ALTHOUGH STILL CONTROVERSIAL (Maxwell, 2004;Olson, 2004), there is renewed enthusiasm for experimentalresearch designs among some well-known educational re- searchers (Feuer, Towne, & Shavelson, 2002; Slavin 2002) of the kind described in this special issue by Ross, Morrison, and Lowther (2005). However, we maintain that this is not the most fruitful path for a design field like instructional technology, and thus we urge higher education faculty members to consider a design research approach. (As described below, design research is distinctly different in both its goals and methods from the developmental research model described in this special issue by Richey and Klein (2005).) Reeves, Herrington, and Oliver 105 The theoretical work of Dewey and others suggests that inquiry-based learning yields outcomes that are more robust and transferable. To pursue socially responsible research in this context, these geosciences instructors wish to establish a substantive, but achievable, goal that will be the focus of their design research. They decide that over the next three years, they will investigate the effectiveness of inquiry- based learning strategies in undergraduate courses in the geosciences, including atmospheric science, geography, and geology. Integration of design principles with technological affordances. Clark (2001) has long maintained that it is pedagogical methods, not technology per se, that most directly influence learning. Sharing this belief, Herrington, Reeves, Oliver, and Woo (2004) have described design principles for integrating authentic inquiry-based tasks into learning environments. To help in their design research initiative focused on inquiry-based course activities, the earth science research- ers could construct a prototype online learning support system that integrates real-world inquiry problems (for example, investigating the potential for mud slides in local communities under normal and extreme precipitation conditions) with Web-based educational re- sources; for example, those found in the Digital Library for Earth System Education (http://www.dlese.org/). The foundation for the learning support system might be developed using a commercial course management system like Blackboard or WebCT, or it might be built using an open source toolkit such as Moodle (http://moodle.org/) or LRN (http://www.dotlrn.org/). Inquiry to refine the learning environment and reveal new design principles. In the earth science scenario described above, the researchers would conduct numerous small and large scale studies aimed at testing and refining the prototype learning environment, and at the same time, investigating the feasibility and effectiveness of inquiry-based learning and digital library resources in undergraduate earth science courses. Quantitative, qualitative, and mixed-methods (Creswell, 2003; Gorard, Roberts, & Taylor, 2004) could be employed to provide the evidence needed to make progress toward the overall goal of enhancing teaching and learning in the geosciences through DESIGN RESEARCH 106 inquiry-based learning strategies while seeking to reveal design prin- ciples that could be applied in other initiatives. A qualitative inves- tigation of student attitudes toward the new learning approach might reveal that they value inquiry-based learning, but believe that the workload required is much heavier than that required in traditional lecture/lab based courses. Accordingly, the faculty researchers might decide to cutback on the number of inquiry-based activities in spe- cific courses but increase the depth of the remaining ones. Alterna- tively, a quantitative investigation of the actual time students put into the various courses in which they are enrolled might reveal that the students are putting much less time into traditional courses than their instructors expect. This might support a design principle that higher education courses should clearly state the time-on-task allocations expected for students beyond actual class meeting time. Long-term engagement and refinement of research method. Design research is not something that is normally undertaken in one month, one semester, or even one year. Two to five years are a more normal cycle, and in some cases, design research will be an ongoing enterprise for even longer periods. Design researchers must also be receptive to changes in the goals, methods, instrumentation, and re- porting cycles of a particular research agenda. Integrating inquiry- based learning methods into undergraduate science courses would never be easy, especially when it appears to be possible for people to graduate from a university without ever having been engaged in any authentic scientific research (National Research Council, 1999). The research- ers in this hypothetical scenario have established a feasible, but ambitious, three-year timeline for their design research project. It is likely that the design research initiative will extend beyond the initial three years, especially if the process and results of the project are rewarding. Intensive collaboration. Whereas previous instructional technol- ogy research in higher education has usually involved one or a few faculty members in a single department, design research requires the collaboration of academic instructors and other staff of diverse stripes. In the earth science scenario, the research team might include faculty Reeves, Herrington, and Oliver 107 members from various geosciences departments (e.g., geology, geogra- phy, and meteorology) as well as instructional design, educational re- search, multimedia production, and programming specialists from other campus units—an office of instructional development or the department of computer and networking services. While a few earth sciences faculty members might form the core of the initiative, remaining together through- out the three-year project, other participants like graduate students and programmers might come and go from time to time. Theory construction and problem solution. Design research has its origins in educators’ pragmatic desire to improve learning, not in a purely functional sense, but from an informed theoretical perspective (Newman, 1990). Design research is grounded in the practical reality of the instructor, from the identification of significant educational problems to the iterative nature of the proposed solutions. However, theoretical foundations and claims are critical to the design of solutions—as noted by Cobb, Confrey, diSessa, Lehrer, and Shauble (2003), “the theory must do real work” (p. 10). Theory informing practice is at the heart of the approach, and the creation of design principles and guidelines enables research outcomes to be transformed into educational practice. If the group of faculty members involved in the earth science ini- tiative simply sought to adopt inquiry-based learning methods without identifying generalizable design principles, they would be engaging in what educators call action research. This action-oriented research strat- egy has been around for more than fifty years. Corey (1953) defined action research as the process through which instructors study their own teaching practice to solve personal challenges in the classroom. Although action research certainly has merit (Reason & Bradbury, 2001), there is much more potential value in design research, because it combines seek- ing practical solutions to classroom problems with the search for design knowledge that others may apply. van den Akker provides a succinct description of design/develop- ment research: More than most other research approaches, development [de- sign] research aims at making both practical and scientific contributions. In the search for innovative ‘solutions’ for educational problems, interaction with practitioners . . . is es- DESIGN RESEARCH 110 • Emphasize content and pedagogy rather than technology. • Give special attention to supporting human interactions and nurturing learning communities. • Modify the learning environments until the pedagogical outcome is reached. • Reflect on the process to reveal design principles that can inform other instructors and researchers, and future devel- opment projects. If design research proliferates, it could contribute more than the ubiq- uitous, but ultimately futile, media comparison studies, and additionally overcome the sterility of most qualitative studies. If it becomes the preferred model in instructional technology research, design research may well advance the quality and usefulness of a field that is presently at risk of becoming inconsequential and irrelevant. Clearly, design research pre- sents a way forward towards more significant and socially responsible research. However, changing the mental models of academic researchers from those that are primarily experimental to those that are focused on design research will not be an easy task, especially given the ongoing preference for media comparison studies using experimental methods that have dominated this field for many decades. Saettler (1990) found evi- dence of experimental comparisons of educational films with classroom instruction as far back as the 1920s, and comparative research designs have been applied to every new educational technology since then. Nonetheless, the dominant mental models must evolve, and it is hoped that this special issue of the Journal of Computing in Higher Education is a positive step in the right direction. Certainly, the need for a more socially responsible research agenda in instructional technology has never been greater. Instead of continuing to tinker around the edges of teaching and learning challenges by conducting quasi-experimental studies focused on small changes in learning environments or even conducting one-off qualitative studies of esoteric cases, instructional technology researchers and their colleagues in other academic disciplines must begin to tackle the huge problems we face in the first quarter of the 21st Century. Design research offers a positive step in that incredibly important quest. Reeves, Herrington, and Oliver 111 REFERENCES Allen K.L., Galvis, D.L., & Katz R.V. (2004) Evaluation of CDs and chewing gum in teaching dental anatomy. Paper presented at the International Association for Dental Research 82nd General Session and Exhibition. Retrieved June 17, 2004, from http://iadr.confex.com/iadr/2004Hawaii/techprogram/abstract_40091.htm Bannan-Ritland, B. (2003). The role of design in research: The integrative learning design framework. Educational Researcher, 32(1), 21-24. Boyer, E.L. (1990). Scholarship reconsidered: Priorities of the professoriate. Princeton, NJ: Carnegie Foundation for the Advancement of Teaching. Bransford, J.D., Brown, A., & Cocking, R. (Eds.). (2000). How people learn: Mind, brain, experience and school. Washington, DC: National Academy Press. Carroll, J.B. (1973). Basic and applied research in education: Definitions, distinc- tions, and implications. In H.S. Broudy, R.H. Ennis, & L.I. Krimerman (Eds.), Philosophy of educational research (pp. 108-121). New York: John Wiley & Sons. Casti, J.L. (1989). Paradigms lost: Images of man in the mirror of science. New York: William Morrow. Clark, R.E. (Ed.). (2001). Learning from media: Arguments, analysis, and evidence. Greenwich, CT: Information Age Publishing. Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9-13. Cognition and Technology Group at Vanderbilt. (1990a). Anchored instruction and its relationship to situated cognition. Educational Researcher, 19(6), 2-10. Cognition and Technology Group at Vanderbilt. (1990b). Technology and the design of generative learning environments. Educational Technology, 31(5), 34-40. Corey, S. (1953). Action research to improve school practice. New York: Teachers College, Columbia University. Creswell, J.W. (2003). Research design: Quantitative, qualitative, and mixed method approaches (2nd ed.). Thousand Oaks, CA: Sage Publications. DESIGN RESEARCH 112 Cronbach, L.J. (1975). Beyond the two disciplines of scientific psychology. Ameri- can Psychologist, 30, 116-126. Cuban, L. (2001). Oversold and underused: Computers in the classroom. Cam- bridge, MA: Harvard University Press. Desforges, C. (2001, August). Familiar challenges and new approaches: Necessary advances in theory and methods in research on teaching and learning. The Desmond Nuttall/Carfax Memorial Lecture, British Educational Research Association (BERA) Annual Conference, Cardiff. Retrieved July 20, 2004, from http://www.tlrp.org/ acadpub/Desforges2000a.pdf Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5-8. Dewey, J. (1938). Experience and education. New York: Simon and Schuster. Dillon, A., & Gabbard, R. (1998). Hypermedia as an educational technology: A review of the quantitative research literature on learning comprehension, control and style. Review of Educational Research, 68(3), 322-349. Exploratorium. (1998). A description of inquiry. Retrieved August 19, 2004, from http://www.exploratorium.edu/IFI/about/inquiry.html Fabos, B., & Young, M.D. (1999). Telecommunications in the classroom: Rhetoric versus reality. Review of Educational Research, 69(3), 217-259. Farley, F.H. (1982). The future of educational research. Educational Researcher, 11(8), 11-19. Feuer, M.J., Towne, L., & Shavelson, R.J. (2002). Scientific culture and educational research. Educational Researcher, 31(8), 4–14. Gorard, S., Roberts, K., & Taylor, C. (2004). What kind of creature is a design experiment? British Educational Research Journal, 30(4), 577-590. Herrington, J., Reeves, T.C., Oliver, R., & Woo, Y. (2004). Designing authentic activities in web-based courses. Journal of Computing in Higher Education, 16(1), 3-29.