Essay Items: Understanding the Use and Benefits of Essay Questions in Assessments, Study notes of Human Development

Download Essay Items: Understanding the Use and Benefits of Essay Questions in Assessments and more Study notes Human Development in PDF only on Docsity! 119 Essay Items Hope You Can Write Difficulty Index ☺☺☺☺ (really pretty easy) One of the things that we will stress throughout this part of Tests & Measurement for People Who Hate Tests & Measurement is that the type of item you use as an assess- ment tool is closely tied to the level of information (or level of thinking skills) you want your test taker to show that he or she does (or does not) have. In Chapter 6, we emphasized how short answer items are best used when trying to assess memorization of factual information. In this chapter on essay items, you’ll find other tools that you can use to assess other types of outcomes. Here, we’ll focus on essay items—those items where the test taker is expected to write a coherent and informative response to a ques- tion. Forget about that Friday spelling test or even the SAT—essay tests are the real thing when it comes to seeing how well test takers integrate ideas and how well they can express them in written form. Essay questions allow for perhaps the most unrestricted type of written assessment item that we will cover in Tests & Measurement for People Who Hate Tests & Measurement. What you want to know is how well the test taker can organize information and express his or her ideas in writing. That’s why the really, really big exams in one’s academic career are usually of the essay type—these types of items just tap more higher-level and complex skills. 7 WHEN WE USE ‘EM AND WHAT THEY LOOK LIKE 07-Salkind (Tests) 4749.qxd 7/5/2005 9:20 PM Page 119 Essay questions come in two basic flavors: open-ended (also called unrestricted or extended) questions and closed-ended (also called restricted) questions. An open-ended (or unrestricted response) essay question is one where there are no restrictions on the response, including the amount of time allowed to finish, the number of pages written, or material included. Now, it is a bit impractical to allow test takers to have 25 hours to answer one essay question or to write hundreds of pages about anything they like. So, of course there are practical limits. It’s just that the limits do not define the scope of the response. For example, here’s an open-ended essay question. 1. Discuss the various theories of human development that have been talked about this semester. Among other things, be sure to compare and contrast the basic assumptions of the theories, the method used to study development, and the criticisms of each. You are free to take as much time as you need. Write as many pages as you like, and organize your answer as you see fit. That is a nice, open-ended question where the response is unrestricted. Now, take a look at this closed-ended question where the response is restricted. 1. Compare and contrast two of the basic theories of human development that have been discussed this semester. Include a brief summary of each theory, the method used to study devel- opment, and the criticisms of each perspective. Limit your response to five written pages, and do not write for longer than 2 hours. These two types of questions reflect different types of experiences. The first question, which is much less restrictive, gives the test taker a lot more flexibility (among other things) and allows for a more creative approach. The more restricted closed-ended question places definite limits on the content as well as the format. 120—— Part III ♦ The Tao and How of Testing Essay items are the item of choice if you want an unrestricted response and want to access higher-order thinking, such as the relationship between ideas and the pros and cons of a particular argument. THINGS TO REMEMBER: WHEN TO WRITE 07-Salkind (Tests) 4749.qxd 7/5/2005 9:20 PM Page 120 4. Have all test takers answer the same questions. This just reduces the burden placed on the developer of the test, but more important, makes the test easier to score—it’s just more practical. Why? This second example just provides additional direction, which may be exactly what the test taker needs to answer the question more completely. 3. Essay questions should be used only to evaluate higher-order outcomes, such as when comparisons, evaluations, analyses, and interpretations are required. Want to know what 643 is? (262,144)—the infant mortality rate of the United States in 2001? (6.9 per 1,000 live births)—what the French called tomatoes? (pomme d’amour for apple of love). If so, an essay question is not what you are looking for—you want the kind of item that tackles lower-level thinking skills such as knowledge or memorization. For the higher-order outcomes, such as evaluation and synthesis, you want the essay format and questions such as the following: 1. Identify the principle factors that contribute to infant mor- tality in the United States and discuss and evaluate the success of efforts that have been taken to close the gap between white and nonwhite populations. Such a question requires a knowledge of rates of infant mortal- ity but goes far beyond the statement of just facts. In fact, here’s this chapter’s cheat sheet about what kinds of words signal higher-order skills (and these words usually show up in essay items). Chapter 7 ♦ Essay Items: Hope You Can Write 123 Analyze Apply Arrange Classify Contrast Compare Create Evaluate Explain Generalize Infer Integrate Justify Organize Persuade Predict Summarize Synthesize 07-Salkind (Tests) 4749.qxd 7/5/2005 9:20 PM Page 123 Why Essay Items Are Good 1. It’s doubtful that you can come up with a better way of finding out not only what test takers know, but also how they relate ideas to one another. You’ve defined your learning goals, and the essay question is your choice of assessment tool because you are interested in finding out how well a test taker understands ideas and can relate ideas to one another. Essay questions most definitely tap into how well test takers can organize and integrate information. And essay questions also provide opportunities to demonstrate creativity. Because essay questions should have one correct answer, and (as you will learn shortly) there should be a model answer against which test takers’ responses should be compared. Allowing for multiple questions requires the test scores to have multiple stan- dards against which to compare them—too much work! Instead, rotate questions from year to year or test to test so there is a nice collection of questions and model answers. THE GOOD AND THE BAD Essay items have their advantages and disadvantages—let’s review them. And, you can find a summary of these in Table 7.1. 124—— Part III ♦ The Tao and How of Testing TABLE 7.1 The Advantages and Disadvantages of Essay Questions Advantages of Essay Items • They help find out how ideas are related to one another. • They increase security. • They provide increased flexibility in item design. • They are relatively easy to construct. Disadvantages of Essay Items • They emphasize writing. • They are difficult to write. • They provide an inadequate sampling of subject matter. • They are hard to score. • They emphasize writing skills over content. 07-Salkind (Tests) 4749.qxd 7/5/2005 9:20 PM Page 124 2. The use of essay questions increases security because it is very dif- ficult to plagiarize during an essay item examination. And, along the same lines, it is almost impossible for test takers to effectively guess the correct answer—so guessing is removed as a legitimate concern. 3. The essay item format has unparalleled flexibility. Take the time necessary to create a really good question—one that is exciting to consider and one that gives the test taker a fair chance at success. 4. Essay questions can be relatively easy to construct. If you know your material well (and we would sure assume such), you can put together four essay questions in less than an hour that can effec- tively tap higher-order learning. Sure beats creating 100 multiple- choice items at 10 minutes each (yep, about 15 hours). Why Essay Items Are Not So Good 1. They emphasize writing. Well, no kidding—that’s what they are supposed to do. But what they don’t do well at all is tapping the test taker who is knowledgeable about ideas and their relationship to one another, but just cannot express it in words. For these kinds of test takers, evaluating them fairly and accurately is always a challenge. 2. They can be tough to write. But didn’t we just say above that they were easy to write? Well, we said they can be. They can be a bear as well! The test designer has to invest a great deal of time in creating each essay question and making sure that it taps the objectives that are to be tested, but does so in a way to encourage the expression of ideas that you can’t find in a true/false or matching test. Chapter 7 ♦ Essay Items: Hope You Can Write 125 Among the many reasons why essays are particularly useful is that they are very flexible in both form (the size and complexity) and purpose (relate simple ideas or elaborate a complex argument). THINGS TO REMEMBER: WHY ESSAYS? 07-Salkind (Tests) 4749.qxd 7/5/2005 9:20 PM Page 125 5. If possible, grade the responses without knowing the test taker’s iden- tity. Because a subjective element can enter into the grading of essay questions, not knowing who the test taker is (and avoiding that pos- sible bias) can be a great help. There are a bunch of ways to do this (using code numbers keyed to names rather than just names is one example), but another way is to have the test taker put his or her name on the back (and blank) side of the first page of the responses. The grader won’t see it and any conflict can be avoided. • Thesis and organization • Content knowledge • Writing style and mechanics Each of these elements, for example, could be scored separately. A model too difficult for you to do? Then at least have the impor- tant points listed on a separate sheet of paper against which you can compare test takers’ responses. 4. Score each question across all test takers. Given that you have a model answer, go through each test focusing the same question each time. For example, score the answer to Question 1 across all test takers, and then go back and score Question 2 across all test takers. This allows you to make absolute judgments in comparison to your model answer, but also to make relative judgments (if nec- essary) within any one item. 128—— Part III ♦ The Tao and How of Testing Essay Items and Unreliability You remember from Chapter 2 that our discussion about reliabil- ity focused in part on the reduction of error variance. The more error variance that is removed from the entire testing situation, the higher the reliability. OK—given that thought, there are a lot of sources of error variance when it comes to essay items, and perhaps the largest source is differences in grading. By its very nature, an essay item has so many indefinite things associated with it (objectivity of the grader, student’s writing skills, etc.) that it is even more important to control what you can. So, do try and ensure anonymity, use a model for scoring, and standardize conditions whenever possible. You’ll have a more reliable test, and the test takers will be treated more fairly. 07-Salkind (Tests) 4749.qxd 7/5/2005 9:20 PM Page 128 SUMMARY Essay questions are terrific to get at those more sophisticated thinking skills that we have to assess to find out if an individual understands more complex ideas and how these ideas relate to one another. But although they might be relatively easy to create (they are short, after all, and only a few are needed), they can underrep- resent content, they are tough to score, and there are lots of reasons why they could be considered unreliable. So, use ’em only when they need to be used! TIME TO PRACTICE 1. In your area of interest, write one stunningly terrific essay question. Then, exchange it with a classmate and evaluate it according to the guidelines that were presented in this chapter. 2. Explain why essay questions are more useful for tapping higher-order thinking skills than just straight memorization. 3. Name two advantages of essay items and provide an explanation as to why they are advantages. 4. Write three essay items that violate at least one of the guidelines we identified in this chapter, and indicate what’s wrong. 5. Name two disadvantages of essay items and discuss how you might compensate for them. ANSWERS TO PRACTICE QUESTIONS 1. Here’s my question: Discuss the origins of the testing movement in the United States and be sure to identify how the social and political events of the times helped encourage the growth of the testing industry (45 minutes). 2. Unlike test items such as short answer or completion (or many other types you have yet to learn about), essay questions provide the flexibility to explore ideas and not just require memorization or the repetition of facts. 3. First, good performance on an essay question demands an understanding of ideas and how they relate to one another. That’s very important in topics where there is a premium on relationships between ideas and concepts rather than rote memo- rization of those concepts. Second, they are very flexible in that they can be adjusted to meet the needs of the assessment setting. One set of essays might be used to look at a description of a particular historical event, whereas a more Chapter 7 ♦ Essay Items: Hope You Can Write 129 07-Salkind (Tests) 4749.qxd 7/5/2005 9:20 PM Page 129 ambitious one (for the test taker, that is) might be used to look at how certain unhistorical events had an impact on others and what the consequences of each event was for future policy decisions. 4. Here are mine: a. What are some of the important things that happened in American history? (unclear and incomplete) b. Name the planets in order of their distance from the sun, with the closest first. (not an essay question) c. When she didn’t establish a foundation for the foundation, she made a terrible mistake that angered her mother. Relate that to how she furthered her career. (Uh—very unclear and include). 5. My two are as follows: a. There’s no getting around the fact that the writing component to essay ques- tions can be very important. But one way to compensate for that is to make sure the test takers are capable of expressing themselves in writing before adminis- tering such an exam. Or, provide adequate outlining assistance and extensive practice organizing and writing such a response. These are tough to get by, but perhaps worth it. b. Essay questions are no fun, tedious, and very draining to score—especially if you have a lot of them to do. One way to help with this task is to use a model answer (as we discussed earlier in this chapter) and to work in small chunks of time so that you remain refreshed. WANT TO KNOW MORE? Further Readings • Hamberg, K., Risberg, G., Johansson, E. E., & Westman, G. (2002). Gender bias in physicians’ management of neck pain: A study of the answers in a Swedish national examination. Journal of Women’s Health and Gender-Based Medicine, 11(7), 653–665. Essay questions can be used in many different settings as exem- plified here, where these researchers examined gender differences in the diagnosis and management of neck pain. Modified essay questions were used to record suggestions about management of neck pain. • Bridgeman, B., & Morgan, R. (1996). Success in college for students with discrepancies between performance on multiple-choice and essay tests. Journal of Educational Psychology, 88(2), 333–340. The title says it all. Students in the top third on the essay portion of a test and the bottom third on the multiple-choice portion of the 130—— Part III ♦ The Tao and How of Testing 07-Salkind (Tests) 4749.qxd 7/5/2005 9:20 PM Page 130