Georgia Tech Writing Outcomes Assessment Report for ENGL 1101 and ENGL 1102 (2003-2004), Exams of Grammar and Composition

Download Georgia Tech Writing Outcomes Assessment Report for ENGL 1101 and ENGL 1102 (2003-2004) and more Exams Grammar and Composition in PDF only on Docsity! General Education Assessment Report: Communications Objective–Writing Outcomes Academic Year 2003-04 Georgia Tech Office of Assessment Jonathan Gordon, Assistant Director August 2004 Georgia Tech Office of Assessment General Education Assessment Report—Communications Objective—Writing Outcomes August 2004 Assessment of Writing Learning Outcome in ENGL 1101, ENGL 1102 and LCC 3401, AY 2003-2004 The Institute Undergraduate Curriculum Committee’s Ad Hoc Subcommittee on General Education developed a set of nine general education objectives for all students earning a Georgia Tech baccalaureate degree. Specifically, the objectives call for: • Technical, mathematical, and scientific competence • An ability to communicate to and productively interact with others • An awareness of culture and values in a diverse world • An understanding of ethical issues surrounding one’s personal and professional activities The Communication Objective as defined by the IUCC and approved by the Faculty Assembly is stated as follows: Georgia Tech students will be able to read a variety of documents critically, acquire and synthesize information, and shape a written or oral presentation that accommodates audience needs and shows a mastery of basic communications skills. The specific assessable learning outcomes stemming from this objective are that students will be able to: 1. Locate the primary thesis in a written work and explain how it is supported by logical arguments. 2. Produce effective writing that supports a given thesis using clear prose, logical organization, and standard spelling, punctuation, and grammar. 3. Deliver a presentation that demonstrates effective core presentation skills, including focus, organization, and delivery. 4. Conduct an effective information search that includes a variety of reference sources (e.g., indexes and library catalogs, bibliographies, and Internet searches). This report focuses on outcome 2: written communication. A separate assessment in LCC 3401: Technical Communications addresses outcome #3, while the Georgia Board of Regents addresses outcome #1 through its Regents examination. An assessment of outcome #4 is also under development. Two separate assessments conducted for the School of Literature, Communication and Culture are presented here. In Fall 2003, an assessment of ENGL 1101: English Composition I was conducted. In Spring 2004, an assessment of ENGL 1102: English Composition II was conducted. ENGL 1101 and ENGL 1102 are both required of all Georgia Tech students. In each course, a random sample of students was drawn from various class sections and selected for the assessment. For each student selected, two short writing assignments were evaluated. The first assignment was completed at the beginning of the semester (pre-instruction); the second assignment was completed toward the end of the semester (post-instruction). Each writing assignment was read by three raters and scored on the basis of a grading rubric. The rubric assessed the persuasiveness of student’s argument (argument), the organization of the document (organization), and the style and mechanics of the writing. In addition, a holistic - 2 - Georgia Tech Office of Assessment General Education Assessment Report—Communications Objective—Writing Outcomes August 2004 Table 4. ANOVA Results by Factors ENGL 1101 ANOVA Results Gender College Final Course Grade Admission Index Quartile SAT Verbal Quartile Argument n.s. n.s. n.s. n.s. n.s. Organization n.s. n.s. n.s. n.s. n.s. Style n.s. n.s. n.s. n.s. ++ Overall n.s. n.s. n.s. n.s. n.s. +F-statistic Pr<.05 ++ F-statistic Pr<.01 +++ F-statistic Pr<.001 n.s. Not significant ENGL 1102 A random sample of 172 students drawn from 44 class sections was selected for the assessment. Based on unweighted averages of the three raters, the number and percent of students whose performance over the course of the semester improved, declined, or remained unchanged is presented below. A majority of students demonstrated improvement in all four categories. Table 5 Argument Organization Style Overall Declined 19 (11.1%) 31 (18.0%) 29 (16.9%) 15 (8.7%) Improved 122 (70.9%) 119 (69.2%) 115 (66.9%) 127 (73.8%) Unchanged 31 (18.0%) 22 (12.8%) 28 (16.3%) 30 (17.4%) Percentages may not total 100 due to rounding The scores for each rater were combined to create an unweighted average for each component of the rubric. The early semester and late semester averages are presented below. A matched-pairs t- test was performed to test the null hypothesis that there is no difference between the scores obtained by a student at the beginning of the semester and the end of the semester. The results indicate a highly significant difference between the scores in all four categories of the grading rubric. The null hypothesis can thus be safely rejected. Table 6 Pre-instruction Post-instruction Difference t-value Argument 1.72 2.18 .46 10.92* Organization 1.57 2.10 .53 10.15* Style 1.76 2.16 .40 9.79* Overall 1.66 2.20 .54 12.72* * Pr<.0001 The following table presents the student performance score frequencies from all of the raters. Chi-square tests for independence indicate that the null hypothesis of equivalence in the distribution of scores between the pre- and post-instruction can confidently be rejected. - 5 - Georgia Tech Office of Assessment General Education Assessment Report—Communications Objective—Writing Outcomes August 2004 Table 7 Pre-instruction Post-instruction Below Average Average Above Average Below Average Average Above Average Argument χ²=121.9, df=2; p<.0001 185 (35.9%) 287 (55.7%) 43 (8.4%) 67 (13.2%) 282 (55.5%) 159 (31.3%) Organization χ²=129.5, df=2; p<.0001 271 (55.6%) 193 (37.5%) 51 (9.9%) 112 (22.1%) 231 (45.5%) 165 (32.5%) Style χ²=97.8, df=2; p<.0001 166 (32.2%) 305 (59.2%) 44 (8.5%) 67 (13.2%) 294 (57.9%) 147 (28.9%) Overall χ²=164.4, df=2; p<.0001 207 (40.3%) 273 (53.1%) 34 (6.6%) 58 (11.4%) 291 (57.4%) 158 (31.2%) Percentages may not total 100 due to rounding Analysis of Variance Results Analysis of variance was run on the late semester scores by several factors. As with the ENGL 1101 results, appropriate post-hoc tests were conducted when significant F-statistics were generated. There were significant gender effects with females scoring higher than males in all four categories. With final course grade, significant effects were found in the argument, style and overall scores. In the argument and overall scores, Tukey’s HSD post hoc tests revealed differences between those receiving A’s in the course versus those receiving C’s, D’s and F’s. In the style score, post-hoc tests could not discern differences among the groups. With the SAT Verbal quartiles, a significant effect was found in the style score, with those in the highest quartile outscoring those in the lowest quartile. No significant differences were found among college or admission index quartile.3 The results from the ENGL 1102 assessment appear to be more robust than those of ENGL 1101. The fact that final course grade was a significant factor for most of the subscores demonstrates a relationship between the learning outcome and overall performance in the course. Table 8 ANOVA Results by Factors ENGL 1102 ANOVA Results Gender College Final Course Grade Admission Index Quartile SAT Verbal Quartile Argument ++ n.s. + n.s. n.s. Organization + n.s. n.s. n.s. n.s. Style + n.s. + n.s. + Overall ++ n.s. + n.s. n.s. +F-statistic Pr<.05 ++ F-statistic Pr<.01 +++ F-statistic Pr<.001 n.s. Not significant 3 Colleges included in this analysis were College of Computing, College of Engineering, Ivan Allen College, and College of Sciences. - 6 - Georgia Tech Office of Assessment General Education Assessment Report—Communications Objective—Writing Outcomes August 2004 Conclusion The assessment data clearly indicate that improvements have occurred over the course of the semester in all aspects of the grading rubric for both ENGL 1101 and ENGL 1102. The differential ANOVA results between these courses might be related to idiosyncrasies in the assessment methods used in each course. Because these courses are both required as part of the general education core and are usually taken in sequence, a single assessment approach might be considered. For example, students might provide writing samples upon beginning ENGL 1101 and then followed up upon completion of ENGL 1102. This approach would simplify the assessment of the written communication objective and could provide more concrete results. - 7 -