Quantitative Research Design: Experimental vs. Non-Experimental - Focus on Paper Tests - P, Study notes of History of Education

Download Quantitative Research Design: Experimental vs. Non-Experimental - Focus on Paper Tests - P and more Study notes History of Education in PDF only on Docsity! 1 Experimental Measures and Non-Experimental Quantitative Research Design Katie Rommel-Esham Education 504 Paper-and Pencil Tests A situation in which a standard set of questions is presented to each subject. The focus is on some cognitive task: what the person knows (achievement), is able to learn (ability or aptitude), chooses or selects (interests, attitudes, or values), or is able to do (skills). All tests measure current performance. 2 Standardized Tests Provide uniform procedures for administration and scoring, which may include timing, a script, specified testing conditions, materials allowed, whether or not students’ questions may be answered Scoring is usually objective Most have been given to some norming group Most are prepared commercially which helps to provide some degree of insurance regarding norms, reliability, and validity May not be specific enough to provide as sensitive a measure as is needed. Criterion-Referenced Interpretation Individual’s scores are compared to a predetermined standard of performance (the criterion), not with the scores of others Score is usually expressed as a percentage or pass/fail Focuses on what individuals are able to do Most result in highly skewed distribution* Good for diagnosis 5 Standard Scores Most common is the z-score: where xi is an individual score, is the mean, and σ is the standard deviation ! z = x i " x # ! x z- Score The z-score provides an indication of how far a given score is from the mean in standard deviation units It allows comparisons across non-equivalent testing situations Scores below the mean will have a negative z - score, while those above the mean will have a positive z -score A z-score is not a score that would be recorded in a grade book, however it provides useful information 6 Aptitude Tests Used to predict future performance on a criterion prior to instruction, placement, or training The test itself, not individual items, is considered to be predictive The terms “intelligence test” and “ability test” are often used interchangeably with aptitude Measures of Noncognitive Traits Personality, Attitude, Value, and Interest Inventories 7 Questionnaires Most widely used technique to gather information from subjects They are economical, anonymous, and consistent Subjects respond to something written Guidelines for Questionnaires Items should be clear Questions should be limited to a single idea or concept (not “double barrelled”) Respondents should be competent to answer (refers to timing, area of expertise, etc) Questions should be relevant Items should be short and simple There should be no negative items There should be no biased items 10 Checklist May provide respondents with a number of items from which to choose May also be used to ask students to reply yes/no, or choose the category to which they belong For categorical responses, respondents can be placed in exactly one category Problems Response set: includes selecting all positive responses or all negative responses (regardless of content), guessing, and sacrificing speed for accuracy Faking: includes responding in socially desirable ways Reliability is generally lower than in cognitive tests Construct validity is difficult to establish Because there are generally not “right” answers, the nature of the comparison group is particularly important 11 Observations Rely on a researcher seeing and hearing things and recording these observations rather than relying on subjects’ self-report responses to questions or statements In quantitative research, the observer acts as a complete observer Observations may be high inference or low inference Observer Effects Observer Bias: due to preconceived notions of the observer Contamination: result of observer’s knowledge of the purpose of the study Halo Effect: based on initial impressions 12 Interviews Interviews are essentially vocal questionnaires Advantages of Interviews They are flexible and adaptable, and may be used with those who are not capable of completing a questionnaire (those who are illiterate or too young to read and write, for example) Responses may be followed up, clarified Result in a higher response rate than questionnaires