Descriptive Research: Survey, Correlational and Observational Techniques, Study notes of History of Education

Download Descriptive Research: Survey, Correlational and Observational Techniques and more Study notes History of Education in PDF only on Docsity! Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 1 1 Survey & Correlational Research Stephen E. Brock, Ph.D., NCSP California State University, Sacramento 2 Introduction Survey (AKA Descriptive) Research is a quantitative methodology that is very similar to qualitative research. that is used to describe the distribution (or make inferences about the distribution) of a variable within a population. that makes use of numbers (instead of words) to describe the variable (or variables) under study. within which sampling is a critical issue (unless your are able to consider all members of the population). that reports results via descriptive statistics (measures of central tendency and variance are the most common). 3 Portfolio Activity #7: Mini-proposal 1 Briefly describe a survey research project relevant to one of your identified research topics. Small group discussions. Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 2 4 Challenges A unique problem faced by some forms of descriptive research is the lack of participant response (“return rate”). Another problem specific to this research method is the fact that researcher’s are often unable to explain to participants exactly what a given word or question means. 5 Strategies 1. Survey or ask people questions about the variable under study (“Self Report Research”). Such an approach is most appropriate when the variable is difficult to observe and/or the sample is large. For example, attitudes, feelings, opinions of a group of any size, or the behaviors of a vary large population. 2. Directly observe the variable This approach is most appropriate when the variable can be reliably observed and/or the population sample is small. For example, the physical aggression or on-task behavior of the students in a specific type of classroom. 6 Types of Surveys Questionnaires An efficient method Requires less time Requires less $ Allows access to more participants Interviews Face to face More difficult and costly Phone Easier to sample from a larger geographic area. What are some of the challenges to result validity presented by the different types of surveys? Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 5 13 Additional Questionnaire Construction Issues Make sure each item asks only one question. Be specific! Avoid vague terms or jargon. Are there situations wherein use of such would be appropriate? Ensure it looks as professional as possible. This includes how the questionnaire is packaged (e.g., stuffed into envelopes). Pilot test all questionnaires with all subgroups of the population to be sampled. 14 Always Include a Cover Letter The letter should … be personal (if possible). be brief. stress the nature and importance of the study. identify your affiliation and endorsements. Including your advisor as appropriate assure the anonymity/confidentiality of the responses. specify the deadline for questionnaire return. Not to long (about 2 weeks) be personally signed. 15 Questionnaire Follow-up Include a return post card (that is separate) from the questionnaire that will allow respondents to identify that they have responded, but not allow their names to be attached to the questionnaire. Can be used for a prize drawing. Short of 100%, never be satisfied with your return rate. As a standard practice, send out a second set of questionnaires. Doing so typically increases return rate by about 20%. Interview a random sample of non-responders to determine if they systematically differ on variables important to your study. Who didn’t respond and what were there reasons? Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 6 16 Selecting Participants The entire population Simple random sampling Stratified random sampling Cluster sampling Systematic Sampling Non-random Sampling 17 Survey Distribution Snail-Mail Strengths: anonymous, easy to score, standardized Weaknesses: response rate, requires reading skills, subjective interpretation of survey items, requires a mailing address. E-Mail Strengths: inexpensive, speed. Weaknesses: requires an e-mail address, could get multiple replies from a single participant. Telephone Strengths: high response rate, speed, over comes geographical limitations. Weaknesses: requires a phone, interviewers need to be trained. 18 Survey Distribution Personal Administration Strengths: efficient when respondents are geographically close to each other. Weaknesses: training required, time consuming. Interview Strengths: return rate, allows for probing, may be recorded. Weaknesses: time consuming, not anonymous, interviewer bias, training required, complex scoring of open ended questions. Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 7 19 Interviews Most appropriate when … the number of variables is large. the population/sample is small. the variables are hard to observe. 20 Interview Development 1. Structured Protocol is identical for all participants. 2. Semi-Structured A core of similar questions, but variations are allowed. 3. Unstructured Depth interviewing, focus groups. 21 Interview Development Closed questions Open ended questions Training Pilot, tape record, practice Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 10 28 Correlational Research A quantitative methodology used to determine whether, and to what degree, a relationship exists between two or more variables within a population (or a sample). The degree of relationships are expressed by correlation coefficients. Coefficients range from +1.00 to -1.00 Higher correlations (coefficients closer to +1.00 or -1.00) indicate stronger relationships. Positive correlations indicate that as the values associated with one variable go up, so do the values associated with the other. e.g., higher grades are associated with higher ???. Negative correlations indicate that as the values associated with one variable go up, the values associated with the other go down e.g., higher grades are associated with lower ???. 29 Portfolio Activity # 8: Mini-proposal 2 Briefly describe a correlational research project relevant to one of your identified research topics. Small group discussion 30 Correlational Research: Scatter Plots http://www.mste.uiuc.edu/courses/ci330ms/youtsey/scatterinfo.html Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 11 31 Correlational Research: Scatter Plots http://www.mste.uiuc.edu/courses/ci330ms/youtsey/scatterinfo.html 32 Correlational Research: Scatter Plots Interactive web page http://math.hws.edu/javamath/config_applets/ ScatterPlotApplet.html More information about developing and using scatter plots http://www.isixsigma.com/tt/scatter_diagram/ 33 Correlation vs. Causation A researcher found that there was a +0.85 correlation between the variable of height and Mental Age among a random sample of 100 individuals. From these data the researcher determines that taller people are smarter than shorter people. What do you think? Interpret this finding. Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 12 34 Correlation vs. Causation Mental Age Height Chronological Age 35 Correlation vs. Causation Even a perfect correlation does not necessarily imply a causal connection between variables. For example, in a recent CDE study, the number of support staff in school districts was positively correlated with poor attendance. An educational research example: Attention span is highly correlated with reading comprehension test scores. But both are also correlated with basic reading skill. The correlation may be the result of a mutual association with these other variables. 36 Correlation vs. Causation Attention Deficits Reading Comprehension Basic Reading Skill Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 15 43 The Correlational Research Process 5. Conduct Data Analysis (continued) Determining statistically significant correlations Look up r in a table of correlation coefficients http://www.ecowin.org/aulas/resources/stats/correlation.htm (ignoring + or - sign). The number of degrees of freedom is two less than the number of pairs. 44 5. Conduct Data Analysis (continued) Correlation’s significance vs. its strength. Just because a correlation is significant does not mean it is high enough to reflect an important relationship. Variance (the correlation coefficient squared) “When two or more variables are correlated, each variable will have a range of scores. Each variable will have some variance; that is not everyone will get the same score. Common or shared variance indicates the extent to which variables vary in a systematic way” (pp. 314-315). r2 is the amount of variance explained (or accounted for) by the correlation coefficient. Determine the amount of variance accounted for by the following r values: 1.0, .95, .75, .50, .25 http://www.calculator.org/jcalc98.html The Correlational Research Process 45 Relationship Studies Often used to study complex variables before beginning an experiment. To identify variables (other than the independent variable) that correlate with the dependent measure. When relationships are identified these variables are then controlled for. For example, before studying how a given IV (like ADHD symptom severity) influences reading comprehension you would want to identify other variables (such as word reading, word attack, vocabulary, background knowledge) that also affect reading comprehension and then control for them. How would this be done? Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 1 1 Survey & Correlational Research Stephen E. Brock, Ph.D., NCSP California State University, Sacramento 2 Introduction Survey (AKA Descriptive) Research is a quantitative methodology that is very similar to qualitative research. that is used to describe the distribution (or make inferences about the distribution) of a variable within a population. that makes use of numbers (instead of words) to describe the variable (or variables) under study. within which sampling is a critical issue (unless your are able to consider all members of the population). that reports results via descriptive statistics (measures of central tendency and variance are the most common). 3 Portfolio Activity #7: Mini-proposal 1 Briefly describe a survey research project relevant to one of your identified research topics. Small group discussions. Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 2 4 Challenges A unique problem faced by some forms of descriptive research is the lack of participant response (“return rate”). Another problem specific to this research method is the fact that researcher’s are often unable to explain to participants exactly what a given word or question means. 5 Strategies 1. Survey or ask people questions about the variable under study (“Self Report Research”). Such an approach is most appropriate when the variable is difficult to observe and/or the sample is large. For example, attitudes, feelings, opinions of a group of any size, or the behaviors of a vary large population. 2. Directly observe the variable This approach is most appropriate when the variable can be reliably observed and/or the population sample is small. For example, the physical aggression or on-task behavior of the students in a specific type of classroom. 6 Types of Surveys Questionnaires An efficient method Requires less time Requires less $ Allows access to more participants Interviews Face to face More difficult and costly Phone Easier to sample from a larger geographic area. What are some of the challenges to result validity presented by the different types of surveys? Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 5 13 Additional Questionnaire Construction Issues Make sure each item asks only one question. Be specific! Avoid vague terms or jargon. Are there situations wherein use of such would be appropriate? Ensure it looks as professional as possible. This includes how the questionnaire is packaged (e.g., stuffed into envelopes). Pilot test all questionnaires with all subgroups of the population to be sampled. 14 Always Include a Cover Letter The letter should … be personal (if possible). be brief. stress the nature and importance of the study. identify your affiliation and endorsements. Including your advisor as appropriate assure the anonymity/confidentiality of the responses. specify the deadline for questionnaire return. Not to long (about 2 weeks) be personally signed. 15 Questionnaire Follow-up Include a return post card (that is separate) from the questionnaire that will allow respondents to identify that they have responded, but not allow their names to be attached to the questionnaire. Can be used for a prize drawing. Short of 100%, never be satisfied with your return rate. As a standard practice, send out a second set of questionnaires. Doing so typically increases return rate by about 20%. Interview a random sample of non-responders to determine if they systematically differ on variables important to your study. Who didn’t respond and what were there reasons? Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 6 16 Selecting Participants The entire population Simple random sampling Stratified random sampling Cluster sampling Systematic Sampling Non-random Sampling 17 Survey Distribution Snail-Mail Strengths: anonymous, easy to score, standardized Weaknesses: response rate, requires reading skills, subjective interpretation of survey items, requires a mailing address. E-Mail Strengths: inexpensive, speed. Weaknesses: requires an e-mail address, could get multiple replies from a single participant. Telephone Strengths: high response rate, speed, over comes geographical limitations. Weaknesses: requires a phone, interviewers need to be trained. 18 Survey Distribution Personal Administration Strengths: efficient when respondents are geographically close to each other. Weaknesses: training required, time consuming. Interview Strengths: return rate, allows for probing, may be recorded. Weaknesses: time consuming, not anonymous, interviewer bias, training required, complex scoring of open ended questions. Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 7 19 Interviews Most appropriate when … the number of variables is large. the population/sample is small. the variables are hard to observe. 20 Interview Development 1. Structured Protocol is identical for all participants. 2. Semi-Structured A core of similar questions, but variations are allowed. 3. Unstructured Depth interviewing, focus groups. 21 Interview Development Closed questions Open ended questions Training Pilot, tape record, practice Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 10 28 Correlational Research A quantitative methodology used to determine whether, and to what degree, a relationship exists between two or more variables within a population (or a sample). The degree of relationships are expressed by correlation coefficients. Coefficients range from +1.00 to -1.00 Higher correlations (coefficients closer to +1.00 or -1.00) indicate stronger relationships. Positive correlations indicate that as the values associated with one variable go up, so do the values associated with the other. e.g., higher grades are associated with higher ???. Negative correlations indicate that as the values associated with one variable go up, the values associated with the other go down e.g., higher grades are associated with lower ???. 29 Portfolio Activity # 8: Mini-proposal 2 Briefly describe a correlational research project relevant to one of your identified research topics. Small group discussion 30 Correlational Research: Scatter Plots http://www.mste.uiuc.edu/courses/ci330ms/youtsey/scatterinfo.html Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 11 31 Correlational Research: Scatter Plots http://www.mste.uiuc.edu/courses/ci330ms/youtsey/scatterinfo.html 32 Correlational Research: Scatter Plots Interactive web page http://math.hws.edu/javamath/config_applets/ ScatterPlotApplet.html More information about developing and using scatter plots http://www.isixsigma.com/tt/scatter_diagram/ 33 Correlation vs. Causation A researcher found that there was a +0.85 correlation between the variable of height and Mental Age among a random sample of 100 individuals. From these data the researcher determines that taller people are smarter than shorter people. What do you think? Interpret this finding. Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 12 34 Correlation vs. Causation Mental Age Height Chronological Age 35 Correlation vs. Causation Even a perfect correlation does not necessarily imply a causal connection between variables. For example, in a recent CDE study, the number of support staff in school districts was positively correlated with poor attendance. An educational research example: Attention span is highly correlated with reading comprehension test scores. But both are also correlated with basic reading skill. The correlation may be the result of a mutual association with these other variables. 36 Correlation vs. Causation Attention Deficits Reading Comprehension Basic Reading Skill Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 15 43 The Correlational Research Process 5. Conduct Data Analysis (continued) Determining statistically significant correlations Look up r in a table of correlation coefficients http://www.ecowin.org/aulas/resources/stats/correlation.htm (ignoring + or - sign). The number of degrees of freedom is two less than the number of pairs. 44 5. Conduct Data Analysis (continued) Correlation’s significance vs. its strength. Just because a correlation is significant does not mean it is high enough to reflect an important relationship. Variance (the correlation coefficient squared) “When two or more variables are correlated, each variable will have a range of scores. Each variable will have some variance; that is not everyone will get the same score. Common or shared variance indicates the extent to which variables vary in a systematic way” (pp. 314-315). r2 is the amount of variance explained (or accounted for) by the correlation coefficient. Determine the amount of variance accounted for by the following r values: 1.0, .95, .75, .50, .25 http://www.calculator.org/jcalc98.html The Correlational Research Process 45 Relationship Studies Often used to study complex variables before beginning an experiment. To identify variables (other than the independent variable) that correlate with the dependent measure. When relationships are identified these variables are then controlled for. For example, before studying how a given IV (like ADHD symptom severity) influences reading comprehension you would want to identify other variables (such as word reading, word attack, vocabulary, background knowledge) that also affect reading comprehension and then control for them. How would this be done? Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 16 46 Relationship Studies Why is it important to be selective when identifying variables to be correlated? What problems might arise if you used a “shotgun approach” and obtained correlations among 100 randomly selected variables and you used a p value of .05? Chances are that 5 of the obtained coefficients will not reflect a true relationship greater than zero. 47 Prediction Studies Regression Analysis A method of analyzing the variability of a criterion variable by examining information available on one or more predictor variables. When only one predictor variable is used, the analysis is referred to as simple regression. When more than one predictor is used, the analysis is referred to as multiple regression 48 Simple Regression A college football coach wishes to use the scores on one variable to predict the scores on another variable. He wishes to determine the best prediction equation for the grade-point averages of potential freshmen recruits. SAT test scores for the current group of recruits as well as their grade point averages are available. From the available SAT and GPA scores for this year’s class, the prediction equation for next year’s class can be calculated. Stephen E. Brock, Ph.D., NCSP EDS 250 Descriptive Research 17 49 Simple Regression A teacher wishes to determine the effects of hours of study (the predictor variable) on vocabulary test performance (the criterion variable). When vocabulary test means associated with different amounts of study differ from each other and lie on a straight line, it is said that there is a simple linear regression of vocabulary test performance on hours of study 50 Hours of study & vocabulary tests 6125 9105 875 765 5124 794 674 554 4103 683 563 443 3102 372 462 342 091 361 251 131 Score BScore AHours SPSS Data Sheet Score Set A Score Set B 51 Multiple Regression A GATE program administrator wishes to determine the best prediction equation for rapid learning among a group of ELL elementary students. The available predictor variables are: 1. SAT-9 scores (X1) 2. Changes in scores on the English LAS (X2) 3. Scores on a non-verbal reasoning test (X3) 4. Primary language vocabulary test scores (X4) Y The criterion variable (the one the administrator wishes to predict) are achievement test gains made between 2nd to 3rd grades (Yobserved). From the regression of the predictor variables (X1, 2, 3, and 4) on the criterion variable (Yobserved), a prediction equation can be developed.