Comparing Data Quality in Surveys: Face-to-Face, Telephone, and Self-Reported Methods, Study notes of Psychology

Download Comparing Data Quality in Surveys: Face-to-Face, Telephone, and Self-Reported Methods and more Study notes Psychology in PDF only on Docsity! Midterm: Things to Know Ways of Knowing/Psychology as a Science: - What are predictor/independent variables, and what are criterion/dependent variables?  Predictor/criterion variables – are general terms used to describe any predictive relationship between variables, regardless of method  Independent/dependent variables – are specific terms used to described a predictive relationship that exists between a variable which has been manipulated by the experimenter and an outcome variable of interest  Subject/cause/predictor variable/independent variable  Object/effect/criterion variable/dependent variable  Both are called constructs - What are constructs? What are operational definitions?  Constructs – are abstract concepts we would like to measure  Operational definitions – is a set of procedures used to measure or manipulate a construct o Indirect, imperfect measurement -How are they related (e.g. what ties them together?)  We apply operational definitions to constructs in order to make them accessible and concrete -Why is the distinction important? - What are hypotheses? What are theories? What is the relationship between hypotheses and theories?  Hypothesis – falsifiable statement of the association between two or more constructs o Usually specify conditions under which a hypothesis holds  Theory – is an integrated set of hypotheses that defines, explains, organizes, and identifies the relevant constructs o Specifies observable variables that define the constructs o Describes relationships among constructs, and under what conditions o To test part of a theory requires that you measure/operationalize/manipulate the abstract concepts - What are characteristics of a good theory?  Falsifiable – explaining everything, explains nothing o Findings that would contradict the theory can be identified and described  Parsimonious – the simpler the explanation, the better – meaning relies on the fewest possible underlying assumptions  Specific – specifies constructs of interest and how to measure them  Consistent – it is consistent w/ known research findings  Stimulation value – spurs on other research - What are sources of support for naïve hypotheses?  Logical analysis  Authority  Consensus  Observation  Past experience -What are the problems with these sources? o Biased conclusions  Logical analysis – logical conclusions are based on beliefs as well as logic  Authority – authorities have their own interests and beliefs as well  Consensus – beliefs and values determine who we see as peers; peers have own interests and beliefs as well; groups of people are poor as independent judges (“group-think”)  Observation – can modify/abandon hypotheses if not consistent w/ observations constructs ca mean different things to different people; inferring that one construct causes another is hard (3rd variable problem, reverse causality); choice of who to observe; bias in decisions of which observations matter  Past experience – memory is reconstructive and inaccurate; store and organize events selectively; consistent info easier to recall than disconfirming info o Confirmation bias – tend to seek out info that is consistent w/ our expectations and discount info that is inconsistent w/ those expectations o Limited data problem – tend to make inferences on the basis of very little info o Expectations problem – expectations tend to influence he way we interpret events o Baserate/comparison group problem – tend to overlook the fact that inferences about contingencies require some kind of comparison or standard o Pleasant truth problem – tend to believe things that make us feel good - What is science and how does it differ from naïve observation?  Science – the process of constructing, testing, and refining theories about natural phenomena through the use of systematic empirical observation o Difference: Use of systematic empirical observation - Can you accept a hypothesis?  Cannot accept a hypothesis – can gather support in favor of or reject - What are the four stages of the scientific method?  1) Review old theories, 2) generate new predictions, 3) conduct empirical observations, 4) test the theory against the observations Evaluating Research -What kinds of articles exist?  Review  Theory  Empirical (research report) -What is a variable?  Variable – is an observable indicator of any attribute that varies across people or things being studied -What are inductive and deductive research? How might they work together? 2 o Multiple operational definitions – multiple ways of measuring, then comparing them to see whether they seem to be measuring the same things -What are four interpretations that could apply to correlational results?  4 explanations exist for the relationship o Causation o Reverse causation o Reciprocal causation – they both cause each other o Third variable problem – a third variable, unaccounted for, variable causes the relationship between constructs -How do we get to “real-world” validity (e.g. the relationship between constructs in reality)? Ethics - What are ethical issues related to . . . Sampling? Deception? Coercion? Use of control groups? Literature review? Generating ideas? Analysis? Writing up your paper? - What are the three principles of the Belmont Report, and what do they mean?  Respect for persons – incorporates at least two ethical convictions o First, that individuals should be treated as autonomous agents o Second, that persons w/ diminished autonomy are entitled to protection o Avoid coercion o Solution: informed consent  Beneficence o Do no harm (or at least minimize it) o Maximize befit for participants o Solutions: risk/benefit analysis  But who conducts the analysis  Justice o Don’t unfairly sample from disadvantaged populations (economically or otherwise) o Control groups must be treated ethically o Solution: informed consent 5 - How do we “solve” the issues raised by these principles? o Informed consent – prospective participant should be informed about what will be required during the study, potential harm, and something about the purpose of the study - What factors affect someone’s ability to provide informed consent?  What if the study involves deception - What factors affect whether or not someone can be seen as participating voluntarily?  If the study involves deception - What is the difference between anonymity and confidentiality?  Anonymity – means that absolutely no identifying characteristics are recorded on the data and that it would be impossible for the researcher to figure out who contributed a given piece of data  Confidentiality – means that although the researcher can figure out whose data are whose, within certain legal limits, the researcher promises never to share that information - What are different types of deception? o Deception by omission – participants are not told the whole truth o Active deception – participants are actively misled about aspects of the exp. o Perseverance effects  Problem created by deception – participants will continue to believe the false feedback even after having been told about the deception  Another problem is participants’’ anger at having been deceived o Double deception or 2nd – order deception – refers to the once standard practice of experimenters telling participants that “the experiment is over” but then going on to collect additional data Reliability and Validity -What is definitional operationalism? o Definitional operationalism - the assumption that the construct is the operational definition -What is a nomological net? How is it useful? o Nomological net – theoretical network of construct-to-construct associations  Derived form relevant theory and stated at an abstract level  Ex. theory might define “social power as the ability to influence other people’s actions” -What are the components of an observed score, and how are they depicted in an equation? o Observed score = true score + error o Equation: X = T + e -What is associated with random error, and what is associated with systematic error? o Random error – nonsystematic and changing  Reflects nonsystematic, ever-changing influences on the score o Systematic error – due to a peripheral construct 6  Reflects influences from other constructs besides the desired one -What is reliability? What is validity? o Reliability – is about consistency. It is the extent to which a measure is free from random error  Product of random error o Validity – is about accuracy. It is the extent to which a measure reflects its underlying construct  Product of systematic error -How is perfect reliability obtained? Is this desirable? o Is not desirable – to maximize validity and reliability, we need balance o Multiple operational definitions are desired in order to capture the entire construct area -Describe the types of reliability o Test-retest reliability – is used to assess the consistency of a measure form one time to another  Test-retest correlation – provides an estimate of the measure’s reliability o Internal consistency reliability – is used to assess the consistency of results across items o Inter-rater reliability – is used to assess the degree to which different raters give consistent estimates of the same phenomenon -What are two ways of obtaining inter-item reliability? What are their features and drawbacks/strengths? o Internal consistency reliability – o Spit-half reliability – split the set of items in a measure in half, then a correlation between the two half-tests provides the basis for an estimate of reliability  Problem: the exact estimate depends on how the split of the items was achieved (odd vs. even, 1st half vs. last half) o Coefficient alpha – estimate derived from the correlations of each item w/ each other item and so does not rest on any arbitrary choice of ways to divide the items into two halves  Strength: can be shown to be equal to the average of all possible split-half reliabilities -Describe the types of construct validity o Face validity – does the test look like it measures what it claims to measure o Content validity – does the measure adequately sample the set of behaviors from the content domain o Convergent validity – does my measure correlate w/ measure of the same construct o Discriminant validity – does my measure correlate w/ measure of different constructs -How are reliability and validity related? -What is the multi-trait multi-method matrix? How is it used to determine reliability and validity? o Multitrait-multimethod (MTMM) matrix – table of correlation coefficients that enable us to simultaneously evaluate the convergent and discriminant validity of a construct o Validity - is based on an assessment of how much one method of measuring a construct agrees with other measures of the same or theoretically similar constructs and disagrees with measures of theoretically dissimilar constructs 7  Can be an hour to 2-3 hours o Disadvantage:  Interviewer effects – the interviewer’s expectations or personal characteristics can influence responses  Consistent w/ the idea that face-to-face situations create the strongest rapport-and hence the strongest tendency for respondents to give invalid, socially desirable answers to suit the interviewer’s expectations or desires  Studies have found larger interviewer effects in face-to-face than in telephone interviews  Primary disadvantage – their high cost, which depends heavily on the geographic coverage required by the study  High cost of travel, etc.  Telephone Interviews o Advantage  Often yield a high response rate, on average five percentage points lower than personal interviews  Higher response rates are attainable by telephone than by personal interviews in special situations such as urban-area samples  Average 10 to 15 percentage points higher than even the best conducted mail surveys, w/ a much larger advantage over more typical mail response rates  Do not impose strict limits on interview length, although they generally do not extend much over an hour  All the other advantages to face-to-face interviews, except visual aids, are also available in telephone interviews  Include: ability to correct misinterpretations, motivate the respondent, and probe for more detail when answers are vague  So, response rates are similar and data quality is comparable  Much lower cost  Supervision of interviewers – b/c telephone interviewers can all be in the same room w/ computer assisted software, they can have one supervisor to advise them all  Rules out a dishonest interviewer faking data  Allows errors in the questionnaire or interviewing procedures to be corrected immediately upon discovery, which is usually impossible w/ face-to-face interviews  Speed – questionnaire can be put together quickly and hundreds of interviews conducted almost overnight  Computer can check for valid data and signal the interviewer to recheck implausible responses, eliminating most coding and data-entry data 10  Biases deriving from question order can be reduced or eliminated by having a set of questions asked in a different, randomly selected order for each respondent; this would be difficult if not impossible for interviewers to do manually o Disadvantages:  Sampling – 1) not all households have telephones, and 2) that not all those who have phones have listings in the directory (b/c they are new arrivals to the area or b/c they choose to be unlisted)  First point is exaggerated – as most homes have telephones  Second point has been circumvented – by researchers using random digit dialing o Random digit dialing – a means of including in a telephone sample the significant number of homes for which the telephone number is unlisted  Interviewer effects are possible – although they’re usually smaller than w/ face-to-face interviews  Inability to use visual aids means some types of questions must be reworded for telephone use  Interviewer does not have visual cues that a misunderstanding has occurred  Direct Questioning via the Internet o Advantage:  Makes it possible to reach large and diverse populations at relatively low cost  Existence of internet communities consisting of like-minded individuals makes it possible to target very specific ad otherwise hard-to-reach populations  Permits participants to provide info at the time and place of their choosing  This sense of anonymity is particularly important in research on sensitive topics in marginalized populations o Disadvantages:  Response rate – often power, often impossible to precisely calculate a response rate in the usual sense b/ c it is not clear how many individuals could have responded but did not  Typically calculated from # of people who completed the survey and # of people who accessed the page  Drawbacks similar to those for mail surveys  There is no way to be certain that participants who respond to an online survey are the person they claim to be  Some individuals have multiple identities  Never clear under what conditions respondents complete the internet-based survey  Experience Sampling o Advantages  It generates detailed information about the experiences of respondents  Relatively short laps in time between the even of interest and participants’ responses to it 11  Considerably fewer participants are required in order to meet the sample size demands of the statistical methods used to formally test hypotheses  The large observations of each participant affords ample statistical power for hypothesis tests o Disadvantages:  Handheld computers typically used for data gathering in experience sampling research are expensive and fragile  Extreme reliance on the participant to generate the data according to the sometimes strict rules of sampling required to test the hypothesis of interest  The immediacy advantage afforded by experience sampling is compromised if participants do not respond to questions when they are signaled or when the events about which they are to provide information take place -What are indirect forms of measurement?  Collateral Reports – 3rd party responses to a questionnaire or interview; gathered in addition to rather than instead of info provided by participants; info provided by informant is in response to the same question posed to participant – therefore by gathering the same info from multiple resources, it is possible to detect biases that might contaminate self- reports  Observation – is an approach to measurement that does not rely at all on participants to report their preferences, opinions, or behaviors. Instead judges are trained by the researcher to detect and record observable indicators of the construct of interest  Physiological Monitoring- targets covert characteristics such as emotions, evaluations, and preferences o Psychophysiology – the study of the interplay of physiological systems and people’s thoughts, feelings, and behaviors o Physiological systems influence and are influenced by people’s experience of themselves and the world around them o It is possible to identify particular patterns of physiological activation that indicate constructs of interest to social scientists -What are some pros and cons of using observation, response times, behaviors, and archival data?  Collateral Reports o Advantages  Potential to overcome biases inherent in self-reports of constructs of interest to social scientists  Through strategic selection of informants, it is possible to use collateral reports to evaluate the veracity of self-reports as well as pinpoint sources of potential bias in self-reports o Disadvantages:  Drawback is the dilemma that must be resolved when the reports are not in agreement either w/ each other or w/ participants’ self-reports; resolution often requires additional research 12  Telephone, and increasingly the internet – are the method of choice for surveys of the general population that cover more than a local geographic area  Experience sampling – is unrivaled for research that requires rich, detailed info from individual respondents in natural settings Questionnaires and Scale Types -What types of content can scales or items cover?  Constructs of interest so social scientists might concern what research participants know (facts); what they think, expect, feel or prefer (beliefs and attitudes); or what they have done (behaviors)  Questionnaire generally includes questions referring to more than one of these categories, and at times a single question has aspects of more than one category -What are problems associated with each type of content?  Questions aimed at facts – memory problems or from response biases of various forms o Memory telescoping – is the tendency to recall events as more recent than their actual dates  Questions aimed at beliefs or attitudes – most difficult to type or write o There is always the possibility the most that respondents do not have an attitude b/c they never thought about he issue until the interviewer asked about it o Attitudes often are complex and multidimensional  A person might not have a single overall attitude toward something but might favor it in some circumstances and reject it in others o Thus expressed attitudes are dependent on details of question wording, question sequence, and interviewer effect to a greater extent than are responses involving facts, for instance  Questions aimed at behavior – the length of time elapsed since the behavior in question influences the accuracy of responses -What are four components of question wording that we need to adhere to?  4 components: simple, unbiased, avoid assumptions, and never be double barreled o simple – Avoid confusing words and grammar o unbiased – by providing equal alternatives and avoiding loaded words o assumptions – can include a “N/A” choice o double barreled- combines two separate ideas into a single question  Terminology – choice of specific terms to convey the question’s concepts o Guideline: terms should be exact, reflecting just what question content intended to mean; terms should be simple-comprehensible even to the least-educated respondents o Ambiguous or vague words are trouble; many commonly used terms are frequently misunderstood, biased words in question can produce biased answers  Question structure – 15 o Complex and lengthy sentences are particularly likely to be misunderstood by respondents, so questions should be short and simple o Questions should simplify the respondents’ task as much as possible o Key idea in the question should come last to avoid premature formulation of an answer o Qualifications and conditional clauses should come first  Expressing all alternatives o Provide all alternatives in the question -What is the funnel principle and why do we use it?  Funnel principle – general questions should come first, followed by increasingly specific and detailed questions, w/ the sequence “funneling down” to the most detailed questions at the end  Reduces bias and confusion -How does ordering affect response bias and how do we try to prevent bias?  Context effects – the effects of preceding questions on the response to later questions, pose the same sorts of issues as effects of question wording, although they seem to be less powerful -What is scaling and what is a response scale?  Scaling – is the assignment of scores to answers to a question so as to yield a measure of a construct  Response scale – is the range of possible answers to a given question -What are the four levels of measurement, and what are their properties? Be able to think up examples of each type.  Ratio, interval, ordinal, nominal  Nominal scales: each category has an arbitrary name rather than having numerical meaning; categories differ in quality, not quantity o Ex. gender, race  Ordinal scales: categories can be rank-ordered on a single continuum; the numerical differences between categories may not be equivalent o Ex. Ranking in running race; weight (body mass index categories) – 1=underweight, 2 =normal, 3 = overweight, 4= obese  Interval scales: numbers can be rank ordered on a single continuum; no true zero point; the relative distance between numbers is equivalent o And therefore you can conduct additions and subtractions o Ex. Farenheit scale; IQ  Ratio scales : numbers can be rank-ordered on a single continuum; o The relative distance between numbers are equivalent  Therefore you can perform additions and subtractions o There is a zero point  Therefore, you can perform multiplications and divisions o Ex. Weight – and 0lbs. means something theoretically 16  Summary o Nominal – numbers only provide category information o Ordinal – numbers provide ranking info o Interval – numbers provide ranking and distance info o Ratio – numbers provide ranking and distance info, and have a true zero point  Levels of measurement determines what you can do to and what you can do w/ your numbers o Higher levels of measurement mean you can do more w/ them, but less to them, statistically o Lower levels of measurement mean you can do more to them, but less w/ them statistically -What determines your level of measurement?  Level of measurement is both a choice and defined by your construct o Choice – decision of how things are measured o Construct – limiting factor -Why do we prefer multiple items, and what kinds of multiple-item measures are there?  More items = better reliability  Differential (Thurstone) scales o Each item represents known positions on an attitude scale associated w/ a scale value o Items represented randomly to which respondent agrees/disagrees o Score is average of scale values of items respondent agrees w/ o Respondents are assumed to only agree w/ a narrow range of items whose position is close to their own  Cumulative (Guttman) scales o Item means agreeing w/ everything below o Score is total number of items agreed w/  Summated (likert) scales o Set of items to which respondents indicate agree/disagree o Score developed by averaging responses o Advantages - Easy, few assumptions o Disadvantages – no item level data o Issues w/ it: reverse coding, anchor order, semantic differential  Reverse coding: ex. “I like the other members of my team” or “I do not like the other member of my team”  Anchor order – direction of strongly disagree to strongly agree (does it start on right or left)  Semantic differential – dislikeable to likeable or strongly agree to disagree  Assumption of unidimensionality  Difficulty in interpretation since people see these rarely  More often used to measure emotions rather than cognitions 17