Online Surveys: A Practical Guide for Researchers, Study notes of Industrial Technology

Download Online Surveys: A Practical Guide for Researchers and more Study notes Industrial Technology in PDF only on Docsity! 1 Introduction Low-cost computing and the rapid development of technology have created new environments for conducting survey research. Like all research methods, online survey research has benefits and drawbacks; the method works well for some research projects but is by no means appropri- ate for all research objectives. This book provides practical information for researchers who are considering using the Internet, mobile devices, and other technologies to conduct surveys. We will evaluate the advantages and disad- vantages of using digital surveys and offer guidelines for the creation and implementation of these surveys. The topics covered herein will be of interest to survey researchers in a wide variety of academic and professional settings who wish to evaluate their options for data collection and analysis. In this chapter, we begin by setting the context within which digital sur- veys are conducted. We review the research process generally, discuss con- cepts common to all surveys, and evaluate the conditions under which online surveys are optimal. The Research Process The research process typically begins with a question that needs an answer or a problem that must be solved. In the case of commissioned research, the questions will be provided to you at the start of the project. Researcher- initiated studies allow investigators more discretion in terms of the specifi- cation of the research goals and objectives. Before framing the goals and objectives of a particular project, it is useful to identify the purpose of the research. Social research projects can be classified into three categories: exploratory, descriptive, and explanatory research. An individual study can have multiple purposes or may be part of a program of research that spans two or all three purposes. C H A P T E R 1 2——Conducting Online Surveys Exploratory Research The goal of exploratory research is to formulate problems, clarify con- cepts, and form hypotheses. Exploration can begin with a literature search, a focus group discussion, or case studies. If a survey is conducted for explor- atory purposes, no attempt is made to examine a random sample of a popu- lation; rather, researchers conducting exploratory research usually look for individuals who are knowledgeable about a topic or process. Exploratory research typically seeks to create hypotheses rather than test them. Data from exploratory studies tends to be qualitative. Examples include brain- storming sessions, interviews with experts, and posting a short survey to a social networking website. Descriptive Research Descriptive studies have more guidelines. They describe people, prod- ucts, and situations. Descriptive studies usually have one or more guiding research questions but generally are not driven by structured research hypotheses. Because this type of research frequently aims to describe char- acteristics of populations based on data collected from samples, it often requires the use of a probability sampling technique, such as simple random sampling. Data from descriptive research may be qualitative or quantita- tive, and quantitative data presentations are normally limited to frequency distributions and summary statistics, such as averages. Customer satisfac- tion surveys, presidential approval polls, and class evaluation surveys are examples of descriptive projects. Explanatory Research The primary purpose of explanatory research is to explain why phenom- ena occur and to predict future occurrences. Explanatory studies are charac- terized by research hypotheses that specify the nature and direction of the relationships between or among variables being studied. Probability sampling is normally a requirement in explanatory research because the goal is often to generalize the results to the population from which the sample is selected. The data are quantitative and almost always require the use of a statistical test to establish the validity of the relationships. For example, explanatory survey research may investigate the factors that contribute to customer satis- faction and determine the relative weight of each factor, or seek to model the variables that lead to shopping cart abandonment. An exploratory survey posted to a social networking website may uncover the fact that an organization’s customers are unhappy. A descriptive study Chapter 1 Introduction——5 the questionnaire. In web-based surveys, the questionnaire resides on a website. Respondents visit the site and access the survey either by clicking on a hyperlink located somewhere on the page or by responding to a pop- up invitation to complete the questionnaire. Mobile surveys take advan- tage of the pervasiveness of cell phones by inviting respondents to participate in SMS (short message service) surveys or those designed to be answered using smartphone applications. Online surveys provide a wealth of opportunities and challenges for researchers. It is important that researchers recognize these opportunities and limitations so that they may make informed decisions when selecting a data collection method (see Table 1.1). It is doubtful that digital surveys will replace other modes of data collection; rather, these methods most likely will be used to complement existing techniques. Table 1.1 Comparison of Survey Methods Survey Type Advantages Disadvantages Postal mail • Low cost • Wide geographic reach • No interviewer bias • Anonymity allows for sensitive questions • Low response rate • Lengthy response period • Contingency questions not effective • Don’t know who is responding to the survey Telephone interview • Limited coverage bias • Fast response • Can ask complex questions • Wide geographic reach • Fewer land phone lines • Confusion with sales calls • Intrusive • Call screening Face-to-face interview • Good response rates • Can ask complex questions • Longer interviews may be tolerated • Limited geographic reach • Time-consuming • Expensive • Susceptible to interviewer bias • Sensitive topics difficult to explore Online • Can be low cost • Fast • Efficient • Contingency questions effective • Direct data entry • Wide geographic reach • Coverage bias • Reliance on software • Too many digital surveys, causing overload 6——Conducting Online Surveys Research Goals and Objectives After identifying the purpose of the research project and deciding that an online survey is an appropriate mode of data collection, it is next necessary to define the study’s goals and objectives. What are you trying to find out by conducting a survey? The objectives of the project determine whom you will survey and what you will ask them. If your objectives are unclear, the results probably will be unclear. Commit to these objectives in writing to help keep the survey focused. Make sure that you can tie in every questionnaire item to one or more of the survey’s objectives. A research goal is more broadly defined than the objectives. Goals are distinguished from objectives in that they are not necessarily measurable. For example, Goals: (A) Establish a solid foundation for Acme’s economic future. (B) Investigate the need for local networking between our industrial and service sectors. Objectives: To conduct a needs assessment survey of our suppliers to determine (A) current use of resources, (B) resource gaps, and the (C) need for outsourcing. Goals: (A) Determine the current state of breast cancer screening among Asian- American women. (B) Understand the psychological, economic, and cultural barriers to breast cancer screening among Asian-American women. (C) Create culturally appropriate, persuasive messages encouraging breast cancer screening among Asian-American women. Objectives: (A) to describe the incidence of breast cancer screening among Asian- American women in the Tri-Valley area; (B) to identify the psychological, economic, and cultural barriers preventing Asian-American women from getting screened for breast cancer; and (C) to test three messages designed to persuade Asian-American women to get screened for breast cancer. Guidelines for Writing Survey Objectives 1. Make objectives specific. To write specific objectives, it is useful to start with a general goal statement that begins with the word to followed by an action verb, such as describe, explain, explore, identify, investigate, gauge, Chapter 1 Introduction——7 measure, assess, or test—for example, “To investigate customer satisfaction levels.” A list of specific objectives can then be generated from this goal state- ment. Example 1.1 shows a general research goal followed by a list of objec- tives. The level of specificity in the objectives will guide the researcher when writing questionnaire items. Example 1.1 Goal: To assess credit union members’ satisfaction with the current services Objectives: To assess credit union members’ satisfaction regarding the following: A. The waiting time to speak to a customer service representative B. The loan application process C. Membership fees D. Telephone banking services E. Checking account services F. Savings account services G. Bilingual services Be sure that the survey objectives are in alignment with the format you choose to administer the survey; e-mail, website, and mobile surveys have innate coverage biases. For example, if your objective is to determine the sat- isfaction level of residents of the XYZ Retirement Village, then placing that survey on the village’s website may not target the correct set of respondents. Evidence indicates that people aged 65 and older access the Internet less often than younger people. The website of a retirement village may be viewed more often by family members of the residents than by the residents themselves. 2. Write measurable objectives. Whether an objective is measurable should be evaluated in light of the proposed survey format. Some objectives, such as those involving physiological variables, may very well be measurable but not in an e-mail or mobile survey. Example 1.2 presents some measurable digital survey objectives. Example 1.2 A. To assess students’ opinions about the proposed mascot B. To determine the percentage of citizens who are likely to vote for Candidate A C. To determine employees’ attitudes about the new delivery system D. To collect members’ ratings of the workshop 10——Conducting Online Surveys When Should an Online Survey Be Used? Digital surveys are not appropriate for every research project. What follows are some questions for researchers who are considering using technology for survey data collection. 1. What is the desired sample size, and how is the sample distributed geographically? If the sample size is fairly large and widely distributed geo- graphically, digital administration is a good option. E-mail and web-based surveys typically involve higher start-up costs than other methods but become cost-effective as the number of completed questionnaires increases. It is, therefore, most efficient to conduct an Internet survey when a large number of participants are to be contacted. Additionally, if the respondents are concentrated in a narrow geographic region, telephone or face-to-face interviews are feasible—albeit more expensive—options, which may result in a higher response rate, thereby reducing nonresponse error. 2. What are the time constraints? Digital surveys have the potential for fast turnaround. E-mail communication is fast, and text messages delivered to mobile phones are instantaneous, whereas postal mail must be physically delivered, obviously taking more time. Researchers should be cautioned, however, about making the general conclusion that digital surveys are always faster than other methods. It is important to consider the total time required to administer the survey; this may include an advance letter or e-mail message plus one or more follow-up reminders. Moreover, to achieve a sufficiently high response rate, a researcher may choose to keep a digital survey in the field for an extended period of time. 3. Does the questionnaire contain sensitive information? If so, anonym- ity might be a concern. Participants who are asked to respond to an e-mail questionnaire may lose their anonymity. If, however, participants are directed to a website to complete the questionnaire, some measure of ano- nymity can be promised. With regard to socially desirable responses, e-mail and website surveys are similar to self-administered postal mail question- naires. Because there is no interviewer on the phone or in person, respon- dents tend to feel safer providing honest answers in an online environment. 4. Who is your target? Clearly, digital surveys require that target respon- dents have access to the appropriate technology, either Internet access or a mobile phone. Physical, psychological, or financial limitations to technology may prohibit the use of digital surveys for certain populations. Digital surveys work well in closed populations where the potential respondents are known Chapter 1 Introduction——11 to have e-mail or Internet access—for example, a group of employees at a company, students at a university, or members of a professional association. They are less ideal when attempting to collect general public opinion data. 5. Is there a sampling frame? Responses to digital surveys are greatest when respondents are prenotified of the upcoming survey request. If you do not have an e-mail list, can one be created or obtained? Government agen- cies, businesses, and educational institutions maintain e-mail lists of their constituencies. Access to the appropriate list makes an e-mail or mobile survey a reasonable choice. Alternatives to using an organization’s list include (a) advertising the survey, perhaps on websites (such as Craig’s List), in promotional literature, or in an online community bulletin board; and (b) purchasing a list from a vendor. Using these alternatives deprives the researcher of the benefits of speed and efficiency that an existing list provides and introduces validity concerns related to the integrity of the sample. 6. Is a convenience sample sufficient, or is a probability sample necessary? To make inferences about underlying populations based on sample statistics, selecting a probability sample of respondents is necessary. Because there is no general population e-mail list and currently no Internet equivalent to telephone random digit dialing, researchers requiring data gathered from probability samples are best advised to consider other types of surveys. The nonprobability samples that can be selected quickly for Internet or mobile surveys work well for exploratory research or as part of a multimethod approach. 7. Would multimedia or interactive features enhance the questionnaire? Unlike paper questionnaires, electronic surveys may include streaming audio or video. Additionally, online questionnaires are arguably the most effective self-administered format for asking contingency questions. Web question- naires can be programmed to avoid logical inconsistencies in follow-up ques- tions. While programming errors may still exist, the automation of skip patterns eliminates the possibility of respondents answering the wrong questions—for example, participants who were not registered to vote respond- ing that they had selected Candidate A in a recent election. 8. Does the researcher have the technical ability to create an online sur- vey, or are funds available to hire someone? If the researcher does not have the technological knowledge or skills to create the digital survey, then either a consultant must be included in the budget or another method should be employed. There are presently hundreds of commercial services available to aid researchers in the creation, distribution, and analysis of digital surveys. These businesses vary greatly in the quality of customer service and data they provide as well as in their pricing structures. 12——Conducting Online Surveys Summary The field of online survey research is rapidly developing. Electronic methods of survey data collection have been touted as the wave of the future, with supporters citing speedy response, low cost, and easy fielding as major ben- efits, while detractors lob harsh criticism about the low response rates and claims that samples do not adequately represent populations. Although the particulars of the technology may be new, the controversy surrounding the research methodology is not. In fact, much of the current debate about digital surveys is reminiscent of a previous era when mail and telephone surveys were met with suspicion. More than 30 years ago, survey authority Don Dillman (1978) noted, Neither mail nor telephone has been considered anything more than a poor substitute for the much heralded face-to-face interview. Perhaps this view is justified, because the two methods had many deficiencies and problems. Surveys by mail typically elicited extremely low response rates, even with short questionnaires. . . . Further, it is not possible to reach many people with mail questionnaires; among those to whom questionnaires could be delivered, the best educated were far more likely to respond. Even completed question- naires left much to be desired. . . . It is not surprising, then, that users of the mail questionnaire treated response rates well below 50 percent as “acceptable” and explained away problems of data quality with disclaimers such as, “this is the best we can expect from a mail questionnaire.” (pp. 1–2) Substitute the word online in place of mail in the above quotation, and you will have a good indication of the contemporary discussion surrounding the use of online surveys. In the decades since Dillman wrote these words, a plethora of methodological research has resulted in techniques for mitigating the deficiencies inherent in the mail and telephone survey methods. In the decades to come, researchers will likely develop procedures to similarly com- pensate for the limitations of online surveys. Although there is still a great deal to learn about electronic surveys, the research to date provides valuable guidance to digital survey developers and forms the basis for our recommen- dations throughout this text. All surveys, whether conducted in person, by mail, e-mail, or mobile phone, have common features. All require clear objectives, well-crafted questionnaires, a sampling strategy, and so on. However, the idiosyncrasies of the technology associated with electronic surveys, with respect to planning, development, distribution, and analysis of the results, warrant detailed attention. Internet and mobile surveys are effective for gathering information quickly and relatively inexpensively from geographically dispersed participants. E-mail