Docsity
Docsity

Prepare for your exams
Prepare for your exams

Study with the several resources on Docsity


Earn points to download
Earn points to download

Earn points by helping other students or get them with a premium plan


Guidelines and tips
Guidelines and tips

Harvard's Guide to Ethical Human Research: Principles, IRB Review, and Consent, Study notes of History

Human Subjects ProtectionInstitutional Review Boards (IRBs)Research EthicsBiomedical Ethics

Guidance for student researchers at Harvard University on ethical obligations, federal regulations, and University policies regarding human subjects research. It covers the importance of respect for persons, beneficence, and justice, the role of Institutional Review Boards (IRBs), and elements of informed consent.

What you will learn

  • What are the key elements of informed consent in human subjects research?
  • What happens during the continuing review of previously approved human subjects research?
  • What role do Institutional Review Boards (IRBs) play in protecting human subjects in research?
  • What are the ethical principles that must be followed in research with human subjects?

Typology: Study notes

2021/2022

Uploaded on 09/27/2022

johnatan
johnatan 🇺🇸

4

(27)

29 documents

1 / 46

Toggle sidebar

Related documents


Partial preview of the text

Download Harvard's Guide to Ethical Human Research: Principles, IRB Review, and Consent and more Study notes History in PDF only on Docsity! INTRODUCTION TO HUMAN SUBJECTS RESEARCH PROTECTION A GUIDE FOR STUDENT RESEARCHERS 1 | P a g e Acknowledgement The Harvard University-Area IRB wishes to thank our colleagues at the University of California, Berkeley, the University of Michigan, the University of Southern California, and the Ohio State University for their willingness to share guidance materials developed for researchers at their own campuses, which served as the basis for this document. 4 | P a g e Introduction to Human Subjects Research Protection for Student Researchers It is a privilege and not a right to conduct research with human subjects. Responsible conduct of research with human volunteers in social, behavioral, or biomedical research requires commitment to the rights and welfare of participants and to the professional standards of the researcher’s academic discipline. This guide is designed to help student researchers at Harvard University understand their ethical obligations as a researcher, the federal regulations governing human subjects research, and the University’s policies and procedures associated with the conduct of research whether or not the activities fall under the oversight of the Harvard University-Area Institutional Review Board (HUA-IRB), also known as the Committee on the Use of Human Subjects (CUHS). The following key resources may be helpful to the student researcher: Harvard University • Harvard University-Area IRB (Committee on the Use of Human Subjects) Richard A. and Susan F. Smith Campus Center 1350 Massachusetts Ave, Suite 935 Cambridge, MA 02138 Phone: 617-496-2847 Email: cuhs@harvard.edu Website: http://cuhs.harvard.edu/ The IRB website includes contact information for IRB staff; guidance materials, including informed consent templates; a schedule of formal training sessions as well as departmental hours; application submission dues dates; and the schedule of meetings of the full IRB. • Electronic Submission, Tracking, and Reporting (ESTR) system Website: http://irb.harvard.edu ESTR is the IRB application tool for all human subjects research at Harvard University. • Undergraduate Research Training Program (URTP) The Undergraduate Research Training Program (URTP) is a comprehensive platform to create better prepared undergraduate researchers. The URTP is comprised of in-person training sessions offered several times throughout the academic calendar, a student-focused curriculum, and an online decision form that will assist students in determining whether their project requires IRB review. All undergraduate researchers who will be conducting research with human subjects (whether IRB review is required or not) must take part in the URTP. More information may be found here: https://cuhs.harvard.edu/urtp-portal • Training in the Ethical Conduct of Research Harvard policies require that all individuals who are involved in human subjects research complete training in the ethical conduct of research. This includes researchers and all study team members who have contact with human subjects or their identifiable data. Faculty sponsors of non-exempt research must also complete the training. More information may be found here: http://cuhs.harvard.edu/required-ethics-training KEY RESOURCES 5 | P a g e • Harvard Research Data Security Policy This policy is particularly focused on the protection of research data that are confidential by reason of applicable law and regulation, agreements covering the acquisition and use of the data, and University policies. http://vpr.harvard.edu/pages/harvard-research-data-security-policy Federal • US Department of Health and Human Services (HHS), Office of Human Research Protections (OHRP) OHRP is the federal agency that has regulatory oversight for research with human subjects. The OHRP website contains many useful resources related to the ethical conduct of research with human subjects and the regulations governing such research. http://www.hhs.gov/ohrp/about 6 | P a g e Chapter 1 Before You Begin Harvard University is responsible for ensuring that the rights and welfare of research participants, or human subjects, are adequately protected in research conducted by its faculty, staff, and students. Federal laws require this protection, and in order for the University to fulfill its responsibility, all research involving human subjects as defined by the federal regulations must receive appropriate review and approval. It is important to note that not all activities that involve people, their data, or specimens are covered by the regulations governing human subjects research and may not require review by an IRB. Please see Chapter 2 for an overview of what types of research require IRB review. The federal regulatory framework governing human subjects research is found in the US Department of Health and Human Services Policy for the Protection of Human Subjects (45 CFR 46). See https://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr-46/index.html. These regulations codify the key ethical principles found in the Belmont Report. See https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/index.html. These principles are: • Respect for persons • Beneficence • Justice In essence, the Belmont Report tells us that for research to be ethical, subjects must choose to participate voluntarily after being fully informed about the research study (respect for persons), that the benefits of the research out‐weigh the risks associated with the research (beneficence), and that the selection of subjects is equitable (justice). Other federal regulations may also apply to the conduct of human subjects research at Harvard University. These are: • Food and Drug Administration (FDA) (21 CFR 50; 56) regulations govern clinical research involving regulated drugs and medical devices. See http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRsearch.cfm?CFRPart=50 and http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRsearch.cfm?CFRPart=5. • Health Insurance Portability and Accountability Act (HIPAA) (46 CFR 164 Subpart E) regulations cover research involving the use of Protected Health Information (PHI). See http://www.hhs.gov/ocr/privacy/hipaa/administrative/privacyrule/index.html. See Appendix I for more information on the ethical and regulatory framework governing the conduct of human subjects research. WHY DOES RESEARCH WITH HUMAN SUBJECTS REQUIRE REVIEW? 9 | P a g e Chapter 2 How do I know if I am conducting human subjects research? Research projects meeting the regulatory definition of research with human subjects require review and approval by an IRB, or a determination that the research is exempt. Not all activities that involve people, their data, or specimens are covered by the regulations governing human subjects research and may not require review by an IRB. The questions that must be considered when determining whether IRB review and approval is required are 1) whether a project fits the regulatory definition of research (“regulated research”), and if so, 2) whether it involves human subjects. Let’s first start with the definition of research. While an activity may be considered research, it is important to highlight that not all research meets the threshold of “regulated research” requiring IRB review. The federal regulations have a very specific definition of what is considered regulated research that requires IRB review. Research generally does not include activities such as the practice of public health, medicine, counseling, or social work. Studies for internal management purposes such as program evaluation, quality assurance, or quality improvement are not research because the intent is not to draw conclusions beyond the activity or program being studied. IS IT RESEARCH? The federal regulations define research as “a systematic investigation, including research development, testing, and evaluation, designed to develop or contribute to generalizable knowledge” (45 CFR 46.102(l)). A systematic investigation is a study or examination that involves a methodical procedure and plan, is theoretically grounded, specifies a focused and well-defined research problem or question, is informed by the empirical findings of others, is analytically robust, and provides a detailed and complete description of data collection methods. Generalizable knowledge is information expressed in theories, principles, and statements of relationships that can be widely applied. A plan to publish findings or present at a professional meeting generally, but not always, indicates an intention to contribute to generalizable knowledge. 10 | P a g e **A note about class/educational “research” activities – Undergraduate class projects and research methods classes may involve data collection activities for training purposes that do not require IRB review and oversight because the intent is to teach methods, not to contribute to generalizable knowledge. The goal of this educational exercise is to prepare the student for future research (i.e., research that will be conducted in their future academic career, such as during graduate school) that *will* meet the federal regulatory definition of research with human subjects, with the understanding that the current research activity most likely will not rise to the level of needing IRB review. However, when the primary focus and initial intent of the class activities are to collect data to be used by students or other researchers beyond the classroom, thereby contributing to “generalizable knowledge,” IRB review may be needed. **A note about student internships – Students within many departments or schools of the University are involved in internships or practica. Some student practica/internships may include research activities that are designed to contribute to generalizable knowledge and, thus, involve research that requires IRB review. If your activity meets the federal regulatory definition of research, the next step is to determine whether your research involves human subjects. The federal regulations define a human subject as “a living individual about whom an investigator (whether professional or student) conducting research: (i) obtains information or biospecimens through intervention or interaction with the individual, and uses, studies, or analyzes the information or biospecimens; or (ii) obtains, uses, studies, analyzes, or generates identifiable private information or identifiable biospecimens” (45 CFR 46.102(e)(1)). • “Living individual” refers to data (information or specimens) collected from living subjects. For example, research using data from the 1880 Census would not be human subjects research. • “About whom” refers to the fact that the information collected must be personal information about an individual. For example, a survey that collects data about the activities of an organization is not human subjects research. • “Intervention” includes physical procedures and manipulations of the subject or the subject's environment for research purposes. For example, taking a saliva or blood sample from a subject or having a subject view a video would be considered a research intervention. • “Interaction” refers to communication between the researcher and the subject. This includes face‐to face, mail, internet and phone interactions, as well as other modes of communication. • “Individually identifiable” means the identity of the subject is or may be readily ascertained by the researcher or others. Research with a de‐identified data set is not research with human subjects because the data are not individually identifiable. • “Private information” includes information about behavior that occurs in a context in which an individual can reasonably expect that no observation or recording is taking place, or information that has been provided for specific purposes by an individual and which the individual can reasonably expect will not be made public. Examples of private information include medical or academic records or personal journals. DOES MY PROJECT INVOLVE HUMAN SUBJECTS? 11 | P a g e Harvard University has identified a number of project types that are “not research” and “not human subjects research” according to the federal regulations and, therefore, do not require review by the IRB: Not Research • case studies; • class activities/research methods classes; • journalism/documentary activities; • quality assurance/quality improvement/program evaluation activities; • oral history; and • standard public health surveillance or prevention activities. Not Human Subjects Research • secondary analysis of publicly available data sets or other de‐identified data sets that have been stripped of all identifiable information; and • research on organizations. If a formal determination is needed: An IRB application is not required for most types of “not research” according to the federal regulations. If you would like a formal “not research” determination from the IRB, or if you are not sure if your project requires review, you can submit a brief application via ESTR, the web‐based IRB application system. The IRB staff will issue a “not research” determination according to the federal regulations or will advise the researcher that the project does involve human subjects research and will recommend the submission of a standard application via ESTR. Undergraduate Research Training Program: The Undergraduate Research Training Program (URTP) is a comprehensive platform to create better prepared undergraduate researchers. The URTP is comprised of in- person training sessions offered several times throughout the academic calendar, a student- focused curriculum (the very one you are reading), and an online decision form that will assist students in determining whether their project requires IRB review (see previous description of the Decision Form). The URTP is required for all undergraduates conducting research with individuals, whether or not the activity rises to the level of requiring IRB review. The Decision Form is an online form administered through Qualtrics that guides the researcher through the various regulatory and institutional requirements for research. Definitions and examples are provided with each question. A decision about whether the researcher needs IRB review is provided. Each form is also reviewed by an IRB regulatory expert to ensure that the form was completed accurately and the correct decision was provided. WHAT IF I AM NOT SURE IF MY PROJECT INVOLVES HUMAN SUBJECTS? DO I NEED TO SUBMIT AN APPLICATION IF MY PROJECT DOES NOT INVOLVE RESEARCH WITH HUMAN SUBJECTS? 14 | P a g e d. Vulnerable participants are individuals who are likely to be susceptible to coercion or undue influence (e.g., students, subordinates, patients). Vulnerable populations also include individuals who cannot give informed consent because of limited autonomy (e.g., children, cognitively impaired, prisoners). e. Restrictions and/or special considerations may also apply where other characteristics render populations vulnerable: i. When recruiting children, both parents and their children must be involved in the recruitment process. Children are not eligible to participate in research without their parents’ permission. ii. Adults living in potentially coercive conditions – e.g., nursing home residents, half-way house residents. iii. People who have experienced or now have: 1. major injuries or acute or chronic disease; 2. disabilities that interfere with the quality of their lives; 3. homelessness; 4. undocumented status; or 5. stigmatized identity. f. Consider what participants will be asked to do, what will be done to them, or what information will be gathered: i. How frequently and over what time period will interviews, tests, etc., be conducted? Will there be breaks? ii. Where will research be conducted? If interviews will be conducted, how will interviewees be made comfortable? What privacy (if any) will be available? iii. Are interviews to be audio or video recorded? This should be disclosed ahead of time to participants and their agreement obtained as part of the consent process. iv. If recordings will be made, will these recordings be stored? Do you have plans for transcription? If you wish to have the option to use recordings in the future you should tell participants this and obtain their consent. g. Consider whether the study will involve either active deception or incomplete disclosure that is likely to significantly mislead participants. h. If so, what is the nature of the deception or incomplete disclosure? Is it likely to be significant to participants? If yes, is there another way to conduct the research that would not involve deception or incomplete disclosure, and, if so, why have you not chosen that alternative? i. What explanation for the deception or incomplete disclosure do you give to participants following their participation? Will participants be "debriefed" or receive information about the research project following its conclusion? Please refer to Chapter 7: Special Considerations for more information on deception and incomplete disclosure. j. Participation in research must always be informed and voluntary (not coerced or unduly influenced). These conditions are met through a consent process. k. Points to consider: i. How will you inform participants about your research and then obtain their consent (e.g., orally, in writing, in person, by phone, by email)? 3. Vulnerable participants 4. Procedures and activities 5. Informed consent 15 | P a g e ii. Will you ask participants to sign a written document – a consent form? iii. Whether the consent process includes a signed consent form or not, you should give participants a document that repeats the explanation of the research, identifies you, and provides contact information. iv. Consent language should be as simple and straightforward as possible, and appropriate for the level of literacy, education, etc. of the participants. v. Will language translation/interpretation be needed? Is there any language barrier that could affect the consent process? If so, be sure to address this and, if needed, make plans for use of translators and translated documents. vi. Will recruitment and consent documents be translated into foreign languages? Children under 18 need to have their parents’ permission in order to participate in research. In addition, they must themselves be asked to agree (“assent”) to participate. l. Consider how confidentiality will be protected: i. Will you use a key or code to identify participants? How will you securely store the information that links codes to identifying information (names, addresses, SSNs)? ii. Will the research data be collected and stored in a manner to keep it separate from the information (names, etc.) that uniquely identify participants? iii. For online studies, will IP addresses or other potentially identifying information be collected? What host site will be used (e.g., SurveyMonkey, iCommons, etc.)? Will those identifiers be removed from the data? If so, at what point, and if not, why must identifiers be retained? iv. Where will data be stored, who has access, and how will it be secured? v. Will research data be destroyed at the end of the study? If not, where and in what format and for how long will the data be stored? To what uses – research, public performance, archiving – might the data be put in future? Note: You should obtain participants' permission for possible future use of their data. vi. If there is a key code connecting participants' data to their identity, when will the link be destroyed? (Include this information during consent process.) m. Think about possible risks of harm to participants that might result from: i. the activities of the research –surveys, interviews, or activities you ask them to engage in; or ii. inadvertent disclosure of the data you will collect about participants. n. Risks can be psychological or emotional (e.g., participants are asked to recall or describe unusually troubling aspects of life); legal (e.g., participants report their illegal statuses or activities); social (e.g., participants are asked to disclose a stigmatized identity, activity, or status like poor grades or HIV status and those data are inadvertently revealed); financial (participants are asked to invest their own money, or disclose private identity information such as social security numbers or private financial information and the research data are inadvertently revealed); and/or physical (activity involves strenuous activity, travel, ingestion of substances, etc.). o. Responsible research requires that risks be minimized, be reasonable in relation to any benefits that might occur, and be clearly communicated to research participants. p. Researchers should take measures to protect participant privacy (e.g., are questions tailored to the research problem only, so participants are not asked to provide unnecessary information?) 6. Confidentiality 7. Risks 16 | P a g e Risks no greater than those that research participants would encounter in their everyday lives are considered minimal risks. The following examples illustrate risks that are potentially greater than minimal risk and strategies to reduce them: Revealing one’s personal experiences of domestic violence would not be, for most people, a normal or everyday occurrence. Most people keep this information private. Revealing it could bring emotional risks (if in the recollection and recounting, painful feelings were aroused); social risks (if the information were revealed to others); and perhaps legal risks (if the individual were a perpetrator or if children were put at risk by the violence, even if the participant were not the perpetrator). Appropriate risk-reduction strategies could include: a. carefully planning the interview ahead of time, and obtaining training and advice in techniques for emotionally sensitive and ethical interviewing; b. preparing a list of appropriate counseling resources to have ready for participants if needed; c. designing and rigorously adhering to methods to protect the confidentiality of data. d. Standard methods include passwords, encryption, and storing research data separately from a key-code linking personal identifiers (e.g., names) from id codes (e.g., numbers). Discussing political organizing and one’s political views in a corrupt and violent political environment might be a normal activity for activists, in that they talk to each other, but it still would not be routine for them to engage in such discussions with a researcher. The possible risks could be reputational (if their colleagues disapproved of them talking to a researcher, or if the data were inadvertently revealed outside the research); legal; and even physical (if disclosure might lead to physical reprisals). Appropriate risk-reduction strategies could include: a. having a compelling and specific justification for questions that would elicit risk-inducing information and avoiding risk-generating questions; b. seeking training in techniques for ethical interviewing in politically sensitive contexts; c. designing and rigorously adhering to methods to protect the confidentiality of data, which probably would include avoiding identifiable information as far as possible, using encryption as well as passwords (for audio recordings as well as written data), and, destroying such information (e.g., by removing it from computers) as rapidly as possible. See the Harvard University Research Data Security policy for information on how to protect your data - http://vpr.harvard.edu/pages/harvard-research-data-security-policy Running a few yards might be a normal physical activity, but running to the point of complete exhaustion would not be for most people; therefore, if it were a research activity, i t would carry research risks beyond what is encountered in daily life. Appropriate risk-reduction strategies could include: a. recruiting only conditioned athletes for whom running to exhaustion might be fairly routine; and/or b. requiring a medical exam ahead of time. 19 | P a g e described in Appendix II and issues a determination. The IRB staff communicates the outcome of the review via the ESTR system. The reviewer may approve the research, require modifications to the research before approval, or refer the submission to the full board for further review. (See Appendix III for a description of the types of determinations that the reviewer may issue for a proposal.) What projects require review by the full (convened) board? Projects that involve MORE THAN MINIMAL RISK or that do not fit into one of the specified expedited review categories must be reviewed by the full board at a convened meeting. Examples of other projects that may require review by the full board include: • projects posing no more than minimal risk to participants but that involve vulnerable populations, sensitive topics, or complex research designs that would benefit from a review by the breadth of expertise represented at the full board. This includes studies that collect sensitive data that may require an NIH Certificate of Confidentiality (CoC) to protect subject data from compelled disclosure. • projects referred to the full board by an expedited reviewer. For example, a reviewer may seek guidance from the full board in determining whether a study meets the regulatory definition of minimal risk or when the scientific question posed by the PI exceeds the expertise of the available expediting reviewers. • research involving prisoners. How is research reviewed by the full board? A designated IRB staff member conducts the initial administrative review of each application identified as requiring full board review. The staff reviewer may return the application to the researcher for changes if the application and supporting materials are incomplete or unclear. Once complete, the application is added to the agenda for the next available meeting of the IRB. A primary IRB reviewer and a secondary reviewer (if needed) are assigned to present the proposed research to the board at the convened meeting. All members receive a copy of the submission materials via ESTR. Consultants may also be invited to assist in the review of research where additional expertise is necessary. In some circumstances, the researcher may be asked to attend the board meeting to respond to the board’s questions. After the meeting, the board’s decision will be communicated to researchers via ESTR. The board may vote to approve the research, determine that there are modifications required to secure approval, or defer action on the application if significant revisions are required. See Appendix III for a description of the types of determinations the IRB may issue for a study. Researchers may not begin research activities until documentation of IRB approval is received via ESTR. FULL (CONVENED) REVIEW 20 | P a g e Chapter 5 What is Informed Consent? Informed consent is the process of telling potential subjects about the key elements of a research study and what their participation will involve. The subjects in the study must participate willingly after having been adequately informed about the research. If the subjects are from a vulnerable population, such as prisoners or children, additional protections are required. Consent documents must be clearly written and understandable to subjects. The informed consent document is an information tool, rather than a legal contract. • Use non‐technical language (comparable to the language in a newspaper or general circulation magazine), free from scientific, legal, or academic jargon. • Be aware of the reading level of your subject population. In general, aim for an 8th grade reading level. • Don’t use the “first person” in the body of the consent (I understand, I agree, etc.), before you have actually explained the research. Use the “second person” to tell the subject about your project. • Regulations preclude the use of exculpatory language that implies that the subject is waiving any legal rights by agreeing to participate. Informed consent must begin with a concise and focused presentation of the key information that is most likely to assist a prospective subject or legally authorized representative in understanding the reasons why one might or might not want to participate in the research. This part of the informed consent must be organized and presented in a way that facilitates comprehension. The regulations further identify the following required elements of informed consent (45 CFR 46.116). • a statement that the activity is research and describing its purpose; • a description of procedures involved in the research, including a statement of the length of time the subject is expected to participate (for example, a one-hour survey, three one-hour interviews over the course of three months); • a description of all reasonably foreseeable risks and discomforts to the subject (includes possible psychological, social, or economic harm, discomfort, or inconvenience, in addition to physical risks); • a description of benefits of the research to the individual subject or to society in general; • description of plans to protect the confidentiality of records identifying the subject (if appropriate); • a disclosure of alternative procedures or treatment available should a subject decide not to participate in the research (rarely applies to social and behavioral research); • for projects involving more than minimal risk to participants, an explanation regarding whether compensation or medical treatment is available if injury occurs (rarely applies to social and behavioral research); ELEMENTS OF INFORMED CONSENT 21 | P a g e • persons (PI and Faculty Advisor) to contact for answers to questions or in the event of a research‐ related injury or emergency; • contact information for the IRB for answers to questions about the subject’s rights as a research participant; • statement that participation is voluntary and that refusal to participate or discontinuing participation will not result in any consequences or any loss of benefits that the person is otherwise entitled to receive; • statement regarding whether participant information or biospecimens will be used for future research studies upon removal of identifiers. Other optional elements as appropriate to the research project in part include: • payment for participation (include information regarding payment if the subjects ends participation before completing the research); • for surveys and interviews, a statement that the subject may skip any question they don’t wish to answer, if possible. See the IRB website at http://cuhs.harvard.edu/ for informed consent templates, sample consent documents, and detailed guidance materials. Informed consent is documented by the use of a written consent form that is signed by the subject or legally authorized representative. A copy of the consent document should be provided to the person signing it ( 45 CFR 46.117). Regulations allow for some exceptions to the requirement for documented (signed) informed consent as well as allowing for waiving informed consent. Waiver of Documentation of Informed Consent According to 45 CFR 46.117, an IRB may waive the requirement for the researcher to obtain a signed consent document if it finds either: • That the only record linking the subject and the research would be the consent document and the principal risk would be potential harm resulting from a breach of confidentiality, OR • That the research presents no more than minimal risk of harm to subjects and involves no procedures for which written consent is normally required outside of the research context, OR • If the subjects or legally authorized representatives are members of a distinct cultural group or community in which signing forms is not the norm, that the research presents no more than minimal risk of harm to subjects and provided there is an appropriate alternative mechanism for documenting that informed consent was obtained. For projects on sensitive topics, use of a waiver of documentation may minimize the risk to participants by making it impossible to link them to the research project. Subjects are still provided with all of the information required for informed consent, either in written or oral form, but no signed consent document is obtained. More commonly, the IRB waives the requirement for documented informed consent for minimal risk projects For projects involving research with children, researchers must obtain both the consent (permission) of one or both parent(s) as well as assent of the minor child depending on the design of the study. Projects involving participants who are cognitively impaired may require consent from a legally authorized representative (LAR). THE INFORMED CONSENT PROCESS 24 | P a g e Please contact the IRB for guidance if one of these events occurs during your research. By submitting a study closure, the researcher confirms that the study is finished and that there will be no further interaction with subjects or their data. Once the IRB receives and acknowledges the study closure, the study is closed in the ESTR system. Note: If the researcher wishes to enroll new subjects for the study, or otherwise engage human subjects in research after the study is closed, a new application must be submitted in ESTR. Therefore, a researcher should only close a study when they are no longer enrolling new subjects, using research interventions on existing subjects, collecting data (including follow‐up data), or performing analysis of identified data as part of the approved study. WHAT DO I DO WHEN I HAVE COMPLETED MY STUDY? 25 | P a g e Chapter 7 Special Considerations Projects that involve collaboration with non‐Harvard University researchers or non‐Harvard University research sites that are “engaged” in the research may require additional external IRB approvals or inter‐institutional agreements before a research project can receive final approval. An institution is engaged in research when its employees or agents intervene or interact with research subjects, including obtaining informed consent, or obtain individually identifiable private information for research purposes. See http://www.hhs.gov/ohrp/humansubjects/guidance/engage08.html. In social and behavioral research, breach of confidentiality is a serious risk posed to participants. Rigorous data security is a key element of protecting subject data from an accidental or malicious breach. Data security includes a plan to manage the physical documentation associated with the project, such as paper surveys, signed consent forms, or documents that contain contact information for subjects to ensure that those materials are not lost or accessed inadvertently by an unauthorized person. Increasingly important is the management of electronic data on desktops or servers as well as on mobile devices such as laptops and flash drives. See the Harvard University Research Data Security policy for information on how to protect your data: http://vpr.harvard.edu/pages/harvard-research-data-security-policy Research conducted outside the United States may create additional challenges for the student researcher and the IRB. Cultural, economic, or political conditions of the host country may alter the risk for participants compared to the same research conducted within the U.S. Other countries and institutions within those countries may have Institutional Review Boards, Ethics Committees, or other research oversight bodies which require review of the research before it can be conducted in that country. Conversely, some may have no mechanism for ethics review of social and behavioral research. Except for research which is federally funded and the international site is engaged, the regulatory authority of the 2018 Rule does not cover research outside the U.S.; therefore, the IRB must ensure that equivalent protections for human subjects participating in research are in place. In its review of your application, the IRB will consider the following information: • description of where the research will be conducted (including geographic location and specific performance sites, where applicable). Note: In some areas, government–issue research visas are required; • information about the local research context, including the current economic, cultural, political, or religious conditions of the area that may affect the conduct of the research, and a COLLABORATION WITH RESEARCHERS OUTSIDE HARVARD UNIVERSITY DATA SECURITY INTERNATIONAL STUDIES 26 | P a g e description of the researcher’s personal experience conducting research (or studying or residing) in the region; • the language(s) in which consent will be sought from participants and the research will be conducted, as well as whether the researcher is fluent in this language or whether an interpreter will be required. If an interpreter will be used, it should be clear what limitations or risks, if any, this might present for participants, as well as how these potential problems will be overcome or minimized; • a description of the informed consent process as appropriate for the culture; • copies of translated study documents (recruitment materials, informed consent, study instruments); • any benefits to the local community that will remain in the community once the research is complete; • if compensation is being offered, a description of its appropriateness for the setting; • procedures for data security and storage in the local setting and for transfer of data and/or specimens to Harvard University; and • a copy of local IRB or equivalent ethics committee approval, where applicable. Depending on the location, this may take the form of a letter of approval from an IRB or research ethics committee, local university department sponsoring the research, institutional oversight committee, or an indigenous council. If the research is federally funded, check with the IRB for other regulatory requirements. If you are traveling to an international setting for your research, submit your IRB application well in advance of your planned travel date. This is particularly crucial for projects that may involve more than minimal risk to participants that will require full board review. See the CUHS website (http://cuhs.harvard.edu) for meeting submission dates. For those research studies that are determined to meet the criteria for expedited or full (convened) IRB review, it will also be necessary for the Office of the Vice Provost for Research (OVPR) to review the study. More information on this Provostial review may be found on the OVPR website http://vpr.harvard.edu/pages/provost-criteria-review as well as in Appendix VI: Provostial Review Process. Research conducted in primary and secondary schools, colleges, and universities that receive U.S. Department of Education funds may be subject to additional federal regulation. Schools that grant access to researchers may also impose requirements, such as district approvals or informed consent processes that would not be required by the IRB. The Family Educational Rights and Privacy Act (FERPA) (34 CFR Part 99) http://www.ed.gov/policy/gen/guid/fpco/ferpa/index.html FERPA applies to research involving student education records for any institution receiving U.S. Department of Education funding, meaning that it applies to most public and private K‐12 schools as well as most public and private colleges and universities. Access to identifiable student records requires written permission from the parent (for minors) or from the adult student unless the research is being conducted by the researcher on behalf of the school. The Protection of Pupil Rights Amendment (PPRA) (34 CFR Part 98) http://www.ed.gov/policy/gen/guid/fpco/ppra/index.html The PPRA, created by the No Child Left Behind Act, applies to survey research conducted in elementary and secondary schools receiving funds under U.S. Department of Education programs. The provisions of the PPRA apply to surveys that involve specific sensitive survey topics. The PPRA includes requirements for parental RESEARCH IN SCHOOLS 29 | P a g e Appendix I Core Ethical Principles and Regulatory Framework Ethical Framework Nuremburg Code The history of the ethical regulations in human subjects research began in the 1940s with the Nuremberg Code. The Nuremberg Code was developed following the Nuremberg Military Tribunal which judged human experimentation conducted by the Nazis. The Code encompasses many of the basic principles governing the ethical conduct of human subjects research today. The Nuremberg Code states that “the voluntary consent of the human subject is absolutely essential” and it further explains the details implied by this requirement: capacity of participants to consent, participants’ rights to participate or not, freedom from coercion, no penalty for withdrawal, and comprehension of the risks and benefits involved. Declaration of Helsinki In 1964, the World Medical Association established recommendations to guide medical doctors in biomedical research involving human subjects. The Declaration governs international research ethics and defines rules for "research combined with clinical care" and "non‐therapeutic research." The Declaration of Helsinki is the basis for Good Clinical Practices (GCP) used today. Issues addressed in the Declaration of Helsinki include: • research involving medical interventions with humans should be based on the results from laboratory and animal experimentation; • research protocols should be reviewed by an independent committee prior to initiation; • informed consent from research participants is necessary; • research should be conducted by medically/scientifically qualified individuals; and • risks should not exceed benefits. The Belmont Report In 1978, the National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research created “The Belmont Report: Ethical Principles and Guidelines for the Protection of Human Subjects of Research.” The Belmont Report sets forth the basic principles governing the ethical conduct of research involving human subjects. The Belmont Report encompasses three key principles: respect for persons (autonomy), beneficence, and justice. 30 | P a g e Principle Application Respect for Persons Respect for persons requires that protocols (including the informed consent process) be designed to promote personal capacity to consider alternatives, make choices, and act without undue influence or interference of others. The principle is reflected in requirements that legally effective informed consent be obtained, unless specific requirements for waiver of informed consent are met and appropriately documented; and that subjects with diminished capacity and others who are vulnerable to coercion or undue influence receive special protection or consideration. Beneficence Beneficence entails an obligation to protect individuals from harm. The principle can be expressed in two general rules: (1) do no harm; and (2) protect from harm by maximizing possible benefits and minimizing possible risks of harm. It is reflected in a requirement that principal investigators design and IRBs approve protocols only under circumstances where the benefits to the subjects and/or the importance of the knowledge to be gained justify the risks to the subjects sufficiently to warrant a decision to allow the subjects to accept those risks. Justice Justice requires fairness in distribution of burdens and benefits. The principle often is expressed in terms of treating persons of similar circumstances or characteristics similarly. It is reflected in the requirement that selection of subjects is equitable and is representative of the group that is intended to benefit from the research. More information can be found at https://www.hhs.gov/ohrp/regulations-and-policy/belmont-report/index.html. Regulatory Framework Governing Human Subjects Research Since the release of the Belmont Report, the federal government has codified the protection of the rights and welfare of human subjects by establishing regulatory codes and regulations. • Federal Policy for the Protection of Human Subjects (45 CFR 46) In 1981, the Department of Health and Human Services codified the Policy for the Protection of Human Subject (45 CFR 46). These regulations, called the “Common Rule,” provided for the basic foundation of the Institutional Review Boards. This federal policy was updated most recently in January 2019, renamed the “New Rule” or “2018 Rule.” More information can be found at https://www.hhs.gov/ohrp/regulations-and-policy/regulations/index.html. • U.S. Food and Drug Administration Regulations The U.S. Food and Drug Administration, under the Department of Health and Human Services, also provides guidance for Institutional Review Boards. FDA regulations differ from the 2018 Rule in some ways as they are intended to regulate research involving drugs, devices, and biologics. See http://irb.jhmi.edu/Guidelines/FDAvsOHRP.html. More information can be found at http://www.fda.gov/oc/ohrt/irbs/. • Health Insurance Portability and Accountability Act (HIPAA)/Privacy Rule The Health Insurance Portability and Accountability Act (HIPAA) is a federal law that generally prohibits health care providers (such as physicians or other health care practitioners, hospitals, nursing facilities, and clinics) from using or disclosing "protected health information" (PHI) without written authorization from the patient. The full text of the Privacy Rule can be found at the HIPAA privacy website of the Office for Civil Rights (OCR): http://www.hhs.gov/ocr/hipaa. 31 | P a g e Appendix II Regulatory Elements of IRB Review When the IRB reviews a non-exempt research protocol, it must make eight specific regulatory determinations in order to grant approval. These determinations find their basis in the ethical principles of the Belmont Report and codified in the Common Rule: respect for persons, beneficence, and justice. Minimal risk means that the probability and magnitude of harm and discomfort anticipated in the proposed research are not greater, in and of themselves, than those ordinarily encountered in daily life or during the performance of routine physical or psychological examination or tests. Types of harm include economic, legal, physical, psychological, and social. Each type of harm may occur in social research either to participants or to people not directly involved in the research, such as family members. Risks In social and behavioral projects, primary risks to subjects tend to be related to psychological distress or harms resulting from a breach of confidentiality. Subjects may feel stress caused by the research questions or procedures. Questions raise painful memories or unresolved issues. Questions about at‐ risk behaviors may cause embarrassment, feelings of guilt, or result in legal liability when that behavior is generally illegal or socially unacceptable. Most psychological risks are minimal and transitory, but researchers must be aware of the potential for serious psychological harm. A breach of confidentiality may be a significant risk to participants in social and behavioral research. If confidentiality of research data is not maintained, a participant might experience risks to reputation, employment, financial standing or insurance coverage. Information about subjects' activities may place them at risk of legal action. For example, if subjects divulge information about their participation in illegal or stigmatizing activities, any disclosure of that information could place the subjects at risk of significant harm. The kind and level of risk is determined by context. For example, research regarding political activism in some countries may put subjects in serious jeopardy, while it would not in other countries. In many cases, risk to privacy/confidentiality can be eliminated or reduced by careful procedures for ensuring confidentiality. Psychological support and referrals can be built into studies when emotional distress may be an outcome. The referrals can be made via an information card or debriefing sheet provided to the subject by the study team. Consent forms that describe the kinds of questions the researcher will ask allow participants to choose whether they wish to divulge certain types of information or participate in the study at all. 1. Risks to subjects are minimized 2. Risks to subjects are reasonable in relation to anticipated benefits, if any 34 | P a g e While any individual who fits the above category can be considered vulnerable, federal regulations offer additional protections to three groups of people: Prisoners are considered to be vulnerable in that their incarceration which could affect their ability to make a truly voluntary and non-coerced decision on whether or not to participate as subjects in research (http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr4 6.htm#subpartc). Research projects involving prisoners as research subjects typically require review by the full IRB. Pregnant women, fetuses, and neonates may be at a greater risk than others due to their physical status. Special regulatory protections, however, are geared toward medical research rather than social/behavioral research. (http://www.hhs.gov/ohrp/humansubje cts/guidance/45cfr46.htm#subpartb) Children are considered vulnerable because they may not be able to completely understand the information presented about a study. (http://www.hhs.gov/ohrp/humansubjects/guidanc e/45cfr46.htm#subpartd) 35 | P a g e Appendix III IRB Determinations After completing its review of an application, the IRB will notify researchers of one of the following actions: Approved – The IRB approves a research proposal when the application is complete and the IRB has determined that the study has met the regulatory criteria for approval. Once the approval determination is issued via ESTR, the research may be conducted according to approved procedures and parameters. Modifications Required to Secure Approval – The IRB makes the determination of modifications required to secure approval when the proposal meets the regulatory criteria for approval but needs specified changes to the protocol, informed consent document(s), or other supporting materials prior to final approval. Such changes must require no more than the simple concurrence of the researcher. The researcher is notified of the study outcome via an ESTR Modifications Required to Secure Approval notice and is provided with detailed instructions regarding required changes to the application or study materials that must be completed before the application can receive final approval. Once all contingencies are met, the IRB will issue an approval notice via ESTR. For initial submissions, no research may be conducted until final approval is released by the IRB. Action Deferred – The IRB full board may vote to defer action on an application when a significant action on the part of the researcher is required before the IRB can consider approval or disapproval of the research. Deferred applications are found to have major deficiencies, such as incomplete procedures and documentation, or major ethical issues, including unreasonable risk to subjects that make it impossible for the IRB to approve the research as proposed. The application is returned to the researcher via a Deferred notice within ESTR that details the issues that must be addressed in the application/materials before it can be reconsidered by the IRB. Upon revision of the application and resubmission to the IRB, the study will be rescheduled for review by the full IRB. Disapproval – The IRB full board may vote to disapprove an application to conduct human subjects research when it determines that the study design does not provide, and is unlikely to be modified to provide, adequate protection to subjects. Disapproval of an application usually follows several attempts by the researcher in conjunction with the efforts of the IRB to modify the study design to afford protection to the subjects. This action can only be taken by the full board at a convened meeting. The researcher will be sent a rationale for the disapproval and may ask that the IRB reconsider the disapproval. 36 | P a g e Appendix IV Categories of Exempt Research EXEMPTION #1: Research, conducted in established or commonly accepted educational settings, that specifically involves normal educational practices that are not likely to adversely impact students' opportunity to learn required educational content or the assessment of educators who provide instruction. This includes most research on regular and special education instructional strategies, and research on the effectiveness of or the comparison among instructional techniques, curricula, or classroom management methods. EXEMPTION #2: Research that only includes interactions involving educational tests (cognitive, diagnostic, aptitude, achievement), survey procedures, interview procedures, or observation of public behavior (including visual or auditory recording) if at least one of the following criteria is met: (i) The information obtained is recorded by the investigator in such a manner that the identity of the human subjects cannot readily be ascertained, directly or through identifiers linked to the subjects; (ii) Any disclosure of the human subjects' responses outside the research would not reasonably place the subjects at risk of criminal or civil liability or be damaging to the subjects' financial standing, employability, educational advancement, or reputation; or (iii) The information obtained is recorded by the investigator in such a manner that the identity of the human subjects can readily be ascertained, directly or through identifiers linked to the subjects, and an IRB conducts a limited IRB review. See more below. EXEMPTION #3: Research involving benign behavioral interventions in conjunction with the collection of information from an adult subject through verbal or written responses (including data entry) or audiovisual recording if the subject prospectively agrees to the intervention and information collection and at least one of the following criteria is met: (i) The information obtained is recorded by the investigator in such a manner that the identity of the human subjects cannot readily be ascertained, directly or through identifiers linked to the subjects; (ii) Any disclosure of the human subjects' responses outside the research would not reasonably place the subjects at risk of criminal or civil liability or be damaging to the subjects' financial standing, employability, educational advancement, or reputation; or (iii) The information obtained is recorded by the investigator in such a manner that the identity of the human subjects can readily be ascertained, directly or through identifiers linked to the subjects, and an IRB conducts a limited IRB review. See more below. For the purpose of this provision, benign behavioral interventions are brief in duration, harmless, painless, not physically invasive, not likely to have a significant adverse lasting impact on the subjects, and the investigator has no reason to think the subjects will find the interventions offensive or embarrassing. Provided all such criteria are met, examples of such benign behavioral interventions would include having the subjects play an online game, having them solve puzzles under various noise conditions, or having them decide how to allocate a nominal amount of received cash between themselves and someone else. If the research involves deceiving the subjects regarding the nature or purposes of the research, this exemption is not applicable unless the subject authorizes the deception through a prospective agreement to participate in research in 39 | P a g e 5. Research involving materials (data, documents, records, or specimens) that have been collected, or will be collected solely for nonresearch purposes (such as medical treatment or diagnosis). (NOTE: Some research in this category may be exempt from the HHS regulations for the protection of human subjects. 45 CFR 46.101(b) (4). This listing refers only to research that is not exempt.) 6. Collection of data from voice, video, digital, or image recordings made for research purposes. 7. Research on individual or group characteristics or behavior (including, but not limited to, research on perception, cognition, motivation, identity, language, communication, cultural beliefs or practices, and social behavior) or research employing survey, interview, oral history, focus group, program evaluation, human factors evaluation, or quality assurance methodologies. (NOTE: Some research in this category may be exempt from the HHS regulations for the protection of human subjects. 45 CFR 46.101(b) (2) and (b) (3). This listing refers only to research that is not exempt.) 8. Continuing review of research previously approved by the convened IRB as follows: (a) where (i) the research is permanently closed to the enrollment of new subjects; (ii) all subjects have completed all research‐ related interventions; and (iii) the research remains active only for long‐term follow‐up of subjects; or (b) where no subjects have been enrolled and no additional risks have been identified; or (c) where the remaining research activities are limited to data analysis. 9. Continuing review of research, not conducted under an investigational new drug application or investigational device exemption where categories two (2) through eight (8) do not apply but the IRB has determined and documented at a convened meeting that the research involves no greater than minimal risk and no additional risks have been identified. 40 | P a g e Appendix VI Provostial Review Process The Provostial Review is a review of research proposals conducted by the Provost’s Office at Harvard University. The Provostial Review process is designed to review projects that pose management challenges and/or reputational risks beyond those routinely covered by the IRB or other review committees. The Provostial Review process also considers the question of whether the proposed research project is within the research and academic mission of the University, as opposed to advocacy or consultancy. There are ten criteria that can trigger a Provostial Review. These criteria are described under the “Criteria and Procedures for Provost's Review of New Projects or Grants” on the Office of the Vice Provost for Research (OVPR) website: http://vpr.harvard.edu/pages/provost-criteria-review. For undergraduate researchers, the most common criterion triggering a Provostial Review is #4: “The project is international and involves human subjects research that requires IRB expedited or full review.” The Provostial Review is undertaken by a team comprised of staff from OVPR, the Office of Sponsored Programs (OSP), and the Office of the Vice Provost for International Affairs. The Provostial Review team relies on the IRB to address human subject protection risks (e.g. risks-benefits to the participants, adequacy of consent forms, protection of privacy, etc.) and focuses its review on other risks. In general, the Provostial Review team reviews the same documents submitted to the IRB (e.g. the protocol, Informed Consents, and Questionnaires). Therefore, the protocol should be complete and detailed, with a full explanation of the researcher’s planned study design to assist with, and accelerate, both the IRB and the Provostial Review Process. The protocol should clearly describe the potential areas of risk in the planned project, and provide plans to mitigate those risks. The following provides a brief summary of the most common risks evaluated in the Provostial Review; researchers should carefully consider if their projects include any of these risks, and if so, provide details on their plans to manage those risk areas. Note: While the next sections will be focused on international research projects, the risks below are not limited to locations outside the U.S. Students planning projects in the United States should also consider these risks and how to mitigate them. 1) Risk to the student Harvard University undergraduates who wish to conduct international research must follow the Harvard College International Travel Policy (https://www.globalsupport.harvard.edu/travel-tools/forms-policies/harvard- college). This policy prohibits travel to high-risk regions, as defined by Harvard Global Support Services (GSS), and restricts travel to elevated-risk regions. A complete listing of GSS risk ratings can be found at https://www.globalsupport.harvard.edu/travel-tools/risk-ratings. Because these risk ratings are dependent on a number of factors, it is recommended that the website be consulted frequently to assess the risks in the region. When planning an international research project, undergraduate researchers should take the following steps: a) Check the GSS risk ratings site to determine if the region of interest is rated as high or elevated risk. 41 | P a g e i) If the region of interest is rated as high-risk, undergraduate researchers cannot conduct research there. The researcher must select another region for the project. ii) If the region is rated as elevated-risk, the researcher should follow the steps described under “Restricted Travel” on the Harvard College International Travel Policy website. When submitting their protocol and related materials to IRB, the researcher should include their correspondence from GSS allowing them to conduct research in the region. b) When preparing the protocol, the researcher should describe how they plan to protect their safety while working internationally. These plans could include checking in with the local U.S. embassy, working with local universities or non-governmental organizations (NGOs) to help navigate the cultural norms, and/or arranging safe meeting places when conducting interviews with research participants. c) Researchers should also provide details of their preparation for working in the area, including their fluency in the local language and any past travel or coursework that has helped them prepare for this project. Example: A student plans to conduct a research project in a region designated as elevated-risk by GSS. In their initial protocol, the student does not include any details on their preparation for the travel; there is no mention of any local connections the student has made, no information on the familiarity of the student with the country, or how the student would choose a safe location for interviews. This lack of information raises concerns that the student has not sufficiently considered the potential risks in the region, and how they would protect themselves. The review will certainly yield a number of key questions that would, at the minimum, delay the approval of the project. How to Fix: The protocol should provide details about the student’s efforts to prepare for this project, including discussions with GSS, contacting local NGOs, working with Harvard faculty who have experience in the region, and describing how they plan to setup a safe space for interviews. With this additional information, the Provostial Review team will be able to approve the work. 2) Reputational risks The Provostial Review team assesses research projects not just on the subject of the research design, but also on the impact the project will have on the reputation of the participants, researchers, and the University. Generally, researchers should consider if anyone involved in the project, or the University itself, is at risk of reputational harm from the work. If there is a possibility of reputational risk, the researcher should explain why the risk is necessary, how they will mitigate that risk, and why other options for conducting the research are not feasible. Example: A researcher’s planned project includes the use of deception to get responses from political officials, with no plans to debrief the officials before the results are published. The results of the work could attract media attention and could potentially embarrass the officials and hurt their reputation in the community. How to Fix: The Provostial Review team would request that a debriefing email be sent to the participating officials once data collection was complete, explaining the research project and notifying them that the results would be published. Including such details would assist with expediting the review process. 3) Risk to populations involved When conducting research in an international location, the researcher should carefully consider the cultural and social norms in the region of interest. Research conducted with marginalized populations, or projects that ask questions about socially-unacceptable or illegal behavior, could lead to negative consequences for the participants. In the research protocol, the researcher should explain how they will protect the participants, including plans to secure the data and to receive local IRB and/or community approvals in the region of interest. 44 | P a g e Appendix VII Useful Resources Harvard University CUHS IRB http://cuhs.harvard.edu ESTR (Electronic Submission, Tracking, and Reporting) Support website http://estrsupport.fss.harvard.edu/ Harvard Office for the Vice Provost of Research http://vpr.harvard.edu/ Federal Department of Health and Human Services (HHS) http://www.hhs.gov Centers for Disease Control and Prevention (CDC) http://www.cdc.gov Human Participant Protection in CDC Research http://www.cdc.gov/od/science/regs/hrpp/ National Institutes of Health (NIH) http://grants.nih.gov/grants/policy/hs/index.htm Certificate of Confidentiality Kiosk http://grants.nih.gov/grants/policy/coc/ Office of Civil Rights (HIPAA policy) http://www.hhs.gov/ocr/privacy/index.html Office of Human Research Protection (OHRP) http://www.hhs.gov/ohrp/ • Regulations http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.htm • Decision Charts http://www.hhs.gov/ohrp/policy/index.html#decision • Guidance and Policy http://www.hhs.gov/ohrp/policy/ • FAQs http://www.hhs.gov/ohrp/faq.html • International Research Policies http://www.hhs.gov/ohrp/international/index.html#NatlPol Food and Drug Administration (FDA) http://www.fda.gov/ Department of Education http://www.ed.gov/ • Human Subjects Research http://www.ed.gov/about/offices/list/ocfo/humansub.html • Family Policy and Compliance Office (FERPA and PPRA) http://www.ed.gov/policy/gen/guid/fpco/index.htl National Science Foundation http://www.nsf.gov • FAQs on Interpreting the Common Rule for the Protection of Human Subjects for Social and Behavioral Research http://www.nsf.gov/bfa/dias/policy/hsfaqs.jsp 45 | P a g e National Science and Technology Council Report on Expedited Review of Social and Behavioral Research Activities http://www.ostp.gov/galleries/NSTC%20Reports/Expedited%20Review%20For%20Web.pdf Discipline‐Specific Resources American Anthropological Association (AAA) http://www.aaanet.org American Educational Research Association (AERA) http://www.aera.org American Psychological Association (APA) http://ww.apa.org American Public Health Association (APHA) http://www.apha.org American Sociologic Association (ASA) http://www.asanet.org National Association of Social Workers http://www.naswd.org/code.htm
Docsity logo



Copyright © 2024 Ladybird Srl - Via Leonardo da Vinci 16, 10126, Torino, Italy - VAT 10816460017 - All rights reserved