Refining and Expanding the Effective Use of 4S: An Evidence-Based Program to Increase Adolescents’ Ability to Self-Manage their School Success, Essays (university) of Financial Management

Download Refining and Expanding the Effective Use of 4S: An Evidence-Based Program to Increase Adolescents’ Ability to Self-Manage their School Success and more Essays (university) Financial Management in PDF only on Docsity! Project Narrative Contents Absolute Priorities ...........................................................................................................................1 A. Significance .................................................................................................................................2 B. Quality of Project Design ............................................................................................................5 B.1. Clearly Specified and Measurable Goals, Objectives, and Outcomes ..................................6 B.2. How Project Design Will Meet the Needs of Target Population ........................................13 B.3. How Proposed Activities Constitute a Coherent, Sustained Program of R & D ................15 B.4. How Proposed Project Will Increase Efficiency .................................................................16 C. Strategy to Scale ........................................................................................................................17 C.1. Strategies to Address Barriers to Scale ...............................................................................17 C.2. Mechanisms to Support Wide-Scale Dissemination ...........................................................19 D. Adequacy of Resources and Quality of the Management Plan .................................................20 D.1. Capacity to Bring the Project to Scale ................................................................................20 D.2. Reasonable Project Costs ....................................................................................................21 D.3. Potential for Continued Project Support .............................................................................22 D.4. Adequacy of Management Plan ..........................................................................................23 E. Quality of the Project Evaluation ..............................................................................................25 E.1. The Evaluation Will Meet What Works Clearinghouse Standards Without Reservations 26 E.2. Key Components, Mediators, Outcomes, Implementation Thresholds, and Cost...............29 E.3. Evaluation Methods Will Provide Valid and Reliable Performance Data on Outcomes ....32 References ......................................................................................................................................34 Tables and Figure Table 1. Goals, Objectives, and Outcomes .....................................................................................9 Figure 1. Theory of Action: How Curricular Intervention Improves Student Outcomes ..............14 Table 2. Key Personnel, Roles, and Responsibilities .....................................................................24 Table 3. Timeline and Management Chart for Accomplishing Project Milestones ......................25 Table 4. Research Questions by 4S Theory of Action Component ...............................................26 Table 5. Fidelity of Implementation ..............................................................................................30 Table 6. Outcomes by Domain, Measure, Baseline Measure, Timing, and Data Source ..............31 Appendix I – Other Materials I.1. CCSR Hypothesized Model of How Non-Cognitive Factors Affect Academic Performance I.2. Mastering the Middle Grades Scope and Sequence I.3. CSOS Studies of Similar Size and Scope I.4. 4S Timeline at a Glance I.5. Power Analysis Assumptions (for Independent Evaluation) I.6. Data Collection Timeline (for Independent Evaluation) PR/Award # S411B200026 Page e16 1 Refining and Expanding the Effective Use of 4S: An Evidence-Based Program to Increase Adolescents’ Ability to Self-Manage their School Success Absolute Priorities This project addresses Absolute Priority 1—Moderate Evidence—by refining and expanding the use of practices identified as having strong evidence of impact on student staying in school outcomes in the IES Practice Guide Preventing Dropout in Secondary Schools (see Evidence Form). Specifically, its third recommendation calls on schools to “engage students by offering curricula and programs that connect schoolwork with college and career success and that improve students’ capacity to manage challenges in and out of school” (Rumberger et al., 2017, p. 3). This project seeks to organize these practices into a Skills for Secondary School Success (4S) course that better prepares 8th graders to navigate the often perilous transition from the middle school to high school and increases students’ capacity to self-manage their school and life success. As such, the project also addresses Absolute Priority 3—Field- Initiated Innovations —Fostering Knowledge and Promoting the Development of Skills That Prepare Students To Be Informed, Thoughtful, and Productive Individuals and Citizens – by validating through a rigorous efficacy study the impact of the 4S course on 8th graders’ social-emotional skills and mindsets identified in existing literature (Aspen Institute, 2019) as most closely associated with students’ ability to self-manage their school success (Claro & Loeb, 2019) and the academic outcomes most predictive of success in high school and beyond, attendance and course grades (GPA) (e.g., Allensworth, 2013; Neild et al., 2008). We will do this in schools serving different populations of historically underserved students in different locales throughout three states. In addition, through a second efficacy study focused on identifying effective but lower cost means of providing teachers the professional development PR/Award # S411B200026 Page e17 4 have been reviewed by CASEL (CASEL, 2015), and RAND (Grant et al., 2017). Most of the evidence for these programs is based on small samples of students and schools or on studies that were not conducted with an RCT design. There are only four SEL programs identified in the RAND report as serving grades 7-12 with positive effects on academic outcomes (including attendance) and only two of them (Building Assets-Reducing Risks and Positive Action) have also met WWC standards.1 Building Assets-Reducing Risk is primarily a 9th grade intervention and requires substantial shifts in school practice (Bos et al., 2019). The evidence of impact for Positive Action is for students in grades 1-5 (Grant et al., 2017). Finally, while there is a growing number of social-psychological mindset interventions with rigorous evidence of positive impact (e.g., Williams et al., 2020; Yeager & Bundick, 2009) and, in the case of growth mindsets, evidence of academic impacts from a scalable intervention tested in a rigorous, national study (Yeager et al., 2019), these social-psychological efforts target just one particular mindset or skill rather than developing the full integrated set of social-emotional orientations and skills needed to make a game-changing impact on academic success (Balfanz & Byrnes, 2020). Thus, there is both a clear area of need – better preparing 8th graders for the transition to high school by enhancing their social emotional development (particularly in the domains of self-management, goal setting, problem solving, belonging, interpersonal skills, and perseverance) – and a lack of evidence-based curricular interventions for middle grades students that meet this need comprehensively, in a manner that can be widely implemented by schools, particularly those schools that serve high concentrations of high-needs students. In short, classroom curricula that integrate multiple high impact SEL skills together and in so doing improve 8th grade students’ ability to better self-manage their school success are needed. 1 The other programs, Student Success Skills, and Mindfulness/Yoga interventions, have been studied in RCTs with small samples and not yet successfully completed WWC review. PR/Award # S411B200026 Page e20 5 To make a difference on a wide scale, these curricula – and the teacher training and implementation supports that accompany them – need to be designed to work within existing school structures and to not require large and sustained investment of school resources nor heroic amounts of teacher time and effort. This project aims to contribute to the knowledge base of how to address this important goal by refining, working to remove barriers to scale, and validating the impact among diverse groups of students in varied locales of an 8th grade Skills for Secondary School Success (4S) course designed to increase adolescents’ ability to self-manage their school success and leave 8th grade on-track to high school graduation. B. Quality of Project Design. Like the interventions in the research evidence cited by the WWC Practice Guide, our proposed project intervention involves a supplementary “Skills for Secondary School Success” course for 8th graders. It contains 40 instructional days of classroom learning activities designed for a 45-60 minute period to be given over the course of nine weeks (scheduled as one of several electives offered in different quarters of the year). The 4S course links the learning activities to career exploration and success in high school to provide a meaningful context and motivation for the course (Yeager, 2017). Its learning activities are designed to follow the conceptual model developed by Chicago Consortium for School Research (CCSR) (Farrington et al., 2012, p. 12) based on an extensive literature review of the relationship between social-emotional and school success skills and students’ academic outcomes. Their model (see Appendix I.1) specifies that academic mindsets, social skills, academic perseverance, and learning strategies all have independent impacts on the academic behaviors that drive academic performance. The learning activities will also incorporate and integrate together social- psychological mindsets with recent evidence of positive impact on success in 9th grade, including growth mindset (Yeager et al., 2016), social belonging (Williams et al., 2020), and purpose in PR/Award # S411B200026 Page e21 6 learning (Yeager, 2017). Thus, the 4S course learning activities aim to build students’ self- management skills and their relationship and collaboration skills (particularly in conflict avoidance and resolution), as well as their academic mindsets, perceptions, and approaches to learning (e.g., goal setting, productive persistence, growth mindset, learning science-based study skills). The 40 learning activities will be culled from a three-year middle grades advisory curriculum already created by the research and development team (see Appendix I.2), with selection guided by existing research literature and the CCSR conceptual model described above to create the highest-impact set of lessons possible. The proposed intervention aligns with practices outlined in supporting evidence cited by the WWC Practice Guide, including integration into the school day and intensity of dosage equivalent to a nine-week quarter (Johnson et al., 2014), building of academic and SEL skills (Dynarski et al., 1998), and development of self-regulation and relationship skills (Heller et al., 2013; Heller et al., 2015). It is also congruent with important SEL program characteristics emphasized by Osher et al. (2016): developmentally appropriate, culturally relevant, and evidence-based. The intervention focuses on grade 8 because of the need to better prepare students to transition successfully to and through ninth grade (Allensworth & Easton, 2007), as well as the finding that as adolescents reach high school age program-based interventions may be less effective (Yeager, 2017). B.1. Clearly Specified and Measurable Goals, Objectives, and Outcomes. The primary outcome of the proposed project is refinement and validation across diverse student populations and locales of a scalable 8th grade curricular/instructional intervention with related professional development supports for teachers, honed for cost-effectiveness. The project will provide middle grade schools serving high-needs populations with an implementable and sustainable means of PR/Award # S411B200026 Page e22 9 Table 1. Goals, Objectives, and Outcomes Goals Objectives Outcomes GOAL 1 – Refine and prepare for widespread use a Skills for Secondary School Success Course designed to increase 8th graders’ SEL skills and improve academic outcomes (attendance and grades), better preparing them to self- manage their school success in 9th grade and beyond 1.1 Revise existing advisory curriculum to leverage highest impact activities in a shorter, more scalable intervention, a quarter long 8th grade Skills for Secondary School Success Course Measure 1.1. Completed course materials (40 days of learning activities), including teacher guide and class/student materials for dissemination in digital format and use in RCT Goal 2 1.2 Collect teacher and student feedback on course materials and learning activities after RCT Goal 2, PD Pilot Study Goal 3, and Efficacy and Cost Effectiveness Test of Alternative PD model, Goal 4 Measure 1.2. Teacher logs/interviews/focus groups and student survey findings provide end user perspective on ease of use, clarity, engagement of the course materials. 1.3 Revise course materials based on teacher and student feedback from RCT Goal 2 and Professional Development Pilot Study Goal 3 Measure 1.3. Revised course materials (40 days of learning activities) ready for widespread use GOAL 2 – Conduct RCT to measure impact of intervention on student outcomes. 2.1 Recruit 10 middle grade schools with diverse student populations in diverse regions of three states as location for randomization at student level Measure 2.1 Signed MOUs will be successfully obtained from at least 10 eligible middle schools during Y1 and Y2. 2.2 Establish school and district data agreements Measure 2.2 Signed data agreements will be successfully obtained from at least 10 eligible middle schools during Y1 and Y2. 2.3 Randomly assign students at each school to treatment and control groups (n=1000) Measure 2.3.1 Baseline data will be collected from the 10 participating schools. Measure 2.3.2 Students will be randomly assigned to treatment and control groups and Measure 2.3.3 Analyses will verify baseline equivalence of treatment and control groups of students. 2.4 Collect and analyze data on fidelity of implementation Measure 2.4 One implementation fidelity report will be completed for each year of the program, for a total of 2 reports. 2.5 Collect and analyze data on SEL outcomes Measure 2.5 Pre and post DESSA assessments will be collected and analyzed on study students, using WWC recommended procedures to measure treatment impact on SEL outcomes. PR/Award # S411B200026 Page e25 10 2.6 Collect and analyze data on academic outcomes Measure 2.6 Analyses of pre and post administrative student data from at least 800 students will be conducted using WWC recommended procedures to measure treatment impact on academic outcomes. 2.7 Collect and analyze cost data Measure 2.7 Per-pupil cost analysis using data collected from schools, as well as cost effectiveness analysis for treatment effects will be written by the end of the Year 3. 2.8 Complete final report Measure 2.8 A final RCT Impact Report will be submitted and disseminated after peer review by invited panel of scholars. GOAL 3 – Develop and pilot alternative means to provide teacher professional development to reduce costs and support large scale dissemination 3.1 Produce PD materials and tools for: 1) micro-creden- tialing, 2) distance learning using medical model (ECHO), and 3) peer coaching with learning communities Measure 3.1.1 One complete set of PD materials in digital/online format for the micro-credentialing. Measure 3.1.2 One complete set of PD materials in digital/online format for the distance learning. Measure 3.1.3 One complete set of PD materials in digital/online format for the peer coaching with learning communities. 3.2 Test PD modalities in 10 or more middle schools-30 or more teachers (Y3) Measure 3.2 Successful implementation of the PD modalities in each of the 10 additional schools. 3.3 Collect and analyze data on implementation of PD and course Measure 3.3 One report of implementation of PD modalities and course implementation will be written by the end of year 4. GOAL 4 – Test alternative professional development models while scaling intervention to 60 schools to establish costs and benefits of different PD modalities under typical conditions 4.1 Recruit a national sample of 60 middle schools Measure 4.1 Signed MOU from 60 additional middle schools 4.2 Implement course and randomly assigned alternative forms of PD in 60 middle schools in Y4 Measure 4.2.1 Baseline data will be collected from the 60 additional middle schools. Measure 4.2.2 Analyses will verify baseline equivalence of treatment and control schools Measure 4.2.3 Implementation completed at 60 schools as evidenced by teacher logs. 4.3 Collect and analyze implementation data and student SEL data from all schools Measure 4.3 A report on the impact of PD Modality on 4S Implementation and Students’ Social-Emotional Learning will be completed by the end of year 5. 4.4 Compare implementation levels and implementation costs for alternative forms of PD Measure 4.4.1 A cost analysis report on alternative forms of PD will be completed by the end of year 5. Measure 4.4.2 The team will submit at least one conference proposal, one manuscript submission to a journal, and the team will present findings in at least one webinar. PR/Award # S411B200026 Page e26 11 Details on the objectives and outcomes of the external evaluation can be found in Section E. Teachers participating in the evaluation will receive two days of training and two in- classroom coaching visits from the R and D team’s implementation support partner, Talent Development Secondary (see Appendix C for letter of support). This will help ensure that teachers receive sufficient training and support to be able to achieve moderate to high implementation of the course, and also establish a baseline of PD effectiveness and cost, to which we can compare the alternative means of professional development being developed and piloted in Goal 3 and tested in Goal 4. Goal 3 of this project is to assemble and pilot test alternative means of providing teacher professional development (PD) and implementation support for the 4S course, which may remove barriers to scaling. In section C (below) we present details on how the time and costs involved in an instructional coaching approach to professional development in schools with significant amounts of teacher turnover were identified as a barrier to scale. Thus, our first objective under Goal 3 is to assemble materials and tools for alternative PD delivery methods aimed at making the professional development and supports teachers need to successfully implement the 4S course more cost-effective and sustainable. In order to find a PD approach that is effective, affordable, and sustainable by schools under typical conditions, it is necessary to test several approaches. We will assemble three with which the R and D team has prior experience in other projects. The first approach is a web-based course through which a teacher can earn a micro-credential based on demonstrating their understanding of the content and purpose of the course, effective instructional delivery methods, and video evidence of their successful application. Micro-credentials are becoming an increasing popular means of PD and provide teachers with a self-initiated, personally controlled, and tailored means of competency-based PD PR/Award # S411B200026 Page e27 14 students from low-income homes, including schools in urban, rural, small city or suburban settings with different racial/ethnic student populations). Figure 1. Theory of Action: How Curricular Intervention Improves Student Outcomes 4S curriculum implemented by school-based faculty for 40 hours over 9 weeks Context of learning how to succeed in high school and how high school relates to careers supports student engagement in 4S course Professional development & supports for 4S teachers Students engage in activities (in collaboration with others in classroom) designed to build skills in relating to others, self- management, persistent effort, etc. Improved SEL competencies supportive of greater self- management of school success Improved attendance and course grades Better able to self- manage school success More successful transition to 9th grade/remain on track to high school graduation through 9th grade (good attendance, no suspensions, no course failures) Improved probability of on- time graduation Participation in the course activities is expected to build SEL skills as well as career awareness and understanding of the relevance of academic learning to future life goals. Because the course focuses on learning how to succeed in high school and how schooling relates to careers, we predict adolescents will view it as more authentic and engaging than a class formally focused on developing social-emotional skills (Yeager, 2017). Thus, course participation is expected to result in improvement in the SEL skills of self-awareness, self-management, social awareness, relationship skills, goal-directed behavior, personal responsibility, decision making, and optimistic thinking. Based on previous research cited in Section A above, we expect this increase INPUTS OUTPUTS SHORT TERM OUTCOMES MEDIUM TERM OUTCOMES LONG TERM OUTCOMES PR/Award # S411B200026 Page e30 15 in SEL skills to be associated with improved academic outcomes in attendance and course grades, which are themselves predictive of a successful transition to and through the ninth grade and ultimately high school graduation outcomes. Prior research also indicates that the largest gains may be achieved by students who have lower grades and attendance prior to the intervention, and by students with lower initial levels of social-emotional development (Balfanz & Byrnes, 2020). This in turn suggests that the intervention should be more impactful in schools with greater percentages of high-needs students (e.g. low income, students with disabilities etc.). B.3. How Proposed Activities Constitute a Coherent, Sustained Program of R & D. The core work of our research and development center, the Center for Social Organization of School (CSOS) at Johns Hopkins University, is providing educators with evidence-based and practice-validated tools, strategies, approaches, and curricula they can effectively use to improve student outcomes in schools that serve high poverty populations. One fundamental challenge of this work, which we aim to help address in the current project, is how schools that serve high needs populations can find the time, capacity, and resources to implement the comprehensive improvements that evidence suggests are required. It is necessary to develop effective interventions that have high returns for amount of capacity required to implement them. Our proposed activities speak directly to that need. We will conduct a series of linked development and evaluation activities designed to improve middle grade students’ social emotional and school success skills – and through them their attendance and course grades – in a shorter concentrated intervention, with the involvement of a smaller number of teachers and more cost-effective means of professional development and coaching. Our proposed activities also represent a coherent sustained program of research and development within the SEL field. The Aspen Institute (2019) calls for research and PR/Award # S411B200026 Page e31 16 development that build upon and seek to further strengthen the integration of social, emotional, and academic development. We seek to advance this by establishing the impact of a curricular intervention – that develops a selected set of key SEL skills and mindsets – on school outcomes (attendance, grades, and on-track rates) during the transition from the middle grades to high school. As noted earlier in Section A, much of the prior SEL research has focused on the impact of either individual social-psychological mindsets (i.e., growth mindset and social belonging – usually at the high school or college level) or comprehensive social emotional development programs focused on a broad range of student outcomes and elementary students. Both of these leave the middle ground and middle grades less explored. Our proposed activities will address that gap, examining how a selected set of SEL skills and mindsets, developed in the authentic context of preparing for success in high school, can improve academic outcomes for 8th graders. B.4. How Proposed Project Will Increase Efficiency. As detailed in the barriers to scale section C.1, the proposed project is focused on taking an existing evidence-based intervention – a three-year middle grades advisory curriculum – and re-constructing it into a quarter-long course while also establishing more cost-effective means of delivering effective professional development and support for the course. This will increase efficiency in the use of time, staff, money and other resources as it transforms an evidence-based approach that previously took three years and could involve up to 10 teachers per middle school into a similar approach that takes one-third the time, and could involve one-tenth the number of teachers, while potentially using a more efficient and lower-cost means of professional development. Moreover, we aim to further improve the results of the course and increase its productivity by focusing on the sub-set of social emotional and school success activities that research indicates will have the greatest impact on academic outcomes and school success. PR/Award # S411B200026 Page e32 19 impact on student academic outcomes. This will increase schools’ ability to make the case that there is a good ratio of academic return to the time and resources invested. Strategy 3 - We will build into the new version of the instructional materials validated pre- and post-tests of social emotional skills (that were not available when the curriculum was initially developed), which will enable schools to have quantifiable student outcomes. Strategy 4 - We will test and validate alternative and cost-effective means of teacher PD, which could provide schools with a more affordable means to maintain the course through inevitable teacher turnover and also support district-wide scale up. C.2. Mechanisms to Support Wide-Scale Dissemination. The Center for Social Organization of Schools (CSOS) at the Johns Hopkins University School of Education has a more than 50- year track record of impactful research, development, and dissemination work. Multiple research and development efforts led by the PI and Co-PIs have been widely disseminated, resulting in further development and replication. These include: 1) Early Warning and Intervention Systems to increase high school graduation rates (involving multiple district and SEA partnerships to help scale them); 2) the Diplomas Now whole school improvement model (funded by i3 and multiple private funders over a ten-year period); and 3) partnerships with school districts and states, through the National Student Attendance, Engagement, and Success Center, to spread Center learnings and approaches on reducing chronic absenteeism. Existing CSOS mechanisms, many developed through the efforts described above, will be used to widely disseminate findings and materials from the project to support further development and replication. These include distributing the findings and materials through our existing networks, including a secondary school redesign network (https://www.hsredesign.org/) working with six states (NY, OH, MA, LA, MS, NM), and the Pathways to Adult Success PR/Award # S411B200026 Page e35 20 network (http://www.pathwaystoadultsuccess.org/) focused on the transition from middle grades to high school and then high school to post-secondary, which involves over 150 school districts, state departments of education, and non-profits and schools working to better support students through these transitions. In addition, the Everyone Graduates Center website (primary dissemination vehicle for CSOS) and on-going webinar series have a wide following and will be used to showcase our findings. We have existing in-house communications, social media, and graphics teams who will help us spread learnings from the project and build support for additional development and replication through webinars, podcasts, and media outreach, as well as the more traditional means of conference presentations and journal articles. Our prior and on- going research, development and dissemination efforts have also enabled us to build a web of relationships among school districts, state departments of education, and funders (prior and on- going work has been funded by the Gates, Arnold, Ford, Schusterman, and Mott Foundations among others) which we will activate to support and enable the replication of this project. Lastly, the PI is currently engaged in an inter-connected set of activities within the SEL field, including serving as a Distinguished Scientist on the Aspen Social, Emotional and Academic Development Commission, which will enable further spread of the project’s findings. D. Adequacy of Resources and Quality of the Management Plan D.1. Capacity to Bring the Project to Scale. The JHU research team has extensive experience in leading and successfully completing other projects of similar scale over the past 15 years (see CVs in Appendix B), including recruiting more than 200 different schools for randomized control trials and dozens of other schools for other studies. All the investigators have been PIs or co-PIs on multiple IES and i3-funded studies, including RCT studies of Diplomas Now, a 9th grade early warning system study, and a high school mathematics curriculum (see Appendix I.3 PR/Award # S411B200026 Page e36 21 for full list of studies of similar scope, and D.4 below for specific responsibilities). Many of these previous projects included program development, implementation support, and teacher training components, as well as multiple forms of qualitative and quantitative data collection and analysis across multiple school sites, districts, and locales. We have successfully worked with third-party evaluators and raised required matching funds on multiple projects. We also have experience in conducting cost-effectiveness analyses (see Mac Iver et al., 2019). The Alabama State Department of Education has agreed to help recruit schools from diverse regions of the state for the study (see letter of support in Appendix C) and we have a long track record of successfully recruiting schools from multiple states and districts for RCTs. Currently, there is widespread interest among school leaders in finding effective SEL interventions that build school success (Atwell & Bridgeland, 2019), and this is likely to further heighten by the impact of Covid-19 and its aftermath. The independent evaluation will be led by a well-experienced team from SRI International, including a principal researcher who served as Project Director and Co-Principal Investigator for the What Works Clearinghouse (WWC) from 2014–18. The team includes members with previous experience conducting RCTs and fidelity of implementation analyses for previous EIR and i3 evaluations and substantial expertise in rigorous econometric methods, as well as content expertise in both social-emotional learning and secondary school reform. Details on their specific experience can be found in their CVs (Appendix B). Their roles and responsibilities are outlined in D.4 below. D.2. Reasonable Project Costs. For Goal 1 and 3 objectives, we have budgeted for teams of researchers and staff practitioners to work together. Time commitments are based on our prior experience with the research and development projects of similar scale and potential impact. For PR/Award # S411B200026 Page e37 24 Table 2. Key Personnel, Roles and Responsibilities Key Personnel and Functional Responsibilities Chart Position Function Principal Investigator Robert Balfanz, Ph.D. Professor, JHU School of Education • Project director with overall responsibility for and contributing to all parts of the project, project management, and writing of reports • Lead contact for U.S. Dept. of Education Senior Leadership Team Douglas Mac Iver, Ph.D Professor, JHU School of Education • Lead for assembly and iterative refinement of intervention curriculum and associated PD variations • Lead for Cost Analysis of Alternative Forms of PD Marcia Davis, Ph.D Associate Professor, JHU School of Education • Lead for recruitment of sites and school/district relations • Lead for development of implementation measures • Lead for Year 4 Implementation of Alternative Forms of PD for RCT Martha Mac Iver, Ph.D Associate Professor, JHU School of Education • Lead liaison to Evaluation Team • Lead for Year 3 PD types pilot study • Lead for Year 4 RCT data collection and analysis Richard Lofton, Ph.D Assistant Professor, JHU School of Education • Lead for qualitative data collection and analysis to inform revisions of course materials and PD supports External Evaluation Team Daniel Princiotta, Ph.D. Senior Education Researcher at SRI International; Independent Evaluation Co-Principal Investigator • Lead impact evaluation execution and cost analysis • Liaison to JHU research team • Produce final report with Dr. Park C.J. Park, M.P.P. Senior Education Researcher at SRI International; Independent Evaluation Co-Principal Investigator. • Lead evaluation project management • Lead implementation evaluation execution and cost analysis • Produce final report with Dr. Princiotta Neil Seftor, Ph.D Principal Researcher at SRI International Independent Evaluation Senior Advisor. • Provide overall guidance on study design, implementation and reporting PR/Award # S411B200026 Page e40 25 Table 3. Timeline and Management Chart for Accomplishing Project Milestones SY1 SY2 SY3 SY4 Milestone Responsible 1/21- 6/21 6 mos 7/21- 6/22 7/22- 6/23 7/23- 6/24 7/24 - 6/25 7/25- 12/25 6 mos Goal 1. Assemble and refine the Skills for Secondary School Success 9 week course, including teacher guide and student materials for 40 days of learning activities designed to develop key SEL skills among 8th graders aligned with the WWC Dropout Prevention Practice Guide Recommendation 3 1.1 JHU (D MacIver & team) X X X X X X 1.2 JHU (R Lofton & team) X X X X 1.3 JHU (D MacIver & team) X X X Goal 2. Conduct RCT to measure impact of 4S Course on student SEL and Academic outcomes. 2.1 JHU (M Davis & team) X X X 2.2 SRI (Princiotta & Park) X X X 2.3 SRI (Princiotta & Park) X X X 2.4 SRI (Princiotta & Park) X X X 2.5 SRI (Princiotta & Park) X X 2.6 SRI (Princiotta & Park) X X X 2.7 SRI (Princiotta & Park) X X X 2.8 SRI (Princiotta & Park) X Goal 3. Develop and pilot alternative means to provide teacher professional development to reduce costs and support large scale dissemination 3.1 JHU (D MacIver & team) X X X X X 3.2 JHU (M Davis & team) X X 3.3 JHU (M MacIver & team) X X Goal 4. Test alternative PD models while scaling intervention to 60 schools to establish costs and benefits of different PD modalities under typical conditions 4.1 JHU (M Davis & team) X X 4.2 JHU (M Davis & team) X 4.3 JHU (M MacIver & team) X X 4.4 JHU (D MacIver & team) X X E. Quality of the Project Evaluation SRI International (SRI) will conduct an independent evaluation of the impact, implementation fidelity, and cost of 4S. The evaluation will address seven confirmatory and four exploratory research questions (RQs and ERQs) aligned with the 4S Theory of Action (Table 4). RQs 1-4 focus on program impact and align with short- and medium-term outcomes. RQs 5-7 align with program inputs and outputs and address implementation and cost. ERQs 1-2 investigate potential moderation effects, and ERQs 3-4 explore the impact of 4S on individual SEL skills and on an On-Track for 9th Grade indicator. PR/Award # S411B200026 Page e41 26 Table 4. Research Questions by 4S Theory of Action Component Research question 4S Theory of Action component Impact: Short-term outcomes RQ1. What is the impact of 4S on grade 8 students’ overall social-emotional competence (SEC) as rated by teachers? Improved SEL skills Impact: Medium-term outcomes RQ2. What is the impact of 4S on grade 8 students’ academic performance as measured by semester 2 core GPA and by no Ds/Fs in any semester 2 core course? Improved grades RQ3. What is the impact of 4S on grade 8 students’ engagement in school as measured by semester 2 attendance rate and by 90% or better semester 2 attendance? Improved attendance RQ4. What is the impact of 4S on grade 8 students’ behavior in school as measured by semester 2 number of suspensions and by any suspension receipt in semester 2? Improved behavior Implementation and cost: Inputs and outputs RQ5. To what extent was the 4S intervention implemented with fidelity? Curriculum and PD RQ6. What were the barriers and facilitators to successful 4S implementation? Implementation context RQ7. What were the costs of implementing 4S overall and per student by school? Curriculum, PD, & context Exploratory research questions: Moderation effects ERQ1. How does the effect of 4S on students’ overall SEC vary by students’ baseline SEC, baseline semester 2 core GPA, and demographic characteristics? Improved SEL skills, implementation context ERQ2. How does the effect of 4S on students’ semester 2 core GPA vary by students’ baseline SEC, baseline semester 2 core GPA, and demographics? Improved grades, implementation context Exploratory research questions: Additional outcomes ERQ3. What is the impact of 4S on grade 8 students’ growth mindset, self- efficacy, self-management, and social awareness, as measured by CORE student survey scales? Improved SEL skills ERQ4. What is the impact of 4S on semester 2 On-Track for 9th Grade composite indicator (no Ds/Fs, 90 percent or better attendance, and no suspensions)? Improved grades, attendance, and behavior E.1. The Evaluation Will Meet What Works Clearinghouse Standards Without Reservations SRI will execute a multi-site, individual-level randomized controlled trial (RCT) where 8th graders will be randomly assigned to treatment and control conditions within 10 schools across 2 waves (5 schools implementing in 2021/22 and the other 5 in 2022/23). Prior to random assignment, each student will have a 50 percent chance of being assigned to either condition. Treatment students will be enrolled in the 4S course in quarter 2, while control students will receive a “business as usual” elective course. To provide sufficient treatment-control contrast, PR/Award # S411B200026 Page e42 29 number of schools in the study (Pustejovsky & Tipton, 2018). In addition, we will examine whether treatment effects vary across students and schools by including interaction terms between treatment status and moderators in our models. We will investigate several student-level moderators: economic disadvantage, race/ethnicity, English Learner status, disability status, prior socioemotional competence, and prior-year 2nd semester core-course GPA. Prior research has shown that SEL interventions have shown greater effects among those with relatively low initial grades (c.f., Paunesku et al., 2015; Yeager et al., 2016). We will also investigate whether treatment effects vary by site by including school-by- treatment status interaction effects. By identifying whether the intervention’s effectiveness varies across groups of students or in different implementation contexts, the evaluation will be able to inform efforts to further develop and scale the 4S initiative. E.2. Key Components, Mediators, Outcomes, Implementation Thresholds, and Cost The proposed evaluation will examine implementation of 4S’s key components, and its relationship with students’ short- and long-term outcomes as described in section B2 and the theory of action. The 4S approach posits that to impact student outcomes, teachers need training and coaching in 4S and must implement 4S daily lessons with sufficient frequency and quality. Consequently, we will measure two key program components: (1) professional development, and (2) 4S curriculum implementation. To be implemented with fidelity, 4S must meet the program- level threshold for both components as specified in Table 5. All teachers implementing the 4S course in the 10-school RCT will be included in the implementation fidelity sample. SRI will use a mix of program records and primary data to measure fidelity of implementation (FOI). After each training session, 4S program staff will share training attendance records with SRI. SRI will also collect 4S program coaches’ teacher coaching records PR/Award # S411B200026 Page e45 30 (teacher visited, date, and focus of the visit). Coaches will also provide a summative teacher rating (rubric to be developed in summer 2022) that SRI will use to measure curriculum implementation quality. Further, SRI will administer a weekly teacher instructional log (developed in summer 2021) to capture the number of unique daily lessons delivered by teachers over the course of the 9-week 4S class to measure quantity of curriculum implementation. Table 5. Fidelity of Implementation Components & Indicators Data Source & Timing Metric Component 1: Professional Development Component-Level Threshold: Receive a rating of “2” or higher on both indicators 1.1 Teacher participates in training prior to course Training attendance records Fall 2021 & Fall 2022 3 = Received more than 6 hours of training 2 = Received 6 hours of training 1 = Received less than 6 hours of training 1.2 Implementation support Coaching records Winter 2022 & Winter 2023 3 = Received 2 in-person visits 2 = Received 1 in-person visit 1 = Received no in-person visits Component 2: 4S Curriculum Implementation Component-Level Threshold: Receive a rating of “2” or higher on both indicators 2.1 Quantity of curriculum implementation Teacher log Fall 2021 & Fall 2022 3 = Delivered at least 35 daily lessons 2 = Delivered 30-34 daily lessons 1 = Delivered less than 30 daily lessons 2.2 Quality of curriculum implementation Summative rating by coach (rubric to be developed) Winter 2022 & Winter 2023 3 = Above expectations 2 = Meets expectations 1 = Does not meet expectations Adequate teacher-level implementation: Meet fidelity threshold for Components 1 & 2 Adequate program-level implementation: 75% of teachers meet teacher-level fidelity threshold. SRI will also collect qualitative data on barriers and supports to implementation. Teacher interviews and student focus groups will be conducted for each wave of 4S schools during the implementation school year. Interviews with teachers will provide information on contextual factors related to implementation and perceptions of the quality and efficacy of the professional development and 4S curriculum. Student focus groups will provide insight into students’ engagement with the curriculum. SRI will share interview, focus group, and FOI findings with JHU annually to support program refinement and scaling. Short- and medium-term evaluation outcome measures as shown in the 4S Theory of Action are listed in Table 6, along with associated measures, timing of baseline and outcome PR/Award # S411B200026 Page e46 31 measure collection, and data source. 4S program outputs are expected to mediate 4S’s impact on students’ social-emotional learning skills (short-term outcomes), which, in turn, are expected to mediate 4S’s impact on student attendance, behavior, and grades (medium-term outcomes). These outcomes are expected to, themselves, mediate students’ long-term academic achievement and eventual high school graduation (which are beyond the scope of the present evaluation). Table 6. Outcomes by Domain, Measure, Baseline Measure, Timing, and Data Source Student outcome by domain Outcome measure Baseline measure Data source Social-emotional learning and behavior Overall socioemotional competence Social-Emotional Total (SET) score (Quarter 4 [Q4]) Social-Emotional Total (SET) score (Quarter 1) Devereux Student Strengths Assessment- mini (teacher respondent) Growth mindset Growth mindset scale (Q4) CORE SEL Skill items included on survey (student respondent) Self-efficacy Self-efficacy scale (Q4) Self-management Self-management scale (Q4) Social awareness Social awareness scale (Q4) Academic achievement Grades Semester 2 core GPA (A-F, 4-0) and dichotomous measure (No Ds/Fs in core courses) (Q3 & Q4) Semester 2 core GPA (A-F, 4-0) (Prior year Q3 & Q4) Administrative data (district) Engagement in school Attendance Proportion of school days attended and dichotomous measure (Attended at least 90%) (Q3 & Q4) Proportion of school days attended (Prior year Q3 & Q4) Administrative data (district) Suspension Received at least one suspension (Q3 & Q4) Received at least one suspension (Prior year Q3 & Q4) Administrative data (district) Note: Data collection by quarter will be the same for schools implementing in the 2021/22 and 2022/23 school year. SRI will perform a cost analysis, documenting the costs of implementing 4S overall and per student by school. SRI will collect cost information using the ingredients method, price the ingredients, generate cost estimates, and create cost-effectiveness ratios of program costs to estimated impacts on student outcomes (Levin et al., 2017). To do so, SRI will draw on the CostOut tool kit (Hollands et al., 2015) and the IES cost analysis tool kit (IES, 2020a). The timeline for implementation, cost, and outcomes data collection is in Appendix I.6. PR/Award # S411B200026 Page e47 34 References Allensworth, E. (2013). The use of ninth grade early warning indicators to improve Chicago schools. Journal of Education for Students Placed at Risk, 18(1), 68-83. https://doi.org/10.1080/10824669.2013.745181 Allensworth, E. M., & Easton, J. Q. (2005). The on-track indicator as a predictor of high school graduation. Consortium on Chicago School Research at the University of Chicago. http://ccsr.uchicago.edu/sites/default/files/publications/p78.pdf Allensworth, E. M., Gwynne, J. A., Moore, P., & de la Torre, M. (2014). Looking forward to high school and college: Middle grade indicators of readiness in Chicago Public Schools. Consortium on Chicago School Research at the University of Chicago. https://consortium.uchicago.edu/publications/looking-forward-high-school-and-college- middle-grade-indicators-readiness-chicago Allensworth, E., & Easton, J. (2007). What matters for staying on-track and graduating in Chicago public high schools. Consortium on Chicago School Research at the University of Chicago. https://consortium.uchicago.edu/publications/what-matters-staying-track- and-graduating-chicago-public-schools Arora, S., Kalishman, S., Thornton, K., Komaromy, M., Katzman, J., Struminger, B., & Rayburn, W. F. (2016). Project ECHO (Project Extension for Community Healthcare Outcomes): A national and global model for continuing professional development. Journal of Continuing Education in the Health Professions, 36, S48-S49. https://doi.org/10.1097/CEH.0000000000000097 PR/Award # S411B200026 Page e50 35 Aspen Institute. (2019). From a nation at risk to a nation at hope: Recommendations from the National Commission on Social, Emotional, & Academic Development. http://nationathope.org/report-from-the-nation/ Atwell, M. N., Balfanz, R., Bridgeland, J., & Ingram, E. (2019). Building a Grad Nation: Progress and challenge in raising high school graduation rates. Civic and Everyone Graduates Center at the School of Education at Johns Hopkins University. http://new.every1graduates.org/wp- content/uploads/2019/06/2019_BuildingaGradNation_FINAL.pdf Atwell, M. N. & Bridgeland, J. (2019). Ready to lead: A 2019 update of principals’ perspectives on how social and emotional learning can prepare children and transform schools. Civic with Hart Research Associates. https://casel.org/wp-content/uploads/2019/10/Ready-to- Lead_FINAL.pdf Balfanz, R., & Byrnes, V. (2020). Connecting social-emotional development, academic achievement, and on-track outcomes: A multi-district study of grades 3 to 9 students supported by City Year Americorps members. Everyone Graduates Center at the Johns Hopkins University School of Education. http://new.every1graduates.org/wp- content/uploads/2020/05/201200507_EGC_CityYearReport_BalfanzByrnesFINAL.pdf Balfanz, R., Herzog, L., & Mac Iver, D. J. (2007). Preventing student disengagement and keeping students on the graduation path in urban middle-grades schools: Early identification and effective interventions. Educational Psychologist, 42(4), 223–235. https://doi.org/10.1080/00461520701621079 PR/Award # S411B200026 Page e51 36 Belfield, C., Bowden, B., Klapp, A., Levin, H., Shand, R., & Zander, S. (2015). The economic value of social and emotional learning. The Center for Benefit-Cost Studies in Education. http://www.casel.org/wp-content/uploads/2016/09/SEL-Revised.pdf Berry, B. (2017). Micro-credentials: The badges of professional growth. The Education Digest, 82(9), 21. Bos, J. M., Dhillon, S., & Borman, T. (2019). Building Assets and Reducing Risks (BARR) validation study: Final report. American Institutes for Research. https://www.air.org/sites/default/files/downloads/report/AIR-BARR-Validation-Final- Report-July-2019.pdf Bowers, A. J., Sprott, R., & Taff, S. A. (2013). Do we know who will drop out? A review of the predictors of dropping out of high school: Precision, sensitivity, and specificity. The High School Journal, 96(2), 77-100. https://doi.org/10.1353/hsj.2013.0000 CASEL. (2015). Effective social and emotional learning programs: Middle and high school edition. https://casel.org/middle-and-high-school-edition-casel-guide/ Claro, S. & Loeb, S. (2019). Self-management skills and student achievement gains: Evidence from California’s CORE districts. PACE. https://files.eric.ed.gov/fulltext/ED600478.pdf Consortium on Chicago School Research (CCSR). (2007). Freshman year: The make-it or break-it year [Teacher edition]. https://consortium.uchicago.edu/sites/default/files/2018- 10/2751whatmatters-teacherfinal.pdf Dong, N., & Maynard, R. (2013). PowerUp!: A tool for calculating minimum detectable effect sizes and minimum required sample sizes for experimental and quasi-experimental design studies. Journal of Research on Educational Effectiveness, 6(1), 24-67. https://doi.org/10.1080/19345747.2012.673143 PR/Award # S411B200026 Page e52 39 http://www.cbcsecosttoolkit.org https://www.mathematica.org/our-publications-and-findings/publications/impacts-of- dropout-prevention-programs Immordino-Yang, M. H., Darling-Hammond, L., & Krone C. (2018). The brain basis for integrated social, emotional, and academic development: How emotions and social relationships drive learning. Aspen Institute: National Commission on Social, Emotional, and Academic Development. https://www.aspeninstitute.org/publications/the-brain-basis- for-integrated-social-emotional-and-academic-development/ Institute of Education Sciences. (2019a). Review of individual studies protocol, version 4.0. Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncee/wwc/Docs/ReferenceResources/wwc_ris_protocol_v4.pdf Institute of Education Sciences. (2019b). Review protocol for supportive learning interventions, version 4.0. Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncee/wwc/Docs/ReferenceResources/wwc_sle_protocol_v4.0_508.pdf Institute of Education Sciences. (2020a). Cost analysis: A starter kit (IES 2020-001). Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/seer/pdf/IES_Cost_Analysis_Starter_Kit_V1.pdf Institute of Education Sciences. (2020b). What Works Clearinghouse standards handbook, version 4.1. Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncee/wwc/Docs/referenceresources/WWC-Standards-Handbook-v4-1- 508.pdf PR/Award # S411B200026 Page e55 40 Johnson, V. L., Simon, P., & Mun, E. (2014). A peer-led high school transition program increases graduation rates among Latino males. Journal of Educational Research, 107(3), 186–196. https://doi.org/10.1080/00220671.2013.788991 Keaton, P. (2012). Numbers and types of public elementary and secondary schools from the Common Core of Data: School year 2010–11 (NCES 2012-325rev). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/pubsearch/pubsinfo.asp?pubid=2012325rev Komaromy, M., Duhigg, D., Metcalf, A., Carlson, C., Kalishman, S., Hayes, L., Burke, T., Thornton, K., & Arora, S. (2016). Project ECHO (Extension for Community Healthcare Outcomes): A new model for educating primary care providers about treatment of substance use disorders. Substance Abuse, 37(1), 20-24. https://dx.doi.org/10.1080/F08897077.2015.1129388 Levin, H. M., McEwan, P. J., Belfield, C., Bowden, A. B., & Shand, R. (2017). Economic evaluation in education: Cost-effectiveness and benefit-cost analysis. SAGE Publications. Liljengren, J., Balfanz, R., & Howell, G. (2017). Early warning systems video training system. Everyone Graduates Center, Johns Hopkins University. https://www.hsredesign.org/ews- videos/ Lindqvist, E., & Vestman. R. (2011). The labor market returns to cognitive and non-cognitive ability: Evidence from the Swedish enlistment. American Economic Journal: Applied Economics, 3(1), 101-128. http://doi.org/10.1257/app.3.1.101 Loeb, S., Christian, M. S., Hough, H. J., Meyer, R. H., Rice, A. B., & West, M. R. (2018). School effects on social-emotional learning: Findings from the first large-scale panel survey of students [Working paper]. Policy Analysis for California Education. PR/Award # S411B200026 Page e56 41 https://eric.ed.gov/?id=ED591089 Mac Iver, M. A., & Messel, M. (2013). The ABCs of keeping on track to graduation: Research findings from Baltimore. Journal of Education for Students Placed at Risk, 18(1), 50-67. http://dx.doi.org/10.1080/10824669.2013.745207 Mac Iver, M. A., Stein, M., L., Davis, M. H., Balfanz, R., & Fox, J. (2019). An efficacy study of a ninth grade early warning indicator intervention. Journal of Research on Educational Effectiveness, 12(3), 363-390. http://dx.doi.org/10.1080/19345747.2019.1615156 Meyer, R. H., Wang, C., & Rice, A. (2018). Measuring students’ social-emotional learning among California’s CORE districts: An IRT modelling approach. Policy Analysis for California Education (PACE). http://www.edpolicyinca.org/publications/sel- measurement Morgan, K. L., & Rubin, D. B. (2012). Rerandomization to improve covariate balance in experiments. The Annals of Statistics, 40(2), 1263-1282. Naglieri, J. A., LeBuffe, P., & Shapiro, V. B. (2011). Universal screening for social-emotional competencies: A study of the reliability and validity of the DESSA-mini. Psychology in the Schools, 48(7), 660-671. https://doi.org/10.1002/pits.20586 Neild, R. C., Stoner-Eby, S., & Furstenberg, F. (2008). Connecting entrance and departure: The transition to ninth grade and high school dropout. Education and Urban Society, 40(5), 543-569. https://doi.org/10.1177/0013124508316438 Orfield, G. (2004). Dropouts in America: Confronting the graduation rate crisis. Harvard Education Press. Osher, D., Kidron, Y., Brackett, M., Dymnicki, A., Jones, S., & Weissberg, R. P. (2016). Advancing the science and practice of social and emotional learning: Looking back and PR/Award # S411B200026 Page e57