Using Algorithms to Track Students: A Seven-College Experiment, Essays (university) of Algorithms and Programming

Download Using Algorithms to Track Students: A Seven-College Experiment and more Essays (university) Algorithms and Programming in PDF only on Docsity! 1 Using Algorithms to Track Students: A Seven-College Experiment∗ Peter Bergman, Elizabeth Kopko, Julio Rodríguez‡ November 2022 ABSTRACT Tracking is widespread in education. In U.S. post-secondary education alone, at least 71% of colleges use a test to track students into various courses. However, there are concerns that placement tests lack validity and unnecessarily reduce education opportunities for students from under-represented groups. While research has shown that tracking can improve student learning, inaccurate placement has consequences: students face misaligned curricula and must pay tuition for remedial courses that do not bear credits toward graduation. We develop an alternative system that uses algorithms to predict college readiness and track students into courses. Compared to the most widely-used placement test system in the country, the algorithms are more predictive of future performance. We conduct an experiment across seven colleges to evaluate the effects of algorithmic placement. Placement rates into college-level courses increase substantially without reducing pass rates. Adjusting for multiple testing, algorithmic placement generally, though not always, narrows gaps in college placement rates and remedial course taking across demographic groups. A detailed cost analysis shows that algorithmic placement is socially efficient: it increases college credits earned while saving costs for students, which more than offsets increased costs for colleges. Costs could be reduced with improved data digitization, as opposed to entering data by hand. ∗ The research reported here was undertaken through the Center for the Analysis of Postsecondary Readiness and supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305C140007 to Teachers College, Columbia University. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education. ‡ Teachers College, Columbia University. Numerous people—including several from the Community College Research Center at Teachers College, Columbia University—have helped make this work happen. We particularly thank Elisabeth Barnett for her leadership, as well as Clive Belfield, Magdalena Bennett, Dan Cullinan, Vikash Reddy, and Susha Roy for their contributions. We also thank Judy Scott-Clayton, Avi Feller, Peter Hull, Sarah Turner, Will Dobbie, and seminar participants for their comments. 2 1. Introduction Tracking students by prior test scores is widespread in U.S. education. In higher education alone, at least 71% of post-secondary institutions use test scores to track students (National Center for Public Policy and Higher Education and Southern Regional Education Board, 2010; Fields and Parsad, 2012).1 These rates are higher in two-year colleges, which enroll nearly half of all post-secondary students but fewer than 40% of whom receive any credential (Bound, Lovenheim, and Turner, 2010; Fields and Parsad, 2012; Chen, 2016; Denning et al., 2022).2 While research has demonstrated large potential benefits of tracking (Duflo et al., 2011; Banerjee et al., 2016; Banerji and Chavan, 2016; Card and Giuliano, 2016; Banerjee et al., 2017), inaccurate placement has consequences: students face misaligned curriculum, and, in higher education, must pay tuition for remedial courses that do not bear credits toward graduation. The potential for inaccurate placement is a concern because there is evidence that several widely-used tests lack validity and unnecessarily reduce education opportunities for students from under-represented groups (Rothstein, 2004; Scott-Clayton et al., 2014). Given most placement tests aim to predict students’ readiness for college-level courses, using an algorithm with multiple inputs, such as high school GPA, to formulate predictions could improve validity (Scott-Clayton et al., 2014; Mullainathan and Spiess, 2017).3 However, algorithmic screening often raises its own concerns about fairness.4 In this paper, we develop and evaluate placement algorithms to track students into college-level courses and implement them via an experiment across seven colleges and 12,544 college students. We recruited community colleges across New York and gathered historical data on their students to estimate models predicting students’ likelihood of passing college-level math and English courses. These predictions 1 Outside the U.S., test scores are frequently used as the sole criterion for admission to colleges (Hastings et al., 2013; Kirkeboen et al., 2016; MacLeod et al., 2017; Riehl, 2019; Aguirre et al., 2020). 2 Community college enrollment has also been significant driver of increases in student borrowing (Chakrabarti, Lovenheim and Morris, 2016). 3 Alternatively, a measure could be constructed to predict treatment effects of specific course placements as opposed to pass rates or readiness. In practice, targeting treatment effects does not seem to be how colleges try to optimize placement systems. 4 For example, this opinion piece in The Washington Post by the director of the American Civil Liberties Union’s Racial Justice Program cites the impact of AI in different contexts, including college admissions. 5 significantly larger for female students in math relative to male students and Black students in English relative to white students. Lower-income students take fewer remedial credits relative to higher-income students. However, the increases in placement into college-level math, though positive and significant, are not as large for Hispanic students compared to white students. In terms of fairness, colleges were concerned algorithmic placement would differentially affect pass rates across subgroups relative to the status quo. The algorithm does not consider any protected characteristics. However, algorithmic placement could reduce or improve pass rates for some groups and not others relative to test-score placement. Looking across multiple subgroups, we find that pass rates in college-level courses remain extremely similar to pass rates in the test-score placement system. Thus placement rates and credit accumulation improve across subgroups without reducing pass rates. The algorithmic placement system also results in cost savings for students. We conducted a detailed cost analysis for colleges and students, separating fixed and variable costs, and costs to students versus costs to colleges. We find that students saved $150, on average, relative to the test-score placement system, which is due to reductions in remedial course taking. This implies an average saving to students equal to $145,200 per cohort, per college in our sample. For colleges to implement such a placement algorithm, decision makers must weigh the potential benefits to students against the costs to the colleges. We estimate that the cost per student in the initial year of the study—above and beyond the test-score placement system—is $70 to $360 dollars, depending on the college. Much of these costs are driven by the need to hand enter data from high school transcripts. Simple process enhancements, such as requesting GPAs on applications, or data-transfer relationships with high schools, could simply and greatly reduce the cost of this data collection. Furthermore, the first year of implementation also involves large fixed costs. We estimate operating costs of the placement algorithm are $40 dollars per student. Our paper contributes to research arguing that data-driven algorithms can improve human decision making and reduce biases (Mullainathan and Spiess, 2017; Li et al., 2020; Arnold et al., 2021; Arnold et al., 2022). Kleinberg et al. (2018) show that a machine-learning algorithm has the potential to reduce bias in bail decisions relative to judges’ decisions alone. At the same time, others are concerned that these algorithms 6 could embed biases into decision making and exacerbate inequalities (Eubanks, 2018). We contribute to this literature by comparing the impacts of a simple, data-driven algorithm to another quantitative measure, test scores. We then evaluate the algorithm by conducting a large-scale experiment. Our paper also relates to a broader literature on tracking students. Historically, tracking is controversial. Oakes (1985) argues that the evidence on tracking is inconsistent, and, in practice, higher-track classes tend to have higher-quality classroom experiences than lower-track classes. More recently, Duflo, Dupas and Kremer (2011) randomize students in Kenya to schools that either tracked students by test scores or assigned students randomly to classrooms. They find that test scores in schools with tracking improved relative to the control group both for students placed in the higher- scoring and the lower-scoring tracks. Card and Giuliano (2016) study a district policy in which students are placed into classrooms based on their test scores. This program caused large increases in the test scores of Black and Hispanic students. Multiple studies look at the effects of being placed into a higher track versus a lower track. Bui et al., (2014) and Card and Giuliano (2014) find that gifted students’ placement into advanced coursework does not change test scores. Cohodes (2020) and Chan (2020), however, find increases in enrollment in advanced high-school coursework and college. 7 In higher education, there is evidence of negative effects on course completion and graduation for marginally admitted students in high-ability classes (de Roux and Riehl, 2022). On the other hand, the evidence that placement into remedial courses improves academic outcomes for marginal students is more mixed, and several regression-discontinuity analyses find no effects (Calcagno and Long, 2008; Bettinger and Long, 2009; Boatman and Long, 2010; Martorell and McFarlin, 2011; Allen and Dadgar, 2012; Hodara, 2012; Scott-Clayton and Rodriguez, 2015). The rest of our paper proceeds as follows. Section 2 provides further background information about tracking in postsecondary institutions and study implementation. Section 3 describes the experimental design, data and empirical strategies. Section 4 7 Several other studies look at the effects of placing into high-test score schools and the results are much more mixed (Jackson, 2010; Pop-Eleches and Urquiola, 2013; Abdulkadiroğlu et al., 2014; Dobbie and Fryer, 2014). 7 presents our findings. Section 5 provides a detailed cost analysis, and Section 6 concludes. 2. Background, Site Recruitment, Algorithm Implementation Tracking students into remedial education is a major component of the higher education system, both in terms of enrollment and cost. In the 2011-12 academic year, 41% of first and second-year students at four-year institutions had taken a remedial course, while at two-year institutions, even more—68% of students—had taken a remedial course (Chen, 2016). The cost of remedial education has been estimated to be as much as $2.9 billion annually (Strong American Schools, 2008). The primary purpose of remedial education is to provide differentiated instruction to under-prepared students, so they have the skills to succeed in college-level coursework (Bettinger and Long, 2009). However, there is evidence that community-colleges’ tracking systems frequently “under place” students—tracking them into remedial courses when they could have succeeded in college-level courses—and “over place” students—tracking them into college-level courses when they were unlikely to be successful (Belfield and Crosta, 2012; Scott-Clayton, 2012). Most institutions administer a multiple-choice test in mathematics, reading, and writing to determine whether incoming students should be placed into remedial or college-level courses. The ACCUPLACER, a computer-adaptive test offered by the College Board, is the most widely-used college placement system in the U.S. (Barnett and Reddy, 2017). Colleges choose a cut score for each test and place students scoring above this score into college-level courses and students below the cut score into various remedial courses.8 Given the placement rules and immediate test results provided by the ACCUPLACER platform, students often learn their placement immediately after completing their exam. Site Selection and Descriptions All the participating colleges are part of the State University of New York (SUNY) system, ranging from large to small, and students’ backgrounds vary from college to 8 Certain colleges may offer exemptions from testing. For instance, this can occur for students who speak English as a second language or who have high SAT scores. 10 years of historical data excluding the most recent year, and then examined fit criteria using data from that most recent year.9 Conditional on placement into the college-level course, exam scores explain very little variation in English course outcomes but more variation in math outcomes; including additional measures adds explanatory power.10 Appendix Figures A.1 and A.2 list the full set of variables used by each college to calculate students’ math and English algorithm scores, respectively. Tables A.2 and A.3 show typical examples of our regression results for math and English. Across colleges, explanatory power is much higher for math course grades than for English course grades. Placement scores typically explain less than 1% of the variation in passing grades for English. Test scores are better predictors for passing math grades, explaining roughly 10% of the variation. This pattern is likely due to the consistency in content between coursework across colleges in math and math tests. ELA college courses may exhibit more heterogeneity across colleges. Prior research has also shown that math exams tend to create stronger incentives for test prep (Riehl, 2019; Riehl and Welch, 2022). Adding high school grades typically explains an additional 10% of the variation in both subjects. We find that indicators for which high school a student attended, which could reflect different grading standards, add little predictive value. Overall, combining multiple measures with predictive analytics is no panacea for predicting future grades, but it does improve the validity of the placement instrument relative to test scores alone. Setting cut probabilities After we selected the final models, we used the coefficients from the regression to simulate placement rates for each college using their historical data. Consider the 9 The focus of this analysis was the overall predictive power of the model. As such, we calculated the Akaike Information Criterion (AIC) statistics for each model (varying variables in 𝐗𝐗𝑖𝑖). The AIC is a penalized-model-fit criterion that combines a model’s log-likelihood with the number of parameters included in a model (Akaike, 1998; Burnham and Anderson, 2002; Mazerolle, 2004; Hastie et al., 2009). Under certain conditions, choosing model specifications according to the AIC is asymptotically equivalent to leave-one-out-cross-validation (Stone, 1977). In practice, we did not have many variables to select from and higher-order and interaction terms had little effect on prediction criteria (and additional complexity was difficult to implement). 10 The unconditional explained variation differs from what we show in the appendix table. For context, pass rates average are around 55% in math and 61% in English. 11 following simplified example where a placement test score (R) and high school GPA (G) are used to predict success in college-level math (Y), defined as earning a grade of C or better. The regression coefficients combined with data on R and G can then predict the probability of earning a C or better in college-level math for incoming students (Y�). A set of decision rules must then be determined based on these predicted probabilities. A hypothetical decision rule would be: Placement𝑖𝑖 = �College Level if Y�𝑖𝑖 ≥ 0.6 Remedial if Y�𝑖𝑖 < 0.6 For each college, we generated spreadsheets projecting the share of students that would place into college-level coursework at any given cut-point as well as the share of those students we would anticipate earning a C or better. These spreadsheets were provided to colleges so that faculty in the pertinent departments could set cut-points for students entering their programs. Figure 1 shows an abbreviated, hypothetical example of one such spreadsheet provided to colleges.11 The top panel shows math placement statistics and the bottom panel shows statistics for English. The highlighted row shows the status quo at the college and the percent of tested students placed into college level is shown in the second column. For instance, for math, the status quo placement rate in a college-level course is 30%. The third column shows the pass or success rate, which is a grade “C” or better in the first college-level course in the relevant subject. In this example, the status-quo pass rate for math is 50% conditional on placement into the college-level math course. Below the highlighted row, we show what would happen to placement and pass rates at different cut points for placement. The first column shows these cut points (“Minimum probability of success”). For instance, for math, the first cut point we show is 45%, which implies that for a student to be placed into college-level math under the algorithm, the student must have a predicted probability of receiving a “C” or better in the gate-keeper math course of at least 45%. If this 45% cut point is used, columns two and three show what would happen to the share of students placed into college-level math under the algorithm (column two) and what would happen to the share who 11 In practice, we showed results from many different cut points. 12 would pass this course conditional on placement (column three). In this example, for math, if the 45% cut point is used, the algorithm would place 40% of students into college-level math and we anticipate 60% of those students would pass. The cut point differs from the expected pass rate because the cut point is the lowest probability of passing for a given student: the cut point implies that every student must have that probability of passing or greater. For instance, if the cut point is 40%, then every student has 40% chance or greater of passing the college-level course. Therefore, most students placed into college-level courses according to this rule will have above a 40% chance of passing the course. Faculty opted to create placement rules that kept pass-rates in college-level courses the same as historical pass rates. In general, this choice implied increases in the predicted number of students placed into college-level coursework. For instance, in the example, the status quo placement and pass rates for English are 60% and 40%, respectively. A cut point of 45% would induce the same pass rate, 60%, but would place 75% of students into the college-level English course. Installation of new placement method in college systems We developed two procedures to implement the algorithms while maintaining the timing of placement decisions. At colleges running our algorithm through the computerized ACCUPLACER-test platform, we programmed custom rules into the ACCUPLACER platform for students selected to be part of the treatment group.12 These rules created the weights on various student characteristics that, when combined with the colleges’ thresholds for placement, produced a placement recommendation for a student. Other colleges ran their placement through a custom server built for the study. Student information was sent to servers to generate the probability of success and the corresponding placement, which was returned to the college. 3. Experimental Design, Data, Empirical Strategy The sample frame consisted of entering cohorts (fall and spring) enrolling at each college 12 As mentioned above, this process placed constraints on the algorithm’s complexity. Interaction terms and non-linear models, for instance, were difficult to implement within the ACCUPLACER system. 15 [2] Y𝑖𝑖𝑖𝑖 = α + βTreatment𝑖𝑖𝑖𝑖 + φ𝑖𝑖 + 𝐗𝐗𝑖𝑖𝑖𝑖 ′ η + 𝐙𝐙𝑖𝑖𝑖𝑖 ′ δ + ε𝑖𝑖𝑖𝑖, where Y𝑖𝑖𝑖𝑖 are academic outcomes for student i in college c, such as placement into a college-level course and passing a college-level course; Treatment𝑖𝑖𝑖𝑖 indicates whether the individual was randomly assigned to be placed using the algorithmic placement system or the test-score placement system; φ𝑖𝑖 are college (strata) fixed-effects; 𝐗𝐗𝑖𝑖𝑖𝑖 is a vector of baseline covariates (gender, race, age, financial aid status); 𝐙𝐙𝑖𝑖𝑖𝑖 is students’ math and English algorithm scores, which are baseline measures of academic preparedness, and ε𝑖𝑖𝑖𝑖 is the error term. The coefficient of interest is β, which is the effect of assignment to the placement algorithm on outcomes discussed above. We estimate Huber-White- Heteroskedasticity robust standard errors following the experimental design (Huber, 1967; White, 1980; Abadie et al., 2020). As not everyone takes a placement exam in both subjects, we estimate these regressions for those who took any placement exam (which cannot be affected by algorithmic assignment), and therefore are assigned to placement by the algorithm for one or two courses. We also estimate these regressions for those who took placement exams in both subjects and therefore can be assigned to placement by the algorithm for two subjects. Treatment Compliance Because not everyone follows their recommended placement, we also estimate the effect of treatment assignment on compliance with the algorithm’s placement recommendation independently of being in the treatment or control group. We observe the algorithm’s and the test-score placement system’s recommendations for each student in our dataset, so compliance is defined as an indicator equal to one for students following the algorithm's placement recommendation and zero otherwise.14 For students who took both math and English exams, compliance is defined as following the algorithm's 14 Therefore, compliance is equal to zero for students in the control group if they follow their test-score placement recommendation (the business as usual), regardless of whether it is the same as the algorithmic recommendation. 16 recommendation in at least one subject. These results are more to inform implementation, and we do not present treatment-on-the-treated results.15 Subgroup Analyses We also study several differential effects of the placement algorithm has on the composition of students placed into remedial and college-level courses. We estimate equation [2] above for each subgroup and also test the significance of the interaction terms, shown below. [3] Y𝑖𝑖𝑖𝑖 = α𝑘𝑘 + β1𝑘𝑘Treatment𝑖𝑖𝑖𝑖 + β2𝑘𝑘Treatment𝑖𝑖𝑖𝑖 × Subgroup𝑘𝑘 + γ𝑖𝑖 + 𝐗𝐗𝑖𝑖𝑖𝑖 ′ η𝑘𝑘 + 𝐙𝐙𝑖𝑖𝑖𝑖 ′ δ𝑘𝑘 + μ𝑖𝑖𝑖𝑖 The outcomes, Y𝑖𝑖𝑖𝑖, are placement in college-level math, placement in college-level English, and credit accumulation. For each k subgroup of interest, we restrict the sample to the reference group and the subgroup. Therefore, the coefficient β1𝑘𝑘 shows the effect for the reference group (listed below), and the coefficient of particular interest is the significance and magnitude of β2𝑘𝑘, which indicates whether the difference between groups of students is widening or narrowing because of algorithmic placement. The subgroups of interest are Black students and Hispanic students compared to white students; female students compared to male students; and Pell recipients compared to non-Pell recipients. This process yields many tests, which increases the likelihood of type-I errors. To control for the Family Wise Error Rate, we use a simple step-down procedure formulated by Holm (1979). Treatment-Control Baseline Balance Randomization should ensure that, in expectation, students assigned to the treatment group are similar to those assigned to the control group. Table 2 provides evidence that random assignment was successfully implemented. Participants’ demographic and academic characteristics are balanced across treatment and control groups. Students’ ACCUPLACER exam scores also are similar across both groups. Overall, the magnitudes of differences between treatment and control groups are small and only one is significant at the 5 percent level, which is unsurprising given the more than 20 15 Treatment-on-the-treated coefficients can be computed as the estimated ITT coefficient divided by the compliance difference between the treatment and the control group (i.e., 1st-stage coefficient) (Imbens and Angrist, 1994). Significance levels for our outcomes remain the same. 17 variables tested. Though not shown, this balance also holds for the subgroup of students who took both the English and math placement exams as well. 4. Results Descriptive Changes in Placements We begin with a descriptive summary of placement changes to show the various ways the algorithm changed students’ placements relative to the test-score placement system. As stated above, it is not obvious how the algorithm will change net placement rates. Table 3 summarizes these changes for students placed by the algorithm. Of the more than 6,000 students assigned to the program-group, 82% were tracked in math and 100% were tracked in English. Among those students who took a math placement exam, 23% experienced a math placement different from what would have been expected under the test-score placement system. Of those with a changed math placement, 67% were placed into a higher-level math course than would have been expected under the test- score placement system, and 33% placed in a lower-level math course. Of those who took the English placement exams, approximately 55% of program-group students experienced a change in the level of their English level placement, of which 90% placed into a higher-level English course and 10% placed into a lower-level course than they would have under the test-score placement system. Table 4 shows compliance with algorithm’s placement recommendations. Overall, the treatment group complies with their algorithmic placement recommendation 80% of the time. Treatment assignment increases compliance with the algorithm’s decision relative to the control group by 76 percentage points. The first stage is slightly lower for the spring cohort, when there are fewer first-time enrollees, but is generally consistent. Students may be placed into a particular course, but they may decide to delay enrollment and enroll in courses in different subjects. Treatment Effects on Placement, Course Taking, and Credits Algorithmic placement caused increases in placement into college-level courses, enrollment in college-level courses, and total college-level credits earned. Table 5 summarizes the first-term results. Students assigned to the placement algorithm are 6.6 percentage points more likely to be placed into a college-level math course, 2.6 percentage points more likely to enroll in a college-level math course, and 1.9 percentage 20 earned also decrease for all subgroups (including male students), and college credits increase a statistically significant amount only for Black students, female students, and Pell-grant recipients. College administrators were interested in how the algorithmic placement system affected gaps in placement rates across subgroups. Given these administrators are making the decisions to maintain the system, we focus our analysis of heterogeneous effects on the extent to which algorithmic placement widened or narrowed gaps in key outcomes across subgroups. This question implies we are interested in the interaction terms from equation [3], which assess whether there are differential effects for Black and Hispanic students (separately) relative to white students, female students relative to male students, and Pell recipients relative to non-Pell recipients. Including outcomes in placement for math and English and credit accumulation in remedial and college-level courses, there are 16 interaction terms of interest. We use the step-down method from Holm (1979) to (conservatively) control for the Family-wise Error Rate at the five percent level. Four interaction terms remain significant after this adjustment. Placement rates for Black students into college-level English increased relative to white students and placement rates for female students into college-level math relative to male students increased as well. Though placement rates into college-level math and English courses increased for Hispanic students overall, relative to white students, the increase in math is smaller than it is for white students. Lastly, the decrease in remedial credits is significantly smaller for Pell recipients than it is for non-Pell recipients. Thus, though all students seem to benefit from algorithmic placement, there is evidence that many of the benefits accrue to students traditionally under-represented in college courses. In terms of fairness, colleges were concerned that algorithmic placement would negatively affect pass rates in college courses for certain subgroups. As mentioned, relative to the test-score placement system, college-level pass rates remained constant overall under the new placement system. While the algorithm did not include any group membership indicator (i.e., there is no disparate treatment), algorithmic placement could have differential effects on college-level pass rates across subgroups. To check this, we independently estimate equation [2] for each subgroup on pass rates and present the results in separate columns of Table 10. While there seems to be a slight reduction in pass rates for white students in English and male students in math 21 and English, the few (4 out of 28) statistically significant coefficients become non- significant (at the five percent level) after adjusting for multiple testing using the Holm’s step-down method. Overall, pass rates in college-level courses are similar across subgroups. This result suggests that the new placement system is largely fair—by this metric—across subgroups. 5. Cost Analysis In this section, we present the cost-effectiveness analysis for the algorithmic placement system and the test-score placement systems for six colleges using the ingredients method (Levin et al., 2017). We could not collect complete cost data at one college.17 The cost estimates reflect the annual expected cost during the first five years of implementing and operating the new placement system at college of similar size and organization as the six sample colleges. Algorithmic placement resulted in cost savings for students: students earned more college credits and took fewer remedial credits with a net effect of lower tuition payments. Relative to the test-score placement system, implementation and operation costs were larger for colleges, $140 per student; operating costs, however, are $40 per student over the status quo. Overall, algorithmic placement is more cost-effective from a social perspective than the existing placement systems. That is, while the implementation and operating costs are larger for colleges, the cost reduction for students more than offsets the increased cost to colleges, so total costs are lower for the algorithmic placement system. Moreover, costs could be reduced substantially if data to estimate the algorithms did not have to be hand entered and if data collection were centralized into a single system. We detail the calculations underpinning these findings below. Defining Costs and Cost Data To better understand the details of our cost-effectiveness analysis, we start by defining several terms. First, fixed costs are those costs that do not vary with college enrollment. Direct costs are the costs of implementing and operating the placement system. 17 What we could collect does not suggest this seventh college had costs significantly different from the others, but personnel changes prevented us from collecting all the necessary data. 22 Implementation costs include one-time costs incurred to develop and test the placement method (e.g., evaluator time) and the operating costs to keep it fully functional. Operating costs refer to running a placement system after the initial method has been developed and tested (i.e., personnel, facilities, administering placement test, etc.). Indirect costs are associated with the price and quantity of credits attempted by the students. The total costs are the sum of the indirect and direct costs. Student costs include only the cost of the credits attempted and not the direct costs, as students do not pay for the additional costs of implementing the algorithmic placement system. In contrast, college costs include direct costs of implementing the alternative system and any costs from course offerings (e.g., changes in the number of remedial courses offered). Finally, cost-efficiency, in our context, compares the costs of the algorithmic placement system to the test-score placement system (Levin et al., 2017). We collected data on ingredients from two primary sources. One source for this information was from direct interviews with faculty and staff who implemented the new testing protocols. The second source for input prices and overhead costs was from secondary sources, such as the Integrated Postsecondary Education Data System (IPEDS), described below. Sources of Costs in the Placement Systems Understanding the different cost components of the placement systems helps to distinguish fixed costs from operating costs. The initial investment to implement the algorithm has three components. First, data on students’ characteristics (including high school transcripts), placements based on test results, and subsequent college outcomes must be collected. In some colleges, these data are already available, but other colleges required more extensive data collection. Second, data must be analyzed to estimate the new placement algorithm. Third, resources must be allocated to create and implement the new system within the college, which includes training personnel. After the initial investment, implementation requires collecting data from entering students and personnel to assign students to either remedial or college-level courses. For the algorithm, one driver of costs was data entry. Data entry costs were lower if the college had all high school information pre-loaded into their databases. In contrast, data entry costs were higher if each student’s information had to be entered into the computing system individually. 25 and administrators asked whether students could be placed via the algorithm as accurately without using these test scores. We examined the extent to which the algorithm would place students differently if test scores were not used for prediction. We find that placement rates would change substantially for math courses—by 18%— however, for English courses, only 5% to 8% of placements would change. This finding is in line with the increased predictive value we find for math test scores over English test scores. 6. Conclusion Our findings indicate that algorithmic placement, which incorporates multiple measures to predict college readiness, significantly impacts how colleges track students into either college-level or remedial courses. First, algorithmic placement allows colleges to choose cut points that explicitly target predicted placement rates and pass rates. Second, the algorithm led to changes in the placement of students. Across the seven study colleges, more students were placed into college-level math and English courses—without reducing pass rates in either course. There were particularly large increases in college- level placements in English courses. By several metrics, the algorithm is more accurate than the widely-used, test-score placement system. While the algorithm’s predictive validity is greater than placement scores alone, the algorithms we developed could be improved. Most notably, our model was constrained by implementation in several ways. To produce rapid placement decisions, we had to embed our algorithm into existing systems, which restricted our modeling choices. We could not for instance, implement a non-linear model. Future models could also use richer transcript data. The colleges we worked with could not readily provide course- level high school grades, which could be predictive of future performance as well. More generally, as colleges develop more consistent ways to record incoming student information, the ability to predict future performance should improve. One question is how our results would differ if all students within a college were placed according to the algorithm. Our interviews with college administrators, department chairs, faculty and counselors at each college documented their impressions to the algorithm’s implementation. Generally, there was no perceived change in classroom composition. However, this could change if all students were placed via the algorithm, especially in English courses where placement changes were more significant. 26 Prior research suggests this might result in improved academic outcomes for students (Duflo et al., 2011). Our results have important implications because the high cost of remedial education falls onto students placed into these courses and indirectly onto taxpayers whose money helps subsidize public postsecondary institutions. As a result, there is both a private and social benefit to ensuring that remedial education is correctly targeted. Colleges recognize this, and some have begun to implement these placement algorithms. Long Beach City College (LBCC) created a placement formula that uses student high school achievement in addition to standardized assessment scores. The formula weights each measure based on how predictive it is of student performance in college courses (Long Beach City College, Office of Institutional Effectiveness, 2013). This paper provides evidence that these placement systems not only affect student outcomes through changes in the placement instrument, but also through colleges’ improved ability to target pass rates explicitly. Future research could test more intricate predictive models than we could implement in the current study, and perhaps focus on algorithms that predict treatment effects of each course rather than pass rates. 27 REFERENCES Abadie, A., Athey, S., Imbens, G. W., & Wooldridge, J. M. (2020). Sampling‐Based versus Design‐Based Uncertainty in Regression Analysis. Econometrica, 88(1), 265-296. Abdulkadiroğlu, A., Angrist, J., & Pathak, P. (2014). The elite illusion: Achievement effects at Boston and New York exam schools. Econometrica, 82(1), 137-196. Aguirre, J., Matta, J., & Montoya, A. M. (2020). Joining the men’s club: The returns to pursuing high-earnings male-dominated fields for women. Unpublished Manuscript. Akaike, H. (1998). Information theory and an extension of the maximum likelihood principle. In E. Parzen, K. Tanabe, & G. Kitagwa (Eds.) Selected Papers of Hirotugu Akaike. Springer Series in Statistics (Perspectives in Statistics). New York, NY: Springer. Allen, D., & Dadgar, M. (2012). Does dual enrollment increase students’ success in college? Evidence from a quasi-experimental analysis of dual enrollment in New York City. New Directions for Higher Education, 11–19. Arnold, D., Dobbie, W., & Hull, P. (2021). Measuring racial discrimination in algorithms. In AEA Papers and Proceedings (Vol. 111, pp. 49-54). Arnold, D., Dobbie, W., & Hull, P. (2022). Measuring racial discrimination in bail decisions. American Economic Review, 112(9), 2992-3038. Bacon, D. R., & Bean, B. (2006). GPA in research studies: An invaluable but neglected opportunity. Journal of Marketing Education, 28, 35– 42. Banerjee, A., Banerji, R., Berry, J., Duflo, E., Kannan, H., Mukerji, S., ... & Walton, M. (2017). From proof of concept to scalable policies: Challenges and solutions, with an application. Journal of Economic Perspectives, 31(4), 73-102. Banerjee, A., Banerji, R., Berry, J., Duflo, E., Kannan, H., Mukherji, S., ... & Walton, M. (2016). Mainstreaming an effective intervention: Evidence from randomized evaluations of “Teaching at the Right Level” in India (No. w22746). National Bureau of Economic Research. Banerji, R., & Chavan, M. (2016). Improving literacy and math instruction at scale in India’s primary schools: The case of Pratham’s Read India program. Journal of Educational Change, 17(4), 453-475. Barnett, E. A., Kopko, E., Cullinan, D., & Belfield, C. R. (2020). Who Should Take College-Level Courses? Impact Findings from an Evaluation of a Multiple 30 Hodara, M. (2012). Language Minority Students at Community College: How Do Developmental Education and English as a Second Language Affect Their Educational Outcomes? (Doctoral dissertation). Retrieved from ProQuest Dissertations and Theses. (Accession Order No. [3505981]). Holm, S. (1979). A Simple Sequentially Rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65-70. Huber, P. J. (1967, June). The behavior of maximum likelihood estimates under nonstandard conditions. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability (Vol. 1, No. 1, pp. 221-233). Imbens, G. W., & Angrist, J. D. (1994). Identification and Estimation of Local Average Treatment Effects. Econometrica, 62(2), 467–475. https://doi.org/10.2307/2951620 Kautz, T., Heckman, J. J., Diris, R., ter Weel, B., and Borghans, L. (2014), Fostering and Measuring Skills: Improving Cognitive and Non‐cognitive Skills to Promote Lifetime Success, OECD, Paris. Kautz, T. D., & Zanoni, W. (2014). Measuring and fostering noncognitive skills in adolescence: Evidence from Chicago Public Schools and the OneGoal Program. Unpublished manuscript, Department of Economics, University of Chicago, Chicago, IL. Kirabo Jackson, C. (2010). Do students benefit from attending better schools? Evidence from rule‐based student assignments in Trinidad and Tobago. The Economic Journal, 120(549), 1399-1429. Kirkeboen, L. J., Leuven, E., & Mogstad, M. (2016). Field of study, earnings, and self- selection. The Quarterly Journal of Economics, 131(3), 1057-1111. Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018). Human decisions and machine predictions. The Quarterly Journal of Economics, 133(1), 237-293. Levin, H.M., McEwan, P.J., Belfield, C.R., Bowden, A.B., & Shand. R. (2017). Economic evaluation of education: Cost-effectiveness analysis and benefit-cost analysis. Thousand Oaks, CA: SAGE Publications. Li, D., Raymond, L., & Bergman, P. (2020). Hiring as Exploration (No. w27736). National Bureau of Economic Research. Lakkaraju, H., J. Kleinberg, J. Leskovec, J. Ludwig, and S. Mullainathan (2017). The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables. Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 275–284. 31 Long Beach City College, Office of Institutional Effectiveness. (2013). Preliminary overview of the effects of the fall 2012 Promise Pathways on key educational milestones. Long Beach, CA: Office of Institutional Effectiveness. MacLeod, W. B., Riehl, E., Saavedra, J. E., & Urquiola, M. (2017). The big sort: College reputation and labor market outcomes. American Economic Journal: Applied Economics, 9(3), 223-61. Martorell, P., & McFarlin, I., Jr. (2011). Help or hindrance? The effects of college remediation on academic and labor market outcomes. The Review of Economics and Statistics, 93(2), 436–454. Mazerolle, M. J. (2004). Appendix 1: Making sense out of Akaike’s Information Criterion (AIC): its use and interpretation in model selection and inference from ecological data. Retrieved from http://theses.ulaval.ca/archimede/fichiers/21842/apa.html Mullainathan, S. and Spiess, J. (2017). Machine learning: an applied econometric approach. Journal of Economic Perspectives, 31(2), 87–106. National Center for Public Policy and Higher Education and the Southern Regional Education Board. (2010). Beyond the Rhetoric: Improving College Readiness Through Coherent State Policy. Retrieved from: http://www.highereducation.org/reports/college_readiness/CollegeReadiness.pdf Oakes, J. (1985). Collaborative Inquiry: A Congenial Paradigm in a Cantankerous World. Pop-Eleches, C., & Urquiola, M. (2013). Going to a better school: Effects and behavioral responses. American Economic Review, 103(4), 1289-1324. Riehl, E. (2019). Fairness in College Admission Exams: From Test Score Gaps to Earnings Equality. Riehl, E., & Welch, M. (2022). Accountability, test prep incentives, and the design of math and English exams. Rodríguez, O., Bowden, A.B., Belfield, C.R., & Scott-Clayton, J. (2014) Remedial placement testing in community colleges: What resources are required, and what does it cost? (CCRC Working Paper No. 73). New York, NY: Community College Research Center. Retrieved from: https://ccrc.tc.columbia.edu/media/k2/attachments/remedial-placement-testing- resources.pdf Rothstein, J. M. (2004) College performance predictions and the SAT. Journal of Econometrics, 121(1-2), 2917-317. Scott-Clayton, J. (2012). Do high-stakes placement exams predict college success? (CCRC Working Paper No. 41). New York, NY: Community College Research Center. Retrieved from 32 http://ccrc.tc.columbia.edu/media/k2/attachments/high-stakes-predict- success.pdf Scott-Clayton, J., Crosta, P. M., & Belfield, C. R. (2014). Improving the targeting of treatment: Evidence from college remediation. Educational Evaluation and Policy Analysis, 36(3), 371–393. Scott-Clayton, J., & Rodriguez, O. (2015). Development, discouragement, or diversion? New evidence on the effects of college remediation policy. Education Finance and Policy, 10(1), 4–45. Stone, M. (1977). An asymptotic equivalence of choice of model by cross‐validation and Akaike's criterion. Journal of the Royal Statistical Society: Series B (Methodological), 39(1), 44-47. Strong American Schools. (2008). Diploma to nowhere. Retrieved from: http://paworldclassmath.webs.com/8534051-Diploma-To-Nowhere-Strong- American-Schools-2008.pdf. White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48(4), 817-838. 35 Table 2. Baseline Characteristics by Treatment Assignment Control Mean Treatment Mean Difference (T - C) P-value Obs. Enrollment 86% 85% -1% 0.26 12,544 Female 50% 50% 1% 0.39 11,901 Race White 44% 42% -1% 0.13 12,544 Asian 3% 2% 0% 0.82 12,544 Black 19% 20% 2% 0.03 12,544 Hispanic 20% 20% 1% 0.43 12,544 Pacific Islander 0% 0% 0% 0.93 12,544 Native American 1% 1% 0% 0.46 12,544 Two or more races 4% 3% 0% 0.25 12,544 Race Missing 10% 10% 0% 0.57 12,544 Age at entry 20.94 20.91 -0.02 0.82 12,544 Pell Grant recip. 42% 43% 1% 0.22 12,544 TAP Grant recip. 31% 31% 0% 0.78 12,544 GED recip. 7% 7% 0% 0.98 12,544 HS GPA (100 scale) 77.96 78.12 0.16 0.34 7,869 HS GPA missing 37% 37% 0% 0.77 12,544 ACCUPLACER Exam score Arithmetic 33.6 34 0.4 0.43 10,191 Algebra 48.1 47.9 -0.2 0.75 10,191 College-level math 8.3 8 -0.3 0.61 3,656 Reading 58.1 58 -0.1 0.81 12,544 Sentence skills 34.9 34.6 -0.3 0.49 10,726 Written exam 3.9 3.9 0.0 0.69 10,979 Total 6,141 6,403 12,544 Notes: Sample is any student who took a placement exam in at least one subject and enrolled at one of the seven study colleges during the study period. Estimates include strata fixed effects (indicators for each college). Observation counts vary for exam scores because students do not necessarily take all exams and gender and HS GPA are not available for all students. 36 Table 3. Changes in Placement for Program-Group Students (1) (2) (3) (4) (5) Took Placement Test Same Placement as Test Score System Placement Changed from Test Score System Higher Placement than Test Score System Lower Placement than Test Score System Math Placement % of sample 81.6% 58.5% 23.1% 15.4% 7.7% N 5,226 3,747 1,479 988 491 English Placement % of sample 100% 45.2% 54.8% 49.1% 5.7% N 6,403 2,891 3,512 3,145 367 Notes: Sample is restricted to treatment group students: students who took a placement exam in at least one subject and enrolled at one of the seven study colleges during the study period and were assigned to the treatment group. Colleges exempted a share of students from placement tests based on certain criteria, shown in column (1). 37 Table 4. Compliance with Algorithm’s Recommendation (1) (2) (3) (4) Overall Sample Fall 2016 Spring 2017 Fall 2017 Treatment 0.757*** 0.813*** 0.699*** 0.729*** (0.005) (0.007) (0.014) (0.008) Control Mean 0.046 0.027 0.065 0.054 Observations 12,544 4,688 1,914 5,942 Notes: Robust standard errors in parentheses. *** p<0.01, ** p<0.05, * p<0.1. Sample is any student who took a placement exam in at least one subject and enrolled at one of the seven study colleges during the study period. Columns (2)-(4) restrict the sample to students tested in the corresponding term. All models include fixed effects for college (strata), controls for demographic indicators (race, gender and age, Pell recipient status), GED/HS indicator, and calculated math and English algorithm scores. Compliance is defined as following the algorithm’s placement recommendation. For students who took both math and English exams, compliance is defined as following the algorithm’s recommendation in at least one subject. Following the test-score placement system’s recommendation in the control group is considered non-compliance regardless of whether it is the same as the algorithmic recommendation. 40 Table 7. Error Rates of the Algorithm v. Test-Score Placement Systems Math Pass Rate (1st-term) English Pass Rate (1st-term) Math Pass Rate (Ever) English Pass Rate (Ever) Treatment Placed in College-Level by Algorithm, Counterfactual was Remedial 0.607 0.624 0.499 0.530 Control Placed in College-level by Test Scores, Counterfactual was Remedial 0.442 0.496 0.401 0.408 Notes: The first row restricts the sample to students assigned to the treatment group who were placed by the algorithm into a college-level course, but their counterfactual test-score placement would have been the remedial course. The second row restricts the sample to students assigned to the control group who were placed by the test-score system (ACCUPLACER) into a college-level course, but their algorithmic placement would have been the in a remedial course. Each column shows pass rates by subject for the first term and ever. Pass rates are (unconditional) college-level credits earned divided by college-level credits attempted in the corresponding period and subject. 41 Table 8. Longer-Run Effects: Fall 2016 Cohort (1) (2) (3) (4) (5) (6) Remedial Credits Attempted College Credits Attempted College Credits Earned Remedial Credits Attempted College Credits Attempted College Credits Earned Treatment -1.181*** 2.503*** 1.618*** -1.224*** 2.692*** 2.041*** (0.129) (0.605) (0.598) (0.164) (0.706) (0.688) Control Mean 3.913 32.82 21.68 4.584 30.32 19.23 Sample All All All Placed in Math and English Placed in Math and English Placed in Math and English Observations 4,688 4,688 4,688 3,277 3,277 3,277 Notes: Robust standard errors shown in parentheses. *** p<0.01, ** p<0.05, * p<0.1. Columns (1)-(3) is the full sample and columns (4)-(6) restrict the sample to students who were placed in both math and English. All models include fixed effects for college (strata), controls for demographic indicators (race, gender and age, Pell recipient status), GED/HS indicator, and calculated math and English algorithm scores. Credits attempted and earned are total college-level credits attempted and earned by students. 42 Table 9. Subgroup Analysis on College-Course Outcomes White Hispanic Black Male Female Pell Non-Pell Placed into College Math 0.106*** 0.034** 0.057*** 0.011 0.139*** 0.068*** 0.065*** (0.013) (0.015) (0.019) (0.012) (0.012) (0.012) (0.011) Observations 3,810 2,116 1,802 4,420 4,486 4,117 5,413 Placed into College English 0.296*** 0.308*** 0.386*** 0.317*** 0.333*** 0.321*** 0.323*** (0.013) (0.019) (0.019) (0.012) (0.012) (0.013) (0.011) Observations 4,085 2,081 2,046 4,894 4,543 4,313 5,735 College Credits Earned 0.000 0.712 1.131* -0.092 1.147** 1.171** 0.037 (0.498) (0.688) (0.622) (0.446) (0.453) (0.459) (0.392) Remedial Credits Attempted -0.692*** -0.792*** -0.734*** -0.555*** -0.834*** -0.887*** -0.500*** (0.066) (0.114) (0.113) (0.064) (0.071) (0.078) (0.053) Observations 5,389 2,485 2,471 5,959 5,942 5,386 7,158 Notes: Robust standard errors shown in parentheses. *** p<0.01, ** p<0.05, * p<0.1. Enrollment and pass-rate outcomes are for “ever enrolled” and “ever passed” the course indicated. Each column restricts the sample to the subgroup in the column header. Each cell is from a separate regression. All models include fixed effects for college (strata), controls for demographic indicators (race, gender and age, Pell recipient status), GED/HS indicator, and calculated math and English algorithm values. Credits attempted and earned are total credits attempted and earned by students. 45 APPENDIX TABLES Table A.1. College Characteristics Institution Cayuga Jefferson Niagara Onondaga Rockland Schenectady Westchester GENERAL INFORMATION Student Population 7,001 5,513 7,712 23,984 10,098 8,458 22,093 Full-time Faculty 69 80 151 194 122 79 215 Part-time Faculty 170 177 0 480 409 0 2 Student/Faculty Ratio 20 18 16 23 23 23 16 % Receiving Financial Aid 92% 91% 92% 92% 56% 92% 70% DEMOGRAPHICS Race/ethnicity: American Indian/Alaska Native 0% 1% 1% 1% 0% 1% 1% Asian 1% 2% 1% 3% 5% 7% 4% Black 5% 7% 11% 12% 18% 14% 21% Hispanic/Latino 3% 11% 3% 5% 20% 6% 32% Native Hawaiian or Other 0% 0% 0% 0% 0% 1% 0% White 85% 73% 80% 49% 39% 67% 33% Multi-Ethnic 2% 3% 2% 3% 2% 2% 2% Race/Ethnicity Unknown 3% 3% 1% 27% 15% 2% 5% Non-Resident Alien 1% 1% 0% 0% 1% 0% 1% Gender: Female 60% 58% 59% 52% 54% 53% 53% Male 40% 42% 41% 48% 46% 47% 47% Age: Under 18 30% 17% 19% 24% 10% 37% 1% 18-24 44% 52% 60% 55% 63% 40% 69% 25-65 26% 31% 21% 21% 26% 23% 30% RETENTION/GRADUATION RATES Retention Full-Time Students 56% 55% 63% 57% 68% 56% 64% Part-Time Students 28% 30% 47% 34% 56% 50% 53% Three-Year Graduation Rate 24% 27% 28% 20% 29% 20% 15% Transfer Out Rate 18% 19% 18% 22% 19% 22% 18% Notes: This table shows summary statistics for all students enrolled at the seven study colleges from historical data. Data are from the U.S. Department of Education, National Center for Education Statistics, IPEDS, Fall 2015, Institutional Characteristics. 46 Table A.2. Math Algorithm Models Model 1 Model 2 Model 3 Model 4 HS GPA1 0.035*** 0.028*** 0.030*** (0.002) (0.003) (0.002) Missing GPA2 2.822*** 2.270*** 2.583*** (0.195) (0.209) (0.210) ACPL Algebra3 0.006*** 0.004*** 0.004*** (0.001) (0.001) (0.001) ACPL Arithmetic Missing2 0.056 0.038 0.065 (0.040) (0.041) (0.042) ACPL Algebra Missing2 0.634*** 0.361** 0.335* (0.141) (0.137) (0.140) ACPL College-level Math Missing2 -0.087 -0.088 -0.084 (0.055) (0.051) (0.051) Years Since HS Graduation 0.020*** (0.004) HS Graduation Year Missing2 -0.056 (0.068) GED2 -0.192** (0.071) Missing Diploma Type2 0.121 (0.100) Constant -2.337*** 0.038 -2.048*** -2.303*** (0.192) (0.122) (0.217) (0.213) N 1,166 1,166 1,166 1,166 R2 0.125 0.105 0.176 0.207 AIC 1,538.4 1,568.6 1,475.5 1,439.5 1 100-point scale 2 Binary indicator 3 Test score range 20-120 Notes: Robust standard errors shown in parentheses. *** p<0.01, ** p<0.05, * p<0.1. This table shows results from regression of the covariates listed on an indicator for getting a C or better in the college-level math course. Models 1 - 3 include different subsets of covariates, with the full model shown in Model 4. 47 Table A.3. English Algorithm Models Model 1 Model 2 Model 3 Model 4 HS GPA1 0.022*** 0.022*** 0.024*** (0.001) (0.001) (0.001) Missing GPA2 1.774*** 1.761*** 1.959*** (0.103) (0.103) (0.114) ACPL Reading3 0.001* 0.001* 0.001 (0.001) (0.001) (0.001) ACPL Sentence Skills3 0.000 0.000 0.000 (0.001) (0.001) (0.001) ACPL Written Exam4 0.000 -0.002 -0.001 (0.002) (0.002) (0.002) ACPL Reading Missing2 0.315*** 0.332*** 0.210** (0.073) (0.074) (0.077) ACPL Sentence Skills Missing2 -0.027 -0.147* -0.154* (0.077) (0.074) (0.074) ACPL Written Exam Missing2 0.021 0.008 0.017 (0.027) (0.026) (0.025) Years Since HS Graduation 0.009*** (0.001) HS Graduation Year Missing2 0.041 (0.087) GED2 -0.190* (0.083) Missing Diploma Type2 0.032 (0.094) High School Rank Percentile 0.000 (0.000) Missing High School Rank2 -0.006 (0.041) Constant -1.147*** 0.478*** -1.218*** -1.301*** (0.101) (0.060) (0.111) (0.118) N 3,786 3,786 3,786 3,786 R2 0.072 0.006 0.078 0.095 AIC 4,893.2 5,161.4 4,879.8 4,823.8 1 100-point scale 2 Binary indicator 3 Test score range 20-120 4 Test score range 1-8 Notes: Robust standard errors shown in parentheses. *** p<0.01, ** p<0.05, * p<0.1. This table shows results from regression of the covariates listed on an indicator for getting a C or better in the college-level English course. Models 1 - 3 include different subsets of covariates, with the full model shown in Model 4. 50 Table A.7. Costs for Implementation and Operation of the Algorithmic Placement System Range Per College Total (six colleges) Lower Per-student Incremental Cost Bound Upper Per-student Incremental Cost Bound Students per semester 5,808 2,750 505 Total Placement Cost: Algorithm $958,810 $268,890 $196,170 Test Score Placement $174,240 $82,590 $15,150 New placement incremental cost: Per semester $784,560 $186,300 $181,020 Per student $140 $70 $360 Notes: 2016 dollars. Present values (discount = 3%). Rounded to $10. Ingredients information on full-time equivalents is from interviews with key personnel at six colleges. Lower and upper bounds represent highest and lowest per-student incremental costs across the six colleges. Cost data not available for one college. Costs amortized over cohorts. Student cohorts rounded to nearest 10. Total placement cost includes all costs to implement and administer the placement test; personnel (i.e., IT, program, senior/faculty, administrative support, and evaluator’s time), fringe benefits, and overheads/facilities. IT personnel salary data from https://www.cs.ny.gov/businesssuite/Compensation/Salary- Schedules/index.cfm?nu=PST&effdt=04/01/2015&archive=1&fullScreen. Program personnel annual salary (step 4, grade 13) from https://www.suny.edu/media/suny/content- assets/documents/hr/UUP_2011-2017_ProfessionalSalarySchedule.pdf. Senior/faculty midpoint MP-IV https://www.suny.edu/hr/compensation/salary/mc-salary-schedule/ https://www.cs.ny.gov/businesssuite/Compensation/Salary- Schedules/index.cfm?nu=CSA&effdt=04/01/2015&archive=1&fullScreen. Evaluator’s time estimated from timesheets. Fringe benefits uprated from ratio of fringe benefits to total salaries (IPEDS data (2013, 846 public community colleges). Overheads/facilities uprated from ratio of all other expenses to total salaries (IPEDS data (2013, 846 public community colleges). Cost to administer placement test from Rodríguez et al. (2014). New placement incremental cost is the difference between the test-score placement system (business-as-usual) and the new, algorithmic placement system’s total placement costs. More than two-thirds of the new placement incremental costs are implementation costs, and approximately 30% are operating costs ($40 per-student), which refer to running of new placement system after initial algorithm has been developed and tested. 51 Table A.8. Cost-Effectiveness Analysis: Social Perspective Per-student Costs Control Treatment Difference Direct cost: Placement $30 $170 $140 Indirect cost: Attempted remedial credits $1,840 $1,270 -$570 Indirect cost: Attempted math and English college credits $3,580 $3,770 $190 Total Cost $5,450 $5,210 -$240 Earned math and English college credits 3.986 4.114 0.128 Cost per earned college credit $1,370 $1,270 -$100 SOURCE: Tables A.5 and A.7 and authors’ calculations. Cost figures rounded to nearest 10.