Whether Standardized Tests Add Value Depends on What Question You're Asking

And why Lowell is like a scary rollercoaster

The Superintendent’s proposal for “re-envisioning high school admissions” says that admission to Lowell should be based on a minimum GPA and should not include admissions tests because “GPA is the best predictor of high school success”.

The source for this assertion is a study done by the University of Chicago on Chicago Public Schools. The Superintendent’s presentation links to this summary of the study but the full report can be found here. We’ll see that the assertion that GPA is the best predictor is true, but elides some data that is germane to the Lowell discussion.

Middle School Indicators

The particular problem that the Chicago report authors were addressing is that, in Chicago, students’ grades and attendance decline sharply when they move from eighth to ninth grade. On average, GPAs go down by half a point when students move from eighth to ninth grade. This happens across the board: students in the top quartile on standardized tests see their GPAs fall even more, by 0.6. The percentage with GPAs below 2.0 rises from 26% to 46%. In eighth grade, 60% of students had attendance rates of 96% or higher. In ninth grade, only 37% of students attended at least 96% of the time.

Among students who earned a mix of As and Bs in eighth grade (with a GPA between 3.0 and 3.5), less than a third (29 percent) finish eleventh grade with a GPA that signals they are likely to succeed in college (at least a 3.0). Another 24 percent finish with grades so low they are not even qualified for college (less than a 2.0).

The goal of the analysis was to identify what factors that could be observed in middle school would affect a child’s chance of success in high school so that interventions could be designed to help students who were at risk of not succeeding. They measured “success” in two ways:

• on-track to graduate at the end of ninth grade (defined as five full-year credits and no more than one semester F in a core course)

• earning As or Bs in 9th grade core classes (equivalent to a 3.0 GPA or higher).

They did look at the same measures at the end of 11th grade but the report focuses on 9th grade. The main findings of the report were:

• “Once we know the grades and attendance of students in middle grade, their test scores provide almost no additional information about whether they will be on track.”

• “Eighth-grade core GPA is the strongest predictor of whether students earn As or Bs in ninth grade.”

That’s the conclusion that found its way into the superintendent’s presentation.

When Standardized Scores Do Help

Let’s dig a little deeper. Go back to the two definitions of “success” that they were using. Note that they are binary outcomes. You’re either on-track to graduate or you’re not. You either have a 3.0 GPA or you don’t. The authors were not asking, for example, which factors will give the best estimate of high-school GPA. If you have a 4.0 GPA in middle school, you are very likely to have at least a 3.0 GPA in high school. It was not relevant to the paper whether it was possible to predict that your high school GPA would be 3.5 or 4.5. All that mattered was that your chance of exceeding 3.0 was super high.

At the time of the study, Chicago students took the ISAT (Illinois Standards Achievement Test - like California’s STAR tests, it was retired at the end of the 2013-14 school year). The report found that:

Attendance or ISAT scores improve the prediction of earning As or Bs beyond core GPA, but only among students with strong middle school grades.

In the chart below, we can see that for students with middle-school GPAs of 3.0-3.3, a child who scored “Low Meets” on the ISAT has a better chance of earning As and Bs in 9th grade than a child who scored “Exceeds” on the ISAT. But if we move up to the students with the best middle-school GPAs (3.7-4.0), we see that the ISAT score really adds value. A student in the range who scores “Exceed” on the ISAT has an 84% of earning As and Bs in 9th grade compared to 62% for a student who scores “Low Meets”.

Source: https://consortium.uchicago.edu/sites/default/files/2019-11/5%20Key%20Findings-MG-Final.pdf

Why is having a higher ISAT score a strong signal that you’ll have a 9th Grade GPA over 3.0 if your middle school GPA is in the range 3.7 - 4.0 but of no use whatsoever in predicting your 9th grade GPA if your middle school GPA is in the range 3.0-3.3 (or, as the main report demonstrates, even lower than 3.0)?

GPA and ISAT score range are highly correlated. 53% of those who scored Exceeds had GPAs over 3.7, compared to only a quarter of those who scored High Meets and only 4% of those who scored Low Meets. Among those who score Exceeds on the ISAT, the number that have GPAs in the 3.7-4.0 range is five times as high as the number with GPAs in the 3.0-3.3 range. Among those who score Low Meets on the ISAT, the proportions are nearly reversed: the number that have GPAs in the 3.0-3.3 range is 3 times as high as the number with GPAs in the 3.7-4.0 range.

If you score Exceeds on the ISAT but only have a GPA in the 3.0-3.3 range (or lower), there may be some issue limiting your achievement. Maybe you have some family issues that prevent you from attending school; maybe you have some health or behavioral issues; maybe you’re so smart that school bores you and you can’t be bothered trying; maybe there is nothing wrong at all but you had a great day when you sat the ISAT with the result that your ISAT score is far higher than your true ability. Whatever the reason, having that great ISAT score didn’t help you get a high middle school GPA and it’s not going to give you a better chance of having a 3.0 GPA in high school than a kid with a similar middle-school GPA who scored much lower on the ISAT.

On the other hand, if you have a middle-school GPA over 3.7, you’ve got your shit together and are able to realize your potential. If you also score Exceeds on the ISAT, you have a much better chance of maintaining at least a 3.0 GPA in high school than a kid with a similar middle-school GPA but a lower ISAT score, who is perhaps over-achieving in middle school.

If the study authors had asked, not which middle school indicators predicted whether a child would have a 3.0 GPA in high school, but which middle school indicators predicted high-school GPA or whether a child would have at least a 4.0 GPA in high school, they would have concluded that the standardized score added significant value to the middle school GPA.

The answer you get depends on the question you ask.

School Effects

The authors considered a huge number of indicators that might have an effect on high school success. In addition to middle school GPA (in various grade combinations) and standardized test scores and attendance, they also evaluated behavioral indicators like suspensions, misconducts, grit, study habits; personal characteristics like race, gender, age relative to grade, neighborhood poverty rate, neighborhood social status, and special education status; and the middle and high schools attended. They considered these indicators individually and in combination. Most of them didn’t add anything that wasn’t already captured in middle school GPA and attendance. The exception was the identity of the middle and high schools.

Students with the same middle grade performance have different probabilities of doing well (whether measured by being on-track or earning As and Bs) in high school, depending on which schools they attend. The effects are opposing: being in a high-achieving middle school increases your chances of doing well in high school while being in a high-achieving high school lowers your chance of doing well in high school.

Among students with the same eighth-grade GPAs, attendance, and test scores, those who went to a high-achieving middle school (i.e. with higher average ISAT scores) with fewer students living in poverty are more likely to be on-track and earn high grades in ninth grade than students who went to a low-achieving middle grade school, with many students living in poverty. Going to a higher-performing, economically advantaged middle school seems to confer some benefits that are not picked up through grades, attendance, and test scores. It is possible that grading standards may be higher at high-achieving middle schools than low-achieving middle schools, so that an A from a high-achieving middle school might mean stronger course performance than an A from a low-achieving middle school. [page 40]

This is certainly what the TNTP study referenced in the Math Vision presentation would suggest. That study concluded that students in lower achieving middle schools don’t get exposed to the enough grade-level material.

Where students attend high school makes an even bigger difference for their likelihood of being on-track or earning high grades than where they attended middle school…Among students with a moderate probability of earning high grades, the probability of doing so ranges from 40 to 77 percent.

When we look at the characteristics of the schools where students earn higher or lower grades than their qualifications would predict, we see the opposite pattern as with the middle grade schools. Students are more likely to be on-track or earn high grades if they attend high schools that serve more low-achieving, disadvantaged peers, compared to students with similar eighth-grade qualifications who attend higher-achieving high schools. This might result from lower standards at high schools serving more disadvantaged students with low incoming test scores, or from grading practices that are based on comparisons among students (e.g., grading on a curve). Students might also feel less capable and withdraw effort if they attend a school with high-achieving peers where they feel everyone else is smarter than they are, or exhibit more self-efficacy and engagement if they are a strong student relative to their school peers. [page 42]

What if a student attends both a high-achieving middle school and a high-achieving high school? In middle school, they “do better than their grades, attendance, and test scores would predict” but in high school they have “a lower likelihood of passing their classes and earning high grades. These two school effects cancel each other out.”

The largest school effects [were] observed among students who attend high schools that are atypical for their middle school. Students who moved from a strong middle school to a weak high school [were] much more likely to pass in ninth grade than their middle grade performance would indicate, while students who moved from a weak middle school to a strong high school [were] much less likely to pass than their middle grade performance would suggest. (page 42)

A more succinct way of putting it is found on page 12 of the report:

Lessons from Chicago’s Selective High Schools

Chicago has 11 selective high schools (SEHSs) with 3,600 total seats in 9th grade. Admission is based on a combination of seventh-grade GPA and a one-hour admissions exam. Seventh-grade standardized test scores used to be factored in as well but were recently removed. Students are organized into groups by census tract and ranked by score.

Census tracts are divided into four tiers based on socio-economic criteria so that each quarter contains approximately one-quarter of Chicago’s school-aged children. Tier 1 represents the lowest-SES quartile; tier 4, the highest-SES quartile. 30% of seats at each SEHS are allocated to the top-scoring applicants, regardless of their SES tier. The remaining seats are divided equally among the four SES tiers.

The selective schools are not all alike. Of the 11 SEHS, one is 90% Hispanic, four are 69%-96% Black, one is 88% Hispanic or Black, and the remaining five have no group with more than 36%. These five are all located near downtown or on the north side of Chicago and are the only ones with significant numbers of White and Asian students. For reference, the district as a whole is 46% Hispanic and 35% Black.

At the most selective of the SEHS, which is about half the size of Lowell in a district that is more than six times bigger, a tier 4 student needs a perfect score to be admitted (i.e. 4.0 GPA and 99th percentile in both Reading and Math), a tier 3 student needs a 4.0 GPA and to be in the 96th percentile, a tier 2 student needs a 4.0 GPA and to be in the 92nd percentile, and a tier 1 student needs a 4.0 GPA and to be in the 80th percentile (or a GPA as low as 3.50 if the admissions test score is in the 97th percentile). By contrast, in the five SEHS that are majority Black or majority Hispanic, tier 4 students need a lower score for admission than tier 1 students, indicating that geographical or racial considerations are a bigger determinant of the attractiveness of a school than whether or not it has selective admissions.

One way to see whether SEHS benefit students is to compare students very close to, but either side of, the admissions cutoff. These students are very similar academically so any difference in their outcomes can be attributed to attendance at the school. What they found in Chicago was that, compared to similar students not admitted to a selective school, SEHS students have similar test scores, high school graduation rates and college enrollment rates but report higher levels of safety at their high school and stronger relationships with peers.

However, tier 1 (low SES) SEHS students have significantly lower GPAs and are less likely to attend a selective college than similar tier 1 students who do not enroll in SEHSs. (emphasis in original).

Implications for San Francisco

We don’t know to what extent the lessons from Chicago apply to San Francisco. It would have been nice if the High School Task Force had found the time to request a data analysis from the district at some point during its work, but it didn’t. If we assume that what’s true in Chicago is also true in San Francisco, it does suggest that much of the debate about Lowell’s future is missing the point.

One group of campaigners measure success by numbers in the door: any policy that leads to more Black and Latino kids at Lowell is a good policy. But if “even students with above-average achievement are at elevated risk of failure if their classroom peers have much higher skill levels”, the prospects of many of the Band Two and Band Three admits may have been hurt rather than helped by going to Lowell and, in particular, the district’s policy of trying to create a pathway from Willie Brown Middle School to Lowell was tragically misguided.

On the other hand, those campaigning to keep selective admissions are often living in the past. Admission to Lowell may have once provided a glide path to top universities for kids at the top of the class but, with the way admissions are done today, that is no longer the case. The UCs evaluates applications “in the local context” and, when the local context is hundreds of people with GPAs over 4.00, it ends up being much harder to get in to the top UCs from Lowell than from any other SFUSD school.

So, if the kids at the bottom of the achievement spectrum get poorer results at Lowell than they would elsewhere, and the kids higher up the achievement spectrum don’t get the boost they once did, the best argument for keeping Lowell a selective admissions school is that it’s voluntary.

The analogy I like is that a selective high-school like Lowell is like the biggest, scariest rollercoaster in the amusement park. Some people are going to love it. Some people are going to throw up after riding it. Others are going to want nothing to do it. And that’s okay because there are plenty of other exciting rides in the amusement park and no-one is forced to ride the scary rollercoaster if they don’t want to. But the people who don’t want to ride the rollercoaster shouldn’t be able to prevent the people who do want to ride it from doing so.

The district’s proposal of a lottery for kids with a minimum GPA of 3.0 has a split-the-baby feel to it. The district never published data on the Band Two and Band Three Admits so it’s impossible to know if 3.0 is above or below the minimum admitted under the old scheme. Regardless, it’ll still be true that kids with a GPA at the bottom of the range would probably be better off elsewhere. Meanwhile, many of the students who would previously have been admitted in Band One will not get admitted under the lottery-with-a-minimum-GPA idea. Those who would have taken advantage of AP Physics C, or other only-at-Lowell classes, will get an inferior education but the rest will do fine elsewhere. And it’s possible to identify those kids who will lose out. I’d be willing to bet that the population of kids taking AP Physics C is drawn overwhelmingly from those who scored in the 98th or 99th percentile on their 7th grade SBAC test, which brings us nicely back to where we started.