Resource Alignment Haste

The DAC has rushed to assign weights to metrics that are not clearly defined.

The District’s Resource Alignment Initiative, and the District Advisory Committee (DAC) that is supposed to provide public feedback on the initiative, seem to have settled on ten criteria that are going to be used to decide which schools to close and have even figured out weights that will enable them to reduce all these criteria to a single composite score. Unfortunately, the metrics that they have chosen to capture these criteria are not clearly defined in their public communications and the DAC members are still discussing things at the vibes level, which means no-one is digging into the details.

I’m going to review each of the criteria and identify real and potential issues I see with the definition or use of the chosen metric, in the hope that the necessary clarity can be added before anyone starts crunching numbers in earnest. I’m going to follow the structure and sequence that the DAC uses.

Equity Category

Criterion 1: School Access.

Metric: the average distance between the three closest schools with the same grade span.

1. The not-yet-open elementary school in Mission Bay should be included in this distance calculation. Is it?

2. Are charter schools included in the distance calculation? In other words, is a charter school seen as a legitimate alternative to a SFUSD school? The argument to include them is that they are public schools after all. The argument to exclude them is that they’re not SFUSD operated.

3. Are K-8 schools being treated as elementary schools or as a separate category? I’m guessing they’re being treated as elementary schools because I have heard that there is a move towards reducing the number of K-8 schools. Monroe ES and SF Community (a K-8 school) are right next door to each other so the answer will greatly affect both schools’ scores on this metric?

4. Are the early childhood centers within the remit of the Resource Alignment Initiative? If so, are they a separate grade span? Do they count as elementary schools for the purpose of the elementary school distance calculation?

5. Does the Lee Newcomer School (now colocated with Lau) count as a separate school? I assume no.

6. Are the continuation schools and Independence High considered as part of the High Schools for the distance calculation? Rising ninth graders can’t choose to attend these schools but some will transfer to them later.

7. It might be necessary to recompute this measure after each closure decision to reflect the schools that remain open. Otherwise, you might end up closing a bunch of schools that are close to each other.

8. The district plans to change to a zone-based system for elementary school assignment. Under such a system, having a school nearby is of no use if it’s not in your zone. Ideally, the list of school closures would be tied to the release of the elementary school zones

Criterion 2: Program Access

Metric: the percentage of students participating in language programs, special education programs, or career technical education and pathway programs.

1. For computing the percentage, if a student is in both special education and a language program is the student counted once or twice?

2. I must confess that the logic of using program access escapes me. I would have no quibble with a stipulation that no particular program or student group should be disproportionately affected by any closures but I don’t understand why students in a language program deserve preferential treatment over students in a general education program. Suppose there are two otherwise identical schools, one of which offers general education and the other offers Cantonese immersion. Why does equity demand that the school offering general education be the one to close?

Criterion 3: Historical Inequities

Metric: The average amount of neighborhood opportunity, as measured by the Opportunity Insight Lab's "upward mobility index," experienced by students in the school.

Do a google search. Opportunity Insight Lab does not have an “upward mobility index”. If you read the paper or watch the video, their preferred measure of upward mobility is the “Household Income at Age 35 for children whose parents were Low Income”. The idea is to identify the areas where low income kids thrive in later life. Let’s assume that this is what is meant by “upward mobility index”.

Figure 1 is a screenshot from their Opportunity Atlas. It shows a map of San Francisco divided by census tract. The color coding represents the household income at age 35 of children who grew up in that tract starting in 1979-1983 and whose parents were low income. The basic picture of high incomes in the west and low incomes in the south-east seems plausible enough but I want to point out a few anomalies that make me hesitant about the quality of the data at this fine-grained a level.

• The values for Sea Cliff ($34k in tract 06075042800) and Presidio Heights ($33k in tract 06075013300) are much lower than the Tenderloin (multiple tracts, all between $37k and $43k). They are also far below part of Portola ($52k in tract 06075025702).

• There are two tracts in Noe Valley between 21st St and 24th St. Low income children who grew up in the blocks west of Castro (i.e. the area around Alvarado ES) had average incomes of $26k but those who grew up east of Castro had average incomes of $50k. I live near here: it’s not plausible that these adjacent tracts differ by so much.

• In Pacific Heights, there’s a tract bounded by California to the South, Jackson to the North, Steiner to the West and Laguna to the east where the mobility index has a value of $22k. The seven tracts that cover the Bayview/Hunters Point area have values of $21k, $30k, $31k, $34k, $39k, $41k, and $45k. Are we really to believe that a low income child who grows up in this part of Pacific Heights is worse off than most low income children who grow up in Bayview/Hunters Point?

These anomalies are due to the statistical approach used to produce the estimates. Presidio Heights and Sea Cliff just don’t have many low income families (and didn’t back in the 1980s either). As Opportunity Insight’s FAQ puts it:

…there may be outliers or imprecision in the estimates. Generally, estimates are more reliable at larger geographies because they are based on more observations in the data. Small geographies, such as Census tracts or even counties, may sometimes lack sufficient data, especially if you are restricting to specific income levels...

Also,

We urge caution when interpreting the results of Census tracts that have colleges in them. Many “low income children” that can be observed in these areas are actually low-income college students or the children of parents classified as low income because they were living on graduate student stipends.

Another unavoidable problem is that some parts of the city have changed dramatically over the last forty years. The Presidio is no longer an army base so the children who live there are no longer the children of soldiers. The character of SOMA and Mission Bay has changed enormously. Data based on how things were may not be valid for how things are today.

Let’s get down to specifics:

1. Let’s assume that the mobility index numbers for each tract are accurate. How do you calculate “the average amount of neighborhood opportunity … experienced by students in the school”. Do you calculate a mobility index for each student based on the student’s home address and then calculate a value for the school by averaging the mobility index of each student? Or do you simply look at the location of the school and assume that the location of the school represents the mobility of the students? The latter is analytically much simpler but will produce quite different results for some schools such as Francisco MS (majority Latino student population but in a census tract with a high value: $61k) or SOTA (fewest disadvantaged kids of any high school but lowest census tract value)

2. Assuming the mobility index is based on the location of the school, I looked up what the values for various schools would be. Here are some of the lower scoring schools: Malcolm X and Carver ($21k), Alvarado ($26k), Grattan ($27k), Starr King ($28k), McKinley ($30k), New Traditions ($31k), Bryant ($31k), Milk ($32k), Chinese Immersion at DeAvila ($33k), Cobb ($34k), Cesar Chavez ($35k), BVHM ($36k), Drew ($39k), Tenderloin ($43k). Do you think the person who voted for this as a vehicle for advancing equity realized that it would give the four schools with the largest White populations in the district (viz. New Traditions, Grattan, Milk, McKinley) scores that were comparable to, and often better than, the three most Latino schools (viz. Cesar Chavez, BVHM, Bryant) and one of the three majority Black schools (Drew)? Did they realize that the high school with the fewest disadvantaged kids (i.e. SOTA) would have a value of $31k, lower than Mission ($36k), Jordan ($39k), Burton ($40k), Galileo ($45k), Marshall ($45k), Lowell ($46k), Balboa ($47k), Lincoln ($58k), Washington ($67k)?

3. How do you deal with schools that have multiple locations such as Claire Lilienthal and Rooftop? I assume the locations are weighted but is the weighting evenly or by number of grades supported or by number of students?

4. For most of the other metrics, schools with high scores are less likely to be closed. As examples, a school is less likely to be closed if it’s located further from other schools or if more of its students are in language or special education programs. As currently written, the schools with the highest scores, and thus less likely to be closed, will be the ones in the Sunset. That is surely not what was intended when this metric was proposed as an “equity” measure.

Excellence Category

Criterion 4: School Culture and Climate

Metric: The percentage of families, staff, and students responding favorably to survey questions about a sense of belonging, safety, or academic support for learning.

1. The results of these surveys used to be public. Some school sites have links to old reports such as this 2021-22 report from Alamo ES. Where are the reports for 2022-23 or 2023-24?

2. Will the metric use one year’s results or average across multiple years?

3. If you look at that old Alamo report, you’ll see that there is no single overall number. There are separate scores for climate of support for academic learning; safety; and sense of belonging from each of families, students, and staff. That’s nine separate numbers. Will the single overall number be based on a simple average of the family, student, and staff numbers or will it be weighted by the number of responses? The choice could have a big impact because, at Alamo, 209 families, 136 students, and 35 staff responded and the staff rankings, in particular, differed markedly from the district average.

Criterion 5: Socio-Emotional Development

Metric: The percentage of students responding favorably to survey questions related to social awareness, self-management, growth mindset, or self-efficacy.

The survey questions for elementary schools can be found here and the Alamo report I linked to above shows the results. Notice that students are generally asked for responses on a five point scale (i.e. not at all; a little; somewhat; mostly; completely) but the report translates this into a binary favorable or not favorable. It’s not clear what constitutes a favorable response. Is “somewhat true” favorable or unfavorable?¹ I’d also be surprised if 4th graders who are below standard in reading could read and understand prompts like “I can increase my intelligence by challenging myself” or “how often did you compliment others’ accomplishments?

This whole growth mindset stuff is quite controversial and I’m among the skeptics. In the community survey, I gave it the lowest possible number of coins. We’re going to decide to close schools based on whether children think they are good at something rather than on whether they actually are good at it? I realize that everything good about a school can’t be captured in something as narrow as SBAC scores but, instead of these surveys, I’d have preferred a Student Retention metric, such as the percentage of children who complete a grade who return for the following year. That would neatly capture both the academic and non-academic aspects of whether a child is thriving at a school. Too late for that now.

Criterion 6: Academic Performance

Metric: State assessments of English Language Arts and Math performance and growth

1. How are they going to produce a single number that measures performance and growth for ELA and Math? You could measure performance by taking the percentage who met or exceeded the standards in ELA or Math and averaging the two. But how do you then fold growth into that? The School Dashboard uses a color-coding mechanism to combine performance and growth into a color on a five-point-scale (red, orange, yellow, green, blue) but that’s done separately for ELA and Math. How is one single number going to be calculated?

2. You could just use a measure of performance, such as the percentage who meet or exceed standards. My standard objection to that is that it’s more a reflection of the parents than the schools. A school filled with Asian children of college graduates will have better scores than a school filled with newcomer Latino students who don’t speak English, no matter how wonderful the teachers in the latter school are. It seems odd to ding a school for things that are outside its control.

3. Alternatively, you could measure growth using the CDE’s growth index which measures how much students’ SBAC scores have increased at each school. I would be a huge advocate of this but (a) it won’t be available until October which is too late for this exercise; (b) it won’t cover high schools anyway.

According to the community survey, Academic Performance is the most important measure of Excellence, and it’s the only one that’s based on actual learning which is the whole purpose of education. It would be good if it were defined clearly.

Effective Use of Resources

Criterion 7: Family Choice and Demand for the School

Metric: The percentage of applicants ranking the school as one of their top three choices in their school application

1. For elementary schools, only kindergarten applications should be counted. Transitional kindergarten applications should not be counted because only some schools offer TK programs. Counting TK applications would overestimate the popularity of the schools with TK programs.

2. Popular language programs can mask the true popularity of a school. Consider Starr King, on the south side of Potrero Hill, which is one of the two schools in the district that offer Mandarin Immersion programs (the other is Ortega, on the south side of Ingleside). In March 2024, 99 families listed Starr King’s Mandarin Immersion program as one of their top three choices, compared to only 22 who listed the GE program as one of their top three. Those 99 are not picking Starr King per se; they are picking Mandarin Immersion and it just happens to be at Starr King. Very few, if any, are also going to have listed Starr King’s GE program as one of their top three. They should not be counted in any evaluation of whether Starr King should stay open.

3. Mandarin Immersion does not belong at Starr King. A dual-language immersion program requires a mix of native and non-native speakers of the language. Where are the Mandarin speakers in SFUSD? Other than Starr King and Ortega, here are the elementary schools with at least 15 Mandarin speakers²: Lau (Chinatown), Stevenson (Sunset), Key (Sunset), Ulloa (Sunset), Jefferson (Sunset), West Portal (near the Sunset), Chinese Immersion School at De Avila (Buena Vista), Sutro (Richmond), Sloat (Balboa Terrace, near the Sunset), Taylor (Portola), Lafayette (Richmond), Alice Fong Yu (Sunset), Argonne (Richmond), Clarendon (Twin Peaks). Twelve of those fourteen are in or near the Sunset or Richmond. It is dumb to locate a Mandarin Immersion program on the south side of Potrero (Starr King), so far from the students that you need in order to make the program popular and successful. In contrast, many Cantonese programs are located in or near the Sunset and many Spanish programs are located near the Mission or Portola.

4. The Vietnamese World Language program at Tenderloin ES is also misplaced³. Little Saigon in the Tenderloin may have once been the center of the Vietnamese population but it isn’t any longer. The Korean program at Lilienthal attracts 96 of the 162 Korean speakers in elementary and middle schools. The Japanese programs at Clarendon and Rosa Parks attract 80 of the 147 Japanese speakers in elementary schools. The Filipino programs at Carmichael and Longfellow attract 109⁴ of the 316 Filipino language speakers in elementary and middle schools. The Arabic program at Redding attracts 59 of the 249 Arabic speakers in elementary schools. But the Vietnamese program at Tenderloin attracts only 19 of the 302 Vietnamese speakers in elementary schools. There are more Arabic than Vietnamese speakers at Tenderloin ES. Vietnamese speakers are not concentrated into any one area but a better location might be somewhere in Portola. Taylor, Hillcrest, and Visitacion Valley have 70 Vietnamese speakers between them.

Criterion 8: Teacher Turnover

Metric: The percentage of teachers who leave a school

1. A key question here is over what time period. One year would seem to be too short because there’s going to be a lot of fluctuation from year to year and, at some of the smaller elementary schools, each individual teacher who leaves has a big impact on its percentage.

2. Also, who is included in the denominator? Do you include teachers who did not have permanent jobs but were substitutes? Do you include the principal and vice-principal(s)? Do you include all certificated staff or just teachers?

3. Are all departures included in the numerator? A teacher retirement is not a bad reflection on the school nor is a teacher leaving to become a principal elsewhere. Ideally, you’d only count teachers switching to become teachers in other schools and those who leave the profession altogether.

4. Low teacher turnover is good. This metric needs to have its sign changed before going into the composite score. Otherwise a school will be penalized for having a low turnover rate.

Criterion 9: Student Enrollment

Metric: A school's 2023-2024 school year enrollment compared to its ideal enrollment

1. Both enrollment and capacity should be central to the discussion of which schools to close but this metric, the ratio of the two, doesn’t really capture both. Closing or merging schools is disruptive: ideally, you would disrupt as few students as possible. The lower the enrollment in a school, the riper it is for closing. Similarly, there is a fixed cost to operating a school. A school of 400 is more efficient than one of 250. The lower the capacity, or ideal enrollment, of a school, the riper it is for closing. But the chosen metric, the ratio of the two, does not distinguish between a school that fills 150 out of 250 seats and another that fills 240 out of 400.

2. The assumption behind this metric is that the ratio of a school’s enrollment to its capacity is an accurate reflection of its health. And that is true provided the capacity is not artificially limited. In response to the declining enrollment post-pandemic, SFUSD had to reduce the number of kindergarten classrooms. It couldn’t touch the schools that already had very low enrollment because reducing classrooms in those would have been tantamount to closing them and that was not politically viable. The cuts were thus concentrated in larger schools that could absorb the loss of a classroom and still be somewhat viable. Those schools that lost classrooms now have enrollment-to-capacity ratios that have effectively been suppressed by the district’s actions. Meanwhile, the schools that were already on life support were not touched.

3. On the flip side, transitional kindergarten is being expanded to all four-year-olds and the district has to find space for all those classrooms. Sometime in the future, every school might offer TK and the whole elementary school enrollment process might take place at the TK stage rather than the kindergarten stage. For now, only a fraction of eligible kids attend TK and only a subset of schools have TK classrooms. The district has naturally placed the TK classrooms in schools that had space available precisely because they were less popular. Those schools have therefore received an artificial boost to their enrollment-to-capacity ratios. For this reason, I would argue that TK enrollment should be excluded from the enrollment to capacity ratios.

4. A school’s ideal enrollment is a function of the number of different programs it offers. A school that offers one program, whether that’s general education (e.g. Sunnyside) or Spanish immersion (e.g. Huerta) or Cantonese immersion (e.g. CIS@DeAvila) will have a higher target enrollment than a school that offers two or three (e.g. Taylor, which offers GE, Spanish bilingual, and Cantonese bilingual) programs. This was discussed at the 4/22/24 DAC Meeting.

Criterion 10: Building Condition

Metric: The school building’s facility condition index (FCI) score.

1. The FCI is an estimate of the amount of work that needs to be done on a school site as a percentage of the replacement value of all the buildings and equipment on site. A low FCI score is good: it means that little money needs to be spent on the school. The composite score formula, as currently written, will make schools that need no work more likely to close. We’ve seen two other metrics where low scores are good. For the other seven, a high score is good. For the metrics where low scores are good, the definition must be changed so that high scores are good or the formula must be changed to reverse the signs of the weights.

2. Some schools (viz. Rooftop, Lilienthal, Carmichael) are split between two sites, each of which has its own FCI. How will these be combined? This is particularly important for Rooftop. Rooftop’s Mayeda campus requires by far the most investment per student of any site in the district.

Conclusion

It seems hasty to have assigned weights to criteria which are so poorly defined. It would be a good use of the DAC’s time to drill into some detail on exactly how these metrics are being calculated to ensure that the resulting numbers can be interpreted correctly.

1. More generally, I don’t understand why you would give students five options when you’re only interested in two. Why not just give each response a score on a 1-5 scale and then average?

2. Including English Learners and those who are already Fluent English Proficient

3. Unlike the immersion and bilingual programs for other languages, the Vietnamese world language program is not one that families have to select in the enrollment application so this whole argument should probably have been kept for a separate post. But I was on a roll and couldn’t stop myself.

4. I made the denominator the number of Filipino speakers in all elementary and middle schools because Carmichael is a K-8 school. But Longfellow is only a K-5 school so this makes the Filipino program seem slightly less popular than it is.