July 16, 2003

Apples to apples evaluation of charter schools

The Manhattan Institute Center for Civic Innovation has just released the Working Paper Apples to Apples: An Evaluation of Charter Schools Serving General Student Populations, by Jay P. Greene, Greg Forster, and Marcus A. Winters (July, 2003). In the report the authors confuse the ability of schools to improve themselves with their ability to improve their students, as this Web contribution will explain. It is an elementary error that completely invalidates the report.

I remark that in February, 2003, the same authors produced a report Testing High Stakes Tests: Can We Believe the Results of Accountability Tests?. I wrote a Web review of that report in which I explained that the authors confused the predictive power of a high stakes test with its validity as a measure of student learning. That too was an elementary error that completely invalidated the report. (In both cases the report's conclusions are plausible, but that is besides the point.)

The present Apples to Apples report sets out to compare the performance of charter schools with that of public schools serving similar populations. (Given the wide range of educational policies in place in charter schools as well as in public schools I'm not sure that the question is all that interesting, but let's accept the question anyway.) In order to compare similar schools, the report focusses on charter schools that serve a general student population, and the control group of public schools is formed by taking for each charter school the nearest public school that also serves a general population.

The measure of performance is whatever standard statewide test is in place. Now I remind the reader of the concept of value-added assessment. See, for example:

Value-added assessment employs, ideally, performance data on individual pupils over multiple years, and looks at improvements over time. It is a way to factor out the effects of different student backgrounds, because these are, one assumes, reflected in their initial test performance. If one doesn't have data on individual pupils then one can use data on grades within a school. In that case the incremental performance that one cares for is that between a certain grade in one year and the next higher grade the next year, on the assumption that this involves approximately the same student population.

Greene et al. could certainly have used such grade-to-grade value added assessment in their work. However, they did something different. They look at the overall performance of each school in one year and compare it to the overall school performance the next year. The school performance is measured in whatever way the state measures it: typically some average scale score or a percentile rank within the state. They do this for each tested subject separately, but not separately for each grade. They then compare the year-to-year changes in performance of the charter schools to the year-to-year changes in performance of the nearest public schools. They find, finally, a small (in fact, very small) advantage for charter schools on this measure. In the executive summary they express their observations as follows:

Measuring test score improvements in eleven states over a one-year period, this study finds that charter schools serving the general student population outperformed nearby regular public schools on math tests by 0.08 standard deviations, equivalent to a benefit of 3 percentile points for a student starting at the 50th percentile. These charter schools also outperformed nearby regular public schools on reading tests by 0.04 standard deviations, equal to a benefit of 2 percentile points for a student starting at the 50th percentile.

And so, the authors completely confuse a measure of the improvement of schools with a measure of the improvement of student performance. Charter schools could be performing wonderfully or they could be performing dismally relative to public schools in improving student performance, and it would not be seen on the whole school year to year test score improvements that are the basis of this report. It would be seen, of course, in traditional value-added assessment at the pupil or grade level.

Posted by Bas Braams at 04:26 PM | Comments (4)

July 14, 2003

Charter schools and segregation

The Civil Rights Project at Harvard University has produced a new study, Charter Schools And Race: A Lost Opportunity for Integrated Education (PDF), by Erica Frankenberg and Chungmei Lee. Please note as well the Executive Summary and the press release. According to the executive summary:

This report details a disappointing set of findings regarding its central question: charter schools are largely more segregated than public schools. Segregation is worse for African American than for Latino students, but is very high for both. In some states, white student isolation in charter schools is as high as that of African Americans. The problems reported here may not be due either to the intent or the desires and values of charter school leaders. They may reflect flaws in state policies, in enforcement, or in methods of approving schools for charters.

It is a strange study, meant primarily, it seems to me, to be quoted in the press. I cannot imagine that this work has anything to offer to a serious student of schools policy.

The situation is that charter schools are found disproportionately in inner cities. It may be of interest to ask why this is so, but the report doesn't address that issue. Taking it for a fact, then it is only natural that charter schools would disproportionately have a high minority population. The study authors might have avoided the loaded and pejorative language of segregation to describe the situation that charter schools are, in their demographics, not much different than public schools in the same geographic environment.

The authors could, of course, have undertaken a sincere study to investigate if charter schools have a special appeal to Black and Hispanic students, beyond what one would expect on the basis of the location of the schools. I don't know if it is a particularly interesting question in this generality - given the diversity of charter schools, as of public schools, it may be more interesting to focus on specific schools or districts rather than on charter schools generally - but in any case, the authors didn't think to ask.

The report was featured in the Boston Globe on Sunday. (That article quotes Frankenberg as saying that charter schools remain more integrated than public schools, but the press contact for the Civil Rights Project describes that as a mis-quote. Should have been: more segregated.) Joanne Jacobs covers the study with reference to a book that she is writing about one overwhelmingly Hispanic charter school. The Center for Education Reform has also paid attention and has put out a press release criticizing the study. Thanks to Education News for the first pointers to this study and the press coverage.

Posted by Bas Braams at 06:35 PM | Comments (0)