EDITOR'S NOTE: This post was co-authored by Christopher Thornberg.
One of the closest races in November’s mid-term election in California was for the normally obscure post of State Superintendent of Public Instruction. The race received relatively intense attention because it became the latest proxy battle in the larger war over ongoing educational reforms across the nation. One side says much more reform is needed, while the other claims the reforms being proposed are the wrong medicine for the malady and argue against changing things. Who’s right? One way to answer that is by metrics – creating ways of measuring school performance that allow us to evaluate whether reforms are actually having an impact.
If only metrics were that easy. How do you measure the success of educational efforts? Standardized tests have been one way, but are fairly controversial for a number of reasons. One of the simplest measures would simply be to ask how many students who enter public high school end up graduating with a diploma. This is seemingly simple and effective - and public funding and educational policies are widely and strongly based on high school graduation rates. But even here we start running into measurement problems.
What to Measure and How
Up until recently, the California Department of Education (CDE) reported high school graduation rates in two ways—which, as it turns out, gave completely different estimates. The first, the “traditional” graduation rate, was based on the percentage of 12th graders who were enrolled at the beginning of a school year and graduated from high school that same school year. But this calculation missed students who may have dropped out of the system before starting their last year of school. As such, the measure dramatically overestimated the success rate of individual schools.
The second measure the CDE used was a four-year graduation rate, calculated by dividing the number of high school graduates by the number of enrollees from the previous four years. This methodology provided a more accurate assessment than the traditional graduation rate, but still had problems identifying changes in enrollment due to student transfers. If more students moved into the district than moved out, the graduation rate would be biased up. Alternatively if more students transferred out, the rate would be biased down. This measure also had a problem with students who graduated earlier or later than the typical four years. Closer—but still no cigar.
Several years ago, the Federal government, in an effort to consistently measure graduation rates across all 50 states, mandated that states implement an on-time graduation rate so that data would be more consistent in cross-state comparisons.
Based on the Federal mandate, beginning in the 2009-10 academic year, the CDE began reporting what is called the four-year adjusted cohort graduation rate. This rate was not calculated based on groups of students, but by actually tracking individual students. Each student is now given a statewide student identifier (SSID) so California can track exactly where students are attending school throughout their K-12 public education. Bingo!
The disparity in the results between the more accurate current measure and the older methods is illustrated in the graphics below. The adjusted cohort graduation rate is significantly lower than the traditional graduation rate. And although the four-year graduation rate is relatively on par with the adjusted cohort graduation rate, the four-year graduation rate was calculated using a flawed methodology (as discussed) and discounts the actual educational progress for a given state. Based on the adjusted cohort graduation rate, when compared to different states, California’s ranking actually rises to the middle of the pack and the state displays a relative upward trend in its graduation rate.
Click to view
Statistics about high school graduation rates form the basis of decisions about what to fund, how much to fund, and what policies to put into place. Without accurate numbers, the decisions cannot be the most effective. The Federal mandate requiring states to measure high school graduation rates in a consistent manner is a strong step towards accurately evaluating the nation’s educational progress. However it is not perfect. There is no one simple measure of school performance, as all measures have issues. But if we don’t measure we can never find out if reform efforts are working. What this means is that one of the most important ‘reforms’ is remaining committed to finding better ways of measuring performance and understanding if our reforms are working at all.