After all their hard work Education Sector's concludes, "Our research shows that Newsweek's methodology is far too focused on one discrete indicator of school quality." Let's review: Challenge Index Score = (# of AP/IB tests given)/(# of graduating seniors). Too focused on one indicator, huh? Gee guys, really?! Don't know if I would have guessed that on my own, so I'm glad you were able to do the heavy lifting on the research side. In fairness, the report goes on to argue that not only is the index too focused on a single indicator, but the indicator they have chosen, is a poor predictor of how the school performs in other important areas: achievement gaps and graduation rate.

I not sure it is quite as "no-duh," self-evident as Rotherham and Mead (the authors of the report) say it is, but it certainly should give one pause that a school could be listed (out of thousands of high schools) as among the nation's best if it has high achievement gaps and low graduation rates, even if AP classes/tests are widely available to students.

This goes to their other point--the way the list is presented in the media--which I think is dead on. If the data from the Newsweek Challenge Index were presented in a table, one might reasonably expect the heading to be something along the lines of "Average number of AP/IB exams given per graduating student". That this could somehow be billed instead as a "Table of the Best High Schools in America" is a ridiculous stretch--not that there isn't plenty of evidence linking a rigorous high school curriculum to collegiate success. A recent longitudinal study released by the Department of Education last week concluded that, "The academic intensity of the student's high school curriculum still counts more than anything else in pre-collegiate history in providing momentum toward completing a bachelor's degree." Taking AP classes would certainly qualify as "academic intensity".

No matter what conclusions you draw about the Challenge Index, the condition that in part spurred Mathews' creation of the index remains: a total inability to compare one high school to another across states. Education Sector suggests that AYP be used instead, but this suggestion is completely worthless for solving Mathews' challenge. Since each individual state sets its own academic standards and its own AYP targets, comparing schools from different states would be a mind-boggling challenge (for those of you who scoff at the challenge, I'll give you a head start and still would wager you dollars to donuts (as my dad would say) that you won't finish in time for the holidays (Christmas if you're Bill O'Reilly)). In a time where there is considerable clamoring for schools and education to be opened up to more "market forces," the ability to compare one school to another reliably and faithfully seems like a crucial step.

In an even broader sense, Matthews' exercise in index creation and Education Sector's subsquent effort to run it through with the sharp blade of their research, belies a problem that I believe will only grow worse. In this era of NCLB and achievement data aplenty, we have created an environment that begs for data comparison and yet our system (lacking a standard robust measurement...AYP need not apply) makes this possible on only a very limited scale. In my mind, it defies logic that we would allow there to persist in this country 50 different definitions of what a high school student should know by the time they graduate. That this makes it nearly impossible for the average person to compare a school in Agoura Hills to one in Virginia Beach using anything but the simplest metrics (see the Challenge Index) is just one of the many unfortunate byproducts. But in a country that creates the BCS in lieu of a simple playoff system, maybe overly complicated is just the way we like things. Speaking of which, I'm not sure which thought is more intriguing: a weekly BCS style ranking system for schools or Newsweek, in the spirit of March Madness, appointing a selection committee to create the bracket for a first ever 64-high school academic rating tournament so they could stop determine the "best high school" in the country. Hmm...