Some years ago, a former chief justice of the Michigan supreme court, Thomas Brennan, sent a questionnaire to a hundred or so of his fellow lawyers, asking them to rank a list of ten law schools in order of quality.
"They included a good sample of the big names. Harvard. Yale. University of Michigan. And some lesser-known schools. John Marshall. Thomas Cooley," Brennan wrote. "As I recall, they ranked Penn State's law school right about in the middle of the pack. Maybe fifth among ten schools listed. Of course, Penn State doesn't have a law school."
Those lawyers put Penn State in the middle of the pack, even though every fact they thought they knew about Penn State's law school was an illusion, because in their minds Penn State is a middle-of-the-pack brand. (Penn State does have a law school today, by the way.)
And where do these kinds of reputational prejudices come from?
According to Michael Bastedo, an educational sociologist at the University of Michigan, "rankings drive reputation." When a person has a task of assessing the relative merits of something he knows nothing about, a school like Penn State can do little to improve its position. To go higher than forty-seventh in the U.S. News ranking of universities, it needs a better reputation score, and to get a better reputation score it needs to be higher than forty-seventh. The U.S. News ratings are a self-fulfilling prophecy.
Bastedo says that reputation ratings can sometimes work very well. It makes sense, for example, to ask professors within a field to rate others in their field: they read one another's, attend the same conferences, and hire one another's graduate students, so they have real knowledge on which to base an opinion. Reputation scores can work for one-dimensional rankings, created by people with specialized knowledge.
Bottom Line: Who comes out on top, in any ranking system, is really about who is doing the ranking.
Source: Malcolm Gladwell, The Order of Things in The New Yorker, February 14, 2011