A Rose by Any Other Name Dept.

Brian Leiter writes Why Did Loyola Law School Fall in US News? Because the Magazine Changed the School's Name, and Its Reputation Score Plunged!:

This really takes the cake for carelessness on the part of U.S. News. Loyola Law School in Los Angeles dropped from 63 to 71 in the overall U.S. News ranking this past spring, and for one primary reason: its reputation score among academics dropped from 2.6 to 2.3. But that kind of drop is extraordinary: the academic reputation scores move .1 in either direction all the time, without rhyme or reason, but only once in the last eight years did another school's peer reputation score drop that much. (The lawyer/judge reputation scores used to fluctuate more wildly, because the response rate was so low; U.S. News this year decided to average two years' worth of these reputation scores to make the results less [meaninglessly] volatile.)

So with only a 1 in 1,000 chance of this kind of movement, what else might explain the precipitous drop in academic reputation? Unfortunately, the explanation seems to be clear: U.S. News unilaterally changed the school's name on the survey: from "Loyola Law School" to "Loyola Marymount University." Loyola was the only school whose name was changed on last year's survey.

I've posted Loyola Dean Victor Gold's full message to the Loyola faculty on this topic after the jump.

Email from Dean Gold to Loyola Law School faculty:

Given the recent faculty email discussion regarding the name US News attaches to our school, I thought it would be appropriate to provide you with the following report on that issue.

As you know, Loyola Law School is ranked 71 in the latest US News survey. Last year we were ranked 63. The decline is the result of a drop in Peer Assessment, the single most heavily weighted of all criteria in the rankings. Last year our score was 2.6. This year it is 2.3. I have uncovered some facts that suggest why this happened.

US News has used a five point scale for Peer Assessment (reputation in the academic community) since 1998. For each of the 11 surveys published between 1998 and 2008, our score was either 2.6 or 2.5. This lack of volatility is typical for Peer Assessment. The last eight surveys show only one other instance when any law school's score dropped as much as .3 in one year. Since about 185 schools are ranked each year, these eight surveys presented well over a thousand opportunities for schools to drop .3 from one year to the next (7x185=1,295). It happened only twice.

What could have produced such an anomaly in Loyola Law School's score this year? It was not a change in the quality or quantity of our faculty's scholarship. Our productivity in the last year or two has been normal. We published articles in the principal law reviews at Stanford, Berkeley, Virginia, Penn, Minnesota, Iowa, Northwestern, Wash U, William & Mary, etc, etc. During the same period we published more than a dozen books with the likes of Oxford, NYU, Aspen, West, etc. etc. We are ranked 40th among all law schools in Social Science Research Network (SSRN) downloads for the last 12 months (5/18/2009 SSRN survey).

Our decline in Peer Assessment may have been caused by the magazine itself. Last fall US News changed the name it assigns to our school on the survey ballot. The ballot that US News distributed in the fall of 2008, which produced our 2.3 Peer Assessment score, referred to us as "Loyola Marymount University." On ballots for all prior years going as far back as I can find, we were called "Loyola Law School." In other words, the first time the ballot called us by a different name, our score fell by a magnitude nearly unprecedented in the history of the magazine's survey of Peer Assessment. No other school's name was changed on the 2008 ballot.

This drop in score following a name change is not surprising. While we are part of Loyola Marymount University, and proud of it, we have been known as Loyola Law School for 80 years. That name has been used in all our branding efforts. Most law school professors and deans know us by that name. We use that name consistently in an effort to avoid confusion with two other schools that have "Loyola" in their names. By changing what we are called on the survey ballot, the magazine may have confused some respondents. To add to the confusion, the version of the rankings published in spring 2008 called us Loyola Marymount University in print, but was based on a survey ballot sent in the fall of 2007 that called us Loyola Law School.

In early May I wrote the director of data research for US News and requested that the magazine return to calling us Loyola Law School. I provided him with the data described above. I also gave him a letter from the President of Loyola Marymount University making the same request and affirming that we are referred to as Loyola Law School even within the university itself. I have referred him to the Best Practices of the American Association for Public Opinion Research, which cautions researchers to take great care in matching question wording to the concepts being measured because the manner in which questions are asked can greatly affect survey results. He admitted to me in writing that the magazine changed what we are called on the ballot without considering whether a name change might affect survey results.

I had hoped that this problem would be easy to resolve. Frankly, I could not understand why US News would want to survey opinion about us by referring to a name that we do not use. The director of data research for US News tells me that the magazine wants to be consistent in the form of names it uses for schools. I hope that, in the end, the magazine will come to realize that its responsibility to give its readership reliable data is more important than format rules.

I have been promised a decision this month. I will keep you posted.

Posted by Rick Hasen at July 17, 2009 09:19 AM