Conglomerate Blog: Business, Law, Economics & Society

As noted below (and on every other law blog, it seems), this year's law school rankings by U.S. News have leaked. Ranking methodologies are not my forte, but I am responding to Dan Solove's invitation to discuss possible improvements. In the interests of full disclosure, one of the sub-items on my agenda is to speak to the alumni and students of BYU's law school about our numbers.

As to the overall ranking, Brian Leiter makes a convincing arguments that it "combines too many factors, in an inexplicable formula, and much of the underlying data isn't reliable, and some of it (e.g., expenditures on secretarial salaries and electriciy) isn't even relevant." Nevertheless, that overall ranking seems to be as important as it has ever been, judging by traffic on discussion boards and local conversations. And it's not clear to me that a proliferation of other rankings (like this new one from Vault) or the distribution of other information is going to change that.

One improvement would be to create clusters of schools (as is done with the lower-ranked schools), rather than the numbered ranking that prevails in the Top 100. Every school that experiences a major shift from one year to the next has a story about a single input or perhaps two inputs that caused the shift. The schools are so tightly clustered that small changes produce huge effects. Last year, for example, BYU dropped 10 spots, a result attributable largely to a small decline in our entering class GPA. You might think that entering class GPA is an important factor in law school quality, but I doubt that BYU is a materially different place as a result of the GPA of one class.

This year, the GPA and LSAT are both up, placing our student numbers comfortably among the Top 20-25 schools on that measure, but our overall ranking declined. We believe that this effect was the result of U.S. News' decision to change the placement calculation. In prior years, students who were not seeking employment were not counted in the denominator. This year, they were counted against our placement numbers. I am told that we have a number of former students who passed the bar, but decided to dedicate themselves to their families rather than to seek employment. I suspect that BYU has an inordinate number of such students, and given the small size of each class (approximately 150 students), this change in calculation dropped our placement number from over 97% to just over 91%. I understand that the category of "students who are not looking" is subject to abuse, but I hope U.S. News revisits their decision to include such students in the denominator.

The bigger issue here is that U.S. News is constantly changing the formula to produce changes in the rankings. This is good marketing, and it curbs to some extent the impulse to play to the rankings. By substituting clusters for numerical ranks, U.S. News would moderate some of the effects of these changes, though that is precisely why it has no incentive to embrace clusters. The interest among readers in small changes from one year to the next is what keeps this machine running. Quite brilliant on their part.

UPDATE: Brian Leiter has posted "An Open Letter to Bob Morse of U.S. News," which focuses on Brian's usual theme of the manipulability of the numbers. Two points in response:

1. Getting data that are not manipulable by the law schools sounds wonderful, but the most meaningful information about a law school may not be objectively verifiable. Brian realizes this, of course, but it bears repeating.

2. I owe Bob Morse an apology in connection with my claim that "U.S. News is constantly changing the formula to produce changes in the rankings." Brian observes,

US News has not made any major adjustments to the ranking formula since 1999, the year you began adjusting expenditures for differences in cost-of-living in different regions of the country. Since then, U.S. News has made some adjustments to definitions of certain items of data, but those were all minor and were, correctly, designed to increase reliability by more closely tracking ABA data.

Whether the changes were "minor" and "correct" depends on where you sit -- the change in the employment number was not "minor" for BYU and several other schools, but it's hard to see how this change was "correct" in any meaningful sense of that word -- but most of the changes that I know about were prompted by the ABA, not by U.S. News. That includes the manner in which employment is tallied, as described in my original post. I am told that the ABA eliminated the separate category for those "not seeking employment," and I understand that schools that are peculiarly affected by the change are lobbying for a return to the old categories.

Law Schools/Lawyering | Bookmark

TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8345157d569e200e5517687168833

Links to weblogs that reference Rankings Talk:

As noted below (and on every other law blog, it seems), this year's law school rankings by U.S. News have leaked. Ranking methodologies are not my forte, but I am responding to Dan Solove's invitation to discuss possible improvements. In the interests of full disclosure, one of the sub-items on my agenda is to speak to the alumni and students of BYU's law school about our numbers.

As to the overall ranking, Brian Leiter makes a convincing arguments that it "combines too many factors, in an inexplicable formula, and much of the underlying data isn't reliable, and some of it (e.g., expenditures on secretarial salaries and electriciy) isn't even relevant." Nevertheless, that overall ranking seems to be as important as it has ever been, judging by traffic on discussion boards and local conversations. And it's not clear to me that a proliferation of other rankings (like this new one from Vault) or the distribution of other information is going to change that.

One improvement would be to create clusters of schools (as is done with the lower-ranked schools), rather than the numbered ranking that prevails in the Top 100. Every school that experiences a major shift from one year to the next has a story about a single input or perhaps two inputs that caused the shift. The schools are so tightly clustered that small changes produce huge effects. Last year, for example, BYU dropped 10 spots, a result attributable largely to a small decline in our entering class GPA. You might think that entering class GPA is an important factor in law school quality, but I doubt that BYU is a materially different place as a result of the GPA of one class.

This year, the GPA and LSAT are both up, placing our student numbers comfortably among the Top 20-25 schools on that measure, but our overall ranking declined. We believe that this effect was the result of U.S. News' decision to change the placement calculation. In prior years, students who were not seeking employment were not counted in the denominator. This year, they were counted against our placement numbers. I am told that we have a number of former students who passed the bar, but decided to dedicate themselves to their families rather than to seek employment. I suspect that BYU has an inordinate number of such students, and given the small size of each class (approximately 150 students), this change in calculation dropped our placement number from over 97% to just over 91%. I understand that the category of "students who are not looking" is subject to abuse, but I hope U.S. News revisits their decision to include such students in the denominator.

The bigger issue here is that U.S. News is constantly changing the formula to produce changes in the rankings. This is good marketing, and it curbs to some extent the impulse to play to the rankings. By substituting clusters for numerical ranks, U.S. News would moderate some of the effects of these changes, though that is precisely why it has no incentive to embrace clusters. The interest among readers in small changes from one year to the next is what keeps this machine running. Quite brilliant on their part.

UPDATE: Brian Leiter has posted "An Open Letter to Bob Morse of U.S. News," which focuses on Brian's usual theme of the manipulability of the numbers. Two points in response:

1. Getting data that are not manipulable by the law schools sounds wonderful, but the most meaningful information about a law school may not be objectively verifiable. Brian realizes this, of course, but it bears repeating.

2. I owe Bob Morse an apology in connection with my claim that "U.S. News is constantly changing the formula to produce changes in the rankings." Brian observes,

US News has not made any major adjustments to the ranking formula since 1999, the year you began adjusting expenditures for differences in cost-of-living in different regions of the country. Since then, U.S. News has made some adjustments to definitions of certain items of data, but those were all minor and were, correctly, designed to increase reliability by more closely tracking ABA data.

Whether the changes were "minor" and "correct" depends on where you sit -- the change in the employment number was not "minor" for BYU and several other schools, but it's hard to see how this change was "correct" in any meaningful sense of that word -- but most of the changes that I know about were prompted by the ABA, not by U.S. News. That includes the manner in which employment is tallied, as described in my original post. I am told that the ABA eliminated the separate category for those "not seeking employment," and I understand that schools that are peculiarly affected by the change are lobbying for a return to the old categories.

Sun	Mon	Tue	Wed	Thu	Fri	Sat
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31