Brian Leiter's law faculty citation analysis is meeting plenty of resistance, to which Brian ably responds. As usual, Paul Caron has all of the links, including a link to a paper that Paul and Bernie Black wrote about using SSRN to rank law schools. According to Paul and Bernie, "citation counts are a respectable proxy for article quality, and correlate reasonably well with other measures." But, of course, "citation counts have limitations," which Paul summarizes in some handy bullet points on his blog. In looking at Brian's rankings in the area of business law, the most obvious "limitation" -- if you want to think of it that way -- is the following:
Dynamism. Cumulative citation counts favor more senior faculty and emphasize older work that accumulates citations over time.
That's inherent in counting citations, which typically start to appear in large numbers only years after an article is published. Of course, this is why Steve Bainbridge can crow about being the only person under 50 among the top 20 most cited faculty in business law.
But this "limitation" is also the strength of citation counts. At least in business law, Brian's rankings capture (mostly) scholars who have been productive for a long period of time. I see no one-hit wonders on the list. If you want a ranking of what's hot, try the SSRN rankings.
Finally, I was intrigued by Mary Dudziak's criticisms (here and here) of the rankings on the Legal History Blog. The gist of her criticisms is this: "The study is confined to the Westlaw JLR database which only includes legal publications." This objection intrigued me because it was one of the oft-voiced complaints about citation counts at Wisconsin, where many of the faculty produce interdisciplinary scholarship. Brian's response is both simple and persuasive: my rankings don't measure the effect of scholarship on disciplines outside of law. I would add something that Brian doesn't say (at least in this response): if you want to measure interdisciplinary impact, create your own rankings!
Last I heard, this suggestion was not lost on the folks in Wisconsin. Such a ranking would be a welcome addition to the rankings universe.
TrackBack URL for this entry:
Links to weblogs that reference The Brouhaha Over Faculty Citation Analysis:
1. Posted by Mary L. Dudziak on November 29, 2007 @ 6:48 | Permalink
Thanks for picking up on this discussion. Just a note regarding your comment: "if you want to measure interdisciplinary impact, create your own rankings!"
As my posts make clear, citations rankings will always be flawed because they favor certain topics. In history, scholars in larger fields (e.g. American legal history) will always received more citations to their work than equally eminent scholars in smaller fields (European and Asian legal history and medievalists). The solution is not to proliferate rankings. Instead, if you want to rank *schools*, go ahead. But don't rank *people*. There are too many flaws in individual rankings and, as my blog notes, too many ways that these rankings can be misused.