During the last several days, Conglomerate bloggers have discussed several factors that might influence standing in the law school hierarchy. Gordon discussed scholarship as a driver of reputation and Bernie Black's critique of the new Harvard Record law school rankings, Vic alluded to an "X Factor" (and read the comments!), and Christine, citing some preliminary numbers, suggested that the X Factor might be faculty blogging. My first post also argued that SSRN may be a better barometer of faculty productivity than most people realize.
So let’s think about translating some of these thoughts into a concrete strategy. Personally, I think there is more to a great law school than just its faculty publication record, but I’ll concede that “more and better” scholarship (however one defines it) is an integral part of most law schools’ strategic plans. Thus, assuming scholarship is the goal, here is one approach that is sure to horrify many readers—a numbers-driven appointments committee.
This seemingly radical idea was first suggested by Paul Caron and Rafael Gely in a well-known book review of Moneyball. 82 Tex. L. Rev. 1483 (2004). The operative strategy is identical to Billy Beane’s famous management of the Oakland Athletics: Since most baseball teams (law schools) lack the budget of the New York Yankees (Harvard), the general manager (appointments committee) needs to find baseball players (teaching candidates) who are undervalued by the market.
To illustrate how this might play out in the AALS hiring market, consider the relative attractiveness of four hypothetical candidates:
- Candidate A has a JD from Yale, Yale Law Journal editor, prestigious judicial clerkship, three published articles, including a student note.
- Candidate B has the same sterling credentials as A but only one published article, no student note and a work-in-progress.
- Candidate C has a JD from Stanford, a PhD in a social science discipline, a work-in-progress but no student note or other publications.
- Candidate D went to a Top 15 law school, no clerkship, was an editor on a secondary law journal, but has three published articles, including a student note.
Caron & Gely’s compared several background characteristics of Leiter’s 50 most-cited young scholars with a random sample of professors who were hired at roughly the same time at comparably ranked law schools. Though their methodology (simple t-tests of means) is too simplistic to yield definitive results, it is worth noting that only two variables had any predictive power for “more and better” scholarship: (1) number of articles before first tenure-track job, and (2) publication of a student note. Judicial clerkships, law review membership, rank of graduating institution, other graduate degrees, and teaching experience were not significant predictors.
Under the typical hiring committee
heuristics, however, Candidate A will get lots of AALS interviews, but there
will be significant disagreements over B, C, and D. Many committees will prefer
B and C because of pedigree (Top 3 law school, PhD,
clerkship, law review, etc.). But
Caron’s & Gely’s statistics suggest that A and D,
because of their publication records, are better bets than B and C. Thus, a number-driven appointments committee
would search out more D’s and less B’s and C’s. (Note that many schools will vainly chase after too many A's.)
I suspect that many faculty members would have a hard time biting the credentials bullet—even if that is clearly the source of the market inefficiency. (Remember the catcher in Moneyball with great stats that the scouts didn’t want because he was “huge in the [posterior]”?)
Caron and Gely ultimately need a more sophisticated methodology (e.g., a multivariate regression model with a larger sample and more refined control variables) to better demonstrate the potential of a Moneyball strategy. But their initial analysis suggests that the traditional heuristics may indeed be overvaluing and undervaluing candidates in predictable ways. Further, the Black & Caron study suggests that SSRN downloads may have some ability to predict future citations. See p. 47, tbl. 10.
If Caron & Gely did collect the data for a refined study, would they publish the results in a law review (or, worse yet, a blog)? There might be more at stake here than a burst of SSRN downloads.
TrackBack URL for this entry:
Links to weblogs that reference A Numbers-Driven Appointments Committee:
1. Posted by Scott Moss on November 11, 2005 @ 13:28 | Permalink
Fascinating. I read Moneyball and have been a Bill James / baseball stats fan since the mid-80s... so it's embarassing that I haven't made the time to read the Gely/Carron article yet. My backlog runs about a year for "things I'm dying to read but have to put on the back burner because reading for my day job" (i.e., my main legal fields).... Anyhow, the point of that disclaimer was to ask a question that may be addressed in their article:
At first, Moneyball-esque statistical analysis was just a weird thing that some obscure authors did that made the rest of the baseball world laugh. At some point in the 90s, the ideas began to gain credence in baseball teams' front offices, and here we are today.
It's great that good idea won out -- but the down side is that all the baseball stat analyses used to be publicly published. Now that it's gone mainstream, much of it is proprietary knowledge. The Oakland A's sign some players that seem mediocre, and the hunch is that they know something, based on some new statistical measure, that they're not telling the competition.
Do you think law profs would necessarily tell us if they did a great study, came up with some whiz-bang statistical measure -- for either student selection or faculty hiring. On the one hand, they gain from keeping it a trade secret... on the other hand, they could publish it... If I were a dean, I'd offer an unusually HUGE research stpend to do a great study and keep it secret....
2. Posted by anon_cd on November 13, 2005 @ 5:33 | Permalink
Speaking as a C-D type AALS candidate (JD-PhD; 10+ articles, etc.) just back from the FRC, I can tell you that very few schools are using "moneyball" strategy. I had only 2 interviews with lower tier schools. Pedigree is king. The real question is why schools dont play moneyball.
3. Posted by Scott Moss on November 13, 2005 @ 18:44 | Permalink
Anon -- yes, "why they don't play moneyball" is the big question that the book Moneyball itself asked; these statistical methods were being published for over a decade before any baseball teams noticed, and even today, not all that many are persuaded. It's a bizarre failure of the free market when businesses keep pursuing empirically disproven strategies and ignore valuable freely available information that their competitiors are ignoring.
The short answer is that if Gely/Carron/Henderson are right, some schools will have figured it out, others will figure it out later, and others much much later.
4. Posted by anon_cd on November 14, 2005 @ 5:36 | Permalink
I think that part of the reason schools dont play moneyball strategy is that there may not be consensus on what a "win" is in law schools. Is it a faculty that publishes more and in better places? Not necessarily. Remember that faculty often have personal motivations that, while not explicitely expressed, may drive their decision making, and collectively, the decision making of the school. People have already addressed the subject of home school currency on this blog - faculty like to hire from their JD granting institutions (which tend to be top 10) because it retains the currency of their degree. Think about it, what's the point of going to Yale/Harvard/Stanford if it's really all about publishing? - Someone from top 50 state law school might very well be able to publish good articles, and lots of them.
However, I dont want to dwell on this particular point too much since it's already been discussed and seems to make everyone nervous. There are a number of other reasons that faculty may not want to emphasize the actual potential publishing capabilities of new hires - but obviously they could never say this and will always give lip service to the importance of scholarship -- while still engaging in strategies that fly in the face of "moneyball" type evidence and analysis.
5. Posted by Christine on November 14, 2005 @ 8:50 | Permalink
Just like the scouts in Moneyball, law schools have become very enamored of proxies -- more so than the end result that the proxy is supposed to predict. That's why you see statements in faculty meetings where persons are discussing candidates with 2 and 3 articles to the effect of "But why didn't X get a clerkship?" At what point does the proxy outweigh the predicted result? When schools send out "New Faculty" brochures, all the proxies are listed and touted, but not any existing scholarship. Look around at your school's literature -- schools still list professors' law school affiliations, decades after a body of scholarship has been created. Fewer words, I guess.