May 09, 2007
Robert Higgs on Peer Review
Posted by Gordon Smith

We do a lot of hand-wringing about student-edited law reviews in the legal academy, so I thought Robert Higg's perspective on peer-reviewed journals, while not entirely novel, was worth quoting because he makes his points with flair:

Peer review, on which lay people place great weight, varies from important, where the editors and the referees are competent and responsible, to a complete farce, where they are not. As a rule, not surprisingly, the process operates somewhere in the middle, being more than a joke but less than the nearly flawless system of Olympian scrutiny that outsiders imagine it to be. Any journal editor who desires, for whatever reason, to knock down a submission can easily do so by choosing referees he knows full well will knock it down; likewise, he can easily obtain favorable referee reports. As I have always counseled young people whose work was rejected, seemingly on improper or insufficient grounds, the system is a crap shoot. Personal vendettas, ideological conflicts, professional jealousies, methodological disagreements, sheer self-promotion and a great deal of plain incompetence and irresponsibility are no strangers to the scientific world; indeed, that world is rife with these all-too-human attributes. In no event can peer review ensure that research is correct in its procedures or its conclusions. The history of every science is a chronicle of one mistake after another. In some sciences these mistakes are largely weeded out in the course of time; in others they persist for extended periods; and in some sciences, such as economics, actual scientific retrogression may continue for generations under the misguided belief that it is really progress.
...
Researchers who employ unorthodox methods or theoretical frameworks have great difficulty under modern conditions in getting their findings published in the "best" journals or, at times, in any scientific journal. Scientific innovators or creative eccentrics always strike the great mass of practitioners as nut cases―until it becomes impossible to deny their findings, a time that often comes only after one generation's professional ring-masters have died off. Science is an odd undertaking: everybody strives to make the next breakthrough, yet when someone does, he is often greeted as if he were carrying the ebola virus. Too many people have too much invested in the reigning ideas; for those people an acknowledgment of their own idea's bankruptcy is tantamount to an admission that they have wasted their lives. Often, perhaps to avoid cognitive dissonance, they never admit that their ideas were wrong. Most important, as a rule, in science as elsewhere, to get along, you must go along.

Commenting on this, Peter Klein speculates about a better world:

The challenge is to design an institution that minimizes both Type I error and Type II error, taking into account the costs of each (presumably the costs of the former — missing an important breakthrough — are higher than the costs of the latter, which are largely wasted trees, electrons, and time, though the lower overall signal-to-noise ratio increases the chance that a published breakthrough will be overlooked).

No one in her right mind would design the law review system from scratch, but if you are worried only about error costs, it doesn't fare badly in comparison to peer review. Law reviews address the potential for Type I errors by allowing for placement of almost everything somewhere. Usually quite fast, too. The absence of standards and the availability of multiple submissions are effective lubricants.

The notion of a "Type II errors" in a world of universal placement makes sense only if the source of the placement matters. As I have noted here, "the problem with the notion that placement is a proxy for quality is that law reviews don't have standards for publication that might distinguish articles in one journal from articles in another. Nevertheless, most of us tend to make some quality judgments based on the placement of an article."

Proceeding from that point, I think most legal academics would agree that Type II errors are numerous. And for the sake of argument, I am willing to concede that Type II errors are more numerous than in peer-reviewed journals. But, as Peter notes, the costs associated with Type II errors may be low. Noise? Yes, lots of it. But most of us do not rely heavily on the placement signal anyway, preferring to sort directly using electronic search mechanisms. So I am positing that the costs associated with Type II errors are fairly small.

So there you have it. Very few Type I errors and a large number of inexpensive Type II errors. Could be worse.

Legal Scholarship | Bookmark

TrackBacks (0)

TrackBack URL for this entry:
https://www.typepad.com/services/trackback/6a00d8345157d569e200d8353b72bb69e2

Links to weblogs that reference Robert Higgs on Peer Review:

Bloggers
Papers
Posts
Recent Comments
Popular Threads
Search The Glom
The Glom on Twitter
Archives by Topic
Archives by Date
January 2019
Sun Mon Tue Wed Thu Fri Sat
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    
Miscellaneous Links