While grading my Business Associations exams in three business days to meet the university's Draconian deadline, I have flashbacked to the NYT article from May 4 that criticized the new essay portion of the SAT. Dr. Les Perelman of MIT studied a sample of these essays and found that the length of the essay correlated with the score the essay received 90% of the time. That week, many bloggers mentioned this, and the general feeling was that something was wrong there.
Why? What if length is a good proxy for content?
Maybe not a perfect proxy, but a good enough proxy that given the alternative of hiring hundreds of essay graders and charging $100 for the SAT, we'll go ahead and tell graders to look at the length and skim over it. I looked back at the NYT article. Dr. Perelman never claims that length was a poor proxy for content. He does not say that the essays given the highest score were bad. He said they had factual errors, but the SAT guide says that factual errors will be ignored. Besides the errors, though, Dr. Perelman did not say that the essays were bad, whether because they were illogical, rambling, incoherent, or internally inconsistent.
Also, the article used as examples the longest essays, which received "4"s were 400 words long, and one short article that was 100 words long was used as an example of a "1." However, the article does not assert that the content did not warrant those grades. If I gave a 4-hour exam, and most of my students filled up four bluebooks (gulp), but one student filled up just one bluebook (holding for handwriting size, etc.), I would take the bet that the exam that was 25% of the size of the others would be the exam that earned the worst grade. I could be surprised, and I can conceive of an exam that is conscise and brilliant, but I would take that bet nevertheless.
Let me assure all my students, I read each and every exam, regardless of length, handwriting, cross-outs, organization, etc. But, I could make the generalization that the best exams are usually longer, usually neater in organization, and actually neater in handwriting, spelling, grammar, and punctuation. I usually have one exam that is beautiful in terms of form but completely devoid of meaningful content and one exam that is horrendously messy, unorganized, and frighteningly brief that captures everything I could expect. However, on average, that is not the case. Again, not a perfect proxy, but not a horrible one. If I ever get more than 3 days to grade, I will try to do a research project to see if length/organization/neatness/handwriting could be a proxy.
My colleague Eric tells me that at Epinions, they found an almost perfect correlation between the length of product reviews and ratings of those reviews. In fact, they discussed creating a computer program that could automatically rate reviews on length, punctuation, grammar, use of headings, and general neatness. So, I'm inclined to say that grading an SAT essay by glancing at the length, organization, spelling, punctuation, grammar, and neatness would not be completely misguided, given the time constraints.
TrackBack URL for this entry:
http://www.typepad.com/services/trackback/6a00d8345157d569e200d835464bbb69e2
Links to weblogs that reference Form as a Proxy for Substance: SAT Essay v. Law School Exam:
| Sun | Mon | Tue | Wed | Thu | Fri | Sat |
|---|---|---|---|---|---|---|
| 1 | 2 | 3 | 4 | |||
| 5 | 6 | 7 | 8 | 9 | 10 | 11 |
| 12 | 13 | 14 | 15 | 16 | 17 | 18 |
| 19 | 20 | 21 | 22 | 23 | 24 | 25 |
| 26 | 27 | 28 | 29 |
