Abstract:

We offer a case-study in irrationality, showing that even in a high stakes, deliberate context, highly intelligent professionals may adopt dominated practices. Multiple-choice tests (MCTs) enjoy many advantages that made them popular tools in educational and psychological measurement. But they suffer from the so-called guessing problem: test-makers cannot distinguish lucky guesses from answers based on knowledge. One way professional test-makers have dealt with this problem is by attempting to lower the incentive to guess, through penalizing errors (called formula scoring). Another is to rid tests of various cues (e.g., a preponderance of correct answers in middle positions) that might help testwise test-takers guess at better than chance odds. Key balancing is the strategy test-takers adopted for ridding tests of positional biases. We show that formula scoring and key balancing, though widespread and intuitively appealing, are in fact "irrational" practices. They do not dispose of the guessing problem and are fraught with problems of their own. Yet they persist, even in the presence of more rational alternatives: Number right scoring is superior to formula scoring, and key randomization is superior to key balancing.

Website