DETECTABILITY OF CORRECTNESS: A SIGNAL DETECTION MEASURE OF KNOWING THAT ONE KNOWS

Abstract

To decide wisely whether to act on the basis of what one knows, whether to rely upon one's knowledge is an important ingredient in managerial skill. It depends upon knowing what one knows and what one can do. Knowing that one knows is conceptualized as the ability to discriminate what one knows from what one does not. It is measured by detectability of correctness A(c), a nonparametric signal detection measure of the capacity to distinguish correct from incorrect responses. The research reported is an exploration of that concept and of the characteristics of A(c) as a measure of it. The importance of knowing that one knows has been recognized, but little work has been done on it. Previous research along three separate lines is critically reviewed: philosophical analysis in epistemology, empirical research on calibration of subjective probabilities, and studies related to feelings of knowing. The measure A(c) is then developed and experimental results indicate that it is independent of the measure of knowing, the proportion of correct responses. It is shown that when there is opportunity to choose which questions in a test are to be answered, the expected score can be increased by an amount depending on the value of A(c). This effect is, essentially, a statement of the condition for validity of A(c) as a measure of knowing that one knows. The prediction is verified empirically. Reliability of A(c), assessed by the split-half method, is found to be higher than the reliability of the proportion of correct responses. Measurement of A(c) requires respondent-generated answers that can be unambiguously scored. The cloze procedure, of filling in missing words in verbal passages, common in reading research and testing, meets these requirements and has been used in the present research. The application is the first, as far as is known, in which confidence judgments have been obtained with cloze, and A(c) is expected to prove useful as a performance measure in cloze testing. In cloze experiments with native speakers of three different languages, who were also fluent in English, it was found that cloze scores did not distinguish between the native language and English but that A(c) was consistently higher for the native language. This finding and the results of another experiment identifying words heard in noise tend strongly to disconfirm the hypothesis that conscious attention to skill learning leads to one's knowing more about what one knows. Perhaps the opposite is the case. Finally, there is significant, but not conclusive, evidence that detectability of correctness can be improved by learning. It is concluded that knowing that one knows is a useful concept separate from knowing and is suitably measured by A(c), but that the determinants of it and its relation to other abilities are yet to be determined.