BRIDGING THE SEMANTIC GAP: EXPLORING DESCRIPTIVE VOCABULARY FOR IMAGE STRUCTURE

Abstract

This research makes a methodological contribution to the development of faceted vocabularies and suggests a potentially significant tool for the development of more effective image retrieval systems. The research project applied an innovative experimental methodology to collect terms used by subjects in the description of images drawn from three domains. The resulting natural language vocabulary was then analyzed to identify a set of concepts that were shared across subjects. These concepts were subsequently organized as a faceted vocabulary that can be used to describe the shapes and relationships between shapes that constitute the internal spatial composition -- or internal contextuality -- of images. Because the vocabulary minimizes the terminological confusion surrounding the representation of the content and internal composition of digital images in Content-Based Image Retrieval [CBIR] systems, it can be applied to develop more effective image retrieval metrics and to enhance the selection of criteria for similarity judgments for CBIR systems. CBIR is a technology made possible by the binary nature of the computer. Although CBIR is used for the representation and retrieval of digital images, these systems make no attempt either to establish a basis for similarity judgments generated by query-by-pictorial-example searches or to address the connection between image content and its internal spatial composition. The disconnect between physical data (the binary code of the computer) and its conceptual interpretation (the intellectual code of the searcher) is known as the semantic gap. A descriptive vocabulary capable of representing the internal visual structure of images has the potential to bridge this gap by connecting physical data with its conceptual interpretation. This research project addressed three questions: Is there a shared vocabulary of terms used by subjects to represent the internal contextuality (i.e., composition) of images? Can the natural language terms be organized into concepts? And, if there is a vocabulary of concepts, is it shared across subject pairs? A natural language vocabulary was identified on the basis of term occurrence in oral descriptions provided by 21 pairs of subjects participating in a referential communication task. In this experiment, each subject pair generated oral descriptions for 14 of 182 images drawn from the domains of abstract art, satellite imagery and photo-microscopy. Analysis of the natural language vocabulary identified a set of 1,319 unique terms; these terms were collapsed into 545 concepts which were subsequently organized into a faceted vocabulary. Frequency of occurrence and domain distribution were tallied for each term and concept of the vocabulary. A shared-ness rating scale was devised to measure subject agreement on concept use. Rank ordering of concepts by shared-ness measure demonstrated which concepts were more broadly shared across subject pairs. To determine if the concepts generated by subject pairs were used consistently by each pair across the three domains the subjects were considered to be â judgesâ and the Spearman rank correlation was computed to indicate inter-rater reliability. Correlation analysis indicated that subject pairs tended to agree in the extent to which they used certain concepts across multiple domains and 14 concepts with the highest shared-ness sums would form the heart of a shared vocabulary. This faceted vocabulary can contribute to the development of more effective image retrieval metrics and interfaces to minimize the terminological confusion and conceptual overlap that currently exists in most CBIR systems. For both the user and the system, the concepts in the faceted vocabulary can be used to represent shapes and relationships between shapes (i.e., internal contextuality) that constitute the internal spatial composition of an image. Representation of internal contextuality would contribute to more effective image search and retrieval by facilitating the construction of more precise feature queries by the user as well as the selection of criteria for similarity judgments in CBIR applications. In addition, reliance of subjects on the use of analogy to describe images suggests that the faceted vocabulary of terms and concepts could be used to provide both the user and the CBIR system with a link to the visual shape represented by a verbal construct. Developing a visual vocabulary of shapes and relationships could be an important application of the controlled vocabulary that emerged from this study. Verbal access to concepts could serve as entry points leading into the visual vocabulary where shapes would be paired with specific low-level terms.