A Path to Concept-based Information Access: From National Collaboratories to Digital Libraries

Abstract

This research aims to provide a semantic, concept-based retrieval option that could supplement existing information retrieval options. Our proposed approach is based on textual analysis of a large corpus of domain-specific documents in order to generate a large set of subject vocabularies. By adopting cluster analysis techniques to analyze the co-occurrence probabilities of the subject vocabularies, a similarity matrix of vocabularies can be built to represent the important concepts and their weighted “relevance” relationships in the subject domain. To create a network of concepts, which we refer to as the “concept space” for the subject domain, we propose to develop general AI-based graph traversal algorithms and graph matching algorithms to automatically translate a searcher’ s preferred vocabularies into a set of the most semantically relevant terms in the database’s underlying subject domain. By providing a more understandable, system-generated, semantics-rich concept space plus algorithms to assist in concept/information spaces traversal, we believe we can greatly alleviate both information overload and the vocabulary problem. In this chapter, we first review our concept space approach and the associated algorithms in Section 2. In Section 3, we describe our experience in using such an approach. In Section 4, we summarize our research findings and our plan for building a semantics-rich Interspace for the Illinois Digital Library project.