Information foraging through clustering and summarization: A self-organizing approach

Abstract

Successful knowledge management requires efficient tools to manage information in the form of text. However, our productivity in generating information has exceeded our ability to process it, and the dream of creating an information-rich society has become a nightmare of information overload. Although researchers and developers believe that interactive information access systems based on clustering and summarization offer a potential remedy to that problem, there is as yet no empirical evidence showing superiority of those tools over traditional keyword search. This dissertation attempted to determine whether automated clustering can help to find relevant information by suggesting an innovative implementation and verifying its potential ability to be of help. Our implementation is based on Kohonen's self-organizing maps and acts as a visualization layer between the user and a keyword-based search engine. We used the clustering properties of self-organizing maps to create a summary of search results. The user relies on this summary when deciding whether and how to provide additional feedback to the system to obtain more relevant documents. We have resolved multiple issues related to the speed and quality of output associated with self-organizing maps and created a version (Adaptive Search) that allows interactive Internet searching. We have performed user studies and a controlled experiment in order to test the proposed approach. In a laboratory experiment, subjects spent less time finding correct answers using Adaptive Search than using the search engine directly. In addition, the documents containing answers were positioned consistently higher in the rank-ordered lists suggested by Adaptive Search as opposed to the lists suggested by the search engine. The search engine that we used was AltaVista, known to be one of the most popular, comprehensive and flexible engines on the Web. Our main conclusion is that indeed information clustering helps information seekers if properly implemented.