An automatic text mining framework for knowledge discovery on the Web

Abstract

As the World Wide Web proliferates, the amounts of data and information available have outpaced human ability to analyze them. Information overload is becoming ever more serious. Effectively and efficiently discovering knowledge on the Web has become a challenge. This dissertation investigates an automatic text mining framework for knowledge discovery on the Web. It consists of five generic steps: collection, conversion, extraction, analysis, and visualization. Input to and output of the framework are respectively Web data and knowledge discovered after applying the steps. Combinations of data and text mining techniques were used to assist human analysis in different scenarios. The research question was determining how knowledge discovery can be enhanced by using the framework. Three empirical studies applying the framework to business intelligence applications were conducted. First, the framework was applied to building a business intelligence search portal that provides meta-searching, Web page summarization, and result categorization. The portal was found to perform comparably to existing search engines in searching and browsing. Users liked its search and analysis capabilities. Thus, the framework can be used to analyze and integrate information distributed in heterogeneous sources. Second, the framework was applied to developing two browsing methods for clustering and visualizing business Web pages. In terms of precision, recall and accuracy, both outperformed list and map displays of search engine results. Users strongly favored the methods' usability and quality. Thus, the framework facilitated exploration of business intelligence from numerous results. Third, the framework was applied to classifying Web pages into different business stakeholder types. Experimental results showed that the framework could effectively help classify certain frequently appearing stakeholder types (e.g., partners). Users strongly preferred the efficiency and capability of this application. Thus, the framework helped identify and extract business stakeholder relationships. In conclusion, our framework alleviated information overload and enhanced human analysis on the Web effectively and efficiently. The research thereby contributes to developing a useful and comprehensive framework for knowledge discovery on the Web and to achieving better understanding of human-computer interaction.