Facilitating Knowledge Discovery by Mining the Content and Link Structure of the Web

Abstract

Given the vast amount of online information covering almost all aspects of human endeavor, the Internet, especially the Web, is clearly a fertile ground for data mining research from which to extract valuable knowledge. Web mining is the application of data mining techniques to extract knowledge from Web data, including Web documents, Web hyperlink structure, and Web usage logs.Traditional Web mining research has been mainly focused on addressing the information overload problem. Many information retrieval (IR) and artificial intelligence (AI) techniques have been adopted or developed to identify relevant information from the Web to meet users' specific information needs. However, most existing studies do not fully explore the social and behavioral aspects of the Web. Thus, the primary goal of this dissertation is to develop an integrated research framework that extends traditional Web mining methodologies to fully explore the technical, social, and behavioral aspects of Web knowledge discovery.My dissertation framework is composed of technical and social/behavioral components. In the technical component of my dissertation, a set of domain specific Web collection building, Web content and link structure mining, and Web knowledge presentation techniques were developed. These techniques were tested in a series of case studies to demonstrate their effectiveness and efficiency in facilitating knowledge discovery in various domains.The social/behavioral component of my dissertation is to explore the application of Web mining technology as a new means to study the social interactions and behavior of Web content providers and users. Several case studies were conducted to extract knowledge on covert organizations' resource allocation plans, information management policies, and technical sophistication using Web mining techniques. Such knowledge would be very difficult to obtain through other means.The major contributions of this dissertation are twofold. First, it proposed a set of new Web mining techniques that can help facilitate knowledge discovery in various domains. Second, it demonstrated the effectiveness and efficiency of applying Web mining techniques in extracting social and behavioral knowledge in different contexts.