A novel within document retrieval tool called ProfileSkim has been designed, developed and evaluated in this internally funded project.

The main components of the skimmer are a Relevance Profile Meter that displays the relevance profile across the document and a document browser that supports highlighting of the query words.

The main function of ProfileSkim is to enable users to identify, efficiently and effectively, relevant passages of text within long documents. The tool integrates passage retrieval, content-based document browsing and an interactive user interface. The key concept underpinning the tool is relevance profiling, in which a profile of retrieval status values is computed across a document in response to a query. Within the user interface, an interactive bar graph provides an overview of this profile, and through interaction with the graph the user can select and browse in situ potentially relevant passages within the document.

A user-centred, task-oriented, comparative evaluation has been conducted in the evaluation of ProfileSkim. FindSkim, which provides similar functionality to the web browser's "Find" command, was developed as a comparison system of ProfileSkim.

Analysis of the experiments confirmed that ProfileSkim was significantly more efficient than Find-Skim, as measured by time for task. The study also indicated that ProfileSkim was as least as effective as FindSkim in identifying relevant pages, as measured by traditional information retrieval measures. Based on qualitative data from questionnaires, there are strong evidences showing that the participants would be more satisfied when using ProfileSkim than FindSkim.

The experimental study confirmed the potential of relevance profiling improving within-document retrieval, and the interactive user interface attracting user's attention. Interactive Relevance Profiling should prove highly beneficial for users trying to identify relevant information within long documents.

Publications

Harper, D. J., Koychev, I., Sun, Y. and Pirie, I. (2004). Within Document Retrieval: A User-Centred Evaluation of Relevance Profiling. Information Retrieval, 2004(7): p. 265-290.

Harper, D. J., Koychev, I. and Sun, Y. (2003). Query-Based Document Skimming: A User-Centred Evaluation of Relevance Profiling. In proceedings of the European Conference on Information Retrieval Research, 377-392, Pisa, Italy, April 2003.

Harper, D. J., Coulthard, S. and Sun, Y. (2002). A Language Modelling Approach to Relevance Profiling for Document Browsing. In proceedings of the Joint Conference on Digital Libraries, 76-83, Oregon, USA, July 2002.

Presentation