The Pagan Agenda

Abstract

This paper is in response to Susan Feldman’s plea for an IR design strategy which compensates for many of the present obstacles to search optimization; these obstacles include the effects of too much information, too little coherence between repositories, and too little pedagogical support for new or novice searchers. An effective and useful IR system needs to perform in anticipation of user intentions. It needs to understand, or at least be able to interpret, users’ goals and strategies prior to their familiarity with a particular retrieval system. Due to the wide variance in relevance judgments regarding delivery modes in information retrieval, it seems pertinent to frame my conclusions on the subject according to a singular perspective on the issue. Thus this paper will examine the dynamic interaction between user and IR system, using the subjective criterion of utility as the true measurement of the system’s value. The user’s criteria for relevance judgments and the situational conceptualizations users employ in the duration of their searches are to be the motivating factors in the design of IR delivery modes for a digital library. From this framework an effective IR design strategy can be devised which includes 1) greater operational transparency to interface layout which explicitly pronounces what is accessible, how it can be retrieved and where to go for help 2) enhanced usability by way of familiarity, adopting as much as feasible the approachable and uncomplicated delivery modes of Google 3) Greater semantic interoperability with respects to the indexing of the database(s) in order to provide emergent functionality and adaptive-search capabilities which keep abreast of the ever-evolving needs of users.

In her article, “The High Cost of Not Finding Information,” Susan Feldman (2004) paints a bleak picture of modern day search optimization. She concludes that knowledge workers are impeded in their tasks by shortcomings in the search services they use, and these shortcomings are costing organizations millions in revenue a year. The three chief deficiencies she observes in modern search services include: too much recall, too little coherence between repositories and too little pedagogical support for new or novice searchers. The field of information science has thus far been unable to adequately address these deficiencies by way of a unified scientific theoretical model, and many scholars in the field are championing alternative approaches to the issue which provide more pragmatic solutions by way of greater emphasis on engineering design. One particular approach is to allow design to some extent grow organically from the records of use, employing patterns of use born out of such resources as user clickstreams and open source participation (O’Connor 2003) . This paper shares this pragmatic emphasis on engineering design as a solution to the above-mentioned obstacles, and contends that users’ criteria for relevance judgments and the situational conceptualizations users employ in the duration of their searches are to be the motivating factors in the design of IR delivery modes. In order to elaborate this point, this paper will use the design of a digital library as a case study through which Feldman’s concerns may be addressed.

Theory: The Utility Criterion

Traditional approaches to user evaluations have tended to cater to the interests of researchers rather than users, adopting in their methodologies an omniscient research-oriented perspective on what constitutes precision and relevancy in information retrieval (Blair 1990). A case in point is the undue consideration placed on unexamined documents in various measures of relevancy, which presupposes value in items that the user has had no direct contact with. As William Cooper (1976, p. 368) warns: “potential utility is no utility at all.” This, in addition to the wide variance in relevance judgments pervading IR research, has lead to approaches to user evaluations which associate relevancy far more purposefully with criteria divulged by the users themselves (Blair 1990; Cooper 1976).

Most notably applied in Cooper’s (1976) utility-theoretic approach, the ‘utility criterion’ is derived from a consensus of user feedback upon the overall usability of a particular site, in order to provide an authentic picture of how users interact with that site within their natural environments. Informally, usability testing collects qualitative information about how users interact with a site by giving them a set of tasks to complete and encouraging them to ‘think out loud’ as they interact with the interface (Nielsen 1995). The testing often incorporates established or potential users so as to utilize their own particular demographic characteristics in the survey.

From these tests, researchers can granulate the meaning of a ‘user’ by placing greater emphasis upon the different kinds of user characteristics interacting with a site. The effects of user demographics and, in a growing branch of study, their ‘mental models’ (Brandt & Uden 2003; Zhang 1998), have shown to have wide ranging implications for the way in which a user interacts with a site and is able to articulate his/her experience. A mental model may be described as an internal representation of knowledge which a person relies upon for problem-solving in a given domain. For example, a novice searcher with no prior computer experience would be expected to have a different mental model from a professional in the field of information science, and as the research of Xiangman Zhang (1998) shows, these initial discrepancies in understanding can result in radically divergent interpretations of usability. Thus consideration should be made to these granularities of user characteristics in the design choices so that they best satisfy the divergent needs of its patronage.

A consensus as to what constitutes representative utility would be incomplete without acknowledging the dynamic process at work in searching. Rather than treat the interaction between users and IR systems as static events, as single queries in need of single solutions, some in the field have placed particular emphasis on searching as a process through which the information seeker develops a series of query permutations in need of answers. Amanda Spink (2002) has developed an IR evaluation measure known as the ‘information problem shift’ which compensates for this dynamic occurrence of search behavior. She determines that the effectiveness of an IR system can be measured in terms of the “change or shift in the human information problems due to IR system interaction.” For this measure to take place data would need to be collected from the user before and after his/her interaction with the IR system.

A final word of caution on implementing the utility criterion in an IR design strategy: due to the possibilities of registering false positives in user evaluations one should not rely solely on the feedback of users, but whenever feasible weigh them alongside expert opinion. Examples of false positives distorting the realities of a systems performance have been documented in a variety of studies (Blair 1990; Hildreth 2001). User criteria for relevance judgments and the situational conceptualizations they employ in their searches should be the decisive factor in the design strategy, but these sources can be aided by the inclusion of auxiliary expert information as to which documents are deemed relevant. As Cooper (p. 370) iterates: “…though the utilities associated with the examined documents are indeed the only ones of direct interest, these utilities can sometimes be better estimated with the help of some auxiliary information about the unexamined documents. The auxiliary information could include information about the potential utility of the unexamined documents had they been retrieved, and in this sense such utilities may be of indirect interest even though it is the utility of the examined documents upon which retrieval performance must ultimately be based.”

Practice: Implementation into a Digital Library

1)Develop greater operational transparency to interface layout to explicitly pronounce what is accessible, how it can be retrieved and where to go for help

A good logical design should allow the user to find the documents he needs and intelligibly place these documents in context to all available documents. One of the recurring criticisms made in user evaluations is to the lack of conceptual help provided in search services (Warren 2005; Brandt & Uden 2003). There is a great anxiety amongst many information seekers as to the seemingly endless potentialities of information from search services, particularly if these services are in some way integrated with the internet. If a digital library chooses to offer portal access to internet-related resources in addition to its collection this option needs to be explicitly pronounced on the search interface and activated by the user. Whenever possible the aspects associated with the digital collection should share a uniform layout, graphic vocabulary and key commands that are explicitly differentiated from the external resources. The screen arrangement should anchor the digital collection and situate external resources in peripheral zones.

Clear and explicit differentiation needs to be sustained within the collection as well. Ideally graphic icons should be used to depict the different document formats, as well as the different retrieval methods (i.e. print, save). Site navigation should be aided by a breadcrumb trail which helps the user conceptualize his particular place in the search process to offset confusion. Another strategy for counteracting user anxiety is to employ some additional visualization techniques to articulate the semantic associations between the retrieved documents, something along the lines of the data mining tools such as Grokker (www.grokker.com) and Kartoo (www.kartoo.com) or by way of a side bar ontology which sets in list form the plausible semantic categories of the query entered (Warren 2005). The categories should be weighted, either numerically or visually by size, to alert the user as to the relative size of recall available to each semantic context in order to browse effectively.
The diversity of mental models interacting with the site needs to be addressed. A possible strategy to consider is to have different versions of the interface accessible according to the initial choice made by the user about his familiarity with the service. For novice searchers, or in accordance to the research pertaining to that particular mental model a scaffolding approach to the site could be employed which caters to the particular needs of the user characteristics, filling in potential knowledge gaps with additional cues for help (Brandt & Uden 2003; Zhang 1998). This scaffolding could include the use of split-screens which provide more but briefer results lists for those who tend not to scroll past one page, or create blinking applets to encourage scrolling down (Brandt & Uden 2003). Assess the particular demographic of user characteristics for the digital library and customize the search service in this fashion to reflect the salient groups, for example, have entry points to the search service which cater specifically to novice searchers with no prior computer experience, secondary and post-secondary students, and information professionals (Zhang 1998).

2) Enhanced usability by way of familiarity, adopting as much as feasible the approachable and uncomplicated delivery modes of Google

Jakob Nielsen (1995), the former head of Human Factors Research at Sun Microsystems, states that a usable interface must be easy to learn, easy to remember, efficient to use, cause the fewest errors and be pleasant to use. In order to accommodate some of this criteria a search service would greatly benefit from a shared familiarity with the delivery modes of Google. Google has remained a popular choice for online query searches due largely to its balance of simplicity and breadth of information. As Susan Augustine and Courtney Greene found in their usability study at the University of Illinois at Chicago, an increasing number of college students are “Google-bred” (Augustine & Greene 2002), and desire a single search box to input their queries.

In order to compete with the ease and simplicity of Google, digital libraries need to entice use by being similarly approachable. As much as feasible, the digital library should adopt the same search box and syntax precedents established by Google, including the use of quotation marks to enclose fixed phrases and pre-designed boxes for Boolean operators. Google’s spell corrector would also be a welcome addition to the digital library design, offering the user alternative spellings of misspelled words at the top of the results page. Similarly, verbose terminology on the site should be replaced with easy to understand vernacular. As indicated in the article by Burton Callicott and Debbie Vaughn (Callicott & Vaughn 2003) many students do not fully understand the meanings of terms such as “periodicals”, “database”, “online catalog”, “e-journals”.

Google unfortunately lacks many of the features for searching that digital libraries possess, most notably, field searching through multiple access points. Most of the popular search engines (i.e. Google, Alta Vista, Lycos, Infoseek) remain fairly simple full-text retrieval systems based on ranking algorithms which do not employ weighted search terms (Blair 2002). In their unprecedented STAIRS experiment (Blair 1990; 2002), Blair and Maron demonstrated that “the effectiveness of full-text retrieval has not been substantiated by reliable recall measures on realistically large databases”, and were in fact limited to a maximum recall value of 20% for a database of 40,000 documents, this despite the perceived effectiveness that was documented in the user evaluations of the study. This linguistic ‘brute fact’ as Blair puts it, leaves much to be desired in the overall efficiency of simple full-text retrieval systems like Google, so that while it is advantageous to carry over some of the superficial appeal of Google into the design for a digital library this should not be done to the detriment of the site’s search functionality.

3) Develop greater semantic interoperability with respects to the indexing of the database(s) in order to provide emergent functionality and adaptive-search capabilities

In their book, “Adaptive Information”, Jeffrey T. Pollock and Ralph Hodgson (p. 6) define ‘semantic interoperability’ as “a dynamic enterprise capability derived from the application of special software technologies (such as reasoners, inference engines, ontologies, and models) that infer, relate, interpret, and classify the implicit meanings of digital content without human involvement.” This should not be confused with the general ambition of the semantic web project. Semantic interoperability is merely a subset of this broader project, used within relatively confined domains for specific goals. Pollock and Hodgson believe semantic interoperability represents a 3- to 5- year vision, whereas the semantic web represents a 5- to 10- year vision. The goals of SI are both to integrate a level of functionality into the software of a given business so that it insulates the business from incurring compatibility issues of the technology evolution and to enrich the conceptual value of the data through the employment of a semantically-enhanced data-modeling framework.

W3C standardized markup languages such as RDF (Resource Description Framework) and OWL (Web Ontology Language) provide suitably general data-modeling frameworks for a universally seamless mapping of all prospective documents and concepts (Warren 2006). Part of the process of these schemas is to identify all objects, concepts and relationships on the web with unique URIs (Warren 2006). This would allow IR systems to break open documents and link them by their salient concepts, a great improvement over the hit and miss strategy of simple full-text retrieval services.

Since the semantic web vision has not yet been realized and markup languages such as RDF and/or OWL are not expected to be universally adopted as the standard(s) of information description on the web for at least another decade, systems such as our digital library will need to bridge this divide by combining an ontological approach with standard text-based search facilities (Warren 2006). This semantic interoperability between the two legacies will allow the system to gain from emergent functionality, and give the user the opportunity to make the transition between the known and the unknown in a seamless fashion.

Greater semantic interoperability with respects to the indexing of the database(s) can also provide adaptive-search capabilities which help to prevent search exhaustivity by customizing the recall to suit the particular needs of individual users or group of users. Efforts have been made in IR research (Warren 2005; Balfe & Smyth 2005) to semi-automate the creation and management of ontologies and metadata; these efforts include the use of 1) AI agents which determine emergent relationships born out of the semantic connections between documents 2) the search history of users or group of users. Ideally, our digital library should take advantage of both approaches, having the indexing of the collection managed semi-automatically; the technical viability of such a enterprise will be tested in user trials in 2006 by Paul Warren as part of the SEKT project (Warren 2005). A corollary to this, I-Spy software uses search caches within communities of users to enhance relevancy. This technique takes advantage of the growing trend of search engines allowing third parties to provide search boxes. So long as there is a relatively narrow and uniform context to the site’s domain of interest (i.e. News, Sports) I-Spy can effectively re-use past search behaviors as a means to re-rank implied preferences (Balfe & Smyth 2005). In order to take advantage of this technology our digital library should make available on its homepage a hyperlink subject directory of the collection which can narrow the domain of interest prior to a search, so that the search terms and strategies employed within each domain provide domain-specific semantic relationships that is learned by the system. So long as there is a log-in feature to the digital library, user clickstreams and feedback can be monitored in order to further enhance the relevancy of recall. Feedback forms should be made available at the end of each recall list to encourage user interaction in the forming of a more adaptive-search service.

References

Augustine, S, & Greene C. (2002). Discovering how students search a library web site: a usability study case. College & Research Libraries, 63 (4): 354-365.

Balfe, E, & Smyth B. (2005). An analysis of query similarity in collaborative web search. Advances in Information Retrieval. 27th European Conference on Information Retrieval Research, ECIR . Santiago de Compostela, Spain, March 21-23 Proceedings: 330-336.

Blair, D. C. (2002). The challenge of commercial document retrieval, part II: A strategy for document searching based on identifiable collection size. Information Processing and Management, 38(2), 293-304.

Blair, D. C. (1990). Language and representation in information retrieval. Amsterdam: Elsevier Science Publishers.

Brandt, D. S., & Uden, L. (2003). Insight into mental models of novice internet searchers. Association for Computing Machinery.Communications of the ACM, 46(7), 133.

Callicott, B., & Vaughn, D. (2003). Broccoli librarianship and google-bred patrons, or what’s wrong with usability testing? College and Undergraduate Libraries, 10(2), 1-18.

Cooper, W. S. (1976). The paradoxical role of unexamined documents in the evaluation of retrieval effectiveness. Information Processing & Management, 12(6), 367-375.

Feldman, S. (2004). The high cost of not finding information. KM World, 13(3), 8-10.
Hildreth, Charles R. (2001). Accounting for users’ inflated assessments of on-line catalogue search performance and usefulness: an experimental study. Information Research, 6. Retrieved April 7th, 2006, from http://informationr.net/ir/6-2/paper101.html

Nielsen, Jakob. (1995). Multimedia and Hypertext: The Internet and Beyond. Boston: AP Professional, 1995, p. 281.

OConnor, B. C. (2003). In Copeland J. H., Kearns J. L. (Eds.), Hunting and gathering on the information savanna : Conversations on modeling human search abilities. Lanham, Md.: Scarecrow Press.

Pollock, J. T. (2004). In Hodgson R. (Ed.), Adaptive information : Improving business through semantic interoperability, grid computing, and enterprise integration. Hoboken, N.J.: Wiley-Interscience.

Spink, A. (2002). A user-centered approach to evaluating human interaction with web search engines: An exploratory study. Information Processing and Management, 38(3), 401-426.
Warren, P. (2006). Knowledge management and the semantic web: From scenario to technology. IEEE Intelligent Systems, 21(1), 53-59.

Warren, P. (2005). Applying Semantic Technology to a Digital Library: A Case Study
Zhang, X. (1998). In University of Toronto. (Ed.), A study of the effects of user characteristics on mental models of information retrieval systems. |cc1998.