2.3.1: Techniques Employed in Formal Studies

The classic formal usability study (also known as a lab study) is conducted in a usability laboratory (Shneiderman and Plaisant, 2004), in which observers can be hidden behind a two-way window. However, a quiet room with a desk, chairs, and computer is oftentimes sufficient. Some studies are recorded with videotape or audiotape, and the content is later transcribed. User behavior can be recorded by people taking notes, but screen-capture programs allow for higher accuracy and greater completeness, especially if paired with click-capture and logging. Some studies use eye-tracking technology (Nielsen and Pernice, 2009) to determine exactly which parts of the screen people view, to better understand the mental process that participants are undergoing when assessing a search interface (see Figure 2.2).

Participants in a usability study should be asked to read and sign a consent form, indicating which modalities are being recorded (audio, video, screen capture, eye tracking) and how the participants' anonymity will be protected. Participants should be told that they are free to leave the study at any time and for any reason.

It is common to give participants warm-up or practice tasks in order to help them get familiar with the system and relax into the tasks. It is also important to pilot test usability studies before running the full study with recruited participants. Initially, pilot tests are often done using co-workers or otherwise easy to recruit individuals, before testing on participants from the desired user base.

Formal (and informal) studies do not necessarily have to be conducted with experimenters sitting in the same room as the participant, or directly observing the participant. Several studies of remote usability testing find that this method yields similar results in most cases to in-person testing (Christos et al., 2007, West and Lehman, 2006, McFadden et al., 2002). Remote testing has the advantage of making it easier to reach a more diverse user pool, and the study can be done in the participants' everyday work environment, and so is potentially more ecologically valid and more cost-effective than travel. Drawbacks to remote testing include difficulties in capturing facial expressions and non-verbal cues and less free-flowing communication between experimenter and participant. It might also be difficult to get experimental software working remotely, but special software has been developed to aid in remote testing that addresses this issue to some degree.

2.3.2: Balancing Condition Ordering Effects

A problem common to all usability studies is that order of exposure to experimental conditions can bias the results. It is common, when working with search interfaces, for participants to learn in several ways: about the tasks, the collection being searched over, and about the search interaction itself. Thus participants might get faster on a set of tasks independent of the different properties of the interfaces being compared, or they might get tired, making them slower on the later conditions. If timing data is being collected as part of a study, it is important to compute statistics that compare time taken at different phases of the study. It is also important to pilot test a study and shorten it if necessary if participants tire towards the end.

Participants learn about a topic while seeking information about it. Thus a participant should never see the same query or topic in two different conditions. The effect of prior exposure is much too great and invalidates any results seen in the second viewing. An example of this can be found in a study in which participants were asked the same question five different times, for five different visual displays. Not surprisingly, participants' performance for a given display was stronger for conditions viewed later in the study. Because the questions were repeated it is not possible to tell if this was the cause or if the task was one that got easier with practice, and any differences seen among the interfaces is called into question.

Participants can be influenced in other ways by order effects. If one interface is much better than another, then seeing the inferior interface second can cause it to have lower usability ratings than if it was viewed first. For example, in a study by the author and students comparing Flamenco, a faceted metadata interface, to a baseline standard search interface for image browsing, the order in which interfaces were viewed had a strong effect on the subjective ratings (such as “This interface is easy to use” or “This interface is flexible,” etc.). When the faceted interface was viewed first, the subjective ratings for the baseline were considerably lower than when baseline was the first interface shown (thus providing additional support for the superiority of the faceted interface) (Yee et al., 2003). Because the order of the interfaces was varied for different participants in this study, it was possible to detect this effect in the data analysis. Thus, to control for order effects, the experimenter should vary the order in which participants are exposed to the key conditions.

Formal experiments must be designed, just as interfaces are (Keppel et al., 1992, Kohavi et al., 2007, Kohavi et al., 2008). A description of a formal study design should state the independent (experimenter chosen) variables, such as the different designs under study or the queries assigned, as well as the dependent (response) variables, such as time elapsed, number of errors made, and the participants' subjective responses. The design must also describe the blocking of the experiment, which refers to how the different experimental conditions are combined, that is, which tasks are assigned to which participants using which interface designs, and in which order. The research questions or hypotheses being tested can then be stated in terms of the independent and dependent variables. The experiment blocking is devised in such a way as to allow statistical tests to determine significant differences and to confirm or refute the hypotheses. A description of the participants is also important, often including demographics of the participant pool, their familiarity with relevant technology, and sometimes the results of cognitive tests.

When comparing interfaces, it is common to use a within-participants design, meaning each participant views each key condition (e.g., each interface, each variation of the new feature). This allows for direct comparisons of the responses of a given individual, which can be useful for comparing subjective responses, timing, and other response variables, in order to control for individual variation. On the other hand, within-participants design has its drawbacks. In some cases participants can form a bias or develop a learning effect after interacting with an interface, or the experimenter may need to have a participant spend a lot of time on one condition, thus making a session last too long. In these cases, it can be better to instead do a between-participants design, in which each participant is shown only one key interface condition. This precludes responses from being compared directly, but it eliminates any artificial effects that occur from one participant seeing multiple designs. In most cases, more participants are required in the between-participants design in order to get statistically meaningful results, but each individual session can be shorter.

Because some tasks may be harder than others, it can also be important to control for which tasks are conducted on each interface condition. In order to control for both the order of exposure of interface and for the combination of interface and task, the experimenter can employ a Latin Square blocking design. A Latin Square can be thought of as a matrix or table with number of rows equalling number of columns. If no row or column shares the same value, the Latin Square condition holds (thus bearing some similarity to the game of Sudoku). An easy way to build an n x n Latin Square is to lay out the first row, and then rotate the values by one position along each column, and repeating until all rows are constructed. So, if the first row of a 3 x 3 Latin Square is 1, 2, 3, the next would be 2, 3, 1, and the last would be 3, 1, 2. Note that this layout does not achieve all possible orderings (of which there are n factorial), but for experiment design it is usually assumed that allowing each condition to appear in each position once is sufficient to capture relevant ordering effects.

As an example, consider a usability study by Aula, 2004 which had the goal of assessing the relative benefits of three methods of showing Web search results summaries (see Figure 2.3). In the Bold style, the summary was left unchanged from what was returned by the Google search engine, with query terms bolded, while in the Plain style, the summary was the same as in Bold but with the boldface highlighting removed. In the List condition, every time an ellipses appeared in the summary, a new list item was created, preceded by a small arrow. Incomplete sentences were marked with ellipses at the start and/or end of the list item.

This study was designed so that participants always saw 10 hits for each query, and in all cases, there were 9 distractors and 1 valid answer summary for the question. The 30 test queries were divided into task sets of 10 queries each (call these A, B, and C), and one-third of the participants saw the List view first, one-third saw the Bold view first, and one-third saw the Plain view first. The Latin Square design might look as shown in Table 2.1. Each row represents a participant group, containing 3 out of the 27 participants. The order in which the participants see the interfaces is shown from left to right. Note that the order in which the task sets were assigned was also varied to ensure that each query group is associated with a different interface type in each position in the test ordering.

This kind of blocking structure allows for statistical tests such as Analysis of Variance (ANOVA) (Stockburger, 1998, Keppel et al., 1992) that can determine whether or not there are interaction effects among the different conditions. In the case of the Aula (Aula, 2004) study, ANOVAs showed that there was a significant effect among interface types and task times, meaning that there is support for the hypothesis that the variation in task completion time was caused by the differences in the interfaces. ANOVAs also showed that although participants made errors 21% of the time, there was no significant effect of error on interface type.

To account for another potential cause of variation, Aula also balanced the location in which the single relevant document would appear in each task set. She ordered the results listings in such a way as to guarantee that for each task set, a correct answer occurred in each of the 10 positions in the list one time. Thus in task set A, a relevant document would appear in position 1 for one query, in position 2 for another query, and so on. This artificial element of the design helps counteract search users' tendency to select the top-ranked hits, allowing each interface an equal chance of having the correct answer in each position in the list. On the other hand, it may introduce an artifact into the study if it is the case that, say, the bolding interface works best when the correct answer falls in the first or second hit.

2.3.3: Obtaining Participants

Ideally, usability study participants are drawn from a pool of potential users of the system under study; for instance, an interface for entering medical record information should be tested by nurse practitioners. Often the true users of such a system are too difficult to recruit for academic studies, and so surrogates are found, such as interns who are training for a given position or graduate students in a field.

For academic HCI research, participants are usually recruited via flyers on buildings, email solicitations, as an (optional) part of a course or by a professional recruiting firm. Often the participants are computer science graduate students, who are not representative of most user populations in terms of their approach to and understanding of computer interfaces. Participants obtained in this manner are usually not a representative sample of the population of potential users, and so the results of such studies must be regarded in this light. More recently, the field of HCI has been moving towards recruiting more realistic participants in usability studies.

The number of participants required to observe statistically significant differences depends on the study design, the variability among the tasks and participants, and the number of factors being compared (Kohavi et al., 2008). Most academic usability studies use a very small number of participants, and often these are drawn from the researchers' environment. Again, these issues suggest that results of studies should be eyed with some skepticism. This is why, throughout this book, the number of participants, as well as the pool they are drawn from, are reported when a usability study is summarized. It is also why results of multiple studies are reported whenever available.

The massive scale of deployed search engines changes the participant-recruitment situation by allowing new ideas to be tested on thousands, or even tens of thousands of users, as discussed below in the section on large-scale log-based usability testing.

Another recent approach to recruiting participants is to use a outsourcing or “crowdsourcing” service, such as Amazon's MTurk service, in which tens of thousands of people sign up online to do quick tasks for small fees. This approach to recruiting participants was found to be a fast, effective way to run studies, particularly for relevance testing (Snow et al., 2008, Alonso et al., 2008, Kaisser et al., 2008).

2.3.4: Measuring Participants' Preferences

As mentioned above, it is important to measure participant preferences, or subjective responses, since interfaces that are not liked will not be used, given a choice. Participant preferences are usually obtained via questionnaires using Likert scales. In Likert scales, participants select a point within a qualitative range, such as “difficult to easy”, “strongly agree to strongly disagree”, “never to always”, and so on (Shneiderman and Plaisant, 2004). Most Likert scales have either 5, 7, or 9 degrees to choose among; odd numbers make it clear what the central or neutral choice is. If the participants can be expected to make fine-grained distinctions in their choices, it is preferable to use a wider scale, otherwise a narrower scale should be used. For example, if comparing a new interface against one that is known already to have strong positive reactions, a wider scale allows for participants to clearly indicate a preference above and beyond what is already familiar.

The meta-analysis by Hornbæk and Law, 2007 suggests ad hoc questionnaires are less reliable and have greater variability than studies that use standard questionnaires. One commonly used questionnaire for assessing subjective reactions to interactive interfaces is the Questionnaire for User Interaction Satisfaction (QUIS) tool (Chin et al., 1988, Shneiderman and Plaisant, 2004). It assesses reactions to questions pertaining to overall reactions to the system, characteristics of the graphical view, terminology, learning difficulty, and system capabilities, on a 9-point Likert scale. A more generally applicable questionnaire consisting of 5 sets of questions was analyzed by van Schaik and Ling, 2005, and was found to distinguish properties useful for measuring the quality of interaction in Web interfaces both for monitoring and improving the design of such sites. Sample questions from this questionnaire are:

• “Learning to use this site was easy.”
• “I felt lost.”
• “I judge the Web page to be: (very disordered 1 2 3 4 5 6 7 very ordered)”

Most of these are answered using a 7-point Likert scale ranging from “strongly agree” to “strongly disagree,” but the questions on aesthetics have their own scales, as shown in the third example.

When comparing two or more designs, it can be revealing to have participants assess each interface in isolation before comparing them directly. This provides a second check on the subjectivity scores for the same questions (Hornbæk and Law, 2007). For example, if in the direct comparison, interface A is rated more flexible than B, the experimenter can check the individual ratings for flexibility for the independent assessment for each interface in order to infer how strongly that difference is felt. If the participant gave a very low score for flexibility for design A and a high score for design B, then the difference can be assumed to be large, but if the scores are similar on the isolated questionnaires, the perceived differences in the comparison can be assumed to be of lesser degree. This is also useful to see if there are effects based on order of presentation. Finally, if the participant for some reason does not complete the entire session, partial subjective information will have been obtained.

Aesthetic impressions are an important form of subjective response, as they have been found to play a role in user acceptance and have been found to correlate with perceptions of an interface's quality, user satisfaction, and overall impression of a site (Hassenzahl, 2004, Lindgaard and Dudek, 2003). van der Heijden, 2003 found that the visual appeal of a Web site affected participants' enjoyment and perception of ease of use, and to a small degree, the usability of the system. Nakarada-Kordic and Lobb, 2005 report that viewers persevere longer in a search task on Web sites whose design appeals to them.

Another reason to evaluate participants' subjective responses is that objective measures and subjective preferences are sometimes only moderately correlated (Hornbæk and Law, 2007). Ben-Bassat et al., 2006 hypothesized that subjective responses to a system may be only partly influenced by its usability. To prove this, they conducted a study with 150 engineering undergraduate students in which they varied the aesthetic quality of a design as well as its usability (in terms of number of keystrokes needed to get a task done). Participants were asked to complete tasks using the different interfaces and then to fill out questionnaires about the interfaces' usability and aesthetics. The experimenters found that the interfaces that required fewer keystrokes were indeed considered more usable, but they also found that the high aesthetic interfaces were seen as slightly more usable than the low-aesthetic interface, even though those were slightly faster. Ben-Bassat et al., 2006 then told participants they would be paid according to how well they did the task in a subsequent experiment. Participants were asked to make bids in an auction format on the different interfaces, to determine which they would use. In this condition, people bid higher on the more efficient designs, independent of their aesthetic scores. Thus, they found that questionnaire-based assessments were affected by aesthetics, but auction-based, financially incentivized assessments were not.

In some cases it is desirable to separate effects of usability from those of branding or of visual design. For instance, to compare the relative benefits of the ranking abilities of search engine A versus B, it is important to re-render the results in some neutral layout that uses the same visual elements for both rankings.

2.5.1: Identifying Session Boundaries

Information seeking is often a complex process consisting of many steps. For this reason, many studies of query logs attempt to infer searchers' intent by observing their behavior across a sequence of steps or activities. This sequence is usually referred to as a search session, which can be defined as a sequence of requests made by a single user for a single navigation purpose (Huang et al., 2004).

There are several approaches for automatically distinguishing the boundaries of search sessions. The simplest method is to set a time threshold; when a user has been inactive for some amount of time, a cutoff is imposed, and all actions that happened before the cutoff are grouped into a session. He and Goker, 2000 found a range of 10 to 15 minutes of inactivity to be optimal for creating session boundaries for web logs. Silverstein et al., 1999 used five minutes, Anick, 2003 used 60 minutes, and Catledge and Pitkow, 1995 recommended 25.5 minutes.

Chen et al., 2002 recognize that timing information varies with both user and task, and so proposed an adaptive timeout method. They defined the “age” of an action to be the interval between an action and the current time. A session boundary was assigned when the age of an action was two times older than the average age of actions in the current session. Thus the time that is allowed to elapse before a session boundary is declared varies with the degree of activity the user is engaged in at different points in time.

Jansen et al., 2007a examined 2,465,145 interactions from 534,507 users of a search engine, comparing several methods for determined session duration, finding that the best method made use of IP address, a cookie (stored client-side information), and query reformulation patterns, the latter to distinguish when a given user is searching for several different information needs in rapid succession. Using this method, they found that 93% of sessions consisted of 3 or fewer queries, with a mean of 2.31 queries per session (SD 1.5). The mean session length using this method was 5 minutes and 15 seconds (SD 39 min).

2.5.2: Issues Surrounding Searcher Identity

In query log analysis, an individual person is usually associated with an IP address, although there are a number of problems with this approach: some people search using multiple different IP addresses, and the same IP address can be used by multiple searchers. Nonetheless, the IP address is a useful starting point for identifying individual searchers.

Note that for typical query log analysis, no attempt is made to determine the actual identity of the users; rather, the IP address is linked to an anonymized unique ID number. Nonetheless, there are important privacy issues associated with the retention and use of query log information. Major search engine companies keep their query logs in confidence, but academic researchers are at a great disadvantage if no such logs are available to them. For this reason, in some cases, query logs have been made available to academic researchers, usually without incident, but in one case to great notoriety (Bar-Ilan, 2007). Adar, 2007 and Xiong and Agichtein, 2007 discuss methods to anonymize the queries themselves, to balance the needs of certain types of research with user privacy, and Cooper, 2008 summarizes the issues from a policy perspective.

Despite these attempts, problems remain (Jones et al., 2007), and so query logs should be carefully handled. An alternative solution is to host a search engine and ask searchers to opt in to having their searches recorded for scientific purposes, with an option for them to remove any information they choose (Cooper, 2008).

Some major web-based companies support “toolbars” which the user must opt in to use and which provide value to the user. However, they monitor all of that user's web-based activity -- not just queries and clicks, but all navigation within the browser. This kind of information can not only lead to a much deeper understanding of peoples' information seeking processes, but can also record more potentially sensitive information. This data must be carefully anonymized and secured.

2.7.1: Avoid Experimenter Bias

A problem common to all usability studies, not just search interface studies, is that designers who evaluate their own designs often unconsciously introduce biases into the evaluation process. It is best to approach interface evaluation from a neutral scientific perspective, but this can be difficult when evaluating one's own design. One solution for getting a more objective read on the usability of an approach is to have the evaluation done by a third party, that is, different people than the designers of the new interface. It is important that the designers trust the evaluators, though, in order to have effective transfer of the results.

Unfortunately, it is often impractical to have outsiders evaluate a research design, so when evaluating one's own design, the experimenter should be aware of ways to avoid introducing bias. It is important that the experimenter does not “leak” information to the participants about which is the favored design. A common mistake is to say something like “We've developed a new interface which we'd like you to evaluate.” This suggests to the participants that the experimenter may be disappointed if the interface does not perform well, and so may discourage less than fully forthcoming responses from some participants. Neutral language such as “we are evaluating several different designs” is both accurate and non-biasing. It is also good practice to assign neutral names to the different designs or conditions; for example, naming the designs after mountain ranges or colors.

2.7.2: Encourage Participant Motivation

Participant motivation or interest can have a significant impact on results (Buchanan et al., 2005, Spool, 2002, Sutcliffe and Ennis, 1998). A highly motivated person will be inventive and try alternative avenues in a way that a bored person will not. Thus motivated participants can demonstrate how an interface will be used on the intense usage end of the spectrum. Unfortunately, many search usability studies ask people to do search on topics about which they are uninformed and/or uninterested. There is evidence that participants do not try hard when paid to participate in search studies on topics they do not care about. Rose, 2004 notes that in one study conducted by AltaVista, 55% of survey results had to be discarded; in some cases participants failed a simple effort test in which they were required to distinguish actual search results from random URLs.

One way to stimulate participant motivation is to allow people to search for information that they are interested in, or perform tasks that they care about (Russell et al., 2006). However, in principle, it is preferable to require all participants to do the same task (under varying conditions), in order to facilitate comparisons within those conditions. One way to help improve motivation but still have some control over the topics tested in a study is to ask participants to select a subset from a pre-defined list of topics, thus allowing them to avoid subjects about which they have no knowledge or interest (Teevan et al., 2005a), but still allowing for direct comparisons of those participants who choose the same tasks to complete.

Another way to motivate participants is to match participants to the task at hand at recruiting time. An early study by the author and students on the Cha-Cha search system (Chen et al., 1999) provides a telling lesson. Undergraduate students were recruited to assess the interface whose main benefit was its ability to familiarize users with the large, diverse campus intranet, and placed search results within that context. The system ended up being popular among campus administrators, but most undergraduates care very little about campus administrative structure, and found little value in seeing this information.

Having learned from this experience, evaluations of the Flamenco system in each case matched the collection to the participant base (Hearst et al., 2002, Yee et al., 2003). For example, when evaluating on recipes, participants who like to cook were recruited; when evaluating on an architecture slide library collection, professional and student architects were recruited, and when evaluating on a fine arts collection graduate and undergraduate art history majors were chosen as participants. In each case the tasks were designed to be of inherent interest to those participant groups. Nonetheless, it is difficult to do this accurately. For the recipes study, one participant disliked the browse interface intensively; upon further questioning it turned out that despite the fact that he loved cooking, he hated recipes, and therefore had a negative reaction to any recipe interface.

Spool, 2002 describes an innovative way to create a highly motivated participant -- ask them to envision a product they would like to buy, show them a Web site that should contain the product, and offer them the money required to buy the product, but not for any other use. To increase the motivation still more, he suggests asking the participant to think about how they want to spend the money, and then send them away to come back a week later. When the participants return, Spool claims they are even more motivated than before, having had time to further mentally refine their desired purchase.

2.7.3: Account for Participants' Individual Differences

Nielsen, 1993 notes that “the two most important issues for usability are the users' task and their individual characteristics and differences” (p. 43). He analyzed 30 published evaluations of hypertext systems and found that four of the 10 largest effects were due to individual differences between participants, and two were due to task differences (Nielsen, 1989a). Nielsen contrasted users' experience with the system under study, with computers in general, and with the task domain. In some domains (such as programming skill), individual ability can vary by a factor of 20 (Nielsen, 1993).

The effects of several types of individual differences on search performance have been studied in the literature: participants' knowledge of the task domain, participants' experience as searchers, and participants' cognitive differences; each of these are discussed below. The existence of these differences underscore the need to use large participant pools when conducting formal studies, in order to balance out the differences.

Many search usability studies have investigated the role of domain knowledge on search outcome, but the results are mixed (Vakkari, 2000a). Hembrooke et al., 2005 found that performance on information-intensive tasks is influenced by the background knowledge and cognitive skills of the study participants. Jacobson and Fusani, 1992 examined the experiences of 59 novice users with a full-text IR system, and used a regression model to assess the relative contributions of computer, system, and subject knowledge to search success. Their results indicated that all three variables played a role in the outcome. Wildemuth, 2004 found that medical students' search tactics changed over time as their knowledge of the domain changed. Sihvonen and Vakkari, 2004 found strong effects of domain knowledge on the use of thesaurus expansion. Vakkari, 2000a studied a set of 11 students as they formulated masters thesis topics, and observed changes in the choice of search terms and tactics as their knowledge of their topics increased. Vakkari and Hakala, 2000 found that the more participants knew about the task, the fewer references they accepted as relevant. Similarly, Spink et al., 1998 found that the less knowledgable their participants were, the more items they marked as partly relevant. Kelly and Cool, 2002 found, in a study of 36 participants, that efficacy of search (measured as number of relevant documents saved divided by the number of documents viewed) was significantly higher for those people who were familiar with the topic than for those who were very unfamiliar with it, meaning that the knowledgeable participants were more accurate at identifying relevant documents.

It can be stated with some assurance that when evaluating with a fixed set of tasks, it is important to ask in advance if participants already know the answer to the question or have knowledge of the topic. For example, Aula, 2004 finds significant differences in the results of a summary browsing study after removing data points corresponding to questions in which participants knew the answers in advance.

A number of search usability studies have assessed the effects of knowledge of the search process itself, contrasting expert and novice searchers, although there is no consensus on the criteria for these classifications (Aula, 2005). Some studies have observed that experts use different strategies than novices (Hölscher and Strube, 2000, Lazonder et al., 2000, White et al., 2005), but perhaps more tellingly, other studies have found interaction effects between search knowledge and domain expertise (Hölscher and Strube, 2000, Jenkins et al., 2003).

Tabatabai and Shore, 2005, in a study of 10 novices, 9 intermediates, and 10 expert searchers, found that novices found less relevant information and were more satisfied, while experts found more relevant results, and were more nervous about having missed information. However, participants who used better strategies had better results, independent of their experience level. The two most important strategies were evaluative (assessment of relevancy of sources, search engines, and results) and metacognition (self-reflection and monitoring). Tabatabai and Shore, 2005 wrote:

“Novices used backtracking to see where it would take them, hoping to get out of a labyrinth and find a comfort zone. On the contrary, intermediates and experts used it to go where they wanted to. Novices were less patient and relied more on trial-and-error. Impatience led them to navigate more, to click more, and to execute before spending enough time exploring or planning. ... Experts were more aware of their feelings, were not surprised by them, and relied on them to modify their strategies. Their positive outlook on the search gave them patience to wait for a response from the system. Novices, on the other hand, felt lost, disoriented, and caught in a labyrinth more often than experts.”

On the other hand, Zhang et al., 2005 found that, in a study including engineering students, increasing domain knowledge changed search behavior, but not search success. Lazonder et al., 2000 tested 25 Dutch students and found that training did not help much after achieving initial familiarity, and saw no effect based on domain knowledge.

Numerous studies show that participants who are generally more cognitively skilled perform better than other participants in information-intensive tasks, independent of the interfaces being compared. Allen, 1992 assessed a range of cognitive factors using the Kit of Factor-Referenced Cognitive Tests (Ekstrom et al., 1976) and then assessed students searching a bibliographic database for pre-assigned topics. (It is common for evaluation of information visualization interfaces to include the paper folding test from the Kit to measure spatial ability (Chen, 2000).) Allen, 1992 found that perceptual speed had an effect on the quality of searches, and logical reasoning, verbal comprehension, and spatial scanning abilities influenced search tactics.

Tombros and Crestani, 2000 note that fast participants remain fast and slow remain slow, independent of the interface. Pirolli et al., 2003 suggests a similar phenomenon when comparing an experimental and a standard browsing interface. When participants were ranked by their performance on an advanced file browser, and then were ranked by their performance on a standard file browser, there was a high correlation between the two rankings (high performers do well regardless of browser). They did another analysis that partitioned the sums of squares (i.e., partitioned the variance), and found that most of the performance effect in their study was due to individual differences rather than differences in the interface designs. To repeat a point made above, it is important to balance the participants in the different conditions when using a small number of participants.

2.7.4: Account for Differences in Tasks and Queries

Task selection in search interface evaluation can greatly affect the outcome of the study. Nielsen, 1993 (p. 185) notes that “the basic rule for test tasks is that they should be chosen to be as representative as possible of the uses to which the system will be eventually put in the field.” He also advises that tasks be small enough to be completed within the time frame but not so small as to be trivial. The following sections describe the effects of task variations, query descriptions, and bias in task selection when evaluating search interfaces.

In fully-automated IR system evaluations, such as those performed for the TREC ad hoc track, it is well-established that differences in the queries can overwhelm the outcome of a comparison between systems. It has been shown that in a typical TREC task, variation within a system across queries is usually higher than variation among top-scoring systems (Buckley and Walz, 2000). Usually no system is the best for all topics in the task, and in fact it is rare for any system to be above average for all topics (Buckley and Walz, 2000). The track administrators typically use 50 different queries in order to control for effects of the task, but even this is insufficient (Voorhees and Harman, 2000).

Adding human participants to the evaluation most likely increases the sensitivity to task variation. A person might respond emotionally to a query, or may have expert knowledge in a topic, or may find a topic excruciatingly boring. Individualized reactions such as these can influence the results of a search usability study. To compound the problem, human participants can tire, and so can only be expected to do a limited number of queries within a test session, thus reducing the number of queries that can be run and increasing the likelihood of irrelevant artifacts having an effect on the outcome.

As mentioned above, Lindgaard and Chattratichart, 2007 found no correlation between number of participants and number of problems found for usability testing of entire interfaces, but did find a correlation between the number of distinct tasks that participants are asked to complete and the number of problems found. This suggests that differences in tasks uncover different problems with the interface. For these reasons, search usability studies are increasingly differentiating query types, and assuming different interfaces will work better or worse when attempting different query types. Examples of varying the information need in this way is given by Woodruff et al., 2001 and Baudisch et al., 2004.

One way to reduce the variability somewhat is to pre-test the queries to determine if they are of similar difficulty, along either subjective or objective measures. It is common practice when conducting a formal study to show queries to one set of participants and have them rate them according to difficulty, interestingness, requiring external knowledge, and so on, and then select a subset with equivalent properties to show to a distinct set of participants in a usability study.

Although post-hoc analyses have found a lack of correlation between human estimates of difficulty and system performance on the TREC ad hoc task (Voorhees and Harman, 2000), and between system estimates of difficulty and system performance (Voorhees, 2004), there is some work suggesting that the weakest-performing queries can be predicted with some accuracy (Kwok, 2005). A recent large-scale analysis of query-system failures suggests that systems often fail for the same reasons on the same queries: the systems emphasize some aspects of a topic while ignoring others, do not handle general concepts well, and lack sufficient natural language understanding (Buckley, 2004). This analysis may prove useful in future for manual identification of query difficulty.

In addition, it does seem that people are good at estimating difficulty for other people (as opposed to for automated ranking evaluations), especially if the pre-testers use the search system that the study participants will be using (Bell and Ruthven, 2004).

As noted above, people find many different ways to express the same concept. Different query terms submitted to the same system can produce radically different results (Buckley and Walz, 2000), thus leading to significant differences in outcome that have little to do with the interface.

To control for this variability, many usability studies pre-assign the query terms that the participant enters. Pre-writing the queries of course greatly reduces the realism of the assessment. Martzoukou, 2004 argues against the artificiality of using pre-chosen queries for evaluation of information seeking studies, and Bilal, 2002 found children were more successful with self-generated queries than with assigned topics and terms.

However, depending on what is being evaluated, pre-assigned query terms can be acceptable. For example, for the search results summary display experiment described above, the query formulation process is far less important than the results assessment process, and so the artificiality of the pre-written queries is less questionable. But for a study of relevance feedback term suggestions, pre-entering the query terms may be quite unrealistic.

A common way to introduce experimenter bias is in the selection of tasks or queries (Käki and Aula, 2008). For example, in numerous studies of information visualization of search results, the tasks chosen are along the lines of “Which interface allows the user to say how many documents have K instances of term A?” or “Which directory has the largest number of documents?” Although visualization can help with tasks of this sort, counting questions like these are not particularly common search tasks for most users, and so violates Nielsen's (Nielsen, 1993) call to evaluate using realistic tasks. This kind of evaluation is quite common because it is often difficult to show that a visualization improves search results in a meaningful way. Separating out different components in order to isolate which makes a difference is good experimental practice, but the components tested must be meaningful or realistic in some way in order to truly inform the field.

As another example, it can be quite difficult to show significant timing differences in Web search evaluations. In one study, the queries were pre-determined and the terms used for the queries were pre-written. The evaluators were successful at showing timing improvements for a category-based view. However, the queries were designed to be maximally ambiguous so that any kind of grouping information would be helpful. For example, a query for the home page of the alternative rock band They Might Be Giants was written as giants, which is unrealistic from a query generation point of view (people searching for music and movie titles use most of the title words in their queries (Rose and Levinson, 2004) ) and is guaranteed to return a wide array of irrelevant results. A much better approach that showed more realistically that participants used grouping to help disambiguate queries was to use a longitudinal study as done by Käki, 2005b and discussed above.

Biasing can of course result from other considerations discussed above; by not varying the order of presentation properly, by asking participants to repeat tasks, and by polluting timing data by having documents pre-loaded for some conditions but not for others.

2.7.5: Control Test Collection Characteristics

When doing a formal usability test, it is important to evaluate the system on a large underlying collection (Willett, 1988). In the early days of experimental IR, much of the testing was done on the tiny Cranfield collection, which consisted of just 1,398 abstracts. In 1985, a revolutionary paper by Blair and Maron, 1985 investigated search over an important legal case consisting of 40,000 documents, showing that high recall remained distressingly elusive with modern systems. The TREC experiments expanded test domains to another order of magnitude in size, starting with about 1 GB of text for training and another gigabyte for testing in 1992 (Harman, 1993).

Unfortunately, the problem of testing IR systems on small collections still occurs today. In one study comparing search information visualization techniques, a collection of only 651 documents was used and another used only 163 documents. Testing a search visualization on such a small collection can yield misleading results, since it is much easier to show a graphic display with 10 or 100 hits than with 1,000 or 10,000; the results on such a small study are unlikely to scale well to a realistic collection.

It is also important to compare two interfaces which index the same underlying collection. Experimenters who wish to compare a new design against a commercial system, or against a research system whose software is unavailable, are tempted to do a study with systems indexing different documents. Unfortunately, the results of a comparison over different collections is rarely valid, as the behavior of search systems is very sensitive both to the queries issued and the underlying collection. If one collection contains a great deal of information that is of interest to a participant and the other does not, the interface is likely to be blamed for the difference. Similarly, if the format of the contents of the two collections is quite different -- for example, one containing full text and the other containing abstracts, or one containing poor content and the other containing high-quality content -- the participants are likely to conflate the differences in the collections with the differences in the interfaces.

2.7.6: Account for Differences in the Timing Response Variable

Some of the most important metrics for evaluation of interfaces include time required to complete tasks, effort level required, and number and severity of errors made. However, comparing search interfaces based on time required to complete tasks can be fraught with problems. As mentioned above, task difficulty and participant differences and query knowledge can have effects that must be factored into the study design. In addition, any search system that allows participants to follow hyperlinks and move away from the search results page is likely to find widely varying timing data. Thus, most timing-based Web search studies require the users to look at a page of results listings, or at most view documents one link away from the results page (Käki and Aula, 2008). Timing data can also be misleading as sometimes the value of a new information access interface is in that it allows users to browse and get to know a collection of information, and thus longer usage times can signal a more successful design. In some cases, a deeper search session, in which the user can comfortably explore, iterate, and refine their search, may take more time than an unsatisfying interface that makes the user want to get in and get out as quickly as possible (Rose, 2004).

Käki and Aula, 2008 point out that there is no standard timing measure for search speed, and suggest a proportional measure called qualified search speed that measures answer per minute, in terms of quality of answers. Using this measure, the experimenter can compare system A with B in terms of relevant answers per minute and irrelevant answers per minute. They found in several studies that the differentiating measure between interfaces was the increase in relevant speed, with irrelevant speed remaining nearly constant.

On a more practical note, timing data can be affected by variable network delays or contrasts between pages that have already been visited, either by the participant or by a preceding participant, and so are in the cache and fast to access. (A related mistake is to allow a participant to see a previous' participants query history in the web browser drop-down menu.) Best practice suggests downloading all pages that will be seen in advance, and having the participants access them locally, if the study allows for that, and clearing out the cache of all prior history when running multiple participants in the same environment.

As another way around the timing quandary, longitudinal studies can show changes in use over time, and can be used to differentiate which types of queries induce different types of response times.

2.7.7: Compare Against a Strong Baseline

Yet another way to bias the outcome of a usability study is to compare a new design against an unrealistic baseline system. For example, one study compared a visualization of search results against a Web search interface that hid the summaries, even though this kind of view is standard for Web search engines and is the one to beat.

All designs should be equally aesthetically pleasing; as discussed in Chapter 1, unattractive designs de-motivate participants, and vice versa. And as mentioned above, all designs being compared should index the same underlying collection.