IBM has traditionally been recognized for the quality and completeness of its product documentation. Many of us can remember the complete library of books that was shipped with the first IBM PCs and PC/XTs (IBM Personal Computer Model XT [extended technology]), with dedicated volumes for detailed setup instructions, the IBM DOS command reference, how-to lessons for pre-installed software such as VisiCalc, as well as comprehensive hardware upgrade and maintenance procedures. As the PC marketplace became increasingly competitive in the 1990s with the emergence of the below-$1000 PC, cost pressures across the industry forced manufacturers to significantly reduce the scope and quality of their printed documentation, forcing the uninitiated to rely on third-party books or the expert in the next cubicle to explain the intricacies of PC use. For example, by 1998, two of four of the top-tier mobile computer vendors (with the exception of those in Japan) had moved to a worldwide documentation strategy that included more on-screen information than print information.

The IBM ThinkPad* product line resisted this trend, and continued to ship a 250-page Users Guide, a detailed Setup Guide, and a mini-manual called Up and Running.¹ By 1998, competitive cost pressures had become too intense to continue this copious library of printed product documentation. The ThinkPad User-Centered Design (UCD) team set out to defend the user's basic right to quality documentation while satisfying an executive mandate to drastically reduce the cost of printed documentation.

Customer research and information

In the same time frame, the ThinkPad marketing team conducted market segmentation research, which revealed several classes of users, who differed in their usage style.² These mobile computer users ranged from the PC-savvy and highly mobile “professional nomads” to the more deskbound and less savvy “convenient mobility” users, who used notebook computers to extend the working day at home. In all cases, however, these users were typically well beyond the novice level for PC use and mobile enough to make printed documentation less practical than softcopy alternatives.

The UCD team had researched mobile computer user frustrations in the U.S., Japan, and Europe, and found several issues that could be addressed by improved documentation and “help” information.^2,3 In particular, these customer frustration issues seemed to be attributable to two factors. First, there was a lack of consistency among the general and problem-solving information provided in print documentation and the marketing and problem-solving information provided on ThinkPad Web sites. Second, there were frustrations with the features that were unique to mobile computers as compared to their desktop counterparts, such as the use of battery power, mobile pointing devices, and communications settings.

This customer research brought new insights that generated IBM interest in on-screen information, as the usage scenario for mobile computers seemed less compatible with printed books. Moreover, an integrated system of on-screen documentation and Internet-based resources had the potential to address several of the frustrations related to finding product and service information from IBM Web sites.

Advantages and disadvantages of on-screen information

On-screen information offers several benefits to mobile computer users. Most important, many of the problems identified in the customer research involved usage while away from the office (maximizing battery life while on an airplane, connecting to other networks, etc). When users were most likely to need to consult documentation, they did not have access to the printed manuals. On-screen documentation had the advantage of traveling with users. When IBM was designing on-screen documentation, some competitors noticed this advantage at the same time and deployed similar documentation on their mobile computers. However, this on-screen documentation only mirrored their printed version and failed to take advantage of many of the capabilities that are unique to on-screen media.

Two of the most important of these capabilities, searching and interactive graphics, were utilized by the UCD team. The searching capability of on-screen documentation allows users to go beyond the usual printed tables of contents and indexes with more efficiency, enabling searches for synonyms or very specific text strings far beyond what would be practical to incorporate into a printed manual. Interactive graphics can vividly illustrate processes such as the physical tasks necessary for manipulating hardware (like removing batteries and ejecting devices), which have always been difficult to illustrate with print documentation.

Although on-screen documentation does have many advantages, there are a few disadvantages as well. Some issues lend themselves better to print documentation,⁴ and this is especially true for issues related to original setup and troubleshooting. For example, what should users do when the screen is blank or the computer cannot be started? Also, studies indicate that under some conditions, reading text from displays may be slower and more error-prone than reading from paper,⁵ and that there is a clear tendency for users to print out long electronic documents.

Given the tension among portability, technological improvements, and cost-reduction goals, as well as the potential usability disadvantages of softcopy documentation, the overall initiative to reengineer ThinkPad information deliverables was well suited for a User Engineering (UE) approach.⁶

Developing the intelligent on-line information system

Our mission was to replace print documentation—to reduce cost and complexity—by changing the way that product information is created, distributed, and used. Although the UCD practitioners were aware of some case studies showing a reduction in customer satisfaction as a result of the use of on-screen documentation, they agreed to set the success criteria for this initiative in terms of cost reduction, so long as customer satisfaction could be maintained by introducing some innovations that took advantage of the on-screen medium. Thus, the team defined ideal usage scenarios for an “intelligent on-line information system” (IOIS) that harnessed the power of Internet technologies to overcome the shortcomings and complexities of print documentation. The principle of “the right information (in the right format) at the right time and place” guided the information and technical architects on the team. However, complexities in achieving this goal meant that the entire method of authoring, producing, publishing, and integrating a ThinkPad's user information had to be transformed.

Setting the success criteria. Our set of success criteria was twofold: the achievement of product cost and development efficiencies and the maintenance of customer satisfaction. This reflected an emerging attitude of UE in IBM—that best-of-breed usability must be judged within the business reality of a product's market landscape. As such, the target for product cost was to reduce the print publication costs by two dollars, as compared with a benchmark predecessor system, in order to be competitive in an increasingly commoditized PC marketplace. Thus, the remaining printed document had to be very small (less than 80 pages) in order to use low-cost printing and binding methods. Development costs were targeted as a zero sum: even with more elaborate multimedia and software-programmed formats, the development team was expected to reallocate resources toward the new skills required. This proved to be very challenging, and finally the management team relented and allowed a small increase in development costs for the first release of the IOIS to account for skill transformation and retraining.

Customer satisfaction was defined in three ways. First, according to IBM's UCD process, the IOIS had to be tested against a best-of-breed competitor and “meet or beat” it on predefined usability metrics. Second, the team tracked press reviews with the goal of observing an increase in positive comments concerning the IOIS. Third, the team used an existing, semi-annual, large-sample customer satisfaction survey, with the goal of maintaining satisfaction with documentation at the same level as that of pre-IOIS customers.

Authoring the right information. Product segmentation and the research into customers' frustrations gave insights into what should be the “right information” to maintain customer satisfaction. The IOIS team set out to define a master list of content topics based on the research (see Reference 2 for a more detailed review). First, mobile computer users in most of the world were found to be more expert PC users than the typical consumer desktop PC user, with the exception of Japan, where mobile computers were often the first PC purchase made by a family. The research indicated that battery use and communications setup were two of the most frequently cited frustrations. Thus, the team focused on topics that differentiated mobile PCs from desktop PCs (e.g., managing battery power, managing communications settings by location, making presentations, etc.) instead of PC basics (e.g., initial setup, starting applications, basic word processing, e-mail, etc.).

Another definition of the “right” information was the most current information. Thus, as new features of the mobile computer were enabled with accessories, as software updates became available, or as the service and support team refined troubleshooting information, users needed access to the most up-to-date information. The entire method of authoring, producing, publishing, and integrating a ThinkPad's user information had to be transformed. In fact, one of the primary drawbacks of print documentation is that it cannot be revised after it is manufactured, except for awkward “readme” files or Web site “hints and tips.” In any case, print documentation often may be outdated and erroneous, causing errors and confusion among users.

This was particularly true for mobile computers. Typically purchased by companies for their employees, mobile computers are highly customized by each customer with software and options that differ from the basic model originally purchased. To address this, the IOIS team defined information that was particularly volatile, and treated it differently. Some of this information, such as troubleshooting, software updates, and accessories information, was made available on the IBM Web site only, so that it could be updated as needed, with the IOIS providing only an intelligent link (see “The right time and place”) to that information. While basic, unchanging procedures were documented directly in the IOIS, dynamic links were given for the user to check for the most updated information.

Finally, equally as important as defining the right information was to define the right format. As the adage goes, “A picture is worth a thousand words,” and the IOIS team felt that complex procedures and concepts would be best communicated visually, through the use of diagrams, images, and animations. Thus, twelve procedures were demonstrated as animations, several hundred supporting diagrams and images were used, and an entire language of informational cues (icons, color schemes, dimensionality) were defined and enforced in a design style guide.

The right time and place. The team had defined “intelligent” in IOIS to mean that IOIS provided reference and task-oriented information with rich media such as animations on the computer, but could also intelligently link to the latest product and service information through the Internet. Additionally, presenting information at the right time and place was part of the intelligence of the IOIS.

The initial concepts for the IOIS assumed that there would be an on-screen navigation panel analogous to the table of contents of a book, a browseable keyword index, and a full-text search engine. Considerable attention was paid to the section headings and topic titles to avoid technical jargon and to use simple, task-oriented language instead. This seemed logical for reference and learning-oriented topics where users had the specific goal of finding information regarding some task they were trying to accomplish. Moreover, the IOIS team defined specific sections at the top of the contents list entitled About Your ThinkPad and Everyday Essentials to provide places for initial setup tasks and an overview of system features, respectively, with particular attention to the most frustrating items, as identified by research. An example of the main IOIS interface, called Access ThinkPad,⁷ is shown in Figure 1.

Figure 1

For device setup and self-maintenance software, the IOIS team presented the content of typical printed procedures for software utilities and applications in a different way to address the “right time and place” requirement. Printed software instruction manuals use step-by-step instructions and pictures of the application screens with references. When viewing the instructions on the computer, pictures of the screen are redundant and awkward because the user is looking at the application on the screen already. Moreover, the step-by-step instructions typically start with the navigation steps to start the software, and then get to the proper place in the software (e.g., “click Start, Control Panel, and then select Power Management … Click the Advanced tab …”). However, users would find this tedious with on-screen instructions, wondering why the instructions “aren't smart enough” to launch the software to the right place automatically. Thus, the IOIS team defined the “cue card” medium, a help panel with a single procedure that, when selected, automatically launched the application, or provided a launch button to the referenced software so that the “reading” and “doing” could be viewed side-by-side (see Figure 2). In other cases, particularly for setup tasks, the user might think the software should be guided, intuitive, self-documented, and thus require no instructions. For important tasks, the IOIS team therefore defined setup wizards that integrated instructions directly into the setup software, with a logical, guided sequence (see Figure 3).

Figure 2   Figure 3

Integration and transformation

For a variety of reasons, HTML was the format of choice for the IOIS. HTML had matured as a rich format with hyperlinking, JavaScript** functions, graphics, and animation capabilities. In addition, the technical writers were already familiar with tag-based authoring. Finally, the integration with Web-based content would be easier and more controlled if the IOIS was rooted in HTML. However, many pitfalls were identified that would make implementing IOIS in HTML much more complex. The requirement to launch utilities and applications directly from the IOIS made the standard HTML browser unusable because launching an application from a Web browser causes a security warning to be displayed. This and other behaviors also require complex “dual-coding” of HTML, to accommodate both the Microsoft Internet Explorer** and Netscape Navigator** browsers. Moreover, dual coding would be required to support multiple operating systems which had different subdirectory structures (e.g., C:\windows directory for Microsoft Windows 98**, versus C:\WINNT for Microsoft Windows 2000**). Ultimately, the architects learned that the various versions of the Microsoft Windows operating system for different languages also had different names for subdirectories (e.g. C:\Programs versus C:\Programmes), adding to the complexity of resolving application launch paths.

Providing full-text search in the HTTP protocol would require server code to be installed on the ThinkPad itself to process the search queries and output the results. ThinkPad customers rejected the HTTP server requirement as a potential security and manageability problem, not warranted for a simple on-screen information system.

Another issue was the integration of the IOIS with the ThinkPad Web site. At the time that the content was to be authored for the IOIS, the structure and address of the Web site would not be known. In addition, the ThinkPad Web site was structured hierarchically, with navigation to the specific content for a particular model of ThinkPad requiring the user to select product category, machine type, model number, configuration, and so forth. Users would not want to be dropped into the Web site at a high level in the navigation, but would want to be placed directly onto the specific page for their exact model of ThinkPad. Thus, the precise link that the user wanted for accessories, service, or upgrade information would be impossible to author into the IOIS because the information system could not be customized to that level. In addition, indexes are difficult to create and maintain in native HTML, particularly when there are multiple translated versions of the HTML.

Due to these and other integration issues, the IOIS architecture could not be implemented with the standard HTML technology of the time. The designers turned to the emerging standard for compiled HTML help. This standard, offering a help engine which is built into the operating system with its own search feature, eliminated the need to perform searches using the client-server HTTP model. The use of HTML help resolved the index, search, and security issues, and also allowed the existing information development staff to use off-the-shelf authoring tools to create the majority of the content. However, the compiled HTML format did not solve all of the issues.

In order to provide a dynamic link to ThinkPad model and configuration-specific Web pages, the architects devised a secure method to transmit model and configuration information (by using cookies) to an IBM Web server that could redirect the user's Web browser to the specific Web page that he or she was looking for. Also, the research and development team developed a technology to dynamically resolve application executable file names and directory paths so that continuous maintenance was avoided. To reduce engineering workload and to meet the goal of making development costs “zero sum” over time, internally developed software for the main Access ThinkPad screen and the setup wizards used a common architecture across all systems and operating systems. Thus, with careful architectural planning, the IOIS was developed to avoid national-language, operating-system, and Web-browser peculiarities, using existing staff with minimal retraining.

Iterative design and test

During development, the UCD team iteratively tested and benchmarked the IOIS. First, the design team scored the existing ThinkPad documentation and compared that score to the proposed IOIS concept using a heuristic (checklist) review. Then, they conducted design walk-throughs,⁶ which were focus group sessions to validate the overall IOIS concept with users, to identify issues with the concept, and to test various high-level design concepts with users. Additionally, formal usability tests⁸ were conducted with individual users using prototype-level systems to validate the implementation, to identify and prioritize user problems when using the system, and to compare user performance between the IBM prototype and some competitors' help systems.

Heuristic benchmarking. Prior to developing the IOIS, a conceptual design and detailed contents listing was benchmarked against the existing ThinkPad documentation in order to predict marketplace reaction to the IOIS and customer satisfaction. The IOIS team found a PC product benchmarking tool for ease of use, called the PC Ease of Use Scoring Script.^9,10 While this heuristic scoring script was designed to measure all aspects of consumer PC usability, it had detailed sections that could be used to rate print documentation, on-line (i.e., on-screen)¹¹ documentation, and troubleshooting information. Figure 4 depicts the result of this analysis. Overall, the IOIS heuristic benchmark showed improvement over the pre-IOIS information set that was more print-centric. However, the specific score for the resulting print documentation in the IOIS case was lower than the pre-IOIS information set. While the team saw this as a risk that some customers might not like the reduction of the print documentation, this was acceptable given that the cost reduction and development goals would be met. Thus the IOIS team recommended developing the IOIS, and they predicted that customer satisfaction would be maintained, or even improved. Again, this was an example of a UE approach where trade-off decisions and risks were recommended in order to meet al. objectives within the business reality of the product's market landscape. The team felt confident that the vast improvement in on-screen information would outweigh any dissatisfaction due to the reduction in print documentation.

Figure 4

Design walk-through research. In the early phase of development, the IOIS design team ran two design walk-through sessions with a total of 27 participants. Participants were briefed on the overall IOIS concept, were given a step-by-step walk-through using two alternative low-level prototype designs, and then asked a series of questions about the concept and prototypes.

Research questions. Of particular interest in this research were questions facing the IOIS design team. First, the team remained concerned that the drastic reduction in print documentation (from over 250 pages to 50 pages) would reduce customer satisfaction. While the score yielded by the usability checklist for the new IOIS concept was better than that of the previous set of on-screen and print documentation, actual user validation was needed before the team would be willing to recommend the IOIS as a cost-saving alternative to print documentation with a low risk of user disaffection. Second, there was little existing research on the optimal use of procedural animations in on-screen documentation, leading to disagreement among the design team regarding the right mixture and presentation of text and animations for a procedure. Finally, the team wanted additional guidance from users regarding the breakdown, or “chunking,” of information into information units and the possible complexity of the resulting navigation.

IOIS concept findings and redesign. Overall, mobile computer users were comfortable with the IOIS concept, whereby on-screen documentation serves as the comprehensive guide with search and print features, and the printed guide is deleted completely, or contains only information about the computer's hardware (e.g., servicing, upgrading, troubleshooting). In fact, 89 percent of the users indicated that they would be happy with either the comprehensive on-screen-only approach with no print documentation (33 percent), or a hardware-focused printed guide and a software-focused on-screen guide (56 percent).

On the other hand, in order to meet the cost reduction goals, the team had to eliminate some of the hardware-oriented information from the printed guide to keep it below the 80-page limit for low-cost printing and binding. Thus, the final IOIS concept was a comprehensive on-screen guide (describing hardware and software) and a duplicate copy in the printed guide of only the troubleshooting, repair, and service information required when the system was broken. For some hardware procedures (e.g., upgrading a memory module or a hard-disk drive) users would have to rely on the on-screen documentation and print capabilities of the IOIS. They would have to read the procedure on the screen, print it if necessary, and then turn the system off to perform the upgrade procedure. However, in cases where a procedure was animated, this caused concern that it might be too difficult for the user to complete the upgrade task, because the printout of the procedure would only have a single image of the animation, and thus would not illustrate all of the steps of the procedure.

To rectify this, the team added a new feature to the concept: a “print-friendly” view of any animated hardware procedure that included additional static graphics (usually the animation “key frames”) to illustrate each step as appropriate. This feature was only required for those tasks that required the system to be turned off.

Animation design and information chunking. Another point of contention in the IOIS team concerned the design of animated procedures. The two competing approaches shown to users in the walk-through exemplified the two extremes of the debate. The first design integrated the text of a task step with the animation so that as the animation “ran,” a bold text callout appeared briefly with the visual depiction of the step. As the animation progressed, each of the text callouts was shown at the appropriate time, and the display included a complete set of multimedia controls (play, pause, skip, etc.). This allowed the most space on the screen for large animations with strong context and clearly legible text. However, all of the steps could not be seen at once. The second design (see Figure 5) presented all of the steps at once, with a smaller animation inset about halfway through the procedure. A combination play/pause toggle button was displayed, but not a full set of animation controls, in order to save space. The text and animation seemed smaller and the screen more cluttered.

Figure 5

Users slightly favored the second alternative, where all of the text instructions were shown at once (58 percent) as compared to the first alternative where only a single step was displayed at a time (42 percent). When users' open-ended comments were analyzed, it was clear that users wanted to see all of the text instructions at once, but that some users were willing to trade that for larger text and less clutter, so long as there was a pause feature. Moreover, users tended to prefer the play/pause control and also a restart feature regardless of the text format. Because this was not an unequivocal result, the team tried to combine the features of both alternatives into a single design. In follow-on discussions, users favored a design that had all of the steps listed, with additional line spacing to reduce clutter, and that also included a separate play and pause control, as well as a control to restart the animation. This redesign was technically feasible and was selected as the final design for the IOIS. In addition, users wanted dynamic text highlighting as a cue to visually link each animation step to a text step, so that the coordination of text and animation could be more easily understood. Unfortunately, this request was not feasible within the IOIS because of difficulties with translation.

There was also disagreement among users about the appropriate chunking of information within procedures. Users were presented with two different styles. First, a “wizard” style procedure was presented, where each step was presented in a separate panel, linked together by back/next controls. In this case, all of the defined terms, related links, and annotations (e.g., warnings, tips, etc.) could be displayed with each step clearly. A second style showed an entire procedure, but defined terms were accessed via a pop-up window, related links were grouped at the bottom, and annotations were displayed below the associated step. The second style had more of a “book-like” appearance overall, but still took advantage of hyperlinking technologies. (Though it was not shown to users, the designers discussed a third style which was simply a book format put online, as is typical with online books in PDF format.)

Users tended to favor the second, “book style” (67 percent) as compared to the sequential “wizard style” (33 percent) for procedural information. It is important to note that the wizard style did not have the functional requirements familiar to users from setup or installation wizards (e.g., user input, branching, functional code), and thus the separation of steps seemed unnecessary. Moreover, some users were concerned that if they wanted to print a procedure, it would be very difficult and wasteful to print each step of the procedure separately. Thus the on-screen “book like” style was selected for the final design (see Figures 2 and 5).

Usability testing

After development was underway, a series of usability tests were conducted to identify implementation problems and to compare the IOIS with competitors' on-screen information systems, as part of the UCD verification process. The IOIS team ran two studies with a total of 10 participants.^12,13 Participants were given probe tasks (15 in the first study and 12 in the second study), in which they were asked to find the answer to a question that could be found in the IOIS. In addition, users were given nine focus tasks, in which they would likely be required to find a documented procedure in the IOIS.

In the first study, participants used only the IOIS, and they were unable to perform searches because that function had not yet been implemented. The second study introduced a slight change to the methodology. First, participants performed the same tasks on a competitor's system. While the IBM help topics were organized by tasks, the topics in the competitor's system were organized by feature. Second, participants performed a smaller number of probe tasks in blocks where they were progressively allowed to use more of the functionality. For the first four probe tasks, users were allowed only to use navigation from the main Access ThinkPad screen and were required to back out to the main screen if they had gone down the wrong path; this method allowed results to be compared to the first study. For the next four probe tasks, users were allowed to use the main screen navigation and embedded topic links to answer the questions. Finally, for the remaining four tasks, the users were given no constraints and were allowed to use all of the functionality, including searching. In both studies, participants were given five minutes to complete benchmark tasks with minimal help. Any comments and errors were recorded, and users completed a survey after completing the tasks.

Study results. Overall, the results of the first usability study demonstrated that the early IOIS prototype needed improvement. In the second study, the IOIS was compared to a competitor's on-screen system, and, because four of the probe tasks were identical to the first usability study, comparisons could be made to that study as well. On the whole, the IOIS version in the second usability test did not fare well. It took participants 50 percent longer to find information in the second-pass IOIS than it did for the competitor's system, and 70 percent longer when compared to the results of the first usability test for the four identical tasks.

One explanation was that the second-pass IOIS had roughly three times more content topics than either the competitor's system or the first-pass IOIS. Information retrieval times had to be improved drastically, and the usability team proposed many changes to address this problem.

Need for linking across topics. In the first study, the topics had a minimal number of links between pages. This meant that users navigating to an incorrect topic deep in the hierarchy would have to navigate back to the top of the hierarchy to find the correct topic located along a different node, making navigation very difficult. The team felt that once the search capability was implemented, satisfaction would increase. However, in the second study, even when users were unrestricted and could use searching, users still felt that the number of cross-referencing links among related topics needed to be increased.

The visual map. The first study showed that many users tried to click on text callouts in the graphics of ThinkPads in the IOIS, presumably in an attempt to link to information about the different ThinkPad components. Users were expecting not just a picture with labels, but rather a graphic way of linking to topics about that component. Although the graphics did not have this expected function, this result identified a user desire that was not satisfied in the IOIS design.

The benefit of being able to group topics by feature was confirmed by the second study. In this study, the intention was to compare task-oriented organization with component-oriented organization. For example, there was one topic that instructed users how to use the ThinkLight*, a feature that illuminated the keyboard for use in dark environments such as a dimly lit classroom or airplane cabin. With task-oriented organization, the user might expect to select a section called “Everyday Use” to find a topic such as “Typing in the Dark.” With component-oriented organization, a user might expect to click on a picture of the keyboard, and then select a topic called “keyboard light.” In the end, it was found that neither organization was better. Some tasks were easier to complete with component organization, especially when the task was clearly identified with a computer component (for example, enabling or disabling a device). Other tasks were easier to complete when they were organized by task. Upon closer examination, the users who had the most problems with task-oriented navigation were more expert; they could more readily find information in the competitor's help system by using technology and component names under such headings as “Ports and Connectors,” “Multimedia,” and “Keyboard and Mouse.”

The conclusion was that, rather than restricting users to a single navigation system, users should navigate to topics in whatever manner made sense to them. On the other hand, multiple views of the same topics were undesirable because this meant that users would not be able to learn a fixed path to navigate to a help topic. Instead, a navigation system was developed in which the organization was mainly organized by tasks, but also provided a visual representation of the ThinkPad computer that linked to all of the important topics about each component. This was called the visual map (see Figure 6). It provided a way to navigate to help topics by component and allowed users to click on components that they could not name.

Figure 6

Search strategies. In addition to finding that it was important to support multiple modes of navigation, the UCD team also found that users did not exclusively use one strategy to find a topic. Instead, they used a variety of strategies, including the keyword index, navigation, linking, and full-text search. The team learned about the help topics best suited to each method of retrieval and then developed strategies to improve each of them. The following improvements were made as a result of these findings.

Expanding keyword synonyms. The team noticed that many of the keywords that users entered were not present in the index. Many of the terms in the index had synonyms that were not included (e.g., hard disk drive instead of hard drive, hard disk, or storage). Thus the team developed a database of words that would be treated as equivalent synonyms to the words already in the index, enabling users to find information in the index by using terminology familiar to them.

Clarifying section titles. Users were confused by many of the section titles in the main Access ThinkPad screen (see Figure 1) and the section and topic titles in the left navigation panel in the underlying IOIS component (see the left side of Figure 6). The IOIS team then focused on improved terminology to clarify language, reduce jargon, and shorten titles as a result of these test findings.

Suggesting related topics and text links. In both studies, users expressed dissatisfaction with navigation among the 300–350 topics contained in the IOIS. This frustration was noted even in the second study after the search function was implemented. Users cited the lack of topic cross-referencing as a source of this frustration. As a result, the team developed a database to manage topics that should be linked. Overall, cross-referencing of related topics was increased by about 200 percent to address the users' concerns.

Quick search. One serendipitous finding of the second study actually convinced the IOIS team to de-emphasize the full-text search feature of the IOIS. From the main Access ThinkPad screen, the original design called for a full-text search field as a “shortcut” navigation strategy. However, due to a “bug” in the HTML help used to implement the underlying IOIS component, search terms could not be passed from the main screen to the search engine, but could only be passed to the keyword index. However, when users could use full-text search, they found that the number of “hits” was overwhelming at times. When the IOIS team improved the index by including many synonyms to fit the variety of user-preferred terminology, the keyword index was actually a more usable search facility. Thus, the search field on the main screen was renamed “quick search” (see Figure 1), and satisfaction with searching was improved.

Epilogue to usability testing

In all, the usability tests identified many issues, some of which precipitated major design changes. As is typical in the fast-paced development of consumer products, there were also usability items that could not be addressed. For example, the Access ThinkPad main screen (Figure 1) and the underlying IOIS component (for example in Figure 6) had considerable overlap in “left-panel” navigation. While this did not seem to impact user performance, many users were initially confused by this design, and it also caused some concern that there were “too many windows open.” However, this was the only architecture at that time that would permit the customized features such as the “Everyday Essentials” section, the customized and IBM-branded GUI (graphic user interface) design, and the intelligent Web integration.

Moreover, there was not enough time to conduct a regression test to verify that all of the changes made actually improved the users' performance and experience. However, in an early prototyping test for the next release of Access ThinkPad,¹³ the usability team got the chance to compare the final Access ThinkPad design in a partial replication. In this regression testing, average topic search times were reduced and were comparable to the competitors' help systems, even though the IOIS contained three times more information than those systems. Thus, the changes recommended as a result of the second usability test had the intended impact, validating the redesign effort.

Marketplace results. The usability testing and heuristic reviews conducted during product concept and development are useful as an iterative design tool. While these tests can predict the success of a UCD initiative, comparing benchmark metrics against competitors in the marketplace is essential in the use of UCD as an integrated part of a business. Consistent with a trend toward a UE model, the ThinkPad UCD team selected from among existing product and marketplace metrics to track the overall success of ThinkPad UCD: positive press reviews, customer satisfaction (from an existing survey), and cost effectiveness (defined as increased profitability due to UCD initiatives).

During the development of IOIS, ease of use measurements based on heuristic assessments were becoming increasingly common in PC industry press reviews. Comparing benchmark metrics against competitors in the marketplace is essential in the use of UCD as an integrated part of a business. Some of these heuristics were variations of the PC Ease of Use Scoring Script that the IOIS team used early in the concept validation time frame.

In 2000, when the PC Magazine roundup (comparing new notebook computers from various manufacturers) was published,¹⁴ two of the ThinkPads reviewed had implemented the IOIS, the ThinkPad A20 and T20. The 2000 roundup rated 31 computers on a variety of benchmark scores. Of particular interest to the IOIS team was the “ease of setup” score, reported as a five-point scale. Unfortunately, the A20 and T20 received low marks (a score of 1 out of 5) for ease of setup. Upon closer examination, the volume of print documentation was the major contributor to the ease of setup score. This risk was identified in the early benchmarking using the PC Magazine heuristic: the print-only documentation score was predicted to be lower than previous ThinkPads. This explanation was further supported by another ThinkPad rated in the roundup. The ThinkPad i-Series had a primarily print-based documentation format, earning it 3 out of 5—the highest score awarded—on ease of setup.

In the overall benchmark, both the ThinkPad A20 and T20 won the Editor's Choice award for their categories (“traveler” and desktop replacement). Moreover, the reviews of the A20 and T20 mentioned the IOIS (Access ThinkPad) as a positive feature that improved ease of use.

To provide further validation for the IOIS initiative, the IOIS team tracked comments in reviews regarding documentation. Documentation was not consistently mentioned in reviews, but when mentioned, comments about print documentation were mostly negative, and comments about on-screen online documentation were always very positive. Thus, the criteria for positive press reviews were not consistently met. Though all reviews that mentioned Access ThinkPad contained very positive comments, there were a number of qualitative and quantitative reviews that cited the lack of print documentation as a negative.

The metric that is most user-centric and quantitatively rigorous is the IBM PC UCD metric. This metric is derived from an existing annual customer satisfaction survey funded by marketing. Each year, computer users, administrators, purchasers, and resellers are recruited for a blind survey administered worldwide. Over 100 questions are administered via phone survey, with one scale specifically measuring satisfaction with documentation. Several others are also related to the impact of quality documentation (ease of use, setup, and installation). Data from the Fall 2000 survey were collected from September to December. Participants were sampled from lists of telephone subscribers generated from a random-digit dialing protocol, and they were paid for their participation. Users had to be the primary users of a portable computer, and only users with systems less than 2 years old were allowed to participate. Data were collected from customers of many different PC manufacturers, with data from IBM ThinkPad users shown in Table 1. It is interesting to note that modest but nonsignificant improvements between the predecessor and redesigned documentation were observed when a quantitative survey was conducted, even for ease of setup, contradicting the heuristic measurement provided by PC Magazine.

The IOIS strategy was specifically aimed at reducing print documentation costs without reducing customer satisfaction with documentation overall. While the printed User's Guide was not completely eliminated, its size was reduced by 80 percent, and thus the cost of documentation was reduced by approximately two dollars per system. This savings translated to a $2.5 million savings in 2000, and over $7 million in 2001. As can be seen in Table 1, customer satisfaction remained unchanged despite this change.¹⁵

Lessons learned from implementation. While the UCD team met many of their objectives for Access ThinkPad, they were not successful in achieving 100 percent of the user-experience and manufacturing-efficiency goals for this first release of the software. The team learned some valuable lessons, which were applied both to minor releases of Access ThinkPad (Versions 2.1, 2.5, and 2.8) as well as to the next major releases (Versions 3.0 and 4.0). Generally, these lessons revealed that a few of the original user-experience and manufacturing-efficiency goals were actually in conflict with one another. The team improved the usability of the product for the customer, but needed to improve the management and delivery aspects as well.

The user-experience goal required implementing dynamic links to related information—shown at the bottom of every help topic. Dynamic links help users searching for more detailed information about a specific topic or information about a related topic. This was one of the major findings of the Access ThinkPad 2.0 usability tests, resulting in a drastic increase in the number of related links coded into the system by the authors. While this worked well for users in meeting the goal of finding information quickly, related links also created dependencies among topics. When groups of topics were compiled together into modules, topics in one module were dynamically linked to topics in other modules. When a change occurred in a topic name or file name, a change also needed to occur inside any modules that referred to that topic as a related link. Such changes conflicted with the goal of easy modification in the reuse of topics and modules across similar systems, adversely affecting predicted manufacturing efficiency.

Translation issues also adversely affected the manufacturing-efficiency goal. Even though Access ThinkPad is essentially electronic documentation, it was treated as software when translated into several languages and tested, before modules were compiled and delivered to manufacturing. Ideally, any last-minute problem identified in translation testing could be quickly changed for a specific topic. However, any change to a topic required recompilation of that module, which took time. Also, difficulties arose when identifying which topic needed to be changed after the topic had been translated into a language not spoken by those making the change. Obviously, a text correction in a translated version would require a native speaker, but software code and hyperlinks embedded into topics also needed to be changed to fix functional bugs. For example, if a link inside a translated help topic needed to be fixed, and the topic was in Swedish, English-speaking developers with the required software and HTML skills had to identify the specific topic that needed to be changed before the change could be made. They could not make this identification because they could not decipher the topic title. This added to time efficiency problems as well. To address this, a change was made in subsequent versions of the IOIS to place a small identifier code at the bottom of each topic, making topic identification for translated files much easier for developers.

Another lesson that was learned concerned terminology. With multiple help topic authors in multiple locations, the editorial team found inconsistencies in terminology emerging. This was particularly problematic because the IOIS contained content including traditional user's guide content, information on software GUIs, marketing materials on the Web site, and service and support information on the Web site. For example, one author referred to the “Trackpoint* pointer,” another, the “TrackPoint device,” and another, the “pointing stick,” to identify the same item. In the final product, this was confusing for users. To better manage the development and use of terms, a terminology database was developed in subsequent releases of Access ThinkPad, for internal use. An editor managed and distributed this database. Authors across the many teams developing information were referred to this central, shared database for the use of terms, some of which had multiple synonyms. Consistent terminology meant fewer changes and more efficient development.

Access ThinkPad continues. The UCD team accomplished their mission of reducing print documentation costs without reducing customer satisfaction with documentation, but that was not the end. The team continued to improve subsequent versions of Access ThinkPad based on the lessons they had learned and a relentless focus on the user. Improvements were identified for the graphic design, navigation, and search capability. Changes were made to better support varying levels of mobile computer experience, from novice to expert. In addition, the development has continuously revised processes to improve development efficiency with this new documentation format.

The next major release of Access ThinkPad (Version 3.0) boasted many improvements based on unmet requirements and direct customer feedback from earlier releases. Access ThinkPad 3.0 had a flatter and more intuitive hierarchical structure based on card sort research.¹³ The visual map was improved with related links and direct launching of configuration utilities where appropriate. Moreover, the visual map was elevated to the main Access ThinkPad panel and doubled as an up-and-running system overview demo (see Figure 7).

Figure 7

Another victory for the IOIS team has been a new animation control that coordinates the steps in the text procedure with the related segment of the animation, a feature suggested by users in the very first IOIS design walk-through sessions. This was accomplished without impacting translation by making a numbered and highlighted animation control correspond with the numbered step below it. In the current release, now called Access IBM 4.0, a new Access IBM Message Center brings system and Web-based messages on relevant tools and support information to the user, an important but technologically challenging feature that had been omitted in earlier releases. A major redesign of the Access IBM 4.0 GUI includes a five-tabbed interface (with tabs labeled “Learn,” “Configure,” “Protect and Restore,” “Help and Support,” and “Stay Current”) to accommodate the wide range of customer experiences identified during new focus-group research conducted for Version 4.0.

In all, Access ThinkPad has been a highly successful UE effort rooted in business and competitive reality, but addressing the user through invention and iterative design. Access ThinkPad can be explored by pressing the ThinkPad button above the keyboard of any new IBM ThinkPad computer.

Acknowledgments

The authors would like to recognize the core IOIS design and development team, many of whom could have co-authored this paper—Chris Nevitt, Atsushi Kumaki, Dr. Stacey Baer, Stevenson Ang, Megumi Izumida, Jeff Ford, Phil Menzies, Rebecca Welles, Seiya Hayabo, Eiki Shibata, Tom Grimes, Adrian Farquharson, and Takanori Hoshiba. Additional thanks are given to the current Access IBM team for a thoughtful review and information about Access IBM 4.0 used in this paper.

Requests for reprints should be sent to David A. Sawin, 9000 S. Rita Road, Tucson, Arizona, 85744, USA

^*Trademark or registered trademark of International Business Machines Corporation.
^**Trademark or registered trademark of Sun Microsystems, Inc, Microsoft Corporation, or Netscape Communications Corporation.

Cited references and notes