China Research Laboratory

Multi-channel Programming Model and Multi-modal Interactive Middleware

In the emerging multi-channel era, multi-modal e-business is characterized by multiple interaction modes, for example voice + WAP, PC + tablet, PDA + phone, and so on, and the flexibility to adapt the interaction mode to the task and the situation - stationary or mobile, hands-on or hands-free, visual or eyes-free, and audible or quiet.

Multi-modal interaction middleware allows access to multiple output modalities. These multiple views can be used collectively to enhance user experience, or independently to select the optimal mode of interaction based on context, environment and user preferences. When used collectively, all views are synchronized; this enables users to switch from one mode to the other at any point during their interaction. When used independently or collectively, application state is saved; this enables users to suspend then resume their interaction where they had left of.

Multi-modal Mobile Banking enables user-centric multi-modal banking service, with several advanced IBM products, WebSphere EveryPlace Suite (WES), WebSphere Business Component Composer(WSBCC), and a key technical component i.e. Interaction Manager(IM) from IBM CRL. Using this solution, mobile users can switch freely among different devices (e.g. desktop browser and mobile phone) and different interaction channels (voice, SMS and WAP of a mobile phone) within any banking transaction, while keeping data synchronization and session persistence. With this function, mobile user can get banking service without any concerns of connectivity change and interaction modality change in mobile environment.

Remote User Interaction:
Networked pervasive devices (such as PDAs) are becoming strategic points while providing end-to-end solutions to enterprise customers. It is one of the major tasks to extend existing enterprise applications to such kind of devices. However, it is difficult/impossible to rewrite/execute every legacy application for PDA due to its limited computing and storage resources. The Remote User Interaction project has developed two technologies. The first is the User Interface Mapping, which enables PvC devices to launch/interact with legacy applications sitting in traditional PC/Server. The second is User Input Mapping assisted by our handwriting recognition technology, which is employed to enhance user experience by mapping pen based input events to keyboard and mice based input events.

Blue Eyes:
A new Chinese input system-EASE(Eye-tracking Assisted Selection and Entry) is invented by CCR&HCI team based on gaze tracking technique. In this technology, we build natural cooperation of hand and eye, and remove the tiresome procedure on candidate selection procedure of Chinese input in order to make the Chinese character typing more natural and more efficient. This invention is based on blue eyes technology from IBM Almadon Research Center.

Multi-modal Information Capture:
With the proliferation of mobile devices, multi - modal information interaction are drawing more and more attention. We prototyped an application to enable the multi-modal information capturing. It captured the information from different channels such as voice, handwriting , keyboard input and eye input. These synchronized information can be cross indexed and retrieved. More research on multi-modal interaction is under exploration. One examples we are working is to implement this new interactive technology in remote meeting system.

Handwriting Recognition

Embedded Handwriting Recognition:
The IBM China Research Laboratory has developed embedded handwriting recognition, which includes these functions:

• Recognizing Simplified Chinese, Traditional Chinese, Alphabets and Numeral, supporting mixed-language recognition.
• Supporting a variety of operating system, Windows, Linux, Palm, etc.
• High recognition accuracy and easy to be ported to other platforms.

Continuous Handwriting Recognition:
IBM China Research Laboratory has been a leader in continuous handwriting recognition research. The advantage of continuous handwriting recognition is its ability to transform the handwriting ink on paper into text files. An application utilizing the continuous handwriting recognition technology has been developed to store, index, retrieve and recognize handwriting ink. Currently, the technology could support simplified Chinese, traditional Chinese, and Japanese. This technology has been implemented into several IBM products, including: IBM ThinkScribe and ThinkPad TransNote.