Audio-Visual Speech Recognition

This project explores the use of visual information in speech recognition systems.

Although significant progress has been made in machine transcription of large vocabulary continuous speech (LVCSR) over the last few years, the technology to date is most effective only under controlled conditions such as low noise, speaker dependent recognition and read speech (as opposed to conversational speech etc.).

The potential for joint audio-visual-based speech recognition is well established on the basis of psychophysical experiments {Summerfeld79}.  Canonical mouth shapes that accompany speech utterances have been categorized, and are known as visual phonemes or ``visemes'' {Stork96}.  Visemes provide information that complements the phonetic stream from the point of view of confusability. For example, ``mi'' and ``ni'' which are confusable acoustically, especially in noise situations, are easy to distinguish visually: in ``mi'' lips close at onset, where as in ``ni'' they do not. The unvoiced fricatives ``f'' and ``s'' which are difficult to recognize acoustically belong to two different viseme groups {Stork96}.

Efforts to use visual information for automatic speech recognition have begun recently on experiments with small vocabulary letter or digit recognition tasks (see e.g., {Bregler94,Potamianos98,Stiefelhagen97}). However, most of these efforts have been limited to small vocabulary (e.g., command, digits) and often to speaker dependent training or isolated word speech where word boundaries are artificially well defined.

In this project, we are investigating the problem of combining visual cues with audio signals for the purpose of improved automatic machine recognition of large-vocabulary, continous speech using sub-phonetic statistical models.
We are interested in demonstrating meaningful improvements for realistic tasks such as broadcast news transcription, large vocabulary dictation and speech reading for the hearing/speech impaired.

Key Research Areas:

Publications List