[ IBM Research ]

Visual Analysis Home
Digital Image Libraries
Semantic Modeling of Images

As multimedia libraries grow in size and complexity, it becomes increasingly important to have good search and navigation tools. Organizing the contents semantically, according to meaningful categories, is an emerging method for achieving this goal. Most approaches to semantic organization are based on text descriptors. In this work, we provide a method for semantic categorization and retrieval of photographic images based on low-level image descriptors derived from perceptual experiments. Using the domain of photographic images as a starting point, our objective was to:

  • discover the most important semantic categories in human similarity perception
  • provide the relationships among these categories
  • find out how much of the semantic information contained in each category can be modeled with calculable image features
  • devise the image-processing model that captures image semantics and simulates human behavior in categorizing and matching images

Our work had three major parts:

  • experiments
  • modeling
  • implementation and testing

In the first part we conducted several subjective experiments aimed at:

  • developing and refining a set of perceptual categories in the domain of photographic images
  • deriving a semantic name for each perceptual category
  • discovering a combination of low-level features which best describe each category
"semantic categories of images"
In the modeling part, we designed an image similarity metric that embodies our findings, to annotate images or to search the database, using the semantic concepts. We have implemented a prototype annotation/retrieval system, and tested it against the judgments of human observers. Our results provide a good match to human performance, thus validating the use of human judgments to develop semantic descriptors. Our method can be used for the enhancement of current image/video retrieval methods, better organization of large image/video databases, and the development of more intuitive navigation schemes, browsing methods and user interfaces.
Image categorization Image categorization
ISee
ISee example One of our research interests is developing efficient and meaningful image features, indexing, annotation and content summarization schemes, and using them for the intelligent search, retrieval and browsing of web documents. Our semantic image classification methodology is currently implemented in the Image Search and Exploration Engine (ISee). ISee is an Internet portal, which incorporates a web image robot, image indexing scheme and a web browser that uses the image metadata to perform search and browse the Internet using the visual attributes.