3D CAPTURE

Visual and Geometric Computing

Three dimensional scanning has recently become a very active area in computer graphics. The requirements for computer graphics are different from those of traditional scanning applications. At IBM Watson Research we are developing scanning systems for producing virtual objects that can be rendered with high visual quality.

The capture of 3D representations of objects has been an area of research for decades in computer vision, robotics and metrology. Traditional uses for digital objects include industrial inspection, autonomous robot navigation and object recognition. Each of these areas has different requirements and different approaches for data acquisition and processing. For inspection applications, metric accuracy is critical, and scanners with great precision have been developed. Autonomous navigation applications, such as the use of robots in hazardous environments, need to be very robust and justify the use of expensive scanning equipment. Object recognition may use both the shape and surface properties, but only to the extent they are needed to compute a unique signature for the object. Unlike traditional applications, the end product of scanning for computer graphics is a model that can be used to render a realistic image of the object under novel conditions, i.e. in a location or under lighting conditions that exist only in computer simulation. The emphasis is on visual, rather than metric accuracy. A systematic error in the shape may be less important than errors in the color or apparent shininess of the object. This translates into a greater concern for estimating the spatial variation of the spectral bidirectional reflectance of a surface, and less concern for precision range measurements.

Applications for computer graphics rendering of scanned objects include virtual museums, e-commerce and games. In virtual museums, institutions can allow visitors to interact with virtual copies of objects that are physically too delicate to touch, or possibly even to keep on display. In e-commerce, a vendor may offer a potential consumer the capability to view an object in a variety of customized configurations. In games, scanned objects can be used to populate visually rich synthetic environments. These applications, while affecting a wide population, can only justify a modest expenditure on individual scanners, dictating the use of commodity components in their construction. Also, these applications require interactive display of the digital objects. This requirement dictates that the output of the scanning processing pipeline must be a form that can be rendered in real time.

At IBM Watson Research, we have focused on the development of scanning systems that acquire surface properties as well as shape, use commodity digital cameras, and produce texture-mapped triangle meshes that can be efficiently rendered by graphics hardware. Our first system was built around a commercial shape scanner that used projected light stripes and multi-baseline stereo. We enhanced the scanner with a novel photometric system that acquired surface normals and albedo at a spatial resolution 4 times the base geometry. We developed a processing pipeline for the acquired data that included new algorithms for meshing point clouds, for computing consistent normals from photometric data, and for the alignment and integration of surface texture maps. Our first major project using our scanning system was creating a digital model of Michelangelo's Florentine Pieta`. We are now exploring the use of alternative shape capture systems based on digital cameras. We are exploring the trade-offs between shape and texture resolution. We are also improving the processing pipeline to reduce user intervention to make scanning accessible to a wide range of consumers.

The goal of the 3D capture project is to produce numerical representations of existing physical objects that can be used to render high visual quality synthetic images of the objects in a computer graphics system. Such synthetic images allow objects to be viewed in places and in lighting environments that do not exist, or in which it is inconvenient to place the object. We seek to produce modifiable models, so that the numerical description of an existing object can also be used as a starting point for designing a new object. Given the numerical description of the object, we may edit the shape or change the surface color or finish.
 

Physical Object + Scanner	Numerical Object

 
The 3D capture problem involves many subproblems. The specific subproblems depend on choices made about  how the initial data is acquired. Here  we describe our approach under the general headings of Data Acquisition, Shape Integration, Texture Processing, and
Shape/Texture Integration.

Raw data  of many sorts that can be processed to give an object description. Touch probes can be used, or a variety of optical techniques, either passive or active. We have concentrated on methods that use controlled, active lighting, and commodity digital cameras. The image shows one of the systems that we have used that combines multi-baseline stereo and photometric stereo. We are continuing to explore variations and improvements of these type of system.

H. Rushmeier, F. Bernardini, J. Mittleman, G. Taubin. Acquiring input for rendering at appropriate level of detail: Digitizing a Pietà. Rendering Techniques '98 (Proceedings of the 9th Eurographics Workshop on Rendering June 29 - July 1, 1998, Vienna, Austria), pages 81-92, Springer-Verlag, 1998

 

After processing the acquired data, the shape is frequently defined by a cloud of points in 3D. While objects can be rendered from point descriptions alone, to facilitate editing and measurement operations we form an integrated triangle mesh to represent the surface. We developed the Ball-Pivoting algorithm to efficienty form such a mesh.

F. Bernardini, J. Mittleman, H. Rushmeier, C.  Silva, G. Taubin. The ball-pivoting algorithm for surface  reconstruction. IEEE Transactions on Visualization and Computer  Graphics 5(4), Oct.-Dec. 1999.

 

We process images captured at a higher resolution than the captured geometry to obtain the color and fine scale geometry of the object. We use variations of  Woodham's original photometric stereo algorithm to do this. A description of some of the considerations involved in this processing are described in:

H. Rushmeier and F. Bernardini, Computing Consistent Normals and Colors  from Photometric Data, Proceedings of the Second International Conference on 3-D Digital Imaging and Modeling , 1999.

This work is described in a presentation given at a course in SIGGRAPH 2000 (12 Mb pdf).

The textures and shape need to be combined into one consistent object that can be readily displayed in computer graphics systems. A major consideration is ensuring that fine details in the color of the object are not blurred when multiple data sets are combined. A method for integrating all the acquired and processed data is described in:

F. Bernardini, I. Martin, H. Rushmeier. High-Quality Texture Reconstruction. IBM  Research Rep. RC 21656, Feb. 2000. Accepted for publication in IEEE Transactions on Visualization and Computer  Graphics.

Read more

We have applied many elements of our current capture technology in creating a digital model of Michelangelo's Florentine Pietà, as describe on our Pietà project page.

We have prepared a survey of existing methods for acquiring  the shape and appearance of objects, which we will be happy to send you:

F. Bernardini, H. Rushmeier. The 3D Model Acquisition Pipeline,  Eurographics 2000, State of the Art Report (STAR). August 2000.