Specular Reflection Reduction with Multi-Flash Imaging | POL S 1, Papers of Political Science

Download Specular Reflection Reduction with Multi-Flash Imaging | POL S 1 and more Papers Political Science in PDF only on Docsity! Specular Reflection Reduction with Multi-Flash Imaging ROGERIO FERIS1, RAMESH RASKAR2 , KARHAN TAN2, MATTHEW TURK1 1UCSB–University of California, Santa Barbara, CA 93106, USA {rferis,mturk}@cs.ucsb.edu 2MERL–Mitsubishi Electric Research Labs, Cambridge, MA 02139, USA {raskar,tan}@merl.com Abstract. We present a novel method to reduce the effect of specularities in digital images. Our approach relies on a simple modification of the capture setup: a multi-flash camera is used to take multiple pictures of the scene, each one with a differently positioned light source. We then formulate the problem of specular highlights reduction as solving a Poisson equation on a gradient field obtained from the input images. Experimental results are demonstrated on real and synthetic images. The entire setup can be conceivably packaged into a self-contained device, no larger than existing digital cameras. 1 Introduction The reflection of light from surfaces in real scenes is gen- erally classified into two main categories: diffuse and spec- ular. The diffuse component results from light rays pen- etrating the surface, undergoing multiple reflections, and re-emerging [8]. In contrast, the specular component is a surface phenomenon - light rays incident on the surface are reflected such that the angle of reflection equals the angle of incidence. Light energy due to specular reflections is often concentrated in a compact lobe, causing strong highlights (bright regions) to appear in the image. These bright spots, also known as specularities, play a major role in many computer graphics and vision prob- lems. They provide a true sense of realism in the envi- ronment, reveal important local curvature information [9] and may even provide additional cues for object recogni- tion [10]. However, in most cases, specular highlights are undesirable in images. They are often considered as an- noyance in traditional photography and cause vision algo- rithms for segmentation and shading analysis to produce er- roneous results. If the sensor direction is varied, highlights shift, diminish rapidly, or suddenly appear in other parts of the scene. This poses a serious problem for vision meth- ods that rely on image correspondences, such as stereo or motion algorithms. A variety of photometric techniques have been pro- posed for detecting and removing specularities using color [6], polarization [15], multiple views [7] and hybrid meth- ods [8]. However, most of these techniques assume that highlights appear in regions with no variation of material type or surface normal. In fact, reliably removing specular- ities in textured regions remains a challenging problem. In this paper, we address the problem of reducing the effect of specularities in digital images by taking a different approach. Rather than relying in one single image as most previous methods, we capture a set of images of the scene with different lighting conditions. Our work belongs to an emerging class of computer graphics techniques that pro- cess multiple images acquired with the same viewpoint but under different conditions, such as different illumination, focus or exposure [2, 1, 12]. Our capture setup is a multi-flash camera, a self-con- tained device that we have recently proposed for depth edge extraction and non-photorealistic rendering [12]. By ob- serving that specularities shift according to the different po- sitions of the flashes, we are able to significantly reduce or completely remove specular highlights in the scene. More specifically, we formulate this problem as solving a Poisson equation on a gradient field obtained from the input images. This allows us to remove specularities in objects with dif- ferent curvatures and textured regions. We will detail our approach in Section 2 and present our experimental results in Section 3. In Section 4, we dis- cuss advantages and limitations of our method. In Section 5, we describe implementation details and conclude the pa- per in Section 6, with final remarks and future work. 2 Specular Highlights Reduction In this section we introduce our proposed method for spec- ular highlights reduction. We start describing our capture setup and then present our gradient domain algorithm. 2.1 Multi-Flash Imaging Instead of taking one single picture of the scene, we use a multi-flash camera with n flashes to acquire n images from the same viewpoint, each one with a differently positioned flash. We have already successfully used this setup for de- Figure 1: Multi-flash camera setups used in [12] for depth edge detection. From left to right: 4-flash setup, 8-flash setup and dynamic scenes setup. Our goal is to exploit multi-flash imaging for specular reflection reduction. tection of depth edges, with applications in stylized render- ing [12], fingerspelling recognition [4] and medical imag- ing [13]. Figure 1 shows our different multi-flash setups for static and dynamic scenes. Note that the position of each flash in these prototypes was chosen to better detect depth edges. We refer to [12] for details about our depth edge de- tection method based on shadows. Now we will show how multi-flash cameras can be used for reliable removal and reduction of specularities. 2.2 Approach Our method is based on the observation that specular spots shift according to the shifting of light sources that created them. We need to consider three cases of how specular spots in different light positions appear in each image: (i) shiny spots remain distinct on a highly specular surface. (ii) some spots overlap. (iii) spots overlap completely (no shift). We show that for cases (i) and (ii), which often oc- cur in practice, our method successfully removes specular highlights. We note that although specularities overlap in the input images, the boundaries (intensity edges) around speculari- ties in general do not overlap. The main idea is to exploit the gradient variation in the n images, taken under the n dif- ferent lighting conditions, at a given pixel location (x,y). If (x,y) is in a specular region, in cases (i) and (ii), the gradient due to the specularity boundary will be high in only one or a minority of the n images. Taking the median of the n gra- dients at that pixel will remove this outlier(s). Our method is motivated by the intrinsic image approach [14], where the author removes shadows in outdoor scenes by noting that shadow boundaries are not static. Let Ik, 1 ≤ k ≤ n be an input image taken with light source k. We reconstruct the specular-reduced image by using median of gradients of input images as follows: • Compute intensity gradient, Gk(x, y) = ∇Ik (x, y) • Find median of gradients, G(x,y) = mediank(Gk(x, y)) • Reconstruct image Î which minimizes ∣∣∣∇Î − G ∣∣∣ This algorithm is illustrated in Figure 2, considering a camera with four flashes. The method used for reconstruct- ing image Î will be described in Section 2.3. Figure 3 shows a simple example to illustrate our me- thod in all three cases mentioned above. For each case, we created four images with manually drawn specularities. The first column in the figure corresponds to the max composite of the four images (Imax), the second corresponds to the median of intensities (Imedian) and the third column is the output of our method - the reconstruction from the median of gradients (Iintrinsic). Note that if we consider Imedian, specularities are not eliminated in case (ii), where spots overlap. On the other hand, our method is able to handle cases (i) and (ii), which often occur in practice. If specularities do not move among images, our method fails to remove them. It is worth mentioning that specularities could also be removed by just taking the min composite of the images, but in this case we would have the presence of shadows, which are not desirable. Since the boundaries of shadows rarely overlap, they are also eliminated in our method. 2.3 Image Reconstruction from Gradient Fields Image reconstruction from gradients fields, an approximate invertibility problem, is still a very active research area. In R2, a modified gradient vector field G may not be in- tegrable. In other words, there might not exist an image Î such that G = ∇Î . In fact, the gradient of a poten- Figure 5: (a-d) Four images taken with our multi-flash camera. (e) Max composite image. (f) Specular-reduced image using our method. (g) Magnitudes of gradients along the scanline showed in (e). Dashed lines correspond to gradients of different images and the solid line is the median of gradient magnitudes. (h) Intensity of max composite along the scanline (dashed line) and intensity of reconstructed specular-reduced image (solid line). Figure 6: More demonstrations of our method. Left: image taken with one of the flashes. Right: our specular-reduced image. sources, we formulate the problem of specularity removal as solving a Poisson equation on a gradient field obtained from the input images. We basically improved and presented another func- tionality for the multi-flash camera, recently proposed for depth edge detection and stylized rendering [12]. As future work, we plan to exploit other imaging parameters, such as variable wavelength (coloured flashes) to improve our tech- niques. Figure 7: Comparison of depth edge detection without spec- ular reflection reduction (left) and using our method (right). References [1] D. Akers, F. Losasso, J. Klingner, M. Agrawala, J. Rick, and P. Hanrahan. Conveying Shape and Fea- tures with Image-Based Relighting. In IEEE Visual- ization, 2003. [2] M. Cohen, A. Colburn, and S. Drucker. Image stacks. Technical Report MSR-TR-2003-40, Microsoft Re- search, 2003. [3] R. Fattal, D. Lischinski, and M. Werman. Gradient Domain High Dynamic Range Compression. In Pro- ceedings of SIGGRAPH 2002, pages 249–256. ACM SIGGRAPH, 2002. [4] R. Feris, M. Turk, R. Raskar, K. Tan, and G. Ohashi. Exploiting Depth Discontinuities for Vision-based Fingerspelling Recognition. In IEEE Workshop on Real-time Vision for Human-Computer Interaction (in conjunction with CVPR’04), Washington DC, USA, 2004. [5] R. Frankot and R. Chellappa. A method for enforcing integrability in shape from shading algorithms. IEEE Transactions on Pattern Analysis and Machine Intel- ligence, 10(4):439–451, 1988. [6] G. Klinker, S. Shafer, and T. Kanade. The measure- ment of highlights in color images. International Journal of Computer Vision, 2:7–32, 1988. [7] S. Lee. Understanding of Surface Reflections in Com- puter Vision by Color and Multiple Views. PhD thesis, University of Pennsylvania, 1991. [8] S. Nayar, X. Fang, and T. Boult. Removal of specu- larities using color and polarization. In International Conference on Computer Vision and Pattern Recogni- tion, pages 583–590, New York City, USA, 1993. [9] M. Oren and S. Nayar. A theory of specular surface geometry. International Journal of Computer Vision, 2:105–124, 1997. [10] M. Osadchy, D. Jacobs, and R. Ramamoorthi. Using specularities for recognition. In International Confer- ence on Computer Vision, Nice, France, 2003. [11] W. Press, S. Teukolsky, W. Vetterling, and B. Flan- nery. Numerical Recipes in C: The Art of Scientific Computing . Pearson Education, 1992. [12] R. Raskar, K. Tan, R. Feris, J. Yu, and M. Turk. A non-photorealistic camera: depth edge detection and stylized rendering using multi-flash imaging. SIG- GRAPH’04 / ACM Transactions on Graphics (to ap- pear), 2004. [13] K. Tan, J. Kobler, P. Dietz, R. Feris, and R. Raskar. Shape-Enhanced Surgical Visualizations and Medical Illustrations with Multi-Flash Imaging. In Conference on Medical Image Computing and Computer Assisted Intervention, France, 2004. [14] Y. Weiss. Deriving intrinsic images from image se- quences. In Proceedings of International Conference on Computer Vision, pages 68–75, 2001. [15] L. Wolff and T. Boult. Constraining object features using a polarization reflectance model. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 16(6):635–657, 1991.