Interactive Matting
description
Transcript of Interactive Matting
Interactive MattingChristoph Rhemann
Supervised by:Margrit Gelautz and Carsten Rother
Matting and compositing
Matting and compositing
Outline
Talk Outline:
• Introduction & previous approaches
• Our matting model
• Evaluation strategy
= +
Cr,g,b = α Fr,g,b + (1 - α) Br,g,b
● ●
● ●
Inverse process of compositing:
Determine: F, B, αGiven:C
Matting is ill posed
= +
Underconstrained problem:7 Unknowns in only 3 Equations
● ●
Cr = α Fr + (1 - α) Br
Cg = α Fg + (1 - α) Bg
Cb = α Fb + (1 - α) Bb
Cr,g,b = α Fr,g,b + (1 - α) Br,g,b● ●
Matting is ill posed
Trimap
Scribbles
Background
Background
Unknown
Foreground
Unknown
Foreground
User interaction
Previous approaches
C = α F + (1 – α) B● ●Recall compositing equation:
Previous approaches
C = α F + (1 – α) B● ●Recall compositing equation:
Closed Form Matting [Levin et al. 06]
R
B
G
Previous approaches
C = α F + (1 – α) B● ●Recall compositing equation:
R
B
G
Closed Form Matting [Levin et al. 06]Assumption: F and B colors in a local window lie on color line
Previous approaches
C = α F + (1 – α) B● ●Recall compositing equation:
R
B
G
Closed Form Matting [Levin et al. 06]Assumption: F and B colors in a local window lie on color line Analytically eliminate F,B. Alpha can be solved in closed form
Result of [Levin et al 06]True Solution Input image + Trimap
Result of Closed Form Matting [Levin et al. 06]:• Result imperfect: Hairs cut off• Problem: Cost function has large solution space
Previous approaches
What are the reasons for pixels to be transparent?
Segmentation – based matting
Defocus Blur
Lens Camera sensor
Point spread
function
Point Spread Function
Focal plane
Lens’ aperture
Lens and defocus
Slides by Anat Levin
LensObject Camera sensor
Point spread
function
Lens’ aperture
Focal planeSlides by Anat Levin
Lens and defocus
Point Spread Function
What are the reasons for pixels to be transparent?
Segmentation – based matting
Defocus Blur Motion Blur
PSF forMotion Blur
What are the reasons for pixels to be transparent?
Segmentation – based matting
Defocus Blur Motion Blur
Discretization
What are the reasons for pixels to be transparent?
Observation: Apart from translucency mixed pixels are caused by camera’s Point Spread Function (PSF)
Segmentation – based matting
Defocus Blur Motion Blur
Discretization Translucency
Basic idea:Model alpha as convolution of a binary segmentation with PSF
Approach taken [Rhemann et al. 08]:Use this model as prior in framework of [Levin et al. 06]
Model for alpha
Binary segmentation PSF Observed alphaInput image + Trimap
Matting process
Initial alpha using [Wang et al. ´07]
(Result is imperfect)
Initialize PSF/deblur alpha
Deblured (sparse) alpha
Binarized (sparse) alpha using gradient
preserving MRF prior
Iterate a few times
Input image
Matting process
Binarized (sparse) alpha using gradient
preserving MRF prior
Segmentation prior
Final alpha
Ground truth
Result for [Levin et al. ’06]
Input image
Input image + trimap
Comparison
Result of [Wang et al. ’07]
Input image
Input image + trimap
Comparison
Input image
Input image + trimap
Result of [Rhemann et al. ’08]
Comparison
Input image + trimap [Levin et al. ’06]
[Wang et al. ’07] [Rhemann et al. ’08] Ground truth alpha
[Levin et al. ’07]
Comparison – Close up
Evaluation of matting algorithms
How to compare performance of algorithms?
Showing some qualitative results
OR
Quantitative evaluation using reference solutions
Evaluation of matting algorithms
• Key Factors for a good quantitative evaluation
• Ground truth dataset
• Online evaluation
• Perceptual error functions
• 35 natural images• High resolution• High quality
Triangulation Matting [Smith, Blinn 96]- Photograph object against 2 different backgrounds True solution to matting problem
Input image Ground truth Zoom in
Ground truth dataset
Data and evaluation scripts online
Advantages:• Investigate results• Upload novel results
www.alphamatting.com
Online evaluation
Motivation:Simple metrics not always correlated with visual quality
Input image Zoom in Result 1SAD: 1215
Result 2SAD: 806
Perceptually motivated error functions
Develop error measures for two properties:• Connectivity of foreground object• Gradient of the alpha matte
Perceptually motivated error functions
Input image Zoom in Result 1SAD: 312
Result 2SAD: 83
User Study:• Goal: Infer visual quality of image compositions• Task: Rank to according to how realistic they appear
Perceptually motivated error functions
Gradient artifacts Connectivity artifacts
Correlation of error measures to average user ranking
Gradient data Connectivity data0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8Grad.
Grad.
Conn.
Conn.
GradientConnectivitySADMSE
Correlation
Perceptually motivated error functions
• Model for alpha overcomes ambiguities
• Model-based algorithm: Performs better than competitors
• Perceptual motivated evaluation
• Message to you: Evaluation of your algorithm is important• Use ground truth data to make quantitative comparisons• Use a large dataset• Use a training / test split
Conclusions
Previous approaches
C = α F + (1 – α) B● ●Recall compositing equation:
R
B
G
Model of F
Model of B
Observed color
Data driven approaches (e.g. [Wang et al. 07])• Model color distribution of F and B (from the user defined trimap)• Observed color more likely under F or B model?• Use likelihood in framework of [Levin et al 06]
Result of data driven approaches [Wang et al. 07]:• Hair is better captured• Many artifacts in the background
Previous approaches
Result of [Levin et al 06]True Solution Input image + Trimap Result of [Wang et al 07]