Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original...
of 34
/34
Embed Size (px)
Transcript of Edge-Adaptive Image Interpolation with Contour …optimization/L1/optseminar/Getreuer...Original...
Edge-Adaptive Image Interpolation with
Contour Stencils
Pascal Getreuer
Dec 27, 2010
TV along Curves
Let u be an image. For C a smooth simple curve, define
‖u‖TV(C) =
∫ T
0
∣∣ ∂∂t
u(γ(t)
)∣∣ dt, γ : [0,T ]→ C .
Strategy: Find approximate con-tours of u by finding curves C suchthat ‖u‖TV(C) is small.
γ′(t)
C
Contour Stencils
A contour stencil is a function S : Z2 × Z2 → R describingedges between pixels, and TV is estimated as
(S ? [u])(k) :=∑
m,n∈Z2
S(m, n) |uk+m − uk+n| ≈ ‖u‖TV(C)
where S describes edges that approximate C .+n1
+n2
(0, 0)(S ? [u])(i , j) =
(|ui,j−1−ui+1,j |
+ |ui−1,j−1 − ui,j | + |ui,j − ui+1,j+1|+ |ui−1,j−ui,j+1|
).
Contour Stencils
Estimate the contours locally by finding a stencil with low TV,
S?(k) = arg minS∈Σ
(S ? [u])(k)
where Σ is a set of candidate stencils.
Σ =
Contour Stencils
Input Contour Stencils Sobel
For each pixel, S?(k) is determined to estimate the localcontour orientation.
Why TV?
Total variation is invariant under diffeomorphisms on space.Consider the change of variables
t = ϕ(s)
dt = ϕ′(s) ds
u(t) u(ϕ(s)
)
and suppose that ϕ′(s) > 0, then∫ T
0
|u′(t)| dt =
∫ ϕ−1(T )
ϕ−1(0)
∣∣u′(ϕ(s))∣∣ϕ′(s) ds
=
∫ ϕ−1(T )
ϕ−1(0)
∣∣ ∂∂s
u(ϕ(s)
)∣∣ ds.
Interpolation Problem
Given discrete image v and point spread function h(x , y), findfunction u(x , y) such that
vi ,j = (h ∗ u)(i , j) for all i , j .
Input v Interpolation u
Edge Directed Interpolation
Theorem (Edge Directed Interpolation)
Consider approximating u(x) by
u(x) = (1− λ)u(a) + λu(b)a
b
x
C1
C2
and let C = C1 ∪ C2 be a curve passing through a, x, and b.Then the approximation error is bounded by
|u(x)− u(x)| ≤ max|1− λ| , |λ|
‖u‖TV(C) .
Choosing the stencil with the smallest TV minimizes theestimated interpolation error:
|u(x)− u(x)| ≤ ‖u‖TV(C) ≈1|S?|(S
??[u])(k) = minS∈Σ
1S (S?[u])(k).
Contour Stencil Windowed Zooming
Local reconstructions:
uk(x) = vk +∑n∈N
cn ϕnS?(k)(x − n), θ
k +N
k
if S =
k +N
k
if S =
where
vk : kth pixel of input image
N : neighborhood
ϕnS?(k) : function oriented with the
best-fitting stencil S?(k)
cn : coefficients such that(h ∗ uk)(m) = vk+m for m ∈ N
Contour Stencil Windowed Zooming
Combine local reconstructions with overlapping windows
u(x) =∑k∈Z2
w(x − k)uk(x − k),
where w satisfies∑k w(x − k) ≡ 1 s.t. method reproduces constants
w(k) = 0 for k 6∈ N s.t. ↓(h ∗ u) ≈ v
w compact support for computational efficiency
Example: Cubic B-spline
w(x) = B(x1)B(x2)
B(t) =(1− |t|+ 1
6|t|3 − 1
3
∣∣1− |t|∣∣3)+
Original Image (332×300) Input Image (83×75)
Estimated Orientations Proposed Interpolation (PSNR 25.97, 0.125 s)
Cubic B-spline (PSNR 25.92, 0.011 s) Fourier (PSNR 25.34, 0.062 s)
TV Minimization (PSNR 25.73, 0.784 s) Proposed Interpolation (PSNR 25.97, 0.125 s)
AQua-2 (PSNR 24.72, 0.016 s) Fractal Zooming (PSNR 24.65)
Roussos (PSNR 25.87, 2.518 s) Proposed Interpolation (PSNR 25.97, 0.125 s)
Zooming Comparison
Average PSNR on the Kodak Image Suite
Zoom Factor 2× 3× 4×AQua-2 25.06 22.48 21.35Fractal Zooming 29.00 27.20 25.25Contour Stencils 29.87 27.77 25.93Roussos 30.56 27.97 26.19
Computation Time (s) vs. Output Image Size
Image Size 128× 128 256× 256 512× 512AQua-2 0.0048 0.017 0.068Contour Stencils 0.025 0.088 0.34Roussos 0.23 2.22 8.64
Analysis of Contour Stencils
CC
C ≈ C =⇒ ‖u‖TV(C) ≈ ‖u‖TV(C)
Curve Perturbation
Let C and C be smooth curves parameterized byγ : [0,T ]→ C and γ : [0,T ]→ C . Then if u is twicecontinuously differentiable,∣∣‖u‖TV(C) − ‖u‖TV(C)
∣∣≤∥∥|∇u|
∥∥∞
∥∥|γ′ − γ′|∥∥1
+ ‖∇2u‖∞( |C |+|C |
2+ 1
4
∥∥|γ′ − γ′|∥∥1
)∥∥|γ − γ|∥∥∞.
Analysis of Contour Stencils
u ∈ C 2 =⇒ discrete TV is first-order accurate
TV Discretization
Suppose u ∈ C 2[0,T ] and 0 = t0 < t1 < · · · < tN = T , anddefine hi = ti − ti−1. Then
‖u‖TV−13T h2
max
havg‖u′′‖∞ ≤
N∑i=1
|u(ti)− u(ti−1)| ≤ ‖u‖TV .
Analysis of Contour Stencils
Let S?2(k) denote the second best-fitting stencil and definethe separation sepv (k) between the first and second best,
S?2(k) := arg minS∈Σ\S?(k)
(S ? [v ])(k),
sepv (k) :=(S?2(k) ? [v ]
)(k)−
(S?(k) ? [v ]
)(k).
Stability of the Best-Fitting Stencil
Suppose that for two images v and v
sepv (k) > 2M ‖v − v‖2 ,
M = maxS∈Σ
[∑m∈Z
(∑n∈Z
∣∣S(m, n) + S(n,m)∣∣)2]1/2
.
Then they have the same best-fitting stencil at k .
Contour Stencil Design
Let f 1, . . . , f J, f j : Z2 → R, be a set of image features.
Example: f j(x) = x1 sin π8j − x2 cos π
8j , j = 0, . . . , 7
We want to design stencils S1, . . . ,SJ that distinguishbetween these features.
Contour Stencil Design
We want stencil S j to be the best-fitting stencil on f j ,
Want: 1|S j |(S
j ? [f j ])(0) < 1|S i |(S
i ? [f j ])(0) for all i 6= j .
Ignoring the 1|S| normalizations, this condition becomes
(S j ? [f j ])(0) < (S i ? [f j ])(0)
=⇒((S j − S i) ? [f j ]
)(0) < 0.
We can try to satisfy this condition by minimizing
minS1,...,SJ
J∑i=1
J∑j=1
((S j − S i) ? [f j ]
)(0) + γ
J∑j=1
‖S j‖1
s.t. 0 ≤ S j(m, n) ≤ 1
Contour Stencil Design
minS1,...,SJ
J∑i=1
J∑j=1
((S j − S i) ? [f j ]
)(0) + γ
J∑j=1
‖S j‖1
s.t. 0 ≤ S j(m, n) ≤ 1
The minimization has closed-form solution
S j(m, n) =
1 if |f j
m − f jn | < 1
J
∑Ji=1|f i
m − f in | −
γJ,
0 if |f jm − f j
n | > 1J
∑Ji=1|f i
m − f in | −
γJ.
Corner-Shaped Stencils
Example: Corner-shaped stencils designed from the features
f j(x) = maxx1 cos π4j − x2 sin π
4j ,
x1 sin π4j + x2 cos π
4j.
3D Stencils
In d dimensions, a stencil S : Zd × Zd → R is applied at voxelk ∈ Zd as
(S ? [u])(k) :=∑
m,n∈Zd
S(m, n) |uk+m − uk+n| .
Small TV detects isosurfaces. Some 3D stencils:
3D Stencils
Example: Stencils applied to ui ,j ,k =√
i2 + j2 + k2
The results are visualized by assigning a color to the region ofspace having a particular best-fitting stencil.
3D Stencils
Example: Stencils applied to an MRI brain volume
Demosaicing
Original Mosaiced Demosaiced
Bayer Grid
Demosaicing Stencils
Centered on a green pixel:
Axially-oriented Diagonally-oriented π8
-oriented
Centered on a red or blue pixel:
Axially-oriented Diagonally-oriented π8
-oriented
Demosaicing Stencils
Stencil orientation estimation on mosaiced data:
Demosaicing Stencils
Stencil orientation estimation on mosaiced data:
Preliminary Demosaicing Method
Let f be the given mosaiced image. We consider demosaicingby the minimization of
arg minu
∑k∈Y ,Cb,Cr
∑n∈Ω
∑m∈N (n)
wm,n
∣∣u(k)m − u(k)
n
∣∣+
λ
2
∑k∈R,G ,B
∑n∈Ω(k)
(fn − u(k)n )2
where
wm,n : weights choosen according the best-fitting stencils
N (n) : neighbors of pixel n
λ : fidelity parameter
Ω(k) : subset of Ω where kth channel is given
Exact
Demosaiced
Exact
Demosaiced
Exact
Demosaiced
Exact
Bilinear
Malvar et al.
Hamilton-Adams
Gunturk et al.
Li
Zhang-Wu
Proposed
Exact
Bilinear
Malvar et al.
Hamilton-Adams
Gunturk et al.
Li
Zhang-Wu
Proposed
Exact
Bilinear
Malvar et al.
Hamilton-Adams
Gunturk et al.
Li
Zhang-Wu
Proposed
