Edge-Adaptive Image Interpolation with

Contour Stencils

Pascal Getreuer

Dec 27, 2010

TV along Curves

Let u be an image. For C a smooth simple curve, define

‖u‖TV(C) =

∫ T

0

∣∣ ∂∂t

u(γ(t)

)∣∣ dt, γ : [0,T ]→ C .

Strategy: Find approximate con-tours of u by finding curves C suchthat ‖u‖TV(C) is small.

γ′(t)

C

Contour Stencils

A contour stencil is a function S : Z2 × Z2 → R describingedges between pixels, and TV is estimated as

(S ? [u])(k) :=∑

m,n∈Z2

S(m, n) |uk+m − uk+n| ≈ ‖u‖TV(C)

where S describes edges that approximate C .+n1

+n2

(0, 0)(S ? [u])(i , j) =

(|ui,j−1−ui+1,j |

+ |ui−1,j−1 − ui,j | + |ui,j − ui+1,j+1|+ |ui−1,j−ui,j+1|

).

Contour Stencils

Estimate the contours locally by finding a stencil with low TV,

S?(k) = arg minS∈Σ

(S ? [u])(k)

where Σ is a set of candidate stencils.

Σ =

Contour Stencils

Input Contour Stencils Sobel

For each pixel, S?(k) is determined to estimate the localcontour orientation.

Why TV?

Total variation is invariant under diffeomorphisms on space.Consider the change of variables

t = ϕ(s)

dt = ϕ′(s) ds

u(t) u(ϕ(s)

)

and suppose that ϕ′(s) > 0, then∫ T

0

|u′(t)| dt =

∫ ϕ−1(T )

ϕ−1(0)

∣∣u′(ϕ(s))∣∣ϕ′(s) ds

=

∫ ϕ−1(T )

ϕ−1(0)

∣∣ ∂∂s

u(ϕ(s)

)∣∣ ds.

Interpolation Problem

Given discrete image v and point spread function h(x , y), findfunction u(x , y) such that

vi ,j = (h ∗ u)(i , j) for all i , j .

Input v Interpolation u

Edge Directed Interpolation

Theorem (Edge Directed Interpolation)

Consider approximating u(x) by

u(x) = (1− λ)u(a) + λu(b)a

b

x

C1

C2

and let C = C1 ∪ C2 be a curve passing through a, x, and b.Then the approximation error is bounded by

|u(x)− u(x)| ≤ max|1− λ| , |λ|

‖u‖TV(C) .

Choosing the stencil with the smallest TV minimizes theestimated interpolation error:

|u(x)− u(x)| ≤ ‖u‖TV(C) ≈1|S?|(S

??[u])(k) = minS∈Σ

1S (S?[u])(k).

Contour Stencil Windowed Zooming

Local reconstructions:

uk(x) = vk +∑n∈N

cn ϕnS?(k)(x − n), θ

k +N

k

if S =

k +N

k

if S =

where

vk : kth pixel of input image

N : neighborhood

ϕnS?(k) : function oriented with the

best-fitting stencil S?(k)

cn : coefficients such that(h ∗ uk)(m) = vk+m for m ∈ N

Contour Stencil Windowed Zooming

Combine local reconstructions with overlapping windows

u(x) =∑k∈Z2

w(x − k)uk(x − k),

where w satisfies∑k w(x − k) ≡ 1 s.t. method reproduces constants

w(k) = 0 for k 6∈ N s.t. ↓(h ∗ u) ≈ v

w compact support for computational efficiency

Example: Cubic B-spline

w(x) = B(x1)B(x2)

B(t) =(1− |t|+ 1

6|t|3 − 1

3

∣∣1− |t|∣∣3)+

Original Image (332×300) Input Image (83×75)

Estimated Orientations Proposed Interpolation (PSNR 25.97, 0.125 s)

Cubic B-spline (PSNR 25.92, 0.011 s) Fourier (PSNR 25.34, 0.062 s)

TV Minimization (PSNR 25.73, 0.784 s) Proposed Interpolation (PSNR 25.97, 0.125 s)

AQua-2 (PSNR 24.72, 0.016 s) Fractal Zooming (PSNR 24.65)

Roussos (PSNR 25.87, 2.518 s) Proposed Interpolation (PSNR 25.97, 0.125 s)

Zooming Comparison

Average PSNR on the Kodak Image Suite

Zoom Factor 2× 3× 4×AQua-2 25.06 22.48 21.35Fractal Zooming 29.00 27.20 25.25Contour Stencils 29.87 27.77 25.93Roussos 30.56 27.97 26.19

Computation Time (s) vs. Output Image Size

Image Size 128× 128 256× 256 512× 512AQua-2 0.0048 0.017 0.068Contour Stencils 0.025 0.088 0.34Roussos 0.23 2.22 8.64

Analysis of Contour Stencils

CC

C ≈ C =⇒ ‖u‖TV(C) ≈ ‖u‖TV(C)

Curve Perturbation

Let C and C be smooth curves parameterized byγ : [0,T ]→ C and γ : [0,T ]→ C . Then if u is twicecontinuously differentiable,∣∣‖u‖TV(C) − ‖u‖TV(C)

∣∣≤∥∥|∇u|

∥∥∞

∥∥|γ′ − γ′|∥∥1

+ ‖∇2u‖∞( |C |+|C |

2+ 1

4

∥∥|γ′ − γ′|∥∥1

)∥∥|γ − γ|∥∥∞.

Analysis of Contour Stencils

u ∈ C 2 =⇒ discrete TV is first-order accurate

TV Discretization

Suppose u ∈ C 2[0,T ] and 0 = t0 < t1 < · · · < tN = T , anddefine hi = ti − ti−1. Then

‖u‖TV−13T h2

max

havg‖u′′‖∞ ≤

N∑i=1

|u(ti)− u(ti−1)| ≤ ‖u‖TV .

Analysis of Contour Stencils

Let S?2(k) denote the second best-fitting stencil and definethe separation sepv (k) between the first and second best,

S?2(k) := arg minS∈Σ\S?(k)

(S ? [v ])(k),

sepv (k) :=(S?2(k) ? [v ]

)(k)−

(S?(k) ? [v ]

)(k).

Stability of the Best-Fitting Stencil

Suppose that for two images v and v

sepv (k) > 2M ‖v − v‖2 ,

M = maxS∈Σ

[∑m∈Z

(∑n∈Z

∣∣S(m, n) + S(n,m)∣∣)2]1/2

.

Then they have the same best-fitting stencil at k .

Contour Stencil Design

Let f 1, . . . , f J, f j : Z2 → R, be a set of image features.

Example: f j(x) = x1 sin π8j − x2 cos π

8j , j = 0, . . . , 7

We want to design stencils S1, . . . ,SJ that distinguishbetween these features.

Contour Stencil Design

We want stencil S j to be the best-fitting stencil on f j ,

Want: 1|S j |(S

j ? [f j ])(0) < 1|S i |(S

i ? [f j ])(0) for all i 6= j .

Ignoring the 1|S| normalizations, this condition becomes

(S j ? [f j ])(0) < (S i ? [f j ])(0)

=⇒((S j − S i) ? [f j ]

)(0) < 0.

We can try to satisfy this condition by minimizing

minS1,...,SJ

J∑i=1

J∑j=1

((S j − S i) ? [f j ]

)(0) + γ

J∑j=1

‖S j‖1

s.t. 0 ≤ S j(m, n) ≤ 1

Contour Stencil Design

minS1,...,SJ

J∑i=1

J∑j=1

((S j − S i) ? [f j ]

)(0) + γ

J∑j=1

‖S j‖1

s.t. 0 ≤ S j(m, n) ≤ 1

The minimization has closed-form solution

S j(m, n) =

1 if |f j

m − f jn | < 1

J

∑Ji=1|f i

m − f in | −

γJ,

0 if |f jm − f j

n | > 1J

∑Ji=1|f i

m − f in | −

γJ.

Corner-Shaped Stencils

Example: Corner-shaped stencils designed from the features

f j(x) = maxx1 cos π4j − x2 sin π

4j ,

x1 sin π4j + x2 cos π

4j.

3D Stencils

In d dimensions, a stencil S : Zd × Zd → R is applied at voxelk ∈ Zd as

(S ? [u])(k) :=∑

m,n∈Zd

S(m, n) |uk+m − uk+n| .

Small TV detects isosurfaces. Some 3D stencils:

3D Stencils

Example: Stencils applied to ui ,j ,k =√

i2 + j2 + k2

The results are visualized by assigning a color to the region ofspace having a particular best-fitting stencil.

3D Stencils

Example: Stencils applied to an MRI brain volume

Demosaicing

Original Mosaiced Demosaiced

Bayer Grid

Demosaicing Stencils

Centered on a green pixel:

Axially-oriented Diagonally-oriented π8

-oriented

Centered on a red or blue pixel:

Axially-oriented Diagonally-oriented π8

-oriented

Demosaicing Stencils

Stencil orientation estimation on mosaiced data:

Demosaicing Stencils

Stencil orientation estimation on mosaiced data:

Preliminary Demosaicing Method

Let f be the given mosaiced image. We consider demosaicingby the minimization of

arg minu

∑k∈Y ,Cb,Cr

∑n∈Ω

∑m∈N (n)

wm,n

∣∣u(k)m − u(k)

n

∣∣+

λ

2

∑k∈R,G ,B

∑n∈Ω(k)

(fn − u(k)n )2

where

wm,n : weights choosen according the best-fitting stencils

N (n) : neighbors of pixel n

λ : fidelity parameter

Ω(k) : subset of Ω where kth channel is given

Exact

Demosaiced

Exact

Demosaiced

Exact

Demosaiced

Exact

Bilinear

Malvar et al.

Hamilton-Adams

Gunturk et al.

Li

Zhang-Wu

Proposed

Exact

Bilinear

Malvar et al.

Hamilton-Adams

Gunturk et al.

Li

Zhang-Wu

Proposed

Exact

Bilinear

Malvar et al.

Hamilton-Adams

Gunturk et al.

Li

Zhang-Wu

Proposed

Thanks!

Webpagehttp://www.math.ucla.edu/∼getreuer/contours

Contactgetreuer@gmail.com