Rounding-based Movesfor Metric Labeling

M. Pawan Kumar

École Centrale ParisINRIA Saclay, Île-de-France

Metric Labeling

Variables V = { V1, V2, …, Vn}

Metric Labeling

Variables V = { V1, V2, …, Vn}

Metric Labeling

Va Vb

Labels L = { l1, l2, …, lh}Variables V = { V1, V2, …, Vn}

Labeling f: { 1, 2, …, n} {1, 2, …, h}

E(f) = Σa θa(f(a)) + Σ(a,b) wabd(f(a),f(b))minf

θa(f(a))

θb(f(b))

wabd(f(a),f(b))

wab ≥ 0

d is metric

Metric Labeling

Va Vb

E(f)minf

NP hard

= Σa θa(f(a)) + Σ(a,b) wabd(f(a),f(b))

Low-level vision applications

Outline

• Approximate Algorithms

• Comparison

• Rounding-based Moves

Boykov, Veksler and Zabih

Kleinberg and Tardos

Efficiency

Accuracy

Move-Making Algorithms

Convex Relaxations

Kolmogorov and Boykov

Move-Making Algorithms

Convex Relaxations

Chekuri, Khanna,Naor and Zosin

Efficiency

Accuracy

Outline

• Approximate Algorithms– Move-Making Algorithms– Linear Programming Relaxation

• Comparison

• Rounding-based Moves

Move-Making Algorithms

Space of All Labelings

f

Expansion Algorithm

Variables take label lα or retain current label

Slide courtesy Pushmeet Kohli Boykov, Veksler and Zabih, 2001

Expansion Algorithm

Sky

House

Tree

Ground

Initialize with TreeStatus: Expand GroundExpand HouseExpand Sky

Slide courtesy Pushmeet Kohli

Variables take label lα or retain current label

Boykov, Veksler and Zabih, 2001

Multiplicative Bounds

f*: Optimal Labeling f: Estimated Labeling

Σa θa(f(a)) + Σ(a,b) wabd(f(a),f(b))

Σa θa(f*(a)) + Σ(a,b) wabd(f*(a),f*(b))

≥

Multiplicative Bounds

f*: Optimal Labeling f: Estimated Labeling

≤

B

Σa θa(f(a)) + Σ(a,b) wabd(f(a),f(b))

Σa θa(f*(a)) + Σ(a,b) wabd(f*(a),f*(b))

Ask me the obvious question

Outline

• Approximate Algorithms– Move-Making Algorithms– Linear Programming Relaxation

• Comparison

• Rounding-based Moves

Integer Linear Program

Number of facets grows exponentially in problem size

Minimize a linear function over a set of feasible solutions

Indicator xa(i) {0,1} for each variable Va and label li

Indicator xab(i,k) {0,1} for each neighbor (Va,Vb) and labels li, lk

Linear Programming Relaxation

Schlesinger, 1976; Chekuri et al., 2001; Wainwright et al., 2003

Indicator xa(i) {0,1} for each variable Va and label li

Indicator xab(i,k) {0,1} for each neighbor (Va,Vb) and labels li, lk

Linear Programming Relaxation

Schlesinger, 1976; Chekuri et al., 2001; Wainwright et al., 2003

Indicator xa(i) [0,1] for each variable Va and label li

Indicator xab(i,k) [0,1] for each neighbor (Va,Vb) and labels li, lk

Approximation Factor

x*: LP Optimal Solution x: Estimated Integral Solution

Σa Σi θa(i)xa(i) + Σ(a,b) Σ(i,k) wabd(i,k)xab(i,k)

≥

Σa Σi θa(i)x*a(i) + Σ(a,b) Σ(i,k) wabd(i,k)x*ab(i,k)

Approximation Factor

x*: LP Optimal Solution x: Estimated Integral Solution

Σa Σi θa(i)xa(i) + Σ(a,b) Σ(i,k) wabd(i,k)xab(i,k)

≤

Σa Σi θa(i)x*a(i) + Σ(a,b) Σ(i,k) wabd(i,k)x*ab(i,k)F

Outline

• Approximate Algorithms

• Comparison

• Rounding-based Moves

Theoretical Guarantees

Expansion LP

Uniform 2 2

Metric 2M O(log h)

TruncatedLinear

2M 2 + √2

TruncatedQuadratic

2M O(√M)

M = ratio of maximum and minimum non-zero distance

Outline

• Approximate Algorithms

• Comparison

• Rounding-based Moves– Complete Rounding– Interval Rounding– Hierarchical Rounding

Complete Rounding

Treat xa(i) [0,1] as probability that f(a) = i

Cumulative probability ya(i) = Σj≤i xa(j)

0 ya(1) ya(2) ya(h) = 1ya(k)ya(i)

Generate a random number r (0,1]

Assign the label next to r

r

Example

0 ya(1) ya(4)ya(3)ya(2)

0.25 0.5 0.75 1.0

0 yb(1) yb(4)yb(3)yb(2)

0.7 0.8 0.9 1.0

0 yc(1) yc(4)yc(3)yc(2)

0.1 0.2 0.3 1.0

r

r

r

Complete Move

A move that mimics complete rounding

Considers all random variables and labels

Assigns labels in one iteration

Key Observation

If d is submodular

d(i,k) + d(i+1,k+1) ≤ d(i,k+1) + d(i+1,k), for all i, k

Schlesinger and Flach, 2003

energy can be minimized via minimum cut

Complete Move

Va Vb

θab(i,k) = wabd(i,k) NP-hard

Complete Move

Va Vb

θab(i,k) = wabd’(i,k)

d’(i,k) ≥ d(i,k)

d’ is submodular

Complete Move

Va Vb

θab(i,k) = wabd’(i,k)

d’(i,k) ≥ d(i,k)

d’ is submodular

Complete Move

New problem can be solved using minimum cut

Same multiplicative bound as complete rounding

Multiplicative bound is tight

Outline

• Approximate Algorithms

• Comparison

• Rounding-based Moves– Complete Rounding– Interval Rounding– Hierarchical Rounding

Interval Rounding

Treat xa(i) [0,1] as probability that f(a) = i

Cumulative probability ya(i) = Σj≤i xa(j)

0 ya(1) ya(2) ya(h) = 1ya(k)ya(i)

Choose an interval of length h’

Interval Rounding

Treat xa(i) [0,1] as probability that f(a) = i

Cumulative probability ya(i) = Σj≤i xa(j)

r

Generate a random number r (0,1]

Assign the label next to r if it is within the interval

ya(k)-ya(i)0

Choose an interval of length h’ REPEAT

Example

0 ya(1) ya(4)ya(3)ya(2)

0.25 0.5 0.75 1.0

0 yb(1) yb(4)yb(3)yb(2)

0.7 0.8 0.9 1.0

0 yc(1) yc(4)yc(3)yc(2)

0.1 0.2 0.3 1.0

Example

0 ya(1) ya(2)

0.25 0.5

0 yb(1) yb(2)

0.7 0.8

0 yc(1) yc(2)

0.1 0.2

r

r

r

Example

0 ya(1) ya(4)ya(3)ya(2)

0.25 0.5 0.75 1.0

0 yb(1) yb(4)yb(3)yb(2)

0.7 0.8 0.9 1.0

0 yc(1) yc(4)yc(3)yc(2)

0.1 0.2 0.3 1.0

Example

0 yc(1) yc(4)yc(3)yc(2)

0.1 0.2 0.3 1.0

Example

0 yc(3)yc(2)

0.1 0.2r

-yc(1) -yc(1)

Example

0 ya(1) ya(4)ya(3)ya(2)

0.25 0.5 0.75 1.0

0 yb(1) yb(4)yb(3)yb(2)

0.7 0.8 0.9 1.0

0 yc(1) yc(4)yc(3)yc(2)

0.1 0.2 0.3 1.0

Interval Move

A move that mimics interval rounding

Considers all variables and an interval of labels

Changes labeling iteratively

Key Observation

If d is submodular

d(i,k) + d(i+1,k+1) ≤ d(i,k+1) + d(i+1,k), for all i, k

Schlesinger and Flach, 2003

energy can be minimized via minimum cut

Interval Move

Va Vb

θab(i,k) = wabd(i,k)

Choose an interval of length h’

Interval Move

Va Vb

θab(i,k) = wabd(i,k)

Choose an interval of length h’

Add the current labels

Interval Move

Va Vb

θab(i,k) = wabd’(i,k)

Choose an interval of length h’

Add the current labels

d’(i,k) ≥ d(i,k)

d’ is submodular

Solve to update labels

Repeat until convergence

Interval Move

Each problem can be solved using minimum cut

Same multiplicative bound as interval rounding

Multiplicative bound is tight

Boykov, Veksler and Zabih

Kleinberg and Tardos

Length of interval = 1

Length of interval = 1

Move-Making Algorithms

Convex Relaxations

Boykov, Veksler and Zabih

Chekuri, Khanna,Naor and Zosin

Length of interval = 1

Optimal interval length

Move-Making Algorithms

Convex Relaxations

Theoretical Guarantees

Moves LP

Uniform 2 2

Metric 2M O(log h)

TruncatedLinear

2 + √2 2 + √2

TruncatedQuadratic

O(√M) O(√M)

M = ratio of maximum and minimum non-zero distance

Outline

• Approximate Algorithms

• Comparison

• Rounding-based Moves– Complete Rounding– Interval Rounding– Hierarchical Rounding

Hierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Hierarchical clustering of labels (e.g. r-HST metrics)

Hierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Assign variables to labels L1, L2 or L3

Move down the hierarchy until the leaf level

Hierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Assign variables to labels l1, l2 or l3

Hierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Assign variables to labels l4, l5 or l6

Hierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Assign variables to labels l7, l8 or l9

Hierarchical Move

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Hierarchical clustering of labels (e.g. r-HST metrics)

Hierarchical Move

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Obtain labeling f1 restricted to labels {l1,l2,l3}

Hierarchical Move

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Obtain labeling f2 restricted to labels {l4,l5,l6}

Hierarchical Move

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Obtain labeling f3 restricted to labels {l7,l8,l9}

Hierarchical Move

L1 L2 L3

Va Vb

f1(a)

f2(a)

f3(a)

Move up the hierarchy until we reach the root

f1(b)

f2(b)

f3(b)

Hierarchical Move

Each problem can be solved using minimum cut

Same multiplicative bound as hierarchical rounding

Multiplicative bound is tight

Boykov, Veksler and Zabih

Kleinberg and Tardos

Flat hierarchy

r-HST hierarchy

Move-Making Algorithms

Convex Relaxations

Theoretical Guarantees

Moves LP

Uniform 2 2

Metric O(log h) O(log h)

TruncatedLinear

2 + √2 2 + √2

TruncatedQuadratic

O(√M) O(√M)

M = ratio of maximum and minimum non-zero distance

Questions?

http://cvn.ecp.fr/personnel/pawan

pawan.kumar@ecp.fr