Multiplicative Boundsfor Metric Labeling

M. Pawan Kumar

École Centrale Paris

Joint work with Phil Torr, Daphne Koller

Metric Labeling

Variables V = { V1, V2, …, Vn}

Metric Labeling

Variables V = { V1, V2, …, Vn}

Metric Labeling

Va Vb

Labels L = { l1, l2, …, lh}Variables V = { V1, V2, …, Vn}

Labeling f: { 1, 2, …, n} {1, 2, …, h}

E(f) = Σa θa(f(a)) + Σ(a,b) wabd(f(a),f(b))minf

θa(f(a))

θb(f(b))wabd(f(a),f(b))

wab ≥ 0

d is metric

Metric Labeling

Va Vb

E(f)minf

NP hard

= Σa θa(f(a)) + Σ(a,b) wabd(f(a),f(b))

Low-level vision applications

Minka. Expectation Propagation for Approximate Bayesian Inference, UAI, 2001 Murphy et al. Loopy Belief Propagation: An Empirical Study, UAI, 1999 Winn et al. Variational Message Passing, JMLR, 2005 Yedidia et al. Generalized Belief Propagation, NIPS, 2001 Besag. On the Statistical Analysis of Dirty Pictures, JRSS, 1986 Boykov et al. Fast Approximate Energy Minimization via Graph Cuts, PAMI, 2001 Komodakis et al. Fast, Approximately Optimal Solutions for Single and Dynamic MRFs, CVPR, 2007 Lempitsky et al. Fusion Moves for Markov Random Field Optimization, PAMI, 2010 Chekuri et al. Approximation Algorithms for Metric Labeling, SODA, 2001 Goemans et al. Improved Approximate Algorithms for Maximum-Cut, JACM, 1995 Muramatsu et al. A New SOCP Relaxation for Max-Cut, JORJ, 2003 Ravikumar et al. QP Relaxations for Metric Labeling, ICML, 2006 Alahari et al. Dynamic Hybrid Algorithms for MAP Inference, PAMI 2010 Kohli et al. On Partial Optimality in Multilabel MRFs, ICML, 2008 Rother et al. Optimizing Binary MRFs via Extended Roof Duality, CVPR, 2007

.

.

.

Approximate Algorithms

Outline

• Linear Programming Relaxation

• Move-Making Algorithms

• Comparison

• Rounding-based Moves

Integer Linear Program

Number of facets grows exponentially in problem size

Minimize a linear function over a set of feasible solutions

Indicator xa(i) {0,1} for each variable Va and label li

Linear Programming Relaxation

Schlesinger, 1976; Chekuri et al., 2001; Wainwright et al., 2003

Indicator xa(i) {0,1} for each variable Va and label li

Linear Programming Relaxation

Schlesinger, 1976; Chekuri et al., 2001; Wainwright et al., 2003

Indicator xa(i) [0,1] for each variable Va and label li

Outline

• Linear Programming Relaxation

• Move-Making Algorithms

• Comparison

• Rounding-based Moves

Move-Making AlgorithmsSpace of All Labelings

f

Expansion AlgorithmVariables take label lα or retain current label

Slide courtesy Pushmeet Kohli Boykov, Veksler and Zabih, 2001

Expansion Algorithm

SkyHouse

TreeGround

Initialize with TreeStatus: Expand GroundExpand HouseExpand Sky

Slide courtesy Pushmeet Kohli

Variables take label lα or retain current label

Boykov, Veksler and Zabih, 2001

Outline

• Linear Programming Relaxation

• Move-Making Algorithms

• Comparison

• Rounding-based Moves

Multiplicative Boundsf*: Optimal Labeling f: Estimated Labeling

Σa θa(f(a)) + Σ(a,b) sabd(f(a),f(b))

Σa θa(f*(a)) + Σ(a,b) sabd(f*(a),f*(b))

≥

Multiplicative Boundsf*: Optimal Labeling f: Estimated Labeling

≤B

Σa θa(f(a)) + Σ(a,b) sabd(f(a),f(b))

Σa θa(f*(a)) + Σ(a,b) sabd(f*(a),f*(b))

Multiplicative Bounds

Expansion LP

Potts 2 2

Metric 2M O(log h)

TruncatedLinear

2M 2 + √2

TruncatedQuadratic

2M O(√M)

M = ratio of maximum and minimum distance

Outline

• Linear Programming Relaxation

• Move-Making Algorithms

• Comparison

• Rounding-based Moves– Complete Rounding– Interval Rounding– Hierarchical Rounding

Complete Rounding

Treat xa(i) [0,1] as probability that f(a) = i

Cumulative probability ya(i) = Σj≤i xa(j)

0 ya(1) ya(2) ya(h) = 1

r

Generate a random number r (0,1]

Assign the label next to r

ya(k)ya(i)

Complete Move

Va Vb

θab(i,k) = sabd(i,k) NP-hard

Complete Move

Va Vb

θab(i,k) = sabd’(i,k)

d’(i,k) ≥ d(i,k)

d’ is submodular

Complete Move

Va Vb

θab(i,k) = sabd’(i,k)

d’(i,k) ≥ d(i,k)

d’ is submodular

Complete Move

New problem can be solved using minimum cut

Same multiplicative bound as complete rounding

Multiplicative bound is tight

Outline

• Linear Programming Relaxation

• Move-Making Algorithms

• Comparison

• Rounding-based Moves– Complete Rounding– Interval Rounding– Hierarchical Rounding

Interval Rounding

Treat xa(i) [0,1] as probability that f(a) = i

Cumulative probability ya(i) = Σj≤i xa(j)

0 ya(1) ya(2) ya(h) = 1ya(k)ya(i)

Choose an interval of length h’

Interval Rounding

Treat xa(i) [0,1] as probability that f(a) = i

Cumulative probability ya(i) = Σj≤i xa(j)

r

Generate a random number r (0,1]

Assign the label next to r if it is within the interval

ya(k)ya(i)

Choose an interval of length h’ REPEAT

Interval Move

Va Vb

θab(i,k) = sabd(i,k)

Choose an interval of length h’

Interval Move

Va Vb

θab(i,k) = sabd(i,k)

Choose an interval of length h’

Add the current labels

Interval Move

Va Vb

θab(i,k) = sabd’(i,k)

Choose an interval of length h’

Add the current labels

d’(i,k) ≥ d(i,k)

d’ is submodular

Solve to update labels

Repeat until convergence

Interval Move

Each problem can be solved using minimum cut

Same multiplicative bound as interval rounding

Multiplicative bound is tight

Kumar and Torr, NIPS 2008

Outline

• Linear Programming Relaxation

• Move-Making Algorithms

• Comparison

• Rounding-based Moves– Complete Rounding– Interval Rounding– Hierarchical Rounding

Hierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Hierarchical clustering of labels (e.g. r-HST metrics)

Hierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Assign variables to labels L1, L2 or L3

Move down the hierarchy until the leaf level

Hierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Assign variables to labels l1, l2 or l3

Hierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Assign variables to labels l4, l5 or l6

Hierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Assign variables to labels l7, l8 or l9

Hierarchical Move

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Hierarchical clustering of labels (e.g. r-HST metrics)

Hierarchical Move

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Obtain labeling f1 restricted to labels {l1,l2,l3}

Hierarchical Move

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Obtain labeling f2 restricted to labels {l4,l5,l6}

Hierarchical Move

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Obtain labeling f3 restricted to labels {l7,l8,l9}

Hierarchical Move

L1 L2 L3

Va Vb

f1(a)

f2(a)

f3(a)

Move up the hierarchy until we reach the root

f1(b)

f2(b)

f3(b)

Hierarchical Move

Each problem can be solved using minimum cut

Same multiplicative bound as hierarchical rounding

Multiplicative bound is tight

Kumar and Koller, UAI 2009

ConclusionMove-Making LP

Potts 2 2

Metric O(log h) O(log h)

TruncatedLinear

2 + √2 2 + √2

TruncatedQuadratic

O(√M) O(√M)

M = ratio of maximum and minimum distance

Open Problems

• Moves for general rounding schemes

• Higher-order energy functions

• Better comparison criterion

Questions?

http://www.centrale-ponts.fr/personnel/pawan