Rounding-based Movesfor Metric Labeling

M. Pawan Kumar

Center for Visual Computing

Ecole Centrale Paris

PostMetric LabelingRandom variables V = {v1, v2, …, vn}

Label set L = {l1, l2, …, lh}

Labelings quantatively distinguished by energy E(y)

Labeling y L∈ n

Unary potential of variable va V∈

∑a θa(ya)

PostMetric LabelingRandom variables V = {v1, v2, …, vn}

Label set L = {l1, l2, …, lh}

Labelings quantatively distinguished by energy E(y)

Labeling y L∈ n

Pairwise potential of variables (va,vb)

∑a θa(ya) + ∑(a,b) wab d(ya,yb)

wab is non-negative d(.,.) is a metric distance function

miny

Post• Existing Work

– Move-Making Algorithms (Efficient)– Linear Programming Relaxation (Accurate)

• Rounding-based Moves– Equivalence– Complete Rounding and Complete Move– Interval Rounding and Interval Move– Hierarchical Rounding and Hierarchical Move

Outline

PostExpansion Algorithm

Sky

House

Tree

GroundInitialize with TreeExpand GroundExpand HouseExpand Sky

Variables take label lα or retain current label

Boykov, Veksler and Zabih, ICCV 1999

PostMove-Making AlgorithmsIteration t

Define St L⊆ n containing current labeling yt

∑a θa(ya) + ∑(a,b) wab d(ya,yb)argminy

s.t. y S∈ t

Sometimes it can even be solved exactly

Above problem is easier than original problem

yt+1 =

Start with an initial labeling y0

Post• Existing Work

– Move-Making Algorithms (Efficient)– Linear Programming Relaxation (Accurate)

• Rounding-based Moves– Equivalence– Complete Rounding and Complete Move– Interval Rounding and Interval Move– Hierarchical Rounding and Hierarchical Move

Outline

PostLinear Programming Relaxation

Chekuri, Khanna, Naor and Zosin, SODA 2001

Binary indicator xa(i) {0,1} ∈

If variable ‘a’ takes the label ‘i’ then xa(i) = 1

∑i xa(i) = 1Each variable ‘a’ takes one label

Similarly, binary indicator xab(i,k) {0,1} ∈

PostLinear Programming Relaxation

Minimize a linear function over feasible x

Indicators xa(i), xab(i,k) {0,1} Relaxed xa(i), xab(i,k) [0,1]

Rounding

Chekuri, Khanna, Naor and Zosin, SODA 2001

Post• Existing Work

– Move-Making Algorithms (Efficient)– Linear Programming Relaxation (Accurate)

• Rounding-based Moves– Equivalence– Complete Rounding and Complete Move– Interval Rounding and Interval Move– Hierarchical Rounding and Hierarchical Move

Outline

PostMove-Making Bound

y*: Optimal Labeling y: Estimated Labeling

Σa θa(ya) + Σ(a,b) wabd(ya,yb)

Σa θa(y*a) + Σ(a,b) wabd(y*a,y*b)

≥

PostMove-Making Bound

y*: Optimal Labeling y: Estimated Labeling

Σa θa(ya) + Σ(a,b) wabd(ya,yb)

Σa θa(y*a) + Σ(a,b) wabd(y*a,y*b)B

≤

For all possible values of θa(i) and wab

PostRounding Approximation

x*: LP Optimal Solution x: Rounded Solution

Σa Σi θa(i)xa(i) + Σ(a,b) Σ(i,k) wabd(i,k)xab(i,k)

≥

Σa Σi θa(i)x*a(i) + Σ(a,b) Σ(i,k) wabd(i,k)x*ab(i,k)

PostRounding Approximation

x*: LP Optimal Solution x: Rounded Solution

Σa Σi θa(i)xa(i) + Σ(a,b) Σ(i,k) wabd(i,k)xab(i,k)

≤

Σa Σi θa(i)x*a(i) + Σ(a,b) Σ(i,k) wabd(i,k)x*ab(i,k)A

For all possible values of θa(i) and wab

PostEquivalence

For any known rounding with approximation A

there exists a move-making algorithm

such that the move-making bound B = A

We know how to design such an algorithm

Post• Existing Work

– Move-Making Algorithms (Efficient)– Linear Programming Relaxation (Accurate)

• Rounding-based Moves– Equivalence– Complete Rounding and Complete Move– Interval Rounding and Interval Move– Hierarchical Rounding and Hierarchical Move

Outline

PostComplete Rounding

Treat x*a(i) [0,1] as probability that ya = li

Cumulative probability za(i) = Σj≤i x*a(j)

0 za(1) za(2) za(h) = 1za(k)za(i)

Generate a random number r (0,1]

Assign the label next to r

r

PostComplete Rounding - Example

0 za(1) za(4)za(3)za(2)

0.25 0.5 0.75 1.0

0 zb(1) zb(4)zb(3)zb(2)

0.7 0.8 0.9 1.0

0 zc(1) zc(4)zc(3)zc(2)

0.1 0.2 0.3 1.0

r

r

r

PostEquivalent Move

Complete Move !!

PostComplete MoveIteration t

Define St ⊆ Ln

∑a θa(ya) + ∑(a,b) wab d(ya,yb)argminy

s.t. y S∈ t

yt+1 =

Start with an initial labeling y0

PostComplete MoveIteration t

Define St = Ln

∑a θa(ya) + ∑(a,b) wab d(ya,yb)argminy

s.t. y S∈ t

How do we solve this problem?

Above problem is the same as the original problem

yt+1 =

Start with an initial labeling y0

PostComplete Move

Define St = Ln

∑a θa(ya) + ∑(a,b) wab d’(ya,yb)argminy

s.t. y S∈ t

How do we solve this problem?

Above problem is the same as the original problem

yt+1 =

PostComplete Move

Define St = Ln

∑a θa(ya) + ∑(a,b) wab d’(ya,yb)argminy

s.t. y S∈ t

Obtained by solving a small LP

Submodular overestimation d’ of d

yt+1 =

PostSubmodular Overestimation

maxi,k d’(li,lk)/d(li,lk)mind’

d’(li,lk) ≥ d(li,lk)s.t.

d’(li,lk+1) + d’(li+1,lk) ≥ d(li,lk) + d(li+1,lk+1)

PostSubmodular Overestimation

bmind’

d’(li,lk) ≥ d(li,lk)s.t.

d’(li,lk+1) + d’(li+1,lk) ≥ d(li,lk) + d(li+1,lk+1)

bd(li,lk) ≥ d’(li,lk)

Dual provides worst-case instance for complete rounding

Post• Existing Work

– Move-Making Algorithms (Efficient)– Linear Programming Relaxation (Accurate)

• Rounding-based Moves– Equivalence– Complete Rounding and Complete Move– Interval Rounding and Interval Move– Hierarchical Rounding and Hierarchical Move

Outline

PostInterval Rounding

Treat x*a(i) [0,1] as probability that ya = li

Cumulative probability za(i) = Σj≤i x*a(j)

0 za(1) za(2) za(h) = 1za(k)za(i)

Choose an interval of length h’

PostInterval Rounding

Treat x*a(i) [0,1] as probability that ya = li

Cumulative probability za(i) = Σj≤i x*a(j)

r

Generate a random number r (0,1]

Assign the label next to r if it is within the interval

za(k)-za(i)0

Choose an interval of length h’ REPEAT

PostInterval Rounding - Example

0 za(1) za(4)za(3)za(2)

0.25 0.5 0.75 1.0

0 zb(1) zb(4)zb(3)zb(2)

0.7 0.8 0.9 1.0

0 zc(1) zc(4)zc(3)zc(2)

0.1 0.2 0.3 1.0

PostInterval Rounding - Example

0 za(1) za(2)

0.25 0.5

0 zb(1) zb(2)

0.7 0.8

0 zc(1) zc(2)

0.1 0.2

r

r

r

PostInterval Rounding - Example

0 za(1) za(4)za(3)za(2)

0.25 0.5 0.75 1.0

0 zb(1) zb(4)zb(3)zb(2)

0.7 0.8 0.9 1.0

0 zc(1) zc(4)zc(3)zc(2)

0.1 0.2 0.3 1.0

PostInterval Rounding - Example

0 zc(1) zc(4)zc(3)zc(2)

0.1 0.2 0.3 1.0

PostInterval Rounding - Example

0 zc(3)zc(2)

0.1 0.2r

-zc(1) -zc(1)

PostInterval Rounding - Example

0 za(1) za(4)za(3)za(2)

0.25 0.5 0.75 1.0

0 zb(1) zb(4)zb(3)zb(2)

0.7 0.8 0.9 1.0

0 zc(1) zc(4)zc(3)zc(2)

0.1 0.2 0.3 1.0

PostEquivalent Move

Interval Move !!

PostInterval MoveIteration t

y S∈ t iff ya = yta or ya interval of labels ∈

∑a θa(ya) + ∑(a,b) wab d(ya,yb)argminy

s.t. y S∈ t

yt+1 =

Start with an initial labeling y0

Choose an interval of labels of length h’

How do we solve this problem?

PostInterval MoveIteration t

y S∈ t iff ya = yta or ya interval of labels ∈

∑a θa(ya) + ∑(a,b) wab d’(ya,yb)argminy

s.t. y S∈ t

yt+1 =

Start with an initial labeling y0

Choose an interval of labels of length h’

Submodular overestimation d’ of d

Post• Existing Work

– Move-Making Algorithms (Efficient)– Linear Programming Relaxation (Accurate)

• Rounding-based Moves– Equivalence– Complete Rounding and Complete Move– Interval Rounding and Interval Move– Hierarchical Rounding and Hierarchical Move

Outline

PostHierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Hierarchical clustering of labels (e.g. r-HST metrics)

PostHierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Assign variables to labels L1, L2 or L3

Move down the hierarchy until the leaf level

PostHierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Assign variables to labels l1, l2 or l3

PostHierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Assign variables to labels l4, l5 or l6

PostHierarchical Rounding

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Assign variables to labels l7, l8 or l9

PostEquivalent Move

Hierarchical Move !!

PostHierarchical Move

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Hierarchical clustering of labels (e.g. r-HST metrics)

PostHierarchical Move

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Obtain labeling y1 restricted to labels {l1,l2,l3}

PostHierarchical Move

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Obtain labeling y2 restricted to labels {l4,l5,l6}

PostHierarchical Move

L1 L2

l1 l2 l3 l4 l5 l6 l7 l8 l9

L3

Obtain labeling y3 restricted to labels {l7,l8,l9}

PostHierarchical Move

L1 L2 L3

Va Vb

y1(a)

y2(a)

y3(a)

Move up the hierarchy until we reach the root

y1(b)

y2(b)

y3(b)

Questions?

http://mpawankumar.info

PostSimple Example - Rounding

θa(1)xa(1) + θa(2)xa(2) + θb(1)xb(1) + θb(2)xb(2)minx≥0

+ d(1,1)xab(1,1) + d(1,2)xab(1,2)+ d(2,1)xab(2,1) + d(2,2)xab(2,2)

xa(1) + xa(2) = 1s.t.

xb(1) + xb(2) = 1

xab(1,1) + xab(1,2) = xa(1)

xab(2,1) + xab(2,2) = xa(2)

xab(1,1) + xab(2,1) = xb(1)

xab(1,2) + xab(2,2) = xb(2)

PostSimple Example - Rounding

x*a(1) + x*a(2) = 1x*a(1)0

x*b(1) + x*b(2) = 1x*b(1)0

Generate a uniform random number r (0,1]

Assign the label next to r

r

r

Probability that Va is assigned label l1? x*a(1)

Probability that Va is assigned label l2? x*a(2)

PostSimple Example - Rounding

x*a(1) + x*a(2) = 1x*a(1)0

x*b(1) + x*b(2) = 1x*b(1)0

Generate a uniform random number r (0,1]

Assign the label next to r

r

r

Probability that Va and Vb are assigned l1 and l1?

min{x*a(1), x*b(1)}

PostSimple Example - Rounding

x*a(1) + x*a(2) = 1x*a(1)0

x*b(1) + x*b(2) = 1x*b(1)0

Generate a uniform random number r (0,1]

Assign the label next to r

r

r

Probability that Va and Vb are assigned l1 and l1?

min{x*ab(1,1)+x*ab(1,2), x*ab(1,1) + x*ab(2,1)}

x*ab(1,1) + min{x*ab(1,2), x*ab(2,1)}

PostSimple Example - Rounding

x*a(1) + x*a(2) = 1x*a(1)0

x*b(1) + x*b(2) = 1x*b(1)0

Generate a uniform random number r (0,1]

Assign the label next to r

r

r

Probability that Va and Vb are assigned l1 and l2?

max{0,x*a(1) - x*b(1)}

PostSimple Example - Rounding

x*a(1) + x*a(2) = 1x*a(1)0

x*b(1) + x*b(2) = 1x*b(1)0

Generate a uniform random number r (0,1]

Assign the label next to r

r

r

Probability that Va and Vb are assigned l1 and l2?

x*ab(1,2) - min{x*ab(1,2), x*ab(2,1)}

max{0,x*ab(1,2) - x*ab(2,1)}

PostSimple Example - Rounding

x*a(1) + x*a(2) = 1x*a(1)0

x*b(1) + x*b(2) = 1x*b(1)0

Generate a uniform random number r (0,1]

Assign the label next to r

r

r

Probability that Va and Vb are assigned l2 and l1?

max{0,x*b(1) - x*a(1)}

PostSimple Example - Rounding

x*a(1) + x*a(2) = 1x*a(1)0

x*b(1) + x*b(2) = 1x*b(1)0

Generate a uniform random number r (0,1]

Assign the label next to r

r

r

Probability that Va and Vb are assigned l2 and l1?

x*ab(2,1) - min{x*ab(1,2), x*ab(2,1)}

max{0,x*ab(2,1) - x*ab(1,2)}

PostSimple Example - Rounding

x*a(1) + x*a(2) = 1x*a(1)0

x*b(1) + x*b(2) = 1x*b(1)0

Generate a uniform random number r (0,1]

Assign the label next to r

r

r

Probability that Va and Vb are assigned l2 and l2?

1-max{x*a(1), x*b(1)}

PostSimple Example - Rounding

x*a(1) + x*a(2) = 1x*a(1)0

x*b(1) + x*b(2) = 1x*b(1)0

Generate a uniform random number r (0,1]

Assign the label next to r

r

r

Probability that Va and Vb are assigned l2 and l2?

min{x*a(2), x*b(2)}

PostSimple Example - Rounding

x*a(1) + x*a(2) = 1x*a(1)0

x*b(1) + x*b(2) = 1x*b(1)0

Generate a uniform random number r (0,1]

Assign the label next to r

r

r

Probability that Va and Vb are assigned l2 and l2?

min{x*ab(2,2)+x*ab(1,2), x*ab(2,2) + x*ab(2,1)}

x*ab(2,2) + min{x*ab(1,2), x*ab(2,1)}

PostSimple Example - Move

θa(ya) + θb(yb)miny + d(ya,yb)

ya ,yb {1,2}∈

If d is submodular, solve using graph cuts

Otherwise

PostSimple Example - Move

θa(ya) + θb(yb)miny + d’(ya,yb)

ya ,yb {0,1}∈

If d is submodular, solve using graph cuts

Otherwise use submodular overestimation d’

Estimate d’ by minimizing distortion

PostSimple Example - Move

bmind'

d’(1,1) ≤ b d(1,1)s.t. d’(1,2) ≤ b d(1,2)

d’(2,1) ≤ b d(2,1) d’(2,2) ≤ b d(2,2)

d(1,1) ≤ d’(1,1) d(1,2) ≤ d’(1,2)

d(2,1) ≤ d’(2,1) d(2,2) ≤ d’(2,2)

d’(1,1) + d’(2,2) ≤ d’(2,1) + d’(2,2)

Dual LP provides worst-case rounding example

LP in the variables d’(i,k)

PostSimple Example - Moved(1,1)β(1,1)+d(1,2)β(1,2)+d(2,1)β(2,1)+d(2,2)β(2,2)minα,β,γ≥0

s.t. d(1,1)α(1,1)+d(1,2)α(1,2)+d(2,1)α(2,1)+d(2,2)α(2,2) = 1

β(1,1) = α(1,1) + γ

β(1,2) = α(1,2) - γ

β(2,1) = α(2,1) - γ

β(2,2) = α(2,2) + γ

Set xab*(i,k) = α(i,k) Set γ = min{xab*(1,2), xab*(2,1)}