Discrete Optimization Lecture 2 – Part 2

M. Pawan Kumar pawan.kumar@ecp.fr

Slides available online http://cvn.ecp.fr/personnel/pawan/

•  Integer Programming Formulation

•  LP Relaxation

Outline

Integer Programming Formulation

Va Vb

Label l0

Label l12

5

4

2

0

1 1

0

2Unary Potentials

θa;0 = 5

θa;1 = 2

θb;0 = 2

θb;1 = 4

Labeling f(a) = 1

f(b) = 0

ya;0 = 0 ya;1 = 1

yb;0 = 1 yb;1 = 0

Any f(.) has equivalent boolean variables ya;i

Integer Programming Formulation

Va Vb

2

5

4

2

0

1 1

0

2Unary Potentials

θa;0 = 5

θa;1 = 2

θb;0 = 2

θb;1 = 4

Labeling f(a) = 1

f(b) = 0

ya;0 = 0 ya;1 = 1

yb;0 = 1 yb;1 = 0

Find the optimal variables ya;i

Label l0

Label l1

Integer Programming Formulation

Va Vb

2

5

4

2

0

1 1

0

2Unary Potentials

θa;0 = 5

θa;1 = 2

θb;0 = 2

θb;1 = 4

Sum of Unary Potentials

∑a ∑i θa;i ya;i

ya;i ∈ {0,1}, for all Va, li ∑i ya;i = 1, for all Va

Label l0

Label l1

Integer Programming Formulation

Va Vb

2

5

4

2

0

1 1

0

2Pairwise Potentials

θab;00 = 0

θab;10 = 1

θab;01 = 1

θab;11 = 0

Sum of Pairwise Potentials

∑(a,b) ∑ik θab;ik ya;iyb;k

ya;i ∈ {0,1} ∑i ya;i = 1

Label l0

Label l1

Integer Programming Formulation

Va Vb

2

5

4

2

0

1 1

0

2Pairwise Potentials

θab;00 = 0

θab;10 = 1

θab;01 = 1

θab;11 = 0

Sum of Pairwise Potentials

∑(a,b) ∑ik θab;ik yab;ik

ya;i ∈ {0,1} ∑i ya;i = 1

yab;ik = ya;i yb;k

Label l0

Label l1

Integer Programming Formulation

min ∑a ∑i θa;i ya;i + ∑(a,b) ∑ik θab;ik yab;ik

ya;i ∈ {0,1}

∑i ya;i = 1

yab;ik = ya;i yb;k

Integer Programming Formulation

min θTy

ya;i ∈ {0,1}

∑i ya;i = 1

yab;ik = ya;i yb;k

θ = [ … θa;i …. ; … θab;ik ….] y = [ … ya;i …. ; … yab;ik ….]

One variable, two labels

ya;0

ya;1

ya;0 ∈ {0,1} ya;1 ∈ {0,1} ya;0 + ya;1 = 1

y = [ ya;0 ya;1] θ = [ θa;0 θa;1]

Two variables, two labels

θ  = [ θa;0 θa;1 θb;0 θb;1

θab;00 θab;01 θab;10 θab;11] y = [ ya;0 ya;1 yb;0 yb;1 yab;00 yab;01 yab;10 yab;11]

ya;0 ∈ {0,1} ya;1 ∈ {0,1} ya;0 + ya;1 = 1

yb;0 ∈ {0,1} yb;1 ∈ {0,1} yb;0 + yb;1 = 1

yab;00 = ya;0 yb;0 yab;01 = ya;0 yb;1

yab;10 = ya;1 yb;0 yab;11 = ya;1 yb;1

In General

Marginal Polytope

In General

θ  ∈ R(|V||L| + |E||L|2)

y ∈ {0,1}(|V||L| + |E||L|2)

Number of constraints

|V||L| + |V| + |E||L|2 ya;i ∈ {0,1} ∑i ya;i = 1 yab;ik = ya;i yb;k

Integer Programming Formulation

min θTy

ya;i ∈ {0,1}

∑i ya;i = 1

yab;ik = ya;i yb;k

θ = [ … θa;i …. ; … θab;ik ….] y = [ … ya;i …. ; … yab;ik ….]

Integer Programming Formulation

min θTy

ya;i ∈ {0,1}

∑i ya;i = 1

yab;ik = ya;i yb;k

Solve to obtain MAP labelling y*

Integer Programming Formulation

min θTy

ya;i ∈ {0,1}

∑i ya;i = 1

yab;ik = ya;i yb;k

But we can’t solve it in general

•  Integer Programming Formulation

•  LP Relaxation

Outline

Linear Programming Relaxation

min θTy

ya;i ∈ {0,1}

∑i ya;i = 1

yab;ik = ya;i yb;k

Two reasons why we can’t solve this

Linear Programming Relaxation

min θTy

ya;i ∈ [0,1]

∑i ya;i = 1

yab;ik = ya;i yb;k

One reason why we can’t solve this

Linear Programming Relaxation

min θTy

ya;i ∈ [0,1]

∑i ya;i = 1

∑k yab;ik = ∑kya;i yb;k

One reason why we can’t solve this

Linear Programming Relaxation

min θTy

ya;i ∈ [0,1]

∑i ya;i = 1

One reason why we can’t solve this

= 1 ∑k yab;ik = ya;i∑k yb;k

Linear Programming Relaxation

min θTy

ya;i ∈ [0,1]

∑i ya;i = 1

∑k yab;ik = ya;i

One reason why we can’t solve this

Linear Programming Relaxation

min θTy

ya;i ∈ [0,1]

∑i ya;i = 1

∑k yab;ik = ya;i

No reason why we can’t solve this * *memory requirements, time complexity

One variable, two labels

ya;0

ya;1

ya;0 ∈ {0,1} ya;1 ∈ {0,1} ya;0 + ya;1 = 1

y = [ ya;0 ya;1] θ = [ θa;0 θa;1]

One variable, two labels

ya;0

ya;1

ya;0 ∈ [0,1] ya;1 ∈ [0,1] ya;0 + ya;1 = 1

y = [ ya;0 ya;1] θ = [ θa;0 θa;1]

Two variables, two labels

θ  = [ θa;0 θa;1 θb;0 θb;1

θab;00 θab;01 θab;10 θab;11] y = [ ya;0 ya;1 yb;0 yb;1 yab;00 yab;01 yab;10 yab;11]

ya;0 ∈ {0,1} ya;1 ∈ {0,1} ya;0 + ya;1 = 1

yb;0 ∈ {0,1} yb;1 ∈ {0,1} yb;0 + yb;1 = 1

yab;00 = ya;0 yb;0 yab;01 = ya;0 yb;1

yab;10 = ya;1 yb;0 yab;11 = ya;1 yb;1

Two variables, two labels

θ  = [ θa;0 θa;1 θb;0 θb;1

θab;00 θab;01 θab;10 θab;11] y = [ ya;0 ya;1 yb;0 yb;1 yab;00 yab;01 yab;10 yab;11]

ya;0 ∈ [0,1] ya;1 ∈ [0,1] ya;0 + ya;1 = 1

yb;0 ∈ [0,1] yb;1 ∈ [0,1] yb;0 + yb;1 = 1

yab;00 = ya;0 yb;0 yab;01 = ya;0 yb;1

yab;10 = ya;1 yb;0 yab;11 = ya;1 yb;1

Two variables, two labels

θ  = [ θa;0 θa;1 θb;0 θb;1

θab;00 θab;01 θab;10 θab;11] y = [ ya;0 ya;1 yb;0 yb;1 yab;00 yab;01 yab;10 yab;11]

ya;0 ∈ [0,1] ya;1 ∈ [0,1] ya;0 + ya;1 = 1

yb;0 ∈ [0,1] yb;1 ∈ [0,1] yb;0 + yb;1 = 1

yab;00 + yab;01 = ya;0

yab;10 = ya;1 yb;0 yab;11 = ya;1 yb;1

Two variables, two labels

θ  = [ θa;0 θa;1 θb;0 θb;1

θab;00 θab;01 θab;10 θab;11] y = [ ya;0 ya;1 yb;0 yb;1 yab;00 yab;01 yab;10 yab;11]

ya;0 ∈ [0,1] ya;1 ∈ [0,1] ya;0 + ya;1 = 1

yb;0 ∈ [0,1] yb;1 ∈ [0,1] yb;0 + yb;1 = 1

yab;00 + yab;01 = ya;0

yab;10 + yab;11 = ya;1

In General

Marginal Polytope

Local Polytope

In General

θ  ∈ R(|V||L| + |E||L|2)

y ∈ [0,1](|V||L| + |E||L|2)

Number of constraints

|V||L| + |V| + |E||L|

Linear Programming Relaxation

min θTy

ya;i ∈ [0,1]

∑i ya;i = 1

∑k yab;ik = ya;i

No reason why we can’t solve this

Linear Programming Relaxation

Extensively studied Optimization

Schlesinger, 1976

Koster, van Hoesel and Kolen, 1998

Theory

Chekuri et al, 2001 Archer et al, 2004

Machine Learning

Wainwright et al., 2001

Linear Programming Relaxation

Many interesting properties

•  Global optimal MAP for trees

Wainwright et al., 2001

But we are interested in NP-hard cases

•  Preserves solution for reparameterization

•  Global optimal MAP for submodular energy

Chekuri et al., 2001

Linear Programming Relaxation

•  Large class of problems

•  Metric Labeling •  Semi-metric Labeling

Many interesting properties - Integrality Gap

Manokaran et al., 2008

•  Most likely, provides best possible integrality gap