Discrete Optimization - M. Pawan Kumar · [email protected] ... Integer Programming Formulation V...
35
Discrete Optimization Lecture 2 – Part 2 M. Pawan Kumar [email protected] Slides available online http://cvn.ecp.fr/personnel/pawan/
Transcript of Discrete Optimization - M. Pawan Kumar · [email protected] ... Integer Programming Formulation V...
Discrete Optimization Lecture 2 – Part 2
M. Pawan Kumar [email protected]
Slides available online http://cvn.ecp.fr/personnel/pawan/
• Integer Programming Formulation
• LP Relaxation
Outline
Integer Programming Formulation
Va Vb
Label l0
Label l12
5
4
2
0
1 1
0
2Unary Potentials
θa;0 = 5
θa;1 = 2
θb;0 = 2
θb;1 = 4
Labeling f(a) = 1
f(b) = 0
ya;0 = 0 ya;1 = 1
yb;0 = 1 yb;1 = 0
Any f(.) has equivalent boolean variables ya;i
Integer Programming Formulation
Va Vb
2
5
4
2
0
1 1
0
2Unary Potentials
θa;0 = 5
θa;1 = 2
θb;0 = 2
θb;1 = 4
Labeling f(a) = 1
f(b) = 0
ya;0 = 0 ya;1 = 1
yb;0 = 1 yb;1 = 0
Find the optimal variables ya;i
Label l0
Label l1
Integer Programming Formulation
Va Vb
2
5
4
2
0
1 1
0
2Unary Potentials
θa;0 = 5
θa;1 = 2
θb;0 = 2
θb;1 = 4
Sum of Unary Potentials
∑a ∑i θa;i ya;i
ya;i ∈ {0,1}, for all Va, li ∑i ya;i = 1, for all Va
Label l0
Label l1
Integer Programming Formulation
Va Vb
2
5
4
2
0
1 1
0
2Pairwise Potentials
θab;00 = 0
θab;10 = 1
θab;01 = 1
θab;11 = 0
Sum of Pairwise Potentials
∑(a,b) ∑ik θab;ik ya;iyb;k
ya;i ∈ {0,1} ∑i ya;i = 1
Label l0
Label l1
Integer Programming Formulation
Va Vb
2
5
4
2
0
1 1
0
2Pairwise Potentials
θab;00 = 0
θab;10 = 1
θab;01 = 1
θab;11 = 0
Sum of Pairwise Potentials
∑(a,b) ∑ik θab;ik yab;ik
ya;i ∈ {0,1} ∑i ya;i = 1
yab;ik = ya;i yb;k
Label l0
Label l1
Integer Programming Formulation
min ∑a ∑i θa;i ya;i + ∑(a,b) ∑ik θab;ik yab;ik
ya;i ∈ {0,1}
∑i ya;i = 1
yab;ik = ya;i yb;k
Integer Programming Formulation
min θTy
ya;i ∈ {0,1}
∑i ya;i = 1
yab;ik = ya;i yb;k
θ = [ … θa;i …. ; … θab;ik ….] y = [ … ya;i …. ; … yab;ik ….]
One variable, two labels
ya;0
ya;1
ya;0 ∈ {0,1} ya;1 ∈ {0,1} ya;0 + ya;1 = 1
y = [ ya;0 ya;1] θ = [ θa;0 θa;1]
Two variables, two labels
θ = [ θa;0 θa;1 θb;0 θb;1
θab;00 θab;01 θab;10 θab;11] y = [ ya;0 ya;1 yb;0 yb;1 yab;00 yab;01 yab;10 yab;11]
ya;0 ∈ {0,1} ya;1 ∈ {0,1} ya;0 + ya;1 = 1
yb;0 ∈ {0,1} yb;1 ∈ {0,1} yb;0 + yb;1 = 1
yab;00 = ya;0 yb;0 yab;01 = ya;0 yb;1
yab;10 = ya;1 yb;0 yab;11 = ya;1 yb;1
In General
Marginal Polytope
In General
θ ∈ R(|V||L| + |E||L|2)
y ∈ {0,1}(|V||L| + |E||L|2)
Number of constraints
|V||L| + |V| + |E||L|2 ya;i ∈ {0,1} ∑i ya;i = 1 yab;ik = ya;i yb;k
Integer Programming Formulation
min θTy
ya;i ∈ {0,1}
∑i ya;i = 1
yab;ik = ya;i yb;k
θ = [ … θa;i …. ; … θab;ik ….] y = [ … ya;i …. ; … yab;ik ….]
Integer Programming Formulation
min θTy
ya;i ∈ {0,1}
∑i ya;i = 1
yab;ik = ya;i yb;k
Solve to obtain MAP labelling y*
Integer Programming Formulation
min θTy
ya;i ∈ {0,1}
∑i ya;i = 1
yab;ik = ya;i yb;k
But we can’t solve it in general
• Integer Programming Formulation
• LP Relaxation
Outline
Linear Programming Relaxation
min θTy
ya;i ∈ {0,1}
∑i ya;i = 1
yab;ik = ya;i yb;k
Two reasons why we can’t solve this
Linear Programming Relaxation
min θTy
ya;i ∈ [0,1]
∑i ya;i = 1
yab;ik = ya;i yb;k
One reason why we can’t solve this
Linear Programming Relaxation
min θTy
ya;i ∈ [0,1]
∑i ya;i = 1
∑k yab;ik = ∑kya;i yb;k
One reason why we can’t solve this
Linear Programming Relaxation
min θTy
ya;i ∈ [0,1]
∑i ya;i = 1
One reason why we can’t solve this
= 1 ∑k yab;ik = ya;i∑k yb;k
Linear Programming Relaxation
min θTy
ya;i ∈ [0,1]
∑i ya;i = 1
∑k yab;ik = ya;i
One reason why we can’t solve this
Linear Programming Relaxation
min θTy
ya;i ∈ [0,1]
∑i ya;i = 1
∑k yab;ik = ya;i
No reason why we can’t solve this * *memory requirements, time complexity
One variable, two labels
ya;0
ya;1
ya;0 ∈ {0,1} ya;1 ∈ {0,1} ya;0 + ya;1 = 1
y = [ ya;0 ya;1] θ = [ θa;0 θa;1]
One variable, two labels
ya;0
ya;1
ya;0 ∈ [0,1] ya;1 ∈ [0,1] ya;0 + ya;1 = 1
y = [ ya;0 ya;1] θ = [ θa;0 θa;1]
Two variables, two labels
θ = [ θa;0 θa;1 θb;0 θb;1
θab;00 θab;01 θab;10 θab;11] y = [ ya;0 ya;1 yb;0 yb;1 yab;00 yab;01 yab;10 yab;11]
ya;0 ∈ {0,1} ya;1 ∈ {0,1} ya;0 + ya;1 = 1
yb;0 ∈ {0,1} yb;1 ∈ {0,1} yb;0 + yb;1 = 1
yab;00 = ya;0 yb;0 yab;01 = ya;0 yb;1
yab;10 = ya;1 yb;0 yab;11 = ya;1 yb;1
Two variables, two labels
θ = [ θa;0 θa;1 θb;0 θb;1
θab;00 θab;01 θab;10 θab;11] y = [ ya;0 ya;1 yb;0 yb;1 yab;00 yab;01 yab;10 yab;11]
ya;0 ∈ [0,1] ya;1 ∈ [0,1] ya;0 + ya;1 = 1
yb;0 ∈ [0,1] yb;1 ∈ [0,1] yb;0 + yb;1 = 1
yab;00 = ya;0 yb;0 yab;01 = ya;0 yb;1
yab;10 = ya;1 yb;0 yab;11 = ya;1 yb;1
Two variables, two labels
θ = [ θa;0 θa;1 θb;0 θb;1
θab;00 θab;01 θab;10 θab;11] y = [ ya;0 ya;1 yb;0 yb;1 yab;00 yab;01 yab;10 yab;11]
ya;0 ∈ [0,1] ya;1 ∈ [0,1] ya;0 + ya;1 = 1
yb;0 ∈ [0,1] yb;1 ∈ [0,1] yb;0 + yb;1 = 1
yab;00 + yab;01 = ya;0
yab;10 = ya;1 yb;0 yab;11 = ya;1 yb;1
Two variables, two labels
θ = [ θa;0 θa;1 θb;0 θb;1
θab;00 θab;01 θab;10 θab;11] y = [ ya;0 ya;1 yb;0 yb;1 yab;00 yab;01 yab;10 yab;11]
ya;0 ∈ [0,1] ya;1 ∈ [0,1] ya;0 + ya;1 = 1
yb;0 ∈ [0,1] yb;1 ∈ [0,1] yb;0 + yb;1 = 1
yab;00 + yab;01 = ya;0
yab;10 + yab;11 = ya;1
In General
Marginal Polytope
Local Polytope
In General
θ ∈ R(|V||L| + |E||L|2)
y ∈ [0,1](|V||L| + |E||L|2)
Number of constraints
|V||L| + |V| + |E||L|
Linear Programming Relaxation
min θTy
ya;i ∈ [0,1]
∑i ya;i = 1
∑k yab;ik = ya;i
No reason why we can’t solve this
Linear Programming Relaxation
Extensively studied Optimization
Schlesinger, 1976
Koster, van Hoesel and Kolen, 1998
Theory
Chekuri et al, 2001 Archer et al, 2004
Machine Learning
Wainwright et al., 2001
Linear Programming Relaxation
Many interesting properties
• Global optimal MAP for trees
Wainwright et al., 2001
But we are interested in NP-hard cases
• Preserves solution for reparameterization
• Global optimal MAP for submodular energy
Chekuri et al., 2001
Linear Programming Relaxation
• Large class of problems
• Metric Labeling • Semi-metric Labeling
Many interesting properties - Integrality Gap
Manokaran et al., 2008
• Most likely, provides best possible integrality gap
