CS6234: Lecture 4

Hon Wai Leong, NUS(CS6234, Spring 2009)

Copyright © 2009 by Leong Hon Wai

CS6234: Lecture 4

Linear ProgrammingLP and Simplex Algorithm [PS82]-Ch2Duality [PS82]-Ch3Primal-Dual Algorithm [PS82]-Ch5

Additional topics: Reading/Presentation by students

Lecture notes adapted from Comb Opt course by Jorisk, Math Dept, Maashricht Univ,

04/21/23 Combinatorial OptimizationMasters OR

Combinatorial Optimization

Chapter 5 [PS82]

The primal dual algorithm


Primal Dual Method

Source: CUHK, CS5160

primal dualrestricted

primalrestricted

dual

y z

found x, succeed!

Construct a better dual


Primal

LP in standard form + Dual

min c’ x

s.t. A x = b (≥ 0 ) (P) x ≥ 0

max π’ b

s.t. π’ A ≤ c (D) π’ free

Dual


Definition of the Dual

Definition 3.1: Given an LP in general form, called the primal, the dual is defined as follows

Primal Dual

Min c’x Max π’b

a’i x = bi i ε M πi free

xj ≥ 0 j ε N π’ Ai ≤ ci


Complementary slackness

Theorem 3.4 A pair, x, π, respectively feasible in a primal-dual pair is optimal if and only if:

ui = πi (a’ix - bi) = 0 for all i (1)

vj = (cj - π’Aj) xj = 0 for all j. (2)


Idea of a Primal-Dual AlgorithmSuppose we have a dual feasible solution π.

If we can find a primal feasible solution x such that xj = 0 whenever cj – π’Aj > 0 then:

Ai x = bi for all i and hence

(1) holds because x is a feasible solution to the primal (2) holds because xj = 0 whenever cj – π’Aj = 0.

Thus the complementary slackness relations hold, and hence x and π are optimal solutions to the primal problem (P) and the dual problem (D) resp.


Outline of the primal dual algorithm

P DRPRPD

x ρπ

Adjustment to π


Primal Dual Method

Source: CUHK, CS5160


primalrestricted

dual

y z

found x, succeed!



Getting started: finding a dual feasible π

If c ≥ 0, π = 0 is a dual feasible solution.

When cj < 0 for some j, introduce variable xm+1, and the additional constraint

x1 + x2 + … + xm+1 = bm+1. (bm+1 large enough)

The dual (D) then becomesmax π’ b + πm+1 bm+1

s.t. π’ A + πm+1 ≤ cj for all j, πm+1 ≤ 0.

The solution πi = 0, i=1…m, and πm+1 = minj cj < 0 is feasible for the dual (D).


Given a dual feasible solution π

Thus, we assume we have a dual feasible solution π. Consider the set

J = { j : π’j Aj = cj }.

A solution x to the primal (P) is optimal iffxj=0 for all j not in J.

Hence, we aim for an x feasible in

Σj=1n aij xj = bi i =1,..,m

xj ≥ 0, for all j in J

xj = 0, for all j not in J


Restricted Primal (RP)

Min ξ = Σj=1m xj

a

Σj=1n aijxj + xj

a = bi i =1,..,m

xj ≥ 0, for all j in J (RP)

xj = 0, for all j not in J

xja ≥ 0, i =1,..,m

If ξopt = 0, the corresponding optimal solution to (RP) yields an x which satisfies (together with π) the complementary slackness conditions.

Thus, in the remainder we consider the case where ξopt > 0.


The dual of the restricted primal: DRP

Min Σj=1m xj

a

s.t. ΣjεJ aij xj + xja = bi i =1,..,m

xj ≥ 0, for all j in J (RP) xj = 0, for all j not in J

xja ≥ 0, i =1,..,m

Max π’ b s.t. π’ Aj ≤ 0 for all j in J

πi ≤ 1, i=1…m,

πi free i=1…m,

We denote by ρ the optimal solution of this (DRP).


Comparison of (D) and (DRP)

(D)

Max π’ b

s.t. π’ Aj ≤ cj j=1…n

πi freei=1…m.

(DRP)

Max π’ b

s.t. π’ Aj ≤ 0 for all j in J

πi ≤ 1, i=1…m,

πi free i=1…m.


A Primal Dual iteration….

Let θ ε R, and consider π* = π + θρ.

Assume θ is such that π* is feasible in (D). Then π*’

b = π’b+ θρ’b.

Obviously ρ’b = ξopt > 0, since otherwise x together with π satisfies the complementary slackness conditions. Thus, by choosing θ > 0, π*’ b > π’b, yielding an improved solution of the dual (D).


A Primal Dual iteration

For π* to be feasible in (D), it must hold that:

π*’Aj = π’Aj + θρ’Aj ≤ cj , j=1…n .

From the definition of (DRP) it holds that

ρ’Aj ≤ 0 for all j in J. Thus if ρ’Aj ≤ 0 for all j, we can choose θ = +∞.But then (D) is unbounded, and hence (P) is infeasible. (Theorem 5.1)

Thus, we assume that ρ’Aj > 0 for some j not in J.


We are going to allow j to ‘enter the basis’ of (D)…

Let θ1 to be the maximum value such that

π’ Aj+ θρ’Aj ≤ cj for all j not in J and ρ’Aj > 0

Define π* = π + θ1 ρ. Then π*’b = π’ b+ θ1 ρ’ b > π*’b.

A Primal Dual iteration

Dual constraint j is satisfied at equality, and then by complementary slackness, primal variable xj can have nonnegative value


The Primal Dual AlgorithmInput: Feasible solution π for (D).Output: Optimal solution x for (P) if it exists.

Infeasible false; opt false;While not (infeasible or opt) begin

J {j: π’ Aj =cj }

Solve (RP) giving solution x.If ξopt = 0

then opt trueelse if ρ’A ≤ 0

then infeasible trueelse π π + θ1 ρ

End {while}


Admissible columns

Definition Given a solution π to the dual (D), let J = { j : π’

Aj =cj }. Then any column Aj , jεJ is called an admissable columns.

Theorem 5.3. A column which is in the optimal basis of (RP) and admissable in some iteration of the Primal Dual algorithm remains admissable at the start of the next iteration.


Admissable columns

Proof. If Aj is in the optimal basis of (RP) then, by complementary slackness dj - ρ’

Aj = 0, where dj is the cost coefficient in the objective function of the (RP). Hence dj = 0, and thus ρ’

Aj = 0.

This in turn implies that π*’ Aj = π’ Aj+ θρ’Aj

= π’ Aj

= cj

Since j was an admissable column. Thus j remains an admissable column.


Consequence

In each iteration, the optimal solution x of (RP), is also a basic feasible solution of the (RP) in the next iteration.

Thus subsequent (RP)’s can be solved taking the optimal solution of the previous one as a starting point for a simplex iteration. Moreover, using an anti cycling rule, the Primal Dual algorithm is finite (see Theorem 5.4).


A Primal Dual method for Shortest Path

node-arc incidence matrix A: 1 if arc k leaves node i

aij = { -1 if arc k enters node i 0 otherwise

Primal (P):

min c’ f

s.t. Σk Ak fk = 0 for all i ≠ s,t.

Σk Ak fk = 1 i = s.f ≥ 0


Dual (D) of the shortest path problem

max πs

s.t. πi - πj ≤ cij,

π free,

πt = 0.

Admissable arcs J = { arcs (i,j) : πi - πj = cij }.


Restricted Primal (RP) of Shortest Path

min Σi xia

s.t. ΣkεJ Ak fk + xia

= 0 for all i ≠ s,t.

Σ{j:(s,j)εJ} A(s,j)f(s,j) + xsa = 1

fk ≥ 0 for all k,

fk = 0 for all k not in J,xi

a ≥ 0 for all i.


Dual of the Restricted Primal (DRP)

max πs

s.t. πi - πj ≤ 0, for all arcs (i,j) in J.

πi ≤ 1, for all i.

πt = 0,

π free,


Solving (DRP)

Obviously πs ≤ 1, and since we aim to maximize πs we try πs = 1. But then all nodes i reachable by an admissable arc from s must also have πi = 1. This argument applies recursively.

Similarly since πt = 0, all nodes i (recursively) reachable from the sink, must have πi = 0.


Optimal solution ρ to (DRP)

If the source can be reached from the sink by a path consisting of admissable arcs πs = 1 end hence (RP) has optimal value zero and we are done.

s

t1 01


Solving (DRP)

Let θ1 to be the maximum value such that

π’ Ak+ θρ’Ak≤ ck for all k not in J and ρ’Ak > 0.

Thus we must consider (i,j) such that ρi -ρj =1.

Therefore θ1 is the minimum over the aforementioned (i,j) of cij – (πi - πj ).

Interpretation on next slide…


Interpretation of Primal Dual iteration

We assume cij > 0 for i,j = 1…m.

Then the solution πi = 0 for i=1…m is feasible in (D), and selected as the starting solution.

We select a non admissable arc from a green or red node i (which has ρi = 1 to a yellow node j, which has ρi = 0. Since arc (i,j) is non admissable it must hold that

πi – πj < cij.


Example

2

s

4

3

2

1

t

1

3

3

1

2

2

5


Initial dual feasible solution

2

s

4

3

2

1

t

1

3

3

1

2

2

5

πs=0

π2=0

π1=0

π4=0

π3=0

πt=0

Admissable arcs: Ø


Initial solution of (RP)

xsa=1, xi

a=0, for all i ≠ s,

no variables fk for admissable columns Ak.


First DRP

2

s

4

3

2

1

t

1

3

3

1

2

2

5

ρs=1

ρ2=1

ρ1=1

ρ4=1

ρ3=1

ρt=0

θ1 = c3t + πt - π3 = 2


Next solution of (RP)

xsa=1, xi

a=0, for all i ≠ s,

variables fk for admissable columns Ak from node 3 to t, but primal variables remain unchanged. (and will remain unchanged, until some arc leaving s is admissable…


Next dual feasible solution

2

s

4

3

2

1

t

1

3

3

1

2

2

5

πs=2

π2=2

π1=2

π4=2

π3=2

πt=0

Admissable arcs: (3,t)


Next DRP

2

s

4

3

2

1

t

1

3

3

1

2

2

5

ρs=1

ρ2=1

ρ1=1

ρ4=1

ρ3=0

ρt=0

θ1 = c43 + π3 - π4 = 2



2

s

4

3

2

1

t

1

3

3

1

2

2

5

πs=4

π2=4

π1=4

π4=4

π3=2

πt=0

Admissable arcs: (3,t), (4,3)


Next DRP

2

s

4

3

2

1

t

1

3

3

1

2

2

5

ρs=1

ρ2=1

ρ1=1

ρ4=0

ρ3=0

ρt=0

θ1 = c24 + π4 - π2 = 1 = c13 + π3 - π1 = 1



2

s

4

3

2

1

t

1

3

3

1

2

2

5

πs=5

π2=5

π1=5

π4=4

π3=2

πt=0

Admissable arcs: (3,t), (4,3), (1,3), (2,4)


Next DRP

2

s

4

3

2

1

t

1

3

3

1

2

2

5

ρs=1

ρ2=0

ρ1=0

ρ4=0

ρ3=0

ρt=0

θ1 = cs2 + π2 - πs = 1



2

s

4

3

2

1

t

1

3

3

1

2

2

5

πs=6

π2=5

π1=5

π4=4

π3=2

πt=0

Admissable arcs: (3,t), (4,3), (1,3), (2,4),(s,2)


Next solution of (RP)

xia=0, for all i,

fs2=1, f24=1, f43=1, f3t=1, all other fij = 0.

Solution value 0 f is feasible in primal (P), and since c ’x = π’b optimal in (P)!


Combinatorialization…(P) has an integer cost vector c, and (0,1) right hand side. (RP)

has a (0,1) cost vector and right hand side b. Therefore in (RP) the complexity is not from the numbers but from the combinatorics.

Similarly (D) has a (0,1) vector b, and integer right hand side c. (DRP) has a (0,1) cost vector and right hand side. Therefore in (DRP) the complexity is not from the numbers but from the combinatorics.

The primal dual algorithm solves a problem with ‘numerical’ complexity, by repeatedly solving a problem which has only ‘combinatorial’ complexity.

This concept, which Papadimitriou calls ‘combinatorialization’ is frequently encountered in combinatorial optimization.



Additional notes from [CUHK]

Primal Dual Method

Lecture 20: March 28


primalrestricted

dual

y z

found x, succeed!


Primal Dual ProgramPrimal Dual Program

Primal Program Dual Program

If there is a feasible primal solution x and a feasible dual solution y,

then both are optimal solutions.

Primal-Dual Method: An algorithm to construct such a pair of solutions.

Optimality Condition

Suppose there is a feasible primal solution x, and a feasible dual solution y.

How do we check that they are optimal solutions?

Avoid strict inequality


Optimality Condition

Complementary Slackness Conditions

Primal complementary slackness condition:

Dual complementary slackness condition:

Primal Dual Method

Start from a feasible dual solution.

Search for a feasible primal solution

satisfying complementary slackness conditions.

If not, improve the objective value of dual solution.

repeat

Restricted Primal

Formulate this as an LP itself!

Given a feasible dual solution y, how do we search for a feasible

primal solution x that satisfies complementary slackness conditions?



Restricted Primal

If j not in J, then we need x(j) to be zero.

If i not in I, then we need

Restricted primal

If zero, we are done

Restricted Dual

Suppose the objective of the restricted primal is not zero, what do we do?

Then we want to find a better dual solution.

Restricted primal Restricted dualnonzero

Restricted Dual

Restricted dualnonzeroDual Program

Consider y+єz as the new dual solution.

Still feasible

Larger value

General Framework


primalrestricted

dual

y z

found x, succeed!


Bipartite Matching


Start from a feasible vertex cover

Find a perfect matching using tight edges

non-zero

Consider y- єz, a better dual

Hungarian Method

Remarks

It is not a polynomial time method.

It reduces the weighted problem to the unweighted problem,

so that the restricted primal linear program is easier to solve,

and often there are combinatorial algorithms to solve it.

Many combinatorial algorithms, like max-flow, matching,

min-cost flow, shortest path, spanning tree, …, can be derived

within this framework.

Approximation Algorithm

How do we adapt the primal-dual method for approximation algorithms?

We want to construct a primal feasible solution x and

a dual feasible solution y so that cx and by are “close”.

Avoid losing too much


Approximate Optimality Condition

Approximate Complementary Slackness Condition


Dual complementary slackness condition: Only a sufficient condition

Vertex Cover



Approximate Optimality Conditions



Pick only vertices that go tight.

Pick only edge with one vertex in the vertex cover.

This would imply a 2-approximation.

This is nothing!

Just focus on this!

Algorithm

Pick only vertices that go tight.

Algorithm (2-approximation for vertex cover)

Initially, x=0, y=0

When there is an uncovered edge

Pick an uncovered edge, and raise y(e) until some vertices go tight.

Add all tight vertices to the vertex cover.

Output the vertex cover x.

Familiar? This is the greedy matching 2-approximation

when every vertex has the same cost.



Thank you.

Q & A

CS6234: Lecture 4

Documents

Transcript of CS6234: Lecture 4