Design and Implementation of Parallel andRandomized Approximation Algorithms

byAjay Shankar Bidyarthy

(b.ajay@iitg.ernet.in)

Under the Guidance ofDr. Gautam K. Das

Department of MathematicsIndian Institute of Technology Guwahati

Guwahati - 781039, India

November 26, 2012

Topics

I Matrix Games solver

I Linear Programs solver

I Semi-Definite Programs solver

I Example: Lovasz ϑ function

I Results

I Conclusions

I Future Works

Matrix Games solver

I The problem is to compute an x such that

Ax ≤ εe, x ∈ S = {x ∈ Rn|eᵀx = 1, x ≥ 0}

whereI A is the payoff matrixI A = −Aᵀ

I Elements of A lie in [−1, 1]I ε ∈ (0, 1]I n ≥ 8

Matrix Games solver

I Our solver returnsI x ∈ Rn, an optimal strategy vector for AI with probability ≥ 1/2

I Time complexityI O( 1

ε2 log2 n) expected time on an n/ log n-processor [Khachiyanet al. [5]] (we will refer it by GK)

I Our solver takes O( 1ε2 n log n) at most time on single processor

Matrix Games solver

I Performance Analysis of our Matrix Games solver

10 20 30 40 50 60 70 80

5

10

15

20ε −

−−

−−

−−

−−

−−

>

Time Taken (min) −−−−−−−−>

Precision Time Tradeoff (n = 5000)

GK

Sedumi

200 400 600 800 1000 1200 1400 1600 1800

0.01

0.02

0.03

0.04

0.05

ε −

−−

−−

−−

−−

−−

>

Time Taken (min) −−−−−−−−>

Precision Time Tradeoff (n = 5000)

GK

Sedumi

: Precision time tradeoff (matrix A5000×5000), ε error accuracy withrespect to CPU time taken by GK algorithm and SeDuMi

Linear Programs solver

I LP of the formPacking:

max{|x | : Ax ≤ 1, x ≥ 0}Covering:

min{|x̂ | : Aᵀx̂ ≥ 1, x̂ ≥ 0}Note: Coefficients of xj ’s and bi ’s are one, for i = 1, 2, ..., rand j = 1, 2, ..., c .

whereI Elements of the constraint matrix A lie in [0, 1]r×c

I r and c is number of rows and column respectively

Linear Programs solver

I Our solver returnsI A (1− 2ε) - approximate feasible primal-dual pair x? and x̂?

I |x?| ≥ (1− 2ε)|x̂?| [Christos et al. [6]] (We will refer it by KY)I with probability at least 1− 1/(rc)I ε ∈ (0, 1]

I Time complexityI Our solver takes O( 1

ε2 n log n) at most time on single processor

Linear Programs solverI Performance Analysis of our LP solver

0 50 100 150 200 250 300 350 400 450 500

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

Time (Seconds)−−−−−−−−−−−>

n:

siz

e o

f m

atr

ix M

n× n

−−

−−

−−

−−

>

Time tradeoff between YK and GLPK LP solver

ε = 0.03 "KY"

ε = 0.05 "KY"

ε = 0.06 "KY"

ε = 0.07 "KY"

ε = 0.09 "KY"

ε = 0.1 "KY"

"GLPK"

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5100

150

200

250

300

350

400

450

500

550

600

Time (Seconds)−−−−−−−−−−−>

n:

siz

e o

f m

atr

ix M

n× n

−−

−−

−−

−−

>

Time tradeoff between YK and GLPK LP solver

ε = 0.03 "KY"

ε = 0.05 "KY"

ε = 0.06 "KY"

ε = 0.07 "KY"

ε = 0.09 "KY"

ε = 0.1 "KY"

"GLPK"

: Precision time tradeoff, Size (r = c) with respect to CPU timetaken by KY algorithm and GLPK

Semi-Definite Programs

I Semidefinite programming (SDP) solves the followingproblem:

min A0 • Xsubject to Aj • X ≥ bj for j = 1, 2, ...,m

X � 0.

WhereI X ∈ Rn×n is a matrix of variablesI A1,A2, ...,Am ∈ Rn×n

I For n × n matrices A and B, A • B =∑

ij AijBij

I A � 0 is notation for A is PSD

Semi-Definite Programs: feasibility engine algorithm

On input ( An×nj , bj for j = 1, 2, ...,m, ε, R (

∑i Xii ≤ R))

Result: Feasible/infeasible solution PSD matrix XInitialize wj s.t.

∑j wj = 1, iter = n log n

ε2, β = min{ ε2 , 0.01}.

while T < iter doUpdate T : T = T + 1.

Compute C : C =∑m

j=1 w(t)j (Aj −

bjR I ).

if C is negative definite report solution is infeasible andstop, else compute the largest eigenvector of C say V .Compute Xt : Xt = V × V ᵀ

Update Wj : w(t+1)j = w

(t)j (1− β(Aj • Xt − bj))/S t , where

S t =∑m

j=1 w(t)j (1− β(Aj • Xt − bj)).

end

Compute X : X =∑T

t=1XtT and return X .algo1

Algorithm 1: Decision making algorithm for primal only SDPusing multiplicative weights update method

Semi-Definite Programs: Algorithm correctness

Objective, the SDP problem P is to be solved.

1. Solve P using the feasibility engine. Let output be Kconstraints feasible

2. Solve P using SDPA, SeDuMi and etc. with exactly Kconstraints and an A0 objective. Let output be α?.

3. Give P return back to the feasibility engine with exactly Kconstraints + A0 • X ≥ α? ± δ, for small δ > 0

feasibility engine should return K constraints feasible and

A0 • X ≥ α? − δ : feasible (satisfiable) and

A0 • X ≥ α? + δ : infeasible (not satisfiable).

Semi-Definite Programs: Example

Let n = 2,m = 3, A0 =

[−11 0

0 23

], A1 =

[10 44 0

],

A2 =

[0 00 −8

], A3 =

[0 −8−8 −2

], b =

−48−8−20

Step 1: feasibility engine output K = 3Step 2: SDPA, SeDuMi etc. output

objValPrimal = +2.3000000262935881e + 01

objValDual = +2.2999999846605416e + 01

Step 3: for n = 2,m = 3 + 1 = K + 1, ε = 0.01, R ≥ 11, (sinceTr(X ) ≤ R, δ = 0.01 and α? = 23.0,feasibility engine output

A0 • X ≥ α? + δ : infeasible (not satisfiable)

A0 • X ≥ α? − δ : feasible (satisfiable)

Semi-Definite Programs

I Performance Analysis of our SDP solver

100 200 300 400 500 600 700 800 9000

100

200

300

400

500

600

700

800

900

1000

n: Problem size−−−−−−−−>

CP

U T

ime (

seco

nd

s)−

−−

−−

−−

−−

−−

>

CPU Time Tradeoff of AHK Algorithm versus SeDuMi with ε = 0.1 (Gaps)

m = 100 "AHK"

m = 200 "AHK"

m = 300 "AHK"

m = 100 "SeDuMi"

: Precision time tradeoff, CPU time taken by algorithm 1 and Se-DuMi with respect to n (problem size)

Example: Lovasz ϑ function

I Consider following SDP problems from SDPLIB 1.2 [2]theta1, theta2 and theta3

I Given a graph G = (V ,E ), the lovasz ϑ-function ϑ(G ) on Gis the optimal value of the following SDP

max J • XI • X = 1

∀{i , j} ∈ E : Eij • X = 0

X � 0.

WhereI I is identity matrixI J is matrix in which every entry is 1I For each edge {i , j} ∈ E ,Eij is the matrix in which both the

(i , j)-th and (j , i)-th entries are 1, and every other entry is 0

Example: Lovasz ϑ function

SDPLIB solves:

SDP

P : min A0 • Xsubject to Aj • X = bj , j = 1, 2, ...,m

X � 0

Relaxation of SDPLib SDPs:

min A0 • Xsubject to Aj • X ≥ bj − ε

−Aj • X ≥ −bj − ε for j = 1, 2, ...,m and ε > 0

X � 0

Tr(X ) ≤ R.

Results

SDP m n Opt ε R δ α? secs T

theta1 104 50 23 0.1 1 0.1 0.40 0.076 10.01 1 0.01 10.91 0.4219 7

theta2 498 100 32.879 0.1 1 0.1 0.40 0.7 10.01 1 0.01 15.44 3.332 7

theta3 1106 150 42.17 0.1 1 0.1 0.40 2.17 10.01 1 0.01 15.9 12.89 9

Table: SDP lower bound for Lovasz ϑ functions

SDP m n Opt ε R δ α? secs T

theta1 104 50 23 0.1 1 0.1 25.4 0.0375 10.01 10 0.01 25.4 0.121 1

theta2 498 100 32.879 0.1 1 0.1 50.2 0.76 10.01 1 0.01 40 0.39 1

theta3 1106 150 42.17 0.1 1 0.1 75.1 1.64 10.01 1 0.01 60 1.56 1

Table: SDP upper bound for Lovasz ϑ functions

Conclusions

I In every iteration interior point methods compute theCholesky decomposition of a PSD matrix. This computationtakes O(n3) time. The top eigenvector of a matrix can becomputed much more efficiently, which is done in algorithm 1.This is where our implementation gets an edge over interiorpoint methods.

I Second advantage of our implementation is that the Choleskydecomposition of the final solution is calculated automatically(X =

∑Tt=1

XtT ).

I We have presented our experimental results on solving relaxedSDPs Lovasz ϑ functions and presented how we approachtowards the optimum one. Our results are good in quality andefficient in compare to existing one.

Future Works

I Presently our implementation runs sequentially, which maynot be good for considerable large combinatorial optimizationproblems.

I In future I am willing to implement approximation algorithmsin a distributed setup. This will give us faster results incompare to existing one. Distributed setup implementationwill also have good quality of results as we have know.

I There has been considerable recent research building on thiswork, developing fast and parallel approximate algorithms toapproximate solutions to packing-covering linear as well assemi-definite programs.

I This research direction seem to be more interesting and canhave reasonable payoff and hence it can not be ignored.

S. Arora and S. Kale.AK07A combinatorial, primal-dual approach to semidefiniteprograms.In Proceedings of the Thirty-Ninth Annual ACM Symposiumon Theory of Computing, pages 227–236, 2007.

Brian Borchers.B99Sdplib 1.2, a library of semidefinite programming testproblems.11(1):683–690, 1999.

Rajiv Raman Dilys Thomas and Ajay Shankar Bidyarthy.RDA12Fast approximations to solve packing-covering lps andconstant-sum games via multiplicative-weights technique.In proceeding of the International Symposium onCombinatorial Optimization - CO 2012, September 17-192012.

S. Arora E. Hazan and S. Kale.AHK05Fast algorithms for approximate semidefinite programmingusing the multiplicative weights update method.

In Proceedings of the 46th Annual IEEE Symposium onFoundations of Computer Science, pages 339–348, 2005.

M.D. Grigoriadis and L.G. Khachiyan.GK95A sublinear-time randomized apprximation algorithm formatrix games.Operations Research Letters, 18(2):53–58, 1995.

Christos Koufogiannakis and Neal E. Young.KY07Beating simplex for fractional packing and covering linearprograms.The 48th IEEE Symposium on Foundation of ComputerScience, 2007.

Jos. F. Strum.SeDuMiSedumi ver. 1.3.Optimization Methods and Software.