Convex Optimization - University of Rochesterjliu/readinggroup/cvx-ch3.pdf · if f is concave then...

Convex OptimizationChapter 3

Lam Tran

October 2, 2015

Lam Tran Convex Optimization

Table of Contents


Convex Functions

Let f : Rn → R be a convex function if ∀x , y ∈ dom f and0 ≤ θ ≤ 1 such that

f (θx + (1− θ)y) ≤ θf (x) + (1− θ)f (y).

It is also known as the Jensen’s inequality.

if f is concave then −f is convex.Lam Tran Convex Optimization

Strictly Convex Functions

Let f : Rn → R be a convex function if ∀x , y ∈ dom f and0 < θ < 1 such that

f (θx + (1− θ)y) < θf (x) + (1− θ)f (y).


Extended Convex Functions

Let f̂ : Rn → R ∪ {∞} be an extended convex function if∀x , y ∈ dom f and 0 ≤ θ ≤ 1 such that

f̂ (x) =

{f (x) x ∈ dom f∞ otherwise

andf̂ (θx + (1− θ)y) ≤ θf̂ (x) + (1− θ)f̂ (y).


Example of Convex Functions


First-order conditions

f is differentiable then f is convex iff dom f is convex and

f (y) > f (x) +∇f (x)T (y − x)

hold ∀x , y ∈ dom f and

∇f (x) =(∂f (x)∂x1

, . . . ,∂f (x)∂xn

).


Second-order conditions

f is twice differentiable then f is convex if

∇2f (x) ≥ 0

hold ∀x ∈ dom f and each element is defined as

∇2f (x)ij =∂f (x)∂xixj

.


Sublevel Sets

The a-sublevel set of a function f : Rn → R is defined as

Ca = {x ∈ dom f | f (x) ≤ a}.

f is a convex function if ∀x , y ∈ Ca and 0 ≤ θ ≤ 1 such thatf (x) ≤ a and f (y) ≤ a then

f (θx + (1− θ)y) < θf (x) + (1− θ)f (y).

hold ∀x , y .


Epigraph

The Ephigraph is defined as

epi f = {(x , t)|x ∈ dom f , f (x) ≤ t}

From a-sublevel sets, epigraph is convex.


Preserve Convexity Operations: Nonnegativeweighted sums

For i = 1, . . . ,n such that wi ≥ 0 and fi is a convex function.Then the function

f (x) =n∑

i=1

wi fi(x).

is convex.Proof: Let x1, x2 ∈ dom g, 0 ≤ θ ≤ 1, and θ̂ = 1− θ. We have

g(θx1 + θ̂x2) =n∑

i=1

wi fi(θx1 + θ̂x2) ≤n∑

i=1

wi

[θfi(θx1) + θ̂fi(x2)

]=

n∑i=1

wiθfi(θx1) +n∑

i=1

wi θ̂fi(x2)

= θg(x1) + θ̂g(x2).


Preserve Convexity Operations: Composition with anaffine mapping

Let x ∈ dom g the affine mapping function is defined as

g(x) = f (Ax + b).

If f is convex function then g is a convex function..Proof: Let x1, x2 ∈ dom g and Ax + b ∈ dom f , 0 ≤ θ ≤ 1, andθ̂ = 1− θ. We have

g(θx1 + θ̂x2) = f (A(θx1 + θ̂x2) + b)

= f (Aθx1 + Aθ̂x2 + b)

= f (θ(Ax1 + b1) + θ̂(Ax2 + b2))

≤ θf (Ax1 + b1) + θ̂f (Ax2 + b2)

= g(x1) + g(x2),

where b = θb1 + θ̂b2.Lam Tran Convex Optimization

Preserve Convexity Operations: Pointwise maximumand supremum

The perspective function: Let x ∈ dom f and f1, f2 be convexfunctions such as

f (x) = max{f1(x), f2(x)}

then f is a convex function.Proof: Let x1, x2 ∈ dom f , 0 ≤ θ ≤ 1, and θ̂ = 1− θ. We have

f (θx1 + θ̂x2) = max(f1(θx1 + θ̂x2), f2(θx1 + θ̂x2))

= max(θf1(x1) + θ̂f (x2), θf2(x1) + θ̂f2(x2))

= max(θf1(x1), θf2(x1)) + max(θ̂f1(x2) + θ̂f2(x2))

= θf (x1) + θ̂f (x2).


Preserve Convexity Operations: Minimization

If f is convex in (x , y), and C is a convex nonempty setthen the function

h(x) = infy∈C

f (x , y)

is a convex function. The domain of g is

dom g = {x |(x , y) ∈ dom f for some y ∈ C}.


Preserve Convexity Operations: Minimization

Proof: Let x1, x2 ∈ dom h, y1, y2 ∈ C, and 0 ≤ θ ≤ 1, andθ̂ = 1− θ. We have

h(θx1 + θ̂x2) = infy∈C

f (θx1 + θ̂x2, y)

≤ f (θx1 + θ̂x2, θy1 + θ̂y2)

≤ θf (x1, y1) + θ̂f (x2, y2)

≤ θg(x1) + θ̂g(x2) + ε

where ε > 0.


Preserve Convexity Operations: Composition

Let x ∈ dom h be a composition function

h(x) = g(f (x)).

1 g is convex & non-decreasing and f is convex, then h isconvex.

2 g is convex & non-increasing and f is concave, then h isconvex.

3 g is concave & non-increasing and f is convex, then h isconcave.

4 g is concave & non-decreasing and f is concave, then h isconcave.

Proof: Let x1, x2 ∈ dom h, 0 ≤ θ ≤ 1, and θ̂ = 1− θ

h(θx1 + θ̂x2) = g(h(θx1 + θ̂x2)) = g(θh(x1) + θ̂h(x2))

= θg(h(x1)) + θ̂g(h(x2)).


Preserve Convexity Operations: Perspective

The perspective function: Let x , t ∈ dom g

g(x , t) = t × f (x/t).

The perspective function g is convex if f is convex.Proof: Let x1, x2, t1, t2 ∈ dom g, 0 ≤ θ ≤ 1, and θ̂ = 1− θ. Wehave

g(θx1 + θ̂x2, θt1 + θ̂t2) =

= (θt1 + θ̂t2)× f

(θx1 + θ̂x2

θt1 + θ̂t2

)

= (θt1 + θ̂t2)× f

(θ t1

t1x1 + θ̂ t2t2x2

(θt1 + θ̂t2)

)

≤ (θt1 + θ̂t2)[

θ1t1θt1 + θ̂t2

f (θx1/t1) +θ2t2

θt1 + θ̂t2f(θ̂x2/t2

)]= t1g(x1, t) + t2g(x2, t2).


Practical Methods to Establish Convexity of a Function

Verify definition (often simplified by restricting to a line).Twice differentiable functions, show

∇2f (x) ≥ 0.

Show that f is obtained from simple convex functions byoperations that preserve convexity.


The Conjugate Function

Let f : Rn → R. The conjugate function of f is f ∗ : Rn → R is

f ∗(x) = supx∈ dom f

(y tx − f (x))

f ∗ is convex (even if f is not convex)


Negative logarithm

f (x) = − log(x)

The deriviate of f (x) w.r.t x is

∂

∂xyx − f (x) = y +

1x= 0 =⇒ x∗ = − 1

y .

We have

yx∗ − f (x∗) = −yy+ log

(−1

y

)= −1− log(−y).

f ∗(y) = supx>0

(xy + log(x)) =

{−1− log(−y) y < 0∞ otherwise


Strictly convex quadratic

f (x) = (1/2)xT Qx

with Q ∈ Sn++. The deriviate of f (x) w.r.t x is

∂

∂xyT x − f (x) = yT − xT Q = 0 =⇒ x∗ = Q−1y .

Plug x∗ into x , we get

yT x∗ + f (x∗) = yT Q−1y − 12

yT Q−1y =12

yT Q−1y .

Thus, we have

f ∗(y) = supx

(y tx − (1/2)x tQ) = 1/2yT Q−1y


Quasiconvex Functions

A function f : Rn → R is quasiconvex if dom f is convex and thesublevel sets

Sa = {x ∈ dom f | f (x) ≤ a}are convex for all a.

-f is quasiconcave.f is quasilinear then every level set {x |f (x) = a} is convex.Any monotonically increasing (decreasing) f is quasilinear.

sublevel of α is Sα = [a,b] and sublevel of β Sβ = [−∞, c].Lam Tran Convex Optimization

Quasiconvex Proof

If f is convex then f is quasiconvex.Proof: Let f be a convex function such as

f (θx + (1− θ)y) ≤ θf (x) + (1− x)f (y)

and the set Sa = {x ∈ dom f |f (x) ≤ a} are convex for all a.Suppose that 0 ≤ θ ≤ 1 and x , y ∈ Sa implies that f (x) ≤ a andf (y) ≤ a. We have

f (θx + (1− θ)y) ≤ θf (x) + (1− θ)f (y)≤ θa + (1− θ)a = a.

Thus, θa + (1− θ)a ∈ Sa and f is quasiconvex.


Quasiconcave Proof

If f is convex then f is quasiconcave.Proof: Let f be a concave function such as

f (θx + (1− θ)y) ≥ θf (x) + (1− x)f (y)

and the set Sa = {x ∈ dom f |f (x) ≥ a} are convex for all a.Suppose that 0 ≤ θ ≤ 1 and x , y ∈ Sa implies that f (x) ≥ a andf (y) ≥ a. We have

f (θx + (1− θ)y) ≥ θf (x) + (1− θ)f (y)≥ θa + (1− θ)a = a.

Thus, θa + (1− θ)a ∈ Sa and f is quasiconcave.


Quasiconvex and Quasiconcave Function Example

Let f : R → R and dom f ∈ R++ defined as

f (x) = log(x)

The 2nd deriviate is

f ′′(x) = − 1x2 < 0

The function f is concave implies that it is quasiconcave (proofnext page).



The function f is quasiconcave if the set

sa = {x ∈ R++| log(x) ≥ a}

is convex for all a. Let x , y ∈ Sa and 0 ≤ θ ≤ 1, implies thatlog(x) ≥ a and log(y) ≥ a. Then we have we have

log(θx + (1− θ)y) ≥ θ log(x) + (1− θ) log(y)≥ θa + (1− θ)a = a.

It implies thatlog(θx + (1− θ)y) ≥ a.

Thus, we have θx + (1− θ)y ∈ Sa and log(x) and the functionis quasiconcave.



The function f is quasiconvex if the set

sa = {x ∈ R++| log(x) ≤ a}

is convex for all a. Let x , y ∈ Sa and 0 ≤ θ ≤ 1, then we havelog(x) ≤ a =⇒ x ≤ exp(a) so as y ≤ exp(a).

θx + (1− θ)y ≤ θ exp(a) + (1− θ)exp(a) = exp(a).

Take the log of both side and we get

log(θx + (1− θ)y) ≤ a.

Thus, we have θx + (1− θ)y ∈ Sa and log(x) is quasiconvex.The above showed that the function log(x) is quasilinear.



Let f : R2 → R and dom f ∈ R+ defined as

f (x1, x2) = x1x2

The Hessian Matrix is

∇f ′′(x1, x2) =

[0 11 0

]

and the eigenvector is[−11

]. The function f is not convex nor

concave. But f is quasiconcave and quasiconvex.



Proof: Sa = {x ∈ R2+|x1x2 ≥ a}, x , y ∈ Sa, 0 ≤ θ ≤ 1 θ̂ = 1− θ.

(θx1 + θ̂x2)(θy1 + θ̂y2) = θ2x1y1 + θ̂2x2y2 + θθ̂(x1y2 + x2y1)

= θ2x1y1 + θ̂2x2y2 + θθ̂

(x1

x2x2y2 +

x2

x1x1y1

)≥ θ2a + θ̂2a + θθ̂

(x1

x2a +

x2

x1a)

= a[θ2 + θ̂2 + θθ̂

(x1

x2+

x2

x1+ 2− 2

√x1

x2

√x2

x1

)]= a

[θ2 + θ̂2 + θθ̂

{(√x1

x2−√

x2

x1

)2

+ 2

}]≥ a

[θ2 + θ̂2 + 2θθ̂

]≥ a(θ2 + θ2 − 2θ + 1− 2θ2 + 2θ) = a. = a.

Thus, we have θx + (1− θ)y ∈ Sa and Sa is a convex for all a.The function xy is quasiconcave.



Proof: Let Sa = {x ∈ R2+|x1x2 ≤ a}, x , y ∈ Sa, 0 ≤ θ ≤ 1, and

θ̂ = 1− θ.

(θx1 + θ̂x2)(θy1 + θ̂y2) = θ2x1y1 + θ̂2x2y2 + θθ̂(x1y2 + x2y1)

≤ θ2x1y1 + θ̂2x2y2

≤ θx1y1 + θ̂x2y2

≤ θa + (1− θ)a = a.

Thus, we have θx + (1− θ)y ∈ Sa and Sa is a convex for all a.The function xy is quasiconvex. Since f is both quasiconvexand quasiconcave, f is quasilinear.



A function f is quasiconvex iff the dom f is convex and for anyx , y ∈ dom f and 0 ≤ θ ≤ 1 such that

f (θx + (1− θ)y) ≤ max{f (x), f (y)}.

It is known as the Jensens inequality for quasiconvex functions.

The value of the function on a segment does not exceed themaximum of its values at the endpoints.



A function f is quasiconvex, that is Sa = {x |f (x) ≤ a} is convexfor all a ∈ R =⇒ f (θx + (1− θ)y) ≤ max{f (x), f (y)}.Proof: WLOG let max(f (x), f (y)) = f (x) = a. Let x , y ∈ Sa suchthat

f (x) ≤ a and f (y) ≤ a

and let 0 ≤ θ ≤ 1. Since Sa is convex, we haveθx + (1− θ)y ∈ Sa, thus we have

f (θx + (1− θ)y) ≤ a = max(f (x), f (y)).

Thus, f is quasiconvex.



f(θx + (1− θ)y) ≤ max(f (x), f (y)) =⇒ Sa = {x |f (x) ≤ a} isconvex for al a ∈ R.

Proof: WLOG Let max(f (x), f (y)) = f (x) = a. Let x , y ∈ Sa and0 ≤ θ ≤ 1. We have f (x) ≤ a and f (y) ≤ a. Since

f (θx + (1− θ)y) ≤ max(f (x), f (y)) = a,

we have f (θx + (1− θ)y) ≤ a. This implies thatθx + (1− θ)y ∈ Sa and Sa is convex.

Quasiconcave can be proved in a similar fashion.



The function f : R→ R is quasiconvex, if and only if at least oneof the following conditions holds

f is nondecreasingf is nonincreasingThere exist a point c ∈ dom f such that for t ≤ c (andt ∈ dom f ), f is nonincreasing, and for t ≥ c (and t ∈ domf ), f is nondecreasing.


Differentiable Quasiconvex Functions

First-order conditions:Suppose f : Rn → R is differentiable. Then f is quasiconvex iffdom f is convex and for all x , y ∈ dom f

f (y) ≤ f (x)⇒ ∇f (x)t(y − x) ≤ 0. (1)



Proof: We assume that f (y) ≤ f (x)⇒ f ′(x)t(y − x) ≤ 0 is true.Let the function f : Rn → R, x , y ∈ dom f such that y > x andf (y) ≤ f (x). Let z ∈ [x , y ] and f (z) ≤ f (x).

Suppose for contradiction that f (z) ≥ f (x), since f isdifferentiable, we have f ′(z) < 0. This implies that

f (x) ≤ f (z) =⇒ f ′(z)(x − z) ≤ 0.

Since (x − z) ≤ 0 and f ′(z) < 0, which contradict the (1).



Need to show the other direction as well.

Proof: We assume that f (y) ≤ f (x)⇒ f ′(x)t(y − x) ≤ 0 is true.Let the function f : Rn → R, x , y ∈ dom f such that y < x andf (y) ≤ f (x). Let z ∈ [y , x ] and f (z) ≤ f (x).

Suppose for contradiction that f (z) ≥ f (x), since f isdifferentiable, we have f ′(z) > 0. This implies that

f (x) ≤ f (z) =⇒ f ′(z)(x − z) ≤ 0.

Since (x − z) ≥ 0 and f ′(z) > 0, which contradict the (1). Thus,we have

f (y) ≤ f (x)⇒ ∇f (x)t(y − x) ≤ 0.



Second-order conditions:If f is quasiconvex, then for all x ∈ dom f , and all y ∈ Rn , wehave

y t∇f (x) = 0⇒ y t∇2f (x)y ≥ 0.


Preserve Quasiconvex Operations

Scalar: If f (x) is quasiconvex and w > 0, theng(x) = wf (x) is also quasiconvex.

Proof: Let Sa = {x |g(x) ≤ a}, x1, x2 ∈ Sa, and 0 ≤ θ ≤ 1.

g(θx + (1− θ)y) = wf (θx + (1− θ)y) ≤ wθf (x) + w(1− θ)f (y)≤ θa + (1− θ)a = a.



Pointwise supremum: if f1 and f2 are quasiconvex then

g(x) = supi∈{1,2}

(fi(x))

is quasiconvex.


g(θx + (1− θ)y) = supi∈{1,2}

(fi(θx + (1− θ)y))

≤ supi∈{1,2}

(θfi(x) + (1− θ)fi(y))

≤ supi∈{1,2}

(θa + (1− θ)a) = a



Pointwise supremum: if f1 and f2 are quasiconvex then

g(x) = supi∈{1,2}

(fi(x))

is quasiconvex.



Compositiong(x) = h(f (x))

is quasiconvex if f : Rn → R is quasiconvex and h : R → Ris nondecreasing.


g(f (θx + (1− θ)y)) = h(f (θx + (1− θ)y))≤ h(θf (x) + (1− θ)f (y))≤ θh(f (x)) + (1− θ)h(f (y)) ≤ a.

Thus, Sa is convex and g(x) is quasiconvex.



Minimization If f (x , y) is quasiconvex jointly in x and y andC is a convex set, then the function

g(x) = infy∈C

f (x , y)

is quasiconvex.

Show Sa = {x |g(x) ≤ a} is convex. Let x1, x2 ∈ Sa, y1, y2 ∈ Cand ε > 0 such that f (x1, y1) ≤ a + ε and f (x2, y2) ≤ a + ε.Since f is quasiconvex, we have

f (θx1 + (1− θ)x2, θy1 + (1− θ)y2) ≤ a + ε

Thus, g(θx1 + (1− θ)x2) ≤ a and Sa is convex.


Log-concave and Log-convex Functions

A log-concave function f : Rn → R if f (x) > 0 for all x ∈ dom fand log f (x) is concave. For x , y ∈ dom f and 0 ≤ θ ≤ 1 then

f (θx + (1− θ)y) = f (x)θf (y)1−θ.


Log-concave and Log-convex Functions


Convex Optimization - University of Rochesterjliu/readinggroup/cvx-ch3.pdf · if f is concave then...

Documents

Transcript of Convex Optimization - University of Rochesterjliu/readinggroup/cvx-ch3.pdf · if f is concave then...