Lecture 3 Convex Functions - University Of Illinoisangelia/L3_convfunc.pdfLecture 3 Convex Functions...

21
Lecture 3 Convex Functions September 2, 2008

Transcript of Lecture 3 Convex Functions - University Of Illinoisangelia/L3_convfunc.pdfLecture 3 Convex Functions...

Lecture 3

Convex Functions

September 2, 2008

Lecture 3

Outline

• Convex Functions

• Examples

• Verifying Convexity of a Function

• Operations on Functions Preserving Convexity

Convex Optimization 1

Lecture 3

Convex Functions

Informally: f is convex when for every segment [x1, x2], as xα =

αx1+(1−α)x2 varies over the line segment [x1, x2], the points (xα, f(xα))

lie below the segment connecting (x1, f(x1)) and (x2, f(x2))

Let f be a function from Rn to R, f : Rn → RThe domain of f is a set in Rn defined by

dom(f) = {x ∈ Rn | f(x) is well defined (finite)}Def. A function f is convex if

(1) Its domain dom(f) is a convex set in Rn and

(2) For all x1, x2 ∈ dom(f) and α ∈ (0,1)

f(αx1 + (1− α)x2) ≤ αf(x1) + (1− α)f(x2)

Convex Optimization 2

Lecture 3

More on Convex Function

Def. A function f is strictly convex when dom(f) is convex and

f(αx1 + (1− α)x2) < αf(x1) + (1− α)f(x2)

for all x1, x2 ∈ dom(f) and α ∈ (0,1)

Def. A function f is concave when −f is convex, i.e.,

(1) Its domain dom(f) is a convex set in Rn and

(2) For all x1, x2 ∈ dom(f) and α ∈ (0,1)

f(αx1 + (1− α)x2) ≥ αf(x1) + (1− α)f(x2)

Def. A function f is strictly concave when −f is strictly convex

Convex Optimization 3

Lecture 3

Examples on R

Convex:

• Affine: ax + b over R for any a, b ∈ R• Exponential: eax over R for any a ∈ R• Power: xp over (0,+∞) for p ≥ 1 or p ≤ 0

• Powers of absolute value: |x|p over R for p ≥ 1

• Negative entropy: x lnx over (0,+∞)

Concave:

• Affine: ax + b over R for any a, b ∈ R• Powers: xp over (0,+∞) for 0 ≤ p ≤ 1

• Logarithm: lnx over (0,+∞)

Convex Optimization 4

Lecture 3

Examples: Affine Functions and Norms

• Affine functions are both convex and concave

• Norms are convex

Examples on Rn

• Affine function f(x) = a′x + b with a ∈ Rn and b ∈ R

• Euclidean, l1, and l∞ norms

• General lp norms

‖x‖p =

(n∑

i=1

|xi|p)1/p

for p ≥ 1

Convex Optimization 5

Lecture 3

Examples on Rm×n

The space Rm×n is the space of m× n matrices

• Affine function

f(X) = tr(ATX) + b =m∑

i=1

n∑j=1

aijxij + b

• Spectral (maximum singular value) norm

f(X) = ‖X‖2 = σmax(X) =√

λmax(XTX)

where λmax(A) denotes the maximum eigenvalue of a matrix A

Convex Optimization 6

Lecture 3

Verifying Convexity of a Function

We can verify that a given function f is convex by

• Using the definition

• Applying some special criteria

• Second-order conditions

• First-order conditions

• Reduction to a scalar function

• Showing that f is obtained through operations preserving convexity

Convex Optimization 7

Lecture 3

Second-Order Conditions

Let f be twice differentiable and let dom(f) = Rn [in general, it is

required that dom(f) is open]

The Hessian ∇2f(x) is a symmetric n × n matrix whose entries are the

second-order partial derivatives of f at x:

[∇2f(x)

]ij

=∂2f(x)

∂xi∂xj

for i, j = 1, . . . , n

2nd-order conditions: For a twice differentiable f with convex domain

• f is convex if and only if

∇2f(x) � 0 for all x ∈ dom(f)

• f is strictly convex if

∇2f(x) � 0 for all x ∈ dom(f)

Convex Optimization 8

Lecture 3

Examples

Quadratic function: f(x) = (1/2)x′Px + q′x + r with a symmetric

n× n matrix P

∇f(x) = Px + q, ∇2f(x) = P

Convex for P � 0

Least-squares objective: f(x) = ‖Ax− b‖2 with an m× n matrix A

∇f(x) = 2AT(Ax− b), ∇2f(x) = 2ATA

Convex for any A

Quadratic-over-linear: f(x, y) = x2/y

∇2f(x, y) = 2y3

[y

−x

] [y

−x

]T

� 0

Convex for y > 0

Convex Optimization 9

Lecture 3

Verifying Convexity of a Function

We can verify that a given function f is convex by

• Using the definition

• Applying some special criteria

• Second-order conditions

• First-order conditions

• Reduction to a scalar function

• Showing that f is obtained through operations preserving convexity

Convex Optimization 10

Lecture 3

First-Order Condition

f is differentiable if dom(f) is open and the gradient

∇f(x) =

(∂f(x)

∂x1,∂f(x)

∂x2, . . . ,

∂f(x)

∂xn

)

exists at each x ∈ domf

1st-order condition: differentiable f is convex if and only if its domain isconvex and

f(x) +∇f(x)T(z − x) ≤ f(z) for all x, z ∈ dom(f)

A first order approximation is a global underestimate of f

Very important property used in algorithm designs and performance analysis

Convex Optimization 11

Lecture 3

Restriction of a convex function to a line

f is convex if and only if domf is convex and the function g : R → R,

g(t) = f(x + tv), domg = {t | x + tv ∈ dom(f)}

is convex (in t) for any x ∈ domf , v ∈ Rn

Checking convexity of multivariable functions can be done by checkingconvexity of functions of one variableExample f : Sn → R with f(X) = − ln detX, domf = Sn

++

g(t) = − ln det(X + tV ) = − ln detX − ln det(I + tX−1/2V X−1/2)

= − ln detX −n∑

i=1

ln(1 + tλi)

where λi are the eigenvalues of X−1/2V X−1/2

g is convex in t (for any choice of V and any X � 0); hence f is concave

Convex Optimization 12

Lecture 3

Operations Preserving Convexity

• Positive Scaling

• Sum

• Composition with affine function

• Pointwise maximum and supremum

• Composition

• Minimization

Convex Optimization 13

Lecture 3

Scaling, Sum, & Composition with Affine Function

Positive multiple For a convex f and λ > 0, the function λf is convex

Sum: For convex f1 and f2, the sum f1 + f2 is convex

(extends to infinite sums, integrals)

Composition with affine function: For a convex f and affine g [i.e.,

g(x) = Ax + b], the composition f ◦ g is convex, where

(f ◦ g)(x) = f(Ax + b)

Examples• Log-barrier for linear inequalities

f(x) = −m∑

i=1

ln(bi−aTi x), domf = {x | aT

i x < bi, i = 1, . . . , m}

• (Any) Norm of affine function: f(x) = ‖Ax + b‖

Convex Optimization 14

Lecture 3

Pointwise maximum

For convex functions f1, . . . , fm, the pointwise-max functionF (x) = max {f1(x), . . . , fm(x)}

is convex (What is domain of F ?)

Examples

• Piecewise-linear function: f(x) = maxi=1,...,m(aTi x + bi) is convex

• Sum of r largest components of a vector x ∈ Rn:

f(x) = x[1] + x[2] + · · ·+ x[r]

is convex (x[i] is i-th largest component of x)

f(x) = max(i1,...,ir)∈Ir

{xi1 + xi2 + · · ·+ xir}

Ir = {(i1, . . . , ir) | i1 < . . . < ir, ij ∈ {1, . . . , m}, j = 1, . . . , n}

Convex Optimization 15

Lecture 3

Pointwise Supremum

Let A ⊆ Rp and f : Rn × Rp → R. Let f(x, z) be convex in x for

each z ∈ A. Then, the supremum function over the set A is convex:

g(x) = supz∈A f(x, z)

Examples

• Set support function is convex for a set C ⊂ Rn,

SC : Rn → R, SC(x) = supz∈C zTx

• Set farthest-distance function is convex for a set C ⊂ Rn,

f : Rn → R, f(x) = supz∈C ‖x− z‖

• Maximum eigenvalue function of a symmetric matrix is convexλmax : Sn → R, λmax(X) = sup‖z‖=1 zTXz

Convex Optimization 16

Lecture 3

Composition with Scalar Functions

Composition of g : Rn → R and h : R → R with dom(g) = Rn and

dom(h) = R:

f(x) = h(g(x))

f is convex if

(1) g is convex, h is nondecreasing and convex

(2) g is concave, h is nonincreasing and convex

Examples

• eg(x) is convex if g is convex

• 1g(x)

is convex if g is concave and positive

Convex Optimization 17

Lecture 3

Composition with Vector Functions

Composition of g : Rn → Rp and h : Rp → R with dom(g) = Rn and

dom(h) = Rp:

f(x) = h(g(x)) = h(g1(x), g2(x), . . . , gp(x))

f is convex if

(1) each gi is convex, h is convex and nondecreasing in each argument

(2) each gi is concave, h is convex and nonincreasing in each argument

Example

•∑m

i=1 egi(x) is convex if gi are convex

Convex Optimization 18

Lecture 3

Extended-Value Functions

A function f is an extended-value function if f : Rn → R ∪ {−∞,+∞}

Example: consider f(x) = infy≥0 xy for x ∈ R

Def. The epigraph of a function f over Rn is the following set in Rn+1:

epif = {(x, w) ∈ Rn+1 | x ∈ Rn, f(x) ≤ w}

General Convex Function Def. A function f is convex if its epigraph

epif is a convex set in Rn+1

This definition is equivalent to the one we have used so far (when reduced

to the function class we have considered thus far). How?

For an f with domain domf , we associate an extended-value function f̃

defined byf̃(x) =

f(x) if x ∈ domf

+∞ otherwise

domf is the projection of epif on Rn; convexity of f by letting w = f(x)

Convex Optimization 19

Lecture 3

Minimization

Let C ⊆ Rn × Rp be a nonempty convex set

Let f : Rn × Rp → R be a convex function [in (x, z) ∈ Rn × Rp]. Then

g(x) = infz∈C

f(x, z) is convex

Example

• Distance to a set: for a nonempty convex C ⊂ Rn,

dist(x, C) = infz∈C ‖x− z‖ is convex

Proof: Let x1, x2 ∈ Rn and α ∈ (0,1) be arbitrary. Let ε > 0 be arbitrarily

small. Then, there exist z1, z2 ∈ C such that f(x1, z1) ≤ g(x1) + ε and

f(x2, z2) ≤ g(x2) + ε. Consider f(αx1 + (1 − α)x2, αz1 + (1 − α)z2)

and use convexity of f and C.

Convex Optimization 20