September 12, 2005 Copyright©2001-5 Erik D. Demaine and Charles E. Leiserson L2.1 Introduction to...

September 12, 2005 Copyright©2001-5 Erik D. Demaine and Charles E. Leiserson L2.1

Introduction to Algorithms

LECTURE2

Asymptotic Notation

•O-, Ω-, and Θ-notation

Recurrences

•Substitution method

•Iterating the recurrence

•Recursion tree

•Master method

September 12, 2005

Copyright ?2001-5 Erik D. Demaine and Charles E. Leiserson L2.2

Asymptotic notation

O-notation (upper bounds):

We write f(n) = O(g(n)) if there Exist constants c>0, n0>0 suchThat 0 ≤f(n) ≤cg(n) for all n≥n0.


September 12, 2005


Asymptotic notation




EXAMPLE: 2n2= O(n3) (c= 1, n0= 2)

September 12, 2005


Asymptotic notation




EXAMPLE: 2n2= O(n3) (c= 1, n0= 2)

functions,Not values

September 12, 2005


Asymptotic notation




EXAMPLE: 2n2= O(n3) (c= 1, n0= 2)

functions,Not values

Funny, “one-way”equality

September 12, 2005


asymptotic upper bound

} )()(0s.t.,|)({))(( 00 nnncgnfncnfngO

n 0

n

cg(n)

f(n)f n O g n( ) ( ( ))

September 12, 2005


Ω-notation (lower bounds)

O-notation is an upper-bound notation. It makes no sense to say f(n) is at least O(n2).

September 12, 2005




Ω(g(n))= { f(n) :there exist constants c> 0, n0> 0such

that 0 ≤ cg(n) ≤ f(n) for all n≥n0}



September 12, 2005








EXAMPLE:

September 12, 2005


asymptotic lower bound

} )()(0 s.t. ,|)({))(( 00 nnnfncgncnfng Ω

n 0

n

cg(n)

f(n)f n g n( ) ( ( ))Ω

September 12, 2005


-notation (tight bounds)

September 12, 2005


-notation (tight bounds)

EXAMPLE:

September 12, 2005


• The definition of required every member of be asymptotically nonnegative.

n 0

n

c 2g(n)

c 1g(n)

f(n)f n g n( ) ( ( ))

Graphic Example of Θ

September 12, 2005


Theorem 3.1.

• For any two functions f(n) and g(n), if and only if and .

f n g n( ) ( ( ))f n O g n( ) ( ( ))f n g n( ) ( ( ))Ω

September 12, 2005


ο-notation and ω-notation

O-notation and Ω-notation are like ≤ and ≥. o-notation and ω-notation are like < and >.

ο(g(n))= { f(n) : for any constant c> 0 there is a constant n0> 0 such that 0 ≤ f(n) < cg(n)

for all n≥ n0}

ο(g(n))= { f(n) : for any constant c> 0 there is a constant n0> 0 such that 0 ≤ f(n) < cg(n)

for all n≥ n0}

2n2= o(n3) (n0= 2/c)EXAMPLE:

But 2n2 ≠ o(n2)

September 12, 2005


ο-notation and ω-notation

O-notation and Ω-notation are like ≤ and ≥. o-notation and ω-notation are like < and >.

(g(n))= { f(n) : for any constant c> 0 there is a constant n0> 0 such that 0 ≤ cg(n) < f(n)

for all n≥ n0}

(g(n))= { f(n) : for any constant c> 0 there is a constant n0> 0 such that 0 ≤ cg(n) < f(n)

for all n≥ n0}

EXAMPLE:

n2/2 =w (n), but n2 /2 ≠ w(n2)

September 12, 2005


Solving recurrences

• The analysis of merge sort from Lecture 1 required us to solve a recurrence.

• Recurrences are like solving integrals, differential equations, etc.

。 Learn a few tricks.

• Lecture 3: Applications of recurrences to divide-and-conquer algorithms.

September 12, 2005


Substitution method

The most general method:

1.Guess the form of the solution.

2.Verify by induction.

3.Solve for constants.

September 12, 2005


Substitution method

The most general method:

1.Guess the form of the solution.

2.Verify by induction.

3.Solve for constants.

EXAMPLE:

• [Assume that T(1) = Θ(1).]• Guess O(n3). (Prove O and Ωseparately.)• Assume that T(k) ≤ ck3 for k< n .• Prove T(n) ≤cn3 by induction.

T(n) = 4T(n/2) + n

September 12, 2005


Example of substitution

T(n) = 4T(n/2) + n ≤ 4c(n/2)3 + n = (c/2)n3 + n = cn3 –((c/2)n3 – n)← desired – residual ≤ cn3 ← desired

Whenever (c/2)n3 – n ≥ 0, for example,if c ≥ 2 and n ≥ 1.

residual

September 12, 2005


Example (continued)

•We must also handle the initial conditions, that is, ground the induction with base cases.

•Base: T(n) = Θ(1) for all n< n0, where n0is a suitable constant.

•For 1 ≤ n < n0, we have “Θ(1)” ≤ cn3, if we pick c big enough.

September 12, 2005


Example (continued)

•We must also handle the initial conditions, that is, ground the induction with base cases.

•Base: T(n) = Θ(1) for all n< n0, where n0is a suitable constant.

•For 1 ≤ n < n0, we have “Θ(1)” ≤ cn3, if we pick c big enough.

This bound is not tight!

September 12, 2005


A tighter upper bound?

We shall prove that T(n) = O(n2).

September 12, 2005




Assume that T(k) ≤ ck2 for k < n:

T(n)= 4T(n/2) + n ≤ 4c(n/2)2 + n = cn2 + n = O(n2)

September 12, 2005





T(n)= 4T(n/2) + n ≤ 4c(n/2)2 + n = cn2 + n = O (n2) Wrong! We must prove the I.H.

September 12, 2005





T(n)= 4T(n/2) + n ≤ 4c(n/2)2 + n = cn2 + n = O (n2) = cn2 – (- n) [ desired - residual] ≤ cn2 for no choice of c > 0. Lose!

Wrong! We must prove the I.H.

September 12, 2005


A tighter upper bound!

IDEA: Strengthen the inductive hypothesis.

• Subtract a low-order term.

Inductive hypothesis: T(k) ≤ c1k2 – c2k for k < n.

September 12, 2005






T(n)= 4T(n/2) + n = 4(c1(n/2)2–c2(n/2))+ n = c1n2–2c2n+ n = c1n2–c2n–(c2n–n) ≤ c1n2–c2n if c2≥1.

September 12, 2005






T(n)= 4T(n/2) + n

= 4(c1(n/2)2–c2(n/2))+ n

= c1n2–2c2n+ n

= c1n2–c2n–(c2n–n)

≤ c1n2–c2n if c2≥1.Pick c1 big enough to handle the initial conditions.

September 12, 2005


Recursion-tree method

• A recursion tree models the costs (time) of a recursive execution of an algorithm.

• The recursion-tree method can be unreliable, just like any method that uses ellipses (…).

• The recursion-tree method promotes intuition, however.

• The recursion tree method is good for generating guesses for the substitution method.

September 12, 2005


Example of recursion tree

Solve T(n) = T(n/4) + T(n/2)+ n2:

September 12, 2005



Solve T(n) = T(n/4) + T(n/2)+ n2:

T(n)

September 12, 2005



Solve T(n) = T(n/4) + T(n/2)+ n2:

n2

T(n/4) T(n/2)

September 12, 2005



September 12, 2005



geometric seriesgeometric series

September 12, 2005


The master method

The master method applies to recurrences of the form

T(n) = aT(n/b) + f(n) ,

where a≥1, b> 1, and f is asymptotically positive.

September 12, 2005


Three common cases

Compare f (n)with n logba:

1. f (n) = O (n logba–ε)for some constant ε> 0.

• f (n) grows polynomially slower than n logba

(by an nεfactor).

Solution: T(n) = (n logba)

September 12, 2005


Three common cases


1. f (n) = O (n logba–ε)for some constant ε> 0.

• f (n) grows polynomially slower than n logba

(by an nεfactor).

Solution: T(n) = (n logba)

2. f (n) = (n logbalgkn)for some constant k≥0.

• f (n) and nl ogba grow at similar rates.

Solution: T(n) = (n logbalgk+1n).

September 12, 2005


Three common cases (cont.)


3. f (n) = Ω(nlogba+ ε)for some constant ε> 0.

• f (n) grows polynomially faster than n logba (by an nεfactor),

and f (n) satisfies the regularity condition that

af(n/b) ≤ cf(n) for some constant c< 1.

Solution: T(n) = (f(n)).

September 12, 2005


Examples

EX. T(n) = 4T(n/2) + n

a =4, b= 2 ⇒n logba =n2; f(n) = n.

CASE1: f(n) = O(n2–ε)for ε= 1.

∴T(n) = (n2).

September 12, 2005


Examples

EX. T(n) = 4T(n/2) + n

a =4, b= 2 ⇒n logba =n2; f(n) = n.

CASE1: f(n) = O(n2–ε)for ε= 1.

∴T(n) = (n2).

EX.T(n) = 4T(n/2) + n2

a =4, b= 2 ⇒ n logba =n2; f(n) = n2.

CASE2: f(n) = (n2lg0n), that is, k = 0.

∴T(n) = (n2lgn).

September 12, 2005


Examples

EX.T(n) = 4T(n/2) + n3

a =4, b= 2 ⇒n logba=n2; f(n) = n3.

CASE3: f(n) = Ω(n2+ ε)for ε= 1

and 4(n/2)3≤cn3 (reg. cond.) for c= 1/2.

∴T(n) = (n3).

September 12, 2005


Examples

EX.T(n) = 4T(n/2) + n3

a =4, b= 2 ⇒n logba=n2; f(n) = n3. CASE3: f(n) = Ω(n2+ ε)for ε= 1 and 4(n/2)3≤cn3 (reg. cond.) for c= 1/2. ∴T(n) = (n3).

EX.T(n) = 4T(n/2) + n2/lgn a =4, b= 2 ⇒ n logba =n2; f(n) = n2/lgn. Master method does not apply. In particular, for every constant ε> 0, we have nε= ω(lgn).

September 12, 2005


Idea of master theorem

Recursion tree:

September 12, 2005



Recursion tree:

September 12, 2005



Recursion tree:

Chien Chen

alogbn= n logn alogbn = n logbn x logna = n logbn x logba/logbn = n logba

September 12, 2005


CASE1: The weight increases geometrically from the root to the leaves. The leaves hold a constant fraction of the total weight.


Recursion tree:

September 12, 2005


CASE2: (k= 0) The weight is approximately the same on each of the logbn levels.


Recursion tree:

September 12, 2005



CASE3: The weight decreases geometrically from the root to the leaves. The root holds a constant fraction of the total weight.

Recursion tree:

September 12, 2005


Polynomials v.s. Exponentials

• Polynomials: – A function is polynomial bounded if

• Exponentials: – Any positive exponential function grows faster

than any polynomial.

P n a ni i

i

d( )

0)()( )1(OnOnf

n o a ab n ( ) ( )1

)1(0lim aa

nn

b

n

September 12, 2005


Logarithms

• A polylogarithmic function in n is a polynomial in the logarithm of n,

• A function f(n) is polylogarithmically bounded if • for any constant a > 0.• Any positive polynomial function grows faster t

han any polylogarithmic function.

)(log)( )1( nOnf O)(log ab non

http://en.wikipedia.org/wiki/Polynomial

http://en.wikipedia.org/wiki/Logarithm

September 12, 2005


Function iteration

.0))((

,0)( )1(

)(

ifｉnff

ifｉnnf i

i

For example, if , thennnf 2)( nnf ii 2)()(

September 12, 2005


• Since the number of atoms in the observable universe is estimated to be about , which is much less than , we rarely encounter a value of n such that .

8010 655362

5lg* n

lg* 2 1lg* 4 2lg*16 3lg* 65536 4lg* 2 565536

Slowly Growing function

September 12, 2005


Appendix: geometric series

Return to last slide viewed.

September 12, 2005 Copyright©2001-5 Erik D. Demaine and Charles E. Leiserson L2.1 Introduction to...

Documents

Transcript of September 12, 2005 Copyright©2001-5 Erik D. Demaine and Charles E. Leiserson L2.1 Introduction to...