Algorithmns and...

Algorithmns and DatastructuresO-Notation, L’Hopital

Albert-Ludwigs-Universität Freiburg

Prof. Dr. Rolf BackofenBioinformatics Group / Department of Computer ScienceAlgorithmns and Datastructures, November 2018

Structure

O-NotationMotivation / DefinitionExamples

Ω-Notation

Θ-Notation

RuntimeSummaryLimit / ConvergenceL’Hôpital / l’HospitalPractical use

November 2018 Prof. Dr. Rolf Backofen – beamer-ufcd 2 / 56

O-NotationMotivation

We are interested in:Example: sorting

Runtime of Minsort “is growing as” n2

Runtime of Heapsort “is growing as” n lognGrowth of a function in runtime T (n)

the role of constants (e.g. 1ns) is minorit is enough if relation holds for some n≥ . . .

Describe the growth of the function more formallyby the means of Landau-Symbols [Wik]):

O(n) (Big O of n),Ω(n) (Omega of n),Θ(n) (Theta of n)


O-NotationDefinition

Big O-Notation:Consider the function: f : N→ R, n 7→ f (n)

N : Natural numbers→ input sizeR : Real numbers→ runtime

Example:f (n) = 3nf (n) = 2n lognf (n) = 1

10n2

f (n) = n2 +3n logn−4n



Big O-Notation:Given two functions f and g:f ,g : N→ RIntuitive: f is Big-O of g (f is O(g))

. . . if f relative to g does not grow faster than gthe growth rate matters, not the absolute values



Big O-Notation:Informal: f = O(g)

“=” corresponds to "is" not "is equal to". . . if for some value n0 for all n≥ n0f (n)≤ C ·g(n) for a constant C(f = O(g): From a value n0 for all n≥ n0→ f (n)≤ C ·g(n))

Formal: f ∈ O(g)

Formal: f ∈ O(g)

O(g) = f : N→R | ∃n0 ∈N, ∃C > 0, ∀n > n0 : f (n)≤C ·g(n)

“set ofall functions”

“for which” “it exists” “for all” “such that”


O-NotationExamples

Illustration of the Big O-Notation:

Figure: Runtime of two algorithms f1, f2

g(n)

f2 ∈ O(g)

f1 ∈ O(g)


O-NotationExamples

Example:f (n) = 5n +7, g(n) = n⇒ 5n +7 ∈ O(g)⇒ f ∈ O(g)Intuitive:f (n) = 5n +7→ linear growth

Attentionf (n)≤ g(n) is not guaranteed, better is f (n)≤ C ·g(n) ∀n > n0.


O-NotationProof

We have to proof: ∃n0, ∃C, ∀n≥ n0 : 5n +7≤ C ·n .

5n +7 ≤ 5n + n (for n≥ 7)= 6n

⇒ n0 = 7, C = 6


O-NotationProof

Alternate proof:

5n +7 ≤ 5n +7n (for n≥ 1)= 12n

⇒ n0 = 1, C = 12


O-NotationExamples

Big O-Notation:We are only interested in the term with the highest-order,the fastest growing summand, the others will be ignoredf (n) is limited from above by C ·g(n)

Examples:

2n2 +7n−20 ∈O(n2)2n2 +7n logn−20 ∈

7n logn−20 ∈5 ∈

2n2 +7n logn + n3 ∈


O-NotationExamples

Harder Example:Polynomes are simpleMore problematic: combination of complex functions

2√

x +3lnx ∈ O(??)


Ω-NotationDefinition

Omega-Notation:Intuitive:

f ∈Ω(g), f is growing at least as fast as gSo the same as Big-O but with at-least and not at-most

Formal: f ∈Ω(g)

Ω(g) = f : N→ R | ∃n0 ∈ N, ∃C > 0, ∀n > n0 : f (n) ≥ C ·g(n)

“in O(n)we had ≤”


Ω-NotationProof

Example:

Proof of f (n) = 5n +7 ∈Ω(n):

5n +7︸︷︷︸f (n)

≥ 1 ·n︸︷︷︸g(n)

(for n≥ 1)

⇒ n0 = 1, C = 1


Ω-NotationExamples

Illustration of the Omega-Notation:

Figure: Runtime of two algorithms f1, f2

f2 ∈Ω(g)

f1 ∈Ω(g)

g(n)


Ω-NotationExamples

Big Omega-Notation:We are only interested in the term with the highest-order,the fastest growing summand, the others will be ignoredf (n) is limited from underneath by C ·g(n)

Examples:

2n2 +7n−20 ∈Ω(n2)2n2 +7n logn−20 ∈

7n logn−20 ∈5 ∈

2n2 +7n logn + n3 ∈


Θ-NotationDefinition

Theta-Notation:Intuitive: f is Theta of g . . .

. . . if f is growing as much as gf ∈Θ(g), f is growing at the same speed as g

Formal: f ∈Θ(g)

Θ(g) = O(g)∩Ω(g)︸︷︷︸Intersection

Example:f (n) = 5n +7, f (n) ∈ O(n), f (n) ∈Ω(n)⇒ f (n) ∈Θ(n)

Proof for O(g) and Ω(g) look at slides 11 and 17


Θ-NotationGraphs

nn0

number of steps(runtime (s))

runtime of algorithm

upper bound

lower bound

f and g have the same “growth”


RuntimeLandau-Symbol Summary

Big O-Notation O(n):

f is growing at most as fast as gC ·g(n) is the upper bound

Big Omega-Notation Ω(n):

f is growing at least as fast as gC ·g(n) is the lower bound

Big Theta-Notation Θ(n):f is growing at the same speed as g

C1 ·g(n) is the lower boundC2 ·g(n) is the upper bound


RuntimeCommon Runtimes

Table: Common runtime types

Runtime Growthf ∈Θ(1) constant timef ∈Θ(logn) = Θ(logk n) logarithmic timef ∈Θ(n) linear timef ∈Θ(n logn) n-log-n time (nearly linear)f ∈Θ(n2) squared timef ∈Θ(n3) cubic timef ∈Θ(nk) polynomial timef ∈Θ(kn), f ∈Θ(2n) exponential time


So far discussed:Membership in O(. . .) proofed by hand:Explicit calculation of n0 and C

However: Both hint at limits in calculus


O-NotationLimit / Convergence

Definition of “Limit”The limit L exists for an infinite sequence f1, f2, f3, . . .if for all ε > 0 one n0 ∈ N exists, such that for alln≥ n0 the following holds true: |fn−L| ≤ ε

A function f : N→ R can be written as a sequence⇒ lim

n→∞fn = L

The limit is converging:

∀ε > 0 ∃n0 ∈ N ∀n≥ n0 : |fn−L| ≤ ε



Example for the proof of a limitFunction f (n) = 2+ 1

n with limes limn→∞ f (n) = 2“Engineering” solution: use n = ∞

1∞

= 0⇒ limn→∞

f (n) = limn→∞

2+ 1n = 2



Now a more formal proof for limn→∞

2+ 1n = 2

We need to show: for all given ε there is an n0 such that forall n≥ n0 ∣∣∣∣2+ 1

n −2∣∣∣∣ =∣∣∣∣1n∣∣∣∣≤ ε

E.g.: for ε = 0.01 we get 1n ≤ ε for n≥ 100

In generaln0 =

⌈1ε

⌉Then we get:∣∣∣∣∣1n

∣∣∣∣∣ = 1n ≤

1n0

= 1⌈1ε

⌉ ≤ 11ε

= ε



Let f ,g : N→ R with an existing limit

limn→∞

f (n)g(n) = L

Hence the following holds:

f ∈ O(g) ⇔ limn→∞

f (n)g(n) < ∞ (1)

f ∈Ω(g) ⇔ limn→∞

f (n)g(n) > 0 (2)

f ∈Θ(g) ⇔ 0< limn→∞

f (n)g(n) < ∞ (3)



f ∈ O(g) ⇔ limn→∞

f (n)g(n) < ∞

Forward proof (⇒):

f ∈ O(g) def. of O(n)⇒ ∃n0, C ∀n≥ n0 : f (n)≤ C ·g(n)

⇒∃n0, C ∀n≥ n0 : f (n)g(n) ≤ C

⇒ limn→∞

f (n)g(n) ≤ C



Backward proof (⇐):

limn→∞

f (n)g(n) < ∞

⇒ limn→∞

f (n)g(n) = C For some C ∈ R (Limit)

def. limes⇒ ∃n0, ∀n≥ n0 : f (n)g(n) ≤ C + ε (e.g. ε = 1)

⇒ ∃n0, ∀n≥ n0 : f (n)≤ (C +1)︸︷︷︸O−notation constant

·g(n)

⇒ f ∈ O(g)


Limits with L’Hôpital

Intuitive:

limn→∞

2+ 1n = 2+ 1

∞= 2

With L’Hôpital:Let f , g : N→ RIf lim

n→∞f (n) = lim

n→∞g(n) = ∞/0

⇒ limn→∞

f (n)g(n) = lim

n→∞

f ′(n)g′(n)

Holy inspirationyou need a doctoral degree for that



The limit can not be determined in the way of an Engineer:

limn→∞

ln(n)n = limn→∞ ln(n)

limn→∞

nplugging in−→ ∞

∞

Determine the limit using L’Hôpital:

limn→∞

f (n)g(n) = lim

n→∞

f ′(n)g′(n)



Using L’Hôpital:

Numerator: f(n) : n 7→ ln(n)Denominator: g(n) : n 7→ n⇒ f ′(n) = 1

n (derivation from Numerator)⇒ g′(n) = 1 (derivation from Denominator)

limn→∞

f ′(n)g′(n) = lim

n→∞

1n = 0 ⇒ lim

n→∞

f (n)g(n) = lim

n→∞

ln(n)n = 0



What can we take for granted without proofing?Only things that are trivialIt is always better to proof it

Examples:

limn→∞

1n = 0 is trivial

limn→∞

1n2 = 0 is trivial

limn→∞

log(n)n = 0 use L’Hopital


O-NotationPractical use

Practical use:It is much easier to determine the runtime of an algorithmby using the O-Notation

1 Computing rules2 Practical use


O-NotationCharacteristics

Transitivity:

f ∈Θ(g) ∧ g ∈Θ(h) → f ∈Θ(h)f ∈ O(g) ∧ g ∈ O(h) → f ∈ O(h)f ∈Ω(g) ∧ g ∈Ω(h) → f ∈Ω(h)

Symmetry:

f ∈Θ(g) ↔ g ∈Θ(f )f ∈ O(g) ↔ g ∈Ω(f )

Reflexivity:

f ∈Θ(f ) f ∈Ω(f ) f ∈ O(f )


O-NotationCalculation Rules

Trivial:

f ∈ O(f )C ·O(f ) = O(f )O(f + C) = O(f )

Addition:

O(f ) + O(g) = O(maxf , g)

Multiplication:

O(f ) ·O(g) = O(f ·g)


O-NotationRuntime Complexity

The input size for all examples is nBasic operations

i1 = 0 O(1)

Sequences of basic operationsi1 = 0 O(1)i2 = 0 O(1)· · · · · ·i327 = 0 O(1)

327 ·O(1) = O(1)



Loopsfor i in range(0, n): O(n)

a[i] = 0 O(1)

O(1) ·O(n) = O(n)

for i in range(0, n): O(n)a1[i] = 0 O(1)· · · · · ·a137[i] = 0 O(1)

137 ·O(1)= O(1)

O(1) ·O(n)

= O(n)



Loopsfor i in range(0, n): O(n)

for j in range(0, n−1): O(n−1)

O(n) ·O(n)

= O(n2)a1[i][j] = 0 O(1)· · · · · ·a137[i][j] = 0 O(1)

137 ·O(1)= O(1)

O(1) ·O(n2)

= O(n2)



Conditionsif x < 100: O(1)

y = x O(1)

O(1)

else:for i in range(0, n): O(n)

if a[i] > y: O(1)y = a[i] O(1)

O(n) ·O(1)= O(n)

O(max1,n)

= O(n)


O-NotationArithmetic mean

Input: List x with n numbersOutput: a[i] is the arithmetic mean of x[0] to x[i]

def ar i thMean ( x ) :a = [ 0 ] ∗ len ( x )f o r i i n range (0 , len ( x ) ) :

s = 0f o r j i n range (0 , i +1) :

s = s + x [ j ]

a [ i ] = s / ( i +1)

r e t u rn a


O-Notation Runtime complexity

for i in range(0, len(x)) : O(n)s = 0 O(1)

O(n)

for j in range(0, i+1): O(i +1)s = s + x[j] O(1)

O(i)

a[i] = s / ( i+1) O(1)

O(n) ·O(i)

= O(n2)

How often will the instructions in the loop be executed,when the problem has size n?

1+2+ . . .+ n = n · (n +1)2 ∈ O(n2)


O-NotationDiscussion

Way of speaking:With the O-Notation we look at the behavior of a functionwhen n→ ∞

We only analyze the runtime when n≥ n0

We talk about asymptotic analysis, when we discuss cost,runtime, etc. as O(. . .), Ω(. . .) or Θ(. . .)



Attention:

If you are using asymptotic analysis, you can not makeany predictions about the runtime of smaller input sizes(n < n0)For small input sizes (mostly n < 10), the runtime ispredictably smalln0 does not necessarily have to be small



Examples:Let A and B be algorithms

A has the runtime f (n) = 80nB has the runtime g(n) = 2n log2n

So f = O(g) but not Θ(g)⇒ A is asymptotic faster than B⇒ There is an n0 for that n≥ n0 : f (n)≤ g(n)



When is A faster then B?We search the minimal n0:

f (n0) = g(n0)80n0 = 2n0 log2n0

40 = log2n0

n0 = 240

= (210)4 = (1024)4

≈ (103)4 = 1012

≈ 1trillion

A ist faster than B if n0 has more than 1 trillion elements


Runtime ExamplesContinued

Logarithm of different bases differ only by a constant

loga n = logb nlogb a = 1

logb a · logb n

Hence: loga n ∈Θ(logb n)For exponent this does not hold

3n 6∈Θ(2n)

Proof: Use equation (1) from Slide 31

3n ∈ O(2n)⇔ limn→∞

3n

2n < ∞

However:limn→∞

3n

2n = limn→∞

(32

)n= ∞


Additional Figure

Figure for slide 28


Further Literature

General[MS08] Kurt Mehlhorn and Peter Sanders.

Algorithms and data structures, 2008.https://people.mpi-inf.mpg.de/~mehlhorn/ftp/Mehlhorn-Sanders-Toolbox.pdf.


https://people.mpi-inf.mpg.de/~mehlhorn/ftp/Mehlhorn-Sanders-Toolbox.pdf

https://people.mpi-inf.mpg.de/~mehlhorn/ftp/Mehlhorn-Sanders-Toolbox.pdf

Further Literature

Big O notation

[Wik] Big O notationhttps://en.wikipedia.org/wiki/Big_O_notation


https://en.wikipedia.org/wiki/Big_O_notation

Algorithmns and...

Documents

Transcript of Algorithmns and...