Linear Time Sorting

Lower Bound for Sorting

Lower Bound

Theorem

Comparison based sorting of n elements requires Ω(n log n) time.

3 / 16

Lower Bound

Recall decision trees.

A[i] < A[j]?

......

. . . ≤ A[i] ≤ A[j] ≤ . . .

No Yes

Decision tree for sorting

I Node decides if A[i] < A[j].I Leaf represents permutation of A.

I n! leafsI Thus, height is at least log n!

4 / 16

Lower Bound

What is log n!?

log n! = log(1 · 2 · . . . · n

)= log 1 + log 2 + . . . + log(n/2) + . . . + log n (≤ n log n)

≥ n2 log(n/2)

=n2 log n− n

2∈ Ω(n log n)

5 / 16

Integer Sorting

Comparison based sorting requires Ω(n log n) time.

I What if we do something different?

Integer Sorting

I Given n integers in 0, . . . , k − 1I Each integer fits in a word.

I All operations for words are allowed, e. g., +, −, . . .

I Allows linear time sorting if k is limited.

6 / 16

Counting Sort

Counting Sort

Idea

I Count for each v ∈ 0, . . . , k − 1 how often it is in A.

I Use this to compute index of A[i] in sorted array.

5 2a 4 6a 3 1 6b 2bA

C

0 1 2 3 4 5 6

0 1 2 1 1 1 2

How do we compute the index?

8 / 16

Counting Sort

Last position of some value v in B:

I IndB(v) = C[0] + C[1] + . . . + C[v ]− 1(−1 because array is 0-based)

B < v v > v

C[0] + C[1] + . . .+ C[v ]

9 / 16

Counting Sort

Observation

I IndB(v) = IndB(v − 1) + C[v]

I Therefore, update C such that C′[i] = C[0] + C[1] + . . . + C[i ]− 1.

C

C′

0

−1

1

0

2

2

1

3

1

4

1

5

2

7

10 / 16

Counting Sort

Algorithm

I Count (in array C) how often each v is in A.

I Update C such that IndB(v) = C[v].

I Iterate backwards over A. Copy each element v = A[i] into B[C[v]]and reduce C[v] by 1.

0 1 2 3 4 5 6 7

5 2a 4 6a 3 1 6b 2bA

C

B

11 / 16

Counting Sort

Algorithm

I Count (in array C) how often each v is in A.

I Update C such that IndB(v) = C[v].

I Iterate backwards over A. Copy each element v = A[i] into B[C[v]]and reduce C[v] by 1.

0 1 2 3 4 5 6 7

5 2a 4 6a 3 1 6b 2bA

C 0 1 2 1 1 1 2

B

11 / 16

Counting Sort

Algorithm

I Count (in array C) how often each v is in A.

I Update C such that IndB(v) = C[v].

I Iterate backwards over A. Copy each element v = A[i] into B[C[v]]and reduce C[v] by 1.

0 1 2 3 4 5 6 7

5 2a 4 6a 3 1 6b 2bA

C −1 0 2 3 4 5 7

B

11 / 16

Counting Sort

Algorithm

I Count (in array C) how often each v is in A.

I Update C such that IndB(v) = C[v].

I Iterate backwards over A. Copy each element v = A[i] into B[C[v]]and reduce C[v] by 1.

0 1 2 3 4 5 6 7

5 2a 4 6a 3 1 6b 2bA

C −1 0 1 3 4 5 7

B 2b

11 / 16

Counting Sort

Algorithm

I Count (in array C) how often each v is in A.

I Update C such that IndB(v) = C[v].

I Iterate backwards over A. Copy each element v = A[i] into B[C[v]]and reduce C[v] by 1.

0 1 2 3 4 5 6 7

5 2a 4 6a 3 1 6b 2bA

C −1 0 1 3 4 5 6

B 2b 6b

11 / 16

Counting Sort

Algorithm

I Count (in array C) how often each v is in A.

I Update C such that IndB(v) = C[v].

I Iterate backwards over A. Copy each element v = A[i] into B[C[v]]and reduce C[v] by 1.

0 1 2 3 4 5 6 7

5 2a 4 6a 3 1 6b 2bA

C −1 −1 1 3 4 5 6

B 1 2b 6b

11 / 16

Counting Sort

Algorithm

I Count (in array C) how often each v is in A.

I Update C such that IndB(v) = C[v].

I Iterate backwards over A. Copy each element v = A[i] into B[C[v]]and reduce C[v] by 1.

0 1 2 3 4 5 6 7

5 2a 4 6a 3 1 6b 2bA

C −1 −1 1 2 4 5 6

B 1 2b 3 6b

11 / 16

Counting Sort

Algorithm

I Count (in array C) how often each v is in A.

I Update C such that IndB(v) = C[v].

I Iterate backwards over A. Copy each element v = A[i] into B[C[v]]and reduce C[v] by 1.

0 1 2 3 4 5 6 7

5 2a 4 6a 3 1 6b 2bA

C −1 −1 1 2 4 5 5

B 1 2b 3 6a 6b

11 / 16

Counting Sort

Algorithm

I Count (in array C) how often each v is in A.

I Update C such that IndB(v) = C[v].

I Iterate backwards over A. Copy each element v = A[i] into B[C[v]]and reduce C[v] by 1.

0 1 2 3 4 5 6 7

5 2a 4 6a 3 1 6b 2bA

C −1 −1 1 2 3 5 5

B 1 2b 3 4 6a 6b

11 / 16

Counting Sort

Algorithm

I Count (in array C) how often each v is in A.

I Update C such that IndB(v) = C[v].

I Iterate backwards over A. Copy each element v = A[i] into B[C[v]]and reduce C[v] by 1.

0 1 2 3 4 5 6 7

5 2a 4 6a 3 1 6b 2bA

C −1 −1 0 2 3 5 5

B 1 2a 2b 3 4 6a 6b

11 / 16

Counting Sort

Algorithm

I Count (in array C) how often each v is in A.

I Update C such that IndB(v) = C[v].

I Iterate backwards over A. Copy each element v = A[i] into B[C[v]]and reduce C[v] by 1.

0 1 2 3 4 5 6 7

5 2a 4 6a 3 1 6b 2bA

C −1 −1 0 2 3 4 5

B 1 2a 2b 3 4 5 6a 6b

11 / 16

Counting Sort

Properties

I Runtime: O(n + k), i. e., O(n) if k ∈ O(n).

I Memory: O(n + k) (for C and B)

I Stable (if implemented correctly)

12 / 16

Counting Sort

Input: An array A such that |A| = n and A[i] ∈ 0, 1, . . . , k − 1.Output: An array B containing the elements of A in sorted order.

1 Create two arrays B and C with |B| = n, |C| = k and initial value 0.

2 For i := 0 To n− 1

3 C[A[i]] := C[A[i]] + 1

4 C[0] := C[0]− 1

5 For i := 1 To k − 1

6 C[i] := C[i] + C[i − 1]

7 For i := n− 1 DownTo 0

8 B[C[A[i]]] := A[i]

9 C[A[i]] := C[A[i]]− 1

13 / 16

Radix Sort

Radix Sort

Idea

I Imagine integers as sequence of d digits with base b.

I Sort integers by digit.

I Start with the least significant digit and use a stable sort(e. g. counting sort).

1 4 12 2 43 4 13 0 41 2 14 3 02 0 23 1 3

4 3 01 4 13 4 11 2 12 0 23 1 32 2 43 0 4

2 0 23 0 43 1 31 2 12 2 44 3 01 4 13 4 1

1 2 11 4 12 0 22 2 43 0 43 1 33 4 14 3 0

15 / 16

Radix Sort – Runtime

Input

I n integers in range [0, k − 1] and a base b

Runtime

I Sorting by a single digit: O(n + b)

I Sorting by all d digits: O((n + b) · d) = O((n + b) logb k)

I O(nc) time if b = n and k ≤ nc

linear if c is constant

Other properties

I Stable

I O(n + b) additional memory is used.

We assume counting sort for sorting by digit.

16 / 16