ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of...

35
ICC Module Computation – Theory of Computation 1 Feedback 75-100% : 41% 50- 75%: 44% <50%: 15% Next Friday: no new topic, instead we will revise the topics learned so far!

Transcript of ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of...

Page 1: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

1

Feedback75-100% : 41%50- 75%: 44%

<50%: 15%

Next Friday: no new topic, instead we will revise the topics learned so far!

Page 2: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

2

Complexity Example Behavior Example Algorithm

Θ(log n) Look at only log n elements of the input Binary search (Recherche dichotomique)

Θ(n) Look at each input element once Linear search

Θ(n・log n) Split input into half (which can be done at most log n times)At each split look at each element once

Merge sort (tri fusion)

Θ(n2) Look at each pair of elments (i,j) Insertion sort

Θ(n3) Look at each triple of element (i,j,k) Floyd’s shortest path algorithm

Θ(2n) Look at all subset of a list orall permutation of a list orall paths in a graph

Create a list of binary numbers of length n, e.g., input: 3output:{{0,0,0},{0,0,1},{0,1,0},{0,1,1},

{1,0,0},{1,0,1},{1,1,0},{1,1,1}}

Representative Algorithms

In this table “look at element x” means to do a fix number instructions for element x, e.g., compare x with 3, add x to another variable,…

Page 3: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

3

Information, Computation, and Communication

Theory of Computation

Page 4: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

4

TopicsStudy the two big questions of the theory of computation:§ Which problems can we solve with an algorithm?§ Which problems can we solve efficiently with an

algorithm?

Important notions:§ Countability§ Decidability (video)§ Complexity classes: P and NP (video)

Page 5: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

5

§ Previously, we have studied algorithms that work regardless of the input data.

§ Each algorithm solves the same question for different input data each time.

§ Running the algorithm on one input gives us, the answer to one instance of the problem

§ Running an algorithm is not the solution of the problem, the solution is the algorithm itself

Algorithms & Problems

Page 6: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

6

§ sort({4, 6, 2,10}) … one instance§ max({4, 6, 2,10}) … one instance§ add(3, 5) … one instance§ add(x, y) … infinitely many instances

(if x and y are not bounded)§ add(x, y), where x, y from {1, 2, 3, 4, 5}

…finitely many instances

Problem Instances

Page 7: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

7

§ If the set of problem instances is finite, it is always possible to resolve the problem by building a Look-Up Table (LUT): we store the answer to every instance. The algorithm consults this table and prints the solution found inside the table – no computation involved!

§ Only the problems involving infinite sets of instances require real work. They are the ones that interest us in today's class.

Infinity of Instances

Page 8: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

8

§ add(x,y) where x, y from {1, 2, 3, 4, 5}Example: Look-Up Table

1 2 3 4 5

1 2 3 4 5 6

2 3 4 5 6 7

3 4 5 6 7 8

4 5 6 7 8 9

5 6 7 8 9 10

xy

Another example: multiplication table until 10

Page 9: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

9

§ add(x,y) where x, y from {1, 2, 3, 4, 5}Example: Look-Up Table

xy

1 2 3 4 5

1 2 3 4 5 6

2 3 4 5 6 7

3 4 5 6 7 8

4 5 6 7 8 9

5 6 7 8 9 10

xy

Page 10: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

10

§ What is the complexity?Example: Look-Up Table

xy x

y

Page 11: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

11

§ Notation: Countability§ Observation:

the natural numbers, N = {1, 2, 3,...}, define counting.

§ Definition: an infinite set S is countable if and only if there exists a bijection f : N → S (a 1-to-1 mapping)

§ Function f defines the enumeration of S.

Comparing Infinite Sets

Page 12: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

12

§ The integers, ℤ = {… ,−3,−2,−1, 0, 1, 2, 3… }, are a countable set.

§ We define a function 𝑓:ℕ → ℤ as follows:• if i is even, then 𝑓 𝑖 = !

"• if i is odd, then 𝑓 𝑖 = − !#$

"

Example: Integers

𝑖 ∈ ℕ 1 2 3 4 5 6 …

f(𝑖) ∈ ℤ 0 1 -1 2 -2 3 …

Page 13: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

13

§ Is the set of even numbers countable?§ Yes, we define a bijection as follows:

𝑓 𝑖 = 2 % 𝑖

Countability: Example

𝑖 ∈ ℕ 1 2 3 4 5 6 …

Even numbers f(𝑖) 2 4 6 8 10 12 …

Page 14: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

14

§ Is the set of rational numbers countable? ℚ =

𝑝𝑞| 𝑝, 𝑞 ∈ ℕ 𝑎𝑛𝑑 𝑞 ≠ 0

Countability: Another Example

1 2 3 4 …

1 1/1 1/2 1/3 1/4 ..

2 2/1 2/2 2/3 2/4 …

3 3/1 3/2 3/3 3/4 …

4 4/1 4/2 4/3 4/4 …

… … … … … …

pq

Page 15: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

15

§ Is the set of rational numbers countable? ℚ =

𝑝𝑞| 𝑝, 𝑞 ∈ ℕ 𝑎𝑛𝑑 𝑞 ≠ 0

Countability: Another Example

1 2 3 4 …

1 1/1 1/2 1/3 1/4 ..

2 2/1 2/2 2/3 2/4 …

3 3/1 3/2 3/3 3/4 …

4 4/1 4/2 4/3 4/4 …

… … … … … …

pq

You can use the same order to counter pairs of integers (i,j)

Page 16: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

16

§ Is the set of rational numbers countable? ℚ =

𝑝𝑞| 𝑝, 𝑞 ∈ ℕ 𝑎𝑛𝑑 𝑞 ≠ 0

Countability: Another Example

1 2 3 4 …

1 1/1 1/2 1/3 1/4 ..

2 2/1 2/2 2/3 2/4 …

3 3/1 3/2 3/3 3/4 …

4 4/1 4/2 4/3 4/4 …

… … … … … …

pq

You can use the same order to counter pairs of integers (i,j)

Page 17: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

17

§ Every algorithm has a finite description, which is a text written in a chosen alphabet. It is easy to enumerate all the possible texts.

§ Let’s start by enumerating the texts of one character (26):

§ Let’s continue with the texts of two characters (26・26 = 676)

§ Now, let’s move to the texts of 3 characters:

§ The set of algorithms is smaller than the set of all possible texts because some texts are not algorithms. Therefore, the set of algorithms is countable.

The Set of Algorithms is Countable

𝑖 ∈ ℕ 1 2 3 4 5 6 …

Text a b c d e f …

𝑖 ∈ ℕ 27 28 29 30 31 32 …

Text aa ab ac ad ae af …

𝑖 ∈ ℕ 703 704 705 706 707 708 …

Text aaa aab aac aad aae aaf …

Page 18: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

18

§ Let’s look at a subset of problems, namely all problems that take an integer as input and return true or false.

§ This is the set of all Boolean functions 𝑓 𝑖 ∶ ℕ → 0, 1

The Set of Problems is not Countable

Page 19: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

19

Consider the set of Boolean functions of an integer variable B = 𝑓 | 𝑓 𝑖 ∶ ℕ → 0, 1

Each function is represented by a list of binary values:

If the set B were countable, we could enumerate the functions:f1, f2, f3, f4, f5, f6, f7, . . .Let’s put them in a table format:

n 1 2 3 4 5 6 7 …f1 0 1 1 0 0 0 1 . . .f2 0 0 0 0 1 0 1 . . .f3 1 1 0 0 1 1 1 . . .f4 0 1 0 1 0 1 1 . . .f5 1 0 1 0 0 0 0 . . .f6 1 1 0 1 1 0 1 . . .f7 0 0 0 1 1 1 1 . . .

The Set of Boolean Functions is Not Countable

n 1 2 3 4 5 6 7 . . .f (n) 0 0 0 1 0 0 1 . . .

Page 20: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

20

Consider the set of Boolean functions of an integer variable B = 𝑓 | 𝑓 𝑖 ∶ ℕ → 0, 1

Each function is represented by a list of binary values:

If the set B were countable, we could enumerate the functions.f1, f2, f3, f4, f5, f6, f7, . . .

The Set of Boolean Functions is Not Countable

Observe the diagonal:a list of binary values

n 1 2 3 4 5 6 7 . . .f (n) 0 0 0 1 0 0 1 . . .

n 1 2 3 4 5 6 7 …f1 0 1 1 0 0 0 1 . . .f2 0 0 0 0 1 0 1 . . .f3 1 1 0 0 1 1 1 . . .f4 0 1 0 1 0 1 1 . . .f5 1 0 1 0 0 0 0 . . .f6 1 1 0 1 1 0 1 . . .f7 0 0 0 1 1 1 1 . . .

Page 21: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

21

§ If B is countable, we can enumerate the functions: f1, f2, f3, f4, f5, f6, f7, . . .

§ The diagonal is a list of binary values 0 0 0 1 0 0 1 …

§ It can be seen as another Boolean function

6𝑓 𝑖 ∶ ℕ → 0, 1that can be written as

6𝑓 𝑖 = 𝑓!(𝑖)

The Set of Boolean Functions is Not Countable

n 1 2 3 4 5 6 7 …f1 0 1 1 0 0 0 1 . . .f2 0 0 0 0 1 0 1 . . .f3 1 1 0 0 1 1 1 . . .f4 0 1 0 1 0 1 1 . . .f5 1 0 1 0 0 0 0 . . .f6 1 1 0 1 1 0 1 . . .f7 0 0 0 1 1 1 1 . . .

Page 22: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

22

§ Let’s define a new Boolean function 𝑓∗ 𝑖 = 1 − 6𝑓 = 1 − 𝑓!(𝑖)𝑓∗ ∶ 1 1 1 0 1 1 0 . . .

§ 𝑓∗ does not appear in the enumeration because for every number i, we have 𝑓∗ 𝑖 ≠ 𝑓!(𝑖)

⇒ B is not countable.For all enumerations, there is always a function 𝑓∗ ∈ 𝐵that is not enumerated.

The Set of Boolean Functions is Not Countable

n 1 2 3 4 5 6 7 …f1 0 1 1 0 0 0 1 . . .f2 0 0 0 0 1 0 1 . . .f3 1 1 0 0 1 1 1 . . .f4 0 1 0 1 0 1 1 . . .f5 1 0 1 0 0 0 0 . . .f6 1 1 0 1 1 0 1 . . .f7 0 0 0 1 1 1 1 . . .

Page 23: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

23

What have we learned?§ All algorithms (whatever the language) are countable.§ Not all functions are countable. There are more

functions (and therefore problems) than algorithm.

In reality, most functions cannot be computed using an algorithm!

(There are simply not enough possible algorithm.)

Not Enough Algorithms!

Page 24: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

24

§ Hilbert's Tenth Problem: "Does a given Diophantine equation have an integer solution?”E.g., the equation 3x2 – 2xy – y2z – 7=0 has an integer solution: x=1, y=2, z=-2By contrast, the equation x2 + y2 + 1=0 has no such solution.

§ Tiling Problem: "Can this set of domino-like tiles tile the plane?” § Post Correspondence Problem:

"Given two lists of strings, is there a list of indices such that the concatenationsfrom both lists are equal?”

§ Security: "Does this computer program have any security vulnerabilities?” (Identifying buffer overflows is undecidable)

§ Deadcode: does a program contain a piece of useless code? (Rice’s Theorem)

§ Program Equivalent: do two programs calculate the same thing? (Rice’s Theorem)

Some Undecidable Problems

Page 25: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

25

§ Yes, if you can find an algorithm to solve P1

§ No, if you can find another problem P2 that is know to be undecidable (e.g., the Halting problem) and you can “reduce P2 to P1”, i.e.,• Show that if you had an algorithm for P1, then you

could also solve P2.• This shows that P1 is at least as difficult as P2. • Since P2 is undecidable, so is P1.

Is Problem P1 is decidable?

Page 26: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

26

§ Several problems do not have an algorithmic solution.

§ Some of these problems are artificial and present no interest, but others are important in practice, such as questions on program characteristics.

§ However, there may exist algorithms that solve common (but not all) instances of such problems.

Summary

Page 27: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

27

TopicsStudy the two big questions of the theory of computation:§ Which problems can we solve with an algorithm?§ Which problems can we solve efficiently with an

algorithm?

Important notions:§ Countability§ Decidability (video)§ Complexity classes: P and NP (video)

Page 28: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

28

§ How can we know if our algorithm is the most efficient?

§ We determine the complexity of the problem§ Like for the algorithms, we do it for the

computation time (time) and the storage requirements (space).

§ The complexity of a problem is the complexity of the best algorithm that solves that problem.

Complexity: Efficiency Measurement

Page 29: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

29

§ The complexity class which has become a synonym for the set of efficient solutions is P, and contains all problems for which there is an algorithm that solves any instance of this problem in polynomial time.

§ Problems in P:• Binary search: log n• Find one’s friends in a social network: n• Sorting: n log n• The shortest path in a graph: n log n• Alignment of two DNA sequences: n2

Solution in Polynomial Time: P

Page 30: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

30

§ The complexity class characterizing the problems whose solutions can be verified efficiently (in polynomial time) is NP (Nondeterministic Polynomial)

§ Example of Problems in NP:• Find a subset of integers whose sum is 0 in a list• Finding a DNA sequence that best fits a collection of

fragments of the sequence• Determine if a mathematical statement has a short

proof

Verification in Polynomial Time: NP

Page 31: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

31

§ The difference between P and NP (both defined in terms of polynomial time) is the same as the difference between solving a problem (compute a solution) and verifying correctness of the solution.

? * ? = 91Is 7 * 13 = 91 correct?Sort the list [3,4,5,2]

Is the list [3,4,5,2] sorted?

From P to NPPolynomial (P) vs Nondeterministic Polynomial (NP)

Page 32: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

32

§ The biggest theoretical problem of today (in Computer Science and in Mathematics):

Is P equal to NP?

§ Option (1): we find a problem in NP which is not in P

§ Option (2): we show that an NP-hard (NP difficile) problem, e.g., the Clique problem, can be solved in a polynomial way

P vs NP

Clique = group in which everyone knows everyone

Millennium problem: the Clay Math Institute named the P versus NPproblem as one of the seven most important open questions in mathematics and offers a 1 million-dollar prize for solving it.

Page 33: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

33

§ Yes, if you can find an algorithm that solves any instance of this problem in polynomial time.

Is a Problem in P?

§ Yes, if you can find an algorithm that verifiesthat a solution is correct in polynomial time.

Is a Problem in NP?

Page 34: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

34

§ Yes, if you can find another problem P2 that is known to be NP-hard (e.g., the Clique problem) and you can reduce P2 to P1 in polynomial time, i.e.,• Show that if you had an algorithm for P1, then you

could also solve P2 using only polynomially many steps to convert a solution for P1 to a solution for P2.

§ This shows that P1 is at least as difficult as P2.§ Since P2 is NP-hard, so is P1.

Is a Problem P1 NP-hard (NP difficile)?

Page 35: ICC Module Computation –Theory of Computation Feedback · ICC Module Computation –Theory of Computation 17 § Every algorithm has a finite description, which is a text written

ICC Module Computation – Theory of Computation

35

Questions ?