Computational Complexity: An Overview - Understanding Algorithm Efficiency - Prof. Edward , Study notes of Computer Architecture and Organization

Download Computational Complexity: An Overview - Understanding Algorithm Efficiency - Prof. Edward and more Study notes Computer Architecture and Organization in PDF only on Docsity! Computational Complexity: An Overview Ed Bosworth Computer Science Department Columbus State University Columbus, GA bosworth_edward@colstate.edu BACKGROUND ASSUMED Some programming experience, maybe in Visual Basic or C++ Have seen some algorithms for searching and sorting Algebra, including logarithms and exponents. Factorial function: N! = N  (N – 1) … N – 1) … ) …  2  1) … Intractable Problems Characteristic: Try every solution and see if one works. Classic problem: Tiling a floor with irregularly shaped tiles Like a jigsaw puzzle if you do it “upside-down”, except you know that the puzzle can be completed. Heuristics: For jigsaw puzzles you use a number of clues, such as colors to aid in putting the puzzle together. Heuristics do not guarantee an answer, but often produce one. Unsolvable Problems Some problems are provably unsolvable, while other problems have no known solution. This is different. Classic unsolvable problem: the Halting Problem. It is provable that no solution exists. Classic problem with no known solution. Is P = NP? (N – 1) … Be patient – I shall define this one) This is a decision problem: Answer “Yes” or “No”. Three possible answers (N – 1) … can you prove one of them?) Yes No The problem is unsolvable. Problems, Strategies, and Algorithms Sample Problems Sort a list by comparison, using a defined ≥ operator. Traveling Salesman – route through each city exactly once. Matrix addition and matrix multiplication Sample Strategies Brute Force Divide and Conquer Dynamic Programming Greedy Sample Algorithms Bubble Sort, Quick Sort, Merge Sort, etc. Breadth-First Search (N – 1) … BFS), Depth-First Search (N – 1) … DFS) Bounds on Functions The “Big-O” and “Big-’ notation for positive functions; f(N – 1) … n) > 0. f(N – 1) … n) is O(N – 1) … g(N – 1) … n)) if there exist positive integers K and N such that for all n  N, f(N – 1) … n)  Kg(N – 1) … n). f(N – 1) … n) is (N – 1) … g(N – 1) … n)) if there exist positive integers K and N such that for all n  N, f(N – 1) … n)  Kg(N – 1) … n). Typical functions for g(N – 1) … n) logarithmic: ln(N – 1) … n), log1) … 0(N – 1) … n), etc. polynomial: n2, n3, etc. nlog(N – 1) … n) is “polynomial” exponential: 2n, en, n!, etc. CLAIM: The Number 100! Is “Astronomical” Let's put a lower bound on the value of 100! 2t=2, 3!=6, 4!=24, 51=120, 6! =720, 7!=5040, 8!= 40320, 9!=362,880, 10! = 3,628,800 Note: The square root of 10 is about 3.1623 Thus 9! > (10) #(10)=(10)"> 100! = 1004994. 290 #894. 9809790... 70069e... #20#19e..¢1 191029! 10 10 10 10 numbers numbers numbers numbers = 90 = 80 270 210 FACT: 1) … 00! Is Bigger Than Astronomical 1) … 00! > 1) … 00  (N – 1) … 90)1) … 0  (N – 1) … 80)1) … 0  (N – 1) … 70)1) … 0 …  (N – 1) … 20)1) … 0  (N – 1) … 1) … 0)1) … 0  9! > 1) … 00  91) … 01) … 01) … 0  81) … 01) … 01) … 0  71) … 01) … 01) … 0 …  21) … 01) … 01) … 0  1) … 1) … 01) … 01) … 0  9! We have 9 occurrences of 1) … 01) … 0, so we group the numbers to get 1) … 00! > 1) … 00  91) … 0  81) … 0  71) … 0 …  21) … 0  1) … 1) … 0 1) … 090  9! > 1) … 00  (N – 1) … 9!)1) … 0 1) … 090  9! = (N – 1) … 9!)1) … 1) … 1) … 092 > (N – 1) … 1) … 05.5)1) … 1) … 1) … 092 = 1) … 060.5 1) … 092 = 1) … 01) … 52.5 > 31) … 01) … 52. There are only 1) … 078 elementary particles in the known universe (N – 1) … Scientific American, March 2003, page 53) Time Complexity: Algorithms vs. Problems Upper Bound: Problem X is O(N – 1) … f(N – 1) … n)) Let problem X be solved by algorithms A1) … , A2, …., An. The time complexity of X is not greater than the minimum of the time complexities of A1) … , A2, …., An. Thus we say that the problem of sorting by comparison is O(N – 1) … nlog(N – 1) … n)) because there exists an O(N – 1) … nlog(N – 1) … n)) algorithm to sort. Big picture: Sorting by comparison has polynomial time complexity. Time Complexity: Algorithms vs. Problems Lower Bound: Problem X is (N – 1) … f(N – 1) … n)) Let problem X be solved by algorithms A1) … , A2, …., An. The time complexity of X is not greater than the minimum of the time complexities of A1) … , A2, …., An. The lower bound for a problem must be established theoretically, not just by comparing every algorithm known to solve it. Sorting by comparisons only has been proven to be (N – 1) … nlog(N – 1) … n)). Example: Lower Bound for Matrix Addition and Multiplication Consider three square matrices (N – 1) … N by N) A, B, C where C is either the sum or product of matrices A and B The basic structure for setting values for array C must be for (int i = 0; i < n; i++) for (int j = 0; j < n; j++) c[i][j] = something It takes n2 steps to fill the n by n array, so both the addition and multiplication problems are (N – 1) … n2). Example: Towers of Hanoi Published by the French mathematician Edouard Lucas in 1) … 883. Question: How many steps to move N disks? RULES: Move all disks from peg 1) … to peg 2 Move one disk at a time Do not place a larger disk on top of a smaller disk. Recursive Solution to Towers of Hanoi To move N disks from peg 1) … to peg 2 1) … ) Move (N – 1) … N – 1) … ) disks from peg 1) … to peg 3 2) Move the big disk from peg 1) … to peg 2 3) Move (N – 1) … N – 1) … ) disks from peg 3 to peg 2 Let T(N – 1) … N) be the number of steps to move N disks. The recurrence equation for T(N – 1) … N) is T(N – 1) … 1) … ) = 1) … T(N – 1) … N) = 2  T(N – 1) … N – 1) … ) + 1) … It can be shown that the solution is T(N – 1) … N) = 2N – 1) … . Decision Problems A decision problem is one that has a “Yes”/”No” answer; i.e., is this list of numbers sorted? An algorithm to verify that a list of numbers is sorted will require at most (N – 1) … n – 1) … ) key comparisons. Decision problems are associated with optimization problems. For the travelling salesman problem (N – 1) … to be defined later): Optimization: What is the length of the shorted tour? Decision: Does there exist a tour of length less than Y? Side Remark: The Real Version of the Problem No such tour exists. All flights must go through Atlanta. Notes on the Solution One can pick the starting city arbitrarily. The only algorithm that provably solves the problem for n cities takes (N – 1) … n – 1) … )! steps, here 5! = 1) … 20. The decision problem can be verified in O(N – 1) … n) time, here 6 steps. I give you a route and a claim it is less that 1) … 000 miles. You just add the length of the n legs of the route to produce the sum. The only algorithms to solve TSP have exponential time complexity. However algorithms exist that frequently solve in polynomial time polynomial time algorithms exist if we are dealing with plane geometry one will accept an answer  1) … 50% of optimal The Problem Class NP A decision problem X is in the set NP if and only if it is solvable by a nondeterministic algorithm of polynomial time complexity. Nondeterministic algorithms do not exist; they are a theoretical abstraction. Think of them as “magic balls” or oracles. For TSP, the magic ball always correctly indicates the next city in the route in polynomial time without any need for calculations, such as the standard search algorithms require. The Problem Class NP-Complete There is a set of more than 1) … ,000 problems with an interesting property, stated two ways: An efficient (N – 1) … polynomial time) solution to any problem in the class implies an efficient solution to all problems in the class. If any problem in the class can be proven to be intractable, then all problems in the class are proven to be intractable. TSP (N – 1) … Decision Version) is NP-Complete. The Big Question: Is P = NP? From the fact that polynomial time solutions exist for problems in P, it follows that P  NP. Question: Is P  NP or P = NP? Nobody seriously believes that P = NP. In 40 years, nobody has offered a convincing proof that P  NP. An efficient solution for any NP-Complete problem would immediately imply that P = NP. NP-Complete Problems: “Where the Action Is” All problems in P have known efficient solutions. No problem in NP is known to have an efficient solution and it is not likely that any such solution will ever be found, but: one can find efficient solutions to related problems. one can find algorithms that usually solve the problem in reasonable time. one can find algorithms that quickly generate approximate answers that are quite useable. This is the topic of most advanced algorithm courses.