Analysis of Sorting Algorithms: Insertion Sort, Heap Sort, and Quick Sort, Assignments of Design and Analysis of Algorithms

Download Analysis of Sorting Algorithms: Insertion Sort, Heap Sort, and Quick Sort and more Assignments Design and Analysis of Algorithms in PDF only on Docsity! ve <4 RIPHAH INTERNATIONAL UNIVERSITY NAME: MUHAMMAD ABDULLAH (FALAK) DEPARTMENT: COMPUTING SECTION: BSCS 4"! C ROLL # 13961 DESIGN AND ANALYSIS OF ALGORITHMS SIR. MUHAMMAD ZUBAIR ASSIGNMENT # 02 (ALGORITHM, ANALYSIS AND TIME COMPLEXITY) QUESTION # 01: Write the algorithm and analysis of that algorithm and find out the T(n): A. Insertion Sort B. Heap Sort C. Quicksort D. Initialize random array of maximum 5000 elements and calculate the sorting time. SOLUTION: INSERTION SORT If the first few objects are already sorted, an unsorted object can be inserted in the sorted set-in proper place. This is called insertion sort. An algorithm considers the elements one at a time, inserting each in its suitable place among those already considered (keeping them sorted). Insertion sort is an example of an incremental algorithm; it builds the sorted sequence one number at a time. This is perhaps the simplest example of the incremental insertion technique, where we build up a complicated structure on n items by first building itonn-1 items and then making the necessary changes to fix things in adding the last item. The given sequences are typically stored in arrays. We also refer the numbers as keys. Along with each key may be additional information, known as satellite data. [Note that "satellite data" does not necessarily come from satellite] ALGORITHM: It works the way you might sort a hand of playing cards: e We start with an empty left hand [sorted array] and the cards face down on the table [unsorted array]. e Then remove one card [key] at a time from the table [unsorted array], and insert it into the correct position in the left hand [sorted array]. ¢ To find the correct position for the card, we compare it with each of the cards already in the hand, from right to left. Note that at all times, the cards held in the left hand are sorted, and these cards were originally the top cards of the pile on the table. T(n) =c1n +c2(n-1)+c4 (n-1)+c5 52 <j <n [n(n +1)/2 +1] + c652<j<n [n(n- 1)/2] + c7 2 <j <n [n(n - 1)/2] + c8(n - 1) T(n) = (c5/2 + c6/2 + c7/2) n2 + (cl +c2+c44+0c5/2 - c6/2 -c7/2+c8)n-(c2+c4+c5 + c8) This running time can be expressed as (an2 + bn +c) for constants a, b, and c that again depend on the statement costs ci. Therefore, T(n) is a quadratic function of n. Here the punch line is that the worst-case occurs, when line 5 executed j times for each j. This can happens if array A starts out in reverse order T(n) = an2 + bn +c = O(n2) It is a quadratic function of n. GRAPHICAL INTERPRETATION: ‘Seconds: 8 IN Z 10 100 1000 10000 e000 50000 75000 100000 RB The graph shows the n2 complexity of the insertion sort. WORST-CASE AND AVERAGE-CASE ANALYSIS: We usually concentrate on finding the worst-case running time: the longest running time for any input size n. The reasons for this choice are as follows: e The worst-case running time gives a guaranteed upper bound on the running time for any input. That is, upper bound gives us a guarantee that the algorithm will never take any longer. e For some algorithms, the worst case occurs often. For example, when searching, the worst case often occurs when the item being searched for is not present, and searches for absent items may be frequent. e Why not analyze the average case? Because it's often about as bad as the worst case. Example: Suppose that we randomly choose n numbers as the input to insertion sort. On average, the key in A[j] is less than half the elements in A[1 .. j - 1] and it is greater than the other half. It implies that on average, the while loop has to look halfway through the sorted subarray A[1 .. j - 1] to decide where to drop key. This means that tj = j/2. Although the average-case running time is approximately half of the worst-case running time, it is still a quadratic function of n. STABILITY: Since multiple keys with the same value are placed in the sorted array in the same order that they appear in the input array, Insertion sort is stable. EXTRA MEMORY: This algorithm does not require extra memory. e For Insertion sort we say the worst-case running time is 6(n2), and the best-case running time is O(n). e Insertion sort use no extra memory it sort in place. ¢ The time of Insertion sort is depends on the original order of a input. It takes a time in Q(n2) in the worst-case, despite the fact that a time in order of n is sufficient to solve large instances in which the items are already sorted. IMPLEMENTATION: void insertionSort(int numbers[], int array_size) { int i, j, index; for (i= 1; i <array_size; i++) { index = numbers[i]; Jeu while ((j > 0) && (numbers|j - 1] > index)) { numbers[j] = numbers[j — 1]; jei-4; } numbers|[j] = index; HEAP SORT The binary heap data structures is an array that can be viewed as a complete binary tree. Each node of the binary tree corresponds to an element of the array. The array is completely filled on all levels except possibly lowest. We represent heaps in level order, going from left to right. The array corresponding to the heap above is [25, 13, 17, 5, 8, 3]. The root of the tree A[1] and given index i of a node, the indices of its parent, left child and right child can be computed: PARENT (i) return floor(i/2) LEFT (i) return 2i RIGHT (i) return 2i+1 smallest element must equal the smallest element, so in general, only entire subtrees of the heap can contain the smallest element. INSERTING ELEMENT IN THE HEAP: Suppose we have a heap as follows: Let's suppose we want to add a node with key 15 to the heap. First, we add the node to the tree at the next spot available at the lowest level of the tree. This is to ensure that the tree remains complete. (20) Cn a @O@ @® © Let's suppose we want to add a node with key 15 to the heap. First, we add the node to the tree at the next spot available at the lowest level of the tree. This is to ensure that the tree remains complete. 0) (i (12 fg ©) @ QD ©@) Now we do the same thing again, comparing the new node to its parent. Since 14 < 15, we have to do another swap: (20) (is) (1? 9 @@ @ oO @ Now we are done, because 15 < 20. BASIC PROCEDURE OF THE HEAP: © Heapify, which runs in O(lg n) time. ¢ Build-Heap, which runs in linear time. ¢ Heap Sort, which runs in O(n Ig n) time. e — Extract-Max, which runs in O(Ig n) time. MAINTAINING THE HEAP PROPERTY: Heapify is a procedure for manipulating heap data structures. It is given an array A and index i into the array. The subtree rooted at the children of A[i] are heap but node A[I] itself may possibly violate the heap property i.e., A[i] < A[2i] or A[i] < A[2i +1]. The procedure 'Heapify' manipulates the tree rooted at Ali] so it becomes a heap. In other words, 'Heapify' is let the value at A[i] "float down" ina heap so that subtree rooted at index i becomes a heap. OUTLINE OF PROCEDURE HEAPIFY: Heapify picks the largest child key and compare it to the parent key. If parent key is larger than heapify quits, otherwise it swaps the parent key with the largest child key. So that the parent is now becomes larger than its children. It is important to note that swap may destroy the heap property of the subtree rooted at the largest child node. If this is the case, Heapify calls itself again using largest child node as the new root. Heapify (A, i) |< left [i] r€ right [i] if | < heap-size [A] and All] > A[i] then largest < | else largest € i if r < heap-size [A] and A[i] > A[largest] then largest € r if largest #i then exchange A[i] <> Allargest] Heapify (A, largest) ANALYSIS: If we put a value at root that is less than every value in the left and right subtree, then 'Heapify' will be called recursively until leaf is reached. To make recursive calls traverse the longest path toa leaf, choose value that make 'Heapify' always recurse on the left child. It follows the left branch when left child is greater than or equal to the right child, so putting 0 at the root and 1 at all other nodes, for example, will accomplished this task. With such values 'Heapify' will called h times, where h is the heap height so its running time will be 6(h) (since each call does (1) work), which is (Ign). Since we have a case in which Heapify's running time (Ig n), its worst- case running time is Q(Ign). EXAMPLE OF HEAPIFY: Suppose we have a complete binary tree somewhere whose subtrees are heaps. In the following complete binary tree, the subtrees of 6 are heaps: @ © @ @ @ @ The Heapify procedure alters the heap so that the tree rooted at 6's position is a heap. Here's how it works. First, we look at the root of our tree and its two children. We then determine which of the three nodes is the greatest. If it is the root, we are done, because we have a heap. If not, we exchange the appropriate child with the root, and continue recursively down the tree. In this case, we exchange 6 and 8, and continue. number of nodes of height h. Since Xp-1 of these subtrees are full, they each contain exactly 251-1 nodes. One of the height h subtrees may not full, but contain at least 1 node at its lower level and has at least 2h nodes. The exact count is 1+2+4+...+2%*141=2". the remaining nodes have height strictly more than h. To connect all subtrees rooted at node of height h., there must be exactly Xp -1 such nodes. The total of nodes is at least. (Xp-1)(2"*! 4 1) + 2" + Xi-1 which is at most n. Simplifying gives Xp <n/2"! + 1/2. n the conclusion, it is a property of binary trees that the number of nodes at any level is half of the total number of nodes up to that level. The number of leaves in a binary heap is equal to n/2, where nis the total number of nodes in the tree, is even and [ n/2 |when nis odd. If these leaves are removed, the number of new leaves will be [ Ig(n/2/2 Tor! n/4 | If this process is continued for h levels the number of leaves at that level will bel n/2t_ |, IMPLEMENTATION: void heapSort(int numbers[], int array_size) { int i, temp; for (i = (array_size / 2)-1; i >= 0; i--) siftDown(numbers, i, array_size); for (i= array_size-1; i >= 1; i--) { temp = numbers[0]; numbers[0] = numbers[i]; numbers[i] = temp; siftDown(numbers, 0, i-1); } } void siftDown(int numbers[], int root, int bottom) { int done, maxChild, temp; done = 0; while ((root*2 <= bottom) && (!done)) { if (root*2 == bottom) maxChild = root * 2; else if (numbers[root * 2] > numbers[root * 2 + 1]) maxChild = root * 2; else maxChild = root * 2 +1; if (numbers[root] < numbers[maxChild]) { temp = numbers[root]; numbers[root] = numbers[maxChild]; numbers[maxChild] = temp; root = maxChild; } else done = 1; } } QUICK SORT The basic version of quick sort algorithm was invented by C. A. R. Hoare in 1960 and formally introduced quick sort in 1962. It is used on the principle of divide-and-conquer. Quick sort is an algorithm of choice in many situations because it is not difficult to implement, it is a good "general purpose" sort and it consumes relatively fewer resources during execution. ADVANTAGES: e Itis in-place since it uses only a small auxiliary stack. e It requires only n log(n) time to sort n items. e thas an extremely short inner loop e This algorithm has been subjected to a thorough mathematical analysis, a very precise statement can be made about performance issues. DISADVANTAGES: e Itis recursive. Especially if recursion is not available, the implementation is extremely complicated. ¢ It requires quadratic (i.e., n2) time in the worst-case. e = Itis fragile i.e., a simple mistake in the implementation can go unnoticed and cause it to perform badly. Quick sort works by partitioning a given array A[p . . r] into two non-empty sub array A[p . . gq] and A[q+1 . . r] such that every key in A[p . . q] is less than or equal to every key in A[q+1 . . r]. Then the two subarrays are sorted by recursive calls to Quick sort. The exact position of the partition depends on the given array and index q is computed as a part of the partitioning procedure. QuickSort 1. If p<rthen 2. q Partition (A, p, r) 3. Recursive call to Quick Sort (A, p, q) 4. Recursive call to Quick Sort (A, q +r, r) Note that to sort entire array, the initial call Quick Sort (A, 1, length[A]) Asa first step, Quick Sort chooses as pivot one of the items in the array to be sorted. Then array is then partitioned on either side of the pivot. Elements that are less than or equal to pivot will move toward the left and elements that are greater than or equal to pivot will move toward the right. PARTITIONING THE ARRAY: Partitioning procedure rearranges the subarrays in-place. PARTITION (A, p, r) 1.x € Alp] 27€p-1 3. fer. 4. while TRUE do 5. Repeatj < j-1 until A[j] < x Repeat i ¢ i+1 until A[/] > x ifi<j 6 7. 8. 9. 10. then exchange A[/] <> A[j] RANDOMIZED_PARTITION (A, p, r) i © RANDOM (p, r) Exchange A[p] © Afi] return PARTITION (A, p, r) Now randomized quick sort call the above procedure in place of PARTITION RANDOMIZED_QUICKSORT (A, p, r) If p<rthen q < RANDOMIZED_PARTITION (A, p, r) RANDOMIZED_QUICKSORT (A, p, q) RANDOMIZED_QUICKSORT (A, q+1, r) Like other randomized algorithms, RANDOMIZED_QUICKSORT has the property that no particular input elicits its worst-case behavior; the behavior of algorithm only depends on the random-number generator. Even intentionally, we cannot produce a bad input for RANDOMIZED_QUICKSORT unless we can predict generator will produce next. ANALYSIS OF QUICK SORT: WORST CASE: Let T(n) be the worst-case time for QUICK SORT on input size n. We have a recurrence: T(n) = maxasqsn-a (T(q) + T(n-q)) +@(n) 1 where q runs from 1 to n-1, since the partition produces two regions, each having size at least 1. Now we guess that 7(n) < cn? for some constant c. Substituting our guess in equation 1.We get T(n) = maxazasn-1 (cq?) + c(n - g?)) + An) = ¢ max (q? + (n-q)*) + Gn) Since the second derivative of expression q* + (n-q)* with respect to q is positive. Therefore, expression achieves a maximum over the range 1< q <n -1 at one of the endpoints. This gives the bound max (q? + (n - q)*)) 1 + (n -1)?=n? + 2(n-1). Continuing with our bounding of T(n) we get: T(n) < c [n? - 2(n-1)] + Qn) = cn? - 2c(n-1) + Bn) Since we can pick the constant so that the 2c(n -1) term dominates the On) term we have: T(n) < cn? Thus the worst-case running time of quick sort is O(n). AVERAGE-CASE ANALYSIS: If the split induced by RANDOMIZED_PARTITION puts constant fraction of elements on one side of the partition, then the recurrence tree has depth @lgn) and @n) work is performed at Ollg n) of these level. This is an intuitive argument why the average-case running time of RANDOMIZED_QUICKSORT is ©(n Ig n). Let T(n) denotes the average time required to sort an array of n elements. A call to RANDOMIZED_QUICKSORT with a 1 element array takes a constant time, so we have 7(1) = 0 (1). After the split RANDOMIZED_QUICKSORT calls itself to sort two subarrays. The average time to sort an array A[1 . . q] is T[q] and the average time to sort an array A[q+1 . . n] is T[n-q]. We have T(n) = 1/n (T(1) + T(n-A) + Sge T(q) + Tn-q))) +n) --—- 1 We know from worst-case analysis: T(1) = (1) and T(n -1) = O(n?) T(n) = 1/n (@X(1) + O(n?) + 1/n "59-1 (r(q) + Tn - q)) +n) = Vn ™Sea(T(q)+T(n-q))+ O(n) ee 2 = I/n[2 ”*Yee(T(k)] + Qn) =2/n™SeeaTkK)+@ln) ete 3 Solve the above recurrence using substitution method. Assume inductively that T(n) < anlgn + b for some constants a > O and b> 0. If we can pick 'a' and 'b' large enough so that n Ig n + b > T(1). Then for n > 1, we have T(n) ="*5421 2/n (akigk + b) + Qn) = 2a/n "3 k-1 kigk - 1/8(n?) + 2b/n(n -1) +@n)— ------- 4 At this point we are claiming that: "1 perkigk < 1/2 n? Ign - 1/8(n?) Stick this claim in the equation 4 above and we get: T(n) S$ 2a/n [1/2 n2 Ign - 1/8(n?)] + 2/n b(n -1) + Qn) Ss anlgn-an/4+2b+@(n) wee 5 In the above equation, we see that Qn) + band an/4 are polynomials and we certainly can choose 'a' large enough so that an/4 dominates Qn) +b. We conclude that QUICKSORT's average running time is Qn Ig(n)). CONCLUSION: Quick sort is an in-place sorting algorithm whose worst-case running time is (n2) and expected running time is (n Ig n) where constants hidden in (n lg n) are small. IMPLEMENTATION: void quickSort(int numbers[], int array_size) { q_sort(numbers, 0, array_size - 1); } void q_sort(int numbers|[], int left, int right) { int pivot, |_hold, r_hold; Lhold = left; r_hold = right; pivot = numbers[left]; while (left < right) { while ((numbers[right] >= pivot) && (left < right)) right--; if (left != right) { numbers[left] = numbers[right]; left++; } while ((numbers[left] <= pivot) && (left < right)) left++; if (left != right) { numbers[right] = numbers[left]; right--; } long j; for(j = i-1; j >= 0 && list[j] > temp; j--) { list{j+1] = list[j]; } list[j+1] = temp; long partition(long left, long right) { int pivot_element = list[left]; int Ib = left, ub = right; int temp; while (left < right) { while(list[left] <= pivot_element) left++; while(list[right] > pivot_element) right--; if (left < right) { temp = list[left]; list[left] = list[right]; list[right] = temp; } list[Ib] = list[right]; list[right] = pivot_element; return right; void quickSort(long left, long right) { if (left < right) { long pivot = partition(left, right); quickSort(left, pivot-1); quickSort(pivot+1, right); int main() { double t1, t2; for (length = 1000; length <= max_length; ) { cout << "\nLength\t: " << length << "\n'; read(); t1 = clock(); bubbleSort(); t2 = clock(); cout << "Bubble Sort\t: " << (t2 - t1)/CLK_TCK <<" sec\n"; read(); t1 = clock(); insertionSort(); t2 = clock(); cout << "Insertion Sort\t: " << (t2 - t1)/CLK_TCK <<" sec\n"; read(); t1 = clock(); quickSort(0, length - 1); t2 = clock(); cout << "Quick Sort\t: "<< (t2 - t1)/CLK_TCK << "sec\n"; switch (length) { case 1000: length = 5000; break; case 5000: length = 10000; break; } return 0;