Branch-and-Bound Algorithm: Decomposing Feasible Domains and Finding Optimal Solutions, Study notes of Mathematics

Download Branch-and-Bound Algorithm: Decomposing Feasible Domains and Finding Optimal Solutions and more Study notes Mathematics in PDF only on Docsity! 3. Branch and Bound. Having studied LP relaxations of IPs and how to solve them using the simplex algorithm, we are now ready to apply these techniques in a framework designed to solve completely general integer programming problems: LP based branch-and-bound. The main idea of branch and bound consists in dividing a computationally hard problem into easier subproblems and systematically exploit the information gained from solving these subproblems. It is thus a divide and conquer approach. Proposition 3.1. Consider the problem z = max{cTx : x ∈ F}, where F denotes the set of feasible solutions (as we saw, F is usually defined im- plicitly via constraints). If F can be decomposed into a union of simpler sets F = F1 ∪ · · · ∪ Fk and if z[j] := max{cTx : x ∈ Fj} (j = 1, . . . , k), then z = max j z[j]. Let us work through an example to see how such decompositions might arise in practice: Example 11. Let F be the set of feasible tours of the travelling salesman problem on a network of 4 cities. Let node 1 be the departure city. F can be subdivided F = F(1,2) ∪ F(1,3) ∪ F(1,4) into the disjoint sets of tours that start with an arc (1, 2), (1, 3) or (1, 4) respectively. Each of the sets F(1,2), F(1,3) and F(1,4) can be further subdivided according to the choice of the second arc, F(1,2) = F(1,2)(2,3) ∪ F(1,2)(2,4) etc. Finally, we see that each of these sets corresponds to a specific TSP tour and cannot be further subdivided. We have found an enumeration tree of the TSP tours. We see that F was decomposed on a first level, and then each of the constituent parts was further decomposed on a further level and so on. Also, each Fj corresponds to a branch (or subtree) of the enumeration tree. Thus, Proposition 3.1 allows us to decompose a hard problem into a possibly large number of easier branch problems, and to find an optimal solution of F by comparing the solutions found for the branch problems. Of course, for even quite moderately sized problems such a tree can no longer be explicitly enumerated, as the number of leaves grows exponentially in the problem size. The idea of implicit enumeration is based on building up the enumeration tree as we explore it, and to prune certain parts that are not worth looking at before those parts are even generated. The pruning mechanisms are based on the following insight: Proposition 3.2. Consider the problem z = max{cTx : x ∈ F}, 28 F F(1,2) F(1,3) F(1,4) F(1,2)(2,3) F(1,2)(2,4) F(1,3)(3,2) F(1,3)(3,4) F(1,4)(4,2) F(1,4)(4,3) Fig. 3.1. The implicit enumeration tree of Example 11. and let F = F1∪ · · ·∪Fk be a decomposition of its feasible domain into smaller sets. Let z[j] ≤ z[j] ≤ z[j] be lower and upper bounds on z[j] = max{cTx : x ∈ Fj} for all j. Then z := max j z[j] ≤ z ≤ max j z[j] =: z gives an upper and lower bound on z. Proof. For all j, z =≥ z[j] ≥ z[j], so z ≥ maxj z[j]. Furthermore, z = maxj z[j] ≤ maxj z[j]. 3.1. Pruning Mechanisms. We speak of “pruning” a branch when we detect that we need no longer explore it further. This can happen for a variety of reasons. 1. Pruning by Bound: A branch Fj can be pruned when z[j] ≤ z. 2. Pruning by Infeasibility: If Fj = ∅, then the corresponding branch can be pruned. Why would an empty subset Fj appear in the decomposition of F? Remember that we don’t set up the decomposition tree explicitly but use an implicit enumeration, typically by introducing more and more constraints as we move towards the leaves of the enumeration tree. As we proceed, the constraints may become incompatible and correspond to an empty set Fj , but this may not be a-priori obvious and has to be detected. For example, in the case where the bounds z[j], z[j] are computed via LP relaxation, the simplex algorithm will detect if the LP relaxation is infeasible, and then Fj is empty as well. 3. Pruning by Optimality: When z[j] = z[j] for some j, then the branch corre- sponding to Fj needs no further consideration, since an optimal solution z[j] = z[j] = 29 F F1 F2 59 7 −∞ x1 ≤ 2 x1 ≥ 3 Fig. 3.4. The partial enumeration tree after the first fractional branching. Of course, futher down the tree we can use the same branching rule to subdivide any Fj thus generated. In our case this leads to the two subproblems (F1) z = max4x1 − x2 s.t. 7x1 − 2x2 ≤ 14 x2 ≤ 3 2x1 − 2x2 ≤ 3 x1 ≤ 2 x ∈ Z2+. and (F2) z = max4x1 − x2 s.t. 7x1 − 2x2 ≤ 14 x2 ≤ 3 2x1 − 2x2 ≤ 3 x1 ≥ 3 x ∈ Z2+. Note that upon solving the LP relaxations of these two problems we will have max{z[1], z[2]} < z, since x1 would have to be allowed to take the value 20 7 for the value z to be attained. Thus, branching on a fractional variable guarantees that the upper bounds decrease strictly in subsequent iterations, unless there exist multiple optimal solutions for the parent LP relaxation. So far we obtain the partial enumeration tree of Figure 3.4. The branches dangling off the nodes F1 and F2 still need to be explored. We mark these nodes as active. The node F on the other hand has been processes and is inactive. 3. Choosing an an active node for processing: During the run of the algorithm, we maintain a list of active nodes. This list currently consists of F1, F2. Later we will 32 discuss breadth-first versus depth-first choices of the next active node to be processed. For now we arbitrarily select F1. 4. Bounding: Next we derive a bound z[1] by solving the LP relaxation (P1) z = max4x1 − x2 s.t. 7x1 − 2x2 ≤ 14 x2 ≤ 3 2x1 − 2x2 ≤ 3 x1 ≤ 2 x ≥ 0 of problem (F1) z[1] = max{cTx : x ∈ F1}. Note that (P1) differs from (P ) by having one more constraint x1 ≤ 2 imposed. In the next section we will see that such problems can be solved via a warm start procedure that works out as follows: Using x1 = 207 − 1 7x3 − 2 7x4 from the optimal primal dictionary of (P ), express the constraint x1 ≤ 2 as 20 7 − 1 7 x3 − 2 7 x4 ≤ 2. Introduce a new slack variable x6 and add the constraint to the dictionary. The old dual solution is still dual feasible for the new dictionary x1 = 20 7 − 1 7 x3 − 2 7 x4 x2 = 3 − 0 · x3 − x4 x5 = 23 7 + 2 7 x3 − 10 7 x4 x6 = − 6 7 + 1 7 x3 + 2 7 x4 z = 59 7 − 4 7 x3 − 1 7 x4. After just a few pivots on the dual dictionary and translating the optimal dual dictionary back into a primal dictionary, we obtain x1 = 2 − x6 x2 = 1 2 + 1 2 x5 + x6 x3 = 1 + x5 + 5x6 x4 = 5 2 + 1 2 x5 + 6x6 z[1] = 15 2 − 1 2 x5 − 3x6. Therefore, z[1] = 152 , ( x[1]1 , x [1] 2 ) = ( 2, 12 ) . 5. Branching: ( x[1]1 , x [1] 2 ) is still not integral. Hence, F1 is not solved to optimality and cannot be pruned. Instead, we need to branch further. 33 F F1 F2 F11 F12 59 7 −∞ 15 2 x1 ≤ 2 x1 ≥ 3 x2 = 0 x2 ≥ 1 Fig. 3.5. The partial enumeration tree after the second fractional branching. Using the same approach as before, we introduce F11 = F1 ∩ {x : x2 ≤ 0} = F1 ∩ {x : x2 = 0} (since x2 ≥ 0), F12 = F1 ∩ {x : x2 ≥ 1}. We now arrive at the partial enumeration tree of Figure 3.5, and the new list of active nodes is F11, F12, F2. 6. Choosing an active node: We arbitrarily choose F2 for processing. 7. Bounding: We compute a bound z[2] by solving the LP relaxation (P2) z = max4x1 − x2 s.t. 7x1 − 2x2 ≤ 14 x2 ≤ 3 2x1 − 2x2 ≤ 3 x1 ≥ 3 x ≥ 0 of the problem (F2) z [1] = max{cTx : x ∈ F2}. Proceeding the same way as above, we express the new constraint x1 ≥ 3 in terms of the optimal nonbasic variables of (LP). We had found x1 = 20/7− 1/7x3 − 2/7x4. Thus, the new constraint is 1 7 x3 + 2 7 x4 ≤ − 1 7 . 34 3.4. LP Based Branch and Bound: The General Case. Let us now de- scribe the LP based branch-and-bound algorithm for general IP instances. We start with a variant that only uses dual bounds. Algorithm 3.3. [LP based branch-and-bound] AN = {F}, z = −∞, x∗ = ∅ (initialisation) while AN .= ∅, repeat choose a problem Fj ∈ AN and apply simplex alg. to x[j] := arg max{cTx : x ∈ Pj} if simplex alg. found Pj = ∅ set AN = AN \ {Fj} elseif simplex alg. found LP relaxation unbounded set z[j] = z = +∞ and AN = (AN \ {Fj}) ∪ {F [1] j , F [2] j } (branch- ing) else (x[j] is finite) set z[j] = cTx[j], z = max(z, z[j]) if z[j] ≤ z set AN = AN \ {Fj} elseif x[j] is feasible for F set z = z[j], x∗ = x[j] and AN = AN \ {Fi : z[i] ≤ z} else (Fj not pruned) set AN = (AN \ {Fj}) ∪ {F [1] j , F [2] j } (branching) end end end. Remarks: The algorithm is constructed to either stop with an optimal incumbent x∗, or with x∗ = ∅ as certificate of infeasibility of the problem. In the first case the final bounds satisfy z = z. The efficiency of the algorithm can be improved if we also compute a lower bound for Fj (using a heuristic) and use this to update the global lower bound z. In LP-based branch-and-bound the LP relaxation of each subproblem is only slightly different from that of its parent problem. In this situation the simplex algo- rithm does not have to be started from scratch. Instead, it is much more economical to use a warm start from a basic feasible solution constructed from the optimal dic- tionary of its parent. We will discuss this technique in the next section. 37 Algorithm 3.4. [Variant with lower bounds] AN = {F}, z = −∞, x∗ = ∅ (initialisation) while AN .= ∅, repeat choose a problem Fj ∈ AN and apply simplex alg. to x[j] := arg max{cTx : x ∈ Pj} if simplex alg. found Pj = ∅ set AN = AN \ {Fj} else using heuristic, compute lower bound z[j] set z = max{z, z[j]} and AN = AN \ {Fi : z[i] ≤ z} if simplex alg. found LP relaxation unbounded set z[j] = z = +∞ and AN = (AN \ {Fj}) ∪ {F [1] j , F [2] j } (branching) else (x[j] is finite) set z[j] = cTx[j], z = max(z, z[j]) if z[j] ≤ z set AN = AN \ {Fj} elseif x[j] is feasible for F set z = z[j], x∗ = x[j] and AN = AN \ {Fi : z[i] ≤ z} else (Fj not pruned) set AN = (AN \ {Fj}) ∪ {F [1] j , F [2] j } (branching) end end end end. 4. Improved Branch and Bound Systems. Let us now discuss an array of tools that can be used to design more efficient branch-and-bound algorithms. 4.1. Warm-Starting the LP Relaxations. In LP-based branch-and-bound the LP relaxation of each subproblem is only slightly different from that of its par- ent problem. In the example of Section 3.3 we have seen that in this situation the simplex algorithm does not have to be started from scratch. Instead, it is much more economical to use a warm start from a basic feasible solution constructed from the optimal dictionary of its parent. Let us now discuss this technique in more detail, distinguishing four different cases. Let x∗, y∗ be optimal solutions for the primal and dual LP instances (P) max x cTx (D) min y bTy s.t. Ax ≤ b, s.t. ATy ≥ c, x ≥ 0 y ≥ 0. 4.1.1. The objective function changes. (P’) max x c̃Tx s.t. Ax ≤ b, x ≥ 0, 38 Note that x∗ is still primal feasible. A few simplex iterations can therefore be applied to the primal dictionary corresponding to x∗ to find the new primal optimal solution x̃∗. The new dual optimal solution ỹ∗ can then be read off the new optimal dictionary. Example 12. Let us once again consider the LP instance (P) max5x1 + 4x2 + 3x3 s.t. 2x1 + 3x2 + x3 ≤ 5 4x1 + x2 + 2x3 ≤ 11 3x1 + 4x2 + 2x3 ≤ 8 x1, x2, x3 ≥ 0 for which we already found the optimal dictionary x3 = 1 + x2 + 3x4 − 2x6 x1 = 2 − 2x2 − 2x4 + x6 x5 = 1 + 5x2 + 2x4 z = 13 − 3x2 − x4 − x6 corresponding to the primal optimal solution x∗1 = 2, x ∗ 3 = 1, x ∗ 2 = 0. Let us now change the objective function to z = 6x1 + 5x2 + 2x3. This can be expressed in terms of the nonbasic variables z = 6(2 − 2x2 − 2x4 + x6) + 5x2 + 2(1 + x2 + 3x4 − 2x6) = 14 − 5x2 − 6x4 + 2x6. Thus, for the changed problem the dictionary corresponding to x∗1 = 2, x ∗ 3 = 1, x∗2 = 0 is as follows, x3 = 1 + x2 + 3x4 − 2x6 x1 = 2 − 2x2 − 2x4 + x6 x5 = 1 + 5x2 + 2x4 z = 14 − 5x2 − 6x4 + 2x6. This dictionary is not optimal, as the nonbasic variable x6 appears with the positive coefficient 2. Thus we use x6 as a pivot and increase it from zero to 1 2 = min (1 2 , 2, +∞ ) . The new dictionary is x6 = 1 2 + 1 2 x2 − 1 2 x3 + 3 2 x4 x1 = 5 2 − 3 2 x2 − 1 2 x3 − 1 2 x4 x5 = 1 + 5x2 + 2x4 z = 15 − 4x2 − x3 − 3x4. 39 4.2.1. Preprocessing. LP or IP models can often be simplified by reducing the number of variables and constraints, and IP models can be tightened before any ac- tual branch-and-bound computations are performed. Example 14. Consider the LP instance max 2x1 + x2 − x3 s.t. 5x1 − 2x2 + 8x3 ≤ 15 8x1 + 3x2 − x3 ≥ 9 x1 + x2 + x3 ≤ 6 0 ≤ x1 ≤ 3 0 ≤ x2 ≤ 1 1 ≤ x3. Tightening bounds: Isolating x1 in the first constraint and using x2 ≤ 1, −x3 ≤ −1 yields 5x1 ≤ 15 + 2x2 − 8x3 ≤ 15 + 2 × 1 − 8 × 1 = 9, and hence, x1 ≤ 9/5, which tightens the bound x1 ≤ 3. Likewise, isolating x3 in the first constraint, and using the bound constraints, we find 8x3 ≤ 15 + 2x2 − 5x1 ≤ 15 + 2 × 1 − 5 × 0 = 17. This implies x3 ≤ 17/8 and tightens x3 ≤ ∞. And finally, isolating x2 in the first constraint, 2x2 ≥ 5x1 + 8x3 − 15 ≥ 5 × 0 + 8 × 1 − 15 = −7 yields x2 ≥ −7/2 which does not tighten x2 ≥ 0. Proceeding similarly with the second and third constraints, we obtain the tight- ened bound 8x1 ≥ 9 − 3x2 + x3 ≥ 9 − 3 + 1 = 7, yielding the improved bound x1 ≥ 7/8. As some of the bounds have changed after the first sweep, we may now go back to the first constraint and tighten the bounds yet further. Isolating x3, we obtain 8x3 ≤ 15 + 2x2 − 5x1 ≤ 15 + 2 − 5 × 7 8 = 101 8 , yielding the improved bound x3 ≤ 101/64. Continuing the second sweep by isolating each variable in turn in each of the con- straints 1–3, and using the bound constraints, several bound constraints may further tighten in general, but not in the present example. 42 How many sweeps of this process are needed? One can show that after two sweeps of all the constraints and variables, the bounds cannot improve any further! Redundant Constraints: Using the final upper bounds in constraint 3, x1 + x2 + x3 ≤ 9 5 + 1 + 101 64 < 6, so that this constraint is redundant and can be omitted. The remaining problem is max 2x1 + x2 − x3 5x1 − 2x2 + 8x3 ≤ 15 8x1 + 3x2 − x3 ≥ 9 7 8 ≤ x1 ≤ 9 5 , 0 ≤ x2 ≤ 1, 1 ≤ x3 ≤ 101 64 . Variable fixing: • Increasing x2 makes the objective function grow and loosens all constraints except x2 ≤ 1. Therefore, in an optimal solution we must have x2 = 1. • Decreasing x3 makes the objective function grow and loosens all constraints except 1 ≤ x3. Thus, in an optimal solution we must have x3 = 1. This leaves the trivial problem max { 2x1 : 7 8 ≤ x1 ≤ 9 5 } . Example 14 shows how to simplify linear programming instances. In the prepro- cessing of IPs we have further possibilities: • For all xj with an integrality constraint xj ∈ Z any bounds lj ≤ xj ≤ uj can be tightened to *lj+ ≤ xj ≤ (uj). • For binary variables new logical or Boolean constraints can be derived that tighten the formulation and hence lead to fewer branching nodes in a branch- and-bound procedure. The latter point is illustrated in the next example: Example 15. Consider a BIP instance whose feasible set is defined by the fol- lowing constraints, 7x1 + 3x2 − 4x3 − 2x4 ≤ 1 −2x1 + 7x2 + 3x3 + x4 ≤ 6 −2x2 − 3x3 − 6x4 ≤ −5 3x1 − 2x3 ≥ −1 x ∈ B4. Generating logical inequalities: The first constraint shows that x1 = 1 ⇒ x3 = 1, which can be written as x1 ≤ x3. 43 Likewise, x1 = 1 ⇒ x4 = 1, or equivalently, x1 ≤ x4. Finally, constraint 1 also shows that the problem is infeasible if x1 = x2 = 1. Therefore, the following constraint must hold, x1 + x2 ≤ 1. We can process the remaining constraints in a similar vein: Constraint 2 yields the inequalities x2 ≤ x1 and x2 +x3 ≤ 1, constraint 3 yields x2 +x4 ≥ 1 and x3 +x4 ≥ 1, and constraint 4 yields x1 ≥ x3. Although the introduction of the new logical constraints makes the problem seem more complicated, the formulation becomes tighter and thus better. Furthermore, we can now process the problem further: Combining pairs of logical inequalities: We now consider pairs involving the same variables: The inequalities x1 ≤ x3 and x1 ≥ x3 yield x1 = x3, x1 + x2 ≤ 1 and x2 ≤ x1 yield x2 = 0, and then x2 + x4 ≥ 1 yields x4 = 1. Simplifying: Substituting the identities x2 = 0, x3 = x1 and x4 = 1 we found, all four constraints become redundant. We are left with the choice x1 ∈ {0, 1}, and hence the feasible set contains only two points S = {(1, 0, 1, 1), (0, 0, 0, 1)}. 4.2.2. Selection of Algorithm to Solve the LPs. The simplex algorithm implemented in a branch-and-bound system typically lets the user choose some pa- rameters. Likewise, interior-point LP solvers can be fine-tuned via certain parameters. We cannot discuss this issue further in the framework of this course. 4.2.3. Node Selection. Depth-First: A depth-first strategy aims at finding a feasible solution quickly, by only choosing an active node that is a direct descendant of the previously processed node. This strategy also makes it easier to warm-start the LP calculations in each iteration. Best-Node-First: To minimise the total number of nodes processed during the run of the algorithm, the optimal strategy is to always choose the node with the largest upper bound, i.e., Sj such that z[j] = max{z[i] : Si ∈ List}. Under this strategy we will never branch on a node St whose upper bound z[t] is smaller than the optimal value z of S. This is called a best-node-first strategy. The depth-first and best-node-first strategies are usually mutually contradictory, so a compromise has to be reached. Usually, depth-first is used initially until a feasible solution is found and a lower bound z is established. 44