Nathan Brixius : combinatorial optimization

Solver Foundation on Hanselminutes

Nathan Brixius — Wed, 09 Dec 2009 15:57:00 GMT

The latest edition of Scott Hanselman's Hanselminutes podcast features yours truly. We talked about Solver Foundation, numerical optimization, and good old-fashioned code optimization. Scott's a great guy and doing the podcast was a lot of fun.

Those of you new to Solver Foundation (or this blog) may be curious about what you can actually do with Solver Foundation. Here is the download location for the free Express version of Solver Foundation. I hope that by listening to the podcast you already know the basics, so here are a few code-oriented posts:

Here is a short description of “what’s new” in our 2.0 release.
Here is a C# sample that solves traveling salesman problems.
Here is a LINQ to SQL sample using Solver Foundation.
Here is an F# sample. The F# integration features a domain specific language for creating optimization problems. It also supports “units of measure”, meaning you can associate real-life units such as miles, gallons, dollars, or touchdowns to parameters and decisions.
Erwin did a really nice post on doing portfolio management using Solver Foundation and Excel. I like the visuals. In fact Erwin's site has a number of great samples and documents.

And I'd be remiss if I didn't mention that Solver Foundation ships with TONS of samples that span languages, scenarios, and solvers.

The last part of our conversation was about writing high-performance code. I'll share some thoughts about that later this week.

Int'l Symposium on Mathematical Programming, Day 2

Nathan Brixius — Fri, 28 Aug 2009 06:22:00 GMT

Here are my notes from the second day of ISMP. (Here are my day 1 notes.) I spent the whole day going to IPM talks:

Meszaros: Unfortunately I missed the plenary talk by Friedrich Eisenbrand due to microbrews. I attended an IPM session after that - first up was Csaba Meszaros - interesting to hear from him because BPMPD has been such an impressive code for so many years. His talk was about recent improvements in BPMPD. BPMPD has been in continuous development for 15 years or so. His recent work appears to focus on a) QCQP b) exploiting "hypersparsity". He is using a primal-dual log barrier method - not a primal-dual HSD method as we at Solver Foundation have chosen for our SOCP solver. So his handling of starting points and infeasibility detection is different. He went into the tradeoffs between using augmented and normal systems for the search direction, which affect both performance and numerical stability. He has implemented both and has logic on top to determine which to use. He also talked about QCQP presolve, which often whittles down the size of "real" models substantially.

An interesting bit of trivia: he showed a table that compared lines of code in various modules, "then and now":

 	2009	1981
Ordering	5700	155
Cholesky	2825	100
Backsolve	820	45

I think the difference is indicative of both Csaba's commitment to improving BPMPD as well as the number of practical issues you need to think about when building an industrial grade solver.

Bliek (IBM): Next up - IPM session chair Christian Bliek from IBM. His talk was about barrier improvements in CPLEX. For the single threaded case, CPLEX 12 is about 5% faster than 11 [note that these numbers are totally unofficial and any errors are my own]. With 4 threads it is about 20% better. There was no magic bullet here, more tuning at the solver and algebra layer. With a sophisticated solver such as CPLEX the improvements come through hard work, no tricks. There was also some data on simplex vs. barrier. For their complete set of their benchmark LPs, for problems that take 100 seconds or more barrier (without crossover) beats dual simplex on average. If you allow crossover, then performance vs dual simplex is about even for small problems (~1s solution time) and after that crossover really takes over. For problems that take 1000s and up, the geometric mean for the performance ratio against dual is 0.55. So almost twice as fast.

For QP, barrier is way better than their active set approach. Of course this does not take into account scenarios where warmstart could be used, where simplex rocks. More on that later. They also have a "concurrent" setting, meaning to race the simplex & IPM solvers. This setting gives the best results. Solver Foundation Services has also supported this type of "transparent parallelism" since version 1.0. The last bit of the talk was about some experiments using PARDISO instead of the solver's own algebra stack. Their conclusion was that using a "black box" linear algebra module shows potential.

Warmstart: The last talk in the session was by Anthony Vannelli, who talked about IPM warmstart. He takes a very pragmatic approach and layed out a very sensible procedure for attacking the difficult problem of using the solution of an IPM problem to seed the solution of a related problem. The net was that using a relatively simple warmstart procedure they were able to get (roughly) a factor of 3 speedup for their test set. There were also some other IPM warmstart researchers in attendance - in particular Alper Yilidrim and Hande Benson. Alper had a great survey talk of existing IPM warmstart methods, and Hande is one of the top researchers in the area. I am sure there have been interesting discussions this week!

Erling Andersen (Mosek): The afternoon IPM session had talks by Erling Andersen from Mosek, Jacek Gondzio, and myself. Erling talked about SOCP improvements in MOSEK 6 (which will be out later this year). He's had SOCP support forever, but for version 6 he has refactored and reworked much of the implementation. He has run into problems getting high accuracy solutions (I feel his pain), and he has had some problems with infeasible or nearly infeasible problems. He had some interesting thoughts on infeasibility criteria, rooted in his experience with real problems. He's going to get about a 15% speedup, which is quite impressive since he is already starting with a very fast solver. Erling has built a great set of solvers and I wish him well with Mosek 6.

Jacek Gondzio talked about the use of indirect methods (such as preconditioned conjugate gradient) in IPM. This hasn't worked out well in the past because such methods are not able to solve the systems to the accuracy required by IPM search directions. And the problem is still not solved, but for his new approach he did show some impressive results for dense problems.

The title of my talk was Interior Point Methods in Solver Foundation. I gave an overview of where IPM sits in the Solver Foundation stack, discussed our user and solver models. (And why different user and solver models are necessary - the "user model" serves Solver Foundation Services as well as programmers, whereas the solver model is structured so as to attain the most numerically stable, high performance solver possible. Separation of concerns!) Then I gave a high-level overview of our solution approach, and talked about a couple of implementation issues including handling of free variables, equality constraints, etc. Lastly I talked about some numerical stability issues. I got nice feedback from people afterwards and there was a lot of curiosity about who is using our stuff and what is next.

Int'l Symposium on Mathematical Programming, Day 1

Nathan Brixius — Wed, 26 Aug 2009 16:35:00 GMT

I am attending the ISMP math programming conference in Chicago, representing the Solver Foundation team. Here are some of my impressions of Day 1 at ISMP:

Plenary [Steven Boyd]: The plenary speaker was Steve Boyd who talked about real-time embedded convex optimization. He considered cases where problems need to be solved in milliseconds or less: control, signal processing, resource allocation, and even finance (think flash trading). The approach is basically to extend "disciplined convex programming". DCP is a modeling system where problems are convex "by construction" because they combine operators with well-known properties; the system is then able to rewrite the problem in the standard form required by a convex programming solver (such as an IPM QP/SOCP solver). The new step is that the system can now generate highly optimized C code for a custom solver for the particular problem family described by the model. (The algorithm itself is standard IPM.) You can do all sorts of optimizations in this case: the problem structure is known so the symbolic ordering can be done in advance, you can ensure better memory locality, etc. He gave many example where small QP instances were solved in tensof microseconds. Fun stuff.

Morning Sessions: I attended a morning session concerning approaches for nonconvex optimization problems. Jon Lee talked about solving a particular class of parametric nonlinear problems, Kurt Anstreicher talked about *nonconvex* QP and associated bounds. I then went to a straight-up theory session talking about convergence rates & asymptotic costs for IPM - there were a couple of new results and an interesting "corrector-predictor" (instead of the other way around) approach that attains better asymptotic convergence. There is an embarrassment of riches at ISMP: I missed by John Hooker about integrating MIP, constraint & global optimization, a great MIP session including Bob Bixby from Gurobi, and all sorts of other stuff. There are a staggering number of good talks to attend.

Afternoon Sessions [Modeling / Stochastic]: I went to a modeling languages track in the afternoon. The first speaker was from LINDO Systems who talked about their new LINGO offering. The primary new feature is to support stochastic modeling. There were some screenshots from What's Best, their spreadsheet solver. They had a few nice samples as well.

Gautam Mitra from OptiRisk was up next - he talked about SAMPL, stochastic extensions for AMPL. They have extended SAMPL to support:

Chance constraints
Integrated chance constraints
Robust optimization

Chance constraints are interesting - they occur in a range of financial problems including VaR and CVaR calculation. Gautam riffed on the difference between stochastic and robust optimization by invoking Rumsfeld: there are known knowns [deterministic optimization], known unknowns [stochastic], and unknown unknowns [robust optimization]. In robust optimization you may have problem coefficients that are in some range with an unknown distribution and you want the constraints to hold in all (or most) possible circumstances. Robust optimization also has SOCP reformulations, and I am a big believer because I think RO can be cleanly expressed in modeling languages. The last talk in the session was about YALMIP support for robust optimization - YALMIP has had widespread adoption by control theory specialists and has had a long history of improvements. This talk was very code-oriented and fun to watch.

A few overall impressions: I think it is fair to say that SOCP is entering the mainstream with LP/QP/MIP - it came up a lot today. On the modeling side, it was interesting to see that stochastic was featured in all three talks in the session I attended. A larger trend is that MINLP (mixed integer nonlinear programming) is hot. There are tons of tracks on it, and it is [unsurprisingly] being taken on by both the MIP and NLP communities. The problem description is very general so of course there are applications, but the dust has not settled on the solver side.

Quadratic Assignment Problems: solution enumeration

Nathan Brixius — Mon, 27 Jul 2009 19:53:00 GMT

In the last few posts we have given code for computing the Gilmore Lawler bound for quadratic assignment problems, and described two different branching techniques. In this post we show how to enumerate solutions of a small QAP. The possible solutions to a QAP range over permutations over unassigned facilities and locations. If lower bounds do not increase after repeated branching, we may eventually assign nearly all the facilities to locations. If there are only 2 or 3 remaining, we may as well just enumerate all the possible solutions and pick the best. In practice, we hope that we will rarely have to do this, otherwise the branch-and-bound tree will be too large to process efficiently. Here is a procedure that enumerates over all solutions of a size 3 QAP. Iterating over permutations is a standard textbook problem, so maybe you can write cleaner code than I did here!

/// <summary>
/// Assume that all facilities have been assigned to locations,
/// except three. EnumerateSolutions enumerates all six remaining assignments
/// returns the best.
/// </summary>
private double EnumerateSolutions(Qap q, BranchAndBoundNode node, int[] pBest) {
double objBest;
int nPerm = perm3.GetLength(0);
int n = perm3.GetLength(1);
double[] obj = new double[nPerm];
int[] r = node.p.UnusedIndices().ToArray();
int[] c = node.pInv.UnusedIndices().ToArray();
int[] p = (int[])node.p.Clone();
Debug.Assert(r.Length == n, "Only call me for size " + n + " QAPs.");

// Try all the possible permutations.
for (int i = 0; i < nPerm; i++) {
    for (int j = 0; j < n; j++) {
      p[r[j]] = c[perm3[i, j]];
    }
    obj[i] = q.Evaluate(p);
}

// Get the best permutation and objective.
int best = obj.ArgMin(out objBest);
p.CopyTo(pBest, 0);
for (int j = 0; j < n; j++) {
    pBest[r[j]] = perm3[best, j];
}
return objBest;
}

If you combine the code from the following posts you will see that we almost have a working B&B solver:

In my next post I will fill in a couple of the gaps, and then I will re-post all of the code. Then we'll think about what we can actually do with a QAP solver!

Quadratic Assignment Problems: strong and weak branching

Nathan Brixius — Mon, 20 Jul 2009 05:19:00 GMT

Let's pick up where we left off last time and write a score-based Branch delegate. It will be used as the basis for both of our "real" branching functions.

private BranchingDecision BranchCore(BranchAndBoundNode node, double bound, double[][] S) {
RowColumnSums(S, node.Size);
double rowVal, colVal;
int rowBest = _rowSum.ArgMax(out rowVal);
int colBest = _colSum.ArgMax(out colVal);
if (rowVal > colVal) {
return new BranchingDecision(true, rowBest);
}
else {
return new BranchingDecision(false, colBest);
}
}

private void RowColumnSums(double[][] U, int size) {
if (_rowSum == null) {
    _rowSum = new double[_qap.Size];
    _colSum = new double[_qap.Size];
}
_rowSum.ConstantFill(Double.MinValue);
_colSum.ConstantFill(Double.MinValue);

for (int i = 0; i < size; i++) {
    for (int j = 0; j < size; j++) {
      _rowSum[i] += U[i][j];
      _colSum[j] += U[i][j];
    }
}
}

If we can create a reasonable score matrix and pass it into BranchCore, then we've got something. Two common techniques for QAP are:

Weak branching. Let S_ij be the reduced cost U_ij that we got from computing the lower bound. (U_ij is a lower bound on the amount that the bound on the subproblem will increase. So large U_ij means the subproblem will be a lot easier.)
Strong branching. Let S_ij be the bound for subproblem ij. The bigger the bound, the easier the subproblem.

Weak branching is trivial to implement for the Gilmore-Lawler (or GLB) bound. As I mentioned a long time ago, the reduced cost matrix essentially comes for free.

private BranchingDecision WeakBranch(BranchAndBoundNode node, double bound, double[][] U) {
return BranchCore(node, bound, U);
}

For strong branching we need to compute bounds for each subproblem. We overwrite the U matrix and pass it into BranchCore. Since the U matrix is used to fathom subproblems later on in the code, we subtract the parent bound from the bound of each subproblem.

private BranchingDecision StrongBranch(BranchAndBoundNode node, double bound, double[][] U) {
Qap r = new Qap(node.Size - 1);
double[][] Ur = MatrixUtilities.NewMatrix(node.Size - 1, node.Size - 1);
for (int i = 0; i < node.Size; i++) {
    for (int j = 0; j < node.Size; j++) {
      Qap.Reduce(node.Qap, r, i, j);
      U[i][j] = GLB.Bound(r, Ur) - bound;
    }
}
return BranchCore(node, bound, U);
}

Strong branching in general produces better guesses that weak branching. However, it comes at a computational cost. Therefore I want to introduce one last complication…er…feature. Since strong branching is smarter but more costly, we should only use it where it is most needed. I want to have a set of rules that governs when to use strong or weak branching. Branching decisions are most important at the top of the tree, or on nodes where the bound sucks. We can measure the suckiness of the bound for a node by computing the relative gap: the ratio of the gap between the bound and the incumbent solution, and the corresponding gap at the root. So our branching rule data structure will have Depth and Gap properties, as well as properties that store the Branch delegate information.

/// <summary>A description of a branching rule: when and how.
/// </summary>
public class BranchingRule {
/// <summary>The ID for this rule.
/// </summary>
public int Index { get; set; }
/// <summary>Maximum depth for which the rule applies.
/// </summary>
public int Depth { get; set; }
/// <summary>Maximum relative gap for which the rule applies.
/// </summary>
public double Gap { get; set; }
/// <summary>Strong or Weak.
/// </summary>
public BranchType Type { get; set; }
/// <summary>StrongBranch() or WeakBranch().
/// </summary>
internal Branch Rule { get; set; }

public BranchingRule(int index, double gap, int depth, BranchType type) {
    Index = index;
    Depth = depth;
    Gap = gap;
    Type = type;
}

public override string ToString() {
    return "Depth = " + Depth + ", Gap = " + Gap + ", Type = " + Type.ToString();
}
}

We will have an array of BranchingRule called _branchingRule. Given a node, we will select the first rule in the array whose depth is larger than the node depth, and whose gap is larger than the node gap.

private Branch SelectBranchingRule(BranchAndBoundNode node) {
double gap = _results.RelativeGap(node.LowerBound);
BranchingRule criterion = _branchingRules.First(entry => (node.Level <= entry.Depth) && (gap >= entry.Gap));
node.RuleIndex = criterion.Index;
return criterion.Rule;
}

I am cheating here because I have not defined the local variable _results. We'll get to that in a few posts. For now, just assume that RelativeGap is defined as I described above: (Objective - bound) / (Objective - RootBound).

At long last, that covers branching. There are many variations, but they are essentially variations of strong or weak branching. At this point we only have two more main topics to cover: what to do when you get to the bottom of the tree, and how do you capture and update results.

Quadratic assignment problems: branching

Nathan Brixius — Mon, 13 Jul 2009 02:24:00 GMT

(This is part of a long-running series on quadratic assignment problems in C#. Click here to catch up.)

Branching means to divide the search space represented by a node into subnodes. QAP is about making assignments of facilities to locations, so two possible approaches come to mind:

Branch into two subproblems: one where an assignment i -> j is allowed, and another where it is disallowed.
Pick a facility and try assigning it to all possible locations, creating N subproblems. (Or pick a location and assign all facilities to it…)

The second approach, called polytomic branching, turns out to be more effective. The problem with the first ("single assignment branching") is that the subproblems aren't that much easier than the parent problem. You'll have to branch a whole bunch of times before you are able to determine anything.

We still need to decide whether to fix a facility ("row branching") or a location ("column branching"). And after that we need to figure out which particular facility or location is best. At this point you may be wondering, "does it even matter?" The answer is that, yes, branching decisions are earth shatteringly important. Where's what I mean: if you make bad branching choices it could take a thousand times longer to solve the problem, or more! This is part of the reason why QAP (and branch-and-bound) are so interesting. Bad branching decisions lead to subproblems that are just about as hard as the original problem. It's kind of like this. Here is a data structure that represents the results of a polytomic branching decision.

/// <summary>A branching decision, the result of applying a branching rule.
/// </summary>
internal class BranchingDecision {
/// <summary>If true, enumerate over all unassigned rows.
/// </summary>
public bool IsRowBranch { get; set; }
/// <summary>The row or column index.
/// </summary>
public int Index { get; set; }
/// <summary>Create a new instance.
/// </summary>
public BranchingDecision(bool isRowBranch, int index) {
IsRowBranch = isRowBranch;
Index = index;
}
}

Ultimately we want to write methods that produce BranchingDecisions. The following delegate represents such a method (we will write two methods in the next blog post):

/// <summary>Makes a branching decision.
/// </summary>
internal delegate BranchingDecision Branch(BranchAndBoundNode node, double bound, double[][] U);

If you accept the claim that branching choices make a big difference, then it is clear that we want to branch so we get the easiest subproblems possible. That is, nodes that will be processed by the branch-and-bound algorithm as quickly as possible. We don't know how to measure that directly (why?), so we have to guess. A general framework for polytomic branching is to compute a score for each of the N^2 subproblems, and store it in a matrix. S_ij is the score for the subproblem created by fixing facility i to location j. Then compute the row and column sums of S, and branch on the row or the column with maximum sum. Making a branching choice reduces to computing scores on subproblems.

The next step will be to write methods that compute scores and return BranchingDecisions. We'll look at two common ways to do that next time. We're about two or three posts away from having a pretty decent branch-and-bound solver for QAP.

Solver Foundation TSP Part II: Directives, Solver Plugins, Model Libraries

Nathan Brixius — Fri, 08 May 2009 06:47:00 GMT

Here is my as-promised "part 2" to my traveling salesman sample. My goal is not to try and do any "production level" modeling, but instead "point the way" by highlighting different aspects of Solver Foundation's architecture. This time I want to talk about directives, plug-in solvers, and model reusability. The Solver Foundation architecture provides the ability to plug-in other solvers to handle one or more problem types. As of version 1.1, the Gurobi solver is Solver Foundation's default MIP (mixed integer programming) solver. Gurobi is a high-performance solver which is capable of taking on industrial strength problems, so my piddling little TSP example is no match for it. Gurobi has a number of parameters that can be set to tune solver performance - you set them by means of Directives. First, let's modify our code to turn on logging. Change the context.Solve() call to:

      GurobiDirective gurobiDirective = new GurobiDirective();
      gurobiDirective.OutputFlag = true;
      Solution solution = context.Solve(gurobiDirective);

Each plug-in solver also defines its own directives - therefore the full power and flexibility of plug-in solvers are available to you as a programmer. In this case I have just flipped on the logging bit - here's what I see.

Optimize a model with 185 Rows, 380 Columns and 1199 NonZeroes
Presolve removed 1 rows and 185 columns
Presolve time: 0.30s
Presolved: 184 Rows, 195 Columns, 832 Nonzeros
Objective GCD is 1

Root relaxation: objective 2.783286e+03, 36 iterations, 0.02 seconds

    Nodes    |    Current Node    |     Objective Bounds      |     Work
 Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

     0     0  2783.2857    0   14          -  2783.2857     -      -    0s
     0     0  3006.1964    0   29          -  3006.1964     -      -    0s
     0     0  3006.1964    0   29          -  3006.1964     -      -    0s
     0     0  3006.1964    0   25          -  3006.1964     -      -    0s
     0     0  3006.1964    0   25          -  3006.1964     -      -    0s
H    0     0                       6029.0000  3006.1964  50.1%     -    0s
     0     0  3006.1964    0   25  6029.0000  3006.1964  50.1%     -    0s
H    0     2                       3805.0000  3006.1964  21.0%     -    0s
     0     2  3006.1964    0   25  3805.0000  3006.1964  21.0%     -    0s
*   75    54              29       3644.0000  3077.0000  15.6%   8.2    0s
*  129    80              27       3381.0000  3077.0000  8.99%   8.7    0s
*  158    62              30       3323.0000  3077.0000  7.40%   8.1    0s

Cutting planes:
  Learned: 3
  Gomory: 1
  MIR: 3

Explored 1231 nodes (5693 simplex iterations) in 0.69 seconds
Thread count was 2 (of 2 available processors)

Optimal solution found (tolerance 1.00e-04)
Best objective 3.3230000000e+03, best bound 3.3230000000e+03, gap 0.0000%

For large models it is sometimes useful to tweak the solver options for performance. For example, if I wanted to spend a slightly larger amount of time doing MIP heuristics, and I wanted to eliminate presolve, then I can set those options easily:

      gurobiDirective.Heuristics = 0.1;
      gurobiDirective.Presolve = PresolveLevel.None;

There is a full reference in the Gurobi Solver Programming Primer that ships with Solver Foundation. My point is that you do not get a "least common denominator" solution - you get all the bells and whistles.

Gurobi is just one example - there are a wide range of plug-ins available, covering most of the popular LP and MIP solvers - check out solverfoundation.com for the full list. To use a plug-in solver, you need to: 1) install the solver (duh), 2) change your exe.config or web.config to point to the solver. No code changes are required. You can find additional documentation on the default MIP directives and the third-party Solver configuration in the SFS Programming Primer.

Last thing - modern programming languages like C# are cool because they promote reusability and maintainability. Don't forget about that when you are writing SFS code. It's common for models in a particular vertical (say, finance or transportation) to have common "blocks" of goals or constraints that are re-used over and over again. It's pretty easy to build reusable model libraries using extension methods. For example, take the assignment constraints in my TSP example. I could factor them out:

  public static class ModelingExtensions {
    public static void AssignmentConstraintsNoDiag(this Model model, Set s, Decision assign) {
      model.AddConstraint("A1", Model.ForEach(s, i => Model.Sum(Model.ForEachWhere(s, j => assign[i, j], j => i != j)) == 1));
      model.AddConstraint("A2", Model.ForEach(s, j => Model.Sum(Model.ForEachWhere(s, i => assign[i, j], i => i != j)) == 1));
    }
  }

Now I can just say model.AssignmentConstraintsNoDiag(city, assign). I could add other variations inside of ModelingExtensions as well. I think this aspect of Solver Foundation has not gotten enough attention - maybe because it doesn't make for a cool demo!

Solving traveling salesman problems using Solver Foundation

Nathan Brixius — Mon, 27 Apr 2009 18:18:00 GMT

Here's an example that I walked through during yesterday's INFORMS session. Erwin has two blog postings about Solver Foundation and the traveling salesman problem, but I want to throw in my two cents because I want to emphasize a couple of points:

By combining C# and Solver Foundation Services it is possible to express complex models clearly and succinctly.
It is very easy to build powerful, reusable model libraries using C# and Solver Foundation Services.
Solver Foundation Services code can be used in many different application environments (ASP.Net, silverlight, DB, command line apps, WPF, …) with minimal changes.

The traveling salesman problem is a classical problem in computer science, and you should bow your head in shame if you don't know about it (and turn in your conference badge if you happen to be in Phoenix). A salesperson needs to make a tour of a number of cities. The restrictions are that she wants to visit each city once and only once, and she wants to minimize the distance travelled. This is perhaps the definitive example of an NP-hard problem.

TSP can be solved using mixed integer programming - optimizing a linear goal with linear constraints, where some of the decision variables are integer. In this first post I will show how to formulate and solve a TSP model using Solver Foundation Services. In my second post I will show how to use the Gurobi MIP solver using SFS. There are many different ways to model the TSP - here is a nice introduction. My goal is to provide a clear, complete example - not build a "production level" TSP model, so I am going to choose a model formulation that dates back to 1960! First, I need to establish a couple of building blocks that will help me construct the data for the model. We need to know the distances between each pair of cities. Typically we are provided the coordinates of the cities and need to derive the distances. So I will introduce a Coordinate class that contains properties for the (x, y) coordinates, and properties to convert to latitude and longitude. Finally, a method that computes the distance between points.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Microsoft.SolverFoundation.Services;
using SolverFoundation.Plugin.Gurobi;

namespace Microsoft.SolverFoundation.Samples.TravelingSalesman {
  class Program {
    // TSP coordinate.
    public class Coordinate {
      public int Name { get; set; }

      // X-coordinate (from TSPLIB)
      public double X { get; set; }

      // Y-coordinate (from TSPLIB)
      public double Y { get; set; }

      public Coordinate(int name, double x, double y) {
        Name = name;
        X = x;
        Y = y;
      }

      // Latitude in radians.
      public double Latitude {
        get { return Math.PI * (Math.Truncate(X) + 5 * (X - Math.Truncate(X)) / 3) / 180; }
      }

      // Longitude in radians.
      public double Longitude {
        get { return Math.PI * (Math.Truncate(Y) + 5 * (Y - Math.Truncate(Y)) / 3) / 180; }
      }

      // Geographic distance between two points (as an integer).
      public int Distance(Coordinate p) {
        double q1 = Math.Cos(Longitude - p.Longitude);
        double q2 = Math.Cos(Latitude - p.Latitude);
        double q3 = Math.Cos(Latitude + p.Latitude);
        // There may rounding difficulties her if the points are close together...just sayin'.
        return (int)(6378.388 * Math.Acos(0.5 * ((1 + q1) * q2 - (1 - q1) * q3)) + 1);
      }
    }

    // TSP city-city arc.
    // 
    public class Arc {
      public int City1 { get; set; }
      public int City2 { get; set; }
      public double Distance { get; set; }
    }

    // Burma14 from TSPLIB. Optimal tour = 3323.
    private static Coordinate[] data = new Coordinate[] {
      new Coordinate(0, 16.47, 96.10),
      new Coordinate(1, 16.47, 94.44),
      new Coordinate(2, 20.09, 92.54),
      new Coordinate(3, 22.39, 93.37),
      new Coordinate(4, 25.23, 97.24),
      new Coordinate(5, 22.00, 96.05),
      new Coordinate(6, 20.47, 97.02),
      new Coordinate(7, 17.20, 96.29),
      new Coordinate(8, 16.30, 97.38),
      new Coordinate(9, 14.05, 98.12),
      new Coordinate(10, 16.53, 97.38),
      new Coordinate(11, 21.52, 95.59),
      new Coordinate(12, 19.41, 97.13),
      new Coordinate(13, 20.09, 94.55)
    };

(The data for this 14-city problem comes from the TSPLIB library). If you've been following my blog you know that the building blocks of a Solver Foundation model are: sets, parameters, decisions, goals, and constraints. I am going to implement a simple formulation that is centered around the following (indexed) decisions:

Assign[i,j]: this is equal to 1 if the optimal tour contains a trip (or arc) from city i to city j.
Rank[i]: this is equal to the number of cities visited after arriving at city i.

We have one parameter in our model:

Distance[I,j]: the distance from city i to city j.

With that in mind, here's the model. Explanation of the goals and constraints follow.

    static void Main(string[] args) {
      SolverContext context = SolverContext.GetContext();
      Model model = context.CreateModel();

      // ------------
      // Parameters
      Set city = new Set(Domain.IntegerNonnegative, "city");
      Parameter dist = new Parameter(Domain.Real, "dist", city, city);
      var arcs = from p1 in data
                 from p2 in data
                 select new Arc { City1 = p1.Name, City2 = p2.Name, Distance = p1.Distance(p2) };
      dist.SetBinding(arcs, "Distance", "City1", "City2");
      model.AddParameters(dist);

      // ------------
      // Decisions
      Decision assign = new Decision(Domain.IntegerRange(0, 1), "assign", city, city);
      Decision rank = new Decision(Domain.RealNonnegative, "rank", city);
      model.AddDecisions(assign, rank);

      // ------------
      // Goal: minimize the length of the tour.
      Goal goal = model.AddGoal("TourLength", GoalKind.Minimize,
        Model.Sum(Model.ForEach(city, i => Model.ForEachWhere(city, j => dist[i, j] * assign[i, j], j => i != j))));

      // ------------
      // Enter and leave each city only once.
      int N = data.Length;
      model.AddConstraint("assign 1",
        Model.ForEach(city, i => Model.Sum(Model.ForEachWhere(city, j => assign[i, j],
          j => i != j)) == 1));
      model.AddConstraint("assign 2",
        Model.ForEach(city, j => Model.Sum(Model.ForEachWhere(city, i => assign[i, j], i => i != j)) == 1));
      model.AssignmentNoDiag(city, assign);

      // Forbid subtours (Miller, Tucker, Zemlin - 1960...)
      model.AddConstraint("no subtours",
        Model.ForEach(city,
          i => Model.ForEachWhere(city,
            j => rank[i] + 1 <= rank[j] + N * (1 - assign[i, j]),
            j => Model.And(i != j, i >= 1, j >= 1)
          )
        )
      );

      Solution solution = context.Solve();

      // Retrieve solution information.
      Console.WriteLine("Cost = {0}", goal.ToDouble());
      Console.WriteLine("Tour:");
      var tour = from p in assign.GetValues() where (double)p[0] > 0.9 select p[2];
      foreach (var i in tour.ToArray()) {
        Console.Write(i + " -> ");
      }
      Console.WriteLine();
    }
  }

In my humble opinion, the "Parameter data =" line is an awesome example of the power of LINQ data binding in Solver Foundation.  We generate the 2D matrix of distances using a single LINQ expression.  It would be incredibly easy to change the code to retrieve the coordinate data from a database (perhaps using a LINQ expression once again), a file, or even a user application.

The goal is straightforward: minimize the distance traveled.  This is a product of the selected arcs and the distance matrix.   We have two types of constraints:

Assignment constraints: these ensure that we enter and leave each city only once.

Subtour constraints: these ensure that we do not have any subtours.  In a four city problem {A, B, C, D}, for example, we cannot have two cycles (A, B), (C, D).  We need to have one tour that contains all the cities.

The assignment constraints are easy using the ForEach and ForEachWhere operations.  I use ForEachWhere because I want to disallow arcs that enter and leave the same city - that doesn't make sense.  The subtour constraint is a little more complicated.   It relates the "assign" and "rank" decisions.  The key fact is that if there is an arc from city i to city j, rank[i] + 1 == j.  Of course, if the (i, j) arc is not part of the optimal tour then all bets are off.   Last note: notice that I can mix parameters, decisions, and C# variables in my expressions.

Once we solve the model we want to print out the cost, and the optimal tour. Getting the cost is very easy using goal.ToDouble(). We can get the tour using either Assign or Rank. I have chosen to use Assign because it gives me another opportunity to use LINQ. When you call GetValues() on a decision, you get arrays that contain the value along with the indexes for each decision. In this case, the last entry in the array is the one we are interested in. There are other ways to conveniently query decsision results, I'll save that for another time.

The next post will show how we can use Solver Foundation's plug-in model to tune the behavior of the Gurobi MIP solver.

Solver Foundation 1.2 released!

Nathan Brixius — Sun, 19 Apr 2009 06:48:00 GMT

I am pleased to announce that Solver Foundation v1.2 is live on solverfoundation.com! Our goal in 1.2 was to make a few key improvements to address feedback that we have received from partners and customers. If you have feedback, questions, suggestions…please post it here, or on our MSDN forum. Positive or negative - that's okay. I am particularly interested in hearing how you use Solver Foundation in your application - or what features you would like to see.

Back to 1.2:

The popular FICO Xpress is now a "certified partner". That means if you are an Xpress user, you can use Solver Foundation's plug-in model to write .Net applications without a hitch.
Amsterdam Optimization is a certified modeling consultant for Solver Foundation. I've linked to Erwin's blog before - he's a member of this group. We are happy to be partnering with them.
MIP start functionality has been added. This allows you to give a starting solution to a MIP model using SetInitialValue on Decision.
Bug fixes - in particular we've taken care of a few QP issues.

Resource constrained scheduling; OML tutorials

Nathan Brixius — Wed, 01 Apr 2009 05:43:00 GMT

Two weeks back I posted two articles showing how easy it is to model critical path scheduling using Microsoft Solver Foundation. I received a few emails asking about various extensions; I will be covering those in upcoming posts. Julian just wrote a great blog post that covers the most commonly requested extension - resource constrained scheduling. If you are getting started with Solver Foundation and want to see an interesting, instructive example, I encourage you to check out his post. Two things that I really like about his OML:

He separates the data from the model description using Parameters.
He relies on the Foreach construct to define his constraints. It results in a very clean model definition.

Julian mentions it in his post, but I also want to call out another great resource for learning OML. Erwin Kalvalagen has written an OML tutorial which includes several interesting examples. It's a great complement to the Excel Programming Primer that is part of the Solver Foundation documentation.

Simple matrix, vector, permutation utilities

Nathan Brixius — Wed, 25 Mar 2009 07:24:00 GMT

I am going to be rolling out the rest of my branch-and-bound algorithm in the next few posts. To make that easier, in this post I introduce some common matrix, vector, and permutation methods. It turns out that for technical computing applications, C#'s extension methods (introduced in 3.0) are awesome. With vectors it's great because you retain the control of having direct array access, but you get the nice object-oriented notation.

I've left out comments and error-checking, and it's not super-optimized, but you get the idea. None of these methods end up being the bottleneck. Most of the methods are obvious, but I do want to comment on the permutation methods. The goal of QAP is to find an optimal assignment of facilities to locations, represented as a permutation. In the course of our branch-and-bound algorithm, we'll be making partial assignments - that is, only some of the entries in the permutation will be filled in. By convention, p[i] == -1 will mean that facility i is unassigned. An index where p[i] < 0 is called "unused". When we branch, we'll want to pick an unused facility (or location), and try assigning all unused locations (or facilities) to it. The use of C#'s iterator and yield concepts really help here.

That said, here's the code.

  public static class MatrixUtilities {
    #region Vector
    public static void ConstantFill(this T[] data, T val) {
      for (int i = 0; i < data.Length; i++) {
        data[i] = val;
      }
    }

    public static int ArgMin(this double[] data, out double best) {
      if (data.Length == 0) {
        best = Double.MinValue;
        return -1;
      }
      int iBest = 0;
      best = data[0];
      for (int i = 1; i < data.Length; i++) {
        if (best > data[i]) {
          best = data[i];
          iBest = i;
        }
      }
      return iBest;
    }

    public static int ArgMax(this double[] data, out double best) {
      if (data.Length == 0) {
        best = Double.MinValue;
        return -1;
      }
      int iBest = 0;
      best = data[0];
      for (int i = 1; i < data.Length; i++) {
        if (best < data[i]) {
          best = data[i];
          iBest = i;
        }
      }
      return iBest;
    }
    #endregion

    #region Permutation
    public static void Swap(this int[] p, int i, int j) {
      int temp = p[i];
      p[i] = p[j];
      p[j] = temp;
    }

    public static int FindUnused(this int[] data, int index) {
      foreach (int unused in data.UnusedIndices()) {
        if (index-- <= 0) {
          return unused;
        }
      }
      return -1;
    }

    public static IEnumerable UnusedIndices(this int[] data) {
      for (int i = 0; i < data.Length; i++) {
        if (data[i] < 0) {
          yield return i;
        }
      }
    }

    public static string PermutationToString(this int[] p, bool oneBased) {
      StringBuilder build = new StringBuilder(p.Length * 4);
      int width = ((int)Math.Log10(p.Length)) + 2;
      build.Append("[");
      for (int i = 0; i < p.Length; i++) {
        int index = oneBased ? p[i] + 1 : p[i];
        build.Append(index.ToString().PadLeft(width));
      }
      build.Append("]");
      return build.ToString();
    }
    #endregion

    #region Matrix

    public static string MatrixToString(this double[][] A) {
      if (A != null) {
        StringBuilder build = new StringBuilder(A.Length * A.Length * 3);
        for (int i = 0; i < A.Length; i++) {
          if (A[i] != null) {
            for (int j = 0; j < A[i].Length; j++) {
              build.AppendFormat("{0,4}", A[i][j]);
              build.Append(" ");
            }
            build.AppendLine();
          }
        }
        return build.ToString();
      }
      return null;
    }

    public static double[][] NewMatrix(int m, int n) {
      double[][] M = new double[m][];
      for (int i = 0; i < M.Length; i++) {
        M[i] = new double[n];
      }
      return M;
    }

    public static double[][] NewMatrix(int n) {
      return NewMatrix(n, n);
    }
    #endregion
  }

Branch-and-bound algorithms for QAP

Nathan Brixius — Mon, 23 Mar 2009 23:22:00 GMT

This is part 4 in a series of posts laying out a simple branch-and-bound solver for QAP in C#. Last time (several months ago!) I provided a simple bounding procedure for QAP. I want to take a step back and give the high-level algorithm, then in subsequent posts I will fill in the implementation.

Branch-and-bound algorithms are a general framework for solving combinatorial optimization problems. The idea is to repeatedly subdivide the search space into smaller subproblems of the same time. For each subproblem we compute bounds, which may allow us to determine that the subproblem cannot lead to an improved solution to the original problem.

  /// Solves QAP using branch-and-bound.
  /// 
  public class BranchAndBound {

    private readonly Qap _qap;
    private BranchAndBoundResults _results;

    /// Create a new instance.
    /// 
    public BranchAndBound(Qap qap) {
      _qap = qap;
    }

    /// Solve a QAP to optimality.
    /// 
    public void Solve() {

      _results = new BranchAndBoundResults();
      Stack stack = new Stack();
      stack.Push(new BranchAndBoundNode(_qap));

      while (stack.Count > 0) {
        BranchAndBoundNode node = stack.Pop();

        if (node.LowerBound <= _results.Objective) {
          if (node_is_easy_to_solve) {
            EnumerateSolutions(node, _results);
          }
          else {
            node.LowerBound = GLB.Bound(node.Qap, U);

            if (node.LowerBound <= _results.Objective) {
              foreach (BranchAndBoundNode subNode in Branch(node, U).OrderBy(n => n.LowerBound)) {
                stack.Push(subNode);
              }
            }
          }
        }
      }
    }
  }

  /// A subproblem in a branch-and-bound algorithm.
  /// 
  public class BranchAndBoundNode {
    private readonly Qap _qap;
    /// The size of the subproblem.
    /// 
    public int Size {
      get { return _qap.Size; }
    }

    /// The lower bound of the subproblem.
    /// 
    public double LowerBound { get; set; }

    /// The QAP subproblem.
    /// 
    public Qap Qap {
      get { return _qap; }
    }
  }

  /// Branch-and-bound solution information.
  /// 
  public class BranchAndBoundResults {
    public int[] p { get; set; }
    public double Objective { get; set; }
  }

Let's work through the pseudocode. Our stack will contain the current set of subproblems that need to be solved. We initialize the stack with our QAP instance, wrapping it inside a BranchAndBoundNode data structure. We'll develop this data structure as we go, but from the code we can already see two uses:

It stores the subproblem (a QAP),
It stores the lower bound computed by the code from my last post.

The other thing we do is allocate a data structure which stores the results: the best assignment found so far, and the best objective value. In the main loop of Solve() we repeatedly pop nodes from the stack. We first compare the node's lower bound to the best objective (also called the incumbent). If the lower bound exceeds the incumbent value, there is no way this subproblem can lead to a better result, so we need not process it. The incumbent value may have changed since the subproblem was created. Then we check to see if the subproblem is "easy" - if so, we can simply enumerate over all possible solutions for the subproblem and update the results if needed. Otherwise, we use our GLB code to compute a bound. Again, we check the lower bound to see if we can eliminate (or fathom) the subproblem. If not, we need to subdivide the node into subproblems, and push them onto the stack.

The top-level algorithm itself is not all that complicated! It's also quite general. We have already described how to compute bounds, so all we need to do is specify:

How to enumerate solutions for small problems.
How to branch - i.e. how to subdivide BranchAndBoundNodes.

The motivation and code for enumerating and branching will be the subject of the next two posts.

Microsoft Solver Foundation version 1.1 released

Nathan Brixius — Wed, 18 Feb 2009 18:34:00 GMT

Microsoft Solver Foundation version 1.1 has been released and the Express version can be downloaded for free here. 1.1 is a big deal because of the introduction of our solver plug-in model. This allows new or existing 3rd party solvers to interface with Solver Foundation Services (SFS) directly - meaning that existing OML models, C#, or other .Net application code will work without changes. As improvements in solver technology emerge, your application code will benefit without a single line of code being changed. We did not just throw the plug-in model over the wall - we have partnered with some of the top names in the field to provide certified plug-ins. The Gurobi MIP solver is included in this release as the default MIP solver (recent, independent benchmarks for Gurobi are available here). We also provide a Mosek partner wrapper. In addition, we provide reference plug-ins (with source code) for CPLEX, Mosek, Xpress-MP, and lp_solve. If you have one of these solvers, it's very easy to drop in the plug-in and get the benefits of Solver Foundation Services: declarative modeling, transparent parallelism, a fully-featured .Net API.

We have also made a number of other improvements. Solver Foundation now supports SOS2 (special ordered sets) - an ordered set of non-negative decisions where at most two consecutive values are nonzero. There is a complete description of the syntax with examples in the Excel Programming Primer document (Appendix 4). We're shipping a number of new examples, including more data binding and ASP.Net samples. Finally, we've improved the performance across the board. In particular, I am proud to say that the performance of our interior-point solvers has improved by 20%, with bigger gains for some problems.

Give it a try and let me know what you think. Post on the Solver Foundation forum with questions or comments, we want to hear from you. 2009 will be a big year for Solver Foundation and it's great to kick it off with this release.

Solver Foundation used to schedule Microsoft Tech Days '09

Nathan Brixius — Thu, 22 Jan 2009 06:57:00 GMT

Pascal Belaud of Microsoft France recently used Microsoft Solver Foundation to schedule the big Microsoft Tech Days '09 conference in sunny Paris, France. The conference has over 16,000 attendees, with hundreds of presentations involving hundreds of speakers and rooms - quite a challenge, but Pascal (with Solver Foundation's help) was up to the task! Solver Foundation was also featured in the keynote. It's quite an application and we were pleased to have the opportunity to work with Pascal. Read more about it (in French - but don't be scared, click the link) here.

In other recent news, we now have an official web presence, check out solverfoundation.com for an overview of what were about and where to get it. Version 1.1 will be on its way soon with many enhancements.

Announcing Microsoft Solver Foundation

Nathan Brixius — Fri, 14 Nov 2008 21:26:00 GMT

I'm pleased to announce the availability of the first version of Microsoft Solver Foundation! Simply put, Solver Foundation is a set of modeling tools and solvers to help you make smart decisions when confronted with complex requirements and priorities. A huge range of business and research problems can be modeled using Solver Foundation, including supply chain optimization, job scheduling, transportation network flows, risk analysis of investment portfolios, etc.

Solver Foundation consists of three types of services:

Programming and Modeling Services which allow .Net and Excel developers to easily integrate Solver Foundation into their applications.
Solver Foundation Services for model interchange, feasibility analysis, and declarative parallel solving.
Solver Runtime and Extensibility Services for plugging in third-party solvers.

Solver Foundation is 100% pure managed code and comes with a number of solvers right out of the box:

Revised, Simplex Linear and Mixed Integer Programming (Primal and Dual Simplex)
Interior Point Method Linear and Quadratic Programming
Constraint Programming with Exhaustive Tree Search, Local Search, and Metaheuristic Techniques
Compact, Quasi-Newton (L-BFGS), Unconstrained Nonlinear Programming

For more information, check out http://code.msdn.microsoft.com/solverfoundation. Click on the Downloads tab to get the Express Edition for free. Solver Foundation includes a bunch of samples for Excel, C#, F# to help you get started.

I'm a new member of the team and I am excited to be a part of the Solver Foundation effort. Before I started at Microsoft I worked on optimization software (interior point methods and combinatorial optimization in particular), so this is a huge passion of mine. In the upcoming months, I'll be blogging in detail about many aspects of Solver Foundation.