Nathan Brixius

Using Solver Foundation's Unconstrained Nonlinear Solver

2009-10-15T22:48:00Z

Hey, did you know that Solver Foundation can solve unconstrained nonlinear optimization problems? Yep, it's true. Few people know about it because 1) we haven't talked about it too much and 2) it is not accessible from the SFS (which means you can't use OML or the Excel add-in to solve such problems). To use the solver you need to work with a solver object directly. The Samples directory contains a simple example, but since there has been a couple of requests I thought I would give a slighly more involved example.

The CompactQuasiNewtonSolver class solves unconstrained optimization problems: it finds the minimum (or maximum) of an unbound objective function f(x). x is an array of doubles, and f needs to be twice differentiable. The CQN solver uses the L-BFGS algorithm. Like other implementations, you need to provide a function that will serve up the function value and gradient on demand, given the variables x. To do that, set the GradientAndValueAtPoint property to a delegate that fills in the gradient, and returns the function value. You'll also need to specify a starting point.

The CompactQuasiNewtonSolver has a Solve method that takes a solver parameters object. It has a few interesting properties, including the solver tolerance and maximum iteration count. On construction you can also pass in an "abort" delegate. This function will be called periodically during the course of execution - if you return true then the solver will return. In my sample below, I abuse it to print out the progress of the solver as it proceeds. Note that I would have liked to have used solver.NumberOfIterations rather than keeping track of the number of calls myself. That's a bug, and it will get fixed in our 2.1 release.

The function can have as many variables as you like. In my example I am minimizing a multidimensional variant of the Rosenbrock function, a standard test problem. You can modify the code to use your own objective function by modifying the ValueAndGradient method. You can also learn more about the solver by consulting the Solver Programming Primer (page 24 or so).

using System;
using Microsoft.SolverFoundation.Solvers;

namespace Microsoft.SolverFoundation.Samples {
  /// <summary> Minimizes the generalized Rosenbrock function, a standard test case
  /// in nonlinear optimization. You can read more about the Rosenbrock
  /// function here:
  ///  http://en.wikipedia.org/wiki/Rosenbrock_function
  /// </summary>
  public class Rosenbrock {
    public static void Main(string[] args) {

      int n = 4; // make sure I'm even, see below.
      double[] x = new double[n];
      for (int i = 0; i < x.Length; i++) {
        x[i] = -1;
      }

      CompactQuasiNewtonSolver solver = new CompactQuasiNewtonSolver(n);
      int iterationCount = 0;
      CompactQuasiNewtonSolverParams param = new CompactQuasiNewtonSolverParams(() => {
        Console.WriteLine("{0} | {1:e6}", /*solver.NumberOfIterations*/ iterationCount++, solver.ToleranceDifference);
        return false;
      }
        );
      
      solver.SetStartingPoint(x);
      solver.GradientAndValueAtPoint = (GetGradientAndValueAtPoint)ValueAndGradient;
      solver.Minimize = true;

      var solutionQuality = solver.Solve(param);

      Console.WriteLine();
      Console.WriteLine("Solution quality = " + solutionQuality);
      Console.WriteLine("f(x) = " + solver.SolutionValue);
      solver.GetSolutionPoint(x);
      for (int i = 0; i < x.Length; i++) {
        Console.WriteLine("x[{0}] = {1}", i, x[i]);
      }
    }

    /// <summary>Get the function value and gradient for the specified point.
    /// </summary>
    /// <param name="x">The point at which the value and gradient should be computed.</param>
    /// <param name="gradient">The gradient vector, the same size as x.</param>
    /// <returns>The function value.</returns>
    private static double ValueAndGradient(double[] x, double[] gradient) {
      // The gradient and value are both computed inside the method for efficiency.
      // It's also possible to refactor value and gradient calculation into submethods.
      double value = 0;
      for (int j = 0; j < x.Length; j += 2) {
        double t1 = 1.0 - x[j];
        double t2 = 10.0 * (x[j + 1] - x[j] * x[j]);
        gradient[j + 1] = 20.0 * t2;
        gradient[j] = -2.0 * (x[j] * gradient[j + 1] + t1);
        value += t1 * t1 + t2 * t2;
      }
      return value;
    }
  }
}

Announcing Brixius 2.0

2009-10-10T03:34:00Z

My wife gave birth to a healthy baby daugther last night! Mom, baby, (and dad) are all doing well. I will be spending some time at home with the newest addition to our family, so I will go easy on the blog posts for awhile. When I come back I will do one on our nonlinear solver, more about Solver Foundation 2.0, and finish up my QAP series.

Announcing Solver Foundation 2.0

2009-10-08T20:25:00Z

I'm pleased to announce the release of Solver Foundation 2.0, available at solverfoundation.com. We've added capabilities that allow Solver Foundation to solve important categories of optimization problems, improved the performance and stability our existing solvers, and improved our Visual Studio and Office tools to make model building and analysis easier. Here are a few of the highlights:

Excel 2007 add-in. We have completely overhauled the Excel 2007 add-in to make it easier to build, debug, and deploy models. Here is a screenshot from the new add-in:

As you can see, the building blocks of a model (Parameters, Decisions, Goals, Constraints) are built through a more task-centered user-interface, reducing the need for hand-coding OML. Data binding has been simplified, and decision (output) data binding is now supported. Models can include random parameters and recourse decisions to build stochastic models. Finally, the add-in now features the ability to deploy models directly to C# (for use within a Visual Studio project) or to OMLX (for use within SharePoint for server-side calculation). We're trying to simplify the modeling process and bridge the gap between modelers, developers, and business decision makers. Export to C# is this easy:

Simulation and stochastic programming. I previewed our stochastic modeling and solving features in a previous post. It is now possible to include random parameters in OML or SFS code to build stochastic models. Solver Foundation includes powerful sampling engines and decomposition techniques to solve large stochastic models quickly, and provides reporting information that tells you how the uncertainty affects the model.

MIP performance improvements. The new Gurobi 2.0 solver is included as the default MIP solver in Solver Foundation. The Gurobi solver's performance and functionality has been improved in a number of important ways, as described here. The combination of Solver Foundation Services' clean, powerful managed code API and Gurobi's world-class MIP solver gets even better in our 2.0. Our own in-house, purely managed MIP solver has been greatly improved as well.

Second Order Conic Programming. As I mentioned in this post, Solver Foundation 2.0's interior-point solver is now capable of solving large-scale second order conic programming models. SOCP models contain nonlinear conic constraints and are often countered in robust optimization and financial models.

Constraint Solver improvements. We have developed new metaheuristics and global constraints for our CP solver in conjunction with the gang in MSR Cambridge. Watch Lengning's blog for more details.

F# integration. We've heard a lot of positive noises about our F# integration - it's better in 2.0. Our F# optimization DSL has been improved and now incorporates units-of-measure as described in this post. Special thanks to the F# team for their help on this important piece of the puzzle.

And more. Submodel support, better SOS1/SOS2 support, improved reporting, improved model analysis and error messages, more solver plug-ins, better performance and memory usage across the board.

The release of Solver Foundation 2.0 is the culmination of one year of hard work by the team. I hope you will give it a try (the Express version is freely available) and provide your feedback either through the blog or through our MSDN forum. Getting this stuff right - and covering the scenarios that people care about - is hard work and it takes time. We've come a long way in a year, but there's so much more we want to do. As always, the team appreciates honest feedback, good and bad. We're serious about listening to that feedback to make Solver Foundation better.

Solver Foundation 2.0 Preview: Second Order Conic Programming

2009-10-02T07:32:00Z

A couple of weeks ago I wrote about one of our big features for Solver Foundation 2.0: stochastic programming. In this post I want to talk about a new solver for 2.0: an interior-point solver for Second Order Conic Programming (SOCP). Second Order Conic Programming is the problem of minimizing a linear function where there are linear terms constrained to be inside a second order cone. Also called conic quadratic programming, SOCP is important because it can be used to model many real-world robust optimization, engineering, and finance problems. You can even model certain types of chance constraints using SOCP. Here are a few favorite SOCP links:

SOCP on wikipedia.
Lobo, Vandenberghe, Boyd & Lebret's paper has several applications.
Alizadeh and Goldfarb's paper is also excellent.
Cornuejols and Tutuncu's book Optimization Methods in Finance has a couple of chapters on conic optimization.
If you want to get truly hardcore, Nemirovski and Vandenberghe have great notes posted online.

An SOCP can contain conventional linear constraints Ax >= b as well as second-order cone constraints. Some solvers define SOCP in terms of conic variables rather than conic constraints. Solver Foundation’s interior point solver uses conic constraints rather than conic variables because (IMHO) it leads to a more flexible, easy-to-program solver API. A second-order cone is a pointed cone where the norm of n - 1 variables t_i is less than the other variable t_0 (so that a cross-section is a circle): t_0 >= || t_i ||, t_0 >= 0. A second-order cone in three dimensions looks like an ice cream cone. A second-order conic constraint is similar, except that each t can be a linear term that may combine several variables. The first term t_0 is called the "primary row".

What this means is that if you know how to build linear programming problems using the ILinearModel API I described last time, then you pretty much know how to build SOCP problems. You need to build a row that represents the conic constraint using AddRow, then you need to call AddRow once for each term in the conic constraint. That's it.

Here's a short code sample: given an InteriorPointSolver, pass in a list of matrices F and a corresponding list of matrices g. We want to find a vector x that minimizes the sum of ||Fx + g|| over all entries in F and g. The steps are as follows:

Create an InteriorPointSolver and the lists F and g.
Create the descision variables x, and add the goal just as in my previous post.
Iterate through the matrices. We will create one conic inequality for each matrix. The primary row is an artificial variable t, which is an upper bound on the norm of Fx + g. The other rows are simply the rows of F, with the bounds coming from g.

/// <summary> Create a simple sum-of-norms problem from F and g.
/// </summary>
/// <remarks>Returns the variable vids so the solution vector x can be 
/// retrieved after Solve().</remarks>
private static int[] BuildSumOfNorms(InteriorPointSolver solver, 
  List<double[][]> F, List<double[]> g) {

  int[] x = new int[F[0][0].Length];
  for (int i = 0; i < x.Length; i++) {
    solver.AddVariable("x" + i, out x[i]);
  }

  // Add the goal.
  int goal;
  solver.AddRow("goal", out goal);
  solver.AddGoal(goal, 1, true);
  for (int k = 0; k < F.Count; k++) {
    int cone, t;
    // Add variable t_k, which is the upper bound on the kth norm.
    solver.AddVariable("t_" + k, out t);
    solver.SetCoefficient(goal, t, Rational.One);
    solver.SetBounds(t, Rational.Zero, Rational.PositiveInfinity);

    // Add a quadratic cone.
    solver.AddRow("cone_" + k, SecondOrderConeType.Quadratic, out cone);
    solver.SetBounds(cone, 0, Rational.PositiveInfinity);

    // Add the conic constraint
    int conicRow1;
    solver.AddRow("cr1_" + k, cone, SecondOrderConeRowType.PrimaryConic, 
      out conicRow1);
    solver.SetCoefficient(conicRow1, t, Rational.One);
    solver.SetLowerBound(conicRow1, Rational.Zero);

    for (int i = 0; i < F[k].Length; i++) {
      int conicRow2;
      solver.AddRow("cr2_" + k + "_" + i, cone, SecondOrderConeRowType.Conic, 
        out conicRow2);
      for (int j = 0; j < F[k][i].Length; j++) {
        if (F[k][i][j] != 0) {
          solver.SetCoefficient(conicRow2, x[j], F[k][i][j]);
        }
      }
      solver.SetLowerBound(conicRow2, -g[k][i]);
    }
  }

  return x;
}

Those are the basics. We will have SOCP analogues of the LinearModel APIs, as well as APIs that let you look at the details of the conic constraints. In Version 2.0 we will support creation and solution of SOCP only using the InteriorPointSolver API. We will not have SFS or OML support for this release.

Using the Solver Foundation Interior Point Solver

2009-09-25T00:24:00Z

I've had a few requests on this topic, so I thought I would give a simple example of how to write code against Solver Foundation's interior point solver. If you are new to Solver Foundation, be warned that this is something of an "advanced topic". The most complete, powerful Solver Foundation programming interface is Solver Foundation Services. The solverfoundation.com site has links to more information about the SFS, and I have worked through several samples on this blog: production planning, two-way data binding, traveling salesman, parameterized models, LINQ-to-SQL.

The InteriorPointSolver class is capable of solving linear, quadratic, and (in our upcoming 2.0 release) second order conic optimization problems. Let's take as an example the production planning problem I wrote about in a previous post. It is a simple linear programming problem with two decisions (the production for each refinery) and three constraints (the demand for each refined product). In our SFS sample we simply declared the decisions, goals, and constraints and expressed them algebraically using the SolverContext and Model. The InteriorPointSolver class represents linear models in a slightly different way: using variables and rows. Variables are the analogue of SFS Decisions, whereas rows can represent either goals or constraints. Both variables and rows can have upper and/or lower bounds. In a linear model, both goals and constraints consist of linear combinations of variables, such as "3*x + 4*y + 5*z". Whereas in the SFS we can express these terms very naturally, with the InteriorPointSolver, the terms are expressed one at a time by specifying the coefficient associated with each variable in the row. Here is the C# code:

  private static void PetrochemLinearModel() {
    InteriorPointSolver solver = new InteriorPointSolver();
    int sa, vz;
    solver.AddVariable("SA", out sa);
    solver.SetBounds(sa, 0, 9000);
    solver.AddVariable("VZ", out vz);
    solver.SetBounds(vz, 0, 6000);

    int goal;
    solver.AddRow("goal", out goal);
    solver.AddGoal(goal, 0, true);
    solver.SetCoefficient(goal, sa, 20);
    solver.SetCoefficient(goal, vz, 15);

    int demand1, demand2, demand3;
    solver.AddRow("demand1", out demand1);
    solver.SetCoefficient(demand1, sa, 0.3);
    solver.SetCoefficient(demand1, vz, 0.4);
    solver.SetLowerBound(demand1, 1126);

    solver.AddRow("demand2", out demand2);
    solver.SetCoefficient(demand2, sa, 0.4);
    solver.SetCoefficient(demand2, vz, 0.2);
    solver.SetLowerBound(demand2, 1500);

    solver.AddRow("demand3", out demand3);
    solver.SetCoefficient(demand3, sa, 0.2);
    solver.SetCoefficient(demand3, vz, 0.3);
    solver.SetLowerBound(demand3, 500);

    var param = new InteriorPointSolverParams();
    var solution = solver.Solve(param);
    Console.WriteLine("goal = " + solution.GetValue(goal).ToDouble());
    Console.WriteLine("sa   = " + solution.GetValue(sa).ToDouble());
    Console.WriteLine("vz   = " + solution.GetValue(vz).ToDouble());
  }

You'll also need to add "using Microsoft.SolverFoundation.Common;" and "using Microsoft.SolverFoundation.Solvers;" to the top of your program. Extending this code to solve a quadratic program is easy: there is an overload of SetCoefficient that takes two variables as input. In my next post I will talk about second order conic programming, and a few other improvements in the 2.0 version of the InteriorPointSolver.

Solver Foundation 2.0 Preview: F# ODSL

2009-09-14T07:08:00Z

Lengning Liu has written a great post that highlights another exciting Solver Foundation 2.0 area: a Solver Foundation ODSL in F#. (F# is a functional programming language for .Net.) The ODSL provides an intuitive, cool way to express LP and MIP problems. My favorite addition to the ODSL is units of measure. Parameters and decisions can be annotated with their units of measure, and the associated consistency checks are enforced by F# at compile time. This is a great way to provide clarity to models as well as enforce correctness. Here's an example from Lengning's post:

let a = 20.0<Dollar/Barrel>

let b = 15.0<Dollar/Barrel>

let sa = var<Barrel/Day>()

let vz = var<_>()

minimise (a * sa + b * vz)

where

[

0.3 * sa + 0.4 * vz >= 2000.<_>;

0.4 * sa + 0.2 * vz >= 1500.<_>;

0.2 * sa + 0.3 * vz >= 500.<_>;

sa <= 9000.<_>;

vz <= 6000.<_>;

sa >= 0.<_>;

vz >= 0.<_>

]

Nash equilibria and 4th down, continued

2009-09-12T07:30:00Z

I left a small cliffhanger in my last post. After a long week I finally had a chance to read through the Adams paper about estimating the value of "going for it" on 4th down. I admit I was a little bit let down. As a reminder - the question is what action a football team should take on fourth down. Failure to gain the necessary yards means the ball is turned over to the opposing side, kicking turns over the ball but with better field position, and making the first down allows the drive to continue, potentially leading to more points. The conclusion of the Romer paper was that coaches are too conservative and kick the ball away in situations where they should go for it instead.

Adams hits the nail on the head by asserting that the results of the Romer paper just do not pass the "smell test". It's nuts to suggest that it's a good idea to go for it on 4th and 4 on your own 25 yard line. But that leaves us only with more questions - is the conclusion of the Romer paper still valid, even if overstated? Can we identify a flaw in the reasoning? Is there a better way to model the problem?

Adams first suggestion for improving the model is to include more historical data. Adams and Romer both claim it's hard to come up with a good model for the "going for it" problem because teams seldom go for it on fourth down in practice - data is hard to come by. Romer and Adams both use game data from the 1998 - 2000 seasons, but Adams uses data from the entire game, not just the first quarter. But why not include more recent data? [The Adams paper was written in '06, so he could have doubled the data set. We have a couple more seasons-worth of data now.] So I'm not sure I even buy the premise that data is lacking.

Adams' second approach is to use Madden '07 to simulate 4th down situations. I initially thought this was a really cool idea, and it kind of is, and then I remembered something I once read. Madden himself asked the designers at EA to make 4th downs more difficult to convert! You cannot find a better example of Galbraith's notion of "conventional wisdom" in action. So as far as I am concerned, you have to throw out the middle section of the paper. Madden is not a simulation: it is pretending to be a simulation. It wants to make you feel like you are experiencing real NFL football. But the problem is that we as players do not make decisions the way that GMs, coaches, and players do. Our motivations are completely different, and there are no real consequences for our actions (other than bragging rights over your roommate). My GM will not fire me if I go for it on 4th and 5 on my own 25. Thus the game must be tuned to correct for this, otherwise you will get Tecmo-like gameplay.

The last section proposes a game-theoretic approach. Adams introduces a zero-sum game with the offense and defense as opponents. The offense and defense both have the choice of choosing a pass- or run-oriented strategy. The payoffs depend on their choices. Adams points out that this is a "simplified version of reality." (It's very close to the original Tecmo Bowl - two choices instead of four.) He uses this approach primarily to make the point that it is not a good idea (as Romer proposes) to use third down data to model fourth down choices, because the payoffs change enough to matter. It is an interesting line of argument for the claim that Romer's conclusions are overstated, but it does not provide insight into how to better model the problem.

Anyway, in the course of poking around the web I came across the ZEUS Football simulation engine. It is frequently referenced in the NYTimes "5th down" blog. For example, here is an interesting discussion about taking an intentional safety late in the game. (I won't bother to explain what that means, because if you have made it this far, you clearly already know what I am talking about.)

All the questions I raised at the beginning of this post are probably best answered by a simulation engine. Which reminds me - did I mention that Solver Foundation is adding stochastic capabilities for our version 2?

Dynamic programming, Nash equilibria, and going for it on 4th down

2009-09-09T01:22:00Z

Last weekend marked the first big college football Saturday of the year. The only game I really cared about (now that I am married and have kids) was the Northern Iowa – Iowa game: I went to Iowa, and I grew up in Cedar Falls (home of the UNI campus). Iowa came from behind and won 17-16 after blocking two field goal attempts in the final seconds. My brother and I were talking on the phone during the 4th quarter. My brother, a UNI fan, was bothered by the conservative coaching that he feels let Iowa back into the game, and that spun into a more general conversation about risk-averse coaching. I don't know anything about sabermetrics, but I did read Moneyball, and I love sports.

Our conversation reminded me of a paper by economist David Romer that I had always intended on reading: “It’s Fourth Down and What Does the Bellman Equation Say?” (I actually recommend this updated 2005 version.) So I read it. It received some attention from the sports world when it came out because Romer’s claim is that the conventional wisdom is wrong: it’s often a much better idea to try to convert on 4th down rather than kick the ball away to the other team. It depends on field position and yards-to-go, of course. He sets up a not-too-complicated dynamic programming model where he is able to place values on particular game situations, and then compares the difference between kicking and “going for it”. It’s interesting but I have some problems with it – in particular the use of 3rd down outcomes rather than 4th down outcomes. The justification is that because teams don’t go for it on 4th down very often there is not enough data, so 3rd down data is a reasonable substitute. I have issues with this because for one thing, playcalling is very different on 3rd down, especially when one is approaching field goal range. Players are also taught to handle 3rd down differently - throw the ball away and avoid taking a sack. Romer does address these sorts of issues, but it still bothers me.

To overcome the lack of 4th down “going for it” data, Christopher Adams from the FTC (!) uses Madden NFL 07 (!!) for simulation purposes and constructs a game theoretic model for 4th down attempts in this paper. He comes to a different conclusion: the conventional wisdom may not be so bad after all. I am looking forward to reading the Adams paper in detail - it's in my backpack. I hope to do some experimentation in this area once I get a grip on the concepts. I would like to write a paper about the prevent defense that Gregg Easterbrook and Bill Simmons despise so much! But right now, I’ve got to get back to work ;)

Solver Foundation 2.0 Preview: Simulation and Stochastic Programming

2009-09-03T19:28:00Z

I'm back from ISMP and helping to put the finishing touches on Solver Foundation 2.0. In our first version we introduced: our declarative modeling language OML; Solver Foundation Services, which provides a powerful, consistent .Net API for modeling and solving a wide range of decision problems; a set of solvers for linear, quadratic, mixed integer, constraint, and unconstrained nonlinear programming; third-party solver plug-in support; and an Excel add-in that provides easy access to all of Solver Foundation's functionality through Office 2007. Our aim in version 2.0 is to make it easier for non-experts to model and solve problems, and to expand the range of problems that can be solved using Solver Foundation. I will spend the next few posts talking about our plans for 2.0.

Many real-world problems involve uncertainty. For example, task durations, sales estimates, rate of return, and so on. Simulation and/or stochastic programming techniques are often used to account for randomness, but applying such techniques can often be quite tricky. In Solver Foundation 2.0 we are adding OML, Solver Foundation Services and solver support for linear stochastic models. The two new modeling concepts are:

Random parameters are used to model randomness. They can be discrete or continuous. They can be used where (non-random) parameters or constants would normally be used in constraints.

A special type of random parameter is a Scenarios parameter. In this case a list of scenarios are given where each scenario contains a value and a probability. For example, to model weather we might have a "rainy" scenario, a "normal" scenario, and a "dry" scenario.

Recourse decisions are decisions that are made in response to the realization of a random parameter. For example, if the weather turns out to be dry, we may not be able to produce enough of a certain crop and may need to purchase it from someone else. In this case, the purchase amount is a recourse decision. Recourse decisions are sometimes called "second-stage" decision because such decisions can be made only after the randomness is resolved.

As you will see in a moment, if you know how to build regular models using OML, building models with random parameters and recourse decisions will be no problem. Solver Foundation Services will contain new APIs that allow for stochastic modeling using random parameters and recourse decisions. Indexed random parameters and data binding will be fully supported. We will support the following distributions in V2:

Continuous: Normal (Gaussian), Uniform, Exponential, Log normal, Scenario-based.
Discrete: Uniform, Geometric, Binomial

Here is a simple OML example. It is a simple production planning problem with scalar input (here is a non-stochastic sample in C#). We introduce recourse decisions that represent the amount of pre-refined product to buy, in case demand cannot be met by production. The demand for each product is random; the demand for gas is modeled using two scenarios of equal probability, the others are uniformly distributed. Not very realistic, but I wanted to cram all the new concepts in a short model! Indexed random parameters are of course possible. In that case you will be able to use data binding to define the parameters associated with the distributions - keeping a clean separation between data and model, and allowing for easy experimentation.

Model[
Decisions[Reals[0, Infinity], SaudiArabia, Venezuela],
Decisions[Reals[0, Infinity],
// To define a recourse decision, just wrap Recourse[] around it.
Recourse[GasBuy], Recourse[JetFuelBuy], Recourse[LubricantBuy]
],

Parameters[
    // Here is a scenario-based random parameter with two scenarios of equal probability.
    Scenarios[Reals[0, Infinity]],
    GasDemand = {{0.5, 1900}, {0.5, 2100}}
],

// Here is a continuous random parameter.
Parameters[UniformDistribution[1500, 1600], JetFuelDemand],
Parameters[UniformDistribution[400, 500], LubricantDemand],

Goals[
    Minimize[ goal -> 20 * SaudiArabia+ 15 * Venezuela + (38.4 * GasBuy + 35.2 * JetFuelBuy + 28.8 * LubricantBuy) ]
],

Constraints[
    0.3 * SaudiArabia + 0.4 * Venezuela + GasBuy >= GasDemand,
    0.4 * SaudiArabia + 0.2 * Venezuela + JetFuelBuy >= JetFuelDemand,
    0.2 * SaudiArabia + 0.3 * Venezuela+ LubricantBuy >= LubricantDemand,
    SaudiArabia <= 9000,
    Venezuela <= 6000
]
]

The modeling constructs are simple, but underneath the hood there is a lot going on:

Solver Foundation Services supports different sampling methods including Monte Carlo and Latin Hypercube. Going beyond simple Monte Carlo often leads to better results in less time.
Large problems can be solved efficiently using Benders decomposition.
Solver Foundation Services will have API support for tuning the behavior of the stochastic solver.
Our primary focus is on providing a satisfactory experience “out-of-the-box”, without having to resort to lots of tuning.

Int'l Symposium on Mathematical Programming, Day 2

2009-08-28T06:22:00Z

Here are my notes from the second day of ISMP. (Here are my day 1 notes.) I spent the whole day going to IPM talks:

Meszaros: Unfortunately I missed the plenary talk by Friedrich Eisenbrand due to microbrews. I attended an IPM session after that - first up was Csaba Meszaros - interesting to hear from him because BPMPD has been such an impressive code for so many years. His talk was about recent improvements in BPMPD. BPMPD has been in continuous development for 15 years or so. His recent work appears to focus on a) QCQP b) exploiting "hypersparsity". He is using a primal-dual log barrier method - not a primal-dual HSD method as we at Solver Foundation have chosen for our SOCP solver. So his handling of starting points and infeasibility detection is different. He went into the tradeoffs between using augmented and normal systems for the search direction, which affect both performance and numerical stability. He has implemented both and has logic on top to determine which to use. He also talked about QCQP presolve, which often whittles down the size of "real" models substantially.

An interesting bit of trivia: he showed a table that compared lines of code in various modules, "then and now":

 	2009	1981
Ordering	5700	155
Cholesky	2825	100
Backsolve	820	45

I think the difference is indicative of both Csaba's commitment to improving BPMPD as well as the number of practical issues you need to think about when building an industrial grade solver.

Bliek (IBM): Next up - IPM session chair Christian Bliek from IBM. His talk was about barrier improvements in CPLEX. For the single threaded case, CPLEX 12 is about 5% faster than 11 [note that these numbers are totally unofficial and any errors are my own]. With 4 threads it is about 20% better. There was no magic bullet here, more tuning at the solver and algebra layer. With a sophisticated solver such as CPLEX the improvements come through hard work, no tricks. There was also some data on simplex vs. barrier. For their complete set of their benchmark LPs, for problems that take 100 seconds or more barrier (without crossover) beats dual simplex on average. If you allow crossover, then performance vs dual simplex is about even for small problems (~1s solution time) and after that crossover really takes over. For problems that take 1000s and up, the geometric mean for the performance ratio against dual is 0.55. So almost twice as fast.

For QP, barrier is way better than their active set approach. Of course this does not take into account scenarios where warmstart could be used, where simplex rocks. More on that later. They also have a "concurrent" setting, meaning to race the simplex & IPM solvers. This setting gives the best results. Solver Foundation Services has also supported this type of "transparent parallelism" since version 1.0. The last bit of the talk was about some experiments using PARDISO instead of the solver's own algebra stack. Their conclusion was that using a "black box" linear algebra module shows potential.

Warmstart: The last talk in the session was by Anthony Vannelli, who talked about IPM warmstart. He takes a very pragmatic approach and layed out a very sensible procedure for attacking the difficult problem of using the solution of an IPM problem to seed the solution of a related problem. The net was that using a relatively simple warmstart procedure they were able to get (roughly) a factor of 3 speedup for their test set. There were also some other IPM warmstart researchers in attendance - in particular Alper Yilidrim and Hande Benson. Alper had a great survey talk of existing IPM warmstart methods, and Hande is one of the top researchers in the area. I am sure there have been interesting discussions this week!

Erling Andersen (Mosek): The afternoon IPM session had talks by Erling Andersen from Mosek, Jacek Gondzio, and myself. Erling talked about SOCP improvements in MOSEK 6 (which will be out later this year). He's had SOCP support forever, but for version 6 he has refactored and reworked much of the implementation. He has run into problems getting high accuracy solutions (I feel his pain), and he has had some problems with infeasible or nearly infeasible problems. He had some interesting thoughts on infeasibility criteria, rooted in his experience with real problems. He's going to get about a 15% speedup, which is quite impressive since he is already starting with a very fast solver. Erling has built a great set of solvers and I wish him well with Mosek 6.

Jacek Gondzio talked about the use of indirect methods (such as preconditioned conjugate gradient) in IPM. This hasn't worked out well in the past because such methods are not able to solve the systems to the accuracy required by IPM search directions. And the problem is still not solved, but for his new approach he did show some impressive results for dense problems.

The title of my talk was Interior Point Methods in Solver Foundation. I gave an overview of where IPM sits in the Solver Foundation stack, discussed our user and solver models. (And why different user and solver models are necessary - the "user model" serves Solver Foundation Services as well as programmers, whereas the solver model is structured so as to attain the most numerically stable, high performance solver possible. Separation of concerns!) Then I gave a high-level overview of our solution approach, and talked about a couple of implementation issues including handling of free variables, equality constraints, etc. Lastly I talked about some numerical stability issues. I got nice feedback from people afterwards and there was a lot of curiosity about who is using our stuff and what is next.

Int'l Symposium on Mathematical Programming, Day 1

2009-08-26T16:35:00Z

I am attending the ISMP math programming conference in Chicago, representing the Solver Foundation team. Here are some of my impressions of Day 1 at ISMP:

Plenary [Steven Boyd]: The plenary speaker was Steve Boyd who talked about real-time embedded convex optimization. He considered cases where problems need to be solved in milliseconds or less: control, signal processing, resource allocation, and even finance (think flash trading). The approach is basically to extend "disciplined convex programming". DCP is a modeling system where problems are convex "by construction" because they combine operators with well-known properties; the system is then able to rewrite the problem in the standard form required by a convex programming solver (such as an IPM QP/SOCP solver). The new step is that the system can now generate highly optimized C code for a custom solver for the particular problem family described by the model. (The algorithm itself is standard IPM.) You can do all sorts of optimizations in this case: the problem structure is known so the symbolic ordering can be done in advance, you can ensure better memory locality, etc. He gave many example where small QP instances were solved in tensof microseconds. Fun stuff.

Morning Sessions: I attended a morning session concerning approaches for nonconvex optimization problems. Jon Lee talked about solving a particular class of parametric nonlinear problems, Kurt Anstreicher talked about *nonconvex* QP and associated bounds. I then went to a straight-up theory session talking about convergence rates & asymptotic costs for IPM - there were a couple of new results and an interesting "corrector-predictor" (instead of the other way around) approach that attains better asymptotic convergence. There is an embarrassment of riches at ISMP: I missed by John Hooker about integrating MIP, constraint & global optimization, a great MIP session including Bob Bixby from Gurobi, and all sorts of other stuff. There are a staggering number of good talks to attend.

Afternoon Sessions [Modeling / Stochastic]: I went to a modeling languages track in the afternoon. The first speaker was from LINDO Systems who talked about their new LINGO offering. The primary new feature is to support stochastic modeling. There were some screenshots from What's Best, their spreadsheet solver. They had a few nice samples as well.

Gautam Mitra from OptiRisk was up next - he talked about SAMPL, stochastic extensions for AMPL. They have extended SAMPL to support:

Chance constraints
Integrated chance constraints
Robust optimization

Chance constraints are interesting - they occur in a range of financial problems including VaR and CVaR calculation. Gautam riffed on the difference between stochastic and robust optimization by invoking Rumsfeld: there are known knowns [deterministic optimization], known unknowns [stochastic], and unknown unknowns [robust optimization]. In robust optimization you may have problem coefficients that are in some range with an unknown distribution and you want the constraints to hold in all (or most) possible circumstances. Robust optimization also has SOCP reformulations, and I am a big believer because I think RO can be cleanly expressed in modeling languages. The last talk in the session was about YALMIP support for robust optimization - YALMIP has had widespread adoption by control theory specialists and has had a long history of improvements. This talk was very code-oriented and fun to watch.

A few overall impressions: I think it is fair to say that SOCP is entering the mainstream with LP/QP/MIP - it came up a lot today. On the modeling side, it was interesting to see that stochastic was featured in all three talks in the session I attended. A larger trend is that MINLP (mixed integer nonlinear programming) is hot. There are tons of tracks on it, and it is [unsurprisingly] being taken on by both the MIP and NLP communities. The problem description is very general so of course there are applications, but the dust has not settled on the solver side.

Quadratic Assignment Problems: solution enumeration

2009-07-27T19:53:00Z

In the last few posts we have given code for computing the Gilmore Lawler bound for quadratic assignment problems, and described two different branching techniques. In this post we show how to enumerate solutions of a small QAP. The possible solutions to a QAP range over permutations over unassigned facilities and locations. If lower bounds do not increase after repeated branching, we may eventually assign nearly all the facilities to locations. If there are only 2 or 3 remaining, we may as well just enumerate all the possible solutions and pick the best. In practice, we hope that we will rarely have to do this, otherwise the branch-and-bound tree will be too large to process efficiently. Here is a procedure that enumerates over all solutions of a size 3 QAP. Iterating over permutations is a standard textbook problem, so maybe you can write cleaner code than I did here!

/// <summary>
/// Assume that all facilities have been assigned to locations,
/// except three. EnumerateSolutions enumerates all six remaining assignments
/// returns the best.
/// </summary>
private double EnumerateSolutions(Qap q, BranchAndBoundNode node, int[] pBest) {
double objBest;
int nPerm = perm3.GetLength(0);
int n = perm3.GetLength(1);
double[] obj = new double[nPerm];
int[] r = node.p.UnusedIndices().ToArray();
int[] c = node.pInv.UnusedIndices().ToArray();
int[] p = (int[])node.p.Clone();
Debug.Assert(r.Length == n, "Only call me for size " + n + " QAPs.");

// Try all the possible permutations.
for (int i = 0; i < nPerm; i++) {
    for (int j = 0; j < n; j++) {
      p[r[j]] = c[perm3[i, j]];
    }
    obj[i] = q.Evaluate(p);
}

// Get the best permutation and objective.
int best = obj.ArgMin(out objBest);
p.CopyTo(pBest, 0);
for (int j = 0; j < n; j++) {
    pBest[r[j]] = perm3[best, j];
}
return objBest;
}

If you combine the code from the following posts you will see that we almost have a working B&B solver:

In my next post I will fill in a couple of the gaps, and then I will re-post all of the code. Then we'll think about what we can actually do with a QAP solver!

Quadratic Assignment Problems: strong and weak branching

2009-07-20T05:19:00Z

Let's pick up where we left off last time and write a score-based Branch delegate. It will be used as the basis for both of our "real" branching functions.

private BranchingDecision BranchCore(BranchAndBoundNode node, double bound, double[][] S) {
RowColumnSums(S, node.Size);
double rowVal, colVal;
int rowBest = _rowSum.ArgMax(out rowVal);
int colBest = _colSum.ArgMax(out colVal);
if (rowVal > colVal) {
return new BranchingDecision(true, rowBest);
}
else {
return new BranchingDecision(false, colBest);
}
}

private void RowColumnSums(double[][] U, int size) {
if (_rowSum == null) {
    _rowSum = new double[_qap.Size];
    _colSum = new double[_qap.Size];
}
_rowSum.ConstantFill(Double.MinValue);
_colSum.ConstantFill(Double.MinValue);

for (int i = 0; i < size; i++) {
    for (int j = 0; j < size; j++) {
      _rowSum[i] += U[i][j];
      _colSum[j] += U[i][j];
    }
}
}

If we can create a reasonable score matrix and pass it into BranchCore, then we've got something. Two common techniques for QAP are:

Weak branching. Let S_ij be the reduced cost U_ij that we got from computing the lower bound. (U_ij is a lower bound on the amount that the bound on the subproblem will increase. So large U_ij means the subproblem will be a lot easier.)
Strong branching. Let S_ij be the bound for subproblem ij. The bigger the bound, the easier the subproblem.

Weak branching is trivial to implement for the Gilmore-Lawler (or GLB) bound. As I mentioned a long time ago, the reduced cost matrix essentially comes for free.

private BranchingDecision WeakBranch(BranchAndBoundNode node, double bound, double[][] U) {
return BranchCore(node, bound, U);
}

For strong branching we need to compute bounds for each subproblem. We overwrite the U matrix and pass it into BranchCore. Since the U matrix is used to fathom subproblems later on in the code, we subtract the parent bound from the bound of each subproblem.

private BranchingDecision StrongBranch(BranchAndBoundNode node, double bound, double[][] U) {
Qap r = new Qap(node.Size - 1);
double[][] Ur = MatrixUtilities.NewMatrix(node.Size - 1, node.Size - 1);
for (int i = 0; i < node.Size; i++) {
    for (int j = 0; j < node.Size; j++) {
      Qap.Reduce(node.Qap, r, i, j);
      U[i][j] = GLB.Bound(r, Ur) - bound;
    }
}
return BranchCore(node, bound, U);
}

Strong branching in general produces better guesses that weak branching. However, it comes at a computational cost. Therefore I want to introduce one last complication…er…feature. Since strong branching is smarter but more costly, we should only use it where it is most needed. I want to have a set of rules that governs when to use strong or weak branching. Branching decisions are most important at the top of the tree, or on nodes where the bound sucks. We can measure the suckiness of the bound for a node by computing the relative gap: the ratio of the gap between the bound and the incumbent solution, and the corresponding gap at the root. So our branching rule data structure will have Depth and Gap properties, as well as properties that store the Branch delegate information.

/// <summary>A description of a branching rule: when and how.
/// </summary>
public class BranchingRule {
/// <summary>The ID for this rule.
/// </summary>
public int Index { get; set; }
/// <summary>Maximum depth for which the rule applies.
/// </summary>
public int Depth { get; set; }
/// <summary>Maximum relative gap for which the rule applies.
/// </summary>
public double Gap { get; set; }
/// <summary>Strong or Weak.
/// </summary>
public BranchType Type { get; set; }
/// <summary>StrongBranch() or WeakBranch().
/// </summary>
internal Branch Rule { get; set; }

public BranchingRule(int index, double gap, int depth, BranchType type) {
    Index = index;
    Depth = depth;
    Gap = gap;
    Type = type;
}

public override string ToString() {
    return "Depth = " + Depth + ", Gap = " + Gap + ", Type = " + Type.ToString();
}
}

We will have an array of BranchingRule called _branchingRule. Given a node, we will select the first rule in the array whose depth is larger than the node depth, and whose gap is larger than the node gap.

private Branch SelectBranchingRule(BranchAndBoundNode node) {
double gap = _results.RelativeGap(node.LowerBound);
BranchingRule criterion = _branchingRules.First(entry => (node.Level <= entry.Depth) && (gap >= entry.Gap));
node.RuleIndex = criterion.Index;
return criterion.Rule;
}

I am cheating here because I have not defined the local variable _results. We'll get to that in a few posts. For now, just assume that RelativeGap is defined as I described above: (Objective - bound) / (Objective - RootBound).

At long last, that covers branching. There are many variations, but they are essentially variations of strong or weak branching. At this point we only have two more main topics to cover: what to do when you get to the bottom of the tree, and how do you capture and update results.

Quadratic assignment problems: branching

2009-07-13T02:24:00Z

(This is part of a long-running series on quadratic assignment problems in C#. Click here to catch up.)

Branching means to divide the search space represented by a node into subnodes. QAP is about making assignments of facilities to locations, so two possible approaches come to mind:

Branch into two subproblems: one where an assignment i -> j is allowed, and another where it is disallowed.
Pick a facility and try assigning it to all possible locations, creating N subproblems. (Or pick a location and assign all facilities to it…)

The second approach, called polytomic branching, turns out to be more effective. The problem with the first ("single assignment branching") is that the subproblems aren't that much easier than the parent problem. You'll have to branch a whole bunch of times before you are able to determine anything.

We still need to decide whether to fix a facility ("row branching") or a location ("column branching"). And after that we need to figure out which particular facility or location is best. At this point you may be wondering, "does it even matter?" The answer is that, yes, branching decisions are earth shatteringly important. Where's what I mean: if you make bad branching choices it could take a thousand times longer to solve the problem, or more! This is part of the reason why QAP (and branch-and-bound) are so interesting. Bad branching decisions lead to subproblems that are just about as hard as the original problem. It's kind of like this. Here is a data structure that represents the results of a polytomic branching decision.

/// <summary>A branching decision, the result of applying a branching rule.
/// </summary>
internal class BranchingDecision {
/// <summary>If true, enumerate over all unassigned rows.
/// </summary>
public bool IsRowBranch { get; set; }
/// <summary>The row or column index.
/// </summary>
public int Index { get; set; }
/// <summary>Create a new instance.
/// </summary>
public BranchingDecision(bool isRowBranch, int index) {
IsRowBranch = isRowBranch;
Index = index;
}
}

Ultimately we want to write methods that produce BranchingDecisions. The following delegate represents such a method (we will write two methods in the next blog post):

/// <summary>Makes a branching decision.
/// </summary>
internal delegate BranchingDecision Branch(BranchAndBoundNode node, double bound, double[][] U);

If you accept the claim that branching choices make a big difference, then it is clear that we want to branch so we get the easiest subproblems possible. That is, nodes that will be processed by the branch-and-bound algorithm as quickly as possible. We don't know how to measure that directly (why?), so we have to guess. A general framework for polytomic branching is to compute a score for each of the N^2 subproblems, and store it in a matrix. S_ij is the score for the subproblem created by fixing facility i to location j. Then compute the row and column sums of S, and branch on the row or the column with maximum sum. Making a branching choice reduces to computing scores on subproblems.

The next step will be to write methods that compute scores and return BranchingDecisions. We'll look at two common ways to do that next time. We're about two or three posts away from having a pretty decent branch-and-bound solver for QAP.

Solver Foundation LINQ to SQL example

2009-06-02T17:45:00Z

Erwin, a modeling consultant and top Solver Foundation user, encountered some problems trying to do two-way data binding using DataTable objects. There are more details on this discussion thread. Ross, a member of the Solver Foundation team, was kind enough to code up a workaround for Erwin's example. In addition to this CS file, you will need to create a new DBML file called SampleDataContext, as I described in a previous post.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Data;
using System.Data.OleDb;
using System.Data.Linq;
using System.Text;
using Microsoft.SolverFoundation.Services;
using System.IO;
 
namespace OML1
{
    class Test
    {
        static void Main(string[] args)
        {
            Test t = new Test();
            t.Solve();
        }
 
        // Holds the OML model
        string strModel = @"Model[
              Parameters[Sets,I],
              Parameters[Reals,p[I]],
 
              Decisions[Reals[0,Infinity],x[I]],
 
              Constraints[
                 Foreach[{i,I}, x[i]==p[i]]
              ]
           ]";
 
 
        //  SFS
        SolverContext context;
        SampleDataContext data;
 
 
        //  Constructor
        public Test()
        {
            context = SolverContext.GetContext();
            data = new SampleDataContext("Data Source=Sql_server_name;Initial Catalog=DataPartitionAllocation20_5;Integrated Security=True");
            context.DataSource = data;
        }
 
 
        // Solve the problem
        public void Solve()
        {
            context.LoadModel(FileFormat.OML, new StringReader(strModel));
 
            Parameter p = context.CurrentModel.Parameters.First(q => q.Name == "p");
            p.SetBinding(data.P, "value", new string[] { "index" });
 
            Decision x = context.CurrentModel.Decisions.First(d => d.Name == "x");
            x.SetBinding(data.X, "value", new string[] { "index" }); 
 
            Solution solution = context.Solve();
            Console.Write("{0}", solution.GetReport());
 
            context.PropagateDecisions();
 
        }
 
    }
 
}