FSharp.ML – industry needs. (Machine Learning for .NET)

Machine Learning is a hot topic for nowadays. ML is a core part of Data Analysis and an auxiliary tool in a lot of domains (NLP, search engines, e-commerce solutions and etc). Many ML related courses available on the Coursera  in “Statistics, Data Analysis, and Scientific Computing” and “Computer Science: Artificial Intelligence, Robotics, Vision” sections. Kaggle holds ML competitions more and more often.

Java has some popular and recognized ML libraries such as Mahout and Weka, but it is much harder to find .NET high performance ML library (which does not run on the IKVM.NET).

What is already available in .NET World?

As Don Syme said, it would be cool to have an independent comparison of already available ML libraries. We need to understand what is suitable for what needs.

Also I want to mention some most promising of them:

What can we do?

We are talking that F# is great for data scientists and statisticians and so it is! We still do not have mature F# ML library, but we have a lot of posts about ML and a lot of interest in this domain:

It is time to put it all together into FShapr.ML.  This can be done in two parts: a complete functional ML framework plus a collection of useful customizable samples.

F#/.NET function minimization (optimization)

I have done some research on function minimization algorithms implemented on .NET. Short summary can be found below.

Gradient descent

Gradient descent is one of the simplest function optimization algorithms. You can implement it by yourself or using one of the following articles:

DotNumerics

DotNumerics is a Numerical Library for .NET. The library is written in pure C# and has more than 100,000 lines of code with the most advanced algorithms for Linear Algebra, Differential Equations and Optimization problems.

Unfortunately, dotNumerics does not have a detailed documentation. Let’s go through all minimization algorithms implemented in dotNumerics. First of all, we implement banana function from simplex method example available on the library site.

#r @"DotNumerics.dll"
open System
open DotNumerics.Optimization

//f(a,b) = 100*(b-a^2)^2 + (1-a)^2
let BananaFunction (x: float array) =
    100.0 * Math.Pow((x.[1] - x.[0] * x.[0]), 2.0) + Math.Pow((1.0 - x.[0]), 2.0)

Downhill Simplex

Downhill Simplex method of Nelder and Mead

The key advantage of Downhill Simplex method is that it does not require the gradient function. All you need is a function and an initial guess.

let initialGuess = [|0.1; 2.0|]

let simplexMin =
    let simplex = Simplex();
    simplex.ComputeMin(BananaFunction,initialGuess);

We have a bit of control over the evaluation model. We can restrict MaxFunEvaluations and specify custom Tolerance in Simplex model. In this case, model instantiation looks like below.

    let simplex = Simplex(MaxFunEvaluations=10000, Tolerance=1e-5);

Truncated Newton

“A Survey of Truncated-Newton Methods”, Journal of Computational and Applied Mathematics.

All other algorithms require gradient function to make calculation.


//f'a(a,b) = (100*(b-a^2)^2 + (1-a)^2)'a = 100*2*(b-a^2)*(-2a) - 2*(1-a)
//f'b(a,b) = (100*(b-a^2)^2 + (1-a)^2)'b = 100*2*(b-a^2)
let BananaFunctionGradient (x: float array) =
    [|100.0 * 2.0 * (x.[1] - x.[0] * x.[0]) * (-2.0 * x.[0]) - 2.0 * (1.0 - x.[0]);
      100.0 * 2.0 * (x.[1] - x.[0] * x.[0])|]

let newtonMin =
    let newton = TruncatedNewton()
    newton.ComputeMin(BananaFunction,BananaFunctionGradient,initialGuess);

Truncated Newton algorithm has three more configuration parameters than Downhill Simplex: Accuracy, MaximunStep and SearchSeverity.

L-BFGS-B

Limited memory Broyden–Fletcher–Goldfarb–Shanno method

let bfgsMin =
    let lbfgsb = L_BFGS_B()
    lbfgsb.ComputeMin(BananaFunction, BananaFunctionGradient, initialGuess);

L-BFGS-B has one more configuration parameters than Downhill Simplex – it is AccuracyFactor.

Results

Below you can find evaluation results received from models with default parameters.

Real: 00:00:00.024, CPU: 00:00:00.062, GC gen0: 0, gen1: 0, gen2: 0
val simplexMin : float [] = [|0.999999998; 0.9999999956|]
Real: 00:00:00.074, CPU: 00:00:00.078, GC gen0: 0, gen1: 0, gen2: 0
val newtonMin : float [] = [|0.9999999999; 0.9999999999|]
Real: 00:00:00.137, CPU: 00:00:00.140, GC gen0: 0, gen1: 0, gen2: 0
val bfgsMin : float [] = [|1.0; 1.0|]