let runFAKE = Download >> Unzip >> IKVMCompile >> Sign >> NuGet

This post is about one more FAKE use case. It will be not usual, but I hope useful script.

The problem I have faced to is recompilation of Stanford NLP products to .NET using IKVM.NET. I am sick of doing it manually. I posted instructions on how to do it, but I think that not many people have tried to do it. I believe that I can automate it end to end from downloading *.jar files to building NuGet packages. Of course, I have chosen FAKE for this task (Thanks to Steffen Forkmann for help with building NuGet packages).

The build scenario is the following:

1. Download zip archive with *.jar files and trained models from Stanford NLP site (They can be large, up to 200Mb like for Stanford Parser, and I do not want to store all this stuff in my repository)
2. Download IKVM.NET compiler as a zip archive. (It is not distributed with NuGet package and is not referenced from IKVM.NET site. It is really tricky to find it for the first time)
3. Unzip all downloaded archives.
4. Carefully recompile all required *.jar files considering all references.
5. Sign all compiled assemblies to be able to deploy them to the GAC if needed.
6. Compile NuGet package.

Steps 1-5 are not covered by FAKE OOTB tasks and I needed to implement them by myself. Since I wanted to use F# 3.0 features and .NET 4.5 capabilities (like System.IO.Compression.FileSystem.ZipFile for unzipping) I have chosen pre-release version of FAKE 2 that uses .NET 4 runtime. Pre-release version of FAKE can be restored from NuGet as follows:

"nuget.exe" "install" "FAKE" "-Pre" "-OutputDirectory" "..\build" "-ExcludeVersion"

Download manager

Requirements: For sure, I do not want to download files from the Internet during each build. Before downloading files, I want to check their presence on the file system, if they are missed then start downloading. During downloading, I want to see the progress status to be sure that everything works. The code that does it:

#r "System.IO.Compression.FileSystem.dll"
let downloadDir = @".\Download\"

let restoreFile url =
    let downloadFile file url =
        printfn "Downloading file '%s' to '%s'..." url file
        let BUFFER_SIZE = 16*1024
        use outputFileStream = File.Create(file, BUFFER_SIZE)
        let req = System.Net.WebRequest.Create(url)
        use response = req.GetResponse()
        use responseStream = response.GetResponseStream()
        let printStep = 100L*1024L
        let buffer = Array.create<byte> BUFFER_SIZE 0uy
        let rec download downloadedBytes =
            let bytesRead = responseStream.Read(buffer, 0, BUFFER_SIZE)
            outputFileStream.Write(buffer, 0, bytesRead)
            if (downloadedBytes/printStep <> (downloadedBytes-int64(bytesRead))/printStep)
                then printfn "\tDownloaded '%d' bytes" downloadedBytes
            if (bytesRead > 0) then download (downloadedBytes + int64(bytesRead))
        download 0L
    let file = downloadDir @@ System.IO.Path.GetFileName(url)
    if (not <| File.Exists(file))
        then url |> downloadFile file
    file
let unZipTo toDir file =
    printfn "Unzipping file '%s' to '%s'" file toDir
    Compression.ZipFile.ExtractToDirectory(file, toDir)
let restoreFolderFromUrl folder url =
    if not <| Directory.Exists folder
        then url |> restoreFile |> unZipTo (folder @@ @"..\")

let restoreFolderFromFile folder zipFile =
    if not <| Directory.Exists folder
        then zipFile |> unZipTo (folder @@ @"..\")

IKVM.NET Compiler

Compiler should be able to rebuild any number of *.jar files with predefined dependencies and sign result *.dll files if required.

let ikvmc =
    restoreFolderFromUrl @".\temp\ikvm-7.3.4830.0" "http://www.frijters.net/ikvmbin-7.3.4830.0.zip"
    @".\temp\ikvm-7.3.4830.0\bin\ikvmc.exe"
let ildasm = @"c:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\x64\ildasm.exe"
let ilasm = @"c:\Windows\Microsoft.NET\Framework64\v2.0.50727\ilasm.exe"
type IKVMcTask(jar:string) =
    member val JarFile = jar
    member val Version = "" with get, set
    member val Dependencies = List.empty<IKVMcTask> with get, set
let timeOut = TimeSpan.FromSeconds(120.0)
let IKVMCompile workingDirectory keyFile tasks =
    let getNewFileName newExtension (fileName:string) =
        Path.GetFileName(fileName).Replace(Path.GetExtension(fileName), newExtension)
    let startProcess fileName args =
        let result =
            ExecProcess
                (fun info ->
                    info.FileName <- fileName
                    info.WorkingDirectory <- FullName workingDirectory
                    info.Arguments <- args)
                timeOut
        if result<> 0 then
            failwithf "Process '%s' failed with exit code '%d'" fileName result
    let newKeyFile =
        let file = workingDirectory @@ (Path.GetFileName(keyFile))
        File.Copy(keyFile, file, true)
        Path.GetFileName(file)
    let rec compile (task:IKVMcTask) =
        let getIKVMCommandLineArgs() =
            let sb = Text.StringBuilder()
            task.Dependencies |> Seq.iter
               (fun x ->
                   compile x
                   x.JarFile |> getNewFileName ".dll" |> bprintf sb " -r:%s")
            if not <| String.IsNullOrEmpty(task.Version)
                then task.Version |> bprintf sb " -version:%s"
            bprintf sb " %s -out:%s"
                (task.JarFile |> getNewFileName ".jar")
                (task.JarFile |> getNewFileName ".dll")
            sb.ToString()
        File.Copy(task.JarFile, workingDirectory @@ (Path.GetFileName(task.JarFile)) ,true)
        startProcess ikvmc (getIKVMCommandLineArgs())

        if (File.Exists(keyFile)) then
            let dllFile = task.JarFile |> getNewFileName ".dll"
            let ilFile = task.JarFile |> getNewFileName ".il"
            startProcess ildasm (sprintf " /all /out=%s %s" ilFile dllFile)
            File.Delete(dllFile)
            startProcess ilasm (sprintf " /dll /key=%s %s" (newKeyFile) ilFile)
    tasks |> Seq.iter compile

Results

Using this helper function, build scripts come out pretty straightforward and easy. For example, recompilation of Stanford Parser looks as follows:

Target "RunIKVMCompiler" (fun _ ->
    restoreFolderFromUrl
        @".\temp\stanford-parser-full-2013-06-20"
        "http://nlp.stanford.edu/software/stanford-parser-full-2013-06-20.zip"
    restoreFolderFromFile
        @".\temp\stanford-parser-full-2013-06-20\edu"
        @".\temp\stanford-parser-full-2013-06-20\stanford-parser-3.2.0-models.jar"

    [IKVMcTask(@"temp\stanford-parser-full-2013-06-20\stanford-parser.jar",
        Version=version,
        Dependencies =
            [IKVMcTask(@"temp\stanford-parser-full-2013-06-20\ejml-0.19-nogui.jar",
                       Version="0.19.0.0")])]
    |> IKVMCompile ikvmDir @".\Stanford.NLP.snk"
)

All source code is available on GitHub.