This post is about one more FAKE use case. It will be not usual, but I hope useful script.
The problem I have faced to is recompilation of Stanford NLP products to .NET using IKVM.NET. I am sick of doing it manually. I posted instructions on how to do it, but I think that not many people have tried to do it. I believe that I can automate it end to end from downloading *.jar files to building NuGet packages. Of course, I have chosen FAKE for this task (Thanks to Steffen Forkmann for help with building NuGet packages).
The build scenario is the following:
- Download zip archive with *.jar files and trained models from Stanford NLP site (They can be large, up to 200Mb like for Stanford Parser, and I do not want to store all this stuff in my repository)
- Download IKVM.NET compiler as a zip archive. (It is not distributed with NuGet package and is not referenced from IKVM.NET site. It is really tricky to find it for the first time)
- Unzip all downloaded archives.
- Carefully recompile all required *.jar files considering all references.
- Sign all compiled assemblies to be able to deploy them to the GAC if needed.
- Compile NuGet package.
Steps 1-5 are not covered by FAKE OOTB tasks and I needed to implement them by myself. Since I wanted to use F# 3.0 features and .NET 4.5 capabilities (like System.IO.Compression.FileSystem.ZipFile for unzipping) I have chosen pre-release version of FAKE 2 that uses .NET 4 runtime. Pre-release version of FAKE can be restored from NuGet as follows:
"nuget.exe" "install" "FAKE" "-Pre" "-OutputDirectory" "..\build" "-ExcludeVersion"
Download manager
Requirements: For sure, I do not want to download files from the Internet during each build. Before downloading files, I want to check their presence on the file system, if they are missed then start downloading. During downloading, I want to see the progress status to be sure that everything works. The code that does it:
#r "System.IO.Compression.FileSystem.dll"
let downloadDir = @".\Download\"
let restoreFile url =
let downloadFile file url =
printfn "Downloading file '%s' to '%s'..." url file
let BUFFER_SIZE = 16*1024
use outputFileStream = File.Create(file, BUFFER_SIZE)
let req = System.Net.WebRequest.Create(url)
use response = req.GetResponse()
use responseStream = response.GetResponseStream()
let printStep = 100L*1024L
let buffer = Array.create<byte> BUFFER_SIZE 0uy
let rec download downloadedBytes =
let bytesRead = responseStream.Read(buffer, 0, BUFFER_SIZE)
outputFileStream.Write(buffer, 0, bytesRead)
if (downloadedBytes/printStep <> (downloadedBytes-int64(bytesRead))/printStep)
then printfn "\tDownloaded '%d' bytes" downloadedBytes
if (bytesRead > 0) then download (downloadedBytes + int64(bytesRead))
download 0L
let file = downloadDir @@ System.IO.Path.GetFileName(url)
if (not <| File.Exists(file))
then url |> downloadFile file
file
let unZipTo toDir file =
printfn "Unzipping file '%s' to '%s'" file toDir
Compression.ZipFile.ExtractToDirectory(file, toDir)
let restoreFolderFromUrl folder url =
if not <| Directory.Exists folder
then url |> restoreFile |> unZipTo (folder @@ @"..\")
let restoreFolderFromFile folder zipFile =
if not <| Directory.Exists folder
then zipFile |> unZipTo (folder @@ @"..\")
IKVM.NET Compiler
Compiler should be able to rebuild any number of *.jar files with predefined dependencies and sign result *.dll files if required.
let ikvmc =
restoreFolderFromUrl @".\temp\ikvm-7.3.4830.0" "http://www.frijters.net/ikvmbin-7.3.4830.0.zip"
@".\temp\ikvm-7.3.4830.0\bin\ikvmc.exe"
let ildasm = @"c:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Bin\x64\ildasm.exe"
let ilasm = @"c:\Windows\Microsoft.NET\Framework64\v2.0.50727\ilasm.exe"
type IKVMcTask(jar:string) =
member val JarFile = jar
member val Version = "" with get, set
member val Dependencies = List.empty<IKVMcTask> with get, set
let timeOut = TimeSpan.FromSeconds(120.0)
let IKVMCompile workingDirectory keyFile tasks =
let getNewFileName newExtension (fileName:string) =
Path.GetFileName(fileName).Replace(Path.GetExtension(fileName), newExtension)
let startProcess fileName args =
let result =
ExecProcess
(fun info ->
info.FileName <- fileName
info.WorkingDirectory <- FullName workingDirectory
info.Arguments <- args)
timeOut
if result<> 0 then
failwithf "Process '%s' failed with exit code '%d'" fileName result
let newKeyFile =
let file = workingDirectory @@ (Path.GetFileName(keyFile))
File.Copy(keyFile, file, true)
Path.GetFileName(file)
let rec compile (task:IKVMcTask) =
let getIKVMCommandLineArgs() =
let sb = Text.StringBuilder()
task.Dependencies |> Seq.iter
(fun x ->
compile x
x.JarFile |> getNewFileName ".dll" |> bprintf sb " -r:%s")
if not <| String.IsNullOrEmpty(task.Version)
then task.Version |> bprintf sb " -version:%s"
bprintf sb " %s -out:%s"
(task.JarFile |> getNewFileName ".jar")
(task.JarFile |> getNewFileName ".dll")
sb.ToString()
File.Copy(task.JarFile, workingDirectory @@ (Path.GetFileName(task.JarFile)) ,true)
startProcess ikvmc (getIKVMCommandLineArgs())
if (File.Exists(keyFile)) then
let dllFile = task.JarFile |> getNewFileName ".dll"
let ilFile = task.JarFile |> getNewFileName ".il"
startProcess ildasm (sprintf " /all /out=%s %s" ilFile dllFile)
File.Delete(dllFile)
startProcess ilasm (sprintf " /dll /key=%s %s" (newKeyFile) ilFile)
tasks |> Seq.iter compile
Results
Using this helper function, build scripts come out pretty straightforward and easy. For example, recompilation of Stanford Parser looks as follows:
Target "RunIKVMCompiler" (fun _ ->
restoreFolderFromUrl
@".\temp\stanford-parser-full-2013-06-20"
"http://nlp.stanford.edu/software/stanford-parser-full-2013-06-20.zip"
restoreFolderFromFile
@".\temp\stanford-parser-full-2013-06-20\edu"
@".\temp\stanford-parser-full-2013-06-20\stanford-parser-3.2.0-models.jar"
[IKVMcTask(@"temp\stanford-parser-full-2013-06-20\stanford-parser.jar",
Version=version,
Dependencies =
[IKVMcTask(@"temp\stanford-parser-full-2013-06-20\ejml-0.19-nogui.jar",
Version="0.19.0.0")])]
|> IKVMCompile ikvmDir @".\Stanford.NLP.snk"
)
All source code is available on GitHub.
