F# Weekly #24 2013

Welcome to F# Weekly,

A roundup of F# content from this past week:

News

Videos/Presentations

Blogs

That’s all for now.  Have a great week.

Previous F# Weekly edition – #23

New Twitter API or “F# Weekly” v1.1

Good news for Twitter and no so good for developers:twitter_app

Today(2013-06-11), we(Twitter) are retiring API v1 and fully transitioning to API v1.1.

What does it all mean? This means that all old services are no longer available. Twitter switched to new ones with mandatory OAuth authentication. From now, to work with twitter services we must register new apps and use OAuth.

Also, it means that:

As I know, there are two alternatives available instead of Twitterizer:

  • Tweetsharp (TweetSharp is a fast, clean wrapper around the Twitter API.)
  • LINQ to Twitter (An open source 3rd party LINQ Provider for the Twitter micro-blogging service.)

I have chosen Tweetsharp because its API similar to Twitterizer. This is a new F# Weekly under the hood script:

#r "Newtonsoft.Json.dll"
#r "Hammock.ClientProfile.dll"
#r "TweetSharp.dll"

open TweetSharp
open System
open System.Net
open System.Text.RegularExpressions

let service = new TwitterService(_consumerKey, _consumerSecret)
service.AuthenticateWith(_accessToken, _accessTokenSecret)

let getTweets query =
    let rec collect maxId =
        let options = SearchOptions(Q = query, Count =Nullable(100), MaxId = Nullable(maxId),
                                    Resulttype = Nullable(TwitterSearchResultType.Recent))
        printfn "Loading %s under id %d" query maxId
        let results = service.Search(options).Statuses |> Seq.toList
        printfn "\t Loaded %d tweets" results.Length
        if (results.Length = 0)
            then List.empty
            else
                let lastTweet = results |> List.rev |> List.head
                if (lastTweet.Id < maxId)                     then results |> List.append (collect (lastTweet.Id))
                    else results
    collect (Int64.MaxValue) |> List.rev

let urlRegexp = Regex("http://([\\w+?\\.\\w+])+([a-zA-Z0-9\\~\\!\\@\\#\\$\\%\\^\\&\\*\\(\\)_\\-\\=\\+\\\\\\/\\?\\.\\:\\;\\'\\,]*)?", RegexOptions.IgnoreCase);

let filterUniqLinks (tweets: TwitterStatus list) =
    let hash = new System.Collections.Generic.HashSet();
    tweets |> List.fold
        (fun acc t ->
             let mathces = urlRegexp.Matches(t.Text)
             if (mathces.Count = 0) then acc
             else let urls =
                     [0 .. (mathces.Count-1)]
                     |> List.map (fun i -> mathces.[i].Value)
                     |> List.filter (fun url -> not(hash.Contains(url)))
                  if (List.isEmpty urls) then acc
                  else urls |> List.iter(fun url -> hash.Add(url) |> ignore)
                       t :: acc)
        [] |> List.rev

let tweets =
    ["#fsharp";"#fsharpx";"@dsyme";"#websharper";"@c4fsharp"]
    |> List.map getTweets
    |> List.concat
    |> List.sortBy (fun t -> t.CreatedDate)
    |> filterUniqLinks

let printTweetsInHtml filename (tweets: TwitterStatus list) =
    let formatTweet (text:string) =
        let matches = urlRegexp.Matches(text)
        seq {0 .. (matches.Count-1)}
            |> Seq.fold (
                fun (t:string) i ->
                    let url = matches.[i].Value
                    t.Replace(url, (sprintf "<a href="\&quot;%s\&quot;" target="\&quot;_blank\&quot;">%s</a>" url url)))
                text
    let rows =
      tweets
        |> List.mapi (fun i t ->
            let id = (tweets.Length - i)
            let text = formatTweet(t.Text)
            sprintf "</pre>
<table id="\&quot;%d\&quot;">
<tbody>
<tr>
<td rowspan="\&quot;2\&quot;" width="\&quot;30\&quot;">%d</td>
<td rowspan="\&quot;2\&quot;" width="\&quot;80\&quot;"><a href="\&quot;javascript:remove('%d')\&quot;">Remove</a></td>
<td rowspan="\&quot;2\&quot;"><a href="\&quot;https://twitter.com/%s\&quot;" target="\&quot;_blank\&quot;"><img alt="" src="\&quot;%s\&quot;/" /></a></td>
<td><b>%s</b></td>
</tr>
<tr>
<td>Created : %s</td>
</tr>
</tbody>
</table>
<pre>
"
id id id t.Author.ScreenName t.Author.ProfileImageUrl text (t.CreatedDate.ToString()))
        |> List.fold (fun s r -> s+" "+r) ""
    let html = sprintf "<script type="text/javascript">// <![CDATA[
function remove(id){return (elem=document.getElementById(id)).parentNode.removeChild(elem);}
// ]]></script>%s" rows
 System.IO.File.WriteAllText(filename, html)

printTweetsInHtml "d:\\tweets.html" tweets

F# Weekly #23 2013

Welcome to F# Weekly,

A roundup of F# content from this past week:

News

Videos/Presentations

Blogs

That’s all for now.  Have a great week.

Previous F# Weekly edition – #22

F# Weekly #22 2013

Welcome to F# Weekly,

So it’s summer, but F# Weekly is still with you. A roundup of F# content from this past week:

News

Videos/Presentations

Blogs

That’s all for now.  Have a great week.

Previous F# Weekly edition – #21

15 Principles for Data Scientists

marksalen's avatarOpen Source Research

I have developed 15 principles for my daily work as a data scientist. These are the principles  that I personally follow :

1- Do not lie with data and do not bullshit: Be honest and frank about empirical evidences. And most importantly do not lie to yourself with data

2- Build everlasting tools and share them with others: Spend a portion of your daily work building tools that makes someone’s life easier. We are freaking humans, we are supposed to be tool builders!

3- Educate yourself continuously: you are a scientist for Bhudda’s sake. Read hardcore math and stats from graduate level textbooks. Never settle down for shitty explanations of a method that you receive from a coworker in the hallway. Learn fundamentals and you can do magic. Read recent papers, go to conferences, publish, and review papers. There is no shortcut for this.

4- Sharpen your skills: learn one language well…

View original post 413 more words

F# Weekly #21 2013

Welcome to F# Weekly,

A roundup of F# content from this past week:

News

Videos/Presentations

Blogs

That’s all for now.  Have a great week.

Previous F# Weekly edition – #20

F# Weekly #20 2013

the lhs of an F# developer’s desk

the rhs of an F# developer’s desk

Welcome to F# Weekly,

This past week was full of interesting events and happenings. If you follow F#, you must read all of this carefully. The roundup of F# content:

News

Videos/Presentations

Blogs

That’s all for now.  Have a great week.

Previous F# Weekly edition – #19
lamdbaLadies

Three easy ways to create simple Web Server with F#

I have tried to find easiest ways to create a simple web server with F#. There are three most simple ways to do it.

The goal is to create a simple web service that maps web request urls to the files in the site folder. If file with such name exists then return its content as html. Assume that all html files located in ‘D:\mySite\‘.

HttpListener

First and probably the most promising option was created by Julian Kay and described in his post “Creating a simple HTTP Server with F#“. I slightly modified source code to satisfy my initial goal. You can find detailed description of how it works in Julian’s post. (Works from FSI)

open System
open System.Net
open System.Text
open System.IO

let siteRoot = @"D:\mySite\"
let host = "http://localhost:8080/"

let listener (handler:(HttpListenerRequest->HttpListenerResponse->Async<unit>)) =
    let hl = new HttpListener()
    hl.Prefixes.Add host
    hl.Start()
    let task = Async.FromBeginEnd(hl.BeginGetContext, hl.EndGetContext)
    async {
        while true do
            let! context = task
            Async.Start(handler context.Request context.Response)
    } |> Async.Start

let output (req:HttpListenerRequest) =
    let file = Path.Combine(siteRoot,
                            Uri(host).MakeRelativeUri(req.Url).OriginalString)
    printfn "Requested : '%s'" file
    if (File.Exists file)
        then File.ReadAllText(file)
        else "File does not exist!"

listener (fun req resp ->
    async {
        let txt = Encoding.ASCII.GetBytes(output req)
        resp.ContentType <- "text/html"
        resp.OutputStream.Write(txt, 0, txt.Length)
        resp.OutputStream.Close()
    })
// TODO: add your code here

Self-hosted WCF service

The second option is a tuned self-hosted WCF service. This approach was proposed by  Brian McNamara as an answer to the StackOverflow question “F# web server library“. (Works from FSI)

#r "System.ServiceModel.dll"
#r "System.ServiceModel.Web.dll"

open System
open System.IO

open System.ServiceModel
open System.ServiceModel.Web

let siteRoot = @"D:\mySite\"

[<ServiceContract>]
type MyContract() =
    [<OperationContract>]
    [<WebGet(UriTemplate="{file}")>]
    member this.Get(file:string) : Stream =
        printfn "Requested : '%s'" file
        WebOperationContext.Current.OutgoingResponse.ContentType <- "text/html"
        let bytes = File.ReadAllBytes(Path.Combine(siteRoot, file))
        upcast new MemoryStream(bytes)

let startAt address =
    let host = new WebServiceHost(typeof<MyContract>, new Uri(address))
    host.AddServiceEndpoint(typeof<MyContract>, new WebHttpBinding(), "")
      |> ignore
    host.Open()
    host

let server = startAt "http://localhost:8080/"
// TODO: add your code here
server.Close()

NancyFx

The third one is based on NancyFx. It is lightweight, low-ceremony, framework for building HTTP based services on .Net and Mono. Nancy is a popular framework in C# world, but does not have a natural support of F#. The F# code looks not so easy and simple as it could be. If you want to make it work, you need to create console application and install the Nancy and Nancy.Hosting.Self NuGet packages.

module WebServers

open System
open System.IO
open Nancy
open Nancy.Hosting.Self
open Nancy.Conventions

let (?) (this : obj) (prop : string) : obj =
    (this :?> DynamicDictionary).[prop]

let siteRoot = @"d:\mySite\"

type WebServerModule() as this =
    inherit NancyModule()
    do this.Get.["{file}"] <-
         fun parameters ->
              new Nancy.Responses.HtmlResponse(
                  HttpStatusCode.OK,
                  (fun (s:Stream) ->
                      let file = (parameters?file).ToString()
                      printfn "Requested : '%s'" file
                      let bytes = File.ReadAllBytes(Path.Combine(siteRoot, file))
                      s.Write(bytes,0,bytes.Length)
              )) |> box

let startAt host =
    let nancyHost = new NancyHost(new Uri(host))
    nancyHost.Start()
    nancyHost

let server = startAt "http://localhost:8080/"
printfn "Press [Enter] to exit."
Console.ReadKey() |> ignore
server.Stop()

Further reading