FsShelter: a Storm shell for F#

I think, therefore I spam.

About a year ago Prolucid adopted Apache Storm as our platform of choice for event stream processing and F# as our language of choice for all of our “cloud” development.

FsStorm was an essential part that let us iterate, scale and deliver quickly, but even from the earliest days it was obvious that the developer experience could be improved. Unfortunately, it meant a complete rewrite of FsStorm:

  • FsStorm DSL is a really thin layer on top of Nimbus API model:
    • has explicit IDs when describing components in a topology
    • uses strings in all the names
    • matching of inputs/outputs is not guaranteed
  • FsStorm uses Json AST as it’s public API:
    • messages, tuples, configuration
    • serialization mechanism is hard-baked into the API

We’ve worked around some of the problems, usually by writing more code.

It actually makes sense that Storm itself doesn’t care about the type of the tuples/fields. It runs on JVM, which is very much typed…

View original post 220 more words

F#ools’ Day Party – April 1, 2016

AprilFools6301st April is coming, Mark Gray and I propose F# community to celebrate this day with funny posts about F#.

All you need is to write a blog post that incorporates the idea of April fool in the content or find out a new extraordinary use of F# and destroy the myth that “F# is for Math and Science”.

Rules are even simpler than in FsAdvent:

  1. Ping @sergey-tihon on Twitter and I will put your name in the table below
  2. Tweet link to your post with  tag on April 1 (+/- 1 day).

Happy F#ools’ Day,
Let’s joke together.

Author Post Title
 Pierre Irrmann  Making program behaviour explicit
 Ross McKinlay
 Phillip Trelford  Silverlight Resurrection
 Vicenç García-Altés  A Premier League team will win the Champions League this year

Application contracts with Swagger powered APIs for .NET or Why SwaggerProvider

This post is part of the F# Advent Calendar in English 2015 project.
And special thank you to Phillip Trelford for the idea of F# Advent in English year ago.

In this post I want to talk about how we design software, what we should care about and what we should not.

The story of The Contract

The initial thing is a problem. Nobody writes code without understanding what he/she is doing and what he/she expects to receive at the end. The task (problem description) may come from anywhere: it may be task from your boss or an awesome idea that pretty soon can change the world and bring you a lot of money or some tool / OSS project that will make your life easier in the future.

When you have a clear problem definition, you can start thinking about the solution. There are plenty of ways to solve any problem. You can choose any programming language, any operating system, any programming paradigm, any framework, any tool-set and so on. In order to choose one (the best in your case) solution you start applying real world constraints to reduce the number of solutions. You will probably decide to use a language and tools, which you are more familiar with or which suits better for this particular task. People do this because we all have very important real-life constraint – it is the time. Time-to-market is always important. Nobody starts writing a new operating system just because he/she can do it. We always try to minimize amount of work that should be done to speed up time-to-market.

When you actually start hacking, the thing you start with is `architecture`. Argh, I said that arch.. word. But architecture is nothing more than decomposition. If you do it right in terms of your constraints and limitations – you are almost done. After that, you should write code for small pieces and compose them to get the solution. But what defines the right decomposition? The answer is contracts!

Sounds really simple, does not it? However, it is not so far from the truth. Real-world solution architecture is always state of art, it cannot be super cool for all things, but it can fit your requirements to certain extent. All you can do mastering the architecture is to decompose your monolith idea to simple manageable components with clear responsibilities and clear contract.

The implementation of contract in programming language level is an interface (the promise of capabilities to the world). The implementation of interfaces in F#/C# is pretty good. You as a developer should only define an interface and implement it somewhere to provide a capability by contract. When you are on the other side and want to consume provided capabilities you should only reference component that defines contract and use it with almost zero overhead.

Based on the experience with interfaces, you have three ways of interaction with contract:

  1. Design (define) contract
  2. Provide (implement) contract
  3. Consume contract

But it is not so simple, when your application is larger than one executable file… What happens when you cross process and/or machine boundaries? In this case, we need a technology that allows us to setup the component that provides contract to one machine and consumes this contract on another machine with the same elegance as interfaces do this. At this moment, higher-level contracts (so called API) come to play.

There are plenty of technologies that can help you to develop some sort of API for your component. Probably first helpful thing that comes to mind is WSDL (Web Service Definition Language). When you choose WSDL as your contract language you receive plenty of tools that help you to manage your API, generate implementation from API contract, generate contract from your implementation, generate code that encapsulate all communication difficulties for consumers and so on. WSDL world is pretty mature and beautiful (I really love it), but it usually comes with SOAP and communication protocol that does not suitable for all types of cross-machine communication.

There are some real-world cases when you need more lightweight and fast communication mechanism that can be “easily” consumed from different languages and run-times. In such cases, people more often decide to provide RESTful APIs. BUT, all of you who design RESTful API for your systems and components SHOULD REMEMBER that technology that you use to design and implement contract SHOULD also provide a way to CONSUME your API. It is your responsibility to care about consumers, they should be focused on solving their own task using your API rather than writing boilerplate code to consume your API.

Swagger

One of technologies that provide such goodness for developers is Swaggernlp-logo-navbar

Swagger is a simple yet powerful representation of your RESTful API. With the largest ecosystem of API tooling on the planet, thousands of developers are supporting Swagger in almost every modern programming language and deployment environment. With a Swagger-enabled API, you get interactive documentation, client SDK generation and discoverability.

Swagger Specification is able to describe pretty wide range of RESTful APIs. Swagger comes along with Swagger Edit that allows contract first API development. APIs that have Swagger schema can easily be integrated with Swagger UI – very powerful interface that dramatically simplify studying of new API (demo). Swagger ecosystem also has tools that generate client code for wide list of languages.

In the .NET world, exists a project called Swashbuckle, which generates Swagger schema to ASP.NET WebAPI services and plugs Swagger UI in your application. I’ve already blogged about how to use Swashbuckle from F# Web Apps.

Swagger is used in ASP.NET API app in Azure App Service and may come to Nancy very soon as well.

F# Swagger Provider

swaggerAs you may already understand, Swagger ecosystem looks very promising in general and Swagger Specification perfectly matches with F# Type Providers approach. So, I am glad to present the SwaggerProvider.

In order to start you need to add reference to SwaggerProvider NuGet package and pass Swagger JSON schema to it (for this example I will use a schema from Petstore – Swagger demo app).

#r "SwaggerProvider/SwaggerProvider.dll"
open SwaggerProvider

type PetStore = SwaggerProvider<"http://petstore.swagger.io/v2/swagger.json">

So, you are generally done. Type Provider parses provided schema and generates all required data types and methods to call API. Note that SwaggerProvider is generative type provider and generates types and methods that exist in run-time, so that means that it can be used from C# code as well.

swagger_defs

swagger_pets

Even if the RESTful API that you need to consume does not gently provide Swagger schema for you, you still can use SwaggerProvider to call it. It should be easier to manually describe schema once than to write and support source code for all REST calls.

Let’s say that we want to call GitHub API to get all fsprojects repositories. For this API call schema will look like this

{
    "swagger": "2.0",
    "info": {
        "description": "This is a Sample for GitHub API.",
        "version": "1.0.0",
        "title": "Swagger GitHub"
    },
    "host": "api.github.com",
    "basePath": "/",
    "tags": [
        {
            "name": "repos",
            "description": "Repositories API"
        }
    ],
    "schemes": [ "https" ],
    "paths": {
        "orgs/{orgId}/repos": {
            "get": {
                "tags": [ "repos" ],
                "summary": "List organization repositories",
                "description": "",
                "operationId": "orgRepos",
                "consumes": [ "application/json" ],
                "produces": [ "application/json" ],
                "parameters": [
                    {
                        "name": "orgId",
                        "in": "path",
                        "description": "Organization name",
                        "required": true,
                        "type": "string"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "successful operation",
                        "schema": {
                            "type": "array",
                            "items": {
                                "$ref": "#/definitions/Repo"
                            }
                        }
                    }
                }
            }
        }
    },
    "definitions": {
        "Repo": {
            "type": "object",
            "properties": {
                "id": {
                    "type": "integer",
                    "format": "int64"
                },
                "name": {
                    "type": "string"
                }
            }
        }
    }
}

If you use Visual Studio to edit Swagger schema, you will be pleasantly surprised by “Intellisense for JSON Schema in the JSON Editor

gh-vs

This JSON describes one data type Repo, that contains simplified version of GitHub Repository data type, and description of one API method /orgs/{orgId}/repos that does the job. Now you are able to use this schema to call GitHub API.

gh_def

gh_repos

But if you really want to make a call to GitHub API, you need to specify real user agent header and SwaggerProvider allows you to do this:

open SwaggerProvider

let [<Literal>] schemaFile = __SOURCE_DIRECTORY__ + "\GitHub.json"
let [<Literal>] headers = "User-Agent=Googlebot/2.1 (+http://www.google.com/bot.html)"
type GitHub = SwaggerProvider< schemaFile, headers >;

let names =
    GitHub.Repos.OrgRepos("fsprojects")
    |> Array.map (fun x -> x.Name)

gh_works

Note: SwaggerProvider does not support schemes in YAML format yet. But this feature is definitely will be implemented, because it dramatically simplifies manual schema creation.

Please share your thoughts about SwaggerProvider and do not be shy to try it and report issues if any.

Special thanks to APIs.guru project for maintenance of wide range of real-world Swagger schemes that were used to test SwaggerProvider.

Simple Web API OWIN host using F#

Very nice starter pack for OWIN Web API with F#. If you use Paket manager you need only one dependency to `Microsoft.AspNet.WebApi.OwinSelfHost` NuGet package

Also Web API can be easily configured to use Swagger UI.

Exception Caught

I thought it would be good to write a simple OWIN Web Api host using F# to show how easy it is to interop between the two languages and as a fun experiment. You can find all of the source code from this blog post at the FSharpWebApi github repository.

If you open the Program.fs solution in the github repository you will see the following two lines of code are reponsible for starting the web server:

The getAppBuilder function is defined in the WebServerBuilder module as follows:

The getAppBuilder function returns a function with the signature (IAppBuilder) -> (). This signature is the same as the one expected by the first parameter of WebApp.Start. The reason for breaking this function off into its own module is so that it can be tested.

The cool thing about the Web Api Owin Self host stack is that there is a nuget…

View original post 491 more words

F# Advent Calendar in English 2015

FsAdvent

Update(10/27/2015): There are a lot of people, who want to participate, so we’ve decided to extend the timeline and double number of slots. Please do not be shy and books free slots.

Last year we ran an amazing event “F# Advent Calendar in English 2014“. It was incredible December full of F# and Christmas spirit. Every day astonishing authors around the globe posted new F#-related articles. It was an extraordinary time.

December is close enough, so it is a good time to plan something special for F# Advent Calendar. Have you done something special this year? Do you have any unique experience you are willing to share? Have your project incredibly evolved this year? Are there any good ideas for the post, but you didn’t have time to write it? The time has come – it is right now! You have a chance to share your story with the globe! Join F# Advent Calendar and hurry up!

Rules

Rules are very simple:

  1. Choose F# related topic for your blog post and reserve the date on Twitter or leave a comment to this post. Please note that you do not have to announce the topic until the date.
  2. Prepare a blog post in English
  3. Publish your post on specified date (according to the calendar)
  4. Post link to your post on Twitter with hashtags #fsharp and #FsAdvent.

Calendar

Date Author Post Title
 Nov 29 (Sunday)  Rachel Reese  How Jet Build Microservices with F#
 Nov 29 (Sunday)  Jamie Dixon  Creating Dynamic Uris For Visual Studio Web Tests
 Nov 30 (Monday)  Steffen Forkmann  F# advent calendar: Using Async.Choice in Paket
 Nov 30 (Monday)  Bohdan Szymanik  Sharpen up your legacy app(s) performance with a bit of F#
 Dec 01 (Tuesday)  Mark Seemann  Recurse
 Dec 01 (Tuesday)  Kristian Schmidt  A roll of the Liar’s dice
 Dec 02 (Wednesday)  Mike Janger  Taking Ionide Out for a Spin
 Dec 02 (Wednesday)  Tomasz Jaskuλa  Data Science tools in F# through univariante linear regression
 Dec 03 (Thursday)  Phillip Trelford  Calendar Types
 Dec 03 (Thursday)  Jeremy Abbott  F# Events, Reactive Programming and Async Workflows
 Dec 04 (Friday)  Richard Dalton  Azure WTF#
 Dec 04 (Friday)  Edgar Sánchez  Calculating a cannon ball trajectory, the fun way
 Dec 05 (Saturday)  Scott Wlaschin  Thirteen ways of looking at a turtle
Thirteen ways of looking at a turtle (part 2)
Thirteen ways of looking at a turtle – addendum
 Dec 05 (Saturday)  Sean Trelford  No 1 at Christmas
 Dec 06 (Sunday)  Andrea Magnorsky  Computation expressions and microphones
More Computation expressions
 Dec 06 (Sunday)  Christopher Atkins  F# 2015 Advent Cookies
 Dec 07 (Monday)  Sergey Tihon  Application contracts with Swagger powered APIs for .NET or Why SwaggerProvider
 Dec 07 (Monday)  Aaron Powell  What’s the time Mr Wolf?
 Dec 08 (Tuesday)  Isaac Abraham  F#, .NET and the Open Source situation
 Dec 08 (Tuesday)  Jonathan Wood  A Quick Look At F# In Visual Studio Code
 Dec 09 (Wednesday)  Reed Copsey, Jr.  Christmas Trees in WPF using FSharp.ViewModule
 Dec 09 (Wednesday)  Peter Bayne  The trips and traps of creating a Generative Type Provider in F#
 Dec 10 (Thursday)  Daniel Egloff  Algo Trading with F# and GPUs
 Dec 11 (Friday)  @TeaDrivenDev  Making Busy Progress in F#
 Dec 11 (Friday)  Reid Evans  Providing Value with Trivial Abstraction in F#
 Dec 12 (Saturday)  Eriawan Kusumawardho  What’s new in F# 4.0 in Visual Studio 2015
 Dec 12 (Saturday)  Riccardo Terrell  Solving the Santa Claus Problem in F#
 Dec 13 (Sunday)  Marcus Griep  Chiron: JSON + Ducks + Monads
 Dec 13 (Sunday)  @lenadroid  Learn the machine! #fsharp #accord.net
 Dec 14 (Monday)  Tomas Jansson  F#, event sourcing and CQRS tutorial… and agents
 Dec 14 (Monday)  Alex Casquete  Building an Hypermedia REST API with F# and Suave.IO
 Dec 15 (Tuesday)  Evelina Gabasova  The Star Wars social network
 Dec 15 (Tuesday)  Stachu Korick  Pseudocode-Driven Development with F#
 Dec 16 (Wednesday)  Yan Cui  Advent of Code F# – Day 16
 Dec 16 (Wednesday)  Paulmichael Blasucci  A Mixed-Paradigm Recipe for Exposing Native Code
 Dec 17 (Thursday)  Kunjan Dalal  1729
 Dec 17 (Thursday)  Jérémie Chassaing  Ukulele Fun for XMas !
 Dec 18 (Friday)  Anton Tcholakov  Using F# for scientific instrument control
 Dec 18 (Friday)  Matt Hawkins  ReST vs CQRS: The Trigger Pattern
 Dec 19 (Saturday)  Michael Newton  Angels From the Realms of Glory
 Dec 19 (Saturday)  Steven Pemberton  Let It Snow! A basic particle system in F# and WPF
 Dec 20 (Sunday)  Juan M Gómez  Developing mobile apps at the speed of light
 Dec 20 (Sunday)  Jorge Fioranelli  Reactive Messaging Patterns with F# and Akka.NET
 Dec 21 (Monday)  Steffen Forkmann  Automatic re-build and background tasks for suave.io websites
 Dec 21 (Monday)  Tomasz Heimowski  Property-based testing XSLT
 Dec 22 (Tuesday)  Mathias Brandewinder  Hacking together @wbfacts, a World Bank Twitter Bot
 Dec 22 (Tuesday)  Chad Boyer  F# Advent Calendar 2015
 Dec 23 (Wednesday)  Carsten König  F# advent 2015 – some fun with lambda calculus
 Dec 23 (Wednesday)  Troy Kershaw  Getting Started with SignalR using F# and OWIN
 Dec 24 (Thursday)  Matthew Sottile  Comparing trees, functionally
 Dec 24 (Thursday)  Craig Stuntz  Designing for Problems Too Big to Test
 Dec 25 (Friday)  Richard Griffiths  Monogame SnowFlakes – 2015
 Dec 25 (Friday)  Louie Bacaj  F# Powered Realtime Dashboard
 Dec 26 (Saturday)  Adam Granicz  WebSharper – a year in review
 Dec 26 (Saturday)  Chris Dobson  F#, Minecraft and a Raspberry Pi
 Dec 27 (Sunday)  @squeekeeper  Generating Markov text from YouTube comments
 Dec 27 (Sunday)  Indy Garcia  Twitter Local
 Dec 28 (Monday)  Pierre Irrmann  Visualizing F# Advent Calendar contributors
 Dec 29 (Tuesday)  Tamizh Vendan  Implementing API Gateway in F# Using Rx and Suave
 Dec 31 (Thursday)  Tomas Petricek  Happy New Year 2016 around the World
Jan 1 (Friday)  F# Software Foundation  Welcome to 2016 – A Call to Action

Swagger for F# Web Apps

nlp-logo-navbarSwagger is a simple yet powerful representation of your RESTful API. With the largest ecosystem of API tooling on the planet, thousands of developers are supporting Swagger in almost every modern programming language and deployment environment. With a Swagger-enabled API, you get interactive documentation, client SDK generation and discoverability.

Swagger is very powerful framework that is able to generate schema and rich UI for your RESTful API. As I know, Swagger is very popular framework especially in the non-.NET world. You probably have already seen Swagger UI (like this) at several resources.

It turns out that it is not hard to use Swagger for .NET F# apps. There is a project called Swashbuckle, which adds Swagger to WebApi projects.

First of all, you need to create F# ASP.NET Web API project. You can do it using “F# Web Application templates (MVC 5 and Web API 2.2) by Ryan Riley and Daniel Mohl“. Choose “Web API 2.2 and Katana 3.0” option in project creation wizard.

So, now you have F# Web App with RESTful API: CarsController with two services ‘/api/cars‘ and ‘/api/car/{id}‘. Everything is awesome but we do not have UI that is able to show a list of available services, their parameters and return types. Swashbuckle will help us here, we need to install ‘Swashbuckle.Core‘ package to our web app.

Install-Package Swashbuckle.Core

The last step is to update HttpConfiguration in Startup.fs in proper way to register Swagger. Add following three lines to the end of RegisterWebApi method.

open Swashbuckle.Application

type Startup() =
    static member RegisterWebApi(config: HttpConfiguration) =
        // ...
        // Swagger configuration
        config
          .EnableSwagger(fun c -> c.SingleApiVersion("v1", "My API") |> ignore)
          .EnableSwaggerUi();

That’s all! When you start your web application and open ‘/swagger/ui/index‘ URI you will see beautiful documentation for your RESTful API.
SwaggerUI

Real-time analytics with Apache Storm – now in F#

I think, therefore I spam.

Over the past several month I’ve been prototyping various aspects of  an IoT platform – or more specifically, exploring the concerns of “soft” real-time handling of communications with potentially hundreds of thousands of devices.

Up to this point, being in .NET ecosystem I’ve been building distributed solutions with a most excellent lightweight ESB – MassTransit, but for IoT we wanted to be a little closer to the wire. Starting with the clean slate and having discovered Apache Storm and Nathan’s presentation and I realized that it addresses exactly the challenges we have.

It appears to be the ultimate reactive microservices platform for lambda architecture: it is fairly simple, fault tolerant overall, yet embracing fire-n-forget and “let it fail” on the component level.

While Storm favours JDK for development, has extensive component support for Java developers and heavily optimizes for JRE components execution, it also supports “shell” components via its multilang protocol. Which is what, unlike Spark…

View original post 206 more words

Building Azure Service Fabric Actors with F# – Part 1

The Cockney Coder

This post is the first part of a brief overview of Service Fabric and how we can model Service Fabric Actors in F#. Part 1 will cover the details of how to get up and running in SF, whilst Part 2 will look at the challenges and solutions to modelling stateful actors in a OO-based framework within F#.

What is Service Fabric?

Service Fabric is a new service on Azure (currently in preview at the time of writing) which is designed to support reliable, scalable (at “hyper scale”) and maintainable distributed applications and services – with automatic support for things like replication of state across nodes, automatic failover & recovery and multi tenanting services on the same instances. It supports (currently) both stateful and stateless micro-services and actor model architectures (more on this shortly). The good thing about Service Fabric (SF) from a risk/reward point of view is that it’s…

View original post 887 more words

Creating a Generative Type Provider

Didactic Code

In my recently released Pluralsight course, Building F# Type Providers, I show how to build a type provider that uses erased types. To keep things simple I opted to not include any discussion of generative type providers beyond explaining the difference between type erasure and type generation. I thought I might get some negative feedback regarding that decision but I still believe it was the right decision for the course. That said, while the feedback I’ve received has been quite positive, especially for my first course, I have indeed heard from a few people that they would have liked to see generated types included as well.

There are a number of existing type providers that use generated types, most notably in my opinion is the SqlEnumProvider from FSharp.Data.SqlClient. That particular type provider generates CLI-enum or enum-like types which represent key/value pairs stored in the source database.

Although SqlEnumProvider…

View original post 898 more words