F# and FAST Search for SharePoint 2010

If you are a SharePoint developer, an Enterprise Search developer or an employee of a large corporation with Global Search through private internal infrastructure then you may be interested in search automation. Deployment of FAST Search Server 2010 for SharePoint (F4SP) is out of the current post’s scope (you can follow TechNet F4SP Deployment Guide if you need).

F# 3.0 comes with feature called “type providers” that helps you to simplify your life in daily routine. For the case of WCF, the Wsdl type provider allows us to automate the proxy generation. Here we need to note that, F# 3.0 works only on the .NET 4.0 and later, but SharePoint 2010 server side runs exclusively on the .NET 3.0 64bit. Let’s see how this works together.

Connecting to the web service

Firstly, we create an empty F# Script file.

#r "System.ServiceModel.dll"
#r "FSharp.Data.TypeProviders.dll"
#r "System.Runtime.Serialization.dll"

open System
open System.Net
open System.Security
open System.ServiceModel
open Microsoft.FSharp.Data.TypeProviders

[<Literal>]
let SearchServiceWsdl = "https://SharePoint2010WebAppUrl/_vti_bin/search.asmx?WSDL"
type SharePointSearch = Microsoft.FSharp.Data.TypeProviders.WsdlService<SearchServiceWsdl>

At this point, the type provider creates proxy classes in the background. The only thing we need to do is to configure the access security. The following code tested on the two SharePoint 2010 farms with NTLM authentication and HTTP/HTTPS access protocols.

let getSharePointSearchService() =
    let binding = new BasicHttpBinding()
    binding.MaxReceivedMessageSize <- 10000000L
    binding.Security.Transport.ClientCredentialType <- HttpClientCredentialType.Ntlm
    binding.Security.Mode <- if (SearchServiceWsdl.StartsWith("https"))
                                 then BasicHttpSecurityMode.Transport
                                 else BasicHttpSecurityMode.TransportCredentialOnly

    let serviceUrl = SearchServiceWsdl.Remove(SearchServiceWsdl.LastIndexOf('?'))
    let service = new SharePointSearch.ServiceTypes.
                        QueryServiceSoapClient(binding, EndpointAddress(serviceUrl))
    //If server located in another domain then we may authenticate manually
    //service.ClientCredentials.Windows.ClientCredential
    //  <- (Net.NetworkCredential("User_Name", "Password"))
    service.ClientCredentials.Windows.AllowedImpersonationLevel
        <- System.Security.Principal.TokenImpersonationLevel.Delegation;
    service

let searchByQueryXml queryXml =
    use searchService = getSharePointSearchService()
    let results = searchService.QueryEx(queryXml)
    let rows = results.Tables.["RelevantResults"].Rows
    [for i in 0..rows.Count-1 do
        yield (rows.[i].ItemArray) |> Array.map (sprintf "%O")]

Building search query XML

To query F4SP we use the same web service as for build-in SharePoint 2010 search, but with a bit different query XML. The last thing that we need to do is to build query.You can find query XML syntax on Microsoft.Search.Query Schema, but it is hard enough to work with it using official documentation. There is a very useful CodePlex project called FAST Search for Sharepoint MOSS 2010 Query Tool which provides a user-friendly query builder interface.

F4SPQueryTool

FAST Query Language (FQL) Syntax

FAST has its own query syntax(FQL Syntax) that can be directly used through SharePoint Search Web Service.

let getFQLQueryXml (fqlString:string) =
  """<QueryPacket Revision="1000">
       <Query>
         <Context>
           <QueryText language="en-US" type="FQL">{0}</QueryText>
         </Context>
         <SupportedFormats Format="urn:Microsoft.Search.Response.Document.Document" />
         <ResultProvider>FASTSearch</ResultProvider>
         <Range>
           <StartAt>1</StartAt>
           <Count>5</Count>
         </Range>
         <EnableStemming>false</EnableStemming>
         <EnableSpellCheck>Off</EnableSpellCheck>
         <IncludeSpecialTermsResults>false</IncludeSpecialTermsResults>
         <IncludeRelevantResults>true</IncludeRelevantResults>
         <ImplicitAndBehavior>false</ImplicitAndBehavior>
         <TrimDuplicates>true</TrimDuplicates>
         <Properties>
           <Property name="Url" />
           <Property name="Write" />
           <Property name="Size" />
         </Properties>
       </Query>
     </QueryPacket>"""
  |> (fun queryTemplate -> String.Format(queryTemplate,fqlString))

let fqlQueryResults =
  """and(string("Functional Programming", annotation_class="user", mode="phrase"),
         or("fileextension":string("ppt", mode="phrase"),
            "fileextension":string("pptx", mode="phrase")))
     AND filter(and(isdocument:1))"""
  |> getFQLQueryXml |> searchByQueryXml

Keyword Query Syntax

FAST also supports native SharePoint Keyword Query Syntax.

let getKeywordQueryXml (keywordString:string) =
  """<QueryPacket Revision="1000">
       <Query>
         <Context>
           <QueryText language="en-US" type="STRING">{0}</QueryText>
         </Context>
         <SupportedFormats Format="urn:Microsoft.Search.Response.Document.Document" />
         <ResultProvider>FASTSearch</ResultProvider>
         <Range>
           <StartAt>1</StartAt>
           <Count>5</Count>
         </Range>
         <EnableStemming>false</EnableStemming>
         <EnableSpellCheck>Off</EnableSpellCheck>
         <IncludeSpecialTermsResults>false</IncludeSpecialTermsResults>
         <IncludeRelevantResults>true</IncludeRelevantResults>
         <ImplicitAndBehavior>false</ImplicitAndBehavior>
         <TrimDuplicates>true</TrimDuplicates>
         <Properties>
           <Property name="Url" />
           <Property name="Write" />
           <Property name="Size" />
         </Properties>
       </Query>
     </QueryPacket>"""
  |> (fun queryTemplate -> String.Format(queryTemplate,keywordString))

let simpleKeywordQueryResults =
  """"Functional Programming" scope:"Documents" (fileextension:"PPT" OR fileextension:"PPTX")"""
  |> getKeywordQueryXml |> searchByQueryXml

Query Syntax Summary

One of the principal differences between two syntaxes is that Keyword Query needs to be converted into FQL on the SharePoint side. Keyword syntax also supports scope conditions, which will be converted into FQL filters. For example “scope:”Documents”” will be translated into ” filter(and(isdocument:1))” (In the case when Documents scope exists in the SharePoint Query Service Application).  Unfortunately, we can not specify SharePoint scope in FQL query.

My wish list for FSI

I have thought about possible directions of FSI development. I have some ideas that I would like to share and discuss.

In general, it would be great to make FSI more than just an execution shell. It can become a really powerful development tool. What exactly we can do:

1) Session Save/Load functionality

It will be nice to have an ability to save current FSI state into the file and continue work in the next time.

2) Native C# interaction with FSI saved sessions

It could sound crazy, but  it would be cool to save FSI session to assembly and provide an easy C#-F# interaction with such assemblies.  For example, it might be useful for Machine Learning tasks. You will be able to create, train and tune your model directly from FSI then save it to assembly and use it from your C# application.

3) Current environment overview

Provide an ability to see the difference between current FSI state and default state. For example:

  1. Print all declared variables with their values
  2. Print all declared function with their signatures
  3. Print list of referenced assemblies with included directories
  4. Provide an ability to print function’s body

There is a tool that provides a part of this functionality. This is FsEye – a visual object tree inspector for the F# Interactive.

4) Intellisense for FSI
5) Detailed type provider description

One of the main cons in type providers is impossibility to explore the schema of the provided world. For example, I want to use Freebase type provider, but I do not know the nature of the provided types. I cannot use provided types fully, because I do not know what data they have.

I am pretty much sure that we need to supply a way to explore provided types. This feature for sure should be tightly integrated with F# IDEs (into Object browser for example).

6)Interactive help

All we use MSDN to understand how things work. We copy type name + function name and paste them to Google, find first MSDN link and go there. Why cannot we teach FSI to do it? We just need to build an easy syntax for that.

7) The ability to use the load directive inside of modules\functions (by @davidkl_)

P.S. I will be happy to hear your feedback or new ideas. Feel free to post comments here or to contact me in twitter (@sergey_tihon).

OData Type Provider with SharePoint 2010

‘Type Providers’ is an extremely cool F# feature that was introduced with F 3.0 and shipped with VS 2012 and .NET 4.5. You can find Type providers’ explanation  here and details about OData type provider here.

It maybe not a news, but OData Type Provider works pretty well with SharePoint 2010.

SharePoint 2010 has a OData Service. You can find mode details about this service at the MSDN article Query SharePoint Foundation with ADO.NET Data Services

One special thing that distinguishes SharePoint service from the others is that you should be authenticated. Just set your credentials into created data content before using it. You can find full code sample below.


#r "FSharp.Data.TypeProviders.dll"
#r "System.Data.Services.Client.dll"

open Microsoft.FSharp.Data.TypeProviders
open System.Net

type sharepoint = ODataService<"http://server_name/_vti_bin/listdata.svc">
let web = sharepoint.GetDataContext() in
    web.Credentials <-
        (NetworkCredential("user_name", "password", "domain") :> ICredentials)

(web.Documents)
    |> Seq.iter (fun item -> printfn "%A : %s" item.Modified item.Name)