F# Weekly | Behind The Scenes

02/01/2024Blog, F# Weekly4 Comments

Update #1: The Newsletter

Since 2014, faithfully every Monday, I’ve dispatched the F# Weekly Newsletter to your mailboxes, amounting to nearly 500 editions to date. From the outset, I’ve adopted Tinyletter, a simple platform allowing seamless subscription and delivery of the F# Weekly newsletter.

However, Mailchimp, who took over TinyLetter in 2011, decided to bring it to an end on February 29, 2024. Consequently, I find myself compelled to seek a new home for the newsletter.

Seeing as the F# Weekly newsletter boasts nearly 1000 subscribers—a number exceeding the free tier limit offered by most services—I made the decision to move the newsletter into my WordPress blog, a resource I’ve already invested in, rather than integrating or purchasing a new service. Some of you are already reading this post from within your inbox.

You may wonder, what does this mean to you? It implies that you will also receive email notifications about my occasional blog posts, aside from the F# Weekly newsletters. If this deviates from your initial subscription interest, I deeply apologize. Feel free to modify or cancel your subscription. Gratefully, I’ve managed to configure WordPress to allow you to set your subscription preferences (as seen in the screenshot below). If my “musings” outnumber your tolerance, you can simply select the F# Weekly category 😁

I am immensely grateful to you for joining me on this journey and consuming every piece of F# news I’ve delivered. Still not subscribed? You can do so from this page 😉

Update #2: Twitter/X API Drama

Several moons ago, I put together an automation system that enabled me to gather #fsharp tweets with links, trimming my time spent crafting each F# Weekly down from 3-4 hours to about a single hour. Half a decade ago, I made this tool publicly available, transitioning it to .NET Core and Azure Functions, and deployed it on Azure via my MVP Azure credits.

But as fate would have it, in March 2023, Elon Musk decided to phase out Twitter API v1.1, charging a hefty $100+ monthly fee from enthusiasts who rely on programmatically reading tweets. This abrupt change drove me back to manually penning F# Weekly by hand, a decision that left me heartbroken. 💔

If you’ve noticed a dip in the quality of F# Weekly or found that I’ve overlooked your posts or videos on occasion:

Make sure to attach the #fsharp tag to your posts if you choose to announce them on X.
Don’t hesitate to tag me directly or @fsharponline if you’d like more visibility or retweets. If either account retweets your post, it’s almost certain to catch my eye during newsletter assembly.
If X isn’t your platform of choice, feel free to mention me on Mastodon or send me an email at sergey.tihon[at]gmail.com if you abstain from social media altogether.

If you have any genius ideas on improving the news collection process, making it more resilient, or possibly even rolling it out on a wider scale, I’m all ears! Don’t hesitate to reach out on social media or leave a comment on this post.

Update #3: Warm Gratitude for Donations

In 2021, I set up the buy me a coffee service as a delightful way for you to treat me to a cup of coffee. I want to pour out a tidal wave of THANKS to everyone who kindly donated over the last 3 years! Your generosity warms my heart. ❤️

The donations have sprinkled extra joy into my everyday life, allowing me to occasionally indulge in non-essential buys without the guilt of dipping into family funds. Here are a few examples of how I’ve put your contributions to use (beyond covering my WordPress subscription and domain expenses):

First and foremost, I’ve made a monthly donation to the Ionide project, and I would earnestly implore you to consider the same if you’re able. It’s crucial that we preserve the presence of a free, cross-platform F# IDE.

I’ve been battling with hand discomfort, which sometimes shoots pain back into my wrists and fingers. I regularly find myself alternating mouses and occasionally keyboards. This year, I decided to give myself an early Christmas present—an ergonomic columnar curved keyboard, the MoErgo Glove 80 with Red Pro Linear 35gf switches, and so far, it has won me over. 😊 I’ve swiftly adjusted to a comfortable typing speed that allows me to work without interruptions throughout the day.

Three years ago, I embarked on a journey to balance my screen time by swapping my digital device for the tactile joy of printed books, finding solace in their immersive stories just before bedtime. The therapeutic effect has improved not only my sleep but also my overall well-being. Over time, I’ve amassed a collection of captivating books. For those who share my love for reading, feel free to send me a friend request on Goodreads. I’m always intrigued to know what tales are keeping you hooked!

Recently, I found it impossible to resist spoiling my “big dog” 🐶 with a cozy new kennel for her to curl up in.

Once again, my heartfelt thanks to all of you for your generosity and for investing your time into reading F# Weekly. Here’s wishing you a 2024 brimming with joy and success!

New IKVM 8.2 & MavenReference for .NET projects

28/08/202228/08/2022Blog, F#14 Comments

Roughly 2 months ago, a new long-awaited version of IKVM was announced.

The killer feature of the 8.2 release is the support of .NET Core. So, now, IKVM can translate JAR files to .NET Core compatible DLLs. Woohoo!

Last week I migrated to the new IKVM two projects and can share my first impressions:

The next one is Stanford.NLP.CoreNLP v4.5.0 with .NET Core support #dotnet #ikvm #nlp https://t.co/V1UnDydOfn https://t.co/pfrd8GEeVC
— Sergey Tihon (@sergey_tihon) August 25, 2022

First impressions

First of all, it works! I am glad to see that project was revived and .NET Core was supported. It really opens up a lot of new possibilities for .NET developers. But, not everything is perfect yet:

No support for JDK higher than JDK 8.
This means that you cannot use JAR files compiled by JDK 9 for example. It is quite an old limitation, and the project requires an incredible amount of work to catch up with later versions. Everyone can help here 😉
IKVM Compiler (IKMVc) is hard to use.
Currently project ships two compilers (IKVMc): one for .NET Framework and one for .NET Core as well as two sets of runtime libraries for two runtimes. You should use .NET Core version of IKVMc with .NET Core set of runtime libraries to produce .NET Core version of a JAR file.
Is it confusing and quite hard to use! In most cases, you should not IKVMc use directly anymore, because the project gives you access to MavenReference / MSBuild integration (read further for details).
You should build on Windows with .NET Core 3.1 installed.
Windows is the only fully supported platform (once again, it only build-time dependency, produced DLLs will work on all platforms including Linux & macOS, and runtimes like .NET 6).
Linux version almost works, except for Sockets support.
The macOS version does not exist yet but looks like something is coming…

MavenReference

MavenReference is a very cute new addition to the IKVM family of tools that hide from you the complexity of IKVMc. All you need is to edit your csprof/fsproj file and reference to Maven package! (You may think about Maven like a Nuget from Java world)

Here is the sample:

  <ItemGroup>
	<PackageReference Include="IKVM" Version="8.2.1" />
	<PackageReference Include="IKVM.Maven.Sdk" Version="1.0.1" />
	<MavenReference Include="org.apache.opennlp:opennlp-tools" Version="1.9.4" />
  </ItemGroup>

IKVM package contains Java runtime libraries that you need to run compiled DLLs
IKVM.Maven.Sdk package provides MSBuild integration to use <MavenReference> (like IKVMc, dependencies resolution and etc)
org.apache.opennlp:opennlp-tools is actually a reference to the maven package that we want to recompile.

That is all you need to get started using Maven dependencies from .NET Core!

P.S. If something co wrong, just check that run build on Windows and Maven package compiled using JDK 8. JDK version (Build-Jdk-Spec) you can find in the manifest inside of the JAR file, do not confuse it with Bundle-RequiredExecutionEnvironment!

Build-Jdk-Spec: 11
Bundle-RequiredExecutionEnvironment: JavaSE-1.8

“Hmm… looks like this file doesn’t have preview we can show you”

19/01/2022SharePoint3 Comments

Microsoft allows you to embed previews of a document stored in SharePoint Online / OneDrive inside your site web page.

There is an API (driveItem: preview – Microsoft Graph v1.0) that allows you to obtain short-lived embeddable URL to render document preview.

Recently some of out users complained that they no longer see previews. Instead, they see very interesting message “Hmm… looks like this file doesn’t have preview we can show you”.

If you try to Google this issue you won’t find much, but if you bing it you will find few questions on MS Tech Community site that aren’t really helpful.

Let’s open Edge/Chrome dev tools (F12) and find out what is going on. On the network tab, I found few failed requests (related to online preview) with 404 status code.

What interesting about this image is that the response comes from ServiceWorker (and not from actual backend). This means that your browser has service worker installed for SharePoint Online that intercept request to the server and reply with 404 instead of passing the request to server.

Solution

We need to find and remove service worker that break our previews.

Open edge://serviceworker-internals/?devtools in Edge
Or open chrome://serviceworker-internals/?devtools in Chrome
Or go to Application > Service Workers > See all registrations

Find service worker registered for your SharePoint Online tenant (*.sharepoint.com) and Unregister it.

Unregister *.sharepoint.com service worker

That it, next time when you reload page with web preview, it should work.

When you visit web preview of a document in SharePoint Online it most likely register the service worker again, but new one should not break your previews.

Announcing OpenXML Package Explorer for VS Code

07/04/202107/04/2021Blog12 Comments

I am excited to announce my new project “OpenXML Package Explorer” extension for Visual Studio Code.

It allows to explore content of Office Open XML packages (*.pptx, *.docx, *.xlsx) inside Visual Studio Code. You can open any valid OpenXML package in Tree View, explore parts inside package, their relationships and content of xml parts (pre-formatted xml with highlighting).

Extension is inspired by windows-only PackageExplorer (lost in CodePlex archive) and Open XML Package Editor for Modern Visual Studios (Visual Studio for Windows) but implemented on top of .NET 5 using F#, Fable, Fable.Remoting, and re-using some pieces from Ionide.

How it works

I believe that source code of this extension can be used as one more sample of Fable-powered VSCode extension. Here is how it currently works:

F# source code compiled to JS using Fable 3 and bundled with webpack.
Code that interacts with OpenXML packages is written in .NET and uses the latest version of System.IO.Packaging
When extension is activated it starts .NET 5 process with API exposed using Fable.Remoting.
Extension assumes that .NET 5 runtime is installed on user machine but it depends on .NET Install Tool for Extension Authors that should help to install runtime to users that do not have it yet.
JS bundle runs inside Node.js VS Code Extension Host process.
Fable.Remoting.Client cannot be used, because it is built on top of XMLHttpRequest that does not exist in Node.js world. Here is feature request for Fable.Remoting.NodeClient that may become a thing one day.
Current client-server communication is built on top of axios library and Fable.Axios bindings with manually written client.
Extension is created using LambdaFactory/fable-vscode-demo sample that uses ionide/ionide-vscode-helpers with bindings for VS Code Extension API.

P.S. Many thanks for Krzysztof Cieślak and Ionide team for ionide-vscode-fsharp, ionide-vscode-helpers and fable-vscode-demo and Stef Levesque for vscode-zipexplorer used as reference implementation and inspiration.

`dotnet watch` with Microsoft.Identity.Web or custom IDistributedCache

05/04/202107/04/2021Blog6 Comments

Microsoft.Identity.Web is new (GA from Sept 30, 2020) library which contains a set of reusable classes used in conjunction with ASP.NET Core for integrating with the Microsoft identity platform (formerly Azure AD v2.0 endpoint) and AAD B2C.
AzureAD/microsoft-identity-web

Microsoft.Identity.Web project template is included in .NET 5.0 with tutorials like “Create a Blazor Server app that uses the Microsoft identity platform for authentication“. The library is really nice and easy to use, but development experience is not ideal yet.

When you run server with `dotnet watch` you will see following errors after each restart.

MsalUiRequiredException: No account or login hint was passed to the AcquireTokenSilent call.
Microsoft.Identity.Client.Internal.Requests.Silent.SilentRequest.ExecuteAsync

MicrosoftIdentityWebChallengeUserException: IDW10502: An MsalUiRequiredException was thrown due to a challenge for the user. See https://aka.ms/ms-id-web/ca_incremental-consent.
Microsoft.Identity.Web.TokenAcquisition.GetAuthenticationResultForUserAsync

The reason is not obvious from the error message. In fact, you browser calls the server with .AspNetCore.Cookies cookies that server cannot accept and cannot renew. What to do? Easy – open dev tool, clean cookies, refresh the page, wait for next server restart and repeat it again. You won’t last long.

Workaround with custom IDistributedCache

In Startup.cs you most likely find code similar

services.AddAuthentication(OpenIdConnectDefaults.AuthenticationScheme)
    .AddMicrosoftIdentityWebApp(Configuration.GetSection("AzureAd"))
        .EnableTokenAcquisitionToCallDownstreamApi(initialScopes)
            .AddMicrosoftGraph(Configuration.GetSection("DownstreamApi"))
            .AddInMemoryTokenCaches();

The last line (AddInMemoryTokenCaches) configure application to use in-memory cache that is empty after each server restart. We need to find a way to store tokens outside the app process and restore the cache after process restart.

Let’s take a look at supported caches out-of-the-box:

The only alternative to AddInMemoryTokenCaches is AddDistributedTokenCaches with ability to store tokens in memory, Redis, CosmosDB, SqlServer. Last three are nice options to distributed application but all of them are complicated for localhost development.

For our use case would be enough to serialise token cache to local file between restart. Luckily, it is not that complicated. We can implement IDistributedCache interface using TestDistributedCache as reference implementation.

public class LocalFileTokenCache :  IDistributedCache
{
    private class FileTransaction : IDisposable
    {
        public FileTransaction(string fileName = "cache.json")
        {
            var root = Path.GetDirectoryName(GetType().Assembly.Location);
            _fileName = Path.Combine(root, fileName);
            
            if (File.Exists(_fileName))
            {
                var str = File.ReadAllText(_fileName);
                Dict = JsonSerializer.Deserialize<Dictionary<string, byte[]>>(str);
            }
            
            Dict ??= new Dictionary<string, byte[]>();
        }

        private readonly string _fileName;
        public Dictionary<string, byte[]> Dict { get; }

        public void Dispose()
        {
            var str =JsonSerializer.Serialize(Dict);
            File.WriteAllText(_fileName, str);
        }
    }
    
    public byte[] Get(string key)
    {
        using var cache = new FileTransaction();
        return cache.Dict.TryGetValue(key, out var value) ? cache.Dict[key] : null;
    }

    public Task<byte[]> GetAsync(string key, CancellationToken token = default)
    {
        return Task.FromResult(Get(key));
    }

    public void Refresh(string key)
    {
        // Don't process anything
    }

    public Task RefreshAsync(string key, CancellationToken token = default)
    {
        Refresh(key);
        return Task.CompletedTask;
    }

    public void Remove(string key)
    {
        using var cache = new FileTransaction();
        cache.Dict.Remove(key, out _);
    }

    public Task RemoveAsync(string key, CancellationToken token = default)
    {
        Remove(key);
        return Task.CompletedTask;
    }

    public void Set(string key, byte[] value, DistributedCacheEntryOptions options)
    {
        using var cache = new FileTransaction();
        cache.Dict[key] = value;
    }

    public Task SetAsync(string key, byte[] value, DistributedCacheEntryOptions options, CancellationToken token = default)
    {
        Set(key, value, options);
        return Task.CompletedTask;
    }
}

LocalFileTokenCache implementation is not suitable for anything rather than local development.

The last step is to register LocalFileTokenCache in DI container as implementation of IDistributedCache instead of MemoryDistributedCache for development environment.

public class Startup
{
    public Startup(IConfiguration configuration, IWebHostEnvironment env)
    {
        Configuration = configuration;
        CurrentEnvironment = env;
    }

    public IConfiguration Configuration { get; }
    private IWebHostEnvironment CurrentEnvironment{ get; set; } 
    
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMicrosoftIdentityWebAppAuthentication(Configuration)
            .EnableTokenAcquisitionToCallDownstreamApi(new[]
            {
                "User.Read", "Files.ReadWrite.AppFolder", "Files.ReadWrite"
            }).AddDistributedTokenCaches();

        if (CurrentEnvironment.IsDevelopment())
        {
            services.AddSingleton<IDistributedCache, LocalFileTokenCache>();
        }
        else
        {
            services.AddSingleton<IDistributedCache, MemoryDistributedCache>();
        }
        //...
    }
    //...
}

P.S. I hope that proper fix will land to official template.

Update 2021-04-06: There is an official guide how to setup a Redis cache in a Docker container for local testing, that also can be used to local development.

ML.NET Recommendation Engine: Pitfall of One-Class Matrix Factorization

05/01/202125/02/2021Machine Learning and NLP10 Comments

During the weekends I decided to take a look at what ML.NET can propose in the area of recommendation engine.

I found a nice picture in Mark Farragher’s blog post that explains three available options:

The choice depends on what information you have.

If you have sophisticated user feedback like rating (or likes and most importantly dislikes) then we can use Matrix Factorization algorithm to estimate unknown ratings.
If we have not only rating but other product fields, we can use more advanced algorithm called “Field-Aware Factorization Machine”
If we have no rating at all then “One Class Matrix Factorization” is the only option for us.

In this post I would like to focus on the last option.

One-Class Matrix Factorization

This algorithm can be used when data is limited. For example:

Books store: We have history of purchases (list of pairs userId + bookId) without user’s feedback and want to recommend new books for existing users.
Amazon store: We have history of co-purchases (list of pairs productId + productId) and want to recommend products in section “Customers Who Bought This Item Also Bought”.
Social network: We have information about user friendship (list of pairs userId + userId) and want to recommend users in section “People You May Know”.

As you already understood, it is applicable for a pair of 2 categorical variables, not only for userId + productId pairs.

Google showed several relevant posts about the usage of ML.NET One Class Matrix Factorizarion:

Official MS “Product Recommendation – Matrix Factorization problem sample“
Build A Product Recommender Using C# and ML.NET Machine Learning by Mark Farragher
Build a Recommendation engine with ML.NET and F# by Riccardo Terrell

After reading all these 3 samples I realised that I do not fully understand what is Label column is used for. Later I came to a conclusion that all three samples most likely are incorrect and here is why.

Mathematical details

Let’s take a look at excellent documentation of MatrixFactorizationTrainer class. The first gem is

There are three input columns required, one for matrix row indexes, one for matrix column indexes, and one for values (i.e., labels) in matrix. They together define a matrix in COO format. The type for label column is a vector of Single (float) while the other two columns are key type scalar.
COO stores a list of (row, column, value) tuples. Ideally, the entries are sorted first by row index and then by column index, to improve random access times. This is another format that is good for incremental matrix construction

So anyway we need three columns. If in the classic Matrix Factorization the Label column is the rating, then for One-Class Matrix Factorization we need to fill it with something else.

The second gem is

The coordinate descent method included is specifically for one-class matrix factorization where all observed ratings are positive signals (that is, all rating values are 1). Notice that the only way to invoke one-class matrix factorization is to assign one-class squared loss to loss function when calling MatrixFactorization(Options). See Page 6 and Page 28 here for a brief introduction to standard matrix factorization and one-class matrix factorization. The default setting induces standard matrix factorization. The underlying library used in ML.NET matrix factorization can be found on a Github repository.

Here is Page 28 from references presentation:

As you see, Label is expected to be always 1, because we watched only One Class (positive rating): user downloaded a book, user purchased 2 items together, there is a friendship between two users.

In the case when data set does not provide rating to us, it is our responsibility to provide 1s to MatrixFactorizationTrainer and specify MatrixFactorizationTrainer.LossFunctionType as loss function.

Here you can find fixes for samples:

Introducing Clippit, get your slides out of PPTX.

15/03/202007/04/2021Blog2 Comments

TL;TR

Clippit is .NETStandard 2.0 library that allows you to easily and efficiently extract all slides from PPTX presentation into one-slide presentations or compose slides back together into one presentation.

Why?

PowerPoint is still the most popular way to present information. Sales and marketing people regularly produce new presentations. But when they work on new presentation they often reuse slides from previous ones. Here is the gap: when you compose presentation you need slides that can be reused, but result of your work is the presentation and you are not generally interested in “slide management”.

One of my projects is enterprise search solution, that help people find Office documents across different enterprise storage systems. One of cool features is that we let our users find particular relevant slide from presentation rather than huge pptx with something relevant on slide 57.

How it was done before

Back in the day, Microsoft PowerPoint allowed us to “Publish slides” (save each individual slide into separate file). I am absolutely sure that this button was in PowerPoint 2013 and as far as I know was removed from PowerPoint 365 and 2019 versions.

When this feature was in the box, you could use Microsoft.Office.Interop.PowerPoint.dll to start instance of PowerPoint and communicate with it using COM Interop.

	public void PublishSlides(string sourceFileName, string resultDirectory)
	{
	Application ppApp = null;
	try
	{
	ppApp = new Application
	{
	DisplayAlerts = PpAlertLevel.ppAlertsNone
	};

	const MsoTriState msoFalse = MsoTriState.msoFalse;
	var presentation = ppApp.Presentations.Open(sourceFileName,
	msoFalse, msoFalse, msoFalse);

	presentation.PublishSlides(resultDirectory, true, true);
	presentation.Close();
	}
	finally
	{
	ppApp?.Quit();
	}
	}

view raw PP-Interop.cs hosted with ❤ by GitHub

But server-side Automation of Office has never been recommended by Microsoft. You were never able to reliably scale your automation, like start multiple instances to work with different document (because you need to think about active window and any on them may stop responding to your command). There is no guarantee that File->Open command will return control to you program, because for example if file is password-protected Office will show popup and ask for password and so on.

That was hard, but doable. PowerPoint guarantees that published slides will be valid PowerPoint documents that user will be able to open and preview. So it is worth playing the ceremony of single-thread automation with retries, timeouts and process kills (when you do not receive control back).

Over the time it became clear that it is the dead end. We need to keep the old version of PowerPoint on some VM and never touch it or find a better way to do it.

The History

Windows only solution that requires MS Office installed on the machine and COM interop is not something that you expect from modern .NET solution.

Ideally it should be .NETStandard library on NuGet that platform-agnostic and able to solve you task anywhere and as fast as possible. But there was nothing on Nuget few months ago.

If you ever work with Office documents from C# you know that there is an OpenXml library that opens office document, deserialize it internals to an object model, let you modify it and then save it back. But OpenXml API is low-level and you need to know a lot about OpenXml internals to be able to extract slides with images, embedding, layouts, masters and cross-references into new presentation correctly.

If you google more you will find that there is a project “Open-Xml-PowerTools” developed by Microsoft since 2015 that have never been officially released on NuGet. Currently this project is archived by OfficeDev team and most actively maintained fork belongs to EricWhiteDev (No NuGet.org feed at this time).

Open-Xml-PowerTools has a feature called PresentationBuilder that was created for similar purpose – compose slide ranges from multiple presentations into one presentation. After playing with this library, I realized that it does a great job but does not fully satisfy my requirements:

Resource usage are not efficient, same streams are opened multiple times and not always properly disposed.
Library is much slower than it could be with proper resource management and less GC pressure.
It generates slides with total size much larger than original presentation, because it copies all layouts when only one is needed.
It does not properly name image parts inside slide, corrupt file extensions and does not support SVG.

It was a great starting point, but I realized that I can improve it. Not only fix all mentioned issues and improve performance 6x times but also add support for new image types, properly extract slide titles and convert then into presentation titles, propagate modification date and erase metadata that does not belong to slide.

How it is done today

So today, I am ready to present the new library Clippit that is technically a fork of most recent version of EricWhiteDev/Open-Xml-PowerTools that is extended and improved for one particular use case: extracting slides from presentation and composing them back efficiently.

All classes were moved to Clippit namespace, so you can load it side-by-side with any version of Open-Xml-PowerTools if you already use it.

The library is already available on NuGet, it is written using C# 8.0 (nullable reference types), compiled for .NET Standard 2.0, tested with .NET Core 3.1. It works perfectly on macOS/Linux and battle-tested on hundreds of real world PowerPoint presentations.

New API is quite simple and easy to use

	var presentation = new PmlDocument(sourceFile);

	// Pubslish slides
	var slides = PresentationBuilder.PublishSlides(presentation).ToList();

	// Save slides into files
	foreach (var slide in slides)
	{
	var targetPath = Path.Combine(targetDir, Path.GetFileName(slide.FileName))
	slide.SaveAs(targetPath);
	}

	// Compose slides back into one presentation
	var sources = slides.Select(x => new SlideSource(x, keepMaster:true)).ToList();
	PresentationBuilder.BuildPresentation(sources)
	.SaveAs(newFileName);

view raw Clippit-101.cs hosted with ❤ by GitHub

P.S. Stay tuned, there will be more OpenXml goodness.

HashiCorp Vault and TLS Certificate Authentication for .NET Applications (Comprehensive guide)

08/10/201907/04/2021Blog4 Comments

HashiCorp Vault is a tool for secrets management, encryption as a service, and privileged access management. It is quite popular nowadays, especially if you own your own infrastructure, private cloud or just cannot store your secrets using Key Vault services provided by Azure/AWS/GCP.

I assume that you already have one up and running instance of HashiCorp Vault, otherwise you may install one using official Installing Vault guide.

Why TLS certificate authentication?

Vault supports many Auth Methods. But what if you are still deploying your app on plain old Windows Server VMs or develop SharePoint application (like I am 😝).

The challenge in this case, that you have to authenticate in Vault in order to get a secret. This means that we need to choose auth method that protects our auth secrets from an accident IT guys who may login on the VM (or malicious code that may find it on file system)

TLS Certificate Auth is a good solution candidate, because we can install certificate into windows certificate store, protect private key (mark it as not-exportable) and even specify list of service accounts, allowed to use this certificate for authentication.

TLS certificate generation

I will be using ssh command on my macOS for certificate generation and Vault configuration, but you can repeat the same step from Window for sure.

For our needs we will use self-signed certificate. You can generate one using OpenSSL. If you do not have OpenSSL installed, you can install from Homebrew.

brew install openssl

First of all we generate private key (it is highly secured, do not share it)

openssl genrsa 2048 > vault_private.pem

Then we generate public part of the key in .pem format (.pem file will be uploaded to Vault for client validation during authentication)

openssl req -x509 -new -key vault_private.pem -out vault_public.pem -days 365

Answer all questions properly, it will help you identify this certificate in future (I’ve created certificate that is valid for 365 days, but you should follow security standards defined in you company).

vault-cfg_sh_—_private-key.png

Note: Common Name cannot be empty, otherwise you will not be able to use this certificate to retrieve the secret (Vault returns ‘missing name in alias’ error). Thank you Vadzim Makarchyk for this note.

The final step is to archive both parts in .pfx format (.pfx file will be deployed into Windows Server certificate store on all machines from where our code should have access to Vault)

openssl pkcs12 -export -in vault_public.pem -inkey vault_private.pem -out vault.pfx

vault-cfg_sh_—_vault-cfg.png

Remember the password entered during *.pfx creation, you’re gonna need it every time you decide to install it on Windows machine.

Vault configuration

In order to configure HashiCorp Vault we will use Vault CLI interface, that can be installed from Homebrew on macOS.

brew install vault

Vault CLI uses environment variables for configuration. My Vault server is hosted on different machine so I need to provide server Url.

VAULT_ADDR=https://my.server.com:8200

export VAULT_ADDR

I uses Enterprise version of Vault that is used by several teams, that it why I also specify namespace (aka folder for my secrets)

VAULT_NAMESPACE=dev/my-team

export VAULT_NAMESPACE

I am lazy to properly setup certificates for Vault CLI, that is why I skip certificate validation (never repeat it in production 😉)

VAULT_SKIP_VERIFY=true

export VAULT_SKIP_VERIFY

We are almost ready to login. The easiest option is to login using Web UI and then reuse issued token in the terminal. Login using your favorite browser, pass authentication and copy token in buffer.

EPAM_Laptop

vault login s.fJTY5S51oIfXKnBAG3Qq5eWp.9GKyY

That is it! Token is saved into ~/.vault-token and CLI is ready to use!

Key/Value secret engine creation

Vault supports multiple Secret Engines, but for our demo we create simple Key/Value storage for secrets (for example to store logins and passwords)

vault secrets enable -path=kv kv

This command enable key/value engine (V1) and name kv (-path param)

NOTE: The kv secrets engine has two versions: kv and kv-v2. To enable versioned kv secrets engine, pass kv-v2 instead.

Engine is ready, but it is empty – let’s fix it.

vault write kv/my-secret value="s3c(eT"

This command effectively creates my-secret secret inside kv secret engine and store one key/value pair inside value=”s3c(eT”

ACL Policy creation

Secret engine is secured, nobody (except you, admin) has access to secrets. We need to create rules/policy that define what access we want to provide. Create new files policy-file.hcl and put following content inside.

path "kv/*" {
  capabilities = ["read", "list"]
}

This policy allows to read and list all secrets inside kv secret engine. All users with this policy will be able to read secrets from our engine. Read more about policies.

Write this policy to the server (and name it policy-name)

vault policy write policy-name policy-file.hcl

TLS Certificates – Auth Method

The last step is to assign this policy. But we want to assign it to all clients authenticated in Vault using TLS certificate created by us earlier.

Fist of all we need to enable certificate authentication in our namespace

vault auth enable cert

and create certificate auth in Vault (name it app), assign policy-name to it and upload the public part of generated key (vault_public.pem)

vault write auth/cert/certs/app policies=policy-name certificate=@vault_public.pem

That is it! Vault is configured and waiting for first connection.

TLS certificate deployment

TLS certificate allows us to deploy it to certain set of machines that should have access to the Vault and then specify which accounts (on these machines) may use it for authentication.

If you are lucky enough and your deployment is automated you can add one more build step in your deployment process that ensures that certificate is provisioned on all target machines. Octopus Deploy is one of such tools that provides built-in template for certificate provisioning. (BTW, it is free for small teams starting from Sept 2, 2019)

On the screenshot you see the step that imports certificate on all target machines with tag SharePoint (in my case) to LocalMachine certificate store to My/Personal store, mark private-key as not exportable and provide access to private key to 2 service accounts.

If your deployment is not automated, you may script the same steps using PowerShell and run it on all machines.

	#Import certificate to local machine personal folder
	$root = Set-Location -PassThru $PSScriptRoot

	$cert = Get-ChildItem -Path $root \| where {$_.Extension -like "*.pfx"}

	$PlainTextPass = Read-Host -Prompt "Type .pfx password for '$cert' certificate"
	$pfxpass = $PlainTextPass \| ConvertTo-SecureString -AsPlainText -Force

	$cert = $cert \| Import-PfxCertificate -CertStoreLocation Cert:\LocalMachine\My -Exportable -Password $pfxpass
	Write-Host "Certificate is imported"


	#Grant permission to selected account on private key and MachineKeys folder
	$fileName = $cert.PrivateKey.CspKeyContainerInfo.UniqueKeyContainerName
	$path = "$env:ALLUSERSPROFILE\Microsoft\Crypto\RSA\MachineKeys\$fileName"

	function SetPermissions([string[]]$accountNames)
	{
	$acl = Get-Acl -Path $path

	# Add the new user and preserve all current permissions: SetAccessRuleProtection(False, X)
	# Add the new user and remove all inherited permissions: SetAccessRuleProtection(True, False)
	# Add the new user and convert all inherited permissions to explicit permissions: SetAccessRuleProtection(True, True)
	$acl.SetAccessRuleProtection($True, $False)

	foreach ($accountName in $accountNames) {
	$rule = New-Object System.Security.AccessControl.FileSystemAccessRule($accountName,"Full","Allow")
	$acl.AddAccessRule($rule)
	}
	Set-Acl -Path $path -AclObject $acl

	Write-Host "Access to certificate is granted for $accountNames"
	}

	SetPermissions(@(
	"me@sergeytihon.com",
	"you@sergeytihon.com"
	))

view raw

importCerfiticate.ps1

hosted with ❤ by GitHub

If you are brave, you can click it even manually! 🙈

Double click on vault.pfx file and choose LocalMachine store location
Click Next, Next and type password used during *.pfx creation and Next again.
Choose Personal certificate store.
Click Next, Finish, OK – your certificated in the store!
Execute mmc (Microsoft Managed Console) from start menu.
File -> Add/Remove Snap-in …
Certificate, Add, Computer account and click Next & Ok
Find our certificate and click Manage Private Keys…
On this screen you can manage the list of accounts that will be able to use this certificate for authentication on the current machine.

.NET client application

Vault is ready, machine is ready (service account / current user is allowed to use certificate from the LocalMachine/Personal store). Few lines of code are separating us from success 😊.

I will use VaultSharp NuGet Package. It is more or less up to date, it supports namespaces feature and starting from next release usage of namespaces will become even more intuitive.

	using System;
	using System.Collections.Generic;
	using System.Data;
	using System.Linq;
	using System.Threading.Tasks;
	using System.Security.Cryptography.X509Certificates;
	using VaultSharp;
	using VaultSharp.V1.AuthMethods.Cert;

	namespace SergeyTihon.App.Configuration
	{
	public class VaultSecretProvider
	{
	public VaultSecretProvider(string vaultUrl, string vaultNamespace, string certificateThumbprint)
	{
	var clientCertificate = GetCertificate(certificateThumbprint);
	var authMethod = new CertAuthMethodInfo(clientCertificate);

	_vaultClient = new VaultClient(new VaultClientSettings(vaultUrl, authMethod)
	{
	BeforeApiRequestAction = (httpClient, httpRequestMessage) =>
	{
	httpRequestMessage.Headers.Add("X-Vault-Namespace", vaultNamespace);
	}
	});
	}

	public static X509Certificate2 GetCertificate(string certThumbprint)
	{
	var store = new X509Store(StoreName.My, StoreLocation.LocalMachine);
	store.Open(OpenFlags.ReadOnly);
	var certCollection = store.Certificates;

	// Find unexpired certificates.
	var currentCerts = certCollection.Find(X509FindType.FindByTimeValid, DateTime.Now, false);

	// From the collection of unexpired certificates, find the ones with the correct thumbprint.
	var signingCert = currentCerts.Find(X509FindType.FindByThumbprint, certThumbprint, false);

	// Return the first certificate in the collection, has the right name and is current.
	var cert = signingCert.OfType<X509Certificate2>().OrderByDescending(c => c.NotBefore).FirstOrDefault();
	store.Close();

	if (cert is null)
	{
	throw new DataException($"Cannot find valid certificate with thumbprint {certThumbprint}");
	}
	return cert;
	}

	private readonly VaultClient _vaultClient;

	public async Task<Dictionary<string, object>> GetValue(string path, string mountPoint)
	{
	var secret = await _vaultClient.V1.Secrets.KeyValue.V1.ReadSecretAsync(path, mountPoint);
	return secret.Data;
	}
	}
	}

view raw

VaultSecretProvider.cs

hosted with ❤ by GitHub

VaultSecretProvider find X509 certificate in StoreName.My / StoreLocation.LocalMachine, then create CertAuthMethodInfo using certificate and VaultClient that X-Vault-Namespace header to each request with vaultNamespace name.

Using configured instance of VaultClient we can request our secret from Vault _vaultClient.V1.Secrets.KeyValue.V1.ReadSecretAsync(path, mountPoint) specifying path to the secret and mountPoint (name of secret engine).

We are ready to call and receive secrets

	new VaultSecretProvider(
	"https://my.server.com:8200", // VAULT_ADDR
	"dev/my-team", // VAULT_NAMESPACE
	"877501d5a018e9344088fd5c89580f6b095f5326" // vault.pfx certificate thumbprint
	).GetValue("my-secret", "kv") // path to the secret – vault write kv/my-secret value="s3c(eT"

view raw

VaultSecretProviderTests.cs

hosted with ❤ by GitHub

Conclusion

Wow, this became a long read, but I hope it was a good one.

TLS certificate authenctication in Vault is a good option for apps that uses Full .NET Framework and runs inside Windows Server VMs.

Just do not forget renew/replace certificates regularly.

Be better WPF / MvvmLight developer in 2018

16/04/201807/04/2021Blog7 Comments

It is 2018, the time of .NET Core, x-plat, clouds, microservices, blockchain and shine of JavaScript. But, there are guys, like me, who still maintain and sometimes develop classic .NET desktop applications for Windows.

I am not a WPF expert, but I spent a couple of days reviewing, testing and fixing one of our desktop apps and I definitely learned a couple of new tips & tricks that worth to share with other non-experts.

Part #1: General Tips

Tip #1.1: Choose right library/framework

It happened that we use MvvmLight. The library is lightweight and already exists for almost 10 years, MVVM pattern is well-known and lets us keep solution code reasonably well-structured.

But this is definitely not the only choice, there are many other different-purpose libraries and frameworks that may suit you better, especially if you do green-field development. So choose carefully:

Tip #1.2: Distribute using Squirrel.Windows

Installers always were hard, the seamless auto-update process is even harder. But, today, we have the solution that works for simple user-oriented applications that don’t do crazy things during installation. This solution is Squirrel.Windows – an installation and update framework for Windows desktop apps, designed for C# apps.

It’s definitely worth to learn it once and use it for all apps that you develop.

Tip #1.3: Think about monitoring

Aggregated analytics from user’s machine is priceless for successful apps. There are plenty amount of data that can help you deliver better apps:

Crash reports
Application version distribution
User’s count / Active user
Performance / Integrations tracking
Custom events / Logs

It is not always possible to collect all kinds of data from user’s machine, but do it if you can. There are a couple of services that may help you, like Application Insights, HockeyApp and others.

Tip #1.4: Use the full power of IDE

Learn tools that MS baked for you and use them

Tip #1.5: Debug Data Binding Issues

When data bindings do not play nice you have a possibility to debug. It is not super intuitive, but there are ways to step into the binding process to better figure out what is actually going on. Check this nice article from Mike Woelmer – How To Debug Data Binding Issues in WPF

Part #2: MVVM Light – Code Tips

C# quickly evolves over time, more and more features become available to us. It is not always obvious how to use new async code with an old API.

Tip #2.1: “New” INotifyPropertyChanged syntax

I think almost any WPF developer knows how to implement INotifyPropertyChanged interface

public class MyViewModel : INotifyPropertyChanged
{
    private string _isBusy;
    public event PropertyChangedEventHandler PropertyChanged;

    public MyViewModel() {}

    public string IsBusy
    {
        get { return _isBusy; }
        set
        {
            _isBusy = value;
            OnPropertyChanged("IsBusy");
        }
    }

    protected void OnPropertyChanged(string name)
    {
        if (PropertyChanged == null)
            return;
        PropertyChanged(this,
            new PropertyChangedEventArgs(name))
    }
}

Using MVVM Light you can do way shorter (such syntax probably exists for a while, but I discovered it only recently)

public class MyViewModel : ViewModelBase
{
    public MyViewModel() {}

    private string _isBusy;
    public string IsBusy
    {
        get => _isBusy;
        set { Set(() => IsBusy, ref _isBusy, value) }
    }
}

All property change events will happen under the hood of Set method. Also, Set method returns true when the value changed so you can use it to do additional actions on property change.

private string _isBusy;
public string IsBusy
{
    get => _isBusy;
    set {
        if (Set(() => IsBusy, ref _isBusy, value)) {
        // Do whatever you need on update
        }
    }
}

Update from Chris Jobson:

There are overloads that allow us to omit first argument – propertyExpression. In this case [CallerMemberName] will be used as the property name, so the code will be even shorter. Not bad for 2018 =)

private string _isBusy;
public string IsBusy
{
    get => _isBusy;
    set => Set(ref _isBusy, value);
}

Tip #2.2: Async to Action glue

C# async was designed to be better compatible with old APIs and consume Action or delegate. Also, it is one of the reasons why async void exists in the language, but we should always use async Task in our own code.

Two following casts are valid

Action task = async () => await Task.Yield();
Func task2 = async () => await Task.Yield();

Read “Do async lambdas return Tasks?” to better understand what’s actually going on here. It means that you can pass your async method as Action to RelayCommand.

new RelayCommand(async() => await Download());

TBH, you should use it like this (explanation in the next tip)

new RelayCommand(async() => await Download(), keepTargetAlive:true);

Tip #2.3: Do not use lambdas with RelayCommand

Lambdas as a parameter for RelayCommand is a bad idea unless you know what can go wrong and use the latest version of MvvmLight.

Actually, I have spent almost 2 days of my life to figure out why at some point of time several buttons in our application stopped working, even though all commands defined in the ViewModel are read-only and assigned once in the constructor.

We had simple commands that do some trivial actions on click, so the developer decided to use lambda in command declaration to save space and simplify the code.

new RelayCommand(() => IsBusy = true);

The code looks simple and correct, but RelayCommand under the hood stores only weak reference to the delegate and any GC cycle can recycle local lambda function. So at some point in time (after next cycle of GC) RelayCommand may not find delegate to call and nothing will happen after the click. For a deeper analysis of this behavior, you can read “RelayCommands and WeakFuncs“.

At the time of writing this post, the issues in MvvmLight library were fixed (Using RelayCommand and Messenger (and WeakAction) with closures) and released in version 5.4.1. But fix does not apply by default.

If you really want to use lambdas with RelayCommand & Messenger you should manually set keepTargetAlive:true (false by default), but probably better do not use them at all.

new RelayCommand(() => IsBusy = true, keepTargetAlive:true);

P.S. Worth to mention that Laurent Bugnion has the course on Pluralsight “MVVM Light Toolkit Fundamentals” that provides detailed MVVM Light overview.