Using .NET Core Kestrel with NGINX and Unix Sockets

I haven’t blogged in quite a while it seems, although in my defence I’ve moved house and done several projects which kept me quite busy.

I’ve been working a lot lately with .NET Core, in particular building out web services through ASP.NET Core and Kestrel, the new light-weight web server. These services were developed on Windows and then deployed out to various Linux servers for production.

When it comes to exposing the web services to the wider world there’s two possibilities, the first is using Kestrel directly, the second is going via a reverse proxy. Kestrel is great for serving dynamic content from ASP.NET Core. However, the web serving capabilities aren’t as feature rich as servers such as IIS, Apache, or NGINX. A reverse proxy server can offload work such as serving static content, caching requests, compressing requests, and SSL termination from the HTTP server. A reverse proxy server may reside on a dedicated machine or may be deployed alongside an HTTP server.

Given the above considerations reverse proxy is usually the way to go.

Now the way this is traditionally achieved with NGINX, and pretty much all the examples you’ll see around, is to run Kestrel on a given port and then configure a reverse proxy to it as part of the server configuration. What you usually end up with is something like below:

server {
    listen 80;
    location / {
        proxy_pass http://localhost:5000;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection keep-alive;
        proxy_set_header Host $http_host;
    }
}

You go to http://www.some-site.com/ and it internally proxies through to http://localhost:5000/. It works pretty well.

However this has some overhead as you’re essentially doing double the amount of HTTP requests, each request to NGINX has to then request to Kestrel and then marshal the requests and responses.

I did some reading around and read that NGINX supports reverse proxy through Unix sockets. What’s a Unix socket I hear you ask? A Unix socket is used for IPC on Unix like systems. I guess the nearest analogy on Windows would be a named pipe. By using a Unix socket you dispense with the overhead of the standard HTTP pipeline. A downside is it’s limited to the immediate system, but since both NGINX and Kestrel are in my case this wasn’t an issue.

After this revelation I checked Kestrel supported it. I’d assumed it didn’t but was pleasantly surprised that it did (as of .NET Core 2 at least). Awesome!

To configure NGINX to reverse proxy to a Unix socket it’s simply a matter of passing the name of the socket. What you end up with is something like this:

server {
    listen 80;
    location / {
        proxy_pass http://unix:/tmp/some-site.sock:/;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection keep-alive;
        proxy_set_header Host $http_host;
    }
}

The name of the socket is /tmp/some-site.sock but we pass it to NGINX as http://unix:/tmp/some-site.sock:/ as that’s the format it expects.

Once I’d configured NGINX I then had to tell Kestrel to bind to the Unix socket. To do this I needed to pass in some options to the UseKestrel() call in the WebHostBuilder, something like this:

.UseKestrel(options =>
{
    options.ListenUnixSocket("/tmp/some-site.sock");
})

A quick restart of the service on Linux and voila, I was now piping through the socket instead of a HTTP connection.

Remember however, I mentioned I develop on Windows and deploy to Linux. Running Kestrel on Windows with the above code will cause an exception. Unix sockets don’t exist on Windows so it doesn’t like it. I found it surprising as other calls like UseIISIntegration() are ignored if IIS isn’t available but they seemed to have overlooked this.

To that end I now needed to check what platform I was on before attempting the bind. If I was on Windows I’d also still want to bind and provide the original HTTP endpoint so I could still perform debugging and testing.

Finally I came up with this:

.UseKestrel(options =>
{
    options.Listen(IPAddress.Any, 5000);

    var isUnix = RuntimeInformation.IsOSPlatform(OSPlatform.Linux);

    if (isUnix)
        options.ListenUnixSocket("/tmp/some-site.sock");
})

What this does is always bind to 0.0.0.0:5000 for HTTP requests but also, if on Unix, bind to the Unix socket /tmp/some-site.sock.

I’ll leave it here for now. Hopefully you’ve found this post interesting and informative. Give it a try! At some point I’ll have to test and provide metrics of both HTTP and Unix socket methods so you can see the performance differences.

4 Comments

  1. Thiago fernandes · 18th April 2018 Reply

    great post!

    I tryed a lot without sucess. Can you help me?

    In my program.cs I have the following code

    var host = new WebHostBuilder()
    .UseKestrel(opt =>
    {
    if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
    {
    opt.ListenUnixSocket(“/tmp/myservice.sock”);
    }
    })
    .Configure(app =>
    {
    app.Map(“health”, Controllers.ProxyController.Handler);
    }).Build();

    And, in Nginx:

    location /health {
    proxy_pass http://unix://tmp/myservice.sock:/;
    }

    But, when I running I get a 502 response (running the same service using http works).

    Do I need to do something else?

    • lloyd · 19th April 2018 Reply

      Shouldn’t this:

      proxy_pass http://unix://tmp/myservice.sock:/;

      Be:

      proxy_pass http://unix:/tmp/myservice.sock:/;

      • Thiago fernandes · 19th April 2018 Reply

        Thanks for the reply.

        The “//” was my mistake when I copied to here… in fact it has only 1 “/” at all and I get a 502 response.

        I tryed on Ubuntu, what distribution are you using? You had to add special rights to the file? There is any other configuration to do?

        Best,
        Thiago

        • Thiago fernandes · 19th April 2018 Reply

          Aspnetcore will create api.socket when its running but Nginx must have permission to write

          sudo chown www-data:www-data /tmp/api.sock

          I just missed that =)

Leave a Reply