Deploying Windows Apps with Kubernetes

Windows and Kubernetes

Windows has now reached production level support for Windows nodes as of Kubernetes v1.14. Due to this we can support various types of Windows workloads in a container and consequently to run in Kubernetes cluster. During the course of this presentation, we’ll go over some of the key considerations such as handling secrets, required node selectors, and other common differences you may encounter in clusters with Windows & Linux nodes.

Once the fundamentals are in place we can then move on to setting up the full developer workflow where we will go over how to create a new app using both .NET Framework (WCF, Console, MVC) and .NET Core using the various tools at our disposal.

This includes a discussion of the following tools:

Roadmap

There has been significant work on getting Kubernetes to support Windows applications and much of the heavy lifting has been done by Microsoft. This is why Azure is the only CSP at the moment to support Windows workloads in Kubernetes.

2016-2017 2017-2018 2018-2019
Containers in Windows Server (preview) Parity with core k8s features (configmap, pvc, dns resolution) Windows Server 2019 / 1809
SIG-Windows formed (Apprenda, TicketMaster, Docker, Huawei, etc) Networking (CNI) and storage improved in k8s 1.6 - 1.10 Product preview coming (AKS, Docker, RedHat)
Kubernetes 1.5 alpha using Windows Server 2016 Windows Server 1709 / 1803 (number of containers + patch) Testing, docs for v1.13+ and stable for v1.14

Note: Windows Server 1709 / 1803 currently need patches to address issues with larger numbers of containers

Building Windows Containers

Kernel / User compatibility

NuGet

The NuGet.config file should have the following added to it if leveraging Artifactory:

<add key="ArtifactoryNuGetV3" value="https://<domain>/artifactory/api/nuget/v3/nuget" protocolVersion="3" />

Then to authenticate against Artifactory with the NuGet Encrypted API key, run the following command:

nuget setapikey <USERNAME>:<PASSWORD> -Source ArtifactoryNuGetV3

To configure the NuGet Visual Studio Extension to use Artifactory, you need to add this repository as another Package Source under Nuget Package Manager.

Tools > NuGet Package Manager > Settings > Package Sources

  1. Go to the “Package Manager Settings”
  2. Add a custom name for the package source
  3. Follow the steps mentioned above

*.csproj

Depending on the project type and architecture you might need to modify the *.csproj file.

Configuration

Where possible you should never bake your configuration settings inside your docker container. A much better approach is to supply the transformation at the time of container startup.

One of the biggest benefits of using containers is the ability to wrap your application and its dependencies into an immutable image. This image is built once and deployed to different environments as it progresses through a continuous delivery pipeline.

As the application is deployed, we can then pass variables or mount configuration files into the container to configure it for the given environment. This ensures that we deploy the exact same application to production as the one tested in other environments. This approach also helps keep secrets out of container images.

Dockerfile

bin
obj
packages
Dockerfile
Jenkinsfile

There are a variety of supported Windows environments:

For each of the Windows environments you have the supported targets:

The following is an example of a full Dockerfile for working with WCF.

WCF

# escape=`

# WCF Server 1803
FROM <domain>:443/docker-remote/microsoft/dotnet-framework:4.7.2-sdk AS build
WORKDIR /app

# Build and package dotnet app
COPY . .
RUN nuget restore
RUN msbuild /p:Configuration=Release /p:DeployOnBuild=true /p:PublishProfile="FolderProfile.pubxml"

# Multi stage build
FROM <domain>:443/docker-remote/microsoft/wcf:4.7.2-windowsservercore-1803 as runtime

RUN MKDIR AuditLog
WORKDIR AuditLog
COPY --from=build /app/bin/Release/Publish ./

# Configuration of the new site in IIS
RUN powershell -NoProfile -Command `
    Import-module IISAdministration; `
    New-IISSite -Name "AuditLog" -PhysicalPath C:\Scripts\AuditLog -BindingInformation "*"

EXPOSE 83

Building Hybrid Clusters

There are currently two ways to leverage Windows workloads in Kubernetes on Azure. For the purposes of this presentation we will only be talking about how to use AKS Engine.

  1. Leverage AKS-Engine
  2. Manually add a Windows virtual machine to the cluster

Note: Managed AKS with Windows support should be general availability by the end of March 2019.

AKS Engine Cluster Definition

The following snippet is the initial cluster definition we pass to AKS Engine in order for it to generate the complete ARM deployments along with the configuration that Kubernetes expects.

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
      "orchestratorRelease": "1.12",
      "kubernetesConfig": {
        "useManagedIdentity": true,
        "userAssignedID": "k8s-prod-msi",
        "enableDataEncryptionAtRest": true,
        "enableEncryptionWithExternalKms": true,
        "enableRbac": true,
        "privateCluster": {
          "enabled": true
        },
        "networkPolicy": "azure",
        "enableRbac": true,
        "kubeletConfig": {
          "--allow-privileged": "true",
          "--anonymous-auth": "false",
          "--max-pods": "60",
          "--network-plugin": "cni"
        },
        "gcHighThreshold":85,
        "gcLowThreshold": 80,
        "apiServerConfig": {
          "--allow-privileged": "true",
          "--anonymous-auth": "false",
        },
        "addons": [
          {
            "name": "tiller",
            "enabled": true
          },
          {
            "name": "kubernetes-dashboard",
            "enabled": true
          },
          {
            "name": "container-monitoring",
            "enabled": true
          },
          {
            "name": "rescheduler",
            "enabled" : true
          }
        ]
      }
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "k8s-aks-production",
      "vmSize": "Standard_D4s_v3",
      "OSDiskSizeGB": 200,
      "vnetSubnetId": "/subscriptions/<subscription_id>/resourceGroups/<network_rg>/providers/Microsoft.Network/virtualNetworks/<vnet>/subnets/Container-Rz-Master",
      "firstConsecutiveStaticIP": "XXX.XX.X.X",
      "vnetCidr": "XXX.XX.X.X/24"
    },
    "agentPoolProfiles": [
      {
        "name": "linuxpool1",
        "count": 2,
        "customNodeLabels": {
          "os": "linux"
        },
        "vmSize": "Standard_D8s_v3",
        "OSDiskSizeGB": 200,
        "storageProfile" : "ManagedDisks",
        "availabilityProfile": "AvailabilitySet",
        "vnetSubnetId": "/subscriptions/<subscription_id>/resourceGroups/<network_rg>/providers/Microsoft.Network/virtualNetworks/<vnet>/subnets/<subnet>"
      },
      {
        "name": "windowspool1",
        "count": 1,
        "customNodeLabels": {
          "os": "windows"
        },
        "osType": "Windows",
        "vmSize": "Standard_D8s_v3",
        "OSDiskSizeGB": 200,
        "storageProfile" : "ManagedDisks",
        "availabilityProfile": "AvailabilitySet",
        "vnetSubnetId": "/subscriptions/<subscription_id>/resourceGroups/<network_rg>/providers/Microsoft.Network/virtualNetworks/<vnet>/subnets/<subnet>"
      }
    ],
    "linuxProfile": {
      "adminUsername": "azureuser",
      "ssh": {
        "publicKeys": [
          {
            "keyData": "ssh-rsa ..."
          }
        ]
      }
    },
    "windowsProfile": {
      "adminUsername": "azureuser",
      "adminPassword": "<pass>"
    },
    "servicePrincipalProfile": {
      "clientId": "<client_id>",
      "secret": "<secret>",
      "objectId": "<object_id>"
    }
  }
}

Note: Remember to make sure that the Windows version you launch with is the same version you want to build and run your containers in.

Orchestration

With our Windows Containers built and a Hybrid Kubernetes cluster instantiated we can now discuss how they can be successfully orchestrated in a distributed fashion.

Considerations

Release Automation

DevOps Pipeline

Jenkins

We have built our own containerized Jenkins JNLP agent against Windows 1803 and using the dotnet-framework base image.

Additionally leveraging chocolatey we have added a few other commonly required libraries.

Repository: windows-jnlp-agent-windows

Jenkinsfile

The following is largely a fully complete Jenkinsfile tailored to work with Windows nodes inside a Kubernetes cluster.

Simply replace the values below with your own configuration.

Note: We makes use of the Kubernetes plugin where you can specify the specific cluster you wish to run CI tests against.

def label = "jnlp-win-${UUID.randomUUID().toString()}"

podTemplate(
  cloud: '<k8s-cluster>',
  serviceAccount: 'jenkins-jenkins
  label: label,
  yaml: """
apiVersion: v1
kind: Pod
metadata:
  labels:
    ci: windows-vs2017
spec:
  containers:
    - name: jnlp
      image: <docker-registry>/jenkins/jenkins-ext:jnlp-windows
      env:
        - name: JENKINS_URL
          value: 'https://<domain>'
        - name: DOCKER_HOST
          value: 'tcp://<ipaddress>:2375'
      tty: true
      working: 'C:\\workspace\\'
  tolerations:
    - effect: NoSchedule
      key: os
      operator: Equal
      value: windows
  nodeSelector:
    agentpool: ciwinpool1
"""
) {
    node (label) {
      // Update the gitlab status to pending
      // https://jenkins.io/doc/pipeline/steps/gitlab-plugin
      updateGitlabCommitStatus name: 'build', state: 'pending'

      bat 'powershell.exe mkdir C:\\repo'
        dir('C:\\repo\\') {
            checkout scm

            withCredentials([usernamePassword(credentialsId: 'docker-pull', passwordVariable: 'pass', usernameVariable: 'user')]) {
                bat "docker login -u ${user} -p ${pass} <docker-registry>"
                bat "docker build . -t <docker-registry>/<repo-name> --no-cache"
                bat "docker push <docker-registry>/<repo-name>"
            }
        }
      bat 'powershell.exe remove-item C:\\repo -recurse -force'

      // Update the gitlab status to success
      updateGitlabCommitStatus name: 'build', state: 'success'
    }
}

Developer Perspective

Anti-Patterns

Right now it is unavoidable that the developer needs to learn a bit about Docker / Kubernetes in order to successfully deploy a Windows application on Kubernetes.

However this can technically be classified as a anti-pattern as we should strive for the developer workflow to remain consistent and for the orchestration to be seamless without any developer intervention. This is one of the main goals of the KNative initiative which brings serverless functionality to Kubernetes.

Debugging

There a quite a few ways to debug Windows Containers and / or your Windows Node on Kubernetes. Additionally the problem scope differs based on the level of the stack where you are experience an issue.

Always important to remember to start small and try to isolate the issue.

Here is some useful information that might help you further troubleshoot.

Debugging Windows Container with Docker Compose

Even though docker swarm is on the way out and being wholly replaced by Kubernetes, docker-compose itself on a single virtual machine can prove very useful in order to prove things work before deploying your applications into a distributed cluster.

This is very useful for debugging as sometimes don’t know where specific issues are in the cluster.

Another important fact is that if everything works in a local docker-compose environment, it is a safe bet your containers should also work in a Kubernetes cluster with certain caveats applied.

Debugging Windows Nodes

Help

There are quite a few places where you can get help with issues related to Windows and Kubernetes. We have had particular success in the Slack channel and GitHub issue queue when their was spefic problems related to our Windows node.

Note: Please remember to be concise with your issue and always reference your Kubernetes version, Windows Kernel version, and specific network configuration.