Deep dive into Kubernetes Deployments

Ben Osborne

Dec 7, 2020·5 min read

Introduction

Hey everyone! 👋 In my first story I guided you through building and deploying a Laravel Lumen application in a micro-services architecture using Kubernetes, Skaffold and Minikube for local testing.

Today we are going to take a deeper dive into a Kubernetes Deployment manifest and explore the following topics:

• Setting resource requests/limits
• Setting pod affinity rules.
• Creating and setting priority classes.

This guide assumes you are familiar with the basics of Kubernetes and Minikube, there will be a chance in each section to participate and run commands against your local cluster, however, you are not required.

Setting Resource Requests / Limits

Let’s kick off with setting request and limits for your containers CPU and Memory, but first we need to understand what request and limit is and how Kubernetes use these values.

Requests

A resource request for CPU or Memory defines the minimum amount of resource for your container to run, for example if your application requires a minimum amount of 256MB to run then this is your memory resource request.

Kubernetes will guarantee that your container has the requested minimum resource for your container to run.

If we do not set resource requests then Kubernetes does not know how much resource your container wants to consume.

When the scheduler puts a pod onto a node it will read the resource requests first and decide which node best fits its requirements based on aggregating resource requests from all containers in the pod.

Limits

As the name suggests, resource limits is the maximum that your container can use, for example you might set your container to request 256MB of memory and a memory limit of 1GB.

Kubernetes will automatically stop the container if it exceeds the resource limits.

Example

Let’ create a simple hello world deployment, and set our resource requests and limits. We will use the tutum/hello-world image in this example.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world-app
  template:
    metadata:
      labels:
        app: hello-world-app
    spec:
      containers:
        - name: hello-world
          image: tutum/hello-world:latest
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: "1"
              memory: "1G"
            requests:
              cpu: "0.5"
              memory: "256M"
          ports:
            - name: http
              containerPort: 80

That’s it, it is important to note that you should always set resource requests and limits, failing to do so could result in a container using up all of the resource on the node causing other deployments/pods to be re-scheduled.

Setting pod affinity rules

Pod affinity rules can be used to constrain a pod to a particular node, you may never need to use pod affinity rules, the scheduler does a good job of determining which node is the best fit for your pod.

However, you may want to spread your pods of the same deployment to one pod per node and this is what we will cover in this topic.

Let’s take a closer look at the deployment specification below:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world-app
  template:
    metadata:
      labels:
        app: hello-world-app
    spec:
      containers:
        - name: hello-world
          image: tutum/hello-world:latest
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: "1"
              memory: "1G"
            requests:
              cpu: "0.5"
              memory: "256M"
          ports:
            - name: http
              containerPort: 80
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - hello-world-app
              topologyKey: "kubernetes.io/hostname"

We have added the podAntiAffinity field with a label selector to select our hello world pod.

This will make sure no other pods that match the label will be placed onto the same node using the node host name.

If you have the existing pod running from previous topic and only one node then you will notice our anti affinity rule working as expected.

Run the following command, you will see the the new pod is stuck in Pending state, this is because we already have a pod running on our node.

> kubectl get pods

Output:

NAME                           READY   STATUS    RESTARTS   AGE
hello-world-58f885df9-99ts2    1/1     Running   0          17m
hello-world-84fc85967b-gx28g   0/1     Pending   0          9s

Let’s find out why the pod is stuck in pending.

> kubectl get event --field-selector involvedObject.name=hello-world-84fc85967b-gx28g

Output:

LAST SEEN   TYPE      REASON             OBJECT                             MESSAGE
77s         Warning   FailedScheduling   pod/hello-world-84fc85967b-gx28g   0/1 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 1 node(s) didn't match pod anti-affinity rules.

Let’s delete the original running pods replica set to allow our new pod to be scheduled.

> kubectl get replicasets.apps

Output:

NAME                     DESIRED   CURRENT   READY   AGE
hello-world-58f885df9    1         1         1       23m
hello-world-84fc85967b   1         1         0       6m16s

The replica set which has ready = 1 is the one we want to delete.

> kubectl delete replicasets.apps hello-world-58f885df9

Great! after a couple of seconds our first version of our pod will be terminated and our new pod will change to running state.

Creating and setting priority classes

You have most likely already come across a situation in your tech stack where some pods are just less important than others, this is where pod priority classes come in.

Lets assume we have a hello-world pod that is of high priority and a grafana-dashboard pod that is of low priority, we first need to create these 2 priority classes in Kubernetes, let’s go ahead and do that now.

Apply the priority-class.yaml file below to your local running cluster:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high
value: 6000000
preemptionPolicy: Never
globalDefault: false
description: "This priority class will not cause other pods to be preempted."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: low
value: 300000
preemptionPolicy: Never
globalDefault: false
description: "This priority class will not cause other pods to be preempted."

We can view the priority classes we have created by running the following command:

> kubectl get priorityclasses.scheduling.k8s.io

Now we have our priority classes created let’s assign them to our pods, using the base deployment YAML (without pod anti affinity) we can add the following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world-app
  template:
    metadata:
      labels:
        app: hello-world-app
    spec:
      containers:
        - name: hello-world
          image: tutum/hello-world:latest
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              cpu: "1"
              memory: "1G"
            requests:
              cpu: "0.5"
              memory: "256M"
          ports:
            - name: http
              containerPort: 80
      priorityClassName: high

That’s it

Thanks for reading, I hope you learned something new today, if you have any questions please ask 😃

The GitHub repository can be found here with all the examples.