888-685-3101, ext. 2

Kubernetes Best Practices, Part 1

by | Feb 14, 2023 | Blog, Cloud, DevOps, Perspectives

Kubernetes Best Practices Part 1

In this article, we will discuss five Kubernetes best practices for using Kubernetes to host your enterprise workloads. This article focuses primarily on best practices for resource definitions that you deploy into a Kubernetes cluster; a subsequent article will focus on best practices around managing your Kubernetes clusters.

Kubernetes Best Practices (Part 1): Table of Contents

Kubernetes Best Practice #1: Plan Resource Requirements Accurately

Kubernetes provides two simple but powerful ways to manage resource requirements for a containerized application; resource requests and resource limits. Resource requests are used to specify the minimum amount of resources a container needs to run effectively, while resource limits control the maximum amount of those same resources that should be assigned to a container.

The Kubernetes scheduler uses the sum total of a pod’s resource requests to determine if there are enough resources available on a node to run the pod. If there are not enough resources available, the pod will not be scheduled to that node; if no nodes have adequate resources to host the pod, it will fail with a FailedScheduling error.

Resource limits are valuable as a tool to manage scarce resources in a kubernetes cluster; they are used to restrict allocation of resources to containers in a pod. Resource limits set a cap for resources that the container runtime will allocate to a container; CPU limits will generally pause execution for a container if they are reached, while memory limits will invoke the Linux kernel’s out-of-memory subsystem; this will likely result in the container being restarted (if possible).

To set resource requests and limits, you can specify the resources in the pod specification YAML file. The following example sets both requests and limits for CPU and memory for a container in a pod:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
	image: my-image
	resources:
  	  requests:
    	    memory: "128Mi"
    	    cpu: "500m"
        limits: 
          memory: "512Mi"
          cpu: "2000m"

In this example, the pod is requesting 128 MiB of memory and 500 milli-CPUs for its single container. The scheduler will only schedule the pod to a node that has at least 128 MiB of memory and 500 milli-CPUs available. We are also setting limits on both memory and CPU; 512 MiB and 2000 milli-CPUs respectively. If the container attempts to use more than these limits, it will be paused and may be terminated and restarted.

One final note: you should specify resource requests for each container in a pod; if a node does not have enough resources to accommodate all containers in that pod, the pod will not be scheduled for that node. It is a best practice to use resource limits as much as possible to keep your applications and cluster running smoothly.

Kubernetes Best Practice #2: Use Namespaces for Resource and Access Management

Namespaces in Kubernetes provide a way to divide a cluster into multiple virtual clusters. Each namespace provides a unique context for resources such as pods, services, and config maps, allowing for better organization and resource management in a cluster.

Namespaces provide a number of benefits for your workloads, including the following:

  1. Resource isolation: Namespaces provide a way to isolate resources, ensuring that resources in one namespace do not interfere with resources in another namespace.
  2. Access control: Namespaces provide a way to control access to resources in a cluster, enabling you to grant or deny access to specific resources based on role or user.
  3. Environment separation: Namespaces can be used to separate resources by environment, such as dev, test, and production, allowing for better organization and management of resources.
  4. Cost optimization: Namespaces can be used to manage resource utilization in a cluster, reducing costs and improving the overall efficiency of the cluster.
  5. Resource management: Namespaces allow you to group resources by team, project, or function, making it easier to manage and monitor resources in a cluster.

Some common patterns for namespace usage include per-environment namespaces (i.e. development, qa, production), per-team namespaces (i.e. ), per-customer namespaces (for single-tenant applications and workloads). It is critical to decide early on how you expect to partition and organize your clusters and namespaces; one very common antipattern that enterprises fall into is to use one large namespace to host all of your applications; you will quickly find that your cluster will become unmanageable and almost impossible to work with as your Kubernetes adoption progresses.

Kubernetes Best Practice #3: Use ConfigMaps and Secrets for Configuration Management

This may seem obvious, but for organizations transitioning to Kubernetes, how best to use ConfigMaps and Secrets can be fairly confusing. Both of these resources are used to manage configuration data; the primary difference (as implied by the name) is that Secrets are mainly used to protect sensitive information (passwords, access tokens, etc.).

Some basic details about ConfigMaps:

  • ConfigMaps are used to store non-sensitive configuration data such as application settings, environment variables, and other data that can change without affecting the security of the application.
  • To use ConfigMaps, you create a ConfigMap object in your Kubernetes cluster and then reference it from within your pod or deployment specification.

A ConfigMap definition would look similar to the following:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-config
data:
  KEY1: VALUE1
  KEY2: VALUE2

In your pod specification, you can reference the ConfigMap like this:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
	image: my-image
	env:
	- name: KEY1
  	valueFrom:
    	configMapKeyRef:
      	name: my-config
      	key: KEY1

The above pod specification will expose the my-config.KEY1 ConfigMap resource as an environment variable named KEY1. There are four ways to use a ConfigMap resource inside of a pod:

  1. Reference the ConfigMap in a container’s command and/or arguments.
  2. Configure the pod to expose the ConfigMap as an Environment variable for a container.
  3. Add a volume that references the ConfigMap in the pod definition, and mount keys from that ConfigMap as files in the volume.
  4. Write code to run inside the pod that uses the Kubernetes API to read the ConfigMap.

A few details about Secrets:

  • Secrets are used to store sensitive information such as API keys, passwords, and other data that should not be exposed in plain text.
  • Secrets are encrypted in the cluster, and only decrypted at runtime, ensuring that the sensitive information is secure.
  • To use Secrets, you create a Secret object in your cluster and then reference it from within your pod or deployment specification.

An example Secret would look like this:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
type: Opaque
data:
  KEY1: BASE64_ENCODED_VALUE1
  KEY2: BASE64_ENCODED_VALUE2

In your pod specification, you can reference the Secret like this:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
	image: my-image
	env:
	- name: KEY1
  	valueFrom:
    	secretKeyRef:
      	name: my-secret
      	key: KEY1

As with ConfigMaps, there are multiple ways to reference a Secret in a pod:

  1. Configure the pod to expose the Secret as an Environment variable for a container.
  2. Add a volume that references the Secret in the pod definition; the keys in that Secret will be created as files in the volume.
  3. A secret can be used by the kubelet when pulling images for a pod; this is primarily used to authenticate against private container registries.

Kubernetes Best Practice #4: Implement Pod Anti-Affinity for High Availability

Pod anti-affinity in Kubernetes is used to control the scheduling of pods within a cluster; it is a powerful Kubernetes feature geared towards controlling pod placement and ensuring high availability, redundancy, and performance in a cluster. It allows you to specify rules that determine which nodes a pod should or should not be scheduled to, based on the labels of other pods running on the same node.

To use pod anti-affinity, you need to specify a podAntiAffinity rule in the pod specification YAML file. The rule defines the conditions that must be met for a pod to be scheduled to a node.

Here is an example of how to specify a pod anti-affinity rule in a pod specification:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
	image: my-image
  affinity:
	podAntiAffinity:
  	requiredDuringSchedulingIgnoredDuringExecution:
  	- labelSelector:
      	matchExpressions:
      	- key: app
        	operator: In
        	values:
        	- my-app
    	topologyKey: "kubernetes.io/hostname"

In this example, the pod anti-affinity rule states that the pod should not be scheduled to a node if another pod with the label app: my-app is already running on that node. The topologyKey specifies the node label that is used to match the anti-affinity rule. In this case, the hostname of the node is used.

It’s important to note that the pod anti-affinity rules are not always guaranteed to be met, and the scheduler may choose to schedule a pod to a node even if it violates the anti-affinity rule. However, the scheduler will make its best effort to satisfy the anti-affinity rule and avoid scheduling multiple pods from the same set on the same node.

Kubernetes Best Practice #5: Consistently Use Liveness and Readiness Probes

Liveness and readiness probes are used in Kubernetes to monitor the health and readiness of containers in a pod. The purpose of these probes is to help the Kubernetes scheduler make decisions about the state of containers and to ensure that only healthy and ready containers are serving traffic.

Liveness probes determine whether a container is alive and healthy. If the liveness probe fails, Kubernetes restarts the container in an attempt to recover it. The number of restarts that Kubernetes will attempt is configurable and can be set in the pod definition. This helps to ensure that containers are running correctly and prevent them from becoming stuck in a crashed or unresponsive state. A liveness probe should fail in situations where the container is unrecoverable and should be terminated; a good example would be to check for a valid 20x or 30x HTTP response from a process that handles HTTP requests.

Readiness probes determine whether a container is ready to receive traffic; if a container’s readiness probe fails, the container is not considered to be fully started up and thus traffic will not be routed to it through Kubernetes services. This helps to prevent traffic from being sent to containers that are not fully initialized or are still in a startup phase. A readiness probe should invoke application logic that checks all parts of the monitored application, including connection pools to external services, internal resource pools, and any other integration points required by the containerized application.

Both liveness and readiness probes are critical for hosting workloads in Kubernetes; by using both effectively, you will ensure that all containers in your cluster are live and operational, and that the Kubernetes scheduler is properly equipped to assist when any container experiences an outage.

Below, we have a simple example of using liveness and readiness probes for a basic nginx-based deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: app-nginx
  name: app-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app-nginx
  template:
    metadata:
      labels:
        app: app-nginx
    spec:
      containers:
      - image: nginx:latest
        name: app-nginx
        ports:
        - containerPort: 8443
          protocol: TCP
        readinessProbe:
          httpGet:
            scheme: HTTPS
            path: /index.html
            port: 8443
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          tcpSocket:
            port: 8443
          initialDelaySeconds: 1
          periodSeconds: 5
          timeoutSeconds: 1

The above example is using a TCP socket probe to determine liveness and an HTTPS GET probe to determine readiness. This is one possible pattern to follow when you have a situation where there is expected latency between container creation and application readiness; the pod will be marked live once the NGiNX process is listening on port 8443, but it won’t have any traffic routed to it until it returns a successful HTTP response on /index.html.

Kubernetes Best Practices (Part 1) Summary

These best practices will help you utilize Kubernetes to its fullest and take advantage of the capabilities that Kubernetes provides to allow you to run highly available, scalable, performant workloads. This series will be continued, with the next article covering five best practices around managing and operating your Kubernetes clusters.

If you are looking for the right Kubernetes specialists to help set up your Kubernetes initiative for success, or are looking for help securing Kubernetes using Aqua Security, then XTIVIA can help.

Please get in touch with us to learn how we can help with your Kubernetes implementation as your Kubernetes partner.