Chapter 5

Let’s dive into our final chapter for this series. This topic is one of the most powerful in Kubernetes and is essential for running complex applications. So far, we have a clear distinction:

Deployments are for stateless applications (a web server where any Pod is identical to any other).
Jobs are for batch tasks (run-to-completion).

But what about the most complex category: stateful applications like databases (e.g., MySQL, PostgreSQL), message queues (e.g., Kafka, RabbitMQ), or any clustered software where each member has a unique identity and state?

Using a Deployment for a database is problematic. Pods get random hostnames, and when one is replaced, the new Pod has a completely new identity and a different IP address. It can’t easily rejoin the database cluster because the other members don’t recognize it. For these applications, we need stability and predictability.

Managing Stateful Applications with StatefulSets

A StatefulSet is a Kubernetes workload object, like a Deployment, but it’s specifically designed to manage stateful applications. It provides strong guarantees about the identity and ordering of its Pods, which Deployments do not.

Here are the key features that make a StatefulSet unique:

Stable, Unique Network Identifiers: Each Pod in a StatefulSet gets a predictable, persistent hostname. The pattern is always (<statefulset-name>)-<ordinal-index>. For example, if your StatefulSet is named web, its Pods will be named web-0, web-1, web-2, and so on. This name sticks to the Pod, even if it’s restarted or rescheduled to a different Node.
Stable, Persistent Storage: Each Pod gets its own unique PersistentVolumeClaim based on a template. The PVC for web-0 is not the same as the PVC for web-1. When a Pod is rescheduled, it always re-attaches to its original storage volume, ensuring it retains its unique state.
Ordered, Graceful Deployment and Scaling:
- Deployment: When you scale up a StatefulSet from 0 to 3 replicas, it happens in strict order. web-0 will be created and must become “Ready” before web-1 is even created. web-1 must be ready before web-2 is created.
- Deletion: When you scale down from 3 to 0, the deletion happens in the reverse order: web-2 is terminated first, then web-1, then web-0. This ordered orchestration is critical for clustered applications that need to rebalance data gracefully.

Headless Services: The Prerequisite

A StatefulSet requires a Headless Service to control the network domain for its Pods. A normal Service gets a single, stable IP (ClusterIP) and load-balances traffic among all its backing Pods.

A Headless Service is created by setting clusterIP: None in the Service definition. It does not get a ClusterIP and does not perform load balancing. Instead, it creates DNS records that point directly to the IP addresses of each individual Pod in the StatefulSet. This is what allows you to connect directly to web-0.my-service or web-1.my-service.

Create the Headless Service This service will define the network identity for our Pods. Create a file named headless-service.yaml:

apiVersion: v1
kind: Service
metadata:
name: my-stateful-app-svc
spec:
# Setting clusterIP to None makes this a "headless" service.
clusterIP: None
selector:
    app: my-stateful-app
ports:
    - protocol: TCP
    port: 80
    targetPort: 80

Apply it:

kubectl apply -f headless-service.yaml

Create the StatefulSet This StatefulSet will run Nginx, but we’ll use a small command to make each Pod display its own unique hostname.

Create a file named statefulset.yaml:

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: my-stateful-app
spec:
# The selector must match the labels of the pod template.
selector:
    matchLabels:
    app: my-stateful-app
# This must match the name of our Headless Service.
serviceName: "my-stateful-app-svc"
replicas: 3
template:
    metadata:
    labels:
        app: my-stateful-app
    spec:
    containers:
    - name: nginx
        image: nginx:1.21.6
        # This command replaces the default nginx page with one that shows the pod's hostname
        command: ["/bin/sh", "-c", "echo \"My hostname is $(hostname)\" > /usr/share/nginx/html/index.html && nginx -g 'daemon off;'"]
        ports:
        - containerPort: 80

Apply it and immediately watch the Pods being created:

kubectl apply -f statefulset.yaml
kubectl get pods -w

You will see my-stateful-app-0 being created first. Only after it becomes Running and Ready will my-stateful-app-1 start creating. The process repeats for my-stateful-app-2. This is the ordered deployment in action.

Test the Stable Network Identities Let’s verify that each Pod has a unique and resolvable DNS name. We’ll launch a temporary pod to act as our client.
Terminal window
```
kubectl run client-pod --image=busybox:1.36 --rm -it -- /bin/sh
```
Once you are inside the client pod’s shell, use wget to query each of the stateful pods by their unique DNS name. The DNS name is <pod-name>.<service-name>:
Terminal window
```
# Inside the client-pod shell

# Query the first pod
wget -q -O - my-stateful-app-0.my-stateful-app-svc
# You will see the output: My hostname is my-stateful-app-0

# Query the second pod
wget -q -O - my-stateful-app-1.my-stateful-app-svc
# You will see the output: My hostname is my-stateful-app-1

exit
```
This proves that each Pod is individually addressable via the Headless Service.
Observe Ordered Scaling Let’s scale the StatefulSet down from 3 replicas to 1.
Terminal window
```
kubectl scale statefulset my-stateful-app --replicas=1
```
Watch the pods with kubectl get pods -w. You will see my-stateful-app-2 terminate first. Once it’s gone, my-stateful-app-1 will terminate, leaving only my-stateful-app-0 running. This reverse-order termination is critical for gracefully shutting down clustered applications.

Clean Up

kubectl delete statefulset my-stateful-app
kubectl delete service my-stateful-app-svc

Key Takeaways for Chapter 5:

Use a Deployment for stateless applications where identity doesn’t matter.
Use a StatefulSet for stateful applications like databases, message queues, and other clustered services that require stable identity and ordered operations.
StatefulSets provide stable hostnames, stable storage, and ordered deployment/scaling.
A StatefulSet always needs a corresponding Headless Service to provide the network DNS entries for its Pods.

This concludes our advanced-core concepts series! You now have a much more robust and production-oriented understanding of how to manage a wide variety of application types on Kubernetes.