Real cluster, recorded, then deleted

Kubernetes on GKE Autopilot, broken on purpose

I set up a GKE Autopilot cluster in Singapore (asia-southeast1), ran a small app on it, and put it through a deploy, a rolling update, autoscaling and self-healing. Then I broke it on purpose in two ways, and a third problem turned up on its own. I tracked down each one with kubectl. Once it was all recorded I deleted the cluster, so this page costs nothing to keep online.

01

What is Kubernetes

Picture a restaurant kitchen. The manager doesn't stand over every cook. They say "always have three plates of this ready", and if a cook goes home sick, someone else steps in. When the lunch rush hits, more cooks come in.

Kubernetes does that job for software. You write down what should be running (the desired state), and it keeps checking and fixing things until reality matches. It restarts apps that crash, moves them off broken machines, and adds or removes copies as needed.

02

The building blocks

Words you'll see on the rest of this page.

Container

An app packed together with everything it needs to run, so it works the same on a laptop as it does in the cloud.

Pod

The smallest thing Kubernetes runs. Usually one container, sometimes a few that need to sit together. Pods are throwaway: when one breaks, Kubernetes replaces it instead of fixing it.

Node

A machine that runs pods. With Autopilot, Google looks after these for you.

Cluster

All the nodes together, plus the control plane that manages them.

Control plane

The part that makes the decisions: which pod goes on which node, and what to do when something drifts from the plan. In the kitchen picture, this is the manager.

Deployment

Where you say which app to run and how many copies to keep. It also handles rolling updates, swapping old pods for new ones a few at a time.

Service

One fixed address for a group of pods. Pods come and go, but traffic always knows where to find them.

Ingress

Rules for traffic coming in from the internet: which web address or path goes to which Service.

NetworkPolicy

Decides which pods are allowed to talk to which, both coming in (ingress) and going out (egress). There's no "Egress" object to match Ingress; this is where outgoing traffic gets controlled. Without one, any pod can reach anything.

HorizontalPodAutoscaler (HPA)

Watches the load, such as CPU use, and adds or removes pods to match. The extra cooks at lunchtime.

Namespace

A way to split one cluster into separate areas, so different apps or teams don't trip over each other.

Requests & limits

Requests are how much CPU and memory a pod asks for, which decides where it can fit. Limits are the most it's allowed to use. Use too much memory and the pod gets killed (OOMKilled). Ask for more than there's room for and it waits (Pending).

kubectl

The command-line tool for talking to a cluster. I used it to apply changes, check status and read logs.

03

Kubernetes vs GKE

Kubernetes is the open-source software. GKE (Google Kubernetes Engine) is Google running it for you. Google keeps the control plane up and patched, and the cluster comes connected to the rest of Google Cloud: load balancers, IAM permissions, logging and monitoring, and Artifact Registry for container images.

AWS has EKS and Azure has AKS, which do the same job. You can also run Kubernetes yourself with tools like kubeadm, but then every upgrade and outage is on you.

04

GKE Standard vs Autopilot

GKE comes in two modes. The main difference is who looks after the machines underneath.

AspectStandardAutopilot
Who manages the nodes You, with help from Google Google
Billing model You pay for the machines, even when they sit idle You pay mostly for the CPU, memory and storage your pods ask for
Control You pick the machine types and node pools, and can SSH in Google picks the machines and scales them
Security defaults Up to you Locked down from the start: no SSH into nodes, privileged pods restricted
Resource requests Optional Google fills in defaults if you leave them out

Google's docs have the full details: Autopilot overview and GKE pricing.

05

Why not just Cloud Run?

This website runs on Cloud Run, not GKE, so it's worth saying why.

Cloud Run

You give it a container and it runs it. There's no cluster to look after, and it scales down to zero when nobody is visiting. For a site like this one, that's all I need.

GKE

Worth it when you need Kubernetes itself: lots of services working together, custom networking, background jobs or apps that keep data, and tools from the Kubernetes world like Helm.

So I ran GKE long enough to record the demo, then deleted the cluster. This site stays on Cloud Run.

06

What I built

One Autopilot cluster in Singapore (asia-southeast1). The app is Google's public sample image hello-app (version 1.0, later 2.0), run as a Deployment called hello with 3 replicas. Each pod asks for 250m CPU and 512Mi memory.

A Service of type LoadBalancer gets a public IP from a Google Cloud load balancer and passes the traffic on to the pods. I didn't use an Ingress. A HorizontalPodAutoscaler keeps the Deployment between 3 and 10 replicas, aiming for 50% CPU.

flowchart LR
    User["Browser"] --> LB["Google Cloud load balancer"]
    LB --> Svc["Service (LoadBalancer)"]
    subgraph Cluster["GKE Autopilot cluster (asia-southeast1)"]
        Svc --> Dep["Deployment"]
        Dep --> P1["Pods"]
        HPA["HorizontalPodAutoscaler"] -.-> Dep
    end
        

Why Autopilot

Two things I saw in the console. On the create-cluster page, the default Standard cluster showed an estimate of $327.93 a month if left running. I only looked and didn't create it. Then the dialog for switching to Autopilot said Google recommends Autopilot for most workloads. I went with Autopilot.

Google Cloud console create-cluster page for a default Standard cluster, showing an estimated monthly cost of $327.93
The default Standard cluster form: $327.93 a month estimated.
Switch from Standard to Autopilot dialog saying Google recommends Autopilot for most workloads
The switch dialog: Google recommends Autopilot for most workloads.

Autopilot bills for what the pods request. The Clusters page showed 0.75 vCPU and 1.5 GB while my 3 pods were running, which is exactly 3 × 250m CPU and 3 × 512Mi.

Kubernetes Engine Clusters page listing gke-demo in asia-southeast1 with 0.75 total vCPUs and 1.5 GB total memory
The Clusters page: 0.75 vCPU and 1.5 GB for 3 pods.
07

The demo, step by step

For each step: what I ran, a trimmed copy of the real output, and a screenshot from the console where I took one.

1. Deploy and load balancing

One command applied the Deployment and the Service. The pods waited in Pending for a short while before they started (the story is in section 08). Then the Service got a public IP. I sent six requests to it and all three pods answered, so the load balancer was spreading the traffic.

$ kubectl apply -f k8s/01-deployment.yaml -f k8s/02-service.yaml
deployment.apps/hello created
service/hello created

$ kubectl get svc hello
NAME    TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)        AGE
hello   LoadBalancer   34.118.238.145   35.247.x.x   80:32361/TCP   109s

$ for i in 1 2 3 4 5 6; do curl -s http://35.247.x.x; echo; done
(output trimmed to requests 3 to 5 of 6)
Hello, world!
Version: 1.0.0
Hostname: hello-56677d9698-5jgrc

Hello, world!
Version: 1.0.0
Hostname: hello-56677d9698-v774l

Hello, world!
Version: 1.0.0
Hostname: hello-56677d9698-48prd
Kubernetes Engine Workloads page showing the hello Deployment, status OK, 3/3 pods, node type Autopilot-managed
Workloads: hello, 3/3 pods, node type Autopilot-managed.
Gateways, Services and Ingress page showing the hello Service of type External load balancer with 3/3 pods
Services: hello is an external load balancer in front of 3/3 pods.

2. Rolling update, 1.0 to 2.0

I changed the image from hello-app:1.0 to 2.0 in the Deployment and applied the file again. Kubernetes replaced the pods one at a time. Afterwards every response said Version: 2.0.0.

$ kubectl apply -f k8s/01-deployment.yaml
deployment.apps/hello configured

$ kubectl rollout status deployment/hello --timeout=600s
...
Waiting for deployment "hello" rollout to finish: 1 old replicas are pending termination...
deployment "hello" successfully rolled out

$ for i in 1 2 3; do curl -s http://35.247.x.x; echo; done  (first response only)
Hello, world!
Version: 2.0.0
Hostname: hello-6996d777cf-sbq6c

3. Scaling with the HPA

The HPA aims for 50% CPU, between 3 and 10 replicas. To make traffic I used a busybox load generator Deployment and added more copies of it. With 4 generators, CPU levelled off around 34 to 37%. With 10 it hovered around 48 to 52%, which is within the HPA's tolerance, so nothing changed. With 25 it reached 55%, then 69%, and the HPA went from 3 to 6 replicas. CPU then settled at 46%. The nodes grew to 4. After I stopped the traffic and the default 5-minute stabilisation window passed, it scaled back from 6 to 3.

$ kubectl apply -f k8s/03-hpa.yaml
horizontalpodautoscaler.autoscaling/hello created

$ kubectl scale deployment/load-generator --replicas=25
deployment.apps/load-generator scaled

$ kubectl get hpa hello --watch   (excerpt)
hello   Deployment/hello   cpu: 55%/50%   3     10    3     8m5s
hello   Deployment/hello   cpu: 69%/50%   3     10    6     8m25s
...
hello   Deployment/hello   cpu: 46%/50%   3     10    6     9m26s

$ kubectl get hpa hello --watch   (after traffic stopped, excerpt)
hello   Deployment/hello   cpu: 0%/50%   3     10    6     15m
hello   Deployment/hello   cpu: 0%/50%   3     10    5     15m
hello   Deployment/hello   cpu: 0%/50%   3     10    3     16m
Deployment details for hello showing 3 updated, 3 ready, 3 available replicas
Before: 3 ready replicas.
Deployment details for hello showing 6 updated, 6 ready, 6 available replicas and CPU rising
After: 6 ready replicas, CPU climbing.

4. Self-healing

I deleted one pod on purpose. The Deployment noticed and started a replacement. In the next listing the new pod was already Running at 3 seconds old, and the count was back to 3.

$ kubectl delete pod hello-6996d777cf-4vtlv --wait=false
pod "hello-6996d777cf-4vtlv" deleted from default namespace

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
hello-6996d777cf-ngvvg   1/1     Running   0          3s
hello-6996d777cf-rmsfs   1/1     Running   0          22s
hello-6996d777cf-sbq6c   1/1     Running   0          32s
08

Breaking it on purpose

I caused two failures on purpose, CrashLoopBackOff and OOMKilled. The third, Pending, I didn't cause. It showed up right after the first deploy and Autopilot fixed it by itself. For each one I worked out what went wrong with kubectl describe pod and kubectl logs --previous. I left the two broken pods as they were until clean-up.

CrashLoopBackOff (on purpose)

payments-worker starts, finds no DATABASE_URL setting and exits with code 1. Kubernetes keeps restarting it, with longer waits each time. The status says it is looping, and the logs from the previous run say why.

$ kubectl get pods payments-worker report-builder
NAME              READY   STATUS             RESTARTS     AGE
payments-worker   0/1     CrashLoopBackOff   1 (9s ago)   12s

$ kubectl logs payments-worker --previous
Starting payments-worker v1.4.2
Connecting to database...
ERROR: DATABASE_URL is not set

$ kubectl describe pod payments-worker | sed -n '/^Events:/,$p'  (events trimmed to the last line)
  Warning  BackOff    9s (x2 over 10s)   kubelet                                spec.containers{worker}: Back-off restarting failed container worker in pod payments-worker_default(81a14308-31da-49c7-a8a5-e9fb21573a6c)
Pod details page for payments-worker with a red banner: Error with exit code 1
The console shows the same thing: Error with exit code 1.

OOMKilled (on purpose)

report-builder has a memory limit of 256Mi and tries to hold 600 MB. It gets killed. describe shows the reason and exit code 137.

$ kubectl get pods payments-worker report-builder
NAME              READY   STATUS             RESTARTS     AGE
report-builder    0/1     OOMKilled          0            11s

$ kubectl describe pod report-builder | sed -n '/State:/,/Restart Count/p'
    State:          Terminated
      Reason:       OOMKilled
      Exit Code:    137
      Started:      Wed, 14 Oct 2026 02:34:56 +0800
      Finished:     Wed, 14 Oct 2026 02:35:01 +0800
    Ready:          False
    Restart Count:  0
Pod details page for report-builder with a red banner: OOMKilled with exit code 137, and an out-of-memory warning
The console: OOMKilled with exit code 137, plus an out-of-memory warning.

Pending (not on purpose)

Right after kubectl apply, all 3 hello pods were Pending. The event says the scheduler had 1 node and it carried a taint the pods don't tolerate. Nothing was wrong with my YAML. The cluster had no room for the pods yet, so Autopilot added 2 new nodes and the pods became Running in about 1.5 minutes. I only had to wait.

$ kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
hello-56677d9698-24km5   0/1     Pending   0          5s    <none>   <none>   <none>           <none>
hello-56677d9698-5jgrc   0/1     Pending   0          5s    <none>   <none>   <none>           <none>
hello-56677d9698-v774l   0/1     Pending   0          5s    <none>   <none>   <none>           <none>

$ kubectl describe pod/hello-56677d9698-24km5 | sed -n '/^Events:/,$p'
Events:
  Type     Reason            Age   From                                   Message
  ----     ------            ----  ----                                   -------
  Warning  FailedScheduling  16s   gke.io/optimize-utilization-scheduler  0/1 nodes are available: 1 node(s) had untolerated taint(s). no new claims to deallocate, preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

$ kubectl get nodes
NAME                                      STATUS   ROLES    AGE     VERSION
gk3-gke-demo-default-pool-be5e45c5-hwqn   Ready    <none>   2m56s   v1.36.4-gke.1391002
gk3-gke-demo-pool-1-8fa8dc24-skfl         Ready    <none>   31s     v1.36.4-gke.1391002
gk3-gke-demo-pool-1-8ff46d88-5cjk         Ready    <none>   4s      v1.36.4-gke.1391002
Kubernetes Engine Workloads page: hello is OK with 6/6 pods, payments-worker shows Error with exit code 1, report-builder shows OOMKilled with exit code 137
Workloads page with both broken pods flagged. hello shows 6/6 pods because autoscaling had already added three.

Cheat sheet

General rules of thumb, not results from my demo.

StatusUsually meansFirst command
CrashLoopBackOff The app crashes as soon as it starts, over and over kubectl logs <pod> --previous
OOMKilled The app used more memory than its limit allows kubectl describe pod <pod> (check Last State)
Pending Kubernetes can't find room for the pod, e.g. it asks for too much or a quota is hit kubectl describe pod <pod> (check Events)
09

Cost and clean-up

Once everything was recorded, I deleted the cluster. Nothing is left running, so keeping this page online costs $0 a month.

The whole demo cost under US$1 (estimated) and ran on Free Trial credit; every resource was deleted afterwards.

Everything existed for about 37 minutes. The cluster was created at 18:11 UTC and was gone by 18:48 UTC on 2026-10-13. What the cost was made of:

  • Autopilot pods. Billed per second for the CPU and memory they request. That was the 3 hello pods (up to 6 while autoscaling), plus the short-lived load generator, CrashLoopBackOff and OOMKilled pods.
  • External load balancer. The forwarding rule behind the LoadBalancer Service, for about 30 minutes.

To clean up I deleted the Service first (that releases the load balancer), then the cluster. Then I listed what was left.

$ kubectl delete service hello
service "hello" deleted from default namespace

$ gcloud container clusters delete gke-demo --region asia-southeast1
Deleting cluster gke-demo...  (progress dots trimmed)  done.
Deleted [https://container.googleapis.com/v1/projects/my-project/zones/asia-southeast1/clusters/gke-demo].

$ gcloud container clusters list

$ gcloud compute forwarding-rules list
Listed 0 items.

$ gcloud compute target-pools list
Listed 0 items.

$ gcloud compute disks list
Listed 0 items.

$ gcloud compute addresses list
Listed 0 items.

$ gcloud compute instances list
Listed 0 items.

$ gcloud compute instance-groups list
Listed 0 items.
10

The manifests

The actual Kubernetes YAML files from the demo. 01-deployment.yaml holds the 2.0 image, as it was after the update. The 1.0 version is kept as its own file.

01-deployment-v1.yaml The hello Deployment as first applied, image version 1.0.
# The demo app (as first applied, version 1.0): Google's public sample "hello-app". 3 replicas, each asking for 250m CPU and 512Mi memory.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello
        image: us-docker.pkg.dev/google-samples/containers/gke/hello-app:1.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: 250m
            memory: 512Mi
        readinessProbe:
          httpGet:
            path: /
            port: 8080
01-deployment.yaml The hello Deployment after the rolling update, image version 2.0.
# The demo app: Google's public sample "hello-app". 3 replicas, each asking for 250m CPU and 512Mi memory.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello
        image: us-docker.pkg.dev/google-samples/containers/gke/hello-app:2.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: 250m
            memory: 512Mi
        readinessProbe:
          httpGet:
            path: /
            port: 8080
02-service.yaml The LoadBalancer Service that exposes the pods.
# One stable address in front of the pods, exposed to the internet through a Google Cloud load balancer.
apiVersion: v1
kind: Service
metadata:
  name: hello
spec:
  type: LoadBalancer
  selector:
    app: hello
  ports:
  - port: 80
    targetPort: 8080
03-hpa.yaml The autoscaler: 50% CPU target, 3 to 10 replicas.
# Horizontal Pod Autoscaler: keep average CPU near 50% of the request, between 3 and 10 pods.
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: hello
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: hello
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50
04-load-generator.yaml A temporary busybox traffic generator for the scaling demo.
# Temporary traffic generator for the scaling demo (deleted afterwards).
apiVersion: apps/v1
kind: Deployment
metadata:
  name: load-generator
spec:
  replicas: 4
  selector:
    matchLabels:
      app: load-generator
  template:
    metadata:
      labels:
        app: load-generator
    spec:
      containers:
      - name: load
        image: busybox:1.36
        command: ["sh", "-c", "while true; do wget -q -O- http://hello > /dev/null; done"]
        resources:
          requests:
            cpu: 250m
            memory: 256Mi
05-crashloop.yaml Troubleshooting demo 1: a pod that exits because DATABASE_URL is missing.
# Troubleshooting demo 1: an app that exits because a required setting is missing -> CrashLoopBackOff.
apiVersion: v1
kind: Pod
metadata:
  name: payments-worker
spec:
  containers:
  - name: worker
    image: busybox:1.36
    command: ["sh", "-c", "echo 'Starting payments-worker v1.4.2'; echo 'Connecting to database...'; echo 'ERROR: DATABASE_URL is not set'; exit 1"]
    resources:
      requests:
        cpu: 250m
        memory: 256Mi
06-oom.yaml Troubleshooting demo 2: a pod that goes over its 256Mi memory limit.
# Troubleshooting demo 2: a container that uses more memory than its 256Mi limit -> OOMKilled.
apiVersion: v1
kind: Pod
metadata:
  name: report-builder
spec:
  restartPolicy: Never
  containers:
  - name: builder
    image: busybox:1.36
    command: ["sh", "-c", "echo 'Loading 600 MB report into memory...'; head -c 600m /dev/zero | tail"]
    resources:
      requests:
        cpu: 250m
        memory: 256Mi
      limits:
        memory: 256Mi