Terraform is a technology for managing infrastructure as code. Many projects would benefit greatly from using it to manage infrastrucutre.

GKE is googles kubernetes offering, and has some great features and offerings to make your experience with kubernetes more amazing.

In this article I'll walk you through some of the options you may want to consider, help you avoid pitfalls, and present my preference for a nice general purpose config.

Feel free to skip to the code , and go back through the article for any arguments that you're curious about.

For tips on writing better cloud native services check out my other article 9 tips for writing better micro-services.

Cluster

We'll start with a minimal config, and add to it.

A basic config will look something like this (the next thing we cover is location which is required).

resource "google_container_cluster" "clustername" {
  name     = "clustername"
}

Location

Location is a required field, and an important one to discuss. This field controls both what regions/zones your cluster runs in, but also whether it runs as a "zonal" or "regional" cluster.

Zonal clusters run in a single availability zone, whereas regional clusters run in multiple but require more instances of every pod (one per availability zone).

If you have enough traffic to make use of multiple instances of each pod, regional will be a great option. If you need to do everything you can to minimize downtime, you should go with a regional cluster.

If your traffic is minimal, and minimizing cost of resources is important, go with a zonal config.

Zonal

resource "google_container_cluster" "clustername" {
  ...
  location = "us-central1-a"
}

Regional

resource "google_container_cluster" "clustername" {
  ...
  location = "us-central1"

  // Optional control over which zones your regional cluster runs in
  node_locations = ["us-central1-a","us-central1-b"]
}

Networking Mode - IMPORTANT

Perhaps the biggest pitfall in setting up your cluster using terraform.

Terraform managed clusters default to routes based clustering, wheras google defaults to VPC_NATIVE. In order to work with private Cloud SQL you'll need VPC_NATIVE.

More info on vpc native clusters here .

resource "google_container_cluster" "clustername" {
  ...
  networking_mode = "VPC_NATIVE"

  // Networking mode VPC_NATIVE also requires an ip allocation policy
  ip_allocation_policy {
    cluster_ipv4_cidr_block  = "/16"
    services_ipv4_cidr_block = "/22"
  }
}

Release channels

Release channels allows you to have google manage your kubernetes update cycle.

Stable is likely good enough for most production workloads, unless you have a specific need for something else.

resource "google_container_cluster" "clustername" {
  ...

  release_channel {
    channel = "STABLE"
  }
}

Workload Identity Config

I need to review this, I haven't been able to get it working properly quite yet.

This option though allows you to specify credentials in a way that new credentials will be generated and used each time a pod is spun up, yet with correct permissions to run what it needs.

Timeouts

Clusters can take some time to create and update. To avoid running into these limits in terraform, let's increase the timeouts.

Without these, there is a higher chance google takes long enough that terraform thinks the operation failed and cancels it.

resource "google_container_cluster" "clustername" {
  ...

  timeouts {
    create = "20m"
    update = "20m"
  }
}

Shielded Nodes

Shielded nodes increases the security of your cluster by verifying nodes that join. Otherwise it's a bit easier to impersonate nodes in your cluster given the ability to exploit a pod.

resource "google_container_cluster" "clustername" {
  ...
  enable_shielded_nodes = "true"
}

Maintance Policy

By default google will run automatic upgrades of your cluster whenever it feels like it. To change it to start in a specific window, this is how. Time is in GMT. There are some other options to choose from in maintenance policy, but this is what I use.

resource "google_container_cluster" "clustername" {
  ...
  maintenance_policy {
    daily_maintenance_window {
      start_time = "03:00"
    }
  }
}

Resource Usage

Gather data on resource usage within the cluster. Good for determining what containers are costing what amount.

resource "google_container_cluster" "clustername" {
  ...
  resource_usage_export_config {
    enable_network_egress_metering = false
    enable_resource_consumption_metering = true

    bigquery_destination {
      dataset_id = "cluster_resource_usage"
    }
  }
}

Initial Node Count and Remove Default Node Pool - IMPORTANT

Node pools specified in the cluster resource require cluster to be destroyed/recreated in order to change them. For this reason I strongly recommend using a separate node pool config. This is what's required for the cluster config in order to enable that.

Clusters spun up by terraform have to have a node pool to start. We'll tell terraform to auto destroy that node pool, and have at most one node in it while creating.

resource "google_container_cluster" "clustername" {
  ...

  initial_node_count       = 1
  remove_default_node_pool = true
}

Other options

Some other options on the cluster object that I generally leave at defaults, but you may be interested in.

enable_autopilot

Autopilot mode gives a more hands off experience for GKE at the expense of less flexibility.

network

Specify what network your cluster runs in. I think default is often good enough, but for more advanced use cases you may want to use another network.

vertical_pod_autoscaling

Still in beta, but sounds like a very nice feature. This will handle tuning cpu/mem requests for you.

Istio config

Note, google seems to discourage using this option. The suggest using anthos if you want service mesh features supported by them. Further the just released a competitor to istio that I imagine will be better supported long term.

Node Pool

Node pools define what VM's are used to run our kubernetes cluster. For the rest of the article we'll be covering the configuration of this resource instead of the cluster's config.

Location (basically the same as cluster)

Zones/Regions should be the same as cluster.

Zonal

resource "google_container_node_pool" "nodepoolname" {
  ...
  location = "us-central1-a"
}

Regional

resource "google_container_node_pool" "nodepoolname" {
  ...
  location = "us-central1"

  // Optional control over which zones your regional cluster runs in
  node_locations = ["us-central1-a","us-central1-b"]
}

Auto-scaling

Auto-scaling is a very important feature allowing your cluster to grow and shrink with your traffic.

I'd suggest having a max node of maybe double what you may need at peak. Basically you want some flexibility and ability to scale, but you don't want a bug making taking up massive CPU to cost massive amounts since your cluster scaled to something huge.

resource "google_container_node_pool" "nodepoolname" {
  ...

  initial_node_count = 1
  autoscaling {
    min_node_count = 1
    max_node_count = 12
  }
}

Node Config - Machine Type

What type of VM your node pool uses. At a rough estimate, I generally target having a low number of nodes, but never less than (number of zones your running in).

Larger nodes will be easier to efficiently make use of, but if your not using all the resources they provide it'll be a waste.

resource "google_container_node_pool" "nodepoolname" {
  ...

  node_config {
    ...

    machine_type = "e2-medium"
  }
}

Machine type docs here .

Node Config - Preemptible

Preemptible VM's are a cheap alternative to regular VM's with the huge caveat that they may be shut down at any point (hopefully with 60s or so warning).

They can save alot of money, but they downside means you'll probably want them only for secondary data processing that isn't directly serving user requests, can shutdown safely and quickly, and/or be restarted/recovered easily.

resource "google_container_node_pool" "nodepoolname" {
  ...

  node_config {
    ...

    preemptible = "true"
  }
}

Node Config - Workload metadata config

Same as workload identity from cluster config, I need to experiment more with this.

Useful for helping secure your cluster more.

Node Config - OAuth scopes

Some permissions you'll need for your node pool.

resource "google_container_node_pool" "nodepoolname" {
  ...

  node_config {
    ...

    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
      "https://www.googleapis.com/auth/cloud-platform",
    ]
  }
}

Node Config - Metadata - disable-legacy-endpoints

Default value set by GKE. Terraform will try to unset it though if you haven't specified it.

resource "google_container_node_pool" "nodepoolname" {
  ...

  node_config {
    metadata = {
      disable-legacy-endpoints = true
    }
  }
}

Management - Auto repair

Should google recover bad nodes for you?

resource "google_container_node_pool" "nodepoolname" {
  ....
  management {
    auto_repair  = true
  }
}

Management - Auto upgrade

Should node pools auto upgrade?

resource "google_container_node_pool" "nodepoolname" {
  ....
  management {
    ....
    auto_upgrade = true
  }
}

Timeouts

Extra time for terraform to apply changes to your node pool. They can take awhile to run.

resource "google_container_node_pool" "nodepoolname" {
  ...

  timeouts {
    create = "20m"
    update = "20m"
  }
}

Other Node pool options

guest accelerators

Allows attaching gpu's to your nodes.

General Purpose Config

This is my take a general purpose config, that should be useful in many cases. But read through and find what changes make sense for your use case.

Find the full example at

Just the cluster and node pool configs

resource "google_container_cluster" "cluster" {
  name     = "cluster"
  location = "us-central1"

  node_locations = ["us-central1-a", "us-central1-b"]

  release_channel {
    channel = "STABLE"
  }

  enable_shielded_nodes = "true"

  workload_identity_config {
    identity_namespace = "${var.project}.svc.id.goog"
  }

  resource_usage_export_config {
    enable_network_egress_metering = true
    enable_resource_consumption_metering = true

    bigquery_destination {
      dataset_id = google_bigquery_dataset.cluster_resource_export.dataset_id
    }
  }

  networking_mode = "VPC_NATIVE"
  ip_allocation_policy {
    cluster_ipv4_cidr_block  = "/16"
    services_ipv4_cidr_block = "/22"
  }

  maintenance_policy {
    daily_maintenance_window {
      start_time = "03:00"
    }
  }

  initial_node_count       = 1
  remove_default_node_pool = true

  timeouts {
    create = "20m"
    update = "20m"
  }
}

resource "google_container_node_pool" "primary" {
  name     = "primary"

  location = "us-central1"
  node_locations = ["us-central1-a", "us-central1-b"]

  cluster  = google_container_cluster.cluster.name

  initial_node_count = 1
  autoscaling {
    min_node_count = 1
    max_node_count = 3
  }

  node_config {
    preemptible  = false
    machine_type = "e2-medium"

    workload_metadata_config {
      node_metadata = "GKE_METADATA_SERVER"
    }

    metadata = {
      disable-legacy-endpoints = true
    }

    oauth_scopes = [
      "https://www.googleapis.com/auth/logging.write",
      "https://www.googleapis.com/auth/monitoring",
      "https://www.googleapis.com/auth/cloud-platform",
    ]
  }

  management {
    auto_repair  = true
    auto_upgrade = true
  }

  timeouts {
    create = "20m"
    update = "20m"
  }
}

Conclusion

GKE is a powerful, full featured and mature offering for kubernetes. With lots of options though it is even more important to manage your config with infrastructure as code.

Terraform is great for managing GKE, but there are a few pitfalls to be aware of (including some defaults that are different in terraform vs gke). It can make it much easier to version control your config, test it in a dev setting, and otherwise be more sure the complexity of configuring your cluster can be tested before YOLO'ing it in your production environment.

Let me know what I've missed, what you disagree with, what was unclear or anything you feel like commenting on below! Your feedback helps me improve.