Homelab Diary Part 4: Talos, Minus the Manual Work

hovorkaobch@gmail.com (Jakub Hovorka) — Wed, 01 Jul 2026 00:00:00 +0000

In the previous part of this series, I finally put the homelab together and installed Proxmox on each machine. In the meantime, I’ve also done some basic setup, such as updating all package repositories on each host, joining all three machines into a Proxmox cluster, and installing a Prometheus exporter to track resource usage and temperatures of each machine.

Now it’s time to finally start running things on the homelab, but to do that, I will need a good foundation. I like to do things in an organized way and make as few manual changes as possible. The reason behind it is that I want to continuously build a library of resources that I can share with other people, or just use in the future, when I need it. If I were to set up everything manually, or with scripts, it would be kind of hard to understand for anyone who is not familiar with my setup. I am also sure I would forget how everything works if I came back to it a year later. To tackle this challenge, I decided to use Infrastructure as Code with tools like OpenTofu and Ansible, and GitOps with ArgoCD, and GitHub Actions. This will allow me to build everything in a way I would do it at a real company, and it will force me to keep the whole setup clean.

Let’s start with OpenTofu. To keep the whole setup modular and reusable, I will create a monorepository, which will hold all of my OpenTofu modules. I will keep this repository public forever, and I will also handle the versioning, so existing setups relying on these modules won’t break anytime I make some changes. Anyone interested in replicating my setup, or a part of it, will then be able to reference the individual modules using the repository URL and a version tag. I will be using this repository in my homelab, which will force me to keep it up-to-date.

My first goal is to be able to create and maintain Kubernetes clusters. I am a big fan of Talos Linux, so that’s what my Kubernetes clusters will run on. Lucky for me, there is siderolabs/talos OpenTofu provider, which, from my experience, is really good. Before I can do anything, I first need to spin up the infrastructure for the clusters, which I will do on Proxmox. There are multiple Proxmox OpenTofu providers, but the best one from my experience is the bpg/proxmox, so that’s what I will be using. I will first go over all the modules, and in the end, I will show an example of how to wire them all together in a nice, scalable way.

Images

The very first module I need is the one to look up Talos images. The module is very simple - I just give it a Talos version and the extensions I need, and it asks the Talos Image Factory for the resulting installer image and ISO URLs.

data "proxmox_virtual_environment_nodes" "this" {}

locals {
 # The Image Factory's ISO URL ends in a fixed name that's the same for
 # every version/schematic of a given platform+arch - it encodes those
 # earlier in the path, not in the final segment. Deriving file_name from
 # it directly would mean every version manually uploaded to the same
 # datastore collides on one name. Embedding version and schematic here
 # keeps each one distinct, and is what image_nodes/iso_file_name hand out
 # for that manual upload.
 file_name = "talos-${var.talos_image_version}-${talos_image_factory_schematic.this.id}-${var.talos_image_platform}.iso"

 target_nodes = var.proxmox_nodes != null ? var.proxmox_nodes : toset(data.proxmox_virtual_environment_nodes.this.names)
}

data "talos_image_factory_extensions_versions" "this" {
 talos_version = var.talos_image_version
 filters = {
 names = var.talos_image_extensions
 }
}

resource "talos_image_factory_schematic" "this" {
 schematic = yamlencode(
 {
 customization = {
 systemExtensions = {
 officialExtensions = data.talos_image_factory_extensions_versions.this.extensions_info[*].name
 }
 }
 }
 )
}

data "talos_image_factory_urls" "this" {
 talos_version = var.talos_image_version
 schematic_id = talos_image_factory_schematic.this.id
 platform = var.talos_image_platform
}

data "proxmox_virtual_environment_nodes" "this" {}
locals {
# The Image Factory's ISO URL ends in a fixed name that's the same for
# every version/schematic of a given platform+arch - it encodes those
# earlier in the path, not in the final segment. Deriving file_name from
# it directly would mean every version manually uploaded to the same
# datastore collides on one name. Embedding version and schematic here
# keeps each one distinct, and is what image_nodes/iso_file_name hand out
# for that manual upload.
file_name = "talos-${var.talos_image_version}-${talos_image_factory_schematic.this.id}-${var.talos_image_platform}.iso"
target_nodes = var.proxmox_nodes != null ? var.proxmox_nodes : toset(data.proxmox_virtual_environment_nodes.this.names)
}
data "talos_image_factory_extensions_versions" "this" {
talos_version = var.talos_image_version
filters = {
names = var.talos_image_extensions
}
}
resource "talos_image_factory_schematic" "this" {
schematic = yamlencode(
{
customization = {
systemExtensions = {
officialExtensions = data.talos_image_factory_extensions_versions.this.extensions_info[*].name
}
}
}
)
}
data "talos_image_factory_urls" "this" {
talos_version = var.talos_image_version
schematic_id = talos_image_factory_schematic.this.id
platform = var.talos_image_platform
}

Show full file (39 lines) Collapse

Worth clarifying early, since it trips people up: the ISO and the installer image you’ll see referenced later (installer_image_url) aren’t the same artifact. The ISO is boot media a VM reads once, on first boot, to install Talos onto its disk. The installer image is a container image (an OCI reference, e.g. factory.talos.dev/installer/<schematic>:<version>) that Talos pulls and runs internally, both during that same initial install and every time talosctl upgrade runs later - an upgrade doesn’t re-attach an ISO, it just tells the already-running Talos to pull a new installer image and reinstall itself in place. Every installer_image_url from here on refers to that second thing, the installer image, not the ISO.

That distinction shaped a decision I initially got wrong. My first instinct was to have this module download the ISO straight to Proxmox on every apply, since that’s what a fresh VM needs to boot. But the ISO is only ever read once per node - after that, upgrades pull the installer image directly, and the ISO just sits there. Tying the download to talos_version meant a real network transfer and a real file on Proxmox storage on every routine version bump, for a file nothing would ever read again. I tried gating it behind a boolean instead, download_iso, defaulting to false and only flipped on when actually provisioning a new node. That fixed the waste, but it opened a sharper problem: flipping the flag back off on a version that already had VMs booted from it would delete the ISO out from under Terraform’s own tracking, and referencing a version whose ISO was never downloaded failed silently at plan time - clean plan, then an opaque Proxmox error mid-apply.

In the end I pulled the download out of the module entirely. It’s now a pure lookup: installer_image, schematic_id, and an iso_url/iso_file_name pair. When I actually need a new VM to boot off a version, I tell Proxmox to fetch it itself, once per node, straight through the API - the same download-url action behind the “Download from URL” button in the web UI, there’s no pvesm subcommand for this:

curl -k -H "Authorization: PVEAPIToken=$PROXMOX_API_TOKEN" \
 -X POST "https://<proxmox-host>:8006/api2/json/nodes/<node>/storage/<datastore>/download-url" \
 --data-urlencode "content=iso" \
 --data-urlencode "filename=$(tofu output -raw iso_file_name)" \
 --data-urlencode "url=$(tofu output -raw iso_url)"

It’s a manual step, but it only comes up when a brand-new node is joining - which is rare - and it means Terraform never owns a destructive action against a file three other VMs might still be quietly depending on.

Virtual Machines

The next module I need is the one to create the VMs on Proxmox. I’ve built this module over the last 2 years, and I think it’s pretty flexible, and sufficient for all the standard use cases. The module looks like this:

resource "proxmox_virtual_environment_vm" "this" {
 for_each = var.virtual_machines

 node_name = each.value.node_name
 name = each.key
 vm_id = try(each.value.vm_id, null)

 description = try(each.value.description, null)
 tags = try(each.value.tags, null)
 on_boot = try(each.value.on_boot, true)

 machine = try(each.value.machine, null)
 scsi_hardware = try(each.value.scsi_hardware, null)
 bios = try(each.value.bios, null)

 agent {
 enabled = try(each.value.agent_enabled, false)
 timeout = try(each.value.agent_timeout, "15m")
 }

 cpu {
 cores = each.value.cpu.cores
 type = try(each.value.cpu.type, null)
 flags = try(each.value.cpu.flags, [""])
 units = try(each.value.cpu.units, null)
 }

 memory {
 dedicated = each.value.memory.dedicated
 # Below dedicated, this enables ballooning (Proxmox can reclaim memory
 # under host pressure); 0 means no ballooning, memory stays pinned at
 # dedicated - the safer default for a module with no opinion on whether
 # a given VM's workload tolerates being squeezed.
 floating = try(each.value.memory.floating, 0)
 }

 dynamic "network_device" {
 for_each = try(each.value.network_devices, [])
 content {
 bridge = network_device.value.bridge
 mac_address = try(network_device.value.mac_address, null)
 model = try(network_device.value.model, null)
 vlan_id = try(network_device.value.vlan_id, null)
 firewall = try(network_device.value.firewall, false)
 }
 }

 dynamic "disk" {
 for_each = coalesce(each.value.disks, [])
 content {
 datastore_id = disk.value.datastore_id
 interface = try(disk.value.interface, "scsi0")
 iothread = try(disk.value.iothread, true)
 cache = try(disk.value.cache, "writethrough")
 discard = try(disk.value.discard, "on")
 ssd = try(disk.value.ssd, true)
 file_format = try(disk.value.file_format, "")
 file_id = try(disk.value.file_id, "")
 size = disk.value.size
 }
 }

 serial_device {
 device = try(each.value.serial_device.device, null)
 }

 # Defaults to ide3 so it doesn't collide with the cloud-init drive, which
 # Proxmox always attaches on ide2.
 cdrom {
 file_id = try(each.value.cdrom.file_id, "none")
 interface = try(each.value.cdrom.interface, "ide3")
 }

 boot_order = try(each.value.boot_order, [])

 operating_system {
 type = try(each.value.operating_system_type, "other")
 }

 dynamic "initialization" {
 for_each = try(each.value.init, null) != null ? [each.value.init] : []
 content {
 datastore_id = initialization.value.datastore_id
 interface = try(initialization.value.interface, null)

 dynamic "dns" {
 for_each = try(initialization.value.dns, null) != null ? { "dns" = initialization.value.dns } : {}
 content {
 servers = initialization.value.dns
 }
 }

 ip_config {
 ipv4 {
 address = initialization.value.ipv4.address
 gateway = initialization.value.ipv4.gateway
 }
 }

 dynamic "user_account" {
 for_each = try(initialization.value.auth, null) != null ? [initialization.value.auth] : []
 content {
 username = try(initialization.value.auth.username, null)
 password = try(initialization.value.auth.password, null)
 keys = try(user_account.value.keys, null)
 }
 }
 }
 }

 dynamic "clone" {
 for_each = try(each.value.clone, null) != null ? [each.value.clone] : []
 content {
 vm_id = clone.value.vm_id
 datastore_id = try(clone.value.datastore_id, null)
 node_name = try(clone.value.node_name, null)
 retries = try(clone.value.retries, null)
 full = try(clone.value.full, true)
 }
 }

 dynamic "hostpci" {
 for_each = coalesce(try(each.value.pci_devices, []), [])
 content {
 device = hostpci.value.device
 mapping = try(hostpci.value.mapping, null)
 pcie = try(hostpci.value.pcie, false)
 rombar = try(hostpci.value.rombar, false)
 xvga = try(hostpci.value.xvga, false)
 }
 }
}

resource "proxmox_virtual_environment_vm" "this" {
for_each = var.virtual_machines
node_name = each.value.node_name
name = each.key
vm_id = try(each.value.vm_id, null)
description = try(each.value.description, null)
tags = try(each.value.tags, null)
on_boot = try(each.value.on_boot, true)
machine = try(each.value.machine, null)
scsi_hardware = try(each.value.scsi_hardware, null)
bios = try(each.value.bios, null)
agent {
enabled = try(each.value.agent_enabled, false)
timeout = try(each.value.agent_timeout, "15m")
}
cpu {
cores = each.value.cpu.cores
type = try(each.value.cpu.type, null)
flags = try(each.value.cpu.flags, [""])
units = try(each.value.cpu.units, null)
}
memory {
dedicated = each.value.memory.dedicated
# Below dedicated, this enables ballooning (Proxmox can reclaim memory
# under host pressure); 0 means no ballooning, memory stays pinned at
# dedicated - the safer default for a module with no opinion on whether
# a given VM's workload tolerates being squeezed.
floating = try(each.value.memory.floating, 0)
}
dynamic "network_device" {
for_each = try(each.value.network_devices, [])
content {
bridge = network_device.value.bridge
mac_address = try(network_device.value.mac_address, null)
model = try(network_device.value.model, null)
vlan_id = try(network_device.value.vlan_id, null)
firewall = try(network_device.value.firewall, false)
}
}
dynamic "disk" {
for_each = coalesce(each.value.disks, [])
content {
datastore_id = disk.value.datastore_id
interface = try(disk.value.interface, "scsi0")
iothread = try(disk.value.iothread, true)
cache = try(disk.value.cache, "writethrough")
discard = try(disk.value.discard, "on")
ssd = try(disk.value.ssd, true)
file_format = try(disk.value.file_format, "")
file_id = try(disk.value.file_id, "")
size = disk.value.size
}
}
serial_device {
device = try(each.value.serial_device.device, null)
}
# Defaults to ide3 so it doesn't collide with the cloud-init drive, which
# Proxmox always attaches on ide2.
cdrom {
file_id = try(each.value.cdrom.file_id, "none")
interface = try(each.value.cdrom.interface, "ide3")
}
boot_order = try(each.value.boot_order, [])
operating_system {
type = try(each.value.operating_system_type, "other")
}
dynamic "initialization" {
for_each = try(each.value.init, null) != null ? [each.value.init] : []
content {
datastore_id = initialization.value.datastore_id
interface = try(initialization.value.interface, null)
dynamic "dns" {
for_each = try(initialization.value.dns, null) != null ? { "dns" = initialization.value.dns } : {}
content {
servers = initialization.value.dns
}
}
ip_config {
ipv4 {
address = initialization.value.ipv4.address
gateway = initialization.value.ipv4.gateway
}
}
dynamic "user_account" {
for_each = try(initialization.value.auth, null) != null ? [initialization.value.auth] : []
content {
username = try(initialization.value.auth.username, null)
password = try(initialization.value.auth.password, null)
keys = try(user_account.value.keys, null)
}
}
}
}
dynamic "clone" {
for_each = try(each.value.clone, null) != null ? [each.value.clone] : []
content {
vm_id = clone.value.vm_id
datastore_id = try(clone.value.datastore_id, null)
node_name = try(clone.value.node_name, null)
retries = try(clone.value.retries, null)
full = try(clone.value.full, true)
}
}
dynamic "hostpci" {
for_each = coalesce(try(each.value.pci_devices, []), [])
content {
device = hostpci.value.device
mapping = try(hostpci.value.mapping, null)
pcie = try(hostpci.value.pcie, false)
rombar = try(hostpci.value.rombar, false)
xvga = try(hostpci.value.xvga, false)
}
}
}

Show full file (132 lines) Collapse

And here are the variables:

variable "virtual_machines" {
 description = "Map of VMs to create. Map key becomes the VM name."
 type = map(object({
 node_name = string
 vm_id = optional(number)
 description = optional(string)
 tags = optional(list(string))
 on_boot = optional(bool)

 machine = optional(string)
 scsi_hardware = optional(string)
 bios = optional(string)

 # Requires the QEMU guest agent running inside the guest OS, otherwise
 # Terraform just waits out `agent_timeout` on every apply.
 agent_enabled = optional(bool)
 agent_timeout = optional(string)

 cpu = object({
 cores = number
 type = optional(string)
 flags = optional(list(string))
 units = optional(number)
 })

 memory = object({
 dedicated = number
 # enables memory ballooning
 floating = optional(number)
 })

 network_devices = optional(list(object({
 bridge = string
 mac_address = optional(string)
 model = optional(string)
 vlan_id = optional(string)
 firewall = optional(bool)
 })))

 disks = optional(list(object({
 datastore_id = string
 interface = optional(string)
 iothread = optional(bool)
 cache = optional(string)
 discard = optional(string)
 ssd = optional(bool)
 file_format = optional(string)
 size = number
 # source image/template to clone from
 file_id = optional(string)
 })))

 # Defaults to ide3: ide2 is reserved for the cloud-init drive below.
 cdrom = optional(object({
 file_id = optional(string)
 interface = optional(string)
 }))

 serial_device = optional(object({
 device = optional(string)
 }))

 boot_order = optional(list(string))
 operating_system_type = optional(string)

 init = optional(object({
 datastore_id = string
 dns = optional(list(string))
 ipv4 = object({
 address = string
 gateway = string
 })
 auth = optional(object({
 username = string
 password = optional(string)
 keys = optional(list(string))
 }))
 }))

 # Clones this VM from an existing template instead of building it fresh.
 clone = optional(object({
 vm_id = number
 datastore_id = optional(string)
 node_name = optional(string)
 retries = optional(number)
 full = optional(bool)
 }))

 # PCI/GPU passthrough. `mapping` refers to a resource mapping configured
 # under Datacenter > Resource Mappings in Proxmox.
 pci_devices = optional(list(object({
 device = string
 mapping = optional(string)
 pcie = optional(bool)
 rombar = optional(bool)
 xvga = optional(bool)
 })))
 }))
}

variable "virtual_machines" {
description = "Map of VMs to create. Map key becomes the VM name."
type = map(object({
node_name = string
vm_id = optional(number)
description = optional(string)
tags = optional(list(string))
on_boot = optional(bool)
machine = optional(string)
scsi_hardware = optional(string)
bios = optional(string)
# Requires the QEMU guest agent running inside the guest OS, otherwise
# Terraform just waits out `agent_timeout` on every apply.
agent_enabled = optional(bool)
agent_timeout = optional(string)
cpu = object({
cores = number
type = optional(string)
flags = optional(list(string))
units = optional(number)
})
memory = object({
dedicated = number
# enables memory ballooning
floating = optional(number)
})
network_devices = optional(list(object({
bridge = string
mac_address = optional(string)
model = optional(string)
vlan_id = optional(string)
firewall = optional(bool)
})))
disks = optional(list(object({
datastore_id = string
interface = optional(string)
iothread = optional(bool)
cache = optional(string)
discard = optional(string)
ssd = optional(bool)
file_format = optional(string)
size = number
# source image/template to clone from
file_id = optional(string)
})))
# Defaults to ide3: ide2 is reserved for the cloud-init drive below.
cdrom = optional(object({
file_id = optional(string)
interface = optional(string)
}))
serial_device = optional(object({
device = optional(string)
}))
boot_order = optional(list(string))
operating_system_type = optional(string)
init = optional(object({
datastore_id = string
dns = optional(list(string))
ipv4 = object({
address = string
gateway = string
})
auth = optional(object({
username = string
password = optional(string)
keys = optional(list(string))
}))
}))
# Clones this VM from an existing template instead of building it fresh.
clone = optional(object({
vm_id = number
datastore_id = optional(string)
node_name = optional(string)
retries = optional(number)
full = optional(bool)
}))
# PCI/GPU passthrough. `mapping` refers to a resource mapping configured
# under Datacenter > Resource Mappings in Proxmox.
pci_devices = optional(list(object({
device = string
mapping = optional(string)
pcie = optional(bool)
rombar = optional(bool)
xvga = optional(bool)
})))
}))
}

Show full file (99 lines) Collapse

There isn’t anything particularly exotic in this module, most of the things are just standard Proxmox VM attributes but there are a few things I want to point out. First is the virtual_machines variable, which, as you might see, is the only variable in this module, and it has a lot of different fields nested in it. This approach allows me to only initiate the module once, no matter if I want to create 1 VM or 100 VMs. This is especially useful for K8s clusters, where all VMs have very similar attributes that would otherwise have to be redefined for every single one.

Second is the cdrom block, which defaults to interface ide3 instead of the more obvious ide2. Reasoning behind this one is simple: Proxmox always reserves ide2 for the cloud-init drive whenever cloud-init is enabled, so if the cdrom also defaulted to ide2, it would just collide with it. ide3 is simply the next slot that’s actually free.

Talos

This is the last module I need for this part, and there is a lot to unpack here. If you are familiar with the Talos cluster creation process, this is basically it, just transformed from individual talosctl commands to OpenTofu code. The module opens with the talos_machine_secrets resource, which generates the secrets shared by the whole cluster. Then there is talos_client_configuration, which generates a talosconfig for the whole cluster, and talos_machine_configuration, which generates a machine config for each node. talos_machine_configuration_apply applies that config - to every node, control planes included, and unsequenced.

resource "talos_machine_secrets" "this" {}

data "talos_client_configuration" "this" {
 cluster_name = var.cluster.name
 client_configuration = talos_machine_secrets.this.client_configuration
 nodes = [for ip in local.talos_api_ips : ip]
 endpoints = local.control_plane_ips
}

data "talos_machine_configuration" "this" {
 for_each = var.nodes

 cluster_name = var.cluster.name
 cluster_endpoint = "https://${local.cluster_endpoint}:6443"
 machine_type = each.value.machine_type
 machine_secrets = talos_machine_secrets.this.machine_secrets
 config_patches = local.node_config_patches[each.key]
}

resource "talos_machine_configuration_apply" "this" {
 for_each = var.nodes

 node = local.talos_api_ips[each.key]
 apply_mode = each.value.apply_mode
 client_configuration = talos_machine_secrets.this.client_configuration
 machine_configuration_input = data.talos_machine_configuration.this[each.key].machine_configuration
}

resource "talos_machine_secrets" "this" {}
data "talos_client_configuration" "this" {
cluster_name = var.cluster.name
client_configuration = talos_machine_secrets.this.client_configuration
nodes = [for ip in local.talos_api_ips : ip]
endpoints = local.control_plane_ips
}
data "talos_machine_configuration" "this" {
for_each = var.nodes
cluster_name = var.cluster.name
cluster_endpoint = "https://${local.cluster_endpoint}:6443"
machine_type = each.value.machine_type
machine_secrets = talos_machine_secrets.this.machine_secrets
config_patches = local.node_config_patches[each.key]
}
resource "talos_machine_configuration_apply" "this" {
for_each = var.nodes
node = local.talos_api_ips[each.key]
apply_mode = each.value.apply_mode
client_configuration = talos_machine_secrets.this.client_configuration
machine_configuration_input = data.talos_machine_configuration.this[each.key].machine_configuration
}

Show full file (27 lines) Collapse

Once every node has its config applied, talos_machine_bootstrap runs the cluster bootstrap against the first control plane node - whichever one happens to come first in the nodes map.

resource "talos_machine_bootstrap" "this" {
 depends_on = [
 talos_machine_configuration_apply.this,
 ]

 node = local.first_control_plane_api_ip
 client_configuration = talos_machine_secrets.this.client_configuration
}

resource "talos_machine_bootstrap" "this" {
depends_on = [
talos_machine_configuration_apply.this,
]
node = local.first_control_plane_api_ip
client_configuration = talos_machine_secrets.this.client_configuration
}

Once the cluster is bootstrapped, I confirm that it becomes healthy using the talos_cluster_health data source, and retrieve the kubeconfig using the talos_cluster_kubeconfig resource. That’s it for main.tf - both Talos OS and Kubernetes upgrades are handled entirely outside it, by scripts/talos.sh, which doesn’t even live inside this module.

An upgrade is a multi-minute, multi-node procedure - snapshot etcd, then proceed node by node, health-gated between each - and that’s a poor fit for a Terraform resource. Terraform’s model is converging to a declared state, not running a multi-step imperative procedure, and doing the latter through a local-exec provisioner means the only way to abort cleanly is to not interrupt a tofu apply for several minutes - fragile by nature, since an interrupted apply can leave a node in an unknown state with no clean way to resume.

So Terraform only owns declaring each node’s target installer_image_url, exposed - along with each node’s reachable IP and role - through a nodes output. Rolling that out to the real cluster is scripts/talos.sh’s job, and it lives at the repo root rather than nested inside this module - it isn’t Terraform, so it doesn’t need fetching through a module source, and it’s a single self-contained file rather than several that could drift out of sync, since it’s meant to be curl’d and run directly. One dispatcher, two subcommands, so there’s one thing to remember instead of a script per operation: talos.sh upgrade-talos <cluster-dir> reads the declared state above via tofu output and reconciles the real cluster to match it, one node at a time. talos.sh upgrade-k8s <cluster-dir> <target-version> does the equivalent for Kubernetes itself, driving talosctl upgrade-k8s - which already sequences its own rollout across every node, so there’s no per-node loop to write for that one.

Both only ever run against an already-bootstrapped, already-running cluster, so they can afford to be thorough: snapshot etcd before touching anything, check Talos and Kubernetes-level health before starting, and avoid the cluster’s VIP throughout - VIP reliability during a control-plane reboot varies by provider and network setup, so rather than assume either way, every check is pinned to a specific node instead, and never to the one currently being upgraded. upgrade-talos additionally waits for each node to report Ready in Kubernetes and uncordons it as a safety net in case a previous failed run left it cordoned - Talos already uncordons automatically after its own reboot, this is just insurance.

One thing worth being careful about: the two subcommands expect the opposite order relative to Terraform. For upgrade-talos, Terraform goes first - bump installer_image_url and tofu apply, then run the script. For upgrade-k8s, the script goes first - talosctl upgrade-k8s doesn’t care what Terraform thinks the version is, so run it against the real cluster, confirm it worked, and only then bump k8s_version and apply, to sync the declaration with what’s now actually running. upgrade-k8s also checks the cluster’s actual current version against the target before doing anything - if they already match, that’s a sign this ran already, or Terraform got applied out of order, so it stops and asks rather than silently proceeding. It can’t check the Terraform side directly (k8s_version isn’t part of this module’s own output surface the way installer_image_url is, and the variable name is caller-specific), but the cluster’s real state is something it can always check.

curl -fsSL https://raw.githubusercontent.com/hovorka-labs/iac-modules/scripts-v1.1.1/scripts/talos.sh -o talos.sh
chmod +x talos.sh

#!/usr/bin/env bash
#
# Operational tooling for a Talos Kubernetes cluster built with the modules
# in this repo (terraform/modules/talos). One file, one entry point - you
# tell it what you want done, it does it, rather than needing to know and
# fetch a different script per operation.
#
# Deliberately outside Terraform entirely. An upgrade (Talos OS or
# Kubernetes) is a multi-minute, multi-node procedure - snapshot etcd, then
# proceed node by node, health-gated between each - and that's a poor fit
# for a Terraform resource: Terraform's model is converging to a declared
# state, not running a multi-step imperative procedure, and an interrupted
# `tofu apply` mid-procedure can leave a node in an unknown state with no
# clean way to resume. This reads/writes real cluster state directly, via
# `tofu output` for the values it needs from your Terraform config.
#
# Single self-contained file on purpose, no separate sourced helper file -
# for something you're expected to curl and run, one atomic download is
# safer than several that could drift out of sync with each other, and it
# means you can read the entire thing before running it.
#
# Usage:
# ./talos.sh upgrade-talos <cluster-dir>
# ./talos.sh upgrade-k8s <cluster-dir> <target-version>
# ./talos.sh --help
#
# cluster-dir is always required: the directory you'd normally run
# `tofu apply` from for this cluster. No default - if you curled this
# script somewhere else first, the current directory is never that
# directory, so guessing would be more likely to be wrong than right.
#
# Get it:
# curl -fsSL https://raw.githubusercontent.com/hovorka-labs/iac-modules/scripts-v1.1.0/scripts/talos.sh -o talos.sh
# chmod +x talos.sh
#
# Env vars:
# TALOSCTL talosctl binary to use (default: talosctl on PATH).
# Override when operating multiple clusters/versions
# at once - upgrade-k8s requires the client to be at
# least as new as the cluster's Talos version, so an
# older PATH talosctl can fail checks a newer one
# wouldn't.
# AUTO_CONFIRM=1 skip every confirmation prompt
# SLEEP_BETWEEN_NODES seconds to pause between nodes, upgrade-talos only (default 15)
# NODE_READY_TIMEOUT kubectl wait timeout per node, upgrade-talos only (default 300s)
# HEALTH_WAIT_TIMEOUT talosctl health --wait-timeout (default 10m)

set -euo pipefail

AUTO_CONFIRM="${AUTO_CONFIRM:-0}"
SLEEP_BETWEEN_NODES="${SLEEP_BETWEEN_NODES:-15}"
NODE_READY_TIMEOUT="${NODE_READY_TIMEOUT:-300s}"
HEALTH_WAIT_TIMEOUT="${HEALTH_WAIT_TIMEOUT:-10m}"
TALOSCTL="${TALOSCTL:-talosctl}"

log() { echo -e "\n[$(date +%H:%M:%S)] $*"; }
die() { echo -e "\n!! $* -- aborting." >&2; exit 1; }

usage() {
 cat <<'EOF'
talos.sh - operational tooling for a Talos Kubernetes cluster

Usage:
 talos.sh upgrade-talos <cluster-dir>
 Roll out the Talos OS version already declared (installer_image_url)
 in your Terraform config, one node at a time. Run AFTER bumping the
 version and `tofu apply`-ing it.

 talos.sh upgrade-k8s <cluster-dir> <target-version>
 Upgrade the Kubernetes control plane and kubelets to <target-version>
 (e.g. v1.35.6), via talosctl's own upgrade-k8s. Run BEFORE touching
 k8s_version in Terraform - bump and apply that afterward, to sync the
 declaration with what's now actually running.

 talos.sh --help
 Show this message.
EOF
}

require_bins() {
 local bin
 for bin in "$@"; do command -v "$bin" >/dev/null || die "$bin not found"; done
 [[ "$TALOSCTL" != "talosctl" ]] && log "Using talosctl override: $TALOSCTL ($("$TALOSCTL" version --client --short 2>/dev/null | tail -1))"
 return 0
}

k8s_node_name_for_ip() {
 kubectl get nodes -o json 2>/dev/null | jq -r --arg ip "$1" \
 '.items[] | select(.status.addresses[]?.address==$ip) | .metadata.name'
}

current_talos_tag() {
 "$TALOSCTL" --endpoints "$1" --nodes "$1" version --short 2>/dev/null \
 | awk '/^Server:/{f=1; next} f && /Tag:/{print $2; exit}'
}

current_k8s_version() {
 kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}' 2>/dev/null
}

# VIP reliability during a control-plane reboot varies by provider/network
# setup - sometimes it fails over cleanly, sometimes it doesn't - so rather
# than assume either way, kubectl is always pointed at a real node instead:
# specifically, any control plane OTHER than the one currently being
# upgraded, so asking about cluster state never depends on the one node
# that might be mid-reboot right now.
point_kubectl_away_from() {
 local avoid="$1" target="$FIRST_CP" cand
 for cand in "${CP_IPS[@]}"; do
 if [[ "$cand" != "$avoid" ]]; then target="$cand"; break; fi
 done
 sed -i.bak -E "s#server: .*#server: https://${target}:6443#" "$TMP_DIR/kubeconfig"
 rm -f "$TMP_DIR/kubeconfig.bak"
}

wait_for_apiserver() {
 local timeout="$1" start elapsed
 start=$(date +%s)
 while ! kubectl get --raw='/readyz' &>/dev/null; do
 elapsed=$(( $(date +%s) - start ))
 (( elapsed > timeout )) && return 1
 sleep 5
 done
}

# talosctl health needs CP vs worker roles, or it expects every node to be
# schedulable -- false for CP nodes carrying the default NoSchedule taint.
health_check() {
 "$TALOSCTL" --endpoints "$FIRST_CP" --nodes "$FIRST_CP" \
 health --control-plane-nodes "$CP_CSV" ${WORKER_CSV:+--worker-nodes "$WORKER_CSV"} \
 --k8s-endpoint "$FIRST_CP" --wait-timeout "$1"
}

# etcd can briefly report HEALTH ? ("Unknown") right after its container
# starts, before its first probe has run -- not the same as unhealthy, so
# this retries for a bit instead of failing on the very first check.
etcd_healthy() {
 local status ok="" i
 for i in $(seq 1 12); do
 status=$("$TALOSCTL" --endpoints "$FIRST_CP" --nodes "$CP_CSV" service etcd 2>&1 || true)
 if echo "$status" | grep -q "^HEALTH" && ! echo "$status" | grep "^HEALTH" | grep -qv "OK$"; then
 ok="1"
 break
 fi
 sleep 5
 done
 echo "$status"
 [[ -n "$ok" ]]
}

take_etcd_snapshot() {
 local label="$1"
 log "Taking etcd snapshot (rollback safety net)"
 mkdir -p "$CLUSTER_DIR/etcd-backup"
 "$TALOSCTL" --endpoints "$FIRST_CP" --nodes "$FIRST_CP" etcd snapshot \
 "$CLUSTER_DIR/etcd-backup/${label}-$(date +%Y%m%d-%H%M%S).snapshot" \
 || die "etcd snapshot failed -- not proceeding without a rollback point"
}

preflight() {
 log "Pre-flight: cluster health"
 health_check 2m || die "cluster is not healthy before starting -- fix this first"

 log "Pre-flight: all Kubernetes nodes Ready"
 local not_ready
 not_ready=$(kubectl get nodes -o json | jq -r \
 '.items[] | select(([.status.conditions[]? | select(.type=="Ready") | .status] | contains(["True"])) | not) | .metadata.name')
 [[ -z "$not_ready" ]] || die "node(s) not Ready: $not_ready"

 log "Pre-flight: etcd status"
 etcd_healthy || die "etcd is not healthy before starting -- fix this first"
}

# Fetches kubeconfig/talosconfig/nodes from tofu output for $1, and derives
# CP_CSV/WORKER_CSV/CP_IPS/FIRST_CP. Sets CLUSTER_DIR/TMP_DIR and exports
# KUBECONFIG/TALOSCONFIG. Shared setup for every subcommand.
setup_cluster_access() {
 CLUSTER_DIR="$1"
 [[ -n "$(ls "$CLUSTER_DIR"/*.tf 2>/dev/null)" ]] || die "$CLUSTER_DIR has no .tf files"

 TMP_DIR=$(mktemp -d)
 trap 'rm -rf "$TMP_DIR"' EXIT

 log "Fetching kubeconfig/talosconfig/nodes from tofu output ($CLUSTER_DIR)"
 tofu -chdir="$CLUSTER_DIR" output -raw kubeconfig > "$TMP_DIR/kubeconfig" || die "tofu output kubeconfig failed -- has this cluster been applied?"
 tofu -chdir="$CLUSTER_DIR" output -raw talosconfig > "$TMP_DIR/talosconfig" || die "tofu output talosconfig failed"
 tofu -chdir="$CLUSTER_DIR" output -json nodes > "$TMP_DIR/nodes.json" || die "tofu output nodes failed -- module needs the 'nodes' output (talos-v1.2.0+)"
 chmod 600 "$TMP_DIR/kubeconfig" "$TMP_DIR/talosconfig"
 export KUBECONFIG="$TMP_DIR/kubeconfig" TALOSCONFIG="$TMP_DIR/talosconfig"

 CP_CSV=$(jq -r '[to_entries[] | select(.value.machine_type=="controlplane") | .value.talos_api_ip] | join(",")' "$TMP_DIR/nodes.json")
 WORKER_CSV=$(jq -r '[to_entries[] | select(.value.machine_type=="worker") | .value.talos_api_ip] | join(",")' "$TMP_DIR/nodes.json")
 [[ -n "$CP_CSV" ]] || die "no control-plane nodes in tofu output nodes"
 IFS=',' read -r -a CP_IPS <<< "$CP_CSV"
 FIRST_CP="${CP_IPS[0]}"

 if [[ ${#CP_IPS[@]} -lt 3 ]]; then
 echo "!! Only ${#CP_IPS[@]} control-plane node(s) -- the API server (and kubectl checks) will go fully unreachable while that node reboots."
 fi

 point_kubectl_away_from ""
 log "Cluster: $CLUSTER_DIR"
 log "Control planes: $CP_CSV"
 log "Workers: ${WORKER_CSV:-<none>}"
}

# ---------------------------------------------------------------------------

cmd_upgrade_talos() {
 require_bins "$TALOSCTL" kubectl jq tofu
 [[ -n "${1:-}" ]] || die "cluster-dir required (usage: $0 upgrade-talos <cluster-dir>)"
 setup_cluster_access "$1"

 if [[ "$AUTO_CONFIRM" != "1" ]]; then
 echo "This only rolls out what's already declared - it doesn't bump installer_image_url or run tofu apply for you."
 read -r -p "Have you already set the target installer_image_url and run tofu apply for $CLUSTER_DIR? [y/N] " answer
 echo
 [[ "$answer" =~ ^[Yy]$ ]] || die "bump installer_image_url and tofu apply first, then rerun"
 fi

 preflight
 take_etcd_snapshot "pre-talos-upgrade"

 # Control planes first (etcd quorum must never see two rebooting at once),
 # then workers, both sorted for a stable order across runs.
 local upgrade_order
 upgrade_order=$(jq -r '
 (to_entries | map(select(.value.machine_type=="controlplane")) | sort_by(.key)) +
 (to_entries | map(select(.value.machine_type=="worker")) | sort_by(.key))
 | .[] | "\(.key)|\(.value.machine_type)|\(.value.talos_api_ip)|\(.value.installer_image_url)"
 ' "$TMP_DIR/nodes.json")

 echo
 echo "Plan:"
 local pending=0 name role ip image target current
 while IFS='|' read -r name role ip image; do
 [[ -z "$name" ]] && continue
 target="${image##*:}"
 current=$(current_talos_tag "$ip" || echo "unknown")
 if [[ "$current" == "$target" ]]; then
 echo " $name ($role, $ip): already on $target"
 else
 echo " $name ($role, $ip): $current -> $target"
 pending=$((pending + 1))
 fi
 done <<< "$upgrade_order"
 echo

 if [[ "$pending" -eq 0 ]]; then
 log "Every node already matches its declared installer_image_url. Nothing to do."
 return 0
 fi

 if [[ "$AUTO_CONFIRM" != "1" ]]; then
 read -r -p "Proceed with the upgrade above? [y/N] " answer
 echo
 [[ "$answer" =~ ^[Yy]$ ]] || die "cancelled"
 fi

 local k8s_node
 while IFS='|' read -r name role ip image; do
 [[ -z "$name" ]] && continue
 target="${image##*:}"
 current=$(current_talos_tag "$ip" || echo "unknown")

 log "=== [$role] $name ($ip) ==="
 if [[ "$current" == "$target" ]]; then
 log "-- already on $target, skipping"
 continue
 fi

 point_kubectl_away_from "$ip"
 k8s_node=$(k8s_node_name_for_ip "$ip" || true)

 log "-- talosctl upgrade: $current -> $target"
 "$TALOSCTL" --endpoints "$ip" --nodes "$ip" upgrade --image "$image" --preserve --wait \
 || die "upgrade failed on $name ($ip) -- it may be in a partial state, investigate before continuing"

 wait_for_apiserver 900 || die "API server did not come back within 15m after upgrading $name"

 if [[ -n "$k8s_node" ]]; then
 log "-- waiting for $k8s_node to be Ready"
 kubectl wait --for=condition=Ready "node/$k8s_node" --timeout="$NODE_READY_TIMEOUT" \
 || die "$k8s_node did not become Ready in time"
 # Talos uncordons after its own reboot; this covers leftovers from a
 # previous failed run.
 kubectl uncordon "$k8s_node" 2>/dev/null || true
 fi

 log "-- confirming etcd is healthy on every control plane before continuing"
 etcd_healthy || die "etcd is not healthy on every control plane after upgrading $name"

 log "=== $name done ==="
 sleep "$SLEEP_BETWEEN_NODES"
 done <<< "$upgrade_order"

 point_kubectl_away_from ""
 log "Final verification"
 health_check 2m
 kubectl get nodes -o wide
 log "All nodes now match their declared installer_image_url."
}

cmd_upgrade_k8s() {
 require_bins "$TALOSCTL" kubectl jq tofu
 [[ -n "${1:-}" && -n "${2:-}" ]] || die "usage: $0 upgrade-k8s <cluster-dir> <target-version>"
 setup_cluster_access "$1"
 local target="$2"

 local current
 current=$(current_k8s_version || echo "unknown")
 log "Kubernetes: $current -> $target"

 # Can't check whether k8s_version in Terraform has already been bumped -
 # unlike installer_image_url, it isn't part of this module's own output
 # surface, and the exact variable/local name is caller-specific, so there's
 # nothing generic to grep. What IS generic: if the cluster is already
 # reporting the target version, something's already happened here, whether
 # that's this script running before, a manual upgrade, or Terraform having
 # been applied out of order - worth a harder stop than the usual confirm.
 if [[ "$current" == "$target" ]]; then
 echo "!! Cluster is already reporting $target. If Terraform's k8s_version was bumped and applied before this ran, the order is backwards - upgrade-k8s is meant to run BEFORE that, not after. Re-running this now is harmless (upgrade-k8s is idempotent), but make sure that's actually what you want."
 if [[ "$AUTO_CONFIRM" != "1" ]]; then
 read -r -p "Proceed anyway? [y/N] " answer
 echo
 [[ "$answer" =~ ^[Yy]$ ]] || die "cancelled"
 fi
 elif [[ "$AUTO_CONFIRM" != "1" ]]; then
 read -r -p "This expects k8s_version in Terraform to still be at the OLD version ($current) - confirm you haven't bumped and applied it yet. [y/N] " answer
 echo
 [[ "$answer" =~ ^[Yy]$ ]] || die "if you already bumped k8s_version in Terraform, reconcile that first (see tofu output), then rerun"
 fi

 preflight
 take_etcd_snapshot "pre-k8s-upgrade"

 log "Dry-run: upgrade-k8s --to $target"
 "$TALOSCTL" --nodes "$FIRST_CP" upgrade-k8s --to "$target" --dry-run \
 || die "dry-run failed -- if it's a compatibility error, your talosctl client may be too old for this cluster (see TALOSCTL env var)"

 if [[ "$AUTO_CONFIRM" != "1" ]]; then
 read -r -p "Dry-run above looks right - proceed with the real upgrade to $target? [y/N] " answer
 echo
 [[ "$answer" =~ ^[Yy]$ ]] || die "cancelled"
 fi

 log "Running upgrade-k8s --to $target"
 "$TALOSCTL" --nodes "$FIRST_CP" upgrade-k8s --to "$target" \
 || die "upgrade-k8s failed -- check the output above before retrying"

 log "Final verification"
 health_check "$HEALTH_WAIT_TIMEOUT"
 kubectl get nodes -o wide

 echo
 echo "Kubernetes is now at $target. Terraform still declares the old version, and will keep drifting from reality until you sync it."
 if [[ "$AUTO_CONFIRM" != "1" ]]; then
 while true; do
 read -r -p "Updated k8s_version to \"$target\" and run tofu apply for $CLUSTER_DIR? [y/N] " answer
 echo
 [[ "$answer" =~ ^[Yy]$ ]] && break
 echo "Do that now, then this is done."
 done
 fi
 log "Done."
}

# ---------------------------------------------------------------------------

case "${1:-}" in
 upgrade-talos)
 shift
 cmd_upgrade_talos "$@"
 ;;
 upgrade-k8s)
 shift
 cmd_upgrade_k8s "$@"
 ;;
 -h|--help|help|"")
 usage
 ;;
 *)
 echo "!! unknown command: $1" >&2
 usage
 exit 1
 ;;
esac

#!/usr/bin/env bash
#
# Operational tooling for a Talos Kubernetes cluster built with the modules
# in this repo (terraform/modules/talos). One file, one entry point - you
# tell it what you want done, it does it, rather than needing to know and
# fetch a different script per operation.
#
# Deliberately outside Terraform entirely. An upgrade (Talos OS or
# Kubernetes) is a multi-minute, multi-node procedure - snapshot etcd, then
# proceed node by node, health-gated between each - and that's a poor fit
# for a Terraform resource: Terraform's model is converging to a declared
# state, not running a multi-step imperative procedure, and an interrupted
# `tofu apply` mid-procedure can leave a node in an unknown state with no
# clean way to resume. This reads/writes real cluster state directly, via
# `tofu output` for the values it needs from your Terraform config.
#
# Single self-contained file on purpose, no separate sourced helper file -
# for something you're expected to curl and run, one atomic download is
# safer than several that could drift out of sync with each other, and it
# means you can read the entire thing before running it.
#
# Usage:
# ./talos.sh upgrade-talos <cluster-dir>
# ./talos.sh upgrade-k8s <cluster-dir> <target-version>
# ./talos.sh --help
#
# cluster-dir is always required: the directory you'd normally run
# `tofu apply` from for this cluster. No default - if you curled this
# script somewhere else first, the current directory is never that
# directory, so guessing would be more likely to be wrong than right.
#
# Get it:
# curl -fsSL https://raw.githubusercontent.com/hovorka-labs/iac-modules/scripts-v1.1.0/scripts/talos.sh -o talos.sh
# chmod +x talos.sh
#
# Env vars:
# TALOSCTL talosctl binary to use (default: talosctl on PATH).
# Override when operating multiple clusters/versions
# at once - upgrade-k8s requires the client to be at
# least as new as the cluster's Talos version, so an
# older PATH talosctl can fail checks a newer one
# wouldn't.
# AUTO_CONFIRM=1 skip every confirmation prompt
# SLEEP_BETWEEN_NODES seconds to pause between nodes, upgrade-talos only (default 15)
# NODE_READY_TIMEOUT kubectl wait timeout per node, upgrade-talos only (default 300s)
# HEALTH_WAIT_TIMEOUT talosctl health --wait-timeout (default 10m)
set -euo pipefail
AUTO_CONFIRM="${AUTO_CONFIRM:-0}"
SLEEP_BETWEEN_NODES="${SLEEP_BETWEEN_NODES:-15}"
NODE_READY_TIMEOUT="${NODE_READY_TIMEOUT:-300s}"
HEALTH_WAIT_TIMEOUT="${HEALTH_WAIT_TIMEOUT:-10m}"
TALOSCTL="${TALOSCTL:-talosctl}"
log() { echo -e "\n[$(date +%H:%M:%S)] $*"; }
die() { echo -e "\n!! $* -- aborting." >&2; exit 1; }
usage() {
cat <<'EOF'
talos.sh - operational tooling for a Talos Kubernetes cluster
Usage:
talos.sh upgrade-talos <cluster-dir>
Roll out the Talos OS version already declared (installer_image_url)
in your Terraform config, one node at a time. Run AFTER bumping the
version and `tofu apply`-ing it.
talos.sh upgrade-k8s <cluster-dir> <target-version>
Upgrade the Kubernetes control plane and kubelets to <target-version>
(e.g. v1.35.6), via talosctl's own upgrade-k8s. Run BEFORE touching
k8s_version in Terraform - bump and apply that afterward, to sync the
declaration with what's now actually running.
talos.sh --help
Show this message.
EOF
}
require_bins() {
local bin
for bin in "$@"; do command -v "$bin" >/dev/null || die "$bin not found"; done
[[ "$TALOSCTL" != "talosctl" ]] && log "Using talosctl override: $TALOSCTL ($("$TALOSCTL" version --client --short 2>/dev/null | tail -1))"
return 0
}
k8s_node_name_for_ip() {
kubectl get nodes -o json 2>/dev/null | jq -r --arg ip "$1" \
'.items[] | select(.status.addresses[]?.address==$ip) | .metadata.name'
}
current_talos_tag() {
"$TALOSCTL" --endpoints "$1" --nodes "$1" version --short 2>/dev/null \
| awk '/^Server:/{f=1; next} f && /Tag:/{print $2; exit}'
}
current_k8s_version() {
kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}' 2>/dev/null
}
# VIP reliability during a control-plane reboot varies by provider/network
# setup - sometimes it fails over cleanly, sometimes it doesn't - so rather
# than assume either way, kubectl is always pointed at a real node instead:
# specifically, any control plane OTHER than the one currently being
# upgraded, so asking about cluster state never depends on the one node
# that might be mid-reboot right now.
point_kubectl_away_from() {
local avoid="$1" target="$FIRST_CP" cand
for cand in "${CP_IPS[@]}"; do
if [[ "$cand" != "$avoid" ]]; then target="$cand"; break; fi
done
sed -i.bak -E "s#server: .*#server: https://${target}:6443#" "$TMP_DIR/kubeconfig"
rm -f "$TMP_DIR/kubeconfig.bak"
}
wait_for_apiserver() {
local timeout="$1" start elapsed
start=$(date +%s)
while ! kubectl get --raw='/readyz' &>/dev/null; do
elapsed=$(( $(date +%s) - start ))
(( elapsed > timeout )) && return 1
sleep 5
done
}
# talosctl health needs CP vs worker roles, or it expects every node to be
# schedulable -- false for CP nodes carrying the default NoSchedule taint.
health_check() {
"$TALOSCTL" --endpoints "$FIRST_CP" --nodes "$FIRST_CP" \
health --control-plane-nodes "$CP_CSV" ${WORKER_CSV:+--worker-nodes "$WORKER_CSV"} \
--k8s-endpoint "$FIRST_CP" --wait-timeout "$1"
}
# etcd can briefly report HEALTH ? ("Unknown") right after its container
# starts, before its first probe has run -- not the same as unhealthy, so
# this retries for a bit instead of failing on the very first check.
etcd_healthy() {
local status ok="" i
for i in $(seq 1 12); do
status=$("$TALOSCTL" --endpoints "$FIRST_CP" --nodes "$CP_CSV" service etcd 2>&1 || true)
if echo "$status" | grep -q "^HEALTH" && ! echo "$status" | grep "^HEALTH" | grep -qv "OK$"; then
ok="1"
break
fi
sleep 5
done
echo "$status"
[[ -n "$ok" ]]
}
take_etcd_snapshot() {
local label="$1"
log "Taking etcd snapshot (rollback safety net)"
mkdir -p "$CLUSTER_DIR/etcd-backup"
"$TALOSCTL" --endpoints "$FIRST_CP" --nodes "$FIRST_CP" etcd snapshot \
"$CLUSTER_DIR/etcd-backup/${label}-$(date +%Y%m%d-%H%M%S).snapshot" \
|| die "etcd snapshot failed -- not proceeding without a rollback point"
}
preflight() {
log "Pre-flight: cluster health"
health_check 2m || die "cluster is not healthy before starting -- fix this first"
log "Pre-flight: all Kubernetes nodes Ready"
local not_ready
not_ready=$(kubectl get nodes -o json | jq -r \
'.items[] | select(([.status.conditions[]? | select(.type=="Ready") | .status] | contains(["True"])) | not) | .metadata.name')
[[ -z "$not_ready" ]] || die "node(s) not Ready: $not_ready"
log "Pre-flight: etcd status"
etcd_healthy || die "etcd is not healthy before starting -- fix this first"
}
# Fetches kubeconfig/talosconfig/nodes from tofu output for $1, and derives
# CP_CSV/WORKER_CSV/CP_IPS/FIRST_CP. Sets CLUSTER_DIR/TMP_DIR and exports
# KUBECONFIG/TALOSCONFIG. Shared setup for every subcommand.
setup_cluster_access() {
CLUSTER_DIR="$1"
[[ -n "$(ls "$CLUSTER_DIR"/*.tf 2>/dev/null)" ]] || die "$CLUSTER_DIR has no .tf files"
TMP_DIR=$(mktemp -d)
trap 'rm -rf "$TMP_DIR"' EXIT
log "Fetching kubeconfig/talosconfig/nodes from tofu output ($CLUSTER_DIR)"
tofu -chdir="$CLUSTER_DIR" output -raw kubeconfig > "$TMP_DIR/kubeconfig" || die "tofu output kubeconfig failed -- has this cluster been applied?"
tofu -chdir="$CLUSTER_DIR" output -raw talosconfig > "$TMP_DIR/talosconfig" || die "tofu output talosconfig failed"
tofu -chdir="$CLUSTER_DIR" output -json nodes > "$TMP_DIR/nodes.json" || die "tofu output nodes failed -- module needs the 'nodes' output (talos-v1.2.0+)"
chmod 600 "$TMP_DIR/kubeconfig" "$TMP_DIR/talosconfig"
export KUBECONFIG="$TMP_DIR/kubeconfig" TALOSCONFIG="$TMP_DIR/talosconfig"
CP_CSV=$(jq -r '[to_entries[] | select(.value.machine_type=="controlplane") | .value.talos_api_ip] | join(",")' "$TMP_DIR/nodes.json")
WORKER_CSV=$(jq -r '[to_entries[] | select(.value.machine_type=="worker") | .value.talos_api_ip] | join(",")' "$TMP_DIR/nodes.json")
[[ -n "$CP_CSV" ]] || die "no control-plane nodes in tofu output nodes"
IFS=',' read -r -a CP_IPS <<< "$CP_CSV"
FIRST_CP="${CP_IPS[0]}"
if [[ ${#CP_IPS[@]} -lt 3 ]]; then
echo "!! Only ${#CP_IPS[@]} control-plane node(s) -- the API server (and kubectl checks) will go fully unreachable while that node reboots."
fi
point_kubectl_away_from ""
log "Cluster: $CLUSTER_DIR"
log "Control planes: $CP_CSV"
log "Workers: ${WORKER_CSV:-<none>}"
}
# ---------------------------------------------------------------------------
cmd_upgrade_talos() {
require_bins "$TALOSCTL" kubectl jq tofu
[[ -n "${1:-}" ]] || die "cluster-dir required (usage: $0 upgrade-talos <cluster-dir>)"
setup_cluster_access "$1"
if [[ "$AUTO_CONFIRM" != "1" ]]; then
echo "This only rolls out what's already declared - it doesn't bump installer_image_url or run tofu apply for you."
read -r -p "Have you already set the target installer_image_url and run tofu apply for $CLUSTER_DIR? [y/N] " answer
echo
[[ "$answer" =~ ^[Yy]$ ]] || die "bump installer_image_url and tofu apply first, then rerun"
fi
preflight
take_etcd_snapshot "pre-talos-upgrade"
# Control planes first (etcd quorum must never see two rebooting at once),
# then workers, both sorted for a stable order across runs.
local upgrade_order
upgrade_order=$(jq -r '
(to_entries | map(select(.value.machine_type=="controlplane")) | sort_by(.key)) +
(to_entries | map(select(.value.machine_type=="worker")) | sort_by(.key))
| .[] | "\(.key)|\(.value.machine_type)|\(.value.talos_api_ip)|\(.value.installer_image_url)"
' "$TMP_DIR/nodes.json")
echo
echo "Plan:"
local pending=0 name role ip image target current
while IFS='|' read -r name role ip image; do
[[ -z "$name" ]] && continue
target="${image##*:}"
current=$(current_talos_tag "$ip" || echo "unknown")
if [[ "$current" == "$target" ]]; then
echo " $name ($role, $ip): already on $target"
else
echo " $name ($role, $ip): $current -> $target"
pending=$((pending + 1))
fi
done <<< "$upgrade_order"
echo
if [[ "$pending" -eq 0 ]]; then
log "Every node already matches its declared installer_image_url. Nothing to do."
return 0
fi
if [[ "$AUTO_CONFIRM" != "1" ]]; then
read -r -p "Proceed with the upgrade above? [y/N] " answer
echo
[[ "$answer" =~ ^[Yy]$ ]] || die "cancelled"
fi
local k8s_node
while IFS='|' read -r name role ip image; do
[[ -z "$name" ]] && continue
target="${image##*:}"
current=$(current_talos_tag "$ip" || echo "unknown")
log "=== [$role] $name ($ip) ==="
if [[ "$current" == "$target" ]]; then
log "-- already on $target, skipping"
continue
fi
point_kubectl_away_from "$ip"
k8s_node=$(k8s_node_name_for_ip "$ip" || true)
log "-- talosctl upgrade: $current -> $target"
"$TALOSCTL" --endpoints "$ip" --nodes "$ip" upgrade --image "$image" --preserve --wait \
|| die "upgrade failed on $name ($ip) -- it may be in a partial state, investigate before continuing"
wait_for_apiserver 900 || die "API server did not come back within 15m after upgrading $name"
if [[ -n "$k8s_node" ]]; then
log "-- waiting for $k8s_node to be Ready"
kubectl wait --for=condition=Ready "node/$k8s_node" --timeout="$NODE_READY_TIMEOUT" \
|| die "$k8s_node did not become Ready in time"
# Talos uncordons after its own reboot; this covers leftovers from a
# previous failed run.
kubectl uncordon "$k8s_node" 2>/dev/null || true
fi
log "-- confirming etcd is healthy on every control plane before continuing"
etcd_healthy || die "etcd is not healthy on every control plane after upgrading $name"
log "=== $name done ==="
sleep "$SLEEP_BETWEEN_NODES"
done <<< "$upgrade_order"
point_kubectl_away_from ""
log "Final verification"
health_check 2m
kubectl get nodes -o wide
log "All nodes now match their declared installer_image_url."
}
cmd_upgrade_k8s() {
require_bins "$TALOSCTL" kubectl jq tofu
[[ -n "${1:-}" && -n "${2:-}" ]] || die "usage: $0 upgrade-k8s <cluster-dir> <target-version>"
setup_cluster_access "$1"
local target="$2"
local current
current=$(current_k8s_version || echo "unknown")
log "Kubernetes: $current -> $target"
# Can't check whether k8s_version in Terraform has already been bumped -
# unlike installer_image_url, it isn't part of this module's own output
# surface, and the exact variable/local name is caller-specific, so there's
# nothing generic to grep. What IS generic: if the cluster is already
# reporting the target version, something's already happened here, whether
# that's this script running before, a manual upgrade, or Terraform having
# been applied out of order - worth a harder stop than the usual confirm.
if [[ "$current" == "$target" ]]; then
echo "!! Cluster is already reporting $target. If Terraform's k8s_version was bumped and applied before this ran, the order is backwards - upgrade-k8s is meant to run BEFORE that, not after. Re-running this now is harmless (upgrade-k8s is idempotent), but make sure that's actually what you want."
if [[ "$AUTO_CONFIRM" != "1" ]]; then
read -r -p "Proceed anyway? [y/N] " answer
echo
[[ "$answer" =~ ^[Yy]$ ]] || die "cancelled"
fi
elif [[ "$AUTO_CONFIRM" != "1" ]]; then
read -r -p "This expects k8s_version in Terraform to still be at the OLD version ($current) - confirm you haven't bumped and applied it yet. [y/N] " answer
echo
[[ "$answer" =~ ^[Yy]$ ]] || die "if you already bumped k8s_version in Terraform, reconcile that first (see tofu output), then rerun"
fi
preflight
take_etcd_snapshot "pre-k8s-upgrade"
log "Dry-run: upgrade-k8s --to $target"
"$TALOSCTL" --nodes "$FIRST_CP" upgrade-k8s --to "$target" --dry-run \
|| die "dry-run failed -- if it's a compatibility error, your talosctl client may be too old for this cluster (see TALOSCTL env var)"
if [[ "$AUTO_CONFIRM" != "1" ]]; then
read -r -p "Dry-run above looks right - proceed with the real upgrade to $target? [y/N] " answer
echo
[[ "$answer" =~ ^[Yy]$ ]] || die "cancelled"
fi
log "Running upgrade-k8s --to $target"
"$TALOSCTL" --nodes "$FIRST_CP" upgrade-k8s --to "$target" \
|| die "upgrade-k8s failed -- check the output above before retrying"
log "Final verification"
health_check "$HEALTH_WAIT_TIMEOUT"
kubectl get nodes -o wide
echo
echo "Kubernetes is now at $target. Terraform still declares the old version, and will keep drifting from reality until you sync it."
if [[ "$AUTO_CONFIRM" != "1" ]]; then
while true; do
read -r -p "Updated k8s_version to \"$target\" and run tofu apply for $CLUSTER_DIR? [y/N] " answer
echo
[[ "$answer" =~ ^[Yy]$ ]] && break
echo "Do that now, then this is done."
done
fi
log "Done."
}
# ---------------------------------------------------------------------------
case "${1:-}" in
upgrade-talos)
shift
cmd_upgrade_talos "$@"
;;
upgrade-k8s)
shift
cmd_upgrade_k8s "$@"
;;
-h|--help|help|"")
usage
;;
*)
echo "!! unknown command: $1" >&2
usage
exit 1
;;
esac

Show full file (387 lines) Collapse

Just like the VM module, everything here is driven by two variables:

variable "cluster" {
 description = "Cluster-wide configuration shared by every node"
 type = object({
 # Talos cluster name, used for cluster registration and the generated client/kubeconfig
 name = string
 # Applied to every node as the topology.kubernetes.io/region label
 region = string
 # Explicit cluster endpoint (host[:port]); overrides vip and the first control plane node's IP
 endpoint = optional(string)

 vip = optional(string)
 api_server_extra_sans = optional(list(string), [])
 # Extra YAML merged into the apiServer block, e.g. extraArgs for OIDC
 api_server_config = optional(string, "")
 allow_scheduling_on_controlplanes = optional(bool, false)
 external_cloud_provider = optional(bool, false)
 disable_kube_proxy = optional(bool, false)

 pod_subnets = optional(list(string), [])
 service_subnets = optional(list(string), [])
 extra_manifests = optional(list(string), [])

 gateway_api_version = string
 })
}

variable "nodes" {
 description = "Map of nodes to configure. The map key is used as the node's identity (hostname, topology zone label unless overridden by zone)."
 type = map(object({
 # controlplane or worker
 machine_type = string
 ip = string
 # Reach the Talos API here instead of ip, e.g. when ip is a private address behind a public one
 talos_api_ip = optional(string)

 mac_address = optional(string, "")
 interface_name = optional(string, "")
 use_dhcp = optional(bool, false)
 gateway = optional(string, "")
 subnet_mask = optional(string, "")

 installer_image_url = string
 k8s_version = string

 # auto applies immediately and reboots if needed; staged/no_reboot/reboot are also valid, see the talos_machine_configuration_apply docs
 apply_mode = optional(string, "auto")

 # topology.kubernetes.io/zone label - defaults to the map key, but some
 # cloud integrations (e.g. Proxmox CSI/CCM) call the underlying provider
 # API using this value directly as a node identifier, so it has to match
 # the provider's own node name rather than whatever this node is called
 # in the nodes map.
 zone = optional(string)

 # Cloud controller manager node ID, in whatever URI format that CCM expects; leave unset on Proxmox, which has no CCM
 provider_id = optional(string)
 node_labels = optional(map(string), {})

 # Applied via a machine.nodeTaints patch, as { key = "value:Effect" }.
 node_taints = optional(map(string), {})
 }))

 validation {
 condition = alltrue([for node in var.nodes : contains(["auto", "reboot", "staged", "no_reboot"], node.apply_mode)])
 error_message = "apply_mode must be one of: auto, reboot, staged, no_reboot."
 }
}

variable "cluster" {
description = "Cluster-wide configuration shared by every node"
type = object({
# Talos cluster name, used for cluster registration and the generated client/kubeconfig
name = string
# Applied to every node as the topology.kubernetes.io/region label
region = string
# Explicit cluster endpoint (host[:port]); overrides vip and the first control plane node's IP
endpoint = optional(string)
vip = optional(string)
api_server_extra_sans = optional(list(string), [])
# Extra YAML merged into the apiServer block, e.g. extraArgs for OIDC
api_server_config = optional(string, "")
allow_scheduling_on_controlplanes = optional(bool, false)
external_cloud_provider = optional(bool, false)
disable_kube_proxy = optional(bool, false)
pod_subnets = optional(list(string), [])
service_subnets = optional(list(string), [])
extra_manifests = optional(list(string), [])
gateway_api_version = string
})
}
variable "nodes" {
description = "Map of nodes to configure. The map key is used as the node's identity (hostname, topology zone label unless overridden by zone)."
type = map(object({
# controlplane or worker
machine_type = string
ip = string
# Reach the Talos API here instead of ip, e.g. when ip is a private address behind a public one
talos_api_ip = optional(string)
mac_address = optional(string, "")
interface_name = optional(string, "")
use_dhcp = optional(bool, false)
gateway = optional(string, "")
subnet_mask = optional(string, "")
installer_image_url = string
k8s_version = string
# auto applies immediately and reboots if needed; staged/no_reboot/reboot are also valid, see the talos_machine_configuration_apply docs
apply_mode = optional(string, "auto")
# topology.kubernetes.io/zone label - defaults to the map key, but some
# cloud integrations (e.g. Proxmox CSI/CCM) call the underlying provider
# API using this value directly as a node identifier, so it has to match
# the provider's own node name rather than whatever this node is called
# in the nodes map.
zone = optional(string)
# Cloud controller manager node ID, in whatever URI format that CCM expects; leave unset on Proxmox, which has no CCM
provider_id = optional(string)
node_labels = optional(map(string), {})
# Applied via a machine.nodeTaints patch, as { key = "value:Effect" }.
node_taints = optional(map(string), {})
}))
validation {
condition = alltrue([for node in var.nodes : contains(["auto", "reboot", "staged", "no_reboot"], node.apply_mode)])
error_message = "apply_mode must be one of: auto, reboot, staged, no_reboot."
}
}

Show full file (67 lines) Collapse

The cluster variable holds the settings shared by every node in the cluster - things like the cluster name, the pod and service subnets, or whether kube-proxy should be disabled because Cilium is handling that instead. The nodes variable is a map, same idea as virtual_machines in the previous module: the key becomes the node’s identity, and the value holds everything specific to that one node, like its IP, MAC address, and whether it’s a controlplane or a worker.

One thing worth pointing out is the name and region fields under the cluster variable. The name field is the actual Talos cluster name, used for cluster registration, while the region field only ends up in a topology.kubernetes.io/region node label. The region field is important because the Proxmox CSI plugin uses it for volume topology matching, and it has to match the Proxmox cluster the VM is on, not the Talos cluster’s own name - two Talos clusters on the same physical Proxmox cluster still need to report the same region for the CSI plugin to work, so coupling it to the Talos cluster name would break that. Keeping name and region separate also means each of those two clusters can still have its own distinct name.

With the variables out of the way, most of the actual logic lives in locals.tf, which does all the prep work before main.tf ever touches a resource. talos_api_ips is a small one, but sets up a pattern I reuse a few times: it defaults to each node’s own IP, but can be overridden per node via talos_api_ip. I added this so the module also works on other cloud providers later, where a node’s private cluster IP and the address you’d actually reach its Talos API on can be different.

locals {
 control_plane_nodes = { for name, node in var.nodes : name => node if node.machine_type == "controlplane" }

 first_control_plane_name = keys(local.control_plane_nodes)[0]

 # Talos API endpoint per node — defaults to the cluster IP, but can be
 # overridden when it isn't directly reachable, e.g. a private IP behind a
 # public one on some cloud providers.
 talos_api_ips = {
 for name, node in var.nodes : name => coalesce(node.talos_api_ip, node.ip)
 }

 first_control_plane_api_ip = local.talos_api_ips[local.first_control_plane_name]

locals {
control_plane_nodes = { for name, node in var.nodes : name => node if node.machine_type == "controlplane" }
first_control_plane_name = keys(local.control_plane_nodes)[0]
# Talos API endpoint per node — defaults to the cluster IP, but can be
# overridden when it isn't directly reachable, e.g. a private IP behind a
# public one on some cloud providers.
talos_api_ips = {
for name, node in var.nodes : name => coalesce(node.talos_api_ip, node.ip)
}
first_control_plane_api_ip = local.talos_api_ips[local.first_control_plane_name]

Right after that comes control_plane_ips and worker_ips, both filtered and pulled out of talos_api_ips above. Both also get exposed as part of the nodes output, which is what talos.sh upgrade-talos uses to build its own control-planes-first, sorted-for-stability upgrade order.

 control_plane_ips = [for name, ip in local.talos_api_ips : ip if var.nodes[name].machine_type == "controlplane"]
 worker_ips = [for name, ip in local.talos_api_ips : ip if var.nodes[name].machine_type == "worker"]

 control_plane_ips = [for name, ip in local.talos_api_ips : ip if var.nodes[name].machine_type == "controlplane"]
worker_ips = [for name, ip in local.talos_api_ips : ip if var.nodes[name].machine_type == "worker"]

cluster_endpoint is the more interesting one. Every machine config needs a cluster_endpoint to be considered valid, but before the cluster exists there’s no external load balancer or DNS record pointing at it yet. So it falls back through three options: an explicit cluster.endpoint override first, then cluster.vip, and only then the first control plane node’s own IP if neither is set. A brand new single-node cluster works with nothing configured, and a proper HA setup is just a matter of setting one variable.

 # Endpoint baked into every machine config: an explicit override wins, then
 # the VIP, then just the first control plane node's IP.
 cluster_endpoint = coalesce(
 var.cluster.endpoint,
 var.cluster.vip,
 local.control_plane_nodes[local.first_control_plane_name].ip
 )

 # Endpoint baked into every machine config: an explicit override wins, then
# the VIP, then just the first control plane node's IP.
cluster_endpoint = coalesce(
var.cluster.endpoint,
var.cluster.vip,
local.control_plane_nodes[local.first_control_plane_name].ip
)

kubelet_extra_args is a small one: provider_id, when set, becomes kubelet’s --provider-id flag, so a cloud controller manager can match a node back to its cloud instance, in whatever URI format that CCM expects. I don’t use it on Proxmox since there’s no CCM here, but I want the module to also work elsewhere eventually, so the field stays in and just stays unset in this homelab.

 # provider-id identifies the node to a cloud controller manager - format is
 # provider-specific; irrelevant without one, e.g. on Proxmox.
 kubelet_extra_args = {
 for name, node in var.nodes : name => (
 node.provider_id != null ? { "provider-id" = node.provider_id } : {}
 )
 }

 # provider-id identifies the node to a cloud controller manager - format is
# provider-specific; irrelevant without one, e.g. on Proxmox.
kubelet_extra_args = {
for name, node in var.nodes : name => (
node.provider_id != null ? { "provider-id" = node.provider_id } : {}
)
}

gateway_api_manifests is just a couple of Gateway API CRD URLs baked into every cluster’s extraManifests, so they exist before Kubernetes even comes up.

The reason why I have this here is that I deploy everything with ArgoCD, so that’s one of the first things I install onto the Kubernetes cluster. However, my ArgoCD deployment creates an HTTPRoute for itself, and if the Gateway API CRDs are not in the cluster yet, the ArgoCD Helm installation fails. That’s why I decided to deploy and maintain the CRDs through OpenTofu instead.

 # Applied before Kubernetes even exists, so anything that needs these CRDs
 # during cluster bootstrap doesn't hit a chicken-and-egg problem waiting on
 # them.
 gateway_api_manifests = [
 "https://github.com/kubernetes-sigs/gateway-api/releases/download/${var.cluster.gateway_api_version}/standard-install.yaml",
 "https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/${var.cluster.gateway_api_version}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml",
 ]

 # Applied before Kubernetes even exists, so anything that needs these CRDs
# during cluster bootstrap doesn't hit a chicken-and-egg problem waiting on
# them.
gateway_api_manifests = [
"https://github.com/kubernetes-sigs/gateway-api/releases/download/${var.cluster.gateway_api_version}/standard-install.yaml",
"https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/${var.cluster.gateway_api_version}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml",
]

The last one, node_config_patches, is where all of that actually turns into a machine config, by combining a few .tftpl templates under templates/machine-config - one shared by every node, and one each for control planes and workers - plus a couple of small inline patches for things that don’t need their own template, like node_taints going straight in as a machine.nodeTaints patch when a node has any set. I could have built the templated parts inline as nested yamlencode() blocks too, but Talos machine configs get long and deeply nested fast, so having the YAML shape visible in its own file is a lot easier to read and diff.

 node_config_patches = {
 for name, node in var.nodes : name => concat(
 [
 templatefile("${path.module}/templates/machine-config/common.yaml.tftpl", {
 zone = coalesce(node.zone, name)
 region = var.cluster.region
 k8s_version = node.k8s_version
 installer_image_url = node.installer_image_url
 machine_type = node.machine_type
 disable_kube_proxy = var.cluster.disable_kube_proxy
 node_labels = node.node_labels
 pod_subnets = var.cluster.pod_subnets
 service_subnets = var.cluster.service_subnets
 }),
 ],
 length(local.kubelet_extra_args[name]) > 0 ? [
 yamlencode({
 machine = {
 kubelet = {
 extraArgs = local.kubelet_extra_args[name]
 }
 }
 })
 ] : [],
 length(node.node_taints) > 0 ? [
 yamlencode({
 machine = {
 nodeTaints = node.node_taints
 }
 })
 ] : [],
 node.machine_type == "controlplane" ? [
 templatefile("${path.module}/templates/machine-config/control-plane.yaml.tftpl", {
 vip = var.cluster.vip
 api_server_extra_sans = var.cluster.api_server_extra_sans
 api_server_config = var.cluster.api_server_config
 allow_scheduling_on_controlplanes = var.cluster.allow_scheduling_on_controlplanes
 external_cloud_provider = var.cluster.external_cloud_provider
 extra_manifests = jsonencode(concat(var.cluster.extra_manifests, local.gateway_api_manifests))
 ip = node.ip
 mac_address = lower(node.mac_address)
 interface_name = node.interface_name
 use_dhcp = node.use_dhcp
 gateway = node.gateway
 subnet_mask = node.subnet_mask
 })
 ] : [
 templatefile("${path.module}/templates/machine-config/worker.yaml.tftpl", {
 ip = node.ip
 mac_address = lower(node.mac_address)
 interface_name = node.interface_name
 use_dhcp = node.use_dhcp
 gateway = node.gateway
 subnet_mask = node.subnet_mask
 })
 ]
 )
 }

 node_config_patches = {
for name, node in var.nodes : name => concat(
[
templatefile("${path.module}/templates/machine-config/common.yaml.tftpl", {
zone = coalesce(node.zone, name)
region = var.cluster.region
k8s_version = node.k8s_version
installer_image_url = node.installer_image_url
machine_type = node.machine_type
disable_kube_proxy = var.cluster.disable_kube_proxy
node_labels = node.node_labels
pod_subnets = var.cluster.pod_subnets
service_subnets = var.cluster.service_subnets
}),
],
length(local.kubelet_extra_args[name]) > 0 ? [
yamlencode({
machine = {
kubelet = {
extraArgs = local.kubelet_extra_args[name]
}
}
})
] : [],
length(node.node_taints) > 0 ? [
yamlencode({
machine = {
nodeTaints = node.node_taints
}
})
] : [],
node.machine_type == "controlplane" ? [
templatefile("${path.module}/templates/machine-config/control-plane.yaml.tftpl", {
vip = var.cluster.vip
api_server_extra_sans = var.cluster.api_server_extra_sans
api_server_config = var.cluster.api_server_config
allow_scheduling_on_controlplanes = var.cluster.allow_scheduling_on_controlplanes
external_cloud_provider = var.cluster.external_cloud_provider
extra_manifests = jsonencode(concat(var.cluster.extra_manifests, local.gateway_api_manifests))
ip = node.ip
mac_address = lower(node.mac_address)
interface_name = node.interface_name
use_dhcp = node.use_dhcp
gateway = node.gateway
subnet_mask = node.subnet_mask
})
] : [
templatefile("${path.module}/templates/machine-config/worker.yaml.tftpl", {
ip = node.ip
mac_address = lower(node.mac_address)
interface_name = node.interface_name
use_dhcp = node.use_dhcp
gateway = node.gateway
subnet_mask = node.subnet_mask
})
]
)
}

Show full file (58 lines) Collapse

Here’s the control plane template, which is the more interesting of the two:

# Control plane specific configuration

cluster:
 allowSchedulingOnControlPlanes: ${allow_scheduling_on_controlplanes}

 apiServer:
 certSANs:
%{ if vip != null ~}
 - ${vip}
%{ endif ~}
%{ for san in api_server_extra_sans ~}
 - ${san}
%{ endfor ~}
 ${indent(4, api_server_config)}

%{ if external_cloud_provider }
 externalCloudProvider:
 enabled: true

%{ endif }
 discovery:
 enabled: true
 registries:
 service:
 disabled: false

 extraManifests: ${extra_manifests}

machine:
%{ if allow_scheduling_on_controlplanes ~}
 # Talos labels every control plane node to keep it out of external/L2
 # load balancer backend pools by default - the right call for a normal
 # HA cluster, but if this node is also meant to take workload traffic
 # (single-node or all-in-one setups), the label has to go, or nothing
 # can reach it through a LoadBalancer Service.
 nodeLabels:
 node.kubernetes.io/exclude-from-external-load-balancers:
 $patch: delete

%{ endif ~}
 network:
%{ if use_dhcp ~}
 interfaces:
 - interface: eth0
 dhcp: true
%{ else ~}
%{ if interface_name != "" ~}
 interfaces:
 - interface: ${interface_name}
 addresses:
 - ${ip}/${subnet_mask}
 dhcp: false
 routes:
 - gateway: ${gateway}
 network: 0.0.0.0/0
%{ else ~}
%{ if mac_address != "" ~}
 interfaces:
 - addresses:
 - ${ip}/${subnet_mask}
 deviceSelector:
 hardwareAddr: ${mac_address}
 dhcp: false
 routes:
 - gateway: ${gateway}
 network: 0.0.0.0/0
%{ if vip != null ~}
 vip:
 ip: ${vip}
%{ endif ~}
%{ endif ~}
%{ endif ~}
%{ endif ~}

# Control plane specific configuration
cluster:
allowSchedulingOnControlPlanes: ${allow_scheduling_on_controlplanes}
apiServer:
certSANs:
%{ if vip != null ~}
- ${vip}
%{ endif ~}
%{ for san in api_server_extra_sans ~}
- ${san}
%{ endfor ~}
${indent(4, api_server_config)}
%{ if external_cloud_provider }
externalCloudProvider:
enabled: true
%{ endif }
discovery:
enabled: true
registries:
service:
disabled: false
extraManifests: ${extra_manifests}
machine:
%{ if allow_scheduling_on_controlplanes ~}
# Talos labels every control plane node to keep it out of external/L2
# load balancer backend pools by default - the right call for a normal
# HA cluster, but if this node is also meant to take workload traffic
# (single-node or all-in-one setups), the label has to go, or nothing
# can reach it through a LoadBalancer Service.
nodeLabels:
node.kubernetes.io/exclude-from-external-load-balancers:
$patch: delete
%{ endif ~}
network:
%{ if use_dhcp ~}
interfaces:
- interface: eth0
dhcp: true
%{ else ~}
%{ if interface_name != "" ~}
interfaces:
- interface: ${interface_name}
addresses:
- ${ip}/${subnet_mask}
dhcp: false
routes:
- gateway: ${gateway}
network: 0.0.0.0/0
%{ else ~}
%{ if mac_address != "" ~}
interfaces:
- addresses:
- ${ip}/${subnet_mask}
deviceSelector:
hardwareAddr: ${mac_address}
dhcp: false
routes:
- gateway: ${gateway}
network: 0.0.0.0/0
%{ if vip != null ~}
vip:
ip: ${vip}
%{ endif ~}
%{ endif ~}
%{ endif ~}
%{ endif ~}

Show full file (73 lines) Collapse

A few fields in there are worth calling out individually. certSANs is where vip and api_server_extra_sans actually land, so the kube-apiserver’s TLS cert covers whatever address I end up hitting it through, VIP or otherwise, instead of just the node’s own IP. api_server_config gets merged in right after, using indent(4, api_server_config) so whatever raw YAML I pass in lines up correctly under apiServer: - that’s the escape hatch for things like OIDC flags I didn’t want to build a dedicated variable for.

allowSchedulingOnControlPlanes and externalCloudProvider are both plain on/off switches, off by default, on when I actually need them - a single control plane node needs scheduling allowed on itself, and any cloud provider with a CCM needs externalCloudProvider enabled for it to work, which matters most exactly in that single-node case, since it’s also the only node a LoadBalancer Service could reach. cni and the pod/service subnets aren’t here anymore, by the way - they live in the shared template now, since they’re genuinely cluster-wide settings, not something specific to control planes; cni stays hardcoded to none on purpose either way, since Talos would happily install Flannel for me, but I always replace it with Cilium afterward, so there’s no point letting Talos install a CNI just to immediately tear it back out.

Turning allowSchedulingOnControlPlanes on has a side effect worth knowing about: Talos labels every control plane node to keep it out of external and L2-announcement load balancer backend pools by default, which is the right call for a normal HA cluster where control planes aren’t meant to serve regular traffic. But if scheduling is allowed on that node - the whole point of a single-node or all-in-one cluster - that label means nothing can reach it through a LoadBalancer Service. So whenever allowSchedulingOnControlPlanes is on, the template also deletes that label via Talos’s $patch: delete syntax, which is the officially documented way to remove a value Talos’s own config generation added rather than something one of my patches set.

The network block is the same three-way fallback in both templates: use DHCP if use_dhcp is set, use a named interface if interface_name is set, or fall back to matching the interface by MAC address via deviceSelector. That last option exists because Proxmox doesn’t guarantee predictable interface names across reboots, so pinning to a MAC address is the more reliable choice in a homelab. On a control plane node, the VIP shows up a second time here too, this time assigned directly to the interface through Talos’s own keepalived integration, not just referenced in the cert.

The worker template follows the same shape, just without the control plane specific bits like the VIP or the API server config - it’s mostly just network setup and a sysctl bump for vm.max_map_count, which most memory-mapped-heavy workloads (Elasticsearch, OpenSearch, various vector databases) expect to already be raised, so I just set it cluster-wide on workers instead of chasing it down per app later.

The module finishes off with six outputs: talosconfig and kubeconfig, so I can talk to the cluster with talosctl and kubectl right away, machine_configs, in case I ever need to inspect what actually got sent to a node, and nodes - each node’s reachable IP, role, and target installer image, which is what talos.sh reads to know what to do. controlplane_ips and worker_ips are just those same IPs pre-filtered by role, for anything else that wants them without having to filter nodes itself. The first three are marked sensitive, since none of them are things I want showing up in a plan output or CI log; the rest aren’t - none of it is secret, and talos.sh needs to be able to read nodes with a plain tofu output -json.

That’s three modules covered - images, virtual machines, and now Talos itself. Next up, I’ll show how all of them wire together into an actual running cluster.

Putting It All Together

This lives in the repo as its own example, terraform/examples/talos-on-proxmox, and it’s deliberately minimal - just the three modules from this post, wired together into a cluster that actually boots. No Cilium, no Proxmox CSI, no GitOps bootstrap yet - those are all separate concerns I’m saving for future parts of this series, so this example stays focused on just standing up the infrastructure and the cluster itself.

# Step 1: Look up a Talos OS image from the Image Factory.
#
# The module queries the Image Factory to build a schematic for the
# requested extensions and returns the installer image URL plus the ISO
# URL/expected file path - it doesn't download or upload anything itself.
# Before running Step 2 for the first time, tell Proxmox to fetch the ISO
# to each node yourself, via the API's download-url endpoint (see the
# module README):
#
# curl -k -H "Authorization: PVEAPIToken=<user>@<realm>!<token-id>=<secret>" \
# -X POST "https://<proxmox-host>:8006/api2/json/nodes/<node>/storage/<datastore>/download-url" \
# --data-urlencode "content=iso" \
# --data-urlencode "filename=$(tofu output -raw iso_file_name)" \
# --data-urlencode "url=$(tofu output -raw iso_url)"
#
# Platform is "nocloud": it's what makes Talos read the static IP we hand
# it below via cloud-init, instead of waiting on DHCP.
module "talos_image" {
 source = "git::https://github.com/hovorka-labs/iac-modules.git//terraform/modules/proxmox/images/talos?ref=proxmox-talos-images-v1.2.0"

 talos_image_version = var.talos_version
 talos_image_platform = "nocloud"

 # Add Talos extensions required by the cluster nodes.
 # The full extension catalogue is at https://factory.talos.dev.
 talos_image_extensions = [
 "siderolabs/qemu-guest-agent",
 ]

 # Scope to specific nodes, or omit to target every node in the cluster.
 proxmox_nodes = var.proxmox_nodes
 proxmox_datastore = var.proxmox_datastore
}

# Step 2: Provision one Proxmox VM per Talos node.
#
# Each VM boots from the image downloaded above (attached as a cdrom); the
# actual disk starts empty and Talos installs itself onto it on first boot.
# The static IP comes from cloud-init, which the nocloud platform picks up
# before Talos even has a machine config to work from.
module "vms" {
 source = "git::https://github.com/hovorka-labs/iac-modules.git//terraform/modules/proxmox/virtual-machines?ref=proxmox-virtual-machines-v1.1.0"

 virtual_machines = local.virtual_machines
}

# Step 3: Bootstrap a Talos Kubernetes cluster on top of the VMs above.
#
# Each node's mac_address comes straight from module.vms, not a variable -
# Proxmox assigns the MAC when the VM is created, and Talos just needs to be
# told the same address so it can match the right NIC in its network config.
#
# region matters once Proxmox CSI is wired in (a future post); for now it
# just reuses the cluster name.
#
# Future step (covered in the next blog post): install Cilium. Without a
# CNI, nodes come up but nothing can actually schedule yet.
module "talos_cluster" {
 source = "git::https://github.com/hovorka-labs/iac-modules.git//terraform/modules/talos?ref=talos-v1.5.2"

 cluster = {
 name = var.talos_cluster_name
 region = var.talos_cluster_name
 gateway_api_version = var.gateway_api_version
 }

 nodes = local.talos_nodes
}

# Step 1: Look up a Talos OS image from the Image Factory.
#
# The module queries the Image Factory to build a schematic for the
# requested extensions and returns the installer image URL plus the ISO
# URL/expected file path - it doesn't download or upload anything itself.
# Before running Step 2 for the first time, tell Proxmox to fetch the ISO
# to each node yourself, via the API's download-url endpoint (see the
# module README):
#
# curl -k -H "Authorization: PVEAPIToken=<user>@<realm>!<token-id>=<secret>" \
# -X POST "https://<proxmox-host>:8006/api2/json/nodes/<node>/storage/<datastore>/download-url" \
# --data-urlencode "content=iso" \
# --data-urlencode "filename=$(tofu output -raw iso_file_name)" \
# --data-urlencode "url=$(tofu output -raw iso_url)"
#
# Platform is "nocloud": it's what makes Talos read the static IP we hand
# it below via cloud-init, instead of waiting on DHCP.
module "talos_image" {
source = "git::https://github.com/hovorka-labs/iac-modules.git//terraform/modules/proxmox/images/talos?ref=proxmox-talos-images-v1.2.0"
talos_image_version = var.talos_version
talos_image_platform = "nocloud"
# Add Talos extensions required by the cluster nodes.
# The full extension catalogue is at https://factory.talos.dev.
talos_image_extensions = [
"siderolabs/qemu-guest-agent",
]
# Scope to specific nodes, or omit to target every node in the cluster.
proxmox_nodes = var.proxmox_nodes
proxmox_datastore = var.proxmox_datastore
}
# Step 2: Provision one Proxmox VM per Talos node.
#
# Each VM boots from the image downloaded above (attached as a cdrom); the
# actual disk starts empty and Talos installs itself onto it on first boot.
# The static IP comes from cloud-init, which the nocloud platform picks up
# before Talos even has a machine config to work from.
module "vms" {
source = "git::https://github.com/hovorka-labs/iac-modules.git//terraform/modules/proxmox/virtual-machines?ref=proxmox-virtual-machines-v1.1.0"
virtual_machines = local.virtual_machines
}
# Step 3: Bootstrap a Talos Kubernetes cluster on top of the VMs above.
#
# Each node's mac_address comes straight from module.vms, not a variable -
# Proxmox assigns the MAC when the VM is created, and Talos just needs to be
# told the same address so it can match the right NIC in its network config.
#
# region matters once Proxmox CSI is wired in (a future post); for now it
# just reuses the cluster name.
#
# Future step (covered in the next blog post): install Cilium. Without a
# CNI, nodes come up but nothing can actually schedule yet.
module "talos_cluster" {
source = "git::https://github.com/hovorka-labs/iac-modules.git//terraform/modules/talos?ref=talos-v1.5.2"
cluster = {
name = var.talos_cluster_name
region = var.talos_cluster_name
gateway_api_version = var.gateway_api_version
}
nodes = local.talos_nodes
}

Show full file (68 lines) Collapse

Three steps, in order: look up the Talos image, provision a VM per node from it, then bootstrap Talos on top of the VMs. The one detail worth pointing out is how mac_address gets into the Talos node config - it’s not a variable I set anywhere, it’s read straight back out of module.vms.mac_addresses. Proxmox assigns the MAC when the VM gets created, and the Talos module just needs to be told the same address so its deviceSelector can match the right NIC. No manual MAC pinning, no coordinating two separate values by hand.

Here’s how that, and the rest of each node’s Talos-facing config, comes together in locals.tf:

 # CIDR notation Proxmox's cloud-init wants above - split it once here.
 talos_nodes = {
 for name, node in var.nodes : name => {
 machine_type = node.role
 # Proxmox CSI/CCM call the Proxmox API using this zone value directly
 # as the node name, so it has to be the real Proxmox node, not this
 # node's own identity in the nodes map.
 zone = node.proxmox_node
 ip = split("/", node.ip)[0]
 subnet_mask = split("/", node.ip)[1]
 gateway = var.network_gateway

 mac_address = module.vms.mac_addresses[name]
 installer_image_url = module.talos_image.installer_image
 k8s_version = var.k8s_version
 }
 }
}

 # CIDR notation Proxmox's cloud-init wants above - split it once here.
talos_nodes = {
for name, node in var.nodes : name => {
machine_type = node.role
# Proxmox CSI/CCM call the Proxmox API using this zone value directly
# as the node name, so it has to be the real Proxmox node, not this
# node's own identity in the nodes map.
zone = node.proxmox_node
ip = split("/", node.ip)[0]
subnet_mask = split("/", node.ip)[1]
gateway = var.network_gateway
mac_address = module.vms.mac_addresses[name]
installer_image_url = module.talos_image.installer_image
k8s_version = var.k8s_version
}
}
}

The rest is just plumbing: talos_cluster_name, k8s_version, and gateway_api_version are new variables feeding cluster.name, nodes[*].k8s_version, and cluster.gateway_api_version. region just reuses the cluster name for now, since nothing in this example actually reads it yet - that only starts to matter once Proxmox CSI gets wired in.

Running tofu apply against this gets me a Talos cluster with a working control plane and a kubeconfig/talosconfig I can pull straight out of the outputs. What it doesn’t get me yet is a cluster that can actually run anything, since there’s still no CNI installed - we will look into setting up Cilium CNI with OpenTofu for this setup in the next episode. This was a pretty extensive post, but there was a lot to cover, and I prepared a base for all the upcoming blog posts. See you at the next one!

Opentofu on Jakub Hovorka

Homelab Diary Part 4: Talos, Minus the Manual Work

Images

Virtual Machines

Talos

Putting It All Together