Homelab Diary Part 4: Talos, Minus the Manual Work
In the previous part of this series, I finally put the homelab together and installed Proxmox on each machine. In the meantime, I’ve also done some basic setup, such as updating all package repositories on each host, joining all three machines into a Proxmox cluster, and installing a Prometheus exporter to track resource usage and temperatures of each machine.
Now it’s time to finally start running things on the homelab, but to do that, I will need a good foundation. I like to do things in an organized way and make as few manual changes as possible. The reason behind it is that I want to continuously build a library of resources that I can share with other people, or just use in the future, when I need it. If I were to set up everything manually, or with scripts, it would be kind of hard to understand for anyone who is not familiar with my setup. I am also sure I would forget how everything works if I came back to it a year later. To tackle this challenge, I decided to use Infrastructure as Code with tools like OpenTofu and Ansible, and GitOps with ArgoCD, and GitHub Actions. This will allow me to build everything in a way I would do it at a real company, and it will force me to keep the whole setup clean.
Let’s start with OpenTofu. To keep the whole setup modular and reusable, I will create a monorepository, which will hold all of my OpenTofu modules. I will keep this repository public forever, and I will also handle the versioning, so existing setups relying on these modules won’t break anytime I make some changes. Anyone interested in replicating my setup, or a part of it, will then be able to reference the individual modules using the repository URL and a version tag. I will be using this repository in my homelab, which will force me to keep it up-to-date.
My first goal is to be able to create and maintain Kubernetes clusters. I am a big fan of Talos Linux, so that’s what my Kubernetes clusters will run on. Lucky for me, there is siderolabs/talos OpenTofu provider, which, from my experience, is really good. Before I can do anything, I first need to spin up the infrastructure for the clusters, which I will do on Proxmox. There are multiple Proxmox OpenTofu providers, but the best one from my experience is the bpg/proxmox, so that’s what I will be using. I will first go over all the modules, and in the end, I will show an example of how to wire them all together in a nice, scalable way.
Images
The very first module I need is the one to look up Talos images. The module is very simple - I just give it a Talos version and the extensions I need, and it asks the Talos Image Factory for the resulting installer image and ISO URLs.
locals {
# Every ISO from the Image Factory has the same file name regardless of
# version - e.g. "metal-amd64.iso" every time, with the version and
# schematic only showing up earlier in the URL, not in the file name
# itself. If we reused that name as-is, uploading a new Talos version to
# Proxmox would silently overwrite the old ISO instead of sitting next to
# it. So we build our own name here, with version and schematic baked in,
# so every upload keeps its own file.
file_name = "talos-${var.talos_image_version}-${talos_image_factory_schematic.this.id}-${var.talos_image_platform}.iso"
}
data "talos_image_factory_extensions_versions" "this" {
talos_version = var.talos_image_version
filters = {
names = var.talos_image_extensions
}
}
resource "talos_image_factory_schematic" "this" {
schematic = yamlencode(
{
customization = {
systemExtensions = {
officialExtensions = data.talos_image_factory_extensions_versions.this.extensions_info[*].name
}
}
}
)
}
data "talos_image_factory_urls" "this" {
talos_version = var.talos_image_version
schematic_id = talos_image_factory_schematic.this.id
platform = var.talos_image_platform
}
Now, I know I mentioned that I don’t like doing things manually, but there are cases where automating something introduces a rabbit hole of issues that need to be patched, and makes the whole solution complex, and fragile. Originally, I was downloading the Talos images on Proxmox nodes automatically, using this module. And that worked fine for a setup I was mainly using for experimentation, but not for running production workloads. Before diving into why it wasn’t good enough for production, I want to lay down some context.
Talos uses 2 images. The first is the ISO, loaded as a CD-ROM in Proxmox’s case, and only read once, on a VM’s very first boot, to bring up Talos in maintenance mode. The more common way to run Talos on Proxmox is importing a raw disk image directly, skipping the install step entirely - but Proxmox’s OpenTofu provider only supports that by SSHing into the node itself, since disk import isn’t something the Proxmox API exposes. I wanted a setup that only ever needed an API token, no SSH access to any host, so I went with the ISO instead.
The second is the installer image, a container image Talos pulls to actually lay itself onto the dedicated disk. That installer-image pull happens both during the first boot and every time talosctl upgrade runs later. An upgrade doesn’t re-attach the ISO, it just has the already-running Talos pull a new installer image and reinstall itself in place. So the ISO really is single-use: once that first install completes, nothing in normal operation ever needs it again.
Now to the reason why my original setup wasn’t good enough for production environments. I first realized it when my friend, Martin Hope, was telling me about issues he encountered when he tried to use similar setup in production. He was trying to use it on OpenStack, and some OpenStack deployments store the VM-to-source-image relationship in metadata, and refuse to delete an image as long as any VM still references it, even long after that VM has been upgraded via the installer image and stopped needing the original at all. Cleanup just fails, and old images pile up in state with no clean way out.
This made me look closer at my own setup to see if something similar could happen to me on Proxmox, and it turned out it could, just not in the same way. An OpenStack instance and a Proxmox VM both stop needing the original image pretty much right after they boot, so deleting it doesn’t actually break anything that’s already running - the real problem is provisioning from it again later, whether that’s a new cluster, rebuilding the current one, or just a new node joining it. My module didn’t protect against that at all. Bumping the Talos version would quietly delete the previous version’s ISO too, so every version bump wasted bandwidth re-downloading a full ISO nobody needed, and if I ever had to rebuild a node that was still declared on an older version, there was nothing left to boot it from.
I tried a couple of fixes that kept the download as a OpenTofu resource, but none of them held up cleanly, so I pulled it out of the module entirely. It now only handles the lookup, to get the installer_image, iso_url, and iso_file_name. Nothing in the module ever downloads or deletes anything on Proxmox, and it’s no longer locked to Proxmox specifically either. The same lookup works unchanged whether the VMs end up on Proxmox, OpenStack, or anywhere else.
When a new node now needs to boot, I just fetch the ISO myself and upload it through Proxmox’s own download-url API by hand. The same action can also be done in the Proxmox web UI, using the “Download from URL” button. There’s no pvesm subcommand for this, so I am using curl:
curl -k -H "Authorization: PVEAPIToken=$PROXMOX_API_TOKEN" \
-X POST "https://<proxmox-host>:8006/api2/json/nodes/<node>/storage/<datastore>/download-url" \
--data-urlencode "content=iso" \
--data-urlencode "filename=$(tofu output -raw iso_file_name)" \
--data-urlencode "url=$(tofu output -raw iso_url)"
It’s a manual step, but it’s a relatively rare one, it only comes up when provisioning a new cluster, reprovisioning the current one, or a new node joining it, never on a version bump or an upgrade. Every ordinary Talos or Kubernetes version change flows entirely through the installer image, which Talos pulls over the network on its own, Proxmox is never involved and nothing gets deleted as a side effect.
Virtual Machines
The next module I need is the one to create the VMs on Proxmox. I’ve built this module over the last 2 years, and I think it’s pretty flexible, and sufficient for all the standard use cases. The module looks like this:
resource "proxmox_virtual_environment_vm" "this" {
for_each = var.virtual_machines
node_name = each.value.node_name
name = each.key
vm_id = try(each.value.vm_id, null)
description = try(each.value.description, null)
tags = try(each.value.tags, null)
on_boot = try(each.value.on_boot, true)
machine = try(each.value.machine, null)
scsi_hardware = try(each.value.scsi_hardware, null)
bios = try(each.value.bios, null)
agent {
enabled = try(each.value.agent_enabled, false)
timeout = try(each.value.agent_timeout, "15m")
}
cpu {
cores = each.value.cpu.cores
type = try(each.value.cpu.type, null)
flags = try(each.value.cpu.flags, [""])
units = try(each.value.cpu.units, null)
}
memory {
dedicated = each.value.memory.dedicated
# Below dedicated, this enables ballooning (Proxmox can reclaim memory
# under host pressure); 0 means no ballooning, memory stays pinned at
# dedicated - the safer default for a module with no opinion on whether
# a given VM's workload tolerates being squeezed.
floating = try(each.value.memory.floating, 0)
}
dynamic "network_device" {
for_each = try(each.value.network_devices, [])
content {
bridge = network_device.value.bridge
mac_address = try(network_device.value.mac_address, null)
model = try(network_device.value.model, null)
vlan_id = try(network_device.value.vlan_id, null)
firewall = try(network_device.value.firewall, false)
}
}
dynamic "disk" {
for_each = coalesce(each.value.disks, [])
content {
datastore_id = disk.value.datastore_id
interface = coalesce(disk.value.interface, "scsi0")
iothread = try(disk.value.iothread, true)
cache = try(disk.value.cache, "writethrough")
discard = try(disk.value.discard, "on")
ssd = try(disk.value.ssd, true)
file_format = try(disk.value.file_format, "")
file_id = try(disk.value.file_id, "")
size = disk.value.size
}
}
serial_device {
device = try(each.value.serial_device.device, null)
}
# Defaults to ide3 so it doesn't collide with the cloud-init drive, which
# Proxmox always attaches on ide2.
cdrom {
file_id = try(each.value.cdrom.file_id, "none")
interface = try(each.value.cdrom.interface, "ide3")
}
boot_order = try(each.value.boot_order, [])
operating_system {
type = try(each.value.operating_system_type, "other")
}
dynamic "initialization" {
for_each = try(each.value.init, null) != null ? [each.value.init] : []
content {
datastore_id = initialization.value.datastore_id
interface = try(initialization.value.interface, null)
dynamic "dns" {
for_each = try(initialization.value.dns, null) != null ? { "dns" = initialization.value.dns } : {}
content {
servers = initialization.value.dns
}
}
ip_config {
ipv4 {
address = initialization.value.ipv4.address
gateway = initialization.value.ipv4.gateway
}
}
dynamic "user_account" {
for_each = try(initialization.value.auth, null) != null ? [initialization.value.auth] : []
content {
username = try(initialization.value.auth.username, null)
password = try(initialization.value.auth.password, null)
keys = try(user_account.value.keys, null)
}
}
}
}
dynamic "clone" {
for_each = try(each.value.clone, null) != null ? [each.value.clone] : []
content {
vm_id = clone.value.vm_id
datastore_id = try(clone.value.datastore_id, null)
node_name = try(clone.value.node_name, null)
retries = try(clone.value.retries, null)
full = try(clone.value.full, true)
}
}
dynamic "hostpci" {
for_each = coalesce(try(each.value.pci_devices, []), [])
content {
device = hostpci.value.device
mapping = try(hostpci.value.mapping, null)
pcie = try(hostpci.value.pcie, false)
rombar = try(hostpci.value.rombar, false)
xvga = try(hostpci.value.xvga, false)
}
}
}
And here are the variables:
variable "virtual_machines" {
description = "Map of VMs to create. Map key becomes the VM name."
type = map(object({
node_name = string
vm_id = optional(number)
description = optional(string)
tags = optional(list(string))
on_boot = optional(bool)
machine = optional(string)
scsi_hardware = optional(string)
bios = optional(string)
# Requires the QEMU guest agent running inside the guest OS, otherwise
# Terraform just waits out `agent_timeout` on every apply.
agent_enabled = optional(bool)
agent_timeout = optional(string)
cpu = object({
cores = number
type = optional(string)
flags = optional(list(string))
units = optional(number)
})
memory = object({
dedicated = number
# enables memory ballooning
floating = optional(number)
})
network_devices = optional(list(object({
bridge = string
mac_address = optional(string)
model = optional(string)
vlan_id = optional(string)
firewall = optional(bool)
})))
disks = optional(list(object({
datastore_id = string
interface = optional(string)
iothread = optional(bool)
cache = optional(string)
discard = optional(string)
ssd = optional(bool)
file_format = optional(string)
size = number
# source image/template to clone from
file_id = optional(string)
})))
# Defaults to ide3: ide2 is reserved for the cloud-init drive below.
cdrom = optional(object({
file_id = optional(string)
interface = optional(string)
}))
serial_device = optional(object({
device = optional(string)
}))
boot_order = optional(list(string))
operating_system_type = optional(string)
init = optional(object({
datastore_id = string
dns = optional(list(string))
ipv4 = object({
address = string
gateway = string
})
auth = optional(object({
username = string
password = optional(string)
keys = optional(list(string))
}))
}))
# Clones this VM from an existing template instead of building it fresh.
clone = optional(object({
vm_id = number
datastore_id = optional(string)
node_name = optional(string)
retries = optional(number)
full = optional(bool)
}))
# PCI/GPU passthrough. `mapping` refers to a resource mapping configured
# under Datacenter > Resource Mappings in Proxmox.
pci_devices = optional(list(object({
device = string
mapping = optional(string)
pcie = optional(bool)
rombar = optional(bool)
xvga = optional(bool)
})))
}))
}
There isn’t anything particularly exotic in this module, most of the things are just standard Proxmox VM attributes but there are a few things I want to point out. First is the virtual_machines variable, which, as you might see, is the only variable in this module, and it has a lot of different fields nested in it. This approach allows me to only initiate the module once, no matter if I want to create 1 VM or 100 VMs. This is especially useful for K8s clusters, where all VMs have very similar attributes that would otherwise have to be redefined for every single one.
Second is the cdrom block, which defaults to interface ide3 instead of the more obvious ide2. Reasoning behind this one is simple: Proxmox always reserves ide2 for the cloud-init drive whenever cloud-init is enabled, so if the cdrom also defaulted to ide2, it would just collide with it. ide3 is simply the next slot that’s actually free.
Talos
If you are familiar with the Talos cluster creation process, this is basically it, just transformed from individual talosctl commands to OpenTofu code. The module opens with the talos_machine_secrets resource, which generates the secrets shared by the whole cluster. Then there is talos_client_configuration, which generates a talosconfig for the whole cluster, and talos_machine_configuration, which generates a machine config for each node. talos_machine_configuration_apply applies that config - to every node, control planes included.
resource "talos_machine_secrets" "this" {}
data "talos_client_configuration" "this" {
cluster_name = var.cluster.name
client_configuration = talos_machine_secrets.this.client_configuration
nodes = [for ip in local.talos_api_ips : ip]
endpoints = local.control_plane_ips
}
data "talos_machine_configuration" "this" {
for_each = var.nodes
cluster_name = var.cluster.name
cluster_endpoint = "https://${local.cluster_endpoint}:6443"
machine_type = each.value.machine_type
machine_secrets = talos_machine_secrets.this.machine_secrets
config_patches = local.node_config_patches[each.key]
}
resource "talos_machine_configuration_apply" "this" {
for_each = var.nodes
node = local.talos_api_ips[each.key]
apply_mode = each.value.apply_mode
client_configuration = talos_machine_secrets.this.client_configuration
machine_configuration_input = data.talos_machine_configuration.this[each.key].machine_configuration
}Once every node has its config applied, talos_machine_bootstrap runs the cluster bootstrap against the first control plane node - specifically, whichever control plane’s node name sorts first alphabetically. That’s a OpenTofu quirk worth knowing: keys() always returns map keys sorted, not in the order you wrote them, so the “first” control plane is picked by name, not by position in the nodes map.
resource "talos_machine_bootstrap" "this" {
depends_on = [
talos_machine_configuration_apply.this,
]
node = local.first_control_plane_api_ip
client_configuration = talos_machine_secrets.this.client_configuration
}Once the cluster is bootstrapped, I confirm that it becomes healthy using the talos_cluster_health data source, and retrieve the kubeconfig using the talos_cluster_kubeconfig resource. That’s it for main.tf - upgrades are handled entirely outside OpenTofu, more on that at the end of this section.
Just like the VM module, everything here is driven by two variables:
variable "cluster" {
description = "Cluster-wide configuration shared by every node"
type = object({
# Talos cluster name, used for cluster registration and the generated client/kubeconfig
name = string
# Applied to every node as the topology.kubernetes.io/region label
region = string
# Explicit cluster endpoint (host[:port]); overrides vip and the first control plane node's IP
endpoint = optional(string)
vip = optional(string)
api_server_extra_sans = optional(list(string), [])
# Extra YAML merged into the apiServer block, e.g. extraArgs for OIDC
api_server_config = optional(string, "")
allow_scheduling_on_controlplanes = optional(bool, false)
external_cloud_provider = optional(bool, false)
disable_kube_proxy = optional(bool, false)
pod_subnets = optional(list(string), [])
service_subnets = optional(list(string), [])
extra_manifests = optional(list(string), [])
gateway_api_version = string
})
}
variable "nodes" {
description = "Map of nodes to configure. The map key is used as the node's identity (hostname, topology zone label unless overridden by zone)."
type = map(object({
# controlplane or worker
machine_type = string
ip = string
# Reach the Talos API here instead of ip, e.g. when ip is a private address behind a public one
talos_api_ip = optional(string)
mac_address = optional(string, "")
interface_name = optional(string, "")
use_dhcp = optional(bool, false)
gateway = optional(string, "")
subnet_mask = optional(string, "")
installer_image_url = string
k8s_version = string
# auto applies immediately and reboots if needed; staged/no_reboot/reboot are also valid, see the talos_machine_configuration_apply docs
apply_mode = optional(string, "auto")
# topology.kubernetes.io/zone label - defaults to the map key, but some
# cloud integrations (e.g. Proxmox CSI/CCM) call the underlying provider
# API using this value directly as a node identifier, so it has to match
# the provider's own node name rather than whatever this node is called
# in the nodes map.
zone = optional(string)
# Cloud controller manager node ID, in whatever URI format that CCM expects; leave unset on Proxmox, which has no CCM
provider_id = optional(string)
node_labels = optional(map(string), {})
# Applied via a machine.nodeTaints patch, as { key = "value:Effect" }.
node_taints = optional(map(string), {})
}))
validation {
condition = alltrue([for node in var.nodes : contains(["auto", "reboot", "staged", "no_reboot"], node.apply_mode)])
error_message = "apply_mode must be one of: auto, reboot, staged, no_reboot."
}
}
The cluster variable holds the settings shared by every node in the cluster - things like the cluster name, the pod and service subnets, or whether kube-proxy should be disabled because Cilium is handling that instead. The nodes variable is a map, same idea as virtual_machines in the previous module: the key becomes the node’s identity, and the value holds everything specific to that one node, like its IP, MAC address, and whether it’s a controlplane or a worker.
Quick note on the name and region fields under the cluster variable. The name field is the actual Talos cluster name, used for cluster registration, while the region field only ends up in a topology.kubernetes.io/region node label. The region field is important because the Proxmox CSI plugin uses it for volume topology matching, and it has to match the Proxmox cluster the VM is on, not the Talos cluster’s own name - two Talos clusters on the same physical Proxmox cluster still need to report the same region for the CSI plugin to work, so coupling it to the Talos cluster name would break that. Keeping name and region separate also means each of those two clusters can still have its own distinct name.
With the variables out of the way, most of the actual logic lives in locals.tf. talos_api_ips is a small one, but sets up a pattern I reuse a few times: it defaults to each node’s own IP, but can be overridden per node via talos_api_ip. I added this so the module also works on other cloud providers later, where a node’s private cluster IP and the address you’d actually reach its Talos API on can be different.
locals {
control_plane_nodes = {
for name, node in var.nodes : name => node if node.machine_type == "controlplane"
}
first_control_plane_name = keys(local.control_plane_nodes)[0]
# Talos API endpoint per node — defaults to the cluster IP, but can be
# overridden when it isn't directly reachable, e.g. a private IP behind a
# public one on some cloud providers.
talos_api_ips = {
for name, node in var.nodes : name => coalesce(node.talos_api_ip, node.ip)
}
first_control_plane_api_ip = local.talos_api_ips[local.first_control_plane_name]Right after that comes control_plane_ips and worker_ips, both filtered and pulled out of talos_api_ips above, and exposed as their own outputs, controlplane_ips and worker_ips.
control_plane_ips = [for name, ip in local.talos_api_ips : ip if var.nodes[name].machine_type == "controlplane"]
worker_ips = [for name, ip in local.talos_api_ips : ip if var.nodes[name].machine_type == "worker"]cluster_endpoint is the more interesting one. Every machine config needs a cluster_endpoint to be considered valid, but before the cluster exists there’s no external load balancer or DNS record pointing at it yet. So it falls back through three options: an explicit cluster.endpoint override first, then cluster.vip, and only then the first control plane node’s own IP if neither is set. A brand new single-node cluster works with nothing configured, and a proper HA setup is just a matter of setting one variable.
# Endpoint baked into every machine config: an explicit override wins, then
# the VIP, then just the first control plane node's IP.
cluster_endpoint = coalesce(
var.cluster.endpoint,
var.cluster.vip,
local.control_plane_nodes[local.first_control_plane_name].ip
)kubelet_extra_args is a small one: provider_id, when set, becomes kubelet’s --provider-id flag, so a cloud controller manager can match a node back to its cloud instance, in whatever URI format that CCM expects. I don’t use it on Proxmox since there’s no CCM here, but I want the module to also work elsewhere eventually, so the field stays in and just stays unset in this homelab.
# provider-id identifies the node to a cloud controller manager - format is
# provider-specific; irrelevant without one, e.g. on Proxmox.
kubelet_extra_args = {
for name, node in var.nodes : name => (
node.provider_id != null ? { "provider-id" = node.provider_id } : {}
)
}gateway_api_manifests is just a couple of Gateway API CRD URLs baked into every cluster’s extraManifests, so they exist before Kubernetes even comes up.
The reason why I have this here is that I deploy everything with ArgoCD, so that’s one of the first things I install onto the Kubernetes cluster. However, my ArgoCD deployment creates an HTTPRoute for itself, and if the Gateway API CRDs are not in the cluster yet, the ArgoCD Helm installation fails. That’s why I decided to deploy and maintain the CRDs through OpenTofu instead.
# Applied before Kubernetes even exists, so anything that needs these CRDs
# during cluster bootstrap doesn't hit a chicken-and-egg problem waiting on
# them.
gateway_api_manifests = [
"https://github.com/kubernetes-sigs/gateway-api/releases/download/${var.cluster.gateway_api_version}/standard-install.yaml",
"https://raw.githubusercontent.com/kubernetes-sigs/gateway-api/${var.cluster.gateway_api_version}/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml",
]The last one, node_config_patches, is where all of that actually turns into a machine config, by combining a few .tftpl templates under templates/machine-config - one shared by every node, and one each for control planes and workers - plus a couple of small inline patches for things that don’t need their own template, like node_taints going straight in as a machine.nodeTaints patch when a node has any set. I could have built the templated parts inline as nested yamlencode() blocks too, but Talos machine configs get long and deeply nested fast, so having the YAML shape visible in its own file is a lot easier to read and diff.
node_config_patches = {
for name, node in var.nodes : name => concat(
[
templatefile("${path.module}/templates/machine-config/common.yaml.tftpl", {
zone = coalesce(node.zone, name)
region = var.cluster.region
k8s_version = node.k8s_version
installer_image_url = node.installer_image_url
machine_type = node.machine_type
disable_kube_proxy = var.cluster.disable_kube_proxy
node_labels = node.node_labels
pod_subnets = var.cluster.pod_subnets
service_subnets = var.cluster.service_subnets
}),
],
length(local.kubelet_extra_args[name]) > 0 ? [
yamlencode({
machine = {
kubelet = {
extraArgs = local.kubelet_extra_args[name]
}
}
})
] : [],
length(node.node_taints) > 0 ? [
yamlencode({
machine = {
nodeTaints = node.node_taints
}
})
] : [],
node.machine_type == "controlplane" ? [
templatefile("${path.module}/templates/machine-config/control-plane.yaml.tftpl", {
vip = var.cluster.vip
api_server_extra_sans = var.cluster.api_server_extra_sans
api_server_config = var.cluster.api_server_config
allow_scheduling_on_controlplanes = var.cluster.allow_scheduling_on_controlplanes
external_cloud_provider = var.cluster.external_cloud_provider
extra_manifests = jsonencode(concat(var.cluster.extra_manifests, local.gateway_api_manifests))
ip = node.ip
mac_address = lower(node.mac_address)
interface_name = node.interface_name
use_dhcp = node.use_dhcp
gateway = node.gateway
subnet_mask = node.subnet_mask
})
] : [
templatefile("${path.module}/templates/machine-config/worker.yaml.tftpl", {
ip = node.ip
mac_address = lower(node.mac_address)
interface_name = node.interface_name
use_dhcp = node.use_dhcp
gateway = node.gateway
subnet_mask = node.subnet_mask
})
]
)
}Here’s the shared template first, since every node gets it:
# Configuration shared by every node in the cluster
cluster:
apiServer:
image: registry.k8s.io/kube-apiserver:${k8s_version}
controllerManager:
image: registry.k8s.io/kube-controller-manager:${k8s_version}
proxy:
image: registry.k8s.io/kube-proxy:${k8s_version}
%{ if disable_kube_proxy }
disabled: true
%{ endif }
scheduler:
image: registry.k8s.io/kube-scheduler:${k8s_version}
# Cluster-wide, so set identically on every node rather than just control
# planes - Talos validates this stays consistent across the cluster.
network:
cni:
name: none # CNI is installed separately
%{ if length(pod_subnets) > 0 }
podSubnets:
%{ for cidr in pod_subnets }
- ${cidr}
%{ endfor }
%{ endif }
%{ if length(service_subnets) > 0 }
serviceSubnets:
%{ for cidr in service_subnets }
- ${cidr}
%{ endfor }
%{ endif }
machine:
install:
image: ${installer_image_url}
kubelet:
image: ghcr.io/siderolabs/kubelet:${k8s_version}
# Topology labels, plus any extra per-node labels (e.g. for dedicated worker pools)
nodeLabels:
topology.kubernetes.io/region: ${region}
topology.kubernetes.io/zone: ${zone}
%{ for key, value in node_labels ~}
${key}: ${value}
%{ endfor ~}
type: ${machine_type}
k8s_version gets baked into every control plane component’s image tag individually - apiServer, controllerManager, proxy, scheduler, kubelet - instead of one central version field, since that’s just how Talos expects it. disable_kube_proxy is a plain conditional right here too, tying back to the cluster variable - when it’s on, kube-proxy just doesn’t get deployed, since Cilium replaces it.
cni is hardcoded to none, and podSubnets/serviceSubnets are set here too, both cluster-wide on purpose - Talos actually validates the network config stays identical across every node, not just control planes, so none of this could live in a role-specific template even if I wanted it to.
machine.install.image is installer_image_url - the installer image from earlier, the same container image Talos pulls to lay itself onto disk. It’s also the value I bump to trigger a Talos OS upgrade later, more on that in Upgrades below. nodeLabels is where region and zone end up too, as topology.kubernetes.io/region and topology.kubernetes.io/zone, plus whatever’s in node_labels for anything more specific, like a dedicated worker pool.
Next is the control plane template, still the more interesting of the two role-specific ones:
# Control plane specific configuration
cluster:
allowSchedulingOnControlPlanes: ${allow_scheduling_on_controlplanes}
apiServer:
certSANs:
%{ if vip != null ~}
- ${vip}
%{ endif ~}
%{ for san in api_server_extra_sans ~}
- ${san}
%{ endfor ~}
${indent(4, api_server_config)}
%{ if external_cloud_provider }
externalCloudProvider:
enabled: true
%{ endif }
discovery:
enabled: true
registries:
service:
disabled: false
extraManifests: ${extra_manifests}
machine:
%{ if allow_scheduling_on_controlplanes ~}
# Talos labels every control plane node to keep it out of external/L2
# load balancer backend pools by default - the right call for a normal
# HA cluster, but if this node is also meant to take workload traffic
# (single-node or all-in-one setups), the label has to go, or nothing
# can reach it through a LoadBalancer Service.
nodeLabels:
node.kubernetes.io/exclude-from-external-load-balancers:
$patch: delete
%{ endif ~}
network:
%{ if use_dhcp ~}
interfaces:
- interface: eth0
dhcp: true
%{ else ~}
%{ if interface_name != "" ~}
interfaces:
- interface: ${interface_name}
addresses:
- ${ip}/${subnet_mask}
dhcp: false
routes:
- gateway: ${gateway}
network: 0.0.0.0/0
%{ else ~}
%{ if mac_address != "" ~}
interfaces:
- addresses:
- ${ip}/${subnet_mask}
deviceSelector:
hardwareAddr: ${mac_address}
dhcp: false
routes:
- gateway: ${gateway}
network: 0.0.0.0/0
%{ if vip != null ~}
vip:
ip: ${vip}
%{ endif ~}
%{ endif ~}
%{ endif ~}
%{ endif ~}
There are a few fields I want to go over in more detail.. certSANs is where vip and api_server_extra_sans land, so the kube-apiserver’s TLS cert covers whatever address I actually hit it through, VIP or otherwise, not just the node’s own IP. api_server_config gets merged in right after, using indent(4, api_server_config) so whatever raw YAML I pass in lines up correctly under apiServer: - that’s the escape hatch for things like OIDC flags I didn’t want a dedicated variable for.
allowSchedulingOnControlPlanes and externalCloudProvider are both plain on/off switches, off by default, on when I actually need them. A single control-plane node needs scheduling allowed on itself, and any cloud provider with a CCM needs externalCloudProvider on to work - which matters most in that single-node case, since it’s also the only node a LoadBalancer Service could ever reach.
Turning allowSchedulingOnControlPlanes on has a side effect I had to work around. Talos labels every control plane node to keep it out of external and L2-announcement load balancer backend pools by default, which makes sense for a normal HA cluster where control planes don’t serve regular traffic. But if scheduling is allowed on that node - the whole point of a single-node or all-in-one cluster - that label means nothing can reach it through a LoadBalancer Service. So whenever allowSchedulingOnControlPlanes is on, the template also deletes that label, via Talos’s $patch: delete syntax - the officially documented way to remove a value Talos’s own config generation added, rather than something one of my patches set.
The network block is the same three-way fallback in both templates: use DHCP if use_dhcp is set, use a named interface if interface_name is set, or fall back to matching the interface by MAC address via deviceSelector. That last option exists because Proxmox doesn’t guarantee predictable interface names across reboots, so pinning to a MAC address is the more reliable choice in a homelab. On a control plane node, the VIP shows up a second time here too, this time assigned directly to the interface through Talos’s own keepalived integration, not just referenced in the cert.
The worker template follows the same shape, just without the control plane specific bits like the VIP or the API server config - it’s mostly just network setup and a sysctl bump for vm.max_map_count, which most memory-mapped-heavy workloads (Elasticsearch, OpenSearch, various vector databases) expect to already be raised, so I just set it cluster-wide on workers instead of chasing it down per app later.
The module finishes off with six outputs: talosconfig and kubeconfig, so I can talk to the cluster with talosctl and kubectl right away, machine_configs, in case I ever need to inspect what actually got sent to a node, and nodes - each node’s reachable IP, role, and target installer image, which is what drives the upgrade tooling covered in Upgrades below. controlplane_ips and worker_ips are just those same IPs pre-filtered by role, for anything else that wants them without having to filter nodes itself. The first three are marked sensitive, since none of them are things I want showing up in a plan output or CI log; the rest aren’t - none of it is secret, and the upgrade tooling needs to be able to read nodes with a plain tofu output -json.
Upgrades
Both Talos OS and Kubernetes upgrades are handled entirely outside main.tf, by scripts/talos.sh, which doesn’t even live inside this module.
An upgrade takes several minutes and touches every node - snapshot etcd, then go node by node, checking health between each one. That’s not something a OpenTofu resource is good at. OpenTofu is built to compare what you declared to what’s actually there and fix the difference, not to run a multi-step procedure like this, and doing that through a local-exec provisioner means the only way to abort cleanly is to just not interrupt tofu apply for several minutes. That’s fragile: if the apply gets interrupted, a node can end up in an unknown state with no clean way to recover.
So OpenTofu only owns declaring each node’s target installer_image_url, exposed through a nodes output along with each node’s reachable IP and role. Actually rolling that out to the cluster is scripts/talos.sh’s job. It lives at the repo root instead of inside this module, since it isn’t OpenTofu and doesn’t need fetching through a module source. It’s also a single self-contained file on purpose, meant to be curl’d and run directly, rather than several files that could drift out of sync with each other.
One dispatcher, two subcommands, so there’s one thing to remember instead of a script per operation. talos.sh upgrade-talos <cluster-dir> reads the declared state via tofu output and rolls the real cluster to match it, one node at a time - control planes first, then workers, both sorted for a stable order across runs. talos.sh upgrade-k8s <cluster-dir> <target-version> does the same for Kubernetes, driving talosctl upgrade-k8s, which already sequences its own rollout across every node, so there’s no per-node loop to write for that one.
Both only ever run against a cluster that’s already bootstrapped and running, so they can afford to be thorough. They snapshot etcd before touching anything, check both Talos and Kubernetes health before starting, and avoid the cluster’s VIP the whole time - VIP reliability during a control-plane reboot depends on the provider and network setup, so instead of assuming either way, every check is pinned to a specific node, never the one currently being upgraded. upgrade-talos also waits for each node to report Ready in Kubernetes and uncordons it afterward, just as insurance in case a previous failed run left it cordoned - Talos already uncordons automatically after its own reboot, so this is just a safety net.
upgrade-talos and upgrade-k8s want things done in a different order.
upgrade-talos reads its target version straight from OpenTofu’s installer_image_url, so OpenTofu has to go first: bump installer_image_url, tofu apply, then run the script to actually roll it out.
upgrade-k8s works the other way around. It still pulls kubeconfig, talosconfig, and the node list from OpenTofu to reach the cluster at all, same as upgrade-talos - what it doesn’t do is read a target version from OpenTofu. You pass that directly on the command line instead, since talosctl upgrade-k8s doesn’t care what OpenTofu thinks the version is. So the order flips: run the script against the real cluster first, confirm it worked, and only then bump k8s_version in OpenTofu and apply, just to keep the declared version in sync with what’s actually running.
Because of that, upgrade-k8s can’t just check whether OpenTofu’s side was already updated - k8s_version isn’t exposed as an output the way installer_image_url is, and the variable name isn’t even fixed, it depends on whoever’s calling the module. So instead it checks the cluster’s actual current version against the target I gave it. If they already match, that’s suspicious: either this already ran, or k8s_version got bumped and applied before the script ran, which is the wrong order. Either way, it stops and asks instead of just proceeding.
curl -fsSL https://raw.githubusercontent.com/hovorka-labs/iac-modules/f4f58a1acecfe2c56ddf1e91776c017ec873e3f4/scripts/talos.sh -o talos.sh
chmod +x talos.sh
#!/usr/bin/env bash
#
# Operational tooling for a Talos Kubernetes cluster built with the modules
# in this repo (terraform/modules/talos). One file, one entry point - you
# tell it what you want done, it does it, rather than needing to know and
# fetch a different script per operation.
#
# Deliberately outside Terraform entirely. An upgrade (Talos OS or
# Kubernetes) is a multi-minute, multi-node procedure - snapshot etcd, then
# proceed node by node, health-gated between each - and that's a poor fit
# for a Terraform resource: Terraform's model is converging to a declared
# state, not running a multi-step imperative procedure, and an interrupted
# apply mid-procedure can leave a node in an unknown state with no clean
# way to resume. This reads/writes real cluster state directly, via an
# `output` command for the values it needs from your Terraform config -
# works with either OpenTofu or Terraform itself, see the TF_BIN env var below.
#
# Single self-contained file on purpose, no separate sourced helper file -
# for something you're expected to curl and run, one atomic download is
# safer than several that could drift out of sync with each other, and it
# means you can read the entire thing before running it.
#
# Usage:
# ./talos.sh upgrade-talos <cluster-dir>
# ./talos.sh upgrade-k8s <cluster-dir> <target-version>
# ./talos.sh --help
#
# cluster-dir is always required: the directory you'd normally run
# an apply from for this cluster. No default - if you curled this
# script somewhere else first, the current directory is never that
# directory, so guessing would be more likely to be wrong than right.
#
# Get it:
# curl -fsSL https://raw.githubusercontent.com/hovorka-labs/iac-modules/main/scripts/talos.sh -o talos.sh
# chmod +x talos.sh
#
# Env vars:
# TALOSCTL talosctl binary to use (default: talosctl on PATH).
# Override when operating multiple clusters/versions
# at once - upgrade-k8s requires the client to be at
# least as new as the cluster's Talos version, so an
# older PATH talosctl can fail checks a newer one
# wouldn't.
# TF_BIN tofu binary to use (default: tofu on PATH). Set
# TF_BIN=terraform if this cluster is managed with
# Terraform instead of OpenTofu - everything else
# works unchanged either way.
# AUTO_CONFIRM=1 skip every confirmation prompt
# SLEEP_BETWEEN_NODES seconds to pause between nodes, upgrade-talos only (default 15)
# NODE_READY_TIMEOUT kubectl wait timeout per node, upgrade-talos only (default 300s)
# HEALTH_WAIT_TIMEOUT talosctl health --wait-timeout (default 10m)
set -euo pipefail
AUTO_CONFIRM="${AUTO_CONFIRM:-0}"
SLEEP_BETWEEN_NODES="${SLEEP_BETWEEN_NODES:-15}"
NODE_READY_TIMEOUT="${NODE_READY_TIMEOUT:-300s}"
HEALTH_WAIT_TIMEOUT="${HEALTH_WAIT_TIMEOUT:-10m}"
TALOSCTL="${TALOSCTL:-talosctl}"
TF_BIN="${TF_BIN:-tofu}"
log() { echo -e "\n[$(date +%H:%M:%S)] $*"; }
die() { echo -e "\n!! $* -- aborting." >&2; exit 1; }
usage() {
cat <<'EOF'
talos.sh - operational tooling for a Talos Kubernetes cluster
Usage:
talos.sh upgrade-talos <cluster-dir>
Roll out the Talos OS version already declared (installer_image_url)
in your Terraform config, one node at a time. Run AFTER bumping the
version and applying it.
talos.sh upgrade-k8s <cluster-dir> <target-version>
Upgrade the Kubernetes control plane and kubelets to <target-version>
(e.g. v1.35.6), via talosctl's own upgrade-k8s. Run BEFORE touching
k8s_version in Terraform - bump and apply that afterward, to sync the
declaration with what's now actually running.
talos.sh --help
Show this message.
EOF
}
require_bins() {
local bin
for bin in "$@"; do command -v "$bin" >/dev/null || die "$bin not found"; done
[[ "$TALOSCTL" != "talosctl" ]] && log "Using talosctl override: $TALOSCTL ($("$TALOSCTL" version --client --short 2>/dev/null | tail -1))"
return 0
}
k8s_node_name_for_ip() {
kubectl get nodes -o json 2>/dev/null | jq -r --arg ip "$1" \
'.items[] | select(.status.addresses[]?.address==$ip) | .metadata.name'
}
current_talos_tag() {
"$TALOSCTL" --endpoints "$1" --nodes "$1" version --short 2>/dev/null \
| awk '/^Server:/{f=1; next} f && /Tag:/{print $2; exit}'
}
current_k8s_version() {
kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}' 2>/dev/null
}
# VIP reliability during a control-plane reboot varies by provider/network
# setup - sometimes it fails over cleanly, sometimes it doesn't - so rather
# than assume either way, kubectl is always pointed at a real node instead:
# specifically, any control plane OTHER than the one currently being
# upgraded, so asking about cluster state never depends on the one node
# that might be mid-reboot right now.
point_kubectl_away_from() {
local avoid="$1" target="$FIRST_CP" cand
for cand in "${CP_IPS[@]}"; do
if [[ "$cand" != "$avoid" ]]; then target="$cand"; break; fi
done
sed -i.bak -E "s#server: .*#server: https://${target}:6443#" "$TMP_DIR/kubeconfig"
rm -f "$TMP_DIR/kubeconfig.bak"
}
wait_for_apiserver() {
local timeout="$1" start elapsed
start=$(date +%s)
while ! kubectl get --raw='/readyz' &>/dev/null; do
elapsed=$(( $(date +%s) - start ))
(( elapsed > timeout )) && return 1
sleep 5
done
}
# talosctl health needs CP vs worker roles, or it expects every node to be
# schedulable -- false for CP nodes carrying the default NoSchedule taint.
health_check() {
"$TALOSCTL" --endpoints "$FIRST_CP" --nodes "$FIRST_CP" \
health --control-plane-nodes "$CP_CSV" ${WORKER_CSV:+--worker-nodes "$WORKER_CSV"} \
--k8s-endpoint "$FIRST_CP" --wait-timeout "$1"
}
# etcd can briefly report HEALTH ? ("Unknown") right after its container
# starts, before its first probe has run -- not the same as unhealthy, so
# this retries for a bit instead of failing on the very first check.
etcd_healthy() {
local status ok="" i
for i in $(seq 1 12); do
status=$("$TALOSCTL" --endpoints "$FIRST_CP" --nodes "$CP_CSV" service etcd 2>&1 || true)
if echo "$status" | grep -q "^HEALTH" && ! echo "$status" | grep "^HEALTH" | grep -qv "OK$"; then
ok="1"
break
fi
sleep 5
done
echo "$status"
[[ -n "$ok" ]]
}
take_etcd_snapshot() {
local label="$1"
log "Taking etcd snapshot (rollback safety net)"
mkdir -p "$CLUSTER_DIR/etcd-backup"
"$TALOSCTL" --endpoints "$FIRST_CP" --nodes "$FIRST_CP" etcd snapshot \
"$CLUSTER_DIR/etcd-backup/${label}-$(date +%Y%m%d-%H%M%S).snapshot" \
|| die "etcd snapshot failed -- not proceeding without a rollback point"
}
preflight() {
log "Pre-flight: cluster health"
health_check 2m || die "cluster is not healthy before starting -- fix this first"
log "Pre-flight: all Kubernetes nodes Ready"
local not_ready
not_ready=$(kubectl get nodes -o json | jq -r \
'.items[] | select(([.status.conditions[]? | select(.type=="Ready") | .status] | contains(["True"])) | not) | .metadata.name')
[[ -z "$not_ready" ]] || die "node(s) not Ready: $not_ready"
log "Pre-flight: etcd status"
etcd_healthy || die "etcd is not healthy before starting -- fix this first"
}
# Fetches kubeconfig/talosconfig/nodes from $TF_BIN output for $1, and derives
# CP_CSV/WORKER_CSV/CP_IPS/FIRST_CP. Sets CLUSTER_DIR/TMP_DIR and exports
# KUBECONFIG/TALOSCONFIG. Shared setup for every subcommand.
setup_cluster_access() {
CLUSTER_DIR="$1"
[[ -n "$(ls "$CLUSTER_DIR"/*.tf 2>/dev/null)" ]] || die "$CLUSTER_DIR has no .tf files"
TMP_DIR=$(mktemp -d)
trap 'rm -rf "$TMP_DIR"' EXIT
log "Fetching kubeconfig/talosconfig/nodes from $TF_BIN output ($CLUSTER_DIR)"
"$TF_BIN" -chdir="$CLUSTER_DIR" output -raw kubeconfig > "$TMP_DIR/kubeconfig" || die "$TF_BIN output kubeconfig failed -- has this cluster been applied?"
"$TF_BIN" -chdir="$CLUSTER_DIR" output -raw talosconfig > "$TMP_DIR/talosconfig" || die "$TF_BIN output talosconfig failed"
"$TF_BIN" -chdir="$CLUSTER_DIR" output -json nodes > "$TMP_DIR/nodes.json" || die "$TF_BIN output nodes failed -- module needs the 'nodes' output"
chmod 600 "$TMP_DIR/kubeconfig" "$TMP_DIR/talosconfig"
export KUBECONFIG="$TMP_DIR/kubeconfig" TALOSCONFIG="$TMP_DIR/talosconfig"
CP_CSV=$(jq -r '[to_entries[] | select(.value.machine_type=="controlplane") | .value.talos_api_ip] | join(",")' "$TMP_DIR/nodes.json")
WORKER_CSV=$(jq -r '[to_entries[] | select(.value.machine_type=="worker") | .value.talos_api_ip] | join(",")' "$TMP_DIR/nodes.json")
[[ -n "$CP_CSV" ]] || die "no control-plane nodes in $TF_BIN output nodes"
IFS=',' read -r -a CP_IPS <<< "$CP_CSV"
FIRST_CP="${CP_IPS[0]}"
if [[ ${#CP_IPS[@]} -lt 3 ]]; then
echo "!! Only ${#CP_IPS[@]} control-plane node(s) -- the API server (and kubectl checks) will go fully unreachable while that node reboots."
fi
point_kubectl_away_from ""
log "Cluster: $CLUSTER_DIR"
log "Control planes: $CP_CSV"
log "Workers: ${WORKER_CSV:-<none>}"
}
# ---------------------------------------------------------------------------
cmd_upgrade_talos() {
require_bins "$TALOSCTL" kubectl jq "$TF_BIN"
[[ -n "${1:-}" ]] || die "cluster-dir required (usage: $0 upgrade-talos <cluster-dir>)"
setup_cluster_access "$1"
if [[ "$AUTO_CONFIRM" != "1" ]]; then
echo "This only rolls out what's already declared - it doesn't bump installer_image_url or apply it for you."
read -r -p "Have you already set the target installer_image_url and run $TF_BIN apply for $CLUSTER_DIR? [y/N] " answer
echo
[[ "$answer" =~ ^[Yy]$ ]] || die "bump installer_image_url and $TF_BIN apply first, then rerun"
fi
preflight
take_etcd_snapshot "pre-talos-upgrade"
# Control planes first (etcd quorum must never see two rebooting at once),
# then workers, both sorted for a stable order across runs.
local upgrade_order
upgrade_order=$(jq -r '
(to_entries | map(select(.value.machine_type=="controlplane")) | sort_by(.key)) +
(to_entries | map(select(.value.machine_type=="worker")) | sort_by(.key))
| .[] | "\(.key)|\(.value.machine_type)|\(.value.talos_api_ip)|\(.value.installer_image_url)"
' "$TMP_DIR/nodes.json")
echo
echo "Plan:"
local pending=0 name role ip image target current
while IFS='|' read -r name role ip image; do
[[ -z "$name" ]] && continue
target="${image##*:}"
current=$(current_talos_tag "$ip" || echo "unknown")
if [[ "$current" == "$target" ]]; then
echo " $name ($role, $ip): already on $target"
else
echo " $name ($role, $ip): $current -> $target"
pending=$((pending + 1))
fi
done <<< "$upgrade_order"
echo
if [[ "$pending" -eq 0 ]]; then
log "Every node already matches its declared installer_image_url. Nothing to do."
return 0
fi
if [[ "$AUTO_CONFIRM" != "1" ]]; then
read -r -p "Proceed with the upgrade above? [y/N] " answer
echo
[[ "$answer" =~ ^[Yy]$ ]] || die "cancelled"
fi
local k8s_node
while IFS='|' read -r name role ip image; do
[[ -z "$name" ]] && continue
target="${image##*:}"
current=$(current_talos_tag "$ip" || echo "unknown")
log "=== [$role] $name ($ip) ==="
if [[ "$current" == "$target" ]]; then
log "-- already on $target, skipping"
continue
fi
point_kubectl_away_from "$ip"
k8s_node=$(k8s_node_name_for_ip "$ip" || true)
log "-- talosctl upgrade: $current -> $target"
"$TALOSCTL" --endpoints "$ip" --nodes "$ip" upgrade --image "$image" --preserve --wait \
|| die "upgrade failed on $name ($ip) -- it may be in a partial state, investigate before continuing"
wait_for_apiserver 900 || die "API server did not come back within 15m after upgrading $name"
if [[ -n "$k8s_node" ]]; then
log "-- waiting for $k8s_node to be Ready"
kubectl wait --for=condition=Ready "node/$k8s_node" --timeout="$NODE_READY_TIMEOUT" \
|| die "$k8s_node did not become Ready in time"
# Talos uncordons after its own reboot; this covers leftovers from a
# previous failed run.
kubectl uncordon "$k8s_node" 2>/dev/null || true
fi
log "-- confirming etcd is healthy on every control plane before continuing"
etcd_healthy || die "etcd is not healthy on every control plane after upgrading $name"
log "=== $name done ==="
sleep "$SLEEP_BETWEEN_NODES"
done <<< "$upgrade_order"
point_kubectl_away_from ""
log "Final verification"
health_check 2m
kubectl get nodes -o wide
log "All nodes now match their declared installer_image_url."
}
cmd_upgrade_k8s() {
require_bins "$TALOSCTL" kubectl jq "$TF_BIN"
[[ -n "${1:-}" && -n "${2:-}" ]] || die "usage: $0 upgrade-k8s <cluster-dir> <target-version>"
setup_cluster_access "$1"
local target="$2"
local current
current=$(current_k8s_version || echo "unknown")
log "Kubernetes: $current -> $target"
# Can't check whether k8s_version in Terraform has already been bumped -
# unlike installer_image_url, it isn't part of this module's own output
# surface, and the exact variable/local name is caller-specific, so there's
# nothing generic to grep. What IS generic: if the cluster is already
# reporting the target version, something's already happened here, whether
# that's this script running before, a manual upgrade, or Terraform having
# been applied out of order - worth a harder stop than the usual confirm.
if [[ "$current" == "$target" ]]; then
echo "!! Cluster is already reporting $target. If Terraform's k8s_version was bumped and applied before this ran, the order is backwards - upgrade-k8s is meant to run BEFORE that, not after. Re-running this now is harmless (upgrade-k8s is idempotent), but make sure that's actually what you want."
if [[ "$AUTO_CONFIRM" != "1" ]]; then
read -r -p "Proceed anyway? [y/N] " answer
echo
[[ "$answer" =~ ^[Yy]$ ]] || die "cancelled"
fi
elif [[ "$AUTO_CONFIRM" != "1" ]]; then
read -r -p "This expects k8s_version in Terraform to still be at the OLD version ($current) - confirm you haven't bumped and applied it yet. [y/N] " answer
echo
[[ "$answer" =~ ^[Yy]$ ]] || die "if you already bumped k8s_version in Terraform, reconcile that first (see $TF_BIN output), then rerun"
fi
preflight
take_etcd_snapshot "pre-k8s-upgrade"
log "Dry-run: upgrade-k8s --to $target"
"$TALOSCTL" --nodes "$FIRST_CP" upgrade-k8s --to "$target" --dry-run \
|| die "dry-run failed -- if it's a compatibility error, your talosctl client may be too old for this cluster (see TALOSCTL env var)"
if [[ "$AUTO_CONFIRM" != "1" ]]; then
read -r -p "Dry-run above looks right - proceed with the real upgrade to $target? [y/N] " answer
echo
[[ "$answer" =~ ^[Yy]$ ]] || die "cancelled"
fi
log "Running upgrade-k8s --to $target"
"$TALOSCTL" --nodes "$FIRST_CP" upgrade-k8s --to "$target" \
|| die "upgrade-k8s failed -- check the output above before retrying"
log "Final verification"
health_check "$HEALTH_WAIT_TIMEOUT"
kubectl get nodes -o wide
echo
echo "Kubernetes is now at $target. Terraform still declares the old version, and will keep drifting from reality until you sync it."
if [[ "$AUTO_CONFIRM" != "1" ]]; then
while true; do
read -r -p "Updated k8s_version to \"$target\" and run $TF_BIN apply for $CLUSTER_DIR? [y/N] " answer
echo
[[ "$answer" =~ ^[Yy]$ ]] && break
echo "Do that now, then this is done."
done
fi
log "Done."
}
# ---------------------------------------------------------------------------
case "${1:-}" in
upgrade-talos)
shift
cmd_upgrade_talos "$@"
;;
upgrade-k8s)
shift
cmd_upgrade_k8s "$@"
;;
-h|--help|help|"")
usage
;;
*)
echo "!! unknown command: $1" >&2
usage
exit 1
;;
esac
That’s three modules covered - images, virtual machines, and now Talos itself. Let’s see how to put them all together, and finally spin up a cluster.
Putting It All Together
This lives in the repo as its own example, terraform/examples/talos-on-proxmox, and it’s deliberately minimal - just the three modules from this post, wired together into a cluster that actually boots. No Cilium, no Proxmox CSI, no GitOps bootstrap yet - those are all separate concerns I’m saving for future parts of this series, so this example stays focused on just standing up the infrastructure and the cluster itself.
# Step 1: Look up a Talos OS image from the Image Factory.
#
# The module queries the Image Factory to build a schematic for the
# requested extensions and returns the installer image URL plus the ISO
# URL/expected file path - it doesn't download or upload anything itself.
# Before running Step 2 for the first time, tell Proxmox to fetch the ISO
# to each node yourself, via the API's download-url endpoint (see the
# module README):
#
# curl -k -H "Authorization: PVEAPIToken=<user>@<realm>!<token-id>=<secret>" \
# -X POST "https://<proxmox-host>:8006/api2/json/nodes/<node>/storage/<datastore>/download-url" \
# --data-urlencode "content=iso" \
# --data-urlencode "filename=$(tofu output -raw iso_file_name)" \
# --data-urlencode "url=$(tofu output -raw iso_url)"
#
# Platform is "nocloud": it's what makes Talos read the static IP we hand
# it below via cloud-init, instead of waiting on DHCP.
module "talos_image" {
source = "git::https://github.com/hovorka-labs/iac-modules.git//terraform/modules/talos/images?ref=blog/homelab-diary-part4"
talos_image_version = var.talos_version
talos_image_platform = "nocloud"
# Add Talos extensions required by the cluster nodes.
# The full extension catalogue is at https://factory.talos.dev.
talos_image_extensions = [
"siderolabs/qemu-guest-agent",
]
}
# Step 2: Provision one Proxmox VM per Talos node.
#
# Each VM boots from the image downloaded above (attached as a cdrom); the
# actual disk starts empty and Talos installs itself onto it on first boot.
# The static IP comes from cloud-init, which the nocloud platform picks up
# before Talos even has a machine config to work from.
module "vms" {
source = "git::https://github.com/hovorka-labs/iac-modules.git//terraform/modules/proxmox/virtual-machines?ref=blog/homelab-diary-part4"
virtual_machines = local.virtual_machines
}
# Step 3: Bootstrap a Talos Kubernetes cluster on top of the VMs above.
#
# Each node's mac_address comes straight from module.vms, not a variable -
# Proxmox assigns the MAC when the VM is created, and Talos just needs to be
# told the same address so it can match the right NIC in its network config.
#
# region matters once Proxmox CSI is wired in (a future post); for now it
# just reuses the cluster name.
#
# Future step (covered in the next blog post): install Cilium. Without a
# CNI, nodes come up but nothing can actually schedule yet.
module "talos_cluster" {
source = "git::https://github.com/hovorka-labs/iac-modules.git//terraform/modules/talos?ref=blog/homelab-diary-part4"
cluster = {
name = var.talos_cluster_name
region = var.talos_cluster_name
gateway_api_version = var.gateway_api_version
}
nodes = local.talos_nodes
}
Three steps, in order: look up the Talos image, provision a VM per node from it, then bootstrap Talos on top of the VMs. One detail: mac_address in the Talos node config isn’t a variable I set anywhere, it’s read straight back out of module.vms.mac_addresses. Proxmox assigns the MAC when the VM gets created, and the Talos module just needs to be told the same address so its deviceSelector can match the right NIC. No manual MAC pinning, no coordinating two separate values by hand.
Here’s how that, and the rest of each node’s Talos-facing config, comes together in locals.tf:
# The Talos module wants a bare IP and a separate subnet mask, not the
# CIDR notation Proxmox's cloud-init wants above - split it once here.
talos_nodes = {
for name, node in var.nodes : name => {
machine_type = node.role
# Proxmox CSI/CCM call the Proxmox API using this zone value directly
# as the node name, so it has to be the real Proxmox node, not this
# node's own identity in the nodes map.
zone = node.proxmox_node
ip = split("/", node.ip)[0]
subnet_mask = split("/", node.ip)[1]
gateway = var.network_gateway
mac_address = module.vms.mac_addresses[name]
installer_image_url = module.talos_image.installer_image
k8s_version = var.k8s_version
}
}The rest is just plumbing: talos_cluster_name, k8s_version, and gateway_api_version are new variables feeding cluster.name, nodes[*].k8s_version, and cluster.gateway_api_version. region just reuses the cluster name for now, since nothing in this example actually reads it yet - that only starts to matter once Proxmox CSI gets wired in.
Running tofu apply against this gets me a Talos cluster with a working control plane and a kubeconfig/talosconfig I can pull straight out of the outputs. What it doesn’t get me yet is a cluster that can actually run anything, since there’s still no CNI installed - we will look into setting up Cilium CNI with OpenTofu for this setup in the next episode. This was a pretty extensive post, but there was a lot to cover, and I prepared a base for all the upcoming blog posts. See you at the next one!