LVM Support in talos

[pranav767]

Existing Issue: #12309
We've had some discussion regarding adding LVM capabilities in talos, here
Addressing the first question would be do we even need to introduce LVM in talos.

First of all, CSIs (as far as I know) today happily manage LVM volumes without any Talos involvement, so I'm not sure why we need to introduce any concept of it to Talos.

To be clear Volumes in talos can be defined as a disk/partition/director/overlay mount, talos handles static provisioning really well, for any k8s use case we can use UserVolumes.
Their are few scenarios where the need for LVM arises:

1. When multiple physical disks need to be pooled into a single logical storage pool that workloads can consume as one volume.
2. Gradual storage expansion by adding new disks over time without reconfiguring applications or Pods.
3. Resizing volumes (primarily growing) without recreating disks or redeploying workloads.
4. Strong per-workload storage isolation, preventing one Pod from exhausting disk space and impacting others.
5. Disk replacement and maintenance on a node without stopping workloads or losing data.
6. Use of Volume Layouts like striping or mirroring
7. Fast, point-in-time snapshots for short-term rollback or experimentation.
8. Cloning volumes for development, testing, or CI workloads.

We can have LVM in talos using any csi drivers like csi-driver-lvm or topolvm
But there are multiple limitations which needs to be addressed while using these, topolvm-limitations

I tried using csi-driver-lvm in my initial attempts.

1. First attempt: The cluster broke because csi-driver-lvm was installed without restricting it to worker nodes. The driver attempted pvcreate on the Talos system disk, CSI drivers will destroy the OS disk if allowed.
2. Second attempt: After restricting the driver to worker nodes, the pod started but failed with failed to mkdir "/etc/lvm/backup": read-only file system, as the CSI driver assumed /etc/lvm was writable.
3. Final attempt: Reinstalled the CSI driver with lvm.hostWritePath=/var/lib/lvm, after which the driver started successfully and LVM provisioning worked as expected.

Later setting up a pod and pvc with proper storage class worked fine:

$ kubectl exec -n kube-system -it csi-driver-lvm-kdlx2 \
  -c csi-driver-lvm -- vgs
  VG      #PV #LV #SN Attr   VSize  VFree 
  csi-lvm   2   1   0 wz--n- 19.99g 18.99g

IMHO, this way works fine, we should work on LVM support in Talos if it simplifies the user experience and helps avoid the following issues:

• Reliance on privileged containers, increasing the risk of host compromise
• Regex-based disk selection, which can accidentally wipe the OS disk
• Lack of Talos-level policies and guardrails to prevent operator mistakes
• Assumptions about mutable system paths (e.g. /etc), which break on Talos
• Ad-hoc disk bootstrapping, leading to inconsistent VG layout and capacity reporting
• Absence of declarative intent, making maintenance and recovery more error-prone
• Also, limitations mentioned in topolvm

If these points make a strong case for adding LVM support in Talos, we can further discuss relevant use cases and scenarios, and how such support could improve usability , as done in common config

[bartlaarhoven]

Adding a concrete production use case where exactly this gap is biting us.

We run a self-hosted K8s cluster on Talos v1.10.6 (6 bare-metal worker nodes, each with 3× 8TB NVMe SSD), with Rook-Ceph providing all storage today.

Our pain point: PostgreSQL (CloudNativePG) WAL fsync latency on Ceph RBD averages 30-100ms (we see peaks of 300ms on hot volumes). On a busy production cluster doing ~3000 commits/s, this is the dominant latency. The same NVMe, accessed locally, does <1ms fsync. Identical hardware - the storage layer is the entire difference.

What we want isn't "drop Ceph for LVM" - it's a mixed-tier architecture, which I suspect is fairly common at the "outgrew Ceph-only, still need RWX" scale:

• Local LVM: latency-critical workloads (Postgres WAL, Elasticsearch hot indices). CNPG handles HA/replication at the application layer, so storage-level redundancy is redundant.
• Ceph: RWX, CephFS, bulk PVCs, cross-node mobility.

What's keeping us hesitant about csi-driver-lvm / TopoLVM today:

1. Disk selection safety. devicePattern regex on a node where another subsystem (Rook) is actively managing disks is genuinely risky. One typo and you've pvcreated on a live OSD or even the OS disk - exactly your "first attempt" experience. Talos-native LVMVolumeGroupConfig with explicit selectors (analogous to UserVolumeConfig) would give the same guardrails Talos already enforces for system partitions.
2. No extension means no reboot to enable the capability. siderolabs/lvm2 doesn't exist as an official extension, so we'd need a custom build + per-node reboot just to enable the userspace tools. Native Talos LVM = MachineConfig change, no schematic regeneration.
3. Declarative coexistence with Rook. Today Rook owns disks via its own CR. With LVM as a first-class Talos resource, both subsystems can declaratively claim what they own and Talos can prevent overlap.

Strongly +1 on landing this. Happy to test against a real production migration once anything is in master.

[briantopping]

+1 as well to this. I've struggled to set up a small amount of local storage via TopoLVM using UserVolume, but my recollection is it would leave the boot process in an unfinished state and eventually just restart itself. We also value the "mixed-tier architecture" @bartlaarhoven lays out above. Our ideal shape would include an option for all system volumes to be in LVM if that was practical.

[bartlaarhoven]

I believe this is related to #12309 and #13529 and therefore seems to be resolved..? 🥳