vm clone --on-demand <hibernate-import>: Windows guest loses NIC after wake (hot-swapped virtio-net not visible to guest)

[tonicmuroq]

Symptom

After cocoon vm clone --on-demand <hibernate-import-snapshot> resumes a Windows 11 guest, the guest has zero network adapters visible to the OS. Networking is completely dead — no DHCP, no ARP, no traffic of any kind on the host TAP.

Reproduce

1. Start a Windows 11 cocoon VM and wait for it to fully boot + agent to come online (DHCP'd, reachable).
2. cocoon snapshot save --name <name> <vm-id> (hibernate)
3. cocoon snapshot export <name> -o - | <push to OCI>
4. Later: pull, cocoon snapshot import --name <name>-hibernate-import
5. cocoon vm clone --output json --name <vm-name> --cpu 4 --memory 4294967296 --network cocoon-dhcp --on-demand <name>-hibernate-import

What we observed

Pre-hibernate VM had:

• Guest MAC a6:a1:d1:30:58:4e
• DHCP-assigned IP 172.20.0.46
• Lease in /var/lib/cocoon/net/leases.json for that MAC

Post-wake (clone):

• cocoon vm list: state=running, first_booted=true
• vm.info (cloud-hypervisor API): state=Running, net[0].mac=6a:54:5c:d8:f7:e0 (a new MAC, regenerated)
• cloud-hypervisor process: alive, ~115% CPU (so guest is executing instructions)
• Guest memory: 2.9 GiB resident (snapshot fully restored)
• 30s tcpdump -nei tap<id>-0 inside the netns: 0 packets from guest, only one host-originated IPv6 router-solicit
• From host bridge cni0 (172.20.0.1): ping 172.20.0.46 → Destination Host Unreachable, ip neigh → FAILED

Then via cocoon vm console <id> SAC → cmd channel:

C:\Windows\System32>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : COCOON-VM
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No

C:\Windows\System32>arp -a
No ARP Entries Found.

C:\Windows\System32>ipconfig /renew
Windows IP Configuration

The operation failed as no adapter is in the state permissible for
this operation.

There is no "Ethernet adapter" section at all in ipconfig /all — Windows sees zero NICs. So this is not a stale-MAC / stale-DHCP-lease issue: the entire device is gone from the guest's view.

Root cause (code-level)

hypervisor/cloudhypervisor/clone.go resume sequence is:

// restoreAndResumeClone, lines ~162–197
restoreVM(ctx, hc, runDir, opts.onDemand)            // vm.restore — VM paused, snapshot loaded
hotSwapNets(ctx, hc, opts.snapshotCfg.Nets, opts.networkConfigs)  // <-- problem
...
resumeVM(ctx, hc)                                     // vm.resume

hotSwapNets() (clone.go:333-353):

// Remove the snapshot-encoded NICs (with their original MACs)
for _, oldNet := range oldNets {
    removeDeviceVM(ctx, hc, oldNet.ID)        // virtio-net hot-removed
}
// Then attach freshly-allocated NICs with new MACs
for _, nc := range networkConfigs {
    addNetVM(ctx, hc, networkConfigToNet(nc)) // virtio-net hot-added with new MAC
}

So between vm.restore and vm.resume:

1. The original virtio-net device — the one Windows had bound at the time of hibernate, with PCI slot/BDF, MAC, queue config baked into its in-memory driver state — is hot-removed.
2. A fresh virtio-net device with a freshly-generated MAC (no preservation; see below) is hot-added.

When vm.resume fires, Windows continues from the snapshot expecting its original NIC. From its perspective, that device is now gone. The newly-added device is a different PCI device that Windows would have to enumerate via PnP — but the snapshot was taken with the OS in a state where that PnP enumeration has already completed for the old device, and the OS doesn't naturally re-enumerate a hot-added virtio-net into a usable adapter on resume. Result: zero adapters.

Linux guests probably tolerate this better (kernel re-binds virtio drivers on hotplug), but for Windows it's effectively fatal.

Why the MAC is regenerated

Even if the same PCI slot were reused, the MAC is regenerated, which would also break DHCP-lease continuity:

• types/snapshot.go:7-16 — SnapshotConfig only persists NICs int (count), not MAC/PCI/queue config. So at restore time we can't know what MAC the snapshot was taken with.
• cmd/vm/run.go:75-145 Clone path calls initNetwork(...) (line 504) never passing the existing parameter.
• network/cni/create.go:64-88 — Config() signature supports existing ...*types.NetworkConfig for MAC reuse, but the clone path doesn't use it.
• network/bridge/bridge_linux.go:201-206 — generateMAC() produces a fresh random MAC via crypto/rand.

Suggested fix direction

The "remove old, add new" approach is fundamentally incompatible with snapshot restore for Windows guests. Two possible fixes:

1. Don't hot-swap at all on wake. Persist enough NIC config in the snapshot (SnapshotConfig.Nets: MAC, PCI slot, num_queues, etc.) that the wake path can reconstruct the exact same virtio-net topology before vm.restore. The TAP/MAC/queue config for the new clone must match the snapshot 1:1.

2. If host-side TAP must be recreated (e.g. node-bound MACs aren't portable across nodes), wire the existing ...*types.NetworkConfig capability already present in network/cni/create.go:64-88 through the clone flow so at least the same MAC is reused. This still won't fix the Windows hot-swap issue but would let Linux guests resume cleanly.

Option 1 is the only one that keeps Windows wakes working. Option 2 alone is insufficient.

Environment

• cocoon: current main (binary deployed on cocoonset-node-1, asia-southeast1-b GKE cocoonset cluster)
• cloud-hypervisor: bundled
• Guest: Windows 11 (cocoon simular/win11-hot:20260505-testing-1), DHCP via cocoon-dhcp
• Network plugin: CNI

Related

• fix(snapshot): fall back to non-sparse tar when segment map exceeds PAX cap #23 (sparse PAX overflow) — fixed, push side now works
• fix(snapshot): bump FetchSnapshotConfig read cap from 1 MiB to 64 MiB epoch#3 (1MB config cap) — fixed, pull side now works
• This issue is the next blocker on Windows hibernate/wake e2e

[tonicmuroq]

tested scenarios — after hotswap, as long as DHCP is in use, SAC sees the new IP

What we're seeing isn't "DHCP didn't get an IP" — Windows has no NIC at all in its device list. Re-pasting the SAC cmd-channel output for visibility (it's in the issue body too):

C:\Windows\System32>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : COCOON-VM
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No

Note that ipconfig /all has no "Ethernet adapter ..." section at all. Even when DHCP fails, Windows still lists the adapter (with "Media disconnected" or "DHCP Enabled... Yes / Autoconfiguration ..."). Here the adapter section is entirely missing.

C:\Windows\System32>arp -a
No ARP Entries Found.

C:\Windows\System32>ipconfig /renew
The operation failed as no adapter is in the state permissible for this operation.

Host-side evidence agrees:

• 30s tcpdump -nei tap<id>-0 inside the netns: 0 packets from guest
• DHCP server: no OFFER/REQUEST for the new MAC 6a:54:5c:d8:f7:e0
• ping 172.20.0.46 from cni0 bridge: Destination Host Unreachable; ip neigh FAILED
• cloud-hypervisor process is alive (~115% CPU, 2.9 GiB RES, state=Running), so the guest is executing instructions — it just can't do any network I/O

Could you share details of the scenario you tested?

1. Guest OS — Linux or Windows? Linux's kernel handles virtio-net hot-add/remove via driver re-bind; Windows PnP behavior under snapshot-restore is different and won't necessarily enumerate a hot-added device after resume.
2. Clone path — vm clone --pull <image> (fresh boot), or vm clone --on-demand <hibernate-import> (memory restore from a hibernate snapshot)? A fresh boot does a boot-time PCI scan and naturally picks up the new NIC; a hibernate restore continues from the existing RAM state past that scan, and PnP usually won't auto-enumerate a hot-added device on resume.
3. cocoon version — anything different from main with fix(snapshot): fall back to non-sparse tar when segment map exceeds PAX cap #23 merged in (the binary we're running on the test node)?

Our image is simular/win11-hot:20260505-testing-1 (Windows 11). If you have a Windows hibernate-wake repro that works, happy to compare against it.

[tonicmuroq]

Local-only reproduction — no vk-cocoon, no epoch, no export/import round-trip

I created a Windows 11 VM via our orchestrator (just to get a real workload state: agent + watchdog + Firebase WebSocket connected), then on the cocoon node I drove the rest with only cocoon CLI.

Setup:

$ sudo cocoon vm list
4HPM4MUSNQSEOIOEM5UAAQV5P7  vk-default-vm-80de7d30-0  running  ...  source MAC: 1e:fb:e5:ab:5d:0a

Stage A — cocoon vm clone (full memory restore, no --on-demand)

$ sudo cocoon snapshot save --name 80de7d30-snap 4HPM4MUSNQSEOIOEM5UAAQV5P7
INF snapshot saved: G6E73LYOPV6YFEAS7WCRUC265D

$ sudo cocoon vm clone --name 80de7d30-test-a --network cocoon-dhcp 80de7d30-snap
INF removed snapshot NIC _net2 (old MAC 1e:fb:e5:ab:5d:0a)
INF added NIC with MAC 82:f7:df:93:90:f3 on TAP tapCLECUOUC-0
INF VM CLECUOUCQRBLFXDUMZBLRMJDAQ cloned from snapshot

# cocoon's own informational message after the clone:
> Windows clone: NICs hot-swapped with new MAC addresses.
>   DHCP networks: no action needed.

30s tcpdump -nei tapCLECUOUC-0 inside the netns:

04:13:43.514703 16:cc:b6:7f:3a:2b > 33:33:00:00:00:02, ICMP6, router solicitation, length 16
04:13:51.706689 16:cc:b6:7f:3a:2b > 33:33:00:00:00:02, ICMP6, router solicitation, length 16
04:14:09.114694 16:cc:b6:7f:3a:2b > 33:33:00:00:00:02, ICMP6, router solicitation, length 16

3 packets captured

All three frames are from 16:cc:b6:7f:3a:2b — the host-side veth peer, not the guest. Guest egress = 0. No DHCP DISCOVER/REQUEST/ARP for the new guest MAC 82:f7:df:93:90:f3. No lease appears in /var/lib/cocoon/net/leases.json for that MAC.

Stage B — cocoon vm clone --on-demand (UFFD lazy memory load)

Same 80de7d30-snap, just adding --on-demand:

$ sudo cocoon vm clone --on-demand --name 80de7d30-test-b --network cocoon-dhcp 80de7d30-snap
INF removed snapshot NIC _net2 (old MAC 1e:fb:e5:ab:5d:0a)
INF added NIC with MAC 0e:12:e1:68:69:7b on TAP tapTEWA3CBE-0

30s tcpdump on tapTEWA3CBE-0:

04:14:34.202922 4a:a7:2f:4e:51:33 > 33:33:00:00:00:02, ICMP6, router solicitation
04:14:43.418693 4a:a7:2f:4e:51:33 > 33:33:00:00:00:02, ICMP6, router solicitation
04:14:43.937552 1e:fb:e5:ab:5d:0a > ff:ff:ff:ff:ff:ff, IPv4: 172.20.0.36.138 > 172.20.0.255.138: NetBIOS, length 201
04:15:01.338687 4a:a7:2f:4e:51:33 > 33:33:00:00:00:02, ICMP6, router solicitation

Same: only host-side IPv6 router-solicits. Zero packets from the new guest MAC 0e:12:e1:68:69:7b. No DHCP, no lease.

(The single NetBIOS broadcast from 1e:fb:e5:ab:5d:0a is the original source VM, which is still running on the same bridge — it's bridge-broadcast leakage onto this TAP, not from our clone.)

What this means

• vm clone and vm clone --on-demand from a local hot snapshot both leave the Windows guest with the same observable failure as the cocoon-issue#28 original report (woken from a hibernate-import after epoch round-trip).
• The bug is therefore not in epoch's snapshot transport, not in the snapshot import path, and not in vk-cocoon's orchestration. It's in cocoon vm clone itself, specifically the hotSwapNets step inside restoreAndResumeClone (hypervisor/cloudhypervisor/clone.go:180).
• cocoon's own post-clone hint says "DHCP networks: no action needed" — but for Windows guests this is incorrect. The hot-removed-then-re-added virtio-net is not visible to the guest after vm.resume.

Reduced repro

Running on a single cocoon node, with a Win11 cocoon image:

# 1. Boot a Windows VM and let it settle (agent etc. fully running)
sudo cocoon vm run --windows --network cocoon-dhcp --cpu 4 --memory 4G \
    --name win11-test simular/win11-hot:20260505-testing-1
# (wait for DHCP lease + workload to settle)

# 2. Hot snapshot it
sudo cocoon snapshot save --name win11-snap <vm-id>

# 3. Clone — with OR without --on-demand, both fail
sudo cocoon vm clone --on-demand --name win11-clone --network cocoon-dhcp win11-snap

# 4. Observe: clone has no guest egress, no DHCP from new MAC, no lease
sudo ip netns exec <clone-netns> tcpdump -nei <clone-tap>

If you have a Windows scenario where this works, I'd love to compare image/cocoon-version to figure out what's different.

[tonicmuroq]

Stage C — full export → rm → import → clone (mirrors vk-cocoon's path locally)

Completing the matrix from the previous comment. This case round-trips the snapshot through the local OCI archive format (snapshot export → snapshot rm → snapshot import) before the clone, mirroring what vk-cocoon does (it just adds an OCI registry hop on top via epoch).

$ sudo cocoon snapshot save --name 80de7d30-snap2 4HPM4MUSNQSEOIOEM5UAAQV5P7
INF snapshot saved: FPXALDWPQA6V5JX5PFSK3NPWOH

$ sudo cocoon snapshot export 80de7d30-snap2 -o /tmp/80de7d30-c.tar
INF exported: /tmp/80de7d30-c.tar       # 5.79 GB

$ sudo cocoon snapshot rm 80de7d30-snap2
INF deleted: FPXALDWPQA6V5JX5PFSK3NPWOH

$ sudo cocoon snapshot import --name 80de7d30-snap2-import /tmp/80de7d30-c.tar
INF snapshot imported: KZWLOL2R3FHHSFPV4FJ6N2N26Q

$ sudo cocoon vm clone --on-demand --name 80de7d30-test-c --network cocoon-dhcp 80de7d30-snap2-import
INF removed snapshot NIC _net2 (old MAC 1e:fb:e5:ab:5d:0a)
INF added NIC with MAC f6:58:57:c1:3c:76 on TAP tapTNSBWEUX-0

30s tcpdump on tapTNSBWEUX-0:

04:19:38.779697 26:c3:0c:96:a3:71 > 33:33:00:00:00:02, ICMP6, router solicitation
04:19:46.522705 26:c3:0c:96:a3:71 > 33:33:00:00:00:02, ICMP6, router solicitation
04:20:01.370697 26:c3:0c:96:a3:71 > 33:33:00:00:00:02, ICMP6, router solicitation

3 packets captured

Same result: only host-side IPv6 router-solicits, 0 packets from the guest's new MAC f6:58:57:c1:3c:76, no DHCP, no lease.

Final test matrix

Stage	Path	guest egress	DHCP for new MAC	Lease
A	vm clone <snap> (full restore)	0	0	none
B	vm clone --on-demand <snap>	0	0	none
C	save → export → rm → import → clone --on-demand	0	0	none

Three different paths, all local on a single cocoon node, no vk-cocoon, no epoch. All three reproduce identically. The bug is in the hotSwapNets step on the local clone path, not in the snapshot transport or orchestrator.

[CMGS]

链式 clone（clone-of-clone）复现 + 根因定位

复现脚本在 testbed-1 (35.240.182.52)：org Win11 → snap → clone1(c1) → snap → clone2(c2)。c1 通，c2 失败——失败形态跟原 issue「丢 NIC」不完全一样，但走到的是同一根因。

现象（c2 阶段）

CH live config 干净——只有 1 个 net 设备：

_net4  MAC 8a:16:07:e7:93:3c  TAP tapT4YCW4W2-0  (chain-c2 hot-swap 后的 NIC)

但 c2 TAP 上 tcpdump 看到的包全部源 MAC 是 ce:82:c7:6d:fe:7d（祖父 c0 的 MAC）+ 源 IP 是 10.99.0.144（c0 的 IP）：

05:06:07.346403 ce:82:c7:6d:fe:7d > ff:ff:ff:ff:ff:ff, ARP Request who-has 169.254.169.254 tell 10.99.0.144
05:06:11.343433 ce:82:c7:6d:fe:7d > ff:ff:ff:ff:ff:ff, ARP Request who-has 169.254.169.254 tell 10.99.0.144
...

过滤 ether src 8a:16:07:e7:93:3c → 0 packets，cocoon-net DHCP 没收到 c2 MAC 的任何包。

关键反向证据

cocoon vm stop chain-c2 && cocoon vm start chain-c2 后立即正常：

• 进程 ELAPSED=02:08（全新进程，不是 resume）
• CH 命令行没有 --restore
• Windows 出包源 MAC 全部是 8a:16:07:e7:93:3c
• DHCP 拿到全新 lease 10.99.0.135

也就是走 fresh boot from disk 路径就没事，问题完全在 vm.restore + hot-swap 路径上。

根因机制（Windows NDIS profile rebinding）

链式状态演化：

阶段	CH 设备	Windows NDIS state
c0 boot	_net1 MAC ce:82:... PCI slot X	GUID-A profile，绑 MAC ce:.../IP 10.99.0.144
c1 hot-swap (rm _net1 + add _net3)	_net3 MAC 8e:... PCI slot X (CH 复用最低空 slot)	GUID-A 标 disconnected（registry 保留），GUID-B profile 新建在 slot X 上，DHCP 10.99.0.46
c1 snap	binary memory 封 GUID-A(stale) + GUID-B(active)	—
c2 hot-swap (rm _net3 + add _net4)	_net4 MAC 8a:... PCI slot X 又被复用	Windows 发现 slot X 上有新设备 → 按 PCI 路径匹配到 GUID-A 的历史 profile（disconnected）→ 复活 GUID-A，用 virtio-net SET_MAC_ADDRESS 把硬件 MAC 改回 ce:...，套回缓存 IP 10.99.0.144，因此根本不发 DHCP

snap 文件级别其实是干净的（snap2 device-manager 只有 _net3，没有 _net1 残留；memory-ranges 里也 grep 不到 c0 的 MAC bytes），ghost 完全在 Windows 注册表 hive + NDIS interface store 里——这部分是 binary memory 的一部分，hot-swap API 接触不到。

副作用（生产风险）

c2 emit 的 ce:82:... 帧到了 cni0 bridge，与 c0 的合法 MAC 在两个 veth 端口上冲突。当下 fdb 稳定指向 c0 的 veth，但只要 c0 网络一空闲，c2 的"phantom"流量就可能抢走 MAC，让 c0 真实流量被错误转发到 c2 → 正常 VM 间歇丢包。

源码层面修复方案

方案 A：调整 hot-swap 顺序（最小代价，首选试验）

hypervisor/cloudhypervisor/clone.go:311 hotSwapNets 当前逻辑：

for _, oldNet := range oldNets {
    removeDeviceVM(...)          // 先卸：释放 PCI slot
}
for i, nc := range networkConfigs {
    addNetVM(...)                // 再装：CH 复用最低空 slot = 老 slot
}

改成 add-then-remove：

for i, nc := range networkConfigs {
    addNetVM(...)                // 先装：老 slot 还占着，新设备拿到下一个空 slot
}
for _, oldNet := range oldNets {
    removeDeviceVM(...)          // 后卸：老 slot 才释放
}

理论效果：新 NIC 的 PCI BDF 跟旧 NIC 不同 → Windows 按 PCI 路径不会匹配到旧 GUID profile → 强制创建新 GUID-C → 走完整 DHCP 流程。

需要验证：CH 是否真的"lowest free slot"分配（看上去是，因为 ID 序列 _net1 → _net3 跳过了 _net2 也佐证 ID counter 不复用，但 PCI BDF 分配策略需要从响应里抓 bdf 字段确认）。addNetVM 现在丢弃了 chPciDeviceInfo，可以顺手记下来打 log。

方案 B：Windows 专用 power-cycle（兜底）

在 restoreAndResumeClone 末尾对 vmCfg.Windows == true 的 case，resume 后立刻：

if opts.vmCfg.Windows {
    powerButton(ctx, hc)            // ACPI shutdown
    waitForShutdown(...)            // poll vm.info state == Shutdown
    bootVM(ctx, hc)                 // CH 直接从磁盘重新 boot，不走 restore
}

代价：失去 on-demand 速度优势（Windows 完整重启 ~30-60s）。但 100% 可靠，因为本质就是绕开整个 vm.restore + hot-swap 路径。

方案 C：CH 上游补 bdf 参数到 vm.add-net

CH 当前的 vm.add-net payload（chNet）没有 pci_segment 之外的 BDF 控制；CH 内部用最低空 slot 分配。理想做法是给 CH 提 PR 加 bdf: "0000:00:0c.0" 这样的可选字段，让 cocoon 显式选远离老 slot 的位置。

但这是上游 Rust 改动，周期长。如果方案 A 成立，方案 C 优先级降低。

方案 D：snapshot 沿袭检测，拒绝链式 Windows on-demand clone

记录 r.SnapshotLineage 计数（c0=0、c0-snap=0、c1=1、c1-snap=1、c2=2…）。vm clone 看到 Windows && OnDemand && lineage >= 1 时打错或自动降级到方案 B。最保守但限制功能。

方案 E：guest-side cleanup（不可行）

理论上可以在 Windows 启动时跑：

pnputil /enum-devices /class Net /status DISCONNECTED
netsh interface ip delete arpcache

清掉 ghost adapter。但需要 cocoon-agent for Windows，目前没有；SAC 不可控。否决。

推荐路线

1. 先实施方案 A——10 行改动，立刻能在 testbed-1 验证。如果 Windows 真的因 PCI slot 不同而新建 GUID-C，问题就此解决。
2. 方案 A 不行就上方案 B 做 Windows-only 兜底。
3. 方案 C 作为长期上游贡献，跟 vsock_override 的 backporting 一样的节奏。
4. 方案 D 只在前 3 都受阻时考虑（限制功能）。

复现环境保留在 testbed-1：chain-c0 (10.99.0.144)、chain-c1 (10.99.0.46)、chain-c2 (现在已 reboot 到 10.99.0.135)。可以随时再做对照实验。