Restoration failed while compacting the snapshots

[ishan16696]

/area disaster-recovery
/kind bug

What happened:
While doing the performance testing for disk iops, I found the that compaction was failing, on digging I found restoration failed with this error:

time="2025-11-16T11:04:55Z" level=fatal msg="Failed to compact snapshot: unable to restore snapshots during compaction: failed to apply events to etcd for delta snapshot Incr-19168042-19177135-1763277147.gz : rpc error: code = ResourceExhausted desc = trying to send message larger than max (51310260 vs. 2097152)"

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):
I will try to reproduce it.

Anything else we need to know?:

[ishan16696]

another occurence of this issue:

time="2026-01-11T17:28:08Z" level=info msg="Applying delta snapshot Incr-56523002-56556387-1768149708.gz [243/265]" actor=restorer
{"level":"info","ts":"2026-01-11T17:28:25.601436Z","caller":"etcdserver/server.go:1520","msg":"triggering snapshot","local-member-id":"8e9e05c52164694d","local-member-applied-index":700008,"local-member-snapshot-index":600007,"local-member-snapshot-count":100000}
{"level":"info","ts":"2026-01-11T17:28:25.611173Z","caller":"etcdserver/server.go:2541","msg":"saved snapshot","snapshot-index":700008}
{"level":"info","ts":"2026-01-11T17:28:25.6125Z","caller":"etcdserver/server.go:2590","msg":"compacted Raft logs","compact-index":695008}
{"level":"info","ts":"2026-01-11T17:28:41.37819Z","caller":"wal/wal.go:784","msg":"created a new WAL segment","path":"/var/etcd/data/compaction.etcd/member/wal/000000000000000a-00000000000acda0.wal"}
{"level":"info","ts":"2026-01-11T17:28:53.277037Z","caller":"fileutil/purge.go:115","msg":"purged","path":"/var/etcd/data/compaction.etcd/member/snap/0000000000000002-0000000000030d43.snap"}
{"level":"info","ts":"2026-01-11T17:28:53.300032Z","caller":"fileutil/purge.go:115","msg":"purged","path":"/var/etcd/data/compaction.etcd/member/wal/0000000000000005-000000000006acd6.wal"}
{"level":"warn","ts":"2026-01-11T17:29:16.143805Z","caller":"clientv3/retry_interceptor.go:63","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00017b860/127.0.0.1:40571","attempt":0,"error":"rpc error: code = ResourceExhausted desc = trying to send message larger than max (56886285 vs. 2097152)"}
time="2026-01-11T17:29:16Z" level=info msg="Cleanup complete" actor=restorer
time="2026-01-11T17:29:16Z" level=error msg="Restoration failed." actor=restorer
time="2026-01-11T17:29:16Z" level=info msg="Closing handleAlarm..." actor=restorer
time="2026-01-11T17:29:16Z" level=fatal msg="Failed to compact snapshot: unable to restore snapshots during compaction: failed to apply events to etcd for delta snapshot Incr-56523002-56556387-1768149708.gz : rpc error: code = ResourceExhausted desc = trying to send message larger than max (56886285 vs. 2097152)"

[gardener-ci-robot]

The Gardener project currently lacks enough active contributors to adequately respond to all issues.
This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 60d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 60d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as active with /lifecycle active
• Mark this issue as frozen with /lifecycle frozen
• Mark this issue as fresh with /remove-lifecycle stale
• Mark this issue as rotten with /lifecycle rotten
• Close this issue with /close

/lifecycle stale