feat(kad): add provider rejection and spillover

[gmelodie]

Summary

Adds overload protection for Kademlia provider records behind the -d:kadProviderRejection compilation flag.

• Without the flag: All existing behavior is preserved byte-for-byte.
• With the flag enabled:
  • Nodes can reject ADD_PROVIDER requests when a configurable per-key provider limit (maxProvidersPerKey) is reached.
  • Advertising nodes automatically backtrack to progressively farther peers ("spillover") when the closest candidates are full.
  • Ensures records can still be stored even on heavily loaded keys.

---

Changes by File

libp2p/protocols/kademlia/protobuf.nim

• Added AddProviderStatus enum:
  • accepted = 0
  • rejected = 1
• Encoded in protobuf field 10 of the Message type.
• Added optional field:
  • providerStatus: Opt[AddProviderStatus]
• Encoding/decoding is always active for clean round-tripping, but runtime population only occurs under the flag.

libp2p/protocols/kademlia/types.nim

• Added constant:
  • DefaultMaxProvidersPerKey = 0 (0 = unlimited)
• Added config field:
  • maxProvidersPerKey: int in KadDHTConfig
• Includes documentation and constructor default.

libp2p/protocols/kademlia/find.nim

• Added:
  • allSortedPeers*(state: LookupState): seq[PeerId]
• Exposes full iterative-lookup shortlist sorted by XOR distance.
• Used for spillover beyond top-k candidates.

libp2p/protocols/kademlia/provider.nim

• Added module-level documentation for both modes (standard and overload-protection).
• Changed:
  • dispatchAddProvider return type:
    • From: Result[void, string]
    • To: Result[AddProviderStatus, string]
  • Under the flag:
    • Reads receiver response.
    • Falls back to accepted for backward compatibility.
• Modified addProvider:
  • Performs full iterativeLookup.
  • Builds larger candidate pool.
  • Sends ADD_PROVIDER in XOR-distance order.
  • Performs spillover when a batch is unanimously rejected.
• Modified handleAddProvider:
  • Enforces maxProvidersPerKey.
  • Exempts re-advertisements by known providers.
  • Sends acceptance/rejection responses.
  • Updates metrics.
• Added helper:
  • sendAddProviderResponse to centralize response writing.

libp2p/protocols/kademlia/kademlia_metrics.nim

• Added metrics:
  • kad_provider_rejections_sent
  • kad_provider_spillover_rounds

tests/libp2p/kademlia/test_add_provider.nim

• Added:
  • {.define: kadProviderRejection.} to enable flag during tests.
• Added five async tests:
  • Per-key limit rejection
  • Same-provider refresh exemption
  • Spillover to farther node
  • Normal replication when all nodes accept
  • Protobuf round-trip for AddProviderStatus

---

Affected Areas

• Gossipsub
• Transports
• Peer Management / Discovery
• Protocol Logic:
  • Kademlia ADD_PROVIDER handling now supports optional rejection responses and spillover logic
• Build / Tooling
• Other

---

Compatibility & Downstream Validation

• Fully backward compatible.
• Protobuf field 10 is optional:
  • Not written without the flag.
  • Unflagged peers decode messages unchanged.
• dispatchAddProvider change:
  • Internal (not exported)
  • No external impact
• Flagged → unflagged interaction:
  • Missing response treated as accepted
  • Preserves interoperability

Downstream Consumers

• Unaffected unless opting in:
  • Nimbus
  • Waku
  • Codex
• Opt-in users should:
  • Set maxProvidersPerKey
  • Validate behavior under rejection load

---

Impact on Library Users

• AddProviderStatus:
  • New public enum (additive)
• Message.providerStatus:
  • New optional field
  • none() unless flag is active and message is a reply
• KadDHTConfig.maxProvidersPerKey:
  • Default: 0 (unlimited)
  • No effect without flag
• KadDHTConfig.new:
  • New optional parameter
• LookupState.allSortedPeers():
  • New public method
• Metrics:
  • kad_provider_rejections_sent
  • kad_provider_spillover_rounds
• dispatchAddProvider:
  • Internal change only

---

Risk Assessment

Iterative Lookup Overhead

• addProvider uses iterativeLookup under the flag.
• Adds extra round-trip and network traffic.
• Low risk:
  • Opt-in only
  • Same traversal already used by getProviders

Response Handling

• Sender waits for response after ADD_PROVIDER.
• If receiver fails or is unflagged:
  • Read error treated as accepted
• Low risk:
  • Correct fallback
  • Tested under partial failure

Test Flag Scope

• {.define: kadProviderRejection.} applies to entire compilation unit.
• All ~115 Kademlia tests run under the flag.
• Low risk:
  • Existing tests verified under flag

Re-advertisement Exemption

• Providers can always refresh their own record.
• Limit applies to distinct providers, not messages.
• Low risk:
  • Behavior documented and tested

---

References

https://github.com/vacp2p/roadmap/pull/445/changes#diff-40a4fb71e95fc9a8144b7c1f1e8ac53f49cb50ddf996d67c960b23d362b68bb1

[codecov-commenter]

Codecov Report

❌ Patch coverage is 82.53968% with 22 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.54%. Comparing base (b1e744a) to head (2a828c5).

Files with missing lines	Patch %	Lines
libp2p/protocols/kademlia/provider.nim	82.30%	20 Missing ⚠️
libp2p/protocols/kademlia/find.nim	75.00%	1 Missing ⚠️
libp2p/protocols/kademlia/protobuf.nim	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2394      +/-   ##
==========================================
- Coverage   73.54%   73.54%   -0.01%     
==========================================
  Files         168      168              
  Lines       22349    22522     +173     
  Branches       20       20              
==========================================
+ Hits        16436    16563     +127     
- Misses       5913     5959      +46     

Files with missing lines	Coverage Δ
libp2p/protocols/kademlia/types.nim	82.32% <100.00%> (-1.02%)	⬇️
libp2p/protocols/kademlia/find.nim	68.34% <75.00%> (-0.31%)	⬇️
libp2p/protocols/kademlia/protobuf.nim	78.99% <83.33%> (-1.28%)	⬇️
libp2p/protocols/kademlia/provider.nim	72.74% <82.30%> (+2.49%)	⬆️

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

This PR adds an opt-in Kademlia overload-protection path for provider records: receivers can reject ADD_PROVIDER requests when a per-key limit is reached, and advertisers can attempt spillover to farther peers. It extends the Kademlia message/config surface and updates tests/metrics around the new behavior.

Changes:

• Added provider rejection status to Kademlia protobuf messages and a new maxProvidersPerKey config option.
• Changed ADD_PROVIDER handling/sending logic to support rejection responses, spillover, and shared batch-waiting behavior.
• Enabled and expanded Kademlia tests for the new feature, plus added related metrics.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/libp2p/kademlia/utils.nim	Adds a test helper to send ADD_PROVIDER and inspect the returned status.
tests/libp2p/kademlia/test_encoding.nim	Adds protobuf round-trip coverage for AddProviderStatus.
tests/libp2p/kademlia/test_add_provider.nim	Adds rejection/spillover behavioral tests and test-only setup changes.
tests/config.nims	Enables the new Kademlia provider-rejection compile flag in test builds.
libp2p/protocols/kademlia/types.nim	Adds config for per-key provider limits and a shared async batch helper.
libp2p/protocols/kademlia/put.nim	Refactors put batching to use the new shared wait helper.
libp2p/protocols/kademlia/provider.nim	Implements response-aware ADD_PROVIDER, rejection handling, and spillover.
libp2p/protocols/kademlia/protobuf.nim	Adds protobuf enum/field support for provider acceptance/rejection status.
libp2p/protocols/kademlia/kademlia_metrics.nim	Adds counters for provider rejections and spillover rounds.
libp2p/protocols/kademlia/get.nim	Refactors get-path batch waiting to use the shared helper.
libp2p/protocols/kademlia/find.nim	Exposes the full lookup shortlist in sorted order for spillover candidate selection.
libp2p/protocols/kademlia.nim	Adds constructor-time handling for the new config in flagged vs. unflagged builds.