[FYI] New open-source “Resonate protocol” in development for multi-room audio streaming synchronized music playback

[Hedda]

Just a heads up to other media player developers; not sure if this information will be usable to other music player projects, but FYI and for reference; several developers from Open Home Foundation, Music Assistant, ESPHome, and Home Assistant projects have started early development working on a new open-source audio streaming and control protocol with a goal to achieve the best possible multi-room music playback experience using 100% open-source code, though initially targeting streaming synchronized audio playback to multiple embedded media player appliances (i.e. music streamers), voice assistants, and smart speakers:

• https://github.com/Resonate-Protocol (Resonate Protocol organization on GitHub including contact information for collaboration)
  • https://github.com/Resonate-Protocol/spec (Specification of the Resontate protocol)
  • https://github.com/Resonate-Protocol/aioresonate (Async Python library implementing the Resonate Protocol.)
  • https://github.com/Resonate-Protocol/audio-sdk-js (Proof of concept of how Improv Audio can work in TypeScipt/JavaScript)

While the "Resonate specification is design from scratch it shared similarities to Snapcast and Squeezelite at a high-level concept, Resonate is not a stand-alone player, but an extension that can turn other existing audio players into a Sonos-like multiroom solutions with time synchronized with the server to play perfectly synced audio. In their own tests they claim to see that average time deviation is below 0.05ms for synced audio (with current testing showing 50 microseconds between two ESP32-S3 devices running ESPHome and connected over WiFi within a local LAN).

"Resonate is a multi-room music experience protocol. The goal of the protocol is to orchestrate all devices that make up the music listening experience. This includes outputting audio on multiple speakers simultaneously, screens and lights visualizing the audio or album art, and wall tablets providing media controls." "Definitions; Server, a Resonate server. Orchestrates all devices. Generates an audio stream, manages all the players, provides metadata etc.. Player, a Resonate client that can play audio, visualize audio or album art or provide music controls".

The ESPHome firmware project is getting a new experimental audio client component (now a draft pull reuest so other developers can test) that adds support for the Resonate Protocol for synchronized music-playback across multiple ESPHome-based devices, i.e. audio synchronizer to enable sound-sync timing for DIY multi-room audio systems:

• WIP: Resonate synchronized audio component esphome/esphome#10183

⚠️ Note! This is a development snapshot - NOT ready for production use. This PR is intended for testing and feedback purposes only. ⚠️

For a WIP reference implementation of a server using their aioresonate (Async Python library see how it is implemented in Music Assistant:

• https://github.com/music-assistant/server/tree/resonate/music_assistant/providers/resonate

Implementation Details:

• Time Synchronization:

  • Employs a Kalman filter to model and compensate for internal clock drift between the client and server
  • Dynamically tracks and adjusts for each device's clock characteristics to maintain tight synchronization
  • In preliminary testing, achieved a median audio synchronization error of approximately 50 microseconds between two ESP32-S3 devices connected over WiFi.

• Architecture:

  • The resonate component implements a flexible hub architecture that allows devices to participate in the audio listening experience with different levels of functionality:
    • Text Sensor: Display track metadata without audio playback (e.g., on screens)
    • Media Player: Output synchronized audio
    • Extensible Design: Ready for future integrations like audio visualization

• mDNS:

  • Implements mDNS advertisement for the component's WebSocket server
  • Enables automatic discovery by resonate servers

• External Dependencies:

  • Integrates esp-libopus (WIP IDF component) for Opus audio codec support

• Testing:

  • Currently compatible with:

    • Music Assistant Server - feat/improv-audio branch
      • Note: This implementation has known issues but provides basic testing capabilities for synchronized playback

  • Example entry for config.yaml:

    • Example Voice Preview Edition firmware is available here: https://github.com/esphome/home-assistant-voice-pe/blob/66f7fdea2e73ef91156ad579aeeb768ecdfa87e1/home-assistant-voice.yaml (requires building with ESPHome dev branch)

• Current Status:

  • August 28-29th updates adds several important changes:
    • Now compatible with the Music Assistant server's resonate branch
      • https://github.com/music-assistant/server/tree/resonate/music_assistant/providers/resonate
    • Fixes issues with rapid stopping and starting (requires using the mixer and resampler speaker platform changes in this PR)
    • Improves memory handling and safety
    • Adds groundwork to support cover art as an image component in ESPHome
      • This PR is intended for testing and feedback purposes only. The implementation contains:
        • Numerous TODO items for simplification and optimization
        • Extensive debug logging for diagnosing synchronization issues
        • Incomplete features pending server-side support (metadata processing, FLAC decoding)
        • Breaking changes expected in future iterations
        • Modified Components

• Future Work:

  • Protocol refinement and stabilization
  • Enhanced audio stack to support additional synchronized audio protocols
  • Complete metadata and FLAC support
  • Production-ready implementation

There was some discussions + further explainations on its concept in a other thead that asked about this new multi-room streaming protocol:

• https://github.com/orgs/music-assistant/discussions/3883

Originally posted by marcelveldt in music-assistant/support#3883 (comment)

"we decided to create (yes, yet another!) protocol. One that would fulfill all our needs and would work across the internet and suitable for low powered devices like the ESP32. It will support (perfect) synced playback by default but can also switch codecs on the fly, making it suitable to stream to a (remote) browser. Based on websockets ensures it can travel across firewalls, reverse proxies and whatnot. It's a bit of shame to not use the strength of an existing audio streaming protocol (snapcast) but we ran into so much trouble it became more of a burden than a help. I even considered rewriting Snapcast from scratch in python, so keep the binary protocol the same but just re-implement it with our issues fixed. I'll make sure we publish the details somewhere soon, currently its still in ideation/POC phase but looking very good. "Be aware that this is nowhere near ready to test something - we are still defining the spec and doing some PoC implementations to test the theory."

If I understand correctly this new Resonate synchronized audio component will add a flexible hub architecture for audio time synchronization

• https://github.com/Resonate-Protocol/spec

This could solve the problem that you normally need more expensive hardware solution if you want to add multi-room audio to existing speakers, as once this is implemented fully you could have almost perfect multi-room audio syncronization using something like the Home Assistant Voice Preview Edition connecte to external speakers using its stereo output jack:

• https://www.home-assistant.io/voice-pe/

But a more relevant follow-up feature request for Music Assistant when this implemented would be if could add support for different types of multi-room audio, (to have options for single source-single zone, single source-multiple zone, or multiple source-multiple zone playback)?

• https://en.wikipedia.org/wiki/Multi-room_audio

Ongoing discussion about this new experimental Resonate audio component is going on in this Discord channel here:

• https://discord.com/channels/753947050995089438/1369735163629801532

KevinA:
"Luckily math is here to save the day 😆 . A Kalman filter is a nice way to combine measurements with different uncertainties, which is exactly what we have! Basically, if the most recent message has a very low delay, then we weigh its value much more than if it had a very high delay. It gives us a much more stable offset than using the median, but it also gives us an estimated error after each update. This means the client can estimate how accurate its own offset is! This would allow us to slow down or increase the pace we send the time messages. The more confident we are in the offset, the less we need to send - and vice versa. This will be especially helpful when we first connect and need to spam a bunch of time messages to quickly build our confidence in the computed offset.

So being accurate is important, of course, but being stable is also very important. The median was problematic, even if its inaccuracy wasn't audible. With the median filter, the offset would shift by over 200 microseconds, on average, per update, which means the client would have to constantly be tweaking the audio to try to stay in sync. With very big jumps, it would require hard syncing which is audible.

The Kalman filter modeling the drift (of 2 seconds per day) had an average offset change of only ~25 microseconds per update. Keep in mind, if the clock does drift 2 seconds per day, we would expect it change 23 microseconds per update if the updates are 1 second apart. If there is no systematic drift in the client clock, the Kalman filter had an average change of less than 1 microsecond per update."

Btw, looking at the commit history it seems like "Resonate" used to previously be refered to as "improv-audio" internally by its developers:

• music-assistant/server@dev...feat/improv-audio

PS: Somewhat ironically the Open Home Foundation recenrly posted a newsletter blog post titiled "Why isn't there one standard?" and then they choose to use xkcd's "standards" classic XKCD comic strip on how standards proliferate as their tumbnail for the Resonate organization on GitHub, however with the Open Home Foundation, Home Assistant , Music Assistant, and Nabu Casa founder backing this they probably have the influence to at least convince many open-source based media layer implementations to also add support for this new protocol:

• https://newsletter.openhomefoundation.org/why-isnt-there-one-standard/

FYI, marcelveldt who is lead developer on this mentioned in this other thread that there is a wish to someday also see the Resonate protocol supported by third-party player clients (e.g. media player appliances like media streamers, open-source voice assistants and smart speakers):

• https://github.com/orgs/music-assistant/discussions/3883

"I'm not sure if big tech will implement this new protocol, but we sure aim to convince the smaller/medium sized companies like WiiM to implement it and make it easy enough to do"

[MaxKellermann]

Oh no, WebSocket! That means wrapping the thing in HTTP and switching protocols to WebSocket is mandatory, just to have something that a raw TCP connection can do as well.

"ensures it can travel across firewalls" .... but how does mDNS travel across these firewalls?

No, thanks.

[Hedda]

Somewhat ironically the Open Home Foundation recenrly posted a newsletter blog post titiled "Why isn't there one standard?" and then they choose to use xkcd's "standards" classic XKCD comic strip on how standards proliferate as their tumbnail for the Resonate organization on GitHub, however with the Open Home Foundation, Home Assistant , Music Assistant, and Nabu Casa founder backing this they probably have the influence to at least convince many open-source based media layer implementations to also add support for this new protocol:

• https://newsletter.openhomefoundation.org/why-isnt-there-one-standard/

FYI, marcelveldt who is lead developer on this mentioned in this other thread that there is a wish to someday also see the Resonate protocol supported by third-party player clients (e.g. media player appliances like media streamers, open-source voice assistants and smart speakers):

• https://github.com/orgs/music-assistant/discussions/3883

"I'm not sure if big tech will implement this new protocol, but we sure aim to convince the smaller/medium sized companies like WiiM to implement it and make it easy enough to do"

By the way, I have previously tried (and so far failed) to encurage developers from Music Assistant. Home Assistant, and other similar projects to at least look at help improving the the new "Matter Casting" (a.k.a. MatterCast) specification, which also is in early development, so see if could try make it fit both commercial and fully open-source implementations. For reference you can see related requests as my attempt to convince other to do with that what I do not have the skills or capacity to do myself:

• [Feature] Matter Casting support for "Audio Player Architecture" with a new "Casting Audio Player device type" and "Audio Player endpoint" ("Casting Audio Player" and adding "Basic Audio Player") + look into adding multi-room music streaming for "Matter Speakers"? project-chip/connectedhomeip#31389

• https://github.com/orgs/music-assistant/discussions/2342

• https://community.home-assistant.io/t/matter-casting-matter-casting-client-support-for-to-in-home-assistant-voice-assistants-new-open-protocol-standard-for-local-video-and-audio-streaming/671645/

• https://github.com/orgs/OpenVoiceOS/discussions/293

Personally think that an open-source “Matter Casting” server/client project would have a greater chance of succeeding with wide adoption.

For the record, yes I am Home Assistant enthusiast but I am also a fan of open technical standards being adopted as industry standards, meaning the combined situation where an open protocol/specification standard that is openly accessible and usable by anyone is widely accepted by established companies to ensure better interoperability and cohesivity in this specific area between both big tech products and open-source software projects. Without a larger open standard for open-source project to unite behind they get even more fractured which will make most average users continue to just buy products that uses propriatory protocols like AirPlay, Google Cast or Sonos.