Music-JEPA: Learning a World Model of Sound from Action

Ziyu Wang, Kun Fang, Yann LeCun

New York University  •  MBZUAI  •  McGill University  •  CIRMMT  •  AMI Labs

This is the demo page for the paper Music-JEPA: Learning a World Model of Sound from Action. We model the temporal dynamics of music audio in latent space, conditioned on action, using a Joint Embedding Predictive Architecture (JEPA). This page provides additional audible materials to supplement the submission. The demo page is organized into three parts:

Part 1: Action-Conditioned Prediction

To reveal the temporal dynmaics of Music-JEPA, we provide samples (corresponding to Figure 3) showing the transition from the current audio state $s_{t}$ (spectrogram) and an action state $a_{t+1}$ (pianoroll and pedal) to the resulting audio state $s_{t+1}$. The ground-truth and prediction rows use the same action, while the counterfactual row uses a modified action. Predictions are performed in latent space and then decoded into spectrograms using a separate decoder.

The audio below each spectrogram is synthesized from the reconstructed spectrogram via the Griffin–Lim algorithm, while the audio below each pianoroll is rendered from MIDI using a standard soundfont. These reconstructions are not intended for high-fidelity audio reproduction, but rather to illustrate the learned latent representations and temporal dynamics. As phase is not explicitly modeled, the resulting audio may exhibit perceptual artifacts.

Click the tabs to switch between samples.

Part 2: Piano Transcription via Planning

Music-JEPA can achieve piano transcription via planning in the latent space (as discussed in Section 4.5). Below are comparisons of transcription outputs on selected audio examples. Each row shows the original audio and synthesized transcription outputs from our method, Yan et al. [2], Kong et al. [3], and the ground truth MIDI.

Transcribed MIDI files are exported to audio using the default piano sound in GarageBand. We do not claim state-of-the-art transcription accuracy; our outputs can contain more errors. However, they are often more expressive and may better reflect the perceived sound.

Part 3: Transcription on Out-of-Distribution (OOD) Samples

In addition to standard piano transcription, we present results on OOD audio. We evaluate on samples from the Piano Concerto Dataset (PCD) [4], which contains both piano and orchestral instruments, and compare against the supervised piano transcription methods of Yan et al. [2] and Kong et al. [3]. As these samples fall outside the typical training distribution, none of the methods are expected to perform optimally.

Interesting observations: supervised transcription methods (Yan et al. and Kong et al.) tend to focus primarily on the piano component, whereas Music-JEPA often produces piano actions that also reflect the surrounding orchestral texture, resulting in outputs that more closely match the overall audio.

References

  1. H. Zhang, K. Fang, Z. Wang, and I. Fujinaga, “ High-resolution sustain pedal depth estimation from piano audio across room acoustics ,” in Proceedings of the 26th International Society for Music Information Retrieval Conference (ISMIR), 2025.
  2. Y. Yan and Z. Duan, “ Scoring time intervals using non-hierarchical transformer for automatic piano transcription ,” in Proceedings of the 26th International Society for Music Information Retrieval Conference (ISMIR), 2024.
  3. Q. Kong, B. Li, X. Song, Y. Wan, and Y. Wang, “ High-resolution piano transcription with pedals by regressing onset and offset times ,” IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), 2021.
  4. Y. Özer, S. Schwär, V. Arifi-Müller, J. Lawrence, E. Sen, and M. Müller, “ Piano Concerto Dataset (PCD): A Multitrack Dataset of Piano Concertos ,” Transactions of the International Society for Music Information Retrieval (TISMIR), 2023.