Abstract

Video-to-Audio (V2A) generation requires balancing four critical perceptual dimensions: semantic consistency, audio-visual temporal synchrony, aesthetic quality, and spatial accuracy; yet existing methods suffer from objective entanglement that conflates competing goals in single loss functions and lack human preference alignment. We introduce PrismAudio, the first framework to integrate Reinforcement Learning into V2A generation with specialized Chain-of-Thought (CoT) planning. Our approach decomposes monolithic reasoning into four specialized CoT modules (Semantic, Temporal, Aesthetic, and Spatial CoT), each paired with targeted reward functions. This CoT-reward correspondence enables multidimensional RL optimization that guides the model to jointly generate better reasoning across all perspectives, solving the objective entanglement problem while preserving interpretability. To make this optimization computationally practical, we propose Fast-GRPO, which employs hybrid ODE-SDE sampling that dramatically reduces the training overhead compared to existing GRPO implementations. We also introduce AudioCanvas, a rigorous benchmark that is more distributionally balanced and covers more realistically diverse and challenging scenarios than existing datasets, with 300 single-event classes and 501 multi-event samples. Experimental results demonstrate that PrismAudio achieves state-of-the-art performance across all four perceptual dimensions on both the in-domain VGGSound test set and out-of-domain AudioCanvas benchmark.

The PRISMAUDIO Framework

Overview of PrismAudio. Left panel: the progress of CoT training data construction using Gemini 2.5 Pro and then fine-tuning VideoLLaMA2 for decomposed CoT generation. Right panel: the Fast-GRPO multi-dimensional CoT-RL framework for post-training the Audio Foundation Model.

Comparison with Baselines in Multi-Event Scenarios on the AudioCanvas Benchmark

Click any card below to load the full high-fidelity comparison and Decomposed Chain-of-Thought analysis for that scene.

Chopping, Thudding

Machining, Tapping

Ice Cutting, Freezer Shutting

Unlocking, Pouring

Smoothing, Flipping

Plastic Pop Opening

Unrolling and Tearing Tape, Smoothing

Piercing, Rubbing

Exchanging Soap, Pouring

Comparison with Baselines in Single-Event Scenarios on the AudioCanvas Benchmark

Click any card below to load the full high-fidelity comparison and Decomposed Chain-of-Thought analysis for that scene.

Trapped Cat

Accelerating, Revving, Vroom

Field Recording

Playing Double Bass

Baby Laughter

Firing Machine Gun

Carpet Cleaning

Rail Transport

Neigh, Whinny

Comparison with Baselines on VGGSound

Click any card below to load the full high-fidelity comparison and Decomposed Chain-of-Thought analysis for that scene.

Baltimore Oriole Calling

Playing Electric Guitar

Playing Drum Kit

Playing Badminton

Car Passing by

Stream Burbling

People Giggling

Panda Sneezing

People Laughing

Quantitative Results

In-domain Evaluation on VGGSound

Out-of-Domain Evaluation on AudioCanvas

BibTex

@misc{liu2025prismaudiodecomposedchainofthoughtsmultidimensional,
          title={PrismAudio: Decomposed Chain-of-Thoughts and Multi-dimensional Rewards for Video-to-Audio Generation}, 
          author={Huadai Liu and Kaicheng Luo and Wen Wang and Qian Chen and Peiwen Sun and Rongjie Huang and Xiangang Li and Jieping Ye and Wei Xue},
          year={2025},
          eprint={2511.18833},
          archivePrefix={arXiv},
          primaryClass={cs.SD},
          url={https://arxiv.org/abs/2511.18833}, 
    }