Measure Before You Operate: Layer-Wise Gradient Surgery in Audio-Video Diffusion RL Is Necessary on Some Backbones and Inert on Others
Abstract
OmniNFT, the current recipe for RL post-training of joint audio–video diffusion models, protects the audio tower from video-loss gradients with a per-block scale (“layer-wise gradient surgery”) whose constants were tuned on one backbone, LTX-2, and published as block indices. We ask three questions: where the relevant blocks are on other backbones, whether the surgery is needed at all, and whether its exact placement matters. Two probes that take hours rather than training runs locate the synchronization-critical blocks and measure per-block gradient leakage. On five architectures the zones sit at different relative depths, and no index-copying or rescaling heuristic recovers them; on LTX-2 itself our probes do not recover the published window. Within a model line the zones do not move under SFT, weight souping or RL. Sixteen controlled RL arms on three backbones then give different answers to the other two questions. Whether surgery is needed depends on the backbone: on a 29B model, fifty epochs of RL without it leave word error rate 39% above the SFT start while the training reward keeps rising, and the calibrated schedule from the same start lowers WER by 25% (); on a 3B model and on the public Ovi, none of the schedules, including no surgery, can be told apart. Where the surgery is placed cannot be resolved at this budget on any backbone: differences between schedules are within noise and change sign between prompt sets. A first-order argument shows that the surgery acts through the product of leakage and gradient conflict, which explains the inert cases but not the 3B one. The recipe also contains the first speech-intelligibility reward for joint AV RL, which lowers WER by 28% on public Harvard sentences and 48% on the VABench speech subset, with smaller gains under independent ASR judges. Probe code and the public evaluation protocol will be released.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.