- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Mean Jacobian, MaJa, a technique that speeds up and matches/improves on decoder-gradient guidance in audio models by removing the need to use the decoder for the update. MaJa relies on the average jacobian computed on a calibration set of the decoder’s gradient for a specific attribute, and a small approximation head of the attribute itself. MaJa’s updates are not perfect approximations of the decoder’s updates, achieving lower cosine similarities than another technique shown in the paper but, despite this, managing to match/increase the guidance’s results, while being 14x faster than the use of a decoder.
Critical strengths
- MaJa is a very clean and understandable technique, which not only has good performances, but is also presented with an auxiliary test to assess whether it is likely to apply.
- The authors try their best to provide adversarial baselines to MaJa, going so far as to train a head that matches the decoder at a cosine of 0.99, nonetheless getting worse results than with MaJa.
- The authors are honest about the work, as they do report where MaJa “fails” (it doesn’t surpass the decoder gradient), namely two attributes (new prompts) on ACE-step Turbo.
Critical weaknesses
- The paper’s writing is terse, the tables could use a bit of a glow up and overall understanding the results is difficult because so many things are scattered and send a reader to the appendix.
- Although extremely explored, the scope of the experiments is a bit narrow, only 5 attributes are explored, across two generators. Furthermore on one of the two generators (ACE) the two losses appear on both the only two attributes tested (on only one of the two types of prompts reported).
Questions for authors
- Could the authors provide results on a third few-step audio model? Even with few attributes tested?
- In Table 7 AudioX gets lots of advantage in running in the later steps, could you show whether the same happens with ACE? Furthermore, do you have an explanation for why ACE doesn’t see the same results as AudioX in Table 1?
2 comments
(continued)
It is not that the Jacobian varies more along ACE's trajectory: one matrix per guidance time, or one per cluster of states, changes the result by at most 0.03 dB. The final version will report these analyses and recommend building the matrix from at least 48 states.
Stable Audio Open Small, per attribute. MaJa error minus decoder-gradient error is shown with the sign flipped, so positive means MaJa is better (dB, 95% CI):
| Attribute | New prompts | Unseen instruments |
|---|---|---|
| Loudness gain | +0.38 [+0.28, +0.50] | +0.23 [−0.03, +0.47] |
| Brightness shelf | +0.36 [−0.12, +1.15] | +0.04 [−0.57, +0.57] |
| High-band gain | +0.12 [−0.12, +0.36] | +0.19 [+0.07, +0.31] |
| Spectral centroid shift | +0.08 [−0.07, +0.22] | −0.19 [−0.37, −0.02] |
| Onset shape | −0.03 [−0.08, +0.02] | −0.06 [−0.20, +0.09] |
A random matrix of the same shape changes the error by at most 0.05 dB on every row.
Response to Reviewer D8S3
We sincerely thank the reviewer for the careful and detailed reading, and for recognising the screen, the adversarial baselines and the reporting of the ACE results. The questions about ACE were particularly valuable to us: following them up showed where the late-step advantage comes from and how the precomputed matrix should be built. We report these findings below.
W1: presentation. We thank the reviewer for this comment and take it on board. Many results were placed in the appendix so that each anticipated objection, including the adversarial baselines the reviewer kindly highlighted, could be answered in full; we accept that this left too much of the reading in the appendix. Please see point 1 of the general response. In the final version the main table will be redesigned so that the results are no longer scattered across the appendix.
W2 / Q1: a third few-step model, and the two ACE losses. This is a very fair request: a third model is the most direct test of whether the result is general, and we thank the reviewer for suggesting it.
Q2a: does the late-step advantage of Table 7 hold on ACE? We thank the reviewer for this question, which turned out to be very informative. The advantage holds on the two generators that re-noise their clean prediction at every step, and on ACE it depends on the sampler and on the step size. We ran three checks.
The late-step pattern is therefore tied to re-noising samplers, and on ACE's deterministic sampler late-only MaJa matches the decoder gradient once the step size is chosen per method.
Q2b: why does ACE differ from AudioX? For loudness the difference is small: on 48 prompts MaJa is within the ±0.2 dB equivalence bound (above). For brightness, a large part of the gap comes from the size of the averaged matrix, which we tested directly.
The ACE brightness matrix in Table 1 averages 24 states at three guidance times. Built from 48 or 80 states covering more guidance times (the recipe we use for ACE loudness), MaJa's brightness error drops by 0.14 to 0.23 dB on every evaluation set, on 8 of 8 or 10 to 11 of 12 prompts. MaJa then moves ahead of the decoder gradient on unseen classes (+0.22 dB [+0.07, +0.43]; +0.30 dB with late-only guidance), and on new prompts the gap narrows from −0.40 dB with the 24-state matrix to −0.17 dB [−0.33, −0.02] on the same prompts.