- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies test-time scaling (TTS) for video world models from a compute-value perspective. It introduces the Compute-Value Audit (CVA) framework, which decomposes the effectiveness of TTS into four requirements: opportunity, state confidence, action value, and compute cost. Through extensive experiments across video generation and reasoning benchmarks, the paper shows that increasing the number of generated candidates can substantially improve oracle quality without translating into meaningful improvements in the quality of the selected output.
Critical strengths
I appreciate the paper's perspective on test-time scaling. The authors distinguish between the potential improvement available in a candidate pool and the improvement that can actually be recovered by a practical selection mechanism. I find this distinction refreshing and important, especially in the context of video world models, where evaluating physical correctness remains challenging.
Critical weaknesses
-
The contribution remains largely diagnostic rather than constructive. While I find the problem formulation interesting, the paper primarily focuses on evaluating and diagnosing the limitations of existing TTS mechanisms. It remains unclear how the proposed framework can directly inform the design of better TTS algorithms for video world models. In particular, the paper does not provide a convincing new mechanism that effectively closes the gap between sampling headroom and realized selection gain in this domain.
-
The main conclusions are somewhat expected, despite the extensive experimental analysis. The central observations—that better candidates do not necessarily lead to better selected outputs, that imperfect verifiers can limit the benefits of additional sampling, and that adaptive computation must outperform compute-matched baselines—are largely consistent with existing insights about verifier alignment and test-time optimization.
Questions for authors
Can the authors demonstrate a case in video world models where the CVA framework leads to a new or improved TTS strategy, rather than merely diagnosing the failure of existing approaches?
4 comments
The resulting improvement is IQ, with a paired 95% confidence interval (CI) of over Physics-IQ event families. This interval applies to the displayed fixed-width comparison. The two policies share a mean-cost cap, but their actual expenditure differs; the cap does not constrain each scene's realized expenditure.
The proposed policy also exceeds the IQ point estimate of uniform WMReward at ( IQ) while using 35.7% less normalized compute ( versus units).
This establishes a positive quality–cost trade-off against the displayed fixed-width baselines. State-independent randomized width policies and a matched-random allocation control are not reported for the new video policy. The result therefore does not complete the full allocation audit required by CVA or isolate the benefit of assigning compute according to state.
The policy and WMReward baseline use the same prompts, conditioning inputs, learned verifier, and candidate-generation model. The evaluated policy changes how computation and selection signals are used across scenes. The aggregate comparison does not attribute the gain separately to motion conditioning, conditional-utility estimation, or score combination. The revised appendix distinguishes this policy objective from a complete executable specification and does not supply unreported parameter values.
The original twelve adaptive-depth controls relied on the previously evaluated score-only state signals and did not pass G4. The new policy combines motion-state information, learned-dynamics verification, and explicit marginal-cost calibration. Its fixed-width comparison tests a richer state–action class while retaining the additional controls required for a full allocation-audit pass.
W2: Contribution beyond previously known verifier-alignment observations
The novelty of CVA is the operational separation of four conditions that are frequently conflated when evaluating inference-time scaling.
Prior TTS research establishes that search and verification are important. CVA determines whether the available opportunity, observable state, actionable intervention, and complete inference economics jointly support a deployable improvement.
The experiments establish that these properties are distinct.
G1: Opportunity. On Physics-IQ, increasing Wan2.2-5B width from to creates oracle IQ points. Thus, failure to improve selected quality is not explained by the absence of better candidates.
G2: State confidence. On MMLU-Pro, disagreement identifies harder prompts, producing a percentage-point state separation, but this predictive signal alone does not establish useful additional sampling.
G3: Action value. Anchor–explorer on PRM800K demonstrates a positive action-value case, improving by percentage points over matched random allocation and over matched uniform allocation. Conversely, the original video refinement experiments fail to produce reliable gains.
G4: Complete entry fee. None of the twelve original video adaptive-depth policies exceeds matched uniform computation; they recover only 42–69% of the measured fee.
The constructive experiment demonstrates how these diagnoses translate into algorithm design. Since G1 passes but existing video verification and allocation are insufficient, the new policy improves the information used for selecting actions and explicitly prices every additional inference operation.
What the comparisons show
The results compare verifier choice, sampling width, and the combined allocation policy.
First, the learned-dynamics verifier improves fixed-pool selection: WMReward reaches IQ at , compared with for G+M. This establishes the value of stronger physical verification.
Second, the multi-budget results demonstrate diminishing selected-quality returns from increasing uniform sampling width. G+M improves from at to only at , despite the substantial increase in oracle quality.
Third, the CVA-guided policy combines these observations to allocate additional computation selectively, reaching IQ at a mean cost substantially below uniform learned-verifier inference.
Thank you once again for taking the time to review our manuscript. We have carefully addressed all the concerns raised in your review and provided detailed responses to fully resolve each of the points raised.
As the discussion period is coming to a close, we would like to kindly follow up on our previous response in case you have not yet had a chance to review it. We appreciate the many demands on your time and would be grateful for any opportunity you may have to review our responses.
Thank you once again for your time and consideration!
Thank you for the clarifications and additional experiments. The new video results partially address my concern about CVA’s practical value. However, as acknowledged in Appendix P, the new policy still lacks important implementation details, and the experiments do not clearly show how much state-aware allocation contributes to the improvement. These issues leave my main concern only partially addressed, so I maintain my rating.
We thank the reviewer for recognizing the importance of distinguishing sampling headroom from deployable selection gain. CVA denotes the Compute-Value Audit, and TTS denotes test-time scaling. We provide a constructive video comparison motivated by the audit's diagnoses.
W1 & Q1: A constructive video TTS strategy derived from CVA
We introduce and evaluate a CVA-guided, state-aware compute allocation strategy that converts the audit's diagnostic findings into an improved video TTS policy.
The design follows directly from the original CVA results.
On Wan2.2-5B, increasing the candidate pool from B=4 to B=16 raises the Physics-IQ quality score (IQ) of the oracle from 35.09 to 44.32, but Native cycle selection improves only from 27.53 to 27.79. Thus, increasing sampling width alone has limited value when the additional candidates cannot be identified reliably.
The original G+M selector, which combines global history evidence (G) and the motion signal (M), improves the B=4 selected quality to 30.81 IQ, but its motion component is not universally reliable. On ABot-14B Dynamic scenes, M achieves 39.86 IQ against a no-TTS baseline of 40.80, whereas G reaches 42.28. This motivates verification conditioned on the physical requirements of the scene.
These observations motivate two coupled design changes.
1. State-aware verification.
The policy estimates motion requirements and verifier disagreement from observable prompts, conditioning frames, and generated candidate statistics. It chooses between the inexpensive G/M signals and additional learned-dynamics verification according to their calibrated conditional utility.
Rather than uniformly applying the same verification rule, the policy emphasizes motion-preserving evidence on dynamic interactions and avoids unnecessarily expensive verification when inexpensive signals already agree.
2. Value-aware compute allocation.
The policy objective balances expected quality improvement against incremental normalized inference cost.
For state s and action a, we define
U(a∣s)=E[ΔQ(a)∣s]−λC(a∣s),
where ΔQ is the improvement over the current selected output, C is the incremental normalized inference cost, and λ controls the quality–compute trade-off. This equation states the allocation objective; the frontier evaluates the combined policy, without isolating the effects of its components.
The action set contains accepting the current candidate, generating another candidate, and invoking the learned-dynamics verifier. Conditional utilities are calibrated using family-disjoint data, and all selection decisions are made without access to the current query's target IQ or privileged future frames.
The policy uses at most eight generated candidates per scene, with an average of 5.6 candidates and 4.0 learned-verifier calls. Its mean normalized cost is 144 Diffusion Transformer (DiT)-equivalent units, including 112 generation units and 32 verification units. The frontier uses fixed relative operation prices for generation and learned-verifier inference; the resulting cost ratios are distinct from strict same-model forward counts and measured latency or energy.
Constructive evaluation: quality versus complete inference cost
The CVA-guided policy improves selected quality over the best reported fixed-width uniform baseline under a common mean-cost cap, and uses less normalized compute than the wider learned-verifier baseline.
At a mean cost of 144 units, CVA-guided selection achieves 32.36 IQ. The best reported fixed-width uniform WMReward baseline under this mean-cost cap uses five candidates and costs 140 units, reaching 31.79 IQ.