- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TTPE, a training-free method for iterative text-to-video refinement. TTPE stores previous refinement trajectories, retrieves similar cases, estimates the expected quality improvement and generation cost of each action, and uses these estimates to decide whether and how to refine the current video.
The problem is important, and reusing previous test-time experience is a sensible idea. However, the mathematical formulation has several fundamental problems. In particular, the reward does not match the final objective, and the quantities called “advantages” are not valid policy advantages. Therefore, the MDP and reinforcement-learning interpretation is not well supported.
Critical strengths
- The paper addresses the practical problem of allocating test-time generation compute.
- Reusing completed refinement trajectories is intuitive and potentially useful.
- The method does not require training the video generator.
- The overall pipeline is easy to implement.
Critical weaknesses
- The system ultimately returns the video with the highest quality observed during the episode. However, the reward is defined as . This penalizes temporary regressions even though such regressions do not affect the final delivered video. For example, suppose the initial quality is 5. A refinement first produces a video with quality 3 and then one with quality 6. The final output improves from 5 to 6. Nevertheless, the two rewards are , so the total return is negative, even before discounting. TTPE therefore considers a successful trajectory worse than immediately stopping.
- The paper estimates (Q(s,a)) by averaging the returns of retrieved historical states where action (a) was selected. But actions in the memory were not randomly assigned. They were selected by the base agent and earlier versions of TTPE. Therefore, different actions are observed in different types of states. For example, “reword” may appear successful simply because the base agent selects it for easier failures, while “resample” may appear unsuccessful because it is used in difficult cases. Subtracting the average return of all neighbors does not remove this selection bias.
- Equation (3) defines the state using the current prompt, scalar quality, best quality, and remaining budget. It does not include the current video, the individual failed requirements, or the refinement history. However, the effect of an action clearly depends on these missing variables. Two videos can have the same prompt and quality score while failing completely different requirements, requiring different refinement actions.
- Every non-stop action has exactly the same immediate cost: one video generation. The “cost return” instead measures how many future generations were used after that action in historical trajectories. This future cost depends mainly on the historical policy’s later stopping decisions. Since the policy evolves as memory grows, these historical costs do not estimate the future cost under the current policy.
Questions for authors
Equation (11) combines quality and cost using the remaining-budget fraction:
.
No optimization argument shows that this weighting solves the stated budget-constrained problem. The hard budget is already enforced by the episode horizon, so the connection between this rule and optimal budget allocation is unclear.
5 comments
Response to Reviewer d8gU (2/3)
W1. Negative Returns and Premature Stopping
Reward design motivation. In test-time generation, only improvements beyond the current best video yield a better output for the user, whereas recovering from regressions consumes budget without improving the delivered result. Our reward therefore credits actions that establish a new best quality. Since the delivered output is always the best video observed, temporary regressions never degrade the final result.
Credit is assigned per action, not per trajectory. The advantage of each action is computed from its return-to-go, i.e., the rewards from that action onward, so earlier regressions never penalize later improving actions. In the reviewer's example 5→3→6, the rewards are −2 and +1, and the returns-to-go are −1 and +1. The action that reaches 6 receives +1, exactly as if it had been reached directly from 5. Only the action that caused the regression receives negative credit, reflecting the extra generation spent on the detour. In practice such detours are usually far longer, making recovery expensive (Table C).
Table C. Regression-and-recovery statistics on StoryEval.
| Video Model | R&R Ratio (%) | Avg. Gen. to Recover 1 Point |
|---|---|---|
| Wan2.1 | 16.35 | 6.36 |
| HunyuanVideo | 10.87 | 4.89 |
| MiniMax-H3 | 4.02 | 2.63 |
Negative returns do not induce stopping.
(i) No within-episode effect. Transitions and their return-to-go values are committed to memory only after the episode terminates, and the retrieval snapshot excludes the current episode and prompt (Alg. 1, lines 4 and 15).
(ii) Memory only re-ranks the agent's candidates. All candidate actions, including stop, are proposed by the frozen agent based on its critique of the current video (Alg. 1, line 5). Retrieved trajectories only contribute normalized, clipped, and evidence-gated advantages (Eqs. 7–9); without supporting neighbors the advantages are zero and the agent's choice stands. Actions are compared by relative merit (Eqs. 10–11), not by the sign of any individual return.
(iii) Stopping before the goal under a limited budget. Stop is only selected when the frozen agent proposes it as a candidate. As the remaining budget decreases, Eq. 11 assigns greater weight to cost advantages, making early stopping more competitive when further refinement offers insufficient quality gains relative to its cost.
Result. Only 1.7%–4.6% of episodes stop with unmet elements (Table A), so premature stopping is rare in practice.
W2. Selection Bias in the Retrieval-Based Q Estimate
We agree that the actions in memory are not randomly assigned, and we do not claim that Eqs. 7–8 are unbiased causal estimates: as stated in Sec. 2.4, they are empirical predictions rather than values under a fixed policy. The failure mode you describe matters only if returns earned in easy states are credited to the same action in hard states; retrieval limits this transfer, as verified below.
Action values are estimated from the most similar states, not from all history. For the current failure state, TTPE first retrieves the most similar historical states. The retrieval document (Alg. 1, line 3) encodes the current score , the remaining budget , and the element-level feedback on the specific unmet elements (see the revised Eq. 3 in W3). It then estimates only from the neighbors whose action family matches (Eq. 7). Easy Reword cases that are dissimilar to the current failure therefore do not enter the Reword estimate for a hard state merely because they once achieved positive returns.
Empirical check on the two most frequent actions. We compare Reword and Switch_repr, the two most frequent actions (Fig. 5), in severe-failure states () where both were valid candidates (N = 482). The states in which each action was selected are comparable in prompt complexity, remaining budget, and step index, so neither action is reserved for easier cases. Their realized outcomes also show no significant difference on either the quality or the cost axis (Table D).
Table D. Reword vs. Switch_repr in zero-score decisions where both were valid (N = 482).
| Reword | Switch_repr | Difference [95% CI] | |
|---|---|---|---|
| State | |||
| Number of elements | 2.73 | 2.75 | −0.03 |
| Remaining budget | 0.67 | 0.68 | −0.01 |
| Step index | 4.00 | 3.89 | +0.11 |
| Realized outcome | |||
| Quality return | −1.34 | −1.32 | −0.02 [−0.55, 0.46] |
| Generation cost | 4.63 | 4.68 | −0.04 [−0.55, 0.43] |
| Immediate improvement rate | 31.4% | 31.8% | −0.4 |
The remaining bias has bounded influence. As noted in W1 (ii), the advantages are clipped and evidence-gated (Eq. 9) and only correct the base-policy prior (Eqs. 10–11), so when the retrieved evidence is weak, the frozen agent's own choice dominates.
Response to Reviewer d8gU (3/3)
W3. State Representation in Eq. 3
Eq. 3 is a compact notation, not a complete list of what the policy observes. As reflected in our Fig. 2 and code, the policy does receive the current video, the textual feedback on each element (which elements are unmet and why), and the textual history of previous refinements. We have revised Eq. 3 to
where is the current video, is the element-level textual feedback on , and is the textual history of previous refinement steps.
W4. Role of the Cost Return
The cost return is not intended as an unbiased estimate of future generation cost under the current policy; it is a relative signal for comparing candidate actions.
All non-stop actions share the same immediate cost of one generation, so they can only be distinguished by how many further generations they lead to. The cost return captures this from historical trajectories. Although it is affected by the later stopping decisions of the historical policy, this effect is shared by all candidates: their estimates are drawn from the same retrieved neighborhood, and the advantage (Eq. 8) subtracts the common state baseline. The comparison is therefore consistent across actions, even though the absolute values are not predictions under the evolving policy.
Empirically, this relative signal is informative. Fig. 3 shows that decisions with higher predicted cost incur larger realized generation costs on all three video models, and the cost advantage reduces generations from 6.30 to 4.47 at comparable completion (Table 3).
Summary
The new evidence addresses the concerns at the center of your review: the budget-conditioned weighting in Eq. 11 is supported by its experiments against fixed weights, the reward assigns credit per action and does not induce premature stopping, action values are estimated from the most similar states and show no selection bias between the most frequent actions, the state definition in Eq. 3 has been updated, and the cost return provides a consistent relative signal that is reflected in realized generation costs. The revised version includes these experiments and the clarifications above, with all changes highlighted in blue. Thank you again for the constructive assessment and specific requests. We hope the new results and clarifications support a stronger acceptance rating.
Response to Reviewer d8gU (1/3)
Thank you for recognizing that the problem is important, that reusing previous test-time experience is a sensible idea, and that our method is easy to implement. Your review raises five issues: the optimization justification of the budget-conditioned weighting (Eq. 11), negative returns and premature stopping, selection bias in the retrieval-based Q estimate, the state representation in Eq. 3, and the role of the cost return. We address them in that order.
Q. Optimization Justification of the Budget-Conditioned Weighting (Eq. 11)
We answer this question from two aspects: why a selection rule is needed for stopping even though the horizon enforces the hard budget, and an analysis of the weight ρ.
1. The horizon enforces feasibility, not allocation. The horizon only guarantees that no prompt exceeds B generations; any policy that runs until the horizon is feasible. Our objective is to reach high quality with as few generations as possible, so the key decision is when to stop, which the horizon does not determine. Without the cost term, the agent keeps refining and consumes most of the budget (Table 3). Table A reports how many episodes end by the stop action and how many end by the budget. On all three video models, about 70% of episodes end by the stop action, using only a fraction of the budget (e.g., 5.8 of 20 generations for Wan-2.1), and almost all of them stop after all prompt elements are satisfied; only 1.7%–4.6% stop with unmet elements.
Table A. Episodes ending by stop action vs. budget on StoryEval (423 prompts).
2. Parameter analysis: a budget-dependent ρt is necessary. Table B compares fixed and dynamic weights. ρ=1 (quality only) reaches similar completion but costs 6.30 generations per prompt; ρ=0 (cost only) reduces generations to 3.97 but drops to 41.17%. A fixed ρ=0.5 is worse on both axes (45.36%, 5.75 generations). Only the budget-conditioned ρt achieves both high completion (47.42%) and low cost (4.47).
Table B. Effect of the weight ρ in Eq. 11 (StoryEval).
We have added these results to the revision.