- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the computational behavior of differentiable token optimization. The authors find that token changes are concentrated in a small number of positions and early optimization steps, while longer optimization often hurts answer quality. Based on these observations, they explore gradient reuse, position-level activity prediction, and early stopping. Experiments on MATH-500 and GSM8K reveal similar patterns, with 15.8% fewer model calls at comparable accuracy on a 40-problem MATH-500 subset.
Critical strengths
- The paper offers an interesting analysis of token-level optimization dynamics, showing that computation is unevenly distributed across positions and optimization steps.
- The empirical findings could be useful, particularly the observation that additional gradient updates can reduce accuracy rather than improve it.
- The paper connects its observations to practical optimization strategies and provides ablations to measure their individual contributions.
Critical weaknesses
- The experiments are limited to a single reward-model-based objective with fixed hyperparameters. The paper does not test whether its findings hold under other optimization settings or DTO methods. In particular, the original ∇-Reasoner configuration and other DTO methods such as GCG are not evaluated.
- The practical benefit of the proposed acceleration is unclear. The optimized outputs perform worse than greedy decoding, and the proposed method saves only 15.8% of model calls. Simply using fewer optimization steps could achieve much larger savings with better accuracy, raising questions about the need for the proposed mechanisms.
- The efficiency evaluation is limited. Savings are measured in model calls rather than wall-clock time, and the joint evaluation covers only 40 problems. Most savings come from gradient reuse, while the other two components contribute relatively little.
- The theoretical analysis is not well connected to the experiments. Some results rely on assumptions that are not verified, and the experiments use a different optimizer from the one analyzed. Moreover, the stopping rule covered by Proposition 3 is not evaluated.
Questions for authors
- Would the findings still hold with different hyperparameters or DTO methods, especially the original ∇-Reasoner setup and GCG? It would be helpful to see results under these settings.
- Could the authors directly compare the proposed approach with a shorter fixed optimization budget in terms of accuracy and compute? This would help clarify when the proposed mechanisms are actually beneficial.
- Could the authors report actual wall-clock speedups and test the method on more problems?
- How well do the theoretical assumptions hold in practice? Also, why wasn't the stopping rule in Proposition 3 tested?
- Could the authors run larger no-cache experiments to check whether stale gradients affect the results?
Dear Reviewer,
With the author response deadline approaching, we wanted to ask whether you have any remaining questions or concerns. We’d be happy to clarify any points while the discussion is still open.
If our responses have addressed your concerns, we would appreciate it if you could let us know whether you would consider raising your score.
Thank you again for your time and thoughtful feedback.
Best regards,
The Authors